2024-05-09T02:55:38.941 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T02:55:38.942 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} --name pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035 --verbose --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/orig.config.yaml 2024-05-09T02:55:38.996 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T02:55:39.080 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035 branch: wip-pdonnell-testing-20240508.183908-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} email: pdonnell@redhat.com first_in_suite: false job_id: '7699035' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240508.183908-debug ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - Error loading MDS rank - missing journal object - Error recovering journal - error decoding table object - failed to read JournalPointer - Corrupt directory entry - Corrupt fnode header - corrupt sessionmap header - Corrupt dentry - Scrub error on inode - Metadata damage detected - MDS_READ_ONLY - force file system read-only - with standby daemon mds - MDS abort because newly corrupt dentry sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240508.183908-debug sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1722 sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240508.183908-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEta9nrY8jGuc+pKfcaGnqi3HrvC9EHyRvkS+7u6fZuMdS0ll8PIkLCuxU6DH1U67FNfRv+HPk3mvCfuiuaEG8= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK78pRSTR6xVAAyJ1HyTjjv1aLm1CxGM5wDjlp7VqkQB5BWRA61Sme2bpZ07flu3DgaPEmE5OQAYdvE+Zm8pe3U= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_damage teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_22:06:20 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T02:55:39.080 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa; will attempt to use it 2024-05-09T02:55:39.081 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa/tasks 2024-05-09T02:55:39.081 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T02:55:39.082 INFO:teuthology.task.internal:Checking packages... 2024-05-09T02:55:39.110 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c9da4049f638c0f84bdd9b4e33c42e86046ac554' 2024-05-09T02:55:39.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T02:55:39.110 INFO:teuthology.packaging:ref: None 2024-05-09T02:55:39.111 INFO:teuthology.packaging:tag: None 2024-05-09T02:55:39.111 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240508.183908-debug 2024-05-09T02:55:39.111 INFO:teuthology.packaging:sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T02:55:39.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240508.183908-debug 2024-05-09T02:55:39.267 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3594.gc9da4049 2024-05-09T02:55:39.268 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T02:55:39.279 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T02:55:39.279 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T02:55:39.338 INFO:teuthology.task.internal:Saving configuration 2024-05-09T02:55:39.356 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T02:55:39.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T02:55:39.473 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:49:30.694668', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEta9nrY8jGuc+pKfcaGnqi3HrvC9EHyRvkS+7u6fZuMdS0ll8PIkLCuxU6DH1U67FNfRv+HPk3mvCfuiuaEG8='} 2024-05-09T02:55:39.498 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 02:49:30.696391', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK78pRSTR6xVAAyJ1HyTjjv1aLm1CxGM5wDjlp7VqkQB5BWRA61Sme2bpZ07flu3DgaPEmE5OQAYdvE+Zm8pe3U='} 2024-05-09T02:55:39.498 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T02:55:39.591 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-05-09T02:55:39.591 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-05-09T02:55:39.591 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T02:55:39.749 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f69b2c0df70>, signals=[15]) 2024-05-09T02:55:39.749 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T02:55:39.815 INFO:teuthology.task.internal:Opening connections... 2024-05-09T02:55:39.815 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T02:55:39.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:55:39.895 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T02:55:39.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:55:39.969 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T02:55:39.978 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-05-09T02:55:39.996 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-05-09T02:55:39.996 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-05-09T02:55:40.054 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-05-09T02:55:40.054 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-05-09T02:55:40.054 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T02:55:40.055 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T02:55:40.056 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-05-09T02:55:40.085 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-05-09T02:55:40.103 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-05-09T02:55:40.104 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-05-09T02:55:40.161 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T02:55:40.162 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T02:55:40.162 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T02:55:40.162 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-05-09T02:55:40.185 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T02:55:40.196 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T02:55:40.263 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T02:55:40.263 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-05-09T02:55:40.265 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-05-09T02:55:40.282 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T02:55:40.345 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T02:55:40.345 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-05-09T02:55:40.348 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-05-09T02:55:40.370 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T02:55:40.472 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T02:55:40.513 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-09T02:55:40.513 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-09T02:55:40.513 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T02:55:40.514 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T02:55:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:55:40.531 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-05-09T02:55:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:55:40.532 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-05-09T02:55:40.589 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-437.el9.x86_64 2024-05-09T02:55:40.589 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-437.el9.x86_64 2024-05-09T02:55:40.589 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-05-09T02:55:40.590 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-437.el9.x86_64 2024-05-09T02:55:40.591 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-437.el9.x86_64 2024-05-09T02:55:40.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-05-09T02:55:42.794 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-05-09T02:55:42.980 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-09T02:55:44.778 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-05-09T02:55:44.831 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-05-09T02:55:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-05-09T02:55:50.899 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-05-09T02:55:51.769 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.2 MB 00:03 2024-05-09T02:55:52.111 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-05-09T02:55:53.187 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-05-09T02:55:55.238 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-05-09T02:55:57.534 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-05-09T02:55:59.646 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T02:55:59.646 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T02:55:59.731 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T02:55:59.741 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 78 M 2024-05-09T02:55:59.742 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 123 M 2024-05-09T02:55:59.743 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-05-09T02:55:59.985 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 2.6 MB/s | 484 kB 00:00 2024-05-09T02:56:00.438 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-05-09T02:56:01.269 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 11 MB/s | 15 MB 00:01 2024-05-09T02:56:01.736 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 18 MB/s | 34 MB 00:01 2024-05-09T02:56:02.069 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 13 MB/s | 28 MB 00:02 2024-05-09T02:56:02.070 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:56:02.070 INFO:teuthology.orchestra.run.smithi046.stdout:Total 33 MB/s | 78 MB 00:02 2024-05-09T02:56:02.469 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T02:56:02.469 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T02:56:02.564 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-09T02:56:02.576 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-09T02:56:02.576 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-05-09T02:56:02.576 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-05-09T02:56:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T02:56:02.578 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 78 M 2024-05-09T02:56:02.578 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 123 M 2024-05-09T02:56:02.578 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-09T02:56:02.780 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-05-09T02:56:03.175 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-05-09T02:56:03.175 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-05-09T02:56:03.380 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 632 kB/s | 484 kB 00:00 2024-05-09T02:56:04.136 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-05-09T02:56:04.137 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-05-09T02:56:06.683 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-05-09T02:56:07.322 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:56:07.411 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:56:10.212 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:56:10.363 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:56:15.440 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:56:15.481 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:19.403 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 941 kB/s | 15 MB 00:16 2024-05-09T02:56:20.336 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 1.6 MB/s | 28 MB 00:16 2024-05-09T02:56:20.576 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:42.724 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 866 kB/s | 34 MB 00:40 2024-05-09T02:56:42.724 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-09T02:56:42.724 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.9 MB/s | 78 MB 00:40 2024-05-09T02:56:43.402 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-09T02:56:43.742 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-09T02:56:43.742 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-09T02:56:44.668 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-09T02:56:44.669 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-09T02:56:47.071 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-09T02:56:47.683 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:56:47.765 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:56:50.434 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:56:50.575 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:56:51.451 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:51.490 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:51.792 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:51.793 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:56:51.793 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:56:51.793 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T02:56:52.621 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-05-09T02:56:52.945 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:56:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:56:53.032 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:56:53.033 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T02:56:53.033 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T02:56:53.033 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T02:56:53.033 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-09T02:56:53.033 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T02:56:53.034 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:56:53.084 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:56:53.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:56:53.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-05-09T02:56:53.116 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-05-09T02:56:53.116 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-05-09T02:56:53.116 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-05-09T02:56:53.116 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T02:56:53.117 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T02:56:53.118 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T02:56:53.119 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-05-09T02:56:53.119 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T02:56:53.119 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T02:56:53.120 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T02:56:53.120 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-444.el9.x86_64 2024-05-09T02:56:53.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T02:56:53.715 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-05-09T02:56:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T02:56:55.105 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-05-09T02:56:55.108 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-05-09T02:56:55.125 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.Z77nJmQBT3 2024-05-09T02:56:55.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z77nJmQBT3 2024-05-09T02:56:55.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.Z77nJmQBT3 2024-05-09T02:56:55.291 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:56:55.331 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.Z77nJmQBT3 is 6KB 2024-05-09T02:56:55.380 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.Z77nJmQBT3 2024-05-09T02:56:55.393 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:56:55.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T02:56:55.462 INFO:teuthology.orchestra.run.smithi046.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T02:56:55.462 INFO:teuthology.orchestra.run.smithi046.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T02:56:55.462 INFO:teuthology.orchestra.run.smithi046.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T02:56:55.462 INFO:teuthology.orchestra.run.smithi046.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T02:56:55.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T02:56:55.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-05-09T02:57:00.157 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:57:25.611 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-09T02:57:25.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-05-09T02:57:25.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:57:29.516 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:57:29.557 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:57:29.928 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:57:29.928 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T02:57:29.928 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T02:57:29.928 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T02:57:30.706 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-09T02:57:31.079 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:57:31.167 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:57:31.167 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:57:31.167 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T02:57:31.168 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T02:57:31.168 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T02:57:31.168 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-09T02:57:31.168 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T02:57:31.168 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T02:57:31.217 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T02:57:31.218 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T02:57:31.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-05-09T02:57:31.248 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-05-09T02:57:31.248 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 68.el9 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8264254 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-05-09T02:57:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T02:57:31.250 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T02:57:31.251 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T02:57:31.251 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-444.el9.x86_64 2024-05-09T02:57:31.251 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T02:57:31.888 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-05-09T02:57:33.329 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T02:57:33.358 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-05-09T02:57:33.361 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-05-09T02:57:33.382 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.rOL3wKLGAs 2024-05-09T02:57:33.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rOL3wKLGAs 2024-05-09T02:57:33.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.rOL3wKLGAs 2024-05-09T02:57:33.594 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.rOL3wKLGAs is 6KB 2024-05-09T02:57:33.621 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.rOL3wKLGAs 2024-05-09T02:57:33.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T02:57:33.701 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T02:57:33.701 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T02:57:33.701 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T02:57:33.701 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T02:57:33.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T02:57:33.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-05-09T02:58:00.095 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-05-09T02:58:03.866 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-09T02:58:03.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T02:58:03.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:58:09.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-05-09T02:58:09.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:58:38.495 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-09T02:58:47.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T02:58:47.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:58:50.562 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-09T02:59:02.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T02:59:02.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:09.109 DEBUG:teuthology.orchestra.remote:timed out 2024-05-09T02:59:20.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-05-09T02:59:21.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-05-09T02:59:21.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:21.379 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-05-09T02:59:21.729 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-05-09T02:59:21.729 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-09T02:59:21.730 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-05-09T02:59:21.747 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-444.el9.x86_64 2024-05-09T02:59:21.747 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T02:59:21.747 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T02:59:21.748 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T02:59:22.749 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-09T02:59:22.749 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T02:59:22.823 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-05-09T02:59:22.857 DEBUG:teuthology.parallel:result is None 2024-05-09T02:59:35.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T02:59:35.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T02:59:36.305 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-05-09T02:59:36.670 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T02:59:36.685 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-444.el9.x86_64"... 2024-05-09T02:59:36.685 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-05-09T02:59:36.700 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-444.el9.x86_64 2024-05-09T02:59:36.700 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T02:59:36.700 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T02:59:36.700 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T02:59:37.701 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-09T02:59:37.701 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T02:59:37.760 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-05-09T02:59:37.790 DEBUG:teuthology.parallel:result is None 2024-05-09T02:59:37.791 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T02:59:37.802 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T02:59:37.802 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T02:59:37.805 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T02:59:37.822 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T02:59:37.851 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T02:59:37.863 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T02:59:37.863 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T02:59:37.866 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T02:59:37.905 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T02:59:37.913 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T02:59:37.913 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T02:59:37.942 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T02:59:37.977 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:59:37.982 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:59:37.989 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:59:37.993 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T02:59:37.995 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T02:59:38.003 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T02:59:38.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T02:59:38.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T02:59:38.063 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T02:59:38.073 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T02:59:38.073 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T02:59:38.106 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T02:59:38.122 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:59:38.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:59:38.257 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:59:38.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:59:38.384 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T02:59:38.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T02:59:38.451 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:59:38.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T02:59:38.560 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:59:38.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T02:59:38.639 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T02:59:38.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T02:59:38.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-05-09T02:59:38.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-05-09T02:59:38.738 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T02:59:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T02:59:39.261 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T02:59:39.270 INFO:teuthology.task.internal:Starting timer... 2024-05-09T02:59:39.270 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T02:59:39.283 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T02:59:39.291 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-09T02:59:39.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-05-09T02:59:39.341 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-05-09T02:59:39.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-05-09T02:59:39.440 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-05-09T02:59:39.442 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T02:59:39.442 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-05-09T02:59:39.479 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-05-09T02:59:39.480 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-05-09T02:59:39.500 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-05-09T02:59:39.500 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive'} 2024-05-09T02:59:39.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T02:59:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:59:39.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T02:59:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T02:59:39.573 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T02:59:39.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-05-09T02:59:39.618 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-05-09T02:59:39.643 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T02:59:39.653 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-09T02:59:39.658 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-09T02:59:39.779 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T02:59:39.795 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T02:59:39.796 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T03:03:41.718 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com')] 2024-05-09T03:03:41.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-05-09T03:03:41.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T03:03:41.801 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-05-09T03:03:41.892 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-05-09T03:03:41.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T03:03:41.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T03:03:41.966 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-05-09T03:03:42.045 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-05-09T03:03:42.045 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T03:03:42.055 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T03:03:42.056 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T03:03:42.056 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:03:42.058 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T03:03:42.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:03:42.089 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T03:03:42.105 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T03:03:42.115 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T03:03:42.129 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T03:03:42.132 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-05-09T03:03:42.144 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-05-09T03:03:42.153 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-05-09T03:03:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T03:03:42.164 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-05-09T03:03:42.174 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T03:03:42.178 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T03:03:42.192 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T03:03:42.229 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:03:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:03:42.232 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-05-09T03:03:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.244 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:03:42.248 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:03:42.248 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-05-09T03:03:42.248 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.248 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.248 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.248 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:03:42.249 INFO:teuthology.run_tasks:Running task install... 2024-05-09T03:03:42.258 DEBUG:teuthology.task.install:project ceph 2024-05-09T03:03:42.258 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}} 2024-05-09T03:03:42.258 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'} 2024-05-09T03:03:42.258 INFO:teuthology.task.install:Using flavor: default 2024-05-09T03:03:42.264 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T03:03:42.264 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T03:03:42.264 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'tag': None, 'wait_for_package': False} 2024-05-09T03:03:42.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T03:03:42.267 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'tag': None, 'wait_for_package': False} 2024-05-09T03:03:42.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T03:03:42.413 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/ 2024-05-09T03:03:42.414 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3594.gc9da4049 2024-05-09T03:03:42.419 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/ 2024-05-09T03:03:42.420 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3594.gc9da4049 2024-05-09T03:03:42.526 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T03:03:42.526 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:03:42.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T03:03:42.532 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T03:03:42.532 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:03:42.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T03:03:42.553 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T03:03:42.553 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c9da4049f638c0f84bdd9b4e33c42e86046ac554/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T03:03:42.558 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T03:03:42.558 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c9da4049f638c0f84bdd9b4e33c42e86046ac554/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T03:03:42.618 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T03:03:42.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T03:03:42.692 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T03:03:42.701 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T03:03:42.756 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-05-09T03:03:42.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-05-09T03:03:42.765 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-05-09T03:03:42.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-05-09T03:03:42.994 INFO:teuthology.orchestra.run.smithi088.stdout:54 files removed 2024-05-09T03:03:43.007 INFO:teuthology.orchestra.run.smithi046.stdout:54 files removed 2024-05-09T03:03:43.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T03:03:43.034 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T03:03:43.641 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 262 kB/s | 84 kB 00:00 2024-05-09T03:03:43.811 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 162 kB/s | 84 kB 00:00 2024-05-09T03:03:44.108 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-05-09T03:03:44.195 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-05-09T03:03:44.517 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-05-09T03:03:44.612 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-05-09T03:03:45.495 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.1 MB 00:00 2024-05-09T03:03:46.384 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-05-09T03:03:47.346 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-05-09T03:03:49.294 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-09T03:03:51.422 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-09T03:03:53.536 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-05-09T03:03:55.020 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-05-09T03:03:56.030 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-05-09T03:04:01.331 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-09T03:04:03.751 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T03:04:03.751 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T03:04:03.761 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T03:04:03.762 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T03:04:03.763 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T03:04:03.764 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T03:04:03.766 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T03:04:03.768 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T03:04:03.769 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T03:04:03.833 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 6.6 k 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.4 M 2024-05-09T03:04:03.851 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 863 k 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 147 k 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 1.5 M 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 142 k 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 3.5 M 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 7.4 M 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 49 k 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 10 M 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 43 M 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 273 k 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 763 k 2024-05-09T03:04:03.852 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 224 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 27 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 18 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 32 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 756 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 127 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T03:04:03.853 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 162 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 321 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 301 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 99 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 86 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.0 M 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 169 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T03:04:03.854 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.3 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.1 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 20 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 34 k 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 2.3 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 274 k 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 4.9 M 2024-05-09T03:04:03.855 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 18 M 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 16 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 25 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 158 k 2024-05-09T03:04:03.856 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 497 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.2 M 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T03:04:03.857 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T03:04:03.858 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 45 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 135 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T03:04:03.859 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T03:04:03.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T03:04:03.861 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T03:04:03.862 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T03:04:03.863 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout:Install 148 Packages 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 8 Packages 2024-05-09T03:04:03.864 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:04:03.865 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 219 M 2024-05-09T03:04:03.865 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-05-09T03:04:04.432 INFO:teuthology.orchestra.run.smithi046.stdout:(1/156): ceph-19.0.0-3594.gc9da4049.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-05-09T03:04:05.008 INFO:teuthology.orchestra.run.smithi046.stdout:(2/156): ceph-fuse-19.0.0-3594.gc9da4049.el9.x8 1.5 MB/s | 863 kB 00:00 2024-05-09T03:04:05.166 INFO:teuthology.orchestra.run.smithi046.stdout:(3/156): ceph-immutable-object-cache-19.0.0-359 927 kB/s | 147 kB 00:00 2024-05-09T03:04:05.533 INFO:teuthology.orchestra.run.smithi046.stdout:(4/156): ceph-common-19.0.0-3594.gc9da4049.el9. 15 MB/s | 20 MB 00:01 2024-05-09T03:04:05.608 INFO:teuthology.orchestra.run.smithi046.stdout:(5/156): ceph-base-19.0.0-3594.gc9da4049.el9.x8 3.9 MB/s | 5.4 MB 00:01 2024-05-09T03:04:05.650 INFO:teuthology.orchestra.run.smithi046.stdout:(6/156): ceph-mgr-19.0.0-3594.gc9da4049.el9.x86 13 MB/s | 1.5 MB 00:00 2024-05-09T03:04:06.100 INFO:teuthology.orchestra.run.smithi046.stdout:(7/156): ceph-mds-19.0.0-3594.gc9da4049.el9.x86 2.4 MB/s | 2.3 MB 00:00 2024-05-09T03:04:06.434 INFO:teuthology.orchestra.run.smithi046.stdout:(8/156): ceph-mon-19.0.0-3594.gc9da4049.el9.x86 5.9 MB/s | 4.9 MB 00:00 2024-05-09T03:04:06.518 INFO:teuthology.orchestra.run.smithi046.stdout:(9/156): ceph-selinux-19.0.0-3594.gc9da4049.el9 302 kB/s | 25 kB 00:00 2024-05-09T03:04:06.693 INFO:teuthology.orchestra.run.smithi046.stdout:(10/156): ceph-osd-19.0.0-3594.gc9da4049.el9.x8 17 MB/s | 18 MB 00:01 2024-05-09T03:04:06.743 INFO:teuthology.orchestra.run.smithi046.stdout:(11/156): cephfs-mirror-19.0.0-3594.gc9da4049.e 4.4 MB/s | 224 kB 00:00 2024-05-09T03:04:06.793 INFO:teuthology.orchestra.run.smithi046.stdout:(12/156): cephfs-shell-19.0.0-3594.gc9da4049.el 544 kB/s | 27 kB 00:00 2024-05-09T03:04:06.852 INFO:teuthology.orchestra.run.smithi046.stdout:(13/156): libcephfs-devel-19.0.0-3594.gc9da4049 543 kB/s | 32 kB 00:00 2024-05-09T03:04:06.944 INFO:teuthology.orchestra.run.smithi046.stdout:(14/156): libcephfs2-19.0.0-3594.gc9da4049.el9. 8.1 MB/s | 756 kB 00:00 2024-05-09T03:04:07.003 INFO:teuthology.orchestra.run.smithi046.stdout:(15/156): libcephsqlite-19.0.0-3594.gc9da4049.e 2.6 MB/s | 158 kB 00:00 2024-05-09T03:04:07.061 INFO:teuthology.orchestra.run.smithi046.stdout:(16/156): librados-devel-19.0.0-3594.gc9da4049. 2.1 MB/s | 127 kB 00:00 2024-05-09T03:04:07.128 INFO:teuthology.orchestra.run.smithi046.stdout:(17/156): libradosstriper1-19.0.0-3594.gc9da404 7.3 MB/s | 497 kB 00:00 2024-05-09T03:04:07.528 INFO:teuthology.orchestra.run.smithi046.stdout:(18/156): librgw2-19.0.0-3594.gc9da4049.el9.x86 13 MB/s | 5.2 MB 00:00 2024-05-09T03:04:07.578 INFO:teuthology.orchestra.run.smithi046.stdout:(19/156): python3-ceph-argparse-19.0.0-3594.gc9 908 kB/s | 45 kB 00:00 2024-05-09T03:04:07.687 INFO:teuthology.orchestra.run.smithi046.stdout:(20/156): ceph-radosgw-19.0.0-3594.gc9da4049.el 6.4 MB/s | 10 MB 00:01 2024-05-09T03:04:07.712 INFO:teuthology.orchestra.run.smithi046.stdout:(21/156): python3-ceph-common-19.0.0-3594.gc9da 1.0 MB/s | 135 kB 00:00 2024-05-09T03:04:07.754 INFO:teuthology.orchestra.run.smithi046.stdout:(22/156): python3-cephfs-19.0.0-3594.gc9da4049. 2.4 MB/s | 162 kB 00:00 2024-05-09T03:04:07.780 INFO:teuthology.orchestra.run.smithi046.stdout:(23/156): python3-rados-19.0.0-3594.gc9da4049.e 4.7 MB/s | 321 kB 00:00 2024-05-09T03:04:07.838 INFO:teuthology.orchestra.run.smithi046.stdout:(24/156): python3-rbd-19.0.0-3594.gc9da4049.el9 3.5 MB/s | 301 kB 00:00 2024-05-09T03:04:07.863 INFO:teuthology.orchestra.run.smithi046.stdout:(25/156): python3-rgw-19.0.0-3594.gc9da4049.el9 1.2 MB/s | 99 kB 00:00 2024-05-09T03:04:07.905 INFO:teuthology.orchestra.run.smithi046.stdout:(26/156): rbd-fuse-19.0.0-3594.gc9da4049.el9.x8 1.3 MB/s | 86 kB 00:00 2024-05-09T03:04:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:(27/156): rbd-nbd-19.0.0-3594.gc9da4049.el9.x86 1.6 MB/s | 169 kB 00:00 2024-05-09T03:04:08.081 INFO:teuthology.orchestra.run.smithi046.stdout:(28/156): rbd-mirror-19.0.0-3594.gc9da4049.el9. 14 MB/s | 3.0 MB 00:00 2024-05-09T03:04:08.115 INFO:teuthology.orchestra.run.smithi046.stdout:(29/156): ceph-grafana-dashboards-19.0.0-3594.g 314 kB/s | 34 kB 00:00 2024-05-09T03:04:08.221 INFO:teuthology.orchestra.run.smithi046.stdout:(30/156): ceph-mgr-cephadm-19.0.0-3594.gc9da404 1.0 MB/s | 142 kB 00:00 2024-05-09T03:04:08.541 INFO:teuthology.orchestra.run.smithi046.stdout:(31/156): ceph-mgr-dashboard-19.0.0-3594.gc9da4 8.3 MB/s | 3.5 MB 00:00 2024-05-09T03:04:08.649 INFO:teuthology.orchestra.run.smithi046.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3594.gc9 2.5 MB/s | 274 kB 00:00 2024-05-09T03:04:08.791 INFO:teuthology.orchestra.run.smithi046.stdout:(33/156): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-05-09T03:04:08.816 INFO:teuthology.orchestra.run.smithi046.stdout:(34/156): ceph-mgr-rook-19.0.0-3594.gc9da4049.e 295 kB/s | 49 kB 00:00 2024-05-09T03:04:08.833 INFO:teuthology.orchestra.run.smithi046.stdout:(35/156): ceph-prometheus-alerts-19.0.0-3594.gc 394 kB/s | 16 kB 00:00 2024-05-09T03:04:08.884 INFO:teuthology.orchestra.run.smithi046.stdout:(36/156): ceph-volume-19.0.0-3594.gc9da4049.el9 4.0 MB/s | 273 kB 00:00 2024-05-09T03:04:08.917 INFO:teuthology.orchestra.run.smithi046.stdout:(37/156): cephadm-19.0.0-3594.gc9da4049.el9.noa 8.9 MB/s | 763 kB 00:00 2024-05-09T03:04:08.951 INFO:teuthology.orchestra.run.smithi046.stdout:(38/156): cephfs-top-19.0.0-3594.gc9da4049.el9. 273 kB/s | 18 kB 00:00 2024-05-09T03:04:09.618 INFO:teuthology.orchestra.run.smithi046.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 115 kB/s | 80 kB 00:00 2024-05-09T03:04:09.926 INFO:teuthology.orchestra.run.smithi046.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 823 kB/s | 803 kB 00:00 2024-05-09T03:04:10.752 INFO:teuthology.orchestra.run.smithi046.stdout:(41/156): ceph-test-19.0.0-3594.gc9da4049.el9.x 10 MB/s | 43 MB 00:04 2024-05-09T03:04:10.777 INFO:teuthology.orchestra.run.smithi046.stdout:(42/156): libquadmath-11.4.1-3.el9.x86_64.rpm 169 kB/s | 195 kB 00:01 2024-05-09T03:04:11.144 INFO:teuthology.orchestra.run.smithi046.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 208 kB/s | 253 kB 00:01 2024-05-09T03:04:11.494 INFO:teuthology.orchestra.run.smithi046.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 149 kB/s | 106 kB 00:00 2024-05-09T03:04:11.778 INFO:teuthology.orchestra.run.smithi046.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:01 2024-05-09T03:04:11.911 INFO:teuthology.orchestra.run.smithi046.stdout:(46/156): python3-pycparser-2.20-6.el9.noarch.r 176 kB/s | 135 kB 00:00 2024-05-09T03:04:12.236 INFO:teuthology.orchestra.run.smithi046.stdout:(47/156): python3-requests-2.25.1-8.el9.noarch. 169 kB/s | 125 kB 00:00 2024-05-09T03:04:12.570 INFO:teuthology.orchestra.run.smithi046.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 271 kB/s | 215 kB 00:00 2024-05-09T03:04:12.662 INFO:teuthology.orchestra.run.smithi046.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 919 kB/s | 689 kB 00:00 2024-05-09T03:04:12.762 INFO:teuthology.orchestra.run.smithi046.stdout:(50/156): elfutils-libelf-devel-0.190-2.el9.x86 215 kB/s | 21 kB 00:00 2024-05-09T03:04:12.871 INFO:teuthology.orchestra.run.smithi046.stdout:(51/156): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-05-09T03:04:12.988 INFO:teuthology.orchestra.run.smithi046.stdout:(52/156): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-05-09T03:04:13.013 INFO:teuthology.orchestra.run.smithi046.stdout:(53/156): boost-program-options-1.75.0-8.el9.x8 241 kB/s | 107 kB 00:00 2024-05-09T03:04:13.038 INFO:teuthology.orchestra.run.smithi046.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 196 kB/s | 33 kB 00:00 2024-05-09T03:04:13.114 INFO:teuthology.orchestra.run.smithi046.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-05-09T03:04:13.221 INFO:teuthology.orchestra.run.smithi046.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-05-09T03:04:13.221 INFO:teuthology.orchestra.run.smithi046.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-05-09T03:04:13.256 INFO:teuthology.orchestra.run.smithi046.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-05-09T03:04:13.400 INFO:teuthology.orchestra.run.smithi046.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-05-09T03:04:13.507 INFO:teuthology.orchestra.run.smithi046.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 235 kB/s | 25 kB 00:00 2024-05-09T03:04:13.607 INFO:teuthology.orchestra.run.smithi046.stdout:(61/156): libtool-2.4.6-45.el9.x86_64.rpm 1.6 MB/s | 582 kB 00:00 2024-05-09T03:04:13.632 INFO:teuthology.orchestra.run.smithi046.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 378 kB/s | 47 kB 00:00 2024-05-09T03:04:13.716 INFO:teuthology.orchestra.run.smithi046.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-09T03:04:13.741 INFO:teuthology.orchestra.run.smithi046.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-09T03:04:13.816 INFO:teuthology.orchestra.run.smithi046.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 352 kB/s | 35 kB 00:00 2024-05-09T03:04:14.284 INFO:teuthology.orchestra.run.smithi046.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-05-09T03:04:15.300 INFO:teuthology.orchestra.run.smithi046.stdout:(67/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-05-09T03:04:16.434 INFO:teuthology.orchestra.run.smithi046.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 756 kB/s | 1.6 MB 00:02 2024-05-09T03:04:16.542 INFO:teuthology.orchestra.run.smithi046.stdout:(69/156): postgresql-contrib-13.14-1.el9.x86_64 734 kB/s | 911 kB 00:01 2024-05-09T03:04:16.609 INFO:teuthology.orchestra.run.smithi046.stdout:(70/156): openblas-openmp-0.3.21-2.el9.x86_64.r 1.7 MB/s | 4.7 MB 00:02 2024-05-09T03:04:16.709 INFO:teuthology.orchestra.run.smithi046.stdout:(71/156): postgresql-private-libs-13.14-1.el9.x 484 kB/s | 133 kB 00:00 2024-05-09T03:04:16.893 INFO:teuthology.orchestra.run.smithi046.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 477 kB/s | 87 kB 00:00 2024-05-09T03:04:17.977 INFO:teuthology.orchestra.run.smithi046.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 756 kB/s | 1.0 MB 00:01 2024-05-09T03:04:18.318 INFO:teuthology.orchestra.run.smithi046.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 715 kB/s | 244 kB 00:00 2024-05-09T03:04:18.660 INFO:teuthology.orchestra.run.smithi046.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 725 kB/s | 248 kB 00:00 2024-05-09T03:04:18.761 INFO:teuthology.orchestra.run.smithi046.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-05-09T03:04:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-05-09T03:04:19.120 INFO:teuthology.orchestra.run.smithi046.stdout:(78/156): python3-mako-1.1.4-6.el9.noarch.rpm 666 kB/s | 172 kB 00:00 2024-05-09T03:04:19.220 INFO:teuthology.orchestra.run.smithi046.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 348 kB/s | 35 kB 00:00 2024-05-09T03:04:23.854 INFO:teuthology.orchestra.run.smithi046.stdout:(80/156): python3-babel-2.9.1-2.el9.noarch.rpm 877 kB/s | 6.0 MB 00:06 2024-05-09T03:04:23.962 INFO:teuthology.orchestra.run.smithi046.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-05-09T03:04:24.245 INFO:teuthology.orchestra.run.smithi046.stdout:(82/156): postgresql-server-13.14-1.el9.x86_64. 768 kB/s | 5.8 MB 00:07 2024-05-09T03:04:24.379 INFO:teuthology.orchestra.run.smithi046.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 533 kB/s | 222 kB 00:00 2024-05-09T03:04:24.479 INFO:teuthology.orchestra.run.smithi046.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 330 kB/s | 77 kB 00:00 2024-05-09T03:04:24.504 INFO:teuthology.orchestra.run.smithi046.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 338 kB/s | 42 kB 00:00 2024-05-09T03:04:24.613 INFO:teuthology.orchestra.run.smithi046.stdout:(86/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-05-09T03:04:24.905 INFO:teuthology.orchestra.run.smithi046.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 633 kB/s | 269 kB 00:00 2024-05-09T03:04:25.014 INFO:teuthology.orchestra.run.smithi046.stdout:(88/156): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-05-09T03:04:25.039 INFO:teuthology.orchestra.run.smithi046.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 655 kB/s | 279 kB 00:00 2024-05-09T03:04:25.114 INFO:teuthology.orchestra.run.smithi046.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 536 kB/s | 54 kB 00:00 2024-05-09T03:04:25.214 INFO:teuthology.orchestra.run.smithi046.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-05-09T03:04:25.315 INFO:teuthology.orchestra.run.smithi046.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-05-09T03:04:25.398 INFO:teuthology.orchestra.run.smithi046.stdout:(93/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 845 kB/s | 5.1 MB 00:06 2024-05-09T03:04:25.498 INFO:teuthology.orchestra.run.smithi046.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 359 kB/s | 36 kB 00:00 2024-05-09T03:04:25.599 INFO:teuthology.orchestra.run.smithi046.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-05-09T03:04:25.658 INFO:teuthology.orchestra.run.smithi046.stdout:(96/156): qatlib-24.02.0-1.el9.x86_64.rpm 644 kB/s | 221 kB 00:00 2024-05-09T03:04:25.758 INFO:teuthology.orchestra.run.smithi046.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 579 kB/s | 58 kB 00:00 2024-05-09T03:04:25.941 INFO:teuthology.orchestra.run.smithi046.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 511 kB/s | 94 kB 00:00 2024-05-09T03:04:26.025 INFO:teuthology.orchestra.run.smithi046.stdout:(99/156): socat-1.7.4.1-5.el9.x86_64.rpm 716 kB/s | 305 kB 00:00 2024-05-09T03:04:26.133 INFO:teuthology.orchestra.run.smithi046.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 332 kB/s | 64 kB 00:00 2024-05-09T03:04:26.159 INFO:teuthology.orchestra.run.smithi046.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 335 kB/s | 45 kB 00:00 2024-05-09T03:04:26.709 INFO:teuthology.orchestra.run.smithi046.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-05-09T03:04:26.993 INFO:teuthology.orchestra.run.smithi046.stdout:(103/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-05-09T03:04:27.068 INFO:teuthology.orchestra.run.smithi046.stdout:(104/156): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-05-09T03:04:27.101 INFO:teuthology.orchestra.run.smithi046.stdout:(105/156): protobuf-compiler-3.14.0-13.el9.x86_ 916 kB/s | 863 kB 00:00 2024-05-09T03:04:27.118 INFO:teuthology.orchestra.run.smithi046.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 386 kB/s | 19 kB 00:00 2024-05-09T03:04:27.169 INFO:teuthology.orchestra.run.smithi046.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-05-09T03:04:27.219 INFO:teuthology.orchestra.run.smithi046.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 975 kB/s | 49 kB 00:00 2024-05-09T03:04:27.278 INFO:teuthology.orchestra.run.smithi046.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-05-09T03:04:27.361 INFO:teuthology.orchestra.run.smithi046.stdout:(110/156): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-05-09T03:04:27.462 INFO:teuthology.orchestra.run.smithi046.stdout:(111/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-05-09T03:04:27.537 INFO:teuthology.orchestra.run.smithi046.stdout:(112/156): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-05-09T03:04:27.562 INFO:teuthology.orchestra.run.smithi046.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-05-09T03:04:27.596 INFO:teuthology.orchestra.run.smithi046.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-05-09T03:04:27.621 INFO:teuthology.orchestra.run.smithi046.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-05-09T03:04:27.654 INFO:teuthology.orchestra.run.smithi046.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-05-09T03:04:27.680 INFO:teuthology.orchestra.run.smithi046.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.8 MB/s | 172 kB 00:00 2024-05-09T03:04:27.713 INFO:teuthology.orchestra.run.smithi046.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-05-09T03:04:27.747 INFO:teuthology.orchestra.run.smithi046.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-05-09T03:04:27.772 INFO:teuthology.orchestra.run.smithi046.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-05-09T03:04:27.798 INFO:teuthology.orchestra.run.smithi046.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-09T03:04:27.873 INFO:teuthology.orchestra.run.smithi046.stdout:(122/156): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-05-09T03:04:27.898 INFO:teuthology.orchestra.run.smithi046.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-09T03:04:27.932 INFO:teuthology.orchestra.run.smithi046.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-05-09T03:04:27.957 INFO:teuthology.orchestra.run.smithi046.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-05-09T03:04:27.991 INFO:teuthology.orchestra.run.smithi046.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-05-09T03:04:28.016 INFO:teuthology.orchestra.run.smithi046.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-05-09T03:04:28.049 INFO:teuthology.orchestra.run.smithi046.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-05-09T03:04:28.100 INFO:teuthology.orchestra.run.smithi046.stdout:(129/156): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-05-09T03:04:28.125 INFO:teuthology.orchestra.run.smithi046.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-05-09T03:04:28.159 INFO:teuthology.orchestra.run.smithi046.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-05-09T03:04:28.184 INFO:teuthology.orchestra.run.smithi046.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 983 kB/s | 58 kB 00:00 2024-05-09T03:04:28.217 INFO:teuthology.orchestra.run.smithi046.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-05-09T03:04:28.243 INFO:teuthology.orchestra.run.smithi046.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-05-09T03:04:28.268 INFO:teuthology.orchestra.run.smithi046.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-05-09T03:04:28.302 INFO:teuthology.orchestra.run.smithi046.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-05-09T03:04:28.327 INFO:teuthology.orchestra.run.smithi046.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-05-09T03:04:28.360 INFO:teuthology.orchestra.run.smithi046.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-05-09T03:04:28.386 INFO:teuthology.orchestra.run.smithi046.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-05-09T03:04:28.419 INFO:teuthology.orchestra.run.smithi046.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-05-09T03:04:28.445 INFO:teuthology.orchestra.run.smithi046.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 873 kB/s | 51 kB 00:00 2024-05-09T03:04:28.478 INFO:teuthology.orchestra.run.smithi046.stdout:(142/156): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-05-09T03:04:28.503 INFO:teuthology.orchestra.run.smithi046.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-05-09T03:04:28.537 INFO:teuthology.orchestra.run.smithi046.stdout:(144/156): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-05-09T03:04:28.562 INFO:teuthology.orchestra.run.smithi046.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-05-09T03:04:28.596 INFO:teuthology.orchestra.run.smithi046.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-09T03:04:28.722 INFO:teuthology.orchestra.run.smithi046.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 9.9 MB/s | 1.6 MB 00:00 2024-05-09T03:04:29.022 INFO:teuthology.orchestra.run.smithi046.stdout:(148/156): librados2-19.0.0-3594.gc9da4049.el9. 7.8 MB/s | 3.3 MB 00:00 2024-05-09T03:04:29.189 INFO:teuthology.orchestra.run.smithi046.stdout:(149/156): librbd1-19.0.0-3594.gc9da4049.el9.x8 6.6 MB/s | 3.1 MB 00:00 2024-05-09T03:04:30.156 INFO:teuthology.orchestra.run.smithi046.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 2.0 MB/s | 2.3 MB 00:01 2024-05-09T03:04:30.289 INFO:teuthology.orchestra.run.smithi046.stdout:(151/156): NetworkManager-libnm-1.47.5-1.el9.x8 1.7 MB/s | 1.8 MB 00:01 2024-05-09T03:04:30.823 INFO:teuthology.orchestra.run.smithi046.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 63 kB/s | 42 kB 00:00 2024-05-09T03:04:31.148 INFO:teuthology.orchestra.run.smithi046.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 289 kB/s | 248 kB 00:00 2024-05-09T03:04:31.824 INFO:teuthology.orchestra.run.smithi046.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 1.3 MB/s | 1.3 MB 00:00 2024-05-09T03:04:32.224 INFO:teuthology.orchestra.run.smithi046.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.2 MB/s | 2.4 MB 00:01 2024-05-09T03:04:44.398 INFO:teuthology.orchestra.run.smithi046.stdout:(156/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 860 kB/s | 16 MB 00:19 2024-05-09T03:04:44.403 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:04:44.403 INFO:teuthology.orchestra.run.smithi046.stdout:Total 5.4 MB/s | 219 MB 00:40 2024-05-09T03:04:45.603 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-05-09T03:04:45.772 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-05-09T03:04:45.772 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-05-09T03:04:47.997 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-05-09T03:04:47.998 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-05-09T03:04:54.131 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-05-09T03:04:54.203 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T03:04:54.241 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T03:04:54.312 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T03:04:54.353 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T03:04:54.379 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T03:04:54.438 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T03:04:54.793 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T03:04:54.905 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T03:04:55.035 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T03:04:55.064 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T03:04:55.156 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T03:04:55.194 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T03:04:55.377 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T03:04:55.433 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T03:04:55.459 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T03:04:55.570 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T03:04:55.594 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T03:04:55.641 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T03:04:55.670 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T03:04:55.690 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T03:04:55.872 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T03:04:56.037 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T03:04:56.200 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T03:04:56.261 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T03:04:56.373 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T03:04:56.462 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T03:04:56.670 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T03:04:56.778 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T03:04:56.808 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T03:04:56.844 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/164 2024-05-09T03:04:56.919 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 22/164 2024-05-09T03:04:56.959 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 23/164 2024-05-09T03:04:57.011 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 24/164 2024-05-09T03:04:57.043 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 25/164 2024-05-09T03:04:57.071 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 26/164 2024-05-09T03:04:57.142 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 27/164 2024-05-09T03:04:57.202 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 28/164 2024-05-09T03:04:57.232 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 29/164 2024-05-09T03:04:57.276 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/164 2024-05-09T03:04:57.346 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/164 2024-05-09T03:04:57.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 32/164 2024-05-09T03:04:57.431 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 33/164 2024-05-09T03:04:57.459 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 34/164 2024-05-09T03:04:57.532 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/164 2024-05-09T03:04:57.574 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 36/164 2024-05-09T03:04:57.651 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 37/164 2024-05-09T03:04:57.789 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 38/164 2024-05-09T03:04:57.844 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 39/164 2024-05-09T03:04:58.398 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 40/164 2024-05-09T03:04:58.482 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 41/164 2024-05-09T03:04:58.517 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 42/164 2024-05-09T03:04:58.586 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 43/164 2024-05-09T03:04:58.624 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 44/164 2024-05-09T03:04:58.666 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 45/164 2024-05-09T03:04:58.823 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 46/164 2024-05-09T03:04:58.876 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 47/164 2024-05-09T03:04:58.971 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/164 2024-05-09T03:04:59.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 49/164 2024-05-09T03:04:59.107 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T03:04:59.144 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T03:04:59.181 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T03:04:59.228 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T03:04:59.263 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T03:04:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T03:04:59.349 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T03:04:59.492 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T03:04:59.554 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T03:04:59.584 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T03:04:59.626 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T03:04:59.732 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T03:04:59.776 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T03:04:59.817 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T03:04:59.849 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T03:04:59.874 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T03:04:59.966 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T03:05:00.106 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T03:05:00.140 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T03:05:00.171 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T03:05:00.202 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T03:05:00.242 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T03:05:00.273 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T03:05:00.302 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T03:05:00.337 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T03:05:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T03:05:00.413 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T03:05:00.447 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T03:05:00.537 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T03:05:00.600 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T03:05:00.633 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T03:05:00.673 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T03:05:00.784 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T03:05:00.836 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T03:05:01.610 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T03:05:01.657 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T03:05:01.690 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T03:05:01.796 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T03:05:01.849 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T03:05:01.940 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T03:05:01.975 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T03:05:02.159 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T03:05:02.190 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T03:05:02.270 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T03:05:02.980 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T03:05:03.099 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T03:05:04.440 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T03:05:04.518 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T03:05:04.546 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T03:05:04.579 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T03:05:04.913 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T03:05:04.958 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T03:05:05.033 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T03:05:05.063 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 102/164 2024-05-09T03:05:05.104 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T03:05:05.615 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T03:05:05.644 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T03:05:05.716 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T03:05:05.746 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 106/164 2024-05-09T03:05:08.036 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T03:05:08.075 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T03:05:08.681 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T03:05:08.721 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T03:05:08.812 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T03:05:08.813 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T03:05:08.813 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T03:05:08.813 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:08.847 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T03:05:20.969 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T03:05:20.969 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-05-09T03:05:20.970 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:21.145 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T03:05:21.164 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T03:05:21.165 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:21.165 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T03:05:21.165 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:22.829 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T03:05:22.858 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T03:05:22.858 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:22.858 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T03:05:22.858 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:05:22.858 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:05:22.859 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:23.107 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T03:05:23.131 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T03:05:23.131 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:23.131 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T03:05:23.132 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:05:23.132 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:05:23.132 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:23.616 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T03:05:23.641 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T03:05:23.641 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:23.641 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T03:05:23.641 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:05:23.641 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:05:23.641 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:23.756 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T03:05:23.814 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T03:05:23.878 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T03:05:24.073 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T03:05:24.125 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T03:05:24.206 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T03:05:24.305 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T03:05:24.448 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T03:05:24.495 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T03:05:24.541 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T03:05:24.576 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T03:05:24.940 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T03:05:24.966 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T03:05:24.999 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 126/164 2024-05-09T03:05:25.063 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 127/164 2024-05-09T03:05:25.098 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T03:05:26.198 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T03:05:26.241 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T03:05:27.257 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T03:05:27.292 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T03:05:27.411 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T03:05:27.582 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 131/164 2024-05-09T03:05:27.610 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T03:05:27.635 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T03:05:27.635 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:27.635 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T03:05:27.636 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:05:27.636 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:05:27.636 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:27.686 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T03:05:27.698 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T03:05:27.800 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 134/164 2024-05-09T03:05:27.862 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 135/164 2024-05-09T03:05:28.855 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T03:05:28.889 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T03:05:28.913 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T03:05:28.913 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:28.914 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T03:05:28.914 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T03:05:28.914 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T03:05:28.914 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:28.956 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T03:05:28.980 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T03:05:28.980 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:28.980 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T03:05:28.980 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:29.032 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T03:05:29.056 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T03:05:29.056 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:29.056 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T03:05:29.057 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T03:05:29.057 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T03:05:29.057 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:29.382 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T03:05:29.407 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T03:05:29.407 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:05:29.407 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T03:05:29.407 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:05:29.407 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:05:29.408 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:33.920 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 141/164 2024-05-09T03:05:33.961 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 142/164 2024-05-09T03:05:34.003 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 143/164 2024-05-09T03:05:34.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T03:05:34.078 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T03:05:34.300 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 146/164 2024-05-09T03:05:34.340 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T03:05:34.370 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 148/164 2024-05-09T03:05:34.400 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T03:05:34.446 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T03:05:35.049 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T03:05:35.085 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T03:05:35.106 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T03:05:36.017 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 152/164 2024-05-09T03:05:36.123 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T03:05:36.174 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T03:05:36.281 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T03:05:36.318 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T03:05:36.344 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T03:05:36.344 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T03:05:36.428 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T03:05:36.469 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T03:05:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T03:05:36.546 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T03:05:36.571 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T03:05:36.585 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T03:05:36.585 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T03:05:36.772 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T03:05:36.773 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T03:05:36.857 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T03:05:36.925 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 1/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 3/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 4/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 5/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 6/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 7/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 11/164 2024-05-09T03:05:42.377 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 12/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 14/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 15/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 16/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 17/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 18/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 19/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 20/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 21/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 22/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 23/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 24/164 2024-05-09T03:05:42.378 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 25/164 2024-05-09T03:05:42.379 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 26/164 2024-05-09T03:05:42.379 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 27/164 2024-05-09T03:05:42.379 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 28/164 2024-05-09T03:05:42.390 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 29/164 2024-05-09T03:05:42.390 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 30/164 2024-05-09T03:05:42.390 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 31/164 2024-05-09T03:05:42.390 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 32/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 33/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 34/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 35/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 36/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 37/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 38/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 39/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T03:05:42.391 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T03:05:42.392 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T03:05:42.393 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T03:05:42.394 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T03:05:42.395 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T03:05:42.396 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T03:05:42.397 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 149/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 151/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T03:05:42.398 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T03:05:42.399 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T03:05:44.997 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T03:05:44.997 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:44.997 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-05-09T03:05:44.997 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T03:05:44.997 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T03:05:44.997 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.998 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:44.999 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T03:05:45.000 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T03:05:45.001 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T03:05:45.002 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T03:05:45.003 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T03:05:45.004 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T03:05:45.005 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T03:05:45.006 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T03:05:45.007 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T03:05:45.008 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T03:05:45.009 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T03:05:45.009 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T03:05:45.009 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T03:05:45.009 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T03:05:45.009 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:05:45.009 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-05-09T03:05:45.400 DEBUG:teuthology.parallel:result is None 2024-05-09T03:07:06.951 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 86 B/s | 16 kB 03:14 2024-05-09T03:07:09.071 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-05-09T03:07:14.201 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-05-09T03:07:16.503 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T03:07:16.503 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T03:07:16.514 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T03:07:16.514 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T03:07:16.515 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T03:07:16.516 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T03:07:16.517 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T03:07:16.520 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T03:07:16.520 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T03:07:16.580 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-05-09T03:07:16.596 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-05-09T03:07:16.596 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-05-09T03:07:16.596 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-05-09T03:07:16.597 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-05-09T03:07:16.597 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T03:07:16.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 6.6 k 2024-05-09T03:07:16.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.4 M 2024-05-09T03:07:16.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 863 k 2024-05-09T03:07:16.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 147 k 2024-05-09T03:07:16.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 1.5 M 2024-05-09T03:07:16.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 142 k 2024-05-09T03:07:16.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 3.5 M 2024-05-09T03:07:16.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 7.4 M 2024-05-09T03:07:16.599 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 49 k 2024-05-09T03:07:16.599 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 10 M 2024-05-09T03:07:16.599 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 43 M 2024-05-09T03:07:16.599 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 273 k 2024-05-09T03:07:16.599 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 763 k 2024-05-09T03:07:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 224 k 2024-05-09T03:07:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-shell x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 27 k 2024-05-09T03:07:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 18 k 2024-05-09T03:07:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T03:07:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T03:07:16.600 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T03:07:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 32 k 2024-05-09T03:07:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 756 k 2024-05-09T03:07:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 127 k 2024-05-09T03:07:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T03:07:16.601 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T03:07:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T03:07:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T03:07:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T03:07:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T03:07:16.602 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 162 k 2024-05-09T03:07:16.603 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 321 k 2024-05-09T03:07:16.603 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 301 k 2024-05-09T03:07:16.603 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 99 k 2024-05-09T03:07:16.603 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 86 k 2024-05-09T03:07:16.603 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.0 M 2024-05-09T03:07:16.603 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 169 k 2024-05-09T03:07:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T03:07:16.604 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-05-09T03:07:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T03:07:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T03:07:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T03:07:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T03:07:16.605 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.3 M 2024-05-09T03:07:16.605 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.1 M 2024-05-09T03:07:16.605 INFO:teuthology.orchestra.run.smithi088.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T03:07:16.605 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T03:07:16.605 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-05-09T03:07:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T03:07:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T03:07:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T03:07:16.606 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 20 M 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 34 k 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 2.3 M 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 274 k 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 4.9 M 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 18 M 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 16 k 2024-05-09T03:07:16.609 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 25 k 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T03:07:16.610 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 158 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T03:07:16.611 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 497 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.2 M 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T03:07:16.612 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T03:07:16.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 45 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 135 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T03:07:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T03:07:16.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T03:07:16.616 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T03:07:16.617 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T03:07:16.618 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T03:07:16.619 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T03:07:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T03:07:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T03:07:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T03:07:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T03:07:16.620 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T03:07:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T03:07:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T03:07:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T03:07:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T03:07:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T03:07:16.621 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T03:07:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T03:07:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T03:07:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T03:07:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T03:07:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T03:07:16.622 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T03:07:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T03:07:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T03:07:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T03:07:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T03:07:16.623 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T03:07:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T03:07:16.624 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-05-09T03:07:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T03:07:16.624 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:07:16.624 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-05-09T03:07:16.624 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-05-09T03:07:16.625 INFO:teuthology.orchestra.run.smithi088.stdout:Install 148 Packages 2024-05-09T03:07:16.625 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 8 Packages 2024-05-09T03:07:16.625 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:07:16.626 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 219 M 2024-05-09T03:07:16.626 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-05-09T03:07:17.428 INFO:teuthology.orchestra.run.smithi088.stdout:(1/156): ceph-19.0.0-3594.gc9da4049.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-05-09T03:07:17.745 INFO:teuthology.orchestra.run.smithi088.stdout:(2/156): ceph-fuse-19.0.0-3594.gc9da4049.el9.x8 2.7 MB/s | 863 kB 00:00 2024-05-09T03:07:18.220 INFO:teuthology.orchestra.run.smithi088.stdout:(3/156): ceph-base-19.0.0-3594.gc9da4049.el9.x8 5.2 MB/s | 5.4 MB 00:01 2024-05-09T03:07:18.553 INFO:teuthology.orchestra.run.smithi088.stdout:(4/156): ceph-mds-19.0.0-3594.gc9da4049.el9.x86 6.8 MB/s | 2.3 MB 00:00 2024-05-09T03:07:18.603 INFO:teuthology.orchestra.run.smithi088.stdout:(5/156): ceph-immutable-object-cache-19.0.0-359 171 kB/s | 147 kB 00:00 2024-05-09T03:07:18.745 INFO:teuthology.orchestra.run.smithi088.stdout:(6/156): ceph-mgr-19.0.0-3594.gc9da4049.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-05-09T03:07:19.862 INFO:teuthology.orchestra.run.smithi088.stdout:(7/156): ceph-common-19.0.0-3594.gc9da4049.el9. 7.3 MB/s | 20 MB 00:02 2024-05-09T03:07:20.004 INFO:teuthology.orchestra.run.smithi088.stdout:(8/156): ceph-mon-19.0.0-3594.gc9da4049.el9.x86 3.5 MB/s | 4.9 MB 00:01 2024-05-09T03:07:20.062 INFO:teuthology.orchestra.run.smithi088.stdout:(9/156): ceph-selinux-19.0.0-3594.gc9da4049.el9 432 kB/s | 25 kB 00:00 2024-05-09T03:07:20.396 INFO:teuthology.orchestra.run.smithi088.stdout:(10/156): ceph-osd-19.0.0-3594.gc9da4049.el9.x8 11 MB/s | 18 MB 00:01 2024-05-09T03:07:20.471 INFO:teuthology.orchestra.run.smithi088.stdout:(11/156): cephfs-mirror-19.0.0-3594.gc9da4049.e 2.9 MB/s | 224 kB 00:00 2024-05-09T03:07:20.539 INFO:teuthology.orchestra.run.smithi088.stdout:(12/156): cephfs-shell-19.0.0-3594.gc9da4049.el 405 kB/s | 27 kB 00:00 2024-05-09T03:07:20.605 INFO:teuthology.orchestra.run.smithi088.stdout:(13/156): libcephfs-devel-19.0.0-3594.gc9da4049 477 kB/s | 32 kB 00:00 2024-05-09T03:07:20.730 INFO:teuthology.orchestra.run.smithi088.stdout:(14/156): libcephfs2-19.0.0-3594.gc9da4049.el9. 5.9 MB/s | 756 kB 00:00 2024-05-09T03:07:20.806 INFO:teuthology.orchestra.run.smithi088.stdout:(15/156): libcephsqlite-19.0.0-3594.gc9da4049.e 2.1 MB/s | 158 kB 00:00 2024-05-09T03:07:20.931 INFO:teuthology.orchestra.run.smithi088.stdout:(16/156): ceph-radosgw-19.0.0-3594.gc9da4049.el 9.5 MB/s | 10 MB 00:01 2024-05-09T03:07:20.956 INFO:teuthology.orchestra.run.smithi088.stdout:(17/156): librados-devel-19.0.0-3594.gc9da4049. 843 kB/s | 127 kB 00:00 2024-05-09T03:07:21.015 INFO:teuthology.orchestra.run.smithi088.stdout:(18/156): libradosstriper1-19.0.0-3594.gc9da404 5.8 MB/s | 497 kB 00:00 2024-05-09T03:07:21.082 INFO:teuthology.orchestra.run.smithi088.stdout:(19/156): python3-ceph-argparse-19.0.0-3594.gc9 678 kB/s | 45 kB 00:00 2024-05-09T03:07:21.132 INFO:teuthology.orchestra.run.smithi088.stdout:(20/156): python3-ceph-common-19.0.0-3594.gc9da 2.6 MB/s | 135 kB 00:00 2024-05-09T03:07:21.191 INFO:teuthology.orchestra.run.smithi088.stdout:(21/156): python3-cephfs-19.0.0-3594.gc9da4049. 2.7 MB/s | 162 kB 00:00 2024-05-09T03:07:21.258 INFO:teuthology.orchestra.run.smithi088.stdout:(22/156): python3-rados-19.0.0-3594.gc9da4049.e 4.7 MB/s | 321 kB 00:00 2024-05-09T03:07:21.317 INFO:teuthology.orchestra.run.smithi088.stdout:(23/156): python3-rbd-19.0.0-3594.gc9da4049.el9 5.0 MB/s | 301 kB 00:00 2024-05-09T03:07:21.375 INFO:teuthology.orchestra.run.smithi088.stdout:(24/156): python3-rgw-19.0.0-3594.gc9da4049.el9 1.7 MB/s | 99 kB 00:00 2024-05-09T03:07:21.417 INFO:teuthology.orchestra.run.smithi088.stdout:(25/156): rbd-fuse-19.0.0-3594.gc9da4049.el9.x8 2.0 MB/s | 86 kB 00:00 2024-05-09T03:07:21.493 INFO:teuthology.orchestra.run.smithi088.stdout:(26/156): librgw2-19.0.0-3594.gc9da4049.el9.x86 9.6 MB/s | 5.2 MB 00:00 2024-05-09T03:07:21.551 INFO:teuthology.orchestra.run.smithi088.stdout:(27/156): rbd-nbd-19.0.0-3594.gc9da4049.el9.x86 2.8 MB/s | 169 kB 00:00 2024-05-09T03:07:21.618 INFO:teuthology.orchestra.run.smithi088.stdout:(28/156): ceph-grafana-dashboards-19.0.0-3594.g 513 kB/s | 34 kB 00:00 2024-05-09T03:07:21.694 INFO:teuthology.orchestra.run.smithi088.stdout:(29/156): ceph-mgr-cephadm-19.0.0-3594.gc9da404 1.8 MB/s | 142 kB 00:00 2024-05-09T03:07:21.744 INFO:teuthology.orchestra.run.smithi088.stdout:(30/156): rbd-mirror-19.0.0-3594.gc9da4049.el9. 9.3 MB/s | 3.0 MB 00:00 2024-05-09T03:07:22.036 INFO:teuthology.orchestra.run.smithi088.stdout:(31/156): ceph-mgr-dashboard-19.0.0-3594.gc9da4 10 MB/s | 3.5 MB 00:00 2024-05-09T03:07:22.103 INFO:teuthology.orchestra.run.smithi088.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3594.gc9 4.0 MB/s | 274 kB 00:00 2024-05-09T03:07:22.170 INFO:teuthology.orchestra.run.smithi088.stdout:(33/156): ceph-mgr-rook-19.0.0-3594.gc9da4049.e 740 kB/s | 49 kB 00:00 2024-05-09T03:07:22.260 INFO:teuthology.orchestra.run.smithi088.stdout:(34/156): ceph-prometheus-alerts-19.0.0-3594.gc 184 kB/s | 16 kB 00:00 2024-05-09T03:07:22.320 INFO:teuthology.orchestra.run.smithi088.stdout:(35/156): ceph-volume-19.0.0-3594.gc9da4049.el9 4.4 MB/s | 273 kB 00:00 2024-05-09T03:07:22.471 INFO:teuthology.orchestra.run.smithi088.stdout:(36/156): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-05-09T03:07:22.512 INFO:teuthology.orchestra.run.smithi088.stdout:(37/156): cephadm-19.0.0-3594.gc9da4049.el9.noa 3.9 MB/s | 763 kB 00:00 2024-05-09T03:07:22.538 INFO:teuthology.orchestra.run.smithi088.stdout:(38/156): cephfs-top-19.0.0-3594.gc9da4049.el9. 273 kB/s | 18 kB 00:00 2024-05-09T03:07:22.680 INFO:teuthology.orchestra.run.smithi088.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-05-09T03:07:22.763 INFO:teuthology.orchestra.run.smithi088.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-09T03:07:22.831 INFO:teuthology.orchestra.run.smithi088.stdout:(41/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-05-09T03:07:22.989 INFO:teuthology.orchestra.run.smithi088.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-05-09T03:07:23.039 INFO:teuthology.orchestra.run.smithi088.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T03:07:23.098 INFO:teuthology.orchestra.run.smithi088.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-09T03:07:23.140 INFO:teuthology.orchestra.run.smithi088.stdout:(45/156): python3-cryptography-36.0.1-4.el9.x86 4.0 MB/s | 1.2 MB 00:00 2024-05-09T03:07:23.165 INFO:teuthology.orchestra.run.smithi088.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-09T03:07:23.232 INFO:teuthology.orchestra.run.smithi088.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-05-09T03:07:23.358 INFO:teuthology.orchestra.run.smithi088.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-05-09T03:07:23.408 INFO:teuthology.orchestra.run.smithi088.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-05-09T03:07:23.466 INFO:teuthology.orchestra.run.smithi088.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 943 kB 00:00 2024-05-09T03:07:23.483 INFO:teuthology.orchestra.run.smithi088.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 285 kB/s | 21 kB 00:00 2024-05-09T03:07:23.525 INFO:teuthology.orchestra.run.smithi088.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 787 kB/s | 33 kB 00:00 2024-05-09T03:07:23.551 INFO:teuthology.orchestra.run.smithi088.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-05-09T03:07:23.593 INFO:teuthology.orchestra.run.smithi088.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 437 kB/s | 18 kB 00:00 2024-05-09T03:07:23.643 INFO:teuthology.orchestra.run.smithi088.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-05-09T03:07:23.702 INFO:teuthology.orchestra.run.smithi088.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-09T03:07:23.752 INFO:teuthology.orchestra.run.smithi088.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-05-09T03:07:23.902 INFO:teuthology.orchestra.run.smithi088.stdout:(58/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-05-09T03:07:24.053 INFO:teuthology.orchestra.run.smithi088.stdout:(59/156): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-05-09T03:07:24.103 INFO:teuthology.orchestra.run.smithi088.stdout:(60/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 512 kB/s | 25 kB 00:00 2024-05-09T03:07:24.195 INFO:teuthology.orchestra.run.smithi088.stdout:(61/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.5 MB/s | 3.0 MB 00:00 2024-05-09T03:07:24.220 INFO:teuthology.orchestra.run.smithi088.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-05-09T03:07:24.279 INFO:teuthology.orchestra.run.smithi088.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-05-09T03:07:24.312 INFO:teuthology.orchestra.run.smithi088.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-05-09T03:07:24.429 INFO:teuthology.orchestra.run.smithi088.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 235 kB/s | 35 kB 00:00 2024-05-09T03:07:25.121 INFO:teuthology.orchestra.run.smithi088.stdout:(66/156): openblas-openmp-0.3.21-2.el9.x86_64.r 5.9 MB/s | 4.7 MB 00:00 2024-05-09T03:07:25.388 INFO:teuthology.orchestra.run.smithi088.stdout:(67/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-05-09T03:07:25.421 INFO:teuthology.orchestra.run.smithi088.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-05-09T03:07:25.480 INFO:teuthology.orchestra.run.smithi088.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-09T03:07:25.606 INFO:teuthology.orchestra.run.smithi088.stdout:(70/156): postgresql-contrib-13.14-1.el9.x86_64 4.1 MB/s | 911 kB 00:00 2024-05-09T03:07:25.814 INFO:teuthology.orchestra.run.smithi088.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-05-09T03:07:25.873 INFO:teuthology.orchestra.run.smithi088.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-05-09T03:07:26.306 INFO:teuthology.orchestra.run.smithi088.stdout:(73/156): postgresql-server-13.14-1.el9.x86_64. 7.0 MB/s | 5.8 MB 00:00 2024-05-09T03:07:26.382 INFO:teuthology.orchestra.run.smithi088.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-05-09T03:07:26.465 INFO:teuthology.orchestra.run.smithi088.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-05-09T03:07:26.532 INFO:teuthology.orchestra.run.smithi088.stdout:(76/156): python3-jsonpatch-1.21-16.el9.noarch. 384 kB/s | 26 kB 00:00 2024-05-09T03:07:26.608 INFO:teuthology.orchestra.run.smithi088.stdout:(77/156): python3-jsonpointer-2.0-4.el9.noarch. 255 kB/s | 19 kB 00:00 2024-05-09T03:07:26.700 INFO:teuthology.orchestra.run.smithi088.stdout:(78/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-05-09T03:07:26.766 INFO:teuthology.orchestra.run.smithi088.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 520 kB/s | 35 kB 00:00 2024-05-09T03:07:27.109 INFO:teuthology.orchestra.run.smithi088.stdout:(80/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.8 MB/s | 6.0 MB 00:01 2024-05-09T03:07:27.175 INFO:teuthology.orchestra.run.smithi088.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-05-09T03:07:27.251 INFO:teuthology.orchestra.run.smithi088.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-05-09T03:07:27.318 INFO:teuthology.orchestra.run.smithi088.stdout:(83/156): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-05-09T03:07:27.385 INFO:teuthology.orchestra.run.smithi088.stdout:(84/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.2 MB/s | 5.1 MB 00:00 2024-05-09T03:07:27.410 INFO:teuthology.orchestra.run.smithi088.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 459 kB/s | 42 kB 00:00 2024-05-09T03:07:27.452 INFO:teuthology.orchestra.run.smithi088.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 269 kB 00:00 2024-05-09T03:07:27.477 INFO:teuthology.orchestra.run.smithi088.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-09T03:07:27.511 INFO:teuthology.orchestra.run.smithi088.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-05-09T03:07:27.536 INFO:teuthology.orchestra.run.smithi088.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-05-09T03:07:27.570 INFO:teuthology.orchestra.run.smithi088.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 914 kB/s | 54 kB 00:00 2024-05-09T03:07:27.628 INFO:teuthology.orchestra.run.smithi088.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 715 kB/s | 42 kB 00:00 2024-05-09T03:07:27.687 INFO:teuthology.orchestra.run.smithi088.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-05-09T03:07:27.754 INFO:teuthology.orchestra.run.smithi088.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-05-09T03:07:27.813 INFO:teuthology.orchestra.run.smithi088.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 618 kB/s | 36 kB 00:00 2024-05-09T03:07:27.871 INFO:teuthology.orchestra.run.smithi088.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-05-09T03:07:27.938 INFO:teuthology.orchestra.run.smithi088.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-05-09T03:07:28.005 INFO:teuthology.orchestra.run.smithi088.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 867 kB/s | 58 kB 00:00 2024-05-09T03:07:28.064 INFO:teuthology.orchestra.run.smithi088.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-05-09T03:07:28.123 INFO:teuthology.orchestra.run.smithi088.stdout:(99/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-09T03:07:28.181 INFO:teuthology.orchestra.run.smithi088.stdout:(100/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 769 kB/s | 45 kB 00:00 2024-05-09T03:07:28.307 INFO:teuthology.orchestra.run.smithi088.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-05-09T03:07:28.432 INFO:teuthology.orchestra.run.smithi088.stdout:(102/156): protobuf-compiler-3.14.0-13.el9.x86_ 6.8 MB/s | 863 kB 00:00 2024-05-09T03:07:28.974 INFO:teuthology.orchestra.run.smithi088.stdout:(103/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-05-09T03:07:29.349 INFO:teuthology.orchestra.run.smithi088.stdout:(104/156): ceph-test-19.0.0-3594.gc9da4049.el9. 4.7 MB/s | 43 MB 00:09 2024-05-09T03:07:29.449 INFO:teuthology.orchestra.run.smithi088.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 539 kB/s | 548 kB 00:01 2024-05-09T03:07:29.524 INFO:teuthology.orchestra.run.smithi088.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 131 kB/s | 19 kB 00:00 2024-05-09T03:07:29.583 INFO:teuthology.orchestra.run.smithi088.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-05-09T03:07:29.658 INFO:teuthology.orchestra.run.smithi088.stdout:(108/156): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-05-09T03:07:29.725 INFO:teuthology.orchestra.run.smithi088.stdout:(109/156): gperftools-libs-2.9.1-2.el9.x86_64.r 411 kB/s | 309 kB 00:00 2024-05-09T03:07:29.767 INFO:teuthology.orchestra.run.smithi088.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-05-09T03:07:29.876 INFO:teuthology.orchestra.run.smithi088.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-05-09T03:07:30.235 INFO:teuthology.orchestra.run.smithi088.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 1.5 MB/s | 548 kB 00:00 2024-05-09T03:07:30.293 INFO:teuthology.orchestra.run.smithi088.stdout:(113/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 747 kB/s | 43 kB 00:00 2024-05-09T03:07:30.352 INFO:teuthology.orchestra.run.smithi088.stdout:(114/156): python3-cachetools-4.2.4-1.el9.noarc 553 kB/s | 32 kB 00:00 2024-05-09T03:07:30.394 INFO:teuthology.orchestra.run.smithi088.stdout:(115/156): python3-certifi-2021.10.8-2.el9.noar 361 kB/s | 15 kB 00:00 2024-05-09T03:07:30.436 INFO:teuthology.orchestra.run.smithi088.stdout:(116/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-05-09T03:07:30.536 INFO:teuthology.orchestra.run.smithi088.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-05-09T03:07:30.695 INFO:teuthology.orchestra.run.smithi088.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 1.4 MB/s | 358 kB 00:00 2024-05-09T03:07:30.729 INFO:teuthology.orchestra.run.smithi088.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.5 MB/s | 303 kB 00:00 2024-05-09T03:07:30.771 INFO:teuthology.orchestra.run.smithi088.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-05-09T03:07:30.887 INFO:teuthology.orchestra.run.smithi088.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 1.3 MB/s | 206 kB 00:00 2024-05-09T03:07:30.988 INFO:teuthology.orchestra.run.smithi088.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-09T03:07:31.029 INFO:teuthology.orchestra.run.smithi088.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 258 kB/s | 11 kB 00:00 2024-05-09T03:07:31.071 INFO:teuthology.orchestra.run.smithi088.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-05-09T03:07:31.113 INFO:teuthology.orchestra.run.smithi088.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-05-09T03:07:31.155 INFO:teuthology.orchestra.run.smithi088.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 467 kB/s | 19 kB 00:00 2024-05-09T03:07:31.197 INFO:teuthology.orchestra.run.smithi088.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 475 kB/s | 20 kB 00:00 2024-05-09T03:07:31.481 INFO:teuthology.orchestra.run.smithi088.stdout:(128/156): libarrow-9.0.0-11.el9.x86_64.rpm 2.2 MB/s | 4.4 MB 00:02 2024-05-09T03:07:31.523 INFO:teuthology.orchestra.run.smithi088.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-05-09T03:07:31.582 INFO:teuthology.orchestra.run.smithi088.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-05-09T03:07:31.640 INFO:teuthology.orchestra.run.smithi088.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 990 kB/s | 58 kB 00:00 2024-05-09T03:07:31.749 INFO:teuthology.orchestra.run.smithi088.stdout:(132/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-05-09T03:07:31.791 INFO:teuthology.orchestra.run.smithi088.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 395 kB/s | 16 kB 00:00 2024-05-09T03:07:31.858 INFO:teuthology.orchestra.run.smithi088.stdout:(134/156): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-05-09T03:07:31.883 INFO:teuthology.orchestra.run.smithi088.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 980 kB/s | 90 kB 00:00 2024-05-09T03:07:31.908 INFO:teuthology.orchestra.run.smithi088.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-05-09T03:07:31.925 INFO:teuthology.orchestra.run.smithi088.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 730 kB/s | 31 kB 00:00 2024-05-09T03:07:31.976 INFO:teuthology.orchestra.run.smithi088.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-09T03:07:32.035 INFO:teuthology.orchestra.run.smithi088.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-09T03:07:32.068 INFO:teuthology.orchestra.run.smithi088.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-05-09T03:07:32.110 INFO:teuthology.orchestra.run.smithi088.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 679 kB/s | 51 kB 00:00 2024-05-09T03:07:32.194 INFO:teuthology.orchestra.run.smithi088.stdout:(142/156): python3-grpcio-1.46.7-10.el9.x86_64. 1.4 MB/s | 2.0 MB 00:01 2024-05-09T03:07:32.219 INFO:teuthology.orchestra.run.smithi088.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 827 kB/s | 90 kB 00:00 2024-05-09T03:07:32.261 INFO:teuthology.orchestra.run.smithi088.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 481 kB/s | 20 kB 00:00 2024-05-09T03:07:32.286 INFO:teuthology.orchestra.run.smithi088.stdout:(145/156): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-05-09T03:07:32.378 INFO:teuthology.orchestra.run.smithi088.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-05-09T03:07:32.420 INFO:teuthology.orchestra.run.smithi088.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 1.8 MB/s | 427 kB 00:00 2024-05-09T03:07:32.837 INFO:teuthology.orchestra.run.smithi088.stdout:(148/156): librbd1-19.0.0-3594.gc9da4049.el9.x8 7.4 MB/s | 3.1 MB 00:00 2024-05-09T03:07:33.004 INFO:teuthology.orchestra.run.smithi088.stdout:(149/156): librados2-19.0.0-3594.gc9da4049.el9. 5.3 MB/s | 3.3 MB 00:00 2024-05-09T03:07:33.104 INFO:teuthology.orchestra.run.smithi088.stdout:(150/156): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-05-09T03:07:33.146 INFO:teuthology.orchestra.run.smithi088.stdout:(151/156): NetworkManager-1.47.5-1.el9.x86_64.r 7.5 MB/s | 2.3 MB 00:00 2024-05-09T03:07:33.230 INFO:teuthology.orchestra.run.smithi088.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 333 kB/s | 42 kB 00:00 2024-05-09T03:07:33.255 INFO:teuthology.orchestra.run.smithi088.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-05-09T03:07:33.322 INFO:teuthology.orchestra.run.smithi088.stdout:(154/156): NetworkManager-libnm-1.47.5-1.el9.x8 5.7 MB/s | 1.8 MB 00:00 2024-05-09T03:07:33.514 INFO:teuthology.orchestra.run.smithi088.stdout:(155/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.3 MB/s | 2.4 MB 00:00 2024-05-09T03:07:33.555 INFO:teuthology.orchestra.run.smithi088.stdout:(156/156): openssl-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 1.3 MB 00:00 2024-05-09T03:07:33.562 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-05-09T03:07:33.562 INFO:teuthology.orchestra.run.smithi088.stdout:Total 13 MB/s | 219 MB 00:16 2024-05-09T03:07:34.731 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-05-09T03:07:34.880 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-05-09T03:07:34.880 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-05-09T03:07:37.004 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-05-09T03:07:37.004 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-05-09T03:07:43.043 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-05-09T03:07:43.143 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T03:07:43.181 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T03:07:43.255 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T03:07:43.293 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T03:07:43.319 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T03:07:43.377 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T03:07:43.892 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T03:07:44.004 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T03:07:44.113 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T03:07:44.146 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T03:07:44.222 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T03:07:44.251 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T03:07:44.426 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T03:07:44.481 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T03:07:44.509 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T03:07:44.613 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T03:07:44.635 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T03:07:44.697 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T03:07:44.719 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T03:07:44.740 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T03:07:44.937 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T03:07:45.117 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T03:07:45.277 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T03:07:45.327 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T03:07:45.439 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T03:07:45.528 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T03:07:45.728 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T03:07:45.836 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T03:07:45.876 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T03:07:45.916 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/164 2024-05-09T03:07:45.981 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 22/164 2024-05-09T03:07:46.024 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 23/164 2024-05-09T03:07:46.077 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 24/164 2024-05-09T03:07:46.109 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 25/164 2024-05-09T03:07:46.137 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 26/164 2024-05-09T03:07:46.205 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 27/164 2024-05-09T03:07:46.259 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 28/164 2024-05-09T03:07:46.290 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 29/164 2024-05-09T03:07:46.325 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/164 2024-05-09T03:07:46.409 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/164 2024-05-09T03:07:46.444 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 32/164 2024-05-09T03:07:46.488 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 33/164 2024-05-09T03:07:46.516 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 34/164 2024-05-09T03:07:46.584 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/164 2024-05-09T03:07:46.631 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-1.el9.noarch 36/164 2024-05-09T03:07:46.705 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 37/164 2024-05-09T03:07:46.837 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 38/164 2024-05-09T03:07:46.901 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 39/164 2024-05-09T03:07:47.419 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 40/164 2024-05-09T03:07:47.504 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 41/164 2024-05-09T03:07:47.533 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 42/164 2024-05-09T03:07:47.591 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 43/164 2024-05-09T03:07:47.631 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 44/164 2024-05-09T03:07:47.679 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 45/164 2024-05-09T03:07:47.850 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 46/164 2024-05-09T03:07:47.909 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 47/164 2024-05-09T03:07:47.999 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/164 2024-05-09T03:07:48.064 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 49/164 2024-05-09T03:07:48.137 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T03:07:48.177 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T03:07:48.213 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T03:07:48.260 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T03:07:48.296 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T03:07:48.339 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T03:07:48.397 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T03:07:48.532 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T03:07:48.645 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T03:07:48.683 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T03:07:48.733 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T03:07:48.827 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T03:07:48.859 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T03:07:48.900 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T03:07:48.932 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T03:07:48.957 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T03:07:49.043 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T03:07:49.167 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T03:07:49.206 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T03:07:49.236 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T03:07:49.277 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T03:07:49.324 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T03:07:49.356 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T03:07:49.477 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T03:07:49.512 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T03:07:49.546 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T03:07:49.587 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T03:07:49.622 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T03:07:49.709 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T03:07:49.756 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T03:07:49.799 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T03:07:49.846 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T03:07:49.944 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T03:07:49.993 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T03:07:50.700 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T03:07:50.748 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T03:07:50.782 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T03:07:50.886 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T03:07:50.948 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T03:07:51.034 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T03:07:51.067 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T03:07:51.251 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T03:07:51.282 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T03:07:51.366 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T03:07:52.015 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T03:07:52.129 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T03:07:53.390 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T03:07:53.469 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T03:07:53.496 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T03:07:53.530 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T03:07:53.848 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T03:07:53.901 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T03:07:53.992 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T03:07:54.022 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 102/164 2024-05-09T03:07:54.068 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T03:07:54.556 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T03:07:54.579 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T03:07:54.650 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T03:07:54.680 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 106/164 2024-05-09T03:07:56.905 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T03:07:56.943 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T03:07:57.522 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T03:07:57.555 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T03:07:57.645 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T03:07:57.645 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T03:07:57.645 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T03:07:57.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:07:57.689 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T03:08:09.692 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T03:08:09.692 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-05-09T03:08:09.692 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-05-09T03:08:09.692 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-05-09T03:08:09.692 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-05-09T03:08:09.693 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-05-09T03:08:09.693 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-05-09T03:08:09.693 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-05-09T03:08:09.693 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:09.865 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T03:08:09.881 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T03:08:09.881 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:09.881 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T03:08:09.881 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:11.468 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T03:08:11.495 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T03:08:11.495 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:11.495 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T03:08:11.495 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:08:11.495 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T03:08:11.495 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:11.718 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T03:08:11.742 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T03:08:11.742 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:11.742 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T03:08:11.742 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:08:11.742 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T03:08:11.742 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:12.219 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T03:08:12.242 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T03:08:12.242 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:12.242 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T03:08:12.242 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:08:12.242 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T03:08:12.243 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:12.349 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T03:08:12.406 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T03:08:12.461 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T03:08:12.645 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T03:08:12.695 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T03:08:12.774 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T03:08:12.860 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T03:08:12.998 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T03:08:13.047 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T03:08:13.086 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T03:08:13.114 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T03:08:13.492 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T03:08:13.521 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T03:08:13.553 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 126/164 2024-05-09T03:08:13.622 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 127/164 2024-05-09T03:08:13.661 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T03:08:14.686 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T03:08:14.730 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T03:08:15.685 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T03:08:15.722 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T03:08:15.831 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T03:08:16.004 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 131/164 2024-05-09T03:08:16.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T03:08:16.056 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T03:08:16.056 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:16.056 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T03:08:16.056 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:08:16.056 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T03:08:16.056 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:16.099 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T03:08:16.111 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T03:08:16.208 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 134/164 2024-05-09T03:08:16.258 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 135/164 2024-05-09T03:08:17.190 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T03:08:17.227 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T03:08:17.251 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T03:08:17.251 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:17.251 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T03:08:17.251 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T03:08:17.251 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T03:08:17.251 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:17.295 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T03:08:17.317 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T03:08:17.318 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:17.318 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T03:08:17.318 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:17.362 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T03:08:17.385 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T03:08:17.385 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:17.385 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T03:08:17.385 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T03:08:17.385 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T03:08:17.385 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:17.696 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T03:08:17.719 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T03:08:17.719 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T03:08:17.719 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T03:08:17.719 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:08:17.719 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T03:08:17.719 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:21.966 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 141/164 2024-05-09T03:08:22.008 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 142/164 2024-05-09T03:08:22.048 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 143/164 2024-05-09T03:08:22.080 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T03:08:22.108 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T03:08:22.298 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 146/164 2024-05-09T03:08:22.346 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T03:08:22.376 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 148/164 2024-05-09T03:08:22.406 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T03:08:22.452 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T03:08:23.078 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T03:08:23.117 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T03:08:23.137 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T03:08:23.968 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 152/164 2024-05-09T03:08:24.047 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T03:08:24.107 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T03:08:24.201 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T03:08:24.242 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T03:08:24.268 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T03:08:24.269 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T03:08:24.386 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T03:08:24.411 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T03:08:24.411 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T03:08:24.497 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T03:08:24.521 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T03:08:24.533 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T03:08:24.533 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T03:08:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T03:08:24.722 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T03:08:24.806 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T03:08:24.871 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T03:08:30.635 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 1/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 3/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 4/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 5/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 6/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 7/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T03:08:30.636 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 11/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 12/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 14/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 15/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 16/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 17/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 18/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 19/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 20/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 21/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 22/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 23/164 2024-05-09T03:08:30.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 24/164 2024-05-09T03:08:30.638 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 25/164 2024-05-09T03:08:30.638 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 26/164 2024-05-09T03:08:30.638 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 27/164 2024-05-09T03:08:30.650 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 28/164 2024-05-09T03:08:30.650 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 29/164 2024-05-09T03:08:30.650 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 30/164 2024-05-09T03:08:30.650 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 31/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 32/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 33/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 34/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 35/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 36/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 37/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 38/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 39/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T03:08:30.651 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T03:08:30.652 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T03:08:30.653 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T03:08:30.654 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T03:08:30.655 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T03:08:30.656 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T03:08:30.657 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T03:08:30.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 149/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 151/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T03:08:30.659 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T03:08:30.660 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T03:08:33.341 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T03:08:33.341 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:33.341 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-05-09T03:08:33.341 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T03:08:33.341 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T03:08:33.341 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T03:08:33.342 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.343 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.344 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.345 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T03:08:33.346 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T03:08:33.347 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T03:08:33.348 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T03:08:33.349 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T03:08:33.350 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T03:08:33.351 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T03:08:33.352 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T03:08:33.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T03:08:33.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T03:08:33.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T03:08:33.356 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T03:08:33.357 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T03:08:33.357 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:33.357 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-05-09T03:08:33.774 DEBUG:teuthology.parallel:result is None 2024-05-09T03:08:33.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T03:08:34.012 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T03:08:34.040 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-3594.gc9da4049.el9 2024-05-09T03:08:34.041 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3594.gc9da4049.el9 2024-05-09T03:08:34.041 INFO:teuthology.task.install:The correct ceph version 19.0.0-3594.gc9da4049 is installed. 2024-05-09T03:08:34.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T03:08:34.261 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T03:08:34.285 INFO:teuthology.orchestra.run.smithi088.stdout:19.0.0-3594.gc9da4049.el9 2024-05-09T03:08:34.285 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3594.gc9da4049.el9 2024-05-09T03:08:34.285 INFO:teuthology.task.install:The correct ceph version 19.0.0-3594.gc9da4049 is installed. 2024-05-09T03:08:34.287 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T03:08:34.287 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:34.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T03:08:34.321 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:34.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T03:08:34.353 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T03:08:34.353 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:34.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T03:08:34.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T03:08:34.461 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:34.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T03:08:34.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T03:08:34.552 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T03:08:34.552 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:34.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T03:08:34.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T03:08:34.651 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:34.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T03:08:34.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T03:08:34.740 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T03:08:34.740 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:34.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T03:08:34.766 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T03:08:34.832 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:34.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T03:08:34.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T03:08:34.922 INFO:teuthology.run_tasks:Running task ceph... 2024-05-09T03:08:35.012 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-09T03:08:35.012 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-05-09T03:08:35.015 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2024-05-09T03:08:35.048 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-09T03:08:35.049 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T03:08:35.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T03:08:35.116 INFO:tasks.ceph:Creating extra log directories... 2024-05-09T03:08:35.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T03:08:35.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T03:08:35.195 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-09T03:08:35.195 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-09T03:08:35.195 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-09T03:08:35.196 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:35.196 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T03:08:35.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T03:08:35.353 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T03:08:35.378 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:35.379 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T03:08:35.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T03:08:35.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T03:08:35.551 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-09T03:08:35.551 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-09T03:08:35.551 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035', 'branch': 'wip-pdonnell-testing-20240508.183908-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7699035', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240508.183908-debug'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240508.183908-debug', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1722, 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'sleep_before_teardown': 0, 'subset': '67/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240508.183908-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHEta9nrY8jGuc+pKfcaGnqi3HrvC9EHyRvkS+7u6fZuMdS0ll8PIkLCuxU6DH1U67FNfRv+HPk3mvCfuiuaEG8=', 'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK78pRSTR6xVAAyJ1HyTjjv1aLm1CxGM5wDjlp7VqkQB5BWRA61Sme2bpZ07flu3DgaPEmE5OQAYdvE+Zm8pe3U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_damage']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-08_22:06:20', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-09T03:08:35.552 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T03:08:35.554 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T03:08:35.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T03:08:35.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T03:08:35.666 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:35.666 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T03:08:35.688 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T03:08:35.689 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-05-09 03:05:37.289019269 +0000 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-05-09 03:03:22.088798530 +0000 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-05-09 03:03:22.088798530 +0000 2024-05-09T03:08:35.744 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-05-09 03:03:22.088798530 +0000 2024-05-09T03:08:35.745 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T03:08:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-05-09T03:08:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-05-09T03:08:35.808 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 8.7649e-05 s, 5.8 MB/s 2024-05-09T03:08:35.809 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T03:08:35.866 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-05-09T03:08:35.922 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-05-09 03:05:37.289019269 +0000 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-05-09 03:03:22.430795081 +0000 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-05-09 03:03:22.430795081 +0000 2024-05-09T03:08:35.923 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-05-09 03:03:22.430795081 +0000 2024-05-09T03:08:35.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T03:08:35.986 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-05-09T03:08:35.986 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-05-09T03:08:35.986 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 8.5936e-05 s, 6.0 MB/s 2024-05-09T03:08:35.988 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T03:08:36.044 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-05-09T03:08:36.100 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T03:08:36.100 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:36.100 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-09T03:08:36.100 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:36.100 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:36.101 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-05-09 03:05:37.290019259 +0000 2024-05-09T03:08:36.101 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-05-09 03:03:22.777791583 +0000 2024-05-09T03:08:36.101 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-05-09 03:03:22.777791583 +0000 2024-05-09T03:08:36.101 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-05-09 03:03:22.777791583 +0000 2024-05-09T03:08:36.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T03:08:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-05-09T03:08:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-05-09T03:08:36.164 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.9429e-05 s, 5.1 MB/s 2024-05-09T03:08:36.165 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T03:08:36.222 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-05-09T03:08:36.278 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T03:08:36.278 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:36.278 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T03:08:36.278 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:36.278 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:36.279 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-05-09 03:05:37.290019259 +0000 2024-05-09T03:08:36.279 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-05-09 03:03:23.102788305 +0000 2024-05-09T03:08:36.279 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-05-09 03:03:23.102788305 +0000 2024-05-09T03:08:36.279 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-05-09 03:03:23.102788305 +0000 2024-05-09T03:08:36.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T03:08:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-05-09T03:08:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-05-09T03:08:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.7759e-05 s, 5.2 MB/s 2024-05-09T03:08:36.343 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T03:08:36.400 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:36.400 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:36.400 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T03:08:36.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T03:08:36.418 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-05-09T03:08:36.474 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T03:08:36.475 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:36.475 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-09T03:08:36.475 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:36.475 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:36.475 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-09 03:08:25.206724215 +0000 2024-05-09T03:08:36.475 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-09 03:03:19.254748722 +0000 2024-05-09T03:08:36.476 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-09 03:03:19.254748722 +0000 2024-05-09T03:08:36.476 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-09 03:03:19.254748722 +0000 2024-05-09T03:08:36.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T03:08:36.539 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-09T03:08:36.539 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-09T03:08:36.539 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000213413 s, 2.4 MB/s 2024-05-09T03:08:36.540 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T03:08:36.597 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-09 03:08:25.207724195 +0000 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-09 03:03:19.582742161 +0000 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-09 03:03:19.582742161 +0000 2024-05-09T03:08:36.654 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-09 03:03:19.582742161 +0000 2024-05-09T03:08:36.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T03:08:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-09T03:08:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-09T03:08:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000208955 s, 2.5 MB/s 2024-05-09T03:08:36.718 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T03:08:36.776 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-09 03:08:25.207724195 +0000 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-09 03:03:19.937735060 +0000 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-09 03:03:19.937735060 +0000 2024-05-09T03:08:36.832 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-09 03:03:19.937735060 +0000 2024-05-09T03:08:36.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T03:08:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-09T03:08:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-09T03:08:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000268289 s, 1.9 MB/s 2024-05-09T03:08:36.896 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T03:08:36.953 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-05-09 03:08:25.208724175 +0000 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-05-09 03:03:20.283728139 +0000 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-05-09 03:03:20.283728139 +0000 2024-05-09T03:08:37.009 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-05-09 03:03:20.283728139 +0000 2024-05-09T03:08:37.010 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T03:08:37.073 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-05-09T03:08:37.073 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-05-09T03:08:37.073 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000204674 s, 2.5 MB/s 2024-05-09T03:08:37.074 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T03:08:37.130 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:37.130 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T03:08:37.130 INFO:tasks.ceph:Generating config... 2024-05-09T03:08:37.132 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-09T03:08:37.132 INFO:tasks.ceph:[client] debug client = 20 2024-05-09T03:08:37.132 INFO:tasks.ceph:[client] debug ms = 1 2024-05-09T03:08:37.132 INFO:tasks.ceph:[client] pid file = 2024-05-09T03:08:37.132 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-09T03:08:37.132 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-09T03:08:37.132 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-05-09T03:08:37.133 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-09T03:08:37.133 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-09T03:08:37.133 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-09T03:08:37.133 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-09T03:08:37.133 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-09T03:08:37.133 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-09T03:08:37.133 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-09T03:08:37.134 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-09T03:08:37.134 INFO:tasks.ceph:Setting up mon.a... 2024-05-09T03:08:37.134 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-09T03:08:37.194 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2024-05-09T03:08:37.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-09T03:08:37.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:08:37.279 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46'), ('mon.b', '172.21.15.88'), ('mon.c', '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]')] 2024-05-09T03:08:37.279 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46,172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-09T03:08:37.280 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:37.280 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T03:08:37.334 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --add b 172.21.15.88 --addv c '[v2:172.21.15.88:3301,v1:172.21.15.88:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 36d71dd3-2926-4712-824e-80cfbafd76c2 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 36d71dd3-2926-4712-824e-80cfbafd76c2 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2024-05-09T03:08:37.432266+0000 2024-05-09T03:08:37.432 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-05-09T03:08:37.432266+0000 2024-05-09T03:08:37.433 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 17 (quincy) 2024-05-09T03:08:37.433 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2024-05-09T03:08:37.433 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-05-09T03:08:37.433 INFO:teuthology.orchestra.run.smithi046.stdout:1: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.b 2024-05-09T03:08:37.433 INFO:teuthology.orchestra.run.smithi046.stdout:2: [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] mon.c 2024-05-09T03:08:37.433 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-09T03:08:37.434 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T03:08:37.449 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 36d71dd3-2926-4712-824e-80cfbafd76c2... 2024-05-09T03:08:37.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T03:08:37.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T03:08:37.534 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.535 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T03:08:37.536 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.46,172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790]" 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 36d71dd3-2926-4712-824e-80cfbafd76c2 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2024-05-09T03:08:37.537 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug trim objects = true 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.538 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: bdev async discard = True 2024-05-09T03:08:37.539 INFO:teuthology.orchestra.run.smithi088.stdout: bdev enable discard = True 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore allocator = bitmap 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 1/20 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 1/20 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 4/10 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2024-05-09T03:08:37.540 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2024-05-09T03:08:37.541 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T03:08:37.542 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: mon down mkfs grace = 300 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: mon op complaint time = 120 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T03:08:37.543 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout: client mount timeout = 600 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = "" 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2024-05-09T03:08:37.544 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout:[mds] 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds = 20 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds balancer = 20 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug frag = True 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug scatterstat = True 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: mds op complaint time = 180 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: mds verify scatter = True 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2024-05-09T03:08:37.545 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-05-09T03:08:37.547 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-05-09T03:08:37.548 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-05-09T03:08:37.549 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,172.21.15.88,[v2:172.21.15.88:3301,v1:172.21.15.88:6790]" 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 36d71dd3-2926-4712-824e-80cfbafd76c2 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.550 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-05-09T03:08:37.551 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: bdev async discard = True 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: bdev enable discard = True 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = bitmap 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 1/20 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 1/20 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 4/10 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-05-09T03:08:37.552 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.553 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T03:08:37.554 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: mon down mkfs grace = 300 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: mon op complaint time = 120 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-05-09T03:08:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: client mount timeout = 600 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = "" 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-05-09T03:08:37.556 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout:[mds] 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 20 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 20 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug frag = True 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug scatterstat = True 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: mds op complaint time = 180 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: mds verify scatter = True 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-05-09T03:08:37.557 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-05-09T03:08:37.557 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-09T03:08:37.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-09T03:08:37.653 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-09T03:08:37.653 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:37.653 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-09T03:08:37.669 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:37.669 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-09T03:08:37.725 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:08:37.725 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:37.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T03:08:37.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:08:37.801 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:08:37.801 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:37.857 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:08:37.857 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:37.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T03:08:37.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T03:08:37.892 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:08:37.892 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:37.947 INFO:tasks.ceph:Setting up mon nodes... 2024-05-09T03:08:37.947 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-09T03:08:37.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T03:08:38.009 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T03:08:38.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T03:08:38.079 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T03:08:38.081 INFO:tasks.ceph:Setting up mds nodes... 2024-05-09T03:08:38.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T03:08:38.144 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T03:08:38.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-09T03:08:38.174 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T03:08:38.276 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T03:08:38.280 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-09T03:08:38.305 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T03:08:38.365 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T03:08:38.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-09T03:08:38.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T03:08:38.492 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T03:08:38.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-09T03:08:38.519 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-09T03:08:38.519 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T03:08:38.572 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-09T03:08:38.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-05-09T03:08:38.637 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-05-09T03:08:38.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-09T03:08:38.701 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-09T03:08:38.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-05-09T03:08:38.763 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-05-09T03:08:38.777 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-09T03:08:38.777 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T03:08:38.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-09T03:08:38.803 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:38.803 INFO:tasks.ceph:role: osd.0 2024-05-09T03:08:38.803 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:08:38.803 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T03:08:38.869 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:38.869 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:38.869 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:38.869 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:38.869 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:38.870 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:38.870 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:38.870 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:38.870 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:38.870 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:38.901 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-05-09T03:08:38.903 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-05-09T03:08:38.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-09T03:08:38.984 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-09T03:08:39.055 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-09T03:08:39.121 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:39.121 INFO:tasks.ceph:role: osd.1 2024-05-09T03:08:39.122 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:08:39.122 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T03:08:39.187 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:39.187 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:39.187 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:39.187 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:39.187 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:39.187 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:39.188 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:39.188 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:39.188 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:39.188 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:39.221 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-05-09T03:08:39.223 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-05-09T03:08:39.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-09T03:08:39.298 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-09T03:08:39.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-09T03:08:39.432 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:39.432 INFO:tasks.ceph:role: osd.2 2024-05-09T03:08:39.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:08:39.432 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T03:08:39.497 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:39.497 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:39.497 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:39.497 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:39.497 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:39.498 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:39.498 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:39.498 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:39.498 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:39.498 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:39.531 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-05-09T03:08:39.534 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-05-09T03:08:39.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-09T03:08:39.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-09T03:08:39.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-09T03:08:39.739 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:39.739 INFO:tasks.ceph:role: osd.3 2024-05-09T03:08:39.739 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:08:39.740 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T03:08:39.804 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:39.804 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:39.805 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:39.837 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-05-09T03:08:39.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-05-09T03:08:39.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-09T03:08:39.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-09T03:08:39.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:40.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:40.081+0000 7f002e7c0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-09T03:08:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:40.081+0000 7f002e7c0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-09T03:08:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:40.081+0000 7f002e7c0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T03:08:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:40.081+0000 7f002e7c0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T03:08:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:40.081+0000 7f002e7c0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T03:08:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:40.082+0000 7f002e7c0640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-09T03:08:42.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-09T03:08:42.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:43.068+0000 7fc7d31cb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-09T03:08:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:43.068+0000 7fc7d31cb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-09T03:08:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:43.068+0000 7fc7d31cb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T03:08:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:43.068+0000 7fc7d31cb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T03:08:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:43.068+0000 7fc7d31cb640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T03:08:43.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:43.068+0000 7fc7d31cb640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-09T03:08:45.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-09T03:08:45.944 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:46.042+0000 7fb2f54dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-09T03:08:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:46.042+0000 7fb2f54dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-09T03:08:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:46.042+0000 7fb2f54dd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T03:08:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:46.042+0000 7fb2f54dd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T03:08:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:46.042+0000 7fb2f54dd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T03:08:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:46.042+0000 7fb2f54dd640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-09T03:08:48.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-09T03:08:48.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:49.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:49.037+0000 7f198ad44640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-09T03:08:49.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:49.037+0000 7f198ad44640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-09T03:08:49.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:49.037+0000 7f198ad44640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T03:08:49.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:49.037+0000 7f198ad44640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T03:08:49.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:49.037+0000 7f198ad44640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T03:08:49.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:08:49.037+0000 7f198ad44640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-09T03:08:51.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-09T03:08:51.924 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-09T03:08:51.954 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:51.954 INFO:tasks.ceph:role: osd.4 2024-05-09T03:08:51.954 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:08:51.955 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T03:08:52.030 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:52.031 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:52.539 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-09T03:08:52.542 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-09T03:08:52.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-09T03:08:52.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-09T03:08:52.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-09T03:08:52.766 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:52.767 INFO:tasks.ceph:role: osd.5 2024-05-09T03:08:52.767 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:08:52.767 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T03:08:52.840 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:52.840 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:52.840 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:52.841 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:53.289 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-09T03:08:53.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-09T03:08:53.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-09T03:08:53.388 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-09T03:08:53.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-09T03:08:53.485 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:53.485 INFO:tasks.ceph:role: osd.6 2024-05-09T03:08:53.485 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:08:53.485 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:53.559 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:54.041 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-09T03:08:54.043 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-09T03:08:54.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-09T03:08:54.159 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-09T03:08:54.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-09T03:08:54.261 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T03:08:54.261 INFO:tasks.ceph:role: osd.7 2024-05-09T03:08:54.261 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:08:54.261 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T03:08:54.336 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T03:08:54.336 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T03:08:54.336 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T03:08:54.336 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T03:08:54.337 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T03:08:54.337 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-05-09T03:08:54.337 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T03:08:54.337 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T03:08:54.337 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T03:08:54.337 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T03:08:54.830 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-05-09T03:08:54.833 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-05-09T03:08:54.833 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-09T03:08:54.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-09T03:08:54.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:55.094+0000 7fba65c18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-09T03:08:55.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:55.094+0000 7fba65c18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-09T03:08:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:55.094+0000 7fba65c18640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T03:08:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:55.094+0000 7fba65c18640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T03:08:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:55.094+0000 7fba65c18640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T03:08:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:55.095+0000 7fba65c18640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-09T03:08:57.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-09T03:08:57.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:08:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:58.074+0000 7f717de15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-09T03:08:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:58.075+0000 7f717de15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-09T03:08:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:58.075+0000 7f717de15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T03:08:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:58.075+0000 7f717de15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T03:08:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:58.075+0000 7f717de15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T03:08:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:08:58.075+0000 7f717de15640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-09T03:09:00.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-09T03:09:00.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:09:01.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:01.063+0000 7ffbddc0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-09T03:09:01.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:01.064+0000 7ffbddc0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-09T03:09:01.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:01.064+0000 7ffbddc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T03:09:01.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:01.064+0000 7ffbddc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T03:09:01.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:01.064+0000 7ffbddc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T03:09:01.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:01.064+0000 7ffbddc0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-09T03:09:03.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-09T03:09:03.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:09:04.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:04.044+0000 7f4515812640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-09T03:09:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:04.044+0000 7f4515812640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-09T03:09:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:04.044+0000 7f4515812640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T03:09:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:04.044+0000 7f4515812640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T03:09:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:04.044+0000 7f4515812640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T03:09:04.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:04.045+0000 7f4515812640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-09T03:09:05.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:09:05.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:09:06.904 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-09T03:09:06.933 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-09T03:09:06.933 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:06.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-09T03:09:06.965 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:06.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-09T03:09:07.035 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-09T03:09:07.105 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-09T03:09:07.175 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-09T03:09:07.244 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-09T03:09:07.313 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-09T03:09:07.382 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.383 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-09T03:09:07.412 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-09T03:09:07.481 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-09T03:09:07.548 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-09T03:09:07.614 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.615 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-09T03:09:07.680 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.680 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-09T03:09:07.746 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-09T03:09:07.812 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.812 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-09T03:09:07.831 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:07.832 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-05-09T03:09:07.890 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.890 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-09T03:09:07.906 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:07.906 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-05-09T03:09:07.961 INFO:tasks.ceph:Adding keys to all mons... 2024-05-09T03:09:07.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T03:09:07.964 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T03:09:08.005 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.y] 2024-05-09T03:09:08.005 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ== 2024-05-09T03:09:08.005 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-05-09T03:09:08.005 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g== 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw== 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw== 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA== 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg== 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA== 2024-05-09T03:09:08.006 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.6] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.7] 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw== 2024-05-09T03:09:08.007 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout:[client.2] 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg== 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout:[client.1] 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ== 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout:[client.3] 2024-05-09T03:09:08.008 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg== 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ== 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g== 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw== 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw== 2024-05-09T03:09:08.025 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA== 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg== 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA== 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg== 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw== 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2024-05-09T03:09:08.026 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[client.2] 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg== 2024-05-09T03:09:08.027 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2024-05-09T03:09:08.028 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ== 2024-05-09T03:09:08.028 INFO:teuthology.orchestra.run.smithi088.stdout:[client.3] 2024-05-09T03:09:08.028 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg== 2024-05-09T03:09:08.029 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:09:08.052 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:09:08.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.201 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.265 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.268 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.329 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.391 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:09:08.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T03:09:08.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.584 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T03:09:08.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.779 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T03:09:08.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:08.968 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T03:09:09.230 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-09T03:09:09.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-09T03:09:09.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T03:09:09.365 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-09T03:09:09.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-09T03:09:09.416 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T03:09:09.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-09T03:09:09.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-09T03:09:09.612 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T03:09:09.720 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-09T03:09:09.744 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:09:09.747 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T03:09:09.800 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-09T03:09:09.800 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T03:09:09.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T03:09:09.804 INFO:tasks.ceph.mon.a:Started 2024-05-09T03:09:09.804 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T03:09:09.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T03:09:09.843 INFO:tasks.ceph.mon.b:Started 2024-05-09T03:09:09.843 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T03:09:09.843 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T03:09:09.845 INFO:tasks.ceph.mon.c:Started 2024-05-09T03:09:09.845 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-09T03:09:09.845 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-09T03:09:09.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-09T03:09:09.850 INFO:tasks.ceph.mgr.y:Started 2024-05-09T03:09:09.850 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-09T03:09:09.850 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-09T03:09:09.852 INFO:tasks.ceph.mgr.x:Started 2024-05-09T03:09:09.852 INFO:tasks.ceph:Setting crush tunables to default 2024-05-09T03:09:09.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2024-05-09T03:09:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.966+0000 7f79fc049640 1 Processor -- start 2024-05-09T03:09:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.967+0000 7f79fc049640 1 -- start start 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fc049640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fc049640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 0x7f79f40d3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fc049640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f406dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fc049640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f79f408dd90 con 0x7f79f406e990 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fc049640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f79f40d2160 con 0x7f79f40b2fd0 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fc049640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f79f40d2440 con 0x7f79f40d3740 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57880/0 (socket says 172.21.15.46:57880) 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 -- 172.21.15.46:0/1211491299 learned_addr learned my addr 172.21.15.46:0/1211491299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:09.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79fa5e6640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f406dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f95e4640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 0x7f79f40d3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 -- 172.21.15.46:0/1211491299 >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 msgr2=0x7f79f408d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 -- 172.21.15.46:0/1211491299 >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 msgr2=0x7f79f408d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f9de5640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f95e4640 1 -- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 msgr2=0x7f79f40d3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f95e4640 1 -- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 msgr2=0x7f79f40d3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.968+0000 7f79f95e4640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 0x7f79f40d3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.969+0000 7f79f95e4640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 0x7f79f40d3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.969+0000 7f79fa5e6640 1 -- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 msgr2=0x7f79f406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.969+0000 7f79fa5e6640 1 -- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 msgr2=0x7f79f406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.969+0000 7f79fa5e6640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f406dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:09:09.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:09.969+0000 7f79fa5e6640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f406dd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T03:09:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.169+0000 7f79f9de5640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.46:3300/0 conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.169+0000 7f79f9de5640 1 -- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 msgr2=0x7f79f406dd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T03:09:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.169+0000 7f79f9de5640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f406dd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.169+0000 7f79f9de5640 1 -- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 msgr2=0x7f79f40d3200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T03:09:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.169+0000 7f79f9de5640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 0x7f79f40d3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.169+0000 7f79f9de5640 1 -- 172.21.15.46:0/1211491299 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f40d2720 con 0x7f79f406e990 2024-05-09T03:09:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.170+0000 7f79f9de5640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f408d850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79f408ce10 tx=0x7f79e802a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d9f947b392ffb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.170+0000 7f79f8de3640 1 -- 172.21.15.46:0/1211491299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8037070 con 0x7f79f406e990 2024-05-09T03:09:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.170+0000 7f79f8de3640 1 -- 172.21.15.46:0/1211491299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e8032640 con 0x7f79f406e990 2024-05-09T03:09:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.170+0000 7f79f8de3640 1 -- 172.21.15.46:0/1211491299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e800f040 con 0x7f79f406e990 2024-05-09T03:09:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 -- 172.21.15.46:0/1211491299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 msgr2=0x7f79f408d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f408d850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79f408ce10 tx=0x7f79e802a370 comp rx=0 tx=0).stop 2024-05-09T03:09:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 -- 172.21.15.46:0/1211491299 shutdown_connections 2024-05-09T03:09:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f406dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 --2- 172.21.15.46:0/1211491299 >> v2:172.21.15.88:3300/0 conn(0x7f79f40b2fd0 0x7f79f40d3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 --2- 172.21.15.46:0/1211491299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f408d850 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 -- 172.21.15.46:0/1211491299 >> 172.21.15.46:0/1211491299 conn(0x7f79f40b5640 msgr2=0x7f79f40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 -- 172.21.15.46:0/1211491299 shutdown_connections 2024-05-09T03:09:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.171+0000 7f79fc049640 1 -- 172.21.15.46:0/1211491299 wait complete. 2024-05-09T03:09:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.173+0000 7f79fc049640 1 Processor -- start 2024-05-09T03:09:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.173+0000 7f79fc049640 1 -- start start 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.174+0000 7f79fc049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f40b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.174+0000 7f79fc049640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79f40b2fd0 0x7f79f41618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.174+0000 7f79fc049640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f415d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.174+0000 7f79fc049640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f79f406e2c0 con 0x7f79f406e990 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.174+0000 7f79fc049640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f79f40d2ce0 con 0x7f79f40b2fd0 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79fc049640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f79f406e440 con 0x7f79f40d3740 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f40b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f40b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57938/0 (socket says 172.21.15.46:57938) 2024-05-09T03:09:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 -- 172.21.15.46:0/3830557655 learned_addr learned my addr 172.21.15.46:0/3830557655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79fa5e6640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f415d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f95e4640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79f40b2fd0 0x7f79f41618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 -- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 msgr2=0x7f79f415d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f415d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 -- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79f40b2fd0 msgr2=0x7f79f41618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79f40b2fd0 0x7f79f41618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 -- 172.21.15.46:0/3830557655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f41f9ca0 con 0x7f79f406e990 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79f9de5640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f40b4ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009970 tx=0x7f79e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8037050 con 0x7f79f406e990 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e8043070 con 0x7f79f406e990 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79fc049640 1 -- 172.21.15.46:0/3830557655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f4161e20 con 0x7f79f406e990 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.175+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8030b40 con 0x7f79f406e990 2024-05-09T03:09:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.176+0000 7f79fc049640 1 -- 172.21.15.46:0/3830557655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f40ca1e0 con 0x7f79f406e990 2024-05-09T03:09:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.176+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f79e8030ce0 con 0x7f79f406e990 2024-05-09T03:09:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.176+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f79e803e450 con 0x7f79f406e990 2024-05-09T03:09:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.177+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f406a270 con 0x7f79f406e990 2024-05-09T03:09:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.179+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f79f41f9ca0 con 0x7f79f406e990 2024-05-09T03:09:10.230 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:10.228+0000 7fdad1033040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T03:09:10.243 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:10.241+0000 7fd063723040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T03:09:10.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.247+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f79f40b62a0 con 0x7f79f406e990 2024-05-09T03:09:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.248+0000 7f79e67fc640 1 -- 172.21.15.46:0/3830557655 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f79e8005990 con 0x7f79f406e990 2024-05-09T03:09:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-05-09T03:09:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 msgr2=0x7f79f40b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f40b4ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009970 tx=0x7f79e8004730 comp rx=0 tx=0).stop 2024-05-09T03:09:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 shutdown_connections 2024-05-09T03:09:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f79f40d3740 0x7f79f415d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79f40b2fd0 0x7f79f41618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 --2- 172.21.15.46:0/3830557655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79f406e990 0x7f79f40b4ea0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 >> 172.21.15.46:0/3830557655 conn(0x7f79f40b5640 msgr2=0x7f79f408cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 shutdown_connections 2024-05-09T03:09:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:10.249+0000 7f79d7fff640 1 -- 172.21.15.46:0/3830557655 wait complete. 2024-05-09T03:09:10.262 INFO:tasks.ceph:check_enable_crimson: False 2024-05-09T03:09:10.262 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-09T03:09:10.263 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:10.263 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-09T03:09:10.289 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:10.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-09T03:09:10.356 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:10.356 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-09T03:09:10.384 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:10.383+0000 7fdad1033040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:09:10.413 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:10.412+0000 7fd063723040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:09:10.422 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:10.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-09T03:09:10.436 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:10.435+0000 7fdad1033040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T03:09:10.470 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:10.468+0000 7fd063723040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T03:09:10.488 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:10.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-09T03:09:10.513 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:10.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-09T03:09:10.578 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:10.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-09T03:09:10.644 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:10.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-09T03:09:10.704 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:10.703+0000 7fdad1033040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T03:09:10.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 2e91e1eb-2bae-45d0-935b-ec079c920ca5 0 2024-05-09T03:09:10.721 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:10.720+0000 7fd063723040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T03:09:10.751 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:10.750+0000 7fdad1033040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T03:09:10.769 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:10.768+0000 7fd063723040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T03:09:10.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.862+0000 7fe56e029640 1 Processor -- start 2024-05-09T03:09:10.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.862+0000 7fe56e029640 1 -- start start 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe56e029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe5680d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe56e029640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5680b7e10 0x7fe5680b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe56e029640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe568160850 con 0x7fe5680ccff0 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe56e029640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe5680d2510 con 0x7fe5680b7e10 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe56e029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe5680cfb90 con 0x7fe5680b4080 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe5680d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe566ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe5680ccff0 0x7fe5680b7700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43662/0 (socket says 172.21.15.88:43662) 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe566ffd640 1 -- 172.21.15.88:0/1171427719 learned_addr learned my addr 172.21.15.88:0/1171427719 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.863+0000 7fe567fff640 1 --2- 172.21.15.88:0/1171427719 >> v2:172.21.15.88:3300/0 conn(0x7fe5680b7e10 0x7fe5680b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5677fe640 1 -- 172.21.15.88:0/1171427719 >> v2:172.21.15.88:3300/0 conn(0x7fe5680b7e10 msgr2=0x7fe5680b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5677fe640 1 --2- 172.21.15.88:0/1171427719 >> v2:172.21.15.88:3300/0 conn(0x7fe5680b7e10 0x7fe5680b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5677fe640 1 -- 172.21.15.88:0/1171427719 >> v1:172.21.15.88:6789/0 conn(0x7fe5680ccff0 legacy=0x7fe5680b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5677fe640 1 -- 172.21.15.88:0/1171427719 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5680cfe70 con 0x7fe5680b4080 2024-05-09T03:09:10.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe567fff640 1 --2- 172.21.15.88:0/1171427719 >> v2:172.21.15.88:3300/0 conn(0x7fe5680b7e10 0x7fe5680b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:10.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5677fe640 1 --2- 172.21.15.88:0/1171427719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe5680d0570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5680d2690 tx=0x7fe55802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73e31454b349e56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:10.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5667fc640 1 -- 172.21.15.88:0/1171427719 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558037070 con 0x7fe5680b4080 2024-05-09T03:09:10.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5667fc640 1 -- 172.21.15.88:0/1171427719 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe558032440 con 0x7fe5680b4080 2024-05-09T03:09:10.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.864+0000 7fe5667fc640 1 -- 172.21.15.88:0/1171427719 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558032810 con 0x7fe5680b4080 2024-05-09T03:09:10.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 -- 172.21.15.88:0/1171427719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 msgr2=0x7fe5680d0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 --2- 172.21.15.88:0/1171427719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe5680d0570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe5680d2690 tx=0x7fe55802a180 comp rx=0 tx=0).stop 2024-05-09T03:09:10.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 -- 172.21.15.88:0/1171427719 shutdown_connections 2024-05-09T03:09:10.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 --2- 172.21.15.88:0/1171427719 >> v2:172.21.15.88:3300/0 conn(0x7fe5680b7e10 0x7fe5680b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 --2- 172.21.15.88:0/1171427719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe5680d0570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 -- 172.21.15.88:0/1171427719 >> 172.21.15.88:0/1171427719 conn(0x7fe56809b7d0 msgr2=0x7fe56809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:10.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 -- 172.21.15.88:0/1171427719 shutdown_connections 2024-05-09T03:09:10.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.865+0000 7fe56e029640 1 -- 172.21.15.88:0/1171427719 wait complete. 2024-05-09T03:09:10.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.867+0000 7fe56e029640 1 Processor -- start 2024-05-09T03:09:10.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.867+0000 7fe56e029640 1 -- start start 2024-05-09T03:09:10.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.868+0000 7fe56e029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe568160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.868+0000 7fe56e029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 0x7fe56814cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.868+0000 7fe56e029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe56814d300 0x7fe5681496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.868+0000 7fe56e029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe5680d1f60 con 0x7fe56814d300 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe56e029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5680d1de0 con 0x7fe5680ccff0 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe56e029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe5680cf7b0 con 0x7fe5680b4080 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 0x7fe56814cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe568160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe5677fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe568160e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40028/0 (socket says 172.21.15.88:40028) 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 0x7fe56814cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49396/0 (socket says 172.21.15.88:49396) 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 -- 172.21.15.88:0/617578769 learned_addr learned my addr 172.21.15.88:0/617578769 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe567fff640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe56814d300 0x7fe5681496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 -- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 msgr2=0x7fe568160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe568160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 -- 172.21.15.88:0/617578769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe56814d300 msgr2=0x7fe5681496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe56814d300 0x7fe5681496e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 -- 172.21.15.88:0/617578769 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5680cbe20 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe5677fe640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe568160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe566ffd640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 0x7fe56814cdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe55c001eb0 tx=0x7fe55c00c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c017070 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe55c005d00 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5681613c0 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c007310 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.869+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5680cc090 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.870+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe55c01a070 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.870+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fe55c012400 con 0x7fe5680ccff0 2024-05-09T03:09:10.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.870+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe568070480 con 0x7fe5680ccff0 2024-05-09T03:09:10.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.873+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe568070480 con 0x7fe5680ccff0 2024-05-09T03:09:10.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.944+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2e91e1eb-2bae-45d0-935b-ec079c920ca5", "id": 0} v 0) -- 0x7fe56806a270 con 0x7fe5680ccff0 2024-05-09T03:09:10.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.948+0000 7fe54ffff640 1 -- 172.21.15.88:0/617578769 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2e91e1eb-2bae-45d0-935b-ec079c920ca5", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fe55c021020 con 0x7fe5680ccff0 2024-05-09T03:09:10.949 INFO:teuthology.orchestra.run.smithi088.stdout:0 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 msgr2=0x7fe56814cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 0x7fe56814cdc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe55c001eb0 tx=0x7fe55c00c7d0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 shutdown_connections 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe56814d300 0x7fe5681496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5680ccff0 0x7fe56814cdc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 --2- 172.21.15.88:0/617578769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5680b4080 0x7fe568160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 >> 172.21.15.88:0/617578769 conn(0x7fe56809b7d0 msgr2=0x7fe5680d2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 shutdown_connections 2024-05-09T03:09:10.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:10.949+0000 7fe56e029640 1 -- 172.21.15.88:0/617578769 wait complete. 2024-05-09T03:09:10.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new a15e320a-d34c-4d31-9f0d-9cfdd145adcd 1 2024-05-09T03:09:11.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.070+0000 7f116acd1640 1 Processor -- start 2024-05-09T03:09:11.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.070+0000 7f116acd1640 1 -- start start 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116acd1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f116415e200 0x7f11640b2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116acd1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 0x7f11640d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116acd1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11640b2e70 con 0x7f11641627e0 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116acd1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f116415e610 con 0x7f116415e200 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116acd1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11640d1690 con 0x7f1164070940 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f1168a6d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f116415e200 0x7f11640b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116926e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 0x7f11640d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f1168a6d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f116415e200 0x7f11640b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49404/0 (socket says 172.21.15.88:49404) 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f116926e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 0x7f11640d23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40030/0 (socket says 172.21.15.88:40030) 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.071+0000 7f1168a6d640 1 -- 172.21.15.88:0/1397369891 learned_addr learned my addr 172.21.15.88:0/1397369891 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.072+0000 7f1168a6d640 1 -- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 msgr2=0x7f11640d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.072+0000 7f1168a6d640 1 --2- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 0x7f11640d23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.072+0000 7f1168a6d640 1 -- 172.21.15.88:0/1397369891 >> v1:172.21.15.46:6789/0 conn(0x7f11641627e0 legacy=0x7f1164070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.072+0000 7f1168a6d640 1 -- 172.21.15.88:0/1397369891 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11640d18e0 con 0x7f116415e200 2024-05-09T03:09:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.072+0000 7f116926e640 1 --2- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 0x7f11640d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.072+0000 7f1168a6d640 1 --2- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f11640b2770 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1158009870 tx=0x7f115802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c70d2011413e82c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f11637fe640 1 -- 172.21.15.88:0/1397369891 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158037070 con 0x7f116415e200 2024-05-09T03:09:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f11637fe640 1 -- 172.21.15.88:0/1397369891 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1158032440 con 0x7f116415e200 2024-05-09T03:09:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f11637fe640 1 -- 172.21.15.88:0/1397369891 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158032810 con 0x7f116415e200 2024-05-09T03:09:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f116acd1640 1 -- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 msgr2=0x7f11640b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f116acd1640 1 --2- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f11640b2770 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1158009870 tx=0x7f115802a270 comp rx=0 tx=0).stop 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f116acd1640 1 -- 172.21.15.88:0/1397369891 shutdown_connections 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f116acd1640 1 --2- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1164070940 0x7f11640d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f116acd1640 1 --2- 172.21.15.88:0/1397369891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f11640b2770 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.073+0000 7f116acd1640 1 -- 172.21.15.88:0/1397369891 >> 172.21.15.88:0/1397369891 conn(0x7f11640d3ac0 msgr2=0x7f11640d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.074+0000 7f116acd1640 1 -- 172.21.15.88:0/1397369891 shutdown_connections 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.074+0000 7f116acd1640 1 -- 172.21.15.88:0/1397369891 wait complete. 2024-05-09T03:09:11.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.075+0000 7f116acd1640 1 Processor -- start 2024-05-09T03:09:11.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.075+0000 7f116acd1640 1 -- start start 2024-05-09T03:09:11.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116acd1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f116414ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116acd1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11641627e0 0x7f116406e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116acd1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f116414f930 0x7f116414fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116acd1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1164162bf0 con 0x7f116414f930 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116acd1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11640d1f30 con 0x7f116415e200 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116acd1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11640d2900 con 0x7f11641627e0 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f116414ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1163fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11641627e0 0x7f116406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f116414ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49414/0 (socket says 172.21.15.88:49414) 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 -- 172.21.15.88:0/2488160982 learned_addr learned my addr 172.21.15.88:0/2488160982 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f116926e640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f116414f930 0x7f116414fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 -- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11641627e0 msgr2=0x7f116406e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11641627e0 0x7f116406e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 -- 172.21.15.88:0/2488160982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f116414f930 msgr2=0x7f116414fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f116414f930 0x7f116414fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 -- 172.21.15.88:0/2488160982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1164150280 con 0x7f116415e200 2024-05-09T03:09:11.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f1168a6d640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f116414ca00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1158000c00 tx=0x7f11580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.077+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158037040 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.078+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1158042070 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.078+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f116406eae0 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.078+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1158030880 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.078+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f116406f0a0 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.078+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1158030a20 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.078+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f1158047c20 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.079+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f116406a270 con 0x7f116415e200 2024-05-09T03:09:11.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.081+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f1164150280 con 0x7f116415e200 2024-05-09T03:09:11.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.154+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a15e320a-d34c-4d31-9f0d-9cfdd145adcd", "id": 1} v 0) -- 0x7f116406d8f0 con 0x7f116415e200 2024-05-09T03:09:11.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.158+0000 7f11617fa640 1 -- 172.21.15.88:0/2488160982 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a15e320a-d34c-4d31-9f0d-9cfdd145adcd", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f1158047460 con 0x7f116415e200 2024-05-09T03:09:11.159 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-05-09T03:09:11.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 msgr2=0x7f116414ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f116414ca00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1158000c00 tx=0x7f11580046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 shutdown_connections 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f116414f930 0x7f116414fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11641627e0 0x7f116406e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 --2- 172.21.15.88:0/2488160982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f116415e200 0x7f116414ca00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 >> 172.21.15.88:0/2488160982 conn(0x7f11640d3ac0 msgr2=0x7f11640b4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 shutdown_connections 2024-05-09T03:09:11.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.159+0000 7f116acd1640 1 -- 172.21.15.88:0/2488160982 wait complete. 2024-05-09T03:09:11.171 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16 2 2024-05-09T03:09:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.280+0000 7fcfda07e640 1 Processor -- start 2024-05-09T03:09:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.280+0000 7fcfda07e640 1 -- start start 2024-05-09T03:09:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.281+0000 7fcfda07e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcfd40ccff0 0x7fcfd40d0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.281+0000 7fcfda07e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd4160850 con 0x7fcfd40b4080 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.281+0000 7fcfda07e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd40cfb90 con 0x7fcfd40b7e10 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.281+0000 7fcfda07e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcfd40d2510 con 0x7fcfd40ccff0 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.281+0000 7fcfd2ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcfd40b7e10 0x7fcfd40b23a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43678/0 (socket says 172.21.15.88:43678) 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.281+0000 7fcfd2ffd640 1 -- 172.21.15.88:0/3660213054 learned_addr learned my addr 172.21.15.88:0/3660213054 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd3fff640 1 --2- 172.21.15.88:0/3660213054 >> v2:172.21.15.46:3300/0 conn(0x7fcfd40ccff0 0x7fcfd40d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167303198 0 0) 0x7fcfd40cfb90 con 0x7fcfd40b7e10 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfbc003520 con 0x7fcfd40b7e10 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249414632 0 0) 0x7fcfd4160850 con 0x7fcfd40b4080 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfd40cfb90 con 0x7fcfd40b4080 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3660213054 >> v1:172.21.15.88:6789/0 conn(0x7fcfd40b7e10 legacy=0x7fcfd40b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2838386725 0 0) 0x7fcfbc003520 con 0x7fcfd40b7e10 2024-05-09T03:09:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3660213054 >> v1:172.21.15.46:6789/0 conn(0x7fcfd40b4080 legacy=0x7fcfd40b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.282+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3660213054 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd40cfe70 con 0x7fcfd40ccff0 2024-05-09T03:09:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.283+0000 7fcfd3fff640 1 --2- 172.21.15.88:0/3660213054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd40d0570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40d2690 tx=0x7fcfcc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9baf096393cc7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.283+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc40016f0 con 0x7fcfd40ccff0 2024-05-09T03:09:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.283+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfcc007540 con 0x7fcfd40ccff0 2024-05-09T03:09:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.283+0000 7fcfd27fc640 1 -- 172.21.15.88:0/3660213054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc007710 con 0x7fcfd40ccff0 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 -- 172.21.15.88:0/3660213054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 msgr2=0x7fcfd40d0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 --2- 172.21.15.88:0/3660213054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd40d0570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40d2690 tx=0x7fcfcc02a6a0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 -- 172.21.15.88:0/3660213054 shutdown_connections 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 --2- 172.21.15.88:0/3660213054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd40d0570 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 -- 172.21.15.88:0/3660213054 >> 172.21.15.88:0/3660213054 conn(0x7fcfd409b7d0 msgr2=0x7fcfd409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 -- 172.21.15.88:0/3660213054 shutdown_connections 2024-05-09T03:09:11.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.284+0000 7fcfda07e640 1 -- 172.21.15.88:0/3660213054 wait complete. 2024-05-09T03:09:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.286+0000 7fcfda07e640 1 Processor -- start 2024-05-09T03:09:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.286+0000 7fcfda07e640 1 -- start start 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfda07e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 0x7fcfd41f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfda07e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd406d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfda07e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 0x7fcfd406dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfda07e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd40d1f60 con 0x7fcfd40ccff0 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfda07e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd40d1de0 con 0x7fcfd414eb50 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfda07e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd40cf7b0 con 0x7fcfd40b4080 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfd3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 0x7fcfd406dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfd37fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 0x7fcfd41f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfd3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 0x7fcfd406dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49430/0 (socket says 172.21.15.88:49430) 2024-05-09T03:09:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3266832116 learned_addr learned my addr 172.21.15.88:0/3266832116 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfd37fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 0x7fcfd41f6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40038/0 (socket says 172.21.15.88:40038) 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.287+0000 7fcfd2ffd640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd406d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 msgr2=0x7fcfd41f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd3fff640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 0x7fcfd41f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3266832116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 msgr2=0x7fcfd406d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd3fff640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd406d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd3fff640 1 -- 172.21.15.88:0/3266832116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd41f68b0 con 0x7fcfd414eb50 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd2ffd640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd406d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd37fe640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 0x7fcfd41f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfd3fff640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 0x7fcfd406dc80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009e50 tx=0x7fcfcc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc03f070 con 0x7fcfd414eb50 2024-05-09T03:09:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfcc03b820 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd4160e70 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc03bbd0 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd40b1b30 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcfcc044460 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.288+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fcfcc03bdc0 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.289+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd406a270 con 0x7fcfd414eb50 2024-05-09T03:09:11.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.291+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcfd41f68b0 con 0x7fcfd414eb50 2024-05-09T03:09:11.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.363+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16", "id": 2} v 0) -- 0x7fcfd41604a0 con 0x7fcfd414eb50 2024-05-09T03:09:11.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.367+0000 7fcfbbfff640 1 -- 172.21.15.88:0/3266832116 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fcfcc031400 con 0x7fcfd414eb50 2024-05-09T03:09:11.368 INFO:teuthology.orchestra.run.smithi088.stdout:2 2024-05-09T03:09:11.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 msgr2=0x7fcfd406dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 0x7fcfd406dc80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009e50 tx=0x7fcfcc004a40 comp rx=0 tx=0).stop 2024-05-09T03:09:11.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 shutdown_connections 2024-05-09T03:09:11.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfd414eb50 0x7fcfd406dc80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfd40ccff0 0x7fcfd406d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 --2- 172.21.15.88:0/3266832116 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcfd40b4080 0x7fcfd41f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 >> 172.21.15.88:0/3266832116 conn(0x7fcfd409b7d0 msgr2=0x7fcfd4070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 shutdown_connections 2024-05-09T03:09:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.368+0000 7fcfda07e640 1 -- 172.21.15.88:0/3266832116 wait complete. 2024-05-09T03:09:11.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 29e7216f-bf54-4c12-9ce2-a567f991c9b2 3 2024-05-09T03:09:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.491+0000 7f78996f9640 1 Processor -- start 2024-05-09T03:09:11.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.491+0000 7f78996f9640 1 -- start start 2024-05-09T03:09:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78996f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f789415de40 0x7f78941611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78996f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940b7760 0x7f78940b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78996f9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78940b80b0 con 0x7f78940d47a0 2024-05-09T03:09:11.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78996f9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f78940d10b0 con 0x7f789415de40 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78996f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f78940d1390 con 0x7f78940b7760 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78927fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f789415de40 0x7f78941611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78937fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940b7760 0x7f78940b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78927fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f789415de40 0x7f78941611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49436/0 (socket says 172.21.15.88:49436) 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.492+0000 7f78927fc640 1 -- 172.21.15.88:0/651107659 learned_addr learned my addr 172.21.15.88:0/651107659 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f78927fc640 1 -- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940b7760 msgr2=0x7f78940b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f78927fc640 1 --2- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940b7760 0x7f78940b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f78927fc640 1 -- 172.21.15.88:0/651107659 >> v1:172.21.15.46:6789/0 conn(0x7f78940d47a0 legacy=0x7f78940d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f78927fc640 1 -- 172.21.15.88:0/651107659 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78940d1670 con 0x7f789415de40 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f78937fe640 1 --2- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940b7760 0x7f78940b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:11.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f78927fc640 1 --2- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78941611c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7888009870 tx=0x7f788802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=818bdc6c043b4ba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.493+0000 7f7891ffb640 1 -- 172.21.15.88:0/651107659 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888037070 con 0x7f789415de40 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f7891ffb640 1 -- 172.21.15.88:0/651107659 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7888032440 con 0x7f789415de40 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f7891ffb640 1 -- 172.21.15.88:0/651107659 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888032810 con 0x7f789415de40 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 -- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 msgr2=0x7f78941611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 --2- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78941611c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7888009870 tx=0x7f788802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 -- 172.21.15.88:0/651107659 shutdown_connections 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 --2- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940b7760 0x7f78940b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 --2- 172.21.15.88:0/651107659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78941611c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 -- 172.21.15.88:0/651107659 >> 172.21.15.88:0/651107659 conn(0x7f78940d2760 msgr2=0x7f78940d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 -- 172.21.15.88:0/651107659 shutdown_connections 2024-05-09T03:09:11.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.494+0000 7f78996f9640 1 -- 172.21.15.88:0/651107659 wait complete. 2024-05-09T03:09:11.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.496+0000 7f78996f9640 1 Processor -- start 2024-05-09T03:09:11.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.496+0000 7f78996f9640 1 -- start start 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78996f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7760 0x7f78940cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78996f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 0x7f789415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78996f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78940cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78996f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7894161700 con 0x7f78940b7760 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78996f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f789415e250 con 0x7f789415de40 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78996f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7894161880 con 0x7f78940d47a0 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78940cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78927fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 0x7f789415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78927fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 0x7f789415c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40052/0 (socket says 172.21.15.88:40052) 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78940cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49448/0 (socket says 172.21.15.88:49448) 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78927fc640 1 -- 172.21.15.88:0/1896002817 learned_addr learned my addr 172.21.15.88:0/1896002817 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f7892ffd640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7760 0x7f78940cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 -- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 msgr2=0x7f789415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 0x7f789415c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 -- 172.21.15.88:0/1896002817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7760 msgr2=0x7f78940cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7760 0x7f78940cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 -- 172.21.15.88:0/1896002817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78940cc8e0 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78927fc640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 0x7f789415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f78937fe640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78940cda50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f788c00ef90 tx=0x7f788c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.498+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c018030 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788c00cd30 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78940cc600 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c003bb0 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78940ca630 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f788c005d10 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f788c01d8a0 con 0x7f789415de40 2024-05-09T03:09:11.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.499+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f789406a270 con 0x7f789415de40 2024-05-09T03:09:11.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.503+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f788c022070 con 0x7f789415de40 2024-05-09T03:09:11.517 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:11.517+0000 7fdad1033040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T03:09:11.532 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:11.530+0000 7fd063723040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T03:09:11.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.576+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "29e7216f-bf54-4c12-9ce2-a567f991c9b2", "id": 3} v 0) -- 0x7f78941603f0 con 0x7f789415de40 2024-05-09T03:09:11.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.580+0000 7f787b7fe640 1 -- 172.21.15.88:0/1896002817 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29e7216f-bf54-4c12-9ce2-a567f991c9b2", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f788c010040 con 0x7f789415de40 2024-05-09T03:09:11.581 INFO:teuthology.orchestra.run.smithi088.stdout:3 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 msgr2=0x7f78940cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78940cda50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f788c00ef90 tx=0x7f788c00c490 comp rx=0 tx=0).stop 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 shutdown_connections 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f789415de40 0x7f78940cda50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d47a0 0x7f789415c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 --2- 172.21.15.88:0/1896002817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7760 0x7f78940cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 >> 172.21.15.88:0/1896002817 conn(0x7f78940d2760 msgr2=0x7f78940b6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 shutdown_connections 2024-05-09T03:09:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.581+0000 7f78996f9640 1 -- 172.21.15.88:0/1896002817 wait complete. 2024-05-09T03:09:11.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new e2b044e3-c608-40af-b9b7-6f04bf54b508 4 2024-05-09T03:09:11.611 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:11.610+0000 7fdad1033040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T03:09:11.624 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:11.622+0000 7fd063723040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T03:09:11.656 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:11.655+0000 7fdad1033040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T03:09:11.669 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:11.667+0000 7fd063723040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T03:09:11.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.702+0000 7f4e16a2b640 1 Processor -- start 2024-05-09T03:09:11.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.702+0000 7f4e16a2b640 1 -- start start 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e16a2b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e1006f920 0x7f4e100ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e16a2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e100b42a0 con 0x7f4e100b3d20 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e16a2b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e100d3730 con 0x7f4e100cfdf0 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e16a2b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4e100b2d40 con 0x7f4e1006f920 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e14fc8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4e100cfdf0 0x7f4e100d3020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43690/0 (socket says 172.21.15.88:43690) 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e14fc8640 1 -- 172.21.15.88:0/2982685640 learned_addr learned my addr 172.21.15.88:0/2982685640 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e0ffff640 1 --2- 172.21.15.88:0/2982685640 >> v2:172.21.15.46:3300/0 conn(0x7f4e1006f920 0x7f4e100ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e0effd640 1 -- 172.21.15.88:0/2982685640 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653447827 0 0) 0x7f4e100d3730 con 0x7f4e100cfdf0 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e0effd640 1 -- 172.21.15.88:0/2982685640 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dfc003520 con 0x7f4e100cfdf0 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e0ffff640 1 -- 172.21.15.88:0/2982685640 >> v1:172.21.15.88:6789/0 conn(0x7f4e100cfdf0 legacy=0x7f4e100d3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e0ffff640 1 -- 172.21.15.88:0/2982685640 >> v1:172.21.15.46:6789/0 conn(0x7f4e100b3d20 legacy=0x7f4e100d2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.704+0000 7f4e0ffff640 1 -- 172.21.15.88:0/2982685640 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e100b31d0 con 0x7f4e1006f920 2024-05-09T03:09:11.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.705+0000 7f4e0ffff640 1 --2- 172.21.15.88:0/2982685640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1006f920 0x7f4e100ce6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e000097a0 tx=0x7f4e0002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c79843669df8e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.705+0000 7f4e0effd640 1 -- 172.21.15.88:0/2982685640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00037070 con 0x7f4e1006f920 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.705+0000 7f4e0effd640 1 -- 172.21.15.88:0/2982685640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e00032440 con 0x7f4e1006f920 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.705+0000 7f4e0effd640 1 -- 172.21.15.88:0/2982685640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00032810 con 0x7f4e1006f920 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 -- 172.21.15.88:0/2982685640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1006f920 msgr2=0x7f4e100ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 --2- 172.21.15.88:0/2982685640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1006f920 0x7f4e100ce6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e000097a0 tx=0x7f4e0002a230 comp rx=0 tx=0).stop 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 -- 172.21.15.88:0/2982685640 shutdown_connections 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 --2- 172.21.15.88:0/2982685640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1006f920 0x7f4e100ce6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 -- 172.21.15.88:0/2982685640 >> 172.21.15.88:0/2982685640 conn(0x7f4e1009b7d0 msgr2=0x7f4e1009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 -- 172.21.15.88:0/2982685640 shutdown_connections 2024-05-09T03:09:11.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.706+0000 7f4e16a2b640 1 -- 172.21.15.88:0/2982685640 wait complete. 2024-05-09T03:09:11.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.708+0000 7f4e16a2b640 1 Processor -- start 2024-05-09T03:09:11.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.708+0000 7f4e16a2b640 1 -- start start 2024-05-09T03:09:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e16a2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 0x7f4e10156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e16a2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e100b3d20 0x7f4e101f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e16a2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e100cfdf0 0x7f4e1015a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e16a2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e100b29d0 con 0x7f4e100cfdf0 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e16a2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4e100b26f0 con 0x7f4e1006f920 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e16a2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4e100cf7a0 con 0x7f4e100b3d20 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.709+0000 7f4e0ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 0x7f4e10156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 0x7f4e10156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49456/0 (socket says 172.21.15.88:49456) 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 -- 172.21.15.88:0/4142525457 learned_addr learned my addr 172.21.15.88:0/4142525457 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0f7fe640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e100b3d20 0x7f4e101f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e14fc8640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e100cfdf0 0x7f4e1015a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 -- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e100b3d20 msgr2=0x7f4e101f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e100b3d20 0x7f4e101f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 -- 172.21.15.88:0/4142525457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e100cfdf0 msgr2=0x7f4e1015a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e100cfdf0 0x7f4e1015a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 -- 172.21.15.88:0/4142525457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e101f52f0 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0ffff640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 0x7f4e10156100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e00035040 tx=0x7f4e000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00037040 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e00030500 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e1015cd30 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e00030910 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e101f45b0 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4e00044020 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.710+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f4e0003bc50 con 0x7f4e1006f920 2024-05-09T03:09:11.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.711+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e101f43a0 con 0x7f4e1006f920 2024-05-09T03:09:11.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.715+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4e0003b430 con 0x7f4e1006f920 2024-05-09T03:09:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.787+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e2b044e3-c608-40af-b9b7-6f04bf54b508", "id": 4} v 0) -- 0x7f4e1006a270 con 0x7f4e1006f920 2024-05-09T03:09:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.791+0000 7f4e0cff9640 1 -- 172.21.15.88:0/4142525457 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2b044e3-c608-40af-b9b7-6f04bf54b508", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4e0003b610 con 0x7f4e1006f920 2024-05-09T03:09:11.792 INFO:teuthology.orchestra.run.smithi088.stdout:4 2024-05-09T03:09:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 msgr2=0x7f4e10156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 0x7f4e10156100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e00035040 tx=0x7f4e000046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 shutdown_connections 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e100cfdf0 0x7f4e1015a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4e100b3d20 0x7f4e101f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 --2- 172.21.15.88:0/4142525457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4e1006f920 0x7f4e10156100 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 >> 172.21.15.88:0/4142525457 conn(0x7f4e1009b7d0 msgr2=0x7f4e1009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 shutdown_connections 2024-05-09T03:09:11.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.792+0000 7f4e16a2b640 1 -- 172.21.15.88:0/4142525457 wait complete. 2024-05-09T03:09:11.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 5c1533a5-2622-4320-9458-107694403b7a 5 2024-05-09T03:09:11.835 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:11.834+0000 7fdad1033040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T03:09:11.843 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:11.841+0000 7fd063723040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T03:09:11.890 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:11.889+0000 7fdad1033040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T03:09:11.892 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:11.891+0000 7fd063723040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T03:09:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.944+0000 7f6a38176640 1 Processor -- start 2024-05-09T03:09:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.945+0000 7f6a38176640 1 -- start start 2024-05-09T03:09:11.946 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:11.946+0000 7fdad1033040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a38176640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 0x7f6a300b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a38176640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 0x7f6a300cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a38176640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a300af850 con 0x7f6a300cf520 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a38176640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6a300cf930 con 0x7f6a300b1450 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a38176640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6a300b8400 con 0x7f6a300cebd0 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 0x7f6a300cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35f12640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 0x7f6a300b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 0x7f6a300cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40062/0 (socket says 172.21.15.88:40062) 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35f12640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 0x7f6a300b77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49466/0 (socket says 172.21.15.88:49466) 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 -- 172.21.15.88:0/3031318232 learned_addr learned my addr 172.21.15.88:0/3031318232 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 -- 172.21.15.88:0/3031318232 >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 msgr2=0x7f6a300b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 --2- 172.21.15.88:0/3031318232 >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 0x7f6a300b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 -- 172.21.15.88:0/3031318232 >> v1:172.21.15.46:6789/0 conn(0x7f6a300cf520 legacy=0x7f6a300cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35711640 1 -- 172.21.15.88:0/3031318232 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a301608f0 con 0x7f6a300cebd0 2024-05-09T03:09:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.946+0000 7f6a35f12640 1 --2- 172.21.15.88:0/3031318232 >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 0x7f6a300b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:11.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.947+0000 7f6a35711640 1 --2- 172.21.15.88:0/3031318232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 0x7f6a300cefe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009870 tx=0x7f6a2c02a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc648bffe8b7466f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.949 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:11.946+0000 7fd063723040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T03:09:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.947+0000 7f6a34f10640 1 -- 172.21.15.88:0/3031318232 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c037070 con 0x7f6a300cebd0 2024-05-09T03:09:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.947+0000 7f6a34f10640 1 -- 172.21.15.88:0/3031318232 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a2c02ae50 con 0x7f6a300cebd0 2024-05-09T03:09:11.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.947+0000 7f6a34f10640 1 -- 172.21.15.88:0/3031318232 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c032860 con 0x7f6a300cebd0 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 -- 172.21.15.88:0/3031318232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 msgr2=0x7f6a300cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 --2- 172.21.15.88:0/3031318232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 0x7f6a300cefe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009870 tx=0x7f6a2c02a300 comp rx=0 tx=0).stop 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 -- 172.21.15.88:0/3031318232 shutdown_connections 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 --2- 172.21.15.88:0/3031318232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cebd0 0x7f6a300cefe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 --2- 172.21.15.88:0/3031318232 >> v2:172.21.15.88:3300/0 conn(0x7f6a300b1450 0x7f6a300b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 -- 172.21.15.88:0/3031318232 >> 172.21.15.88:0/3031318232 conn(0x7f6a3009b7d0 msgr2=0x7f6a3009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 -- 172.21.15.88:0/3031318232 shutdown_connections 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.948+0000 7f6a38176640 1 -- 172.21.15.88:0/3031318232 wait complete. 2024-05-09T03:09:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.950+0000 7f6a38176640 1 Processor -- start 2024-05-09T03:09:11.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.950+0000 7f6a38176640 1 -- start start 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a38176640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a300b1450 0x7f6a300b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a38176640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a300cebd0 0x7f6a301e9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a38176640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cf520 0x7f6a301ea2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a38176640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6a300d1f90 con 0x7f6a300b1450 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a38176640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6a300d1cb0 con 0x7f6a300cebd0 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a38176640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6a300d2270 con 0x7f6a300cf520 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.951+0000 7f6a35711640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a300cebd0 0x7f6a301e9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cf520 0x7f6a301ea2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a35711640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a300cebd0 0x7f6a301e9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49474/0 (socket says 172.21.15.88:49474) 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a35711640 1 -- 172.21.15.88:0/213078741 learned_addr learned my addr 172.21.15.88:0/213078741 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a35f12640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a300b1450 0x7f6a300b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 -- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a300cebd0 msgr2=0x7f6a301e9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a300cebd0 0x7f6a301e9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 -- 172.21.15.88:0/213078741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a300b1450 msgr2=0x7f6a300b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a300b1450 0x7f6a300b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 -- 172.21.15.88:0/213078741 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a300c7a00 con 0x7f6a300cf520 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a36713640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cf520 0x7f6a301ea2c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a30074170 tx=0x7f6a2800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28003b90 con 0x7f6a300cf520 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a28003d50 con 0x7f6a300cf520 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28005310 con 0x7f6a300cf520 2024-05-09T03:09:11.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a300b25c0 con 0x7f6a300cf520 2024-05-09T03:09:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a300b2b80 con 0x7f6a300cf520 2024-05-09T03:09:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a280054b0 con 0x7f6a300cf520 2024-05-09T03:09:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.952+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f6a280129a0 con 0x7f6a300cf520 2024-05-09T03:09:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.953+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3006a270 con 0x7f6a300cf520 2024-05-09T03:09:11.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:11.955+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6a28012c40 con 0x7f6a300cf520 2024-05-09T03:09:12.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.027+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5c1533a5-2622-4320-9458-107694403b7a", "id": 5} v 0) -- 0x7f6a300d2730 con 0x7f6a300cf520 2024-05-09T03:09:12.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.031+0000 7f6a267fc640 1 -- 172.21.15.88:0/213078741 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5c1533a5-2622-4320-9458-107694403b7a", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f6a28012e20 con 0x7f6a300cf520 2024-05-09T03:09:12.032 INFO:teuthology.orchestra.run.smithi088.stdout:5 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cf520 msgr2=0x7f6a301ea2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cf520 0x7f6a301ea2c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a30074170 tx=0x7f6a2800cb90 comp rx=0 tx=0).stop 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 shutdown_connections 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a300cf520 0x7f6a301ea2c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a300cebd0 0x7f6a301e9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 --2- 172.21.15.88:0/213078741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a300b1450 0x7f6a300b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 >> 172.21.15.88:0/213078741 conn(0x7f6a3009b7d0 msgr2=0x7f6a301e8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 shutdown_connections 2024-05-09T03:09:12.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.032+0000 7f6a38176640 1 -- 172.21.15.88:0/213078741 wait complete. 2024-05-09T03:09:12.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 7ac7f6ad-996f-448d-b0f6-2c055736ef8e 6 2024-05-09T03:09:12.065 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:12.064+0000 7fd063723040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:09:12.066 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:12.065+0000 7fdad1033040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T03:09:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.154+0000 7f93be535640 1 Processor -- start 2024-05-09T03:09:12.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.154+0000 7f93be535640 1 -- start start 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93be535640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b80cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93be535640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f93b80b77a0 0x7f93b80b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93be535640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93b806f910 con 0x7f93b806e020 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93be535640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f93b80b80f0 con 0x7f93b80b77a0 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93be535640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93b80d3880 con 0x7f93b80d33b0 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b80cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b80cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40084/0 (socket says 172.21.15.88:40084) 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93bcad2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f93b806e020 0x7f93b806e430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43700/0 (socket says 172.21.15.88:43700) 2024-05-09T03:09:12.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 -- 172.21.15.88:0/2789679172 learned_addr learned my addr 172.21.15.88:0/2789679172 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b77fe640 1 --2- 172.21.15.88:0/2789679172 >> v2:172.21.15.46:3300/0 conn(0x7f93b80b77a0 0x7f93b80b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b6ffd640 1 -- 172.21.15.88:0/2789679172 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884548433 0 0) 0x7f93b806f910 con 0x7f93b806e020 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b6ffd640 1 -- 172.21.15.88:0/2789679172 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93a4003520 con 0x7f93b806e020 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 -- 172.21.15.88:0/2789679172 >> v2:172.21.15.46:3300/0 conn(0x7f93b80b77a0 msgr2=0x7f93b80b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 --2- 172.21.15.88:0/2789679172 >> v2:172.21.15.46:3300/0 conn(0x7f93b80b77a0 0x7f93b80b7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 -- 172.21.15.88:0/2789679172 >> v1:172.21.15.88:6789/0 conn(0x7f93b806e020 legacy=0x7f93b806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.156+0000 7f93b7fff640 1 -- 172.21.15.88:0/2789679172 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b806f370 con 0x7f93b80d33b0 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.157+0000 7f93b7fff640 1 --2- 172.21.15.88:0/2789679172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b80cf950 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009870 tx=0x7f93a802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d31bd2a542d8fd60 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.157+0000 7f93b6ffd640 1 -- 172.21.15.88:0/2789679172 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8037070 con 0x7f93b80d33b0 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.157+0000 7f93b6ffd640 1 -- 172.21.15.88:0/2789679172 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a8032440 con 0x7f93b80d33b0 2024-05-09T03:09:12.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.157+0000 7f93b6ffd640 1 -- 172.21.15.88:0/2789679172 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8032810 con 0x7f93b80d33b0 2024-05-09T03:09:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 -- 172.21.15.88:0/2789679172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 msgr2=0x7f93b80cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 --2- 172.21.15.88:0/2789679172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b80cf950 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009870 tx=0x7f93a802a270 comp rx=0 tx=0).stop 2024-05-09T03:09:12.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 -- 172.21.15.88:0/2789679172 shutdown_connections 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 --2- 172.21.15.88:0/2789679172 >> v2:172.21.15.46:3300/0 conn(0x7f93b80b77a0 0x7f93b80b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 --2- 172.21.15.88:0/2789679172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b80cf950 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 -- 172.21.15.88:0/2789679172 >> 172.21.15.88:0/2789679172 conn(0x7f93b809b7d0 msgr2=0x7f93b809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 -- 172.21.15.88:0/2789679172 shutdown_connections 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.158+0000 7f93be535640 1 -- 172.21.15.88:0/2789679172 wait complete. 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.160+0000 7f93be535640 1 Processor -- start 2024-05-09T03:09:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.160+0000 7f93be535640 1 -- start start 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93be535640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93b806e020 0x7f93b815f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93be535640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 0x7f93b8151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93be535640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b8151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93be535640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80b4230 con 0x7f93b806e020 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93be535640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80d0720 con 0x7f93b80b77a0 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93be535640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93b80b43b0 con 0x7f93b80d33b0 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93bcad2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b8151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93b77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 0x7f93b8151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93bcad2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b8151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.88:40094/0 (socket says 172.21.15.88:40094) 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93b77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 0x7f93b8151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49488/0 (socket says 172.21.15.88:49488) 2024-05-09T03:09:12.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93b7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93b806e020 0x7f93b815f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.161+0000 7f93bcad2640 1 -- 172.21.15.88:0/3248218602 learned_addr learned my addr 172.21.15.88:0/3248218602 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93bcad2640 1 -- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 msgr2=0x7f93b8151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93bcad2640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 0x7f93b8151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93bcad2640 1 -- 172.21.15.88:0/3248218602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93b806e020 msgr2=0x7f93b815f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93bcad2640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93b806e020 0x7f93b815f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93bcad2640 1 -- 172.21.15.88:0/3248218602 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b80ceea0 con 0x7f93b80d33b0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b77fe640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 0x7f93b8151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b7fff640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93b806e020 0x7f93b815f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93bcad2640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b8151d90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f93b000c760 tx=0x7f93b000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0017070 con 0x7f93b80d33b0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b000a640 con 0x7f93b80d33b0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b000aa30 con 0x7f93b80d33b0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b815d7e0 con 0x7f93b80d33b0 2024-05-09T03:09:12.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b8152900 con 0x7f93b80d33b0 2024-05-09T03:09:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f93b001a070 con 0x7f93b80d33b0 2024-05-09T03:09:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.162+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f93b0007750 con 0x7f93b80d33b0 2024-05-09T03:09:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.163+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b806a270 con 0x7f93b80d33b0 2024-05-09T03:09:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.165+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f93b0021020 con 0x7f93b80d33b0 2024-05-09T03:09:12.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.237+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7ac7f6ad-996f-448d-b0f6-2c055736ef8e", "id": 6} v 0) -- 0x7f93b8151160 con 0x7f93b80d33b0 2024-05-09T03:09:12.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.241+0000 7f93b4ff9640 1 -- 172.21.15.88:0/3248218602 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7ac7f6ad-996f-448d-b0f6-2c055736ef8e", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f93b000abd0 con 0x7f93b80d33b0 2024-05-09T03:09:12.242 INFO:teuthology.orchestra.run.smithi088.stdout:6 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 msgr2=0x7f93b8151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b8151d90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f93b000c760 tx=0x7f93b000cc20 comp rx=0 tx=0).stop 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 shutdown_connections 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93b80d33b0 0x7f93b8151d90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b80b77a0 0x7f93b8151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 --2- 172.21.15.88:0/3248218602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93b806e020 0x7f93b815f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 >> 172.21.15.88:0/3248218602 conn(0x7f93b809b7d0 msgr2=0x7f93b815c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 shutdown_connections 2024-05-09T03:09:12.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.242+0000 7f93be535640 1 -- 172.21.15.88:0/3248218602 wait complete. 2024-05-09T03:09:12.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd new 6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea 7 2024-05-09T03:09:12.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.404+0000 7f0f79bce640 1 Processor -- start 2024-05-09T03:09:12.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.404+0000 7f0f79bce640 1 -- start start 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f79bce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f7406e020 0x7f0f740b7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f79bce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f740b7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f79bce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f7415afa0 con 0x7f0f7415ab90 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f79bce640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0f740cfec0 con 0x7f0f7406e020 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f79bce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0f74160c20 con 0x7f0f7406e430 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f737fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0f7415ab90 0x7f0f740cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43716/0 (socket says 172.21.15.88:43716) 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f73fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f740b7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f737fe640 1 -- 172.21.15.88:0/1620043746 learned_addr learned my addr 172.21.15.88:0/1620043746 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f6bfff640 1 --2- 172.21.15.88:0/1620043746 >> v2:172.21.15.46:3300/0 conn(0x7f0f7406e020 0x7f0f740b7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f72ffd640 1 -- 172.21.15.88:0/1620043746 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136764700 0 0) 0x7f0f7415afa0 con 0x7f0f7415ab90 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f72ffd640 1 -- 172.21.15.88:0/1620043746 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f60003520 con 0x7f0f7415ab90 2024-05-09T03:09:12.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f73fff640 1 -- 172.21.15.88:0/1620043746 >> v2:172.21.15.46:3300/0 conn(0x7f0f7406e020 msgr2=0x7f0f740b7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f73fff640 1 --2- 172.21.15.88:0/1620043746 >> v2:172.21.15.46:3300/0 conn(0x7f0f7406e020 0x7f0f740b7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f73fff640 1 -- 172.21.15.88:0/1620043746 >> v1:172.21.15.88:6789/0 conn(0x7f0f7415ab90 legacy=0x7f0f740cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.406+0000 7f0f73fff640 1 -- 172.21.15.88:0/1620043746 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7415dad0 con 0x7f0f7406e430 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.407+0000 7f0f73fff640 1 --2- 172.21.15.88:0/1620043746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f740b7c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009d90 tx=0x7f0f6c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc99d1c87aec902 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.407+0000 7f0f72ffd640 1 -- 172.21.15.88:0/1620043746 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c007740 con 0x7f0f7406e430 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.407+0000 7f0f72ffd640 1 -- 172.21.15.88:0/1620043746 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f6c0078e0 con 0x7f0f7406e430 2024-05-09T03:09:12.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.407+0000 7f0f72ffd640 1 -- 172.21.15.88:0/1620043746 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c007cb0 con 0x7f0f7406e430 2024-05-09T03:09:12.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 -- 172.21.15.88:0/1620043746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 msgr2=0x7f0f740b7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 --2- 172.21.15.88:0/1620043746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f740b7c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009d90 tx=0x7f0f6c02a660 comp rx=0 tx=0).stop 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 -- 172.21.15.88:0/1620043746 shutdown_connections 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 --2- 172.21.15.88:0/1620043746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f740b7c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 --2- 172.21.15.88:0/1620043746 >> v2:172.21.15.46:3300/0 conn(0x7f0f7406e020 0x7f0f740b7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 -- 172.21.15.88:0/1620043746 >> 172.21.15.88:0/1620043746 conn(0x7f0f7409b7d0 msgr2=0x7f0f7409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 -- 172.21.15.88:0/1620043746 shutdown_connections 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.408+0000 7f0f79bce640 1 -- 172.21.15.88:0/1620043746 wait complete. 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.410+0000 7f0f79bce640 1 Processor -- start 2024-05-09T03:09:12.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.410+0000 7f0f79bce640 1 -- start start 2024-05-09T03:09:12.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f79bce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7406e020 0x7f0f74157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f79bce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f7409fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f79bce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 0x7f0f7406eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f79bce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7415e370 con 0x7f0f7406e020 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f79bce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7415e090 con 0x7f0f7415ab90 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f79bce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0f7415d630 con 0x7f0f7406e430 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f73fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 0x7f0f7406eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f6bfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f7409fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f73fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 0x7f0f7406eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49504/0 (socket says 172.21.15.88:49504) 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f73fff640 1 -- 172.21.15.88:0/2384405534 learned_addr learned my addr 172.21.15.88:0/2384405534 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-05-09T03:09:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f737fe640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7406e020 0x7f0f74157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.411+0000 7f0f73fff640 1 -- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 msgr2=0x7f0f7409fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f73fff640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f7409fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f73fff640 1 -- 172.21.15.88:0/2384405534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7406e020 msgr2=0x7f0f74157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f73fff640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7406e020 0x7f0f74157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f73fff640 1 -- 172.21.15.88:0/2384405534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f74158500 con 0x7f0f7415ab90 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f6bfff640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f7409fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f73fff640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 0x7f0f7406eff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009eb0 tx=0x7f0f6c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c03f070 con 0x7f0f7415ab90 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f6c031bc0 con 0x7f0f7415ab90 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f740c65c0 con 0x7f0f7415ab90 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c009450 con 0x7f0f7415ab90 2024-05-09T03:09:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f740c6b80 con 0x7f0f7415ab90 2024-05-09T03:09:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0f6c044460 con 0x7f0f7415ab90 2024-05-09T03:09:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.412+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f0f6c03ba70 con 0x7f0f7415ab90 2024-05-09T03:09:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.413+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7406a270 con 0x7f0f7415ab90 2024-05-09T03:09:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.415+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f0f74158500 con 0x7f0f7415ab90 2024-05-09T03:09:12.467 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:12.466+0000 7fd063723040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T03:09:12.468 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:12.467+0000 7fdad1033040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T03:09:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.487+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea", "id": 7} v 0) -- 0x7f0f7415cc50 con 0x7f0f7415ab90 2024-05-09T03:09:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.491+0000 7f0f70ff9640 1 -- 172.21.15.88:0/2384405534 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f0f6c03bd50 con 0x7f0f7415ab90 2024-05-09T03:09:12.492 INFO:teuthology.orchestra.run.smithi088.stdout:7 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 msgr2=0x7f0f7406eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 0x7f0f7406eff0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c009eb0 tx=0x7f0f6c004a40 comp rx=0 tx=0).stop 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 shutdown_connections 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f7415ab90 0x7f0f7406eff0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f7406e430 0x7f0f7409fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 --2- 172.21.15.88:0/2384405534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f7406e020 0x7f0f74157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 >> 172.21.15.88:0/2384405534 conn(0x7f0f7409b7d0 msgr2=0x7f0f740708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 shutdown_connections 2024-05-09T03:09:12.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:12.492+0000 7f0f79bce640 1 -- 172.21.15.88:0/2384405534 wait complete. 2024-05-09T03:09:12.504 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-09T03:09:12.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-09T03:09:12.506 INFO:tasks.ceph.osd.0:Started 2024-05-09T03:09:12.506 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-09T03:09:12.506 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-09T03:09:12.508 INFO:tasks.ceph.osd.1:Started 2024-05-09T03:09:12.508 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-09T03:09:12.509 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-09T03:09:12.511 INFO:tasks.ceph.osd.2:Started 2024-05-09T03:09:12.511 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-09T03:09:12.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-09T03:09:12.513 INFO:tasks.ceph.osd.3:Started 2024-05-09T03:09:12.513 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-09T03:09:12.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-09T03:09:12.519 INFO:tasks.ceph.osd.4:Started 2024-05-09T03:09:12.519 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-09T03:09:12.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-09T03:09:12.521 INFO:tasks.ceph.osd.5:Started 2024-05-09T03:09:12.521 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-09T03:09:12.521 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-09T03:09:12.523 INFO:tasks.ceph.osd.6:Started 2024-05-09T03:09:12.524 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-09T03:09:12.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-09T03:09:12.526 INFO:tasks.ceph.osd.7:Started 2024-05-09T03:09:12.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:12.642 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:12.641+0000 7fdad1033040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T03:09:12.651 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:12.650+0000 7fd063723040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T03:09:12.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.673+0000 7f9ba7100640 1 Processor -- start 2024-05-09T03:09:12.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.673+0000 7f9ba7100640 1 -- start start 2024-05-09T03:09:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba7100640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9ba0164c70 0x7f9ba01615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba7100640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba00d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba7100640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ba0160b20 con 0x7f9ba015dfe0 2024-05-09T03:09:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba7100640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9ba0161150 con 0x7f9ba0164c70 2024-05-09T03:09:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba7100640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba00d15f0 con 0x7f9ba00d2480 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba4e9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9ba0164c70 0x7f9ba01615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba4e9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9ba0164c70 0x7f9ba01615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57988/0 (socket says 172.21.15.46:57988) 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/4046846768 learned_addr learned my addr 172.21.15.46:0/4046846768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba569d640 1 --2- 172.21.15.46:0/4046846768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba00d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.674+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/4046846768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 msgr2=0x7f9ba00d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9ba4e9c640 1 --2- 172.21.15.46:0/4046846768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba00d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/4046846768 >> v1:172.21.15.46:6789/0 conn(0x7f9ba015dfe0 legacy=0x7f9ba00d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/4046846768 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba00d18d0 con 0x7f9ba0164c70 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9ba4e9c640 1 --2- 172.21.15.46:0/4046846768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba01615e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009870 tx=0x7f9b9402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cc000c6501226c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9b9f7fe640 1 -- 172.21.15.46:0/4046846768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94037070 con 0x7f9ba0164c70 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9b9f7fe640 1 -- 172.21.15.46:0/4046846768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b94032440 con 0x7f9ba0164c70 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.675+0000 7f9b9f7fe640 1 -- 172.21.15.46:0/4046846768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94032810 con 0x7f9ba0164c70 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 -- 172.21.15.46:0/4046846768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 msgr2=0x7f9ba01615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 --2- 172.21.15.46:0/4046846768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba01615e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009870 tx=0x7f9b9402a250 comp rx=0 tx=0).stop 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 -- 172.21.15.46:0/4046846768 shutdown_connections 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 --2- 172.21.15.46:0/4046846768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba00d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 --2- 172.21.15.46:0/4046846768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba01615e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 -- 172.21.15.46:0/4046846768 >> 172.21.15.46:0/4046846768 conn(0x7f9ba0073db0 msgr2=0x7f9ba00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 -- 172.21.15.46:0/4046846768 shutdown_connections 2024-05-09T03:09:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.676+0000 7f9ba7100640 1 -- 172.21.15.46:0/4046846768 wait complete. 2024-05-09T03:09:12.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.678+0000 7f9ba7100640 1 Processor -- start 2024-05-09T03:09:12.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.678+0000 7f9ba7100640 1 -- start start 2024-05-09T03:09:12.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba7100640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba0158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba7100640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ba015dfe0 0x7f9ba0155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba7100640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:12.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba7100640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba00d2dd0 con 0x7f9ba0164c70 2024-05-09T03:09:12.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba7100640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba015e3f0 con 0x7f9ba015dfe0 2024-05-09T03:09:12.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba7100640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba00d45c0 con 0x7f9ba00d2480 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba4e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba0158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba4e9c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba0158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:46778/0 (socket says 172.21.15.46:46778) 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/3644087206 learned_addr learned my addr 172.21.15.46:0/3644087206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9ba569d640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.680+0000 7f9b9ffff640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ba015dfe0 0x7f9ba0155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ba015dfe0 msgr2=0x7f9ba0155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba4e9c640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ba015dfe0 0x7f9ba0155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/3644087206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 msgr2=0x7f9ba00cba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba4e9c640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba00cba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba4e9c640 1 -- 172.21.15.46:0/3644087206 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba0157e80 con 0x7f9ba00d2480 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba4e9c640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba0158aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9b94000c00 tx=0x7f9b940046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94041020 con 0x7f9ba00d2480 2024-05-09T03:09:12.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b940306c0 con 0x7f9ba00d2480 2024-05-09T03:09:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba01f6e40 con 0x7f9ba00d2480 2024-05-09T03:09:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94037040 con 0x7f9ba00d2480 2024-05-09T03:09:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01f7400 con 0x7f9ba00d2480 2024-05-09T03:09:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9b94044020 con 0x7f9ba00d2480 2024-05-09T03:09:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.681+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f9b9403dc90 con 0x7f9ba00d2480 2024-05-09T03:09:12.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.682+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba008cfd0 con 0x7f9ba00d2480 2024-05-09T03:09:12.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.684+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f9ba008cfd0 con 0x7f9ba00d2480 2024-05-09T03:09:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.746+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ba006a270 con 0x7f9ba00d2480 2024-05-09T03:09:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.747+0000 7f9b9d7fa640 1 -- 172.21.15.46:0/3644087206 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f9b9404b030 con 0x7f9ba00d2480 2024-05-09T03:09:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:12.749 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:12.489163+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 msgr2=0x7f9ba0158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba0158aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9b94000c00 tx=0x7f9b940046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 shutdown_connections 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ba0164c70 0x7f9ba00cba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ba015dfe0 0x7f9ba0155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 --2- 172.21.15.46:0/3644087206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ba00d2480 0x7f9ba0158aa0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 >> 172.21.15.46:0/3644087206 conn(0x7f9ba0073db0 msgr2=0x7f9ba015d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 shutdown_connections 2024-05-09T03:09:12.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:12.748+0000 7f9ba7100640 1 -- 172.21.15.46:0/3644087206 wait complete. 2024-05-09T03:09:12.761 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-09T03:09:12.761 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-09T03:09:12.902 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:12.901+0000 7fdad1033040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T03:09:12.928 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:12.927+0000 7fd063723040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T03:09:12.954 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:12.953+0000 7fdad1033040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T03:09:12.983 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:12.982+0000 7fd063723040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T03:09:13.071 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.070+0000 7fdad1033040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T03:09:13.105 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.103+0000 7fd063723040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T03:09:13.132 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-09T03:09:13.131+0000 7f65ef140640 -1 Falling back to public interface 2024-05-09T03:09:13.133 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-09T03:09:13.132+0000 7f6f86f01640 -1 Falling back to public interface 2024-05-09T03:09:13.140 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-05-09T03:09:13.138+0000 7f3d7e224640 -1 Falling back to public interface 2024-05-09T03:09:13.148 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-05-09T03:09:13.146+0000 7f59c3c13640 -1 Falling back to public interface 2024-05-09T03:09:13.152 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-05-09T03:09:13.150+0000 7fe87ce0d640 -1 Falling back to public interface 2024-05-09T03:09:13.156 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-09T03:09:13.155+0000 7f74b6ef0640 -1 Falling back to public interface 2024-05-09T03:09:13.156 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-05-09T03:09:13.155+0000 7f2360830640 -1 Falling back to public interface 2024-05-09T03:09:13.170 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-09T03:09:13.169+0000 7fb2dec16640 -1 Falling back to public interface 2024-05-09T03:09:13.372 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.371+0000 7fdad1033040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T03:09:13.413 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.411+0000 7fd063723040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T03:09:13.421 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.420+0000 7fdad1033040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T03:09:13.464 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.462+0000 7fd063723040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T03:09:13.567 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:13.580 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.579+0000 7fdad1033040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T03:09:13.629 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.627+0000 7fd063723040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T03:09:13.672 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.671+0000 7f12039f3640 1 Processor -- start 2024-05-09T03:09:13.673 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.671+0000 7f12039f3640 1 -- start start 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f12039f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b43f0 0x7f11fc0cd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f12039f3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11fc06f990 con 0x7f11fc06f580 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f12039f3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11fc0d26b0 con 0x7f11fc0b4800 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f12039f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc0d2990 con 0x7f11fc0b43f0 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f120178f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11fc06f580 0x7f11fc0b3ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55124/0 (socket says 172.21.15.46:55124) 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f120178f640 1 -- 172.21.15.46:0/318134234 learned_addr learned my addr 172.21.15.46:0/318134234 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.672+0000 7f1200f8e640 1 --2- 172.21.15.46:0/318134234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b43f0 0x7f11fc0cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744283666 0 0) 0x7f11fc06f990 con 0x7f11fc06f580 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11e8003520 con 0x7f11fc06f580 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431801072 0 0) 0x7f11fc0d26b0 con 0x7f11fc0b4800 2024-05-09T03:09:13.675 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11fc06f990 con 0x7f11fc0b4800 2024-05-09T03:09:13.676 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f1200f8e640 1 -- 172.21.15.46:0/318134234 >> v1:172.21.15.88:6789/0 conn(0x7f11fc0b4800 legacy=0x7f11fc0cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:13.676 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f1200f8e640 1 -- 172.21.15.46:0/318134234 >> v1:172.21.15.46:6789/0 conn(0x7f11fc06f580 legacy=0x7f11fc0b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:13.676 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.673+0000 7f1200f8e640 1 -- 172.21.15.46:0/318134234 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc0d2c70 con 0x7f11fc0b43f0 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.674+0000 7f1200f8e640 1 --2- 172.21.15.46:0/318134234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b43f0 0x7f11fc0cd120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11f00097d0 tx=0x7f11f002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=555c1046c90e9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.674+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0037070 con 0x7f11fc0b43f0 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.674+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f0032440 con 0x7f11fc0b43f0 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.674+0000 7f11effff640 1 -- 172.21.15.46:0/318134234 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0032810 con 0x7f11fc0b43f0 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 -- 172.21.15.46:0/318134234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b43f0 msgr2=0x7f11fc0cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 --2- 172.21.15.46:0/318134234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b43f0 0x7f11fc0cd120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11f00097d0 tx=0x7f11f002a210 comp rx=0 tx=0).stop 2024-05-09T03:09:13.677 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 -- 172.21.15.46:0/318134234 shutdown_connections 2024-05-09T03:09:13.678 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 --2- 172.21.15.46:0/318134234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b43f0 0x7f11fc0cd120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.678 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 -- 172.21.15.46:0/318134234 >> 172.21.15.46:0/318134234 conn(0x7f11fc09b7d0 msgr2=0x7f11fc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:13.678 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 -- 172.21.15.46:0/318134234 shutdown_connections 2024-05-09T03:09:13.678 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.675+0000 7f12039f3640 1 -- 172.21.15.46:0/318134234 wait complete. 2024-05-09T03:09:13.678 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.677+0000 7f12039f3640 1 Processor -- start 2024-05-09T03:09:13.678 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.677+0000 7f12039f3640 1 -- start start 2024-05-09T03:09:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f12039f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11fc06f580 0x7f11fc0ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f12039f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 0x7f11fc1f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f12039f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b4800 0x7f11fc0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f12039f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc0cf1b0 con 0x7f11fc0b43f0 2024-05-09T03:09:13.681 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f12039f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11fc0d3230 con 0x7f11fc06f580 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f12039f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11fc0cf330 con 0x7f11fc0b4800 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 0x7f11fc1f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 0x7f11fc1f6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58008/0 (socket says 172.21.15.46:58008) 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 -- 172.21.15.46:0/1419383251 learned_addr learned my addr 172.21.15.46:0/1419383251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1201f90640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b4800 0x7f11fc0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f120178f640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11fc06f580 0x7f11fc0ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 -- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b4800 msgr2=0x7f11fc0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b4800 0x7f11fc0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 -- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11fc06f580 msgr2=0x7f11fc0ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11fc06f580 0x7f11fc0ca990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.678+0000 7f1200f8e640 1 -- 172.21.15.46:0/1419383251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11fc15bbd0 con 0x7f11fc0b43f0 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f1200f8e640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 0x7f11fc1f6320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11f00097d0 tx=0x7f11f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0037040 con 0x7f11fc0b43f0 2024-05-09T03:09:13.682 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f0004230 con 0x7f11fc0b43f0 2024-05-09T03:09:13.683 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11fc15beb0 con 0x7f11fc0b43f0 2024-05-09T03:09:13.683 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f0006e10 con 0x7f11fc0b43f0 2024-05-09T03:09:13.683 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11fc0c7370 con 0x7f11fc0b43f0 2024-05-09T03:09:13.683 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f11f003a070 con 0x7f11fc0b43f0 2024-05-09T03:09:13.683 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.679+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f11f0003d30 con 0x7f11fc0b43f0 2024-05-09T03:09:13.683 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.680+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11fc0b1690 con 0x7f11fc0b43f0 2024-05-09T03:09:13.684 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.682+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f11fc0b1690 con 0x7f11fc0b43f0 2024-05-09T03:09:13.718 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.717+0000 7fdad1033040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T03:09:13.746 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.745+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11fc0722c0 con 0x7f11fc0b43f0 2024-05-09T03:09:13.747 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.745+0000 7f11edffb640 1 -- 172.21.15.46:0/1419383251 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f11fc0722c0 con 0x7f11fc0b43f0 2024-05-09T03:09:13.747 INFO:teuthology.misc.health.smithi046.stdout: 2024-05-09T03:09:13.747 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:12.489163+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:13.748 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 msgr2=0x7f11fc1f6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:13.748 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 0x7f11fc1f6320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11f00097d0 tx=0x7f11f00046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.748 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 shutdown_connections 2024-05-09T03:09:13.748 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11fc0b4800 0x7f11fc0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.748 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11fc0b43f0 0x7f11fc1f6320 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.749 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 --2- 172.21.15.46:0/1419383251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11fc06f580 0x7f11fc0ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:13.749 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 >> 172.21.15.46:0/1419383251 conn(0x7f11fc09b7d0 msgr2=0x7f11fc160480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:13.749 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 shutdown_connections 2024-05-09T03:09:13.749 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:13.746+0000 7f12039f3640 1 -- 172.21.15.46:0/1419383251 wait complete. 2024-05-09T03:09:13.758 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-09T03:09:13.768 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.768+0000 7fdad1033040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T03:09:13.778 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.776+0000 7fd063723040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T03:09:13.830 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.829+0000 7fd063723040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T03:09:13.838 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.837+0000 7fdad1033040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T03:09:13.882 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:13.882+0000 7fdad1033040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T03:09:13.904 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.902+0000 7fd063723040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T03:09:13.952 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:13.950+0000 7fd063723040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T03:09:14.364 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:14.363+0000 7fdad1033040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T03:09:14.454 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:14.452+0000 7fd063723040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T03:09:14.478 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:14.477+0000 7fdad1033040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T03:09:14.578 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-05-09T03:09:14.576+0000 7fd063723040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T03:09:15.495 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:15.494+0000 7fda701ee640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-09T03:09:15.495 INFO:tasks.ceph.mgr.x.smithi088.stderr:2024-05-09T03:09:15.494+0000 7fda701ee640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-09T03:09:16.617 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-09T03:09:16.616+0000 7f65ef140640 -1 osd.5 0 log_to_monitors true 2024-05-09T03:09:16.622 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-05-09T03:09:16.620+0000 7f3d7e224640 -1 osd.0 0 log_to_monitors true 2024-05-09T03:09:16.640 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-05-09T03:09:16.638+0000 7f59c3c13640 -1 osd.2 0 log_to_monitors true 2024-05-09T03:09:16.654 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-09T03:09:16.654+0000 7f6f86f01640 -1 osd.6 0 log_to_monitors true 2024-05-09T03:09:16.675 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-05-09T03:09:16.674+0000 7fe87ce0d640 -1 osd.3 0 log_to_monitors true 2024-05-09T03:09:16.698 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-09T03:09:16.697+0000 7fb2dec16640 -1 osd.4 0 log_to_monitors true 2024-05-09T03:09:16.704 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-09T03:09:16.703+0000 7f74b6ef0640 -1 osd.7 0 log_to_monitors true 2024-05-09T03:09:16.707 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-05-09T03:09:16.705+0000 7f2360830640 -1 osd.1 0 log_to_monitors true 2024-05-09T03:09:20.006 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-05-09T03:09:20.004+0000 7f59bf504640 -1 osd.2 0 waiting for initial osdmap 2024-05-09T03:09:20.020 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-05-09T03:09:20.019+0000 7f59b72ff640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.050 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-05-09T03:09:20.048+0000 7f3d7a32e640 -1 osd.0 0 waiting for initial osdmap 2024-05-09T03:09:20.057 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-05-09T03:09:20.055+0000 7f235c7c6640 -1 osd.1 0 waiting for initial osdmap 2024-05-09T03:09:20.061 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-05-09T03:09:20.060+0000 7f3d71915640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.063 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-05-09T03:09:20.062+0000 7fe878851640 -1 osd.3 0 waiting for initial osdmap 2024-05-09T03:09:20.068 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-05-09T03:09:20.067+0000 7f23545c1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.074 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-05-09T03:09:20.073+0000 7fe87064c640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.565 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:20.586 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-09T03:09:20.585+0000 7f65eb0d6640 -1 osd.5 0 waiting for initial osdmap 2024-05-09T03:09:20.602 INFO:tasks.ceph.osd.5.smithi088.stderr:2024-05-09T03:09:20.602+0000 7f65e2ed1640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.612 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-09T03:09:20.612+0000 7f74b2e86640 -1 osd.7 0 waiting for initial osdmap 2024-05-09T03:09:20.625 INFO:tasks.ceph.osd.7.smithi088.stderr:2024-05-09T03:09:20.625+0000 7f74aac81640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.634 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-09T03:09:20.633+0000 7fb2da5e7640 -1 osd.4 0 waiting for initial osdmap 2024-05-09T03:09:20.647 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-09T03:09:20.646+0000 7f6f82e97640 -1 osd.6 0 waiting for initial osdmap 2024-05-09T03:09:20.655 INFO:tasks.ceph.osd.4.smithi088.stderr:2024-05-09T03:09:20.654+0000 7fb2d23e2640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.658 INFO:tasks.ceph.osd.6.smithi088.stderr:2024-05-09T03:09:20.658+0000 7f6f7ac92640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T03:09:20.679 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.677+0000 7fef7bd0d640 1 Processor -- start 2024-05-09T03:09:20.679 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.677+0000 7fef7bd0d640 1 -- start start 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7bd0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef7406cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7bd0d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef740d2a40 0x7fef740d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7bd0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef740d3390 con 0x7fef740d36c0 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7bd0d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fef7406d860 con 0x7fef740d2a40 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7bd0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fef7406db40 con 0x7fef741588c0 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7a2aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef740d2a40 0x7fef740d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7a2aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef740d2a40 0x7fef740d2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33562/0 (socket says 172.21.15.46:33562) 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef7a2aa640 1 -- 172.21.15.46:0/894109324 learned_addr learned my addr 172.21.15.46:0/894109324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.678+0000 7fef792a8640 1 --2- 172.21.15.46:0/894109324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef7406cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.679+0000 7fef7a2aa640 1 -- 172.21.15.46:0/894109324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 msgr2=0x7fef7406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.679+0000 7fef7a2aa640 1 --2- 172.21.15.46:0/894109324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef7406cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.685 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.679+0000 7fef7a2aa640 1 -- 172.21.15.46:0/894109324 >> v1:172.21.15.88:6789/0 conn(0x7fef740d36c0 legacy=0x7fef7406e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.679+0000 7fef78aa7640 1 -- 172.21.15.46:0/894109324 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855446590 0 0) 0x7fef70001490 con 0x7fef740d36c0 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.679+0000 7fef7a2aa640 1 -- 172.21.15.46:0/894109324 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7406de20 con 0x7fef740d2a40 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.679+0000 7fef7a2aa640 1 --2- 172.21.15.46:0/894109324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef740d2e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009870 tx=0x7fef6c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcd728632ff0530 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef78aa7640 1 -- 172.21.15.46:0/894109324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c037070 con 0x7fef740d2a40 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef78aa7640 1 -- 172.21.15.46:0/894109324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef6c032440 con 0x7fef740d2a40 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef78aa7640 1 -- 172.21.15.46:0/894109324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6c032810 con 0x7fef740d2a40 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 -- 172.21.15.46:0/894109324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 msgr2=0x7fef740d2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/894109324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef740d2e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fef6c009870 tx=0x7fef6c02a270 comp rx=0 tx=0).stop 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 -- 172.21.15.46:0/894109324 shutdown_connections 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/894109324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef740d2e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/894109324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef7406cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 -- 172.21.15.46:0/894109324 >> 172.21.15.46:0/894109324 conn(0x7fef7409b7d0 msgr2=0x7fef7409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:20.686 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 -- 172.21.15.46:0/894109324 shutdown_connections 2024-05-09T03:09:20.687 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.680+0000 7fef7bd0d640 1 -- 172.21.15.46:0/894109324 wait complete. 2024-05-09T03:09:20.687 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.682+0000 7fef7bd0d640 1 Processor -- start 2024-05-09T03:09:20.687 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.682+0000 7fef7bd0d640 1 -- start start 2024-05-09T03:09:20.689 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7bd0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef7414dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7bd0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef740c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7bd0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef740ca2f0 0x7fef740c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7bd0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fef7406d450 con 0x7fef740d2a40 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7bd0d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fef7406e3e0 con 0x7fef740ca2f0 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7bd0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fef7406d5d0 con 0x7fef741588c0 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef7414dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef7414dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33576/0 (socket says 172.21.15.46:33576) 2024-05-09T03:09:20.690 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 -- 172.21.15.46:0/2325720571 learned_addr learned my addr 172.21.15.46:0/2325720571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:20.691 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef792a8640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef740c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:20.691 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef7a2aa640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef740ca2f0 0x7fef740c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:20.691 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 -- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 msgr2=0x7fef740c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:20.691 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef740c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.692 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 -- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef740ca2f0 msgr2=0x7fef740c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:20.692 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef740ca2f0 0x7fef740c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.692 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 -- 172.21.15.46:0/2325720571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef740c9450 con 0x7fef740d2a40 2024-05-09T03:09:20.692 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef79aa9640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef7414dee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c740 tx=0x7fef7000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:20.692 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.684+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70017070 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.685+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7000ce50 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.685+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef740c66d0 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.685+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70005070 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.685+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef740c6c50 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.686+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef7406a270 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.686+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7fef7001a070 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.687+0000 7fef6a7fc640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef58076240 0x7fef58078750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.687+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7fef70095d80 con 0x7fef740d2a40 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.687+0000 7fef792a8640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef58076240 0x7fef58078750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:20.693 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.688+0000 7fef792a8640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef58076240 0x7fef58078750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef740b1940 tx=0x7fef64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:20.694 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.692+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fef740d3390 con 0x7fef740d2a40 2024-05-09T03:09:20.886 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.885+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef740ce460 con 0x7fef740d2a40 2024-05-09T03:09:20.887 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.885+0000 7fef6a7fc640 1 -- 172.21.15.46:0/2325720571 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7fef70060550 con 0x7fef740d2a40 2024-05-09T03:09:20.887 INFO:teuthology.misc.health.smithi046.stdout: 2024-05-09T03:09:20.887 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":12,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:18.492842+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef58076240 msgr2=0x7fef58078750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef58076240 0x7fef58078750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fef740b1940 tx=0x7fef64008040 comp rx=0 tx=0).stop 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 msgr2=0x7fef7414dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef7414dee0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef7000c740 tx=0x7fef7000cc00 comp rx=0 tx=0).stop 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 shutdown_connections 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef740ca2f0 0x7fef740c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef741588c0 0x7fef740c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef58076240 0x7fef58078750 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 --2- 172.21.15.46:0/2325720571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef740d2a40 0x7fef7414dee0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 >> 172.21.15.46:0/2325720571 conn(0x7fef7409b7d0 msgr2=0x7fef740b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 shutdown_connections 2024-05-09T03:09:20.890 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:20.888+0000 7fef7bd0d640 1 -- 172.21.15.46:0/2325720571 wait complete. 2024-05-09T03:09:20.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-09T03:09:27.708 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:27.828 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.826+0000 7f81b985b640 1 Processor -- start 2024-05-09T03:09:27.828 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.826+0000 7f81b985b640 1 -- start start 2024-05-09T03:09:27.832 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.827+0000 7f81b985b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 0x7f81b4161a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.832 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.827+0000 7f81b985b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b40d4890 con 0x7f81b40d5350 2024-05-09T03:09:27.832 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.827+0000 7f81b985b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b4161330 con 0x7f81b4161f60 2024-05-09T03:09:27.832 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.827+0000 7f81b985b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81b40d4ec0 con 0x7f81b40d2d30 2024-05-09T03:09:27.832 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81b40d5350 0x7f81b40d2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37974/0 (socket says 172.21.15.46:37974) 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b2ffd640 1 -- 172.21.15.46:0/3291351508 learned_addr learned my addr 172.21.15.46:0/3291351508 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b27fc640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 0x7f81b4161a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089091091 0 0) 0x7f81b40d4890 con 0x7f81b40d5350 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f819c003520 con 0x7f81b40d5350 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653176855 0 0) 0x7f81b4161330 con 0x7f81b4161f60 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81b40d4890 con 0x7f81b4161f60 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2496263736 0 0) 0x7f819c003520 con 0x7f81b40d5350 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 msgr2=0x7f81b4161a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 0x7f81b4161a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 >> v1:172.21.15.88:6789/0 conn(0x7f81b4161f60 legacy=0x7f81b40b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b4161600 con 0x7f81b40d5350 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f81a40016f0 con 0x7f81b40d5350 2024-05-09T03:09:27.833 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 >> v1:172.21.15.46:6789/0 conn(0x7f81b40d5350 legacy=0x7f81b40d2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 0x7f819c003c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c004210 0x7f819c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f819c004db0 0x7f819c005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81b40d4ec0 con 0x7f819c004210 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f819c00bc70 con 0x7f819c004db0 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.828+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f819c00be70 con 0x7f81b40d2d30 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b27fc640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c004210 0x7f819c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b2ffd640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 0x7f819c003c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b37fe640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f819c004db0 0x7f819c005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b27fc640 1 -- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 msgr2=0x7f819c003c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b27fc640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b40d2d30 0x7f819c003c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b27fc640 1 -- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f819c004db0 msgr2=0x7f819c005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b27fc640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f819c004db0 0x7f819c005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:27.834 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.829+0000 7f81b27fc640 1 -- 172.21.15.46:0/3291351508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c003520 con 0x7f819c004210 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.830+0000 7f81b27fc640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c004210 0x7f819c004800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009ac0 tx=0x7f819c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bad96f4f4a65e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.830+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8036070 con 0x7f819c004210 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.830+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a8031400 con 0x7f819c004210 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.830+0000 7f81b1ffb640 1 -- 172.21.15.46:0/3291351508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8009690 con 0x7f819c004210 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.830+0000 7f81b985b640 1 -- 172.21.15.46:0/3291351508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c004210 msgr2=0x7f819c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.830+0000 7f81b985b640 1 --2- 172.21.15.46:0/3291351508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c004210 0x7f819c004800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f81a8009ac0 tx=0x7f819c005830 comp rx=0 tx=0).stop 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.831+0000 7f81b37fe640 1 -- 172.21.15.46:0/3291351508 reap_dead start 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.831+0000 7f81b985b640 1 -- 172.21.15.46:0/3291351508 shutdown_connections 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.831+0000 7f81b985b640 1 -- 172.21.15.46:0/3291351508 >> 172.21.15.46:0/3291351508 conn(0x7f81b406edc0 msgr2=0x7f81b4073db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.831+0000 7f81b985b640 1 -- 172.21.15.46:0/3291351508 shutdown_connections 2024-05-09T03:09:27.835 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.831+0000 7f81b985b640 1 -- 172.21.15.46:0/3291351508 wait complete. 2024-05-09T03:09:27.837 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.832+0000 7f81b985b640 1 Processor -- start 2024-05-09T03:09:27.837 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.833+0000 7f81b985b640 1 -- start start 2024-05-09T03:09:27.837 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b985b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f819c004210 0x7f81b415ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.837 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b985b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 0x7f81b41f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.837 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b985b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b4157f70 0x7f81b4158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b985b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81b4161a20 con 0x7f81b41e9020 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b985b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81b40d5760 con 0x7f81b4157f70 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b985b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81b4161ba0 con 0x7f819c004210 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 0x7f81b41f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b2ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f819c004210 0x7f81b415ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 0x7f81b41f5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33160/0 (socket says 172.21.15.46:33160) 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 -- 172.21.15.46:0/2942506209 learned_addr learned my addr 172.21.15.46:0/2942506209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b37fe640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b4157f70 0x7f81b4158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 -- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f819c004210 msgr2=0x7f81b415ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f819c004210 0x7f81b415ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 -- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b4157f70 msgr2=0x7f81b4158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b4157f70 0x7f81b4158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.834+0000 7f81b27fc640 1 -- 172.21.15.46:0/2942506209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b41588c0 con 0x7f81b41e9020 2024-05-09T03:09:27.838 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.835+0000 7f81b27fc640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 0x7f81b41f5f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81a8034040 tx=0x7f81a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.835+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8040020 con 0x7f81b41e9020 2024-05-09T03:09:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.835+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a8003a40 con 0x7f81b41e9020 2024-05-09T03:09:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.835+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8036040 con 0x7f81b41e9020 2024-05-09T03:09:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.835+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b41f6450 con 0x7f81b41e9020 2024-05-09T03:09:27.839 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.835+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b415b7a0 con 0x7f81b41e9020 2024-05-09T03:09:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.836+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b415b210 con 0x7f81b41e9020 2024-05-09T03:09:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.836+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81a8043020 con 0x7f81b41e9020 2024-05-09T03:09:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.837+0000 7f819bfff640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81a00762d0 0x7f81a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.837+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f81a80b7400 con 0x7f81b41e9020 2024-05-09T03:09:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.837+0000 7f81b2ffd640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81a00762d0 0x7f81a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:27.840 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.838+0000 7f81b2ffd640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81a00762d0 0x7f81a00787e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81a4001050 tx=0x7f81a4001be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:27.843 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:27.842+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f81a804a070 con 0x7f81b41e9020 2024-05-09T03:09:28.033 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.031+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81b406a270 con 0x7f81b41e9020 2024-05-09T03:09:28.034 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.032+0000 7f819bfff640 1 -- 172.21.15.46:0/2942506209 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12812 (secure 0 0 0) 0x7f81b41588c0 con 0x7f81b41e9020 2024-05-09T03:09:28.034 INFO:teuthology.misc.health.smithi046.stdout: 2024-05-09T03:09:28.034 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":16,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:24.649380+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:28.036 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81a00762d0 msgr2=0x7f81a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.036 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81a00762d0 0x7f81a00787e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81a4001050 tx=0x7f81a4001be0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.036 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 msgr2=0x7f81b41f5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.036 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 0x7f81b41f5f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f81a8034040 tx=0x7f81a8004730 comp rx=0 tx=0).stop 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 shutdown_connections 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b4157f70 0x7f81b4158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81a00762d0 0x7f81a00787e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b41e9020 0x7f81b41f5f10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 --2- 172.21.15.46:0/2942506209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f819c004210 0x7f81b415ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 >> 172.21.15.46:0/2942506209 conn(0x7f81b406edc0 msgr2=0x7f81b40d1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 shutdown_connections 2024-05-09T03:09:28.037 INFO:teuthology.misc.health.smithi046.stderr:2024-05-09T03:09:28.034+0000 7f81b985b640 1 -- 172.21.15.46:0/2942506209 wait complete. 2024-05-09T03:09:28.046 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-09T03:09:28.046 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-09T03:09:28.047 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:09:28.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:09:28.049 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:09:28.049 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:09:28.049 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:09:28.051 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:09:28.051 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:09:28.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:09:28.054 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:09:28.054 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:09:28.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:09:28.057 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:09:28.057 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-09T03:09:28.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:09:28.140 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:09:28.140 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:09:28.151 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:09:28.153 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:09:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.177+0000 7fd3b3447640 1 Processor -- start 2024-05-09T03:09:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.177+0000 7fd3b3447640 1 -- start start 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.178+0000 7fd3b3447640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac15d180 0x7fd3ac0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.178+0000 7fd3b3447640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd3ac0d30f0 0x7fd3ac0b4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.178+0000 7fd3b3447640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3ac15d590 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.178+0000 7fd3b3447640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd3ac06eaf0 con 0x7fd3ac0d30f0 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.178+0000 7fd3b3447640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac0b2af0 con 0x7fd3ac15d180 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b09e2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3ac0b26e0 0x7fd3ac0d29e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38004/0 (socket says 172.21.15.46:38004) 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b09e2640 1 -- 172.21.15.46:0/3852907434 learned_addr learned my addr 172.21.15.46:0/3852907434 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b11e3640 1 --2- 172.21.15.46:0/3852907434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac15d180 0x7fd3ac0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b19e4640 1 --2- 172.21.15.46:0/3852907434 >> v2:172.21.15.88:3300/0 conn(0x7fd3ac0d30f0 0x7fd3ac0b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3a3fff640 1 -- 172.21.15.46:0/3852907434 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439118324 0 0) 0x7fd39c001110 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3a3fff640 1 -- 172.21.15.46:0/3852907434 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd398003520 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b11e3640 1 -- 172.21.15.46:0/3852907434 >> v2:172.21.15.88:3300/0 conn(0x7fd3ac0d30f0 msgr2=0x7fd3ac0b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b11e3640 1 --2- 172.21.15.46:0/3852907434 >> v2:172.21.15.88:3300/0 conn(0x7fd3ac0d30f0 0x7fd3ac0b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b11e3640 1 -- 172.21.15.46:0/3852907434 >> v1:172.21.15.46:6789/0 conn(0x7fd3ac0b26e0 legacy=0x7fd3ac0d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.179+0000 7fd3b11e3640 1 -- 172.21.15.46:0/3852907434 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac06e1c0 con 0x7fd3ac15d180 2024-05-09T03:09:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.180+0000 7fd3b11e3640 1 --2- 172.21.15.46:0/3852907434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac15d180 0x7fd3ac0b21a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009790 tx=0x7fd3a802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d804f51bfadf9c2a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.180+0000 7fd3a3fff640 1 -- 172.21.15.46:0/3852907434 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8037070 con 0x7fd3ac15d180 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.180+0000 7fd3a3fff640 1 -- 172.21.15.46:0/3852907434 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a8032440 con 0x7fd3ac15d180 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.180+0000 7fd3a3fff640 1 -- 172.21.15.46:0/3852907434 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8032810 con 0x7fd3ac15d180 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 -- 172.21.15.46:0/3852907434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac15d180 msgr2=0x7fd3ac0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 --2- 172.21.15.46:0/3852907434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac15d180 0x7fd3ac0b21a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009790 tx=0x7fd3a802a180 comp rx=0 tx=0).stop 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 -- 172.21.15.46:0/3852907434 shutdown_connections 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 --2- 172.21.15.46:0/3852907434 >> v2:172.21.15.88:3300/0 conn(0x7fd3ac0d30f0 0x7fd3ac0b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 --2- 172.21.15.46:0/3852907434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac15d180 0x7fd3ac0b21a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 -- 172.21.15.46:0/3852907434 >> 172.21.15.46:0/3852907434 conn(0x7fd3ac09b7d0 msgr2=0x7fd3ac09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 -- 172.21.15.46:0/3852907434 shutdown_connections 2024-05-09T03:09:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.181+0000 7fd3b3447640 1 -- 172.21.15.46:0/3852907434 wait complete. 2024-05-09T03:09:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.183+0000 7fd3b3447640 1 Processor -- start 2024-05-09T03:09:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.183+0000 7fd3b3447640 1 -- start start 2024-05-09T03:09:28.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b3447640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 0x7fd3ac0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:28.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b3447640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac0d30f0 0x7fd3ac0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:28.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b3447640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3ac15d180 0x7fd3ac0c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:28.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b3447640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac06ef80 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b3447640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac06e780 con 0x7fd3ac15d180 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b3447640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac0b4570 con 0x7fd3ac0d30f0 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b11e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 0x7fd3ac0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b11e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 0x7fd3ac0cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33194/0 (socket says 172.21.15.46:33194) 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b11e3640 1 -- 172.21.15.46:0/1644911195 learned_addr learned my addr 172.21.15.46:0/1644911195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b09e2640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac0d30f0 0x7fd3ac0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.184+0000 7fd3b19e4640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3ac15d180 0x7fd3ac0c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b11e3640 1 -- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac0d30f0 msgr2=0x7fd3ac0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b11e3640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac0d30f0 0x7fd3ac0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b11e3640 1 -- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3ac15d180 msgr2=0x7fd3ac0c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b11e3640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3ac15d180 0x7fd3ac0c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b11e3640 1 -- 172.21.15.46:0/1644911195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac0cc370 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b11e3640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 0x7fd3ac0cb4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8035040 tx=0x7fd3a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:28.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8037040 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a8042070 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac1602f0 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8006030 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.185+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac147c20 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.186+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ac06a270 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.187+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd3a80061d0 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.188+0000 7fd3a1ffb640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd390076300 0x7fd390078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.188+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd3a80b85e0 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.188+0000 7fd3b09e2640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd390076300 0x7fd390078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.188+0000 7fd3b09e2640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd390076300 0x7fd390078810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0c9580 tx=0x7fd39c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.192+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd3a8081d50 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.420+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3ac147e50 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.421+0000 7fd3a1ffb640 1 -- 172.21.15.46:0/1644911195 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7fd3a808a390 con 0x7fd3ac0b26e0 2024-05-09T03:09:28.423 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:28.423 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T03:09:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-05-09T03:09:28.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd390076300 msgr2=0x7fd390078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd390076300 0x7fd390078810 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0c9580 tx=0x7fd39c009040 comp rx=0 tx=0).stop 2024-05-09T03:09:28.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 msgr2=0x7fd3ac0cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:28.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 0x7fd3ac0cb4a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8035040 tx=0x7fd3a80046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 shutdown_connections 2024-05-09T03:09:28.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3ac15d180 0x7fd3ac0c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3ac0d30f0 0x7fd3ac0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd390076300 0x7fd390078810 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 --2- 172.21.15.46:0/1644911195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ac0b26e0 0x7fd3ac0cb4a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:28.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 >> 172.21.15.46:0/1644911195 conn(0x7fd3ac09b7d0 msgr2=0x7fd3ac160ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:28.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 shutdown_connections 2024-05-09T03:09:28.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:28.423+0000 7fd3b3447640 1 -- 172.21.15.46:0/1644911195 wait complete. 2024-05-09T03:09:30.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:09:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.564+0000 7fa0aa805640 1 Processor -- start 2024-05-09T03:09:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.565+0000 7fa0aa805640 1 -- start start 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0aa805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a4164c70 0x7fa0a41615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0aa805640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0a415dfe0 0x7fa0a40d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0aa805640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a4160b20 con 0x7fa0a40d2480 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0aa805640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa0a4161150 con 0x7fa0a415dfe0 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0aa805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a40d15f0 con 0x7fa0a4164c70 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0a8da2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa0a40d2480 0x7fa0a40d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38014/0 (socket says 172.21.15.46:38014) 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0a8da2640 1 -- 172.21.15.46:0/760847789 learned_addr learned my addr 172.21.15.46:0/760847789 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0a3fff640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a4164c70 0x7fa0a41615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0a37fe640 1 --2- 172.21.15.46:0/760847789 >> v2:172.21.15.88:3300/0 conn(0x7fa0a415dfe0 0x7fa0a40d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.566+0000 7fa0a2ffd640 1 -- 172.21.15.46:0/760847789 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185650174 0 0) 0x7fa0a4160b20 con 0x7fa0a40d2480 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.567+0000 7fa0a2ffd640 1 -- 172.21.15.46:0/760847789 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c003520 con 0x7fa0a40d2480 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.567+0000 7fa0a37fe640 1 -- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a4164c70 msgr2=0x7fa0a41615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.567+0000 7fa0a37fe640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a4164c70 0x7fa0a41615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.567+0000 7fa0a37fe640 1 -- 172.21.15.46:0/760847789 >> v1:172.21.15.46:6789/0 conn(0x7fa0a40d2480 legacy=0x7fa0a40d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.567+0000 7fa0a37fe640 1 -- 172.21.15.46:0/760847789 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a40d18d0 con 0x7fa0a415dfe0 2024-05-09T03:09:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.567+0000 7fa0a3fff640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a4164c70 0x7fa0a41615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0a37fe640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a415dfe0 0x7fa0a40d0dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097a0 tx=0x7fa09802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fba52f0e44faba3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0a2ffd640 1 -- 172.21.15.46:0/760847789 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098037070 con 0x7fa0a415dfe0 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0a2ffd640 1 -- 172.21.15.46:0/760847789 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa09802ae50 con 0x7fa0a415dfe0 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0a2ffd640 1 -- 172.21.15.46:0/760847789 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098032660 con 0x7fa0a415dfe0 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0aa805640 1 -- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a415dfe0 msgr2=0x7fa0a40d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0aa805640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a415dfe0 0x7fa0a40d0dc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097a0 tx=0x7fa09802a100 comp rx=0 tx=0).stop 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0aa805640 1 -- 172.21.15.46:0/760847789 shutdown_connections 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0aa805640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a415dfe0 0x7fa0a40d0dc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0aa805640 1 --2- 172.21.15.46:0/760847789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a4164c70 0x7fa0a41615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.568+0000 7fa0aa805640 1 -- 172.21.15.46:0/760847789 >> 172.21.15.46:0/760847789 conn(0x7fa0a4073db0 msgr2=0x7fa0a40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.569+0000 7fa0aa805640 1 -- 172.21.15.46:0/760847789 shutdown_connections 2024-05-09T03:09:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.569+0000 7fa0aa805640 1 -- 172.21.15.46:0/760847789 wait complete. 2024-05-09T03:09:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.570+0000 7fa0aa805640 1 Processor -- start 2024-05-09T03:09:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.571+0000 7fa0aa805640 1 -- start start 2024-05-09T03:09:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0aa805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a40d2480 0x7fa0a4152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0aa805640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 0x7fa0a4161ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0aa805640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a4164c70 0x7fa0a40ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0aa805640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a415e3f0 con 0x7fa0a415dfe0 2024-05-09T03:09:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0aa805640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40d1300 con 0x7fa0a4164c70 2024-05-09T03:09:30.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0aa805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a40d45c0 con 0x7fa0a40d2480 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 0x7fa0a4161ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 0x7fa0a4161ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33208/0 (socket says 172.21.15.46:33208) 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a37fe640 1 -- 172.21.15.46:0/2513090920 learned_addr learned my addr 172.21.15.46:0/2513090920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a3fff640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a40d2480 0x7fa0a4152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a8da2640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a4164c70 0x7fa0a40ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a3fff640 1 -- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a4164c70 msgr2=0x7fa0a40ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a3fff640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a4164c70 0x7fa0a40ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a3fff640 1 -- 172.21.15.46:0/2513090920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 msgr2=0x7fa0a4161ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a3fff640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 0x7fa0a4161ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a3fff640 1 -- 172.21.15.46:0/2513090920 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a4162430 con 0x7fa0a40d2480 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a8da2640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a4164c70 0x7fa0a40ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.572+0000 7fa0a37fe640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 0x7fa0a4161ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.573+0000 7fa0a3fff640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a40d2480 0x7fa0a4152610 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa09400e8a0 tx=0x7fa09400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.573+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa094017070 con 0x7fa0a40d2480 2024-05-09T03:09:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.573+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa09400c940 con 0x7fa0a40d2480 2024-05-09T03:09:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.573+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a4153920 con 0x7fa0a40d2480 2024-05-09T03:09:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.573+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09400cd30 con 0x7fa0a40d2480 2024-05-09T03:09:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.573+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a4153ee0 con 0x7fa0a40d2480 2024-05-09T03:09:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.574+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a408cfd0 con 0x7fa0a40d2480 2024-05-09T03:09:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.575+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa09401a070 con 0x7fa0a40d2480 2024-05-09T03:09:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.575+0000 7fa0a0ff9640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa090076300 0x7fa090078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.575+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa094097670 con 0x7fa0a40d2480 2024-05-09T03:09:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.575+0000 7fa0a37fe640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa090076300 0x7fa090078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.576+0000 7fa0a37fe640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa090076300 0x7fa090078810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097a0 tx=0x7fa09802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:30.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.581+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa0a408cfd0 con 0x7fa0a40d2480 2024-05-09T03:09:30.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.812+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0a406a270 con 0x7fa0a40d2480 2024-05-09T03:09:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.813+0000 7fa0a0ff9640 1 -- 172.21.15.46:0/2513090920 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3855 (secure 0 0 0) 0x7fa094060db0 con 0x7fa0a40d2480 2024-05-09T03:09:30.815 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:30.815 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T03:09:30.815 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa090076300 msgr2=0x7fa090078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa090076300 0x7fa090078810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097a0 tx=0x7fa09802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a40d2480 msgr2=0x7fa0a4152610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a40d2480 0x7fa0a4152610 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa09400e8a0 tx=0x7fa09400ed60 comp rx=0 tx=0).stop 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 shutdown_connections 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0a4164c70 0x7fa0a40ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa090076300 0x7fa090078810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0a415dfe0 0x7fa0a4161ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 --2- 172.21.15.46:0/2513090920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0a40d2480 0x7fa0a4152610 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 >> 172.21.15.46:0/2513090920 conn(0x7fa0a4073db0 msgr2=0x7fa0a40b9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 shutdown_connections 2024-05-09T03:09:30.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.816+0000 7fa0aa805640 1 -- 172.21.15.46:0/2513090920 wait complete. 2024-05-09T03:09:30.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:09:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.942+0000 7ff495584640 1 Processor -- start 2024-05-09T03:09:30.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.942+0000 7ff495584640 1 -- start start 2024-05-09T03:09:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.943+0000 7ff495584640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900b40a0 0x7ff4900b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.943+0000 7ff495584640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff490160e20 con 0x7ff4900d2a80 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.943+0000 7ff495584640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff490161100 con 0x7ff4900d1ec0 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.943+0000 7ff495584640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4901613e0 con 0x7ff4900b40a0 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff487fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4900d2a80 0x7ff4900d2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38024/0 (socket says 172.21.15.46:38024) 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff487fff640 1 -- 172.21.15.46:0/2696465917 learned_addr learned my addr 172.21.15.46:0/2696465917 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48f7fe640 1 --2- 172.21.15.46:0/2696465917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900b40a0 0x7ff4900b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48e7fc640 1 -- 172.21.15.46:0/2696465917 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124700284 0 0) 0x7ff490160e20 con 0x7ff4900d2a80 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48e7fc640 1 -- 172.21.15.46:0/2696465917 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff47c003520 con 0x7ff4900d2a80 2024-05-09T03:09:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48e7fc640 1 -- 172.21.15.46:0/2696465917 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1258290700 0 0) 0x7ff47c003520 con 0x7ff4900d2a80 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48f7fe640 1 -- 172.21.15.46:0/2696465917 >> v1:172.21.15.88:6789/0 conn(0x7ff4900d1ec0 legacy=0x7ff4900b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48f7fe640 1 -- 172.21.15.46:0/2696465917 >> v1:172.21.15.46:6789/0 conn(0x7ff4900d2a80 legacy=0x7ff4900d2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.944+0000 7ff48f7fe640 1 -- 172.21.15.46:0/2696465917 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4901616c0 con 0x7ff4900b40a0 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.945+0000 7ff48f7fe640 1 --2- 172.21.15.46:0/2696465917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900b40a0 0x7ff4900b44b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff488009dc0 tx=0x7ff48802a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d42a347eed15a11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.945+0000 7ff48e7fc640 1 -- 172.21.15.46:0/2696465917 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488035020 con 0x7ff4900b40a0 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.945+0000 7ff48e7fc640 1 -- 172.21.15.46:0/2696465917 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff488007540 con 0x7ff4900b40a0 2024-05-09T03:09:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.945+0000 7ff48e7fc640 1 -- 172.21.15.46:0/2696465917 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4780016f0 con 0x7ff4900b40a0 2024-05-09T03:09:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 -- 172.21.15.46:0/2696465917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900b40a0 msgr2=0x7ff4900b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 --2- 172.21.15.46:0/2696465917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900b40a0 0x7ff4900b44b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff488009dc0 tx=0x7ff48802a500 comp rx=0 tx=0).stop 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 -- 172.21.15.46:0/2696465917 shutdown_connections 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 --2- 172.21.15.46:0/2696465917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900b40a0 0x7ff4900b44b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 -- 172.21.15.46:0/2696465917 >> 172.21.15.46:0/2696465917 conn(0x7ff49009b7d0 msgr2=0x7ff49009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 -- 172.21.15.46:0/2696465917 shutdown_connections 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.946+0000 7ff495584640 1 -- 172.21.15.46:0/2696465917 wait complete. 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.948+0000 7ff495584640 1 Processor -- start 2024-05-09T03:09:30.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.948+0000 7ff495584640 1 -- start start 2024-05-09T03:09:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff495584640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 0x7ff490155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff495584640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4900d1ec0 0x7ff4901f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff495584640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900d2a80 0x7ff490150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff495584640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4900d2450 con 0x7ff4900b40a0 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff495584640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4900d22d0 con 0x7ff4900d1ec0 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff495584640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4900b2900 con 0x7ff4900d2a80 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 0x7ff490155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 0x7ff490155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33218/0 (socket says 172.21.15.46:33218) 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 -- 172.21.15.46:0/1532172609 learned_addr learned my addr 172.21.15.46:0/1532172609 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff487fff640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4900d1ec0 0x7ff4901f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48f7fe640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900d2a80 0x7ff490150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 -- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900d2a80 msgr2=0x7ff490150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900d2a80 0x7ff490150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 -- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4900d1ec0 msgr2=0x7ff4901f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4900d1ec0 0x7ff4901f6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff48effd640 1 -- 172.21.15.46:0/1532172609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4900b0d80 con 0x7ff4900b40a0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.949+0000 7ff487fff640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4900d1ec0 0x7ff4901f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.950+0000 7ff48effd640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 0x7ff490155cb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c500 tx=0x7ff48000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.950+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480017070 con 0x7ff4900b40a0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.950+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4800035f0 con 0x7ff4900b40a0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.950+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4900b1060 con 0x7ff4900b40a0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.950+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480003a20 con 0x7ff4900b40a0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.950+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4900b1620 con 0x7ff4900b40a0 2024-05-09T03:09:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.951+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff49006a270 con 0x7ff4900b40a0 2024-05-09T03:09:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.952+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff48001a070 con 0x7ff4900b40a0 2024-05-09T03:09:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.952+0000 7ff4877fe640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff470076300 0x7ff470078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.952+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff480096670 con 0x7ff4900b40a0 2024-05-09T03:09:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.952+0000 7ff487fff640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff470076300 0x7ff470078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.953+0000 7ff487fff640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff470076300 0x7ff470078810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff49006db20 tx=0x7ff478009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:30.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:30.958+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff4800603b0 con 0x7ff4900b40a0 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.148+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff4900b3500 con 0x7ff4900b40a0 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.148+0000 7ff4877fe640 1 -- 172.21.15.46:0/1532172609 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7ff4800683a0 con 0x7ff4900b40a0 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:09:31.150 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:09:31.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:09:31.151 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff470076300 msgr2=0x7ff470078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff470076300 0x7ff470078810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff49006db20 tx=0x7ff478009040 comp rx=0 tx=0).stop 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 msgr2=0x7ff490155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 0x7ff490155cb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c500 tx=0x7ff48000c9c0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 shutdown_connections 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4900d2a80 0x7ff490150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4900d1ec0 0x7ff4901f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff470076300 0x7ff470078810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 --2- 172.21.15.46:0/1532172609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4900b40a0 0x7ff490155cb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 >> 172.21.15.46:0/1532172609 conn(0x7ff49009b7d0 msgr2=0x7ff49009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 shutdown_connections 2024-05-09T03:09:31.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.150+0000 7ff495584640 1 -- 172.21.15.46:0/1532172609 wait complete. 2024-05-09T03:09:31.165 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:09:31.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:09:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.282+0000 7fc4bc6b8640 1 Processor -- start 2024-05-09T03:09:31.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.282+0000 7fc4bc6b8640 1 -- start start 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bc6b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4b40b5140 0x7fc4b40b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bc6b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc4b40b8070 0x7fc4b406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bc6b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b406e250 0x7fc4b40d48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bc6b8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc4b40b9b60 con 0x7fc4b40b5140 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bc6b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc4b40b5550 con 0x7fc4b40b8070 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bc6b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b40b8480 con 0x7fc4b406e250 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4b40b5140 0x7fc4b40b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4b40b5140 0x7fc4b40b7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33222/0 (socket says 172.21.15.46:33222) 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 -- 172.21.15.46:0/3933165358 learned_addr learned my addr 172.21.15.46:0/3933165358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4bac55640 1 --2- 172.21.15.46:0/3933165358 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b406e250 0x7fc4b40d48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 -- 172.21.15.46:0/3933165358 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b406e250 msgr2=0x7fc4b40d48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 --2- 172.21.15.46:0/3933165358 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b406e250 0x7fc4b40d48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4b9c53640 1 --2- 172.21.15.46:0/3933165358 >> v2:172.21.15.88:3300/0 conn(0x7fc4b40b8070 0x7fc4b406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 -- 172.21.15.46:0/3933165358 >> v2:172.21.15.88:3300/0 conn(0x7fc4b40b8070 msgr2=0x7fc4b406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 --2- 172.21.15.46:0/3933165358 >> v2:172.21.15.88:3300/0 conn(0x7fc4b40b8070 0x7fc4b406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 -- 172.21.15.46:0/3933165358 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b406e660 con 0x7fc4b40b5140 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4ba454640 1 --2- 172.21.15.46:0/3933165358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b40b5140 0x7fc4b40b7b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40b7ac0 tx=0x7fc4b002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467b62d5033edf77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4b9452640 1 -- 172.21.15.46:0/3933165358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0036070 con 0x7fc4b40b5140 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4b9452640 1 -- 172.21.15.46:0/3933165358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b0031600 con 0x7fc4b40b5140 2024-05-09T03:09:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.285+0000 7fc4b9452640 1 -- 172.21.15.46:0/3933165358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00319d0 con 0x7fc4b40b5140 2024-05-09T03:09:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3933165358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b40b5140 msgr2=0x7fc4b40b7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3933165358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b40b5140 0x7fc4b40b7b30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40b7ac0 tx=0x7fc4b002a6a0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3933165358 shutdown_connections 2024-05-09T03:09:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3933165358 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b406e250 0x7fc4b40d48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3933165358 >> v2:172.21.15.88:3300/0 conn(0x7fc4b40b8070 0x7fc4b406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3933165358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b40b5140 0x7fc4b40b7b30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3933165358 >> 172.21.15.46:0/3933165358 conn(0x7fc4b40d5e90 msgr2=0x7fc4b40d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3933165358 shutdown_connections 2024-05-09T03:09:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.286+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3933165358 wait complete. 2024-05-09T03:09:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.289+0000 7fc4bc6b8640 1 Processor -- start 2024-05-09T03:09:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.289+0000 7fc4bc6b8640 1 -- start start 2024-05-09T03:09:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 0x7fc4b415e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4b40b5140 0x7fc4b40ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b40b8070 0x7fc4b40ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b40b9510 con 0x7fc4b406e250 2024-05-09T03:09:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b40b9270 con 0x7fc4b40b5140 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b40b97f0 con 0x7fc4b40b8070 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 0x7fc4b415e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 0x7fc4b415e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33232/0 (socket says 172.21.15.46:33232) 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 -- 172.21.15.46:0/3875709015 learned_addr learned my addr 172.21.15.46:0/3875709015 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bac55640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b40b8070 0x7fc4b40ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4b9c53640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4b40b5140 0x7fc4b40ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 -- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b40b8070 msgr2=0x7fc4b40ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b40b8070 0x7fc4b40ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 -- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4b40b5140 msgr2=0x7fc4b40ae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4b40b5140 0x7fc4b40ae470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 -- 172.21.15.46:0/3875709015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b4161c90 con 0x7fc4b406e250 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4b9c53640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4b40b5140 0x7fc4b40ae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4ba454640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 0x7fc4b415e360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0034040 tx=0x7fc4b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:31.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0036050 con 0x7fc4b406e250 2024-05-09T03:09:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b4161f70 con 0x7fc4b406e250 2024-05-09T03:09:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b0042070 con 0x7fc4b406e250 2024-05-09T03:09:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0009510 con 0x7fc4b406e250 2024-05-09T03:09:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.290+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b4162530 con 0x7fc4b406e250 2024-05-09T03:09:31.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.291+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b406a270 con 0x7fc4b406e250 2024-05-09T03:09:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.292+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc4b0007c30 con 0x7fc4b406e250 2024-05-09T03:09:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.292+0000 7fc4aaffd640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc4980762d0 0x7fc4980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.292+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fc4b00b8d40 con 0x7fc4b406e250 2024-05-09T03:09:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.292+0000 7fc4b9c53640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc4980762d0 0x7fc4980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:31.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.293+0000 7fc4b9c53640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc4980762d0 0x7fc4980787e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40cf7c0 tx=0x7fc4a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.297+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc4b00829f0 con 0x7fc4b406e250 2024-05-09T03:09:31.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:31.486+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc4b40d1290 con 0x7fc4b406e250 2024-05-09T03:09:32.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.161+0000 7fc4aaffd640 1 -- 172.21.15.46:0/3875709015 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fc4b00822f0 con 0x7fc4b406e250 2024-05-09T03:09:32.164 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:09:32.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.164+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc4980762d0 msgr2=0x7fc4980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:32.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.164+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc4980762d0 0x7fc4980787e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40cf7c0 tx=0x7fc4a4008040 comp rx=0 tx=0).stop 2024-05-09T03:09:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.164+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 msgr2=0x7fc4b415e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.164+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 0x7fc4b415e360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0034040 tx=0x7fc4b0004a40 comp rx=0 tx=0).stop 2024-05-09T03:09:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 shutdown_connections 2024-05-09T03:09:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc4b40b8070 0x7fc4b40ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4b40b5140 0x7fc4b40ae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc4980762d0 0x7fc4980787e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 --2- 172.21.15.46:0/3875709015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4b406e250 0x7fc4b415e360 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 >> 172.21.15.46:0/3875709015 conn(0x7fc4b40d5e90 msgr2=0x7fc4b40ca490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 shutdown_connections 2024-05-09T03:09:32.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.165+0000 7fc4bc6b8640 1 -- 172.21.15.46:0/3875709015 wait complete. 2024-05-09T03:09:32.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:09:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.313+0000 7f94e6278640 1 Processor -- start 2024-05-09T03:09:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.313+0000 7f94e6278640 1 -- start start 2024-05-09T03:09:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94e6278640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94e00b52a0 0x7f94e006e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94e6278640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e00d1730 con 0x7f94e00d1000 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94e6278640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e00b96c0 con 0x7f94e00b7e50 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94e6278640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f94e0160b40 con 0x7f94e00b52a0 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94dffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94e00b52a0 0x7f94e006e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94dffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94e00b52a0 0x7f94e006e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33244/0 (socket says 172.21.15.46:33244) 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94dffff640 1 -- 172.21.15.46:0/1151638132 learned_addr learned my addr 172.21.15.46:0/1151638132 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94d7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94e00d1000 0x7f94e00b7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38034/0 (socket says 172.21.15.46:38034) 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94dffff640 1 -- 172.21.15.46:0/1151638132 >> v1:172.21.15.88:6789/0 conn(0x7f94e00b7e50 legacy=0x7f94e00ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94dffff640 1 -- 172.21.15.46:0/1151638132 >> v1:172.21.15.46:6789/0 conn(0x7f94e00d1000 legacy=0x7f94e00b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.314+0000 7f94dffff640 1 -- 172.21.15.46:0/1151638132 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e00b99a0 con 0x7f94e00b52a0 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.315+0000 7f94dffff640 1 --2- 172.21.15.46:0/1151638132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e00b52a0 0x7f94e006e7f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009870 tx=0x7f94d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4831230a068d82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.315+0000 7f94df7fe640 1 -- 172.21.15.46:0/1151638132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0037070 con 0x7f94e00b52a0 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.315+0000 7f94df7fe640 1 -- 172.21.15.46:0/1151638132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d0032440 con 0x7f94e00b52a0 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.315+0000 7f94df7fe640 1 -- 172.21.15.46:0/1151638132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d0032810 con 0x7f94e00b52a0 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 -- 172.21.15.46:0/1151638132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e00b52a0 msgr2=0x7f94e006e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 --2- 172.21.15.46:0/1151638132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e00b52a0 0x7f94e006e7f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009870 tx=0x7f94d002a270 comp rx=0 tx=0).stop 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 -- 172.21.15.46:0/1151638132 shutdown_connections 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 --2- 172.21.15.46:0/1151638132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e00b52a0 0x7f94e006e7f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 -- 172.21.15.46:0/1151638132 >> 172.21.15.46:0/1151638132 conn(0x7f94e00d2920 msgr2=0x7f94e00d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 -- 172.21.15.46:0/1151638132 shutdown_connections 2024-05-09T03:09:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.316+0000 7f94e6278640 1 -- 172.21.15.46:0/1151638132 wait complete. 2024-05-09T03:09:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.317+0000 7f94e6278640 1 Processor -- start 2024-05-09T03:09:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.318+0000 7f94e6278640 1 -- start start 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e6278640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94e00b52a0 0x7f94e0161f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e6278640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00d1000 0x7f94e0151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e6278640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 0x7f94e015d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e6278640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94e00d1410 con 0x7f94e015a3d0 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e6278640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f94e00b9d10 con 0x7f94e00d1000 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e6278640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f94e00ba8b0 con 0x7f94e00b52a0 2024-05-09T03:09:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 0x7f94e015d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 0x7f94e015d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33258/0 (socket says 172.21.15.46:33258) 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 -- 172.21.15.46:0/2728883848 learned_addr learned my addr 172.21.15.46:0/2728883848 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94dffff640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94e00b52a0 0x7f94e0161f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94d7fff640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00d1000 0x7f94e0151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 -- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94e00b52a0 msgr2=0x7f94e0161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94e00b52a0 0x7f94e0161f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 -- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00d1000 msgr2=0x7f94e0151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00d1000 0x7f94e0151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 -- 172.21.15.46:0/2728883848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e0162830 con 0x7f94e015a3d0 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.319+0000 7f94e4815640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 0x7f94e015d650 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94d800b9e0 tx=0x7f94d800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.320+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8003b70 con 0x7f94e015a3d0 2024-05-09T03:09:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.320+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d8003d10 con 0x7f94e015a3d0 2024-05-09T03:09:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.320+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e00ce020 con 0x7f94e015a3d0 2024-05-09T03:09:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.320+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d800cb60 con 0x7f94e015a3d0 2024-05-09T03:09:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.320+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e00ce5e0 con 0x7f94e015a3d0 2024-05-09T03:09:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.321+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e0068f60 con 0x7f94e015a3d0 2024-05-09T03:09:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.323+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f94d800cd00 con 0x7f94e015a3d0 2024-05-09T03:09:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.323+0000 7f94dd7fa640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94c00762d0 0x7f94c00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.323+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f94d8097220 con 0x7f94e015a3d0 2024-05-09T03:09:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.323+0000 7f94dffff640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94c00762d0 0x7f94c00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.323+0000 7f94dffff640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94c00762d0 0x7f94c00787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94d0034000 tx=0x7f94d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.328+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f94e0068f60 con 0x7f94e015a3d0 2024-05-09T03:09:32.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:32.521+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f94e006a270 con 0x7f94e015a3d0 2024-05-09T03:09:33.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.168+0000 7f94dd7fa640 1 -- 172.21.15.46:0/2728883848 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f94d8060680 con 0x7f94e015a3d0 2024-05-09T03:09:33.169 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94c00762d0 msgr2=0x7f94c00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94c00762d0 0x7f94c00787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94d0034000 tx=0x7f94d002e000 comp rx=0 tx=0).stop 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 msgr2=0x7f94e015d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 0x7f94e015d650 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94d800b9e0 tx=0x7f94d800bea0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 shutdown_connections 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94c00762d0 0x7f94c00787e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94e015a3d0 0x7f94e015d650 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94e00d1000 0x7f94e0151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 --2- 172.21.15.46:0/2728883848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94e00b52a0 0x7f94e0161f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.172+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 >> 172.21.15.46:0/2728883848 conn(0x7f94e00d2920 msgr2=0x7f94e0161480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.173+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 shutdown_connections 2024-05-09T03:09:33.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.173+0000 7f94e6278640 1 -- 172.21.15.46:0/2728883848 wait complete. 2024-05-09T03:09:33.201 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:09:33.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.320+0000 7f9b6f1b2640 1 Processor -- start 2024-05-09T03:09:33.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.320+0000 7f9b6f1b2640 1 -- start start 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.321+0000 7f9b6f1b2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9b680d2eb0 0x7f9b680d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.321+0000 7f9b6f1b2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b680b95a0 0x7f9b680b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.321+0000 7f9b6f1b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b680d4a60 con 0x7f9b680b7760 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.321+0000 7f9b6f1b2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9b680d4d40 con 0x7f9b680b95a0 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.321+0000 7f9b6f1b2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9b680d5020 con 0x7f9b680d2eb0 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.321+0000 7f9b6d74f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b680b95a0 0x7f9b680b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6d74f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9b680b95a0 0x7f9b680b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33270/0 (socket says 172.21.15.46:33270) 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6d74f640 1 -- 172.21.15.46:0/496853064 learned_addr learned my addr 172.21.15.46:0/496853064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6cf4e640 1 --2- 172.21.15.46:0/496853064 >> v2:172.21.15.88:3300/0 conn(0x7f9b680d2eb0 0x7f9b680d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6d74f640 1 -- 172.21.15.46:0/496853064 >> v2:172.21.15.88:3300/0 conn(0x7f9b680d2eb0 msgr2=0x7f9b680d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6d74f640 1 --2- 172.21.15.46:0/496853064 >> v2:172.21.15.88:3300/0 conn(0x7f9b680d2eb0 0x7f9b680d32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6d74f640 1 -- 172.21.15.46:0/496853064 >> v1:172.21.15.88:6789/0 conn(0x7f9b680b7760 legacy=0x7f9b680b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6d74f640 1 -- 172.21.15.46:0/496853064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b680d5300 con 0x7f9b680b95a0 2024-05-09T03:09:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.322+0000 7f9b6cf4e640 1 --2- 172.21.15.46:0/496853064 >> v2:172.21.15.88:3300/0 conn(0x7f9b680d2eb0 0x7f9b680d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6d74f640 1 --2- 172.21.15.46:0/496853064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b680b99b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b680d6d90 tx=0x7f9b6002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d396df295dde358 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b5f7fe640 1 -- 172.21.15.46:0/496853064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60036070 con 0x7f9b680b95a0 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b5f7fe640 1 -- 172.21.15.46:0/496853064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b60031600 con 0x7f9b680b95a0 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b5f7fe640 1 -- 172.21.15.46:0/496853064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b600319d0 con 0x7f9b680b95a0 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/496853064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 msgr2=0x7f9b680b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/496853064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b680b99b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b680d6d90 tx=0x7f9b6002a730 comp rx=0 tx=0).stop 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/496853064 shutdown_connections 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/496853064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b680b99b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/496853064 >> v2:172.21.15.88:3300/0 conn(0x7f9b680d2eb0 0x7f9b680d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.323+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/496853064 >> 172.21.15.46:0/496853064 conn(0x7f9b6808d390 msgr2=0x7f9b6808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.324+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/496853064 shutdown_connections 2024-05-09T03:09:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.324+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/496853064 wait complete. 2024-05-09T03:09:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.325+0000 7f9b6f1b2640 1 Processor -- start 2024-05-09T03:09:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.325+0000 7f9b6f1b2640 1 -- start start 2024-05-09T03:09:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6f1b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b680b7760 0x7f9b6815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6f1b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b681e92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6f1b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b680d2eb0 0x7f9b6815caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6f1b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b680b5360 con 0x7f9b680b95a0 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6f1b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9b680b51e0 con 0x7f9b680d2eb0 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6f1b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9b68163bd0 con 0x7f9b680b7760 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b681e92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b681e92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33284/0 (socket says 172.21.15.46:33284) 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 -- 172.21.15.46:0/518654314 learned_addr learned my addr 172.21.15.46:0/518654314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6cf4e640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b680b7760 0x7f9b6815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b6d74f640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b680d2eb0 0x7f9b6815caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 -- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b680b7760 msgr2=0x7f9b6815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b680b7760 0x7f9b6815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 -- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b680d2eb0 msgr2=0x7f9b6815caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b680d2eb0 0x7f9b6815caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 -- 172.21.15.46:0/518654314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6815e350 con 0x7f9b680b95a0 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5ffff640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b681e92b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c3d0 tx=0x7f9b5800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.327+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58017070 con 0x7f9b680b95a0 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.328+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b580052d0 con 0x7f9b680b95a0 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.328+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b680cb8f0 con 0x7f9b680b95a0 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.328+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58003830 con 0x7f9b680b95a0 2024-05-09T03:09:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.328+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b680cbeb0 con 0x7f9b680b95a0 2024-05-09T03:09:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.328+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6806a270 con 0x7f9b680b95a0 2024-05-09T03:09:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.329+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9b5801a070 con 0x7f9b680b95a0 2024-05-09T03:09:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.330+0000 7f9b5d7fa640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9b540762d0 0x7f9b540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.330+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f9b58096980 con 0x7f9b680b95a0 2024-05-09T03:09:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.330+0000 7f9b6cf4e640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9b540762d0 0x7f9b540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.330+0000 7f9b6cf4e640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9b540762d0 0x7f9b540787e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b64006fd0 tx=0x7f9b64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.335+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9b5805fbc0 con 0x7f9b680b95a0 2024-05-09T03:09:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:33.567+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9b680c9420 con 0x7f9b680b95a0 2024-05-09T03:09:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.170+0000 7f9b5d7fa640 1 -- 172.21.15.46:0/518654314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f9b58068290 con 0x7f9b680b95a0 2024-05-09T03:09:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T03:09:34.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.175+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9b540762d0 msgr2=0x7f9b540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.175+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9b540762d0 0x7f9b540787e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9b64006fd0 tx=0x7f9b64008040 comp rx=0 tx=0).stop 2024-05-09T03:09:34.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.175+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 msgr2=0x7f9b681e92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.175+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b681e92b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800c3d0 tx=0x7f9b5800c890 comp rx=0 tx=0).stop 2024-05-09T03:09:34.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.177+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 shutdown_connections 2024-05-09T03:09:34.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.177+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9b680d2eb0 0x7f9b6815caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.177+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9b540762d0 0x7f9b540787e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.177+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b680b95a0 0x7f9b681e92b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.177+0000 7f9b6f1b2640 1 --2- 172.21.15.46:0/518654314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9b680b7760 0x7f9b6815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.177+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 >> 172.21.15.46:0/518654314 conn(0x7f9b6808d390 msgr2=0x7f9b68162260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:34.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.178+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 shutdown_connections 2024-05-09T03:09:34.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.182+0000 7f9b6f1b2640 1 -- 172.21.15.46:0/518654314 wait complete. 2024-05-09T03:09:34.217 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:34.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.334+0000 7f0e36338640 1 Processor -- start 2024-05-09T03:09:34.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.334+0000 7f0e36338640 1 -- start start 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e36338640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e301604a0 0x7f0e300d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e36338640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e300b7ef0 0x7f0e300b9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e36338640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e300d1920 con 0x7f0e300d1170 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e36338640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0e300d5130 con 0x7f0e301604a0 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e36338640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0e300d1640 con 0x7f0e300b7ef0 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e301604a0 0x7f0e300d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0e300d1170 0x7f0e300b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38038/0 (socket says 172.21.15.46:38038) 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2f7fe640 1 -- 172.21.15.46:0/1546003601 learned_addr learned my addr 172.21.15.46:0/1546003601 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e348d5640 1 --2- 172.21.15.46:0/1546003601 >> v2:172.21.15.88:3300/0 conn(0x7f0e300b7ef0 0x7f0e300b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1546003601 >> v2:172.21.15.88:3300/0 conn(0x7f0e300b7ef0 msgr2=0x7f0e300b9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2ffff640 1 --2- 172.21.15.46:0/1546003601 >> v2:172.21.15.88:3300/0 conn(0x7f0e300b7ef0 0x7f0e300b9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1546003601 >> v1:172.21.15.46:6789/0 conn(0x7f0e300d1170 legacy=0x7f0e300b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.336+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1546003601 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e300d4ae0 con 0x7f0e301604a0 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.337+0000 7f0e2ffff640 1 --2- 172.21.15.46:0/1546003601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e301604a0 0x7f0e300d0c30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e300d2e80 tx=0x7f0e2002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb2c0409a10580c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.337+0000 7f0e2effd640 1 -- 172.21.15.46:0/1546003601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e20037070 con 0x7f0e301604a0 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.337+0000 7f0e2effd640 1 -- 172.21.15.46:0/1546003601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e20032440 con 0x7f0e301604a0 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.337+0000 7f0e2effd640 1 -- 172.21.15.46:0/1546003601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e20032810 con 0x7f0e301604a0 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 -- 172.21.15.46:0/1546003601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e301604a0 msgr2=0x7f0e300d0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 --2- 172.21.15.46:0/1546003601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e301604a0 0x7f0e300d0c30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e300d2e80 tx=0x7f0e2002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 -- 172.21.15.46:0/1546003601 shutdown_connections 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 --2- 172.21.15.46:0/1546003601 >> v2:172.21.15.88:3300/0 conn(0x7f0e300b7ef0 0x7f0e300b9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 --2- 172.21.15.46:0/1546003601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e301604a0 0x7f0e300d0c30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 -- 172.21.15.46:0/1546003601 >> 172.21.15.46:0/1546003601 conn(0x7f0e300741d0 msgr2=0x7f0e300745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:34.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 -- 172.21.15.46:0/1546003601 shutdown_connections 2024-05-09T03:09:34.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.338+0000 7f0e36338640 1 -- 172.21.15.46:0/1546003601 wait complete. 2024-05-09T03:09:34.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.340+0000 7f0e36338640 1 Processor -- start 2024-05-09T03:09:34.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.340+0000 7f0e36338640 1 -- start start 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e36338640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 0x7f0e300b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e36338640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e301604a0 0x7f0e30159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e36338640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e300b6e90 0x7f0e300b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e36338640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0e300d55c0 con 0x7f0e300d1170 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e36338640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0e301608b0 con 0x7f0e301604a0 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e36338640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0e300ce1c0 con 0x7f0e300b6e90 2024-05-09T03:09:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 0x7f0e300b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 0x7f0e300b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33300/0 (socket says 172.21.15.46:33300) 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1057063122 learned_addr learned my addr 172.21.15.46:0/1057063122 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e348d5640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e300b6e90 0x7f0e300b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.341+0000 7f0e2f7fe640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e301604a0 0x7f0e30159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e300b6e90 msgr2=0x7f0e300b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2ffff640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e300b6e90 0x7f0e300b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e301604a0 msgr2=0x7f0e30159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2ffff640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e301604a0 0x7f0e30159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2ffff640 1 -- 172.21.15.46:0/1057063122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e300b77e0 con 0x7f0e300d1170 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2f7fe640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e301604a0 0x7f0e30159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e348d5640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e300b6e90 0x7f0e300b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2ffff640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 0x7f0e300b4af0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e20000c00 tx=0x7f0e200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e20037040 con 0x7f0e300d1170 2024-05-09T03:09:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e20042070 con 0x7f0e300d1170 2024-05-09T03:09:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e300b5460 con 0x7f0e300d1170 2024-05-09T03:09:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e20030bd0 con 0x7f0e300d1170 2024-05-09T03:09:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.342+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e300b5a20 con 0x7f0e300d1170 2024-05-09T03:09:34.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.343+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3006a270 con 0x7f0e300d1170 2024-05-09T03:09:34.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.344+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0e20047460 con 0x7f0e300d1170 2024-05-09T03:09:34.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.344+0000 7f0e2cff9640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e1c0762d0 0x7f0e1c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.344+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f0e200b8090 con 0x7f0e300d1170 2024-05-09T03:09:34.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.345+0000 7f0e2f7fe640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e1c0762d0 0x7f0e1c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.345+0000 7f0e2f7fe640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e1c0762d0 0x7f0e1c0787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e300c9420 tx=0x7f0e24009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.349+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0e300b77e0 con 0x7f0e300d1170 2024-05-09T03:09:34.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.541+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e301e9e30 con 0x7f0e300d1170 2024-05-09T03:09:34.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.542+0000 7f0e2cff9640 1 -- 172.21.15.46:0/1057063122 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16546 (secure 0 0 0) 0x7f0e20032e20 con 0x7f0e300d1170 2024-05-09T03:09:34.543 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:34.543 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:33.568527+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e1c0762d0 msgr2=0x7f0e1c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e1c0762d0 0x7f0e1c0787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e300c9420 tx=0x7f0e24009040 comp rx=0 tx=0).stop 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 msgr2=0x7f0e300b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 0x7f0e300b4af0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0e20000c00 tx=0x7f0e200046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 shutdown_connections 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e300b6e90 0x7f0e300b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e301604a0 0x7f0e30159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e1c0762d0 0x7f0e1c0787e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 --2- 172.21.15.46:0/1057063122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e300d1170 0x7f0e300b4af0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 >> 172.21.15.46:0/1057063122 conn(0x7f0e300741d0 msgr2=0x7f0e300d6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 shutdown_connections 2024-05-09T03:09:34.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.544+0000 7f0e36338640 1 -- 172.21.15.46:0/1057063122 wait complete. 2024-05-09T03:09:34.556 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:34.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.670+0000 7fa3f0a61640 1 Processor -- start 2024-05-09T03:09:34.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.670+0000 7fa3f0a61640 1 -- start start 2024-05-09T03:09:34.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3f0a61640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3ec161ce0 0x7fa3ec1620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3f0a61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec15b7e0 0x7fa3ec15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3f0a61640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3ec162630 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3f0a61640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa3ec15eab0 con 0x7fa3ec161ce0 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3f0a61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ec15c130 con 0x7fa3ec15b7e0 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3e9d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3ec161ce0 0x7fa3ec1620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3e9d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3ec161ce0 0x7fa3ec1620f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33312/0 (socket says 172.21.15.46:33312) 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1653819185 learned_addr learned my addr 172.21.15.46:0/1653819185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.671+0000 7fa3ead9d640 1 --2- 172.21.15.46:0/1653819185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec15b7e0 0x7fa3ec15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.672+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1653819185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec15b7e0 msgr2=0x7fa3ec15bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.672+0000 7fa3e9d9b640 1 --2- 172.21.15.46:0/1653819185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec15b7e0 0x7fa3ec15bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.672+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1653819185 >> v1:172.21.15.88:6789/0 conn(0x7fa3ec15e6a0 legacy=0x7fa3ec070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.672+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1653819185 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec070a60 con 0x7fa3ec161ce0 2024-05-09T03:09:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.672+0000 7fa3e9d9b640 1 --2- 172.21.15.46:0/1653819185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec161ce0 0x7fa3ec1620f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009870 tx=0x7fa3d402a1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a8fb4c8f7499c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.672+0000 7fa3e959a640 1 -- 172.21.15.46:0/1653819185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4037070 con 0x7fa3ec161ce0 2024-05-09T03:09:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3e959a640 1 -- 172.21.15.46:0/1653819185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3d4032440 con 0x7fa3ec161ce0 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3e959a640 1 -- 172.21.15.46:0/1653819185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4032810 con 0x7fa3ec161ce0 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1653819185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec161ce0 msgr2=0x7fa3ec1620f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1653819185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec161ce0 0x7fa3ec1620f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4009870 tx=0x7fa3d402a1b0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1653819185 shutdown_connections 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1653819185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec15b7e0 0x7fa3ec15bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1653819185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec161ce0 0x7fa3ec1620f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1653819185 >> 172.21.15.46:0/1653819185 conn(0x7fa3ec0d0630 msgr2=0x7fa3ec0d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1653819185 shutdown_connections 2024-05-09T03:09:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.673+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1653819185 wait complete. 2024-05-09T03:09:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.675+0000 7fa3f0a61640 1 Processor -- start 2024-05-09T03:09:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.675+0000 7fa3f0a61640 1 -- start start 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3ec15b7e0 0x7fa3ec06fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 0x7fa3ec152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec161ce0 0x7fa3ec0d2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec0d1900 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa3ec071020 con 0x7fa3ec15b7e0 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa3ec0b3160 con 0x7fa3ec161ce0 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 0x7fa3ec152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 0x7fa3ec152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33320/0 (socket says 172.21.15.46:33320) 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1337971702 learned_addr learned my addr 172.21.15.46:0/1337971702 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3ead9d640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec161ce0 0x7fa3ec0d2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3ea59c640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3ec15b7e0 0x7fa3ec06fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec161ce0 msgr2=0x7fa3ec0d2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec161ce0 0x7fa3ec0d2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3ec15b7e0 msgr2=0x7fa3ec06fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3ec15b7e0 0x7fa3ec06fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 -- 172.21.15.46:0/1337971702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3ec06f010 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3ead9d640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec161ce0 0x7fa3ec0d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e9d9b640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 0x7fa3ec152130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa3d402a1f0 tx=0x7fa3d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4037040 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3d4042070 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3ec0cacd0 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4030880 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.677+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3ec0cb290 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.678+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3ec06ed80 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.679+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa3d4030a20 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.680+0000 7fa3e2ffd640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3cc076300 0x7fa3cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.680+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fa3d40b7ec0 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.680+0000 7fa3ea59c640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3cc076300 0x7fa3cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.681+0000 7fa3ea59c640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3cc076300 0x7fa3cc078810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40015e0 tx=0x7fa3e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.685+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa3d4081820 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.875+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3ec06a270 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.876+0000 7fa3e2ffd640 1 -- 172.21.15.46:0/1337971702 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16546 (secure 0 0 0) 0x7fa3d4081120 con 0x7fa3ec15e6a0 2024-05-09T03:09:34.877 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:34.877 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":19,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:33.568527+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:34.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3cc076300 msgr2=0x7fa3cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3cc076300 0x7fa3cc078810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40015e0 tx=0x7fa3e4009040 comp rx=0 tx=0).stop 2024-05-09T03:09:34.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 msgr2=0x7fa3ec152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:34.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 0x7fa3ec152130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa3d402a1f0 tx=0x7fa3d40046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 shutdown_connections 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3ec161ce0 0x7fa3ec0d2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3cc076300 0x7fa3cc078810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3ec15e6a0 0x7fa3ec152130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 --2- 172.21.15.46:0/1337971702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3ec15b7e0 0x7fa3ec06fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 >> 172.21.15.46:0/1337971702 conn(0x7fa3ec0d0630 msgr2=0x7fa3ec0b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 shutdown_connections 2024-05-09T03:09:34.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:34.878+0000 7fa3f0a61640 1 -- 172.21.15.46:0/1337971702 wait complete. 2024-05-09T03:09:34.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:09:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.007+0000 7f4000c69640 1 Processor -- start 2024-05-09T03:09:35.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.007+0000 7f4000c69640 1 -- start start 2024-05-09T03:09:35.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f4000c69640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ffc0cc350 0x7f3ffc06c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f4000c69640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ffc068c70 con 0x7f3ffc164620 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f4000c69640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ffc0b7a40 con 0x7f3ffc0cbc20 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f4000c69640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3ffc0b77f0 con 0x7f3ffc0cc350 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f3ff1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ffc0cc350 0x7f3ffc06c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f3ffad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ffc164620 0x7f3ffc0cd220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38042/0 (socket says 172.21.15.46:38042) 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.008+0000 7f3ffad9d640 1 -- 172.21.15.46:0/1273402396 learned_addr learned my addr 172.21.15.46:0/1273402396 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.009+0000 7f3ff1d9b640 1 -- 172.21.15.46:0/1273402396 >> v1:172.21.15.88:6789/0 conn(0x7f3ffc0cbc20 legacy=0x7f3ffc163f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.009+0000 7f3ff1d9b640 1 -- 172.21.15.46:0/1273402396 >> v1:172.21.15.46:6789/0 conn(0x7f3ffc164620 legacy=0x7f3ffc0cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.009+0000 7f3ff9d9b640 1 -- 172.21.15.46:0/1273402396 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998184479 0 0) 0x7f3ffc068c70 con 0x7f3ffc164620 2024-05-09T03:09:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.009+0000 7f3ff1d9b640 1 -- 172.21.15.46:0/1273402396 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc163ba0 con 0x7f3ffc0cc350 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f3ff1d9b640 1 --2- 172.21.15.46:0/1273402396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc06c680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40097a0 tx=0x7f3fe402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deca27025869033d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f3ff9d9b640 1 -- 172.21.15.46:0/1273402396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4037070 con 0x7f3ffc0cc350 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f3ff9d9b640 1 -- 172.21.15.46:0/1273402396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fe4032440 con 0x7f3ffc0cc350 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f3ff9d9b640 1 -- 172.21.15.46:0/1273402396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4032810 con 0x7f3ffc0cc350 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 -- 172.21.15.46:0/1273402396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc0cc350 msgr2=0x7f3ffc06c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 --2- 172.21.15.46:0/1273402396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc06c680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40097a0 tx=0x7f3fe402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 -- 172.21.15.46:0/1273402396 shutdown_connections 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 --2- 172.21.15.46:0/1273402396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc06c680 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 -- 172.21.15.46:0/1273402396 >> 172.21.15.46:0/1273402396 conn(0x7f3ffc09b7d0 msgr2=0x7f3ffc0b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 -- 172.21.15.46:0/1273402396 shutdown_connections 2024-05-09T03:09:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.010+0000 7f4000c69640 1 -- 172.21.15.46:0/1273402396 wait complete. 2024-05-09T03:09:35.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.012+0000 7f4000c69640 1 Processor -- start 2024-05-09T03:09:35.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.013+0000 7f4000c69640 1 -- start start 2024-05-09T03:09:35.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f4000c69640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ffc0cbc20 0x7f3ffc157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:35.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f4000c69640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc0d1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f4000c69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 0x7f3ffc15cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f4000c69640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc0cd930 con 0x7f3ffc164620 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f4000c69640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc164c80 con 0x7f3ffc0cc350 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f4000c69640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3ffc0cdab0 con 0x7f3ffc0cbc20 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 0x7f3ffc15cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 0x7f3ffc15cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33352/0 (socket says 172.21.15.46:33352) 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 -- 172.21.15.46:0/2437464345 learned_addr learned my addr 172.21.15.46:0/2437464345 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ff1d9b640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc0d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffa59c640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ffc0cbc20 0x7f3ffc157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 -- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ffc0cbc20 msgr2=0x7f3ffc157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ffc0cbc20 0x7f3ffc157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 -- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cc350 msgr2=0x7f3ffc0d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc0d1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.014+0000 7f3ffad9d640 1 -- 172.21.15.46:0/2437464345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc1ea360 con 0x7f3ffc164620 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.015+0000 7f3ffad9d640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 0x7f3ffc15cf00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400c680 tx=0x7f3ff400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:35.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.015+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4018070 con 0x7f3ffc164620 2024-05-09T03:09:35.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.015+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff4004d60 con 0x7f3ffc164620 2024-05-09T03:09:35.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.015+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ffc1ea600 con 0x7f3ffc164620 2024-05-09T03:09:35.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.015+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff400a820 con 0x7f3ffc164620 2024-05-09T03:09:35.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.015+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ffc1eabc0 con 0x7f3ffc164620 2024-05-09T03:09:35.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.016+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ffc06a270 con 0x7f3ffc164620 2024-05-09T03:09:35.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.017+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3ff401b070 con 0x7f3ffc164620 2024-05-09T03:09:35.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.017+0000 7f3ff37fe640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fdc0762d0 0x7f3fdc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:35.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.017+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f3ff4096a90 con 0x7f3ffc164620 2024-05-09T03:09:35.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.017+0000 7f3ffa59c640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fdc0762d0 0x7f3fdc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:35.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.018+0000 7f3ffa59c640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fdc0762d0 0x7f3fdc0787e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3fec000f60 tx=0x7f3fec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:35.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.022+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3ffc1ea360 con 0x7f3ffc164620 2024-05-09T03:09:35.155 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:09:35.154+0000 7fa389a6f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:09:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:35.253+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3ffc0b7df0 con 0x7f3ffc164620 2024-05-09T03:09:35.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:09:35.638 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:09:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.190+0000 7f3ff37fe640 1 -- 172.21.15.46:0/2437464345 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 118+0+0 (secure 0 0 0) 0x7f3ff405fd10 con 0x7f3ffc164620 2024-05-09T03:09:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fdc0762d0 msgr2=0x7f3fdc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fdc0762d0 0x7f3fdc0787e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3fec000f60 tx=0x7f3fec009040 comp rx=0 tx=0).stop 2024-05-09T03:09:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 msgr2=0x7f3ffc15cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 0x7f3ffc15cf00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400c680 tx=0x7f3ff400cb40 comp rx=0 tx=0).stop 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 shutdown_connections 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fdc0762d0 0x7f3fdc0787e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ffc164620 0x7f3ffc15cf00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cc350 0x7f3ffc0d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 --2- 172.21.15.46:0/2437464345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ffc0cbc20 0x7f3ffc157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 >> 172.21.15.46:0/2437464345 conn(0x7f3ffc09b7d0 msgr2=0x7f3ffc06fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 shutdown_connections 2024-05-09T03:09:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.193+0000 7f4000c69640 1 -- 172.21.15.46:0/2437464345 wait complete. 2024-05-09T03:09:36.208 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:09:36.208 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:09:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.331+0000 7f2dcea54640 1 Processor -- start 2024-05-09T03:09:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.332+0000 7f2dcea54640 1 -- start start 2024-05-09T03:09:36.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dcea54640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2dc80739e0 0x7f2dc806f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:36.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dcea54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dc80d0450 con 0x7f2dc80b52a0 2024-05-09T03:09:36.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dcea54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dc80b56b0 con 0x7f2dc8071100 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dcea54640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2dc80cf8b0 con 0x7f2dc80739e0 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dc77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2dc80b52a0 0x7f2dc80732d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45534/0 (socket says 172.21.15.46:45534) 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dc77fe640 1 -- 172.21.15.46:0/373914694 learned_addr learned my addr 172.21.15.46:0/373914694 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dccff1640 1 --2- 172.21.15.46:0/373914694 >> v2:172.21.15.88:3300/0 conn(0x7f2dc80739e0 0x7f2dc806f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dc6ffd640 1 -- 172.21.15.46:0/373914694 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108093145 0 0) 0x7f2dc80d0450 con 0x7f2dc80b52a0 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.333+0000 7f2dc6ffd640 1 -- 172.21.15.46:0/373914694 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2db4003520 con 0x7f2dc80b52a0 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.334+0000 7f2dccff1640 1 -- 172.21.15.46:0/373914694 >> v1:172.21.15.88:6789/0 conn(0x7f2dc8071100 legacy=0x7f2dc80cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.334+0000 7f2dccff1640 1 -- 172.21.15.46:0/373914694 >> v1:172.21.15.46:6789/0 conn(0x7f2dc80b52a0 legacy=0x7f2dc80732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.334+0000 7f2dccff1640 1 -- 172.21.15.46:0/373914694 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc80b5930 con 0x7f2dc80739e0 2024-05-09T03:09:36.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.334+0000 7f2dccff1640 1 --2- 172.21.15.46:0/373914694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80739e0 0x7f2dc806f9a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009d90 tx=0x7f2dc002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f54fd25e41bcf895 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:36.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dc6ffd640 1 -- 172.21.15.46:0/373914694 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0007740 con 0x7f2dc80739e0 2024-05-09T03:09:36.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dc6ffd640 1 -- 172.21.15.46:0/373914694 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc00078e0 con 0x7f2dc80739e0 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dc6ffd640 1 -- 172.21.15.46:0/373914694 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0007c90 con 0x7f2dc80739e0 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 -- 172.21.15.46:0/373914694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80739e0 msgr2=0x7f2dc806f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 --2- 172.21.15.46:0/373914694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80739e0 0x7f2dc806f9a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009d90 tx=0x7f2dc002a6f0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 -- 172.21.15.46:0/373914694 shutdown_connections 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 --2- 172.21.15.46:0/373914694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80739e0 0x7f2dc806f9a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 -- 172.21.15.46:0/373914694 >> 172.21.15.46:0/373914694 conn(0x7f2dc80b3620 msgr2=0x7f2dc80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 -- 172.21.15.46:0/373914694 shutdown_connections 2024-05-09T03:09:36.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.335+0000 7f2dcea54640 1 -- 172.21.15.46:0/373914694 wait complete. 2024-05-09T03:09:36.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.337+0000 7f2dcea54640 1 Processor -- start 2024-05-09T03:09:36.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.337+0000 7f2dcea54640 1 -- start start 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dcea54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dc8071100 0x7f2dc8153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dcea54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 0x7f2dc814a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dcea54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80afde0 0x7f2dc814ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dcea54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc80d1540 con 0x7f2dc80b52a0 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dcea54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc80cf540 con 0x7f2dc80afde0 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dcea54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc80b5ef0 con 0x7f2dc8071100 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 0x7f2dc814a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 0x7f2dc814a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55688/0 (socket says 172.21.15.46:55688) 2024-05-09T03:09:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 -- 172.21.15.46:0/2839223809 learned_addr learned my addr 172.21.15.46:0/2839223809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc7fff640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dc8071100 0x7f2dc8153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dccff1640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80afde0 0x7f2dc814ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 -- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dc8071100 msgr2=0x7f2dc8153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dc8071100 0x7f2dc8153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 -- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80afde0 msgr2=0x7f2dc814ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80afde0 0x7f2dc814ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 -- 172.21.15.46:0/2839223809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc80aed40 con 0x7f2dc80b52a0 2024-05-09T03:09:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dccff1640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80afde0 0x7f2dc814ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.339+0000 7f2dc77fe640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 0x7f2dc814a8b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2db000c440 tx=0x7f2db000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.340+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0017070 con 0x7f2dc80b52a0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.340+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2db0005ce0 con 0x7f2dc80b52a0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.340+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc80af020 con 0x7f2dc80b52a0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.340+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0007270 con 0x7f2dc80b52a0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.340+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc80af5e0 con 0x7f2dc80b52a0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.341+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc80aeab0 con 0x7f2dc80b52a0 2024-05-09T03:09:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.341+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2db001a070 con 0x7f2dc80b52a0 2024-05-09T03:09:36.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.342+0000 7f2dc4ff9640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2da8076300 0x7f2da8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:36.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.342+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2db00967f0 con 0x7f2dc80b52a0 2024-05-09T03:09:36.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.342+0000 7f2dc7fff640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2da8076300 0x7f2da8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:36.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.343+0000 7f2dc7fff640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2da8076300 0x7f2da8078810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2db8000f60 tx=0x7f2db8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:36.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.346+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2db005fa70 con 0x7f2dc80b52a0 2024-05-09T03:09:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:36.575+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f2dc806a270 con 0x7f2dc80b52a0 2024-05-09T03:09:37.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.199+0000 7f2dc4ff9640 1 -- 172.21.15.46:0/2839223809 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7f2db0063d70 con 0x7f2dc80b52a0 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2da8076300 msgr2=0x7f2da8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2da8076300 0x7f2da8078810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2db8000f60 tx=0x7f2db8009040 comp rx=0 tx=0).stop 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 msgr2=0x7f2dc814a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 0x7f2dc814a8b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2db000c440 tx=0x7f2db000c900 comp rx=0 tx=0).stop 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 shutdown_connections 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dc80afde0 0x7f2dc814ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2da8076300 0x7f2da8078810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dc80b52a0 0x7f2dc814a8b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 --2- 172.21.15.46:0/2839223809 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dc8071100 0x7f2dc8153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 >> 172.21.15.46:0/2839223809 conn(0x7f2dc80b3620 msgr2=0x7f2dc80b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:37.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 shutdown_connections 2024-05-09T03:09:37.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.202+0000 7f2dcea54640 1 -- 172.21.15.46:0/2839223809 wait complete. 2024-05-09T03:09:37.217 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:09:37.217 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:09:37.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.338+0000 7fa6523d6640 1 Processor -- start 2024-05-09T03:09:37.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.338+0000 7fa6523d6640 1 -- start start 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa6523d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 0x7fa64c1606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa6523d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa64c06cd90 0x7fa64c09c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa6523d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa64c072100 con 0x7fa64c0b1e70 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa6523d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa64c0b8770 con 0x7fa64c06cd90 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa6523d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa64c09c780 con 0x7fa64c0b8360 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa64b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa64c0b1e70 0x7fa64c06c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45538/0 (socket says 172.21.15.46:45538) 2024-05-09T03:09:37.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa64b7fe640 1 -- 172.21.15.46:0/3569374180 learned_addr learned my addr 172.21.15.46:0/3569374180 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa64bfff640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 0x7fa64c1606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa650973640 1 --2- 172.21.15.46:0/3569374180 >> v2:172.21.15.88:3300/0 conn(0x7fa64c06cd90 0x7fa64c09c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.339+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014739357 0 0) 0x7fa64c072100 con 0x7fa64c0b1e70 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa638003520 con 0x7fa64c0b1e70 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2497060972 0 0) 0x7fa638003520 con 0x7fa64c0b1e70 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 msgr2=0x7fa64c1606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 0x7fa64c1606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 >> v2:172.21.15.88:3300/0 conn(0x7fa64c06cd90 msgr2=0x7fa64c09c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 --2- 172.21.15.46:0/3569374180 >> v2:172.21.15.88:3300/0 conn(0x7fa64c06cd90 0x7fa64c09c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c0723e0 con 0x7fa64c0b1e70 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fa6340016f0 con 0x7fa64c0b1e70 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 >> v1:172.21.15.46:6789/0 conn(0x7fa64c0b1e70 legacy=0x7fa64c06c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 0x7fa638003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa638003fd0 0x7fa638004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa638005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c0b8770 con 0x7fa638004d40 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c09c780 con 0x7fa638003fd0 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa63800bb50 con 0x7fa64c0b8360 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64bfff640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 0x7fa638003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa650973640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa638005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa64b7fe640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa638003fd0 0x7fa638004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa650973640 1 -- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 msgr2=0x7fa638003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa650973640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c0b8360 0x7fa638003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa650973640 1 -- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa638003fd0 msgr2=0x7fa638004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa650973640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa638003fd0 0x7fa638004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.340+0000 7fa650973640 1 -- 172.21.15.46:0/3569374180 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa638003520 con 0x7fa638004d40 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.341+0000 7fa650973640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa638005150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0d06d0 tx=0x7fa638005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c61cb6a3f969fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:37.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.341+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644009c10 con 0x7fa638004d40 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.341+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa644036070 con 0x7fa638004d40 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.341+0000 7fa64affd640 1 -- 172.21.15.46:0/3569374180 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644031850 con 0x7fa638004d40 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa6523d6640 1 -- 172.21.15.46:0/3569374180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 msgr2=0x7fa638005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3569374180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa638005150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0d06d0 tx=0x7fa638005830 comp rx=0 tx=0).stop 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa650973640 1 -- 172.21.15.46:0/3569374180 reap_dead start 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa6523d6640 1 -- 172.21.15.46:0/3569374180 shutdown_connections 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa6523d6640 1 -- 172.21.15.46:0/3569374180 >> 172.21.15.46:0/3569374180 conn(0x7fa64c0d2c70 msgr2=0x7fa64c0d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa6523d6640 1 -- 172.21.15.46:0/3569374180 shutdown_connections 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.342+0000 7fa6523d6640 1 -- 172.21.15.46:0/3569374180 wait complete. 2024-05-09T03:09:37.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.344+0000 7fa6523d6640 1 Processor -- start 2024-05-09T03:09:37.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.344+0000 7fa6523d6640 1 -- start start 2024-05-09T03:09:37.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa6523d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa64c14aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa6523d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa64c1627b0 0x7fa64c162bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa6523d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c15e350 0x7fa64c15e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa6523d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c0726c0 con 0x7fa638004d40 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa6523d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c09bd20 con 0x7fa64c1627b0 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa6523d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa64c0b2280 con 0x7fa64c15e350 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa64bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa64c14aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa64bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa64c14aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55714/0 (socket says 172.21.15.46:55714) 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa64bfff640 1 -- 172.21.15.46:0/3870456770 learned_addr learned my addr 172.21.15.46:0/3870456770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa64b7fe640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa64c1627b0 0x7fa64c162bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.345+0000 7fa650973640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c15e350 0x7fa64c15e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64bfff640 1 -- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c15e350 msgr2=0x7fa64c15e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64bfff640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c15e350 0x7fa64c15e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64bfff640 1 -- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa64c1627b0 msgr2=0x7fa64c162bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64bfff640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa64c1627b0 0x7fa64c162bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64bfff640 1 -- 172.21.15.46:0/3870456770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c15eca0 con 0x7fa638004d40 2024-05-09T03:09:37.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64b7fe640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa64c1627b0 0x7fa64c162bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa64bfff640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa64c14aed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa63c011960 tx=0x7fa63c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00faf0 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63c00fc90 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64c1f9e00 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c014440 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.346+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64c1f9040 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.347+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64c06a270 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.348+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa63c012040 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.348+0000 7fa648ff9640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa62c0762d0 0x7fa62c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.349+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa63c097bf0 con 0x7fa638004d40 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.349+0000 7fa64b7fe640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa62c0762d0 0x7fa62c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.349+0000 7fa64b7fe640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa62c0762d0 0x7fa62c0787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0b17b0 tx=0x7fa634009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:37.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.353+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa64c15eca0 con 0x7fa638004d40 2024-05-09T03:09:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.584+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa64c1f99e0 con 0x7fa638004d40 2024-05-09T03:09:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.584+0000 7fa648ff9640 1 -- 172.21.15.46:0/3870456770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7fa63c019030 con 0x7fa638004d40 2024-05-09T03:09:37.585 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:37.585 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:09:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:09:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa62c0762d0 msgr2=0x7fa62c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa62c0762d0 0x7fa62c0787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0b17b0 tx=0x7fa634009040 comp rx=0 tx=0).stop 2024-05-09T03:09:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 msgr2=0x7fa64c14aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa64c14aed0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa63c011960 tx=0x7fa63c011e20 comp rx=0 tx=0).stop 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 shutdown_connections 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa64c15e350 0x7fa64c15e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa64c1627b0 0x7fa64c162bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa62c0762d0 0x7fa62c0787e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 --2- 172.21.15.46:0/3870456770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa638004d40 0x7fa64c14aed0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 >> 172.21.15.46:0/3870456770 conn(0x7fa64c0d2c70 msgr2=0x7fa64c0b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 shutdown_connections 2024-05-09T03:09:37.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.586+0000 7fa6523d6640 1 -- 172.21.15.46:0/3870456770 wait complete. 2024-05-09T03:09:37.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.713+0000 7f8bbfffa640 1 Processor -- start 2024-05-09T03:09:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.713+0000 7f8bbfffa640 1 -- start start 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbfffa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80d3170 0x7f8bb815e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbfffa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb80b5410 con 0x7f8bb815ee20 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbfffa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb815f630 con 0x7f8bb80b5000 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbfffa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb80cf500 con 0x7f8bb80d3170 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbe597640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8bb815ee20 0x7f8bb80ce460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45540/0 (socket says 172.21.15.46:45540) 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbe597640 1 -- 172.21.15.46:0/2713337903 learned_addr learned my addr 172.21.15.46:0/2713337903 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.714+0000 7f8bbd595640 1 --2- 172.21.15.46:0/2713337903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80d3170 0x7f8bb815e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831398879 0 0) 0x7f8ba8001490 con 0x7f8bb815ee20 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba0003520 con 0x7f8bb815ee20 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744349292 0 0) 0x7f8bb815f630 con 0x7f8bb80b5000 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba8001490 con 0x7f8bb80b5000 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbd595640 1 -- 172.21.15.46:0/2713337903 >> v1:172.21.15.88:6789/0 conn(0x7f8bb80b5000 legacy=0x7f8bb80d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbd595640 1 -- 172.21.15.46:0/2713337903 >> v1:172.21.15.46:6789/0 conn(0x7f8bb815ee20 legacy=0x7f8bb80ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.715+0000 7f8bbd595640 1 -- 172.21.15.46:0/2713337903 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb80cf7e0 con 0x7f8bb80d3170 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.716+0000 7f8bbd595640 1 --2- 172.21.15.46:0/2713337903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80d3170 0x7f8bb815e8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0097d0 tx=0x7f8bac02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1560f9decde5f0c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.716+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac037070 con 0x7f8bb80d3170 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.716+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bac032440 con 0x7f8bb80d3170 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.716+0000 7f8bbcd94640 1 -- 172.21.15.46:0/2713337903 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac032810 con 0x7f8bb80d3170 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.716+0000 7f8bbfffa640 1 -- 172.21.15.46:0/2713337903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80d3170 msgr2=0x7f8bb815e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.716+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/2713337903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80d3170 0x7f8bb815e8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0097d0 tx=0x7f8bac02a210 comp rx=0 tx=0).stop 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.717+0000 7f8bbfffa640 1 -- 172.21.15.46:0/2713337903 shutdown_connections 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.717+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/2713337903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80d3170 0x7f8bb815e8e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.717+0000 7f8bbfffa640 1 -- 172.21.15.46:0/2713337903 >> 172.21.15.46:0/2713337903 conn(0x7f8bb809b7d0 msgr2=0x7f8bb809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.717+0000 7f8bbfffa640 1 -- 172.21.15.46:0/2713337903 shutdown_connections 2024-05-09T03:09:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.717+0000 7f8bbfffa640 1 -- 172.21.15.46:0/2713337903 wait complete. 2024-05-09T03:09:37.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.718+0000 7f8bbfffa640 1 Processor -- start 2024-05-09T03:09:37.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.718+0000 7f8bbfffa640 1 -- start start 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbfffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bb80b5000 0x7f8bb806e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbfffa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 0x7f8bb80c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbfffa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80ca230 0x7f8bb81499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbfffa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb815f230 con 0x7f8bb815ee20 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbfffa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb80cfda0 con 0x7f8bb80b5000 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbfffa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8bb815f3b0 con 0x7f8bb80ca230 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 0x7f8bb80c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 0x7f8bb80c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55726/0 (socket says 172.21.15.46:55726) 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 -- 172.21.15.46:0/1807829704 learned_addr learned my addr 172.21.15.46:0/1807829704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbe597640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80ca230 0x7f8bb81499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbdd96640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bb80b5000 0x7f8bb806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 -- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80ca230 msgr2=0x7f8bb81499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80ca230 0x7f8bb81499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 -- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bb80b5000 msgr2=0x7f8bb806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bb80b5000 0x7f8bb806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 -- 172.21.15.46:0/1807829704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb806eaa0 con 0x7f8bb815ee20 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.720+0000 7f8bbd595640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 0x7f8bb80c9cf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8bac035040 tx=0x7f8bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.721+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac041070 con 0x7f8bb815ee20 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.721+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bac004220 con 0x7f8bb815ee20 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.721+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb81603e0 con 0x7f8bb815ee20 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.721+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bac037040 con 0x7f8bb815ee20 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.721+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb81f42a0 con 0x7f8bb815ee20 2024-05-09T03:09:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.721+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bb806a270 con 0x7f8bb815ee20 2024-05-09T03:09:37.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.722+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8bac003610 con 0x7f8bb815ee20 2024-05-09T03:09:37.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.723+0000 7f8bb67fc640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ba40762d0 0x7f8ba40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:37.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.723+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8bac0b7b20 con 0x7f8bb815ee20 2024-05-09T03:09:37.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.723+0000 7f8bbdd96640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ba40762d0 0x7f8ba40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:37.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.723+0000 7f8bbdd96640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ba40762d0 0x7f8ba40787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0001120 tx=0x7f8bb0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.727+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8bac04b020 con 0x7f8bb815ee20 2024-05-09T03:09:37.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.923+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8bb806e830 con 0x7f8bb815ee20 2024-05-09T03:09:37.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.925+0000 7f8bb67fc640 1 -- 172.21.15.46:0/1807829704 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7f8bac0a0e20 con 0x7f8bb815ee20 2024-05-09T03:09:37.926 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:37.926 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:37.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ba40762d0 msgr2=0x7f8ba40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ba40762d0 0x7f8ba40787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0001120 tx=0x7f8bb0009040 comp rx=0 tx=0).stop 2024-05-09T03:09:37.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 msgr2=0x7f8bb80c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:37.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 0x7f8bb80c9cf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8bac035040 tx=0x7f8bac0046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 shutdown_connections 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8bb80ca230 0x7f8bb81499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ba40762d0 0x7f8ba40787e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bb815ee20 0x7f8bb80c9cf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 --2- 172.21.15.46:0/1807829704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bb80b5000 0x7f8bb806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 >> 172.21.15.46:0/1807829704 conn(0x7f8bb809b7d0 msgr2=0x7f8bb815c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 shutdown_connections 2024-05-09T03:09:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:37.927+0000 7f8bbfffa640 1 -- 172.21.15.46:0/1807829704 wait complete. 2024-05-09T03:09:37.939 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:09:37.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:09:38.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.052+0000 7f517fa8b640 1 Processor -- start 2024-05-09T03:09:38.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.052+0000 7f517fa8b640 1 -- start start 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517fa8b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51780d57a0 0x7f5178160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517fa8b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51780b7ef0 0x7f51780b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517fa8b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780d49e0 con 0x7f51780b4af0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517fa8b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f51780d5bb0 con 0x7f51780b7ef0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517fa8b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f51780d4cc0 con 0x7f51780d57a0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517e028640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f51780b7ef0 0x7f51780b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517d026640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f51780b4af0 0x7f51780b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45556/0 (socket says 172.21.15.46:45556) 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517d026640 1 -- 172.21.15.46:0/2312531999 learned_addr learned my addr 172.21.15.46:0/2312531999 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517d827640 1 --2- 172.21.15.46:0/2312531999 >> v2:172.21.15.88:3300/0 conn(0x7f51780d57a0 0x7f5178160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517c825640 1 -- 172.21.15.46:0/2312531999 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404246474 0 0) 0x7f51780d49e0 con 0x7f51780b4af0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517c825640 1 -- 172.21.15.46:0/2312531999 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5160003520 con 0x7f51780b4af0 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517e028640 1 -- 172.21.15.46:0/2312531999 >> v2:172.21.15.88:3300/0 conn(0x7f51780d57a0 msgr2=0x7f5178160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:38.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517e028640 1 --2- 172.21.15.46:0/2312531999 >> v2:172.21.15.88:3300/0 conn(0x7f51780d57a0 0x7f5178160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517e028640 1 -- 172.21.15.46:0/2312531999 >> v1:172.21.15.46:6789/0 conn(0x7f51780b4af0 legacy=0x7f51780b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.054+0000 7f517e028640 1 -- 172.21.15.46:0/2312531999 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51780d4fa0 con 0x7f51780b7ef0 2024-05-09T03:09:38.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.055+0000 7f517e028640 1 --2- 172.21.15.46:0/2312531999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780b7ef0 0x7f51780b4f00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f517808ed20 tx=0x7f516802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d866f89b414d73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.055+0000 7f517c825640 1 -- 172.21.15.46:0/2312531999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168007740 con 0x7f51780b7ef0 2024-05-09T03:09:38.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.055+0000 7f517c825640 1 -- 172.21.15.46:0/2312531999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51680078e0 con 0x7f51780b7ef0 2024-05-09T03:09:38.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.055+0000 7f517c825640 1 -- 172.21.15.46:0/2312531999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168007cb0 con 0x7f51780b7ef0 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 -- 172.21.15.46:0/2312531999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780b7ef0 msgr2=0x7f51780b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 --2- 172.21.15.46:0/2312531999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780b7ef0 0x7f51780b4f00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f517808ed20 tx=0x7f516802a680 comp rx=0 tx=0).stop 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 -- 172.21.15.46:0/2312531999 shutdown_connections 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 --2- 172.21.15.46:0/2312531999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780b7ef0 0x7f51780b4f00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 --2- 172.21.15.46:0/2312531999 >> v2:172.21.15.88:3300/0 conn(0x7f51780d57a0 0x7f5178160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 -- 172.21.15.46:0/2312531999 >> 172.21.15.46:0/2312531999 conn(0x7f51780743e0 msgr2=0x7f51780d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 -- 172.21.15.46:0/2312531999 shutdown_connections 2024-05-09T03:09:38.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.056+0000 7f517fa8b640 1 -- 172.21.15.46:0/2312531999 wait complete. 2024-05-09T03:09:38.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.058+0000 7f517fa8b640 1 Processor -- start 2024-05-09T03:09:38.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.058+0000 7f517fa8b640 1 -- start start 2024-05-09T03:09:38.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517fa8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51780b4af0 0x7f5178156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517fa8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 0x7f51780d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517fa8b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5178158940 0x7f5178158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517fa8b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f51780d1920 con 0x7f51780d57a0 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517fa8b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f51780d15b0 con 0x7f5178158940 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517fa8b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f517815c280 con 0x7f51780b4af0 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517d026640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 0x7f51780d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517d026640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 0x7f51780d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55744/0 (socket says 172.21.15.46:55744) 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517d026640 1 -- 172.21.15.46:0/1935917863 learned_addr learned my addr 172.21.15.46:0/1935917863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.059+0000 7f517d827640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51780b4af0 0x7f5178156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517e028640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5178158940 0x7f5178158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d026640 1 -- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51780b4af0 msgr2=0x7f5178156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d026640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51780b4af0 0x7f5178156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d026640 1 -- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5178158940 msgr2=0x7f5178158d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d026640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5178158940 0x7f5178158d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d026640 1 -- 172.21.15.46:0/1935917863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5178159a50 con 0x7f51780d57a0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517e028640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5178158940 0x7f5178158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d827640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51780b4af0 0x7f5178156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517d026640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 0x7f51780d08f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c410 tx=0x7f516c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c017070 con 0x7f51780d57a0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f516c005ce0 con 0x7f51780d57a0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5178159d30 con 0x7f51780d57a0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516c0072d0 con 0x7f51780d57a0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.060+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f517815aa20 con 0x7f51780d57a0 2024-05-09T03:09:38.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.061+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51781597c0 con 0x7f51780d57a0 2024-05-09T03:09:38.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.062+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f516c01a070 con 0x7f51780d57a0 2024-05-09T03:09:38.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.062+0000 7f51767fc640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51640762d0 0x7f51640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.062+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f516c096e90 con 0x7f51780d57a0 2024-05-09T03:09:38.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.063+0000 7f517d827640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51640762d0 0x7f51640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.063+0000 7f517d827640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51640762d0 0x7f51640787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f517806d3b0 tx=0x7f5170008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.066+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f516c0600f0 con 0x7f51780d57a0 2024-05-09T03:09:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.259+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f517815a250 con 0x7f51640762d0 2024-05-09T03:09:38.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.265+0000 7f51767fc640 1 -- 172.21.15.46:0/1935917863 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237066 (secure 0 0 0) 0x7f517815a250 con 0x7f51640762d0 2024-05-09T03:09:38.269 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:38.271 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:09:38.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51640762d0 msgr2=0x7f51640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51640762d0 0x7f51640787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f517806d3b0 tx=0x7f5170008040 comp rx=0 tx=0).stop 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 msgr2=0x7f51780d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 0x7f51780d08f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f516c00c410 tx=0x7f516c00c8d0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 shutdown_connections 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5178158940 0x7f5178158d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51640762d0 0x7f51640787e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51780d57a0 0x7f51780d08f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 --2- 172.21.15.46:0/1935917863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f51780b4af0 0x7f5178156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 >> 172.21.15.46:0/1935917863 conn(0x7f51780743e0 msgr2=0x7f51781e5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 shutdown_connections 2024-05-09T03:09:38.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.269+0000 7f517fa8b640 1 -- 172.21.15.46:0/1935917863 wait complete. 2024-05-09T03:09:38.281 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-05-09T03:09:36.491162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":235700,"kb_used_data":2376,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739020,"statfs":{"total":773094113280,"available":772852756480,"internally_reserved":0,"allocated":2433024,"data_stored":1546872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.327025"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":102,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-09T03:09:25.312488+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:09:25.312488+0000","last_peered":"2024-05-09T03:09:25.312488+0000","last_clean":"2024-05-09T03:09:25.312488+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:09:25.312488+0000","last_undegraded":"2024-05-09T03:09:25.312488+0000","last_fullsized":"2024-05-09T03:09:25.312488+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:33.163887+0000","last_change":"2024-05-09T03:09:33.163887+0000","last_active":"2024-05-09T03:09:33.163887+0000","last_peered":"2024-05-09T03:09:33.163887+0000","last_clean":"2024-05-09T03:09:33.163887+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:33.163887+0000","last_undegraded":"2024-05-09T03:09:33.163887+0000","last_fullsized":"2024-05-09T03:09:33.163887+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:09:32.158675+0000","last_change":"2024-05-09T03:09:32.158675+0000","last_active":"2024-05-09T03:09:32.158675+0000","last_peered":"2024-05-09T03:09:32.158675+0000","last_clean":"2024-05-09T03:09:32.158675+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:09:32.158675+0000","last_undegraded":"2024-05-09T03:09:32.158675+0000","last_fullsized":"2024-05-09T03:09:32.158675+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31336,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31340,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340500,"statfs":{"total":96636764160,"available":96604672000,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31332,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":749568,"data_stored":636135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:09:38.285 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:09:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.435+0000 7fccf35b2640 1 Processor -- start 2024-05-09T03:09:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.435+0000 7fccf35b2640 1 -- start start 2024-05-09T03:09:38.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.436+0000 7fccf35b2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccec160610 0x7fccec0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.436+0000 7fccf35b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf35b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccec0b8130 con 0x7fccec0d57a0 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf35b2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fccec0d5bb0 con 0x7fccec160610 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf35b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fccec0d49e0 con 0x7fccec0b7d20 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf0b4d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccec160610 0x7fccec0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43168/0 (socket says 172.21.15.46:43168) 2024-05-09T03:09:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 -- 172.21.15.46:0/2005863472 learned_addr learned my addr 172.21.15.46:0/2005863472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 -- 172.21.15.46:0/2005863472 >> v2:172.21.15.88:3300/0 conn(0x7fccec160610 msgr2=0x7fccec0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 --2- 172.21.15.46:0/2005863472 >> v2:172.21.15.88:3300/0 conn(0x7fccec160610 0x7fccec0b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 -- 172.21.15.46:0/2005863472 >> v1:172.21.15.88:6789/0 conn(0x7fccec0d57a0 legacy=0x7fccec0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf1b4f640 1 -- 172.21.15.46:0/2005863472 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec0d4cc0 con 0x7fccec0b7d20 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.437+0000 7fccf0b4d640 1 --2- 172.21.15.46:0/2005863472 >> v2:172.21.15.88:3300/0 conn(0x7fccec160610 0x7fccec0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.438+0000 7fccf1b4f640 1 --2- 172.21.15.46:0/2005863472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0b9450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009870 tx=0x7fcce402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6087c99563a0d51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.438+0000 7fcce3fff640 1 -- 172.21.15.46:0/2005863472 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4037070 con 0x7fccec0b7d20 2024-05-09T03:09:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.438+0000 7fcce3fff640 1 -- 172.21.15.46:0/2005863472 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce4032440 con 0x7fccec0b7d20 2024-05-09T03:09:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.438+0000 7fcce3fff640 1 -- 172.21.15.46:0/2005863472 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4032810 con 0x7fccec0b7d20 2024-05-09T03:09:38.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 -- 172.21.15.46:0/2005863472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 msgr2=0x7fccec0b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 --2- 172.21.15.46:0/2005863472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0b9450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009870 tx=0x7fcce402a270 comp rx=0 tx=0).stop 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 -- 172.21.15.46:0/2005863472 shutdown_connections 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 --2- 172.21.15.46:0/2005863472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0b9450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 --2- 172.21.15.46:0/2005863472 >> v2:172.21.15.88:3300/0 conn(0x7fccec160610 0x7fccec0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 -- 172.21.15.46:0/2005863472 >> 172.21.15.46:0/2005863472 conn(0x7fccec0743e0 msgr2=0x7fccec0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 -- 172.21.15.46:0/2005863472 shutdown_connections 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.439+0000 7fccf35b2640 1 -- 172.21.15.46:0/2005863472 wait complete. 2024-05-09T03:09:38.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.441+0000 7fccf35b2640 1 Processor -- start 2024-05-09T03:09:38.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.441+0000 7fccf35b2640 1 -- start start 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf35b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0d0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf35b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccec0d57a0 0x7fccec1548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf35b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 0x7fccec1e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf35b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fccec0d15b0 con 0x7fccec160610 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf35b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fccec160a20 con 0x7fccec0d57a0 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf35b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fccec0d1920 con 0x7fccec0b7d20 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf1b4f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 0x7fccec1e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf1b4f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 0x7fccec1e9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55748/0 (socket says 172.21.15.46:55748) 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf1b4f640 1 -- 172.21.15.46:0/1515539723 learned_addr learned my addr 172.21.15.46:0/1515539723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:38.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.442+0000 7fccf0b4d640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccec0d57a0 0x7fccec1548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf134e640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf1b4f640 1 -- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 msgr2=0x7fccec0d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf1b4f640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf1b4f640 1 -- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccec0d57a0 msgr2=0x7fccec1548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf1b4f640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccec0d57a0 0x7fccec1548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf1b4f640 1 -- 172.21.15.46:0/1515539723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec1eaac0 con 0x7fccec160610 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf1b4f640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 0x7fccec1e9e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009870 tx=0x7fcce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4041020 con 0x7fccec160610 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce403d640 con 0x7fccec160610 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccec153b10 con 0x7fccec160610 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4037040 con 0x7fccec160610 2024-05-09T03:09:38.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.443+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccec1540d0 con 0x7fccec160610 2024-05-09T03:09:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.444+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccec06a270 con 0x7fccec160610 2024-05-09T03:09:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.445+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcce4044020 con 0x7fccec160610 2024-05-09T03:09:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.446+0000 7fcce1ffb640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccd00762d0 0x7fccd00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.446+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcce40b7c50 con 0x7fccec160610 2024-05-09T03:09:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.446+0000 7fccf134e640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccd00762d0 0x7fccd00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.447+0000 7fccf134e640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccd00762d0 0x7fccd00787e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccec06d3b0 tx=0x7fcce8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.451+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcce40bd020 con 0x7fccec160610 2024-05-09T03:09:38.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.641+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fccec0d36d0 con 0x7fccd00762d0 2024-05-09T03:09:38.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.647+0000 7fcce1ffb640 1 -- 172.21.15.46:0/1515539723 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251902 (secure 0 0 0) 0x7fccec0d36d0 con 0x7fccd00762d0 2024-05-09T03:09:38.651 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:38.652 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:09:38.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccd00762d0 msgr2=0x7fccd00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccd00762d0 0x7fccd00787e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fccec06d3b0 tx=0x7fcce8009040 comp rx=0 tx=0).stop 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 msgr2=0x7fccec1e9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 0x7fccec1e9e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009870 tx=0x7fcce40046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 shutdown_connections 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccd00762d0 0x7fccd00787e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec160610 0x7fccec1e9e40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccec0d57a0 0x7fccec1548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.650+0000 7fccf35b2640 1 --2- 172.21.15.46:0/1515539723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fccec0b7d20 0x7fccec0d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.651+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 >> 172.21.15.46:0/1515539723 conn(0x7fccec0743e0 msgr2=0x7fccec0bb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.651+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 shutdown_connections 2024-05-09T03:09:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.651+0000 7fccf35b2640 1 -- 172.21.15.46:0/1515539723 wait complete. 2024-05-09T03:09:38.663 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-05-09T03:09:38.491617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235592,"kb_used_data":2776,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739128,"statfs":{"total":773094113280,"available":772852867072,"internally_reserved":0,"allocated":2842624,"data_stored":1727828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001358"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197915+0000","last_change":"2024-05-09T03:09:34.191121+0000","last_active":"2024-05-09T03:09:34.197915+0000","last_peered":"2024-05-09T03:09:34.197915+0000","last_clean":"2024-05-09T03:09:34.197915+0000","last_became_active":"2024-05-09T03:09:34.189608+0000","last_became_peered":"2024-05-09T03:09:34.189608+0000","last_unstale":"2024-05-09T03:09:34.197915+0000","last_undegraded":"2024-05-09T03:09:34.197915+0000","last_fullsized":"2024-05-09T03:09:34.197915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191404+0000","last_change":"2024-05-09T03:09:34.200351+0000","last_active":"2024-05-09T03:09:35.191404+0000","last_peered":"2024-05-09T03:09:35.191404+0000","last_clean":"2024-05-09T03:09:35.191404+0000","last_became_active":"2024-05-09T03:09:34.200009+0000","last_became_peered":"2024-05-09T03:09:34.200009+0000","last_unstale":"2024-05-09T03:09:35.191404+0000","last_undegraded":"2024-05-09T03:09:35.191404+0000","last_fullsized":"2024-05-09T03:09:35.191404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.206972+0000","last_change":"2024-05-09T03:09:34.204156+0000","last_active":"2024-05-09T03:09:34.206972+0000","last_peered":"2024-05-09T03:09:34.206972+0000","last_clean":"2024-05-09T03:09:34.206972+0000","last_became_active":"2024-05-09T03:09:34.203843+0000","last_became_peered":"2024-05-09T03:09:34.203843+0000","last_unstale":"2024-05-09T03:09:34.206972+0000","last_undegraded":"2024-05-09T03:09:34.206972+0000","last_fullsized":"2024-05-09T03:09:34.206972+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:21.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223412+0000","last_change":"2024-05-09T03:09:34.206782+0000","last_active":"2024-05-09T03:09:35.223412+0000","last_peered":"2024-05-09T03:09:35.223412+0000","last_clean":"2024-05-09T03:09:35.223412+0000","last_became_active":"2024-05-09T03:09:34.206376+0000","last_became_peered":"2024-05-09T03:09:34.206376+0000","last_unstale":"2024-05-09T03:09:35.223412+0000","last_undegraded":"2024-05-09T03:09:35.223412+0000","last_fullsized":"2024-05-09T03:09:35.223412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:33.100263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.207163+0000","last_change":"2024-05-09T03:09:34.203941+0000","last_active":"2024-05-09T03:09:34.207163+0000","last_peered":"2024-05-09T03:09:34.207163+0000","last_clean":"2024-05-09T03:09:34.207163+0000","last_became_active":"2024-05-09T03:09:34.203530+0000","last_became_peered":"2024-05-09T03:09:34.203530+0000","last_unstale":"2024-05-09T03:09:34.207163+0000","last_undegraded":"2024-05-09T03:09:34.207163+0000","last_fullsized":"2024-05-09T03:09:34.207163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223482+0000","last_change":"2024-05-09T03:09:34.201477+0000","last_active":"2024-05-09T03:09:35.223482+0000","last_peered":"2024-05-09T03:09:35.223482+0000","last_clean":"2024-05-09T03:09:35.223482+0000","last_became_active":"2024-05-09T03:09:34.200622+0000","last_became_peered":"2024-05-09T03:09:34.200622+0000","last_unstale":"2024-05-09T03:09:35.223482+0000","last_undegraded":"2024-05-09T03:09:35.223482+0000","last_fullsized":"2024-05-09T03:09:35.223482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.196052+0000","last_change":"2024-05-09T03:09:34.190313+0000","last_active":"2024-05-09T03:09:34.196052+0000","last_peered":"2024-05-09T03:09:34.196052+0000","last_clean":"2024-05-09T03:09:34.196052+0000","last_became_active":"2024-05-09T03:09:34.189973+0000","last_became_peered":"2024-05-09T03:09:34.189973+0000","last_unstale":"2024-05-09T03:09:34.196052+0000","last_undegraded":"2024-05-09T03:09:34.196052+0000","last_fullsized":"2024-05-09T03:09:34.196052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:16.118667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.208052+0000","last_change":"2024-05-09T03:09:34.206587+0000","last_active":"2024-05-09T03:09:34.208052+0000","last_peered":"2024-05-09T03:09:34.208052+0000","last_clean":"2024-05-09T03:09:34.208052+0000","last_became_active":"2024-05-09T03:09:34.206317+0000","last_became_peered":"2024-05-09T03:09:34.206317+0000","last_unstale":"2024-05-09T03:09:34.208052+0000","last_undegraded":"2024-05-09T03:09:34.208052+0000","last_fullsized":"2024-05-09T03:09:34.208052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:20.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.207281+0000","last_change":"2024-05-09T03:09:34.202532+0000","last_active":"2024-05-09T03:09:34.207281+0000","last_peered":"2024-05-09T03:09:34.207281+0000","last_clean":"2024-05-09T03:09:34.207281+0000","last_became_active":"2024-05-09T03:09:34.202253+0000","last_became_peered":"2024-05-09T03:09:34.202253+0000","last_unstale":"2024-05-09T03:09:34.207281+0000","last_undegraded":"2024-05-09T03:09:34.207281+0000","last_fullsized":"2024-05-09T03:09:34.207281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195921+0000","last_change":"2024-05-09T03:09:34.199946+0000","last_active":"2024-05-09T03:09:35.195921+0000","last_peered":"2024-05-09T03:09:35.195921+0000","last_clean":"2024-05-09T03:09:35.195921+0000","last_became_active":"2024-05-09T03:09:34.199583+0000","last_became_peered":"2024-05-09T03:09:34.199583+0000","last_unstale":"2024-05-09T03:09:35.195921+0000","last_undegraded":"2024-05-09T03:09:35.195921+0000","last_fullsized":"2024-05-09T03:09:35.195921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194810+0000","last_change":"2024-05-09T03:09:34.199326+0000","last_active":"2024-05-09T03:09:35.194810+0000","last_peered":"2024-05-09T03:09:35.194810+0000","last_clean":"2024-05-09T03:09:35.194810+0000","last_became_active":"2024-05-09T03:09:34.198952+0000","last_became_peered":"2024-05-09T03:09:34.198952+0000","last_unstale":"2024-05-09T03:09:35.194810+0000","last_undegraded":"2024-05-09T03:09:35.194810+0000","last_fullsized":"2024-05-09T03:09:35.194810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195230+0000","last_change":"2024-05-09T03:09:34.186397+0000","last_active":"2024-05-09T03:09:35.195230+0000","last_peered":"2024-05-09T03:09:35.195230+0000","last_clean":"2024-05-09T03:09:35.195230+0000","last_became_active":"2024-05-09T03:09:34.186102+0000","last_became_peered":"2024-05-09T03:09:34.186102+0000","last_unstale":"2024-05-09T03:09:35.195230+0000","last_undegraded":"2024-05-09T03:09:35.195230+0000","last_fullsized":"2024-05-09T03:09:35.195230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:17.126340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.204285+0000","last_change":"2024-05-09T03:09:34.198335+0000","last_active":"2024-05-09T03:09:34.204285+0000","last_peered":"2024-05-09T03:09:34.204285+0000","last_clean":"2024-05-09T03:09:34.204285+0000","last_became_active":"2024-05-09T03:09:34.197960+0000","last_became_peered":"2024-05-09T03:09:34.197960+0000","last_unstale":"2024-05-09T03:09:34.204285+0000","last_undegraded":"2024-05-09T03:09:34.204285+0000","last_fullsized":"2024-05-09T03:09:34.204285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:41.508915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.198009+0000","last_change":"2024-05-09T03:09:34.189212+0000","last_active":"2024-05-09T03:09:34.198009+0000","last_peered":"2024-05-09T03:09:34.198009+0000","last_clean":"2024-05-09T03:09:34.198009+0000","last_became_active":"2024-05-09T03:09:34.188580+0000","last_became_peered":"2024-05-09T03:09:34.188580+0000","last_unstale":"2024-05-09T03:09:34.198009+0000","last_undegraded":"2024-05-09T03:09:34.198009+0000","last_fullsized":"2024-05-09T03:09:34.198009+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:06.254267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191292+0000","last_change":"2024-05-09T03:09:34.186444+0000","last_active":"2024-05-09T03:09:35.191292+0000","last_peered":"2024-05-09T03:09:35.191292+0000","last_clean":"2024-05-09T03:09:35.191292+0000","last_became_active":"2024-05-09T03:09:34.186119+0000","last_became_peered":"2024-05-09T03:09:34.186119+0000","last_unstale":"2024-05-09T03:09:35.191292+0000","last_undegraded":"2024-05-09T03:09:35.191292+0000","last_fullsized":"2024-05-09T03:09:35.191292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223239+0000","last_change":"2024-05-09T03:09:34.201163+0000","last_active":"2024-05-09T03:09:35.223239+0000","last_peered":"2024-05-09T03:09:35.223239+0000","last_clean":"2024-05-09T03:09:35.223239+0000","last_became_active":"2024-05-09T03:09:34.200814+0000","last_became_peered":"2024-05-09T03:09:34.200814+0000","last_unstale":"2024-05-09T03:09:35.223239+0000","last_undegraded":"2024-05-09T03:09:35.223239+0000","last_fullsized":"2024-05-09T03:09:35.223239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223892+0000","last_change":"2024-05-09T03:09:34.201044+0000","last_active":"2024-05-09T03:09:35.223892+0000","last_peered":"2024-05-09T03:09:35.223892+0000","last_clean":"2024-05-09T03:09:35.223892+0000","last_became_active":"2024-05-09T03:09:34.200392+0000","last_became_peered":"2024-05-09T03:09:34.200392+0000","last_unstale":"2024-05-09T03:09:35.223892+0000","last_undegraded":"2024-05-09T03:09:35.223892+0000","last_fullsized":"2024-05-09T03:09:35.223892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:49.231849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.206062+0000","last_change":"2024-05-09T03:09:34.203929+0000","last_active":"2024-05-09T03:09:34.206062+0000","last_peered":"2024-05-09T03:09:34.206062+0000","last_clean":"2024-05-09T03:09:34.206062+0000","last_became_active":"2024-05-09T03:09:34.203562+0000","last_became_peered":"2024-05-09T03:09:34.203562+0000","last_unstale":"2024-05-09T03:09:34.206062+0000","last_undegraded":"2024-05-09T03:09:34.206062+0000","last_fullsized":"2024-05-09T03:09:34.206062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:00.147452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.209286+0000","last_change":"2024-05-09T03:09:34.204594+0000","last_active":"2024-05-09T03:09:34.209286+0000","last_peered":"2024-05-09T03:09:34.209286+0000","last_clean":"2024-05-09T03:09:34.209286+0000","last_became_active":"2024-05-09T03:09:34.204289+0000","last_became_peered":"2024-05-09T03:09:34.204289+0000","last_unstale":"2024-05-09T03:09:34.209286+0000","last_undegraded":"2024-05-09T03:09:34.209286+0000","last_fullsized":"2024-05-09T03:09:34.209286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:33.100263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195935+0000","last_change":"2024-05-09T03:09:34.186570+0000","last_active":"2024-05-09T03:09:35.195935+0000","last_peered":"2024-05-09T03:09:35.195935+0000","last_clean":"2024-05-09T03:09:35.195935+0000","last_became_active":"2024-05-09T03:09:34.186249+0000","last_became_peered":"2024-05-09T03:09:34.186249+0000","last_unstale":"2024-05-09T03:09:35.195935+0000","last_undegraded":"2024-05-09T03:09:35.195935+0000","last_fullsized":"2024-05-09T03:09:35.195935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.198472+0000","last_change":"2024-05-09T03:09:34.195784+0000","last_active":"2024-05-09T03:09:34.198472+0000","last_peered":"2024-05-09T03:09:34.198472+0000","last_clean":"2024-05-09T03:09:34.198472+0000","last_became_active":"2024-05-09T03:09:34.195111+0000","last_became_peered":"2024-05-09T03:09:34.195111+0000","last_unstale":"2024-05-09T03:09:34.198472+0000","last_undegraded":"2024-05-09T03:09:34.198472+0000","last_fullsized":"2024-05-09T03:09:34.198472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:17.126340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223417+0000","last_change":"2024-05-09T03:09:34.201228+0000","last_active":"2024-05-09T03:09:35.223417+0000","last_peered":"2024-05-09T03:09:35.223417+0000","last_clean":"2024-05-09T03:09:35.223417+0000","last_became_active":"2024-05-09T03:09:34.200894+0000","last_became_peered":"2024-05-09T03:09:34.200894+0000","last_unstale":"2024-05-09T03:09:35.223417+0000","last_undegraded":"2024-05-09T03:09:35.223417+0000","last_fullsized":"2024-05-09T03:09:35.223417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197797+0000","last_change":"2024-05-09T03:09:34.189092+0000","last_active":"2024-05-09T03:09:34.197797+0000","last_peered":"2024-05-09T03:09:34.197797+0000","last_clean":"2024-05-09T03:09:34.197797+0000","last_became_active":"2024-05-09T03:09:34.188774+0000","last_became_peered":"2024-05-09T03:09:34.188774+0000","last_unstale":"2024-05-09T03:09:34.197797+0000","last_undegraded":"2024-05-09T03:09:34.197797+0000","last_fullsized":"2024-05-09T03:09:34.197797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:06.254267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223434+0000","last_change":"2024-05-09T03:09:34.201623+0000","last_active":"2024-05-09T03:09:35.223434+0000","last_peered":"2024-05-09T03:09:35.223434+0000","last_clean":"2024-05-09T03:09:35.223434+0000","last_became_active":"2024-05-09T03:09:34.200070+0000","last_became_peered":"2024-05-09T03:09:34.200070+0000","last_unstale":"2024-05-09T03:09:35.223434+0000","last_undegraded":"2024-05-09T03:09:35.223434+0000","last_fullsized":"2024-05-09T03:09:35.223434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.191391+0000","last_change":"2024-05-09T03:09:34.189949+0000","last_active":"2024-05-09T03:09:34.191391+0000","last_peered":"2024-05-09T03:09:34.191391+0000","last_clean":"2024-05-09T03:09:34.191391+0000","last_became_active":"2024-05-09T03:09:34.189595+0000","last_became_peered":"2024-05-09T03:09:34.189595+0000","last_unstale":"2024-05-09T03:09:34.191391+0000","last_undegraded":"2024-05-09T03:09:34.191391+0000","last_fullsized":"2024-05-09T03:09:34.191391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:00.147452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.196011+0000","last_change":"2024-05-09T03:09:34.188475+0000","last_active":"2024-05-09T03:09:35.196011+0000","last_peered":"2024-05-09T03:09:35.196011+0000","last_clean":"2024-05-09T03:09:35.196011+0000","last_became_active":"2024-05-09T03:09:34.188128+0000","last_became_peered":"2024-05-09T03:09:34.188128+0000","last_unstale":"2024-05-09T03:09:35.196011+0000","last_undegraded":"2024-05-09T03:09:35.196011+0000","last_fullsized":"2024-05-09T03:09:35.196011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191594+0000","last_change":"2024-05-09T03:09:34.191453+0000","last_active":"2024-05-09T03:09:35.191594+0000","last_peered":"2024-05-09T03:09:35.191594+0000","last_clean":"2024-05-09T03:09:35.191594+0000","last_became_active":"2024-05-09T03:09:34.191094+0000","last_became_peered":"2024-05-09T03:09:34.191094+0000","last_unstale":"2024-05-09T03:09:35.191594+0000","last_undegraded":"2024-05-09T03:09:35.191594+0000","last_fullsized":"2024-05-09T03:09:35.191594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.204685+0000","last_change":"2024-05-09T03:09:34.201091+0000","last_active":"2024-05-09T03:09:34.204685+0000","last_peered":"2024-05-09T03:09:34.204685+0000","last_clean":"2024-05-09T03:09:34.204685+0000","last_became_active":"2024-05-09T03:09:34.198235+0000","last_became_peered":"2024-05-09T03:09:34.198235+0000","last_unstale":"2024-05-09T03:09:34.204685+0000","last_undegraded":"2024-05-09T03:09:34.204685+0000","last_fullsized":"2024-05-09T03:09:34.204685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:21.923965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.198496+0000","last_change":"2024-05-09T03:09:34.196239+0000","last_active":"2024-05-09T03:09:34.198496+0000","last_peered":"2024-05-09T03:09:34.198496+0000","last_clean":"2024-05-09T03:09:34.198496+0000","last_became_active":"2024-05-09T03:09:34.195895+0000","last_became_peered":"2024-05-09T03:09:34.195895+0000","last_unstale":"2024-05-09T03:09:34.198496+0000","last_undegraded":"2024-05-09T03:09:34.198496+0000","last_fullsized":"2024-05-09T03:09:34.198496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223858+0000","last_change":"2024-05-09T03:09:34.206560+0000","last_active":"2024-05-09T03:09:35.223858+0000","last_peered":"2024-05-09T03:09:35.223858+0000","last_clean":"2024-05-09T03:09:35.223858+0000","last_became_active":"2024-05-09T03:09:34.205417+0000","last_became_peered":"2024-05-09T03:09:34.205417+0000","last_unstale":"2024-05-09T03:09:35.223858+0000","last_undegraded":"2024-05-09T03:09:35.223858+0000","last_fullsized":"2024-05-09T03:09:35.223858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:59:37.733696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.207954+0000","last_change":"2024-05-09T03:09:34.206717+0000","last_active":"2024-05-09T03:09:34.207954+0000","last_peered":"2024-05-09T03:09:34.207954+0000","last_clean":"2024-05-09T03:09:34.207954+0000","last_became_active":"2024-05-09T03:09:34.206391+0000","last_became_peered":"2024-05-09T03:09:34.206391+0000","last_unstale":"2024-05-09T03:09:34.207954+0000","last_undegraded":"2024-05-09T03:09:34.207954+0000","last_fullsized":"2024-05-09T03:09:34.207954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:17.126340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194450+0000","last_change":"2024-05-09T03:09:34.181522+0000","last_active":"2024-05-09T03:09:35.194450+0000","last_peered":"2024-05-09T03:09:35.194450+0000","last_clean":"2024-05-09T03:09:35.194450+0000","last_became_active":"2024-05-09T03:09:34.180437+0000","last_became_peered":"2024-05-09T03:09:34.180437+0000","last_unstale":"2024-05-09T03:09:35.194450+0000","last_undegraded":"2024-05-09T03:09:35.194450+0000","last_fullsized":"2024-05-09T03:09:35.194450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194687+0000","last_change":"2024-05-09T03:09:34.187744+0000","last_active":"2024-05-09T03:09:35.194687+0000","last_peered":"2024-05-09T03:09:35.194687+0000","last_clean":"2024-05-09T03:09:35.194687+0000","last_became_active":"2024-05-09T03:09:34.187413+0000","last_became_peered":"2024-05-09T03:09:34.187413+0000","last_unstale":"2024-05-09T03:09:35.194687+0000","last_undegraded":"2024-05-09T03:09:35.194687+0000","last_fullsized":"2024-05-09T03:09:35.194687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:00.147452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.205285+0000","last_change":"2024-05-09T03:09:33.189245+0000","last_active":"2024-05-09T03:09:34.205285+0000","last_peered":"2024-05-09T03:09:34.205285+0000","last_clean":"2024-05-09T03:09:34.205285+0000","last_became_active":"2024-05-09T03:09:33.188865+0000","last_became_peered":"2024-05-09T03:09:33.188865+0000","last_unstale":"2024-05-09T03:09:34.205285+0000","last_undegraded":"2024-05-09T03:09:34.205285+0000","last_fullsized":"2024-05-09T03:09:34.205285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.189991+0000","last_change":"2024-05-09T03:09:34.183515+0000","last_active":"2024-05-09T03:09:34.189991+0000","last_peered":"2024-05-09T03:09:34.189991+0000","last_clean":"2024-05-09T03:09:34.189991+0000","last_became_active":"2024-05-09T03:09:34.183151+0000","last_became_peered":"2024-05-09T03:09:34.183151+0000","last_unstale":"2024-05-09T03:09:34.189991+0000","last_undegraded":"2024-05-09T03:09:34.189991+0000","last_fullsized":"2024-05-09T03:09:34.189991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:04.848193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.196036+0000","last_change":"2024-05-09T03:09:33.177972+0000","last_active":"2024-05-09T03:09:35.196036+0000","last_peered":"2024-05-09T03:09:35.196036+0000","last_clean":"2024-05-09T03:09:35.196036+0000","last_became_active":"2024-05-09T03:09:33.177585+0000","last_became_peered":"2024-05-09T03:09:33.177585+0000","last_unstale":"2024-05-09T03:09:35.196036+0000","last_undegraded":"2024-05-09T03:09:35.196036+0000","last_fullsized":"2024-05-09T03:09:35.196036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.180386+0000","last_change":"2024-05-09T03:09:33.193553+0000","last_active":"2024-05-09T03:09:34.180386+0000","last_peered":"2024-05-09T03:09:34.180386+0000","last_clean":"2024-05-09T03:09:34.180386+0000","last_became_active":"2024-05-09T03:09:33.193050+0000","last_became_peered":"2024-05-09T03:09:33.193050+0000","last_unstale":"2024-05-09T03:09:34.180386+0000","last_undegraded":"2024-05-09T03:09:34.180386+0000","last_fullsized":"2024-05-09T03:09:34.180386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:28.743738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197637+0000","last_change":"2024-05-09T03:09:34.193946+0000","last_active":"2024-05-09T03:09:34.197637+0000","last_peered":"2024-05-09T03:09:34.197637+0000","last_clean":"2024-05-09T03:09:34.197637+0000","last_became_active":"2024-05-09T03:09:34.192276+0000","last_became_peered":"2024-05-09T03:09:34.192276+0000","last_unstale":"2024-05-09T03:09:34.197637+0000","last_undegraded":"2024-05-09T03:09:34.197637+0000","last_fullsized":"2024-05-09T03:09:34.197637+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223499+0000","last_change":"2024-05-09T03:09:33.183226+0000","last_active":"2024-05-09T03:09:35.223499+0000","last_peered":"2024-05-09T03:09:35.223499+0000","last_clean":"2024-05-09T03:09:35.223499+0000","last_became_active":"2024-05-09T03:09:33.182875+0000","last_became_peered":"2024-05-09T03:09:33.182875+0000","last_unstale":"2024-05-09T03:09:35.223499+0000","last_undegraded":"2024-05-09T03:09:35.223499+0000","last_fullsized":"2024-05-09T03:09:35.223499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191627+0000","last_change":"2024-05-09T03:09:34.192348+0000","last_active":"2024-05-09T03:09:35.191627+0000","last_peered":"2024-05-09T03:09:35.191627+0000","last_clean":"2024-05-09T03:09:35.191627+0000","last_became_active":"2024-05-09T03:09:34.192007+0000","last_became_peered":"2024-05-09T03:09:34.192007+0000","last_unstale":"2024-05-09T03:09:35.191627+0000","last_undegraded":"2024-05-09T03:09:35.191627+0000","last_fullsized":"2024-05-09T03:09:35.191627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195519+0000","last_change":"2024-05-09T03:09:33.189847+0000","last_active":"2024-05-09T03:09:35.195519+0000","last_peered":"2024-05-09T03:09:35.195519+0000","last_clean":"2024-05-09T03:09:35.195519+0000","last_became_active":"2024-05-09T03:09:33.189107+0000","last_became_peered":"2024-05-09T03:09:33.189107+0000","last_unstale":"2024-05-09T03:09:35.195519+0000","last_undegraded":"2024-05-09T03:09:35.195519+0000","last_fullsized":"2024-05-09T03:09:35.195519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223574+0000","last_change":"2024-05-09T03:09:34.198289+0000","last_active":"2024-05-09T03:09:35.223574+0000","last_peered":"2024-05-09T03:09:35.223574+0000","last_clean":"2024-05-09T03:09:35.223574+0000","last_became_active":"2024-05-09T03:09:34.197367+0000","last_became_peered":"2024-05-09T03:09:34.197367+0000","last_unstale":"2024-05-09T03:09:35.223574+0000","last_undegraded":"2024-05-09T03:09:35.223574+0000","last_fullsized":"2024-05-09T03:09:35.223574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223764+0000","last_change":"2024-05-09T03:09:33.188844+0000","last_active":"2024-05-09T03:09:35.223764+0000","last_peered":"2024-05-09T03:09:35.223764+0000","last_clean":"2024-05-09T03:09:35.223764+0000","last_became_active":"2024-05-09T03:09:33.188348+0000","last_became_peered":"2024-05-09T03:09:33.188348+0000","last_unstale":"2024-05-09T03:09:35.223764+0000","last_undegraded":"2024-05-09T03:09:35.223764+0000","last_fullsized":"2024-05-09T03:09:35.223764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.201227+0000","last_change":"2024-05-09T03:09:34.199361+0000","last_active":"2024-05-09T03:09:34.201227+0000","last_peered":"2024-05-09T03:09:34.201227+0000","last_clean":"2024-05-09T03:09:34.201227+0000","last_became_active":"2024-05-09T03:09:34.199023+0000","last_became_peered":"2024-05-09T03:09:34.199023+0000","last_unstale":"2024-05-09T03:09:34.201227+0000","last_undegraded":"2024-05-09T03:09:34.201227+0000","last_fullsized":"2024-05-09T03:09:34.201227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191595+0000","last_change":"2024-05-09T03:09:33.193443+0000","last_active":"2024-05-09T03:09:35.191595+0000","last_peered":"2024-05-09T03:09:35.191595+0000","last_clean":"2024-05-09T03:09:35.191595+0000","last_became_active":"2024-05-09T03:09:33.193095+0000","last_became_peered":"2024-05-09T03:09:33.193095+0000","last_unstale":"2024-05-09T03:09:35.191595+0000","last_undegraded":"2024-05-09T03:09:35.191595+0000","last_fullsized":"2024-05-09T03:09:35.191595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195302+0000","last_change":"2024-05-09T03:09:34.190211+0000","last_active":"2024-05-09T03:09:35.195302+0000","last_peered":"2024-05-09T03:09:35.195302+0000","last_clean":"2024-05-09T03:09:35.195302+0000","last_became_active":"2024-05-09T03:09:34.189747+0000","last_became_peered":"2024-05-09T03:09:34.189747+0000","last_unstale":"2024-05-09T03:09:35.195302+0000","last_undegraded":"2024-05-09T03:09:35.195302+0000","last_fullsized":"2024-05-09T03:09:35.195302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195680+0000","last_change":"2024-05-09T03:09:33.186806+0000","last_active":"2024-05-09T03:09:35.195680+0000","last_peered":"2024-05-09T03:09:35.195680+0000","last_clean":"2024-05-09T03:09:35.195680+0000","last_became_active":"2024-05-09T03:09:33.185580+0000","last_became_peered":"2024-05-09T03:09:33.185580+0000","last_unstale":"2024-05-09T03:09:35.195680+0000","last_undegraded":"2024-05-09T03:09:35.195680+0000","last_fullsized":"2024-05-09T03:09:35.195680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197135+0000","last_change":"2024-05-09T03:09:34.191009+0000","last_active":"2024-05-09T03:09:34.197135+0000","last_peered":"2024-05-09T03:09:34.197135+0000","last_clean":"2024-05-09T03:09:34.197135+0000","last_became_active":"2024-05-09T03:09:34.190131+0000","last_became_peered":"2024-05-09T03:09:34.190131+0000","last_unstale":"2024-05-09T03:09:34.197135+0000","last_undegraded":"2024-05-09T03:09:34.197135+0000","last_fullsized":"2024-05-09T03:09:34.197135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:43.778183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.200401+0000","last_change":"2024-05-09T03:09:33.185616+0000","last_active":"2024-05-09T03:09:34.200401+0000","last_peered":"2024-05-09T03:09:34.200401+0000","last_clean":"2024-05-09T03:09:34.200401+0000","last_became_active":"2024-05-09T03:09:33.185302+0000","last_became_peered":"2024-05-09T03:09:33.185302+0000","last_unstale":"2024-05-09T03:09:34.200401+0000","last_undegraded":"2024-05-09T03:09:34.200401+0000","last_fullsized":"2024-05-09T03:09:34.200401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:28.743738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.198033+0000","last_change":"2024-05-09T03:09:34.190554+0000","last_active":"2024-05-09T03:09:34.198033+0000","last_peered":"2024-05-09T03:09:34.198033+0000","last_clean":"2024-05-09T03:09:34.198033+0000","last_became_active":"2024-05-09T03:09:34.190165+0000","last_became_peered":"2024-05-09T03:09:34.190165+0000","last_unstale":"2024-05-09T03:09:34.198033+0000","last_undegraded":"2024-05-09T03:09:34.198033+0000","last_fullsized":"2024-05-09T03:09:34.198033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:00.147452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223697+0000","last_change":"2024-05-09T03:09:33.191326+0000","last_active":"2024-05-09T03:09:35.223697+0000","last_peered":"2024-05-09T03:09:35.223697+0000","last_clean":"2024-05-09T03:09:35.223697+0000","last_became_active":"2024-05-09T03:09:33.190974+0000","last_became_peered":"2024-05-09T03:09:33.190974+0000","last_unstale":"2024-05-09T03:09:35.223697+0000","last_undegraded":"2024-05-09T03:09:35.223697+0000","last_fullsized":"2024-05-09T03:09:35.223697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.198235+0000","last_change":"2024-05-09T03:09:34.195650+0000","last_active":"2024-05-09T03:09:34.198235+0000","last_peered":"2024-05-09T03:09:34.198235+0000","last_clean":"2024-05-09T03:09:34.198235+0000","last_became_active":"2024-05-09T03:09:34.195236+0000","last_became_peered":"2024-05-09T03:09:34.195236+0000","last_unstale":"2024-05-09T03:09:34.198235+0000","last_undegraded":"2024-05-09T03:09:34.198235+0000","last_fullsized":"2024-05-09T03:09:34.198235+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195554+0000","last_change":"2024-05-09T03:09:33.184933+0000","last_active":"2024-05-09T03:09:35.195554+0000","last_peered":"2024-05-09T03:09:35.195554+0000","last_clean":"2024-05-09T03:09:35.195554+0000","last_became_active":"2024-05-09T03:09:33.184584+0000","last_became_peered":"2024-05-09T03:09:33.184584+0000","last_unstale":"2024-05-09T03:09:35.195554+0000","last_undegraded":"2024-05-09T03:09:35.195554+0000","last_fullsized":"2024-05-09T03:09:35.195554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195463+0000","last_change":"2024-05-09T03:09:34.179608+0000","last_active":"2024-05-09T03:09:35.195463+0000","last_peered":"2024-05-09T03:09:35.195463+0000","last_clean":"2024-05-09T03:09:35.195463+0000","last_became_active":"2024-05-09T03:09:34.179281+0000","last_became_peered":"2024-05-09T03:09:34.179281+0000","last_unstale":"2024-05-09T03:09:35.195463+0000","last_undegraded":"2024-05-09T03:09:35.195463+0000","last_fullsized":"2024-05-09T03:09:35.195463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.210135+0000","last_change":"2024-05-09T03:09:33.183294+0000","last_active":"2024-05-09T03:09:34.210135+0000","last_peered":"2024-05-09T03:09:34.210135+0000","last_clean":"2024-05-09T03:09:34.210135+0000","last_became_active":"2024-05-09T03:09:33.182789+0000","last_became_peered":"2024-05-09T03:09:33.182789+0000","last_unstale":"2024-05-09T03:09:34.210135+0000","last_undegraded":"2024-05-09T03:09:34.210135+0000","last_fullsized":"2024-05-09T03:09:34.210135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.198411+0000","last_change":"2024-05-09T03:09:34.195805+0000","last_active":"2024-05-09T03:09:34.198411+0000","last_peered":"2024-05-09T03:09:34.198411+0000","last_clean":"2024-05-09T03:09:34.198411+0000","last_became_active":"2024-05-09T03:09:34.195460+0000","last_became_peered":"2024-05-09T03:09:34.195460+0000","last_unstale":"2024-05-09T03:09:34.198411+0000","last_undegraded":"2024-05-09T03:09:34.198411+0000","last_fullsized":"2024-05-09T03:09:34.198411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195504+0000","last_change":"2024-05-09T03:09:33.182939+0000","last_active":"2024-05-09T03:09:35.195504+0000","last_peered":"2024-05-09T03:09:35.195504+0000","last_clean":"2024-05-09T03:09:35.195504+0000","last_became_active":"2024-05-09T03:09:33.182611+0000","last_became_peered":"2024-05-09T03:09:33.182611+0000","last_unstale":"2024-05-09T03:09:35.195504+0000","last_undegraded":"2024-05-09T03:09:35.195504+0000","last_fullsized":"2024-05-09T03:09:35.195504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:18.783039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.196015+0000","last_change":"2024-05-09T03:09:34.190860+0000","last_active":"2024-05-09T03:09:34.196015+0000","last_peered":"2024-05-09T03:09:34.196015+0000","last_clean":"2024-05-09T03:09:34.196015+0000","last_became_active":"2024-05-09T03:09:34.190509+0000","last_became_peered":"2024-05-09T03:09:34.190509+0000","last_unstale":"2024-05-09T03:09:34.196015+0000","last_undegraded":"2024-05-09T03:09:34.196015+0000","last_fullsized":"2024-05-09T03:09:34.196015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:20.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.205970+0000","last_change":"2024-05-09T03:09:33.187823+0000","last_active":"2024-05-09T03:09:34.205970+0000","last_peered":"2024-05-09T03:09:34.205970+0000","last_clean":"2024-05-09T03:09:34.205970+0000","last_became_active":"2024-05-09T03:09:33.187487+0000","last_became_peered":"2024-05-09T03:09:33.187487+0000","last_unstale":"2024-05-09T03:09:34.205970+0000","last_undegraded":"2024-05-09T03:09:34.205970+0000","last_fullsized":"2024-05-09T03:09:34.205970+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:03.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.199314+0000","last_change":"2024-05-09T03:09:34.196111+0000","last_active":"2024-05-09T03:09:34.199314+0000","last_peered":"2024-05-09T03:09:34.199314+0000","last_clean":"2024-05-09T03:09:34.199314+0000","last_became_active":"2024-05-09T03:09:34.195741+0000","last_became_peered":"2024-05-09T03:09:34.195741+0000","last_unstale":"2024-05-09T03:09:34.199314+0000","last_undegraded":"2024-05-09T03:09:34.199314+0000","last_fullsized":"2024-05-09T03:09:34.199314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191690+0000","last_change":"2024-05-09T03:09:33.189105+0000","last_active":"2024-05-09T03:09:35.191690+0000","last_peered":"2024-05-09T03:09:35.191690+0000","last_clean":"2024-05-09T03:09:35.191690+0000","last_became_active":"2024-05-09T03:09:33.188557+0000","last_became_peered":"2024-05-09T03:09:33.188557+0000","last_unstale":"2024-05-09T03:09:35.191690+0000","last_undegraded":"2024-05-09T03:09:35.191690+0000","last_fullsized":"2024-05-09T03:09:35.191690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:40.503702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223831+0000","last_change":"2024-05-09T03:09:34.205860+0000","last_active":"2024-05-09T03:09:35.223831+0000","last_peered":"2024-05-09T03:09:35.223831+0000","last_clean":"2024-05-09T03:09:35.223831+0000","last_became_active":"2024-05-09T03:09:34.205494+0000","last_became_peered":"2024-05-09T03:09:34.205494+0000","last_unstale":"2024-05-09T03:09:35.223831+0000","last_undegraded":"2024-05-09T03:09:35.223831+0000","last_fullsized":"2024-05-09T03:09:35.223831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.607505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.193895+0000","last_change":"2024-05-09T03:09:33.191678+0000","last_active":"2024-05-09T03:09:34.193895+0000","last_peered":"2024-05-09T03:09:34.193895+0000","last_clean":"2024-05-09T03:09:34.193895+0000","last_became_active":"2024-05-09T03:09:33.191347+0000","last_became_peered":"2024-05-09T03:09:33.191347+0000","last_unstale":"2024-05-09T03:09:34.193895+0000","last_undegraded":"2024-05-09T03:09:34.193895+0000","last_fullsized":"2024-05-09T03:09:34.193895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.207159+0000","last_change":"2024-05-09T03:09:34.200500+0000","last_active":"2024-05-09T03:09:34.207159+0000","last_peered":"2024-05-09T03:09:34.207159+0000","last_clean":"2024-05-09T03:09:34.207159+0000","last_became_active":"2024-05-09T03:09:34.200204+0000","last_became_peered":"2024-05-09T03:09:34.200204+0000","last_unstale":"2024-05-09T03:09:34.207159+0000","last_undegraded":"2024-05-09T03:09:34.207159+0000","last_fullsized":"2024-05-09T03:09:34.207159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":108,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.181190+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:09:34.181190+0000","last_peered":"2024-05-09T03:09:34.181190+0000","last_clean":"2024-05-09T03:09:34.181190+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:09:34.181190+0000","last_undegraded":"2024-05-09T03:09:34.181190+0000","last_fullsized":"2024-05-09T03:09:34.181190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223311+0000","last_change":"2024-05-09T03:09:33.189141+0000","last_active":"2024-05-09T03:09:35.223311+0000","last_peered":"2024-05-09T03:09:35.223311+0000","last_clean":"2024-05-09T03:09:35.223311+0000","last_became_active":"2024-05-09T03:09:33.188576+0000","last_became_peered":"2024-05-09T03:09:33.188576+0000","last_unstale":"2024-05-09T03:09:35.223311+0000","last_undegraded":"2024-05-09T03:09:35.223311+0000","last_fullsized":"2024-05-09T03:09:35.223311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.205537+0000","last_change":"2024-05-09T03:09:34.202021+0000","last_active":"2024-05-09T03:09:34.205537+0000","last_peered":"2024-05-09T03:09:34.205537+0000","last_clean":"2024-05-09T03:09:34.205537+0000","last_became_active":"2024-05-09T03:09:34.201661+0000","last_became_peered":"2024-05-09T03:09:34.201661+0000","last_unstale":"2024-05-09T03:09:34.205537+0000","last_undegraded":"2024-05-09T03:09:34.205537+0000","last_fullsized":"2024-05-09T03:09:34.205537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:16.118667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.184849+0000","last_change":"2024-05-09T03:09:33.189640+0000","last_active":"2024-05-09T03:09:34.184849+0000","last_peered":"2024-05-09T03:09:34.184849+0000","last_clean":"2024-05-09T03:09:34.184849+0000","last_became_active":"2024-05-09T03:09:33.189291+0000","last_became_peered":"2024-05-09T03:09:33.189291+0000","last_unstale":"2024-05-09T03:09:34.184849+0000","last_undegraded":"2024-05-09T03:09:34.184849+0000","last_fullsized":"2024-05-09T03:09:34.184849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:56.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197775+0000","last_change":"2024-05-09T03:09:34.186522+0000","last_active":"2024-05-09T03:09:34.197775+0000","last_peered":"2024-05-09T03:09:34.197775+0000","last_clean":"2024-05-09T03:09:34.197775+0000","last_became_active":"2024-05-09T03:09:34.186183+0000","last_became_peered":"2024-05-09T03:09:34.186183+0000","last_unstale":"2024-05-09T03:09:34.197775+0000","last_undegraded":"2024-05-09T03:09:34.197775+0000","last_fullsized":"2024-05-09T03:09:34.197775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:16.433251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.193931+0000","last_change":"2024-05-09T03:09:33.186397+0000","last_active":"2024-05-09T03:09:34.193931+0000","last_peered":"2024-05-09T03:09:34.193931+0000","last_clean":"2024-05-09T03:09:34.193931+0000","last_became_active":"2024-05-09T03:09:33.186026+0000","last_became_peered":"2024-05-09T03:09:33.186026+0000","last_unstale":"2024-05-09T03:09:34.193931+0000","last_undegraded":"2024-05-09T03:09:34.193931+0000","last_fullsized":"2024-05-09T03:09:34.193931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191563+0000","last_change":"2024-05-09T03:09:34.206024+0000","last_active":"2024-05-09T03:09:35.191563+0000","last_peered":"2024-05-09T03:09:35.191563+0000","last_clean":"2024-05-09T03:09:35.191563+0000","last_became_active":"2024-05-09T03:09:34.205694+0000","last_became_peered":"2024-05-09T03:09:34.205694+0000","last_unstale":"2024-05-09T03:09:35.191563+0000","last_undegraded":"2024-05-09T03:09:35.191563+0000","last_fullsized":"2024-05-09T03:09:35.191563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223876+0000","last_change":"2024-05-09T03:09:33.189135+0000","last_active":"2024-05-09T03:09:35.223876+0000","last_peered":"2024-05-09T03:09:35.223876+0000","last_clean":"2024-05-09T03:09:35.223876+0000","last_became_active":"2024-05-09T03:09:33.188869+0000","last_became_peered":"2024-05-09T03:09:33.188869+0000","last_unstale":"2024-05-09T03:09:35.223876+0000","last_undegraded":"2024-05-09T03:09:35.223876+0000","last_fullsized":"2024-05-09T03:09:35.223876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195454+0000","last_change":"2024-05-09T03:09:34.189609+0000","last_active":"2024-05-09T03:09:35.195454+0000","last_peered":"2024-05-09T03:09:35.195454+0000","last_clean":"2024-05-09T03:09:35.195454+0000","last_became_active":"2024-05-09T03:09:34.189088+0000","last_became_peered":"2024-05-09T03:09:34.189088+0000","last_unstale":"2024-05-09T03:09:35.195454+0000","last_undegraded":"2024-05-09T03:09:35.195454+0000","last_fullsized":"2024-05-09T03:09:35.195454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:41.508915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191407+0000","last_change":"2024-05-09T03:09:33.184995+0000","last_active":"2024-05-09T03:09:35.191407+0000","last_peered":"2024-05-09T03:09:35.191407+0000","last_clean":"2024-05-09T03:09:35.191407+0000","last_became_active":"2024-05-09T03:09:33.184596+0000","last_became_peered":"2024-05-09T03:09:33.184596+0000","last_unstale":"2024-05-09T03:09:35.191407+0000","last_undegraded":"2024-05-09T03:09:35.191407+0000","last_fullsized":"2024-05-09T03:09:35.191407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223976+0000","last_change":"2024-05-09T03:09:34.205632+0000","last_active":"2024-05-09T03:09:35.223976+0000","last_peered":"2024-05-09T03:09:35.223976+0000","last_clean":"2024-05-09T03:09:35.223976+0000","last_became_active":"2024-05-09T03:09:34.205275+0000","last_became_peered":"2024-05-09T03:09:34.205275+0000","last_unstale":"2024-05-09T03:09:35.223976+0000","last_undegraded":"2024-05-09T03:09:35.223976+0000","last_fullsized":"2024-05-09T03:09:35.223976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.707927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.184369+0000","last_change":"2024-05-09T03:09:33.193320+0000","last_active":"2024-05-09T03:09:34.184369+0000","last_peered":"2024-05-09T03:09:34.184369+0000","last_clean":"2024-05-09T03:09:34.184369+0000","last_became_active":"2024-05-09T03:09:33.193005+0000","last_became_peered":"2024-05-09T03:09:33.193005+0000","last_unstale":"2024-05-09T03:09:34.184369+0000","last_undegraded":"2024-05-09T03:09:34.184369+0000","last_fullsized":"2024-05-09T03:09:34.184369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:40.503702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191290+0000","last_change":"2024-05-09T03:09:34.204993+0000","last_active":"2024-05-09T03:09:35.191290+0000","last_peered":"2024-05-09T03:09:35.191290+0000","last_clean":"2024-05-09T03:09:35.191290+0000","last_became_active":"2024-05-09T03:09:34.204647+0000","last_became_peered":"2024-05-09T03:09:34.204647+0000","last_unstale":"2024-05-09T03:09:35.191290+0000","last_undegraded":"2024-05-09T03:09:35.191290+0000","last_fullsized":"2024-05-09T03:09:35.191290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191516+0000","last_change":"2024-05-09T03:09:33.188474+0000","last_active":"2024-05-09T03:09:35.191516+0000","last_peered":"2024-05-09T03:09:35.191516+0000","last_clean":"2024-05-09T03:09:35.191516+0000","last_became_active":"2024-05-09T03:09:33.187643+0000","last_became_peered":"2024-05-09T03:09:33.187643+0000","last_unstale":"2024-05-09T03:09:35.191516+0000","last_undegraded":"2024-05-09T03:09:35.191516+0000","last_fullsized":"2024-05-09T03:09:35.191516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:04.365646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.193947+0000","last_change":"2024-05-09T03:09:34.190064+0000","last_active":"2024-05-09T03:09:34.193947+0000","last_peered":"2024-05-09T03:09:34.193947+0000","last_clean":"2024-05-09T03:09:34.193947+0000","last_became_active":"2024-05-09T03:09:34.189695+0000","last_became_peered":"2024-05-09T03:09:34.189695+0000","last_unstale":"2024-05-09T03:09:34.193947+0000","last_undegraded":"2024-05-09T03:09:34.193947+0000","last_fullsized":"2024-05-09T03:09:34.193947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:17.126340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195648+0000","last_change":"2024-05-09T03:09:33.188283+0000","last_active":"2024-05-09T03:09:35.195648+0000","last_peered":"2024-05-09T03:09:35.195648+0000","last_clean":"2024-05-09T03:09:35.195648+0000","last_became_active":"2024-05-09T03:09:33.187986+0000","last_became_peered":"2024-05-09T03:09:33.187986+0000","last_unstale":"2024-05-09T03:09:35.195648+0000","last_undegraded":"2024-05-09T03:09:35.195648+0000","last_fullsized":"2024-05-09T03:09:35.195648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:59.142240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197975+0000","last_change":"2024-05-09T03:09:34.188118+0000","last_active":"2024-05-09T03:09:34.197975+0000","last_peered":"2024-05-09T03:09:34.197975+0000","last_clean":"2024-05-09T03:09:34.197975+0000","last_became_active":"2024-05-09T03:09:34.187756+0000","last_became_peered":"2024-05-09T03:09:34.187756+0000","last_unstale":"2024-05-09T03:09:34.197975+0000","last_undegraded":"2024-05-09T03:09:34.197975+0000","last_fullsized":"2024-05-09T03:09:34.197975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:11.191879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.180347+0000","last_change":"2024-05-09T03:09:33.191538+0000","last_active":"2024-05-09T03:09:34.180347+0000","last_peered":"2024-05-09T03:09:34.180347+0000","last_clean":"2024-05-09T03:09:34.180347+0000","last_became_active":"2024-05-09T03:09:33.190940+0000","last_became_peered":"2024-05-09T03:09:33.190940+0000","last_unstale":"2024-05-09T03:09:34.180347+0000","last_undegraded":"2024-05-09T03:09:34.180347+0000","last_fullsized":"2024-05-09T03:09:34.180347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195374+0000","last_change":"2024-05-09T03:09:34.203831+0000","last_active":"2024-05-09T03:09:35.195374+0000","last_peered":"2024-05-09T03:09:35.195374+0000","last_clean":"2024-05-09T03:09:35.195374+0000","last_became_active":"2024-05-09T03:09:34.203509+0000","last_became_peered":"2024-05-09T03:09:34.203509+0000","last_unstale":"2024-05-09T03:09:35.195374+0000","last_undegraded":"2024-05-09T03:09:35.195374+0000","last_fullsized":"2024-05-09T03:09:35.195374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191428+0000","last_change":"2024-05-09T03:09:33.188345+0000","last_active":"2024-05-09T03:09:35.191428+0000","last_peered":"2024-05-09T03:09:35.191428+0000","last_clean":"2024-05-09T03:09:35.191428+0000","last_became_active":"2024-05-09T03:09:33.187996+0000","last_became_peered":"2024-05-09T03:09:33.187996+0000","last_unstale":"2024-05-09T03:09:35.191428+0000","last_undegraded":"2024-05-09T03:09:35.191428+0000","last_fullsized":"2024-05-09T03:09:35.191428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.190202+0000","last_change":"2024-05-09T03:09:34.188063+0000","last_active":"2024-05-09T03:09:34.190202+0000","last_peered":"2024-05-09T03:09:34.190202+0000","last_clean":"2024-05-09T03:09:34.190202+0000","last_became_active":"2024-05-09T03:09:34.187703+0000","last_became_peered":"2024-05-09T03:09:34.187703+0000","last_unstale":"2024-05-09T03:09:34.190202+0000","last_undegraded":"2024-05-09T03:09:34.190202+0000","last_fullsized":"2024-05-09T03:09:34.190202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:11.191879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191401+0000","last_change":"2024-05-09T03:09:33.191733+0000","last_active":"2024-05-09T03:09:35.191401+0000","last_peered":"2024-05-09T03:09:35.191401+0000","last_clean":"2024-05-09T03:09:35.191401+0000","last_became_active":"2024-05-09T03:09:33.190856+0000","last_became_peered":"2024-05-09T03:09:33.190856+0000","last_unstale":"2024-05-09T03:09:35.191401+0000","last_undegraded":"2024-05-09T03:09:35.191401+0000","last_fullsized":"2024-05-09T03:09:35.191401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.204311+0000","last_change":"2024-05-09T03:09:34.201161+0000","last_active":"2024-05-09T03:09:34.204311+0000","last_peered":"2024-05-09T03:09:34.204311+0000","last_clean":"2024-05-09T03:09:34.204311+0000","last_became_active":"2024-05-09T03:09:34.200789+0000","last_became_peered":"2024-05-09T03:09:34.200789+0000","last_unstale":"2024-05-09T03:09:34.204311+0000","last_undegraded":"2024-05-09T03:09:34.204311+0000","last_fullsized":"2024-05-09T03:09:34.204311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:57.557495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.212243+0000","last_change":"2024-05-09T03:09:33.191322+0000","last_active":"2024-05-09T03:09:34.212243+0000","last_peered":"2024-05-09T03:09:34.212243+0000","last_clean":"2024-05-09T03:09:34.212243+0000","last_became_active":"2024-05-09T03:09:33.190925+0000","last_became_peered":"2024-05-09T03:09:33.190925+0000","last_unstale":"2024-05-09T03:09:34.212243+0000","last_undegraded":"2024-05-09T03:09:34.212243+0000","last_fullsized":"2024-05-09T03:09:34.212243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:20.918752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195950+0000","last_change":"2024-05-09T03:09:34.181003+0000","last_active":"2024-05-09T03:09:35.195950+0000","last_peered":"2024-05-09T03:09:35.195950+0000","last_clean":"2024-05-09T03:09:35.195950+0000","last_became_active":"2024-05-09T03:09:34.180247+0000","last_became_peered":"2024-05-09T03:09:34.180247+0000","last_unstale":"2024-05-09T03:09:35.195950+0000","last_undegraded":"2024-05-09T03:09:35.195950+0000","last_fullsized":"2024-05-09T03:09:35.195950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223857+0000","last_change":"2024-05-09T03:09:34.207920+0000","last_active":"2024-05-09T03:09:35.223857+0000","last_peered":"2024-05-09T03:09:35.223857+0000","last_clean":"2024-05-09T03:09:35.223857+0000","last_became_active":"2024-05-09T03:09:34.207645+0000","last_became_peered":"2024-05-09T03:09:34.207645+0000","last_unstale":"2024-05-09T03:09:35.223857+0000","last_undegraded":"2024-05-09T03:09:35.223857+0000","last_fullsized":"2024-05-09T03:09:35.223857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:06.122864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.212235+0000","last_change":"2024-05-09T03:09:33.194189+0000","last_active":"2024-05-09T03:09:34.212235+0000","last_peered":"2024-05-09T03:09:34.212235+0000","last_clean":"2024-05-09T03:09:34.212235+0000","last_became_active":"2024-05-09T03:09:33.193816+0000","last_became_peered":"2024-05-09T03:09:33.193816+0000","last_unstale":"2024-05-09T03:09:34.212235+0000","last_undegraded":"2024-05-09T03:09:34.212235+0000","last_fullsized":"2024-05-09T03:09:34.212235+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223608+0000","last_change":"2024-05-09T03:09:34.207981+0000","last_active":"2024-05-09T03:09:35.223608+0000","last_peered":"2024-05-09T03:09:35.223608+0000","last_clean":"2024-05-09T03:09:35.223608+0000","last_became_active":"2024-05-09T03:09:34.207609+0000","last_became_peered":"2024-05-09T03:09:34.207609+0000","last_unstale":"2024-05-09T03:09:35.223608+0000","last_undegraded":"2024-05-09T03:09:35.223608+0000","last_fullsized":"2024-05-09T03:09:35.223608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:41.381783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.209694+0000","last_change":"2024-05-09T03:09:33.194110+0000","last_active":"2024-05-09T03:09:34.209694+0000","last_peered":"2024-05-09T03:09:34.209694+0000","last_clean":"2024-05-09T03:09:34.209694+0000","last_became_active":"2024-05-09T03:09:33.193682+0000","last_became_peered":"2024-05-09T03:09:33.193682+0000","last_unstale":"2024-05-09T03:09:34.209694+0000","last_undegraded":"2024-05-09T03:09:34.209694+0000","last_fullsized":"2024-05-09T03:09:34.209694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:04.365646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.202980+0000","last_change":"2024-05-09T03:09:34.201024+0000","last_active":"2024-05-09T03:09:34.202980+0000","last_peered":"2024-05-09T03:09:34.202980+0000","last_clean":"2024-05-09T03:09:34.202980+0000","last_became_active":"2024-05-09T03:09:34.200708+0000","last_became_peered":"2024-05-09T03:09:34.200708+0000","last_unstale":"2024-05-09T03:09:34.202980+0000","last_undegraded":"2024-05-09T03:09:34.202980+0000","last_fullsized":"2024-05-09T03:09:34.202980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:41.508915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.208312+0000","last_change":"2024-05-09T03:09:33.190754+0000","last_active":"2024-05-09T03:09:34.208312+0000","last_peered":"2024-05-09T03:09:34.208312+0000","last_clean":"2024-05-09T03:09:34.208312+0000","last_became_active":"2024-05-09T03:09:33.189371+0000","last_became_peered":"2024-05-09T03:09:33.189371+0000","last_unstale":"2024-05-09T03:09:34.208312+0000","last_undegraded":"2024-05-09T03:09:34.208312+0000","last_fullsized":"2024-05-09T03:09:34.208312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:03.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.197112+0000","last_change":"2024-05-09T03:09:34.193444+0000","last_active":"2024-05-09T03:09:34.197112+0000","last_peered":"2024-05-09T03:09:34.197112+0000","last_clean":"2024-05-09T03:09:34.197112+0000","last_became_active":"2024-05-09T03:09:34.193113+0000","last_became_peered":"2024-05-09T03:09:34.193113+0000","last_unstale":"2024-05-09T03:09:34.197112+0000","last_undegraded":"2024-05-09T03:09:34.197112+0000","last_fullsized":"2024-05-09T03:09:34.197112+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-09T03:09:34.200009+0000","last_change":"2024-05-09T03:09:33.191591+0000","last_active":"2024-05-09T03:09:34.200009+0000","last_peered":"2024-05-09T03:09:34.200009+0000","last_clean":"2024-05-09T03:09:34.200009+0000","last_became_active":"2024-05-09T03:09:33.191275+0000","last_became_peered":"2024-05-09T03:09:33.191275+0000","last_unstale":"2024-05-09T03:09:34.200009+0000","last_undegraded":"2024-05-09T03:09:34.200009+0000","last_fullsized":"2024-05-09T03:09:34.200009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:18.783039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":204800,"data_stored":66246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":192512,"data_stored":64634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":798720,"data_stored":655640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:09:38.667 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:09:38.667 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T03:09:38.667 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-09T03:09:38.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-09T03:09:38.668 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-09T03:09:38.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.817+0000 7fe43c48b640 1 Processor -- start 2024-05-09T03:09:38.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.817+0000 7fe43c48b640 1 -- start start 2024-05-09T03:09:38.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.818+0000 7fe43c48b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 0x7fe434160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.818+0000 7fe43c48b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4340d49e0 con 0x7fe4340b4af0 2024-05-09T03:09:38.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe43c48b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4340d4cc0 con 0x7fe4340b7ef0 2024-05-09T03:09:38.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe43c48b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4340d5bb0 con 0x7fe4340d57a0 2024-05-09T03:09:38.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439a26640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4340b4af0 0x7fe4340b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45560/0 (socket says 172.21.15.46:45560) 2024-05-09T03:09:38.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439a26640 1 -- 172.21.15.46:0/886465097 learned_addr learned my addr 172.21.15.46:0/886465097 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe43a227640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 0x7fe434160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3028689913 0 0) 0x7fe4340d49e0 con 0x7fe4340b4af0 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe420003520 con 0x7fe4340b4af0 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3754506152 0 0) 0x7fe420003520 con 0x7fe4340b4af0 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 msgr2=0x7fe434160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 0x7fe434160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 >> v1:172.21.15.88:6789/0 conn(0x7fe4340b7ef0 legacy=0x7fe4340b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4340d4fa0 con 0x7fe4340b4af0 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fe4240016f0 con 0x7fe4340b4af0 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 >> v1:172.21.15.46:6789/0 conn(0x7fe4340b4af0 legacy=0x7fe4340b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420003a30 0x7fe420003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 0x7fe420004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe420004d40 0x7fe4200051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4340d5bb0 con 0x7fe420004d40 2024-05-09T03:09:38.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe42000bb50 con 0x7fe420003a30 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe42000bd50 con 0x7fe4340d57a0 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe43aa28640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe420004d40 0x7fe4200051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe43a227640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420003a30 0x7fe420003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.819+0000 7fe439a26640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 0x7fe420004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.820+0000 7fe43aa28640 1 -- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 msgr2=0x7fe420004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.820+0000 7fe43aa28640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4340d57a0 0x7fe420004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.820+0000 7fe43aa28640 1 -- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420003a30 msgr2=0x7fe420003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.820+0000 7fe43aa28640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420003a30 0x7fe420003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.820+0000 7fe43aa28640 1 -- 172.21.15.46:0/886465097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe420003520 con 0x7fe420004d40 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.820+0000 7fe43aa28640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe420004d40 0x7fe4200051b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe42c001ee0 tx=0x7fe420005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8843dd591de4de0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c037070 con 0x7fe420004d40 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe42c032440 con 0x7fe420004d40 2024-05-09T03:09:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe439225640 1 -- 172.21.15.46:0/886465097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c001b30 con 0x7fe420004d40 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43c48b640 1 -- 172.21.15.46:0/886465097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe420004d40 msgr2=0x7fe4200051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43c48b640 1 --2- 172.21.15.46:0/886465097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe420004d40 0x7fe4200051b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe42c001ee0 tx=0x7fe420005830 comp rx=0 tx=0).stop 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43aa28640 1 -- 172.21.15.46:0/886465097 reap_dead start 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43c48b640 1 -- 172.21.15.46:0/886465097 shutdown_connections 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43c48b640 1 -- 172.21.15.46:0/886465097 >> 172.21.15.46:0/886465097 conn(0x7fe4340743e0 msgr2=0x7fe4340d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43c48b640 1 -- 172.21.15.46:0/886465097 shutdown_connections 2024-05-09T03:09:38.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.821+0000 7fe43c48b640 1 -- 172.21.15.46:0/886465097 wait complete. 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.823+0000 7fe43c48b640 1 Processor -- start 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.823+0000 7fe43c48b640 1 -- start start 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420004d40 0x7fe43414cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4341588b0 0x7fe434158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 0x7fe43415f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4340d1920 con 0x7fe43415f5c0 2024-05-09T03:09:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe4340d15b0 con 0x7fe420004d40 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4341580a0 con 0x7fe4341588b0 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 0x7fe43415f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 0x7fe43415f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55766/0 (socket says 172.21.15.46:55766) 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 -- 172.21.15.46:0/2075586794 learned_addr learned my addr 172.21.15.46:0/2075586794 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43a227640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420004d40 0x7fe43414cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe439a26640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4341588b0 0x7fe434158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 -- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4341588b0 msgr2=0x7fe434158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4341588b0 0x7fe434158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 -- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420004d40 msgr2=0x7fe43414cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420004d40 0x7fe43414cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 -- 172.21.15.46:0/2075586794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43415ff10 con 0x7fe43415f5c0 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe439a26640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4341588b0 0x7fe434158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:38.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43a227640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420004d40 0x7fe43414cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43aa28640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 0x7fe43415f9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe42c001fd0 tx=0x7fe42c001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c037040 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe42c032e10 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c007d90 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.825+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4341f9b70 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.826+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4341fa130 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.827+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe43406a270 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.827+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe42c043020 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.828+0000 7fe42affd640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4180762d0 0x7fe4180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.828+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe42c0b6f50 con 0x7fe43415f5c0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.828+0000 7fe43a227640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4180762d0 0x7fe4180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:38.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.829+0000 7fe43a227640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4180762d0 0x7fe4180787e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe43406d3b0 tx=0x7fe430008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:38.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:38.832+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe43415ff10 con 0x7fe43415f5c0 2024-05-09T03:09:39.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.073+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fe4340b79d0 con 0x7fe43415f5c0 2024-05-09T03:09:39.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.076+0000 7fe42affd640 1 -- 172.21.15.46:0/2075586794 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+189518 (secure 0 0 0) 0x7fe42c080820 con 0x7fe43415f5c0 2024-05-09T03:09:39.079 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:39.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4180762d0 msgr2=0x7fe4180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4180762d0 0x7fe4180787e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe43406d3b0 tx=0x7fe430008040 comp rx=0 tx=0).stop 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 msgr2=0x7fe43415f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 0x7fe43415f9d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe42c001fd0 tx=0x7fe42c001490 comp rx=0 tx=0).stop 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 shutdown_connections 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4180762d0 0x7fe4180787e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe43415f5c0 0x7fe43415f9d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4341588b0 0x7fe434158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 --2- 172.21.15.46:0/2075586794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe420004d40 0x7fe43414cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 >> 172.21.15.46:0/2075586794 conn(0x7fe4340743e0 msgr2=0x7fe43415bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 shutdown_connections 2024-05-09T03:09:39.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.079+0000 7fe43c48b640 1 -- 172.21.15.46:0/2075586794 wait complete. 2024-05-09T03:09:39.090 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":663422477},{"type":"v1","addr":"172.21.15.88:6833","nonce":663422477}]},"active_addr":"172.21.15.88:6833/663422477","active_change":"2024-05-09T03:09:14.480076+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3124857729}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":2359673674}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":1274374593}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3739868797}]}]} 2024-05-09T03:09:39.094 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-09T03:09:39.094 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-09T03:09:39.095 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.209+0000 7f6f55e0c640 1 Processor -- start 2024-05-09T03:09:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.209+0000 7f6f55e0c640 1 -- start start 2024-05-09T03:09:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f55e0c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f500d33a0 0x7f6f500cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f55e0c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f500b4ee0 0x7f6f500d1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f55e0c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f50160910 con 0x7f6f500d20e0 2024-05-09T03:09:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f55e0c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6f500b3e40 con 0x7f6f500b4ee0 2024-05-09T03:09:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f55e0c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6f500b4120 con 0x7f6f500d33a0 2024-05-09T03:09:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f500b4ee0 0x7f6f500d1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6f500b4ee0 0x7f6f500d1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55778/0 (socket says 172.21.15.46:55778) 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f4ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6f500d20e0 0x7f6f500cd140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45562/0 (socket says 172.21.15.46:45562) 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 -- 172.21.15.46:0/815097751 learned_addr learned my addr 172.21.15.46:0/815097751 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f4f7fe640 1 --2- 172.21.15.46:0/815097751 >> v2:172.21.15.88:3300/0 conn(0x7f6f500d33a0 0x7f6f500cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 -- 172.21.15.46:0/815097751 >> v2:172.21.15.88:3300/0 conn(0x7f6f500d33a0 msgr2=0x7f6f500cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 --2- 172.21.15.46:0/815097751 >> v2:172.21.15.88:3300/0 conn(0x7f6f500d33a0 0x7f6f500cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 -- 172.21.15.46:0/815097751 >> v1:172.21.15.46:6789/0 conn(0x7f6f500d20e0 legacy=0x7f6f500cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f46ffd640 1 -- 172.21.15.46:0/815097751 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f500b4400 con 0x7f6f500b4ee0 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.211+0000 7f6f4effd640 1 -- 172.21.15.46:0/815097751 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2337320501 0 0) 0x7f6f50160910 con 0x7f6f500d20e0 2024-05-09T03:09:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.212+0000 7f6f46ffd640 1 --2- 172.21.15.46:0/815097751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f500d1ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f50074d90 tx=0x7f6f3802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b52b8413a33251d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.212+0000 7f6f4effd640 1 -- 172.21.15.46:0/815097751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38037070 con 0x7f6f500b4ee0 2024-05-09T03:09:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.212+0000 7f6f4effd640 1 -- 172.21.15.46:0/815097751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f38032440 con 0x7f6f500b4ee0 2024-05-09T03:09:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.212+0000 7f6f4effd640 1 -- 172.21.15.46:0/815097751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38032810 con 0x7f6f500b4ee0 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 -- 172.21.15.46:0/815097751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 msgr2=0x7f6f500d1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/815097751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f500d1ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6f50074d90 tx=0x7f6f3802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 -- 172.21.15.46:0/815097751 shutdown_connections 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/815097751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f500d1ba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/815097751 >> v2:172.21.15.88:3300/0 conn(0x7f6f500d33a0 0x7f6f500cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 -- 172.21.15.46:0/815097751 >> 172.21.15.46:0/815097751 conn(0x7f6f5009b7d0 msgr2=0x7f6f5009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 -- 172.21.15.46:0/815097751 shutdown_connections 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.213+0000 7f6f55e0c640 1 -- 172.21.15.46:0/815097751 wait complete. 2024-05-09T03:09:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.215+0000 7f6f55e0c640 1 Processor -- start 2024-05-09T03:09:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.215+0000 7f6f55e0c640 1 -- start start 2024-05-09T03:09:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f55e0c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f5015d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f55e0c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f500d20e0 0x7f6f50156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f55e0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f500d33a0 0x7f6f5015d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f55e0c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6f500b5470 con 0x7f6f500b4ee0 2024-05-09T03:09:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f55e0c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6f500b52f0 con 0x7f6f500d20e0 2024-05-09T03:09:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f55e0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6f500d24f0 con 0x7f6f500d33a0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f5015d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f5015d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55780/0 (socket says 172.21.15.46:55780) 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f4f7fe640 1 -- 172.21.15.46:0/4266567591 learned_addr learned my addr 172.21.15.46:0/4266567591 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f4ffff640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f500d33a0 0x7f6f5015d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.216+0000 7f6f46ffd640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f500d20e0 0x7f6f50156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4f7fe640 1 -- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f500d33a0 msgr2=0x7f6f5015d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4f7fe640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f500d33a0 0x7f6f5015d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4f7fe640 1 -- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f500d20e0 msgr2=0x7f6f50156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4f7fe640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f500d20e0 0x7f6f50156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4f7fe640 1 -- 172.21.15.46:0/4266567591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f50157550 con 0x7f6f500b4ee0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4f7fe640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f5015d230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000e8a0 tx=0x7f6f4000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f40017070 con 0x7f6f500b4ee0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4000ee90 con 0x7f6f500b4ee0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f50157830 con 0x7f6f500b4ee0 2024-05-09T03:09:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000cb60 con 0x7f6f500b4ee0 2024-05-09T03:09:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.217+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f501596c0 con 0x7f6f500b4ee0 2024-05-09T03:09:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.218+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f50157d50 con 0x7f6f500b4ee0 2024-05-09T03:09:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.219+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6f4001a070 con 0x7f6f500b4ee0 2024-05-09T03:09:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.219+0000 7f6f4cff9640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6f30076300 0x7f6f30078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.220+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6f40097420 con 0x7f6f500b4ee0 2024-05-09T03:09:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.220+0000 7f6f46ffd640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6f30076300 0x7f6f30078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.220+0000 7f6f46ffd640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6f30076300 0x7f6f30078810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6f38034000 tx=0x7f6f3802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.224+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6f400606a0 con 0x7f6f500b4ee0 2024-05-09T03:09:39.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.416+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f50159920 con 0x7f6f500b4ee0 2024-05-09T03:09:39.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.418+0000 7f6f4cff9640 1 -- 172.21.15.46:0/4266567591 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7f6f400649a0 con 0x7f6f500b4ee0 2024-05-09T03:09:39.418 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:39.419 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6f30076300 msgr2=0x7f6f30078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6f30076300 0x7f6f30078810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6f38034000 tx=0x7f6f3802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 msgr2=0x7f6f5015d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f5015d230 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6f4000e8a0 tx=0x7f6f4000ed60 comp rx=0 tx=0).stop 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 shutdown_connections 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6f500d33a0 0x7f6f5015d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f500d20e0 0x7f6f50156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6f30076300 0x7f6f30078810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 --2- 172.21.15.46:0/4266567591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f500b4ee0 0x7f6f5015d230 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 >> 172.21.15.46:0/4266567591 conn(0x7f6f5009b7d0 msgr2=0x7f6f5006d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 shutdown_connections 2024-05-09T03:09:39.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.420+0000 7f6f55e0c640 1 -- 172.21.15.46:0/4266567591 wait complete. 2024-05-09T03:09:39.432 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-09T03:09:39.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.545+0000 7f2d3d172640 1 Processor -- start 2024-05-09T03:09:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.545+0000 7f2d3d172640 1 -- start start 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3d172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38160610 0x7f2d380b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3d172640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d380b7d20 0x7f2d380b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3d172640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d380b8130 con 0x7f2d380d57a0 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3d172640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2d380d5bb0 con 0x7f2d380b7d20 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3d172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2d380d49e0 con 0x7f2d38160610 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d36d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d380d57a0 0x7f2d380b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45572/0 (socket says 172.21.15.46:45572) 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d380b7d20 0x7f2d380b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d36d9d640 1 -- 172.21.15.46:0/2771365515 learned_addr learned my addr 172.21.15.46:0/2771365515 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3659c640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38160610 0x7f2d380b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3759e640 1 -- 172.21.15.46:0/2771365515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38160610 msgr2=0x7f2d380b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3759e640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38160610 0x7f2d380b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3759e640 1 -- 172.21.15.46:0/2771365515 >> v1:172.21.15.46:6789/0 conn(0x7f2d380d57a0 legacy=0x7f2d380b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d35d9b640 1 -- 172.21.15.46:0/2771365515 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452989911 0 0) 0x7f2d28001490 con 0x7f2d380d57a0 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3759e640 1 -- 172.21.15.46:0/2771365515 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d380d4cc0 con 0x7f2d380b7d20 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.547+0000 7f2d3659c640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38160610 0x7f2d380b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.548+0000 7f2d3759e640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d380b7d20 0x7f2d380b9450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d300097a0 tx=0x7f2d3002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c6107e65b54e4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.548+0000 7f2d35d9b640 1 -- 172.21.15.46:0/2771365515 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30037070 con 0x7f2d380b7d20 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.548+0000 7f2d35d9b640 1 -- 172.21.15.46:0/2771365515 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d30032440 con 0x7f2d380b7d20 2024-05-09T03:09:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.548+0000 7f2d35d9b640 1 -- 172.21.15.46:0/2771365515 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30032810 con 0x7f2d380b7d20 2024-05-09T03:09:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 -- 172.21.15.46:0/2771365515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d380b7d20 msgr2=0x7f2d380b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d380b7d20 0x7f2d380b9450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2d300097a0 tx=0x7f2d3002a180 comp rx=0 tx=0).stop 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 -- 172.21.15.46:0/2771365515 shutdown_connections 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d380b7d20 0x7f2d380b9450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2771365515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d38160610 0x7f2d380b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 -- 172.21.15.46:0/2771365515 >> 172.21.15.46:0/2771365515 conn(0x7f2d380743e0 msgr2=0x7f2d380d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 -- 172.21.15.46:0/2771365515 shutdown_connections 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.549+0000 7f2d3d172640 1 -- 172.21.15.46:0/2771365515 wait complete. 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.551+0000 7f2d3d172640 1 Processor -- start 2024-05-09T03:09:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.551+0000 7f2d3d172640 1 -- start start 2024-05-09T03:09:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3d172640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380b7d20 0x7f2d38151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3d172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d380d57a0 0x7f2d38151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3d172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 0x7f2d380ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3d172640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d38160a20 con 0x7f2d38160610 2024-05-09T03:09:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3d172640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2d380d5280 con 0x7f2d380b7d20 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3d172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2d380d15b0 con 0x7f2d380d57a0 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 0x7f2d380ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 0x7f2d380ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55798/0 (socket says 172.21.15.46:55798) 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.552+0000 7f2d3759e640 1 -- 172.21.15.46:0/2895124203 learned_addr learned my addr 172.21.15.46:0/2895124203 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3659c640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d380d57a0 0x7f2d38151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d36d9d640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380b7d20 0x7f2d38151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3759e640 1 -- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d380d57a0 msgr2=0x7f2d38151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3759e640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d380d57a0 0x7f2d38151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3759e640 1 -- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380b7d20 msgr2=0x7f2d38151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3759e640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380b7d20 0x7f2d38151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3759e640 1 -- 172.21.15.46:0/2895124203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d380cd890 con 0x7f2d38160610 2024-05-09T03:09:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d36d9d640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380b7d20 0x7f2d38151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3759e640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 0x7f2d380ce930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d30035040 tx=0x7f2d300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30037050 con 0x7f2d38160610 2024-05-09T03:09:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d30038040 con 0x7f2d38160610 2024-05-09T03:09:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d380cdb70 con 0x7f2d38160610 2024-05-09T03:09:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30006e10 con 0x7f2d38160610 2024-05-09T03:09:39.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.553+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d380ce130 con 0x7f2d38160610 2024-05-09T03:09:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.554+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d380cd600 con 0x7f2d38160610 2024-05-09T03:09:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.555+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d30030980 con 0x7f2d38160610 2024-05-09T03:09:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.555+0000 7f2d2f7fe640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d180762d0 0x7f2d180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.556+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2d300b8060 con 0x7f2d38160610 2024-05-09T03:09:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.556+0000 7f2d36d9d640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d180762d0 0x7f2d180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.556+0000 7f2d36d9d640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d180762d0 0x7f2d180787e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d28007fd0 tx=0x7f2d28009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.560+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2d300819c0 con 0x7f2d38160610 2024-05-09T03:09:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.751+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d380b66e0 con 0x7f2d38160610 2024-05-09T03:09:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.752+0000 7f2d2f7fe640 1 -- 172.21.15.46:0/2895124203 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7f2d30032e20 con 0x7f2d38160610 2024-05-09T03:09:39.753 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:39.753 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d180762d0 msgr2=0x7f2d180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d180762d0 0x7f2d180787e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2d28007fd0 tx=0x7f2d28009040 comp rx=0 tx=0).stop 2024-05-09T03:09:39.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 msgr2=0x7f2d380ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 0x7f2d380ce930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2d30035040 tx=0x7f2d300046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 shutdown_connections 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d180762d0 0x7f2d180787e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d38160610 0x7f2d380ce930 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d380d57a0 0x7f2d38151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 --2- 172.21.15.46:0/2895124203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d380b7d20 0x7f2d38151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.754+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 >> 172.21.15.46:0/2895124203 conn(0x7f2d380743e0 msgr2=0x7f2d380b54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.755+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 shutdown_connections 2024-05-09T03:09:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.755+0000 7f2d3d172640 1 -- 172.21.15.46:0/2895124203 wait complete. 2024-05-09T03:09:39.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-09T03:09:39.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-09T03:09:39.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-09T03:09:39.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-09T03:09:39.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-09T03:09:39.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-09T03:09:39.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-09T03:09:39.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-09T03:09:39.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.996+0000 7fcf2715f640 1 Processor -- start 2024-05-09T03:09:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.997+0000 7fcf2715f640 1 -- start start 2024-05-09T03:09:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7fabac443640 1 Processor -- start 2024-05-09T03:09:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.996+0000 7f820119d640 1 Processor -- start 2024-05-09T03:09:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.996+0000 7f820119d640 1 -- start start 2024-05-09T03:09:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7fabac443640 1 -- start start 2024-05-09T03:09:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7fbc0b9b7640 1 Processor -- start 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f820119d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81fc0b4bb0 0x7f81fc163de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f820119d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0739e0 0x7f81fc178ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f820119d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81fc16d8e0 con 0x7f81fc0b5560 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f820119d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f81fc16e3c0 con 0x7f81fc0b4bb0 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f820119d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc16dbc0 con 0x7f81fc0739e0 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fa59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f81fc0b5560 0x7f81fc0732d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:35120/0 (socket says 172.21.15.46:35120) 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fa59c640 1 -- 172.21.15.46:0/1570465512 learned_addr learned my addr 172.21.15.46:0/1570465512 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fb59e640 1 --2- 172.21.15.46:0/1570465512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0739e0 0x7f81fc178ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fad9d640 1 --2- 172.21.15.46:0/1570465512 >> v2:172.21.15.46:3300/0 conn(0x7f81fc0b4bb0 0x7f81fc163de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fb59e640 1 -- 172.21.15.46:0/1570465512 >> v2:172.21.15.46:3300/0 conn(0x7f81fc0b4bb0 msgr2=0x7f81fc163de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fb59e640 1 --2- 172.21.15.46:0/1570465512 >> v2:172.21.15.46:3300/0 conn(0x7f81fc0b4bb0 0x7f81fc163de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fb59e640 1 -- 172.21.15.46:0/1570465512 >> v1:172.21.15.88:6789/0 conn(0x7f81fc0b5560 legacy=0x7f81fc0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fb59e640 1 -- 172.21.15.46:0/1570465512 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc164320 con 0x7f81fc0739e0 2024-05-09T03:09:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7f81fad9d640 1 --2- 172.21.15.46:0/1570465512 >> v2:172.21.15.46:3300/0 conn(0x7f81fc0b4bb0 0x7f81fc163de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f6aeffff640 1 Processor -- start 2024-05-09T03:09:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f00b42d9640 1 Processor -- start 2024-05-09T03:09:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fcf2715f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200d2ae0 0x7fcf200ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fcf2715f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf200cfa90 con 0x7fcf200b3000 2024-05-09T03:09:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fcf2715f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf200cef10 con 0x7fcf20164c60 2024-05-09T03:09:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fcf2715f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcf200c7450 con 0x7fcf200d2ae0 2024-05-09T03:09:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fcf1ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcf200b3000 0x7fcf200ce1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45588/0 (socket says 172.21.15.46:45588) 2024-05-09T03:09:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fcf1ffff640 1 -- 172.21.15.46:0/552704570 learned_addr learned my addr 172.21.15.46:0/552704570 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.000+0000 7fabac443640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fab9c0e16a0 0x7fab9c034140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabac443640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab9c050930 con 0x7fab9c038c20 2024-05-09T03:09:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabaa1df640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fab9c03a510 0x7fab9c038510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:35132/0 (socket says 172.21.15.46:35132) 2024-05-09T03:09:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabac443640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab9c03aaa0 con 0x7fab9c03a510 2024-05-09T03:09:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabac443640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fab9c0e1ab0 con 0x7fab9c0e16a0 2024-05-09T03:09:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabaa1df640 1 -- 172.21.15.46:0/2931172555 learned_addr learned my addr 172.21.15.46:0/2931172555 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabaa9e0640 1 --2- 172.21.15.46:0/2931172555 >> v2:172.21.15.46:3300/0 conn(0x7fab9c0e16a0 0x7fab9c034140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3861255770 0 0) 0x7fcf200cef10 con 0x7fcf20164c60 2024-05-09T03:09:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf08003520 con 0x7fcf20164c60 2024-05-09T03:09:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f00b42d9640 1 -- start start 2024-05-09T03:09:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f00b42d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4052fb0 0x7f00a404f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868512193 0 0) 0x7fcf18001490 con 0x7fcf200b3000 2024-05-09T03:09:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf200cef10 con 0x7fcf200b3000 2024-05-09T03:09:40.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.998+0000 7f6aeffff640 1 -- start start 2024-05-09T03:09:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3211384932 0 0) 0x7fcf08003520 con 0x7fcf20164c60 2024-05-09T03:09:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6af015d340 0x7f6af015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af00735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6af00711a0 0x7f6af0073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200d2ae0 msgr2=0x7fcf200ce8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200d2ae0 0x7fcf200ce8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 >> v1:172.21.15.46:6789/0 conn(0x7fcf200b3000 legacy=0x7fcf200ce1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeffff640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6af006f4e0 con 0x7f6af00711a0 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeffff640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6af006f780 con 0x7f6af015d340 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeffff640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6af006fa60 con 0x7f6af0070d90 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeeffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6af015d340 0x7f6af015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeeffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6af015d340 0x7f6af015c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56688/0 (socket says 172.21.15.46:56688) 2024-05-09T03:09:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6aeeffd640 1 -- 172.21.15.46:0/3773412073 learned_addr learned my addr 172.21.15.46:0/3773412073 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f6ae7fff640 1 --2- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af00735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf200cf1f0 con 0x7fcf20164c60 2024-05-09T03:09:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fcf100016f0 con 0x7fcf20164c60 2024-05-09T03:09:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f00b42d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00a4034180 0x7f00a4038340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 >> v1:172.21.15.88:6789/0 conn(0x7fcf20164c60 legacy=0x7fcf200b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf08003be0 0x7fcf08004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf08004800 0x7fcf08004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f81fb59e640 1 --2- 172.21.15.46:0/1570465512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0739e0 0x7f81fc178ff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009d90 tx=0x7f81f402a4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8261abfbab3fba7b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:39.999+0000 7fbc0b9b7640 1 -- start start 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fbc0b9b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbfc03a8c0 0x7fbbfc0e19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fbc0b9b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0525d0 0x7fbbfc0529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fbc0b9b7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbfc054b40 con 0x7fbbfc0389e0 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fbc0b9b7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbbfc0556e0 con 0x7fbbfc03a8c0 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fbc0b9b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbfc052f20 con 0x7fbbfc0525d0 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fbc09753640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbfc03a8c0 0x7fbbfc0e19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fbc09753640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbfc03a8c0 0x7fbbfc0e19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55846/0 (socket says 172.21.15.46:55846) 2024-05-09T03:09:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fbc09753640 1 -- 172.21.15.46:0/3057703035 learned_addr learned my addr 172.21.15.46:0/3057703035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf080099b0 0x7fcf0800bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f6aeeffd640 1 -- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 msgr2=0x7f6af00735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f6aeeffd640 1 --2- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af00735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f6aeeffd640 1 -- 172.21.15.46:0/3773412073 >> v2:172.21.15.46:3300/0 conn(0x7f6af00711a0 msgr2=0x7f6af0073ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f6aeeffd640 1 --2- 172.21.15.46:0/3773412073 >> v2:172.21.15.46:3300/0 conn(0x7f6af00711a0 0x7f6af0073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f6aeeffd640 1 -- 172.21.15.46:0/3773412073 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af006fd40 con 0x7f6af015d340 2024-05-09T03:09:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf200c7450 con 0x7fcf080099b0 2024-05-09T03:09:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0800c3c0 con 0x7fcf08003be0 2024-05-09T03:09:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0800c5c0 con 0x7fcf08004800 2024-05-09T03:09:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf08004800 0x7fcf08004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabaa9e0640 1 -- 172.21.15.46:0/2931172555 >> v1:172.21.15.88:6789/0 conn(0x7fab9c03a510 legacy=0x7fab9c038510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabaa9e0640 1 -- 172.21.15.46:0/2931172555 >> v1:172.21.15.46:6789/0 conn(0x7fab9c038c20 legacy=0x7fab9c0064c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7fabaa9e0640 1 -- 172.21.15.46:0/2931172555 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c050bb0 con 0x7fab9c0e16a0 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7fbc08f52640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0525d0 0x7fbbfc0529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1865529533 0 0) 0x7fbbfc054b40 con 0x7fbbfc0389e0 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbf0003520 con 0x7fbbfc0389e0 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7f81f9d9b640 1 -- 172.21.15.46:0/1570465512 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4007740 con 0x7f81fc0739e0 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7f81f9d9b640 1 -- 172.21.15.46:0/1570465512 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f40078e0 con 0x7f81fc0739e0 2024-05-09T03:09:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7f81f9d9b640 1 -- 172.21.15.46:0/1570465512 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4007cb0 con 0x7f81fc0739e0 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f00b42d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a40e8230 con 0x7f00a4007320 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f00b42d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f00a40533c0 con 0x7f00a4034180 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f00b42d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00a4006280 con 0x7f00a4052fb0 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.001+0000 7f00b2876640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00a4034180 0x7f00a4038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2876640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00a4034180 0x7f00a4038340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56674/0 (socket says 172.21.15.46:56674) 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2876640 1 -- 172.21.15.46:0/1060081380 learned_addr learned my addr 172.21.15.46:0/1060081380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2075640 1 --2- 172.21.15.46:0/1060081380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4052fb0 0x7f00a404f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2075640 1 -- 172.21.15.46:0/1060081380 >> v2:172.21.15.88:3300/0 conn(0x7f00a4034180 msgr2=0x7f00a4038340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2075640 1 --2- 172.21.15.46:0/1060081380 >> v2:172.21.15.88:3300/0 conn(0x7f00a4034180 0x7f00a4038340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2075640 1 -- 172.21.15.46:0/1060081380 >> v1:172.21.15.88:6789/0 conn(0x7f00a4007320 legacy=0x7f00a4033a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T03:09:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f00b2075640 1 -- 172.21.15.46:0/1060081380 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a4006560 con 0x7f00a4052fb0 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.002+0000 7f7c911c7640 1 Processor -- start 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3887130077 0 0) 0x7fbbf0003520 con 0x7fbbfc0389e0 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0525d0 msgr2=0x7fbbfc0529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0525d0 0x7fbbfc0529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 >> v2:172.21.15.46:3300/0 conn(0x7fbbfc03a8c0 msgr2=0x7fbbfc0e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 --2- 172.21.15.46:0/3057703035 >> v2:172.21.15.46:3300/0 conn(0x7fbbfc03a8c0 0x7fbbfc0e19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc054db0 con 0x7fbbfc0389e0 2024-05-09T03:09:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fbbf40016f0 con 0x7fbbfc0389e0 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 >> v1:172.21.15.88:6789/0 conn(0x7fbbfc0389e0 legacy=0x7fbbfc038df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbf0003a50 0x7fbbf0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbf0004800 0x7fbbf0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf24efb640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf08003be0 0x7fcf08004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf256fc640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf080099b0 0x7fcf0800bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf1ffff640 1 -- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf08003be0 msgr2=0x7fcf08004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf08003be0 0x7fcf08004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf1ffff640 1 -- 172.21.15.46:0/552704570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf080099b0 msgr2=0x7fcf0800bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf080099b0 0x7fcf0800bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fcf1ffff640 1 -- 172.21.15.46:0/552704570 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf200cf1f0 con 0x7fcf08004800 2024-05-09T03:09:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbf0005150 0x7fbbf000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc0556e0 con 0x7fbbf0004800 2024-05-09T03:09:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc052f20 con 0x7fbbf0005150 2024-05-09T03:09:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf000c030 con 0x7fbbf0003a50 2024-05-09T03:09:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7f7c911c7640 1 -- start start 2024-05-09T03:09:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbc09753640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbf0003a50 0x7fbbf0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fbc09f54640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbf0005150 0x7fbbf000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7fabaa9e0640 1 --2- 172.21.15.46:0/2931172555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0e16a0 0x7fab9c034140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faba4000b60 tx=0x7faba40a7140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3217ccc0a8dcf9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7faba91dd640 1 -- 172.21.15.46:0/2931172555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba40b4070 con 0x7fab9c0e16a0 2024-05-09T03:09:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f00b2075640 1 --2- 172.21.15.46:0/1060081380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4052fb0 0x7f00a404f340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00a80097a0 tx=0x7f00a802a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9040d21a5e42aaac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7faba91dd640 1 -- 172.21.15.46:0/2931172555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba40a7d40 con 0x7fab9c0e16a0 2024-05-09T03:09:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f6aeeffd640 1 --2- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af015c120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6af0073050 tx=0x7f6ae002e8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eca92e5afba9a22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf08004800 0x7fcf08004c10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c4c0 tx=0x7fcf08005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20bfb983935a8f67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7fbc09753640 1 -- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbf0005150 msgr2=0x7fbbf000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7fbc09753640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbf0005150 0x7fbbf000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7fbc09753640 1 -- 172.21.15.46:0/3057703035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbf0004800 msgr2=0x7fbbf0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7fbc09753640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbf0004800 0x7fbbf0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7fbc09753640 1 -- 172.21.15.46:0/3057703035 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc054db0 con 0x7fbbf0003a50 2024-05-09T03:09:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f00b1073640 1 -- 172.21.15.46:0/1060081380 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a8037070 con 0x7f00a4052fb0 2024-05-09T03:09:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.003+0000 7faba91dd640 1 -- 172.21.15.46:0/2931172555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba40af640 con 0x7fab9c0e16a0 2024-05-09T03:09:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f6aee7fc640 1 -- 172.21.15.46:0/3773412073 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0031070 con 0x7f6af015d340 2024-05-09T03:09:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f6aee7fc640 1 -- 172.21.15.46:0/3773412073 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ae0035750 con 0x7f6af015d340 2024-05-09T03:09:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f00b1073640 1 -- 172.21.15.46:0/1060081380 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a802adb0 con 0x7f00a4052fb0 2024-05-09T03:09:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f820119d640 1 -- 172.21.15.46:0/1570465512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0739e0 msgr2=0x7f81fc178ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f820119d640 1 --2- 172.21.15.46:0/1570465512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0739e0 0x7f81fc178ff0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009d90 tx=0x7f81f402a4c0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.004+0000 7f6aee7fc640 1 -- 172.21.15.46:0/3773412073 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0035b20 con 0x7f6af015d340 2024-05-09T03:09:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fabac443640 1 -- 172.21.15.46:0/2931172555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0e16a0 msgr2=0x7fab9c034140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fabac443640 1 --2- 172.21.15.46:0/2931172555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0e16a0 0x7fab9c034140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faba4000b60 tx=0x7faba40a7140 comp rx=0 tx=0).stop 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf18032020 con 0x7fcf08004800 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf1802e650 con 0x7fcf08004800 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fcf1f7fe640 1 -- 172.21.15.46:0/552704570 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf1800c0e0 con 0x7fcf08004800 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f820119d640 1 -- 172.21.15.46:0/1570465512 shutdown_connections 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f820119d640 1 --2- 172.21.15.46:0/1570465512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0739e0 0x7f81fc178ff0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f820119d640 1 --2- 172.21.15.46:0/1570465512 >> v2:172.21.15.46:3300/0 conn(0x7f81fc0b4bb0 0x7f81fc163de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f820119d640 1 -- 172.21.15.46:0/1570465512 >> 172.21.15.46:0/1570465512 conn(0x7f81fc0ce370 msgr2=0x7f81fc0ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fbc09753640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbf0003a50 0x7fbbf0003ec0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc0406c160 tx=0x7fbbf0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74e791ba4a844f92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f820119d640 1 -- 172.21.15.46:0/1570465512 shutdown_connections 2024-05-09T03:09:40.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f820119d640 1 -- 172.21.15.46:0/1570465512 wait complete. 2024-05-09T03:09:40.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc040b3070 con 0x7fbbf0003a50 2024-05-09T03:09:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc040ae600 con 0x7fbbf0003a50 2024-05-09T03:09:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fbbfbfff640 1 -- 172.21.15.46:0/3057703035 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04064120 con 0x7fbbf0003a50 2024-05-09T03:09:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f6aeffff640 1 -- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 msgr2=0x7f6af015c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f6aeffff640 1 --2- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af015c120 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6af0073050 tx=0x7f6ae002e8e0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fabac443640 1 -- 172.21.15.46:0/2931172555 shutdown_connections 2024-05-09T03:09:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fabac443640 1 --2- 172.21.15.46:0/2931172555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c0e16a0 0x7fab9c034140 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7fabac443640 1 -- 172.21.15.46:0/2931172555 >> 172.21.15.46:0/2931172555 conn(0x7fab9c054cf0 msgr2=0x7fab9c055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f7c911c7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7c8c1566b0 0x7f7c8c15c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.005+0000 7f7c911c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c8c0d0e30 0x7f7c8c0d1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fabac443640 1 -- 172.21.15.46:0/2931172555 shutdown_connections 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 -- 172.21.15.46:0/3773412073 shutdown_connections 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 --2- 172.21.15.46:0/3773412073 >> v2:172.21.15.46:3300/0 conn(0x7f6af00711a0 0x7f6af0073ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 --2- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af00735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 --2- 172.21.15.46:0/3773412073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af015c120 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 -- 172.21.15.46:0/3773412073 >> 172.21.15.46:0/3773412073 conn(0x7f6af00b3910 msgr2=0x7f6af00b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b42d9640 1 -- 172.21.15.46:0/1060081380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4052fb0 msgr2=0x7f00a404f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b1073640 1 -- 172.21.15.46:0/1060081380 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a8032660 con 0x7f00a4052fb0 2024-05-09T03:09:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b42d9640 1 --2- 172.21.15.46:0/1060081380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4052fb0 0x7f00a404f340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00a80097a0 tx=0x7f00a802a060 comp rx=0 tx=0).stop 2024-05-09T03:09:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c8ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c8c0d0e30 0x7f7c8c0d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fabac443640 1 -- 172.21.15.46:0/2931172555 wait complete. 2024-05-09T03:09:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c8ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c8c0d0e30 0x7f7c8c0d1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55864/0 (socket says 172.21.15.46:55864) 2024-05-09T03:09:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8c0d0690 con 0x7f7c8c0cee30 2024-05-09T03:09:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8c156ac0 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8c0b7860 con 0x7f7c8c1566b0 2024-05-09T03:09:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c89d9b640 1 --2- 172.21.15.46:0/1080261737 >> v2:172.21.15.88:3300/0 conn(0x7f7c8c1566b0 0x7f7c8c15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f7c8ad9d640 1 -- 172.21.15.46:0/1080261737 learned_addr learned my addr 172.21.15.46:0/1080261737 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fcf2715f640 1 -- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf08004800 msgr2=0x7fcf08004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fcf2715f640 1 --2- 172.21.15.46:0/552704570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf08004800 0x7fcf08004c10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcf1800c4c0 tx=0x7fcf08005830 comp rx=0 tx=0).stop 2024-05-09T03:09:40.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fcf256fc640 1 -- 172.21.15.46:0/552704570 reap_dead start 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fcf2715f640 1 -- 172.21.15.46:0/552704570 shutdown_connections 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fcf2715f640 1 -- 172.21.15.46:0/552704570 >> 172.21.15.46:0/552704570 conn(0x7fcf200d3b90 msgr2=0x7fcf200d3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 -- 172.21.15.46:0/3773412073 shutdown_connections 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f6aeffff640 1 -- 172.21.15.46:0/3773412073 wait complete. 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7f7c8ad9d640 1 -- 172.21.15.46:0/1080261737 >> v2:172.21.15.88:3300/0 conn(0x7f7c8c1566b0 msgr2=0x7f7c8c15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7f7c8ad9d640 1 --2- 172.21.15.46:0/1080261737 >> v2:172.21.15.88:3300/0 conn(0x7f7c8c1566b0 0x7f7c8c15c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7f7c8ad9d640 1 -- 172.21.15.46:0/1080261737 >> v1:172.21.15.46:6789/0 conn(0x7f7c8c0cee30 legacy=0x7f7c8c15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7f7c8ad9d640 1 -- 172.21.15.46:0/1080261737 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c15cd40 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b42d9640 1 -- 172.21.15.46:0/1060081380 shutdown_connections 2024-05-09T03:09:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b42d9640 1 --2- 172.21.15.46:0/1060081380 >> v2:172.21.15.88:3300/0 conn(0x7f00a4034180 0x7f00a4038340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b42d9640 1 --2- 172.21.15.46:0/1060081380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4052fb0 0x7f00a404f340 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7f00b42d9640 1 -- 172.21.15.46:0/1060081380 >> 172.21.15.46:0/1060081380 conn(0x7f00a4054cf0 msgr2=0x7f00a4055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.006+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbf0003a50 msgr2=0x7fbbf0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7fbc0b9b7640 1 --2- 172.21.15.46:0/3057703035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbf0003a50 0x7fbbf0003ec0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc0406c160 tx=0x7fbbf0005830 comp rx=0 tx=0).stop 2024-05-09T03:09:40.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7fcf2715f640 1 -- 172.21.15.46:0/552704570 shutdown_connections 2024-05-09T03:09:40.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7fe581c9f640 1 Processor -- start 2024-05-09T03:09:40.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7fcf2715f640 1 -- 172.21.15.46:0/552704570 wait complete. 2024-05-09T03:09:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.007+0000 7fe581c9f640 1 -- start start 2024-05-09T03:09:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7fbc09f54640 1 -- 172.21.15.46:0/3057703035 reap_dead start 2024-05-09T03:09:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/3057703035 shutdown_connections 2024-05-09T03:09:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/3057703035 >> 172.21.15.46:0/3057703035 conn(0x7fbbfc056f60 msgr2=0x7fbbfc057370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/3057703035 shutdown_connections 2024-05-09T03:09:40.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/3057703035 wait complete. 2024-05-09T03:09:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7f820119d640 1 Processor -- start 2024-05-09T03:09:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7f820119d640 1 -- start start 2024-05-09T03:09:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7f00b42d9640 1 -- 172.21.15.46:0/1060081380 shutdown_connections 2024-05-09T03:09:40.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.008+0000 7f00b42d9640 1 -- 172.21.15.46:0/1060081380 wait complete. 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe581c9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5740e3db0 0x7fe5740e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe57b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5740e3db0 0x7fe5740e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe57b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5740e3db0 0x7fe5740e7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55872/0 (socket says 172.21.15.46:55872) 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe57b7fe640 1 -- 172.21.15.46:0/1593825628 learned_addr learned my addr 172.21.15.46:0/1593825628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe581c9f640 1 --2- 172.21.15.46:0/1593825628 >> v2:172.21.15.88:3300/0 conn(0x7fe574058510 0x7fe574037b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe574006450 con 0x7fe5740e73a0 2024-05-09T03:09:40.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe5740e4320 con 0x7fe5740e3db0 2024-05-09T03:09:40.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7f6aeffff640 1 Processor -- start 2024-05-09T03:09:40.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe5740e4600 con 0x7fe574058510 2024-05-09T03:09:40.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fabac443640 1 Processor -- start 2024-05-09T03:09:40.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fe57bfff640 1 --2- 172.21.15.46:0/1593825628 >> v2:172.21.15.88:3300/0 conn(0x7fe574058510 0x7fe574037b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7fabac443640 1 -- start start 2024-05-09T03:09:40.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7fe57bfff640 1 -- 172.21.15.46:0/1593825628 >> v2:172.21.15.46:3300/0 conn(0x7fe5740e3db0 msgr2=0x7fe5740e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7f820119d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0b4bb0 0x7f81fc164580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7fe57bfff640 1 --2- 172.21.15.46:0/1593825628 >> v2:172.21.15.46:3300/0 conn(0x7fe5740e3db0 0x7fe5740e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7fe57bfff640 1 -- 172.21.15.46:0/1593825628 >> v1:172.21.15.88:6789/0 conn(0x7fe5740e73a0 legacy=0x7fe574057e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7f820119d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81fc0b5560 0x7f81fc08b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7fe57bfff640 1 -- 172.21.15.46:0/1593825628 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5740e48e0 con 0x7fe574058510 2024-05-09T03:09:40.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7f820119d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81fc15c160 0x7f81fc170850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.010+0000 7fcf2715f640 1 Processor -- start 2024-05-09T03:09:40.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f820119d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc0b5af0 con 0x7f81fc15c160 2024-05-09T03:09:40.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f820119d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc0b5970 con 0x7f81fc0b5560 2024-05-09T03:09:40.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f820119d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc174dd0 con 0x7f81fc0b4bb0 2024-05-09T03:09:40.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7fcf2715f640 1 -- start start 2024-05-09T03:09:40.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.009+0000 7f6aeffff640 1 -- start start 2024-05-09T03:09:40.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7fbc0b9b7640 1 Processor -- start 2024-05-09T03:09:40.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7fbc0b9b7640 1 -- start start 2024-05-09T03:09:40.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f81fb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81fc15c160 0x7f81fc170850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f7c8ad9d640 1 --2- 172.21.15.46:0/1080261737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c0d1240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0b7cc0 tx=0x7f7c8402ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6342168102a03259 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f81fb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81fc15c160 0x7f81fc170850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55886/0 (socket says 172.21.15.46:55886) 2024-05-09T03:09:40.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.011+0000 7f81fb59e640 1 -- 172.21.15.46:0/1936408700 learned_addr learned my addr 172.21.15.46:0/1936408700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81fb59e640 1 -- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0b4bb0 msgr2=0x7f81fc164580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T03:09:40.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81fb59e640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81fc0b4bb0 0x7f81fc164580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81fb59e640 1 -- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81fc0b5560 msgr2=0x7f81fc08b8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T03:09:40.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f00b42d9640 1 Processor -- start 2024-05-09T03:09:40.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81fb59e640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81fc0b5560 0x7f81fc08b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81fb59e640 1 -- 172.21.15.46:0/1936408700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc164e80 con 0x7f81fc15c160 2024-05-09T03:09:40.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7fe57bfff640 1 --2- 172.21.15.46:0/1593825628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe574058510 0x7fe574037b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe57c000b60 tx=0x7fe57c0a70a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99cb669e8a512e9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f00b42d9640 1 -- start start 2024-05-09T03:09:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81fb59e640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81fc15c160 0x7f81fc170850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009eb0 tx=0x7f81f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f7c909c6640 1 -- 172.21.15.46:0/1080261737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84038070 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f7c909c6640 1 -- 172.21.15.46:0/1080261737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8402bb50 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f7c909c6640 1 -- 172.21.15.46:0/1080261737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84033440 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7fe57a7fc640 1 -- 172.21.15.46:0/1593825628 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0b4070 con 0x7fe574058510 2024-05-09T03:09:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7fabac443640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c03a510 0x7fab9c0e2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7fabaa1df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c03a510 0x7fab9c0e2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f403f070 con 0x7f81fc15c160 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af0154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af00711a0 0x7f6af0147dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af00735a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeffff640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6af0069ab0 con 0x7f6af00711a0 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeffff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6af0069930 con 0x7f6af015d340 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeffff640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6af015fc90 con 0x7f6af0070d90 2024-05-09T03:09:40.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af0154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af0154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43270/0 (socket says 172.21.15.46:43270) 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aeeffd640 1 -- 172.21.15.46:0/2419203415 learned_addr learned my addr 172.21.15.46:0/2419203415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6ae7fff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af00711a0 0x7f6af0147dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.012+0000 7f6aef7fe640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af00735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fe57a7fc640 1 -- 172.21.15.46:0/1593825628 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57c0a7ce0 con 0x7fe574058510 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f4003b90 con 0x7f81fc15c160 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f40318c0 con 0x7f81fc15c160 2024-05-09T03:09:40.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc165160 con 0x7f81fc15c160 2024-05-09T03:09:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fe57a7fc640 1 -- 172.21.15.46:0/1593825628 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0af640 con 0x7fe574058510 2024-05-09T03:09:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc165720 con 0x7f81fc15c160 2024-05-09T03:09:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0d0e30 msgr2=0x7f7c8c0d1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f7c911c7640 1 --2- 172.21.15.46:0/1080261737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c0d1240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0b7cc0 tx=0x7f7c8402ae00 comp rx=0 tx=0).stop 2024-05-09T03:09:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 shutdown_connections 2024-05-09T03:09:40.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7f7c911c7640 1 --2- 172.21.15.46:0/1080261737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c0d1240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f7c911c7640 1 --2- 172.21.15.46:0/1080261737 >> v2:172.21.15.88:3300/0 conn(0x7f7c8c1566b0 0x7f7c8c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 >> 172.21.15.46:0/1080261737 conn(0x7f7c8c0732c0 msgr2=0x7f7c8c0736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf2715f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200c9650 0x7fcf200cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc0b9b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbfc0cc7b0 0x7fbbfc0e3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc0b9b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0e19d0 0x7fbbfc0e1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc0b9b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbfc0e2320 0x7fbbfc179130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc0b9b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc0e61e0 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc0b9b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc055370 con 0x7fbbfc0e2320 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc0b9b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbfc0e6360 con 0x7fbbfc0e19d0 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fbc09753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbfc0cc7b0 0x7fbbfc0e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbfc0cc7b0 0x7fbbfc0e3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55924/0 (socket says 172.21.15.46:55924) 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 -- 172.21.15.46:0/1393005451 learned_addr learned my addr 172.21.15.46:0/1393005451 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf2715f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20070600 0x7fcf200c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe574058510 msgr2=0x7fe574037b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 --2- 172.21.15.46:0/1593825628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe574058510 0x7fe574037b50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe57c000b60 tx=0x7fe57c0a70a0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf24efb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200c9650 0x7fcf200cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf24efb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200c9650 0x7fcf200cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43286/0 (socket says 172.21.15.46:43286) 2024-05-09T03:09:40.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf24efb640 1 -- 172.21.15.46:0/711275942 learned_addr learned my addr 172.21.15.46:0/711275942 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf2715f640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf200c7b10 0x7fcf201fe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 shutdown_connections 2024-05-09T03:09:40.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 --2- 172.21.15.46:0/1593825628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe574058510 0x7fe574037b50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 --2- 172.21.15.46:0/1593825628 >> v2:172.21.15.46:3300/0 conn(0x7fe5740e3db0 0x7fe5740e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 >> 172.21.15.46:0/1593825628 conn(0x7fe57403ac60 msgr2=0x7fe57403b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabac443640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0e16a0 0x7fab9c17ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabac443640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c04e0c0 0x7fab9c0db090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabac443640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c033f30 con 0x7fab9c03a510 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabaa1df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c03a510 0x7fab9c0e2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55908/0 (socket says 172.21.15.46:55908) 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabaa1df640 1 -- 172.21.15.46:0/2783958765 learned_addr learned my addr 172.21.15.46:0/2783958765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabac443640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c033db0 con 0x7fab9c04e0c0 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.013+0000 7fabac443640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c006bd0 con 0x7fab9c0e16a0 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7faba99de640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0e16a0 0x7fab9c17ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f00b42d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4007320 0x7f00a40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f00b42d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a4052fb0 0x7f00a416f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b42d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a416feb0 0x7f00a41702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20070600 0x7fcf200c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fabaa9e0640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c04e0c0 0x7fab9c0db090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b2075640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4007320 0x7f00a40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf200ce8f0 con 0x7fcf20070600 2024-05-09T03:09:40.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf200cf7b0 con 0x7fcf200c7b10 2024-05-09T03:09:40.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcf200cea70 con 0x7fcf200c9650 2024-05-09T03:09:40.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81fc06a270 con 0x7f81fc15c160 2024-05-09T03:09:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b1874640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a4052fb0 0x7f00a416f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b1874640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a4052fb0 0x7f00a416f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56762/0 (socket says 172.21.15.46:56762) 2024-05-09T03:09:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b2075640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4007320 0x7f00a40cd4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43300/0 (socket says 172.21.15.46:43300) 2024-05-09T03:09:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b1874640 1 -- 172.21.15.46:0/4101326214 learned_addr learned my addr 172.21.15.46:0/4101326214 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b2876640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a416feb0 0x7f00a41702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 shutdown_connections 2024-05-09T03:09:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4006e00 con 0x7f00a416feb0 2024-05-09T03:09:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4006b20 con 0x7f00a4052fb0 2024-05-09T03:09:40.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fe581c9f640 1 -- 172.21.15.46:0/1593825628 wait complete. 2024-05-09T03:09:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00a4007730 con 0x7f00a4007320 2024-05-09T03:09:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fcf256fc640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf200c7b10 0x7fcf201fe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fabaa1df640 1 -- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0e16a0 msgr2=0x7fab9c17ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fabaa1df640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fab9c0e16a0 0x7fab9c17ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 shutdown_connections 2024-05-09T03:09:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f7c911c7640 1 -- 172.21.15.46:0/1080261737 wait complete. 2024-05-09T03:09:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fabaa1df640 1 -- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c04e0c0 msgr2=0x7fab9c0db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fabaa1df640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab9c04e0c0 0x7fab9c0db090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7fabaa1df640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c17b3d0 con 0x7fab9c03a510 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f6aef7fe640 1 -- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 msgr2=0x7f6af0154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f6aef7fe640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6af0070d90 0x7f6af0154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f6aef7fe640 1 -- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af00711a0 msgr2=0x7f6af0147dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f6aef7fe640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af00711a0 0x7f6af0147dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7f6aef7fe640 1 -- 172.21.15.46:0/2419203415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af0154ba0 con 0x7f6af015d340 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f6ae7fff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af00711a0 0x7f6af0147dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.015+0000 7f6aef7fe640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af00735a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800be50 tx=0x7f6ae800bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b1874640 1 -- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4007320 msgr2=0x7f00a40cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b1874640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4007320 0x7f00a40cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b1874640 1 -- 172.21.15.46:0/4101326214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a416feb0 msgr2=0x7f00a41702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b1874640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a416feb0 0x7f00a41702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b1874640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a40e71d0 con 0x7f00a4052fb0 2024-05-09T03:09:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81f4031a60 con 0x7f81fc15c160 2024-05-09T03:09:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b2876640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a416feb0 0x7f00a41702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:40.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b2075640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a4007320 0x7f00a40cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 -- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0e19d0 msgr2=0x7fbbfc0e1de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbfc0e19d0 0x7fbbfc0e1de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 -- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbfc0e2320 msgr2=0x7fbbfc179130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbfc0e2320 0x7fbbfc179130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.014+0000 7fbc09753640 1 -- 172.21.15.46:0/1393005451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc0d8a90 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fbc09753640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbfc0cc7b0 0x7fbbfc0e3050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc04076920 tx=0x7fbc04075a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc040bd030 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00b1874640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a4052fb0 0x7f00a416f970 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00ac06b8a0 tx=0x7f00ac06ebe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8019070 con 0x7f6af015d340 2024-05-09T03:09:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ae8014440 con 0x7f6af015d340 2024-05-09T03:09:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f81e37fe640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81e8076300 0x7f81e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f81f40b8a50 con 0x7f81fc15c160 2024-05-09T03:09:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac076290 con 0x7f00a4052fb0 2024-05-09T03:09:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f81e37fe640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f81e807e5d0 0x7f81e8080a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81e80810f0 con 0x7f81e807e5d0 2024-05-09T03:09:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f81f40bd2a0 con 0x7f81fc15c160 2024-05-09T03:09:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f81fad9d640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81e8076300 0x7f81e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbfc0c9d80 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbfc0ca340 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc040ad750 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f81fa59c640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f81e807e5d0 0x7f81e8080a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f7c911c7640 1 Processor -- start 2024-05-09T03:09:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc040b3040 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7f7c911c7640 1 -- start start 2024-05-09T03:09:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8014830 con 0x7f6af015d340 2024-05-09T03:09:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af0154e80 con 0x7f6af015d340 2024-05-09T03:09:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af0155440 con 0x7f6af015d340 2024-05-09T03:09:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fcf256fc640 1 -- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200c9650 msgr2=0x7fcf200cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fcf256fc640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf200c9650 0x7fcf200cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fcf256fc640 1 -- 172.21.15.46:0/711275942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20070600 msgr2=0x7fcf200c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fcf256fc640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf20070600 0x7fcf200c75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7fcf256fc640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf200c9a60 con 0x7fcf200c7b10 2024-05-09T03:09:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7fabaa1df640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c03a510 0x7fab9c0e2610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faba0001b10 tx=0x7faba0014420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7f81fad9d640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81e8076300 0x7f81e8078810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006fd0 tx=0x7f81ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.016+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00ac090070 con 0x7f00a4052fb0 2024-05-09T03:09:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac076670 con 0x7f00a4052fb0 2024-05-09T03:09:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a40e7450 con 0x7f00a4052fb0 2024-05-09T03:09:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.017+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a40e26a0 con 0x7f00a4052fb0 2024-05-09T03:09:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7fe581c9f640 1 Processor -- start 2024-05-09T03:09:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7fcf256fc640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf200c7b10 0x7fcf201fe3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf200d4b60 tx=0x7fcf1400b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.018+0000 7fe581c9f640 1 -- start start 2024-05-09T03:09:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf140125f0 con 0x7fcf200c7b10 2024-05-09T03:09:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6af0155820 con 0x7f6af015d340 2024-05-09T03:09:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf14018070 con 0x7fcf200c7b10 2024-05-09T03:09:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbbe4001030 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7f81fa59c640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f81e807e5d0 0x7f81e8080a20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f00a4170800 con 0x7f00a4052fb0 2024-05-09T03:09:40.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00ac095470 con 0x7f00a4052fb0 2024-05-09T03:09:40.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf140129f0 con 0x7fcf200c7b10 2024-05-09T03:09:40.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.019+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== osd.2 v2:172.21.15.46:6808/2404849282 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81e80810f0 con 0x7f81e807e5d0 2024-05-09T03:09:40.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0014b30 con 0x7fab9c03a510 2024-05-09T03:09:40.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba0014cd0 con 0x7fab9c03a510 2024-05-09T03:09:40.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f7c911c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0cee30 0x7f7c8c143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ae801c070 con 0x7f6af015d340 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f7c911c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c8a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0cee30 0x7f7c8c143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c911c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c8c1566b0 0x7f7c8c0b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c8a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0cee30 0x7f7c8c143580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55950/0 (socket says 172.21.15.46:55950) 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c8a59c640 1 -- 172.21.15.46:0/1071694924 learned_addr learned my addr 172.21.15.46:0/1071694924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c911c7640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c15de60 con 0x7f7c8c0cee30 2024-05-09T03:09:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c911c7640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c15d300 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c911c7640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c15dfe0 con 0x7f7c8c1566b0 2024-05-09T03:09:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0005ab0 con 0x7fab9c03a510 2024-05-09T03:09:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f00a2ffd640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00900763a0 0x7f00900788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.020+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f00ac113c90 con 0x7f00a4052fb0 2024-05-09T03:09:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f00a2ffd640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f009007e5f0 0x7f0090080a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf200c7f20 con 0x7fcf200c7b10 2024-05-09T03:09:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00ac096e50 con 0x7f009007e5f0 2024-05-09T03:09:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fabac443640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c0e2e70 con 0x7fab9c03a510 2024-05-09T03:09:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf200cbc40 con 0x7fcf200c7b10 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fabac443640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c0e3430 con 0x7fab9c03a510 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f6ae77fe640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ad0076320 0x7f6ad0078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6ae8097ff0 con 0x7f6af015d340 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f6ae77fe640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f6ad007e590 0x7f6ad00809e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f6aeeffd640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ad0076320 0x7f6ad0078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f6ae7fff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f6ad007e590 0x7f6ad00809e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ae8017e70 con 0x7f6ad007e590 2024-05-09T03:09:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6ae8098440 con 0x7f6af015d340 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f00ac0d9570 con 0x7f00a4052fb0 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f00b2876640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f009007e5f0 0x7f0090080a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f00b2075640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00900763a0 0x7f00900788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f00b2876640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f009007e5f0 0x7f0090080a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c8ad9d640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c8c1566b0 0x7f7c8c0b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7f7c89d9b640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f7c89d9b640 1 -- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c8c1566b0 msgr2=0x7f7c8c0b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f7c89d9b640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7c8c1566b0 0x7f7c8c0b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f7c89d9b640 1 -- 172.21.15.46:0/1071694924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0cee30 msgr2=0x7f7c8c143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f7c89d9b640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0cee30 0x7f7c8c143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f7c89d9b640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c143cf0 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fe581c9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5740e3db0 0x7fe5740330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fe581c9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5740e73a0 0x7fe5740324e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fe581c9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe574030c10 0x7fe574034ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fe581c9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe574058a80 con 0x7fe5740e73a0 2024-05-09T03:09:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f7c8a59c640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c8c0cee30 0x7f7c8c143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcf2006a270 con 0x7fcf200c7b10 2024-05-09T03:09:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f00b2075640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00900763a0 0x7f00900788b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00a8034000 tx=0x7f00a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fe57affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5740e73a0 0x7fe5740324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fe57affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5740e73a0 0x7fe5740324e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55964/0 (socket says 172.21.15.46:55964) 2024-05-09T03:09:40.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fe57affd640 1 -- 172.21.15.46:0/4033841701 learned_addr learned my addr 172.21.15.46:0/4033841701 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7f6aeeffd640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ad0076320 0x7f6ad0078830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000a6d0 tx=0x7f6ae0033000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fe57bfff640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe574030c10 0x7fe574034ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.021+0000 7fe581c9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740e81d0 con 0x7fe5740e3db0 2024-05-09T03:09:40.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe574058c00 con 0x7fe574030c10 2024-05-09T03:09:40.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fe57b7fe640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5740e3db0 0x7fe5740330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.022+0000 7fabac443640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab9c0054a0 con 0x7fab9c03a510 2024-05-09T03:09:40.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f6ae7fff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f6ad007e590 0x7f6ad00809e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbc040c0070 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f7c89d9b640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c157600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000d670 tx=0x7f7c8000db30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8001a070 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f00ac096e50 con 0x7f009007e5f0 2024-05-09T03:09:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8000c730 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8001a070 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fbbf9ffb640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbbe80762d0 0x7fbbe80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbc04134510 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fbbf9ffb640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7fbbe807e4c0 0x7fbbe8080910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc040af610 con 0x7fbbe807e4c0 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fbc0413b050 con 0x7fbbfc0cc7b0 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fbc08f52640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbbe80762d0 0x7fbbe80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ae8017e70 con 0x7f6ad007e590 2024-05-09T03:09:40.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf14012b90 con 0x7fcf200c7b10 2024-05-09T03:09:40.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f7c911c7640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8c143fd0 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7f7c911c7640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8c144590 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.024+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faba0005c50 con 0x7fab9c03a510 2024-05-09T03:09:40.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7fe57affd640 1 -- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe574030c10 msgr2=0x7fe574034ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7fe57affd640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe574030c10 0x7fe574034ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7fe57affd640 1 -- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5740e3db0 msgr2=0x7fe5740330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7fe57affd640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5740e3db0 0x7fe5740330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.023+0000 7fe57affd640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe574032a20 con 0x7fe5740e73a0 2024-05-09T03:09:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c8c06a270 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fe57affd640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5740e73a0 0x7fe5740324e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe56800b3f0 tx=0x7fe56800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fbc08f52640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbbe80762d0 0x7fbbe80787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0cc760 tx=0x7fbc000069c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fcf1d7fa640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf0c0763a0 0x7fcf0c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568015020 con 0x7fe5740e73a0 2024-05-09T03:09:40.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcf14097e60 con 0x7fcf200c7b10 2024-05-09T03:09:40.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe568011830 con 0x7fe5740e73a0 2024-05-09T03:09:40.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fab96ffd640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fab880763a0 0x7fab880788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568011c20 con 0x7fe5740e73a0 2024-05-09T03:09:40.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fcf24efb640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf0c0763a0 0x7fcf0c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faba009f400 con 0x7fab9c03a510 2024-05-09T03:09:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fcf1d7fa640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7fcf0c07e670 0x7fcf0c080ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf140067f0 con 0x7fcf0c07e670 2024-05-09T03:09:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.025+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fcf14098250 con 0x7fcf200c7b10 2024-05-09T03:09:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fab96ffd640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7fab8807e610 0x7fab88080a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faba0019e50 con 0x7fab8807e610 2024-05-09T03:09:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fabaa9e0640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7fab8807e610 0x7fab88080a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7faba00a6050 con 0x7fab9c03a510 2024-05-09T03:09:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe574031440 con 0x7fe5740e73a0 2024-05-09T03:09:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe574031a00 con 0x7fe5740e73a0 2024-05-09T03:09:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7fcf0c07e670 0x7fcf0c080ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.026+0000 7fcf24efb640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf0c0763a0 0x7fcf0c0788b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcf10001130 tx=0x7fcf10001080 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7fcf1ffff640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7fcf0c07e670 0x7fcf0c080ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe57403b2a0 con 0x7fe5740e73a0 2024-05-09T03:09:40.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7fbc09f54640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7fbbe807e4c0 0x7fbbe8080910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7fabaa9e0640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7fab8807e610 0x7fab88080a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== osd.4 v2:172.21.15.88:6824/902225877 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf140067f0 con 0x7fcf0c07e670 2024-05-09T03:09:40.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.028+0000 7fbc09f54640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7fbbe807e4c0 0x7fbbe8080910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7faba99de640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fab880763a0 0x7fab880788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faba0019e50 con 0x7fab8807e610 2024-05-09T03:09:40.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc040af610 con 0x7fbbe807e4c0 2024-05-09T03:09:40.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7faba99de640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fab880763a0 0x7fab880788b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fab98000f60 tx=0x7fab98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.027+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7c8001d070 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.028+0000 7f7c737fe640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7c780766d0 0x7f7c78078be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.028+0000 7f7c8a59c640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7c780766d0 0x7f7c78078be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.028+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7c80012070 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7f7c737fe640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f7c7807e920 0x7f7c78080d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c80017ec0 con 0x7f7c7807e920 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7c8009d050 con 0x7f7c8c0d0e30 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.029+0000 7f7c8ad9d640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f7c7807e920 0x7f7c78080d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.030+0000 7f7c8ad9d640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f7c7807e920 0x7f7c78080d70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.030+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe568018020 con 0x7fe5740e73a0 2024-05-09T03:09:40.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.030+0000 7f7c8a59c640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7c780766d0 0x7f7c78078be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0016a0 tx=0x7f7c7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c80017ec0 con 0x7f7c7807e920 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe567fff640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5600762d0 0x7fe5600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe568097b30 con 0x7fe5740e73a0 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe567fff640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fe56007e520 0x7fe560080970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe560081040 con 0x7fe56007e520 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe568097f20 con 0x7fe5740e73a0 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe57b7fe640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5600762d0 0x7fe5600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.031+0000 7fe57bfff640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fe56007e520 0x7fe560080970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.032+0000 7fe57b7fe640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5600762d0 0x7fe5600787e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe570009b30 tx=0x7fe570009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.032+0000 7fe57bfff640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fe56007e520 0x7fe560080970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.033+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe560081040 con 0x7fe56007e520 2024-05-09T03:09:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.051+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f81fc164c30 con 0x7f81e807e5d0 2024-05-09T03:09:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.052+0000 7f81e37fe640 1 -- 172.21.15.46:0/1936408700 <== osd.2 v2:172.21.15.46:6808/2404849282 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f81fc164c30 con 0x7f81e807e5d0 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.052+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f81e807e5d0 msgr2=0x7f81e8080a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.052+0000 7f820119d640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f81e807e5d0 0x7f81e8080a20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81e8076300 msgr2=0x7f81e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f820119d640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81e8076300 0x7f81e8078810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006fd0 tx=0x7f81ec008040 comp rx=0 tx=0).stop 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81fc15c160 msgr2=0x7f81fc170850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f820119d640 1 --2- 172.21.15.46:0/1936408700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81fc15c160 0x7f81fc170850 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009eb0 tx=0x7f81f4004a40 comp rx=0 tx=0).stop 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f81fb59e640 1 -- 172.21.15.46:0/1936408700 reap_dead start 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 shutdown_connections 2024-05-09T03:09:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.053+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 >> 172.21.15.46:0/1936408700 conn(0x7f81fc0ce370 msgr2=0x7f81fc0731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.054+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 shutdown_connections 2024-05-09T03:09:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.057+0000 7f820119d640 1 -- 172.21.15.46:0/1936408700 wait complete. 2024-05-09T03:09:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.057+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6af0154950 con 0x7f6ad007e590 2024-05-09T03:09:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.058+0000 7f6ae77fe640 1 -- 172.21.15.46:0/2419203415 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6af0154950 con 0x7f6ad007e590 2024-05-09T03:09:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.058+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f00a40e4c90 con 0x7f009007e5f0 2024-05-09T03:09:40.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.058+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f6ad007e590 msgr2=0x7f6ad00809e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.058+0000 7f6aeffff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f6ad007e590 0x7f6ad00809e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.058+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ad0076320 msgr2=0x7f6ad0078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.058+0000 7f6aeffff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ad0076320 0x7f6ad0078830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000a6d0 tx=0x7f6ae0033000 comp rx=0 tx=0).stop 2024-05-09T03:09:40.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c8c0b49a0 con 0x7f7c7807e920 2024-05-09T03:09:40.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 msgr2=0x7f6af00735a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aeffff640 1 --2- 172.21.15.46:0/2419203415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6af015d340 0x7f6af00735a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800be50 tx=0x7f6ae800bf40 comp rx=0 tx=0).stop 2024-05-09T03:09:40.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aef7fe640 1 -- 172.21.15.46:0/2419203415 reap_dead start 2024-05-09T03:09:40.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 shutdown_connections 2024-05-09T03:09:40.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f00a2ffd640 1 -- 172.21.15.46:0/4101326214 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f00a40e4c90 con 0x7f009007e5f0 2024-05-09T03:09:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 >> 172.21.15.46:0/2419203415 conn(0x7f6af00b3910 msgr2=0x7f6af00b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 shutdown_connections 2024-05-09T03:09:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f6aeffff640 1 -- 172.21.15.46:0/2419203415 wait complete. 2024-05-09T03:09:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f009007e5f0 msgr2=0x7f0090080a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f00b42d9640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f009007e5f0 0x7f0090080a40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00900763a0 msgr2=0x7f00900788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f00b42d9640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00900763a0 0x7f00900788b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f00a8034000 tx=0x7f00a802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a4052fb0 msgr2=0x7f00a416f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b42d9640 1 --2- 172.21.15.46:0/4101326214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a4052fb0 0x7f00a416f970 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00ac06b8a0 tx=0x7f00ac06ebe0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b2876640 1 -- 172.21.15.46:0/4101326214 reap_dead start 2024-05-09T03:09:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 shutdown_connections 2024-05-09T03:09:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 >> 172.21.15.46:0/4101326214 conn(0x7f00a4054cf0 msgr2=0x7f00a416e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.059+0000 7f7c737fe640 1 -- 172.21.15.46:0/1071694924 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7c8c0b49a0 con 0x7f7c7807e920 2024-05-09T03:09:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 shutdown_connections 2024-05-09T03:09:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f00b42d9640 1 -- 172.21.15.46:0/4101326214 wait complete. 2024-05-09T03:09:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f7c7807e920 msgr2=0x7f7c78080d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.060+0000 7f7c717fa640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f7c7807e920 0x7f7c78080d70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7c780766d0 msgr2=0x7f7c78078be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c717fa640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7c780766d0 0x7f7c78078be0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0016a0 tx=0x7f7c7c009040 comp rx=0 tx=0).stop 2024-05-09T03:09:40.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c8c0d0e30 msgr2=0x7f7c8c157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c717fa640 1 --2- 172.21.15.46:0/1071694924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c8c0d0e30 0x7f7c8c157600 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000d670 tx=0x7f7c8000db30 comp rx=0 tx=0).stop 2024-05-09T03:09:40.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c8ad9d640 1 -- 172.21.15.46:0/1071694924 reap_dead start 2024-05-09T03:09:40.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.062+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcf200b97b0 con 0x7fcf0c07e670 2024-05-09T03:09:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 shutdown_connections 2024-05-09T03:09:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.061+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 >> 172.21.15.46:0/1071694924 conn(0x7f7c8c0732c0 msgr2=0x7f7c8c0736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.062+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 shutdown_connections 2024-05-09T03:09:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.062+0000 7f7c717fa640 1 -- 172.21.15.46:0/1071694924 wait complete. 2024-05-09T03:09:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.062+0000 7fabac443640 1 -- 172.21.15.46:0/2783958765 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fab9c0db850 con 0x7fab8807e610 2024-05-09T03:09:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbbe4002dd0 con 0x7fbbe807e4c0 2024-05-09T03:09:40.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fcf1d7fa640 1 -- 172.21.15.46:0/711275942 <== osd.4 v2:172.21.15.88:6824/902225877 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcf200b97b0 con 0x7fcf0c07e670 2024-05-09T03:09:40.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fab96ffd640 1 -- 172.21.15.46:0/2783958765 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fab9c0db850 con 0x7fab8807e610 2024-05-09T03:09:40.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fbbf9ffb640 1 -- 172.21.15.46:0/1393005451 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbbe4002dd0 con 0x7fbbe807e4c0 2024-05-09T03:09:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7fcf0c07e670 msgr2=0x7fcf0c080ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fcf2715f640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7fcf0c07e670 0x7fcf0c080ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7fbbe807e4c0 msgr2=0x7fbbe8080910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.063+0000 7fbc0b9b7640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7fbbe807e4c0 0x7fbbe8080910 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf0c0763a0 msgr2=0x7fcf0c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf2715f640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf0c0763a0 0x7fcf0c0788b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcf10001130 tx=0x7fcf10001080 comp rx=0 tx=0).stop 2024-05-09T03:09:40.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf200c7b10 msgr2=0x7fcf201fe3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf2715f640 1 --2- 172.21.15.46:0/711275942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf200c7b10 0x7fcf201fe3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcf200d4b60 tx=0x7fcf1400b880 comp rx=0 tx=0).stop 2024-05-09T03:09:40.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbbe80762d0 msgr2=0x7fbbe80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf256fc640 1 -- 172.21.15.46:0/711275942 reap_dead start 2024-05-09T03:09:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc0b9b7640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbbe80762d0 0x7fbbe80787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc0cc760 tx=0x7fbc000069c0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbfc0cc7b0 msgr2=0x7fbbfc0e3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc0b9b7640 1 --2- 172.21.15.46:0/1393005451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbfc0cc7b0 0x7fbbfc0e3050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbc04076920 tx=0x7fbc04075a10 comp rx=0 tx=0).stop 2024-05-09T03:09:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 shutdown_connections 2024-05-09T03:09:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc09f54640 1 -- 172.21.15.46:0/1393005451 reap_dead start 2024-05-09T03:09:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 >> 172.21.15.46:0/711275942 conn(0x7fcf200d3b90 msgr2=0x7fcf200b3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 shutdown_connections 2024-05-09T03:09:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.064+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 >> 172.21.15.46:0/1393005451 conn(0x7fbbfc056f60 msgr2=0x7fbbfc036cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.065+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe5740322a0 con 0x7fe56007e520 2024-05-09T03:09:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.065+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7fab8807e610 msgr2=0x7fab88080a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.065+0000 7fab94ff9640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7fab8807e610 0x7fab88080a60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.065+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 shutdown_connections 2024-05-09T03:09:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fab880763a0 msgr2=0x7fab880788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fab880763a0 0x7fab880788b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fab98000f60 tx=0x7fab98009040 comp rx=0 tx=0).stop 2024-05-09T03:09:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fcf2715f640 1 -- 172.21.15.46:0/711275942 wait complete. 2024-05-09T03:09:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c03a510 msgr2=0x7fab9c0e2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 --2- 172.21.15.46:0/2783958765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab9c03a510 0x7fab9c0e2610 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faba0001b10 tx=0x7faba0014420 comp rx=0 tx=0).stop 2024-05-09T03:09:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fabaa9e0640 1 -- 172.21.15.46:0/2783958765 reap_dead start 2024-05-09T03:09:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 shutdown_connections 2024-05-09T03:09:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fbc0b9b7640 1 -- 172.21.15.46:0/1393005451 wait complete. 2024-05-09T03:09:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 shutdown_connections 2024-05-09T03:09:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 >> 172.21.15.46:0/2783958765 conn(0x7fab9c054cf0 msgr2=0x7fab9c004940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 shutdown_connections 2024-05-09T03:09:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fab94ff9640 1 -- 172.21.15.46:0/2783958765 wait complete. 2024-05-09T03:09:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.066+0000 7fe567fff640 1 -- 172.21.15.46:0/4033841701 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe5740322a0 con 0x7fe56007e520 2024-05-09T03:09:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.067+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fe56007e520 msgr2=0x7fe560080970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.067+0000 7fe581c9f640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fe56007e520 0x7fe560080970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5600762d0 msgr2=0x7fe5600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe581c9f640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5600762d0 0x7fe5600787e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe570009b30 tx=0x7fe570009240 comp rx=0 tx=0).stop 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5740e73a0 msgr2=0x7fe5740324e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe581c9f640 1 --2- 172.21.15.46:0/4033841701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5740e73a0 0x7fe5740324e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe56800b3f0 tx=0x7fe56800b8b0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe57bfff640 1 -- 172.21.15.46:0/4033841701 reap_dead start 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 shutdown_connections 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.068+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 >> 172.21.15.46:0/4033841701 conn(0x7fe57403ac60 msgr2=0x7fe574038df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.069+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 shutdown_connections 2024-05-09T03:09:40.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.069+0000 7fe581c9f640 1 -- 172.21.15.46:0/4033841701 wait complete. 2024-05-09T03:09:40.120 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:40.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T03:09:40.121 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.121 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T03:09:40.122 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-05-09T03:09:40.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T03:09:40.123 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:40.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T03:09:40.124 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T03:09:40.125 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:40.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T03:09:40.126 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:40.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T03:09:40.127 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:40.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T03:09:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.385+0000 7f96f7fff640 1 Processor -- start 2024-05-09T03:09:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.385+0000 7f96f7fff640 1 -- start start 2024-05-09T03:09:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7f96f7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96f004fb30 0x7f96f0104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7f96f7fff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f0050440 con 0x7f96f0105440 2024-05-09T03:09:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7f96f7fff640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f0054700 con 0x7f96f0105850 2024-05-09T03:09:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7f96f7fff640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f96f0034250 con 0x7f96f004fb30 2024-05-09T03:09:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7f96f67fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96f0105440 0x7f96f00538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45612/0 (socket says 172.21.15.46:45612) 2024-05-09T03:09:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7f96f67fc640 1 -- 172.21.15.46:0/4185579349 learned_addr learned my addr 172.21.15.46:0/4185579349 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/4185579349 >> v2:172.21.15.88:3300/0 conn(0x7f96f004fb30 0x7f96f0104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477807925 0 0) 0x7f96f0054700 con 0x7f96f0105850 2024-05-09T03:09:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96e0003520 con 0x7f96f0105850 2024-05-09T03:09:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3092518223 0 0) 0x7f96f0050440 con 0x7f96f0105440 2024-05-09T03:09:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96f0054700 con 0x7f96f0105440 2024-05-09T03:09:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7fa694f27640 1 Processor -- start 2024-05-09T03:09:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7fa694f27640 1 -- start start 2024-05-09T03:09:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.387+0000 7fe94abb4640 1 Processor -- start 2024-05-09T03:09:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7fe94abb4640 1 -- start start 2024-05-09T03:09:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7fbf49d3f640 1 Processor -- start 2024-05-09T03:09:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.388+0000 7fbf49d3f640 1 -- start start 2024-05-09T03:09:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3744028658 0 0) 0x7f96e0003520 con 0x7f96f0105850 2024-05-09T03:09:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 >> v2:172.21.15.88:3300/0 conn(0x7f96f004fb30 msgr2=0x7f96f0104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 --2- 172.21.15.46:0/4185579349 >> v2:172.21.15.88:3300/0 conn(0x7f96f004fb30 0x7f96f0104f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 >> v1:172.21.15.46:6789/0 conn(0x7f96f0105440 legacy=0x7f96f00538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f0050710 con 0x7f96f0105850 2024-05-09T03:09:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f96f8076c00 con 0x7f96f0105850 2024-05-09T03:09:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 >> v1:172.21.15.88:6789/0 conn(0x7f96f0105850 legacy=0x7f96f0053ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96e0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e0004800 0x7f96e0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96e0005150 0x7f96e00145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96e0014b00 con 0x7f96e0003b40 2024-05-09T03:09:40.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96e0014c80 con 0x7f96e0005150 2024-05-09T03:09:40.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.389+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96e0014e00 con 0x7f96e0004800 2024-05-09T03:09:40.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7f14262cb640 1 Processor -- start 2024-05-09T03:09:40.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7f96f77fe640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96e0005150 0x7f96e00145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7fbf49d3f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c04f330 0x7fbf3c0e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7fbf49d3f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbf3c048310 0x7fbf3c0382a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7fbf49d3f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf3c0e80c0 con 0x7fbf3c033940 2024-05-09T03:09:40.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7fbf49d3f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3c053200 con 0x7fbf3c048310 2024-05-09T03:09:40.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7fbf49d3f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c047dc0 con 0x7fbf3c04f330 2024-05-09T03:09:40.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7f14262cb640 1 -- start start 2024-05-09T03:09:40.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7efcc50cb640 1 Processor -- start 2024-05-09T03:09:40.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7efcc50cb640 1 -- start start 2024-05-09T03:09:40.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe94abb4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 0x7fe9440b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96e0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.390+0000 7f96f67fc640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e0004800 0x7f96e0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f96f6ffd640 1 -- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e0004800 msgr2=0x7f96e0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e0004800 0x7f96e0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f96f6ffd640 1 -- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96e0005150 msgr2=0x7f96e00145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96e0005150 0x7f96e00145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f96f6ffd640 1 -- 172.21.15.46:0/4185579349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e0003520 con 0x7f96e0003b40 2024-05-09T03:09:40.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f96f77fe640 1 -- 172.21.15.46:0/4185579349 reap_dead start 2024-05-09T03:09:40.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe94abb4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9440b5220 0x7fe9440b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe943fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 0x7fe9440b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbaf908b640 1 Processor -- start 2024-05-09T03:09:40.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe94abb4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe944160ba0 con 0x7fe944160790 2024-05-09T03:09:40.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe943fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 0x7fe9440b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43332/0 (socket says 172.21.15.46:43332) 2024-05-09T03:09:40.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe943fff640 1 -- 172.21.15.46:0/287343054 learned_addr learned my addr 172.21.15.46:0/287343054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fe949151640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9440b5220 0x7fe9440b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe94415cbc0 con 0x7fe9440b5220 2024-05-09T03:09:40.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe94415cea0 con 0x7fe9440739e0 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fa694f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6880512b0 0x7fa688007e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fa694f27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa688054090 0x7fa688008360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fa694f27640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa688054590 con 0x7fa6880517d0 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fa694f27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa688054870 con 0x7fa688054090 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fa694f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa688054b50 con 0x7fa6880512b0 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fa68e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6880517d0 0x7fa688050ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45636/0 (socket says 172.21.15.46:45636) 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fa68e59c640 1 -- 172.21.15.46:0/101435647 learned_addr learned my addr 172.21.15.46:0/101435647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fa68ed9d640 1 --2- 172.21.15.46:0/101435647 >> v2:172.21.15.88:3300/0 conn(0x7fa688054090 0x7fa688008360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbaf908b640 1 -- start start 2024-05-09T03:09:40.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f1294db3640 1 Processor -- start 2024-05-09T03:09:40.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7f1294db3640 1 -- start start 2024-05-09T03:09:40.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbf43fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbf3c033940 0x7fbf3c033d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:35174/0 (socket says 172.21.15.46:35174) 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbf43fff640 1 -- 172.21.15.46:0/3977860979 learned_addr learned my addr 172.21.15.46:0/3977860979 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbf42ffd640 1 --2- 172.21.15.46:0/3977860979 >> v2:172.21.15.88:3300/0 conn(0x7fbf3c048310 0x7fbf3c0382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbf437fe640 1 --2- 172.21.15.46:0/3977860979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c04f330 0x7fbf3c0e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbf427fc640 1 -- 172.21.15.46:0/3977860979 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346044020 0 0) 0x7fbf3c0e80c0 con 0x7fbf3c033940 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.391+0000 7fbf427fc640 1 -- 172.21.15.46:0/3977860979 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf34003520 con 0x7fbf3c033940 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fbf437fe640 1 -- 172.21.15.46:0/3977860979 >> v2:172.21.15.88:3300/0 conn(0x7fbf3c048310 msgr2=0x7fbf3c0382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fbf437fe640 1 --2- 172.21.15.46:0/3977860979 >> v2:172.21.15.88:3300/0 conn(0x7fbf3c048310 0x7fbf3c0382a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fbf437fe640 1 -- 172.21.15.46:0/3977860979 >> v1:172.21.15.88:6789/0 conn(0x7fbf3c033940 legacy=0x7fbf3c033d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fbf437fe640 1 -- 172.21.15.46:0/3977860979 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c048010 con 0x7fbf3c04f330 2024-05-09T03:09:40.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fe943fff640 1 -- 172.21.15.46:0/287343054 >> v2:172.21.15.88:3300/0 conn(0x7fe9440b5220 msgr2=0x7fe9440b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fe943fff640 1 --2- 172.21.15.46:0/287343054 >> v2:172.21.15.88:3300/0 conn(0x7fe9440b5220 0x7fe9440b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fe943fff640 1 -- 172.21.15.46:0/287343054 >> v1:172.21.15.46:6789/0 conn(0x7fe944160790 legacy=0x7fe9440732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fe943fff640 1 -- 172.21.15.46:0/287343054 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94415d180 con 0x7fe9440739e0 2024-05-09T03:09:40.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.392+0000 7fbf437fe640 1 --2- 172.21.15.46:0/3977860979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c04f330 0x7fbf3c0e7b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf380097a0 tx=0x7fbf3802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=784d5313d7df036d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fbf427fc640 1 -- 172.21.15.46:0/3977860979 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf38037070 con 0x7fbf3c04f330 2024-05-09T03:09:40.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fbf427fc640 1 -- 172.21.15.46:0/3977860979 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf38032440 con 0x7fbf3c04f330 2024-05-09T03:09:40.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fbf427fc640 1 -- 172.21.15.46:0/3977860979 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf38032810 con 0x7fbf3c04f330 2024-05-09T03:09:40.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fa68ed9d640 1 -- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6880512b0 msgr2=0x7fa688007e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T03:09:40.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fa68ed9d640 1 --2- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6880512b0 0x7fa688007e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fa68ed9d640 1 -- 172.21.15.46:0/101435647 >> v1:172.21.15.46:6789/0 conn(0x7fa6880517d0 legacy=0x7fa688050ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7fa68ed9d640 1 -- 172.21.15.46:0/101435647 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa688054e30 con 0x7fa688054090 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f14262cb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 0x7f14180485a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f14262cb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1418054170 0x7f1418006ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f14262cb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1418053260 con 0x7f1418053650 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f14262cb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f14180e7fa0 con 0x7f1418054170 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f14262cb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1418034810 con 0x7f14180382f0 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc50cb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcc00b57c0 0x7efcc006f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc50cb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efcc00b0db0 0x7efcc0153120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc50cb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcc015d4a0 con 0x7efcc00712b0 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc50cb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7efcc0153660 con 0x7efcc00b57c0 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f1294db3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f129006f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f1294db3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12900b57c0 0x7f12900b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f1294db3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12900b6b00 con 0x7f1290153300 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f1294db3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f12900b12f0 con 0x7f12900b57c0 2024-05-09T03:09:40.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f1294db3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f129015d4a0 con 0x7f12900712b0 2024-05-09T03:09:40.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc50cb640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efcc00b64b0 con 0x7efcc00b0db0 2024-05-09T03:09:40.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7f14252c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 0x7f14180485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12900b57c0 0x7f12900b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf908b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40d06a0 0x7fbaf40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f14252c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 0x7f14180485a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43336/0 (socket says 172.21.15.46:43336) 2024-05-09T03:09:40.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f14252c9640 1 -- 172.21.15.46:0/476425779 learned_addr learned my addr 172.21.15.46:0/476425779 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f1425aca640 1 --2- 172.21.15.46:0/476425779 >> v2:172.21.15.46:3300/0 conn(0x7f1418054170 0x7f1418006ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc48ca640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efcc00b0db0 0x7efcc0153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fe943fff640 1 --2- 172.21.15.46:0/287343054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 0x7fe9440b4ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe930007040 tx=0x7fe93002b1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ca8d1ecac6090d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f129006f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf908b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaf40ce340 0x7fbaf40b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f129006f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43342/0 (socket says 172.21.15.46:43342) 2024-05-09T03:09:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 -- 172.21.15.46:0/4135461958 learned_addr learned my addr 172.21.15.46:0/4135461958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf908b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaf40d0ab0 con 0x7fbaf40cf4a0 2024-05-09T03:09:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96e0003f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009aa0 tx=0x7f96e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c615c11c64566a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf908b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbaf40ceab0 con 0x7fbaf40ce340 2024-05-09T03:09:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf908b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbaf40cfc30 con 0x7fbaf40d06a0 2024-05-09T03:09:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf2d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40d06a0 0x7fbaf40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaf40ce340 0x7fbaf40b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaf40ce340 0x7fbaf40b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56850/0 (socket says 172.21.15.46:56850) 2024-05-09T03:09:40.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbaf359e640 1 -- 172.21.15.46:0/1354739943 learned_addr learned my addr 172.21.15.46:0/1354739943 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 -- 172.21.15.46:0/3977860979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c04f330 msgr2=0x7fbf3c0e7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/3977860979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c04f330 0x7fbf3c0e7b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf380097a0 tx=0x7fbf3802a180 comp rx=0 tx=0).stop 2024-05-09T03:09:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 -- 172.21.15.46:0/3977860979 shutdown_connections 2024-05-09T03:09:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/3977860979 >> v2:172.21.15.88:3300/0 conn(0x7fbf3c048310 0x7fbf3c0382a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/3977860979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c04f330 0x7fbf3c0e7b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 -- 172.21.15.46:0/3977860979 >> 172.21.15.46:0/3977860979 conn(0x7fbf3c054ce0 msgr2=0x7fbf3c0550f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 -- 172.21.15.46:0/3977860979 shutdown_connections 2024-05-09T03:09:40.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fbf49d3f640 1 -- 172.21.15.46:0/3977860979 wait complete. 2024-05-09T03:09:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fe9437fe640 1 -- 172.21.15.46:0/287343054 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9300078d0 con 0x7fe9440739e0 2024-05-09T03:09:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fe9437fe640 1 -- 172.21.15.46:0/287343054 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe930007a70 con 0x7fe9440739e0 2024-05-09T03:09:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 -- 172.21.15.46:0/4135461958 >> v2:172.21.15.88:3300/0 conn(0x7f12900b57c0 msgr2=0x7f12900b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 --2- 172.21.15.46:0/4135461958 >> v2:172.21.15.88:3300/0 conn(0x7f12900b57c0 0x7f12900b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 -- 172.21.15.46:0/4135461958 >> v1:172.21.15.46:6789/0 conn(0x7f1290153300 legacy=0x7f12900b50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f128f7fe640 1 -- 172.21.15.46:0/4135461958 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12900b64b0 con 0x7f12900712b0 2024-05-09T03:09:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7fe9437fe640 1 -- 172.21.15.46:0/287343054 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930007e40 con 0x7fe9440739e0 2024-05-09T03:09:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec038070 con 0x7f96e0003b40 2024-05-09T03:09:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ec033410 con 0x7f96e0003b40 2024-05-09T03:09:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7f96f5ffb640 1 -- 172.21.15.46:0/4185579349 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec009720 con 0x7f96e0003b40 2024-05-09T03:09:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc48ca640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efcc00b0db0 0x7efcc0153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56844/0 (socket says 172.21.15.46:56844) 2024-05-09T03:09:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.393+0000 7efcc48ca640 1 -- 172.21.15.46:0/4043952781 learned_addr learned my addr 172.21.15.46:0/4043952781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7efcc48ca640 1 -- 172.21.15.46:0/4043952781 >> v2:172.21.15.46:3300/0 conn(0x7efcc00b57c0 msgr2=0x7efcc006f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7efcc48ca640 1 --2- 172.21.15.46:0/4043952781 >> v2:172.21.15.46:3300/0 conn(0x7efcc00b57c0 0x7efcc006f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7efcc48ca640 1 -- 172.21.15.46:0/4043952781 >> v1:172.21.15.88:6789/0 conn(0x7efcc00712b0 legacy=0x7efcc00b50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fa68ed9d640 1 --2- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa688054090 0x7fa688008360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6800097a0 tx=0x7fa68002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266f3904065431e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7efcc48ca640 1 -- 172.21.15.46:0/4043952781 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc00b6790 con 0x7efcc00b0db0 2024-05-09T03:09:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f14252c9640 1 -- 172.21.15.46:0/476425779 >> v2:172.21.15.46:3300/0 conn(0x7f1418054170 msgr2=0x7f1418006ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f14252c9640 1 --2- 172.21.15.46:0/476425779 >> v2:172.21.15.46:3300/0 conn(0x7f1418054170 0x7f1418006ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f14252c9640 1 -- 172.21.15.46:0/476425779 >> v1:172.21.15.46:6789/0 conn(0x7f1418053650 legacy=0x7f1418053a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.394+0000 7f14252c9640 1 -- 172.21.15.46:0/476425779 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1418050330 con 0x7f14180382f0 2024-05-09T03:09:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 msgr2=0x7fe9440b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fe94abb4640 1 --2- 172.21.15.46:0/287343054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 0x7fe9440b4ce0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe930007040 tx=0x7fe93002b1b0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fbaf359e640 1 -- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40d06a0 msgr2=0x7fbaf40cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:40.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fbaf359e640 1 --2- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40d06a0 0x7fbaf40cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fbaf359e640 1 -- 172.21.15.46:0/1354739943 >> v1:172.21.15.46:6789/0 conn(0x7fbaf40cf4a0 legacy=0x7fbaf40b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fbaf359e640 1 -- 172.21.15.46:0/1354739943 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf40d01b0 con 0x7fbaf40ce340 2024-05-09T03:09:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7fbaf2d9d640 1 --2- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40d06a0 0x7fbaf40cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f128f7fe640 1 --2- 172.21.15.46:0/4135461958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f129006f220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1280009870 tx=0x7f128002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4de9e1f6343d89a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.395+0000 7efcc48ca640 1 --2- 172.21.15.46:0/4043952781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0153120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efcc0074640 tx=0x7efcb002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f1ef37f15913e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 shutdown_connections 2024-05-09T03:09:40.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fa68d59a640 1 -- 172.21.15.46:0/101435647 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680037070 con 0x7fa688054090 2024-05-09T03:09:40.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fe94abb4640 1 --2- 172.21.15.46:0/287343054 >> v2:172.21.15.88:3300/0 conn(0x7fe9440b5220 0x7fe9440b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fe94abb4640 1 --2- 172.21.15.46:0/287343054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440739e0 0x7fe9440b4ce0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 >> 172.21.15.46:0/287343054 conn(0x7fe9440ce370 msgr2=0x7fe9440ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f14252c9640 1 --2- 172.21.15.46:0/476425779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 0x7f14180485a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1410009870 tx=0x7f141002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43e6c390b6f398d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f128e7fc640 1 -- 172.21.15.46:0/4135461958 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1280037070 con 0x7f12900712b0 2024-05-09T03:09:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 -- 172.21.15.46:0/4185579349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 msgr2=0x7f96e0003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96e0003f90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009aa0 tx=0x7f96e0005830 comp rx=0 tx=0).stop 2024-05-09T03:09:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7efcbeffd640 1 -- 172.21.15.46:0/4043952781 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0037070 con 0x7efcc00b0db0 2024-05-09T03:09:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7efcbeffd640 1 -- 172.21.15.46:0/4043952781 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb0032440 con 0x7efcc00b0db0 2024-05-09T03:09:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7efcbeffd640 1 -- 172.21.15.46:0/4043952781 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0032810 con 0x7efcc00b0db0 2024-05-09T03:09:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f1417fff640 1 -- 172.21.15.46:0/476425779 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410037070 con 0x7f14180382f0 2024-05-09T03:09:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f1417fff640 1 -- 172.21.15.46:0/476425779 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1410032440 con 0x7f14180382f0 2024-05-09T03:09:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f1417fff640 1 -- 172.21.15.46:0/476425779 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1410032810 con 0x7f14180382f0 2024-05-09T03:09:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fa68d59a640 1 -- 172.21.15.46:0/101435647 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa680032440 con 0x7fa688054090 2024-05-09T03:09:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 shutdown_connections 2024-05-09T03:09:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 -- 172.21.15.46:0/4185579349 shutdown_connections 2024-05-09T03:09:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 --2- 172.21.15.46:0/4185579349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96e0003f90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 -- 172.21.15.46:0/4185579349 >> 172.21.15.46:0/4185579349 conn(0x7f96f0054cf0 msgr2=0x7f96f0055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7fe94abb4640 1 -- 172.21.15.46:0/287343054 wait complete. 2024-05-09T03:09:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 -- 172.21.15.46:0/4185579349 shutdown_connections 2024-05-09T03:09:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f128e7fc640 1 -- 172.21.15.46:0/4135461958 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1280032440 con 0x7f12900712b0 2024-05-09T03:09:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f128e7fc640 1 -- 172.21.15.46:0/4135461958 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1280037070 con 0x7f12900712b0 2024-05-09T03:09:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f1294db3640 1 -- 172.21.15.46:0/4135461958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 msgr2=0x7f129006f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f1294db3640 1 --2- 172.21.15.46:0/4135461958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f129006f220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1280009870 tx=0x7f128002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.396+0000 7f96f7fff640 1 -- 172.21.15.46:0/4185579349 wait complete. 2024-05-09T03:09:40.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 -- 172.21.15.46:0/4043952781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 msgr2=0x7efcc0153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 --2- 172.21.15.46:0/4043952781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0153120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efcc0074640 tx=0x7efcb002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7fbf49d3f640 1 Processor -- start 2024-05-09T03:09:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 -- 172.21.15.46:0/4043952781 shutdown_connections 2024-05-09T03:09:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 --2- 172.21.15.46:0/4043952781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0153120 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 --2- 172.21.15.46:0/4043952781 >> v2:172.21.15.46:3300/0 conn(0x7efcc00b57c0 0x7efcc006f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7fa68d59a640 1 -- 172.21.15.46:0/101435647 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680032810 con 0x7fa688054090 2024-05-09T03:09:40.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7fa694f27640 1 -- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa688054090 msgr2=0x7fa688008360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7fa694f27640 1 --2- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa688054090 0x7fa688008360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6800097a0 tx=0x7fa68002a180 comp rx=0 tx=0).stop 2024-05-09T03:09:40.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 -- 172.21.15.46:0/4043952781 >> 172.21.15.46:0/4043952781 conn(0x7efcc006c680 msgr2=0x7efcc006ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f14262cb640 1 -- 172.21.15.46:0/476425779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 msgr2=0x7f14180485a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f14262cb640 1 --2- 172.21.15.46:0/476425779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 0x7f14180485a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1410009870 tx=0x7f141002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 -- 172.21.15.46:0/4043952781 shutdown_connections 2024-05-09T03:09:40.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f1294db3640 1 -- 172.21.15.46:0/4135461958 shutdown_connections 2024-05-09T03:09:40.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f1294db3640 1 --2- 172.21.15.46:0/4135461958 >> v2:172.21.15.88:3300/0 conn(0x7f12900b57c0 0x7f12900b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f1294db3640 1 --2- 172.21.15.46:0/4135461958 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f129006f220 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f1294db3640 1 -- 172.21.15.46:0/4135461958 >> 172.21.15.46:0/4135461958 conn(0x7f129006c680 msgr2=0x7f129006ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7fbaf359e640 1 --2- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40ce340 0x7fbaf40b64f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009e50 tx=0x7fbaec02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=537405d37e8645d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f14262cb640 1 -- 172.21.15.46:0/476425779 shutdown_connections 2024-05-09T03:09:40.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7f14262cb640 1 --2- 172.21.15.46:0/476425779 >> v2:172.21.15.46:3300/0 conn(0x7f1418054170 0x7f1418006ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.397+0000 7efcc50cb640 1 -- 172.21.15.46:0/4043952781 wait complete. 2024-05-09T03:09:40.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f14262cb640 1 --2- 172.21.15.46:0/476425779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f14180382f0 0x7f14180485a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f14262cb640 1 -- 172.21.15.46:0/476425779 >> 172.21.15.46:0/476425779 conn(0x7f1418055130 msgr2=0x7f1418055540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fa694f27640 1 -- 172.21.15.46:0/101435647 shutdown_connections 2024-05-09T03:09:40.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fa694f27640 1 --2- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa688054090 0x7fa688008360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fa694f27640 1 --2- 172.21.15.46:0/101435647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6880512b0 0x7fa688007e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fa694f27640 1 -- 172.21.15.46:0/101435647 >> 172.21.15.46:0/101435647 conn(0x7fa688055df0 msgr2=0x7fa688056200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fbf49d3f640 1 -- start start 2024-05-09T03:09:40.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f1294db3640 1 -- 172.21.15.46:0/4135461958 shutdown_connections 2024-05-09T03:09:40.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f1294db3640 1 -- 172.21.15.46:0/4135461958 wait complete. 2024-05-09T03:09:40.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f14262cb640 1 -- 172.21.15.46:0/476425779 shutdown_connections 2024-05-09T03:09:40.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fa694f27640 1 -- 172.21.15.46:0/101435647 shutdown_connections 2024-05-09T03:09:40.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7fa694f27640 1 -- 172.21.15.46:0/101435647 wait complete. 2024-05-09T03:09:40.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.398+0000 7f14262cb640 1 -- 172.21.15.46:0/476425779 wait complete. 2024-05-09T03:09:40.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.399+0000 7fbaf1d9b640 1 -- 172.21.15.46:0/1354739943 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec036070 con 0x7fbaf40ce340 2024-05-09T03:09:40.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.399+0000 7fbaf1d9b640 1 -- 172.21.15.46:0/1354739943 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec031600 con 0x7fbaf40ce340 2024-05-09T03:09:40.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.399+0000 7fbaf1d9b640 1 -- 172.21.15.46:0/1354739943 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0319d0 con 0x7fbaf40ce340 2024-05-09T03:09:40.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fe94abb4640 1 Processor -- start 2024-05-09T03:09:40.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7f96f7fff640 1 Processor -- start 2024-05-09T03:09:40.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf49d3f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3c033940 0x7fbf3c0d8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fe94abb4640 1 -- start start 2024-05-09T03:09:40.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf49d3f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 0x7fbf3c0438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7efcc50cb640 1 Processor -- start 2024-05-09T03:09:40.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbaf908b640 1 -- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40ce340 msgr2=0x7fbaf40b64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbaf908b640 1 --2- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40ce340 0x7fbaf40b64f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009e50 tx=0x7fbaec02a680 comp rx=0 tx=0).stop 2024-05-09T03:09:40.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf49d3f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf3c04f330 0x7fbf3c046190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf42ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 0x7fbf3c0438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7efcc50cb640 1 -- start start 2024-05-09T03:09:40.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf42ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 0x7fbf3c0438e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43360/0 (socket says 172.21.15.46:43360) 2024-05-09T03:09:40.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf49d3f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c006450 con 0x7fbf3c033940 2024-05-09T03:09:40.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf42ffd640 1 -- 172.21.15.46:0/2947822368 learned_addr learned my addr 172.21.15.46:0/2947822368 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3c006aa0 con 0x7fbf3c04f330 2024-05-09T03:09:40.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7f1294db3640 1 Processor -- start 2024-05-09T03:09:40.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbf3c006730 con 0x7fbf3c048310 2024-05-09T03:09:40.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7fbf43fff640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf3c04f330 0x7fbf3c046190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbf42ffd640 1 -- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf3c04f330 msgr2=0x7fbf3c046190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbf42ffd640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf3c04f330 0x7fbf3c046190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbaf908b640 1 -- 172.21.15.46:0/1354739943 shutdown_connections 2024-05-09T03:09:40.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbaf908b640 1 --2- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40ce340 0x7fbaf40b64f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbf42ffd640 1 -- 172.21.15.46:0/2947822368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3c033940 msgr2=0x7fbf3c0d8120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T03:09:40.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbaf908b640 1 --2- 172.21.15.46:0/1354739943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40d06a0 0x7fbaf40cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbaf908b640 1 -- 172.21.15.46:0/1354739943 >> 172.21.15.46:0/1354739943 conn(0x7fbaf40b4490 msgr2=0x7fbaf40b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7f14262cb640 1 Processor -- start 2024-05-09T03:09:40.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fa694f27640 1 Processor -- start 2024-05-09T03:09:40.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbf42ffd640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3c033940 0x7fbf3c0d8120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbaf908b640 1 -- 172.21.15.46:0/1354739943 shutdown_connections 2024-05-09T03:09:40.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7f14262cb640 1 -- start start 2024-05-09T03:09:40.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbf42ffd640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3c0442c0 con 0x7fbf3c048310 2024-05-09T03:09:40.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbf42ffd640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 0x7fbf3c0438e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e840 tx=0x7fbf3000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.400+0000 7f96f7fff640 1 -- start start 2024-05-09T03:09:40.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fbaf908b640 1 -- 172.21.15.46:0/1354739943 wait complete. 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96f0048820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f7fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96f019a170 0x7f96f01973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f0197910 0x7f96f0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f7fff640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96f0006dd0 con 0x7f96e0003b40 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f7fff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96f0007420 con 0x7f96f0197910 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f7fff640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96f00070b0 con 0x7f96f019a170 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f67fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96f019a170 0x7f96f01973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96f0048820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96f0048820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55990/0 (socket says 172.21.15.46:55990) 2024-05-09T03:09:40.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7efcc50cb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 0x7efcc013b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7efcc50cb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f96f6ffd640 1 -- 172.21.15.46:0/3332038084 learned_addr learned my addr 172.21.15.46:0/3332038084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcbffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 0x7efcc013b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcc50cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0154650 0x7efcc0145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fe94abb4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440b5220 0x7fe944141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f6ffd640 1 -- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96f019a170 msgr2=0x7f96f01973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fe94abb4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe944160790 0x7fe944144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96f019a170 0x7f96f01973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fe94abb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 0x7fe944156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f6ffd640 1 -- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f0197910 msgr2=0x7f96f0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f0197910 0x7f96f0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fe949151640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 0x7fe944156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcbffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 0x7efcc013b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43386/0 (socket says 172.21.15.46:43386) 2024-05-09T03:09:40.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0145050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56870/0 (socket says 172.21.15.46:56870) 2024-05-09T03:09:40.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcc48ca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0154650 0x7efcc0145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcc48ca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0154650 0x7efcc0145590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56002/0 (socket says 172.21.15.46:56002) 2024-05-09T03:09:40.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcc50cb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efcc00b5da0 con 0x7efcc0154650 2024-05-09T03:09:40.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f6ffd640 1 -- 172.21.15.46:0/3332038084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f0048d60 con 0x7f96e0003b40 2024-05-09T03:09:40.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f67fc640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96f019a170 0x7f96f01973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:40.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf30017070 con 0x7fbf3c048310 2024-05-09T03:09:40.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcbffff640 1 -- 172.21.15.46:0/1588640530 learned_addr learned my addr 172.21.15.46:0/1588640530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcc50cb640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efcc00b11c0 con 0x7efcc00b0db0 2024-05-09T03:09:40.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7efcc50cb640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efcc00b5f20 con 0x7efcc00712b0 2024-05-09T03:09:40.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.402+0000 7f1294db3640 1 -- start start 2024-05-09T03:09:40.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.401+0000 7fa694f27640 1 -- start start 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7f96f6ffd640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96f0048820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009b90 tx=0x7f96ec008180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fa694f27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6880517d0 0x7fa6880d9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fa694f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 0x7fa6880d9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fa694f27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6880d14f0 0x7fa68803a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fa694f27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880354b0 con 0x7fa6880517d0 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fa694f27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa688035330 con 0x7fa6880d14f0 2024-05-09T03:09:40.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fa694f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa688055630 con 0x7fa688054090 2024-05-09T03:09:40.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf3c0445a0 con 0x7fbf3c048310 2024-05-09T03:09:40.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf3c044b60 con 0x7fbf3c048310 2024-05-09T03:09:40.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.403+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3000c940 con 0x7fbf3c048310 2024-05-09T03:09:40.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000cd30 con 0x7fbf3c048310 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec038040 con 0x7f96e0003b40 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f0049270 con 0x7f96e0003b40 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f0049830 con 0x7f96e0003b40 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 0x7fa6880d9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 0x7fa6880d9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43388/0 (socket says 172.21.15.46:43388) 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 -- 172.21.15.46:0/3364819938 learned_addr learned my addr 172.21.15.46:0/3364819938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68e59c640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6880517d0 0x7fa6880d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fe949151640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 0x7fe944156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56018/0 (socket says 172.21.15.46:56018) 2024-05-09T03:09:40.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fe949151640 1 -- 172.21.15.46:0/2834034859 learned_addr learned my addr 172.21.15.46:0/2834034859 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fe94abb4640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9440d12e0 con 0x7fe944141260 2024-05-09T03:09:40.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fe94abb4640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe94415a030 con 0x7fe944160790 2024-05-09T03:09:40.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fe94abb4640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe9440d1460 con 0x7fe9440b5220 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14262cb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14180382f0 0x7f141802c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14262cb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 0x7f14180e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14262cb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1418054170 0x7f1418189e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14252c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14180382f0 0x7f141802c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f1424ac8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 0x7f14180e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14252c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14180382f0 0x7f141802c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56036/0 (socket says 172.21.15.46:56036) 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f1424ac8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 0x7f14180e75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43398/0 (socket says 172.21.15.46:43398) 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14252c9640 1 -- 172.21.15.46:0/3623366425 learned_addr learned my addr 172.21.15.46:0/3623366425 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f1425aca640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1418054170 0x7f1418189e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1418007400 con 0x7f14180382f0 2024-05-09T03:09:40.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f14180066c0 con 0x7f1418054170 2024-05-09T03:09:40.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1418007580 con 0x7f1418053650 2024-05-09T03:09:40.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fbaf908b640 1 Processor -- start 2024-05-09T03:09:40.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1294db3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f1290140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 -- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6880d14f0 msgr2=0x7fa68803a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6880d14f0 0x7fa68803a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 -- 172.21.15.46:0/3364819938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6880517d0 msgr2=0x7fa6880d9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6880517d0 0x7fa6880d9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.404+0000 7fa68dd9b640 1 -- 172.21.15.46:0/3364819938 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6880d0cd0 con 0x7fa688054090 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fa68dd9b640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 0x7fa6880d9ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa69006b8a0 tx=0x7fa69006ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690076290 con 0x7fa688054090 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa690090070 con 0x7fa688054090 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690095470 con 0x7fa688054090 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6880d6f50 con 0x7fa688054090 2024-05-09T03:09:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fe943fff640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe944160790 0x7fe944144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fbaf908b640 1 -- start start 2024-05-09T03:09:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1294db3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1290153300 0x7f129014a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1294db3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1290153950 0x7f1290143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1294db3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12900b5da0 con 0x7f1290153950 2024-05-09T03:09:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1294db3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f129015d170 con 0x7f1290153300 2024-05-09T03:09:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1294db3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12900b5f20 con 0x7f12900712b0 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa688039930 con 0x7fa688054090 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7efcbf7fe640 1 -- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 msgr2=0x7efcc013b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7efcbf7fe640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 0x7efcc013b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7efcbf7fe640 1 -- 172.21.15.46:0/1588640530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0154650 msgr2=0x7efcc0145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7efcbf7fe640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0154650 0x7efcc0145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7efcbf7fe640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc0156350 con 0x7efcc00b0db0 2024-05-09T03:09:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ec004230 con 0x7f96e0003b40 2024-05-09T03:09:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec038040 con 0x7f96e0003b40 2024-05-09T03:09:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7efcbf7fe640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0145050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcb400b440 tx=0x7efcb400b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f128ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1290153950 0x7f1290143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f128ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1290153950 0x7f1290143010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56038/0 (socket says 172.21.15.46:56038) 2024-05-09T03:09:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f128ffff640 1 -- 172.21.15.46:0/381117530 learned_addr learned my addr 172.21.15.46:0/381117530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7fe943fff640 1 -- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440b5220 msgr2=0x7fe944141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7fe943fff640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440b5220 0x7fe944141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7fe943fff640 1 -- 172.21.15.46:0/2834034859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 msgr2=0x7fe944156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7fe943fff640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 0x7fe944156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7fe943fff640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9440d02b0 con 0x7fe944160790 2024-05-09T03:09:40.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1424ac8640 1 -- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1418054170 msgr2=0x7f1418189e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1424ac8640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1418054170 0x7f1418189e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1424ac8640 1 -- 172.21.15.46:0/3623366425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14180382f0 msgr2=0x7f141802c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1424ac8640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14180382f0 0x7f141802c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.405+0000 7f1424ac8640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141802bca0 con 0x7f1418053650 2024-05-09T03:09:40.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7fe943fff640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe944160790 0x7fe944144880 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe930031000 tx=0x7fe9300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128effd640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1290153300 0x7f129014a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128f7fe640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f1290140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128effd640 1 -- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 msgr2=0x7f1290140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128effd640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f1290140fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128effd640 1 -- 172.21.15.46:0/381117530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1290153950 msgr2=0x7f1290143010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128effd640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1290153950 0x7f1290143010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f128effd640 1 -- 172.21.15.46:0/381117530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129014af20 con 0x7f1290153300 2024-05-09T03:09:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.406+0000 7f1424ac8640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 0x7f14180e75d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1420069fe0 tx=0x7f142006ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7efcbffff640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 0x7efcc013b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:40.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fe949151640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 0x7fe944156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:40.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f128effd640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1290153300 0x7f129014a9e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f128800bd20 tx=0x7f128800be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe930042020 con 0x7fe944160790 2024-05-09T03:09:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf24005290 con 0x7fbf3c048310 2024-05-09T03:09:40.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fe94abb4640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9440cfa10 con 0x7fe944160790 2024-05-09T03:09:40.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fe94abb4640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9440cffd0 con 0x7fe944160790 2024-05-09T03:09:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf908b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf40ce340 0x7fbaf4159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f96dd7fa640 1 -- 172.21.15.46:0/3332038084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d8005290 con 0x7f96e0003b40 2024-05-09T03:09:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f96ec03b070 con 0x7f96e0003b40 2024-05-09T03:09:40.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa674005290 con 0x7fa688054090 2024-05-09T03:09:40.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf908b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40cf4a0 0x7fbaf4158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf908b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40d06a0 0x7fbaf409e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf40ce340 0x7fbaf4159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf908b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf40ce750 con 0x7fbaf40ce340 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf908b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf40b58c0 con 0x7fbaf40d06a0 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf908b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbaf40cf8b0 con 0x7fbaf40cf4a0 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf40ce340 0x7fbaf4159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56054/0 (socket says 172.21.15.46:56054) 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7fbaf2d9d640 1 -- 172.21.15.46:0/3408838278 learned_addr learned my addr 172.21.15.46:0/3408838278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fbaf359e640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40d06a0 0x7fbaf409e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fbaf359e640 1 -- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40cf4a0 msgr2=0x7fbaf4158040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T03:09:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fbaf359e640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40cf4a0 0x7fbaf4158040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fbaf359e640 1 -- 172.21.15.46:0/3408838278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf40ce340 msgr2=0x7fbaf4159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fbaf359e640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf40ce340 0x7fbaf4159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fbaf359e640 1 -- 172.21.15.46:0/3408838278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf409d030 con 0x7fbaf40d06a0 2024-05-09T03:09:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288019070 con 0x7f1290153300 2024-05-09T03:09:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1288014440 con 0x7f1290153300 2024-05-09T03:09:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288014870 con 0x7f1290153300 2024-05-09T03:09:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1294db3640 1 -- 172.21.15.46:0/381117530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1290141f70 con 0x7f1290153300 2024-05-09T03:09:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1294db3640 1 -- 172.21.15.46:0/381117530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1290142530 con 0x7f1290153300 2024-05-09T03:09:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1420076490 con 0x7f1418053650 2024-05-09T03:09:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1420090070 con 0x7f1418053650 2024-05-09T03:09:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14200956b0 con 0x7f1418053650 2024-05-09T03:09:40.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141802bf40 con 0x7f1418053650 2024-05-09T03:09:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4011830 con 0x7efcc00b0db0 2024-05-09T03:09:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb4016070 con 0x7efcc00b0db0 2024-05-09T03:09:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4011c10 con 0x7efcc00b0db0 2024-05-09T03:09:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.407+0000 7efcc50cb640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc0156630 con 0x7efcc00b0db0 2024-05-09T03:09:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7efcc50cb640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc0142c30 con 0x7efcc00b0db0 2024-05-09T03:09:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fbaf359e640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40d06a0 0x7fbaf409e0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbaec030000 tx=0x7fbaec004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7f96df7fe640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96e40764d0 0x7f96e40789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f96ec0b7600 con 0x7f96e0003b40 2024-05-09T03:09:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec041020 con 0x7fbaf40d06a0 2024-05-09T03:09:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9300339b0 con 0x7fe944160790 2024-05-09T03:09:40.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec0086b0 con 0x7fbaf40d06a0 2024-05-09T03:09:40.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec036040 con 0x7fbaf40d06a0 2024-05-09T03:09:40.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa690095610 con 0x7fa688054090 2024-05-09T03:09:40.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fa67effd640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa670076320 0x7fa670078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.409+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf409d310 con 0x7fbaf40d06a0 2024-05-09T03:09:40.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.410+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9300078d0 con 0x7fe944160790 2024-05-09T03:09:40.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.410+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf409d8d0 con 0x7fbaf40d06a0 2024-05-09T03:09:40.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.410+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbf3001a070 con 0x7fbf3c048310 2024-05-09T03:09:40.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.408+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1418035fd0 con 0x7f1418053650 2024-05-09T03:09:40.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1268005290 con 0x7f1290153300 2024-05-09T03:09:40.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.410+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca0005290 con 0x7efcc00b0db0 2024-05-09T03:09:40.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efcb401b460 con 0x7efcc00b0db0 2024-05-09T03:09:40.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14180058e0 con 0x7f1418053650 2024-05-09T03:09:40.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7fbf2ffff640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbf280763a0 0x7fbf280788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa6900d95e0 con 0x7fa688054090 2024-05-09T03:09:40.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7fbf437fe640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbf280763a0 0x7fbf280788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7efcbcff9640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcac0763a0 0x7efcac0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbf300977a0 con 0x7fbf3c048310 2024-05-09T03:09:40.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7f96f67fc640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96e40764d0 0x7f96e40789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7fa68e59c640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa670076320 0x7fa670078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7efcbffff640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcac0763a0 0x7efcac0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf4076c00 con 0x7fbaf40d06a0 2024-05-09T03:09:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efcb4097210 con 0x7efcc00b0db0 2024-05-09T03:09:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe91c005290 con 0x7fe944160790 2024-05-09T03:09:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe930045020 con 0x7fe944160790 2024-05-09T03:09:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.411+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1420095850 con 0x7f1418053650 2024-05-09T03:09:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7f1415ffb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14040763a0 0x7f14040788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1420113cc0 con 0x7f1418053650 2024-05-09T03:09:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f128801c070 con 0x7f1290153300 2024-05-09T03:09:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.412+0000 7f14252c9640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14040763a0 0x7f14040788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7efcbffff640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcac0763a0 0x7efcac0788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcb8007350 tx=0x7efcb8009610 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fa68e59c640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa670076320 0x7fa670078830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa684005880 tx=0x7fa684009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fbf437fe640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbf280763a0 0x7fbf280788b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf38034000 tx=0x7fbf3802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbaec044020 con 0x7fbaf40d06a0 2024-05-09T03:09:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7f14252c9640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14040763a0 0x7f14040788b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f14100090b0 tx=0x7f141002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7f1277fff640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f127c0763d0 0x7f127c0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1288097580 con 0x7f1290153300 2024-05-09T03:09:40.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fe9417fa640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe9280785e0 0x7fe92807aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7f128f7fe640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f127c0763d0 0x7f127c0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7f96f67fc640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96e40764d0 0x7f96e40789e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96f002bc50 tx=0x7f96e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fbadb7fe640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbae00763d0 0x7fbae00788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbaec0b8270 con 0x7fbaf40d06a0 2024-05-09T03:09:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.414+0000 7fbaf2d9d640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbae00763d0 0x7fbae00788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.413+0000 7fe948950640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe9280785e0 0x7fe92807aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.414+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe9300b8fb0 con 0x7fe944160790 2024-05-09T03:09:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.414+0000 7f128f7fe640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f127c0763d0 0x7f127c0788e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1280034000 tx=0x7f128002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.414+0000 7fbaf2d9d640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbae00763d0 0x7fbae00788e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006a90 tx=0x7fbae4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.414+0000 7fe948950640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe9280785e0 0x7fe92807aaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe938006fd0 tx=0x7fe938006bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.416+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbf30060a00 con 0x7fbf3c048310 2024-05-09T03:09:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.417+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f96ec0809a0 con 0x7f96e0003b40 2024-05-09T03:09:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.417+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efcb4060b90 con 0x7efcc00b0db0 2024-05-09T03:09:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.421+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f14200dcfb0 con 0x7f1418053650 2024-05-09T03:09:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.422+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe930082a20 con 0x7fe944160790 2024-05-09T03:09:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.426+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa6900dd6f0 con 0x7fa688054090 2024-05-09T03:09:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.428+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1288060ed0 con 0x7f1290153300 2024-05-09T03:09:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.436+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbaec04b030 con 0x7fbaf40d06a0 2024-05-09T03:09:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.844+0000 7f96dd7fa640 1 -- 172.21.15.46:0/3332038084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f96d8005850 con 0x7f96e0003b40 2024-05-09T03:09:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.845+0000 7f96df7fe640 1 -- 172.21.15.46:0/3332038084 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96ec084ca0 con 0x7f96e0003b40 2024-05-09T03:09:40.846 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.849+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96e40764d0 msgr2=0x7f96e40789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.849+0000 7f96f7fff640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96e40764d0 0x7f96e40789e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f96f002bc50 tx=0x7f96e8009040 comp rx=0 tx=0).stop 2024-05-09T03:09:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.849+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 msgr2=0x7f96f0048820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.849+0000 7f96f7fff640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96f0048820 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009b90 tx=0x7f96ec008180 comp rx=0 tx=0).stop 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.849+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 shutdown_connections 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f0197910 0x7f96f0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96f019a170 0x7f96f01973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96e40764d0 0x7f96e40789e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 --2- 172.21.15.46:0/3332038084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e0003b40 0x7f96f0048820 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 >> 172.21.15.46:0/3332038084 conn(0x7f96f0054cf0 msgr2=0x7f96f0101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 shutdown_connections 2024-05-09T03:09:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.850+0000 7f96f7fff640 1 -- 172.21.15.46:0/3332038084 wait complete. 2024-05-09T03:09:40.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.862+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbf24005580 con 0x7fbf3c048310 2024-05-09T03:09:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.862+0000 7fbf2ffff640 1 -- 172.21.15.46:0/2947822368 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf30064d00 con 0x7fbf3c048310 2024-05-09T03:09:40.863 INFO:teuthology.orchestra.run.smithi046.stdout:55834574852 2024-05-09T03:09:40.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.865+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7efca0005580 con 0x7efcc00b0db0 2024-05-09T03:09:40.866 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.866+0000 7efcbcff9640 1 -- 172.21.15.46:0/1588640530 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efcb4060490 con 0x7efcc00b0db0 2024-05-09T03:09:40.868 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.867+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbf280763a0 msgr2=0x7fbf280788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.867+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbf280763a0 0x7fbf280788b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbf38034000 tx=0x7fbf3802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 msgr2=0x7fbf3c0438e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 0x7fbf3c0438e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000e840 tx=0x7fbf3000ed00 comp rx=0 tx=0).stop 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 shutdown_connections 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbf3c04f330 0x7fbf3c046190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbf3c048310 0x7fbf3c0438e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbf280763a0 0x7fbf280788b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 --2- 172.21.15.46:0/2947822368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf3c033940 0x7fbf3c0d8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 >> 172.21.15.46:0/2947822368 conn(0x7fbf3c054ce0 msgr2=0x7fbf3c039a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 shutdown_connections 2024-05-09T03:09:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.868+0000 7fbf49d3f640 1 -- 172.21.15.46:0/2947822368 wait complete. 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcac0763a0 msgr2=0x7efcac0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcac0763a0 0x7efcac0788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efcb8007350 tx=0x7efcb8009610 comp rx=0 tx=0).stop 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 msgr2=0x7efcc0145050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0145050 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efcb400b440 tx=0x7efcb400b900 comp rx=0 tx=0).stop 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 shutdown_connections 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcac0763a0 0x7efcac0788b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0154650 0x7efcc0145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc00b0db0 0x7efcc0145050 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 --2- 172.21.15.46:0/1588640530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc00712b0 0x7efcc013b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.869+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 >> 172.21.15.46:0/1588640530 conn(0x7efcc006c680 msgr2=0x7efcc00b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.870+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 shutdown_connections 2024-05-09T03:09:40.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.870+0000 7efca67fc640 1 -- 172.21.15.46:0/1588640530 wait complete. 2024-05-09T03:09:40.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.872+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f141802c110 con 0x7f1418053650 2024-05-09T03:09:40.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.873+0000 7f1415ffb640 1 -- 172.21.15.46:0/3623366425 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f14200e12b0 con 0x7f1418053650 2024-05-09T03:09:40.874 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.874+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe91c005580 con 0x7fe944160790 2024-05-09T03:09:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.874+0000 7fe9417fa640 1 -- 172.21.15.46:0/2834034859 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe930082320 con 0x7fe944160790 2024-05-09T03:09:40.875 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.875+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa674005850 con 0x7fa688054090 2024-05-09T03:09:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.876+0000 7fa67effd640 1 -- 172.21.15.46:0/3364819938 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa6900dcff0 con 0x7fa688054090 2024-05-09T03:09:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.876+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14040763a0 msgr2=0x7f14040788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.876+0000 7f14262cb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14040763a0 0x7f14040788b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f14100090b0 tx=0x7f141002e000 comp rx=0 tx=0).stop 2024-05-09T03:09:40.877 INFO:teuthology.orchestra.run.smithi046.stdout:55834574852 2024-05-09T03:09:40.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.876+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 msgr2=0x7f14180e75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.876+0000 7f14262cb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 0x7f14180e75d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1420069fe0 tx=0x7f142006ebe0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 shutdown_connections 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1418054170 0x7f1418189e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1418053650 0x7f14180e75d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14040763a0 0x7f14040788b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 --2- 172.21.15.46:0/3623366425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14180382f0 0x7f141802c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 >> 172.21.15.46:0/3623366425 conn(0x7f1418055130 msgr2=0x7f1418039e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 shutdown_connections 2024-05-09T03:09:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7f14262cb640 1 -- 172.21.15.46:0/3623366425 wait complete. 2024-05-09T03:09:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.877+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbaf40d0c80 con 0x7fbaf40d06a0 2024-05-09T03:09:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.878+0000 7fbadb7fe640 1 -- 172.21.15.46:0/3408838278 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbaec081be0 con 0x7fbaf40d06a0 2024-05-09T03:09:40.879 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe9280785e0 msgr2=0x7fe92807aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe9280785e0 0x7fe92807aaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe938006fd0 tx=0x7fe938006bc0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe944160790 msgr2=0x7fe944144880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe944160790 0x7fe944144880 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe930031000 tx=0x7fe9300046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 shutdown_connections 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe9280785e0 0x7fe92807aaf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe944141260 0x7fe944156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe944160790 0x7fe944144880 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 --2- 172.21.15.46:0/2834034859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9440b5220 0x7fe944141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 >> 172.21.15.46:0/2834034859 conn(0x7fe9440ce370 msgr2=0x7fe9440708d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 shutdown_connections 2024-05-09T03:09:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa670076320 msgr2=0x7fa670078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fa694f27640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa670076320 0x7fa670078830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa684005880 tx=0x7fa684009040 comp rx=0 tx=0).stop 2024-05-09T03:09:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 msgr2=0x7fa6880d9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.879+0000 7fa694f27640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 0x7fa6880d9ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa69006b8a0 tx=0x7fa69006ebe0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fe92effd640 1 -- 172.21.15.46:0/2834034859 wait complete. 2024-05-09T03:09:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 shutdown_connections 2024-05-09T03:09:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6880d14f0 0x7fa68803a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa688054090 0x7fa6880d9ca0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa670076320 0x7fa670078830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 --2- 172.21.15.46:0/3364819938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6880517d0 0x7fa6880d9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 >> 172.21.15.46:0/3364819938 conn(0x7fa688055df0 msgr2=0x7fa6880eb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 shutdown_connections 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.880+0000 7fa694f27640 1 -- 172.21.15.46:0/3364819938 wait complete. 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.881+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1268005580 con 0x7f1290153300 2024-05-09T03:09:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbae00763d0 msgr2=0x7fbae00788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbae00763d0 0x7fbae00788e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbae4006a90 tx=0x7fbae4008040 comp rx=0 tx=0).stop 2024-05-09T03:09:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40d06a0 msgr2=0x7fbaf409e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40d06a0 0x7fbaf409e0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbaec030000 tx=0x7fbaec004a40 comp rx=0 tx=0).stop 2024-05-09T03:09:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7f1277fff640 1 -- 172.21.15.46:0/381117530 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f12880607d0 con 0x7f1290153300 2024-05-09T03:09:40.885 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 shutdown_connections 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaf40d06a0 0x7fbaf409e0d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbaf40cf4a0 0x7fbaf4158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbae00763d0 0x7fbae00788e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 --2- 172.21.15.46:0/3408838278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbaf40ce340 0x7fbaf4159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 >> 172.21.15.46:0/3408838278 conn(0x7fbaf40b4490 msgr2=0x7fbaf41692f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.882+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 shutdown_connections 2024-05-09T03:09:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.883+0000 7fbaf908b640 1 -- 172.21.15.46:0/3408838278 wait complete. 2024-05-09T03:09:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f127c0763d0 msgr2=0x7f127c0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f127c0763d0 0x7f127c0788e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1280034000 tx=0x7f128002e000 comp rx=0 tx=0).stop 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1290153300 msgr2=0x7f129014a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1290153300 0x7f129014a9e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f128800bd20 tx=0x7f128800be10 comp rx=0 tx=0).stop 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 shutdown_connections 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f127c0763d0 0x7f127c0788e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1290153950 0x7f1290143010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1290153300 0x7f129014a9e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 --2- 172.21.15.46:0/381117530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12900712b0 0x7f1290140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 >> 172.21.15.46:0/381117530 conn(0x7f129006c680 msgr2=0x7f129006ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 shutdown_connections 2024-05-09T03:09:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:40.886+0000 7f1275ffb640 1 -- 172.21.15.46:0/381117530 wait complete. 2024-05-09T03:09:40.890 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-05-09T03:09:40.892 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-05-09T03:09:40.899 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2024-05-09T03:09:40.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2024-05-09T03:09:40.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2024-05-09T03:09:40.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.3 2024-05-09T03:09:40.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-05-09T03:09:41.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T03:09:41.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T03:09:41.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T03:09:41.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T03:09:41.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T03:09:41.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T03:09:41.905 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T03:09:41.907 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T03:09:42.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.070+0000 7fc083fc7640 1 Processor -- start 2024-05-09T03:09:42.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.072+0000 7fc083fc7640 1 -- start start 2024-05-09T03:09:42.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.076+0000 7fc083fc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c156940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.077+0000 7fc083fc7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc07c0d1740 0x7fc07c06fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.077+0000 7fc081562640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc07c0d1740 0x7fc07c06fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.077+0000 7fc081562640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc07c0d1740 0x7fc07c06fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56924/0 (socket says 172.21.15.46:56924) 2024-05-09T03:09:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.077+0000 7fc081562640 1 -- 172.21.15.46:0/2714790865 learned_addr learned my addr 172.21.15.46:0/2714790865 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.078+0000 7fc081d63640 1 --2- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.078+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc07c070690 con 0x7fc07c070280 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.078+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc07c15d900 con 0x7fc07c0d1740 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.078+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c0cec80 con 0x7fc07c0ce870 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.079+0000 7fc081562640 1 -- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 msgr2=0x7fc07c156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.079+0000 7fc081562640 1 --2- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.079+0000 7fc081562640 1 -- 172.21.15.46:0/2714790865 >> v1:172.21.15.88:6789/0 conn(0x7fc07c070280 legacy=0x7fc07c06c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.079+0000 7fc081562640 1 -- 172.21.15.46:0/2714790865 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c15ff80 con 0x7fc07c0d1740 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.079+0000 7fc080d61640 1 -- 172.21.15.46:0/2714790865 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911771643 0 0) 0x7fc074001490 con 0x7fc07c070280 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.081+0000 7fc081562640 1 --2- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c0d1740 0x7fc07c06fd40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc06c007360 tx=0x7fc06c02c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6be604b4e4e7b23e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.081+0000 7fc080d61640 1 -- 172.21.15.46:0/2714790865 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c008be0 con 0x7fc07c0d1740 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.081+0000 7fc080d61640 1 -- 172.21.15.46:0/2714790865 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc06c008d80 con 0x7fc07c0d1740 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.083+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c0d1740 msgr2=0x7fc07c06fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.083+0000 7fc083fc7640 1 --2- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c0d1740 0x7fc07c06fd40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc06c007360 tx=0x7fc06c02c370 comp rx=0 tx=0).stop 2024-05-09T03:09:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.083+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 shutdown_connections 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.083+0000 7fc083fc7640 1 --2- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c0d1740 0x7fc07c06fd40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.083+0000 7fc083fc7640 1 --2- 172.21.15.46:0/2714790865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.084+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 >> 172.21.15.46:0/2714790865 conn(0x7fc07c0b4400 msgr2=0x7fc07c06f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.084+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 shutdown_connections 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.084+0000 7fc083fc7640 1 -- 172.21.15.46:0/2714790865 wait complete. 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.088+0000 7fc083fc7640 1 Processor -- start 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.088+0000 7fc083fc7640 1 -- start start 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc083fc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 0x7fc07c0aa660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc083fc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc081d63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 0x7fc07c0aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc081d63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 0x7fc07c0aa660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56940/0 (socket says 172.21.15.46:56940) 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc081562640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc081562640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c1578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43432/0 (socket says 172.21.15.46:43432) 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.091+0000 7fc081562640 1 -- 172.21.15.46:0/1517897326 learned_addr learned my addr 172.21.15.46:0/1517897326 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.092+0000 7fc083fc7640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07c0d1740 0x7fc07c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.092+0000 7fc082564640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07c0d1740 0x7fc07c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.092+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c160820 con 0x7fc07c0d1740 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.092+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c160540 con 0x7fc07c070280 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.092+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c0d1b50 con 0x7fc07c0ce870 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.093+0000 7fc082564640 1 -- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 msgr2=0x7fc07c1578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.093+0000 7fc082564640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c1578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.093+0000 7fc082564640 1 -- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 msgr2=0x7fc07c0aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.093+0000 7fc082564640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 0x7fc07c0aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.093+0000 7fc082564640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c1581f0 con 0x7fc07c0d1740 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.094+0000 7fc082564640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07c0d1740 0x7fc07c0b04b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0740013b0 tx=0x7fc07400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.094+0000 7fc081d63640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 0x7fc07c0aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.094+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07401dc30 con 0x7fc07c0d1740 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.095+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc07c1584d0 con 0x7fc07c0d1740 2024-05-09T03:09:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.095+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc07c158a90 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.098+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074015070 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.098+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074003970 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.098+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc074003b90 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.099+0000 7fc07a7fc640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc05c0763d0 0x7fc05c0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.100+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc07409f320 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.100+0000 7fc081d63640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc05c0763d0 0x7fc05c0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.100+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07c06a270 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.110+0000 7fc081d63640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc05c0763d0 0x7fc05c0788e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0725a0 tx=0x7fc064008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.111+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc074022030 con 0x7fc07c0d1740 2024-05-09T03:09:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.142+0000 7f924c3f3640 1 Processor -- start 2024-05-09T03:09:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.142+0000 7fc1748a6640 1 Processor -- start 2024-05-09T03:09:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.142+0000 7fc1748a6640 1 -- start start 2024-05-09T03:09:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.142+0000 7f924c3f3640 1 -- start start 2024-05-09T03:09:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.142+0000 7fb5b0049640 1 Processor -- start 2024-05-09T03:09:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f924c3f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9244160790 0x7f92440732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f924c3f3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9244073810 0x7f92440cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f924c3f3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9244073c20 con 0x7f92440cf9e0 2024-05-09T03:09:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f924c3f3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9244160ba0 con 0x7f9244073810 2024-05-09T03:09:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f924c3f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f924415cbc0 con 0x7f9244160790 2024-05-09T03:09:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7fc1748a6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc168052fb0 0x7fc16804f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7fc1748a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc168034180 0x7fc168038340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7fdcbbfff640 1 Processor -- start 2024-05-09T03:09:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7fdcbbfff640 1 -- start start 2024-05-09T03:09:42.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.142+0000 7fb5b0049640 1 -- start start 2024-05-09T03:09:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f8acbfff640 1 Processor -- start 2024-05-09T03:09:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.144+0000 7f8acbfff640 1 -- start start 2024-05-09T03:09:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc1748a6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1680e8230 con 0x7fc168007320 2024-05-09T03:09:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc1748a6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc1680533c0 con 0x7fc168052fb0 2024-05-09T03:09:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc1748a6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc168006280 con 0x7fc168034180 2024-05-09T03:09:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7f7e6d3b4640 1 Processor -- start 2024-05-09T03:09:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7f7e6d3b4640 1 -- start start 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc16effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc168007320 0x7fc168033a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45682/0 (socket says 172.21.15.46:45682) 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc16effd640 1 -- 172.21.15.46:0/367408309 learned_addr learned my addr 172.21.15.46:0/367408309 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc16ffff640 1 --2- 172.21.15.46:0/367408309 >> v2:172.21.15.88:3300/0 conn(0x7fc168034180 0x7fc168038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc16f7fe640 1 --2- 172.21.15.46:0/367408309 >> v2:172.21.15.46:3300/0 conn(0x7fc168052fb0 0x7fc16804f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fc16e7fc640 1 -- 172.21.15.46:0/367408309 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3645913107 0 0) 0x7fc1680e8230 con 0x7fc168007320 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fc16e7fc640 1 -- 172.21.15.46:0/367408309 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc160003520 con 0x7fc168007320 2024-05-09T03:09:42.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fc16e7fc640 1 -- 172.21.15.46:0/367408309 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2384271682 0 0) 0x7fc160003520 con 0x7fc168007320 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fdcbbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdcb4007870 0x7fdcb404f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fdcbbfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40545c0 0x7fdcb40062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fdcbbfff640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb404fe60 con 0x7fdcb40549d0 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fdcbbfff640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdcb4054130 con 0x7fdcb4007870 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fdcbbfff640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb4053ae0 con 0x7fdcb40545c0 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fc16f7fe640 1 -- 172.21.15.46:0/367408309 >> v2:172.21.15.88:3300/0 conn(0x7fc168034180 msgr2=0x7fc168038340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fc16f7fe640 1 --2- 172.21.15.46:0/367408309 >> v2:172.21.15.88:3300/0 conn(0x7fc168034180 0x7fc168038340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fc16f7fe640 1 -- 172.21.15.46:0/367408309 >> v1:172.21.15.46:6789/0 conn(0x7fc168007320 legacy=0x7fc168033a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8acc0b7230 0x7f8acc0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8acc0b6210 0x7f8acc15d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acbfff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8acc0b0280 con 0x7f8acc0b3c10 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acbfff640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8acc0b0520 con 0x7f8acc0b6210 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acbfff640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8acc0b0800 con 0x7f8acc0b7230 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8acc0b6210 0x7f8acc15d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8acc0b6210 0x7f8acc15d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56090/0 (socket says 172.21.15.46:56090) 2024-05-09T03:09:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f8acb7fe640 1 -- 172.21.15.46:0/786885552 learned_addr learned my addr 172.21.15.46:0/786885552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fc16f7fe640 1 -- 172.21.15.46:0/367408309 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc168006560 con 0x7fc168052fb0 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f8aca7fc640 1 --2- 172.21.15.46:0/786885552 >> v2:172.21.15.88:3300/0 conn(0x7f8acc0b7230 0x7f8acc0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f8acb7fe640 1 -- 172.21.15.46:0/786885552 >> v2:172.21.15.88:3300/0 conn(0x7f8acc0b7230 msgr2=0x7f8acc0b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f8acb7fe640 1 --2- 172.21.15.46:0/786885552 >> v2:172.21.15.88:3300/0 conn(0x7f8acc0b7230 0x7f8acc0b15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f8acb7fe640 1 -- 172.21.15.46:0/786885552 >> v1:172.21.15.46:6789/0 conn(0x7f8acc0b3c10 legacy=0x7f8acc0b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f8acb7fe640 1 -- 172.21.15.46:0/786885552 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8acc0b0ae0 con 0x7f8acc0b6210 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f8aca7fc640 1 --2- 172.21.15.46:0/786885552 >> v2:172.21.15.88:3300/0 conn(0x7f8acc0b7230 0x7f8acc0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:42.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fc16ffff640 1 --2- 172.21.15.46:0/367408309 >> v2:172.21.15.88:3300/0 conn(0x7fc168034180 0x7fc168038340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fdcc14f0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdcb40549d0 0x7fdcb40dfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:35216/0 (socket says 172.21.15.46:35216) 2024-05-09T03:09:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fdcc14f0640 1 -- 172.21.15.46:0/3155981940 learned_addr learned my addr 172.21.15.46:0/3155981940 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7f924a18f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9244160790 0x7f92440732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7f924a990640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f92440cf9e0 0x7f92440b4ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:35202/0 (socket says 172.21.15.46:35202) 2024-05-09T03:09:42.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fdcbaffd640 1 --2- 172.21.15.46:0/3155981940 >> v2:172.21.15.88:3300/0 conn(0x7fdcb4007870 0x7fdcb404f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7f924a990640 1 -- 172.21.15.46:0/491230572 learned_addr learned my addr 172.21.15.46:0/491230572 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f924a18f640 1 -- 172.21.15.46:0/491230572 >> v2:172.21.15.88:3300/0 conn(0x7f9244073810 msgr2=0x7f92440cf4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f924a18f640 1 --2- 172.21.15.46:0/491230572 >> v2:172.21.15.88:3300/0 conn(0x7f9244073810 0x7f92440cf4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f924a18f640 1 -- 172.21.15.46:0/491230572 >> v1:172.21.15.88:6789/0 conn(0x7f92440cf9e0 legacy=0x7f92440b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7f924a18f640 1 -- 172.21.15.46:0/491230572 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924415cea0 con 0x7f9244160790 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f924a18f640 1 --2- 172.21.15.46:0/491230572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9244160790 0x7f92440732d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9240010b90 tx=0x7f9240032b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ed1f341d82d8588 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f924918d640 1 -- 172.21.15.46:0/491230572 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92400348a0 con 0x7f9244160790 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f924918d640 1 -- 172.21.15.46:0/491230572 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9240034a40 con 0x7f9244160790 2024-05-09T03:09:42.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f924918d640 1 -- 172.21.15.46:0/491230572 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240034e10 con 0x7f9244160790 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fc16f7fe640 1 --2- 172.21.15.46:0/367408309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168052fb0 0x7fc16804f340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc17006be50 tx=0x7fc1700a7020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af44606ba71da28b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f7e6d3b4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 0x7f7e60037b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f7e6d3b4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e600e8140 con 0x7f7e600384c0 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f7e6d3b4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e60057900 con 0x7f7e600380b0 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7f7e6d3b4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6003c180 con 0x7f7e600e47b0 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 0x7f7e60037b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 0x7f7e60037b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43480/0 (socket says 172.21.15.46:43480) 2024-05-09T03:09:42.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1174008941 learned_addr learned my addr 172.21.15.46:0/1174008941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40545c0 0x7fdcb40062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1174008941 >> v1:172.21.15.88:6789/0 conn(0x7f7e600380b0 legacy=0x7f7e600e7320 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T03:09:42.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1174008941 >> v1:172.21.15.46:6789/0 conn(0x7f7e600384c0 legacy=0x7f7e600e7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71621050 0 0) 0x7fdcb404fe60 con 0x7fdcb40549d0 2024-05-09T03:09:42.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdca4003520 con 0x7fdcb40549d0 2024-05-09T03:09:42.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 <== mon.1 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1499606268 0 0) 0x7fdca4003520 con 0x7fdcb40549d0 2024-05-09T03:09:42.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40545c0 msgr2=0x7fdcb40062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fc16e7fc640 1 -- 172.21.15.46:0/367408309 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1700b4070 con 0x7fc168052fb0 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40545c0 0x7fdcb40062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 >> v2:172.21.15.88:3300/0 conn(0x7fdcb4007870 msgr2=0x7fdcb404f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 --2- 172.21.15.46:0/3155981940 >> v2:172.21.15.88:3300/0 conn(0x7fdcb4007870 0x7fdcb404f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb4053dc0 con 0x7fdcb40549d0 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 <== mon.1 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fdcbc076c00 con 0x7fdcb40549d0 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 >> v1:172.21.15.88:6789/0 conn(0x7fdcb40549d0 legacy=0x7fdcb40dfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca40039b0 0x7fdca4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4004360 0x7fdca4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca4004d40 0x7fdca4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb4054130 con 0x7fdca4004360 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb4053ae0 con 0x7fdca40039b0 2024-05-09T03:09:42.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdca40102e0 con 0x7fdca4004d40 2024-05-09T03:09:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fc16e7fc640 1 -- 172.21.15.46:0/367408309 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1700a7960 con 0x7fc168052fb0 2024-05-09T03:09:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7fc16e7fc640 1 -- 172.21.15.46:0/367408309 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1640016f0 con 0x7fc168052fb0 2024-05-09T03:09:42.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.148+0000 7f8acb7fe640 1 --2- 172.21.15.46:0/786885552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc0b6210 0x7f8acc15d100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0727f0 tx=0x7f8ac402c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a9722d2e68a080 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fdcbaffd640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca40039b0 0x7fdca4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4004360 0x7fdca4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fb5b0049640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5a80b2490 0x7fb5a8164500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.145+0000 7fb5b0049640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80b3790 0x7fb5a80b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fb5b0049640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a806ddf0 con 0x7fb5a80ce270 2024-05-09T03:09:42.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fdcba7fc640 1 -- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca4004d40 msgr2=0x7fdca4005210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T03:09:42.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca4004d40 0x7fdca4005210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fdcba7fc640 1 -- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca40039b0 msgr2=0x7fdca4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca40039b0 0x7fdca4003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fdcba7fc640 1 -- 172.21.15.46:0/3155981940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb4053dc0 con 0x7fdca4004360 2024-05-09T03:09:42.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fda35fee640 1 Processor -- start 2024-05-09T03:09:42.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f8ac9ffb640 1 -- 172.21.15.46:0/786885552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4039070 con 0x7f8acc0b6210 2024-05-09T03:09:42.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7f924c3f3640 1 -- 172.21.15.46:0/491230572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9244160790 msgr2=0x7f92440732d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7f924c3f3640 1 --2- 172.21.15.46:0/491230572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9244160790 0x7f92440732d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9240010b90 tx=0x7f9240032b40 comp rx=0 tx=0).stop 2024-05-09T03:09:42.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 -- 172.21.15.46:0/367408309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168052fb0 msgr2=0x7fc16804f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 --2- 172.21.15.46:0/367408309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168052fb0 0x7fc16804f340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc17006be50 tx=0x7fc1700a7020 comp rx=0 tx=0).stop 2024-05-09T03:09:42.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1174008941 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60057be0 con 0x7f7e600e47b0 2024-05-09T03:09:42.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f924c3f3640 1 -- 172.21.15.46:0/491230572 shutdown_connections 2024-05-09T03:09:42.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f924c3f3640 1 --2- 172.21.15.46:0/491230572 >> v2:172.21.15.88:3300/0 conn(0x7f9244073810 0x7f92440cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f924c3f3640 1 --2- 172.21.15.46:0/491230572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9244160790 0x7f92440732d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f924c3f3640 1 -- 172.21.15.46:0/491230572 >> 172.21.15.46:0/491230572 conn(0x7f92440ce370 msgr2=0x7f92440ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 -- 172.21.15.46:0/367408309 shutdown_connections 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 --2- 172.21.15.46:0/367408309 >> v2:172.21.15.88:3300/0 conn(0x7fc168034180 0x7fc168038340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fb5b0049640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a80b2a70 con 0x7fb5a80b2490 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fb5b0049640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a806d250 con 0x7fb5a80b3790 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fb5adde5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5a80b2490 0x7fb5a8164500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fb5ad5e4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb5a80ce270 0x7fb5a806d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:35212/0 (socket says 172.21.15.46:35212) 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.146+0000 7fb5ad5e4640 1 -- 172.21.15.46:0/2643715324 learned_addr learned my addr 172.21.15.46:0/2643715324 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fb5adde5640 1 -- 172.21.15.46:0/2643715324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80b3790 msgr2=0x7fb5a80b3ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fb5adde5640 1 --2- 172.21.15.46:0/2643715324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80b3790 0x7fb5a80b3ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fb5adde5640 1 -- 172.21.15.46:0/2643715324 >> v1:172.21.15.88:6789/0 conn(0x7fb5a80ce270 legacy=0x7fb5a806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fb5adde5640 1 -- 172.21.15.46:0/2643715324 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a8068880 con 0x7fb5a80b2490 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fb5adde5640 1 --2- 172.21.15.46:0/2643715324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b2490 0x7fb5a8164500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb598009a00 tx=0x7fb59802a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bc65157a309848 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fb5acde3640 1 -- 172.21.15.46:0/2643715324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598037070 con 0x7fb5a80b2490 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fb5acde3640 1 -- 172.21.15.46:0/2643715324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59802ae20 con 0x7fb5a80b2490 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5acde3640 1 -- 172.21.15.46:0/2643715324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598032660 con 0x7fb5a80b2490 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f8ac9ffb640 1 -- 172.21.15.46:0/786885552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac4034410 con 0x7f8acc0b6210 2024-05-09T03:09:42.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f8ac9ffb640 1 -- 172.21.15.46:0/786885552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40347e0 con 0x7f8acc0b6210 2024-05-09T03:09:42.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 --2- 172.21.15.46:0/367408309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168052fb0 0x7fc16804f340 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 -- 172.21.15.46:0/367408309 >> 172.21.15.46:0/367408309 conn(0x7fc168054cf0 msgr2=0x7fc168055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f924c3f3640 1 -- 172.21.15.46:0/491230572 shutdown_connections 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.147+0000 7fda35fee640 1 -- start start 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fda35fee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fda2804f330 0x7fda280e7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.149+0000 7fda35fee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda28033940 0x7fda28033d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda35fee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda28034290 con 0x7fda28048310 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda35fee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fda28053200 con 0x7fda2804f330 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda35fee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fda28047dc0 con 0x7fda28033940 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fda2804f330 0x7fda280e7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fda2804f330 0x7fda280e7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:56998/0 (socket says 172.21.15.46:56998) 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2f7fe640 1 -- 172.21.15.46:0/2713716044 learned_addr learned my addr 172.21.15.46:0/2713716044 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2ffff640 1 --2- 172.21.15.46:0/2713716044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda28033940 0x7fda28033d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2ffff640 1 -- 172.21.15.46:0/2713716044 >> v2:172.21.15.88:3300/0 conn(0x7fda2804f330 msgr2=0x7fda280e7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2ffff640 1 --2- 172.21.15.46:0/2713716044 >> v2:172.21.15.88:3300/0 conn(0x7fda2804f330 0x7fda280e7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2ffff640 1 -- 172.21.15.46:0/2713716044 >> v1:172.21.15.88:6789/0 conn(0x7fda28048310 legacy=0x7fda280382a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fda2ffff640 1 -- 172.21.15.46:0/2713716044 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda280e80c0 con 0x7fda28033940 2024-05-09T03:09:42.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 -- 172.21.15.46:0/367408309 shutdown_connections 2024-05-09T03:09:42.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fc1748a6640 1 -- 172.21.15.46:0/367408309 wait complete. 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5b0049640 1 -- 172.21.15.46:0/2643715324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b2490 msgr2=0x7fb5a8164500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5b0049640 1 --2- 172.21.15.46:0/2643715324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b2490 0x7fb5a8164500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb598009a00 tx=0x7fb59802a2e0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5b0049640 1 -- 172.21.15.46:0/2643715324 shutdown_connections 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5b0049640 1 --2- 172.21.15.46:0/2643715324 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80b3790 0x7fb5a80b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5b0049640 1 --2- 172.21.15.46:0/2643715324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b2490 0x7fb5a8164500 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7fb5b0049640 1 -- 172.21.15.46:0/2643715324 >> 172.21.15.46:0/2643715324 conn(0x7fb5a8073b10 msgr2=0x7fb5a80b7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7fb5b0049640 1 -- 172.21.15.46:0/2643715324 shutdown_connections 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7fb5b0049640 1 -- 172.21.15.46:0/2643715324 wait complete. 2024-05-09T03:09:42.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7f7e66ffd640 1 --2- 172.21.15.46:0/1174008941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 0x7f7e60037b70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009880 tx=0x7f7e5802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93f3d69e0b5146fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.150+0000 7f924c3f3640 1 -- 172.21.15.46:0/491230572 wait complete. 2024-05-09T03:09:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4004360 0x7fdca4004800 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40384e0 tx=0x7fdca4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f92b40c928bc955 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7fda2ffff640 1 --2- 172.21.15.46:0/2713716044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda28033940 0x7fda28033d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fda30000b60 tx=0x7fda300a7120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6905e817617ad144 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7f7e65ffb640 1 -- 172.21.15.46:0/1174008941 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5802ae20 con 0x7f7e600e47b0 2024-05-09T03:09:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7f7e65ffb640 1 -- 172.21.15.46:0/1174008941 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e58004030 con 0x7f7e600e47b0 2024-05-09T03:09:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7f7e65ffb640 1 -- 172.21.15.46:0/1174008941 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58004420 con 0x7f7e600e47b0 2024-05-09T03:09:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7fda2e7fc640 1 -- 172.21.15.46:0/2713716044 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda300b4070 con 0x7fda28033940 2024-05-09T03:09:42.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.151+0000 7fda2e7fc640 1 -- 172.21.15.46:0/2713716044 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda300a7d60 con 0x7fda28033940 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 -- 172.21.15.46:0/786885552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc0b6210 msgr2=0x7f8acc15d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 --2- 172.21.15.46:0/786885552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc0b6210 0x7f8acc15d100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8acc0727f0 tx=0x7f8ac402c820 comp rx=0 tx=0).stop 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 -- 172.21.15.46:0/786885552 shutdown_connections 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 --2- 172.21.15.46:0/786885552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc0b6210 0x7f8acc15d100 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 --2- 172.21.15.46:0/786885552 >> v2:172.21.15.88:3300/0 conn(0x7f8acc0b7230 0x7f8acc0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 -- 172.21.15.46:0/786885552 >> 172.21.15.46:0/786885552 conn(0x7f8acc09b7d0 msgr2=0x7f8acc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1174008941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 msgr2=0x7f7e60037b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1174008941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 0x7f7e60037b70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009880 tx=0x7f7e5802a0d0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac036070 con 0x7fdca4004360 2024-05-09T03:09:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7f8acbfff640 1 -- 172.21.15.46:0/786885552 shutdown_connections 2024-05-09T03:09:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcac031400 con 0x7fdca4004360 2024-05-09T03:09:42.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.152+0000 7fdcb9ffb640 1 -- 172.21.15.46:0/3155981940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac009630 con 0x7fdca4004360 2024-05-09T03:09:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1174008941 shutdown_connections 2024-05-09T03:09:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1174008941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600e47b0 0x7f7e60037b70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1174008941 >> 172.21.15.46:0/1174008941 conn(0x7f7e60059100 msgr2=0x7f7e60059510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7f8acbfff640 1 -- 172.21.15.46:0/786885552 wait complete. 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1174008941 shutdown_connections 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1174008941 wait complete. 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fb5b0049640 1 Processor -- start 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fda35fee640 1 -- 172.21.15.46:0/2713716044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda28033940 msgr2=0x7fda28033d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fda35fee640 1 --2- 172.21.15.46:0/2713716044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda28033940 0x7fda28033d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fda30000b60 tx=0x7fda300a7120 comp rx=0 tx=0).stop 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fda35fee640 1 -- 172.21.15.46:0/2713716044 shutdown_connections 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fda35fee640 1 --2- 172.21.15.46:0/2713716044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda28033940 0x7fda28033d50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fda35fee640 1 --2- 172.21.15.46:0/2713716044 >> v2:172.21.15.88:3300/0 conn(0x7fda2804f330 0x7fda280e7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fda35fee640 1 -- 172.21.15.46:0/2713716044 >> 172.21.15.46:0/2713716044 conn(0x7fda28054ce0 msgr2=0x7fda280550f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fc1748a6640 1 Processor -- start 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fdcbbfff640 1 -- 172.21.15.46:0/3155981940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4004360 msgr2=0x7fdca4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/3155981940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca4004360 0x7fdca4004800 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdcb40384e0 tx=0x7fdca4005830 comp rx=0 tx=0).stop 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fdcc14f0640 1 -- 172.21.15.46:0/3155981940 reap_dead start 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fdcbbfff640 1 -- 172.21.15.46:0/3155981940 shutdown_connections 2024-05-09T03:09:42.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.153+0000 7fdcbbfff640 1 -- 172.21.15.46:0/3155981940 >> 172.21.15.46:0/3155981940 conn(0x7fdcb4054ec0 msgr2=0x7fdcb40552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7fb5b0049640 1 -- start start 2024-05-09T03:09:42.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7fdcbbfff640 1 -- 172.21.15.46:0/3155981940 shutdown_connections 2024-05-09T03:09:42.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7fdcbbfff640 1 -- 172.21.15.46:0/3155981940 wait complete. 2024-05-09T03:09:42.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7fc1748a6640 1 -- start start 2024-05-09T03:09:42.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7f924c3f3640 1 Processor -- start 2024-05-09T03:09:42.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7fda35fee640 1 -- 172.21.15.46:0/2713716044 shutdown_connections 2024-05-09T03:09:42.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7f924c3f3640 1 -- start start 2024-05-09T03:09:42.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7f8acbfff640 1 Processor -- start 2024-05-09T03:09:42.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7f8acbfff640 1 -- start start 2024-05-09T03:09:42.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7fb5b0049640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 0x7fb5a8166f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7f7e6d3b4640 1 Processor -- start 2024-05-09T03:09:42.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7fc1748a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 0x7fc168181910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7fb5b0049640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 0x7fb5a80cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 0x7fc168181910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5ad5e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 0x7fb5a80cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc16f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 0x7fc168181910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56108/0 (socket says 172.21.15.46:56108) 2024-05-09T03:09:42.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc1748a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168052fb0 0x7fc1680dddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc16f7fe640 1 -- 172.21.15.46:0/1929792008 learned_addr learned my addr 172.21.15.46:0/1929792008 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5ad5e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 0x7fb5a80cfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56100/0 (socket says 172.21.15.46:56100) 2024-05-09T03:09:42.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.156+0000 7f7e6d3b4640 1 -- start start 2024-05-09T03:09:42.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5adde5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 0x7fb5a8166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fdcbbfff640 1 Processor -- start 2024-05-09T03:09:42.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5adde5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 0x7fb5a8166f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:57006/0 (socket says 172.21.15.46:57006) 2024-05-09T03:09:42.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5ad5e4640 1 -- 172.21.15.46:0/3889647499 learned_addr learned my addr 172.21.15.46:0/3889647499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80ce270 0x7fb5a81626a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc16effd640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168052fb0 0x7fc1680dddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fdcbbfff640 1 -- start start 2024-05-09T03:09:42.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680e2f00 0x7fc1680e3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7f924c3f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 0x7f92440ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80b40e0 con 0x7fb5a80b3790 2024-05-09T03:09:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80ce680 con 0x7fb5a80b2490 2024-05-09T03:09:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5ae5e6640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80ce270 0x7fb5a81626a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc168006e00 con 0x7fc168007320 2024-05-09T03:09:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fc16ffff640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680e2f00 0x7fc1680e3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a80af2c0 con 0x7fb5a80ce270 2024-05-09T03:09:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc168006b20 con 0x7fc168052fb0 2024-05-09T03:09:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924c3f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92440cf9e0 0x7f92441f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924c3f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9244160790 0x7f92441f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc168007730 con 0x7fc1680e2f00 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924a18f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 0x7f92440ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924c3f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f924415a030 con 0x7f9244160790 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924c3f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f924415d460 con 0x7f9244073810 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924a990640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9244160790 0x7f92441f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924a18f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 0x7f92440ad210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:57020/0 (socket says 172.21.15.46:57020) 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924a18f640 1 -- 172.21.15.46:0/2398524117 learned_addr learned my addr 172.21.15.46:0/2398524117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fb5ad5e4640 1 -- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80ce270 msgr2=0x7fb5a81626a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fb5ad5e4640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80ce270 0x7fb5a81626a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fb5ad5e4640 1 -- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 msgr2=0x7fb5a8166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fb5ad5e4640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 0x7fb5a8166f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fb5ad5e4640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a80d03a0 con 0x7fb5a80b3790 2024-05-09T03:09:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f92440d12e0 con 0x7f92440cf9e0 2024-05-09T03:09:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7f8acbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 0x7f8acc15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fc16ffff640 1 -- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168052fb0 msgr2=0x7fc1680dddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8acbfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8acc0b6210 0x7f8acc152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fc16ffff640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168052fb0 0x7fc1680dddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e6d3b4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 0x7f7e6000de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fc16ffff640 1 -- 172.21.15.46:0/1929792008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 msgr2=0x7fc168181910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fc16ffff640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 0x7fc168181910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e6d3b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e600384c0 0x7f7e600d4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8acaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 0x7f8acc15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fc16ffff640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1680e3850 con 0x7fc1680e2f00 2024-05-09T03:09:42.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fb5ad5e4640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 0x7fb5a80cfe60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000ce00 tx=0x7fb5a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 0x7f7e6000de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e66ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 0x7f7e6000de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43512/0 (socket says 172.21.15.46:43512) 2024-05-09T03:09:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1219988765 learned_addr learned my addr 172.21.15.46:0/1219988765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0017070 con 0x7fb5a80b3790 2024-05-09T03:09:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8acaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 0x7f8acc15af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:57034/0 (socket says 172.21.15.46:57034) 2024-05-09T03:09:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e600e47b0 0x7f7e600dd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fb5adde5640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 0x7fb5a8166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fb5ae5e6640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80ce270 0x7fb5a81626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8acaffd640 1 -- 172.21.15.46:0/2730632755 learned_addr learned my addr 172.21.15.46:0/2730632755 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e60058480 con 0x7f7e600384c0 2024-05-09T03:09:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e600581a0 con 0x7f7e600e47b0 2024-05-09T03:09:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7e600388d0 con 0x7f7e600380b0 2024-05-09T03:09:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a815c2d0 con 0x7fb5a80b3790 2024-05-09T03:09:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc154dd0 0x7f8acc1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8aca7fc640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8acc0b6210 0x7f8acc152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a81fe220 con 0x7fb5a80b3790 2024-05-09T03:09:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f7e677fe640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e600e47b0 0x7f7e600dd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc0b6620 con 0x7f8acc154dd0 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc0b10a0 con 0x7f8acc0b3c10 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8acc15d640 con 0x7f8acc0b6210 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcbbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca4004360 0x7fdcb4175f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f924998e640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92440cf9e0 0x7f92441f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f924a990640 1 -- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92440cf9e0 msgr2=0x7f92441f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f924a990640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92440cf9e0 0x7f92441f5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f924a990640 1 -- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 msgr2=0x7f92440ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f924a990640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 0x7f92440ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.159+0000 7f924a990640 1 -- 172.21.15.46:0/2398524117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92440abfc0 con 0x7f9244160790 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f924a18f640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 0x7f92440ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f924a990640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9244160790 0x7f92441f6000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f92340047c0 tx=0x7f923400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e600e47b0 msgr2=0x7f7e600dd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e66ffd640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e600e47b0 0x7f7e600dd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1219988765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e600384c0 msgr2=0x7f7e600d4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e66ffd640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e600384c0 0x7f7e600d4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e66ffd640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6017e490 con 0x7f7e600380b0 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcbbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcb404f240 0x7fdcb40da2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcbbfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40da820 0x7fdcb40d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcbbfff640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb404fbd0 con 0x7fdcb404f240 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcbbfff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb404fa50 con 0x7fdca4004360 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcbbfff640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb40e0980 con 0x7fdcb40da820 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.154+0000 7fda35fee640 1 -- 172.21.15.46:0/2713716044 wait complete. 2024-05-09T03:09:42.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acaffd640 1 -- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8acc0b6210 msgr2=0x7f8acc152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acaffd640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8acc0b6210 0x7f8acc152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acaffd640 1 -- 172.21.15.46:0/2730632755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc154dd0 msgr2=0x7f8acc1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acaffd640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc154dd0 0x7f8acc1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f8acaffd640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8acc155720 con 0x7f8acc0b3c10 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.157+0000 7fda35fee640 1 Processor -- start 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.158+0000 7fda35fee640 1 -- start start 2024-05-09T03:09:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcc14f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40da820 0x7fdcb40d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e66ffd640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 0x7f7e6000de30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e580090b0 tx=0x7f7e580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcba7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcb404f240 0x7fdcb40da2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcc14f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40da820 0x7fdcb40d4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43538/0 (socket says 172.21.15.46:43538) 2024-05-09T03:09:42.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fdcc14f0640 1 -- 172.21.15.46:0/700442097 learned_addr learned my addr 172.21.15.46:0/700442097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7fda35fee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda28033940 0x7fda280e9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.160+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58041020 con 0x7f7e600380b0 2024-05-09T03:09:42.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda35fee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 0x7fda28053ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fdcba7fc640 1 -- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40da820 msgr2=0x7fdcb40d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40da820 0x7fdcb40d4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e58003d60 con 0x7f7e600380b0 2024-05-09T03:09:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58006260 con 0x7f7e600380b0 2024-05-09T03:09:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fdcba7fc640 1 -- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca4004360 msgr2=0x7fdcb4175f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T03:09:42.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda35fee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 0x7fda280e6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca4004360 0x7fdcb4175f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fdcba7fc640 1 -- 172.21.15.46:0/700442097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb4176460 con 0x7fdcb404f240 2024-05-09T03:09:42.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda35fee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda28006170 con 0x7fda28033940 2024-05-09T03:09:42.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda35fee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fda28006aa0 con 0x7fda28048310 2024-05-09T03:09:42.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda35fee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fda28006450 con 0x7fda2804f330 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fc16f7fe640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 0x7fc168181910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 0x7fda28053ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 0x7fda280e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 0x7fda28053ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:57070/0 (socket says 172.21.15.46:57070) 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 0x7fda280e6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43544/0 (socket says 172.21.15.46:43544) 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 -- 172.21.15.46:0/512859974 learned_addr learned my addr 172.21.15.46:0/512859974 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a8076c00 con 0x7fb5a80b3790 2024-05-09T03:09:42.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fc16ffff640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680e2f00 0x7fc1680e3310 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc16804f2f0 tx=0x7fc15c00d9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2f7fe640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda28033940 0x7fda280e9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 -- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 msgr2=0x7fda280e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 0x7fda280e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 -- 172.21.15.46:0/512859974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda28033940 msgr2=0x7fda280e9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda28033940 0x7fda280e9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2effd640 1 -- 172.21.15.46:0/512859974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda280e9fb0 con 0x7fda28048310 2024-05-09T03:09:42.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2f7fe640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda28033940 0x7fda280e9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:42.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fda2ffff640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 0x7fda280e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:42.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.161+0000 7fdcba7fc640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcb404f240 0x7fdcb40da2e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdca4010230 tx=0x7fdcac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fda2effd640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 0x7fda28053ab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b420 tx=0x7fda1c00b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004030 con 0x7fc1680e2f00 2024-05-09T03:09:42.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92340107d0 con 0x7f9244160790 2024-05-09T03:09:42.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1680ded40 con 0x7fc1680e2f00 2024-05-09T03:09:42.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9234016070 con 0x7f9244160790 2024-05-09T03:09:42.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c011830 con 0x7fda28048310 2024-05-09T03:09:42.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1c016070 con 0x7fda28048310 2024-05-09T03:09:42.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92440aaf20 con 0x7f9244160790 2024-05-09T03:09:42.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16816d500 con 0x7fc1680e2f00 2024-05-09T03:09:42.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92440ab4e0 con 0x7f9244160790 2024-05-09T03:09:42.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234010b70 con 0x7f9244160790 2024-05-09T03:09:42.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15c0041d0 con 0x7fc1680e2f00 2024-05-09T03:09:42.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a000ad40 con 0x7fb5a80b3790 2024-05-09T03:09:42.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0021060 con 0x7fb5a80b3790 2024-05-09T03:09:42.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb5a001a070 con 0x7fb5a80b3790 2024-05-09T03:09:42.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e600d14d0 con 0x7f7e600380b0 2024-05-09T03:09:42.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c011c10 con 0x7fda28048310 2024-05-09T03:09:42.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e600d1a90 con 0x7f7e600380b0 2024-05-09T03:09:42.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda280ea290 con 0x7fda28048310 2024-05-09T03:09:42.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda280ea850 con 0x7fda28048310 2024-05-09T03:09:42.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004030 con 0x7fc1680e2f00 2024-05-09T03:09:42.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.164+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc150005290 con 0x7fc1680e2f00 2024-05-09T03:09:42.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac041030 con 0x7fdcb404f240 2024-05-09T03:09:42.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7f8acaffd640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 0x7f8acc15af20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000b3f0 tx=0x7f8ac000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0015020 con 0x7f8acc0b3c10 2024-05-09T03:09:42.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.164+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8acc159190 con 0x7f8acc0b3c10 2024-05-09T03:09:42.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.164+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8acc159750 con 0x7f8acc0b3c10 2024-05-09T03:09:42.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.162+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcac037040 con 0x7fdcb404f240 2024-05-09T03:09:42.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.163+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac036070 con 0x7fdcb404f240 2024-05-09T03:09:42.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.164+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb40dac30 con 0x7fdcb404f240 2024-05-09T03:09:42.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.164+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb40d89b0 con 0x7fdcb404f240 2024-05-09T03:09:42.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.164+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924406a270 con 0x7f9244160790 2024-05-09T03:09:42.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.165+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac0011830 con 0x7f8acc0b3c10 2024-05-09T03:09:42.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.166+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac0011c20 con 0x7f8acc0b3c10 2024-05-09T03:09:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.165+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e600058e0 con 0x7f7e600380b0 2024-05-09T03:09:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.165+0000 7fb59e7fc640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb58c076330 0x7fb58c078840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.166+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb5a0097210 con 0x7fb5a80b3790 2024-05-09T03:09:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.165+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda28005490 con 0x7fda28048310 2024-05-09T03:09:42.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.166+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7e58044020 con 0x7f7e600380b0 2024-05-09T03:09:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac005290 con 0x7f8acc0b3c10 2024-05-09T03:09:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.166+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc15c004380 con 0x7fc1680e2f00 2024-05-09T03:09:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fc15bfff640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1540764d0 0x7fc1540789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7f7e4f7fe640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e50076510 0x7f7e50078a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.166+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fda1c01b460 con 0x7fda28048310 2024-05-09T03:09:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fda1bfff640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fda140762d0 0x7fda140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fb5adde5640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb58c076330 0x7fb58c078840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc15c097890 con 0x7fc1680e2f00 2024-05-09T03:09:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7f7e667fc640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e50076510 0x7f7e50078a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7e580b8520 con 0x7f7e600380b0 2024-05-09T03:09:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fda1c05c840 con 0x7fda28048310 2024-05-09T03:09:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fda2f7fe640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fda140762d0 0x7fda140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.167+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb4005670 con 0x7fdcb404f240 2024-05-09T03:09:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.168+0000 7fc16f7fe640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1540764d0 0x7fc1540789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.168+0000 7f7e667fc640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e50076510 0x7f7e50078a20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e600e4d30 tx=0x7f7e5c006be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.168+0000 7fb5adde5640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb58c076330 0x7fb58c078840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb598034000 tx=0x7fb59802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.168+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9234010d10 con 0x7f9244160790 2024-05-09T03:09:42.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.168+0000 7fda2f7fe640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fda140762d0 0x7fda140787e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fda280516c0 tx=0x7fda24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.168+0000 7fc16f7fe640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1540764d0 0x7fc1540789e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc17002e1c0 tx=0x7fc1700ab000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f923affd640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92280763a0 0x7f92280788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8ac0018020 con 0x7f8acc0b3c10 2024-05-09T03:09:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9234097490 con 0x7f9244160790 2024-05-09T03:09:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f924a18f640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92280763a0 0x7f92280788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f8ab37fe640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ab80763a0 0x7f8ab80788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f924a18f640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92280763a0 0x7f92280788b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9240032b80 tx=0x7f9240038000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.169+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8ac0097920 con 0x7f8acc0b3c10 2024-05-09T03:09:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.170+0000 7f8aca7fc640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ab80763a0 0x7f8ab80788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.170+0000 7f8aca7fc640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ab80763a0 0x7f8ab80788b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8acc1f5160 tx=0x7f8abc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.172+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb5a00604e0 con 0x7fb5a80b3790 2024-05-09T03:09:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.173+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdcac044070 con 0x7fdcb404f240 2024-05-09T03:09:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.174+0000 7fdca37fe640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdca8076320 0x7fdca8078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.174+0000 7fdcbaffd640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdca8076320 0x7fdca8078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.174+0000 7fdcbaffd640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdca8076320 0x7fdca8078830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005fd0 tx=0x7fdcb0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.175+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1680e3850 con 0x7fc1680e2f00 2024-05-09T03:09:42.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.179+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fda1c020020 con 0x7fda28048310 2024-05-09T03:09:42.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.182+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdcac03ed50 con 0x7fdcb404f240 2024-05-09T03:09:42.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.182+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9244073c20 con 0x7f9244160790 2024-05-09T03:09:42.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.187+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7e5804b030 con 0x7f7e600380b0 2024-05-09T03:09:42.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.188+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8ac001f020 con 0x7f8acc0b3c10 2024-05-09T03:09:42.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.193+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdcb40dac30 con 0x7fdcb404f240 2024-05-09T03:09:42.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.531+0000 7fc083fc7640 1 -- 172.21.15.46:0/1517897326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc07c0741f0 con 0x7fc07c0d1740 2024-05-09T03:09:42.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.532+0000 7fc07a7fc640 1 -- 172.21.15.46:0/1517897326 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc07400f070 con 0x7fc07c0d1740 2024-05-09T03:09:42.532 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.535+0000 7fc073fff640 1 -- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc05c0763d0 msgr2=0x7fc05c0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.535+0000 7fc073fff640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc05c0763d0 0x7fc05c0788e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0725a0 tx=0x7fc064008040 comp rx=0 tx=0).stop 2024-05-09T03:09:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 -- 172.21.15.46:0/1517897326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07c0d1740 msgr2=0x7fc07c0b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07c0d1740 0x7fc07c0b04b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc0740013b0 tx=0x7fc07400cc70 comp rx=0 tx=0).stop 2024-05-09T03:09:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 -- 172.21.15.46:0/1517897326 shutdown_connections 2024-05-09T03:09:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc05c0763d0 0x7fc05c0788e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07c0d1740 0x7fc07c0b04b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc07c0ce870 0x7fc07c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 --2- 172.21.15.46:0/1517897326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc07c070280 0x7fc07c0aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.536+0000 7fc073fff640 1 -- 172.21.15.46:0/1517897326 >> 172.21.15.46:0/1517897326 conn(0x7fc07c0b4400 msgr2=0x7fc07c154090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.537+0000 7fc073fff640 1 -- 172.21.15.46:0/1517897326 shutdown_connections 2024-05-09T03:09:42.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.537+0000 7fc073fff640 1 -- 172.21.15.46:0/1517897326 wait complete. 2024-05-09T03:09:42.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-05-09T03:09:42.555 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.585+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7e6003f080 con 0x7f7e600380b0 2024-05-09T03:09:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.587+0000 7f7e4f7fe640 1 -- 172.21.15.46:0/1219988765 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7e58081dd0 con 0x7f7e600380b0 2024-05-09T03:09:42.588 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-05-09T03:09:42.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.590+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e50076510 msgr2=0x7f7e50078a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.590+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e50076510 0x7f7e50078a20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e600e4d30 tx=0x7f7e5c006be0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.590+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 msgr2=0x7f7e6000de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.590+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 0x7f7e6000de30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7e580090b0 tx=0x7f7e580046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.590+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 shutdown_connections 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.590+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e600e47b0 0x7f7e600dd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.591+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e50076510 0x7f7e50078a20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.591+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e600384c0 0x7f7e600d4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.591+0000 7f7e6d3b4640 1 --2- 172.21.15.46:0/1219988765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e600380b0 0x7f7e6000de30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.591+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 >> 172.21.15.46:0/1219988765 conn(0x7f7e60059100 msgr2=0x7f7e6003e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.591+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 shutdown_connections 2024-05-09T03:09:42.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.591+0000 7f7e6d3b4640 1 -- 172.21.15.46:0/1219988765 wait complete. 2024-05-09T03:09:42.603 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.4 2024-05-09T03:09:42.603 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.618+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fc150005580 con 0x7fc1680e2f00 2024-05-09T03:09:42.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.619+0000 7fc15bfff640 1 -- 172.21.15.46:0/1929792008 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc15c060a80 con 0x7fc1680e2f00 2024-05-09T03:09:42.619 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-05-09T03:09:42.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1540764d0 msgr2=0x7fc1540789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1540764d0 0x7fc1540789e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc17002e1c0 tx=0x7fc1700ab000 comp rx=0 tx=0).stop 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680e2f00 msgr2=0x7fc1680e3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680e2f00 0x7fc1680e3310 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc16804f2f0 tx=0x7fc15c00d9c0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 shutdown_connections 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680e2f00 0x7fc1680e3310 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168052fb0 0x7fc1680dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1540764d0 0x7fc1540789e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 --2- 172.21.15.46:0/1929792008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc168007320 0x7fc168181910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 >> 172.21.15.46:0/1929792008 conn(0x7fc168054cf0 msgr2=0x7fc168053980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.621+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 shutdown_connections 2024-05-09T03:09:42.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.622+0000 7fc1748a6640 1 -- 172.21.15.46:0/1929792008 wait complete. 2024-05-09T03:09:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.622+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fda280d71e0 con 0x7fda28048310 2024-05-09T03:09:42.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.623+0000 7fda1bfff640 1 -- 172.21.15.46:0/512859974 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fda1c060950 con 0x7fda28048310 2024-05-09T03:09:42.624 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-05-09T03:09:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fda140762d0 msgr2=0x7fda140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fda140762d0 0x7fda140787e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fda280516c0 tx=0x7fda24008040 comp rx=0 tx=0).stop 2024-05-09T03:09:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 msgr2=0x7fda28053ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 0x7fda28053ab0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fda1c00b420 tx=0x7fda1c00b8e0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 shutdown_connections 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fda2804f330 0x7fda280e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda28048310 0x7fda28053ab0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fda140762d0 0x7fda140787e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 --2- 172.21.15.46:0/512859974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda28033940 0x7fda280e9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 >> 172.21.15.46:0/512859974 conn(0x7fda28054ce0 msgr2=0x7fda28034720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 shutdown_connections 2024-05-09T03:09:42.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.627+0000 7fda35fee640 1 -- 172.21.15.46:0/512859974 wait complete. 2024-05-09T03:09:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.628+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb5a8163cb0 con 0x7fb5a80b3790 2024-05-09T03:09:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.628+0000 7fb59e7fc640 1 -- 172.21.15.46:0/3889647499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb5a00647e0 con 0x7fb5a80b3790 2024-05-09T03:09:42.630 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2024-05-09T03:09:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.630+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f92440aacb0 con 0x7f9244160790 2024-05-09T03:09:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.630+0000 7f923affd640 1 -- 172.21.15.46:0/2398524117 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9234060dd0 con 0x7f9244160790 2024-05-09T03:09:42.631 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f8aac005850 con 0x7f8acc0b3c10 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb58c076330 msgr2=0x7fb58c078840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb58c076330 0x7fb58c078840 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb598034000 tx=0x7fb59802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 msgr2=0x7fb5a80cfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 0x7fb5a80cfe60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000ce00 tx=0x7fb5a000a430 comp rx=0 tx=0).stop 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 shutdown_connections 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5a80ce270 0x7fb5a81626a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb58c076330 0x7fb58c078840 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a80b3790 0x7fb5a80cfe60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 --2- 172.21.15.46:0/3889647499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5a80b2490 0x7fb5a8166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 >> 172.21.15.46:0/3889647499 conn(0x7fb5a8073b10 msgr2=0x7fb5a80b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.632+0000 7f8ab37fe640 1 -- 172.21.15.46:0/2730632755 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8acc155720 con 0x7f8acc0b3c10 2024-05-09T03:09:42.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.633+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 shutdown_connections 2024-05-09T03:09:42.634 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2024-05-09T03:09:42.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.633+0000 7fb5b0049640 1 -- 172.21.15.46:0/3889647499 wait complete. 2024-05-09T03:09:42.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92280763a0 msgr2=0x7f92280788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92280763a0 0x7f92280788b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9240032b80 tx=0x7f9240038000 comp rx=0 tx=0).stop 2024-05-09T03:09:42.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9244160790 msgr2=0x7f92441f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9244160790 0x7f92441f6000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f92340047c0 tx=0x7f923400c890 comp rx=0 tx=0).stop 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 shutdown_connections 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92280763a0 0x7f92280788b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9244160790 0x7f92441f6000 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92440cf9e0 0x7f92441f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 --2- 172.21.15.46:0/2398524117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9244073810 0x7f92440ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 >> 172.21.15.46:0/2398524117 conn(0x7f92440ce370 msgr2=0x7f92440b5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 shutdown_connections 2024-05-09T03:09:42.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.634+0000 7f924c3f3640 1 -- 172.21.15.46:0/2398524117 wait complete. 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ab80763a0 msgr2=0x7f8ab80788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ab80763a0 0x7f8ab80788b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8acc1f5160 tx=0x7f8abc008040 comp rx=0 tx=0).stop 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 msgr2=0x7f8acc15af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 0x7f8acc15af20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000b3f0 tx=0x7f8ac000b8b0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 shutdown_connections 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8ab80763a0 0x7f8ab80788b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8acc154dd0 0x7f8acc1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8acc0b6210 0x7f8acc152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 --2- 172.21.15.46:0/2730632755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8acc0b3c10 0x7f8acc15af20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 >> 172.21.15.46:0/2730632755 conn(0x7f8acc09b7d0 msgr2=0x7f8acc0b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.638 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2024-05-09T03:09:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 shutdown_connections 2024-05-09T03:09:42.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.636+0000 7f8acbfff640 1 -- 172.21.15.46:0/2730632755 wait complete. 2024-05-09T03:09:42.638 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.644+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fdcb40535f0 con 0x7fdcb404f240 2024-05-09T03:09:42.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.645+0000 7fdca37fe640 1 -- 172.21.15.46:0/700442097 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdcb4176460 con 0x7fdcb404f240 2024-05-09T03:09:42.645 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2024-05-09T03:09:42.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.0 2024-05-09T03:09:42.646 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdca8076320 msgr2=0x7fdca8078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdca8076320 0x7fdca8078830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005fd0 tx=0x7fdcb0009040 comp rx=0 tx=0).stop 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcb404f240 msgr2=0x7fdcb40da2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcb404f240 0x7fdcb40da2e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdca4010230 tx=0x7fdcac004730 comp rx=0 tx=0).stop 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 shutdown_connections 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdcb40da820 0x7fdcb40d4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdca8076320 0x7fdca8078830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcb404f240 0x7fdcb40da2e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 --2- 172.21.15.46:0/700442097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca4004360 0x7fdcb4175f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 >> 172.21.15.46:0/700442097 conn(0x7fdcb4054ec0 msgr2=0x7fdcb4005a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 shutdown_connections 2024-05-09T03:09:42.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.648+0000 7fdcbbfff640 1 -- 172.21.15.46:0/700442097 wait complete. 2024-05-09T03:09:42.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.5 2024-05-09T03:09:42.651 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2024-05-09T03:09:42.651 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.3 2024-05-09T03:09:42.654 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2024-05-09T03:09:42.661 DEBUG:teuthology.parallel:result is None 2024-05-09T03:09:42.661 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:09:42.661 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:09:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.815+0000 7f27a0029640 1 Processor -- start 2024-05-09T03:09:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.815+0000 7f27a0029640 1 -- start start 2024-05-09T03:09:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f27a0029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f27980b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f27a0029640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27980b7d20 0x7f27980b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f27a0029640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27980b8130 con 0x7f27980d57a0 2024-05-09T03:09:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f27a0029640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f27980d5bb0 con 0x7f27980b7d20 2024-05-09T03:09:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f27a0029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27980d49e0 con 0x7f2798160610 2024-05-09T03:09:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f279e5c6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27980b7d20 0x7f27980b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.816+0000 7f279d5c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f27980b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279d5c4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f27980b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43548/0 (socket says 172.21.15.46:43548) 2024-05-09T03:09:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279d5c4640 1 -- 172.21.15.46:0/2963351619 learned_addr learned my addr 172.21.15.46:0/2963351619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279d5c4640 1 -- 172.21.15.46:0/2963351619 >> v2:172.21.15.88:3300/0 conn(0x7f27980b7d20 msgr2=0x7f27980b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279d5c4640 1 --2- 172.21.15.46:0/2963351619 >> v2:172.21.15.88:3300/0 conn(0x7f27980b7d20 0x7f27980b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279d5c4640 1 -- 172.21.15.46:0/2963351619 >> v1:172.21.15.88:6789/0 conn(0x7f27980d57a0 legacy=0x7f27980b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279d5c4640 1 -- 172.21.15.46:0/2963351619 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27980d4cc0 con 0x7f2798160610 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279cdc3640 1 -- 172.21.15.46:0/2963351619 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3995626847 0 0) 0x7f2788001490 con 0x7f27980d57a0 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.817+0000 7f279e5c6640 1 --2- 172.21.15.46:0/2963351619 >> v2:172.21.15.88:3300/0 conn(0x7f27980b7d20 0x7f27980b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.818+0000 7f279d5c4640 1 --2- 172.21.15.46:0/2963351619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f27980b77e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f278c009870 tx=0x7f278c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=897f20f8cca109e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.818+0000 7f279cdc3640 1 -- 172.21.15.46:0/2963351619 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c037070 con 0x7f2798160610 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.818+0000 7f279cdc3640 1 -- 172.21.15.46:0/2963351619 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f278c032440 con 0x7f2798160610 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.818+0000 7f279cdc3640 1 -- 172.21.15.46:0/2963351619 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c032810 con 0x7f2798160610 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 -- 172.21.15.46:0/2963351619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 msgr2=0x7f27980b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 --2- 172.21.15.46:0/2963351619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f27980b77e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f278c009870 tx=0x7f278c02a270 comp rx=0 tx=0).stop 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 -- 172.21.15.46:0/2963351619 shutdown_connections 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 --2- 172.21.15.46:0/2963351619 >> v2:172.21.15.88:3300/0 conn(0x7f27980b7d20 0x7f27980b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 --2- 172.21.15.46:0/2963351619 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f27980b77e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 -- 172.21.15.46:0/2963351619 >> 172.21.15.46:0/2963351619 conn(0x7f27980743e0 msgr2=0x7f27980d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 -- 172.21.15.46:0/2963351619 shutdown_connections 2024-05-09T03:09:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.819+0000 7f27a0029640 1 -- 172.21.15.46:0/2963351619 wait complete. 2024-05-09T03:09:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.820+0000 7f27a0029640 1 Processor -- start 2024-05-09T03:09:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.820+0000 7f27a0029640 1 -- start start 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f27a0029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27980b7d20 0x7f2798159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f27a0029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 0x7f27981450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f27a0029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f2798145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f27a0029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2798160a20 con 0x7f27980d57a0 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f27a0029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27980d5280 con 0x7f27980b7d20 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f27a0029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27980d15b0 con 0x7f2798160610 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 0x7f27981450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 0x7f27981450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56152/0 (socket says 172.21.15.46:56152) 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 -- 172.21.15.46:0/3258996534 learned_addr learned my addr 172.21.15.46:0/3258996534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279ddc5640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27980b7d20 0x7f2798159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279e5c6640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f2798145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 -- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 msgr2=0x7f2798145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f2798145630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 -- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27980b7d20 msgr2=0x7f2798159a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27980b7d20 0x7f2798159a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.822+0000 7f279d5c4640 1 -- 172.21.15.46:0/3258996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2798159fd0 con 0x7f27980d57a0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.823+0000 7f279d5c4640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 0x7f27981450f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f278c0090b0 tx=0x7f278c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.823+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c037040 con 0x7f27980d57a0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.823+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f278c004030 con 0x7f27980d57a0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.823+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2798144330 con 0x7f27980d57a0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.823+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c004420 con 0x7f27980d57a0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.823+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27981448f0 con 0x7f27980d57a0 2024-05-09T03:09:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.824+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2798076490 con 0x7f27980d57a0 2024-05-09T03:09:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.825+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f278c03a070 con 0x7f27980d57a0 2024-05-09T03:09:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.825+0000 7f27967fc640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27840762d0 0x7f27840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.825+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f278c0b7a20 con 0x7f27980d57a0 2024-05-09T03:09:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.826+0000 7f279ddc5640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27840762d0 0x7f27840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.826+0000 7f279ddc5640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27840762d0 0x7f27840787e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f27880012e0 tx=0x7f2788009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:42.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:42.830+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2798076490 con 0x7f27980d57a0 2024-05-09T03:09:43.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.019+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f279806a270 con 0x7f27840762d0 2024-05-09T03:09:43.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.026+0000 7f27967fc640 1 -- 172.21.15.46:0/3258996534 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f279806a270 con 0x7f27840762d0 2024-05-09T03:09:43.029 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:43.030 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27840762d0 msgr2=0x7f27840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27840762d0 0x7f27840787e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f27880012e0 tx=0x7f2788009040 comp rx=0 tx=0).stop 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 msgr2=0x7f27981450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 0x7f27981450f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f278c0090b0 tx=0x7f278c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 shutdown_connections 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798160610 0x7f2798145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27840762d0 0x7f27840787e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27980d57a0 0x7f27981450f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 --2- 172.21.15.46:0/3258996534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27980b7d20 0x7f2798159a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 >> 172.21.15.46:0/3258996534 conn(0x7f27980743e0 msgr2=0x7f27980bae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:43.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.029+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 shutdown_connections 2024-05-09T03:09:43.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.030+0000 7f27a0029640 1 -- 172.21.15.46:0/3258996534 wait complete. 2024-05-09T03:09:43.041 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-05-09T03:09:42.492701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235624,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739096,"statfs":{"total":773094113280,"available":772852834304,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.320665"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914921+0000","last_change":"2024-05-09T03:09:34.191121+0000","last_active":"2024-05-09T03:09:36.914921+0000","last_peered":"2024-05-09T03:09:36.914921+0000","last_clean":"2024-05-09T03:09:36.914921+0000","last_became_active":"2024-05-09T03:09:34.189608+0000","last_became_peered":"2024-05-09T03:09:34.189608+0000","last_unstale":"2024-05-09T03:09:36.914921+0000","last_undegraded":"2024-05-09T03:09:36.914921+0000","last_fullsized":"2024-05-09T03:09:36.914921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:49.231849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191404+0000","last_change":"2024-05-09T03:09:34.200351+0000","last_active":"2024-05-09T03:09:35.191404+0000","last_peered":"2024-05-09T03:09:35.191404+0000","last_clean":"2024-05-09T03:09:35.191404+0000","last_became_active":"2024-05-09T03:09:34.200009+0000","last_became_peered":"2024-05-09T03:09:34.200009+0000","last_unstale":"2024-05-09T03:09:35.191404+0000","last_undegraded":"2024-05-09T03:09:35.191404+0000","last_fullsized":"2024-05-09T03:09:35.191404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914510+0000","last_change":"2024-05-09T03:09:34.204156+0000","last_active":"2024-05-09T03:09:36.914510+0000","last_peered":"2024-05-09T03:09:36.914510+0000","last_clean":"2024-05-09T03:09:36.914510+0000","last_became_active":"2024-05-09T03:09:34.203843+0000","last_became_peered":"2024-05-09T03:09:34.203843+0000","last_unstale":"2024-05-09T03:09:36.914510+0000","last_undegraded":"2024-05-09T03:09:36.914510+0000","last_fullsized":"2024-05-09T03:09:36.914510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:16.118667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223412+0000","last_change":"2024-05-09T03:09:34.206782+0000","last_active":"2024-05-09T03:09:35.223412+0000","last_peered":"2024-05-09T03:09:35.223412+0000","last_clean":"2024-05-09T03:09:35.223412+0000","last_became_active":"2024-05-09T03:09:34.206376+0000","last_became_peered":"2024-05-09T03:09:34.206376+0000","last_unstale":"2024-05-09T03:09:35.223412+0000","last_undegraded":"2024-05-09T03:09:35.223412+0000","last_fullsized":"2024-05-09T03:09:35.223412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:33.100263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914552+0000","last_change":"2024-05-09T03:09:34.203941+0000","last_active":"2024-05-09T03:09:36.914552+0000","last_peered":"2024-05-09T03:09:36.914552+0000","last_clean":"2024-05-09T03:09:36.914552+0000","last_became_active":"2024-05-09T03:09:34.203530+0000","last_became_peered":"2024-05-09T03:09:34.203530+0000","last_unstale":"2024-05-09T03:09:36.914552+0000","last_undegraded":"2024-05-09T03:09:36.914552+0000","last_fullsized":"2024-05-09T03:09:36.914552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223482+0000","last_change":"2024-05-09T03:09:34.201477+0000","last_active":"2024-05-09T03:09:35.223482+0000","last_peered":"2024-05-09T03:09:35.223482+0000","last_clean":"2024-05-09T03:09:35.223482+0000","last_became_active":"2024-05-09T03:09:34.200622+0000","last_became_peered":"2024-05-09T03:09:34.200622+0000","last_unstale":"2024-05-09T03:09:35.223482+0000","last_undegraded":"2024-05-09T03:09:35.223482+0000","last_fullsized":"2024-05-09T03:09:35.223482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914369+0000","last_change":"2024-05-09T03:09:34.190313+0000","last_active":"2024-05-09T03:09:36.914369+0000","last_peered":"2024-05-09T03:09:36.914369+0000","last_clean":"2024-05-09T03:09:36.914369+0000","last_became_active":"2024-05-09T03:09:34.189973+0000","last_became_peered":"2024-05-09T03:09:34.189973+0000","last_unstale":"2024-05-09T03:09:36.914369+0000","last_undegraded":"2024-05-09T03:09:36.914369+0000","last_fullsized":"2024-05-09T03:09:36.914369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913936+0000","last_change":"2024-05-09T03:09:34.206587+0000","last_active":"2024-05-09T03:09:36.913936+0000","last_peered":"2024-05-09T03:09:36.913936+0000","last_clean":"2024-05-09T03:09:36.913936+0000","last_became_active":"2024-05-09T03:09:34.206317+0000","last_became_peered":"2024-05-09T03:09:34.206317+0000","last_unstale":"2024-05-09T03:09:36.913936+0000","last_undegraded":"2024-05-09T03:09:36.913936+0000","last_fullsized":"2024-05-09T03:09:36.913936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914964+0000","last_change":"2024-05-09T03:09:34.202532+0000","last_active":"2024-05-09T03:09:36.914964+0000","last_peered":"2024-05-09T03:09:36.914964+0000","last_clean":"2024-05-09T03:09:36.914964+0000","last_became_active":"2024-05-09T03:09:34.202253+0000","last_became_peered":"2024-05-09T03:09:34.202253+0000","last_unstale":"2024-05-09T03:09:36.914964+0000","last_undegraded":"2024-05-09T03:09:36.914964+0000","last_fullsized":"2024-05-09T03:09:36.914964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.707927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195921+0000","last_change":"2024-05-09T03:09:34.199946+0000","last_active":"2024-05-09T03:09:35.195921+0000","last_peered":"2024-05-09T03:09:35.195921+0000","last_clean":"2024-05-09T03:09:35.195921+0000","last_became_active":"2024-05-09T03:09:34.199583+0000","last_became_peered":"2024-05-09T03:09:34.199583+0000","last_unstale":"2024-05-09T03:09:35.195921+0000","last_undegraded":"2024-05-09T03:09:35.195921+0000","last_fullsized":"2024-05-09T03:09:35.195921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194810+0000","last_change":"2024-05-09T03:09:34.199326+0000","last_active":"2024-05-09T03:09:35.194810+0000","last_peered":"2024-05-09T03:09:35.194810+0000","last_clean":"2024-05-09T03:09:35.194810+0000","last_became_active":"2024-05-09T03:09:34.198952+0000","last_became_peered":"2024-05-09T03:09:34.198952+0000","last_unstale":"2024-05-09T03:09:35.194810+0000","last_undegraded":"2024-05-09T03:09:35.194810+0000","last_fullsized":"2024-05-09T03:09:35.194810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195230+0000","last_change":"2024-05-09T03:09:34.186397+0000","last_active":"2024-05-09T03:09:35.195230+0000","last_peered":"2024-05-09T03:09:35.195230+0000","last_clean":"2024-05-09T03:09:35.195230+0000","last_became_active":"2024-05-09T03:09:34.186102+0000","last_became_peered":"2024-05-09T03:09:34.186102+0000","last_unstale":"2024-05-09T03:09:35.195230+0000","last_undegraded":"2024-05-09T03:09:35.195230+0000","last_fullsized":"2024-05-09T03:09:35.195230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:17.126340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915391+0000","last_change":"2024-05-09T03:09:34.198335+0000","last_active":"2024-05-09T03:09:36.915391+0000","last_peered":"2024-05-09T03:09:36.915391+0000","last_clean":"2024-05-09T03:09:36.915391+0000","last_became_active":"2024-05-09T03:09:34.197960+0000","last_became_peered":"2024-05-09T03:09:34.197960+0000","last_unstale":"2024-05-09T03:09:36.915391+0000","last_undegraded":"2024-05-09T03:09:36.915391+0000","last_fullsized":"2024-05-09T03:09:36.915391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914604+0000","last_change":"2024-05-09T03:09:34.189212+0000","last_active":"2024-05-09T03:09:36.914604+0000","last_peered":"2024-05-09T03:09:36.914604+0000","last_clean":"2024-05-09T03:09:36.914604+0000","last_became_active":"2024-05-09T03:09:34.188580+0000","last_became_peered":"2024-05-09T03:09:34.188580+0000","last_unstale":"2024-05-09T03:09:36.914604+0000","last_undegraded":"2024-05-09T03:09:36.914604+0000","last_fullsized":"2024-05-09T03:09:36.914604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:20.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191292+0000","last_change":"2024-05-09T03:09:34.186444+0000","last_active":"2024-05-09T03:09:35.191292+0000","last_peered":"2024-05-09T03:09:35.191292+0000","last_clean":"2024-05-09T03:09:35.191292+0000","last_became_active":"2024-05-09T03:09:34.186119+0000","last_became_peered":"2024-05-09T03:09:34.186119+0000","last_unstale":"2024-05-09T03:09:35.191292+0000","last_undegraded":"2024-05-09T03:09:35.191292+0000","last_fullsized":"2024-05-09T03:09:35.191292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223239+0000","last_change":"2024-05-09T03:09:34.201163+0000","last_active":"2024-05-09T03:09:35.223239+0000","last_peered":"2024-05-09T03:09:35.223239+0000","last_clean":"2024-05-09T03:09:35.223239+0000","last_became_active":"2024-05-09T03:09:34.200814+0000","last_became_peered":"2024-05-09T03:09:34.200814+0000","last_unstale":"2024-05-09T03:09:35.223239+0000","last_undegraded":"2024-05-09T03:09:35.223239+0000","last_fullsized":"2024-05-09T03:09:35.223239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223892+0000","last_change":"2024-05-09T03:09:34.201044+0000","last_active":"2024-05-09T03:09:35.223892+0000","last_peered":"2024-05-09T03:09:35.223892+0000","last_clean":"2024-05-09T03:09:35.223892+0000","last_became_active":"2024-05-09T03:09:34.200392+0000","last_became_peered":"2024-05-09T03:09:34.200392+0000","last_unstale":"2024-05-09T03:09:35.223892+0000","last_undegraded":"2024-05-09T03:09:35.223892+0000","last_fullsized":"2024-05-09T03:09:35.223892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:49.231849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914697+0000","last_change":"2024-05-09T03:09:34.203929+0000","last_active":"2024-05-09T03:09:36.914697+0000","last_peered":"2024-05-09T03:09:36.914697+0000","last_clean":"2024-05-09T03:09:36.914697+0000","last_became_active":"2024-05-09T03:09:34.203562+0000","last_became_peered":"2024-05-09T03:09:34.203562+0000","last_unstale":"2024-05-09T03:09:36.914697+0000","last_undegraded":"2024-05-09T03:09:36.914697+0000","last_fullsized":"2024-05-09T03:09:36.914697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915098+0000","last_change":"2024-05-09T03:09:34.204594+0000","last_active":"2024-05-09T03:09:36.915098+0000","last_peered":"2024-05-09T03:09:36.915098+0000","last_clean":"2024-05-09T03:09:36.915098+0000","last_became_active":"2024-05-09T03:09:34.204289+0000","last_became_peered":"2024-05-09T03:09:34.204289+0000","last_unstale":"2024-05-09T03:09:36.915098+0000","last_undegraded":"2024-05-09T03:09:36.915098+0000","last_fullsized":"2024-05-09T03:09:36.915098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:09.418002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195935+0000","last_change":"2024-05-09T03:09:34.186570+0000","last_active":"2024-05-09T03:09:35.195935+0000","last_peered":"2024-05-09T03:09:35.195935+0000","last_clean":"2024-05-09T03:09:35.195935+0000","last_became_active":"2024-05-09T03:09:34.186249+0000","last_became_peered":"2024-05-09T03:09:34.186249+0000","last_unstale":"2024-05-09T03:09:35.195935+0000","last_undegraded":"2024-05-09T03:09:35.195935+0000","last_fullsized":"2024-05-09T03:09:35.195935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914719+0000","last_change":"2024-05-09T03:09:34.195784+0000","last_active":"2024-05-09T03:09:36.914719+0000","last_peered":"2024-05-09T03:09:36.914719+0000","last_clean":"2024-05-09T03:09:36.914719+0000","last_became_active":"2024-05-09T03:09:34.195111+0000","last_became_peered":"2024-05-09T03:09:34.195111+0000","last_unstale":"2024-05-09T03:09:36.914719+0000","last_undegraded":"2024-05-09T03:09:36.914719+0000","last_fullsized":"2024-05-09T03:09:36.914719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:43.778183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223417+0000","last_change":"2024-05-09T03:09:34.201228+0000","last_active":"2024-05-09T03:09:35.223417+0000","last_peered":"2024-05-09T03:09:35.223417+0000","last_clean":"2024-05-09T03:09:35.223417+0000","last_became_active":"2024-05-09T03:09:34.200894+0000","last_became_peered":"2024-05-09T03:09:34.200894+0000","last_unstale":"2024-05-09T03:09:35.223417+0000","last_undegraded":"2024-05-09T03:09:35.223417+0000","last_fullsized":"2024-05-09T03:09:35.223417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914339+0000","last_change":"2024-05-09T03:09:34.189092+0000","last_active":"2024-05-09T03:09:36.914339+0000","last_peered":"2024-05-09T03:09:36.914339+0000","last_clean":"2024-05-09T03:09:36.914339+0000","last_became_active":"2024-05-09T03:09:34.188774+0000","last_became_peered":"2024-05-09T03:09:34.188774+0000","last_unstale":"2024-05-09T03:09:36.914339+0000","last_undegraded":"2024-05-09T03:09:36.914339+0000","last_fullsized":"2024-05-09T03:09:36.914339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223434+0000","last_change":"2024-05-09T03:09:34.201623+0000","last_active":"2024-05-09T03:09:35.223434+0000","last_peered":"2024-05-09T03:09:35.223434+0000","last_clean":"2024-05-09T03:09:35.223434+0000","last_became_active":"2024-05-09T03:09:34.200070+0000","last_became_peered":"2024-05-09T03:09:34.200070+0000","last_unstale":"2024-05-09T03:09:35.223434+0000","last_undegraded":"2024-05-09T03:09:35.223434+0000","last_fullsized":"2024-05-09T03:09:35.223434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915317+0000","last_change":"2024-05-09T03:09:34.189949+0000","last_active":"2024-05-09T03:09:36.915317+0000","last_peered":"2024-05-09T03:09:36.915317+0000","last_clean":"2024-05-09T03:09:36.915317+0000","last_became_active":"2024-05-09T03:09:34.189595+0000","last_became_peered":"2024-05-09T03:09:34.189595+0000","last_unstale":"2024-05-09T03:09:36.915317+0000","last_undegraded":"2024-05-09T03:09:36.915317+0000","last_fullsized":"2024-05-09T03:09:36.915317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:33.100263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.196011+0000","last_change":"2024-05-09T03:09:34.188475+0000","last_active":"2024-05-09T03:09:35.196011+0000","last_peered":"2024-05-09T03:09:35.196011+0000","last_clean":"2024-05-09T03:09:35.196011+0000","last_became_active":"2024-05-09T03:09:34.188128+0000","last_became_peered":"2024-05-09T03:09:34.188128+0000","last_unstale":"2024-05-09T03:09:35.196011+0000","last_undegraded":"2024-05-09T03:09:35.196011+0000","last_fullsized":"2024-05-09T03:09:35.196011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191594+0000","last_change":"2024-05-09T03:09:34.191453+0000","last_active":"2024-05-09T03:09:35.191594+0000","last_peered":"2024-05-09T03:09:35.191594+0000","last_clean":"2024-05-09T03:09:35.191594+0000","last_became_active":"2024-05-09T03:09:34.191094+0000","last_became_peered":"2024-05-09T03:09:34.191094+0000","last_unstale":"2024-05-09T03:09:35.191594+0000","last_undegraded":"2024-05-09T03:09:35.191594+0000","last_fullsized":"2024-05-09T03:09:35.191594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915216+0000","last_change":"2024-05-09T03:09:34.201091+0000","last_active":"2024-05-09T03:09:36.915216+0000","last_peered":"2024-05-09T03:09:36.915216+0000","last_clean":"2024-05-09T03:09:36.915216+0000","last_became_active":"2024-05-09T03:09:34.198235+0000","last_became_peered":"2024-05-09T03:09:34.198235+0000","last_unstale":"2024-05-09T03:09:36.915216+0000","last_undegraded":"2024-05-09T03:09:36.915216+0000","last_fullsized":"2024-05-09T03:09:36.915216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914899+0000","last_change":"2024-05-09T03:09:34.196239+0000","last_active":"2024-05-09T03:09:36.914899+0000","last_peered":"2024-05-09T03:09:36.914899+0000","last_clean":"2024-05-09T03:09:36.914899+0000","last_became_active":"2024-05-09T03:09:34.195895+0000","last_became_peered":"2024-05-09T03:09:34.195895+0000","last_unstale":"2024-05-09T03:09:36.914899+0000","last_undegraded":"2024-05-09T03:09:36.914899+0000","last_fullsized":"2024-05-09T03:09:36.914899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223858+0000","last_change":"2024-05-09T03:09:34.206560+0000","last_active":"2024-05-09T03:09:35.223858+0000","last_peered":"2024-05-09T03:09:35.223858+0000","last_clean":"2024-05-09T03:09:35.223858+0000","last_became_active":"2024-05-09T03:09:34.205417+0000","last_became_peered":"2024-05-09T03:09:34.205417+0000","last_unstale":"2024-05-09T03:09:35.223858+0000","last_undegraded":"2024-05-09T03:09:35.223858+0000","last_fullsized":"2024-05-09T03:09:35.223858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:59:37.733696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913999+0000","last_change":"2024-05-09T03:09:34.206717+0000","last_active":"2024-05-09T03:09:36.913999+0000","last_peered":"2024-05-09T03:09:36.913999+0000","last_clean":"2024-05-09T03:09:36.913999+0000","last_became_active":"2024-05-09T03:09:34.206391+0000","last_became_peered":"2024-05-09T03:09:34.206391+0000","last_unstale":"2024-05-09T03:09:36.913999+0000","last_undegraded":"2024-05-09T03:09:36.913999+0000","last_fullsized":"2024-05-09T03:09:36.913999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194450+0000","last_change":"2024-05-09T03:09:34.181522+0000","last_active":"2024-05-09T03:09:35.194450+0000","last_peered":"2024-05-09T03:09:35.194450+0000","last_clean":"2024-05-09T03:09:35.194450+0000","last_became_active":"2024-05-09T03:09:34.180437+0000","last_became_peered":"2024-05-09T03:09:34.180437+0000","last_unstale":"2024-05-09T03:09:35.194450+0000","last_undegraded":"2024-05-09T03:09:35.194450+0000","last_fullsized":"2024-05-09T03:09:35.194450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194687+0000","last_change":"2024-05-09T03:09:34.187744+0000","last_active":"2024-05-09T03:09:35.194687+0000","last_peered":"2024-05-09T03:09:35.194687+0000","last_clean":"2024-05-09T03:09:35.194687+0000","last_became_active":"2024-05-09T03:09:34.187413+0000","last_became_peered":"2024-05-09T03:09:34.187413+0000","last_unstale":"2024-05-09T03:09:35.194687+0000","last_undegraded":"2024-05-09T03:09:35.194687+0000","last_fullsized":"2024-05-09T03:09:35.194687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:00.147452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915534+0000","last_change":"2024-05-09T03:09:33.189245+0000","last_active":"2024-05-09T03:09:36.915534+0000","last_peered":"2024-05-09T03:09:36.915534+0000","last_clean":"2024-05-09T03:09:36.915534+0000","last_became_active":"2024-05-09T03:09:33.188865+0000","last_became_peered":"2024-05-09T03:09:33.188865+0000","last_unstale":"2024-05-09T03:09:36.915534+0000","last_undegraded":"2024-05-09T03:09:36.915534+0000","last_fullsized":"2024-05-09T03:09:36.915534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915442+0000","last_change":"2024-05-09T03:09:34.183515+0000","last_active":"2024-05-09T03:09:36.915442+0000","last_peered":"2024-05-09T03:09:36.915442+0000","last_clean":"2024-05-09T03:09:36.915442+0000","last_became_active":"2024-05-09T03:09:34.183151+0000","last_became_peered":"2024-05-09T03:09:34.183151+0000","last_unstale":"2024-05-09T03:09:36.915442+0000","last_undegraded":"2024-05-09T03:09:36.915442+0000","last_fullsized":"2024-05-09T03:09:36.915442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.196036+0000","last_change":"2024-05-09T03:09:33.177972+0000","last_active":"2024-05-09T03:09:35.196036+0000","last_peered":"2024-05-09T03:09:35.196036+0000","last_clean":"2024-05-09T03:09:35.196036+0000","last_became_active":"2024-05-09T03:09:33.177585+0000","last_became_peered":"2024-05-09T03:09:33.177585+0000","last_unstale":"2024-05-09T03:09:35.196036+0000","last_undegraded":"2024-05-09T03:09:35.196036+0000","last_fullsized":"2024-05-09T03:09:35.196036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914511+0000","last_change":"2024-05-09T03:09:33.193553+0000","last_active":"2024-05-09T03:09:36.914511+0000","last_peered":"2024-05-09T03:09:36.914511+0000","last_clean":"2024-05-09T03:09:36.914511+0000","last_became_active":"2024-05-09T03:09:33.193050+0000","last_became_peered":"2024-05-09T03:09:33.193050+0000","last_unstale":"2024-05-09T03:09:36.914511+0000","last_undegraded":"2024-05-09T03:09:36.914511+0000","last_fullsized":"2024-05-09T03:09:36.914511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:28.743738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914265+0000","last_change":"2024-05-09T03:09:34.193946+0000","last_active":"2024-05-09T03:09:36.914265+0000","last_peered":"2024-05-09T03:09:36.914265+0000","last_clean":"2024-05-09T03:09:36.914265+0000","last_became_active":"2024-05-09T03:09:34.192276+0000","last_became_peered":"2024-05-09T03:09:34.192276+0000","last_unstale":"2024-05-09T03:09:36.914265+0000","last_undegraded":"2024-05-09T03:09:36.914265+0000","last_fullsized":"2024-05-09T03:09:36.914265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223499+0000","last_change":"2024-05-09T03:09:33.183226+0000","last_active":"2024-05-09T03:09:35.223499+0000","last_peered":"2024-05-09T03:09:35.223499+0000","last_clean":"2024-05-09T03:09:35.223499+0000","last_became_active":"2024-05-09T03:09:33.182875+0000","last_became_peered":"2024-05-09T03:09:33.182875+0000","last_unstale":"2024-05-09T03:09:35.223499+0000","last_undegraded":"2024-05-09T03:09:35.223499+0000","last_fullsized":"2024-05-09T03:09:35.223499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191627+0000","last_change":"2024-05-09T03:09:34.192348+0000","last_active":"2024-05-09T03:09:35.191627+0000","last_peered":"2024-05-09T03:09:35.191627+0000","last_clean":"2024-05-09T03:09:35.191627+0000","last_became_active":"2024-05-09T03:09:34.192007+0000","last_became_peered":"2024-05-09T03:09:34.192007+0000","last_unstale":"2024-05-09T03:09:35.191627+0000","last_undegraded":"2024-05-09T03:09:35.191627+0000","last_fullsized":"2024-05-09T03:09:35.191627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195519+0000","last_change":"2024-05-09T03:09:33.189847+0000","last_active":"2024-05-09T03:09:35.195519+0000","last_peered":"2024-05-09T03:09:35.195519+0000","last_clean":"2024-05-09T03:09:35.195519+0000","last_became_active":"2024-05-09T03:09:33.189107+0000","last_became_peered":"2024-05-09T03:09:33.189107+0000","last_unstale":"2024-05-09T03:09:35.195519+0000","last_undegraded":"2024-05-09T03:09:35.195519+0000","last_fullsized":"2024-05-09T03:09:35.195519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223574+0000","last_change":"2024-05-09T03:09:34.198289+0000","last_active":"2024-05-09T03:09:35.223574+0000","last_peered":"2024-05-09T03:09:35.223574+0000","last_clean":"2024-05-09T03:09:35.223574+0000","last_became_active":"2024-05-09T03:09:34.197367+0000","last_became_peered":"2024-05-09T03:09:34.197367+0000","last_unstale":"2024-05-09T03:09:35.223574+0000","last_undegraded":"2024-05-09T03:09:35.223574+0000","last_fullsized":"2024-05-09T03:09:35.223574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223764+0000","last_change":"2024-05-09T03:09:33.188844+0000","last_active":"2024-05-09T03:09:35.223764+0000","last_peered":"2024-05-09T03:09:35.223764+0000","last_clean":"2024-05-09T03:09:35.223764+0000","last_became_active":"2024-05-09T03:09:33.188348+0000","last_became_peered":"2024-05-09T03:09:33.188348+0000","last_unstale":"2024-05-09T03:09:35.223764+0000","last_undegraded":"2024-05-09T03:09:35.223764+0000","last_fullsized":"2024-05-09T03:09:35.223764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915657+0000","last_change":"2024-05-09T03:09:34.199361+0000","last_active":"2024-05-09T03:09:36.915657+0000","last_peered":"2024-05-09T03:09:36.915657+0000","last_clean":"2024-05-09T03:09:36.915657+0000","last_became_active":"2024-05-09T03:09:34.199023+0000","last_became_peered":"2024-05-09T03:09:34.199023+0000","last_unstale":"2024-05-09T03:09:36.915657+0000","last_undegraded":"2024-05-09T03:09:36.915657+0000","last_fullsized":"2024-05-09T03:09:36.915657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191595+0000","last_change":"2024-05-09T03:09:33.193443+0000","last_active":"2024-05-09T03:09:35.191595+0000","last_peered":"2024-05-09T03:09:35.191595+0000","last_clean":"2024-05-09T03:09:35.191595+0000","last_became_active":"2024-05-09T03:09:33.193095+0000","last_became_peered":"2024-05-09T03:09:33.193095+0000","last_unstale":"2024-05-09T03:09:35.191595+0000","last_undegraded":"2024-05-09T03:09:35.191595+0000","last_fullsized":"2024-05-09T03:09:35.191595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195302+0000","last_change":"2024-05-09T03:09:34.190211+0000","last_active":"2024-05-09T03:09:35.195302+0000","last_peered":"2024-05-09T03:09:35.195302+0000","last_clean":"2024-05-09T03:09:35.195302+0000","last_became_active":"2024-05-09T03:09:34.189747+0000","last_became_peered":"2024-05-09T03:09:34.189747+0000","last_unstale":"2024-05-09T03:09:35.195302+0000","last_undegraded":"2024-05-09T03:09:35.195302+0000","last_fullsized":"2024-05-09T03:09:35.195302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195680+0000","last_change":"2024-05-09T03:09:33.186806+0000","last_active":"2024-05-09T03:09:35.195680+0000","last_peered":"2024-05-09T03:09:35.195680+0000","last_clean":"2024-05-09T03:09:35.195680+0000","last_became_active":"2024-05-09T03:09:33.185580+0000","last_became_peered":"2024-05-09T03:09:33.185580+0000","last_unstale":"2024-05-09T03:09:35.195680+0000","last_undegraded":"2024-05-09T03:09:35.195680+0000","last_fullsized":"2024-05-09T03:09:35.195680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914632+0000","last_change":"2024-05-09T03:09:34.191009+0000","last_active":"2024-05-09T03:09:36.914632+0000","last_peered":"2024-05-09T03:09:36.914632+0000","last_clean":"2024-05-09T03:09:36.914632+0000","last_became_active":"2024-05-09T03:09:34.190131+0000","last_became_peered":"2024-05-09T03:09:34.190131+0000","last_unstale":"2024-05-09T03:09:36.914632+0000","last_undegraded":"2024-05-09T03:09:36.914632+0000","last_fullsized":"2024-05-09T03:09:36.914632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.607505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915064+0000","last_change":"2024-05-09T03:09:33.185616+0000","last_active":"2024-05-09T03:09:36.915064+0000","last_peered":"2024-05-09T03:09:36.915064+0000","last_clean":"2024-05-09T03:09:36.915064+0000","last_became_active":"2024-05-09T03:09:33.185302+0000","last_became_peered":"2024-05-09T03:09:33.185302+0000","last_unstale":"2024-05-09T03:09:36.915064+0000","last_undegraded":"2024-05-09T03:09:36.915064+0000","last_fullsized":"2024-05-09T03:09:36.915064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:28.743738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914805+0000","last_change":"2024-05-09T03:09:34.190554+0000","last_active":"2024-05-09T03:09:36.914805+0000","last_peered":"2024-05-09T03:09:36.914805+0000","last_clean":"2024-05-09T03:09:36.914805+0000","last_became_active":"2024-05-09T03:09:34.190165+0000","last_became_peered":"2024-05-09T03:09:34.190165+0000","last_unstale":"2024-05-09T03:09:36.914805+0000","last_undegraded":"2024-05-09T03:09:36.914805+0000","last_fullsized":"2024-05-09T03:09:36.914805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223697+0000","last_change":"2024-05-09T03:09:33.191326+0000","last_active":"2024-05-09T03:09:35.223697+0000","last_peered":"2024-05-09T03:09:35.223697+0000","last_clean":"2024-05-09T03:09:35.223697+0000","last_became_active":"2024-05-09T03:09:33.190974+0000","last_became_peered":"2024-05-09T03:09:33.190974+0000","last_unstale":"2024-05-09T03:09:35.223697+0000","last_undegraded":"2024-05-09T03:09:35.223697+0000","last_fullsized":"2024-05-09T03:09:35.223697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914599+0000","last_change":"2024-05-09T03:09:34.195650+0000","last_active":"2024-05-09T03:09:36.914599+0000","last_peered":"2024-05-09T03:09:36.914599+0000","last_clean":"2024-05-09T03:09:36.914599+0000","last_became_active":"2024-05-09T03:09:34.195236+0000","last_became_peered":"2024-05-09T03:09:34.195236+0000","last_unstale":"2024-05-09T03:09:36.914599+0000","last_undegraded":"2024-05-09T03:09:36.914599+0000","last_fullsized":"2024-05-09T03:09:36.914599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:41.381783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195554+0000","last_change":"2024-05-09T03:09:33.184933+0000","last_active":"2024-05-09T03:09:35.195554+0000","last_peered":"2024-05-09T03:09:35.195554+0000","last_clean":"2024-05-09T03:09:35.195554+0000","last_became_active":"2024-05-09T03:09:33.184584+0000","last_became_peered":"2024-05-09T03:09:33.184584+0000","last_unstale":"2024-05-09T03:09:35.195554+0000","last_undegraded":"2024-05-09T03:09:35.195554+0000","last_fullsized":"2024-05-09T03:09:35.195554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195463+0000","last_change":"2024-05-09T03:09:34.179608+0000","last_active":"2024-05-09T03:09:35.195463+0000","last_peered":"2024-05-09T03:09:35.195463+0000","last_clean":"2024-05-09T03:09:35.195463+0000","last_became_active":"2024-05-09T03:09:34.179281+0000","last_became_peered":"2024-05-09T03:09:34.179281+0000","last_unstale":"2024-05-09T03:09:35.195463+0000","last_undegraded":"2024-05-09T03:09:35.195463+0000","last_fullsized":"2024-05-09T03:09:35.195463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913853+0000","last_change":"2024-05-09T03:09:33.183294+0000","last_active":"2024-05-09T03:09:36.913853+0000","last_peered":"2024-05-09T03:09:36.913853+0000","last_clean":"2024-05-09T03:09:36.913853+0000","last_became_active":"2024-05-09T03:09:33.182789+0000","last_became_peered":"2024-05-09T03:09:33.182789+0000","last_unstale":"2024-05-09T03:09:36.913853+0000","last_undegraded":"2024-05-09T03:09:36.913853+0000","last_fullsized":"2024-05-09T03:09:36.913853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914435+0000","last_change":"2024-05-09T03:09:34.195805+0000","last_active":"2024-05-09T03:09:36.914435+0000","last_peered":"2024-05-09T03:09:36.914435+0000","last_clean":"2024-05-09T03:09:36.914435+0000","last_became_active":"2024-05-09T03:09:34.195460+0000","last_became_peered":"2024-05-09T03:09:34.195460+0000","last_unstale":"2024-05-09T03:09:36.914435+0000","last_undegraded":"2024-05-09T03:09:36.914435+0000","last_fullsized":"2024-05-09T03:09:36.914435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195504+0000","last_change":"2024-05-09T03:09:33.182939+0000","last_active":"2024-05-09T03:09:35.195504+0000","last_peered":"2024-05-09T03:09:35.195504+0000","last_clean":"2024-05-09T03:09:35.195504+0000","last_became_active":"2024-05-09T03:09:33.182611+0000","last_became_peered":"2024-05-09T03:09:33.182611+0000","last_unstale":"2024-05-09T03:09:35.195504+0000","last_undegraded":"2024-05-09T03:09:35.195504+0000","last_fullsized":"2024-05-09T03:09:35.195504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:18.783039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914887+0000","last_change":"2024-05-09T03:09:34.190860+0000","last_active":"2024-05-09T03:09:36.914887+0000","last_peered":"2024-05-09T03:09:36.914887+0000","last_clean":"2024-05-09T03:09:36.914887+0000","last_became_active":"2024-05-09T03:09:34.190509+0000","last_became_peered":"2024-05-09T03:09:34.190509+0000","last_unstale":"2024-05-09T03:09:36.914887+0000","last_undegraded":"2024-05-09T03:09:36.914887+0000","last_fullsized":"2024-05-09T03:09:36.914887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:06.122864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914801+0000","last_change":"2024-05-09T03:09:33.187823+0000","last_active":"2024-05-09T03:09:36.914801+0000","last_peered":"2024-05-09T03:09:36.914801+0000","last_clean":"2024-05-09T03:09:36.914801+0000","last_became_active":"2024-05-09T03:09:33.187487+0000","last_became_peered":"2024-05-09T03:09:33.187487+0000","last_unstale":"2024-05-09T03:09:36.914801+0000","last_undegraded":"2024-05-09T03:09:36.914801+0000","last_fullsized":"2024-05-09T03:09:36.914801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:03.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915118+0000","last_change":"2024-05-09T03:09:34.196111+0000","last_active":"2024-05-09T03:09:36.915118+0000","last_peered":"2024-05-09T03:09:36.915118+0000","last_clean":"2024-05-09T03:09:36.915118+0000","last_became_active":"2024-05-09T03:09:34.195741+0000","last_became_peered":"2024-05-09T03:09:34.195741+0000","last_unstale":"2024-05-09T03:09:36.915118+0000","last_undegraded":"2024-05-09T03:09:36.915118+0000","last_fullsized":"2024-05-09T03:09:36.915118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:13:32.501193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191690+0000","last_change":"2024-05-09T03:09:33.189105+0000","last_active":"2024-05-09T03:09:35.191690+0000","last_peered":"2024-05-09T03:09:35.191690+0000","last_clean":"2024-05-09T03:09:35.191690+0000","last_became_active":"2024-05-09T03:09:33.188557+0000","last_became_peered":"2024-05-09T03:09:33.188557+0000","last_unstale":"2024-05-09T03:09:35.191690+0000","last_undegraded":"2024-05-09T03:09:35.191690+0000","last_fullsized":"2024-05-09T03:09:35.191690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:40.503702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223831+0000","last_change":"2024-05-09T03:09:34.205860+0000","last_active":"2024-05-09T03:09:35.223831+0000","last_peered":"2024-05-09T03:09:35.223831+0000","last_clean":"2024-05-09T03:09:35.223831+0000","last_became_active":"2024-05-09T03:09:34.205494+0000","last_became_peered":"2024-05-09T03:09:34.205494+0000","last_unstale":"2024-05-09T03:09:35.223831+0000","last_undegraded":"2024-05-09T03:09:35.223831+0000","last_fullsized":"2024-05-09T03:09:35.223831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.607505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914248+0000","last_change":"2024-05-09T03:09:33.191678+0000","last_active":"2024-05-09T03:09:36.914248+0000","last_peered":"2024-05-09T03:09:36.914248+0000","last_clean":"2024-05-09T03:09:36.914248+0000","last_became_active":"2024-05-09T03:09:33.191347+0000","last_became_peered":"2024-05-09T03:09:33.191347+0000","last_unstale":"2024-05-09T03:09:36.914248+0000","last_undegraded":"2024-05-09T03:09:36.914248+0000","last_fullsized":"2024-05-09T03:09:36.914248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914843+0000","last_change":"2024-05-09T03:09:34.200500+0000","last_active":"2024-05-09T03:09:36.914843+0000","last_peered":"2024-05-09T03:09:36.914843+0000","last_clean":"2024-05-09T03:09:36.914843+0000","last_became_active":"2024-05-09T03:09:34.200204+0000","last_became_peered":"2024-05-09T03:09:34.200204+0000","last_unstale":"2024-05-09T03:09:36.914843+0000","last_undegraded":"2024-05-09T03:09:36.914843+0000","last_fullsized":"2024-05-09T03:09:36.914843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:59:37.733696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914436+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:09:36.914436+0000","last_peered":"2024-05-09T03:09:36.914436+0000","last_clean":"2024-05-09T03:09:36.914436+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:09:36.914436+0000","last_undegraded":"2024-05-09T03:09:36.914436+0000","last_fullsized":"2024-05-09T03:09:36.914436+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223311+0000","last_change":"2024-05-09T03:09:33.189141+0000","last_active":"2024-05-09T03:09:35.223311+0000","last_peered":"2024-05-09T03:09:35.223311+0000","last_clean":"2024-05-09T03:09:35.223311+0000","last_became_active":"2024-05-09T03:09:33.188576+0000","last_became_peered":"2024-05-09T03:09:33.188576+0000","last_unstale":"2024-05-09T03:09:35.223311+0000","last_undegraded":"2024-05-09T03:09:35.223311+0000","last_fullsized":"2024-05-09T03:09:35.223311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914857+0000","last_change":"2024-05-09T03:09:34.202021+0000","last_active":"2024-05-09T03:09:36.914857+0000","last_peered":"2024-05-09T03:09:36.914857+0000","last_clean":"2024-05-09T03:09:36.914857+0000","last_became_active":"2024-05-09T03:09:34.201661+0000","last_became_peered":"2024-05-09T03:09:34.201661+0000","last_unstale":"2024-05-09T03:09:36.914857+0000","last_undegraded":"2024-05-09T03:09:36.914857+0000","last_fullsized":"2024-05-09T03:09:36.914857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914241+0000","last_change":"2024-05-09T03:09:33.189640+0000","last_active":"2024-05-09T03:09:36.914241+0000","last_peered":"2024-05-09T03:09:36.914241+0000","last_clean":"2024-05-09T03:09:36.914241+0000","last_became_active":"2024-05-09T03:09:33.189291+0000","last_became_peered":"2024-05-09T03:09:33.189291+0000","last_unstale":"2024-05-09T03:09:36.914241+0000","last_undegraded":"2024-05-09T03:09:36.914241+0000","last_fullsized":"2024-05-09T03:09:36.914241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:56.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915084+0000","last_change":"2024-05-09T03:09:34.186522+0000","last_active":"2024-05-09T03:09:36.915084+0000","last_peered":"2024-05-09T03:09:36.915084+0000","last_clean":"2024-05-09T03:09:36.915084+0000","last_became_active":"2024-05-09T03:09:34.186183+0000","last_became_peered":"2024-05-09T03:09:34.186183+0000","last_unstale":"2024-05-09T03:09:36.915084+0000","last_undegraded":"2024-05-09T03:09:36.915084+0000","last_fullsized":"2024-05-09T03:09:36.915084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913979+0000","last_change":"2024-05-09T03:09:33.186397+0000","last_active":"2024-05-09T03:09:36.913979+0000","last_peered":"2024-05-09T03:09:36.913979+0000","last_clean":"2024-05-09T03:09:36.913979+0000","last_became_active":"2024-05-09T03:09:33.186026+0000","last_became_peered":"2024-05-09T03:09:33.186026+0000","last_unstale":"2024-05-09T03:09:36.913979+0000","last_undegraded":"2024-05-09T03:09:36.913979+0000","last_fullsized":"2024-05-09T03:09:36.913979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191563+0000","last_change":"2024-05-09T03:09:34.206024+0000","last_active":"2024-05-09T03:09:35.191563+0000","last_peered":"2024-05-09T03:09:35.191563+0000","last_clean":"2024-05-09T03:09:35.191563+0000","last_became_active":"2024-05-09T03:09:34.205694+0000","last_became_peered":"2024-05-09T03:09:34.205694+0000","last_unstale":"2024-05-09T03:09:35.191563+0000","last_undegraded":"2024-05-09T03:09:35.191563+0000","last_fullsized":"2024-05-09T03:09:35.191563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223876+0000","last_change":"2024-05-09T03:09:33.189135+0000","last_active":"2024-05-09T03:09:35.223876+0000","last_peered":"2024-05-09T03:09:35.223876+0000","last_clean":"2024-05-09T03:09:35.223876+0000","last_became_active":"2024-05-09T03:09:33.188869+0000","last_became_peered":"2024-05-09T03:09:33.188869+0000","last_unstale":"2024-05-09T03:09:35.223876+0000","last_undegraded":"2024-05-09T03:09:35.223876+0000","last_fullsized":"2024-05-09T03:09:35.223876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195454+0000","last_change":"2024-05-09T03:09:34.189609+0000","last_active":"2024-05-09T03:09:35.195454+0000","last_peered":"2024-05-09T03:09:35.195454+0000","last_clean":"2024-05-09T03:09:35.195454+0000","last_became_active":"2024-05-09T03:09:34.189088+0000","last_became_peered":"2024-05-09T03:09:34.189088+0000","last_unstale":"2024-05-09T03:09:35.195454+0000","last_undegraded":"2024-05-09T03:09:35.195454+0000","last_fullsized":"2024-05-09T03:09:35.195454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:41.508915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191407+0000","last_change":"2024-05-09T03:09:33.184995+0000","last_active":"2024-05-09T03:09:35.191407+0000","last_peered":"2024-05-09T03:09:35.191407+0000","last_clean":"2024-05-09T03:09:35.191407+0000","last_became_active":"2024-05-09T03:09:33.184596+0000","last_became_peered":"2024-05-09T03:09:33.184596+0000","last_unstale":"2024-05-09T03:09:35.191407+0000","last_undegraded":"2024-05-09T03:09:35.191407+0000","last_fullsized":"2024-05-09T03:09:35.191407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223976+0000","last_change":"2024-05-09T03:09:34.205632+0000","last_active":"2024-05-09T03:09:35.223976+0000","last_peered":"2024-05-09T03:09:35.223976+0000","last_clean":"2024-05-09T03:09:35.223976+0000","last_became_active":"2024-05-09T03:09:34.205275+0000","last_became_peered":"2024-05-09T03:09:34.205275+0000","last_unstale":"2024-05-09T03:09:35.223976+0000","last_undegraded":"2024-05-09T03:09:35.223976+0000","last_fullsized":"2024-05-09T03:09:35.223976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.707927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915021+0000","last_change":"2024-05-09T03:09:33.193320+0000","last_active":"2024-05-09T03:09:36.915021+0000","last_peered":"2024-05-09T03:09:36.915021+0000","last_clean":"2024-05-09T03:09:36.915021+0000","last_became_active":"2024-05-09T03:09:33.193005+0000","last_became_peered":"2024-05-09T03:09:33.193005+0000","last_unstale":"2024-05-09T03:09:36.915021+0000","last_undegraded":"2024-05-09T03:09:36.915021+0000","last_fullsized":"2024-05-09T03:09:36.915021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:40.503702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191290+0000","last_change":"2024-05-09T03:09:34.204993+0000","last_active":"2024-05-09T03:09:35.191290+0000","last_peered":"2024-05-09T03:09:35.191290+0000","last_clean":"2024-05-09T03:09:35.191290+0000","last_became_active":"2024-05-09T03:09:34.204647+0000","last_became_peered":"2024-05-09T03:09:34.204647+0000","last_unstale":"2024-05-09T03:09:35.191290+0000","last_undegraded":"2024-05-09T03:09:35.191290+0000","last_fullsized":"2024-05-09T03:09:35.191290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191516+0000","last_change":"2024-05-09T03:09:33.188474+0000","last_active":"2024-05-09T03:09:35.191516+0000","last_peered":"2024-05-09T03:09:35.191516+0000","last_clean":"2024-05-09T03:09:35.191516+0000","last_became_active":"2024-05-09T03:09:33.187643+0000","last_became_peered":"2024-05-09T03:09:33.187643+0000","last_unstale":"2024-05-09T03:09:35.191516+0000","last_undegraded":"2024-05-09T03:09:35.191516+0000","last_fullsized":"2024-05-09T03:09:35.191516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:04.365646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915505+0000","last_change":"2024-05-09T03:09:34.190064+0000","last_active":"2024-05-09T03:09:36.915505+0000","last_peered":"2024-05-09T03:09:36.915505+0000","last_clean":"2024-05-09T03:09:36.915505+0000","last_became_active":"2024-05-09T03:09:34.189695+0000","last_became_peered":"2024-05-09T03:09:34.189695+0000","last_unstale":"2024-05-09T03:09:36.915505+0000","last_undegraded":"2024-05-09T03:09:36.915505+0000","last_fullsized":"2024-05-09T03:09:36.915505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195648+0000","last_change":"2024-05-09T03:09:33.188283+0000","last_active":"2024-05-09T03:09:35.195648+0000","last_peered":"2024-05-09T03:09:35.195648+0000","last_clean":"2024-05-09T03:09:35.195648+0000","last_became_active":"2024-05-09T03:09:33.187986+0000","last_became_peered":"2024-05-09T03:09:33.187986+0000","last_unstale":"2024-05-09T03:09:35.195648+0000","last_undegraded":"2024-05-09T03:09:35.195648+0000","last_fullsized":"2024-05-09T03:09:35.195648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:59.142240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915245+0000","last_change":"2024-05-09T03:09:34.188118+0000","last_active":"2024-05-09T03:09:36.915245+0000","last_peered":"2024-05-09T03:09:36.915245+0000","last_clean":"2024-05-09T03:09:36.915245+0000","last_became_active":"2024-05-09T03:09:34.187756+0000","last_became_peered":"2024-05-09T03:09:34.187756+0000","last_unstale":"2024-05-09T03:09:36.915245+0000","last_undegraded":"2024-05-09T03:09:36.915245+0000","last_fullsized":"2024-05-09T03:09:36.915245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914221+0000","last_change":"2024-05-09T03:09:33.191538+0000","last_active":"2024-05-09T03:09:36.914221+0000","last_peered":"2024-05-09T03:09:36.914221+0000","last_clean":"2024-05-09T03:09:36.914221+0000","last_became_active":"2024-05-09T03:09:33.190940+0000","last_became_peered":"2024-05-09T03:09:33.190940+0000","last_unstale":"2024-05-09T03:09:36.914221+0000","last_undegraded":"2024-05-09T03:09:36.914221+0000","last_fullsized":"2024-05-09T03:09:36.914221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195374+0000","last_change":"2024-05-09T03:09:34.203831+0000","last_active":"2024-05-09T03:09:35.195374+0000","last_peered":"2024-05-09T03:09:35.195374+0000","last_clean":"2024-05-09T03:09:35.195374+0000","last_became_active":"2024-05-09T03:09:34.203509+0000","last_became_peered":"2024-05-09T03:09:34.203509+0000","last_unstale":"2024-05-09T03:09:35.195374+0000","last_undegraded":"2024-05-09T03:09:35.195374+0000","last_fullsized":"2024-05-09T03:09:35.195374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191428+0000","last_change":"2024-05-09T03:09:33.188345+0000","last_active":"2024-05-09T03:09:35.191428+0000","last_peered":"2024-05-09T03:09:35.191428+0000","last_clean":"2024-05-09T03:09:35.191428+0000","last_became_active":"2024-05-09T03:09:33.187996+0000","last_became_peered":"2024-05-09T03:09:33.187996+0000","last_unstale":"2024-05-09T03:09:35.191428+0000","last_undegraded":"2024-05-09T03:09:35.191428+0000","last_fullsized":"2024-05-09T03:09:35.191428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914440+0000","last_change":"2024-05-09T03:09:34.188063+0000","last_active":"2024-05-09T03:09:36.914440+0000","last_peered":"2024-05-09T03:09:36.914440+0000","last_clean":"2024-05-09T03:09:36.914440+0000","last_became_active":"2024-05-09T03:09:34.187703+0000","last_became_peered":"2024-05-09T03:09:34.187703+0000","last_unstale":"2024-05-09T03:09:36.914440+0000","last_undegraded":"2024-05-09T03:09:36.914440+0000","last_fullsized":"2024-05-09T03:09:36.914440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191401+0000","last_change":"2024-05-09T03:09:33.191733+0000","last_active":"2024-05-09T03:09:35.191401+0000","last_peered":"2024-05-09T03:09:35.191401+0000","last_clean":"2024-05-09T03:09:35.191401+0000","last_became_active":"2024-05-09T03:09:33.190856+0000","last_became_peered":"2024-05-09T03:09:33.190856+0000","last_unstale":"2024-05-09T03:09:35.191401+0000","last_undegraded":"2024-05-09T03:09:35.191401+0000","last_fullsized":"2024-05-09T03:09:35.191401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915388+0000","last_change":"2024-05-09T03:09:34.201161+0000","last_active":"2024-05-09T03:09:36.915388+0000","last_peered":"2024-05-09T03:09:36.915388+0000","last_clean":"2024-05-09T03:09:36.915388+0000","last_became_active":"2024-05-09T03:09:34.200789+0000","last_became_peered":"2024-05-09T03:09:34.200789+0000","last_unstale":"2024-05-09T03:09:36.915388+0000","last_undegraded":"2024-05-09T03:09:36.915388+0000","last_fullsized":"2024-05-09T03:09:36.915388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914778+0000","last_change":"2024-05-09T03:09:33.191322+0000","last_active":"2024-05-09T03:09:36.914778+0000","last_peered":"2024-05-09T03:09:36.914778+0000","last_clean":"2024-05-09T03:09:36.914778+0000","last_became_active":"2024-05-09T03:09:33.190925+0000","last_became_peered":"2024-05-09T03:09:33.190925+0000","last_unstale":"2024-05-09T03:09:36.914778+0000","last_undegraded":"2024-05-09T03:09:36.914778+0000","last_fullsized":"2024-05-09T03:09:36.914778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:20.918752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195950+0000","last_change":"2024-05-09T03:09:34.181003+0000","last_active":"2024-05-09T03:09:35.195950+0000","last_peered":"2024-05-09T03:09:35.195950+0000","last_clean":"2024-05-09T03:09:35.195950+0000","last_became_active":"2024-05-09T03:09:34.180247+0000","last_became_peered":"2024-05-09T03:09:34.180247+0000","last_unstale":"2024-05-09T03:09:35.195950+0000","last_undegraded":"2024-05-09T03:09:35.195950+0000","last_fullsized":"2024-05-09T03:09:35.195950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223857+0000","last_change":"2024-05-09T03:09:34.207920+0000","last_active":"2024-05-09T03:09:35.223857+0000","last_peered":"2024-05-09T03:09:35.223857+0000","last_clean":"2024-05-09T03:09:35.223857+0000","last_became_active":"2024-05-09T03:09:34.207645+0000","last_became_peered":"2024-05-09T03:09:34.207645+0000","last_unstale":"2024-05-09T03:09:35.223857+0000","last_undegraded":"2024-05-09T03:09:35.223857+0000","last_fullsized":"2024-05-09T03:09:35.223857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:06.122864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914557+0000","last_change":"2024-05-09T03:09:33.194189+0000","last_active":"2024-05-09T03:09:36.914557+0000","last_peered":"2024-05-09T03:09:36.914557+0000","last_clean":"2024-05-09T03:09:36.914557+0000","last_became_active":"2024-05-09T03:09:33.193816+0000","last_became_peered":"2024-05-09T03:09:33.193816+0000","last_unstale":"2024-05-09T03:09:36.914557+0000","last_undegraded":"2024-05-09T03:09:36.914557+0000","last_fullsized":"2024-05-09T03:09:36.914557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223608+0000","last_change":"2024-05-09T03:09:34.207981+0000","last_active":"2024-05-09T03:09:35.223608+0000","last_peered":"2024-05-09T03:09:35.223608+0000","last_clean":"2024-05-09T03:09:35.223608+0000","last_became_active":"2024-05-09T03:09:34.207609+0000","last_became_peered":"2024-05-09T03:09:34.207609+0000","last_unstale":"2024-05-09T03:09:35.223608+0000","last_undegraded":"2024-05-09T03:09:35.223608+0000","last_fullsized":"2024-05-09T03:09:35.223608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:41.381783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914861+0000","last_change":"2024-05-09T03:09:33.194110+0000","last_active":"2024-05-09T03:09:36.914861+0000","last_peered":"2024-05-09T03:09:36.914861+0000","last_clean":"2024-05-09T03:09:36.914861+0000","last_became_active":"2024-05-09T03:09:33.193682+0000","last_became_peered":"2024-05-09T03:09:33.193682+0000","last_unstale":"2024-05-09T03:09:36.914861+0000","last_undegraded":"2024-05-09T03:09:36.914861+0000","last_fullsized":"2024-05-09T03:09:36.914861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:04.365646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915459+0000","last_change":"2024-05-09T03:09:34.201024+0000","last_active":"2024-05-09T03:09:36.915459+0000","last_peered":"2024-05-09T03:09:36.915459+0000","last_clean":"2024-05-09T03:09:36.915459+0000","last_became_active":"2024-05-09T03:09:34.200708+0000","last_became_peered":"2024-05-09T03:09:34.200708+0000","last_unstale":"2024-05-09T03:09:36.915459+0000","last_undegraded":"2024-05-09T03:09:36.915459+0000","last_fullsized":"2024-05-09T03:09:36.915459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915071+0000","last_change":"2024-05-09T03:09:33.190754+0000","last_active":"2024-05-09T03:09:36.915071+0000","last_peered":"2024-05-09T03:09:36.915071+0000","last_clean":"2024-05-09T03:09:36.915071+0000","last_became_active":"2024-05-09T03:09:33.189371+0000","last_became_peered":"2024-05-09T03:09:33.189371+0000","last_unstale":"2024-05-09T03:09:36.915071+0000","last_undegraded":"2024-05-09T03:09:36.915071+0000","last_fullsized":"2024-05-09T03:09:36.915071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:03.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914867+0000","last_change":"2024-05-09T03:09:34.193444+0000","last_active":"2024-05-09T03:09:36.914867+0000","last_peered":"2024-05-09T03:09:36.914867+0000","last_clean":"2024-05-09T03:09:36.914867+0000","last_became_active":"2024-05-09T03:09:34.193113+0000","last_became_peered":"2024-05-09T03:09:34.193113+0000","last_unstale":"2024-05-09T03:09:36.914867+0000","last_undegraded":"2024-05-09T03:09:36.914867+0000","last_fullsized":"2024-05-09T03:09:36.914867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915644+0000","last_change":"2024-05-09T03:09:33.191591+0000","last_active":"2024-05-09T03:09:36.915644+0000","last_peered":"2024-05-09T03:09:36.915644+0000","last_clean":"2024-05-09T03:09:36.915644+0000","last_became_active":"2024-05-09T03:09:33.191275+0000","last_became_peered":"2024-05-09T03:09:33.191275+0000","last_unstale":"2024-05-09T03:09:36.915644+0000","last_undegraded":"2024-05-09T03:09:36.915644+0000","last_fullsized":"2024-05-09T03:09:36.915644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:18.783039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:09:43.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:09:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.195+0000 7fe379870640 1 Processor -- start 2024-05-09T03:09:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.195+0000 7fe379870640 1 -- start start 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe379870640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe374164c70 0x7fe3741615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe379870640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe37415dfe0 0x7fe3740d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe379870640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe374160b20 con 0x7fe3740d2480 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe379870640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe374161150 con 0x7fe374164c70 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe379870640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe3740d15f0 con 0x7fe37415dfe0 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe372ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe374164c70 0x7fe3741615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe372ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe374164c70 0x7fe3741615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56156/0 (socket says 172.21.15.46:56156) 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.196+0000 7fe372ffd640 1 -- 172.21.15.46:0/793946220 learned_addr learned my addr 172.21.15.46:0/793946220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.197+0000 7fe3727fc640 1 --2- 172.21.15.46:0/793946220 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe37415dfe0 0x7fe3740d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.197+0000 7fe372ffd640 1 -- 172.21.15.46:0/793946220 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe37415dfe0 msgr2=0x7fe3740d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.197+0000 7fe372ffd640 1 --2- 172.21.15.46:0/793946220 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe37415dfe0 0x7fe3740d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.197+0000 7fe372ffd640 1 -- 172.21.15.46:0/793946220 >> v1:172.21.15.88:6789/0 conn(0x7fe3740d2480 legacy=0x7fe3740d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.197+0000 7fe372ffd640 1 -- 172.21.15.46:0/793946220 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3740d18d0 con 0x7fe374164c70 2024-05-09T03:09:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.197+0000 7fe372ffd640 1 --2- 172.21.15.46:0/793946220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe374164c70 0x7fe3741615e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe364009870 tx=0x7fe36402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e7d08172b53731 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe371ffb640 1 -- 172.21.15.46:0/793946220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364037070 con 0x7fe374164c70 2024-05-09T03:09:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe371ffb640 1 -- 172.21.15.46:0/793946220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe364032440 con 0x7fe374164c70 2024-05-09T03:09:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe371ffb640 1 -- 172.21.15.46:0/793946220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364032810 con 0x7fe374164c70 2024-05-09T03:09:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 -- 172.21.15.46:0/793946220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe374164c70 msgr2=0x7fe3741615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 --2- 172.21.15.46:0/793946220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe374164c70 0x7fe3741615e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe364009870 tx=0x7fe36402a130 comp rx=0 tx=0).stop 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 -- 172.21.15.46:0/793946220 shutdown_connections 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 --2- 172.21.15.46:0/793946220 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe37415dfe0 0x7fe3740d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 --2- 172.21.15.46:0/793946220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe374164c70 0x7fe3741615e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 -- 172.21.15.46:0/793946220 >> 172.21.15.46:0/793946220 conn(0x7fe374073db0 msgr2=0x7fe3740741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 -- 172.21.15.46:0/793946220 shutdown_connections 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.198+0000 7fe379870640 1 -- 172.21.15.46:0/793946220 wait complete. 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.200+0000 7fe379870640 1 Processor -- start 2024-05-09T03:09:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.200+0000 7fe379870640 1 -- start start 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.201+0000 7fe379870640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 0x7fe3740ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.201+0000 7fe379870640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37415dfe0 0x7fe374152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe379870640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe374164c70 0x7fe3740cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe379870640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe37415e3f0 con 0x7fe3740d2480 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe379870640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe3740d1300 con 0x7fe37415dfe0 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe379870640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe3740d45c0 con 0x7fe374164c70 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 0x7fe3740ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 0x7fe3740ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56160/0 (socket says 172.21.15.46:56160) 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 -- 172.21.15.46:0/2221676857 learned_addr learned my addr 172.21.15.46:0/2221676857 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe3727fc640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37415dfe0 0x7fe374152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe3737fe640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe374164c70 0x7fe3740cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 -- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe374164c70 msgr2=0x7fe3740cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe374164c70 0x7fe3740cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 -- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37415dfe0 msgr2=0x7fe374152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37415dfe0 0x7fe374152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 -- 172.21.15.46:0/2221676857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe374153110 con 0x7fe3740d2480 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe3737fe640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe374164c70 0x7fe3740cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe372ffd640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 0x7fe3740ce830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe364009870 tx=0x7fe3640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364041020 con 0x7fe3740d2480 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3640306c0 con 0x7fe3740d2480 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3741533b0 con 0x7fe3740d2480 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe364037040 con 0x7fe3740d2480 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.202+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3741f7660 con 0x7fe3740d2480 2024-05-09T03:09:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.203+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37408cfd0 con 0x7fe3740d2480 2024-05-09T03:09:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.204+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe364044020 con 0x7fe3740d2480 2024-05-09T03:09:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.205+0000 7fe35bfff640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3600762d0 0x7fe3600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.205+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe3640b7e40 con 0x7fe3740d2480 2024-05-09T03:09:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.205+0000 7fe3727fc640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3600762d0 0x7fe3600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.205+0000 7fe3727fc640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3600762d0 0x7fe3600787e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe37415d360 tx=0x7fe368008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.209+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe37408cfd0 con 0x7fe3740d2480 2024-05-09T03:09:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.396+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe3741f7900 con 0x7fe3600762d0 2024-05-09T03:09:43.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.403+0000 7fe35bfff640 1 -- 172.21.15.46:0/2221676857 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7fe3741f7900 con 0x7fe3600762d0 2024-05-09T03:09:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:43.407 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:09:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3600762d0 msgr2=0x7fe3600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3600762d0 0x7fe3600787e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe37415d360 tx=0x7fe368008040 comp rx=0 tx=0).stop 2024-05-09T03:09:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 msgr2=0x7fe3740ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 0x7fe3740ce830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe364009870 tx=0x7fe3640046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 shutdown_connections 2024-05-09T03:09:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe374164c70 0x7fe3740cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe37415dfe0 0x7fe374152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3600762d0 0x7fe3600787e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 --2- 172.21.15.46:0/2221676857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3740d2480 0x7fe3740ce830 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 >> 172.21.15.46:0/2221676857 conn(0x7fe374073db0 msgr2=0x7fe37415e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 shutdown_connections 2024-05-09T03:09:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.407+0000 7fe379870640 1 -- 172.21.15.46:0/2221676857 wait complete. 2024-05-09T03:09:43.419 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-05-09T03:09:42.492701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235624,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739096,"statfs":{"total":773094113280,"available":772852834304,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.320665"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914921+0000","last_change":"2024-05-09T03:09:34.191121+0000","last_active":"2024-05-09T03:09:36.914921+0000","last_peered":"2024-05-09T03:09:36.914921+0000","last_clean":"2024-05-09T03:09:36.914921+0000","last_became_active":"2024-05-09T03:09:34.189608+0000","last_became_peered":"2024-05-09T03:09:34.189608+0000","last_unstale":"2024-05-09T03:09:36.914921+0000","last_undegraded":"2024-05-09T03:09:36.914921+0000","last_fullsized":"2024-05-09T03:09:36.914921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:49.231849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191404+0000","last_change":"2024-05-09T03:09:34.200351+0000","last_active":"2024-05-09T03:09:35.191404+0000","last_peered":"2024-05-09T03:09:35.191404+0000","last_clean":"2024-05-09T03:09:35.191404+0000","last_became_active":"2024-05-09T03:09:34.200009+0000","last_became_peered":"2024-05-09T03:09:34.200009+0000","last_unstale":"2024-05-09T03:09:35.191404+0000","last_undegraded":"2024-05-09T03:09:35.191404+0000","last_fullsized":"2024-05-09T03:09:35.191404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914510+0000","last_change":"2024-05-09T03:09:34.204156+0000","last_active":"2024-05-09T03:09:36.914510+0000","last_peered":"2024-05-09T03:09:36.914510+0000","last_clean":"2024-05-09T03:09:36.914510+0000","last_became_active":"2024-05-09T03:09:34.203843+0000","last_became_peered":"2024-05-09T03:09:34.203843+0000","last_unstale":"2024-05-09T03:09:36.914510+0000","last_undegraded":"2024-05-09T03:09:36.914510+0000","last_fullsized":"2024-05-09T03:09:36.914510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:34:16.118667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223412+0000","last_change":"2024-05-09T03:09:34.206782+0000","last_active":"2024-05-09T03:09:35.223412+0000","last_peered":"2024-05-09T03:09:35.223412+0000","last_clean":"2024-05-09T03:09:35.223412+0000","last_became_active":"2024-05-09T03:09:34.206376+0000","last_became_peered":"2024-05-09T03:09:34.206376+0000","last_unstale":"2024-05-09T03:09:35.223412+0000","last_undegraded":"2024-05-09T03:09:35.223412+0000","last_fullsized":"2024-05-09T03:09:35.223412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:33.100263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914552+0000","last_change":"2024-05-09T03:09:34.203941+0000","last_active":"2024-05-09T03:09:36.914552+0000","last_peered":"2024-05-09T03:09:36.914552+0000","last_clean":"2024-05-09T03:09:36.914552+0000","last_became_active":"2024-05-09T03:09:34.203530+0000","last_became_peered":"2024-05-09T03:09:34.203530+0000","last_unstale":"2024-05-09T03:09:36.914552+0000","last_undegraded":"2024-05-09T03:09:36.914552+0000","last_fullsized":"2024-05-09T03:09:36.914552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223482+0000","last_change":"2024-05-09T03:09:34.201477+0000","last_active":"2024-05-09T03:09:35.223482+0000","last_peered":"2024-05-09T03:09:35.223482+0000","last_clean":"2024-05-09T03:09:35.223482+0000","last_became_active":"2024-05-09T03:09:34.200622+0000","last_became_peered":"2024-05-09T03:09:34.200622+0000","last_unstale":"2024-05-09T03:09:35.223482+0000","last_undegraded":"2024-05-09T03:09:35.223482+0000","last_fullsized":"2024-05-09T03:09:35.223482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914369+0000","last_change":"2024-05-09T03:09:34.190313+0000","last_active":"2024-05-09T03:09:36.914369+0000","last_peered":"2024-05-09T03:09:36.914369+0000","last_clean":"2024-05-09T03:09:36.914369+0000","last_became_active":"2024-05-09T03:09:34.189973+0000","last_became_peered":"2024-05-09T03:09:34.189973+0000","last_unstale":"2024-05-09T03:09:36.914369+0000","last_undegraded":"2024-05-09T03:09:36.914369+0000","last_fullsized":"2024-05-09T03:09:36.914369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913936+0000","last_change":"2024-05-09T03:09:34.206587+0000","last_active":"2024-05-09T03:09:36.913936+0000","last_peered":"2024-05-09T03:09:36.913936+0000","last_clean":"2024-05-09T03:09:36.913936+0000","last_became_active":"2024-05-09T03:09:34.206317+0000","last_became_peered":"2024-05-09T03:09:34.206317+0000","last_unstale":"2024-05-09T03:09:36.913936+0000","last_undegraded":"2024-05-09T03:09:36.913936+0000","last_fullsized":"2024-05-09T03:09:36.913936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914964+0000","last_change":"2024-05-09T03:09:34.202532+0000","last_active":"2024-05-09T03:09:36.914964+0000","last_peered":"2024-05-09T03:09:36.914964+0000","last_clean":"2024-05-09T03:09:36.914964+0000","last_became_active":"2024-05-09T03:09:34.202253+0000","last_became_peered":"2024-05-09T03:09:34.202253+0000","last_unstale":"2024-05-09T03:09:36.914964+0000","last_undegraded":"2024-05-09T03:09:36.914964+0000","last_fullsized":"2024-05-09T03:09:36.914964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.707927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195921+0000","last_change":"2024-05-09T03:09:34.199946+0000","last_active":"2024-05-09T03:09:35.195921+0000","last_peered":"2024-05-09T03:09:35.195921+0000","last_clean":"2024-05-09T03:09:35.195921+0000","last_became_active":"2024-05-09T03:09:34.199583+0000","last_became_peered":"2024-05-09T03:09:34.199583+0000","last_unstale":"2024-05-09T03:09:35.195921+0000","last_undegraded":"2024-05-09T03:09:35.195921+0000","last_fullsized":"2024-05-09T03:09:35.195921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194810+0000","last_change":"2024-05-09T03:09:34.199326+0000","last_active":"2024-05-09T03:09:35.194810+0000","last_peered":"2024-05-09T03:09:35.194810+0000","last_clean":"2024-05-09T03:09:35.194810+0000","last_became_active":"2024-05-09T03:09:34.198952+0000","last_became_peered":"2024-05-09T03:09:34.198952+0000","last_unstale":"2024-05-09T03:09:35.194810+0000","last_undegraded":"2024-05-09T03:09:35.194810+0000","last_fullsized":"2024-05-09T03:09:35.194810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195230+0000","last_change":"2024-05-09T03:09:34.186397+0000","last_active":"2024-05-09T03:09:35.195230+0000","last_peered":"2024-05-09T03:09:35.195230+0000","last_clean":"2024-05-09T03:09:35.195230+0000","last_became_active":"2024-05-09T03:09:34.186102+0000","last_became_peered":"2024-05-09T03:09:34.186102+0000","last_unstale":"2024-05-09T03:09:35.195230+0000","last_undegraded":"2024-05-09T03:09:35.195230+0000","last_fullsized":"2024-05-09T03:09:35.195230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:17.126340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915391+0000","last_change":"2024-05-09T03:09:34.198335+0000","last_active":"2024-05-09T03:09:36.915391+0000","last_peered":"2024-05-09T03:09:36.915391+0000","last_clean":"2024-05-09T03:09:36.915391+0000","last_became_active":"2024-05-09T03:09:34.197960+0000","last_became_peered":"2024-05-09T03:09:34.197960+0000","last_unstale":"2024-05-09T03:09:36.915391+0000","last_undegraded":"2024-05-09T03:09:36.915391+0000","last_fullsized":"2024-05-09T03:09:36.915391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914604+0000","last_change":"2024-05-09T03:09:34.189212+0000","last_active":"2024-05-09T03:09:36.914604+0000","last_peered":"2024-05-09T03:09:36.914604+0000","last_clean":"2024-05-09T03:09:36.914604+0000","last_became_active":"2024-05-09T03:09:34.188580+0000","last_became_peered":"2024-05-09T03:09:34.188580+0000","last_unstale":"2024-05-09T03:09:36.914604+0000","last_undegraded":"2024-05-09T03:09:36.914604+0000","last_fullsized":"2024-05-09T03:09:36.914604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:20.896373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191292+0000","last_change":"2024-05-09T03:09:34.186444+0000","last_active":"2024-05-09T03:09:35.191292+0000","last_peered":"2024-05-09T03:09:35.191292+0000","last_clean":"2024-05-09T03:09:35.191292+0000","last_became_active":"2024-05-09T03:09:34.186119+0000","last_became_peered":"2024-05-09T03:09:34.186119+0000","last_unstale":"2024-05-09T03:09:35.191292+0000","last_undegraded":"2024-05-09T03:09:35.191292+0000","last_fullsized":"2024-05-09T03:09:35.191292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223239+0000","last_change":"2024-05-09T03:09:34.201163+0000","last_active":"2024-05-09T03:09:35.223239+0000","last_peered":"2024-05-09T03:09:35.223239+0000","last_clean":"2024-05-09T03:09:35.223239+0000","last_became_active":"2024-05-09T03:09:34.200814+0000","last_became_peered":"2024-05-09T03:09:34.200814+0000","last_unstale":"2024-05-09T03:09:35.223239+0000","last_undegraded":"2024-05-09T03:09:35.223239+0000","last_fullsized":"2024-05-09T03:09:35.223239+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223892+0000","last_change":"2024-05-09T03:09:34.201044+0000","last_active":"2024-05-09T03:09:35.223892+0000","last_peered":"2024-05-09T03:09:35.223892+0000","last_clean":"2024-05-09T03:09:35.223892+0000","last_became_active":"2024-05-09T03:09:34.200392+0000","last_became_peered":"2024-05-09T03:09:34.200392+0000","last_unstale":"2024-05-09T03:09:35.223892+0000","last_undegraded":"2024-05-09T03:09:35.223892+0000","last_fullsized":"2024-05-09T03:09:35.223892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:49.231849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914697+0000","last_change":"2024-05-09T03:09:34.203929+0000","last_active":"2024-05-09T03:09:36.914697+0000","last_peered":"2024-05-09T03:09:36.914697+0000","last_clean":"2024-05-09T03:09:36.914697+0000","last_became_active":"2024-05-09T03:09:34.203562+0000","last_became_peered":"2024-05-09T03:09:34.203562+0000","last_unstale":"2024-05-09T03:09:36.914697+0000","last_undegraded":"2024-05-09T03:09:36.914697+0000","last_fullsized":"2024-05-09T03:09:36.914697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915098+0000","last_change":"2024-05-09T03:09:34.204594+0000","last_active":"2024-05-09T03:09:36.915098+0000","last_peered":"2024-05-09T03:09:36.915098+0000","last_clean":"2024-05-09T03:09:36.915098+0000","last_became_active":"2024-05-09T03:09:34.204289+0000","last_became_peered":"2024-05-09T03:09:34.204289+0000","last_unstale":"2024-05-09T03:09:36.915098+0000","last_undegraded":"2024-05-09T03:09:36.915098+0000","last_fullsized":"2024-05-09T03:09:36.915098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:09.418002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195935+0000","last_change":"2024-05-09T03:09:34.186570+0000","last_active":"2024-05-09T03:09:35.195935+0000","last_peered":"2024-05-09T03:09:35.195935+0000","last_clean":"2024-05-09T03:09:35.195935+0000","last_became_active":"2024-05-09T03:09:34.186249+0000","last_became_peered":"2024-05-09T03:09:34.186249+0000","last_unstale":"2024-05-09T03:09:35.195935+0000","last_undegraded":"2024-05-09T03:09:35.195935+0000","last_fullsized":"2024-05-09T03:09:35.195935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914719+0000","last_change":"2024-05-09T03:09:34.195784+0000","last_active":"2024-05-09T03:09:36.914719+0000","last_peered":"2024-05-09T03:09:36.914719+0000","last_clean":"2024-05-09T03:09:36.914719+0000","last_became_active":"2024-05-09T03:09:34.195111+0000","last_became_peered":"2024-05-09T03:09:34.195111+0000","last_unstale":"2024-05-09T03:09:36.914719+0000","last_undegraded":"2024-05-09T03:09:36.914719+0000","last_fullsized":"2024-05-09T03:09:36.914719+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:43.778183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223417+0000","last_change":"2024-05-09T03:09:34.201228+0000","last_active":"2024-05-09T03:09:35.223417+0000","last_peered":"2024-05-09T03:09:35.223417+0000","last_clean":"2024-05-09T03:09:35.223417+0000","last_became_active":"2024-05-09T03:09:34.200894+0000","last_became_peered":"2024-05-09T03:09:34.200894+0000","last_unstale":"2024-05-09T03:09:35.223417+0000","last_undegraded":"2024-05-09T03:09:35.223417+0000","last_fullsized":"2024-05-09T03:09:35.223417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914339+0000","last_change":"2024-05-09T03:09:34.189092+0000","last_active":"2024-05-09T03:09:36.914339+0000","last_peered":"2024-05-09T03:09:36.914339+0000","last_clean":"2024-05-09T03:09:36.914339+0000","last_became_active":"2024-05-09T03:09:34.188774+0000","last_became_peered":"2024-05-09T03:09:34.188774+0000","last_unstale":"2024-05-09T03:09:36.914339+0000","last_undegraded":"2024-05-09T03:09:36.914339+0000","last_fullsized":"2024-05-09T03:09:36.914339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223434+0000","last_change":"2024-05-09T03:09:34.201623+0000","last_active":"2024-05-09T03:09:35.223434+0000","last_peered":"2024-05-09T03:09:35.223434+0000","last_clean":"2024-05-09T03:09:35.223434+0000","last_became_active":"2024-05-09T03:09:34.200070+0000","last_became_peered":"2024-05-09T03:09:34.200070+0000","last_unstale":"2024-05-09T03:09:35.223434+0000","last_undegraded":"2024-05-09T03:09:35.223434+0000","last_fullsized":"2024-05-09T03:09:35.223434+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915317+0000","last_change":"2024-05-09T03:09:34.189949+0000","last_active":"2024-05-09T03:09:36.915317+0000","last_peered":"2024-05-09T03:09:36.915317+0000","last_clean":"2024-05-09T03:09:36.915317+0000","last_became_active":"2024-05-09T03:09:34.189595+0000","last_became_peered":"2024-05-09T03:09:34.189595+0000","last_unstale":"2024-05-09T03:09:36.915317+0000","last_undegraded":"2024-05-09T03:09:36.915317+0000","last_fullsized":"2024-05-09T03:09:36.915317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:33.100263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.196011+0000","last_change":"2024-05-09T03:09:34.188475+0000","last_active":"2024-05-09T03:09:35.196011+0000","last_peered":"2024-05-09T03:09:35.196011+0000","last_clean":"2024-05-09T03:09:35.196011+0000","last_became_active":"2024-05-09T03:09:34.188128+0000","last_became_peered":"2024-05-09T03:09:34.188128+0000","last_unstale":"2024-05-09T03:09:35.196011+0000","last_undegraded":"2024-05-09T03:09:35.196011+0000","last_fullsized":"2024-05-09T03:09:35.196011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191594+0000","last_change":"2024-05-09T03:09:34.191453+0000","last_active":"2024-05-09T03:09:35.191594+0000","last_peered":"2024-05-09T03:09:35.191594+0000","last_clean":"2024-05-09T03:09:35.191594+0000","last_became_active":"2024-05-09T03:09:34.191094+0000","last_became_peered":"2024-05-09T03:09:34.191094+0000","last_unstale":"2024-05-09T03:09:35.191594+0000","last_undegraded":"2024-05-09T03:09:35.191594+0000","last_fullsized":"2024-05-09T03:09:35.191594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915216+0000","last_change":"2024-05-09T03:09:34.201091+0000","last_active":"2024-05-09T03:09:36.915216+0000","last_peered":"2024-05-09T03:09:36.915216+0000","last_clean":"2024-05-09T03:09:36.915216+0000","last_became_active":"2024-05-09T03:09:34.198235+0000","last_became_peered":"2024-05-09T03:09:34.198235+0000","last_unstale":"2024-05-09T03:09:36.915216+0000","last_undegraded":"2024-05-09T03:09:36.915216+0000","last_fullsized":"2024-05-09T03:09:36.915216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914899+0000","last_change":"2024-05-09T03:09:34.196239+0000","last_active":"2024-05-09T03:09:36.914899+0000","last_peered":"2024-05-09T03:09:36.914899+0000","last_clean":"2024-05-09T03:09:36.914899+0000","last_became_active":"2024-05-09T03:09:34.195895+0000","last_became_peered":"2024-05-09T03:09:34.195895+0000","last_unstale":"2024-05-09T03:09:36.914899+0000","last_undegraded":"2024-05-09T03:09:36.914899+0000","last_fullsized":"2024-05-09T03:09:36.914899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223858+0000","last_change":"2024-05-09T03:09:34.206560+0000","last_active":"2024-05-09T03:09:35.223858+0000","last_peered":"2024-05-09T03:09:35.223858+0000","last_clean":"2024-05-09T03:09:35.223858+0000","last_became_active":"2024-05-09T03:09:34.205417+0000","last_became_peered":"2024-05-09T03:09:34.205417+0000","last_unstale":"2024-05-09T03:09:35.223858+0000","last_undegraded":"2024-05-09T03:09:35.223858+0000","last_fullsized":"2024-05-09T03:09:35.223858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:59:37.733696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913999+0000","last_change":"2024-05-09T03:09:34.206717+0000","last_active":"2024-05-09T03:09:36.913999+0000","last_peered":"2024-05-09T03:09:36.913999+0000","last_clean":"2024-05-09T03:09:36.913999+0000","last_became_active":"2024-05-09T03:09:34.206391+0000","last_became_peered":"2024-05-09T03:09:34.206391+0000","last_unstale":"2024-05-09T03:09:36.913999+0000","last_undegraded":"2024-05-09T03:09:36.913999+0000","last_fullsized":"2024-05-09T03:09:36.913999+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194450+0000","last_change":"2024-05-09T03:09:34.181522+0000","last_active":"2024-05-09T03:09:35.194450+0000","last_peered":"2024-05-09T03:09:35.194450+0000","last_clean":"2024-05-09T03:09:35.194450+0000","last_became_active":"2024-05-09T03:09:34.180437+0000","last_became_peered":"2024-05-09T03:09:34.180437+0000","last_unstale":"2024-05-09T03:09:35.194450+0000","last_undegraded":"2024-05-09T03:09:35.194450+0000","last_fullsized":"2024-05-09T03:09:35.194450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.194687+0000","last_change":"2024-05-09T03:09:34.187744+0000","last_active":"2024-05-09T03:09:35.194687+0000","last_peered":"2024-05-09T03:09:35.194687+0000","last_clean":"2024-05-09T03:09:35.194687+0000","last_became_active":"2024-05-09T03:09:34.187413+0000","last_became_peered":"2024-05-09T03:09:34.187413+0000","last_unstale":"2024-05-09T03:09:35.194687+0000","last_undegraded":"2024-05-09T03:09:35.194687+0000","last_fullsized":"2024-05-09T03:09:35.194687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:00.147452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915534+0000","last_change":"2024-05-09T03:09:33.189245+0000","last_active":"2024-05-09T03:09:36.915534+0000","last_peered":"2024-05-09T03:09:36.915534+0000","last_clean":"2024-05-09T03:09:36.915534+0000","last_became_active":"2024-05-09T03:09:33.188865+0000","last_became_peered":"2024-05-09T03:09:33.188865+0000","last_unstale":"2024-05-09T03:09:36.915534+0000","last_undegraded":"2024-05-09T03:09:36.915534+0000","last_fullsized":"2024-05-09T03:09:36.915534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915442+0000","last_change":"2024-05-09T03:09:34.183515+0000","last_active":"2024-05-09T03:09:36.915442+0000","last_peered":"2024-05-09T03:09:36.915442+0000","last_clean":"2024-05-09T03:09:36.915442+0000","last_became_active":"2024-05-09T03:09:34.183151+0000","last_became_peered":"2024-05-09T03:09:34.183151+0000","last_unstale":"2024-05-09T03:09:36.915442+0000","last_undegraded":"2024-05-09T03:09:36.915442+0000","last_fullsized":"2024-05-09T03:09:36.915442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.196036+0000","last_change":"2024-05-09T03:09:33.177972+0000","last_active":"2024-05-09T03:09:35.196036+0000","last_peered":"2024-05-09T03:09:35.196036+0000","last_clean":"2024-05-09T03:09:35.196036+0000","last_became_active":"2024-05-09T03:09:33.177585+0000","last_became_peered":"2024-05-09T03:09:33.177585+0000","last_unstale":"2024-05-09T03:09:35.196036+0000","last_undegraded":"2024-05-09T03:09:35.196036+0000","last_fullsized":"2024-05-09T03:09:35.196036+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914511+0000","last_change":"2024-05-09T03:09:33.193553+0000","last_active":"2024-05-09T03:09:36.914511+0000","last_peered":"2024-05-09T03:09:36.914511+0000","last_clean":"2024-05-09T03:09:36.914511+0000","last_became_active":"2024-05-09T03:09:33.193050+0000","last_became_peered":"2024-05-09T03:09:33.193050+0000","last_unstale":"2024-05-09T03:09:36.914511+0000","last_undegraded":"2024-05-09T03:09:36.914511+0000","last_fullsized":"2024-05-09T03:09:36.914511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:28.743738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914265+0000","last_change":"2024-05-09T03:09:34.193946+0000","last_active":"2024-05-09T03:09:36.914265+0000","last_peered":"2024-05-09T03:09:36.914265+0000","last_clean":"2024-05-09T03:09:36.914265+0000","last_became_active":"2024-05-09T03:09:34.192276+0000","last_became_peered":"2024-05-09T03:09:34.192276+0000","last_unstale":"2024-05-09T03:09:36.914265+0000","last_undegraded":"2024-05-09T03:09:36.914265+0000","last_fullsized":"2024-05-09T03:09:36.914265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223499+0000","last_change":"2024-05-09T03:09:33.183226+0000","last_active":"2024-05-09T03:09:35.223499+0000","last_peered":"2024-05-09T03:09:35.223499+0000","last_clean":"2024-05-09T03:09:35.223499+0000","last_became_active":"2024-05-09T03:09:33.182875+0000","last_became_peered":"2024-05-09T03:09:33.182875+0000","last_unstale":"2024-05-09T03:09:35.223499+0000","last_undegraded":"2024-05-09T03:09:35.223499+0000","last_fullsized":"2024-05-09T03:09:35.223499+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191627+0000","last_change":"2024-05-09T03:09:34.192348+0000","last_active":"2024-05-09T03:09:35.191627+0000","last_peered":"2024-05-09T03:09:35.191627+0000","last_clean":"2024-05-09T03:09:35.191627+0000","last_became_active":"2024-05-09T03:09:34.192007+0000","last_became_peered":"2024-05-09T03:09:34.192007+0000","last_unstale":"2024-05-09T03:09:35.191627+0000","last_undegraded":"2024-05-09T03:09:35.191627+0000","last_fullsized":"2024-05-09T03:09:35.191627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195519+0000","last_change":"2024-05-09T03:09:33.189847+0000","last_active":"2024-05-09T03:09:35.195519+0000","last_peered":"2024-05-09T03:09:35.195519+0000","last_clean":"2024-05-09T03:09:35.195519+0000","last_became_active":"2024-05-09T03:09:33.189107+0000","last_became_peered":"2024-05-09T03:09:33.189107+0000","last_unstale":"2024-05-09T03:09:35.195519+0000","last_undegraded":"2024-05-09T03:09:35.195519+0000","last_fullsized":"2024-05-09T03:09:35.195519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223574+0000","last_change":"2024-05-09T03:09:34.198289+0000","last_active":"2024-05-09T03:09:35.223574+0000","last_peered":"2024-05-09T03:09:35.223574+0000","last_clean":"2024-05-09T03:09:35.223574+0000","last_became_active":"2024-05-09T03:09:34.197367+0000","last_became_peered":"2024-05-09T03:09:34.197367+0000","last_unstale":"2024-05-09T03:09:35.223574+0000","last_undegraded":"2024-05-09T03:09:35.223574+0000","last_fullsized":"2024-05-09T03:09:35.223574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223764+0000","last_change":"2024-05-09T03:09:33.188844+0000","last_active":"2024-05-09T03:09:35.223764+0000","last_peered":"2024-05-09T03:09:35.223764+0000","last_clean":"2024-05-09T03:09:35.223764+0000","last_became_active":"2024-05-09T03:09:33.188348+0000","last_became_peered":"2024-05-09T03:09:33.188348+0000","last_unstale":"2024-05-09T03:09:35.223764+0000","last_undegraded":"2024-05-09T03:09:35.223764+0000","last_fullsized":"2024-05-09T03:09:35.223764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915657+0000","last_change":"2024-05-09T03:09:34.199361+0000","last_active":"2024-05-09T03:09:36.915657+0000","last_peered":"2024-05-09T03:09:36.915657+0000","last_clean":"2024-05-09T03:09:36.915657+0000","last_became_active":"2024-05-09T03:09:34.199023+0000","last_became_peered":"2024-05-09T03:09:34.199023+0000","last_unstale":"2024-05-09T03:09:36.915657+0000","last_undegraded":"2024-05-09T03:09:36.915657+0000","last_fullsized":"2024-05-09T03:09:36.915657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:35:44.725666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191595+0000","last_change":"2024-05-09T03:09:33.193443+0000","last_active":"2024-05-09T03:09:35.191595+0000","last_peered":"2024-05-09T03:09:35.191595+0000","last_clean":"2024-05-09T03:09:35.191595+0000","last_became_active":"2024-05-09T03:09:33.193095+0000","last_became_peered":"2024-05-09T03:09:33.193095+0000","last_unstale":"2024-05-09T03:09:35.191595+0000","last_undegraded":"2024-05-09T03:09:35.191595+0000","last_fullsized":"2024-05-09T03:09:35.191595+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195302+0000","last_change":"2024-05-09T03:09:34.190211+0000","last_active":"2024-05-09T03:09:35.195302+0000","last_peered":"2024-05-09T03:09:35.195302+0000","last_clean":"2024-05-09T03:09:35.195302+0000","last_became_active":"2024-05-09T03:09:34.189747+0000","last_became_peered":"2024-05-09T03:09:34.189747+0000","last_unstale":"2024-05-09T03:09:35.195302+0000","last_undegraded":"2024-05-09T03:09:35.195302+0000","last_fullsized":"2024-05-09T03:09:35.195302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:52.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195680+0000","last_change":"2024-05-09T03:09:33.186806+0000","last_active":"2024-05-09T03:09:35.195680+0000","last_peered":"2024-05-09T03:09:35.195680+0000","last_clean":"2024-05-09T03:09:35.195680+0000","last_became_active":"2024-05-09T03:09:33.185580+0000","last_became_peered":"2024-05-09T03:09:33.185580+0000","last_unstale":"2024-05-09T03:09:35.195680+0000","last_undegraded":"2024-05-09T03:09:35.195680+0000","last_fullsized":"2024-05-09T03:09:35.195680+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914632+0000","last_change":"2024-05-09T03:09:34.191009+0000","last_active":"2024-05-09T03:09:36.914632+0000","last_peered":"2024-05-09T03:09:36.914632+0000","last_clean":"2024-05-09T03:09:36.914632+0000","last_became_active":"2024-05-09T03:09:34.190131+0000","last_became_peered":"2024-05-09T03:09:34.190131+0000","last_unstale":"2024-05-09T03:09:36.914632+0000","last_undegraded":"2024-05-09T03:09:36.914632+0000","last_fullsized":"2024-05-09T03:09:36.914632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.607505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915064+0000","last_change":"2024-05-09T03:09:33.185616+0000","last_active":"2024-05-09T03:09:36.915064+0000","last_peered":"2024-05-09T03:09:36.915064+0000","last_clean":"2024-05-09T03:09:36.915064+0000","last_became_active":"2024-05-09T03:09:33.185302+0000","last_became_peered":"2024-05-09T03:09:33.185302+0000","last_unstale":"2024-05-09T03:09:36.915064+0000","last_undegraded":"2024-05-09T03:09:36.915064+0000","last_fullsized":"2024-05-09T03:09:36.915064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:28.743738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914805+0000","last_change":"2024-05-09T03:09:34.190554+0000","last_active":"2024-05-09T03:09:36.914805+0000","last_peered":"2024-05-09T03:09:36.914805+0000","last_clean":"2024-05-09T03:09:36.914805+0000","last_became_active":"2024-05-09T03:09:34.190165+0000","last_became_peered":"2024-05-09T03:09:34.190165+0000","last_unstale":"2024-05-09T03:09:36.914805+0000","last_undegraded":"2024-05-09T03:09:36.914805+0000","last_fullsized":"2024-05-09T03:09:36.914805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:58.717521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223697+0000","last_change":"2024-05-09T03:09:33.191326+0000","last_active":"2024-05-09T03:09:35.223697+0000","last_peered":"2024-05-09T03:09:35.223697+0000","last_clean":"2024-05-09T03:09:35.223697+0000","last_became_active":"2024-05-09T03:09:33.190974+0000","last_became_peered":"2024-05-09T03:09:33.190974+0000","last_unstale":"2024-05-09T03:09:35.223697+0000","last_undegraded":"2024-05-09T03:09:35.223697+0000","last_fullsized":"2024-05-09T03:09:35.223697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914599+0000","last_change":"2024-05-09T03:09:34.195650+0000","last_active":"2024-05-09T03:09:36.914599+0000","last_peered":"2024-05-09T03:09:36.914599+0000","last_clean":"2024-05-09T03:09:36.914599+0000","last_became_active":"2024-05-09T03:09:34.195236+0000","last_became_peered":"2024-05-09T03:09:34.195236+0000","last_unstale":"2024-05-09T03:09:36.914599+0000","last_undegraded":"2024-05-09T03:09:36.914599+0000","last_fullsized":"2024-05-09T03:09:36.914599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:41.381783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195554+0000","last_change":"2024-05-09T03:09:33.184933+0000","last_active":"2024-05-09T03:09:35.195554+0000","last_peered":"2024-05-09T03:09:35.195554+0000","last_clean":"2024-05-09T03:09:35.195554+0000","last_became_active":"2024-05-09T03:09:33.184584+0000","last_became_peered":"2024-05-09T03:09:33.184584+0000","last_unstale":"2024-05-09T03:09:35.195554+0000","last_undegraded":"2024-05-09T03:09:35.195554+0000","last_fullsized":"2024-05-09T03:09:35.195554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195463+0000","last_change":"2024-05-09T03:09:34.179608+0000","last_active":"2024-05-09T03:09:35.195463+0000","last_peered":"2024-05-09T03:09:35.195463+0000","last_clean":"2024-05-09T03:09:35.195463+0000","last_became_active":"2024-05-09T03:09:34.179281+0000","last_became_peered":"2024-05-09T03:09:34.179281+0000","last_unstale":"2024-05-09T03:09:35.195463+0000","last_undegraded":"2024-05-09T03:09:35.195463+0000","last_fullsized":"2024-05-09T03:09:35.195463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913853+0000","last_change":"2024-05-09T03:09:33.183294+0000","last_active":"2024-05-09T03:09:36.913853+0000","last_peered":"2024-05-09T03:09:36.913853+0000","last_clean":"2024-05-09T03:09:36.913853+0000","last_became_active":"2024-05-09T03:09:33.182789+0000","last_became_peered":"2024-05-09T03:09:33.182789+0000","last_unstale":"2024-05-09T03:09:36.913853+0000","last_undegraded":"2024-05-09T03:09:36.913853+0000","last_fullsized":"2024-05-09T03:09:36.913853+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914435+0000","last_change":"2024-05-09T03:09:34.195805+0000","last_active":"2024-05-09T03:09:36.914435+0000","last_peered":"2024-05-09T03:09:36.914435+0000","last_clean":"2024-05-09T03:09:36.914435+0000","last_became_active":"2024-05-09T03:09:34.195460+0000","last_became_peered":"2024-05-09T03:09:34.195460+0000","last_unstale":"2024-05-09T03:09:36.914435+0000","last_undegraded":"2024-05-09T03:09:36.914435+0000","last_fullsized":"2024-05-09T03:09:36.914435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:07:13.552991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195504+0000","last_change":"2024-05-09T03:09:33.182939+0000","last_active":"2024-05-09T03:09:35.195504+0000","last_peered":"2024-05-09T03:09:35.195504+0000","last_clean":"2024-05-09T03:09:35.195504+0000","last_became_active":"2024-05-09T03:09:33.182611+0000","last_became_peered":"2024-05-09T03:09:33.182611+0000","last_unstale":"2024-05-09T03:09:35.195504+0000","last_undegraded":"2024-05-09T03:09:35.195504+0000","last_fullsized":"2024-05-09T03:09:35.195504+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:18.783039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914887+0000","last_change":"2024-05-09T03:09:34.190860+0000","last_active":"2024-05-09T03:09:36.914887+0000","last_peered":"2024-05-09T03:09:36.914887+0000","last_clean":"2024-05-09T03:09:36.914887+0000","last_became_active":"2024-05-09T03:09:34.190509+0000","last_became_peered":"2024-05-09T03:09:34.190509+0000","last_unstale":"2024-05-09T03:09:36.914887+0000","last_undegraded":"2024-05-09T03:09:36.914887+0000","last_fullsized":"2024-05-09T03:09:36.914887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:06.122864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914801+0000","last_change":"2024-05-09T03:09:33.187823+0000","last_active":"2024-05-09T03:09:36.914801+0000","last_peered":"2024-05-09T03:09:36.914801+0000","last_clean":"2024-05-09T03:09:36.914801+0000","last_became_active":"2024-05-09T03:09:33.187487+0000","last_became_peered":"2024-05-09T03:09:33.187487+0000","last_unstale":"2024-05-09T03:09:36.914801+0000","last_undegraded":"2024-05-09T03:09:36.914801+0000","last_fullsized":"2024-05-09T03:09:36.914801+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:03.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915118+0000","last_change":"2024-05-09T03:09:34.196111+0000","last_active":"2024-05-09T03:09:36.915118+0000","last_peered":"2024-05-09T03:09:36.915118+0000","last_clean":"2024-05-09T03:09:36.915118+0000","last_became_active":"2024-05-09T03:09:34.195741+0000","last_became_peered":"2024-05-09T03:09:34.195741+0000","last_unstale":"2024-05-09T03:09:36.915118+0000","last_undegraded":"2024-05-09T03:09:36.915118+0000","last_fullsized":"2024-05-09T03:09:36.915118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:13:32.501193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191690+0000","last_change":"2024-05-09T03:09:33.189105+0000","last_active":"2024-05-09T03:09:35.191690+0000","last_peered":"2024-05-09T03:09:35.191690+0000","last_clean":"2024-05-09T03:09:35.191690+0000","last_became_active":"2024-05-09T03:09:33.188557+0000","last_became_peered":"2024-05-09T03:09:33.188557+0000","last_unstale":"2024-05-09T03:09:35.191690+0000","last_undegraded":"2024-05-09T03:09:35.191690+0000","last_fullsized":"2024-05-09T03:09:35.191690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:40.503702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223831+0000","last_change":"2024-05-09T03:09:34.205860+0000","last_active":"2024-05-09T03:09:35.223831+0000","last_peered":"2024-05-09T03:09:35.223831+0000","last_clean":"2024-05-09T03:09:35.223831+0000","last_became_active":"2024-05-09T03:09:34.205494+0000","last_became_peered":"2024-05-09T03:09:34.205494+0000","last_unstale":"2024-05-09T03:09:35.223831+0000","last_undegraded":"2024-05-09T03:09:35.223831+0000","last_fullsized":"2024-05-09T03:09:35.223831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.607505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914248+0000","last_change":"2024-05-09T03:09:33.191678+0000","last_active":"2024-05-09T03:09:36.914248+0000","last_peered":"2024-05-09T03:09:36.914248+0000","last_clean":"2024-05-09T03:09:36.914248+0000","last_became_active":"2024-05-09T03:09:33.191347+0000","last_became_peered":"2024-05-09T03:09:33.191347+0000","last_unstale":"2024-05-09T03:09:36.914248+0000","last_undegraded":"2024-05-09T03:09:36.914248+0000","last_fullsized":"2024-05-09T03:09:36.914248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914843+0000","last_change":"2024-05-09T03:09:34.200500+0000","last_active":"2024-05-09T03:09:36.914843+0000","last_peered":"2024-05-09T03:09:36.914843+0000","last_clean":"2024-05-09T03:09:36.914843+0000","last_became_active":"2024-05-09T03:09:34.200204+0000","last_became_peered":"2024-05-09T03:09:34.200204+0000","last_unstale":"2024-05-09T03:09:36.914843+0000","last_undegraded":"2024-05-09T03:09:36.914843+0000","last_fullsized":"2024-05-09T03:09:36.914843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:59:37.733696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":110,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914436+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:09:36.914436+0000","last_peered":"2024-05-09T03:09:36.914436+0000","last_clean":"2024-05-09T03:09:36.914436+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:09:36.914436+0000","last_undegraded":"2024-05-09T03:09:36.914436+0000","last_fullsized":"2024-05-09T03:09:36.914436+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223311+0000","last_change":"2024-05-09T03:09:33.189141+0000","last_active":"2024-05-09T03:09:35.223311+0000","last_peered":"2024-05-09T03:09:35.223311+0000","last_clean":"2024-05-09T03:09:35.223311+0000","last_became_active":"2024-05-09T03:09:33.188576+0000","last_became_peered":"2024-05-09T03:09:33.188576+0000","last_unstale":"2024-05-09T03:09:35.223311+0000","last_undegraded":"2024-05-09T03:09:35.223311+0000","last_fullsized":"2024-05-09T03:09:35.223311+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:15.428039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914857+0000","last_change":"2024-05-09T03:09:34.202021+0000","last_active":"2024-05-09T03:09:36.914857+0000","last_peered":"2024-05-09T03:09:36.914857+0000","last_clean":"2024-05-09T03:09:36.914857+0000","last_became_active":"2024-05-09T03:09:34.201661+0000","last_became_peered":"2024-05-09T03:09:34.201661+0000","last_unstale":"2024-05-09T03:09:36.914857+0000","last_undegraded":"2024-05-09T03:09:36.914857+0000","last_fullsized":"2024-05-09T03:09:36.914857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914241+0000","last_change":"2024-05-09T03:09:33.189640+0000","last_active":"2024-05-09T03:09:36.914241+0000","last_peered":"2024-05-09T03:09:36.914241+0000","last_clean":"2024-05-09T03:09:36.914241+0000","last_became_active":"2024-05-09T03:09:33.189291+0000","last_became_peered":"2024-05-09T03:09:33.189291+0000","last_unstale":"2024-05-09T03:09:36.914241+0000","last_undegraded":"2024-05-09T03:09:36.914241+0000","last_fullsized":"2024-05-09T03:09:36.914241+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:56.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915084+0000","last_change":"2024-05-09T03:09:34.186522+0000","last_active":"2024-05-09T03:09:36.915084+0000","last_peered":"2024-05-09T03:09:36.915084+0000","last_clean":"2024-05-09T03:09:36.915084+0000","last_became_active":"2024-05-09T03:09:34.186183+0000","last_became_peered":"2024-05-09T03:09:34.186183+0000","last_unstale":"2024-05-09T03:09:36.915084+0000","last_undegraded":"2024-05-09T03:09:36.915084+0000","last_fullsized":"2024-05-09T03:09:36.915084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:08:26.347742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.913979+0000","last_change":"2024-05-09T03:09:33.186397+0000","last_active":"2024-05-09T03:09:36.913979+0000","last_peered":"2024-05-09T03:09:36.913979+0000","last_clean":"2024-05-09T03:09:36.913979+0000","last_became_active":"2024-05-09T03:09:33.186026+0000","last_became_peered":"2024-05-09T03:09:33.186026+0000","last_unstale":"2024-05-09T03:09:36.913979+0000","last_undegraded":"2024-05-09T03:09:36.913979+0000","last_fullsized":"2024-05-09T03:09:36.913979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191563+0000","last_change":"2024-05-09T03:09:34.206024+0000","last_active":"2024-05-09T03:09:35.191563+0000","last_peered":"2024-05-09T03:09:35.191563+0000","last_clean":"2024-05-09T03:09:35.191563+0000","last_became_active":"2024-05-09T03:09:34.205694+0000","last_became_peered":"2024-05-09T03:09:34.205694+0000","last_unstale":"2024-05-09T03:09:35.191563+0000","last_undegraded":"2024-05-09T03:09:35.191563+0000","last_fullsized":"2024-05-09T03:09:35.191563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:37:08.150041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223876+0000","last_change":"2024-05-09T03:09:33.189135+0000","last_active":"2024-05-09T03:09:35.223876+0000","last_peered":"2024-05-09T03:09:35.223876+0000","last_clean":"2024-05-09T03:09:35.223876+0000","last_became_active":"2024-05-09T03:09:33.188869+0000","last_became_peered":"2024-05-09T03:09:33.188869+0000","last_unstale":"2024-05-09T03:09:35.223876+0000","last_undegraded":"2024-05-09T03:09:35.223876+0000","last_fullsized":"2024-05-09T03:09:35.223876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195454+0000","last_change":"2024-05-09T03:09:34.189609+0000","last_active":"2024-05-09T03:09:35.195454+0000","last_peered":"2024-05-09T03:09:35.195454+0000","last_clean":"2024-05-09T03:09:35.195454+0000","last_became_active":"2024-05-09T03:09:34.189088+0000","last_became_peered":"2024-05-09T03:09:34.189088+0000","last_unstale":"2024-05-09T03:09:35.195454+0000","last_undegraded":"2024-05-09T03:09:35.195454+0000","last_fullsized":"2024-05-09T03:09:35.195454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:41.508915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191407+0000","last_change":"2024-05-09T03:09:33.184995+0000","last_active":"2024-05-09T03:09:35.191407+0000","last_peered":"2024-05-09T03:09:35.191407+0000","last_clean":"2024-05-09T03:09:35.191407+0000","last_became_active":"2024-05-09T03:09:33.184596+0000","last_became_peered":"2024-05-09T03:09:33.184596+0000","last_unstale":"2024-05-09T03:09:35.191407+0000","last_undegraded":"2024-05-09T03:09:35.191407+0000","last_fullsized":"2024-05-09T03:09:35.191407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223976+0000","last_change":"2024-05-09T03:09:34.205632+0000","last_active":"2024-05-09T03:09:35.223976+0000","last_peered":"2024-05-09T03:09:35.223976+0000","last_clean":"2024-05-09T03:09:35.223976+0000","last_became_active":"2024-05-09T03:09:34.205275+0000","last_became_peered":"2024-05-09T03:09:34.205275+0000","last_unstale":"2024-05-09T03:09:35.223976+0000","last_undegraded":"2024-05-09T03:09:35.223976+0000","last_fullsized":"2024-05-09T03:09:35.223976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:02.707927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915021+0000","last_change":"2024-05-09T03:09:33.193320+0000","last_active":"2024-05-09T03:09:36.915021+0000","last_peered":"2024-05-09T03:09:36.915021+0000","last_clean":"2024-05-09T03:09:36.915021+0000","last_became_active":"2024-05-09T03:09:33.193005+0000","last_became_peered":"2024-05-09T03:09:33.193005+0000","last_unstale":"2024-05-09T03:09:36.915021+0000","last_undegraded":"2024-05-09T03:09:36.915021+0000","last_fullsized":"2024-05-09T03:09:36.915021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:44:40.503702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191290+0000","last_change":"2024-05-09T03:09:34.204993+0000","last_active":"2024-05-09T03:09:35.191290+0000","last_peered":"2024-05-09T03:09:35.191290+0000","last_clean":"2024-05-09T03:09:35.191290+0000","last_became_active":"2024-05-09T03:09:34.204647+0000","last_became_peered":"2024-05-09T03:09:34.204647+0000","last_unstale":"2024-05-09T03:09:35.191290+0000","last_undegraded":"2024-05-09T03:09:35.191290+0000","last_fullsized":"2024-05-09T03:09:35.191290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:10:33.852073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191516+0000","last_change":"2024-05-09T03:09:33.188474+0000","last_active":"2024-05-09T03:09:35.191516+0000","last_peered":"2024-05-09T03:09:35.191516+0000","last_clean":"2024-05-09T03:09:35.191516+0000","last_became_active":"2024-05-09T03:09:33.187643+0000","last_became_peered":"2024-05-09T03:09:33.187643+0000","last_unstale":"2024-05-09T03:09:35.191516+0000","last_undegraded":"2024-05-09T03:09:35.191516+0000","last_fullsized":"2024-05-09T03:09:35.191516+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:04.365646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915505+0000","last_change":"2024-05-09T03:09:34.190064+0000","last_active":"2024-05-09T03:09:36.915505+0000","last_peered":"2024-05-09T03:09:36.915505+0000","last_clean":"2024-05-09T03:09:36.915505+0000","last_became_active":"2024-05-09T03:09:34.189695+0000","last_became_peered":"2024-05-09T03:09:34.189695+0000","last_unstale":"2024-05-09T03:09:36.915505+0000","last_undegraded":"2024-05-09T03:09:36.915505+0000","last_fullsized":"2024-05-09T03:09:36.915505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:06.036989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195648+0000","last_change":"2024-05-09T03:09:33.188283+0000","last_active":"2024-05-09T03:09:35.195648+0000","last_peered":"2024-05-09T03:09:35.195648+0000","last_clean":"2024-05-09T03:09:35.195648+0000","last_became_active":"2024-05-09T03:09:33.187986+0000","last_became_peered":"2024-05-09T03:09:33.187986+0000","last_unstale":"2024-05-09T03:09:35.195648+0000","last_undegraded":"2024-05-09T03:09:35.195648+0000","last_fullsized":"2024-05-09T03:09:35.195648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:59.142240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915245+0000","last_change":"2024-05-09T03:09:34.188118+0000","last_active":"2024-05-09T03:09:36.915245+0000","last_peered":"2024-05-09T03:09:36.915245+0000","last_clean":"2024-05-09T03:09:36.915245+0000","last_became_active":"2024-05-09T03:09:34.187756+0000","last_became_peered":"2024-05-09T03:09:34.187756+0000","last_unstale":"2024-05-09T03:09:36.915245+0000","last_undegraded":"2024-05-09T03:09:36.915245+0000","last_fullsized":"2024-05-09T03:09:36.915245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914221+0000","last_change":"2024-05-09T03:09:33.191538+0000","last_active":"2024-05-09T03:09:36.914221+0000","last_peered":"2024-05-09T03:09:36.914221+0000","last_clean":"2024-05-09T03:09:36.914221+0000","last_became_active":"2024-05-09T03:09:33.190940+0000","last_became_peered":"2024-05-09T03:09:33.190940+0000","last_unstale":"2024-05-09T03:09:36.914221+0000","last_undegraded":"2024-05-09T03:09:36.914221+0000","last_fullsized":"2024-05-09T03:09:36.914221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:29:16.121128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195374+0000","last_change":"2024-05-09T03:09:34.203831+0000","last_active":"2024-05-09T03:09:35.195374+0000","last_peered":"2024-05-09T03:09:35.195374+0000","last_clean":"2024-05-09T03:09:35.195374+0000","last_became_active":"2024-05-09T03:09:34.203509+0000","last_became_peered":"2024-05-09T03:09:34.203509+0000","last_unstale":"2024-05-09T03:09:35.195374+0000","last_undegraded":"2024-05-09T03:09:35.195374+0000","last_fullsized":"2024-05-09T03:09:35.195374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:05.370858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191428+0000","last_change":"2024-05-09T03:09:33.188345+0000","last_active":"2024-05-09T03:09:35.191428+0000","last_peered":"2024-05-09T03:09:35.191428+0000","last_clean":"2024-05-09T03:09:35.191428+0000","last_became_active":"2024-05-09T03:09:33.187996+0000","last_became_peered":"2024-05-09T03:09:33.187996+0000","last_unstale":"2024-05-09T03:09:35.191428+0000","last_undegraded":"2024-05-09T03:09:35.191428+0000","last_fullsized":"2024-05-09T03:09:35.191428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914440+0000","last_change":"2024-05-09T03:09:34.188063+0000","last_active":"2024-05-09T03:09:36.914440+0000","last_peered":"2024-05-09T03:09:36.914440+0000","last_clean":"2024-05-09T03:09:36.914440+0000","last_became_active":"2024-05-09T03:09:34.187703+0000","last_became_peered":"2024-05-09T03:09:34.187703+0000","last_unstale":"2024-05-09T03:09:36.914440+0000","last_undegraded":"2024-05-09T03:09:36.914440+0000","last_fullsized":"2024-05-09T03:09:36.914440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.191401+0000","last_change":"2024-05-09T03:09:33.191733+0000","last_active":"2024-05-09T03:09:35.191401+0000","last_peered":"2024-05-09T03:09:35.191401+0000","last_clean":"2024-05-09T03:09:35.191401+0000","last_became_active":"2024-05-09T03:09:33.190856+0000","last_became_peered":"2024-05-09T03:09:33.190856+0000","last_unstale":"2024-05-09T03:09:35.191401+0000","last_undegraded":"2024-05-09T03:09:35.191401+0000","last_fullsized":"2024-05-09T03:09:35.191401+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:05.249055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915388+0000","last_change":"2024-05-09T03:09:34.201161+0000","last_active":"2024-05-09T03:09:36.915388+0000","last_peered":"2024-05-09T03:09:36.915388+0000","last_clean":"2024-05-09T03:09:36.915388+0000","last_became_active":"2024-05-09T03:09:34.200789+0000","last_became_peered":"2024-05-09T03:09:34.200789+0000","last_unstale":"2024-05-09T03:09:36.915388+0000","last_undegraded":"2024-05-09T03:09:36.915388+0000","last_fullsized":"2024-05-09T03:09:36.915388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:07.581376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914778+0000","last_change":"2024-05-09T03:09:33.191322+0000","last_active":"2024-05-09T03:09:36.914778+0000","last_peered":"2024-05-09T03:09:36.914778+0000","last_clean":"2024-05-09T03:09:36.914778+0000","last_became_active":"2024-05-09T03:09:33.190925+0000","last_became_peered":"2024-05-09T03:09:33.190925+0000","last_unstale":"2024-05-09T03:09:36.914778+0000","last_undegraded":"2024-05-09T03:09:36.914778+0000","last_fullsized":"2024-05-09T03:09:36.914778+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:33:20.918752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.195950+0000","last_change":"2024-05-09T03:09:34.181003+0000","last_active":"2024-05-09T03:09:35.195950+0000","last_peered":"2024-05-09T03:09:35.195950+0000","last_clean":"2024-05-09T03:09:35.195950+0000","last_became_active":"2024-05-09T03:09:34.180247+0000","last_became_peered":"2024-05-09T03:09:34.180247+0000","last_unstale":"2024-05-09T03:09:35.195950+0000","last_undegraded":"2024-05-09T03:09:35.195950+0000","last_fullsized":"2024-05-09T03:09:35.195950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:13:50.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223857+0000","last_change":"2024-05-09T03:09:34.207920+0000","last_active":"2024-05-09T03:09:35.223857+0000","last_peered":"2024-05-09T03:09:35.223857+0000","last_clean":"2024-05-09T03:09:35.223857+0000","last_became_active":"2024-05-09T03:09:34.207645+0000","last_became_peered":"2024-05-09T03:09:34.207645+0000","last_unstale":"2024-05-09T03:09:35.223857+0000","last_undegraded":"2024-05-09T03:09:35.223857+0000","last_fullsized":"2024-05-09T03:09:35.223857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:06.122864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914557+0000","last_change":"2024-05-09T03:09:33.194189+0000","last_active":"2024-05-09T03:09:36.914557+0000","last_peered":"2024-05-09T03:09:36.914557+0000","last_clean":"2024-05-09T03:09:36.914557+0000","last_became_active":"2024-05-09T03:09:33.193816+0000","last_became_peered":"2024-05-09T03:09:33.193816+0000","last_unstale":"2024-05-09T03:09:36.914557+0000","last_undegraded":"2024-05-09T03:09:36.914557+0000","last_fullsized":"2024-05-09T03:09:36.914557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:51.846139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:35.223608+0000","last_change":"2024-05-09T03:09:34.207981+0000","last_active":"2024-05-09T03:09:35.223608+0000","last_peered":"2024-05-09T03:09:35.223608+0000","last_clean":"2024-05-09T03:09:35.223608+0000","last_became_active":"2024-05-09T03:09:34.207609+0000","last_became_peered":"2024-05-09T03:09:34.207609+0000","last_unstale":"2024-05-09T03:09:35.223608+0000","last_undegraded":"2024-05-09T03:09:35.223608+0000","last_fullsized":"2024-05-09T03:09:35.223608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:41.381783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914861+0000","last_change":"2024-05-09T03:09:33.194110+0000","last_active":"2024-05-09T03:09:36.914861+0000","last_peered":"2024-05-09T03:09:36.914861+0000","last_clean":"2024-05-09T03:09:36.914861+0000","last_became_active":"2024-05-09T03:09:33.193682+0000","last_became_peered":"2024-05-09T03:09:33.193682+0000","last_unstale":"2024-05-09T03:09:36.914861+0000","last_undegraded":"2024-05-09T03:09:36.914861+0000","last_fullsized":"2024-05-09T03:09:36.914861+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:48:04.365646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915459+0000","last_change":"2024-05-09T03:09:34.201024+0000","last_active":"2024-05-09T03:09:36.915459+0000","last_peered":"2024-05-09T03:09:36.915459+0000","last_clean":"2024-05-09T03:09:36.915459+0000","last_became_active":"2024-05-09T03:09:34.200708+0000","last_became_peered":"2024-05-09T03:09:34.200708+0000","last_unstale":"2024-05-09T03:09:36.915459+0000","last_undegraded":"2024-05-09T03:09:36.915459+0000","last_fullsized":"2024-05-09T03:09:36.915459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915071+0000","last_change":"2024-05-09T03:09:33.190754+0000","last_active":"2024-05-09T03:09:36.915071+0000","last_peered":"2024-05-09T03:09:36.915071+0000","last_clean":"2024-05-09T03:09:36.915071+0000","last_became_active":"2024-05-09T03:09:33.189371+0000","last_became_peered":"2024-05-09T03:09:33.189371+0000","last_unstale":"2024-05-09T03:09:36.915071+0000","last_undegraded":"2024-05-09T03:09:36.915071+0000","last_fullsized":"2024-05-09T03:09:36.915071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:19:03.842981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.914867+0000","last_change":"2024-05-09T03:09:34.193444+0000","last_active":"2024-05-09T03:09:36.914867+0000","last_peered":"2024-05-09T03:09:36.914867+0000","last_clean":"2024-05-09T03:09:36.914867+0000","last_became_active":"2024-05-09T03:09:34.193113+0000","last_became_peered":"2024-05-09T03:09:34.193113+0000","last_unstale":"2024-05-09T03:09:36.914867+0000","last_undegraded":"2024-05-09T03:09:36.914867+0000","last_fullsized":"2024-05-09T03:09:36.914867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:33.163887+0000","last_clean_scrub_stamp":"2024-05-09T03:09:33.163887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:21:58.855985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T03:09:36.915644+0000","last_change":"2024-05-09T03:09:33.191591+0000","last_active":"2024-05-09T03:09:36.915644+0000","last_peered":"2024-05-09T03:09:36.915644+0000","last_clean":"2024-05-09T03:09:36.915644+0000","last_became_active":"2024-05-09T03:09:33.191275+0000","last_became_peered":"2024-05-09T03:09:33.191275+0000","last_unstale":"2024-05-09T03:09:36.915644+0000","last_undegraded":"2024-05-09T03:09:36.915644+0000","last_fullsized":"2024-05-09T03:09:36.915644+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:32.158675+0000","last_clean_scrub_stamp":"2024-05-09T03:09:32.158675+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:18.783039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:09:43.424 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:09:43.424 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T03:09:43.424 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-09T03:09:43.424 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T03:09:43.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.572+0000 7f9122c49640 1 Processor -- start 2024-05-09T03:09:43.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.572+0000 7f9122c49640 1 -- start start 2024-05-09T03:09:43.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f9122c49640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f911c160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f9122c49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f911c0d49e0 con 0x7f911c0b7ef0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f9122c49640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f911c0d4cc0 con 0x7f911c0b4af0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f9122c49640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f911c0d5bb0 con 0x7f911c0d57a0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f91211e6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f911c0b7ef0 0x7f911c0b9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45698/0 (socket says 172.21.15.46:45698) 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f91211e6640 1 -- 172.21.15.46:0/1870890260 learned_addr learned my addr 172.21.15.46:0/1870890260 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f91209e5640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f911c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541073317 0 0) 0x7f911c0d49e0 con 0x7f911c0b7ef0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.573+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9104003520 con 0x7f911c0b7ef0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356622663 0 0) 0x7f911c0d4cc0 con 0x7f911c0b4af0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f911c0d49e0 con 0x7f911c0b4af0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2139196760 0 0) 0x7f9104003520 con 0x7f911c0b7ef0 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 msgr2=0x7f911c160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f911c160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 >> v1:172.21.15.88:6789/0 conn(0x7f911c0b4af0 legacy=0x7f911c0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911c0d4fa0 con 0x7f911c0b7ef0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f910c001670 con 0x7f911c0b7ef0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 >> v1:172.21.15.46:6789/0 conn(0x7f911c0b7ef0 legacy=0x7f911c0b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9104003cb0 0x7f9104004100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104004800 0x7f9104004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f910400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f911c0d5bb0 con 0x7f9104003cb0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9104005560 con 0x7f9104004800 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f910400bf00 con 0x7f911c0d57a0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91209e5640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9104003cb0 0x7f9104004100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f911bfff640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104004800 0x7f9104004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91211e6640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f910400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91209e5640 1 -- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 msgr2=0x7f910400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91209e5640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f910400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91209e5640 1 -- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104004800 msgr2=0x7f9104004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91209e5640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104004800 0x7f9104004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91209e5640 1 -- 172.21.15.46:0/1870890260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9104003520 con 0x7f9104003cb0 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.574+0000 7f91211e6640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c0d57a0 0x7f910400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.575+0000 7f91209e5640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9104003cb0 0x7f9104004100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9114009f90 tx=0x7f9104005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58071dc31688d33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.575+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114009c10 con 0x7f9104003cb0 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.575+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9114036070 con 0x7f9104003cb0 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.575+0000 7f911b7fe640 1 -- 172.21.15.46:0/1870890260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114031810 con 0x7f9104003cb0 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f9122c49640 1 -- 172.21.15.46:0/1870890260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9104003cb0 msgr2=0x7f9104004100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f9122c49640 1 --2- 172.21.15.46:0/1870890260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9104003cb0 0x7f9104004100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9114009f90 tx=0x7f9104005830 comp rx=0 tx=0).stop 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f91211e6640 1 -- 172.21.15.46:0/1870890260 reap_dead start 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f9122c49640 1 -- 172.21.15.46:0/1870890260 shutdown_connections 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f9122c49640 1 -- 172.21.15.46:0/1870890260 >> 172.21.15.46:0/1870890260 conn(0x7f911c0743e0 msgr2=0x7f911c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f9122c49640 1 -- 172.21.15.46:0/1870890260 shutdown_connections 2024-05-09T03:09:43.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.576+0000 7f9122c49640 1 -- 172.21.15.46:0/1870890260 wait complete. 2024-05-09T03:09:43.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.578+0000 7f9122c49640 1 Processor -- start 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.578+0000 7f9122c49640 1 -- start start 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f9122c49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104003cb0 0x7f911c163590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f9122c49640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c161040 0x7f911c156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f9122c49640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 0x7f911c15b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f9122c49640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f911c0d1920 con 0x7f911c15b340 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f9122c49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f911c0d15b0 con 0x7f9104003cb0 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f9122c49640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f911c159ea0 con 0x7f911c161040 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f91211e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 0x7f911c15b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f91211e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 0x7f911c15b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56172/0 (socket says 172.21.15.46:56172) 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f91211e6640 1 -- 172.21.15.46:0/927784930 learned_addr learned my addr 172.21.15.46:0/927784930 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f91209e5640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104003cb0 0x7f911c163590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.579+0000 7f911bfff640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c161040 0x7f911c156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91211e6640 1 -- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c161040 msgr2=0x7f911c156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91211e6640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c161040 0x7f911c156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91211e6640 1 -- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104003cb0 msgr2=0x7f911c163590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91211e6640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104003cb0 0x7f911c163590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91211e6640 1 -- 172.21.15.46:0/927784930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911c15bc90 con 0x7f911c15b340 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91211e6640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 0x7f911c15b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f911c0d2790 tx=0x7f910c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c019070 con 0x7f911c15b340 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910c010510 con 0x7f911c15b340 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910c0108e0 con 0x7f911c15b340 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911c1580c0 con 0x7f911c15b340 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.580+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911c158360 con 0x7f911c15b340 2024-05-09T03:09:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.581+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f911c0d0c50 con 0x7f911c15b340 2024-05-09T03:09:43.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.582+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f910c01c070 con 0x7f911c15b340 2024-05-09T03:09:43.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.582+0000 7f91197fa640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9108076300 0x7f9108078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.582+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f910c096b60 con 0x7f911c15b340 2024-05-09T03:09:43.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.582+0000 7f91209e5640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9108076300 0x7f9108078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.583+0000 7f91209e5640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9108076300 0x7f9108078810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9114035000 tx=0x7f911402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.587+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f911c0d0c50 con 0x7f911c15b340 2024-05-09T03:09:43.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.817+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f911c06a270 con 0x7f911c15b340 2024-05-09T03:09:43.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.817+0000 7f91197fa640 1 -- 172.21.15.46:0/927784930 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f911c15bc90 con 0x7f911c15b340 2024-05-09T03:09:43.818 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:43.818 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T03:09:43.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9108076300 msgr2=0x7f9108078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9108076300 0x7f9108078810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9114035000 tx=0x7f911402f000 comp rx=0 tx=0).stop 2024-05-09T03:09:43.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 msgr2=0x7f911c15b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 0x7f911c15b750 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f911c0d2790 tx=0x7f910c00e960 comp rx=0 tx=0).stop 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 shutdown_connections 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9108076300 0x7f9108078810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c15b340 0x7f911c15b750 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f911c161040 0x7f911c156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 --2- 172.21.15.46:0/927784930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9104003cb0 0x7f911c163590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.819+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 >> 172.21.15.46:0/927784930 conn(0x7f911c0743e0 msgr2=0x7f911c0d3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.820+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 shutdown_connections 2024-05-09T03:09:43.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.820+0000 7f9122c49640 1 -- 172.21.15.46:0/927784930 wait complete. 2024-05-09T03:09:43.831 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-09T03:09:43.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:09:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.945+0000 7f28fd627640 1 Processor -- start 2024-05-09T03:09:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.945+0000 7f28fd627640 1 -- start start 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.946+0000 7f28fd627640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80b92e0 0x7f28f80b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.946+0000 7f28fd627640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28f80d0480 0x7f28f80d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28fd627640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f28f80ce6a0 0x7f28f80ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28fd627640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f28f80d0dd0 con 0x7f28f80d0480 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28fd627640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f28f80d2b80 con 0x7f28f80ce6a0 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28fd627640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f28f80d2e60 con 0x7f28f80b92e0 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28f80d0480 0x7f28f80d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28f80d0480 0x7f28f80d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56182/0 (socket says 172.21.15.46:56182) 2024-05-09T03:09:43.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 -- 172.21.15.46:0/245914819 learned_addr learned my addr 172.21.15.46:0/245914819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f6ffd640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80b92e0 0x7f28f80b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f77fe640 1 --2- 172.21.15.46:0/245914819 >> v2:172.21.15.88:3300/0 conn(0x7f28f80ce6a0 0x7f28f80ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 -- 172.21.15.46:0/245914819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80b92e0 msgr2=0x7f28f80b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80b92e0 0x7f28f80b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 -- 172.21.15.46:0/245914819 >> v2:172.21.15.88:3300/0 conn(0x7f28f80ce6a0 msgr2=0x7f28f80ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 --2- 172.21.15.46:0/245914819 >> v2:172.21.15.88:3300/0 conn(0x7f28f80ce6a0 0x7f28f80ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f67fc640 1 -- 172.21.15.46:0/245914819 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f80d3140 con 0x7f28f80d0480 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f6ffd640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80b92e0 0x7f28f80b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.947+0000 7f28f77fe640 1 --2- 172.21.15.46:0/245914819 >> v2:172.21.15.88:3300/0 conn(0x7f28f80ce6a0 0x7f28f80ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.948+0000 7f28f67fc640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80d0480 0x7f28f80d0890 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f28f8163f40 tx=0x7f28ec02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fa9e28e3631943 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.948+0000 7f28f5ffb640 1 -- 172.21.15.46:0/245914819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec037070 con 0x7f28f80d0480 2024-05-09T03:09:43.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.948+0000 7f28f5ffb640 1 -- 172.21.15.46:0/245914819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28ec032440 con 0x7f28f80d0480 2024-05-09T03:09:43.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.948+0000 7f28f5ffb640 1 -- 172.21.15.46:0/245914819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec032810 con 0x7f28f80d0480 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 -- 172.21.15.46:0/245914819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80d0480 msgr2=0x7f28f80d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80d0480 0x7f28f80d0890 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f28f8163f40 tx=0x7f28ec02a140 comp rx=0 tx=0).stop 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 -- 172.21.15.46:0/245914819 shutdown_connections 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 --2- 172.21.15.46:0/245914819 >> v2:172.21.15.88:3300/0 conn(0x7f28f80ce6a0 0x7f28f80ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80d0480 0x7f28f80d0890 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 --2- 172.21.15.46:0/245914819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80b92e0 0x7f28f80b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 -- 172.21.15.46:0/245914819 >> 172.21.15.46:0/245914819 conn(0x7f28f80d3c00 msgr2=0x7f28f80d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:43.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 -- 172.21.15.46:0/245914819 shutdown_connections 2024-05-09T03:09:43.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.949+0000 7f28fd627640 1 -- 172.21.15.46:0/245914819 wait complete. 2024-05-09T03:09:43.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.951+0000 7f28fd627640 1 Processor -- start 2024-05-09T03:09:43.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.951+0000 7f28fd627640 1 -- start start 2024-05-09T03:09:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28fd627640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28f80b92e0 0x7f28f80b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28fd627640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 0x7f28f80b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28fd627640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80d0480 0x7f28f80b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28fd627640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f28f80cf170 con 0x7f28f80ce6a0 2024-05-09T03:09:43.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28fd627640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f28f80ceff0 con 0x7f28f80b92e0 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28fd627640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f28f80cf2f0 con 0x7f28f80d0480 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 0x7f28f80b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 0x7f28f80b75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56190/0 (socket says 172.21.15.46:56190) 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 -- 172.21.15.46:0/3241403975 learned_addr learned my addr 172.21.15.46:0/3241403975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f6ffd640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28f80b92e0 0x7f28f80b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f77fe640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80d0480 0x7f28f80b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 -- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80d0480 msgr2=0x7f28f80b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80d0480 0x7f28f80b7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 -- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28f80b92e0 msgr2=0x7f28f80b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28f80b92e0 0x7f28f80b67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:43.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.952+0000 7f28f67fc640 1 -- 172.21.15.46:0/3241403975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f81586e0 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.953+0000 7f28f67fc640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 0x7f28f80b75e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28ec02a180 tx=0x7f28ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.953+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec037040 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.953+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28ec042070 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.953+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f81589c0 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.953+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec030bb0 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.953+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f8158f80 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.954+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f806a270 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.955+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f28ec047460 con 0x7f28f80ce6a0 2024-05-09T03:09:43.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.955+0000 7f28df7fe640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f28e40762d0 0x7f28e40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.955+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f28ec0b8110 con 0x7f28f80ce6a0 2024-05-09T03:09:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.955+0000 7f28f6ffd640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f28e40762d0 0x7f28e40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:43.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.956+0000 7f28f6ffd640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f28e40762d0 0x7f28e40787e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f28f8071390 tx=0x7f28e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:43.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:43.960+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f28ec081420 con 0x7f28f80ce6a0 2024-05-09T03:09:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.191+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28f8158470 con 0x7f28f80ce6a0 2024-05-09T03:09:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.192+0000 7f28df7fe640 1 -- 172.21.15.46:0/3241403975 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7f28ec085720 con 0x7f28f80ce6a0 2024-05-09T03:09:44.193 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:44.193 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:09:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f28e40762d0 msgr2=0x7f28e40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f28e40762d0 0x7f28e40787e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f28f8071390 tx=0x7f28e8008040 comp rx=0 tx=0).stop 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 msgr2=0x7f28f80b75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 0x7f28f80b75e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f28ec02a180 tx=0x7f28ec0046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 shutdown_connections 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28f80d0480 0x7f28f80b7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f28e40762d0 0x7f28e40787e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f80ce6a0 0x7f28f80b75e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 --2- 172.21.15.46:0/3241403975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28f80b92e0 0x7f28f80b67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 >> 172.21.15.46:0/3241403975 conn(0x7f28f80d3c00 msgr2=0x7f28f80b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 shutdown_connections 2024-05-09T03:09:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.194+0000 7f28fd627640 1 -- 172.21.15.46:0/3241403975 wait complete. 2024-05-09T03:09:44.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:09:44.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.319+0000 7f172eae0640 1 Processor -- start 2024-05-09T03:09:44.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.319+0000 7f172eae0640 1 -- start start 2024-05-09T03:09:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172eae0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281604a0 0x7f17280d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172eae0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f17280b7ef0 0x7f17280b9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172eae0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17280d1920 con 0x7f17280d1170 2024-05-09T03:09:44.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172eae0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f17280d5130 con 0x7f17280b7ef0 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172eae0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f17280d1640 con 0x7f17281604a0 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f1727fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17280d1170 0x7f17280b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45714/0 (socket says 172.21.15.46:45714) 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f1727fff640 1 -- 172.21.15.46:0/2968475092 learned_addr learned my addr 172.21.15.46:0/2968475092 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172d07d640 1 --2- 172.21.15.46:0/2968475092 >> v2:172.21.15.88:3300/0 conn(0x7f17280b7ef0 0x7f17280b9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172c87c640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281604a0 0x7f17280d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f17277fe640 1 -- 172.21.15.46:0/2968475092 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 452402228 0 0) 0x7f1710001110 con 0x7f17280d1170 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f17277fe640 1 -- 172.21.15.46:0/2968475092 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1714003520 con 0x7f17280d1170 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172d07d640 1 -- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281604a0 msgr2=0x7f17280d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172d07d640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281604a0 0x7f17280d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172d07d640 1 -- 172.21.15.46:0/2968475092 >> v1:172.21.15.46:6789/0 conn(0x7f17280d1170 legacy=0x7f17280b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172d07d640 1 -- 172.21.15.46:0/2968475092 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17280d4ae0 con 0x7f17280b7ef0 2024-05-09T03:09:44.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.321+0000 7f172c87c640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281604a0 0x7f17280d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.322+0000 7f172d07d640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17280b7ef0 0x7f17280b9320 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17200097a0 tx=0x7f172002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df27bab156084133 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:44.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.322+0000 7f17277fe640 1 -- 172.21.15.46:0/2968475092 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720037070 con 0x7f17280b7ef0 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.322+0000 7f17277fe640 1 -- 172.21.15.46:0/2968475092 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1720032440 con 0x7f17280b7ef0 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f17277fe640 1 -- 172.21.15.46:0/2968475092 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17200327f0 con 0x7f17280b7ef0 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 -- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17280b7ef0 msgr2=0x7f17280b9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17280b7ef0 0x7f17280b9320 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f17200097a0 tx=0x7f172002a180 comp rx=0 tx=0).stop 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 -- 172.21.15.46:0/2968475092 shutdown_connections 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17280b7ef0 0x7f17280b9320 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 --2- 172.21.15.46:0/2968475092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281604a0 0x7f17280d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 -- 172.21.15.46:0/2968475092 >> 172.21.15.46:0/2968475092 conn(0x7f17280741d0 msgr2=0x7f17280745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 -- 172.21.15.46:0/2968475092 shutdown_connections 2024-05-09T03:09:44.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.323+0000 7f172eae0640 1 -- 172.21.15.46:0/2968475092 wait complete. 2024-05-09T03:09:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.325+0000 7f172eae0640 1 Processor -- start 2024-05-09T03:09:44.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.325+0000 7f172eae0640 1 -- start start 2024-05-09T03:09:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.326+0000 7f172eae0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 0x7f172813f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.326+0000 7f172eae0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17281604a0 0x7f17281f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.326+0000 7f172eae0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281424e0 0x7f1728141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.326+0000 7f172eae0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17280d55c0 con 0x7f17280d1170 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.326+0000 7f172eae0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f17281608b0 con 0x7f17281604a0 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.326+0000 7f172eae0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f17280cbb10 con 0x7f17281424e0 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 0x7f172813f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 0x7f172813f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56194/0 (socket says 172.21.15.46:56194) 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 -- 172.21.15.46:0/2162370776 learned_addr learned my addr 172.21.15.46:0/2162370776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172d07d640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281424e0 0x7f1728141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f1727fff640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17281604a0 0x7f17281f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 -- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281424e0 msgr2=0x7f1728141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281424e0 0x7f1728141440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 -- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17281604a0 msgr2=0x7f17281f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17281604a0 0x7f17281f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 -- 172.21.15.46:0/2162370776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17281434a0 con 0x7f17280d1170 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172d07d640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281424e0 0x7f1728141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:44.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f1727fff640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17281604a0 0x7f17281f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172c87c640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 0x7f172813f640 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1718009e60 tx=0x7f171800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1718018020 con 0x7f17280d1170 2024-05-09T03:09:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1718014480 con 0x7f17280d1170 2024-05-09T03:09:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1728143780 con 0x7f17280d1170 2024-05-09T03:09:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1718014850 con 0x7f17280d1170 2024-05-09T03:09:44.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.327+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1728143d40 con 0x7f17280d1170 2024-05-09T03:09:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.328+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f172806a270 con 0x7f17280d1170 2024-05-09T03:09:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.329+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1718005d10 con 0x7f17280d1170 2024-05-09T03:09:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.329+0000 7f17257fa640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17080762d0 0x7f17080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.330+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1718098380 con 0x7f17280d1170 2024-05-09T03:09:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.330+0000 7f1727fff640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17080762d0 0x7f17080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.330+0000 7f1727fff640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17080762d0 0x7f17080787e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1710007fd0 tx=0x7f1710009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.334+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1718022030 con 0x7f17280d1170 2024-05-09T03:09:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.563+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f172813f120 con 0x7f17280d1170 2024-05-09T03:09:44.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.564+0000 7f17257fa640 1 -- 172.21.15.46:0/2162370776 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7f1718010040 con 0x7f17280d1170 2024-05-09T03:09:44.564 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:44.564 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:09:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:09:44.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17080762d0 msgr2=0x7f17080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17080762d0 0x7f17080787e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1710007fd0 tx=0x7f1710009040 comp rx=0 tx=0).stop 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 msgr2=0x7f172813f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 0x7f172813f640 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1718009e60 tx=0x7f171800c490 comp rx=0 tx=0).stop 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 shutdown_connections 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17281424e0 0x7f1728141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17281604a0 0x7f17281f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17080762d0 0x7f17080787e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 --2- 172.21.15.46:0/2162370776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17280d1170 0x7f172813f640 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 >> 172.21.15.46:0/2162370776 conn(0x7f17280741d0 msgr2=0x7f1728162a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 shutdown_connections 2024-05-09T03:09:44.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.566+0000 7f172eae0640 1 -- 172.21.15.46:0/2162370776 wait complete. 2024-05-09T03:09:44.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:44.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.691+0000 7f0f4c4a7640 1 Processor -- start 2024-05-09T03:09:44.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.691+0000 7f0f4c4a7640 1 -- start start 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4c4a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 0x7f0f440b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4c4a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0f440b9320 0x7f0f440b9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4c4a7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f440d1920 con 0x7f0f441604a0 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4c4a7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0f440d5130 con 0x7f0f440b9320 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4c4a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0f440b8130 con 0x7f0f440b77e0 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 0x7f0f440b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4aa44640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0f440b9320 0x7f0f440b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 0x7f0f440b7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43652/0 (socket says 172.21.15.46:43652) 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 -- 172.21.15.46:0/433615584 learned_addr learned my addr 172.21.15.46:0/433615584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 -- 172.21.15.46:0/433615584 >> v2:172.21.15.88:3300/0 conn(0x7f0f440b9320 msgr2=0x7f0f440b9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:44.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 --2- 172.21.15.46:0/433615584 >> v2:172.21.15.88:3300/0 conn(0x7f0f440b9320 0x7f0f440b9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 -- 172.21.15.46:0/433615584 >> v1:172.21.15.88:6789/0 conn(0x7f0f441604a0 legacy=0x7f0f440d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f49a42640 1 -- 172.21.15.46:0/433615584 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f440d1640 con 0x7f0f440b77e0 2024-05-09T03:09:44.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.693+0000 7f0f4aa44640 1 --2- 172.21.15.46:0/433615584 >> v2:172.21.15.88:3300/0 conn(0x7f0f440b9320 0x7f0f440b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.694+0000 7f0f49a42640 1 --2- 172.21.15.46:0/433615584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 0x7f0f440b7bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009870 tx=0x7f0f3802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d955c767b7ec458 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.694+0000 7f0f49241640 1 -- 172.21.15.46:0/433615584 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38037070 con 0x7f0f440b77e0 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.694+0000 7f0f49241640 1 -- 172.21.15.46:0/433615584 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f38032440 con 0x7f0f440b77e0 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.694+0000 7f0f49241640 1 -- 172.21.15.46:0/433615584 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f38032810 con 0x7f0f440b77e0 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/433615584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 msgr2=0x7f0f440b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/433615584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 0x7f0f440b7bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009870 tx=0x7f0f3802a270 comp rx=0 tx=0).stop 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/433615584 shutdown_connections 2024-05-09T03:09:44.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/433615584 >> v2:172.21.15.88:3300/0 conn(0x7f0f440b9320 0x7f0f440b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/433615584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b77e0 0x7f0f440b7bf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/433615584 >> 172.21.15.46:0/433615584 conn(0x7f0f440741d0 msgr2=0x7f0f440745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/433615584 shutdown_connections 2024-05-09T03:09:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.695+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/433615584 wait complete. 2024-05-09T03:09:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.697+0000 7f0f4c4a7640 1 Processor -- start 2024-05-09T03:09:44.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.697+0000 7f0f4c4a7640 1 -- start start 2024-05-09T03:09:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4c4a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f440b77e0 0x7f0f441479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4c4a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b9320 0x7f0f44155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4c4a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 0x7f0f441626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4c4a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0f441608b0 con 0x7f0f441604a0 2024-05-09T03:09:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4c4a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0f440d4dc0 con 0x7f0f440b77e0 2024-05-09T03:09:44.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4c4a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0f440d1340 con 0x7f0f440b9320 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4aa44640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 0x7f0f441626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4aa44640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 0x7f0f441626f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56202/0 (socket says 172.21.15.46:56202) 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.698+0000 7f0f4aa44640 1 -- 172.21.15.46:0/3271638863 learned_addr learned my addr 172.21.15.46:0/3271638863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f49a42640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b9320 0x7f0f44155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4a243640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f440b77e0 0x7f0f441479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4aa44640 1 -- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b9320 msgr2=0x7f0f44155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4aa44640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b9320 0x7f0f44155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4aa44640 1 -- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f440b77e0 msgr2=0x7f0f441479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4aa44640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f440b77e0 0x7f0f441479c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4aa44640 1 -- 172.21.15.46:0/3271638863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f440c9da0 con 0x7f0f441604a0 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f49a42640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b9320 0x7f0f44155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4aa44640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 0x7f0f441626f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400efc0 tx=0x7f0f3400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34018020 con 0x7f0f441604a0 2024-05-09T03:09:44.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f34014480 con 0x7f0f441604a0 2024-05-09T03:09:44.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f440ca080 con 0x7f0f441604a0 2024-05-09T03:09:44.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f34014870 con 0x7f0f441604a0 2024-05-09T03:09:44.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.699+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f440ca640 con 0x7f0f441604a0 2024-05-09T03:09:44.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.700+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f4406a270 con 0x7f0f441604a0 2024-05-09T03:09:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.701+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0f34005d10 con 0x7f0f441604a0 2024-05-09T03:09:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.702+0000 7f0f42ffd640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f300762d0 0x7f0f300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.702+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0f340982d0 con 0x7f0f441604a0 2024-05-09T03:09:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.702+0000 7f0f4a243640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f300762d0 0x7f0f300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:44.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.703+0000 7f0f4a243640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f300762d0 0x7f0f300787e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0f440d6e40 tx=0x7f0f3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:44.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.707+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0f34022030 con 0x7f0f441604a0 2024-05-09T03:09:44.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.900+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0f440c9230 con 0x7f0f441604a0 2024-05-09T03:09:44.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.901+0000 7f0f42ffd640 1 -- 172.21.15.46:0/3271638863 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7f0f34010040 con 0x7f0f441604a0 2024-05-09T03:09:44.902 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:44.903 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f300762d0 msgr2=0x7f0f300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f300762d0 0x7f0f300787e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0f440d6e40 tx=0x7f0f3c009040 comp rx=0 tx=0).stop 2024-05-09T03:09:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 msgr2=0x7f0f441626f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 0x7f0f441626f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400efc0 tx=0x7f0f3400c490 comp rx=0 tx=0).stop 2024-05-09T03:09:44.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 shutdown_connections 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f300762d0 0x7f0f300787e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f441604a0 0x7f0f441626f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f440b9320 0x7f0f44155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 --2- 172.21.15.46:0/3271638863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f440b77e0 0x7f0f441479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 >> 172.21.15.46:0/3271638863 conn(0x7f0f440741d0 msgr2=0x7f0f4415d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 shutdown_connections 2024-05-09T03:09:44.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:44.903+0000 7f0f4c4a7640 1 -- 172.21.15.46:0/3271638863 wait complete. 2024-05-09T03:09:44.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:09:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.030+0000 7fa8ce029640 1 Processor -- start 2024-05-09T03:09:45.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.030+0000 7fa8ce029640 1 -- start start 2024-05-09T03:09:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8ce029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c8164c70 0x7fa8c81615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8ce029640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8ce029640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c8160b20 con 0x7fa8c80d2480 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8ce029640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c8161150 con 0x7fa8c815dfe0 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8ce029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c80d15f0 con 0x7fa8c8164c70 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8c7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8c80d2480 0x7fa8c80d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45718/0 (socket says 172.21.15.46:45718) 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8c7fff640 1 -- 172.21.15.46:0/1629466161 learned_addr learned my addr 172.21.15.46:0/1629466161 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8c77fe640 1 --2- 172.21.15.46:0/1629466161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c8164c70 0x7fa8c81615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.031+0000 7fa8c6ffd640 1 --2- 172.21.15.46:0/1629466161 >> v2:172.21.15.88:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c67fc640 1 -- 172.21.15.46:0/1629466161 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 41381231 0 0) 0x7fa8c8160b20 con 0x7fa8c80d2480 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c67fc640 1 -- 172.21.15.46:0/1629466161 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8b0003520 con 0x7fa8c80d2480 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c77fe640 1 -- 172.21.15.46:0/1629466161 >> v2:172.21.15.88:3300/0 conn(0x7fa8c815dfe0 msgr2=0x7fa8c80d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c77fe640 1 --2- 172.21.15.46:0/1629466161 >> v2:172.21.15.88:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c77fe640 1 -- 172.21.15.46:0/1629466161 >> v1:172.21.15.46:6789/0 conn(0x7fa8c80d2480 legacy=0x7fa8c80d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c77fe640 1 -- 172.21.15.46:0/1629466161 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c80d18d0 con 0x7fa8c8164c70 2024-05-09T03:09:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.032+0000 7fa8c6ffd640 1 --2- 172.21.15.46:0/1629466161 >> v2:172.21.15.88:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8c77fe640 1 --2- 172.21.15.46:0/1629466161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c8164c70 0x7fa8c81615e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80097a0 tx=0x7fa8b802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dbf164621a44d1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8c67fc640 1 -- 172.21.15.46:0/1629466161 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8037070 con 0x7fa8c8164c70 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8c67fc640 1 -- 172.21.15.46:0/1629466161 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b802ae50 con 0x7fa8c8164c70 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8c67fc640 1 -- 172.21.15.46:0/1629466161 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8032660 con 0x7fa8c8164c70 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8ce029640 1 -- 172.21.15.46:0/1629466161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c8164c70 msgr2=0x7fa8c81615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8ce029640 1 --2- 172.21.15.46:0/1629466161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c8164c70 0x7fa8c81615e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8b80097a0 tx=0x7fa8b802a100 comp rx=0 tx=0).stop 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8ce029640 1 -- 172.21.15.46:0/1629466161 shutdown_connections 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8ce029640 1 --2- 172.21.15.46:0/1629466161 >> v2:172.21.15.88:3300/0 conn(0x7fa8c815dfe0 0x7fa8c80d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.033+0000 7fa8ce029640 1 --2- 172.21.15.46:0/1629466161 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c8164c70 0x7fa8c81615e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.034+0000 7fa8ce029640 1 -- 172.21.15.46:0/1629466161 >> 172.21.15.46:0/1629466161 conn(0x7fa8c8073db0 msgr2=0x7fa8c80741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.034+0000 7fa8ce029640 1 -- 172.21.15.46:0/1629466161 shutdown_connections 2024-05-09T03:09:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.034+0000 7fa8ce029640 1 -- 172.21.15.46:0/1629466161 wait complete. 2024-05-09T03:09:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.035+0000 7fa8ce029640 1 Processor -- start 2024-05-09T03:09:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.035+0000 7fa8ce029640 1 -- start start 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8ce029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c80d2480 0x7fa8c80cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8ce029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8c815dfe0 0x7fa8c814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8ce029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 0x7fa8c81483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8ce029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c815e3f0 con 0x7fa8c8164c70 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8ce029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c80d1300 con 0x7fa8c815dfe0 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8ce029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c80d45c0 con 0x7fa8c80d2480 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 0x7fa8c81483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 0x7fa8c81483a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56218/0 (socket says 172.21.15.46:56218) 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 -- 172.21.15.46:0/117417867 learned_addr learned my addr 172.21.15.46:0/117417867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c77fe640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c80d2480 0x7fa8c80cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c6ffd640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8c815dfe0 0x7fa8c814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 -- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c80d2480 msgr2=0x7fa8c80cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c80d2480 0x7fa8c80cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 -- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8c815dfe0 msgr2=0x7fa8c814dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8c815dfe0 0x7fa8c814dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 -- 172.21.15.46:0/117417867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c80cad50 con 0x7fa8c8164c70 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.037+0000 7fa8c7fff640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 0x7fa8c81483a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000cad0 tx=0x7fa8c000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.038+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c000ac30 con 0x7fa8c8164c70 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.038+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8c000add0 con 0x7fa8c8164c70 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.038+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c80cb030 con 0x7fa8c8164c70 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.038+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0012600 con 0x7fa8c8164c70 2024-05-09T03:09:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.038+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c80cb5f0 con 0x7fa8c8164c70 2024-05-09T03:09:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.039+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c808cfd0 con 0x7fa8c8164c70 2024-05-09T03:09:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.039+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa8c00127a0 con 0x7fa8c8164c70 2024-05-09T03:09:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.040+0000 7fa8affff640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8b40762d0 0x7fa8b40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.040+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa8c0097b50 con 0x7fa8c8164c70 2024-05-09T03:09:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.040+0000 7fa8c77fe640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8b40762d0 0x7fa8b40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.041+0000 7fa8c77fe640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8b40762d0 0x7fa8b40787e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8034000 tx=0x7fa8b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.045+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa8c808cfd0 con 0x7fa8c8164c70 2024-05-09T03:09:45.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.274+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8c80caae0 con 0x7fa8c8164c70 2024-05-09T03:09:45.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.274+0000 7fa8affff640 1 -- 172.21.15.46:0/117417867 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7fa8c0060dd0 con 0x7fa8c8164c70 2024-05-09T03:09:45.275 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:45.275 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:09:45.276 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:09:45.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8b40762d0 msgr2=0x7fa8b40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8b40762d0 0x7fa8b40787e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8034000 tx=0x7fa8b802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 msgr2=0x7fa8c81483a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 0x7fa8c81483a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa8c000cad0 tx=0x7fa8c000cf90 comp rx=0 tx=0).stop 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 shutdown_connections 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8b40762d0 0x7fa8b40787e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8c8164c70 0x7fa8c81483a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8c815dfe0 0x7fa8c814dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 --2- 172.21.15.46:0/117417867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8c80d2480 0x7fa8c80cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 >> 172.21.15.46:0/117417867 conn(0x7fa8c8073db0 msgr2=0x7fa8c815e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 shutdown_connections 2024-05-09T03:09:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.277+0000 7fa8ce029640 1 -- 172.21.15.46:0/117417867 wait complete. 2024-05-09T03:09:45.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:09:33.568483+0000', 'modified': '2024-05-09T03:09:36.576285+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4287': {'gid': 4287, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/59422413', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 59422413}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 59422413}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4287, 'qdb_cluster': [4287]} 2024-05-09T03:09:45.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:09:45.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:09:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.812+0000 7fd0c7784640 1 Processor -- start 2024-05-09T03:09:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.812+0000 7fd0c7784640 1 -- start start 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.813+0000 7fd0c7784640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0c00b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.813+0000 7fd0c7784640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0c00d3400 con 0x7fd0c00d2e90 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.813+0000 7fd0c7784640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0c00d4a60 con 0x7fd0c00b7e70 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.813+0000 7fd0c7784640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd0c00d4d40 con 0x7fd0c01605b0 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c5520640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd0c00d2e90 0x7fd0c00b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46192/0 (socket says 172.21.15.46:46192) 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c5520640 1 -- 172.21.15.46:0/3523672452 learned_addr learned my addr 172.21.15.46:0/3523672452 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0c00b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557562568 0 0) 0x7fd0c00d3400 con 0x7fd0c00d2e90 2024-05-09T03:09:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0ac003520 con 0x7fd0c00d2e90 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483055135 0 0) 0x7fd0c00d4a60 con 0x7fd0c00b7e70 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0c00d3400 con 0x7fd0c00b7e70 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1877046255 0 0) 0x7fd0ac003520 con 0x7fd0c00d2e90 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 msgr2=0x7fd0c00b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0c00b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 >> v1:172.21.15.88:6789/0 conn(0x7fd0c00b7e70 legacy=0x7fd0c00b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c00d5020 con 0x7fd0c00d2e90 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fd0bc0016f0 con 0x7fd0c00d2e90 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0c00b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 >> v1:172.21.15.46:6789/0 conn(0x7fd0c00d2e90 legacy=0x7fd0c00b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0ac003c60 0x7fd0ac0040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c00d4d40 con 0x7fd0ac004800 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac0054e0 con 0x7fd0ac003c60 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac00bf00 con 0x7fd0c01605b0 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c5520640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0ac003c60 0x7fd0ac0040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.814+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.815+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 msgr2=0x7fd0ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.815+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c01605b0 0x7fd0ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.815+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0ac003c60 msgr2=0x7fd0ac0040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.815+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0ac003c60 0x7fd0ac0040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.815+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3523672452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac003520 con 0x7fd0ac004800 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.815+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0ac004c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0001900 tx=0x7fd0ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8355739ab0ddcc6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0001520 con 0x7fd0ac004800 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b002acd0 con 0x7fd0ac004800 2024-05-09T03:09:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0b7fff640 1 -- 172.21.15.46:0/3523672452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0005310 con 0x7fd0ac004800 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c7784640 1 -- 172.21.15.46:0/3523672452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 msgr2=0x7fd0ac004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c7784640 1 --2- 172.21.15.46:0/3523672452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0ac004c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0001900 tx=0x7fd0ac005830 comp rx=0 tx=0).stop 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c5d21640 1 -- 172.21.15.46:0/3523672452 reap_dead start 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c7784640 1 -- 172.21.15.46:0/3523672452 shutdown_connections 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c7784640 1 -- 172.21.15.46:0/3523672452 >> 172.21.15.46:0/3523672452 conn(0x7fd0c008d390 msgr2=0x7fd0c008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c7784640 1 -- 172.21.15.46:0/3523672452 shutdown_connections 2024-05-09T03:09:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.816+0000 7fd0c7784640 1 -- 172.21.15.46:0/3523672452 wait complete. 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.818+0000 7fd0c7784640 1 Processor -- start 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.818+0000 7fd0c7784640 1 -- start start 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.819+0000 7fd0c7784640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0c00c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.819+0000 7fd0c7784640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c014e6d0 0x7fd0c00cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.819+0000 7fd0c7784640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c00cb8a0 0x7fd0c00ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.819+0000 7fd0c7784640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c00b4d80 con 0x7fd0ac004800 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.819+0000 7fd0c7784640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c00b4c00 con 0x7fd0c00cb8a0 2024-05-09T03:09:45.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.819+0000 7fd0c7784640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd0c00b4fc0 con 0x7fd0c014e6d0 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0c00c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0c00c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57042/0 (socket says 172.21.15.46:57042) 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 -- 172.21.15.46:0/292931359 learned_addr learned my addr 172.21.15.46:0/292931359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c014e6d0 0x7fd0c00cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c00cb8a0 0x7fd0c00ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 -- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c014e6d0 msgr2=0x7fd0c00cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c014e6d0 0x7fd0c00cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 -- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c00cb8a0 msgr2=0x7fd0c00ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c00cb8a0 0x7fd0c00ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 -- 172.21.15.46:0/292931359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c00cbcb0 con 0x7fd0ac004800 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c5520640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0c00c96d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc004ea0 tx=0x7fd0bc004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0b5ffb640 1 -- 172.21.15.46:0/292931359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00f680 con 0x7fd0ac004800 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0b5ffb640 1 -- 172.21.15.46:0/292931359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0bc018070 con 0x7fd0ac004800 2024-05-09T03:09:45.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c0162fd0 con 0x7fd0ac004800 2024-05-09T03:09:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.820+0000 7fd0b5ffb640 1 -- 172.21.15.46:0/292931359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00fa20 con 0x7fd0ac004800 2024-05-09T03:09:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.821+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c00cdcd0 con 0x7fd0ac004800 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.822+0000 7fd0b5ffb640 1 -- 172.21.15.46:0/292931359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd0bc00fbc0 con 0x7fd0ac004800 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.823+0000 7fd0b5ffb640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0a4076300 0x7fd0a4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.823+0000 7fd0b5ffb640 1 -- 172.21.15.46:0/292931359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd0bc0970e0 con 0x7fd0ac004800 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.823+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0a4076300 0x7fd0a4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.823+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0a4076300 0x7fd0a4078810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0031000 tx=0x7fd0b002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.824+0000 7fd0ce637740 1 Processor -- start 2024-05-09T03:09:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.824+0000 7fd0ce637740 1 -- start start 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3ef7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0ce637740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563dd3e144e0 con 0x563dd3aeb590 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563dd3dafff0 con 0x563dd3ae33b0 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563dd3e104a0 con 0x563dd3b0c270 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3ef7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3ef7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57054/0 (socket says 172.21.15.46:57054) 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 -- 172.21.15.46:0/1133794456 learned_addr learned my addr 172.21.15.46:0/1133794456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5520640 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 -- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 msgr2=0x563dd3edbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 -- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 msgr2=0x563dd3ef6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.825+0000 7fd0c5d21640 1 -- 172.21.15.46:0/1133794456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd3d96720 con 0x563dd3aeb590 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.826+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3ef7120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00d02e0 tx=0x7fd0b8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d382b5e586e06e49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.826+0000 7fd0aaffd640 1 -- 172.21.15.46:0/1133794456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b801a030 con 0x563dd3aeb590 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.826+0000 7fd0aaffd640 1 -- 172.21.15.46:0/1133794456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b8016640 con 0x563dd3aeb590 2024-05-09T03:09:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.826+0000 7fd0aaffd640 1 -- 172.21.15.46:0/1133794456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b80169f0 con 0x563dd3aeb590 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.826+0000 7fd0ce637740 1 -- 172.21.15.46:0/1133794456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 msgr2=0x563dd3ef7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.826+0000 7fd0ce637740 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3ef7120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00d02e0 tx=0x7fd0b8009750 comp rx=0 tx=0).stop 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 -- 172.21.15.46:0/1133794456 shutdown_connections 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3ef7120 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 --2- 172.21.15.46:0/1133794456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 -- 172.21.15.46:0/1133794456 >> 172.21.15.46:0/1133794456 conn(0x563dd3b003c0 msgr2=0x563dd3eb1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 -- 172.21.15.46:0/1133794456 shutdown_connections 2024-05-09T03:09:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.827+0000 7fd0ce637740 1 -- 172.21.15.46:0/1133794456 wait complete. 2024-05-09T03:09:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.828+0000 7fd0ce637740 1 -- start start 2024-05-09T03:09:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.828+0000 7fd0ce637740 1 Processor -- start 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3ae33b0 0x563dd3f09710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3f09c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3b0c270 0x563dd3f0a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563dd3d9b760 con 0x563dd3aeb590 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563dd3eddef0 con 0x563dd3b0c270 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563dd3ef3b70 con 0x563dd3ae33b0 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0ce637740 10 client.0 _ll_register_callbacks cb 0x563dd3dbe4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0c4d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3f09c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0c4d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3f09c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57056/0 (socket says 172.21.15.46:57056) 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3592244518 learned_addr learned my addr 172.21.15.46:0/3592244518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0c5520640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3ae33b0 0x563dd3f09710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.829+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3b0c270 0x563dd3f0a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3ae33b0 msgr2=0x563dd3f09710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3ae33b0 0x563dd3f09710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3b0c270 msgr2=0x563dd3f0a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3b0c270 0x563dd3f0a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/3592244518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd3ddcbd0 con 0x563dd3aeb590 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3b0c270 0x563dd3f0a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3f09c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00019f0 tx=0x7fd0b002c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc832b8b9d728e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd09ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd09ffff640 1 -- 172.21.15.46:0/3592244518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b002ac80 con 0x563dd3aeb590 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd09ffff640 1 -- 172.21.15.46:0/3592244518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd0b0003b70 con 0x563dd3aeb590 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0b8016640 con 0x563dd3aeb590 2024-05-09T03:09:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd09ffff640 1 -- 172.21.15.46:0/3592244518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fd0b0003db0 con 0x563dd3aeb590 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 10 client.4569 fetch_fsmap learned FSMap version 7 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x563dd3dc2280 con 0x563dd3aeb590 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd09ffff640 1 -- 172.21.15.46:0/3592244518 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b8016640 con 0x563dd3aeb590 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd09ffff640 1 -- 172.21.15.46:0/3592244518 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7fd0b0007020 con 0x563dd3aeb590 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 10 client.4569 fetch_fsmap finished waiting for FSMap version 7 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 10 client.4569 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 20 client.4569 populate_metadata read hostname 'smithi046' 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3acae40 0x563dd3edd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 4 client.4569 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.830+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 --> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563dd3e220f0 con 0x563dd3acae40 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.831+0000 7fd0c5520640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3acae40 0x563dd3edd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.831+0000 7fd09ffff640 10 client.4569 ms_handle_connect on v2:172.21.15.46:6833/59422413 2024-05-09T03:09:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.831+0000 7fd0c5520640 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3acae40 0x563dd3edd0e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e7388200c0f72ca server_cookie=f5478ae602c5a637 in_seq=0 out_seq=0 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd09ffff640 1 -- 172.21.15.46:0/3592244518 <== mds.0 v2:172.21.15.46:6833/59422413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563dd3e220f0 con 0x563dd3acae40 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd09ffff640 10 client.4569 handle_command_reply: tid=0 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 client.4569 shutdown 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 msgr2=0x563dd3f09c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3f09c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00019f0 tx=0x7fd0b002c070 comp rx=0 tx=0).stop 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 shutdown_connections 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3acae40 0x563dd3edd0e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3b0c270 0x563dd3f0a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3f09c50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd09ffff640 0 client.4569 ms_handle_reset on v2:172.21.15.46:6833/59422413 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 --2- 172.21.15.46:0/3592244518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3ae33b0 0x563dd3f09710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 >> 172.21.15.46:0/3592244518 conn(0x563dd3b003c0 msgr2=0x563dd3edbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 shutdown_connections 2024-05-09T03:09:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 1 -- 172.21.15.46:0/3592244518 wait complete. 2024-05-09T03:09:45.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.832+0000 7fd0ce637740 20 client.4569 trim_cache size 0 max 16384 2024-05-09T03:09:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.851+0000 7fd0ce637740 1 Processor -- start 2024-05-09T03:09:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.851+0000 7fd0ce637740 1 -- start start 2024-05-09T03:09:45.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.852+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.852+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3edc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.852+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.852+0000 7fd0ce637740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0202000 con 0x563dd3aeb590 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.852+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0162720 con 0x563dd3ae33b0 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.852+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd0c0201a10 con 0x563dd3b0c270 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3edc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3edc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57066/0 (socket says 172.21.15.46:57066) 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2919435250 learned_addr learned my addr 172.21.15.46:0/2919435250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2919435250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 msgr2=0x563dd3edd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2919435250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 msgr2=0x563dd3ef8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2919435250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563dd3b77850 con 0x563dd3aeb590 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3edc0a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd0b002a6a0 tx=0x7fd0b003a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3624a7c66a072f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0aa7fc640 1 -- 172.21.15.46:0/2919435250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0003b70 con 0x563dd3aeb590 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0aa7fc640 1 -- 172.21.15.46:0/2919435250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b0007570 con 0x563dd3aeb590 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.853+0000 7fd0aa7fc640 1 -- 172.21.15.46:0/2919435250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b002ae00 con 0x563dd3aeb590 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 -- 172.21.15.46:0/2919435250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 msgr2=0x563dd3edc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3edc0a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd0b002a6a0 tx=0x7fd0b003a000 comp rx=0 tx=0).stop 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 -- 172.21.15.46:0/2919435250 shutdown_connections 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3edd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3edc0a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2919435250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3ef8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 -- 172.21.15.46:0/2919435250 >> 172.21.15.46:0/2919435250 conn(0x563dd3b003c0 msgr2=0x563dd3ede080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 -- 172.21.15.46:0/2919435250 shutdown_connections 2024-05-09T03:09:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.854+0000 7fd0ce637740 1 -- 172.21.15.46:0/2919435250 wait complete. 2024-05-09T03:09:45.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.855+0000 7fd0ce637740 1 -- start start 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.855+0000 7fd0ce637740 1 Processor -- start 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3eb1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3eb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3ef8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563dd3ef17c0 con 0x563dd3aeb590 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563dd3edfa40 con 0x563dd3ae33b0 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563dd3ef1de0 con 0x563dd3b0c270 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0ce637740 10 client.0 _ll_register_callbacks cb 0x7fd0ac003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0c4d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3eb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0c4d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3eb1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57070/0 (socket says 172.21.15.46:57070) 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.856+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2416590987 learned_addr learned my addr 172.21.15.46:0/2416590987 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c5520640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3eb1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c5d21640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3ef8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 msgr2=0x563dd3ef8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3ef8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 msgr2=0x563dd3eb1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3eb1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c4d1f640 1 -- 172.21.15.46:0/2416590987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563dd3bee6b0 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0c4d1f640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3eb1b80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00019f0 tx=0x7fd0b0001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796b1a0667489e2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd09ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd09ffff640 1 -- 172.21.15.46:0/2416590987 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b00069e0 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd09ffff640 1 -- 172.21.15.46:0/2416590987 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd0b0061ad0 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0b0007570 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd09ffff640 1 -- 172.21.15.46:0/2416590987 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fd0b0061cd0 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 10 client.4575 fetch_fsmap learned FSMap version 7 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x563dd3dd0630 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd09ffff640 1 -- 172.21.15.46:0/2416590987 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b0007570 con 0x563dd3aeb590 2024-05-09T03:09:45.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd09ffff640 1 -- 172.21.15.46:0/2416590987 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7fd0b0006b80 con 0x563dd3aeb590 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 10 client.4575 fetch_fsmap finished waiting for FSMap version 7 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 10 client.4575 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 20 client.4575 populate_metadata read hostname 'smithi046' 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3f77630 0x563dd3f77a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 4 client.4575 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.857+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 --> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] -- command(tid 0: {"prefix": "status"}) -- 0x563dd3e1bf60 con 0x563dd3f77630 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0c5520640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3f77630 0x563dd3f77a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd09ffff640 10 client.4575 ms_handle_connect on v2:172.21.15.46:6833/59422413 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0c5520640 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3f77630 0x563dd3f77a40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d38437a54cb52fcb server_cookie=52a91b4f6ddceede in_seq=0 out_seq=0 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd09ffff640 1 -- 172.21.15.46:0/2416590987 <== mds.0 v2:172.21.15.46:6833/59422413 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x563dd3e1bf60 con 0x563dd3f77630 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd09ffff640 10 client.4575 handle_command_reply: tid=0 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 client.4575 shutdown 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 msgr2=0x563dd3eb1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3eb1b80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd0b00019f0 tx=0x7fd0b0001360 comp rx=0 tx=0).stop 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 shutdown_connections 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:6833/59422413,v1:172.21.15.46:6835/59422413] conn(0x563dd3f77630 0x563dd3f77a40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563dd3b0c270 0x563dd3ef8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563dd3aeb590 0x563dd3eb1b80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 --2- 172.21.15.46:0/2416590987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563dd3ae33b0 0x563dd3eb1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd09ffff640 0 client.4575 ms_handle_reset on v2:172.21.15.46:6833/59422413 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.858+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 >> 172.21.15.46:0/2416590987 conn(0x563dd3b003c0 msgr2=0x563dd3edec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 shutdown_connections 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0ce637740 1 -- 172.21.15.46:0/2416590987 wait complete. 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0ce637740 20 client.4575 trim_cache size 0 max 16384 2024-05-09T03:09:45.864 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4287, 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 11.687852317000001, 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 7, 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 20, 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 20, 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 17.717055447 2024-05-09T03:09:45.865 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:09:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0a4076300 msgr2=0x7fd0a4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0a4076300 0x7fd0a4078810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0031000 tx=0x7fd0b002e000 comp rx=0 tx=0).stop 2024-05-09T03:09:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 msgr2=0x7fd0c00c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0c00c96d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc004ea0 tx=0x7fd0bc004f90 comp rx=0 tx=0).stop 2024-05-09T03:09:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 shutdown_connections 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0c00cb8a0 0x7fd0c00ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0c014e6d0 0x7fd0c00cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0a4076300 0x7fd0a4078810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 --2- 172.21.15.46:0/292931359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0ac004800 0x7fd0c00c96d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 >> 172.21.15.46:0/292931359 conn(0x7fd0c008d390 msgr2=0x7fd0c00d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 shutdown_connections 2024-05-09T03:09:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:45.859+0000 7fd0c7784640 1 -- 172.21.15.46:0/292931359 wait complete. 2024-05-09T03:09:45.871 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T03:09:45.880 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T03:09:45.880 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T03:09:45.881 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T03:09:45.881 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-09T03:09:45.903 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-09T03:09:45.904 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-05-09T03:09:45.904 INFO:tasks.ceph_fuse:client.0 config is {} 2024-05-09T03:09:45.904 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T03:09:45.904 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:45.904 INFO:tasks.ceph_fuse:client.2 config is {} 2024-05-09T03:09:45.904 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T03:09:45.904 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:45.904 INFO:tasks.ceph_fuse:client.1 config is {} 2024-05-09T03:09:45.904 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T03:09:45.904 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:45.905 INFO:tasks.ceph_fuse:client.3 config is {} 2024-05-09T03:09:45.905 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T03:09:45.905 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:45.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:45.905 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-05-09T03:09:45.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:45.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2024-05-09T03:09:45.997 INFO:teuthology.orchestra.run.smithi088.stderr:Cannot find device "ceph-brx" 2024-05-09T03:09:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:09:45.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:45.999 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:09:46.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:46.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-05-09T03:09:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:Cannot find device "ceph-brx" 2024-05-09T03:09:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:09:46.086 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-09T03:09:46.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:09:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.240+0000 7f6ed9f27640 1 Processor -- start 2024-05-09T03:09:46.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.240+0000 7f6ed9f27640 1 -- start start 2024-05-09T03:09:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.241+0000 7f6ed9f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 0x7f6ed40b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.241+0000 7f6ed9f27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ed4070d80 0x7f6ed40aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.241+0000 7f6ed9f27640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed415d980 con 0x7f6ed40aeb80 2024-05-09T03:09:46.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.241+0000 7f6ed9f27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6ed415dc60 con 0x7f6ed4070d80 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.241+0000 7f6ed9f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed415df40 con 0x7f6ed40b1210 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.241+0000 7f6ed2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6ed40aeb80 0x7f6ed4070670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46208/0 (socket says 172.21.15.46:46208) 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed2ffd640 1 -- 172.21.15.46:0/4286470128 learned_addr learned my addr 172.21.15.46:0/4286470128 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed37fe640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 0x7f6ed40b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed3fff640 1 --2- 172.21.15.46:0/4286470128 >> v2:172.21.15.88:3300/0 conn(0x7f6ed4070d80 0x7f6ed40aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997090703 0 0) 0x7f6ed415d980 con 0x7f6ed40aeb80 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ebc005730 con 0x7f6ed40aeb80 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 515166011 0 0) 0x7f6ebc005730 con 0x7f6ed40aeb80 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 msgr2=0x7f6ed40b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 0x7f6ed40b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 >> v2:172.21.15.88:3300/0 conn(0x7f6ed4070d80 msgr2=0x7f6ed40aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 --2- 172.21.15.46:0/4286470128 >> v2:172.21.15.88:3300/0 conn(0x7f6ed4070d80 0x7f6ed40aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed415e220 con 0x7f6ed40aeb80 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f6ec80016f0 con 0x7f6ed40aeb80 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 >> v1:172.21.15.46:6789/0 conn(0x7f6ed40aeb80 legacy=0x7f6ed4070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 0x7f6ebc005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ebc006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ebc006f50 0x7f6ebc007360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed415dc60 con 0x7f6ebc0061e0 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed415df40 con 0x7f6ebc006f50 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6ebc00dd60 con 0x7f6ed40b1210 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ebc006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed37fe640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 0x7f6ebc005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.242+0000 7f6ed3fff640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ebc006f50 0x7f6ebc007360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed2ffd640 1 -- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 msgr2=0x7f6ebc005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed40b1210 0x7f6ebc005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed2ffd640 1 -- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ebc006f50 msgr2=0x7f6ebc007360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ebc006f50 0x7f6ebc007360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed2ffd640 1 -- 172.21.15.46:0/4286470128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc005730 con 0x7f6ebc0061e0 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ebc006a10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4072710 tx=0x7f6ebc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3815ed857034f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8007d50 con 0x7f6ebc0061e0 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec8005ad0 con 0x7f6ebc0061e0 2024-05-09T03:09:46.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.243+0000 7f6ed27fc640 1 -- 172.21.15.46:0/4286470128 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8007070 con 0x7f6ebc0061e0 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed9f27640 1 -- 172.21.15.46:0/4286470128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 msgr2=0x7f6ebc006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/4286470128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ebc006a10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4072710 tx=0x7f6ebc007a40 comp rx=0 tx=0).stop 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed3fff640 1 -- 172.21.15.46:0/4286470128 reap_dead start 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed9f27640 1 -- 172.21.15.46:0/4286470128 shutdown_connections 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed9f27640 1 -- 172.21.15.46:0/4286470128 >> 172.21.15.46:0/4286470128 conn(0x7f6ed40b3910 msgr2=0x7f6ed40b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed9f27640 1 -- 172.21.15.46:0/4286470128 shutdown_connections 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.244+0000 7f6ed9f27640 1 -- 172.21.15.46:0/4286470128 wait complete. 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.246+0000 7f6ed9f27640 1 Processor -- start 2024-05-09T03:09:46.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.246+0000 7f6ed9f27640 1 -- start start 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.247+0000 7f6ed9f27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ed413d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.247+0000 7f6ed9f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed415e570 0x7f6ed41f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.247+0000 7f6ed9f27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ed413c950 0x7f6ed4142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.247+0000 7f6ed9f27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed40b5170 con 0x7f6ebc0061e0 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.247+0000 7f6ed9f27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed40b4ff0 con 0x7f6ed413c950 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.247+0000 7f6ed9f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed40b52f0 con 0x7f6ed415e570 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ed413d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ed413d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57092/0 (socket says 172.21.15.46:57092) 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 -- 172.21.15.46:0/818083854 learned_addr learned my addr 172.21.15.46:0/818083854 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed3fff640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ed413c950 0x7f6ed4142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed415e570 0x7f6ed41f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 -- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed415e570 msgr2=0x7f6ed41f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed415e570 0x7f6ed41f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 -- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ed413c950 msgr2=0x7f6ed4142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ed413c950 0x7f6ed4142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 -- 172.21.15.46:0/818083854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed40b19b0 con 0x7f6ebc0061e0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed37fe640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ed413d780 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4011940 tx=0x7f6ec4011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec400faf0 con 0x7f6ebc0061e0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec400fc90 con 0x7f6ebc0061e0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed40b1c90 con 0x7f6ebc0061e0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4014440 con 0x7f6ebc0061e0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.248+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed40b2250 con 0x7f6ebc0061e0 2024-05-09T03:09:46.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.249+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed4069320 con 0x7f6ebc0061e0 2024-05-09T03:09:46.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.250+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ec4012040 con 0x7f6ebc0061e0 2024-05-09T03:09:46.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.251+0000 7f6ebbfff640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ec00762d0 0x7f6ec00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:46.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.251+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6ec4097d90 con 0x7f6ebc0061e0 2024-05-09T03:09:46.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.251+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ec00762d0 0x7f6ec00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:46.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.251+0000 7f6ed2ffd640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ec00762d0 0x7f6ec00787e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8032000 tx=0x7f6ec8009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:46.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.254+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6ed4069320 con 0x7f6ebc0061e0 2024-05-09T03:09:46.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.484+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6ed406a270 con 0x7f6ebc0061e0 2024-05-09T03:09:46.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.484+0000 7f6ebbfff640 1 -- 172.21.15.46:0/818083854 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f6ec4019030 con 0x7f6ebc0061e0 2024-05-09T03:09:46.485 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:09:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ec00762d0 msgr2=0x7f6ec00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ec00762d0 0x7f6ec00787e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8032000 tx=0x7f6ec8009030 comp rx=0 tx=0).stop 2024-05-09T03:09:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 msgr2=0x7f6ed413d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:46.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ed413d780 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4011940 tx=0x7f6ec4011e00 comp rx=0 tx=0).stop 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 shutdown_connections 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ed413c950 0x7f6ed4142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ed415e570 0x7f6ed41f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ec00762d0 0x7f6ec00787e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 --2- 172.21.15.46:0/818083854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ebc0061e0 0x7f6ed413d780 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.486+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 >> 172.21.15.46:0/818083854 conn(0x7f6ed40b3910 msgr2=0x7f6ed406fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.487+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 shutdown_connections 2024-05-09T03:09:46.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46.487+0000 7f6ed9f27640 1 -- 172.21.15.46:0/818083854 wait complete. 2024-05-09T03:09:46.498 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:09:46.499 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:09:46.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:46.499 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:46.516 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42531sec preferred_lft 42531sec 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:46.517 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:09:46.518 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T03:09:46.518 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:46.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:46.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:46.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:46.712 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:09:46.738 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:09:46.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:46.740 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:09:46.802 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:09:46.802 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:09:46.802 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:09:46.803 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:09:46.803 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:09:46.803 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:46.803 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:46.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:09:46.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:09:46.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:09:46.803 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:46.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:47.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:47.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:47.104 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:09:47.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:47.120 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-05-09T03:09:47.177 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:47.177 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:47.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:47.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T03:09:47.178 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:47.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:47.307 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T03:09:47.308 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:47.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:47.443 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:47.443 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:47.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:09:47.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T03:09:47.443 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:47.574 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:09:47.574 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:47.574 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:47.590 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:09:47.590 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:47.591 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:47.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:09:47.724 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:47.788 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:09:47.788 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:09:47.788 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:09:47.789 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:09:47.790 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:09:47.791 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:47.791 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:47.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:47.792 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:09:47.853 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:09:47.853 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:09:47.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:09:47.930 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:47.988 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:09:47.988 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:09:47.988 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:09:47.988 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:09:47.989 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:47.990 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:47.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:47.991 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:09:48.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:09:48.008+0000 7f590c996480 -1 init, newargv = 0x7f58f80057c0 newargc=15 2024-05-09T03:09:48.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[43238]: starting ceph client 2024-05-09T03:09:48.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[43238]: starting fuse 2024-05-09T03:09:48.027 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:09:48.027 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:09:48.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:09:48.027 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:09:48.027 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:09:48.028 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:09:48.029 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:09:48.150 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.43238.asok 2024-05-09T03:09:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:48.161 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43238.asok 2024-05-09T03:09:48.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:48.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43238.asok status 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "43238", 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:09:48.312 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4587, 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4587 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 754061489 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:09:48.313 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 754061489 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4587 192.168.0.1:0/754061489", 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/754061489", 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 7, 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 20, 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:09:48.314 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:09:48.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:09:48.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.435+0000 7f43f5ac8640 1 Processor -- start 2024-05-09T03:09:48.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.435+0000 7f43f5ac8640 1 -- start start 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.436+0000 7f43f5ac8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f0073500 0x7f43f0073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.436+0000 7f43f5ac8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f43f0161cd0 0x7f43f01620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.436+0000 7f43f5ac8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f015e6e0 con 0x7f43f00af670 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43f5ac8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f43f00afa80 con 0x7f43f0161cd0 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43f5ac8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f43f0073e50 con 0x7f43f0073500 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43ef7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f43f00af670 0x7f43f00d1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46212/0 (socket says 172.21.15.46:46212) 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43ef7fe640 1 -- 172.21.15.46:0/2986548498 learned_addr learned my addr 172.21.15.46:0/2986548498 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43effff640 1 --2- 172.21.15.46:0/2986548498 >> v2:172.21.15.46:3300/0 conn(0x7f43f0161cd0 0x7f43f01620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:48.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43eeffd640 1 --2- 172.21.15.46:0/2986548498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f0073500 0x7f43f0073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43ee7fc640 1 -- 172.21.15.46:0/2986548498 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047169599 0 0) 0x7f43e0001110 con 0x7f43f00af670 2024-05-09T03:09:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43ee7fc640 1 -- 172.21.15.46:0/2986548498 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43d8003520 con 0x7f43f00af670 2024-05-09T03:09:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43effff640 1 -- 172.21.15.46:0/2986548498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f0073500 msgr2=0x7f43f0073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43effff640 1 --2- 172.21.15.46:0/2986548498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f0073500 0x7f43f0073910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43effff640 1 -- 172.21.15.46:0/2986548498 >> v1:172.21.15.46:6789/0 conn(0x7f43f00af670 legacy=0x7f43f00d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:48.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.437+0000 7f43effff640 1 -- 172.21.15.46:0/2986548498 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f00d1880 con 0x7f43f0161cd0 2024-05-09T03:09:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.438+0000 7f43effff640 1 --2- 172.21.15.46:0/2986548498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0161cd0 0x7f43f01620e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009d90 tx=0x7f43e802a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816cc82f93e47c88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.438+0000 7f43ee7fc640 1 -- 172.21.15.46:0/2986548498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e802ae60 con 0x7f43f0161cd0 2024-05-09T03:09:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.438+0000 7f43ee7fc640 1 -- 172.21.15.46:0/2986548498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e8007740 con 0x7f43f0161cd0 2024-05-09T03:09:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.438+0000 7f43ee7fc640 1 -- 172.21.15.46:0/2986548498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8007b10 con 0x7f43f0161cd0 2024-05-09T03:09:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 -- 172.21.15.46:0/2986548498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0161cd0 msgr2=0x7f43f01620e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:48.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/2986548498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0161cd0 0x7f43f01620e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009d90 tx=0x7f43e802a640 comp rx=0 tx=0).stop 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 -- 172.21.15.46:0/2986548498 shutdown_connections 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/2986548498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0161cd0 0x7f43f01620e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/2986548498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f0073500 0x7f43f0073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 -- 172.21.15.46:0/2986548498 >> 172.21.15.46:0/2986548498 conn(0x7f43f00b3910 msgr2=0x7f43f00b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 -- 172.21.15.46:0/2986548498 shutdown_connections 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.439+0000 7f43f5ac8640 1 -- 172.21.15.46:0/2986548498 wait complete. 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.440+0000 7f43f5ac8640 1 Processor -- start 2024-05-09T03:09:48.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.441+0000 7f43f5ac8640 1 -- start start 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43f5ac8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 0x7f43f014b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43f5ac8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f00af670 0x7f43f0160e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43f5ac8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f0161cd0 0x7f43f015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43f5ac8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43f015ef20 con 0x7f43f0073500 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43f5ac8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f43f015ec40 con 0x7f43f0161cd0 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43f5ac8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f43f015f200 con 0x7f43f00af670 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 0x7f43f014b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 0x7f43f014b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57108/0 (socket says 172.21.15.46:57108) 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 -- 172.21.15.46:0/3229617928 learned_addr learned my addr 172.21.15.46:0/3229617928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43eeffd640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f00af670 0x7f43f0160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43effff640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f0161cd0 0x7f43f015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:48.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 -- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f00af670 msgr2=0x7f43f0160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f00af670 0x7f43f0160e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 -- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f0161cd0 msgr2=0x7f43f015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f0161cd0 0x7f43f015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.442+0000 7f43ef7fe640 1 -- 172.21.15.46:0/3229617928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f0157230 con 0x7f43f0073500 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.443+0000 7f43ef7fe640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 0x7f43f014b7b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f43e000c760 tx=0x7f43e000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.443+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0017070 con 0x7f43f0073500 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.443+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e0005ab0 con 0x7f43f0073500 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.443+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f0161a90 con 0x7f43f0073500 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.443+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0001d10 con 0x7f43f0073500 2024-05-09T03:09:48.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.443+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f01f4690 con 0x7f43f0073500 2024-05-09T03:09:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.444+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f015f800 con 0x7f43f0073500 2024-05-09T03:09:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.445+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f43e001a070 con 0x7f43f0073500 2024-05-09T03:09:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.445+0000 7f43d7fff640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43dc0762d0 0x7f43dc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.445+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f43e0096900 con 0x7f43f0073500 2024-05-09T03:09:48.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.446+0000 7f43eeffd640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43dc0762d0 0x7f43dc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:48.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.446+0000 7f43eeffd640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43dc0762d0 0x7f43dc0787e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006fd0 tx=0x7f43e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.450+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f43f015e6e0 con 0x7f43f0073500 2024-05-09T03:09:48.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.681+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f43f006a270 con 0x7f43f0073500 2024-05-09T03:09:48.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.681+0000 7f43d7fff640 1 -- 172.21.15.46:0/3229617928 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f43e0096020 con 0x7f43f0073500 2024-05-09T03:09:48.683 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:09:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.683+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43dc0762d0 msgr2=0x7f43dc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.683+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43dc0762d0 0x7f43dc0787e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006fd0 tx=0x7f43e4008040 comp rx=0 tx=0).stop 2024-05-09T03:09:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.683+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 msgr2=0x7f43f014b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.683+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 0x7f43f014b7b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f43e000c760 tx=0x7f43e000cc20 comp rx=0 tx=0).stop 2024-05-09T03:09:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 shutdown_connections 2024-05-09T03:09:48.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43f0161cd0 0x7f43f015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43f00af670 0x7f43f0160e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43dc0762d0 0x7f43dc0787e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 --2- 172.21.15.46:0/3229617928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43f0073500 0x7f43f014b7b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 >> 172.21.15.46:0/3229617928 conn(0x7f43f00b3910 msgr2=0x7f43f00b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 shutdown_connections 2024-05-09T03:09:48.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48.684+0000 7f43f5ac8640 1 -- 172.21.15.46:0/3229617928 wait complete. 2024-05-09T03:09:48.695 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:09:48.695 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:09:48.696 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-05-09T03:09:48.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:48.696 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:09:48.714 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:48.714 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:48.714 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:48.714 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:48.715 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:48.715 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:48.715 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:48.715 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:48.715 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:09:48.715 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:48.716 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:48.716 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:09:48.716 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:09:48.716 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:48.716 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:09:48.716 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:09:48.717 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42529sec preferred_lft 42529sec 2024-05-09T03:09:48.717 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:09:48.717 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:48.717 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:48.717 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:48.717 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:09:48.718 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:09:48.718 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:48.718 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:09:48.718 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:48.718 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:09:48.719 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:48.719 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:09:48.719 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:48.719 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link tentative 2024-05-09T03:09:48.719 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:48.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:48.720 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:09:48.779 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:09:48.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:48.780 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:09:48.838 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:09:48.838 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:09:48.838 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:09:48.838 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:09:48.838 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:09:48.839 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:48.839 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:48.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:09:48.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:09:48.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:09:48.840 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:48.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:48.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:48.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:48.968 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:09:48.984 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:09:48.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:48.985 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-05-09T03:09:49.041 INFO:teuthology.orchestra.run.smithi046.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-05-09T03:09:49.042 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:49.042 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:49.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:09:49.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-05-09T03:09:49.043 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:49.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:49.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:49.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:49.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-09T03:09:49.187 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:49.188 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:49.188 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:49.188 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.188 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:49.188 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.188 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:09:49.189 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 7a:17:e1:72:34:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T03:09:49.190 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T03:09:49.190 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.190 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7817:e1ff:fe72:3425/64 scope link tentative 2024-05-09T03:09:49.190 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:49.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-05-09T03:09:49.257 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 7a:17:e1:72:34:25 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.258 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7817:e1ff:fe72:3425/64 scope link tentative 2024-05-09T03:09:49.259 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:49.260 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-05-09T03:09:49.260 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:49.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:49.416 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:49.416 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:09:49.416 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.1 up 2024-05-09T03:09:49.416 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.1 master ceph-brx 2024-05-09T03:09:49.417 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:09:49.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:49.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:49.496 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-05-09T03:09:49.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:49.497 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:49.512 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-05-09T03:09:49.512 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:49.513 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:49.569 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:09:49.636 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:09:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:09:49.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:49.697 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:09:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:09:49.751 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-05-09T03:09:49.751 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-05-09T03:09:49.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:09:49.819 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:09:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:49.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:09:49.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:09:49.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:49.881 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:09:49.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:2024-05-09T03:09:49.888+0000 7fe9dd80e480 -1 init, newargv = 0x7fe9c0001250 newargc=15 2024-05-09T03:09:49.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[43736]: starting ceph client 2024-05-09T03:09:49.894 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[43736]: starting fuse 2024-05-09T03:09:49.905 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:09:49.905 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-05-09T03:09:49.906 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 43] 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-05-09T03:09:49.906 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.2")) 2024-05-09T03:09:49.907 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:09:50.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.2.43736.asok 2024-05-09T03:09:50.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:50.037 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.43736.asok 2024-05-09T03:09:50.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:50.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.43736.asok status 2024-05-09T03:09:50.186 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:09:50.186 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:09:50.186 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "2", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "43736", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4599, 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4599 2024-05-09T03:09:50.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 984723164 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 984723164 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4599 192.168.0.2:0/984723164", 2024-05-09T03:09:50.188 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.2:0/984723164", 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 7, 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 20, 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:09:50.189 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:09:50.196 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:09:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.310+0000 7f5fb4d7d640 1 Processor -- start 2024-05-09T03:09:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.311+0000 7f5fb4d7d640 1 -- start start 2024-05-09T03:09:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5fb4d7d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5fb00d57a0 0x7f5fb0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5fb4d7d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5fb00b7ef0 0x7f5fb00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5fb4d7d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb00d49e0 con 0x7f5fb00b4af0 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5fb4d7d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb00d5bb0 con 0x7f5fb00b7ef0 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5fb4d7d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb00d4cc0 con 0x7f5fb00d57a0 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5faed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5fb00b7ef0 0x7f5fb00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5faed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5fb00b7ef0 0x7f5fb00b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57116/0 (socket says 172.21.15.46:57116) 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5faed9d640 1 -- 172.21.15.46:0/2718139554 learned_addr learned my addr 172.21.15.46:0/2718139554 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.312+0000 7f5fae59c640 1 --2- 172.21.15.46:0/2718139554 >> v2:172.21.15.88:3300/0 conn(0x7f5fb00d57a0 0x7f5fb0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.313+0000 7f5faed9d640 1 -- 172.21.15.46:0/2718139554 >> v2:172.21.15.88:3300/0 conn(0x7f5fb00d57a0 msgr2=0x7f5fb0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.313+0000 7f5faed9d640 1 --2- 172.21.15.46:0/2718139554 >> v2:172.21.15.88:3300/0 conn(0x7f5fb00d57a0 0x7f5fb0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.313+0000 7f5faed9d640 1 -- 172.21.15.46:0/2718139554 >> v1:172.21.15.88:6789/0 conn(0x7f5fb00b4af0 legacy=0x7f5fb00b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.313+0000 7f5faed9d640 1 -- 172.21.15.46:0/2718139554 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb00d4fa0 con 0x7f5fb00b7ef0 2024-05-09T03:09:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.313+0000 7f5faed9d640 1 --2- 172.21.15.46:0/2718139554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00b7ef0 0x7f5fb00b4f00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fb008ed20 tx=0x7f5fa802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aaa5d4e02fe919c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.313+0000 7f5fad59a640 1 -- 172.21.15.46:0/2718139554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8037070 con 0x7f5fb00b7ef0 2024-05-09T03:09:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fad59a640 1 -- 172.21.15.46:0/2718139554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa8032440 con 0x7f5fb00b7ef0 2024-05-09T03:09:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fad59a640 1 -- 172.21.15.46:0/2718139554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8032810 con 0x7f5fb00b7ef0 2024-05-09T03:09:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2718139554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00b7ef0 msgr2=0x7f5fb00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:50.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2718139554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00b7ef0 0x7f5fb00b4f00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5fb008ed20 tx=0x7f5fa802a270 comp rx=0 tx=0).stop 2024-05-09T03:09:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2718139554 shutdown_connections 2024-05-09T03:09:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2718139554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00b7ef0 0x7f5fb00b4f00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2718139554 >> v2:172.21.15.88:3300/0 conn(0x7f5fb00d57a0 0x7f5fb0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2718139554 >> 172.21.15.46:0/2718139554 conn(0x7f5fb00743e0 msgr2=0x7f5fb00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2718139554 shutdown_connections 2024-05-09T03:09:50.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.314+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2718139554 wait complete. 2024-05-09T03:09:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.316+0000 7f5fb4d7d640 1 Processor -- start 2024-05-09T03:09:50.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.316+0000 7f5fb4d7d640 1 -- start start 2024-05-09T03:09:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fb4d7d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5fb00b4af0 0x7f5fb0156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fb4d7d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fb00d57a0 0x7f5fb01e8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fb4d7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 0x7f5fb01e6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:50.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fb4d7d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00d1920 con 0x7f5fb00c2060 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fb4d7d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb00d15b0 con 0x7f5fb00d57a0 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fb4d7d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb00c2bc0 con 0x7f5fb00b4af0 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 0x7f5fb01e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 0x7f5fb01e6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57122/0 (socket says 172.21.15.46:57122) 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5fb00b4af0 0x7f5fb0156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 -- 172.21.15.46:0/2586467856 learned_addr learned my addr 172.21.15.46:0/2586467856 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fadd9b640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fb00d57a0 0x7f5fb01e8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 -- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5fb00b4af0 msgr2=0x7f5fb0156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5fb00b4af0 0x7f5fb0156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 -- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fb00d57a0 msgr2=0x7f5fb01e8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fb00d57a0 0x7f5fb01e8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5faed9d640 1 -- 172.21.15.46:0/2586467856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb01e7160 con 0x7f5fb00c2060 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.318+0000 7f5fadd9b640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fb00d57a0 0x7f5fb01e8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:50.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.319+0000 7f5faed9d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 0x7f5fb01e6430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80090b0 tx=0x7f5fa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.319+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8041020 con 0x7f5fb00c2060 2024-05-09T03:09:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.319+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa803d640 con 0x7f5fb00c2060 2024-05-09T03:09:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.319+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb01e7440 con 0x7f5fb00c2060 2024-05-09T03:09:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.319+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8037040 con 0x7f5fb00c2060 2024-05-09T03:09:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.319+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb01e7a00 con 0x7f5fb00c2060 2024-05-09T03:09:50.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.320+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb01e6ed0 con 0x7f5fb00c2060 2024-05-09T03:09:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.321+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5fa8044020 con 0x7f5fb00c2060 2024-05-09T03:09:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.321+0000 7f5fa6ffd640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f900762d0 0x7f5f900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.322+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5fa80b7bf0 con 0x7f5fb00c2060 2024-05-09T03:09:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.322+0000 7f5fae59c640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f900762d0 0x7f5f900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:50.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.322+0000 7f5fae59c640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f900762d0 0x7f5f900787e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5fb006d3b0 tx=0x7f5fa000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:50.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.326+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5fa804b030 con 0x7f5fb00c2060 2024-05-09T03:09:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.557+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5fb006a270 con 0x7f5fb00c2060 2024-05-09T03:09:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.558+0000 7f5fa6ffd640 1 -- 172.21.15.46:0/2586467856 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f5fa8080f10 con 0x7f5fb00c2060 2024-05-09T03:09:50.559 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f900762d0 msgr2=0x7f5f900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f900762d0 0x7f5f900787e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5fb006d3b0 tx=0x7f5fa000b040 comp rx=0 tx=0).stop 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 msgr2=0x7f5fb01e6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 0x7f5fb01e6430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80090b0 tx=0x7f5fa80046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 shutdown_connections 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f900762d0 0x7f5f900787e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb00c2060 0x7f5fb01e6430 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fb00d57a0 0x7f5fb01e8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 --2- 172.21.15.46:0/2586467856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5fb00b4af0 0x7f5fb0156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:50.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 >> 172.21.15.46:0/2586467856 conn(0x7f5fb00743e0 msgr2=0x7f5fb015d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:50.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 shutdown_connections 2024-05-09T03:09:50.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:50.560+0000 7f5fb4d7d640 1 -- 172.21.15.46:0/2586467856 wait complete. 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:09:50.572 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-05-09T03:09:50.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:50.573 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-05-09T03:09:50.592 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:50.592 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:09:50.593 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:50.594 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T03:09:50.594 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42583sec preferred_lft 42583sec 2024-05-09T03:09:50.594 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link 2024-05-09T03:09:50.594 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:50.594 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T03:09:50.594 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:50.594 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:50.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add name ceph-brx type bridge 2024-05-09T03:09:50.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr flush dev ceph-brx 2024-05-09T03:09:50.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set ceph-brx up 2024-05-09T03:09:50.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T03:09:50.595 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:50.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:50.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:50.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:50.867 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:09:50.892 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-05-09T03:09:50.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:50.894 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-05-09T03:09:50.997 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-05-09T03:09:50.997 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:09:50.997 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T03:09:50.997 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T03:09:50.997 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:09:50.997 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:50.997 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:50.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T03:09:50.997 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T03:09:50.998 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T03:09:50.998 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:51.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:51.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:51.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:51.230 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-05-09T03:09:51.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:51.245 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2024-05-09T03:09:51.301 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:51.301 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:51.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:51.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-05-09T03:09:51.301 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:51.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:51.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:51.419 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-05-09T03:09:51.419 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:51.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:51.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:51.557 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:51.557 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:51.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-05-09T03:09:51.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T03:09:51.557 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:51.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:51.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:51.634 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-05-09T03:09:51.634 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:51.634 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:51.654 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-05-09T03:09:51.654 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:51.654 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:51.711 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2024-05-09T03:09:51.776 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2024-05-09T03:09:51.833 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/selinux 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:51.834 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-4 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-5 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-6 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-7 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:51.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:51.835 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2024-05-09T03:09:51.889 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:09:51.889 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-05-09T03:09:51.932 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2024-05-09T03:09:51.956 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:52.012 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2024-05-09T03:09:52.012 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/selinux 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2024-05-09T03:09:52.013 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-4 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-5 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-6 2024-05-09T03:09:52.014 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-7 2024-05-09T03:09:52.015 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2024-05-09T03:09:52.015 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:52.015 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:52.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:52.016 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2024-05-09T03:09:52.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi088.stderr:2024-05-09T03:09:52.031+0000 7fb1838f3480 -1 init, newargv = 0x7fb1740054e0 newargc=15 2024-05-09T03:09:52.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi088.stderr:ceph-fuse[41676]: starting ceph client 2024-05-09T03:09:52.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi088.stderr:ceph-fuse[41676]: starting fuse 2024-05-09T03:09:52.048 INFO:teuthology.orchestra.run.smithi088.stdout:40 2024-05-09T03:09:52.048 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-05-09T03:09:52.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:09:52.048 DEBUG:teuthology.orchestra.run.smithi088:> import glob 2024-05-09T03:09:52.048 DEBUG:teuthology.orchestra.run.smithi088:> import re 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> import subprocess 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> def _find_admin_socket(client_name): 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> files = glob.glob(asok_path) 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> # Given a non-glob path, it better be there 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> if "*" not in asok_path: 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> assert(len(files) == 1) 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> return files[0] 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> for f in files: 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:09:52.049 DEBUG:teuthology.orchestra.run.smithi088:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> contents = proc_f.read() 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> if mountpoint in contents: 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> return f 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> print(_find_admin_socket("client.1")) 2024-05-09T03:09:52.050 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-05-09T03:09:52.167 INFO:teuthology.orchestra.run.smithi088.stdout:/var/run/ceph/ceph-client.1.41676.asok 2024-05-09T03:09:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:52.178 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41676.asok 2024-05-09T03:09:52.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:52.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41676.asok status 2024-05-09T03:09:52.323 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-09T03:09:52.323 INFO:teuthology.orchestra.run.smithi088.stdout: "metadata": { 2024-05-09T03:09:52.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:09:52.323 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:09:52.323 INFO:teuthology.orchestra.run.smithi088.stdout: "entity_id": "1", 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "hostname": "smithi088", 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "pid": "41676", 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "root": "/" 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_count": 0, 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_pinned_count": 0, 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4508, 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "inst": { 2024-05-09T03:09:52.324 INFO:teuthology.orchestra.run.smithi088.stdout: "name": { 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "client", 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 4508 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 833144650 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:09:52.325 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 833144650 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "inst_str": "client.4508 192.168.0.1:0/833144650", 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "addr_str": "192.168.0.1:0/833144650", 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "inode_count": 1, 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_epoch": 7, 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch": 20, 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklisted": false, 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs" 2024-05-09T03:09:52.326 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-09T03:09:52.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:09:52.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.460+0000 7f588c1d5640 1 Processor -- start 2024-05-09T03:09:52.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.460+0000 7f588c1d5640 1 -- start start 2024-05-09T03:09:52.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f588c1d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d0b90 0x7f58840b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:52.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f588c1d5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58840d2b80 con 0x7f58840cf6b0 2024-05-09T03:09:52.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f588c1d5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58840d2e60 con 0x7f58840b92e0 2024-05-09T03:09:52.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f588c1d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f58840d3140 con 0x7f58840d0b90 2024-05-09T03:09:52.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f588a772640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f58840cf6b0 0x7f588406d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46218/0 (socket says 172.21.15.46:46218) 2024-05-09T03:09:52.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f588a772640 1 -- 172.21.15.46:0/2072654 learned_addr learned my addr 172.21.15.46:0/2072654 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5889770640 1 --2- 172.21.15.46:0/2072654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d0b90 0x7f58840b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182698574 0 0) 0x7f58840d2b80 con 0x7f58840cf6b0 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5870003520 con 0x7f58840cf6b0 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068252157 0 0) 0x7f58840d2e60 con 0x7f58840b92e0 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58840d2b80 con 0x7f58840b92e0 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5889770640 1 -- 172.21.15.46:0/2072654 >> v1:172.21.15.88:6789/0 conn(0x7f58840b92e0 legacy=0x7f58840d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5889770640 1 -- 172.21.15.46:0/2072654 >> v1:172.21.15.46:6789/0 conn(0x7f58840cf6b0 legacy=0x7f588406d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3339792498 0 0) 0x7f5870003520 con 0x7f58840cf6b0 2024-05-09T03:09:52.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.462+0000 7f5889770640 1 -- 172.21.15.46:0/2072654 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58840d3420 con 0x7f58840d0b90 2024-05-09T03:09:52.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.463+0000 7f5889770640 1 --2- 172.21.15.46:0/2072654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d0b90 0x7f58840b51a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f58740097d0 tx=0x7f587402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ae273afa6aaeb4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:52.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.463+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874037070 con 0x7f58840d0b90 2024-05-09T03:09:52.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f587402ab90 con 0x7f58840d0b90 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f5888f6f640 1 -- 172.21.15.46:0/2072654 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c001a70 con 0x7f58840d0b90 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 -- 172.21.15.46:0/2072654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d0b90 msgr2=0x7f58840b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 --2- 172.21.15.46:0/2072654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d0b90 0x7f58840b51a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f58740097d0 tx=0x7f587402a100 comp rx=0 tx=0).stop 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 -- 172.21.15.46:0/2072654 shutdown_connections 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 --2- 172.21.15.46:0/2072654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d0b90 0x7f58840b51a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 -- 172.21.15.46:0/2072654 >> 172.21.15.46:0/2072654 conn(0x7f58840d3c00 msgr2=0x7f58840d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 -- 172.21.15.46:0/2072654 shutdown_connections 2024-05-09T03:09:52.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.464+0000 7f588c1d5640 1 -- 172.21.15.46:0/2072654 wait complete. 2024-05-09T03:09:52.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.466+0000 7f588c1d5640 1 Processor -- start 2024-05-09T03:09:52.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.466+0000 7f588c1d5640 1 -- start start 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588c1d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840b92e0 0x7f58840cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588c1d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 0x7f5884160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588c1d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884158fe0 0x7f5884158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588c1d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f588416b3d0 con 0x7f58840cf6b0 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588c1d5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f588416b250 con 0x7f5884158fe0 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588c1d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f588416b550 con 0x7f58840b92e0 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 0x7f5884160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:52.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 0x7f5884160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57134/0 (socket says 172.21.15.46:57134) 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 -- 172.21.15.46:0/3904165886 learned_addr learned my addr 172.21.15.46:0/3904165886 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f588a772640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884158fe0 0x7f5884158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889f71640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840b92e0 0x7f58840cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 -- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840b92e0 msgr2=0x7f58840cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840b92e0 0x7f58840cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 -- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884158fe0 msgr2=0x7f5884158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884158fe0 0x7f5884158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 -- 172.21.15.46:0/3904165886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588415f090 con 0x7f58840cf6b0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f5889770640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 0x7f5884160130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f58740097d0 tx=0x7f58740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.468+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874042020 con 0x7f58840cf6b0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.469+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f587403d670 con 0x7f58840cf6b0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.469+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5874037040 con 0x7f58840cf6b0 2024-05-09T03:09:52.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.469+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588415f370 con 0x7f58840cf6b0 2024-05-09T03:09:52.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.469+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588415f930 con 0x7f58840cf6b0 2024-05-09T03:09:52.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.469+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588406a270 con 0x7f58840cf6b0 2024-05-09T03:09:52.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.470+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5874045020 con 0x7f58840cf6b0 2024-05-09T03:09:52.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.471+0000 7f587a7fc640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f58680762d0 0x7f58680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:52.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.471+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f58740b7e30 con 0x7f58840cf6b0 2024-05-09T03:09:52.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.471+0000 7f5889f71640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f58680762d0 0x7f58680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:52.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.472+0000 7f5889f71640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f58680762d0 0x7f58680787e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5880001120 tx=0x7f5880009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:52.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.476+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f587404c030 con 0x7f58840cf6b0 2024-05-09T03:09:52.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.707+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f58840cded0 con 0x7f58840cf6b0 2024-05-09T03:09:52.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.708+0000 7f587a7fc640 1 -- 172.21.15.46:0/3904165886 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f5874081820 con 0x7f58840cf6b0 2024-05-09T03:09:52.708 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f58680762d0 msgr2=0x7f58680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f58680762d0 0x7f58680787e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5880001120 tx=0x7f5880009040 comp rx=0 tx=0).stop 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 msgr2=0x7f5884160130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 0x7f5884160130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f58740097d0 tx=0x7f58740046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 shutdown_connections 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884158fe0 0x7f5884158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f58680762d0 0x7f58680787e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840cf6b0 0x7f5884160130 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 --2- 172.21.15.46:0/3904165886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840b92e0 0x7f58840cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 >> 172.21.15.46:0/3904165886 conn(0x7f58840d3c00 msgr2=0x7f58840b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 shutdown_connections 2024-05-09T03:09:52.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:52.710+0000 7f588c1d5640 1 -- 172.21.15.46:0/3904165886 wait complete. 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:09:52.722 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-05-09T03:09:52.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:52.722 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-05-09T03:09:52.739 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:52.739 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:52.739 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:52.739 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:52.739 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42581sec preferred_lft 42581sec 2024-05-09T03:09:52.740 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 62:28:f5:4c:a5:dd brd ff:ff:ff:ff:ff:ff 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::7455:65ff:fe93:43f/64 scope link 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 62:28:f5:4c:a5:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::6028:f5ff:fe4c:a5dd/64 scope link tentative 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:52.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:52.742 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:09:52.804 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-05-09T03:09:52.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:52.805 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-05-09T03:09:52.862 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-05-09T03:09:52.863 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:09:52.863 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T03:09:52.863 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T03:09:52.863 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:09:52.863 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:52.863 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:52.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T03:09:52.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T03:09:52.863 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T03:09:52.864 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:52.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:52.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:52.989 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-05-09T03:09:53.004 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T03:09:53.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:53.005 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2024-05-09T03:09:53.060 INFO:teuthology.orchestra.run.smithi088.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-05-09T03:09:53.061 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:53.061 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:53.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T03:09:53.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-05-09T03:09:53.061 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:53.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:53.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:53.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether c2:28:06:2a:c1:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T03:09:53.203 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T03:09:53.204 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.204 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c028:6ff:fe2a:c116/64 scope link 2024-05-09T03:09:53.204 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:53.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-05-09T03:09:53.269 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether c2:28:06:2a:c1:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c028:6ff:fe2a:c116/64 scope link 2024-05-09T03:09:53.270 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:09:53.271 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-05-09T03:09:53.272 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:53.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:53.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:53.424 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:09:53.424 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-05-09T03:09:53.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.1 up 2024-05-09T03:09:53.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.1 master ceph-brx 2024-05-09T03:09:53.424 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-05-09T03:09:53.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:09:53.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:53.501 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-05-09T03:09:53.501 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:53.502 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:53.516 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-05-09T03:09:53.517 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:09:53.517 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:53.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2024-05-09T03:09:53.637 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:53.694 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2024-05-09T03:09:53.694 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2024-05-09T03:09:53.694 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2024-05-09T03:09:53.694 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/selinux 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2024-05-09T03:09:53.695 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-4 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-5 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-6 2024-05-09T03:09:53.696 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-7 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run.smithi088.stdout:/home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T03:09:53.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:53.697 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2024-05-09T03:09:53.750 INFO:teuthology.orchestra.run.smithi088.stdout:40 2024-05-09T03:09:53.750 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-05-09T03:09:53.750 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-05-09T03:09:53.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2024-05-09T03:09:53.816 DEBUG:teuthology.orchestra.run.smithi088:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:09:53.874 INFO:teuthology.orchestra.run.smithi088.stdout:/proc 2024-05-09T03:09:53.874 INFO:teuthology.orchestra.run.smithi088.stdout:/sys 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/dev 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/security 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/shm 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/pts 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/run 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/cgroup 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/pstore 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/bpf 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/ 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/selinux 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/mqueue 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/debug 2024-05-09T03:09:53.875 INFO:teuthology.orchestra.run.smithi088.stdout:/dev/hugepages 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/tracing 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/fs/fuse/connections 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/sys/kernel/config 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/run/user/1000 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-4 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-5 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-6 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/var/lib/ceph/osd/ceph-7 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns 2024-05-09T03:09:53.876 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:53.877 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:09:53.877 INFO:teuthology.orchestra.run.smithi088.stdout:/home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:53.877 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T03:09:53.877 INFO:teuthology.orchestra.run.smithi088.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T03:09:53.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:53.877 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2024-05-09T03:09:53.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi088.stderr:2024-05-09T03:09:53.883+0000 7f4c509e7480 -1 init, newargv = 0x7f4c3400a420 newargc=15 2024-05-09T03:09:53.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi088.stderr:ceph-fuse[42129]: starting ceph client 2024-05-09T03:09:53.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi088.stderr:ceph-fuse[42129]: starting fuse 2024-05-09T03:09:53.899 INFO:teuthology.orchestra.run.smithi088.stdout:40 2024-05-09T03:09:53.900 INFO:teuthology.orchestra.run.smithi088.stdout:43 2024-05-09T03:09:53.900 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> import glob 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> import re 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> import subprocess 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> def _find_admin_socket(client_name): 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-05-09T03:09:53.900 DEBUG:teuthology.orchestra.run.smithi088:> files = glob.glob(asok_path) 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> # Given a non-glob path, it better be there 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> if "*" not in asok_path: 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> assert(len(files) == 1) 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> return files[0] 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> for f in files: 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> contents = proc_f.read() 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> if mountpoint in contents: 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> return f 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:09:53.901 DEBUG:teuthology.orchestra.run.smithi088:> 2024-05-09T03:09:53.902 DEBUG:teuthology.orchestra.run.smithi088:> print(_find_admin_socket("client.3")) 2024-05-09T03:09:53.902 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-05-09T03:09:54.018 INFO:teuthology.orchestra.run.smithi088.stdout:/var/run/ceph/ceph-client.3.42129.asok 2024-05-09T03:09:54.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-05-09T03:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:09:54.029 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.42129.asok 2024-05-09T03:09:54.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.42129.asok status 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "metadata": { 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "entity_id": "3", 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "hostname": "smithi088", 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "pid": "42129", 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "root": "/" 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_count": 0, 2024-05-09T03:09:54.177 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_pinned_count": 0, 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4588, 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "inst": { 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "name": { 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "client", 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 4588 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.2:0", 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 683449792 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2024-05-09T03:09:54.178 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.2:0", 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 683449792 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "inst_str": "client.4588 192.168.0.2:0/683449792", 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "addr_str": "192.168.0.2:0/683449792", 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "inode_count": 1, 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_epoch": 7, 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch": 20, 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklisted": false, 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs" 2024-05-09T03:09:54.179 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-05-09T03:09:54.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.186 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:09:54.186 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:54.207 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:09:54.207 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:54.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:54.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.280 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:09:54.280 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:54.340 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:09:54.340 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:54.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:54.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.413 DEBUG:teuthology.orchestra.run.smithi088:> stat --file-system '--printf=%T 2024-05-09T03:09:54.414 DEBUG:teuthology.orchestra.run.smithi088:> ' -- /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:54.429 INFO:teuthology.orchestra.run.smithi088.stdout:fuseblk 2024-05-09T03:09:54.429 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:54.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:54.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.497 DEBUG:teuthology.orchestra.run.smithi088:> stat --file-system '--printf=%T 2024-05-09T03:09:54.497 DEBUG:teuthology.orchestra.run.smithi088:> ' -- /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:54.554 INFO:teuthology.orchestra.run.smithi088.stdout:fuseblk 2024-05-09T03:09:54.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:54.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:54.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-05-09T03:09:54.621 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-05-09T03:09:54.656 INFO:tasks.cephfs_test_runner:Starting test: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2024-05-09T03:09:54.657 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.785+0000 7f4cad8f0640 1 Processor -- start 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.785+0000 7f4cad8f0640 1 -- start start 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4cad8f0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ca80d57a0 0x7f4ca8160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4cad8f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4cad8f0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca80d49e0 con 0x7f4ca80b77e0 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4cad8f0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca80d5bb0 con 0x7f4ca80d57a0 2024-05-09T03:09:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4cad8f0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca80d4cc0 con 0x7f4ca80b4af0 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ca80d57a0 0x7f4ca8160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4ca80d57a0 0x7f4ca8160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57146/0 (socket says 172.21.15.46:57146) 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1377365064 learned_addr learned my addr 172.21.15.46:0/1377365064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca67fc640 1 --2- 172.21.15.46:0/1377365064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1377365064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 msgr2=0x7f4ca80b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 --2- 172.21.15.46:0/1377365064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1377365064 >> v1:172.21.15.88:6789/0 conn(0x7f4ca80b77e0 legacy=0x7f4ca80b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:54.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.787+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1377365064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca80d4fa0 con 0x7f4ca80d57a0 2024-05-09T03:09:54.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.788+0000 7f4ca6ffd640 1 --2- 172.21.15.46:0/1377365064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80d57a0 0x7f4ca8160610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009870 tx=0x7f4c9802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0cfeb40c185d21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:54.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.788+0000 7f4ca5ffb640 1 -- 172.21.15.46:0/1377365064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98037070 con 0x7f4ca80d57a0 2024-05-09T03:09:54.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.788+0000 7f4ca5ffb640 1 -- 172.21.15.46:0/1377365064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c98032440 con 0x7f4ca80d57a0 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.788+0000 7f4ca5ffb640 1 -- 172.21.15.46:0/1377365064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98032810 con 0x7f4ca80d57a0 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 -- 172.21.15.46:0/1377365064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80d57a0 msgr2=0x7f4ca8160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/1377365064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80d57a0 0x7f4ca8160610 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009870 tx=0x7f4c9802a270 comp rx=0 tx=0).stop 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 -- 172.21.15.46:0/1377365064 shutdown_connections 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/1377365064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/1377365064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80d57a0 0x7f4ca8160610 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 -- 172.21.15.46:0/1377365064 >> 172.21.15.46:0/1377365064 conn(0x7f4ca80743e0 msgr2=0x7f4ca80d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 -- 172.21.15.46:0/1377365064 shutdown_connections 2024-05-09T03:09:54.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.789+0000 7f4cad8f0640 1 -- 172.21.15.46:0/1377365064 wait complete. 2024-05-09T03:09:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.791+0000 7f4cad8f0640 1 Processor -- start 2024-05-09T03:09:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.791+0000 7f4cad8f0640 1 -- start start 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4cad8f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4cad8f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 0x7f4ca8156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4cad8f0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ca80d57a0 0x7f4ca81f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4cad8f0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca80d1920 con 0x7f4ca80b77e0 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4cad8f0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca80d15b0 con 0x7f4ca80d57a0 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4cad8f0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca80cb890 con 0x7f4ca80b4af0 2024-05-09T03:09:54.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4ca67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 0x7f4ca8156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4ca67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 0x7f4ca8156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57148/0 (socket says 172.21.15.46:57148) 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4ca67fc640 1 -- 172.21.15.46:0/351016219 learned_addr learned my addr 172.21.15.46:0/351016219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4ca6ffd640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.792+0000 7f4ca77fe640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ca80d57a0 0x7f4ca81f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4ca67fc640 1 -- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 msgr2=0x7f4ca80c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4ca67fc640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4ca67fc640 1 -- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ca80d57a0 msgr2=0x7f4ca81f9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4ca67fc640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ca80d57a0 0x7f4ca81f9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4ca67fc640 1 -- 172.21.15.46:0/351016219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca8159290 con 0x7f4ca80b77e0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4ca67fc640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 0x7f4ca8156590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e8a0 tx=0x7f4c9c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c017070 con 0x7f4ca80b77e0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9c00c940 con 0x7f4ca80b77e0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c00cd30 con 0x7f4ca80b77e0 2024-05-09T03:09:54.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.793+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca8159570 con 0x7f4ca80b77e0 2024-05-09T03:09:54.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.794+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca8159b30 con 0x7f4ca80b77e0 2024-05-09T03:09:54.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.795+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca806a270 con 0x7f4ca80b77e0 2024-05-09T03:09:54.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.795+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4c9c01a070 con 0x7f4ca80b77e0 2024-05-09T03:09:54.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.796+0000 7f4c8ffff640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c94076300 0x7f4c94078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:54.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.796+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4c9c097810 con 0x7f4ca80b77e0 2024-05-09T03:09:54.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.796+0000 7f4ca6ffd640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c94076300 0x7f4c94078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:54.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.797+0000 7f4ca6ffd640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c94076300 0x7f4c94078810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4c98034000 tx=0x7f4c9802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:54.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.801+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4c9c060aa0 con 0x7f4ca80b77e0 2024-05-09T03:09:54.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.992+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ca80d32c0 con 0x7f4ca80b77e0 2024-05-09T03:09:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.993+0000 7f4c8ffff640 1 -- 172.21.15.46:0/351016219 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7f4c9c064da0 con 0x7f4ca80b77e0 2024-05-09T03:09:54.994 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:09:54.994 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:09:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c94076300 msgr2=0x7f4c94078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c94076300 0x7f4c94078810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4c98034000 tx=0x7f4c9802e000 comp rx=0 tx=0).stop 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 msgr2=0x7f4ca8156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 0x7f4ca8156590 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e8a0 tx=0x7f4c9c00ed60 comp rx=0 tx=0).stop 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 shutdown_connections 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ca80d57a0 0x7f4ca81f9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c94076300 0x7f4c94078810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca80b77e0 0x7f4ca8156590 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.995+0000 7f4cad8f0640 1 --2- 172.21.15.46:0/351016219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ca80b4af0 0x7f4ca80c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.996+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 >> 172.21.15.46:0/351016219 conn(0x7f4ca80743e0 msgr2=0x7f4ca80b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.996+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 shutdown_connections 2024-05-09T03:09:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:54.996+0000 7f4cad8f0640 1 -- 172.21.15.46:0/351016219 wait complete. 2024-05-09T03:09:55.008 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T03:09:22.506928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T03:09:31.491289+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T03:09:32.526431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T03:09:55.008 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T03:09:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.126+0000 7f990dc68640 1 Processor -- start 2024-05-09T03:09:55.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.126+0000 7f990dc68640 1 -- start start 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f990dc68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f99080b4700 0x7f99080d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f990dc68640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99080d2630 0x7f99080b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f990dc68640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990806ebe0 con 0x7f99080b3f50 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f990dc68640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f990806eec0 con 0x7f99080d2630 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f990dc68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f990806f1a0 con 0x7f99080b4700 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99080d2630 0x7f99080b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f99080d2630 0x7f99080b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57160/0 (socket says 172.21.15.46:57160) 2024-05-09T03:09:55.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 -- 172.21.15.46:0/1163340405 learned_addr learned my addr 172.21.15.46:0/1163340405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9907fff640 1 --1- 172.21.15.46:0/1163340405 >> v1:172.21.15.88:6789/0 conn(0x7f99080b3f50 0x7f990808cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:45360/0 (socket says 172.21.15.46:45360) 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f99077fe640 1 --2- 172.21.15.46:0/1163340405 >> v2:172.21.15.88:3300/0 conn(0x7f99080b4700 0x7f99080d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 -- 172.21.15.46:0/1163340405 >> v2:172.21.15.88:3300/0 conn(0x7f99080b4700 msgr2=0x7f99080d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 --2- 172.21.15.46:0/1163340405 >> v2:172.21.15.88:3300/0 conn(0x7f99080b4700 0x7f99080d20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 -- 172.21.15.46:0/1163340405 >> v1:172.21.15.88:6789/0 conn(0x7f99080b3f50 legacy=0x7f990808cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f99067fc640 1 -- 172.21.15.46:0/1163340405 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404537901 0 0) 0x7f990806ebe0 con 0x7f99080b3f50 2024-05-09T03:09:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.127+0000 7f9906ffd640 1 -- 172.21.15.46:0/1163340405 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990806f480 con 0x7f99080d2630 2024-05-09T03:09:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.128+0000 7f9906ffd640 1 --2- 172.21.15.46:0/1163340405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080d2630 0x7f99080b3a10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9908070130 tx=0x7f98fc02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4bbce8459002c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.128+0000 7f99067fc640 1 -- 172.21.15.46:0/1163340405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc037070 con 0x7f99080d2630 2024-05-09T03:09:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.128+0000 7f99067fc640 1 -- 172.21.15.46:0/1163340405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98fc032440 con 0x7f99080d2630 2024-05-09T03:09:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.128+0000 7f99067fc640 1 -- 172.21.15.46:0/1163340405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc032810 con 0x7f99080d2630 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 -- 172.21.15.46:0/1163340405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080d2630 msgr2=0x7f99080b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 --2- 172.21.15.46:0/1163340405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080d2630 0x7f99080b3a10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9908070130 tx=0x7f98fc02a150 comp rx=0 tx=0).stop 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 -- 172.21.15.46:0/1163340405 shutdown_connections 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 --2- 172.21.15.46:0/1163340405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080d2630 0x7f99080b3a10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 --2- 172.21.15.46:0/1163340405 >> v2:172.21.15.88:3300/0 conn(0x7f99080b4700 0x7f99080d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 -- 172.21.15.46:0/1163340405 >> 172.21.15.46:0/1163340405 conn(0x7f99080b55d0 msgr2=0x7f99080b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 -- 172.21.15.46:0/1163340405 shutdown_connections 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.129+0000 7f990dc68640 1 -- 172.21.15.46:0/1163340405 wait complete. 2024-05-09T03:09:55.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.131+0000 7f990dc68640 1 Processor -- start 2024-05-09T03:09:55.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.131+0000 7f990dc68640 1 -- start start 2024-05-09T03:09:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.132+0000 7f990dc68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 0x7f99081f5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.132+0000 7f990dc68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99080b4700 0x7f990809ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.132+0000 7f990dc68640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99080d2630 0x7f99080cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.132+0000 7f990dc68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f99080b51d0 con 0x7f99080b3f50 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.132+0000 7f990dc68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f99080d2a40 con 0x7f99080b4700 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.132+0000 7f990dc68640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f99080cd080 con 0x7f99080d2630 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 0x7f99081f5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 0x7f99081f5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57166/0 (socket says 172.21.15.46:57166) 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 -- 172.21.15.46:0/448903235 learned_addr learned my addr 172.21.15.46:0/448903235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f9907fff640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99080d2630 0x7f99080cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f9906ffd640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99080b4700 0x7f990809ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 -- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99080d2630 msgr2=0x7f99080cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99080d2630 0x7f99080cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 -- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99080b4700 msgr2=0x7f990809ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99080b4700 0x7f990809ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 -- 172.21.15.46:0/448903235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990809fc90 con 0x7f99080b3f50 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f99077fe640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 0x7f99081f5eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e8a0 tx=0x7f98f800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8017070 con 0x7f99080b3f50 2024-05-09T03:09:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f800c940 con 0x7f99080b3f50 2024-05-09T03:09:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800cd30 con 0x7f99080b3f50 2024-05-09T03:09:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990809ff70 con 0x7f99080b3f50 2024-05-09T03:09:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.133+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99080a0530 con 0x7f99080b3f50 2024-05-09T03:09:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.134+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990806a270 con 0x7f99080b3f50 2024-05-09T03:09:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.136+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f98f801a070 con 0x7f99080b3f50 2024-05-09T03:09:55.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.136+0000 7f98effff640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98f4076300 0x7f98f4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.136+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f98f8097b90 con 0x7f99080b3f50 2024-05-09T03:09:55.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.136+0000 7f9906ffd640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98f4076300 0x7f98f4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.137+0000 7f9906ffd640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98f4076300 0x7f98f4078810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98fc034000 tx=0x7f98fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.140+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f98f8060ef0 con 0x7f99080b3f50 2024-05-09T03:09:55.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.332+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f99080b8560 con 0x7f99080b3f50 2024-05-09T03:09:55.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.332+0000 7f98effff640 1 -- 172.21.15.46:0/448903235 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f98f80651f0 con 0x7f99080b3f50 2024-05-09T03:09:55.333 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-05-09T03:09:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98f4076300 msgr2=0x7f98f4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98f4076300 0x7f98f4078810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98fc034000 tx=0x7f98fc02e000 comp rx=0 tx=0).stop 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 msgr2=0x7f99081f5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 0x7f99081f5eb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e8a0 tx=0x7f98f800ed60 comp rx=0 tx=0).stop 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 shutdown_connections 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99080d2630 0x7f99080cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99080b4700 0x7f990809ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98f4076300 0x7f98f4078810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 --2- 172.21.15.46:0/448903235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99080b3f50 0x7f99081f5eb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 >> 172.21.15.46:0/448903235 conn(0x7f99080b55d0 msgr2=0x7f9908071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 shutdown_connections 2024-05-09T03:09:55.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.334+0000 7f990dc68640 1 -- 172.21.15.46:0/448903235 wait complete. 2024-05-09T03:09:55.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T03:09:55.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.463+0000 7f36bb4bd640 1 Processor -- start 2024-05-09T03:09:55.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.463+0000 7f36bb4bd640 1 -- start start 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.464+0000 7f36bb4bd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36b406e4e0 0x7f36b41603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.464+0000 7f36bb4bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.464+0000 7f36bb4bd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b40cf5b0 con 0x7f36b4071570 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.464+0000 7f36bb4bd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f36b4072090 con 0x7f36b406e4e0 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.464+0000 7f36bb4bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f36b40d2c30 con 0x7f36b40cec60 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.464+0000 7f36b9259640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36b406e4e0 0x7f36b41603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b9259640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36b406e4e0 0x7f36b41603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57182/0 (socket says 172.21.15.46:57182) 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b9259640 1 -- 172.21.15.46:0/2505261643 learned_addr learned my addr 172.21.15.46:0/2505261643 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:55.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b8a58640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b9259640 1 -- 172.21.15.46:0/2505261643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 msgr2=0x7f36b40cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b9259640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b9259640 1 -- 172.21.15.46:0/2505261643 >> v1:172.21.15.88:6789/0 conn(0x7f36b4071570 legacy=0x7f36b4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b9259640 1 -- 172.21.15.46:0/2505261643 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b40d2f10 con 0x7f36b406e4e0 2024-05-09T03:09:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.465+0000 7f36b8a58640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:09:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.466+0000 7f36b9259640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b406e4e0 0x7f36b41603d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009870 tx=0x7f36b002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d205f549c25c5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.466+0000 7f36abfff640 1 -- 172.21.15.46:0/2505261643 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0037070 con 0x7f36b406e4e0 2024-05-09T03:09:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.466+0000 7f36abfff640 1 -- 172.21.15.46:0/2505261643 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b002ae50 con 0x7f36b406e4e0 2024-05-09T03:09:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.466+0000 7f36abfff640 1 -- 172.21.15.46:0/2505261643 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b0032860 con 0x7f36b406e4e0 2024-05-09T03:09:55.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.466+0000 7f36bb4bd640 1 -- 172.21.15.46:0/2505261643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b406e4e0 msgr2=0x7f36b41603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.466+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b406e4e0 0x7f36b41603d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009870 tx=0x7f36b002a300 comp rx=0 tx=0).stop 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.467+0000 7f36bb4bd640 1 -- 172.21.15.46:0/2505261643 shutdown_connections 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.467+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.467+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/2505261643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b406e4e0 0x7f36b41603d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.467+0000 7f36bb4bd640 1 -- 172.21.15.46:0/2505261643 >> 172.21.15.46:0/2505261643 conn(0x7f36b40d3ae0 msgr2=0x7f36b40d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.467+0000 7f36bb4bd640 1 -- 172.21.15.46:0/2505261643 shutdown_connections 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.467+0000 7f36bb4bd640 1 -- 172.21.15.46:0/2505261643 wait complete. 2024-05-09T03:09:55.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.468+0000 7f36bb4bd640 1 Processor -- start 2024-05-09T03:09:55.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.468+0000 7f36bb4bd640 1 -- start start 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36bb4bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b406e4e0 0x7f36b40cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36bb4bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 0x7f36b40b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36bb4bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36bb4bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f36b40d37b0 con 0x7f36b4071570 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36bb4bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f36b40d34d0 con 0x7f36b406e4e0 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36bb4bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f36b4160910 con 0x7f36b40cec60 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 0x7f36b40b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 0x7f36b40b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57192/0 (socket says 172.21.15.46:57192) 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 -- 172.21.15.46:0/924841036 learned_addr learned my addr 172.21.15.46:0/924841036 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b9259640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b406e4e0 0x7f36b40cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b9a5a640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 -- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 msgr2=0x7f36b40cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 -- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b406e4e0 msgr2=0x7f36b40cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b406e4e0 0x7f36b40cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 -- 172.21.15.46:0/924841036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b40cab90 con 0x7f36b4071570 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.470+0000 7f36b8a58640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 0x7f36b40b00f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f36a400d800 tx=0x7f36a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.471+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4004030 con 0x7f36b4071570 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.471+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a40041d0 con 0x7f36b4071570 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.471+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b40cb090 con 0x7f36b4071570 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.471+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a40096a0 con 0x7f36b4071570 2024-05-09T03:09:55.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.471+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b40cb650 con 0x7f36b4071570 2024-05-09T03:09:55.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.472+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b406a270 con 0x7f36b4071570 2024-05-09T03:09:55.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.472+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f36a4009840 con 0x7f36b4071570 2024-05-09T03:09:55.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.473+0000 7f36a9ffb640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36980762d0 0x7f36980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.473+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f36a40977e0 con 0x7f36b4071570 2024-05-09T03:09:55.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.473+0000 7f36b9259640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36980762d0 0x7f36980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.474+0000 7f36b9259640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36980762d0 0x7f36980787e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36b0034000 tx=0x7f36b002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.478+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f36a4060a50 con 0x7f36b4071570 2024-05-09T03:09:55.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.670+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f36b4072280 con 0x7f36b4071570 2024-05-09T03:09:55.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.670+0000 7f36a9ffb640 1 -- 172.21.15.46:0/924841036 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f36a4064d50 con 0x7f36b4071570 2024-05-09T03:09:55.671 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.672+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36980762d0 msgr2=0x7f36980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36980762d0 0x7f36980787e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36b0034000 tx=0x7f36b002e000 comp rx=0 tx=0).stop 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 msgr2=0x7f36b40b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 0x7f36b40b00f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f36a400d800 tx=0x7f36a400dcc0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 shutdown_connections 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36b40cec60 0x7f36b40cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36980762d0 0x7f36980787e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b4071570 0x7f36b40b00f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 --2- 172.21.15.46:0/924841036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b406e4e0 0x7f36b40cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 >> 172.21.15.46:0/924841036 conn(0x7f36b40d3ae0 msgr2=0x7f36b415da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 shutdown_connections 2024-05-09T03:09:55.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.673+0000 7f36bb4bd640 1 -- 172.21.15.46:0/924841036 wait complete. 2024-05-09T03:09:55.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T03:09:55.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.801+0000 7f5bb2b5e640 1 Processor -- start 2024-05-09T03:09:55.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.802+0000 7f5bb2b5e640 1 -- start start 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb2b5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5bac0b7e70 0x7f5bac0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb2b5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 0x7f5bac0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb2b5e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bac0b5550 con 0x7f5bac0b5140 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb2b5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5bac0d3480 con 0x7f5bac0b7e70 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb2b5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5bac0d44e0 con 0x7f5bac0d3df0 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb08fa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5bac0b5140 0x7f5bac0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43296/0 (socket says 172.21.15.46:43296) 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb08fa640 1 -- 172.21.15.46:0/270994628 learned_addr learned my addr 172.21.15.46:0/270994628 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bb10fb640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 0x7f5bac0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797873718 0 0) 0x7f5bac0b5550 con 0x7f5bac0b5140 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5babfff640 1 --2- 172.21.15.46:0/270994628 >> v2:172.21.15.88:3300/0 conn(0x7f5bac0b7e70 0x7f5bac0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.803+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b98003520 con 0x7f5bac0b5140 2024-05-09T03:09:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2986019617 0 0) 0x7f5b98003520 con 0x7f5bac0b5140 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 msgr2=0x7f5bac0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 0x7f5bac0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 >> v2:172.21.15.88:3300/0 conn(0x7f5bac0b7e70 msgr2=0x7f5bac0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 --2- 172.21.15.46:0/270994628 >> v2:172.21.15.88:3300/0 conn(0x7f5bac0b7e70 0x7f5bac0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac0d47c0 con 0x7f5bac0b5140 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f5ba40016f0 con 0x7f5bac0b5140 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 >> v1:172.21.15.46:6789/0 conn(0x7f5bac0b5140 legacy=0x7f5bac0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b98003a00 0x7f5b98003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 0x7f5b98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b98004d40 0x7f5b98005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0d44e0 con 0x7f5b98003a00 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9800ba70 con 0x7f5b98004d40 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5b9800bc70 con 0x7f5bac0d3df0 2024-05-09T03:09:55.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b98003a00 0x7f5b98003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5babfff640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 0x7f5b98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb10fb640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b98004d40 0x7f5b98005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb08fa640 1 -- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 msgr2=0x7f5b98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5bac0d3df0 0x7f5b98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb08fa640 1 -- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b98004d40 msgr2=0x7f5b98005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b98004d40 0x7f5b98005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.804+0000 7f5bb08fa640 1 -- 172.21.15.46:0/270994628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b98003520 con 0x7f5b98003a00 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.805+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b98003a00 0x7f5b98003e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c290 tx=0x7f5b98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0603a6999ce637 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.805+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba402e070 con 0x7f5b98003a00 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.805+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba4005ad0 con 0x7f5b98003a00 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.805+0000 7f5bab7fe640 1 -- 172.21.15.46:0/270994628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4007d50 con 0x7f5b98003a00 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/270994628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b98003a00 msgr2=0x7f5b98003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/270994628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b98003a00 0x7f5b98003e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c290 tx=0x7f5b98005830 comp rx=0 tx=0).stop 2024-05-09T03:09:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb10fb640 1 -- 172.21.15.46:0/270994628 reap_dead start 2024-05-09T03:09:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/270994628 shutdown_connections 2024-05-09T03:09:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/270994628 >> 172.21.15.46:0/270994628 conn(0x7f5bac08d3f0 msgr2=0x7f5bac08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/270994628 shutdown_connections 2024-05-09T03:09:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.806+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/270994628 wait complete. 2024-05-09T03:09:55.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.808+0000 7f5bb2b5e640 1 Processor -- start 2024-05-09T03:09:55.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.808+0000 7f5bb2b5e640 1 -- start start 2024-05-09T03:09:55.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb2b5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b98003a00 0x7f5bac1471c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb2b5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bac0c8cd0 0x7f5bac159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb2b5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 0x7f5bac151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb2b5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0d4200 con 0x7f5bac159760 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb2b5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5bac0d4d80 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb2b5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5bac0d5580 con 0x7f5b98003a00 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb10fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 0x7f5bac151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb10fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 0x7f5bac151e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55372/0 (socket says 172.21.15.46:55372) 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb10fb640 1 -- 172.21.15.46:0/3060242728 learned_addr learned my addr 172.21.15.46:0/3060242728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5babfff640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bac0c8cd0 0x7f5bac159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.809+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b98003a00 0x7f5bac1471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5babfff640 1 -- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b98003a00 msgr2=0x7f5bac1471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5babfff640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b98003a00 0x7f5bac1471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5babfff640 1 -- 172.21.15.46:0/3060242728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 msgr2=0x7f5bac151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:55.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5babfff640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 0x7f5bac151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5babfff640 1 -- 172.21.15.46:0/3060242728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bac159b70 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5bb10fb640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 0x7f5bac151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5babfff640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bac0c8cd0 0x7f5bac159220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b94011960 tx=0x7f5b94011e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94019070 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b94004300 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bac0c90e0 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9400f9e0 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.810+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bac15b730 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.811+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bac06a270 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.812+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5b9401c070 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.812+0000 7f5ba97fa640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b8c0762d0 0x7f5b8c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:55.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.812+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5b94098360 con 0x7f5bac0c8cd0 2024-05-09T03:09:55.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.812+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b8c0762d0 0x7f5b8c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:55.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.813+0000 7f5bb08fa640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b8c0762d0 0x7f5b8c0787e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c290 tx=0x7f5ba402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:55.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:55.817+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5bac159b70 con 0x7f5bac0c8cd0 2024-05-09T03:09:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.006+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5bac0d0f00 con 0x7f5bac0c8cd0 2024-05-09T03:09:56.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.007+0000 7f5ba97fa640 1 -- 172.21.15.46:0/3060242728 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f5b94061540 con 0x7f5bac0c8cd0 2024-05-09T03:09:56.008 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-05-09T03:09:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b8c0762d0 msgr2=0x7f5b8c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b8c0762d0 0x7f5b8c0787e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c290 tx=0x7f5ba402e000 comp rx=0 tx=0).stop 2024-05-09T03:09:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bac0c8cd0 msgr2=0x7f5bac159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bac0c8cd0 0x7f5bac159220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5b94011960 tx=0x7f5b94011e20 comp rx=0 tx=0).stop 2024-05-09T03:09:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 shutdown_connections 2024-05-09T03:09:56.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b8c0762d0 0x7f5b8c0787e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bac159760 0x7f5bac151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5bac0c8cd0 0x7f5bac159220 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 --2- 172.21.15.46:0/3060242728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b98003a00 0x7f5bac1471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 >> 172.21.15.46:0/3060242728 conn(0x7f5bac08d3f0 msgr2=0x7f5bac0d1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 shutdown_connections 2024-05-09T03:09:56.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.009+0000 7f5bb2b5e640 1 -- 172.21.15.46:0/3060242728 wait complete. 2024-05-09T03:09:56.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-05-09T03:09:56.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.135+0000 7fe2fced9640 1 Processor -- start 2024-05-09T03:09:56.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.135+0000 7fe2fced9640 1 -- start start 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2fced9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2f80b92e0 0x7fe2f80b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2fced9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2f80d0480 0x7fe2f80d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2fced9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f80d0dd0 con 0x7fe2f80ce6a0 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2fced9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f80d2b80 con 0x7fe2f80d0480 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2fced9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f80d2e60 con 0x7fe2f80b92e0 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2f5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2f80d0480 0x7fe2f80d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2f6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2f80ce6a0 0x7fe2f80ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43304/0 (socket says 172.21.15.46:43304) 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2f6d9d640 1 -- 172.21.15.46:0/2425987380 learned_addr learned my addr 172.21.15.46:0/2425987380 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.137+0000 7fe2f659c640 1 --2- 172.21.15.46:0/2425987380 >> v2:172.21.15.88:3300/0 conn(0x7fe2f80b92e0 0x7fe2f80b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f5d9b640 1 -- 172.21.15.46:0/2425987380 >> v2:172.21.15.88:3300/0 conn(0x7fe2f80b92e0 msgr2=0x7fe2f80b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f5d9b640 1 --2- 172.21.15.46:0/2425987380 >> v2:172.21.15.88:3300/0 conn(0x7fe2f80b92e0 0x7fe2f80b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f5d9b640 1 -- 172.21.15.46:0/2425987380 >> v1:172.21.15.46:6789/0 conn(0x7fe2f80ce6a0 legacy=0x7fe2f80ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f5d9b640 1 -- 172.21.15.46:0/2425987380 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f80d3140 con 0x7fe2f80d0480 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f559a640 1 -- 172.21.15.46:0/2425987380 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3972977941 0 0) 0x7fe2f0001110 con 0x7fe2f80ce6a0 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f5d9b640 1 --2- 172.21.15.46:0/2425987380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f80d0890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2f806d120 tx=0x7fe2e002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde0f93902940c07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f559a640 1 -- 172.21.15.46:0/2425987380 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0037070 con 0x7fe2f80d0480 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f559a640 1 -- 172.21.15.46:0/2425987380 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e0032440 con 0x7fe2f80d0480 2024-05-09T03:09:56.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.138+0000 7fe2f559a640 1 -- 172.21.15.46:0/2425987380 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e002ae60 con 0x7fe2f80d0480 2024-05-09T03:09:56.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 -- 172.21.15.46:0/2425987380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 msgr2=0x7fe2f80d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2425987380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f80d0890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe2f806d120 tx=0x7fe2e002a110 comp rx=0 tx=0).stop 2024-05-09T03:09:56.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 -- 172.21.15.46:0/2425987380 shutdown_connections 2024-05-09T03:09:56.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2425987380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f80d0890 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2425987380 >> v2:172.21.15.88:3300/0 conn(0x7fe2f80b92e0 0x7fe2f80b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 -- 172.21.15.46:0/2425987380 >> 172.21.15.46:0/2425987380 conn(0x7fe2f80d3c00 msgr2=0x7fe2f80d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:56.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 -- 172.21.15.46:0/2425987380 shutdown_connections 2024-05-09T03:09:56.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.139+0000 7fe2fced9640 1 -- 172.21.15.46:0/2425987380 wait complete. 2024-05-09T03:09:56.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.141+0000 7fe2fced9640 1 Processor -- start 2024-05-09T03:09:56.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.141+0000 7fe2fced9640 1 -- start start 2024-05-09T03:09:56.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.142+0000 7fe2fced9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2f80b92e0 0x7fe2f80cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.142+0000 7fe2fced9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2f80ce6a0 0x7fe2f81fa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2fced9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f81f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2fced9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f806e920 con 0x7fe2f80d0480 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2fced9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f806e7a0 con 0x7fe2f80ce6a0 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2fced9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f806eaa0 con 0x7fe2f80b92e0 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f81f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f81f3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55402/0 (socket says 172.21.15.46:55402) 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 -- 172.21.15.46:0/2898955372 learned_addr learned my addr 172.21.15.46:0/2898955372 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f5d9b640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2f80ce6a0 0x7fe2f81fa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f659c640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2f80b92e0 0x7fe2f80cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 -- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2f80b92e0 msgr2=0x7fe2f80cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2f80b92e0 0x7fe2f80cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 -- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2f80ce6a0 msgr2=0x7fe2f81fa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2f80ce6a0 0x7fe2f81fa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 -- 172.21.15.46:0/2898955372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f80ccac0 con 0x7fe2f80d0480 2024-05-09T03:09:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2f6d9d640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f81f3980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c3f0 tx=0x7fe2f000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f00074e0 con 0x7fe2f80d0480 2024-05-09T03:09:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2f00047e0 con 0x7fe2f80d0480 2024-05-09T03:09:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f80ccda0 con 0x7fe2f80d0480 2024-05-09T03:09:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.143+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0004bd0 con 0x7fe2f80d0480 2024-05-09T03:09:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.144+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f815b4d0 con 0x7fe2f80d0480 2024-05-09T03:09:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.144+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f80cc820 con 0x7fe2f80d0480 2024-05-09T03:09:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.145+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe2f0011070 con 0x7fe2f80d0480 2024-05-09T03:09:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.146+0000 7fe2eeffd640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2d8076300 0x7fe2d8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.146+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe2f0097270 con 0x7fe2f80d0480 2024-05-09T03:09:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.146+0000 7fe2f659c640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2d8076300 0x7fe2d8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.147+0000 7fe2f659c640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2d8076300 0x7fe2d8078810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8071390 tx=0x7fe2e800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:56.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.151+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe2f80cc820 con 0x7fe2f80d0480 2024-05-09T03:09:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.418+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) -- 0x7fe2f806a270 con 0x7fe2f80d0480 2024-05-09T03:09:56.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.624+0000 7fe2eeffd640 1 -- 172.21.15.46:0/2898955372 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v43) ==== 137+0+0 (secure 0 0 0) 0x7fe2f80d0dd0 con 0x7fe2f80d0480 2024-05-09T03:09:56.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2d8076300 msgr2=0x7fe2d8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2d8076300 0x7fe2d8078810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8071390 tx=0x7fe2e800b040 comp rx=0 tx=0).stop 2024-05-09T03:09:56.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 msgr2=0x7fe2f81f3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f81f3980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c3f0 tx=0x7fe2f000c8b0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 shutdown_connections 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2d8076300 0x7fe2d8078810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f80d0480 0x7fe2f81f3980 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2f80ce6a0 0x7fe2f81fa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.626+0000 7fe2fced9640 1 --2- 172.21.15.46:0/2898955372 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2f80b92e0 0x7fe2f80cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.627+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 >> 172.21.15.46:0/2898955372 conn(0x7fe2f80d3c00 msgr2=0x7fe2f80b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.627+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 shutdown_connections 2024-05-09T03:09:56.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.627+0000 7fe2fced9640 1 -- 172.21.15.46:0/2898955372 wait complete. 2024-05-09T03:09:56.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T03:09:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.757+0000 7f8a0e47c640 1 Processor -- start 2024-05-09T03:09:56.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.757+0000 7f8a0e47c640 1 -- start start 2024-05-09T03:09:56.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a0e47c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a080bb770 0x7f8a080d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a0e47c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a08164900 con 0x7f8a080d3180 2024-05-09T03:09:56.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a0e47c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a08164be0 con 0x7f8a08163eb0 2024-05-09T03:09:56.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a0e47c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8a080d4bb0 con 0x7f8a080bb770 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a0ca19640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8a080d3180 0x7f8a080b9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43310/0 (socket says 172.21.15.46:43310) 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a0ca19640 1 -- 172.21.15.46:0/129310645 learned_addr learned my addr 172.21.15.46:0/129310645 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a07fff640 1 --2- 172.21.15.46:0/129310645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a080bb770 0x7f8a080d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705124335 0 0) 0x7f8a08164900 con 0x7f8a080d3180 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.758+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89f0003520 con 0x7f8a080d3180 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.759+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821077960 0 0) 0x7f8a08164be0 con 0x7f8a08163eb0 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.759+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a08164900 con 0x7f8a08163eb0 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.759+0000 7f8a07fff640 1 -- 172.21.15.46:0/129310645 >> v1:172.21.15.88:6789/0 conn(0x7f8a08163eb0 legacy=0x7f8a080d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.759+0000 7f8a07fff640 1 -- 172.21.15.46:0/129310645 >> v1:172.21.15.46:6789/0 conn(0x7f8a080d3180 legacy=0x7f8a080b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.759+0000 7f8a07fff640 1 -- 172.21.15.46:0/129310645 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a080d4840 con 0x7f8a080bb770 2024-05-09T03:09:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.759+0000 7f8a07fff640 1 --2- 172.21.15.46:0/129310645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a080bb770 0x7f8a080d5040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f89f80097d0 tx=0x7f89f802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b097b906afa6fb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8037070 con 0x7f8a080bb770 2024-05-09T03:09:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f8032440 con 0x7f8a080bb770 2024-05-09T03:09:56.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a06ffd640 1 -- 172.21.15.46:0/129310645 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8032810 con 0x7f8a080bb770 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 -- 172.21.15.46:0/129310645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a080bb770 msgr2=0x7f8a080d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/129310645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a080bb770 0x7f8a080d5040 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f89f80097d0 tx=0x7f89f802a210 comp rx=0 tx=0).stop 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 -- 172.21.15.46:0/129310645 shutdown_connections 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/129310645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a080bb770 0x7f8a080d5040 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 -- 172.21.15.46:0/129310645 >> 172.21.15.46:0/129310645 conn(0x7f8a0806e370 msgr2=0x7f8a0806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 -- 172.21.15.46:0/129310645 shutdown_connections 2024-05-09T03:09:56.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.760+0000 7f8a0e47c640 1 -- 172.21.15.46:0/129310645 wait complete. 2024-05-09T03:09:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.762+0000 7f8a0e47c640 1 Processor -- start 2024-05-09T03:09:56.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.762+0000 7f8a0e47c640 1 -- start start 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0e47c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 0x7f8a0815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0e47c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a08163eb0 0x7f8a08151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0e47c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a0815eac0 0x7f8a08162520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0e47c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8a080d0bc0 con 0x7f8a080bb770 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0e47c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8a080bbb80 con 0x7f8a0815eac0 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0e47c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8a080d0d40 con 0x7f8a08163eb0 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 0x7f8a0815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 0x7f8a0815ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55408/0 (socket says 172.21.15.46:55408) 2024-05-09T03:09:56.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 -- 172.21.15.46:0/1622645891 learned_addr learned my addr 172.21.15.46:0/1622645891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a0ca19640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a0815eac0 0x7f8a08162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a077fe640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a08163eb0 0x7f8a08151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 -- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a08163eb0 msgr2=0x7f8a08151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a08163eb0 0x7f8a08151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 -- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a0815eac0 msgr2=0x7f8a08162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a0815eac0 0x7f8a08162520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 -- 172.21.15.46:0/1622645891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0815d3b0 con 0x7f8a080bb770 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.764+0000 7f8a07fff640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 0x7f8a0815ca40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89f80097d0 tx=0x7f89f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.765+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8037040 con 0x7f8a080bb770 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.765+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f8004230 con 0x7f8a080bb770 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.765+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0815d690 con 0x7f8a080bb770 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.765+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8003610 con 0x7f8a080bb770 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.765+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a080c9870 con 0x7f8a080bb770 2024-05-09T03:09:56.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.765+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0806a270 con 0x7f8a080bb770 2024-05-09T03:09:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.766+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f89f80037b0 con 0x7f8a080bb770 2024-05-09T03:09:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.767+0000 7f8a04ff9640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89f4076300 0x7f89f4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:56.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.767+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f89f80b80f0 con 0x7f8a080bb770 2024-05-09T03:09:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.767+0000 7f8a077fe640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89f4076300 0x7f89f4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:56.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.768+0000 7f8a077fe640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89f4076300 0x7f89f4078810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f89fc001120 tx=0x7f89fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:56.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.772+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f89f804b070 con 0x7f8a080bb770 2024-05-09T03:09:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.965+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8a080d4130 con 0x7f8a080bb770 2024-05-09T03:09:56.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.969+0000 7f8a04ff9640 1 -- 172.21.15.46:0/1622645891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f89f8081250 con 0x7f8a080bb770 2024-05-09T03:09:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89f4076300 msgr2=0x7f89f4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89f4076300 0x7f89f4078810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f89fc001120 tx=0x7f89fc009040 comp rx=0 tx=0).stop 2024-05-09T03:09:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 msgr2=0x7f8a0815ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 0x7f8a0815ca40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89f80097d0 tx=0x7f89f80046c0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 shutdown_connections 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a0815eac0 0x7f8a08162520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8a08163eb0 0x7f8a08151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89f4076300 0x7f89f4078810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 --2- 172.21.15.46:0/1622645891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a080bb770 0x7f8a0815ca40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 >> 172.21.15.46:0/1622645891 conn(0x7f8a0806e370 msgr2=0x7f8a080b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 shutdown_connections 2024-05-09T03:09:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:56.971+0000 7f8a0e47c640 1 -- 172.21.15.46:0/1622645891 wait complete. 2024-05-09T03:09:56.984 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:09:56.984 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:09:57.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:09:57.072 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:09:57.073 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:57.073 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:57.089 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:57.090 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:57.146 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:09:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.301+0000 7fa63d984640 1 Processor -- start 2024-05-09T03:09:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.301+0000 7fa63d984640 1 -- start start 2024-05-09T03:09:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.302+0000 7fa63d984640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6380d2e90 0x7fa6380b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.302+0000 7fa63d984640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 0x7fa6380b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa63d984640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6380d3400 con 0x7fa6381605b0 2024-05-09T03:09:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa63d984640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa6380d4a60 con 0x7fa6380d2e90 2024-05-09T03:09:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa63d984640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa6380d4d40 con 0x7fa6380b4fc0 2024-05-09T03:09:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa6367fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6381605b0 0x7fa6380d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43322/0 (socket says 172.21.15.46:43322) 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa6367fc640 1 -- 172.21.15.46:0/61941351 learned_addr learned my addr 172.21.15.46:0/61941351 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa636ffd640 1 --2- 172.21.15.46:0/61941351 >> v2:172.21.15.88:3300/0 conn(0x7fa6380d2e90 0x7fa6380b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa6377fe640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 0x7fa6380b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634638936 0 0) 0x7fa6380d3400 con 0x7fa6381605b0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa624003520 con 0x7fa6381605b0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1141979714 0 0) 0x7fa624003520 con 0x7fa6381605b0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 msgr2=0x7fa6380b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 0x7fa6380b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 >> v2:172.21.15.88:3300/0 conn(0x7fa6380d2e90 msgr2=0x7fa6380b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 --2- 172.21.15.46:0/61941351 >> v2:172.21.15.88:3300/0 conn(0x7fa6380d2e90 0x7fa6380b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6380d5020 con 0x7fa6381605b0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fa6200016f0 con 0x7fa6381605b0 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 >> v1:172.21.15.46:6789/0 conn(0x7fa6381605b0 legacy=0x7fa6380d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 0x7fa624003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa624004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa624004d40 0x7fa624005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380d4a60 con 0x7fa624003fd0 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380d4d40 con 0x7fa624004d40 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa62400bac0 con 0x7fa6380b4fc0 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.303+0000 7fa6367fc640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa624004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa636ffd640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 0x7fa624003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6377fe640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa624004d40 0x7fa624005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6367fc640 1 -- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 msgr2=0x7fa624003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6367fc640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6380b4fc0 0x7fa624003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6367fc640 1 -- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa624004d40 msgr2=0x7fa624005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6367fc640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa624004d40 0x7fa624005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6367fc640 1 -- 172.21.15.46:0/61941351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa624003520 con 0x7fa624003fd0 2024-05-09T03:09:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.304+0000 7fa6367fc640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa624004800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa62000c210 tx=0x7fa624005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af51a73b764656e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620007d30 con 0x7fa624003fd0 2024-05-09T03:09:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa620005af0 con 0x7fa624003fd0 2024-05-09T03:09:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa635ffb640 1 -- 172.21.15.46:0/61941351 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620007070 con 0x7fa624003fd0 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa63d984640 1 -- 172.21.15.46:0/61941351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 msgr2=0x7fa624004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa63d984640 1 --2- 172.21.15.46:0/61941351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa624004800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa62000c210 tx=0x7fa624005830 comp rx=0 tx=0).stop 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa6377fe640 1 -- 172.21.15.46:0/61941351 reap_dead start 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa63d984640 1 -- 172.21.15.46:0/61941351 shutdown_connections 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa63d984640 1 -- 172.21.15.46:0/61941351 >> 172.21.15.46:0/61941351 conn(0x7fa63808d390 msgr2=0x7fa63808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa63d984640 1 -- 172.21.15.46:0/61941351 shutdown_connections 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.305+0000 7fa63d984640 1 -- 172.21.15.46:0/61941351 wait complete. 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.307+0000 7fa63d984640 1 Processor -- start 2024-05-09T03:09:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.307+0000 7fa63d984640 1 -- start start 2024-05-09T03:09:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa6381fa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6381fa8c0 0x7fa6381560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa638156600 0x7fa638162010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380d5540 con 0x7fa624003fd0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380b4c00 con 0x7fa6381fa8c0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa638143d50 con 0x7fa638156600 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa6381fa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa6381fa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55434/0 (socket says 172.21.15.46:55434) 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 -- 172.21.15.46:0/1506511398 learned_addr learned my addr 172.21.15.46:0/1506511398 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa6367fc640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6381fa8c0 0x7fa6381560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 -- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa638156600 msgr2=0x7fa638162010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa6377fe640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa638156600 0x7fa638162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa638156600 0x7fa638162010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 -- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6381fa8c0 msgr2=0x7fa6381560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6381fa8c0 0x7fa6381560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 -- 172.21.15.46:0/1506511398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa638157390 con 0x7fa624003fd0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa636ffd640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa6381fa380 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa628011930 tx=0x7fa628011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62800faf0 con 0x7fa624003fd0 2024-05-09T03:09:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62800fc90 con 0x7fa624003fd0 2024-05-09T03:09:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa638156dd0 con 0x7fa624003fd0 2024-05-09T03:09:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.309+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628004030 con 0x7fa624003fd0 2024-05-09T03:09:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.310+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa638161810 con 0x7fa624003fd0 2024-05-09T03:09:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.310+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6380b8c60 con 0x7fa624003fd0 2024-05-09T03:09:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.311+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa628012040 con 0x7fa624003fd0 2024-05-09T03:09:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.312+0000 7fa62ffff640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa618076300 0x7fa618078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.312+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa628097880 con 0x7fa624003fd0 2024-05-09T03:09:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.312+0000 7fa6367fc640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa618076300 0x7fa618078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.313+0000 7fa6367fc640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa618076300 0x7fa618078810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa620032000 tx=0x7fa620009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.316+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa6380b8c60 con 0x7fa624003fd0 2024-05-09T03:09:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.517+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa6380b54c0 con 0x7fa624003fd0 2024-05-09T03:09:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.518+0000 7fa62ffff640 1 -- 172.21.15.46:0/1506511398 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7fa6380b54c0 con 0x7fa624003fd0 2024-05-09T03:09:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa618076300 msgr2=0x7fa618078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa618076300 0x7fa618078810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa620032000 tx=0x7fa620009030 comp rx=0 tx=0).stop 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 msgr2=0x7fa6381fa380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa6381fa380 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa628011930 tx=0x7fa628011df0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 shutdown_connections 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa638156600 0x7fa638162010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6381fa8c0 0x7fa6381560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa618076300 0x7fa618078810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 --2- 172.21.15.46:0/1506511398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa624003fd0 0x7fa6381fa380 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 >> 172.21.15.46:0/1506511398 conn(0x7fa63808d390 msgr2=0x7fa6380b6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 shutdown_connections 2024-05-09T03:09:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.520+0000 7fa63d984640 1 -- 172.21.15.46:0/1506511398 wait complete. 2024-05-09T03:09:57.532 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:57.532 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:57.548 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:09:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.705+0000 7f320e5a8640 1 Processor -- start 2024-05-09T03:09:57.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.706+0000 7f320e5a8640 1 -- start start 2024-05-09T03:09:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f320e5a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3208163eb0 0x7f32081642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f320e5a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3208164900 con 0x7f32080d2a70 2024-05-09T03:09:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f320e5a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3208164be0 con 0x7f32080bb770 2024-05-09T03:09:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f320e5a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f32080d4bb0 con 0x7f3208163eb0 2024-05-09T03:09:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f32077fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3208163eb0 0x7f32081642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f32077fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3208163eb0 0x7f32081642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55440/0 (socket says 172.21.15.46:55440) 2024-05-09T03:09:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f32077fe640 1 -- 172.21.15.46:0/1812153232 learned_addr learned my addr 172.21.15.46:0/1812153232 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f32077fe640 1 -- 172.21.15.46:0/1812153232 >> v1:172.21.15.88:6789/0 conn(0x7f32080bb770 legacy=0x7f32080d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f32077fe640 1 -- 172.21.15.46:0/1812153232 >> v1:172.21.15.46:6789/0 conn(0x7f32080d2a70 legacy=0x7f32080d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.707+0000 7f32077fe640 1 -- 172.21.15.46:0/1812153232 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32080d4840 con 0x7f3208163eb0 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.708+0000 7f32077fe640 1 --2- 172.21.15.46:0/1812153232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32081642c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009870 tx=0x7f31fc02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f6b156e211e000 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.708+0000 7f3206ffd640 1 -- 172.21.15.46:0/1812153232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc037070 con 0x7f3208163eb0 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.708+0000 7f3206ffd640 1 -- 172.21.15.46:0/1812153232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc02ae50 con 0x7f3208163eb0 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.708+0000 7f3206ffd640 1 -- 172.21.15.46:0/1812153232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc032860 con 0x7f3208163eb0 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 -- 172.21.15.46:0/1812153232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 msgr2=0x7f32081642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 --2- 172.21.15.46:0/1812153232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32081642c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009870 tx=0x7f31fc02a300 comp rx=0 tx=0).stop 2024-05-09T03:09:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 -- 172.21.15.46:0/1812153232 shutdown_connections 2024-05-09T03:09:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 --2- 172.21.15.46:0/1812153232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32081642c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 -- 172.21.15.46:0/1812153232 >> 172.21.15.46:0/1812153232 conn(0x7f320806e370 msgr2=0x7f320806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 -- 172.21.15.46:0/1812153232 shutdown_connections 2024-05-09T03:09:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.709+0000 7f320e5a8640 1 -- 172.21.15.46:0/1812153232 wait complete. 2024-05-09T03:09:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.711+0000 7f320e5a8640 1 Processor -- start 2024-05-09T03:09:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.711+0000 7f320e5a8640 1 -- start start 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320e5a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32080bb770 0x7f32081eae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320e5a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32080d2a70 0x7f320814b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320e5a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32080ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320e5a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f32080d0d40 con 0x7f3208163eb0 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320e5a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32080d0bc0 con 0x7f32080d2a70 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320e5a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32080b9210 con 0x7f32080bb770 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.712+0000 7f320cb45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32080ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32080ce7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55446/0 (socket says 172.21.15.46:55446) 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 -- 172.21.15.46:0/426273078 learned_addr learned my addr 172.21.15.46:0/426273078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f32077fe640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32080d2a70 0x7f320814b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f3207fff640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32080bb770 0x7f32081eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 -- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32080bb770 msgr2=0x7f32081eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32080bb770 0x7f32081eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 -- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32080d2a70 msgr2=0x7f320814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32080d2a70 0x7f320814b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 -- 172.21.15.46:0/426273078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32081eb7f0 con 0x7f3208163eb0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320cb45640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32080ce7d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3200002230 tx=0x7f320000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200003b70 con 0x7f3208163eb0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3200003d10 con 0x7f3208163eb0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32081ebad0 con 0x7f3208163eb0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32000052d0 con 0x7f3208163eb0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.713+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32081ec090 con 0x7f3208163eb0 2024-05-09T03:09:57.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.714+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320806a270 con 0x7f3208163eb0 2024-05-09T03:09:57.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.715+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3200005680 con 0x7f3208163eb0 2024-05-09T03:09:57.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.716+0000 7f3204ff9640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31f40762d0 0x7f31f40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:57.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.716+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3200097690 con 0x7f3208163eb0 2024-05-09T03:09:57.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.716+0000 7f3207fff640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31f40762d0 0x7f31f40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:57.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.716+0000 7f3207fff640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31f40762d0 0x7f31f40787e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f32080b8880 tx=0x7f31f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.721+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3200060870 con 0x7f3208163eb0 2024-05-09T03:09:57.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.921+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f32080d4130 con 0x7f3208163eb0 2024-05-09T03:09:57.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.922+0000 7f3204ff9640 1 -- 172.21.15.46:0/426273078 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f3200064b70 con 0x7f3208163eb0 2024-05-09T03:09:57.922 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31f40762d0 msgr2=0x7f31f40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31f40762d0 0x7f31f40787e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f32080b8880 tx=0x7f31f8009040 comp rx=0 tx=0).stop 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 msgr2=0x7f32080ce7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32080ce7d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3200002230 tx=0x7f320000cb90 comp rx=0 tx=0).stop 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 shutdown_connections 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31f40762d0 0x7f31f40787e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3208163eb0 0x7f32080ce7d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32080d2a70 0x7f320814b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 --2- 172.21.15.46:0/426273078 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32080bb770 0x7f32081eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 >> 172.21.15.46:0/426273078 conn(0x7f320806e370 msgr2=0x7f32080b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 shutdown_connections 2024-05-09T03:09:57.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:57.924+0000 7f320e5a8640 1 -- 172.21.15.46:0/426273078 wait complete. 2024-05-09T03:09:57.936 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:09:57.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:57.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:09:57.963 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:09:57.963 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:09:57.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:57.964 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:09:58.022 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:58.022 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:58.043 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:58.043 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:58.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:09:58.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.218+0000 7f1a82b15640 1 Processor -- start 2024-05-09T03:09:58.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.218+0000 7f1a82b15640 1 -- start start 2024-05-09T03:09:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a82b15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 0x7f1a7c0d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a82b15640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a7c0b7760 0x7f1a7c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a82b15640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a7c0b80b0 con 0x7f1a7c0d38b0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a82b15640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1a7c0d4a60 con 0x7f1a7c0b7760 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a82b15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7c0d4d40 con 0x7f1a7c0d2eb0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a810b2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1a7c0d38b0 0x7f1a7c0d3cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43338/0 (socket says 172.21.15.46:43338) 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a810b2640 1 -- 172.21.15.46:0/1763185595 learned_addr learned my addr 172.21.15.46:0/1763185595 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a808b1640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 0x7f1a7c0d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7bfff640 1 --2- 172.21.15.46:0/1763185595 >> v2:172.21.15.88:3300/0 conn(0x7f1a7c0b7760 0x7f1a7c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270885143 0 0) 0x7f1a74001110 con 0x7f1a7c0d38b0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a64003520 con 0x7f1a7c0d38b0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1545177915 0 0) 0x7f1a64003520 con 0x7f1a7c0d38b0 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 msgr2=0x7f1a7c0d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 0x7f1a7c0d32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 >> v2:172.21.15.88:3300/0 conn(0x7f1a7c0b7760 msgr2=0x7f1a7c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 --2- 172.21.15.46:0/1763185595 >> v2:172.21.15.88:3300/0 conn(0x7f1a7c0b7760 0x7f1a7c0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c0d5020 con 0x7f1a7c0d38b0 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f1a740018c0 con 0x7f1a7c0d38b0 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 >> v1:172.21.15.46:6789/0 conn(0x7f1a7c0d38b0 legacy=0x7f1a7c0d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a64003a00 0x7f1a64003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.220+0000 7f1a7b7fe640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a64004800 0x7f1a64004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a7b7fe640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 0x7f1a64005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0d4a60 con 0x7f1a64003a00 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0d4d40 con 0x7f1a64004800 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a6400ba70 con 0x7f1a7c0d2eb0 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a808b1640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a64003a00 0x7f1a64003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a7bfff640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a64004800 0x7f1a64004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a810b2640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 0x7f1a64005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a808b1640 1 -- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 msgr2=0x7f1a64005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a808b1640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0d2eb0 0x7f1a64005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a808b1640 1 -- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a64004800 msgr2=0x7f1a64004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a808b1640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a64004800 0x7f1a64004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.221+0000 7f1a808b1640 1 -- 172.21.15.46:0/1763185595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a64003520 con 0x7f1a64003a00 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a808b1640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a64003a00 0x7f1a64003e70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009f90 tx=0x7f1a64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35ba471bda6a41f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c009c10 con 0x7f1a64003a00 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6c036070 con 0x7f1a64003a00 2024-05-09T03:09:58.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a7b7fe640 1 -- 172.21.15.46:0/1763185595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c031810 con 0x7f1a64003a00 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a82b15640 1 -- 172.21.15.46:0/1763185595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a64003a00 msgr2=0x7f1a64003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a82b15640 1 --2- 172.21.15.46:0/1763185595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a64003a00 0x7f1a64003e70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009f90 tx=0x7f1a64005830 comp rx=0 tx=0).stop 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a810b2640 1 -- 172.21.15.46:0/1763185595 reap_dead start 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a82b15640 1 -- 172.21.15.46:0/1763185595 shutdown_connections 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.222+0000 7f1a82b15640 1 -- 172.21.15.46:0/1763185595 >> 172.21.15.46:0/1763185595 conn(0x7f1a7c08d390 msgr2=0x7f1a7c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.223+0000 7f1a82b15640 1 -- 172.21.15.46:0/1763185595 shutdown_connections 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.223+0000 7f1a82b15640 1 -- 172.21.15.46:0/1763185595 wait complete. 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.224+0000 7f1a82b15640 1 Processor -- start 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.224+0000 7f1a82b15640 1 -- start start 2024-05-09T03:09:58.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a82b15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a7c154700 0x7f1a7c0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a82b15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0cb280 0x7f1a7c0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a82b15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 0x7f1a7c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a82b15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0b51e0 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a82b15640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7c0d55e0 con 0x7f1a7c154700 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a82b15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a7c0b5360 con 0x7f1a7c0cb280 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 0x7f1a7c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 0x7f1a7c1558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55458/0 (socket says 172.21.15.46:55458) 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 -- 172.21.15.46:0/2717719017 learned_addr learned my addr 172.21.15.46:0/2717719017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a7bfff640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0cb280 0x7f1a7c0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a808b1640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a7c154700 0x7f1a7c0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 -- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0cb280 msgr2=0x7f1a7c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0cb280 0x7f1a7c0ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 -- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a7c154700 msgr2=0x7f1a7c0cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a7c154700 0x7f1a7c0cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.226+0000 7f1a810b2640 1 -- 172.21.15.46:0/2717719017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a7c0cb790 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.227+0000 7f1a810b2640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 0x7f1a7c1558b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1a74001eb0 tx=0x7f1a74007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.227+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a74001180 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.227+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a74018070 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.227+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7400fb30 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.227+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a7c0cba70 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.227+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a7c1565a0 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.228+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a7c06a270 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.229+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a7400fcd0 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.230+0000 7f1a797fa640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a680762d0 0x7f1a680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.230+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1a74097460 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.230+0000 7f1a808b1640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a680762d0 0x7f1a680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.230+0000 7f1a808b1640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a680762d0 0x7f1a680787e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009f90 tx=0x7f1a6c02f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:58.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.234+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1a7c0b80b0 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.437+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1a7c0b4e00 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.438+0000 7f1a797fa640 1 -- 172.21.15.46:0/2717719017 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f1a74060f10 con 0x7f1a7c0cd2d0 2024-05-09T03:09:58.439 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a680762d0 msgr2=0x7f1a680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a680762d0 0x7f1a680787e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009f90 tx=0x7f1a6c02f000 comp rx=0 tx=0).stop 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 msgr2=0x7f1a7c1558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 0x7f1a7c1558b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1a74001eb0 tx=0x7f1a74007660 comp rx=0 tx=0).stop 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 shutdown_connections 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a680762d0 0x7f1a680787e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a7c0cd2d0 0x7f1a7c1558b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a7c0cb280 0x7f1a7c0ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 --2- 172.21.15.46:0/2717719017 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a7c154700 0x7f1a7c0cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 >> 172.21.15.46:0/2717719017 conn(0x7f1a7c08d390 msgr2=0x7f1a7c1624a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 shutdown_connections 2024-05-09T03:09:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.440+0000 7f1a82b15640 1 -- 172.21.15.46:0/2717719017 wait complete. 2024-05-09T03:09:58.452 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:58.452 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:58.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:09:58.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.586+0000 7f1d30e8b640 1 Processor -- start 2024-05-09T03:09:58.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.586+0000 7f1d30e8b640 1 -- start start 2024-05-09T03:09:58.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d30e8b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d2c0d3fc0 0x7f1d2c16b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d30e8b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2c0b5550 con 0x7f1d2c0b5140 2024-05-09T03:09:58.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d30e8b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2c0d44e0 con 0x7f1d2c0b7e70 2024-05-09T03:09:58.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d30e8b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1d2c0d3480 con 0x7f1d2c0d3fc0 2024-05-09T03:09:58.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d2ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d2c0d3fc0 0x7f1d2c16b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d2ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d2c0d3fc0 0x7f1d2c16b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55470/0 (socket says 172.21.15.46:55470) 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/1101144614 learned_addr learned my addr 172.21.15.46:0/1101144614 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/1101144614 >> v1:172.21.15.88:6789/0 conn(0x7f1d2c0b7e70 legacy=0x7f1d2c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/1101144614 >> v1:172.21.15.46:6789/0 conn(0x7f1d2c0b5140 legacy=0x7f1d2c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.588+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/1101144614 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c0d47c0 con 0x7f1d2c0d3fc0 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.589+0000 7f1d2ad9d640 1 --2- 172.21.15.46:0/1101144614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c0d3fc0 0x7f1d2c16b0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1d24009e50 tx=0x7f1d2402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454bbb45c8ba0044 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.589+0000 7f1d2959a640 1 -- 172.21.15.46:0/1101144614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24036070 con 0x7f1d2c0d3fc0 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.589+0000 7f1d2959a640 1 -- 172.21.15.46:0/1101144614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d24031600 con 0x7f1d2c0d3fc0 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.589+0000 7f1d2959a640 1 -- 172.21.15.46:0/1101144614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d240319d0 con 0x7f1d2c0d3fc0 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 -- 172.21.15.46:0/1101144614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c0d3fc0 msgr2=0x7f1d2c16b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/1101144614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c0d3fc0 0x7f1d2c16b0d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1d24009e50 tx=0x7f1d2402a7a0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 -- 172.21.15.46:0/1101144614 shutdown_connections 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/1101144614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c0d3fc0 0x7f1d2c16b0d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 -- 172.21.15.46:0/1101144614 >> 172.21.15.46:0/1101144614 conn(0x7f1d2c08d3f0 msgr2=0x7f1d2c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:58.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 -- 172.21.15.46:0/1101144614 shutdown_connections 2024-05-09T03:09:58.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.590+0000 7f1d30e8b640 1 -- 172.21.15.46:0/1101144614 wait complete. 2024-05-09T03:09:58.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.592+0000 7f1d30e8b640 1 Processor -- start 2024-05-09T03:09:58.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.592+0000 7f1d30e8b640 1 -- start start 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d30e8b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d2c0b5140 0x7f1d2c1f9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d30e8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d2c0d3fc0 0x7f1d2c1f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d30e8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 0x7f1d2c1539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d30e8b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c0d5580 con 0x7f1d2c143000 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d30e8b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2c0d4d80 con 0x7f1d2c0b5140 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d30e8b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2c0d5700 con 0x7f1d2c0d3fc0 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.593+0000 7f1d2ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 0x7f1d2c1539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 0x7f1d2c1539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55478/0 (socket says 172.21.15.46:55478) 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/2767552964 learned_addr learned my addr 172.21.15.46:0/2767552964 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:58.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2a59c640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d2c0b5140 0x7f1d2c1f9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d29d9b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d2c0d3fc0 0x7f1d2c1f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d2c0d3fc0 msgr2=0x7f1d2c1f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d2c0d3fc0 0x7f1d2c1f9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d2c0b5140 msgr2=0x7f1d2c1f9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d2c0b5140 0x7f1d2c1f9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 -- 172.21.15.46:0/2767552964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2c15a0b0 con 0x7f1d2c143000 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d2ad9d640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 0x7f1d2c1539f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1d24009f70 tx=0x7f1d24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24041020 con 0x7f1d2c143000 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d24007800 con 0x7f1d2c143000 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2c15d710 con 0x7f1d2c143000 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24036040 con 0x7f1d2c143000 2024-05-09T03:09:58.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.594+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2c15dcd0 con 0x7f1d2c143000 2024-05-09T03:09:58.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.595+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d2c15cef0 con 0x7f1d2c143000 2024-05-09T03:09:58.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.596+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1d24044020 con 0x7f1d2c143000 2024-05-09T03:09:58.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.597+0000 7f1d1affd640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1d0c076300 0x7f1d0c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:58.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.597+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1d240b8580 con 0x7f1d2c143000 2024-05-09T03:09:58.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.597+0000 7f1d2a59c640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1d0c076300 0x7f1d0c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:58.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.597+0000 7f1d2a59c640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1d0c076300 0x7f1d0c078810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0015e0 tx=0x7f1d1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:58.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.601+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1d2404b030 con 0x7f1d2c143000 2024-05-09T03:09:58.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.809+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1d2c06a270 con 0x7f1d2c143000 2024-05-09T03:09:58.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.809+0000 7f1d1affd640 1 -- 172.21.15.46:0/2767552964 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f1d24081dd0 con 0x7f1d2c143000 2024-05-09T03:09:58.810 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:09:58.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1d0c076300 msgr2=0x7f1d0c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1d0c076300 0x7f1d0c078810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0015e0 tx=0x7f1d1c009040 comp rx=0 tx=0).stop 2024-05-09T03:09:58.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 msgr2=0x7f1d2c1539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 0x7f1d2c1539f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1d24009f70 tx=0x7f1d24004a40 comp rx=0 tx=0).stop 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 shutdown_connections 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1d0c076300 0x7f1d0c078810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c143000 0x7f1d2c1539f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1d2c0d3fc0 0x7f1d2c1f9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 --2- 172.21.15.46:0/2767552964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d2c0b5140 0x7f1d2c1f9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.811+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 >> 172.21.15.46:0/2767552964 conn(0x7f1d2c08d3f0 msgr2=0x7f1d2c0b7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.812+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 shutdown_connections 2024-05-09T03:09:58.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:58.812+0000 7f1d30e8b640 1 -- 172.21.15.46:0/2767552964 wait complete. 2024-05-09T03:09:58.823 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi088.front.sepia.ceph.com... 2024-05-09T03:09:58.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:58.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-05-09T03:09:58.855 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:09:58.855 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:09:58.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:58.856 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-05-09T03:09:59.008 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:59.008 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:59.025 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:59.025 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:59.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:09:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.238+0000 7feb6ccf9640 1 Processor -- start 2024-05-09T03:09:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.238+0000 7feb6ccf9640 1 -- start start 2024-05-09T03:09:59.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb6ccf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b4c20 0x7feb680cd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb6ccf9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb680b26b0 0x7feb680cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb6ccf9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb680b41d0 con 0x7feb680b22a0 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb6ccf9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7feb6806f920 con 0x7feb680b26b0 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb6ccf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7feb6806ed10 con 0x7feb680b4c20 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb66d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb680b26b0 0x7feb680cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb66d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb680b26b0 0x7feb680cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55488/0 (socket says 172.21.15.46:55488) 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.239+0000 7feb66d9d640 1 -- 172.21.15.46:0/68025614 learned_addr learned my addr 172.21.15.46:0/68025614 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.240+0000 7feb5ffff640 1 --2- 172.21.15.46:0/68025614 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b4c20 0x7feb680cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.240+0000 7feb66d9d640 1 -- 172.21.15.46:0/68025614 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b4c20 msgr2=0x7feb680cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.240+0000 7feb66d9d640 1 --2- 172.21.15.46:0/68025614 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b4c20 0x7feb680cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.240+0000 7feb66d9d640 1 -- 172.21.15.46:0/68025614 >> v1:172.21.15.88:6789/0 conn(0x7feb680b22a0 legacy=0x7feb680b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.240+0000 7feb66d9d640 1 -- 172.21.15.46:0/68025614 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6806eff0 con 0x7feb680b26b0 2024-05-09T03:09:59.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.240+0000 7feb66d9d640 1 --2- 172.21.15.46:0/68025614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb680cd890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7feb60009e50 tx=0x7feb6002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3911885477d6ac1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb65d9b640 1 -- 172.21.15.46:0/68025614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60036070 con 0x7feb680b26b0 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb65d9b640 1 -- 172.21.15.46:0/68025614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb60031600 con 0x7feb680b26b0 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb65d9b640 1 -- 172.21.15.46:0/68025614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600319d0 con 0x7feb680b26b0 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 -- 172.21.15.46:0/68025614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 msgr2=0x7feb680cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/68025614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb680cd890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7feb60009e50 tx=0x7feb6002a710 comp rx=0 tx=0).stop 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 -- 172.21.15.46:0/68025614 shutdown_connections 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/68025614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb680cd890 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/68025614 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b4c20 0x7feb680cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 -- 172.21.15.46:0/68025614 >> 172.21.15.46:0/68025614 conn(0x7feb6809b7d0 msgr2=0x7feb6809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 -- 172.21.15.46:0/68025614 shutdown_connections 2024-05-09T03:09:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.241+0000 7feb6ccf9640 1 -- 172.21.15.46:0/68025614 wait complete. 2024-05-09T03:09:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.243+0000 7feb6ccf9640 1 Processor -- start 2024-05-09T03:09:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.243+0000 7feb6ccf9640 1 -- start start 2024-05-09T03:09:59.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6ccf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b22a0 0x7feb6815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6ccf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb6814b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6ccf9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb6814b660 0x7feb68071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6ccf9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb6806ea00 con 0x7feb680b26b0 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6ccf9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7feb6806f5b0 con 0x7feb6814b660 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6ccf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7feb680d3460 con 0x7feb680b22a0 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb6814b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb6814b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55498/0 (socket says 172.21.15.46:55498) 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 -- 172.21.15.46:0/2115926945 learned_addr learned my addr 172.21.15.46:0/2115926945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb6659c640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b22a0 0x7feb6815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb66d9d640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb6814b660 0x7feb68071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 -- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b22a0 msgr2=0x7feb6815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b22a0 0x7feb6815a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 -- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb6814b660 msgr2=0x7feb68071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb6814b660 0x7feb68071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 -- 172.21.15.46:0/2115926945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb680cab10 con 0x7feb680b26b0 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.245+0000 7feb5ffff640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb6814b120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb5000d5f0 tx=0x7feb5000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.246+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb500125f0 con 0x7feb680b26b0 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.246+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb50019070 con 0x7feb680b26b0 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.246+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb68149000 con 0x7feb680b26b0 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.246+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb500129d0 con 0x7feb680b26b0 2024-05-09T03:09:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.246+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb681495c0 con 0x7feb680b26b0 2024-05-09T03:09:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.247+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb68070ea0 con 0x7feb680b26b0 2024-05-09T03:09:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.247+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feb50012b70 con 0x7feb680b26b0 2024-05-09T03:09:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.248+0000 7feb5effd640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb480762d0 0x7feb480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.248+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7feb500978a0 con 0x7feb680b26b0 2024-05-09T03:09:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.248+0000 7feb6659c640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb480762d0 0x7feb480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.249+0000 7feb6659c640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb480762d0 0x7feb480787e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feb68073fa0 tx=0x7feb58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.253+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7feb50060bb0 con 0x7feb680b26b0 2024-05-09T03:09:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.455+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7feb6814bbe0 con 0x7feb680b26b0 2024-05-09T03:09:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.455+0000 7feb5effd640 1 -- 172.21.15.46:0/2115926945 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7feb50064eb0 con 0x7feb680b26b0 2024-05-09T03:09:59.456 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:09:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb480762d0 msgr2=0x7feb480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb480762d0 0x7feb480787e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7feb68073fa0 tx=0x7feb58009040 comp rx=0 tx=0).stop 2024-05-09T03:09:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 msgr2=0x7feb6814b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb6814b120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb5000d5f0 tx=0x7feb5000dab0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 shutdown_connections 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb6814b660 0x7feb68071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb480762d0 0x7feb480787e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb680b26b0 0x7feb6814b120 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 --2- 172.21.15.46:0/2115926945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb680b22a0 0x7feb6815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.457+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 >> 172.21.15.46:0/2115926945 conn(0x7feb6809b7d0 msgr2=0x7feb680cf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.458+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 shutdown_connections 2024-05-09T03:09:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.458+0000 7feb6ccf9640 1 -- 172.21.15.46:0/2115926945 wait complete. 2024-05-09T03:09:59.470 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:09:59.470 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:59.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:09:59.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.641+0000 7fcd0c4e9640 1 Processor -- start 2024-05-09T03:09:59.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.642+0000 7fcd0c4e9640 1 -- start start 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0c4e9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd04163eb0 0x7fcd041642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0c4e9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd040d2a70 0x7fcd040d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0c4e9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd040d33c0 con 0x7fcd040bb770 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0c4e9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcd040d4bb0 con 0x7fcd040d2a70 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0c4e9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcd04164900 con 0x7fcd04163eb0 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0a285640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd040bb770 0x7fcd040d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43346/0 (socket says 172.21.15.46:43346) 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0a285640 1 -- 172.21.15.46:0/2558296740 learned_addr learned my addr 172.21.15.46:0/2558296740 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0aa86640 1 --2- 172.21.15.46:0/2558296740 >> v2:172.21.15.46:3300/0 conn(0x7fcd040d2a70 0x7fcd040d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd09a84640 1 --2- 172.21.15.46:0/2558296740 >> v2:172.21.15.88:3300/0 conn(0x7fcd04163eb0 0x7fcd041642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0aa86640 1 -- 172.21.15.46:0/2558296740 >> v2:172.21.15.88:3300/0 conn(0x7fcd04163eb0 msgr2=0x7fcd041642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0aa86640 1 --2- 172.21.15.46:0/2558296740 >> v2:172.21.15.88:3300/0 conn(0x7fcd04163eb0 0x7fcd041642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.643+0000 7fcd0aa86640 1 -- 172.21.15.46:0/2558296740 >> v1:172.21.15.46:6789/0 conn(0x7fcd040bb770 legacy=0x7fcd040d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.644+0000 7fcd09283640 1 -- 172.21.15.46:0/2558296740 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900884783 0 0) 0x7fccfc001110 con 0x7fcd040bb770 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.644+0000 7fcd0aa86640 1 -- 172.21.15.46:0/2558296740 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd04164be0 con 0x7fcd040d2a70 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.644+0000 7fcd0aa86640 1 --2- 172.21.15.46:0/2558296740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040d2a70 0x7fcd040d2e80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcd040d6d00 tx=0x7fccf402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b86bfae492ff57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.644+0000 7fcd09283640 1 -- 172.21.15.46:0/2558296740 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4007740 con 0x7fcd040d2a70 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.644+0000 7fcd09283640 1 -- 172.21.15.46:0/2558296740 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf40078e0 con 0x7fcd040d2a70 2024-05-09T03:09:59.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.644+0000 7fcd09283640 1 -- 172.21.15.46:0/2558296740 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4007cb0 con 0x7fcd040d2a70 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2558296740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040d2a70 msgr2=0x7fcd040d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2558296740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040d2a70 0x7fcd040d2e80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcd040d6d00 tx=0x7fccf402a680 comp rx=0 tx=0).stop 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2558296740 shutdown_connections 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2558296740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040d2a70 0x7fcd040d2e80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2558296740 >> v2:172.21.15.88:3300/0 conn(0x7fcd04163eb0 0x7fcd041642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2558296740 >> 172.21.15.46:0/2558296740 conn(0x7fcd0406e370 msgr2=0x7fcd0406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:59.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2558296740 shutdown_connections 2024-05-09T03:09:59.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.645+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2558296740 wait complete. 2024-05-09T03:09:59.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.647+0000 7fcd0c4e9640 1 Processor -- start 2024-05-09T03:09:59.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.647+0000 7fcd0c4e9640 1 -- start start 2024-05-09T03:09:59.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0c4e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 0x7fcd0415a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0c4e9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040d2a70 0x7fcd0415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0c4e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04163eb0 0x7fcd04156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0c4e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd040d0bc0 con 0x7fcd040bb770 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0c4e9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcd040bbb80 con 0x7fcd04163eb0 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0c4e9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcd040d0d40 con 0x7fcd040d2a70 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0a285640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 0x7fcd0415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0a285640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 0x7fcd0415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55524/0 (socket says 172.21.15.46:55524) 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.648+0000 7fcd0a285640 1 -- 172.21.15.46:0/2119550279 learned_addr learned my addr 172.21.15.46:0/2119550279 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0aa86640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04163eb0 0x7fcd04156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd09a84640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040d2a70 0x7fcd0415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0a285640 1 -- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040d2a70 msgr2=0x7fcd0415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0a285640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040d2a70 0x7fcd0415d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0a285640 1 -- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04163eb0 msgr2=0x7fcd04156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0a285640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04163eb0 0x7fcd04156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0a285640 1 -- 172.21.15.46:0/2119550279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0415a850 con 0x7fcd040bb770 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0aa86640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04163eb0 0x7fcd04156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:59.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd09a84640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040d2a70 0x7fcd0415d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:09:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0a285640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 0x7fcd0415a090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c3c0 tx=0x7fccfc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc0074e0 con 0x7fcd040bb770 2024-05-09T03:09:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccfc0047e0 con 0x7fcd040bb770 2024-05-09T03:09:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0415e1c0 con 0x7fcd040bb770 2024-05-09T03:09:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc004c10 con 0x7fcd040bb770 2024-05-09T03:09:59.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.649+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd0415e780 con 0x7fcd040bb770 2024-05-09T03:09:59.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.650+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd0406a270 con 0x7fcd040bb770 2024-05-09T03:09:59.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.651+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fccfc011070 con 0x7fcd040bb770 2024-05-09T03:09:59.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.652+0000 7fcd02ffd640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccf0076300 0x7fccf0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:09:59.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.652+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fccfc097260 con 0x7fcd040bb770 2024-05-09T03:09:59.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.652+0000 7fcd09a84640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccf0076300 0x7fccf0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:09:59.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.652+0000 7fcd09a84640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccf0076300 0x7fccf0078810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fccf8006fd0 tx=0x7fccf8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:09:59.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.656+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcd0415a850 con 0x7fcd040bb770 2024-05-09T03:09:59.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.863+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcd040d09a0 con 0x7fcd040bb770 2024-05-09T03:09:59.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.863+0000 7fcd02ffd640 1 -- 172.21.15.46:0/2119550279 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7fcd040d33c0 con 0x7fcd040bb770 2024-05-09T03:09:59.864 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:09:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccf0076300 msgr2=0x7fccf0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccf0076300 0x7fccf0078810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fccf8006fd0 tx=0x7fccf8008040 comp rx=0 tx=0).stop 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 msgr2=0x7fcd0415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 0x7fcd0415a090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c3c0 tx=0x7fccfc00c880 comp rx=0 tx=0).stop 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 shutdown_connections 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04163eb0 0x7fcd04156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd040d2a70 0x7fcd0415d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccf0076300 0x7fccf0078810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 --2- 172.21.15.46:0/2119550279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd040bb770 0x7fcd0415a090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.865+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 >> 172.21.15.46:0/2119550279 conn(0x7fcd0406e370 msgr2=0x7fcd040b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.866+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 shutdown_connections 2024-05-09T03:09:59.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:09:59.866+0000 7fcd0c4e9640 1 -- 172.21.15.46:0/2119550279 wait complete. 2024-05-09T03:09:59.877 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:09:59.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:59.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-05-09T03:09:59.904 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:09:59.904 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:09:59.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:09:59.904 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-05-09T03:09:59.961 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:59.961 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:09:59.980 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:09:59.981 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:10:00.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:10:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.156+0000 7fb88415f640 1 Processor -- start 2024-05-09T03:10:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.156+0000 7fb88415f640 1 -- start start 2024-05-09T03:10:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.157+0000 7fb88415f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb87c0b5370 0x7fb87c0d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.157+0000 7fb88415f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb87c0d1da0 0x7fb87c0cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.157+0000 7fb88415f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb87c0d21b0 con 0x7fb87c143410 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.157+0000 7fb88415f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb87c0b4920 con 0x7fb87c0b5370 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.157+0000 7fb88415f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb87c0b4ee0 con 0x7fb87c0d1da0 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb87c0b5370 0x7fb87c0d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb87c0b5370 0x7fb87c0d1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55536/0 (socket says 172.21.15.46:55536) 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 -- 172.21.15.46:0/2900090782 learned_addr learned my addr 172.21.15.46:0/2900090782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8826fc640 1 --2- 172.21.15.46:0/2900090782 >> v2:172.21.15.88:3300/0 conn(0x7fb87c0d1da0 0x7fb87c0cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 -- 172.21.15.46:0/2900090782 >> v2:172.21.15.88:3300/0 conn(0x7fb87c0d1da0 msgr2=0x7fb87c0cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 --2- 172.21.15.46:0/2900090782 >> v2:172.21.15.88:3300/0 conn(0x7fb87c0d1da0 0x7fb87c0cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 -- 172.21.15.46:0/2900090782 >> v1:172.21.15.88:6789/0 conn(0x7fb87c143410 legacy=0x7fb87c160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8816fa640 1 -- 172.21.15.46:0/2900090782 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c1616b0 con 0x7fb87c0b5370 2024-05-09T03:10:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.158+0000 7fb8826fc640 1 --2- 172.21.15.46:0/2900090782 >> v2:172.21.15.88:3300/0 conn(0x7fb87c0d1da0 0x7fb87c0cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb8816fa640 1 --2- 172.21.15.46:0/2900090782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0b5370 0x7fb87c0d1860 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0cf410 tx=0x7fb86c02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339987ba7ad663a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb880ef9640 1 -- 172.21.15.46:0/2900090782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c037070 con 0x7fb87c0b5370 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb880ef9640 1 -- 172.21.15.46:0/2900090782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86c032440 con 0x7fb87c0b5370 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb880ef9640 1 -- 172.21.15.46:0/2900090782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c032810 con 0x7fb87c0b5370 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb88415f640 1 -- 172.21.15.46:0/2900090782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0b5370 msgr2=0x7fb87c0d1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb88415f640 1 --2- 172.21.15.46:0/2900090782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0b5370 0x7fb87c0d1860 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0cf410 tx=0x7fb86c02a200 comp rx=0 tx=0).stop 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb88415f640 1 -- 172.21.15.46:0/2900090782 shutdown_connections 2024-05-09T03:10:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb88415f640 1 --2- 172.21.15.46:0/2900090782 >> v2:172.21.15.88:3300/0 conn(0x7fb87c0d1da0 0x7fb87c0cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb88415f640 1 --2- 172.21.15.46:0/2900090782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0b5370 0x7fb87c0d1860 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.159+0000 7fb88415f640 1 -- 172.21.15.46:0/2900090782 >> 172.21.15.46:0/2900090782 conn(0x7fb87c0b3620 msgr2=0x7fb87c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.160+0000 7fb88415f640 1 -- 172.21.15.46:0/2900090782 shutdown_connections 2024-05-09T03:10:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.160+0000 7fb88415f640 1 -- 172.21.15.46:0/2900090782 wait complete. 2024-05-09T03:10:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.161+0000 7fb88415f640 1 Processor -- start 2024-05-09T03:10:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.161+0000 7fb88415f640 1 -- start start 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb88415f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb87c0b5370 0x7fb87c1e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb88415f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 0x7fb87c1f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb88415f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb87c143410 0x7fb87c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb88415f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c143820 con 0x7fb87c0d1da0 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb88415f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c0b4b70 con 0x7fb87c0b5370 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb88415f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb87c161370 con 0x7fb87c143410 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 0x7fb87c1f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 0x7fb87c1f9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55540/0 (socket says 172.21.15.46:55540) 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 -- 172.21.15.46:0/2993325994 learned_addr learned my addr 172.21.15.46:0/2993325994 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8826fc640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb87c143410 0x7fb87c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb881efb640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb87c0b5370 0x7fb87c1e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 -- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb87c143410 msgr2=0x7fb87c158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb87c143410 0x7fb87c158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 -- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb87c0b5370 msgr2=0x7fb87c1e8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb87c0b5370 0x7fb87c1e8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 -- 172.21.15.46:0/2993325994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c1facf0 con 0x7fb87c0d1da0 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8826fc640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb87c143410 0x7fb87c158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb881efb640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb87c0b5370 0x7fb87c1e8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8816fa640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 0x7fb87c1f9f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb86c02a240 tx=0x7fb86c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.163+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c041020 con 0x7fb87c0d1da0 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.164+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86c0306c0 con 0x7fb87c0d1da0 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.164+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c037040 con 0x7fb87c0d1da0 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.164+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c156b10 con 0x7fb87c0d1da0 2024-05-09T03:10:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.164+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c1570d0 con 0x7fb87c0d1da0 2024-05-09T03:10:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.164+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c06a270 con 0x7fb87c0d1da0 2024-05-09T03:10:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.165+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb86c044020 con 0x7fb87c0d1da0 2024-05-09T03:10:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.166+0000 7fb8727fc640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8600762d0 0x7fb8600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.166+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb86c0b7e30 con 0x7fb87c0d1da0 2024-05-09T03:10:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.166+0000 7fb881efb640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8600762d0 0x7fb8600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.167+0000 7fb881efb640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8600762d0 0x7fb8600787e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb878007fd0 tx=0x7fb878009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.171+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb86c04b030 con 0x7fb87c0d1da0 2024-05-09T03:10:00.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.375+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb87c1faa80 con 0x7fb87c0d1da0 2024-05-09T03:10:00.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.375+0000 7fb8727fc640 1 -- 172.21.15.46:0/2993325994 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7fb86c081790 con 0x7fb87c0d1da0 2024-05-09T03:10:00.376 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8600762d0 msgr2=0x7fb8600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8600762d0 0x7fb8600787e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb878007fd0 tx=0x7fb878009040 comp rx=0 tx=0).stop 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 msgr2=0x7fb87c1f9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 0x7fb87c1f9f80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb86c02a240 tx=0x7fb86c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 shutdown_connections 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb87c143410 0x7fb87c158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8600762d0 0x7fb8600787e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb87c0d1da0 0x7fb87c1f9f80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 --2- 172.21.15.46:0/2993325994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb87c0b5370 0x7fb87c1e8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 >> 172.21.15.46:0/2993325994 conn(0x7fb87c0b3620 msgr2=0x7fb87c0cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:00.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 shutdown_connections 2024-05-09T03:10:00.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.377+0000 7fb88415f640 1 -- 172.21.15.46:0/2993325994 wait complete. 2024-05-09T03:10:00.389 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:10:00.389 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:10:00.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:10:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.524+0000 7f1752c89640 1 Processor -- start 2024-05-09T03:10:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.524+0000 7f1752c89640 1 -- start start 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1752c89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f174c0d32c0 0x7f174c0b9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1752c89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f174c0d4a60 con 0x7f174c0b7e70 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1752c89640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f174c0d4d40 con 0x7f174c0d2eb0 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1752c89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f174c0d5020 con 0x7f174c0d32c0 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1751226640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f174c0d32c0 0x7f174c0b9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f174bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f174c0b7e70 0x7f174c0b95a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43352/0 (socket says 172.21.15.46:43352) 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f174bfff640 1 -- 172.21.15.46:0/3890639791 learned_addr learned my addr 172.21.15.46:0/3890639791 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1751226640 1 -- 172.21.15.46:0/3890639791 >> v1:172.21.15.88:6789/0 conn(0x7f174c0d2eb0 legacy=0x7f174c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1751226640 1 -- 172.21.15.46:0/3890639791 >> v1:172.21.15.46:6789/0 conn(0x7f174c0b7e70 legacy=0x7f174c0b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.526+0000 7f1751226640 1 -- 172.21.15.46:0/3890639791 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c0d5300 con 0x7f174c0d32c0 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.527+0000 7f1751226640 1 --2- 172.21.15.46:0/3890639791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b9cb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1744009d90 tx=0x7f174402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64ffcdfc188d303 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.527+0000 7f174b7fe640 1 -- 172.21.15.46:0/3890639791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744007740 con 0x7f174c0d32c0 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.527+0000 7f174b7fe640 1 -- 172.21.15.46:0/3890639791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17440078e0 con 0x7f174c0d32c0 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.527+0000 7f174b7fe640 1 -- 172.21.15.46:0/3890639791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744007cb0 con 0x7f174c0d32c0 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 -- 172.21.15.46:0/3890639791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 msgr2=0x7f174c0b9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 --2- 172.21.15.46:0/3890639791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b9cb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1744009d90 tx=0x7f174402a6f0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 -- 172.21.15.46:0/3890639791 shutdown_connections 2024-05-09T03:10:00.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 --2- 172.21.15.46:0/3890639791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b9cb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 -- 172.21.15.46:0/3890639791 >> 172.21.15.46:0/3890639791 conn(0x7f174c08d390 msgr2=0x7f174c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 -- 172.21.15.46:0/3890639791 shutdown_connections 2024-05-09T03:10:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.528+0000 7f1752c89640 1 -- 172.21.15.46:0/3890639791 wait complete. 2024-05-09T03:10:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.530+0000 7f1752c89640 1 Processor -- start 2024-05-09T03:10:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.530+0000 7f1752c89640 1 -- start start 2024-05-09T03:10:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.531+0000 7f1752c89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f174c0d2eb0 0x7f174c14bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.531+0000 7f1752c89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.531+0000 7f1752c89640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f174c0bae00 0x7f174c0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.531+0000 7f1752c89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f174c0b5360 con 0x7f174c0d32c0 2024-05-09T03:10:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.531+0000 7f1752c89640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f174c0b51e0 con 0x7f174c0d2eb0 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.531+0000 7f1752c89640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f174c163bd0 con 0x7f174c0bae00 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55568/0 (socket says 172.21.15.46:55568) 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 -- 172.21.15.46:0/1720027998 learned_addr learned my addr 172.21.15.46:0/1720027998 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f1750a25640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f174c0d2eb0 0x7f174c14bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f1751226640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f174c0bae00 0x7f174c0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 -- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f174c0bae00 msgr2=0x7f174c0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f174c0bae00 0x7f174c0c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 -- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f174c0d2eb0 msgr2=0x7f174c14bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f174c0d2eb0 0x7f174c14bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 -- 172.21.15.46:0/1720027998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c0b9d60 con 0x7f174c0d32c0 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f1750a25640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f174c0d2eb0 0x7f174c14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f174bfff640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b94a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f173400c400 tx=0x7f173400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734017070 con 0x7f174c0d32c0 2024-05-09T03:10:00.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1734005ce0 con 0x7f174c0d32c0 2024-05-09T03:10:00.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c0ba040 con 0x7f174c0d32c0 2024-05-09T03:10:00.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17340072d0 con 0x7f174c0d32c0 2024-05-09T03:10:00.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.532+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c0ba600 con 0x7f174c0d32c0 2024-05-09T03:10:00.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.533+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174c06a270 con 0x7f174c0d32c0 2024-05-09T03:10:00.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.534+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f173401a070 con 0x7f174c0d32c0 2024-05-09T03:10:00.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.535+0000 7f17497fa640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f172c0762d0 0x7f172c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.535+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1734096890 con 0x7f174c0d32c0 2024-05-09T03:10:00.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.535+0000 7f1750a25640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f172c0762d0 0x7f172c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.536+0000 7f1750a25640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f172c0762d0 0x7f172c0787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f173c007fd0 tx=0x7f173c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.540+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f173405fc30 con 0x7f174c0d32c0 2024-05-09T03:10:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.743+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f174c0c9420 con 0x7f174c0d32c0 2024-05-09T03:10:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.743+0000 7f17497fa640 1 -- 172.21.15.46:0/1720027998 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f1734063f30 con 0x7f174c0d32c0 2024-05-09T03:10:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f172c0762d0 msgr2=0x7f172c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f172c0762d0 0x7f172c0787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f173c007fd0 tx=0x7f173c009040 comp rx=0 tx=0).stop 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 msgr2=0x7f174c0b94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b94a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f173400c400 tx=0x7f173400c8c0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 shutdown_connections 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f174c0bae00 0x7f174c0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f172c0762d0 0x7f172c0787e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0d32c0 0x7f174c0b94a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 --2- 172.21.15.46:0/1720027998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f174c0d2eb0 0x7f174c14bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 >> 172.21.15.46:0/1720027998 conn(0x7f174c08d390 msgr2=0x7f174c0b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:00.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 shutdown_connections 2024-05-09T03:10:00.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.745+0000 7f1752c89640 1 -- 172.21.15.46:0/1720027998 wait complete. 2024-05-09T03:10:00.757 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi088.front.sepia.ceph.com... 2024-05-09T03:10:00.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:00.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-05-09T03:10:00.789 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:00.789 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:10:00.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:00.790 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-05-09T03:10:00.848 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T03:10:00.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.964+0000 7f4b49e43640 1 Processor -- start 2024-05-09T03:10:00.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.964+0000 7f4b49e43640 1 -- start start 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b49e43640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b440d0b90 0x7f4b440b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b49e43640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b440d2b80 con 0x7f4b440b92e0 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b49e43640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b440d2e60 con 0x7f4b440cf6b0 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b49e43640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4b440d3140 con 0x7f4b440d0b90 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b42ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b440d0b90 0x7f4b440b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b437fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b440b92e0 0x7f4b440d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43366/0 (socket says 172.21.15.46:43366) 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b437fe640 1 -- 172.21.15.46:0/4110451220 learned_addr learned my addr 172.21.15.46:0/4110451220 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:00.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b42ffd640 1 -- 172.21.15.46:0/4110451220 >> v1:172.21.15.88:6789/0 conn(0x7f4b440cf6b0 legacy=0x7f4b4406d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:00.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b42ffd640 1 -- 172.21.15.46:0/4110451220 >> v1:172.21.15.46:6789/0 conn(0x7f4b440b92e0 legacy=0x7f4b440d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.965+0000 7f4b42ffd640 1 -- 172.21.15.46:0/4110451220 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b440d3420 con 0x7f4b440d0b90 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.966+0000 7f4b42ffd640 1 --2- 172.21.15.46:0/4110451220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440d0b90 0x7f4b440b51a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4b380097a0 tx=0x7f4b3802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf22d68413e5a7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.966+0000 7f4b427fc640 1 -- 172.21.15.46:0/4110451220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38037070 con 0x7f4b440d0b90 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.966+0000 7f4b427fc640 1 -- 172.21.15.46:0/4110451220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b3802ae50 con 0x7f4b440d0b90 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.966+0000 7f4b427fc640 1 -- 172.21.15.46:0/4110451220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38032660 con 0x7f4b440d0b90 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 -- 172.21.15.46:0/4110451220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440d0b90 msgr2=0x7f4b440b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 --2- 172.21.15.46:0/4110451220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440d0b90 0x7f4b440b51a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4b380097a0 tx=0x7f4b3802a100 comp rx=0 tx=0).stop 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 -- 172.21.15.46:0/4110451220 shutdown_connections 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 --2- 172.21.15.46:0/4110451220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440d0b90 0x7f4b440b51a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 -- 172.21.15.46:0/4110451220 >> 172.21.15.46:0/4110451220 conn(0x7f4b440d3c00 msgr2=0x7f4b440d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 -- 172.21.15.46:0/4110451220 shutdown_connections 2024-05-09T03:10:00.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.967+0000 7f4b49e43640 1 -- 172.21.15.46:0/4110451220 wait complete. 2024-05-09T03:10:00.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.969+0000 7f4b49e43640 1 Processor -- start 2024-05-09T03:10:00.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.969+0000 7f4b49e43640 1 -- start start 2024-05-09T03:10:00.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b49e43640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b440b92e0 0x7f4b441545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b49e43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 0x7f4b44155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b49e43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b440cd890 0x7f4b441560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b49e43640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4416b3d0 con 0x7f4b440cf6b0 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b49e43640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4416b250 con 0x7f4b440cd890 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b49e43640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4b4416b550 con 0x7f4b440b92e0 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.970+0000 7f4b42ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 0x7f4b44155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 0x7f4b44155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55576/0 (socket says 172.21.15.46:55576) 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 -- 172.21.15.46:0/1296379553 learned_addr learned my addr 172.21.15.46:0/1296379553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b437fe640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b440b92e0 0x7f4b441545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 -- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b440b92e0 msgr2=0x7f4b441545d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b43fff640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b440cd890 0x7f4b441560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b440b92e0 0x7f4b441545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 -- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b440cd890 msgr2=0x7f4b441560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b440cd890 0x7f4b441560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:00.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 -- 172.21.15.46:0/1296379553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b44154b10 con 0x7f4b440cf6b0 2024-05-09T03:10:00.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b42ffd640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 0x7f4b44155b70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4b380097a0 tx=0x7f4b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38042070 con 0x7f4b440cf6b0 2024-05-09T03:10:00.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b3803e670 con 0x7f4b440cf6b0 2024-05-09T03:10:00.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b38037040 con 0x7f4b440cf6b0 2024-05-09T03:10:00.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b440cc0c0 con 0x7f4b440cf6b0 2024-05-09T03:10:00.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.971+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b4415b0e0 con 0x7f4b440cf6b0 2024-05-09T03:10:00.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.973+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b4415abf0 con 0x7f4b440cf6b0 2024-05-09T03:10:00.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.973+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4b38045070 con 0x7f4b440cf6b0 2024-05-09T03:10:00.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.974+0000 7f4b2bfff640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b30076300 0x7f4b30078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:00.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.974+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4b380b7e30 con 0x7f4b440cf6b0 2024-05-09T03:10:00.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.974+0000 7f4b437fe640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b30076300 0x7f4b30078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:00.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.975+0000 7f4b437fe640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b30076300 0x7f4b30078810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4b34000f60 tx=0x7f4b34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:00.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:00.978+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4b44154b10 con 0x7f4b440cf6b0 2024-05-09T03:10:01.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.207+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b4406a270 con 0x7f4b440cf6b0 2024-05-09T03:10:01.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.207+0000 7f4b2bfff640 1 -- 172.21.15.46:0/1296379553 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7f4b38081790 con 0x7f4b440cf6b0 2024-05-09T03:10:01.208 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:01.208 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:10:01.208 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b30076300 msgr2=0x7f4b30078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b30076300 0x7f4b30078810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4b34000f60 tx=0x7f4b34009040 comp rx=0 tx=0).stop 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 msgr2=0x7f4b44155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 0x7f4b44155b70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4b380097a0 tx=0x7f4b380046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 shutdown_connections 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b440cd890 0x7f4b441560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b30076300 0x7f4b30078810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b440cf6b0 0x7f4b44155b70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 --2- 172.21.15.46:0/1296379553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b440b92e0 0x7f4b441545d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 >> 172.21.15.46:0/1296379553 conn(0x7f4b440d3c00 msgr2=0x7f4b440b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 shutdown_connections 2024-05-09T03:10:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.210+0000 7f4b49e43640 1 -- 172.21.15.46:0/1296379553 wait complete. 2024-05-09T03:10:01.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.336+0000 7f69c0a29640 1 Processor -- start 2024-05-09T03:10:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.336+0000 7f69c0a29640 1 -- start start 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69c0a29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc15e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69c0a29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69bc0cfc20 0x7f69bc0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69c0a29640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69bc0ceb80 con 0x7f69bc09bfe0 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69c0a29640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f69bc0d0570 con 0x7f69bc0cfc20 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69c0a29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f69bc0cf790 con 0x7f69bc0ce5e0 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69bad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69bc0cfc20 0x7f69bc0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.337+0000 7f69bad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69bc0cfc20 0x7f69bc0d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55586/0 (socket says 172.21.15.46:55586) 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.338+0000 7f69bad9d640 1 -- 172.21.15.46:0/3187336776 learned_addr learned my addr 172.21.15.46:0/3187336776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.338+0000 7f69ba59c640 1 --2- 172.21.15.46:0/3187336776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc15e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.338+0000 7f69bad9d640 1 -- 172.21.15.46:0/3187336776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 msgr2=0x7f69bc15e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.338+0000 7f69bad9d640 1 --2- 172.21.15.46:0/3187336776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc15e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.338+0000 7f69bad9d640 1 -- 172.21.15.46:0/3187336776 >> v1:172.21.15.88:6789/0 conn(0x7f69bc09bfe0 legacy=0x7f69bc0afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.338+0000 7f69bad9d640 1 -- 172.21.15.46:0/3187336776 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc0cee60 con 0x7f69bc0cfc20 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69bad9d640 1 --2- 172.21.15.46:0/3187336776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc0d0030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009e50 tx=0x7f69b402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e6e2cd12ce3356 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69b9d9b640 1 -- 172.21.15.46:0/3187336776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4036070 con 0x7f69bc0cfc20 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69b9d9b640 1 -- 172.21.15.46:0/3187336776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b4031600 con 0x7f69bc0cfc20 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69b9d9b640 1 -- 172.21.15.46:0/3187336776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40319d0 con 0x7f69bc0cfc20 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 -- 172.21.15.46:0/3187336776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 msgr2=0x7f69bc0d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 --2- 172.21.15.46:0/3187336776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc0d0030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009e50 tx=0x7f69b402a680 comp rx=0 tx=0).stop 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 -- 172.21.15.46:0/3187336776 shutdown_connections 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 --2- 172.21.15.46:0/3187336776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc0d0030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 --2- 172.21.15.46:0/3187336776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc15e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 -- 172.21.15.46:0/3187336776 >> 172.21.15.46:0/3187336776 conn(0x7f69bc0d2c70 msgr2=0x7f69bc0d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 -- 172.21.15.46:0/3187336776 shutdown_connections 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.339+0000 7f69c0a29640 1 -- 172.21.15.46:0/3187336776 wait complete. 2024-05-09T03:10:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.341+0000 7f69c0a29640 1 Processor -- start 2024-05-09T03:10:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.342+0000 7f69c0a29640 1 -- start start 2024-05-09T03:10:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69c0a29640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69bc09bfe0 0x7f69bc1fa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69c0a29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc14ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69c0a29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc14f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69c0a29640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc0b0130 con 0x7f69bc0cfc20 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69c0a29640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc0cf420 con 0x7f69bc09bfe0 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69c0a29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f69bc14e2d0 con 0x7f69bc0ce5e0 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc14f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc14f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55590/0 (socket says 172.21.15.46:55590) 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 -- 172.21.15.46:0/2758058768 learned_addr learned my addr 172.21.15.46:0/2758058768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69b1d9b640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc14ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69ba59c640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69bc09bfe0 0x7f69bc1fa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 -- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 msgr2=0x7f69bc14ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc14ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 -- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69bc09bfe0 msgr2=0x7f69bc1fa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69bc09bfe0 0x7f69bc1fa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.343+0000 7f69bad9d640 1 -- 172.21.15.46:0/2758058768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc14d230 con 0x7f69bc0cfc20 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.344+0000 7f69bad9d640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc14f590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009f70 tx=0x7f69b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.344+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4041020 con 0x7f69bc0cfc20 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.344+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b40077a0 con 0x7f69bc0cfc20 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.344+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69bc1501e0 con 0x7f69bc0cfc20 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.344+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4036040 con 0x7f69bc0cfc20 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.344+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69bc1f9ec0 con 0x7f69bc0cfc20 2024-05-09T03:10:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.345+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69bc14ff50 con 0x7f69bc0cfc20 2024-05-09T03:10:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.346+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f69b4044020 con 0x7f69bc0cfc20 2024-05-09T03:10:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.346+0000 7f69b37fe640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f699c0762d0 0x7f699c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.346+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f69b40b8480 con 0x7f69bc0cfc20 2024-05-09T03:10:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.347+0000 7f69ba59c640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f699c0762d0 0x7f699c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.347+0000 7f69ba59c640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f699c0762d0 0x7f699c0787e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009a90 tx=0x7f69ac009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:01.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.350+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f69b404b030 con 0x7f69bc0cfc20 2024-05-09T03:10:01.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.578+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69bc076490 con 0x7f69bc0cfc20 2024-05-09T03:10:01.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.578+0000 7f69b37fe640 1 -- 172.21.15.46:0/2758058768 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7f69bc076490 con 0x7f69bc0cfc20 2024-05-09T03:10:01.579 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:01.579 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:10:01.580 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:10:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.580+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f699c0762d0 msgr2=0x7f699c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.580+0000 7f69c0a29640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f699c0762d0 0x7f699c0787e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009a90 tx=0x7f69ac009110 comp rx=0 tx=0).stop 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.580+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 msgr2=0x7f69bc14f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.580+0000 7f69c0a29640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc14f590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009f70 tx=0x7f69b4004a40 comp rx=0 tx=0).stop 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 shutdown_connections 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f699c0762d0 0x7f699c0787e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69bc0cfc20 0x7f69bc14f590 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69bc0ce5e0 0x7f69bc14ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 --2- 172.21.15.46:0/2758058768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69bc09bfe0 0x7f69bc1fa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 >> 172.21.15.46:0/2758058768 conn(0x7f69bc0d2c70 msgr2=0x7f69bc15ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 shutdown_connections 2024-05-09T03:10:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.581+0000 7f69c0a29640 1 -- 172.21.15.46:0/2758058768 wait complete. 2024-05-09T03:10:01.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.707+0000 7f44547e8640 1 Processor -- start 2024-05-09T03:10:01.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.707+0000 7f44547e8640 1 -- start start 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f44547e8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f444c0d57a0 0x7f444c160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f44547e8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c0b7ef0 0x7f444c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f44547e8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444c0d49e0 con 0x7f444c0b4af0 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f44547e8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f444c0d5bb0 con 0x7f444c0d57a0 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f44547e8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f444c0d4cc0 con 0x7f444c0b7ef0 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f444c0d57a0 0x7f444c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f444c0d57a0 0x7f444c160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55600/0 (socket says 172.21.15.46:55600) 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 -- 172.21.15.46:0/3348788300 learned_addr learned my addr 172.21.15.46:0/3348788300 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452d85640 1 --2- 172.21.15.46:0/3348788300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c0b7ef0 0x7f444c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 -- 172.21.15.46:0/3348788300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c0b7ef0 msgr2=0x7f444c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 --2- 172.21.15.46:0/3348788300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c0b7ef0 0x7f444c0b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 -- 172.21.15.46:0/3348788300 >> v1:172.21.15.88:6789/0 conn(0x7f444c0b4af0 legacy=0x7f444c0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:01.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.709+0000 7f4452584640 1 -- 172.21.15.46:0/3348788300 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c0d4fa0 con 0x7f444c0d57a0 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.710+0000 7f4452584640 1 --2- 172.21.15.46:0/3348788300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c160610 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4448009e50 tx=0x7f444802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b3ac52af1fb3cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.710+0000 7f4451582640 1 -- 172.21.15.46:0/3348788300 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448036070 con 0x7f444c0d57a0 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.710+0000 7f4451582640 1 -- 172.21.15.46:0/3348788300 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4448031600 con 0x7f444c0d57a0 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.710+0000 7f4451582640 1 -- 172.21.15.46:0/3348788300 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44480319d0 con 0x7f444c0d57a0 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 -- 172.21.15.46:0/3348788300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 msgr2=0x7f444c160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 --2- 172.21.15.46:0/3348788300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c160610 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4448009e50 tx=0x7f444802a680 comp rx=0 tx=0).stop 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 -- 172.21.15.46:0/3348788300 shutdown_connections 2024-05-09T03:10:01.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 --2- 172.21.15.46:0/3348788300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c0b7ef0 0x7f444c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 --2- 172.21.15.46:0/3348788300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c160610 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 -- 172.21.15.46:0/3348788300 >> 172.21.15.46:0/3348788300 conn(0x7f444c0743e0 msgr2=0x7f444c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 -- 172.21.15.46:0/3348788300 shutdown_connections 2024-05-09T03:10:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.711+0000 7f44547e8640 1 -- 172.21.15.46:0/3348788300 wait complete. 2024-05-09T03:10:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.713+0000 7f44547e8640 1 Processor -- start 2024-05-09T03:10:01.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.713+0000 7f44547e8640 1 -- start start 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f44547e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f444c0b4af0 0x7f444c1f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f44547e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c1538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f44547e8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c151320 0x7f444c0b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f44547e8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f444c0d1920 con 0x7f444c0d57a0 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f44547e8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f444c0d15b0 con 0x7f444c0b4af0 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f44547e8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f444c1f6d40 con 0x7f444c151320 2024-05-09T03:10:01.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f4451d83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c1538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f4451d83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c1538d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55610/0 (socket says 172.21.15.46:55610) 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f4451d83640 1 -- 172.21.15.46:0/3747184521 learned_addr learned my addr 172.21.15.46:0/3747184521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f4452584640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f444c0b4af0 0x7f444c1f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.714+0000 7f4452d85640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c151320 0x7f444c0b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4451d83640 1 -- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c151320 msgr2=0x7f444c0b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4451d83640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c151320 0x7f444c0b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4451d83640 1 -- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f444c0b4af0 msgr2=0x7f444c1f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4451d83640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f444c0b4af0 0x7f444c1f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4451d83640 1 -- 172.21.15.46:0/3747184521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c1627a0 con 0x7f444c0d57a0 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4451d83640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c1538d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f443c00b410 tx=0x7f443c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c015020 con 0x7f444c0d57a0 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f443c011830 con 0x7f444c0d57a0 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444c162a80 con 0x7f444c0d57a0 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c011c20 con 0x7f444c0d57a0 2024-05-09T03:10:01.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.715+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444c1f6a80 con 0x7f444c0d57a0 2024-05-09T03:10:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.716+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444c06a270 con 0x7f444c0d57a0 2024-05-09T03:10:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.717+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f443c018020 con 0x7f444c0d57a0 2024-05-09T03:10:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.717+0000 7f4442ffd640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4430076300 0x7f4430078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.717+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f443c097450 con 0x7f444c0d57a0 2024-05-09T03:10:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.718+0000 7f4452584640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4430076300 0x7f4430078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:01.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.718+0000 7f4452584640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4430076300 0x7f4430078810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4448033000 tx=0x7f4448009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:01.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.722+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f443c01f020 con 0x7f444c0d57a0 2024-05-09T03:10:01.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.913+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f444c15f850 con 0x7f444c0d57a0 2024-05-09T03:10:01.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.915+0000 7f4442ffd640 1 -- 172.21.15.46:0/3747184521 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7f443c060760 con 0x7f444c0d57a0 2024-05-09T03:10:01.916 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:01.916 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4430076300 msgr2=0x7f4430078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4430076300 0x7f4430078810 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4448033000 tx=0x7f4448009000 comp rx=0 tx=0).stop 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 msgr2=0x7f444c1538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c1538d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f443c00b410 tx=0x7f443c00b8d0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 shutdown_connections 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f444c151320 0x7f444c0b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4430076300 0x7f4430078810 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f444c0d57a0 0x7f444c1538d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 --2- 172.21.15.46:0/3747184521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f444c0b4af0 0x7f444c1f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 >> 172.21.15.46:0/3747184521 conn(0x7f444c0743e0 msgr2=0x7f444c153130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 shutdown_connections 2024-05-09T03:10:01.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:01.917+0000 7f44547e8640 1 -- 172.21.15.46:0/3747184521 wait complete. 2024-05-09T03:10:01.929 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:10:01.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.043+0000 7f95cee14640 1 Processor -- start 2024-05-09T03:10:02.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.043+0000 7f95cee14640 1 -- start start 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cee14640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95c80d57a0 0x7f95c8160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cee14640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95c80b4af0 0x7f95c80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cee14640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95c80d49e0 con 0x7f95c80b77e0 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cee14640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95c80d5bb0 con 0x7f95c80b4af0 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cee14640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95c80d4cc0 con 0x7f95c80d57a0 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95c7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95c80b4af0 0x7f95c80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cd3b1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95c80b77e0 0x7f95c80b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43372/0 (socket says 172.21.15.46:43372) 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95cd3b1640 1 -- 172.21.15.46:0/4237234154 learned_addr learned my addr 172.21.15.46:0/4237234154 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95ccbb0640 1 --2- 172.21.15.46:0/4237234154 >> v2:172.21.15.88:3300/0 conn(0x7f95c80d57a0 0x7f95c8160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95c7fff640 1 -- 172.21.15.46:0/4237234154 >> v2:172.21.15.88:3300/0 conn(0x7f95c80d57a0 msgr2=0x7f95c8160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:02.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95c7fff640 1 --2- 172.21.15.46:0/4237234154 >> v2:172.21.15.88:3300/0 conn(0x7f95c80d57a0 0x7f95c8160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95c7fff640 1 -- 172.21.15.46:0/4237234154 >> v1:172.21.15.46:6789/0 conn(0x7f95c80b77e0 legacy=0x7f95c80b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95c77fe640 1 -- 172.21.15.46:0/4237234154 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13381596 0 0) 0x7f95c80d49e0 con 0x7f95c80b77e0 2024-05-09T03:10:02.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.045+0000 7f95c7fff640 1 -- 172.21.15.46:0/4237234154 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c80d4fa0 con 0x7f95c80b4af0 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.046+0000 7f95c7fff640 1 --2- 172.21.15.46:0/4237234154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b4af0 0x7f95c80b4f00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f95c808ed20 tx=0x7f95bc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b639a2882b14e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.046+0000 7f95c77fe640 1 -- 172.21.15.46:0/4237234154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc037070 con 0x7f95c80b4af0 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.046+0000 7f95c77fe640 1 -- 172.21.15.46:0/4237234154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc032440 con 0x7f95c80b4af0 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.046+0000 7f95c77fe640 1 -- 172.21.15.46:0/4237234154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc032810 con 0x7f95c80b4af0 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 -- 172.21.15.46:0/4237234154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b4af0 msgr2=0x7f95c80b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 --2- 172.21.15.46:0/4237234154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b4af0 0x7f95c80b4f00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f95c808ed20 tx=0x7f95bc02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 -- 172.21.15.46:0/4237234154 shutdown_connections 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 --2- 172.21.15.46:0/4237234154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b4af0 0x7f95c80b4f00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 --2- 172.21.15.46:0/4237234154 >> v2:172.21.15.88:3300/0 conn(0x7f95c80d57a0 0x7f95c8160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 -- 172.21.15.46:0/4237234154 >> 172.21.15.46:0/4237234154 conn(0x7f95c80743e0 msgr2=0x7f95c80d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 -- 172.21.15.46:0/4237234154 shutdown_connections 2024-05-09T03:10:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.047+0000 7f95cee14640 1 -- 172.21.15.46:0/4237234154 wait complete. 2024-05-09T03:10:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.049+0000 7f95cee14640 1 Processor -- start 2024-05-09T03:10:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.049+0000 7f95cee14640 1 -- start start 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cee14640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b4af0 0x7f95c814d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cee14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 0x7f95c8161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cee14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d57a0 0x7f95c814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cee14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95c80d1920 con 0x7f95c80b77e0 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cee14640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95c80d15b0 con 0x7f95c80d57a0 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cee14640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95c814ead0 con 0x7f95c80b4af0 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 0x7f95c8161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 0x7f95c8161660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55640/0 (socket says 172.21.15.46:55640) 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 -- 172.21.15.46:0/781686302 learned_addr learned my addr 172.21.15.46:0/781686302 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95ccbb0640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b4af0 0x7f95c814d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95cd3b1640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d57a0 0x7f95c814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 -- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b4af0 msgr2=0x7f95c814d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b4af0 0x7f95c814d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 -- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d57a0 msgr2=0x7f95c814daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.050+0000 7f95c7fff640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d57a0 0x7f95c814daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95c7fff640 1 -- 172.21.15.46:0/781686302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c814d690 con 0x7f95c80b77e0 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95c7fff640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 0x7f95c8161660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f95bc000c00 tx=0x7f95bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc037040 con 0x7f95c80b77e0 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc042070 con 0x7f95c80b77e0 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c8162120 con 0x7f95c80b77e0 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc030960 con 0x7f95c80b77e0 2024-05-09T03:10:02.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.051+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c81626e0 con 0x7f95c80b77e0 2024-05-09T03:10:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.052+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c8161e90 con 0x7f95c80b77e0 2024-05-09T03:10:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.053+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95bc030b00 con 0x7f95c80b77e0 2024-05-09T03:10:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.053+0000 7f95c57fa640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b4076300 0x7f95b4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.053+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f95bc0b7d00 con 0x7f95c80b77e0 2024-05-09T03:10:02.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.053+0000 7f95ccbb0640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b4076300 0x7f95b4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.054+0000 7f95ccbb0640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b4076300 0x7f95b4078810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95c806d3b0 tx=0x7f95b8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.058+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95bc0815d0 con 0x7f95c80b77e0 2024-05-09T03:10:02.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.287+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95c806a270 con 0x7f95c80b77e0 2024-05-09T03:10:02.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.288+0000 7f95c57fa640 1 -- 172.21.15.46:0/781686302 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7f95bc080ed0 con 0x7f95c80b77e0 2024-05-09T03:10:02.288 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:02.288 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:10:02.289 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:10:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b4076300 msgr2=0x7f95b4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b4076300 0x7f95b4078810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95c806d3b0 tx=0x7f95b8009190 comp rx=0 tx=0).stop 2024-05-09T03:10:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 msgr2=0x7f95c8161660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 0x7f95c8161660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f95bc000c00 tx=0x7f95bc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 shutdown_connections 2024-05-09T03:10:02.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d57a0 0x7f95c814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b4076300 0x7f95b4078810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b77e0 0x7f95c8161660 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 --2- 172.21.15.46:0/781686302 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b4af0 0x7f95c814d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 >> 172.21.15.46:0/781686302 conn(0x7f95c80743e0 msgr2=0x7f95c80bac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 shutdown_connections 2024-05-09T03:10:02.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.290+0000 7f95cee14640 1 -- 172.21.15.46:0/781686302 wait complete. 2024-05-09T03:10:02.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:02.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.415+0000 7f24fc75e640 1 Processor -- start 2024-05-09T03:10:02.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.415+0000 7f24fc75e640 1 -- start start 2024-05-09T03:10:02.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fc75e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f24f40d57a0 0x7f24f4160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fc75e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24f40b4af0 0x7f24f40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fc75e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b77e0 0x7f24f40b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fc75e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f24f40d5bb0 con 0x7f24f40b4af0 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fc75e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f24f40b8130 con 0x7f24f40d57a0 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fc75e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f24f40d49e0 con 0x7f24f40b77e0 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24f40b4af0 0x7f24f40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24f40b4af0 0x7f24f40b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55646/0 (socket says 172.21.15.46:55646) 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 -- 172.21.15.46:0/796086534 learned_addr learned my addr 172.21.15.46:0/796086534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24facfb640 1 --2- 172.21.15.46:0/796086534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b77e0 0x7f24f40b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24fa4fa640 1 --2- 172.21.15.46:0/796086534 >> v2:172.21.15.88:3300/0 conn(0x7f24f40d57a0 0x7f24f4160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 -- 172.21.15.46:0/796086534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b77e0 msgr2=0x7f24f40b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 --2- 172.21.15.46:0/796086534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b77e0 0x7f24f40b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 -- 172.21.15.46:0/796086534 >> v2:172.21.15.88:3300/0 conn(0x7f24f40d57a0 msgr2=0x7f24f4160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 --2- 172.21.15.46:0/796086534 >> v2:172.21.15.88:3300/0 conn(0x7f24f40d57a0 0x7f24f4160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.417+0000 7f24f9cf9640 1 -- 172.21.15.46:0/796086534 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f40d4cc0 con 0x7f24f40b4af0 2024-05-09T03:10:02.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.418+0000 7f24f9cf9640 1 --2- 172.21.15.46:0/796086534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40b4af0 0x7f24f40b4f00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24f408ed20 tx=0x7f24e802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2454102d85d360 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.418+0000 7f24f94f8640 1 -- 172.21.15.46:0/796086534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8037070 con 0x7f24f40b4af0 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.418+0000 7f24f94f8640 1 -- 172.21.15.46:0/796086534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8032440 con 0x7f24f40b4af0 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.418+0000 7f24f94f8640 1 -- 172.21.15.46:0/796086534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8032810 con 0x7f24f40b4af0 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 -- 172.21.15.46:0/796086534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40b4af0 msgr2=0x7f24f40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 --2- 172.21.15.46:0/796086534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40b4af0 0x7f24f40b4f00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f24f408ed20 tx=0x7f24e802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 -- 172.21.15.46:0/796086534 shutdown_connections 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 --2- 172.21.15.46:0/796086534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b77e0 0x7f24f40b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 --2- 172.21.15.46:0/796086534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40b4af0 0x7f24f40b4f00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 --2- 172.21.15.46:0/796086534 >> v2:172.21.15.88:3300/0 conn(0x7f24f40d57a0 0x7f24f4160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 -- 172.21.15.46:0/796086534 >> 172.21.15.46:0/796086534 conn(0x7f24f40743e0 msgr2=0x7f24f40d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 -- 172.21.15.46:0/796086534 shutdown_connections 2024-05-09T03:10:02.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.419+0000 7f24fc75e640 1 -- 172.21.15.46:0/796086534 wait complete. 2024-05-09T03:10:02.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.421+0000 7f24fc75e640 1 Processor -- start 2024-05-09T03:10:02.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.421+0000 7f24fc75e640 1 -- start start 2024-05-09T03:10:02.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fc75e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b4af0 0x7f24f414c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fc75e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f40b77e0 0x7f24f40ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fc75e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 0x7f24f414ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fc75e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24f40d15b0 con 0x7f24f40d57a0 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fc75e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24f40d5280 con 0x7f24f40b77e0 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fc75e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f24f40d1920 con 0x7f24f40b4af0 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24facfb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 0x7f24f414ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24facfb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 0x7f24f414ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55662/0 (socket says 172.21.15.46:55662) 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24facfb640 1 -- 172.21.15.46:0/3761062728 learned_addr learned my addr 172.21.15.46:0/3761062728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24fa4fa640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b4af0 0x7f24f414c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.422+0000 7f24f9cf9640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f40b77e0 0x7f24f40ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24facfb640 1 -- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b4af0 msgr2=0x7f24f414c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24facfb640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b4af0 0x7f24f414c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24facfb640 1 -- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f40b77e0 msgr2=0x7f24f40ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24facfb640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f40b77e0 0x7f24f40ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24facfb640 1 -- 172.21.15.46:0/3761062728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f40caee0 con 0x7f24f40d57a0 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24facfb640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 0x7f24f414ea60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00e820 tx=0x7f24ec00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec017070 con 0x7f24f40d57a0 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ec00c730 con 0x7f24f40d57a0 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f40cb1c0 con 0x7f24f40d57a0 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec00cb20 con 0x7f24f40d57a0 2024-05-09T03:10:02.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.423+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f414cd90 con 0x7f24f40d57a0 2024-05-09T03:10:02.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.424+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f406a270 con 0x7f24f40d57a0 2024-05-09T03:10:02.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.425+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f24ec01a070 con 0x7f24f40d57a0 2024-05-09T03:10:02.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.425+0000 7f24f2ffd640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f24e00762d0 0x7f24e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.426+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f24ec097450 con 0x7f24f40d57a0 2024-05-09T03:10:02.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.426+0000 7f24fa4fa640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f24e00762d0 0x7f24e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.426+0000 7f24fa4fa640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f24e00762d0 0x7f24e00787e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f24f406d3b0 tx=0x7f24e4009830 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.430+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f24ec060630 con 0x7f24f40d57a0 2024-05-09T03:10:02.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.660+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24f40b5e00 con 0x7f24f40d57a0 2024-05-09T03:10:02.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.660+0000 7f24f2ffd640 1 -- 172.21.15.46:0/3761062728 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5166 (secure 0 0 0) 0x7f24ec064930 con 0x7f24f40d57a0 2024-05-09T03:10:02.661 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:02.661 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4177137588","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4177137588},{"type":"v1","addr":"172.21.15.46:6834","nonce":4177137588}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2318243136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2318243136},{"type":"v1","addr":"172.21.15.88:6835","nonce":2318243136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/638010562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":638010562},{"type":"v1","addr":"172.21.15.88:6837","nonce":638010562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:09:33.568483+0000","modified":"2024-05-09T03:09:36.576285+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4287":{"gid":4287,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/59422413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":59422413},{"type":"v1","addr":"172.21.15.46:6835","nonce":59422413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4287,"qdb_cluster":[4287]},"id":1}]} 2024-05-09T03:10:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-05-09T03:10:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.662+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f24e00762d0 msgr2=0x7f24e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.662+0000 7f24fc75e640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f24e00762d0 0x7f24e00787e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f24f406d3b0 tx=0x7f24e4009830 comp rx=0 tx=0).stop 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 msgr2=0x7f24f414ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 0x7f24f414ea60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00e820 tx=0x7f24ec00ece0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 shutdown_connections 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f24e00762d0 0x7f24e00787e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f40d57a0 0x7f24f414ea60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24f40b77e0 0x7f24f40ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 --2- 172.21.15.46:0/3761062728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24f40b4af0 0x7f24f414c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 >> 172.21.15.46:0/3761062728 conn(0x7f24f40743e0 msgr2=0x7f24f40d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 shutdown_connections 2024-05-09T03:10:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.663+0000 7f24fc75e640 1 -- 172.21.15.46:0/3761062728 wait complete. 2024-05-09T03:10:02.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:02.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.789+0000 7fe3cff1f640 1 Processor -- start 2024-05-09T03:10:02.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.789+0000 7fe3cff1f640 1 -- start start 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cff1f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c80d57a0 0x7fe3c8160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cff1f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cff1f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3c80b77e0 0x7fe3c80b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cff1f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c80d5bb0 con 0x7fe3c80d57a0 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cff1f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c80b8130 con 0x7fe3c80b77e0 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cff1f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c80d49e0 con 0x7fe3c80b4af0 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c80d57a0 0x7fe3c8160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3c80d57a0 0x7fe3c8160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55676/0 (socket says 172.21.15.46:55676) 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 -- 172.21.15.46:0/2957727350 learned_addr learned my addr 172.21.15.46:0/2957727350 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cd4ba640 1 --2- 172.21.15.46:0/2957727350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3ce4bc640 1 --2- 172.21.15.46:0/2957727350 >> v2:172.21.15.88:3300/0 conn(0x7fe3c80b77e0 0x7fe3c80b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 -- 172.21.15.46:0/2957727350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 msgr2=0x7fe3c80b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/2957727350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 -- 172.21.15.46:0/2957727350 >> v2:172.21.15.88:3300/0 conn(0x7fe3c80b77e0 msgr2=0x7fe3c80b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/2957727350 >> v2:172.21.15.88:3300/0 conn(0x7fe3c80b77e0 0x7fe3c80b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3cdcbb640 1 -- 172.21.15.46:0/2957727350 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c80d4cc0 con 0x7fe3c80d57a0 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.791+0000 7fe3ce4bc640 1 --2- 172.21.15.46:0/2957727350 >> v2:172.21.15.88:3300/0 conn(0x7fe3c80b77e0 0x7fe3c80b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.792+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/2957727350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80d57a0 0x7fe3c8160610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3c808f200 tx=0x7fe3c402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293a27b32d20c054 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.792+0000 7fe3cccb9640 1 -- 172.21.15.46:0/2957727350 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4036070 con 0x7fe3c80d57a0 2024-05-09T03:10:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.792+0000 7fe3cccb9640 1 -- 172.21.15.46:0/2957727350 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c4031600 con 0x7fe3c80d57a0 2024-05-09T03:10:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.792+0000 7fe3cccb9640 1 -- 172.21.15.46:0/2957727350 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c40319d0 con 0x7fe3c80d57a0 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 -- 172.21.15.46:0/2957727350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80d57a0 msgr2=0x7fe3c8160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/2957727350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80d57a0 0x7fe3c8160610 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe3c808f200 tx=0x7fe3c402a6a0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 -- 172.21.15.46:0/2957727350 shutdown_connections 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/2957727350 >> v2:172.21.15.88:3300/0 conn(0x7fe3c80b77e0 0x7fe3c80b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/2957727350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/2957727350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80d57a0 0x7fe3c8160610 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 -- 172.21.15.46:0/2957727350 >> 172.21.15.46:0/2957727350 conn(0x7fe3c80743e0 msgr2=0x7fe3c80d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 -- 172.21.15.46:0/2957727350 shutdown_connections 2024-05-09T03:10:02.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.793+0000 7fe3cff1f640 1 -- 172.21.15.46:0/2957727350 wait complete. 2024-05-09T03:10:02.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.795+0000 7fe3cff1f640 1 Processor -- start 2024-05-09T03:10:02.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.795+0000 7fe3cff1f640 1 -- start start 2024-05-09T03:10:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cff1f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cff1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 0x7fe3c814d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cff1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3c80d57a0 0x7fe3c81eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cff1f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80d15b0 con 0x7fe3c80b77e0 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cff1f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80d5280 con 0x7fe3c80d57a0 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cff1f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c80d1920 con 0x7fe3c80b4af0 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cd4ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 0x7fe3c814d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cd4ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 0x7fe3c814d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55686/0 (socket says 172.21.15.46:55686) 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cd4ba640 1 -- 172.21.15.46:0/3316944452 learned_addr learned my addr 172.21.15.46:0/3316944452 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.796+0000 7fe3ce4bc640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3c80d57a0 0x7fe3c81eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cd4ba640 1 -- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 msgr2=0x7fe3c80cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cd4ba640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cd4ba640 1 -- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3c80d57a0 msgr2=0x7fe3c81eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cd4ba640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3c80d57a0 0x7fe3c81eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cd4ba640 1 -- 172.21.15.46:0/3316944452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c814b6c0 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cd4ba640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 0x7fe3c814d720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800e870 tx=0x7fe3b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8017070 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b800ee60 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c814b9a0 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800cb80 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.797+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c814bf60 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.798+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c806a270 con 0x7fe3c80b77e0 2024-05-09T03:10:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.799+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe3b801a070 con 0x7fe3c80b77e0 2024-05-09T03:10:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.800+0000 7fe3be7fc640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3ac0762d0 0x7fe3ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.800+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe3b8097400 con 0x7fe3c80b77e0 2024-05-09T03:10:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.800+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3ac0762d0 0x7fe3ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:02.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.800+0000 7fe3cdcbb640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3ac0762d0 0x7fe3ac0787e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4033000 tx=0x7fe3c4009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.804+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe3b8060680 con 0x7fe3c80b77e0 2024-05-09T03:10:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.997+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3c815fcc0 con 0x7fe3c80b77e0 2024-05-09T03:10:02.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:02.998+0000 7fe3be7fc640 1 -- 172.21.15.46:0/3316944452 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16533 (secure 0 0 0) 0x7fe3b8064980 con 0x7fe3c80b77e0 2024-05-09T03:10:02.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:02.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:09:35.182593+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:09:31.491289+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T03:09:32.526431+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.000+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3ac0762d0 msgr2=0x7fe3ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.000+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3ac0762d0 0x7fe3ac0787e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4033000 tx=0x7fe3c4009000 comp rx=0 tx=0).stop 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.000+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 msgr2=0x7fe3c814d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.000+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 0x7fe3c814d720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800e870 tx=0x7fe3b800ed30 comp rx=0 tx=0).stop 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 shutdown_connections 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3c80d57a0 0x7fe3c81eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3ac0762d0 0x7fe3ac0787e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3c80b77e0 0x7fe3c814d720 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 --2- 172.21.15.46:0/3316944452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3c80b4af0 0x7fe3c80cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 >> 172.21.15.46:0/3316944452 conn(0x7fe3c80743e0 msgr2=0x7fe3c80b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 shutdown_connections 2024-05-09T03:10:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.001+0000 7fe3cff1f640 1 -- 172.21.15.46:0/3316944452 wait complete. 2024-05-09T03:10:03.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:10:03.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.128+0000 7fa0bcc3b640 1 Processor -- start 2024-05-09T03:10:03.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.128+0000 7fa0bcc3b640 1 -- start start 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0bcc3b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0b80bb770 0x7fa0b80d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0bcc3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0b80d3180 0x7fa0b81642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0bcc3b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b8164900 con 0x7fa0b8163eb0 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0bcc3b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b80d4bb0 con 0x7fa0b80d3180 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0bcc3b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b8164be0 con 0x7fa0b80bb770 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0b6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0b80d3180 0x7fa0b81642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0affff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa0b8163eb0 0x7fa0b80d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43386/0 (socket says 172.21.15.46:43386) 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.129+0000 7fa0affff640 1 -- 172.21.15.46:0/1405011921 learned_addr learned my addr 172.21.15.46:0/1405011921 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b659c640 1 --2- 172.21.15.46:0/1405011921 >> v2:172.21.15.88:3300/0 conn(0x7fa0b80bb770 0x7fa0b80d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/1405011921 >> v2:172.21.15.88:3300/0 conn(0x7fa0b80bb770 msgr2=0x7fa0b80d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b6d9d640 1 --2- 172.21.15.46:0/1405011921 >> v2:172.21.15.88:3300/0 conn(0x7fa0b80bb770 0x7fa0b80d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/1405011921 >> v1:172.21.15.46:6789/0 conn(0x7fa0b8163eb0 legacy=0x7fa0b80d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/1405011921 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b80d4840 con 0x7fa0b80d3180 2024-05-09T03:10:03.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b5d9b640 1 -- 172.21.15.46:0/1405011921 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495015351 0 0) 0x7fa0b8164900 con 0x7fa0b8163eb0 2024-05-09T03:10:03.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.130+0000 7fa0b659c640 1 --2- 172.21.15.46:0/1405011921 >> v2:172.21.15.88:3300/0 conn(0x7fa0b80bb770 0x7fa0b80d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0b6d9d640 1 --2- 172.21.15.46:0/1405011921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80d3180 0x7fa0b81642c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80d6d80 tx=0x7fa0b002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dba73e321335ad2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0b5d9b640 1 -- 172.21.15.46:0/1405011921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0007740 con 0x7fa0b80d3180 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0b5d9b640 1 -- 172.21.15.46:0/1405011921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b00078e0 con 0x7fa0b80d3180 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0b5d9b640 1 -- 172.21.15.46:0/1405011921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0007cb0 con 0x7fa0b80d3180 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/1405011921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80d3180 msgr2=0x7fa0b81642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/1405011921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80d3180 0x7fa0b81642c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80d6d80 tx=0x7fa0b002a680 comp rx=0 tx=0).stop 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/1405011921 shutdown_connections 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/1405011921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80d3180 0x7fa0b81642c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/1405011921 >> v2:172.21.15.88:3300/0 conn(0x7fa0b80bb770 0x7fa0b80d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.131+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/1405011921 >> 172.21.15.46:0/1405011921 conn(0x7fa0b806e370 msgr2=0x7fa0b806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:03.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.132+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/1405011921 shutdown_connections 2024-05-09T03:10:03.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.132+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/1405011921 wait complete. 2024-05-09T03:10:03.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.133+0000 7fa0bcc3b640 1 Processor -- start 2024-05-09T03:10:03.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.134+0000 7fa0bcc3b640 1 -- start start 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0bcc3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0b80bb770 0x7fa0b8156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0bcc3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0b8163eb0 0x7fa0b8153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0bcc3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 0x7fa0b80b9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0bcc3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80d0bc0 con 0x7fa0b80b9210 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0bcc3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b80bbb80 con 0x7fa0b80bb770 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0bcc3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b80d0d40 con 0x7fa0b8163eb0 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 0x7fa0b80b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 0x7fa0b80b9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55704/0 (socket says 172.21.15.46:55704) 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/3220070670 learned_addr learned my addr 172.21.15.46:0/3220070670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b659c640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0b80bb770 0x7fa0b8156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0affff640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0b8163eb0 0x7fa0b8153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0b8163eb0 msgr2=0x7fa0b8153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0b8163eb0 0x7fa0b8153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0b80bb770 msgr2=0x7fa0b8156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0b80bb770 0x7fa0b8156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 -- 172.21.15.46:0/3220070670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b80ad970 con 0x7fa0b80b9210 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.135+0000 7fa0b6d9d640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 0x7fa0b80b9620 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa0b002ac70 tx=0x7fa0b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.136+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b003f070 con 0x7fa0b80b9210 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.136+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0b0031c40 con 0x7fa0b80b9210 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.136+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b80b9b60 con 0x7fa0b80b9210 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.136+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0003990 con 0x7fa0b80b9210 2024-05-09T03:10:03.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.136+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b80ad0d0 con 0x7fa0b80b9210 2024-05-09T03:10:03.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.137+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b806a270 con 0x7fa0b80b9210 2024-05-09T03:10:03.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.137+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa0b0003b30 con 0x7fa0b80b9210 2024-05-09T03:10:03.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.138+0000 7fa0aeffd640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa098076300 0x7fa098078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.138+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa0b00b8880 con 0x7fa0b80b9210 2024-05-09T03:10:03.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.138+0000 7fa0b659c640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa098076300 0x7fa098078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.139+0000 7fa0b659c640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa098076300 0x7fa098078810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80b8880 tx=0x7fa0a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:03.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.142+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa0b80b9b60 con 0x7fa0b80b9210 2024-05-09T03:10:03.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.371+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa0b80d4130 con 0x7fa0b80b9210 2024-05-09T03:10:03.372 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:10:03.371+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:10:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.373+0000 7fa0aeffd640 1 -- 172.21.15.46:0/3220070670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) ==== 161+0+0 (secure 0 0 0) 0x7fa0b80ad970 con 0x7fa0b80b9210 2024-05-09T03:10:03.374 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:10:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa098076300 msgr2=0x7fa098078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa098076300 0x7fa098078810 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80b8880 tx=0x7fa0a8008040 comp rx=0 tx=0).stop 2024-05-09T03:10:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 msgr2=0x7fa0b80b9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 0x7fa0b80b9620 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa0b002ac70 tx=0x7fa0b0004a40 comp rx=0 tx=0).stop 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 shutdown_connections 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa098076300 0x7fa098078810 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0b80b9210 0x7fa0b80b9620 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0b8163eb0 0x7fa0b8153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 --2- 172.21.15.46:0/3220070670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0b80bb770 0x7fa0b8156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 >> 172.21.15.46:0/3220070670 conn(0x7fa0b806e370 msgr2=0x7fa0b80d2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 shutdown_connections 2024-05-09T03:10:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.376+0000 7fa0bcc3b640 1 -- 172.21.15.46:0/3220070670 wait complete. 2024-05-09T03:10:03.378 INFO:tasks.ceph.mds.c.smithi046.stderr: -2168> 2024-05-09T03:09:35.154+0000 7fa389a6f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:10:03.393 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:10:03.437 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:10:03.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.510+0000 7f5c7bd0d640 1 Processor -- start 2024-05-09T03:10:03.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.511+0000 7f5c7bd0d640 1 -- start start 2024-05-09T03:10:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c7bd0d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5c740d1460 0x7f5c740cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c7bd0d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c7406fe50 con 0x7f5c740cfff0 2024-05-09T03:10:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c7bd0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c74070130 con 0x7f5c74070f00 2024-05-09T03:10:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c7bd0d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5c740b4d00 con 0x7f5c740d1460 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c7a2aa640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c740cfff0 0x7f5c74073b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43406/0 (socket says 172.21.15.46:43406) 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c7a2aa640 1 -- 172.21.15.46:0/2716609228 learned_addr learned my addr 172.21.15.46:0/2716609228 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c792a8640 1 --2- 172.21.15.46:0/2716609228 >> v2:172.21.15.88:3300/0 conn(0x7f5c740d1460 0x7f5c740cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.512+0000 7f5c78aa7640 1 -- 172.21.15.46:0/2716609228 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2800132588 0 0) 0x7f5c7406fe50 con 0x7f5c740cfff0 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.513+0000 7f5c78aa7640 1 -- 172.21.15.46:0/2716609228 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c60003520 con 0x7f5c740cfff0 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.513+0000 7f5c792a8640 1 -- 172.21.15.46:0/2716609228 >> v1:172.21.15.88:6789/0 conn(0x7f5c74070f00 legacy=0x7f5c7415c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.513+0000 7f5c78aa7640 1 -- 172.21.15.46:0/2716609228 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754406608 0 0) 0x7f5c74070130 con 0x7f5c74070f00 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.513+0000 7f5c792a8640 1 -- 172.21.15.46:0/2716609228 >> v1:172.21.15.46:6789/0 conn(0x7f5c740cfff0 legacy=0x7f5c74073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.513+0000 7f5c792a8640 1 -- 172.21.15.46:0/2716609228 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c740b4990 con 0x7f5c740d1460 2024-05-09T03:10:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.513+0000 7f5c792a8640 1 --2- 172.21.15.46:0/2716609228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c740cfab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400a940 tx=0x7f5c6402b550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10401377fc7c5b02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c78aa7640 1 -- 172.21.15.46:0/2716609228 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400b040 con 0x7f5c740d1460 2024-05-09T03:10:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c78aa7640 1 -- 172.21.15.46:0/2716609228 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c64030710 con 0x7f5c740d1460 2024-05-09T03:10:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c78aa7640 1 -- 172.21.15.46:0/2716609228 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c64030ae0 con 0x7f5c740d1460 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/2716609228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 msgr2=0x7f5c740cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/2716609228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c740cfab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400a940 tx=0x7f5c6402b550 comp rx=0 tx=0).stop 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/2716609228 shutdown_connections 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/2716609228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c740cfab0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.514+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/2716609228 >> 172.21.15.46:0/2716609228 conn(0x7f5c740b3690 msgr2=0x7f5c740b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.515+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/2716609228 shutdown_connections 2024-05-09T03:10:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.515+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/2716609228 wait complete. 2024-05-09T03:10:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.516+0000 7f5c7bd0d640 1 Processor -- start 2024-05-09T03:10:03.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.516+0000 7f5c7bd0d640 1 -- start start 2024-05-09T03:10:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7bd0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c74070f00 0x7f5c74155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7bd0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 0x7f5c74148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7bd0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c741597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7bd0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5c740743d0 con 0x7f5c740cfff0 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7bd0d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5c74074250 con 0x7f5c740d1460 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7bd0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5c740732b0 con 0x7f5c74070f00 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 0x7f5c74148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 0x7f5c74148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55738/0 (socket says 172.21.15.46:55738) 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 -- 172.21.15.46:0/955388119 learned_addr learned my addr 172.21.15.46:0/955388119 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c79aa9640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c74070f00 0x7f5c74155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c7a2aa640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c741597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 -- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c74070f00 msgr2=0x7f5c74155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c74070f00 0x7f5c74155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 -- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 msgr2=0x7f5c741597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c741597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 -- 172.21.15.46:0/955388119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c74155d90 con 0x7f5c740cfff0 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.518+0000 7f5c792a8640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 0x7f5c74148340 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400a940 tx=0x7f5c640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.519+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6403f030 con 0x7f5c740cfff0 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.519+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c64030510 con 0x7f5c740cfff0 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.519+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c74154430 con 0x7f5c740cfff0 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.519+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c64033c70 con 0x7f5c740cfff0 2024-05-09T03:10:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.519+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c741549f0 con 0x7f5c740cfff0 2024-05-09T03:10:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.519+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c74154180 con 0x7f5c740cfff0 2024-05-09T03:10:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.520+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5c6403b430 con 0x7f5c740cfff0 2024-05-09T03:10:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.521+0000 7f5c6a7fc640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c58076300 0x7f5c58078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.521+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5795+0+0 (secure 0 0 0) 0x7f5c640b7bf0 con 0x7f5c740cfff0 2024-05-09T03:10:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.521+0000 7f5c79aa9640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c58076300 0x7f5c58078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.522+0000 7f5c79aa9640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c58076300 0x7f5c58078810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000f60 tx=0x7f5c70009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.525+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5c74155d90 con 0x7f5c740cfff0 2024-05-09T03:10:03.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:03.754+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5c7406a270 con 0x7f5c740cfff0 2024-05-09T03:10:04.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.375+0000 7f5c6a7fc640 1 -- 172.21.15.46:0/955388119 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) ==== 104+0+0 (secure 0 0 0) 0x7f5c6400a070 con 0x7f5c740cfff0 2024-05-09T03:10:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c58076300 msgr2=0x7f5c58078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c58076300 0x7f5c58078810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000f60 tx=0x7f5c70009040 comp rx=0 tx=0).stop 2024-05-09T03:10:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 msgr2=0x7f5c74148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 0x7f5c74148340 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400a940 tx=0x7f5c640046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 shutdown_connections 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c740d1460 0x7f5c741597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c58076300 0x7f5c58078810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c740cfff0 0x7f5c74148340 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 --2- 172.21.15.46:0/955388119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c74070f00 0x7f5c74155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 >> 172.21.15.46:0/955388119 conn(0x7f5c740b3690 msgr2=0x7f5c740b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 shutdown_connections 2024-05-09T03:10:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.377+0000 7f5c7bd0d640 1 -- 172.21.15.46:0/955388119 wait complete. 2024-05-09T03:10:04.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:10:04.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.514+0000 7f83df615640 1 Processor -- start 2024-05-09T03:10:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.514+0000 7f83df615640 1 -- start start 2024-05-09T03:10:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83df615640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f83d80b77d0 0x7f83d80b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83df615640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83d80b23d0 0x7f83d80b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83df615640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d8161f80 con 0x7f83d80b3a90 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83df615640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f83d8161d30 con 0x7f83d80b23d0 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83df615640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f83d8162260 con 0x7f83d80b77d0 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83ddbb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83d80b23d0 0x7f83d80b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83ddbb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83d80b23d0 0x7f83d80b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55752/0 (socket says 172.21.15.46:55752) 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83ddbb2640 1 -- 172.21.15.46:0/3867915684 learned_addr learned my addr 172.21.15.46:0/3867915684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.515+0000 7f83dcbb0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f83d80b3a90 0x7f83d80b3ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43410/0 (socket says 172.21.15.46:43410) 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83ddbb2640 1 -- 172.21.15.46:0/3867915684 >> v2:172.21.15.88:3300/0 conn(0x7f83d80b77d0 msgr2=0x7f83d80b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83dd3b1640 1 --2- 172.21.15.46:0/3867915684 >> v2:172.21.15.88:3300/0 conn(0x7f83d80b77d0 0x7f83d80b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83ddbb2640 1 --2- 172.21.15.46:0/3867915684 >> v2:172.21.15.88:3300/0 conn(0x7f83d80b77d0 0x7f83d80b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83cffff640 1 -- 172.21.15.46:0/3867915684 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096506605 0 0) 0x7f83d8161f80 con 0x7f83d80b3a90 2024-05-09T03:10:04.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83ddbb2640 1 -- 172.21.15.46:0/3867915684 >> v1:172.21.15.46:6789/0 conn(0x7f83d80b3a90 legacy=0x7f83d80b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:04.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83ddbb2640 1 -- 172.21.15.46:0/3867915684 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d8162540 con 0x7f83d80b23d0 2024-05-09T03:10:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.516+0000 7f83ddbb2640 1 --2- 172.21.15.46:0/3867915684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d80b27e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f83d80734a0 tx=0x7f83d002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aca70a6562c2df3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83cffff640 1 -- 172.21.15.46:0/3867915684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0036070 con 0x7f83d80b23d0 2024-05-09T03:10:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83cffff640 1 -- 172.21.15.46:0/3867915684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d0031600 con 0x7f83d80b23d0 2024-05-09T03:10:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83cffff640 1 -- 172.21.15.46:0/3867915684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00319d0 con 0x7f83d80b23d0 2024-05-09T03:10:04.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 -- 172.21.15.46:0/3867915684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 msgr2=0x7f83d80b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 --2- 172.21.15.46:0/3867915684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d80b27e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f83d80734a0 tx=0x7f83d002a6a0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 -- 172.21.15.46:0/3867915684 shutdown_connections 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 --2- 172.21.15.46:0/3867915684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d80b27e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 --2- 172.21.15.46:0/3867915684 >> v2:172.21.15.88:3300/0 conn(0x7f83d80b77d0 0x7f83d80b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 -- 172.21.15.46:0/3867915684 >> 172.21.15.46:0/3867915684 conn(0x7f83d809b7d0 msgr2=0x7f83d809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.517+0000 7f83df615640 1 -- 172.21.15.46:0/3867915684 shutdown_connections 2024-05-09T03:10:04.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.518+0000 7f83df615640 1 -- 172.21.15.46:0/3867915684 wait complete. 2024-05-09T03:10:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.519+0000 7f83df615640 1 Processor -- start 2024-05-09T03:10:04.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.519+0000 7f83df615640 1 -- start start 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83d80b3a90 0x7f83d815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83d80b77d0 0x7f83d80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83d80af8c0 con 0x7f83d80b23d0 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f83d80cf4c0 con 0x7f83d80b77d0 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f83d806f920 con 0x7f83d80b3a90 2024-05-09T03:10:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d815aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55762/0 (socket says 172.21.15.46:55762) 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 -- 172.21.15.46:0/2931801290 learned_addr learned my addr 172.21.15.46:0/2931801290 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83ddbb2640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83d80b77d0 0x7f83d80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dcbb0640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83d80b3a90 0x7f83d815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 -- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83d80b3a90 msgr2=0x7f83d815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83d80b3a90 0x7f83d815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 -- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83d80b77d0 msgr2=0x7f83d80b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83d80b77d0 0x7f83d80b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 -- 172.21.15.46:0/2931801290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d815c600 con 0x7f83d80b23d0 2024-05-09T03:10:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dcbb0640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83d80b3a90 0x7f83d815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83ddbb2640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83d80b77d0 0x7f83d80b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83dd3b1640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d815aab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f83d400c5f0 tx=0x7f83d400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4015070 con 0x7f83d80b23d0 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d4004220 con 0x7f83d80b23d0 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4003610 con 0x7f83d80b23d0 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.521+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d815c8e0 con 0x7f83d80b23d0 2024-05-09T03:10:04.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.522+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d815cea0 con 0x7f83d80b23d0 2024-05-09T03:10:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.523+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d806a270 con 0x7f83d80b23d0 2024-05-09T03:10:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.523+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f83d40037b0 con 0x7f83d80b23d0 2024-05-09T03:10:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.524+0000 7f83cdffb640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83bc0762d0 0x7f83bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.524+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5795+0+0 (secure 0 0 0) 0x7f83d40973b0 con 0x7f83d80b23d0 2024-05-09T03:10:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.524+0000 7f83dcbb0640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83bc0762d0 0x7f83bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.525+0000 7f83dcbb0640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83bc0762d0 0x7f83bc0787e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000f60 tx=0x7f83c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.528+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f83d4060cb0 con 0x7f83d80b23d0 2024-05-09T03:10:04.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:04.718+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f83d80cf730 con 0x7f83d80b23d0 2024-05-09T03:10:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.653+0000 7f83cdffb640 1 -- 172.21.15.46:0/2931801290 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v22) ==== 181+0+0 (secure 0 0 0) 0x7f83d40605b0 con 0x7f83d80b23d0 2024-05-09T03:10:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.656+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83bc0762d0 msgr2=0x7f83bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.656+0000 7f83df615640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83bc0762d0 0x7f83bc0787e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f83c8000f60 tx=0x7f83c8009040 comp rx=0 tx=0).stop 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.656+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 msgr2=0x7f83d815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.656+0000 7f83df615640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d815aab0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f83d400c5f0 tx=0x7f83d400cab0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 shutdown_connections 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83d80b77d0 0x7f83d80b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83d80b3a90 0x7f83d815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83bc0762d0 0x7f83bc0787e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 --2- 172.21.15.46:0/2931801290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d80b23d0 0x7f83d815aab0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 >> 172.21.15.46:0/2931801290 conn(0x7f83d809b7d0 msgr2=0x7f83d8075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 shutdown_connections 2024-05-09T03:10:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.657+0000 7f83df615640 1 -- 172.21.15.46:0/2931801290 wait complete. 2024-05-09T03:10:05.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:10:05.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:10:05.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:10:05.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.797+0000 7f4cbc82b640 1 Processor -- start 2024-05-09T03:10:05.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.797+0000 7f4cbc82b640 1 -- start start 2024-05-09T03:10:05.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cbc82b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cb80b9240 0x7f4cb8160250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:05.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cbc82b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb80b7e50 0x7f4cb80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:05.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cbc82b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb8164340 con 0x7f4cb8163df0 2024-05-09T03:10:05.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cbc82b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4cb80d4370 con 0x7f4cb80b9240 2024-05-09T03:10:05.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cbc82b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cb8164620 con 0x7f4cb80b7e50 2024-05-09T03:10:05.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cb80b9240 0x7f4cb8160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cb80b9240 0x7f4cb8160250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32780/0 (socket says 172.21.15.46:32780) 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 -- 172.21.15.46:0/2507627921 learned_addr learned my addr 172.21.15.46:0/2507627921 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb6d9d640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb80b7e50 0x7f4cb80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 -- 172.21.15.46:0/2507627921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb80b7e50 msgr2=0x7f4cb80d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb80b7e50 0x7f4cb80d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 -- 172.21.15.46:0/2507627921 >> v1:172.21.15.88:6789/0 conn(0x7f4cb8163df0 legacy=0x7f4cb80b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb659c640 1 -- 172.21.15.46:0/2507627921 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb8164900 con 0x7f4cb80b9240 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.799+0000 7f4cb6d9d640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb80b7e50 0x7f4cb80d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:05.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.800+0000 7f4cb659c640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb8160250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009870 tx=0x7f4ca802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2015274319354693 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:05.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.800+0000 7f4cb559a640 1 -- 172.21.15.46:0/2507627921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8037070 con 0x7f4cb80b9240 2024-05-09T03:10:05.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.800+0000 7f4cb559a640 1 -- 172.21.15.46:0/2507627921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca8032440 con 0x7f4cb80b9240 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.800+0000 7f4cb559a640 1 -- 172.21.15.46:0/2507627921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8032810 con 0x7f4cb80b9240 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 -- 172.21.15.46:0/2507627921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 msgr2=0x7f4cb8160250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb8160250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009870 tx=0x7f4ca802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 -- 172.21.15.46:0/2507627921 shutdown_connections 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb80b7e50 0x7f4cb80d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/2507627921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb8160250 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 -- 172.21.15.46:0/2507627921 >> 172.21.15.46:0/2507627921 conn(0x7f4cb80d2920 msgr2=0x7f4cb80d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 -- 172.21.15.46:0/2507627921 shutdown_connections 2024-05-09T03:10:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.801+0000 7f4cbc82b640 1 -- 172.21.15.46:0/2507627921 wait complete. 2024-05-09T03:10:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.803+0000 7f4cbc82b640 1 Processor -- start 2024-05-09T03:10:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.803+0000 7f4cbc82b640 1 -- start start 2024-05-09T03:10:05.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cbc82b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb80cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cbc82b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb8163df0 0x7f4cb8160250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cbc82b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8151210 0x7f4cb8162d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cbc82b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb80b5550 con 0x7f4cb80b9240 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cbc82b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb80b53d0 con 0x7f4cb8151210 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cbc82b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cb80d0a00 con 0x7f4cb8163df0 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cb659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb80cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cb659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb80cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32792/0 (socket says 172.21.15.46:32792) 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.804+0000 7f4cb659c640 1 -- 172.21.15.46:0/1843370976 learned_addr learned my addr 172.21.15.46:0/1843370976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb6d9d640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8151210 0x7f4cb8162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb5d9b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb8163df0 0x7f4cb8160250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb659c640 1 -- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb8163df0 msgr2=0x7f4cb8160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb659c640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb8163df0 0x7f4cb8160250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb659c640 1 -- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8151210 msgr2=0x7f4cb8162d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:05.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb659c640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8151210 0x7f4cb8162d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb659c640 1 -- 172.21.15.46:0/1843370976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb8160a80 con 0x7f4cb80b9240 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb5d9b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb8163df0 0x7f4cb8160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cb659c640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb80cab40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009870 tx=0x7f4ca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8041070 con 0x7f4cb80b9240 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca803c870 con 0x7f4cb80b9240 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb8160d60 con 0x7f4cb80b9240 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8037040 con 0x7f4cb80b9240 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.805+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb8161320 con 0x7f4cb80b9240 2024-05-09T03:10:05.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.806+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb806a270 con 0x7f4cb80b9240 2024-05-09T03:10:05.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.807+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4ca8044070 con 0x7f4cb80b9240 2024-05-09T03:10:05.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.808+0000 7f4ca6ffd640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c980762d0 0x7f4c980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:05.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.808+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5357+0+0 (secure 0 0 0) 0x7f4ca80b8430 con 0x7f4cb80b9240 2024-05-09T03:10:05.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.808+0000 7f4cb5d9b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c980762d0 0x7f4c980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:05.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.808+0000 7f4cb5d9b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c980762d0 0x7f4c980787e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006fd0 tx=0x7f4cac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:05.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:05.812+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4ca80818a0 con 0x7f4cb80b9240 2024-05-09T03:10:06.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.005+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4cb80d4150 con 0x7f4cb80b9240 2024-05-09T03:10:06.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.661+0000 7f4ca6ffd640 1 -- 172.21.15.46:0/1843370976 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v23) ==== 169+0+0 (secure 0 0 0) 0x7f4ca8085ba0 con 0x7f4cb80b9240 2024-05-09T03:10:06.662 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:10:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.664+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c980762d0 msgr2=0x7f4c980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.664+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c980762d0 0x7f4c980787e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006fd0 tx=0x7f4cac008040 comp rx=0 tx=0).stop 2024-05-09T03:10:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.664+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 msgr2=0x7f4cb80cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:06.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.664+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb80cab40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009870 tx=0x7f4ca80046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 shutdown_connections 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb8151210 0x7f4cb8162d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb8163df0 0x7f4cb8160250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c980762d0 0x7f4c980787e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 --2- 172.21.15.46:0/1843370976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb80b9240 0x7f4cb80cab40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 >> 172.21.15.46:0/1843370976 conn(0x7f4cb80d2920 msgr2=0x7f4cb80bac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 shutdown_connections 2024-05-09T03:10:06.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.665+0000 7f4cbc82b640 1 -- 172.21.15.46:0/1843370976 wait complete. 2024-05-09T03:10:06.680 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:10:06.680 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:10:06.680 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:10:06.680 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:06.680 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:10:06.680 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T03:10:06.681 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:10:06.681 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:06.681 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:10:06.681 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T03:10:06.681 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:10:06.681 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:06.681 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:10:06.682 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T03:10:06.682 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:10:06.682 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:06.693 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:10:06.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:10:06.695 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:10:06.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:10:06.730 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:10:06.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:10:06.734 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:10:06.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:10:06.737 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:10:06.737 DEBUG:teuthology.parallel:result is None 2024-05-09T03:10:06.738 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:10:06.738 DEBUG:teuthology.parallel:result is None 2024-05-09T03:10:06.776 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:10:06.777 DEBUG:teuthology.parallel:result is None 2024-05-09T03:10:06.778 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:10:06.778 DEBUG:teuthology.parallel:result is None 2024-05-09T03:10:06.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T03:10:06.833 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:10:06.833 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:10:06.869 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:10:06.869 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:10:06.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.901+0000 7f253dffa640 1 Processor -- start 2024-05-09T03:10:06.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.901+0000 7f253dffa640 1 -- start start 2024-05-09T03:10:06.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f253dffa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380d1110 0x7f25380d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f253dffa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25380d1920 0x7f25380d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f253dffa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25380b92c0 con 0x7f25380b32c0 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f253dffa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f25380b2f10 con 0x7f25380d1920 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f253dffa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f25380b9560 con 0x7f25380d1110 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f2536ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25380d1920 0x7f25380d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f2536ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25380d1920 0x7f25380d1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32824/0 (socket says 172.21.15.46:32824) 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.903+0000 7f2536ffd640 1 -- 172.21.15.46:0/785685237 learned_addr learned my addr 172.21.15.46:0/785685237 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.904+0000 7f25377fe640 1 --2- 172.21.15.46:0/785685237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380d1110 0x7f25380d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.904+0000 7f2536ffd640 1 -- 172.21.15.46:0/785685237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380d1110 msgr2=0x7f25380d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.904+0000 7f2536ffd640 1 --2- 172.21.15.46:0/785685237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380d1110 0x7f25380d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.904+0000 7f2536ffd640 1 -- 172.21.15.46:0/785685237 >> v1:172.21.15.88:6789/0 conn(0x7f25380b32c0 legacy=0x7f25380b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.904+0000 7f2536ffd640 1 -- 172.21.15.46:0/785685237 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25380cfb20 con 0x7f25380d1920 2024-05-09T03:10:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.905+0000 7f2536ffd640 1 --2- 172.21.15.46:0/785685237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380d1920 0x7f25380d1d30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f252c009870 tx=0x7f252c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6369a9e14e7aec41 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.905+0000 7f25367fc640 1 -- 172.21.15.46:0/785685237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c037070 con 0x7f25380d1920 2024-05-09T03:10:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.905+0000 7f25367fc640 1 -- 172.21.15.46:0/785685237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252c032440 con 0x7f25380d1920 2024-05-09T03:10:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.905+0000 7f25367fc640 1 -- 172.21.15.46:0/785685237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c032810 con 0x7f25380d1920 2024-05-09T03:10:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.905+0000 7f253dffa640 1 -- 172.21.15.46:0/785685237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380d1920 msgr2=0x7f25380d1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.905+0000 7f253dffa640 1 --2- 172.21.15.46:0/785685237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380d1920 0x7f25380d1d30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f252c009870 tx=0x7f252c02a130 comp rx=0 tx=0).stop 2024-05-09T03:10:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.906+0000 7f253dffa640 1 -- 172.21.15.46:0/785685237 shutdown_connections 2024-05-09T03:10:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.906+0000 7f253dffa640 1 --2- 172.21.15.46:0/785685237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380d1920 0x7f25380d1d30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.906+0000 7f253dffa640 1 --2- 172.21.15.46:0/785685237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380d1110 0x7f25380d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.906+0000 7f253dffa640 1 -- 172.21.15.46:0/785685237 >> 172.21.15.46:0/785685237 conn(0x7f25380d05f0 msgr2=0x7f25380d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.906+0000 7f253dffa640 1 -- 172.21.15.46:0/785685237 shutdown_connections 2024-05-09T03:10:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.906+0000 7f253dffa640 1 -- 172.21.15.46:0/785685237 wait complete. 2024-05-09T03:10:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.907+0000 7f253dffa640 1 Processor -- start 2024-05-09T03:10:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.908+0000 7f253dffa640 1 -- start start 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f253dffa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380b32c0 0x7f25380a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f253dffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25380d1920 0x7f25380cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f253dffa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 0x7f2538161420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f253dffa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25380b8f80 con 0x7f25380cc780 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f253dffa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f25380b8e00 con 0x7f25380d1920 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f253dffa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f25380b3de0 con 0x7f25380b32c0 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 0x7f2538161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 0x7f2538161420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32830/0 (socket says 172.21.15.46:32830) 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 -- 172.21.15.46:0/2295212350 learned_addr learned my addr 172.21.15.46:0/2295212350 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f25377fe640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380b32c0 0x7f25380a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2536ffd640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25380d1920 0x7f25380cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 -- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380b32c0 msgr2=0x7f25380a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380b32c0 0x7f25380a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 -- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25380d1920 msgr2=0x7f25380cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25380d1920 0x7f25380cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2537fff640 1 -- 172.21.15.46:0/2295212350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25380a1910 con 0x7f25380cc780 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.909+0000 7f2536ffd640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25380d1920 0x7f25380cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.910+0000 7f2537fff640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 0x7f2538161420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f253000c730 tx=0x7f253000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.910+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530003b70 con 0x7f25380cc780 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.910+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2530003d10 con 0x7f25380cc780 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.910+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253000aa70 con 0x7f25380cc780 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.910+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253815f0a0 con 0x7f25380cc780 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.910+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253815f660 con 0x7f25380cc780 2024-05-09T03:10:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.911+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f253806a270 con 0x7f25380cc780 2024-05-09T03:10:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.913+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f253000ac10 con 0x7f25380cc780 2024-05-09T03:10:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.914+0000 7f251ffff640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2524076300 0x7f2524078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.914+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 4943+0+0 (secure 0 0 0) 0x7f2530097b40 con 0x7f25380cc780 2024-05-09T03:10:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.914+0000 7f25377fe640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2524076300 0x7f2524078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.914+0000 7f25377fe640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2524076300 0x7f2524078810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2528006fd0 tx=0x7f2528008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:06.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:06.917+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f25380a1910 con 0x7f25380cc780 2024-05-09T03:10:06.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-05-09T03:10:06.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-05-09T03:10:07.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.124+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f253815d9b0 con 0x7f25380cc780 2024-05-09T03:10:07.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.675+0000 7f251ffff640 1 -- 172.21.15.46:0/2295212350 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v25) ==== 97+0+0 (secure 0 0 0) 0x7f2530061880 con 0x7f25380cc780 2024-05-09T03:10:07.676 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-05-09T03:10:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2524076300 msgr2=0x7f2524078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2524076300 0x7f2524078810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2528006fd0 tx=0x7f2528008040 comp rx=0 tx=0).stop 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 msgr2=0x7f2538161420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 0x7f2538161420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f253000c730 tx=0x7f253000cbf0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 shutdown_connections 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2524076300 0x7f2524078810 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25380cc780 0x7f2538161420 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25380d1920 0x7f25380cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 --2- 172.21.15.46:0/2295212350 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f25380b32c0 0x7f25380a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.677+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 >> 172.21.15.46:0/2295212350 conn(0x7f25380d05f0 msgr2=0x7f25380b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.678+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 shutdown_connections 2024-05-09T03:10:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.678+0000 7f253dffa640 1 -- 172.21.15.46:0/2295212350 wait complete. 2024-05-09T03:10:07.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T03:10:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.807+0000 7fb2ffb3f640 1 Processor -- start 2024-05-09T03:10:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.808+0000 7fb2ffb3f640 1 -- start start 2024-05-09T03:10:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2ffb3f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb2f8164c70 0x7fb2f81615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2ffb3f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2ffb3f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2f80d2480 0x7fb2f80d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2ffb3f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f8161150 con 0x7fb2f80d2480 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2ffb3f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f8160b20 con 0x7fb2f8164c70 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2ffb3f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f80d15f0 con 0x7fb2f815dfe0 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2fe0dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2f80d2480 0x7fb2f80d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2fe0dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2f80d2480 0x7fb2f80d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32832/0 (socket says 172.21.15.46:32832) 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2fe0dc640 1 -- 172.21.15.46:0/294351549 learned_addr learned my addr 172.21.15.46:0/294351549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2fd8db640 1 --2- 172.21.15.46:0/294351549 >> v2:172.21.15.88:3300/0 conn(0x7fb2f8164c70 0x7fb2f81615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.809+0000 7fb2fd0da640 1 --2- 172.21.15.46:0/294351549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fe0dc640 1 -- 172.21.15.46:0/294351549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 msgr2=0x7fb2f80d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fe0dc640 1 --2- 172.21.15.46:0/294351549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fe0dc640 1 -- 172.21.15.46:0/294351549 >> v2:172.21.15.88:3300/0 conn(0x7fb2f8164c70 msgr2=0x7fb2f81615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fe0dc640 1 --2- 172.21.15.46:0/294351549 >> v2:172.21.15.88:3300/0 conn(0x7fb2f8164c70 0x7fb2f81615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fe0dc640 1 -- 172.21.15.46:0/294351549 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f80d18d0 con 0x7fb2f80d2480 2024-05-09T03:10:07.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fd8db640 1 --2- 172.21.15.46:0/294351549 >> v2:172.21.15.88:3300/0 conn(0x7fb2f8164c70 0x7fb2f81615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fe0dc640 1 --2- 172.21.15.46:0/294351549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f80d2890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80b8900 tx=0x7fb2f002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506fefb08803f3a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fc8d9640 1 -- 172.21.15.46:0/294351549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0037070 con 0x7fb2f80d2480 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fc8d9640 1 -- 172.21.15.46:0/294351549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f0032440 con 0x7fb2f80d2480 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.810+0000 7fb2fc8d9640 1 -- 172.21.15.46:0/294351549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0032810 con 0x7fb2f80d2480 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/294351549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 msgr2=0x7fb2f80d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/294351549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f80d2890 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80b8900 tx=0x7fb2f002a140 comp rx=0 tx=0).stop 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/294351549 shutdown_connections 2024-05-09T03:10:07.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/294351549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f80d2890 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/294351549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/294351549 >> v2:172.21.15.88:3300/0 conn(0x7fb2f8164c70 0x7fb2f81615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/294351549 >> 172.21.15.46:0/294351549 conn(0x7fb2f8073db0 msgr2=0x7fb2f80741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/294351549 shutdown_connections 2024-05-09T03:10:07.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.811+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/294351549 wait complete. 2024-05-09T03:10:07.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.813+0000 7fb2ffb3f640 1 Processor -- start 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.813+0000 7fb2ffb3f640 1 -- start start 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2ffb3f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f8151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2ffb3f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2ffb3f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2f8164c70 0x7fb2f80cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2ffb3f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f815e3f0 con 0x7fb2f80d2480 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2ffb3f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f80d1300 con 0x7fb2f8164c70 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2ffb3f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f80d2dd0 con 0x7fb2f815dfe0 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2fd8db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f8151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2fd8db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f8151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32842/0 (socket says 172.21.15.46:32842) 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.814+0000 7fb2fd8db640 1 -- 172.21.15.46:0/521249055 learned_addr learned my addr 172.21.15.46:0/521249055 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:07.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd0da640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd8db640 1 -- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 msgr2=0x7fb2f80cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd8db640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd8db640 1 -- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2f8164c70 msgr2=0x7fb2f80cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fe0dc640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2f8164c70 0x7fb2f80cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd8db640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2f8164c70 0x7fb2f80cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd8db640 1 -- 172.21.15.46:0/521249055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f81526c0 con 0x7fb2f80d2480 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2fd8db640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f8151d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8007ef0 tx=0x7fb2e800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e801a070 con 0x7fb2f80d2480 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e8003b90 con 0x7fb2f80d2480 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f80ccd50 con 0x7fb2f80d2480 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8004030 con 0x7fb2f80d2480 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.815+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f80cd530 con 0x7fb2f80d2480 2024-05-09T03:10:07.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.816+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f808cfd0 con 0x7fb2f80d2480 2024-05-09T03:10:07.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.817+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb2e801d070 con 0x7fb2f80d2480 2024-05-09T03:10:07.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.817+0000 7fb2f67fc640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb2e40762d0 0x7fb2e40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:07.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.817+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7fb2e80974b0 con 0x7fb2f80d2480 2024-05-09T03:10:07.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.818+0000 7fb2fd0da640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb2e40762d0 0x7fb2e40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:07.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.818+0000 7fb2fd0da640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb2e40762d0 0x7fb2e40787e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec006fd0 tx=0x7fb2ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:07.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:07.822+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb2f808cfd0 con 0x7fb2f80d2480 2024-05-09T03:10:08.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.091+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb2f806a270 con 0x7fb2f80d2480 2024-05-09T03:10:08.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.092+0000 7fb2f67fc640 1 -- 172.21.15.46:0/521249055 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) ==== 80+0+6645 (secure 0 0 0) 0x7fb2f81526c0 con 0x7fb2f80d2480 2024-05-09T03:10:08.093 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:08.093 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:08.093 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-05-09T03:10:08.093 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.093 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-05-09T03:10:08.093 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g==", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw==", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.094 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw==", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.095 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw==", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw==", 2024-05-09T03:10:08.096 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA==", 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg==", 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA==", 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.098 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g==", 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.099 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA==", 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ==", 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.100 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw==", 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.101 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw==", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ==", 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.102 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg==", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.103 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg==", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA1PjxmuVPDJhAAHOgxPpTSEd3pibPqobcHCg==", 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mds", 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWPjxmXIDTBhAAQevJZS6bPdJPiRnx/moQ5A==", 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mds" 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.105 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mgr", 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWPjxmPI/TBhAAlHRDV3ZrBPi+kEsPZ7OlTQ==", 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mgr" 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-osd", 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWPjxms5zTBhAAuBT6J6KlhzW52aC7lcnsHw==", 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-osd" 2024-05-09T03:10:08.106 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd", 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWPjxmS6XTBhAAMrNMDaYmNp+fObZfwU107A==", 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd" 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWPjxmVq7TBhAA7oFhNpVO0plkCPIRwIGKXQ==", 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.107 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rgw", 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBWPjxmtrbTBhAAbq7TksFBXMq9XdRT3ImHsg==", 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rgw" 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg==", 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.108 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ==", 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-05-09T03:10:08.109 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb2e40762d0 msgr2=0x7fb2e40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb2e40762d0 0x7fb2e40787e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec006fd0 tx=0x7fb2ec008040 comp rx=0 tx=0).stop 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 msgr2=0x7fb2f8151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f8151d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8007ef0 tx=0x7fb2e800d4a0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 shutdown_connections 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2f8164c70 0x7fb2f80cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2f815dfe0 0x7fb2f80cf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb2e40762d0 0x7fb2e40787e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 --2- 172.21.15.46:0/521249055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f80d2480 0x7fb2f8151d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 >> 172.21.15.46:0/521249055 conn(0x7fb2f8073db0 msgr2=0x7fb2f815fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 shutdown_connections 2024-05-09T03:10:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.094+0000 7fb2ffb3f640 1 -- 172.21.15.46:0/521249055 wait complete. 2024-05-09T03:10:08.112 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-05-09T03:10:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.227+0000 7fc78a572640 1 Processor -- start 2024-05-09T03:10:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.227+0000 7fc78a572640 1 -- start start 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc78a572640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc7840b7e70 0x7fc7840d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc78a572640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7840b95a0 0x7fc7840b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc78a572640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7840d4a60 con 0x7fc7840d2eb0 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc78a572640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc7840d4d40 con 0x7fc7840b95a0 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc78a572640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc7840d5020 con 0x7fc7840b7e70 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7840b95a0 0x7fc7840b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7840b95a0 0x7fc7840b99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32850/0 (socket says 172.21.15.46:32850) 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 -- 172.21.15.46:0/3620379205 learned_addr learned my addr 172.21.15.46:0/3620379205 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc7837fe640 1 --2- 172.21.15.46:0/3620379205 >> v2:172.21.15.88:3300/0 conn(0x7fc7840b7e70 0x7fc7840d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 -- 172.21.15.46:0/3620379205 >> v2:172.21.15.88:3300/0 conn(0x7fc7840b7e70 msgr2=0x7fc7840d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 --2- 172.21.15.46:0/3620379205 >> v2:172.21.15.88:3300/0 conn(0x7fc7840b7e70 0x7fc7840d32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 -- 172.21.15.46:0/3620379205 >> v1:172.21.15.88:6789/0 conn(0x7fc7840d2eb0 legacy=0x7fc7840b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.229+0000 7fc788b0f640 1 -- 172.21.15.46:0/3620379205 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7840d5300 con 0x7fc7840b95a0 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.230+0000 7fc788b0f640 1 --2- 172.21.15.46:0/3620379205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840b95a0 0x7fc7840b99b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc7840d03c0 tx=0x7fc77c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed4dacc557f4c7c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.230+0000 7fc782ffd640 1 -- 172.21.15.46:0/3620379205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c036070 con 0x7fc7840b95a0 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.230+0000 7fc782ffd640 1 -- 172.21.15.46:0/3620379205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc77c031600 con 0x7fc7840b95a0 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.230+0000 7fc782ffd640 1 -- 172.21.15.46:0/3620379205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0319d0 con 0x7fc7840b95a0 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 -- 172.21.15.46:0/3620379205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840b95a0 msgr2=0x7fc7840b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 --2- 172.21.15.46:0/3620379205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840b95a0 0x7fc7840b99b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc7840d03c0 tx=0x7fc77c02a6a0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 -- 172.21.15.46:0/3620379205 shutdown_connections 2024-05-09T03:10:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 --2- 172.21.15.46:0/3620379205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840b95a0 0x7fc7840b99b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 --2- 172.21.15.46:0/3620379205 >> v2:172.21.15.88:3300/0 conn(0x7fc7840b7e70 0x7fc7840d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 -- 172.21.15.46:0/3620379205 >> 172.21.15.46:0/3620379205 conn(0x7fc78408d390 msgr2=0x7fc78408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 -- 172.21.15.46:0/3620379205 shutdown_connections 2024-05-09T03:10:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.231+0000 7fc78a572640 1 -- 172.21.15.46:0/3620379205 wait complete. 2024-05-09T03:10:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.233+0000 7fc78a572640 1 Processor -- start 2024-05-09T03:10:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.233+0000 7fc78a572640 1 -- start start 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc78a572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7840b95a0 0x7fc7841e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc78a572640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 0x7fc7840cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc78a572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7840cfa50 0x7fc78414c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc78a572640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc7840b5360 con 0x7fc7840d2eb0 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc78a572640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc7840b51e0 con 0x7fc7840b95a0 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc78a572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc784163bd0 con 0x7fc7840cfa50 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc7837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 0x7fc7840cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc7837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 0x7fc7840cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32864/0 (socket says 172.21.15.46:32864) 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc7837fe640 1 -- 172.21.15.46:0/2852166276 learned_addr learned my addr 172.21.15.46:0/2852166276 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.234+0000 7fc783fff640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7840b95a0 0x7fc7841e8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc788b0f640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7840cfa50 0x7fc78414c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc7837fe640 1 -- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7840cfa50 msgr2=0x7fc78414c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc7837fe640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7840cfa50 0x7fc78414c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc7837fe640 1 -- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7840b95a0 msgr2=0x7fc7841e8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc7837fe640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7840b95a0 0x7fc7841e8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc7837fe640 1 -- 172.21.15.46:0/2852166276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7841e9170 con 0x7fc7840d2eb0 2024-05-09T03:10:08.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc783fff640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7840b95a0 0x7fc7841e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc7837fe640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 0x7fc7840cf510 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc77800d620 tx=0x7fc77800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77800b620 con 0x7fc7840d2eb0 2024-05-09T03:10:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc778019070 con 0x7fc7840d2eb0 2024-05-09T03:10:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77800ba00 con 0x7fc7840d2eb0 2024-05-09T03:10:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7840cab40 con 0x7fc7840d2eb0 2024-05-09T03:10:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.235+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7840cb100 con 0x7fc7840d2eb0 2024-05-09T03:10:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.236+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc78406a270 con 0x7fc7840d2eb0 2024-05-09T03:10:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.237+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc77800bba0 con 0x7fc7840d2eb0 2024-05-09T03:10:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.237+0000 7fc780ff9640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7700762d0 0x7fc7700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.238+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7fc778097700 con 0x7fc7840d2eb0 2024-05-09T03:10:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.238+0000 7fc783fff640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7700762d0 0x7fc7700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.238+0000 7fc783fff640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7700762d0 0x7fc7700787e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc774006fd0 tx=0x7fc774008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.242+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc7780613b0 con 0x7fc7840d2eb0 2024-05-09T03:10:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.507+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fc7840c9420 con 0x7fc7840d2eb0 2024-05-09T03:10:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.510+0000 7fc780ff9640 1 -- 172.21.15.46:0/2852166276 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) ==== 90+0+0 (secure 0 0 0) 0x7fc778060cb0 con 0x7fc7840d2eb0 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7700762d0 msgr2=0x7fc7700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7700762d0 0x7fc7700787e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc774006fd0 tx=0x7fc774008040 comp rx=0 tx=0).stop 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 msgr2=0x7fc7840cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 0x7fc7840cf510 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc77800d620 tx=0x7fc77800dae0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 shutdown_connections 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc7840cfa50 0x7fc78414c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7700762d0 0x7fc7700787e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7840d2eb0 0x7fc7840cf510 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 --2- 172.21.15.46:0/2852166276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7840b95a0 0x7fc7841e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 >> 172.21.15.46:0/2852166276 conn(0x7fc78408d390 msgr2=0x7fc7840bb2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 shutdown_connections 2024-05-09T03:10:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.512+0000 7fc78a572640 1 -- 172.21.15.46:0/2852166276 wait complete. 2024-05-09T03:10:08.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-05-09T03:10:08.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.639+0000 7fc28736c640 1 Processor -- start 2024-05-09T03:10:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.639+0000 7fc28736c640 1 -- start start 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.640+0000 7fc28736c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc2801604a0 0x7fc280164830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.640+0000 7fc28736c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc280071710 0x7fc280071b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.640+0000 7fc28736c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2800b3380 con 0x7fc2800d16c0 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.640+0000 7fc28736c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc2800b7f10 con 0x7fc2801604a0 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.640+0000 7fc28736c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc2800b9350 con 0x7fc280071710 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285909640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2800d16c0 0x7fc2800d1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40976/0 (socket says 172.21.15.46:40976) 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285909640 1 -- 172.21.15.46:0/3244955462 learned_addr learned my addr 172.21.15.46:0/3244955462 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285108640 1 --2- 172.21.15.46:0/3244955462 >> v2:172.21.15.88:3300/0 conn(0x7fc2801604a0 0x7fc280164830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc284907640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc280071710 0x7fc280071b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc277fff640 1 -- 172.21.15.46:0/3244955462 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2111028348 0 0) 0x7fc2800b3380 con 0x7fc2800d16c0 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc277fff640 1 -- 172.21.15.46:0/3244955462 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc26c003520 con 0x7fc2800d16c0 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc277fff640 1 -- 172.21.15.46:0/3244955462 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 526244932 0 0) 0x7fc26c003520 con 0x7fc2800d16c0 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285108640 1 -- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc280071710 msgr2=0x7fc280071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285108640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc280071710 0x7fc280071b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285108640 1 -- 172.21.15.46:0/3244955462 >> v1:172.21.15.46:6789/0 conn(0x7fc2800d16c0 legacy=0x7fc2800d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc285108640 1 -- 172.21.15.46:0/3244955462 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc280164d70 con 0x7fc2801604a0 2024-05-09T03:10:08.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.641+0000 7fc284907640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc280071710 0x7fc280071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.642+0000 7fc285108640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc280164830 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0097a0 tx=0x7fc27c02a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91b61ebe508ccdf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.642+0000 7fc277fff640 1 -- 172.21.15.46:0/3244955462 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c037070 con 0x7fc2801604a0 2024-05-09T03:10:08.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.642+0000 7fc277fff640 1 -- 172.21.15.46:0/3244955462 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc27c02aba0 con 0x7fc2801604a0 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.642+0000 7fc277fff640 1 -- 172.21.15.46:0/3244955462 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278001a90 con 0x7fc2801604a0 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 -- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 msgr2=0x7fc280164830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc280164830 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc27c0097a0 tx=0x7fc27c02a110 comp rx=0 tx=0).stop 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 -- 172.21.15.46:0/3244955462 shutdown_connections 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc280071710 0x7fc280071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 --2- 172.21.15.46:0/3244955462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc280164830 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 -- 172.21.15.46:0/3244955462 >> 172.21.15.46:0/3244955462 conn(0x7fc2800d0570 msgr2=0x7fc2800d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 -- 172.21.15.46:0/3244955462 shutdown_connections 2024-05-09T03:10:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.643+0000 7fc28736c640 1 -- 172.21.15.46:0/3244955462 wait complete. 2024-05-09T03:10:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.645+0000 7fc28736c640 1 Processor -- start 2024-05-09T03:10:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.645+0000 7fc28736c640 1 -- start start 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc28736c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 0x7fc280162740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc28736c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2800d16c0 0x7fc28006f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc28736c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc28006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc28736c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc2800d02e0 con 0x7fc280071710 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc28736c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc280072060 con 0x7fc2801604a0 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc28736c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc2800cfd20 con 0x7fc2800d16c0 2024-05-09T03:10:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc285108640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 0x7fc280162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc285108640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 0x7fc280162740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32870/0 (socket says 172.21.15.46:32870) 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.646+0000 7fc285108640 1 -- 172.21.15.46:0/3853806280 learned_addr learned my addr 172.21.15.46:0/3853806280 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285108640 1 -- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2800d16c0 msgr2=0x7fc28006f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc284907640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2800d16c0 0x7fc28006f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285909640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc28006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285108640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2800d16c0 0x7fc28006f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285108640 1 -- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 msgr2=0x7fc28006f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285108640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc28006f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285108640 1 -- 172.21.15.46:0/3853806280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc280070250 con 0x7fc280071710 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc285108640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 0x7fc280162740 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc27c035040 tx=0x7fc27c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c041070 con 0x7fc280071710 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc27c03c660 con 0x7fc280071710 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2801ead90 con 0x7fc280071710 2024-05-09T03:10:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c037040 con 0x7fc280071710 2024-05-09T03:10:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.647+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc280161a90 con 0x7fc280071710 2024-05-09T03:10:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.648+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc280161cf0 con 0x7fc280071710 2024-05-09T03:10:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.649+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc27c044070 con 0x7fc280071710 2024-05-09T03:10:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.650+0000 7fc275ffb640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc264076300 0x7fc264078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.650+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7fc27c0b7840 con 0x7fc280071710 2024-05-09T03:10:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.650+0000 7fc284907640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc264076300 0x7fc264078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:08.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.650+0000 7fc284907640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc264076300 0x7fc264078810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc280072030 tx=0x7fc270008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.654+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc27c0813c0 con 0x7fc280071710 2024-05-09T03:10:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.919+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fc2800d2e20 con 0x7fc280071710 2024-05-09T03:10:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.922+0000 7fc275ffb640 1 -- 172.21.15.46:0/3853806280 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) ==== 90+0+0 (secure 0 0 0) 0x7fc27c080cc0 con 0x7fc280071710 2024-05-09T03:10:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc264076300 msgr2=0x7fc264078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc264076300 0x7fc264078810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc280072030 tx=0x7fc270008040 comp rx=0 tx=0).stop 2024-05-09T03:10:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 msgr2=0x7fc280162740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 0x7fc280162740 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc27c035040 tx=0x7fc27c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 shutdown_connections 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2801604a0 0x7fc28006f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2800d16c0 0x7fc28006f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc264076300 0x7fc264078810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 --2- 172.21.15.46:0/3853806280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc280071710 0x7fc280162740 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 >> 172.21.15.46:0/3853806280 conn(0x7fc2800d0570 msgr2=0x7fc2800d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 shutdown_connections 2024-05-09T03:10:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:08.924+0000 7fc28736c640 1 -- 172.21.15.46:0/3853806280 wait complete. 2024-05-09T03:10:08.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-05-09T03:10:09.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.051+0000 7f77cfa2b640 1 Processor -- start 2024-05-09T03:10:09.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.051+0000 7f77cfa2b640 1 -- start start 2024-05-09T03:10:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77cfa2b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f77c80b7e70 0x7f77c80b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77cfa2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77c80d3400 con 0x7f77c81605b0 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77cfa2b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77c80d4a60 con 0x7f77c80d2e90 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77cfa2b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f77c80d4d40 con 0x7f77c80b7e70 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77cdfc8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f77c81605b0 0x7f77c80b97e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40984/0 (socket says 172.21.15.46:40984) 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77cdfc8640 1 -- 172.21.15.46:0/1328267174 learned_addr learned my addr 172.21.15.46:0/1328267174 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77ccfc6640 1 --2- 172.21.15.46:0/1328267174 >> v2:172.21.15.88:3300/0 conn(0x7f77c80b7e70 0x7f77c80b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330645600 0 0) 0x7f77c80d3400 con 0x7f77c81605b0 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.053+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77b0003520 con 0x7f77c81605b0 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.054+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471856160 0 0) 0x7f77c80d4a60 con 0x7f77c80d2e90 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.054+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c80d3400 con 0x7f77c80d2e90 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.054+0000 7f77ccfc6640 1 -- 172.21.15.46:0/1328267174 >> v1:172.21.15.88:6789/0 conn(0x7f77c80d2e90 legacy=0x7f77c80b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.054+0000 7f77ccfc6640 1 -- 172.21.15.46:0/1328267174 >> v1:172.21.15.46:6789/0 conn(0x7f77c81605b0 legacy=0x7f77c80b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.054+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2537791296 0 0) 0x7f77b0003520 con 0x7f77c81605b0 2024-05-09T03:10:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.054+0000 7f77ccfc6640 1 -- 172.21.15.46:0/1328267174 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c80d5020 con 0x7f77c80b7e70 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77ccfc6640 1 --2- 172.21.15.46:0/1328267174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80b7e70 0x7f77c80b92a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0097a0 tx=0x7f77bc02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8671aa99872a3252 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc037070 con 0x7f77c80b7e70 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77bc02abc0 con 0x7f77c80b7e70 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77c7fff640 1 -- 172.21.15.46:0/1328267174 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0001710 con 0x7f77c80b7e70 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77cfa2b640 1 -- 172.21.15.46:0/1328267174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80b7e70 msgr2=0x7f77c80b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/1328267174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80b7e70 0x7f77c80b92a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0097a0 tx=0x7f77bc02a130 comp rx=0 tx=0).stop 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77cfa2b640 1 -- 172.21.15.46:0/1328267174 shutdown_connections 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/1328267174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80b7e70 0x7f77c80b92a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.055+0000 7f77cfa2b640 1 -- 172.21.15.46:0/1328267174 >> 172.21.15.46:0/1328267174 conn(0x7f77c808d390 msgr2=0x7f77c808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.056+0000 7f77cfa2b640 1 -- 172.21.15.46:0/1328267174 shutdown_connections 2024-05-09T03:10:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.056+0000 7f77cfa2b640 1 -- 172.21.15.46:0/1328267174 wait complete. 2024-05-09T03:10:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.057+0000 7f77cfa2b640 1 Processor -- start 2024-05-09T03:10:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.057+0000 7f77cfa2b640 1 -- start start 2024-05-09T03:10:09.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80d2e90 0x7f77c8151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c81605b0 0x7f77c80cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 0x7f77c8161f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77c80b4d80 con 0x7f77c8161b80 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f77c80b4c00 con 0x7f77c80d2e90 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77c80b4fc0 con 0x7f77c81605b0 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 0x7f77c8161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 0x7f77c8161f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32886/0 (socket says 172.21.15.46:32886) 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 -- 172.21.15.46:0/3660472423 learned_addr learned my addr 172.21.15.46:0/3660472423 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cd7c7640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80d2e90 0x7f77c8151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77ccfc6640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c81605b0 0x7f77c80cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 -- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c81605b0 msgr2=0x7f77c80cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c81605b0 0x7f77c80cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 -- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80d2e90 msgr2=0x7f77c8151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80d2e90 0x7f77c8151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 -- 172.21.15.46:0/3660472423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c8160dc0 con 0x7f77c8161b80 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cd7c7640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80d2e90 0x7f77c8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77ccfc6640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c81605b0 0x7f77c80cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cdfc8640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 0x7f77c8161f90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c480 tx=0x7f77c000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0018070 con 0x7f77c8161b80 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c0007500 con 0x7f77c8161b80 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c81610a0 con 0x7f77c8161b80 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.059+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0004a20 con 0x7f77c8161b80 2024-05-09T03:10:09.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.060+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c8161660 con 0x7f77c8161b80 2024-05-09T03:10:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.061+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c806a270 con 0x7f77c8161b80 2024-05-09T03:10:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.061+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f77c0004bc0 con 0x7f77c8161b80 2024-05-09T03:10:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.062+0000 7f77c5ffb640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b40762d0 0x7f77b40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.062+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7f77c0096640 con 0x7f77c8161b80 2024-05-09T03:10:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.062+0000 7f77cd7c7640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b40762d0 0x7f77b40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.063+0000 7f77cd7c7640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b40762d0 0x7f77b40787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f77b8007fd0 tx=0x7f77b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.067+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f77c0060260 con 0x7f77c8161b80 2024-05-09T03:10:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.329+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f77c80d1360 con 0x7f77c8161b80 2024-05-09T03:10:09.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.332+0000 7f77c5ffb640 1 -- 172.21.15.46:0/3660472423 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f77c005fb60 con 0x7f77c8161b80 2024-05-09T03:10:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b40762d0 msgr2=0x7f77b40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b40762d0 0x7f77b40787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f77b8007fd0 tx=0x7f77b8009040 comp rx=0 tx=0).stop 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 msgr2=0x7f77c8161f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 0x7f77c8161f90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c480 tx=0x7f77c000c940 comp rx=0 tx=0).stop 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 shutdown_connections 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b40762d0 0x7f77b40787e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c8161b80 0x7f77c8161f90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c81605b0 0x7f77c80cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 --2- 172.21.15.46:0/3660472423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c80d2e90 0x7f77c8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 >> 172.21.15.46:0/3660472423 conn(0x7f77c808d390 msgr2=0x7f77c815ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 shutdown_connections 2024-05-09T03:10:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.334+0000 7f77cfa2b640 1 -- 172.21.15.46:0/3660472423 wait complete. 2024-05-09T03:10:09.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-05-09T03:10:09.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.461+0000 7f26056a7640 1 Processor -- start 2024-05-09T03:10:09.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.461+0000 7f26056a7640 1 -- start start 2024-05-09T03:10:09.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.462+0000 7f26056a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000d2e90 0x7f26000b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f26056a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 0x7f26000b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f26056a7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26000d3400 con 0x7f26000b4fc0 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f26056a7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f26000d4a60 con 0x7f26001605b0 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f26056a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f26000d4d40 con 0x7f26000d2e90 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25fe7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 0x7f26000b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25feffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000d2e90 0x7f26000b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25ff7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f26000b4fc0 0x7f26000d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34536/0 (socket says 172.21.15.46:34536) 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25fe7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 0x7f26000b7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:55662/0 (socket says 172.21.15.46:55662) 2024-05-09T03:10:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25ff7fe640 1 -- 172.21.15.46:0/4176613897 learned_addr learned my addr 172.21.15.46:0/4176613897 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25feffd640 1 -- 172.21.15.46:0/4176613897 >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 msgr2=0x7f26000b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25feffd640 1 --2- 172.21.15.46:0/4176613897 >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 0x7f26000b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25feffd640 1 -- 172.21.15.46:0/4176613897 >> v1:172.21.15.88:6789/0 conn(0x7f26000b4fc0 legacy=0x7f26000d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25feffd640 1 -- 172.21.15.46:0/4176613897 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26000d5020 con 0x7f26000d2e90 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.463+0000 7f25fe7fc640 1 --2- 172.21.15.46:0/4176613897 >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 0x7f26000b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.464+0000 7f25feffd640 1 --2- 172.21.15.46:0/4176613897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000d2e90 0x7f26000b7760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25f00097a0 tx=0x7f25f002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87e1915f3499f962 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.464+0000 7f25fdffb640 1 -- 172.21.15.46:0/4176613897 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0037070 con 0x7f26000d2e90 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.464+0000 7f25fdffb640 1 -- 172.21.15.46:0/4176613897 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f002ae50 con 0x7f26000d2e90 2024-05-09T03:10:09.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.464+0000 7f25fdffb640 1 -- 172.21.15.46:0/4176613897 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0032660 con 0x7f26000d2e90 2024-05-09T03:10:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 -- 172.21.15.46:0/4176613897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000d2e90 msgr2=0x7f26000b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 --2- 172.21.15.46:0/4176613897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000d2e90 0x7f26000b7760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25f00097a0 tx=0x7f25f002a100 comp rx=0 tx=0).stop 2024-05-09T03:10:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 -- 172.21.15.46:0/4176613897 shutdown_connections 2024-05-09T03:10:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 --2- 172.21.15.46:0/4176613897 >> v2:172.21.15.88:3300/0 conn(0x7f26001605b0 0x7f26000b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 --2- 172.21.15.46:0/4176613897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000d2e90 0x7f26000b7760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 -- 172.21.15.46:0/4176613897 >> 172.21.15.46:0/4176613897 conn(0x7f260008d390 msgr2=0x7f260008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 -- 172.21.15.46:0/4176613897 shutdown_connections 2024-05-09T03:10:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.465+0000 7f26056a7640 1 -- 172.21.15.46:0/4176613897 wait complete. 2024-05-09T03:10:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.467+0000 7f26056a7640 1 Processor -- start 2024-05-09T03:10:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.467+0000 7f26056a7640 1 -- start start 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f26056a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000b4fc0 0x7f26001f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f26056a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26000d2e90 0x7f26000cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f26056a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 0x7f26000ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f26056a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26000d5540 con 0x7f26001605b0 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f26056a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f26000b4c00 con 0x7f26000d2e90 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f26056a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f26001f73e0 con 0x7f26000b4fc0 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f25ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 0x7f26000ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f25ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 0x7f26000ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32900/0 (socket says 172.21.15.46:32900) 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.468+0000 7f25ff7fe640 1 -- 172.21.15.46:0/1751761860 learned_addr learned my addr 172.21.15.46:0/1751761860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:09.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25feffd640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000b4fc0 0x7f26001f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25fe7fc640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26000d2e90 0x7f26000cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25ff7fe640 1 -- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000b4fc0 msgr2=0x7f26001f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25ff7fe640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000b4fc0 0x7f26001f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25ff7fe640 1 -- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26000d2e90 msgr2=0x7f26000cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25ff7fe640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26000d2e90 0x7f26000cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25ff7fe640 1 -- 172.21.15.46:0/1751761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2600151660 con 0x7f26001605b0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25fe7fc640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26000d2e90 0x7f26000cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25ff7fe640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 0x7f26000ce770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f25f800caa0 tx=0x7f25f800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8017070 con 0x7f26001605b0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f800ac00 con 0x7f26001605b0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8012400 con 0x7f26001605b0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26000cd6d0 con 0x7f26001605b0 2024-05-09T03:10:09.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.469+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26000cdc90 con 0x7f26001605b0 2024-05-09T03:10:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.470+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f260015de40 con 0x7f26001605b0 2024-05-09T03:10:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.471+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25f801a070 con 0x7f26001605b0 2024-05-09T03:10:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.471+0000 7f25e77fe640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f25ec076300 0x7f25ec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.472+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7f25f8096f70 con 0x7f26001605b0 2024-05-09T03:10:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.472+0000 7f25feffd640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f25ec076300 0x7f25ec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.472+0000 7f25feffd640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f25ec076300 0x7f25ec078810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f25f0034000 tx=0x7f25f002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.476+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f260015de40 con 0x7f26001605b0 2024-05-09T03:10:09.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.745+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f260006a270 con 0x7f26001605b0 2024-05-09T03:10:09.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.748+0000 7f25e77fe640 1 -- 172.21.15.46:0/1751761860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f25f80604a0 con 0x7f26001605b0 2024-05-09T03:10:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f25ec076300 msgr2=0x7f25ec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f25ec076300 0x7f25ec078810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f25f0034000 tx=0x7f25f002e000 comp rx=0 tx=0).stop 2024-05-09T03:10:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 msgr2=0x7f26000ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 0x7f26000ce770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f25f800caa0 tx=0x7f25f800cf60 comp rx=0 tx=0).stop 2024-05-09T03:10:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 shutdown_connections 2024-05-09T03:10:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f25ec076300 0x7f25ec078810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26001605b0 0x7f26000ce770 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26000d2e90 0x7f26000cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 --2- 172.21.15.46:0/1751761860 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26000b4fc0 0x7f26001f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 >> 172.21.15.46:0/1751761860 conn(0x7f260008d390 msgr2=0x7f260015d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:09.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 shutdown_connections 2024-05-09T03:10:09.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.750+0000 7f26056a7640 1 -- 172.21.15.46:0/1751761860 wait complete. 2024-05-09T03:10:09.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-05-09T03:10:09.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.877+0000 7fb20453f640 1 Processor -- start 2024-05-09T03:10:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.877+0000 7fb20453f640 1 -- start start 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb20453f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1fc071100 0x7fb1fc0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb20453f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0cfd40 0x7fb1fc0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb20453f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1fc0b5930 con 0x7fb1fc0d0150 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb20453f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb1fc0cf8b0 con 0x7fb1fc071100 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb20453f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb1fc0b5c10 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb2022db640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1fc071100 0x7fb1fc0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.878+0000 7fb202adc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb1fc0d0150 0x7fb1fc06f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40994/0 (socket says 172.21.15.46:40994) 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb202adc640 1 -- 172.21.15.46:0/3375150765 learned_addr learned my addr 172.21.15.46:0/3375150765 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb201ada640 1 --2- 172.21.15.46:0/3375150765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0cfd40 0x7fb1fc0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb2022db640 1 -- 172.21.15.46:0/3375150765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0cfd40 msgr2=0x7fb1fc0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb2022db640 1 --2- 172.21.15.46:0/3375150765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0cfd40 0x7fb1fc0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb2022db640 1 -- 172.21.15.46:0/3375150765 >> v1:172.21.15.46:6789/0 conn(0x7fb1fc0d0150 legacy=0x7fb1fc06f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb2022db640 1 -- 172.21.15.46:0/3375150765 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc0cf540 con 0x7fb1fc071100 2024-05-09T03:10:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.879+0000 7fb2012d9640 1 -- 172.21.15.46:0/3375150765 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242923175 0 0) 0x7fb1fc0b5930 con 0x7fb1fc0d0150 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.880+0000 7fb2022db640 1 --2- 172.21.15.46:0/3375150765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc071100 0x7fb1fc0b52a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800aea0 tx=0x7fb1f802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9783ec824e2b8c2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.880+0000 7fb2012d9640 1 -- 172.21.15.46:0/3375150765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8037070 con 0x7fb1fc071100 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.880+0000 7fb2012d9640 1 -- 172.21.15.46:0/3375150765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1f8008640 con 0x7fb1fc071100 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.880+0000 7fb2012d9640 1 -- 172.21.15.46:0/3375150765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8008a10 con 0x7fb1fc071100 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.880+0000 7fb20453f640 1 -- 172.21.15.46:0/3375150765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc071100 msgr2=0x7fb1fc0b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.880+0000 7fb20453f640 1 --2- 172.21.15.46:0/3375150765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc071100 0x7fb1fc0b52a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800aea0 tx=0x7fb1f802b7b0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.881+0000 7fb20453f640 1 -- 172.21.15.46:0/3375150765 shutdown_connections 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.881+0000 7fb20453f640 1 --2- 172.21.15.46:0/3375150765 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0cfd40 0x7fb1fc0732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.881+0000 7fb20453f640 1 --2- 172.21.15.46:0/3375150765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc071100 0x7fb1fc0b52a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.881+0000 7fb20453f640 1 -- 172.21.15.46:0/3375150765 >> 172.21.15.46:0/3375150765 conn(0x7fb1fc0b3620 msgr2=0x7fb1fc0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.881+0000 7fb20453f640 1 -- 172.21.15.46:0/3375150765 shutdown_connections 2024-05-09T03:10:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.881+0000 7fb20453f640 1 -- 172.21.15.46:0/3375150765 wait complete. 2024-05-09T03:10:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.882+0000 7fb20453f640 1 Processor -- start 2024-05-09T03:10:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.882+0000 7fb20453f640 1 -- start start 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1fc071100 0x7fb1fc15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 0x7fb1fc140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0d0150 0x7fb1fc1f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0b6070 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc0b5ef0 con 0x7fb1fc071100 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb1fc15c9d0 con 0x7fb1fc0d0150 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 0x7fb1fc140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 0x7fb1fc140d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32910/0 (socket says 172.21.15.46:32910) 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 -- 172.21.15.46:0/2480242088 learned_addr learned my addr 172.21.15.46:0/2480242088 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb202adc640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0d0150 0x7fb1fc1f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb2022db640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1fc071100 0x7fb1fc15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 -- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0d0150 msgr2=0x7fb1fc1f7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0d0150 0x7fb1fc1f7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 -- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1fc071100 msgr2=0x7fb1fc15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1fc071100 0x7fb1fc15d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 -- 172.21.15.46:0/2480242088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc15dd10 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb202adc640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0d0150 0x7fb1fc1f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb201ada640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 0x7fb1fc140d60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00d5f0 tx=0x7fb1ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec0125f0 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1ec019070 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc15e220 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec0129d0 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.884+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc15e7e0 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.885+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1fc06a270 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.886+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1ec012b70 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.887+0000 7fb1f2ffd640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1e00762d0 0x7fb1e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.887+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7fb1ec0976c0 con 0x7fb1fc0cfd40 2024-05-09T03:10:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.887+0000 7fb2022db640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1e00762d0 0x7fb1e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.887+0000 7fb2022db640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1e00762d0 0x7fb1e00787e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8034000 tx=0x7fb1f802f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:09.891+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb1ec061370 con 0x7fb1fc0cfd40 2024-05-09T03:10:10.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.154+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fb1fc15dfb0 con 0x7fb1fc0cfd40 2024-05-09T03:10:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.157+0000 7fb1f2ffd640 1 -- 172.21.15.46:0/2480242088 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) ==== 97+0+0 (secure 0 0 0) 0x7fb1ec0670a0 con 0x7fb1fc0cfd40 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1e00762d0 msgr2=0x7fb1e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1e00762d0 0x7fb1e00787e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8034000 tx=0x7fb1f802f000 comp rx=0 tx=0).stop 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 msgr2=0x7fb1fc140d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 0x7fb1fc140d60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00d5f0 tx=0x7fb1ec00dab0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 shutdown_connections 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1fc0d0150 0x7fb1fc1f7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1e00762d0 0x7fb1e00787e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1fc0cfd40 0x7fb1fc140d60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 --2- 172.21.15.46:0/2480242088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1fc071100 0x7fb1fc15d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 >> 172.21.15.46:0/2480242088 conn(0x7fb1fc0b3620 msgr2=0x7fb1fc15ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 shutdown_connections 2024-05-09T03:10:10.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.159+0000 7fb20453f640 1 -- 172.21.15.46:0/2480242088 wait complete. 2024-05-09T03:10:10.171 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-05-09T03:10:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.285+0000 7fb38d269640 1 Processor -- start 2024-05-09T03:10:10.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.285+0000 7fb38d269640 1 -- start start 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb38d269640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb3880d2e90 0x7fb3880b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb38d269640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 0x7fb3880b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb38d269640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3880d3400 con 0x7fb3880b4fc0 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb38d269640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb3880d4a60 con 0x7fb3880d2e90 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb38d269640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb3880d4d40 con 0x7fb3881605b0 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb3877fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb3880b4fc0 0x7fb3880d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41010/0 (socket says 172.21.15.46:41010) 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb3877fe640 1 -- 172.21.15.46:0/3561175205 learned_addr learned my addr 172.21.15.46:0/3561175205 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb386ffd640 1 --2- 172.21.15.46:0/3561175205 >> v2:172.21.15.88:3300/0 conn(0x7fb3880d2e90 0x7fb3880b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb3867fc640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 0x7fb3880b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783863674 0 0) 0x7fb3880d3400 con 0x7fb3880b4fc0 2024-05-09T03:10:10.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb374003520 con 0x7fb3880b4fc0 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2164427788 0 0) 0x7fb374003520 con 0x7fb3880b4fc0 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 msgr2=0x7fb3880b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 0x7fb3880b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 >> v2:172.21.15.88:3300/0 conn(0x7fb3880d2e90 msgr2=0x7fb3880b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 --2- 172.21.15.46:0/3561175205 >> v2:172.21.15.88:3300/0 conn(0x7fb3880d2e90 0x7fb3880b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.287+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3880d5020 con 0x7fb3880b4fc0 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fb3800016f0 con 0x7fb3880b4fc0 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb386ffd640 1 --2- 172.21.15.46:0/3561175205 >> v2:172.21.15.88:3300/0 conn(0x7fb3880d2e90 0x7fb3880b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 >> v1:172.21.15.46:6789/0 conn(0x7fb3880b4fc0 legacy=0x7fb3880d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 0x7fb374003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb374003fd0 0x7fb374004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb374004d40 0x7fb374005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880d4a60 con 0x7fb374004d40 2024-05-09T03:10:10.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880d4d40 con 0x7fb374003fd0 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb37400bac0 con 0x7fb3881605b0 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3877fe640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb374004d40 0x7fb374005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb386ffd640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 0x7fb374003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3867fc640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb374003fd0 0x7fb374004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3877fe640 1 -- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 msgr2=0x7fb374003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3877fe640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881605b0 0x7fb374003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3877fe640 1 -- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb374003fd0 msgr2=0x7fb374004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3877fe640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb374003fd0 0x7fb374004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.288+0000 7fb3877fe640 1 -- 172.21.15.46:0/3561175205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb374003520 con 0x7fb374004d40 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.289+0000 7fb3877fe640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb374004d40 0x7fb374005150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb374005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1f3aa5880c0df9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.289+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380007d50 con 0x7fb374004d40 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.289+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb380005af0 con 0x7fb374004d40 2024-05-09T03:10:10.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.289+0000 7fb385ffb640 1 -- 172.21.15.46:0/3561175205 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380007070 con 0x7fb374004d40 2024-05-09T03:10:10.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb38d269640 1 -- 172.21.15.46:0/3561175205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb374004d40 msgr2=0x7fb374005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb38d269640 1 --2- 172.21.15.46:0/3561175205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb374004d40 0x7fb374005150 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb374005830 comp rx=0 tx=0).stop 2024-05-09T03:10:10.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb3877fe640 1 -- 172.21.15.46:0/3561175205 reap_dead start 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb38d269640 1 -- 172.21.15.46:0/3561175205 shutdown_connections 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb38d269640 1 -- 172.21.15.46:0/3561175205 >> 172.21.15.46:0/3561175205 conn(0x7fb38808d390 msgr2=0x7fb38808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb38d269640 1 -- 172.21.15.46:0/3561175205 shutdown_connections 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.290+0000 7fb38d269640 1 -- 172.21.15.46:0/3561175205 wait complete. 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.292+0000 7fb38d269640 1 Processor -- start 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.292+0000 7fb38d269640 1 -- start start 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb38d269640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb374004d40 0x7fb3881625b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb38d269640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 0x7fb3880cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb38d269640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3880cb840 0x7fb3880ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb38d269640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880d5540 con 0x7fb388162af0 2024-05-09T03:10:10.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb38d269640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880b4c00 con 0x7fb3880cb840 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb38d269640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb3881f7390 con 0x7fb374004d40 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 0x7fb3880cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 0x7fb3880cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32928/0 (socket says 172.21.15.46:32928) 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 -- 172.21.15.46:0/1884956461 learned_addr learned my addr 172.21.15.46:0/1884956461 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb386ffd640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb374004d40 0x7fb3881625b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3877fe640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3880cb840 0x7fb3880ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 -- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb374004d40 msgr2=0x7fb3881625b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb374004d40 0x7fb3881625b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 -- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3880cb840 msgr2=0x7fb3880ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3880cb840 0x7fb3880ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.293+0000 7fb3867fc640 1 -- 172.21.15.46:0/1884956461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb388161530 con 0x7fb388162af0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.294+0000 7fb3867fc640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 0x7fb3880cb300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb3700119d0 tx=0x7fb370011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.294+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37000faf0 con 0x7fb388162af0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.294+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37000fc90 con 0x7fb388162af0 2024-05-09T03:10:10.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.294+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3881617f0 con 0x7fb388162af0 2024-05-09T03:10:10.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.294+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb370014440 con 0x7fb388162af0 2024-05-09T03:10:10.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.294+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb388161db0 con 0x7fb388162af0 2024-05-09T03:10:10.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.295+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb38806a270 con 0x7fb388162af0 2024-05-09T03:10:10.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.296+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb370012040 con 0x7fb388162af0 2024-05-09T03:10:10.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.296+0000 7fb37f7fe640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3680762d0 0x7fb3680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.297+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7fb370097a80 con 0x7fb388162af0 2024-05-09T03:10:10.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.297+0000 7fb386ffd640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3680762d0 0x7fb3680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.297+0000 7fb386ffd640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3680762d0 0x7fb3680787e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb38806f130 tx=0x7fb37800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:10.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.301+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb370019030 con 0x7fb388162af0 2024-05-09T03:10:10.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.565+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fb3880d0f00 con 0x7fb388162af0 2024-05-09T03:10:10.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.567+0000 7fb37f7fe640 1 -- 172.21.15.46:0/1884956461 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fb370061620 con 0x7fb388162af0 2024-05-09T03:10:10.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3680762d0 msgr2=0x7fb3680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3680762d0 0x7fb3680787e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb38806f130 tx=0x7fb37800b040 comp rx=0 tx=0).stop 2024-05-09T03:10:10.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 msgr2=0x7fb3880cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 0x7fb3880cb300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb3700119d0 tx=0x7fb370011e90 comp rx=0 tx=0).stop 2024-05-09T03:10:10.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 shutdown_connections 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3880cb840 0x7fb3880ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3680762d0 0x7fb3680787e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb388162af0 0x7fb3880cb300 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 --2- 172.21.15.46:0/1884956461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb374004d40 0x7fb3881625b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 >> 172.21.15.46:0/1884956461 conn(0x7fb38808d390 msgr2=0x7fb3880d1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 shutdown_connections 2024-05-09T03:10:10.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.569+0000 7fb38d269640 1 -- 172.21.15.46:0/1884956461 wait complete. 2024-05-09T03:10:10.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:10:10.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.697+0000 7f543d2e0640 1 Processor -- start 2024-05-09T03:10:10.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.697+0000 7f543d2e0640 1 -- start start 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f543d2e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54381613c0 0x7f54380b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f543d2e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54380b7d00 0x7f5438163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f543d2e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54380b8110 con 0x7f54380b5210 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f543d2e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f54380b5620 con 0x7f54380b7d00 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f543d2e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f54381609a0 con 0x7f54381613c0 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f5436ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f54380b5210 0x7f54380d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41018/0 (socket says 172.21.15.46:41018) 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.698+0000 7f5436ffd640 1 -- 172.21.15.46:0/4271106019 learned_addr learned my addr 172.21.15.46:0/4271106019 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54377fe640 1 --2- 172.21.15.46:0/4271106019 >> v2:172.21.15.88:3300/0 conn(0x7f54380b7d00 0x7f5438163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54367fc640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54381613c0 0x7f54380b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f5435ffb640 1 -- 172.21.15.46:0/4271106019 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1689772177 0 0) 0x7f5428001110 con 0x7f54380b5210 2024-05-09T03:10:10.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f5435ffb640 1 -- 172.21.15.46:0/4271106019 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5420003520 con 0x7f54380b5210 2024-05-09T03:10:10.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f5435ffb640 1 -- 172.21.15.46:0/4271106019 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 145563841 0 0) 0x7f5420003520 con 0x7f54380b5210 2024-05-09T03:10:10.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54377fe640 1 -- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54381613c0 msgr2=0x7f54380b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:10.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54377fe640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54381613c0 0x7f54380b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54377fe640 1 -- 172.21.15.46:0/4271106019 >> v1:172.21.15.46:6789/0 conn(0x7f54380b5210 legacy=0x7f54380d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54377fe640 1 -- 172.21.15.46:0/4271106019 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54381617d0 con 0x7f54380b7d00 2024-05-09T03:10:10.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.699+0000 7f54367fc640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54381613c0 0x7f54380b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.700+0000 7f54377fe640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54380b7d00 0x7f5438163e60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5430009d90 tx=0x7f543002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64270634eb76db0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.700+0000 7f5435ffb640 1 -- 172.21.15.46:0/4271106019 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54280018c0 con 0x7f54380b7d00 2024-05-09T03:10:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.700+0000 7f5435ffb640 1 -- 172.21.15.46:0/4271106019 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5430007540 con 0x7f54380b7d00 2024-05-09T03:10:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.700+0000 7f5435ffb640 1 -- 172.21.15.46:0/4271106019 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300078f0 con 0x7f54380b7d00 2024-05-09T03:10:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 -- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54380b7d00 msgr2=0x7f5438163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54380b7d00 0x7f5438163e60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5430009d90 tx=0x7f543002a5f0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 -- 172.21.15.46:0/4271106019 shutdown_connections 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54380b7d00 0x7f5438163e60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 --2- 172.21.15.46:0/4271106019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54381613c0 0x7f54380b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 -- 172.21.15.46:0/4271106019 >> 172.21.15.46:0/4271106019 conn(0x7f54380d29a0 msgr2=0x7f54380d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 -- 172.21.15.46:0/4271106019 shutdown_connections 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.701+0000 7f543d2e0640 1 -- 172.21.15.46:0/4271106019 wait complete. 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.703+0000 7f543d2e0640 1 Processor -- start 2024-05-09T03:10:10.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.703+0000 7f543d2e0640 1 -- start start 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f543d2e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54380b5210 0x7f54380b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f543d2e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 0x7f543815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f543d2e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54381613c0 0x7f5438161b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f543d2e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5438160bc0 con 0x7f54380b7d00 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f543d2e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54380d5560 con 0x7f54381613c0 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f543d2e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5438160ea0 con 0x7f54380b5210 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f54367fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 0x7f543815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f54367fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 0x7f543815bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32930/0 (socket says 172.21.15.46:32930) 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f54367fc640 1 -- 172.21.15.46:0/604647391 learned_addr learned my addr 172.21.15.46:0/604647391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:10.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f5436ffd640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54380b5210 0x7f54380b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.704+0000 7f54377fe640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54381613c0 0x7f5438161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f54367fc640 1 -- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54380b5210 msgr2=0x7f54380b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f54367fc640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54380b5210 0x7f54380b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f54367fc640 1 -- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54381613c0 msgr2=0x7f5438161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f54367fc640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54381613c0 0x7f5438161b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f54367fc640 1 -- 172.21.15.46:0/604647391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54380cd550 con 0x7f54380b7d00 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f54367fc640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 0x7f543815bef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f542c00e990 tx=0x7f542c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c017070 con 0x7f54380b7d00 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f542c00c9e0 con 0x7f54380b7d00 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c00cdb0 con 0x7f54380b7d00 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54380cde20 con 0x7f54380b7d00 2024-05-09T03:10:10.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.705+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54380cd270 con 0x7f54380b7d00 2024-05-09T03:10:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.706+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543806a270 con 0x7f54380b7d00 2024-05-09T03:10:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.707+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f542c01a070 con 0x7f54380b7d00 2024-05-09T03:10:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.708+0000 7f541f7fe640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54240762d0 0x7f54240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.708+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7f542c097510 con 0x7f54380b7d00 2024-05-09T03:10:10.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.708+0000 7f5436ffd640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54240762d0 0x7f54240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:10.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.708+0000 7f5436ffd640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54240762d0 0x7f54240787e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54280014d0 tx=0x7f5428001460 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.712+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f542c060ad0 con 0x7f54380b7d00 2024-05-09T03:10:10.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.905+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f54380d40d0 con 0x7f54380b7d00 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.905+0000 7f541f7fe640 1 -- 172.21.15.46:0/604647391 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v27) ==== 84+0+65 (secure 0 0 0) 0x7f542c069110 con 0x7f54380b7d00 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:10.906 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:10:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54240762d0 msgr2=0x7f54240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54240762d0 0x7f54240787e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54280014d0 tx=0x7f5428001460 comp rx=0 tx=0).stop 2024-05-09T03:10:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 msgr2=0x7f543815bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 0x7f543815bef0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f542c00e990 tx=0x7f542c00ee50 comp rx=0 tx=0).stop 2024-05-09T03:10:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 shutdown_connections 2024-05-09T03:10:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54381613c0 0x7f5438161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54240762d0 0x7f54240787e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54380b7d00 0x7f543815bef0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 --2- 172.21.15.46:0/604647391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54380b5210 0x7f54380b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 >> 172.21.15.46:0/604647391 conn(0x7f54380d29a0 msgr2=0x7f543815e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 shutdown_connections 2024-05-09T03:10:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:10.907+0000 7f543d2e0640 1 -- 172.21.15.46:0/604647391 wait complete. 2024-05-09T03:10:10.919 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:10:10.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:10:11.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.037+0000 7f3c564d9640 1 Processor -- start 2024-05-09T03:10:11.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.037+0000 7f3c564d9640 1 -- start start 2024-05-09T03:10:11.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c564d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c50071650 0x7f3c500b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c564d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c5015f070 0x7f3c50161cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c564d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c500b5520 0x7f3c500b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c564d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5015ebe0 con 0x7f3c50071650 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c564d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3c50162b80 con 0x7f3c500b5520 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c564d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3c500b5e70 con 0x7f3c5015f070 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c4ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c50071650 0x7f3c500b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c4ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c50071650 0x7f3c500b2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32946/0 (socket says 172.21.15.46:32946) 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.038+0000 7f3c4ffff640 1 -- 172.21.15.46:0/1687554363 learned_addr learned my addr 172.21.15.46:0/1687554363 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c54a76640 1 --2- 172.21.15.46:0/1687554363 >> v2:172.21.15.88:3300/0 conn(0x7f3c500b5520 0x7f3c500b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4f7fe640 1 --2- 172.21.15.46:0/1687554363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c5015f070 0x7f3c50161cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4ffff640 1 -- 172.21.15.46:0/1687554363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c5015f070 msgr2=0x7f3c50161cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4ffff640 1 --2- 172.21.15.46:0/1687554363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c5015f070 0x7f3c50161cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4ffff640 1 -- 172.21.15.46:0/1687554363 >> v2:172.21.15.88:3300/0 conn(0x7f3c500b5520 msgr2=0x7f3c500b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:11.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4ffff640 1 --2- 172.21.15.46:0/1687554363 >> v2:172.21.15.88:3300/0 conn(0x7f3c500b5520 0x7f3c500b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4ffff640 1 -- 172.21.15.46:0/1687554363 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c501625c0 con 0x7f3c50071650 2024-05-09T03:10:11.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c54a76640 1 --2- 172.21.15.46:0/1687554363 >> v2:172.21.15.88:3300/0 conn(0x7f3c500b5520 0x7f3c500b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.039+0000 7f3c4ffff640 1 --2- 172.21.15.46:0/1687554363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c500b2ed0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c500b6a20 tx=0x7f3c4002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b395a38b7c3343 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c4effd640 1 -- 172.21.15.46:0/1687554363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40037070 con 0x7f3c50071650 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c4effd640 1 -- 172.21.15.46:0/1687554363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c40032440 con 0x7f3c50071650 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c4effd640 1 -- 172.21.15.46:0/1687554363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40032810 con 0x7f3c50071650 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 -- 172.21.15.46:0/1687554363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 msgr2=0x7f3c500b2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 --2- 172.21.15.46:0/1687554363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c500b2ed0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c500b6a20 tx=0x7f3c4002a140 comp rx=0 tx=0).stop 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 -- 172.21.15.46:0/1687554363 shutdown_connections 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 --2- 172.21.15.46:0/1687554363 >> v2:172.21.15.88:3300/0 conn(0x7f3c500b5520 0x7f3c500b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 --2- 172.21.15.46:0/1687554363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c5015f070 0x7f3c50161cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 --2- 172.21.15.46:0/1687554363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c500b2ed0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 -- 172.21.15.46:0/1687554363 >> 172.21.15.46:0/1687554363 conn(0x7f3c500d0520 msgr2=0x7f3c500d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 -- 172.21.15.46:0/1687554363 shutdown_connections 2024-05-09T03:10:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.040+0000 7f3c564d9640 1 -- 172.21.15.46:0/1687554363 wait complete. 2024-05-09T03:10:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.042+0000 7f3c564d9640 1 Processor -- start 2024-05-09T03:10:11.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.042+0000 7f3c564d9640 1 -- start start 2024-05-09T03:10:11.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c564d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c50158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c564d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c500b5520 0x7f3c5014d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c564d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c5015f070 0x7f3c501fa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c564d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3c50162200 con 0x7f3c50071650 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c564d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5015e870 con 0x7f3c5015f070 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c564d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3c50162380 con 0x7f3c500b5520 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c50158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c50158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32950/0 (socket says 172.21.15.46:32950) 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 -- 172.21.15.46:0/3575642775 learned_addr learned my addr 172.21.15.46:0/3575642775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c54a76640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c5015f070 0x7f3c501fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4f7fe640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c500b5520 0x7f3c5014d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 -- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c500b5520 msgr2=0x7f3c5014d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c500b5520 0x7f3c5014d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 -- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c5015f070 msgr2=0x7f3c501fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c5015f070 0x7f3c501fa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 -- 172.21.15.46:0/3575642775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c501590a0 con 0x7f3c50071650 2024-05-09T03:10:11.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4f7fe640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c500b5520 0x7f3c5014d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:11.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4ffff640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c50158b60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c4002a180 tx=0x7f3c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:11.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.044+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40037040 con 0x7f3c50071650 2024-05-09T03:10:11.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.045+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c40042070 con 0x7f3c50071650 2024-05-09T03:10:11.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.045+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c40030bb0 con 0x7f3c50071650 2024-05-09T03:10:11.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.045+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5014d740 con 0x7f3c50071650 2024-05-09T03:10:11.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.045+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5015b790 con 0x7f3c50071650 2024-05-09T03:10:11.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.046+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c5006a270 con 0x7f3c50071650 2024-05-09T03:10:11.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.047+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3c40047460 con 0x7f3c50071650 2024-05-09T03:10:11.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.047+0000 7f3c4cff9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3c3c076300 0x7f3c3c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:11.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.047+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 4857+0+0 (secure 0 0 0) 0x7f3c400b7dc0 con 0x7f3c50071650 2024-05-09T03:10:11.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.047+0000 7f3c4f7fe640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3c3c076300 0x7f3c3c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:11.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.048+0000 7f3c4f7fe640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3c3c076300 0x7f3c3c078810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3c44006fd0 tx=0x7f3c44008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:11.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.052+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3c40081a70 con 0x7f3c50071650 2024-05-09T03:10:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:11.245+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3c500b9470 con 0x7f3c50071650 2024-05-09T03:10:12.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.160+0000 7f3c4cff9640 1 -- 172.21.15.46:0/3575642775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v28) ==== 138+0+0 (secure 0 0 0) 0x7f3c40081370 con 0x7f3c50071650 2024-05-09T03:10:12.161 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3c3c076300 msgr2=0x7f3c3c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3c3c076300 0x7f3c3c078810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3c44006fd0 tx=0x7f3c44008040 comp rx=0 tx=0).stop 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 msgr2=0x7f3c50158b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c50158b60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c4002a180 tx=0x7f3c400046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 shutdown_connections 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c5015f070 0x7f3c501fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3c500b5520 0x7f3c5014d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3c3c076300 0x7f3c3c078810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 --2- 172.21.15.46:0/3575642775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c50071650 0x7f3c50158b60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 >> 172.21.15.46:0/3575642775 conn(0x7f3c500d0520 msgr2=0x7f3c50075f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 shutdown_connections 2024-05-09T03:10:12.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.164+0000 7f3c564d9640 1 -- 172.21.15.46:0/3575642775 wait complete. 2024-05-09T03:10:12.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:10:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.311+0000 7fca3ff5e640 1 Processor -- start 2024-05-09T03:10:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.312+0000 7fca3ff5e640 1 -- start start 2024-05-09T03:10:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3ff5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b52a0 0x7fca3806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3ff5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 0x7fca380b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3ff5e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca380b8090 con 0x7fca380b7c80 2024-05-09T03:10:12.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3ff5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fca38160b40 con 0x7fca380d1000 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3ff5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fca380b9d10 con 0x7fca380b52a0 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3dcfa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b52a0 0x7fca3806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3d4f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 0x7fca380b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3e4fb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fca380b7c80 0x7fca380d49c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34552/0 (socket says 172.21.15.46:34552) 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3e4fb640 1 -- 172.21.15.46:0/1626868037 learned_addr learned my addr 172.21.15.46:0/1626868037 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.313+0000 7fca3d4f9640 1 --2- 172.21.15.46:0/1626868037 >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 0x7fca380b7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:55752/0 (socket says 172.21.15.46:55752) 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.314+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1626868037 >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 msgr2=0x7fca380b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.314+0000 7fca3dcfa640 1 --2- 172.21.15.46:0/1626868037 >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 0x7fca380b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.314+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1626868037 >> v1:172.21.15.88:6789/0 conn(0x7fca380b7c80 legacy=0x7fca380d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.314+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1626868037 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca380d1730 con 0x7fca380b52a0 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.314+0000 7fca3d4f9640 1 --2- 172.21.15.46:0/1626868037 >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 0x7fca380b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.314+0000 7fca3dcfa640 1 --2- 172.21.15.46:0/1626868037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b52a0 0x7fca3806e7f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca28009dc0 tx=0x7fca2802a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e1703a9998884aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ccf8640 1 -- 172.21.15.46:0/1626868037 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2802ae60 con 0x7fca380b52a0 2024-05-09T03:10:12.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ccf8640 1 -- 172.21.15.46:0/1626868037 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28007740 con 0x7fca380b52a0 2024-05-09T03:10:12.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ccf8640 1 -- 172.21.15.46:0/1626868037 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28007b10 con 0x7fca380b52a0 2024-05-09T03:10:12.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1626868037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b52a0 msgr2=0x7fca3806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1626868037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b52a0 0x7fca3806e7f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca28009dc0 tx=0x7fca2802a680 comp rx=0 tx=0).stop 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1626868037 shutdown_connections 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1626868037 >> v2:172.21.15.88:3300/0 conn(0x7fca380d1000 0x7fca380b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1626868037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b52a0 0x7fca3806e7f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.315+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1626868037 >> 172.21.15.46:0/1626868037 conn(0x7fca380d2920 msgr2=0x7fca380d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.316+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1626868037 shutdown_connections 2024-05-09T03:10:12.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.316+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1626868037 wait complete. 2024-05-09T03:10:12.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.318+0000 7fca3ff5e640 1 Processor -- start 2024-05-09T03:10:12.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.318+0000 7fca3ff5e640 1 -- start start 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3ff5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 0x7fca381f7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3ff5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b7c80 0x7fca381f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3ff5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca380d1000 0x7fca381563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3ff5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fca380b93b0 con 0x7fca380b52a0 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3ff5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca380b99a0 con 0x7fca380d1000 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3ff5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fca380d1410 con 0x7fca380b7c80 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 0x7fca381f7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 0x7fca381f7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32964/0 (socket says 172.21.15.46:32964) 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1095313902 learned_addr learned my addr 172.21.15.46:0/1095313902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3d4f9640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b7c80 0x7fca381f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3e4fb640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca380d1000 0x7fca381563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b7c80 msgr2=0x7fca381f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:12.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b7c80 0x7fca381f6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca380d1000 msgr2=0x7fca381563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca380d1000 0x7fca381563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.319+0000 7fca3dcfa640 1 -- 172.21.15.46:0/1095313902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca381f82e0 con 0x7fca380b52a0 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.320+0000 7fca3dcfa640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 0x7fca381f7da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fca2802ac70 tx=0x7fca28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.320+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2803f070 con 0x7fca380b52a0 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.320+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2802ae60 con 0x7fca380b52a0 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.320+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca380ac880 con 0x7fca380b52a0 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.320+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28031c60 con 0x7fca380b52a0 2024-05-09T03:10:12.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.320+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca381f7980 con 0x7fca380b52a0 2024-05-09T03:10:12.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.321+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca38068f60 con 0x7fca380b52a0 2024-05-09T03:10:12.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.322+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fca2803b410 con 0x7fca380b52a0 2024-05-09T03:10:12.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.322+0000 7fca367fc640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca240762d0 0x7fca240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:12.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.323+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5227+0+0 (secure 0 0 0) 0x7fca280b88a0 con 0x7fca380b52a0 2024-05-09T03:10:12.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.323+0000 7fca3d4f9640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca240762d0 0x7fca240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:12.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.323+0000 7fca3d4f9640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca240762d0 0x7fca240787e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca381fa130 tx=0x7fca2c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:12.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.327+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fca38068f60 con 0x7fca380b52a0 2024-05-09T03:10:12.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:12.517+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fca3806a270 con 0x7fca380b52a0 2024-05-09T03:10:13.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.168+0000 7fca367fc640 1 -- 172.21.15.46:0/1095313902 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v29) ==== 170+0+0 (secure 0 0 0) 0x7fca2804a070 con 0x7fca380b52a0 2024-05-09T03:10:13.169 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca240762d0 msgr2=0x7fca240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca240762d0 0x7fca240787e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca381fa130 tx=0x7fca2c009190 comp rx=0 tx=0).stop 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 msgr2=0x7fca381f7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 0x7fca381f7da0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fca2802ac70 tx=0x7fca28004a40 comp rx=0 tx=0).stop 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 shutdown_connections 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca380d1000 0x7fca381563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fca380b7c80 0x7fca381f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca240762d0 0x7fca240787e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 --2- 172.21.15.46:0/1095313902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca380b52a0 0x7fca381f7da0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 >> 172.21.15.46:0/1095313902 conn(0x7fca380d2920 msgr2=0x7fca3815f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 shutdown_connections 2024-05-09T03:10:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.172+0000 7fca3ff5e640 1 -- 172.21.15.46:0/1095313902 wait complete. 2024-05-09T03:10:13.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:10:13.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.327+0000 7f00c9534640 1 Processor -- start 2024-05-09T03:10:13.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.327+0000 7f00c9534640 1 -- start start 2024-05-09T03:10:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c9534640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40cffe0 0x7f00c40b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c9534640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00c40b77a0 0x7f00c40b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c9534640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00c40b80f0 con 0x7f00c40d0610 2024-05-09T03:10:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c9534640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f00c40b4340 con 0x7f00c40b77a0 2024-05-09T03:10:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c9534640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00c40b3aa0 con 0x7f00c40cffe0 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00c40b77a0 0x7f00c40b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f00c40d0610 0x7f00c40cd090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41022/0 (socket says 172.21.15.46:41022) 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c27fc640 1 -- 172.21.15.46:0/4136226675 learned_addr learned my addr 172.21.15.46:0/4136226675 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.328+0000 7f00c2ffd640 1 --2- 172.21.15.46:0/4136226675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40cffe0 0x7f00c40b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c37fe640 1 -- 172.21.15.46:0/4136226675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40cffe0 msgr2=0x7f00c40b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c37fe640 1 --2- 172.21.15.46:0/4136226675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40cffe0 0x7f00c40b2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c37fe640 1 -- 172.21.15.46:0/4136226675 >> v1:172.21.15.46:6789/0 conn(0x7f00c40d0610 legacy=0x7f00c40cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:13.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c37fe640 1 -- 172.21.15.46:0/4136226675 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c40b3d80 con 0x7f00c40b77a0 2024-05-09T03:10:13.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c37fe640 1 --2- 172.21.15.46:0/4136226675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40b77a0 0x7f00c40b7bb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f00b40097a0 tx=0x7f00b402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a975e8e653acc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:13.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c1ffb640 1 -- 172.21.15.46:0/4136226675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4037070 con 0x7f00c40b77a0 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.329+0000 7f00c1ffb640 1 -- 172.21.15.46:0/4136226675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b4032440 con 0x7f00c40b77a0 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c1ffb640 1 -- 172.21.15.46:0/4136226675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4032810 con 0x7f00c40b77a0 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 -- 172.21.15.46:0/4136226675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40b77a0 msgr2=0x7f00c40b7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 --2- 172.21.15.46:0/4136226675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40b77a0 0x7f00c40b7bb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f00b40097a0 tx=0x7f00b402a180 comp rx=0 tx=0).stop 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 -- 172.21.15.46:0/4136226675 shutdown_connections 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 --2- 172.21.15.46:0/4136226675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40b77a0 0x7f00c40b7bb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 --2- 172.21.15.46:0/4136226675 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40cffe0 0x7f00c40b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 -- 172.21.15.46:0/4136226675 >> 172.21.15.46:0/4136226675 conn(0x7f00c409b7d0 msgr2=0x7f00c409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 -- 172.21.15.46:0/4136226675 shutdown_connections 2024-05-09T03:10:13.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.330+0000 7f00c9534640 1 -- 172.21.15.46:0/4136226675 wait complete. 2024-05-09T03:10:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.332+0000 7f00c9534640 1 Processor -- start 2024-05-09T03:10:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.332+0000 7f00c9534640 1 -- start start 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.333+0000 7f00c9534640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40b77a0 0x7f00c40a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.333+0000 7f00c9534640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 0x7f00c40b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.333+0000 7f00c9534640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40d0610 0x7f00c4150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c9534640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00c40cd7a0 con 0x7f00c40cffe0 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c9534640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00c40d2970 con 0x7f00c40d0610 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c9534640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00c40b3700 con 0x7f00c40b77a0 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 0x7f00c40b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 0x7f00c40b12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32974/0 (socket says 172.21.15.46:32974) 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 -- 172.21.15.46:0/914820521 learned_addr learned my addr 172.21.15.46:0/914820521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:13.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c2ffd640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40b77a0 0x7f00c40a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c37fe640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40d0610 0x7f00c4150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 -- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40b77a0 msgr2=0x7f00c40a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40b77a0 0x7f00c40a02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 -- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40d0610 msgr2=0x7f00c4150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40d0610 0x7f00c4150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 -- 172.21.15.46:0/914820521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c40b1a50 con 0x7f00c40cffe0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c37fe640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40d0610 0x7f00c4150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c27fc640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 0x7f00c40b12b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00c340 tx=0x7f00ac00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac017070 con 0x7f00c40cffe0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00ac005d00 con 0x7f00c40cffe0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c40b1d30 con 0x7f00c40cffe0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.334+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac007310 con 0x7f00c40cffe0 2024-05-09T03:10:13.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.335+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c41f9020 con 0x7f00c40cffe0 2024-05-09T03:10:13.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.335+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c406a270 con 0x7f00c40cffe0 2024-05-09T03:10:13.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.336+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00ac01a070 con 0x7f00c40cffe0 2024-05-09T03:10:13.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.337+0000 7f00bb7fe640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00a40762d0 0x7f00a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.337+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5609+0+0 (secure 0 0 0) 0x7f00ac096e60 con 0x7f00c40cffe0 2024-05-09T03:10:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.337+0000 7f00c2ffd640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00a40762d0 0x7f00a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.338+0000 7f00c2ffd640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00a40762d0 0x7f00a40787e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0099c0 tx=0x7f00bc003990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:13.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.341+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f00ac021020 con 0x7f00c40cffe0 2024-05-09T03:10:13.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:13.571+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f00c40d3550 con 0x7f00c40cffe0 2024-05-09T03:10:13.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-05-09T03:10:13.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-05-09T03:10:14.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.170+0000 7f00bb7fe640 1 -- 172.21.15.46:0/914820521 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7f00c40b80f0 con 0x7f00c40cffe0 2024-05-09T03:10:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 4 and data pool 5 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00a40762d0 msgr2=0x7f00a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00a40762d0 0x7f00a40787e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0099c0 tx=0x7f00bc003990 comp rx=0 tx=0).stop 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 msgr2=0x7f00c40b12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 0x7f00c40b12b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00c340 tx=0x7f00ac00c800 comp rx=0 tx=0).stop 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 shutdown_connections 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40d0610 0x7f00c4150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00a40762d0 0x7f00a40787e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00c40cffe0 0x7f00c40b12b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.173+0000 7f00c9534640 1 --2- 172.21.15.46:0/914820521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00c40b77a0 0x7f00c40a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.174+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 >> 172.21.15.46:0/914820521 conn(0x7f00c409b7d0 msgr2=0x7f00c40702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.174+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 shutdown_connections 2024-05-09T03:10:14.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.174+0000 7f00c9534640 1 -- 172.21.15.46:0/914820521 wait complete. 2024-05-09T03:10:14.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.321+0000 7fcf94e3b640 1 Processor -- start 2024-05-09T03:10:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.321+0000 7fcf94e3b640 1 -- start start 2024-05-09T03:10:14.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf94e3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf901615f0 0x7fcf900b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf94e3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf900d2cf0 0x7fcf900d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf94e3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf900d15b0 con 0x7fcf900d0b80 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf94e3b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcf90160b10 con 0x7fcf900d2cf0 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf94e3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcf90160df0 con 0x7fcf901615f0 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf8ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf900d2cf0 0x7fcf900d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf8ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf900d2cf0 0x7fcf900d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32976/0 (socket says 172.21.15.46:32976) 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf8ed9d640 1 -- 172.21.15.46:0/1764798216 learned_addr learned my addr 172.21.15.46:0/1764798216 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.322+0000 7fcf8e59c640 1 --2- 172.21.15.46:0/1764798216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf901615f0 0x7fcf900b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.323+0000 7fcf8ed9d640 1 -- 172.21.15.46:0/1764798216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf901615f0 msgr2=0x7fcf900b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.323+0000 7fcf8ed9d640 1 --2- 172.21.15.46:0/1764798216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf901615f0 0x7fcf900b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.323+0000 7fcf8ed9d640 1 -- 172.21.15.46:0/1764798216 >> v1:172.21.15.88:6789/0 conn(0x7fcf900d0b80 legacy=0x7fcf900d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.323+0000 7fcf8ed9d640 1 -- 172.21.15.46:0/1764798216 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf901610d0 con 0x7fcf900d2cf0 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.323+0000 7fcf8ed9d640 1 --2- 172.21.15.46:0/1764798216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d2cf0 0x7fcf900d0f90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009870 tx=0x7fcf8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a89e623bca255e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf8d59a640 1 -- 172.21.15.46:0/1764798216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf88037070 con 0x7fcf900d2cf0 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf8d59a640 1 -- 172.21.15.46:0/1764798216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf88032440 con 0x7fcf900d2cf0 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf8d59a640 1 -- 172.21.15.46:0/1764798216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf88032810 con 0x7fcf900d2cf0 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 -- 172.21.15.46:0/1764798216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d2cf0 msgr2=0x7fcf900d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/1764798216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d2cf0 0x7fcf900d0f90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009870 tx=0x7fcf8802a270 comp rx=0 tx=0).stop 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 -- 172.21.15.46:0/1764798216 shutdown_connections 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/1764798216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d2cf0 0x7fcf900d0f90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/1764798216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf901615f0 0x7fcf900b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 -- 172.21.15.46:0/1764798216 >> 172.21.15.46:0/1764798216 conn(0x7fcf9006e190 msgr2=0x7fcf9006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.324+0000 7fcf94e3b640 1 -- 172.21.15.46:0/1764798216 shutdown_connections 2024-05-09T03:10:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.325+0000 7fcf94e3b640 1 -- 172.21.15.46:0/1764798216 wait complete. 2024-05-09T03:10:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.326+0000 7fcf94e3b640 1 Processor -- start 2024-05-09T03:10:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.327+0000 7fcf94e3b640 1 -- start start 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf94e3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 0x7fcf901f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf94e3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900d2cf0 0x7fcf900cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf94e3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf901615f0 0x7fcf90162ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf94e3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf901640c0 con 0x7fcf900d0b80 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf94e3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf90163f40 con 0x7fcf901615f0 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf94e3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcf90164240 con 0x7fcf900d2cf0 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 0x7fcf901f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 0x7fcf901f6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32980/0 (socket says 172.21.15.46:32980) 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 -- 172.21.15.46:0/67485397 learned_addr learned my addr 172.21.15.46:0/67485397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8dd9b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900d2cf0 0x7fcf900cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8ed9d640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf901615f0 0x7fcf90162ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 -- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900d2cf0 msgr2=0x7fcf900cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900d2cf0 0x7fcf900cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 -- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf901615f0 msgr2=0x7fcf90162ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf901615f0 0x7fcf90162ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.328+0000 7fcf8e59c640 1 -- 172.21.15.46:0/67485397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf900cd290 con 0x7fcf900d0b80 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.329+0000 7fcf8e59c640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 0x7fcf901f6b80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf80007e60 tx=0x7fcf8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.329+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80014620 con 0x7fcf900d0b80 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.329+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf80019070 con 0x7fcf900d0b80 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.329+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf80014a20 con 0x7fcf900d0b80 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.329+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf901f76b0 con 0x7fcf900d0b80 2024-05-09T03:10:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.329+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf900cd550 con 0x7fcf900d0b80 2024-05-09T03:10:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.330+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf9006a270 con 0x7fcf900d0b80 2024-05-09T03:10:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.330+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf80014bc0 con 0x7fcf900d0b80 2024-05-09T03:10:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.331+0000 7fcf7effd640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf700762d0 0x7fcf700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.331+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fcf80097f60 con 0x7fcf900d0b80 2024-05-09T03:10:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.331+0000 7fcf8dd9b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf700762d0 0x7fcf700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.332+0000 7fcf8dd9b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf700762d0 0x7fcf700787e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcf900caa90 tx=0x7fcf84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.336+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcf80061830 con 0x7fcf900d0b80 2024-05-09T03:10:14.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.533+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf90160780 con 0x7fcf900d0b80 2024-05-09T03:10:14.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.534+0000 7fcf7effd640 1 -- 172.21.15.46:0/67485397 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16610 (secure 0 0 0) 0x7fcf80061130 con 0x7fcf900d0b80 2024-05-09T03:10:14.536 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:14.536 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:10:13.572744+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:10:11.250168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T03:10:12.522105+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:14.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf700762d0 msgr2=0x7fcf700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf700762d0 0x7fcf700787e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcf900caa90 tx=0x7fcf84009040 comp rx=0 tx=0).stop 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 msgr2=0x7fcf901f6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 0x7fcf901f6b80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf80007e60 tx=0x7fcf8000e4b0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 shutdown_connections 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf901615f0 0x7fcf90162ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf900d2cf0 0x7fcf900cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf700762d0 0x7fcf700787e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 --2- 172.21.15.46:0/67485397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf900d0b80 0x7fcf901f6b80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 >> 172.21.15.46:0/67485397 conn(0x7fcf9006e190 msgr2=0x7fcf900babb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 shutdown_connections 2024-05-09T03:10:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.537+0000 7fcf94e3b640 1 -- 172.21.15.46:0/67485397 wait complete. 2024-05-09T03:10:14.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:14.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.666+0000 7f41cfbb2640 1 Processor -- start 2024-05-09T03:10:14.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.666+0000 7f41cfbb2640 1 -- start start 2024-05-09T03:10:14.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cfbb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41c80b9200 0x7f41c808d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cfbb2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41c80d2270 0x7f41c80b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cfbb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41c80b4ee0 con 0x7f41c808d6e0 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cfbb2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f41c80b5460 con 0x7f41c80b9200 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cfbb2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f41c80cfac0 con 0x7f41c80d2270 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cd94e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41c80b9200 0x7f41c808d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cd94e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f41c80b9200 0x7f41c808d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32994/0 (socket says 172.21.15.46:32994) 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.667+0000 7f41cd94e640 1 -- 172.21.15.46:0/538401040 learned_addr learned my addr 172.21.15.46:0/538401040 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.668+0000 7f41ce14f640 1 --2- 172.21.15.46:0/538401040 >> v2:172.21.15.88:3300/0 conn(0x7f41c80d2270 0x7f41c80b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.668+0000 7f41cd94e640 1 -- 172.21.15.46:0/538401040 >> v2:172.21.15.88:3300/0 conn(0x7f41c80d2270 msgr2=0x7f41c80b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.668+0000 7f41cd94e640 1 --2- 172.21.15.46:0/538401040 >> v2:172.21.15.88:3300/0 conn(0x7f41c80d2270 0x7f41c80b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.668+0000 7f41cd94e640 1 -- 172.21.15.46:0/538401040 >> v1:172.21.15.88:6789/0 conn(0x7f41c808d6e0 legacy=0x7f41c80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.668+0000 7f41cd94e640 1 -- 172.21.15.46:0/538401040 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c80b62e0 con 0x7f41c80b9200 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.668+0000 7f41cd94e640 1 --2- 172.21.15.46:0/538401040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c808d1a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f41c808e3e0 tx=0x7f41c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a0ad16a5054442 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cc94c640 1 -- 172.21.15.46:0/538401040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4037070 con 0x7f41c80b9200 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cc94c640 1 -- 172.21.15.46:0/538401040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c4032440 con 0x7f41c80b9200 2024-05-09T03:10:14.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cc94c640 1 -- 172.21.15.46:0/538401040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4032810 con 0x7f41c80b9200 2024-05-09T03:10:14.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 -- 172.21.15.46:0/538401040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 msgr2=0x7f41c808d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/538401040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c808d1a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f41c808e3e0 tx=0x7f41c402a270 comp rx=0 tx=0).stop 2024-05-09T03:10:14.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 -- 172.21.15.46:0/538401040 shutdown_connections 2024-05-09T03:10:14.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/538401040 >> v2:172.21.15.88:3300/0 conn(0x7f41c80d2270 0x7f41c80b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/538401040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c808d1a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 -- 172.21.15.46:0/538401040 >> 172.21.15.46:0/538401040 conn(0x7f41c80b56e0 msgr2=0x7f41c80b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:14.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 -- 172.21.15.46:0/538401040 shutdown_connections 2024-05-09T03:10:14.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.669+0000 7f41cfbb2640 1 -- 172.21.15.46:0/538401040 wait complete. 2024-05-09T03:10:14.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.671+0000 7f41cfbb2640 1 Processor -- start 2024-05-09T03:10:14.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.671+0000 7f41cfbb2640 1 -- start start 2024-05-09T03:10:14.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.672+0000 7f41cfbb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41c808d6e0 0x7f41c814b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.672+0000 7f41cfbb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c8161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.672+0000 7f41cfbb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41c80d2270 0x7f41c8156b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.672+0000 7f41cfbb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80cf750 con 0x7f41c80b9200 2024-05-09T03:10:14.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.672+0000 7f41cfbb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41c80cf470 con 0x7f41c808d6e0 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.672+0000 7f41cfbb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f41c80b4a20 con 0x7f41c80d2270 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c8161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c8161ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33010/0 (socket says 172.21.15.46:33010) 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 -- 172.21.15.46:0/1548048944 learned_addr learned my addr 172.21.15.46:0/1548048944 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd94e640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41c808d6e0 0x7f41c814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41ce14f640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41c80d2270 0x7f41c8156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 -- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41c80d2270 msgr2=0x7f41c8156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41c80d2270 0x7f41c8156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 -- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41c808d6e0 msgr2=0x7f41c814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41c808d6e0 0x7f41c814b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 -- 172.21.15.46:0/1548048944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c8159c90 con 0x7f41c80b9200 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cd14d640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c8161ce0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c370 tx=0x7f41b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8017070 con 0x7f41c80b9200 2024-05-09T03:10:14.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b80052d0 con 0x7f41c80b9200 2024-05-09T03:10:14.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8003810 con 0x7f41c80b9200 2024-05-09T03:10:14.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c815a100 con 0x7f41c80b9200 2024-05-09T03:10:14.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.673+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c815a6c0 con 0x7f41c80b9200 2024-05-09T03:10:14.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.674+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c806a270 con 0x7f41c80b9200 2024-05-09T03:10:14.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.675+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f41b801a070 con 0x7f41c80b9200 2024-05-09T03:10:14.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.676+0000 7f41be7fc640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41ac0762d0 0x7f41ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:14.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.676+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f41b8096900 con 0x7f41c80b9200 2024-05-09T03:10:14.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.676+0000 7f41cd94e640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41ac0762d0 0x7f41ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:14.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.676+0000 7f41cd94e640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41ac0762d0 0x7f41ac0787e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f41c4034000 tx=0x7f41c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:14.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.681+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f41b805fb80 con 0x7f41c80b9200 2024-05-09T03:10:14.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.870+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41c8159e60 con 0x7f41c80b9200 2024-05-09T03:10:14.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.872+0000 7f41be7fc640 1 -- 172.21.15.46:0/1548048944 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16610 (secure 0 0 0) 0x7f41b8068250 con 0x7f41c80b9200 2024-05-09T03:10:14.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:14.873 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:10:13.572744+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:10:11.250168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T03:10:12.522105+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:14.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41ac0762d0 msgr2=0x7f41ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41ac0762d0 0x7f41ac0787e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f41c4034000 tx=0x7f41c402e000 comp rx=0 tx=0).stop 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 msgr2=0x7f41c8161ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c8161ce0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41b800c370 tx=0x7f41b800c830 comp rx=0 tx=0).stop 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 shutdown_connections 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41c80d2270 0x7f41c8156b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41ac0762d0 0x7f41ac0787e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c80b9200 0x7f41c8161ce0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 --2- 172.21.15.46:0/1548048944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41c808d6e0 0x7f41c814b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:14.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 >> 172.21.15.46:0/1548048944 conn(0x7f41c80b56e0 msgr2=0x7f41c80cda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:14.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 shutdown_connections 2024-05-09T03:10:14.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:14.874+0000 7f41cfbb2640 1 -- 172.21.15.46:0/1548048944 wait complete. 2024-05-09T03:10:14.886 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:10:15.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.004+0000 7f910e87e640 1 Processor -- start 2024-05-09T03:10:15.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.004+0000 7f910e87e640 1 -- start start 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.005+0000 7f910e87e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 0x7f91080b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.005+0000 7f910e87e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91080b4fc0 0x7f91080b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.005+0000 7f910e87e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91080d3400 con 0x7f91081605b0 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.005+0000 7f910e87e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f91080d4a60 con 0x7f91080b4fc0 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.005+0000 7f910e87e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f91080d4d40 con 0x7f91080d2e90 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 0x7f91080b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f910ce1b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91080b4fc0 0x7f91080b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 0x7f91080b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:33214/0 (socket says 172.21.15.46:33214) 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 -- 172.21.15.46:0/1766035029 learned_addr learned my addr 172.21.15.46:0/1766035029 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:15.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 -- 172.21.15.46:0/1766035029 >> v2:172.21.15.88:3300/0 conn(0x7f91080b4fc0 msgr2=0x7f91080b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:15.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 --2- 172.21.15.46:0/1766035029 >> v2:172.21.15.88:3300/0 conn(0x7f91080b4fc0 0x7f91080b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:15.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 -- 172.21.15.46:0/1766035029 >> v1:172.21.15.88:6789/0 conn(0x7f91081605b0 legacy=0x7f91080d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:15.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f9107fff640 1 -- 172.21.15.46:0/1766035029 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91080d5020 con 0x7f91080d2e90 2024-05-09T03:10:15.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.006+0000 7f910ce1b640 1 --2- 172.21.15.46:0/1766035029 >> v2:172.21.15.88:3300/0 conn(0x7f91080b4fc0 0x7f91080b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:15.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.007+0000 7f9107fff640 1 --2- 172.21.15.46:0/1766035029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 0x7f91080b7760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009870 tx=0x7f90f802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a03bdf62f4acb90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:15.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.007+0000 7f9106ffd640 1 -- 172.21.15.46:0/1766035029 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8037070 con 0x7f91080d2e90 2024-05-09T03:10:15.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.007+0000 7f9106ffd640 1 -- 172.21.15.46:0/1766035029 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f8032440 con 0x7f91080d2e90 2024-05-09T03:10:15.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.007+0000 7f9106ffd640 1 -- 172.21.15.46:0/1766035029 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8032810 con 0x7f91080d2e90 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 -- 172.21.15.46:0/1766035029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 msgr2=0x7f91080b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 --2- 172.21.15.46:0/1766035029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 0x7f91080b7760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009870 tx=0x7f90f802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 -- 172.21.15.46:0/1766035029 shutdown_connections 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 --2- 172.21.15.46:0/1766035029 >> v2:172.21.15.88:3300/0 conn(0x7f91080b4fc0 0x7f91080b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 --2- 172.21.15.46:0/1766035029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080d2e90 0x7f91080b7760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 -- 172.21.15.46:0/1766035029 >> 172.21.15.46:0/1766035029 conn(0x7f910808d390 msgr2=0x7f910808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 -- 172.21.15.46:0/1766035029 shutdown_connections 2024-05-09T03:10:15.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.008+0000 7f910e87e640 1 -- 172.21.15.46:0/1766035029 wait complete. 2024-05-09T03:10:15.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.010+0000 7f910e87e640 1 Processor -- start 2024-05-09T03:10:15.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.010+0000 7f910e87e640 1 -- start start 2024-05-09T03:10:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f910e87e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080b4fc0 0x7f91081531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f910e87e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 0x7f91081f9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f910e87e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91081605b0 0x7f91081f9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f910e87e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91080d5540 con 0x7f91080d2e90 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f910e87e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91080b4c00 con 0x7f91081605b0 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f910e87e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f91081f7850 con 0x7f91080b4fc0 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f91077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 0x7f91081f9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f91077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 0x7f91081f9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33018/0 (socket says 172.21.15.46:33018) 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.011+0000 7f91077fe640 1 -- 172.21.15.46:0/3878403994 learned_addr learned my addr 172.21.15.46:0/3878403994 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f910ce1b640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91081605b0 0x7f91081f9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f9107fff640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080b4fc0 0x7f91081531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f91077fe640 1 -- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080b4fc0 msgr2=0x7f91081531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f91077fe640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080b4fc0 0x7f91081531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f91077fe640 1 -- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91081605b0 msgr2=0x7f91081f9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f91077fe640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91081605b0 0x7f91081f9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f91077fe640 1 -- 172.21.15.46:0/3878403994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91081f7640 con 0x7f91080d2e90 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f9107fff640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080b4fc0 0x7f91081531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f91077fe640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 0x7f91081f9a70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00c3a0 tx=0x7f90fc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc017070 con 0x7f91080d2e90 2024-05-09T03:10:15.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90fc0052d0 con 0x7f91080d2e90 2024-05-09T03:10:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910815a4f0 con 0x7f91080d2e90 2024-05-09T03:10:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc003830 con 0x7f91080d2e90 2024-05-09T03:10:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.012+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9108161d30 con 0x7f91080d2e90 2024-05-09T03:10:15.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.013+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910806a270 con 0x7f91080d2e90 2024-05-09T03:10:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.014+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f90fc01a070 con 0x7f91080d2e90 2024-05-09T03:10:15.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.015+0000 7f9104ff9640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90f40762d0 0x7f90f40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.015+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f90fc096a90 con 0x7f91080d2e90 2024-05-09T03:10:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.015+0000 7f9107fff640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90f40762d0 0x7f90f40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:15.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.016+0000 7f9107fff640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90f40762d0 0x7f90f40787e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f90f8034000 tx=0x7f90f802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:15.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.020+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f90fc0602c0 con 0x7f91080d2e90 2024-05-09T03:10:15.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:15.253+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f910815e850 con 0x7f91080d2e90 2024-05-09T03:10:16.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.186+0000 7f9104ff9640 1 -- 172.21.15.46:0/3878403994 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7f90fc068340 con 0x7f91080d2e90 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.188+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90f40762d0 msgr2=0x7f90f40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.188+0000 7f910e87e640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90f40762d0 0x7f90f40787e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f90f8034000 tx=0x7f90f802e000 comp rx=0 tx=0).stop 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 msgr2=0x7f91081f9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 0x7f91081f9a70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00c3a0 tx=0x7f90fc00c860 comp rx=0 tx=0).stop 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 shutdown_connections 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91081605b0 0x7f91081f9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90f40762d0 0x7f90f40787e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91080d2e90 0x7f91081f9a70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 --2- 172.21.15.46:0/3878403994 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91080b4fc0 0x7f91081531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 >> 172.21.15.46:0/3878403994 conn(0x7f910808d390 msgr2=0x7f910815d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 shutdown_connections 2024-05-09T03:10:16.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.189+0000 7f910e87e640 1 -- 172.21.15.46:0/3878403994 wait complete. 2024-05-09T03:10:16.204 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:10:16.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:10:16.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.330+0000 7f28b5b5e640 1 Processor -- start 2024-05-09T03:10:16.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.330+0000 7f28b5b5e640 1 -- start start 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28b5b5e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28b00b7e70 0x7f28b00b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28b5b5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f28b01605b0 0x7f28b00b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28b5b5e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28b00d3400 con 0x7f28b00d2e90 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28b5b5e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f28b00d4a60 con 0x7f28b00b7e70 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28b5b5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f28b00d4d40 con 0x7f28b01605b0 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28a6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28b00b7e70 0x7f28b00b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28af7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f28b00d2e90 0x7f28b00b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35722/0 (socket says 172.21.15.46:35722) 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28af7fe640 1 -- 172.21.15.46:0/1823971860 learned_addr learned my addr 172.21.15.46:0/1823971860 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.331+0000 7f28affff640 1 --2- 172.21.15.46:0/1823971860 >> v2:172.21.15.88:3300/0 conn(0x7f28b01605b0 0x7f28b00b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28a6ffd640 1 -- 172.21.15.46:0/1823971860 >> v2:172.21.15.88:3300/0 conn(0x7f28b01605b0 msgr2=0x7f28b00b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28a6ffd640 1 --2- 172.21.15.46:0/1823971860 >> v2:172.21.15.88:3300/0 conn(0x7f28b01605b0 0x7f28b00b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28a6ffd640 1 -- 172.21.15.46:0/1823971860 >> v1:172.21.15.46:6789/0 conn(0x7f28b00d2e90 legacy=0x7f28b00b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28aeffd640 1 -- 172.21.15.46:0/1823971860 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824152056 0 0) 0x7f28b00d3400 con 0x7f28b00d2e90 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28a6ffd640 1 -- 172.21.15.46:0/1823971860 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b00d5020 con 0x7f28b00b7e70 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28a6ffd640 1 --2- 172.21.15.46:0/1823971860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b00b7e70 0x7f28b00b92a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f28b00c95f0 tx=0x7f289802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94184983bca62f9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.332+0000 7f28aeffd640 1 -- 172.21.15.46:0/1823971860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898037070 con 0x7f28b00b7e70 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28aeffd640 1 -- 172.21.15.46:0/1823971860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2898032440 con 0x7f28b00b7e70 2024-05-09T03:10:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28aeffd640 1 -- 172.21.15.46:0/1823971860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289802ae60 con 0x7f28b00b7e70 2024-05-09T03:10:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 -- 172.21.15.46:0/1823971860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b00b7e70 msgr2=0x7f28b00b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/1823971860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b00b7e70 0x7f28b00b92a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f28b00c95f0 tx=0x7f289802a110 comp rx=0 tx=0).stop 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 -- 172.21.15.46:0/1823971860 shutdown_connections 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/1823971860 >> v2:172.21.15.88:3300/0 conn(0x7f28b01605b0 0x7f28b00b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/1823971860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b00b7e70 0x7f28b00b92a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 -- 172.21.15.46:0/1823971860 >> 172.21.15.46:0/1823971860 conn(0x7f28b008d390 msgr2=0x7f28b008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 -- 172.21.15.46:0/1823971860 shutdown_connections 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.333+0000 7f28b5b5e640 1 -- 172.21.15.46:0/1823971860 wait complete. 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.335+0000 7f28b5b5e640 1 Processor -- start 2024-05-09T03:10:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.335+0000 7f28b5b5e640 1 -- start start 2024-05-09T03:10:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.336+0000 7f28b5b5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28b00d2e90 0x7f28b00cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.336+0000 7f28b5b5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 0x7f28b00ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28b5b5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28b01e9a30 0x7f28b01e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28b5b5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00b4d80 con 0x7f28b01605b0 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28b5b5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00b4c00 con 0x7f28b01e9a30 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28b5b5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f28b00b4fc0 con 0x7f28b00d2e90 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 0x7f28b00ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 0x7f28b00ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39728/0 (socket says 172.21.15.46:39728) 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 -- 172.21.15.46:0/93977738 learned_addr learned my addr 172.21.15.46:0/93977738 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28affff640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28b01e9a30 0x7f28b01e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28af7fe640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28b00d2e90 0x7f28b00cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 -- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28b00d2e90 msgr2=0x7f28b00cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28b00d2e90 0x7f28b00cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 -- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28b01e9a30 msgr2=0x7f28b01e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28b01e9a30 0x7f28b01e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 -- 172.21.15.46:0/93977738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b01ea380 con 0x7f28b01605b0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28a6ffd640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 0x7f28b00ce620 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2898000c00 tx=0x7f28980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898037040 con 0x7f28b01605b0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2898042070 con 0x7f28b01605b0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b00cf0e0 con 0x7f28b01605b0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.337+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898030bd0 con 0x7f28b01605b0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.338+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b00cf6a0 con 0x7f28b01605b0 2024-05-09T03:10:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.338+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b006a270 con 0x7f28b01605b0 2024-05-09T03:10:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.339+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2898047460 con 0x7f28b01605b0 2024-05-09T03:10:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.340+0000 7f28acff9640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2890076300 0x7f2890078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.340+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f28980b80f0 con 0x7f28b01605b0 2024-05-09T03:10:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.340+0000 7f28af7fe640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2890076300 0x7f2890078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:16.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.341+0000 7f28af7fe640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2890076300 0x7f2890078810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28a0000f60 tx=0x7f28a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.345+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f28b01ea380 con 0x7f28b01605b0 2024-05-09T03:10:16.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:16.575+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f28b00d4560 con 0x7f28b01605b0 2024-05-09T03:10:17.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.194+0000 7f28acff9640 1 -- 172.21.15.46:0/93977738 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f2898032e20 con 0x7f28b01605b0 2024-05-09T03:10:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2890076300 msgr2=0x7f2890078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2890076300 0x7f2890078810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28a0000f60 tx=0x7f28a0009040 comp rx=0 tx=0).stop 2024-05-09T03:10:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 msgr2=0x7f28b00ce620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 0x7f28b00ce620 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2898000c00 tx=0x7f28980046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 shutdown_connections 2024-05-09T03:10:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28b01e9a30 0x7f28b01e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2890076300 0x7f2890078810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b01605b0 0x7f28b00ce620 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 --2- 172.21.15.46:0/93977738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f28b00d2e90 0x7f28b00cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 >> 172.21.15.46:0/93977738 conn(0x7f28b008d390 msgr2=0x7f28b008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 shutdown_connections 2024-05-09T03:10:17.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.197+0000 7f28b5b5e640 1 -- 172.21.15.46:0/93977738 wait complete. 2024-05-09T03:10:17.211 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:10:17.211 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:17.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.331+0000 7ff468029640 1 Processor -- start 2024-05-09T03:10:17.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.332+0000 7ff468029640 1 -- start start 2024-05-09T03:10:17.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff468029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4600b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff468029640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4600d0480 0x7ff4600d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff468029640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4600d0dd0 con 0x7ff4600ce6a0 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff468029640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff4600d2b80 con 0x7ff4600d0480 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff468029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4600d2e60 con 0x7ff4600b92e0 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff4665c6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4600ce6a0 0x7ff4600ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35738/0 (socket says 172.21.15.46:35738) 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff4665c6640 1 -- 172.21.15.46:0/3316275650 learned_addr learned my addr 172.21.15.46:0/3316275650 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff465dc5640 1 --2- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4600b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff4655c4640 1 --2- 172.21.15.46:0/3316275650 >> v2:172.21.15.88:3300/0 conn(0x7ff4600d0480 0x7ff4600d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff464dc3640 1 -- 172.21.15.46:0/3316275650 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032580411 0 0) 0x7ff458001110 con 0x7ff4600ce6a0 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.333+0000 7ff464dc3640 1 -- 172.21.15.46:0/3316275650 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff44c003520 con 0x7ff4600ce6a0 2024-05-09T03:10:17.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.334+0000 7ff464dc3640 1 -- 172.21.15.46:0/3316275650 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 149255390 0 0) 0x7ff44c003520 con 0x7ff4600ce6a0 2024-05-09T03:10:17.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.334+0000 7ff4655c4640 1 -- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 msgr2=0x7ff4600b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:17.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.334+0000 7ff4655c4640 1 --2- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4600b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.334+0000 7ff4655c4640 1 -- 172.21.15.46:0/3316275650 >> v1:172.21.15.46:6789/0 conn(0x7ff4600ce6a0 legacy=0x7ff4600ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.334+0000 7ff4655c4640 1 -- 172.21.15.46:0/3316275650 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4600d3140 con 0x7ff4600d0480 2024-05-09T03:10:17.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff4655c4640 1 --2- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600d0480 0x7ff4600d0890 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff4500097a0 tx=0x7ff45002a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e493719a85c4a2c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:17.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff464dc3640 1 -- 172.21.15.46:0/3316275650 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff450037070 con 0x7ff4600d0480 2024-05-09T03:10:17.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff464dc3640 1 -- 172.21.15.46:0/3316275650 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff45002ab10 con 0x7ff4600d0480 2024-05-09T03:10:17.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff464dc3640 1 -- 172.21.15.46:0/3316275650 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4580018c0 con 0x7ff4600d0480 2024-05-09T03:10:17.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff468029640 1 -- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600d0480 msgr2=0x7ff4600d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff468029640 1 --2- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600d0480 0x7ff4600d0890 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff4500097a0 tx=0x7ff45002a080 comp rx=0 tx=0).stop 2024-05-09T03:10:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff468029640 1 -- 172.21.15.46:0/3316275650 shutdown_connections 2024-05-09T03:10:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff468029640 1 --2- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600d0480 0x7ff4600d0890 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff468029640 1 --2- 172.21.15.46:0/3316275650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4600b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.335+0000 7ff468029640 1 -- 172.21.15.46:0/3316275650 >> 172.21.15.46:0/3316275650 conn(0x7ff4600d3c00 msgr2=0x7ff4600d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.336+0000 7ff468029640 1 -- 172.21.15.46:0/3316275650 shutdown_connections 2024-05-09T03:10:17.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.336+0000 7ff468029640 1 -- 172.21.15.46:0/3316275650 wait complete. 2024-05-09T03:10:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.337+0000 7ff468029640 1 Processor -- start 2024-05-09T03:10:17.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.337+0000 7ff468029640 1 -- start start 2024-05-09T03:10:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff468029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4601628e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff468029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600ce6a0 0x7ff4601600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff468029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 0x7ff460160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff468029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff46006e920 con 0x7ff4600d0480 2024-05-09T03:10:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff468029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff46006e7a0 con 0x7ff4600ce6a0 2024-05-09T03:10:17.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff468029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff46006eaa0 con 0x7ff4600b92e0 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 0x7ff460160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 0x7ff460160600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39730/0 (socket says 172.21.15.46:39730) 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 -- 172.21.15.46:0/1475978807 learned_addr learned my addr 172.21.15.46:0/1475978807 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4655c4640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600ce6a0 0x7ff4601600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff465dc5640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4601628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 -- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 msgr2=0x7ff4601628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4601628e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 -- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600ce6a0 msgr2=0x7ff4601600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600ce6a0 0x7ff4601600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 -- 172.21.15.46:0/1475978807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4601f7e40 con 0x7ff4600d0480 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4665c6640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 0x7ff460160600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff45800c500 tx=0x7ff45800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458005ce0 con 0x7ff4600d0480 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.339+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff458007070 con 0x7ff4600d0480 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.340+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4600cb7e0 con 0x7ff4600d0480 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.340+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4580074a0 con 0x7ff4600d0480 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.340+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4600cbda0 con 0x7ff4600d0480 2024-05-09T03:10:17.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.340+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff460071ca0 con 0x7ff4600d0480 2024-05-09T03:10:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.341+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff45801b400 con 0x7ff4600d0480 2024-05-09T03:10:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.342+0000 7ff4567fc640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff444076300 0x7ff444078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.342+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7ff458097200 con 0x7ff4600d0480 2024-05-09T03:10:17.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.342+0000 7ff465dc5640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff444076300 0x7ff444078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.343+0000 7ff465dc5640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff444076300 0x7ff444078810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff460071390 tx=0x7ff45c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:17.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.347+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff460071ca0 con 0x7ff4600d0480 2024-05-09T03:10:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.577+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4600cb5d0 con 0x7ff4600d0480 2024-05-09T03:10:17.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.578+0000 7ff4567fc640 1 -- 172.21.15.46:0/1475978807 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7ff4600d0dd0 con 0x7ff4600d0480 2024-05-09T03:10:17.579 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:17.579 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:16.576454+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4809":{"gid":4809,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/3490266983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3490266983},{"type":"v1","addr":"172.21.15.46:6834","nonce":3490266983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4809,"qdb_cluster":[4809]},"id":2}]} 2024-05-09T03:10:17.579 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 15 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff444076300 msgr2=0x7ff444078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff444076300 0x7ff444078810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff460071390 tx=0x7ff45c00b040 comp rx=0 tx=0).stop 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 msgr2=0x7ff460160600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 0x7ff460160600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff45800c500 tx=0x7ff45800c9c0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 shutdown_connections 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff444076300 0x7ff444078810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4600d0480 0x7ff460160600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4600ce6a0 0x7ff4601600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 --2- 172.21.15.46:0/1475978807 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4600b92e0 0x7ff4601628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 >> 172.21.15.46:0/1475978807 conn(0x7ff4600d3c00 msgr2=0x7ff460072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 shutdown_connections 2024-05-09T03:10:17.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.580+0000 7ff468029640 1 -- 172.21.15.46:0/1475978807 wait complete. 2024-05-09T03:10:17.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:10:17.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.705+0000 7f1b71f72640 1 Processor -- start 2024-05-09T03:10:17.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.705+0000 7f1b71f72640 1 -- start start 2024-05-09T03:10:17.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b71f72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b6c0ce5e0 0x7f1b6c15e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b71f72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c09bfe0 0x7f1b6c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b71f72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b6c0ceb80 con 0x7f1b6c0aff60 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b71f72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1b6c0cf790 con 0x7f1b6c0ce5e0 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b71f72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6c0cee60 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b6c0ce5e0 0x7f1b6c15e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b6c0aff60 0x7f1b6c0cfc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35748/0 (socket says 172.21.15.46:35748) 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/3580637637 >> v2:172.21.15.46:3300/0 conn(0x7f1b6c0ce5e0 0x7f1b6c15e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39734/0 (socket says 172.21.15.46:39734) 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6bfff640 1 -- 172.21.15.46:0/3580637637 learned_addr learned my addr 172.21.15.46:0/3580637637 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6affd640 1 --2- 172.21.15.46:0/3580637637 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c09bfe0 0x7f1b6c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/3580637637 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c09bfe0 msgr2=0x7f1b6c0afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/3580637637 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c09bfe0 0x7f1b6c0afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/3580637637 >> v1:172.21.15.46:6789/0 conn(0x7f1b6c0aff60 legacy=0x7f1b6c0cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6a7fc640 1 -- 172.21.15.46:0/3580637637 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 808109750 0 0) 0x7f1b6c0ceb80 con 0x7f1b6c0aff60 2024-05-09T03:10:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.707+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/3580637637 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c0cf140 con 0x7f1b6c0ce5e0 2024-05-09T03:10:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.708+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/3580637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0ce5e0 0x7f1b6c15e330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009790 tx=0x7f1b5c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec34488b0850058 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.708+0000 7f1b6a7fc640 1 -- 172.21.15.46:0/3580637637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c037070 con 0x7f1b6c0ce5e0 2024-05-09T03:10:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.708+0000 7f1b6a7fc640 1 -- 172.21.15.46:0/3580637637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b5c02ae50 con 0x7f1b6c0ce5e0 2024-05-09T03:10:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.708+0000 7f1b6a7fc640 1 -- 172.21.15.46:0/3580637637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c032660 con 0x7f1b6c0ce5e0 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 -- 172.21.15.46:0/3580637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0ce5e0 msgr2=0x7f1b6c15e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 --2- 172.21.15.46:0/3580637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0ce5e0 0x7f1b6c15e330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009790 tx=0x7f1b5c02a100 comp rx=0 tx=0).stop 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 -- 172.21.15.46:0/3580637637 shutdown_connections 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 --2- 172.21.15.46:0/3580637637 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c09bfe0 0x7f1b6c0afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 --2- 172.21.15.46:0/3580637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0ce5e0 0x7f1b6c15e330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 -- 172.21.15.46:0/3580637637 >> 172.21.15.46:0/3580637637 conn(0x7f1b6c0d2c70 msgr2=0x7f1b6c0d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 -- 172.21.15.46:0/3580637637 shutdown_connections 2024-05-09T03:10:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.709+0000 7f1b71f72640 1 -- 172.21.15.46:0/3580637637 wait complete. 2024-05-09T03:10:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.711+0000 7f1b71f72640 1 Processor -- start 2024-05-09T03:10:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.711+0000 7f1b71f72640 1 -- start start 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b71f72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 0x7f1b6c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b71f72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c0aff60 0x7f1b6c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b71f72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6c0ce5e0 0x7f1b6c0c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b71f72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c0d04b0 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b71f72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c0d0330 con 0x7f1b6c0ce5e0 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b71f72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6c09c3f0 con 0x7f1b6c0aff60 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 0x7f1b6c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 0x7f1b6c15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39744/0 (socket says 172.21.15.46:39744) 2024-05-09T03:10:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/2367024049 learned_addr learned my addr 172.21.15.46:0/2367024049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c0aff60 msgr2=0x7f1b6c0ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c0aff60 0x7f1b6c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6c0ce5e0 msgr2=0x7f1b6c0c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6c0ce5e0 0x7f1b6c0c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.712+0000 7f1b6b7fe640 1 -- 172.21.15.46:0/2367024049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c0c7860 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.713+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 0x7f1b6c15a380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0098b0 tx=0x7f1b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.713+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c041070 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.713+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b5c03c670 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.713+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6c0c7b40 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.713+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c037040 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.713+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6c0c8100 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.714+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b6c06a270 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.715+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1b5c044070 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.715+0000 7f1b53fff640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b580762d0 0x7f1b580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.715+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f1b5c0b83c0 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.716+0000 7f1b6affd640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b580762d0 0x7f1b580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:17.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.716+0000 7f1b6affd640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b580762d0 0x7f1b580787e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c15d7f0 tx=0x7f1b60008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:17.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.719+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1b5c081720 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.912+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b6c15db00 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.913+0000 7f1b53fff640 1 -- 172.21.15.46:0/2367024049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16597 (secure 0 0 0) 0x7f1b5c085a20 con 0x7f1b6c09bfe0 2024-05-09T03:10:17.914 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:17.914 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:10:15.170660+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:10:11.250168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T03:10:12.522105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:10:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b580762d0 msgr2=0x7f1b580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b580762d0 0x7f1b580787e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c15d7f0 tx=0x7f1b60008040 comp rx=0 tx=0).stop 2024-05-09T03:10:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 msgr2=0x7f1b6c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 0x7f1b6c15a380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c0098b0 tx=0x7f1b5c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 shutdown_connections 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b6c0ce5e0 0x7f1b6c0c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b6c0aff60 0x7f1b6c0ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b580762d0 0x7f1b580787e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 --2- 172.21.15.46:0/2367024049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c09bfe0 0x7f1b6c15a380 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 >> 172.21.15.46:0/2367024049 conn(0x7f1b6c0d2c70 msgr2=0x7f1b6c15abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 shutdown_connections 2024-05-09T03:10:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:17.915+0000 7f1b71f72640 1 -- 172.21.15.46:0/2367024049 wait complete. 2024-05-09T03:10:17.928 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:10:17.928 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:10:18.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.042+0000 7f7083fee640 1 Processor -- start 2024-05-09T03:10:18.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.042+0000 7f7083fee640 1 -- start start 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.043+0000 7f7083fee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f707c0b9200 0x7f707c08d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.043+0000 7f7083fee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c0b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.043+0000 7f7083fee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f707c0b4ee0 con 0x7f707c08d6e0 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.043+0000 7f7083fee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f707c0b5460 con 0x7f707c0b9200 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.043+0000 7f7083fee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f707c0cfac0 con 0x7f707c0d2270 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f707c0b9200 0x7f707c08d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f707c0b9200 0x7f707c08d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39748/0 (socket says 172.21.15.46:39748) 2024-05-09T03:10:18.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 -- 172.21.15.46:0/3013574769 learned_addr learned my addr 172.21.15.46:0/3013574769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f708258b640 1 --2- 172.21.15.46:0/3013574769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c0b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 -- 172.21.15.46:0/3013574769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 msgr2=0x7f707c0b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 --2- 172.21.15.46:0/3013574769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c0b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 -- 172.21.15.46:0/3013574769 >> v1:172.21.15.88:6789/0 conn(0x7f707c08d6e0 legacy=0x7f707c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.044+0000 7f7081d8a640 1 -- 172.21.15.46:0/3013574769 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707c0b62e0 con 0x7f707c0b9200 2024-05-09T03:10:18.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.045+0000 7f7081d8a640 1 --2- 172.21.15.46:0/3013574769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c08d1a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7078009870 tx=0x7f707802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f9b7c5f73d41bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.045+0000 7f7080d88640 1 -- 172.21.15.46:0/3013574769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078037070 con 0x7f707c0b9200 2024-05-09T03:10:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.045+0000 7f7080d88640 1 -- 172.21.15.46:0/3013574769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7078032440 con 0x7f707c0b9200 2024-05-09T03:10:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.045+0000 7f7080d88640 1 -- 172.21.15.46:0/3013574769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078032810 con 0x7f707c0b9200 2024-05-09T03:10:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.045+0000 7f7083fee640 1 -- 172.21.15.46:0/3013574769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 msgr2=0x7f707c08d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.045+0000 7f7083fee640 1 --2- 172.21.15.46:0/3013574769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c08d1a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7078009870 tx=0x7f707802a270 comp rx=0 tx=0).stop 2024-05-09T03:10:18.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.046+0000 7f7083fee640 1 -- 172.21.15.46:0/3013574769 shutdown_connections 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.046+0000 7f7083fee640 1 --2- 172.21.15.46:0/3013574769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c0b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.046+0000 7f7083fee640 1 --2- 172.21.15.46:0/3013574769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c08d1a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.046+0000 7f7083fee640 1 -- 172.21.15.46:0/3013574769 >> 172.21.15.46:0/3013574769 conn(0x7f707c0b56e0 msgr2=0x7f707c0b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.046+0000 7f7083fee640 1 -- 172.21.15.46:0/3013574769 shutdown_connections 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.046+0000 7f7083fee640 1 -- 172.21.15.46:0/3013574769 wait complete. 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.047+0000 7f7083fee640 1 Processor -- start 2024-05-09T03:10:18.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.047+0000 7f7083fee640 1 -- start start 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7083fee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707c08d6e0 0x7f707c0b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7083fee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c071950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7083fee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c071e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7083fee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f707c0cf750 con 0x7f707c0b9200 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7083fee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f707c0cf470 con 0x7f707c08d6e0 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7083fee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f707c0b4a20 con 0x7f707c0d2270 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c071950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c071950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39750/0 (socket says 172.21.15.46:39750) 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 -- 172.21.15.46:0/1754159788 learned_addr learned my addr 172.21.15.46:0/1754159788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:18.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f708258b640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081d8a640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707c08d6e0 0x7f707c0b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 -- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 msgr2=0x7f707c071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c071e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 -- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707c08d6e0 msgr2=0x7f707c0b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707c08d6e0 0x7f707c0b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 -- 172.21.15.46:0/1754159788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707c0b41a0 con 0x7f707c0b9200 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081d8a640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707c08d6e0 0x7f707c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f7081589640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c071950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f706c00c3a0 tx=0x7f706c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.049+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c017070 con 0x7f707c0b9200 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.050+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706c0052d0 con 0x7f707c0b9200 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.050+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0037f0 con 0x7f707c0b9200 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.050+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707c0b4480 con 0x7f707c0b9200 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.050+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f707c1f5660 con 0x7f707c0b9200 2024-05-09T03:10:18.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.050+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f707c1f51a0 con 0x7f707c0b9200 2024-05-09T03:10:18.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.051+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f706c01a070 con 0x7f707c0b9200 2024-05-09T03:10:18.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.052+0000 7f70727fc640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70600762d0 0x7f70600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.052+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f706c096930 con 0x7f707c0b9200 2024-05-09T03:10:18.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.052+0000 7f7081d8a640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70600762d0 0x7f70600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.052+0000 7f7081d8a640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70600762d0 0x7f70600787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7078034000 tx=0x7f707802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.056+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f707c1f51a0 con 0x7f707c0b9200 2024-05-09T03:10:18.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.246+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f707c1623c0 con 0x7f70600762d0 2024-05-09T03:10:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.252+0000 7f70727fc640 1 -- 172.21.15.46:0/1754159788 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237186 (secure 0 0 0) 0x7f707c1623c0 con 0x7f70600762d0 2024-05-09T03:10:18.254 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:18.263 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70600762d0 msgr2=0x7f70600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70600762d0 0x7f70600787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7078034000 tx=0x7f707802e000 comp rx=0 tx=0).stop 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 msgr2=0x7f707c071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c071950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f706c00c3a0 tx=0x7f706c00c860 comp rx=0 tx=0).stop 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 shutdown_connections 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f707c0d2270 0x7f707c071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70600762d0 0x7f70600787e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f707c0b9200 0x7f707c071950 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 --2- 172.21.15.46:0/1754159788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f707c08d6e0 0x7f707c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 >> 172.21.15.46:0/1754159788 conn(0x7f707c0b56e0 msgr2=0x7f707c161040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:18.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 shutdown_connections 2024-05-09T03:10:18.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.255+0000 7f7083fee640 1 -- 172.21.15.46:0/1754159788 wait complete. 2024-05-09T03:10:18.268 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-05-09T03:10:16.500329+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":21,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236352,"kb_used_data":3496,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738368,"statfs":{"total":773094113280,"available":772852088832,"internally_reserved":0,"allocated":3579904,"data_stored":2052472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.333673"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:13.161299+0000","last_change":"2024-05-09T03:10:13.161299+0000","last_active":"2024-05-09T03:10:13.161299+0000","last_peered":"2024-05-09T03:10:13.161299+0000","last_clean":"2024-05-09T03:10:13.161299+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:13.161299+0000","last_undegraded":"2024-05-09T03:10:13.161299+0000","last_fullsized":"2024-05-09T03:10:13.161299+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-05-09T03:10:12.158723+0000","last_change":"2024-05-09T03:10:12.158723+0000","last_active":"2024-05-09T03:10:12.158723+0000","last_peered":"2024-05-09T03:10:12.158723+0000","last_clean":"2024-05-09T03:10:12.158723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:10:12.158723+0000","last_undegraded":"2024-05-09T03:10:12.158723+0000","last_fullsized":"2024-05-09T03:10:12.158723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":121,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-05-09T03:10:07.910076+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:10:07.910076+0000","last_peered":"2024-05-09T03:10:07.910076+0000","last_clean":"2024-05-09T03:10:07.910076+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:10:07.910076+0000","last_undegraded":"2024-05-09T03:10:07.910076+0000","last_fullsized":"2024-05-09T03:10:07.910076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":892928,"data_stored":699335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27332,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":299008,"data_stored":108967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":108967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27328,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":299008,"data_stored":108967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31472,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":299008,"data_stored":108967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31484,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":299008,"data_stored":108967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31472,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340368,"statfs":{"total":96636764160,"available":96604536832,"internally_reserved":0,"allocated":299008,"data_stored":108967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":892928,"data_stored":699335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:10:18.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:10:18.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.430+0000 7fb82a1b4640 1 Processor -- start 2024-05-09T03:10:18.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.431+0000 7fb82a1b4640 1 -- start start 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb82a1b4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8241615f0 0x7fb8240b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb82a1b4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8240d15b0 con 0x7fb8240d2cf0 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb82a1b4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb824160b10 con 0x7fb8240d0b80 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb82a1b4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb824160df0 con 0x7fb8241615f0 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb8237fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8241615f0 0x7fb8240b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb823fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8240d2cf0 0x7fb824163f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35752/0 (socket says 172.21.15.46:35752) 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb823fff640 1 -- 172.21.15.46:0/3053630352 learned_addr learned my addr 172.21.15.46:0/3053630352 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:18.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb8237fe640 1 --2- 172.21.15.46:0/3053630352 >> v2:172.21.15.46:3300/0 conn(0x7fb8241615f0 0x7fb8240b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39756/0 (socket says 172.21.15.46:39756) 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb8237fe640 1 -- 172.21.15.46:0/3053630352 >> v1:172.21.15.88:6789/0 conn(0x7fb8240d0b80 legacy=0x7fb8240d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb8237fe640 1 -- 172.21.15.46:0/3053630352 >> v1:172.21.15.46:6789/0 conn(0x7fb8240d2cf0 legacy=0x7fb824163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.432+0000 7fb8237fe640 1 -- 172.21.15.46:0/3053630352 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8241610d0 con 0x7fb8241615f0 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.433+0000 7fb8237fe640 1 --2- 172.21.15.46:0/3053630352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8241615f0 0x7fb8240b81b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb8140097a0 tx=0x7fb81402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fdf8fb5fd2fe48 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.433+0000 7fb8227fc640 1 -- 172.21.15.46:0/3053630352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814037070 con 0x7fb8241615f0 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.433+0000 7fb8227fc640 1 -- 172.21.15.46:0/3053630352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb814032440 con 0x7fb8241615f0 2024-05-09T03:10:18.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.433+0000 7fb8227fc640 1 -- 172.21.15.46:0/3053630352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814032810 con 0x7fb8241615f0 2024-05-09T03:10:18.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3053630352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8241615f0 msgr2=0x7fb8240b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3053630352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8241615f0 0x7fb8240b81b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb8140097a0 tx=0x7fb81402a230 comp rx=0 tx=0).stop 2024-05-09T03:10:18.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3053630352 shutdown_connections 2024-05-09T03:10:18.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3053630352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8241615f0 0x7fb8240b81b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3053630352 >> 172.21.15.46:0/3053630352 conn(0x7fb82406e190 msgr2=0x7fb82406e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:18.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3053630352 shutdown_connections 2024-05-09T03:10:18.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.434+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3053630352 wait complete. 2024-05-09T03:10:18.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.436+0000 7fb82a1b4640 1 Processor -- start 2024-05-09T03:10:18.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.436+0000 7fb82a1b4640 1 -- start start 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.437+0000 7fb82a1b4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8240d0b80 0x7fb8240ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.437+0000 7fb82a1b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 0x7fb82415e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.437+0000 7fb82a1b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8241615f0 0x7fb8241eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.437+0000 7fb82a1b4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8240d1110 con 0x7fb8240d2cf0 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.437+0000 7fb82a1b4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb8240d0f90 con 0x7fb8241615f0 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.437+0000 7fb82a1b4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb8240d1290 con 0x7fb8240d0b80 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 0x7fb82415e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 0x7fb82415e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39768/0 (socket says 172.21.15.46:39768) 2024-05-09T03:10:18.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 -- 172.21.15.46:0/3170291644 learned_addr learned my addr 172.21.15.46:0/3170291644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb823fff640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8241615f0 0x7fb8241eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 -- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8240d0b80 msgr2=0x7fb8240ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb8237fe640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8240d0b80 0x7fb8240ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8240d0b80 0x7fb8240ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 -- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8241615f0 msgr2=0x7fb8241eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8241615f0 0x7fb8241eae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 -- 172.21.15.46:0/3170291644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8241ebbb0 con 0x7fb8240d2cf0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb823fff640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8241615f0 0x7fb8241eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb8237fe640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8240d0b80 0x7fb8240ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb822ffd640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 0x7fb82415e2b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb81800c3a0 tx=0x7fb81800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818017070 con 0x7fb8240d2cf0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8180052d0 con 0x7fb8240d2cf0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8240c9ac0 con 0x7fb8240d2cf0 2024-05-09T03:10:18.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818003830 con 0x7fb8240d2cf0 2024-05-09T03:10:18.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.438+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8240ca080 con 0x7fb8240d2cf0 2024-05-09T03:10:18.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.439+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb82406a270 con 0x7fb8240d2cf0 2024-05-09T03:10:18.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.440+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb81801a070 con 0x7fb8240d2cf0 2024-05-09T03:10:18.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.441+0000 7fb80bfff640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8100762d0 0x7fb8100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.441+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7fb8180969d0 con 0x7fb8240d2cf0 2024-05-09T03:10:18.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.441+0000 7fb8237fe640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8100762d0 0x7fb8100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.442+0000 7fb8237fe640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8100762d0 0x7fb8100787e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb814034000 tx=0x7fb81402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.446+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb8180602a0 con 0x7fb8240d2cf0 2024-05-09T03:10:18.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.633+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb8240b5480 con 0x7fb8100762d0 2024-05-09T03:10:18.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.640+0000 7fb80bfff640 1 -- 172.21.15.46:0/3170291644 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251939 (secure 0 0 0) 0x7fb8240b5480 con 0x7fb8100762d0 2024-05-09T03:10:18.643 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:18.645 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:10:18.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8100762d0 msgr2=0x7fb8100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8100762d0 0x7fb8100787e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb814034000 tx=0x7fb81402e000 comp rx=0 tx=0).stop 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 msgr2=0x7fb82415e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 0x7fb82415e2b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb81800c3a0 tx=0x7fb81800c860 comp rx=0 tx=0).stop 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 shutdown_connections 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8241615f0 0x7fb8241eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8100762d0 0x7fb8100787e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8240d2cf0 0x7fb82415e2b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 --2- 172.21.15.46:0/3170291644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8240d0b80 0x7fb8240ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 >> 172.21.15.46:0/3170291644 conn(0x7fb82406e190 msgr2=0x7fb8240cfb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:18.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 shutdown_connections 2024-05-09T03:10:18.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.643+0000 7fb82a1b4640 1 -- 172.21.15.46:0/3170291644 wait complete. 2024-05-09T03:10:18.655 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-05-09T03:10:18.500677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":163,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236736,"kb_used_data":3908,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737984,"statfs":{"total":773094113280,"available":772851695616,"internally_reserved":0,"allocated":4001792,"data_stored":2239664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000985"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175298+0000","last_change":"2024-05-09T03:10:14.190175+0000","last_active":"2024-05-09T03:10:15.175298+0000","last_peered":"2024-05-09T03:10:15.175298+0000","last_clean":"2024-05-09T03:10:15.175298+0000","last_became_active":"2024-05-09T03:10:14.189594+0000","last_became_peered":"2024-05-09T03:10:14.189594+0000","last_unstale":"2024-05-09T03:10:15.175298+0000","last_undegraded":"2024-05-09T03:10:15.175298+0000","last_fullsized":"2024-05-09T03:10:15.175298+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:21.549186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188337+0000","last_change":"2024-05-09T03:10:14.190454+0000","last_active":"2024-05-09T03:10:15.188337+0000","last_peered":"2024-05-09T03:10:15.188337+0000","last_clean":"2024-05-09T03:10:15.188337+0000","last_became_active":"2024-05-09T03:10:14.189503+0000","last_became_peered":"2024-05-09T03:10:14.189503+0000","last_unstale":"2024-05-09T03:10:15.188337+0000","last_undegraded":"2024-05-09T03:10:15.188337+0000","last_fullsized":"2024-05-09T03:10:15.188337+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:41:28.658547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289904+0000","last_change":"2024-05-09T03:10:14.196225+0000","last_active":"2024-05-09T03:10:16.289904+0000","last_peered":"2024-05-09T03:10:16.289904+0000","last_clean":"2024-05-09T03:10:16.289904+0000","last_became_active":"2024-05-09T03:10:14.195930+0000","last_became_peered":"2024-05-09T03:10:14.195930+0000","last_unstale":"2024-05-09T03:10:16.289904+0000","last_undegraded":"2024-05-09T03:10:16.289904+0000","last_fullsized":"2024-05-09T03:10:16.289904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:38.376803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177758+0000","last_change":"2024-05-09T03:10:14.202598+0000","last_active":"2024-05-09T03:10:15.177758+0000","last_peered":"2024-05-09T03:10:15.177758+0000","last_clean":"2024-05-09T03:10:15.177758+0000","last_became_active":"2024-05-09T03:10:14.202277+0000","last_became_peered":"2024-05-09T03:10:14.202277+0000","last_unstale":"2024-05-09T03:10:15.177758+0000","last_undegraded":"2024-05-09T03:10:15.177758+0000","last_fullsized":"2024-05-09T03:10:15.177758+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:27.169580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288652+0000","last_change":"2024-05-09T03:10:14.198765+0000","last_active":"2024-05-09T03:10:16.288652+0000","last_peered":"2024-05-09T03:10:16.288652+0000","last_clean":"2024-05-09T03:10:16.288652+0000","last_became_active":"2024-05-09T03:10:14.198198+0000","last_became_peered":"2024-05-09T03:10:14.198198+0000","last_unstale":"2024-05-09T03:10:16.288652+0000","last_undegraded":"2024-05-09T03:10:16.288652+0000","last_fullsized":"2024-05-09T03:10:16.288652+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:20.733331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187937+0000","last_change":"2024-05-09T03:10:14.191468+0000","last_active":"2024-05-09T03:10:15.187937+0000","last_peered":"2024-05-09T03:10:15.187937+0000","last_clean":"2024-05-09T03:10:15.187937+0000","last_became_active":"2024-05-09T03:10:14.191109+0000","last_became_peered":"2024-05-09T03:10:14.191109+0000","last_unstale":"2024-05-09T03:10:15.187937+0000","last_undegraded":"2024-05-09T03:10:15.187937+0000","last_fullsized":"2024-05-09T03:10:15.187937+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:38.376803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289309+0000","last_change":"2024-05-09T03:10:14.190202+0000","last_active":"2024-05-09T03:10:16.289309+0000","last_peered":"2024-05-09T03:10:16.289309+0000","last_clean":"2024-05-09T03:10:16.289309+0000","last_became_active":"2024-05-09T03:10:14.189878+0000","last_became_peered":"2024-05-09T03:10:14.189878+0000","last_unstale":"2024-05-09T03:10:16.289309+0000","last_undegraded":"2024-05-09T03:10:16.289309+0000","last_fullsized":"2024-05-09T03:10:16.289309+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:24:29.032308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289377+0000","last_change":"2024-05-09T03:10:14.187974+0000","last_active":"2024-05-09T03:10:16.289377+0000","last_peered":"2024-05-09T03:10:16.289377+0000","last_clean":"2024-05-09T03:10:16.289377+0000","last_became_active":"2024-05-09T03:10:14.187671+0000","last_became_peered":"2024-05-09T03:10:14.187671+0000","last_unstale":"2024-05-09T03:10:16.289377+0000","last_undegraded":"2024-05-09T03:10:16.289377+0000","last_fullsized":"2024-05-09T03:10:16.289377+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:15.211067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.176372+0000","last_change":"2024-05-09T03:10:14.195962+0000","last_active":"2024-05-09T03:10:15.176372+0000","last_peered":"2024-05-09T03:10:15.176372+0000","last_clean":"2024-05-09T03:10:15.176372+0000","last_became_active":"2024-05-09T03:10:14.195690+0000","last_became_peered":"2024-05-09T03:10:14.195690+0000","last_unstale":"2024-05-09T03:10:15.176372+0000","last_undegraded":"2024-05-09T03:10:15.176372+0000","last_fullsized":"2024-05-09T03:10:15.176372+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:39.990487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.178177+0000","last_change":"2024-05-09T03:10:14.194353+0000","last_active":"2024-05-09T03:10:15.178177+0000","last_peered":"2024-05-09T03:10:15.178177+0000","last_clean":"2024-05-09T03:10:15.178177+0000","last_became_active":"2024-05-09T03:10:14.193920+0000","last_became_peered":"2024-05-09T03:10:14.193920+0000","last_unstale":"2024-05-09T03:10:15.178177+0000","last_undegraded":"2024-05-09T03:10:15.178177+0000","last_fullsized":"2024-05-09T03:10:15.178177+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:52.778192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289595+0000","last_change":"2024-05-09T03:10:14.189862+0000","last_active":"2024-05-09T03:10:16.289595+0000","last_peered":"2024-05-09T03:10:16.289595+0000","last_clean":"2024-05-09T03:10:16.289595+0000","last_became_active":"2024-05-09T03:10:14.189560+0000","last_became_peered":"2024-05-09T03:10:14.189560+0000","last_unstale":"2024-05-09T03:10:16.289595+0000","last_undegraded":"2024-05-09T03:10:16.289595+0000","last_fullsized":"2024-05-09T03:10:16.289595+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:01:38.376803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289230+0000","last_change":"2024-05-09T03:10:14.193332+0000","last_active":"2024-05-09T03:10:16.289230+0000","last_peered":"2024-05-09T03:10:16.289230+0000","last_clean":"2024-05-09T03:10:16.289230+0000","last_became_active":"2024-05-09T03:10:14.193004+0000","last_became_peered":"2024-05-09T03:10:14.193004+0000","last_unstale":"2024-05-09T03:10:16.289230+0000","last_undegraded":"2024-05-09T03:10:16.289230+0000","last_fullsized":"2024-05-09T03:10:16.289230+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:02:15.873742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288631+0000","last_change":"2024-05-09T03:10:14.198496+0000","last_active":"2024-05-09T03:10:16.288631+0000","last_peered":"2024-05-09T03:10:16.288631+0000","last_clean":"2024-05-09T03:10:16.288631+0000","last_became_active":"2024-05-09T03:10:14.197782+0000","last_became_peered":"2024-05-09T03:10:14.197782+0000","last_unstale":"2024-05-09T03:10:16.288631+0000","last_undegraded":"2024-05-09T03:10:16.288631+0000","last_fullsized":"2024-05-09T03:10:16.288631+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:14:12.498605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188060+0000","last_change":"2024-05-09T03:10:14.189718+0000","last_active":"2024-05-09T03:10:15.188060+0000","last_peered":"2024-05-09T03:10:15.188060+0000","last_clean":"2024-05-09T03:10:15.188060+0000","last_became_active":"2024-05-09T03:10:14.189368+0000","last_became_peered":"2024-05-09T03:10:14.189368+0000","last_unstale":"2024-05-09T03:10:15.188060+0000","last_undegraded":"2024-05-09T03:10:15.188060+0000","last_fullsized":"2024-05-09T03:10:15.188060+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:04:28.779074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289435+0000","last_change":"2024-05-09T03:10:14.186723+0000","last_active":"2024-05-09T03:10:16.289435+0000","last_peered":"2024-05-09T03:10:16.289435+0000","last_clean":"2024-05-09T03:10:16.289435+0000","last_became_active":"2024-05-09T03:10:14.186379+0000","last_became_peered":"2024-05-09T03:10:14.186379+0000","last_unstale":"2024-05-09T03:10:16.289435+0000","last_undegraded":"2024-05-09T03:10:16.289435+0000","last_fullsized":"2024-05-09T03:10:16.289435+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:20:29.672597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289927+0000","last_change":"2024-05-09T03:10:14.189858+0000","last_active":"2024-05-09T03:10:16.289927+0000","last_peered":"2024-05-09T03:10:16.289927+0000","last_clean":"2024-05-09T03:10:16.289927+0000","last_became_active":"2024-05-09T03:10:14.189522+0000","last_became_peered":"2024-05-09T03:10:14.189522+0000","last_unstale":"2024-05-09T03:10:16.289927+0000","last_undegraded":"2024-05-09T03:10:16.289927+0000","last_fullsized":"2024-05-09T03:10:16.289927+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:55:40.059780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289874+0000","last_change":"2024-05-09T03:10:14.191527+0000","last_active":"2024-05-09T03:10:16.289874+0000","last_peered":"2024-05-09T03:10:16.289874+0000","last_clean":"2024-05-09T03:10:16.289874+0000","last_became_active":"2024-05-09T03:10:14.191202+0000","last_became_peered":"2024-05-09T03:10:14.191202+0000","last_unstale":"2024-05-09T03:10:16.289874+0000","last_undegraded":"2024-05-09T03:10:16.289874+0000","last_fullsized":"2024-05-09T03:10:16.289874+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:07.408762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188432+0000","last_change":"2024-05-09T03:10:14.190731+0000","last_active":"2024-05-09T03:10:15.188432+0000","last_peered":"2024-05-09T03:10:15.188432+0000","last_clean":"2024-05-09T03:10:15.188432+0000","last_became_active":"2024-05-09T03:10:14.189870+0000","last_became_peered":"2024-05-09T03:10:14.189870+0000","last_unstale":"2024-05-09T03:10:15.188432+0000","last_undegraded":"2024-05-09T03:10:15.188432+0000","last_fullsized":"2024-05-09T03:10:15.188432+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:02.981209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187985+0000","last_change":"2024-05-09T03:10:14.198124+0000","last_active":"2024-05-09T03:10:15.187985+0000","last_peered":"2024-05-09T03:10:15.187985+0000","last_clean":"2024-05-09T03:10:15.187985+0000","last_became_active":"2024-05-09T03:10:14.197767+0000","last_became_peered":"2024-05-09T03:10:14.197767+0000","last_unstale":"2024-05-09T03:10:15.187985+0000","last_undegraded":"2024-05-09T03:10:15.187985+0000","last_fullsized":"2024-05-09T03:10:15.187985+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:41.531958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288851+0000","last_change":"2024-05-09T03:10:14.203471+0000","last_active":"2024-05-09T03:10:16.288851+0000","last_peered":"2024-05-09T03:10:16.288851+0000","last_clean":"2024-05-09T03:10:16.288851+0000","last_became_active":"2024-05-09T03:10:14.202369+0000","last_became_peered":"2024-05-09T03:10:14.202369+0000","last_unstale":"2024-05-09T03:10:16.288851+0000","last_undegraded":"2024-05-09T03:10:16.288851+0000","last_fullsized":"2024-05-09T03:10:16.288851+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:12:04.695764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175603+0000","last_change":"2024-05-09T03:10:14.184287+0000","last_active":"2024-05-09T03:10:15.175603+0000","last_peered":"2024-05-09T03:10:15.175603+0000","last_clean":"2024-05-09T03:10:15.175603+0000","last_became_active":"2024-05-09T03:10:14.183979+0000","last_became_peered":"2024-05-09T03:10:14.183979+0000","last_unstale":"2024-05-09T03:10:15.175603+0000","last_undegraded":"2024-05-09T03:10:15.175603+0000","last_fullsized":"2024-05-09T03:10:15.175603+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:43:05.572998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175174+0000","last_change":"2024-05-09T03:10:14.192596+0000","last_active":"2024-05-09T03:10:15.175174+0000","last_peered":"2024-05-09T03:10:15.175174+0000","last_clean":"2024-05-09T03:10:15.175174+0000","last_became_active":"2024-05-09T03:10:14.192072+0000","last_became_peered":"2024-05-09T03:10:14.192072+0000","last_unstale":"2024-05-09T03:10:15.175174+0000","last_undegraded":"2024-05-09T03:10:15.175174+0000","last_fullsized":"2024-05-09T03:10:15.175174+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:30:39.880900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.173816+0000","last_change":"2024-05-09T03:10:14.183029+0000","last_active":"2024-05-09T03:10:15.173816+0000","last_peered":"2024-05-09T03:10:15.173816+0000","last_clean":"2024-05-09T03:10:15.173816+0000","last_became_active":"2024-05-09T03:10:14.182723+0000","last_became_peered":"2024-05-09T03:10:14.182723+0000","last_unstale":"2024-05-09T03:10:15.173816+0000","last_undegraded":"2024-05-09T03:10:15.173816+0000","last_fullsized":"2024-05-09T03:10:15.173816+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:38.411297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288914+0000","last_change":"2024-05-09T03:10:14.189261+0000","last_active":"2024-05-09T03:10:16.288914+0000","last_peered":"2024-05-09T03:10:16.288914+0000","last_clean":"2024-05-09T03:10:16.288914+0000","last_became_active":"2024-05-09T03:10:14.188920+0000","last_became_peered":"2024-05-09T03:10:14.188920+0000","last_unstale":"2024-05-09T03:10:16.288914+0000","last_undegraded":"2024-05-09T03:10:16.288914+0000","last_fullsized":"2024-05-09T03:10:16.288914+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:52.778192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177988+0000","last_change":"2024-05-09T03:10:14.203151+0000","last_active":"2024-05-09T03:10:15.177988+0000","last_peered":"2024-05-09T03:10:15.177988+0000","last_clean":"2024-05-09T03:10:15.177988+0000","last_became_active":"2024-05-09T03:10:14.202839+0000","last_became_peered":"2024-05-09T03:10:14.202839+0000","last_unstale":"2024-05-09T03:10:15.177988+0000","last_undegraded":"2024-05-09T03:10:15.177988+0000","last_fullsized":"2024-05-09T03:10:15.177988+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:34:48.208384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188487+0000","last_change":"2024-05-09T03:10:14.190182+0000","last_active":"2024-05-09T03:10:15.188487+0000","last_peered":"2024-05-09T03:10:15.188487+0000","last_clean":"2024-05-09T03:10:15.188487+0000","last_became_active":"2024-05-09T03:10:14.189184+0000","last_became_peered":"2024-05-09T03:10:14.189184+0000","last_unstale":"2024-05-09T03:10:15.188487+0000","last_undegraded":"2024-05-09T03:10:15.188487+0000","last_fullsized":"2024-05-09T03:10:15.188487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:43:11.492819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.178069+0000","last_change":"2024-05-09T03:10:14.197334+0000","last_active":"2024-05-09T03:10:15.178069+0000","last_peered":"2024-05-09T03:10:15.178069+0000","last_clean":"2024-05-09T03:10:15.178069+0000","last_became_active":"2024-05-09T03:10:14.197032+0000","last_became_peered":"2024-05-09T03:10:14.197032+0000","last_unstale":"2024-05-09T03:10:15.178069+0000","last_undegraded":"2024-05-09T03:10:15.178069+0000","last_fullsized":"2024-05-09T03:10:15.178069+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:59.805800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289669+0000","last_change":"2024-05-09T03:10:14.191869+0000","last_active":"2024-05-09T03:10:16.289669+0000","last_peered":"2024-05-09T03:10:16.289669+0000","last_clean":"2024-05-09T03:10:16.289669+0000","last_became_active":"2024-05-09T03:10:14.191506+0000","last_became_peered":"2024-05-09T03:10:14.191506+0000","last_unstale":"2024-05-09T03:10:16.289669+0000","last_undegraded":"2024-05-09T03:10:16.289669+0000","last_fullsized":"2024-05-09T03:10:16.289669+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:02:37.244609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175582+0000","last_change":"2024-05-09T03:10:14.193955+0000","last_active":"2024-05-09T03:10:15.175582+0000","last_peered":"2024-05-09T03:10:15.175582+0000","last_clean":"2024-05-09T03:10:15.175582+0000","last_became_active":"2024-05-09T03:10:14.193157+0000","last_became_peered":"2024-05-09T03:10:14.193157+0000","last_unstale":"2024-05-09T03:10:15.175582+0000","last_undegraded":"2024-05-09T03:10:15.175582+0000","last_fullsized":"2024-05-09T03:10:15.175582+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:01:54.360142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289122+0000","last_change":"2024-05-09T03:10:14.191195+0000","last_active":"2024-05-09T03:10:16.289122+0000","last_peered":"2024-05-09T03:10:16.289122+0000","last_clean":"2024-05-09T03:10:16.289122+0000","last_became_active":"2024-05-09T03:10:14.190700+0000","last_became_peered":"2024-05-09T03:10:14.190700+0000","last_unstale":"2024-05-09T03:10:16.289122+0000","last_undegraded":"2024-05-09T03:10:16.289122+0000","last_fullsized":"2024-05-09T03:10:16.289122+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:52.778192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175496+0000","last_change":"2024-05-09T03:10:14.189206+0000","last_active":"2024-05-09T03:10:15.175496+0000","last_peered":"2024-05-09T03:10:15.175496+0000","last_clean":"2024-05-09T03:10:15.175496+0000","last_became_active":"2024-05-09T03:10:14.188877+0000","last_became_peered":"2024-05-09T03:10:14.188877+0000","last_unstale":"2024-05-09T03:10:15.175496+0000","last_undegraded":"2024-05-09T03:10:15.175496+0000","last_fullsized":"2024-05-09T03:10:15.175496+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:30:39.880900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289854+0000","last_change":"2024-05-09T03:10:14.187298+0000","last_active":"2024-05-09T03:10:16.289854+0000","last_peered":"2024-05-09T03:10:16.289854+0000","last_clean":"2024-05-09T03:10:16.289854+0000","last_became_active":"2024-05-09T03:10:14.185101+0000","last_became_peered":"2024-05-09T03:10:14.185101+0000","last_unstale":"2024-05-09T03:10:16.289854+0000","last_undegraded":"2024-05-09T03:10:16.289854+0000","last_fullsized":"2024-05-09T03:10:16.289854+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:04:28.779074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.173971+0000","last_change":"2024-05-09T03:10:14.184964+0000","last_active":"2024-05-09T03:10:15.173971+0000","last_peered":"2024-05-09T03:10:15.173971+0000","last_clean":"2024-05-09T03:10:15.173971+0000","last_became_active":"2024-05-09T03:10:14.183486+0000","last_became_peered":"2024-05-09T03:10:14.183486+0000","last_unstale":"2024-05-09T03:10:15.173971+0000","last_undegraded":"2024-05-09T03:10:15.173971+0000","last_fullsized":"2024-05-09T03:10:15.173971+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:43:05.572998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'4","reported_seq":18,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175540+0000","last_change":"2024-05-09T03:10:13.189003+0000","last_active":"2024-05-09T03:10:15.175540+0000","last_peered":"2024-05-09T03:10:15.175540+0000","last_clean":"2024-05-09T03:10:15.175540+0000","last_became_active":"2024-05-09T03:10:13.188179+0000","last_became_peered":"2024-05-09T03:10:13.188179+0000","last_unstale":"2024-05-09T03:10:15.175540+0000","last_undegraded":"2024-05-09T03:10:15.175540+0000","last_fullsized":"2024-05-09T03:10:15.175540+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:12.038216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175408+0000","last_change":"2024-05-09T03:10:14.190854+0000","last_active":"2024-05-09T03:10:15.175408+0000","last_peered":"2024-05-09T03:10:15.175408+0000","last_clean":"2024-05-09T03:10:15.175408+0000","last_became_active":"2024-05-09T03:10:14.190520+0000","last_became_peered":"2024-05-09T03:10:14.190520+0000","last_unstale":"2024-05-09T03:10:15.175408+0000","last_undegraded":"2024-05-09T03:10:15.175408+0000","last_fullsized":"2024-05-09T03:10:15.175408+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:34:19.373988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"30'1","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.173956+0000","last_change":"2024-05-09T03:10:13.187760+0000","last_active":"2024-05-09T03:10:15.173956+0000","last_peered":"2024-05-09T03:10:15.173956+0000","last_clean":"2024-05-09T03:10:15.173956+0000","last_became_active":"2024-05-09T03:10:13.187320+0000","last_became_peered":"2024-05-09T03:10:13.187320+0000","last_unstale":"2024-05-09T03:10:15.173956+0000","last_undegraded":"2024-05-09T03:10:15.173956+0000","last_fullsized":"2024-05-09T03:10:15.173956+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:35.080152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.176427+0000","last_change":"2024-05-09T03:10:14.192931+0000","last_active":"2024-05-09T03:10:15.176427+0000","last_peered":"2024-05-09T03:10:15.176427+0000","last_clean":"2024-05-09T03:10:15.176427+0000","last_became_active":"2024-05-09T03:10:14.192188+0000","last_became_peered":"2024-05-09T03:10:14.192188+0000","last_unstale":"2024-05-09T03:10:15.176427+0000","last_undegraded":"2024-05-09T03:10:15.176427+0000","last_fullsized":"2024-05-09T03:10:15.176427+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:30.362161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"30'2","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187998+0000","last_change":"2024-05-09T03:10:13.196671+0000","last_active":"2024-05-09T03:10:15.187998+0000","last_peered":"2024-05-09T03:10:15.187998+0000","last_clean":"2024-05-09T03:10:15.187998+0000","last_became_active":"2024-05-09T03:10:13.196295+0000","last_became_peered":"2024-05-09T03:10:13.196295+0000","last_unstale":"2024-05-09T03:10:15.187998+0000","last_undegraded":"2024-05-09T03:10:15.187998+0000","last_fullsized":"2024-05-09T03:10:15.187998+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:43:04.570422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.178221+0000","last_change":"2024-05-09T03:10:14.196604+0000","last_active":"2024-05-09T03:10:15.178221+0000","last_peered":"2024-05-09T03:10:15.178221+0000","last_clean":"2024-05-09T03:10:15.178221+0000","last_became_active":"2024-05-09T03:10:14.196292+0000","last_became_peered":"2024-05-09T03:10:14.196292+0000","last_unstale":"2024-05-09T03:10:15.178221+0000","last_undegraded":"2024-05-09T03:10:15.178221+0000","last_fullsized":"2024-05-09T03:10:15.178221+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:02:15.873742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"30'2","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288815+0000","last_change":"2024-05-09T03:10:13.182175+0000","last_active":"2024-05-09T03:10:16.288815+0000","last_peered":"2024-05-09T03:10:16.288815+0000","last_clean":"2024-05-09T03:10:16.288815+0000","last_became_active":"2024-05-09T03:10:13.181796+0000","last_became_peered":"2024-05-09T03:10:13.181796+0000","last_unstale":"2024-05-09T03:10:16.288815+0000","last_undegraded":"2024-05-09T03:10:16.288815+0000","last_fullsized":"2024-05-09T03:10:16.288815+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:22:12.095099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289716+0000","last_change":"2024-05-09T03:10:14.186528+0000","last_active":"2024-05-09T03:10:16.289716+0000","last_peered":"2024-05-09T03:10:16.289716+0000","last_clean":"2024-05-09T03:10:16.289716+0000","last_became_active":"2024-05-09T03:10:14.186206+0000","last_became_peered":"2024-05-09T03:10:14.186206+0000","last_unstale":"2024-05-09T03:10:16.289716+0000","last_undegraded":"2024-05-09T03:10:16.289716+0000","last_fullsized":"2024-05-09T03:10:16.289716+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:41.531958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"30'1","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175716+0000","last_change":"2024-05-09T03:10:13.195670+0000","last_active":"2024-05-09T03:10:15.175716+0000","last_peered":"2024-05-09T03:10:15.175716+0000","last_clean":"2024-05-09T03:10:15.175716+0000","last_became_active":"2024-05-09T03:10:13.195340+0000","last_became_peered":"2024-05-09T03:10:13.195340+0000","last_unstale":"2024-05-09T03:10:15.175716+0000","last_undegraded":"2024-05-09T03:10:15.175716+0000","last_fullsized":"2024-05-09T03:10:15.175716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:48.412838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289482+0000","last_change":"2024-05-09T03:10:14.192583+0000","last_active":"2024-05-09T03:10:16.289482+0000","last_peered":"2024-05-09T03:10:16.289482+0000","last_clean":"2024-05-09T03:10:16.289482+0000","last_became_active":"2024-05-09T03:10:14.192251+0000","last_became_peered":"2024-05-09T03:10:14.192251+0000","last_unstale":"2024-05-09T03:10:16.289482+0000","last_undegraded":"2024-05-09T03:10:16.289482+0000","last_fullsized":"2024-05-09T03:10:16.289482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:24:29.032308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"30'1","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175285+0000","last_change":"2024-05-09T03:10:13.195492+0000","last_active":"2024-05-09T03:10:15.175285+0000","last_peered":"2024-05-09T03:10:15.175285+0000","last_clean":"2024-05-09T03:10:15.175285+0000","last_became_active":"2024-05-09T03:10:13.195109+0000","last_became_peered":"2024-05-09T03:10:13.195109+0000","last_unstale":"2024-05-09T03:10:15.175285+0000","last_undegraded":"2024-05-09T03:10:15.175285+0000","last_fullsized":"2024-05-09T03:10:15.175285+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:19.730755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175814+0000","last_change":"2024-05-09T03:10:14.190128+0000","last_active":"2024-05-09T03:10:15.175814+0000","last_peered":"2024-05-09T03:10:15.175814+0000","last_clean":"2024-05-09T03:10:15.175814+0000","last_became_active":"2024-05-09T03:10:14.189812+0000","last_became_peered":"2024-05-09T03:10:14.189812+0000","last_unstale":"2024-05-09T03:10:15.175814+0000","last_undegraded":"2024-05-09T03:10:15.175814+0000","last_fullsized":"2024-05-09T03:10:15.175814+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:49.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187747+0000","last_change":"2024-05-09T03:10:13.192723+0000","last_active":"2024-05-09T03:10:15.187747+0000","last_peered":"2024-05-09T03:10:15.187747+0000","last_clean":"2024-05-09T03:10:15.187747+0000","last_became_active":"2024-05-09T03:10:13.192346+0000","last_became_peered":"2024-05-09T03:10:13.192346+0000","last_unstale":"2024-05-09T03:10:15.187747+0000","last_undegraded":"2024-05-09T03:10:15.187747+0000","last_fullsized":"2024-05-09T03:10:15.187747+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:30:38.878324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177519+0000","last_change":"2024-05-09T03:10:14.189146+0000","last_active":"2024-05-09T03:10:15.177519+0000","last_peered":"2024-05-09T03:10:15.177519+0000","last_clean":"2024-05-09T03:10:15.177519+0000","last_became_active":"2024-05-09T03:10:14.188861+0000","last_became_peered":"2024-05-09T03:10:14.188861+0000","last_unstale":"2024-05-09T03:10:15.177519+0000","last_undegraded":"2024-05-09T03:10:15.177519+0000","last_fullsized":"2024-05-09T03:10:15.177519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:30.362161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175766+0000","last_change":"2024-05-09T03:10:13.185536+0000","last_active":"2024-05-09T03:10:15.175766+0000","last_peered":"2024-05-09T03:10:15.175766+0000","last_clean":"2024-05-09T03:10:15.175766+0000","last_became_active":"2024-05-09T03:10:13.185232+0000","last_became_peered":"2024-05-09T03:10:13.185232+0000","last_unstale":"2024-05-09T03:10:15.175766+0000","last_undegraded":"2024-05-09T03:10:15.175766+0000","last_fullsized":"2024-05-09T03:10:15.175766+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:41.702763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289924+0000","last_change":"2024-05-09T03:10:14.193911+0000","last_active":"2024-05-09T03:10:16.289924+0000","last_peered":"2024-05-09T03:10:16.289924+0000","last_clean":"2024-05-09T03:10:16.289924+0000","last_became_active":"2024-05-09T03:10:14.193496+0000","last_became_peered":"2024-05-09T03:10:14.193496+0000","last_unstale":"2024-05-09T03:10:16.289924+0000","last_undegraded":"2024-05-09T03:10:16.289924+0000","last_fullsized":"2024-05-09T03:10:16.289924+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:41.531958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188311+0000","last_change":"2024-05-09T03:10:13.195253+0000","last_active":"2024-05-09T03:10:15.188311+0000","last_peered":"2024-05-09T03:10:15.188311+0000","last_clean":"2024-05-09T03:10:15.188311+0000","last_became_active":"2024-05-09T03:10:13.194925+0000","last_became_peered":"2024-05-09T03:10:13.194925+0000","last_unstale":"2024-05-09T03:10:15.188311+0000","last_undegraded":"2024-05-09T03:10:15.188311+0000","last_fullsized":"2024-05-09T03:10:15.188311+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:53.293921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188380+0000","last_change":"2024-05-09T03:10:14.201538+0000","last_active":"2024-05-09T03:10:15.188380+0000","last_peered":"2024-05-09T03:10:15.188380+0000","last_clean":"2024-05-09T03:10:15.188380+0000","last_became_active":"2024-05-09T03:10:14.201240+0000","last_became_peered":"2024-05-09T03:10:14.201240+0000","last_unstale":"2024-05-09T03:10:15.188380+0000","last_undegraded":"2024-05-09T03:10:15.188380+0000","last_fullsized":"2024-05-09T03:10:15.188380+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:25.200800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.174078+0000","last_change":"2024-05-09T03:10:13.190325+0000","last_active":"2024-05-09T03:10:15.174078+0000","last_peered":"2024-05-09T03:10:15.174078+0000","last_clean":"2024-05-09T03:10:15.174078+0000","last_became_active":"2024-05-09T03:10:13.189970+0000","last_became_peered":"2024-05-09T03:10:13.189970+0000","last_unstale":"2024-05-09T03:10:15.174078+0000","last_undegraded":"2024-05-09T03:10:15.174078+0000","last_fullsized":"2024-05-09T03:10:15.174078+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:54.992976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188244+0000","last_change":"2024-05-09T03:10:14.188689+0000","last_active":"2024-05-09T03:10:15.188244+0000","last_peered":"2024-05-09T03:10:15.188244+0000","last_clean":"2024-05-09T03:10:15.188244+0000","last_became_active":"2024-05-09T03:10:14.188338+0000","last_became_peered":"2024-05-09T03:10:14.188338+0000","last_unstale":"2024-05-09T03:10:15.188244+0000","last_undegraded":"2024-05-09T03:10:15.188244+0000","last_fullsized":"2024-05-09T03:10:15.188244+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:55:40.059780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"30'3","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289813+0000","last_change":"2024-05-09T03:10:13.189057+0000","last_active":"2024-05-09T03:10:16.289813+0000","last_peered":"2024-05-09T03:10:16.289813+0000","last_clean":"2024-05-09T03:10:16.289813+0000","last_became_active":"2024-05-09T03:10:13.188543+0000","last_became_peered":"2024-05-09T03:10:13.188543+0000","last_unstale":"2024-05-09T03:10:16.289813+0000","last_undegraded":"2024-05-09T03:10:16.289813+0000","last_fullsized":"2024-05-09T03:10:16.289813+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:37.408721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289664+0000","last_change":"2024-05-09T03:10:14.195816+0000","last_active":"2024-05-09T03:10:16.289664+0000","last_peered":"2024-05-09T03:10:16.289664+0000","last_clean":"2024-05-09T03:10:16.289664+0000","last_became_active":"2024-05-09T03:10:14.195363+0000","last_became_peered":"2024-05-09T03:10:14.195363+0000","last_unstale":"2024-05-09T03:10:16.289664+0000","last_undegraded":"2024-05-09T03:10:16.289664+0000","last_fullsized":"2024-05-09T03:10:16.289664+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:15.211067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"30'6","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188167+0000","last_change":"2024-05-09T03:10:13.196415+0000","last_active":"2024-05-09T03:10:15.188167+0000","last_peered":"2024-05-09T03:10:15.188167+0000","last_clean":"2024-05-09T03:10:15.188167+0000","last_became_active":"2024-05-09T03:10:13.195796+0000","last_became_peered":"2024-05-09T03:10:13.195796+0000","last_unstale":"2024-05-09T03:10:15.188167+0000","last_undegraded":"2024-05-09T03:10:15.188167+0000","last_fullsized":"2024-05-09T03:10:15.188167+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:48.970501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177514+0000","last_change":"2024-05-09T03:10:14.199845+0000","last_active":"2024-05-09T03:10:15.177514+0000","last_peered":"2024-05-09T03:10:15.177514+0000","last_clean":"2024-05-09T03:10:15.177514+0000","last_became_active":"2024-05-09T03:10:14.199489+0000","last_became_peered":"2024-05-09T03:10:14.199489+0000","last_unstale":"2024-05-09T03:10:15.177514+0000","last_undegraded":"2024-05-09T03:10:15.177514+0000","last_fullsized":"2024-05-09T03:10:15.177514+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:02:39.990487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"30'2","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175820+0000","last_change":"2024-05-09T03:10:13.189980+0000","last_active":"2024-05-09T03:10:15.175820+0000","last_peered":"2024-05-09T03:10:15.175820+0000","last_clean":"2024-05-09T03:10:15.175820+0000","last_became_active":"2024-05-09T03:10:13.189577+0000","last_became_peered":"2024-05-09T03:10:13.189577+0000","last_unstale":"2024-05-09T03:10:15.175820+0000","last_undegraded":"2024-05-09T03:10:15.175820+0000","last_fullsized":"2024-05-09T03:10:15.175820+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:14:11.496029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.188169+0000","last_change":"2024-05-09T03:10:14.191816+0000","last_active":"2024-05-09T03:10:15.188169+0000","last_peered":"2024-05-09T03:10:15.188169+0000","last_clean":"2024-05-09T03:10:15.188169+0000","last_became_active":"2024-05-09T03:10:14.191414+0000","last_became_peered":"2024-05-09T03:10:14.191414+0000","last_unstale":"2024-05-09T03:10:15.188169+0000","last_undegraded":"2024-05-09T03:10:15.188169+0000","last_fullsized":"2024-05-09T03:10:15.188169+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:07.408762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177492+0000","last_change":"2024-05-09T03:10:13.180210+0000","last_active":"2024-05-09T03:10:15.177492+0000","last_peered":"2024-05-09T03:10:15.177492+0000","last_clean":"2024-05-09T03:10:15.177492+0000","last_became_active":"2024-05-09T03:10:13.179745+0000","last_became_peered":"2024-05-09T03:10:13.179745+0000","last_unstale":"2024-05-09T03:10:15.177492+0000","last_undegraded":"2024-05-09T03:10:15.177492+0000","last_fullsized":"2024-05-09T03:10:15.177492+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:54.992976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289479+0000","last_change":"2024-05-09T03:10:14.185764+0000","last_active":"2024-05-09T03:10:16.289479+0000","last_peered":"2024-05-09T03:10:16.289479+0000","last_clean":"2024-05-09T03:10:16.289479+0000","last_became_active":"2024-05-09T03:10:14.185380+0000","last_became_peered":"2024-05-09T03:10:14.185380+0000","last_unstale":"2024-05-09T03:10:16.289479+0000","last_undegraded":"2024-05-09T03:10:16.289479+0000","last_fullsized":"2024-05-09T03:10:16.289479+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:02:37.244609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187801+0000","last_change":"2024-05-09T03:10:13.194866+0000","last_active":"2024-05-09T03:10:15.187801+0000","last_peered":"2024-05-09T03:10:15.187801+0000","last_clean":"2024-05-09T03:10:15.187801+0000","last_became_active":"2024-05-09T03:10:13.194487+0000","last_became_peered":"2024-05-09T03:10:13.194487+0000","last_unstale":"2024-05-09T03:10:15.187801+0000","last_undegraded":"2024-05-09T03:10:15.187801+0000","last_fullsized":"2024-05-09T03:10:15.187801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:01:53.357566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175685+0000","last_change":"2024-05-09T03:10:14.191757+0000","last_active":"2024-05-09T03:10:15.175685+0000","last_peered":"2024-05-09T03:10:15.175685+0000","last_clean":"2024-05-09T03:10:15.175685+0000","last_became_active":"2024-05-09T03:10:14.191442+0000","last_became_peered":"2024-05-09T03:10:14.191442+0000","last_unstale":"2024-05-09T03:10:15.175685+0000","last_undegraded":"2024-05-09T03:10:15.175685+0000","last_fullsized":"2024-05-09T03:10:15.175685+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:01:54.360142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"30'8","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187679+0000","last_change":"2024-05-09T03:10:13.196071+0000","last_active":"2024-05-09T03:10:15.187679+0000","last_peered":"2024-05-09T03:10:15.187679+0000","last_clean":"2024-05-09T03:10:15.187679+0000","last_became_active":"2024-05-09T03:10:13.195657+0000","last_became_peered":"2024-05-09T03:10:13.195657+0000","last_unstale":"2024-05-09T03:10:15.187679+0000","last_undegraded":"2024-05-09T03:10:15.187679+0000","last_fullsized":"2024-05-09T03:10:15.187679+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:34:18.371412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.176276+0000","last_change":"2024-05-09T03:10:14.197281+0000","last_active":"2024-05-09T03:10:15.176276+0000","last_peered":"2024-05-09T03:10:15.176276+0000","last_clean":"2024-05-09T03:10:15.176276+0000","last_became_active":"2024-05-09T03:10:14.196719+0000","last_became_peered":"2024-05-09T03:10:14.196719+0000","last_unstale":"2024-05-09T03:10:15.176276+0000","last_undegraded":"2024-05-09T03:10:15.176276+0000","last_fullsized":"2024-05-09T03:10:15.176276+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:13:54.296498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"30'1","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.178072+0000","last_change":"2024-05-09T03:10:13.193371+0000","last_active":"2024-05-09T03:10:15.178072+0000","last_peered":"2024-05-09T03:10:15.178072+0000","last_clean":"2024-05-09T03:10:15.178072+0000","last_became_active":"2024-05-09T03:10:13.193021+0000","last_became_peered":"2024-05-09T03:10:13.193021+0000","last_unstale":"2024-05-09T03:10:15.178072+0000","last_undegraded":"2024-05-09T03:10:15.178072+0000","last_fullsized":"2024-05-09T03:10:15.178072+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:12.038216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289965+0000","last_change":"2024-05-09T03:10:14.196922+0000","last_active":"2024-05-09T03:10:16.289965+0000","last_peered":"2024-05-09T03:10:16.289965+0000","last_clean":"2024-05-09T03:10:16.289965+0000","last_became_active":"2024-05-09T03:10:14.196597+0000","last_became_peered":"2024-05-09T03:10:14.196597+0000","last_unstale":"2024-05-09T03:10:16.289965+0000","last_undegraded":"2024-05-09T03:10:16.289965+0000","last_fullsized":"2024-05-09T03:10:16.289965+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:07.408762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"30'2","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.176188+0000","last_change":"2024-05-09T03:10:13.186282+0000","last_active":"2024-05-09T03:10:15.176188+0000","last_peered":"2024-05-09T03:10:15.176188+0000","last_clean":"2024-05-09T03:10:15.176188+0000","last_became_active":"2024-05-09T03:10:13.185976+0000","last_became_peered":"2024-05-09T03:10:13.185976+0000","last_unstale":"2024-05-09T03:10:15.176188+0000","last_undegraded":"2024-05-09T03:10:15.176188+0000","last_fullsized":"2024-05-09T03:10:15.176188+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:41.190703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.178144+0000","last_change":"2024-05-09T03:10:14.198462+0000","last_active":"2024-05-09T03:10:15.178144+0000","last_peered":"2024-05-09T03:10:15.178144+0000","last_clean":"2024-05-09T03:10:15.178144+0000","last_became_active":"2024-05-09T03:10:14.198155+0000","last_became_peered":"2024-05-09T03:10:14.198155+0000","last_unstale":"2024-05-09T03:10:15.178144+0000","last_undegraded":"2024-05-09T03:10:15.178144+0000","last_fullsized":"2024-05-09T03:10:15.178144+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:00.128577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"30'1","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.176287+0000","last_change":"2024-05-09T03:10:13.189132+0000","last_active":"2024-05-09T03:10:15.176287+0000","last_peered":"2024-05-09T03:10:15.176287+0000","last_clean":"2024-05-09T03:10:15.176287+0000","last_became_active":"2024-05-09T03:10:13.188751+0000","last_became_peered":"2024-05-09T03:10:13.188751+0000","last_unstale":"2024-05-09T03:10:15.176287+0000","last_undegraded":"2024-05-09T03:10:15.176287+0000","last_fullsized":"2024-05-09T03:10:15.176287+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:21:28.226685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288861+0000","last_change":"2024-05-09T03:10:14.201363+0000","last_active":"2024-05-09T03:10:16.288861+0000","last_peered":"2024-05-09T03:10:16.288861+0000","last_clean":"2024-05-09T03:10:16.288861+0000","last_became_active":"2024-05-09T03:10:14.200547+0000","last_became_peered":"2024-05-09T03:10:14.200547+0000","last_unstale":"2024-05-09T03:10:16.288861+0000","last_undegraded":"2024-05-09T03:10:16.288861+0000","last_fullsized":"2024-05-09T03:10:16.288861+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:42.193279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.176266+0000","last_change":"2024-05-09T03:10:13.195590+0000","last_active":"2024-05-09T03:10:15.176266+0000","last_peered":"2024-05-09T03:10:15.176266+0000","last_clean":"2024-05-09T03:10:15.176266+0000","last_became_active":"2024-05-09T03:10:13.195146+0000","last_became_peered":"2024-05-09T03:10:13.195146+0000","last_unstale":"2024-05-09T03:10:15.176266+0000","last_undegraded":"2024-05-09T03:10:15.176266+0000","last_fullsized":"2024-05-09T03:10:15.176266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:12:03.693188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288650+0000","last_change":"2024-05-09T03:10:14.197451+0000","last_active":"2024-05-09T03:10:16.288650+0000","last_peered":"2024-05-09T03:10:16.288650+0000","last_clean":"2024-05-09T03:10:16.288650+0000","last_became_active":"2024-05-09T03:10:14.197139+0000","last_became_peered":"2024-05-09T03:10:14.197139+0000","last_unstale":"2024-05-09T03:10:16.288650+0000","last_undegraded":"2024-05-09T03:10:16.288650+0000","last_fullsized":"2024-05-09T03:10:16.288650+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:42.705339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"30'2","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.174186+0000","last_change":"2024-05-09T03:10:13.185454+0000","last_active":"2024-05-09T03:10:15.174186+0000","last_peered":"2024-05-09T03:10:15.174186+0000","last_clean":"2024-05-09T03:10:15.174186+0000","last_became_active":"2024-05-09T03:10:13.185132+0000","last_became_peered":"2024-05-09T03:10:13.185132+0000","last_unstale":"2024-05-09T03:10:15.174186+0000","last_undegraded":"2024-05-09T03:10:15.174186+0000","last_fullsized":"2024-05-09T03:10:15.174186+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:12.038216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288817+0000","last_change":"2024-05-09T03:10:14.205634+0000","last_active":"2024-05-09T03:10:16.288817+0000","last_peered":"2024-05-09T03:10:16.288817+0000","last_clean":"2024-05-09T03:10:16.288817+0000","last_became_active":"2024-05-09T03:10:14.204656+0000","last_became_peered":"2024-05-09T03:10:14.204656+0000","last_unstale":"2024-05-09T03:10:16.288817+0000","last_undegraded":"2024-05-09T03:10:16.288817+0000","last_fullsized":"2024-05-09T03:10:16.288817+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:39.482190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175727+0000","last_change":"2024-05-09T03:10:13.184457+0000","last_active":"2024-05-09T03:10:15.175727+0000","last_peered":"2024-05-09T03:10:15.175727+0000","last_clean":"2024-05-09T03:10:15.175727+0000","last_became_active":"2024-05-09T03:10:13.184152+0000","last_became_peered":"2024-05-09T03:10:13.184152+0000","last_unstale":"2024-05-09T03:10:15.175727+0000","last_undegraded":"2024-05-09T03:10:15.175727+0000","last_fullsized":"2024-05-09T03:10:15.175727+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:41.190703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288631+0000","last_change":"2024-05-09T03:10:14.203268+0000","last_active":"2024-05-09T03:10:16.288631+0000","last_peered":"2024-05-09T03:10:16.288631+0000","last_clean":"2024-05-09T03:10:16.288631+0000","last_became_active":"2024-05-09T03:10:14.202487+0000","last_became_peered":"2024-05-09T03:10:14.202487+0000","last_unstale":"2024-05-09T03:10:16.288631+0000","last_undegraded":"2024-05-09T03:10:16.288631+0000","last_fullsized":"2024-05-09T03:10:16.288631+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:49.415414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175435+0000","last_change":"2024-05-09T03:10:13.188068+0000","last_active":"2024-05-09T03:10:15.175435+0000","last_peered":"2024-05-09T03:10:15.175435+0000","last_clean":"2024-05-09T03:10:15.175435+0000","last_became_active":"2024-05-09T03:10:13.187642+0000","last_became_peered":"2024-05-09T03:10:13.187642+0000","last_unstale":"2024-05-09T03:10:15.175435+0000","last_undegraded":"2024-05-09T03:10:15.175435+0000","last_fullsized":"2024-05-09T03:10:15.175435+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:21.503239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289196+0000","last_change":"2024-05-09T03:10:14.185439+0000","last_active":"2024-05-09T03:10:16.289196+0000","last_peered":"2024-05-09T03:10:16.289196+0000","last_clean":"2024-05-09T03:10:16.289196+0000","last_became_active":"2024-05-09T03:10:14.185130+0000","last_became_peered":"2024-05-09T03:10:14.185130+0000","last_unstale":"2024-05-09T03:10:16.289196+0000","last_undegraded":"2024-05-09T03:10:16.289196+0000","last_fullsized":"2024-05-09T03:10:16.289196+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:02:15.873742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288738+0000","last_change":"2024-05-09T03:10:13.188915+0000","last_active":"2024-05-09T03:10:16.288738+0000","last_peered":"2024-05-09T03:10:16.288738+0000","last_clean":"2024-05-09T03:10:16.288738+0000","last_became_active":"2024-05-09T03:10:13.188595+0000","last_became_peered":"2024-05-09T03:10:13.188595+0000","last_unstale":"2024-05-09T03:10:16.288738+0000","last_undegraded":"2024-05-09T03:10:16.288738+0000","last_fullsized":"2024-05-09T03:10:16.288738+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:30:38.878324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.288811+0000","last_change":"2024-05-09T03:10:14.202960+0000","last_active":"2024-05-09T03:10:16.288811+0000","last_peered":"2024-05-09T03:10:16.288811+0000","last_clean":"2024-05-09T03:10:16.288811+0000","last_became_active":"2024-05-09T03:10:14.202317+0000","last_became_peered":"2024-05-09T03:10:14.202317+0000","last_unstale":"2024-05-09T03:10:16.288811+0000","last_undegraded":"2024-05-09T03:10:16.288811+0000","last_fullsized":"2024-05-09T03:10:16.288811+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:47:22.505815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"30'2","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.178076+0000","last_change":"2024-05-09T03:10:13.189007+0000","last_active":"2024-05-09T03:10:15.178076+0000","last_peered":"2024-05-09T03:10:15.178076+0000","last_clean":"2024-05-09T03:10:15.178076+0000","last_became_active":"2024-05-09T03:10:13.188639+0000","last_became_peered":"2024-05-09T03:10:13.188639+0000","last_unstale":"2024-05-09T03:10:15.178076+0000","last_undegraded":"2024-05-09T03:10:15.178076+0000","last_fullsized":"2024-05-09T03:10:15.178076+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:29.950760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289962+0000","last_change":"2024-05-09T03:10:14.195682+0000","last_active":"2024-05-09T03:10:16.289962+0000","last_peered":"2024-05-09T03:10:16.289962+0000","last_clean":"2024-05-09T03:10:16.289962+0000","last_became_active":"2024-05-09T03:10:14.195391+0000","last_became_peered":"2024-05-09T03:10:14.195391+0000","last_unstale":"2024-05-09T03:10:16.289962+0000","last_undegraded":"2024-05-09T03:10:16.289962+0000","last_fullsized":"2024-05-09T03:10:16.289962+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:04:28.779074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"31'1","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177005+0000","last_change":"2024-05-09T03:10:13.184687+0000","last_active":"2024-05-09T03:10:15.177005+0000","last_peered":"2024-05-09T03:10:15.177005+0000","last_clean":"2024-05-09T03:10:15.177005+0000","last_became_active":"2024-05-09T03:10:13.184319+0000","last_became_peered":"2024-05-09T03:10:13.184319+0000","last_unstale":"2024-05-09T03:10:15.177005+0000","last_undegraded":"2024-05-09T03:10:15.177005+0000","last_fullsized":"2024-05-09T03:10:15.177005+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:17.683246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.173959+0000","last_change":"2024-05-09T03:10:14.189905+0000","last_active":"2024-05-09T03:10:15.173959+0000","last_peered":"2024-05-09T03:10:15.173959+0000","last_clean":"2024-05-09T03:10:15.173959+0000","last_became_active":"2024-05-09T03:10:14.189631+0000","last_became_peered":"2024-05-09T03:10:14.189631+0000","last_unstale":"2024-05-09T03:10:15.173959+0000","last_undegraded":"2024-05-09T03:10:15.173959+0000","last_fullsized":"2024-05-09T03:10:15.173959+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:01:54.360142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177951+0000","last_change":"2024-05-09T03:10:13.182329+0000","last_active":"2024-05-09T03:10:15.177951+0000","last_peered":"2024-05-09T03:10:15.177951+0000","last_clean":"2024-05-09T03:10:15.177951+0000","last_became_active":"2024-05-09T03:10:13.181341+0000","last_became_peered":"2024-05-09T03:10:13.181341+0000","last_unstale":"2024-05-09T03:10:15.177951+0000","last_undegraded":"2024-05-09T03:10:15.177951+0000","last_fullsized":"2024-05-09T03:10:15.177951+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:17:35.080152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175597+0000","last_change":"2024-05-09T03:10:14.185813+0000","last_active":"2024-05-09T03:10:15.175597+0000","last_peered":"2024-05-09T03:10:15.175597+0000","last_clean":"2024-05-09T03:10:15.175597+0000","last_became_active":"2024-05-09T03:10:14.185485+0000","last_became_peered":"2024-05-09T03:10:14.185485+0000","last_unstale":"2024-05-09T03:10:15.175597+0000","last_undegraded":"2024-05-09T03:10:15.175597+0000","last_fullsized":"2024-05-09T03:10:15.175597+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:38.411297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.174227+0000","last_change":"2024-05-09T03:10:13.188877+0000","last_active":"2024-05-09T03:10:15.174227+0000","last_peered":"2024-05-09T03:10:15.174227+0000","last_clean":"2024-05-09T03:10:15.174227+0000","last_became_active":"2024-05-09T03:10:13.188503+0000","last_became_peered":"2024-05-09T03:10:13.188503+0000","last_unstale":"2024-05-09T03:10:15.174227+0000","last_undegraded":"2024-05-09T03:10:15.174227+0000","last_fullsized":"2024-05-09T03:10:15.174227+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:29.950760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.174117+0000","last_change":"2024-05-09T03:10:14.177229+0000","last_active":"2024-05-09T03:10:15.174117+0000","last_peered":"2024-05-09T03:10:15.174117+0000","last_clean":"2024-05-09T03:10:15.174117+0000","last_became_active":"2024-05-09T03:10:14.176926+0000","last_became_peered":"2024-05-09T03:10:14.176926+0000","last_unstale":"2024-05-09T03:10:15.174117+0000","last_undegraded":"2024-05-09T03:10:15.174117+0000","last_fullsized":"2024-05-09T03:10:15.174117+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:21:49.973077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.177605+0000","last_change":"2024-05-09T03:10:13.190300+0000","last_active":"2024-05-09T03:10:15.177605+0000","last_peered":"2024-05-09T03:10:15.177605+0000","last_clean":"2024-05-09T03:10:15.177605+0000","last_became_active":"2024-05-09T03:10:13.189467+0000","last_became_peered":"2024-05-09T03:10:13.189467+0000","last_unstale":"2024-05-09T03:10:15.177605+0000","last_undegraded":"2024-05-09T03:10:15.177605+0000","last_fullsized":"2024-05-09T03:10:15.177605+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:17.683246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187660+0000","last_change":"2024-05-09T03:10:14.195733+0000","last_active":"2024-05-09T03:10:15.187660+0000","last_peered":"2024-05-09T03:10:15.187660+0000","last_clean":"2024-05-09T03:10:15.187660+0000","last_became_active":"2024-05-09T03:10:14.195416+0000","last_became_peered":"2024-05-09T03:10:14.195416+0000","last_unstale":"2024-05-09T03:10:15.187660+0000","last_undegraded":"2024-05-09T03:10:15.187660+0000","last_fullsized":"2024-05-09T03:10:15.187660+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:02:37.244609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175537+0000","last_change":"2024-05-09T03:10:13.189469+0000","last_active":"2024-05-09T03:10:15.175537+0000","last_peered":"2024-05-09T03:10:15.175537+0000","last_clean":"2024-05-09T03:10:15.175537+0000","last_became_active":"2024-05-09T03:10:13.188143+0000","last_became_peered":"2024-05-09T03:10:13.188143+0000","last_unstale":"2024-05-09T03:10:15.175537+0000","last_undegraded":"2024-05-09T03:10:15.175537+0000","last_fullsized":"2024-05-09T03:10:15.175537+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:45:38.479614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175336+0000","last_change":"2024-05-09T03:10:14.196276+0000","last_active":"2024-05-09T03:10:15.175336+0000","last_peered":"2024-05-09T03:10:15.175336+0000","last_clean":"2024-05-09T03:10:15.175336+0000","last_became_active":"2024-05-09T03:10:14.195923+0000","last_became_peered":"2024-05-09T03:10:14.195923+0000","last_unstale":"2024-05-09T03:10:15.175336+0000","last_undegraded":"2024-05-09T03:10:15.175336+0000","last_fullsized":"2024-05-09T03:10:15.175336+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:38.411297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.187769+0000","last_change":"2024-05-09T03:10:13.194832+0000","last_active":"2024-05-09T03:10:15.187769+0000","last_peered":"2024-05-09T03:10:15.187769+0000","last_clean":"2024-05-09T03:10:15.187769+0000","last_became_active":"2024-05-09T03:10:13.194466+0000","last_became_peered":"2024-05-09T03:10:13.194466+0000","last_unstale":"2024-05-09T03:10:15.187769+0000","last_undegraded":"2024-05-09T03:10:15.187769+0000","last_fullsized":"2024-05-09T03:10:15.187769+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:37.408721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289293+0000","last_change":"2024-05-09T03:10:14.207270+0000","last_active":"2024-05-09T03:10:16.289293+0000","last_peered":"2024-05-09T03:10:16.289293+0000","last_clean":"2024-05-09T03:10:16.289293+0000","last_became_active":"2024-05-09T03:10:14.206979+0000","last_became_peered":"2024-05-09T03:10:14.206979+0000","last_unstale":"2024-05-09T03:10:16.289293+0000","last_undegraded":"2024-05-09T03:10:16.289293+0000","last_fullsized":"2024-05-09T03:10:16.289293+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:13.161299+0000","last_clean_scrub_stamp":"2024-05-09T03:10:13.161299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:20:29.672597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:15.175137+0000","last_change":"2024-05-09T03:10:13.188534+0000","last_active":"2024-05-09T03:10:15.175137+0000","last_peered":"2024-05-09T03:10:15.175137+0000","last_clean":"2024-05-09T03:10:15.175137+0000","last_became_active":"2024-05-09T03:10:13.188195+0000","last_became_peered":"2024-05-09T03:10:13.188195+0000","last_unstale":"2024-05-09T03:10:15.175137+0000","last_undegraded":"2024-05-09T03:10:15.175137+0000","last_fullsized":"2024-05-09T03:10:15.175137+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:10:12.158723+0000","last_clean_scrub_stamp":"2024-05-09T03:10:12.158723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:24:10.686565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":129,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-05-09T03:10:16.289919+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:10:16.289919+0000","last_peered":"2024-05-09T03:10:16.289919+0000","last_clean":"2024-05-09T03:10:16.289919+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:10:16.289919+0000","last_undegraded":"2024-05-09T03:10:16.289919+0000","last_fullsized":"2024-05-09T03:10:16.289919+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":933888,"data_stored":718284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":356352,"data_stored":134742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":344064,"data_stored":128498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":356352,"data_stored":134230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":364544,"data_stored":135760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":339968,"data_stored":127896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":364544,"data_stored":135760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":724494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:10:18.659 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:10:18.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T03:10:18.674 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:10:18.674+0000 7f1f69357640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:10:18.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.795+0000 7f137d6d1640 1 Processor -- start 2024-05-09T03:10:18.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.795+0000 7f137d6d1640 1 -- start start 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.796+0000 7f137d6d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 0x7f13780b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.796+0000 7f137d6d1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13780af220 0x7f13780b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.796+0000 7f137d6d1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13780b10f0 con 0x7f13780b5b50 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.796+0000 7f137d6d1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f13780b6d30 con 0x7f13780af220 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.796+0000 7f137d6d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f13780b0ea0 con 0x7f13781577b0 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f13767fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13780b5b50 0x7f13780aeb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35768/0 (socket says 172.21.15.46:35768) 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f13767fc640 1 -- 172.21.15.46:0/3419744564 learned_addr learned my addr 172.21.15.46:0/3419744564 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:18.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1376ffd640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 0x7f13780b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f13777fe640 1 --2- 172.21.15.46:0/3419744564 >> v2:172.21.15.88:3300/0 conn(0x7f13780af220 0x7f13780b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1005284059 0 0) 0x7f13780b10f0 con 0x7f13780b5b50 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1364003520 con 0x7f13780b5b50 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 524910617 0 0) 0x7f1364003520 con 0x7f13780b5b50 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 msgr2=0x7f13780b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 0x7f13780b67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 >> v2:172.21.15.88:3300/0 conn(0x7f13780af220 msgr2=0x7f13780b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 --2- 172.21.15.46:0/3419744564 >> v2:172.21.15.88:3300/0 conn(0x7f13780af220 0x7f13780b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13780b5070 con 0x7f13780b5b50 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f13600016f0 con 0x7f13780b5b50 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 >> v1:172.21.15.46:6789/0 conn(0x7f13780b5b50 legacy=0x7f13780aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1364003a00 0x7f1364003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f1364004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 0x7f1364005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13780b6d30 con 0x7f1364004800 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f13780b0ea0 con 0x7f1364003a00 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f136400baf0 con 0x7f13781577b0 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f13767fc640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f1364004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.797+0000 7f13777fe640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 0x7f1364005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f1376ffd640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1364003a00 0x7f1364003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f13767fc640 1 -- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 msgr2=0x7f1364005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f13767fc640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13781577b0 0x7f1364005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f13767fc640 1 -- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1364003a00 msgr2=0x7f1364003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f13767fc640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1364003a00 0x7f1364003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f13767fc640 1 -- 172.21.15.46:0/3419744564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1364003520 con 0x7f1364004800 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f1376ffd640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1364003a00 0x7f1364003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.798+0000 7f13767fc640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f1364004c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1378161ca0 tx=0x7f1364005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a565d9616db07188 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360007d50 con 0x7f1364004800 2024-05-09T03:10:18.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1360005ad0 con 0x7f1364004800 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f1375ffb640 1 -- 172.21.15.46:0/3419744564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360007070 con 0x7f1364004800 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f137d6d1640 1 -- 172.21.15.46:0/3419744564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 msgr2=0x7f1364004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3419744564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f1364004c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1378161ca0 tx=0x7f1364005830 comp rx=0 tx=0).stop 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f13777fe640 1 -- 172.21.15.46:0/3419744564 reap_dead start 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f137d6d1640 1 -- 172.21.15.46:0/3419744564 shutdown_connections 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f137d6d1640 1 -- 172.21.15.46:0/3419744564 >> 172.21.15.46:0/3419744564 conn(0x7f13780d1980 msgr2=0x7f13780d1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f137d6d1640 1 -- 172.21.15.46:0/3419744564 shutdown_connections 2024-05-09T03:10:18.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.799+0000 7f137d6d1640 1 -- 172.21.15.46:0/3419744564 wait complete. 2024-05-09T03:10:18.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.801+0000 7f137d6d1640 1 Processor -- start 2024-05-09T03:10:18.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.801+0000 7f137d6d1640 1 -- start start 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f137813cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f137813d220 0x7f137813fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13781400e0 0x7f13781404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13780b13d0 con 0x7f1364004800 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f13780b5630 con 0x7f13781400e0 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f13780b67f0 con 0x7f137813d220 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f137813cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f137813cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39786/0 (socket says 172.21.15.46:39786) 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 -- 172.21.15.46:0/3376491748 learned_addr learned my addr 172.21.15.46:0/3376491748 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f13777fe640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13781400e0 0x7f13781404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 -- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f137813d220 msgr2=0x7f137813fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f137813d220 0x7f137813fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 -- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13781400e0 msgr2=0x7f13781404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13781400e0 0x7f13781404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 -- 172.21.15.46:0/3376491748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137813d630 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f1376ffd640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f137813cce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1370007ea0 tx=0x7f1370007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137001a020 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1370016470 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137815f110 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370016840 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.803+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137813d9d0 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.804+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f137806a270 con 0x7f1364004800 2024-05-09T03:10:18.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.805+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1370005d10 con 0x7f1364004800 2024-05-09T03:10:18.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.806+0000 7f136f7fe640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13580762d0 0x7f13580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:18.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.806+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f13700988b0 con 0x7f1364004800 2024-05-09T03:10:18.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.806+0000 7f13767fc640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13580762d0 0x7f13580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:18.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.806+0000 7f13767fc640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13580762d0 0x7f13580787e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1378161ca0 tx=0x7f1360009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:18.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:18.809+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1370024030 con 0x7f1364004800 2024-05-09T03:10:19.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.080+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f13780d1140 con 0x7f1364004800 2024-05-09T03:10:19.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.083+0000 7f136f7fe640 1 -- 172.21.15.46:0/3376491748 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) ==== 190+0+147 (secure 0 0 0) 0x7f1370062260 con 0x7f1364004800 2024-05-09T03:10:19.084 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:10:19.084 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:10:19.085 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-05-09T03:10:19.085 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-05-09T03:10:19.085 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T03:10:19.085 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-05-09T03:10:19.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13580762d0 msgr2=0x7f13580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13580762d0 0x7f13580787e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1378161ca0 tx=0x7f1360009030 comp rx=0 tx=0).stop 2024-05-09T03:10:19.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 msgr2=0x7f137813cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f137813cce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1370007ea0 tx=0x7f1370007f90 comp rx=0 tx=0).stop 2024-05-09T03:10:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 shutdown_connections 2024-05-09T03:10:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13781400e0 0x7f13781404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f137813d220 0x7f137813fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13580762d0 0x7f13580787e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 --2- 172.21.15.46:0/3376491748 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1364004800 0x7f137813cce0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.086+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 >> 172.21.15.46:0/3376491748 conn(0x7f13780d1980 msgr2=0x7f13780732f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.087+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 shutdown_connections 2024-05-09T03:10:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.087+0000 7f137d6d1640 1 -- 172.21.15.46:0/3376491748 wait complete. 2024-05-09T03:10:19.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:19.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.216+0000 7f71925a8640 1 Processor -- start 2024-05-09T03:10:19.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.216+0000 7f71925a8640 1 -- start start 2024-05-09T03:10:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f71925a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f718c0b9200 0x7f718c08d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f71925a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c08d6e0 0x7f718c0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f71925a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f718c0b4ee0 con 0x7f718c0d20a0 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f71925a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f718c0b5460 con 0x7f718c0b9200 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f71925a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f718c0cfac0 con 0x7f718c08d6e0 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f718bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f718c0b9200 0x7f718c08d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f718bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f718c0b9200 0x7f718c08d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39798/0 (socket says 172.21.15.46:39798) 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f718bfff640 1 -- 172.21.15.46:0/2649794997 learned_addr learned my addr 172.21.15.46:0/2649794997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.217+0000 7f7183fff640 1 --2- 172.21.15.46:0/2649794997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c08d6e0 0x7f718c0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.218+0000 7f718bfff640 1 -- 172.21.15.46:0/2649794997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c08d6e0 msgr2=0x7f718c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.218+0000 7f718bfff640 1 --2- 172.21.15.46:0/2649794997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c08d6e0 0x7f718c0d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.218+0000 7f718bfff640 1 -- 172.21.15.46:0/2649794997 >> v1:172.21.15.88:6789/0 conn(0x7f718c0d20a0 legacy=0x7f718c162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.218+0000 7f718bfff640 1 -- 172.21.15.46:0/2649794997 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718c0b62e0 con 0x7f718c0b9200 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.218+0000 7f718bfff640 1 --2- 172.21.15.46:0/2649794997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0b9200 0x7f718c08d1a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f717c009870 tx=0x7f717c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f590b0d186521f15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f718b7fe640 1 -- 172.21.15.46:0/2649794997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c037070 con 0x7f718c0b9200 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f718b7fe640 1 -- 172.21.15.46:0/2649794997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717c032440 con 0x7f718c0b9200 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f718b7fe640 1 -- 172.21.15.46:0/2649794997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c032810 con 0x7f718c0b9200 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 -- 172.21.15.46:0/2649794997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0b9200 msgr2=0x7f718c08d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 --2- 172.21.15.46:0/2649794997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0b9200 0x7f718c08d1a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f717c009870 tx=0x7f717c02a270 comp rx=0 tx=0).stop 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 -- 172.21.15.46:0/2649794997 shutdown_connections 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 --2- 172.21.15.46:0/2649794997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c08d6e0 0x7f718c0d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 --2- 172.21.15.46:0/2649794997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0b9200 0x7f718c08d1a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 -- 172.21.15.46:0/2649794997 >> 172.21.15.46:0/2649794997 conn(0x7f718c0b56e0 msgr2=0x7f718c0b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 -- 172.21.15.46:0/2649794997 shutdown_connections 2024-05-09T03:10:19.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.219+0000 7f71925a8640 1 -- 172.21.15.46:0/2649794997 wait complete. 2024-05-09T03:10:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.221+0000 7f71925a8640 1 Processor -- start 2024-05-09T03:10:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.221+0000 7f71925a8640 1 -- start start 2024-05-09T03:10:19.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.222+0000 7f71925a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f718c08d6e0 0x7f718c157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:19.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.222+0000 7f71925a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c0b9200 0x7f718c145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.222+0000 7f71925a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 0x7f718c148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.222+0000 7f71925a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f718c0cf750 con 0x7f718c0d20a0 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.222+0000 7f71925a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f718c0cf470 con 0x7f718c08d6e0 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f71925a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f718c0b4a20 con 0x7f718c0b9200 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 0x7f718c148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 0x7f718c148cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39802/0 (socket says 172.21.15.46:39802) 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 -- 172.21.15.46:0/2560135813 learned_addr learned my addr 172.21.15.46:0/2560135813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7183fff640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c0b9200 0x7f718c145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f718bfff640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f718c08d6e0 0x7f718c157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 -- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c0b9200 msgr2=0x7f718c145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c0b9200 0x7f718c145a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 -- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f718c08d6e0 msgr2=0x7f718c157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f718c08d6e0 0x7f718c157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 -- 172.21.15.46:0/2560135813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718c143840 con 0x7f718c0d20a0 2024-05-09T03:10:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f7190b45640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 0x7f718c148cb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f718400c6f0 tx=0x7f718400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184003970 con 0x7f718c0d20a0 2024-05-09T03:10:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184003b10 con 0x7f718c0d20a0 2024-05-09T03:10:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718400a850 con 0x7f718c0d20a0 2024-05-09T03:10:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.223+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718c143b20 con 0x7f718c0d20a0 2024-05-09T03:10:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.224+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f718c160110 con 0x7f718c0d20a0 2024-05-09T03:10:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.224+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f718c06a270 con 0x7f718c0d20a0 2024-05-09T03:10:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.225+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f718400a9f0 con 0x7f718c0d20a0 2024-05-09T03:10:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.226+0000 7f71897fa640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f716c0762d0 0x7f716c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.226+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f7184097420 con 0x7f718c0d20a0 2024-05-09T03:10:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.226+0000 7f718bfff640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f716c0762d0 0x7f716c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.226+0000 7f718bfff640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f716c0762d0 0x7f716c0787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f717c034000 tx=0x7f717c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:19.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.230+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7184060600 con 0x7f718c0d20a0 2024-05-09T03:10:19.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.458+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f718c1e8140 con 0x7f718c0d20a0 2024-05-09T03:10:19.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.459+0000 7f71897fa640 1 -- 172.21.15.46:0/2560135813 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7f7184064900 con 0x7f718c0d20a0 2024-05-09T03:10:19.459 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:19.460 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:16.576454+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4809":{"gid":4809,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/3490266983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3490266983},{"type":"v1","addr":"172.21.15.46:6834","nonce":3490266983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4809,"qdb_cluster":[4809]},"id":2}]} 2024-05-09T03:10:19.460 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 15 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f716c0762d0 msgr2=0x7f716c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f716c0762d0 0x7f716c0787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f717c034000 tx=0x7f717c02e000 comp rx=0 tx=0).stop 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 msgr2=0x7f718c148cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 0x7f718c148cb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f718400c6f0 tx=0x7f718400cbb0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 shutdown_connections 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f716c0762d0 0x7f716c0787e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f718c0d20a0 0x7f718c148cb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f718c0b9200 0x7f718c145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 --2- 172.21.15.46:0/2560135813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f718c08d6e0 0x7f718c157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 >> 172.21.15.46:0/2560135813 conn(0x7f718c0b56e0 msgr2=0x7f718c1e72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 shutdown_connections 2024-05-09T03:10:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:19.461+0000 7f71925a8640 1 -- 172.21.15.46:0/2560135813 wait complete. 2024-05-09T03:10:19.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:10:13.572699+0000', 'modified': '2024-05-09T03:10:16.576454+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4809}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4809': {'gid': 4809, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6834/3490266983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 3490266983}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 3490266983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4809, 'qdb_cluster': [4809]} 2024-05-09T03:10:19.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:10:19.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T03:10:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.000+0000 7ffa27927640 1 Processor -- start 2024-05-09T03:10:20.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.000+0000 7ffa27927640 1 -- start start 2024-05-09T03:10:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa27927640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffa200d2e90 0x7ffa200b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa27927640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa200d3400 con 0x7ffa201605b0 2024-05-09T03:10:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa27927640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa200d4a60 con 0x7ffa200b4fc0 2024-05-09T03:10:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa27927640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ffa200d4d40 con 0x7ffa200d2e90 2024-05-09T03:10:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa24ec2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffa201605b0 0x7ffa200d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35774/0 (socket says 172.21.15.46:35774) 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1668203293 learned_addr learned my addr 172.21.15.46:0/1668203293 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1668203293 >> v2:172.21.15.88:3300/0 conn(0x7ffa200d2e90 0x7ffa200b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609194890 0 0) 0x7ffa200d3400 con 0x7ffa201605b0 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa0c003520 con 0x7ffa201605b0 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1317201619 0 0) 0x7ffa0c003520 con 0x7ffa201605b0 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 >> v2:172.21.15.88:3300/0 conn(0x7ffa200d2e90 msgr2=0x7ffa200b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 --2- 172.21.15.46:0/1668203293 >> v2:172.21.15.88:3300/0 conn(0x7ffa200d2e90 0x7ffa200b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 >> v1:172.21.15.88:6789/0 conn(0x7ffa200b4fc0 legacy=0x7ffa200d3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa200d5020 con 0x7ffa201605b0 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ffa10001710 con 0x7ffa201605b0 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 >> v1:172.21.15.46:6789/0 conn(0x7ffa201605b0 legacy=0x7ffa200d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c003a00 0x7ffa0c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa0c005270 0x7ffa0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.002+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200d4d40 con 0x7ffa0c004800 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c00c200 con 0x7ffa0c005270 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0c00c400 con 0x7ffa0c003a00 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c003a00 0x7ffa0c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa0c005270 0x7ffa0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c003a00 msgr2=0x7ffa0c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c003a00 0x7ffa0c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa0c005270 msgr2=0x7ffa0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa0c005270 0x7ffa0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa25ec4640 1 -- 172.21.15.46:0/1668203293 reap_dead start 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.003+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1668203293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c003520 con 0x7ffa0c004800 2024-05-09T03:10:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.004+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa200d3330 tx=0x7ffa0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6998d6cabc196a98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.004+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10007db0 con 0x7ffa0c004800 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.004+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10005ad0 con 0x7ffa0c004800 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.004+0000 7ffa17fff640 1 -- 172.21.15.46:0/1668203293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10007070 con 0x7ffa0c004800 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.004+0000 7ffa27927640 1 -- 172.21.15.46:0/1668203293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c004800 msgr2=0x7ffa0c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.004+0000 7ffa27927640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffa200d3330 tx=0x7ffa0c005830 comp rx=0 tx=0).stop 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.005+0000 7ffa27927640 1 -- 172.21.15.46:0/1668203293 shutdown_connections 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.005+0000 7ffa27927640 1 --2- 172.21.15.46:0/1668203293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa0c004800 0x7ffa0c004c10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.005+0000 7ffa27927640 1 -- 172.21.15.46:0/1668203293 >> 172.21.15.46:0/1668203293 conn(0x7ffa2008d390 msgr2=0x7ffa2008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.005+0000 7ffa27927640 1 -- 172.21.15.46:0/1668203293 shutdown_connections 2024-05-09T03:10:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.005+0000 7ffa27927640 1 -- 172.21.15.46:0/1668203293 wait complete. 2024-05-09T03:10:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.006+0000 7ffa27927640 1 Processor -- start 2024-05-09T03:10:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.007+0000 7ffa27927640 1 -- start start 2024-05-09T03:10:20.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa27927640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c004800 0x7ffa201531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa27927640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 0x7ffa20153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa27927640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa200cade0 0x7ffa200cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa27927640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200b4d80 con 0x7ffa200ceff0 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa27927640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200b4c00 con 0x7ffa200cade0 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa27927640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ffa200b7760 con 0x7ffa0c004800 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 0x7ffa20153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 0x7ffa20153d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39820/0 (socket says 172.21.15.46:39820) 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 -- 172.21.15.46:0/2920579019 learned_addr learned my addr 172.21.15.46:0/2920579019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa256c3640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c004800 0x7ffa201531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa200cade0 0x7ffa200cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 -- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c004800 msgr2=0x7ffa201531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c004800 0x7ffa201531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 -- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa200cade0 msgr2=0x7ffa200cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa200cade0 0x7ffa200cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 -- 172.21.15.46:0/2920579019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa200c9d40 con 0x7ffa200ceff0 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.008+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 0x7ffa20153d30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c300 tx=0x7ffa200d4c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.009+0000 7ffa15ffb640 1 -- 172.21.15.46:0/2920579019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10001510 con 0x7ffa200ceff0 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.009+0000 7ffa15ffb640 1 -- 172.21.15.46:0/2920579019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10040070 con 0x7ffa200ceff0 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.009+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa200ca020 con 0x7ffa200ceff0 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.009+0000 7ffa15ffb640 1 -- 172.21.15.46:0/2920579019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa100047e0 con 0x7ffa200ceff0 2024-05-09T03:10:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.009+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa200ca5e0 con 0x7ffa200ceff0 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.011+0000 7ffa15ffb640 1 -- 172.21.15.46:0/2920579019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa1003a070 con 0x7ffa200ceff0 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.012+0000 7ffa15ffb640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa04076300 0x7ffa04078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.012+0000 7ffa15ffb640 1 -- 172.21.15.46:0/2920579019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7ffa100b6990 con 0x7ffa200ceff0 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.012+0000 7ffa256c3640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa04076300 0x7ffa04078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.012+0000 7ffa2e7da740 1 Processor -- start 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.012+0000 7ffa2e7da740 1 -- start start 2024-05-09T03:10:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.013+0000 7ffa256c3640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa04076300 0x7ffa04078810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffa2006f130 tx=0x7ffa1c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703defeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e0abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e0b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c703d284e0 con 0x55c7039ff590 2024-05-09T03:10:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c703cc3ff0 con 0x55c7039f73b0 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c703d244a0 con 0x55c703a20270 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e0b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e0b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39834/0 (socket says 172.21.15.46:39834) 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 -- 172.21.15.46:0/1999658239 learned_addr learned my addr 172.21.15.46:0/1999658239 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703defeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e0abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 -- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 msgr2=0x55c703defeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703defeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 -- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 msgr2=0x55c703e0abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e0abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 -- 172.21.15.46:0/1999658239 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c703caa720 con 0x55c7039ff590 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.014+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e0b120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffa18001520 tx=0x7ffa180016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa2d47177c8e601 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa0affd640 1 -- 172.21.15.46:0/1999658239 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18005480 con 0x55c7039ff590 2024-05-09T03:10:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa0affd640 1 -- 172.21.15.46:0/1999658239 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa18005620 con 0x55c7039ff590 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa0affd640 1 -- 172.21.15.46:0/1999658239 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa180057d0 con 0x55c7039ff590 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1999658239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 msgr2=0x55c703e0b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e0b120 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffa18001520 tx=0x7ffa180016f0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1999658239 shutdown_connections 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e0b120 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e0abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1999658239 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703defeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1999658239 >> 172.21.15.46:0/1999658239 conn(0x55c703a143c0 msgr2=0x55c703dc5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1999658239 shutdown_connections 2024-05-09T03:10:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.015+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1999658239 wait complete. 2024-05-09T03:10:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.016+0000 7ffa2e7da740 1 -- start start 2024-05-09T03:10:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.016+0000 7ffa2e7da740 1 Processor -- start 2024-05-09T03:10:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703de7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703df1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703e07df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c703caf760 con 0x55c7039ff590 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c703e06ba0 con 0x55c7039f73b0 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c703df2730 con 0x55c703a20270 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa2e7da740 10 client.0 _ll_register_callbacks cb 0x55c703cd24b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703df1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703df1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39840/0 (socket says 172.21.15.46:39840) 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1343182146 learned_addr learned my addr 172.21.15.46:0/1343182146 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703de7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703e07df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 msgr2=0x55c703e07df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703e07df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 msgr2=0x55c703de7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703de7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1343182146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c703cf0bd0 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703e07df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703df1e10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa1007c150 tx=0x7ffa1007f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c6b3158f15539 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.018+0000 7ff9fffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ff9fffff640 1 -- 172.21.15.46:0/1343182146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1007fb60 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ff9fffff640 1 -- 172.21.15.46:0/1343182146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7ffa100884a0 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa18005620 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ff9fffff640 1 -- 172.21.15.46:0/1343182146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7ffa1009f050 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 10 client.4962 fetch_fsmap learned FSMap version 15 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55c703cd6280 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ff9fffff640 1 -- 172.21.15.46:0/1343182146 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa18005620 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ff9fffff640 1 -- 172.21.15.46:0/1343182146 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7ffa100a1070 con 0x55c7039ff590 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 10 client.4962 fetch_fsmap finished waiting for FSMap version 15 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 10 client.4962 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 20 client.4962 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c7039dee40 0x55c703e05b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 4 client.4962 mds_command: new command op to 4809 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 --> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c703d360f0 con 0x55c7039dee40 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.019+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c7039dee40 0x55c703e05b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ff9fffff640 10 client.4962 ms_handle_connect on v2:172.21.15.46:6832/3490266983 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c7039dee40 0x55c703e05b30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1b384781ab2b4a server_cookie=23c9f54430e359aa in_seq=0 out_seq=0 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ff9fffff640 1 -- 172.21.15.46:0/1343182146 <== mds.0 v2:172.21.15.46:6832/3490266983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c703d360f0 con 0x55c7039dee40 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ff9fffff640 10 client.4962 handle_command_reply: tid=0 2024-05-09T03:10:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ffa2e7da740 1 client.4962 shutdown 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 msgr2=0x55c703df1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.020+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703df1e10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa1007c150 tx=0x7ffa1007f2d0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 shutdown_connections 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c7039dee40 0x55c703e05b30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c703a20270 0x55c703e07df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703df1e10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ff9fffff640 0 client.4962 ms_handle_reset on v2:172.21.15.46:6832/3490266983 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1343182146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703de7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 >> 172.21.15.46:0/1343182146 conn(0x55c703a143c0 msgr2=0x55c703dc5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 shutdown_connections 2024-05-09T03:10:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1343182146 wait complete. 2024-05-09T03:10:20.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.021+0000 7ffa2e7da740 20 client.4962 trim_cache size 0 max 16384 2024-05-09T03:10:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.040+0000 7ffa2e7da740 1 Processor -- start 2024-05-09T03:10:20.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.040+0000 7ffa2e7da740 1 -- start start 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039f73b0 0x55c703e08da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e05b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c703a20270 0x55c703e06070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa20202200 con 0x55c7039ff590 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200cd820 con 0x55c703a20270 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ffa20201be0 con 0x55c7039f73b0 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e05b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e05b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39852/0 (socket says 172.21.15.46:39852) 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1420161492 learned_addr learned my addr 172.21.15.46:0/1420161492 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa256c3640 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039f73b0 0x55c703e08da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c703a20270 0x55c703e06070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039f73b0 msgr2=0x55c703e08da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039f73b0 0x55c703e08da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c703a20270 msgr2=0x55c703e06070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c703a20270 0x55c703e06070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 -- 172.21.15.46:0/1420161492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c703a8b850 con 0x55c7039ff590 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c703a20270 0x55c703e06070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.041+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e05b30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa10082010 tx=0x7ffa10083000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac9a1ae2c05c78b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/1420161492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa100b3040 con 0x55c7039ff590 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/1420161492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10088930 con 0x55c7039ff590 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa0a7fc640 1 -- 172.21.15.46:0/1420161492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1007fb60 con 0x55c7039ff590 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1420161492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 msgr2=0x55c703e05b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e05b30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ffa10082010 tx=0x7ffa10083000 comp rx=0 tx=0).stop 2024-05-09T03:10:20.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1420161492 shutdown_connections 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c703a20270 0x55c703e06070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c7039ff590 0x55c703e05b30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/1420161492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039f73b0 0x55c703e08da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1420161492 >> 172.21.15.46:0/1420161492 conn(0x55c703a143c0 msgr2=0x55c703de7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1420161492 shutdown_connections 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.042+0000 7ffa2e7da740 1 -- 172.21.15.46:0/1420161492 wait complete. 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.043+0000 7ffa2e7da740 1 -- start start 2024-05-09T03:10:20.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.043+0000 7ffa2e7da740 1 Processor -- start 2024-05-09T03:10:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e8a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039ff590 0x55c703e05b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 0x55c703e06040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c703e09f80 con 0x55c703a20270 2024-05-09T03:10:20.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c703e06a00 con 0x55c7039f73b0 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c703e0a5a0 con 0x55c7039ff590 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa2e7da740 10 client.0 _ll_register_callbacks cb 0x7ffa10007d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 0x55c703e06040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 0x55c703e06040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39860/0 (socket says 172.21.15.46:39860) 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 -- 172.21.15.46:0/998200090 learned_addr learned my addr 172.21.15.46:0/998200090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa256c3640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e8a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa24ec2640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039ff590 0x55c703e05b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 -- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039ff590 msgr2=0x55c703e05b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039ff590 0x55c703e05b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 -- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 msgr2=0x55c703e8a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e8a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.045+0000 7ffa25ec4640 1 -- 172.21.15.46:0/998200090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c703b026b0 con 0x55c703a20270 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa25ec4640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 0x55c703e06040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa18001730 tx=0x7ffa180015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c612cd7ee0ebc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ff9fffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ff9fffff640 1 -- 172.21.15.46:0/998200090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18005810 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ff9fffff640 1 -- 172.21.15.46:0/998200090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7ffa18015680 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa10088930 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ff9fffff640 1 -- 172.21.15.46:0/998200090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7ffa18032050 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 10 client.4968 fetch_fsmap learned FSMap version 15 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55c703ce4630 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ff9fffff640 1 -- 172.21.15.46:0/998200090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10088930 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ff9fffff640 1 -- 172.21.15.46:0/998200090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7ffa18035020 con 0x55c703a20270 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 10 client.4968 fetch_fsmap finished waiting for FSMap version 15 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 10 client.4968 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 20 client.4968 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c703deafc0 0x55c703e8aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 4 client.4968 mds_command: new command op to 4809 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 --> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] -- command(tid 0: {"prefix": "status"}) -- 0x55c703d2ff60 con 0x55c703deafc0 2024-05-09T03:10:20.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.046+0000 7ffa256c3640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c703deafc0 0x55c703e8aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ff9fffff640 10 client.4968 ms_handle_connect on v2:172.21.15.46:6832/3490266983 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa256c3640 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c703deafc0 0x55c703e8aea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f7fdf174a2a89b server_cookie=5659167f0c44dce1 in_seq=0 out_seq=0 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ff9fffff640 1 -- 172.21.15.46:0/998200090 <== mds.0 v2:172.21.15.46:6832/3490266983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55c703d2ff60 con 0x55c703deafc0 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ff9fffff640 10 client.4968 handle_command_reply: tid=0 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 client.4968 shutdown 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 msgr2=0x55c703e06040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 0x55c703e06040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ffa18001730 tx=0x7ffa180015f0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 shutdown_connections 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:6832/3490266983,v1:172.21.15.46:6834/3490266983] conn(0x55c703deafc0 0x55c703e8aea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c703a20270 0x55c703e06040 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c7039ff590 0x55c703e05b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ff9fffff640 0 client.4968 ms_handle_reset on v2:172.21.15.46:6832/3490266983 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 --2- 172.21.15.46:0/998200090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c7039f73b0 0x55c703e8a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 >> 172.21.15.46:0/998200090 conn(0x55c703a143c0 msgr2=0x55c703dc6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 shutdown_connections 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.047+0000 7ffa2e7da740 1 -- 172.21.15.46:0/998200090 wait complete. 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa2e7da740 20 client.4968 trim_cache size 0 max 16384 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4809, 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:10:20.054 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:10:20.055 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 5.8767193180000001, 2024-05-09T03:10:20.055 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 15, 2024-05-09T03:10:20.055 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 31, 2024-05-09T03:10:20.055 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 31, 2024-05-09T03:10:20.055 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 13.177230543 2024-05-09T03:10:20.055 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa04076300 msgr2=0x7ffa04078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa04076300 0x7ffa04078810 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffa2006f130 tx=0x7ffa1c009190 comp rx=0 tx=0).stop 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 msgr2=0x7ffa20153d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 0x7ffa20153d30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa1000c300 tx=0x7ffa200d4c30 comp rx=0 tx=0).stop 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 shutdown_connections 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffa200cade0 0x7ffa200cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa04076300 0x7ffa04078810 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa200ceff0 0x7ffa20153d30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 --2- 172.21.15.46:0/2920579019 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ffa0c004800 0x7ffa201531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 >> 172.21.15.46:0/2920579019 conn(0x7ffa2008d390 msgr2=0x7ffa2015e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 shutdown_connections 2024-05-09T03:10:20.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.048+0000 7ffa27927640 1 -- 172.21.15.46:0/2920579019 wait complete. 2024-05-09T03:10:20.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:10:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.174+0000 7f7b35d84640 1 Processor -- start 2024-05-09T03:10:20.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.174+0000 7f7b35d84640 1 -- start start 2024-05-09T03:10:20.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.175+0000 7f7b35d84640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b301615f0 0x7f7b300b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.175+0000 7f7b35d84640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d0b80 0x7f7b300d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b35d84640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b300d25e0 0x7f7b300d29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b35d84640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7b300d15b0 con 0x7f7b300d25e0 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b35d84640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7b30160b10 con 0x7f7b301615f0 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b35d84640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7b30160df0 con 0x7f7b300d0b80 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b300d25e0 0x7f7b300d29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b300d25e0 0x7f7b300d29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39874/0 (socket says 172.21.15.46:39874) 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 -- 172.21.15.46:0/3159288997 learned_addr learned my addr 172.21.15.46:0/3159288997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2f7fe640 1 --2- 172.21.15.46:0/3159288997 >> v2:172.21.15.88:3300/0 conn(0x7f7b301615f0 0x7f7b300b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2effd640 1 --2- 172.21.15.46:0/3159288997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d0b80 0x7f7b300d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 -- 172.21.15.46:0/3159288997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d0b80 msgr2=0x7f7b300d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 --2- 172.21.15.46:0/3159288997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d0b80 0x7f7b300d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 -- 172.21.15.46:0/3159288997 >> v2:172.21.15.88:3300/0 conn(0x7f7b301615f0 msgr2=0x7f7b300b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 --2- 172.21.15.46:0/3159288997 >> v2:172.21.15.88:3300/0 conn(0x7f7b301615f0 0x7f7b300b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.176+0000 7f7b2ffff640 1 -- 172.21.15.46:0/3159288997 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b301610d0 con 0x7f7b300d25e0 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.177+0000 7f7b2ffff640 1 --2- 172.21.15.46:0/3159288997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d25e0 0x7f7b300d29f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7b3008e2a0 tx=0x7f7b2802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca1795767c74600 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.177+0000 7f7b2e7fc640 1 -- 172.21.15.46:0/3159288997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b28036070 con 0x7f7b300d25e0 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.177+0000 7f7b2e7fc640 1 -- 172.21.15.46:0/3159288997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b28031600 con 0x7f7b300d25e0 2024-05-09T03:10:20.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.177+0000 7f7b2e7fc640 1 -- 172.21.15.46:0/3159288997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b280319d0 con 0x7f7b300d25e0 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 -- 172.21.15.46:0/3159288997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d25e0 msgr2=0x7f7b300d29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 --2- 172.21.15.46:0/3159288997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d25e0 0x7f7b300d29f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7b3008e2a0 tx=0x7f7b2802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 -- 172.21.15.46:0/3159288997 shutdown_connections 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 --2- 172.21.15.46:0/3159288997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d25e0 0x7f7b300d29f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 --2- 172.21.15.46:0/3159288997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d0b80 0x7f7b300d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 --2- 172.21.15.46:0/3159288997 >> v2:172.21.15.88:3300/0 conn(0x7f7b301615f0 0x7f7b300b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 -- 172.21.15.46:0/3159288997 >> 172.21.15.46:0/3159288997 conn(0x7f7b3006e190 msgr2=0x7f7b3006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 -- 172.21.15.46:0/3159288997 shutdown_connections 2024-05-09T03:10:20.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.178+0000 7f7b35d84640 1 -- 172.21.15.46:0/3159288997 wait complete. 2024-05-09T03:10:20.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.180+0000 7f7b35d84640 1 Processor -- start 2024-05-09T03:10:20.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.180+0000 7f7b35d84640 1 -- start start 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b35d84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 0x7f7b301f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b35d84640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d25e0 0x7f7b3015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b35d84640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b301615f0 0x7f7b30162d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b35d84640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7b300d48e0 con 0x7f7b300d0b80 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b35d84640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7b300d2f30 con 0x7f7b301615f0 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b35d84640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7b300d4a60 con 0x7f7b300d25e0 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 0x7f7b301f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 0x7f7b301f64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39890/0 (socket says 172.21.15.46:39890) 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b2f7fe640 1 -- 172.21.15.46:0/2616907254 learned_addr learned my addr 172.21.15.46:0/2616907254 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b2effd640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d25e0 0x7f7b3015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.181+0000 7f7b2ffff640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b301615f0 0x7f7b30162d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2f7fe640 1 -- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d25e0 msgr2=0x7f7b3015ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2f7fe640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d25e0 0x7f7b3015ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2f7fe640 1 -- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b301615f0 msgr2=0x7f7b30162d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2f7fe640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b301615f0 0x7f7b30162d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2f7fe640 1 -- 172.21.15.46:0/2616907254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b301f6a10 con 0x7f7b300d0b80 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2ffff640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b301615f0 0x7f7b30162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2effd640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d25e0 0x7f7b3015ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b2f7fe640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 0x7f7b301f64d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000e870 tx=0x7f7b2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20017070 con 0x7f7b300d0b80 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b2000c940 con 0x7f7b300d0b80 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3015b970 con 0x7f7b300d0b80 2024-05-09T03:10:20.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2000cd30 con 0x7f7b300d0b80 2024-05-09T03:10:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.182+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3015bf30 con 0x7f7b300d0b80 2024-05-09T03:10:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.183+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3006a270 con 0x7f7b300d0b80 2024-05-09T03:10:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.184+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7b2001a070 con 0x7f7b300d0b80 2024-05-09T03:10:20.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.184+0000 7f7b17fff640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b1c076300 0x7f7b1c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.184+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f7b200974f0 con 0x7f7b300d0b80 2024-05-09T03:10:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.185+0000 7f7b2effd640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b1c076300 0x7f7b1c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:20.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.185+0000 7f7b2effd640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b1c076300 0x7f7b1c078810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b300cf380 tx=0x7f7b24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:20.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.189+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7b301f6a10 con 0x7f7b300d0b80 2024-05-09T03:10:20.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.420+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7b3015c380 con 0x7f7b300d0b80 2024-05-09T03:10:20.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.420+0000 7f7b17fff640 1 -- 172.21.15.46:0/2616907254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f7b20060dc0 con 0x7f7b300d0b80 2024-05-09T03:10:20.421 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:10:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b1c076300 msgr2=0x7f7b1c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b1c076300 0x7f7b1c078810 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7b300cf380 tx=0x7f7b24008040 comp rx=0 tx=0).stop 2024-05-09T03:10:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 msgr2=0x7f7b301f64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 0x7f7b301f64d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000e870 tx=0x7f7b2000ed30 comp rx=0 tx=0).stop 2024-05-09T03:10:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 shutdown_connections 2024-05-09T03:10:20.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b301615f0 0x7f7b30162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b300d25e0 0x7f7b3015ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b1c076300 0x7f7b1c078810 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 --2- 172.21.15.46:0/2616907254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b300d0b80 0x7f7b301f64d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 >> 172.21.15.46:0/2616907254 conn(0x7f7b3006e190 msgr2=0x7f7b300d1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.422+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 shutdown_connections 2024-05-09T03:10:20.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20.423+0000 7f7b35d84640 1 -- 172.21.15.46:0/2616907254 wait complete. 2024-05-09T03:10:20.434 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:10:20.435 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:10:20.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:20.435 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:10:20.452 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:10:20.452 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:10:20.452 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:10:20.452 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.452 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:10:20.452 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42497sec preferred_lft 42497sec 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:10:20.453 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:10:20.454 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:20.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:20.455 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:10:20.517 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:10:20.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:20.519 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:10:20.576 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:10:20.576 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:10:20.577 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:10:20.577 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:10:20.577 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:10:20.577 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:10:20.577 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:10:20.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:10:20.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:10:20.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:10:20.577 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:10:20.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:10:20.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:10:20.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:20.707 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:10:20.723 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:10:20.723 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:10:20.724 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:10:20.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:20.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:10:20.790 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:10:20.790 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:10:20.790 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:20.845 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:10:20.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:10:20.846 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:20.901 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:10:20.967 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:10:21.024 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:10:21.024 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:10:21.025 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:10:21.026 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:10:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:10:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:21.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.027 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:10:21.080 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:10:21.080 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:10:21.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:10:21.148 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:10:21.207 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:10:21.208 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:21.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.209 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:10:21.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:10:21.218+0000 7f76c09e7480 -1 init, newargv = 0x55ef9d837580 newargc=15 2024-05-09T03:10:21.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[46601]: starting ceph client 2024-05-09T03:10:21.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[46601]: starting fuse 2024-05-09T03:10:21.236 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:10:21.236 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:10:21.237 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:10:21.238 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:10:21.358 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.46601.asok 2024-05-09T03:10:21.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:10:21.368 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46601.asok 2024-05-09T03:10:21.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46601.asok status 2024-05-09T03:10:21.517 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:21.517 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:10:21.517 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:10:21.517 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "46601", 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4980, 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:10:21.518 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4980 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1572323204 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1572323204 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:21.519 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4980 192.168.0.1:0/1572323204", 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1572323204", 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 15, 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 31, 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:10:21.520 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:21.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.525 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:10:21.526 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:21.542 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:10:21.542 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:21.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:21.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.610 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-05-09T03:10:21.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.672 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-05-09T03:10:21.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:21.732 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-05-09T03:10:21.790 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:10:21.791 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:10:21.791 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:10:21.791 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:21.791 DEBUG:teuthology.orchestra.run.smithi046:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-05-09T03:10:21.791 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:10:21.899 INFO:teuthology.orchestra.run.smithi046.stdout:1099511627776 2024-05-09T03:10:21.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:10:21.908 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:10:21.908 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:10:21.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:10:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.120+0000 7f2cb2cc8640 1 Processor -- start 2024-05-09T03:10:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.120+0000 7f2cb2cc8640 1 -- start start 2024-05-09T03:10:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb2cc8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2cac0b6660 0x7f2cac0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb2cc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2cac0d1f00 0x7f2cac144d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb2cc8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cac0b55c0 con 0x7f2cac147370 2024-05-09T03:10:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb2cc8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2cac0b58a0 con 0x7f2cac0d1f00 2024-05-09T03:10:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb2cc8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2cac0b5b80 con 0x7f2cac0b6660 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb1265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2cac0d1f00 0x7f2cac144d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb1265640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2cac0d1f00 0x7f2cac144d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39900/0 (socket says 172.21.15.46:39900) 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb1265640 1 -- 172.21.15.46:0/2015695845 learned_addr learned my addr 172.21.15.46:0/2015695845 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.121+0000 7f2cb0a64640 1 --2- 172.21.15.46:0/2015695845 >> v2:172.21.15.88:3300/0 conn(0x7f2cac0b6660 0x7f2cac0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.122+0000 7f2cb1265640 1 -- 172.21.15.46:0/2015695845 >> v2:172.21.15.88:3300/0 conn(0x7f2cac0b6660 msgr2=0x7f2cac0b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.122+0000 7f2cb1265640 1 --2- 172.21.15.46:0/2015695845 >> v2:172.21.15.88:3300/0 conn(0x7f2cac0b6660 0x7f2cac0b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.122+0000 7f2cb1265640 1 -- 172.21.15.46:0/2015695845 >> v1:172.21.15.88:6789/0 conn(0x7f2cac147370 legacy=0x7f2cac0d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.122+0000 7f2cb1265640 1 -- 172.21.15.46:0/2015695845 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cac0b5e60 con 0x7f2cac0d1f00 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.122+0000 7f2cb1265640 1 --2- 172.21.15.46:0/2015695845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac0d1f00 0x7f2cac144d50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0727c0 tx=0x7f2ca402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3e1baa701f417 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cab7fe640 1 -- 172.21.15.46:0/2015695845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4037070 con 0x7f2cac0d1f00 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cab7fe640 1 -- 172.21.15.46:0/2015695845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca4032440 con 0x7f2cac0d1f00 2024-05-09T03:10:22.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cab7fe640 1 -- 172.21.15.46:0/2015695845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4032810 con 0x7f2cac0d1f00 2024-05-09T03:10:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/2015695845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac0d1f00 msgr2=0x7f2cac144d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/2015695845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac0d1f00 0x7f2cac144d50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0727c0 tx=0x7f2ca402a270 comp rx=0 tx=0).stop 2024-05-09T03:10:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/2015695845 shutdown_connections 2024-05-09T03:10:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/2015695845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac0d1f00 0x7f2cac144d50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/2015695845 >> v2:172.21.15.88:3300/0 conn(0x7f2cac0b6660 0x7f2cac0b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/2015695845 >> 172.21.15.46:0/2015695845 conn(0x7f2cac0b3620 msgr2=0x7f2cac0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/2015695845 shutdown_connections 2024-05-09T03:10:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.123+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/2015695845 wait complete. 2024-05-09T03:10:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.125+0000 7f2cb2cc8640 1 Processor -- start 2024-05-09T03:10:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.125+0000 7f2cb2cc8640 1 -- start start 2024-05-09T03:10:22.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb2cc8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2cac0b6660 0x7f2cac1d9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb2cc8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cac0d1f00 0x7f2cac143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb2cc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 0x7f2cac1e3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb2cc8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac145290 con 0x7f2cac147370 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb2cc8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac1447e0 con 0x7f2cac0d1f00 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb2cc8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2cac145410 con 0x7f2cac0b6660 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 0x7f2cac1e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 0x7f2cac1e3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39904/0 (socket says 172.21.15.46:39904) 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 -- 172.21.15.46:0/1878581813 learned_addr learned my addr 172.21.15.46:0/1878581813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cabfff640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cac0d1f00 0x7f2cac143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb0a64640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2cac0b6660 0x7f2cac1d9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 -- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2cac0b6660 msgr2=0x7f2cac1d9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2cac0b6660 0x7f2cac1d9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 -- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cac0d1f00 msgr2=0x7f2cac143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cac0d1f00 0x7f2cac143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 -- 172.21.15.46:0/1878581813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cac1f8270 con 0x7f2cac147370 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.127+0000 7f2cb1265640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 0x7f2cac1e3230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2ca402a2b0 tx=0x7f2ca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.128+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4041020 con 0x7f2cac147370 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.128+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca403d640 con 0x7f2cac147370 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.128+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cac1f8550 con 0x7f2cac147370 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.128+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4037040 con 0x7f2cac147370 2024-05-09T03:10:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.128+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cac1f8b10 con 0x7f2cac147370 2024-05-09T03:10:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.129+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cac06a270 con 0x7f2cac147370 2024-05-09T03:10:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.129+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2ca4044020 con 0x7f2cac147370 2024-05-09T03:10:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.130+0000 7f2ca97fa640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c8c0762d0 0x7f2c8c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.130+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f2ca40b7d70 con 0x7f2cac147370 2024-05-09T03:10:22.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.130+0000 7f2cb0a64640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c8c0762d0 0x7f2c8c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.131+0000 7f2cb0a64640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c8c0762d0 0x7f2c8c0787e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009a30 tx=0x7f2c9c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.135+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2ca404b030 con 0x7f2cac147370 2024-05-09T03:10:22.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.346+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2cac0715f0 con 0x7f2cac147370 2024-05-09T03:10:22.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.346+0000 7f2ca97fa640 1 -- 172.21.15.46:0/1878581813 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v31) ==== 80+0+0 (secure 0 0 0) 0x7f2ca4081650 con 0x7f2cac147370 2024-05-09T03:10:22.347 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c8c0762d0 msgr2=0x7f2c8c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c8c0762d0 0x7f2c8c0787e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009a30 tx=0x7f2c9c008040 comp rx=0 tx=0).stop 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 msgr2=0x7f2cac1e3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 0x7f2cac1e3230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2ca402a2b0 tx=0x7f2ca40046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 shutdown_connections 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c8c0762d0 0x7f2c8c0787e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cac147370 0x7f2cac1e3230 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2cac0d1f00 0x7f2cac143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 --2- 172.21.15.46:0/1878581813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2cac0b6660 0x7f2cac1d9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 >> 172.21.15.46:0/1878581813 conn(0x7f2cac0b3620 msgr2=0x7f2cac0b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 shutdown_connections 2024-05-09T03:10:22.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.349+0000 7f2cb2cc8640 1 -- 172.21.15.46:0/1878581813 wait complete. 2024-05-09T03:10:22.361 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:10:22.361 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:10:22.378 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:10:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.532+0000 7fbd8b300640 1 Processor -- start 2024-05-09T03:10:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.533+0000 7fbd8b300640 1 -- start start 2024-05-09T03:10:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8b300640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd840b5140 0x7fbd840b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8b300640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd840d3fc0 0x7fbd840b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8b300640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd840b80b0 con 0x7fbd840b7ca0 2024-05-09T03:10:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8b300640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbd840d3480 con 0x7fbd840d3fc0 2024-05-09T03:10:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8b300640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbd840b5550 con 0x7fbd840b5140 2024-05-09T03:10:22.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8989d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd840d3fc0 0x7fbd840b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8889b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbd840b7ca0 0x7fbd840d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35786/0 (socket says 172.21.15.46:35786) 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8889b640 1 -- 172.21.15.46:0/893942365 learned_addr learned my addr 172.21.15.46:0/893942365 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.534+0000 7fbd8909c640 1 --2- 172.21.15.46:0/893942365 >> v2:172.21.15.88:3300/0 conn(0x7fbd840b5140 0x7fbd840b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd8989d640 1 -- 172.21.15.46:0/893942365 >> v2:172.21.15.88:3300/0 conn(0x7fbd840b5140 msgr2=0x7fbd840b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd8989d640 1 --2- 172.21.15.46:0/893942365 >> v2:172.21.15.88:3300/0 conn(0x7fbd840b5140 0x7fbd840b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd8989d640 1 -- 172.21.15.46:0/893942365 >> v1:172.21.15.46:6789/0 conn(0x7fbd840b7ca0 legacy=0x7fbd840d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd8989d640 1 -- 172.21.15.46:0/893942365 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd840d44e0 con 0x7fbd840d3fc0 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd83fff640 1 -- 172.21.15.46:0/893942365 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663272368 0 0) 0x7fbd78001110 con 0x7fbd840b7ca0 2024-05-09T03:10:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd8909c640 1 --2- 172.21.15.46:0/893942365 >> v2:172.21.15.88:3300/0 conn(0x7fbd840b5140 0x7fbd840b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd8989d640 1 --2- 172.21.15.46:0/893942365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840d3fc0 0x7fbd840b9380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd840b88d0 tx=0x7fbd7c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a681595454987f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd83fff640 1 -- 172.21.15.46:0/893942365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c037070 con 0x7fbd840d3fc0 2024-05-09T03:10:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd83fff640 1 -- 172.21.15.46:0/893942365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd7c032440 con 0x7fbd840d3fc0 2024-05-09T03:10:22.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.535+0000 7fbd83fff640 1 -- 172.21.15.46:0/893942365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c032810 con 0x7fbd840d3fc0 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 -- 172.21.15.46:0/893942365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840d3fc0 msgr2=0x7fbd840b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 --2- 172.21.15.46:0/893942365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840d3fc0 0x7fbd840b9380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd840b88d0 tx=0x7fbd7c02a230 comp rx=0 tx=0).stop 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 -- 172.21.15.46:0/893942365 shutdown_connections 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 --2- 172.21.15.46:0/893942365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840d3fc0 0x7fbd840b9380 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 --2- 172.21.15.46:0/893942365 >> v2:172.21.15.88:3300/0 conn(0x7fbd840b5140 0x7fbd840b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 -- 172.21.15.46:0/893942365 >> 172.21.15.46:0/893942365 conn(0x7fbd8408d3f0 msgr2=0x7fbd8408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 -- 172.21.15.46:0/893942365 shutdown_connections 2024-05-09T03:10:22.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.536+0000 7fbd8b300640 1 -- 172.21.15.46:0/893942365 wait complete. 2024-05-09T03:10:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.538+0000 7fbd8b300640 1 Processor -- start 2024-05-09T03:10:22.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.538+0000 7fbd8b300640 1 -- start start 2024-05-09T03:10:22.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8b300640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 0x7fbd840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8b300640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd840b7ca0 0x7fbd841512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8b300640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd840d3fc0 0x7fbd84151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8b300640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbd840d4d80 con 0x7fbd840b5140 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8b300640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbd840d4aa0 con 0x7fbd840d3fc0 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8b300640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbd840d5580 con 0x7fbd840b7ca0 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8909c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 0x7fbd840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8909c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 0x7fbd840cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39924/0 (socket says 172.21.15.46:39924) 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.539+0000 7fbd8909c640 1 -- 172.21.15.46:0/420870432 learned_addr learned my addr 172.21.15.46:0/420870432 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8889b640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd840b7ca0 0x7fbd841512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8989d640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd840d3fc0 0x7fbd84151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8909c640 1 -- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd840b7ca0 msgr2=0x7fbd841512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8909c640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd840b7ca0 0x7fbd841512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8909c640 1 -- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd840d3fc0 msgr2=0x7fbd84151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8909c640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd840d3fc0 0x7fbd84151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8909c640 1 -- 172.21.15.46:0/420870432 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8415ed20 con 0x7fbd840b5140 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8989d640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd840d3fc0 0x7fbd84151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8909c640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 0x7fbd840cf2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbd74007f20 tx=0x7fbd7400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7401a070 con 0x7fbd840b5140 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd74003b90 con 0x7fbd840b5140 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd840cdd80 con 0x7fbd840b5140 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.540+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74004030 con 0x7fbd840b5140 2024-05-09T03:10:22.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.541+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd840ad450 con 0x7fbd840b5140 2024-05-09T03:10:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.541+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd8406a270 con 0x7fbd840b5140 2024-05-09T03:10:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.542+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbd7401d070 con 0x7fbd840b5140 2024-05-09T03:10:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.543+0000 7fbd81ffb640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd700762d0 0x7fbd700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.543+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7fbd74097670 con 0x7fbd840b5140 2024-05-09T03:10:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.543+0000 7fbd8889b640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd700762d0 0x7fbd700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:22.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.544+0000 7fbd8889b640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd700762d0 0x7fbd700787e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd840b5cd0 tx=0x7fbd78007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:22.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.548+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbd74010070 con 0x7fbd840b5140 2024-05-09T03:10:22.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.751+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbd840cf830 con 0x7fbd840b5140 2024-05-09T03:10:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.751+0000 7fbd81ffb640 1 -- 172.21.15.46:0/420870432 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v31) ==== 80+0+0 (secure 0 0 0) 0x7fbd74060a10 con 0x7fbd840b5140 2024-05-09T03:10:22.752 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd700762d0 msgr2=0x7fbd700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd700762d0 0x7fbd700787e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbd840b5cd0 tx=0x7fbd78007c00 comp rx=0 tx=0).stop 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 msgr2=0x7fbd840cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 0x7fbd840cf2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbd74007f20 tx=0x7fbd7400d560 comp rx=0 tx=0).stop 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 shutdown_connections 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd840d3fc0 0x7fbd84151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbd840b7ca0 0x7fbd841512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd700762d0 0x7fbd700787e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 --2- 172.21.15.46:0/420870432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd840b5140 0x7fbd840cf2f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 >> 172.21.15.46:0/420870432 conn(0x7fbd8408d3f0 msgr2=0x7fbd840b7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 shutdown_connections 2024-05-09T03:10:22.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:22.754+0000 7fbd8b300640 1 -- 172.21.15.46:0/420870432 wait complete. 2024-05-09T03:10:22.765 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:10:22.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:22.766 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:22.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[46601]: fuse finished with error 0 and tester_r 0 2024-05-09T03:10:22.792 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:28.894 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:10:28.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:28.894 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:10:28.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:29.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.083+0000 7fb4530b8640 1 Processor -- start 2024-05-09T03:10:29.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.083+0000 7fb4530b8640 1 -- start start 2024-05-09T03:10:29.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.084+0000 7fb4530b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb44c1613c0 0x7fb44c0b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.084+0000 7fb4530b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 0x7fb44c163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.084+0000 7fb4530b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb44c0b8110 con 0x7fb44c0b5210 2024-05-09T03:10:29.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.084+0000 7fb4530b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb44c0b5620 con 0x7fb44c1613c0 2024-05-09T03:10:29.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.084+0000 7fb4530b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb44c1609a0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb451655640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 0x7fb44c163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb443fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb44c1613c0 0x7fb44c0b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb450e54640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb44c0b5210 0x7fb44c0d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34964/0 (socket says 172.21.15.46:34964) 2024-05-09T03:10:29.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb451655640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 0x7fb44c163e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:35098/0 (socket says 172.21.15.46:35098) 2024-05-09T03:10:29.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb450e54640 1 -- 172.21.15.46:0/4178271858 learned_addr learned my addr 172.21.15.46:0/4178271858 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb451655640 1 -- 172.21.15.46:0/4178271858 >> v2:172.21.15.88:3300/0 conn(0x7fb44c1613c0 msgr2=0x7fb44c0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb451655640 1 --2- 172.21.15.46:0/4178271858 >> v2:172.21.15.88:3300/0 conn(0x7fb44c1613c0 0x7fb44c0b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb451655640 1 -- 172.21.15.46:0/4178271858 >> v1:172.21.15.88:6789/0 conn(0x7fb44c0b5210 legacy=0x7fb44c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb451655640 1 -- 172.21.15.46:0/4178271858 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c1617d0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb4437fe640 1 -- 172.21.15.46:0/4178271858 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355848986 0 0) 0x7fb448001110 con 0x7fb44c0b5210 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.085+0000 7fb443fff640 1 --2- 172.21.15.46:0/4178271858 >> v2:172.21.15.88:3300/0 conn(0x7fb44c1613c0 0x7fb44c0b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.086+0000 7fb451655640 1 --2- 172.21.15.46:0/4178271858 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 0x7fb44c163e60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb444009d90 tx=0x7fb44402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e90b556029fa8eee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.086+0000 7fb4437fe640 1 -- 172.21.15.46:0/4178271858 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444007740 con 0x7fb44c0b7d00 2024-05-09T03:10:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.086+0000 7fb4437fe640 1 -- 172.21.15.46:0/4178271858 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4440078e0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.086+0000 7fb4437fe640 1 -- 172.21.15.46:0/4178271858 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444007cb0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 -- 172.21.15.46:0/4178271858 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 msgr2=0x7fb44c163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 --2- 172.21.15.46:0/4178271858 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 0x7fb44c163e60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb444009d90 tx=0x7fb44402a660 comp rx=0 tx=0).stop 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 -- 172.21.15.46:0/4178271858 shutdown_connections 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 --2- 172.21.15.46:0/4178271858 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c0b7d00 0x7fb44c163e60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 --2- 172.21.15.46:0/4178271858 >> v2:172.21.15.88:3300/0 conn(0x7fb44c1613c0 0x7fb44c0b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 -- 172.21.15.46:0/4178271858 >> 172.21.15.46:0/4178271858 conn(0x7fb44c0d29a0 msgr2=0x7fb44c0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 -- 172.21.15.46:0/4178271858 shutdown_connections 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.087+0000 7fb4530b8640 1 -- 172.21.15.46:0/4178271858 wait complete. 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.089+0000 7fb4530b8640 1 Processor -- start 2024-05-09T03:10:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.089+0000 7fb4530b8640 1 -- start start 2024-05-09T03:10:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.090+0000 7fb4530b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb44c0b5210 0x7fb44c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.090+0000 7fb4530b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 0x7fb44c0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.090+0000 7fb4530b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c1613c0 0x7fb44c0cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.090+0000 7fb4530b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c160bc0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.090+0000 7fb4530b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb44c0d5560 con 0x7fb44c0b5210 2024-05-09T03:10:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.090+0000 7fb4530b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb44c160ea0 con 0x7fb44c1613c0 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 0x7fb44c0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 0x7fb44c0cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53062/0 (socket says 172.21.15.46:53062) 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 -- 172.21.15.46:0/843628141 learned_addr learned my addr 172.21.15.46:0/843628141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb451655640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c1613c0 0x7fb44c0cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb450e54640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb44c0b5210 0x7fb44c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 -- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c1613c0 msgr2=0x7fb44c0cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c1613c0 0x7fb44c0cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 -- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb44c0b5210 msgr2=0x7fb44c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb44c0b5210 0x7fb44c0ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 -- 172.21.15.46:0/843628141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb44c0cbc80 con 0x7fb44c0b7d00 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb443fff640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 0x7fb44c0cae90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00e960 tx=0x7fb43c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43c017070 con 0x7fb44c0b7d00 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb43c00c9a0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb44c0cbf60 con 0x7fb44c0b7d00 2024-05-09T03:10:29.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb43c00cd70 con 0x7fb44c0b7d00 2024-05-09T03:10:29.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.091+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb44c0cc520 con 0x7fb44c0b7d00 2024-05-09T03:10:29.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.092+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb44c06a270 con 0x7fb44c0b7d00 2024-05-09T03:10:29.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.093+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb43c01a070 con 0x7fb44c0b7d00 2024-05-09T03:10:29.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.094+0000 7fb4417fa640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb438076300 0x7fb438078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.094+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7fb43c097800 con 0x7fb44c0b7d00 2024-05-09T03:10:29.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.094+0000 7fb450e54640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb438076300 0x7fb438078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.095+0000 7fb450e54640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb438076300 0x7fb438078810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb448007fd0 tx=0x7fb448007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.098+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb43c0609e0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.329+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb44c0d40d0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.329+0000 7fb4417fa640 1 -- 172.21.15.46:0/843628141 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7fb43c064ce0 con 0x7fb44c0b7d00 2024-05-09T03:10:29.330 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:29.330 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:16.576454+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4809":{"gid":4809,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/3490266983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3490266983},{"type":"v1","addr":"172.21.15.46:6834","nonce":3490266983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4809,"qdb_cluster":[4809]},"id":2}]} 2024-05-09T03:10:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 15 2024-05-09T03:10:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb438076300 msgr2=0x7fb438078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb438076300 0x7fb438078810 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb448007fd0 tx=0x7fb448007de0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 msgr2=0x7fb44c0cae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 0x7fb44c0cae90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb43c00e960 tx=0x7fb43c00ee20 comp rx=0 tx=0).stop 2024-05-09T03:10:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 shutdown_connections 2024-05-09T03:10:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44c1613c0 0x7fb44c0cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb438076300 0x7fb438078810 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb44c0b7d00 0x7fb44c0cae90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 --2- 172.21.15.46:0/843628141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb44c0b5210 0x7fb44c0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 >> 172.21.15.46:0/843628141 conn(0x7fb44c0d29a0 msgr2=0x7fb44c15c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:29.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 shutdown_connections 2024-05-09T03:10:29.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.331+0000 7fb4530b8640 1 -- 172.21.15.46:0/843628141 wait complete. 2024-05-09T03:10:29.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-05-09T03:10:29.481 INFO:teuthology.orchestra.run.smithi046.stdout:{"return_code":0} 2024-05-09T03:10:29.482 DEBUG:tasks.cephfs.filesystem:_json_asok output { "return_code": 0 } 2024-05-09T03:10:29.482 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:10:29.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.597+0000 7ff77b0ee640 1 Processor -- start 2024-05-09T03:10:29.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.597+0000 7ff77b0ee640 1 -- start start 2024-05-09T03:10:29.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.598+0000 7ff77b0ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7740d2e90 0x7ff7740b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.598+0000 7ff77b0ee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7740d3400 con 0x7ff7741605b0 2024-05-09T03:10:29.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.598+0000 7ff77b0ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7740d4a60 con 0x7ff7740b4fc0 2024-05-09T03:10:29.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.598+0000 7ff77b0ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff7740d4d40 con 0x7ff7740d2e90 2024-05-09T03:10:29.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.598+0000 7ff773fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff7741605b0 0x7ff7740d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54782/0 (socket says 172.21.15.46:54782) 2024-05-09T03:10:29.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff773fff640 1 -- 172.21.15.46:0/1752558862 learned_addr learned my addr 172.21.15.46:0/1752558862 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:29.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4213532247 0 0) 0x7ff7740d3400 con 0x7ff7741605b0 2024-05-09T03:10:29.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff778e8a640 1 --2- 172.21.15.46:0/1752558862 >> v2:172.21.15.88:3300/0 conn(0x7ff7740d2e90 0x7ff7740b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff764003520 con 0x7ff7741605b0 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 336799677 0 0) 0x7ff764003520 con 0x7ff7741605b0 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 >> v2:172.21.15.88:3300/0 conn(0x7ff7740d2e90 msgr2=0x7ff7740b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 --2- 172.21.15.46:0/1752558862 >> v2:172.21.15.88:3300/0 conn(0x7ff7740d2e90 0x7ff7740b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 >> v1:172.21.15.88:6789/0 conn(0x7ff7740b4fc0 legacy=0x7ff7740d3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7740d5020 con 0x7ff7741605b0 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ff760001710 con 0x7ff7741605b0 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 >> v1:172.21.15.46:6789/0 conn(0x7ff7741605b0 legacy=0x7ff7740d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff764003a00 0x7ff764003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff764004800 0x7ff764004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764005270 0x7ff76400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740d4d40 con 0x7ff764003a00 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff76400c200 con 0x7ff764005270 2024-05-09T03:10:29.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff76400c400 con 0x7ff764004800 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff778e8a640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff764003a00 0x7ff764003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff773fff640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff764004800 0x7ff764004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.599+0000 7ff77968b640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764005270 0x7ff76400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff778e8a640 1 -- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff764004800 msgr2=0x7ff764004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff778e8a640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff764004800 0x7ff764004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff778e8a640 1 -- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764005270 msgr2=0x7ff76400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:29.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff778e8a640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764005270 0x7ff76400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff77968b640 1 -- 172.21.15.46:0/1752558862 reap_dead start 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff778e8a640 1 -- 172.21.15.46:0/1752558862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff764003520 con 0x7ff764003a00 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff773fff640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff764004800 0x7ff764004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.600+0000 7ff778e8a640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff764003a00 0x7ff764003e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff768009a90 tx=0x7ff764005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece09afce840093f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768036070 con 0x7ff764003a00 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff768031400 con 0x7ff764003a00 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff7737fe640 1 -- 172.21.15.46:0/1752558862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768009690 con 0x7ff764003a00 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 -- 172.21.15.46:0/1752558862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff764003a00 msgr2=0x7ff764003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff764003a00 0x7ff764003e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff768009a90 tx=0x7ff764005830 comp rx=0 tx=0).stop 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 -- 172.21.15.46:0/1752558862 shutdown_connections 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/1752558862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff764003a00 0x7ff764003e30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 -- 172.21.15.46:0/1752558862 >> 172.21.15.46:0/1752558862 conn(0x7ff77408d390 msgr2=0x7ff77408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 -- 172.21.15.46:0/1752558862 shutdown_connections 2024-05-09T03:10:29.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.601+0000 7ff77b0ee640 1 -- 172.21.15.46:0/1752558862 wait complete. 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.603+0000 7ff77b0ee640 1 Processor -- start 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.603+0000 7ff77b0ee640 1 -- start start 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77b0ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764003a00 0x7ff7740ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77b0ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 0x7ff7741531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77b0ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7740cb350 0x7ff7740cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77b0ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740b4d80 con 0x7ff7741f7cd0 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77b0ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740b4c00 con 0x7ff764003a00 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77b0ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff7740b7760 con 0x7ff7740cb350 2024-05-09T03:10:29.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 0x7ff7741531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 0x7ff7741531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53080/0 (socket says 172.21.15.46:53080) 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 -- 172.21.15.46:0/524889335 learned_addr learned my addr 172.21.15.46:0/524889335 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff778e8a640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764003a00 0x7ff7740ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77968b640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7740cb350 0x7ff7740cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 -- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7740cb350 msgr2=0x7ff7740cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7740cb350 0x7ff7740cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 -- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764003a00 msgr2=0x7ff7740ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764003a00 0x7ff7740ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 -- 172.21.15.46:0/524889335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7740cbca0 con 0x7ff7741f7cd0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff77968b640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7740cb350 0x7ff7740cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.605+0000 7ff773fff640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 0x7ff7741531f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff760001eb0 tx=0x7ff76000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.606+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76000c660 con 0x7ff7741f7cd0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.606+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff760018070 con 0x7ff7741f7cd0 2024-05-09T03:10:29.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.606+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7740ccde0 con 0x7ff7741f7cd0 2024-05-09T03:10:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.606+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76000ca20 con 0x7ff7741f7cd0 2024-05-09T03:10:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.606+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7740cd3a0 con 0x7ff7741f7cd0 2024-05-09T03:10:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.606+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff77406a270 con 0x7ff7741f7cd0 2024-05-09T03:10:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.607+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff76000cbc0 con 0x7ff7741f7cd0 2024-05-09T03:10:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.608+0000 7ff7717fa640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7580762d0 0x7ff7580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.608+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7ff760097180 con 0x7ff7741f7cd0 2024-05-09T03:10:29.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.608+0000 7ff778e8a640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7580762d0 0x7ff7580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.609+0000 7ff778e8a640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7580762d0 0x7ff7580787e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff768033000 tx=0x7ff768008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.613+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff7740cbca0 con 0x7ff7741f7cd0 2024-05-09T03:10:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.844+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff7740ca7c0 con 0x7ff7741f7cd0 2024-05-09T03:10:29.845 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:10:29.845+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:10:29.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.847+0000 7ff7717fa640 1 -- 172.21.15.46:0/524889335 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) ==== 161+0+0 (secure 0 0 0) 0x7ff760060b70 con 0x7ff7741f7cd0 2024-05-09T03:10:29.847 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7580762d0 msgr2=0x7ff7580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7580762d0 0x7ff7580787e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff768033000 tx=0x7ff768008030 comp rx=0 tx=0).stop 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 msgr2=0x7ff7741531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 0x7ff7741531f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff760001eb0 tx=0x7ff76000ec10 comp rx=0 tx=0).stop 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 shutdown_connections 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7740cb350 0x7ff7740cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7580762d0 0x7ff7580787e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7741f7cd0 0x7ff7741531f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 --2- 172.21.15.46:0/524889335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff764003a00 0x7ff7740ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 >> 172.21.15.46:0/524889335 conn(0x7ff77408d390 msgr2=0x7ff77415f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 shutdown_connections 2024-05-09T03:10:29.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.850+0000 7ff77b0ee640 1 -- 172.21.15.46:0/524889335 wait complete. 2024-05-09T03:10:29.852 INFO:tasks.ceph.mds.a.smithi046.stderr: -3382> 2024-05-09T03:10:18.674+0000 7f1f69357640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:10:29.864 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:10:29.911 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:10:29.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.923+0000 7f041a91df40 1 Processor -- start 2024-05-09T03:10:29.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.923+0000 7f041a91df40 1 -- start start 2024-05-09T03:10:29.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041a91df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788dc640 0x55aa78987050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041a91df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788de930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041a91df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041a91df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55aa78857f50 con 0x55aa7897b520 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041a91df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aa7885bf90 con 0x55aa788dc640 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041a91df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55aa788fa8e0 con 0x55aa788df540 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788de930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53130/0 (socket says 172.21.15.46:53130) 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 -- 172.21.15.46:0/1286461943 learned_addr learned my addr 172.21.15.46:0/1286461943 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f04139a0640 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788dc640 0x55aa78987050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f04141a1640 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 -- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788df540 msgr2=0x55aa788df950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 -- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788dc640 msgr2=0x55aa78987050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788dc640 0x55aa78987050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 -- 172.21.15.46:0/1286461943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa788f9ab0 con 0x55aa7897b520 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f04139a0640 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788dc640 0x55aa78987050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:29.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.924+0000 7f041319f640 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788de930 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f04080017e0 tx=0x7f04080099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a47d00a783937cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041219d640 1 -- 172.21.15.46:0/1286461943 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040800e7e0 con 0x55aa7897b520 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041219d640 1 -- 172.21.15.46:0/1286461943 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0408013030 con 0x55aa7897b520 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041219d640 1 -- 172.21.15.46:0/1286461943 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040800ebc0 con 0x55aa7897b520 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 -- 172.21.15.46:0/1286461943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 msgr2=0x55aa788de930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788de930 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f04080017e0 tx=0x7f04080099b0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 -- 172.21.15.46:0/1286461943 shutdown_connections 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788de930 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 --2- 172.21.15.46:0/1286461943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788dc640 0x55aa78987050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 -- 172.21.15.46:0/1286461943 >> 172.21.15.46:0/1286461943 conn(0x55aa787d3ac0 msgr2=0x55aa788e07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 -- 172.21.15.46:0/1286461943 shutdown_connections 2024-05-09T03:10:29.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.925+0000 7f041a91df40 1 -- 172.21.15.46:0/1286461943 wait complete. 2024-05-09T03:10:29.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.926+0000 7f041a91df40 1 Processor -- start 2024-05-09T03:10:29.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.926+0000 7f041a91df40 1 -- start start 2024-05-09T03:10:29.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.927+0000 7f041a91df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788dc640 0x55aa789739e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.927+0000 7f041a91df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.927+0000 7f041a91df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788dfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.927+0000 7f041a91df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55aa78983f80 con 0x55aa7897b520 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.927+0000 7f041a91df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aa78983d30 con 0x55aa788df540 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.927+0000 7f041a91df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55aa788b32e0 con 0x55aa788dc640 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788dfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788dfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53138/0 (socket says 172.21.15.46:53138) 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 -- 172.21.15.46:0/807690617 learned_addr learned my addr 172.21.15.46:0/807690617 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 -- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788dc640 msgr2=0x55aa789739e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa788dc640 0x55aa789739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041319f640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 -- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788df540 msgr2=0x55aa788df950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 -- 172.21.15.46:0/807690617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa78985640 con 0x55aa7897b520 2024-05-09T03:10:29.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041319f640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa788df540 0x55aa788df950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f04141a1640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788dfe90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f040c056040 tx=0x7f040c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041119b640 1 -- 172.21.15.46:0/807690617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c058040 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041119b640 1 -- 172.21.15.46:0/807690617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f040c065070 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aa78985920 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041119b640 1 -- 172.21.15.46:0/807690617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f040c0066e0 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.928+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa78978ec0 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.930+0000 7f041119b640 1 -- 172.21.15.46:0/807690617 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f040c053a30 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.930+0000 7f041119b640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0400074cb0 0x7f04000771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f041119b640 1 -- 172.21.15.46:0/807690617 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f040c0d9d30 con 0x55aa7897b520 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f04139a0640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0400074cb0 0x7f04000771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f041a91df40 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55aa78979bd0 0x55aa78979fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) -- 0x55aa78a27250 con 0x55aa78979bd0 2024-05-09T03:10:29.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f041319f640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55aa78979bd0 0x55aa78979fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:29.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f041319f640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55aa78979bd0 0x55aa78979fe0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.931+0000 7f04139a0640 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0400074cb0 0x7f04000771c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55aa788f55a0 tx=0x7f0404005a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:29.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041319f640 1 -- 172.21.15.46:0/807690617 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v32'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f040800e7c0 con 0x55aa78979bd0 2024-05-09T03:10:29.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55aa78979bd0 msgr2=0x55aa78979fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55aa78979bd0 0x55aa78979fe0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0400074cb0 msgr2=0x7f04000771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0400074cb0 0x7f04000771c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55aa788f55a0 tx=0x7f0404005a00 comp rx=0 tx=0).stop 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 msgr2=0x55aa788dfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 --2- 172.21.15.46:0/807690617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa7897b520 0x55aa788dfe90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f040c056040 tx=0x7f040c0014a0 comp rx=0 tx=0).stop 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f04141a1640 1 -- 172.21.15.46:0/807690617 reap_dead start 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 shutdown_connections 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 >> 172.21.15.46:0/807690617 conn(0x55aa787d3ac0 msgr2=0x55aa788dbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 shutdown_connections 2024-05-09T03:10:29.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:29.933+0000 7f041a91df40 1 -- 172.21.15.46:0/807690617 wait complete. 2024-05-09T03:10:29.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:10:30.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.054+0000 7f191a9e0640 1 Processor -- start 2024-05-09T03:10:30.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.054+0000 7f191a9e0640 1 -- start start 2024-05-09T03:10:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.055+0000 7f191a9e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f19140b7e70 0x7f19140b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f191a9e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19140d3400 con 0x7f19140d2e90 2024-05-09T03:10:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f191a9e0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19140d4a60 con 0x7f19141605b0 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f191a9e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f19140d4d40 con 0x7f19140b7e70 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1913fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f19140d2e90 0x7f19140b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54810/0 (socket says 172.21.15.46:54810) 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1913fff640 1 -- 172.21.15.46:0/3199245911 learned_addr learned my addr 172.21.15.46:0/3199245911 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f19137fe640 1 --2- 172.21.15.46:0/3199245911 >> v2:172.21.15.88:3300/0 conn(0x7f19140b7e70 0x7f19140b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 555478591 0 0) 0x7f19140d3400 con 0x7f19140d2e90 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18fc003520 con 0x7f19140d2e90 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1601588930 0 0) 0x7f18fc003520 con 0x7f19140d2e90 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 >> v2:172.21.15.88:3300/0 conn(0x7f19140b7e70 msgr2=0x7f19140b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 --2- 172.21.15.46:0/3199245911 >> v2:172.21.15.88:3300/0 conn(0x7f19140b7e70 0x7f19140b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 >> v1:172.21.15.88:6789/0 conn(0x7f19141605b0 legacy=0x7f19140b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19140d5020 con 0x7f19140d2e90 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f1904001710 con 0x7f19140d2e90 2024-05-09T03:10:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 >> v1:172.21.15.46:6789/0 conn(0x7f19140d2e90 legacy=0x7f19140b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18fc003aa0 0x7f18fc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18fc004800 0x7f18fc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f18fc00ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f19140d4d40 con 0x7f18fc005350 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18fc00c2e0 con 0x7f18fc003aa0 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.056+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f18fc00c4e0 con 0x7f18fc004800 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1918f7d640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f18fc00ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1913fff640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18fc003aa0 0x7f18fc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f19137fe640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18fc004800 0x7f18fc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1918f7d640 1 -- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18fc004800 msgr2=0x7f18fc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1918f7d640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18fc004800 0x7f18fc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1918f7d640 1 -- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18fc003aa0 msgr2=0x7f18fc003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1918f7d640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18fc003aa0 0x7f18fc003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.057+0000 7f1918f7d640 1 -- 172.21.15.46:0/3199245911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18fc003520 con 0x7f18fc005350 2024-05-09T03:10:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f1918f7d640 1 -- 172.21.15.46:0/3199245911 reap_dead start 2024-05-09T03:10:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f1918f7d640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f18fc00ba00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f190c001e90 tx=0x7f18fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db1ce5511fccbec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c001ab0 con 0x7f18fc005350 2024-05-09T03:10:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f190c02ae50 con 0x7f18fc005350 2024-05-09T03:10:30.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f1912ffd640 1 -- 172.21.15.46:0/3199245911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c003b70 con 0x7f18fc005350 2024-05-09T03:10:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f191a9e0640 1 -- 172.21.15.46:0/3199245911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 msgr2=0x7f18fc00ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.058+0000 7f191a9e0640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f18fc00ba00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f190c001e90 tx=0x7f18fc005830 comp rx=0 tx=0).stop 2024-05-09T03:10:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.059+0000 7f191a9e0640 1 -- 172.21.15.46:0/3199245911 shutdown_connections 2024-05-09T03:10:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.059+0000 7f191a9e0640 1 --2- 172.21.15.46:0/3199245911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f18fc00ba00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.059+0000 7f191a9e0640 1 -- 172.21.15.46:0/3199245911 >> 172.21.15.46:0/3199245911 conn(0x7f191408d390 msgr2=0x7f191408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.059+0000 7f191a9e0640 1 -- 172.21.15.46:0/3199245911 shutdown_connections 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.059+0000 7f191a9e0640 1 -- 172.21.15.46:0/3199245911 wait complete. 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.060+0000 7f191a9e0640 1 Processor -- start 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.061+0000 7f191a9e0640 1 -- start start 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f191a9e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f191414e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f191a9e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19141f9280 0x7f19141f9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f191a9e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19141f9bd0 0x7f19140cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f191a9e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f19140b4d80 con 0x7f18fc005350 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f191a9e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f19140b4c00 con 0x7f19141f9280 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f191a9e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f19140b4fc0 con 0x7f19141f9bd0 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f191414e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f191414e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53144/0 (socket says 172.21.15.46:53144) 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 -- 172.21.15.46:0/681729463 learned_addr learned my addr 172.21.15.46:0/681729463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f19137fe640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19141f9280 0x7f19141f9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1918f7d640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19141f9bd0 0x7f19140cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 -- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19141f9bd0 msgr2=0x7f19140cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19141f9bd0 0x7f19140cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 -- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19141f9280 msgr2=0x7f19141f9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19141f9280 0x7f19141f9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1913fff640 1 -- 172.21.15.46:0/681729463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1914160bc0 con 0x7f18fc005350 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.062+0000 7f1918f7d640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19141f9bd0 0x7f19140cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.063+0000 7f1913fff640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f191414e100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1904004f40 tx=0x7f1904007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.063+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190400f750 con 0x7f18fc005350 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.063+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1904018070 con 0x7f18fc005350 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.063+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19140cbfe0 con 0x7f18fc005350 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.063+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190400fb10 con 0x7f18fc005350 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.063+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19140ce470 con 0x7f18fc005350 2024-05-09T03:10:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.064+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f191406a270 con 0x7f18fc005350 2024-05-09T03:10:30.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.065+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f190400fcb0 con 0x7f18fc005350 2024-05-09T03:10:30.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.065+0000 7f1910ff9640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f19000762d0 0x7f19000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.065+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f1904097ed0 con 0x7f18fc005350 2024-05-09T03:10:30.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.065+0000 7f19137fe640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f19000762d0 0x7f19000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.066+0000 7f19137fe640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f19000762d0 0x7f19000787e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1914069e20 tx=0x7f1908009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:30.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.069+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f19040618b0 con 0x7f18fc005350 2024-05-09T03:10:30.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.303+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f191415d3a0 con 0x7f18fc005350 2024-05-09T03:10:30.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.851+0000 7f1910ff9640 1 -- 172.21.15.46:0/681729463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v17) ==== 162+0+0 (secure 0 0 0) 0x7f19040611b0 con 0x7f18fc005350 2024-05-09T03:10:30.852 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f19000762d0 msgr2=0x7f19000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f19000762d0 0x7f19000787e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1914069e20 tx=0x7f1908009190 comp rx=0 tx=0).stop 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 msgr2=0x7f191414e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f191414e100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1904004f40 tx=0x7f1904007430 comp rx=0 tx=0).stop 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 shutdown_connections 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f19141f9bd0 0x7f19140cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19141f9280 0x7f19141f9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f19000762d0 0x7f19000787e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 --2- 172.21.15.46:0/681729463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18fc005350 0x7f191414e100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 >> 172.21.15.46:0/681729463 conn(0x7f191408d390 msgr2=0x7f191415efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 shutdown_connections 2024-05-09T03:10:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.854+0000 7f191a9e0640 1 -- 172.21.15.46:0/681729463 wait complete. 2024-05-09T03:10:30.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:30.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.987+0000 7f2c36813640 1 Processor -- start 2024-05-09T03:10:30.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.987+0000 7f2c36813640 1 -- start start 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c36813640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c300b9310 0x7f2c3008cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c36813640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c3008d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c36813640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c3006ebe0 con 0x7f2c30164830 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c36813640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2c3008dbf0 con 0x7f2c300b9310 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c36813640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c3006f760 con 0x7f2c3008d2a0 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c34db0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2c30164830 0x7f2c30164c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54826/0 (socket says 172.21.15.46:54826) 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c34db0640 1 -- 172.21.15.46:0/4280396678 learned_addr learned my addr 172.21.15.46:0/4280396678 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c2f7fe640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c3008d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c2ffff640 1 --2- 172.21.15.46:0/4280396678 >> v2:172.21.15.88:3300/0 conn(0x7f2c300b9310 0x7f2c3008cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1600559676 0 0) 0x7f2c3006ebe0 con 0x7f2c30164830 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c18003520 con 0x7f2c30164830 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 675601421 0 0) 0x7f2c18003520 con 0x7f2c30164830 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.989+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 msgr2=0x7f2c3008d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c3008d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 >> v2:172.21.15.88:3300/0 conn(0x7f2c300b9310 msgr2=0x7f2c3008cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 --2- 172.21.15.46:0/4280396678 >> v2:172.21.15.88:3300/0 conn(0x7f2c300b9310 0x7f2c3008cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3006eec0 con 0x7f2c30164830 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2f7fe640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c3008d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f2c28001a70 con 0x7f2c30164830 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 >> v1:172.21.15.46:6789/0 conn(0x7f2c30164830 legacy=0x7f2c30164c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c18003a00 0x7f2c18003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c18004d40 0x7f2c18005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3008dbf0 con 0x7f2c18004d40 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3006f760 con 0x7f2c18003a00 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c1800ba70 con 0x7f2c3008d2a0 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c34db0640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c18004d40 0x7f2c18005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2ffff640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c18003a00 0x7f2c18003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c2f7fe640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c34db0640 1 -- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 msgr2=0x7f2c18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c34db0640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3008d2a0 0x7f2c18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c34db0640 1 -- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c18003a00 msgr2=0x7f2c18003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:30.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c34db0640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c18003a00 0x7f2c18003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.990+0000 7f2c34db0640 1 -- 172.21.15.46:0/4280396678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c18003520 con 0x7f2c18004d40 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.991+0000 7f2c34db0640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c18004d40 0x7f2c18005150 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800a8a0 tx=0x7f2c18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3757a6c5d9d5c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.991+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2800a4c0 con 0x7f2c18004d40 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.991+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c2802f070 con 0x7f2c18004d40 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.991+0000 7f2c2effd640 1 -- 172.21.15.46:0/4280396678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2802bc60 con 0x7f2c18004d40 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c36813640 1 -- 172.21.15.46:0/4280396678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c18004d40 msgr2=0x7f2c18005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c36813640 1 --2- 172.21.15.46:0/4280396678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c18004d40 0x7f2c18005150 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2c2800a8a0 tx=0x7f2c18005830 comp rx=0 tx=0).stop 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c34db0640 1 -- 172.21.15.46:0/4280396678 reap_dead start 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c36813640 1 -- 172.21.15.46:0/4280396678 shutdown_connections 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c36813640 1 -- 172.21.15.46:0/4280396678 >> 172.21.15.46:0/4280396678 conn(0x7f2c300b5660 msgr2=0x7f2c300b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c36813640 1 -- 172.21.15.46:0/4280396678 shutdown_connections 2024-05-09T03:10:30.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.992+0000 7f2c36813640 1 -- 172.21.15.46:0/4280396678 wait complete. 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.994+0000 7f2c36813640 1 Processor -- start 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.994+0000 7f2c36813640 1 -- start start 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c36813640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 0x7f2c3015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c36813640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3015b460 0x7f2c3014aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c36813640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c3014d370 0x7f2c3014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c36813640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c30165350 con 0x7f2c3009f0e0 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c36813640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3006f480 con 0x7f2c3014d370 2024-05-09T03:10:30.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c36813640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c301654d0 con 0x7f2c3015b460 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 0x7f2c3015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 0x7f2c3015eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53160/0 (socket says 172.21.15.46:53160) 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2ffff640 1 -- 172.21.15.46:0/1532235251 learned_addr learned my addr 172.21.15.46:0/1532235251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c34db0640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c3014d370 0x7f2c3014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2f7fe640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3015b460 0x7f2c3014aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2ffff640 1 -- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3015b460 msgr2=0x7f2c3014aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2ffff640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3015b460 0x7f2c3014aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.995+0000 7f2c2ffff640 1 -- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c3014d370 msgr2=0x7f2c3014d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c2ffff640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c3014d370 0x7f2c3014d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c2ffff640 1 -- 172.21.15.46:0/1532235251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3014afe0 con 0x7f2c3009f0e0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c2ffff640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 0x7f2c3015eac0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c200119b0 tx=0x7f2c20011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20019070 con 0x7f2c3009f0e0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c20004240 con 0x7f2c3009f0e0 2024-05-09T03:10:31.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3014e260 con 0x7f2c3009f0e0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000f8e0 con 0x7f2c3009f0e0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.996+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c3014b660 con 0x7f2c3009f0e0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.997+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c3006a270 con 0x7f2c3009f0e0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.998+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2c2001c070 con 0x7f2c3009f0e0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.998+0000 7f2c2cff9640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c1c076300 0x7f2c1c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.998+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f2c200980b0 con 0x7f2c3009f0e0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.998+0000 7f2c2f7fe640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c1c076300 0x7f2c1c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:31.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:30.999+0000 7f2c2f7fe640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c1c076300 0x7f2c1c078810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2c300b8960 tx=0x7f2c24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:31.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.002+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2c20061920 con 0x7f2c3009f0e0 2024-05-09T03:10:31.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.229+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c300b4f60 con 0x7f2c3009f0e0 2024-05-09T03:10:31.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.230+0000 7f2c2cff9640 1 -- 172.21.15.46:0/1532235251 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5172 (secure 0 0 0) 0x7f2c20061220 con 0x7f2c3009f0e0 2024-05-09T03:10:31.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:31.231 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:30.854644+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-05-09T03:10:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 18 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c1c076300 msgr2=0x7f2c1c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c1c076300 0x7f2c1c078810 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2c300b8960 tx=0x7f2c24008040 comp rx=0 tx=0).stop 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 msgr2=0x7f2c3015eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 0x7f2c3015eac0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c200119b0 tx=0x7f2c20011e70 comp rx=0 tx=0).stop 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 shutdown_connections 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c3014d370 0x7f2c3014d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c3015b460 0x7f2c3014aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c1c076300 0x7f2c1c078810 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 --2- 172.21.15.46:0/1532235251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c3009f0e0 0x7f2c3015eac0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 >> 172.21.15.46:0/1532235251 conn(0x7f2c300b5660 msgr2=0x7f2c30161bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 shutdown_connections 2024-05-09T03:10:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:31.232+0000 7f2c36813640 1 -- 172.21.15.46:0/1532235251 wait complete. 2024-05-09T03:10:31.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:10:13.572699+0000', 'modified': '2024-05-09T03:10:30.854644+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5010': {'gid': 5010, 'name': 'a', 'rank': 0, 'incarnation': 18, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/2536776736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2536776736}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2536776736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:10:31.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5010:up:replay 2024-05-09T03:10:32.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.776+0000 7efcce470640 1 Processor -- start 2024-05-09T03:10:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.776+0000 7efcce470640 1 -- start start 2024-05-09T03:10:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.777+0000 7efcce470640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc81615f0 0x7efcc80b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.777+0000 7efcce470640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcc80d15b0 con 0x7efcc80d0b80 2024-05-09T03:10:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.777+0000 7efcce470640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcc8160b10 con 0x7efcc80d2cf0 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.777+0000 7efcce470640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efcc8160df0 con 0x7efcc81615f0 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efcc80d0b80 0x7efcc80d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54838/0 (socket says 172.21.15.46:54838) 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc77fe640 1 -- 172.21.15.46:0/2055628069 learned_addr learned my addr 172.21.15.46:0/2055628069 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc7fff640 1 --2- 172.21.15.46:0/2055628069 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc81615f0 0x7efcc80b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 12566090 0 0) 0x7efcc80d15b0 con 0x7efcc80d0b80 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcb0003520 con 0x7efcc80d0b80 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522105569 0 0) 0x7efcc8160b10 con 0x7efcc80d2cf0 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcc80d15b0 con 0x7efcc80d2cf0 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 598719861 0 0) 0x7efcb0003520 con 0x7efcc80d0b80 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc7fff640 1 -- 172.21.15.46:0/2055628069 >> v1:172.21.15.88:6789/0 conn(0x7efcc80d2cf0 legacy=0x7efcc8163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc7fff640 1 -- 172.21.15.46:0/2055628069 >> v1:172.21.15.46:6789/0 conn(0x7efcc80d0b80 legacy=0x7efcc80d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.778+0000 7efcc7fff640 1 -- 172.21.15.46:0/2055628069 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc81610d0 con 0x7efcc81615f0 2024-05-09T03:10:32.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.779+0000 7efcc7fff640 1 --2- 172.21.15.46:0/2055628069 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc81615f0 0x7efcc80b81b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efcb80097d0 tx=0x7efcb802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc855ba17c7393df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.779+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8037070 con 0x7efcc81615f0 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.779+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb802ac30 con 0x7efcc81615f0 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.779+0000 7efcc6ffd640 1 -- 172.21.15.46:0/2055628069 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc001710 con 0x7efcc81615f0 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 -- 172.21.15.46:0/2055628069 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc81615f0 msgr2=0x7efcc80b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 --2- 172.21.15.46:0/2055628069 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc81615f0 0x7efcc80b81b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efcb80097d0 tx=0x7efcb802a1a0 comp rx=0 tx=0).stop 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 -- 172.21.15.46:0/2055628069 shutdown_connections 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 --2- 172.21.15.46:0/2055628069 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc81615f0 0x7efcc80b81b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 -- 172.21.15.46:0/2055628069 >> 172.21.15.46:0/2055628069 conn(0x7efcc806e190 msgr2=0x7efcc806e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 -- 172.21.15.46:0/2055628069 shutdown_connections 2024-05-09T03:10:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.780+0000 7efcce470640 1 -- 172.21.15.46:0/2055628069 wait complete. 2024-05-09T03:10:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.782+0000 7efcce470640 1 Processor -- start 2024-05-09T03:10:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.782+0000 7efcce470640 1 -- start start 2024-05-09T03:10:32.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efcce470640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc80d0b80 0x7efcc8159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:32.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efcce470640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc80d2cf0 0x7efcc80c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:32.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efcce470640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 0x7efcc81fa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efcce470640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efcc80d1110 con 0x7efcc81615f0 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efcce470640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efcc80d0f90 con 0x7efcc80d2cf0 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efcce470640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efcc80d1290 con 0x7efcc80d0b80 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.783+0000 7efccca0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 0x7efcc81fa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 0x7efcc81fa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53170/0 (socket says 172.21.15.46:53170) 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 -- 172.21.15.46:0/3938153810 learned_addr learned my addr 172.21.15.46:0/3938153810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcc7fff640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc80d0b80 0x7efcc8159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcc77fe640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc80d2cf0 0x7efcc80c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 -- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc80d0b80 msgr2=0x7efcc8159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc80d0b80 0x7efcc8159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 -- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc80d2cf0 msgr2=0x7efcc80c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc80d2cf0 0x7efcc80c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:32.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 -- 172.21.15.46:0/3938153810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc8159a40 con 0x7efcc81615f0 2024-05-09T03:10:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efccca0d640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 0x7efcc81fa250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efcc000c730 tx=0x7efcc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc0003b70 con 0x7efcc81615f0 2024-05-09T03:10:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc0003d50 con 0x7efcc81615f0 2024-05-09T03:10:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc815b2d0 con 0x7efcc81615f0 2024-05-09T03:10:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc0005310 con 0x7efcc81615f0 2024-05-09T03:10:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.784+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc815b890 con 0x7efcc81615f0 2024-05-09T03:10:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.785+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc806a270 con 0x7efcc81615f0 2024-05-09T03:10:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.786+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efcc00056c0 con 0x7efcc81615f0 2024-05-09T03:10:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.787+0000 7efcc4ff9640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcb4076300 0x7efcb4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.787+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7efcc00975f0 con 0x7efcc81615f0 2024-05-09T03:10:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.787+0000 7efcc7fff640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcb4076300 0x7efcb4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.788+0000 7efcc7fff640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcb4076300 0x7efcb4078810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcb8034000 tx=0x7efcb802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:32.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:32.792+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efcc00608b0 con 0x7efcc81615f0 2024-05-09T03:10:33.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.020+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcc8159c10 con 0x7efcc81615f0 2024-05-09T03:10:33.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.020+0000 7efcc4ff9640 1 -- 172.21.15.46:0/3938153810 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5172 (secure 0 0 0) 0x7efcc0064bb0 con 0x7efcc81615f0 2024-05-09T03:10:33.022 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:33.022 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:31.863008+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-05-09T03:10:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2024-05-09T03:10:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcb4076300 msgr2=0x7efcb4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:33.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcb4076300 0x7efcb4078810 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcb8034000 tx=0x7efcb802e000 comp rx=0 tx=0).stop 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 msgr2=0x7efcc81fa250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 0x7efcc81fa250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efcc000c730 tx=0x7efcc000cbf0 comp rx=0 tx=0).stop 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 shutdown_connections 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcb4076300 0x7efcb4078810 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc81615f0 0x7efcc81fa250 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcc80d2cf0 0x7efcc80c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 --2- 172.21.15.46:0/3938153810 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efcc80d0b80 0x7efcc8159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 >> 172.21.15.46:0/3938153810 conn(0x7efcc806e190 msgr2=0x7efcc815ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 shutdown_connections 2024-05-09T03:10:33.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:33.022+0000 7efcce470640 1 -- 172.21.15.46:0/3938153810 wait complete. 2024-05-09T03:10:33.438 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:10:13.572699+0000', 'modified': '2024-05-09T03:10:31.863008+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5010': {'gid': 5010, 'name': 'a', 'rank': 0, 'incarnation': 18, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.46:6834/2536776736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2536776736}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2536776736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:10:33.438 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5010:up:rejoin 2024-05-09T03:10:34.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:34.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.568+0000 7f5d25643640 1 Processor -- start 2024-05-09T03:10:34.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.568+0000 7f5d25643640 1 -- start start 2024-05-09T03:10:34.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.569+0000 7f5d25643640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5d200d57a0 0x7f5d20160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.569+0000 7f5d25643640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d200b4af0 0x7f5d200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.569+0000 7f5d25643640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.569+0000 7f5d25643640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5d200d5bb0 con 0x7f5d200b4af0 2024-05-09T03:10:34.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.569+0000 7f5d25643640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5d200b8130 con 0x7f5d200d57a0 2024-05-09T03:10:34.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.569+0000 7f5d25643640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5d200d49e0 con 0x7f5d200b77e0 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d200b4af0 0x7f5d200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d200b4af0 0x7f5d200b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53184/0 (socket says 172.21.15.46:53184) 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 -- 172.21.15.46:0/2832498483 learned_addr learned my addr 172.21.15.46:0/2832498483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d1f7fe640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d1effd640 1 --2- 172.21.15.46:0/2832498483 >> v2:172.21.15.88:3300/0 conn(0x7f5d200d57a0 0x7f5d20160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 -- 172.21.15.46:0/2832498483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 msgr2=0x7f5d200b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 -- 172.21.15.46:0/2832498483 >> v2:172.21.15.88:3300/0 conn(0x7f5d200d57a0 msgr2=0x7f5d20160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 --2- 172.21.15.46:0/2832498483 >> v2:172.21.15.88:3300/0 conn(0x7f5d200d57a0 0x7f5d20160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d167fc640 1 -- 172.21.15.46:0/2832498483 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d200d4cc0 con 0x7f5d200b4af0 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.570+0000 7f5d1f7fe640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.571+0000 7f5d167fc640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200b4af0 0x7f5d200b4f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5d2008ed20 tx=0x7f5d0802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845ebb8bbd5cec25 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:34.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.571+0000 7f5d1e7fc640 1 -- 172.21.15.46:0/2832498483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08037070 con 0x7f5d200b4af0 2024-05-09T03:10:34.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.571+0000 7f5d1e7fc640 1 -- 172.21.15.46:0/2832498483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d08032440 con 0x7f5d200b4af0 2024-05-09T03:10:34.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.571+0000 7f5d1e7fc640 1 -- 172.21.15.46:0/2832498483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08032810 con 0x7f5d200b4af0 2024-05-09T03:10:34.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 -- 172.21.15.46:0/2832498483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200b4af0 msgr2=0x7f5d200b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:34.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200b4af0 0x7f5d200b4f00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5d2008ed20 tx=0x7f5d0802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 -- 172.21.15.46:0/2832498483 shutdown_connections 2024-05-09T03:10:34.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 --2- 172.21.15.46:0/2832498483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200b4af0 0x7f5d200b4f00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 --2- 172.21.15.46:0/2832498483 >> v2:172.21.15.88:3300/0 conn(0x7f5d200d57a0 0x7f5d20160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 -- 172.21.15.46:0/2832498483 >> 172.21.15.46:0/2832498483 conn(0x7f5d200743e0 msgr2=0x7f5d200d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:34.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 -- 172.21.15.46:0/2832498483 shutdown_connections 2024-05-09T03:10:34.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.572+0000 7f5d25643640 1 -- 172.21.15.46:0/2832498483 wait complete. 2024-05-09T03:10:34.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.574+0000 7f5d25643640 1 Processor -- start 2024-05-09T03:10:34.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.574+0000 7f5d25643640 1 -- start start 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d25643640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d200b4af0 0x7f5d2014c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d25643640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d25643640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 0x7f5d20162d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d25643640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d200d15b0 con 0x7f5d200d57a0 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d25643640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5d200d5280 con 0x7f5d200b4af0 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d25643640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5d200d1920 con 0x7f5d200b77e0 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d1f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 0x7f5d20162d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d1f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 0x7f5d20162d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53200/0 (socket says 172.21.15.46:53200) 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d1f7fe640 1 -- 172.21.15.46:0/3333966878 learned_addr learned my addr 172.21.15.46:0/3333966878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:34.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d167fc640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.575+0000 7f5d1effd640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d200b4af0 0x7f5d2014c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d1f7fe640 1 -- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 msgr2=0x7f5d200c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d1f7fe640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d1f7fe640 1 -- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d200b4af0 msgr2=0x7f5d2014c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d1f7fe640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d200b4af0 0x7f5d2014c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d1f7fe640 1 -- 172.21.15.46:0/3333966878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d200ca490 con 0x7f5d200d57a0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d1f7fe640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 0x7f5d20162d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007db0 tx=0x7f5d1800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1801a070 con 0x7f5d200d57a0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d18015400 con 0x7f5d200d57a0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d180157f0 con 0x7f5d200d57a0 2024-05-09T03:10:34.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2014f140 con 0x7f5d200d57a0 2024-05-09T03:10:34.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.576+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2014f700 con 0x7f5d200d57a0 2024-05-09T03:10:34.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.577+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d200ca200 con 0x7f5d200d57a0 2024-05-09T03:10:34.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.578+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5d1801d070 con 0x7f5d200d57a0 2024-05-09T03:10:34.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.578+0000 7f5d17fff640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5d000762d0 0x7f5d000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:34.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.579+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f5d18097cf0 con 0x7f5d200d57a0 2024-05-09T03:10:34.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.579+0000 7f5d1effd640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5d000762d0 0x7f5d000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:34.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.579+0000 7f5d1effd640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5d000762d0 0x7f5d000787e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d2006d3b0 tx=0x7f5d10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:34.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.582+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5d18024020 con 0x7f5d200d57a0 2024-05-09T03:10:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.811+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d2006a270 con 0x7f5d200d57a0 2024-05-09T03:10:34.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.812+0000 7f5d17fff640 1 -- 172.21.15.46:0/3333966878 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7f5d18010070 con 0x7f5d200d57a0 2024-05-09T03:10:34.814 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:34.814 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:10:34.814 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:10:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5d000762d0 msgr2=0x7f5d000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5d000762d0 0x7f5d000787e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d2006d3b0 tx=0x7f5d10008040 comp rx=0 tx=0).stop 2024-05-09T03:10:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 msgr2=0x7f5d20162d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 0x7f5d20162d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007db0 tx=0x7f5d1800d4a0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 shutdown_connections 2024-05-09T03:10:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5d000762d0 0x7f5d000787e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d200d57a0 0x7f5d20162d80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d200b77e0 0x7f5d200c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 --2- 172.21.15.46:0/3333966878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d200b4af0 0x7f5d2014c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 >> 172.21.15.46:0/3333966878 conn(0x7f5d200743e0 msgr2=0x7f5d200d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 shutdown_connections 2024-05-09T03:10:34.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:34.814+0000 7f5d25643640 1 -- 172.21.15.46:0/3333966878 wait complete. 2024-05-09T03:10:35.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:10:13.572699+0000', 'modified': '2024-05-09T03:10:33.866474+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 32, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5010}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5010': {'gid': 5010, 'name': 'a', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/2536776736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2536776736}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2536776736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5010, 'qdb_cluster': [5010]} 2024-05-09T03:10:35.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:10:35.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T03:10:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.346+0000 7f32f4bb2640 1 Processor -- start 2024-05-09T03:10:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.346+0000 7f32f4bb2640 1 -- start start 2024-05-09T03:10:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32f4bb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f00b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32f4bb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32f00b3f50 0x7f32f00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32f4bb2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32f006ebe0 con 0x7f32f00b4700 2024-05-09T03:10:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32f4bb2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f32f006eec0 con 0x7f32f00b3f50 2024-05-09T03:10:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32f4bb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32f006f1a0 con 0x7f32f00d2800 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32eed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32f00b3f50 0x7f32f00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32ee59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f32f00b4700 0x7f32f00d20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54852/0 (socket says 172.21.15.46:54852) 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32ee59c640 1 -- 172.21.15.46:0/3838426260 learned_addr learned my addr 172.21.15.46:0/3838426260 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.348+0000 7f32edd9b640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f00b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32eed9d640 1 -- 172.21.15.46:0/3838426260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 msgr2=0x7f32f00b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32eed9d640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f00b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32eed9d640 1 -- 172.21.15.46:0/3838426260 >> v1:172.21.15.46:6789/0 conn(0x7f32f00b4700 legacy=0x7f32f00d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32eed9d640 1 -- 172.21.15.46:0/3838426260 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f006f480 con 0x7f32f00b3f50 2024-05-09T03:10:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32edd9b640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f00b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:35.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32eed9d640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b3f50 0x7f32f00ce130 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f32e80097a0 tx=0x7f32e802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c1bd7681f6cef6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32ed59a640 1 -- 172.21.15.46:0/3838426260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8037070 con 0x7f32f00b3f50 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.349+0000 7f32ed59a640 1 -- 172.21.15.46:0/3838426260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e802ae50 con 0x7f32f00b3f50 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32ed59a640 1 -- 172.21.15.46:0/3838426260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e8032660 con 0x7f32f00b3f50 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 -- 172.21.15.46:0/3838426260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b3f50 msgr2=0x7f32f00ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b3f50 0x7f32f00ce130 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f32e80097a0 tx=0x7f32e802a100 comp rx=0 tx=0).stop 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 -- 172.21.15.46:0/3838426260 shutdown_connections 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b3f50 0x7f32f00ce130 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/3838426260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f00b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 -- 172.21.15.46:0/3838426260 >> 172.21.15.46:0/3838426260 conn(0x7f32f00b55d0 msgr2=0x7f32f00b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 -- 172.21.15.46:0/3838426260 shutdown_connections 2024-05-09T03:10:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.350+0000 7f32f4bb2640 1 -- 172.21.15.46:0/3838426260 wait complete. 2024-05-09T03:10:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.352+0000 7f32f4bb2640 1 Processor -- start 2024-05-09T03:10:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.352+0000 7f32f4bb2640 1 -- start start 2024-05-09T03:10:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32f4bb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32f00b3f50 0x7f32f01dbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32f4bb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 0x7f32f01fa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32f4bb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f01fa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32f4bb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f32f00b51d0 con 0x7f32f00b4700 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32f4bb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32f00b4360 con 0x7f32f00b3f50 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32f4bb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32f009eae0 con 0x7f32f00d2800 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 0x7f32f01fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 0x7f32f01fa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53216/0 (socket says 172.21.15.46:53216) 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 -- 172.21.15.46:0/1235010446 learned_addr learned my addr 172.21.15.46:0/1235010446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32eed9d640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f01fa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32ee59c640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32f00b3f50 0x7f32f01dbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 -- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 msgr2=0x7f32f01fa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f01fa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 -- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32f00b3f50 msgr2=0x7f32f01dbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32f00b3f50 0x7f32f01dbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 -- 172.21.15.46:0/1235010446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f00b2bd0 con 0x7f32f00b4700 2024-05-09T03:10:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.354+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 0x7f32f01fa170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f32d800d800 tx=0x7f32d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.355+0000 7f32e6ffd640 1 -- 172.21.15.46:0/1235010446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8004030 con 0x7f32f00b4700 2024-05-09T03:10:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.355+0000 7f32e6ffd640 1 -- 172.21.15.46:0/1235010446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d80041d0 con 0x7f32f00b4700 2024-05-09T03:10:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.355+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f00b2eb0 con 0x7f32f00b4700 2024-05-09T03:10:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.355+0000 7f32e6ffd640 1 -- 172.21.15.46:0/1235010446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d80096a0 con 0x7f32f00b4700 2024-05-09T03:10:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.355+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f009e820 con 0x7f32f00b4700 2024-05-09T03:10:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.356+0000 7f32e6ffd640 1 -- 172.21.15.46:0/1235010446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f32d8009840 con 0x7f32f00b4700 2024-05-09T03:10:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.357+0000 7f32e6ffd640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32d00762d0 0x7f32d00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.357+0000 7f32e6ffd640 1 -- 172.21.15.46:0/1235010446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f32d8097770 con 0x7f32f00b4700 2024-05-09T03:10:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.357+0000 7f32ee59c640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32d00762d0 0x7f32d00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.358+0000 7f32fba65740 1 Processor -- start 2024-05-09T03:10:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.358+0000 7f32ee59c640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32d00762d0 0x7f32d00787e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f32e00012e0 tx=0x7f32e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.358+0000 7f32fba65740 1 -- start start 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32fba65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 0x557fbf4f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf50ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32fba65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557fbf42c4e0 con 0x557fbf124270 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557fbf3c7ff0 con 0x557fbf0fb3b0 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557fbf4284a0 con 0x557fbf103590 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32ee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 0x557fbf4f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32ee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 0x557fbf4f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53226/0 (socket says 172.21.15.46:53226) 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.359+0000 7f32ee59c640 1 -- 172.21.15.46:0/3612023274 learned_addr learned my addr 172.21.15.46:0/3612023274 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32eed9d640 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32edd9b640 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf50ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32ee59c640 1 -- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 msgr2=0x557fbf50f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32ee59c640 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32ee59c640 1 -- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 msgr2=0x557fbf50ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32ee59c640 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf50ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32ee59c640 1 -- 172.21.15.46:0/3612023274 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fbf3ae720 con 0x557fbf124270 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32ee59c640 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 0x557fbf4f3eb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f32e00144d0 tx=0x7f32e0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4815fa02954adcc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32cffff640 1 -- 172.21.15.46:0/3612023274 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0025070 con 0x557fbf124270 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32cffff640 1 -- 172.21.15.46:0/3612023274 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e001b050 con 0x557fbf124270 2024-05-09T03:10:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.360+0000 7f32cffff640 1 -- 172.21.15.46:0/3612023274 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e00137c0 con 0x557fbf124270 2024-05-09T03:10:35.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 -- 172.21.15.46:0/3612023274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 msgr2=0x557fbf4f3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 0x557fbf4f3eb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f32e00144d0 tx=0x7f32e0012000 comp rx=0 tx=0).stop 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 -- 172.21.15.46:0/3612023274 shutdown_connections 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf50ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 --2- 172.21.15.46:0/3612023274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf124270 0x557fbf4f3eb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 -- 172.21.15.46:0/3612023274 >> 172.21.15.46:0/3612023274 conn(0x557fbf1183c0 msgr2=0x557fbf4c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 -- 172.21.15.46:0/3612023274 shutdown_connections 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.361+0000 7f32fba65740 1 -- 172.21.15.46:0/3612023274 wait complete. 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.362+0000 7f32fba65740 1 -- start start 2024-05-09T03:10:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.362+0000 7f32fba65740 1 Processor -- start 2024-05-09T03:10:35.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.363+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf509b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.363+0000 7f32fba65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.363+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf124270 0x557fbf50b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.363+0000 7f32fba65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557fbf3b3760 con 0x557fbf103590 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557fbf50d6f0 con 0x557fbf0fb3b0 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557fbf4f5c60 con 0x557fbf124270 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32fba65740 10 client.0 _ll_register_callbacks cb 0x557fbf3d64b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53240/0 (socket says 172.21.15.46:53240) 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 -- 172.21.15.46:0/3563673787 learned_addr learned my addr 172.21.15.46:0/3563673787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32eed9d640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf124270 0x557fbf50b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32ee59c640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf509b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 -- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf124270 msgr2=0x557fbf50b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf124270 0x557fbf50b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 -- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 msgr2=0x557fbf509b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf509b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 -- 172.21.15.46:0/3563673787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fbf3f4bd0 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32eed9d640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf124270 0x557fbf50b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32edd9b640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50ad30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32d805ca90 tx=0x7f32d8060180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583eecb73553b4a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32cd7fa640 1 -- 172.21.15.46:0/3563673787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8093060 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32cd7fa640 1 -- 172.21.15.46:0/3563673787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f32d8009d30 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e001b050 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32cd7fa640 1 -- 172.21.15.46:0/3563673787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f32d8079030 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32fba65740 10 client.5058 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.364+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x557fbf3da280 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32cd7fa640 1 -- 172.21.15.46:0/3563673787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e001b050 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32cd7fa640 1 -- 172.21.15.46:0/3563673787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f32d8013070 con 0x557fbf103590 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32fba65740 10 client.5058 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32fba65740 10 client.5058 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32fba65740 20 client.5058 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:35.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32fba65740 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf0e2e40 0x557fbf521a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32fba65740 4 client.5058 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fbf43a0f0 con 0x557fbf0e2e40 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32ee59c640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf0e2e40 0x557fbf521a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32cd7fa640 10 client.5058 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.365+0000 7f32ee59c640 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf0e2e40 0x557fbf521a10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe4b600f93f5cb9b server_cookie=c938a3e3263fe782 in_seq=0 out_seq=0 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32cd7fa640 1 -- 172.21.15.46:0/3563673787 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557fbf43a0f0 con 0x557fbf0e2e40 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32cd7fa640 10 client.5058 handle_command_reply: tid=0 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 client.5058 shutdown 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 msgr2=0x557fbf50ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50ad30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32d805ca90 tx=0x7f32d8060180 comp rx=0 tx=0).stop 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 shutdown_connections 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf124270 0x557fbf50b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf0e2e40 0x557fbf521a10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50ad30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 --2- 172.21.15.46:0/3563673787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf0fb3b0 0x557fbf509b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 >> 172.21.15.46:0/3563673787 conn(0x557fbf1183c0 msgr2=0x557fbf4c9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32cd7fa640 0 client.5058 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.366+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 shutdown_connections 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.367+0000 7f32fba65740 1 -- 172.21.15.46:0/3563673787 wait complete. 2024-05-09T03:10:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.367+0000 7f32fba65740 20 client.5058 trim_cache size 0 max 16384 2024-05-09T03:10:35.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.385+0000 7f32fba65740 1 Processor -- start 2024-05-09T03:10:35.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.385+0000 7f32fba65740 1 -- start start 2024-05-09T03:10:35.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf0fb3b0 0x557fbf5280a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32fba65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32fba65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f32f01f7760 con 0x557fbf103590 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32f01f6ac0 con 0x557fbf124270 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32f01f7110 con 0x557fbf0fb3b0 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53252/0 (socket says 172.21.15.46:53252) 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 -- 172.21.15.46:0/957036197 learned_addr learned my addr 172.21.15.46:0/957036197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32ee59c640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf0fb3b0 0x557fbf5280a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32eed9d640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 -- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf0fb3b0 msgr2=0x557fbf5280a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf0fb3b0 0x557fbf5280a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 -- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 msgr2=0x557fbf50afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 -- 172.21.15.46:0/957036197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fbf18f850 con 0x557fbf103590 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32ee59c640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf0fb3b0 0x557fbf5280a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32eed9d640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32edd9b640 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50aa90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f32d800bfd0 tx=0x7f32d8067050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706a657970b03d2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32cf7fe640 1 -- 172.21.15.46:0/957036197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d806d070 con 0x557fbf103590 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32cf7fe640 1 -- 172.21.15.46:0/957036197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d8009d30 con 0x557fbf103590 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.387+0000 7f32cf7fe640 1 -- 172.21.15.46:0/957036197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d805fe70 con 0x557fbf103590 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 -- 172.21.15.46:0/957036197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 msgr2=0x557fbf50aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50aa90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f32d800bfd0 tx=0x7f32d8067050 comp rx=0 tx=0).stop 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 -- 172.21.15.46:0/957036197 shutdown_connections 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 --2- 172.21.15.46:0/957036197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf0fb3b0 0x557fbf5280a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 -- 172.21.15.46:0/957036197 >> 172.21.15.46:0/957036197 conn(0x557fbf1183c0 msgr2=0x557fbf511480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 -- 172.21.15.46:0/957036197 shutdown_connections 2024-05-09T03:10:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.388+0000 7f32fba65740 1 -- 172.21.15.46:0/957036197 wait complete. 2024-05-09T03:10:35.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.389+0000 7f32fba65740 1 -- start start 2024-05-09T03:10:35.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.389+0000 7f32fba65740 1 Processor -- start 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 0x557fbf4f5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557fbf4eeac0 con 0x557fbf0fb3b0 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557fbf50cfa0 con 0x557fbf124270 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557fbf4f61e0 con 0x557fbf103590 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 10 client.0 _ll_register_callbacks cb 0x7f32d800d750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 0x557fbf4f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 0x557fbf4f5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53262/0 (socket says 172.21.15.46:53262) 2024-05-09T03:10:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 -- 172.21.15.46:0/1449198747 learned_addr learned my addr 172.21.15.46:0/1449198747 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32eed9d640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 -- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 msgr2=0x557fbf50aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 -- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 msgr2=0x557fbf50afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 -- 172.21.15.46:0/1449198747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fbf2066b0 con 0x557fbf0fb3b0 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32eed9d640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32ee59c640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 0x557fbf4f5830 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f32e0020000 tx=0x7f32e0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace47af9e4fc38cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32cd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32cd7fa640 1 -- 172.21.15.46:0/1449198747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0025040 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32cd7fa640 1 -- 172.21.15.46:0/1449198747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f32e0013b50 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.391+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32d8009d30 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32cd7fa640 1 -- 172.21.15.46:0/1449198747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f32e0041050 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 10 client.5064 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x557fbf3e8630 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32cd7fa640 1 -- 172.21.15.46:0/1449198747 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d8009d30 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32cd7fa640 1 -- 172.21.15.46:0/1449198747 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f32e0044020 con 0x557fbf0fb3b0 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 10 client.5064 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 10 client.5064 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 20 client.5064 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf58c3f0 0x557fbf58c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 4 client.5064 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:10:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "status"}) -- 0x557fbf4f1240 con 0x557fbf58c3f0 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf58c3f0 0x557fbf58c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.392+0000 7f32cd7fa640 10 client.5064 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32edd9b640 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf58c3f0 0x557fbf58c800 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66f4722159119263 server_cookie=a2060945ec7ff1f9 in_seq=0 out_seq=0 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32cd7fa640 1 -- 172.21.15.46:0/1449198747 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x557fbf4f1240 con 0x557fbf58c3f0 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32cd7fa640 10 client.5064 handle_command_reply: tid=0 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 client.5064 shutdown 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 msgr2=0x557fbf4f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 0x557fbf4f5830 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f32e0020000 tx=0x7f32e0001250 comp rx=0 tx=0).stop 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 shutdown_connections 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557fbf124270 0x557fbf50afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557fbf103590 0x557fbf50aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x557fbf58c3f0 0x557fbf58c800 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 --2- 172.21.15.46:0/1449198747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557fbf0fb3b0 0x557fbf4f5830 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32cd7fa640 0 client.5064 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 >> 172.21.15.46:0/1449198747 conn(0x557fbf1183c0 msgr2=0x557fbf4bf660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 shutdown_connections 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 1 -- 172.21.15.46:0/1449198747 wait complete. 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.393+0000 7f32fba65740 20 client.5064 trim_cache size 0 max 16384 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5010, 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:10:35.401 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.5356848699999999, 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 21, 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 32, 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 32, 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 5.4819400229999999 2024-05-09T03:10:35.402 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32d00762d0 msgr2=0x7f32d00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32d00762d0 0x7f32d00787e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f32e00012e0 tx=0x7f32e0009040 comp rx=0 tx=0).stop 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 msgr2=0x7f32f01fa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 0x7f32f01fa170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f32d800d800 tx=0x7f32d800dcc0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 shutdown_connections 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32f00d2800 0x7f32f01fa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32d00762d0 0x7f32d00787e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f00b4700 0x7f32f01fa170 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 --2- 172.21.15.46:0/1235010446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32f00b3f50 0x7f32f01dbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 >> 172.21.15.46:0/1235010446 conn(0x7f32f00b55d0 msgr2=0x7f32f00b6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 shutdown_connections 2024-05-09T03:10:35.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.394+0000 7f32f4bb2640 1 -- 172.21.15.46:0/1235010446 wait complete. 2024-05-09T03:10:35.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:10:35.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.519+0000 7ff2af376640 1 Processor -- start 2024-05-09T03:10:35.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.519+0000 7ff2af376640 1 -- start start 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.520+0000 7ff2af376640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 0x7ff2a80b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.520+0000 7ff2af376640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a80b8130 con 0x7ff2a80d57a0 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.520+0000 7ff2af376640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a80d49e0 con 0x7ff2a80b7d20 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.520+0000 7ff2af376640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff2a80d5bb0 con 0x7ff2a8160610 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff2ad112640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff2a80d57a0 0x7ff2a80b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54866/0 (socket says 172.21.15.46:54866) 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff2ad112640 1 -- 172.21.15.46:0/4116268414 learned_addr learned my addr 172.21.15.46:0/4116268414 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff2ac911640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 0x7ff2a80b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030597880 0 0) 0x7ff2a4001110 con 0x7ff2a80d57a0 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff294003520 con 0x7ff2a80d57a0 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3818906763 0 0) 0x7ff294003520 con 0x7ff2a80d57a0 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 msgr2=0x7ff2a80b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 0x7ff2a80b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 >> v1:172.21.15.88:6789/0 conn(0x7ff2a80b7d20 legacy=0x7ff2a80b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a80d4cc0 con 0x7ff2a80d57a0 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ff2a40018c0 con 0x7ff2a80d57a0 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 >> v1:172.21.15.46:6789/0 conn(0x7ff2a80d57a0 legacy=0x7ff2a80b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294003a30 0x7ff294003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff294004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 0x7ff2940052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a80d5bb0 con 0x7ff294004800 2024-05-09T03:10:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff29400bbb0 con 0x7ff294003a30 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff29400bdb0 con 0x7ff2a8160610 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff2ac911640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff294004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff2ad112640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294003a30 0x7ff294003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.521+0000 7ff2ad913640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 0x7ff2940052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.522+0000 7ff2ac911640 1 -- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 msgr2=0x7ff2940052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.522+0000 7ff2ac911640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8160610 0x7ff2940052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.522+0000 7ff2ac911640 1 -- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294003a30 msgr2=0x7ff294003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.522+0000 7ff2ac911640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294003a30 0x7ff294003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.522+0000 7ff2ac911640 1 -- 172.21.15.46:0/4116268414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff294003520 con 0x7ff294004800 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.522+0000 7ff2ac911640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff294004c10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff298009a90 tx=0x7ff294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23efc01e12cf4ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298036070 con 0x7ff294004800 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff298031400 con 0x7ff294004800 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff29ffff640 1 -- 172.21.15.46:0/4116268414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff298009690 con 0x7ff294004800 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2af376640 1 -- 172.21.15.46:0/4116268414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 msgr2=0x7ff294004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2af376640 1 --2- 172.21.15.46:0/4116268414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff294004c10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff298009a90 tx=0x7ff294005830 comp rx=0 tx=0).stop 2024-05-09T03:10:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2ad913640 1 -- 172.21.15.46:0/4116268414 reap_dead start 2024-05-09T03:10:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2af376640 1 -- 172.21.15.46:0/4116268414 shutdown_connections 2024-05-09T03:10:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2af376640 1 -- 172.21.15.46:0/4116268414 >> 172.21.15.46:0/4116268414 conn(0x7ff2a80743e0 msgr2=0x7ff2a80d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2af376640 1 -- 172.21.15.46:0/4116268414 shutdown_connections 2024-05-09T03:10:35.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.523+0000 7ff2af376640 1 -- 172.21.15.46:0/4116268414 wait complete. 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.525+0000 7ff2af376640 1 Processor -- start 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.525+0000 7ff2af376640 1 -- start start 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.526+0000 7ff2af376640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff2a8161550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2af376640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a8161a90 0x7ff2a80b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2af376640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8159ce0 0x7ff2a80b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2af376640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a8160a20 con 0x7ff294004800 2024-05-09T03:10:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2af376640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff2a80d5280 con 0x7ff2a8161a90 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2af376640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff2a80d15b0 con 0x7ff2a8159ce0 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff2a8161550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff2a8161550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53276/0 (socket says 172.21.15.46:53276) 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 -- 172.21.15.46:0/240568133 learned_addr learned my addr 172.21.15.46:0/240568133 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ac911640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a8161a90 0x7ff2a80b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad913640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8159ce0 0x7ff2a80b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 -- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8159ce0 msgr2=0x7ff2a80b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8159ce0 0x7ff2a80b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 -- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a8161a90 msgr2=0x7ff2a80b6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a8161a90 0x7ff2a80b6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 -- 172.21.15.46:0/240568133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a80b7950 con 0x7ff294004800 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2ad112640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff2a8161550 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4007340 tx=0x7ff2a4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a401a070 con 0x7ff294004800 2024-05-09T03:10:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2a400f9c0 con 0x7ff294004800 2024-05-09T03:10:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a400fd90 con 0x7ff294004800 2024-05-09T03:10:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.527+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a81f6bd0 con 0x7ff294004800 2024-05-09T03:10:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.528+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2a81f7190 con 0x7ff294004800 2024-05-09T03:10:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.528+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2a806a270 con 0x7ff294004800 2024-05-09T03:10:35.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.529+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff2a401d070 con 0x7ff294004800 2024-05-09T03:10:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.530+0000 7ff29dffb640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff28c076300 0x7ff28c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.530+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7ff2a4097250 con 0x7ff294004800 2024-05-09T03:10:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.530+0000 7ff2ac911640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff28c076300 0x7ff28c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.530+0000 7ff2ac911640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff28c076300 0x7ff28c078810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff298033000 tx=0x7ff298008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:35.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.534+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff2a80b8130 con 0x7ff294004800 2024-05-09T03:10:35.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.763+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff2a80bb370 con 0x7ff294004800 2024-05-09T03:10:35.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.763+0000 7ff29dffb640 1 -- 172.21.15.46:0/240568133 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7ff2a4060a20 con 0x7ff294004800 2024-05-09T03:10:35.765 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:10:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff28c076300 msgr2=0x7ff28c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff28c076300 0x7ff28c078810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff298033000 tx=0x7ff298008030 comp rx=0 tx=0).stop 2024-05-09T03:10:35.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 msgr2=0x7ff2a8161550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff2a8161550 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4007340 tx=0x7ff2a4007800 comp rx=0 tx=0).stop 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 shutdown_connections 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2a8159ce0 0x7ff2a80b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a8161a90 0x7ff2a80b6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff28c076300 0x7ff28c078810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 --2- 172.21.15.46:0/240568133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294004800 0x7ff2a8161550 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 >> 172.21.15.46:0/240568133 conn(0x7ff2a80743e0 msgr2=0x7ff2a815dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 shutdown_connections 2024-05-09T03:10:35.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:35.765+0000 7ff2af376640 1 -- 172.21.15.46:0/240568133 wait complete. 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:10:35.778 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:10:35.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:35.779 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:10:35.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:10:35.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:35.796 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42482sec preferred_lft 42482sec 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:10:35.797 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:10:35.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:35.799 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:10:35.892 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:10:35.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:35.894 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:10:35.951 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:10:35.952 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:10:35.952 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:10:35.952 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:10:35.952 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:10:35.952 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:10:35.952 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:10:35.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:10:35.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:10:35.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:10:35.953 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:10:36.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:10:36.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:10:36.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.082 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:10:36.098 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:10:36.099 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:10:36.099 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:10:36.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:10:36.165 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:10:36.165 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:10:36.166 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:36.221 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:10:36.221 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:10:36.222 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:36.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:10:36.344 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:10:36.401 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:10:36.401 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:10:36.401 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:10:36.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:10:36.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:10:36.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:36.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:36.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:36.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:36.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.404 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:10:36.457 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:10:36.458 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:10:36.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:10:36.525 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:10:36.583 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:10:36.583 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:10:36.583 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:10:36.584 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:36.585 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:10:36.586 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:36.586 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:10:36.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.586 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:10:36.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:10:36.594+0000 7f235a435480 -1 init, newargv = 0x55fe07639780 newargc=15 2024-05-09T03:10:36.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[47664]: starting ceph client 2024-05-09T03:10:36.600 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[47664]: starting fuse 2024-05-09T03:10:36.612 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:10:36.612 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:10:36.613 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:10:36.614 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:10:36.733 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.47664.asok 2024-05-09T03:10:36.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:10:36.744 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47664.asok 2024-05-09T03:10:36.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47664.asok status 2024-05-09T03:10:36.859 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:10:36.857+0000 7fe4db00a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:10:36.893 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "47664", 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5076, 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5076 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:10:36.894 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3581742119 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3581742119 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5076 192.168.0.1:0/3581742119", 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3581742119", 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 21, 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 32, 2024-05-09T03:10:36.895 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:10:36.896 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:10:36.896 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:10:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:36.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.902 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:10:36.902 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:36.918 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:10:36.918 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:36.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:36.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:36.985 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-05-09T03:10:37.045 INFO:teuthology.orchestra.run.smithi046.stdout:file_undamaged 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> import json 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(e.errno) 2024-05-09T03:10:37.046 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:37.047 DEBUG:teuthology.orchestra.run.smithi046:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-09T03:10:37.047 DEBUG:teuthology.orchestra.run.smithi046:> print(json.dumps( 2024-05-09T03:10:37.047 DEBUG:teuthology.orchestra.run.smithi046:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-09T03:10:37.047 DEBUG:teuthology.orchestra.run.smithi046:> indent=2)) 2024-05-09T03:10:37.047 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:10:37.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-05-09T03:10:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-09T03:10:37.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:10:37.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.319+0000 7fe804842640 1 Processor -- start 2024-05-09T03:10:37.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.319+0000 7fe804842640 1 -- start start 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe804842640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8000b92e0 0x7fe8000b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe804842640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8000d0480 0x7fe8000d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe804842640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe8000ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe804842640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe8000d0dd0 con 0x7fe8000d0480 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe804842640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe8000d2b80 con 0x7fe8000b92e0 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe804842640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe8000d2e60 con 0x7fe8000ce6a0 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe7f5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8000d0480 0x7fe8000d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe7f5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8000d0480 0x7fe8000d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44926/0 (socket says 172.21.15.46:44926) 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe7f5d9b640 1 -- 172.21.15.46:0/1287048573 learned_addr learned my addr 172.21.15.46:0/1287048573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe7fe59c640 1 --2- 172.21.15.46:0/1287048573 >> v2:172.21.15.88:3300/0 conn(0x7fe8000b92e0 0x7fe8000b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.320+0000 7fe7fed9d640 1 --2- 172.21.15.46:0/1287048573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe8000ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.321+0000 7fe7f5d9b640 1 -- 172.21.15.46:0/1287048573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 msgr2=0x7fe8000ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.321+0000 7fe7f5d9b640 1 --2- 172.21.15.46:0/1287048573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe8000ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.321+0000 7fe7f5d9b640 1 -- 172.21.15.46:0/1287048573 >> v2:172.21.15.88:3300/0 conn(0x7fe8000b92e0 msgr2=0x7fe8000b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.321+0000 7fe7f5d9b640 1 --2- 172.21.15.46:0/1287048573 >> v2:172.21.15.88:3300/0 conn(0x7fe8000b92e0 0x7fe8000b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.321+0000 7fe7f5d9b640 1 -- 172.21.15.46:0/1287048573 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8000d3140 con 0x7fe8000d0480 2024-05-09T03:10:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.321+0000 7fe7f5d9b640 1 --2- 172.21.15.46:0/1287048573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe8000d0890 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe80006d120 tx=0x7fe7e802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b19b394fc548c1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe7fdd9b640 1 -- 172.21.15.46:0/1287048573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8037070 con 0x7fe8000d0480 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe7fdd9b640 1 -- 172.21.15.46:0/1287048573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e8032440 con 0x7fe8000d0480 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe7fdd9b640 1 -- 172.21.15.46:0/1287048573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8032810 con 0x7fe8000d0480 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 -- 172.21.15.46:0/1287048573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 msgr2=0x7fe8000d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 --2- 172.21.15.46:0/1287048573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe8000d0890 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe80006d120 tx=0x7fe7e802a140 comp rx=0 tx=0).stop 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 -- 172.21.15.46:0/1287048573 shutdown_connections 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 --2- 172.21.15.46:0/1287048573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe8000ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 --2- 172.21.15.46:0/1287048573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe8000d0890 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 --2- 172.21.15.46:0/1287048573 >> v2:172.21.15.88:3300/0 conn(0x7fe8000b92e0 0x7fe8000b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 -- 172.21.15.46:0/1287048573 >> 172.21.15.46:0/1287048573 conn(0x7fe8000d3c00 msgr2=0x7fe8000d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 -- 172.21.15.46:0/1287048573 shutdown_connections 2024-05-09T03:10:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.322+0000 7fe804842640 1 -- 172.21.15.46:0/1287048573 wait complete. 2024-05-09T03:10:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.324+0000 7fe804842640 1 Processor -- start 2024-05-09T03:10:37.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.325+0000 7fe804842640 1 -- start start 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe804842640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000b92e0 0x7fe8001600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe804842640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe80006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe804842640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe80006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe804842640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000cf170 con 0x7fe8000d0480 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe804842640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000ceff0 con 0x7fe8000b92e0 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe804842640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe8000cf2f0 con 0x7fe8000ce6a0 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe80006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe80006eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44942/0 (socket says 172.21.15.46:44942) 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 -- 172.21.15.46:0/223012386 learned_addr learned my addr 172.21.15.46:0/223012386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7f5d9b640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe80006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fe59c640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000b92e0 0x7fe8001600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 -- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 msgr2=0x7fe80006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe80006e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 -- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000b92e0 msgr2=0x7fe8001600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000b92e0 0x7fe8001600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.326+0000 7fe7fed9d640 1 -- 172.21.15.46:0/223012386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8001565b0 con 0x7fe8000d0480 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.327+0000 7fe7fed9d640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe80006eb60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800ef90 tx=0x7fe7f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.327+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8018020 con 0x7fe8000d0480 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.327+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f8014480 con 0x7fe8000d0480 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.327+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe800160c30 con 0x7fe8000d0480 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.327+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8009e60 con 0x7fe8000d0480 2024-05-09T03:10:37.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.327+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8001611f0 con 0x7fe8000d0480 2024-05-09T03:10:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.328+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8001609a0 con 0x7fe8000d0480 2024-05-09T03:10:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.329+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe7f8005d10 con 0x7fe8000d0480 2024-05-09T03:10:37.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.329+0000 7fe7f77fe640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7e00762d0 0x7fe7e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.329+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fe7f8098250 con 0x7fe8000d0480 2024-05-09T03:10:37.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.329+0000 7fe7fe59c640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7e00762d0 0x7fe7e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.330+0000 7fe7fe59c640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7e00762d0 0x7fe7e00787e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe800071390 tx=0x7fe7f0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.334+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe7f8022030 con 0x7fe8000d0480 2024-05-09T03:10:37.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.561+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe80006a270 con 0x7fe8000d0480 2024-05-09T03:10:37.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.561+0000 7fe7f77fe640 1 -- 172.21.15.46:0/223012386 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7fe7f8010040 con 0x7fe8000d0480 2024-05-09T03:10:37.563 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:37.563 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:10:37.563 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:10:37.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7e00762d0 msgr2=0x7fe7e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7e00762d0 0x7fe7e00787e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe800071390 tx=0x7fe7f0009190 comp rx=0 tx=0).stop 2024-05-09T03:10:37.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 msgr2=0x7fe80006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe80006eb60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800ef90 tx=0x7fe7f800c490 comp rx=0 tx=0).stop 2024-05-09T03:10:37.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 shutdown_connections 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7e00762d0 0x7fe7e00787e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000d0480 0x7fe80006eb60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000ce6a0 0x7fe80006e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 --2- 172.21.15.46:0/223012386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000b92e0 0x7fe8001600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 >> 172.21.15.46:0/223012386 conn(0x7fe8000d3c00 msgr2=0x7fe800072d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 shutdown_connections 2024-05-09T03:10:37.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.563+0000 7fe804842640 1 -- 172.21.15.46:0/223012386 wait complete. 2024-05-09T03:10:37.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-05-09T03:10:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.691+0000 7fc5c0ad9640 1 Processor -- start 2024-05-09T03:10:37.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.691+0000 7fc5c0ad9640 1 -- start start 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.692+0000 7fc5c0ad9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5bc0b52b0 0x7fc5bc0b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5c0ad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5bc0b7c80 0x7fc5bc0b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5c0ad9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5bc0b8090 con 0x7fc5bc164bf0 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5c0ad9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc5bc164760 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5c0ad9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc5bc0d17a0 con 0x7fc5bc0b52b0 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5bad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5bc0b7c80 0x7fc5bc0b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5ba59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc5bc164bf0 0x7fc5bc0d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56844/0 (socket says 172.21.15.46:56844) 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5ba59c640 1 -- 172.21.15.46:0/75534230 learned_addr learned my addr 172.21.15.46:0/75534230 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/75534230 >> v2:172.21.15.88:3300/0 conn(0x7fc5bc0b52b0 0x7fc5bc0b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5bad9d640 1 -- 172.21.15.46:0/75534230 >> v2:172.21.15.88:3300/0 conn(0x7fc5bc0b52b0 msgr2=0x7fc5bc0b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/75534230 >> v2:172.21.15.88:3300/0 conn(0x7fc5bc0b52b0 0x7fc5bc0b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5bad9d640 1 -- 172.21.15.46:0/75534230 >> v1:172.21.15.46:6789/0 conn(0x7fc5bc164bf0 legacy=0x7fc5bc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5bad9d640 1 -- 172.21.15.46:0/75534230 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc0d14c0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5b959a640 1 -- 172.21.15.46:0/75534230 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055417420 0 0) 0x7fc5ac001110 con 0x7fc5bc164bf0 2024-05-09T03:10:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.693+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/75534230 >> v2:172.21.15.88:3300/0 conn(0x7fc5bc0b52b0 0x7fc5bc0b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.694+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/75534230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0b9190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0d57d0 tx=0x7fc5b402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c276a840ee68fc1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.694+0000 7fc5b959a640 1 -- 172.21.15.46:0/75534230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4007740 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.694+0000 7fc5b959a640 1 -- 172.21.15.46:0/75534230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b40078e0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.694+0000 7fc5b959a640 1 -- 172.21.15.46:0/75534230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4007cb0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/75534230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 msgr2=0x7fc5bc0b9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/75534230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0b9190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0d57d0 tx=0x7fc5b402a680 comp rx=0 tx=0).stop 2024-05-09T03:10:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/75534230 shutdown_connections 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/75534230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0b9190 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/75534230 >> v2:172.21.15.88:3300/0 conn(0x7fc5bc0b52b0 0x7fc5bc0b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/75534230 >> 172.21.15.46:0/75534230 conn(0x7fc5bc0d2920 msgr2=0x7fc5bc0d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/75534230 shutdown_connections 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.695+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/75534230 wait complete. 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.697+0000 7fc5c0ad9640 1 Processor -- start 2024-05-09T03:10:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.697+0000 7fc5c0ad9640 1 -- start start 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5c0ad9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5bc0b52b0 0x7fc5bc162fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5c0ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5c0ad9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc164bf0 0x7fc5bc0cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5c0ad9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc0d1180 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5c0ad9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc1643f0 con 0x7fc5bc164bf0 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5c0ad9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc06eb30 con 0x7fc5bc0b52b0 2024-05-09T03:10:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44952/0 (socket says 172.21.15.46:44952) 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1104640165 learned_addr learned my addr 172.21.15.46:0/1104640165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc164bf0 0x7fc5bc0cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.698+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5bc0b52b0 0x7fc5bc162fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5bc0b52b0 msgr2=0x7fc5bc162fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5bc0b52b0 0x7fc5bc162fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc164bf0 msgr2=0x7fc5bc0cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc164bf0 0x7fc5bc0cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1104640165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc162530 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0ca540 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d7d0 tx=0x7fc5b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5aaffd640 1 -- 172.21.15.46:0/1104640165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0004030 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5aaffd640 1 -- 172.21.15.46:0/1104640165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b00041d0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5aaffd640 1 -- 172.21.15.46:0/1104640165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00096a0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5bc0cc610 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.699+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5bc1627d0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.701+0000 7fc5aaffd640 1 -- 172.21.15.46:0/1104640165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc5b0009840 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.702+0000 7fc5aaffd640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc59c0762d0 0x7fc59c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.702+0000 7fc5aaffd640 1 -- 172.21.15.46:0/1104640165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fc5b00977e0 con 0x7fc5bc0b7c80 2024-05-09T03:10:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.702+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc59c0762d0 0x7fc59c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.703+0000 7fc5c798d740 1 Processor -- start 2024-05-09T03:10:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.703+0000 7fc5c798d740 1 -- start start 2024-05-09T03:10:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.703+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc59c0762d0 0x7fc59c0787e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac007fd0 tx=0x7fc5ac007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019ab36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019a90f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a91450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5c798d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c0199f3d10 con 0x55c0196eb350 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c01997b750 con 0x55c0196ca670 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c0199efcd0 con 0x55c0196c2490 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019ab36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019ab36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44956/0 (socket says 172.21.15.46:44956) 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 -- 172.21.15.46:0/311917768 learned_addr learned my addr 172.21.15.46:0/311917768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019a90f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a91450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 -- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 msgr2=0x55c019a90f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019a90f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 -- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 msgr2=0x55c019a91450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a91450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.704+0000 7fc5ba59c640 1 -- 172.21.15.46:0/311917768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0197cd790 con 0x55c0196eb350 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.705+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019ab36d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac004f90 tx=0x7fc5ac013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4195f898b3cc6c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.705+0000 7fc59bfff640 1 -- 172.21.15.46:0/311917768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac026070 con 0x55c0196eb350 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.705+0000 7fc59bfff640 1 -- 172.21.15.46:0/311917768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac013b80 con 0x55c0196eb350 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.705+0000 7fc59bfff640 1 -- 172.21.15.46:0/311917768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac012350 con 0x55c0196eb350 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 -- 172.21.15.46:0/311917768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 msgr2=0x55c019ab36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019ab36d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac004f90 tx=0x7fc5ac013000 comp rx=0 tx=0).stop 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 -- 172.21.15.46:0/311917768 shutdown_connections 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a91450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019a90f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 --2- 172.21.15.46:0/311917768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019ab36d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 -- 172.21.15.46:0/311917768 >> 172.21.15.46:0/311917768 conn(0x55c0196df4a0 msgr2=0x55c019a8e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 -- 172.21.15.46:0/311917768 shutdown_connections 2024-05-09T03:10:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.706+0000 7fc5c798d740 1 -- 172.21.15.46:0/311917768 wait complete. 2024-05-09T03:10:37.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.707+0000 7fc5c798d740 1 -- start start 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.707+0000 7fc5c798d740 1 Processor -- start 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 0x55c019ae5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196ca670 0x55c019ab0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196eb350 0x55c019ad4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c0199b0bd0 con 0x55c0196c2490 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c019ae64d0 con 0x55c0196eb350 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c019ad2160 con 0x55c0196ca670 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5c798d740 10 client.0 _ll_register_callbacks cb 0x55c0199a5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5ba59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 0x55c019ae5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5ba59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 0x55c019ae5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44968/0 (socket says 172.21.15.46:44968) 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.708+0000 7fc5ba59c640 1 -- 172.21.15.46:0/3838729511 learned_addr learned my addr 172.21.15.46:0/3838729511 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196eb350 0x55c019ad4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196ca670 0x55c019ab0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5ba59c640 1 -- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196ca670 msgr2=0x55c019ab0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196ca670 0x55c019ab0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5ba59c640 1 -- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196eb350 msgr2=0x55c019ad4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196eb350 0x55c019ad4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5ba59c640 1 -- 172.21.15.46:0/3838729511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c01969c130 con 0x55c0196c2490 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 0x55c019ae5e40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac007450 tx=0x7fc5ac000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ab8a5ed375b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5997fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5997fa640 1 -- 172.21.15.46:0/3838729511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac031060 con 0x55c0196c2490 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5997fa640 1 -- 172.21.15.46:0/3838729511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fc5ac021bb0 con 0x55c0196c2490 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5ac013b80 con 0x55c0196c2490 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5997fa640 1 -- 172.21.15.46:0/3838729511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc5ac037050 con 0x55c0196c2490 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5c798d740 10 client.5097 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:37.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x55c0197cd5c0 con 0x55c0196c2490 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.709+0000 7fc5997fa640 1 -- 172.21.15.46:0/3838729511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac013b80 con 0x55c0196c2490 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5997fa640 1 -- 172.21.15.46:0/3838729511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7fc5ac03a020 con 0x55c0196c2490 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5c798d740 10 client.5097 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5c798d740 10 client.5097 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5c798d740 20 client.5097 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c0196aa130 0x55c019ad0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5c798d740 4 client.5097 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c019a01920 con 0x55c0196aa130 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c0196aa130 0x55c019ad0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5997fa640 10 client.5097 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.710+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c0196aa130 0x55c019ad0510 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3bb1ea7f4ca3e3c server_cookie=2e9c2f856577c033 in_seq=0 out_seq=0 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5997fa640 1 -- 172.21.15.46:0/3838729511 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c019a01920 con 0x55c0196aa130 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5997fa640 10 client.5097 handle_command_reply: tid=0 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 client.5097 shutdown 2024-05-09T03:10:37.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 msgr2=0x55c019ae5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 0x55c019ae5e40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac007450 tx=0x7fc5ac000ed0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 shutdown_connections 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196eb350 0x55c019ad4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196ca670 0x55c019ab0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c0196aa130 0x55c019ad0510 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3838729511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196c2490 0x55c019ae5e40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 >> 172.21.15.46:0/3838729511 conn(0x55c0196df4a0 msgr2=0x55c019a8a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5997fa640 0 client.5097 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 shutdown_connections 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 1 -- 172.21.15.46:0/3838729511 wait complete. 2024-05-09T03:10:37.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.711+0000 7fc5c798d740 20 client.5097 trim_cache size 0 max 16384 2024-05-09T03:10:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.725+0000 7fc5c798d740 1 Processor -- start 2024-05-09T03:10:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.725+0000 7fc5c798d740 1 -- start start 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196c2490 0x55c019ae5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 0x55c019ab8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196eb350 0x55c019aef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5c798d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc202040 con 0x55c0196ca670 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc1fa8b0 con 0x55c0196c2490 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc15bf00 con 0x55c0196eb350 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 0x55c019ab8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 0x55c019ab8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44972/0 (socket says 172.21.15.46:44972) 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1528839533 learned_addr learned my addr 172.21.15.46:0/1528839533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196eb350 0x55c019aef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196c2490 0x55c019ae5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196eb350 msgr2=0x55c019aef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196eb350 0x55c019aef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196c2490 msgr2=0x55c019ae5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196c2490 0x55c019ae5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 -- 172.21.15.46:0/1528839533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0196cf150 con 0x55c0196ca670 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196eb350 0x55c019aef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196c2490 0x55c019ae5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 0x55c019ab8520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0060780 tx=0x7fc5b005d200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b98e88719363de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc59b7fe640 1 -- 172.21.15.46:0/1528839533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0064d80 con 0x55c0196ca670 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc59b7fe640 1 -- 172.21.15.46:0/1528839533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b0094070 con 0x55c0196ca670 2024-05-09T03:10:37.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.727+0000 7fc59b7fe640 1 -- 172.21.15.46:0/1528839533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00697a0 con 0x55c0196ca670 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 -- 172.21.15.46:0/1528839533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 msgr2=0x55c019ab8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 0x55c019ab8520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0060780 tx=0x7fc5b005d200 comp rx=0 tx=0).stop 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 -- 172.21.15.46:0/1528839533 shutdown_connections 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196eb350 0x55c019aef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196ca670 0x55c019ab8520 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 --2- 172.21.15.46:0/1528839533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196c2490 0x55c019ae5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 -- 172.21.15.46:0/1528839533 >> 172.21.15.46:0/1528839533 conn(0x55c0196df4a0 msgr2=0x55c019a8a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 -- 172.21.15.46:0/1528839533 shutdown_connections 2024-05-09T03:10:37.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.728+0000 7fc5c798d740 1 -- 172.21.15.46:0/1528839533 wait complete. 2024-05-09T03:10:37.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.729+0000 7fc5c798d740 1 -- start start 2024-05-09T03:10:37.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.729+0000 7fc5c798d740 1 Processor -- start 2024-05-09T03:10:37.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.730+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019b52df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.730+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a90bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.730+0000 7fc5c798d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019a910f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.730+0000 7fc5c798d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c019aaefe0 con 0x55c0196eb350 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.730+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c019ab1440 con 0x55c0196ca670 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.730+0000 7fc5c798d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c019ab00e0 con 0x55c0196c2490 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5c798d740 10 client.0 _ll_register_callbacks cb 0x7fc5ac024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019a910f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019a910f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44984/0 (socket says 172.21.15.46:44984) 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 -- 172.21.15.46:0/3398352742 learned_addr learned my addr 172.21.15.46:0/3398352742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019b52df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5b9d9b640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a90bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 -- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 msgr2=0x55c019b52df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019b52df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 -- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 msgr2=0x55c019a90bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:37.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a90bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 -- 172.21.15.46:0/3398352742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c019a8ea70 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5bad9d640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019a910f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009e50 tx=0x7fc5b4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4779dd4e05b6b78 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5997fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5997fa640 1 -- 172.21.15.46:0/3398352742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4004730 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b0094070 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.731+0000 7fc5997fa640 1 -- 172.21.15.46:0/3398352742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fc5b4007540 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5997fa640 1 -- 172.21.15.46:0/3398352742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fc5b4050050 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 10 client.5103 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x55c019ab0e20 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5997fa640 1 -- 172.21.15.46:0/3398352742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b0094070 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5997fa640 1 -- 172.21.15.46:0/3398352742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7fc5b4053020 con 0x55c0196eb350 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 10 client.5103 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 10 client.5103 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:10:37.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 20 client.5103 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c019ad0ed0 0x55c019ad5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 4 client.5103 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55c0199fb790 con 0x55c019ad0ed0 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c019ad0ed0 0x55c019ad5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.732+0000 7fc5997fa640 10 client.5103 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5ba59c640 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c019ad0ed0 0x55c019ad5bf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a60dbb93070d36d server_cookie=feb02cdcc5e5e644 in_seq=0 out_seq=0 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5997fa640 1 -- 172.21.15.46:0/3398352742 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+236 (crc 0 0 0) 0x55c0199fb790 con 0x55c019ad0ed0 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5997fa640 10 client.5103 handle_command_reply: tid=0 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 client.5103 shutdown 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 msgr2=0x55c019a910f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019a910f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009e50 tx=0x7fc5b4004ab0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 shutdown_connections 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55c019ad0ed0 0x55c019ad5bf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0196eb350 0x55c019a910f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c0196ca670 0x55c019a90bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 --2- 172.21.15.46:0/3398352742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c0196c2490 0x55c019b52df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5997fa640 0 client.5103 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 >> 172.21.15.46:0/3398352742 conn(0x55c0196df4a0 msgr2=0x55c019ad2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 shutdown_connections 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 1 -- 172.21.15.46:0/3398352742 wait complete. 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.733+0000 7fc5c798d740 20 client.5103 trim_cache size 0 max 16384 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:10:37.740 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "dentry", 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 548358141, 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511627776, 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: "frag": "*", 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: "dname": "file_to_be_damaged", 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: "snap_id": "head", 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/subdir/file_to_be_damaged" 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:37.741 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc59c0762d0 msgr2=0x7fc59c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc59c0762d0 0x7fc59c0787e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac007fd0 tx=0x7fc5ac007de0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 msgr2=0x7fc5bc0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0ca540 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000d7d0 tx=0x7fc5b000dc90 comp rx=0 tx=0).stop 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 shutdown_connections 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc164bf0 0x7fc5bc0cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc59c0762d0 0x7fc59c0787e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5bc0b7c80 0x7fc5bc0ca540 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 --2- 172.21.15.46:0/1104640165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5bc0b52b0 0x7fc5bc162fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:37.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 >> 172.21.15.46:0/1104640165 conn(0x7fc5bc0d2920 msgr2=0x7fc5bc0ba0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:37.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 shutdown_connections 2024-05-09T03:10:37.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37.734+0000 7fc5c0ad9640 1 -- 172.21.15.46:0/1104640165 wait complete. 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> import errno 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> f.close() 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> except IOError as e: 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(errno.EIO) 2024-05-09T03:10:37.747 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:10:37.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-05-09T03:10:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> import json 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(e.errno) 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> print(json.dumps( 2024-05-09T03:10:37.825 DEBUG:teuthology.orchestra.run.smithi046:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-09T03:10:37.826 DEBUG:teuthology.orchestra.run.smithi046:> indent=2)) 2024-05-09T03:10:37.826 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:10:37.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:37 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-05-09T03:10:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-05-09T03:10:37.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:37.945 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-05-09T03:10:38.003 INFO:teuthology.orchestra.run.smithi046.stdout:2 2024-05-09T03:10:38.004 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:10:38.004 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:10:38.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:10:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.216+0000 7f734b10d640 1 Processor -- start 2024-05-09T03:10:38.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.216+0000 7f734b10d640 1 -- start start 2024-05-09T03:10:38.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f734b10d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7344163eb0 0x7f73441642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f734b10d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73440d2a70 0x7f73440d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f734b10d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73440d33c0 con 0x7f73440bb770 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f734b10d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f73440d4bb0 con 0x7f73440d2a70 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f734b10d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7344164900 con 0x7f7344163eb0 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f73496aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73440d2a70 0x7f73440d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f7348ea9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73440bb770 0x7f73440d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56850/0 (socket says 172.21.15.46:56850) 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f73496aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73440d2a70 0x7f73440d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44988/0 (socket says 172.21.15.46:44988) 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.217+0000 7f7348ea9640 1 -- 172.21.15.46:0/2837522224 learned_addr learned my addr 172.21.15.46:0/2837522224 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.218+0000 7f7343fff640 1 --2- 172.21.15.46:0/2837522224 >> v2:172.21.15.88:3300/0 conn(0x7f7344163eb0 0x7f73441642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.218+0000 7f73496aa640 1 -- 172.21.15.46:0/2837522224 >> v2:172.21.15.88:3300/0 conn(0x7f7344163eb0 msgr2=0x7f73441642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.218+0000 7f73496aa640 1 --2- 172.21.15.46:0/2837522224 >> v2:172.21.15.88:3300/0 conn(0x7f7344163eb0 0x7f73441642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.218+0000 7f73496aa640 1 -- 172.21.15.46:0/2837522224 >> v1:172.21.15.46:6789/0 conn(0x7f73440bb770 legacy=0x7f73440d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.218+0000 7f73437fe640 1 -- 172.21.15.46:0/2837522224 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935359568 0 0) 0x7f7338001110 con 0x7f73440bb770 2024-05-09T03:10:38.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.218+0000 7f73496aa640 1 -- 172.21.15.46:0/2837522224 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7344164be0 con 0x7f73440d2a70 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f73496aa640 1 --2- 172.21.15.46:0/2837522224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73440d2a70 0x7f73440d2e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f73440d6d00 tx=0x7f733c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3594e72e2f585934 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f73437fe640 1 -- 172.21.15.46:0/2837522224 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c007740 con 0x7f73440d2a70 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f73437fe640 1 -- 172.21.15.46:0/2837522224 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733c0078e0 con 0x7f73440d2a70 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f73437fe640 1 -- 172.21.15.46:0/2837522224 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c007cb0 con 0x7f73440d2a70 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f734b10d640 1 -- 172.21.15.46:0/2837522224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73440d2a70 msgr2=0x7f73440d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f734b10d640 1 --2- 172.21.15.46:0/2837522224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73440d2a70 0x7f73440d2e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f73440d6d00 tx=0x7f733c02a680 comp rx=0 tx=0).stop 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f734b10d640 1 -- 172.21.15.46:0/2837522224 shutdown_connections 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f734b10d640 1 --2- 172.21.15.46:0/2837522224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73440d2a70 0x7f73440d2e80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f734b10d640 1 --2- 172.21.15.46:0/2837522224 >> v2:172.21.15.88:3300/0 conn(0x7f7344163eb0 0x7f73441642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.219+0000 7f734b10d640 1 -- 172.21.15.46:0/2837522224 >> 172.21.15.46:0/2837522224 conn(0x7f734406e370 msgr2=0x7f734406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:38.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.220+0000 7f734b10d640 1 -- 172.21.15.46:0/2837522224 shutdown_connections 2024-05-09T03:10:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.220+0000 7f734b10d640 1 -- 172.21.15.46:0/2837522224 wait complete. 2024-05-09T03:10:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.221+0000 7f734b10d640 1 Processor -- start 2024-05-09T03:10:38.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.221+0000 7f734b10d640 1 -- start start 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f734b10d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73440bb770 0x7f73441ec250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f734b10d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73440d2a70 0x7f734414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f734b10d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 0x7f734414c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f734b10d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73440d0bc0 con 0x7f7344163eb0 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f734b10d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73440bbb80 con 0x7f73440d2a70 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f734b10d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f73440d0d40 con 0x7f73440bb770 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 0x7f734414c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 0x7f734414c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45002/0 (socket says 172.21.15.46:45002) 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 -- 172.21.15.46:0/2221080662 learned_addr learned my addr 172.21.15.46:0/2221080662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f7343fff640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73440d2a70 0x7f734414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f7348ea9640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73440bb770 0x7f73441ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 -- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73440bb770 msgr2=0x7f73441ec250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73440bb770 0x7f73441ec250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 -- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73440d2a70 msgr2=0x7f734414b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73440d2a70 0x7f734414b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f73496aa640 1 -- 172.21.15.46:0/2221080662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73441eacb0 con 0x7f7344163eb0 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.223+0000 7f7348ea9640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73440bb770 0x7f73441ec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.224+0000 7f73496aa640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 0x7f734414c320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f733c009e50 tx=0x7f733c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.224+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c03f070 con 0x7f7344163eb0 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.224+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733c031bf0 con 0x7f7344163eb0 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.224+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73441eb1b0 con 0x7f7344163eb0 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.224+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c044460 con 0x7f7344163eb0 2024-05-09T03:10:38.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.224+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73441eb770 con 0x7f7344163eb0 2024-05-09T03:10:38.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.225+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f734406a270 con 0x7f7344163eb0 2024-05-09T03:10:38.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.226+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f733c03b410 con 0x7f7344163eb0 2024-05-09T03:10:38.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.226+0000 7f73417fa640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73280762d0 0x7f73280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.226+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f733c0b8fe0 con 0x7f7344163eb0 2024-05-09T03:10:38.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.227+0000 7f7348ea9640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73280762d0 0x7f73280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.227+0000 7f7348ea9640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73280762d0 0x7f73280787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7338000f60 tx=0x7f7338007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:38.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.231+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f733c082850 con 0x7f7344163eb0 2024-05-09T03:10:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.433+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f73440d4130 con 0x7f7344163eb0 2024-05-09T03:10:38.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.434+0000 7f73417fa640 1 -- 172.21.15.46:0/2221080662 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v32) ==== 80+0+122 (secure 0 0 0) 0x7f733c082150 con 0x7f7344163eb0 2024-05-09T03:10:38.435 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3490266983 2024-05-10T03:10:29.845801+0000 2024-05-09T03:10:38.436 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3490266983 2024-05-10T03:10:29.845801+0000 2024-05-09T03:10:38.436 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73280762d0 msgr2=0x7f73280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73280762d0 0x7f73280787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7338000f60 tx=0x7f7338007ca0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 msgr2=0x7f734414c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 0x7f734414c320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f733c009e50 tx=0x7f733c004a40 comp rx=0 tx=0).stop 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 shutdown_connections 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73280762d0 0x7f73280787e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7344163eb0 0x7f734414c320 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73440d2a70 0x7f734414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 --2- 172.21.15.46:0/2221080662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73440bb770 0x7f73441ec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 >> 172.21.15.46:0/2221080662 conn(0x7f734406e370 msgr2=0x7f73440b9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 shutdown_connections 2024-05-09T03:10:38.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.436+0000 7f734b10d640 1 -- 172.21.15.46:0/2221080662 wait complete. 2024-05-09T03:10:38.448 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:10:38.449 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:10:38.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:10:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.619+0000 7fce5a60c640 1 Processor -- start 2024-05-09T03:10:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.619+0000 7fce5a60c640 1 -- start start 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce5a60c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce540b7e70 0x7fce540d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce5a60c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 0x7fce540b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce5a60c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce540b5550 con 0x7fce540b5140 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce5a60c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fce540d3480 con 0x7fce540b7e70 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce5a60c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fce540d44e0 con 0x7fce540d3df0 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce53fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce540b5140 0x7fce540b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56856/0 (socket says 172.21.15.46:56856) 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce53fff640 1 -- 172.21.15.46:0/3977913381 learned_addr learned my addr 172.21.15.46:0/3977913381 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce58ba9640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 0x7fce540b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3977913381 >> v2:172.21.15.88:3300/0 conn(0x7fce540b7e70 0x7fce540d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2868325116 0 0) 0x7fce540b5550 con 0x7fce540b5140 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.621+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce40003520 con 0x7fce540b5140 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2889094943 0 0) 0x7fce40003520 con 0x7fce540b5140 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 msgr2=0x7fce540b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 0x7fce540b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 >> v2:172.21.15.88:3300/0 conn(0x7fce540b7e70 msgr2=0x7fce540d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 --2- 172.21.15.46:0/3977913381 >> v2:172.21.15.88:3300/0 conn(0x7fce540b7e70 0x7fce540d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce540d47c0 con 0x7fce540b5140 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fce440016f0 con 0x7fce540b5140 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 >> v1:172.21.15.46:6789/0 conn(0x7fce540b5140 legacy=0x7fce540b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce40003a00 0x7fce40003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce40004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 0x7fce40005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fce540d44e0 con 0x7fce40004800 2024-05-09T03:10:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fce4000ba70 con 0x7fce40003a00 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fce4000bc70 con 0x7fce540d3df0 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce40004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce53fff640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce40003a00 0x7fce40003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce58ba9640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 0x7fce40005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce4bfff640 1 -- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 msgr2=0x7fce40005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce540d3df0 0x7fce40005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce4bfff640 1 -- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce40003a00 msgr2=0x7fce40003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce40003a00 0x7fce40003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.622+0000 7fce4bfff640 1 -- 172.21.15.46:0/3977913381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce40003520 con 0x7fce40004800 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.623+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce40004c10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ac0 tx=0x7fce40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91baab1625841762 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.623+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c036070 con 0x7fce40004800 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.623+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c031400 con 0x7fce40004800 2024-05-09T03:10:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.623+0000 7fce537fe640 1 -- 172.21.15.46:0/3977913381 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c009690 con 0x7fce40004800 2024-05-09T03:10:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce5a60c640 1 -- 172.21.15.46:0/3977913381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 msgr2=0x7fce40004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3977913381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce40004c10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ac0 tx=0x7fce40005830 comp rx=0 tx=0).stop 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce58ba9640 1 -- 172.21.15.46:0/3977913381 reap_dead start 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce5a60c640 1 -- 172.21.15.46:0/3977913381 shutdown_connections 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce5a60c640 1 -- 172.21.15.46:0/3977913381 >> 172.21.15.46:0/3977913381 conn(0x7fce5408d3f0 msgr2=0x7fce5408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce5a60c640 1 -- 172.21.15.46:0/3977913381 shutdown_connections 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.624+0000 7fce5a60c640 1 -- 172.21.15.46:0/3977913381 wait complete. 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.626+0000 7fce5a60c640 1 Processor -- start 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.626+0000 7fce5a60c640 1 -- start start 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce5a60c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce54152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce5a60c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce5414dc60 0x7fce5414e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce5a60c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce54158980 0x7fce54158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce5a60c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fce540d4200 con 0x7fce40004800 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce5a60c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fce540d4d80 con 0x7fce54158980 2024-05-09T03:10:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce5a60c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fce540d5580 con 0x7fce5414dc60 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce54152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce54152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45010/0 (socket says 172.21.15.46:45010) 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 -- 172.21.15.46:0/3538243451 learned_addr learned my addr 172.21.15.46:0/3538243451 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce5414dc60 0x7fce5414e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce58ba9640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce54158980 0x7fce54158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 -- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce5414dc60 msgr2=0x7fce5414e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce5414dc60 0x7fce5414e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 -- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce54158980 msgr2=0x7fce54158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce54158980 0x7fce54158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce53fff640 1 -- 172.21.15.46:0/3538243451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce541f6260 con 0x7fce40004800 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.627+0000 7fce58ba9640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce54158980 0x7fce54158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.628+0000 7fce53fff640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce54152900 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fce44004ef0 tx=0x7fce44007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.628+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4400f7b0 con 0x7fce40004800 2024-05-09T03:10:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.628+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce44018070 con 0x7fce40004800 2024-05-09T03:10:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.628+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4400fb50 con 0x7fce40004800 2024-05-09T03:10:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.628+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce541f6540 con 0x7fce40004800 2024-05-09T03:10:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.628+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce541623b0 con 0x7fce40004800 2024-05-09T03:10:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.629+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce541f6c50 con 0x7fce40004800 2024-05-09T03:10:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.630+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fce4400fcf0 con 0x7fce40004800 2024-05-09T03:10:38.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.630+0000 7fce517fa640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce340762d0 0x7fce340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:38.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.630+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fce44097610 con 0x7fce40004800 2024-05-09T03:10:38.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.630+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce340762d0 0x7fce340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:38.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.631+0000 7fce4bfff640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce340762d0 0x7fce340787e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ac0 tx=0x7fce3c008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.635+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fce44060f10 con 0x7fce40004800 2024-05-09T03:10:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.841+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fce5406a270 con 0x7fce40004800 2024-05-09T03:10:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.841+0000 7fce517fa640 1 -- 172.21.15.46:0/3538243451 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v32) ==== 80+0+122 (secure 0 0 0) 0x7fce44060810 con 0x7fce40004800 2024-05-09T03:10:38.843 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3490266983 2024-05-10T03:10:29.845801+0000 2024-05-09T03:10:38.843 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3490266983 2024-05-10T03:10:29.845801+0000 2024-05-09T03:10:38.843 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:10:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce340762d0 msgr2=0x7fce340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce340762d0 0x7fce340787e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ac0 tx=0x7fce3c008030 comp rx=0 tx=0).stop 2024-05-09T03:10:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 msgr2=0x7fce54152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:38.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce54152900 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fce44004ef0 tx=0x7fce44007430 comp rx=0 tx=0).stop 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 shutdown_connections 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce54158980 0x7fce54158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce5414dc60 0x7fce5414e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce340762d0 0x7fce340787e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 --2- 172.21.15.46:0/3538243451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce40004800 0x7fce54152900 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 >> 172.21.15.46:0/3538243451 conn(0x7fce5408d3f0 msgr2=0x7fce540d1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 shutdown_connections 2024-05-09T03:10:38.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:38.843+0000 7fce5a60c640 1 -- 172.21.15.46:0/3538243451 wait complete. 2024-05-09T03:10:38.856 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:10:38.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:38.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:10:38.881 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[47664]: fuse finished with error 0 and tester_r 0 2024-05-09T03:10:38.883 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:10:38.974 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:10:38.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:38.975 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:10:38.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:38.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub start /subdir repair 2024-05-09T03:10:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.148+0000 7f30c0e61640 1 Processor -- start 2024-05-09T03:10:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.148+0000 7f30c0e61640 1 -- start start 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30c0e61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f30bc0d2da0 0x7f30bc160480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30c0e61640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30bc0d4700 0x7f30bc0b9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30c0e61640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30bc0d4e00 con 0x7f30bc0d3f30 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30c0e61640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f30bc0d4340 con 0x7f30bc0d4700 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30c0e61640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f30bc0d50e0 con 0x7f30bc0d2da0 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30bad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30bc0d4700 0x7f30bc0b9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30ba59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f30bc0d3f30 0x7f30bc0d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56872/0 (socket says 172.21.15.46:56872) 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30ba59c640 1 -- 172.21.15.46:0/3760785376 learned_addr learned my addr 172.21.15.46:0/3760785376 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3760785376 >> v2:172.21.15.88:3300/0 conn(0x7f30bc0d2da0 0x7f30bc160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30bad9d640 1 -- 172.21.15.46:0/3760785376 >> v2:172.21.15.88:3300/0 conn(0x7f30bc0d2da0 msgr2=0x7f30bc160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30bad9d640 1 --2- 172.21.15.46:0/3760785376 >> v2:172.21.15.88:3300/0 conn(0x7f30bc0d2da0 0x7f30bc160480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30bad9d640 1 -- 172.21.15.46:0/3760785376 >> v1:172.21.15.46:6789/0 conn(0x7f30bc0d3f30 legacy=0x7f30bc0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30b959a640 1 -- 172.21.15.46:0/3760785376 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 180383542 0 0) 0x7f30bc0d4e00 con 0x7f30bc0d3f30 2024-05-09T03:10:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.150+0000 7f30bad9d640 1 -- 172.21.15.46:0/3760785376 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc0d53c0 con 0x7f30bc0d4700 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.151+0000 7f30bad9d640 1 --2- 172.21.15.46:0/3760785376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc0b9930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0d0280 tx=0x7f30b402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6164335bed405717 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.151+0000 7f30b959a640 1 -- 172.21.15.46:0/3760785376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4037070 con 0x7f30bc0d4700 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.151+0000 7f30b959a640 1 -- 172.21.15.46:0/3760785376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b4032440 con 0x7f30bc0d4700 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.151+0000 7f30b959a640 1 -- 172.21.15.46:0/3760785376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4032810 con 0x7f30bc0d4700 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 -- 172.21.15.46:0/3760785376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 msgr2=0x7f30bc0b9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3760785376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc0b9930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f30bc0d0280 tx=0x7f30b402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 -- 172.21.15.46:0/3760785376 shutdown_connections 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3760785376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc0b9930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3760785376 >> v2:172.21.15.88:3300/0 conn(0x7f30bc0d2da0 0x7f30bc160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 -- 172.21.15.46:0/3760785376 >> 172.21.15.46:0/3760785376 conn(0x7f30bc06e120 msgr2=0x7f30bc06e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 -- 172.21.15.46:0/3760785376 shutdown_connections 2024-05-09T03:10:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.152+0000 7f30c0e61640 1 -- 172.21.15.46:0/3760785376 wait complete. 2024-05-09T03:10:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.154+0000 7f30c0e61640 1 Processor -- start 2024-05-09T03:10:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.154+0000 7f30c0e61640 1 -- start start 2024-05-09T03:10:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30c0e61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30bc0d2da0 0x7f30bc0ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30c0e61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30bc0d3f30 0x7f30bc15d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30c0e61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc15daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30c0e61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc0d4b10 con 0x7f30bc0d4700 2024-05-09T03:10:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30c0e61640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc0b9410 con 0x7f30bc0d3f30 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30c0e61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f30bc0b9e70 con 0x7f30bc0d2da0 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc15daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc15daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45024/0 (socket says 172.21.15.46:45024) 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30bad9d640 1 -- 172.21.15.46:0/3311925363 learned_addr learned my addr 172.21.15.46:0/3311925363 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30bc0d3f30 0x7f30bc15d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30ba59c640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30bc0d2da0 0x7f30bc0ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.155+0000 7f30bad9d640 1 -- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30bc0d2da0 msgr2=0x7f30bc0ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30bad9d640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30bc0d2da0 0x7f30bc0ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30bad9d640 1 -- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30bc0d3f30 msgr2=0x7f30bc15d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30bad9d640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30bc0d3f30 0x7f30bc15d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30bad9d640 1 -- 172.21.15.46:0/3311925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc0acbf0 con 0x7f30bc0d4700 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30bc0d3f30 0x7f30bc15d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30bad9d640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc15daf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f30b4035040 tx=0x7f30b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30aaffd640 1 -- 172.21.15.46:0/3311925363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4037040 con 0x7f30bc0d4700 2024-05-09T03:10:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30aaffd640 1 -- 172.21.15.46:0/3311925363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b4004230 con 0x7f30bc0d4700 2024-05-09T03:10:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30bc0ad070 con 0x7f30bc0d4700 2024-05-09T03:10:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30aaffd640 1 -- 172.21.15.46:0/3311925363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4006e10 con 0x7f30bc0d4700 2024-05-09T03:10:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.156+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30bc0ad630 con 0x7f30bc0d4700 2024-05-09T03:10:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.158+0000 7f30aaffd640 1 -- 172.21.15.46:0/3311925363 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f30b4030770 con 0x7f30bc0d4700 2024-05-09T03:10:39.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.159+0000 7f30aaffd640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f309c0762d0 0x7f309c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.159+0000 7f30aaffd640 1 -- 172.21.15.46:0/3311925363 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f30b40b8860 con 0x7f30bc0d4700 2024-05-09T03:10:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.159+0000 7f30ba59c640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f309c0762d0 0x7f309c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.159+0000 7f30c7dfa740 1 Processor -- start 2024-05-09T03:10:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.159+0000 7f30ba59c640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f309c0762d0 0x7f309c0787e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30ac007ed0 tx=0x7f30ac007e60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.160+0000 7f30c7dfa740 1 -- start start 2024-05-09T03:10:39.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a9f5b0 0x558ae6e6e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e89260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e897a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558ae6da79e0 con 0x558ae6a766f0 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558ae6d61ac0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558ae6da39a0 con 0x558ae6a7e8d0 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e89260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e89260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45034/0 (socket says 172.21.15.46:45034) 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 -- 172.21.15.46:0/4036309296 learned_addr learned my addr 172.21.15.46:0/4036309296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30bad9d640 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e897a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30ba59c640 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a9f5b0 0x558ae6e6e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 -- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 msgr2=0x558ae6e897a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e897a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 -- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a9f5b0 msgr2=0x558ae6e6e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a9f5b0 0x558ae6e6e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.161+0000 7f30b9d9b640 1 -- 172.21.15.46:0/4036309296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae6b819f0 con 0x558ae6a766f0 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.162+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e89260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f30b000b8d0 tx=0x7f30b000bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e86ad61bdab7e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.162+0000 7f309bfff640 1 -- 172.21.15.46:0/4036309296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0018030 con 0x558ae6a766f0 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.162+0000 7f309bfff640 1 -- 172.21.15.46:0/4036309296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b0004020 con 0x558ae6a766f0 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.162+0000 7f309bfff640 1 -- 172.21.15.46:0/4036309296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b00043d0 con 0x558ae6a766f0 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.162+0000 7f30c7dfa740 1 -- 172.21.15.46:0/4036309296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 msgr2=0x558ae6e89260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.162+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e89260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f30b000b8d0 tx=0x7f30b000bd90 comp rx=0 tx=0).stop 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 -- 172.21.15.46:0/4036309296 shutdown_connections 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e897a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e89260 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/4036309296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a9f5b0 0x558ae6e6e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 -- 172.21.15.46:0/4036309296 >> 172.21.15.46:0/4036309296 conn(0x558ae6a93700 msgr2=0x558ae6e43b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 -- 172.21.15.46:0/4036309296 shutdown_connections 2024-05-09T03:10:39.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.163+0000 7f30c7dfa740 1 -- 172.21.15.46:0/4036309296 wait complete. 2024-05-09T03:10:39.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.164+0000 7f30c7dfa740 1 -- start start 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.164+0000 7f30c7dfa740 1 Processor -- start 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e9bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a7e8d0 0x558ae6e717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e5e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558ae6d2f380 con 0x558ae6a766f0 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558ae6e9c4e0 con 0x558ae6a7e8d0 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558ae6e5f420 con 0x558ae6a9f5b0 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30c7dfa740 10 client.0 _ll_register_callbacks cb 0x558ae6d501b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:39.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30ba59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e9bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30ba59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e9bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45042/0 (socket says 172.21.15.46:45042) 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.165+0000 7f30ba59c640 1 -- 172.21.15.46:0/3851835087 learned_addr learned my addr 172.21.15.46:0/3851835087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a7e8d0 0x558ae6e717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30bad9d640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e5e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30ba59c640 1 -- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 msgr2=0x558ae6e5e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30ba59c640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e5e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30ba59c640 1 -- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a7e8d0 msgr2=0x558ae6e717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30ba59c640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a7e8d0 0x558ae6e717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30ba59c640 1 -- 172.21.15.46:0/3851835087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae6b0ab90 con 0x558ae6a766f0 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a7e8d0 0x558ae6e717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30ba59c640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e9bfa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0146e0 tx=0x7f30ac001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593480ef272935d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30997fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30997fa640 1 -- 172.21.15.46:0/3851835087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac012590 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30997fa640 1 -- 172.21.15.46:0/3851835087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f30ac010040 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30b0004020 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30997fa640 1 -- 172.21.15.46:0/3851835087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f30ac02e050 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30c7dfa740 10 client.5133 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.166+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x558ae6a50180 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30997fa640 1 -- 172.21.15.46:0/3851835087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b0004020 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30997fa640 1 -- 172.21.15.46:0/3851835087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f30ac031020 con 0x558ae6a766f0 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30c7dfa740 10 client.5133 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30c7dfa740 10 client.5133 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-05-09T03:10:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30c7dfa740 20 client.5133 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6a5e180 0x558ae6e83cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30c7dfa740 4 client.5133 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ae6db5430 con 0x558ae6a5e180 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6a5e180 0x558ae6e83cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30997fa640 10 client.5133 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.167+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6a5e180 0x558ae6e83cc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32fe83d2fb07cb99 server_cookie=c8fd2ae626257723 in_seq=0 out_seq=0 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30997fa640 1 -- 172.21.15.46:0/3851835087 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558ae6db5430 con 0x558ae6a5e180 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30997fa640 10 client.5133 handle_command_reply: tid=0 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 client.5133 shutdown 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 msgr2=0x558ae6e9bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e9bfa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0146e0 tx=0x7f30ac001250 comp rx=0 tx=0).stop 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 shutdown_connections 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e5e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a7e8d0 0x558ae6e717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6a5e180 0x558ae6e83cc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/3851835087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a766f0 0x558ae6e9bfa0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30997fa640 0 client.5133 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 >> 172.21.15.46:0/3851835087 conn(0x558ae6a93700 msgr2=0x558ae6e6b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 shutdown_connections 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 1 -- 172.21.15.46:0/3851835087 wait complete. 2024-05-09T03:10:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.168+0000 7f30c7dfa740 20 client.5133 trim_cache size 0 max 16384 2024-05-09T03:10:39.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.187+0000 7f30c7dfa740 1 Processor -- start 2024-05-09T03:10:39.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.187+0000 7f30c7dfa740 1 -- start start 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e39b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 0x558ae6e6df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e6e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc202d50 con 0x558ae6a7e8d0 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc201de0 con 0x558ae6a766f0 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f30bc14c720 con 0x558ae6a9f5b0 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.188+0000 7f30b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 0x558ae6e6df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 0x558ae6e6df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45046/0 (socket says 172.21.15.46:45046) 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 -- 172.21.15.46:0/1477367556 learned_addr learned my addr 172.21.15.46:0/1477367556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30bad9d640 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e6e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30ba59c640 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e39b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 -- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 msgr2=0x558ae6e6e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e6e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 -- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 msgr2=0x558ae6e39b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e39b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 -- 172.21.15.46:0/1477367556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae6e39930 con 0x558ae6a7e8d0 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 0x558ae6e6df30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f30b000b8d0 tx=0x7f30b0009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126b803cd5b29d2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f309b7fe640 1 -- 172.21.15.46:0/1477367556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b002c070 con 0x558ae6a7e8d0 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f309b7fe640 1 -- 172.21.15.46:0/1477367556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b000f7c0 con 0x558ae6a7e8d0 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.189+0000 7f309b7fe640 1 -- 172.21.15.46:0/1477367556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b000f9d0 con 0x558ae6a7e8d0 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1477367556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 msgr2=0x558ae6e6df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 0x558ae6e6df30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f30b000b8d0 tx=0x7f30b0009000 comp rx=0 tx=0).stop 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1477367556 shutdown_connections 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a9f5b0 0x558ae6e6e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a7e8d0 0x558ae6e6df30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1477367556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e39b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1477367556 >> 172.21.15.46:0/1477367556 conn(0x558ae6a93700 msgr2=0x558ae6e715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1477367556 shutdown_connections 2024-05-09T03:10:39.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.190+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1477367556 wait complete. 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.191+0000 7f30c7dfa740 1 -- start start 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.191+0000 7f30c7dfa740 1 Processor -- start 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e70a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e43880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 0x558ae6e43dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558ae6e8a2e0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc14cd70 con 0x558ae6a766f0 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558ae6e6a650 con 0x558ae6a7e8d0 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30c7dfa740 10 client.0 _ll_register_callbacks cb 0x7f30ac001fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.192+0000 7f30bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 0x558ae6e43dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 0x558ae6e43dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45058/0 (socket says 172.21.15.46:45058) 2024-05-09T03:10:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 -- 172.21.15.46:0/1852487137 learned_addr learned my addr 172.21.15.46:0/1852487137 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30ba59c640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e70a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30b9d9b640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e43880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 -- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 msgr2=0x558ae6e43880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e43880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 -- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 msgr2=0x558ae6e70a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e70a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 -- 172.21.15.46:0/1852487137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae6e87f10 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30bad9d640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 0x558ae6e43dc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f30b407ec30 tx=0x7f30b407e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26d47a75f43df12 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30997fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30997fa640 1 -- 172.21.15.46:0/1852487137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40882c0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30997fa640 1 -- 172.21.15.46:0/1852487137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f30b4081c20 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30b000f7c0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30997fa640 1 -- 172.21.15.46:0/1852487137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f30b40814d0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30c7dfa740 10 client.5139 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x558ae6e9a3a0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.193+0000 7f30997fa640 1 -- 172.21.15.46:0/1852487137 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b000f7c0 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30997fa640 1 -- 172.21.15.46:0/1852487137 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f30b40a0030 con 0x558ae6a9f5b0 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30c7dfa740 10 client.5139 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30c7dfa740 10 client.5139 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-05-09T03:10:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30c7dfa740 20 client.5139 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6e70f90 0x558ae6e42040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30c7dfa740 4 client.5139 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}] 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}) -- 0x558ae6daf460 con 0x558ae6e70f90 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30ba59c640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6e70f90 0x558ae6e42040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30997fa640 10 client.5139 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.194+0000 7f30ba59c640 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6e70f90 0x558ae6e42040 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a03a6cc836a919a server_cookie=184d58339a7e33f7 in_seq=0 out_seq=0 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30997fa640 1 -- 172.21.15.46:0/1852487137 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558ae6daf460 con 0x558ae6e70f90 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30997fa640 10 client.5139 handle_command_reply: tid=0 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 client.5139 shutdown 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 msgr2=0x558ae6e43dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 0x558ae6e43dc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f30b407ec30 tx=0x7f30b407e0c0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 shutdown_connections 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x558ae6e70f90 0x558ae6e42040 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ae6a9f5b0 0x558ae6e43dc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ae6a7e8d0 0x558ae6e43880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 --2- 172.21.15.46:0/1852487137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ae6a766f0 0x558ae6e70a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 >> 172.21.15.46:0/1852487137 conn(0x558ae6a93700 msgr2=0x558ae6e3ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30997fa640 0 client.5139 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 shutdown_connections 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 1 -- 172.21.15.46:0/1852487137 wait complete. 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.195+0000 7f30c7dfa740 20 client.5139 trim_cache size 0 max 16384 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:39.203 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-05-09T03:10:39.204 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "ff8743d4-d453-439e-be14-57a10077d696", 2024-05-09T03:10:39.204 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-05-09T03:10:39.204 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:39.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f309c0762d0 msgr2=0x7f309c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f309c0762d0 0x7f309c0787e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30ac007ed0 tx=0x7f30ac007e60 comp rx=0 tx=0).stop 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 msgr2=0x7f30bc15daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc15daf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f30b4035040 tx=0x7f30b40046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 shutdown_connections 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f309c0762d0 0x7f309c0787e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30bc0d4700 0x7f30bc15daf0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30bc0d3f30 0x7f30bc15d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 --2- 172.21.15.46:0/3311925363 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30bc0d2da0 0x7f30bc0ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 >> 172.21.15.46:0/3311925363 conn(0x7f30bc06e120 msgr2=0x7f30bc15eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 shutdown_connections 2024-05-09T03:10:39.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.196+0000 7f30c0e61640 1 -- 172.21.15.46:0/3311925363 wait complete. 2024-05-09T03:10:39.209 INFO:tasks.cephfs.test_damage:{ "return_code": 0, "scrub_tag": "ff8743d4-d453-439e-be14-57a10077d696", "mode": "asynchronous" } 2024-05-09T03:10:39.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:10:39.209 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-05-09T03:10:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.325+0000 7fb1f2700640 1 Processor -- start 2024-05-09T03:10:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.325+0000 7fb1f2700640 1 -- start start 2024-05-09T03:10:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1f2700640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1ec0d2eb0 0x7fb1ec0d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1f2700640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec0d4a60 con 0x7fb1ec0b7760 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1f2700640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec0d4d40 con 0x7fb1ec0b95a0 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1f2700640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb1ec0d5020 con 0x7fb1ec0d2eb0 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1e3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb1ec0b7760 0x7fb1ec0b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56884/0 (socket says 172.21.15.46:56884) 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1e3fff640 1 -- 172.21.15.46:0/1333687334 learned_addr learned my addr 172.21.15.46:0/1333687334 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/1333687334 >> v2:172.21.15.88:3300/0 conn(0x7fb1ec0d2eb0 0x7fb1ec0d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.326+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549790993 0 0) 0x7fb1ec0d4a60 con 0x7fb1ec0b7760 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.327+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d8003520 con 0x7fb1ec0b7760 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.327+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282122823 0 0) 0x7fb1ec0d4d40 con 0x7fb1ec0b95a0 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.327+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1ec0d4a60 con 0x7fb1ec0b95a0 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.327+0000 7fb1ebfff640 1 -- 172.21.15.46:0/1333687334 >> v1:172.21.15.88:6789/0 conn(0x7fb1ec0b95a0 legacy=0x7fb1ec0b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.327+0000 7fb1ebfff640 1 -- 172.21.15.46:0/1333687334 >> v1:172.21.15.46:6789/0 conn(0x7fb1ec0b7760 legacy=0x7fb1ec0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.327+0000 7fb1ebfff640 1 -- 172.21.15.46:0/1333687334 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec0d5300 con 0x7fb1ec0d2eb0 2024-05-09T03:10:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.328+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/1333687334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec0d32c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0097a0 tx=0x7fb1dc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a082ce7ad57b10be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.328+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc037070 con 0x7fb1ec0d2eb0 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.328+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc032440 con 0x7fb1ec0d2eb0 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.328+0000 7fb1eb7fe640 1 -- 172.21.15.46:0/1333687334 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc0327f0 con 0x7fb1ec0d2eb0 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.328+0000 7fb1f2700640 1 -- 172.21.15.46:0/1333687334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 msgr2=0x7fb1ec0d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.328+0000 7fb1f2700640 1 --2- 172.21.15.46:0/1333687334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec0d32c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0097a0 tx=0x7fb1dc02a230 comp rx=0 tx=0).stop 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.329+0000 7fb1f2700640 1 -- 172.21.15.46:0/1333687334 shutdown_connections 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.329+0000 7fb1f2700640 1 --2- 172.21.15.46:0/1333687334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec0d32c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.329+0000 7fb1f2700640 1 -- 172.21.15.46:0/1333687334 >> 172.21.15.46:0/1333687334 conn(0x7fb1ec08d390 msgr2=0x7fb1ec08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.329+0000 7fb1f2700640 1 -- 172.21.15.46:0/1333687334 shutdown_connections 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.329+0000 7fb1f2700640 1 -- 172.21.15.46:0/1333687334 wait complete. 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.330+0000 7fb1f2700640 1 Processor -- start 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.330+0000 7fb1f2700640 1 -- start start 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f2700640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 0x7fb1ec158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f2700640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1ec0b95a0 0x7fb1ec0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f2700640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec15fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f2700640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0b5360 con 0x7fb1ec0b7760 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f2700640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0b51e0 con 0x7fb1ec0d2eb0 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f2700640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec163bd0 con 0x7fb1ec0b95a0 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 0x7fb1ec158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 0x7fb1ec158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45072/0 (socket says 172.21.15.46:45072) 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 -- 172.21.15.46:0/384632904 learned_addr learned my addr 172.21.15.46:0/384632904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec15fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1ec0b95a0 0x7fb1ec0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 -- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1ec0b95a0 msgr2=0x7fb1ec0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1ec0b95a0 0x7fb1ec0c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 -- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 msgr2=0x7fb1ec15fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec15fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 -- 172.21.15.46:0/384632904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec158b70 con 0x7fb1ec0b7760 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec15fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 0x7fb1ec158380 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc000c00 tx=0x7fb1dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.332+0000 7fb1e97fa640 1 -- 172.21.15.46:0/384632904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc037040 con 0x7fb1ec0b7760 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.333+0000 7fb1e97fa640 1 -- 172.21.15.46:0/384632904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc042070 con 0x7fb1ec0b7760 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.333+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec0ca1b0 con 0x7fb1ec0b7760 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.333+0000 7fb1e97fa640 1 -- 172.21.15.46:0/384632904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc032830 con 0x7fb1ec0b7760 2024-05-09T03:10:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.333+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec15c0f0 con 0x7fb1ec0b7760 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.334+0000 7fb1e97fa640 1 -- 172.21.15.46:0/384632904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1dc0329d0 con 0x7fb1ec0b7760 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.335+0000 7fb1e97fa640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1cc076300 0x7fb1cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.335+0000 7fb1e97fa640 1 -- 172.21.15.46:0/384632904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fb1dc0b8010 con 0x7fb1ec0b7760 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.335+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1cc076300 0x7fb1cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.335+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1cc076300 0x7fb1cc078810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4007a60 tx=0x7fb1d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.336+0000 7fb1f95b1740 1 Processor -- start 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.336+0000 7fb1f95b1740 1 -- start start 2024-05-09T03:10:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf61390 0x55924d330510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 0x55924d34b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d34b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55924d2691a0 con 0x55924cf384d0 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55924d222170 con 0x55924cf61390 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55924d265160 con 0x55924cf406b0 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 0x55924d34b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 0x55924d34b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45074/0 (socket says 172.21.15.46:45074) 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1e3fff640 1 -- 172.21.15.46:0/1991083179 learned_addr learned my addr 172.21.15.46:0/1991083179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf61390 0x55924d330510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.337+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d34b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e3fff640 1 -- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 msgr2=0x55924d34b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d34b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e3fff640 1 -- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf61390 msgr2=0x55924d330510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf61390 0x55924d330510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e3fff640 1 -- 172.21.15.46:0/1991083179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55924d234390 con 0x55924cf384d0 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf61390 0x55924d330510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 0x55924d34b240 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4001e10 tx=0x7fb1d4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8847a2700cd69bb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e1ffb640 1 -- 172.21.15.46:0/1991083179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400e710 con 0x55924cf384d0 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e1ffb640 1 -- 172.21.15.46:0/1991083179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d400e8b0 con 0x55924cf384d0 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.338+0000 7fb1e1ffb640 1 -- 172.21.15.46:0/1991083179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400ec60 con 0x55924cf384d0 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 -- 172.21.15.46:0/1991083179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 msgr2=0x55924d34b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 0x55924d34b240 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4001e10 tx=0x7fb1d4011000 comp rx=0 tx=0).stop 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 -- 172.21.15.46:0/1991083179 shutdown_connections 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d34b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf384d0 0x55924d34b240 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/1991083179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf61390 0x55924d330510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 -- 172.21.15.46:0/1991083179 >> 172.21.15.46:0/1991083179 conn(0x55924cf554e0 msgr2=0x55924d305b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 -- 172.21.15.46:0/1991083179 shutdown_connections 2024-05-09T03:10:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.339+0000 7fb1f95b1740 1 -- 172.21.15.46:0/1991083179 wait complete. 2024-05-09T03:10:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.340+0000 7fb1f95b1740 1 -- start start 2024-05-09T03:10:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.340+0000 7fb1f95b1740 1 Processor -- start 2024-05-09T03:10:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf384d0 0x55924d346d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d2ffdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d3002f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55924d1ef8c0 con 0x55924cf61390 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55924d3493c0 con 0x55924cf384d0 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55924d333e40 con 0x55924cf406b0 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f95b1740 10 client.0 _ll_register_callbacks cb 0x55924d20b860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d3002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d3002f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45088/0 (socket says 172.21.15.46:45088) 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3546195980 learned_addr learned my addr 172.21.15.46:0/3546195980 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d2ffdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.341+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf384d0 0x55924d346d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 msgr2=0x55924d2ffdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d2ffdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf384d0 msgr2=0x55924d346d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf384d0 0x55924d346d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3546195980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55924d0437d0 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d3002f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005950 tx=0x7fb1e40019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d6fd715bd9b6d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1caffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1caffd640 1 -- 172.21.15.46:0/3546195980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4014400 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1caffd640 1 -- 172.21.15.46:0/3546195980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fb1e40145a0 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d400e8b0 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1caffd640 1 -- 172.21.15.46:0/3546195980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb1e4021050 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f95b1740 10 client.5151 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x55924cfcc970 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.342+0000 7fb1caffd640 1 -- 172.21.15.46:0/3546195980 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d400e8b0 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1caffd640 1 -- 172.21.15.46:0/3546195980 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7fb1e4024020 con 0x55924cf61390 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1f95b1740 10 client.5151 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1f95b1740 10 client.5151 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1f95b1740 20 client.5151 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924cf1ff60 0x55924d348840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1f95b1740 4 client.5151 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55924d276db0 con 0x55924cf1ff60 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924cf1ff60 0x55924d348840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1caffd640 10 client.5151 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.343+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924cf1ff60 0x55924d348840 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4635b74dc88c994 server_cookie=b05b959ad80a51e5 in_seq=0 out_seq=0 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1caffd640 1 -- 172.21.15.46:0/3546195980 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55924d276db0 con 0x55924cf1ff60 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1caffd640 10 client.5151 handle_command_reply: tid=0 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 client.5151 shutdown 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 msgr2=0x55924d3002f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d3002f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005950 tx=0x7fb1e40019c0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 shutdown_connections 2024-05-09T03:10:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924cf1ff60 0x55924d348840 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d3002f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf406b0 0x55924d2ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1caffd640 0 client.5151 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3546195980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf384d0 0x55924d346d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 >> 172.21.15.46:0/3546195980 conn(0x55924cf554e0 msgr2=0x55924d305b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 shutdown_connections 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.344+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3546195980 wait complete. 2024-05-09T03:10:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.345+0000 7fb1f95b1740 20 client.5151 trim_cache size 0 max 16384 2024-05-09T03:10:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.363+0000 7fb1f95b1740 1 Processor -- start 2024-05-09T03:10:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.363+0000 7fb1f95b1740 1 -- start start 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d32b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3489a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d303e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec202db0 con 0x55924cf61390 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec15d620 con 0x55924cf406b0 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec1fae50 con 0x55924cf384d0 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.364+0000 7fb1f0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d303e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d303e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45092/0 (socket says 172.21.15.46:45092) 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3223372266 learned_addr learned my addr 172.21.15.46:0/3223372266 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d32b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 msgr2=0x55924d32b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d32b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 msgr2=0x55924d3489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3223372266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55924cf11f60 con 0x55924cf61390 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d303e90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40059a0 tx=0x7fb1e400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c5c414c882ccad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1e17fa640 1 -- 172.21.15.46:0/3223372266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4014910 con 0x55924cf61390 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1e17fa640 1 -- 172.21.15.46:0/3223372266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e4029070 con 0x55924cf61390 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.365+0000 7fb1e17fa640 1 -- 172.21.15.46:0/3223372266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4014600 con 0x55924cf61390 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3223372266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 msgr2=0x55924d303e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d303e90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40059a0 tx=0x7fb1e400e000 comp rx=0 tx=0).stop 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3223372266 shutdown_connections 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d303e90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3223372266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d32b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3223372266 >> 172.21.15.46:0/3223372266 conn(0x55924cf554e0 msgr2=0x55924d346d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3223372266 shutdown_connections 2024-05-09T03:10:39.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.366+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3223372266 wait complete. 2024-05-09T03:10:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.367+0000 7fb1f95b1740 1 -- start start 2024-05-09T03:10:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.367+0000 7fb1f95b1740 1 Processor -- start 2024-05-09T03:10:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d3c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d35ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec15cfd0 con 0x55924cf61390 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55924d3cbc20 con 0x55924cf406b0 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55924d3483a0 con 0x55924cf384d0 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f95b1740 10 client.0 _ll_register_callbacks cb 0x7fb1e4003b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d35ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.368+0000 7fb1f0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d35ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45108/0 (socket says 172.21.15.46:45108) 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3970874548 learned_addr learned my addr 172.21.15.46:0/3970874548 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d3c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 msgr2=0x55924d3c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d3c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 msgr2=0x55924d3cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 -- 172.21.15.46:0/3970874548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55924d3004c0 con 0x55924cf61390 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1e3fff640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:10:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f0c9d640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d35ce70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40014a0 tx=0x7fb1e4001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2306b0718c39bc89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1caffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1caffd640 1 -- 172.21.15.46:0/3970874548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4033070 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1caffd640 1 -- 172.21.15.46:0/3970874548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fb1e402d610 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1e4029070 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1caffd640 1 -- 172.21.15.46:0/3970874548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb1e403f050 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f95b1740 10 client.5157 fetch_fsmap learned FSMap version 21 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x55924d34d680 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.369+0000 7fb1caffd640 1 -- 172.21.15.46:0/3970874548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e4029070 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1caffd640 1 -- 172.21.15.46:0/3970874548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7fb1e4042020 con 0x55924cf61390 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 10 client.5157 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 10 client.5157 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 20 client.5157 populate_metadata read hostname 'smithi046' 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924d31f3f0 0x55924d2ff860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:10:39.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 4 client.5157 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55924d270c20 con 0x55924d31f3f0 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924d31f3f0 0x55924d2ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1caffd640 10 client.5157 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1ebfff640 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924d31f3f0 0x55924d2ff860 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4e027ffdd4604aa server_cookie=a013691034624003 in_seq=0 out_seq=0 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1caffd640 1 -- 172.21.15.46:0/3970874548 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+256 (crc 0 0 0) 0x55924d270c20 con 0x55924d31f3f0 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1caffd640 10 client.5157 handle_command_reply: tid=0 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.370+0000 7fb1f95b1740 1 client.5157 shutdown 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 msgr2=0x55924d35ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d35ce70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40014a0 tx=0x7fb1e4001360 comp rx=0 tx=0).stop 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 shutdown_connections 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x55924d31f3f0 0x55924d2ff860 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55924cf61390 0x55924d35ce70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55924cf406b0 0x55924d3cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 --2- 172.21.15.46:0/3970874548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55924cf384d0 0x55924d3c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1caffd640 0 client.5157 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 >> 172.21.15.46:0/3970874548 conn(0x55924cf554e0 msgr2=0x55924d3489a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 shutdown_connections 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 1 -- 172.21.15.46:0/3970874548 wait complete. 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f95b1740 20 client.5157 trim_cache size 0 max 16384 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stdout: "ff8743d4-d453-439e-be14-57a10077d696": { 2024-05-09T03:10:39.378 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/subdir", 2024-05-09T03:10:39.379 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "ff8743d4-d453-439e-be14-57a10077d696", 2024-05-09T03:10:39.379 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "repair" 2024-05-09T03:10:39.379 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:39.379 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:10:39.379 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1cc076300 msgr2=0x7fb1cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1cc076300 0x7fb1cc078810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4007a60 tx=0x7fb1d4009040 comp rx=0 tx=0).stop 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 msgr2=0x7fb1ec158380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 0x7fb1ec158380 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc000c00 tx=0x7fb1dc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 shutdown_connections 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1ec0d2eb0 0x7fb1ec15fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1ec0b95a0 0x7fb1ec0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1cc076300 0x7fb1cc078810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.371+0000 7fb1f2700640 1 --2- 172.21.15.46:0/384632904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1ec0b7760 0x7fb1ec158380 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.372+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 >> 172.21.15.46:0/384632904 conn(0x7fb1ec08d390 msgr2=0x7fb1ec0d0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:10:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.372+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 shutdown_connections 2024-05-09T03:10:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:10:39.372+0000 7fb1f2700640 1 -- 172.21.15.46:0/384632904 wait complete. 2024-05-09T03:10:39.384 INFO:tasks.cephfs.filesystem:scrub status for tag:ff8743d4-d453-439e-be14-57a10077d696 - {'path': '/subdir', 'tag': 'ff8743d4-d453-439e-be14-57a10077d696', 'options': 'repair'} 2024-05-09T03:11:05.841 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:11:05.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:11:09.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:09.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-05-09T03:11:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.517+0000 7f0280f46640 1 Processor -- start 2024-05-09T03:11:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.517+0000 7f0280f46640 1 -- start start 2024-05-09T03:11:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0280f46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0b7e70 0x7f027c0d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0280f46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f027c0b95a0 0x7f027c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0280f46640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f027c0d4a60 con 0x7f027c0d2eb0 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0280f46640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f027c0d4d40 con 0x7f027c0b95a0 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0280f46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f027c0d5020 con 0x7f027c0b7e70 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f027c0b95a0 0x7f027c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f027c0d2eb0 0x7f027c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47012/0 (socket says 172.21.15.46:47012) 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027a59c640 1 -- 172.21.15.46:0/4024689000 learned_addr learned my addr 172.21.15.46:0/4024689000 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0273fff640 1 --2- 172.21.15.46:0/4024689000 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0b7e70 0x7f027c0d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027ad9d640 1 -- 172.21.15.46:0/4024689000 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0b7e70 msgr2=0x7f027c0d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027ad9d640 1 --2- 172.21.15.46:0/4024689000 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0b7e70 0x7f027c0d32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027ad9d640 1 -- 172.21.15.46:0/4024689000 >> v1:172.21.15.46:6789/0 conn(0x7f027c0d2eb0 legacy=0x7f027c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f027ad9d640 1 -- 172.21.15.46:0/4024689000 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c0d5300 con 0x7f027c0b95a0 2024-05-09T03:11:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.518+0000 7f0279d9b640 1 -- 172.21.15.46:0/4024689000 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1027385872 0 0) 0x7f027c0d4a60 con 0x7f027c0d2eb0 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.519+0000 7f027ad9d640 1 --2- 172.21.15.46:0/4024689000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0b95a0 0x7f027c0b99b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0274009d90 tx=0x7f027402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cb16b50d53ebd9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.519+0000 7f0279d9b640 1 -- 172.21.15.46:0/4024689000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274007740 con 0x7f027c0b95a0 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.519+0000 7f0279d9b640 1 -- 172.21.15.46:0/4024689000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02740078e0 con 0x7f027c0b95a0 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.519+0000 7f0279d9b640 1 -- 172.21.15.46:0/4024689000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274007cb0 con 0x7f027c0b95a0 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.519+0000 7f0280f46640 1 -- 172.21.15.46:0/4024689000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0b95a0 msgr2=0x7f027c0b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.519+0000 7f0280f46640 1 --2- 172.21.15.46:0/4024689000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0b95a0 0x7f027c0b99b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0274009d90 tx=0x7f027402a660 comp rx=0 tx=0).stop 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.520+0000 7f0280f46640 1 -- 172.21.15.46:0/4024689000 shutdown_connections 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.520+0000 7f0280f46640 1 --2- 172.21.15.46:0/4024689000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0b95a0 0x7f027c0b99b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.520+0000 7f0280f46640 1 --2- 172.21.15.46:0/4024689000 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0b7e70 0x7f027c0d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.520+0000 7f0280f46640 1 -- 172.21.15.46:0/4024689000 >> 172.21.15.46:0/4024689000 conn(0x7f027c08d390 msgr2=0x7f027c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.520+0000 7f0280f46640 1 -- 172.21.15.46:0/4024689000 shutdown_connections 2024-05-09T03:11:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.520+0000 7f0280f46640 1 -- 172.21.15.46:0/4024689000 wait complete. 2024-05-09T03:11:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.522+0000 7f0280f46640 1 Processor -- start 2024-05-09T03:11:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.522+0000 7f0280f46640 1 -- start start 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0280f46640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f027c0b95a0 0x7f027c14ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0280f46640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0d2eb0 0x7f027c14f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0280f46640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 0x7f027c1542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0280f46640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0b5360 con 0x7f027c0cf440 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0280f46640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f027c0b51e0 con 0x7f027c0b95a0 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0280f46640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f027c163bd0 con 0x7f027c0d2eb0 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 0x7f027c1542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 0x7f027c1542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39754/0 (socket says 172.21.15.46:39754) 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027ad9d640 1 -- 172.21.15.46:0/575842462 learned_addr learned my addr 172.21.15.46:0/575842462 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027a59c640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f027c0b95a0 0x7f027c14ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f0273fff640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0d2eb0 0x7f027c14f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027ad9d640 1 -- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0d2eb0 msgr2=0x7f027c14f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027ad9d640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0d2eb0 0x7f027c14f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.523+0000 7f027ad9d640 1 -- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f027c0b95a0 msgr2=0x7f027c14ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f027ad9d640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f027c0b95a0 0x7f027c14ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f027ad9d640 1 -- 172.21.15.46:0/575842462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c155160 con 0x7f027c0cf440 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f027a59c640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f027c0b95a0 0x7f027c14ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f027ad9d640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 0x7f027c1542a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0274009eb0 tx=0x7f0274004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f0272ffd640 1 -- 172.21.15.46:0/575842462 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027403f070 con 0x7f027c0cf440 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f0272ffd640 1 -- 172.21.15.46:0/575842462 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0274031bc0 con 0x7f027c0cf440 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f0272ffd640 1 -- 172.21.15.46:0/575842462 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274009450 con 0x7f027c0cf440 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027c0cc640 con 0x7f027c0cf440 2024-05-09T03:11:09.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.524+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f027c0cec40 con 0x7f027c0cf440 2024-05-09T03:11:09.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.526+0000 7f0272ffd640 1 -- 172.21.15.46:0/575842462 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f027403b410 con 0x7f027c0cf440 2024-05-09T03:11:09.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.526+0000 7f0272ffd640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f025c0762d0 0x7f025c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.526+0000 7f0272ffd640 1 -- 172.21.15.46:0/575842462 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f02740b87b0 con 0x7f027c0cf440 2024-05-09T03:11:09.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.526+0000 7f027a59c640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f025c0762d0 0x7f025c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.527+0000 7f027a59c640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f025c0762d0 0x7f025c0787e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f026c000f60 tx=0x7f026c007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.527+0000 7f0287dff740 1 Processor -- start 2024-05-09T03:11:09.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.527+0000 7f0287dff740 1 -- start start 2024-05-09T03:11:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0287dff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f7881a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f78835240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78835780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0287dff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562f787531a0 con 0x562f7844b390 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562f7870c170 con 0x562f784224d0 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562f7874f160 con 0x562f7842a6b0 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f7881a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f7881a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39762/0 (socket says 172.21.15.46:39762) 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 -- 172.21.15.46:0/1854209518 learned_addr learned my addr 172.21.15.46:0/1854209518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f0273fff640 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f78835240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027ad9d640 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78835780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 -- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 msgr2=0x562f78835780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78835780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 -- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 msgr2=0x562f78835240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f78835240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 -- 172.21.15.46:0/1854209518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f7871e390 con 0x562f7844b390 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.529+0000 7f027a59c640 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f7881a510 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f026c005e20 tx=0x7f026c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2580eac5cf67975b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f025bfff640 1 -- 172.21.15.46:0/1854209518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c015110 con 0x562f7844b390 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f025bfff640 1 -- 172.21.15.46:0/1854209518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026c0159c0 con 0x562f7844b390 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f025bfff640 1 -- 172.21.15.46:0/1854209518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c015d70 con 0x562f7844b390 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 -- 172.21.15.46:0/1854209518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 msgr2=0x562f7881a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f7881a510 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f026c005e20 tx=0x7f026c012000 comp rx=0 tx=0).stop 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 -- 172.21.15.46:0/1854209518 shutdown_connections 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78835780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f78835240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 --2- 172.21.15.46:0/1854209518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f7881a510 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.530+0000 7f0287dff740 1 -- 172.21.15.46:0/1854209518 >> 172.21.15.46:0/1854209518 conn(0x562f7843f4e0 msgr2=0x562f787efb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.531+0000 7f0287dff740 1 -- 172.21.15.46:0/1854209518 shutdown_connections 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.531+0000 7f0287dff740 1 -- 172.21.15.46:0/1854209518 wait complete. 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.531+0000 7f0287dff740 1 -- start start 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.532+0000 7f0287dff740 1 Processor -- start 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f7881ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78844b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f788450d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562f786d98c0 con 0x562f7844b390 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562f78845720 con 0x562f784224d0 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562f78838f80 con 0x562f7842a6b0 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0287dff740 10 client.0 _ll_register_callbacks cb 0x562f786f5860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f788450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f788450d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39774/0 (socket says 172.21.15.46:39774) 2024-05-09T03:11:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 -- 172.21.15.46:0/3719131006 learned_addr learned my addr 172.21.15.46:0/3719131006 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027a59c640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f7881ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f0273fff640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78844b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 -- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 msgr2=0x562f78844b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78844b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 -- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 msgr2=0x562f7881ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f7881ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.533+0000 7f027ad9d640 1 -- 172.21.15.46:0/3719131006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f7852d7d0 con 0x562f7844b390 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f027ad9d640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f788450d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f027407d9a0 tx=0x7f0274081150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e4112b06b7ab44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f02597fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f02597fa640 1 -- 172.21.15.46:0/3719131006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274082020 con 0x562f7844b390 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f02597fa640 1 -- 172.21.15.46:0/3719131006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f027408a2f0 con 0x562f7844b390 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f026c0159c0 con 0x562f7844b390 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f02597fa640 1 -- 172.21.15.46:0/3719131006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0274081920 con 0x562f7844b390 2024-05-09T03:11:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 10 client.5172 fetch_fsmap learned FSMap version 21 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x562f784b6970 con 0x562f7844b390 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f02597fa640 1 -- 172.21.15.46:0/3719131006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026c0159c0 con 0x562f7844b390 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f02597fa640 1 -- 172.21.15.46:0/3719131006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f0274080b80 con 0x562f7844b390 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 10 client.5172 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 10 client.5172 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 20 client.5172 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f78409f60 0x562f78815ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 4 client.5172 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.534+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f78760db0 con 0x562f78409f60 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.535+0000 7f027a59c640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f78409f60 0x562f78815ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.535+0000 7f02597fa640 10 client.5172 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.535+0000 7f027a59c640 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f78409f60 0x562f78815ad0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8406145d4660c7c server_cookie=114dc4accc1fb469 in_seq=0 out_seq=0 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f02597fa640 1 -- 172.21.15.46:0/3719131006 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562f78760db0 con 0x562f78409f60 2024-05-09T03:11:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f02597fa640 10 client.5172 handle_command_reply: tid=0 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 client.5172 shutdown 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 msgr2=0x562f788450d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f788450d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f027407d9a0 tx=0x7f0274081150 comp rx=0 tx=0).stop 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 shutdown_connections 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f78409f60 0x562f78815ad0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f02597fa640 0 client.5172 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f7844b390 0x562f788450d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78844b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 --2- 172.21.15.46:0/3719131006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f784224d0 0x562f7881ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 >> 172.21.15.46:0/3719131006 conn(0x562f7843f4e0 msgr2=0x562f787efb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 shutdown_connections 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 1 -- 172.21.15.46:0/3719131006 wait complete. 2024-05-09T03:11:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.536+0000 7f0287dff740 20 client.5172 trim_cache size 0 max 16384 2024-05-09T03:11:09.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.555+0000 7f0287dff740 1 Processor -- start 2024-05-09T03:11:09.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.555+0000 7f0287dff740 1 -- start start 2024-05-09T03:11:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f78409f60 0x562f78816790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0287dff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78816cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f7842a6b0 0x562f78844920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0287dff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f027c202fd0 con 0x562f784224d0 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f027c150de0 con 0x562f7842a6b0 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f027c1fa020 con 0x562f78409f60 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78816cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78816cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39788/0 (socket says 172.21.15.46:39788) 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 -- 172.21.15.46:0/62975996 learned_addr learned my addr 172.21.15.46:0/62975996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f027a59c640 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f78409f60 0x562f78816790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f027ad9d640 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f7842a6b0 0x562f78844920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 -- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f78409f60 msgr2=0x562f78816790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f78409f60 0x562f78816790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 -- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f7842a6b0 msgr2=0x562f78844920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f7842a6b0 0x562f78844920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.556+0000 7f0273fff640 1 -- 172.21.15.46:0/62975996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f783fbf60 con 0x562f784224d0 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.557+0000 7f0273fff640 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78816cd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0264009c80 tx=0x7f0264009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22af227da57f00b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.557+0000 7f025b7fe640 1 -- 172.21.15.46:0/62975996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264009e30 con 0x562f784224d0 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.557+0000 7f025b7fe640 1 -- 172.21.15.46:0/62975996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0264004240 con 0x562f784224d0 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.557+0000 7f025b7fe640 1 -- 172.21.15.46:0/62975996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0264003610 con 0x562f784224d0 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 -- 172.21.15.46:0/62975996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 msgr2=0x562f78816cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78816cd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0264009c80 tx=0x7f0264009190 comp rx=0 tx=0).stop 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 -- 172.21.15.46:0/62975996 shutdown_connections 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f7842a6b0 0x562f78844920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78816cd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 --2- 172.21.15.46:0/62975996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f78409f60 0x562f78816790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 -- 172.21.15.46:0/62975996 >> 172.21.15.46:0/62975996 conn(0x562f7843f4e0 msgr2=0x562f78812240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 -- 172.21.15.46:0/62975996 shutdown_connections 2024-05-09T03:11:09.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.558+0000 7f0287dff740 1 -- 172.21.15.46:0/62975996 wait complete. 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.559+0000 7f0287dff740 1 -- start start 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.559+0000 7f0287dff740 1 Processor -- start 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f78409f60 0x562f788b2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78856340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78856880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f027c1fa670 con 0x562f784224d0 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562f788b2c00 con 0x562f78409f60 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562f78843a40 con 0x562f7842a6b0 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0287dff740 10 client.0 _ll_register_callbacks cb 0x7f02740816f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0273fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78856340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0273fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78856340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39794/0 (socket says 172.21.15.46:39794) 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f0273fff640 1 -- 172.21.15.46:0/364339111 learned_addr learned my addr 172.21.15.46:0/364339111 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f027a59c640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f78409f60 0x562f788b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.560+0000 7f027ad9d640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78856880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0273fff640 1 -- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 msgr2=0x562f78856880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0273fff640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78856880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0273fff640 1 -- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f78409f60 msgr2=0x562f788b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0273fff640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f78409f60 0x562f788b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0273fff640 1 -- 172.21.15.46:0/364339111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f787e9c50 con 0x562f784224d0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0273fff640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78856340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0264009c50 tx=0x7f0264004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5b06b5bac027ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f02597fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f02597fa640 1 -- 172.21.15.46:0/364339111 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026402f060 con 0x562f784224d0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f02597fa640 1 -- 172.21.15.46:0/364339111 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f02640043f0 con 0x562f784224d0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0264004240 con 0x562f784224d0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f02597fa640 1 -- 172.21.15.46:0/364339111 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0264035050 con 0x562f784224d0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 10 client.5178 fetch_fsmap learned FSMap version 21 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x562f787f1f00 con 0x562f784224d0 2024-05-09T03:11:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f02597fa640 1 -- 172.21.15.46:0/364339111 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0264004240 con 0x562f784224d0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f02597fa640 1 -- 172.21.15.46:0/364339111 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7f0264038020 con 0x562f784224d0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 10 client.5178 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 10 client.5178 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 20 client.5178 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f783df3a0 0x562f78816760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 4 client.5178 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.561+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562f7881e4b0 con 0x562f783df3a0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f027a59c640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f783df3a0 0x562f78816760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f02597fa640 10 client.5178 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f027a59c640 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f783df3a0 0x562f78816760 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ebd41c6bb5e39b5 server_cookie=e029f119c28375f1 in_seq=0 out_seq=0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f02597fa640 1 -- 172.21.15.46:0/364339111 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562f7881e4b0 con 0x562f783df3a0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f02597fa640 10 client.5178 handle_command_reply: tid=0 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f0287dff740 1 client.5178 shutdown 2024-05-09T03:11:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 msgr2=0x562f78856340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.562+0000 7f0287dff740 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78856340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0264009c50 tx=0x7f0264004730 comp rx=0 tx=0).stop 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 shutdown_connections 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562f7842a6b0 0x562f78856880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x562f783df3a0 0x562f78816760 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562f784224d0 0x562f78856340 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 --2- 172.21.15.46:0/364339111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f78409f60 0x562f788b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 >> 172.21.15.46:0/364339111 conn(0x562f7843f4e0 msgr2=0x562f787e93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f02597fa640 0 client.5178 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 shutdown_connections 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 1 -- 172.21.15.46:0/364339111 wait complete. 2024-05-09T03:11:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0287dff740 20 client.5178 trim_cache size 0 max 16384 2024-05-09T03:11:09.571 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:09.571 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-05-09T03:11:09.571 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-05-09T03:11:09.571 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f025c0762d0 msgr2=0x7f025c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f025c0762d0 0x7f025c0787e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f026c000f60 tx=0x7f026c007ca0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 msgr2=0x7f027c1542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 0x7f027c1542a0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0274009eb0 tx=0x7f0274004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 shutdown_connections 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f025c0762d0 0x7f025c0787e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f027c0cf440 0x7f027c1542a0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f027c0d2eb0 0x7f027c14f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 --2- 172.21.15.46:0/575842462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f027c0b95a0 0x7f027c14ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 >> 172.21.15.46:0/575842462 conn(0x7f027c08d390 msgr2=0x7f027c0bb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 shutdown_connections 2024-05-09T03:11:09.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.563+0000 7f0280f46640 1 -- 172.21.15.46:0/575842462 wait complete. 2024-05-09T03:11:09.576 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T03:11:09.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:11:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.690+0000 7ff42c927640 1 Processor -- start 2024-05-09T03:11:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.690+0000 7ff42c927640 1 -- start start 2024-05-09T03:11:09.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff42c927640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4281615e0 0x7ff42815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff42c927640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff428160b20 con 0x7ff428164c70 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff42c927640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4280d15f0 con 0x7ff4280d2480 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff42c927640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff42815e520 con 0x7ff4281615e0 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff425d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4281615e0 0x7ff42815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff42659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff428164c70 0x7ff4280d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47028/0 (socket says 172.21.15.46:47028) 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff42659c640 1 -- 172.21.15.46:0/578567314 learned_addr learned my addr 172.21.15.46:0/578567314 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff425d9b640 1 -- 172.21.15.46:0/578567314 >> v1:172.21.15.88:6789/0 conn(0x7ff4280d2480 legacy=0x7ff4280d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff425d9b640 1 -- 172.21.15.46:0/578567314 >> v1:172.21.15.46:6789/0 conn(0x7ff428164c70 legacy=0x7ff4280d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.692+0000 7ff425d9b640 1 -- 172.21.15.46:0/578567314 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4280d18d0 con 0x7ff4281615e0 2024-05-09T03:11:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.693+0000 7ff425d9b640 1 --2- 172.21.15.46:0/578567314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4281615e0 0x7ff42815dfe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0097a0 tx=0x7ff41c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574d4fdfcc3a16e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.693+0000 7ff42559a640 1 -- 172.21.15.46:0/578567314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c037070 con 0x7ff4281615e0 2024-05-09T03:11:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.693+0000 7ff42559a640 1 -- 172.21.15.46:0/578567314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff41c032440 con 0x7ff4281615e0 2024-05-09T03:11:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.693+0000 7ff42559a640 1 -- 172.21.15.46:0/578567314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41c032810 con 0x7ff4281615e0 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 -- 172.21.15.46:0/578567314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4281615e0 msgr2=0x7ff42815dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 --2- 172.21.15.46:0/578567314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4281615e0 0x7ff42815dfe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0097a0 tx=0x7ff41c02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 -- 172.21.15.46:0/578567314 shutdown_connections 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 --2- 172.21.15.46:0/578567314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4281615e0 0x7ff42815dfe0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 -- 172.21.15.46:0/578567314 >> 172.21.15.46:0/578567314 conn(0x7ff428073db0 msgr2=0x7ff4280741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 -- 172.21.15.46:0/578567314 shutdown_connections 2024-05-09T03:11:09.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.694+0000 7ff42c927640 1 -- 172.21.15.46:0/578567314 wait complete. 2024-05-09T03:11:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.696+0000 7ff42c927640 1 Processor -- start 2024-05-09T03:11:09.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.696+0000 7ff42c927640 1 -- start start 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42c927640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4280d2480 0x7ff428151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42c927640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4281615e0 0x7ff4281590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42c927640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 0x7ff428155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42c927640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4281619f0 con 0x7ff428164c70 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42c927640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff428161150 con 0x7ff4281615e0 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42c927640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4280d45c0 con 0x7ff4280d2480 2024-05-09T03:11:09.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff426d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 0x7ff428155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff426d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 0x7ff428155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39816/0 (socket says 172.21.15.46:39816) 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff426d9d640 1 -- 172.21.15.46:0/371199051 learned_addr learned my addr 172.21.15.46:0/371199051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff42659c640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4280d2480 0x7ff428151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.697+0000 7ff425d9b640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4281615e0 0x7ff4281590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff426d9d640 1 -- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4280d2480 msgr2=0x7ff428151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff426d9d640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4280d2480 0x7ff428151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff426d9d640 1 -- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4281615e0 msgr2=0x7ff4281590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff426d9d640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4281615e0 0x7ff4281590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff426d9d640 1 -- 172.21.15.46:0/371199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff428152800 con 0x7ff428164c70 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff426d9d640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 0x7ff428155160 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff42000c700 tx=0x7ff42000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff420003b70 con 0x7ff428164c70 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff420003d10 con 0x7ff428164c70 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42000aa50 con 0x7ff428164c70 2024-05-09T03:11:09.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff428163e80 con 0x7ff428164c70 2024-05-09T03:11:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.698+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff428164440 con 0x7ff428164c70 2024-05-09T03:11:09.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.699+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff42808cfd0 con 0x7ff428164c70 2024-05-09T03:11:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.700+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff42000abf0 con 0x7ff428164c70 2024-05-09T03:11:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.700+0000 7ff416ffd640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff408076300 0x7ff408078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.701+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7ff4200972d0 con 0x7ff428164c70 2024-05-09T03:11:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.701+0000 7ff42659c640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff408076300 0x7ff408078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:09.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.701+0000 7ff42659c640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff408076300 0x7ff408078810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff418000f60 tx=0x7ff418009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.706+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff42808cfd0 con 0x7ff428164c70 2024-05-09T03:11:09.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.939+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff4280ca1d0 con 0x7ff428164c70 2024-05-09T03:11:09.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.939+0000 7ff416ffd640 1 -- 172.21.15.46:0/371199051 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) ==== 53+0+73 (secure 0 0 0) 0x7ff420060610 con 0x7ff428164c70 2024-05-09T03:11:09.940 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff408076300 msgr2=0x7ff408078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff408076300 0x7ff408078810 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff418000f60 tx=0x7ff418009040 comp rx=0 tx=0).stop 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 msgr2=0x7ff428155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 0x7ff428155160 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff42000c700 tx=0x7ff42000cbc0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 shutdown_connections 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff408076300 0x7ff408078810 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff428164c70 0x7ff428155160 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4281615e0 0x7ff4281590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 --2- 172.21.15.46:0/371199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4280d2480 0x7ff428151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 >> 172.21.15.46:0/371199051 conn(0x7ff428073db0 msgr2=0x7ff4280ba8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 shutdown_connections 2024-05-09T03:11:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:09.941+0000 7ff42c927640 1 -- 172.21.15.46:0/371199051 wait complete. 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:11:09.953 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:11:09.954 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:11:09.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:09.954 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:11:09.970 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:11:09.970 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:11:09.970 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:11:09.970 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42448sec preferred_lft 42448sec 2024-05-09T03:11:09.971 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:11:09.972 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.973 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:11:09.973 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:09.973 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:11:09.973 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:09.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:09.973 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:11:10.035 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:11:10.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:10.036 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:11:10.093 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:11:10.093 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:11:10.093 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:11:10.093 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:11:10.093 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:11:10.093 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:11:10.094 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:11:10.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:11:10.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:11:10.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:11:10.094 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:11:10.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:11:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:10.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:10.225 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:11:10.242 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:11:10.242 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:11:10.243 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:11:10.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:10.243 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:11:10.308 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:11:10.308 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:11:10.309 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:10.364 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:11:10.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:11:10.365 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:10.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:11:10.487 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:11:10.544 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:11:10.545 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:10.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:10.547 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:11:10.600 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:11:10.600 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:11:10.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:11:10.668 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:11:10.727 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:11:10.727 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:11:10.727 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:11:10.727 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:11:10.727 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:11:10.727 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:11:10.728 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:11:10.729 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:11:10.730 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:11:10.730 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:10.730 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:10.730 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:10.730 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:10.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:10.730 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:11:10.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:11:10.737+0000 7f8c88d1a480 -1 init, newargv = 0x7f8c78005700 newargc=15 2024-05-09T03:11:10.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[48642]: starting ceph client 2024-05-09T03:11:10.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[48642]: starting fuse 2024-05-09T03:11:10.754 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:11:10.755 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:11:10.755 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:11:10.756 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:11:10.875 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.48642.asok 2024-05-09T03:11:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:10.886 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48642.asok 2024-05-09T03:11:10.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:10.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48642.asok status 2024-05-09T03:11:11.034 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:11.034 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:11:11.034 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:11:11.034 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "48642", 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5190, 2024-05-09T03:11:11.035 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5190 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1612556873 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:11:11.036 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1612556873 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5190 192.168.0.1:0/1612556873", 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1612556873", 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 21, 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 32, 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:11:11.037 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:11.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:11.043 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:11:11.043 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:11.060 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:11:11.060 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:11.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:11.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:11.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:11.127 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-05-09T03:11:11.185 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:11:11.186 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:11:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.283+0000 7f28a16f0f40 1 Processor -- start 2024-05-09T03:11:11.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.283+0000 7f28a16f0f40 1 -- start start 2024-05-09T03:11:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.284+0000 7f28a16f0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b443930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.284+0000 7f28a16f0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 0x55fd3b449760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.284+0000 7f28a16f0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b44c160 0x55fd3b449ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f28a16f0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fd3b3c1600 con 0x55fd3b3c0600 2024-05-09T03:11:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f28a16f0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fd3b31ef50 con 0x55fd3b44c160 2024-05-09T03:11:11.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f28a16f0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fd3b3c18e0 con 0x55fd3b3b83b0 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 0x55fd3b449760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 0x55fd3b449760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39828/0 (socket says 172.21.15.46:39828) 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 -- 172.21.15.46:0/2323041016 learned_addr learned my addr 172.21.15.46:0/2323041016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289b9b9640 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b44c160 0x55fd3b449ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289b1b8640 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b443930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 -- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 msgr2=0x55fd3b443930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b443930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 -- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b44c160 msgr2=0x55fd3b449ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b44c160 0x55fd3b449ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 -- 172.21.15.46:0/2323041016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd3b3bc7f0 con 0x55fd3b3c0600 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289b9b9640 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b44c160 0x55fd3b449ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f289a9b7640 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 0x55fd3b449760 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f288400a4f0 tx=0x7f288404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d5b4892d2ba2f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f28999b5640 1 -- 172.21.15.46:0/2323041016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2884059040 con 0x55fd3b3c0600 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f28999b5640 1 -- 172.21.15.46:0/2323041016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28840549b0 con 0x55fd3b3c0600 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.285+0000 7f28999b5640 1 -- 172.21.15.46:0/2323041016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2884054d80 con 0x55fd3b3c0600 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 -- 172.21.15.46:0/2323041016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 msgr2=0x55fd3b449760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 0x55fd3b449760 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f288400a4f0 tx=0x7f288404e000 comp rx=0 tx=0).stop 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 -- 172.21.15.46:0/2323041016 shutdown_connections 2024-05-09T03:11:11.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b44c160 0x55fd3b449ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b3c0600 0x55fd3b449760 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/2323041016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b443930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 -- 172.21.15.46:0/2323041016 >> 172.21.15.46:0/2323041016 conn(0x55fd3b29aac0 msgr2=0x55fd3b3a3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 -- 172.21.15.46:0/2323041016 shutdown_connections 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.286+0000 7f28a16f0f40 1 -- 172.21.15.46:0/2323041016 wait complete. 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.287+0000 7f28a16f0f40 1 Processor -- start 2024-05-09T03:11:11.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.287+0000 7f28a16f0f40 1 -- start start 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f28a16f0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b3a6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f28a16f0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b3c0600 0x55fd3b3a7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f28a16f0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b44c160 0x55fd3b4e3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f28a16f0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fd3b3a60c0 con 0x55fd3b44c160 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f28a16f0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fd3b3a5de0 con 0x55fd3b3c0600 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f28a16f0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fd3b3a63a0 con 0x55fd3b3b83b0 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f289b9b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b44c160 0x55fd3b4e3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f289b9b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b44c160 0x55fd3b4e3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39832/0 (socket says 172.21.15.46:39832) 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.288+0000 7f289b9b9640 1 -- 172.21.15.46:0/3937036342 learned_addr learned my addr 172.21.15.46:0/3937036342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289a9b7640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b3c0600 0x55fd3b3a7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b1b8640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b3a6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b9b9640 1 -- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 msgr2=0x55fd3b3a6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b9b9640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd3b3b83b0 0x55fd3b3a6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b9b9640 1 -- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b3c0600 msgr2=0x55fd3b3a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b9b9640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3b3c0600 0x55fd3b3a7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b9b9640 1 -- 172.21.15.46:0/3937036342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd3b4e3570 con 0x55fd3b44c160 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f289b9b9640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b44c160 0x55fd3b4e3a70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28940086e0 tx=0x7f28940087d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f28989b3640 1 -- 172.21.15.46:0/3937036342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28940145f0 con 0x55fd3b44c160 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f28989b3640 1 -- 172.21.15.46:0/3937036342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2894019070 con 0x55fd3b44c160 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd3b375d80 con 0x55fd3b44c160 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f28989b3640 1 -- 172.21.15.46:0/3937036342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28940149d0 con 0x55fd3b44c160 2024-05-09T03:11:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.289+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd3b376340 con 0x55fd3b44c160 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.291+0000 7f28989b3640 1 -- 172.21.15.46:0/3937036342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2894014b70 con 0x55fd3b44c160 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.291+0000 7f28989b3640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f287c074cb0 0x7f287c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.291+0000 7f28989b3640 1 -- 172.21.15.46:0/3937036342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f2894096330 con 0x55fd3b44c160 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.292+0000 7f289b1b8640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f287c074cb0 0x7f287c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.292+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fd3b43fc30 0x55fd3b4e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.292+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32) -- 0x55fd3b4e44f0 con 0x55fd3b43fc30 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.292+0000 7f289a9b7640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fd3b43fc30 0x55fd3b4e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.292+0000 7f289b1b8640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f287c074cb0 0x7f287c0771c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f288c004540 tx=0x7f288c002070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.292+0000 7f289a9b7640 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fd3b43fc30 0x55fd3b4e3fb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.293+0000 7f289a9b7640 1 -- 172.21.15.46:0/3937036342 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v32'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2884066070 con 0x55fd3b43fc30 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.293+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fd3b43fc30 msgr2=0x55fd3b4e3fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fd3b43fc30 0x55fd3b4e3fb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f287c074cb0 msgr2=0x7f287c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f287c074cb0 0x7f287c0771c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f288c004540 tx=0x7f288c002070 comp rx=0 tx=0).stop 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b44c160 msgr2=0x55fd3b4e3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 --2- 172.21.15.46:0/3937036342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3b44c160 0x55fd3b4e3a70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f28940086e0 tx=0x7f28940087d0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f289b9b9640 1 -- 172.21.15.46:0/3937036342 reap_dead start 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 shutdown_connections 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 >> 172.21.15.46:0/3937036342 conn(0x55fd3b29aac0 msgr2=0x55fd3b3bb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 shutdown_connections 2024-05-09T03:11:11.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.294+0000 7f28a16f0f40 1 -- 172.21.15.46:0/3937036342 wait complete. 2024-05-09T03:11:11.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.413+0000 7fc614757640 1 Processor -- start 2024-05-09T03:11:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.413+0000 7fc614757640 1 -- start start 2024-05-09T03:11:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.414+0000 7fc614757640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc60c1615e0 0x7fc60c15dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.414+0000 7fc614757640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60c160b20 con 0x7fc60c0d2480 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.414+0000 7fc614757640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60c0d15f0 con 0x7fc60c164c70 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.414+0000 7fc614757640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc60c15e520 con 0x7fc60c1615e0 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.415+0000 7fc611cf2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc60c1615e0 0x7fc60c15dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.415+0000 7fc612cf4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc60c0d2480 0x7fc60c0d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47054/0 (socket says 172.21.15.46:47054) 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.415+0000 7fc612cf4640 1 -- 172.21.15.46:0/1533470668 learned_addr learned my addr 172.21.15.46:0/1533470668 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.415+0000 7fc611cf2640 1 -- 172.21.15.46:0/1533470668 >> v1:172.21.15.88:6789/0 conn(0x7fc60c164c70 legacy=0x7fc60c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.415+0000 7fc611cf2640 1 -- 172.21.15.46:0/1533470668 >> v1:172.21.15.46:6789/0 conn(0x7fc60c0d2480 legacy=0x7fc60c0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.415+0000 7fc611cf2640 1 -- 172.21.15.46:0/1533470668 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c0d18d0 con 0x7fc60c1615e0 2024-05-09T03:11:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.416+0000 7fc611cf2640 1 --2- 172.21.15.46:0/1533470668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c1615e0 0x7fc60c15dfe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0097a0 tx=0x7fc5fc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fef38fcc551ab32 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.416+0000 7fc6114f1640 1 -- 172.21.15.46:0/1533470668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc037070 con 0x7fc60c1615e0 2024-05-09T03:11:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.416+0000 7fc6114f1640 1 -- 172.21.15.46:0/1533470668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc02ae50 con 0x7fc60c1615e0 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.416+0000 7fc6114f1640 1 -- 172.21.15.46:0/1533470668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc032660 con 0x7fc60c1615e0 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.416+0000 7fc614757640 1 -- 172.21.15.46:0/1533470668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c1615e0 msgr2=0x7fc60c15dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.416+0000 7fc614757640 1 --2- 172.21.15.46:0/1533470668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c1615e0 0x7fc60c15dfe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0097a0 tx=0x7fc5fc02a100 comp rx=0 tx=0).stop 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.417+0000 7fc614757640 1 -- 172.21.15.46:0/1533470668 shutdown_connections 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.417+0000 7fc614757640 1 --2- 172.21.15.46:0/1533470668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c1615e0 0x7fc60c15dfe0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.417+0000 7fc614757640 1 -- 172.21.15.46:0/1533470668 >> 172.21.15.46:0/1533470668 conn(0x7fc60c073db0 msgr2=0x7fc60c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.417+0000 7fc614757640 1 -- 172.21.15.46:0/1533470668 shutdown_connections 2024-05-09T03:11:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.417+0000 7fc614757640 1 -- 172.21.15.46:0/1533470668 wait complete. 2024-05-09T03:11:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.418+0000 7fc614757640 1 Processor -- start 2024-05-09T03:11:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.418+0000 7fc614757640 1 -- start start 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc614757640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc60c0d2480 0x7fc60c0cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc614757640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1615e0 0x7fc60c0c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc614757640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 0x7fc60c0cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc614757640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c1619f0 con 0x7fc60c164c70 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc614757640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c161150 con 0x7fc60c0d2480 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc614757640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c0d45c0 con 0x7fc60c1615e0 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 0x7fc60c0cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 0x7fc60c0cd4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39846/0 (socket says 172.21.15.46:39846) 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 -- 172.21.15.46:0/1368249662 learned_addr learned my addr 172.21.15.46:0/1368249662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc6124f3640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc60c0d2480 0x7fc60c0cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc611cf2640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1615e0 0x7fc60c0c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 -- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1615e0 msgr2=0x7fc60c0c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1615e0 0x7fc60c0c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 -- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc60c0d2480 msgr2=0x7fc60c0cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc60c0d2480 0x7fc60c0cf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.420+0000 7fc612cf4640 1 -- 172.21.15.46:0/1368249662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c0c8dd0 con 0x7fc60c164c70 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.421+0000 7fc612cf4640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 0x7fc60c0cd4d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc60400cab0 tx=0x7fc60400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.421+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604017070 con 0x7fc60c164c70 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.421+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc60400ac30 con 0x7fc60c164c70 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.421+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604012400 con 0x7fc60c164c70 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.421+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c0c9fb0 con 0x7fc60c164c70 2024-05-09T03:11:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.421+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c0ca570 con 0x7fc60c164c70 2024-05-09T03:11:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.422+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c08cfd0 con 0x7fc60c164c70 2024-05-09T03:11:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.422+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc60401a070 con 0x7fc60c164c70 2024-05-09T03:11:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.423+0000 7fc602ffd640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5f00762d0 0x7fc5f00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.423+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fc604097210 con 0x7fc60c164c70 2024-05-09T03:11:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.423+0000 7fc6124f3640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5f00762d0 0x7fc5f00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.424+0000 7fc6124f3640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5f00762d0 0x7fc5f00787e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc608006fd0 tx=0x7fc608009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.428+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc60c08cfd0 con 0x7fc60c164c70 2024-05-09T03:11:11.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.660+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc60c0c9d40 con 0x7fc60c164c70 2024-05-09T03:11:11.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.661+0000 7fc602ffd640 1 -- 172.21.15.46:0/1368249662 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7fc6040604c0 con 0x7fc60c164c70 2024-05-09T03:11:11.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:11.662 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:11:11.662 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5f00762d0 msgr2=0x7fc5f00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5f00762d0 0x7fc5f00787e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc608006fd0 tx=0x7fc608009060 comp rx=0 tx=0).stop 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 msgr2=0x7fc60c0cd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 0x7fc60c0cd4d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc60400cab0 tx=0x7fc60400cf70 comp rx=0 tx=0).stop 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 shutdown_connections 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5f00762d0 0x7fc5f00787e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc60c164c70 0x7fc60c0cd4d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc60c1615e0 0x7fc60c0c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 --2- 172.21.15.46:0/1368249662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc60c0d2480 0x7fc60c0cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 >> 172.21.15.46:0/1368249662 conn(0x7fc60c073db0 msgr2=0x7fc60c0baa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 shutdown_connections 2024-05-09T03:11:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.663+0000 7fc614757640 1 -- 172.21.15.46:0/1368249662 wait complete. 2024-05-09T03:11:11.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage rm 548358141 2024-05-09T03:11:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.792+0000 7fd3df708640 1 Processor -- start 2024-05-09T03:11:11.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.792+0000 7fd3df708640 1 -- start start 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3df708640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3d8143410 0x7fd3d80b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3df708640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d80d1f70 0x7fd3d8161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3df708640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d81616b0 con 0x7fd3d8160c60 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3df708640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d80b58b0 con 0x7fd3d8143410 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3df708640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d80b4920 con 0x7fd3d80d1f70 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3d8143410 0x7fd3d80b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3d8143410 0x7fd3d80b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39848/0 (socket says 172.21.15.46:39848) 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/1942801902 learned_addr learned my addr 172.21.15.46:0/1942801902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/1942801902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d80d1f70 0x7fd3d8161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/1942801902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d80d1f70 msgr2=0x7fd3d8161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/1942801902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d80d1f70 0x7fd3d8161070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/1942801902 >> v1:172.21.15.88:6789/0 conn(0x7fd3d8160c60 legacy=0x7fd3d80d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.794+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/1942801902 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d8161990 con 0x7fd3d8143410 2024-05-09T03:11:11.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.795+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/1942801902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8143410 0x7fd3d80b5370 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009e50 tx=0x7fd3c802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cf158b7be6079a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.795+0000 7fd3d7fff640 1 -- 172.21.15.46:0/1942801902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8036070 con 0x7fd3d8143410 2024-05-09T03:11:11.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.795+0000 7fd3d7fff640 1 -- 172.21.15.46:0/1942801902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c8031600 con 0x7fd3d8143410 2024-05-09T03:11:11.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.795+0000 7fd3d7fff640 1 -- 172.21.15.46:0/1942801902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c80319d0 con 0x7fd3d8143410 2024-05-09T03:11:11.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 -- 172.21.15.46:0/1942801902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8143410 msgr2=0x7fd3d80b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 --2- 172.21.15.46:0/1942801902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8143410 0x7fd3d80b5370 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009e50 tx=0x7fd3c802a680 comp rx=0 tx=0).stop 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 -- 172.21.15.46:0/1942801902 shutdown_connections 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 --2- 172.21.15.46:0/1942801902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d80d1f70 0x7fd3d8161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 --2- 172.21.15.46:0/1942801902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d8143410 0x7fd3d80b5370 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 -- 172.21.15.46:0/1942801902 >> 172.21.15.46:0/1942801902 conn(0x7fd3d80b3620 msgr2=0x7fd3d80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 -- 172.21.15.46:0/1942801902 shutdown_connections 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.796+0000 7fd3df708640 1 -- 172.21.15.46:0/1942801902 wait complete. 2024-05-09T03:11:11.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.798+0000 7fd3df708640 1 Processor -- start 2024-05-09T03:11:11.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.798+0000 7fd3df708640 1 -- start start 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3df708640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3d8143410 0x7fd3d8158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3df708640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d8160c60 0x7fd3d81f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3df708640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 0x7fd3d8141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3df708640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d8143820 con 0x7fd3d815bf30 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3df708640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d80b4ee0 con 0x7fd3d8143410 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3df708640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d80d1210 con 0x7fd3d8160c60 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 0x7fd3d8141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 0x7fd3d8141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39854/0 (socket says 172.21.15.46:39854) 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 -- 172.21.15.46:0/1787986548 learned_addr learned my addr 172.21.15.46:0/1787986548 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3d8143410 0x7fd3d8158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d8160c60 0x7fd3d81f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 -- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d8160c60 msgr2=0x7fd3d81f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d8160c60 0x7fd3d81f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 -- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3d8143410 msgr2=0x7fd3d8158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3d8143410 0x7fd3d8158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3ddca5640 1 -- 172.21.15.46:0/1787986548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d80caa00 con 0x7fd3d815bf30 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.799+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d8160c60 0x7fd3d81f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.800+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 0x7fd3d8141660 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000fa40 tx=0x7fd3d000ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.800+0000 7fd3d5ffb640 1 -- 172.21.15.46:0/1787986548 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0007bd0 con 0x7fd3d815bf30 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.800+0000 7fd3d5ffb640 1 -- 172.21.15.46:0/1787986548 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d0007d90 con 0x7fd3d815bf30 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.800+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d81f77b0 con 0x7fd3d815bf30 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.800+0000 7fd3d5ffb640 1 -- 172.21.15.46:0/1787986548 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0011690 con 0x7fd3d815bf30 2024-05-09T03:11:11.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.800+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d80c69c0 con 0x7fd3d815bf30 2024-05-09T03:11:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.802+0000 7fd3d5ffb640 1 -- 172.21.15.46:0/1787986548 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd3d0019020 con 0x7fd3d815bf30 2024-05-09T03:11:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.803+0000 7fd3d5ffb640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3c40762d0 0x7fd3c40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.803+0000 7fd3d5ffb640 1 -- 172.21.15.46:0/1787986548 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fd3d0097250 con 0x7fd3d815bf30 2024-05-09T03:11:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.803+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3c40762d0 0x7fd3c40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.803+0000 7fd3e65bd740 1 Processor -- start 2024-05-09T03:11:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.804+0000 7fd3e65bd740 1 -- start start 2024-05-09T03:11:11.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.804+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3c40762d0 0x7fd3c40787e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8033000 tx=0x7fd3c8009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d19640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2cf2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634a2c558c0 con 0x5634a29245d0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5634a2c0ee10 con 0x5634a292c7b0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5634a2c51880 con 0x5634a294d490 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d19640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d19640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39856/0 (socket says 172.21.15.46:39856) 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 -- 172.21.15.46:0/1992871435 learned_addr learned my addr 172.21.15.46:0/1992871435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2cf2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 -- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 msgr2=0x5634a2d1cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 -- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 msgr2=0x5634a2cf2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2cf2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 -- 172.21.15.46:0/1992871435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634a2a2f8d0 con 0x5634a29245d0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.805+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d19640 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0015e0 tx=0x7fd3cc001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d5950344feeb1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3beffd640 1 -- 172.21.15.46:0/1992871435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc019070 con 0x5634a29245d0 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3beffd640 1 -- 172.21.15.46:0/1992871435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc001cf0 con 0x5634a29245d0 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3beffd640 1 -- 172.21.15.46:0/1992871435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc00e6e0 con 0x5634a29245d0 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 -- 172.21.15.46:0/1992871435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 msgr2=0x5634a2d19640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d19640 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0015e0 tx=0x7fd3cc001aa0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 -- 172.21.15.46:0/1992871435 shutdown_connections 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2cf2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d19640 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/1992871435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.806+0000 7fd3e65bd740 1 -- 172.21.15.46:0/1992871435 >> 172.21.15.46:0/1992871435 conn(0x5634a29415e0 msgr2=0x5634a2cf2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.807+0000 7fd3e65bd740 1 -- 172.21.15.46:0/1992871435 shutdown_connections 2024-05-09T03:11:11.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.807+0000 7fd3e65bd740 1 -- 172.21.15.46:0/1992871435 wait complete. 2024-05-09T03:11:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.807+0000 7fd3e65bd740 1 -- start start 2024-05-09T03:11:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.808+0000 7fd3e65bd740 1 Processor -- start 2024-05-09T03:11:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d1fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a292c7b0 0x5634a2d1ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d36c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634a2bdc760 con 0x5634a29245d0 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5634a2d37fb0 con 0x5634a294d490 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5634a2d18db0 con 0x5634a292c7b0 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3e65bd740 10 client.0 _ll_register_callbacks cb 0x5634a2bfe920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d1fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d1fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39868/0 (socket says 172.21.15.46:39868) 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/2631188399 learned_addr learned my addr 172.21.15.46:0/2631188399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d36c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a292c7b0 0x5634a2d1ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a292c7b0 msgr2=0x5634a2d1ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a292c7b0 0x5634a2d1ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 msgr2=0x5634a2d36c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d36c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.809+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/2631188399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634a29b8a70 con 0x5634a29245d0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d36c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d1fa20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009f70 tx=0x7fd3c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae11e8357b16feb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3b7fff640 1 -- 172.21.15.46:0/2631188399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8036040 con 0x5634a29245d0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3b7fff640 1 -- 172.21.15.46:0/2631188399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fd3c803cbe0 con 0x5634a29245d0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3cc001cf0 con 0x5634a29245d0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3b7fff640 1 -- 172.21.15.46:0/2631188399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd3c803cde0 con 0x5634a29245d0 2024-05-09T03:11:11.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 10 client.5220 fetch_fsmap learned FSMap version 21 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x5634a28fe060 con 0x5634a29245d0 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3b7fff640 1 -- 172.21.15.46:0/2631188399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc001cf0 con 0x5634a29245d0 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3b7fff640 1 -- 172.21.15.46:0/2631188399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7fd3c8058080 con 0x5634a29245d0 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 10 client.5220 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 10 client.5220 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 20 client.5220 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a290c060 0x5634a2d54650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 4 client.5220 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.810+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634a2c63310 con 0x5634a290c060 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.811+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a290c060 0x5634a2d54650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.811+0000 7fd3b7fff640 10 client.5220 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:11.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.811+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a290c060 0x5634a2d54650 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25ae3a040c0d7b1b server_cookie=4e97b8ef5b7f5af2 in_seq=0 out_seq=0 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3b7fff640 1 -- 172.21.15.46:0/2631188399 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5634a2c63310 con 0x5634a290c060 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3b7fff640 10 client.5220 handle_command_reply: tid=0 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 client.5220 shutdown 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 msgr2=0x5634a2d1fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d1fa20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009f70 tx=0x7fd3c8004ab0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 shutdown_connections 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d36c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a292c7b0 0x5634a2d1ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a290c060 0x5634a2d54650 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/2631188399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d1fa20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3b7fff640 0 client.5220 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 >> 172.21.15.46:0/2631188399 conn(0x5634a29415e0 msgr2=0x5634a2cf1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 shutdown_connections 2024-05-09T03:11:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 1 -- 172.21.15.46:0/2631188399 wait complete. 2024-05-09T03:11:11.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.812+0000 7fd3e65bd740 20 client.5220 trim_cache size 0 max 16384 2024-05-09T03:11:11.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.826+0000 7fd3e65bd740 1 Processor -- start 2024-05-09T03:11:11.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.826+0000 7fd3e65bd740 1 -- start start 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.827+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a29245d0 0x5634a2d2d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.827+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 0x5634a2d2d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.827+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d17ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.827+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d815ad30 con 0x5634a292c7b0 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.827+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d815a090 con 0x5634a294d490 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.827+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d815a6e0 con 0x5634a29245d0 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 0x5634a2d2d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 0x5634a2d2d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39876/0 (socket says 172.21.15.46:39876) 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 -- 172.21.15.46:0/915204753 learned_addr learned my addr 172.21.15.46:0/915204753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 -- 172.21.15.46:0/915204753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a29245d0 msgr2=0x5634a2d2d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a29245d0 0x5634a2d2d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 -- 172.21.15.46:0/915204753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 msgr2=0x5634a2d17ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d17ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 -- 172.21.15.46:0/915204753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634a2d36350 con 0x5634a292c7b0 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 0x5634a2d2d670 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc013000 tx=0x7fd3cc00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde3b723ea8e32f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3be7fc640 1 -- 172.21.15.46:0/915204753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc019040 con 0x5634a292c7b0 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3be7fc640 1 -- 172.21.15.46:0/915204753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc01a040 con 0x5634a292c7b0 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.828+0000 7fd3be7fc640 1 -- 172.21.15.46:0/915204753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc0069d0 con 0x5634a292c7b0 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 -- 172.21.15.46:0/915204753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 msgr2=0x5634a2d2d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 0x5634a2d2d670 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc013000 tx=0x7fd3cc00a000 comp rx=0 tx=0).stop 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 -- 172.21.15.46:0/915204753 shutdown_connections 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a294d490 0x5634a2d17ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a292c7b0 0x5634a2d2d670 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/915204753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a29245d0 0x5634a2d2d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 -- 172.21.15.46:0/915204753 >> 172.21.15.46:0/915204753 conn(0x5634a29415e0 msgr2=0x5634a2d1f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 -- 172.21.15.46:0/915204753 shutdown_connections 2024-05-09T03:11:11.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.829+0000 7fd3e65bd740 1 -- 172.21.15.46:0/915204753 wait complete. 2024-05-09T03:11:11.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.830+0000 7fd3e65bd740 1 -- start start 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.830+0000 7fd3e65bd740 1 Processor -- start 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d15b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2d1f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634a2d1ed30 con 0x5634a29245d0 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5634a2d18a20 con 0x5634a292c7b0 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5634a2d37470 con 0x5634a294d490 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3e65bd740 10 client.0 _ll_register_callbacks cb 0x7fd3c8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3dd4a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d15b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3dd4a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d15b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39884/0 (socket says 172.21.15.46:39884) 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.831+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/3718064862 learned_addr learned my addr 172.21.15.46:0/3718064862 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3ddca5640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2d1f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 msgr2=0x5634a2d1f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 msgr2=0x5634a2d1f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2d1f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dd4a4640 1 -- 172.21.15.46:0/3718064862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634a2d18000 con 0x5634a29245d0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3dd4a4640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d15b60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd3c802a6c0 tx=0x7fd3c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d563c87c3e7314dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3b7fff640 1 -- 172.21.15.46:0/3718064862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8036040 con 0x5634a29245d0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3b7fff640 1 -- 172.21.15.46:0/3718064862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fd3c803cbe0 con 0x5634a29245d0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3cc01a040 con 0x5634a29245d0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3b7fff640 1 -- 172.21.15.46:0/3718064862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd3c8031790 con 0x5634a29245d0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3e65bd740 10 client.5226 fetch_fsmap learned FSMap version 21 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) -- 0x5634a2d20170 con 0x5634a29245d0 2024-05-09T03:11:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.832+0000 7fd3b7fff640 1 -- 172.21.15.46:0/3718064862 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc01a040 con 0x5634a29245d0 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3b7fff640 1 -- 172.21.15.46:0/3718064862 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 21) ==== 3494+0+0 (secure 0 0 0) 0x7fd3c8048c90 con 0x5634a29245d0 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 10 client.5226 fetch_fsmap finished waiting for FSMap version 21 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 10 client.5226 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 20 client.5226 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a2db7d30 0x5634a2cf2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 4 client.5226 mds_command: new command op to 5010 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 548358141}] 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 --> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] -- command(tid 0: {"prefix": "damage rm", "damage_id": 548358141}) -- 0x5634a2c5d340 con 0x5634a2db7d30 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a2db7d30 0x5634a2cf2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3b7fff640 10 client.5226 ms_handle_connect on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3dcca3640 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a2db7d30 0x5634a2cf2330 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f397bfef11189416 server_cookie=4ee318740329c50c in_seq=0 out_seq=0 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3b7fff640 1 -- 172.21.15.46:0/3718064862 <== mds.0 v2:172.21.15.46:6832/2536776736 1 ==== command_reply(tid 0: 0 ) ==== 8+0+0 (crc 0 0 0) 0x5634a2c5d340 con 0x5634a2db7d30 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3b7fff640 10 client.5226 handle_command_reply: tid=0 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.833+0000 7fd3e65bd740 1 client.5226 shutdown 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 msgr2=0x5634a2d15b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d15b60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd3c802a6c0 tx=0x7fd3c8004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 shutdown_connections 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634a294d490 0x5634a2d1f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634a292c7b0 0x5634a2d1f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:6832/2536776736,v1:172.21.15.46:6834/2536776736] conn(0x5634a2db7d30 0x5634a2cf2330 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 --2- 172.21.15.46:0/3718064862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634a29245d0 0x5634a2d15b60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 >> 172.21.15.46:0/3718064862 conn(0x5634a29415e0 msgr2=0x5634a2d2d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3b7fff640 0 client.5226 ms_handle_reset on v2:172.21.15.46:6832/2536776736 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 shutdown_connections 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 1 -- 172.21.15.46:0/3718064862 wait complete. 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3e65bd740 20 client.5226 trim_cache size 0 max 16384 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3c40762d0 msgr2=0x7fd3c40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3df708640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3c40762d0 0x7fd3c40787e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8033000 tx=0x7fd3c8009000 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 msgr2=0x7fd3d8141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.834+0000 7fd3df708640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 0x7fd3d8141660 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000fa40 tx=0x7fd3d000ff00 comp rx=0 tx=0).stop 2024-05-09T03:11:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 shutdown_connections 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3c40762d0 0x7fd3c40787e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3d815bf30 0x7fd3d8141660 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3d8160c60 0x7fd3d81f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 --2- 172.21.15.46:0/1787986548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3d8143410 0x7fd3d8158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 >> 172.21.15.46:0/1787986548 conn(0x7fd3d80b3620 msgr2=0x7fd3d80cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 shutdown_connections 2024-05-09T03:11:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11.835+0000 7fd3df708640 1 -- 172.21.15.46:0/1787986548 wait complete. 2024-05-09T03:11:11.846 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> import errno 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> f.close() 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> except IOError as e: 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(errno.EIO) 2024-05-09T03:11:11.847 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:11:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:11.929 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:11:11.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:11:12.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:11:12.057 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:11:12.058 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:12.058 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:12.058 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:12.074 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:12.074 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:12.131 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:11:12.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.286+0000 7f6dd50f9640 1 Processor -- start 2024-05-09T03:11:12.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.286+0000 7f6dd50f9640 1 -- start start 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dd50f9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6dd00b5140 0x7f6dd00b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dd50f9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dd00b80b0 con 0x7f6dd00b7ca0 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dd50f9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dd00b5550 con 0x7f6dd00d3fc0 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dd50f9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6dd00d3480 con 0x7f6dd00b5140 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dced9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6dd00b5140 0x7f6dd00b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dce59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6dd00b7ca0 0x7f6dd00d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47056/0 (socket says 172.21.15.46:47056) 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dce59c640 1 -- 172.21.15.46:0/3588797224 learned_addr learned my addr 172.21.15.46:0/3588797224 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dced9d640 1 -- 172.21.15.46:0/3588797224 >> v1:172.21.15.88:6789/0 conn(0x7f6dd00d3fc0 legacy=0x7f6dd00b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dced9d640 1 -- 172.21.15.46:0/3588797224 >> v1:172.21.15.46:6789/0 conn(0x7f6dd00b7ca0 legacy=0x7f6dd00d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dced9d640 1 -- 172.21.15.46:0/3588797224 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd00d44e0 con 0x7f6dd00b5140 2024-05-09T03:11:12.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.288+0000 7f6dcdd9b640 1 -- 172.21.15.46:0/3588797224 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 947579800 0 0) 0x7f6db8001110 con 0x7f6dd00b7ca0 2024-05-09T03:11:12.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.289+0000 7f6dced9d640 1 --2- 172.21.15.46:0/3588797224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd00b7760 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009d90 tx=0x7f6dc002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3daf0d246affa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.289+0000 7f6dcdd9b640 1 -- 172.21.15.46:0/3588797224 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0037070 con 0x7f6dd00b5140 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.289+0000 7f6dcdd9b640 1 -- 172.21.15.46:0/3588797224 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc0007740 con 0x7f6dd00b5140 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.289+0000 7f6dcdd9b640 1 -- 172.21.15.46:0/3588797224 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0007b10 con 0x7f6dd00b5140 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 -- 172.21.15.46:0/3588797224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 msgr2=0x7f6dd00b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/3588797224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd00b7760 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009d90 tx=0x7f6dc002a780 comp rx=0 tx=0).stop 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 -- 172.21.15.46:0/3588797224 shutdown_connections 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/3588797224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd00b7760 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 -- 172.21.15.46:0/3588797224 >> 172.21.15.46:0/3588797224 conn(0x7f6dd008d3f0 msgr2=0x7f6dd008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 -- 172.21.15.46:0/3588797224 shutdown_connections 2024-05-09T03:11:12.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.290+0000 7f6dd50f9640 1 -- 172.21.15.46:0/3588797224 wait complete. 2024-05-09T03:11:12.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.292+0000 7f6dd50f9640 1 Processor -- start 2024-05-09T03:11:12.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.292+0000 7f6dd50f9640 1 -- start start 2024-05-09T03:11:12.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dd50f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd015a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dd50f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dd00b7ca0 0x7f6dd0152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dd50f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dd00d3fc0 0x7f6dd01eaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dd50f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd00d4d80 con 0x7f6dd00b5140 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dd50f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd00d4aa0 con 0x7f6dd00b7ca0 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dd50f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6dd00d5580 con 0x7f6dd00d3fc0 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dced9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd015a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dced9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd015a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39916/0 (socket says 172.21.15.46:39916) 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.293+0000 7f6dced9d640 1 -- 172.21.15.46:0/905169286 learned_addr learned my addr 172.21.15.46:0/905169286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dce59c640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dd00b7ca0 0x7f6dd0152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dcf59e640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dd00d3fc0 0x7f6dd01eaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dced9d640 1 -- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dd00d3fc0 msgr2=0x7f6dd01eaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dced9d640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dd00d3fc0 0x7f6dd01eaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dced9d640 1 -- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dd00b7ca0 msgr2=0x7f6dd0152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dced9d640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dd00b7ca0 0x7f6dd0152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dced9d640 1 -- 172.21.15.46:0/905169286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd01e9e30 con 0x7f6dd00b5140 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dcf59e640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dd00d3fc0 0x7f6dd01eaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dce59c640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dd00b7ca0 0x7f6dd0152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dced9d640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd015a1d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009d90 tx=0x7f6dc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0042020 con 0x7f6dd00b5140 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc0032a70 con 0x7f6dd00b5140 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd01ea110 con 0x7f6dd00b5140 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0037040 con 0x7f6dd00b5140 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.294+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd01ea6d0 con 0x7f6dd00b5140 2024-05-09T03:11:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.295+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dd00d3890 con 0x7f6dd00b5140 2024-05-09T03:11:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.296+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6dc0045020 con 0x7f6dd00b5140 2024-05-09T03:11:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.296+0000 7f6dc77fe640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6db00762d0 0x7f6db00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.297+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f6dc00b8ad0 con 0x7f6dd00b5140 2024-05-09T03:11:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.297+0000 7f6dce59c640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6db00762d0 0x7f6db00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.297+0000 7f6dce59c640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6db00762d0 0x7f6db00787e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6db8007fd0 tx=0x7f6db8007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.301+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6dc004c030 con 0x7f6dd00b5140 2024-05-09T03:11:12.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.502+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6dd006a270 con 0x7f6dd00b5140 2024-05-09T03:11:12.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.503+0000 7f6dc77fe640 1 -- 172.21.15.46:0/905169286 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v32) ==== 80+0+122 (secure 0 0 0) 0x7f6dc0082220 con 0x7f6dd00b5140 2024-05-09T03:11:12.504 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3490266983 2024-05-10T03:10:29.845801+0000 2024-05-09T03:11:12.504 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3490266983 2024-05-10T03:10:29.845801+0000 2024-05-09T03:11:12.504 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6db00762d0 msgr2=0x7f6db00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6db00762d0 0x7f6db00787e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6db8007fd0 tx=0x7f6db8007de0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 msgr2=0x7f6dd015a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd015a1d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009d90 tx=0x7f6dc0004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 shutdown_connections 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6dd00d3fc0 0x7f6dd01eaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6dd00b7ca0 0x7f6dd0152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6db00762d0 0x7f6db00787e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 --2- 172.21.15.46:0/905169286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6dd00b5140 0x7f6dd015a1d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 >> 172.21.15.46:0/905169286 conn(0x7f6dd008d3f0 msgr2=0x7f6dd0161bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:12.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 shutdown_connections 2024-05-09T03:11:12.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.505+0000 7f6dd50f9640 1 -- 172.21.15.46:0/905169286 wait complete. 2024-05-09T03:11:12.516 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:11:12.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:12.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:12.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[48642]: fuse finished with error 0 and tester_r 0 2024-05-09T03:11:12.543 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:11:12.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:12.544 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:11:12.711 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:12.711 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:11:12.711 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:12.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:11:12.731 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:12.732 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:12.732 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:12.768 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:11:12.768 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:12.768 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:11:12.768 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:12.790 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:11:12.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.923+0000 7f1b8429f640 1 Processor -- start 2024-05-09T03:11:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.923+0000 7f1b8429f640 1 -- start start 2024-05-09T03:11:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.924+0000 7f1b8429f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1b7c164c70 0x7f1b7c1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.924+0000 7f1b8429f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7c160b20 con 0x7f1b7c0d2480 2024-05-09T03:11:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.924+0000 7f1b8429f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b7c0d15f0 con 0x7f1b7c15dfe0 2024-05-09T03:11:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.924+0000 7f1b8429f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1b7c161150 con 0x7f1b7c164c70 2024-05-09T03:11:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.924+0000 7f1b8283c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1b7c0d2480 0x7f1b7c0d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47062/0 (socket says 172.21.15.46:47062) 2024-05-09T03:11:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.924+0000 7f1b8283c640 1 -- 172.21.15.46:0/2359894226 learned_addr learned my addr 172.21.15.46:0/2359894226 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b8203b640 1 --2- 172.21.15.46:0/2359894226 >> v2:172.21.15.88:3300/0 conn(0x7f1b7c164c70 0x7f1b7c1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127029029 0 0) 0x7f1b7c160b20 con 0x7f1b7c0d2480 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b68003520 con 0x7f1b7c0d2480 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582785765 0 0) 0x7f1b68003520 con 0x7f1b7c0d2480 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 >> v2:172.21.15.88:3300/0 conn(0x7f1b7c164c70 msgr2=0x7f1b7c1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 --2- 172.21.15.46:0/2359894226 >> v2:172.21.15.88:3300/0 conn(0x7f1b7c164c70 0x7f1b7c1615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 >> v1:172.21.15.88:6789/0 conn(0x7f1b7c15dfe0 legacy=0x7f1b7c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7c0d18d0 con 0x7f1b7c0d2480 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f1b74001a70 con 0x7f1b7c0d2480 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 >> v1:172.21.15.46:6789/0 conn(0x7f1b7c0d2480 legacy=0x7f1b7c0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b68003a50 0x7f1b68003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b68004800 0x7f1b68004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c161150 con 0x7f1b68005350 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6800c280 con 0x7f1b68003a50 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6800c480 con 0x7f1b68004800 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b8283c640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b8203b640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b68003a50 0x7f1b68003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b8183a640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b68004800 0x7f1b68004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.925+0000 7f1b8283c640 1 -- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b68004800 msgr2=0x7f1b68004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b8283c640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b68004800 0x7f1b68004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b8283c640 1 -- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b68003a50 msgr2=0x7f1b68003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b8283c640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b68003a50 0x7f1b68003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b8283c640 1 -- 172.21.15.46:0/2359894226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b68003520 con 0x7f1b68005350 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b8283c640 1 -- 172.21.15.46:0/2359894226 reap_dead start 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b8283c640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b6800b9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c161100 tx=0x7f1b68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21b0a4152133f51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7400a480 con 0x7f1b68005350 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7402f070 con 0x7f1b68005350 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.926+0000 7f1b81039640 1 -- 172.21.15.46:0/2359894226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7402bc00 con 0x7f1b68005350 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 -- 172.21.15.46:0/2359894226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 msgr2=0x7f1b6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b6800b9c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c161100 tx=0x7f1b68005830 comp rx=0 tx=0).stop 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 -- 172.21.15.46:0/2359894226 shutdown_connections 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 --2- 172.21.15.46:0/2359894226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b6800b9c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 -- 172.21.15.46:0/2359894226 >> 172.21.15.46:0/2359894226 conn(0x7f1b7c073db0 msgr2=0x7f1b7c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 -- 172.21.15.46:0/2359894226 shutdown_connections 2024-05-09T03:11:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.927+0000 7f1b8429f640 1 -- 172.21.15.46:0/2359894226 wait complete. 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.929+0000 7f1b8429f640 1 Processor -- start 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.929+0000 7f1b8429f640 1 -- start start 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8429f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b7c1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8429f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b7c1eb070 0x7f1b7c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8429f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b7c159260 0x7f1b7c163140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8429f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c1615e0 con 0x7f1b68005350 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8429f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7c15e3f0 con 0x7f1b7c159260 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8429f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7c161760 con 0x7f1b7c1eb070 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.930+0000 7f1b8203b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b7c1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b7c1525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39938/0 (socket says 172.21.15.46:39938) 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 -- 172.21.15.46:0/1441541381 learned_addr learned my addr 172.21.15.46:0/1441541381 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 -- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b7c1eb070 msgr2=0x7f1b7c158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b7c1eb070 0x7f1b7c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 -- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b7c159260 msgr2=0x7f1b7c163140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8283c640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b7c159260 0x7f1b7c163140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b7c159260 0x7f1b7c163140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 -- 172.21.15.46:0/1441541381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7c159670 con 0x7f1b68005350 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8203b640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b7c1525f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1b78011910 tx=0x7f1b78011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7800faf0 con 0x7f1b68005350 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7800fc90 con 0x7f1b68005350 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7c153070 con 0x7f1b68005350 2024-05-09T03:11:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78014440 con 0x7f1b68005350 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.931+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7c162380 con 0x7f1b68005350 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.932+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b7c08cfd0 con 0x7f1b68005350 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.933+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1b78012040 con 0x7f1b68005350 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.934+0000 7f1b72ffd640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b60076300 0x7f1b60078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.934+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f1b78097e10 con 0x7f1b68005350 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.934+0000 7f1b8183a640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b60076300 0x7f1b60078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:12.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.934+0000 7f1b8183a640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b60076300 0x7f1b60078810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c1eafc0 tx=0x7f1b6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:12.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:12.939+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1b7c08cfd0 con 0x7f1b68005350 2024-05-09T03:11:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.170+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b7c06a270 con 0x7f1b68005350 2024-05-09T03:11:13.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.171+0000 7f1b72ffd640 1 -- 172.21.15.46:0/1441541381 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7f1b7809b030 con 0x7f1b68005350 2024-05-09T03:11:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:13.172 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:11:13.172 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b60076300 msgr2=0x7f1b60078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b60076300 0x7f1b60078810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c1eafc0 tx=0x7f1b6c009040 comp rx=0 tx=0).stop 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 msgr2=0x7f1b7c1525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b7c1525f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1b78011910 tx=0x7f1b78011dd0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 shutdown_connections 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b7c159260 0x7f1b7c163140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b7c1eb070 0x7f1b7c158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b60076300 0x7f1b60078810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 --2- 172.21.15.46:0/1441541381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b68005350 0x7f1b7c1525f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 >> 172.21.15.46:0/1441541381 conn(0x7f1b7c073db0 msgr2=0x7f1b7c0d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:13.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 shutdown_connections 2024-05-09T03:11:13.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.173+0000 7f1b8429f640 1 -- 172.21.15.46:0/1441541381 wait complete. 2024-05-09T03:11:13.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:13.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.299+0000 7faae66f9640 1 Processor -- start 2024-05-09T03:11:13.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.299+0000 7faae66f9640 1 -- start start 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faae66f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faae0073500 0x7faae0073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faae66f9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae015e6e0 con 0x7faae00af670 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faae66f9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae00afa80 con 0x7faae0161cd0 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faae66f9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faae0073e50 con 0x7faae0073500 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faae00af670 0x7faae00d1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47066/0 (socket says 172.21.15.46:47066) 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadffff640 1 -- 172.21.15.46:0/314381933 learned_addr learned my addr 172.21.15.46:0/314381933 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:13.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadf7fe640 1 --2- 172.21.15.46:0/314381933 >> v2:172.21.15.88:3300/0 conn(0x7faae0073500 0x7faae0073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643687441 0 0) 0x7faad0001110 con 0x7faae00af670 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac8003520 con 0x7faae00af670 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 448330899 0 0) 0x7faad8001490 con 0x7faae0161cd0 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.301+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faad0001110 con 0x7faae0161cd0 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.302+0000 7faadf7fe640 1 -- 172.21.15.46:0/314381933 >> v1:172.21.15.88:6789/0 conn(0x7faae0161cd0 legacy=0x7faae01620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.302+0000 7faadf7fe640 1 -- 172.21.15.46:0/314381933 >> v1:172.21.15.46:6789/0 conn(0x7faae00af670 legacy=0x7faae00d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.302+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4155521930 0 0) 0x7faac8003520 con 0x7faae00af670 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.302+0000 7faadf7fe640 1 -- 172.21.15.46:0/314381933 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae00d1880 con 0x7faae0073500 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.302+0000 7faadf7fe640 1 --2- 172.21.15.46:0/314381933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0073500 0x7faae0073910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faad40097a0 tx=0x7faad402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5ba3a123395e49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4037070 con 0x7faae0073500 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad402ac50 con 0x7faae0073500 2024-05-09T03:11:13.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faadeffd640 1 -- 172.21.15.46:0/314381933 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad00018c0 con 0x7faae0073500 2024-05-09T03:11:13.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 -- 172.21.15.46:0/314381933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0073500 msgr2=0x7faae0073910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 --2- 172.21.15.46:0/314381933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0073500 0x7faae0073910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faad40097a0 tx=0x7faad402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 -- 172.21.15.46:0/314381933 shutdown_connections 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 --2- 172.21.15.46:0/314381933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0073500 0x7faae0073910 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 -- 172.21.15.46:0/314381933 >> 172.21.15.46:0/314381933 conn(0x7faae00b3910 msgr2=0x7faae00b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 -- 172.21.15.46:0/314381933 shutdown_connections 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.303+0000 7faae66f9640 1 -- 172.21.15.46:0/314381933 wait complete. 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.305+0000 7faae66f9640 1 Processor -- start 2024-05-09T03:11:13.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.305+0000 7faae66f9640 1 -- start start 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae66f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faae0073500 0x7faae01f8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae66f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 0x7faae00d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae66f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0161cd0 0x7faae0154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae66f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faae015ef20 con 0x7faae00af670 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae66f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faae015ec40 con 0x7faae0161cd0 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae66f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faae015f200 con 0x7faae0073500 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 0x7faae00d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 0x7faae00d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39952/0 (socket says 172.21.15.46:39952) 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 -- 172.21.15.46:0/2096619328 learned_addr learned my addr 172.21.15.46:0/2096619328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadffff640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faae0073500 0x7faae01f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faae4c96640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0161cd0 0x7faae0154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 -- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faae0073500 msgr2=0x7faae01f8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faae0073500 0x7faae01f8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 -- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0161cd0 msgr2=0x7faae0154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0161cd0 0x7faae0154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 -- 172.21.15.46:0/2096619328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae01e80f0 con 0x7faae00af670 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadf7fe640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 0x7faae00d01e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faad40098c0 tx=0x7faad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.307+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4041020 con 0x7faae00af670 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.308+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad403d420 con 0x7faae00af670 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.308+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4037040 con 0x7faae00af670 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.308+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae01e83d0 con 0x7faae00af670 2024-05-09T03:11:13.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.308+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae01e8990 con 0x7faae00af670 2024-05-09T03:11:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.309+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faae006a270 con 0x7faae00af670 2024-05-09T03:11:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.309+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faad4044020 con 0x7faae00af670 2024-05-09T03:11:13.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.310+0000 7faadcff9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faacc076300 0x7faacc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.310+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7faad40b77c0 con 0x7faae00af670 2024-05-09T03:11:13.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.310+0000 7faadffff640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faacc076300 0x7faacc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.310+0000 7faadffff640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faacc076300 0x7faacc078810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faad0005880 tx=0x7faad0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:13.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.315+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faad404b070 con 0x7faae00af670 2024-05-09T03:11:13.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.542+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faae015da80 con 0x7faae00af670 2024-05-09T03:11:13.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.543+0000 7faadcff9640 1 -- 172.21.15.46:0/2096619328 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7faad4081030 con 0x7faae00af670 2024-05-09T03:11:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:13.544 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:11:13.545 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:11:13.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faacc076300 msgr2=0x7faacc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faacc076300 0x7faacc078810 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faad0005880 tx=0x7faad0009040 comp rx=0 tx=0).stop 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 msgr2=0x7faae00d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 0x7faae00d01e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7faad40098c0 tx=0x7faad40046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 shutdown_connections 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faae0161cd0 0x7faae0154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faacc076300 0x7faacc078810 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faae00af670 0x7faae00d01e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 --2- 172.21.15.46:0/2096619328 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faae0073500 0x7faae01f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 >> 172.21.15.46:0/2096619328 conn(0x7faae00b3910 msgr2=0x7faae00b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 shutdown_connections 2024-05-09T03:11:13.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.545+0000 7faae66f9640 1 -- 172.21.15.46:0/2096619328 wait complete. 2024-05-09T03:11:13.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.671+0000 7fc8d4614640 1 Processor -- start 2024-05-09T03:11:13.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.671+0000 7fc8d4614640 1 -- start start 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.672+0000 7fc8d4614640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0d57a0 0x7fc8cc160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d4614640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8cc0b4af0 0x7fc8cc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d4614640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8cc0d49e0 con 0x7fc8cc0b77e0 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d4614640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc8cc0d5bb0 con 0x7fc8cc0b4af0 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d4614640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc8cc0d4cc0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d2bb1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc8cc0b77e0 0x7fc8cc0b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47078/0 (socket says 172.21.15.46:47078) 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d2bb1640 1 -- 172.21.15.46:0/2503223791 learned_addr learned my addr 172.21.15.46:0/2503223791 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d23b0640 1 --2- 172.21.15.46:0/2503223791 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0d57a0 0x7fc8cc160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d1baf640 1 --2- 172.21.15.46:0/2503223791 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc0b4af0 0x7fc8cc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d13ae640 1 -- 172.21.15.46:0/2503223791 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889261154 0 0) 0x7fc8cc0d49e0 con 0x7fc8cc0b77e0 2024-05-09T03:11:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d13ae640 1 -- 172.21.15.46:0/2503223791 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8b4003520 con 0x7fc8cc0b77e0 2024-05-09T03:11:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d23b0640 1 -- 172.21.15.46:0/2503223791 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc0b4af0 msgr2=0x7fc8cc0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d23b0640 1 --2- 172.21.15.46:0/2503223791 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc0b4af0 0x7fc8cc0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d23b0640 1 -- 172.21.15.46:0/2503223791 >> v1:172.21.15.46:6789/0 conn(0x7fc8cc0b77e0 legacy=0x7fc8cc0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.673+0000 7fc8d23b0640 1 -- 172.21.15.46:0/2503223791 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc0d4fa0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.674+0000 7fc8d23b0640 1 --2- 172.21.15.46:0/2503223791 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0d57a0 0x7fc8cc160610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009d90 tx=0x7fc8bc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76c63c974650cc5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.674+0000 7fc8d13ae640 1 -- 172.21.15.46:0/2503223791 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc007740 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.674+0000 7fc8d13ae640 1 -- 172.21.15.46:0/2503223791 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8bc0078e0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.674+0000 7fc8d13ae640 1 -- 172.21.15.46:0/2503223791 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8bc007c90 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 -- 172.21.15.46:0/2503223791 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0d57a0 msgr2=0x7fc8cc160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 --2- 172.21.15.46:0/2503223791 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0d57a0 0x7fc8cc160610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009d90 tx=0x7fc8bc02a660 comp rx=0 tx=0).stop 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 -- 172.21.15.46:0/2503223791 shutdown_connections 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 --2- 172.21.15.46:0/2503223791 >> v2:172.21.15.88:3300/0 conn(0x7fc8cc0b4af0 0x7fc8cc0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 --2- 172.21.15.46:0/2503223791 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0d57a0 0x7fc8cc160610 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 -- 172.21.15.46:0/2503223791 >> 172.21.15.46:0/2503223791 conn(0x7fc8cc0743e0 msgr2=0x7fc8cc0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 -- 172.21.15.46:0/2503223791 shutdown_connections 2024-05-09T03:11:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.675+0000 7fc8d4614640 1 -- 172.21.15.46:0/2503223791 wait complete. 2024-05-09T03:11:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.677+0000 7fc8d4614640 1 Processor -- start 2024-05-09T03:11:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.677+0000 7fc8d4614640 1 -- start start 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d4614640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0b4af0 0x7fc8cc0ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d4614640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0b77e0 0x7fc8cc1f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d4614640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 0x7fc8cc14dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d4614640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc0d1920 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d4614640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc8cc0d15b0 con 0x7fc8cc0b4af0 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d4614640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc8cc14e7d0 con 0x7fc8cc0b77e0 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d2bb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 0x7fc8cc14dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d2bb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 0x7fc8cc14dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39954/0 (socket says 172.21.15.46:39954) 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.678+0000 7fc8d2bb1640 1 -- 172.21.15.46:0/1794416434 learned_addr learned my addr 172.21.15.46:0/1794416434 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d1baf640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0b77e0 0x7fc8cc1f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d23b0640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0b4af0 0x7fc8cc0ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d2bb1640 1 -- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0b77e0 msgr2=0x7fc8cc1f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d2bb1640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0b77e0 0x7fc8cc1f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d2bb1640 1 -- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0b4af0 msgr2=0x7fc8cc0ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d2bb1640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0b4af0 0x7fc8cc0ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d2bb1640 1 -- 172.21.15.46:0/1794416434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8cc0cad10 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d2bb1640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 0x7fc8cc14dbc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400c390 tx=0x7fc8c400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4017070 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c4005ce0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8cc1f6340 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4007310 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.679+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8cc0c9cf0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.680+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8cc15a2d0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.681+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc8c401a070 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.681+0000 7fc8caffd640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8b80762d0 0x7fc8b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.682+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fc8c4096ad0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.682+0000 7fc8d23b0640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8b80762d0 0x7fc8b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.682+0000 7fc8d23b0640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8b80762d0 0x7fc8b80787e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc034000 tx=0x7fc8bc02f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:13.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.686+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc8c40602c0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.876+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8cc06a270 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.878+0000 7fc8caffd640 1 -- 172.21.15.46:0/1794416434 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16726 (secure 0 0 0) 0x7fc8c405fbc0 con 0x7fc8cc0d57a0 2024-05-09T03:11:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:13.879 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":32,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:10:29.845835+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:10:11.250168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T03:10:12.522105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/3490266983":"2024-05-10T03:10:29.845801+0000","172.21.15.46:6832/3490266983":"2024-05-10T03:10:29.845801+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:13.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8b80762d0 msgr2=0x7fc8b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8b80762d0 0x7fc8b80787e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc034000 tx=0x7fc8bc02f000 comp rx=0 tx=0).stop 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 msgr2=0x7fc8cc14dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 0x7fc8cc14dbc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400c390 tx=0x7fc8c400c850 comp rx=0 tx=0).stop 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 shutdown_connections 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8b80762d0 0x7fc8b80787e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8cc0d57a0 0x7fc8cc14dbc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8cc0b77e0 0x7fc8cc1f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 --2- 172.21.15.46:0/1794416434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8cc0b4af0 0x7fc8cc0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 >> 172.21.15.46:0/1794416434 conn(0x7fc8cc0743e0 msgr2=0x7fc8cc0b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 shutdown_connections 2024-05-09T03:11:13.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:13.880+0000 7fc8d4614640 1 -- 172.21.15.46:0/1794416434 wait complete. 2024-05-09T03:11:13.892 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:11:13.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:14.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.006+0000 7f2c9e414640 1 Processor -- start 2024-05-09T03:11:14.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.006+0000 7f2c9e414640 1 -- start start 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c9e414640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c981615f0 0x7f2c980b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c9e414640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c980d0b80 0x7f2c980d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c9e414640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c980d15b0 con 0x7f2c980d25e0 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c9e414640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2c98160b10 con 0x7f2c981615f0 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c9e414640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c98160df0 con 0x7f2c980d0b80 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c981615f0 0x7f2c980b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c981615f0 0x7f2c980b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39964/0 (socket says 172.21.15.46:39964) 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 -- 172.21.15.46:0/2205103882 learned_addr learned my addr 172.21.15.46:0/2205103882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:14.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c977fe640 1 --2- 172.21.15.46:0/2205103882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c980d0b80 0x7f2c980d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 -- 172.21.15.46:0/2205103882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c980d0b80 msgr2=0x7f2c980d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:14.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 --2- 172.21.15.46:0/2205103882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c980d0b80 0x7f2c980d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 -- 172.21.15.46:0/2205103882 >> v1:172.21.15.88:6789/0 conn(0x7f2c980d25e0 legacy=0x7f2c980d29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.008+0000 7f2c97fff640 1 -- 172.21.15.46:0/2205103882 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c981610d0 con 0x7f2c981615f0 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.009+0000 7f2c97fff640 1 --2- 172.21.15.46:0/2205103882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c981615f0 0x7f2c980b81b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2c88009870 tx=0x7f2c8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db9600793bb5176 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.009+0000 7f2c96ffd640 1 -- 172.21.15.46:0/2205103882 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c88037070 con 0x7f2c981615f0 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.009+0000 7f2c96ffd640 1 -- 172.21.15.46:0/2205103882 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c88032440 con 0x7f2c981615f0 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.009+0000 7f2c96ffd640 1 -- 172.21.15.46:0/2205103882 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c88032810 con 0x7f2c981615f0 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 -- 172.21.15.46:0/2205103882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c981615f0 msgr2=0x7f2c980b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 --2- 172.21.15.46:0/2205103882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c981615f0 0x7f2c980b81b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2c88009870 tx=0x7f2c8802a270 comp rx=0 tx=0).stop 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 -- 172.21.15.46:0/2205103882 shutdown_connections 2024-05-09T03:11:14.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 --2- 172.21.15.46:0/2205103882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c980d0b80 0x7f2c980d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 --2- 172.21.15.46:0/2205103882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c981615f0 0x7f2c980b81b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 -- 172.21.15.46:0/2205103882 >> 172.21.15.46:0/2205103882 conn(0x7f2c9806e190 msgr2=0x7f2c9806e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 -- 172.21.15.46:0/2205103882 shutdown_connections 2024-05-09T03:11:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.010+0000 7f2c9e414640 1 -- 172.21.15.46:0/2205103882 wait complete. 2024-05-09T03:11:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.012+0000 7f2c9e414640 1 Processor -- start 2024-05-09T03:11:14.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.012+0000 7f2c9e414640 1 -- start start 2024-05-09T03:11:14.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9e414640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 0x7f2c98153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9e414640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980d25e0 0x7f2c981f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9e414640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c981615f0 0x7f2c980cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9e414640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c981640c0 con 0x7f2c980d0b80 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9e414640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2c98163f40 con 0x7f2c980d25e0 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9e414640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c98164240 con 0x7f2c981615f0 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 0x7f2c98153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 0x7f2c98153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39972/0 (socket says 172.21.15.46:39972) 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c97fff640 1 -- 172.21.15.46:0/1500010025 learned_addr learned my addr 172.21.15.46:0/1500010025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c977fe640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980d25e0 0x7f2c981f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.013+0000 7f2c9c9b1640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c981615f0 0x7f2c980cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c97fff640 1 -- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c981615f0 msgr2=0x7f2c980cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c97fff640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c981615f0 0x7f2c980cc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c97fff640 1 -- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980d25e0 msgr2=0x7f2c981f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c97fff640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980d25e0 0x7f2c981f6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c97fff640 1 -- 172.21.15.46:0/1500010025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c980cb650 con 0x7f2c980d0b80 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c97fff640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 0x7f2c98153090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2c8802a2b0 tx=0x7f2c880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c88041020 con 0x7f2c980d0b80 2024-05-09T03:11:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c880306c0 con 0x7f2c980d0b80 2024-05-09T03:11:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c980cb930 con 0x7f2c980d0b80 2024-05-09T03:11:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c88037040 con 0x7f2c980d0b80 2024-05-09T03:11:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.014+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c980cbef0 con 0x7f2c980d0b80 2024-05-09T03:11:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.015+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c9806a270 con 0x7f2c980d0b80 2024-05-09T03:11:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.016+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2c88044020 con 0x7f2c980d0b80 2024-05-09T03:11:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.017+0000 7f2c94ff9640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c840762d0 0x7f2c840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.017+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f2c880b7de0 con 0x7f2c980d0b80 2024-05-09T03:11:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.017+0000 7f2c977fe640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c840762d0 0x7f2c840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.018+0000 7f2c977fe640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c840762d0 0x7f2c840787e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2c980b7150 tx=0x7f2c8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.021+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2c8804b030 con 0x7f2c980d0b80 2024-05-09T03:11:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.255+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c981607f0 con 0x7f2c980d0b80 2024-05-09T03:11:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.256+0000 7f2c94ff9640 1 -- 172.21.15.46:0/1500010025 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7f2c88081000 con 0x7f2c980d0b80 2024-05-09T03:11:14.256 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:14.256 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:11:14.257 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:11:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.257+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c840762d0 msgr2=0x7f2c840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c840762d0 0x7f2c840787e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2c980b7150 tx=0x7f2c8c008040 comp rx=0 tx=0).stop 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 msgr2=0x7f2c98153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 0x7f2c98153090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f2c8802a2b0 tx=0x7f2c880046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 shutdown_connections 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c981615f0 0x7f2c980cc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980d25e0 0x7f2c981f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c840762d0 0x7f2c840787e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 --2- 172.21.15.46:0/1500010025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c980d0b80 0x7f2c98153090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 >> 172.21.15.46:0/1500010025 conn(0x7f2c9806e190 msgr2=0x7f2c98075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 shutdown_connections 2024-05-09T03:11:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.258+0000 7f2c9e414640 1 -- 172.21.15.46:0/1500010025 wait complete. 2024-05-09T03:11:14.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.383+0000 7fe633d5e640 1 Processor -- start 2024-05-09T03:11:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.383+0000 7fe633d5e640 1 -- start start 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.384+0000 7fe633d5e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe62c0b9310 0x7fe62c08cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.384+0000 7fe633d5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c08d2a0 0x7fe62c08d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.384+0000 7fe633d5e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62c06ebe0 con 0x7fe62c164830 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.384+0000 7fe633d5e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe62c08dbf0 con 0x7fe62c0b9310 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.384+0000 7fe633d5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c06f760 con 0x7fe62c08d2a0 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe62c0b9310 0x7fe62c08cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe62c0b9310 0x7fe62c08cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39988/0 (socket says 172.21.15.46:39988) 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 -- 172.21.15.46:0/4278341684 learned_addr learned my addr 172.21.15.46:0/4278341684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe6312f9640 1 --2- 172.21.15.46:0/4278341684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c08d2a0 0x7fe62c08d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 -- 172.21.15.46:0/4278341684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c08d2a0 msgr2=0x7fe62c08d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 --2- 172.21.15.46:0/4278341684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c08d2a0 0x7fe62c08d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 -- 172.21.15.46:0/4278341684 >> v1:172.21.15.88:6789/0 conn(0x7fe62c164830 legacy=0x7fe62c164c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.385+0000 7fe631afa640 1 -- 172.21.15.46:0/4278341684 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c06eec0 con 0x7fe62c0b9310 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.386+0000 7fe631afa640 1 --2- 172.21.15.46:0/4278341684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c0b9310 0x7fe62c08cd60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba1042381ab1368 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.386+0000 7fe630af8640 1 -- 172.21.15.46:0/4278341684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628037070 con 0x7fe62c0b9310 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.386+0000 7fe630af8640 1 -- 172.21.15.46:0/4278341684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe628032440 con 0x7fe62c0b9310 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.386+0000 7fe630af8640 1 -- 172.21.15.46:0/4278341684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628032810 con 0x7fe62c0b9310 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 -- 172.21.15.46:0/4278341684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c0b9310 msgr2=0x7fe62c08cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 --2- 172.21.15.46:0/4278341684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c0b9310 0x7fe62c08cd60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62802a270 comp rx=0 tx=0).stop 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 -- 172.21.15.46:0/4278341684 shutdown_connections 2024-05-09T03:11:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 --2- 172.21.15.46:0/4278341684 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c08d2a0 0x7fe62c08d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 --2- 172.21.15.46:0/4278341684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c0b9310 0x7fe62c08cd60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 -- 172.21.15.46:0/4278341684 >> 172.21.15.46:0/4278341684 conn(0x7fe62c0b5660 msgr2=0x7fe62c0b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 -- 172.21.15.46:0/4278341684 shutdown_connections 2024-05-09T03:11:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.387+0000 7fe633d5e640 1 -- 172.21.15.46:0/4278341684 wait complete. 2024-05-09T03:11:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.389+0000 7fe633d5e640 1 Processor -- start 2024-05-09T03:11:14.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.389+0000 7fe633d5e640 1 -- start start 2024-05-09T03:11:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe633d5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 0x7fe62c1f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe633d5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe62c0b9310 0x7fe62c1590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe633d5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c164830 0x7fe62c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe633d5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c165350 con 0x7fe62c08d2a0 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe633d5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe62c06f480 con 0x7fe62c0b9310 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe633d5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe62c1654d0 con 0x7fe62c164830 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 0x7fe62c1f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 0x7fe62c1f6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39994/0 (socket says 172.21.15.46:39994) 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 -- 172.21.15.46:0/617621653 learned_addr learned my addr 172.21.15.46:0/617621653 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe6312f9640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe62c0b9310 0x7fe62c1590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe6322fb640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c164830 0x7fe62c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 -- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c164830 msgr2=0x7fe62c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c164830 0x7fe62c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 -- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe62c0b9310 msgr2=0x7fe62c1590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe62c0b9310 0x7fe62c1590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.390+0000 7fe631afa640 1 -- 172.21.15.46:0/617621653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62c0b46f0 con 0x7fe62c08d2a0 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.391+0000 7fe631afa640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 0x7fe62c1f6db0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe62802a2b0 tx=0x7fe6280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.391+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628037040 con 0x7fe62c08d2a0 2024-05-09T03:11:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.391+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe628042070 con 0x7fe62c08d2a0 2024-05-09T03:11:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.391+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62c0b8260 con 0x7fe62c08d2a0 2024-05-09T03:11:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.391+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe628030880 con 0x7fe62c08d2a0 2024-05-09T03:11:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.391+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62c15d660 con 0x7fe62c08d2a0 2024-05-09T03:11:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.392+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62c06a270 con 0x7fe62c08d2a0 2024-05-09T03:11:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.393+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe628030a20 con 0x7fe62c08d2a0 2024-05-09T03:11:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.393+0000 7fe6227fc640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe610076300 0x7fe610078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.394+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fe6280b7e10 con 0x7fe62c08d2a0 2024-05-09T03:11:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.394+0000 7fe6312f9640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe610076300 0x7fe610078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.394+0000 7fe6312f9640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe610076300 0x7fe610078810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe62c15b800 tx=0x7fe61c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.398+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe628081710 con 0x7fe62c08d2a0 2024-05-09T03:11:14.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.626+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe62c0cdb70 con 0x7fe62c08d2a0 2024-05-09T03:11:14.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.626+0000 7fe6227fc640 1 -- 172.21.15.46:0/617621653 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5179 (secure 0 0 0) 0x7fe628081010 con 0x7fe62c08d2a0 2024-05-09T03:11:14.627 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:14.627 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:10:13.572699+0000","modified":"2024-05-09T03:10:33.866474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5010},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5010":{"gid":5010,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2536776736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2536776736},{"type":"v1","addr":"172.21.15.46:6834","nonce":2536776736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5010,"qdb_cluster":[5010]},"id":2}]} 2024-05-09T03:11:14.628 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 21 2024-05-09T03:11:14.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.628+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe610076300 msgr2=0x7fe610078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.628+0000 7fe633d5e640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe610076300 0x7fe610078810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe62c15b800 tx=0x7fe61c009190 comp rx=0 tx=0).stop 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.628+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 msgr2=0x7fe62c1f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.628+0000 7fe633d5e640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 0x7fe62c1f6db0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe62802a2b0 tx=0x7fe6280046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 shutdown_connections 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe62c164830 0x7fe62c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe62c0b9310 0x7fe62c1590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe610076300 0x7fe610078810 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 --2- 172.21.15.46:0/617621653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe62c08d2a0 0x7fe62c1f6db0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 >> 172.21.15.46:0/617621653 conn(0x7fe62c0b5660 msgr2=0x7fe62c0d05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:14.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 shutdown_connections 2024-05-09T03:11:14.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.629+0000 7fe633d5e640 1 -- 172.21.15.46:0/617621653 wait complete. 2024-05-09T03:11:14.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:14.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.755+0000 7f6d2869a640 1 Processor -- start 2024-05-09T03:11:14.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.755+0000 7f6d2869a640 1 -- start start 2024-05-09T03:11:14.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d2869a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d20160610 0x7f6d200b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d2869a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d200b7d20 0x7f6d200b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d2869a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d200b8130 con 0x7f6d200d57a0 2024-05-09T03:11:14.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d2869a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6d200d5bb0 con 0x7f6d200b7d20 2024-05-09T03:11:14.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d2869a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d200d49e0 con 0x7f6d20160610 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d26c37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d200b7d20 0x7f6d200b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d26c37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6d200b7d20 0x7f6d200b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40000/0 (socket says 172.21.15.46:40000) 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d26c37640 1 -- 172.21.15.46:0/1354212570 learned_addr learned my addr 172.21.15.46:0/1354212570 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.756+0000 7f6d25c35640 1 --2- 172.21.15.46:0/1354212570 >> v2:172.21.15.88:3300/0 conn(0x7f6d20160610 0x7f6d200b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.757+0000 7f6d26c37640 1 -- 172.21.15.46:0/1354212570 >> v2:172.21.15.88:3300/0 conn(0x7f6d20160610 msgr2=0x7f6d200b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.757+0000 7f6d26c37640 1 --2- 172.21.15.46:0/1354212570 >> v2:172.21.15.88:3300/0 conn(0x7f6d20160610 0x7f6d200b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.757+0000 7f6d26c37640 1 -- 172.21.15.46:0/1354212570 >> v1:172.21.15.88:6789/0 conn(0x7f6d200d57a0 legacy=0x7f6d200b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.757+0000 7f6d26c37640 1 -- 172.21.15.46:0/1354212570 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d200d4cc0 con 0x7f6d200b7d20 2024-05-09T03:11:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.757+0000 7f6d26c37640 1 --2- 172.21.15.46:0/1354212570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200b7d20 0x7f6d200b9450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6d200d2f80 tx=0x7f6d1802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9af7f4cfae30582 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d25434640 1 -- 172.21.15.46:0/1354212570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18036070 con 0x7f6d200b7d20 2024-05-09T03:11:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d25434640 1 -- 172.21.15.46:0/1354212570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d18031600 con 0x7f6d200b7d20 2024-05-09T03:11:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d25434640 1 -- 172.21.15.46:0/1354212570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d180319d0 con 0x7f6d200b7d20 2024-05-09T03:11:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 -- 172.21.15.46:0/1354212570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200b7d20 msgr2=0x7f6d200b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 --2- 172.21.15.46:0/1354212570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200b7d20 0x7f6d200b9450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6d200d2f80 tx=0x7f6d1802a730 comp rx=0 tx=0).stop 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 -- 172.21.15.46:0/1354212570 shutdown_connections 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 --2- 172.21.15.46:0/1354212570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200b7d20 0x7f6d200b9450 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 --2- 172.21.15.46:0/1354212570 >> v2:172.21.15.88:3300/0 conn(0x7f6d20160610 0x7f6d200b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 -- 172.21.15.46:0/1354212570 >> 172.21.15.46:0/1354212570 conn(0x7f6d200743e0 msgr2=0x7f6d200d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 -- 172.21.15.46:0/1354212570 shutdown_connections 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.758+0000 7f6d2869a640 1 -- 172.21.15.46:0/1354212570 wait complete. 2024-05-09T03:11:14.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.760+0000 7f6d2869a640 1 Processor -- start 2024-05-09T03:11:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.760+0000 7f6d2869a640 1 -- start start 2024-05-09T03:11:14.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.761+0000 7f6d2869a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d200b7d20 0x7f6d201e9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.761+0000 7f6d2869a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 0x7f6d2015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d2869a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6d20160610 0x7f6d20159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d2869a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6d20160a20 con 0x7f6d200d57a0 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d2869a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d200d5280 con 0x7f6d200b7d20 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d2869a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6d200d15b0 con 0x7f6d20160610 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 0x7f6d2015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 0x7f6d2015e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40016/0 (socket says 172.21.15.46:40016) 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 -- 172.21.15.46:0/3644149788 learned_addr learned my addr 172.21.15.46:0/3644149788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d26436640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d200b7d20 0x7f6d201e9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d26c37640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6d20160610 0x7f6d20159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 -- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6d20160610 msgr2=0x7f6d20159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6d20160610 0x7f6d20159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 -- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d200b7d20 msgr2=0x7f6d201e9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d200b7d20 0x7f6d201e9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 -- 172.21.15.46:0/3644149788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d201fa3e0 con 0x7f6d200d57a0 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d25c35640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 0x7f6d2015e2d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e870 tx=0x7f6d1000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10017070 con 0x7f6d200d57a0 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1000c940 con 0x7f6d200d57a0 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d201fa6c0 con 0x7f6d200d57a0 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.762+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1000cd30 con 0x7f6d200d57a0 2024-05-09T03:11:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.763+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d201fac80 con 0x7f6d200d57a0 2024-05-09T03:11:14.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.763+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d201fa130 con 0x7f6d200d57a0 2024-05-09T03:11:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.764+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6d1001a070 con 0x7f6d200d57a0 2024-05-09T03:11:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.765+0000 7f6d16ffd640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6d040762d0 0x7f6d040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.765+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7f6d10097540 con 0x7f6d200d57a0 2024-05-09T03:11:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.765+0000 7f6d26436640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6d040762d0 0x7f6d040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:14.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.765+0000 7f6d26436640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6d040762d0 0x7f6d040787e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6d2006d3b0 tx=0x7f6d1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:14.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.769+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6d100606c0 con 0x7f6d200d57a0 2024-05-09T03:11:14.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.963+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d2006a270 con 0x7f6d200d57a0 2024-05-09T03:11:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.964+0000 7f6d16ffd640 1 -- 172.21.15.46:0/3644149788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16726 (secure 0 0 0) 0x7f6d100649c0 con 0x7f6d200d57a0 2024-05-09T03:11:14.965 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:14.965 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":32,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:10:29.845835+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:10:11.250168+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-05-09T03:10:12.522105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/3490266983":"2024-05-10T03:10:29.845801+0000","172.21.15.46:6832/3490266983":"2024-05-10T03:10:29.845801+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:14.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6d040762d0 msgr2=0x7f6d040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6d040762d0 0x7f6d040787e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6d2006d3b0 tx=0x7f6d1c009040 comp rx=0 tx=0).stop 2024-05-09T03:11:14.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 msgr2=0x7f6d2015e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 0x7f6d2015e2d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000e870 tx=0x7f6d1000ed30 comp rx=0 tx=0).stop 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 shutdown_connections 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6d20160610 0x7f6d20159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6d040762d0 0x7f6d040787e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d200d57a0 0x7f6d2015e2d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 --2- 172.21.15.46:0/3644149788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d200b7d20 0x7f6d201e9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 >> 172.21.15.46:0/3644149788 conn(0x7f6d200743e0 msgr2=0x7f6d200b75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 shutdown_connections 2024-05-09T03:11:14.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:14.966+0000 7f6d2869a640 1 -- 172.21.15.46:0/3644149788 wait complete. 2024-05-09T03:11:14.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:11:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.093+0000 7fc9f7914640 1 Processor -- start 2024-05-09T03:11:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.093+0000 7fc9f7914640 1 -- start start 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f7914640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9f00d2e90 0x7fc9f00b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f7914640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f01605b0 0x7fc9f00b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f7914640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9f00b4fc0 0x7fc9f00d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f7914640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f00d3400 con 0x7fc9f00b4fc0 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f7914640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f00d4a60 con 0x7fc9f00d2e90 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f7914640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f00d4d40 con 0x7fc9f01605b0 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9f00b4fc0 0x7fc9f00d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9f00b4fc0 0x7fc9f00d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40032/0 (socket says 172.21.15.46:40032) 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/864222662 learned_addr learned my addr 172.21.15.46:0/864222662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f56b0640 1 --2- 172.21.15.46:0/864222662 >> v2:172.21.15.88:3300/0 conn(0x7fc9f00d2e90 0x7fc9f00b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f4eaf640 1 --2- 172.21.15.46:0/864222662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f01605b0 0x7fc9f00b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/864222662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f01605b0 msgr2=0x7fc9f00b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 --2- 172.21.15.46:0/864222662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f01605b0 0x7fc9f00b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/864222662 >> v2:172.21.15.88:3300/0 conn(0x7fc9f00d2e90 msgr2=0x7fc9f00b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 --2- 172.21.15.46:0/864222662 >> v2:172.21.15.88:3300/0 conn(0x7fc9f00d2e90 0x7fc9f00b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.095+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/864222662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f00d5020 con 0x7fc9f00b4fc0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.096+0000 7fc9f5eb1640 1 --2- 172.21.15.46:0/864222662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f00d38b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009870 tx=0x7fc9e802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7354217363af101 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.096+0000 7fc9effff640 1 -- 172.21.15.46:0/864222662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8037070 con 0x7fc9f00b4fc0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.096+0000 7fc9effff640 1 -- 172.21.15.46:0/864222662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e8032440 con 0x7fc9f00b4fc0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.096+0000 7fc9effff640 1 -- 172.21.15.46:0/864222662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8032810 con 0x7fc9f00b4fc0 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 -- 172.21.15.46:0/864222662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f00b4fc0 msgr2=0x7fc9f00d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 --2- 172.21.15.46:0/864222662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f00d38b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009870 tx=0x7fc9e802a210 comp rx=0 tx=0).stop 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 -- 172.21.15.46:0/864222662 shutdown_connections 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 --2- 172.21.15.46:0/864222662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f00d38b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 --2- 172.21.15.46:0/864222662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f01605b0 0x7fc9f00b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 --2- 172.21.15.46:0/864222662 >> v2:172.21.15.88:3300/0 conn(0x7fc9f00d2e90 0x7fc9f00b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 -- 172.21.15.46:0/864222662 >> 172.21.15.46:0/864222662 conn(0x7fc9f008d390 msgr2=0x7fc9f008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 -- 172.21.15.46:0/864222662 shutdown_connections 2024-05-09T03:11:15.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.097+0000 7fc9f7914640 1 -- 172.21.15.46:0/864222662 wait complete. 2024-05-09T03:11:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.099+0000 7fc9f7914640 1 Processor -- start 2024-05-09T03:11:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.099+0000 7fc9f7914640 1 -- start start 2024-05-09T03:11:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f7914640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f01fa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f7914640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f00d2e90 0x7fc9f00cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f7914640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 0x7fc9f01f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f7914640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f00b4d80 con 0x7fc9f01605b0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f7914640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f00b4c00 con 0x7fc9f00b4fc0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f7914640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f00d3df0 con 0x7fc9f00d2e90 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f5eb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 0x7fc9f01f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f5eb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 0x7fc9f01f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40042/0 (socket says 172.21.15.46:40042) 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.100+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/838130210 learned_addr learned my addr 172.21.15.46:0/838130210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f56b0640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f01fa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f4eaf640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f00d2e90 0x7fc9f00cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f00d2e90 msgr2=0x7fc9f00cb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f5eb1640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f00d2e90 0x7fc9f00cb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9f00b4fc0 msgr2=0x7fc9f01fa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f5eb1640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f01fa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f5eb1640 1 -- 172.21.15.46:0/838130210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f00cbbd0 con 0x7fc9f01605b0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f5eb1640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 0x7fc9f01f62a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9e802a250 tx=0x7fc9e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:15.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8041020 con 0x7fc9f01605b0 2024-05-09T03:11:15.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e803d640 con 0x7fc9f01605b0 2024-05-09T03:11:15.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f01f9e90 con 0x7fc9f01605b0 2024-05-09T03:11:15.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8037040 con 0x7fc9f01605b0 2024-05-09T03:11:15.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.101+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f00cbeb0 con 0x7fc9f01605b0 2024-05-09T03:11:15.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.102+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f00cb940 con 0x7fc9f01605b0 2024-05-09T03:11:15.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.103+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc9e8044020 con 0x7fc9f01605b0 2024-05-09T03:11:15.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.104+0000 7fc9edffb640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9dc0762d0 0x7fc9dc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.104+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5795+0+0 (secure 0 0 0) 0x7fc9e80b8150 con 0x7fc9f01605b0 2024-05-09T03:11:15.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.104+0000 7fc9f56b0640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9dc0762d0 0x7fc9dc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.105+0000 7fc9f56b0640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9dc0762d0 0x7fc9dc0787e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc9f006f130 tx=0x7fc9e000d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:15.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.108+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc9e804b030 con 0x7fc9f01605b0 2024-05-09T03:11:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.335+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc9f0069b40 con 0x7fc9f01605b0 2024-05-09T03:11:15.865 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:11:15.336+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:11:15.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.337+0000 7fc9edffb640 1 -- 172.21.15.46:0/838130210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) ==== 161+0+0 (secure 0 0 0) 0x7fc9e80813e0 con 0x7fc9f01605b0 2024-05-09T03:11:15.866 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:11:15.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9dc0762d0 msgr2=0x7fc9dc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9dc0762d0 0x7fc9dc0787e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc9f006f130 tx=0x7fc9e000d040 comp rx=0 tx=0).stop 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 msgr2=0x7fc9f01f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 0x7fc9f01f62a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9e802a250 tx=0x7fc9e80046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 shutdown_connections 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9dc0762d0 0x7fc9dc0787e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9f01605b0 0x7fc9f01f62a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9f00d2e90 0x7fc9f00cb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 --2- 172.21.15.46:0/838130210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9f00b4fc0 0x7fc9f01fa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 >> 172.21.15.46:0/838130210 conn(0x7fc9f008d390 msgr2=0x7fc9f0075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 shutdown_connections 2024-05-09T03:11:15.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.341+0000 7fc9f7914640 1 -- 172.21.15.46:0/838130210 wait complete. 2024-05-09T03:11:15.868 INFO:tasks.ceph.mds.a.smithi046.stderr: -3478> 2024-05-09T03:10:36.857+0000 7fe4db00a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:11:15.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:11:15.869 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:11:15.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.992+0000 7fd0aace9640 1 Processor -- start 2024-05-09T03:11:15.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.992+0000 7fd0aace9640 1 -- start start 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0aace9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0a40b50b0 0x7fd0a40d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0aace9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0a40b8220 0x7fd0a40b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0aace9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0a40d5740 con 0x7fd0a40d11f0 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0aace9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd0a406e760 con 0x7fd0a40b8220 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0aace9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd0a406de30 con 0x7fd0a40b50b0 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0a40b8220 0x7fd0a40b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0a40b8220 0x7fd0a40b92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35512/0 (socket says 172.21.15.46:35512) 2024-05-09T03:11:15.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 -- 172.21.15.46:0/2129918890 learned_addr learned my addr 172.21.15.46:0/2129918890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a3fff640 1 --1- 172.21.15.46:0/2129918890 >> v1:172.21.15.46:6789/0 conn(0x7fd0a40d11f0 0x7fd0a40b7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40676/0 (socket says 172.21.15.46:40676) 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a8a85640 1 --2- 172.21.15.46:0/2129918890 >> v2:172.21.15.88:3300/0 conn(0x7fd0a40b50b0 0x7fd0a40d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 -- 172.21.15.46:0/2129918890 >> v2:172.21.15.88:3300/0 conn(0x7fd0a40b50b0 msgr2=0x7fd0a40d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 --2- 172.21.15.46:0/2129918890 >> v2:172.21.15.88:3300/0 conn(0x7fd0a40b50b0 0x7fd0a40d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 -- 172.21.15.46:0/2129918890 >> v1:172.21.15.46:6789/0 conn(0x7fd0a40d11f0 legacy=0x7fd0a40b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a9286640 1 -- 172.21.15.46:0/2129918890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a406e110 con 0x7fd0a40b8220 2024-05-09T03:11:15.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.994+0000 7fd0a37fe640 1 -- 172.21.15.46:0/2129918890 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2705686357 0 0) 0x7fd0a40d5740 con 0x7fd0a40d11f0 2024-05-09T03:11:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.995+0000 7fd0a9286640 1 --2- 172.21.15.46:0/2129918890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40b8220 0x7fd0a40b92a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4163f80 tx=0x7fd09c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5828b96c52a97fe4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.995+0000 7fd0a37fe640 1 -- 172.21.15.46:0/2129918890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c036070 con 0x7fd0a40b8220 2024-05-09T03:11:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.995+0000 7fd0a37fe640 1 -- 172.21.15.46:0/2129918890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09c031600 con 0x7fd0a40b8220 2024-05-09T03:11:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.995+0000 7fd0a37fe640 1 -- 172.21.15.46:0/2129918890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0319d0 con 0x7fd0a40b8220 2024-05-09T03:11:15.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 -- 172.21.15.46:0/2129918890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40b8220 msgr2=0x7fd0a40b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 --2- 172.21.15.46:0/2129918890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40b8220 0x7fd0a40b92a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4163f80 tx=0x7fd09c02a730 comp rx=0 tx=0).stop 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 -- 172.21.15.46:0/2129918890 shutdown_connections 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 --2- 172.21.15.46:0/2129918890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40b8220 0x7fd0a40b92a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 --2- 172.21.15.46:0/2129918890 >> v2:172.21.15.88:3300/0 conn(0x7fd0a40b50b0 0x7fd0a40d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 -- 172.21.15.46:0/2129918890 >> 172.21.15.46:0/2129918890 conn(0x7fd0a40d2b80 msgr2=0x7fd0a40d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 -- 172.21.15.46:0/2129918890 shutdown_connections 2024-05-09T03:11:15.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.996+0000 7fd0aace9640 1 -- 172.21.15.46:0/2129918890 wait complete. 2024-05-09T03:11:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.998+0000 7fd0aace9640 1 Processor -- start 2024-05-09T03:11:15.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.998+0000 7fd0aace9640 1 -- start start 2024-05-09T03:11:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0aace9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0a40b50b0 0x7fd0a415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0aace9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0a40d11f0 0x7fd0a41604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0aace9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 0x7fd0a40bb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0aace9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a406ebf0 con 0x7fd0a40baf50 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0aace9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a40b54c0 con 0x7fd0a40d11f0 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0aace9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd0a40d1800 con 0x7fd0a40b50b0 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0a9286640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 0x7fd0a40bb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0a9286640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 0x7fd0a40bb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35516/0 (socket says 172.21.15.46:35516) 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0a9286640 1 -- 172.21.15.46:0/3996298647 learned_addr learned my addr 172.21.15.46:0/3996298647 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0a8a85640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0a40b50b0 0x7fd0a415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:15.999+0000 7fd0a3fff640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0a40d11f0 0x7fd0a41604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a9286640 1 -- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0a40b50b0 msgr2=0x7fd0a415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a9286640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0a40b50b0 0x7fd0a415d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a9286640 1 -- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0a40d11f0 msgr2=0x7fd0a41604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a9286640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0a40d11f0 0x7fd0a41604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a9286640 1 -- 172.21.15.46:0/3996298647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a40bb8a0 con 0x7fd0a40baf50 2024-05-09T03:11:16.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a8a85640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0a40b50b0 0x7fd0a415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a9286640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 0x7fd0a40bb360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd09c034040 tx=0x7fd09c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c041020 con 0x7fd0a40baf50 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09c037040 con 0x7fd0a40baf50 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a40ada20 con 0x7fd0a40baf50 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c036050 con 0x7fd0a40baf50 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.000+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a41611a0 con 0x7fd0a40baf50 2024-05-09T03:11:16.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.001+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0a406a270 con 0x7fd0a40baf50 2024-05-09T03:11:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.002+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd09c007c50 con 0x7fd0a40baf50 2024-05-09T03:11:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.002+0000 7fd0a17fa640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd090076300 0x7fd090078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.002+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5881+0+0 (secure 0 0 0) 0x7fd09c0b7d60 con 0x7fd0a40baf50 2024-05-09T03:11:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.002+0000 7fd0a8a85640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd090076300 0x7fd090078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.003+0000 7fd0a8a85640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd090076300 0x7fd090078810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd094006fd0 tx=0x7fd094008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:16.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.007+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd09c04b060 con 0x7fd0a40baf50 2024-05-09T03:11:16.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.242+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fd0a40ccb20 con 0x7fd0a40baf50 2024-05-09T03:11:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.339+0000 7fd0a17fa640 1 -- 172.21.15.46:0/3996298647 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7fd0a40bb8a0 con 0x7fd0a40baf50 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd090076300 msgr2=0x7fd090078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd090076300 0x7fd090078810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd094006fd0 tx=0x7fd094008040 comp rx=0 tx=0).stop 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 msgr2=0x7fd0a40bb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 0x7fd0a40bb360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd09c034040 tx=0x7fd09c004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 shutdown_connections 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd090076300 0x7fd090078810 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0a40baf50 0x7fd0a40bb360 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0a40d11f0 0x7fd0a41604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 --2- 172.21.15.46:0/3996298647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0a40b50b0 0x7fd0a415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 >> 172.21.15.46:0/3996298647 conn(0x7fd0a40d2b80 msgr2=0x7fd0a40b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:16.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 shutdown_connections 2024-05-09T03:11:16.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.341+0000 7fd0aace9640 1 -- 172.21.15.46:0/3996298647 wait complete. 2024-05-09T03:11:16.353 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:11:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.471+0000 7f556b926640 1 Processor -- start 2024-05-09T03:11:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.471+0000 7f556b926640 1 -- start start 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f556b926640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55640bb770 0x7f55640d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f556b926640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5564163eb0 0x7f55641642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f556b926640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5564164900 con 0x7f55640d2a70 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f556b926640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f55640d4bb0 con 0x7f5564163eb0 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f556b926640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5564164be0 con 0x7f55640bb770 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5564163eb0 0x7f55641642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5564163eb0 0x7f55641642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35526/0 (socket says 172.21.15.46:35526) 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 -- 172.21.15.46:0/979330315 learned_addr learned my addr 172.21.15.46:0/979330315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f55696c2640 1 --2- 172.21.15.46:0/979330315 >> v2:172.21.15.88:3300/0 conn(0x7f55640bb770 0x7f55640d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 -- 172.21.15.46:0/979330315 >> v2:172.21.15.88:3300/0 conn(0x7f55640bb770 msgr2=0x7f55640d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 --2- 172.21.15.46:0/979330315 >> v2:172.21.15.88:3300/0 conn(0x7f55640bb770 0x7f55640d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 -- 172.21.15.46:0/979330315 >> v1:172.21.15.88:6789/0 conn(0x7f55640d2a70 legacy=0x7f55640d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.473+0000 7f5568ec1640 1 -- 172.21.15.46:0/979330315 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55640d4840 con 0x7f5564163eb0 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.474+0000 7f5568ec1640 1 --2- 172.21.15.46:0/979330315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5564163eb0 0x7f55641642c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f55640d6d80 tx=0x7f555402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fe954121f7c000 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.474+0000 7f555bfff640 1 -- 172.21.15.46:0/979330315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554037070 con 0x7f5564163eb0 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.474+0000 7f555bfff640 1 -- 172.21.15.46:0/979330315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5554032440 con 0x7f5564163eb0 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.474+0000 7f555bfff640 1 -- 172.21.15.46:0/979330315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554032810 con 0x7f5564163eb0 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 -- 172.21.15.46:0/979330315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5564163eb0 msgr2=0x7f55641642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 --2- 172.21.15.46:0/979330315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5564163eb0 0x7f55641642c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f55640d6d80 tx=0x7f555402a270 comp rx=0 tx=0).stop 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 -- 172.21.15.46:0/979330315 shutdown_connections 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 --2- 172.21.15.46:0/979330315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5564163eb0 0x7f55641642c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 --2- 172.21.15.46:0/979330315 >> v2:172.21.15.88:3300/0 conn(0x7f55640bb770 0x7f55640d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 -- 172.21.15.46:0/979330315 >> 172.21.15.46:0/979330315 conn(0x7f556406e370 msgr2=0x7f556406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:16.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 -- 172.21.15.46:0/979330315 shutdown_connections 2024-05-09T03:11:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.475+0000 7f556b926640 1 -- 172.21.15.46:0/979330315 wait complete. 2024-05-09T03:11:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.477+0000 7f556b926640 1 Processor -- start 2024-05-09T03:11:16.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.477+0000 7f556b926640 1 -- start start 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f556b926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 0x7f5564162240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f556b926640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55640d2a70 0x7f5564162780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f556b926640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5564163eb0 0x7f556414e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f556b926640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f55640d0bc0 con 0x7f55640bb770 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f556b926640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f55640bbb80 con 0x7f55640d2a70 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f556b926640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f55640d0d40 con 0x7f5564163eb0 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 0x7f5564162240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 0x7f5564162240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35528/0 (socket says 172.21.15.46:35528) 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 -- 172.21.15.46:0/1433029699 learned_addr learned my addr 172.21.15.46:0/1433029699 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f5568ec1640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55640d2a70 0x7f5564162780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f5569ec3640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5564163eb0 0x7f556414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 -- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5564163eb0 msgr2=0x7f556414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5564163eb0 0x7f556414e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 -- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55640d2a70 msgr2=0x7f5564162780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55640d2a70 0x7f5564162780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.478+0000 7f55696c2640 1 -- 172.21.15.46:0/1433029699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5564150b50 con 0x7f55640bb770 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.479+0000 7f55696c2640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 0x7f5564162240 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f556000d5d0 tx=0x7f556000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.479+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560004240 con 0x7f55640bb770 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.479+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55600043e0 con 0x7f55640bb770 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.479+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5564150e30 con 0x7f55640bb770 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.479+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560003830 con 0x7f55640bb770 2024-05-09T03:11:16.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.479+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5564152c00 con 0x7f55640bb770 2024-05-09T03:11:16.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.480+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556406a270 con 0x7f55640bb770 2024-05-09T03:11:16.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.481+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f55600039d0 con 0x7f55640bb770 2024-05-09T03:11:16.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.481+0000 7f5559ffb640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5548076300 0x7f5548078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:16.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.481+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5881+0+0 (secure 0 0 0) 0x7f5560097820 con 0x7f55640bb770 2024-05-09T03:11:16.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.482+0000 7f5568ec1640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5548076300 0x7f5548078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:16.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.482+0000 7f5568ec1640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5548076300 0x7f5548078810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5554034000 tx=0x7f555402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:16.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.486+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5560060a80 con 0x7f55640bb770 2024-05-09T03:11:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:16.683+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f55640d4130 con 0x7f55640bb770 2024-05-09T03:11:17.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.348+0000 7f5559ffb640 1 -- 172.21.15.46:0/1433029699 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) ==== 181+0+0 (secure 0 0 0) 0x7f5560064d80 con 0x7f55640bb770 2024-05-09T03:11:17.349 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5548076300 msgr2=0x7f5548078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5548076300 0x7f5548078810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5554034000 tx=0x7f555402e000 comp rx=0 tx=0).stop 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 msgr2=0x7f5564162240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 0x7f5564162240 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f556000d5d0 tx=0x7f556000da90 comp rx=0 tx=0).stop 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 shutdown_connections 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5564163eb0 0x7f556414e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55640d2a70 0x7f5564162780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5548076300 0x7f5548078810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 --2- 172.21.15.46:0/1433029699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640bb770 0x7f5564162240 secure :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f556000d5d0 tx=0x7f556000da90 comp rx=0 tx=0).stop 2024-05-09T03:11:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.352+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 >> 172.21.15.46:0/1433029699 conn(0x7f556406e370 msgr2=0x7f55640d2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.353+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 shutdown_connections 2024-05-09T03:11:17.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.353+0000 7f556b926640 1 -- 172.21.15.46:0/1433029699 wait complete. 2024-05-09T03:11:17.368 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:11:17.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.492+0000 7fd4c351b640 1 Processor -- start 2024-05-09T03:11:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.493+0000 7fd4c351b640 1 -- start start 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c351b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4bc0b4fa0 0x7fd4bc164830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c351b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4bc164d70 0x7fd4bc165180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c351b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4bc0b4700 con 0x7fd4bc0b92e0 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c351b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd4bc1656c0 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c351b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd4bc0b3c10 con 0x7fd4bc164d70 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4bc0b4fa0 0x7fd4bc164830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd4bc0b4fa0 0x7fd4bc164830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35532/0 (socket says 172.21.15.46:35532) 2024-05-09T03:11:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 -- 172.21.15.46:0/3143347439 learned_addr learned my addr 172.21.15.46:0/3143347439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c1ab8640 1 --2- 172.21.15.46:0/3143347439 >> v2:172.21.15.88:3300/0 conn(0x7fd4bc164d70 0x7fd4bc165180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 -- 172.21.15.46:0/3143347439 >> v2:172.21.15.88:3300/0 conn(0x7fd4bc164d70 msgr2=0x7fd4bc165180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 --2- 172.21.15.46:0/3143347439 >> v2:172.21.15.88:3300/0 conn(0x7fd4bc164d70 0x7fd4bc165180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 -- 172.21.15.46:0/3143347439 >> v1:172.21.15.88:6789/0 conn(0x7fd4bc0b92e0 legacy=0x7fd4bc08d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:17.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.494+0000 7fd4c0ab6640 1 -- 172.21.15.46:0/3143347439 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4bc0b3e60 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.495+0000 7fd4c0ab6640 1 --2- 172.21.15.46:0/3143347439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc164830 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc08e350 tx=0x7fd4ac02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54eb508cf678c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.495+0000 7fd4b3fff640 1 -- 172.21.15.46:0/3143347439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac037070 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.495+0000 7fd4b3fff640 1 -- 172.21.15.46:0/3143347439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ac032440 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.495+0000 7fd4b3fff640 1 -- 172.21.15.46:0/3143347439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac032810 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 -- 172.21.15.46:0/3143347439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 msgr2=0x7fd4bc164830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 --2- 172.21.15.46:0/3143347439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc164830 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc08e350 tx=0x7fd4ac02a150 comp rx=0 tx=0).stop 2024-05-09T03:11:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 -- 172.21.15.46:0/3143347439 shutdown_connections 2024-05-09T03:11:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 --2- 172.21.15.46:0/3143347439 >> v2:172.21.15.88:3300/0 conn(0x7fd4bc164d70 0x7fd4bc165180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 --2- 172.21.15.46:0/3143347439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc164830 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 -- 172.21.15.46:0/3143347439 >> 172.21.15.46:0/3143347439 conn(0x7fd4bc0b55f0 msgr2=0x7fd4bc0b5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 -- 172.21.15.46:0/3143347439 shutdown_connections 2024-05-09T03:11:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.496+0000 7fd4c351b640 1 -- 172.21.15.46:0/3143347439 wait complete. 2024-05-09T03:11:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.498+0000 7fd4c351b640 1 Processor -- start 2024-05-09T03:11:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.498+0000 7fd4c351b640 1 -- start start 2024-05-09T03:11:17.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c351b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc1f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:17.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c351b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4bc0b92e0 0x7fd4bc0a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:17.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c351b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4bc164d70 0x7fd4bc09fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c351b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd4bc08d840 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c351b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd4bc0b4420 con 0x7fd4bc164d70 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c351b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd4bc0b4b40 con 0x7fd4bc0b92e0 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c12b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc1f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c12b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc1f7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35536/0 (socket says 172.21.15.46:35536) 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.499+0000 7fd4c12b7640 1 -- 172.21.15.46:0/2993820407 learned_addr learned my addr 172.21.15.46:0/2993820407 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c1ab8640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4bc164d70 0x7fd4bc09fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c0ab6640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4bc0b92e0 0x7fd4bc0a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:17.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c12b7640 1 -- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4bc0b92e0 msgr2=0x7fd4bc0a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c12b7640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4bc0b92e0 0x7fd4bc0a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c12b7640 1 -- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4bc164d70 msgr2=0x7fd4bc09fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c12b7640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4bc164d70 0x7fd4bc09fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c12b7640 1 -- 172.21.15.46:0/2993820407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4bc09eda0 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c0ab6640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4bc0b92e0 0x7fd4bc0a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c12b7640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc1f7870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c3a0 tx=0x7fd4b800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8017070 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b80052d0 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8003830 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4bc0a14f0 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.500+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4bc09f320 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.501+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4bc06a270 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.502+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd4b801a070 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.503+0000 7fd4b1ffb640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd4a00762d0 0x7fd4a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.503+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5443+0+0 (secure 0 0 0) 0x7fd4b80968e0 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.503+0000 7fd4c0ab6640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd4a00762d0 0x7fd4a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.504+0000 7fd4c0ab6640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd4a00762d0 0x7fd4a00787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac034000 tx=0x7fd4ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:17.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.507+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd4b805fbc0 con 0x7fd4bc0b4fa0 2024-05-09T03:11:17.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:17.705+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd4bc072090 con 0x7fd4bc0b4fa0 2024-05-09T03:11:18.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.350+0000 7fd4b1ffb640 1 -- 172.21.15.46:0/2993820407 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) ==== 169+0+0 (secure 0 0 0) 0x7fd4b8068290 con 0x7fd4bc0b4fa0 2024-05-09T03:11:18.351 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:11:18.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.353+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd4a00762d0 msgr2=0x7fd4a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.353+0000 7fd4c351b640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd4a00762d0 0x7fd4a00787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac034000 tx=0x7fd4ac02e000 comp rx=0 tx=0).stop 2024-05-09T03:11:18.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.353+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 msgr2=0x7fd4bc1f7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.353+0000 7fd4c351b640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc1f7870 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4b800c3a0 tx=0x7fd4b800c860 comp rx=0 tx=0).stop 2024-05-09T03:11:18.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.354+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 shutdown_connections 2024-05-09T03:11:18.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.354+0000 7fd4c351b640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4bc164d70 0x7fd4bc09fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.354+0000 7fd4c351b640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4bc0b92e0 0x7fd4bc0a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.354+0000 7fd4c351b640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd4a00762d0 0x7fd4a00787e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.354+0000 7fd4c351b640 1 --2- 172.21.15.46:0/2993820407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4bc0b4fa0 0x7fd4bc1f7870 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.354+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 >> 172.21.15.46:0/2993820407 conn(0x7fd4bc0b55f0 msgr2=0x7fd4bc072a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:18.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.355+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 shutdown_connections 2024-05-09T03:11:18.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.355+0000 7fd4c351b640 1 -- 172.21.15.46:0/2993820407 wait complete. 2024-05-09T03:11:18.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T03:11:18.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.493+0000 7f9fb8a43640 1 Processor -- start 2024-05-09T03:11:18.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.493+0000 7f9fb8a43640 1 -- start start 2024-05-09T03:11:18.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb8a43640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b7e70 0x7f9fb4160320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb8a43640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fb4163d80 0x7f9fb4164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb8a43640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb4161700 con 0x7f9fb40b4d80 2024-05-09T03:11:18.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb8a43640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9fb40d1230 con 0x7f9fb4163d80 2024-05-09T03:11:18.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb8a43640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb40d0900 con 0x7f9fb40b7e70 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fb4163d80 0x7f9fb4164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fb4163d80 0x7f9fb4164190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35542/0 (socket says 172.21.15.46:35542) 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.494+0000 7f9fb2d9d640 1 -- 172.21.15.46:0/4222121014 learned_addr learned my addr 172.21.15.46:0/4222121014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.495+0000 7f9fa9d9b640 1 --2- 172.21.15.46:0/4222121014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b7e70 0x7f9fb4160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.495+0000 7f9fb2d9d640 1 -- 172.21.15.46:0/4222121014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b7e70 msgr2=0x7f9fb4160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.495+0000 7f9fb2d9d640 1 --2- 172.21.15.46:0/4222121014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b7e70 0x7f9fb4160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.495+0000 7f9fb2d9d640 1 -- 172.21.15.46:0/4222121014 >> v1:172.21.15.88:6789/0 conn(0x7f9fb40b4d80 legacy=0x7f9fb40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:18.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.495+0000 7f9fb2d9d640 1 -- 172.21.15.46:0/4222121014 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb40d0be0 con 0x7f9fb4163d80 2024-05-09T03:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.495+0000 7f9fb2d9d640 1 --2- 172.21.15.46:0/4222121014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb4164190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009e50 tx=0x7f9fac02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca92acb39cc8e51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb1d9b640 1 -- 172.21.15.46:0/4222121014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac036070 con 0x7f9fb4163d80 2024-05-09T03:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb1d9b640 1 -- 172.21.15.46:0/4222121014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fac031600 con 0x7f9fb4163d80 2024-05-09T03:11:18.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb1d9b640 1 -- 172.21.15.46:0/4222121014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac0319d0 con 0x7f9fb4163d80 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 -- 172.21.15.46:0/4222121014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 msgr2=0x7f9fb4164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/4222121014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb4164190 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009e50 tx=0x7f9fac02a680 comp rx=0 tx=0).stop 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 -- 172.21.15.46:0/4222121014 shutdown_connections 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/4222121014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb4164190 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/4222121014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b7e70 0x7f9fb4160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 -- 172.21.15.46:0/4222121014 >> 172.21.15.46:0/4222121014 conn(0x7f9fb40d2760 msgr2=0x7f9fb40d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 -- 172.21.15.46:0/4222121014 shutdown_connections 2024-05-09T03:11:18.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.496+0000 7f9fb8a43640 1 -- 172.21.15.46:0/4222121014 wait complete. 2024-05-09T03:11:18.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.498+0000 7f9fb8a43640 1 Processor -- start 2024-05-09T03:11:18.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.498+0000 7f9fb8a43640 1 -- start start 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb8a43640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b4d80 0x7f9fb415de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb8a43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb40cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb8a43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fb415e660 0x7f9fb40cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb8a43640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb40b8390 con 0x7f9fb4163d80 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb8a43640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb40d16c0 con 0x7f9fb415e660 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb8a43640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb40b8510 con 0x7f9fb40b4d80 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb40cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb40cefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35558/0 (socket says 172.21.15.46:35558) 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 -- 172.21.15.46:0/1757139520 learned_addr learned my addr 172.21.15.46:0/1757139520 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb2d9d640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fb415e660 0x7f9fb40cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fb259c640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b4d80 0x7f9fb415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 -- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b4d80 msgr2=0x7f9fb415de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b4d80 0x7f9fb415de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 -- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fb415e660 msgr2=0x7f9fb40cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fb415e660 0x7f9fb40cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 -- 172.21.15.46:0/1757139520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb40ce530 con 0x7f9fb4163d80 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.500+0000 7f9fa9d9b640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb40cefd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00c820 tx=0x7f9f9c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.501+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c015070 con 0x7f9fb4163d80 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.501+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f9c004020 con 0x7f9fb4163d80 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.501+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb40aca00 con 0x7f9fb4163d80 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.501+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c0043f0 con 0x7f9fb4163d80 2024-05-09T03:11:18.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.501+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb40ceab0 con 0x7f9fb4163d80 2024-05-09T03:11:18.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.502+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb40ac770 con 0x7f9fb4163d80 2024-05-09T03:11:18.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.503+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9f9c007830 con 0x7f9fb4163d80 2024-05-09T03:11:18.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.503+0000 7f9fab7fe640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f940762d0 0x7f9f940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.503+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7f9f9c0970a0 con 0x7f9fb4163d80 2024-05-09T03:11:18.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.503+0000 7f9fb259c640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f940762d0 0x7f9f940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.504+0000 7f9fb259c640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f940762d0 0x7f9f940787e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4006fd0 tx=0x7f9fa4009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:18.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.508+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9f9c0605c0 con 0x7f9fb4163d80 2024-05-09T03:11:18.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.702+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f9fb406a270 con 0x7f9fb4163d80 2024-05-09T03:11:18.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.706+0000 7f9fab7fe640 1 -- 172.21.15.46:0/1757139520 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f9f9c0648c0 con 0x7f9fb4163d80 2024-05-09T03:11:18.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f940762d0 msgr2=0x7f9f940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f940762d0 0x7f9f940787e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4006fd0 tx=0x7f9fa4009060 comp rx=0 tx=0).stop 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 msgr2=0x7f9fb40cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb40cefd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c00c820 tx=0x7f9f9c00cce0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 shutdown_connections 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fb415e660 0x7f9fb40cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f940762d0 0x7f9f940787e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb4163d80 0x7f9fb40cefd0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 --2- 172.21.15.46:0/1757139520 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9fb40b4d80 0x7f9fb415de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 >> 172.21.15.46:0/1757139520 conn(0x7f9fb40d2760 msgr2=0x7f9fb40bb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 shutdown_connections 2024-05-09T03:11:18.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.709+0000 7f9fb8a43640 1 -- 172.21.15.46:0/1757139520 wait complete. 2024-05-09T03:11:18.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-05-09T03:11:18.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.834+0000 7fc170765640 1 Processor -- start 2024-05-09T03:11:18.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.834+0000 7fc170765640 1 -- start start 2024-05-09T03:11:18.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc170765640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1680b9310 0x7fc16808cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc170765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc168164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc170765640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc16806ebe0 con 0x7fc16808d2a0 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc170765640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc16806f760 con 0x7fc1680b9310 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc170765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc16806eec0 con 0x7fc16808d6b0 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16ed02640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc168164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16dd00640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc16808d2a0 0x7fc168164830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:48852/0 (socket says 172.21.15.46:48852) 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16e501640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1680b9310 0x7fc16808cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16dd00640 1 -- 172.21.15.46:0/1578937226 learned_addr learned my addr 172.21.15.46:0/1578937226 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16ed02640 1 -- 172.21.15.46:0/1578937226 >> v2:172.21.15.88:3300/0 conn(0x7fc1680b9310 msgr2=0x7fc16808cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16ed02640 1 --2- 172.21.15.46:0/1578937226 >> v2:172.21.15.88:3300/0 conn(0x7fc1680b9310 0x7fc16808cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16ed02640 1 -- 172.21.15.46:0/1578937226 >> v1:172.21.15.88:6789/0 conn(0x7fc16808d2a0 legacy=0x7fc168164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16ed02640 1 -- 172.21.15.46:0/1578937226 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16806f1a0 con 0x7fc16808d6b0 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.836+0000 7fc16e501640 1 --2- 172.21.15.46:0/1578937226 >> v2:172.21.15.88:3300/0 conn(0x7fc1680b9310 0x7fc16808cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.837+0000 7fc16ed02640 1 --2- 172.21.15.46:0/1578937226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc168164f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc1600097a0 tx=0x7fc16002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad2185d0336754c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:18.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.837+0000 7fc16d4ff640 1 -- 172.21.15.46:0/1578937226 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160037070 con 0x7fc16808d6b0 2024-05-09T03:11:18.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc16d4ff640 1 -- 172.21.15.46:0/1578937226 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc160032440 con 0x7fc16808d6b0 2024-05-09T03:11:18.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc16d4ff640 1 -- 172.21.15.46:0/1578937226 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160032810 con 0x7fc16808d6b0 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 -- 172.21.15.46:0/1578937226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 msgr2=0x7fc168164f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 --2- 172.21.15.46:0/1578937226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc168164f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc1600097a0 tx=0x7fc16002a180 comp rx=0 tx=0).stop 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 -- 172.21.15.46:0/1578937226 shutdown_connections 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 --2- 172.21.15.46:0/1578937226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc168164f40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 --2- 172.21.15.46:0/1578937226 >> v2:172.21.15.88:3300/0 conn(0x7fc1680b9310 0x7fc16808cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 -- 172.21.15.46:0/1578937226 >> 172.21.15.46:0/1578937226 conn(0x7fc1680b5660 msgr2=0x7fc1680b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 -- 172.21.15.46:0/1578937226 shutdown_connections 2024-05-09T03:11:18.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.838+0000 7fc170765640 1 -- 172.21.15.46:0/1578937226 wait complete. 2024-05-09T03:11:18.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.840+0000 7fc170765640 1 Processor -- start 2024-05-09T03:11:18.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.840+0000 7fc170765640 1 -- start start 2024-05-09T03:11:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.841+0000 7fc170765640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16808d2a0 0x7fc168155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.841+0000 7fc170765640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc1681fa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.841+0000 7fc170765640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 0x7fc16815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.841+0000 7fc170765640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc168165480 con 0x7fc1680b9310 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.841+0000 7fc170765640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc16808dac0 con 0x7fc16808d2a0 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.841+0000 7fc170765640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc168165600 con 0x7fc16808d6b0 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 0x7fc16815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 0x7fc16815e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35572/0 (socket says 172.21.15.46:35572) 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 -- 172.21.15.46:0/3911445557 learned_addr learned my addr 172.21.15.46:0/3911445557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16dd00640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc1681fa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16e501640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16808d2a0 0x7fc168155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 -- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 msgr2=0x7fc1681fa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc1681fa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 -- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16808d2a0 msgr2=0x7fc168155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16808d2a0 0x7fc168155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 -- 172.21.15.46:0/3911445557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16815a9a0 con 0x7fc1680b9310 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc16ed02640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 0x7fc16815e5e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc160035040 tx=0x7fc1600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:18.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160037040 con 0x7fc1680b9310 2024-05-09T03:11:18.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1681fa8f0 con 0x7fc1680b9310 2024-05-09T03:11:18.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc160042070 con 0x7fc1680b9310 2024-05-09T03:11:18.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1600308a0 con 0x7fc1680b9310 2024-05-09T03:11:18.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.842+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1681f9bb0 con 0x7fc1680b9310 2024-05-09T03:11:18.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.843+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16806a270 con 0x7fc1680b9310 2024-05-09T03:11:18.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.844+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc160030a40 con 0x7fc1680b9310 2024-05-09T03:11:18.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.845+0000 7fc15effd640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc14c0762d0 0x7fc14c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:18.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.845+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7fc1600b7b10 con 0x7fc1680b9310 2024-05-09T03:11:18.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.845+0000 7fc16e501640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc14c0762d0 0x7fc14c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:18.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.846+0000 7fc16e501640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc14c0762d0 0x7fc14c0787e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc164009bd0 tx=0x7fc16400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:18.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:18.850+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc16004c020 con 0x7fc1680b9310 2024-05-09T03:11:19.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.112+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) -- 0x7fc1680ce540 con 0x7fc1680b9310 2024-05-09T03:11:19.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.356+0000 7fc15effd640 1 -- 172.21.15.46:0/3911445557 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v107) ==== 134+0+0 (secure 0 0 0) 0x7fc160080fb0 con 0x7fc1680b9310 2024-05-09T03:11:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc14c0762d0 msgr2=0x7fc14c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc14c0762d0 0x7fc14c0787e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc164009bd0 tx=0x7fc16400b040 comp rx=0 tx=0).stop 2024-05-09T03:11:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 msgr2=0x7fc16815e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 0x7fc16815e5e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc160035040 tx=0x7fc1600046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 shutdown_connections 2024-05-09T03:11:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc14c0762d0 0x7fc14c0787e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680b9310 0x7fc16815e5e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc16808d6b0 0x7fc1681fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 --2- 172.21.15.46:0/3911445557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16808d2a0 0x7fc168155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 >> 172.21.15.46:0/3911445557 conn(0x7fc1680b5660 msgr2=0x7fc1680ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 shutdown_connections 2024-05-09T03:11:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.359+0000 7fc170765640 1 -- 172.21.15.46:0/3911445557 wait complete. 2024-05-09T03:11:19.371 INFO:tasks.cephfs_test_runner:test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) ... ok 2024-05-09T03:11:19.371 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) That the MDS won't abort when the dentry is already known to be damaged. 2024-05-09T03:11:19.372 INFO:tasks.cephfs_test_runner:test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) 2024-05-09T03:11:19.372 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:19.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.485+0000 7fc22e76b640 1 Processor -- start 2024-05-09T03:11:19.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.485+0000 7fc22e76b640 1 -- start start 2024-05-09T03:11:19.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc22e76b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 0x7fc2280b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc22e76b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2280b8130 con 0x7fc2280d57a0 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc22e76b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2280d49e0 con 0x7fc2280b7d20 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc22e76b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc2280d5bb0 con 0x7fc228160610 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc227fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2280d57a0 0x7fc2280b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40690/0 (socket says 172.21.15.46:40690) 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc227fff640 1 -- 172.21.15.46:0/3710895327 learned_addr learned my addr 172.21.15.46:0/3710895327 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973715302 0 0) 0x7fc218001110 con 0x7fc2280d57a0 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc2277fe640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 0x7fc2280b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc210003520 con 0x7fc2280d57a0 2024-05-09T03:11:19.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3065129740 0 0) 0x7fc210003520 con 0x7fc2280d57a0 2024-05-09T03:11:19.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 msgr2=0x7fc2280b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.487+0000 7fc226ffd640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 0x7fc2280b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 >> v1:172.21.15.88:6789/0 conn(0x7fc2280b7d20 legacy=0x7fc2280b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2280d4cc0 con 0x7fc2280d57a0 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fc2180018c0 con 0x7fc2280d57a0 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 >> v1:172.21.15.46:6789/0 conn(0x7fc2280d57a0 legacy=0x7fc2280b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210003a80 0x7fc210003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 0x7fc210004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc210004d90 0x7fc210005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc2280d5bb0 con 0x7fc210004d90 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000bc50 con 0x7fc210003a80 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc21000be50 con 0x7fc228160610 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc22cd08640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc210004d90 0x7fc210005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc227fff640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210003a80 0x7fc210003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc2277fe640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 0x7fc210004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc22cd08640 1 -- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 msgr2=0x7fc210004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc22cd08640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228160610 0x7fc210004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc22cd08640 1 -- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210003a80 msgr2=0x7fc210003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc22cd08640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210003a80 0x7fc210003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.488+0000 7fc22cd08640 1 -- 172.21.15.46:0/3710895327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc210003520 con 0x7fc210004d90 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.489+0000 7fc22cd08640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc210004d90 0x7fc210005240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc220001e60 tx=0x7fc210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb11366ae1d54901 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.489+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220001ab0 con 0x7fc210004d90 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.489+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc22002ae40 con 0x7fc210004d90 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.489+0000 7fc226ffd640 1 -- 172.21.15.46:0/3710895327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220003b70 con 0x7fc210004d90 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22e76b640 1 -- 172.21.15.46:0/3710895327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc210004d90 msgr2=0x7fc210005240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22e76b640 1 --2- 172.21.15.46:0/3710895327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc210004d90 0x7fc210005240 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc220001e60 tx=0x7fc210005830 comp rx=0 tx=0).stop 2024-05-09T03:11:19.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22cd08640 1 -- 172.21.15.46:0/3710895327 reap_dead start 2024-05-09T03:11:19.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22e76b640 1 -- 172.21.15.46:0/3710895327 shutdown_connections 2024-05-09T03:11:19.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22e76b640 1 -- 172.21.15.46:0/3710895327 >> 172.21.15.46:0/3710895327 conn(0x7fc2280743e0 msgr2=0x7fc2280d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:19.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22e76b640 1 -- 172.21.15.46:0/3710895327 shutdown_connections 2024-05-09T03:11:19.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.490+0000 7fc22e76b640 1 -- 172.21.15.46:0/3710895327 wait complete. 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.492+0000 7fc22e76b640 1 Processor -- start 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.492+0000 7fc22e76b640 1 -- start start 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22e76b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210004d90 0x7fc2281f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22e76b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228153c80 0x7fc228157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22e76b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 0x7fc228158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22e76b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc228160a20 con 0x7fc228154090 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22e76b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2280d5280 con 0x7fc210004d90 2024-05-09T03:11:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22e76b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc2280d15b0 con 0x7fc228153c80 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22cd08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 0x7fc228158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22cd08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 0x7fc228158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35592/0 (socket says 172.21.15.46:35592) 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.493+0000 7fc22cd08640 1 -- 172.21.15.46:0/215162903 learned_addr learned my addr 172.21.15.46:0/215162903 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc2277fe640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228153c80 0x7fc228157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc227fff640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210004d90 0x7fc2281f5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22cd08640 1 -- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228153c80 msgr2=0x7fc228157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22cd08640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228153c80 0x7fc228157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22cd08640 1 -- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210004d90 msgr2=0x7fc2281f5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22cd08640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210004d90 0x7fc2281f5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22cd08640 1 -- 172.21.15.46:0/215162903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc228158a40 con 0x7fc228154090 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22cd08640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 0x7fc228158500 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc220001f80 tx=0x7fc2200014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220007df0 con 0x7fc228154090 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc220041070 con 0x7fc228154090 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2200050d0 con 0x7fc228154090 2024-05-09T03:11:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2281547f0 con 0x7fc228154090 2024-05-09T03:11:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.494+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc228154db0 con 0x7fc228154090 2024-05-09T03:11:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.495+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc22806a270 con 0x7fc228154090 2024-05-09T03:11:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.496+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc22003a070 con 0x7fc228154090 2024-05-09T03:11:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.496+0000 7fc224ff9640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc214076300 0x7fc214078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.497+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7fc2200b6e40 con 0x7fc228154090 2024-05-09T03:11:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.497+0000 7fc227fff640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc214076300 0x7fc214078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.497+0000 7fc227fff640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc214076300 0x7fc214078810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc218007fd0 tx=0x7fc218009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:19.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.501+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc220080910 con 0x7fc228154090 2024-05-09T03:11:19.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.690+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2280ba2e0 con 0x7fc228154090 2024-05-09T03:11:19.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.691+0000 7fc224ff9640 1 -- 172.21.15.46:0/215162903 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13078 (secure 0 0 0) 0x7fc220080210 con 0x7fc228154090 2024-05-09T03:11:19.692 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:19.692 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":35,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:18.349081+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/2536776736":"2024-05-10T03:11:15.336546+0000","172.21.15.46:6832/2536776736":"2024-05-10T03:11:15.336546+0000","172.21.15.46:6834/3490266983":"2024-05-10T03:10:29.845801+0000","172.21.15.46:6832/3490266983":"2024-05-10T03:10:29.845801+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc214076300 msgr2=0x7fc214078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc214076300 0x7fc214078810 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc218007fd0 tx=0x7fc218009040 comp rx=0 tx=0).stop 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 msgr2=0x7fc228158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 0x7fc228158500 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc220001f80 tx=0x7fc2200014a0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 shutdown_connections 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc214076300 0x7fc214078810 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc228154090 0x7fc228158500 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc228153c80 0x7fc228157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 --2- 172.21.15.46:0/215162903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc210004d90 0x7fc2281f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 >> 172.21.15.46:0/215162903 conn(0x7fc2280743e0 msgr2=0x7fc2280baae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:19.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 shutdown_connections 2024-05-09T03:11:19.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.693+0000 7fc22e76b640 1 -- 172.21.15.46:0/215162903 wait complete. 2024-05-09T03:11:19.705 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T03:09:22.506928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T03:11:19.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T03:11:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.822+0000 7f4affa2b640 1 Processor -- start 2024-05-09T03:11:19.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.822+0000 7f4affa2b640 1 -- start start 2024-05-09T03:11:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4affa2b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4af80d3180 0x7f4af81642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4affa2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af8164900 con 0x7f4af80bb770 2024-05-09T03:11:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4affa2b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af8164be0 con 0x7f4af8163eb0 2024-05-09T03:11:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4affa2b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4af80d4bb0 con 0x7f4af80d3180 2024-05-09T03:11:19.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4afd7c7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4af80bb770 0x7f4af80d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40706/0 (socket says 172.21.15.46:40706) 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4afd7c7640 1 -- 172.21.15.46:0/1191725279 learned_addr learned my addr 172.21.15.46:0/1191725279 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.823+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1191725279 >> v2:172.21.15.88:3300/0 conn(0x7f4af80d3180 0x7f4af81642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518912090 0 0) 0x7f4af8164900 con 0x7f4af80bb770 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ae0003520 con 0x7f4af80bb770 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1375691435 0 0) 0x7f4af8164be0 con 0x7f4af8163eb0 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4af8164900 con 0x7f4af8163eb0 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2894715725 0 0) 0x7f4ae0003520 con 0x7f4af80bb770 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 >> v2:172.21.15.88:3300/0 conn(0x7f4af80d3180 msgr2=0x7f4af81642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 --2- 172.21.15.46:0/1191725279 >> v2:172.21.15.88:3300/0 conn(0x7f4af80d3180 0x7f4af81642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 >> v1:172.21.15.88:6789/0 conn(0x7f4af8163eb0 legacy=0x7f4af80d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af80d4840 con 0x7f4af80bb770 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1191725279 >> v2:172.21.15.88:3300/0 conn(0x7f4af80d3180 0x7f4af81642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f4af00016f0 con 0x7f4af80bb770 2024-05-09T03:11:19.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 >> v1:172.21.15.46:6789/0 conn(0x7f4af80bb770 legacy=0x7f4af80d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ae0003c30 0x7f4ae00040a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0004800 0x7f4ae0004c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00052a0 0x7f4ae000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4af80d4bb0 con 0x7f4ae0003c30 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4ae000c230 con 0x7f4ae00052a0 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae000c430 con 0x7f4ae0004800 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ae0003c30 0x7f4ae00040a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00052a0 0x7f4ae000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.824+0000 7f4afcfc6640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0004800 0x7f4ae0004c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afd7c7640 1 -- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0004800 msgr2=0x7f4ae0004c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0004800 0x7f4ae0004c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afd7c7640 1 -- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00052a0 msgr2=0x7f4ae000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00052a0 0x7f4ae000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afd7c7640 1 -- 172.21.15.46:0/1191725279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae0003520 con 0x7f4ae0003c30 2024-05-09T03:11:19.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ae00052a0 0x7f4ae000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afdfc8640 1 -- 172.21.15.46:0/1191725279 reap_dead start 2024-05-09T03:11:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ae0003c30 0x7f4ae00040a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4af000c160 tx=0x7f4ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2d91134c6c6bdc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0007c80 con 0x7f4ae0003c30 2024-05-09T03:11:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4af0005af0 con 0x7f4ae0003c30 2024-05-09T03:11:19.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.825+0000 7f4af7fff640 1 -- 172.21.15.46:0/1191725279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0007070 con 0x7f4ae0003c30 2024-05-09T03:11:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 -- 172.21.15.46:0/1191725279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ae0003c30 msgr2=0x7f4ae00040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ae0003c30 0x7f4ae00040a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4af000c160 tx=0x7f4ae0005830 comp rx=0 tx=0).stop 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 -- 172.21.15.46:0/1191725279 shutdown_connections 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1191725279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ae0003c30 0x7f4ae00040a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 -- 172.21.15.46:0/1191725279 >> 172.21.15.46:0/1191725279 conn(0x7f4af806e370 msgr2=0x7f4af806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 -- 172.21.15.46:0/1191725279 shutdown_connections 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.826+0000 7f4affa2b640 1 -- 172.21.15.46:0/1191725279 wait complete. 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.828+0000 7f4affa2b640 1 Processor -- start 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.828+0000 7f4affa2b640 1 -- start start 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0003c30 0x7f4af814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4af814e370 0x7f4af80cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 0x7f4af80cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4af80d0d40 con 0x7f4af80ce670 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4af80d0bc0 con 0x7f4af814e370 2024-05-09T03:11:19.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4af81642c0 con 0x7f4ae0003c30 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 0x7f4af80cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 0x7f4af80cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35606/0 (socket says 172.21.15.46:35606) 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 -- 172.21.15.46:0/1875030113 learned_addr learned my addr 172.21.15.46:0/1875030113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0003c30 0x7f4af814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afcfc6640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4af814e370 0x7f4af80cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 -- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0003c30 msgr2=0x7f4af814eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0003c30 0x7f4af814eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 -- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4af814e370 msgr2=0x7f4af80cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4af814e370 0x7f4af80cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 -- 172.21.15.46:0/1875030113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af80cc190 con 0x7f4af80ce670 2024-05-09T03:11:19.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0003c30 0x7f4af814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afcfc6640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4af814e370 0x7f4af80cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4afdfc8640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 0x7f4af80cea80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4af80c9460 tx=0x7f4ae8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8009860 con 0x7f4af80ce670 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae801b070 con 0x7f4af80ce670 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af80cd5d0 con 0x7f4af80ce670 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.830+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800f8f0 con 0x7f4af80ce670 2024-05-09T03:11:19.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.831+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af80cdb90 con 0x7f4af80ce670 2024-05-09T03:11:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.831+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af80cd340 con 0x7f4af80ce670 2024-05-09T03:11:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.832+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4ae8012040 con 0x7f4af80ce670 2024-05-09T03:11:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.833+0000 7f4af5ffb640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4ae4076300 0x7f4ae4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.833+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7f4ae8097e80 con 0x7f4af80ce670 2024-05-09T03:11:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.833+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4ae4076300 0x7f4ae4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:19.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.834+0000 7f4afd7c7640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4ae4076300 0x7f4ae4078810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4af0032000 tx=0x7f4af0009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:19.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:19.837+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4ae8061ad0 con 0x7f4af80ce670 2024-05-09T03:11:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.034+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4af806a270 con 0x7f4af80ce670 2024-05-09T03:11:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.034+0000 7f4af5ffb640 1 -- 172.21.15.46:0/1875030113 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7f4ae80613d0 con 0x7f4af80ce670 2024-05-09T03:11:20.035 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-05-09T03:11:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4ae4076300 msgr2=0x7f4ae4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4ae4076300 0x7f4ae4078810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4af0032000 tx=0x7f4af0009030 comp rx=0 tx=0).stop 2024-05-09T03:11:20.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 msgr2=0x7f4af80cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 0x7f4af80cea80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4af80c9460 tx=0x7f4ae8009f80 comp rx=0 tx=0).stop 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 shutdown_connections 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4ae4076300 0x7f4ae4078810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4af80ce670 0x7f4af80cea80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4af814e370 0x7f4af80cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 --2- 172.21.15.46:0/1875030113 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4ae0003c30 0x7f4af814eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.036+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 >> 172.21.15.46:0/1875030113 conn(0x7f4af806e370 msgr2=0x7f4af80cfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.037+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 shutdown_connections 2024-05-09T03:11:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.037+0000 7f4affa2b640 1 -- 172.21.15.46:0/1875030113 wait complete. 2024-05-09T03:11:20.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-05-09T03:11:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.163+0000 7fe503b72640 1 Processor -- start 2024-05-09T03:11:20.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.163+0000 7fe503b72640 1 -- start start 2024-05-09T03:11:20.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe503b72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4fc06e160 0x7fe4fc160670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:20.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe503b72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe4fc0d23b0 0x7fe4fc0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:20.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe503b72640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4fc0d2d20 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe503b72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe4fc0d2ad0 con 0x7fe4fc06e160 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe503b72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe4fc0d3650 con 0x7fe4fc0d23b0 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe50110d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4fc06e160 0x7fe4fc160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe50110d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4fc06e160 0x7fe4fc160670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35622/0 (socket says 172.21.15.46:35622) 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe50110d640 1 -- 172.21.15.46:0/149271194 learned_addr learned my addr 172.21.15.46:0/149271194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.164+0000 7fe50210f640 1 --2- 172.21.15.46:0/149271194 >> v2:172.21.15.88:3300/0 conn(0x7fe4fc0d23b0 0x7fe4fc0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.165+0000 7fe50110d640 1 -- 172.21.15.46:0/149271194 >> v2:172.21.15.88:3300/0 conn(0x7fe4fc0d23b0 msgr2=0x7fe4fc0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.165+0000 7fe50110d640 1 --2- 172.21.15.46:0/149271194 >> v2:172.21.15.88:3300/0 conn(0x7fe4fc0d23b0 0x7fe4fc0cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.165+0000 7fe50110d640 1 -- 172.21.15.46:0/149271194 >> v1:172.21.15.88:6789/0 conn(0x7fe4fc0cfeb0 legacy=0x7fe4fc0d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:20.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.165+0000 7fe50110d640 1 -- 172.21.15.46:0/149271194 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4fc0d3000 con 0x7fe4fc06e160 2024-05-09T03:11:20.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.165+0000 7fe50110d640 1 --2- 172.21.15.46:0/149271194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc06e160 0x7fe4fc160670 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc074290 tx=0x7fe4f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9558b46a8ff90b05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:20.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe50090c640 1 -- 172.21.15.46:0/149271194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0037070 con 0x7fe4fc06e160 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe50090c640 1 -- 172.21.15.46:0/149271194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f0032440 con 0x7fe4fc06e160 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe50090c640 1 -- 172.21.15.46:0/149271194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0032810 con 0x7fe4fc06e160 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 -- 172.21.15.46:0/149271194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc06e160 msgr2=0x7fe4fc160670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 --2- 172.21.15.46:0/149271194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc06e160 0x7fe4fc160670 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc074290 tx=0x7fe4f002a270 comp rx=0 tx=0).stop 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 -- 172.21.15.46:0/149271194 shutdown_connections 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 --2- 172.21.15.46:0/149271194 >> v2:172.21.15.88:3300/0 conn(0x7fe4fc0d23b0 0x7fe4fc0cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 --2- 172.21.15.46:0/149271194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc06e160 0x7fe4fc160670 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 -- 172.21.15.46:0/149271194 >> 172.21.15.46:0/149271194 conn(0x7fe4fc09b7d0 msgr2=0x7fe4fc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 -- 172.21.15.46:0/149271194 shutdown_connections 2024-05-09T03:11:20.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.166+0000 7fe503b72640 1 -- 172.21.15.46:0/149271194 wait complete. 2024-05-09T03:11:20.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.168+0000 7fe503b72640 1 Processor -- start 2024-05-09T03:11:20.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.168+0000 7fe503b72640 1 -- start start 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4fc06e160 0x7fe4fc15c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 0x7fe4fc162480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4fc0d23b0 0x7fe4fc1629c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4fc0cd850 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe4fc0cd6d0 con 0x7fe4fc06e160 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4fc0cd9d0 con 0x7fe4fc0d23b0 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 0x7fe4fc162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 0x7fe4fc162480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35634/0 (socket says 172.21.15.46:35634) 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 -- 172.21.15.46:0/746401015 learned_addr learned my addr 172.21.15.46:0/746401015 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50190e640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4fc06e160 0x7fe4fc15c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50210f640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4fc0d23b0 0x7fe4fc1629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 -- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4fc0d23b0 msgr2=0x7fe4fc1629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:20.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4fc0d23b0 0x7fe4fc1629c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 -- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4fc06e160 msgr2=0x7fe4fc15c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4fc06e160 0x7fe4fc15c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 -- 172.21.15.46:0/746401015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4fc0cb6f0 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50190e640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4fc06e160 0x7fe4fc15c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe50110d640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 0x7fe4fc162480 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0000c00 tx=0x7fe4f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0041020 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f00306c0 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4fc161940 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0037040 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.170+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4fc0cbc40 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.171+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4fc06f470 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.172+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe4f0044020 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.173+0000 7fe4fa7fc640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4e80762d0 0x7fe4e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.173+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7fe4f00b7c00 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.173+0000 7fe50190e640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4e80762d0 0x7fe4e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:20.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.173+0000 7fe50190e640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4e80762d0 0x7fe4e80787e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec007fd0 tx=0x7fe4ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:20.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.177+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe4f00be080 con 0x7fe4fc0cfeb0 2024-05-09T03:11:20.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:20.442+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) -- 0x7fe4fc06a270 con 0x7fe4fc0cfeb0 2024-05-09T03:11:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.362+0000 7fe4fa7fc640 1 -- 172.21.15.46:0/746401015 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v109) ==== 144+0+0 (secure 0 0 0) 0x7fe4f004b030 con 0x7fe4fc0cfeb0 2024-05-09T03:11:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4e80762d0 msgr2=0x7fe4e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4e80762d0 0x7fe4e80787e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec007fd0 tx=0x7fe4ec009040 comp rx=0 tx=0).stop 2024-05-09T03:11:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 msgr2=0x7fe4fc162480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 0x7fe4fc162480 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0000c00 tx=0x7fe4f00046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 shutdown_connections 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4fc0d23b0 0x7fe4fc1629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4e80762d0 0x7fe4e80787e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4fc0cfeb0 0x7fe4fc162480 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 --2- 172.21.15.46:0/746401015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4fc06e160 0x7fe4fc15c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.365+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 >> 172.21.15.46:0/746401015 conn(0x7fe4fc09b7d0 msgr2=0x7fe4fc0d1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.366+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 shutdown_connections 2024-05-09T03:11:21.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.366+0000 7fe503b72640 1 -- 172.21.15.46:0/746401015 wait complete. 2024-05-09T03:11:21.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T03:11:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.504+0000 7f5b24986640 1 Processor -- start 2024-05-09T03:11:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.504+0000 7f5b24986640 1 -- start start 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b24986640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b24986640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b200d2a70 0x7f5b200d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b24986640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b200d33c0 con 0x7f5b200bb770 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b24986640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5b200d4bb0 con 0x7f5b200d2a70 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b24986640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5b20164900 con 0x7f5b20163eb0 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b1e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5b200bb770 0x7f5b200d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40718/0 (socket says 172.21.15.46:40718) 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.505+0000 7f5b1e59c640 1 -- 172.21.15.46:0/3317866547 learned_addr learned my addr 172.21.15.46:0/3317866547 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1ed9d640 1 --2- 172.21.15.46:0/3317866547 >> v2:172.21.15.88:3300/0 conn(0x7f5b200d2a70 0x7f5b200d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b17fff640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1dd9b640 1 -- 172.21.15.46:0/3317866547 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636084620 0 0) 0x7f5b10001110 con 0x7f5b200bb770 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1dd9b640 1 -- 172.21.15.46:0/3317866547 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b0c003520 con 0x7f5b200bb770 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1dd9b640 1 -- 172.21.15.46:0/3317866547 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 400012056 0 0) 0x7f5b0c003520 con 0x7f5b200bb770 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1ed9d640 1 -- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 msgr2=0x7f5b201642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1ed9d640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1ed9d640 1 -- 172.21.15.46:0/3317866547 >> v1:172.21.15.46:6789/0 conn(0x7f5b200bb770 legacy=0x7f5b200d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b1ed9d640 1 -- 172.21.15.46:0/3317866547 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b20164be0 con 0x7f5b200d2a70 2024-05-09T03:11:21.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.506+0000 7f5b17fff640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.507+0000 7f5b1ed9d640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200d2a70 0x7f5b200d2e80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009d90 tx=0x7f5b1802a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98f150c3b24fb2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.507+0000 7f5b1dd9b640 1 -- 172.21.15.46:0/3317866547 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b100018e0 con 0x7f5b200d2a70 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.507+0000 7f5b1dd9b640 1 -- 172.21.15.46:0/3317866547 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b18007540 con 0x7f5b200d2a70 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.507+0000 7f5b1dd9b640 1 -- 172.21.15.46:0/3317866547 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b18007910 con 0x7f5b200d2a70 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 -- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200d2a70 msgr2=0x7f5b200d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200d2a70 0x7f5b200d2e80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009d90 tx=0x7f5b1802a5f0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 -- 172.21.15.46:0/3317866547 shutdown_connections 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200d2a70 0x7f5b200d2e80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 --2- 172.21.15.46:0/3317866547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 -- 172.21.15.46:0/3317866547 >> 172.21.15.46:0/3317866547 conn(0x7f5b2006e370 msgr2=0x7f5b2006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 -- 172.21.15.46:0/3317866547 shutdown_connections 2024-05-09T03:11:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.508+0000 7f5b24986640 1 -- 172.21.15.46:0/3317866547 wait complete. 2024-05-09T03:11:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.510+0000 7f5b24986640 1 Processor -- start 2024-05-09T03:11:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.510+0000 7f5b24986640 1 -- start start 2024-05-09T03:11:21.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b24986640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200bb770 0x7f5b200ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:21.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b24986640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 0x7f5b201eba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:21.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b24986640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:21.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b24986640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b200d0bc0 con 0x7f5b200d2a70 2024-05-09T03:11:21.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b24986640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5b200bbb80 con 0x7f5b200bb770 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b24986640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5b200d0d40 con 0x7f5b20163eb0 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 0x7f5b201eba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 0x7f5b201eba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35636/0 (socket says 172.21.15.46:35636) 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b17fff640 1 -- 172.21.15.46:0/4105940463 learned_addr learned my addr 172.21.15.46:0/4105940463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.511+0000 7f5b1e59c640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200bb770 0x7f5b200ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b17fff640 1 -- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 msgr2=0x7f5b201ebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b1ed9d640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b17fff640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201ebf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b17fff640 1 -- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200bb770 msgr2=0x7f5b200ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b17fff640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200bb770 0x7f5b200ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b17fff640 1 -- 172.21.15.46:0/4105940463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2014c9f0 con 0x7f5b200d2a70 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b1e59c640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200bb770 0x7f5b200ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:21.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b17fff640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 0x7f5b201eba50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e960 tx=0x7f5b0800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:21.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08017070 con 0x7f5b200d2a70 2024-05-09T03:11:21.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0800ca00 con 0x7f5b200d2a70 2024-05-09T03:11:21.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0800cdd0 con 0x7f5b200d2a70 2024-05-09T03:11:21.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2015e340 con 0x7f5b200d2a70 2024-05-09T03:11:21.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.512+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2015e900 con 0x7f5b200d2a70 2024-05-09T03:11:21.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.513+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b200ca950 con 0x7f5b200d2a70 2024-05-09T03:11:21.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.514+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5b0801a070 con 0x7f5b200d2a70 2024-05-09T03:11:21.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.514+0000 7f5b16ffd640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b000762d0 0x7f5b000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:21.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.515+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7f5b08097250 con 0x7f5b200d2a70 2024-05-09T03:11:21.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.515+0000 7f5b1e59c640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b000762d0 0x7f5b000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:21.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.515+0000 7f5b1e59c640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b000762d0 0x7f5b000787e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b10001eb0 tx=0x7f5b10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:21.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.519+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5b08060dc0 con 0x7f5b200d2a70 2024-05-09T03:11:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.714+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5b2006a270 con 0x7f5b200d2a70 2024-05-09T03:11:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.718+0000 7f5b16ffd640 1 -- 172.21.15.46:0/4105940463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f5b080606c0 con 0x7f5b200d2a70 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b000762d0 msgr2=0x7f5b000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b000762d0 0x7f5b000787e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b10001eb0 tx=0x7f5b10009040 comp rx=0 tx=0).stop 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 msgr2=0x7f5b201eba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 0x7f5b201eba50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e960 tx=0x7f5b0800ee20 comp rx=0 tx=0).stop 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 shutdown_connections 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b20163eb0 0x7f5b201ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5b000762d0 0x7f5b000787e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b200d2a70 0x7f5b201eba50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 --2- 172.21.15.46:0/4105940463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b200bb770 0x7f5b200ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 >> 172.21.15.46:0/4105940463 conn(0x7f5b2006e370 msgr2=0x7f5b200b5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 shutdown_connections 2024-05-09T03:11:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:21.720+0000 7f5b24986640 1 -- 172.21.15.46:0/4105940463 wait complete. 2024-05-09T03:11:21.732 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:11:21.732 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:11:21.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:11:21.828 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:11:21.828 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:21.829 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:21.845 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:11:21.845 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:11:21.845 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:21.863 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:11:21.863 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:21.863 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:21.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:11:21.902 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:11:21.902 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:21.921 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:11:21.921 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T03:11:21.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:22.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.056+0000 7ff5e13e0640 1 Processor -- start 2024-05-09T03:11:22.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.056+0000 7ff5e13e0640 1 -- start start 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.057+0000 7ff5e13e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5dc0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.057+0000 7ff5e13e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5dc0cef80 con 0x7ff5dc0ce590 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.057+0000 7ff5e13e0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5dc0b2540 con 0x7ff5dc0d31e0 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.057+0000 7ff5e13e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff5dc0b2b90 con 0x7ff5dc0d0280 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.057+0000 7ff5d3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5dc0ce590 0x7ff5dc0d2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40728/0 (socket says 172.21.15.46:40728) 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.057+0000 7ff5d3fff640 1 -- 172.21.15.46:0/873507863 learned_addr learned my addr 172.21.15.46:0/873507863 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5daffd640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5dc0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404792143 0 0) 0x7ff5dc0cef80 con 0x7ff5dc0ce590 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5c8003520 con 0x7ff5dc0ce590 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 859319467 0 0) 0x7ff5c8003520 con 0x7ff5dc0ce590 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 msgr2=0x7ff5dc0b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5dc0b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 >> v1:172.21.15.88:6789/0 conn(0x7ff5dc0d31e0 legacy=0x7ff5dc0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc0b2820 con 0x7ff5dc0ce590 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ff5c40016f0 con 0x7ff5dc0ce590 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 >> v1:172.21.15.46:6789/0 conn(0x7ff5dc0ce590 legacy=0x7ff5dc0d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c8004d40 0x7ff5c8005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc0b2b90 con 0x7ff5c8004d40 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff5c800bb50 con 0x7ff5c80039e0 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff5c800bd00 con 0x7ff5dc0d0280 2024-05-09T03:11:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5daffd640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5db7fe640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c8004d40 0x7ff5c8005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.058+0000 7ff5d3fff640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5daffd640 1 -- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 msgr2=0x7ff5c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5daffd640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5daffd640 1 -- 172.21.15.46:0/873507863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c8004d40 msgr2=0x7ff5c8005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5daffd640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c8004d40 0x7ff5c8005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5daffd640 1 -- 172.21.15.46:0/873507863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c8003520 con 0x7ff5c80039e0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5d3fff640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc0d0280 0x7ff5c8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5db7fe640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5c8004d40 0x7ff5c8005170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.059+0000 7ff5daffd640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009ab0 tx=0x7ff5c8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c83a54bdba573ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc036070 con 0x7ff5c80039e0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5cc005990 con 0x7ff5c80039e0 2024-05-09T03:11:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5da7fc640 1 -- 172.21.15.46:0/873507863 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc009680 con 0x7ff5c80039e0 2024-05-09T03:11:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5e13e0640 1 -- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 msgr2=0x7ff5c8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/873507863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5c8003e10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009ab0 tx=0x7ff5c8005830 comp rx=0 tx=0).stop 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5db7fe640 1 -- 172.21.15.46:0/873507863 reap_dead start 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5e13e0640 1 -- 172.21.15.46:0/873507863 shutdown_connections 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5e13e0640 1 -- 172.21.15.46:0/873507863 >> 172.21.15.46:0/873507863 conn(0x7ff5dc09b7d0 msgr2=0x7ff5dc09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5e13e0640 1 -- 172.21.15.46:0/873507863 shutdown_connections 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.060+0000 7ff5e13e0640 1 -- 172.21.15.46:0/873507863 wait complete. 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.062+0000 7ff5e13e0640 1 Processor -- start 2024-05-09T03:11:22.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.062+0000 7ff5e13e0640 1 -- start start 2024-05-09T03:11:22.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5dc15c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 0x7ff5dc14ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc1f4200 0x7ff5dc1f4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc15e660 con 0x7ff5dc1611a0 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc15e4e0 con 0x7ff5c80039e0 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff5dc0b3730 con 0x7ff5dc1f4200 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 0x7ff5dc14ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 0x7ff5dc14ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35654/0 (socket says 172.21.15.46:35654) 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 -- 172.21.15.46:0/3759812868 learned_addr learned my addr 172.21.15.46:0/3759812868 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5daffd640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5dc15c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5db7fe640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc1f4200 0x7ff5dc1f4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 -- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc1f4200 msgr2=0x7ff5dc1f4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc1f4200 0x7ff5dc1f4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 -- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 msgr2=0x7ff5dc15c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5dc15c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 -- 172.21.15.46:0/3759812868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc1f4b50 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d3fff640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 0x7ff5dc14ee80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4007370 tx=0x7ff5c4007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c401a070 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c400fa50 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5dc15af80 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400fe40 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.064+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5dc15b540 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.066+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5dc06a270 con 0x7ff5dc1611a0 2024-05-09T03:11:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.066+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff5c401d070 con 0x7ff5dc1611a0 2024-05-09T03:11:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.067+0000 7ff5d37fe640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5bc0762d0 0x7ff5bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.067+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7ff5c4096f90 con 0x7ff5dc1611a0 2024-05-09T03:11:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.067+0000 7ff5daffd640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5bc0762d0 0x7ff5bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.067+0000 7ff5daffd640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5bc0762d0 0x7ff5bc0787e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc033000 tx=0x7ff5cc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:22.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.071+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff5c4060410 con 0x7ff5dc1611a0 2024-05-09T03:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.305+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5dc15ad10 con 0x7ff5dc1611a0 2024-05-09T03:11:22.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.306+0000 7ff5d37fe640 1 -- 172.21.15.46:0/3759812868 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+3866 (secure 0 0 0) 0x7ff5c4064710 con 0x7ff5dc1611a0 2024-05-09T03:11:22.307 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:22.307 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2687931875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2687931875},{"type":"v1","addr":"172.21.15.88:6836","nonce":2687931875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/508485608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":508485608},{"type":"v1","addr":"172.21.15.88:6837","nonce":508485608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4634,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/4145068237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":4145068237},{"type":"v1","addr":"172.21.15.46:6835","nonce":4145068237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5286,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1101093163","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1101093163},{"type":"v1","addr":"172.21.15.46:6834","nonce":1101093163}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":23}],"filesystems":[]} 2024-05-09T03:11:22.307 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-05-09T03:11:22.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5bc0762d0 msgr2=0x7ff5bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5bc0762d0 0x7ff5bc0787e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc033000 tx=0x7ff5cc008030 comp rx=0 tx=0).stop 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 msgr2=0x7ff5dc14ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 0x7ff5dc14ee80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4007370 tx=0x7ff5c4007830 comp rx=0 tx=0).stop 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 shutdown_connections 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff5dc1f4200 0x7ff5dc1f4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5bc0762d0 0x7ff5bc0787e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1611a0 0x7ff5dc14ee80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 --2- 172.21.15.46:0/3759812868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5c80039e0 0x7ff5dc15c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 >> 172.21.15.46:0/3759812868 conn(0x7ff5dc09b7d0 msgr2=0x7ff5dc162290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 shutdown_connections 2024-05-09T03:11:22.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.308+0000 7ff5e13e0640 1 -- 172.21.15.46:0/3759812868 wait complete. 2024-05-09T03:11:22.320 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:11:22.320 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:11:22.321 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:11:22.321 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:11:22.321 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:11:22.321 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T03:11:22.321 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:11:22.321 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:11:22.321 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:11:22.322 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T03:11:22.322 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:11:22.322 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:11:22.322 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:11:22.322 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T03:11:22.322 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:11:22.322 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:11:22.331 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:11:22.331 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:11:22.334 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:11:22.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:11:22.335 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:11:22.335 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:11:22.336 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:11:22.336 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:11:22.375 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:11:22.376 DEBUG:teuthology.parallel:result is None 2024-05-09T03:11:22.377 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:11:22.377 DEBUG:teuthology.parallel:result is None 2024-05-09T03:11:22.378 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:11:22.378 DEBUG:teuthology.parallel:result is None 2024-05-09T03:11:22.379 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:11:22.379 DEBUG:teuthology.parallel:result is None 2024-05-09T03:11:22.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T03:11:22.465 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:11:22.466 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:11:22.471 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:11:22.471 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:11:22.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.500+0000 7f6c76b9f640 1 Processor -- start 2024-05-09T03:11:22.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.500+0000 7f6c76b9f640 1 -- start start 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c76b9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d2930 0x7f6c700d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c76b9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c700b9300 0x7f6c700b9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c76b9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c700d3280 con 0x7f6c700d4650 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c76b9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6c700d5760 con 0x7f6c700b9300 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c76b9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6c700d4b50 con 0x7f6c700d2930 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c7639e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c700b9300 0x7f6c700b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c75b9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6c700d4650 0x7f6c700b4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40740/0 (socket says 172.21.15.46:40740) 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c75b9d640 1 -- 172.21.15.46:0/3287891073 learned_addr learned my addr 172.21.15.46:0/3287891073 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c7539c640 1 --2- 172.21.15.46:0/3287891073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d2930 0x7f6c700d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c7639e640 1 -- 172.21.15.46:0/3287891073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d2930 msgr2=0x7f6c700d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c7639e640 1 --2- 172.21.15.46:0/3287891073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d2930 0x7f6c700d2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c7639e640 1 -- 172.21.15.46:0/3287891073 >> v1:172.21.15.46:6789/0 conn(0x7f6c700d4650 legacy=0x7f6c700b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.502+0000 7f6c7639e640 1 -- 172.21.15.46:0/3287891073 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c700d4e30 con 0x7f6c700b9300 2024-05-09T03:11:22.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.503+0000 7f6c7639e640 1 --2- 172.21.15.46:0/3287891073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700b9300 0x7f6c700b9710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009d90 tx=0x7f6c6802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6865bb853c5ef6c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.503+0000 7f6c74b9b640 1 -- 172.21.15.46:0/3287891073 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68031600 con 0x7f6c700b9300 2024-05-09T03:11:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.503+0000 7f6c74b9b640 1 -- 172.21.15.46:0/3287891073 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c680317a0 con 0x7f6c700b9300 2024-05-09T03:11:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.503+0000 7f6c74b9b640 1 -- 172.21.15.46:0/3287891073 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68031b70 con 0x7f6c700b9300 2024-05-09T03:11:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3287891073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700b9300 msgr2=0x7f6c700b9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3287891073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700b9300 0x7f6c700b9710 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009d90 tx=0x7f6c6802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3287891073 shutdown_connections 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3287891073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700b9300 0x7f6c700b9710 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3287891073 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d2930 0x7f6c700d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3287891073 >> 172.21.15.46:0/3287891073 conn(0x7f6c7006e110 msgr2=0x7f6c7006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3287891073 shutdown_connections 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.504+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3287891073 wait complete. 2024-05-09T03:11:22.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.506+0000 7f6c76b9f640 1 Processor -- start 2024-05-09T03:11:22.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.506+0000 7f6c76b9f640 1 -- start start 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c76b9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c700b9300 0x7f6c70153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c76b9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 0x7f6c700cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c76b9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d4650 0x7f6c70161f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c76b9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6c70160540 con 0x7f6c700d2930 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c76b9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6c700d53f0 con 0x7f6c700b9300 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c76b9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6c701606c0 con 0x7f6c700d4650 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 0x7f6c700cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 0x7f6c700cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35708/0 (socket says 172.21.15.46:35708) 2024-05-09T03:11:22.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 -- 172.21.15.46:0/3149614109 learned_addr learned my addr 172.21.15.46:0/3149614109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c75b9d640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c700b9300 0x7f6c70153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7639e640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d4650 0x7f6c70161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 -- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d4650 msgr2=0x7f6c70161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d4650 0x7f6c70161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 -- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c700b9300 msgr2=0x7f6c70153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c700b9300 0x7f6c70153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.507+0000 7f6c7539c640 1 -- 172.21.15.46:0/3149614109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c70153a40 con 0x7f6c700d2930 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.508+0000 7f6c7539c640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 0x7f6c700cbca0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000e8a0 tx=0x7f6c6000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.508+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60017070 con 0x7f6c700d2930 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.508+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6000c940 con 0x7f6c700d2930 2024-05-09T03:11:22.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.508+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000cd30 con 0x7f6c700d2930 2024-05-09T03:11:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.508+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c700ca3e0 con 0x7f6c700d2930 2024-05-09T03:11:22.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.508+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c700ca9a0 con 0x7f6c700d2930 2024-05-09T03:11:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.509+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c70076c00 con 0x7f6c700d2930 2024-05-09T03:11:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.509+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6c6001a070 con 0x7f6c700d2930 2024-05-09T03:11:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.510+0000 7f6c667fc640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c540762d0 0x7f6c540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.510+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5029+0+0 (secure 0 0 0) 0x7f6c60097480 con 0x7f6c700d2930 2024-05-09T03:11:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.510+0000 7f6c75b9d640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c540762d0 0x7f6c540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:22.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.511+0000 7f6c75b9d640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c540762d0 0x7f6c540787e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c000f60 tx=0x7f6c6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:22.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.515+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6c60060ac0 con 0x7f6c700d2930 2024-05-09T03:11:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:22.719+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6c70163b30 con 0x7f6c700d2930 2024-05-09T03:11:23.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.379+0000 7f6c667fc640 1 -- 172.21.15.46:0/3149614109 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v37) ==== 97+0+0 (secure 0 0 0) 0x7f6c60064dc0 con 0x7f6c700d2930 2024-05-09T03:11:23.380 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-05-09T03:11:23.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c540762d0 msgr2=0x7f6c540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c540762d0 0x7f6c540787e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c000f60 tx=0x7f6c6c009040 comp rx=0 tx=0).stop 2024-05-09T03:11:23.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 msgr2=0x7f6c700cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 0x7f6c700cbca0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000e8a0 tx=0x7f6c6000ed60 comp rx=0 tx=0).stop 2024-05-09T03:11:23.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 shutdown_connections 2024-05-09T03:11:23.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c700d4650 0x7f6c70161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c540762d0 0x7f6c540787e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c700d2930 0x7f6c700cbca0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 --2- 172.21.15.46:0/3149614109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c700b9300 0x7f6c70153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 >> 172.21.15.46:0/3149614109 conn(0x7f6c7006e110 msgr2=0x7f6c700ba500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:23.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 shutdown_connections 2024-05-09T03:11:23.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.382+0000 7f6c76b9f640 1 -- 172.21.15.46:0/3149614109 wait complete. 2024-05-09T03:11:23.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T03:11:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.517+0000 7f34d2384640 1 Processor -- start 2024-05-09T03:11:23.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.518+0000 7f34d2384640 1 -- start start 2024-05-09T03:11:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34d2384640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34cc1615e0 0x7f34cc15dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34d2384640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34cc0d2480 0x7f34cc0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34d2384640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34cc160b20 con 0x7f34cc164c70 2024-05-09T03:11:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34d2384640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f34cc15e520 con 0x7f34cc1615e0 2024-05-09T03:11:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34d2384640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f34cc161150 con 0x7f34cc0d2480 2024-05-09T03:11:23.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34cc1615e0 0x7f34cc15dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34cc1615e0 0x7f34cc15dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35722/0 (socket says 172.21.15.46:35722) 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 -- 172.21.15.46:0/460448572 learned_addr learned my addr 172.21.15.46:0/460448572 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34d0921640 1 --2- 172.21.15.46:0/460448572 >> v2:172.21.15.88:3300/0 conn(0x7f34cc0d2480 0x7f34cc0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 -- 172.21.15.46:0/460448572 >> v2:172.21.15.88:3300/0 conn(0x7f34cc0d2480 msgr2=0x7f34cc0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 --2- 172.21.15.46:0/460448572 >> v2:172.21.15.88:3300/0 conn(0x7f34cc0d2480 0x7f34cc0d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 -- 172.21.15.46:0/460448572 >> v1:172.21.15.88:6789/0 conn(0x7f34cc164c70 legacy=0x7f34cc0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.519+0000 7f34cb7fe640 1 -- 172.21.15.46:0/460448572 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc0d15f0 con 0x7f34cc1615e0 2024-05-09T03:11:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.520+0000 7f34cb7fe640 1 --2- 172.21.15.46:0/460448572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc1615e0 0x7f34cc15dfe0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0d3e30 tx=0x7f34c002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a40e3b08b46e82a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.520+0000 7f34caffd640 1 -- 172.21.15.46:0/460448572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0037070 con 0x7f34cc1615e0 2024-05-09T03:11:23.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.520+0000 7f34caffd640 1 -- 172.21.15.46:0/460448572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c0032440 con 0x7f34cc1615e0 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.520+0000 7f34caffd640 1 -- 172.21.15.46:0/460448572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0032810 con 0x7f34cc1615e0 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 -- 172.21.15.46:0/460448572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc1615e0 msgr2=0x7f34cc15dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 --2- 172.21.15.46:0/460448572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc1615e0 0x7f34cc15dfe0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0d3e30 tx=0x7f34c002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 -- 172.21.15.46:0/460448572 shutdown_connections 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 --2- 172.21.15.46:0/460448572 >> v2:172.21.15.88:3300/0 conn(0x7f34cc0d2480 0x7f34cc0d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 --2- 172.21.15.46:0/460448572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc1615e0 0x7f34cc15dfe0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 -- 172.21.15.46:0/460448572 >> 172.21.15.46:0/460448572 conn(0x7f34cc073db0 msgr2=0x7f34cc0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 -- 172.21.15.46:0/460448572 shutdown_connections 2024-05-09T03:11:23.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.521+0000 7f34d2384640 1 -- 172.21.15.46:0/460448572 wait complete. 2024-05-09T03:11:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.523+0000 7f34d2384640 1 Processor -- start 2024-05-09T03:11:23.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.523+0000 7f34d2384640 1 -- start start 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.524+0000 7f34d2384640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34cc0d2480 0x7f34cc1e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d2384640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34cc1615e0 0x7f34cc0cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d2384640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 0x7f34cc0baad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d2384640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc1619f0 con 0x7f34cc164c70 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d2384640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc160de0 con 0x7f34cc1615e0 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d2384640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc0d2dd0 con 0x7f34cc0d2480 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 0x7f34cc0baad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 0x7f34cc0baad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35730/0 (socket says 172.21.15.46:35730) 2024-05-09T03:11:23.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 -- 172.21.15.46:0/2242509235 learned_addr learned my addr 172.21.15.46:0/2242509235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34cb7fe640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34cc1615e0 0x7f34cc0cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34cbfff640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34cc0d2480 0x7f34cc1e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 -- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34cc0d2480 msgr2=0x7f34cc1e9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34cc0d2480 0x7f34cc1e9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 -- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34cc1615e0 msgr2=0x7f34cc0cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34cc1615e0 0x7f34cc0cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 -- 172.21.15.46:0/2242509235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc1ea470 con 0x7f34cc164c70 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34cb7fe640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34cc1615e0 0x7f34cc0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34cbfff640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34cc0d2480 0x7f34cc1e9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34d0921640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 0x7f34cc0baad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f34c400efc0 tx=0x7f34c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.525+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4018020 con 0x7f34cc164c70 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.526+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c4014480 con 0x7f34cc164c70 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.526+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c4009e60 con 0x7f34cc164c70 2024-05-09T03:11:23.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.526+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc1ea8f0 con 0x7f34cc164c70 2024-05-09T03:11:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.526+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc0cd800 con 0x7f34cc164c70 2024-05-09T03:11:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.527+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34cc08cfd0 con 0x7f34cc164c70 2024-05-09T03:11:23.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.527+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34c4005d10 con 0x7f34cc164c70 2024-05-09T03:11:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.528+0000 7f34c8ff9640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34b8076300 0x7f34b8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.528+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4857+0+0 (secure 0 0 0) 0x7f34c4098030 con 0x7f34cc164c70 2024-05-09T03:11:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.528+0000 7f34cbfff640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34b8076300 0x7f34b8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.529+0000 7f34cbfff640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34b8076300 0x7f34b8078810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0016a0 tx=0x7f34bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:23.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.533+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34cc08cfd0 con 0x7f34cc164c70 2024-05-09T03:11:23.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.797+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f34cc1ea640 con 0x7f34cc164c70 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.797+0000 7f34c8ff9640 1 -- 172.21.15.46:0/2242509235 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) ==== 80+0+5304 (secure 0 0 0) 0x7f34c4022030 con 0x7f34cc164c70 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g==", 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.798 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw==", 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.799 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw==", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw==", 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw==", 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA==", 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.802 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg==", 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.803 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA==", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g==", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.804 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA==", 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ==", 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.805 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw==", 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.806 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw==", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ==", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:11:23.807 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg==", 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.808 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg==", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA1PjxmuVPDJhAAHOgxPpTSEd3pibPqobcHCg==", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:11:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg==", 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:23.810 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ==", 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-05-09T03:11:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34b8076300 msgr2=0x7f34b8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34b8076300 0x7f34b8078810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0016a0 tx=0x7f34bc009040 comp rx=0 tx=0).stop 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 msgr2=0x7f34cc0baad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 0x7f34cc0baad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f34c400efc0 tx=0x7f34c400c490 comp rx=0 tx=0).stop 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 shutdown_connections 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34b8076300 0x7f34b8078810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34cc164c70 0x7f34cc0baad0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34cc1615e0 0x7f34cc0cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 --2- 172.21.15.46:0/2242509235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34cc0d2480 0x7f34cc1e9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 >> 172.21.15.46:0/2242509235 conn(0x7f34cc073db0 msgr2=0x7f34cc0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:23.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 shutdown_connections 2024-05-09T03:11:23.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.799+0000 7f34d2384640 1 -- 172.21.15.46:0/2242509235 wait complete. 2024-05-09T03:11:23.814 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:11:23.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.929+0000 7f1139b94640 1 Processor -- start 2024-05-09T03:11:23.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.929+0000 7f1139b94640 1 -- start start 2024-05-09T03:11:23.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1139b94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11341613c0 0x7f11340b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1139b94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11340b7d00 0x7f1134163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1139b94640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11340b8110 con 0x7f11340b5210 2024-05-09T03:11:23.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1139b94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f11340b5620 con 0x7f11341613c0 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1139b94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11341609a0 con 0x7f11340b7d00 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1132ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11341613c0 0x7f11340b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1132ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11341613c0 0x7f11340b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35738/0 (socket says 172.21.15.46:35738) 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1132ffd640 1 -- 172.21.15.46:0/90057484 learned_addr learned my addr 172.21.15.46:0/90057484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1133fff640 1 --2- 172.21.15.46:0/90057484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11340b7d00 0x7f1134163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.930+0000 7f1132ffd640 1 -- 172.21.15.46:0/90057484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11340b7d00 msgr2=0x7f1134163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.931+0000 7f1132ffd640 1 --2- 172.21.15.46:0/90057484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11340b7d00 0x7f1134163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.931+0000 7f1132ffd640 1 -- 172.21.15.46:0/90057484 >> v1:172.21.15.88:6789/0 conn(0x7f11340b5210 legacy=0x7f11340d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.931+0000 7f1132ffd640 1 -- 172.21.15.46:0/90057484 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11341617d0 con 0x7f11341613c0 2024-05-09T03:11:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.931+0000 7f1132ffd640 1 --2- 172.21.15.46:0/90057484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11341613c0 0x7f11340b77c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1128009870 tx=0x7f112802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6273497573ebc80c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.931+0000 7f11327fc640 1 -- 172.21.15.46:0/90057484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128037070 con 0x7f11341613c0 2024-05-09T03:11:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f11327fc640 1 -- 172.21.15.46:0/90057484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1128032440 con 0x7f11341613c0 2024-05-09T03:11:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f11327fc640 1 -- 172.21.15.46:0/90057484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128032810 con 0x7f11341613c0 2024-05-09T03:11:23.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 -- 172.21.15.46:0/90057484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11341613c0 msgr2=0x7f11340b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 --2- 172.21.15.46:0/90057484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11341613c0 0x7f11340b77c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1128009870 tx=0x7f112802a270 comp rx=0 tx=0).stop 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 -- 172.21.15.46:0/90057484 shutdown_connections 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 --2- 172.21.15.46:0/90057484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11340b7d00 0x7f1134163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 --2- 172.21.15.46:0/90057484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11341613c0 0x7f11340b77c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 -- 172.21.15.46:0/90057484 >> 172.21.15.46:0/90057484 conn(0x7f11340d29a0 msgr2=0x7f11340d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 -- 172.21.15.46:0/90057484 shutdown_connections 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.932+0000 7f1139b94640 1 -- 172.21.15.46:0/90057484 wait complete. 2024-05-09T03:11:23.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.934+0000 7f1139b94640 1 Processor -- start 2024-05-09T03:11:23.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.934+0000 7f1139b94640 1 -- start start 2024-05-09T03:11:23.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1139b94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11340b5210 0x7f11341520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1139b94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 0x7f11340cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1139b94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11341613c0 0x7f11340ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1139b94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1134160bc0 con 0x7f11340b7d00 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1139b94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11340d5560 con 0x7f11340b5210 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1139b94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1134160ea0 con 0x7f11341613c0 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 0x7f11340cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 0x7f11340cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35752/0 (socket says 172.21.15.46:35752) 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 -- 172.21.15.46:0/3035352495 learned_addr learned my addr 172.21.15.46:0/3035352495 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f11337fe640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11340b5210 0x7f11341520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1133fff640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11341613c0 0x7f11340ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 -- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11341613c0 msgr2=0x7f11340ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11341613c0 0x7f11340ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 -- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11340b5210 msgr2=0x7f11341520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11340b5210 0x7f11341520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:23.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 -- 172.21.15.46:0/3035352495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11340cfd70 con 0x7f11340b7d00 2024-05-09T03:11:23.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f1132ffd640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 0x7f11340cd080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1128000c00 tx=0x7f11280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:23.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128041020 con 0x7f11340b7d00 2024-05-09T03:11:23.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11280306c0 con 0x7f11340b7d00 2024-05-09T03:11:23.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.936+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128037040 con 0x7f11340b7d00 2024-05-09T03:11:23.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.937+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11340d0050 con 0x7f11340b7d00 2024-05-09T03:11:23.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.937+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11340ca0e0 con 0x7f11340b7d00 2024-05-09T03:11:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.938+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f113406a270 con 0x7f11340b7d00 2024-05-09T03:11:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.938+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1128044020 con 0x7f11340b7d00 2024-05-09T03:11:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.939+0000 7f111bfff640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11200762d0 0x7f11200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.939+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4857+0+0 (secure 0 0 0) 0x7f11280b7ab0 con 0x7f11340b7d00 2024-05-09T03:11:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.939+0000 7f11337fe640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11200762d0 0x7f11200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:23.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.939+0000 7f11337fe640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11200762d0 0x7f11200787e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1124001720 tx=0x7f1124009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:23.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:23.943+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f112804b030 con 0x7f11340b7d00 2024-05-09T03:11:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.136+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f11340c9610 con 0x7f11340b7d00 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.136+0000 7f111bfff640 1 -- 172.21.15.46:0/3035352495 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v39) ==== 84+0+65 (secure 0 0 0) 0x7f1128081630 con 0x7f11340b7d00 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:24.137 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:11:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11200762d0 msgr2=0x7f11200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11200762d0 0x7f11200787e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1124001720 tx=0x7f1124009040 comp rx=0 tx=0).stop 2024-05-09T03:11:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 msgr2=0x7f11340cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 0x7f11340cd080 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1128000c00 tx=0x7f11280046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 shutdown_connections 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11341613c0 0x7f11340ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11200762d0 0x7f11200787e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11340b7d00 0x7f11340cd080 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 --2- 172.21.15.46:0/3035352495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11340b5210 0x7f11341520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 >> 172.21.15.46:0/3035352495 conn(0x7f11340d29a0 msgr2=0x7f113415c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 shutdown_connections 2024-05-09T03:11:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.138+0000 7f1139b94640 1 -- 172.21.15.46:0/3035352495 wait complete. 2024-05-09T03:11:24.150 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:11:24.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:11:24.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.268+0000 7f2d7d496640 1 Processor -- start 2024-05-09T03:11:24.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.268+0000 7f2d7d496640 1 -- start start 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d7d496640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d78163eb0 0x7f2d781642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d7d496640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2d780d2a70 0x7f2d780d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d7d496640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d780d33c0 con 0x7f2d780bb770 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d7d496640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2d780d4bb0 con 0x7f2d78163eb0 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d7d496640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2d78164900 con 0x7f2d780d2a70 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d76ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d780bb770 0x7f2d780d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40750/0 (socket says 172.21.15.46:40750) 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d6e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d78163eb0 0x7f2d781642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d76ffd640 1 -- 172.21.15.46:0/2324072427 learned_addr learned my addr 172.21.15.46:0/2324072427 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d777fe640 1 --2- 172.21.15.46:0/2324072427 >> v2:172.21.15.88:3300/0 conn(0x7f2d780d2a70 0x7f2d780d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:24.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d6e7fc640 1 -- 172.21.15.46:0/2324072427 >> v2:172.21.15.88:3300/0 conn(0x7f2d780d2a70 msgr2=0x7f2d780d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:24.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d6e7fc640 1 --2- 172.21.15.46:0/2324072427 >> v2:172.21.15.88:3300/0 conn(0x7f2d780d2a70 0x7f2d780d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d6e7fc640 1 -- 172.21.15.46:0/2324072427 >> v1:172.21.15.46:6789/0 conn(0x7f2d780bb770 legacy=0x7f2d780d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:24.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.270+0000 7f2d6e7fc640 1 -- 172.21.15.46:0/2324072427 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d78164be0 con 0x7f2d78163eb0 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.271+0000 7f2d6e7fc640 1 --2- 172.21.15.46:0/2324072427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78163eb0 0x7f2d781642c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2d7808e340 tx=0x7f2d6002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1b44eca027a05a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.271+0000 7f2d767fc640 1 -- 172.21.15.46:0/2324072427 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60037070 con 0x7f2d78163eb0 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.271+0000 7f2d767fc640 1 -- 172.21.15.46:0/2324072427 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d60032440 con 0x7f2d78163eb0 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.271+0000 7f2d767fc640 1 -- 172.21.15.46:0/2324072427 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60032810 con 0x7f2d78163eb0 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 -- 172.21.15.46:0/2324072427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78163eb0 msgr2=0x7f2d781642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 --2- 172.21.15.46:0/2324072427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78163eb0 0x7f2d781642c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2d7808e340 tx=0x7f2d6002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 -- 172.21.15.46:0/2324072427 shutdown_connections 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 --2- 172.21.15.46:0/2324072427 >> v2:172.21.15.88:3300/0 conn(0x7f2d780d2a70 0x7f2d780d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 --2- 172.21.15.46:0/2324072427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d78163eb0 0x7f2d781642c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 -- 172.21.15.46:0/2324072427 >> 172.21.15.46:0/2324072427 conn(0x7f2d7806e370 msgr2=0x7f2d7806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:24.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 -- 172.21.15.46:0/2324072427 shutdown_connections 2024-05-09T03:11:24.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.272+0000 7f2d7d496640 1 -- 172.21.15.46:0/2324072427 wait complete. 2024-05-09T03:11:24.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.274+0000 7f2d7d496640 1 Processor -- start 2024-05-09T03:11:24.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.274+0000 7f2d7d496640 1 -- start start 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d7d496640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 0x7f2d7814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d7d496640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d780d2a70 0x7f2d7814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d7d496640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d78163eb0 0x7f2d781ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d7d496640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d780d0bc0 con 0x7f2d780bb770 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d7d496640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2d780bbb80 con 0x7f2d78163eb0 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d7d496640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2d780d0d40 con 0x7f2d780d2a70 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 0x7f2d7814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 0x7f2d7814f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35778/0 (socket says 172.21.15.46:35778) 2024-05-09T03:11:24.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.275+0000 7f2d76ffd640 1 -- 172.21.15.46:0/922244011 learned_addr learned my addr 172.21.15.46:0/922244011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d777fe640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d78163eb0 0x7f2d781ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d6e7fc640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d780d2a70 0x7f2d7814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d76ffd640 1 -- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d780d2a70 msgr2=0x7f2d7814be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d76ffd640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d780d2a70 0x7f2d7814be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d76ffd640 1 -- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d78163eb0 msgr2=0x7f2d781ebe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:24.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d76ffd640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d78163eb0 0x7f2d781ebe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d76ffd640 1 -- 172.21.15.46:0/922244011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7814f730 con 0x7f2d780bb770 2024-05-09T03:11:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d76ffd640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 0x7f2d7814f1f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c3d0 tx=0x7f2d6800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:24.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68017070 con 0x7f2d780bb770 2024-05-09T03:11:24.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d68005ce0 con 0x7f2d780bb770 2024-05-09T03:11:24.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7814c9f0 con 0x7f2d780bb770 2024-05-09T03:11:24.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68007310 con 0x7f2d780bb770 2024-05-09T03:11:24.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.276+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d7814cfb0 con 0x7f2d780bb770 2024-05-09T03:11:24.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.277+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d7806a270 con 0x7f2d780bb770 2024-05-09T03:11:24.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.278+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d6801a070 con 0x7f2d780bb770 2024-05-09T03:11:24.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.279+0000 7f2d6ffff640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d58076300 0x7f2d58078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:24.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.279+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4857+0+0 (secure 0 0 0) 0x7f2d680965b0 con 0x7f2d780bb770 2024-05-09T03:11:24.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.279+0000 7f2d6e7fc640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d58076300 0x7f2d58078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:24.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.279+0000 7f2d6e7fc640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d58076300 0x7f2d58078810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d60034000 tx=0x7f2d6002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:24.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.283+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2d7814f730 con 0x7f2d780bb770 2024-05-09T03:11:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:24.476+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2d781639f0 con 0x7f2d780bb770 2024-05-09T03:11:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.379+0000 7f2d6ffff640 1 -- 172.21.15.46:0/922244011 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v40) ==== 138+0+0 (secure 0 0 0) 0x7f2d780d33c0 con 0x7f2d780bb770 2024-05-09T03:11:25.380 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d58076300 msgr2=0x7f2d58078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d58076300 0x7f2d58078810 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d60034000 tx=0x7f2d6002e000 comp rx=0 tx=0).stop 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 msgr2=0x7f2d7814f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 0x7f2d7814f1f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2d6800c3d0 tx=0x7f2d6800c890 comp rx=0 tx=0).stop 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 shutdown_connections 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d78163eb0 0x7f2d781ebe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2d780d2a70 0x7f2d7814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d58076300 0x7f2d58078810 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 --2- 172.21.15.46:0/922244011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d780bb770 0x7f2d7814f1f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 >> 172.21.15.46:0/922244011 conn(0x7f2d7806e370 msgr2=0x7f2d780b60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 shutdown_connections 2024-05-09T03:11:25.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.382+0000 7f2d7d496640 1 -- 172.21.15.46:0/922244011 wait complete. 2024-05-09T03:11:25.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:11:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.529+0000 7faa05eee640 1 Processor -- start 2024-05-09T03:11:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.529+0000 7faa05eee640 1 -- start start 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7faa05eee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa000b4660 0x7faa0008d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7faa05eee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000b4de0 con 0x7faa0008d640 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7faa05eee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa0006e010 con 0x7faa000b40f0 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7faa05eee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faa0008e1d0 con 0x7faa000b4660 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa000b4660 0x7faa0008d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9feffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faa0008d640 0x7faa000b39e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40766/0 (socket says 172.21.15.46:40766) 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9feffd640 1 -- 172.21.15.46:0/3624721433 learned_addr learned my addr 172.21.15.46:0/3624721433 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/3624721433 >> v1:172.21.15.88:6789/0 conn(0x7faa000b40f0 legacy=0x7faa000ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/3624721433 >> v1:172.21.15.46:6789/0 conn(0x7faa0008d640 legacy=0x7faa000b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:25.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/3624721433 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0006e2b0 con 0x7faa000b4660 2024-05-09T03:11:25.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.531+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/3624721433 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133401817 0 0) 0x7faa000b4de0 con 0x7faa0008d640 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.532+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/3624721433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000b4660 0x7faa0008d100 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faa00070060 tx=0x7fa9f002a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77f7766e4f595e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.532+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/3624721433 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0037070 con 0x7faa000b4660 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.532+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/3624721433 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f002ad80 con 0x7faa000b4660 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.532+0000 7fa9fe7fc640 1 -- 172.21.15.46:0/3624721433 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0032660 con 0x7faa000b4660 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 -- 172.21.15.46:0/3624721433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000b4660 msgr2=0x7faa0008d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 --2- 172.21.15.46:0/3624721433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000b4660 0x7faa0008d100 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faa00070060 tx=0x7fa9f002a030 comp rx=0 tx=0).stop 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 -- 172.21.15.46:0/3624721433 shutdown_connections 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 --2- 172.21.15.46:0/3624721433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000b4660 0x7faa0008d100 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 -- 172.21.15.46:0/3624721433 >> 172.21.15.46:0/3624721433 conn(0x7faa000b55c0 msgr2=0x7faa000b59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 -- 172.21.15.46:0/3624721433 shutdown_connections 2024-05-09T03:11:25.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.533+0000 7faa05eee640 1 -- 172.21.15.46:0/3624721433 wait complete. 2024-05-09T03:11:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.535+0000 7faa05eee640 1 Processor -- start 2024-05-09T03:11:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.535+0000 7faa05eee640 1 -- start start 2024-05-09T03:11:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7faa05eee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 0x7faa001d5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7faa05eee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b40f0 0x7faa001e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7faa05eee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa000b4660 0x7faa001ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:25.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7faa05eee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa0006f5f0 con 0x7faa0008d640 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7faa05eee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa0006f030 con 0x7faa000b4660 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7faa05eee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faa0008dec0 con 0x7faa000b40f0 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7fa9ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 0x7faa001d5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7fa9ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 0x7faa001d5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35794/0 (socket says 172.21.15.46:35794) 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.536+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/4112705458 learned_addr learned my addr 172.21.15.46:0/4112705458 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9feffd640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b40f0 0x7faa001e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9fffff640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa000b4660 0x7faa001ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b40f0 msgr2=0x7faa001e9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b40f0 0x7faa001e9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa000b4660 msgr2=0x7faa001ea4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:25.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa000b4660 0x7faa001ea4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9ff7fe640 1 -- 172.21.15.46:0/4112705458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa001d6650 con 0x7faa0008d640 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9feffd640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b40f0 0x7faa001e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9fffff640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa000b4660 0x7faa001ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9ff7fe640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 0x7faa001d5900 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009890 tx=0x7fa9f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0041070 con 0x7faa0008d640 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f003c670 con 0x7faa0008d640 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa001d6930 con 0x7faa0008d640 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0037040 con 0x7faa0008d640 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.537+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa001d6ef0 con 0x7faa0008d640 2024-05-09T03:11:25.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.538+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0006a270 con 0x7faa0008d640 2024-05-09T03:11:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.539+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa9f0044070 con 0x7faa0008d640 2024-05-09T03:11:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.540+0000 7fa9e7fff640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9ec0762d0 0x7fa9ec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.540+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 5227+0+0 (secure 0 0 0) 0x7fa9f00b7d30 con 0x7faa0008d640 2024-05-09T03:11:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.540+0000 7fa9feffd640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9ec0762d0 0x7fa9ec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:25.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.540+0000 7fa9feffd640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9ec0762d0 0x7fa9ec0787e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4000f60 tx=0x7fa9f4007cc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:25.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.545+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa9f0081740 con 0x7faa0008d640 2024-05-09T03:11:25.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:25.737+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7faa000b8cb0 con 0x7faa0008d640 2024-05-09T03:11:26.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.381+0000 7fa9e7fff640 1 -- 172.21.15.46:0/4112705458 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v41) ==== 170+0+0 (secure 0 0 0) 0x7fa9f0081040 con 0x7faa0008d640 2024-05-09T03:11:26.382 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:11:26.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.386+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9ec0762d0 msgr2=0x7fa9ec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:26.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.386+0000 7faa05eee640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9ec0762d0 0x7fa9ec0787e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4000f60 tx=0x7fa9f4007cc0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.386+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 msgr2=0x7faa001d5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.386+0000 7faa05eee640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 0x7faa001d5900 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009890 tx=0x7fa9f00046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.388+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 shutdown_connections 2024-05-09T03:11:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.388+0000 7faa05eee640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa000b4660 0x7faa001ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.388+0000 7faa05eee640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b40f0 0x7faa001e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.388+0000 7faa05eee640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9ec0762d0 0x7fa9ec0787e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.388+0000 7faa05eee640 1 --2- 172.21.15.46:0/4112705458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa0008d640 0x7faa001d5900 secure :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009890 tx=0x7fa9f00046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.388+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 >> 172.21.15.46:0/4112705458 conn(0x7faa000b55c0 msgr2=0x7faa000d2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.391+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 shutdown_connections 2024-05-09T03:11:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.391+0000 7faa05eee640 1 -- 172.21.15.46:0/4112705458 wait complete. 2024-05-09T03:11:26.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:11:26.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.542+0000 7f23a2cf9640 1 Processor -- start 2024-05-09T03:11:26.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.542+0000 7f23a2cf9640 1 -- start start 2024-05-09T03:11:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.543+0000 7f23a2cf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b7e70 0x7f239c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.543+0000 7f23a2cf9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f239c0d3df0 0x7f239c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.543+0000 7f23a2cf9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f239c0b5550 con 0x7f239c0b5140 2024-05-09T03:11:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.543+0000 7f23a2cf9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f239c0d3480 con 0x7f239c0d3df0 2024-05-09T03:11:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.543+0000 7f23a2cf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f239c0d44e0 con 0x7f239c0b7e70 2024-05-09T03:11:26.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f239c0d3df0 0x7f239c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f239c0d3df0 0x7f239c0b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60064/0 (socket says 172.21.15.46:60064) 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 -- 172.21.15.46:0/3292087221 learned_addr learned my addr 172.21.15.46:0/3292087221 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f2393fff640 1 --2- 172.21.15.46:0/3292087221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b7e70 0x7f239c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 -- 172.21.15.46:0/3292087221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b7e70 msgr2=0x7f239c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 --2- 172.21.15.46:0/3292087221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b7e70 0x7f239c0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 -- 172.21.15.46:0/3292087221 >> v1:172.21.15.88:6789/0 conn(0x7f239c0b5140 legacy=0x7f239c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.544+0000 7f23a1296640 1 -- 172.21.15.46:0/3292087221 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c0d47c0 con 0x7f239c0d3df0 2024-05-09T03:11:26.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.545+0000 7f23a1296640 1 --2- 172.21.15.46:0/3292087221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0d3df0 0x7f239c0b9380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2394009e50 tx=0x7f239402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b592831e76abd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.545+0000 7f23937fe640 1 -- 172.21.15.46:0/3292087221 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2394036070 con 0x7f239c0d3df0 2024-05-09T03:11:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.545+0000 7f23937fe640 1 -- 172.21.15.46:0/3292087221 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2394031600 con 0x7f239c0d3df0 2024-05-09T03:11:26.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.545+0000 7f23937fe640 1 -- 172.21.15.46:0/3292087221 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23940319d0 con 0x7f239c0d3df0 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 -- 172.21.15.46:0/3292087221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0d3df0 msgr2=0x7f239c0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/3292087221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0d3df0 0x7f239c0b9380 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2394009e50 tx=0x7f239402a7a0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 -- 172.21.15.46:0/3292087221 shutdown_connections 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/3292087221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0d3df0 0x7f239c0b9380 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/3292087221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b7e70 0x7f239c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 -- 172.21.15.46:0/3292087221 >> 172.21.15.46:0/3292087221 conn(0x7f239c08d3f0 msgr2=0x7f239c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 -- 172.21.15.46:0/3292087221 shutdown_connections 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.546+0000 7f23a2cf9640 1 -- 172.21.15.46:0/3292087221 wait complete. 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.547+0000 7f23a2cf9640 1 Processor -- start 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.547+0000 7f23a2cf9640 1 -- start start 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a2cf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 0x7f239c1558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:26.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a2cf9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f239c0d3df0 0x7f239c1eb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a2cf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b6b90 0x7f239c0b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a2cf9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f239c0d4200 con 0x7f239c0b5140 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a2cf9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f239c0d4d80 con 0x7f239c0d3df0 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a2cf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f239c0d5580 con 0x7f239c0b6b90 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 0x7f239c1558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 0x7f239c1558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60076/0 (socket says 172.21.15.46:60076) 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 -- 172.21.15.46:0/2112644430 learned_addr learned my addr 172.21.15.46:0/2112644430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a1296640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b6b90 0x7f239c0b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f2393fff640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f239c0d3df0 0x7f239c1eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 -- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b6b90 msgr2=0x7f239c0b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b6b90 0x7f239c0b6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 -- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f239c0d3df0 msgr2=0x7f239c1eb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.549+0000 7f23a0a95640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f239c0d3df0 0x7f239c1eb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23a0a95640 1 -- 172.21.15.46:0/2112644430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c0b74e0 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23a0a95640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 0x7f239c1558b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f239800c3a0 tx=0x7f239800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2398017070 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23980052d0 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239c1ea4f0 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2398003830 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.550+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239c1eaab0 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.551+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239c06a270 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.552+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f239801a070 con 0x7f239c0b5140 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.552+0000 7f23917fa640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2388076300 0x7f2388078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:26.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.552+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5609+0+0 (secure 0 0 0) 0x7f2398096860 con 0x7f239c0b5140 2024-05-09T03:11:26.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.553+0000 7f2393fff640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2388076300 0x7f2388078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:26.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.553+0000 7f2393fff640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2388076300 0x7f2388078810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f239c1627c0 tx=0x7f238c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:26.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.557+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f239c0b74e0 con 0x7f239c0b5140 2024-05-09T03:11:26.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:26.785+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f239c162db0 con 0x7f239c0b5140 2024-05-09T03:11:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.408+0000 7f23917fa640 1 -- 172.21.15.46:0/2112644430 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v26) ==== 172+0+0 (secure 0 0 0) 0x7f2398068210 con 0x7f239c0b5140 2024-05-09T03:11:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 6 and data pool 7 2024-05-09T03:11:27.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.416+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2388076300 msgr2=0x7f2388078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.416+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2388076300 0x7f2388078810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f239c1627c0 tx=0x7f238c009190 comp rx=0 tx=0).stop 2024-05-09T03:11:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.416+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 msgr2=0x7f239c1558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.416+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 0x7f239c1558b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f239800c3a0 tx=0x7f239800c860 comp rx=0 tx=0).stop 2024-05-09T03:11:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 shutdown_connections 2024-05-09T03:11:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f239c0b6b90 0x7f239c0b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f239c0d3df0 0x7f239c1eb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2388076300 0x7f2388078810 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 --2- 172.21.15.46:0/2112644430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f239c0b5140 0x7f239c1558b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 >> 172.21.15.46:0/2112644430 conn(0x7f239c08d3f0 msgr2=0x7f239c15fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 shutdown_connections 2024-05-09T03:11:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.423+0000 7f23a2cf9640 1 -- 172.21.15.46:0/2112644430 wait complete. 2024-05-09T03:11:27.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.557+0000 7f3a3fef9640 1 Processor -- start 2024-05-09T03:11:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.558+0000 7f3a3fef9640 1 -- start start 2024-05-09T03:11:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3fef9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380d57a0 0x7f3a38160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3fef9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a380b4af0 0x7f3a380b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3fef9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a380d49e0 con 0x7f3a380b77e0 2024-05-09T03:11:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3fef9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3a380d5bb0 con 0x7f3a380b4af0 2024-05-09T03:11:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3fef9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3a380d4cc0 con 0x7f3a380d57a0 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3d494640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a380b4af0 0x7f3a380b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3d494640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a380b4af0 0x7f3a380b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60086/0 (socket says 172.21.15.46:60086) 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3d494640 1 -- 172.21.15.46:0/2755014552 learned_addr learned my addr 172.21.15.46:0/2755014552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.559+0000 7f3a3dc95640 1 --2- 172.21.15.46:0/2755014552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380d57a0 0x7f3a38160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.560+0000 7f3a3d494640 1 -- 172.21.15.46:0/2755014552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380d57a0 msgr2=0x7f3a38160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.560+0000 7f3a3d494640 1 --2- 172.21.15.46:0/2755014552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380d57a0 0x7f3a38160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.560+0000 7f3a3d494640 1 -- 172.21.15.46:0/2755014552 >> v1:172.21.15.88:6789/0 conn(0x7f3a380b77e0 legacy=0x7f3a380b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.560+0000 7f3a3d494640 1 -- 172.21.15.46:0/2755014552 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a380d4fa0 con 0x7f3a380b4af0 2024-05-09T03:11:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.560+0000 7f3a3d494640 1 --2- 172.21.15.46:0/2755014552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380b4af0 0x7f3a380b4f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009870 tx=0x7f3a2802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43110d7d1cd6abe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.560+0000 7f3a3cc93640 1 -- 172.21.15.46:0/2755014552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28037070 con 0x7f3a380b4af0 2024-05-09T03:11:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3cc93640 1 -- 172.21.15.46:0/2755014552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a28032440 con 0x7f3a380b4af0 2024-05-09T03:11:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3cc93640 1 -- 172.21.15.46:0/2755014552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28032810 con 0x7f3a380b4af0 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 -- 172.21.15.46:0/2755014552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380b4af0 msgr2=0x7f3a380b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/2755014552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380b4af0 0x7f3a380b4f00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009870 tx=0x7f3a2802a270 comp rx=0 tx=0).stop 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 -- 172.21.15.46:0/2755014552 shutdown_connections 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/2755014552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380b4af0 0x7f3a380b4f00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/2755014552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380d57a0 0x7f3a38160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 -- 172.21.15.46:0/2755014552 >> 172.21.15.46:0/2755014552 conn(0x7f3a380743e0 msgr2=0x7f3a380d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 -- 172.21.15.46:0/2755014552 shutdown_connections 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.561+0000 7f3a3fef9640 1 -- 172.21.15.46:0/2755014552 wait complete. 2024-05-09T03:11:27.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.563+0000 7f3a3fef9640 1 Processor -- start 2024-05-09T03:11:27.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.563+0000 7f3a3fef9640 1 -- start start 2024-05-09T03:11:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.564+0000 7f3a3fef9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a380b4af0 0x7f3a380c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.564+0000 7f3a3fef9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380b77e0 0x7f3a381f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.564+0000 7f3a3fef9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 0x7f3a381587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.564+0000 7f3a3fef9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3a380d1920 con 0x7f3a380d57a0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.564+0000 7f3a3fef9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3a380d15b0 con 0x7f3a380b4af0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.564+0000 7f3a3fef9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3a381f70b0 con 0x7f3a380b77e0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 0x7f3a381587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 0x7f3a381587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60092/0 (socket says 172.21.15.46:60092) 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 -- 172.21.15.46:0/1893834242 learned_addr learned my addr 172.21.15.46:0/1893834242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3dc95640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a380b4af0 0x7f3a380c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3d494640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380b77e0 0x7f3a381f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 -- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380b77e0 msgr2=0x7f3a381f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380b77e0 0x7f3a381f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 -- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a380b4af0 msgr2=0x7f3a380c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a380b4af0 0x7f3a380c8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 -- 172.21.15.46:0/1893834242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a380c8dd0 con 0x7f3a380d57a0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3e496640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 0x7f3a381587c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3a30002230 tx=0x7f3a3000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:27.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30003b70 con 0x7f3a380d57a0 2024-05-09T03:11:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30003d10 con 0x7f3a380d57a0 2024-05-09T03:11:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a38157720 con 0x7f3a380d57a0 2024-05-09T03:11:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300052d0 con 0x7f3a380d57a0 2024-05-09T03:11:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.565+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a38157ce0 con 0x7f3a380d57a0 2024-05-09T03:11:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.566+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3806a270 con 0x7f3a380d57a0 2024-05-09T03:11:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.567+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3a30005680 con 0x7f3a380d57a0 2024-05-09T03:11:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.568+0000 7f3a2e7fc640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3a1c0762d0 0x7f3a1c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.568+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5709+0+0 (secure 0 0 0) 0x7f3a30097660 con 0x7f3a380d57a0 2024-05-09T03:11:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.568+0000 7f3a3dc95640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3a1c0762d0 0x7f3a1c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.568+0000 7f3a3dc95640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3a1c0762d0 0x7f3a1c0787e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3a3806d3b0 tx=0x7f3a34009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:27.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.572+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3a30060840 con 0x7f3a380d57a0 2024-05-09T03:11:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.768+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a380ba760 con 0x7f3a380d57a0 2024-05-09T03:11:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.769+0000 7f3a2e7fc640 1 -- 172.21.15.46:0/1893834242 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+16554 (secure 0 0 0) 0x7f3a30064b40 con 0x7f3a380d57a0 2024-05-09T03:11:27.770 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:27.770 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:26.786049+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:11:24.480773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T03:11:25.741329+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3a1c0762d0 msgr2=0x7f3a1c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3a1c0762d0 0x7f3a1c0787e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3a3806d3b0 tx=0x7f3a34009190 comp rx=0 tx=0).stop 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 msgr2=0x7f3a381587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 0x7f3a381587c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3a30002230 tx=0x7f3a3000cb90 comp rx=0 tx=0).stop 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 shutdown_connections 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3a1c0762d0 0x7f3a1c0787e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a380d57a0 0x7f3a381587c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3a380b77e0 0x7f3a381f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 --2- 172.21.15.46:0/1893834242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a380b4af0 0x7f3a380c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 >> 172.21.15.46:0/1893834242 conn(0x7f3a380743e0 msgr2=0x7f3a380b6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 shutdown_connections 2024-05-09T03:11:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.772+0000 7f3a3fef9640 1 -- 172.21.15.46:0/1893834242 wait complete. 2024-05-09T03:11:27.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:27.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.898+0000 7f8c75b38640 1 Processor -- start 2024-05-09T03:11:27.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.898+0000 7f8c75b38640 1 -- start start 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c75b38640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c700b7a50 0x7f8c701698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c75b38640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c700b4ee0 con 0x7f8c70162610 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c75b38640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c700b62e0 con 0x7f8c700b9200 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c75b38640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8c700b5460 con 0x7f8c700b7a50 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c6ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c700b7a50 0x7f8c701698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c6effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c70162610 0x7f8c70162a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57892/0 (socket says 172.21.15.46:57892) 2024-05-09T03:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c6ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c700b7a50 0x7f8c701698d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60108/0 (socket says 172.21.15.46:60108) 2024-05-09T03:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.899+0000 7f8c6effd640 1 -- 172.21.15.46:0/2192082886 learned_addr learned my addr 172.21.15.46:0/2192082886 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.900+0000 7f8c6ffff640 1 -- 172.21.15.46:0/2192082886 >> v1:172.21.15.88:6789/0 conn(0x7f8c700b9200 legacy=0x7f8c7008d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.900+0000 7f8c6ffff640 1 -- 172.21.15.46:0/2192082886 >> v1:172.21.15.46:6789/0 conn(0x7f8c70162610 legacy=0x7f8c70162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.900+0000 7f8c6ffff640 1 -- 172.21.15.46:0/2192082886 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c700b5180 con 0x7f8c700b7a50 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.900+0000 7f8c6ffff640 1 --2- 172.21.15.46:0/2192082886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b7a50 0x7f8c701698d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009d90 tx=0x7f8c6802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d14a433d837d36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c6e7fc640 1 -- 172.21.15.46:0/2192082886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68007740 con 0x7f8c700b7a50 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c6e7fc640 1 -- 172.21.15.46:0/2192082886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c680078e0 con 0x7f8c700b7a50 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c6e7fc640 1 -- 172.21.15.46:0/2192082886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68007cb0 con 0x7f8c700b7a50 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 -- 172.21.15.46:0/2192082886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b7a50 msgr2=0x7f8c701698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 --2- 172.21.15.46:0/2192082886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b7a50 0x7f8c701698d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009d90 tx=0x7f8c6802a660 comp rx=0 tx=0).stop 2024-05-09T03:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 -- 172.21.15.46:0/2192082886 shutdown_connections 2024-05-09T03:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 --2- 172.21.15.46:0/2192082886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b7a50 0x7f8c701698d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 -- 172.21.15.46:0/2192082886 >> 172.21.15.46:0/2192082886 conn(0x7f8c700b56e0 msgr2=0x7f8c700b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 -- 172.21.15.46:0/2192082886 shutdown_connections 2024-05-09T03:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.901+0000 7f8c75b38640 1 -- 172.21.15.46:0/2192082886 wait complete. 2024-05-09T03:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.903+0000 7f8c75b38640 1 Processor -- start 2024-05-09T03:11:27.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.903+0000 7f8c75b38640 1 -- start start 2024-05-09T03:11:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.904+0000 7f8c75b38640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c700b7a50 0x7f8c700c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 0x7f8c7015f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70162610 0x7f8c7015bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700cf750 con 0x7f8c700b9200 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c700cfac0 con 0x7f8c70162610 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7008d8b0 con 0x7f8c700b7a50 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 0x7f8c7015f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 0x7f8c7015f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60116/0 (socket says 172.21.15.46:60116) 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 -- 172.21.15.46:0/139610139 learned_addr learned my addr 172.21.15.46:0/139610139 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6f7fe640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c700b7a50 0x7f8c700c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6ffff640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70162610 0x7f8c7015bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 -- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c700b7a50 msgr2=0x7f8c700c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c700b7a50 0x7f8c700c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 -- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70162610 msgr2=0x7f8c7015bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70162610 0x7f8c7015bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 -- 172.21.15.46:0/139610139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7015fbf0 con 0x7f8c700b9200 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6ffff640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70162610 0x7f8c7015bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c6effd640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 0x7f8c7015f6b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400c340 tx=0x7f8c6400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64017070 con 0x7f8c700b9200 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c64005ce0 con 0x7f8c700b9200 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64007310 con 0x7f8c700b9200 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c70156010 con 0x7f8c700b9200 2024-05-09T03:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.905+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c701565d0 con 0x7f8c700b9200 2024-05-09T03:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.906+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7006a270 con 0x7f8c700b9200 2024-05-09T03:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.907+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c6401a070 con 0x7f8c700b9200 2024-05-09T03:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.908+0000 7f8c57fff640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c5c0762d0 0x7f8c5c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.908+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5709+0+0 (secure 0 0 0) 0x7f8c64096820 con 0x7f8c700b9200 2024-05-09T03:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.908+0000 7f8c6f7fe640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c5c0762d0 0x7f8c5c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.908+0000 7f8c6f7fe640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c5c0762d0 0x7f8c5c0787e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8c60006fd0 tx=0x7f8c60009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:27.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:27.913+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8c6405fb30 con 0x7f8c700b9200 2024-05-09T03:11:28.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.103+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c70070440 con 0x7f8c700b9200 2024-05-09T03:11:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.104+0000 7f8c57fff640 1 -- 172.21.15.46:0/139610139 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+16554 (secure 0 0 0) 0x7f8c64063e30 con 0x7f8c700b9200 2024-05-09T03:11:28.105 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:28.105 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:26.786049+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:11:24.480773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T03:11:25.741329+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c5c0762d0 msgr2=0x7f8c5c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c5c0762d0 0x7f8c5c0787e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8c60006fd0 tx=0x7f8c60009060 comp rx=0 tx=0).stop 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 msgr2=0x7f8c7015f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 0x7f8c7015f6b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400c340 tx=0x7f8c6400c800 comp rx=0 tx=0).stop 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 shutdown_connections 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70162610 0x7f8c7015bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c5c0762d0 0x7f8c5c0787e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c700b9200 0x7f8c7015f6b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 --2- 172.21.15.46:0/139610139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c700b7a50 0x7f8c700c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 >> 172.21.15.46:0/139610139 conn(0x7f8c700b56e0 msgr2=0x7f8c70072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:28.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 shutdown_connections 2024-05-09T03:11:28.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.106+0000 7f8c75b38640 1 -- 172.21.15.46:0/139610139 wait complete. 2024-05-09T03:11:28.118 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:11:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.235+0000 7fa3a8661640 1 Processor -- start 2024-05-09T03:11:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.236+0000 7fa3a8661640 1 -- start start 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a8661640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa3a00d1110 0x7fa3a00d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a8661640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a00b92c0 con 0x7fa3a00d1920 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a8661640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a00b2f10 con 0x7fa3a00d1d30 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a8661640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a00b9560 con 0x7fa3a00d1110 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a5bfc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa3a00d1920 0x7fa3a00b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57894/0 (socket says 172.21.15.46:57894) 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/460808075 learned_addr learned my addr 172.21.15.46:0/460808075 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.237+0000 7fa3a63fd640 1 --2- 172.21.15.46:0/460808075 >> v2:172.21.15.88:3300/0 conn(0x7fa3a00d1110 0x7fa3a00d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883315577 0 0) 0x7fa3a00b92c0 con 0x7fa3a00d1920 2024-05-09T03:11:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa38c003520 con 0x7fa3a00d1920 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 436166798 0 0) 0x7fa38c003520 con 0x7fa3a00d1920 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 >> v2:172.21.15.88:3300/0 conn(0x7fa3a00d1110 msgr2=0x7fa3a00d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 --2- 172.21.15.46:0/460808075 >> v2:172.21.15.88:3300/0 conn(0x7fa3a00d1110 0x7fa3a00d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 >> v1:172.21.15.88:6789/0 conn(0x7fa3a00d1d30 legacy=0x7fa3a00b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a00cfb20 con 0x7fa3a00d1920 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fa390001710 con 0x7fa3a00d1920 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 >> v1:172.21.15.46:6789/0 conn(0x7fa3a00d1920 legacy=0x7fa3a00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa38c003a50 0x7fa38c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c004800 0x7fa38c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c0053a0 0x7fa38c00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a00b9560 con 0x7fa38c004800 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa38c00c330 con 0x7fa38c0053a0 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa38c00c530 con 0x7fa38c003a50 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c004800 0x7fa38c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a6bfe640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c0053a0 0x7fa38c00ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a63fd640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa38c003a50 0x7fa38c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa38c003a50 msgr2=0x7fa38c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa38c003a50 0x7fa38c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c0053a0 msgr2=0x7fa38c00ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c0053a0 0x7fa38c00ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/460808075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38c003520 con 0x7fa38c004800 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.238+0000 7fa3a6bfe640 1 -- 172.21.15.46:0/460808075 reap_dead start 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.239+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c004800 0x7fa38c004c10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa3a006f9c0 tx=0x7fa38c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6415c0660fbf7e64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.239+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390007db0 con 0x7fa38c004800 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.239+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa390005ad0 con 0x7fa38c004800 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.239+0000 7fa3a53fb640 1 -- 172.21.15.46:0/460808075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390007070 con 0x7fa38c004800 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 -- 172.21.15.46:0/460808075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c004800 msgr2=0x7fa38c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c004800 0x7fa38c004c10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa3a006f9c0 tx=0x7fa38c005830 comp rx=0 tx=0).stop 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 -- 172.21.15.46:0/460808075 shutdown_connections 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 --2- 172.21.15.46:0/460808075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa38c004800 0x7fa38c004c10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 -- 172.21.15.46:0/460808075 >> 172.21.15.46:0/460808075 conn(0x7fa3a00d05f0 msgr2=0x7fa3a00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 -- 172.21.15.46:0/460808075 shutdown_connections 2024-05-09T03:11:28.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.240+0000 7fa3a8661640 1 -- 172.21.15.46:0/460808075 wait complete. 2024-05-09T03:11:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.242+0000 7fa3a8661640 1 Processor -- start 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.242+0000 7fa3a8661640 1 -- start start 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a8661640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c004800 0x7fa3a0157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a8661640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 0x7fa3a01f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a8661640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3a00cb2f0 0x7fa3a00cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a8661640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a00b8f80 con 0x7fa3a01f5580 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a8661640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a00b8e00 con 0x7fa38c004800 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a8661640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a00d1110 con 0x7fa3a00cb2f0 2024-05-09T03:11:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a5bfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 0x7fa3a01f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a5bfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 0x7fa3a01f5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60138/0 (socket says 172.21.15.46:60138) 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.243+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/911380924 learned_addr learned my addr 172.21.15.46:0/911380924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a63fd640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c004800 0x7fa3a0157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3a00cb2f0 msgr2=0x7fa3a00cb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a6bfe640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3a00cb2f0 0x7fa3a00cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3a00cb2f0 0x7fa3a00cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c004800 msgr2=0x7fa3a0157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c004800 0x7fa3a0157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a5bfc640 1 -- 172.21.15.46:0/911380924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a00cbc40 con 0x7fa3a01f5580 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a63fd640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c004800 0x7fa3a0157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a5bfc640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 0x7fa3a01f5990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00b9490 tx=0x7fa390001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39003f070 con 0x7fa3a01f5580 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa390005c80 con 0x7fa3a01f5580 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a00cd800 con 0x7fa3a01f5580 2024-05-09T03:11:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3900016d0 con 0x7fa3a01f5580 2024-05-09T03:11:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.244+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a006e3f0 con 0x7fa3a01f5580 2024-05-09T03:11:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.245+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a006a270 con 0x7fa3a01f5580 2024-05-09T03:11:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.246+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa390042070 con 0x7fa3a01f5580 2024-05-09T03:11:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.247+0000 7fa396ffd640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3840762d0 0x7fa3840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.247+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5709+0+0 (secure 0 0 0) 0x7fa3900b7300 con 0x7fa3a01f5580 2024-05-09T03:11:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.247+0000 7fa3a63fd640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3840762d0 0x7fa3840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:28.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.247+0000 7fa3a63fd640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3840762d0 0x7fa3840787e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0076c00 tx=0x7fa39c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:28.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.251+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa390049030 con 0x7fa3a01f5580 2024-05-09T03:11:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:28.486+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa3a00cd9d0 con 0x7fa3a01f5580 2024-05-09T03:11:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.420+0000 7fa396ffd640 1 -- 172.21.15.46:0/911380924 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fa3a00cbc40 con 0x7fa3a01f5580 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3840762d0 msgr2=0x7fa3840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3840762d0 0x7fa3840787e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0076c00 tx=0x7fa39c008040 comp rx=0 tx=0).stop 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 msgr2=0x7fa3a01f5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 0x7fa3a01f5990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00b9490 tx=0x7fa390001a10 comp rx=0 tx=0).stop 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 shutdown_connections 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa3a00cb2f0 0x7fa3a00cb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3840762d0 0x7fa3840787e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a01f5580 0x7fa3a01f5990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 --2- 172.21.15.46:0/911380924 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa38c004800 0x7fa3a0157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 >> 172.21.15.46:0/911380924 conn(0x7fa3a00d05f0 msgr2=0x7fa3a0071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:29.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 shutdown_connections 2024-05-09T03:11:29.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.423+0000 7fa3a8661640 1 -- 172.21.15.46:0/911380924 wait complete. 2024-05-09T03:11:29.438 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:11:29.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:11:29.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.563+0000 7f0a68b0d640 1 Processor -- start 2024-05-09T03:11:29.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.563+0000 7f0a68b0d640 1 -- start start 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a68b0d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a64167060 0x7f0a64169480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a68b0d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a640b3f30 con 0x7f0a640b77a0 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a68b0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a640b41a0 con 0x7f0a640ceff0 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a68b0d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0a640cf400 con 0x7f0a64167060 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a59d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a640b77a0 0x7f0a640b7bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57904/0 (socket says 172.21.15.46:57904) 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a59d9b640 1 -- 172.21.15.46:0/1551159631 learned_addr learned my addr 172.21.15.46:0/1551159631 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a62d9d640 1 --2- 172.21.15.46:0/1551159631 >> v2:172.21.15.88:3300/0 conn(0x7f0a64167060 0x7f0a64169480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.564+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696281080 0 0) 0x7f0a640b3f30 con 0x7f0a640b77a0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a50003520 con 0x7f0a640b77a0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135398114 0 0) 0x7f0a640b41a0 con 0x7f0a640ceff0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a640b3f30 con 0x7f0a640ceff0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2765643288 0 0) 0x7f0a50003520 con 0x7f0a640b77a0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 >> v2:172.21.15.88:3300/0 conn(0x7f0a64167060 msgr2=0x7f0a64169480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 --2- 172.21.15.46:0/1551159631 >> v2:172.21.15.88:3300/0 conn(0x7f0a64167060 0x7f0a64169480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 >> v1:172.21.15.88:6789/0 conn(0x7f0a640ceff0 legacy=0x7f0a640b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a64161f50 con 0x7f0a640b77a0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f0a4c0016f0 con 0x7f0a640b77a0 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 >> v1:172.21.15.46:6789/0 conn(0x7f0a640b77a0 legacy=0x7f0a640b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50003c80 0x7f0a500040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a50004800 0x7f0a50004c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a50005380 0x7f0a5000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a640cf400 con 0x7f0a50004800 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5000c320 con 0x7f0a50005380 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5000c520 con 0x7f0a50003c80 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a50004800 0x7f0a50004c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a6259c640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50003c80 0x7f0a500040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a62d9d640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a50005380 0x7f0a5000ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a59d9b640 1 -- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50003c80 msgr2=0x7f0a500040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50003c80 0x7f0a500040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a59d9b640 1 -- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a50005380 msgr2=0x7f0a5000ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:29.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a50005380 0x7f0a5000ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a59d9b640 1 -- 172.21.15.46:0/1551159631 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a50003520 con 0x7f0a50004800 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.565+0000 7f0a62d9d640 1 -- 172.21.15.46:0/1551159631 reap_dead start 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.566+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a50004800 0x7f0a50004c70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00c110 tx=0x7f0a50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa49fe2e369cfea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.566+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c007c40 con 0x7f0a50004800 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.566+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c005ad0 con 0x7f0a50004800 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.566+0000 7f0a61d9b640 1 -- 172.21.15.46:0/1551159631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c007070 con 0x7f0a50004800 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 -- 172.21.15.46:0/1551159631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a50004800 msgr2=0x7f0a50004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a50004800 0x7f0a50004c70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00c110 tx=0x7f0a50005830 comp rx=0 tx=0).stop 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 -- 172.21.15.46:0/1551159631 shutdown_connections 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/1551159631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a50004800 0x7f0a50004c70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 -- 172.21.15.46:0/1551159631 >> 172.21.15.46:0/1551159631 conn(0x7f0a6409b7d0 msgr2=0x7f0a6409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 -- 172.21.15.46:0/1551159631 shutdown_connections 2024-05-09T03:11:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.567+0000 7f0a68b0d640 1 -- 172.21.15.46:0/1551159631 wait complete. 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.569+0000 7f0a68b0d640 1 Processor -- start 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.569+0000 7f0a68b0d640 1 -- start start 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50004800 0x7f0a640c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 0x7f0a6415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a64158250 0x7f0a64158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64162510 con 0x7f0a6415d820 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64161d00 con 0x7f0a64158250 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0a64162a30 con 0x7f0a50004800 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 0x7f0a6415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 0x7f0a6415dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60148/0 (socket says 172.21.15.46:60148) 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 -- 172.21.15.46:0/529297190 learned_addr learned my addr 172.21.15.46:0/529297190 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a62d9d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a64158250 0x7f0a64158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a6259c640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50004800 0x7f0a640c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 -- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50004800 msgr2=0x7f0a640c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50004800 0x7f0a640c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 -- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a64158250 msgr2=0x7f0a64158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a64158250 0x7f0a64158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 -- 172.21.15.46:0/529297190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a640b1b70 con 0x7f0a6415d820 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a62d9d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a64158250 0x7f0a64158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a6259c640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50004800 0x7f0a640c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a59d9b640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 0x7f0a6415dc30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c02aca0 tx=0x7f0a4c001890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c037040 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c005e00 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.571+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a64157770 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.572+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c0047e0 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.572+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a640b1e50 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.572+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a6406a270 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.573+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a4c043030 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.574+0000 7f0a5b7fe640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a440762d0 0x7f0a440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.574+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0a4c0b69b0 con 0x7f0a6415d820 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.574+0000 7f0a6259c640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a440762d0 0x7f0a440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.574+0000 7f0a6259c640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a440762d0 0x7f0a440787e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0a540055c0 tx=0x7f0a54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.578+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0a4c04a060 con 0x7f0a6415d820 2024-05-09T03:11:29.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:29.814+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0a640cf6c0 con 0x7f0a6415d820 2024-05-09T03:11:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.421+0000 7f0a5b7fe640 1 -- 172.21.15.46:0/529297190 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7f0a4c080310 con 0x7f0a6415d820 2024-05-09T03:11:30.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.423+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a440762d0 msgr2=0x7f0a440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.423+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a440762d0 0x7f0a440787e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0a540055c0 tx=0x7f0a54009040 comp rx=0 tx=0).stop 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.423+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 msgr2=0x7f0a6415dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.423+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 0x7f0a6415dc30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c02aca0 tx=0x7f0a4c001890 comp rx=0 tx=0).stop 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.423+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 shutdown_connections 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.423+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a64158250 0x7f0a64158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.424+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a440762d0 0x7f0a440787e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.424+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a6415d820 0x7f0a6415dc30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.424+0000 7f0a68b0d640 1 --2- 172.21.15.46:0/529297190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a50004800 0x7f0a640c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.424+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 >> 172.21.15.46:0/529297190 conn(0x7f0a6409b7d0 msgr2=0x7f0a64075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.424+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 shutdown_connections 2024-05-09T03:11:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.424+0000 7f0a68b0d640 1 -- 172.21.15.46:0/529297190 wait complete. 2024-05-09T03:11:30.439 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:11:30.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:30.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.560+0000 7fc0c227e640 1 Processor -- start 2024-05-09T03:11:30.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.560+0000 7fc0c227e640 1 -- start start 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.561+0000 7fc0c227e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0bc0b9200 0x7fc0bc08d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.561+0000 7fc0c227e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0bc0b4ee0 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.561+0000 7fc0c227e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0bc0b62e0 con 0x7fc0bc0d2270 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.561+0000 7fc0c227e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc0bc0b5460 con 0x7fc0bc0b9200 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0b3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc0bc08d6e0 0x7fc0bc0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57916/0 (socket says 172.21.15.46:57916) 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0b3fff640 1 -- 172.21.15.46:0/2245841253 learned_addr learned my addr 172.21.15.46:0/2245841253 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bbfff640 1 --2- 172.21.15.46:0/2245841253 >> v2:172.21.15.88:3300/0 conn(0x7fc0bc0b9200 0x7fc0bc08d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bb7fe640 1 -- 172.21.15.46:0/2245841253 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030249957 0 0) 0x7fc0bc0b4ee0 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bb7fe640 1 -- 172.21.15.46:0/2245841253 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0a8003520 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bbfff640 1 -- 172.21.15.46:0/2245841253 >> v1:172.21.15.88:6789/0 conn(0x7fc0bc0d2270 legacy=0x7fc0bc0b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bb7fe640 1 -- 172.21.15.46:0/2245841253 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842725590 0 0) 0x7fc0bc0b62e0 con 0x7fc0bc0d2270 2024-05-09T03:11:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bbfff640 1 -- 172.21.15.46:0/2245841253 >> v1:172.21.15.46:6789/0 conn(0x7fc0bc08d6e0 legacy=0x7fc0bc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.562+0000 7fc0bbfff640 1 -- 172.21.15.46:0/2245841253 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc0b5180 con 0x7fc0bc0b9200 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.563+0000 7fc0bbfff640 1 --2- 172.21.15.46:0/2245841253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0b9200 0x7fc0bc08d1a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0097a0 tx=0x7fc0ac02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c362658dd616a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.563+0000 7fc0bb7fe640 1 -- 172.21.15.46:0/2245841253 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac037070 con 0x7fc0bc0b9200 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.563+0000 7fc0bb7fe640 1 -- 172.21.15.46:0/2245841253 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0ac02ac30 con 0x7fc0bc0b9200 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.563+0000 7fc0bb7fe640 1 -- 172.21.15.46:0/2245841253 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40016f0 con 0x7fc0bc0b9200 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 -- 172.21.15.46:0/2245841253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0b9200 msgr2=0x7fc0bc08d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 --2- 172.21.15.46:0/2245841253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0b9200 0x7fc0bc08d1a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0097a0 tx=0x7fc0ac02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 -- 172.21.15.46:0/2245841253 shutdown_connections 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 --2- 172.21.15.46:0/2245841253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0b9200 0x7fc0bc08d1a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 -- 172.21.15.46:0/2245841253 >> 172.21.15.46:0/2245841253 conn(0x7fc0bc0b56e0 msgr2=0x7fc0bc0b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:30.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 -- 172.21.15.46:0/2245841253 shutdown_connections 2024-05-09T03:11:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.564+0000 7fc0c227e640 1 -- 172.21.15.46:0/2245841253 wait complete. 2024-05-09T03:11:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.566+0000 7fc0c227e640 1 Processor -- start 2024-05-09T03:11:30.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.566+0000 7fc0c227e640 1 -- start start 2024-05-09T03:11:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c227e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 0x7fc0bc160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c227e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0bc0b9200 0x7fc0bc14bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c227e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0d2270 0x7fc0bc158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c227e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc0cf750 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c227e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc0cfac0 con 0x7fc0bc0d2270 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c227e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc0bc0b4a20 con 0x7fc0bc0b9200 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 0x7fc0bc160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 0x7fc0bc160cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60156/0 (socket says 172.21.15.46:60156) 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0bbfff640 1 -- 172.21.15.46:0/34164132 learned_addr learned my addr 172.21.15.46:0/34164132 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.567+0000 7fc0c081b640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0d2270 0x7fc0bc158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0bbfff640 1 -- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0bc0b9200 msgr2=0x7fc0bc14bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0b3fff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0bc0b9200 0x7fc0bc14bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0bbfff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0bc0b9200 0x7fc0bc14bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0bbfff640 1 -- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0d2270 msgr2=0x7fc0bc158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0bbfff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0d2270 0x7fc0bc158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0bbfff640 1 -- 172.21.15.46:0/34164132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc160090 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0c081b640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0d2270 0x7fc0bc158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0b3fff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0bc0b9200 0x7fc0bc14bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0bbfff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 0x7fc0bc160cd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac035040 tx=0x7fc0ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac037040 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0ac004030 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0bc15f010 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac004400 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.568+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0bc15f5b0 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.569+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0bc06a270 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.570+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc0ac03a040 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.570+0000 7fc0b97fa640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc09c076300 0x7fc09c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.571+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fc0ac0b7b30 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.571+0000 7fc0b3fff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc09c076300 0x7fc09c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.571+0000 7fc0b3fff640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc09c076300 0x7fc09c078810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0c71f0 tx=0x7fc0a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.575+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc0ac042020 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.807+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0bc0d2030 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.808+0000 7fc0b97fa640 1 -- 172.21.15.46:0/34164132 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5172 (secure 0 0 0) 0x7fc0ac081400 con 0x7fc0bc08d6e0 2024-05-09T03:11:30.809 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:30.809 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:29.815636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2630572362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2630572362},{"type":"v1","addr":"172.21.15.46:6835","nonce":2630572362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":3}]} 2024-05-09T03:11:30.809 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2024-05-09T03:11:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc09c076300 msgr2=0x7fc09c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc09c076300 0x7fc09c078810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0c71f0 tx=0x7fc0a4009040 comp rx=0 tx=0).stop 2024-05-09T03:11:30.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 msgr2=0x7fc0bc160cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 0x7fc0bc160cd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac035040 tx=0x7fc0ac0046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 shutdown_connections 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0bc0d2270 0x7fc0bc158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0bc0b9200 0x7fc0bc14bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc09c076300 0x7fc09c078810 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 --2- 172.21.15.46:0/34164132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0bc08d6e0 0x7fc0bc160cd0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 >> 172.21.15.46:0/34164132 conn(0x7fc0bc0b56e0 msgr2=0x7fc0bc0d2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.810+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 shutdown_connections 2024-05-09T03:11:30.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.811+0000 7fc0c227e640 1 -- 172.21.15.46:0/34164132 wait complete. 2024-05-09T03:11:30.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:30.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.937+0000 7fc30061e640 1 Processor -- start 2024-05-09T03:11:30.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.937+0000 7fc30061e640 1 -- start start 2024-05-09T03:11:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.938+0000 7fc30061e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2f80cf6b0 0x7fc2f80b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.938+0000 7fc30061e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f80d2b80 con 0x7fc2f80d0b90 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.938+0000 7fc30061e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f80d2e60 con 0x7fc2f80b92e0 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.938+0000 7fc30061e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f80d3140 con 0x7fc2f80cf6b0 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2febbb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2f80cf6b0 0x7fc2f80b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2fdbb9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc2f80d0b90 0x7fc2f806d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57932/0 (socket says 172.21.15.46:57932) 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2febbb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2f80cf6b0 0x7fc2f80b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60166/0 (socket says 172.21.15.46:60166) 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2fdbb9640 1 -- 172.21.15.46:0/4286013606 learned_addr learned my addr 172.21.15.46:0/4286013606 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2febbb640 1 -- 172.21.15.46:0/4286013606 >> v1:172.21.15.88:6789/0 conn(0x7fc2f80b92e0 legacy=0x7fc2f80d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2febbb640 1 -- 172.21.15.46:0/4286013606 >> v1:172.21.15.46:6789/0 conn(0x7fc2f80d0b90 legacy=0x7fc2f806d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2fd3b8640 1 -- 172.21.15.46:0/4286013606 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606171384 0 0) 0x7fc2f80d2b80 con 0x7fc2f80d0b90 2024-05-09T03:11:30.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.939+0000 7fc2febbb640 1 -- 172.21.15.46:0/4286013606 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f80d3420 con 0x7fc2f80cf6b0 2024-05-09T03:11:30.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.940+0000 7fc2febbb640 1 --2- 172.21.15.46:0/4286013606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80cf6b0 0x7fc2f80b51a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009d90 tx=0x7fc2f002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754ba91d1f6a79f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.940+0000 7fc2fd3b8640 1 -- 172.21.15.46:0/4286013606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0007740 con 0x7fc2f80cf6b0 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.940+0000 7fc2fd3b8640 1 -- 172.21.15.46:0/4286013606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f00078e0 con 0x7fc2f80cf6b0 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.940+0000 7fc2fd3b8640 1 -- 172.21.15.46:0/4286013606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0007cb0 con 0x7fc2f80cf6b0 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.940+0000 7fc30061e640 1 -- 172.21.15.46:0/4286013606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80cf6b0 msgr2=0x7fc2f80b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.940+0000 7fc30061e640 1 --2- 172.21.15.46:0/4286013606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80cf6b0 0x7fc2f80b51a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009d90 tx=0x7fc2f002a660 comp rx=0 tx=0).stop 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.941+0000 7fc30061e640 1 -- 172.21.15.46:0/4286013606 shutdown_connections 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.941+0000 7fc30061e640 1 --2- 172.21.15.46:0/4286013606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80cf6b0 0x7fc2f80b51a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.941+0000 7fc30061e640 1 -- 172.21.15.46:0/4286013606 >> 172.21.15.46:0/4286013606 conn(0x7fc2f80d3c00 msgr2=0x7fc2f80d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.941+0000 7fc30061e640 1 -- 172.21.15.46:0/4286013606 shutdown_connections 2024-05-09T03:11:30.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.941+0000 7fc30061e640 1 -- 172.21.15.46:0/4286013606 wait complete. 2024-05-09T03:11:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.942+0000 7fc30061e640 1 Processor -- start 2024-05-09T03:11:30.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.942+0000 7fc30061e640 1 -- start start 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc30061e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f80b92e0 0x7fc2f80b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc30061e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2f80cf6b0 0x7fc2f80b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc30061e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 0x7fc2f81597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc30061e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f816b3d0 con 0x7fc2f80b7e90 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc30061e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f816b250 con 0x7fc2f80b92e0 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc30061e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f816b550 con 0x7fc2f80cf6b0 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 0x7fc2f81597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 0x7fc2f81597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60182/0 (socket says 172.21.15.46:60182) 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 -- 172.21.15.46:0/869062898 learned_addr learned my addr 172.21.15.46:0/869062898 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:30.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2fdbb9640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2f80cf6b0 0x7fc2f80b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2fe3ba640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f80b92e0 0x7fc2f80b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 -- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2f80cf6b0 msgr2=0x7fc2f80b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2f80cf6b0 0x7fc2f80b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 -- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f80b92e0 msgr2=0x7fc2f80b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f80b92e0 0x7fc2f80b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 -- 172.21.15.46:0/869062898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f8154eb0 con 0x7fc2f80b7e90 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.944+0000 7fc2febbb640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 0x7fc2f81597e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc2f002ac50 tx=0x7fc2f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.945+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0040070 con 0x7fc2f80b7e90 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.945+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f0031c90 con 0x7fc2f80b7e90 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.945+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f8155810 con 0x7fc2f80b7e90 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.945+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0045680 con 0x7fc2f80b7e90 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.945+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f80b4870 con 0x7fc2f80b7e90 2024-05-09T03:11:30.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.945+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f806a270 con 0x7fc2f80b7e90 2024-05-09T03:11:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.946+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc2f0039040 con 0x7fc2f80b7e90 2024-05-09T03:11:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.947+0000 7fc2eeffd640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc2dc0762d0 0x7fc2dc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.947+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fc2f00b9030 con 0x7fc2f80b7e90 2024-05-09T03:11:30.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.947+0000 7fc2fe3ba640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc2dc0762d0 0x7fc2dc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:30.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.948+0000 7fc2fe3ba640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc2dc0762d0 0x7fc2dc0787e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4000f60 tx=0x7fc2f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:30.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:30.952+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc2f0082a50 con 0x7fc2f80b7e90 2024-05-09T03:11:31.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.147+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2f806de80 con 0x7fc2f80b7e90 2024-05-09T03:11:31.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.148+0000 7fc2eeffd640 1 -- 172.21.15.46:0/869062898 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16541 (secure 0 0 0) 0x7fc2f0082350 con 0x7fc2f80b7e90 2024-05-09T03:11:31.149 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:31.149 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":43,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:28.391962+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:11:24.480773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T03:11:25.741329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:31.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.150+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc2dc0762d0 msgr2=0x7fc2dc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.150+0000 7fc30061e640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc2dc0762d0 0x7fc2dc0787e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4000f60 tx=0x7fc2f4009040 comp rx=0 tx=0).stop 2024-05-09T03:11:31.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.150+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 msgr2=0x7fc2f81597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.150+0000 7fc30061e640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 0x7fc2f81597e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc2f002ac50 tx=0x7fc2f0004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 shutdown_connections 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc2dc0762d0 0x7fc2dc0787e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2f80b7e90 0x7fc2f81597e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc2f80cf6b0 0x7fc2f80b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 --2- 172.21.15.46:0/869062898 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f80b92e0 0x7fc2f80b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 >> 172.21.15.46:0/869062898 conn(0x7fc2f80d3c00 msgr2=0x7fc2f80b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 shutdown_connections 2024-05-09T03:11:31.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.151+0000 7fc30061e640 1 -- 172.21.15.46:0/869062898 wait complete. 2024-05-09T03:11:31.163 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:11:31.163 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:11:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.277+0000 7fe8c25e8640 1 Processor -- start 2024-05-09T03:11:31.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.277+0000 7fe8c25e8640 1 -- start start 2024-05-09T03:11:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.278+0000 7fe8c25e8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8bc0d57a0 0x7fe8bc160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.278+0000 7fe8c25e8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8bc0d49e0 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.278+0000 7fe8c25e8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8bc0d4cc0 con 0x7fe8bc0b7ef0 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.278+0000 7fe8c25e8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe8bc0d5bb0 con 0x7fe8bc0d57a0 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8bc0d57a0 0x7fe8bc160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8bc0d57a0 0x7fe8bc160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60184/0 (socket says 172.21.15.46:60184) 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3716977620 learned_addr learned my addr 172.21.15.46:0/3716977620 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bb7fe640 1 --1- 172.21.15.46:0/3716977620 >> v1:172.21.15.46:6789/0 conn(0x7fe8bc0b4af0 0x7fe8bc0b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57934/0 (socket says 172.21.15.46:57934) 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3716977620 >> v1:172.21.15.88:6789/0 conn(0x7fe8bc0b7ef0 legacy=0x7fe8bc0b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3716977620 >> v1:172.21.15.46:6789/0 conn(0x7fe8bc0b4af0 legacy=0x7fe8bc0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.279+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3716977620 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc0d4fa0 con 0x7fe8bc0d57a0 2024-05-09T03:11:31.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.280+0000 7fe8bbfff640 1 --2- 172.21.15.46:0/3716977620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc160610 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009870 tx=0x7fe8ac02a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cd620c328cce35 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:31.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.280+0000 7fe8baffd640 1 -- 172.21.15.46:0/3716977620 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac037070 con 0x7fe8bc0d57a0 2024-05-09T03:11:31.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.280+0000 7fe8baffd640 1 -- 172.21.15.46:0/3716977620 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac032440 con 0x7fe8bc0d57a0 2024-05-09T03:11:31.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.280+0000 7fe8baffd640 1 -- 172.21.15.46:0/3716977620 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac032810 con 0x7fe8bc0d57a0 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3716977620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0d57a0 msgr2=0x7fe8bc160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3716977620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc160610 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009870 tx=0x7fe8ac02a200 comp rx=0 tx=0).stop 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3716977620 shutdown_connections 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3716977620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc160610 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3716977620 >> 172.21.15.46:0/3716977620 conn(0x7fe8bc0743e0 msgr2=0x7fe8bc0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3716977620 shutdown_connections 2024-05-09T03:11:31.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.281+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3716977620 wait complete. 2024-05-09T03:11:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.282+0000 7fe8c25e8640 1 Processor -- start 2024-05-09T03:11:31.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.283+0000 7fe8c25e8640 1 -- start start 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c25e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 0x7fe8bc1e9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c25e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc1e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c25e8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8bc1e7560 0x7fe8bc1e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c25e8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc0d1920 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c25e8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc0d15b0 con 0x7fe8bc0d57a0 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c25e8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe8bc0c9c20 con 0x7fe8bc1e7560 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 0x7fe8bc1e9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 0x7fe8bc1e9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60186/0 (socket says 172.21.15.46:60186) 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3523198479 learned_addr learned my addr 172.21.15.46:0/3523198479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bb7fe640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc1e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8c0b85640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8bc1e7560 0x7fe8bc1e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8bc1e7560 msgr2=0x7fe8bc1e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8bc1e7560 0x7fe8bc1e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8bc0d57a0 msgr2=0x7fe8bc1e9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc1e9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.284+0000 7fe8bbfff640 1 -- 172.21.15.46:0/3523198479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1eaad0 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.285+0000 7fe8bbfff640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 0x7fe8bc1e9a30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000c00 tx=0x7fe8ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.285+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac037040 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.285+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac004230 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.285+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac030620 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.285+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc1e6ac0 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.285+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1e6d60 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.286+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8bc06a270 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.287+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe8ac0307c0 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.287+0000 7fe8b8ff9640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe8a8076300 0x7fe8a8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.287+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fe8ac0b8400 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.288+0000 7fe8bb7fe640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe8a8076300 0x7fe8a8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.288+0000 7fe8bb7fe640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe8a8076300 0x7fe8a8078810 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc1e8660 tx=0x7fe8b0007e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:31.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.292+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe8ac081700 con 0x7fe8bc0b4af0 2024-05-09T03:11:31.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.484+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8bc06d0f0 con 0x7fe8a8076300 2024-05-09T03:11:31.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.491+0000 7fe8b8ff9640 1 -- 172.21.15.46:0/3523198479 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7fe8bc06d0f0 con 0x7fe8a8076300 2024-05-09T03:11:31.494 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:31.496 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:11:31.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe8a8076300 msgr2=0x7fe8a8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe8a8076300 0x7fe8a8078810 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc1e8660 tx=0x7fe8b0007e00 comp rx=0 tx=0).stop 2024-05-09T03:11:31.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 msgr2=0x7fe8bc1e9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 0x7fe8bc1e9a30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000c00 tx=0x7fe8ac0046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 shutdown_connections 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8bc1e7560 0x7fe8bc1e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8bc0d57a0 0x7fe8bc1e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe8a8076300 0x7fe8a8078810 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 --2- 172.21.15.46:0/3523198479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8bc0b4af0 0x7fe8bc1e9a30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 >> 172.21.15.46:0/3523198479 conn(0x7fe8bc0743e0 msgr2=0x7fe8bc0b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 shutdown_connections 2024-05-09T03:11:31.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.494+0000 7fe8c25e8640 1 -- 172.21.15.46:0/3523198479 wait complete. 2024-05-09T03:11:31.506 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-05-09T03:11:30.520630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237672,"kb_used_data":4832,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737048,"statfs":{"total":773094113280,"available":772850737152,"internally_reserved":0,"allocated":4947968,"data_stored":2669038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.137100"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386053+0000","last_change":"2024-05-09T03:11:26.386053+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386053+0000","last_undegraded":"2024-05-09T03:11:26.386053+0000","last_fullsized":"2024-05-09T03:11:26.386053+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393641+0000","last_change":"2024-05-09T03:11:26.393641+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393641+0000","last_undegraded":"2024-05-09T03:11:26.393641+0000","last_fullsized":"2024-05-09T03:11:26.393641+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386800+0000","last_change":"2024-05-09T03:11:26.386800+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386800+0000","last_undegraded":"2024-05-09T03:11:26.386800+0000","last_fullsized":"2024-05-09T03:11:26.386800+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399458+0000","last_change":"2024-05-09T03:11:26.399458+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399458+0000","last_undegraded":"2024-05-09T03:11:26.399458+0000","last_fullsized":"2024-05-09T03:11:26.399458+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393056+0000","last_change":"2024-05-09T03:11:26.393056+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393056+0000","last_undegraded":"2024-05-09T03:11:26.393056+0000","last_fullsized":"2024-05-09T03:11:26.393056+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393193+0000","last_change":"2024-05-09T03:11:26.393193+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393193+0000","last_undegraded":"2024-05-09T03:11:26.393193+0000","last_fullsized":"2024-05-09T03:11:26.393193+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.387803+0000","last_change":"2024-05-09T03:11:26.387803+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.387803+0000","last_undegraded":"2024-05-09T03:11:26.387803+0000","last_fullsized":"2024-05-09T03:11:26.387803+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392140+0000","last_change":"2024-05-09T03:11:26.392140+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392140+0000","last_undegraded":"2024-05-09T03:11:26.392140+0000","last_fullsized":"2024-05-09T03:11:26.392140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399153+0000","last_change":"2024-05-09T03:11:26.399153+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399153+0000","last_undegraded":"2024-05-09T03:11:26.399153+0000","last_fullsized":"2024-05-09T03:11:26.399153+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399862+0000","last_change":"2024-05-09T03:11:26.399862+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399862+0000","last_undegraded":"2024-05-09T03:11:26.399862+0000","last_fullsized":"2024-05-09T03:11:26.399862+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398110+0000","last_change":"2024-05-09T03:11:26.398110+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398110+0000","last_undegraded":"2024-05-09T03:11:26.398110+0000","last_fullsized":"2024-05-09T03:11:26.398110+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393757+0000","last_change":"2024-05-09T03:11:26.393757+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393757+0000","last_undegraded":"2024-05-09T03:11:26.393757+0000","last_fullsized":"2024-05-09T03:11:26.393757+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389234+0000","last_change":"2024-05-09T03:11:26.389234+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389234+0000","last_undegraded":"2024-05-09T03:11:26.389234+0000","last_fullsized":"2024-05-09T03:11:26.389234+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391877+0000","last_change":"2024-05-09T03:11:26.391877+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391877+0000","last_undegraded":"2024-05-09T03:11:26.391877+0000","last_fullsized":"2024-05-09T03:11:26.391877+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393846+0000","last_change":"2024-05-09T03:11:26.393846+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393846+0000","last_undegraded":"2024-05-09T03:11:26.393846+0000","last_fullsized":"2024-05-09T03:11:26.393846+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392475+0000","last_change":"2024-05-09T03:11:26.392475+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392475+0000","last_undegraded":"2024-05-09T03:11:26.392475+0000","last_fullsized":"2024-05-09T03:11:26.392475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388512+0000","last_change":"2024-05-09T03:11:26.388512+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388512+0000","last_undegraded":"2024-05-09T03:11:26.388512+0000","last_fullsized":"2024-05-09T03:11:26.388512+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388619+0000","last_change":"2024-05-09T03:11:26.388619+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388619+0000","last_undegraded":"2024-05-09T03:11:26.388619+0000","last_fullsized":"2024-05-09T03:11:26.388619+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388811+0000","last_change":"2024-05-09T03:11:26.388811+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388811+0000","last_undegraded":"2024-05-09T03:11:26.388811+0000","last_fullsized":"2024-05-09T03:11:26.388811+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391447+0000","last_change":"2024-05-09T03:11:26.391447+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391447+0000","last_undegraded":"2024-05-09T03:11:26.391447+0000","last_fullsized":"2024-05-09T03:11:26.391447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393040+0000","last_change":"2024-05-09T03:11:26.393040+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393040+0000","last_undegraded":"2024-05-09T03:11:26.393040+0000","last_fullsized":"2024-05-09T03:11:26.393040+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391316+0000","last_change":"2024-05-09T03:11:26.391316+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391316+0000","last_undegraded":"2024-05-09T03:11:26.391316+0000","last_fullsized":"2024-05-09T03:11:26.391316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389605+0000","last_change":"2024-05-09T03:11:26.389605+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389605+0000","last_undegraded":"2024-05-09T03:11:26.389605+0000","last_fullsized":"2024-05-09T03:11:26.389605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391465+0000","last_change":"2024-05-09T03:11:26.391465+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391465+0000","last_undegraded":"2024-05-09T03:11:26.391465+0000","last_fullsized":"2024-05-09T03:11:26.391465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390564+0000","last_change":"2024-05-09T03:11:26.390564+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390564+0000","last_undegraded":"2024-05-09T03:11:26.390564+0000","last_fullsized":"2024-05-09T03:11:26.390564+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390514+0000","last_change":"2024-05-09T03:11:26.390514+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390514+0000","last_undegraded":"2024-05-09T03:11:26.390514+0000","last_fullsized":"2024-05-09T03:11:26.390514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385973+0000","last_change":"2024-05-09T03:11:26.385973+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385973+0000","last_undegraded":"2024-05-09T03:11:26.385973+0000","last_fullsized":"2024-05-09T03:11:26.385973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392423+0000","last_change":"2024-05-09T03:11:26.392423+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392423+0000","last_undegraded":"2024-05-09T03:11:26.392423+0000","last_fullsized":"2024-05-09T03:11:26.392423+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391928+0000","last_change":"2024-05-09T03:11:26.391928+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391928+0000","last_undegraded":"2024-05-09T03:11:26.391928+0000","last_fullsized":"2024-05-09T03:11:26.391928+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388354+0000","last_change":"2024-05-09T03:11:26.388354+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388354+0000","last_undegraded":"2024-05-09T03:11:26.388354+0000","last_fullsized":"2024-05-09T03:11:26.388354+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393240+0000","last_change":"2024-05-09T03:11:26.393240+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393240+0000","last_undegraded":"2024-05-09T03:11:26.393240+0000","last_fullsized":"2024-05-09T03:11:26.393240+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391939+0000","last_change":"2024-05-09T03:11:26.391939+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391939+0000","last_undegraded":"2024-05-09T03:11:26.391939+0000","last_fullsized":"2024-05-09T03:11:26.391939+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389988+0000","last_change":"2024-05-09T03:11:26.389988+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389988+0000","last_undegraded":"2024-05-09T03:11:26.389988+0000","last_fullsized":"2024-05-09T03:11:26.389988+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406501+0000","last_change":"2024-05-09T03:11:26.395704+0000","last_active":"2024-05-09T03:11:26.406501+0000","last_peered":"2024-05-09T03:11:26.406501+0000","last_clean":"2024-05-09T03:11:26.406501+0000","last_became_active":"2024-05-09T03:11:26.394437+0000","last_became_peered":"2024-05-09T03:11:26.394437+0000","last_unstale":"2024-05-09T03:11:26.406501+0000","last_undegraded":"2024-05-09T03:11:26.406501+0000","last_fullsized":"2024-05-09T03:11:26.406501+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391552+0000","last_change":"2024-05-09T03:11:26.391552+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391552+0000","last_undegraded":"2024-05-09T03:11:26.391552+0000","last_fullsized":"2024-05-09T03:11:26.391552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402507+0000","last_change":"2024-05-09T03:11:26.395710+0000","last_active":"2024-05-09T03:11:26.402507+0000","last_peered":"2024-05-09T03:11:26.402507+0000","last_clean":"2024-05-09T03:11:26.402507+0000","last_became_active":"2024-05-09T03:11:26.395376+0000","last_became_peered":"2024-05-09T03:11:26.395376+0000","last_unstale":"2024-05-09T03:11:26.402507+0000","last_undegraded":"2024-05-09T03:11:26.402507+0000","last_fullsized":"2024-05-09T03:11:26.402507+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:15.450135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391936+0000","last_change":"2024-05-09T03:11:26.391936+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391936+0000","last_undegraded":"2024-05-09T03:11:26.391936+0000","last_fullsized":"2024-05-09T03:11:26.391936+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401360+0000","last_change":"2024-05-09T03:11:26.400247+0000","last_active":"2024-05-09T03:11:26.401360+0000","last_peered":"2024-05-09T03:11:26.401360+0000","last_clean":"2024-05-09T03:11:26.401360+0000","last_became_active":"2024-05-09T03:11:26.399921+0000","last_became_peered":"2024-05-09T03:11:26.399921+0000","last_unstale":"2024-05-09T03:11:26.401360+0000","last_undegraded":"2024-05-09T03:11:26.401360+0000","last_fullsized":"2024-05-09T03:11:26.401360+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390640+0000","last_change":"2024-05-09T03:11:26.390640+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390640+0000","last_undegraded":"2024-05-09T03:11:26.390640+0000","last_fullsized":"2024-05-09T03:11:26.390640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402127+0000","last_change":"2024-05-09T03:11:26.400502+0000","last_active":"2024-05-09T03:11:26.402127+0000","last_peered":"2024-05-09T03:11:26.402127+0000","last_clean":"2024-05-09T03:11:26.402127+0000","last_became_active":"2024-05-09T03:11:26.400182+0000","last_became_peered":"2024-05-09T03:11:26.400182+0000","last_unstale":"2024-05-09T03:11:26.402127+0000","last_undegraded":"2024-05-09T03:11:26.402127+0000","last_fullsized":"2024-05-09T03:11:26.402127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:49.354980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386654+0000","last_change":"2024-05-09T03:11:26.386654+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386654+0000","last_undegraded":"2024-05-09T03:11:26.386654+0000","last_fullsized":"2024-05-09T03:11:26.386654+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406565+0000","last_change":"2024-05-09T03:11:26.397298+0000","last_active":"2024-05-09T03:11:26.406565+0000","last_peered":"2024-05-09T03:11:26.406565+0000","last_clean":"2024-05-09T03:11:26.406565+0000","last_became_active":"2024-05-09T03:11:26.397003+0000","last_became_peered":"2024-05-09T03:11:26.397003+0000","last_unstale":"2024-05-09T03:11:26.406565+0000","last_undegraded":"2024-05-09T03:11:26.406565+0000","last_fullsized":"2024-05-09T03:11:26.406565+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.400278+0000","last_change":"2024-05-09T03:11:26.400278+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.400278+0000","last_undegraded":"2024-05-09T03:11:26.400278+0000","last_fullsized":"2024-05-09T03:11:26.400278+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405578+0000","last_change":"2024-05-09T03:11:26.397547+0000","last_active":"2024-05-09T03:11:26.405578+0000","last_peered":"2024-05-09T03:11:26.405578+0000","last_clean":"2024-05-09T03:11:26.405578+0000","last_became_active":"2024-05-09T03:11:26.397248+0000","last_became_peered":"2024-05-09T03:11:26.397248+0000","last_unstale":"2024-05-09T03:11:26.405578+0000","last_undegraded":"2024-05-09T03:11:26.405578+0000","last_fullsized":"2024-05-09T03:11:26.405578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:10:57.344324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391644+0000","last_change":"2024-05-09T03:11:26.391644+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391644+0000","last_undegraded":"2024-05-09T03:11:26.391644+0000","last_fullsized":"2024-05-09T03:11:26.391644+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406549+0000","last_change":"2024-05-09T03:11:26.402930+0000","last_active":"2024-05-09T03:11:26.406549+0000","last_peered":"2024-05-09T03:11:26.406549+0000","last_clean":"2024-05-09T03:11:26.406549+0000","last_became_active":"2024-05-09T03:11:26.402595+0000","last_became_peered":"2024-05-09T03:11:26.402595+0000","last_unstale":"2024-05-09T03:11:26.406549+0000","last_undegraded":"2024-05-09T03:11:26.406549+0000","last_fullsized":"2024-05-09T03:11:26.406549+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:12.461211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393099+0000","last_change":"2024-05-09T03:11:26.393099+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393099+0000","last_undegraded":"2024-05-09T03:11:26.393099+0000","last_fullsized":"2024-05-09T03:11:26.393099+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407684+0000","last_change":"2024-05-09T03:11:26.405390+0000","last_active":"2024-05-09T03:11:26.407684+0000","last_peered":"2024-05-09T03:11:26.407684+0000","last_clean":"2024-05-09T03:11:26.407684+0000","last_became_active":"2024-05-09T03:11:26.404946+0000","last_became_peered":"2024-05-09T03:11:26.404946+0000","last_unstale":"2024-05-09T03:11:26.407684+0000","last_undegraded":"2024-05-09T03:11:26.407684+0000","last_fullsized":"2024-05-09T03:11:26.407684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:25:32.094306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.387537+0000","last_change":"2024-05-09T03:11:26.387537+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.387537+0000","last_undegraded":"2024-05-09T03:11:26.387537+0000","last_fullsized":"2024-05-09T03:11:26.387537+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405664+0000","last_change":"2024-05-09T03:11:26.398230+0000","last_active":"2024-05-09T03:11:26.405664+0000","last_peered":"2024-05-09T03:11:26.405664+0000","last_clean":"2024-05-09T03:11:26.405664+0000","last_became_active":"2024-05-09T03:11:26.397938+0000","last_became_peered":"2024-05-09T03:11:26.397938+0000","last_unstale":"2024-05-09T03:11:26.405664+0000","last_undegraded":"2024-05-09T03:11:26.405664+0000","last_fullsized":"2024-05-09T03:11:26.405664+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385422+0000","last_change":"2024-05-09T03:11:26.385422+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385422+0000","last_undegraded":"2024-05-09T03:11:26.385422+0000","last_fullsized":"2024-05-09T03:11:26.385422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408635+0000","last_change":"2024-05-09T03:11:26.407940+0000","last_active":"2024-05-09T03:11:26.408635+0000","last_peered":"2024-05-09T03:11:26.408635+0000","last_clean":"2024-05-09T03:11:26.408635+0000","last_became_active":"2024-05-09T03:11:26.407561+0000","last_became_peered":"2024-05-09T03:11:26.407561+0000","last_unstale":"2024-05-09T03:11:26.408635+0000","last_undegraded":"2024-05-09T03:11:26.408635+0000","last_fullsized":"2024-05-09T03:11:26.408635+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:15.450135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393376+0000","last_change":"2024-05-09T03:11:26.393376+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393376+0000","last_undegraded":"2024-05-09T03:11:26.393376+0000","last_fullsized":"2024-05-09T03:11:26.393376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402424+0000","last_change":"2024-05-09T03:11:26.401361+0000","last_active":"2024-05-09T03:11:26.402424+0000","last_peered":"2024-05-09T03:11:26.402424+0000","last_clean":"2024-05-09T03:11:26.402424+0000","last_became_active":"2024-05-09T03:11:26.401038+0000","last_became_peered":"2024-05-09T03:11:26.401038+0000","last_unstale":"2024-05-09T03:11:26.402424+0000","last_undegraded":"2024-05-09T03:11:26.402424+0000","last_fullsized":"2024-05-09T03:11:26.402424+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:28.972509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386048+0000","last_change":"2024-05-09T03:11:26.386048+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386048+0000","last_undegraded":"2024-05-09T03:11:26.386048+0000","last_fullsized":"2024-05-09T03:11:26.386048+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407226+0000","last_change":"2024-05-09T03:11:26.406530+0000","last_active":"2024-05-09T03:11:26.407226+0000","last_peered":"2024-05-09T03:11:26.407226+0000","last_clean":"2024-05-09T03:11:26.407226+0000","last_became_active":"2024-05-09T03:11:26.406173+0000","last_became_peered":"2024-05-09T03:11:26.406173+0000","last_unstale":"2024-05-09T03:11:26.407226+0000","last_undegraded":"2024-05-09T03:11:26.407226+0000","last_fullsized":"2024-05-09T03:11:26.407226+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:03:28.087361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391660+0000","last_change":"2024-05-09T03:11:26.391660+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391660+0000","last_undegraded":"2024-05-09T03:11:26.391660+0000","last_fullsized":"2024-05-09T03:11:26.391660+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407154+0000","last_change":"2024-05-09T03:11:26.402684+0000","last_active":"2024-05-09T03:11:26.407154+0000","last_peered":"2024-05-09T03:11:26.407154+0000","last_clean":"2024-05-09T03:11:26.407154+0000","last_became_active":"2024-05-09T03:11:26.402325+0000","last_became_peered":"2024-05-09T03:11:26.402325+0000","last_unstale":"2024-05-09T03:11:26.407154+0000","last_undegraded":"2024-05-09T03:11:26.407154+0000","last_fullsized":"2024-05-09T03:11:26.407154+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:24.688956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399627+0000","last_change":"2024-05-09T03:11:26.399627+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399627+0000","last_undegraded":"2024-05-09T03:11:26.399627+0000","last_fullsized":"2024-05-09T03:11:26.399627+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401609+0000","last_change":"2024-05-09T03:11:26.399378+0000","last_active":"2024-05-09T03:11:26.401609+0000","last_peered":"2024-05-09T03:11:26.401609+0000","last_clean":"2024-05-09T03:11:26.401609+0000","last_became_active":"2024-05-09T03:11:26.399091+0000","last_became_peered":"2024-05-09T03:11:26.399091+0000","last_unstale":"2024-05-09T03:11:26.401609+0000","last_undegraded":"2024-05-09T03:11:26.401609+0000","last_fullsized":"2024-05-09T03:11:26.401609+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:38.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393393+0000","last_change":"2024-05-09T03:11:26.393393+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393393+0000","last_undegraded":"2024-05-09T03:11:26.393393+0000","last_fullsized":"2024-05-09T03:11:26.393393+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406631+0000","last_change":"2024-05-09T03:11:26.398576+0000","last_active":"2024-05-09T03:11:26.406631+0000","last_peered":"2024-05-09T03:11:26.406631+0000","last_clean":"2024-05-09T03:11:26.406631+0000","last_became_active":"2024-05-09T03:11:26.398273+0000","last_became_peered":"2024-05-09T03:11:26.398273+0000","last_unstale":"2024-05-09T03:11:26.406631+0000","last_undegraded":"2024-05-09T03:11:26.406631+0000","last_fullsized":"2024-05-09T03:11:26.406631+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385398+0000","last_change":"2024-05-09T03:11:26.385398+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385398+0000","last_undegraded":"2024-05-09T03:11:26.385398+0000","last_fullsized":"2024-05-09T03:11:26.385398+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405242+0000","last_change":"2024-05-09T03:11:26.403484+0000","last_active":"2024-05-09T03:11:26.405242+0000","last_peered":"2024-05-09T03:11:26.405242+0000","last_clean":"2024-05-09T03:11:26.405242+0000","last_became_active":"2024-05-09T03:11:26.403147+0000","last_became_peered":"2024-05-09T03:11:26.403147+0000","last_unstale":"2024-05-09T03:11:26.405242+0000","last_undegraded":"2024-05-09T03:11:26.405242+0000","last_fullsized":"2024-05-09T03:11:26.405242+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:48.730535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390014+0000","last_change":"2024-05-09T03:11:26.390014+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390014+0000","last_undegraded":"2024-05-09T03:11:26.390014+0000","last_fullsized":"2024-05-09T03:11:26.390014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.409491+0000","last_change":"2024-05-09T03:11:26.408975+0000","last_active":"2024-05-09T03:11:26.409491+0000","last_peered":"2024-05-09T03:11:26.409491+0000","last_clean":"2024-05-09T03:11:26.409491+0000","last_became_active":"2024-05-09T03:11:26.408602+0000","last_became_peered":"2024-05-09T03:11:26.408602+0000","last_unstale":"2024-05-09T03:11:26.409491+0000","last_undegraded":"2024-05-09T03:11:26.409491+0000","last_fullsized":"2024-05-09T03:11:26.409491+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.886217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398258+0000","last_change":"2024-05-09T03:11:26.398258+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398258+0000","last_undegraded":"2024-05-09T03:11:26.398258+0000","last_fullsized":"2024-05-09T03:11:26.398258+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405558+0000","last_change":"2024-05-09T03:11:26.399038+0000","last_active":"2024-05-09T03:11:26.405558+0000","last_peered":"2024-05-09T03:11:26.405558+0000","last_clean":"2024-05-09T03:11:26.405558+0000","last_became_active":"2024-05-09T03:11:26.398728+0000","last_became_peered":"2024-05-09T03:11:26.398728+0000","last_unstale":"2024-05-09T03:11:26.405558+0000","last_undegraded":"2024-05-09T03:11:26.405558+0000","last_fullsized":"2024-05-09T03:11:26.405558+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391987+0000","last_change":"2024-05-09T03:11:26.391987+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391987+0000","last_undegraded":"2024-05-09T03:11:26.391987+0000","last_fullsized":"2024-05-09T03:11:26.391987+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405551+0000","last_change":"2024-05-09T03:11:26.402375+0000","last_active":"2024-05-09T03:11:26.405551+0000","last_peered":"2024-05-09T03:11:26.405551+0000","last_clean":"2024-05-09T03:11:26.405551+0000","last_became_active":"2024-05-09T03:11:26.402033+0000","last_became_peered":"2024-05-09T03:11:26.402033+0000","last_unstale":"2024-05-09T03:11:26.405551+0000","last_undegraded":"2024-05-09T03:11:26.405551+0000","last_fullsized":"2024-05-09T03:11:26.405551+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:38.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389422+0000","last_change":"2024-05-09T03:11:26.389422+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389422+0000","last_undegraded":"2024-05-09T03:11:26.389422+0000","last_fullsized":"2024-05-09T03:11:26.389422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407153+0000","last_change":"2024-05-09T03:11:26.401461+0000","last_active":"2024-05-09T03:11:26.407153+0000","last_peered":"2024-05-09T03:11:26.407153+0000","last_clean":"2024-05-09T03:11:26.407153+0000","last_became_active":"2024-05-09T03:11:26.401148+0000","last_became_peered":"2024-05-09T03:11:26.401148+0000","last_unstale":"2024-05-09T03:11:26.407153+0000","last_undegraded":"2024-05-09T03:11:26.407153+0000","last_fullsized":"2024-05-09T03:11:26.407153+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:08.673574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392466+0000","last_change":"2024-05-09T03:11:26.392466+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392466+0000","last_undegraded":"2024-05-09T03:11:26.392466+0000","last_fullsized":"2024-05-09T03:11:26.392466+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407109+0000","last_change":"2024-05-09T03:11:26.400381+0000","last_active":"2024-05-09T03:11:26.407109+0000","last_peered":"2024-05-09T03:11:26.407109+0000","last_clean":"2024-05-09T03:11:26.407109+0000","last_became_active":"2024-05-09T03:11:26.400059+0000","last_became_peered":"2024-05-09T03:11:26.400059+0000","last_unstale":"2024-05-09T03:11:26.407109+0000","last_undegraded":"2024-05-09T03:11:26.407109+0000","last_fullsized":"2024-05-09T03:11:26.407109+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:11.134274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392034+0000","last_change":"2024-05-09T03:11:26.392034+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392034+0000","last_undegraded":"2024-05-09T03:11:26.392034+0000","last_fullsized":"2024-05-09T03:11:26.392034+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406578+0000","last_change":"2024-05-09T03:11:26.403917+0000","last_active":"2024-05-09T03:11:26.406578+0000","last_peered":"2024-05-09T03:11:26.406578+0000","last_clean":"2024-05-09T03:11:26.406578+0000","last_became_active":"2024-05-09T03:11:26.403487+0000","last_became_peered":"2024-05-09T03:11:26.403487+0000","last_unstale":"2024-05-09T03:11:26.406578+0000","last_undegraded":"2024-05-09T03:11:26.406578+0000","last_fullsized":"2024-05-09T03:11:26.406578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:49.354980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392477+0000","last_change":"2024-05-09T03:11:26.392477+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392477+0000","last_undegraded":"2024-05-09T03:11:26.392477+0000","last_fullsized":"2024-05-09T03:11:26.392477+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408299+0000","last_change":"2024-05-09T03:11:26.406762+0000","last_active":"2024-05-09T03:11:26.408299+0000","last_peered":"2024-05-09T03:11:26.408299+0000","last_clean":"2024-05-09T03:11:26.408299+0000","last_became_active":"2024-05-09T03:11:26.406432+0000","last_became_peered":"2024-05-09T03:11:26.406432+0000","last_unstale":"2024-05-09T03:11:26.408299+0000","last_undegraded":"2024-05-09T03:11:26.408299+0000","last_fullsized":"2024-05-09T03:11:26.408299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:49.151611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390722+0000","last_change":"2024-05-09T03:11:26.390722+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390722+0000","last_undegraded":"2024-05-09T03:11:26.390722+0000","last_fullsized":"2024-05-09T03:11:26.390722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401829+0000","last_change":"2024-05-09T03:11:26.399861+0000","last_active":"2024-05-09T03:11:26.401829+0000","last_peered":"2024-05-09T03:11:26.401829+0000","last_clean":"2024-05-09T03:11:26.401829+0000","last_became_active":"2024-05-09T03:11:26.399383+0000","last_became_peered":"2024-05-09T03:11:26.399383+0000","last_unstale":"2024-05-09T03:11:26.401829+0000","last_undegraded":"2024-05-09T03:11:26.401829+0000","last_fullsized":"2024-05-09T03:11:26.401829+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:12.461211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391073+0000","last_change":"2024-05-09T03:11:26.391073+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391073+0000","last_undegraded":"2024-05-09T03:11:26.391073+0000","last_fullsized":"2024-05-09T03:11:26.391073+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406669+0000","last_change":"2024-05-09T03:11:26.398315+0000","last_active":"2024-05-09T03:11:26.406669+0000","last_peered":"2024-05-09T03:11:26.406669+0000","last_clean":"2024-05-09T03:11:26.406669+0000","last_became_active":"2024-05-09T03:11:26.397997+0000","last_became_peered":"2024-05-09T03:11:26.397997+0000","last_unstale":"2024-05-09T03:11:26.406669+0000","last_undegraded":"2024-05-09T03:11:26.406669+0000","last_fullsized":"2024-05-09T03:11:26.406669+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392309+0000","last_change":"2024-05-09T03:11:26.392309+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392309+0000","last_undegraded":"2024-05-09T03:11:26.392309+0000","last_fullsized":"2024-05-09T03:11:26.392309+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402447+0000","last_change":"2024-05-09T03:11:26.401410+0000","last_active":"2024-05-09T03:11:26.402447+0000","last_peered":"2024-05-09T03:11:26.402447+0000","last_clean":"2024-05-09T03:11:26.402447+0000","last_became_active":"2024-05-09T03:11:26.401111+0000","last_became_peered":"2024-05-09T03:11:26.401111+0000","last_unstale":"2024-05-09T03:11:26.402447+0000","last_undegraded":"2024-05-09T03:11:26.402447+0000","last_fullsized":"2024-05-09T03:11:26.402447+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385409+0000","last_change":"2024-05-09T03:11:26.385409+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385409+0000","last_undegraded":"2024-05-09T03:11:26.385409+0000","last_fullsized":"2024-05-09T03:11:26.385409+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401845+0000","last_change":"2024-05-09T03:11:26.401170+0000","last_active":"2024-05-09T03:11:26.401845+0000","last_peered":"2024-05-09T03:11:26.401845+0000","last_clean":"2024-05-09T03:11:26.401845+0000","last_became_active":"2024-05-09T03:11:26.400832+0000","last_became_peered":"2024-05-09T03:11:26.400832+0000","last_unstale":"2024-05-09T03:11:26.401845+0000","last_undegraded":"2024-05-09T03:11:26.401845+0000","last_fullsized":"2024-05-09T03:11:26.401845+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389437+0000","last_change":"2024-05-09T03:11:26.389437+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389437+0000","last_undegraded":"2024-05-09T03:11:26.389437+0000","last_fullsized":"2024-05-09T03:11:26.389437+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402144+0000","last_change":"2024-05-09T03:11:26.400568+0000","last_active":"2024-05-09T03:11:26.402144+0000","last_peered":"2024-05-09T03:11:26.402144+0000","last_clean":"2024-05-09T03:11:26.402144+0000","last_became_active":"2024-05-09T03:11:26.400253+0000","last_became_peered":"2024-05-09T03:11:26.400253+0000","last_unstale":"2024-05-09T03:11:26.402144+0000","last_undegraded":"2024-05-09T03:11:26.402144+0000","last_fullsized":"2024-05-09T03:11:26.402144+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:40.713887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389149+0000","last_change":"2024-05-09T03:11:26.389149+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389149+0000","last_undegraded":"2024-05-09T03:11:26.389149+0000","last_fullsized":"2024-05-09T03:11:26.389149+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405674+0000","last_change":"2024-05-09T03:11:26.397676+0000","last_active":"2024-05-09T03:11:26.405674+0000","last_peered":"2024-05-09T03:11:26.405674+0000","last_clean":"2024-05-09T03:11:26.405674+0000","last_became_active":"2024-05-09T03:11:26.396811+0000","last_became_peered":"2024-05-09T03:11:26.396811+0000","last_unstale":"2024-05-09T03:11:26.405674+0000","last_undegraded":"2024-05-09T03:11:26.405674+0000","last_fullsized":"2024-05-09T03:11:26.405674+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:30.350454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391430+0000","last_change":"2024-05-09T03:11:26.391430+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391430+0000","last_undegraded":"2024-05-09T03:11:26.391430+0000","last_fullsized":"2024-05-09T03:11:26.391430+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.403855+0000","last_change":"2024-05-09T03:11:26.402849+0000","last_active":"2024-05-09T03:11:26.403855+0000","last_peered":"2024-05-09T03:11:26.403855+0000","last_clean":"2024-05-09T03:11:26.403855+0000","last_became_active":"2024-05-09T03:11:26.402533+0000","last_became_peered":"2024-05-09T03:11:26.402533+0000","last_unstale":"2024-05-09T03:11:26.403855+0000","last_undegraded":"2024-05-09T03:11:26.403855+0000","last_fullsized":"2024-05-09T03:11:26.403855+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385911+0000","last_change":"2024-05-09T03:11:26.385911+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385911+0000","last_undegraded":"2024-05-09T03:11:26.385911+0000","last_fullsized":"2024-05-09T03:11:26.385911+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.398696+0000","last_change":"2024-05-09T03:11:26.396367+0000","last_active":"2024-05-09T03:11:26.398696+0000","last_peered":"2024-05-09T03:11:26.398696+0000","last_clean":"2024-05-09T03:11:26.398696+0000","last_became_active":"2024-05-09T03:11:26.396006+0000","last_became_peered":"2024-05-09T03:11:26.396006+0000","last_unstale":"2024-05-09T03:11:26.398696+0000","last_undegraded":"2024-05-09T03:11:26.398696+0000","last_fullsized":"2024-05-09T03:11:26.398696+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:42.938942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398327+0000","last_change":"2024-05-09T03:11:26.398327+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398327+0000","last_undegraded":"2024-05-09T03:11:26.398327+0000","last_fullsized":"2024-05-09T03:11:26.398327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408710+0000","last_change":"2024-05-09T03:11:26.407676+0000","last_active":"2024-05-09T03:11:26.408710+0000","last_peered":"2024-05-09T03:11:26.408710+0000","last_clean":"2024-05-09T03:11:26.408710+0000","last_became_active":"2024-05-09T03:11:26.407284+0000","last_became_peered":"2024-05-09T03:11:26.407284+0000","last_unstale":"2024-05-09T03:11:26.408710+0000","last_undegraded":"2024-05-09T03:11:26.408710+0000","last_fullsized":"2024-05-09T03:11:26.408710+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:40.713887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.390235+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:11:26.390235+0000","last_peered":"2024-05-09T03:11:26.390235+0000","last_clean":"2024-05-09T03:11:26.390235+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:11:26.390235+0000","last_undegraded":"2024-05-09T03:11:26.390235+0000","last_fullsized":"2024-05-09T03:11:26.390235+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":774839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":774839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:11:31.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:11:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.660+0000 7f45c9805640 1 Processor -- start 2024-05-09T03:11:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.661+0000 7f45c9805640 1 -- start start 2024-05-09T03:11:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c9805640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45c40b9310 0x7f45c408cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c9805640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c4165650 con 0x7f45c408d6b0 2024-05-09T03:11:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c9805640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c406ebe0 con 0x7f45c408d2a0 2024-05-09T03:11:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c9805640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f45c406f760 con 0x7f45c40b9310 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45c40b9310 0x7f45c408cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f45c408d6b0 0x7f45c4164f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57946/0 (socket says 172.21.15.46:57946) 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45c40b9310 0x7f45c408cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60196/0 (socket says 172.21.15.46:60196) 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c37fe640 1 -- 172.21.15.46:0/3405192033 learned_addr learned my addr 172.21.15.46:0/3405192033 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3405192033 >> v1:172.21.15.88:6789/0 conn(0x7f45c408d2a0 legacy=0x7f45c4164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3405192033 >> v1:172.21.15.46:6789/0 conn(0x7f45c408d6b0 legacy=0x7f45c4164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c1ffb640 1 -- 172.21.15.46:0/3405192033 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801123397 0 0) 0x7f45c4165650 con 0x7f45c408d6b0 2024-05-09T03:11:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.662+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3405192033 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c406eec0 con 0x7f45c40b9310 2024-05-09T03:11:31.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.663+0000 7f45c2ffd640 1 --2- 172.21.15.46:0/3405192033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c40b9310 0x7f45c408cd60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009d90 tx=0x7f45b402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f23495d6ebf654d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.663+0000 7f45c1ffb640 1 -- 172.21.15.46:0/3405192033 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4007740 con 0x7f45c40b9310 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.663+0000 7f45c1ffb640 1 -- 172.21.15.46:0/3405192033 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b40078e0 con 0x7f45c40b9310 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.663+0000 7f45c1ffb640 1 -- 172.21.15.46:0/3405192033 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4007cb0 con 0x7f45c40b9310 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 -- 172.21.15.46:0/3405192033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c40b9310 msgr2=0x7f45c408cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 --2- 172.21.15.46:0/3405192033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c40b9310 0x7f45c408cd60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009d90 tx=0x7f45b402a660 comp rx=0 tx=0).stop 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 -- 172.21.15.46:0/3405192033 shutdown_connections 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 --2- 172.21.15.46:0/3405192033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c40b9310 0x7f45c408cd60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 -- 172.21.15.46:0/3405192033 >> 172.21.15.46:0/3405192033 conn(0x7f45c40b5660 msgr2=0x7f45c40b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 -- 172.21.15.46:0/3405192033 shutdown_connections 2024-05-09T03:11:31.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.664+0000 7f45c9805640 1 -- 172.21.15.46:0/3405192033 wait complete. 2024-05-09T03:11:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.666+0000 7f45c9805640 1 Processor -- start 2024-05-09T03:11:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.666+0000 7f45c9805640 1 -- start start 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c9805640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 0x7f45c415ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c9805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45c408d6b0 0x7f45c41f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c9805640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c40b9310 0x7f45c40b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c9805640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45c408dac0 con 0x7f45c408d2a0 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c9805640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f45c406f480 con 0x7f45c40b9310 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c9805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f45c40cf590 con 0x7f45c408d6b0 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 0x7f45c415ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 0x7f45c415ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60206/0 (socket says 172.21.15.46:60206) 2024-05-09T03:11:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.667+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3644909576 learned_addr learned my addr 172.21.15.46:0/3644909576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c37fe640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c40b9310 0x7f45c40b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c27fc640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45c408d6b0 0x7f45c41f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45c408d6b0 msgr2=0x7f45c41f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c2ffd640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45c408d6b0 0x7f45c41f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c40b9310 msgr2=0x7f45c40b7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c2ffd640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c40b9310 0x7f45c40b7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c2ffd640 1 -- 172.21.15.46:0/3644909576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c415d9f0 con 0x7f45c408d2a0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c2ffd640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 0x7f45c415ad50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009d90 tx=0x7f45b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b4040070 con 0x7f45c408d2a0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b4031c40 con 0x7f45c408d2a0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c415b290 con 0x7f45c408d2a0 2024-05-09T03:11:31.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b403c420 con 0x7f45c408d2a0 2024-05-09T03:11:31.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.668+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c415b660 con 0x7f45c408d2a0 2024-05-09T03:11:31.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.669+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c415ded0 con 0x7f45c408d2a0 2024-05-09T03:11:31.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.670+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f45b4039040 con 0x7f45c408d2a0 2024-05-09T03:11:31.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.671+0000 7f45abfff640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f45b00762d0 0x7f45b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:31.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.671+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f45b40b9070 con 0x7f45c408d2a0 2024-05-09T03:11:31.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.671+0000 7f45c27fc640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f45b00762d0 0x7f45b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:31.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.671+0000 7f45c27fc640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f45b00762d0 0x7f45b00787e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45b8006fd0 tx=0x7f45b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:31.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.674+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f45b40829b0 con 0x7f45c408d2a0 2024-05-09T03:11:31.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.861+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f45c406a270 con 0x7f45b00762d0 2024-05-09T03:11:31.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.868+0000 7f45abfff640 1 -- 172.21.15.46:0/3644909576 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7f45c406a270 con 0x7f45b00762d0 2024-05-09T03:11:31.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:31.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f45b00762d0 msgr2=0x7f45b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f45b00762d0 0x7f45b00787e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45b8006fd0 tx=0x7f45b8008040 comp rx=0 tx=0).stop 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 msgr2=0x7f45c415ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 0x7f45c415ad50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f45b4009d90 tx=0x7f45b4004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 shutdown_connections 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c40b9310 0x7f45c40b7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45c408d6b0 0x7f45c41f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f45b00762d0 0x7f45b00787e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 --2- 172.21.15.46:0/3644909576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45c408d2a0 0x7f45c415ad50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 >> 172.21.15.46:0/3644909576 conn(0x7f45c40b5660 msgr2=0x7f45c4071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 shutdown_connections 2024-05-09T03:11:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:31.871+0000 7f45c9805640 1 -- 172.21.15.46:0/3644909576 wait complete. 2024-05-09T03:11:31.884 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-05-09T03:11:30.520630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237672,"kb_used_data":4832,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737048,"statfs":{"total":773094113280,"available":772850737152,"internally_reserved":0,"allocated":4947968,"data_stored":2669038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.137100"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386053+0000","last_change":"2024-05-09T03:11:26.386053+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386053+0000","last_undegraded":"2024-05-09T03:11:26.386053+0000","last_fullsized":"2024-05-09T03:11:26.386053+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393641+0000","last_change":"2024-05-09T03:11:26.393641+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393641+0000","last_undegraded":"2024-05-09T03:11:26.393641+0000","last_fullsized":"2024-05-09T03:11:26.393641+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386800+0000","last_change":"2024-05-09T03:11:26.386800+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386800+0000","last_undegraded":"2024-05-09T03:11:26.386800+0000","last_fullsized":"2024-05-09T03:11:26.386800+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399458+0000","last_change":"2024-05-09T03:11:26.399458+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399458+0000","last_undegraded":"2024-05-09T03:11:26.399458+0000","last_fullsized":"2024-05-09T03:11:26.399458+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393056+0000","last_change":"2024-05-09T03:11:26.393056+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393056+0000","last_undegraded":"2024-05-09T03:11:26.393056+0000","last_fullsized":"2024-05-09T03:11:26.393056+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393193+0000","last_change":"2024-05-09T03:11:26.393193+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393193+0000","last_undegraded":"2024-05-09T03:11:26.393193+0000","last_fullsized":"2024-05-09T03:11:26.393193+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.387803+0000","last_change":"2024-05-09T03:11:26.387803+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.387803+0000","last_undegraded":"2024-05-09T03:11:26.387803+0000","last_fullsized":"2024-05-09T03:11:26.387803+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392140+0000","last_change":"2024-05-09T03:11:26.392140+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392140+0000","last_undegraded":"2024-05-09T03:11:26.392140+0000","last_fullsized":"2024-05-09T03:11:26.392140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399153+0000","last_change":"2024-05-09T03:11:26.399153+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399153+0000","last_undegraded":"2024-05-09T03:11:26.399153+0000","last_fullsized":"2024-05-09T03:11:26.399153+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399862+0000","last_change":"2024-05-09T03:11:26.399862+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399862+0000","last_undegraded":"2024-05-09T03:11:26.399862+0000","last_fullsized":"2024-05-09T03:11:26.399862+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398110+0000","last_change":"2024-05-09T03:11:26.398110+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398110+0000","last_undegraded":"2024-05-09T03:11:26.398110+0000","last_fullsized":"2024-05-09T03:11:26.398110+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393757+0000","last_change":"2024-05-09T03:11:26.393757+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393757+0000","last_undegraded":"2024-05-09T03:11:26.393757+0000","last_fullsized":"2024-05-09T03:11:26.393757+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389234+0000","last_change":"2024-05-09T03:11:26.389234+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389234+0000","last_undegraded":"2024-05-09T03:11:26.389234+0000","last_fullsized":"2024-05-09T03:11:26.389234+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391877+0000","last_change":"2024-05-09T03:11:26.391877+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391877+0000","last_undegraded":"2024-05-09T03:11:26.391877+0000","last_fullsized":"2024-05-09T03:11:26.391877+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393846+0000","last_change":"2024-05-09T03:11:26.393846+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393846+0000","last_undegraded":"2024-05-09T03:11:26.393846+0000","last_fullsized":"2024-05-09T03:11:26.393846+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392475+0000","last_change":"2024-05-09T03:11:26.392475+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392475+0000","last_undegraded":"2024-05-09T03:11:26.392475+0000","last_fullsized":"2024-05-09T03:11:26.392475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388512+0000","last_change":"2024-05-09T03:11:26.388512+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388512+0000","last_undegraded":"2024-05-09T03:11:26.388512+0000","last_fullsized":"2024-05-09T03:11:26.388512+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388619+0000","last_change":"2024-05-09T03:11:26.388619+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388619+0000","last_undegraded":"2024-05-09T03:11:26.388619+0000","last_fullsized":"2024-05-09T03:11:26.388619+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388811+0000","last_change":"2024-05-09T03:11:26.388811+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388811+0000","last_undegraded":"2024-05-09T03:11:26.388811+0000","last_fullsized":"2024-05-09T03:11:26.388811+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391447+0000","last_change":"2024-05-09T03:11:26.391447+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391447+0000","last_undegraded":"2024-05-09T03:11:26.391447+0000","last_fullsized":"2024-05-09T03:11:26.391447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393040+0000","last_change":"2024-05-09T03:11:26.393040+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393040+0000","last_undegraded":"2024-05-09T03:11:26.393040+0000","last_fullsized":"2024-05-09T03:11:26.393040+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391316+0000","last_change":"2024-05-09T03:11:26.391316+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391316+0000","last_undegraded":"2024-05-09T03:11:26.391316+0000","last_fullsized":"2024-05-09T03:11:26.391316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389605+0000","last_change":"2024-05-09T03:11:26.389605+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389605+0000","last_undegraded":"2024-05-09T03:11:26.389605+0000","last_fullsized":"2024-05-09T03:11:26.389605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391465+0000","last_change":"2024-05-09T03:11:26.391465+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391465+0000","last_undegraded":"2024-05-09T03:11:26.391465+0000","last_fullsized":"2024-05-09T03:11:26.391465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390564+0000","last_change":"2024-05-09T03:11:26.390564+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390564+0000","last_undegraded":"2024-05-09T03:11:26.390564+0000","last_fullsized":"2024-05-09T03:11:26.390564+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390514+0000","last_change":"2024-05-09T03:11:26.390514+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390514+0000","last_undegraded":"2024-05-09T03:11:26.390514+0000","last_fullsized":"2024-05-09T03:11:26.390514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385973+0000","last_change":"2024-05-09T03:11:26.385973+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385973+0000","last_undegraded":"2024-05-09T03:11:26.385973+0000","last_fullsized":"2024-05-09T03:11:26.385973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392423+0000","last_change":"2024-05-09T03:11:26.392423+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392423+0000","last_undegraded":"2024-05-09T03:11:26.392423+0000","last_fullsized":"2024-05-09T03:11:26.392423+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391928+0000","last_change":"2024-05-09T03:11:26.391928+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391928+0000","last_undegraded":"2024-05-09T03:11:26.391928+0000","last_fullsized":"2024-05-09T03:11:26.391928+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388354+0000","last_change":"2024-05-09T03:11:26.388354+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388354+0000","last_undegraded":"2024-05-09T03:11:26.388354+0000","last_fullsized":"2024-05-09T03:11:26.388354+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393240+0000","last_change":"2024-05-09T03:11:26.393240+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393240+0000","last_undegraded":"2024-05-09T03:11:26.393240+0000","last_fullsized":"2024-05-09T03:11:26.393240+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391939+0000","last_change":"2024-05-09T03:11:26.391939+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391939+0000","last_undegraded":"2024-05-09T03:11:26.391939+0000","last_fullsized":"2024-05-09T03:11:26.391939+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389988+0000","last_change":"2024-05-09T03:11:26.389988+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389988+0000","last_undegraded":"2024-05-09T03:11:26.389988+0000","last_fullsized":"2024-05-09T03:11:26.389988+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406501+0000","last_change":"2024-05-09T03:11:26.395704+0000","last_active":"2024-05-09T03:11:26.406501+0000","last_peered":"2024-05-09T03:11:26.406501+0000","last_clean":"2024-05-09T03:11:26.406501+0000","last_became_active":"2024-05-09T03:11:26.394437+0000","last_became_peered":"2024-05-09T03:11:26.394437+0000","last_unstale":"2024-05-09T03:11:26.406501+0000","last_undegraded":"2024-05-09T03:11:26.406501+0000","last_fullsized":"2024-05-09T03:11:26.406501+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391552+0000","last_change":"2024-05-09T03:11:26.391552+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391552+0000","last_undegraded":"2024-05-09T03:11:26.391552+0000","last_fullsized":"2024-05-09T03:11:26.391552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402507+0000","last_change":"2024-05-09T03:11:26.395710+0000","last_active":"2024-05-09T03:11:26.402507+0000","last_peered":"2024-05-09T03:11:26.402507+0000","last_clean":"2024-05-09T03:11:26.402507+0000","last_became_active":"2024-05-09T03:11:26.395376+0000","last_became_peered":"2024-05-09T03:11:26.395376+0000","last_unstale":"2024-05-09T03:11:26.402507+0000","last_undegraded":"2024-05-09T03:11:26.402507+0000","last_fullsized":"2024-05-09T03:11:26.402507+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:15.450135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391936+0000","last_change":"2024-05-09T03:11:26.391936+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391936+0000","last_undegraded":"2024-05-09T03:11:26.391936+0000","last_fullsized":"2024-05-09T03:11:26.391936+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401360+0000","last_change":"2024-05-09T03:11:26.400247+0000","last_active":"2024-05-09T03:11:26.401360+0000","last_peered":"2024-05-09T03:11:26.401360+0000","last_clean":"2024-05-09T03:11:26.401360+0000","last_became_active":"2024-05-09T03:11:26.399921+0000","last_became_peered":"2024-05-09T03:11:26.399921+0000","last_unstale":"2024-05-09T03:11:26.401360+0000","last_undegraded":"2024-05-09T03:11:26.401360+0000","last_fullsized":"2024-05-09T03:11:26.401360+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390640+0000","last_change":"2024-05-09T03:11:26.390640+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390640+0000","last_undegraded":"2024-05-09T03:11:26.390640+0000","last_fullsized":"2024-05-09T03:11:26.390640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402127+0000","last_change":"2024-05-09T03:11:26.400502+0000","last_active":"2024-05-09T03:11:26.402127+0000","last_peered":"2024-05-09T03:11:26.402127+0000","last_clean":"2024-05-09T03:11:26.402127+0000","last_became_active":"2024-05-09T03:11:26.400182+0000","last_became_peered":"2024-05-09T03:11:26.400182+0000","last_unstale":"2024-05-09T03:11:26.402127+0000","last_undegraded":"2024-05-09T03:11:26.402127+0000","last_fullsized":"2024-05-09T03:11:26.402127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:49.354980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386654+0000","last_change":"2024-05-09T03:11:26.386654+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386654+0000","last_undegraded":"2024-05-09T03:11:26.386654+0000","last_fullsized":"2024-05-09T03:11:26.386654+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406565+0000","last_change":"2024-05-09T03:11:26.397298+0000","last_active":"2024-05-09T03:11:26.406565+0000","last_peered":"2024-05-09T03:11:26.406565+0000","last_clean":"2024-05-09T03:11:26.406565+0000","last_became_active":"2024-05-09T03:11:26.397003+0000","last_became_peered":"2024-05-09T03:11:26.397003+0000","last_unstale":"2024-05-09T03:11:26.406565+0000","last_undegraded":"2024-05-09T03:11:26.406565+0000","last_fullsized":"2024-05-09T03:11:26.406565+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.400278+0000","last_change":"2024-05-09T03:11:26.400278+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.400278+0000","last_undegraded":"2024-05-09T03:11:26.400278+0000","last_fullsized":"2024-05-09T03:11:26.400278+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405578+0000","last_change":"2024-05-09T03:11:26.397547+0000","last_active":"2024-05-09T03:11:26.405578+0000","last_peered":"2024-05-09T03:11:26.405578+0000","last_clean":"2024-05-09T03:11:26.405578+0000","last_became_active":"2024-05-09T03:11:26.397248+0000","last_became_peered":"2024-05-09T03:11:26.397248+0000","last_unstale":"2024-05-09T03:11:26.405578+0000","last_undegraded":"2024-05-09T03:11:26.405578+0000","last_fullsized":"2024-05-09T03:11:26.405578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:10:57.344324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391644+0000","last_change":"2024-05-09T03:11:26.391644+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391644+0000","last_undegraded":"2024-05-09T03:11:26.391644+0000","last_fullsized":"2024-05-09T03:11:26.391644+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406549+0000","last_change":"2024-05-09T03:11:26.402930+0000","last_active":"2024-05-09T03:11:26.406549+0000","last_peered":"2024-05-09T03:11:26.406549+0000","last_clean":"2024-05-09T03:11:26.406549+0000","last_became_active":"2024-05-09T03:11:26.402595+0000","last_became_peered":"2024-05-09T03:11:26.402595+0000","last_unstale":"2024-05-09T03:11:26.406549+0000","last_undegraded":"2024-05-09T03:11:26.406549+0000","last_fullsized":"2024-05-09T03:11:26.406549+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:12.461211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393099+0000","last_change":"2024-05-09T03:11:26.393099+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393099+0000","last_undegraded":"2024-05-09T03:11:26.393099+0000","last_fullsized":"2024-05-09T03:11:26.393099+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407684+0000","last_change":"2024-05-09T03:11:26.405390+0000","last_active":"2024-05-09T03:11:26.407684+0000","last_peered":"2024-05-09T03:11:26.407684+0000","last_clean":"2024-05-09T03:11:26.407684+0000","last_became_active":"2024-05-09T03:11:26.404946+0000","last_became_peered":"2024-05-09T03:11:26.404946+0000","last_unstale":"2024-05-09T03:11:26.407684+0000","last_undegraded":"2024-05-09T03:11:26.407684+0000","last_fullsized":"2024-05-09T03:11:26.407684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:25:32.094306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.387537+0000","last_change":"2024-05-09T03:11:26.387537+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.387537+0000","last_undegraded":"2024-05-09T03:11:26.387537+0000","last_fullsized":"2024-05-09T03:11:26.387537+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405664+0000","last_change":"2024-05-09T03:11:26.398230+0000","last_active":"2024-05-09T03:11:26.405664+0000","last_peered":"2024-05-09T03:11:26.405664+0000","last_clean":"2024-05-09T03:11:26.405664+0000","last_became_active":"2024-05-09T03:11:26.397938+0000","last_became_peered":"2024-05-09T03:11:26.397938+0000","last_unstale":"2024-05-09T03:11:26.405664+0000","last_undegraded":"2024-05-09T03:11:26.405664+0000","last_fullsized":"2024-05-09T03:11:26.405664+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385422+0000","last_change":"2024-05-09T03:11:26.385422+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385422+0000","last_undegraded":"2024-05-09T03:11:26.385422+0000","last_fullsized":"2024-05-09T03:11:26.385422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408635+0000","last_change":"2024-05-09T03:11:26.407940+0000","last_active":"2024-05-09T03:11:26.408635+0000","last_peered":"2024-05-09T03:11:26.408635+0000","last_clean":"2024-05-09T03:11:26.408635+0000","last_became_active":"2024-05-09T03:11:26.407561+0000","last_became_peered":"2024-05-09T03:11:26.407561+0000","last_unstale":"2024-05-09T03:11:26.408635+0000","last_undegraded":"2024-05-09T03:11:26.408635+0000","last_fullsized":"2024-05-09T03:11:26.408635+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:15.450135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393376+0000","last_change":"2024-05-09T03:11:26.393376+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393376+0000","last_undegraded":"2024-05-09T03:11:26.393376+0000","last_fullsized":"2024-05-09T03:11:26.393376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402424+0000","last_change":"2024-05-09T03:11:26.401361+0000","last_active":"2024-05-09T03:11:26.402424+0000","last_peered":"2024-05-09T03:11:26.402424+0000","last_clean":"2024-05-09T03:11:26.402424+0000","last_became_active":"2024-05-09T03:11:26.401038+0000","last_became_peered":"2024-05-09T03:11:26.401038+0000","last_unstale":"2024-05-09T03:11:26.402424+0000","last_undegraded":"2024-05-09T03:11:26.402424+0000","last_fullsized":"2024-05-09T03:11:26.402424+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:28.972509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386048+0000","last_change":"2024-05-09T03:11:26.386048+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386048+0000","last_undegraded":"2024-05-09T03:11:26.386048+0000","last_fullsized":"2024-05-09T03:11:26.386048+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407226+0000","last_change":"2024-05-09T03:11:26.406530+0000","last_active":"2024-05-09T03:11:26.407226+0000","last_peered":"2024-05-09T03:11:26.407226+0000","last_clean":"2024-05-09T03:11:26.407226+0000","last_became_active":"2024-05-09T03:11:26.406173+0000","last_became_peered":"2024-05-09T03:11:26.406173+0000","last_unstale":"2024-05-09T03:11:26.407226+0000","last_undegraded":"2024-05-09T03:11:26.407226+0000","last_fullsized":"2024-05-09T03:11:26.407226+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:03:28.087361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391660+0000","last_change":"2024-05-09T03:11:26.391660+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391660+0000","last_undegraded":"2024-05-09T03:11:26.391660+0000","last_fullsized":"2024-05-09T03:11:26.391660+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407154+0000","last_change":"2024-05-09T03:11:26.402684+0000","last_active":"2024-05-09T03:11:26.407154+0000","last_peered":"2024-05-09T03:11:26.407154+0000","last_clean":"2024-05-09T03:11:26.407154+0000","last_became_active":"2024-05-09T03:11:26.402325+0000","last_became_peered":"2024-05-09T03:11:26.402325+0000","last_unstale":"2024-05-09T03:11:26.407154+0000","last_undegraded":"2024-05-09T03:11:26.407154+0000","last_fullsized":"2024-05-09T03:11:26.407154+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:24.688956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399627+0000","last_change":"2024-05-09T03:11:26.399627+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399627+0000","last_undegraded":"2024-05-09T03:11:26.399627+0000","last_fullsized":"2024-05-09T03:11:26.399627+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401609+0000","last_change":"2024-05-09T03:11:26.399378+0000","last_active":"2024-05-09T03:11:26.401609+0000","last_peered":"2024-05-09T03:11:26.401609+0000","last_clean":"2024-05-09T03:11:26.401609+0000","last_became_active":"2024-05-09T03:11:26.399091+0000","last_became_peered":"2024-05-09T03:11:26.399091+0000","last_unstale":"2024-05-09T03:11:26.401609+0000","last_undegraded":"2024-05-09T03:11:26.401609+0000","last_fullsized":"2024-05-09T03:11:26.401609+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:38.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393393+0000","last_change":"2024-05-09T03:11:26.393393+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393393+0000","last_undegraded":"2024-05-09T03:11:26.393393+0000","last_fullsized":"2024-05-09T03:11:26.393393+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406631+0000","last_change":"2024-05-09T03:11:26.398576+0000","last_active":"2024-05-09T03:11:26.406631+0000","last_peered":"2024-05-09T03:11:26.406631+0000","last_clean":"2024-05-09T03:11:26.406631+0000","last_became_active":"2024-05-09T03:11:26.398273+0000","last_became_peered":"2024-05-09T03:11:26.398273+0000","last_unstale":"2024-05-09T03:11:26.406631+0000","last_undegraded":"2024-05-09T03:11:26.406631+0000","last_fullsized":"2024-05-09T03:11:26.406631+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385398+0000","last_change":"2024-05-09T03:11:26.385398+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385398+0000","last_undegraded":"2024-05-09T03:11:26.385398+0000","last_fullsized":"2024-05-09T03:11:26.385398+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405242+0000","last_change":"2024-05-09T03:11:26.403484+0000","last_active":"2024-05-09T03:11:26.405242+0000","last_peered":"2024-05-09T03:11:26.405242+0000","last_clean":"2024-05-09T03:11:26.405242+0000","last_became_active":"2024-05-09T03:11:26.403147+0000","last_became_peered":"2024-05-09T03:11:26.403147+0000","last_unstale":"2024-05-09T03:11:26.405242+0000","last_undegraded":"2024-05-09T03:11:26.405242+0000","last_fullsized":"2024-05-09T03:11:26.405242+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:48.730535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390014+0000","last_change":"2024-05-09T03:11:26.390014+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390014+0000","last_undegraded":"2024-05-09T03:11:26.390014+0000","last_fullsized":"2024-05-09T03:11:26.390014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.409491+0000","last_change":"2024-05-09T03:11:26.408975+0000","last_active":"2024-05-09T03:11:26.409491+0000","last_peered":"2024-05-09T03:11:26.409491+0000","last_clean":"2024-05-09T03:11:26.409491+0000","last_became_active":"2024-05-09T03:11:26.408602+0000","last_became_peered":"2024-05-09T03:11:26.408602+0000","last_unstale":"2024-05-09T03:11:26.409491+0000","last_undegraded":"2024-05-09T03:11:26.409491+0000","last_fullsized":"2024-05-09T03:11:26.409491+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.886217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398258+0000","last_change":"2024-05-09T03:11:26.398258+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398258+0000","last_undegraded":"2024-05-09T03:11:26.398258+0000","last_fullsized":"2024-05-09T03:11:26.398258+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405558+0000","last_change":"2024-05-09T03:11:26.399038+0000","last_active":"2024-05-09T03:11:26.405558+0000","last_peered":"2024-05-09T03:11:26.405558+0000","last_clean":"2024-05-09T03:11:26.405558+0000","last_became_active":"2024-05-09T03:11:26.398728+0000","last_became_peered":"2024-05-09T03:11:26.398728+0000","last_unstale":"2024-05-09T03:11:26.405558+0000","last_undegraded":"2024-05-09T03:11:26.405558+0000","last_fullsized":"2024-05-09T03:11:26.405558+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391987+0000","last_change":"2024-05-09T03:11:26.391987+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391987+0000","last_undegraded":"2024-05-09T03:11:26.391987+0000","last_fullsized":"2024-05-09T03:11:26.391987+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405551+0000","last_change":"2024-05-09T03:11:26.402375+0000","last_active":"2024-05-09T03:11:26.405551+0000","last_peered":"2024-05-09T03:11:26.405551+0000","last_clean":"2024-05-09T03:11:26.405551+0000","last_became_active":"2024-05-09T03:11:26.402033+0000","last_became_peered":"2024-05-09T03:11:26.402033+0000","last_unstale":"2024-05-09T03:11:26.405551+0000","last_undegraded":"2024-05-09T03:11:26.405551+0000","last_fullsized":"2024-05-09T03:11:26.405551+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:38.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389422+0000","last_change":"2024-05-09T03:11:26.389422+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389422+0000","last_undegraded":"2024-05-09T03:11:26.389422+0000","last_fullsized":"2024-05-09T03:11:26.389422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407153+0000","last_change":"2024-05-09T03:11:26.401461+0000","last_active":"2024-05-09T03:11:26.407153+0000","last_peered":"2024-05-09T03:11:26.407153+0000","last_clean":"2024-05-09T03:11:26.407153+0000","last_became_active":"2024-05-09T03:11:26.401148+0000","last_became_peered":"2024-05-09T03:11:26.401148+0000","last_unstale":"2024-05-09T03:11:26.407153+0000","last_undegraded":"2024-05-09T03:11:26.407153+0000","last_fullsized":"2024-05-09T03:11:26.407153+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:08.673574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392466+0000","last_change":"2024-05-09T03:11:26.392466+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392466+0000","last_undegraded":"2024-05-09T03:11:26.392466+0000","last_fullsized":"2024-05-09T03:11:26.392466+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407109+0000","last_change":"2024-05-09T03:11:26.400381+0000","last_active":"2024-05-09T03:11:26.407109+0000","last_peered":"2024-05-09T03:11:26.407109+0000","last_clean":"2024-05-09T03:11:26.407109+0000","last_became_active":"2024-05-09T03:11:26.400059+0000","last_became_peered":"2024-05-09T03:11:26.400059+0000","last_unstale":"2024-05-09T03:11:26.407109+0000","last_undegraded":"2024-05-09T03:11:26.407109+0000","last_fullsized":"2024-05-09T03:11:26.407109+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:11.134274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392034+0000","last_change":"2024-05-09T03:11:26.392034+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392034+0000","last_undegraded":"2024-05-09T03:11:26.392034+0000","last_fullsized":"2024-05-09T03:11:26.392034+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406578+0000","last_change":"2024-05-09T03:11:26.403917+0000","last_active":"2024-05-09T03:11:26.406578+0000","last_peered":"2024-05-09T03:11:26.406578+0000","last_clean":"2024-05-09T03:11:26.406578+0000","last_became_active":"2024-05-09T03:11:26.403487+0000","last_became_peered":"2024-05-09T03:11:26.403487+0000","last_unstale":"2024-05-09T03:11:26.406578+0000","last_undegraded":"2024-05-09T03:11:26.406578+0000","last_fullsized":"2024-05-09T03:11:26.406578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:49.354980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392477+0000","last_change":"2024-05-09T03:11:26.392477+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392477+0000","last_undegraded":"2024-05-09T03:11:26.392477+0000","last_fullsized":"2024-05-09T03:11:26.392477+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408299+0000","last_change":"2024-05-09T03:11:26.406762+0000","last_active":"2024-05-09T03:11:26.408299+0000","last_peered":"2024-05-09T03:11:26.408299+0000","last_clean":"2024-05-09T03:11:26.408299+0000","last_became_active":"2024-05-09T03:11:26.406432+0000","last_became_peered":"2024-05-09T03:11:26.406432+0000","last_unstale":"2024-05-09T03:11:26.408299+0000","last_undegraded":"2024-05-09T03:11:26.408299+0000","last_fullsized":"2024-05-09T03:11:26.408299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:49.151611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390722+0000","last_change":"2024-05-09T03:11:26.390722+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390722+0000","last_undegraded":"2024-05-09T03:11:26.390722+0000","last_fullsized":"2024-05-09T03:11:26.390722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401829+0000","last_change":"2024-05-09T03:11:26.399861+0000","last_active":"2024-05-09T03:11:26.401829+0000","last_peered":"2024-05-09T03:11:26.401829+0000","last_clean":"2024-05-09T03:11:26.401829+0000","last_became_active":"2024-05-09T03:11:26.399383+0000","last_became_peered":"2024-05-09T03:11:26.399383+0000","last_unstale":"2024-05-09T03:11:26.401829+0000","last_undegraded":"2024-05-09T03:11:26.401829+0000","last_fullsized":"2024-05-09T03:11:26.401829+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:12.461211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391073+0000","last_change":"2024-05-09T03:11:26.391073+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391073+0000","last_undegraded":"2024-05-09T03:11:26.391073+0000","last_fullsized":"2024-05-09T03:11:26.391073+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406669+0000","last_change":"2024-05-09T03:11:26.398315+0000","last_active":"2024-05-09T03:11:26.406669+0000","last_peered":"2024-05-09T03:11:26.406669+0000","last_clean":"2024-05-09T03:11:26.406669+0000","last_became_active":"2024-05-09T03:11:26.397997+0000","last_became_peered":"2024-05-09T03:11:26.397997+0000","last_unstale":"2024-05-09T03:11:26.406669+0000","last_undegraded":"2024-05-09T03:11:26.406669+0000","last_fullsized":"2024-05-09T03:11:26.406669+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392309+0000","last_change":"2024-05-09T03:11:26.392309+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392309+0000","last_undegraded":"2024-05-09T03:11:26.392309+0000","last_fullsized":"2024-05-09T03:11:26.392309+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402447+0000","last_change":"2024-05-09T03:11:26.401410+0000","last_active":"2024-05-09T03:11:26.402447+0000","last_peered":"2024-05-09T03:11:26.402447+0000","last_clean":"2024-05-09T03:11:26.402447+0000","last_became_active":"2024-05-09T03:11:26.401111+0000","last_became_peered":"2024-05-09T03:11:26.401111+0000","last_unstale":"2024-05-09T03:11:26.402447+0000","last_undegraded":"2024-05-09T03:11:26.402447+0000","last_fullsized":"2024-05-09T03:11:26.402447+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385409+0000","last_change":"2024-05-09T03:11:26.385409+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385409+0000","last_undegraded":"2024-05-09T03:11:26.385409+0000","last_fullsized":"2024-05-09T03:11:26.385409+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401845+0000","last_change":"2024-05-09T03:11:26.401170+0000","last_active":"2024-05-09T03:11:26.401845+0000","last_peered":"2024-05-09T03:11:26.401845+0000","last_clean":"2024-05-09T03:11:26.401845+0000","last_became_active":"2024-05-09T03:11:26.400832+0000","last_became_peered":"2024-05-09T03:11:26.400832+0000","last_unstale":"2024-05-09T03:11:26.401845+0000","last_undegraded":"2024-05-09T03:11:26.401845+0000","last_fullsized":"2024-05-09T03:11:26.401845+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389437+0000","last_change":"2024-05-09T03:11:26.389437+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389437+0000","last_undegraded":"2024-05-09T03:11:26.389437+0000","last_fullsized":"2024-05-09T03:11:26.389437+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402144+0000","last_change":"2024-05-09T03:11:26.400568+0000","last_active":"2024-05-09T03:11:26.402144+0000","last_peered":"2024-05-09T03:11:26.402144+0000","last_clean":"2024-05-09T03:11:26.402144+0000","last_became_active":"2024-05-09T03:11:26.400253+0000","last_became_peered":"2024-05-09T03:11:26.400253+0000","last_unstale":"2024-05-09T03:11:26.402144+0000","last_undegraded":"2024-05-09T03:11:26.402144+0000","last_fullsized":"2024-05-09T03:11:26.402144+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:40.713887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389149+0000","last_change":"2024-05-09T03:11:26.389149+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389149+0000","last_undegraded":"2024-05-09T03:11:26.389149+0000","last_fullsized":"2024-05-09T03:11:26.389149+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405674+0000","last_change":"2024-05-09T03:11:26.397676+0000","last_active":"2024-05-09T03:11:26.405674+0000","last_peered":"2024-05-09T03:11:26.405674+0000","last_clean":"2024-05-09T03:11:26.405674+0000","last_became_active":"2024-05-09T03:11:26.396811+0000","last_became_peered":"2024-05-09T03:11:26.396811+0000","last_unstale":"2024-05-09T03:11:26.405674+0000","last_undegraded":"2024-05-09T03:11:26.405674+0000","last_fullsized":"2024-05-09T03:11:26.405674+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:30.350454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391430+0000","last_change":"2024-05-09T03:11:26.391430+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391430+0000","last_undegraded":"2024-05-09T03:11:26.391430+0000","last_fullsized":"2024-05-09T03:11:26.391430+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.403855+0000","last_change":"2024-05-09T03:11:26.402849+0000","last_active":"2024-05-09T03:11:26.403855+0000","last_peered":"2024-05-09T03:11:26.403855+0000","last_clean":"2024-05-09T03:11:26.403855+0000","last_became_active":"2024-05-09T03:11:26.402533+0000","last_became_peered":"2024-05-09T03:11:26.402533+0000","last_unstale":"2024-05-09T03:11:26.403855+0000","last_undegraded":"2024-05-09T03:11:26.403855+0000","last_fullsized":"2024-05-09T03:11:26.403855+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385911+0000","last_change":"2024-05-09T03:11:26.385911+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385911+0000","last_undegraded":"2024-05-09T03:11:26.385911+0000","last_fullsized":"2024-05-09T03:11:26.385911+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.398696+0000","last_change":"2024-05-09T03:11:26.396367+0000","last_active":"2024-05-09T03:11:26.398696+0000","last_peered":"2024-05-09T03:11:26.398696+0000","last_clean":"2024-05-09T03:11:26.398696+0000","last_became_active":"2024-05-09T03:11:26.396006+0000","last_became_peered":"2024-05-09T03:11:26.396006+0000","last_unstale":"2024-05-09T03:11:26.398696+0000","last_undegraded":"2024-05-09T03:11:26.398696+0000","last_fullsized":"2024-05-09T03:11:26.398696+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:42.938942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398327+0000","last_change":"2024-05-09T03:11:26.398327+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398327+0000","last_undegraded":"2024-05-09T03:11:26.398327+0000","last_fullsized":"2024-05-09T03:11:26.398327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408710+0000","last_change":"2024-05-09T03:11:26.407676+0000","last_active":"2024-05-09T03:11:26.408710+0000","last_peered":"2024-05-09T03:11:26.408710+0000","last_clean":"2024-05-09T03:11:26.408710+0000","last_became_active":"2024-05-09T03:11:26.407284+0000","last_became_peered":"2024-05-09T03:11:26.407284+0000","last_unstale":"2024-05-09T03:11:26.408710+0000","last_undegraded":"2024-05-09T03:11:26.408710+0000","last_fullsized":"2024-05-09T03:11:26.408710+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:40.713887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.390235+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:11:26.390235+0000","last_peered":"2024-05-09T03:11:26.390235+0000","last_clean":"2024-05-09T03:11:26.390235+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:11:26.390235+0000","last_undegraded":"2024-05-09T03:11:26.390235+0000","last_fullsized":"2024-05-09T03:11:26.390235+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":774839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":774839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:11:31.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:11:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.038+0000 7f3403ec8640 1 Processor -- start 2024-05-09T03:11:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.038+0000 7f3403ec8640 1 -- start start 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3403ec8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33fc164a40 0x7f33fc160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3403ec8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f33fc161380 0x7f33fc0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3403ec8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33fc071300 con 0x7f33fc070ef0 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3403ec8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f33fc163fa0 con 0x7f33fc164a40 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3403ec8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f33fc0cfcb0 con 0x7f33fc161380 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33fc164a40 0x7f33fc160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33fc164a40 0x7f33fc160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60216/0 (socket says 172.21.15.46:60216) 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 -- 172.21.15.46:0/561898715 learned_addr learned my addr 172.21.15.46:0/561898715 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:32.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401463640 1 --2- 172.21.15.46:0/561898715 >> v2:172.21.15.88:3300/0 conn(0x7f33fc161380 0x7f33fc0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 -- 172.21.15.46:0/561898715 >> v2:172.21.15.88:3300/0 conn(0x7f33fc161380 msgr2=0x7f33fc0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 --2- 172.21.15.46:0/561898715 >> v2:172.21.15.88:3300/0 conn(0x7f33fc161380 0x7f33fc0709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 -- 172.21.15.46:0/561898715 >> v1:172.21.15.88:6789/0 conn(0x7f33fc070ef0 legacy=0x7f33fc073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.040+0000 7f3401c64640 1 -- 172.21.15.46:0/561898715 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc0cff90 con 0x7f33fc164a40 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.041+0000 7f3401c64640 1 --2- 172.21.15.46:0/561898715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc164a40 0x7f33fc160340 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f33fc06e9c0 tx=0x7f33ec02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932d3767ae6e5ced server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.041+0000 7f3400c62640 1 -- 172.21.15.46:0/561898715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec036070 con 0x7f33fc164a40 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.041+0000 7f3400c62640 1 -- 172.21.15.46:0/561898715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33ec031600 con 0x7f33fc164a40 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.041+0000 7f3400c62640 1 -- 172.21.15.46:0/561898715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0319d0 con 0x7f33fc164a40 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 -- 172.21.15.46:0/561898715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc164a40 msgr2=0x7f33fc160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 --2- 172.21.15.46:0/561898715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc164a40 0x7f33fc160340 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f33fc06e9c0 tx=0x7f33ec02a6a0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 -- 172.21.15.46:0/561898715 shutdown_connections 2024-05-09T03:11:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 --2- 172.21.15.46:0/561898715 >> v2:172.21.15.88:3300/0 conn(0x7f33fc161380 0x7f33fc0709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 --2- 172.21.15.46:0/561898715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc164a40 0x7f33fc160340 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 -- 172.21.15.46:0/561898715 >> 172.21.15.46:0/561898715 conn(0x7f33fc0b3c40 msgr2=0x7f33fc0b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 -- 172.21.15.46:0/561898715 shutdown_connections 2024-05-09T03:11:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.042+0000 7f3403ec8640 1 -- 172.21.15.46:0/561898715 wait complete. 2024-05-09T03:11:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.044+0000 7f3403ec8640 1 Processor -- start 2024-05-09T03:11:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.044+0000 7f3403ec8640 1 -- start start 2024-05-09T03:11:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3403ec8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33fc070ef0 0x7f33fc1f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3403ec8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 0x7f33fc1fa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3403ec8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc164a40 0x7f33fc073730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3403ec8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc161790 con 0x7f33fc161380 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3403ec8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc164520 con 0x7f33fc164a40 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3403ec8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0d0270 con 0x7f33fc070ef0 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3401463640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 0x7f33fc1fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3401463640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 0x7f33fc1fa080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60220/0 (socket says 172.21.15.46:60220) 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.045+0000 7f3401463640 1 -- 172.21.15.46:0/1892141714 learned_addr learned my addr 172.21.15.46:0/1892141714 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3402465640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc164a40 0x7f33fc073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401c64640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33fc070ef0 0x7f33fc1f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401463640 1 -- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33fc070ef0 msgr2=0x7f33fc1f01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401463640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33fc070ef0 0x7f33fc1f01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401463640 1 -- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc164a40 msgr2=0x7f33fc073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401463640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc164a40 0x7f33fc073730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401463640 1 -- 172.21.15.46:0/1892141714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc149fc0 con 0x7f33fc161380 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3402465640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc164a40 0x7f33fc073730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401c64640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33fc070ef0 0x7f33fc1f01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3401463640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 0x7f33fc1fa080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f33f000e870 tx=0x7f33f000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f0017070 con 0x7f33fc161380 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f000c940 con 0x7f33fc161380 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f000cd30 con 0x7f33fc161380 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc149440 con 0x7f33fc161380 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.046+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc149a00 con 0x7f33fc161380 2024-05-09T03:11:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.047+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33fc06a270 con 0x7f33fc161380 2024-05-09T03:11:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.048+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f33f001a070 con 0x7f33fc161380 2024-05-09T03:11:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.048+0000 7f33fa7fc640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33e80762d0 0x7f33e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.049+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f33f0097520 con 0x7f33fc161380 2024-05-09T03:11:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.049+0000 7f3401c64640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33e80762d0 0x7f33e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.049+0000 7f3401c64640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33e80762d0 0x7f33e80787e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33ec033000 tx=0x7f33ec009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.053+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f33f0060df0 con 0x7f33fc161380 2024-05-09T03:11:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.238+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33fc074300 con 0x7f33e80762d0 2024-05-09T03:11:32.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.245+0000 7f33fa7fc640 1 -- 172.21.15.46:0/1892141714 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7f33fc074300 con 0x7f33e80762d0 2024-05-09T03:11:32.248 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:32.249 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33e80762d0 msgr2=0x7f33e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33e80762d0 0x7f33e80787e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f33ec033000 tx=0x7f33ec009000 comp rx=0 tx=0).stop 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 msgr2=0x7f33fc1fa080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 0x7f33fc1fa080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f33f000e870 tx=0x7f33f000ed30 comp rx=0 tx=0).stop 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 shutdown_connections 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33fc164a40 0x7f33fc073730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33e80762d0 0x7f33e80787e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33fc161380 0x7f33fc1fa080 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 --2- 172.21.15.46:0/1892141714 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33fc070ef0 0x7f33fc1f01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 >> 172.21.15.46:0/1892141714 conn(0x7f33fc0b3c40 msgr2=0x7f33fc0b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 shutdown_connections 2024-05-09T03:11:32.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.248+0000 7f3403ec8640 1 -- 172.21.15.46:0/1892141714 wait complete. 2024-05-09T03:11:32.260 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-05-09T03:11:30.520630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237672,"kb_used_data":4832,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737048,"statfs":{"total":773094113280,"available":772850737152,"internally_reserved":0,"allocated":4947968,"data_stored":2669038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.137100"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386053+0000","last_change":"2024-05-09T03:11:26.386053+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386053+0000","last_undegraded":"2024-05-09T03:11:26.386053+0000","last_fullsized":"2024-05-09T03:11:26.386053+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393641+0000","last_change":"2024-05-09T03:11:26.393641+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393641+0000","last_undegraded":"2024-05-09T03:11:26.393641+0000","last_fullsized":"2024-05-09T03:11:26.393641+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386800+0000","last_change":"2024-05-09T03:11:26.386800+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386800+0000","last_undegraded":"2024-05-09T03:11:26.386800+0000","last_fullsized":"2024-05-09T03:11:26.386800+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399458+0000","last_change":"2024-05-09T03:11:26.399458+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399458+0000","last_undegraded":"2024-05-09T03:11:26.399458+0000","last_fullsized":"2024-05-09T03:11:26.399458+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393056+0000","last_change":"2024-05-09T03:11:26.393056+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393056+0000","last_undegraded":"2024-05-09T03:11:26.393056+0000","last_fullsized":"2024-05-09T03:11:26.393056+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393193+0000","last_change":"2024-05-09T03:11:26.393193+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393193+0000","last_undegraded":"2024-05-09T03:11:26.393193+0000","last_fullsized":"2024-05-09T03:11:26.393193+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.387803+0000","last_change":"2024-05-09T03:11:26.387803+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.387803+0000","last_undegraded":"2024-05-09T03:11:26.387803+0000","last_fullsized":"2024-05-09T03:11:26.387803+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392140+0000","last_change":"2024-05-09T03:11:26.392140+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392140+0000","last_undegraded":"2024-05-09T03:11:26.392140+0000","last_fullsized":"2024-05-09T03:11:26.392140+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399153+0000","last_change":"2024-05-09T03:11:26.399153+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399153+0000","last_undegraded":"2024-05-09T03:11:26.399153+0000","last_fullsized":"2024-05-09T03:11:26.399153+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399862+0000","last_change":"2024-05-09T03:11:26.399862+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399862+0000","last_undegraded":"2024-05-09T03:11:26.399862+0000","last_fullsized":"2024-05-09T03:11:26.399862+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398110+0000","last_change":"2024-05-09T03:11:26.398110+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398110+0000","last_undegraded":"2024-05-09T03:11:26.398110+0000","last_fullsized":"2024-05-09T03:11:26.398110+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393757+0000","last_change":"2024-05-09T03:11:26.393757+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393757+0000","last_undegraded":"2024-05-09T03:11:26.393757+0000","last_fullsized":"2024-05-09T03:11:26.393757+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389234+0000","last_change":"2024-05-09T03:11:26.389234+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389234+0000","last_undegraded":"2024-05-09T03:11:26.389234+0000","last_fullsized":"2024-05-09T03:11:26.389234+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391877+0000","last_change":"2024-05-09T03:11:26.391877+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391877+0000","last_undegraded":"2024-05-09T03:11:26.391877+0000","last_fullsized":"2024-05-09T03:11:26.391877+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393846+0000","last_change":"2024-05-09T03:11:26.393846+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393846+0000","last_undegraded":"2024-05-09T03:11:26.393846+0000","last_fullsized":"2024-05-09T03:11:26.393846+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392475+0000","last_change":"2024-05-09T03:11:26.392475+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392475+0000","last_undegraded":"2024-05-09T03:11:26.392475+0000","last_fullsized":"2024-05-09T03:11:26.392475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388512+0000","last_change":"2024-05-09T03:11:26.388512+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388512+0000","last_undegraded":"2024-05-09T03:11:26.388512+0000","last_fullsized":"2024-05-09T03:11:26.388512+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388619+0000","last_change":"2024-05-09T03:11:26.388619+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388619+0000","last_undegraded":"2024-05-09T03:11:26.388619+0000","last_fullsized":"2024-05-09T03:11:26.388619+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388811+0000","last_change":"2024-05-09T03:11:26.388811+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388811+0000","last_undegraded":"2024-05-09T03:11:26.388811+0000","last_fullsized":"2024-05-09T03:11:26.388811+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391447+0000","last_change":"2024-05-09T03:11:26.391447+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391447+0000","last_undegraded":"2024-05-09T03:11:26.391447+0000","last_fullsized":"2024-05-09T03:11:26.391447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393040+0000","last_change":"2024-05-09T03:11:26.393040+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393040+0000","last_undegraded":"2024-05-09T03:11:26.393040+0000","last_fullsized":"2024-05-09T03:11:26.393040+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391316+0000","last_change":"2024-05-09T03:11:26.391316+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391316+0000","last_undegraded":"2024-05-09T03:11:26.391316+0000","last_fullsized":"2024-05-09T03:11:26.391316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389605+0000","last_change":"2024-05-09T03:11:26.389605+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389605+0000","last_undegraded":"2024-05-09T03:11:26.389605+0000","last_fullsized":"2024-05-09T03:11:26.389605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391465+0000","last_change":"2024-05-09T03:11:26.391465+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391465+0000","last_undegraded":"2024-05-09T03:11:26.391465+0000","last_fullsized":"2024-05-09T03:11:26.391465+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390564+0000","last_change":"2024-05-09T03:11:26.390564+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390564+0000","last_undegraded":"2024-05-09T03:11:26.390564+0000","last_fullsized":"2024-05-09T03:11:26.390564+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390514+0000","last_change":"2024-05-09T03:11:26.390514+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390514+0000","last_undegraded":"2024-05-09T03:11:26.390514+0000","last_fullsized":"2024-05-09T03:11:26.390514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385973+0000","last_change":"2024-05-09T03:11:26.385973+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385973+0000","last_undegraded":"2024-05-09T03:11:26.385973+0000","last_fullsized":"2024-05-09T03:11:26.385973+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392423+0000","last_change":"2024-05-09T03:11:26.392423+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392423+0000","last_undegraded":"2024-05-09T03:11:26.392423+0000","last_fullsized":"2024-05-09T03:11:26.392423+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391928+0000","last_change":"2024-05-09T03:11:26.391928+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391928+0000","last_undegraded":"2024-05-09T03:11:26.391928+0000","last_fullsized":"2024-05-09T03:11:26.391928+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.388354+0000","last_change":"2024-05-09T03:11:26.388354+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.388354+0000","last_undegraded":"2024-05-09T03:11:26.388354+0000","last_fullsized":"2024-05-09T03:11:26.388354+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393240+0000","last_change":"2024-05-09T03:11:26.393240+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393240+0000","last_undegraded":"2024-05-09T03:11:26.393240+0000","last_fullsized":"2024-05-09T03:11:26.393240+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391939+0000","last_change":"2024-05-09T03:11:26.391939+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391939+0000","last_undegraded":"2024-05-09T03:11:26.391939+0000","last_fullsized":"2024-05-09T03:11:26.391939+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389988+0000","last_change":"2024-05-09T03:11:26.389988+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389988+0000","last_undegraded":"2024-05-09T03:11:26.389988+0000","last_fullsized":"2024-05-09T03:11:26.389988+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406501+0000","last_change":"2024-05-09T03:11:26.395704+0000","last_active":"2024-05-09T03:11:26.406501+0000","last_peered":"2024-05-09T03:11:26.406501+0000","last_clean":"2024-05-09T03:11:26.406501+0000","last_became_active":"2024-05-09T03:11:26.394437+0000","last_became_peered":"2024-05-09T03:11:26.394437+0000","last_unstale":"2024-05-09T03:11:26.406501+0000","last_undegraded":"2024-05-09T03:11:26.406501+0000","last_fullsized":"2024-05-09T03:11:26.406501+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391552+0000","last_change":"2024-05-09T03:11:26.391552+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391552+0000","last_undegraded":"2024-05-09T03:11:26.391552+0000","last_fullsized":"2024-05-09T03:11:26.391552+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402507+0000","last_change":"2024-05-09T03:11:26.395710+0000","last_active":"2024-05-09T03:11:26.402507+0000","last_peered":"2024-05-09T03:11:26.402507+0000","last_clean":"2024-05-09T03:11:26.402507+0000","last_became_active":"2024-05-09T03:11:26.395376+0000","last_became_peered":"2024-05-09T03:11:26.395376+0000","last_unstale":"2024-05-09T03:11:26.402507+0000","last_undegraded":"2024-05-09T03:11:26.402507+0000","last_fullsized":"2024-05-09T03:11:26.402507+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:15.450135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391936+0000","last_change":"2024-05-09T03:11:26.391936+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391936+0000","last_undegraded":"2024-05-09T03:11:26.391936+0000","last_fullsized":"2024-05-09T03:11:26.391936+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401360+0000","last_change":"2024-05-09T03:11:26.400247+0000","last_active":"2024-05-09T03:11:26.401360+0000","last_peered":"2024-05-09T03:11:26.401360+0000","last_clean":"2024-05-09T03:11:26.401360+0000","last_became_active":"2024-05-09T03:11:26.399921+0000","last_became_peered":"2024-05-09T03:11:26.399921+0000","last_unstale":"2024-05-09T03:11:26.401360+0000","last_undegraded":"2024-05-09T03:11:26.401360+0000","last_fullsized":"2024-05-09T03:11:26.401360+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390640+0000","last_change":"2024-05-09T03:11:26.390640+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390640+0000","last_undegraded":"2024-05-09T03:11:26.390640+0000","last_fullsized":"2024-05-09T03:11:26.390640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402127+0000","last_change":"2024-05-09T03:11:26.400502+0000","last_active":"2024-05-09T03:11:26.402127+0000","last_peered":"2024-05-09T03:11:26.402127+0000","last_clean":"2024-05-09T03:11:26.402127+0000","last_became_active":"2024-05-09T03:11:26.400182+0000","last_became_peered":"2024-05-09T03:11:26.400182+0000","last_unstale":"2024-05-09T03:11:26.402127+0000","last_undegraded":"2024-05-09T03:11:26.402127+0000","last_fullsized":"2024-05-09T03:11:26.402127+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:49.354980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386654+0000","last_change":"2024-05-09T03:11:26.386654+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386654+0000","last_undegraded":"2024-05-09T03:11:26.386654+0000","last_fullsized":"2024-05-09T03:11:26.386654+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406565+0000","last_change":"2024-05-09T03:11:26.397298+0000","last_active":"2024-05-09T03:11:26.406565+0000","last_peered":"2024-05-09T03:11:26.406565+0000","last_clean":"2024-05-09T03:11:26.406565+0000","last_became_active":"2024-05-09T03:11:26.397003+0000","last_became_peered":"2024-05-09T03:11:26.397003+0000","last_unstale":"2024-05-09T03:11:26.406565+0000","last_undegraded":"2024-05-09T03:11:26.406565+0000","last_fullsized":"2024-05-09T03:11:26.406565+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.400278+0000","last_change":"2024-05-09T03:11:26.400278+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.400278+0000","last_undegraded":"2024-05-09T03:11:26.400278+0000","last_fullsized":"2024-05-09T03:11:26.400278+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405578+0000","last_change":"2024-05-09T03:11:26.397547+0000","last_active":"2024-05-09T03:11:26.405578+0000","last_peered":"2024-05-09T03:11:26.405578+0000","last_clean":"2024-05-09T03:11:26.405578+0000","last_became_active":"2024-05-09T03:11:26.397248+0000","last_became_peered":"2024-05-09T03:11:26.397248+0000","last_unstale":"2024-05-09T03:11:26.405578+0000","last_undegraded":"2024-05-09T03:11:26.405578+0000","last_fullsized":"2024-05-09T03:11:26.405578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:10:57.344324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391644+0000","last_change":"2024-05-09T03:11:26.391644+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391644+0000","last_undegraded":"2024-05-09T03:11:26.391644+0000","last_fullsized":"2024-05-09T03:11:26.391644+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406549+0000","last_change":"2024-05-09T03:11:26.402930+0000","last_active":"2024-05-09T03:11:26.406549+0000","last_peered":"2024-05-09T03:11:26.406549+0000","last_clean":"2024-05-09T03:11:26.406549+0000","last_became_active":"2024-05-09T03:11:26.402595+0000","last_became_peered":"2024-05-09T03:11:26.402595+0000","last_unstale":"2024-05-09T03:11:26.406549+0000","last_undegraded":"2024-05-09T03:11:26.406549+0000","last_fullsized":"2024-05-09T03:11:26.406549+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:12.461211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393099+0000","last_change":"2024-05-09T03:11:26.393099+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393099+0000","last_undegraded":"2024-05-09T03:11:26.393099+0000","last_fullsized":"2024-05-09T03:11:26.393099+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407684+0000","last_change":"2024-05-09T03:11:26.405390+0000","last_active":"2024-05-09T03:11:26.407684+0000","last_peered":"2024-05-09T03:11:26.407684+0000","last_clean":"2024-05-09T03:11:26.407684+0000","last_became_active":"2024-05-09T03:11:26.404946+0000","last_became_peered":"2024-05-09T03:11:26.404946+0000","last_unstale":"2024-05-09T03:11:26.407684+0000","last_undegraded":"2024-05-09T03:11:26.407684+0000","last_fullsized":"2024-05-09T03:11:26.407684+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:25:32.094306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.387537+0000","last_change":"2024-05-09T03:11:26.387537+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.387537+0000","last_undegraded":"2024-05-09T03:11:26.387537+0000","last_fullsized":"2024-05-09T03:11:26.387537+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405664+0000","last_change":"2024-05-09T03:11:26.398230+0000","last_active":"2024-05-09T03:11:26.405664+0000","last_peered":"2024-05-09T03:11:26.405664+0000","last_clean":"2024-05-09T03:11:26.405664+0000","last_became_active":"2024-05-09T03:11:26.397938+0000","last_became_peered":"2024-05-09T03:11:26.397938+0000","last_unstale":"2024-05-09T03:11:26.405664+0000","last_undegraded":"2024-05-09T03:11:26.405664+0000","last_fullsized":"2024-05-09T03:11:26.405664+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385422+0000","last_change":"2024-05-09T03:11:26.385422+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385422+0000","last_undegraded":"2024-05-09T03:11:26.385422+0000","last_fullsized":"2024-05-09T03:11:26.385422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408635+0000","last_change":"2024-05-09T03:11:26.407940+0000","last_active":"2024-05-09T03:11:26.408635+0000","last_peered":"2024-05-09T03:11:26.408635+0000","last_clean":"2024-05-09T03:11:26.408635+0000","last_became_active":"2024-05-09T03:11:26.407561+0000","last_became_peered":"2024-05-09T03:11:26.407561+0000","last_unstale":"2024-05-09T03:11:26.408635+0000","last_undegraded":"2024-05-09T03:11:26.408635+0000","last_fullsized":"2024-05-09T03:11:26.408635+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:15.450135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393376+0000","last_change":"2024-05-09T03:11:26.393376+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393376+0000","last_undegraded":"2024-05-09T03:11:26.393376+0000","last_fullsized":"2024-05-09T03:11:26.393376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402424+0000","last_change":"2024-05-09T03:11:26.401361+0000","last_active":"2024-05-09T03:11:26.402424+0000","last_peered":"2024-05-09T03:11:26.402424+0000","last_clean":"2024-05-09T03:11:26.402424+0000","last_became_active":"2024-05-09T03:11:26.401038+0000","last_became_peered":"2024-05-09T03:11:26.401038+0000","last_unstale":"2024-05-09T03:11:26.402424+0000","last_undegraded":"2024-05-09T03:11:26.402424+0000","last_fullsized":"2024-05-09T03:11:26.402424+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:28.972509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.386048+0000","last_change":"2024-05-09T03:11:26.386048+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.386048+0000","last_undegraded":"2024-05-09T03:11:26.386048+0000","last_fullsized":"2024-05-09T03:11:26.386048+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407226+0000","last_change":"2024-05-09T03:11:26.406530+0000","last_active":"2024-05-09T03:11:26.407226+0000","last_peered":"2024-05-09T03:11:26.407226+0000","last_clean":"2024-05-09T03:11:26.407226+0000","last_became_active":"2024-05-09T03:11:26.406173+0000","last_became_peered":"2024-05-09T03:11:26.406173+0000","last_unstale":"2024-05-09T03:11:26.407226+0000","last_undegraded":"2024-05-09T03:11:26.407226+0000","last_fullsized":"2024-05-09T03:11:26.407226+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:03:28.087361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391660+0000","last_change":"2024-05-09T03:11:26.391660+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391660+0000","last_undegraded":"2024-05-09T03:11:26.391660+0000","last_fullsized":"2024-05-09T03:11:26.391660+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407154+0000","last_change":"2024-05-09T03:11:26.402684+0000","last_active":"2024-05-09T03:11:26.407154+0000","last_peered":"2024-05-09T03:11:26.407154+0000","last_clean":"2024-05-09T03:11:26.407154+0000","last_became_active":"2024-05-09T03:11:26.402325+0000","last_became_peered":"2024-05-09T03:11:26.402325+0000","last_unstale":"2024-05-09T03:11:26.407154+0000","last_undegraded":"2024-05-09T03:11:26.407154+0000","last_fullsized":"2024-05-09T03:11:26.407154+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:24.688956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.399627+0000","last_change":"2024-05-09T03:11:26.399627+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.399627+0000","last_undegraded":"2024-05-09T03:11:26.399627+0000","last_fullsized":"2024-05-09T03:11:26.399627+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401609+0000","last_change":"2024-05-09T03:11:26.399378+0000","last_active":"2024-05-09T03:11:26.401609+0000","last_peered":"2024-05-09T03:11:26.401609+0000","last_clean":"2024-05-09T03:11:26.401609+0000","last_became_active":"2024-05-09T03:11:26.399091+0000","last_became_peered":"2024-05-09T03:11:26.399091+0000","last_unstale":"2024-05-09T03:11:26.401609+0000","last_undegraded":"2024-05-09T03:11:26.401609+0000","last_fullsized":"2024-05-09T03:11:26.401609+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:38.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.393393+0000","last_change":"2024-05-09T03:11:26.393393+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.393393+0000","last_undegraded":"2024-05-09T03:11:26.393393+0000","last_fullsized":"2024-05-09T03:11:26.393393+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406631+0000","last_change":"2024-05-09T03:11:26.398576+0000","last_active":"2024-05-09T03:11:26.406631+0000","last_peered":"2024-05-09T03:11:26.406631+0000","last_clean":"2024-05-09T03:11:26.406631+0000","last_became_active":"2024-05-09T03:11:26.398273+0000","last_became_peered":"2024-05-09T03:11:26.398273+0000","last_unstale":"2024-05-09T03:11:26.406631+0000","last_undegraded":"2024-05-09T03:11:26.406631+0000","last_fullsized":"2024-05-09T03:11:26.406631+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385398+0000","last_change":"2024-05-09T03:11:26.385398+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385398+0000","last_undegraded":"2024-05-09T03:11:26.385398+0000","last_fullsized":"2024-05-09T03:11:26.385398+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405242+0000","last_change":"2024-05-09T03:11:26.403484+0000","last_active":"2024-05-09T03:11:26.405242+0000","last_peered":"2024-05-09T03:11:26.405242+0000","last_clean":"2024-05-09T03:11:26.405242+0000","last_became_active":"2024-05-09T03:11:26.403147+0000","last_became_peered":"2024-05-09T03:11:26.403147+0000","last_unstale":"2024-05-09T03:11:26.405242+0000","last_undegraded":"2024-05-09T03:11:26.405242+0000","last_fullsized":"2024-05-09T03:11:26.405242+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:48.730535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390014+0000","last_change":"2024-05-09T03:11:26.390014+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390014+0000","last_undegraded":"2024-05-09T03:11:26.390014+0000","last_fullsized":"2024-05-09T03:11:26.390014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.409491+0000","last_change":"2024-05-09T03:11:26.408975+0000","last_active":"2024-05-09T03:11:26.409491+0000","last_peered":"2024-05-09T03:11:26.409491+0000","last_clean":"2024-05-09T03:11:26.409491+0000","last_became_active":"2024-05-09T03:11:26.408602+0000","last_became_peered":"2024-05-09T03:11:26.408602+0000","last_unstale":"2024-05-09T03:11:26.409491+0000","last_undegraded":"2024-05-09T03:11:26.409491+0000","last_fullsized":"2024-05-09T03:11:26.409491+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:21:41.886217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398258+0000","last_change":"2024-05-09T03:11:26.398258+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398258+0000","last_undegraded":"2024-05-09T03:11:26.398258+0000","last_fullsized":"2024-05-09T03:11:26.398258+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405558+0000","last_change":"2024-05-09T03:11:26.399038+0000","last_active":"2024-05-09T03:11:26.405558+0000","last_peered":"2024-05-09T03:11:26.405558+0000","last_clean":"2024-05-09T03:11:26.405558+0000","last_became_active":"2024-05-09T03:11:26.398728+0000","last_became_peered":"2024-05-09T03:11:26.398728+0000","last_unstale":"2024-05-09T03:11:26.405558+0000","last_undegraded":"2024-05-09T03:11:26.405558+0000","last_fullsized":"2024-05-09T03:11:26.405558+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391987+0000","last_change":"2024-05-09T03:11:26.391987+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391987+0000","last_undegraded":"2024-05-09T03:11:26.391987+0000","last_fullsized":"2024-05-09T03:11:26.391987+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405551+0000","last_change":"2024-05-09T03:11:26.402375+0000","last_active":"2024-05-09T03:11:26.405551+0000","last_peered":"2024-05-09T03:11:26.405551+0000","last_clean":"2024-05-09T03:11:26.405551+0000","last_became_active":"2024-05-09T03:11:26.402033+0000","last_became_peered":"2024-05-09T03:11:26.402033+0000","last_unstale":"2024-05-09T03:11:26.405551+0000","last_undegraded":"2024-05-09T03:11:26.405551+0000","last_fullsized":"2024-05-09T03:11:26.405551+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:38.001424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389422+0000","last_change":"2024-05-09T03:11:26.389422+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389422+0000","last_undegraded":"2024-05-09T03:11:26.389422+0000","last_fullsized":"2024-05-09T03:11:26.389422+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407153+0000","last_change":"2024-05-09T03:11:26.401461+0000","last_active":"2024-05-09T03:11:26.407153+0000","last_peered":"2024-05-09T03:11:26.407153+0000","last_clean":"2024-05-09T03:11:26.407153+0000","last_became_active":"2024-05-09T03:11:26.401148+0000","last_became_peered":"2024-05-09T03:11:26.401148+0000","last_unstale":"2024-05-09T03:11:26.407153+0000","last_undegraded":"2024-05-09T03:11:26.407153+0000","last_fullsized":"2024-05-09T03:11:26.407153+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:08.673574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392466+0000","last_change":"2024-05-09T03:11:26.392466+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392466+0000","last_undegraded":"2024-05-09T03:11:26.392466+0000","last_fullsized":"2024-05-09T03:11:26.392466+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.407109+0000","last_change":"2024-05-09T03:11:26.400381+0000","last_active":"2024-05-09T03:11:26.407109+0000","last_peered":"2024-05-09T03:11:26.407109+0000","last_clean":"2024-05-09T03:11:26.407109+0000","last_became_active":"2024-05-09T03:11:26.400059+0000","last_became_peered":"2024-05-09T03:11:26.400059+0000","last_unstale":"2024-05-09T03:11:26.407109+0000","last_undegraded":"2024-05-09T03:11:26.407109+0000","last_fullsized":"2024-05-09T03:11:26.407109+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:11.134274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392034+0000","last_change":"2024-05-09T03:11:26.392034+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392034+0000","last_undegraded":"2024-05-09T03:11:26.392034+0000","last_fullsized":"2024-05-09T03:11:26.392034+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406578+0000","last_change":"2024-05-09T03:11:26.403917+0000","last_active":"2024-05-09T03:11:26.406578+0000","last_peered":"2024-05-09T03:11:26.406578+0000","last_clean":"2024-05-09T03:11:26.406578+0000","last_became_active":"2024-05-09T03:11:26.403487+0000","last_became_peered":"2024-05-09T03:11:26.403487+0000","last_unstale":"2024-05-09T03:11:26.406578+0000","last_undegraded":"2024-05-09T03:11:26.406578+0000","last_fullsized":"2024-05-09T03:11:26.406578+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:49.354980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392477+0000","last_change":"2024-05-09T03:11:26.392477+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392477+0000","last_undegraded":"2024-05-09T03:11:26.392477+0000","last_fullsized":"2024-05-09T03:11:26.392477+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408299+0000","last_change":"2024-05-09T03:11:26.406762+0000","last_active":"2024-05-09T03:11:26.408299+0000","last_peered":"2024-05-09T03:11:26.408299+0000","last_clean":"2024-05-09T03:11:26.408299+0000","last_became_active":"2024-05-09T03:11:26.406432+0000","last_became_peered":"2024-05-09T03:11:26.406432+0000","last_unstale":"2024-05-09T03:11:26.408299+0000","last_undegraded":"2024-05-09T03:11:26.408299+0000","last_fullsized":"2024-05-09T03:11:26.408299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:49.151611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.390722+0000","last_change":"2024-05-09T03:11:26.390722+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.390722+0000","last_undegraded":"2024-05-09T03:11:26.390722+0000","last_fullsized":"2024-05-09T03:11:26.390722+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401829+0000","last_change":"2024-05-09T03:11:26.399861+0000","last_active":"2024-05-09T03:11:26.401829+0000","last_peered":"2024-05-09T03:11:26.401829+0000","last_clean":"2024-05-09T03:11:26.401829+0000","last_became_active":"2024-05-09T03:11:26.399383+0000","last_became_peered":"2024-05-09T03:11:26.399383+0000","last_unstale":"2024-05-09T03:11:26.401829+0000","last_undegraded":"2024-05-09T03:11:26.401829+0000","last_fullsized":"2024-05-09T03:11:26.401829+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:12.461211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391073+0000","last_change":"2024-05-09T03:11:26.391073+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391073+0000","last_undegraded":"2024-05-09T03:11:26.391073+0000","last_fullsized":"2024-05-09T03:11:26.391073+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.406669+0000","last_change":"2024-05-09T03:11:26.398315+0000","last_active":"2024-05-09T03:11:26.406669+0000","last_peered":"2024-05-09T03:11:26.406669+0000","last_clean":"2024-05-09T03:11:26.406669+0000","last_became_active":"2024-05-09T03:11:26.397997+0000","last_became_peered":"2024-05-09T03:11:26.397997+0000","last_unstale":"2024-05-09T03:11:26.406669+0000","last_undegraded":"2024-05-09T03:11:26.406669+0000","last_fullsized":"2024-05-09T03:11:26.406669+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.392309+0000","last_change":"2024-05-09T03:11:26.392309+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.392309+0000","last_undegraded":"2024-05-09T03:11:26.392309+0000","last_fullsized":"2024-05-09T03:11:26.392309+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402447+0000","last_change":"2024-05-09T03:11:26.401410+0000","last_active":"2024-05-09T03:11:26.402447+0000","last_peered":"2024-05-09T03:11:26.402447+0000","last_clean":"2024-05-09T03:11:26.402447+0000","last_became_active":"2024-05-09T03:11:26.401111+0000","last_became_peered":"2024-05-09T03:11:26.401111+0000","last_unstale":"2024-05-09T03:11:26.402447+0000","last_undegraded":"2024-05-09T03:11:26.402447+0000","last_fullsized":"2024-05-09T03:11:26.402447+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385409+0000","last_change":"2024-05-09T03:11:26.385409+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385409+0000","last_undegraded":"2024-05-09T03:11:26.385409+0000","last_fullsized":"2024-05-09T03:11:26.385409+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.401845+0000","last_change":"2024-05-09T03:11:26.401170+0000","last_active":"2024-05-09T03:11:26.401845+0000","last_peered":"2024-05-09T03:11:26.401845+0000","last_clean":"2024-05-09T03:11:26.401845+0000","last_became_active":"2024-05-09T03:11:26.400832+0000","last_became_peered":"2024-05-09T03:11:26.400832+0000","last_unstale":"2024-05-09T03:11:26.401845+0000","last_undegraded":"2024-05-09T03:11:26.401845+0000","last_fullsized":"2024-05-09T03:11:26.401845+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389437+0000","last_change":"2024-05-09T03:11:26.389437+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389437+0000","last_undegraded":"2024-05-09T03:11:26.389437+0000","last_fullsized":"2024-05-09T03:11:26.389437+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.402144+0000","last_change":"2024-05-09T03:11:26.400568+0000","last_active":"2024-05-09T03:11:26.402144+0000","last_peered":"2024-05-09T03:11:26.402144+0000","last_clean":"2024-05-09T03:11:26.402144+0000","last_became_active":"2024-05-09T03:11:26.400253+0000","last_became_peered":"2024-05-09T03:11:26.400253+0000","last_unstale":"2024-05-09T03:11:26.402144+0000","last_undegraded":"2024-05-09T03:11:26.402144+0000","last_fullsized":"2024-05-09T03:11:26.402144+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:40.713887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.389149+0000","last_change":"2024-05-09T03:11:26.389149+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.389149+0000","last_undegraded":"2024-05-09T03:11:26.389149+0000","last_fullsized":"2024-05-09T03:11:26.389149+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.405674+0000","last_change":"2024-05-09T03:11:26.397676+0000","last_active":"2024-05-09T03:11:26.405674+0000","last_peered":"2024-05-09T03:11:26.405674+0000","last_clean":"2024-05-09T03:11:26.405674+0000","last_became_active":"2024-05-09T03:11:26.396811+0000","last_became_peered":"2024-05-09T03:11:26.396811+0000","last_unstale":"2024-05-09T03:11:26.405674+0000","last_undegraded":"2024-05-09T03:11:26.405674+0000","last_fullsized":"2024-05-09T03:11:26.405674+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:30.350454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.391430+0000","last_change":"2024-05-09T03:11:26.391430+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.391430+0000","last_undegraded":"2024-05-09T03:11:26.391430+0000","last_fullsized":"2024-05-09T03:11:26.391430+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.403855+0000","last_change":"2024-05-09T03:11:26.402849+0000","last_active":"2024-05-09T03:11:26.403855+0000","last_peered":"2024-05-09T03:11:26.403855+0000","last_clean":"2024-05-09T03:11:26.403855+0000","last_became_active":"2024-05-09T03:11:26.402533+0000","last_became_peered":"2024-05-09T03:11:26.402533+0000","last_unstale":"2024-05-09T03:11:26.403855+0000","last_undegraded":"2024-05-09T03:11:26.403855+0000","last_fullsized":"2024-05-09T03:11:26.403855+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.385911+0000","last_change":"2024-05-09T03:11:26.385911+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.385911+0000","last_undegraded":"2024-05-09T03:11:26.385911+0000","last_fullsized":"2024-05-09T03:11:26.385911+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.398696+0000","last_change":"2024-05-09T03:11:26.396367+0000","last_active":"2024-05-09T03:11:26.398696+0000","last_peered":"2024-05-09T03:11:26.398696+0000","last_clean":"2024-05-09T03:11:26.398696+0000","last_became_active":"2024-05-09T03:11:26.396006+0000","last_became_peered":"2024-05-09T03:11:26.396006+0000","last_unstale":"2024-05-09T03:11:26.398696+0000","last_undegraded":"2024-05-09T03:11:26.398696+0000","last_fullsized":"2024-05-09T03:11:26.398696+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:42.938942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":41,"state":"creating+peering","last_fresh":"2024-05-09T03:11:26.398327+0000","last_change":"2024-05-09T03:11:26.398327+0000","last_active":"2024-05-09T03:11:26.379689+0000","last_peered":"2024-05-09T03:11:26.379689+0000","last_clean":"2024-05-09T03:11:26.379689+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:11:26.398327+0000","last_undegraded":"2024-05-09T03:11:26.398327+0000","last_fullsized":"2024-05-09T03:11:26.398327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.408710+0000","last_change":"2024-05-09T03:11:26.407676+0000","last_active":"2024-05-09T03:11:26.408710+0000","last_peered":"2024-05-09T03:11:26.408710+0000","last_clean":"2024-05-09T03:11:26.408710+0000","last_became_active":"2024-05-09T03:11:26.407284+0000","last_became_peered":"2024-05-09T03:11:26.407284+0000","last_unstale":"2024-05-09T03:11:26.408710+0000","last_undegraded":"2024-05-09T03:11:26.408710+0000","last_fullsized":"2024-05-09T03:11:26.408710+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:40.713887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":147,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-05-09T03:11:26.390235+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:11:26.390235+0000","last_peered":"2024-05-09T03:11:26.390235+0000","last_clean":"2024-05-09T03:11:26.390235+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:11:26.390235+0000","last_undegraded":"2024-05-09T03:11:26.390235+0000","last_fullsized":"2024-05-09T03:11:26.390235+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1060864,"data_stored":774839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31648,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340192,"statfs":{"total":96636764160,"available":96604356608,"internally_reserved":0,"allocated":479232,"data_stored":190738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":466944,"data_stored":184471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":774839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3f is not active+clean 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.386053+0000', 'last_change': '2024-05-09T03:11:26.386053+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.386053+0000', 'last_undegraded': '2024-05-09T03:11:26.386053+0000', 'last_fullsized': '2024-05-09T03:11:26.386053+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3e is not active+clean 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393641+0000', 'last_change': '2024-05-09T03:11:26.393641+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393641+0000', 'last_undegraded': '2024-05-09T03:11:26.393641+0000', 'last_fullsized': '2024-05-09T03:11:26.393641+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3d is not active+clean 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.386800+0000', 'last_change': '2024-05-09T03:11:26.386800+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.386800+0000', 'last_undegraded': '2024-05-09T03:11:26.386800+0000', 'last_fullsized': '2024-05-09T03:11:26.386800+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.264 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3c is not active+clean 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.399458+0000', 'last_change': '2024-05-09T03:11:26.399458+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.399458+0000', 'last_undegraded': '2024-05-09T03:11:26.399458+0000', 'last_fullsized': '2024-05-09T03:11:26.399458+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3b is not active+clean 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393056+0000', 'last_change': '2024-05-09T03:11:26.393056+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393056+0000', 'last_undegraded': '2024-05-09T03:11:26.393056+0000', 'last_fullsized': '2024-05-09T03:11:26.393056+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3a is not active+clean 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393193+0000', 'last_change': '2024-05-09T03:11:26.393193+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393193+0000', 'last_undegraded': '2024-05-09T03:11:26.393193+0000', 'last_fullsized': '2024-05-09T03:11:26.393193+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:PG 7.39 is not active+clean 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.387803+0000', 'last_change': '2024-05-09T03:11:26.387803+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.387803+0000', 'last_undegraded': '2024-05-09T03:11:26.387803+0000', 'last_fullsized': '2024-05-09T03:11:26.387803+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:PG 7.38 is not active+clean 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392140+0000', 'last_change': '2024-05-09T03:11:26.392140+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392140+0000', 'last_undegraded': '2024-05-09T03:11:26.392140+0000', 'last_fullsized': '2024-05-09T03:11:26.392140+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:PG 7.37 is not active+clean 2024-05-09T03:11:32.265 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.399153+0000', 'last_change': '2024-05-09T03:11:26.399153+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.399153+0000', 'last_undegraded': '2024-05-09T03:11:26.399153+0000', 'last_fullsized': '2024-05-09T03:11:26.399153+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:PG 7.36 is not active+clean 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.399862+0000', 'last_change': '2024-05-09T03:11:26.399862+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.399862+0000', 'last_undegraded': '2024-05-09T03:11:26.399862+0000', 'last_fullsized': '2024-05-09T03:11:26.399862+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:PG 7.35 is not active+clean 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.398110+0000', 'last_change': '2024-05-09T03:11:26.398110+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.398110+0000', 'last_undegraded': '2024-05-09T03:11:26.398110+0000', 'last_fullsized': '2024-05-09T03:11:26.398110+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:PG 7.34 is not active+clean 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393757+0000', 'last_change': '2024-05-09T03:11:26.393757+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393757+0000', 'last_undegraded': '2024-05-09T03:11:26.393757+0000', 'last_fullsized': '2024-05-09T03:11:26.393757+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:PG 7.33 is not active+clean 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.389234+0000', 'last_change': '2024-05-09T03:11:26.389234+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.389234+0000', 'last_undegraded': '2024-05-09T03:11:26.389234+0000', 'last_fullsized': '2024-05-09T03:11:26.389234+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:PG 7.32 is not active+clean 2024-05-09T03:11:32.266 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391877+0000', 'last_change': '2024-05-09T03:11:26.391877+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391877+0000', 'last_undegraded': '2024-05-09T03:11:26.391877+0000', 'last_fullsized': '2024-05-09T03:11:26.391877+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:PG 7.31 is not active+clean 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393846+0000', 'last_change': '2024-05-09T03:11:26.393846+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393846+0000', 'last_undegraded': '2024-05-09T03:11:26.393846+0000', 'last_fullsized': '2024-05-09T03:11:26.393846+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:PG 7.30 is not active+clean 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392475+0000', 'last_change': '2024-05-09T03:11:26.392475+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392475+0000', 'last_undegraded': '2024-05-09T03:11:26.392475+0000', 'last_fullsized': '2024-05-09T03:11:26.392475+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2f is not active+clean 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.388512+0000', 'last_change': '2024-05-09T03:11:26.388512+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.388512+0000', 'last_undegraded': '2024-05-09T03:11:26.388512+0000', 'last_fullsized': '2024-05-09T03:11:26.388512+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2e is not active+clean 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.388619+0000', 'last_change': '2024-05-09T03:11:26.388619+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.388619+0000', 'last_undegraded': '2024-05-09T03:11:26.388619+0000', 'last_fullsized': '2024-05-09T03:11:26.388619+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2d is not active+clean 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.388811+0000', 'last_change': '2024-05-09T03:11:26.388811+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.388811+0000', 'last_undegraded': '2024-05-09T03:11:26.388811+0000', 'last_fullsized': '2024-05-09T03:11:26.388811+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2c is not active+clean 2024-05-09T03:11:32.267 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391447+0000', 'last_change': '2024-05-09T03:11:26.391447+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391447+0000', 'last_undegraded': '2024-05-09T03:11:26.391447+0000', 'last_fullsized': '2024-05-09T03:11:26.391447+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2b is not active+clean 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393040+0000', 'last_change': '2024-05-09T03:11:26.393040+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393040+0000', 'last_undegraded': '2024-05-09T03:11:26.393040+0000', 'last_fullsized': '2024-05-09T03:11:26.393040+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2a is not active+clean 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391316+0000', 'last_change': '2024-05-09T03:11:26.391316+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391316+0000', 'last_undegraded': '2024-05-09T03:11:26.391316+0000', 'last_fullsized': '2024-05-09T03:11:26.391316+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:PG 7.29 is not active+clean 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.389605+0000', 'last_change': '2024-05-09T03:11:26.389605+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.389605+0000', 'last_undegraded': '2024-05-09T03:11:26.389605+0000', 'last_fullsized': '2024-05-09T03:11:26.389605+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:PG 7.28 is not active+clean 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391465+0000', 'last_change': '2024-05-09T03:11:26.391465+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391465+0000', 'last_undegraded': '2024-05-09T03:11:26.391465+0000', 'last_fullsized': '2024-05-09T03:11:26.391465+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:PG 7.27 is not active+clean 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.390564+0000', 'last_change': '2024-05-09T03:11:26.390564+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.390564+0000', 'last_undegraded': '2024-05-09T03:11:26.390564+0000', 'last_fullsized': '2024-05-09T03:11:26.390564+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:PG 7.26 is not active+clean 2024-05-09T03:11:32.268 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.390514+0000', 'last_change': '2024-05-09T03:11:26.390514+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.390514+0000', 'last_undegraded': '2024-05-09T03:11:26.390514+0000', 'last_fullsized': '2024-05-09T03:11:26.390514+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:PG 7.25 is not active+clean 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.385973+0000', 'last_change': '2024-05-09T03:11:26.385973+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.385973+0000', 'last_undegraded': '2024-05-09T03:11:26.385973+0000', 'last_fullsized': '2024-05-09T03:11:26.385973+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:PG 7.24 is not active+clean 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392423+0000', 'last_change': '2024-05-09T03:11:26.392423+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392423+0000', 'last_undegraded': '2024-05-09T03:11:26.392423+0000', 'last_fullsized': '2024-05-09T03:11:26.392423+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:PG 7.23 is not active+clean 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391928+0000', 'last_change': '2024-05-09T03:11:26.391928+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391928+0000', 'last_undegraded': '2024-05-09T03:11:26.391928+0000', 'last_fullsized': '2024-05-09T03:11:26.391928+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:PG 7.22 is not active+clean 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.388354+0000', 'last_change': '2024-05-09T03:11:26.388354+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.388354+0000', 'last_undegraded': '2024-05-09T03:11:26.388354+0000', 'last_fullsized': '2024-05-09T03:11:26.388354+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:PG 7.21 is not active+clean 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393240+0000', 'last_change': '2024-05-09T03:11:26.393240+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393240+0000', 'last_undegraded': '2024-05-09T03:11:26.393240+0000', 'last_fullsized': '2024-05-09T03:11:26.393240+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:PG 7.20 is not active+clean 2024-05-09T03:11:32.269 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391939+0000', 'last_change': '2024-05-09T03:11:26.391939+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391939+0000', 'last_undegraded': '2024-05-09T03:11:26.391939+0000', 'last_fullsized': '2024-05-09T03:11:26.391939+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1e is not active+clean 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.389988+0000', 'last_change': '2024-05-09T03:11:26.389988+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.389988+0000', 'last_undegraded': '2024-05-09T03:11:26.389988+0000', 'last_fullsized': '2024-05-09T03:11:26.389988+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1f is not active+clean 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391552+0000', 'last_change': '2024-05-09T03:11:26.391552+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391552+0000', 'last_undegraded': '2024-05-09T03:11:26.391552+0000', 'last_fullsized': '2024-05-09T03:11:26.391552+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1c is not active+clean 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391936+0000', 'last_change': '2024-05-09T03:11:26.391936+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391936+0000', 'last_undegraded': '2024-05-09T03:11:26.391936+0000', 'last_fullsized': '2024-05-09T03:11:26.391936+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1d is not active+clean 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.390640+0000', 'last_change': '2024-05-09T03:11:26.390640+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.390640+0000', 'last_undegraded': '2024-05-09T03:11:26.390640+0000', 'last_fullsized': '2024-05-09T03:11:26.390640+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1a is not active+clean 2024-05-09T03:11:32.270 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.386654+0000', 'last_change': '2024-05-09T03:11:26.386654+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.386654+0000', 'last_undegraded': '2024-05-09T03:11:26.386654+0000', 'last_fullsized': '2024-05-09T03:11:26.386654+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1b is not active+clean 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.400278+0000', 'last_change': '2024-05-09T03:11:26.400278+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.400278+0000', 'last_undegraded': '2024-05-09T03:11:26.400278+0000', 'last_fullsized': '2024-05-09T03:11:26.400278+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:PG 7.18 is not active+clean 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391644+0000', 'last_change': '2024-05-09T03:11:26.391644+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391644+0000', 'last_undegraded': '2024-05-09T03:11:26.391644+0000', 'last_fullsized': '2024-05-09T03:11:26.391644+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:PG 7.19 is not active+clean 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393099+0000', 'last_change': '2024-05-09T03:11:26.393099+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393099+0000', 'last_undegraded': '2024-05-09T03:11:26.393099+0000', 'last_fullsized': '2024-05-09T03:11:26.393099+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:PG 7.16 is not active+clean 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.387537+0000', 'last_change': '2024-05-09T03:11:26.387537+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.387537+0000', 'last_undegraded': '2024-05-09T03:11:26.387537+0000', 'last_fullsized': '2024-05-09T03:11:26.387537+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:PG 7.17 is not active+clean 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.385422+0000', 'last_change': '2024-05-09T03:11:26.385422+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.385422+0000', 'last_undegraded': '2024-05-09T03:11:26.385422+0000', 'last_fullsized': '2024-05-09T03:11:26.385422+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:PG 7.14 is not active+clean 2024-05-09T03:11:32.271 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393376+0000', 'last_change': '2024-05-09T03:11:26.393376+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393376+0000', 'last_undegraded': '2024-05-09T03:11:26.393376+0000', 'last_fullsized': '2024-05-09T03:11:26.393376+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:PG 7.15 is not active+clean 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.386048+0000', 'last_change': '2024-05-09T03:11:26.386048+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.386048+0000', 'last_undegraded': '2024-05-09T03:11:26.386048+0000', 'last_fullsized': '2024-05-09T03:11:26.386048+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:PG 7.12 is not active+clean 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391660+0000', 'last_change': '2024-05-09T03:11:26.391660+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391660+0000', 'last_undegraded': '2024-05-09T03:11:26.391660+0000', 'last_fullsized': '2024-05-09T03:11:26.391660+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:PG 7.13 is not active+clean 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.399627+0000', 'last_change': '2024-05-09T03:11:26.399627+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.399627+0000', 'last_undegraded': '2024-05-09T03:11:26.399627+0000', 'last_fullsized': '2024-05-09T03:11:26.399627+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:PG 7.10 is not active+clean 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.393393+0000', 'last_change': '2024-05-09T03:11:26.393393+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.393393+0000', 'last_undegraded': '2024-05-09T03:11:26.393393+0000', 'last_fullsized': '2024-05-09T03:11:26.393393+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:PG 7.11 is not active+clean 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.385398+0000', 'last_change': '2024-05-09T03:11:26.385398+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.385398+0000', 'last_undegraded': '2024-05-09T03:11:26.385398+0000', 'last_fullsized': '2024-05-09T03:11:26.385398+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:PG 7.e is not active+clean 2024-05-09T03:11:32.272 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.390014+0000', 'last_change': '2024-05-09T03:11:26.390014+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.390014+0000', 'last_undegraded': '2024-05-09T03:11:26.390014+0000', 'last_fullsized': '2024-05-09T03:11:26.390014+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:PG 7.f is not active+clean 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.398258+0000', 'last_change': '2024-05-09T03:11:26.398258+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.398258+0000', 'last_undegraded': '2024-05-09T03:11:26.398258+0000', 'last_fullsized': '2024-05-09T03:11:26.398258+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:PG 7.c is not active+clean 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391987+0000', 'last_change': '2024-05-09T03:11:26.391987+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391987+0000', 'last_undegraded': '2024-05-09T03:11:26.391987+0000', 'last_fullsized': '2024-05-09T03:11:26.391987+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:PG 7.d is not active+clean 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.389422+0000', 'last_change': '2024-05-09T03:11:26.389422+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.389422+0000', 'last_undegraded': '2024-05-09T03:11:26.389422+0000', 'last_fullsized': '2024-05-09T03:11:26.389422+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:PG 7.a is not active+clean 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392466+0000', 'last_change': '2024-05-09T03:11:26.392466+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392466+0000', 'last_undegraded': '2024-05-09T03:11:26.392466+0000', 'last_fullsized': '2024-05-09T03:11:26.392466+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:PG 7.b is not active+clean 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392034+0000', 'last_change': '2024-05-09T03:11:26.392034+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392034+0000', 'last_undegraded': '2024-05-09T03:11:26.392034+0000', 'last_fullsized': '2024-05-09T03:11:26.392034+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:PG 7.8 is not active+clean 2024-05-09T03:11:32.273 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392477+0000', 'last_change': '2024-05-09T03:11:26.392477+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392477+0000', 'last_undegraded': '2024-05-09T03:11:26.392477+0000', 'last_fullsized': '2024-05-09T03:11:26.392477+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:PG 7.9 is not active+clean 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.390722+0000', 'last_change': '2024-05-09T03:11:26.390722+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.390722+0000', 'last_undegraded': '2024-05-09T03:11:26.390722+0000', 'last_fullsized': '2024-05-09T03:11:26.390722+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:PG 7.7 is not active+clean 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391073+0000', 'last_change': '2024-05-09T03:11:26.391073+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391073+0000', 'last_undegraded': '2024-05-09T03:11:26.391073+0000', 'last_fullsized': '2024-05-09T03:11:26.391073+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:PG 7.4 is not active+clean 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.392309+0000', 'last_change': '2024-05-09T03:11:26.392309+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.392309+0000', 'last_undegraded': '2024-05-09T03:11:26.392309+0000', 'last_fullsized': '2024-05-09T03:11:26.392309+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:PG 7.5 is not active+clean 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.385409+0000', 'last_change': '2024-05-09T03:11:26.385409+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.385409+0000', 'last_undegraded': '2024-05-09T03:11:26.385409+0000', 'last_fullsized': '2024-05-09T03:11:26.385409+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2 is not active+clean 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.389437+0000', 'last_change': '2024-05-09T03:11:26.389437+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.389437+0000', 'last_undegraded': '2024-05-09T03:11:26.389437+0000', 'last_fullsized': '2024-05-09T03:11:26.389437+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3 is not active+clean 2024-05-09T03:11:32.274 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.389149+0000', 'last_change': '2024-05-09T03:11:26.389149+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.389149+0000', 'last_undegraded': '2024-05-09T03:11:26.389149+0000', 'last_fullsized': '2024-05-09T03:11:26.389149+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:11:32.275 INFO:tasks.ceph.ceph_manager.ceph:PG 7.0 is not active+clean 2024-05-09T03:11:32.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.391430+0000', 'last_change': '2024-05-09T03:11:26.391430+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.391430+0000', 'last_undegraded': '2024-05-09T03:11:26.391430+0000', 'last_fullsized': '2024-05-09T03:11:26.391430+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:11:32.275 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1 is not active+clean 2024-05-09T03:11:32.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.385911+0000', 'last_change': '2024-05-09T03:11:26.385911+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.385911+0000', 'last_undegraded': '2024-05-09T03:11:26.385911+0000', 'last_fullsized': '2024-05-09T03:11:26.385911+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:11:32.275 INFO:tasks.ceph.ceph_manager.ceph:PG 7.6 is not active+clean 2024-05-09T03:11:32.275 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 41, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:11:26.398327+0000', 'last_change': '2024-05-09T03:11:26.398327+0000', 'last_active': '2024-05-09T03:11:26.379689+0000', 'last_peered': '2024-05-09T03:11:26.379689+0000', 'last_clean': '2024-05-09T03:11:26.379689+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:11:26.398327+0000', 'last_undegraded': '2024-05-09T03:11:26.398327+0000', 'last_fullsized': '2024-05-09T03:11:26.398327+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 41, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'last_clean_scrub_stamp': '2024-05-09T03:11:26.379689+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:11:32.276 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-09T03:11:32.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.413+0000 7fb7d0d72640 1 Processor -- start 2024-05-09T03:11:32.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.413+0000 7fb7d0d72640 1 -- start start 2024-05-09T03:11:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7d0d72640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7cc0d57a0 0x7fb7cc160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7d0d72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7d0d72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7cc0d49e0 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7d0d72640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb7cc0d5bb0 con 0x7fb7cc0d57a0 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7d0d72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d4cc0 con 0x7fb7cc0b4af0 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7ca59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb7cc0d57a0 0x7fb7cc160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7cad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7cc0b77e0 0x7fb7cc0b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57954/0 (socket says 172.21.15.46:57954) 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7cad9d640 1 -- 172.21.15.46:0/642823826 learned_addr learned my addr 172.21.15.46:0/642823826 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.415+0000 7fb7c9d9b640 1 --2- 172.21.15.46:0/642823826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7ca59c640 1 -- 172.21.15.46:0/642823826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 msgr2=0x7fb7cc0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7ca59c640 1 --2- 172.21.15.46:0/642823826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7ca59c640 1 -- 172.21.15.46:0/642823826 >> v1:172.21.15.46:6789/0 conn(0x7fb7cc0b77e0 legacy=0x7fb7cc0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7ca59c640 1 -- 172.21.15.46:0/642823826 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc0d4fa0 con 0x7fb7cc0d57a0 2024-05-09T03:11:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7c959a640 1 -- 172.21.15.46:0/642823826 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2875714084 0 0) 0x7fb7cc0d49e0 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7ca59c640 1 --2- 172.21.15.46:0/642823826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc160610 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40097a0 tx=0x7fb7c402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9caa13125125cb3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7c959a640 1 -- 172.21.15.46:0/642823826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4037070 con 0x7fb7cc0d57a0 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7c959a640 1 -- 172.21.15.46:0/642823826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c4032440 con 0x7fb7cc0d57a0 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.416+0000 7fb7c959a640 1 -- 172.21.15.46:0/642823826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4032810 con 0x7fb7cc0d57a0 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 -- 172.21.15.46:0/642823826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0d57a0 msgr2=0x7fb7cc160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/642823826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc160610 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40097a0 tx=0x7fb7c402a180 comp rx=0 tx=0).stop 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 -- 172.21.15.46:0/642823826 shutdown_connections 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/642823826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/642823826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc160610 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 -- 172.21.15.46:0/642823826 >> 172.21.15.46:0/642823826 conn(0x7fb7cc0743e0 msgr2=0x7fb7cc0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 -- 172.21.15.46:0/642823826 shutdown_connections 2024-05-09T03:11:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.417+0000 7fb7d0d72640 1 -- 172.21.15.46:0/642823826 wait complete. 2024-05-09T03:11:32.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.419+0000 7fb7d0d72640 1 Processor -- start 2024-05-09T03:11:32.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.419+0000 7fb7d0d72640 1 -- start start 2024-05-09T03:11:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 0x7fb7cc15c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d1920 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc0d15b0 con 0x7fb7cc0d57a0 2024-05-09T03:11:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc153b70 con 0x7fb7cc0b4af0 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 0x7fb7cc15c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 0x7fb7cc15c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60240/0 (socket says 172.21.15.46:60240) 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 -- 172.21.15.46:0/3421088743 learned_addr learned my addr 172.21.15.46:0/3421088743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7ca59c640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7cad9d640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 -- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 msgr2=0x7fb7cc153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 -- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7cc0d57a0 msgr2=0x7fb7cc15c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc15c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 -- 172.21.15.46:0/3421088743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc159820 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7cad9d640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7ca59c640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c9d9b640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 0x7fb7cc15c460 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400d5f0 tx=0x7fb7b400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40125f0 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b4019070 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40129d0 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.421+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7cc150e40 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.422+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7cc151400 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.422+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7cc06a270 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.423+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb7b4012b70 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.424+0000 7fb7c2ffd640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7ac0762d0 0x7fb7ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.424+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fb7b4097a90 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.424+0000 7fb7ca59c640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7ac0762d0 0x7fb7ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.424+0000 7fb7ca59c640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7ac0762d0 0x7fb7ac0787e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4034000 tx=0x7fb7c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.428+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb7b4061360 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.692+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fb7cc0b75a0 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.692+0000 7fb7c2ffd640 1 -- 172.21.15.46:0/3421088743 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1195 (secure 0 0 0) 0x7fb7b4060c60 con 0x7fb7cc0b77e0 2024-05-09T03:11:32.693 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:32.693 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":142,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":8,"num_up_osds":8,"osd_up_since":1715224161,"num_in_osds":8,"osd_in_since":1715224152,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":243376128,"bytes_avail":772850737152,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":30,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5358}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-05-09T03:11:10.515812+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-05-09T03:11:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7ac0762d0 msgr2=0x7fb7ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7ac0762d0 0x7fb7ac0787e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4034000 tx=0x7fb7c402e000 comp rx=0 tx=0).stop 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 msgr2=0x7fb7cc15c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 0x7fb7cc15c460 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb7b400d5f0 tx=0x7fb7b400dab0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 shutdown_connections 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7cc0d57a0 0x7fb7cc15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7ac0762d0 0x7fb7ac0787e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7cc0b77e0 0x7fb7cc15c460 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 --2- 172.21.15.46:0/3421088743 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb7cc0b4af0 0x7fb7cc153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.694+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 >> 172.21.15.46:0/3421088743 conn(0x7fb7cc0743e0 msgr2=0x7fb7cc0b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.695+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 shutdown_connections 2024-05-09T03:11:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.695+0000 7fb7d0d72640 1 -- 172.21.15.46:0/3421088743 wait complete. 2024-05-09T03:11:32.707 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-05-09T03:11:32.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:11:32.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.821+0000 7fa68037d640 1 Processor -- start 2024-05-09T03:11:32.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.821+0000 7fa68037d640 1 -- start start 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa68037d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6780b7a50 0x7fa6781698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa68037d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6780b4ee0 con 0x7fa6780b9200 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa68037d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6780b62e0 con 0x7fa678162610 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa68037d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa6780b5460 con 0x7fa6780b7a50 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa67e119640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6780b9200 0x7fa67808d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57968/0 (socket says 172.21.15.46:57968) 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa67e119640 1 -- 172.21.15.46:0/2904401785 learned_addr learned my addr 172.21.15.46:0/2904401785 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.822+0000 7fa67e91a640 1 --2- 172.21.15.46:0/2904401785 >> v2:172.21.15.88:3300/0 conn(0x7fa6780b7a50 0x7fa6781698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 776626131 0 0) 0x7fa6780b4ee0 con 0x7fa6780b9200 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa660003520 con 0x7fa6780b9200 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2036847836 0 0) 0x7fa6780b62e0 con 0x7fa678162610 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6780b4ee0 con 0x7fa678162610 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67e91a640 1 -- 172.21.15.46:0/2904401785 >> v1:172.21.15.88:6789/0 conn(0x7fa678162610 legacy=0x7fa678162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67e91a640 1 -- 172.21.15.46:0/2904401785 >> v1:172.21.15.46:6789/0 conn(0x7fa6780b9200 legacy=0x7fa67808d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3462145530 0 0) 0x7fa660003520 con 0x7fa6780b9200 2024-05-09T03:11:32.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.823+0000 7fa67e91a640 1 -- 172.21.15.46:0/2904401785 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6780b5180 con 0x7fa6780b7a50 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.824+0000 7fa67e91a640 1 --2- 172.21.15.46:0/2904401785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b7a50 0x7fa6781698d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa6700097a0 tx=0x7fa67002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18dba85916a48fe0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.824+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670037070 con 0x7fa6780b7a50 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.824+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa67002ac50 con 0x7fa6780b7a50 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.824+0000 7fa67d117640 1 -- 172.21.15.46:0/2904401785 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668001a90 con 0x7fa6780b7a50 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 -- 172.21.15.46:0/2904401785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b7a50 msgr2=0x7fa6781698d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 --2- 172.21.15.46:0/2904401785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b7a50 0x7fa6781698d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa6700097a0 tx=0x7fa67002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 -- 172.21.15.46:0/2904401785 shutdown_connections 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 --2- 172.21.15.46:0/2904401785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b7a50 0x7fa6781698d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 -- 172.21.15.46:0/2904401785 >> 172.21.15.46:0/2904401785 conn(0x7fa6780b56e0 msgr2=0x7fa6780b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 -- 172.21.15.46:0/2904401785 shutdown_connections 2024-05-09T03:11:32.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.825+0000 7fa68037d640 1 -- 172.21.15.46:0/2904401785 wait complete. 2024-05-09T03:11:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.827+0000 7fa68037d640 1 Processor -- start 2024-05-09T03:11:32.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.827+0000 7fa68037d640 1 -- start start 2024-05-09T03:11:32.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa68037d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 0x7fa67806f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa68037d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b9200 0x7fa6780ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa68037d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa678162610 0x7fa6780cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa68037d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cf750 con 0x7fa6780b7a50 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa68037d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cfac0 con 0x7fa6780b9200 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa68037d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa67808d8b0 con 0x7fa678162610 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 0x7fa67806f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 0x7fa67806f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60244/0 (socket says 172.21.15.46:60244) 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 -- 172.21.15.46:0/2731467480 learned_addr learned my addr 172.21.15.46:0/2731467480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67d918640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b9200 0x7fa6780ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e91a640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa678162610 0x7fa6780cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 -- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa678162610 msgr2=0x7fa6780cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa678162610 0x7fa6780cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 -- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b9200 msgr2=0x7fa6780ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:32.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b9200 0x7fa6780ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.828+0000 7fa67e119640 1 -- 172.21.15.46:0/2731467480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67806e830 con 0x7fa6780b7a50 2024-05-09T03:11:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.829+0000 7fa67e119640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 0x7fa67806f310 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa66800ca80 tx=0x7fa66800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.829+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668018070 con 0x7fa6780b7a50 2024-05-09T03:11:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.829+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66800a5e0 con 0x7fa6780b7a50 2024-05-09T03:11:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.829+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6780cc080 con 0x7fa6780b7a50 2024-05-09T03:11:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.829+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66800a9d0 con 0x7fa6780b7a50 2024-05-09T03:11:32.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.829+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6781f79b0 con 0x7fa6780b7a50 2024-05-09T03:11:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.830+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6781f7c10 con 0x7fa6780b7a50 2024-05-09T03:11:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.831+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa66801b070 con 0x7fa6780b7a50 2024-05-09T03:11:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.831+0000 7fa676ffd640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6640762d0 0x7fa6640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.831+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fa668096fa0 con 0x7fa6780b7a50 2024-05-09T03:11:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.831+0000 7fa67d918640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6640762d0 0x7fa6640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:32.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.832+0000 7fa67d918640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6640762d0 0x7fa6640787e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa6780b4860 tx=0x7fa66c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:32.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:32.835+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa6680609e0 con 0x7fa6780b7a50 2024-05-09T03:11:33.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.022+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa67806a270 con 0x7fa6640762d0 2024-05-09T03:11:33.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.029+0000 7fa676ffd640 1 -- 172.21.15.46:0/2731467480 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7fa67806a270 con 0x7fa6640762d0 2024-05-09T03:11:33.032 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:33.033 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6640762d0 msgr2=0x7fa6640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6640762d0 0x7fa6640787e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa6780b4860 tx=0x7fa66c009040 comp rx=0 tx=0).stop 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 msgr2=0x7fa67806f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 0x7fa67806f310 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa66800ca80 tx=0x7fa66800cf40 comp rx=0 tx=0).stop 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 shutdown_connections 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa678162610 0x7fa6780cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6780b9200 0x7fa6780ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:33.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6640762d0 0x7fa6640787e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 --2- 172.21.15.46:0/2731467480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6780b7a50 0x7fa67806f310 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 >> 172.21.15.46:0/2731467480 conn(0x7fa6780b56e0 msgr2=0x7fa678160d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 shutdown_connections 2024-05-09T03:11:33.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:33.032+0000 7fa68037d640 1 -- 172.21.15.46:0/2731467480 wait complete. 2024-05-09T03:11:33.044 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-05-09T03:11:32.521086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":179,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238008,"kb_used_data":5160,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736712,"statfs":{"total":773094113280,"available":772850393088,"internally_reserved":0,"allocated":5283840,"data_stored":2816060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.137801"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766498+0000","last_change":"2024-05-09T03:11:27.416315+0000","last_active":"2024-05-09T03:11:28.766498+0000","last_peered":"2024-05-09T03:11:28.766498+0000","last_clean":"2024-05-09T03:11:28.766498+0000","last_became_active":"2024-05-09T03:11:27.415726+0000","last_became_peered":"2024-05-09T03:11:27.415726+0000","last_unstale":"2024-05-09T03:11:28.766498+0000","last_undegraded":"2024-05-09T03:11:28.766498+0000","last_fullsized":"2024-05-09T03:11:28.766498+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:42:24.049436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763776+0000","last_change":"2024-05-09T03:11:27.414806+0000","last_active":"2024-05-09T03:11:28.763776+0000","last_peered":"2024-05-09T03:11:28.763776+0000","last_clean":"2024-05-09T03:11:28.763776+0000","last_became_active":"2024-05-09T03:11:27.414490+0000","last_became_peered":"2024-05-09T03:11:27.414490+0000","last_unstale":"2024-05-09T03:11:28.763776+0000","last_undegraded":"2024-05-09T03:11:28.763776+0000","last_fullsized":"2024-05-09T03:11:28.763776+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:30.226498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398863+0000","last_change":"2024-05-09T03:11:27.407445+0000","last_active":"2024-05-09T03:11:28.398863+0000","last_peered":"2024-05-09T03:11:28.398863+0000","last_clean":"2024-05-09T03:11:28.398863+0000","last_became_active":"2024-05-09T03:11:27.407132+0000","last_became_peered":"2024-05-09T03:11:27.407132+0000","last_unstale":"2024-05-09T03:11:28.398863+0000","last_undegraded":"2024-05-09T03:11:28.398863+0000","last_fullsized":"2024-05-09T03:11:28.398863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:25.693725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763247+0000","last_change":"2024-05-09T03:11:27.420521+0000","last_active":"2024-05-09T03:11:28.763247+0000","last_peered":"2024-05-09T03:11:28.763247+0000","last_clean":"2024-05-09T03:11:28.763247+0000","last_became_active":"2024-05-09T03:11:27.420134+0000","last_became_peered":"2024-05-09T03:11:27.420134+0000","last_unstale":"2024-05-09T03:11:28.763247+0000","last_undegraded":"2024-05-09T03:11:28.763247+0000","last_fullsized":"2024-05-09T03:11:28.763247+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:25.175611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763575+0000","last_change":"2024-05-09T03:11:27.414109+0000","last_active":"2024-05-09T03:11:28.763575+0000","last_peered":"2024-05-09T03:11:28.763575+0000","last_clean":"2024-05-09T03:11:28.763575+0000","last_became_active":"2024-05-09T03:11:27.413743+0000","last_became_peered":"2024-05-09T03:11:27.413743+0000","last_unstale":"2024-05-09T03:11:28.763575+0000","last_undegraded":"2024-05-09T03:11:28.763575+0000","last_fullsized":"2024-05-09T03:11:28.763575+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:23.507112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421588+0000","last_change":"2024-05-09T03:11:27.416637+0000","last_active":"2024-05-09T03:11:28.421588+0000","last_peered":"2024-05-09T03:11:28.421588+0000","last_clean":"2024-05-09T03:11:28.421588+0000","last_became_active":"2024-05-09T03:11:27.416223+0000","last_became_peered":"2024-05-09T03:11:27.416223+0000","last_unstale":"2024-05-09T03:11:28.421588+0000","last_undegraded":"2024-05-09T03:11:28.421588+0000","last_fullsized":"2024-05-09T03:11:28.421588+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:18.454432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766185+0000","last_change":"2024-05-09T03:11:27.417640+0000","last_active":"2024-05-09T03:11:28.766185+0000","last_peered":"2024-05-09T03:11:28.766185+0000","last_clean":"2024-05-09T03:11:28.766185+0000","last_became_active":"2024-05-09T03:11:27.417188+0000","last_became_peered":"2024-05-09T03:11:27.417188+0000","last_unstale":"2024-05-09T03:11:28.766185+0000","last_undegraded":"2024-05-09T03:11:28.766185+0000","last_fullsized":"2024-05-09T03:11:28.766185+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:30.226498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763468+0000","last_change":"2024-05-09T03:11:27.418976+0000","last_active":"2024-05-09T03:11:28.763468+0000","last_peered":"2024-05-09T03:11:28.763468+0000","last_clean":"2024-05-09T03:11:28.763468+0000","last_became_active":"2024-05-09T03:11:27.418232+0000","last_became_peered":"2024-05-09T03:11:27.418232+0000","last_unstale":"2024-05-09T03:11:28.763468+0000","last_undegraded":"2024-05-09T03:11:28.763468+0000","last_fullsized":"2024-05-09T03:11:28.763468+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:23.507112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763556+0000","last_change":"2024-05-09T03:11:27.410452+0000","last_active":"2024-05-09T03:11:28.763556+0000","last_peered":"2024-05-09T03:11:28.763556+0000","last_clean":"2024-05-09T03:11:28.763556+0000","last_became_active":"2024-05-09T03:11:27.410080+0000","last_became_peered":"2024-05-09T03:11:27.410080+0000","last_unstale":"2024-05-09T03:11:28.763556+0000","last_undegraded":"2024-05-09T03:11:28.763556+0000","last_fullsized":"2024-05-09T03:11:28.763556+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:24:52.213030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763838+0000","last_change":"2024-05-09T03:11:27.418629+0000","last_active":"2024-05-09T03:11:28.763838+0000","last_peered":"2024-05-09T03:11:28.763838+0000","last_clean":"2024-05-09T03:11:28.763838+0000","last_became_active":"2024-05-09T03:11:27.418191+0000","last_became_peered":"2024-05-09T03:11:27.418191+0000","last_unstale":"2024-05-09T03:11:28.763838+0000","last_undegraded":"2024-05-09T03:11:28.763838+0000","last_fullsized":"2024-05-09T03:11:28.763838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:42:24.049436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763516+0000","last_change":"2024-05-09T03:11:27.410458+0000","last_active":"2024-05-09T03:11:28.763516+0000","last_peered":"2024-05-09T03:11:28.763516+0000","last_clean":"2024-05-09T03:11:28.763516+0000","last_became_active":"2024-05-09T03:11:27.410113+0000","last_became_peered":"2024-05-09T03:11:27.410113+0000","last_unstale":"2024-05-09T03:11:28.763516+0000","last_undegraded":"2024-05-09T03:11:28.763516+0000","last_fullsized":"2024-05-09T03:11:28.763516+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:55.297209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763640+0000","last_change":"2024-05-09T03:11:27.406120+0000","last_active":"2024-05-09T03:11:28.763640+0000","last_peered":"2024-05-09T03:11:28.763640+0000","last_clean":"2024-05-09T03:11:28.763640+0000","last_became_active":"2024-05-09T03:11:27.405636+0000","last_became_peered":"2024-05-09T03:11:27.405636+0000","last_unstale":"2024-05-09T03:11:28.763640+0000","last_undegraded":"2024-05-09T03:11:28.763640+0000","last_fullsized":"2024-05-09T03:11:28.763640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:55.297209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395368+0000","last_change":"2024-05-09T03:11:27.413654+0000","last_active":"2024-05-09T03:11:28.395368+0000","last_peered":"2024-05-09T03:11:28.395368+0000","last_clean":"2024-05-09T03:11:28.395368+0000","last_became_active":"2024-05-09T03:11:27.413316+0000","last_became_peered":"2024-05-09T03:11:27.413316+0000","last_unstale":"2024-05-09T03:11:28.395368+0000","last_undegraded":"2024-05-09T03:11:28.395368+0000","last_fullsized":"2024-05-09T03:11:28.395368+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:04:16.199599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763673+0000","last_change":"2024-05-09T03:11:27.415386+0000","last_active":"2024-05-09T03:11:28.763673+0000","last_peered":"2024-05-09T03:11:28.763673+0000","last_clean":"2024-05-09T03:11:28.763673+0000","last_became_active":"2024-05-09T03:11:27.412137+0000","last_became_peered":"2024-05-09T03:11:27.412137+0000","last_unstale":"2024-05-09T03:11:28.763673+0000","last_undegraded":"2024-05-09T03:11:28.763673+0000","last_fullsized":"2024-05-09T03:11:28.763673+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:30.226498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421546+0000","last_change":"2024-05-09T03:11:27.416475+0000","last_active":"2024-05-09T03:11:28.421546+0000","last_peered":"2024-05-09T03:11:28.421546+0000","last_clean":"2024-05-09T03:11:28.421546+0000","last_became_active":"2024-05-09T03:11:27.416157+0000","last_became_peered":"2024-05-09T03:11:27.416157+0000","last_unstale":"2024-05-09T03:11:28.421546+0000","last_undegraded":"2024-05-09T03:11:28.421546+0000","last_fullsized":"2024-05-09T03:11:28.421546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:10.279184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398865+0000","last_change":"2024-05-09T03:11:27.421834+0000","last_active":"2024-05-09T03:11:28.398865+0000","last_peered":"2024-05-09T03:11:28.398865+0000","last_clean":"2024-05-09T03:11:28.398865+0000","last_became_active":"2024-05-09T03:11:27.421442+0000","last_became_peered":"2024-05-09T03:11:27.421442+0000","last_unstale":"2024-05-09T03:11:28.398865+0000","last_undegraded":"2024-05-09T03:11:28.398865+0000","last_fullsized":"2024-05-09T03:11:28.398865+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:26.498723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399435+0000","last_change":"2024-05-09T03:11:27.413670+0000","last_active":"2024-05-09T03:11:28.399435+0000","last_peered":"2024-05-09T03:11:28.399435+0000","last_clean":"2024-05-09T03:11:28.399435+0000","last_became_active":"2024-05-09T03:11:27.413362+0000","last_became_peered":"2024-05-09T03:11:27.413362+0000","last_unstale":"2024-05-09T03:11:28.399435+0000","last_undegraded":"2024-05-09T03:11:28.399435+0000","last_fullsized":"2024-05-09T03:11:28.399435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:28.857415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398949+0000","last_change":"2024-05-09T03:11:27.415156+0000","last_active":"2024-05-09T03:11:28.398949+0000","last_peered":"2024-05-09T03:11:28.398949+0000","last_clean":"2024-05-09T03:11:28.398949+0000","last_became_active":"2024-05-09T03:11:27.414821+0000","last_became_peered":"2024-05-09T03:11:27.414821+0000","last_unstale":"2024-05-09T03:11:28.398949+0000","last_undegraded":"2024-05-09T03:11:28.398949+0000","last_fullsized":"2024-05-09T03:11:28.398949+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:39.037525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400154+0000","last_change":"2024-05-09T03:11:27.412965+0000","last_active":"2024-05-09T03:11:28.400154+0000","last_peered":"2024-05-09T03:11:28.400154+0000","last_clean":"2024-05-09T03:11:28.400154+0000","last_became_active":"2024-05-09T03:11:27.412660+0000","last_became_peered":"2024-05-09T03:11:27.412660+0000","last_unstale":"2024-05-09T03:11:28.400154+0000","last_undegraded":"2024-05-09T03:11:28.400154+0000","last_fullsized":"2024-05-09T03:11:28.400154+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:06.230758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395530+0000","last_change":"2024-05-09T03:11:27.422078+0000","last_active":"2024-05-09T03:11:28.395530+0000","last_peered":"2024-05-09T03:11:28.395530+0000","last_clean":"2024-05-09T03:11:28.395530+0000","last_became_active":"2024-05-09T03:11:27.421684+0000","last_became_peered":"2024-05-09T03:11:27.421684+0000","last_unstale":"2024-05-09T03:11:28.395530+0000","last_undegraded":"2024-05-09T03:11:28.395530+0000","last_fullsized":"2024-05-09T03:11:28.395530+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:49.735304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421684+0000","last_change":"2024-05-09T03:11:27.415241+0000","last_active":"2024-05-09T03:11:28.421684+0000","last_peered":"2024-05-09T03:11:28.421684+0000","last_clean":"2024-05-09T03:11:28.421684+0000","last_became_active":"2024-05-09T03:11:27.414898+0000","last_became_peered":"2024-05-09T03:11:27.414898+0000","last_unstale":"2024-05-09T03:11:28.421684+0000","last_undegraded":"2024-05-09T03:11:28.421684+0000","last_fullsized":"2024-05-09T03:11:28.421684+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:57.365456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763756+0000","last_change":"2024-05-09T03:11:27.420886+0000","last_active":"2024-05-09T03:11:28.763756+0000","last_peered":"2024-05-09T03:11:28.763756+0000","last_clean":"2024-05-09T03:11:28.763756+0000","last_became_active":"2024-05-09T03:11:27.420585+0000","last_became_peered":"2024-05-09T03:11:27.420585+0000","last_unstale":"2024-05-09T03:11:28.763756+0000","last_undegraded":"2024-05-09T03:11:28.763756+0000","last_fullsized":"2024-05-09T03:11:28.763756+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:42.288267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400015+0000","last_change":"2024-05-09T03:11:27.410820+0000","last_active":"2024-05-09T03:11:28.400015+0000","last_peered":"2024-05-09T03:11:28.400015+0000","last_clean":"2024-05-09T03:11:28.400015+0000","last_became_active":"2024-05-09T03:11:27.409738+0000","last_became_peered":"2024-05-09T03:11:27.409738+0000","last_unstale":"2024-05-09T03:11:28.400015+0000","last_undegraded":"2024-05-09T03:11:28.400015+0000","last_fullsized":"2024-05-09T03:11:28.400015+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:29.977279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398901+0000","last_change":"2024-05-09T03:11:27.410340+0000","last_active":"2024-05-09T03:11:28.398901+0000","last_peered":"2024-05-09T03:11:28.398901+0000","last_clean":"2024-05-09T03:11:28.398901+0000","last_became_active":"2024-05-09T03:11:27.409999+0000","last_became_peered":"2024-05-09T03:11:27.409999+0000","last_unstale":"2024-05-09T03:11:28.398901+0000","last_undegraded":"2024-05-09T03:11:28.398901+0000","last_fullsized":"2024-05-09T03:11:28.398901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:36:10.742856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400286+0000","last_change":"2024-05-09T03:11:27.416485+0000","last_active":"2024-05-09T03:11:28.400286+0000","last_peered":"2024-05-09T03:11:28.400286+0000","last_clean":"2024-05-09T03:11:28.400286+0000","last_became_active":"2024-05-09T03:11:27.415236+0000","last_became_peered":"2024-05-09T03:11:27.415236+0000","last_unstale":"2024-05-09T03:11:28.400286+0000","last_undegraded":"2024-05-09T03:11:28.400286+0000","last_fullsized":"2024-05-09T03:11:28.400286+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:12.139044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395263+0000","last_change":"2024-05-09T03:11:27.422564+0000","last_active":"2024-05-09T03:11:28.395263+0000","last_peered":"2024-05-09T03:11:28.395263+0000","last_clean":"2024-05-09T03:11:28.395263+0000","last_became_active":"2024-05-09T03:11:27.422200+0000","last_became_peered":"2024-05-09T03:11:27.422200+0000","last_unstale":"2024-05-09T03:11:28.395263+0000","last_undegraded":"2024-05-09T03:11:28.395263+0000","last_fullsized":"2024-05-09T03:11:28.395263+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:38.419189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399215+0000","last_change":"2024-05-09T03:11:27.405881+0000","last_active":"2024-05-09T03:11:28.399215+0000","last_peered":"2024-05-09T03:11:28.399215+0000","last_clean":"2024-05-09T03:11:28.399215+0000","last_became_active":"2024-05-09T03:11:27.404751+0000","last_became_peered":"2024-05-09T03:11:27.404751+0000","last_unstale":"2024-05-09T03:11:28.399215+0000","last_undegraded":"2024-05-09T03:11:28.399215+0000","last_fullsized":"2024-05-09T03:11:28.399215+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:25.175611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763751+0000","last_change":"2024-05-09T03:11:27.405410+0000","last_active":"2024-05-09T03:11:28.763751+0000","last_peered":"2024-05-09T03:11:28.763751+0000","last_clean":"2024-05-09T03:11:28.763751+0000","last_became_active":"2024-05-09T03:11:27.404135+0000","last_became_peered":"2024-05-09T03:11:27.404135+0000","last_unstale":"2024-05-09T03:11:28.763751+0000","last_undegraded":"2024-05-09T03:11:28.763751+0000","last_fullsized":"2024-05-09T03:11:28.763751+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:24:52.213030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399815+0000","last_change":"2024-05-09T03:11:27.407820+0000","last_active":"2024-05-09T03:11:28.399815+0000","last_peered":"2024-05-09T03:11:28.399815+0000","last_clean":"2024-05-09T03:11:28.399815+0000","last_became_active":"2024-05-09T03:11:27.407497+0000","last_became_peered":"2024-05-09T03:11:27.407497+0000","last_unstale":"2024-05-09T03:11:28.399815+0000","last_undegraded":"2024-05-09T03:11:28.399815+0000","last_fullsized":"2024-05-09T03:11:28.399815+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:41.718657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398572+0000","last_change":"2024-05-09T03:11:27.410973+0000","last_active":"2024-05-09T03:11:28.398572+0000","last_peered":"2024-05-09T03:11:28.398572+0000","last_clean":"2024-05-09T03:11:28.398572+0000","last_became_active":"2024-05-09T03:11:27.410693+0000","last_became_peered":"2024-05-09T03:11:27.410693+0000","last_unstale":"2024-05-09T03:11:28.398572+0000","last_undegraded":"2024-05-09T03:11:28.398572+0000","last_fullsized":"2024-05-09T03:11:28.398572+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:09.678343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421631+0000","last_change":"2024-05-09T03:11:27.407906+0000","last_active":"2024-05-09T03:11:28.421631+0000","last_peered":"2024-05-09T03:11:28.421631+0000","last_clean":"2024-05-09T03:11:28.421631+0000","last_became_active":"2024-05-09T03:11:27.404754+0000","last_became_peered":"2024-05-09T03:11:27.404754+0000","last_unstale":"2024-05-09T03:11:28.421631+0000","last_undegraded":"2024-05-09T03:11:28.421631+0000","last_fullsized":"2024-05-09T03:11:28.421631+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:49.585557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395498+0000","last_change":"2024-05-09T03:11:27.416539+0000","last_active":"2024-05-09T03:11:28.395498+0000","last_peered":"2024-05-09T03:11:28.395498+0000","last_clean":"2024-05-09T03:11:28.395498+0000","last_became_active":"2024-05-09T03:11:27.416229+0000","last_became_peered":"2024-05-09T03:11:27.416229+0000","last_unstale":"2024-05-09T03:11:28.395498+0000","last_undegraded":"2024-05-09T03:11:28.395498+0000","last_fullsized":"2024-05-09T03:11:28.395498+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:43.943712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763348+0000","last_change":"2024-05-09T03:11:27.407857+0000","last_active":"2024-05-09T03:11:28.763348+0000","last_peered":"2024-05-09T03:11:28.763348+0000","last_clean":"2024-05-09T03:11:28.763348+0000","last_became_active":"2024-05-09T03:11:27.407515+0000","last_became_peered":"2024-05-09T03:11:27.407515+0000","last_unstale":"2024-05-09T03:11:28.763348+0000","last_undegraded":"2024-05-09T03:11:28.763348+0000","last_fullsized":"2024-05-09T03:11:28.763348+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:41.718657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421983+0000","last_change":"2024-05-09T03:11:26.395704+0000","last_active":"2024-05-09T03:11:28.421983+0000","last_peered":"2024-05-09T03:11:28.421983+0000","last_clean":"2024-05-09T03:11:28.421983+0000","last_became_active":"2024-05-09T03:11:26.394437+0000","last_became_peered":"2024-05-09T03:11:26.394437+0000","last_unstale":"2024-05-09T03:11:28.421983+0000","last_undegraded":"2024-05-09T03:11:28.421983+0000","last_fullsized":"2024-05-09T03:11:28.421983+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:36.728436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400128+0000","last_change":"2024-05-09T03:11:27.406857+0000","last_active":"2024-05-09T03:11:28.400128+0000","last_peered":"2024-05-09T03:11:28.400128+0000","last_clean":"2024-05-09T03:11:28.400128+0000","last_became_active":"2024-05-09T03:11:27.405935+0000","last_became_peered":"2024-05-09T03:11:27.405935+0000","last_unstale":"2024-05-09T03:11:28.400128+0000","last_undegraded":"2024-05-09T03:11:28.400128+0000","last_fullsized":"2024-05-09T03:11:28.400128+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:43.768405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399164+0000","last_change":"2024-05-09T03:11:26.395710+0000","last_active":"2024-05-09T03:11:28.399164+0000","last_peered":"2024-05-09T03:11:28.399164+0000","last_clean":"2024-05-09T03:11:28.399164+0000","last_became_active":"2024-05-09T03:11:26.395376+0000","last_became_peered":"2024-05-09T03:11:26.395376+0000","last_unstale":"2024-05-09T03:11:28.399164+0000","last_undegraded":"2024-05-09T03:11:28.399164+0000","last_fullsized":"2024-05-09T03:11:28.399164+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:51.356924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763887+0000","last_change":"2024-05-09T03:11:27.408971+0000","last_active":"2024-05-09T03:11:28.763887+0000","last_peered":"2024-05-09T03:11:28.763887+0000","last_clean":"2024-05-09T03:11:28.763887+0000","last_became_active":"2024-05-09T03:11:27.408422+0000","last_became_peered":"2024-05-09T03:11:27.408422+0000","last_unstale":"2024-05-09T03:11:28.763887+0000","last_undegraded":"2024-05-09T03:11:28.763887+0000","last_fullsized":"2024-05-09T03:11:28.763887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:19.073921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763688+0000","last_change":"2024-05-09T03:11:26.400247+0000","last_active":"2024-05-09T03:11:28.763688+0000","last_peered":"2024-05-09T03:11:28.763688+0000","last_clean":"2024-05-09T03:11:28.763688+0000","last_became_active":"2024-05-09T03:11:26.399921+0000","last_became_peered":"2024-05-09T03:11:26.399921+0000","last_unstale":"2024-05-09T03:11:28.763688+0000","last_undegraded":"2024-05-09T03:11:28.763688+0000","last_fullsized":"2024-05-09T03:11:28.763688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:36:09.738086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763576+0000","last_change":"2024-05-09T03:11:27.415016+0000","last_active":"2024-05-09T03:11:28.763576+0000","last_peered":"2024-05-09T03:11:28.763576+0000","last_clean":"2024-05-09T03:11:28.763576+0000","last_became_active":"2024-05-09T03:11:27.414667+0000","last_became_peered":"2024-05-09T03:11:27.414667+0000","last_unstale":"2024-05-09T03:11:28.763576+0000","last_undegraded":"2024-05-09T03:11:28.763576+0000","last_fullsized":"2024-05-09T03:11:28.763576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:29.977279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766350+0000","last_change":"2024-05-09T03:11:26.400502+0000","last_active":"2024-05-09T03:11:28.766350+0000","last_peered":"2024-05-09T03:11:28.766350+0000","last_clean":"2024-05-09T03:11:28.766350+0000","last_became_active":"2024-05-09T03:11:26.400182+0000","last_became_peered":"2024-05-09T03:11:26.400182+0000","last_unstale":"2024-05-09T03:11:28.766350+0000","last_undegraded":"2024-05-09T03:11:28.766350+0000","last_fullsized":"2024-05-09T03:11:28.766350+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398563+0000","last_change":"2024-05-09T03:11:27.409079+0000","last_active":"2024-05-09T03:11:28.398563+0000","last_peered":"2024-05-09T03:11:28.398563+0000","last_clean":"2024-05-09T03:11:28.398563+0000","last_became_active":"2024-05-09T03:11:27.408723+0000","last_became_peered":"2024-05-09T03:11:27.408723+0000","last_unstale":"2024-05-09T03:11:28.398563+0000","last_undegraded":"2024-05-09T03:11:28.398563+0000","last_fullsized":"2024-05-09T03:11:28.398563+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:12.139044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421796+0000","last_change":"2024-05-09T03:11:26.397298+0000","last_active":"2024-05-09T03:11:28.421796+0000","last_peered":"2024-05-09T03:11:28.421796+0000","last_clean":"2024-05-09T03:11:28.421796+0000","last_became_active":"2024-05-09T03:11:26.397003+0000","last_became_peered":"2024-05-09T03:11:26.397003+0000","last_unstale":"2024-05-09T03:11:28.421796+0000","last_undegraded":"2024-05-09T03:11:28.421796+0000","last_fullsized":"2024-05-09T03:11:28.421796+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:27.852645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763204+0000","last_change":"2024-05-09T03:11:27.409367+0000","last_active":"2024-05-09T03:11:28.763204+0000","last_peered":"2024-05-09T03:11:28.763204+0000","last_clean":"2024-05-09T03:11:28.763204+0000","last_became_active":"2024-05-09T03:11:27.409015+0000","last_became_peered":"2024-05-09T03:11:27.409015+0000","last_unstale":"2024-05-09T03:11:28.763204+0000","last_undegraded":"2024-05-09T03:11:28.763204+0000","last_fullsized":"2024-05-09T03:11:28.763204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:39.037525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399033+0000","last_change":"2024-05-09T03:11:26.397547+0000","last_active":"2024-05-09T03:11:28.399033+0000","last_peered":"2024-05-09T03:11:28.399033+0000","last_clean":"2024-05-09T03:11:28.399033+0000","last_became_active":"2024-05-09T03:11:26.397248+0000","last_became_peered":"2024-05-09T03:11:26.397248+0000","last_unstale":"2024-05-09T03:11:28.399033+0000","last_undegraded":"2024-05-09T03:11:28.399033+0000","last_fullsized":"2024-05-09T03:11:28.399033+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:11:34.691002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763806+0000","last_change":"2024-05-09T03:11:27.418606+0000","last_active":"2024-05-09T03:11:28.763806+0000","last_peered":"2024-05-09T03:11:28.763806+0000","last_clean":"2024-05-09T03:11:28.763806+0000","last_became_active":"2024-05-09T03:11:27.417825+0000","last_became_peered":"2024-05-09T03:11:27.417825+0000","last_unstale":"2024-05-09T03:11:28.763806+0000","last_undegraded":"2024-05-09T03:11:28.763806+0000","last_fullsized":"2024-05-09T03:11:28.763806+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:02.195884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763362+0000","last_change":"2024-05-09T03:11:26.402930+0000","last_active":"2024-05-09T03:11:28.763362+0000","last_peered":"2024-05-09T03:11:28.763362+0000","last_clean":"2024-05-09T03:11:28.763362+0000","last_became_active":"2024-05-09T03:11:26.402595+0000","last_became_peered":"2024-05-09T03:11:26.402595+0000","last_unstale":"2024-05-09T03:11:28.763362+0000","last_undegraded":"2024-05-09T03:11:28.763362+0000","last_fullsized":"2024-05-09T03:11:28.763362+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398905+0000","last_change":"2024-05-09T03:11:27.411617+0000","last_active":"2024-05-09T03:11:28.398905+0000","last_peered":"2024-05-09T03:11:28.398905+0000","last_clean":"2024-05-09T03:11:28.398905+0000","last_became_active":"2024-05-09T03:11:27.411165+0000","last_became_peered":"2024-05-09T03:11:27.411165+0000","last_unstale":"2024-05-09T03:11:28.398905+0000","last_undegraded":"2024-05-09T03:11:28.398905+0000","last_fullsized":"2024-05-09T03:11:28.398905+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:39.006194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763890+0000","last_change":"2024-05-09T03:11:26.405390+0000","last_active":"2024-05-09T03:11:28.763890+0000","last_peered":"2024-05-09T03:11:28.763890+0000","last_clean":"2024-05-09T03:11:28.763890+0000","last_became_active":"2024-05-09T03:11:26.404946+0000","last_became_peered":"2024-05-09T03:11:26.404946+0000","last_unstale":"2024-05-09T03:11:28.763890+0000","last_undegraded":"2024-05-09T03:11:28.763890+0000","last_fullsized":"2024-05-09T03:11:28.763890+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:30.350454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399129+0000","last_change":"2024-05-09T03:11:27.412937+0000","last_active":"2024-05-09T03:11:28.399129+0000","last_peered":"2024-05-09T03:11:28.399129+0000","last_clean":"2024-05-09T03:11:28.399129+0000","last_became_active":"2024-05-09T03:11:27.412599+0000","last_became_peered":"2024-05-09T03:11:27.412599+0000","last_unstale":"2024-05-09T03:11:28.399129+0000","last_undegraded":"2024-05-09T03:11:28.399129+0000","last_fullsized":"2024-05-09T03:11:28.399129+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:37.733206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399521+0000","last_change":"2024-05-09T03:11:26.398230+0000","last_active":"2024-05-09T03:11:28.399521+0000","last_peered":"2024-05-09T03:11:28.399521+0000","last_clean":"2024-05-09T03:11:28.399521+0000","last_became_active":"2024-05-09T03:11:26.397938+0000","last_became_peered":"2024-05-09T03:11:26.397938+0000","last_unstale":"2024-05-09T03:11:28.399521+0000","last_undegraded":"2024-05-09T03:11:28.399521+0000","last_fullsized":"2024-05-09T03:11:28.399521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:36:09.738086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766544+0000","last_change":"2024-05-09T03:11:27.416047+0000","last_active":"2024-05-09T03:11:28.766544+0000","last_peered":"2024-05-09T03:11:28.766544+0000","last_clean":"2024-05-09T03:11:28.766544+0000","last_became_active":"2024-05-09T03:11:27.415330+0000","last_became_peered":"2024-05-09T03:11:27.415330+0000","last_unstale":"2024-05-09T03:11:28.766544+0000","last_undegraded":"2024-05-09T03:11:28.766544+0000","last_fullsized":"2024-05-09T03:11:28.766544+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:24.775162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763955+0000","last_change":"2024-05-09T03:11:26.407940+0000","last_active":"2024-05-09T03:11:28.763955+0000","last_peered":"2024-05-09T03:11:28.763955+0000","last_clean":"2024-05-09T03:11:28.763955+0000","last_became_active":"2024-05-09T03:11:26.407561+0000","last_became_peered":"2024-05-09T03:11:26.407561+0000","last_unstale":"2024-05-09T03:11:28.763955+0000","last_undegraded":"2024-05-09T03:11:28.763955+0000","last_fullsized":"2024-05-09T03:11:28.763955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:11:34.691002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421938+0000","last_change":"2024-05-09T03:11:27.406773+0000","last_active":"2024-05-09T03:11:28.421938+0000","last_peered":"2024-05-09T03:11:28.421938+0000","last_clean":"2024-05-09T03:11:28.421938+0000","last_became_active":"2024-05-09T03:11:27.406458+0000","last_became_peered":"2024-05-09T03:11:27.406458+0000","last_unstale":"2024-05-09T03:11:28.421938+0000","last_undegraded":"2024-05-09T03:11:28.421938+0000","last_fullsized":"2024-05-09T03:11:28.421938+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:00.963026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766581+0000","last_change":"2024-05-09T03:11:26.401361+0000","last_active":"2024-05-09T03:11:28.766581+0000","last_peered":"2024-05-09T03:11:28.766581+0000","last_clean":"2024-05-09T03:11:28.766581+0000","last_became_active":"2024-05-09T03:11:26.401038+0000","last_became_peered":"2024-05-09T03:11:26.401038+0000","last_unstale":"2024-05-09T03:11:28.766581+0000","last_undegraded":"2024-05-09T03:11:28.766581+0000","last_fullsized":"2024-05-09T03:11:28.766581+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:11.134274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766452+0000","last_change":"2024-05-09T03:11:27.416724+0000","last_active":"2024-05-09T03:11:28.766452+0000","last_peered":"2024-05-09T03:11:28.766452+0000","last_clean":"2024-05-09T03:11:28.766452+0000","last_became_active":"2024-05-09T03:11:27.416337+0000","last_became_peered":"2024-05-09T03:11:27.416337+0000","last_unstale":"2024-05-09T03:11:28.766452+0000","last_undegraded":"2024-05-09T03:11:28.766452+0000","last_fullsized":"2024-05-09T03:11:28.766452+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:02.195884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":20,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395496+0000","last_change":"2024-05-09T03:11:26.406530+0000","last_active":"2024-05-09T03:11:28.395496+0000","last_peered":"2024-05-09T03:11:28.395496+0000","last_clean":"2024-05-09T03:11:28.395496+0000","last_became_active":"2024-05-09T03:11:26.406173+0000","last_became_peered":"2024-05-09T03:11:26.406173+0000","last_unstale":"2024-05-09T03:11:28.395496+0000","last_undegraded":"2024-05-09T03:11:28.395496+0000","last_fullsized":"2024-05-09T03:11:28.395496+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:27.424687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399443+0000","last_change":"2024-05-09T03:11:27.413674+0000","last_active":"2024-05-09T03:11:28.399443+0000","last_peered":"2024-05-09T03:11:28.399443+0000","last_clean":"2024-05-09T03:11:28.399443+0000","last_became_active":"2024-05-09T03:11:27.412932+0000","last_became_peered":"2024-05-09T03:11:27.412932+0000","last_unstale":"2024-05-09T03:11:28.399443+0000","last_undegraded":"2024-05-09T03:11:28.399443+0000","last_fullsized":"2024-05-09T03:11:28.399443+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:50.359749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421913+0000","last_change":"2024-05-09T03:11:26.402684+0000","last_active":"2024-05-09T03:11:28.421913+0000","last_peered":"2024-05-09T03:11:28.421913+0000","last_clean":"2024-05-09T03:11:28.421913+0000","last_became_active":"2024-05-09T03:11:26.402325+0000","last_became_peered":"2024-05-09T03:11:26.402325+0000","last_unstale":"2024-05-09T03:11:28.421913+0000","last_undegraded":"2024-05-09T03:11:28.421913+0000","last_fullsized":"2024-05-09T03:11:28.421913+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:22.502342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763263+0000","last_change":"2024-05-09T03:11:27.406566+0000","last_active":"2024-05-09T03:11:28.763263+0000","last_peered":"2024-05-09T03:11:28.763263+0000","last_clean":"2024-05-09T03:11:28.763263+0000","last_became_active":"2024-05-09T03:11:27.406191+0000","last_became_peered":"2024-05-09T03:11:27.406191+0000","last_unstale":"2024-05-09T03:11:28.763263+0000","last_undegraded":"2024-05-09T03:11:28.763263+0000","last_fullsized":"2024-05-09T03:11:28.763263+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:37.733206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766188+0000","last_change":"2024-05-09T03:11:26.399378+0000","last_active":"2024-05-09T03:11:28.766188+0000","last_peered":"2024-05-09T03:11:28.766188+0000","last_clean":"2024-05-09T03:11:28.766188+0000","last_became_active":"2024-05-09T03:11:26.399091+0000","last_became_peered":"2024-05-09T03:11:26.399091+0000","last_unstale":"2024-05-09T03:11:28.766188+0000","last_undegraded":"2024-05-09T03:11:28.766188+0000","last_fullsized":"2024-05-09T03:11:28.766188+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421546+0000","last_change":"2024-05-09T03:11:27.409088+0000","last_active":"2024-05-09T03:11:28.421546+0000","last_peered":"2024-05-09T03:11:28.421546+0000","last_clean":"2024-05-09T03:11:28.421546+0000","last_became_active":"2024-05-09T03:11:27.406411+0000","last_became_peered":"2024-05-09T03:11:27.406411+0000","last_unstale":"2024-05-09T03:11:28.421546+0000","last_undegraded":"2024-05-09T03:11:28.421546+0000","last_fullsized":"2024-05-09T03:11:28.421546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:38:36.959072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421706+0000","last_change":"2024-05-09T03:11:26.398576+0000","last_active":"2024-05-09T03:11:28.421706+0000","last_peered":"2024-05-09T03:11:28.421706+0000","last_clean":"2024-05-09T03:11:28.421706+0000","last_became_active":"2024-05-09T03:11:26.398273+0000","last_became_peered":"2024-05-09T03:11:26.398273+0000","last_unstale":"2024-05-09T03:11:28.421706+0000","last_undegraded":"2024-05-09T03:11:28.421706+0000","last_fullsized":"2024-05-09T03:11:28.421706+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:38.032755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766335+0000","last_change":"2024-05-09T03:11:27.414943+0000","last_active":"2024-05-09T03:11:28.766335+0000","last_peered":"2024-05-09T03:11:28.766335+0000","last_clean":"2024-05-09T03:11:28.766335+0000","last_became_active":"2024-05-09T03:11:27.414642+0000","last_became_peered":"2024-05-09T03:11:27.414642+0000","last_unstale":"2024-05-09T03:11:28.766335+0000","last_undegraded":"2024-05-09T03:11:28.766335+0000","last_fullsized":"2024-05-09T03:11:28.766335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:19.073921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":22,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763905+0000","last_change":"2024-05-09T03:11:26.403484+0000","last_active":"2024-05-09T03:11:28.763905+0000","last_peered":"2024-05-09T03:11:28.763905+0000","last_clean":"2024-05-09T03:11:28.763905+0000","last_became_active":"2024-05-09T03:11:26.403147+0000","last_became_peered":"2024-05-09T03:11:26.403147+0000","last_unstale":"2024-05-09T03:11:28.763905+0000","last_undegraded":"2024-05-09T03:11:28.763905+0000","last_fullsized":"2024-05-09T03:11:28.763905+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395493+0000","last_change":"2024-05-09T03:11:27.411377+0000","last_active":"2024-05-09T03:11:28.395493+0000","last_peered":"2024-05-09T03:11:28.395493+0000","last_clean":"2024-05-09T03:11:28.395493+0000","last_became_active":"2024-05-09T03:11:27.411100+0000","last_became_peered":"2024-05-09T03:11:27.411100+0000","last_unstale":"2024-05-09T03:11:28.395493+0000","last_undegraded":"2024-05-09T03:11:28.395493+0000","last_fullsized":"2024-05-09T03:11:28.395493+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:44:24.711209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395528+0000","last_change":"2024-05-09T03:11:26.408975+0000","last_active":"2024-05-09T03:11:28.395528+0000","last_peered":"2024-05-09T03:11:28.395528+0000","last_clean":"2024-05-09T03:11:28.395528+0000","last_became_active":"2024-05-09T03:11:26.408602+0000","last_became_peered":"2024-05-09T03:11:26.408602+0000","last_unstale":"2024-05-09T03:11:28.395528+0000","last_undegraded":"2024-05-09T03:11:28.395528+0000","last_fullsized":"2024-05-09T03:11:28.395528+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:25:41.245927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763685+0000","last_change":"2024-05-09T03:11:27.407905+0000","last_active":"2024-05-09T03:11:28.763685+0000","last_peered":"2024-05-09T03:11:28.763685+0000","last_clean":"2024-05-09T03:11:28.763685+0000","last_became_active":"2024-05-09T03:11:27.407565+0000","last_became_peered":"2024-05-09T03:11:27.407565+0000","last_unstale":"2024-05-09T03:11:28.763685+0000","last_undegraded":"2024-05-09T03:11:28.763685+0000","last_fullsized":"2024-05-09T03:11:28.763685+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:19.073921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399325+0000","last_change":"2024-05-09T03:11:26.399038+0000","last_active":"2024-05-09T03:11:28.399325+0000","last_peered":"2024-05-09T03:11:28.399325+0000","last_clean":"2024-05-09T03:11:28.399325+0000","last_became_active":"2024-05-09T03:11:26.398728+0000","last_became_peered":"2024-05-09T03:11:26.398728+0000","last_unstale":"2024-05-09T03:11:28.399325+0000","last_undegraded":"2024-05-09T03:11:28.399325+0000","last_fullsized":"2024-05-09T03:11:28.399325+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:25.493953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399311+0000","last_change":"2024-05-09T03:11:27.418360+0000","last_active":"2024-05-09T03:11:28.399311+0000","last_peered":"2024-05-09T03:11:28.399311+0000","last_clean":"2024-05-09T03:11:28.399311+0000","last_became_active":"2024-05-09T03:11:27.417813+0000","last_became_peered":"2024-05-09T03:11:27.417813+0000","last_unstale":"2024-05-09T03:11:28.399311+0000","last_undegraded":"2024-05-09T03:11:28.399311+0000","last_fullsized":"2024-05-09T03:11:28.399311+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:13.465981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763746+0000","last_change":"2024-05-09T03:11:26.402375+0000","last_active":"2024-05-09T03:11:28.763746+0000","last_peered":"2024-05-09T03:11:28.763746+0000","last_clean":"2024-05-09T03:11:28.763746+0000","last_became_active":"2024-05-09T03:11:26.402033+0000","last_became_peered":"2024-05-09T03:11:26.402033+0000","last_unstale":"2024-05-09T03:11:28.763746+0000","last_undegraded":"2024-05-09T03:11:28.763746+0000","last_fullsized":"2024-05-09T03:11:28.763746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400060+0000","last_change":"2024-05-09T03:11:27.408673+0000","last_active":"2024-05-09T03:11:28.400060+0000","last_peered":"2024-05-09T03:11:28.400060+0000","last_clean":"2024-05-09T03:11:28.400060+0000","last_became_active":"2024-05-09T03:11:27.408398+0000","last_became_peered":"2024-05-09T03:11:27.408398+0000","last_unstale":"2024-05-09T03:11:28.400060+0000","last_undegraded":"2024-05-09T03:11:28.400060+0000","last_fullsized":"2024-05-09T03:11:28.400060+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:54.060589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.422052+0000","last_change":"2024-05-09T03:11:26.401461+0000","last_active":"2024-05-09T03:11:28.422052+0000","last_peered":"2024-05-09T03:11:28.422052+0000","last_clean":"2024-05-09T03:11:28.422052+0000","last_became_active":"2024-05-09T03:11:26.401148+0000","last_became_peered":"2024-05-09T03:11:26.401148+0000","last_unstale":"2024-05-09T03:11:28.422052+0000","last_undegraded":"2024-05-09T03:11:28.422052+0000","last_fullsized":"2024-05-09T03:11:28.422052+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:24:51.208260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421894+0000","last_change":"2024-05-09T03:11:27.407523+0000","last_active":"2024-05-09T03:11:28.421894+0000","last_peered":"2024-05-09T03:11:28.421894+0000","last_clean":"2024-05-09T03:11:28.421894+0000","last_became_active":"2024-05-09T03:11:27.407238+0000","last_became_peered":"2024-05-09T03:11:27.407238+0000","last_unstale":"2024-05-09T03:11:28.421894+0000","last_undegraded":"2024-05-09T03:11:28.421894+0000","last_fullsized":"2024-05-09T03:11:28.421894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:04.924505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.422054+0000","last_change":"2024-05-09T03:11:26.400381+0000","last_active":"2024-05-09T03:11:28.422054+0000","last_peered":"2024-05-09T03:11:28.422054+0000","last_clean":"2024-05-09T03:11:28.422054+0000","last_became_active":"2024-05-09T03:11:26.400059+0000","last_became_peered":"2024-05-09T03:11:26.400059+0000","last_unstale":"2024-05-09T03:11:28.422054+0000","last_undegraded":"2024-05-09T03:11:28.422054+0000","last_fullsized":"2024-05-09T03:11:28.422054+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:54.292440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.764149+0000","last_change":"2024-05-09T03:11:27.402555+0000","last_active":"2024-05-09T03:11:28.764149+0000","last_peered":"2024-05-09T03:11:28.764149+0000","last_clean":"2024-05-09T03:11:28.764149+0000","last_became_active":"2024-05-09T03:11:27.402229+0000","last_became_peered":"2024-05-09T03:11:27.402229+0000","last_unstale":"2024-05-09T03:11:28.764149+0000","last_undegraded":"2024-05-09T03:11:28.764149+0000","last_fullsized":"2024-05-09T03:11:28.764149+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:06.230758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763397+0000","last_change":"2024-05-09T03:11:26.403917+0000","last_active":"2024-05-09T03:11:28.763397+0000","last_peered":"2024-05-09T03:11:28.763397+0000","last_clean":"2024-05-09T03:11:28.763397+0000","last_became_active":"2024-05-09T03:11:26.403487+0000","last_became_peered":"2024-05-09T03:11:26.403487+0000","last_unstale":"2024-05-09T03:11:28.763397+0000","last_undegraded":"2024-05-09T03:11:28.763397+0000","last_fullsized":"2024-05-09T03:11:28.763397+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395221+0000","last_change":"2024-05-09T03:11:27.412961+0000","last_active":"2024-05-09T03:11:28.395221+0000","last_peered":"2024-05-09T03:11:28.395221+0000","last_clean":"2024-05-09T03:11:28.395221+0000","last_became_active":"2024-05-09T03:11:27.412631+0000","last_became_peered":"2024-05-09T03:11:27.412631+0000","last_unstale":"2024-05-09T03:11:28.395221+0000","last_undegraded":"2024-05-09T03:11:28.395221+0000","last_fullsized":"2024-05-09T03:11:28.395221+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:42:41.876936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763297+0000","last_change":"2024-05-09T03:11:26.406762+0000","last_active":"2024-05-09T03:11:28.763297+0000","last_peered":"2024-05-09T03:11:28.763297+0000","last_clean":"2024-05-09T03:11:28.763297+0000","last_became_active":"2024-05-09T03:11:26.406432+0000","last_became_peered":"2024-05-09T03:11:26.406432+0000","last_unstale":"2024-05-09T03:11:28.763297+0000","last_undegraded":"2024-05-09T03:11:28.763297+0000","last_fullsized":"2024-05-09T03:11:28.763297+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763119+0000","last_change":"2024-05-09T03:11:27.411420+0000","last_active":"2024-05-09T03:11:28.763119+0000","last_peered":"2024-05-09T03:11:28.763119+0000","last_clean":"2024-05-09T03:11:28.763119+0000","last_became_active":"2024-05-09T03:11:27.411089+0000","last_became_peered":"2024-05-09T03:11:27.411089+0000","last_unstale":"2024-05-09T03:11:28.763119+0000","last_undegraded":"2024-05-09T03:11:28.763119+0000","last_fullsized":"2024-05-09T03:11:28.763119+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:25.693725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.765995+0000","last_change":"2024-05-09T03:11:26.399861+0000","last_active":"2024-05-09T03:11:28.765995+0000","last_peered":"2024-05-09T03:11:28.765995+0000","last_clean":"2024-05-09T03:11:28.765995+0000","last_became_active":"2024-05-09T03:11:26.399383+0000","last_became_peered":"2024-05-09T03:11:26.399383+0000","last_unstale":"2024-05-09T03:11:28.765995+0000","last_undegraded":"2024-05-09T03:11:28.765995+0000","last_fullsized":"2024-05-09T03:11:28.765995+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:08.673574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400194+0000","last_change":"2024-05-09T03:11:27.412290+0000","last_active":"2024-05-09T03:11:28.400194+0000","last_peered":"2024-05-09T03:11:28.400194+0000","last_clean":"2024-05-09T03:11:28.400194+0000","last_became_active":"2024-05-09T03:11:27.411955+0000","last_became_peered":"2024-05-09T03:11:27.411955+0000","last_unstale":"2024-05-09T03:11:28.400194+0000","last_undegraded":"2024-05-09T03:11:28.400194+0000","last_fullsized":"2024-05-09T03:11:28.400194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:31.446242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"43'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421921+0000","last_change":"2024-05-09T03:11:26.398315+0000","last_active":"2024-05-09T03:11:28.421921+0000","last_peered":"2024-05-09T03:11:28.421921+0000","last_clean":"2024-05-09T03:11:28.421921+0000","last_became_active":"2024-05-09T03:11:26.397997+0000","last_became_peered":"2024-05-09T03:11:26.397997+0000","last_unstale":"2024-05-09T03:11:28.421921+0000","last_undegraded":"2024-05-09T03:11:28.421921+0000","last_fullsized":"2024-05-09T03:11:28.421921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:24.170842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763457+0000","last_change":"2024-05-09T03:11:27.409274+0000","last_active":"2024-05-09T03:11:28.763457+0000","last_peered":"2024-05-09T03:11:28.763457+0000","last_clean":"2024-05-09T03:11:28.763457+0000","last_became_active":"2024-05-09T03:11:27.408465+0000","last_became_peered":"2024-05-09T03:11:27.408465+0000","last_unstale":"2024-05-09T03:11:28.763457+0000","last_undegraded":"2024-05-09T03:11:28.763457+0000","last_fullsized":"2024-05-09T03:11:28.763457+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:28.857415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766587+0000","last_change":"2024-05-09T03:11:26.401410+0000","last_active":"2024-05-09T03:11:28.766587+0000","last_peered":"2024-05-09T03:11:28.766587+0000","last_clean":"2024-05-09T03:11:28.766587+0000","last_became_active":"2024-05-09T03:11:26.401111+0000","last_became_peered":"2024-05-09T03:11:26.401111+0000","last_unstale":"2024-05-09T03:11:28.766587+0000","last_undegraded":"2024-05-09T03:11:28.766587+0000","last_fullsized":"2024-05-09T03:11:28.766587+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:24.688956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766438+0000","last_change":"2024-05-09T03:11:27.411038+0000","last_active":"2024-05-09T03:11:28.766438+0000","last_peered":"2024-05-09T03:11:28.766438+0000","last_clean":"2024-05-09T03:11:28.766438+0000","last_became_active":"2024-05-09T03:11:27.410703+0000","last_became_peered":"2024-05-09T03:11:27.410703+0000","last_unstale":"2024-05-09T03:11:28.766438+0000","last_undegraded":"2024-05-09T03:11:28.766438+0000","last_fullsized":"2024-05-09T03:11:28.766438+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:42.288267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763032+0000","last_change":"2024-05-09T03:11:26.401170+0000","last_active":"2024-05-09T03:11:28.763032+0000","last_peered":"2024-05-09T03:11:28.763032+0000","last_clean":"2024-05-09T03:11:28.763032+0000","last_became_active":"2024-05-09T03:11:26.400832+0000","last_became_peered":"2024-05-09T03:11:26.400832+0000","last_unstale":"2024-05-09T03:11:28.763032+0000","last_undegraded":"2024-05-09T03:11:28.763032+0000","last_fullsized":"2024-05-09T03:11:28.763032+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:25.493953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.422198+0000","last_change":"2024-05-09T03:11:27.417138+0000","last_active":"2024-05-09T03:11:28.422198+0000","last_peered":"2024-05-09T03:11:28.422198+0000","last_clean":"2024-05-09T03:11:28.422198+0000","last_became_active":"2024-05-09T03:11:27.416597+0000","last_became_peered":"2024-05-09T03:11:27.416597+0000","last_unstale":"2024-05-09T03:11:28.422198+0000","last_undegraded":"2024-05-09T03:11:28.422198+0000","last_fullsized":"2024-05-09T03:11:28.422198+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:22.285522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766335+0000","last_change":"2024-05-09T03:11:26.400568+0000","last_active":"2024-05-09T03:11:28.766335+0000","last_peered":"2024-05-09T03:11:28.766335+0000","last_clean":"2024-05-09T03:11:28.766335+0000","last_became_active":"2024-05-09T03:11:26.400253+0000","last_became_peered":"2024-05-09T03:11:26.400253+0000","last_unstale":"2024-05-09T03:11:28.766335+0000","last_undegraded":"2024-05-09T03:11:28.766335+0000","last_fullsized":"2024-05-09T03:11:28.766335+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421813+0000","last_change":"2024-05-09T03:11:27.409547+0000","last_active":"2024-05-09T03:11:28.421813+0000","last_peered":"2024-05-09T03:11:28.421813+0000","last_clean":"2024-05-09T03:11:28.421813+0000","last_became_active":"2024-05-09T03:11:27.408810+0000","last_became_peered":"2024-05-09T03:11:27.408810+0000","last_unstale":"2024-05-09T03:11:28.421813+0000","last_undegraded":"2024-05-09T03:11:28.421813+0000","last_fullsized":"2024-05-09T03:11:28.421813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:11:42.434473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399161+0000","last_change":"2024-05-09T03:11:26.397676+0000","last_active":"2024-05-09T03:11:28.399161+0000","last_peered":"2024-05-09T03:11:28.399161+0000","last_clean":"2024-05-09T03:11:28.399161+0000","last_became_active":"2024-05-09T03:11:26.396811+0000","last_became_peered":"2024-05-09T03:11:26.396811+0000","last_unstale":"2024-05-09T03:11:28.399161+0000","last_undegraded":"2024-05-09T03:11:28.399161+0000","last_fullsized":"2024-05-09T03:11:28.399161+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763469+0000","last_change":"2024-05-09T03:11:27.407941+0000","last_active":"2024-05-09T03:11:28.763469+0000","last_peered":"2024-05-09T03:11:28.763469+0000","last_clean":"2024-05-09T03:11:28.763469+0000","last_became_active":"2024-05-09T03:11:27.407612+0000","last_became_peered":"2024-05-09T03:11:27.407612+0000","last_unstale":"2024-05-09T03:11:28.763469+0000","last_undegraded":"2024-05-09T03:11:28.763469+0000","last_fullsized":"2024-05-09T03:11:28.763469+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:28.857415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763221+0000","last_change":"2024-05-09T03:11:26.402849+0000","last_active":"2024-05-09T03:11:28.763221+0000","last_peered":"2024-05-09T03:11:28.763221+0000","last_clean":"2024-05-09T03:11:28.763221+0000","last_became_active":"2024-05-09T03:11:26.402533+0000","last_became_peered":"2024-05-09T03:11:26.402533+0000","last_unstale":"2024-05-09T03:11:28.763221+0000","last_undegraded":"2024-05-09T03:11:28.763221+0000","last_fullsized":"2024-05-09T03:11:28.763221+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:51.356924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398534+0000","last_change":"2024-05-09T03:11:27.420355+0000","last_active":"2024-05-09T03:11:28.398534+0000","last_peered":"2024-05-09T03:11:28.398534+0000","last_clean":"2024-05-09T03:11:28.398534+0000","last_became_active":"2024-05-09T03:11:27.420026+0000","last_became_peered":"2024-05-09T03:11:27.420026+0000","last_unstale":"2024-05-09T03:11:28.398534+0000","last_undegraded":"2024-05-09T03:11:28.398534+0000","last_fullsized":"2024-05-09T03:11:28.398534+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:31.446242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763983+0000","last_change":"2024-05-09T03:11:26.396367+0000","last_active":"2024-05-09T03:11:28.763983+0000","last_peered":"2024-05-09T03:11:28.763983+0000","last_clean":"2024-05-09T03:11:28.763983+0000","last_became_active":"2024-05-09T03:11:26.396006+0000","last_became_peered":"2024-05-09T03:11:26.396006+0000","last_unstale":"2024-05-09T03:11:28.763983+0000","last_undegraded":"2024-05-09T03:11:28.763983+0000","last_fullsized":"2024-05-09T03:11:28.763983+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763884+0000","last_change":"2024-05-09T03:11:27.406342+0000","last_active":"2024-05-09T03:11:28.763884+0000","last_peered":"2024-05-09T03:11:28.763884+0000","last_clean":"2024-05-09T03:11:28.763884+0000","last_became_active":"2024-05-09T03:11:27.406026+0000","last_became_peered":"2024-05-09T03:11:27.406026+0000","last_unstale":"2024-05-09T03:11:28.763884+0000","last_undegraded":"2024-05-09T03:11:28.763884+0000","last_fullsized":"2024-05-09T03:11:28.763884+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:24.775162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763896+0000","last_change":"2024-05-09T03:11:26.407676+0000","last_active":"2024-05-09T03:11:28.763896+0000","last_peered":"2024-05-09T03:11:28.763896+0000","last_clean":"2024-05-09T03:11:28.763896+0000","last_became_active":"2024-05-09T03:11:26.407284+0000","last_became_peered":"2024-05-09T03:11:26.407284+0000","last_unstale":"2024-05-09T03:11:28.763896+0000","last_undegraded":"2024-05-09T03:11:28.763896+0000","last_fullsized":"2024-05-09T03:11:28.763896+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":151,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766074+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:11:28.766074+0000","last_peered":"2024-05-09T03:11:28.766074+0000","last_clean":"2024-05-09T03:11:28.766074+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:11:28.766074+0000","last_undegraded":"2024-05-09T03:11:28.766074+0000","last_fullsized":"2024-05-09T03:11:28.766074+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1101824,"data_stored":794271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":512000,"data_stored":204485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":507904,"data_stored":203883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":516096,"data_stored":205425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31692,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340148,"statfs":{"total":96636764160,"available":96604311552,"internally_reserved":0,"allocated":516096,"data_stored":204451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":205447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":507904,"data_stored":203883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:11:33.380 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:11:33.379+0000 7f19c41a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:11:35.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:11:35.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:11:36.048 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:11:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.177+0000 7f9964e96640 1 Processor -- start 2024-05-09T03:11:36.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.177+0000 7f9964e96640 1 -- start start 2024-05-09T03:11:36.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.178+0000 7f9964e96640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9960164c70 0x7f99601615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.178+0000 7f9964e96640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f996015dfe0 0x7f99600d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.178+0000 7f9964e96640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9960160b20 con 0x7f99600d2480 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.178+0000 7f9964e96640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9960161150 con 0x7f996015dfe0 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.178+0000 7f9964e96640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f99600d15f0 con 0x7f9960164c70 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f996015dfe0 0x7f99600d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f996015dfe0 0x7f99600d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39386/0 (socket says 172.21.15.46:39386) 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 -- 172.21.15.46:0/3912511702 learned_addr learned my addr 172.21.15.46:0/3912511702 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995e59c640 1 --2- 172.21.15.46:0/3912511702 >> v2:172.21.15.88:3300/0 conn(0x7f9960164c70 0x7f99601615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 -- 172.21.15.46:0/3912511702 >> v2:172.21.15.88:3300/0 conn(0x7f9960164c70 msgr2=0x7f99601615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 --2- 172.21.15.46:0/3912511702 >> v2:172.21.15.88:3300/0 conn(0x7f9960164c70 0x7f99601615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 -- 172.21.15.46:0/3912511702 >> v1:172.21.15.88:6789/0 conn(0x7f99600d2480 legacy=0x7f99600d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995dd9b640 1 -- 172.21.15.46:0/3912511702 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99600d18d0 con 0x7f996015dfe0 2024-05-09T03:11:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.179+0000 7f995e59c640 1 --2- 172.21.15.46:0/3912511702 >> v2:172.21.15.88:3300/0 conn(0x7f9960164c70 0x7f99601615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:36.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.180+0000 7f995dd9b640 1 --2- 172.21.15.46:0/3912511702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996015dfe0 0x7f99600d0dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f99600b8900 tx=0x7f995402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef80bd441c13849 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.180+0000 7f995d59a640 1 -- 172.21.15.46:0/3912511702 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954037070 con 0x7f996015dfe0 2024-05-09T03:11:36.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.180+0000 7f995d59a640 1 -- 172.21.15.46:0/3912511702 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9954032440 con 0x7f996015dfe0 2024-05-09T03:11:36.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.180+0000 7f995d59a640 1 -- 172.21.15.46:0/3912511702 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954032810 con 0x7f996015dfe0 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 -- 172.21.15.46:0/3912511702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996015dfe0 msgr2=0x7f99600d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 --2- 172.21.15.46:0/3912511702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996015dfe0 0x7f99600d0dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f99600b8900 tx=0x7f995402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 -- 172.21.15.46:0/3912511702 shutdown_connections 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 --2- 172.21.15.46:0/3912511702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f996015dfe0 0x7f99600d0dc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 --2- 172.21.15.46:0/3912511702 >> v2:172.21.15.88:3300/0 conn(0x7f9960164c70 0x7f99601615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 -- 172.21.15.46:0/3912511702 >> 172.21.15.46:0/3912511702 conn(0x7f9960073db0 msgr2=0x7f99600741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 -- 172.21.15.46:0/3912511702 shutdown_connections 2024-05-09T03:11:36.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.181+0000 7f9964e96640 1 -- 172.21.15.46:0/3912511702 wait complete. 2024-05-09T03:11:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.183+0000 7f9964e96640 1 Processor -- start 2024-05-09T03:11:36.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.183+0000 7f9964e96640 1 -- start start 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f9964e96640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99600d2480 0x7f9960163360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f9964e96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f996015dfe0 0x7f9960157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f9964e96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 0x7f9960156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f9964e96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f996015e3f0 con 0x7f9960164c70 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f9964e96640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f99600d1300 con 0x7f996015dfe0 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f9964e96640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f99600d45c0 con 0x7f99600d2480 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f995ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 0x7f9960156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f995ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 0x7f9960156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39396/0 (socket says 172.21.15.46:39396) 2024-05-09T03:11:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f995ed9d640 1 -- 172.21.15.46:0/1868073025 learned_addr learned my addr 172.21.15.46:0/1868073025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.184+0000 7f995e59c640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99600d2480 0x7f9960163360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995dd9b640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f996015dfe0 0x7f9960157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995ed9d640 1 -- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99600d2480 msgr2=0x7f9960163360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995ed9d640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99600d2480 0x7f9960163360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995ed9d640 1 -- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f996015dfe0 msgr2=0x7f9960157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995ed9d640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f996015dfe0 0x7f9960157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995ed9d640 1 -- 172.21.15.46:0/1868073025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9960157030 con 0x7f9960164c70 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995dd9b640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f996015dfe0 0x7f9960157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f995ed9d640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 0x7f9960156690 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f995800c700 tx=0x7f995800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9958003b70 con 0x7f9960164c70 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9958003d10 con 0x7f9960164c70 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9960157310 con 0x7f9960164c70 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995800aa50 con 0x7f9960164c70 2024-05-09T03:11:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.185+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9960150910 con 0x7f9960164c70 2024-05-09T03:11:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.186+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f996008cfd0 con 0x7f9960164c70 2024-05-09T03:11:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.187+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f995800abf0 con 0x7f9960164c70 2024-05-09T03:11:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.187+0000 7f994effd640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99400762d0 0x7f99400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.187+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f9958097470 con 0x7f9960164c70 2024-05-09T03:11:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.188+0000 7f995e59c640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99400762d0 0x7f99400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.188+0000 7f995e59c640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99400762d0 0x7f99400787e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9950006fd0 tx=0x7f9950008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.192+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f996008cfd0 con 0x7f9960164c70 2024-05-09T03:11:36.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.378+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f99601644a0 con 0x7f99400762d0 2024-05-09T03:11:36.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.385+0000 7f994effd640 1 -- 172.21.15.46:0/1868073025 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251949 (secure 0 0 0) 0x7f99601644a0 con 0x7f99400762d0 2024-05-09T03:11:36.388 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:36.389 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.387+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99400762d0 msgr2=0x7f99400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.387+0000 7f9964e96640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99400762d0 0x7f99400787e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9950006fd0 tx=0x7f9950008040 comp rx=0 tx=0).stop 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.387+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 msgr2=0x7f9960156690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.387+0000 7f9964e96640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 0x7f9960156690 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f995800c700 tx=0x7f995800cbc0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 shutdown_connections 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99400762d0 0x7f99400787e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9960164c70 0x7f9960156690 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f996015dfe0 0x7f9960157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 --2- 172.21.15.46:0/1868073025 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f99600d2480 0x7f9960163360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 >> 172.21.15.46:0/1868073025 conn(0x7f9960073db0 msgr2=0x7f99600b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 shutdown_connections 2024-05-09T03:11:36.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.388+0000 7f9964e96640 1 -- 172.21.15.46:0/1868073025 wait complete. 2024-05-09T03:11:36.400 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-05-09T03:11:34.521848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":179,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238008,"kb_used_data":5160,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736712,"statfs":{"total":773094113280,"available":772850393088,"internally_reserved":0,"allocated":5283840,"data_stored":2816060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002387"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766498+0000","last_change":"2024-05-09T03:11:27.416315+0000","last_active":"2024-05-09T03:11:28.766498+0000","last_peered":"2024-05-09T03:11:28.766498+0000","last_clean":"2024-05-09T03:11:28.766498+0000","last_became_active":"2024-05-09T03:11:27.415726+0000","last_became_peered":"2024-05-09T03:11:27.415726+0000","last_unstale":"2024-05-09T03:11:28.766498+0000","last_undegraded":"2024-05-09T03:11:28.766498+0000","last_fullsized":"2024-05-09T03:11:28.766498+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:42:24.049436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763776+0000","last_change":"2024-05-09T03:11:27.414806+0000","last_active":"2024-05-09T03:11:28.763776+0000","last_peered":"2024-05-09T03:11:28.763776+0000","last_clean":"2024-05-09T03:11:28.763776+0000","last_became_active":"2024-05-09T03:11:27.414490+0000","last_became_peered":"2024-05-09T03:11:27.414490+0000","last_unstale":"2024-05-09T03:11:28.763776+0000","last_undegraded":"2024-05-09T03:11:28.763776+0000","last_fullsized":"2024-05-09T03:11:28.763776+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:30.226498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398863+0000","last_change":"2024-05-09T03:11:27.407445+0000","last_active":"2024-05-09T03:11:28.398863+0000","last_peered":"2024-05-09T03:11:28.398863+0000","last_clean":"2024-05-09T03:11:28.398863+0000","last_became_active":"2024-05-09T03:11:27.407132+0000","last_became_peered":"2024-05-09T03:11:27.407132+0000","last_unstale":"2024-05-09T03:11:28.398863+0000","last_undegraded":"2024-05-09T03:11:28.398863+0000","last_fullsized":"2024-05-09T03:11:28.398863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:25.693725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763247+0000","last_change":"2024-05-09T03:11:27.420521+0000","last_active":"2024-05-09T03:11:28.763247+0000","last_peered":"2024-05-09T03:11:28.763247+0000","last_clean":"2024-05-09T03:11:28.763247+0000","last_became_active":"2024-05-09T03:11:27.420134+0000","last_became_peered":"2024-05-09T03:11:27.420134+0000","last_unstale":"2024-05-09T03:11:28.763247+0000","last_undegraded":"2024-05-09T03:11:28.763247+0000","last_fullsized":"2024-05-09T03:11:28.763247+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:25.175611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763575+0000","last_change":"2024-05-09T03:11:27.414109+0000","last_active":"2024-05-09T03:11:28.763575+0000","last_peered":"2024-05-09T03:11:28.763575+0000","last_clean":"2024-05-09T03:11:28.763575+0000","last_became_active":"2024-05-09T03:11:27.413743+0000","last_became_peered":"2024-05-09T03:11:27.413743+0000","last_unstale":"2024-05-09T03:11:28.763575+0000","last_undegraded":"2024-05-09T03:11:28.763575+0000","last_fullsized":"2024-05-09T03:11:28.763575+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:23.507112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421588+0000","last_change":"2024-05-09T03:11:27.416637+0000","last_active":"2024-05-09T03:11:28.421588+0000","last_peered":"2024-05-09T03:11:28.421588+0000","last_clean":"2024-05-09T03:11:28.421588+0000","last_became_active":"2024-05-09T03:11:27.416223+0000","last_became_peered":"2024-05-09T03:11:27.416223+0000","last_unstale":"2024-05-09T03:11:28.421588+0000","last_undegraded":"2024-05-09T03:11:28.421588+0000","last_fullsized":"2024-05-09T03:11:28.421588+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:18.454432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766185+0000","last_change":"2024-05-09T03:11:27.417640+0000","last_active":"2024-05-09T03:11:28.766185+0000","last_peered":"2024-05-09T03:11:28.766185+0000","last_clean":"2024-05-09T03:11:28.766185+0000","last_became_active":"2024-05-09T03:11:27.417188+0000","last_became_peered":"2024-05-09T03:11:27.417188+0000","last_unstale":"2024-05-09T03:11:28.766185+0000","last_undegraded":"2024-05-09T03:11:28.766185+0000","last_fullsized":"2024-05-09T03:11:28.766185+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:30.226498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763468+0000","last_change":"2024-05-09T03:11:27.418976+0000","last_active":"2024-05-09T03:11:28.763468+0000","last_peered":"2024-05-09T03:11:28.763468+0000","last_clean":"2024-05-09T03:11:28.763468+0000","last_became_active":"2024-05-09T03:11:27.418232+0000","last_became_peered":"2024-05-09T03:11:27.418232+0000","last_unstale":"2024-05-09T03:11:28.763468+0000","last_undegraded":"2024-05-09T03:11:28.763468+0000","last_fullsized":"2024-05-09T03:11:28.763468+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:23.507112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763556+0000","last_change":"2024-05-09T03:11:27.410452+0000","last_active":"2024-05-09T03:11:28.763556+0000","last_peered":"2024-05-09T03:11:28.763556+0000","last_clean":"2024-05-09T03:11:28.763556+0000","last_became_active":"2024-05-09T03:11:27.410080+0000","last_became_peered":"2024-05-09T03:11:27.410080+0000","last_unstale":"2024-05-09T03:11:28.763556+0000","last_undegraded":"2024-05-09T03:11:28.763556+0000","last_fullsized":"2024-05-09T03:11:28.763556+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:24:52.213030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763838+0000","last_change":"2024-05-09T03:11:27.418629+0000","last_active":"2024-05-09T03:11:28.763838+0000","last_peered":"2024-05-09T03:11:28.763838+0000","last_clean":"2024-05-09T03:11:28.763838+0000","last_became_active":"2024-05-09T03:11:27.418191+0000","last_became_peered":"2024-05-09T03:11:27.418191+0000","last_unstale":"2024-05-09T03:11:28.763838+0000","last_undegraded":"2024-05-09T03:11:28.763838+0000","last_fullsized":"2024-05-09T03:11:28.763838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:42:24.049436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763516+0000","last_change":"2024-05-09T03:11:27.410458+0000","last_active":"2024-05-09T03:11:28.763516+0000","last_peered":"2024-05-09T03:11:28.763516+0000","last_clean":"2024-05-09T03:11:28.763516+0000","last_became_active":"2024-05-09T03:11:27.410113+0000","last_became_peered":"2024-05-09T03:11:27.410113+0000","last_unstale":"2024-05-09T03:11:28.763516+0000","last_undegraded":"2024-05-09T03:11:28.763516+0000","last_fullsized":"2024-05-09T03:11:28.763516+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:55.297209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763640+0000","last_change":"2024-05-09T03:11:27.406120+0000","last_active":"2024-05-09T03:11:28.763640+0000","last_peered":"2024-05-09T03:11:28.763640+0000","last_clean":"2024-05-09T03:11:28.763640+0000","last_became_active":"2024-05-09T03:11:27.405636+0000","last_became_peered":"2024-05-09T03:11:27.405636+0000","last_unstale":"2024-05-09T03:11:28.763640+0000","last_undegraded":"2024-05-09T03:11:28.763640+0000","last_fullsized":"2024-05-09T03:11:28.763640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:55.297209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395368+0000","last_change":"2024-05-09T03:11:27.413654+0000","last_active":"2024-05-09T03:11:28.395368+0000","last_peered":"2024-05-09T03:11:28.395368+0000","last_clean":"2024-05-09T03:11:28.395368+0000","last_became_active":"2024-05-09T03:11:27.413316+0000","last_became_peered":"2024-05-09T03:11:27.413316+0000","last_unstale":"2024-05-09T03:11:28.395368+0000","last_undegraded":"2024-05-09T03:11:28.395368+0000","last_fullsized":"2024-05-09T03:11:28.395368+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:04:16.199599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763673+0000","last_change":"2024-05-09T03:11:27.415386+0000","last_active":"2024-05-09T03:11:28.763673+0000","last_peered":"2024-05-09T03:11:28.763673+0000","last_clean":"2024-05-09T03:11:28.763673+0000","last_became_active":"2024-05-09T03:11:27.412137+0000","last_became_peered":"2024-05-09T03:11:27.412137+0000","last_unstale":"2024-05-09T03:11:28.763673+0000","last_undegraded":"2024-05-09T03:11:28.763673+0000","last_fullsized":"2024-05-09T03:11:28.763673+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:30.226498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421546+0000","last_change":"2024-05-09T03:11:27.416475+0000","last_active":"2024-05-09T03:11:28.421546+0000","last_peered":"2024-05-09T03:11:28.421546+0000","last_clean":"2024-05-09T03:11:28.421546+0000","last_became_active":"2024-05-09T03:11:27.416157+0000","last_became_peered":"2024-05-09T03:11:27.416157+0000","last_unstale":"2024-05-09T03:11:28.421546+0000","last_undegraded":"2024-05-09T03:11:28.421546+0000","last_fullsized":"2024-05-09T03:11:28.421546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:10.279184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398865+0000","last_change":"2024-05-09T03:11:27.421834+0000","last_active":"2024-05-09T03:11:28.398865+0000","last_peered":"2024-05-09T03:11:28.398865+0000","last_clean":"2024-05-09T03:11:28.398865+0000","last_became_active":"2024-05-09T03:11:27.421442+0000","last_became_peered":"2024-05-09T03:11:27.421442+0000","last_unstale":"2024-05-09T03:11:28.398865+0000","last_undegraded":"2024-05-09T03:11:28.398865+0000","last_fullsized":"2024-05-09T03:11:28.398865+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:26.498723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399435+0000","last_change":"2024-05-09T03:11:27.413670+0000","last_active":"2024-05-09T03:11:28.399435+0000","last_peered":"2024-05-09T03:11:28.399435+0000","last_clean":"2024-05-09T03:11:28.399435+0000","last_became_active":"2024-05-09T03:11:27.413362+0000","last_became_peered":"2024-05-09T03:11:27.413362+0000","last_unstale":"2024-05-09T03:11:28.399435+0000","last_undegraded":"2024-05-09T03:11:28.399435+0000","last_fullsized":"2024-05-09T03:11:28.399435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:28.857415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398949+0000","last_change":"2024-05-09T03:11:27.415156+0000","last_active":"2024-05-09T03:11:28.398949+0000","last_peered":"2024-05-09T03:11:28.398949+0000","last_clean":"2024-05-09T03:11:28.398949+0000","last_became_active":"2024-05-09T03:11:27.414821+0000","last_became_peered":"2024-05-09T03:11:27.414821+0000","last_unstale":"2024-05-09T03:11:28.398949+0000","last_undegraded":"2024-05-09T03:11:28.398949+0000","last_fullsized":"2024-05-09T03:11:28.398949+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:39.037525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400154+0000","last_change":"2024-05-09T03:11:27.412965+0000","last_active":"2024-05-09T03:11:28.400154+0000","last_peered":"2024-05-09T03:11:28.400154+0000","last_clean":"2024-05-09T03:11:28.400154+0000","last_became_active":"2024-05-09T03:11:27.412660+0000","last_became_peered":"2024-05-09T03:11:27.412660+0000","last_unstale":"2024-05-09T03:11:28.400154+0000","last_undegraded":"2024-05-09T03:11:28.400154+0000","last_fullsized":"2024-05-09T03:11:28.400154+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:06.230758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395530+0000","last_change":"2024-05-09T03:11:27.422078+0000","last_active":"2024-05-09T03:11:28.395530+0000","last_peered":"2024-05-09T03:11:28.395530+0000","last_clean":"2024-05-09T03:11:28.395530+0000","last_became_active":"2024-05-09T03:11:27.421684+0000","last_became_peered":"2024-05-09T03:11:27.421684+0000","last_unstale":"2024-05-09T03:11:28.395530+0000","last_undegraded":"2024-05-09T03:11:28.395530+0000","last_fullsized":"2024-05-09T03:11:28.395530+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:49.735304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421684+0000","last_change":"2024-05-09T03:11:27.415241+0000","last_active":"2024-05-09T03:11:28.421684+0000","last_peered":"2024-05-09T03:11:28.421684+0000","last_clean":"2024-05-09T03:11:28.421684+0000","last_became_active":"2024-05-09T03:11:27.414898+0000","last_became_peered":"2024-05-09T03:11:27.414898+0000","last_unstale":"2024-05-09T03:11:28.421684+0000","last_undegraded":"2024-05-09T03:11:28.421684+0000","last_fullsized":"2024-05-09T03:11:28.421684+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:57.365456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763756+0000","last_change":"2024-05-09T03:11:27.420886+0000","last_active":"2024-05-09T03:11:28.763756+0000","last_peered":"2024-05-09T03:11:28.763756+0000","last_clean":"2024-05-09T03:11:28.763756+0000","last_became_active":"2024-05-09T03:11:27.420585+0000","last_became_peered":"2024-05-09T03:11:27.420585+0000","last_unstale":"2024-05-09T03:11:28.763756+0000","last_undegraded":"2024-05-09T03:11:28.763756+0000","last_fullsized":"2024-05-09T03:11:28.763756+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:42.288267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400015+0000","last_change":"2024-05-09T03:11:27.410820+0000","last_active":"2024-05-09T03:11:28.400015+0000","last_peered":"2024-05-09T03:11:28.400015+0000","last_clean":"2024-05-09T03:11:28.400015+0000","last_became_active":"2024-05-09T03:11:27.409738+0000","last_became_peered":"2024-05-09T03:11:27.409738+0000","last_unstale":"2024-05-09T03:11:28.400015+0000","last_undegraded":"2024-05-09T03:11:28.400015+0000","last_fullsized":"2024-05-09T03:11:28.400015+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:29.977279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398901+0000","last_change":"2024-05-09T03:11:27.410340+0000","last_active":"2024-05-09T03:11:28.398901+0000","last_peered":"2024-05-09T03:11:28.398901+0000","last_clean":"2024-05-09T03:11:28.398901+0000","last_became_active":"2024-05-09T03:11:27.409999+0000","last_became_peered":"2024-05-09T03:11:27.409999+0000","last_unstale":"2024-05-09T03:11:28.398901+0000","last_undegraded":"2024-05-09T03:11:28.398901+0000","last_fullsized":"2024-05-09T03:11:28.398901+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:36:10.742856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400286+0000","last_change":"2024-05-09T03:11:27.416485+0000","last_active":"2024-05-09T03:11:28.400286+0000","last_peered":"2024-05-09T03:11:28.400286+0000","last_clean":"2024-05-09T03:11:28.400286+0000","last_became_active":"2024-05-09T03:11:27.415236+0000","last_became_peered":"2024-05-09T03:11:27.415236+0000","last_unstale":"2024-05-09T03:11:28.400286+0000","last_undegraded":"2024-05-09T03:11:28.400286+0000","last_fullsized":"2024-05-09T03:11:28.400286+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:12.139044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395263+0000","last_change":"2024-05-09T03:11:27.422564+0000","last_active":"2024-05-09T03:11:28.395263+0000","last_peered":"2024-05-09T03:11:28.395263+0000","last_clean":"2024-05-09T03:11:28.395263+0000","last_became_active":"2024-05-09T03:11:27.422200+0000","last_became_peered":"2024-05-09T03:11:27.422200+0000","last_unstale":"2024-05-09T03:11:28.395263+0000","last_undegraded":"2024-05-09T03:11:28.395263+0000","last_fullsized":"2024-05-09T03:11:28.395263+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:38.419189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399215+0000","last_change":"2024-05-09T03:11:27.405881+0000","last_active":"2024-05-09T03:11:28.399215+0000","last_peered":"2024-05-09T03:11:28.399215+0000","last_clean":"2024-05-09T03:11:28.399215+0000","last_became_active":"2024-05-09T03:11:27.404751+0000","last_became_peered":"2024-05-09T03:11:27.404751+0000","last_unstale":"2024-05-09T03:11:28.399215+0000","last_undegraded":"2024-05-09T03:11:28.399215+0000","last_fullsized":"2024-05-09T03:11:28.399215+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:25.175611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763751+0000","last_change":"2024-05-09T03:11:27.405410+0000","last_active":"2024-05-09T03:11:28.763751+0000","last_peered":"2024-05-09T03:11:28.763751+0000","last_clean":"2024-05-09T03:11:28.763751+0000","last_became_active":"2024-05-09T03:11:27.404135+0000","last_became_peered":"2024-05-09T03:11:27.404135+0000","last_unstale":"2024-05-09T03:11:28.763751+0000","last_undegraded":"2024-05-09T03:11:28.763751+0000","last_fullsized":"2024-05-09T03:11:28.763751+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:24:52.213030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399815+0000","last_change":"2024-05-09T03:11:27.407820+0000","last_active":"2024-05-09T03:11:28.399815+0000","last_peered":"2024-05-09T03:11:28.399815+0000","last_clean":"2024-05-09T03:11:28.399815+0000","last_became_active":"2024-05-09T03:11:27.407497+0000","last_became_peered":"2024-05-09T03:11:27.407497+0000","last_unstale":"2024-05-09T03:11:28.399815+0000","last_undegraded":"2024-05-09T03:11:28.399815+0000","last_fullsized":"2024-05-09T03:11:28.399815+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:41.718657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398572+0000","last_change":"2024-05-09T03:11:27.410973+0000","last_active":"2024-05-09T03:11:28.398572+0000","last_peered":"2024-05-09T03:11:28.398572+0000","last_clean":"2024-05-09T03:11:28.398572+0000","last_became_active":"2024-05-09T03:11:27.410693+0000","last_became_peered":"2024-05-09T03:11:27.410693+0000","last_unstale":"2024-05-09T03:11:28.398572+0000","last_undegraded":"2024-05-09T03:11:28.398572+0000","last_fullsized":"2024-05-09T03:11:28.398572+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:09.678343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421631+0000","last_change":"2024-05-09T03:11:27.407906+0000","last_active":"2024-05-09T03:11:28.421631+0000","last_peered":"2024-05-09T03:11:28.421631+0000","last_clean":"2024-05-09T03:11:28.421631+0000","last_became_active":"2024-05-09T03:11:27.404754+0000","last_became_peered":"2024-05-09T03:11:27.404754+0000","last_unstale":"2024-05-09T03:11:28.421631+0000","last_undegraded":"2024-05-09T03:11:28.421631+0000","last_fullsized":"2024-05-09T03:11:28.421631+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:49.585557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395498+0000","last_change":"2024-05-09T03:11:27.416539+0000","last_active":"2024-05-09T03:11:28.395498+0000","last_peered":"2024-05-09T03:11:28.395498+0000","last_clean":"2024-05-09T03:11:28.395498+0000","last_became_active":"2024-05-09T03:11:27.416229+0000","last_became_peered":"2024-05-09T03:11:27.416229+0000","last_unstale":"2024-05-09T03:11:28.395498+0000","last_undegraded":"2024-05-09T03:11:28.395498+0000","last_fullsized":"2024-05-09T03:11:28.395498+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:43.943712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763348+0000","last_change":"2024-05-09T03:11:27.407857+0000","last_active":"2024-05-09T03:11:28.763348+0000","last_peered":"2024-05-09T03:11:28.763348+0000","last_clean":"2024-05-09T03:11:28.763348+0000","last_became_active":"2024-05-09T03:11:27.407515+0000","last_became_peered":"2024-05-09T03:11:27.407515+0000","last_unstale":"2024-05-09T03:11:28.763348+0000","last_undegraded":"2024-05-09T03:11:28.763348+0000","last_fullsized":"2024-05-09T03:11:28.763348+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:04:41.718657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421983+0000","last_change":"2024-05-09T03:11:26.395704+0000","last_active":"2024-05-09T03:11:28.421983+0000","last_peered":"2024-05-09T03:11:28.421983+0000","last_clean":"2024-05-09T03:11:28.421983+0000","last_became_active":"2024-05-09T03:11:26.394437+0000","last_became_peered":"2024-05-09T03:11:26.394437+0000","last_unstale":"2024-05-09T03:11:28.421983+0000","last_undegraded":"2024-05-09T03:11:28.421983+0000","last_fullsized":"2024-05-09T03:11:28.421983+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:36.728436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400128+0000","last_change":"2024-05-09T03:11:27.406857+0000","last_active":"2024-05-09T03:11:28.400128+0000","last_peered":"2024-05-09T03:11:28.400128+0000","last_clean":"2024-05-09T03:11:28.400128+0000","last_became_active":"2024-05-09T03:11:27.405935+0000","last_became_peered":"2024-05-09T03:11:27.405935+0000","last_unstale":"2024-05-09T03:11:28.400128+0000","last_undegraded":"2024-05-09T03:11:28.400128+0000","last_fullsized":"2024-05-09T03:11:28.400128+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:43.768405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399164+0000","last_change":"2024-05-09T03:11:26.395710+0000","last_active":"2024-05-09T03:11:28.399164+0000","last_peered":"2024-05-09T03:11:28.399164+0000","last_clean":"2024-05-09T03:11:28.399164+0000","last_became_active":"2024-05-09T03:11:26.395376+0000","last_became_peered":"2024-05-09T03:11:26.395376+0000","last_unstale":"2024-05-09T03:11:28.399164+0000","last_undegraded":"2024-05-09T03:11:28.399164+0000","last_fullsized":"2024-05-09T03:11:28.399164+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:51.356924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763887+0000","last_change":"2024-05-09T03:11:27.408971+0000","last_active":"2024-05-09T03:11:28.763887+0000","last_peered":"2024-05-09T03:11:28.763887+0000","last_clean":"2024-05-09T03:11:28.763887+0000","last_became_active":"2024-05-09T03:11:27.408422+0000","last_became_peered":"2024-05-09T03:11:27.408422+0000","last_unstale":"2024-05-09T03:11:28.763887+0000","last_undegraded":"2024-05-09T03:11:28.763887+0000","last_fullsized":"2024-05-09T03:11:28.763887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:19.073921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763688+0000","last_change":"2024-05-09T03:11:26.400247+0000","last_active":"2024-05-09T03:11:28.763688+0000","last_peered":"2024-05-09T03:11:28.763688+0000","last_clean":"2024-05-09T03:11:28.763688+0000","last_became_active":"2024-05-09T03:11:26.399921+0000","last_became_peered":"2024-05-09T03:11:26.399921+0000","last_unstale":"2024-05-09T03:11:28.763688+0000","last_undegraded":"2024-05-09T03:11:28.763688+0000","last_fullsized":"2024-05-09T03:11:28.763688+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:36:09.738086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763576+0000","last_change":"2024-05-09T03:11:27.415016+0000","last_active":"2024-05-09T03:11:28.763576+0000","last_peered":"2024-05-09T03:11:28.763576+0000","last_clean":"2024-05-09T03:11:28.763576+0000","last_became_active":"2024-05-09T03:11:27.414667+0000","last_became_peered":"2024-05-09T03:11:27.414667+0000","last_unstale":"2024-05-09T03:11:28.763576+0000","last_undegraded":"2024-05-09T03:11:28.763576+0000","last_fullsized":"2024-05-09T03:11:28.763576+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:02:29.977279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766350+0000","last_change":"2024-05-09T03:11:26.400502+0000","last_active":"2024-05-09T03:11:28.766350+0000","last_peered":"2024-05-09T03:11:28.766350+0000","last_clean":"2024-05-09T03:11:28.766350+0000","last_became_active":"2024-05-09T03:11:26.400182+0000","last_became_peered":"2024-05-09T03:11:26.400182+0000","last_unstale":"2024-05-09T03:11:28.766350+0000","last_undegraded":"2024-05-09T03:11:28.766350+0000","last_fullsized":"2024-05-09T03:11:28.766350+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398563+0000","last_change":"2024-05-09T03:11:27.409079+0000","last_active":"2024-05-09T03:11:28.398563+0000","last_peered":"2024-05-09T03:11:28.398563+0000","last_clean":"2024-05-09T03:11:28.398563+0000","last_became_active":"2024-05-09T03:11:27.408723+0000","last_became_peered":"2024-05-09T03:11:27.408723+0000","last_unstale":"2024-05-09T03:11:28.398563+0000","last_undegraded":"2024-05-09T03:11:28.398563+0000","last_fullsized":"2024-05-09T03:11:28.398563+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:12.139044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421796+0000","last_change":"2024-05-09T03:11:26.397298+0000","last_active":"2024-05-09T03:11:28.421796+0000","last_peered":"2024-05-09T03:11:28.421796+0000","last_clean":"2024-05-09T03:11:28.421796+0000","last_became_active":"2024-05-09T03:11:26.397003+0000","last_became_peered":"2024-05-09T03:11:26.397003+0000","last_unstale":"2024-05-09T03:11:28.421796+0000","last_undegraded":"2024-05-09T03:11:28.421796+0000","last_fullsized":"2024-05-09T03:11:28.421796+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:27.852645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763204+0000","last_change":"2024-05-09T03:11:27.409367+0000","last_active":"2024-05-09T03:11:28.763204+0000","last_peered":"2024-05-09T03:11:28.763204+0000","last_clean":"2024-05-09T03:11:28.763204+0000","last_became_active":"2024-05-09T03:11:27.409015+0000","last_became_peered":"2024-05-09T03:11:27.409015+0000","last_unstale":"2024-05-09T03:11:28.763204+0000","last_undegraded":"2024-05-09T03:11:28.763204+0000","last_fullsized":"2024-05-09T03:11:28.763204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:39.037525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399033+0000","last_change":"2024-05-09T03:11:26.397547+0000","last_active":"2024-05-09T03:11:28.399033+0000","last_peered":"2024-05-09T03:11:28.399033+0000","last_clean":"2024-05-09T03:11:28.399033+0000","last_became_active":"2024-05-09T03:11:26.397248+0000","last_became_peered":"2024-05-09T03:11:26.397248+0000","last_unstale":"2024-05-09T03:11:28.399033+0000","last_undegraded":"2024-05-09T03:11:28.399033+0000","last_fullsized":"2024-05-09T03:11:28.399033+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:11:34.691002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763806+0000","last_change":"2024-05-09T03:11:27.418606+0000","last_active":"2024-05-09T03:11:28.763806+0000","last_peered":"2024-05-09T03:11:28.763806+0000","last_clean":"2024-05-09T03:11:28.763806+0000","last_became_active":"2024-05-09T03:11:27.417825+0000","last_became_peered":"2024-05-09T03:11:27.417825+0000","last_unstale":"2024-05-09T03:11:28.763806+0000","last_undegraded":"2024-05-09T03:11:28.763806+0000","last_fullsized":"2024-05-09T03:11:28.763806+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:02.195884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763362+0000","last_change":"2024-05-09T03:11:26.402930+0000","last_active":"2024-05-09T03:11:28.763362+0000","last_peered":"2024-05-09T03:11:28.763362+0000","last_clean":"2024-05-09T03:11:28.763362+0000","last_became_active":"2024-05-09T03:11:26.402595+0000","last_became_peered":"2024-05-09T03:11:26.402595+0000","last_unstale":"2024-05-09T03:11:28.763362+0000","last_undegraded":"2024-05-09T03:11:28.763362+0000","last_fullsized":"2024-05-09T03:11:28.763362+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:53.055819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398905+0000","last_change":"2024-05-09T03:11:27.411617+0000","last_active":"2024-05-09T03:11:28.398905+0000","last_peered":"2024-05-09T03:11:28.398905+0000","last_clean":"2024-05-09T03:11:28.398905+0000","last_became_active":"2024-05-09T03:11:27.411165+0000","last_became_peered":"2024-05-09T03:11:27.411165+0000","last_unstale":"2024-05-09T03:11:28.398905+0000","last_undegraded":"2024-05-09T03:11:28.398905+0000","last_fullsized":"2024-05-09T03:11:28.398905+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:39.006194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763890+0000","last_change":"2024-05-09T03:11:26.405390+0000","last_active":"2024-05-09T03:11:28.763890+0000","last_peered":"2024-05-09T03:11:28.763890+0000","last_clean":"2024-05-09T03:11:28.763890+0000","last_became_active":"2024-05-09T03:11:26.404946+0000","last_became_peered":"2024-05-09T03:11:26.404946+0000","last_unstale":"2024-05-09T03:11:28.763890+0000","last_undegraded":"2024-05-09T03:11:28.763890+0000","last_fullsized":"2024-05-09T03:11:28.763890+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:26:30.350454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399129+0000","last_change":"2024-05-09T03:11:27.412937+0000","last_active":"2024-05-09T03:11:28.399129+0000","last_peered":"2024-05-09T03:11:28.399129+0000","last_clean":"2024-05-09T03:11:28.399129+0000","last_became_active":"2024-05-09T03:11:27.412599+0000","last_became_peered":"2024-05-09T03:11:27.412599+0000","last_unstale":"2024-05-09T03:11:28.399129+0000","last_undegraded":"2024-05-09T03:11:28.399129+0000","last_fullsized":"2024-05-09T03:11:28.399129+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:37.733206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399521+0000","last_change":"2024-05-09T03:11:26.398230+0000","last_active":"2024-05-09T03:11:28.399521+0000","last_peered":"2024-05-09T03:11:28.399521+0000","last_clean":"2024-05-09T03:11:28.399521+0000","last_became_active":"2024-05-09T03:11:26.397938+0000","last_became_peered":"2024-05-09T03:11:26.397938+0000","last_unstale":"2024-05-09T03:11:28.399521+0000","last_undegraded":"2024-05-09T03:11:28.399521+0000","last_fullsized":"2024-05-09T03:11:28.399521+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:36:09.738086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766544+0000","last_change":"2024-05-09T03:11:27.416047+0000","last_active":"2024-05-09T03:11:28.766544+0000","last_peered":"2024-05-09T03:11:28.766544+0000","last_clean":"2024-05-09T03:11:28.766544+0000","last_became_active":"2024-05-09T03:11:27.415330+0000","last_became_peered":"2024-05-09T03:11:27.415330+0000","last_unstale":"2024-05-09T03:11:28.766544+0000","last_undegraded":"2024-05-09T03:11:28.766544+0000","last_fullsized":"2024-05-09T03:11:28.766544+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:24.775162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763955+0000","last_change":"2024-05-09T03:11:26.407940+0000","last_active":"2024-05-09T03:11:28.763955+0000","last_peered":"2024-05-09T03:11:28.763955+0000","last_clean":"2024-05-09T03:11:28.763955+0000","last_became_active":"2024-05-09T03:11:26.407561+0000","last_became_peered":"2024-05-09T03:11:26.407561+0000","last_unstale":"2024-05-09T03:11:28.763955+0000","last_undegraded":"2024-05-09T03:11:28.763955+0000","last_fullsized":"2024-05-09T03:11:28.763955+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:11:34.691002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421938+0000","last_change":"2024-05-09T03:11:27.406773+0000","last_active":"2024-05-09T03:11:28.421938+0000","last_peered":"2024-05-09T03:11:28.421938+0000","last_clean":"2024-05-09T03:11:28.421938+0000","last_became_active":"2024-05-09T03:11:27.406458+0000","last_became_peered":"2024-05-09T03:11:27.406458+0000","last_unstale":"2024-05-09T03:11:28.421938+0000","last_undegraded":"2024-05-09T03:11:28.421938+0000","last_fullsized":"2024-05-09T03:11:28.421938+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:00.963026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":17,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766581+0000","last_change":"2024-05-09T03:11:26.401361+0000","last_active":"2024-05-09T03:11:28.766581+0000","last_peered":"2024-05-09T03:11:28.766581+0000","last_clean":"2024-05-09T03:11:28.766581+0000","last_became_active":"2024-05-09T03:11:26.401038+0000","last_became_peered":"2024-05-09T03:11:26.401038+0000","last_unstale":"2024-05-09T03:11:28.766581+0000","last_undegraded":"2024-05-09T03:11:28.766581+0000","last_fullsized":"2024-05-09T03:11:28.766581+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:11.134274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766452+0000","last_change":"2024-05-09T03:11:27.416724+0000","last_active":"2024-05-09T03:11:28.766452+0000","last_peered":"2024-05-09T03:11:28.766452+0000","last_clean":"2024-05-09T03:11:28.766452+0000","last_became_active":"2024-05-09T03:11:27.416337+0000","last_became_peered":"2024-05-09T03:11:27.416337+0000","last_unstale":"2024-05-09T03:11:28.766452+0000","last_undegraded":"2024-05-09T03:11:28.766452+0000","last_fullsized":"2024-05-09T03:11:28.766452+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:02.195884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":20,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395496+0000","last_change":"2024-05-09T03:11:26.406530+0000","last_active":"2024-05-09T03:11:28.395496+0000","last_peered":"2024-05-09T03:11:28.395496+0000","last_clean":"2024-05-09T03:11:28.395496+0000","last_became_active":"2024-05-09T03:11:26.406173+0000","last_became_peered":"2024-05-09T03:11:26.406173+0000","last_unstale":"2024-05-09T03:11:28.395496+0000","last_undegraded":"2024-05-09T03:11:28.395496+0000","last_fullsized":"2024-05-09T03:11:28.395496+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:27.424687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399443+0000","last_change":"2024-05-09T03:11:27.413674+0000","last_active":"2024-05-09T03:11:28.399443+0000","last_peered":"2024-05-09T03:11:28.399443+0000","last_clean":"2024-05-09T03:11:28.399443+0000","last_became_active":"2024-05-09T03:11:27.412932+0000","last_became_peered":"2024-05-09T03:11:27.412932+0000","last_unstale":"2024-05-09T03:11:28.399443+0000","last_undegraded":"2024-05-09T03:11:28.399443+0000","last_fullsized":"2024-05-09T03:11:28.399443+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:50.359749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421913+0000","last_change":"2024-05-09T03:11:26.402684+0000","last_active":"2024-05-09T03:11:28.421913+0000","last_peered":"2024-05-09T03:11:28.421913+0000","last_clean":"2024-05-09T03:11:28.421913+0000","last_became_active":"2024-05-09T03:11:26.402325+0000","last_became_peered":"2024-05-09T03:11:26.402325+0000","last_unstale":"2024-05-09T03:11:28.421913+0000","last_undegraded":"2024-05-09T03:11:28.421913+0000","last_fullsized":"2024-05-09T03:11:28.421913+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:22.502342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763263+0000","last_change":"2024-05-09T03:11:27.406566+0000","last_active":"2024-05-09T03:11:28.763263+0000","last_peered":"2024-05-09T03:11:28.763263+0000","last_clean":"2024-05-09T03:11:28.763263+0000","last_became_active":"2024-05-09T03:11:27.406191+0000","last_became_peered":"2024-05-09T03:11:27.406191+0000","last_unstale":"2024-05-09T03:11:28.763263+0000","last_undegraded":"2024-05-09T03:11:28.763263+0000","last_fullsized":"2024-05-09T03:11:28.763263+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:37.733206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766188+0000","last_change":"2024-05-09T03:11:26.399378+0000","last_active":"2024-05-09T03:11:28.766188+0000","last_peered":"2024-05-09T03:11:28.766188+0000","last_clean":"2024-05-09T03:11:28.766188+0000","last_became_active":"2024-05-09T03:11:26.399091+0000","last_became_peered":"2024-05-09T03:11:26.399091+0000","last_unstale":"2024-05-09T03:11:28.766188+0000","last_undegraded":"2024-05-09T03:11:28.766188+0000","last_fullsized":"2024-05-09T03:11:28.766188+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421546+0000","last_change":"2024-05-09T03:11:27.409088+0000","last_active":"2024-05-09T03:11:28.421546+0000","last_peered":"2024-05-09T03:11:28.421546+0000","last_clean":"2024-05-09T03:11:28.421546+0000","last_became_active":"2024-05-09T03:11:27.406411+0000","last_became_peered":"2024-05-09T03:11:27.406411+0000","last_unstale":"2024-05-09T03:11:28.421546+0000","last_undegraded":"2024-05-09T03:11:28.421546+0000","last_fullsized":"2024-05-09T03:11:28.421546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:38:36.959072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421706+0000","last_change":"2024-05-09T03:11:26.398576+0000","last_active":"2024-05-09T03:11:28.421706+0000","last_peered":"2024-05-09T03:11:28.421706+0000","last_clean":"2024-05-09T03:11:28.421706+0000","last_became_active":"2024-05-09T03:11:26.398273+0000","last_became_peered":"2024-05-09T03:11:26.398273+0000","last_unstale":"2024-05-09T03:11:28.421706+0000","last_undegraded":"2024-05-09T03:11:28.421706+0000","last_fullsized":"2024-05-09T03:11:28.421706+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:38.032755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766335+0000","last_change":"2024-05-09T03:11:27.414943+0000","last_active":"2024-05-09T03:11:28.766335+0000","last_peered":"2024-05-09T03:11:28.766335+0000","last_clean":"2024-05-09T03:11:28.766335+0000","last_became_active":"2024-05-09T03:11:27.414642+0000","last_became_peered":"2024-05-09T03:11:27.414642+0000","last_unstale":"2024-05-09T03:11:28.766335+0000","last_undegraded":"2024-05-09T03:11:28.766335+0000","last_fullsized":"2024-05-09T03:11:28.766335+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:19.073921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":22,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763905+0000","last_change":"2024-05-09T03:11:26.403484+0000","last_active":"2024-05-09T03:11:28.763905+0000","last_peered":"2024-05-09T03:11:28.763905+0000","last_clean":"2024-05-09T03:11:28.763905+0000","last_became_active":"2024-05-09T03:11:26.403147+0000","last_became_peered":"2024-05-09T03:11:26.403147+0000","last_unstale":"2024-05-09T03:11:28.763905+0000","last_undegraded":"2024-05-09T03:11:28.763905+0000","last_fullsized":"2024-05-09T03:11:28.763905+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:07.861834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395493+0000","last_change":"2024-05-09T03:11:27.411377+0000","last_active":"2024-05-09T03:11:28.395493+0000","last_peered":"2024-05-09T03:11:28.395493+0000","last_clean":"2024-05-09T03:11:28.395493+0000","last_became_active":"2024-05-09T03:11:27.411100+0000","last_became_peered":"2024-05-09T03:11:27.411100+0000","last_unstale":"2024-05-09T03:11:28.395493+0000","last_undegraded":"2024-05-09T03:11:28.395493+0000","last_fullsized":"2024-05-09T03:11:28.395493+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:44:24.711209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395528+0000","last_change":"2024-05-09T03:11:26.408975+0000","last_active":"2024-05-09T03:11:28.395528+0000","last_peered":"2024-05-09T03:11:28.395528+0000","last_clean":"2024-05-09T03:11:28.395528+0000","last_became_active":"2024-05-09T03:11:26.408602+0000","last_became_peered":"2024-05-09T03:11:26.408602+0000","last_unstale":"2024-05-09T03:11:28.395528+0000","last_undegraded":"2024-05-09T03:11:28.395528+0000","last_fullsized":"2024-05-09T03:11:28.395528+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:25:41.245927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763685+0000","last_change":"2024-05-09T03:11:27.407905+0000","last_active":"2024-05-09T03:11:28.763685+0000","last_peered":"2024-05-09T03:11:28.763685+0000","last_clean":"2024-05-09T03:11:28.763685+0000","last_became_active":"2024-05-09T03:11:27.407565+0000","last_became_peered":"2024-05-09T03:11:27.407565+0000","last_unstale":"2024-05-09T03:11:28.763685+0000","last_undegraded":"2024-05-09T03:11:28.763685+0000","last_fullsized":"2024-05-09T03:11:28.763685+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:19.073921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399325+0000","last_change":"2024-05-09T03:11:26.399038+0000","last_active":"2024-05-09T03:11:28.399325+0000","last_peered":"2024-05-09T03:11:28.399325+0000","last_clean":"2024-05-09T03:11:28.399325+0000","last_became_active":"2024-05-09T03:11:26.398728+0000","last_became_peered":"2024-05-09T03:11:26.398728+0000","last_unstale":"2024-05-09T03:11:28.399325+0000","last_undegraded":"2024-05-09T03:11:28.399325+0000","last_fullsized":"2024-05-09T03:11:28.399325+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:25.493953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399311+0000","last_change":"2024-05-09T03:11:27.418360+0000","last_active":"2024-05-09T03:11:28.399311+0000","last_peered":"2024-05-09T03:11:28.399311+0000","last_clean":"2024-05-09T03:11:28.399311+0000","last_became_active":"2024-05-09T03:11:27.417813+0000","last_became_peered":"2024-05-09T03:11:27.417813+0000","last_unstale":"2024-05-09T03:11:28.399311+0000","last_undegraded":"2024-05-09T03:11:28.399311+0000","last_fullsized":"2024-05-09T03:11:28.399311+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:13.465981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763746+0000","last_change":"2024-05-09T03:11:26.402375+0000","last_active":"2024-05-09T03:11:28.763746+0000","last_peered":"2024-05-09T03:11:28.763746+0000","last_clean":"2024-05-09T03:11:28.763746+0000","last_became_active":"2024-05-09T03:11:26.402033+0000","last_became_peered":"2024-05-09T03:11:26.402033+0000","last_unstale":"2024-05-09T03:11:28.763746+0000","last_undegraded":"2024-05-09T03:11:28.763746+0000","last_fullsized":"2024-05-09T03:11:28.763746+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400060+0000","last_change":"2024-05-09T03:11:27.408673+0000","last_active":"2024-05-09T03:11:28.400060+0000","last_peered":"2024-05-09T03:11:28.400060+0000","last_clean":"2024-05-09T03:11:28.400060+0000","last_became_active":"2024-05-09T03:11:27.408398+0000","last_became_peered":"2024-05-09T03:11:27.408398+0000","last_unstale":"2024-05-09T03:11:28.400060+0000","last_undegraded":"2024-05-09T03:11:28.400060+0000","last_fullsized":"2024-05-09T03:11:28.400060+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:54.060589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.422052+0000","last_change":"2024-05-09T03:11:26.401461+0000","last_active":"2024-05-09T03:11:28.422052+0000","last_peered":"2024-05-09T03:11:28.422052+0000","last_clean":"2024-05-09T03:11:28.422052+0000","last_became_active":"2024-05-09T03:11:26.401148+0000","last_became_peered":"2024-05-09T03:11:26.401148+0000","last_unstale":"2024-05-09T03:11:28.422052+0000","last_undegraded":"2024-05-09T03:11:28.422052+0000","last_fullsized":"2024-05-09T03:11:28.422052+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:24:51.208260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421894+0000","last_change":"2024-05-09T03:11:27.407523+0000","last_active":"2024-05-09T03:11:28.421894+0000","last_peered":"2024-05-09T03:11:28.421894+0000","last_clean":"2024-05-09T03:11:28.421894+0000","last_became_active":"2024-05-09T03:11:27.407238+0000","last_became_peered":"2024-05-09T03:11:27.407238+0000","last_unstale":"2024-05-09T03:11:28.421894+0000","last_undegraded":"2024-05-09T03:11:28.421894+0000","last_fullsized":"2024-05-09T03:11:28.421894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:04.924505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.422054+0000","last_change":"2024-05-09T03:11:26.400381+0000","last_active":"2024-05-09T03:11:28.422054+0000","last_peered":"2024-05-09T03:11:28.422054+0000","last_clean":"2024-05-09T03:11:28.422054+0000","last_became_active":"2024-05-09T03:11:26.400059+0000","last_became_peered":"2024-05-09T03:11:26.400059+0000","last_unstale":"2024-05-09T03:11:28.422054+0000","last_undegraded":"2024-05-09T03:11:28.422054+0000","last_fullsized":"2024-05-09T03:11:28.422054+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:54.292440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.764149+0000","last_change":"2024-05-09T03:11:27.402555+0000","last_active":"2024-05-09T03:11:28.764149+0000","last_peered":"2024-05-09T03:11:28.764149+0000","last_clean":"2024-05-09T03:11:28.764149+0000","last_became_active":"2024-05-09T03:11:27.402229+0000","last_became_peered":"2024-05-09T03:11:27.402229+0000","last_unstale":"2024-05-09T03:11:28.764149+0000","last_undegraded":"2024-05-09T03:11:28.764149+0000","last_fullsized":"2024-05-09T03:11:28.764149+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:06.230758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763397+0000","last_change":"2024-05-09T03:11:26.403917+0000","last_active":"2024-05-09T03:11:28.763397+0000","last_peered":"2024-05-09T03:11:28.763397+0000","last_clean":"2024-05-09T03:11:28.763397+0000","last_became_active":"2024-05-09T03:11:26.403487+0000","last_became_peered":"2024-05-09T03:11:26.403487+0000","last_unstale":"2024-05-09T03:11:28.763397+0000","last_undegraded":"2024-05-09T03:11:28.763397+0000","last_fullsized":"2024-05-09T03:11:28.763397+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:42.763635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.395221+0000","last_change":"2024-05-09T03:11:27.412961+0000","last_active":"2024-05-09T03:11:28.395221+0000","last_peered":"2024-05-09T03:11:28.395221+0000","last_clean":"2024-05-09T03:11:28.395221+0000","last_became_active":"2024-05-09T03:11:27.412631+0000","last_became_peered":"2024-05-09T03:11:27.412631+0000","last_unstale":"2024-05-09T03:11:28.395221+0000","last_undegraded":"2024-05-09T03:11:28.395221+0000","last_fullsized":"2024-05-09T03:11:28.395221+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:42:41.876936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763297+0000","last_change":"2024-05-09T03:11:26.406762+0000","last_active":"2024-05-09T03:11:28.763297+0000","last_peered":"2024-05-09T03:11:28.763297+0000","last_clean":"2024-05-09T03:11:28.763297+0000","last_became_active":"2024-05-09T03:11:26.406432+0000","last_became_peered":"2024-05-09T03:11:26.406432+0000","last_unstale":"2024-05-09T03:11:28.763297+0000","last_undegraded":"2024-05-09T03:11:28.763297+0000","last_fullsized":"2024-05-09T03:11:28.763297+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:34.156082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763119+0000","last_change":"2024-05-09T03:11:27.411420+0000","last_active":"2024-05-09T03:11:28.763119+0000","last_peered":"2024-05-09T03:11:28.763119+0000","last_clean":"2024-05-09T03:11:28.763119+0000","last_became_active":"2024-05-09T03:11:27.411089+0000","last_became_peered":"2024-05-09T03:11:27.411089+0000","last_unstale":"2024-05-09T03:11:28.763119+0000","last_undegraded":"2024-05-09T03:11:28.763119+0000","last_fullsized":"2024-05-09T03:11:28.763119+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:25.693725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.765995+0000","last_change":"2024-05-09T03:11:26.399861+0000","last_active":"2024-05-09T03:11:28.765995+0000","last_peered":"2024-05-09T03:11:28.765995+0000","last_clean":"2024-05-09T03:11:28.765995+0000","last_became_active":"2024-05-09T03:11:26.399383+0000","last_became_peered":"2024-05-09T03:11:26.399383+0000","last_unstale":"2024-05-09T03:11:28.765995+0000","last_undegraded":"2024-05-09T03:11:28.765995+0000","last_fullsized":"2024-05-09T03:11:28.765995+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:08.673574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.400194+0000","last_change":"2024-05-09T03:11:27.412290+0000","last_active":"2024-05-09T03:11:28.400194+0000","last_peered":"2024-05-09T03:11:28.400194+0000","last_clean":"2024-05-09T03:11:28.400194+0000","last_became_active":"2024-05-09T03:11:27.411955+0000","last_became_peered":"2024-05-09T03:11:27.411955+0000","last_unstale":"2024-05-09T03:11:28.400194+0000","last_undegraded":"2024-05-09T03:11:28.400194+0000","last_fullsized":"2024-05-09T03:11:28.400194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:31.446242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"43'1","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421921+0000","last_change":"2024-05-09T03:11:26.398315+0000","last_active":"2024-05-09T03:11:28.421921+0000","last_peered":"2024-05-09T03:11:28.421921+0000","last_clean":"2024-05-09T03:11:28.421921+0000","last_became_active":"2024-05-09T03:11:26.397997+0000","last_became_peered":"2024-05-09T03:11:26.397997+0000","last_unstale":"2024-05-09T03:11:28.421921+0000","last_undegraded":"2024-05-09T03:11:28.421921+0000","last_fullsized":"2024-05-09T03:11:28.421921+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:20:24.170842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763457+0000","last_change":"2024-05-09T03:11:27.409274+0000","last_active":"2024-05-09T03:11:28.763457+0000","last_peered":"2024-05-09T03:11:28.763457+0000","last_clean":"2024-05-09T03:11:28.763457+0000","last_became_active":"2024-05-09T03:11:27.408465+0000","last_became_peered":"2024-05-09T03:11:27.408465+0000","last_unstale":"2024-05-09T03:11:28.763457+0000","last_undegraded":"2024-05-09T03:11:28.763457+0000","last_fullsized":"2024-05-09T03:11:28.763457+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:28.857415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766587+0000","last_change":"2024-05-09T03:11:26.401410+0000","last_active":"2024-05-09T03:11:28.766587+0000","last_peered":"2024-05-09T03:11:28.766587+0000","last_clean":"2024-05-09T03:11:28.766587+0000","last_became_active":"2024-05-09T03:11:26.401111+0000","last_became_peered":"2024-05-09T03:11:26.401111+0000","last_unstale":"2024-05-09T03:11:28.766587+0000","last_undegraded":"2024-05-09T03:11:28.766587+0000","last_fullsized":"2024-05-09T03:11:28.766587+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:28:24.688956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766438+0000","last_change":"2024-05-09T03:11:27.411038+0000","last_active":"2024-05-09T03:11:28.766438+0000","last_peered":"2024-05-09T03:11:28.766438+0000","last_clean":"2024-05-09T03:11:28.766438+0000","last_became_active":"2024-05-09T03:11:27.410703+0000","last_became_peered":"2024-05-09T03:11:27.410703+0000","last_unstale":"2024-05-09T03:11:28.766438+0000","last_undegraded":"2024-05-09T03:11:28.766438+0000","last_fullsized":"2024-05-09T03:11:28.766438+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:42.288267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763032+0000","last_change":"2024-05-09T03:11:26.401170+0000","last_active":"2024-05-09T03:11:28.763032+0000","last_peered":"2024-05-09T03:11:28.763032+0000","last_clean":"2024-05-09T03:11:28.763032+0000","last_became_active":"2024-05-09T03:11:26.400832+0000","last_became_peered":"2024-05-09T03:11:26.400832+0000","last_unstale":"2024-05-09T03:11:28.763032+0000","last_undegraded":"2024-05-09T03:11:28.763032+0000","last_fullsized":"2024-05-09T03:11:28.763032+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:13:25.493953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.422198+0000","last_change":"2024-05-09T03:11:27.417138+0000","last_active":"2024-05-09T03:11:28.422198+0000","last_peered":"2024-05-09T03:11:28.422198+0000","last_clean":"2024-05-09T03:11:28.422198+0000","last_became_active":"2024-05-09T03:11:27.416597+0000","last_became_peered":"2024-05-09T03:11:27.416597+0000","last_unstale":"2024-05-09T03:11:28.422198+0000","last_undegraded":"2024-05-09T03:11:28.422198+0000","last_fullsized":"2024-05-09T03:11:28.422198+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:22.285522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766335+0000","last_change":"2024-05-09T03:11:26.400568+0000","last_active":"2024-05-09T03:11:28.766335+0000","last_peered":"2024-05-09T03:11:28.766335+0000","last_clean":"2024-05-09T03:11:28.766335+0000","last_became_active":"2024-05-09T03:11:26.400253+0000","last_became_peered":"2024-05-09T03:11:26.400253+0000","last_unstale":"2024-05-09T03:11:28.766335+0000","last_undegraded":"2024-05-09T03:11:28.766335+0000","last_fullsized":"2024-05-09T03:11:28.766335+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:30.441472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.421813+0000","last_change":"2024-05-09T03:11:27.409547+0000","last_active":"2024-05-09T03:11:28.421813+0000","last_peered":"2024-05-09T03:11:28.421813+0000","last_clean":"2024-05-09T03:11:28.421813+0000","last_became_active":"2024-05-09T03:11:27.408810+0000","last_became_peered":"2024-05-09T03:11:27.408810+0000","last_unstale":"2024-05-09T03:11:28.421813+0000","last_undegraded":"2024-05-09T03:11:28.421813+0000","last_fullsized":"2024-05-09T03:11:28.421813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:11:42.434473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.399161+0000","last_change":"2024-05-09T03:11:26.397676+0000","last_active":"2024-05-09T03:11:28.399161+0000","last_peered":"2024-05-09T03:11:28.399161+0000","last_clean":"2024-05-09T03:11:28.399161+0000","last_became_active":"2024-05-09T03:11:26.396811+0000","last_became_peered":"2024-05-09T03:11:26.396811+0000","last_unstale":"2024-05-09T03:11:28.399161+0000","last_undegraded":"2024-05-09T03:11:28.399161+0000","last_fullsized":"2024-05-09T03:11:28.399161+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763469+0000","last_change":"2024-05-09T03:11:27.407941+0000","last_active":"2024-05-09T03:11:28.763469+0000","last_peered":"2024-05-09T03:11:28.763469+0000","last_clean":"2024-05-09T03:11:28.763469+0000","last_became_active":"2024-05-09T03:11:27.407612+0000","last_became_peered":"2024-05-09T03:11:27.407612+0000","last_unstale":"2024-05-09T03:11:28.763469+0000","last_undegraded":"2024-05-09T03:11:28.763469+0000","last_fullsized":"2024-05-09T03:11:28.763469+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:28.857415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763221+0000","last_change":"2024-05-09T03:11:26.402849+0000","last_active":"2024-05-09T03:11:28.763221+0000","last_peered":"2024-05-09T03:11:28.763221+0000","last_clean":"2024-05-09T03:11:28.763221+0000","last_became_active":"2024-05-09T03:11:26.402533+0000","last_became_peered":"2024-05-09T03:11:26.402533+0000","last_unstale":"2024-05-09T03:11:28.763221+0000","last_undegraded":"2024-05-09T03:11:28.763221+0000","last_fullsized":"2024-05-09T03:11:28.763221+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:51.356924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.398534+0000","last_change":"2024-05-09T03:11:27.420355+0000","last_active":"2024-05-09T03:11:28.398534+0000","last_peered":"2024-05-09T03:11:28.398534+0000","last_clean":"2024-05-09T03:11:28.398534+0000","last_became_active":"2024-05-09T03:11:27.420026+0000","last_became_peered":"2024-05-09T03:11:27.420026+0000","last_unstale":"2024-05-09T03:11:28.398534+0000","last_undegraded":"2024-05-09T03:11:28.398534+0000","last_fullsized":"2024-05-09T03:11:28.398534+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:56:31.446242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763983+0000","last_change":"2024-05-09T03:11:26.396367+0000","last_active":"2024-05-09T03:11:28.763983+0000","last_peered":"2024-05-09T03:11:28.763983+0000","last_clean":"2024-05-09T03:11:28.763983+0000","last_became_active":"2024-05-09T03:11:26.396006+0000","last_became_peered":"2024-05-09T03:11:26.396006+0000","last_unstale":"2024-05-09T03:11:28.763983+0000","last_undegraded":"2024-05-09T03:11:28.763983+0000","last_fullsized":"2024-05-09T03:11:28.763983+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:11.485652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763884+0000","last_change":"2024-05-09T03:11:27.406342+0000","last_active":"2024-05-09T03:11:28.763884+0000","last_peered":"2024-05-09T03:11:28.763884+0000","last_clean":"2024-05-09T03:11:28.763884+0000","last_became_active":"2024-05-09T03:11:27.406026+0000","last_became_peered":"2024-05-09T03:11:27.406026+0000","last_unstale":"2024-05-09T03:11:28.763884+0000","last_undegraded":"2024-05-09T03:11:28.763884+0000","last_fullsized":"2024-05-09T03:11:28.763884+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:26.379689+0000","last_clean_scrub_stamp":"2024-05-09T03:11:26.379689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:24.775162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.763896+0000","last_change":"2024-05-09T03:11:26.407676+0000","last_active":"2024-05-09T03:11:28.763896+0000","last_peered":"2024-05-09T03:11:28.763896+0000","last_clean":"2024-05-09T03:11:28.763896+0000","last_became_active":"2024-05-09T03:11:26.407284+0000","last_became_peered":"2024-05-09T03:11:26.407284+0000","last_unstale":"2024-05-09T03:11:28.763896+0000","last_undegraded":"2024-05-09T03:11:28.763896+0000","last_fullsized":"2024-05-09T03:11:28.763896+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:11:25.374919+0000","last_clean_scrub_stamp":"2024-05-09T03:11:25.374919+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:05.225988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":151,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-05-09T03:11:28.766074+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:11:28.766074+0000","last_peered":"2024-05-09T03:11:28.766074+0000","last_clean":"2024-05-09T03:11:28.766074+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:11:28.766074+0000","last_undegraded":"2024-05-09T03:11:28.766074+0000","last_fullsized":"2024-05-09T03:11:28.766074+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343744,"statfs":{"total":96636764160,"available":96607993856,"internally_reserved":0,"allocated":1101824,"data_stored":794271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":512000,"data_stored":204485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27524,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344316,"statfs":{"total":96636764160,"available":96608579584,"internally_reserved":0,"allocated":507904,"data_stored":203883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":516096,"data_stored":205425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31692,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340148,"statfs":{"total":96636764160,"available":96604311552,"internally_reserved":0,"allocated":516096,"data_stored":204451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":205447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":507904,"data_stored":203883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:11:36.403 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:11:36.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T03:11:36.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.562+0000 7f236c84c640 1 Processor -- start 2024-05-09T03:11:36.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.562+0000 7f236c84c640 1 -- start start 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.563+0000 7f236c84c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f23680bb5b0 0x7f23680d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.563+0000 7f236c84c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23680d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.563+0000 7f236c84c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23680d45d0 con 0x7f23680d52d0 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.563+0000 7f236c84c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f23680d56e0 con 0x7f23680bb5b0 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.563+0000 7f236c84c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f23681645e0 con 0x7f2368163dd0 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.563+0000 7f236659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f23680bb5b0 0x7f23680d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f2366d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f23680d52d0 0x7f23680b91b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57008/0 (socket says 172.21.15.46:57008) 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f2366d9d640 1 -- 172.21.15.46:0/751730647 learned_addr learned my addr 172.21.15.46:0/751730647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f235ffff640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23680d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f236659c640 1 -- 172.21.15.46:0/751730647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 msgr2=0x7f23680d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f236659c640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23680d4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f236659c640 1 -- 172.21.15.46:0/751730647 >> v1:172.21.15.46:6789/0 conn(0x7f23680d52d0 legacy=0x7f23680b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f236659c640 1 -- 172.21.15.46:0/751730647 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23681648c0 con 0x7f23680bb5b0 2024-05-09T03:11:36.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f2365d9b640 1 -- 172.21.15.46:0/751730647 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1493761898 0 0) 0x7f23680d45d0 con 0x7f23680d52d0 2024-05-09T03:11:36.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.564+0000 7f235ffff640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23680d4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.565+0000 7f236659c640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f23680d1380 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f23580097a0 tx=0x7f235802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6ef714008513c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.565+0000 7f2365d9b640 1 -- 172.21.15.46:0/751730647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358037070 con 0x7f23680bb5b0 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.565+0000 7f2365d9b640 1 -- 172.21.15.46:0/751730647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2358032440 con 0x7f23680bb5b0 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.565+0000 7f2365d9b640 1 -- 172.21.15.46:0/751730647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358032810 con 0x7f23680bb5b0 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 -- 172.21.15.46:0/751730647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 msgr2=0x7f23680d1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f23680d1380 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f23580097a0 tx=0x7f235802a180 comp rx=0 tx=0).stop 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 -- 172.21.15.46:0/751730647 shutdown_connections 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23680d4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 --2- 172.21.15.46:0/751730647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f23680d1380 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 -- 172.21.15.46:0/751730647 >> 172.21.15.46:0/751730647 conn(0x7f23680d5e80 msgr2=0x7f23680d6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:36.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 -- 172.21.15.46:0/751730647 shutdown_connections 2024-05-09T03:11:36.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.566+0000 7f236c84c640 1 -- 172.21.15.46:0/751730647 wait complete. 2024-05-09T03:11:36.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.568+0000 7f236c84c640 1 Processor -- start 2024-05-09T03:11:36.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.568+0000 7f236c84c640 1 -- start start 2024-05-09T03:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236c84c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f2368157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236c84c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23680d52d0 0x7f23681f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236c84c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23681f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236c84c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2368164360 con 0x7f23680bb5b0 2024-05-09T03:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236c84c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f23681641e0 con 0x7f23680d52d0 2024-05-09T03:11:36.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236c84c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f23680b98c0 con 0x7f2368163dd0 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f2368157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f2368157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39418/0 (socket says 172.21.15.46:39418) 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 -- 172.21.15.46:0/1175715813 learned_addr learned my addr 172.21.15.46:0/1175715813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f235ffff640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23680d52d0 0x7f23681f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f2366d9d640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23681f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 -- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 msgr2=0x7f23681f6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23681f6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 -- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23680d52d0 msgr2=0x7f23681f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23680d52d0 0x7f23681f63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f236659c640 1 -- 172.21.15.46:0/1175715813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23680cc710 con 0x7f23680bb5b0 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f235ffff640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23680d52d0 0x7f23681f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.569+0000 7f2366d9d640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23681f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.570+0000 7f236659c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f2368157250 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2358035040 tx=0x7f23580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.570+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358037040 con 0x7f23680bb5b0 2024-05-09T03:11:36.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.570+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2358030770 con 0x7f23680bb5b0 2024-05-09T03:11:36.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.570+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358030b40 con 0x7f23680bb5b0 2024-05-09T03:11:36.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.570+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23680adc00 con 0x7f23680bb5b0 2024-05-09T03:11:36.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.570+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23680cbc30 con 0x7f23680bb5b0 2024-05-09T03:11:36.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.571+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236806a270 con 0x7f23680bb5b0 2024-05-09T03:11:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.571+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2358044020 con 0x7f23680bb5b0 2024-05-09T03:11:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.572+0000 7f235effd640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2348076300 0x7f2348078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.572+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f23580b7b10 con 0x7f23680bb5b0 2024-05-09T03:11:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.572+0000 7f235ffff640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2348076300 0x7f2348078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.573+0000 7f235ffff640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2348076300 0x7f2348078810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f23680b0140 tx=0x7f2350008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.577+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2358032e20 con 0x7f23680bb5b0 2024-05-09T03:11:36.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.844+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f23680d0ce0 con 0x7f23680bb5b0 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.846+0000 7f235effd640 1 -- 172.21.15.46:0/1175715813 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) ==== 190+0+147 (secure 0 0 0) 0x7f23580813e0 con 0x7f23680bb5b0 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T03:11:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-05-09T03:11:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2348076300 msgr2=0x7f2348078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2348076300 0x7f2348078810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f23680b0140 tx=0x7f2350008040 comp rx=0 tx=0).stop 2024-05-09T03:11:36.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 msgr2=0x7f2368157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f2368157250 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2358035040 tx=0x7f23580046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 shutdown_connections 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2368163dd0 0x7f23681f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23680d52d0 0x7f23681f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2348076300 0x7f2348078810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 --2- 172.21.15.46:0/1175715813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23680bb5b0 0x7f2368157250 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.848+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 >> 172.21.15.46:0/1175715813 conn(0x7f23680d5e80 msgr2=0x7f23680b6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.849+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 shutdown_connections 2024-05-09T03:11:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.849+0000 7f236c84c640 1 -- 172.21.15.46:0/1175715813 wait complete. 2024-05-09T03:11:36.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:36.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.975+0000 7f3417e2b640 1 Processor -- start 2024-05-09T03:11:36.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.976+0000 7f3417e2b640 1 -- start start 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3417e2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f3410160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3417e2b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34100b7ef0 0x7f34100b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3417e2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34100d49e0 con 0x7f34100b4af0 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3417e2b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f34100d5bb0 con 0x7f34100b7ef0 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3417e2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34100d4cc0 con 0x7f34100d57a0 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34163c8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f34100b7ef0 0x7f34100b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34153c6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34100b4af0 0x7f34100b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57020/0 (socket says 172.21.15.46:57020) 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34153c6640 1 -- 172.21.15.46:0/2478432357 learned_addr learned my addr 172.21.15.46:0/2478432357 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3415bc7640 1 --2- 172.21.15.46:0/2478432357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f3410160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34163c8640 1 -- 172.21.15.46:0/2478432357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 msgr2=0x7f3410160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34163c8640 1 --2- 172.21.15.46:0/2478432357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f3410160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34163c8640 1 -- 172.21.15.46:0/2478432357 >> v1:172.21.15.46:6789/0 conn(0x7f34100b4af0 legacy=0x7f34100b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f34163c8640 1 -- 172.21.15.46:0/2478432357 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34100d4fa0 con 0x7f34100b7ef0 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.977+0000 7f3414bc5640 1 -- 172.21.15.46:0/2478432357 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294085555 0 0) 0x7f34100d49e0 con 0x7f34100b4af0 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.978+0000 7f34163c8640 1 --2- 172.21.15.46:0/2478432357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34100b7ef0 0x7f34100b4f00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f34080097a0 tx=0x7f340802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9dfc45d1c2e1bea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.978+0000 7f3414bc5640 1 -- 172.21.15.46:0/2478432357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408037070 con 0x7f34100b7ef0 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.978+0000 7f3414bc5640 1 -- 172.21.15.46:0/2478432357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3408032440 con 0x7f34100b7ef0 2024-05-09T03:11:36.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.978+0000 7f3414bc5640 1 -- 172.21.15.46:0/2478432357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408032810 con 0x7f34100b7ef0 2024-05-09T03:11:36.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 -- 172.21.15.46:0/2478432357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34100b7ef0 msgr2=0x7f34100b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2478432357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34100b7ef0 0x7f34100b4f00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f34080097a0 tx=0x7f340802a180 comp rx=0 tx=0).stop 2024-05-09T03:11:36.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 -- 172.21.15.46:0/2478432357 shutdown_connections 2024-05-09T03:11:36.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2478432357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34100b7ef0 0x7f34100b4f00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2478432357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f3410160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 -- 172.21.15.46:0/2478432357 >> 172.21.15.46:0/2478432357 conn(0x7f34100743e0 msgr2=0x7f34100d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:36.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 -- 172.21.15.46:0/2478432357 shutdown_connections 2024-05-09T03:11:36.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.979+0000 7f3417e2b640 1 -- 172.21.15.46:0/2478432357 wait complete. 2024-05-09T03:11:36.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.981+0000 7f3417e2b640 1 Processor -- start 2024-05-09T03:11:36.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.981+0000 7f3417e2b640 1 -- start start 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3417e2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34100b4af0 0x7f341015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3417e2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f341015cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3417e2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 0x7f34100b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3417e2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34100d1920 con 0x7f3410157770 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3417e2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f34100d15b0 con 0x7f34100b4af0 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3417e2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3410152360 con 0x7f34100d57a0 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f34163c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 0x7f34100b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f34163c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 0x7f34100b6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39430/0 (socket says 172.21.15.46:39430) 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f34163c8640 1 -- 172.21.15.46:0/2348332196 learned_addr learned my addr 172.21.15.46:0/2348332196 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.982+0000 7f3415bc7640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34100b4af0 0x7f341015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34153c6640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f341015cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34163c8640 1 -- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 msgr2=0x7f341015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34163c8640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f341015cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34163c8640 1 -- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34100b4af0 msgr2=0x7f341015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34163c8640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34100b4af0 0x7f341015a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34163c8640 1 -- 172.21.15.46:0/2348332196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341015d330 con 0x7f3410157770 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34153c6640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f341015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f34163c8640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 0x7f34100b6c30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f34080098c0 tx=0x7f34080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408037040 con 0x7f3410157770 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3408042070 con 0x7f3410157770 2024-05-09T03:11:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408030b50 con 0x7f3410157770 2024-05-09T03:11:36.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341015add0 con 0x7f3410157770 2024-05-09T03:11:36.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.983+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3410156f70 con 0x7f3410157770 2024-05-09T03:11:36.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.984+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341015ab40 con 0x7f3410157770 2024-05-09T03:11:36.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.985+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3408030cf0 con 0x7f3410157770 2024-05-09T03:11:36.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.985+0000 7f340e7fc640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33fc0762d0 0x7f33fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:36.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.986+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f34080b81c0 con 0x7f3410157770 2024-05-09T03:11:36.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.986+0000 7f3415bc7640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33fc0762d0 0x7f33fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:36.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.986+0000 7f3415bc7640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33fc0762d0 0x7f33fc0787e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f341006d3b0 tx=0x7f3400009830 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:36.990+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3408032e20 con 0x7f3410157770 2024-05-09T03:11:37.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.221+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f341006a270 con 0x7f3410157770 2024-05-09T03:11:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.222+0000 7f340e7fc640 1 -- 172.21.15.46:0/2348332196 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5172 (secure 0 0 0) 0x7f3408081a80 con 0x7f3410157770 2024-05-09T03:11:37.223 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:37.223 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:29.815636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2630572362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2630572362},{"type":"v1","addr":"172.21.15.46:6835","nonce":2630572362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":3}]} 2024-05-09T03:11:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2024-05-09T03:11:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33fc0762d0 msgr2=0x7f33fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33fc0762d0 0x7f33fc0787e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f341006d3b0 tx=0x7f3400009830 comp rx=0 tx=0).stop 2024-05-09T03:11:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 msgr2=0x7f34100b6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 0x7f34100b6c30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f34080098c0 tx=0x7f34080046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 shutdown_connections 2024-05-09T03:11:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33fc0762d0 0x7f33fc0787e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3410157770 0x7f34100b6c30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34100d57a0 0x7f341015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 --2- 172.21.15.46:0/2348332196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34100b4af0 0x7f341015a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 >> 172.21.15.46:0/2348332196 conn(0x7f34100743e0 msgr2=0x7f34100baab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 shutdown_connections 2024-05-09T03:11:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.224+0000 7f3417e2b640 1 -- 172.21.15.46:0/2348332196 wait complete. 2024-05-09T03:11:37.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:11:26.786006+0000', 'modified': '2024-05-09T03:11:29.815636+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5358}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5358': {'gid': 5358, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/2630572362', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 2630572362}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2630572362}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5358, 'qdb_cluster': [5358]} 2024-05-09T03:11:37.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:11:37.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T03:11:37.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.757+0000 7f0aabea7640 1 Processor -- start 2024-05-09T03:11:37.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.757+0000 7f0aabea7640 1 -- start start 2024-05-09T03:11:37.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aabea7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aabea7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0aa406e7e0 0x7f0aa4164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aabea7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aa409d470 con 0x7f0aa4164ec0 2024-05-09T03:11:37.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aabea7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa409cfe0 con 0x7f0aa406e7e0 2024-05-09T03:11:37.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aabea7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa40b94a0 con 0x7f0aa40d3730 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa9442640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0aa406e7e0 0x7f0aa4164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aaa444640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0aa4164ec0 0x7f0aa40ceba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57024/0 (socket says 172.21.15.46:57024) 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aaa444640 1 -- 172.21.15.46:0/748150114 learned_addr learned my addr 172.21.15.46:0/748150114 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/748150114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa9442640 1 -- 172.21.15.46:0/748150114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 msgr2=0x7f0aa406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa9442640 1 --2- 172.21.15.46:0/748150114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa406e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa9442640 1 -- 172.21.15.46:0/748150114 >> v1:172.21.15.46:6789/0 conn(0x7f0aa4164ec0 legacy=0x7f0aa40ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa9442640 1 -- 172.21.15.46:0/748150114 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa40d11b0 con 0x7f0aa406e7e0 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.759+0000 7f0aa8c41640 1 -- 172.21.15.46:0/748150114 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632199225 0 0) 0x7f0a9c001110 con 0x7f0aa4164ec0 2024-05-09T03:11:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.760+0000 7f0aa9442640 1 --2- 172.21.15.46:0/748150114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa406e7e0 0x7f0aa4164980 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0a980097a0 tx=0x7f0a9802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79039b3453787f0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.760+0000 7f0aa8c41640 1 -- 172.21.15.46:0/748150114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98037070 con 0x7f0aa406e7e0 2024-05-09T03:11:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.760+0000 7f0aa8c41640 1 -- 172.21.15.46:0/748150114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9802ae50 con 0x7f0aa406e7e0 2024-05-09T03:11:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.760+0000 7f0aa8c41640 1 -- 172.21.15.46:0/748150114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98032660 con 0x7f0aa406e7e0 2024-05-09T03:11:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 -- 172.21.15.46:0/748150114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa406e7e0 msgr2=0x7f0aa4164980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 --2- 172.21.15.46:0/748150114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa406e7e0 0x7f0aa4164980 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0a980097a0 tx=0x7f0a9802a100 comp rx=0 tx=0).stop 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 -- 172.21.15.46:0/748150114 shutdown_connections 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 --2- 172.21.15.46:0/748150114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa406e7e0 0x7f0aa4164980 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 --2- 172.21.15.46:0/748150114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa406e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 -- 172.21.15.46:0/748150114 >> 172.21.15.46:0/748150114 conn(0x7f0aa40d05f0 msgr2=0x7f0aa40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 -- 172.21.15.46:0/748150114 shutdown_connections 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.761+0000 7f0aabea7640 1 -- 172.21.15.46:0/748150114 wait complete. 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.763+0000 7f0aabea7640 1 Processor -- start 2024-05-09T03:11:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.763+0000 7f0aabea7640 1 -- start start 2024-05-09T03:11:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aabea7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aa406e7e0 0x7f0aa40cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aabea7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa4151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aabea7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 0x7f0aa40cc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aabea7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa408cd60 con 0x7f0aa4164ec0 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aabea7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa40b9160 con 0x7f0aa406e7e0 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aabea7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa408cee0 con 0x7f0aa40d3730 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 0x7f0aa40cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 0x7f0aa40cc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39452/0 (socket says 172.21.15.46:39452) 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 -- 172.21.15.46:0/394311326 learned_addr learned my addr 172.21.15.46:0/394311326 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aa9442640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa4151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aa406e7e0 0x7f0aa40cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 -- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 msgr2=0x7f0aa4151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa4151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 -- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aa406e7e0 msgr2=0x7f0aa40cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aa406e7e0 0x7f0aa40cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aaa444640 1 -- 172.21.15.46:0/394311326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa40cd390 con 0x7f0aa4164ec0 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.764+0000 7f0aa9442640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa4151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.765+0000 7f0aaa444640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 0x7f0aa40cc660 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c390 tx=0x7f0a9c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.765+0000 7f0aa27fc640 1 -- 172.21.15.46:0/394311326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0074e0 con 0x7f0aa4164ec0 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.765+0000 7f0aa27fc640 1 -- 172.21.15.46:0/394311326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9c0047e0 con 0x7f0aa4164ec0 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.765+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa40cad50 con 0x7f0aa4164ec0 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.765+0000 7f0aa27fc640 1 -- 172.21.15.46:0/394311326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c004bd0 con 0x7f0aa4164ec0 2024-05-09T03:11:37.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.765+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa40cb2d0 con 0x7f0aa4164ec0 2024-05-09T03:11:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.767+0000 7f0aa27fc640 1 -- 172.21.15.46:0/394311326 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a9c011070 con 0x7f0aa4164ec0 2024-05-09T03:11:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.767+0000 7f0aa27fc640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a90076300 0x7f0a90078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.768+0000 7f0aa27fc640 1 -- 172.21.15.46:0/394311326 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0a9c097090 con 0x7f0aa4164ec0 2024-05-09T03:11:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.768+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a90076300 0x7f0a90078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.768+0000 7f0ab2e30740 1 Processor -- start 2024-05-09T03:11:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.768+0000 7f0ab2e30740 1 -- start start 2024-05-09T03:11:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.768+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a90076300 0x7f0a90078810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a940099c0 tx=0x7f0a94003990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.769+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd0a5270 0x5573dd474eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.769+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd48fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.769+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd490120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.769+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5573dd3ad4e0 con 0x5573dd07c3b0 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.769+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573dd348ff0 con 0x5573dd0a5270 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.769+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5573dd3a94a0 con 0x5573dd084590 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd48fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd48fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39464/0 (socket says 172.21.15.46:39464) 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 -- 172.21.15.46:0/2910616348 learned_addr learned my addr 172.21.15.46:0/2910616348 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd0a5270 0x5573dd474eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aaa444640 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd490120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 -- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 msgr2=0x5573dd490120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd490120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 -- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd0a5270 msgr2=0x5573dd474eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd0a5270 0x5573dd474eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 -- 172.21.15.46:0/2910616348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573dd32f720 con 0x5573dd07c3b0 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd0a5270 0x5573dd474eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0aa9442640 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd48fbe0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0a98000c00 tx=0x7f0a9802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d662e5bb14288573 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0a8b7fe640 1 -- 172.21.15.46:0/2910616348 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98037040 con 0x5573dd07c3b0 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0a8b7fe640 1 -- 172.21.15.46:0/2910616348 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a98004140 con 0x5573dd07c3b0 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.770+0000 7f0a8b7fe640 1 -- 172.21.15.46:0/2910616348 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a98003610 con 0x5573dd07c3b0 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2910616348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 msgr2=0x5573dd48fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd48fbe0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0a98000c00 tx=0x7f0a9802c000 comp rx=0 tx=0).stop 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2910616348 shutdown_connections 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd490120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd48fbe0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2910616348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd0a5270 0x5573dd474eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2910616348 >> 172.21.15.46:0/2910616348 conn(0x5573dd0993c0 msgr2=0x5573dd44a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2910616348 shutdown_connections 2024-05-09T03:11:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.771+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2910616348 wait complete. 2024-05-09T03:11:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.772+0000 7f0ab2e30740 1 -- start start 2024-05-09T03:11:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.772+0000 7f0ab2e30740 1 Processor -- start 2024-05-09T03:11:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.773+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd07c3b0 0x5573dd4705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.773+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd48cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.773+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 0x5573dd48e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.773+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5573dd334760 con 0x5573dd0a5270 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.773+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573dd48a4f0 con 0x5573dd07c3b0 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.773+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5573dd48a7d0 con 0x5573dd084590 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0ab2e30740 10 client.0 _ll_register_callbacks cb 0x5573dd3574b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 0x5573dd48e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 0x5573dd48e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39472/0 (socket says 172.21.15.46:39472) 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 -- 172.21.15.46:0/2897829671 learned_addr learned my addr 172.21.15.46:0/2897829671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd07c3b0 0x5573dd4705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aa9442640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd48cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 -- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 msgr2=0x5573dd48cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd48cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 -- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd07c3b0 msgr2=0x5573dd4705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd07c3b0 0x5573dd4705b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 -- 172.21.15.46:0/2897829671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573dd375bd0 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd07c3b0 0x5573dd4705b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aa9442640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd48cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0aaa444640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 0x5573dd48e8e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c05d560 tx=0x7f0a9c007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e02db7b6324671b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0a88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0a88ff9640 1 -- 172.21.15.46:0/2897829671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c079070 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0a88ff9640 1 -- 172.21.15.46:0/2897829671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0a9c068e30 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a98004140 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0a88ff9640 1 -- 172.21.15.46:0/2897829671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a9c0602f0 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0ab2e30740 10 client.5541 fetch_fsmap learned FSMap version 30 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.774+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5573dd35b280 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0a88ff9640 1 -- 172.21.15.46:0/2897829671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a98004140 con 0x5573dd0a5270 2024-05-09T03:11:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0a88ff9640 1 -- 172.21.15.46:0/2897829671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7f0a9c05f550 con 0x5573dd0a5270 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0ab2e30740 10 client.5541 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0ab2e30740 10 client.5541 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0ab2e30740 20 client.5541 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd063e40 0x5573dd4acb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0ab2e30740 4 client.5541 mds_command: new command op to 5358 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 --> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573dd3bb0f0 con 0x5573dd063e40 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd063e40 0x5573dd4acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0a88ff9640 10 client.5541 ms_handle_connect on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.775+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd063e40 0x5573dd4acb10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39510b960a5bd973 server_cookie=4b190988051eacf1 in_seq=0 out_seq=0 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0a88ff9640 1 -- 172.21.15.46:0/2897829671 <== mds.0 v2:172.21.15.46:6833/2630572362 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5573dd3bb0f0 con 0x5573dd063e40 2024-05-09T03:11:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0a88ff9640 10 client.5541 handle_command_reply: tid=0 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 client.5541 shutdown 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 msgr2=0x5573dd48e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 0x5573dd48e8e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c05d560 tx=0x7f0a9c007740 comp rx=0 tx=0).stop 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 shutdown_connections 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd063e40 0x5573dd4acb10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd0a5270 0x5573dd48e8e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd084590 0x5573dd48cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0a88ff9640 0 client.5541 ms_handle_reset on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/2897829671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd07c3b0 0x5573dd4705b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 >> 172.21.15.46:0/2897829671 conn(0x5573dd0993c0 msgr2=0x5573dd471a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 shutdown_connections 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.776+0000 7f0ab2e30740 1 -- 172.21.15.46:0/2897829671 wait complete. 2024-05-09T03:11:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.777+0000 7f0ab2e30740 20 client.5541 trim_cache size 0 max 16384 2024-05-09T03:11:37.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.796+0000 7f0ab2e30740 1 Processor -- start 2024-05-09T03:11:37.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.796+0000 7f0ab2e30740 1 -- start start 2024-05-09T03:11:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd44b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd44ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd471260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa4201da0 con 0x5573dd07c3b0 2024-05-09T03:11:37.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa40c7960 con 0x5573dd084590 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa40c7fb0 con 0x5573dd0a5270 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd44b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd44b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39486/0 (socket says 172.21.15.46:39486) 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 -- 172.21.15.46:0/3530049999 learned_addr learned my addr 172.21.15.46:0/3530049999 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 -- 172.21.15.46:0/3530049999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 msgr2=0x5573dd471260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd471260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 -- 172.21.15.46:0/3530049999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 msgr2=0x5573dd44ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd44ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 -- 172.21.15.46:0/3530049999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573dd110850 con 0x5573dd07c3b0 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.797+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd44b4f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0a94016770 tx=0x7f0a94015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14dda2f4f274d0e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0a8affd640 1 -- 172.21.15.46:0/3530049999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9401f040 con 0x5573dd07c3b0 2024-05-09T03:11:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0a8affd640 1 -- 172.21.15.46:0/3530049999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9402e070 con 0x5573dd07c3b0 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0a8affd640 1 -- 172.21.15.46:0/3530049999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94028610 con 0x5573dd07c3b0 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 -- 172.21.15.46:0/3530049999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 msgr2=0x5573dd44b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd44b4f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0a94016770 tx=0x7f0a94015000 comp rx=0 tx=0).stop 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 -- 172.21.15.46:0/3530049999 shutdown_connections 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd471260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd44ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/3530049999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd44b4f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 -- 172.21.15.46:0/3530049999 >> 172.21.15.46:0/3530049999 conn(0x5573dd0993c0 msgr2=0x5573dd4468e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 -- 172.21.15.46:0/3530049999 shutdown_connections 2024-05-09T03:11:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.798+0000 7f0ab2e30740 1 -- 172.21.15.46:0/3530049999 wait complete. 2024-05-09T03:11:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.799+0000 7f0ab2e30740 1 -- start start 2024-05-09T03:11:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.799+0000 7f0ab2e30740 1 Processor -- start 2024-05-09T03:11:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd50fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd48ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd44b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5573dd48ae30 con 0x5573dd07c3b0 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573dd48d5b0 con 0x5573dd084590 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5573dd48b3e0 con 0x5573dd0a5270 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0ab2e30740 10 client.0 _ll_register_callbacks cb 0x7f0a9c0600e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd50fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd50fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39492/0 (socket says 172.21.15.46:39492) 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 -- 172.21.15.46:0/1748329034 learned_addr learned my addr 172.21.15.46:0/1748329034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aaa444640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd44b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9442640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd48ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 -- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 msgr2=0x5573dd44b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd44b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 -- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 msgr2=0x5573dd48ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd48ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 -- 172.21.15.46:0/1748329034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573dd1876b0 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aaa444640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd44b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9442640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd48ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0aa9c43640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd50fc90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402a000 tx=0x7f0a94004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947ca3ff33f58a36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.801+0000 7f0a88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0a88ff9640 1 -- 172.21.15.46:0/1748329034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94028880 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0a88ff9640 1 -- 172.21.15.46:0/1748329034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f0a94028a20 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a9402e070 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0a88ff9640 1 -- 172.21.15.46:0/1748329034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0a94041050 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 10 client.5547 fetch_fsmap learned FSMap version 30 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5573dd369630 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0a88ff9640 1 -- 172.21.15.46:0/1748329034 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9402e070 con 0x5573dd07c3b0 2024-05-09T03:11:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0a88ff9640 1 -- 172.21.15.46:0/1748329034 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7f0a94044020 con 0x5573dd07c3b0 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 10 client.5547 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 10 client.5547 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 20 client.5547 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd44b880 0x5573dd443c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 4 client.5547 mds_command: new command op to 5358 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 --> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] -- command(tid 0: {"prefix": "status"}) -- 0x5573dd3b4f60 con 0x5573dd44b880 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.802+0000 7f0aa9442640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd44b880 0x5573dd443c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0a88ff9640 10 client.5547 ms_handle_connect on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0aa9442640 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd44b880 0x5573dd443c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a6f8ef19a673fda server_cookie=50658563309d1aa in_seq=0 out_seq=0 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0a88ff9640 1 -- 172.21.15.46:0/1748329034 <== mds.0 v2:172.21.15.46:6833/2630572362 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5573dd3b4f60 con 0x5573dd44b880 2024-05-09T03:11:37.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0a88ff9640 10 client.5547 handle_command_reply: tid=0 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 client.5547 shutdown 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 msgr2=0x5573dd50fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd50fc90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0a9402a000 tx=0x7f0a94004ab0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 shutdown_connections 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x5573dd44b880 0x5573dd443c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5573dd0a5270 0x5573dd44b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573dd084590 0x5573dd48ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0a88ff9640 0 client.5547 ms_handle_reset on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 --2- 172.21.15.46:0/1748329034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5573dd07c3b0 0x5573dd50fc90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 >> 172.21.15.46:0/1748329034 conn(0x5573dd0993c0 msgr2=0x5573dd48dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 shutdown_connections 2024-05-09T03:11:37.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 1 -- 172.21.15.46:0/1748329034 wait complete. 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.803+0000 7f0ab2e30740 20 client.5547 trim_cache size 0 max 16384 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5358, 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:11:37.811 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:11:37.812 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.396736625000001, 2024-05-09T03:11:37.812 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 30, 2024-05-09T03:11:37.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 43, 2024-05-09T03:11:37.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 43, 2024-05-09T03:11:37.812 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15.336752840000001 2024-05-09T03:11:37.812 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a90076300 msgr2=0x7f0a90078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a90076300 0x7f0a90078810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0a940099c0 tx=0x7f0a94003990 comp rx=0 tx=0).stop 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 msgr2=0x7f0aa40cc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 0x7f0aa40cc660 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00c390 tx=0x7f0a9c00c850 comp rx=0 tx=0).stop 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 shutdown_connections 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a90076300 0x7f0a90078810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa4164ec0 0x7f0aa40cc660 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aa40d3730 0x7f0aa4151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 --2- 172.21.15.46:0/394311326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aa406e7e0 0x7f0aa40cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 >> 172.21.15.46:0/394311326 conn(0x7f0aa40d05f0 msgr2=0x7f0aa4163f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 shutdown_connections 2024-05-09T03:11:37.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.804+0000 7f0aabea7640 1 -- 172.21.15.46:0/394311326 wait complete. 2024-05-09T03:11:37.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:11:37.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.931+0000 7f37e7c9f640 1 Processor -- start 2024-05-09T03:11:37.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.931+0000 7f37e7c9f640 1 -- start start 2024-05-09T03:11:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e7c9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37e00b28b0 0x7f37e00d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e7c9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f37e00d2f20 0x7f37e00b4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e7c9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e015d590 con 0x7f37e015d180 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e7c9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f37e006eaf0 con 0x7f37e00b28b0 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e7c9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f37e006e1c0 con 0x7f37e00d2f20 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e523a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37e00b28b0 0x7f37e00d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e5a3b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f37e015d180 0x7f37e00b21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57034/0 (socket says 172.21.15.46:57034) 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.932+0000 7f37e5a3b640 1 -- 172.21.15.46:0/1409778706 learned_addr learned my addr 172.21.15.46:0/1409778706 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e623c640 1 --2- 172.21.15.46:0/1409778706 >> v2:172.21.15.88:3300/0 conn(0x7f37e00d2f20 0x7f37e00b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e4a39640 1 -- 172.21.15.46:0/1409778706 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 339535889 0 0) 0x7f37d0001610 con 0x7f37e015d180 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e4a39640 1 -- 172.21.15.46:0/1409778706 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c8003520 con 0x7f37e015d180 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e523a640 1 -- 172.21.15.46:0/1409778706 >> v2:172.21.15.88:3300/0 conn(0x7f37e00d2f20 msgr2=0x7f37e00b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e523a640 1 --2- 172.21.15.46:0/1409778706 >> v2:172.21.15.88:3300/0 conn(0x7f37e00d2f20 0x7f37e00b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e523a640 1 -- 172.21.15.46:0/1409778706 >> v1:172.21.15.46:6789/0 conn(0x7f37e015d180 legacy=0x7f37e00b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.933+0000 7f37e523a640 1 -- 172.21.15.46:0/1409778706 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e006e4a0 con 0x7f37e00b28b0 2024-05-09T03:11:37.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e523a640 1 --2- 172.21.15.46:0/1409778706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00b28b0 0x7f37e00d29e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f37e006ec70 tx=0x7f37d402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a3027d8368584d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e4a39640 1 -- 172.21.15.46:0/1409778706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4037070 con 0x7f37e00b28b0 2024-05-09T03:11:37.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e4a39640 1 -- 172.21.15.46:0/1409778706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d4032440 con 0x7f37e00b28b0 2024-05-09T03:11:37.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e4a39640 1 -- 172.21.15.46:0/1409778706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4032810 con 0x7f37e00b28b0 2024-05-09T03:11:37.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1409778706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00b28b0 msgr2=0x7f37e00d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1409778706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00b28b0 0x7f37e00d29e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f37e006ec70 tx=0x7f37d402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1409778706 shutdown_connections 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1409778706 >> v2:172.21.15.88:3300/0 conn(0x7f37e00d2f20 0x7f37e00b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1409778706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00b28b0 0x7f37e00d29e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.934+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1409778706 >> 172.21.15.46:0/1409778706 conn(0x7f37e009b7d0 msgr2=0x7f37e009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.935+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1409778706 shutdown_connections 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.935+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1409778706 wait complete. 2024-05-09T03:11:37.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.936+0000 7f37e7c9f640 1 Processor -- start 2024-05-09T03:11:37.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.936+0000 7f37e7c9f640 1 -- start start 2024-05-09T03:11:37.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 0x7f37e01f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37e015d180 0x7f37e014fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37e0150280 0x7f37e0070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f37e00d3330 con 0x7f37e00d2f20 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f37e006ef80 con 0x7f37e0150280 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f37e00d34b0 con 0x7f37e015d180 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 0x7f37e01f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 0x7f37e01f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39518/0 (socket says 172.21.15.46:39518) 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 -- 172.21.15.46:0/1701564076 learned_addr learned my addr 172.21.15.46:0/1701564076 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e623c640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37e0150280 0x7f37e0070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e523a640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37e015d180 0x7f37e014fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 -- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37e015d180 msgr2=0x7f37e014fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37e015d180 0x7f37e014fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 -- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37e0150280 msgr2=0x7f37e0070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:37.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37e0150280 0x7f37e0070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 -- 172.21.15.46:0/1701564076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e015f090 con 0x7f37e00d2f20 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e623c640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37e0150280 0x7f37e0070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e5a3b640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 0x7f37e01f3ab0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f37d000c7d0 tx=0x7f37d000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d0017070 con 0x7f37e00d2f20 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d000a640 con 0x7f37e00d2f20 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e00cf260 con 0x7f37e00d2f20 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.938+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d000a9f0 con 0x7f37e00d2f20 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.939+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e00cf820 con 0x7f37e00d2f20 2024-05-09T03:11:37.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.939+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e006a270 con 0x7f37e00d2f20 2024-05-09T03:11:37.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.940+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f37d001a070 con 0x7f37e00d2f20 2024-05-09T03:11:37.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.941+0000 7f37de7fc640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37cc0762d0 0x7f37cc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:37.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.941+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f37d0096c40 con 0x7f37e00d2f20 2024-05-09T03:11:37.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.941+0000 7f37e523a640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37cc0762d0 0x7f37cc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:37.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.942+0000 7f37e523a640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37cc0762d0 0x7f37cc0787e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37d4034000 tx=0x7f37d402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:37.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:37.946+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f37e015d590 con 0x7f37e00d2f20 2024-05-09T03:11:38.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.177+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f37e0160e50 con 0x7f37e00d2f20 2024-05-09T03:11:38.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.177+0000 7f37de7fc640 1 -- 172.21.15.46:0/1701564076 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7f37d00605a0 con 0x7f37e00d2f20 2024-05-09T03:11:38.178 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:11:38.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37cc0762d0 msgr2=0x7f37cc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:38.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37cc0762d0 0x7f37cc0787e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f37d4034000 tx=0x7f37d402e000 comp rx=0 tx=0).stop 2024-05-09T03:11:38.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 msgr2=0x7f37e01f3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:38.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 0x7f37e01f3ab0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f37d000c7d0 tx=0x7f37d000cc90 comp rx=0 tx=0).stop 2024-05-09T03:11:38.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 shutdown_connections 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37e0150280 0x7f37e0070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37e015d180 0x7f37e014fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37cc0762d0 0x7f37cc0787e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 --2- 172.21.15.46:0/1701564076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37e00d2f20 0x7f37e01f3ab0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 >> 172.21.15.46:0/1701564076 conn(0x7f37e009b7d0 msgr2=0x7f37e01511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 shutdown_connections 2024-05-09T03:11:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38.179+0000 7f37e7c9f640 1 -- 172.21.15.46:0/1701564076 wait complete. 2024-05-09T03:11:38.191 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:11:38.191 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:11:38.191 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:11:38.191 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:11:38.192 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:38.192 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:11:38.192 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:11:38.192 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:11:38.192 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:11:38.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.192 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:38.209 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42419sec preferred_lft 42419sec 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:11:38.210 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:11:38.211 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:38.212 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:11:38.212 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:11:38.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.212 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:11:38.273 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:11:38.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.275 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:11:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:11:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:11:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:11:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:11:38.332 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:11:38.333 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:11:38.333 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:11:38.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:11:38.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:11:38.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:11:38.333 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:11:38.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:11:38.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:38.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.463 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:11:38.479 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:11:38.479 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:11:38.479 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:11:38.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.479 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:11:38.545 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:11:38.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:11:38.545 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:38.602 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:11:38.602 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:11:38.602 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:38.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:11:38.725 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:11:38.782 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:11:38.783 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:38.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.784 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:11:38.838 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:11:38.838 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:11:38.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:11:38.906 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:11:38.965 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:11:38.966 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:11:38.967 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:11:38.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:38.968 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:11:38.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:11:38.976+0000 7efe435ea480 -1 init, newargv = 0x5610fc299780 newargc=15 2024-05-09T03:11:38.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[51190]: starting ceph client 2024-05-09T03:11:38.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[51190]: starting fuse 2024-05-09T03:11:38.993 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:11:38.994 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:11:38.994 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:11:38.995 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:11:39.115 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.51190.asok 2024-05-09T03:11:39.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:39.126 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51190.asok 2024-05-09T03:11:39.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:39.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51190.asok status 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "51190", 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:11:39.274 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5562, 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5562 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2037454469 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:11:39.275 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2037454469 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5562 192.168.0.1:0/2037454469", 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2037454469", 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 30, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 43, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:11:39.276 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:39.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:39.283 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:11:39.283 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:39.299 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:11:39.300 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:39.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:39.300 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:39.367 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/b') 2024-05-09T03:11:39.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:11:39.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:39.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-05-09T03:11:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.592+0000 7f9364d4d640 1 Processor -- start 2024-05-09T03:11:39.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.592+0000 7f9364d4d640 1 -- start start 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f9364d4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936006e5a0 0x7f936006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f9364d4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93600b93f0 con 0x7f936009d0b0 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f9364d4d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93600b9690 con 0x7f936009cca0 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f9364d4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93600d0020 con 0x7f936006e5a0 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f936009d0b0 0x7f93600d1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57050/0 (socket says 172.21.15.46:57050) 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935ed9d640 1 -- 172.21.15.46:0/678227859 learned_addr learned my addr 172.21.15.46:0/678227859 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:39.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935e59c640 1 --2- 172.21.15.46:0/678227859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936006e5a0 0x7f936006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443143453 0 0) 0x7f93600b93f0 con 0x7f936009d0b0 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9348003520 con 0x7f936009d0b0 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4292425816 0 0) 0x7f93600b9690 con 0x7f936009cca0 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93600b93f0 con 0x7f936009cca0 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2044149063 0 0) 0x7f9348003520 con 0x7f936009d0b0 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935e59c640 1 -- 172.21.15.46:0/678227859 >> v1:172.21.15.88:6789/0 conn(0x7f936009cca0 legacy=0x7f93600cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935e59c640 1 -- 172.21.15.46:0/678227859 >> v1:172.21.15.46:6789/0 conn(0x7f936009d0b0 legacy=0x7f93600d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.594+0000 7f935e59c640 1 -- 172.21.15.46:0/678227859 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93600d3220 con 0x7f936006e5a0 2024-05-09T03:11:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.595+0000 7f935e59c640 1 --2- 172.21.15.46:0/678227859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936006e5a0 0x7f936006e9b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93500097d0 tx=0x7f935002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c66fbd408a2f820 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.595+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350037070 con 0x7f936006e5a0 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.595+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f935002ab10 con 0x7f936006e5a0 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.595+0000 7f935d59a640 1 -- 172.21.15.46:0/678227859 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9358001a90 con 0x7f936006e5a0 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 -- 172.21.15.46:0/678227859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936006e5a0 msgr2=0x7f936006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 --2- 172.21.15.46:0/678227859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936006e5a0 0x7f936006e9b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93500097d0 tx=0x7f935002a080 comp rx=0 tx=0).stop 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 -- 172.21.15.46:0/678227859 shutdown_connections 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 --2- 172.21.15.46:0/678227859 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936006e5a0 0x7f936006e9b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 -- 172.21.15.46:0/678227859 >> 172.21.15.46:0/678227859 conn(0x7f93600d05f0 msgr2=0x7f93600d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 -- 172.21.15.46:0/678227859 shutdown_connections 2024-05-09T03:11:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.596+0000 7f9364d4d640 1 -- 172.21.15.46:0/678227859 wait complete. 2024-05-09T03:11:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.598+0000 7f9364d4d640 1 Processor -- start 2024-05-09T03:11:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.598+0000 7f9364d4d640 1 -- start start 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f9364d4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 0x7f93601f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f9364d4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f936009cca0 0x7f936009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f9364d4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936009d0b0 0x7f93601f9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f9364d4d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f93600cf2e0 con 0x7f936006e5a0 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f9364d4d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9360074b20 con 0x7f936009cca0 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f9364d4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93600c9430 con 0x7f936009d0b0 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 0x7f93601f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 0x7f93601f5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39522/0 (socket says 172.21.15.46:39522) 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 -- 172.21.15.46:0/2503401670 learned_addr learned my addr 172.21.15.46:0/2503401670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935ed9d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936009d0b0 0x7f93601f9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f936009cca0 0x7f936009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 -- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936009d0b0 msgr2=0x7f93601f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936009d0b0 0x7f93601f9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 -- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f936009cca0 msgr2=0x7f936009e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f936009cca0 0x7f936009e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 -- 172.21.15.46:0/2503401670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936009f220 con 0x7f936006e5a0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.600+0000 7f935e59c640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 0x7f93601f5d10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9350009720 tx=0x7f93500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.601+0000 7f934effd640 1 -- 172.21.15.46:0/2503401670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350041070 con 0x7f936006e5a0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.601+0000 7f934effd640 1 -- 172.21.15.46:0/2503401670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f935003c660 con 0x7f936006e5a0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.601+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936009f500 con 0x7f936006e5a0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.601+0000 7f934effd640 1 -- 172.21.15.46:0/2503401670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350037040 con 0x7f936006e5a0 2024-05-09T03:11:39.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.601+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93600c9170 con 0x7f936006e5a0 2024-05-09T03:11:39.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.602+0000 7f934effd640 1 -- 172.21.15.46:0/2503401670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9350044070 con 0x7f936006e5a0 2024-05-09T03:11:39.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.603+0000 7f934effd640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9340076300 0x7f9340078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.603+0000 7f934effd640 1 -- 172.21.15.46:0/2503401670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f93500b7a40 con 0x7f936006e5a0 2024-05-09T03:11:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.603+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9340076300 0x7f9340078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.604+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9340076300 0x7f9340078810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f93600d1300 tx=0x7f9354009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.604+0000 7f936bc02740 1 Processor -- start 2024-05-09T03:11:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.604+0000 7f936bc02740 1 -- start start 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.605+0000 7f936bc02740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 0x55f59825f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.605+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e674d0 0x55f59827a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.605+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e6f6b0 0x55f59827a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.605+0000 7f936bc02740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f5981981a0 con 0x55f597e90390 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.605+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f598151170 con 0x55f597e6f6b0 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.605+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f598194160 con 0x55f597e674d0 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 0x55f59825f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 0x55f59825f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39528/0 (socket says 172.21.15.46:39528) 2024-05-09T03:11:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 -- 172.21.15.46:0/2500281652 learned_addr learned my addr 172.21.15.46:0/2500281652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935ed9d640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e6f6b0 0x55f59827a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e674d0 0x55f59827a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 -- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e674d0 msgr2=0x55f59827a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e674d0 0x55f59827a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 -- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e6f6b0 msgr2=0x55f59827a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e6f6b0 0x55f59827a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 -- 172.21.15.46:0/2500281652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f598163390 con 0x55f597e90390 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935ed9d640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e6f6b0 0x55f59827a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e674d0 0x55f59827a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f935e59c640 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 0x55f59825f510 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9350032fd0 tx=0x7f9350080be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994d4445da47f707 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f933ffff640 1 -- 172.21.15.46:0/2500281652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935004b070 con 0x55f597e90390 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f933ffff640 1 -- 172.21.15.46:0/2500281652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9350080860 con 0x55f597e90390 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.606+0000 7f933ffff640 1 -- 172.21.15.46:0/2500281652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350080d50 con 0x55f597e90390 2024-05-09T03:11:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 -- 172.21.15.46:0/2500281652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 msgr2=0x55f59825f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 0x55f59825f510 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9350032fd0 tx=0x7f9350080be0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 -- 172.21.15.46:0/2500281652 shutdown_connections 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e6f6b0 0x55f59827a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e674d0 0x55f59827a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 --2- 172.21.15.46:0/2500281652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e90390 0x55f59825f510 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 -- 172.21.15.46:0/2500281652 >> 172.21.15.46:0/2500281652 conn(0x55f597e844e0 msgr2=0x55f598234b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 -- 172.21.15.46:0/2500281652 shutdown_connections 2024-05-09T03:11:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.607+0000 7f936bc02740 1 -- 172.21.15.46:0/2500281652 wait complete. 2024-05-09T03:11:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.608+0000 7f936bc02740 1 -- start start 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.608+0000 7f936bc02740 1 Processor -- start 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 0x55f598261d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e6f6b0 0x55f598272dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e90390 0x55f59822ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f59811e8c0 con 0x55f597e674d0 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f59828a350 con 0x55f597e90390 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f598262540 con 0x55f597e6f6b0 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f936bc02740 10 client.0 _ll_register_callbacks cb 0x55f59813a860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 0x55f598261d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 0x55f598261d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39534/0 (socket says 172.21.15.46:39534) 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 -- 172.21.15.46:0/2513685613 learned_addr learned my addr 172.21.15.46:0/2513685613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935ed9d640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e90390 0x55f59822ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e6f6b0 0x55f598272dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 -- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e6f6b0 msgr2=0x55f598272dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e6f6b0 0x55f598272dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 -- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e90390 msgr2=0x55f59822ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e90390 0x55f59822ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 -- 172.21.15.46:0/2513685613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f597f727d0 con 0x55f597e674d0 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935ed9d640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e90390 0x55f59822ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e6f6b0 0x55f598272dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f935e59c640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 0x55f598261d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9350000c00 tx=0x7f9350032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21082efcbcfc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f933d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.610+0000 7f933d7fa640 1 -- 172.21.15.46:0/2513685613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9350085050 con 0x55f597e674d0 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f933d7fa640 1 -- 172.21.15.46:0/2513685613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f93500a2120 con 0x55f597e674d0 2024-05-09T03:11:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9350080860 con 0x55f597e674d0 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f933d7fa640 1 -- 172.21.15.46:0/2513685613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9350089a70 con 0x55f597e674d0 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 10 client.5574 fetch_fsmap learned FSMap version 30 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x55f597efb970 con 0x55f597e674d0 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f933d7fa640 1 -- 172.21.15.46:0/2513685613 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9350080860 con 0x55f597e674d0 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f933d7fa640 1 -- 172.21.15.46:0/2513685613 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7f935007ffb0 con 0x55f597e674d0 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 10 client.5574 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 10 client.5574 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 20 client.5574 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f597e4ef60 0x55f5982897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 4 client.5574 mds_command: new command op to 5358 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 --> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5981a5db0 con 0x55f597e4ef60 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.611+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f597e4ef60 0x55f5982897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f933d7fa640 10 client.5574 ms_handle_connect on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f935dd9b640 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f597e4ef60 0x55f5982897c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4dd454f627132ba server_cookie=c6e9ed6fb4220d41 in_seq=0 out_seq=0 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f933d7fa640 1 -- 172.21.15.46:0/2513685613 <== mds.0 v2:172.21.15.46:6833/2630572362 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f5981a5db0 con 0x55f597e4ef60 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f933d7fa640 10 client.5574 handle_command_reply: tid=0 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f936bc02740 1 client.5574 shutdown 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 msgr2=0x55f598261d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.612+0000 7f936bc02740 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 0x55f598261d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9350000c00 tx=0x7f9350032f60 comp rx=0 tx=0).stop 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 shutdown_connections 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f597e4ef60 0x55f5982897c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e90390 0x55f59822ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e6f6b0 0x55f598272dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f933d7fa640 0 client.5574 ms_handle_reset on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 --2- 172.21.15.46:0/2513685613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e674d0 0x55f598261d40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 >> 172.21.15.46:0/2513685613 conn(0x55f597e844e0 msgr2=0x55f598234b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 shutdown_connections 2024-05-09T03:11:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 1 -- 172.21.15.46:0/2513685613 wait complete. 2024-05-09T03:11:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.613+0000 7f936bc02740 20 client.5574 trim_cache size 0 max 16384 2024-05-09T03:11:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.628+0000 7f936bc02740 1 Processor -- start 2024-05-09T03:11:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.628+0000 7f936bc02740 1 -- start start 2024-05-09T03:11:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f5982354a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f936bc02740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f59827bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f59827c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f936bc02740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9360202d00 con 0x55f597e6f6b0 2024-05-09T03:11:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f93600c8f50 con 0x55f597e674d0 2024-05-09T03:11:39.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93600c82b0 con 0x55f597e90390 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f59827bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f59827bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39536/0 (socket says 172.21.15.46:39536) 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 -- 172.21.15.46:0/1042524535 learned_addr learned my addr 172.21.15.46:0/1042524535 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 -- 172.21.15.46:0/1042524535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 msgr2=0x55f59827c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f59827c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 -- 172.21.15.46:0/1042524535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 msgr2=0x55f5982354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f5982354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 -- 172.21.15.46:0/1042524535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f597e40f60 con 0x55f597e6f6b0 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f935dd9b640 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f59827bbd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f93600b93a0 tx=0x7f9354012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ffc7ba47253c44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f933f7fe640 1 -- 172.21.15.46:0/1042524535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935402a030 con 0x55f597e6f6b0 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f933f7fe640 1 -- 172.21.15.46:0/1042524535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93540238a0 con 0x55f597e6f6b0 2024-05-09T03:11:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.630+0000 7f933f7fe640 1 -- 172.21.15.46:0/1042524535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354023a50 con 0x55f597e6f6b0 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 -- 172.21.15.46:0/1042524535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 msgr2=0x55f59827bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f59827bbd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f93600b93a0 tx=0x7f9354012000 comp rx=0 tx=0).stop 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 -- 172.21.15.46:0/1042524535 shutdown_connections 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f59827c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f59827bbd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 --2- 172.21.15.46:0/1042524535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f5982354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 -- 172.21.15.46:0/1042524535 >> 172.21.15.46:0/1042524535 conn(0x55f597e844e0 msgr2=0x55f598234860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 -- 172.21.15.46:0/1042524535 shutdown_connections 2024-05-09T03:11:39.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.631+0000 7f936bc02740 1 -- 172.21.15.46:0/1042524535 wait complete. 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.632+0000 7f936bc02740 1 -- start start 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.632+0000 7f936bc02740 1 Processor -- start 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f598288e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f598293340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f598270170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f598261900 con 0x55f597e6f6b0 2024-05-09T03:11:39.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f93600c8900 con 0x55f597e674d0 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f5982571f0 con 0x55f597e90390 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f936bc02740 10 client.0 _ll_register_callbacks cb 0x7f935007cc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f598293340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f598293340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39550/0 (socket says 172.21.15.46:39550) 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 -- 172.21.15.46:0/1355642576 learned_addr learned my addr 172.21.15.46:0/1355642576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935ed9d640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f598270170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935e59c640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f598288e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 -- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 msgr2=0x55f598270170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f598270170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 -- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 msgr2=0x55f598288e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f598288e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 -- 172.21.15.46:0/1355642576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5982f6570 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f935dd9b640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f598293340 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f935400f7e0 tx=0x7f9354007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af360c174dd24635 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f933d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.634+0000 7f933d7fa640 1 -- 172.21.15.46:0/1355642576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9354023ce0 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f933d7fa640 1 -- 172.21.15.46:0/1355642576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f9354026150 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93540238a0 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f933d7fa640 1 -- 172.21.15.46:0/1355642576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9354026350 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 10 client.5580 fetch_fsmap learned FSMap version 30 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x55f598272350 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f933d7fa640 1 -- 172.21.15.46:0/1355642576 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93540238a0 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f933d7fa640 1 -- 172.21.15.46:0/1355642576 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7f93540264f0 con 0x55f597e6f6b0 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 10 client.5580 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T03:11:39.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 10 client.5580 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 20 client.5580 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f5982706b0 0x55f59827b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 4 client.5580 mds_command: new command op to 5358 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 --> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f59819fc20 con 0x55f5982706b0 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.635+0000 7f935e59c640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f5982706b0 0x55f59827b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.636+0000 7f933d7fa640 10 client.5580 ms_handle_connect on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:39.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.636+0000 7f935e59c640 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f5982706b0 0x55f59827b960 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29eff40e6940a602 server_cookie=8c68eee1c894cd40 in_seq=0 out_seq=0 2024-05-09T03:11:39.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f933d7fa640 1 -- 172.21.15.46:0/1355642576 <== mds.0 v2:172.21.15.46:6833/2630572362 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55f59819fc20 con 0x55f5982706b0 2024-05-09T03:11:39.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f933d7fa640 10 client.5580 handle_command_reply: tid=0 2024-05-09T03:11:39.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 client.5580 shutdown 2024-05-09T03:11:39.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 msgr2=0x55f598293340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f598293340 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f935400f7e0 tx=0x7f9354007d90 comp rx=0 tx=0).stop 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 shutdown_connections 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:6833/2630572362,v1:172.21.15.46:6835/2630572362] conn(0x55f5982706b0 0x55f59827b960 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f597e90390 0x55f598270170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f933d7fa640 0 client.5580 ms_handle_reset on v2:172.21.15.46:6833/2630572362 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f597e6f6b0 0x55f598293340 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 --2- 172.21.15.46:0/1355642576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f597e674d0 0x55f598288e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 >> 172.21.15.46:0/1355642576 conn(0x55f597e844e0 msgr2=0x55f59825f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 shutdown_connections 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 1 -- 172.21.15.46:0/1355642576 wait complete. 2024-05-09T03:11:39.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.646+0000 7f936bc02740 20 client.5580 trim_cache size 0 max 16384 2024-05-09T03:11:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:39.649 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0 2024-05-09T03:11:39.649 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9340076300 msgr2=0x7f9340078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9340076300 0x7f9340078810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f93600d1300 tx=0x7f9354009040 comp rx=0 tx=0).stop 2024-05-09T03:11:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 msgr2=0x7f93601f5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 0x7f93601f5d10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9350009720 tx=0x7f93500046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 shutdown_connections 2024-05-09T03:11:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f936009d0b0 0x7f93601f9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f936009cca0 0x7f936009e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9340076300 0x7f9340078810 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 --2- 172.21.15.46:0/2503401670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f936006e5a0 0x7f93601f5d10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 >> 172.21.15.46:0/2503401670 conn(0x7f93600d05f0 msgr2=0x7f93600d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 shutdown_connections 2024-05-09T03:11:39.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.647+0000 7f9364d4d640 1 -- 172.21.15.46:0/2503401670 wait complete. 2024-05-09T03:11:39.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-05-09T03:11:39.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.776+0000 7f20ed9c7640 1 Processor -- start 2024-05-09T03:11:39.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.776+0000 7f20ed9c7640 1 -- start start 2024-05-09T03:11:39.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20ed9c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 0x7f20e80cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20ed9c7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e80cf5b0 con 0x7f20e8071570 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20ed9c7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20e8072090 con 0x7f20e806e4e0 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20ed9c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20e80d2c30 con 0x7f20e80cec60 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20e8071570 0x7f20e8071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57052/0 (socket says 172.21.15.46:57052) 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e77fe640 1 -- 172.21.15.46:0/927361896 learned_addr learned my addr 172.21.15.46:0/927361896 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e67fc640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 0x7f20e80cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 853400308 0 0) 0x7f20d8001490 con 0x7f20e8071570 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20d4003520 con 0x7f20e8071570 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 697418139 0 0) 0x7f20d4003520 con 0x7f20e8071570 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 msgr2=0x7f20e80cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 0x7f20e80cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 >> v1:172.21.15.88:6789/0 conn(0x7f20e806e4e0 legacy=0x7f20e81603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.778+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e80d2f10 con 0x7f20e8071570 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f20d8001c40 con 0x7f20e8071570 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 >> v1:172.21.15.46:6789/0 conn(0x7f20e8071570 legacy=0x7f20e8071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 0x7f20d40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20d4003f90 0x7f20d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d4004d40 0x7f20d4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20e80d2c30 con 0x7f20d4003f90 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20d400bb80 con 0x7f20d4004d40 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20d400bd80 con 0x7f20e80cec60 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e67fc640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20d4003f90 0x7f20d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e6ffd640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 0x7f20d40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e77fe640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d4004d40 0x7f20d4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e67fc640 1 -- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 msgr2=0x7f20d40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e67fc640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20e80cec60 0x7f20d40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e67fc640 1 -- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d4004d40 msgr2=0x7f20d4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e67fc640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d4004d40 0x7f20d4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.779+0000 7f20e67fc640 1 -- 172.21.15.46:0/927361896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d4003520 con 0x7f20d4003f90 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.780+0000 7f20e67fc640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20d4003f90 0x7f20d4004800 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009ac0 tx=0x7f20d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97714fa24283935c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.780+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0036070 con 0x7f20d4003f90 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.780+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d0031400 con 0x7f20d4003f90 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.780+0000 7f20e5ffb640 1 -- 172.21.15.46:0/927361896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0009690 con 0x7f20d4003f90 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20ed9c7640 1 -- 172.21.15.46:0/927361896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20d4003f90 msgr2=0x7f20d4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/927361896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20d4003f90 0x7f20d4004800 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009ac0 tx=0x7f20d4005830 comp rx=0 tx=0).stop 2024-05-09T03:11:39.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20e77fe640 1 -- 172.21.15.46:0/927361896 reap_dead start 2024-05-09T03:11:39.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20ed9c7640 1 -- 172.21.15.46:0/927361896 shutdown_connections 2024-05-09T03:11:39.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20ed9c7640 1 -- 172.21.15.46:0/927361896 >> 172.21.15.46:0/927361896 conn(0x7f20e80d3ae0 msgr2=0x7f20e80d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20ed9c7640 1 -- 172.21.15.46:0/927361896 shutdown_connections 2024-05-09T03:11:39.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.781+0000 7f20ed9c7640 1 -- 172.21.15.46:0/927361896 wait complete. 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.782+0000 7f20ed9c7640 1 Processor -- start 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.783+0000 7f20ed9c7640 1 -- start start 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20ed9c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20d4003f90 0x7f20e80c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20ed9c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e8150d90 0x7f20e815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20ed9c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 0x7f20e80c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20ed9c7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20e80d37b0 con 0x7f20e815ad60 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20ed9c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20e80d34d0 con 0x7f20e8150d90 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20ed9c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20e806e8f0 con 0x7f20d4003f90 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 0x7f20e80c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 0x7f20e80c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39570/0 (socket says 172.21.15.46:39570) 2024-05-09T03:11:39.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 -- 172.21.15.46:0/1518842713 learned_addr learned my addr 172.21.15.46:0/1518842713 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e67fc640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e8150d90 0x7f20e815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e6ffd640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20d4003f90 0x7f20e80c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 -- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20d4003f90 msgr2=0x7f20e80c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20d4003f90 0x7f20e80c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 -- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e8150d90 msgr2=0x7f20e815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.784+0000 7f20e77fe640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e8150d90 0x7f20e815a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20e77fe640 1 -- 172.21.15.46:0/1518842713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e815b170 con 0x7f20e815ad60 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20e77fe640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 0x7f20e80c98c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f20d4004780 tx=0x7f20d800af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d80077e0 con 0x7f20e815ad60 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d8018070 con 0x7f20e815ad60 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e80cbc30 con 0x7f20e815ad60 2024-05-09T03:11:39.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d80133f0 con 0x7f20e815ad60 2024-05-09T03:11:39.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.785+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e80cc1f0 con 0x7f20e815ad60 2024-05-09T03:11:39.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.786+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e806a270 con 0x7f20e815ad60 2024-05-09T03:11:39.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.787+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f20d8013590 con 0x7f20e815ad60 2024-05-09T03:11:39.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.787+0000 7f20dffff640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20c80762d0 0x7f20c80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:39.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.787+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f20d8097b00 con 0x7f20e815ad60 2024-05-09T03:11:39.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.788+0000 7f20e6ffd640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20c80762d0 0x7f20c80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:39.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.788+0000 7f20e6ffd640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20c80762d0 0x7f20c80787e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f20e00011a0 tx=0x7f20e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:39.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.792+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f20e815b170 con 0x7f20e815ad60 2024-05-09T03:11:39.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.984+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) -- 0x7f20e80cb9c0 con 0x7f20e815ad60 2024-05-09T03:11:39.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.988+0000 7f20dffff640 1 -- 172.21.15.46:0/1518842713 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v13)=0 v13) ==== 135+0+0 (secure 0 0 0) 0x7f20e80cf5b0 con 0x7f20e815ad60 2024-05-09T03:11:39.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20c80762d0 msgr2=0x7f20c80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20c80762d0 0x7f20c80787e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f20e00011a0 tx=0x7f20e0009040 comp rx=0 tx=0).stop 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 msgr2=0x7f20e80c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 0x7f20e80c98c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f20d4004780 tx=0x7f20d800af60 comp rx=0 tx=0).stop 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 shutdown_connections 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20c80762d0 0x7f20c80787e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20e815ad60 0x7f20e80c98c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20e8150d90 0x7f20e815a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 --2- 172.21.15.46:0/1518842713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20d4003f90 0x7f20e80c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 >> 172.21.15.46:0/1518842713 conn(0x7f20e80d3ae0 msgr2=0x7f20e80b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 shutdown_connections 2024-05-09T03:11:39.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:39.990+0000 7f20ed9c7640 1 -- 172.21.15.46:0/1518842713 wait complete. 2024-05-09T03:11:40.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-05-09T03:11:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.120+0000 7fc9db6b1640 1 Processor -- start 2024-05-09T03:11:40.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.120+0000 7fc9db6b1640 1 -- start start 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9db6b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9d40b5140 0x7fc9d40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9db6b1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9d40d3fc0 0x7fc9d40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9db6b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9d40b80b0 con 0x7fc9d40b7ca0 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9db6b1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d40d3480 con 0x7fc9d40b5140 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9db6b1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d40b5550 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d944d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9d40b5140 0x7fc9d40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d8c4c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9d40b7ca0 0x7fc9d40d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57066/0 (socket says 172.21.15.46:57066) 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d8c4c640 1 -- 172.21.15.46:0/2692406410 learned_addr learned my addr 172.21.15.46:0/2692406410 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d9c4e640 1 --2- 172.21.15.46:0/2692406410 >> v2:172.21.15.88:3300/0 conn(0x7fc9d40d3fc0 0x7fc9d40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:40.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d944d640 1 -- 172.21.15.46:0/2692406410 >> v2:172.21.15.88:3300/0 conn(0x7fc9d40d3fc0 msgr2=0x7fc9d40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d944d640 1 --2- 172.21.15.46:0/2692406410 >> v2:172.21.15.88:3300/0 conn(0x7fc9d40d3fc0 0x7fc9d40b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d944d640 1 -- 172.21.15.46:0/2692406410 >> v1:172.21.15.46:6789/0 conn(0x7fc9d40b7ca0 legacy=0x7fc9d40d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9d944d640 1 -- 172.21.15.46:0/2692406410 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d40d44e0 con 0x7fc9d40b5140 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.122+0000 7fc9cbfff640 1 -- 172.21.15.46:0/2692406410 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123116805 0 0) 0x7fc9c4001110 con 0x7fc9d40b7ca0 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.123+0000 7fc9d944d640 1 --2- 172.21.15.46:0/2692406410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40b5140 0x7fc9d40b7760 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc9d408f180 tx=0x7fc9d002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b59482d17c0908c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.123+0000 7fc9cbfff640 1 -- 172.21.15.46:0/2692406410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0037070 con 0x7fc9d40b5140 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.123+0000 7fc9cbfff640 1 -- 172.21.15.46:0/2692406410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d0032440 con 0x7fc9d40b5140 2024-05-09T03:11:40.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.123+0000 7fc9cbfff640 1 -- 172.21.15.46:0/2692406410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0032810 con 0x7fc9d40b5140 2024-05-09T03:11:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 -- 172.21.15.46:0/2692406410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40b5140 msgr2=0x7fc9d40b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/2692406410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40b5140 0x7fc9d40b7760 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc9d408f180 tx=0x7fc9d002a230 comp rx=0 tx=0).stop 2024-05-09T03:11:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 -- 172.21.15.46:0/2692406410 shutdown_connections 2024-05-09T03:11:40.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/2692406410 >> v2:172.21.15.88:3300/0 conn(0x7fc9d40d3fc0 0x7fc9d40b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/2692406410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40b5140 0x7fc9d40b7760 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 -- 172.21.15.46:0/2692406410 >> 172.21.15.46:0/2692406410 conn(0x7fc9d408d3f0 msgr2=0x7fc9d408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 -- 172.21.15.46:0/2692406410 shutdown_connections 2024-05-09T03:11:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.124+0000 7fc9db6b1640 1 -- 172.21.15.46:0/2692406410 wait complete. 2024-05-09T03:11:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.126+0000 7fc9db6b1640 1 Processor -- start 2024-05-09T03:11:40.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.126+0000 7fc9db6b1640 1 -- start start 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9db6b1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9d40b5140 0x7fc9d4159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9db6b1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d40b7ca0 0x7fc9d415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9db6b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 0x7fc9d415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9db6b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d40d4d80 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9db6b1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d40d4aa0 con 0x7fc9d40b7ca0 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9db6b1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d40d5580 con 0x7fc9d40b5140 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9d9c4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 0x7fc9d415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9d9c4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 0x7fc9d415a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39596/0 (socket says 172.21.15.46:39596) 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9d9c4e640 1 -- 172.21.15.46:0/3380016642 learned_addr learned my addr 172.21.15.46:0/3380016642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9d8c4c640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d40b7ca0 0x7fc9d415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:40.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.127+0000 7fc9d944d640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9d40b5140 0x7fc9d4159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d9c4e640 1 -- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9d40b5140 msgr2=0x7fc9d4159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d9c4e640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9d40b5140 0x7fc9d4159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d9c4e640 1 -- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d40b7ca0 msgr2=0x7fc9d415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d9c4e640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d40b7ca0 0x7fc9d415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d9c4e640 1 -- 172.21.15.46:0/3380016642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d41488c0 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d944d640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9d40b5140 0x7fc9d4159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9d9c4e640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 0x7fc9d415a8a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc007e90 tx=0x7fc9cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc014420 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9cc019070 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d4159060 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc014800 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.128+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d4159620 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.129+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d4148a90 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.130+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc9cc0149a0 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.130+0000 7fc9c9ffb640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9b8076300 0x7fc9b8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.130+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7fc9cc097da0 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.131+0000 7fc9d944d640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9b8076300 0x7fc9b8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.131+0000 7fc9d944d640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9b8076300 0x7fc9b8078810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0034000 tx=0x7fc9d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.135+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc9cc0610b0 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.328+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) -- 0x7fc9d406a270 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.332+0000 7fc9c9ffb640 1 -- 172.21.15.46:0/3380016642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v14)=0 v14) ==== 138+0+0 (secure 0 0 0) 0x7fc9cc0653b0 con 0x7fc9d40d3fc0 2024-05-09T03:11:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9b8076300 msgr2=0x7fc9b8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:40.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9b8076300 0x7fc9b8078810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0034000 tx=0x7fc9d002e000 comp rx=0 tx=0).stop 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 msgr2=0x7fc9d415a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 0x7fc9d415a8a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc007e90 tx=0x7fc9cc00e4b0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 shutdown_connections 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9b8076300 0x7fc9b8078810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9d40d3fc0 0x7fc9d415a8a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d40b7ca0 0x7fc9d415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 --2- 172.21.15.46:0/3380016642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9d40b5140 0x7fc9d4159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.334+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 >> 172.21.15.46:0/3380016642 conn(0x7fc9d408d3f0 msgr2=0x7fc9d40b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.335+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 shutdown_connections 2024-05-09T03:11:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:40.335+0000 7fc9db6b1640 1 -- 172.21.15.46:0/3380016642 wait complete. 2024-05-09T03:11:45.347 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mv a/b a/c; sync .') 2024-05-09T03:11:45.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:11:45.412 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:11:45.412+0000 7f19c69a7640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5562:17 nref=7 cr=0x55e08fdb4e00)) (dversion lock w=1 last_client=5562) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55e08faab400] 2024-05-09T03:11:45.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:11:45.421 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:45.421 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:45.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:11:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.597+0000 7faaa9053640 1 Processor -- start 2024-05-09T03:11:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.598+0000 7faaa9053640 1 -- start start 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa9053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa40d3730 0x7faaa406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa9053640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaa406e7e0 0x7faaa4164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa9053640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaa409d470 con 0x7faaa4164ec0 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa9053640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faaa409cfe0 con 0x7faaa406e7e0 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa9053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faaa40b94a0 con 0x7faaa40d3730 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaa406e7e0 0x7faaa4164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaa406e7e0 0x7faaa4164980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39600/0 (socket says 172.21.15.46:39600) 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 -- 172.21.15.46:0/3140106897 learned_addr learned my addr 172.21.15.46:0/3140106897 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 -- 172.21.15.46:0/3140106897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa40d3730 msgr2=0x7faaa406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa2d9d640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa40d3730 0x7faaa406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa40d3730 0x7faaa406e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 -- 172.21.15.46:0/3140106897 >> v1:172.21.15.88:6789/0 conn(0x7faaa4164ec0 legacy=0x7faaa40ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa259c640 1 -- 172.21.15.46:0/3140106897 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa40d11b0 con 0x7faaa406e7e0 2024-05-09T03:11:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.599+0000 7faaa2d9d640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa40d3730 0x7faaa406e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.600+0000 7faaa259c640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa406e7e0 0x7faaa4164980 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009870 tx=0x7faa8c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d0f9e18a47ebd8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.600+0000 7faaa1d9b640 1 -- 172.21.15.46:0/3140106897 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c037070 con 0x7faaa406e7e0 2024-05-09T03:11:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.600+0000 7faaa1d9b640 1 -- 172.21.15.46:0/3140106897 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c032440 con 0x7faaa406e7e0 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.600+0000 7faaa1d9b640 1 -- 172.21.15.46:0/3140106897 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c032810 con 0x7faaa406e7e0 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 -- 172.21.15.46:0/3140106897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa406e7e0 msgr2=0x7faaa4164980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa406e7e0 0x7faaa4164980 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009870 tx=0x7faa8c02a130 comp rx=0 tx=0).stop 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 -- 172.21.15.46:0/3140106897 shutdown_connections 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa406e7e0 0x7faaa4164980 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 --2- 172.21.15.46:0/3140106897 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa40d3730 0x7faaa406e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 -- 172.21.15.46:0/3140106897 >> 172.21.15.46:0/3140106897 conn(0x7faaa40d05f0 msgr2=0x7faaa40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 -- 172.21.15.46:0/3140106897 shutdown_connections 2024-05-09T03:11:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.601+0000 7faaa9053640 1 -- 172.21.15.46:0/3140106897 wait complete. 2024-05-09T03:11:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.603+0000 7faaa9053640 1 Processor -- start 2024-05-09T03:11:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.603+0000 7faaa9053640 1 -- start start 2024-05-09T03:11:45.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.604+0000 7faaa9053640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaa406e7e0 0x7faaa415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.604+0000 7faaa9053640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 0x7faaa41fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.604+0000 7faaa9053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa4164ec0 0x7faaa40cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.604+0000 7faaa9053640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faaa408cd60 con 0x7faaa40d3730 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.604+0000 7faaa9053640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faaa40b9160 con 0x7faaa406e7e0 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.604+0000 7faaa9053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faaa408cee0 con 0x7faaa4164ec0 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 0x7faaa41fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 0x7faaa41fa5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39608/0 (socket says 172.21.15.46:39608) 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 -- 172.21.15.46:0/693363362 learned_addr learned my addr 172.21.15.46:0/693363362 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa2d9d640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaa406e7e0 0x7faaa415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa359e640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa4164ec0 0x7faaa40cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 -- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa4164ec0 msgr2=0x7faaa40cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa4164ec0 0x7faaa40cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 -- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaa406e7e0 msgr2=0x7faaa415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaa406e7e0 0x7faaa415bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 -- 172.21.15.46:0/693363362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa40c8ca0 con 0x7faaa40d3730 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa359e640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa4164ec0 0x7faaa40cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa259c640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 0x7faaa41fa5a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009870 tx=0x7faa8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c041020 con 0x7faaa40d3730 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c030970 con 0x7faaa40d3730 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa40c8f80 con 0x7faaa40d3730 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c037040 con 0x7faaa40d3730 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.605+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa40c9540 con 0x7faaa40d3730 2024-05-09T03:11:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.606+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaa40c8a10 con 0x7faaa40d3730 2024-05-09T03:11:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.607+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faa8c044020 con 0x7faaa40d3730 2024-05-09T03:11:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.608+0000 7faa9b7fe640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa840762d0 0x7faa840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.608+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7faa8c0b8000 con 0x7faaa40d3730 2024-05-09T03:11:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.608+0000 7faaa2d9d640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa840762d0 0x7faa840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.608+0000 7faaa2d9d640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa840762d0 0x7faa840787e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faa9400aac0 tx=0x7faa94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.612+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faa8c04b030 con 0x7faaa40d3730 2024-05-09T03:11:45.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.816+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faaa40ccd20 con 0x7faaa40d3730 2024-05-09T03:11:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.816+0000 7faa9b7fe640 1 -- 172.21.15.46:0/693363362 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7faaa409d470 con 0x7faaa40d3730 2024-05-09T03:11:45.817 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:11:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa840762d0 msgr2=0x7faa840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:45.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa840762d0 0x7faa840787e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faa9400aac0 tx=0x7faa94008040 comp rx=0 tx=0).stop 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 msgr2=0x7faaa41fa5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 0x7faaa41fa5a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7faa8c009870 tx=0x7faa8c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 shutdown_connections 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaa4164ec0 0x7faaa40cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa840762d0 0x7faa840787e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa40d3730 0x7faaa41fa5a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 --2- 172.21.15.46:0/693363362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaa406e7e0 0x7faaa415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 >> 172.21.15.46:0/693363362 conn(0x7faaa40d05f0 msgr2=0x7faaa40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 shutdown_connections 2024-05-09T03:11:45.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:45.819+0000 7faaa9053640 1 -- 172.21.15.46:0/693363362 wait complete. 2024-05-09T03:11:45.831 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:11:45.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:45.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:11:45.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[51190]: fuse finished with error 0 and tester_r 0 2024-05-09T03:11:45.857 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:11:45.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:11:45.857 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:11:45.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.069+0000 7f9a2be29640 1 Processor -- start 2024-05-09T03:11:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.070+0000 7f9a2be29640 1 -- start start 2024-05-09T03:11:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a2be29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a241613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a2be29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a240b7ed0 0x7f9a240d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a2be29640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a240d5280 con 0x7f9a240d4800 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a2be29640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9a240b5620 con 0x7f9a240b7ed0 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a2be29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9a241609a0 con 0x7f9a240b5210 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a2a3c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a240b7ed0 0x7f9a240d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a293c4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9a240d4800 0x7f9a240b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59902/0 (socket says 172.21.15.46:59902) 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a293c4640 1 -- 172.21.15.46:0/3457225865 learned_addr learned my addr 172.21.15.46:0/3457225865 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.071+0000 7f9a29bc5640 1 --2- 172.21.15.46:0/3457225865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a241613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.072+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/3457225865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 msgr2=0x7f9a241613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.072+0000 7f9a2a3c6640 1 --2- 172.21.15.46:0/3457225865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a241613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.072+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/3457225865 >> v1:172.21.15.46:6789/0 conn(0x7f9a240d4800 legacy=0x7f9a240b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.072+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/3457225865 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a240d5560 con 0x7f9a240b7ed0 2024-05-09T03:11:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.072+0000 7f9a28bc3640 1 -- 172.21.15.46:0/3457225865 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 525270071 0 0) 0x7f9a240d5280 con 0x7f9a240d4800 2024-05-09T03:11:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.072+0000 7f9a2a3c6640 1 --2- 172.21.15.46:0/3457225865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240b7ed0 0x7f9a240d4c10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009d90 tx=0x7f9a1402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59936b7c549f29f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a28bc3640 1 -- 172.21.15.46:0/3457225865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14007740 con 0x7f9a240b7ed0 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a28bc3640 1 -- 172.21.15.46:0/3457225865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a140078e0 con 0x7f9a240b7ed0 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a28bc3640 1 -- 172.21.15.46:0/3457225865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14007cb0 con 0x7f9a240b7ed0 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 -- 172.21.15.46:0/3457225865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240b7ed0 msgr2=0x7f9a240d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 --2- 172.21.15.46:0/3457225865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240b7ed0 0x7f9a240d4c10 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009d90 tx=0x7f9a1402a5f0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 -- 172.21.15.46:0/3457225865 shutdown_connections 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 --2- 172.21.15.46:0/3457225865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240b7ed0 0x7f9a240d4c10 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 --2- 172.21.15.46:0/3457225865 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a241613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 -- 172.21.15.46:0/3457225865 >> 172.21.15.46:0/3457225865 conn(0x7f9a240d29a0 msgr2=0x7f9a240d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 -- 172.21.15.46:0/3457225865 shutdown_connections 2024-05-09T03:11:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.073+0000 7f9a2be29640 1 -- 172.21.15.46:0/3457225865 wait complete. 2024-05-09T03:11:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.075+0000 7f9a2be29640 1 Processor -- start 2024-05-09T03:11:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.075+0000 7f9a2be29640 1 -- start start 2024-05-09T03:11:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.076+0000 7f9a2be29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a240ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.076+0000 7f9a2be29640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a240d4800 0x7f9a240caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.076+0000 7f9a2be29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 0x7f9a240cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.076+0000 7f9a2be29640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9a240d0ae0 con 0x7f9a240cbf60 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.076+0000 7f9a2be29640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9a24160ea0 con 0x7f9a240d4800 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.076+0000 7f9a2be29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9a24163e60 con 0x7f9a240b5210 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 0x7f9a240cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 0x7f9a240cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39896/0 (socket says 172.21.15.46:39896) 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/98468441 learned_addr learned my addr 172.21.15.46:0/98468441 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a293c4640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a240d4800 0x7f9a240caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a29bc5640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a240ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 msgr2=0x7f9a240ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a240ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a240d4800 msgr2=0x7f9a240caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a240d4800 0x7f9a240caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 -- 172.21.15.46:0/98468441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a240cf360 con 0x7f9a240cbf60 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a293c4640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a240d4800 0x7f9a240caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2a3c6640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 0x7f9a240cb410 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9a1402abe0 tx=0x7f9a14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1403f070 con 0x7f9a240cbf60 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a14031bc0 con 0x7f9a240cbf60 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14044460 con 0x7f9a240cbf60 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a240cf640 con 0x7f9a240cbf60 2024-05-09T03:11:46.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.077+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a240cfc00 con 0x7f9a240cbf60 2024-05-09T03:11:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.078+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a2406a270 con 0x7f9a240cbf60 2024-05-09T03:11:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.079+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9a1403b410 con 0x7f9a240cbf60 2024-05-09T03:11:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.079+0000 7f9a227fc640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a10076300 0x7f9a10078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.080+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f9a140b8e70 con 0x7f9a240cbf60 2024-05-09T03:11:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.080+0000 7f9a29bc5640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a10076300 0x7f9a10078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.080+0000 7f9a29bc5640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a10076300 0x7f9a10078810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c006fd0 tx=0x7f9a1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.084+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9a140820f0 con 0x7f9a240cbf60 2024-05-09T03:11:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.314+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a24162480 con 0x7f9a240cbf60 2024-05-09T03:11:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.315+0000 7f9a227fc640 1 -- 172.21.15.46:0/98468441 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5172 (secure 0 0 0) 0x7f9a140863f0 con 0x7f9a240cbf60 2024-05-09T03:11:46.316 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:46.316 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:29.815636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2630572362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2630572362},{"type":"v1","addr":"172.21.15.46:6835","nonce":2630572362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":3}]} 2024-05-09T03:11:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2024-05-09T03:11:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a10076300 msgr2=0x7f9a10078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a10076300 0x7f9a10078810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c006fd0 tx=0x7f9a1c008040 comp rx=0 tx=0).stop 2024-05-09T03:11:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 msgr2=0x7f9a240cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 0x7f9a240cb410 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9a1402abe0 tx=0x7f9a14004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 shutdown_connections 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a10076300 0x7f9a10078810 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a240cbf60 0x7f9a240cb410 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a240d4800 0x7f9a240caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 --2- 172.21.15.46:0/98468441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9a240b5210 0x7f9a240ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 >> 172.21.15.46:0/98468441 conn(0x7f9a240d29a0 msgr2=0x7f9a24160240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 shutdown_connections 2024-05-09T03:11:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.317+0000 7f9a2be29640 1 -- 172.21.15.46:0/98468441 wait complete. 2024-05-09T03:11:46.329 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.443+0000 7feb8fb00640 1 Processor -- start 2024-05-09T03:11:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.444+0000 7feb8fb00640 1 -- start start 2024-05-09T03:11:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8fb00640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb880b7ef0 0x7feb8816b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8fb00640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb880d49e0 con 0x7feb88160610 2024-05-09T03:11:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8fb00640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb880d4cc0 con 0x7feb880d57a0 2024-05-09T03:11:46.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8fb00640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7feb880d5bb0 con 0x7feb880b7ef0 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8e09d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb880b7ef0 0x7feb8816b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8d09b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feb88160610 0x7feb880b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59916/0 (socket says 172.21.15.46:59916) 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8d09b640 1 -- 172.21.15.46:0/1570670638 learned_addr learned my addr 172.21.15.46:0/1570670638 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.445+0000 7feb8e09d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb880b7ef0 0x7feb8816b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39898/0 (socket says 172.21.15.46:39898) 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.446+0000 7feb8e09d640 1 -- 172.21.15.46:0/1570670638 >> v1:172.21.15.88:6789/0 conn(0x7feb880d57a0 legacy=0x7feb880b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.446+0000 7feb8e09d640 1 -- 172.21.15.46:0/1570670638 >> v1:172.21.15.46:6789/0 conn(0x7feb88160610 legacy=0x7feb880b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.446+0000 7feb8e09d640 1 -- 172.21.15.46:0/1570670638 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb880d4fa0 con 0x7feb880b7ef0 2024-05-09T03:11:46.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.446+0000 7feb8e09d640 1 --2- 172.21.15.46:0/1570670638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb880b7ef0 0x7feb8816b1f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7feb800097a0 tx=0x7feb8002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18a64204340a843 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.446+0000 7feb8c89a640 1 -- 172.21.15.46:0/1570670638 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80037070 con 0x7feb880b7ef0 2024-05-09T03:11:46.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8c89a640 1 -- 172.21.15.46:0/1570670638 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb80032440 con 0x7feb880b7ef0 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8c89a640 1 -- 172.21.15.46:0/1570670638 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80032810 con 0x7feb880b7ef0 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 -- 172.21.15.46:0/1570670638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb880b7ef0 msgr2=0x7feb8816b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 --2- 172.21.15.46:0/1570670638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb880b7ef0 0x7feb8816b1f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7feb800097a0 tx=0x7feb8002a230 comp rx=0 tx=0).stop 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 -- 172.21.15.46:0/1570670638 shutdown_connections 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 --2- 172.21.15.46:0/1570670638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb880b7ef0 0x7feb8816b1f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 -- 172.21.15.46:0/1570670638 >> 172.21.15.46:0/1570670638 conn(0x7feb880743e0 msgr2=0x7feb880d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 -- 172.21.15.46:0/1570670638 shutdown_connections 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.447+0000 7feb8fb00640 1 -- 172.21.15.46:0/1570670638 wait complete. 2024-05-09T03:11:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.449+0000 7feb8fb00640 1 Processor -- start 2024-05-09T03:11:46.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.449+0000 7feb8fb00640 1 -- start start 2024-05-09T03:11:46.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.450+0000 7feb8fb00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb880d57a0 0x7feb881e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.450+0000 7feb8fb00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb88160610 0x7feb881e9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.450+0000 7feb8fb00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 0x7feb881e96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.450+0000 7feb8fb00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb880d15b0 con 0x7feb88147850 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.450+0000 7feb8fb00640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7feb88160a20 con 0x7feb880d57a0 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.450+0000 7feb8fb00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7feb880d1920 con 0x7feb88160610 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 0x7feb881e96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 0x7feb881e96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39914/0 (socket says 172.21.15.46:39914) 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 -- 172.21.15.46:0/3672716285 learned_addr learned my addr 172.21.15.46:0/3672716285 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8d89c640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb880d57a0 0x7feb881e8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8d09b640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb88160610 0x7feb881e9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 -- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb88160610 msgr2=0x7feb881e9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb88160610 0x7feb881e9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 -- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb880d57a0 msgr2=0x7feb881e8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb880d57a0 0x7feb881e8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 -- 172.21.15.46:0/3672716285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb88158220 con 0x7feb88147850 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8e09d640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 0x7feb881e96b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7feb80035040 tx=0x7feb800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80037040 con 0x7feb88147850 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb800306f0 con 0x7feb88147850 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb88157f40 con 0x7feb88147850 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80030ac0 con 0x7feb88147850 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.451+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb88158900 con 0x7feb88147850 2024-05-09T03:11:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.452+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb8806a270 con 0x7feb88147850 2024-05-09T03:11:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.453+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feb80044020 con 0x7feb88147850 2024-05-09T03:11:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.454+0000 7feb867fc640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb74076300 0x7feb74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.454+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7feb800b7a00 con 0x7feb88147850 2024-05-09T03:11:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.454+0000 7feb8d89c640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb74076300 0x7feb74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.454+0000 7feb8d89c640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb74076300 0x7feb74078810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb8806d3b0 tx=0x7feb78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.458+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7feb80080be0 con 0x7feb88147850 2024-05-09T03:11:46.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.689+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb881e61e0 con 0x7feb88147850 2024-05-09T03:11:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.689+0000 7feb867fc640 1 -- 172.21.15.46:0/3672716285 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5172 (secure 0 0 0) 0x7feb80084ee0 con 0x7feb88147850 2024-05-09T03:11:46.690 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:46.690 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:29.815636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5358},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5358":{"gid":5358,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2630572362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2630572362},{"type":"v1","addr":"172.21.15.46:6835","nonce":2630572362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5358,"qdb_cluster":[5358]},"id":3}]} 2024-05-09T03:11:46.690 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 30 2024-05-09T03:11:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb74076300 msgr2=0x7feb74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb74076300 0x7feb74078810 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb8806d3b0 tx=0x7feb78009040 comp rx=0 tx=0).stop 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 msgr2=0x7feb881e96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 0x7feb881e96b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7feb80035040 tx=0x7feb800046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 shutdown_connections 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feb74076300 0x7feb74078810 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb88147850 0x7feb881e96b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feb88160610 0x7feb881e9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 --2- 172.21.15.46:0/3672716285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feb880d57a0 0x7feb881e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 >> 172.21.15.46:0/3672716285 conn(0x7feb880743e0 msgr2=0x7feb880b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 shutdown_connections 2024-05-09T03:11:46.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.691+0000 7feb8fb00640 1 -- 172.21.15.46:0/3672716285 wait complete. 2024-05-09T03:11:46.704 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /a 0 2024-05-09T03:11:46.832 INFO:teuthology.orchestra.run.smithi046.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-05-09T03:11:45.411836+0000","btime":"2024-05-09T03:11:39.464882+0000","mode":16877,"uid":1000,"gid":1316,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-05-09T03:11:45.411836+0000","atime":"2024-05-09T03:11:39.464882+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-05-09T03:11:45.411836+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-05-09T03:11:45.411836+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-05-09T03:11:45.411836+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"dirfrag":1,"caps":0,"dirtyparent":0,"dirty":1,"waiter":0,"authpin":0},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"3","committed_version":"3","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"a/c","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":0,"dirty":1,"authpin":0},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-05-09T03:11:46.833 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-05-09T03:11:45.411836+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-05-09T03:11:39.464882+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-05-09T03:11:39.464882+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-05-09T03:11:45.411836+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "3", "committing_version": "3", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 0, "lock": 0, "request": 0 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/a", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-05-09T03:11:45.411836+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1316, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-05-09T03:11:45.411836+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 0, "caps": 0, "dirfrag": 1, "dirty": 1, "dirtyparent": 0, "lock": 0, "request": 0, "waiter": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iquiesce", "xlock_by": {} }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-05-09T03:11:45.411836+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-05-09T03:11:46.834 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-05-09T03:11:45.411836+0000', 'btime': '2024-05-09T03:11:39.464882+0000', 'mode': 16877, 'uid': 1000, 'gid': 1316, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-05-09T03:11:45.411836+0000', 'atime': '2024-05-09T03:11:39.464882+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'quiesce_block': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-05-09T03:11:45.411836+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-05-09T03:11:45.411836+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-05-09T03:11:45.411836+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'dirfrag': 1, 'caps': 0, 'dirtyparent': 0, 'dirty': 1, 'waiter': 0, 'authpin': 0}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {'gather_set': [], 'state': 'lock', 'type': 'inest', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'flocklock': {}, 'policylock': {}, 'quiescelock': {'gather_set': [], 'state': 'lock', 'type': 'iquiesce', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '3', 'committed_version': '3', 'is_rep': False, 'dir_auth': '', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'dentries': [{'path': 'a/c', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 0, 'dirty': 1, 'authpin': 0}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-05-09T03:11:46.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:11:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.949+0000 7f1671c53640 1 Processor -- start 2024-05-09T03:11:46.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.950+0000 7f1671c53640 1 -- start start 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f1671c53640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f166c0b7e70 0x7f166c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f1671c53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f166c0d3df0 0x7f166c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f1671c53640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f166c0b5550 con 0x7f166c0b5140 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f1671c53640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f166c0d3480 con 0x7f166c0b7e70 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f1671c53640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f166c0d44e0 con 0x7f166c0d3df0 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f166c0b5140 0x7f166c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59920/0 (socket says 172.21.15.46:59920) 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f166c0b7e70 0x7f166c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166b7fe640 1 -- 172.21.15.46:0/3693632567 learned_addr learned my addr 172.21.15.46:0/3693632567 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166bfff640 1 --2- 172.21.15.46:0/3693632567 >> v2:172.21.15.88:3300/0 conn(0x7f166c0d3df0 0x7f166c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166affd640 1 -- 172.21.15.46:0/3693632567 >> v2:172.21.15.88:3300/0 conn(0x7f166c0d3df0 msgr2=0x7f166c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166affd640 1 --2- 172.21.15.46:0/3693632567 >> v2:172.21.15.88:3300/0 conn(0x7f166c0d3df0 0x7f166c0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166affd640 1 -- 172.21.15.46:0/3693632567 >> v1:172.21.15.46:6789/0 conn(0x7f166c0b5140 legacy=0x7f166c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166affd640 1 -- 172.21.15.46:0/3693632567 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c0d47c0 con 0x7f166c0b7e70 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166a7fc640 1 -- 172.21.15.46:0/3693632567 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1272110923 0 0) 0x7f166c0b5550 con 0x7f166c0b5140 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.951+0000 7f166bfff640 1 --2- 172.21.15.46:0/3693632567 >> v2:172.21.15.88:3300/0 conn(0x7f166c0d3df0 0x7f166c0b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.952+0000 7f166affd640 1 --2- 172.21.15.46:0/3693632567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b7e70 0x7f166c0d38b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f166c08f180 tx=0x7f166002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805f538ee35261e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.952+0000 7f166a7fc640 1 -- 172.21.15.46:0/3693632567 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660037070 con 0x7f166c0b7e70 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.952+0000 7f166a7fc640 1 -- 172.21.15.46:0/3693632567 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1660032440 con 0x7f166c0b7e70 2024-05-09T03:11:46.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.952+0000 7f166a7fc640 1 -- 172.21.15.46:0/3693632567 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660032810 con 0x7f166c0b7e70 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 -- 172.21.15.46:0/3693632567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b7e70 msgr2=0x7f166c0d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 --2- 172.21.15.46:0/3693632567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b7e70 0x7f166c0d38b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f166c08f180 tx=0x7f166002a230 comp rx=0 tx=0).stop 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 -- 172.21.15.46:0/3693632567 shutdown_connections 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 --2- 172.21.15.46:0/3693632567 >> v2:172.21.15.88:3300/0 conn(0x7f166c0d3df0 0x7f166c0b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 --2- 172.21.15.46:0/3693632567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b7e70 0x7f166c0d38b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 -- 172.21.15.46:0/3693632567 >> 172.21.15.46:0/3693632567 conn(0x7f166c08d3f0 msgr2=0x7f166c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 -- 172.21.15.46:0/3693632567 shutdown_connections 2024-05-09T03:11:46.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.953+0000 7f1671c53640 1 -- 172.21.15.46:0/3693632567 wait complete. 2024-05-09T03:11:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.955+0000 7f1671c53640 1 Processor -- start 2024-05-09T03:11:46.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.955+0000 7f1671c53640 1 -- start start 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f1671c53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 0x7f166c15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f1671c53640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f166c0d3df0 0x7f166c1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f1671c53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f166c153f40 0x7f166c162230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f1671c53640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f166c0d4200 con 0x7f166c0b5140 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f1671c53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f166c0d4d80 con 0x7f166c153f40 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f1671c53640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f166c0d5580 con 0x7f166c0d3df0 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f166b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 0x7f166c15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f166b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 0x7f166c15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39928/0 (socket says 172.21.15.46:39928) 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.956+0000 7f166b7fe640 1 -- 172.21.15.46:0/1299528447 learned_addr learned my addr 172.21.15.46:0/1299528447 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166bfff640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f166c153f40 0x7f166c162230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166affd640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f166c0d3df0 0x7f166c1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166b7fe640 1 -- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f166c0d3df0 msgr2=0x7f166c1537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166b7fe640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f166c0d3df0 0x7f166c1537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166b7fe640 1 -- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f166c153f40 msgr2=0x7f166c162230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166b7fe640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f166c153f40 0x7f166c162230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166b7fe640 1 -- 172.21.15.46:0/1299528447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166c1567a0 con 0x7f166c0b5140 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166bfff640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f166c153f40 0x7f166c162230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166affd640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f166c0d3df0 0x7f166c1537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f166b7fe640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 0x7f166c15b3a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c3e0 tx=0x7f165c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c0074e0 con 0x7f166c0b5140 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f165c0047e0 con 0x7f166c0b5140 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166c156a80 con 0x7f166c0b5140 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c004bd0 con 0x7f166c0b5140 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.957+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f166c157040 con 0x7f166c0b5140 2024-05-09T03:11:46.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.958+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f166c06a270 con 0x7f166c0b5140 2024-05-09T03:11:46.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.959+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f165c011070 con 0x7f166c0b5140 2024-05-09T03:11:46.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.960+0000 7f1653fff640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658076300 0x7f1658078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:46.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.960+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f165c0972e0 con 0x7f166c0b5140 2024-05-09T03:11:46.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.960+0000 7f166affd640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658076300 0x7f1658078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:46.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.960+0000 7f166affd640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658076300 0x7f1658078810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1660034000 tx=0x7f166002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:46.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:46.964+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f165c060b10 con 0x7f166c0b5140 2024-05-09T03:11:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.199+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f166c153ce0 con 0x7f166c0b5140 2024-05-09T03:11:47.200 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:11:47.199+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:11:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.201+0000 7f1653fff640 1 -- 172.21.15.46:0/1299528447 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) ==== 161+0+0 (secure 0 0 0) 0x7f165c060410 con 0x7f166c0b5140 2024-05-09T03:11:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:11:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658076300 msgr2=0x7f1658078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658076300 0x7f1658078810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1660034000 tx=0x7f166002e000 comp rx=0 tx=0).stop 2024-05-09T03:11:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 msgr2=0x7f166c15b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 0x7f166c15b3a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f165c00c3e0 tx=0x7f165c00c8a0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 shutdown_connections 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f166c153f40 0x7f166c162230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f166c0d3df0 0x7f166c1537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658076300 0x7f1658078810 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 --2- 172.21.15.46:0/1299528447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f166c0b5140 0x7f166c15b3a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 >> 172.21.15.46:0/1299528447 conn(0x7f166c08d3f0 msgr2=0x7f166c0d10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.204+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 shutdown_connections 2024-05-09T03:11:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.205+0000 7f1671c53640 1 -- 172.21.15.46:0/1299528447 wait complete. 2024-05-09T03:11:47.206 INFO:tasks.ceph.mds.a.smithi046.stderr: -2872> 2024-05-09T03:11:33.379+0000 7f19c41a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:11:47.210 INFO:tasks.ceph.mds.a.smithi046.stderr: -312> 2024-05-09T03:11:45.412+0000 7f19c69a7640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5562:17 nref=7 cr=0x55e08fdb4e00)) (dversion lock w=1 last_client=5562) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55e08faab400] 2024-05-09T03:11:47.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-05-09T03:11:47.263 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:11:47.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.336+0000 7fb428dd5640 1 Processor -- start 2024-05-09T03:11:47.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.336+0000 7fb428dd5640 1 -- start start 2024-05-09T03:11:47.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb428dd5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4240b4d80 0x7fb424160320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb428dd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb428dd5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb42406ddf0 0x7fb42406e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb428dd5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb4240d1230 con 0x7fb4240b4d80 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb428dd5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb4240b80b0 con 0x7fb42406ddf0 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb428dd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb424161700 con 0x7fb4240b7760 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb42259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4240b4d80 0x7fb424160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb42259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4240b4d80 0x7fb424160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39956/0 (socket says 172.21.15.46:39956) 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.337+0000 7fb42259c640 1 -- 172.21.15.46:0/314786051 learned_addr learned my addr 172.21.15.46:0/314786051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb422d9d640 1 --2- 172.21.15.46:0/314786051 >> v2:172.21.15.88:3300/0 conn(0x7fb42406ddf0 0x7fb42406e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb41bfff640 1 --2- 172.21.15.46:0/314786051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb42259c640 1 -- 172.21.15.46:0/314786051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 msgr2=0x7fb4240b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb42259c640 1 --2- 172.21.15.46:0/314786051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb42259c640 1 -- 172.21.15.46:0/314786051 >> v2:172.21.15.88:3300/0 conn(0x7fb42406ddf0 msgr2=0x7fb42406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb42259c640 1 --2- 172.21.15.46:0/314786051 >> v2:172.21.15.88:3300/0 conn(0x7fb42406ddf0 0x7fb42406e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.338+0000 7fb42259c640 1 -- 172.21.15.46:0/314786051 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4240d0900 con 0x7fb4240b4d80 2024-05-09T03:11:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.339+0000 7fb42259c640 1 --2- 172.21.15.46:0/314786051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb424160320 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb4240d2540 tx=0x7fb41402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb7b0f9ecabd49f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.339+0000 7fb421d9b640 1 -- 172.21.15.46:0/314786051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414037070 con 0x7fb4240b4d80 2024-05-09T03:11:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.339+0000 7fb421d9b640 1 -- 172.21.15.46:0/314786051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb414032440 con 0x7fb4240b4d80 2024-05-09T03:11:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.339+0000 7fb421d9b640 1 -- 172.21.15.46:0/314786051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb414032810 con 0x7fb4240b4d80 2024-05-09T03:11:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.339+0000 7fb428dd5640 1 -- 172.21.15.46:0/314786051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 msgr2=0x7fb424160320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.339+0000 7fb428dd5640 1 --2- 172.21.15.46:0/314786051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb424160320 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb4240d2540 tx=0x7fb41402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 -- 172.21.15.46:0/314786051 shutdown_connections 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 --2- 172.21.15.46:0/314786051 >> v2:172.21.15.88:3300/0 conn(0x7fb42406ddf0 0x7fb42406e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 --2- 172.21.15.46:0/314786051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 --2- 172.21.15.46:0/314786051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb424160320 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 -- 172.21.15.46:0/314786051 >> 172.21.15.46:0/314786051 conn(0x7fb4240d2760 msgr2=0x7fb4240d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 -- 172.21.15.46:0/314786051 shutdown_connections 2024-05-09T03:11:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.340+0000 7fb428dd5640 1 -- 172.21.15.46:0/314786051 wait complete. 2024-05-09T03:11:47.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.341+0000 7fb428dd5640 1 Processor -- start 2024-05-09T03:11:47.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.342+0000 7fb428dd5640 1 -- start start 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb428dd5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42406ddf0 0x7fb42415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb428dd5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb42415dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb428dd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb428dd5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4240d16c0 con 0x7fb4240b4d80 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb428dd5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb4240d0ec0 con 0x7fb42406ddf0 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb428dd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb4240b8390 con 0x7fb4240b7760 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb42415dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb42415dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39958/0 (socket says 172.21.15.46:39958) 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 -- 172.21.15.46:0/2050256822 learned_addr learned my addr 172.21.15.46:0/2050256822 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb422d9d640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb42259c640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42406ddf0 0x7fb42415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 -- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 msgr2=0x7fb4240ca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240ca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 -- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42406ddf0 msgr2=0x7fb42415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42406ddf0 0x7fb42415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 -- 172.21.15.46:0/2050256822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42415e1f0 con 0x7fb4240b4d80 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.343+0000 7fb41bfff640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb42415dcb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00d620 tx=0x7fb40c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.344+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c00b620 con 0x7fb4240b4d80 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.344+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb40c019070 con 0x7fb4240b4d80 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.344+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42415cef0 con 0x7fb4240b4d80 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.344+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c00ba00 con 0x7fb4240b4d80 2024-05-09T03:11:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.344+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42415d4b0 con 0x7fb4240b4d80 2024-05-09T03:11:47.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.345+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb42406a270 con 0x7fb4240b4d80 2024-05-09T03:11:47.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.346+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb40c00bba0 con 0x7fb4240b4d80 2024-05-09T03:11:47.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.346+0000 7fb41affd640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4040762d0 0x7fb4040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.346+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7fb40c097900 con 0x7fb4240b4d80 2024-05-09T03:11:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.346+0000 7fb42259c640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4040762d0 0x7fb4040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.347+0000 7fb42259c640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4040762d0 0x7fb4040787e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb414034000 tx=0x7fb41402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:47.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.351+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb40c060b20 con 0x7fb4240b4d80 2024-05-09T03:11:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.541+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7fb4240d4100 con 0x7fb4240b4d80 2024-05-09T03:11:47.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.545+0000 7fb41affd640 1 -- 172.21.15.46:0/2050256822 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7fb40c064e20 con 0x7fb4240b4d80 2024-05-09T03:11:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4040762d0 msgr2=0x7fb4040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4040762d0 0x7fb4040787e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb414034000 tx=0x7fb41402e000 comp rx=0 tx=0).stop 2024-05-09T03:11:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 msgr2=0x7fb42415dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb42415dcb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00d620 tx=0x7fb40c00dae0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 shutdown_connections 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4240b7760 0x7fb4240ca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4040762d0 0x7fb4040787e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4240b4d80 0x7fb42415dcb0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 --2- 172.21.15.46:0/2050256822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb42406ddf0 0x7fb42415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 >> 172.21.15.46:0/2050256822 conn(0x7fb4240d2760 msgr2=0x7fb424163180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 shutdown_connections 2024-05-09T03:11:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.547+0000 7fb428dd5640 1 -- 172.21.15.46:0/2050256822 wait complete. 2024-05-09T03:11:47.559 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-05-09T03:11:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.676+0000 7f2b53853640 1 Processor -- start 2024-05-09T03:11:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.676+0000 7f2b53853640 1 -- start start 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b53853640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b4c0b7e70 0x7f2b4c0b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b53853640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b4c1605b0 0x7f2b4c0b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b53853640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b4c0d3400 con 0x7f2b4c0d2e90 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b53853640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4c0d4a60 con 0x7f2b4c1605b0 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b53853640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4c0d4d40 con 0x7f2b4c0b7e70 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b51df0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b4c1605b0 0x7f2b4c0b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b515ef640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2b4c0d2e90 0x7f2b4c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59928/0 (socket says 172.21.15.46:59928) 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.677+0000 7f2b515ef640 1 -- 172.21.15.46:0/3112417141 learned_addr learned my addr 172.21.15.46:0/3112417141 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b50dee640 1 --2- 172.21.15.46:0/3112417141 >> v2:172.21.15.88:3300/0 conn(0x7f2b4c0b7e70 0x7f2b4c0b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b51df0640 1 -- 172.21.15.46:0/3112417141 >> v2:172.21.15.88:3300/0 conn(0x7f2b4c0b7e70 msgr2=0x7f2b4c0b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b51df0640 1 --2- 172.21.15.46:0/3112417141 >> v2:172.21.15.88:3300/0 conn(0x7f2b4c0b7e70 0x7f2b4c0b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b51df0640 1 -- 172.21.15.46:0/3112417141 >> v1:172.21.15.46:6789/0 conn(0x7f2b4c0d2e90 legacy=0x7f2b4c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b3ffff640 1 -- 172.21.15.46:0/3112417141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355361871 0 0) 0x7f2b4c0d3400 con 0x7f2b4c0d2e90 2024-05-09T03:11:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b51df0640 1 -- 172.21.15.46:0/3112417141 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c0d5020 con 0x7f2b4c1605b0 2024-05-09T03:11:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.678+0000 7f2b50dee640 1 --2- 172.21.15.46:0/3112417141 >> v2:172.21.15.88:3300/0 conn(0x7f2b4c0b7e70 0x7f2b4c0b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.679+0000 7f2b51df0640 1 --2- 172.21.15.46:0/3112417141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c0b97e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c08e880 tx=0x7f2b4402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bcec289d7523ea9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.679+0000 7f2b3ffff640 1 -- 172.21.15.46:0/3112417141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44007740 con 0x7f2b4c1605b0 2024-05-09T03:11:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.679+0000 7f2b3ffff640 1 -- 172.21.15.46:0/3112417141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b440078e0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.679+0000 7f2b3ffff640 1 -- 172.21.15.46:0/3112417141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b44007cb0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.679+0000 7f2b53853640 1 -- 172.21.15.46:0/3112417141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 msgr2=0x7f2b4c0b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.679+0000 7f2b53853640 1 --2- 172.21.15.46:0/3112417141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c0b97e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c08e880 tx=0x7f2b4402a5f0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.680+0000 7f2b53853640 1 -- 172.21.15.46:0/3112417141 shutdown_connections 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.680+0000 7f2b53853640 1 --2- 172.21.15.46:0/3112417141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c0b97e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.680+0000 7f2b53853640 1 --2- 172.21.15.46:0/3112417141 >> v2:172.21.15.88:3300/0 conn(0x7f2b4c0b7e70 0x7f2b4c0b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.680+0000 7f2b53853640 1 -- 172.21.15.46:0/3112417141 >> 172.21.15.46:0/3112417141 conn(0x7f2b4c08d390 msgr2=0x7f2b4c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.680+0000 7f2b53853640 1 -- 172.21.15.46:0/3112417141 shutdown_connections 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.680+0000 7f2b53853640 1 -- 172.21.15.46:0/3112417141 wait complete. 2024-05-09T03:11:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.681+0000 7f2b53853640 1 Processor -- start 2024-05-09T03:11:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.681+0000 7f2b53853640 1 -- start start 2024-05-09T03:11:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b53853640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c0d2e90 0x7f2b4c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b53853640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b53853640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b4c15c150 0x7f2b4c0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b53853640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0b4d80 con 0x7f2b4c1605b0 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b53853640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c0b4c00 con 0x7f2b4c0d2e90 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b53853640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c0b4fc0 con 0x7f2b4c15c150 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c15bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39976/0 (socket says 172.21.15.46:39976) 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 -- 172.21.15.46:0/3405493713 learned_addr learned my addr 172.21.15.46:0/3405493713 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b515ef640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c0d2e90 0x7f2b4c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b51df0640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b4c15c150 0x7f2b4c0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 -- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b4c15c150 msgr2=0x7f2b4c0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b4c15c150 0x7f2b4c0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 -- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c0d2e90 msgr2=0x7f2b4c15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c0d2e90 0x7f2b4c15b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 -- 172.21.15.46:0/3405493713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c15c560 con 0x7f2b4c1605b0 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b51df0640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b4c15c150 0x7f2b4c0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b515ef640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c0d2e90 0x7f2b4c15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.683+0000 7f2b50dee640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c15bc10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800c820 tx=0x7f2b4800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.684+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b48015070 con 0x7f2b4c1605b0 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.684+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b480079d0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.684+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c0b40f0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.684+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b48007d80 con 0x7f2b4c1605b0 2024-05-09T03:11:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.684+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c0b46b0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.685+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c0b3e60 con 0x7f2b4c1605b0 2024-05-09T03:11:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.685+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2b48018070 con 0x7f2b4c1605b0 2024-05-09T03:11:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.686+0000 7f2b3dffb640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b300762d0 0x7f2b300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.686+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f2b48097540 con 0x7f2b4c1605b0 2024-05-09T03:11:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.686+0000 7f2b515ef640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b300762d0 0x7f2b300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:47.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.687+0000 7f2b515ef640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b300762d0 0x7f2b300787e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b40000f60 tx=0x7f2b40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.691+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2b48060db0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.882+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) -- 0x7f2b4c06a270 con 0x7f2b4c1605b0 2024-05-09T03:11:47.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.883+0000 7f2b3dffb640 1 -- 172.21.15.46:0/3405493713 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v15)=0 v15) ==== 135+0+0 (secure 0 0 0) 0x7f2b480606b0 con 0x7f2b4c1605b0 2024-05-09T03:11:47.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b300762d0 msgr2=0x7f2b300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b300762d0 0x7f2b300787e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b40000f60 tx=0x7f2b40009040 comp rx=0 tx=0).stop 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 msgr2=0x7f2b4c15bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c15bc10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800c820 tx=0x7f2b4800cce0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 shutdown_connections 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b4c15c150 0x7f2b4c0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b300762d0 0x7f2b300787e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b4c1605b0 0x7f2b4c15bc10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 --2- 172.21.15.46:0/3405493713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c0d2e90 0x7f2b4c15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 >> 172.21.15.46:0/3405493713 conn(0x7f2b4c08d390 msgr2=0x7f2b4c075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 shutdown_connections 2024-05-09T03:11:47.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:47.885+0000 7f2b53853640 1 -- 172.21.15.46:0/3405493713 wait complete. 2024-05-09T03:11:47.897 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:11:48.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.013+0000 7f3db8d94640 1 Processor -- start 2024-05-09T03:11:48.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.014+0000 7f3db8d94640 1 -- start start 2024-05-09T03:11:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db8d94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3db4163eb0 0x7f3db41642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db8d94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3db40d2a70 0x7f3db40d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db8d94640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db40d33c0 con 0x7f3db40bb770 2024-05-09T03:11:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db8d94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3db40d4bb0 con 0x7f3db40d2a70 2024-05-09T03:11:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db8d94640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3db4164900 con 0x7f3db4163eb0 2024-05-09T03:11:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3db40bb770 0x7f3db40d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59940/0 (socket says 172.21.15.46:59940) 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db259c640 1 -- 172.21.15.46:0/845725229 learned_addr learned my addr 172.21.15.46:0/845725229 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.015+0000 7f3db2d9d640 1 --2- 172.21.15.46:0/845725229 >> v2:172.21.15.46:3300/0 conn(0x7f3db40d2a70 0x7f3db40d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db2d9d640 1 -- 172.21.15.46:0/845725229 >> v2:172.21.15.88:3300/0 conn(0x7f3db4163eb0 msgr2=0x7f3db41642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db1d9b640 1 --2- 172.21.15.46:0/845725229 >> v2:172.21.15.88:3300/0 conn(0x7f3db4163eb0 0x7f3db41642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db2d9d640 1 --2- 172.21.15.46:0/845725229 >> v2:172.21.15.88:3300/0 conn(0x7f3db4163eb0 0x7f3db41642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db2d9d640 1 -- 172.21.15.46:0/845725229 >> v1:172.21.15.46:6789/0 conn(0x7f3db40bb770 legacy=0x7f3db40d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db159a640 1 -- 172.21.15.46:0/845725229 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2628309743 0 0) 0x7f3dac001110 con 0x7f3db40bb770 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db2d9d640 1 -- 172.21.15.46:0/845725229 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db4164be0 con 0x7f3db40d2a70 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db1d9b640 1 --2- 172.21.15.46:0/845725229 >> v2:172.21.15.88:3300/0 conn(0x7f3db4163eb0 0x7f3db41642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db2d9d640 1 --2- 172.21.15.46:0/845725229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db40d2e80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3db40d6d00 tx=0x7f3da402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b622a0dc5f1cdb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.016+0000 7f3db159a640 1 -- 172.21.15.46:0/845725229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4007740 con 0x7f3db40d2a70 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db159a640 1 -- 172.21.15.46:0/845725229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da40078e0 con 0x7f3db40d2a70 2024-05-09T03:11:48.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db159a640 1 -- 172.21.15.46:0/845725229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4007cb0 con 0x7f3db40d2a70 2024-05-09T03:11:48.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 -- 172.21.15.46:0/845725229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 msgr2=0x7f3db40d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:48.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 --2- 172.21.15.46:0/845725229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db40d2e80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3db40d6d00 tx=0x7f3da402a680 comp rx=0 tx=0).stop 2024-05-09T03:11:48.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 -- 172.21.15.46:0/845725229 shutdown_connections 2024-05-09T03:11:48.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 --2- 172.21.15.46:0/845725229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db40d2e80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:48.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 --2- 172.21.15.46:0/845725229 >> v2:172.21.15.88:3300/0 conn(0x7f3db4163eb0 0x7f3db41642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:48.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 -- 172.21.15.46:0/845725229 >> 172.21.15.46:0/845725229 conn(0x7f3db406e370 msgr2=0x7f3db406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:48.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 -- 172.21.15.46:0/845725229 shutdown_connections 2024-05-09T03:11:48.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.017+0000 7f3db8d94640 1 -- 172.21.15.46:0/845725229 wait complete. 2024-05-09T03:11:48.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.019+0000 7f3db8d94640 1 Processor -- start 2024-05-09T03:11:48.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.019+0000 7f3db8d94640 1 -- start start 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db40bb770 0x7f3db40c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db415e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db4163eb0 0x7f3db415ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40d0bc0 con 0x7f3db40d2a70 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40bbb80 con 0x7f3db40bb770 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3db40d0d40 con 0x7f3db4163eb0 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db415e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db415e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39996/0 (socket says 172.21.15.46:39996) 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 -- 172.21.15.46:0/1269077028 learned_addr learned my addr 172.21.15.46:0/1269077028 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:48.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db259c640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db40bb770 0x7f3db40c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db2d9d640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db4163eb0 0x7f3db415ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 -- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db4163eb0 msgr2=0x7f3db415ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db4163eb0 0x7f3db415ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 -- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db40bb770 msgr2=0x7f3db40c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db40bb770 0x7f3db40c87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 -- 172.21.15.46:0/1269077028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db41eacd0 con 0x7f3db40d2a70 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db259c640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db40bb770 0x7f3db40c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db2d9d640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db4163eb0 0x7f3db415ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db1d9b640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db415e6f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00c820 tx=0x7f3d9c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c015070 con 0x7f3db40d2a70 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d9c007a30 con 0x7f3db40d2a70 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db41ea710 con 0x7f3db40d2a70 2024-05-09T03:11:48.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.021+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c007de0 con 0x7f3db40d2a70 2024-05-09T03:11:48.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.022+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db41469b0 con 0x7f3db40d2a70 2024-05-09T03:11:48.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.022+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db415aff0 con 0x7f3db40d2a70 2024-05-09T03:11:48.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.023+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3d9c018070 con 0x7f3db40d2a70 2024-05-09T03:11:48.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.024+0000 7f3daaffd640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3d94076300 0x7f3d94078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:48.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.024+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f3d9c097680 con 0x7f3db40d2a70 2024-05-09T03:11:48.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.024+0000 7f3db259c640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3d94076300 0x7f3d94078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:48.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.024+0000 7f3db259c640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3d94076300 0x7f3d94078810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3dac000f60 tx=0x7f3dac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:48.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.028+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3d9c060e60 con 0x7f3db40d2a70 2024-05-09T03:11:48.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:48.258+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f3db406a270 con 0x7f3db40d2a70 2024-05-09T03:11:49.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.207+0000 7f3daaffd640 1 -- 172.21.15.46:0/1269077028 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v33) ==== 162+0+0 (secure 0 0 0) 0x7f3d9c060760 con 0x7f3db40d2a70 2024-05-09T03:11:49.208 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:11:49.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3d94076300 msgr2=0x7f3d94078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3d94076300 0x7f3d94078810 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3dac000f60 tx=0x7f3dac009040 comp rx=0 tx=0).stop 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 msgr2=0x7f3db415e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db415e6f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00c820 tx=0x7f3d9c00cce0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 shutdown_connections 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3db4163eb0 0x7f3db415ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3d94076300 0x7f3d94078810 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3db40d2a70 0x7f3db415e6f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 --2- 172.21.15.46:0/1269077028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3db40bb770 0x7f3db40c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.210+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 >> 172.21.15.46:0/1269077028 conn(0x7f3db406e370 msgr2=0x7f3db40b4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.211+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 shutdown_connections 2024-05-09T03:11:49.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.211+0000 7f3db8d94640 1 -- 172.21.15.46:0/1269077028 wait complete. 2024-05-09T03:11:49.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:49.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.349+0000 7f0a4b0b8640 1 Processor -- start 2024-05-09T03:11:49.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.349+0000 7f0a4b0b8640 1 -- start start 2024-05-09T03:11:49.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a4b0b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a440b9200 0x7f0a4408d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a4b0b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a4408d6e0 0x7f0a440d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a4b0b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a440b4ee0 con 0x7f0a440d20a0 2024-05-09T03:11:49.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a4b0b8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0a440b5460 con 0x7f0a440b9200 2024-05-09T03:11:49.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a4b0b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0a440cfac0 con 0x7f0a4408d6e0 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a48e54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a440b9200 0x7f0a4408d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a48e54640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a440b9200 0x7f0a4408d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40008/0 (socket says 172.21.15.46:40008) 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.350+0000 7f0a48e54640 1 -- 172.21.15.46:0/3858975788 learned_addr learned my addr 172.21.15.46:0/3858975788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a43fff640 1 --2- 172.21.15.46:0/3858975788 >> v2:172.21.15.88:3300/0 conn(0x7f0a4408d6e0 0x7f0a440d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a48e54640 1 -- 172.21.15.46:0/3858975788 >> v2:172.21.15.88:3300/0 conn(0x7f0a4408d6e0 msgr2=0x7f0a440d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a48e54640 1 --2- 172.21.15.46:0/3858975788 >> v2:172.21.15.88:3300/0 conn(0x7f0a4408d6e0 0x7f0a440d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a48e54640 1 -- 172.21.15.46:0/3858975788 >> v1:172.21.15.88:6789/0 conn(0x7f0a440d20a0 legacy=0x7f0a44162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a48e54640 1 -- 172.21.15.46:0/3858975788 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a440b62e0 con 0x7f0a440b9200 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a43fff640 1 --2- 172.21.15.46:0/3858975788 >> v2:172.21.15.88:3300/0 conn(0x7f0a4408d6e0 0x7f0a440d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.351+0000 7f0a48e54640 1 --2- 172.21.15.46:0/3858975788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a4408d1a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0a44074b60 tx=0x7f0a2c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1691f18534c779e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a437fe640 1 -- 172.21.15.46:0/3858975788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c037070 con 0x7f0a440b9200 2024-05-09T03:11:49.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a437fe640 1 -- 172.21.15.46:0/3858975788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a2c032440 con 0x7f0a440b9200 2024-05-09T03:11:49.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a437fe640 1 -- 172.21.15.46:0/3858975788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2c032810 con 0x7f0a440b9200 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/3858975788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 msgr2=0x7f0a4408d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/3858975788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a4408d1a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0a44074b60 tx=0x7f0a2c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/3858975788 shutdown_connections 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/3858975788 >> v2:172.21.15.88:3300/0 conn(0x7f0a4408d6e0 0x7f0a440d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/3858975788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a4408d1a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/3858975788 >> 172.21.15.46:0/3858975788 conn(0x7f0a440b56e0 msgr2=0x7f0a440b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/3858975788 shutdown_connections 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.352+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/3858975788 wait complete. 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.354+0000 7f0a4b0b8640 1 Processor -- start 2024-05-09T03:11:49.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.354+0000 7f0a4b0b8640 1 -- start start 2024-05-09T03:11:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a4b0b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a4408d6e0 0x7f0a44155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a4b0b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a44153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a4b0b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a440d20a0 0x7f0a44161370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a4b0b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a440cf750 con 0x7f0a440b9200 2024-05-09T03:11:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a4b0b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0a440cf470 con 0x7f0a4408d6e0 2024-05-09T03:11:49.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a4b0b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0a440b4a20 con 0x7f0a440d20a0 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a44153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a44153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40024/0 (socket says 172.21.15.46:40024) 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 -- 172.21.15.46:0/1183472336 learned_addr learned my addr 172.21.15.46:0/1183472336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a48e54640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a4408d6e0 0x7f0a44155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a49655640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a440d20a0 0x7f0a44161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 -- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a440d20a0 msgr2=0x7f0a44161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a440d20a0 0x7f0a44161370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 -- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a4408d6e0 msgr2=0x7f0a44155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a4408d6e0 0x7f0a44155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 -- 172.21.15.46:0/1183472336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a441559b0 con 0x7f0a440b9200 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a49655640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a440d20a0 0x7f0a44161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:49.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.356+0000 7f0a43fff640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a44153640 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400ce60 tx=0x7f0a3400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.357+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3400f890 con 0x7f0a440b9200 2024-05-09T03:11:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.357+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3400fa30 con 0x7f0a440b9200 2024-05-09T03:11:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.357+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3400fe00 con 0x7f0a440b9200 2024-05-09T03:11:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.357+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a44154130 con 0x7f0a440b9200 2024-05-09T03:11:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.357+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4415b0d0 con 0x7f0a440b9200 2024-05-09T03:11:49.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.358+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4406a270 con 0x7f0a440b9200 2024-05-09T03:11:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.358+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a34018400 con 0x7f0a440b9200 2024-05-09T03:11:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.359+0000 7f0a417fa640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a24076300 0x7f0a24078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.359+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f0a340982f0 con 0x7f0a440b9200 2024-05-09T03:11:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.359+0000 7f0a48e54640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a24076300 0x7f0a24078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.360+0000 7f0a48e54640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a24076300 0x7f0a24078810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c034000 tx=0x7f0a2c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.363+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0a34014030 con 0x7f0a440b9200 2024-05-09T03:11:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.593+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a441e8600 con 0x7f0a440b9200 2024-05-09T03:11:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.594+0000 7f0a417fa640 1 -- 172.21.15.46:0/1183472336 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5163 (secure 0 0 0) 0x7f0a3400ae20 con 0x7f0a440b9200 2024-05-09T03:11:49.595 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:49.595 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:49.211594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-05-09T03:11:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 34 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a24076300 msgr2=0x7f0a24078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a24076300 0x7f0a24078810 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c034000 tx=0x7f0a2c02e000 comp rx=0 tx=0).stop 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 msgr2=0x7f0a44153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a44153640 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400ce60 tx=0x7f0a3400a650 comp rx=0 tx=0).stop 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 shutdown_connections 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a440d20a0 0x7f0a44161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a24076300 0x7f0a24078810 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a440b9200 0x7f0a44153640 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 --2- 172.21.15.46:0/1183472336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a4408d6e0 0x7f0a44155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 >> 172.21.15.46:0/1183472336 conn(0x7f0a440b56e0 msgr2=0x7f0a440cdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 shutdown_connections 2024-05-09T03:11:49.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.596+0000 7f0a4b0b8640 1 -- 172.21.15.46:0/1183472336 wait complete. 2024-05-09T03:11:49.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-05-09T03:11:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.724+0000 7fd2dd500640 1 Processor -- start 2024-05-09T03:11:49.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.724+0000 7fd2dd500640 1 -- start start 2024-05-09T03:11:49.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2dd500640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2d80b52a0 0x7fd2d806e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2dd500640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd2d80b7e50 0x7fd2d80ba1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2dd500640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d80d1730 con 0x7fd2d80d1000 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2dd500640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d8160b40 con 0x7fd2d80b52a0 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2dd500640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d80b9d10 con 0x7fd2d80b7e50 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2d80b52a0 0x7fd2d806e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d67fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2d80d1000 0x7fd2d80b7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59946/0 (socket says 172.21.15.46:59946) 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d67fc640 1 -- 172.21.15.46:0/3048229678 learned_addr learned my addr 172.21.15.46:0/3048229678 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/3048229678 >> v2:172.21.15.88:3300/0 conn(0x7fd2d80b7e50 0x7fd2d80ba1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d6ffd640 1 -- 172.21.15.46:0/3048229678 >> v2:172.21.15.88:3300/0 conn(0x7fd2d80b7e50 msgr2=0x7fd2d80ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/3048229678 >> v2:172.21.15.88:3300/0 conn(0x7fd2d80b7e50 0x7fd2d80ba1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d6ffd640 1 -- 172.21.15.46:0/3048229678 >> v1:172.21.15.46:6789/0 conn(0x7fd2d80d1000 legacy=0x7fd2d80b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d5ffb640 1 -- 172.21.15.46:0/3048229678 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2289746769 0 0) 0x7fd2d80d1730 con 0x7fd2d80d1000 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d6ffd640 1 -- 172.21.15.46:0/3048229678 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d80b96c0 con 0x7fd2d80b52a0 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.726+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/3048229678 >> v2:172.21.15.88:3300/0 conn(0x7fd2d80b7e50 0x7fd2d80ba1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.727+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/3048229678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80b52a0 0x7fd2d806e7f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd2d808f1b0 tx=0x7fd2d002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1776b86088dfdac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.727+0000 7fd2d5ffb640 1 -- 172.21.15.46:0/3048229678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0037070 con 0x7fd2d80b52a0 2024-05-09T03:11:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.727+0000 7fd2d5ffb640 1 -- 172.21.15.46:0/3048229678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d0032440 con 0x7fd2d80b52a0 2024-05-09T03:11:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.727+0000 7fd2d5ffb640 1 -- 172.21.15.46:0/3048229678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0032810 con 0x7fd2d80b52a0 2024-05-09T03:11:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 -- 172.21.15.46:0/3048229678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80b52a0 msgr2=0x7fd2d806e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3048229678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80b52a0 0x7fd2d806e7f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fd2d808f1b0 tx=0x7fd2d002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 -- 172.21.15.46:0/3048229678 shutdown_connections 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3048229678 >> v2:172.21.15.88:3300/0 conn(0x7fd2d80b7e50 0x7fd2d80ba1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3048229678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80b52a0 0x7fd2d806e7f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 -- 172.21.15.46:0/3048229678 >> 172.21.15.46:0/3048229678 conn(0x7fd2d80d2920 msgr2=0x7fd2d80d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 -- 172.21.15.46:0/3048229678 shutdown_connections 2024-05-09T03:11:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.728+0000 7fd2dd500640 1 -- 172.21.15.46:0/3048229678 wait complete. 2024-05-09T03:11:49.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.730+0000 7fd2dd500640 1 Processor -- start 2024-05-09T03:11:49.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.730+0000 7fd2dd500640 1 -- start start 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2dd500640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2d80b52a0 0x7fd2d815d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2dd500640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 0x7fd2d80cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2dd500640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2d80cf510 0x7fd2d80d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2dd500640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d80d1410 con 0x7fd2d80d1000 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2dd500640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d80b93b0 con 0x7fd2d80cf510 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2dd500640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80ba6e0 con 0x7fd2d80b52a0 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 0x7fd2d80cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 0x7fd2d80cefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40046/0 (socket says 172.21.15.46:40046) 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2d67fc640 1 -- 172.21.15.46:0/3608664028 learned_addr learned my addr 172.21.15.46:0/3608664028 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2d80b52a0 0x7fd2d815d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2d80cf510 0x7fd2d80d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.731+0000 7fd2d67fc640 1 -- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2d80b52a0 msgr2=0x7fd2d815d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2d80b52a0 0x7fd2d815d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2d67fc640 1 -- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2d80cf510 msgr2=0x7fd2d80d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2d80cf510 0x7fd2d80d5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2d67fc640 1 -- 172.21.15.46:0/3608664028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d81f64c0 con 0x7fd2d80d1000 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2d80cf510 0x7fd2d80d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 0x7fd2d80cefd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c410 tx=0x7fd2c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3608664028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c00074e0 con 0x7fd2d80d1000 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3608664028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c00047e0 con 0x7fd2d80d1000 2024-05-09T03:11:49.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3608664028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0004bd0 con 0x7fd2d80d1000 2024-05-09T03:11:49.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d81f67a0 con 0x7fd2d80d1000 2024-05-09T03:11:49.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.732+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d81f6d60 con 0x7fd2d80d1000 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.734+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3608664028 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd2c0011070 con 0x7fd2d80d1000 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.734+0000 7fd2cf7fe640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2b80762d0 0x7fd2b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.735+0000 7fd2cf7fe640 1 -- 172.21.15.46:0/3608664028 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7fd2c0097300 con 0x7fd2d80d1000 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.735+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2b80762d0 0x7fd2b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.735+0000 7fd2e43ba740 1 Processor -- start 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.735+0000 7fd2e43ba740 1 -- start start 2024-05-09T03:11:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2b80762d0 0x7fd2b80787e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0034000 tx=0x7fd2d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d05510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 0x557c14d20240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d20780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c14c3e1a0 con 0x557c1490d4d0 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557c14bf7170 con 0x557c14936390 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557c14c3a160 con 0x557c149156b0 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.736+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 0x557c14d20240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 0x557c14d20240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40052/0 (socket says 172.21.15.46:40052) 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 -- 172.21.15.46:0/2285954316 learned_addr learned my addr 172.21.15.46:0/2285954316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d20780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d05510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 -- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 msgr2=0x557c14d20780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d20780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 -- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 msgr2=0x557c14d05510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d05510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 -- 172.21.15.46:0/2285954316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c14c09390 con 0x557c1490d4d0 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 0x557c14d20240 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd2c005c4e0 tx=0x7fd2c005cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc657594e0820a82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2ccff9640 1 -- 172.21.15.46:0/2285954316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0060b60 con 0x557c1490d4d0 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2ccff9640 1 -- 172.21.15.46:0/2285954316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c0060460 con 0x557c1490d4d0 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.737+0000 7fd2ccff9640 1 -- 172.21.15.46:0/2285954316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0064810 con 0x557c1490d4d0 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2285954316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 msgr2=0x557c14d20240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 0x557c14d20240 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd2c005c4e0 tx=0x7fd2c005cab0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2285954316 shutdown_connections 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d20780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1490d4d0 0x557c14d20240 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2285954316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d05510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2285954316 >> 172.21.15.46:0/2285954316 conn(0x557c1492a4e0 msgr2=0x557c14cdab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2285954316 shutdown_connections 2024-05-09T03:11:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.738+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2285954316 wait complete. 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.739+0000 7fd2e43ba740 1 -- start start 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.739+0000 7fd2e43ba740 1 Processor -- start 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.740+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d06420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.740+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 0x557c14d040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.740+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d04600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.740+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c14bc48c0 con 0x557c149156b0 2024-05-09T03:11:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.740+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557c14d06bb0 con 0x557c14936390 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.740+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557c14d06960 con 0x557c1490d4d0 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2e43ba740 10 client.0 _ll_register_callbacks cb 0x557c14be0860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 0x557c14d040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 0x557c14d040c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40058/0 (socket says 172.21.15.46:40058) 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 -- 172.21.15.46:0/4049794008 learned_addr learned my addr 172.21.15.46:0/4049794008 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d06420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d04600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 -- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 msgr2=0x557c14d06420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d06420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 -- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 msgr2=0x557c14d04600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d04600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 -- 172.21.15.46:0/4049794008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c14a187d0 con 0x557c149156b0 2024-05-09T03:11:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 0x557c14d040c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd2c005c4e0 tx=0x7fd2c0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58f560494769659 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/4049794008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0069230 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/4049794008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7fd2c00970e0 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2c0060460 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/4049794008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd2c0080050 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2e43ba740 10 client.5682 fetch_fsmap learned FSMap version 34 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.741+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x557c149a1970 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/4049794008 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c0060460 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/4049794008 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 34) ==== 3486+0+0 (secure 0 0 0) 0x7fd2c0079bc0 con 0x557c149156b0 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2e43ba740 10 client.5682 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2e43ba740 10 client.5682 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2e43ba740 20 client.5682 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c148f4f60 0x557c14d17c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2e43ba740 4 client.5682 mds_command: new command op to 5637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 --> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c14c4bdb0 con 0x557c148f4f60 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c148f4f60 0x557c14d17c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c148f4f60 0x557c14d17c70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=595c76c5a583097f server_cookie=27e81de0d06330a0 in_seq=0 out_seq=0 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.742+0000 7fd2b5ffb640 10 client.5682 ms_handle_connect on v2:172.21.15.46:6833/881221292 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/4049794008 <== mds.0 v2:172.21.15.46:6833/881221292 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557c14c4bdb0 con 0x557c148f4f60 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2b5ffb640 10 client.5682 handle_command_reply: tid=0 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 client.5682 shutdown 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 msgr2=0x557c14d040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 0x557c14d040c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd2c005c4e0 tx=0x7fd2c0000ed0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 shutdown_connections 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c148f4f60 0x557c14d17c70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c14936390 0x557c14d04600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c149156b0 0x557c14d040c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4049794008 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d06420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2b5ffb640 0 client.5682 ms_handle_reset on v2:172.21.15.46:6833/881221292 2024-05-09T03:11:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 >> 172.21.15.46:0/4049794008 conn(0x557c1492a4e0 msgr2=0x557c14cdab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 shutdown_connections 2024-05-09T03:11:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4049794008 wait complete. 2024-05-09T03:11:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.743+0000 7fd2e43ba740 20 client.5682 trim_cache size 0 max 16384 2024-05-09T03:11:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.759+0000 7fd2e43ba740 1 Processor -- start 2024-05-09T03:11:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.759+0000 7fd2e43ba740 1 -- start start 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d05d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c149156b0 0x557c14d06280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d8202c00 con 0x557c14936390 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d8201ca0 con 0x557c149156b0 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80cdae0 con 0x557c1490d4d0 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40070/0 (socket says 172.21.15.46:40070) 2024-05-09T03:11:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2d77fe640 1 -- 172.21.15.46:0/4185698431 learned_addr learned my addr 172.21.15.46:0/4185698431 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d05d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.760+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c149156b0 0x557c14d06280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2d77fe640 1 -- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 msgr2=0x557c14d05d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d05d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2d77fe640 1 -- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c149156b0 msgr2=0x557c14d06280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c149156b0 0x557c14d06280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2d77fe640 1 -- 172.21.15.46:0/4185698431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c148e6f60 con 0x557c14936390 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07920 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8010490 tx=0x7fd2c8010950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf8616a83a97eee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2b7fff640 1 -- 172.21.15.46:0/4185698431 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c801e070 con 0x557c14936390 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2b7fff640 1 -- 172.21.15.46:0/4185698431 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c800e6b0 con 0x557c14936390 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.761+0000 7fd2b7fff640 1 -- 172.21.15.46:0/4185698431 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c800ea60 con 0x557c14936390 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4185698431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 msgr2=0x557c14d07920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07920 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8010490 tx=0x7fd2c8010950 comp rx=0 tx=0).stop 2024-05-09T03:11:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4185698431 shutdown_connections 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07920 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c149156b0 0x557c14d06280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/4185698431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1490d4d0 0x557c14d05d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4185698431 >> 172.21.15.46:0/4185698431 conn(0x557c1492a4e0 msgr2=0x557c14cda8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4185698431 shutdown_connections 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 -- 172.21.15.46:0/4185698431 wait complete. 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.762+0000 7fd2e43ba740 1 -- start start 2024-05-09T03:11:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.763+0000 7fd2e43ba740 1 Processor -- start 2024-05-09T03:11:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1490d4d0 0x557c14d9ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d07920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d80ca350 con 0x557c14936390 2024-05-09T03:11:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557c14cd48e0 con 0x557c1490d4d0 2024-05-09T03:11:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557c14d01870 con 0x557c149156b0 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2e43ba740 10 client.0 _ll_register_callbacks cb 0x7fd2d0035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40076/0 (socket says 172.21.15.46:40076) 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 -- 172.21.15.46:0/2695825381 learned_addr learned my addr 172.21.15.46:0/2695825381 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1490d4d0 0x557c14d9ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d07920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 -- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 msgr2=0x557c14d07920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d07920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 -- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1490d4d0 msgr2=0x557c14d9ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1490d4d0 0x557c14d9ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 -- 172.21.15.46:0/2695825381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c14d01b50 con 0x557c14936390 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1490d4d0 0x557c14d9ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d67fc640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d07920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2d77fe640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07e60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8018000 tx=0x7fd2c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad76f0c501992711 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.764+0000 7fd2b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/2695825381 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8029020 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/2695825381 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7fd2c8024670 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2c800e6b0 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/2695825381 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fd2c8033050 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 10 client.5688 fetch_fsmap learned FSMap version 34 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x557c14cd9fd0 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/2695825381 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c800e6b0 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/2695825381 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 34) ==== 3486+0+0 (secure 0 0 0) 0x7fd2c8036020 con 0x557c14936390 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 10 client.5688 fetch_fsmap finished waiting for FSMap version 34 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 10 client.5688 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 20 client.5688 populate_metadata read hostname 'smithi046' 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c14cff000 0x557c14cff410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 4 client.5688 mds_command: new command op to 5637 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 --> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557c14c45c20 con 0x557c14cff000 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.765+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c14cff000 0x557c14cff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2d6ffd640 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c14cff000 0x557c14cff410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbaca375b78fc1df server_cookie=122eb04ca5b66fea in_seq=0 out_seq=0 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2b5ffb640 10 client.5688 ms_handle_connect on v2:172.21.15.46:6833/881221292 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2b5ffb640 1 -- 172.21.15.46:0/2695825381 <== mds.0 v2:172.21.15.46:6833/881221292 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x557c14c45c20 con 0x557c14cff000 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2b5ffb640 10 client.5688 handle_command_reply: tid=0 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 client.5688 shutdown 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 msgr2=0x557c14d07e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07e60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8018000 tx=0x7fd2c8004ab0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 shutdown_connections 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:6833/881221292,v1:172.21.15.46:6835/881221292] conn(0x557c14cff000 0x557c14cff410 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c14936390 0x557c14d07e60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c149156b0 0x557c14d07920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2b5ffb640 0 client.5688 ms_handle_reset on v2:172.21.15.46:6833/881221292 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 --2- 172.21.15.46:0/2695825381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1490d4d0 0x557c14d9ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 >> 172.21.15.46:0/2695825381 conn(0x557c1492a4e0 msgr2=0x557c14cdb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 shutdown_connections 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 1 -- 172.21.15.46:0/2695825381 wait complete. 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.766+0000 7fd2e43ba740 20 client.5688 trim_cache size 0 max 16384 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0 2024-05-09T03:11:49.773 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:11:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2b80762d0 msgr2=0x7fd2b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2b80762d0 0x7fd2b80787e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0034000 tx=0x7fd2d002e000 comp rx=0 tx=0).stop 2024-05-09T03:11:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 msgr2=0x7fd2d80cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 0x7fd2d80cefd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd2c000c410 tx=0x7fd2c000c8d0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 shutdown_connections 2024-05-09T03:11:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2d80cf510 0x7fd2d80d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2b80762d0 0x7fd2b80787e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d80d1000 0x7fd2d80cefd0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 --2- 172.21.15.46:0/3608664028 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2d80b52a0 0x7fd2d815d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 >> 172.21.15.46:0/3608664028 conn(0x7fd2d80d2920 msgr2=0x7fd2d81e8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 shutdown_connections 2024-05-09T03:11:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.767+0000 7fd2dd500640 1 -- 172.21.15.46:0/3608664028 wait complete. 2024-05-09T03:11:49.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.892+0000 7f4ce2366640 1 Processor -- start 2024-05-09T03:11:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.892+0000 7f4ce2366640 1 -- start start 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.893+0000 7f4ce2366640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cdc0b9200 0x7f4cdc08d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.893+0000 7f4ce2366640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc08d6e0 0x7f4cdc0d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.893+0000 7f4ce2366640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cdc0b4ee0 con 0x7f4cdc0d20a0 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.893+0000 7f4ce2366640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4cdc0b5460 con 0x7f4cdc0b9200 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.893+0000 7f4ce2366640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cdc0cfac0 con 0x7f4cdc08d6e0 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cdc0b9200 0x7f4cdc08d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4ce0903640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4cdc0d20a0 0x7f4cdc162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:33416/0 (socket says 172.21.15.46:33416) 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4ce0903640 1 -- 172.21.15.46:0/1695941268 learned_addr learned my addr 172.21.15.46:0/1695941268 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdb7fe640 1 --2- 172.21.15.46:0/1695941268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc08d6e0 0x7f4cdc0d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/1695941268 >> v2:172.21.15.88:3300/0 conn(0x7f4cdc0b9200 msgr2=0x7f4cdc08d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdb7fe640 1 --2- 172.21.15.46:0/1695941268 >> v2:172.21.15.88:3300/0 conn(0x7f4cdc0b9200 0x7f4cdc08d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/1695941268 >> v1:172.21.15.88:6789/0 conn(0x7f4cdc0d20a0 legacy=0x7f4cdc162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/1695941268 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc0b62e0 con 0x7f4cdc08d6e0 2024-05-09T03:11:49.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.894+0000 7f4cdbfff640 1 --2- 172.21.15.46:0/1695941268 >> v2:172.21.15.88:3300/0 conn(0x7f4cdc0b9200 0x7f4cdc08d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.895+0000 7f4cdb7fe640 1 --2- 172.21.15.46:0/1695941268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc08d6e0 0x7f4cdc0d1b60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00097a0 tx=0x7f4cd002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf2cc3b86b43b51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.895+0000 7f4cdaffd640 1 -- 172.21.15.46:0/1695941268 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0037070 con 0x7f4cdc08d6e0 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.895+0000 7f4cdaffd640 1 -- 172.21.15.46:0/1695941268 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd0032440 con 0x7f4cdc08d6e0 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.895+0000 7f4cdaffd640 1 -- 172.21.15.46:0/1695941268 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0032810 con 0x7f4cdc08d6e0 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 -- 172.21.15.46:0/1695941268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc08d6e0 msgr2=0x7f4cdc0d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 --2- 172.21.15.46:0/1695941268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc08d6e0 0x7f4cdc0d1b60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00097a0 tx=0x7f4cd002a180 comp rx=0 tx=0).stop 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 -- 172.21.15.46:0/1695941268 shutdown_connections 2024-05-09T03:11:49.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 --2- 172.21.15.46:0/1695941268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc08d6e0 0x7f4cdc0d1b60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 --2- 172.21.15.46:0/1695941268 >> v2:172.21.15.88:3300/0 conn(0x7f4cdc0b9200 0x7f4cdc08d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 -- 172.21.15.46:0/1695941268 >> 172.21.15.46:0/1695941268 conn(0x7f4cdc0b56e0 msgr2=0x7f4cdc0b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 -- 172.21.15.46:0/1695941268 shutdown_connections 2024-05-09T03:11:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.896+0000 7f4ce2366640 1 -- 172.21.15.46:0/1695941268 wait complete. 2024-05-09T03:11:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.897+0000 7f4ce2366640 1 Processor -- start 2024-05-09T03:11:49.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.898+0000 7f4ce2366640 1 -- start start 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce2366640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc08d6e0 0x7f4cdc14ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce2366640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 0x7f4cdc0b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce2366640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc0d20a0 0x7f4cdc160e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce2366640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc0cf750 con 0x7f4cdc0b9200 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce2366640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4cdc0cf470 con 0x7f4cdc08d6e0 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce2366640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cdc0b4a20 con 0x7f4cdc0d20a0 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 0x7f4cdc0b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 0x7f4cdc0b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40080/0 (socket says 172.21.15.46:40080) 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/4218038492 learned_addr learned my addr 172.21.15.46:0/4218038492 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdbfff640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc08d6e0 0x7f4cdc14ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4ce0903640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc0d20a0 0x7f4cdc160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc0d20a0 msgr2=0x7f4cdc160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc0d20a0 0x7f4cdc160e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc08d6e0 msgr2=0x7f4cdc14ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.899+0000 7f4cdb7fe640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc08d6e0 0x7f4cdc14ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4cdb7fe640 1 -- 172.21.15.46:0/4218038492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cdc0b4270 con 0x7f4cdc0b9200 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4cdb7fe640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 0x7f4cdc0b3d30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0035040 tx=0x7f4cd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0037040 con 0x7f4cdc0b9200 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd0030500 con 0x7f4cdc0b9200 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd00308d0 con 0x7f4cdc0b9200 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cdc15fdb0 con 0x7f4cdc0b9200 2024-05-09T03:11:49.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.900+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cdc160370 con 0x7f4cdc0b9200 2024-05-09T03:11:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.901+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cdc06a270 con 0x7f4cdc0b9200 2024-05-09T03:11:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.902+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4cd0044020 con 0x7f4cdc0b9200 2024-05-09T03:11:49.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.902+0000 7f4cd8ff9640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4cc80762d0 0x7f4cc80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.902+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f4cd00b7700 con 0x7f4cdc0b9200 2024-05-09T03:11:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.903+0000 7f4cdbfff640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4cc80762d0 0x7f4cc80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.903+0000 7f4cdbfff640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4cc80762d0 0x7f4cc80787e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009560 tx=0x7f4ccc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:49.907+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4cd0080930 con 0x7f4cdc0b9200 2024-05-09T03:11:50.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.138+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cdc15b930 con 0x7f4cdc0b9200 2024-05-09T03:11:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.138+0000 7f4cd8ff9640 1 -- 172.21.15.46:0/4218038492 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5163 (secure 0 0 0) 0x7f4cd0084c30 con 0x7f4cdc0b9200 2024-05-09T03:11:50.139 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:50.139 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:49.211594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-05-09T03:11:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 34 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4cc80762d0 msgr2=0x7f4cc80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4cc80762d0 0x7f4cc80787e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc009560 tx=0x7f4ccc009190 comp rx=0 tx=0).stop 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 msgr2=0x7f4cdc0b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 0x7f4cdc0b3d30 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0035040 tx=0x7f4cd00046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 shutdown_connections 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cdc0d20a0 0x7f4cdc160e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4cc80762d0 0x7f4cc80787e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cdc0b9200 0x7f4cdc0b3d30 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 --2- 172.21.15.46:0/4218038492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cdc08d6e0 0x7f4cdc14ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 >> 172.21.15.46:0/4218038492 conn(0x7f4cdc0b56e0 msgr2=0x7f4cdc161e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 shutdown_connections 2024-05-09T03:11:50.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.141+0000 7f4ce2366640 1 -- 172.21.15.46:0/4218038492 wait complete. 2024-05-09T03:11:50.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:50.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.267+0000 7f647aed9640 1 Processor -- start 2024-05-09T03:11:50.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.267+0000 7f647aed9640 1 -- start start 2024-05-09T03:11:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f647aed9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f647aed9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f647406de60 0x7f647406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f647aed9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f647406e7b0 con 0x7f64740d34e0 2024-05-09T03:11:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f647aed9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f647408de80 con 0x7f647406de60 2024-05-09T03:11:50.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f647aed9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f64740d2440 con 0x7f64740d1f60 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f6479476640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f647406de60 0x7f647406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f6479476640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f647406de60 0x7f647406e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40088/0 (socket says 172.21.15.46:40088) 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.268+0000 7f6479476640 1 -- 172.21.15.46:0/4009446029 learned_addr learned my addr 172.21.15.46:0/4009446029 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f6478c75640 1 --2- 172.21.15.46:0/4009446029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f6479476640 1 -- 172.21.15.46:0/4009446029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 msgr2=0x7f64740b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f6479476640 1 --2- 172.21.15.46:0/4009446029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740b2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f6479476640 1 -- 172.21.15.46:0/4009446029 >> v1:172.21.15.88:6789/0 conn(0x7f64740d34e0 legacy=0x7f647408d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:50.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f6479476640 1 -- 172.21.15.46:0/4009446029 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64740d2720 con 0x7f647406de60 2024-05-09T03:11:50.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f6479476640 1 --2- 172.21.15.46:0/4009446029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f647406de60 0x7f647406e270 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f646c009870 tx=0x7f646c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fdb815248c4c1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:50.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.269+0000 7f64737fe640 1 -- 172.21.15.46:0/4009446029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c037070 con 0x7f647406de60 2024-05-09T03:11:50.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f64737fe640 1 -- 172.21.15.46:0/4009446029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f646c032440 con 0x7f647406de60 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f64737fe640 1 -- 172.21.15.46:0/4009446029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c032810 con 0x7f647406de60 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 -- 172.21.15.46:0/4009446029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f647406de60 msgr2=0x7f647406e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 --2- 172.21.15.46:0/4009446029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f647406de60 0x7f647406e270 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f646c009870 tx=0x7f646c02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 -- 172.21.15.46:0/4009446029 shutdown_connections 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 --2- 172.21.15.46:0/4009446029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f647406de60 0x7f647406e270 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 --2- 172.21.15.46:0/4009446029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 -- 172.21.15.46:0/4009446029 >> 172.21.15.46:0/4009446029 conn(0x7f64740b56e0 msgr2=0x7f64740b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 -- 172.21.15.46:0/4009446029 shutdown_connections 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.270+0000 7f647aed9640 1 -- 172.21.15.46:0/4009446029 wait complete. 2024-05-09T03:11:50.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.272+0000 7f647aed9640 1 Processor -- start 2024-05-09T03:11:50.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.272+0000 7f647aed9640 1 -- start start 2024-05-09T03:11:50.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.273+0000 7f647aed9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f647406de60 0x7f6474162b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.273+0000 7f647aed9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.273+0000 7f647aed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 0x7f64740b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f647aed9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64740d2fc0 con 0x7f64740d34e0 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f647aed9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f64740d2ce0 con 0x7f647406de60 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f647aed9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f64740d38f0 con 0x7f64740d1f60 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 0x7f64740b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 0x7f64740b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40102/0 (socket says 172.21.15.46:40102) 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 -- 172.21.15.46:0/221067067 learned_addr learned my addr 172.21.15.46:0/221067067 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6478c75640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f647406de60 0x7f6474162b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6473fff640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 -- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 msgr2=0x7f64740cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 -- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f647406de60 msgr2=0x7f6474162b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f647406de60 0x7f6474162b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 -- 172.21.15.46:0/221067067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64740cba80 con 0x7f64740d34e0 2024-05-09T03:11:50.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f6479476640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 0x7f64740b7c30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f646c02a200 tx=0x7f646c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:50.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c041070 con 0x7f64740d34e0 2024-05-09T03:11:50.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.274+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f646c03c950 con 0x7f64740d34e0 2024-05-09T03:11:50.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.275+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64740b86b0 con 0x7f64740d34e0 2024-05-09T03:11:50.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.275+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c037040 con 0x7f64740d34e0 2024-05-09T03:11:50.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.275+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6474161160 con 0x7f64740d34e0 2024-05-09T03:11:50.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.275+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f647406a270 con 0x7f64740d34e0 2024-05-09T03:11:50.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.276+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f646c044070 con 0x7f64740d34e0 2024-05-09T03:11:50.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.277+0000 7f64717fa640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454076300 0x7f6454078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.277+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f646c0b82b0 con 0x7f64740d34e0 2024-05-09T03:11:50.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.277+0000 7f6478c75640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454076300 0x7f6454078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.277+0000 7f6478c75640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454076300 0x7f6454078810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f64740c76f0 tx=0x7f645c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:50.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.281+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f647406e7b0 con 0x7f64740d34e0 2024-05-09T03:11:50.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.511+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64740b9660 con 0x7f64740d34e0 2024-05-09T03:11:50.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.512+0000 7f64717fa640 1 -- 172.21.15.46:0/221067067 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5166 (secure 0 0 0) 0x7f646c081450 con 0x7f64740d34e0 2024-05-09T03:11:50.513 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:50.513 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:49.224734+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-05-09T03:11:50.513 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 35 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454076300 msgr2=0x7f6454078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454076300 0x7f6454078810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f64740c76f0 tx=0x7f645c008040 comp rx=0 tx=0).stop 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 msgr2=0x7f64740b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 0x7f64740b7c30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f646c02a200 tx=0x7f646c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 shutdown_connections 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454076300 0x7f6454078810 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64740d34e0 0x7f64740b7c30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64740d1f60 0x7f64740cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 --2- 172.21.15.46:0/221067067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f647406de60 0x7f6474162b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 >> 172.21.15.46:0/221067067 conn(0x7f64740b56e0 msgr2=0x7f647408d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 shutdown_connections 2024-05-09T03:11:50.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.514+0000 7f647aed9640 1 -- 172.21.15.46:0/221067067 wait complete. 2024-05-09T03:11:50.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:50.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.639+0000 7f1221cbd640 1 Processor -- start 2024-05-09T03:11:50.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.640+0000 7f1221cbd640 1 -- start start 2024-05-09T03:11:50.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f1221cbd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c0d1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f1221cbd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121c06ebe0 con 0x7f121c0b9310 2024-05-09T03:11:50.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f1221cbd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f121c06eec0 con 0x7f121c08d880 2024-05-09T03:11:50.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f1221cbd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f121c06f760 con 0x7f121c08d470 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f121b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f121c0b9310 0x7f121c08cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59952/0 (socket says 172.21.15.46:59952) 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f121b7fe640 1 -- 172.21.15.46:0/4054521231 learned_addr learned my addr 172.21.15.46:0/4054521231 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f121affd640 1 --2- 172.21.15.46:0/4054521231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c0d1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020748719 0 0) 0x7f121c06ebe0 con 0x7f121c0b9310 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.641+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1204003520 con 0x7f121c0b9310 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586068734 0 0) 0x7f121c06eec0 con 0x7f121c08d880 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f121c06ebe0 con 0x7f121c08d880 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1309067385 0 0) 0x7f1204003520 con 0x7f121c0b9310 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121affd640 1 -- 172.21.15.46:0/4054521231 >> v1:172.21.15.88:6789/0 conn(0x7f121c08d880 legacy=0x7f121c0d24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121affd640 1 -- 172.21.15.46:0/4054521231 >> v1:172.21.15.46:6789/0 conn(0x7f121c0b9310 legacy=0x7f121c08cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121affd640 1 -- 172.21.15.46:0/4054521231 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c06f1a0 con 0x7f121c08d470 2024-05-09T03:11:50.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.642+0000 7f121affd640 1 --2- 172.21.15.46:0/4054521231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c0d1f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12100097d0 tx=0x7f121002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=438572e0ce60347 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210037070 con 0x7f121c08d470 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121002ac30 con 0x7f121c08d470 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f121a7fc640 1 -- 172.21.15.46:0/4054521231 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120c001a70 con 0x7f121c08d470 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f1221cbd640 1 -- 172.21.15.46:0/4054521231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 msgr2=0x7f121c0d1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f1221cbd640 1 --2- 172.21.15.46:0/4054521231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c0d1f60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12100097d0 tx=0x7f121002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f1221cbd640 1 -- 172.21.15.46:0/4054521231 shutdown_connections 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f1221cbd640 1 --2- 172.21.15.46:0/4054521231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c0d1f60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f1221cbd640 1 -- 172.21.15.46:0/4054521231 >> 172.21.15.46:0/4054521231 conn(0x7f121c0b5660 msgr2=0x7f121c0b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.643+0000 7f1221cbd640 1 -- 172.21.15.46:0/4054521231 shutdown_connections 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.644+0000 7f1221cbd640 1 -- 172.21.15.46:0/4054521231 wait complete. 2024-05-09T03:11:50.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.645+0000 7f1221cbd640 1 Processor -- start 2024-05-09T03:11:50.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.646+0000 7f1221cbd640 1 -- start start 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f1221cbd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c1e3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f1221cbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f121c08d880 0x7f121c1dc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f1221cbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 0x7f121c1de4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f1221cbd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f121c0d2d30 con 0x7f121c0b9310 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f1221cbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f121c0d2bb0 con 0x7f121c08d880 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f1221cbd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f121c0cf590 con 0x7f121c08d470 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 0x7f121c1de4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f121c08d880 0x7f121c1dc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 0x7f121c1de4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40108/0 (socket says 172.21.15.46:40108) 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 -- 172.21.15.46:0/2443382532 learned_addr learned my addr 172.21.15.46:0/2443382532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c1e3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 -- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 msgr2=0x7f121c1e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c1e3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 -- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f121c08d880 msgr2=0x7f121c1dc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f121c08d880 0x7f121c1dc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.647+0000 7f121bfff640 1 -- 172.21.15.46:0/2443382532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c1ddcc0 con 0x7f121c0b9310 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.648+0000 7f121bfff640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 0x7f121c1de4e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f121400c3c0 tx=0x7f121400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.648+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214017070 con 0x7f121c0b9310 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.648+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12140052d0 con 0x7f121c0b9310 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.648+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214003830 con 0x7f121c0b9310 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.648+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121c1ddfa0 con 0x7f121c0b9310 2024-05-09T03:11:50.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.648+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121c1e1cd0 con 0x7f121c0b9310 2024-05-09T03:11:50.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.649+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f121c06a270 con 0x7f121c0b9310 2024-05-09T03:11:50.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.650+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f121401a070 con 0x7f121c0b9310 2024-05-09T03:11:50.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.650+0000 7f1203fff640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12080762d0 0x7f12080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:50.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.650+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f1214096970 con 0x7f121c0b9310 2024-05-09T03:11:50.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.650+0000 7f121b7fe640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12080762d0 0x7f12080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:50.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.651+0000 7f121b7fe640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12080762d0 0x7f12080787e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f120c002230 tx=0x7f120c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:50.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.655+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f121405fb90 con 0x7f121c0b9310 2024-05-09T03:11:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.883+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f121c1e35c0 con 0x7f121c0b9310 2024-05-09T03:11:50.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.884+0000 7f1203fff640 1 -- 172.21.15.46:0/2443382532 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5166 (secure 0 0 0) 0x7f1214068260 con 0x7f121c0b9310 2024-05-09T03:11:50.884 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:50.884 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:49.224734+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-05-09T03:11:50.885 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 35 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12080762d0 msgr2=0x7f12080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12080762d0 0x7f12080787e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f120c002230 tx=0x7f120c009040 comp rx=0 tx=0).stop 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 msgr2=0x7f121c1de4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 0x7f121c1de4e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f121400c3c0 tx=0x7f121400c880 comp rx=0 tx=0).stop 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 shutdown_connections 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12080762d0 0x7f12080787e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f121c0b9310 0x7f121c1de4e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f121c08d880 0x7f121c1dc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 --2- 172.21.15.46:0/2443382532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f121c08d470 0x7f121c1e3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 >> 172.21.15.46:0/2443382532 conn(0x7f121c0b5660 msgr2=0x7f121c0b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 shutdown_connections 2024-05-09T03:11:50.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:50.886+0000 7f1221cbd640 1 -- 172.21.15.46:0/2443382532 wait complete. 2024-05-09T03:11:50.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /a 0 2024-05-09T03:11:51.026 INFO:teuthology.orchestra.run.smithi046.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-05-09T03:11:45.411836+0000","btime":"2024-05-09T03:11:39.464882+0000","mode":16877,"uid":1000,"gid":1316,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-05-09T03:11:45.411836+0000","atime":"2024-05-09T03:11:39.464882+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-05-09T03:11:45.411836+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-05-09T03:11:45.411836+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-05-09T03:11:45.411836+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirfrag":1,"dirty":1},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1},"nref":1,"dentries":[{"path":"a/c","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"inodepin":1,"dirty":1},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirty":1},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-05-09T03:11:51.027 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-05-09T03:11:45.411836+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-05-09T03:11:39.464882+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-05-09T03:11:39.464882+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-05-09T03:11:45.411836+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "dirty": 1, "inodepin": 1 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "dirty": 1 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/a", "pins": { "child": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-05-09T03:11:45.411836+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1316, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-05-09T03:11:45.411836+0000", "nestlock": {}, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "dirfrag": 1, "dirty": 1 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iquiesce", "xlock_by": {} }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-05-09T03:11:45.411836+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-05-09T03:11:51.027 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-05-09T03:11:45.411836+0000', 'btime': '2024-05-09T03:11:39.464882+0000', 'mode': 16877, 'uid': 1000, 'gid': 1316, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-05-09T03:11:45.411836+0000', 'atime': '2024-05-09T03:11:39.464882+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'quiesce_block': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-05-09T03:11:45.411836+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-05-09T03:11:45.411836+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-05-09T03:11:45.411836+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirfrag': 1, 'dirty': 1}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {}, 'flocklock': {}, 'policylock': {}, 'quiescelock': {'gather_set': [], 'state': 'lock', 'type': 'iquiesce', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '0', 'committed_version': '0', 'is_rep': False, 'dir_auth': '', 'states': ['auth'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1}, 'nref': 1, 'dentries': [{'path': 'a/c', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'inodepin': 1, 'dirty': 1}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirty': 1}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-05-09T03:11:51.027 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:11:51.028 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:11:51.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:11:51.112 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:11:51.112 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:51.113 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:51.113 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:51.131 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:11:51.131 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:51.131 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:11:51.131 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:51.172 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:11:51.172 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:51.173 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:11:51.173 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:51.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:11:51.196 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:11:51.196 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:11:51.196 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:11:51.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:11:51.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.356+0000 7f3468042640 1 Processor -- start 2024-05-09T03:11:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.356+0000 7f3468042640 1 -- start start 2024-05-09T03:11:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3468042640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34600b4ff0 0x7f34600aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3468042640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3460070d80 0x7f3460073870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3468042640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f346015d980 con 0x7f34600af0c0 2024-05-09T03:11:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3468042640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f346015dc60 con 0x7f3460070d80 2024-05-09T03:11:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3468042640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f346015df40 con 0x7f34600b4ff0 2024-05-09T03:11:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34665df640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3460070d80 0x7f3460073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34655dd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34600af0c0 0x7f3460070670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59966/0 (socket says 172.21.15.46:59966) 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34655dd640 1 -- 172.21.15.46:0/430382129 learned_addr learned my addr 172.21.15.46:0/430382129 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3465dde640 1 --2- 172.21.15.46:0/430382129 >> v2:172.21.15.88:3300/0 conn(0x7f34600b4ff0 0x7f34600aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34665df640 1 -- 172.21.15.46:0/430382129 >> v2:172.21.15.88:3300/0 conn(0x7f34600b4ff0 msgr2=0x7f34600aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34665df640 1 --2- 172.21.15.46:0/430382129 >> v2:172.21.15.88:3300/0 conn(0x7f34600b4ff0 0x7f34600aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34665df640 1 -- 172.21.15.46:0/430382129 >> v1:172.21.15.46:6789/0 conn(0x7f34600af0c0 legacy=0x7f3460070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f34665df640 1 -- 172.21.15.46:0/430382129 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f346015e220 con 0x7f3460070d80 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.358+0000 7f3464ddc640 1 -- 172.21.15.46:0/430382129 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68756260 0 0) 0x7f346015d980 con 0x7f34600af0c0 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.359+0000 7f34665df640 1 --2- 172.21.15.46:0/430382129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460070d80 0x7f3460073870 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f34600713a0 tx=0x7f345802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b84e53ad5db251 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.359+0000 7f3464ddc640 1 -- 172.21.15.46:0/430382129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458037070 con 0x7f3460070d80 2024-05-09T03:11:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.359+0000 7f3464ddc640 1 -- 172.21.15.46:0/430382129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3458032440 con 0x7f3460070d80 2024-05-09T03:11:51.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.359+0000 7f3464ddc640 1 -- 172.21.15.46:0/430382129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345802ae60 con 0x7f3460070d80 2024-05-09T03:11:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 -- 172.21.15.46:0/430382129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460070d80 msgr2=0x7f3460073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 --2- 172.21.15.46:0/430382129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460070d80 0x7f3460073870 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f34600713a0 tx=0x7f345802a110 comp rx=0 tx=0).stop 2024-05-09T03:11:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 -- 172.21.15.46:0/430382129 shutdown_connections 2024-05-09T03:11:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 --2- 172.21.15.46:0/430382129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3460070d80 0x7f3460073870 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 --2- 172.21.15.46:0/430382129 >> v2:172.21.15.88:3300/0 conn(0x7f34600b4ff0 0x7f34600aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 -- 172.21.15.46:0/430382129 >> 172.21.15.46:0/430382129 conn(0x7f34600b3910 msgr2=0x7f34600b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 -- 172.21.15.46:0/430382129 shutdown_connections 2024-05-09T03:11:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.360+0000 7f3468042640 1 -- 172.21.15.46:0/430382129 wait complete. 2024-05-09T03:11:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.362+0000 7f3468042640 1 Processor -- start 2024-05-09T03:11:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.362+0000 7f3468042640 1 -- start start 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f3468042640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34600af0c0 0x7f34601441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f3468042640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 0x7f34601446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f3468042640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34601419d0 0x7f34601f9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f3468042640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f346015d6c0 con 0x7f34600b4ff0 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f3468042640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f346015d540 con 0x7f34600af0c0 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f3468042640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34600b1310 con 0x7f34601419d0 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f34655dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 0x7f34601446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.363+0000 7f34655dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 0x7f34601446f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40122/0 (socket says 172.21.15.46:40122) 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 -- 172.21.15.46:0/4162619627 learned_addr learned my addr 172.21.15.46:0/4162619627 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f3465dde640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34600af0c0 0x7f34601441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34665df640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34601419d0 0x7f34601f9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 -- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34601419d0 msgr2=0x7f34601f9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34601419d0 0x7f34601f9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 -- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34600af0c0 msgr2=0x7f34601441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34600af0c0 0x7f34601441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 -- 172.21.15.46:0/4162619627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3460143040 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34655dd640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 0x7f34601446f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f345000c3e0 tx=0x7f345000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34500074e0 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34500047e0 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450004bd0 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3460143320 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.364+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34601438e0 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.365+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f346006a270 con 0x7f34600b4ff0 2024-05-09T03:11:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.366+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3450011070 con 0x7f34600b4ff0 2024-05-09T03:11:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.366+0000 7f34567fc640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34440762d0 0x7f34440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.367+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f3450097190 con 0x7f34600b4ff0 2024-05-09T03:11:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.367+0000 7f3465dde640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34440762d0 0x7f34440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.367+0000 7f3465dde640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34440762d0 0x7f34440787e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f34600734b0 tx=0x7f345c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:51.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.371+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34500603b0 con 0x7f34600b4ff0 2024-05-09T03:11:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.604+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3460142410 con 0x7f34600b4ff0 2024-05-09T03:11:51.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.605+0000 7f34567fc640 1 -- 172.21.15.46:0/4162619627 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5163 (secure 0 0 0) 0x7f34500646b0 con 0x7f34600b4ff0 2024-05-09T03:11:51.606 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:51.606 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:50.215538+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-05-09T03:11:51.606 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34440762d0 msgr2=0x7f34440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34440762d0 0x7f34440787e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f34600734b0 tx=0x7f345c00b040 comp rx=0 tx=0).stop 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 msgr2=0x7f34601446f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 0x7f34601446f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f345000c3e0 tx=0x7f345000c8a0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 shutdown_connections 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34601419d0 0x7f34601f9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34440762d0 0x7f34440787e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34600b4ff0 0x7f34601446f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 --2- 172.21.15.46:0/4162619627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34600af0c0 0x7f34601441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 >> 172.21.15.46:0/4162619627 conn(0x7f34600b3910 msgr2=0x7f34600b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 shutdown_connections 2024-05-09T03:11:51.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.607+0000 7f3468042640 1 -- 172.21.15.46:0/4162619627 wait complete. 2024-05-09T03:11:51.619 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:51.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.732+0000 7fdfca078640 1 Processor -- start 2024-05-09T03:11:51.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.732+0000 7fdfca078640 1 -- start start 2024-05-09T03:11:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfca078640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdfc4164c70 0x7fdfc41615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfca078640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc4160b20 con 0x7fdfc40d2480 2024-05-09T03:11:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfca078640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfc40d15f0 con 0x7fdfc415dfe0 2024-05-09T03:11:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfca078640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdfc4161150 con 0x7fdfc4164c70 2024-05-09T03:11:51.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfc37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdfc4164c70 0x7fdfc41615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfc3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdfc40d2480 0x7fdfc40d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59980/0 (socket says 172.21.15.46:59980) 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfc37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdfc4164c70 0x7fdfc41615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40126/0 (socket says 172.21.15.46:40126) 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.733+0000 7fdfc3fff640 1 -- 172.21.15.46:0/3288956734 learned_addr learned my addr 172.21.15.46:0/3288956734 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.734+0000 7fdfc37fe640 1 -- 172.21.15.46:0/3288956734 >> v1:172.21.15.88:6789/0 conn(0x7fdfc415dfe0 legacy=0x7fdfc40d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.734+0000 7fdfc27fc640 1 -- 172.21.15.46:0/3288956734 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403942213 0 0) 0x7fdfc4160b20 con 0x7fdfc40d2480 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.734+0000 7fdfc37fe640 1 -- 172.21.15.46:0/3288956734 >> v1:172.21.15.46:6789/0 conn(0x7fdfc40d2480 legacy=0x7fdfc40d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.734+0000 7fdfc37fe640 1 -- 172.21.15.46:0/3288956734 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc40d18d0 con 0x7fdfc4164c70 2024-05-09T03:11:51.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.735+0000 7fdfc37fe640 1 --2- 172.21.15.46:0/3288956734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc4164c70 0x7fdfc41615e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0097a0 tx=0x7fdfac02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83afa7aff397db7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:51.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.735+0000 7fdfc27fc640 1 -- 172.21.15.46:0/3288956734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac037070 con 0x7fdfc4164c70 2024-05-09T03:11:51.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.735+0000 7fdfc27fc640 1 -- 172.21.15.46:0/3288956734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfac032440 con 0x7fdfc4164c70 2024-05-09T03:11:51.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.735+0000 7fdfc27fc640 1 -- 172.21.15.46:0/3288956734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac032810 con 0x7fdfc4164c70 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.735+0000 7fdfca078640 1 -- 172.21.15.46:0/3288956734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc4164c70 msgr2=0x7fdfc41615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.735+0000 7fdfca078640 1 --2- 172.21.15.46:0/3288956734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc4164c70 0x7fdfc41615e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0097a0 tx=0x7fdfac02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.736+0000 7fdfca078640 1 -- 172.21.15.46:0/3288956734 shutdown_connections 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.736+0000 7fdfca078640 1 --2- 172.21.15.46:0/3288956734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc4164c70 0x7fdfc41615e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.736+0000 7fdfca078640 1 -- 172.21.15.46:0/3288956734 >> 172.21.15.46:0/3288956734 conn(0x7fdfc4073db0 msgr2=0x7fdfc40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.736+0000 7fdfca078640 1 -- 172.21.15.46:0/3288956734 shutdown_connections 2024-05-09T03:11:51.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.736+0000 7fdfca078640 1 -- 172.21.15.46:0/3288956734 wait complete. 2024-05-09T03:11:51.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.737+0000 7fdfca078640 1 Processor -- start 2024-05-09T03:11:51.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.738+0000 7fdfca078640 1 -- start start 2024-05-09T03:11:51.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfca078640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 0x7fdfc40ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfca078640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4162910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfca078640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4164c70 0x7fdfc41f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfca078640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc40d45c0 con 0x7fdfc40d2480 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfca078640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc415e3f0 con 0x7fdfc4164c70 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfca078640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdfc40d4740 con 0x7fdfc415dfe0 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 0x7fdfc40ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 0x7fdfc40ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40130/0 (socket says 172.21.15.46:40130) 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 -- 172.21.15.46:0/2985768804 learned_addr learned my addr 172.21.15.46:0/2985768804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc3fff640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4164c70 0x7fdfc41f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 -- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfc415dfe0 msgr2=0x7fdfc4162910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc2ffd640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4162910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4162910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 -- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4164c70 msgr2=0x7fdfc41f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4164c70 0x7fdfc41f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc37fe640 1 -- 172.21.15.46:0/2985768804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc40c9e90 con 0x7fdfc40d2480 2024-05-09T03:11:51.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc3fff640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4164c70 0x7fdfc41f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.739+0000 7fdfc2ffd640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4162910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.740+0000 7fdfc37fe640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 0x7fdfc40ccd50 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0097a0 tx=0x7fdfac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.740+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac037050 con 0x7fdfc40d2480 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.740+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfac038040 con 0x7fdfc40d2480 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.740+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc40ca100 con 0x7fdfc40d2480 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.740+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac004490 con 0x7fdfc40d2480 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.740+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc40ca6c0 con 0x7fdfc40d2480 2024-05-09T03:11:51.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.741+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc408cfd0 con 0x7fdfc40d2480 2024-05-09T03:11:51.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.741+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdfac030900 con 0x7fdfc40d2480 2024-05-09T03:11:51.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.742+0000 7fdfbbfff640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfa40762d0 0x7fdfa40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:51.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.742+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7fdfac0b82f0 con 0x7fdfc40d2480 2024-05-09T03:11:51.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.742+0000 7fdfc2ffd640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfa40762d0 0x7fdfa40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:51.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.743+0000 7fdfc2ffd640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfa40762d0 0x7fdfa40787e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4000f60 tx=0x7fdfb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:51.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.747+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdfc408cfd0 con 0x7fdfc40d2480 2024-05-09T03:11:51.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.982+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfc406a270 con 0x7fdfc40d2480 2024-05-09T03:11:51.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.982+0000 7fdfbbfff640 1 -- 172.21.15.46:0/2985768804 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+5163 (secure 0 0 0) 0x7fdfac081b60 con 0x7fdfc40d2480 2024-05-09T03:11:51.987 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:51.987 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:50.215538+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 36 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfa40762d0 msgr2=0x7fdfa40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfa40762d0 0x7fdfa40787e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4000f60 tx=0x7fdfb4009040 comp rx=0 tx=0).stop 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 msgr2=0x7fdfc40ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 0x7fdfc40ccd50 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0097a0 tx=0x7fdfac0046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 shutdown_connections 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfc4164c70 0x7fdfc41f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdfc415dfe0 0x7fdfc4162910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfa40762d0 0x7fdfa40787e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 --2- 172.21.15.46:0/2985768804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdfc40d2480 0x7fdfc40ccd50 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.984+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 >> 172.21.15.46:0/2985768804 conn(0x7fdfc4073db0 msgr2=0x7fdfc40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.985+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 shutdown_connections 2024-05-09T03:11:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:51.985+0000 7fdfca078640 1 -- 172.21.15.46:0/2985768804 wait complete. 2024-05-09T03:11:51.996 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:52.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.110+0000 7f463c96c640 1 Processor -- start 2024-05-09T03:11:52.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.110+0000 7f463c96c640 1 -- start start 2024-05-09T03:11:52.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463c96c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46380b7ed0 0x7f4638163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463c96c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46381617d0 con 0x7f46380b5210 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463c96c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46380d5280 con 0x7f46381613c0 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463c96c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46380b5620 con 0x7f46380b7ed0 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f46380b5210 0x7f46380d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59992/0 (socket says 172.21.15.46:59992) 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463659c640 1 -- 172.21.15.46:0/938866297 learned_addr learned my addr 172.21.15.46:0/938866297 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f4636d9d640 1 --2- 172.21.15.46:0/938866297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46380b7ed0 0x7f4638163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 460704521 0 0) 0x7f46381617d0 con 0x7f46380b5210 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.112+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4620003520 con 0x7f46380b5210 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.113+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319363922 0 0) 0x7f46380d5280 con 0x7f46381613c0 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.113+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46381617d0 con 0x7f46381613c0 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.113+0000 7f4636d9d640 1 -- 172.21.15.46:0/938866297 >> v1:172.21.15.88:6789/0 conn(0x7f46381613c0 legacy=0x7f46380b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.113+0000 7f4636d9d640 1 -- 172.21.15.46:0/938866297 >> v1:172.21.15.46:6789/0 conn(0x7f46380b5210 legacy=0x7f46380d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.113+0000 7f4636d9d640 1 -- 172.21.15.46:0/938866297 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46380d5560 con 0x7f46380b7ed0 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.113+0000 7f4636d9d640 1 --2- 172.21.15.46:0/938866297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46380b7ed0 0x7f4638163e60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4630009dc0 tx=0x7f463002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20c06d186f7375d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630007740 con 0x7f46380b7ed0 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46300078e0 con 0x7f46380b7ed0 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463559a640 1 -- 172.21.15.46:0/938866297 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630007c90 con 0x7f46380b7ed0 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463c96c640 1 -- 172.21.15.46:0/938866297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46380b7ed0 msgr2=0x7f4638163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463c96c640 1 --2- 172.21.15.46:0/938866297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46380b7ed0 0x7f4638163e60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4630009dc0 tx=0x7f463002a6d0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463c96c640 1 -- 172.21.15.46:0/938866297 shutdown_connections 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463c96c640 1 --2- 172.21.15.46:0/938866297 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46380b7ed0 0x7f4638163e60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.114+0000 7f463c96c640 1 -- 172.21.15.46:0/938866297 >> 172.21.15.46:0/938866297 conn(0x7f46380d29a0 msgr2=0x7f46380d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.115+0000 7f463c96c640 1 -- 172.21.15.46:0/938866297 shutdown_connections 2024-05-09T03:11:52.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.115+0000 7f463c96c640 1 -- 172.21.15.46:0/938866297 wait complete. 2024-05-09T03:11:52.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.116+0000 7f463c96c640 1 Processor -- start 2024-05-09T03:11:52.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.116+0000 7f463c96c640 1 -- start start 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463c96c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380b5210 0x7f46381f9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463c96c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46381613c0 0x7f46381fa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463c96c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 0x7f46381f8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463c96c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4638160ea0 con 0x7f46380b61b0 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463c96c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f46381609a0 con 0x7f46380b5210 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463c96c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46380d4f10 con 0x7f46381613c0 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 0x7f46381f8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 0x7f46381f8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40134/0 (socket says 172.21.15.46:40134) 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 -- 172.21.15.46:0/567511753 learned_addr learned my addr 172.21.15.46:0/567511753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463659c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380b5210 0x7f46381f9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4635d9b640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46381613c0 0x7f46381fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 -- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46381613c0 msgr2=0x7f46381fa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46381613c0 0x7f46381fa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 -- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380b5210 msgr2=0x7f46381f9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380b5210 0x7f46381f9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 -- 172.21.15.46:0/567511753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46381f6570 con 0x7f46380b61b0 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f463659c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380b5210 0x7f46381f9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4635d9b640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46381613c0 0x7f46381fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.118+0000 7f4636d9d640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 0x7f46381f8e10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f463002acc0 tx=0x7f4630004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.119+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630009450 con 0x7f46380b61b0 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.119+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4630004240 con 0x7f46380b61b0 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.119+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46380b6920 con 0x7f46380b61b0 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.119+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630004610 con 0x7f46380b61b0 2024-05-09T03:11:52.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.119+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46380b6ee0 con 0x7f46380b61b0 2024-05-09T03:11:52.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.120+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46380b6670 con 0x7f46380b61b0 2024-05-09T03:11:52.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.121+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f463003b410 con 0x7f46380b61b0 2024-05-09T03:11:52.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.121+0000 7f4626ffd640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4618076300 0x7f4618078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.121+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f46300b8d20 con 0x7f46380b61b0 2024-05-09T03:11:52.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.122+0000 7f463659c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4618076300 0x7f4618078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.122+0000 7f463659c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4618076300 0x7f4618078810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4628007fd0 tx=0x7f4628009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.126+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f46381f6570 con 0x7f46380b61b0 2024-05-09T03:11:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.321+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f463806a270 con 0x7f46380b61b0 2024-05-09T03:11:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.322+0000 7f4626ffd640 1 -- 172.21.15.46:0/567511753 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16670 (secure 0 0 0) 0x7f463003f030 con 0x7f46380b61b0 2024-05-09T03:11:52.323 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:52.323 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":44,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:47.200197+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:11:24.480773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T03:11:25.741329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/2630572362":"2024-05-10T03:11:47.200166+0000","172.21.15.46:6833/2630572362":"2024-05-10T03:11:47.200166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4618076300 msgr2=0x7f4618078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4618076300 0x7f4618078810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4628007fd0 tx=0x7f4628009040 comp rx=0 tx=0).stop 2024-05-09T03:11:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 msgr2=0x7f46381f8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 0x7f46381f8e10 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f463002acc0 tx=0x7f4630004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 shutdown_connections 2024-05-09T03:11:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4618076300 0x7f4618078810 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46380b61b0 0x7f46381f8e10 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46381613c0 0x7f46381fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 --2- 172.21.15.46:0/567511753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380b5210 0x7f46381f9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 >> 172.21.15.46:0/567511753 conn(0x7f46380d29a0 msgr2=0x7f463806ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 shutdown_connections 2024-05-09T03:11:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.324+0000 7f463c96c640 1 -- 172.21.15.46:0/567511753 wait complete. 2024-05-09T03:11:52.336 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:11:52.336 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.450+0000 7f6370db2640 1 Processor -- start 2024-05-09T03:11:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.450+0000 7f6370db2640 1 -- start start 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6370db2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f636c0d57a0 0x7f636c160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6370db2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f636c0b4af0 0x7f636c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6370db2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f636c0d49e0 con 0x7f636c0b77e0 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6370db2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f636c0d5bb0 con 0x7f636c0b4af0 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6370db2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f636c0d4cc0 con 0x7f636c0d57a0 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6369d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f636c0b4af0 0x7f636c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f636ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f636c0b77e0 0x7f636c0b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59994/0 (socket says 172.21.15.46:59994) 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f636ad9d640 1 -- 172.21.15.46:0/2687251141 learned_addr learned my addr 172.21.15.46:0/2687251141 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:52.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f636a59c640 1 --2- 172.21.15.46:0/2687251141 >> v2:172.21.15.88:3300/0 conn(0x7f636c0d57a0 0x7f636c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6369d9b640 1 -- 172.21.15.46:0/2687251141 >> v2:172.21.15.88:3300/0 conn(0x7f636c0d57a0 msgr2=0x7f636c160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6369d9b640 1 --2- 172.21.15.46:0/2687251141 >> v2:172.21.15.88:3300/0 conn(0x7f636c0d57a0 0x7f636c160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6369d9b640 1 -- 172.21.15.46:0/2687251141 >> v1:172.21.15.46:6789/0 conn(0x7f636c0b77e0 legacy=0x7f636c0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f636959a640 1 -- 172.21.15.46:0/2687251141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 81889930 0 0) 0x7f636c0d49e0 con 0x7f636c0b77e0 2024-05-09T03:11:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.452+0000 7f6369d9b640 1 -- 172.21.15.46:0/2687251141 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636c0d4fa0 con 0x7f636c0b4af0 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.453+0000 7f6369d9b640 1 --2- 172.21.15.46:0/2687251141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c0b4f00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f636c08ed20 tx=0x7f636002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d2ec2dfdd696fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.453+0000 7f636959a640 1 -- 172.21.15.46:0/2687251141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360037070 con 0x7f636c0b4af0 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.453+0000 7f636959a640 1 -- 172.21.15.46:0/2687251141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6360032440 con 0x7f636c0b4af0 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.453+0000 7f636959a640 1 -- 172.21.15.46:0/2687251141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360032810 con 0x7f636c0b4af0 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 -- 172.21.15.46:0/2687251141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 msgr2=0x7f636c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 --2- 172.21.15.46:0/2687251141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c0b4f00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f636c08ed20 tx=0x7f636002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 -- 172.21.15.46:0/2687251141 shutdown_connections 2024-05-09T03:11:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 --2- 172.21.15.46:0/2687251141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c0b4f00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 --2- 172.21.15.46:0/2687251141 >> v2:172.21.15.88:3300/0 conn(0x7f636c0d57a0 0x7f636c160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 -- 172.21.15.46:0/2687251141 >> 172.21.15.46:0/2687251141 conn(0x7f636c0743e0 msgr2=0x7f636c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 -- 172.21.15.46:0/2687251141 shutdown_connections 2024-05-09T03:11:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.454+0000 7f6370db2640 1 -- 172.21.15.46:0/2687251141 wait complete. 2024-05-09T03:11:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.456+0000 7f6370db2640 1 Processor -- start 2024-05-09T03:11:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.456+0000 7f6370db2640 1 -- start start 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f6370db2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c14d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f6370db2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c0b77e0 0x7f636c161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f6370db2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c0d57a0 0x7f636c14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f6370db2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f636c0d1920 con 0x7f636c0b4af0 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f6370db2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f636c0d15b0 con 0x7f636c0b77e0 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f6370db2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f636c14ead0 con 0x7f636c0d57a0 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f636a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c14d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f636a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c14d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40160/0 (socket says 172.21.15.46:40160) 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.457+0000 7f636a59c640 1 -- 172.21.15.46:0/2782273916 learned_addr learned my addr 172.21.15.46:0/2782273916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:52.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636ad9d640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c0d57a0 0x7f636c14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f6369d9b640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c0b77e0 0x7f636c161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636a59c640 1 -- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c0d57a0 msgr2=0x7f636c14daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636a59c640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c0d57a0 0x7f636c14daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636a59c640 1 -- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c0b77e0 msgr2=0x7f636c161660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636a59c640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c0b77e0 0x7f636c161660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636a59c640 1 -- 172.21.15.46:0/2782273916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636c14d690 con 0x7f636c0b4af0 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f636a59c640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c14d150 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f635c00e840 tx=0x7f635c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c017070 con 0x7f636c0b4af0 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635c00c940 con 0x7f636c0b4af0 2024-05-09T03:11:52.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c00cd10 con 0x7f636c0b4af0 2024-05-09T03:11:52.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636c162120 con 0x7f636c0b4af0 2024-05-09T03:11:52.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.458+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f636c1626e0 con 0x7f636c0b4af0 2024-05-09T03:11:52.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.459+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f636c06a270 con 0x7f636c0b4af0 2024-05-09T03:11:52.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.460+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f635c01a070 con 0x7f636c0b4af0 2024-05-09T03:11:52.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.460+0000 7f635affd640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f634c076300 0x7f634c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.460+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f635c097540 con 0x7f636c0b4af0 2024-05-09T03:11:52.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.460+0000 7f6369d9b640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f634c076300 0x7f634c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.461+0000 7f6369d9b640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f634c076300 0x7f634c078810 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6360034000 tx=0x7f636002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.465+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f635c060db0 con 0x7f636c0b4af0 2024-05-09T03:11:52.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.692+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f636c0c9300 con 0x7f636c0b4af0 2024-05-09T03:11:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.693+0000 7f635affd640 1 -- 172.21.15.46:0/2782273916 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5170 (secure 0 0 0) 0x7f635c0606b0 con 0x7f636c0b4af0 2024-05-09T03:11:52.694 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:52.694 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:52.220626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5637,"qdb_cluster":[5637]},"id":3}]} 2024-05-09T03:11:52.694 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2024-05-09T03:11:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f634c076300 msgr2=0x7f634c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f634c076300 0x7f634c078810 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6360034000 tx=0x7f636002e000 comp rx=0 tx=0).stop 2024-05-09T03:11:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 msgr2=0x7f636c14d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c14d150 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f635c00e840 tx=0x7f635c00ed00 comp rx=0 tx=0).stop 2024-05-09T03:11:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 shutdown_connections 2024-05-09T03:11:52.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f636c0d57a0 0x7f636c14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f636c0b77e0 0x7f636c161660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f634c076300 0x7f634c078810 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 --2- 172.21.15.46:0/2782273916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f636c0b4af0 0x7f636c14d150 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 >> 172.21.15.46:0/2782273916 conn(0x7f636c0743e0 msgr2=0x7f636c0bac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 shutdown_connections 2024-05-09T03:11:52.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.695+0000 7f6370db2640 1 -- 172.21.15.46:0/2782273916 wait complete. 2024-05-09T03:11:52.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:11:52.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.821+0000 7f904bcd8640 1 Processor -- start 2024-05-09T03:11:52.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.821+0000 7f904bcd8640 1 -- start start 2024-05-09T03:11:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904bcd8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90440b5210 0x7f90441613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904bcd8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90440d4800 0x7f90440d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904bcd8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90440d5280 con 0x7f90440b77c0 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904bcd8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f90440b5620 con 0x7f90440b5210 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904bcd8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f90441609a0 con 0x7f90440d4800 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f9049a74640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90440b5210 0x7f90441613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904a275640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90440b77c0 0x7f90440b7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60000/0 (socket says 172.21.15.46:60000) 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.822+0000 7f904a275640 1 -- 172.21.15.46:0/3980444039 learned_addr learned my addr 172.21.15.46:0/3980444039 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.823+0000 7f9049273640 1 --2- 172.21.15.46:0/3980444039 >> v2:172.21.15.88:3300/0 conn(0x7f90440d4800 0x7f90440d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.823+0000 7f9049a74640 1 -- 172.21.15.46:0/3980444039 >> v2:172.21.15.88:3300/0 conn(0x7f90440d4800 msgr2=0x7f90440d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.823+0000 7f9049a74640 1 --2- 172.21.15.46:0/3980444039 >> v2:172.21.15.88:3300/0 conn(0x7f90440d4800 0x7f90440d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.823+0000 7f9049a74640 1 -- 172.21.15.46:0/3980444039 >> v1:172.21.15.46:6789/0 conn(0x7f90440b77c0 legacy=0x7f90440b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.823+0000 7f9049a74640 1 -- 172.21.15.46:0/3980444039 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90440d5560 con 0x7f90440b5210 2024-05-09T03:11:52.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.823+0000 7f9049a74640 1 --2- 172.21.15.46:0/3980444039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440b5210 0x7f90441613c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f904406ee20 tx=0x7f904002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e1332f5e4cdcdb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f9048a72640 1 -- 172.21.15.46:0/3980444039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9040037070 con 0x7f90440b5210 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f9048a72640 1 -- 172.21.15.46:0/3980444039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9040032440 con 0x7f90440b5210 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f9048a72640 1 -- 172.21.15.46:0/3980444039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904002ae60 con 0x7f90440b5210 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f904bcd8640 1 -- 172.21.15.46:0/3980444039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440b5210 msgr2=0x7f90441613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f904bcd8640 1 --2- 172.21.15.46:0/3980444039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440b5210 0x7f90441613c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f904406ee20 tx=0x7f904002a110 comp rx=0 tx=0).stop 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f904bcd8640 1 -- 172.21.15.46:0/3980444039 shutdown_connections 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f904bcd8640 1 --2- 172.21.15.46:0/3980444039 >> v2:172.21.15.88:3300/0 conn(0x7f90440d4800 0x7f90440d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f904bcd8640 1 --2- 172.21.15.46:0/3980444039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440b5210 0x7f90441613c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.824+0000 7f904bcd8640 1 -- 172.21.15.46:0/3980444039 >> 172.21.15.46:0/3980444039 conn(0x7f90440d29a0 msgr2=0x7f90440d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.825+0000 7f904bcd8640 1 -- 172.21.15.46:0/3980444039 shutdown_connections 2024-05-09T03:11:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.825+0000 7f904bcd8640 1 -- 172.21.15.46:0/3980444039 wait complete. 2024-05-09T03:11:52.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.826+0000 7f904bcd8640 1 Processor -- start 2024-05-09T03:11:52.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.826+0000 7f904bcd8640 1 -- start start 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90440b5210 0x7f90440c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90440b77c0 0x7f90441f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 0x7f90440ba350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90440d0ae0 con 0x7f90440d4800 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9044160ea0 con 0x7f90440b77c0 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9044163e60 con 0x7f90440b5210 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 0x7f90440ba350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 0x7f90440ba350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40184/0 (socket says 172.21.15.46:40184) 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 -- 172.21.15.46:0/646561024 learned_addr learned my addr 172.21.15.46:0/646561024 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f9049273640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90440b77c0 0x7f90441f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f9049a74640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90440b5210 0x7f90440c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 -- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90440b5210 msgr2=0x7f90440c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90440b5210 0x7f90440c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 -- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90440b77c0 msgr2=0x7f90441f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90440b77c0 0x7f90441f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 -- 172.21.15.46:0/646561024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90440cc6f0 con 0x7f90440d4800 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904a275640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 0x7f90440ba350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f903c00caa0 tx=0x7f903c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c017070 con 0x7f90440d4800 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903c00ac00 con 0x7f90440d4800 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90440c9eb0 con 0x7f90440d4800 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c012400 con 0x7f90440d4800 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.828+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90440cf830 con 0x7f90440d4800 2024-05-09T03:11:52.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.829+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f904406a270 con 0x7f90440d4800 2024-05-09T03:11:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.830+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f903c01a070 con 0x7f90440d4800 2024-05-09T03:11:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.831+0000 7f903a7fc640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9028076300 0x7f9028078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:52.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.831+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f903c097990 con 0x7f90440d4800 2024-05-09T03:11:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.831+0000 7f9049a74640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9028076300 0x7f9028078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:52.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.832+0000 7f9049a74640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9028076300 0x7f9028078810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9040034000 tx=0x7f904002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:52.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:52.835+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f903c060cb0 con 0x7f90440d4800 2024-05-09T03:11:53.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.064+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90440cc480 con 0x7f90440d4800 2024-05-09T03:11:53.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.065+0000 7f903a7fc640 1 -- 172.21.15.46:0/646561024 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5170 (secure 0 0 0) 0x7f903c064fb0 con 0x7f90440d4800 2024-05-09T03:11:53.066 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:53.066 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:11:26.786006+0000","modified":"2024-05-09T03:11:52.220626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5637":{"gid":5637,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/881221292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":881221292},{"type":"v1","addr":"172.21.15.46:6835","nonce":881221292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5637,"qdb_cluster":[5637]},"id":3}]} 2024-05-09T03:11:53.066 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 37 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9028076300 msgr2=0x7f9028078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9028076300 0x7f9028078810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9040034000 tx=0x7f904002e000 comp rx=0 tx=0).stop 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 msgr2=0x7f90440ba350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 0x7f90440ba350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f903c00caa0 tx=0x7f903c00cf60 comp rx=0 tx=0).stop 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 shutdown_connections 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9028076300 0x7f9028078810 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90440d4800 0x7f90440ba350 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90440b77c0 0x7f90441f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 --2- 172.21.15.46:0/646561024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90440b5210 0x7f90440c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 >> 172.21.15.46:0/646561024 conn(0x7f90440d29a0 msgr2=0x7f90440b7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 shutdown_connections 2024-05-09T03:11:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.067+0000 7f904bcd8640 1 -- 172.21.15.46:0/646561024 wait complete. 2024-05-09T03:11:53.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.193+0000 7f6622334640 1 Processor -- start 2024-05-09T03:11:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.193+0000 7f6622334640 1 -- start start 2024-05-09T03:11:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f6622334640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f661c164c70 0x7f661c1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f6622334640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f661c15dfe0 0x7f661c0d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f6622334640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f661c160b20 con 0x7f661c0d2480 2024-05-09T03:11:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f6622334640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f661c161150 con 0x7f661c164c70 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f6622334640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f661c0d15f0 con 0x7f661c15dfe0 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f661bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f661c164c70 0x7f661c1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f66208d1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f661c0d2480 0x7f661c0d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60016/0 (socket says 172.21.15.46:60016) 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.194+0000 7f66208d1640 1 -- 172.21.15.46:0/2604841064 learned_addr learned my addr 172.21.15.46:0/2604841064 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.195+0000 7f661b7fe640 1 --2- 172.21.15.46:0/2604841064 >> v2:172.21.15.88:3300/0 conn(0x7f661c15dfe0 0x7f661c0d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.195+0000 7f661bfff640 1 -- 172.21.15.46:0/2604841064 >> v2:172.21.15.88:3300/0 conn(0x7f661c15dfe0 msgr2=0x7f661c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.195+0000 7f661bfff640 1 --2- 172.21.15.46:0/2604841064 >> v2:172.21.15.88:3300/0 conn(0x7f661c15dfe0 0x7f661c0d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.195+0000 7f661bfff640 1 -- 172.21.15.46:0/2604841064 >> v1:172.21.15.46:6789/0 conn(0x7f661c0d2480 legacy=0x7f661c0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.195+0000 7f661affd640 1 -- 172.21.15.46:0/2604841064 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129174259 0 0) 0x7f661c160b20 con 0x7f661c0d2480 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.195+0000 7f661bfff640 1 -- 172.21.15.46:0/2604841064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c0d18d0 con 0x7f661c164c70 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f661bfff640 1 --2- 172.21.15.46:0/2604841064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c164c70 0x7f661c1615e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f661c0cce00 tx=0x7f660c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a59b3bf64f4ca5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f661affd640 1 -- 172.21.15.46:0/2604841064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c037070 con 0x7f661c164c70 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f661affd640 1 -- 172.21.15.46:0/2604841064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c032440 con 0x7f661c164c70 2024-05-09T03:11:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f661affd640 1 -- 172.21.15.46:0/2604841064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c02ae60 con 0x7f661c164c70 2024-05-09T03:11:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 -- 172.21.15.46:0/2604841064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c164c70 msgr2=0x7f661c1615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 --2- 172.21.15.46:0/2604841064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c164c70 0x7f661c1615e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f661c0cce00 tx=0x7f660c02a110 comp rx=0 tx=0).stop 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 -- 172.21.15.46:0/2604841064 shutdown_connections 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 --2- 172.21.15.46:0/2604841064 >> v2:172.21.15.88:3300/0 conn(0x7f661c15dfe0 0x7f661c0d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 --2- 172.21.15.46:0/2604841064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c164c70 0x7f661c1615e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 -- 172.21.15.46:0/2604841064 >> 172.21.15.46:0/2604841064 conn(0x7f661c073db0 msgr2=0x7f661c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.196+0000 7f6622334640 1 -- 172.21.15.46:0/2604841064 shutdown_connections 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.197+0000 7f6622334640 1 -- 172.21.15.46:0/2604841064 wait complete. 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.198+0000 7f6622334640 1 Processor -- start 2024-05-09T03:11:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.199+0000 7f6622334640 1 -- start start 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f6622334640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 0x7f661c1f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f6622334640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f661c15dfe0 0x7f661c164060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f6622334640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f661c164c70 0x7f661c1542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f6622334640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f661c15e3f0 con 0x7f661c0d2480 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f6622334640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f661c0d1300 con 0x7f661c15dfe0 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f6622334640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f661c0d45c0 con 0x7f661c164c70 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 0x7f661c1f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 0x7f661c1f6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40206/0 (socket says 172.21.15.46:40206) 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 -- 172.21.15.46:0/351665915 learned_addr learned my addr 172.21.15.46:0/351665915 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f66208d1640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f661c164c70 0x7f661c1542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661b7fe640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f661c15dfe0 0x7f661c164060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 -- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f661c164c70 msgr2=0x7f661c1542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f661c164c70 0x7f661c1542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 -- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f661c15dfe0 msgr2=0x7f661c164060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f661c15dfe0 0x7f661c164060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 -- 172.21.15.46:0/351665915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661c153220 con 0x7f661c0d2480 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661b7fe640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f661c15dfe0 0x7f661c164060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.200+0000 7f661bfff640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 0x7f661c1f6590 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f660c000c00 tx=0x7f660c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.201+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c037040 con 0x7f661c0d2480 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.201+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c042070 con 0x7f661c0d2480 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.201+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c030bd0 con 0x7f661c0d2480 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.201+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f661c153500 con 0x7f661c0d2480 2024-05-09T03:11:53.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.201+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f661c153ac0 con 0x7f661c0d2480 2024-05-09T03:11:53.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.202+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661c08cfd0 con 0x7f661c0d2480 2024-05-09T03:11:53.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.203+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f660c047460 con 0x7f661c0d2480 2024-05-09T03:11:53.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.203+0000 7f6618ff9640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66080762d0 0x7f66080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.203+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f660c0b82a0 con 0x7f661c0d2480 2024-05-09T03:11:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.203+0000 7f661b7fe640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66080762d0 0x7f66080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.204+0000 7f661b7fe640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66080762d0 0x7f66080787e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6610006fd0 tx=0x7f6610008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.208+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f661c08cfd0 con 0x7f661c0d2480 2024-05-09T03:11:53.209 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:11:53.208+0000 7febacc12640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:11:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.402+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f661c06a270 con 0x7f661c0d2480 2024-05-09T03:11:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.403+0000 7f6618ff9640 1 -- 172.21.15.46:0/351665915 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16670 (secure 0 0 0) 0x7f660c032e20 con 0x7f661c0d2480 2024-05-09T03:11:53.404 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:53.405 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":44,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:47.200197+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:11:24.480773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-05-09T03:11:25.741329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/2630572362":"2024-05-10T03:11:47.200166+0000","172.21.15.46:6833/2630572362":"2024-05-10T03:11:47.200166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66080762d0 msgr2=0x7f66080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66080762d0 0x7f66080787e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6610006fd0 tx=0x7f6610008040 comp rx=0 tx=0).stop 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 msgr2=0x7f661c1f6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 0x7f661c1f6590 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f660c000c00 tx=0x7f660c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 shutdown_connections 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f661c164c70 0x7f661c1542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f661c15dfe0 0x7f661c164060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66080762d0 0x7f66080787e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 --2- 172.21.15.46:0/351665915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f661c0d2480 0x7f661c1f6590 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 >> 172.21.15.46:0/351665915 conn(0x7f661c073db0 msgr2=0x7f661c0b9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 shutdown_connections 2024-05-09T03:11:53.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.406+0000 7f6622334640 1 -- 172.21.15.46:0/351665915 wait complete. 2024-05-09T03:11:53.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:11:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.533+0000 7f4bd39d6640 1 Processor -- start 2024-05-09T03:11:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.533+0000 7f4bd39d6640 1 -- start start 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.534+0000 7f4bd39d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bcc0b5140 0x7f4bcc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd39d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd39d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bcc0b80b0 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd39d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4bcc0d3480 con 0x7f4bcc0b5140 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd39d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4bcc0b5550 con 0x7f4bcc0b7ca0 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1772640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bcc0b5140 0x7f4bcc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1f73640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4bcc0d3df0 0x7f4bcc163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60032/0 (socket says 172.21.15.46:60032) 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1f73640 1 -- 172.21.15.46:0/1701932612 learned_addr learned my addr 172.21.15.46:0/1701932612 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1772640 1 -- 172.21.15.46:0/1701932612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 msgr2=0x7f4bcc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1772640 1 --2- 172.21.15.46:0/1701932612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1772640 1 -- 172.21.15.46:0/1701932612 >> v1:172.21.15.46:6789/0 conn(0x7f4bcc0d3df0 legacy=0x7f4bcc163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bd1772640 1 -- 172.21.15.46:0/1701932612 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc0d44e0 con 0x7f4bcc0b5140 2024-05-09T03:11:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.535+0000 7f4bc3fff640 1 -- 172.21.15.46:0/1701932612 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647742654 0 0) 0x7f4bc4001490 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.536+0000 7f4bd1772640 1 --2- 172.21.15.46:0/1701932612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc0b7760 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80097a0 tx=0x7f4bc802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7c1f5b6beff12 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.536+0000 7f4bc3fff640 1 -- 172.21.15.46:0/1701932612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8037070 con 0x7f4bcc0b5140 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.536+0000 7f4bc3fff640 1 -- 172.21.15.46:0/1701932612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc8032440 con 0x7f4bcc0b5140 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.536+0000 7f4bc3fff640 1 -- 172.21.15.46:0/1701932612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8032810 con 0x7f4bcc0b5140 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 -- 172.21.15.46:0/1701932612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0b5140 msgr2=0x7f4bcc0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/1701932612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc0b7760 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80097a0 tx=0x7f4bc802a210 comp rx=0 tx=0).stop 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 -- 172.21.15.46:0/1701932612 shutdown_connections 2024-05-09T03:11:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/1701932612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/1701932612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc0b7760 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 -- 172.21.15.46:0/1701932612 >> 172.21.15.46:0/1701932612 conn(0x7f4bcc08d3f0 msgr2=0x7f4bcc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 -- 172.21.15.46:0/1701932612 shutdown_connections 2024-05-09T03:11:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.537+0000 7f4bd39d6640 1 -- 172.21.15.46:0/1701932612 wait complete. 2024-05-09T03:11:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.539+0000 7f4bd39d6640 1 Processor -- start 2024-05-09T03:11:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.539+0000 7f4bd39d6640 1 -- start start 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd39d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd39d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd39d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 0x7f4bcc159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd39d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc0d4d80 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd39d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4bcc0d4aa0 con 0x7f4bcc0b5140 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd39d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4bcc0d4200 con 0x7f4bcc0b7ca0 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 0x7f4bcc159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 0x7f4bcc159670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40234/0 (socket says 172.21.15.46:40234) 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 -- 172.21.15.46:0/2910392730 learned_addr learned my addr 172.21.15.46:0/2910392730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:53.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd0f71640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1772640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 -- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 msgr2=0x7f4bcc159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 -- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bcc0b5140 msgr2=0x7f4bcc1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc1571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.540+0000 7f4bd1f73640 1 -- 172.21.15.46:0/2910392730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bcc1576f0 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.541+0000 7f4bd1f73640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 0x7f4bcc159670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c760 tx=0x7f4bc400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.541+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4018070 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.541+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc400ce10 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.541+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bcc155950 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.541+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc400a880 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.541+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bcc161880 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.542+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bcc06a270 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.543+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4bc401b070 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.543+0000 7f4bc1ffb640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bb00762d0 0x7f4bb00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.543+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5795+0+0 (secure 0 0 0) 0x7f4bc4096a60 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.544+0000 7f4bd1772640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bb00762d0 0x7f4bb00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.544+0000 7f4bd1772640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bb00762d0 0x7f4bb00787e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8034000 tx=0x7f4bc802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.548+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4bcc0b80b0 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.784+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4bcc0bae60 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.785 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:11:53.785+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:11:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.786+0000 7f4bc1ffb640 1 -- 172.21.15.46:0/2910392730 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f4bc405fd10 con 0x7f4bcc0d3df0 2024-05-09T03:11:53.787 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:11:53.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bb00762d0 msgr2=0x7f4bb00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bb00762d0 0x7f4bb00787e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8034000 tx=0x7f4bc802e000 comp rx=0 tx=0).stop 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 msgr2=0x7f4bcc159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 0x7f4bcc159670 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4bc400c760 tx=0x7f4bc400cc20 comp rx=0 tx=0).stop 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 shutdown_connections 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bb00762d0 0x7f4bb00787e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bcc0d3df0 0x7f4bcc159670 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bcc0b7ca0 0x7f4bcc159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 --2- 172.21.15.46:0/2910392730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bcc0b5140 0x7f4bcc1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 >> 172.21.15.46:0/2910392730 conn(0x7f4bcc08d3f0 msgr2=0x7f4bcc0ba340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 shutdown_connections 2024-05-09T03:11:53.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.789+0000 7f4bd39d6640 1 -- 172.21.15.46:0/2910392730 wait complete. 2024-05-09T03:11:53.792 INFO:tasks.ceph.mds.a.smithi046.stderr: -40> 2024-05-09T03:11:53.208+0000 7febacc12640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:11:53.802 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:11:53.848 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:11:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.922+0000 7f3794661640 1 Processor -- start 2024-05-09T03:11:53.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.922+0000 7f3794661640 1 -- start start 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.923+0000 7f3794661640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f378c0d00f0 0x7f378c09cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.923+0000 7f3794661640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f378c0d1740 0x7f378c0ce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.923+0000 7f3794661640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378c0b2d30 con 0x7f378c0b9260 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.923+0000 7f3794661640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f378c0d2a20 con 0x7f378c0d1740 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.923+0000 7f3794661640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f378c0b2af0 con 0x7f378c0d00f0 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3792bfe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f378c0d1740 0x7f378c0ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3791bfc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f378c0b9260 0x7f378c0d2310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60056/0 (socket says 172.21.15.46:60056) 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3791bfc640 1 -- 172.21.15.46:0/2703918028 learned_addr learned my addr 172.21.15.46:0/2703918028 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f37923fd640 1 --2- 172.21.15.46:0/2703918028 >> v2:172.21.15.88:3300/0 conn(0x7f378c0d00f0 0x7f378c09cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3792bfe640 1 -- 172.21.15.46:0/2703918028 >> v2:172.21.15.88:3300/0 conn(0x7f378c0d00f0 msgr2=0x7f378c09cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3792bfe640 1 --2- 172.21.15.46:0/2703918028 >> v2:172.21.15.88:3300/0 conn(0x7f378c0d00f0 0x7f378c09cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3792bfe640 1 -- 172.21.15.46:0/2703918028 >> v1:172.21.15.46:6789/0 conn(0x7f378c0b9260 legacy=0x7f378c0d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f37913fb640 1 -- 172.21.15.46:0/2703918028 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2869223321 0 0) 0x7f378c0b2d30 con 0x7f378c0b9260 2024-05-09T03:11:53.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f3792bfe640 1 -- 172.21.15.46:0/2703918028 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c0d1360 con 0x7f378c0d1740 2024-05-09T03:11:53.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.924+0000 7f37923fd640 1 --2- 172.21.15.46:0/2703918028 >> v2:172.21.15.88:3300/0 conn(0x7f378c0d00f0 0x7f378c09cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.925+0000 7f3792bfe640 1 --2- 172.21.15.46:0/2703918028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c0ce780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3784009d90 tx=0x7f378402a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb0ee0454f38712 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.925+0000 7f37913fb640 1 -- 172.21.15.46:0/2703918028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784007740 con 0x7f378c0d1740 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.925+0000 7f37913fb640 1 -- 172.21.15.46:0/2703918028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37840078e0 con 0x7f378c0d1740 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.925+0000 7f37913fb640 1 -- 172.21.15.46:0/2703918028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784007cb0 con 0x7f378c0d1740 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 -- 172.21.15.46:0/2703918028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 msgr2=0x7f378c0ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 --2- 172.21.15.46:0/2703918028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c0ce780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f3784009d90 tx=0x7f378402a580 comp rx=0 tx=0).stop 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 -- 172.21.15.46:0/2703918028 shutdown_connections 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 --2- 172.21.15.46:0/2703918028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c0ce780 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 --2- 172.21.15.46:0/2703918028 >> v2:172.21.15.88:3300/0 conn(0x7f378c0d00f0 0x7f378c09cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 -- 172.21.15.46:0/2703918028 >> 172.21.15.46:0/2703918028 conn(0x7f378c0d05f0 msgr2=0x7f378c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 -- 172.21.15.46:0/2703918028 shutdown_connections 2024-05-09T03:11:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.926+0000 7f3794661640 1 -- 172.21.15.46:0/2703918028 wait complete. 2024-05-09T03:11:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.927+0000 7f3794661640 1 Processor -- start 2024-05-09T03:11:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.928+0000 7f3794661640 1 -- start start 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3794661640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f378c0b9260 0x7f378c0c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3794661640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f378c0d00f0 0x7f378c0cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3794661640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3794661640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f378c074540 con 0x7f378c0d1740 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3794661640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f378c0d1110 con 0x7f378c0d00f0 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3794661640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f378c0b8c70 con 0x7f378c0b9260 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40282/0 (socket says 172.21.15.46:40282) 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 -- 172.21.15.46:0/3154831532 learned_addr learned my addr 172.21.15.46:0/3154831532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f37923fd640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f378c0b9260 0x7f378c0c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3791bfc640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f378c0d00f0 0x7f378c0cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 -- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f378c0b9260 msgr2=0x7f378c0c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f378c0b9260 0x7f378c0c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 -- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f378c0d00f0 msgr2=0x7f378c0cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f378c0d00f0 0x7f378c0cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3792bfe640 1 -- 172.21.15.46:0/3154831532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c0cd2a0 con 0x7f378c0d1740 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.929+0000 7f3791bfc640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f378c0d00f0 0x7f378c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.930+0000 7f3792bfe640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c071d00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f378402a5c0 tx=0x7f3784004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.930+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378403f070 con 0x7f378c0d1740 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.930+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3784031bc0 con 0x7f378c0d1740 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.930+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378c0cd580 con 0x7f378c0d1740 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.930+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3784009450 con 0x7f378c0d1740 2024-05-09T03:11:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.930+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378c0ccd20 con 0x7f378c0d1740 2024-05-09T03:11:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.931+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378c0cdaa0 con 0x7f378c0d1740 2024-05-09T03:11:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.932+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3784003990 con 0x7f378c0d1740 2024-05-09T03:11:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.932+0000 7f3782ffd640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3770076300 0x7f3770078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.932+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5881+0+0 (secure 0 0 0) 0x7f37840b8880 con 0x7f378c0d1740 2024-05-09T03:11:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.932+0000 7f37923fd640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3770076300 0x7f3770078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.933+0000 7f37923fd640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3770076300 0x7f3770078810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f378c051250 tx=0x7f3788008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:53.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:53.937+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f378c0cdaa0 con 0x7f378c0d1740 2024-05-09T03:11:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.167+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f378c06a270 con 0x7f378c0d1740 2024-05-09T03:11:54.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.788+0000 7f3782ffd640 1 -- 172.21.15.46:0/3154831532 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7f37840820a0 con 0x7f378c0d1740 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3770076300 msgr2=0x7f3770078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3770076300 0x7f3770078810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f378c051250 tx=0x7f3788008040 comp rx=0 tx=0).stop 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 msgr2=0x7f378c071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c071d00 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f378402a5c0 tx=0x7f3784004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 shutdown_connections 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3770076300 0x7f3770078810 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f378c0d1740 0x7f378c071d00 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f378c0d00f0 0x7f378c0cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 --2- 172.21.15.46:0/3154831532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f378c0b9260 0x7f378c0c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 >> 172.21.15.46:0/3154831532 conn(0x7f378c0d05f0 msgr2=0x7f378c0b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:54.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 shutdown_connections 2024-05-09T03:11:54.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.791+0000 7f3794661640 1 -- 172.21.15.46:0/3154831532 wait complete. 2024-05-09T03:11:54.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:11:54.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.931+0000 7f30762d9640 1 Processor -- start 2024-05-09T03:11:54.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.931+0000 7f30762d9640 1 -- start start 2024-05-09T03:11:54.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f30762d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f30700d16b0 0x7f30700d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f30762d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30700b2da0 0x7f3070070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f30762d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3070070770 0x7f30700ce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f30762d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3070070b80 con 0x7f30700b2da0 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f30762d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f30700d2db0 con 0x7f30700d16b0 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f30762d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f30700d3090 con 0x7f3070070770 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f306f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30700b2da0 0x7f3070070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.932+0000 7f306f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30700b2da0 0x7f3070070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40290/0 (socket says 172.21.15.46:40290) 2024-05-09T03:11:54.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 -- 172.21.15.46:0/3356010236 learned_addr learned my addr 172.21.15.46:0/3356010236 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306ffff640 1 --2- 172.21.15.46:0/3356010236 >> v2:172.21.15.88:3300/0 conn(0x7f30700d16b0 0x7f30700d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f3074876640 1 --2- 172.21.15.46:0/3356010236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3070070770 0x7f30700ce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 -- 172.21.15.46:0/3356010236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3070070770 msgr2=0x7f30700ce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 --2- 172.21.15.46:0/3356010236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3070070770 0x7f30700ce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 -- 172.21.15.46:0/3356010236 >> v2:172.21.15.88:3300/0 conn(0x7f30700d16b0 msgr2=0x7f30700d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 --2- 172.21.15.46:0/3356010236 >> v2:172.21.15.88:3300/0 conn(0x7f30700d16b0 0x7f30700d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 -- 172.21.15.46:0/3356010236 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30700d3370 con 0x7f30700b2da0 2024-05-09T03:11:54.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.933+0000 7f306f7fe640 1 --2- 172.21.15.46:0/3356010236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30700b2da0 0x7f3070070230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3070064360 tx=0x7f306402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bb21bd365f856c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:54.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f306effd640 1 -- 172.21.15.46:0/3356010236 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064037070 con 0x7f30700b2da0 2024-05-09T03:11:54.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f306effd640 1 -- 172.21.15.46:0/3356010236 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3064032440 con 0x7f30700b2da0 2024-05-09T03:11:54.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f306effd640 1 -- 172.21.15.46:0/3356010236 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3064032810 con 0x7f30700b2da0 2024-05-09T03:11:54.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 -- 172.21.15.46:0/3356010236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30700b2da0 msgr2=0x7f3070070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:54.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 --2- 172.21.15.46:0/3356010236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30700b2da0 0x7f3070070230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3070064360 tx=0x7f306402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 -- 172.21.15.46:0/3356010236 shutdown_connections 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 --2- 172.21.15.46:0/3356010236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3070070770 0x7f30700ce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 --2- 172.21.15.46:0/3356010236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30700b2da0 0x7f3070070230 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 --2- 172.21.15.46:0/3356010236 >> v2:172.21.15.88:3300/0 conn(0x7f30700d16b0 0x7f30700d2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 -- 172.21.15.46:0/3356010236 >> 172.21.15.46:0/3356010236 conn(0x7f30700d3ae0 msgr2=0x7f30700d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.934+0000 7f30762d9640 1 -- 172.21.15.46:0/3356010236 shutdown_connections 2024-05-09T03:11:54.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.935+0000 7f30762d9640 1 -- 172.21.15.46:0/3356010236 wait complete. 2024-05-09T03:11:54.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.936+0000 7f30762d9640 1 Processor -- start 2024-05-09T03:11:54.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.937+0000 7f30762d9640 1 -- start start 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f30762d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 0x7f3070149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f30762d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30700b2da0 0x7f30700ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f30762d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30700d16b0 0x7f30700b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f30762d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f30700b6ed0 con 0x7f3070070770 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f30762d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f30700d2ad0 con 0x7f30700d16b0 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f30762d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f30700ceba0 con 0x7f30700b2da0 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 0x7f3070149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 0x7f3070149060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40306/0 (socket says 172.21.15.46:40306) 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 -- 172.21.15.46:0/363297314 learned_addr learned my addr 172.21.15.46:0/363297314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306f7fe640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30700b2da0 0x7f30700ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f3074876640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30700d16b0 0x7f30700b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 -- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30700b2da0 msgr2=0x7f30700ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30700b2da0 0x7f30700ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 -- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30700d16b0 msgr2=0x7f30700b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30700d16b0 0x7f30700b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 -- 172.21.15.46:0/363297314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30700ca980 con 0x7f3070070770 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306f7fe640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30700b2da0 0x7f30700ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f3074876640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30700d16b0 0x7f30700b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.938+0000 7f306ffff640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 0x7f3070149060 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f306000d5f0 tx=0x7f306000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.939+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30600125f0 con 0x7f3070070770 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.939+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3060019070 con 0x7f3070070770 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.939+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30600129d0 con 0x7f3070070770 2024-05-09T03:11:54.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.939+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30700cb320 con 0x7f3070070770 2024-05-09T03:11:54.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.939+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30700cb8e0 con 0x7f3070070770 2024-05-09T03:11:54.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.940+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f307006a270 con 0x7f3070070770 2024-05-09T03:11:54.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.941+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3060012b70 con 0x7f3070070770 2024-05-09T03:11:54.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.941+0000 7f306cff9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f305c0762d0 0x7f305c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:54.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.941+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5881+0+0 (secure 0 0 0) 0x7f3060097ab0 con 0x7f3070070770 2024-05-09T03:11:54.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.941+0000 7f306f7fe640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f305c0762d0 0x7f305c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:54.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.942+0000 7f306f7fe640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f305c0762d0 0x7f305c0787e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3064034000 tx=0x7f306402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:54.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:54.946+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f30700ca980 con 0x7f3070070770 2024-05-09T03:11:55.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.139+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f307015ea00 con 0x7f3070070770 2024-05-09T03:11:55.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.790+0000 7f306cff9640 1 -- 172.21.15.46:0/363297314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) ==== 181+0+0 (secure 0 0 0) 0x7f3060061360 con 0x7f3070070770 2024-05-09T03:11:55.791 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:11:55.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f305c0762d0 msgr2=0x7f305c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f305c0762d0 0x7f305c0787e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3064034000 tx=0x7f306402e000 comp rx=0 tx=0).stop 2024-05-09T03:11:55.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 msgr2=0x7f3070149060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 0x7f3070149060 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f306000d5f0 tx=0x7f306000dab0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 shutdown_connections 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30700d16b0 0x7f30700b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f30700b2da0 0x7f30700ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f305c0762d0 0x7f305c0787e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 --2- 172.21.15.46:0/363297314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3070070770 0x7f3070149060 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.794+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 >> 172.21.15.46:0/363297314 conn(0x7f30700d3ae0 msgr2=0x7f30700d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.795+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 shutdown_connections 2024-05-09T03:11:55.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.795+0000 7f30762d9640 1 -- 172.21.15.46:0/363297314 wait complete. 2024-05-09T03:11:55.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:11:55.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.936+0000 7ff2891e3640 1 Processor -- start 2024-05-09T03:11:55.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.936+0000 7ff2891e3640 1 -- start start 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff2891e3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff28406e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff2891e3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2840b9420 con 0x7ff28406ecf0 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff2891e3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2840d1300 con 0x7ff284164f20 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff2891e3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff2840cfbc0 con 0x7ff28406e3a0 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff27a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff28406ecf0 0x7ff284164810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40240/0 (socket says 172.21.15.46:40240) 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff27a59c640 1 -- 172.21.15.46:0/2645757366 learned_addr learned my addr 172.21.15.46:0/2645757366 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.937+0000 7ff282d9d640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff28406e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255126380 0 0) 0x7ff2840b9420 con 0x7ff28406ecf0 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff270003520 con 0x7ff28406ecf0 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451011697 0 0) 0x7ff2840d1300 con 0x7ff284164f20 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2840b9420 con 0x7ff284164f20 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1656205972 0 0) 0x7ff270003520 con 0x7ff28406ecf0 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 msgr2=0x7ff28406e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff28406e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 >> v1:172.21.15.88:6789/0 conn(0x7ff284164f20 legacy=0x7ff2840d1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28409d250 con 0x7ff28406ecf0 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ff26c0016f0 con 0x7ff28406ecf0 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff282d9d640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff28406e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 >> v1:172.21.15.46:6789/0 conn(0x7ff28406ecf0 legacy=0x7ff284164810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff270003c10 0x7ff270004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff270004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff270004d40 0x7ff270005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2840cfbc0 con 0x7ff270004d40 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff27000bc10 con 0x7ff270003c10 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff27000be10 con 0x7ff28406e3a0 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28359e640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff270004d40 0x7ff270005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff282d9d640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff270003c10 0x7ff270004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff27a59c640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff270004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28359e640 1 -- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 msgr2=0x7ff270004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28359e640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff28406e3a0 0x7ff270004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28359e640 1 -- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff270003c10 msgr2=0x7ff270004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28359e640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff270003c10 0x7ff270004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.938+0000 7ff28359e640 1 -- 172.21.15.46:0/2645757366 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff270003520 con 0x7ff270004d40 2024-05-09T03:11:55.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.939+0000 7ff28359e640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff270004d40 0x7ff270005210 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff284074540 tx=0x7ff270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abad16ceb2b7007a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:55.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.939+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c001c20 con 0x7ff270004d40 2024-05-09T03:11:55.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.939+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff27c006d70 con 0x7ff270004d40 2024-05-09T03:11:55.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.939+0000 7ff28259c640 1 -- 172.21.15.46:0/2645757366 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c003bb0 con 0x7ff270004d40 2024-05-09T03:11:55.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff2891e3640 1 -- 172.21.15.46:0/2645757366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff270004d40 msgr2=0x7ff270005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff2891e3640 1 --2- 172.21.15.46:0/2645757366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff270004d40 0x7ff270005210 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff284074540 tx=0x7ff270005830 comp rx=0 tx=0).stop 2024-05-09T03:11:55.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff28359e640 1 -- 172.21.15.46:0/2645757366 reap_dead start 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff2891e3640 1 -- 172.21.15.46:0/2645757366 shutdown_connections 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff2891e3640 1 -- 172.21.15.46:0/2645757366 >> 172.21.15.46:0/2645757366 conn(0x7ff2840d05d0 msgr2=0x7ff2840d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff2891e3640 1 -- 172.21.15.46:0/2645757366 shutdown_connections 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.940+0000 7ff2891e3640 1 -- 172.21.15.46:0/2645757366 wait complete. 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.942+0000 7ff2891e3640 1 Processor -- start 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.942+0000 7ff2891e3640 1 -- start start 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff2891e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2840a14a0 0x7ff2841f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff2891e3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2840c95d0 0x7ff2841fa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff2891e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 0x7ff284162b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff2891e3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff28406ddf0 con 0x7ff2841fa6d0 2024-05-09T03:11:55.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff2891e3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff28406dc70 con 0x7ff2840a14a0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff2891e3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff284165330 con 0x7ff2840c95d0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 0x7ff284162b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 0x7ff284162b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47694/0 (socket says 172.21.15.46:47694) 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 -- 172.21.15.46:0/1400658879 learned_addr learned my addr 172.21.15.46:0/1400658879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff27a59c640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2840c95d0 0x7ff2841fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff282d9d640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2840a14a0 0x7ff2841f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 -- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2840c95d0 msgr2=0x7ff2841fa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2840c95d0 0x7ff2841fa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 -- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2840a14a0 msgr2=0x7ff2841f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2840a14a0 0x7ff2841f6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 -- 172.21.15.46:0/1400658879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2841f99f0 con 0x7ff2841fa6d0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff28359e640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 0x7ff284162b20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0020c0 tx=0x7ff27c001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.944+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c001c20 con 0x7ff2841fa6d0 2024-05-09T03:11:55.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.945+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff27c041070 con 0x7ff2841fa6d0 2024-05-09T03:11:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.945+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2841f6940 con 0x7ff2841fa6d0 2024-05-09T03:11:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.945+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c007d90 con 0x7ff2841fa6d0 2024-05-09T03:11:55.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.945+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2841f6ef0 con 0x7ff2841fa6d0 2024-05-09T03:11:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.946+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28406a270 con 0x7ff2841fa6d0 2024-05-09T03:11:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.946+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff27c039070 con 0x7ff2841fa6d0 2024-05-09T03:11:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.947+0000 7ff27bfff640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2640762d0 0x7ff2640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.947+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5443+0+0 (secure 0 0 0) 0x7ff27c0b70b0 con 0x7ff2841fa6d0 2024-05-09T03:11:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.947+0000 7ff282d9d640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2640762d0 0x7ff2640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:55.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.948+0000 7ff282d9d640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2640762d0 0x7ff2640787e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2840cdb60 tx=0x7ff274008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:55.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:55.952+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff2841f99f0 con 0x7ff2841fa6d0 2024-05-09T03:11:56.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.144+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff28406e8e0 con 0x7ff2841fa6d0 2024-05-09T03:11:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.795+0000 7ff27bfff640 1 -- 172.21.15.46:0/1400658879 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) ==== 169+0+0 (secure 0 0 0) 0x7ff27c080a80 con 0x7ff2841fa6d0 2024-05-09T03:11:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:11:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2640762d0 msgr2=0x7ff2640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2640762d0 0x7ff2640787e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2840cdb60 tx=0x7ff274008040 comp rx=0 tx=0).stop 2024-05-09T03:11:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 msgr2=0x7ff284162b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 0x7ff284162b20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0020c0 tx=0x7ff27c001670 comp rx=0 tx=0).stop 2024-05-09T03:11:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 shutdown_connections 2024-05-09T03:11:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2640762d0 0x7ff2640787e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2841fa6d0 0x7ff284162b20 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2840c95d0 0x7ff2841fa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 --2- 172.21.15.46:0/1400658879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2840a14a0 0x7ff2841f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 >> 172.21.15.46:0/1400658879 conn(0x7ff2840d05d0 msgr2=0x7ff2840b7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.799+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 shutdown_connections 2024-05-09T03:11:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.800+0000 7ff2891e3640 1 -- 172.21.15.46:0/1400658879 wait complete. 2024-05-09T03:11:56.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-05-09T03:11:56.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.939+0000 7fa4d15f3640 1 Processor -- start 2024-05-09T03:11:56.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.940+0000 7fa4d15f3640 1 -- start start 2024-05-09T03:11:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4d15f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4cc0b7e70 0x7fa4cc0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4d15f3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa4cc160520 0x7fa4cc0b9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4d15f3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4cc0d5ad0 con 0x7fa4cc0b4fc0 2024-05-09T03:11:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4d15f3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa4cc0d47a0 con 0x7fa4cc0b7e70 2024-05-09T03:11:56.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4d15f3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa4cc0d39e0 con 0x7fa4cc160520 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4cc0b7e70 0x7fa4cc0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4cc0b7e70 0x7fa4cc0b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47696/0 (socket says 172.21.15.46:47696) 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 -- 172.21.15.46:0/1084216668 learned_addr learned my addr 172.21.15.46:0/1084216668 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4cb7fe640 1 --2- 172.21.15.46:0/1084216668 >> v2:172.21.15.88:3300/0 conn(0x7fa4cc160520 0x7fa4cc0b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 -- 172.21.15.46:0/1084216668 >> v2:172.21.15.88:3300/0 conn(0x7fa4cc160520 msgr2=0x7fa4cc0b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 --2- 172.21.15.46:0/1084216668 >> v2:172.21.15.88:3300/0 conn(0x7fa4cc160520 0x7fa4cc0b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 -- 172.21.15.46:0/1084216668 >> v1:172.21.15.88:6789/0 conn(0x7fa4cc0b4fc0 legacy=0x7fa4cc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.941+0000 7fa4ca7fc640 1 -- 172.21.15.46:0/1084216668 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4cc0d3cc0 con 0x7fa4cc0b7e70 2024-05-09T03:11:56.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.942+0000 7fa4ca7fc640 1 --2- 172.21.15.46:0/1084216668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc0b7e70 0x7fa4cc0b4a00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc0b4980 tx=0x7fa4b402a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69882407abcbf529 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:56.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.942+0000 7fa4c9ffb640 1 -- 172.21.15.46:0/1084216668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4037070 con 0x7fa4cc0b7e70 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.942+0000 7fa4c9ffb640 1 -- 172.21.15.46:0/1084216668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b4032440 con 0x7fa4cc0b7e70 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.942+0000 7fa4c9ffb640 1 -- 172.21.15.46:0/1084216668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4032810 con 0x7fa4cc0b7e70 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 -- 172.21.15.46:0/1084216668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc0b7e70 msgr2=0x7fa4cc0b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/1084216668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc0b7e70 0x7fa4cc0b4a00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc0b4980 tx=0x7fa4b402a150 comp rx=0 tx=0).stop 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 -- 172.21.15.46:0/1084216668 shutdown_connections 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/1084216668 >> v2:172.21.15.88:3300/0 conn(0x7fa4cc160520 0x7fa4cc0b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/1084216668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc0b7e70 0x7fa4cc0b4a00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 -- 172.21.15.46:0/1084216668 >> 172.21.15.46:0/1084216668 conn(0x7fa4cc08d3f0 msgr2=0x7fa4cc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 -- 172.21.15.46:0/1084216668 shutdown_connections 2024-05-09T03:11:56.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.943+0000 7fa4d15f3640 1 -- 172.21.15.46:0/1084216668 wait complete. 2024-05-09T03:11:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.945+0000 7fa4d15f3640 1 Processor -- start 2024-05-09T03:11:56.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.945+0000 7fa4d15f3640 1 -- start start 2024-05-09T03:11:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.946+0000 7fa4d15f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4cc0b4fc0 0x7fa4cc147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.946+0000 7fa4d15f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4cc160520 0x7fa4cc1e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:56.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.946+0000 7fa4d15f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 0x7fa4cc159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.946+0000 7fa4d15f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4cc160930 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.946+0000 7fa4d15f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa4cc0d4280 con 0x7fa4cc160520 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.946+0000 7fa4d15f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa4cc0d5750 con 0x7fa4cc0b4fc0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 0x7fa4cc159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 0x7fa4cc159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47700/0 (socket says 172.21.15.46:47700) 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 -- 172.21.15.46:0/2619357629 learned_addr learned my addr 172.21.15.46:0/2619357629 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4ca7fc640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4cc160520 0x7fa4cc1e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4caffd640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4cc0b4fc0 0x7fa4cc147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 -- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4cc0b4fc0 msgr2=0x7fa4cc147200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4cc0b4fc0 0x7fa4cc147200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 -- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4cc160520 msgr2=0x7fa4cc1e7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4cc160520 0x7fa4cc1e7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 -- 172.21.15.46:0/2619357629 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4cc1e88c0 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4cb7fe640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 0x7fa4cc159cc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d600 tx=0x7fa4c400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40125f0 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c4019070 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40129f0 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4cc1e8ba0 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.947+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4cc1e9160 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.948+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4cc06a270 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.949+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa4c4012b90 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.950+0000 7fa4c37fe640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa4ac076300 0x7fa4ac078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.950+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7fa4c4097650 con 0x7fa4cc1e80d0 2024-05-09T03:11:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.950+0000 7fa4caffd640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa4ac076300 0x7fa4ac078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:56.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.950+0000 7fa4caffd640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa4ac076300 0x7fa4ac078810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc0bbe50 tx=0x7fa4bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:56.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:56.954+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa4c4060b80 con 0x7fa4cc1e80d0 2024-05-09T03:11:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.149+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7fa4cc1e8650 con 0x7fa4cc1e80d0 2024-05-09T03:11:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.149+0000 7fa4c37fe640 1 -- 172.21.15.46:0/2619357629 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7fa4c4064e80 con 0x7fa4cc1e80d0 2024-05-09T03:11:57.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.151+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa4ac076300 msgr2=0x7fa4ac078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.151+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa4ac076300 0x7fa4ac078810 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc0bbe50 tx=0x7fa4bc009040 comp rx=0 tx=0).stop 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 msgr2=0x7fa4cc159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 0x7fa4cc159cc0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d600 tx=0x7fa4c400dac0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 shutdown_connections 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa4ac076300 0x7fa4ac078810 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc1e80d0 0x7fa4cc159cc0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4cc160520 0x7fa4cc1e7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 --2- 172.21.15.46:0/2619357629 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4cc0b4fc0 0x7fa4cc147200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 >> 172.21.15.46:0/2619357629 conn(0x7fa4cc08d3f0 msgr2=0x7fa4cc15dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 shutdown_connections 2024-05-09T03:11:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.152+0000 7fa4d15f3640 1 -- 172.21.15.46:0/2619357629 wait complete. 2024-05-09T03:11:57.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_abort_on_newly_corrupt_dentry 2024-05-09T03:11:57.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.279+0000 7f29dd9ee640 1 Processor -- start 2024-05-09T03:11:57.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.279+0000 7f29dd9ee640 1 -- start start 2024-05-09T03:11:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29dd9ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f29d8164bf0 0x7f29d80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29dd9ee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29d80d17a0 con 0x7f29d80b7e50 2024-05-09T03:11:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29dd9ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29d80d14c0 con 0x7f29d80d0fb0 2024-05-09T03:11:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29dd9ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f29d8164760 con 0x7f29d8164bf0 2024-05-09T03:11:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f29d80b7e50 0x7f29d80b9190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40254/0 (socket says 172.21.15.46:40254) 2024-05-09T03:11:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d77fe640 1 -- 172.21.15.46:0/996330444 learned_addr learned my addr 172.21.15.46:0/996330444 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/996330444 >> v2:172.21.15.88:3300/0 conn(0x7f29d8164bf0 0x7f29d80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096750907 0 0) 0x7f29d80d17a0 con 0x7f29d80b7e50 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c0003520 con 0x7f29d80b7e50 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1636494784 0 0) 0x7f29c0003520 con 0x7f29d80b7e50 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 >> v2:172.21.15.88:3300/0 conn(0x7f29d8164bf0 msgr2=0x7f29d80d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 --2- 172.21.15.46:0/996330444 >> v2:172.21.15.88:3300/0 conn(0x7f29d8164bf0 0x7f29d80d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 >> v1:172.21.15.88:6789/0 conn(0x7f29d80d0fb0 legacy=0x7f29d80b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d8164110 con 0x7f29d80b7e50 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f29d00016f0 con 0x7f29d80b7e50 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.281+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 >> v1:172.21.15.46:6789/0 conn(0x7f29d80b7e50 legacy=0x7f29d80b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d5ffb640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c0003a50 0x7f29c0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d5ffb640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c0004800 0x7f29c0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d5ffb640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0005250 0x7f29c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f29d8164760 con 0x7f29c0003a50 2024-05-09T03:11:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f29c000c1e0 con 0x7f29c0005250 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f29c000c3e0 con 0x7f29c0004800 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c0003a50 0x7f29c0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d67fc640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c0004800 0x7f29c0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d77fe640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0005250 0x7f29c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d6ffd640 1 -- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c0004800 msgr2=0x7f29c0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c0004800 0x7f29c0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d6ffd640 1 -- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0005250 msgr2=0x7f29c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0005250 0x7f29c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d6ffd640 1 -- 172.21.15.46:0/996330444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c0003520 con 0x7f29c0003a50 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d77fe640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0005250 0x7f29c000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.282+0000 7f29d77fe640 1 -- 172.21.15.46:0/996330444 reap_dead start 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.283+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c0003a50 0x7f29c0003ec0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009fc0 tx=0x7f29c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53438baca201015e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.283+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8009c10 con 0x7f29c0003a50 2024-05-09T03:11:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.283+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c8036070 con 0x7f29c0003a50 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.283+0000 7f29d5ffb640 1 -- 172.21.15.46:0/996330444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8031880 con 0x7f29c0003a50 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 -- 172.21.15.46:0/996330444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c0003a50 msgr2=0x7f29c0003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c0003a50 0x7f29c0003ec0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009fc0 tx=0x7f29c0005830 comp rx=0 tx=0).stop 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 -- 172.21.15.46:0/996330444 shutdown_connections 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/996330444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c0003a50 0x7f29c0003ec0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 -- 172.21.15.46:0/996330444 >> 172.21.15.46:0/996330444 conn(0x7f29d80d2920 msgr2=0x7f29d80d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 -- 172.21.15.46:0/996330444 shutdown_connections 2024-05-09T03:11:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.284+0000 7f29dd9ee640 1 -- 172.21.15.46:0/996330444 wait complete. 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.285+0000 7f29dd9ee640 1 Processor -- start 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.286+0000 7f29dd9ee640 1 -- start start 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29dd9ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0003a50 0x7f29d814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29dd9ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 0x7f29d81599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29dd9ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d8159f10 0x7f29d815ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29dd9ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f29d806ecb0 con 0x7f29d80ca840 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29dd9ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f29d806eb30 con 0x7f29c0003a50 2024-05-09T03:11:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29dd9ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f29d80b52b0 con 0x7f29d8159f10 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 0x7f29d81599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 0x7f29d81599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47714/0 (socket says 172.21.15.46:47714) 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 -- 172.21.15.46:0/801685500 learned_addr learned my addr 172.21.15.46:0/801685500 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d77fe640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d8159f10 0x7f29d815ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0003a50 0x7f29d814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 -- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d8159f10 msgr2=0x7f29d815ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d8159f10 0x7f29d815ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 -- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0003a50 msgr2=0x7f29d814f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0003a50 0x7f29d814f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d67fc640 1 -- 172.21.15.46:0/801685500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d815a320 con 0x7f29d80ca840 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.287+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0003a50 0x7f29d814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:11:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.288+0000 7f29d67fc640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 0x7f29d81599d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00f820 tx=0x7f29cc00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.288+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0052e0 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.288+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29cc018070 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.288+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d81f61f0 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.288+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc00d950 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.288+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d80cc000 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.289+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d806a270 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.290+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f29cc00daf0 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.290+0000 7f29bffff640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29c4076300 0x7f29c4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.290+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7f29cc0971f0 con 0x7f29d80ca840 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.290+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29c4076300 0x7f29c4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.291+0000 7f29d6ffd640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29c4076300 0x7f29c4078810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f29c8035000 tx=0x7f29c802f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.295+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f29d815a320 con 0x7f29d80ca840 2024-05-09T03:11:57.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.484+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_abort_on_newly_corrupt_dentry"} v 0) -- 0x7f29d80d40c0 con 0x7f29d80ca840 2024-05-09T03:11:57.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.488+0000 7f29bffff640 1 -- 172.21.15.46:0/801685500 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_abort_on_newly_corrupt_dentry"}]=0 v16) ==== 116+0+0 (secure 0 0 0) 0x7f29cc067e20 con 0x7f29d80ca840 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29c4076300 msgr2=0x7f29c4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29c4076300 0x7f29c4078810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f29c8035000 tx=0x7f29c802f000 comp rx=0 tx=0).stop 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 msgr2=0x7f29d81599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 0x7f29d81599d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00f820 tx=0x7f29cc00fce0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 shutdown_connections 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29d8159f10 0x7f29d815ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29c4076300 0x7f29c4078810 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29d80ca840 0x7f29d81599d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 --2- 172.21.15.46:0/801685500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c0003a50 0x7f29d814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 >> 172.21.15.46:0/801685500 conn(0x7f29d80d2920 msgr2=0x7f29d80d4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 shutdown_connections 2024-05-09T03:11:57.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.490+0000 7f29dd9ee640 1 -- 172.21.15.46:0/801685500 wait complete. 2024-05-09T03:11:57.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T03:11:57.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.618+0000 7ff67c849640 1 Processor -- start 2024-05-09T03:11:57.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.618+0000 7ff67c849640 1 -- start start 2024-05-09T03:11:57.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.619+0000 7ff67c849640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff678070420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.619+0000 7ff67c849640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff678070960 0x7ff678160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.619+0000 7ff67c849640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff678070d70 con 0x7ff6781610b0 2024-05-09T03:11:57.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.619+0000 7ff67c849640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff678073650 con 0x7ff678070960 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.619+0000 7ff67c849640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff678073930 con 0x7ff67815dfb0 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff66ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff678070960 0x7ff678160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff676d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff6781610b0 0x7ff6780d0680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40262/0 (socket says 172.21.15.46:40262) 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff676d9d640 1 -- 172.21.15.46:0/1974540627 learned_addr learned my addr 172.21.15.46:0/1974540627 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff67659c640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff678070420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff66ffff640 1 -- 172.21.15.46:0/1974540627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 msgr2=0x7ff678070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff66ffff640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff678070420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff66ffff640 1 -- 172.21.15.46:0/1974540627 >> v1:172.21.15.46:6789/0 conn(0x7ff6781610b0 legacy=0x7ff6780d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff66ffff640 1 -- 172.21.15.46:0/1974540627 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff678073c10 con 0x7ff678070960 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff675d9b640 1 -- 172.21.15.46:0/1974540627 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614260814 0 0) 0x7ff670001490 con 0x7ff6781610b0 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.620+0000 7ff67659c640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff678070420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.621+0000 7ff66ffff640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff678070960 0x7ff678160b70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff6600097a0 tx=0x7ff66002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff4d98648187fe0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.621+0000 7ff675d9b640 1 -- 172.21.15.46:0/1974540627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660037070 con 0x7ff678070960 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.621+0000 7ff675d9b640 1 -- 172.21.15.46:0/1974540627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff660032440 con 0x7ff678070960 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.621+0000 7ff675d9b640 1 -- 172.21.15.46:0/1974540627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660032810 con 0x7ff678070960 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 -- 172.21.15.46:0/1974540627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff678070960 msgr2=0x7ff678160b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff678070960 0x7ff678160b70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff6600097a0 tx=0x7ff66002a180 comp rx=0 tx=0).stop 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 -- 172.21.15.46:0/1974540627 shutdown_connections 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff678070960 0x7ff678160b70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 --2- 172.21.15.46:0/1974540627 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff678070420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 -- 172.21.15.46:0/1974540627 >> 172.21.15.46:0/1974540627 conn(0x7ff6780b3690 msgr2=0x7ff6780b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 -- 172.21.15.46:0/1974540627 shutdown_connections 2024-05-09T03:11:57.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.622+0000 7ff67c849640 1 -- 172.21.15.46:0/1974540627 wait complete. 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.624+0000 7ff67c849640 1 Processor -- start 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.624+0000 7ff67c849640 1 -- start start 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67c849640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff678070960 0x7ff67814ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67c849640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff6781f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67c849640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 0x7ff6780aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67c849640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6780732b0 con 0x7ff6781610b0 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67c849640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff6780ce820 con 0x7ff678070960 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67c849640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff678073430 con 0x7ff67815dfb0 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 0x7ff6780aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 0x7ff6780aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47742/0 (socket says 172.21.15.46:47742) 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 -- 172.21.15.46:0/729842469 learned_addr learned my addr 172.21.15.46:0/729842469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:57.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff66ffff640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff6781f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff67659c640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff678070960 0x7ff67814ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 -- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 msgr2=0x7ff6781f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff6781f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 -- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff678070960 msgr2=0x7ff67814ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff678070960 0x7ff67814ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.625+0000 7ff676d9d640 1 -- 172.21.15.46:0/729842469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff67814f690 con 0x7ff6781610b0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.626+0000 7ff676d9d640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 0x7ff6780aa230 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c6b0 tx=0x7ff67000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.626+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff670018070 con 0x7ff6781610b0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.626+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff670004da0 con 0x7ff6781610b0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.626+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67000a820 con 0x7ff6781610b0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.626+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67814f970 con 0x7ff6781610b0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.626+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6780c9480 con 0x7ff6781610b0 2024-05-09T03:11:57.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.627+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff67814f400 con 0x7ff6781610b0 2024-05-09T03:11:57.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.628+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff67000a9c0 con 0x7ff6781610b0 2024-05-09T03:11:57.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.628+0000 7ff66effd640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6580762d0 0x7ff6580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.628+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7ff670096810 con 0x7ff6781610b0 2024-05-09T03:11:57.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.629+0000 7ff67659c640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6580762d0 0x7ff6580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.629+0000 7ff67659c640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6580762d0 0x7ff6580787e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff668006fd0 tx=0x7ff668008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.633+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff678070d70 con 0x7ff6781610b0 2024-05-09T03:11:57.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.823+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7ff6780b3c40 con 0x7ff6781610b0 2024-05-09T03:11:57.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.826+0000 7ff66effd640 1 -- 172.21.15.46:0/729842469 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7ff67005fd30 con 0x7ff6781610b0 2024-05-09T03:11:57.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6580762d0 msgr2=0x7ff6580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6580762d0 0x7ff6580787e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff668006fd0 tx=0x7ff668008040 comp rx=0 tx=0).stop 2024-05-09T03:11:57.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 msgr2=0x7ff6780aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 0x7ff6780aa230 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c6b0 tx=0x7ff67000cb70 comp rx=0 tx=0).stop 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 shutdown_connections 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6580762d0 0x7ff6580787e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6781610b0 0x7ff6780aa230 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff67815dfb0 0x7ff6781f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 --2- 172.21.15.46:0/729842469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff678070960 0x7ff67814ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 >> 172.21.15.46:0/729842469 conn(0x7ff6780b3690 msgr2=0x7ff6780b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 shutdown_connections 2024-05-09T03:11:57.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.828+0000 7ff67c849640 1 -- 172.21.15.46:0/729842469 wait complete. 2024-05-09T03:11:57.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-05-09T03:11:57.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.954+0000 7f9f65453640 1 Processor -- start 2024-05-09T03:11:57.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.954+0000 7f9f65453640 1 -- start start 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f65453640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f60160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f65453640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9f600b7ef0 0x7f9f600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f65453640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f600d49e0 con 0x7f9f600b4af0 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f65453640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9f600d5bb0 con 0x7f9f600b7ef0 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f65453640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9f600d4cc0 con 0x7f9f600d57a0 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9f600b7ef0 0x7f9f600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5e7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9f600b4af0 0x7f9f600b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40278/0 (socket says 172.21.15.46:40278) 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5e7fc640 1 -- 172.21.15.46:0/3030839200 learned_addr learned my addr 172.21.15.46:0/3030839200 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5effd640 1 --2- 172.21.15.46:0/3030839200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f60160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/3030839200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 msgr2=0x7f9f60160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:57.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5f7fe640 1 --2- 172.21.15.46:0/3030839200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f60160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/3030839200 >> v1:172.21.15.46:6789/0 conn(0x7f9f600b4af0 legacy=0x7f9f600b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5dffb640 1 -- 172.21.15.46:0/3030839200 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2229193582 0 0) 0x7f9f600d49e0 con 0x7f9f600b4af0 2024-05-09T03:11:57.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.956+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/3030839200 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f600d4fa0 con 0x7f9f600b7ef0 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.957+0000 7f9f5f7fe640 1 --2- 172.21.15.46:0/3030839200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600b7ef0 0x7f9f600b4f00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009d90 tx=0x7f9f5802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba18647e46b7d0f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.957+0000 7f9f5dffb640 1 -- 172.21.15.46:0/3030839200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58007740 con 0x7f9f600b7ef0 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.957+0000 7f9f5dffb640 1 -- 172.21.15.46:0/3030839200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f580078e0 con 0x7f9f600b7ef0 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.957+0000 7f9f5dffb640 1 -- 172.21.15.46:0/3030839200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58007cb0 con 0x7f9f600b7ef0 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 -- 172.21.15.46:0/3030839200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600b7ef0 msgr2=0x7f9f600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 --2- 172.21.15.46:0/3030839200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600b7ef0 0x7f9f600b4f00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009d90 tx=0x7f9f5802a660 comp rx=0 tx=0).stop 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 -- 172.21.15.46:0/3030839200 shutdown_connections 2024-05-09T03:11:57.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 --2- 172.21.15.46:0/3030839200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600b7ef0 0x7f9f600b4f00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 --2- 172.21.15.46:0/3030839200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f60160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 -- 172.21.15.46:0/3030839200 >> 172.21.15.46:0/3030839200 conn(0x7f9f600743e0 msgr2=0x7f9f600d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:57.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 -- 172.21.15.46:0/3030839200 shutdown_connections 2024-05-09T03:11:57.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.958+0000 7f9f65453640 1 -- 172.21.15.46:0/3030839200 wait complete. 2024-05-09T03:11:57.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.960+0000 7f9f65453640 1 Processor -- start 2024-05-09T03:11:57.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.960+0000 7f9f65453640 1 -- start start 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f65453640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f600b4af0 0x7f9f600cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f65453640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f600cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f65453640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 0x7f9f600cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f65453640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f600d1920 con 0x7f9f600cef40 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f65453640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9f600d15b0 con 0x7f9f600b4af0 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f65453640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9f600cd6a0 con 0x7f9f600d57a0 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f5f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 0x7f9f600cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f5f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 0x7f9f600cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47772/0 (socket says 172.21.15.46:47772) 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.961+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/4183511812 learned_addr learned my addr 172.21.15.46:0/4183511812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5e7fc640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f600cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5effd640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f600b4af0 0x7f9f600cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 msgr2=0x7f9f600cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5f7fe640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f600cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f600b4af0 msgr2=0x7f9f600cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5f7fe640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f600b4af0 0x7f9f600cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5f7fe640 1 -- 172.21.15.46:0/4183511812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6015c700 con 0x7f9f600cef40 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5effd640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f600b4af0 0x7f9f600cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5e7fc640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f600cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f5f7fe640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 0x7f9f600cc8a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9f5802ac50 tx=0x7f9f58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5803f070 con 0x7f9f600cef40 2024-05-09T03:11:57.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f58031bc0 con 0x7f9f600cef40 2024-05-09T03:11:57.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f600ca520 con 0x7f9f600cef40 2024-05-09T03:11:57.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58009450 con 0x7f9f600cef40 2024-05-09T03:11:57.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.962+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f600caae0 con 0x7f9f600cef40 2024-05-09T03:11:57.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.963+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f600bbc40 con 0x7f9f600cef40 2024-05-09T03:11:57.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.964+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9f58044460 con 0x7f9f600cef40 2024-05-09T03:11:57.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.964+0000 7f9f477fe640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f4c0762d0 0x7f9f4c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:57.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.965+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7f9f580b8c30 con 0x7f9f600cef40 2024-05-09T03:11:57.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.965+0000 7f9f5effd640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f4c0762d0 0x7f9f4c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:57.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.965+0000 7f9f5effd640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f4c0762d0 0x7f9f4c0787e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9f6006d3b0 tx=0x7f9f50008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:57.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:57.969+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9f600bbc40 con 0x7f9f600cef40 2024-05-09T03:11:58.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.240+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) -- 0x7f9f6006a270 con 0x7f9f600cef40 2024-05-09T03:11:58.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.797+0000 7f9f477fe640 1 -- 172.21.15.46:0/4183511812 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v141) ==== 141+0+0 (secure 0 0 0) 0x7f9f580820b0 con 0x7f9f600cef40 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f4c0762d0 msgr2=0x7f9f4c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f4c0762d0 0x7f9f4c0787e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9f6006d3b0 tx=0x7f9f50008040 comp rx=0 tx=0).stop 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 msgr2=0x7f9f600cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 0x7f9f600cc8a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9f5802ac50 tx=0x7f9f58004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 shutdown_connections 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f4c0762d0 0x7f9f4c0787e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f600cef40 0x7f9f600cc8a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f600d57a0 0x7f9f600cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 --2- 172.21.15.46:0/4183511812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f600b4af0 0x7f9f600cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 >> 172.21.15.46:0/4183511812 conn(0x7f9f600743e0 msgr2=0x7f9f600b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:58.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 shutdown_connections 2024-05-09T03:11:58.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.800+0000 7f9f65453640 1 -- 172.21.15.46:0/4183511812 wait complete. 2024-05-09T03:11:58.815 INFO:tasks.cephfs_test_runner:That the MDS won't abort when the dentry is already known to be damaged. ... ok 2024-05-09T03:11:58.815 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-05-09T03:11:58.815 INFO:tasks.cephfs_test_runner:test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) 2024-05-09T03:11:58.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:11:58.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.936+0000 7f7651fb7640 1 Processor -- start 2024-05-09T03:11:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.936+0000 7f7651fb7640 1 -- start start 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.937+0000 7f7651fb7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f764c164c70 0x7f764c1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.937+0000 7f7651fb7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f764c0d2480 0x7f764c0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.937+0000 7f7651fb7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f764c160b20 con 0x7f764c15dfe0 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.937+0000 7f7651fb7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f764c161150 con 0x7f764c0d2480 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.937+0000 7f7651fb7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f764c0d15f0 con 0x7f764c164c70 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.937+0000 7f764bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f764c0d2480 0x7f764c0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.938+0000 7f764bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f764c0d2480 0x7f764c0d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47776/0 (socket says 172.21.15.46:47776) 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.938+0000 7f764bfff640 1 -- 172.21.15.46:0/2691265794 learned_addr learned my addr 172.21.15.46:0/2691265794 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.938+0000 7f764bfff640 1 -- 172.21.15.46:0/2691265794 >> v2:172.21.15.88:3300/0 conn(0x7f764c164c70 msgr2=0x7f764c1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.938+0000 7f764bfff640 1 --2- 172.21.15.46:0/2691265794 >> v2:172.21.15.88:3300/0 conn(0x7f764c164c70 0x7f764c1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.938+0000 7f764bfff640 1 -- 172.21.15.46:0/2691265794 >> v1:172.21.15.88:6789/0 conn(0x7f764c15dfe0 legacy=0x7f764c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.938+0000 7f764bfff640 1 -- 172.21.15.46:0/2691265794 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764c0d18d0 con 0x7f764c0d2480 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.939+0000 7f764bfff640 1 --2- 172.21.15.46:0/2691265794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c0d2480 0x7f764c0d2890 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f764c0b8900 tx=0x7f764402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cca1b4ebea2fb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.939+0000 7f764affd640 1 -- 172.21.15.46:0/2691265794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644036070 con 0x7f764c0d2480 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.939+0000 7f764affd640 1 -- 172.21.15.46:0/2691265794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7644031600 con 0x7f764c0d2480 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.939+0000 7f764affd640 1 -- 172.21.15.46:0/2691265794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440319d0 con 0x7f764c0d2480 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.939+0000 7f7651fb7640 1 -- 172.21.15.46:0/2691265794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c0d2480 msgr2=0x7f764c0d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.939+0000 7f7651fb7640 1 --2- 172.21.15.46:0/2691265794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c0d2480 0x7f764c0d2890 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f764c0b8900 tx=0x7f764402a610 comp rx=0 tx=0).stop 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.940+0000 7f7651fb7640 1 -- 172.21.15.46:0/2691265794 shutdown_connections 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.940+0000 7f7651fb7640 1 --2- 172.21.15.46:0/2691265794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c0d2480 0x7f764c0d2890 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.940+0000 7f7651fb7640 1 --2- 172.21.15.46:0/2691265794 >> v2:172.21.15.88:3300/0 conn(0x7f764c164c70 0x7f764c1615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.940+0000 7f7651fb7640 1 -- 172.21.15.46:0/2691265794 >> 172.21.15.46:0/2691265794 conn(0x7f764c073db0 msgr2=0x7f764c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.940+0000 7f7651fb7640 1 -- 172.21.15.46:0/2691265794 shutdown_connections 2024-05-09T03:11:58.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.940+0000 7f7651fb7640 1 -- 172.21.15.46:0/2691265794 wait complete. 2024-05-09T03:11:58.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.941+0000 7f7651fb7640 1 Processor -- start 2024-05-09T03:11:58.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.941+0000 7f7651fb7640 1 -- start start 2024-05-09T03:11:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7651fb7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f764c0d2480 0x7f764c0cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7651fb7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f764c15dfe0 0x7f764c1f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7651fb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 0x7f764c163940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7651fb7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f764c0d2dd0 con 0x7f764c164c70 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7651fb7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f764c15e3f0 con 0x7f764c15dfe0 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7651fb7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f764c0d45c0 con 0x7f764c0d2480 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 0x7f764c163940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 0x7f764c163940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47790/0 (socket says 172.21.15.46:47790) 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 -- 172.21.15.46:0/1834401990 learned_addr learned my addr 172.21.15.46:0/1834401990 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f7643fff640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f764c15dfe0 0x7f764c1f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764b7fe640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f764c0d2480 0x7f764c0cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 -- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f764c0d2480 msgr2=0x7f764c0cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f764c0d2480 0x7f764c0cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 -- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f764c15dfe0 msgr2=0x7f764c1f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f764c15dfe0 0x7f764c1f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 -- 172.21.15.46:0/1834401990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764c1626d0 con 0x7f764c164c70 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.943+0000 7f764bfff640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 0x7f764c163940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7644009f40 tx=0x7f7644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.944+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644041020 con 0x7f764c164c70 2024-05-09T03:11:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.944+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76440077a0 con 0x7f764c164c70 2024-05-09T03:11:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.944+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644036040 con 0x7f764c164c70 2024-05-09T03:11:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.944+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f764c1629b0 con 0x7f764c164c70 2024-05-09T03:11:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.944+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f764c162f70 con 0x7f764c164c70 2024-05-09T03:11:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.945+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f764c08cfd0 con 0x7f764c164c70 2024-05-09T03:11:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.946+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7644044020 con 0x7f764c164c70 2024-05-09T03:11:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.946+0000 7f7648ff9640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f762c0762d0 0x7f762c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.946+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7f76440b8520 con 0x7f764c164c70 2024-05-09T03:11:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.946+0000 7f764b7fe640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f762c0762d0 0x7f762c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.947+0000 7f764b7fe640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f762c0762d0 0x7f762c0787e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f763c0098e0 tx=0x7f763c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:58.950+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f764c08cfd0 con 0x7f764c164c70 2024-05-09T03:11:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.142+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f764c06a270 con 0x7f764c164c70 2024-05-09T03:11:59.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.143+0000 7f7648ff9640 1 -- 172.21.15.46:0/1834401990 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+13076 (secure 0 0 0) 0x7f764404b030 con 0x7f764c164c70 2024-05-09T03:11:59.143 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:11:59.143 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":47,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:11:56.790987+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6833/881221292":"2024-05-10T03:11:53.785422+0000","172.21.15.46:6835/2630572362":"2024-05-10T03:11:47.200166+0000","172.21.15.46:6835/881221292":"2024-05-10T03:11:53.785422+0000","172.21.15.46:6833/2630572362":"2024-05-10T03:11:47.200166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f762c0762d0 msgr2=0x7f762c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f762c0762d0 0x7f762c0787e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f763c0098e0 tx=0x7f763c009060 comp rx=0 tx=0).stop 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 msgr2=0x7f764c163940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 0x7f764c163940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7644009f40 tx=0x7f7644004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 shutdown_connections 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f762c0762d0 0x7f762c0787e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f764c164c70 0x7f764c163940 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f764c15dfe0 0x7f764c1f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 --2- 172.21.15.46:0/1834401990 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f764c0d2480 0x7f764c0cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 >> 172.21.15.46:0/1834401990 conn(0x7f764c073db0 msgr2=0x7f764c15fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 shutdown_connections 2024-05-09T03:11:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.145+0000 7f7651fb7640 1 -- 172.21.15.46:0/1834401990 wait complete. 2024-05-09T03:11:59.157 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T03:09:22.506928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T03:11:59.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T03:11:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.274+0000 7faafbfd5640 1 Processor -- start 2024-05-09T03:11:59.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.274+0000 7faafbfd5640 1 -- start start 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.275+0000 7faafbfd5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faaf40d2e90 0x7faaf40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.275+0000 7faafbfd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf41605b0 0x7faaf40b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.275+0000 7faafbfd5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaf40b4fc0 0x7faaf40d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafbfd5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faaf40d3400 con 0x7faaf40b4fc0 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafbfd5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faaf40d4a60 con 0x7faaf40d2e90 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafbfd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faaf40d4d40 con 0x7faaf41605b0 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaf40b4fc0 0x7faaf40d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaf40b4fc0 0x7faaf40d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47794/0 (socket says 172.21.15.46:47794) 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 -- 172.21.15.46:0/888849843 learned_addr learned my addr 172.21.15.46:0/888849843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faaf9d71640 1 --2- 172.21.15.46:0/888849843 >> v2:172.21.15.88:3300/0 conn(0x7faaf40d2e90 0x7faaf40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faaf9570640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf41605b0 0x7faaf40b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 -- 172.21.15.46:0/888849843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf41605b0 msgr2=0x7faaf40b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf41605b0 0x7faaf40b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 -- 172.21.15.46:0/888849843 >> v2:172.21.15.88:3300/0 conn(0x7faaf40d2e90 msgr2=0x7faaf40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 --2- 172.21.15.46:0/888849843 >> v2:172.21.15.88:3300/0 conn(0x7faaf40d2e90 0x7faaf40b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faafa572640 1 -- 172.21.15.46:0/888849843 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf40d5020 con 0x7faaf40b4fc0 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faaf9d71640 1 --2- 172.21.15.46:0/888849843 >> v2:172.21.15.88:3300/0 conn(0x7faaf40d2e90 0x7faaf40b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.276+0000 7faaf9570640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf41605b0 0x7faaf40b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.277+0000 7faafa572640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf40b4fc0 0x7faaf40d38b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7faaec009e50 tx=0x7faaec02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d888dafe1fa2f7ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.277+0000 7faaf8d6f640 1 -- 172.21.15.46:0/888849843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec036070 con 0x7faaf40b4fc0 2024-05-09T03:11:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.277+0000 7faaf8d6f640 1 -- 172.21.15.46:0/888849843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaec031600 con 0x7faaf40b4fc0 2024-05-09T03:11:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.277+0000 7faaf8d6f640 1 -- 172.21.15.46:0/888849843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec0319d0 con 0x7faaf40b4fc0 2024-05-09T03:11:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 -- 172.21.15.46:0/888849843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf40b4fc0 msgr2=0x7faaf40d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf40b4fc0 0x7faaf40d38b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7faaec009e50 tx=0x7faaec02a6d0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 -- 172.21.15.46:0/888849843 shutdown_connections 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf40b4fc0 0x7faaf40d38b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 --2- 172.21.15.46:0/888849843 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf41605b0 0x7faaf40b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 --2- 172.21.15.46:0/888849843 >> v2:172.21.15.88:3300/0 conn(0x7faaf40d2e90 0x7faaf40b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 -- 172.21.15.46:0/888849843 >> 172.21.15.46:0/888849843 conn(0x7faaf408d390 msgr2=0x7faaf408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 -- 172.21.15.46:0/888849843 shutdown_connections 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.278+0000 7faafbfd5640 1 -- 172.21.15.46:0/888849843 wait complete. 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.280+0000 7faafbfd5640 1 Processor -- start 2024-05-09T03:11:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.280+0000 7faafbfd5640 1 -- start start 2024-05-09T03:11:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafbfd5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf40b4fc0 0x7faaf41f9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafbfd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf40d2e90 0x7faaf40cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafbfd5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 0x7faaf40cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafbfd5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faaf40b4d80 con 0x7faaf41605b0 2024-05-09T03:11:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafbfd5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faaf40b4c00 con 0x7faaf40b4fc0 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafbfd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faaf40d3df0 con 0x7faaf40d2e90 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 0x7faaf40cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 0x7faaf40cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47796/0 (socket says 172.21.15.46:47796) 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 -- 172.21.15.46:0/1869618628 learned_addr learned my addr 172.21.15.46:0/1869618628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faaf9d71640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf40b4fc0 0x7faaf41f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faaf9570640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf40d2e90 0x7faaf40cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 -- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf40d2e90 msgr2=0x7faaf40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf40d2e90 0x7faaf40cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 -- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf40b4fc0 msgr2=0x7faaf41f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf40b4fc0 0x7faaf41f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faafa572640 1 -- 172.21.15.46:0/1869618628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf41f9760 con 0x7faaf41605b0 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faaf9d71640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf40b4fc0 0x7faaf41f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.281+0000 7faaf9570640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf40d2e90 0x7faaf40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.282+0000 7faafa572640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 0x7faaf40cb7e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7faaec034040 tx=0x7faaec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.282+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec036050 con 0x7faaf41605b0 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.282+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaec042070 con 0x7faaf41605b0 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.282+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf4161c50 con 0x7faaf41605b0 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.282+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec009510 con 0x7faaf41605b0 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.282+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf40ce520 con 0x7faaf41605b0 2024-05-09T03:11:59.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.283+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf406a270 con 0x7faaf41605b0 2024-05-09T03:11:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.284+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faaec007c50 con 0x7faaf41605b0 2024-05-09T03:11:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.284+0000 7faaea7fc640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad80762d0 0x7faad80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.284+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7faaec0b8280 con 0x7faaf41605b0 2024-05-09T03:11:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.285+0000 7faaf9d71640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad80762d0 0x7faad80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.285+0000 7faaf9d71640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad80762d0 0x7faad80787e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faaf0006fd0 tx=0x7faaf0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:59.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.289+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faaec081e90 con 0x7faaf41605b0 2024-05-09T03:11:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.477+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faaf40b7e30 con 0x7faaf41605b0 2024-05-09T03:11:59.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.478+0000 7faaea7fc640 1 -- 172.21.15.46:0/1869618628 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7faaec081790 con 0x7faaf41605b0 2024-05-09T03:11:59.479 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad80762d0 msgr2=0x7faad80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad80762d0 0x7faad80787e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7faaf0006fd0 tx=0x7faaf0008040 comp rx=0 tx=0).stop 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 msgr2=0x7faaf40cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 0x7faaf40cb7e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7faaec034040 tx=0x7faaec004a40 comp rx=0 tx=0).stop 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 shutdown_connections 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad80762d0 0x7faad80787e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf41605b0 0x7faaf40cb7e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf40d2e90 0x7faaf40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 --2- 172.21.15.46:0/1869618628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf40b4fc0 0x7faaf41f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 >> 172.21.15.46:0/1869618628 conn(0x7faaf408d390 msgr2=0x7faaf4075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:59.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 shutdown_connections 2024-05-09T03:11:59.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.480+0000 7faafbfd5640 1 -- 172.21.15.46:0/1869618628 wait complete. 2024-05-09T03:11:59.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-05-09T03:11:59.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.606+0000 7feef6dba640 1 Processor -- start 2024-05-09T03:11:59.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.606+0000 7feef6dba640 1 -- start start 2024-05-09T03:11:59.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef6dba640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef6dba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feef00d2480 0x7feef00d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef6dba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feef0160b20 con 0x7feef0164c70 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef6dba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7feef015e520 con 0x7feef00d2480 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef6dba640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7feef0161150 con 0x7feef01615e0 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef5357640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feef00d2480 0x7feef00d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef5357640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feef00d2480 0x7feef00d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47800/0 (socket says 172.21.15.46:47800) 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.607+0000 7feef5357640 1 -- 172.21.15.46:0/2029152570 learned_addr learned my addr 172.21.15.46:0/2029152570 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.608+0000 7feeeffff640 1 --2- 172.21.15.46:0/2029152570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.608+0000 7feef5357640 1 -- 172.21.15.46:0/2029152570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 msgr2=0x7feef015dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.608+0000 7feef5357640 1 --2- 172.21.15.46:0/2029152570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef015dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.608+0000 7feef5357640 1 -- 172.21.15.46:0/2029152570 >> v1:172.21.15.88:6789/0 conn(0x7feef0164c70 legacy=0x7feef00d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.608+0000 7feef5357640 1 -- 172.21.15.46:0/2029152570 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef00d15f0 con 0x7feef00d2480 2024-05-09T03:11:59.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.608+0000 7feef5357640 1 --2- 172.21.15.46:0/2029152570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00d2890 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7feee8009e50 tx=0x7feee802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962b98b3b7c0e75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feeef7fe640 1 -- 172.21.15.46:0/2029152570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8036070 con 0x7feef00d2480 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feeef7fe640 1 -- 172.21.15.46:0/2029152570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8031600 con 0x7feef00d2480 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feeef7fe640 1 -- 172.21.15.46:0/2029152570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee80319d0 con 0x7feef00d2480 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 -- 172.21.15.46:0/2029152570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 msgr2=0x7feef00d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 --2- 172.21.15.46:0/2029152570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00d2890 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7feee8009e50 tx=0x7feee802a710 comp rx=0 tx=0).stop 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 -- 172.21.15.46:0/2029152570 shutdown_connections 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 --2- 172.21.15.46:0/2029152570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00d2890 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 --2- 172.21.15.46:0/2029152570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef015dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 -- 172.21.15.46:0/2029152570 >> 172.21.15.46:0/2029152570 conn(0x7feef0073db0 msgr2=0x7feef00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 -- 172.21.15.46:0/2029152570 shutdown_connections 2024-05-09T03:11:59.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.609+0000 7feef6dba640 1 -- 172.21.15.46:0/2029152570 wait complete. 2024-05-09T03:11:59.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.611+0000 7feef6dba640 1 Processor -- start 2024-05-09T03:11:59.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.611+0000 7feef6dba640 1 -- start start 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef6dba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef6dba640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef01508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef6dba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feef0164c70 0x7feef00d4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef6dba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feef01619f0 con 0x7feef00d2480 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef6dba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7feef0160de0 con 0x7feef0164c70 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef6dba640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7feef00d2dd0 con 0x7feef01615e0 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47804/0 (socket says 172.21.15.46:47804) 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 -- 172.21.15.46:0/2466368784 learned_addr learned my addr 172.21.15.46:0/2466368784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef5357640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feef0164c70 0x7feef00d4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feeeffff640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef01508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 -- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 msgr2=0x7feef01508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef01508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 -- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feef0164c70 msgr2=0x7feef00d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feef0164c70 0x7feef00d4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 -- 172.21.15.46:0/2466368784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef00c9950 con 0x7feef00d2480 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.613+0000 7feef4b56640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00cb990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7feee400c3a0 tx=0x7feee400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.614+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4017070 con 0x7feef00d2480 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.614+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee40052d0 con 0x7feef00d2480 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.614+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef00cc370 con 0x7feef00d2480 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.614+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4003830 con 0x7feef00d2480 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.614+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef00c9ea0 con 0x7feef00d2480 2024-05-09T03:11:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.614+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef008cfd0 con 0x7feef00d2480 2024-05-09T03:11:59.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.615+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feee401a070 con 0x7feef00d2480 2024-05-09T03:11:59.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.616+0000 7feeed7fa640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed4076300 0x7feed4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:11:59.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.616+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7feee40965b0 con 0x7feef00d2480 2024-05-09T03:11:59.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.616+0000 7feeeffff640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed4076300 0x7feed4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:11:59.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.616+0000 7feeeffff640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed4076300 0x7feed4078810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feef00cb920 tx=0x7feedc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:11:59.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.620+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7feef008cfd0 con 0x7feef00d2480 2024-05-09T03:11:59.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:11:59.889+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) -- 0x7feef006a270 con 0x7feef00d2480 2024-05-09T03:12:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.804+0000 7feeed7fa640 1 -- 172.21.15.46:0/2466368784 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v143) ==== 145+0+0 (secure 0 0 0) 0x7feee40681a0 con 0x7feef00d2480 2024-05-09T03:12:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.806+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed4076300 msgr2=0x7feed4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.806+0000 7feef6dba640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed4076300 0x7feed4078810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feef00cb920 tx=0x7feedc008040 comp rx=0 tx=0).stop 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.806+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 msgr2=0x7feef00cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.806+0000 7feef6dba640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00cb990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7feee400c3a0 tx=0x7feee400c860 comp rx=0 tx=0).stop 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 shutdown_connections 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feef0164c70 0x7feef00d4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feef01615e0 0x7feef01508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed4076300 0x7feed4078810 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 --2- 172.21.15.46:0/2466368784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feef00d2480 0x7feef00cb990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 >> 172.21.15.46:0/2466368784 conn(0x7feef0073db0 msgr2=0x7feef015fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 shutdown_connections 2024-05-09T03:12:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.807+0000 7feef6dba640 1 -- 172.21.15.46:0/2466368784 wait complete. 2024-05-09T03:12:00.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T03:12:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.946+0000 7f49d2805640 1 Processor -- start 2024-05-09T03:12:00.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.946+0000 7f49d2805640 1 -- start start 2024-05-09T03:12:00.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49d2805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc1605b0 0x7f49cc0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49d2805640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49cc0d3400 con 0x7f49cc0b7e70 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49d2805640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49cc0d4a60 con 0x7f49cc0d2e90 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49d2805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f49cc0d4d40 con 0x7f49cc1605b0 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49c3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f49cc0b7e70 0x7f49cc0b92a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40280/0 (socket says 172.21.15.46:40280) 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49c3fff640 1 -- 172.21.15.46:0/3771227882 learned_addr learned my addr 172.21.15.46:0/3771227882 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49d0da2640 1 --2- 172.21.15.46:0/3771227882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc1605b0 0x7f49cc0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3758016713 0 0) 0x7f49cc0d3400 con 0x7f49cc0b7e70 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.948+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49b8003520 con 0x7f49cc0b7e70 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1868826932 0 0) 0x7f49cc0d4a60 con 0x7f49cc0d2e90 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49cc0d3400 con 0x7f49cc0d2e90 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1847065099 0 0) 0x7f49b8003520 con 0x7f49cc0b7e70 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49d0da2640 1 -- 172.21.15.46:0/3771227882 >> v1:172.21.15.88:6789/0 conn(0x7f49cc0d2e90 legacy=0x7f49cc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49d0da2640 1 -- 172.21.15.46:0/3771227882 >> v1:172.21.15.46:6789/0 conn(0x7f49cc0b7e70 legacy=0x7f49cc0b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49d0da2640 1 -- 172.21.15.46:0/3771227882 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc0d5020 con 0x7f49cc1605b0 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.949+0000 7f49d0da2640 1 --2- 172.21.15.46:0/3771227882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc1605b0 0x7f49cc0b99b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009dc0 tx=0x7f49c402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e8b7756b2647252 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b40016f0 con 0x7f49cc1605b0 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c4007540 con 0x7f49cc1605b0 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49cb7fe640 1 -- 172.21.15.46:0/3771227882 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c40078f0 con 0x7f49cc1605b0 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49d2805640 1 -- 172.21.15.46:0/3771227882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc1605b0 msgr2=0x7f49cc0b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49d2805640 1 --2- 172.21.15.46:0/3771227882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc1605b0 0x7f49cc0b99b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009dc0 tx=0x7f49c402a660 comp rx=0 tx=0).stop 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49d2805640 1 -- 172.21.15.46:0/3771227882 shutdown_connections 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49d2805640 1 --2- 172.21.15.46:0/3771227882 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc1605b0 0x7f49cc0b99b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49d2805640 1 -- 172.21.15.46:0/3771227882 >> 172.21.15.46:0/3771227882 conn(0x7f49cc08d390 msgr2=0x7f49cc08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.950+0000 7f49d2805640 1 -- 172.21.15.46:0/3771227882 shutdown_connections 2024-05-09T03:12:00.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.951+0000 7f49d2805640 1 -- 172.21.15.46:0/3771227882 wait complete. 2024-05-09T03:12:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.952+0000 7f49d2805640 1 Processor -- start 2024-05-09T03:12:00.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.952+0000 7f49d2805640 1 -- start start 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc0d2e90 0x7f49cc15c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 0x7f49cc0b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49cc0cf620 0x7f49cc0cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc0b4d80 con 0x7f49cc1605b0 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc0b4c00 con 0x7f49cc0cf620 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f49cc0b4fc0 con 0x7f49cc0d2e90 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 0x7f49cc0b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 0x7f49cc0b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47814/0 (socket says 172.21.15.46:47814) 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 -- 172.21.15.46:0/3398793989 learned_addr learned my addr 172.21.15.46:0/3398793989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:00.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49cbfff640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc0d2e90 0x7f49cc15c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d0da2640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49cc0cf620 0x7f49cc0cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 -- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc0d2e90 msgr2=0x7f49cc15c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc0d2e90 0x7f49cc15c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 -- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49cc0cf620 msgr2=0x7f49cc0cdb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49cc0cf620 0x7f49cc0cdb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 -- 172.21.15.46:0/3398793989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc0b4740 con 0x7f49cc1605b0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c3fff640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 0x7f49cc0b4200 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f49b400c490 tx=0x7f49b400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4017070 con 0x7f49cc1605b0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b4005ce0 con 0x7f49cc1605b0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.954+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49cc0cfba0 con 0x7f49cc1605b0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.955+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b40072b0 con 0x7f49cc1605b0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.955+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49cc0cd030 con 0x7f49cc1605b0 2024-05-09T03:12:00.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.955+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49cc06a270 con 0x7f49cc1605b0 2024-05-09T03:12:00.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.956+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f49b401a070 con 0x7f49cc1605b0 2024-05-09T03:12:00.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.957+0000 7f49c97fa640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f49ac076300 0x7f49ac078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:00.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.957+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7f49b4096740 con 0x7f49cc1605b0 2024-05-09T03:12:00.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.957+0000 7f49cbfff640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f49ac076300 0x7f49ac078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:00.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.958+0000 7f49cbfff640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f49ac076300 0x7f49ac078810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f49bc001120 tx=0x7f49bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:00.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:00.961+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f49b4060340 con 0x7f49cc1605b0 2024-05-09T03:12:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.154+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f49cc1e8d60 con 0x7f49cc1605b0 2024-05-09T03:12:01.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.157+0000 7f49c97fa640 1 -- 172.21.15.46:0/3398793989 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7f49b405fc40 con 0x7f49cc1605b0 2024-05-09T03:12:01.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f49ac076300 msgr2=0x7f49ac078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f49ac076300 0x7f49ac078810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f49bc001120 tx=0x7f49bc009040 comp rx=0 tx=0).stop 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 msgr2=0x7f49cc0b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 0x7f49cc0b4200 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f49b400c490 tx=0x7f49b400c950 comp rx=0 tx=0).stop 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 shutdown_connections 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49cc0cf620 0x7f49cc0cdb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f49ac076300 0x7f49ac078810 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49cc1605b0 0x7f49cc0b4200 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 --2- 172.21.15.46:0/3398793989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f49cc0d2e90 0x7f49cc15c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 >> 172.21.15.46:0/3398793989 conn(0x7f49cc08d390 msgr2=0x7f49cc0d38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 shutdown_connections 2024-05-09T03:12:01.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.160+0000 7f49d2805640 1 -- 172.21.15.46:0/3398793989 wait complete. 2024-05-09T03:12:01.172 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:12:01.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:12:01.220 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:12:01.266 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:12:01.266 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:12:01.267 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:12:01.283 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:12:01.283 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:12:01.283 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:12:01.301 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:12:01.301 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:12:01.301 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:12:01.340 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:12:01.341 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:12:01.341 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:12:01.358 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:12:01.358 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T03:12:01.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:01.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.494+0000 7f487c576640 1 Processor -- start 2024-05-09T03:12:01.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.494+0000 7f487c576640 1 -- start start 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.495+0000 7f487c576640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48741615e0 0x7f487415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.495+0000 7f487c576640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48740d2480 0x7f48740d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.495+0000 7f487c576640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4874160b20 con 0x7f4874164c70 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.495+0000 7f487c576640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f487415e520 con 0x7f48741615e0 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.495+0000 7f487c576640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4874161150 con 0x7f48740d2480 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f487a312640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4874164c70 0x7f48740d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40288/0 (socket says 172.21.15.46:40288) 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f4879b11640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48741615e0 0x7f487415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f487a312640 1 -- 172.21.15.46:0/424731245 learned_addr learned my addr 172.21.15.46:0/424731245 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f487ab13640 1 --2- 172.21.15.46:0/424731245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48740d2480 0x7f48740d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f4879b11640 1 -- 172.21.15.46:0/424731245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48740d2480 msgr2=0x7f48740d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f4879b11640 1 --2- 172.21.15.46:0/424731245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48740d2480 0x7f48740d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f4879b11640 1 -- 172.21.15.46:0/424731245 >> v1:172.21.15.46:6789/0 conn(0x7f4874164c70 legacy=0x7f48740d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f4879310640 1 -- 172.21.15.46:0/424731245 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955827985 0 0) 0x7f4874160b20 con 0x7f4874164c70 2024-05-09T03:12:01.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.496+0000 7f4879b11640 1 -- 172.21.15.46:0/424731245 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48740d15f0 con 0x7f48741615e0 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.497+0000 7f4879b11640 1 --2- 172.21.15.46:0/424731245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48741615e0 0x7f487415dfe0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48640097a0 tx=0x7f486402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0b66dc1cf263dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.497+0000 7f4879310640 1 -- 172.21.15.46:0/424731245 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864037070 con 0x7f48741615e0 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.497+0000 7f4879310640 1 -- 172.21.15.46:0/424731245 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4864032440 con 0x7f48741615e0 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.497+0000 7f4879310640 1 -- 172.21.15.46:0/424731245 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4864032810 con 0x7f48741615e0 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 -- 172.21.15.46:0/424731245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48741615e0 msgr2=0x7f487415dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 --2- 172.21.15.46:0/424731245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48741615e0 0x7f487415dfe0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f48640097a0 tx=0x7f486402a180 comp rx=0 tx=0).stop 2024-05-09T03:12:01.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 -- 172.21.15.46:0/424731245 shutdown_connections 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 --2- 172.21.15.46:0/424731245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48740d2480 0x7f48740d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 --2- 172.21.15.46:0/424731245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48741615e0 0x7f487415dfe0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 -- 172.21.15.46:0/424731245 >> 172.21.15.46:0/424731245 conn(0x7f4874073db0 msgr2=0x7f48740741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 -- 172.21.15.46:0/424731245 shutdown_connections 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.498+0000 7f487c576640 1 -- 172.21.15.46:0/424731245 wait complete. 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.499+0000 7f487c576640 1 Processor -- start 2024-05-09T03:12:01.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.499+0000 7f487c576640 1 -- start start 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487c576640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 0x7f48740ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487c576640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48741615e0 0x7f487414c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487c576640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4874164c70 0x7f4874162070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487c576640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48741619f0 con 0x7f48740d2480 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487c576640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4874160de0 con 0x7f48741615e0 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487c576640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48740d2dd0 con 0x7f4874164c70 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 0x7f48740ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 0x7f48740ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47832/0 (socket says 172.21.15.46:47832) 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 -- 172.21.15.46:0/1531825760 learned_addr learned my addr 172.21.15.46:0/1531825760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:01.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487ab13640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4874164c70 0x7f4874162070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f4879b11640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48741615e0 0x7f487414c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 -- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4874164c70 msgr2=0x7f4874162070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4874164c70 0x7f4874162070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 -- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48741615e0 msgr2=0x7f487414c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48741615e0 0x7f487414c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 -- 172.21.15.46:0/1531825760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48740cd5a0 con 0x7f48740d2480 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f4879b11640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48741615e0 0x7f487414c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.501+0000 7f487a312640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 0x7f48740ccd50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f487000c410 tx=0x7f487000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.502+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48700074e0 con 0x7f48740d2480 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.502+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48700047e0 con 0x7f48740d2480 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.502+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48740cd880 con 0x7f48740d2480 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.502+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4870004bd0 con 0x7f48740d2480 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.502+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48740cde40 con 0x7f48740d2480 2024-05-09T03:12:01.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.502+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487408cfd0 con 0x7f48740d2480 2024-05-09T03:12:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.503+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4870011070 con 0x7f48740d2480 2024-05-09T03:12:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.504+0000 7f486affd640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48580762d0 0x7f48580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.504+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7f4870096ec0 con 0x7f48740d2480 2024-05-09T03:12:01.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.504+0000 7f4879b11640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48580762d0 0x7f48580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.505+0000 7f4879b11640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48580762d0 0x7f48580787e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4864034000 tx=0x7f486402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:01.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.509+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f487408cfd0 con 0x7f48740d2480 2024-05-09T03:12:01.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.736+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f487406a270 con 0x7f48740d2480 2024-05-09T03:12:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.736+0000 7f486affd640 1 -- 172.21.15.46:0/1531825760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+3860 (secure 0 0 0) 0x7f4870060ac0 con 0x7f48740d2480 2024-05-09T03:12:01.737 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:01.737 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5194,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/97590007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":97590007},{"type":"v1","addr":"172.21.15.46:6834","nonce":97590007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5197,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/350451000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":350451000},{"type":"v1","addr":"172.21.15.88:6836","nonce":350451000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5200,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2819198314","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2819198314},{"type":"v1","addr":"172.21.15.88:6837","nonce":2819198314}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25},{"gid":5772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1734999265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1734999265},{"type":"v1","addr":"172.21.15.46:6835","nonce":1734999265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[]} 2024-05-09T03:12:01.737 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 39 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48580762d0 msgr2=0x7f48580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48580762d0 0x7f48580787e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4864034000 tx=0x7f486402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 msgr2=0x7f48740ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 0x7f48740ccd50 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f487000c410 tx=0x7f487000c8d0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 shutdown_connections 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4874164c70 0x7f4874162070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48741615e0 0x7f487414c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48580762d0 0x7f48580787e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 --2- 172.21.15.46:0/1531825760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48740d2480 0x7f48740ccd50 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 >> 172.21.15.46:0/1531825760 conn(0x7f4874073db0 msgr2=0x7f487415e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 shutdown_connections 2024-05-09T03:12:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.739+0000 7f487c576640 1 -- 172.21.15.46:0/1531825760 wait complete. 2024-05-09T03:12:01.751 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:12:01.751 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:12:01.751 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:12:01.751 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:12:01.751 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:12:01.751 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T03:12:01.752 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:12:01.752 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:12:01.752 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:12:01.752 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T03:12:01.752 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:12:01.752 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:12:01.752 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:12:01.753 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T03:12:01.753 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:12:01.753 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:12:01.761 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:12:01.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:12:01.764 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:12:01.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:12:01.765 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:12:01.765 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:12:01.766 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:12:01.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:12:01.806 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:12:01.807 DEBUG:teuthology.parallel:result is None 2024-05-09T03:12:01.808 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:12:01.808 DEBUG:teuthology.parallel:result is None 2024-05-09T03:12:01.809 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:12:01.809 DEBUG:teuthology.parallel:result is None 2024-05-09T03:12:01.810 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:12:01.810 DEBUG:teuthology.parallel:result is None 2024-05-09T03:12:01.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T03:12:01.897 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:12:01.898 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:12:01.904 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:12:01.905 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:12:01.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.932+0000 7f534f09f640 1 Processor -- start 2024-05-09T03:12:01.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.932+0000 7f534f09f640 1 -- start start 2024-05-09T03:12:01.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534f09f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53480d0020 0x7f53480b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534f09f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53480b3800 0x7f53480b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534f09f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53480b92c0 con 0x7f53480d1110 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534f09f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f53480d0430 con 0x7f53480b3800 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534f09f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53480b2f10 con 0x7f53480d0020 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53480b3800 0x7f53480b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53480b3800 0x7f53480b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47878/0 (socket says 172.21.15.46:47878) 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 -- 172.21.15.46:0/3332830260 learned_addr learned my addr 172.21.15.46:0/3332830260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f5347fff640 1 --2- 172.21.15.46:0/3332830260 >> v2:172.21.15.88:3300/0 conn(0x7f53480d0020 0x7f53480b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f53477fe640 1 -- 172.21.15.46:0/3332830260 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2318932731 0 0) 0x7f53480b92c0 con 0x7f53480d1110 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f53477fe640 1 -- 172.21.15.46:0/3332830260 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5330003520 con 0x7f53480d1110 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 -- 172.21.15.46:0/3332830260 >> v2:172.21.15.88:3300/0 conn(0x7f53480d0020 msgr2=0x7f53480b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 --2- 172.21.15.46:0/3332830260 >> v2:172.21.15.88:3300/0 conn(0x7f53480d0020 0x7f53480b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 -- 172.21.15.46:0/3332830260 >> v1:172.21.15.88:6789/0 conn(0x7f53480d1110 legacy=0x7f53480d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f534d63c640 1 -- 172.21.15.46:0/3332830260 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53480b9560 con 0x7f53480b3800 2024-05-09T03:12:01.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.934+0000 7f5347fff640 1 --2- 172.21.15.46:0/3332830260 >> v2:172.21.15.88:3300/0 conn(0x7f53480d0020 0x7f53480b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:01.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.935+0000 7f534d63c640 1 --2- 172.21.15.46:0/3332830260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480b3800 0x7f53480b3c10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f534808cef0 tx=0x7f534002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9dcc2fe51bf1a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.935+0000 7f53477fe640 1 -- 172.21.15.46:0/3332830260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340036070 con 0x7f53480b3800 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.935+0000 7f53477fe640 1 -- 172.21.15.46:0/3332830260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5340031600 con 0x7f53480b3800 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.935+0000 7f53477fe640 1 -- 172.21.15.46:0/3332830260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400319d0 con 0x7f53480b3800 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 -- 172.21.15.46:0/3332830260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480b3800 msgr2=0x7f53480b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 --2- 172.21.15.46:0/3332830260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480b3800 0x7f53480b3c10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f534808cef0 tx=0x7f534002a6a0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 -- 172.21.15.46:0/3332830260 shutdown_connections 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 --2- 172.21.15.46:0/3332830260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480b3800 0x7f53480b3c10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 --2- 172.21.15.46:0/3332830260 >> v2:172.21.15.88:3300/0 conn(0x7f53480d0020 0x7f53480b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 -- 172.21.15.46:0/3332830260 >> 172.21.15.46:0/3332830260 conn(0x7f53480d05f0 msgr2=0x7f53480d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:01.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 -- 172.21.15.46:0/3332830260 shutdown_connections 2024-05-09T03:12:01.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.936+0000 7f534f09f640 1 -- 172.21.15.46:0/3332830260 wait complete. 2024-05-09T03:12:01.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.938+0000 7f534f09f640 1 Processor -- start 2024-05-09T03:12:01.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.938+0000 7f534f09f640 1 -- start start 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534f09f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53480b3800 0x7f53481e7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534f09f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 0x7f534815a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534f09f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5348145bb0 0x7f53480b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534f09f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f534809ceb0 con 0x7f53480d0020 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534f09f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53480d2030 con 0x7f5348145bb0 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534f09f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f534809d030 con 0x7f53480b3800 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f5347fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 0x7f534815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f5347fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 0x7f534815a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47894/0 (socket says 172.21.15.46:47894) 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f5347fff640 1 -- 172.21.15.46:0/203101558 learned_addr learned my addr 172.21.15.46:0/203101558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.939+0000 7f534ce3b640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53480b3800 0x7f53481e7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f534d63c640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5348145bb0 0x7f53480b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f5347fff640 1 -- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53480b3800 msgr2=0x7f53481e7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f5347fff640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53480b3800 0x7f53481e7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f5347fff640 1 -- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5348145bb0 msgr2=0x7f53480b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f5347fff640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5348145bb0 0x7f53480b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f5347fff640 1 -- 172.21.15.46:0/203101558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53481e78c0 con 0x7f53480d0020 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f5347fff640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 0x7f534815a850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f533c00e910 tx=0x7f533c00edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c017070 con 0x7f53480d0020 2024-05-09T03:12:01.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f533c00c9a0 con 0x7f53480d0020 2024-05-09T03:12:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53481e7ba0 con 0x7f53480d0020 2024-05-09T03:12:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c00cd50 con 0x7f53480d0020 2024-05-09T03:12:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.940+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5348157c80 con 0x7f53480d0020 2024-05-09T03:12:01.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.941+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53481e80c0 con 0x7f53480d0020 2024-05-09T03:12:01.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.942+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f533c01a070 con 0x7f53480d0020 2024-05-09T03:12:01.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.943+0000 7f53457fa640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53340762d0 0x7f53340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:01.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.943+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5029+0+0 (secure 0 0 0) 0x7f533c097440 con 0x7f53480d0020 2024-05-09T03:12:01.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.943+0000 7f534ce3b640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53340762d0 0x7f53340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:01.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.943+0000 7f534ce3b640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53340762d0 0x7f53340787e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5348076c00 tx=0x7f5338009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:01.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:01.947+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f53481e80c0 con 0x7f53480d0020 2024-05-09T03:12:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.151+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f53481e7650 con 0x7f53480d0020 2024-05-09T03:12:02.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.817+0000 7f53457fa640 1 -- 172.21.15.46:0/203101558 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v49) ==== 97+0+0 (secure 0 0 0) 0x7f533c0609f0 con 0x7f53480d0020 2024-05-09T03:12:02.818 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-05-09T03:12:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.820+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53340762d0 msgr2=0x7f53340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.820+0000 7f534f09f640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53340762d0 0x7f53340787e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5348076c00 tx=0x7f5338009040 comp rx=0 tx=0).stop 2024-05-09T03:12:02.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.820+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 msgr2=0x7f534815a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.820+0000 7f534f09f640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 0x7f534815a850 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f533c00e910 tx=0x7f533c00edd0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 shutdown_connections 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5348145bb0 0x7f53480b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53340762d0 0x7f53340787e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d0020 0x7f534815a850 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 --2- 172.21.15.46:0/203101558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53480b3800 0x7f53481e7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 >> 172.21.15.46:0/203101558 conn(0x7f53480d05f0 msgr2=0x7f53480b7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 shutdown_connections 2024-05-09T03:12:02.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.821+0000 7f534f09f640 1 -- 172.21.15.46:0/203101558 wait complete. 2024-05-09T03:12:02.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T03:12:02.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.958+0000 7f9872b7d640 1 Processor -- start 2024-05-09T03:12:02.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.959+0000 7f9872b7d640 1 -- start start 2024-05-09T03:12:02.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f9872b7d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:02.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f9872b7d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f986c0d0b80 0x7f986c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f9872b7d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f986c0d15b0 con 0x7f986c0d25e0 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f9872b7d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f986c160b10 con 0x7f986c0d0b80 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f9872b7d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f986c160df0 con 0x7f986c1615f0 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f986bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f986c0d0b80 0x7f986c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f9870919640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f987111a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f986c0d25e0 0x7f986c0d29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:48436/0 (socket says 172.21.15.46:48436) 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f986bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f986c0d0b80 0x7f986c0d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:46322/0 (socket says 172.21.15.46:46322) 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.960+0000 7f987111a640 1 -- 172.21.15.46:0/1372168755 learned_addr learned my addr 172.21.15.46:0/1372168755 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.961+0000 7f986bfff640 1 -- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 msgr2=0x7f986c0b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.961+0000 7f986bfff640 1 --2- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.961+0000 7f986bfff640 1 -- 172.21.15.46:0/1372168755 >> v1:172.21.15.88:6789/0 conn(0x7f986c0d25e0 legacy=0x7f986c0d29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.961+0000 7f986bfff640 1 -- 172.21.15.46:0/1372168755 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986c1610d0 con 0x7f986c0d0b80 2024-05-09T03:12:02.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.961+0000 7f9870919640 1 --2- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.961+0000 7f986bfff640 1 --2- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d0b80 0x7f986c0d0f90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98600097a0 tx=0x7f986002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c997b6c968675205 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f986b7fe640 1 -- 172.21.15.46:0/1372168755 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860037070 con 0x7f986c0d0b80 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f986b7fe640 1 -- 172.21.15.46:0/1372168755 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9860032440 con 0x7f986c0d0b80 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f986b7fe640 1 -- 172.21.15.46:0/1372168755 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860032810 con 0x7f986c0d0b80 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 -- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d0b80 msgr2=0x7f986c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 --2- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d0b80 0x7f986c0d0f90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98600097a0 tx=0x7f986002a180 comp rx=0 tx=0).stop 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 -- 172.21.15.46:0/1372168755 shutdown_connections 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 --2- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d0b80 0x7f986c0d0f90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 --2- 172.21.15.46:0/1372168755 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 -- 172.21.15.46:0/1372168755 >> 172.21.15.46:0/1372168755 conn(0x7f986c06e190 msgr2=0x7f986c06e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 -- 172.21.15.46:0/1372168755 shutdown_connections 2024-05-09T03:12:02.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.962+0000 7f9872b7d640 1 -- 172.21.15.46:0/1372168755 wait complete. 2024-05-09T03:12:02.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.964+0000 7f9872b7d640 1 Processor -- start 2024-05-09T03:12:02.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.964+0000 7f9872b7d640 1 -- start start 2024-05-09T03:12:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9872b7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 0x7f986c0cfc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9872b7d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d25e0 0x7f986c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9872b7d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9872b7d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f986c1640c0 con 0x7f986c0d0b80 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9872b7d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f986c163f40 con 0x7f986c0d25e0 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9872b7d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f986c164240 con 0x7f986c1615f0 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 0x7f986c0cfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 0x7f986c0cfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47898/0 (socket says 172.21.15.46:47898) 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 -- 172.21.15.46:0/3113219938 learned_addr learned my addr 172.21.15.46:0/3113219938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f987111a640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f986bfff640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d25e0 0x7f986c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 -- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 msgr2=0x7f986c0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 -- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d25e0 msgr2=0x7f986c0ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d25e0 0x7f986c0ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.966+0000 7f9870919640 1 -- 172.21.15.46:0/3113219938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986c0d0170 con 0x7f986c0d0b80 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.967+0000 7f9870919640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 0x7f986c0cfc30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f985c00e990 tx=0x7f985c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.967+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c017070 con 0x7f986c0d0b80 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.967+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f985c00c9e0 con 0x7f986c0d0b80 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.967+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986c0c9ec0 con 0x7f986c0d0b80 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.967+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c00cdb0 con 0x7f986c0d0b80 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.967+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f986c0ca480 con 0x7f986c0d0b80 2024-05-09T03:12:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.968+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f986c06a270 con 0x7f986c0d0b80 2024-05-09T03:12:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.968+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f985c01a070 con 0x7f986c0d0b80 2024-05-09T03:12:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.969+0000 7f98697fa640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98580762d0 0x7f98580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.969+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4857+0+0 (secure 0 0 0) 0x7f985c097170 con 0x7f986c0d0b80 2024-05-09T03:12:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.969+0000 7f986bfff640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98580762d0 0x7f98580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.970+0000 7f986bfff640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98580762d0 0x7f98580787e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9860034000 tx=0x7f986002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:02.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:02.974+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f986c0d0170 con 0x7f986c0d0b80 2024-05-09T03:12:03.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.237+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f986c06e850 con 0x7f986c0d0b80 2024-05-09T03:12:03.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.238+0000 7f98697fa640 1 -- 172.21.15.46:0/3113219938 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) ==== 80+0+5304 (secure 0 0 0) 0x7f985c0606c0 con 0x7f986c0d0b80 2024-05-09T03:12:03.238 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g==", 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.239 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw==", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw==", 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.240 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw==", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:12:03.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw==", 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.242 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA==", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg==", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.243 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA==", 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-05-09T03:12:03.244 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g==", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA==", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ==", 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw==", 2024-05-09T03:12:03.246 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw==", 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ==", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg==", 2024-05-09T03:12:03.248 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg==", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:12:03.249 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA1PjxmuVPDJhAAHOgxPpTSEd3pibPqobcHCg==", 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.250 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg==", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ==", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:12:03.251 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:12:03.252 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.252 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.252 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-05-09T03:12:03.252 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:03.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98580762d0 msgr2=0x7f98580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98580762d0 0x7f98580787e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9860034000 tx=0x7f986002e000 comp rx=0 tx=0).stop 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 msgr2=0x7f986c0cfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 0x7f986c0cfc30 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f985c00e990 tx=0x7f985c00ee50 comp rx=0 tx=0).stop 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 shutdown_connections 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f986c1615f0 0x7f986c0cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f986c0d25e0 0x7f986c0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98580762d0 0x7f98580787e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 --2- 172.21.15.46:0/3113219938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f986c0d0b80 0x7f986c0cfc30 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 >> 172.21.15.46:0/3113219938 conn(0x7f986c06e190 msgr2=0x7f986c0b60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 shutdown_connections 2024-05-09T03:12:03.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.240+0000 7f9872b7d640 1 -- 172.21.15.46:0/3113219938 wait complete. 2024-05-09T03:12:03.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:12:03.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.369+0000 7ff124500640 1 Processor -- start 2024-05-09T03:12:03.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.369+0000 7ff124500640 1 -- start start 2024-05-09T03:12:03.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff124500640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff124500640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff11c0b7ed0 0x7ff11c0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff124500640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff11c0d5280 con 0x7ff11c0d4800 2024-05-09T03:12:03.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff124500640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff11c0b5620 con 0x7ff11c0b7ed0 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff124500640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c1609a0 con 0x7ff11c0b5210 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff122a9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff11c0b7ed0 0x7ff11c0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff121a9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff11c0d4800 0x7ff11c0b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40308/0 (socket says 172.21.15.46:40308) 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff121a9b640 1 -- 172.21.15.46:0/526744613 learned_addr learned my addr 172.21.15.46:0/526744613 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff12229c640 1 --2- 172.21.15.46:0/526744613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff122a9d640 1 -- 172.21.15.46:0/526744613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 msgr2=0x7ff11c1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff122a9d640 1 --2- 172.21.15.46:0/526744613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff122a9d640 1 -- 172.21.15.46:0/526744613 >> v1:172.21.15.46:6789/0 conn(0x7ff11c0d4800 legacy=0x7ff11c0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff122a9d640 1 -- 172.21.15.46:0/526744613 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c0d5560 con 0x7ff11c0b7ed0 2024-05-09T03:12:03.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.371+0000 7ff12129a640 1 -- 172.21.15.46:0/526744613 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048017433 0 0) 0x7ff11c0d5280 con 0x7ff11c0d4800 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.372+0000 7ff122a9d640 1 --2- 172.21.15.46:0/526744613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0b7ed0 0x7ff11c0d4c10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff114009d90 tx=0x7ff11402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500170b55d18cda1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.372+0000 7ff12129a640 1 -- 172.21.15.46:0/526744613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114007740 con 0x7ff11c0b7ed0 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.372+0000 7ff12129a640 1 -- 172.21.15.46:0/526744613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1140078e0 con 0x7ff11c0b7ed0 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.372+0000 7ff12129a640 1 -- 172.21.15.46:0/526744613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114007cb0 con 0x7ff11c0b7ed0 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 -- 172.21.15.46:0/526744613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0b7ed0 msgr2=0x7ff11c0d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 --2- 172.21.15.46:0/526744613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0b7ed0 0x7ff11c0d4c10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff114009d90 tx=0x7ff11402a5f0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 -- 172.21.15.46:0/526744613 shutdown_connections 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 --2- 172.21.15.46:0/526744613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0b7ed0 0x7ff11c0d4c10 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 --2- 172.21.15.46:0/526744613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 -- 172.21.15.46:0/526744613 >> 172.21.15.46:0/526744613 conn(0x7ff11c0d29a0 msgr2=0x7ff11c0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:03.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 -- 172.21.15.46:0/526744613 shutdown_connections 2024-05-09T03:12:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.373+0000 7ff124500640 1 -- 172.21.15.46:0/526744613 wait complete. 2024-05-09T03:12:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.375+0000 7ff124500640 1 Processor -- start 2024-05-09T03:12:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.375+0000 7ff124500640 1 -- start start 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff124500640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff124500640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 0x7ff11c0cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff124500640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11c0d5cd0 0x7ff11c0d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff124500640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c0d0ae0 con 0x7ff11c0d4800 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff124500640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff11c160ea0 con 0x7ff11c0d5cd0 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff124500640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff11c163e60 con 0x7ff11c0b5210 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff121a9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 0x7ff11c0cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff121a9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 0x7ff11c0cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47918/0 (socket says 172.21.15.46:47918) 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.376+0000 7ff121a9b640 1 -- 172.21.15.46:0/2571598582 learned_addr learned my addr 172.21.15.46:0/2571598582 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff12229c640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff122a9d640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11c0d5cd0 0x7ff11c0d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff121a9b640 1 -- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 msgr2=0x7ff11c151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff121a9b640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff121a9b640 1 -- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11c0d5cd0 msgr2=0x7ff11c0d60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff121a9b640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11c0d5cd0 0x7ff11c0d60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff121a9b640 1 -- 172.21.15.46:0/2571598582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11c0d6620 con 0x7ff11c0d4800 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff121a9b640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 0x7ff11c0cbfc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00c3c0 tx=0x7ff10c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0074e0 con 0x7ff11c0d4800 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10c0047e0 con 0x7ff11c0d4800 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff11c1e9cc0 con 0x7ff11c0d4800 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c004b90 con 0x7ff11c0d4800 2024-05-09T03:12:03.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.377+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11c1ea130 con 0x7ff11c0d4800 2024-05-09T03:12:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.378+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff11c1e9e90 con 0x7ff11c0d4800 2024-05-09T03:12:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.379+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff10c011070 con 0x7ff11c0d4800 2024-05-09T03:12:03.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.379+0000 7ff112ffd640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff100076300 0x7ff100078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.379+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4857+0+0 (secure 0 0 0) 0x7ff10c096d40 con 0x7ff11c0d4800 2024-05-09T03:12:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.380+0000 7ff12229c640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff100076300 0x7ff100078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.380+0000 7ff12229c640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff100076300 0x7ff100078810 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff118009a30 tx=0x7ff118008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:03.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.384+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff11c1e9e90 con 0x7ff11c0d4800 2024-05-09T03:12:03.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.577+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff11c06a270 con 0x7ff11c0d4800 2024-05-09T03:12:03.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.578+0000 7ff112ffd640 1 -- 172.21.15.46:0/2571598582 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) ==== 84+0+65 (secure 0 0 0) 0x7ff11c0d6620 con 0x7ff11c0d4800 2024-05-09T03:12:03.578 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:03.579 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:12:03.579 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:12:03.579 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:12:03.579 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:12:03.579 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:03.579 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff100076300 msgr2=0x7ff100078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff100076300 0x7ff100078810 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff118009a30 tx=0x7ff118008040 comp rx=0 tx=0).stop 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 msgr2=0x7ff11c0cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 0x7ff11c0cbfc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00c3c0 tx=0x7ff10c00c880 comp rx=0 tx=0).stop 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 shutdown_connections 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff11c0d5cd0 0x7ff11c0d60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff100076300 0x7ff100078810 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff11c0d4800 0x7ff11c0cbfc0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 --2- 172.21.15.46:0/2571598582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff11c0b5210 0x7ff11c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 >> 172.21.15.46:0/2571598582 conn(0x7ff11c0d29a0 msgr2=0x7ff11c0b9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 shutdown_connections 2024-05-09T03:12:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.580+0000 7ff124500640 1 -- 172.21.15.46:0/2571598582 wait complete. 2024-05-09T03:12:03.592 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:12:03.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:12:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.710+0000 7fd6655e3640 1 Processor -- start 2024-05-09T03:12:03.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.710+0000 7fd6655e3640 1 -- start start 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd6655e3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd66006e3a0 0x7fd66006e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd6655e3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd660164810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd6655e3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6600b9420 con 0x7fd660164d50 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd6655e3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd6600d1300 con 0x7fd66006e3a0 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd6655e3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd66009d250 con 0x7fd66006ecf0 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd65effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd66006e3a0 0x7fd66006e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd65e7fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd660164810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd65f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd660164d50 0x7fd6600cebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:48444/0 (socket says 172.21.15.46:48444) 2024-05-09T03:12:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.712+0000 7fd65f7fe640 1 -- 172.21.15.46:0/4250648224 learned_addr learned my addr 172.21.15.46:0/4250648224 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.713+0000 7fd65effd640 1 -- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 msgr2=0x7fd660164810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.713+0000 7fd65effd640 1 --2- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd660164810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.713+0000 7fd65effd640 1 -- 172.21.15.46:0/4250648224 >> v1:172.21.15.88:6789/0 conn(0x7fd660164d50 legacy=0x7fd6600cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.713+0000 7fd65effd640 1 -- 172.21.15.46:0/4250648224 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66006d820 con 0x7fd66006e3a0 2024-05-09T03:12:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.713+0000 7fd65effd640 1 --2- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd66006e3a0 0x7fd66006e7b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6500097a0 tx=0x7fd65002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b803d8c5290c8389 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd65dffb640 1 -- 172.21.15.46:0/4250648224 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650037070 con 0x7fd66006e3a0 2024-05-09T03:12:03.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd65dffb640 1 -- 172.21.15.46:0/4250648224 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65002ae50 con 0x7fd66006e3a0 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd65dffb640 1 -- 172.21.15.46:0/4250648224 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650032660 con 0x7fd66006e3a0 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 -- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd66006e3a0 msgr2=0x7fd66006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 --2- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd66006e3a0 0x7fd66006e7b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd6500097a0 tx=0x7fd65002a100 comp rx=0 tx=0).stop 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 -- 172.21.15.46:0/4250648224 shutdown_connections 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 --2- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd660164810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 --2- 172.21.15.46:0/4250648224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd66006e3a0 0x7fd66006e7b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 -- 172.21.15.46:0/4250648224 >> 172.21.15.46:0/4250648224 conn(0x7fd6600d05d0 msgr2=0x7fd6600d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 -- 172.21.15.46:0/4250648224 shutdown_connections 2024-05-09T03:12:03.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.714+0000 7fd6655e3640 1 -- 172.21.15.46:0/4250648224 wait complete. 2024-05-09T03:12:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.716+0000 7fd6655e3640 1 Processor -- start 2024-05-09T03:12:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.716+0000 7fd6655e3640 1 -- start start 2024-05-09T03:12:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.717+0000 7fd6655e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 0x7fd66014c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.717+0000 7fd6655e3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd6600c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.717+0000 7fd6655e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd660164d50 0x7fd66015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.717+0000 7fd6655e3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6600d2fd0 con 0x7fd66006e3a0 2024-05-09T03:12:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.717+0000 7fd6655e3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd66006de80 con 0x7fd660164d50 2024-05-09T03:12:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.717+0000 7fd6655e3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd6600d3150 con 0x7fd66006ecf0 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 0x7fd66014c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 0x7fd66014c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47922/0 (socket says 172.21.15.46:47922) 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 -- 172.21.15.46:0/1168292346 learned_addr learned my addr 172.21.15.46:0/1168292346 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65f7fe640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd660164d50 0x7fd66015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65e7fc640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd6600c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 -- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 msgr2=0x7fd6600c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd6600c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 -- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd660164d50 msgr2=0x7fd66015b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd660164d50 0x7fd66015b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 -- 172.21.15.46:0/1168292346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66014db90 con 0x7fd66006e3a0 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd65effd640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 0x7fd66014c100 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd6500098c0 tx=0x7fd6500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650041030 con 0x7fd66006e3a0 2024-05-09T03:12:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65002ac50 con 0x7fd66006e3a0 2024-05-09T03:12:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650037040 con 0x7fd66006e3a0 2024-05-09T03:12:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66014de70 con 0x7fd66006e3a0 2024-05-09T03:12:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.718+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66014e430 con 0x7fd66006e3a0 2024-05-09T03:12:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.719+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd66006a270 con 0x7fd66006e3a0 2024-05-09T03:12:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.720+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd650044070 con 0x7fd66006e3a0 2024-05-09T03:12:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.721+0000 7fd6477fe640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd64c0762d0 0x7fd64c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.721+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4857+0+0 (secure 0 0 0) 0x7fd65002ae00 con 0x7fd66006e3a0 2024-05-09T03:12:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.721+0000 7fd65e7fc640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd64c0762d0 0x7fd64c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:03.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.721+0000 7fd65e7fc640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd64c0762d0 0x7fd64c0787e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd66009c550 tx=0x7fd65400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:03.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.725+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd65004b070 con 0x7fd66006e3a0 2024-05-09T03:12:03.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:03.919+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd66009c110 con 0x7fd66006e3a0 2024-05-09T03:12:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.823+0000 7fd6477fe640 1 -- 172.21.15.46:0/1168292346 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v52) ==== 138+0+0 (secure 0 0 0) 0x7fd650081640 con 0x7fd66006e3a0 2024-05-09T03:12:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd64c0762d0 msgr2=0x7fd64c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd64c0762d0 0x7fd64c0787e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd66009c550 tx=0x7fd65400b040 comp rx=0 tx=0).stop 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 msgr2=0x7fd66014c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 0x7fd66014c100 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd6500098c0 tx=0x7fd6500046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 shutdown_connections 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd660164d50 0x7fd66015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd66006ecf0 0x7fd6600c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd64c0762d0 0x7fd64c0787e0 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd66009c550 tx=0x7fd65400b040 comp rx=0 tx=0).stop 2024-05-09T03:12:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 --2- 172.21.15.46:0/1168292346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd66006e3a0 0x7fd66014c100 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 >> 172.21.15.46:0/1168292346 conn(0x7fd6600d05d0 msgr2=0x7fd6600b8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 shutdown_connections 2024-05-09T03:12:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.826+0000 7fd6655e3640 1 -- 172.21.15.46:0/1168292346 wait complete. 2024-05-09T03:12:04.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:12:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.974+0000 7fe85607e640 1 Processor -- start 2024-05-09T03:12:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.974+0000 7fe85607e640 1 -- start start 2024-05-09T03:12:04.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe85607e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe85015bdf0 0x7fe85015e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:04.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe85607e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe850070c60 0x7fe8501623e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe85607e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe85015eea0 con 0x7fe850162920 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe85607e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe85015ec70 con 0x7fe850070c60 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe85607e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe850161f50 con 0x7fe85015bdf0 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe84effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe850070c60 0x7fe8501623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe84ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe850162920 0x7fe8500aeb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40312/0 (socket says 172.21.15.46:40312) 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe84ffff640 1 -- 172.21.15.46:0/1108244058 learned_addr learned my addr 172.21.15.46:0/1108244058 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.975+0000 7fe84f7fe640 1 --2- 172.21.15.46:0/1108244058 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe85015bdf0 0x7fe85015e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.976+0000 7fe84effd640 1 -- 172.21.15.46:0/1108244058 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe85015bdf0 msgr2=0x7fe85015e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.976+0000 7fe84effd640 1 --2- 172.21.15.46:0/1108244058 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe85015bdf0 0x7fe85015e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.976+0000 7fe84effd640 1 -- 172.21.15.46:0/1108244058 >> v1:172.21.15.46:6789/0 conn(0x7fe850162920 legacy=0x7fe8500aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.976+0000 7fe84e7fc640 1 -- 172.21.15.46:0/1108244058 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941346803 0 0) 0x7fe85015eea0 con 0x7fe850162920 2024-05-09T03:12:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.976+0000 7fe84effd640 1 -- 172.21.15.46:0/1108244058 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85015f180 con 0x7fe850070c60 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.976+0000 7fe84effd640 1 --2- 172.21.15.46:0/1108244058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe8501623e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe8440097a0 tx=0x7fe84402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2cab7194a4d6c66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe84e7fc640 1 -- 172.21.15.46:0/1108244058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844037070 con 0x7fe850070c60 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe84e7fc640 1 -- 172.21.15.46:0/1108244058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe84402ae50 con 0x7fe850070c60 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe84e7fc640 1 -- 172.21.15.46:0/1108244058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844032660 con 0x7fe850070c60 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 -- 172.21.15.46:0/1108244058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 msgr2=0x7fe8501623e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 --2- 172.21.15.46:0/1108244058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe8501623e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fe8440097a0 tx=0x7fe84402a100 comp rx=0 tx=0).stop 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 -- 172.21.15.46:0/1108244058 shutdown_connections 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 --2- 172.21.15.46:0/1108244058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe8501623e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 --2- 172.21.15.46:0/1108244058 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe85015bdf0 0x7fe85015e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 -- 172.21.15.46:0/1108244058 >> 172.21.15.46:0/1108244058 conn(0x7fe8500d1980 msgr2=0x7fe8500d1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 -- 172.21.15.46:0/1108244058 shutdown_connections 2024-05-09T03:12:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.977+0000 7fe85607e640 1 -- 172.21.15.46:0/1108244058 wait complete. 2024-05-09T03:12:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.979+0000 7fe85607e640 1 Processor -- start 2024-05-09T03:12:04.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.979+0000 7fe85607e640 1 -- start start 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe85607e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe85014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe85607e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85015bdf0 0x7fe850159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe85607e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe850162920 0x7fe85015d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe85607e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8500af220 con 0x7fe850070c60 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe85607e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe850071070 con 0x7fe85015bdf0 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe85607e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe8500af3a0 con 0x7fe850162920 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe85014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe85014eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47934/0 (socket says 172.21.15.46:47934) 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 -- 172.21.15.46:0/3697902593 learned_addr learned my addr 172.21.15.46:0/3697902593 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84effd640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85015bdf0 0x7fe850159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84ffff640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe850162920 0x7fe85015d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 -- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe850162920 msgr2=0x7fe85015d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe850162920 0x7fe85015d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 -- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85015bdf0 msgr2=0x7fe850159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85015bdf0 0x7fe850159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 -- 172.21.15.46:0/3697902593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8500cadb0 con 0x7fe850070c60 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84effd640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85015bdf0 0x7fe850159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84ffff640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe850162920 0x7fe85015d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.981+0000 7fe84f7fe640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe85014eb40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe840009c90 tx=0x7fe84000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.982+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840018020 con 0x7fe850070c60 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.982+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe840014480 con 0x7fe850070c60 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.982+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8500cb2e0 con 0x7fe850070c60 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.982+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840014850 con 0x7fe850070c60 2024-05-09T03:12:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.982+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8500cb8a0 con 0x7fe850070c60 2024-05-09T03:12:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.983+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe85006a270 con 0x7fe850070c60 2024-05-09T03:12:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.983+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe840005d10 con 0x7fe850070c60 2024-05-09T03:12:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.984+0000 7fe837fff640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe83c076300 0x7fe83c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.984+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5227+0+0 (secure 0 0 0) 0x7fe840098240 con 0x7fe850070c60 2024-05-09T03:12:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.984+0000 7fe84effd640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe83c076300 0x7fe83c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.985+0000 7fe84effd640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe83c076300 0x7fe83c078810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe844034000 tx=0x7fe84402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:04.989+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe840022030 con 0x7fe850070c60 2024-05-09T03:12:05.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.181+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe85015afa0 con 0x7fe850070c60 2024-05-09T03:12:05.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.825+0000 7fe837fff640 1 -- 172.21.15.46:0/3697902593 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v53) ==== 170+0+0 (secure 0 0 0) 0x7fe840010040 con 0x7fe850070c60 2024-05-09T03:12:05.827 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:12:05.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe83c076300 msgr2=0x7fe83c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe83c076300 0x7fe83c078810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe844034000 tx=0x7fe84402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 msgr2=0x7fe85014eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe85014eb40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe840009c90 tx=0x7fe84000c490 comp rx=0 tx=0).stop 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 shutdown_connections 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe850162920 0x7fe85015d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe85015bdf0 0x7fe850159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe83c076300 0x7fe83c078810 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 --2- 172.21.15.46:0/3697902593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe850070c60 0x7fe85014eb40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.829+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 >> 172.21.15.46:0/3697902593 conn(0x7fe8500d1980 msgr2=0x7fe8500af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:05.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.830+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 shutdown_connections 2024-05-09T03:12:05.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.830+0000 7fe85607e640 1 -- 172.21.15.46:0/3697902593 wait complete. 2024-05-09T03:12:05.859 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:12:05.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.983+0000 7f1f718c8640 1 Processor -- start 2024-05-09T03:12:05.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.983+0000 7f1f718c8640 1 -- start start 2024-05-09T03:12:05.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.984+0000 7f1f718c8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f6c0b4700 0x7f1f6c0d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.984+0000 7f1f718c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.984+0000 7f1f718c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6c06ebe0 con 0x7f1f6c0d2630 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.984+0000 7f1f718c8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1f6c0b4530 con 0x7f1f6c0b4700 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.984+0000 7f1f718c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6c06eec0 con 0x7f1f6c0b4120 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f6c0b4700 0x7f1f6c0d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f6c0b4700 0x7f1f6c0d20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53126/0 (socket says 172.21.15.46:53126) 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 -- 172.21.15.46:0/149446472 learned_addr learned my addr 172.21.15.46:0/149446472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6b7fe640 1 --2- 172.21.15.46:0/149446472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 -- 172.21.15.46:0/149446472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 msgr2=0x7f1f6c0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 --2- 172.21.15.46:0/149446472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c0ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 -- 172.21.15.46:0/149446472 >> v1:172.21.15.88:6789/0 conn(0x7f1f6c0d2630 legacy=0x7f1f6c0b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.985+0000 7f1f6affd640 1 -- 172.21.15.46:0/149446472 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c06f1a0 con 0x7f1f6c0b4700 2024-05-09T03:12:05.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.986+0000 7f1f6affd640 1 --2- 172.21.15.46:0/149446472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c0d20f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009870 tx=0x7f1f5c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390c827be463e9a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:05.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.986+0000 7f1f6a7fc640 1 -- 172.21.15.46:0/149446472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c037070 con 0x7f1f6c0b4700 2024-05-09T03:12:05.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.986+0000 7f1f6a7fc640 1 -- 172.21.15.46:0/149446472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f5c032440 con 0x7f1f6c0b4700 2024-05-09T03:12:05.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.986+0000 7f1f6a7fc640 1 -- 172.21.15.46:0/149446472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c032810 con 0x7f1f6c0b4700 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.986+0000 7f1f718c8640 1 -- 172.21.15.46:0/149446472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0b4700 msgr2=0x7f1f6c0d20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.986+0000 7f1f718c8640 1 --2- 172.21.15.46:0/149446472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c0d20f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009870 tx=0x7f1f5c02a130 comp rx=0 tx=0).stop 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.987+0000 7f1f718c8640 1 -- 172.21.15.46:0/149446472 shutdown_connections 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.987+0000 7f1f718c8640 1 --2- 172.21.15.46:0/149446472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.987+0000 7f1f718c8640 1 --2- 172.21.15.46:0/149446472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c0d20f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.987+0000 7f1f718c8640 1 -- 172.21.15.46:0/149446472 >> 172.21.15.46:0/149446472 conn(0x7f1f6c0b55d0 msgr2=0x7f1f6c0b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.987+0000 7f1f718c8640 1 -- 172.21.15.46:0/149446472 shutdown_connections 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.987+0000 7f1f718c8640 1 -- 172.21.15.46:0/149446472 wait complete. 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.988+0000 7f1f718c8640 1 Processor -- start 2024-05-09T03:12:05.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.988+0000 7f1f718c8640 1 -- start start 2024-05-09T03:12:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f718c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c09e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f718c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c09e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f718c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 0x7f1f6c1fa380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f718c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c0d2a40 con 0x7f1f6c0d2630 2024-05-09T03:12:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f718c8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c0d2db0 con 0x7f1f6c0b4700 2024-05-09T03:12:05.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f718c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6c0b51d0 con 0x7f1f6c0b4120 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 0x7f1f6c1fa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 0x7f1f6c1fa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53140/0 (socket says 172.21.15.46:53140) 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 -- 172.21.15.46:0/2250555359 learned_addr learned my addr 172.21.15.46:0/2250555359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6affd640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c09e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f63fff640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c09e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 -- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 msgr2=0x7f1f6c09e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c09e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 -- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f6c0b4700 msgr2=0x7f1f6c09e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c09e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 -- 172.21.15.46:0/2250555359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c1f6e10 con 0x7f1f6c0d2630 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.990+0000 7f1f6b7fe640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 0x7f1f6c1fa380 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400cc30 tx=0x7f1f6400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.991+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6400ce40 con 0x7f1f6c0d2630 2024-05-09T03:12:05.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.991+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f64003990 con 0x7f1f6c0d2630 2024-05-09T03:12:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.991+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6c09f910 con 0x7f1f6c0d2630 2024-05-09T03:12:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.991+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64003d40 con 0x7f1f6c0d2630 2024-05-09T03:12:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.991+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6c09fed0 con 0x7f1f6c0d2630 2024-05-09T03:12:05.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.991+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6c06a270 con 0x7f1f6c0d2630 2024-05-09T03:12:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.992+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1f64003ee0 con 0x7f1f6c0d2630 2024-05-09T03:12:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.993+0000 7f1f68ff9640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f4c076300 0x7f1f4c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.993+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5609+0+0 (secure 0 0 0) 0x7f1f64097e60 con 0x7f1f6c0d2630 2024-05-09T03:12:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.993+0000 7f1f6affd640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f4c076300 0x7f1f4c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:05.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.994+0000 7f1f6affd640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f4c076300 0x7f1f4c078810 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c034000 tx=0x7f1f5c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:05.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:05.997+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1f64014030 con 0x7f1f6c0d2630 2024-05-09T03:12:06.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:12:06.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:12:06.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.229+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1f6c0720b0 con 0x7f1f6c0d2630 2024-05-09T03:12:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.853+0000 7f1f68ff9640 1 -- 172.21.15.46:0/2250555359 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v42) ==== 172+0+0 (secure 0 0 0) 0x7f1f64061120 con 0x7f1f6c0d2630 2024-05-09T03:12:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 8 and data pool 9 2024-05-09T03:12:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.860+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f4c076300 msgr2=0x7f1f4c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:06.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.860+0000 7f1f718c8640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f4c076300 0x7f1f4c078810 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c034000 tx=0x7f1f5c02e000 comp rx=0 tx=0).stop 2024-05-09T03:12:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.860+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 msgr2=0x7f1f6c1fa380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.860+0000 7f1f718c8640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 0x7f1f6c1fa380 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f1f6400cc30 tx=0x7f1f6400a5c0 comp rx=0 tx=0).stop 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.861+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 shutdown_connections 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.861+0000 7f1f718c8640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f4c076300 0x7f1f4c078810 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.861+0000 7f1f718c8640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f6c0d2630 0x7f1f6c1fa380 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.861+0000 7f1f718c8640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f6c0b4700 0x7f1f6c09e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.861+0000 7f1f718c8640 1 --2- 172.21.15.46:0/2250555359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f6c0b4120 0x7f1f6c09e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.861+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 >> 172.21.15.46:0/2250555359 conn(0x7f1f6c0b55d0 msgr2=0x7f1f6c08cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.862+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 shutdown_connections 2024-05-09T03:12:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:06.862+0000 7f1f718c8640 1 -- 172.21.15.46:0/2250555359 wait complete. 2024-05-09T03:12:06.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:07.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.009+0000 7f4f7380d640 1 Processor -- start 2024-05-09T03:12:07.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.009+0000 7f4f7380d640 1 -- start start 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f7380d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f6c0cf6b0 0x7f4f6c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f7380d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6c0d2b80 con 0x7f4f6c0d0b90 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f7380d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f6c0d2e60 con 0x7f4f6c0b92e0 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f7380d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4f6c0d3140 con 0x7f4f6c0cf6b0 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f70da8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f6c0d0b90 0x7f4f6c06d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48574/0 (socket says 172.21.15.46:48574) 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f70da8640 1 -- 172.21.15.46:0/4012298362 learned_addr learned my addr 172.21.15.46:0/4012298362 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f71daa640 1 --2- 172.21.15.46:0/4012298362 >> v2:172.21.15.88:3300/0 conn(0x7f4f6c0cf6b0 0x7f4f6c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932598811 0 0) 0x7f4f6c0d2b80 con 0x7f4f6c0d0b90 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f54003520 con 0x7f4f6c0d0b90 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645399522 0 0) 0x7f4f54003520 con 0x7f4f6c0d0b90 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 >> v2:172.21.15.88:3300/0 conn(0x7f4f6c0cf6b0 msgr2=0x7f4f6c0b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:07.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 --2- 172.21.15.46:0/4012298362 >> v2:172.21.15.88:3300/0 conn(0x7f4f6c0cf6b0 0x7f4f6c0b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 >> v1:172.21.15.88:6789/0 conn(0x7f4f6c0b92e0 legacy=0x7f4f6c0d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.011+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c0d3420 con 0x7f4f6c0d0b90 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f4f600016f0 con 0x7f4f6c0d0b90 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 >> v1:172.21.15.46:6789/0 conn(0x7f4f6c0d0b90 legacy=0x7f4f6c06d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f54003a50 0x7f4f54003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f54004800 0x7f4f54004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54005350 0x7f4f5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c0d3140 con 0x7f4f54005350 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f5400c280 con 0x7f4f54003a50 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5400c480 con 0x7f4f54004800 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f71daa640 1 --2- 172.21.15.46:0/4012298362 >> v2:172.21.15.88:3300/0 conn(0x7f4f6c0cf6b0 0x7f4f6c0b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f715a9640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f54003a50 0x7f4f54003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f71daa640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54005350 0x7f4f5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f70da8640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f54004800 0x7f4f54004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f715a9640 1 -- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f54004800 msgr2=0x7f4f54004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f715a9640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f54004800 0x7f4f54004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f715a9640 1 -- 172.21.15.46:0/4012298362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54005350 msgr2=0x7f4f5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f715a9640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54005350 0x7f4f5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f715a9640 1 -- 172.21.15.46:0/4012298362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f54003520 con 0x7f4f54003a50 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f71daa640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54005350 0x7f4f5400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.012+0000 7f4f71daa640 1 -- 172.21.15.46:0/4012298362 reap_dead start 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.013+0000 7f4f715a9640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f54003a50 0x7f4f54003ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c001e60 tx=0x7f4f54005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3336920c3a8354f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.013+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c001ab0 con 0x7f4f54003a50 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.013+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c02add0 con 0x7f4f54003a50 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.013+0000 7f4f6bfff640 1 -- 172.21.15.46:0/4012298362 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c003b70 con 0x7f4f54003a50 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 -- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f54003a50 msgr2=0x7f4f54003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f54003a50 0x7f4f54003ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c001e60 tx=0x7f4f54005830 comp rx=0 tx=0).stop 2024-05-09T03:12:07.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 -- 172.21.15.46:0/4012298362 shutdown_connections 2024-05-09T03:12:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 --2- 172.21.15.46:0/4012298362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f54003a50 0x7f4f54003ec0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 -- 172.21.15.46:0/4012298362 >> 172.21.15.46:0/4012298362 conn(0x7f4f6c0d3c00 msgr2=0x7f4f6c0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 -- 172.21.15.46:0/4012298362 shutdown_connections 2024-05-09T03:12:07.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.014+0000 7f4f7380d640 1 -- 172.21.15.46:0/4012298362 wait complete. 2024-05-09T03:12:07.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.016+0000 7f4f7380d640 1 Processor -- start 2024-05-09T03:12:07.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.016+0000 7f4f7380d640 1 -- start start 2024-05-09T03:12:07.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f7380d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 0x7f4f6c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f7380d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c1618f0 0x7f4f6c0b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f7380d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f6c161d00 0x7f4f6c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f7380d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c0b5530 con 0x7f4f54003a50 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f7380d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c0b53b0 con 0x7f4f6c1618f0 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f7380d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6c0d0480 con 0x7f4f6c161d00 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f715a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 0x7f4f6c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f715a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 0x7f4f6c153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53158/0 (socket says 172.21.15.46:53158) 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f715a9640 1 -- 172.21.15.46:0/1013259851 learned_addr learned my addr 172.21.15.46:0/1013259851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f71daa640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f6c161d00 0x7f4f6c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.017+0000 7f4f70da8640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c1618f0 0x7f4f6c0b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f715a9640 1 -- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f6c161d00 msgr2=0x7f4f6c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f715a9640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f6c161d00 0x7f4f6c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f715a9640 1 -- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c1618f0 msgr2=0x7f4f6c0b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f715a9640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c1618f0 0x7f4f6c0b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f715a9640 1 -- 172.21.15.46:0/1013259851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c160280 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f715a9640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 0x7f4f6c153ee0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c001f80 tx=0x7f4f5c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c042070 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c005500 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6c0b5750 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c037040 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.018+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f6c1627b0 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.019+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f6c06a270 con 0x7f4f54003a50 2024-05-09T03:12:07.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.020+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4f5c045070 con 0x7f4f54003a50 2024-05-09T03:12:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.020+0000 7f4f69ffb640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f580762d0 0x7f4f580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.021+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5709+0+0 (secure 0 0 0) 0x7f4f5c0b7420 con 0x7f4f54003a50 2024-05-09T03:12:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.021+0000 7f4f70da8640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f580762d0 0x7f4f580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.021+0000 7f4f70da8640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f580762d0 0x7f4f580787e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c1f9960 tx=0x7f4f60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.025+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4f5c080c50 con 0x7f4f54003a50 2024-05-09T03:12:07.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.215+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f6c06d830 con 0x7f4f54003a50 2024-05-09T03:12:07.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.216+0000 7f4f69ffb640 1 -- 172.21.15.46:0/1013259851 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+16606 (secure 0 0 0) 0x7f4f5c080550 con 0x7f4f54003a50 2024-05-09T03:12:07.217 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:07.217 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:12:06.230117+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:12:03.923965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T03:12:05.185724+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f580762d0 msgr2=0x7f4f580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f580762d0 0x7f4f580787e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c1f9960 tx=0x7f4f60009040 comp rx=0 tx=0).stop 2024-05-09T03:12:07.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 msgr2=0x7f4f6c153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 0x7f4f6c153ee0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c001f80 tx=0x7f4f5c0014a0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 shutdown_connections 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f6c161d00 0x7f4f6c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c1618f0 0x7f4f6c0b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f580762d0 0x7f4f580787e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 --2- 172.21.15.46:0/1013259851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f54003a50 0x7f4f6c153ee0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.218+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 >> 172.21.15.46:0/1013259851 conn(0x7f4f6c0d3c00 msgr2=0x7f4f6c0b8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.219+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 shutdown_connections 2024-05-09T03:12:07.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.219+0000 7f4f7380d640 1 -- 172.21.15.46:0/1013259851 wait complete. 2024-05-09T03:12:07.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.345+0000 7f2fddd14640 1 Processor -- start 2024-05-09T03:12:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.345+0000 7f2fddd14640 1 -- start start 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fddd14640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fd8164c70 0x7f2fd81615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fddd14640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2fd815dfe0 0x7f2fd80d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fddd14640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd8160b20 con 0x7f2fd80d2480 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fddd14640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd8161150 con 0x7f2fd8164c70 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fddd14640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd80d15f0 con 0x7f2fd815dfe0 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fd77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fd8164c70 0x7f2fd81615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fd7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2fd80d2480 0x7f2fd80d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48590/0 (socket says 172.21.15.46:48590) 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.346+0000 7f2fd7fff640 1 -- 172.21.15.46:0/768848512 learned_addr learned my addr 172.21.15.46:0/768848512 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.347+0000 7f2fd6ffd640 1 --2- 172.21.15.46:0/768848512 >> v2:172.21.15.88:3300/0 conn(0x7f2fd815dfe0 0x7f2fd80d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.347+0000 7f2fd77fe640 1 -- 172.21.15.46:0/768848512 >> v2:172.21.15.88:3300/0 conn(0x7f2fd815dfe0 msgr2=0x7f2fd80d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.347+0000 7f2fd77fe640 1 --2- 172.21.15.46:0/768848512 >> v2:172.21.15.88:3300/0 conn(0x7f2fd815dfe0 0x7f2fd80d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.347+0000 7f2fd77fe640 1 -- 172.21.15.46:0/768848512 >> v1:172.21.15.46:6789/0 conn(0x7f2fd80d2480 legacy=0x7f2fd80d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.347+0000 7f2fd77fe640 1 -- 172.21.15.46:0/768848512 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd80d18d0 con 0x7f2fd8164c70 2024-05-09T03:12:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.347+0000 7f2fd67fc640 1 -- 172.21.15.46:0/768848512 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 356220518 0 0) 0x7f2fd8160b20 con 0x7f2fd80d2480 2024-05-09T03:12:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fd77fe640 1 --2- 172.21.15.46:0/768848512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd8164c70 0x7f2fd81615e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80cce00 tx=0x7f2fc802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24267f84a662522e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fd67fc640 1 -- 172.21.15.46:0/768848512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8037070 con 0x7f2fd8164c70 2024-05-09T03:12:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fd67fc640 1 -- 172.21.15.46:0/768848512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc8032440 con 0x7f2fd8164c70 2024-05-09T03:12:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fd67fc640 1 -- 172.21.15.46:0/768848512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc802ae60 con 0x7f2fd8164c70 2024-05-09T03:12:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fddd14640 1 -- 172.21.15.46:0/768848512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd8164c70 msgr2=0x7f2fd81615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fddd14640 1 --2- 172.21.15.46:0/768848512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd8164c70 0x7f2fd81615e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80cce00 tx=0x7f2fc802a110 comp rx=0 tx=0).stop 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fddd14640 1 -- 172.21.15.46:0/768848512 shutdown_connections 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fddd14640 1 --2- 172.21.15.46:0/768848512 >> v2:172.21.15.88:3300/0 conn(0x7f2fd815dfe0 0x7f2fd80d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fddd14640 1 --2- 172.21.15.46:0/768848512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd8164c70 0x7f2fd81615e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.348+0000 7f2fddd14640 1 -- 172.21.15.46:0/768848512 >> 172.21.15.46:0/768848512 conn(0x7f2fd8073db0 msgr2=0x7f2fd80741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.349+0000 7f2fddd14640 1 -- 172.21.15.46:0/768848512 shutdown_connections 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.349+0000 7f2fddd14640 1 -- 172.21.15.46:0/768848512 wait complete. 2024-05-09T03:12:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.350+0000 7f2fddd14640 1 Processor -- start 2024-05-09T03:12:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.351+0000 7f2fddd14640 1 -- start start 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fddd14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80d2480 0x7f2fd81f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fddd14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 0x7f2fd80b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fddd14640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fd8164c70 0x7f2fd815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fddd14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd815e3f0 con 0x7f2fd815dfe0 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fddd14640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd80d1300 con 0x7f2fd80d2480 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fddd14640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd80d45c0 con 0x7f2fd8164c70 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 0x7f2fd80b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 0x7f2fd80b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53182/0 (socket says 172.21.15.46:53182) 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 -- 172.21.15.46:0/2563944725 learned_addr learned my addr 172.21.15.46:0/2563944725 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd77fe640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80d2480 0x7f2fd81f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd7fff640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fd8164c70 0x7f2fd815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 -- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fd8164c70 msgr2=0x7f2fd815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fd8164c70 0x7f2fd815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 -- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80d2480 msgr2=0x7f2fd81f6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80d2480 0x7f2fd81f6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 -- 172.21.15.46:0/2563944725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd80b8550 con 0x7f2fd815dfe0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.352+0000 7f2fd6ffd640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 0x7f2fd80b8010 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00e870 tx=0x7f2fcc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.353+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc017070 con 0x7f2fd815dfe0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.353+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fcc00c940 con 0x7f2fd815dfe0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.353+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd80ce310 con 0x7f2fd815dfe0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.353+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00cd30 con 0x7f2fd815dfe0 2024-05-09T03:12:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.353+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd80ce8d0 con 0x7f2fd815dfe0 2024-05-09T03:12:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.353+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd80ce080 con 0x7f2fd815dfe0 2024-05-09T03:12:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.354+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2fcc01a070 con 0x7f2fd815dfe0 2024-05-09T03:12:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.355+0000 7f2fbffff640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fc4076300 0x7f2fc4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.355+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5709+0+0 (secure 0 0 0) 0x7f2fcc097b40 con 0x7f2fd815dfe0 2024-05-09T03:12:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.355+0000 7f2fd77fe640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fc4076300 0x7f2fc4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.356+0000 7f2fd77fe640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fc4076300 0x7f2fc4078810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8034000 tx=0x7f2fc802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.359+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2fcc061580 con 0x7f2fd815dfe0 2024-05-09T03:12:07.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.550+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2fd808cfd0 con 0x7f2fd815dfe0 2024-05-09T03:12:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.551+0000 7f2fbffff640 1 -- 172.21.15.46:0/2563944725 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+16606 (secure 0 0 0) 0x7f2fd808cfd0 con 0x7f2fd815dfe0 2024-05-09T03:12:07.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:07.552 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:12:06.230117+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:12:03.923965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T03:12:05.185724+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fc4076300 msgr2=0x7f2fc4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fc4076300 0x7f2fc4078810 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8034000 tx=0x7f2fc802e000 comp rx=0 tx=0).stop 2024-05-09T03:12:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 msgr2=0x7f2fd80b8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 0x7f2fd80b8010 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc00e870 tx=0x7f2fcc00ed30 comp rx=0 tx=0).stop 2024-05-09T03:12:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 shutdown_connections 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fd8164c70 0x7f2fd815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fc4076300 0x7f2fc4078810 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fd815dfe0 0x7f2fd80b8010 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 --2- 172.21.15.46:0/2563944725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fd80d2480 0x7f2fd81f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 >> 172.21.15.46:0/2563944725 conn(0x7f2fd8073db0 msgr2=0x7f2fd80baf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 shutdown_connections 2024-05-09T03:12:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.553+0000 7f2fddd14640 1 -- 172.21.15.46:0/2563944725 wait complete. 2024-05-09T03:12:07.565 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:12:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.683+0000 7fb9a5cf9640 1 Processor -- start 2024-05-09T03:12:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.683+0000 7fb9a5cf9640 1 -- start start 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.684+0000 7fb9a5cf9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb9a00b7e70 0x7fb9a00d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.684+0000 7fb9a5cf9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a00b5550 con 0x7fb9a00b5140 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb9a5cf9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a00d44e0 con 0x7fb9a00d3df0 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb9a5cf9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a00d3480 con 0x7fb9a00b7e70 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9a00b5140 0x7fb9a00b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48594/0 (socket says 172.21.15.46:48594) 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99f7fe640 1 -- 172.21.15.46:0/632064682 learned_addr learned my addr 172.21.15.46:0/632064682 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99effd640 1 --2- 172.21.15.46:0/632064682 >> v2:172.21.15.46:3300/0 conn(0x7fb9a00b7e70 0x7fb9a00d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99e7fc640 1 -- 172.21.15.46:0/632064682 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2551908862 0 0) 0x7fb9a00b5550 con 0x7fb9a00b5140 2024-05-09T03:12:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99e7fc640 1 -- 172.21.15.46:0/632064682 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb988003520 con 0x7fb9a00b5140 2024-05-09T03:12:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99effd640 1 -- 172.21.15.46:0/632064682 >> v1:172.21.15.88:6789/0 conn(0x7fb9a00d3df0 legacy=0x7fb9a00b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99effd640 1 -- 172.21.15.46:0/632064682 >> v1:172.21.15.46:6789/0 conn(0x7fb9a00b5140 legacy=0x7fb9a00b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.685+0000 7fb99effd640 1 -- 172.21.15.46:0/632064682 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a00d47c0 con 0x7fb9a00b7e70 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.686+0000 7fb99effd640 1 --2- 172.21.15.46:0/632064682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b7e70 0x7fb9a00d38b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb9940097a0 tx=0x7fb99402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0cee876dcf0964 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.686+0000 7fb99e7fc640 1 -- 172.21.15.46:0/632064682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994037070 con 0x7fb9a00b7e70 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.686+0000 7fb99e7fc640 1 -- 172.21.15.46:0/632064682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb994032640 con 0x7fb9a00b7e70 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.686+0000 7fb99e7fc640 1 -- 172.21.15.46:0/632064682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9940329f0 con 0x7fb9a00b7e70 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/632064682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b7e70 msgr2=0x7fb9a00d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/632064682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b7e70 0x7fb9a00d38b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb9940097a0 tx=0x7fb99402a2c0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/632064682 shutdown_connections 2024-05-09T03:12:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/632064682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b7e70 0x7fb9a00d38b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/632064682 >> 172.21.15.46:0/632064682 conn(0x7fb9a008d3f0 msgr2=0x7fb9a008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/632064682 shutdown_connections 2024-05-09T03:12:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.687+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/632064682 wait complete. 2024-05-09T03:12:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.689+0000 7fb9a5cf9640 1 Processor -- start 2024-05-09T03:12:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.689+0000 7fb9a5cf9640 1 -- start start 2024-05-09T03:12:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb9a5cf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 0x7fb9a01f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb9a5cf9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9a00d3df0 0x7fb9a00c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb9a5cf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9a0156b20 0x7fb9a0162340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb9a5cf9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a00d4200 con 0x7fb9a00b5140 2024-05-09T03:12:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb9a5cf9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a00d4d80 con 0x7fb9a00d3df0 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb9a5cf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a00d5580 con 0x7fb9a0156b20 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb99f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 0x7fb9a01f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb99f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 0x7fb9a01f62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53192/0 (socket says 172.21.15.46:53192) 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb99f7fe640 1 -- 172.21.15.46:0/816620191 learned_addr learned my addr 172.21.15.46:0/816620191 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb99effd640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9a00d3df0 0x7fb9a00c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.690+0000 7fb99ffff640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9a0156b20 0x7fb9a0162340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99f7fe640 1 -- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9a0156b20 msgr2=0x7fb9a0162340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99f7fe640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9a0156b20 0x7fb9a0162340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99f7fe640 1 -- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9a00d3df0 msgr2=0x7fb9a00c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99f7fe640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9a00d3df0 0x7fb9a00c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99f7fe640 1 -- 172.21.15.46:0/816620191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a015a150 con 0x7fb9a00b5140 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99effd640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9a00d3df0 0x7fb9a00c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb99f7fe640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 0x7fb9a01f62c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb99000c400 tx=0x7fb99000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990017070 con 0x7fb9a00b5140 2024-05-09T03:12:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb990005d00 con 0x7fb9a00b5140 2024-05-09T03:12:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a015a430 con 0x7fb9a00b5140 2024-05-09T03:12:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990007310 con 0x7fb9a00b5140 2024-05-09T03:12:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.691+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a015a9f0 con 0x7fb9a00b5140 2024-05-09T03:12:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.692+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a006a270 con 0x7fb9a00b5140 2024-05-09T03:12:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.693+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb99001a070 con 0x7fb9a00b5140 2024-05-09T03:12:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.693+0000 7fb987fff640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb98c076300 0x7fb98c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.694+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5709+0+0 (secure 0 0 0) 0x7fb990096db0 con 0x7fb9a00b5140 2024-05-09T03:12:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.694+0000 7fb99effd640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb98c076300 0x7fb98c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.694+0000 7fb99effd640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb98c076300 0x7fb98c078810 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb994034000 tx=0x7fb99402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:07.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.697+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb990021020 con 0x7fb9a00b5140 2024-05-09T03:12:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:07.931+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb9a0164580 con 0x7fb9a00b5140 2024-05-09T03:12:08.821 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:12:08.820+0000 7fe170391640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:12:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.864+0000 7fb987fff640 1 -- 172.21.15.46:0/816620191 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) ==== 118+0+0 (secure 0 0 0) 0x7fb990060760 con 0x7fb9a00b5140 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb98c076300 msgr2=0x7fb98c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb98c076300 0x7fb98c078810 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb994034000 tx=0x7fb99402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 msgr2=0x7fb9a01f62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 0x7fb9a01f62c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb99000c400 tx=0x7fb99000c8c0 comp rx=0 tx=0).stop 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 shutdown_connections 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9a0156b20 0x7fb9a0162340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9a00d3df0 0x7fb9a00c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb98c076300 0x7fb98c078810 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 --2- 172.21.15.46:0/816620191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a00b5140 0x7fb9a01f62c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 >> 172.21.15.46:0/816620191 conn(0x7fb9a008d3f0 msgr2=0x7fb9a00b7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 shutdown_connections 2024-05-09T03:12:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:08.868+0000 7fb9a5cf9640 1 -- 172.21.15.46:0/816620191 wait complete. 2024-05-09T03:12:08.883 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:12:08.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:12:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.007+0000 7f7f362d9640 1 Processor -- start 2024-05-09T03:12:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.007+0000 7f7f362d9640 1 -- start start 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.008+0000 7f7f362d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f300d3180 0x7f7f301642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.008+0000 7f7f362d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f30164900 con 0x7f7f30163eb0 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f362d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f30164be0 con 0x7f7f300bb770 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f362d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7f300d4bb0 con 0x7f7f300d3180 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f27fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7f30163eb0 0x7f7f300d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48602/0 (socket says 172.21.15.46:48602) 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f27fff640 1 -- 172.21.15.46:0/3283879107 learned_addr learned my addr 172.21.15.46:0/3283879107 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f34876640 1 --2- 172.21.15.46:0/3283879107 >> v2:172.21.15.88:3300/0 conn(0x7f7f300d3180 0x7f7f301642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601643423 0 0) 0x7f7f30164900 con 0x7f7f30163eb0 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f1c003520 con 0x7f7f30163eb0 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2283571030 0 0) 0x7f7f30164be0 con 0x7f7f300bb770 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f30164900 con 0x7f7f300bb770 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f34876640 1 -- 172.21.15.46:0/3283879107 >> v1:172.21.15.88:6789/0 conn(0x7f7f300bb770 legacy=0x7f7f300d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f34876640 1 -- 172.21.15.46:0/3283879107 >> v1:172.21.15.46:6789/0 conn(0x7f7f30163eb0 legacy=0x7f7f300d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.009+0000 7f7f34876640 1 -- 172.21.15.46:0/3283879107 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f300d4840 con 0x7f7f300d3180 2024-05-09T03:12:09.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.010+0000 7f7f34876640 1 --2- 172.21.15.46:0/3283879107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300d3180 0x7f7f301642c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7f300d4d30 tx=0x7f7f2802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5bc299151ac402a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:09.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.010+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28007740 con 0x7f7f300d3180 2024-05-09T03:12:09.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.010+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f280078e0 con 0x7f7f300d3180 2024-05-09T03:12:09.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.010+0000 7f7f2f7fe640 1 -- 172.21.15.46:0/3283879107 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28007cb0 con 0x7f7f300d3180 2024-05-09T03:12:09.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 -- 172.21.15.46:0/3283879107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300d3180 msgr2=0x7f7f301642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 --2- 172.21.15.46:0/3283879107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300d3180 0x7f7f301642c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7f300d4d30 tx=0x7f7f2802a6f0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 -- 172.21.15.46:0/3283879107 shutdown_connections 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 --2- 172.21.15.46:0/3283879107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300d3180 0x7f7f301642c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 -- 172.21.15.46:0/3283879107 >> 172.21.15.46:0/3283879107 conn(0x7f7f3006e370 msgr2=0x7f7f3006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 -- 172.21.15.46:0/3283879107 shutdown_connections 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.011+0000 7f7f362d9640 1 -- 172.21.15.46:0/3283879107 wait complete. 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.013+0000 7f7f362d9640 1 Processor -- start 2024-05-09T03:12:09.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.013+0000 7f7f362d9640 1 -- start start 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f362d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300bb770 0x7f7f301f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f362d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f30163eb0 0x7f7f3015de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f362d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 0x7f7f3015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f362d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f300d0d40 con 0x7f7f301526f0 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f362d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7f300d0bc0 con 0x7f7f300bb770 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f362d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7f300b9210 con 0x7f7f30163eb0 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f34876640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 0x7f7f3015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f34876640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 0x7f7f3015e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53208/0 (socket says 172.21.15.46:53208) 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f34876640 1 -- 172.21.15.46:0/942153488 learned_addr learned my addr 172.21.15.46:0/942153488 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:09.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f2ffff640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300bb770 0x7f7f301f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.014+0000 7f7f27fff640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f30163eb0 0x7f7f3015de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f34876640 1 -- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f30163eb0 msgr2=0x7f7f3015de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f34876640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f30163eb0 0x7f7f3015de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f34876640 1 -- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300bb770 msgr2=0x7f7f301f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f34876640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300bb770 0x7f7f301f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f34876640 1 -- 172.21.15.46:0/942153488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f30152f90 con 0x7f7f301526f0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f34876640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 0x7f7f3015e360 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009e50 tx=0x7f7f28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2803f070 con 0x7f7f301526f0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f28031bf0 con 0x7f7f301526f0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f3014cc40 con 0x7f7f301526f0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28009450 con 0x7f7f301526f0 2024-05-09T03:12:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.015+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f300cf040 con 0x7f7f301526f0 2024-05-09T03:12:09.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.016+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f300cedb0 con 0x7f7f301526f0 2024-05-09T03:12:09.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.017+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7f28003990 con 0x7f7f301526f0 2024-05-09T03:12:09.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.017+0000 7f7f2d7fa640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f10076300 0x7f7f10078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:09.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.018+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f7f280b87b0 con 0x7f7f301526f0 2024-05-09T03:12:09.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.018+0000 7f7f2ffff640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f10076300 0x7f7f10078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.018+0000 7f7f2ffff640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f10076300 0x7f7f10078810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7f20001120 tx=0x7f7f20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.021+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7f30152f90 con 0x7f7f301526f0 2024-05-09T03:12:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.249+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f7f3006ecb0 con 0x7f7f301526f0 2024-05-09T03:12:09.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.865+0000 7f7f2d7fa640 1 -- 172.21.15.46:0/942153488 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7f7f3006ecb0 con 0x7f7f301526f0 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f10076300 msgr2=0x7f7f10078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f10076300 0x7f7f10078810 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7f20001120 tx=0x7f7f20009040 comp rx=0 tx=0).stop 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 msgr2=0x7f7f3015e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 0x7f7f3015e360 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7f28009e50 tx=0x7f7f28004a40 comp rx=0 tx=0).stop 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 shutdown_connections 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f10076300 0x7f7f10078810 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f301526f0 0x7f7f3015e360 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f30163eb0 0x7f7f3015de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 --2- 172.21.15.46:0/942153488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f300bb770 0x7f7f301f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 >> 172.21.15.46:0/942153488 conn(0x7f7f3006e370 msgr2=0x7f7f300ba840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 shutdown_connections 2024-05-09T03:12:09.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:09.868+0000 7f7f362d9640 1 -- 172.21.15.46:0/942153488 wait complete. 2024-05-09T03:12:09.884 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:12:09.884 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:10.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.005+0000 7f965c540640 1 Processor -- start 2024-05-09T03:12:10.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.005+0000 7f965c540640 1 -- start start 2024-05-09T03:12:10.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965c540640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96541615e0 0x7f965415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965c540640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96540d2480 0x7f96540d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965c540640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9654160b20 con 0x7f9654164c70 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965c540640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f965415e520 con 0x7f96540d2480 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965c540640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9654161150 con 0x7f96541615e0 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965aadd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96540d2480 0x7f96540d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965aadd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96540d2480 0x7f96540d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53224/0 (socket says 172.21.15.46:53224) 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.006+0000 7f965aadd640 1 -- 172.21.15.46:0/1745276063 learned_addr learned my addr 172.21.15.46:0/1745276063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.007+0000 7f9659adb640 1 --2- 172.21.15.46:0/1745276063 >> v2:172.21.15.88:3300/0 conn(0x7f96541615e0 0x7f965415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.007+0000 7f965aadd640 1 -- 172.21.15.46:0/1745276063 >> v2:172.21.15.88:3300/0 conn(0x7f96541615e0 msgr2=0x7f965415dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.007+0000 7f965aadd640 1 --2- 172.21.15.46:0/1745276063 >> v2:172.21.15.88:3300/0 conn(0x7f96541615e0 0x7f965415dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.007+0000 7f965aadd640 1 -- 172.21.15.46:0/1745276063 >> v1:172.21.15.88:6789/0 conn(0x7f9654164c70 legacy=0x7f96540d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.007+0000 7f965aadd640 1 -- 172.21.15.46:0/1745276063 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96540d15f0 con 0x7f96540d2480 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.008+0000 7f965aadd640 1 --2- 172.21.15.46:0/1745276063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96540d2480 0x7f96540d2890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f96540cce00 tx=0x7f964c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dec6695defcaa1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.008+0000 7f96592da640 1 -- 172.21.15.46:0/1745276063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c036070 con 0x7f96540d2480 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.008+0000 7f96592da640 1 -- 172.21.15.46:0/1745276063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f964c031600 con 0x7f96540d2480 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.008+0000 7f96592da640 1 -- 172.21.15.46:0/1745276063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c0319d0 con 0x7f96540d2480 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.008+0000 7f965c540640 1 -- 172.21.15.46:0/1745276063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96540d2480 msgr2=0x7f96540d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.008+0000 7f965c540640 1 --2- 172.21.15.46:0/1745276063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96540d2480 0x7f96540d2890 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f96540cce00 tx=0x7f964c02a6a0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.009+0000 7f965c540640 1 -- 172.21.15.46:0/1745276063 shutdown_connections 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.009+0000 7f965c540640 1 --2- 172.21.15.46:0/1745276063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96540d2480 0x7f96540d2890 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.009+0000 7f965c540640 1 --2- 172.21.15.46:0/1745276063 >> v2:172.21.15.88:3300/0 conn(0x7f96541615e0 0x7f965415dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.009+0000 7f965c540640 1 -- 172.21.15.46:0/1745276063 >> 172.21.15.46:0/1745276063 conn(0x7f9654073db0 msgr2=0x7f96540741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:10.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.009+0000 7f965c540640 1 -- 172.21.15.46:0/1745276063 shutdown_connections 2024-05-09T03:12:10.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.009+0000 7f965c540640 1 -- 172.21.15.46:0/1745276063 wait complete. 2024-05-09T03:12:10.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.010+0000 7f965c540640 1 Processor -- start 2024-05-09T03:12:10.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.011+0000 7f965c540640 1 -- start start 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965c540640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96540d2480 0x7f96540cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965c540640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96541615e0 0x7f96540c8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965c540640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 0x7f965415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965c540640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96541619f0 con 0x7f9654164c70 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965c540640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9654160de0 con 0x7f96541615e0 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965c540640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96540d2dd0 con 0x7f96540d2480 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 0x7f965415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 0x7f965415bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53234/0 (socket says 172.21.15.46:53234) 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 -- 172.21.15.46:0/3512377321 learned_addr learned my addr 172.21.15.46:0/3512377321 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f9659adb640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96541615e0 0x7f96540c8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965a2dc640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96540d2480 0x7f96540cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 -- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96540d2480 msgr2=0x7f96540cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96540d2480 0x7f96540cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 -- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96541615e0 msgr2=0x7f96540c8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96541615e0 0x7f96540c8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 -- 172.21.15.46:0/3512377321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9654148b90 con 0x7f9654164c70 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965a2dc640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96540d2480 0x7f96540cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.012+0000 7f965aadd640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 0x7f965415bcc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f964c009f40 tx=0x7f964c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.013+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c03f020 con 0x7f9654164c70 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.013+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f964c037040 con 0x7f9654164c70 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.013+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96540c98b0 con 0x7f9654164c70 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.013+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c036070 con 0x7f9654164c70 2024-05-09T03:12:10.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.013+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96540c9e70 con 0x7f9654164c70 2024-05-09T03:12:10.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.014+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96540cc340 con 0x7f9654164c70 2024-05-09T03:12:10.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.015+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f964c042020 con 0x7f9654164c70 2024-05-09T03:12:10.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.015+0000 7f964affd640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9638076300 0x7f9638078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.015+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f964c0b7d80 con 0x7f9654164c70 2024-05-09T03:12:10.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.015+0000 7f965a2dc640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9638076300 0x7f9638078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.016+0000 7f965a2dc640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9638076300 0x7f9638078810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9650007fd0 tx=0x7f9650009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.020+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f964c049020 con 0x7f9654164c70 2024-05-09T03:12:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.249+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9654148d60 con 0x7f9654164c70 2024-05-09T03:12:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.250+0000 7f964affd640 1 -- 172.21.15.46:0/3512377321 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5181 (secure 0 0 0) 0x7f964c081650 con 0x7f9654164c70 2024-05-09T03:12:10.252 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:10.252 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:09.251952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/4263404372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":4263404372},{"type":"v1","addr":"172.21.15.46:6835","nonce":4263404372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5874,"qdb_cluster":[5874]},"id":4}]} 2024-05-09T03:12:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 46 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9638076300 msgr2=0x7f9638078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9638076300 0x7f9638078810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9650007fd0 tx=0x7f9650009040 comp rx=0 tx=0).stop 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 msgr2=0x7f965415bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 0x7f965415bcc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f964c009f40 tx=0x7f964c004a40 comp rx=0 tx=0).stop 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 shutdown_connections 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9638076300 0x7f9638078810 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9654164c70 0x7f965415bcc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96541615e0 0x7f96540c8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 --2- 172.21.15.46:0/3512377321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96540d2480 0x7f96540cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 >> 172.21.15.46:0/3512377321 conn(0x7f9654073db0 msgr2=0x7f96540ba240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:10.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 shutdown_connections 2024-05-09T03:12:10.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.252+0000 7f965c540640 1 -- 172.21.15.46:0/3512377321 wait complete. 2024-05-09T03:12:10.265 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:12:10.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.378+0000 7f523c62b640 1 Processor -- start 2024-05-09T03:12:10.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.378+0000 7f523c62b640 1 -- start start 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523c62b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340b5210 0x7f52341613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523c62b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340d5280 con 0x7f52340b7ed0 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523c62b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340d5560 con 0x7f52340d4800 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523c62b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52340b5620 con 0x7f52340b5210 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523abc8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52340b7ed0 0x7f5234163e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48610/0 (socket says 172.21.15.46:48610) 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523abc8640 1 -- 172.21.15.46:0/2777056493 learned_addr learned my addr 172.21.15.46:0/2777056493 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f523a3c7640 1 --2- 172.21.15.46:0/2777056493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340b5210 0x7f52341613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630706893 0 0) 0x7f52340d5280 con 0x7f52340b7ed0 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.380+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5220003520 con 0x7f52340b7ed0 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.381+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405680402 0 0) 0x7f52340d5560 con 0x7f52340d4800 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.381+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52340d5280 con 0x7f52340d4800 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.381+0000 7f523a3c7640 1 -- 172.21.15.46:0/2777056493 >> v1:172.21.15.88:6789/0 conn(0x7f52340d4800 legacy=0x7f52340b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.381+0000 7f523a3c7640 1 -- 172.21.15.46:0/2777056493 >> v1:172.21.15.46:6789/0 conn(0x7f52340b7ed0 legacy=0x7f5234163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.381+0000 7f523a3c7640 1 -- 172.21.15.46:0/2777056493 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5234160bc0 con 0x7f52340b5210 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.381+0000 7f523a3c7640 1 --2- 172.21.15.46:0/2777056493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340b5210 0x7f52341613c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5230009dc0 tx=0x7f523002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57025d0fdc4f7679 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.382+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230007740 con 0x7f52340b5210 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.382+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52300078e0 con 0x7f52340b5210 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.382+0000 7f52393c5640 1 -- 172.21.15.46:0/2777056493 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230007cb0 con 0x7f52340b5210 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.382+0000 7f523c62b640 1 -- 172.21.15.46:0/2777056493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340b5210 msgr2=0x7f52341613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.382+0000 7f523c62b640 1 --2- 172.21.15.46:0/2777056493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340b5210 0x7f52341613c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5230009dc0 tx=0x7f523002a6d0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.383+0000 7f523c62b640 1 -- 172.21.15.46:0/2777056493 shutdown_connections 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.383+0000 7f523c62b640 1 --2- 172.21.15.46:0/2777056493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340b5210 0x7f52341613c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.383+0000 7f523c62b640 1 -- 172.21.15.46:0/2777056493 >> 172.21.15.46:0/2777056493 conn(0x7f52340d29a0 msgr2=0x7f52340d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.383+0000 7f523c62b640 1 -- 172.21.15.46:0/2777056493 shutdown_connections 2024-05-09T03:12:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.383+0000 7f523c62b640 1 -- 172.21.15.46:0/2777056493 wait complete. 2024-05-09T03:12:10.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.384+0000 7f523c62b640 1 Processor -- start 2024-05-09T03:12:10.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.384+0000 7f523c62b640 1 -- start start 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523c62b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 0x7f5234159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523c62b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340d4800 0x7f52340ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523c62b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523415be10 0x7f52340cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523c62b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52340d0ae0 con 0x7f52340b5210 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523c62b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52341609a0 con 0x7f523415be10 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523c62b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52340d4c10 con 0x7f52340d4800 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 0x7f5234159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 0x7f5234159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53240/0 (socket says 172.21.15.46:53240) 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 -- 172.21.15.46:0/1365506242 learned_addr learned my addr 172.21.15.46:0/1365506242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f5239bc6640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340d4800 0x7f52340ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523abc8640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523415be10 0x7f52340cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 -- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340d4800 msgr2=0x7f52340ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340d4800 0x7f52340ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 -- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523415be10 msgr2=0x7f52340cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523415be10 0x7f52340cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 -- 172.21.15.46:0/1365506242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52341e9e60 con 0x7f52340b5210 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523abc8640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523415be10 0x7f52340cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f523a3c7640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 0x7f5234159100 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5230009dc0 tx=0x7f5230004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.386+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230040070 con 0x7f52340b5210 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.387+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5230031bc0 con 0x7f52340b5210 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.387+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52341ea140 con 0x7f52340b5210 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.387+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230009450 con 0x7f52340b5210 2024-05-09T03:12:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.387+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52340cb9d0 con 0x7f52340b5210 2024-05-09T03:12:10.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.388+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52340b4a90 con 0x7f52340b5210 2024-05-09T03:12:10.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.388+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5230039040 con 0x7f52340b5210 2024-05-09T03:12:10.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.389+0000 7f522affd640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52180762d0 0x7f52180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.389+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f52300b83b0 con 0x7f52340b5210 2024-05-09T03:12:10.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.389+0000 7f5239bc6640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52180762d0 0x7f52180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.389+0000 7f5239bc6640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52180762d0 0x7f52180787e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52340c95e0 tx=0x7f5224009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.394+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f52340b4a90 con 0x7f52340b5210 2024-05-09T03:12:10.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.589+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f523406a270 con 0x7f52340b5210 2024-05-09T03:12:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.590+0000 7f522affd640 1 -- 172.21.15.46:0/1365506242 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16593 (secure 0 0 0) 0x7f523004b070 con 0x7f52340b5210 2024-05-09T03:12:10.592 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:10.592 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":55,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:12:07.843968+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:12:03.923965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T03:12:05.185724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:12:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.592+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52180762d0 msgr2=0x7f52180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.592+0000 7f523c62b640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52180762d0 0x7f52180787e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f52340c95e0 tx=0x7f5224009040 comp rx=0 tx=0).stop 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 msgr2=0x7f5234159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 0x7f5234159100 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5230009dc0 tx=0x7f5230004a40 comp rx=0 tx=0).stop 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 shutdown_connections 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523415be10 0x7f52340cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52340d4800 0x7f52340ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52180762d0 0x7f52180787e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 --2- 172.21.15.46:0/1365506242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52340b5210 0x7f5234159100 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 >> 172.21.15.46:0/1365506242 conn(0x7f52340d29a0 msgr2=0x7f52340ba0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 shutdown_connections 2024-05-09T03:12:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.593+0000 7f523c62b640 1 -- 172.21.15.46:0/1365506242 wait complete. 2024-05-09T03:12:10.606 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:12:10.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:12:10.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.719+0000 7fe4760f9640 1 Processor -- start 2024-05-09T03:12:10.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.719+0000 7fe4760f9640 1 -- start start 2024-05-09T03:12:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe4760f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47008d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe4760f9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe47008d6e0 0x7fe4700d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe4760f9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4700b4ee0 con 0x7fe4700d20a0 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe4760f9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe4700b5460 con 0x7fe47008d6e0 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe4760f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4700cfac0 con 0x7fe4700b9200 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe46effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe47008d6e0 0x7fe4700d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe46ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4700d20a0 0x7fe470162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48618/0 (socket says 172.21.15.46:48618) 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.720+0000 7fe46ffff640 1 -- 172.21.15.46:0/1911098926 learned_addr learned my addr 172.21.15.46:0/1911098926 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46f7fe640 1 --2- 172.21.15.46:0/1911098926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47008d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46e7fc640 1 -- 172.21.15.46:0/1911098926 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147123749 0 0) 0x7fe4700b4ee0 con 0x7fe4700d20a0 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46e7fc640 1 -- 172.21.15.46:0/1911098926 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe458003520 con 0x7fe4700d20a0 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46effd640 1 -- 172.21.15.46:0/1911098926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 msgr2=0x7fe47008d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46effd640 1 --2- 172.21.15.46:0/1911098926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47008d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46effd640 1 -- 172.21.15.46:0/1911098926 >> v1:172.21.15.46:6789/0 conn(0x7fe4700d20a0 legacy=0x7fe470162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.721+0000 7fe46effd640 1 -- 172.21.15.46:0/1911098926 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4700b62e0 con 0x7fe47008d6e0 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe46effd640 1 --2- 172.21.15.46:0/1911098926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe4700d1b60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe4640097a0 tx=0x7fe46402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d211f213e8139dc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe46e7fc640 1 -- 172.21.15.46:0/1911098926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe464037070 con 0x7fe47008d6e0 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe46e7fc640 1 -- 172.21.15.46:0/1911098926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe464032440 con 0x7fe47008d6e0 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe46e7fc640 1 -- 172.21.15.46:0/1911098926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe464032810 con 0x7fe47008d6e0 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe4760f9640 1 -- 172.21.15.46:0/1911098926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 msgr2=0x7fe4700d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe4760f9640 1 --2- 172.21.15.46:0/1911098926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe4700d1b60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe4640097a0 tx=0x7fe46402a180 comp rx=0 tx=0).stop 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe4760f9640 1 -- 172.21.15.46:0/1911098926 shutdown_connections 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe4760f9640 1 --2- 172.21.15.46:0/1911098926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe4700d1b60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe4760f9640 1 --2- 172.21.15.46:0/1911098926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47008d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.722+0000 7fe4760f9640 1 -- 172.21.15.46:0/1911098926 >> 172.21.15.46:0/1911098926 conn(0x7fe4700b56e0 msgr2=0x7fe4700b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.723+0000 7fe4760f9640 1 -- 172.21.15.46:0/1911098926 shutdown_connections 2024-05-09T03:12:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.723+0000 7fe4760f9640 1 -- 172.21.15.46:0/1911098926 wait complete. 2024-05-09T03:12:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.724+0000 7fe4760f9640 1 Processor -- start 2024-05-09T03:12:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.724+0000 7fe4760f9640 1 -- start start 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe4760f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe470152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe4760f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe4760f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4700d20a0 0x7fe47015d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe4760f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4700cf750 con 0x7fe47008d6e0 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe4760f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe4700cf470 con 0x7fe4700d20a0 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe4760f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4700b4a20 con 0x7fe4700b9200 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe470152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe470152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53252/0 (socket says 172.21.15.46:53252) 2024-05-09T03:12:10.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 -- 172.21.15.46:0/345420912 learned_addr learned my addr 172.21.15.46:0/345420912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46effd640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46ffff640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4700d20a0 0x7fe47015d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 -- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 msgr2=0x7fe47006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47006e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 -- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4700d20a0 msgr2=0x7fe47015d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4700d20a0 0x7fe47015d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 -- 172.21.15.46:0/345420912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe470156490 con 0x7fe47008d6e0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.726+0000 7fe46f7fe640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe470152ff0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe46000d5f0 tx=0x7fe46000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.727+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4600125f0 con 0x7fe47008d6e0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.727+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe460019070 con 0x7fe47008d6e0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.727+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4701553f0 con 0x7fe47008d6e0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.727+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4600129d0 con 0x7fe47008d6e0 2024-05-09T03:12:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.727+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4701559b0 con 0x7fe47008d6e0 2024-05-09T03:12:10.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.728+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47006a270 con 0x7fe47008d6e0 2024-05-09T03:12:10.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.729+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe460012b70 con 0x7fe47008d6e0 2024-05-09T03:12:10.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.730+0000 7fe457fff640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe45c0762d0 0x7fe45c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:10.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.730+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fe460098310 con 0x7fe47008d6e0 2024-05-09T03:12:10.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.730+0000 7fe46effd640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe45c0762d0 0x7fe45c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:10.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.730+0000 7fe46effd640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe45c0762d0 0x7fe45c0787e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe464034000 tx=0x7fe46402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.734+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe460061700 con 0x7fe47008d6e0 2024-05-09T03:12:10.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.921+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe470156070 con 0x7fe45c0762d0 2024-05-09T03:12:10.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.927+0000 7fe457fff640 1 -- 172.21.15.46:0/345420912 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7fe470156070 con 0x7fe45c0762d0 2024-05-09T03:12:10.932 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:10.934 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe45c0762d0 msgr2=0x7fe45c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe45c0762d0 0x7fe45c0787e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe464034000 tx=0x7fe46402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 msgr2=0x7fe470152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe470152ff0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe46000d5f0 tx=0x7fe46000dab0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 shutdown_connections 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4700d20a0 0x7fe47015d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4700b9200 0x7fe47006e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe45c0762d0 0x7fe45c0787e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 --2- 172.21.15.46:0/345420912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe47008d6e0 0x7fe470152ff0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.930+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 >> 172.21.15.46:0/345420912 conn(0x7fe4700b56e0 msgr2=0x7fe4700b7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.931+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 shutdown_connections 2024-05-09T03:12:10.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:10.931+0000 7fe4760f9640 1 -- 172.21.15.46:0/345420912 wait complete. 2024-05-09T03:12:10.944 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-05-09T03:12:10.531012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238908,"kb_used_data":6048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735812,"statfs":{"total":773094113280,"available":772849471488,"internally_reserved":0,"allocated":6193152,"data_stored":3226534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.706220"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841057+0000","last_change":"2024-05-09T03:12:05.841057+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841057+0000","last_undegraded":"2024-05-09T03:12:05.841057+0000","last_fullsized":"2024-05-09T03:12:05.841057+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836609+0000","last_change":"2024-05-09T03:12:05.836609+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836609+0000","last_undegraded":"2024-05-09T03:12:05.836609+0000","last_fullsized":"2024-05-09T03:12:05.836609+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837976+0000","last_change":"2024-05-09T03:12:05.837976+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837976+0000","last_undegraded":"2024-05-09T03:12:05.837976+0000","last_fullsized":"2024-05-09T03:12:05.837976+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.843259+0000","last_change":"2024-05-09T03:12:05.843259+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.843259+0000","last_undegraded":"2024-05-09T03:12:05.843259+0000","last_fullsized":"2024-05-09T03:12:05.843259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839531+0000","last_change":"2024-05-09T03:12:05.839531+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839531+0000","last_undegraded":"2024-05-09T03:12:05.839531+0000","last_fullsized":"2024-05-09T03:12:05.839531+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834452+0000","last_change":"2024-05-09T03:12:05.834452+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834452+0000","last_undegraded":"2024-05-09T03:12:05.834452+0000","last_fullsized":"2024-05-09T03:12:05.834452+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836296+0000","last_change":"2024-05-09T03:12:05.836296+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836296+0000","last_undegraded":"2024-05-09T03:12:05.836296+0000","last_fullsized":"2024-05-09T03:12:05.836296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835798+0000","last_change":"2024-05-09T03:12:05.835798+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835798+0000","last_undegraded":"2024-05-09T03:12:05.835798+0000","last_fullsized":"2024-05-09T03:12:05.835798+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837939+0000","last_change":"2024-05-09T03:12:05.837939+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837939+0000","last_undegraded":"2024-05-09T03:12:05.837939+0000","last_fullsized":"2024-05-09T03:12:05.837939+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833750+0000","last_change":"2024-05-09T03:12:05.833750+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833750+0000","last_undegraded":"2024-05-09T03:12:05.833750+0000","last_fullsized":"2024-05-09T03:12:05.833750+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833154+0000","last_change":"2024-05-09T03:12:05.833154+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833154+0000","last_undegraded":"2024-05-09T03:12:05.833154+0000","last_fullsized":"2024-05-09T03:12:05.833154+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842732+0000","last_change":"2024-05-09T03:12:05.842732+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842732+0000","last_undegraded":"2024-05-09T03:12:05.842732+0000","last_fullsized":"2024-05-09T03:12:05.842732+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833533+0000","last_change":"2024-05-09T03:12:05.833533+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833533+0000","last_undegraded":"2024-05-09T03:12:05.833533+0000","last_fullsized":"2024-05-09T03:12:05.833533+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836888+0000","last_change":"2024-05-09T03:12:05.836888+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836888+0000","last_undegraded":"2024-05-09T03:12:05.836888+0000","last_fullsized":"2024-05-09T03:12:05.836888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835706+0000","last_change":"2024-05-09T03:12:05.835706+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835706+0000","last_undegraded":"2024-05-09T03:12:05.835706+0000","last_fullsized":"2024-05-09T03:12:05.835706+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833981+0000","last_change":"2024-05-09T03:12:05.833981+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833981+0000","last_undegraded":"2024-05-09T03:12:05.833981+0000","last_fullsized":"2024-05-09T03:12:05.833981+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840026+0000","last_change":"2024-05-09T03:12:05.840026+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840026+0000","last_undegraded":"2024-05-09T03:12:05.840026+0000","last_fullsized":"2024-05-09T03:12:05.840026+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838628+0000","last_change":"2024-05-09T03:12:05.838628+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838628+0000","last_undegraded":"2024-05-09T03:12:05.838628+0000","last_fullsized":"2024-05-09T03:12:05.838628+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839714+0000","last_change":"2024-05-09T03:12:05.839714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839714+0000","last_undegraded":"2024-05-09T03:12:05.839714+0000","last_fullsized":"2024-05-09T03:12:05.839714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837245+0000","last_change":"2024-05-09T03:12:05.837245+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837245+0000","last_undegraded":"2024-05-09T03:12:05.837245+0000","last_fullsized":"2024-05-09T03:12:05.837245+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838405+0000","last_change":"2024-05-09T03:12:05.838405+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838405+0000","last_undegraded":"2024-05-09T03:12:05.838405+0000","last_fullsized":"2024-05-09T03:12:05.838405+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837199+0000","last_change":"2024-05-09T03:12:05.837199+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837199+0000","last_undegraded":"2024-05-09T03:12:05.837199+0000","last_fullsized":"2024-05-09T03:12:05.837199+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836251+0000","last_change":"2024-05-09T03:12:05.836251+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836251+0000","last_undegraded":"2024-05-09T03:12:05.836251+0000","last_fullsized":"2024-05-09T03:12:05.836251+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834399+0000","last_change":"2024-05-09T03:12:05.834399+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834399+0000","last_undegraded":"2024-05-09T03:12:05.834399+0000","last_fullsized":"2024-05-09T03:12:05.834399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840127+0000","last_change":"2024-05-09T03:12:05.840127+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840127+0000","last_undegraded":"2024-05-09T03:12:05.840127+0000","last_fullsized":"2024-05-09T03:12:05.840127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.844753+0000","last_change":"2024-05-09T03:12:05.844753+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.844753+0000","last_undegraded":"2024-05-09T03:12:05.844753+0000","last_fullsized":"2024-05-09T03:12:05.844753+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836714+0000","last_change":"2024-05-09T03:12:05.836714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836714+0000","last_undegraded":"2024-05-09T03:12:05.836714+0000","last_fullsized":"2024-05-09T03:12:05.836714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842327+0000","last_change":"2024-05-09T03:12:05.842327+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842327+0000","last_undegraded":"2024-05-09T03:12:05.842327+0000","last_fullsized":"2024-05-09T03:12:05.842327+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837936+0000","last_change":"2024-05-09T03:12:05.837936+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837936+0000","last_undegraded":"2024-05-09T03:12:05.837936+0000","last_fullsized":"2024-05-09T03:12:05.837936+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838187+0000","last_change":"2024-05-09T03:12:05.838187+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838187+0000","last_undegraded":"2024-05-09T03:12:05.838187+0000","last_fullsized":"2024-05-09T03:12:05.838187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840098+0000","last_change":"2024-05-09T03:12:05.840098+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840098+0000","last_undegraded":"2024-05-09T03:12:05.840098+0000","last_fullsized":"2024-05-09T03:12:05.840098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832938+0000","last_change":"2024-05-09T03:12:05.832938+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832938+0000","last_undegraded":"2024-05-09T03:12:05.832938+0000","last_fullsized":"2024-05-09T03:12:05.832938+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837077+0000","last_change":"2024-05-09T03:12:05.837077+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837077+0000","last_undegraded":"2024-05-09T03:12:05.837077+0000","last_fullsized":"2024-05-09T03:12:05.837077+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852354+0000","last_change":"2024-05-09T03:12:05.849076+0000","last_active":"2024-05-09T03:12:05.852354+0000","last_peered":"2024-05-09T03:12:05.852354+0000","last_clean":"2024-05-09T03:12:05.852354+0000","last_became_active":"2024-05-09T03:12:05.848768+0000","last_became_peered":"2024-05-09T03:12:05.848768+0000","last_unstale":"2024-05-09T03:12:05.852354+0000","last_undegraded":"2024-05-09T03:12:05.852354+0000","last_fullsized":"2024-05-09T03:12:05.852354+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:05.791451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832809+0000","last_change":"2024-05-09T03:12:05.832809+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832809+0000","last_undegraded":"2024-05-09T03:12:05.832809+0000","last_fullsized":"2024-05-09T03:12:05.832809+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851128+0000","last_change":"2024-05-09T03:12:05.846715+0000","last_active":"2024-05-09T03:12:05.851128+0000","last_peered":"2024-05-09T03:12:05.851128+0000","last_clean":"2024-05-09T03:12:05.851128+0000","last_became_active":"2024-05-09T03:12:05.846448+0000","last_became_peered":"2024-05-09T03:12:05.846448+0000","last_unstale":"2024-05-09T03:12:05.851128+0000","last_undegraded":"2024-05-09T03:12:05.851128+0000","last_fullsized":"2024-05-09T03:12:05.851128+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:53:45.376505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833746+0000","last_change":"2024-05-09T03:12:05.833746+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833746+0000","last_undegraded":"2024-05-09T03:12:05.833746+0000","last_fullsized":"2024-05-09T03:12:05.833746+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849635+0000","last_change":"2024-05-09T03:12:05.845815+0000","last_active":"2024-05-09T03:12:05.849635+0000","last_peered":"2024-05-09T03:12:05.849635+0000","last_clean":"2024-05-09T03:12:05.849635+0000","last_became_active":"2024-05-09T03:12:05.845509+0000","last_became_peered":"2024-05-09T03:12:05.845509+0000","last_unstale":"2024-05-09T03:12:05.849635+0000","last_undegraded":"2024-05-09T03:12:05.849635+0000","last_fullsized":"2024-05-09T03:12:05.849635+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:07.083261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836173+0000","last_change":"2024-05-09T03:12:05.836173+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836173+0000","last_undegraded":"2024-05-09T03:12:05.836173+0000","last_fullsized":"2024-05-09T03:12:05.836173+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853894+0000","last_change":"2024-05-09T03:12:05.849601+0000","last_active":"2024-05-09T03:12:05.853894+0000","last_peered":"2024-05-09T03:12:05.853894+0000","last_clean":"2024-05-09T03:12:05.853894+0000","last_became_active":"2024-05-09T03:12:05.849230+0000","last_became_peered":"2024-05-09T03:12:05.849230+0000","last_unstale":"2024-05-09T03:12:05.853894+0000","last_undegraded":"2024-05-09T03:12:05.853894+0000","last_fullsized":"2024-05-09T03:12:05.853894+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:43.363860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835661+0000","last_change":"2024-05-09T03:12:05.835661+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835661+0000","last_undegraded":"2024-05-09T03:12:05.835661+0000","last_fullsized":"2024-05-09T03:12:05.835661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.856188+0000","last_change":"2024-05-09T03:12:05.853186+0000","last_active":"2024-05-09T03:12:05.856188+0000","last_peered":"2024-05-09T03:12:05.856188+0000","last_clean":"2024-05-09T03:12:05.856188+0000","last_became_active":"2024-05-09T03:12:05.852815+0000","last_became_peered":"2024-05-09T03:12:05.852815+0000","last_unstale":"2024-05-09T03:12:05.856188+0000","last_undegraded":"2024-05-09T03:12:05.856188+0000","last_fullsized":"2024-05-09T03:12:05.856188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838497+0000","last_change":"2024-05-09T03:12:05.838497+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838497+0000","last_undegraded":"2024-05-09T03:12:05.838497+0000","last_fullsized":"2024-05-09T03:12:05.838497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849662+0000","last_change":"2024-05-09T03:12:05.844443+0000","last_active":"2024-05-09T03:12:05.849662+0000","last_peered":"2024-05-09T03:12:05.849662+0000","last_clean":"2024-05-09T03:12:05.849662+0000","last_became_active":"2024-05-09T03:12:05.844119+0000","last_became_peered":"2024-05-09T03:12:05.844119+0000","last_unstale":"2024-05-09T03:12:05.849662+0000","last_undegraded":"2024-05-09T03:12:05.849662+0000","last_fullsized":"2024-05-09T03:12:05.849662+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833532+0000","last_change":"2024-05-09T03:12:05.833532+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833532+0000","last_undegraded":"2024-05-09T03:12:05.833532+0000","last_fullsized":"2024-05-09T03:12:05.833532+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852899+0000","last_change":"2024-05-09T03:12:05.847498+0000","last_active":"2024-05-09T03:12:05.852899+0000","last_peered":"2024-05-09T03:12:05.852899+0000","last_clean":"2024-05-09T03:12:05.852899+0000","last_became_active":"2024-05-09T03:12:05.847170+0000","last_became_peered":"2024-05-09T03:12:05.847170+0000","last_unstale":"2024-05-09T03:12:05.852899+0000","last_undegraded":"2024-05-09T03:12:05.852899+0000","last_fullsized":"2024-05-09T03:12:05.852899+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835683+0000","last_change":"2024-05-09T03:12:05.835683+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835683+0000","last_undegraded":"2024-05-09T03:12:05.835683+0000","last_fullsized":"2024-05-09T03:12:05.835683+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853539+0000","last_change":"2024-05-09T03:12:05.847308+0000","last_active":"2024-05-09T03:12:05.853539+0000","last_peered":"2024-05-09T03:12:05.853539+0000","last_clean":"2024-05-09T03:12:05.853539+0000","last_became_active":"2024-05-09T03:12:05.846406+0000","last_became_peered":"2024-05-09T03:12:05.846406+0000","last_unstale":"2024-05-09T03:12:05.853539+0000","last_undegraded":"2024-05-09T03:12:05.853539+0000","last_fullsized":"2024-05-09T03:12:05.853539+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838513+0000","last_change":"2024-05-09T03:12:05.838513+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838513+0000","last_undegraded":"2024-05-09T03:12:05.838513+0000","last_fullsized":"2024-05-09T03:12:05.838513+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851570+0000","last_change":"2024-05-09T03:12:05.847459+0000","last_active":"2024-05-09T03:12:05.851570+0000","last_peered":"2024-05-09T03:12:05.851570+0000","last_clean":"2024-05-09T03:12:05.851570+0000","last_became_active":"2024-05-09T03:12:05.847106+0000","last_became_peered":"2024-05-09T03:12:05.847106+0000","last_unstale":"2024-05-09T03:12:05.851570+0000","last_undegraded":"2024-05-09T03:12:05.851570+0000","last_fullsized":"2024-05-09T03:12:05.851570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835774+0000","last_change":"2024-05-09T03:12:05.835774+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835774+0000","last_undegraded":"2024-05-09T03:12:05.835774+0000","last_fullsized":"2024-05-09T03:12:05.835774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852846+0000","last_change":"2024-05-09T03:12:05.849037+0000","last_active":"2024-05-09T03:12:05.852846+0000","last_peered":"2024-05-09T03:12:05.852846+0000","last_clean":"2024-05-09T03:12:05.852846+0000","last_became_active":"2024-05-09T03:12:05.848283+0000","last_became_peered":"2024-05-09T03:12:05.848283+0000","last_unstale":"2024-05-09T03:12:05.852846+0000","last_undegraded":"2024-05-09T03:12:05.852846+0000","last_fullsized":"2024-05-09T03:12:05.852846+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:50.152051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833184+0000","last_change":"2024-05-09T03:12:05.833184+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833184+0000","last_undegraded":"2024-05-09T03:12:05.833184+0000","last_fullsized":"2024-05-09T03:12:05.833184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851612+0000","last_change":"2024-05-09T03:12:05.847630+0000","last_active":"2024-05-09T03:12:05.851612+0000","last_peered":"2024-05-09T03:12:05.851612+0000","last_clean":"2024-05-09T03:12:05.851612+0000","last_became_active":"2024-05-09T03:12:05.847302+0000","last_became_peered":"2024-05-09T03:12:05.847302+0000","last_unstale":"2024-05-09T03:12:05.851612+0000","last_undegraded":"2024-05-09T03:12:05.851612+0000","last_fullsized":"2024-05-09T03:12:05.851612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837431+0000","last_change":"2024-05-09T03:12:05.837431+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837431+0000","last_undegraded":"2024-05-09T03:12:05.837431+0000","last_fullsized":"2024-05-09T03:12:05.837431+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853524+0000","last_change":"2024-05-09T03:12:05.845847+0000","last_active":"2024-05-09T03:12:05.853524+0000","last_peered":"2024-05-09T03:12:05.853524+0000","last_clean":"2024-05-09T03:12:05.853524+0000","last_became_active":"2024-05-09T03:12:05.845474+0000","last_became_peered":"2024-05-09T03:12:05.845474+0000","last_unstale":"2024-05-09T03:12:05.853524+0000","last_undegraded":"2024-05-09T03:12:05.853524+0000","last_fullsized":"2024-05-09T03:12:05.853524+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:56.893788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839951+0000","last_change":"2024-05-09T03:12:05.839951+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839951+0000","last_undegraded":"2024-05-09T03:12:05.839951+0000","last_fullsized":"2024-05-09T03:12:05.839951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854903+0000","last_change":"2024-05-09T03:12:05.854030+0000","last_active":"2024-05-09T03:12:05.854903+0000","last_peered":"2024-05-09T03:12:05.854903+0000","last_clean":"2024-05-09T03:12:05.854903+0000","last_became_active":"2024-05-09T03:12:05.853677+0000","last_became_peered":"2024-05-09T03:12:05.853677+0000","last_unstale":"2024-05-09T03:12:05.854903+0000","last_undegraded":"2024-05-09T03:12:05.854903+0000","last_fullsized":"2024-05-09T03:12:05.854903+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:25.968267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837833+0000","last_change":"2024-05-09T03:12:05.837833+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837833+0000","last_undegraded":"2024-05-09T03:12:05.837833+0000","last_fullsized":"2024-05-09T03:12:05.837833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846449+0000","last_change":"2024-05-09T03:12:05.843325+0000","last_active":"2024-05-09T03:12:05.846449+0000","last_peered":"2024-05-09T03:12:05.846449+0000","last_clean":"2024-05-09T03:12:05.846449+0000","last_became_active":"2024-05-09T03:12:05.842923+0000","last_became_peered":"2024-05-09T03:12:05.842923+0000","last_unstale":"2024-05-09T03:12:05.846449+0000","last_undegraded":"2024-05-09T03:12:05.846449+0000","last_fullsized":"2024-05-09T03:12:05.846449+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:43:02.488792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837016+0000","last_change":"2024-05-09T03:12:05.837016+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837016+0000","last_undegraded":"2024-05-09T03:12:05.837016+0000","last_fullsized":"2024-05-09T03:12:05.837016+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853796+0000","last_change":"2024-05-09T03:12:05.848661+0000","last_active":"2024-05-09T03:12:05.853796+0000","last_peered":"2024-05-09T03:12:05.853796+0000","last_clean":"2024-05-09T03:12:05.853796+0000","last_became_active":"2024-05-09T03:12:05.848010+0000","last_became_peered":"2024-05-09T03:12:05.848010+0000","last_unstale":"2024-05-09T03:12:05.853796+0000","last_undegraded":"2024-05-09T03:12:05.853796+0000","last_fullsized":"2024-05-09T03:12:05.853796+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:20.873828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835668+0000","last_change":"2024-05-09T03:12:05.835668+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835668+0000","last_undegraded":"2024-05-09T03:12:05.835668+0000","last_fullsized":"2024-05-09T03:12:05.835668+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852646+0000","last_change":"2024-05-09T03:12:05.849792+0000","last_active":"2024-05-09T03:12:05.852646+0000","last_peered":"2024-05-09T03:12:05.852646+0000","last_clean":"2024-05-09T03:12:05.852646+0000","last_became_active":"2024-05-09T03:12:05.849439+0000","last_became_peered":"2024-05-09T03:12:05.849439+0000","last_unstale":"2024-05-09T03:12:05.852646+0000","last_undegraded":"2024-05-09T03:12:05.852646+0000","last_fullsized":"2024-05-09T03:12:05.852646+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836209+0000","last_change":"2024-05-09T03:12:05.836209+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836209+0000","last_undegraded":"2024-05-09T03:12:05.836209+0000","last_fullsized":"2024-05-09T03:12:05.836209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854627+0000","last_change":"2024-05-09T03:12:05.853093+0000","last_active":"2024-05-09T03:12:05.854627+0000","last_peered":"2024-05-09T03:12:05.854627+0000","last_clean":"2024-05-09T03:12:05.854627+0000","last_became_active":"2024-05-09T03:12:05.852692+0000","last_became_peered":"2024-05-09T03:12:05.852692+0000","last_unstale":"2024-05-09T03:12:05.854627+0000","last_undegraded":"2024-05-09T03:12:05.854627+0000","last_fullsized":"2024-05-09T03:12:05.854627+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:43.389393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841432+0000","last_change":"2024-05-09T03:12:05.841432+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841432+0000","last_undegraded":"2024-05-09T03:12:05.841432+0000","last_fullsized":"2024-05-09T03:12:05.841432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852440+0000","last_change":"2024-05-09T03:12:05.849439+0000","last_active":"2024-05-09T03:12:05.852440+0000","last_peered":"2024-05-09T03:12:05.852440+0000","last_clean":"2024-05-09T03:12:05.852440+0000","last_became_active":"2024-05-09T03:12:05.849134+0000","last_became_peered":"2024-05-09T03:12:05.849134+0000","last_unstale":"2024-05-09T03:12:05.852440+0000","last_undegraded":"2024-05-09T03:12:05.852440+0000","last_fullsized":"2024-05-09T03:12:05.852440+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:43:46.511932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835494+0000","last_change":"2024-05-09T03:12:05.835494+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835494+0000","last_undegraded":"2024-05-09T03:12:05.835494+0000","last_fullsized":"2024-05-09T03:12:05.835494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854065+0000","last_change":"2024-05-09T03:12:05.852504+0000","last_active":"2024-05-09T03:12:05.854065+0000","last_peered":"2024-05-09T03:12:05.854065+0000","last_clean":"2024-05-09T03:12:05.854065+0000","last_became_active":"2024-05-09T03:12:05.851857+0000","last_became_peered":"2024-05-09T03:12:05.851857+0000","last_unstale":"2024-05-09T03:12:05.854065+0000","last_undegraded":"2024-05-09T03:12:05.854065+0000","last_fullsized":"2024-05-09T03:12:05.854065+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836813+0000","last_change":"2024-05-09T03:12:05.836813+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836813+0000","last_undegraded":"2024-05-09T03:12:05.836813+0000","last_fullsized":"2024-05-09T03:12:05.836813+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849695+0000","last_change":"2024-05-09T03:12:05.844269+0000","last_active":"2024-05-09T03:12:05.849695+0000","last_peered":"2024-05-09T03:12:05.849695+0000","last_clean":"2024-05-09T03:12:05.849695+0000","last_became_active":"2024-05-09T03:12:05.843939+0000","last_became_peered":"2024-05-09T03:12:05.843939+0000","last_unstale":"2024-05-09T03:12:05.849695+0000","last_undegraded":"2024-05-09T03:12:05.849695+0000","last_fullsized":"2024-05-09T03:12:05.849695+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839358+0000","last_change":"2024-05-09T03:12:05.839358+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839358+0000","last_undegraded":"2024-05-09T03:12:05.839358+0000","last_fullsized":"2024-05-09T03:12:05.839358+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851603+0000","last_change":"2024-05-09T03:12:05.847432+0000","last_active":"2024-05-09T03:12:05.851603+0000","last_peered":"2024-05-09T03:12:05.851603+0000","last_clean":"2024-05-09T03:12:05.851603+0000","last_became_active":"2024-05-09T03:12:05.847153+0000","last_became_peered":"2024-05-09T03:12:05.847153+0000","last_unstale":"2024-05-09T03:12:05.851603+0000","last_undegraded":"2024-05-09T03:12:05.851603+0000","last_fullsized":"2024-05-09T03:12:05.851603+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:02:27.480835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833512+0000","last_change":"2024-05-09T03:12:05.833512+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833512+0000","last_undegraded":"2024-05-09T03:12:05.833512+0000","last_fullsized":"2024-05-09T03:12:05.833512+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853273+0000","last_change":"2024-05-09T03:12:05.850218+0000","last_active":"2024-05-09T03:12:05.853273+0000","last_peered":"2024-05-09T03:12:05.853273+0000","last_clean":"2024-05-09T03:12:05.853273+0000","last_became_active":"2024-05-09T03:12:05.849743+0000","last_became_peered":"2024-05-09T03:12:05.849743+0000","last_unstale":"2024-05-09T03:12:05.853273+0000","last_undegraded":"2024-05-09T03:12:05.853273+0000","last_fullsized":"2024-05-09T03:12:05.853273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:40.635239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833538+0000","last_change":"2024-05-09T03:12:05.833538+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833538+0000","last_undegraded":"2024-05-09T03:12:05.833538+0000","last_fullsized":"2024-05-09T03:12:05.833538+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852904+0000","last_change":"2024-05-09T03:12:05.848391+0000","last_active":"2024-05-09T03:12:05.852904+0000","last_peered":"2024-05-09T03:12:05.852904+0000","last_clean":"2024-05-09T03:12:05.852904+0000","last_became_active":"2024-05-09T03:12:05.847525+0000","last_became_peered":"2024-05-09T03:12:05.847525+0000","last_unstale":"2024-05-09T03:12:05.852904+0000","last_undegraded":"2024-05-09T03:12:05.852904+0000","last_fullsized":"2024-05-09T03:12:05.852904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841979+0000","last_change":"2024-05-09T03:12:05.841979+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841979+0000","last_undegraded":"2024-05-09T03:12:05.841979+0000","last_fullsized":"2024-05-09T03:12:05.841979+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.847707+0000","last_change":"2024-05-09T03:12:05.846693+0000","last_active":"2024-05-09T03:12:05.847707+0000","last_peered":"2024-05-09T03:12:05.847707+0000","last_clean":"2024-05-09T03:12:05.847707+0000","last_became_active":"2024-05-09T03:12:05.846379+0000","last_became_peered":"2024-05-09T03:12:05.846379+0000","last_unstale":"2024-05-09T03:12:05.847707+0000","last_undegraded":"2024-05-09T03:12:05.847707+0000","last_fullsized":"2024-05-09T03:12:05.847707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:57.513277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840940+0000","last_change":"2024-05-09T03:12:05.840940+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840940+0000","last_undegraded":"2024-05-09T03:12:05.840940+0000","last_fullsized":"2024-05-09T03:12:05.840940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846195+0000","last_change":"2024-05-09T03:12:05.843597+0000","last_active":"2024-05-09T03:12:05.846195+0000","last_peered":"2024-05-09T03:12:05.846195+0000","last_clean":"2024-05-09T03:12:05.846195+0000","last_became_active":"2024-05-09T03:12:05.843281+0000","last_became_peered":"2024-05-09T03:12:05.843281+0000","last_unstale":"2024-05-09T03:12:05.846195+0000","last_undegraded":"2024-05-09T03:12:05.846195+0000","last_fullsized":"2024-05-09T03:12:05.846195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832473+0000","last_change":"2024-05-09T03:12:05.832473+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832473+0000","last_undegraded":"2024-05-09T03:12:05.832473+0000","last_fullsized":"2024-05-09T03:12:05.832473+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852261+0000","last_change":"2024-05-09T03:12:05.845237+0000","last_active":"2024-05-09T03:12:05.852261+0000","last_peered":"2024-05-09T03:12:05.852261+0000","last_clean":"2024-05-09T03:12:05.852261+0000","last_became_active":"2024-05-09T03:12:05.844921+0000","last_became_peered":"2024-05-09T03:12:05.844921+0000","last_unstale":"2024-05-09T03:12:05.852261+0000","last_undegraded":"2024-05-09T03:12:05.852261+0000","last_fullsized":"2024-05-09T03:12:05.852261+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:08:03.214517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835634+0000","last_change":"2024-05-09T03:12:05.835634+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835634+0000","last_undegraded":"2024-05-09T03:12:05.835634+0000","last_fullsized":"2024-05-09T03:12:05.835634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854023+0000","last_change":"2024-05-09T03:12:05.850855+0000","last_active":"2024-05-09T03:12:05.854023+0000","last_peered":"2024-05-09T03:12:05.854023+0000","last_clean":"2024-05-09T03:12:05.854023+0000","last_became_active":"2024-05-09T03:12:05.850518+0000","last_became_peered":"2024-05-09T03:12:05.850518+0000","last_unstale":"2024-05-09T03:12:05.854023+0000","last_undegraded":"2024-05-09T03:12:05.854023+0000","last_fullsized":"2024-05-09T03:12:05.854023+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834748+0000","last_change":"2024-05-09T03:12:05.834748+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834748+0000","last_undegraded":"2024-05-09T03:12:05.834748+0000","last_fullsized":"2024-05-09T03:12:05.834748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851159+0000","last_change":"2024-05-09T03:12:05.848645+0000","last_active":"2024-05-09T03:12:05.851159+0000","last_peered":"2024-05-09T03:12:05.851159+0000","last_clean":"2024-05-09T03:12:05.851159+0000","last_became_active":"2024-05-09T03:12:05.848326+0000","last_became_peered":"2024-05-09T03:12:05.848326+0000","last_unstale":"2024-05-09T03:12:05.851159+0000","last_undegraded":"2024-05-09T03:12:05.851159+0000","last_fullsized":"2024-05-09T03:12:05.851159+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:20.727622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835236+0000","last_change":"2024-05-09T03:12:05.835236+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835236+0000","last_undegraded":"2024-05-09T03:12:05.835236+0000","last_fullsized":"2024-05-09T03:12:05.835236+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853515+0000","last_change":"2024-05-09T03:12:05.843892+0000","last_active":"2024-05-09T03:12:05.853515+0000","last_peered":"2024-05-09T03:12:05.853515+0000","last_clean":"2024-05-09T03:12:05.853515+0000","last_became_active":"2024-05-09T03:12:05.843459+0000","last_became_peered":"2024-05-09T03:12:05.843459+0000","last_unstale":"2024-05-09T03:12:05.853515+0000","last_undegraded":"2024-05-09T03:12:05.853515+0000","last_fullsized":"2024-05-09T03:12:05.853515+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:48.718539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832475+0000","last_change":"2024-05-09T03:12:05.832475+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832475+0000","last_undegraded":"2024-05-09T03:12:05.832475+0000","last_fullsized":"2024-05-09T03:12:05.832475+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.845868+0000","last_change":"2024-05-09T03:12:05.844488+0000","last_active":"2024-05-09T03:12:05.845868+0000","last_peered":"2024-05-09T03:12:05.845868+0000","last_clean":"2024-05-09T03:12:05.845868+0000","last_became_active":"2024-05-09T03:12:05.844183+0000","last_became_peered":"2024-05-09T03:12:05.844183+0000","last_unstale":"2024-05-09T03:12:05.845868+0000","last_undegraded":"2024-05-09T03:12:05.845868+0000","last_fullsized":"2024-05-09T03:12:05.845868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:08:08.665854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834642+0000","last_change":"2024-05-09T03:12:05.834642+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834642+0000","last_undegraded":"2024-05-09T03:12:05.834642+0000","last_fullsized":"2024-05-09T03:12:05.834642+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852985+0000","last_change":"2024-05-09T03:12:05.849987+0000","last_active":"2024-05-09T03:12:05.852985+0000","last_peered":"2024-05-09T03:12:05.852985+0000","last_clean":"2024-05-09T03:12:05.852985+0000","last_became_active":"2024-05-09T03:12:05.848380+0000","last_became_peered":"2024-05-09T03:12:05.848380+0000","last_unstale":"2024-05-09T03:12:05.852985+0000","last_undegraded":"2024-05-09T03:12:05.852985+0000","last_fullsized":"2024-05-09T03:12:05.852985+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:09:38.070718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835177+0000","last_change":"2024-05-09T03:12:05.835177+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835177+0000","last_undegraded":"2024-05-09T03:12:05.835177+0000","last_fullsized":"2024-05-09T03:12:05.835177+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852583+0000","last_change":"2024-05-09T03:12:05.851689+0000","last_active":"2024-05-09T03:12:05.852583+0000","last_peered":"2024-05-09T03:12:05.852583+0000","last_clean":"2024-05-09T03:12:05.852583+0000","last_became_active":"2024-05-09T03:12:05.851357+0000","last_became_peered":"2024-05-09T03:12:05.851357+0000","last_unstale":"2024-05-09T03:12:05.852583+0000","last_undegraded":"2024-05-09T03:12:05.852583+0000","last_fullsized":"2024-05-09T03:12:05.852583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:49.416402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":170,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.836038+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:12:05.836038+0000","last_peered":"2024-05-09T03:12:05.836038+0000","last_clean":"2024-05-09T03:12:05.836038+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:12:05.836038+0000","last_undegraded":"2024-05-09T03:12:05.836038+0000","last_fullsized":"2024-05-09T03:12:05.836038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1228800,"data_stored":850793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":260425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":844526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:12:10.948 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:12:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.097+0000 7fc3548c0640 1 Processor -- start 2024-05-09T03:12:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.097+0000 7fc3548c0640 1 -- start start 2024-05-09T03:12:11.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc3548c0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3500b5210 0x7fc3501613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc3548c0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 0x7fc3500d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc3548c0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3500d5280 con 0x7fc3500d4800 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc3548c0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc3500b5620 con 0x7fc3500b5210 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc3548c0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc3501609a0 con 0x7fc3500b7ed0 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34ed9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 0x7fc3500d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3500b5210 0x7fc3501613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc3500d4800 0x7fc3500b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:33118/0 (socket says 172.21.15.46:33118) 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34ed9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 0x7fc3500d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:40394/0 (socket says 172.21.15.46:40394) 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34dd9b640 1 -- 172.21.15.46:0/4115462779 learned_addr learned my addr 172.21.15.46:0/4115462779 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3500b5210 0x7fc3501613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:41770/0 (socket says 172.21.15.46:41770) 2024-05-09T03:12:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34ed9d640 1 -- 172.21.15.46:0/4115462779 >> v2:172.21.15.88:3300/0 conn(0x7fc3500b5210 msgr2=0x7fc3501613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34ed9d640 1 --2- 172.21.15.46:0/4115462779 >> v2:172.21.15.88:3300/0 conn(0x7fc3500b5210 0x7fc3501613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.099+0000 7fc34ed9d640 1 -- 172.21.15.46:0/4115462779 >> v1:172.21.15.88:6789/0 conn(0x7fc3500d4800 legacy=0x7fc3500b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.100+0000 7fc34ed9d640 1 -- 172.21.15.46:0/4115462779 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3500d5560 con 0x7fc3500b7ed0 2024-05-09T03:12:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.100+0000 7fc34ed9d640 1 --2- 172.21.15.46:0/4115462779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 0x7fc3500d4c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc348009d90 tx=0x7fc34802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19741fe547e4049a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc34d59a640 1 -- 172.21.15.46:0/4115462779 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348031600 con 0x7fc3500b7ed0 2024-05-09T03:12:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc34d59a640 1 -- 172.21.15.46:0/4115462779 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3480317a0 con 0x7fc3500b7ed0 2024-05-09T03:12:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc34d59a640 1 -- 172.21.15.46:0/4115462779 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348031b70 con 0x7fc3500b7ed0 2024-05-09T03:12:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 -- 172.21.15.46:0/4115462779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 msgr2=0x7fc3500d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 --2- 172.21.15.46:0/4115462779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 0x7fc3500d4c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc348009d90 tx=0x7fc34802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 -- 172.21.15.46:0/4115462779 shutdown_connections 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 --2- 172.21.15.46:0/4115462779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500b7ed0 0x7fc3500d4c10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 --2- 172.21.15.46:0/4115462779 >> v2:172.21.15.88:3300/0 conn(0x7fc3500b5210 0x7fc3501613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 -- 172.21.15.46:0/4115462779 >> 172.21.15.46:0/4115462779 conn(0x7fc3500d29a0 msgr2=0x7fc3500d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 -- 172.21.15.46:0/4115462779 shutdown_connections 2024-05-09T03:12:11.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.101+0000 7fc3548c0640 1 -- 172.21.15.46:0/4115462779 wait complete. 2024-05-09T03:12:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.103+0000 7fc3548c0640 1 Processor -- start 2024-05-09T03:12:11.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.103+0000 7fc3548c0640 1 -- start start 2024-05-09T03:12:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc3548c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3500b5210 0x7fc350148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc3548c0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500d4800 0x7fc3501f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc3548c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 0x7fc3501f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc3548c0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3500d0ae0 con 0x7fc3501f62c0 2024-05-09T03:12:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc3548c0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc350160ea0 con 0x7fc3500b5210 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc3548c0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc350163e60 con 0x7fc3500d4800 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 0x7fc3501f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 0x7fc3501f66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53262/0 (socket says 172.21.15.46:53262) 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 -- 172.21.15.46:0/2530381734 learned_addr learned my addr 172.21.15.46:0/2530381734 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34dd9b640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500d4800 0x7fc3501f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34e59c640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3500b5210 0x7fc350148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 -- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500d4800 msgr2=0x7fc3501f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500d4800 0x7fc3501f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 -- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3500b5210 msgr2=0x7fc350148820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3500b5210 0x7fc350148820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 -- 172.21.15.46:0/2530381734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3500ca900 con 0x7fc3501f62c0 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.105+0000 7fc34ed9d640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 0x7fc3501f66d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc34802abc0 tx=0x7fc348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.106+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34803f070 con 0x7fc3501f62c0 2024-05-09T03:12:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.106+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc348030b60 con 0x7fc3501f62c0 2024-05-09T03:12:11.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.106+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc348009450 con 0x7fc3501f62c0 2024-05-09T03:12:11.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.106+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3500cabe0 con 0x7fc3501f62c0 2024-05-09T03:12:11.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.106+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3500cb6c0 con 0x7fc3501f62c0 2024-05-09T03:12:11.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.106+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35006a270 con 0x7fc3501f62c0 2024-05-09T03:12:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.107+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc34803b410 con 0x7fc3501f62c0 2024-05-09T03:12:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.108+0000 7fc33effd640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc330076300 0x7fc330078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.108+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fc3480b8bd0 con 0x7fc3501f62c0 2024-05-09T03:12:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.108+0000 7fc34e59c640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc330076300 0x7fc330078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.109+0000 7fc34e59c640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc330076300 0x7fc330078810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc340009aa0 tx=0x7fc340009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.112+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc348082050 con 0x7fc3501f62c0 2024-05-09T03:12:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.303+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3500b8700 con 0x7fc330076300 2024-05-09T03:12:11.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.309+0000 7fc33effd640 1 -- 172.21.15.46:0/2530381734 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7fc3500b8700 con 0x7fc330076300 2024-05-09T03:12:11.313 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.312+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc330076300 msgr2=0x7fc330078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.312+0000 7fc3548c0640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc330076300 0x7fc330078810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc340009aa0 tx=0x7fc340009190 comp rx=0 tx=0).stop 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.312+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 msgr2=0x7fc3501f66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.312+0000 7fc3548c0640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 0x7fc3501f66d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc34802abc0 tx=0x7fc348004a40 comp rx=0 tx=0).stop 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 shutdown_connections 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc330076300 0x7fc330078810 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3501f62c0 0x7fc3501f66d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc3500d4800 0x7fc3501f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 --2- 172.21.15.46:0/2530381734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3500b5210 0x7fc350148820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 >> 172.21.15.46:0/2530381734 conn(0x7fc3500d29a0 msgr2=0x7fc3500ba770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 shutdown_connections 2024-05-09T03:12:11.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.313+0000 7fc3548c0640 1 -- 172.21.15.46:0/2530381734 wait complete. 2024-05-09T03:12:11.325 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-05-09T03:12:10.531012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238908,"kb_used_data":6048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735812,"statfs":{"total":773094113280,"available":772849471488,"internally_reserved":0,"allocated":6193152,"data_stored":3226534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.706220"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841057+0000","last_change":"2024-05-09T03:12:05.841057+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841057+0000","last_undegraded":"2024-05-09T03:12:05.841057+0000","last_fullsized":"2024-05-09T03:12:05.841057+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836609+0000","last_change":"2024-05-09T03:12:05.836609+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836609+0000","last_undegraded":"2024-05-09T03:12:05.836609+0000","last_fullsized":"2024-05-09T03:12:05.836609+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837976+0000","last_change":"2024-05-09T03:12:05.837976+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837976+0000","last_undegraded":"2024-05-09T03:12:05.837976+0000","last_fullsized":"2024-05-09T03:12:05.837976+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.843259+0000","last_change":"2024-05-09T03:12:05.843259+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.843259+0000","last_undegraded":"2024-05-09T03:12:05.843259+0000","last_fullsized":"2024-05-09T03:12:05.843259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839531+0000","last_change":"2024-05-09T03:12:05.839531+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839531+0000","last_undegraded":"2024-05-09T03:12:05.839531+0000","last_fullsized":"2024-05-09T03:12:05.839531+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834452+0000","last_change":"2024-05-09T03:12:05.834452+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834452+0000","last_undegraded":"2024-05-09T03:12:05.834452+0000","last_fullsized":"2024-05-09T03:12:05.834452+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836296+0000","last_change":"2024-05-09T03:12:05.836296+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836296+0000","last_undegraded":"2024-05-09T03:12:05.836296+0000","last_fullsized":"2024-05-09T03:12:05.836296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835798+0000","last_change":"2024-05-09T03:12:05.835798+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835798+0000","last_undegraded":"2024-05-09T03:12:05.835798+0000","last_fullsized":"2024-05-09T03:12:05.835798+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837939+0000","last_change":"2024-05-09T03:12:05.837939+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837939+0000","last_undegraded":"2024-05-09T03:12:05.837939+0000","last_fullsized":"2024-05-09T03:12:05.837939+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833750+0000","last_change":"2024-05-09T03:12:05.833750+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833750+0000","last_undegraded":"2024-05-09T03:12:05.833750+0000","last_fullsized":"2024-05-09T03:12:05.833750+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833154+0000","last_change":"2024-05-09T03:12:05.833154+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833154+0000","last_undegraded":"2024-05-09T03:12:05.833154+0000","last_fullsized":"2024-05-09T03:12:05.833154+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842732+0000","last_change":"2024-05-09T03:12:05.842732+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842732+0000","last_undegraded":"2024-05-09T03:12:05.842732+0000","last_fullsized":"2024-05-09T03:12:05.842732+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833533+0000","last_change":"2024-05-09T03:12:05.833533+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833533+0000","last_undegraded":"2024-05-09T03:12:05.833533+0000","last_fullsized":"2024-05-09T03:12:05.833533+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836888+0000","last_change":"2024-05-09T03:12:05.836888+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836888+0000","last_undegraded":"2024-05-09T03:12:05.836888+0000","last_fullsized":"2024-05-09T03:12:05.836888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835706+0000","last_change":"2024-05-09T03:12:05.835706+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835706+0000","last_undegraded":"2024-05-09T03:12:05.835706+0000","last_fullsized":"2024-05-09T03:12:05.835706+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833981+0000","last_change":"2024-05-09T03:12:05.833981+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833981+0000","last_undegraded":"2024-05-09T03:12:05.833981+0000","last_fullsized":"2024-05-09T03:12:05.833981+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840026+0000","last_change":"2024-05-09T03:12:05.840026+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840026+0000","last_undegraded":"2024-05-09T03:12:05.840026+0000","last_fullsized":"2024-05-09T03:12:05.840026+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838628+0000","last_change":"2024-05-09T03:12:05.838628+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838628+0000","last_undegraded":"2024-05-09T03:12:05.838628+0000","last_fullsized":"2024-05-09T03:12:05.838628+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839714+0000","last_change":"2024-05-09T03:12:05.839714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839714+0000","last_undegraded":"2024-05-09T03:12:05.839714+0000","last_fullsized":"2024-05-09T03:12:05.839714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837245+0000","last_change":"2024-05-09T03:12:05.837245+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837245+0000","last_undegraded":"2024-05-09T03:12:05.837245+0000","last_fullsized":"2024-05-09T03:12:05.837245+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838405+0000","last_change":"2024-05-09T03:12:05.838405+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838405+0000","last_undegraded":"2024-05-09T03:12:05.838405+0000","last_fullsized":"2024-05-09T03:12:05.838405+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837199+0000","last_change":"2024-05-09T03:12:05.837199+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837199+0000","last_undegraded":"2024-05-09T03:12:05.837199+0000","last_fullsized":"2024-05-09T03:12:05.837199+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836251+0000","last_change":"2024-05-09T03:12:05.836251+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836251+0000","last_undegraded":"2024-05-09T03:12:05.836251+0000","last_fullsized":"2024-05-09T03:12:05.836251+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834399+0000","last_change":"2024-05-09T03:12:05.834399+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834399+0000","last_undegraded":"2024-05-09T03:12:05.834399+0000","last_fullsized":"2024-05-09T03:12:05.834399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840127+0000","last_change":"2024-05-09T03:12:05.840127+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840127+0000","last_undegraded":"2024-05-09T03:12:05.840127+0000","last_fullsized":"2024-05-09T03:12:05.840127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.844753+0000","last_change":"2024-05-09T03:12:05.844753+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.844753+0000","last_undegraded":"2024-05-09T03:12:05.844753+0000","last_fullsized":"2024-05-09T03:12:05.844753+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836714+0000","last_change":"2024-05-09T03:12:05.836714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836714+0000","last_undegraded":"2024-05-09T03:12:05.836714+0000","last_fullsized":"2024-05-09T03:12:05.836714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842327+0000","last_change":"2024-05-09T03:12:05.842327+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842327+0000","last_undegraded":"2024-05-09T03:12:05.842327+0000","last_fullsized":"2024-05-09T03:12:05.842327+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837936+0000","last_change":"2024-05-09T03:12:05.837936+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837936+0000","last_undegraded":"2024-05-09T03:12:05.837936+0000","last_fullsized":"2024-05-09T03:12:05.837936+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838187+0000","last_change":"2024-05-09T03:12:05.838187+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838187+0000","last_undegraded":"2024-05-09T03:12:05.838187+0000","last_fullsized":"2024-05-09T03:12:05.838187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840098+0000","last_change":"2024-05-09T03:12:05.840098+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840098+0000","last_undegraded":"2024-05-09T03:12:05.840098+0000","last_fullsized":"2024-05-09T03:12:05.840098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832938+0000","last_change":"2024-05-09T03:12:05.832938+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832938+0000","last_undegraded":"2024-05-09T03:12:05.832938+0000","last_fullsized":"2024-05-09T03:12:05.832938+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837077+0000","last_change":"2024-05-09T03:12:05.837077+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837077+0000","last_undegraded":"2024-05-09T03:12:05.837077+0000","last_fullsized":"2024-05-09T03:12:05.837077+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852354+0000","last_change":"2024-05-09T03:12:05.849076+0000","last_active":"2024-05-09T03:12:05.852354+0000","last_peered":"2024-05-09T03:12:05.852354+0000","last_clean":"2024-05-09T03:12:05.852354+0000","last_became_active":"2024-05-09T03:12:05.848768+0000","last_became_peered":"2024-05-09T03:12:05.848768+0000","last_unstale":"2024-05-09T03:12:05.852354+0000","last_undegraded":"2024-05-09T03:12:05.852354+0000","last_fullsized":"2024-05-09T03:12:05.852354+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:05.791451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832809+0000","last_change":"2024-05-09T03:12:05.832809+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832809+0000","last_undegraded":"2024-05-09T03:12:05.832809+0000","last_fullsized":"2024-05-09T03:12:05.832809+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851128+0000","last_change":"2024-05-09T03:12:05.846715+0000","last_active":"2024-05-09T03:12:05.851128+0000","last_peered":"2024-05-09T03:12:05.851128+0000","last_clean":"2024-05-09T03:12:05.851128+0000","last_became_active":"2024-05-09T03:12:05.846448+0000","last_became_peered":"2024-05-09T03:12:05.846448+0000","last_unstale":"2024-05-09T03:12:05.851128+0000","last_undegraded":"2024-05-09T03:12:05.851128+0000","last_fullsized":"2024-05-09T03:12:05.851128+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:53:45.376505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833746+0000","last_change":"2024-05-09T03:12:05.833746+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833746+0000","last_undegraded":"2024-05-09T03:12:05.833746+0000","last_fullsized":"2024-05-09T03:12:05.833746+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849635+0000","last_change":"2024-05-09T03:12:05.845815+0000","last_active":"2024-05-09T03:12:05.849635+0000","last_peered":"2024-05-09T03:12:05.849635+0000","last_clean":"2024-05-09T03:12:05.849635+0000","last_became_active":"2024-05-09T03:12:05.845509+0000","last_became_peered":"2024-05-09T03:12:05.845509+0000","last_unstale":"2024-05-09T03:12:05.849635+0000","last_undegraded":"2024-05-09T03:12:05.849635+0000","last_fullsized":"2024-05-09T03:12:05.849635+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:07.083261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836173+0000","last_change":"2024-05-09T03:12:05.836173+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836173+0000","last_undegraded":"2024-05-09T03:12:05.836173+0000","last_fullsized":"2024-05-09T03:12:05.836173+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853894+0000","last_change":"2024-05-09T03:12:05.849601+0000","last_active":"2024-05-09T03:12:05.853894+0000","last_peered":"2024-05-09T03:12:05.853894+0000","last_clean":"2024-05-09T03:12:05.853894+0000","last_became_active":"2024-05-09T03:12:05.849230+0000","last_became_peered":"2024-05-09T03:12:05.849230+0000","last_unstale":"2024-05-09T03:12:05.853894+0000","last_undegraded":"2024-05-09T03:12:05.853894+0000","last_fullsized":"2024-05-09T03:12:05.853894+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:43.363860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835661+0000","last_change":"2024-05-09T03:12:05.835661+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835661+0000","last_undegraded":"2024-05-09T03:12:05.835661+0000","last_fullsized":"2024-05-09T03:12:05.835661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.856188+0000","last_change":"2024-05-09T03:12:05.853186+0000","last_active":"2024-05-09T03:12:05.856188+0000","last_peered":"2024-05-09T03:12:05.856188+0000","last_clean":"2024-05-09T03:12:05.856188+0000","last_became_active":"2024-05-09T03:12:05.852815+0000","last_became_peered":"2024-05-09T03:12:05.852815+0000","last_unstale":"2024-05-09T03:12:05.856188+0000","last_undegraded":"2024-05-09T03:12:05.856188+0000","last_fullsized":"2024-05-09T03:12:05.856188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838497+0000","last_change":"2024-05-09T03:12:05.838497+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838497+0000","last_undegraded":"2024-05-09T03:12:05.838497+0000","last_fullsized":"2024-05-09T03:12:05.838497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849662+0000","last_change":"2024-05-09T03:12:05.844443+0000","last_active":"2024-05-09T03:12:05.849662+0000","last_peered":"2024-05-09T03:12:05.849662+0000","last_clean":"2024-05-09T03:12:05.849662+0000","last_became_active":"2024-05-09T03:12:05.844119+0000","last_became_peered":"2024-05-09T03:12:05.844119+0000","last_unstale":"2024-05-09T03:12:05.849662+0000","last_undegraded":"2024-05-09T03:12:05.849662+0000","last_fullsized":"2024-05-09T03:12:05.849662+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833532+0000","last_change":"2024-05-09T03:12:05.833532+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833532+0000","last_undegraded":"2024-05-09T03:12:05.833532+0000","last_fullsized":"2024-05-09T03:12:05.833532+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852899+0000","last_change":"2024-05-09T03:12:05.847498+0000","last_active":"2024-05-09T03:12:05.852899+0000","last_peered":"2024-05-09T03:12:05.852899+0000","last_clean":"2024-05-09T03:12:05.852899+0000","last_became_active":"2024-05-09T03:12:05.847170+0000","last_became_peered":"2024-05-09T03:12:05.847170+0000","last_unstale":"2024-05-09T03:12:05.852899+0000","last_undegraded":"2024-05-09T03:12:05.852899+0000","last_fullsized":"2024-05-09T03:12:05.852899+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835683+0000","last_change":"2024-05-09T03:12:05.835683+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835683+0000","last_undegraded":"2024-05-09T03:12:05.835683+0000","last_fullsized":"2024-05-09T03:12:05.835683+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853539+0000","last_change":"2024-05-09T03:12:05.847308+0000","last_active":"2024-05-09T03:12:05.853539+0000","last_peered":"2024-05-09T03:12:05.853539+0000","last_clean":"2024-05-09T03:12:05.853539+0000","last_became_active":"2024-05-09T03:12:05.846406+0000","last_became_peered":"2024-05-09T03:12:05.846406+0000","last_unstale":"2024-05-09T03:12:05.853539+0000","last_undegraded":"2024-05-09T03:12:05.853539+0000","last_fullsized":"2024-05-09T03:12:05.853539+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838513+0000","last_change":"2024-05-09T03:12:05.838513+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838513+0000","last_undegraded":"2024-05-09T03:12:05.838513+0000","last_fullsized":"2024-05-09T03:12:05.838513+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851570+0000","last_change":"2024-05-09T03:12:05.847459+0000","last_active":"2024-05-09T03:12:05.851570+0000","last_peered":"2024-05-09T03:12:05.851570+0000","last_clean":"2024-05-09T03:12:05.851570+0000","last_became_active":"2024-05-09T03:12:05.847106+0000","last_became_peered":"2024-05-09T03:12:05.847106+0000","last_unstale":"2024-05-09T03:12:05.851570+0000","last_undegraded":"2024-05-09T03:12:05.851570+0000","last_fullsized":"2024-05-09T03:12:05.851570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835774+0000","last_change":"2024-05-09T03:12:05.835774+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835774+0000","last_undegraded":"2024-05-09T03:12:05.835774+0000","last_fullsized":"2024-05-09T03:12:05.835774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852846+0000","last_change":"2024-05-09T03:12:05.849037+0000","last_active":"2024-05-09T03:12:05.852846+0000","last_peered":"2024-05-09T03:12:05.852846+0000","last_clean":"2024-05-09T03:12:05.852846+0000","last_became_active":"2024-05-09T03:12:05.848283+0000","last_became_peered":"2024-05-09T03:12:05.848283+0000","last_unstale":"2024-05-09T03:12:05.852846+0000","last_undegraded":"2024-05-09T03:12:05.852846+0000","last_fullsized":"2024-05-09T03:12:05.852846+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:50.152051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833184+0000","last_change":"2024-05-09T03:12:05.833184+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833184+0000","last_undegraded":"2024-05-09T03:12:05.833184+0000","last_fullsized":"2024-05-09T03:12:05.833184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851612+0000","last_change":"2024-05-09T03:12:05.847630+0000","last_active":"2024-05-09T03:12:05.851612+0000","last_peered":"2024-05-09T03:12:05.851612+0000","last_clean":"2024-05-09T03:12:05.851612+0000","last_became_active":"2024-05-09T03:12:05.847302+0000","last_became_peered":"2024-05-09T03:12:05.847302+0000","last_unstale":"2024-05-09T03:12:05.851612+0000","last_undegraded":"2024-05-09T03:12:05.851612+0000","last_fullsized":"2024-05-09T03:12:05.851612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837431+0000","last_change":"2024-05-09T03:12:05.837431+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837431+0000","last_undegraded":"2024-05-09T03:12:05.837431+0000","last_fullsized":"2024-05-09T03:12:05.837431+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853524+0000","last_change":"2024-05-09T03:12:05.845847+0000","last_active":"2024-05-09T03:12:05.853524+0000","last_peered":"2024-05-09T03:12:05.853524+0000","last_clean":"2024-05-09T03:12:05.853524+0000","last_became_active":"2024-05-09T03:12:05.845474+0000","last_became_peered":"2024-05-09T03:12:05.845474+0000","last_unstale":"2024-05-09T03:12:05.853524+0000","last_undegraded":"2024-05-09T03:12:05.853524+0000","last_fullsized":"2024-05-09T03:12:05.853524+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:56.893788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839951+0000","last_change":"2024-05-09T03:12:05.839951+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839951+0000","last_undegraded":"2024-05-09T03:12:05.839951+0000","last_fullsized":"2024-05-09T03:12:05.839951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854903+0000","last_change":"2024-05-09T03:12:05.854030+0000","last_active":"2024-05-09T03:12:05.854903+0000","last_peered":"2024-05-09T03:12:05.854903+0000","last_clean":"2024-05-09T03:12:05.854903+0000","last_became_active":"2024-05-09T03:12:05.853677+0000","last_became_peered":"2024-05-09T03:12:05.853677+0000","last_unstale":"2024-05-09T03:12:05.854903+0000","last_undegraded":"2024-05-09T03:12:05.854903+0000","last_fullsized":"2024-05-09T03:12:05.854903+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:25.968267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837833+0000","last_change":"2024-05-09T03:12:05.837833+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837833+0000","last_undegraded":"2024-05-09T03:12:05.837833+0000","last_fullsized":"2024-05-09T03:12:05.837833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846449+0000","last_change":"2024-05-09T03:12:05.843325+0000","last_active":"2024-05-09T03:12:05.846449+0000","last_peered":"2024-05-09T03:12:05.846449+0000","last_clean":"2024-05-09T03:12:05.846449+0000","last_became_active":"2024-05-09T03:12:05.842923+0000","last_became_peered":"2024-05-09T03:12:05.842923+0000","last_unstale":"2024-05-09T03:12:05.846449+0000","last_undegraded":"2024-05-09T03:12:05.846449+0000","last_fullsized":"2024-05-09T03:12:05.846449+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:43:02.488792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837016+0000","last_change":"2024-05-09T03:12:05.837016+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837016+0000","last_undegraded":"2024-05-09T03:12:05.837016+0000","last_fullsized":"2024-05-09T03:12:05.837016+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853796+0000","last_change":"2024-05-09T03:12:05.848661+0000","last_active":"2024-05-09T03:12:05.853796+0000","last_peered":"2024-05-09T03:12:05.853796+0000","last_clean":"2024-05-09T03:12:05.853796+0000","last_became_active":"2024-05-09T03:12:05.848010+0000","last_became_peered":"2024-05-09T03:12:05.848010+0000","last_unstale":"2024-05-09T03:12:05.853796+0000","last_undegraded":"2024-05-09T03:12:05.853796+0000","last_fullsized":"2024-05-09T03:12:05.853796+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:20.873828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835668+0000","last_change":"2024-05-09T03:12:05.835668+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835668+0000","last_undegraded":"2024-05-09T03:12:05.835668+0000","last_fullsized":"2024-05-09T03:12:05.835668+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852646+0000","last_change":"2024-05-09T03:12:05.849792+0000","last_active":"2024-05-09T03:12:05.852646+0000","last_peered":"2024-05-09T03:12:05.852646+0000","last_clean":"2024-05-09T03:12:05.852646+0000","last_became_active":"2024-05-09T03:12:05.849439+0000","last_became_peered":"2024-05-09T03:12:05.849439+0000","last_unstale":"2024-05-09T03:12:05.852646+0000","last_undegraded":"2024-05-09T03:12:05.852646+0000","last_fullsized":"2024-05-09T03:12:05.852646+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836209+0000","last_change":"2024-05-09T03:12:05.836209+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836209+0000","last_undegraded":"2024-05-09T03:12:05.836209+0000","last_fullsized":"2024-05-09T03:12:05.836209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854627+0000","last_change":"2024-05-09T03:12:05.853093+0000","last_active":"2024-05-09T03:12:05.854627+0000","last_peered":"2024-05-09T03:12:05.854627+0000","last_clean":"2024-05-09T03:12:05.854627+0000","last_became_active":"2024-05-09T03:12:05.852692+0000","last_became_peered":"2024-05-09T03:12:05.852692+0000","last_unstale":"2024-05-09T03:12:05.854627+0000","last_undegraded":"2024-05-09T03:12:05.854627+0000","last_fullsized":"2024-05-09T03:12:05.854627+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:43.389393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841432+0000","last_change":"2024-05-09T03:12:05.841432+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841432+0000","last_undegraded":"2024-05-09T03:12:05.841432+0000","last_fullsized":"2024-05-09T03:12:05.841432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852440+0000","last_change":"2024-05-09T03:12:05.849439+0000","last_active":"2024-05-09T03:12:05.852440+0000","last_peered":"2024-05-09T03:12:05.852440+0000","last_clean":"2024-05-09T03:12:05.852440+0000","last_became_active":"2024-05-09T03:12:05.849134+0000","last_became_peered":"2024-05-09T03:12:05.849134+0000","last_unstale":"2024-05-09T03:12:05.852440+0000","last_undegraded":"2024-05-09T03:12:05.852440+0000","last_fullsized":"2024-05-09T03:12:05.852440+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:43:46.511932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835494+0000","last_change":"2024-05-09T03:12:05.835494+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835494+0000","last_undegraded":"2024-05-09T03:12:05.835494+0000","last_fullsized":"2024-05-09T03:12:05.835494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854065+0000","last_change":"2024-05-09T03:12:05.852504+0000","last_active":"2024-05-09T03:12:05.854065+0000","last_peered":"2024-05-09T03:12:05.854065+0000","last_clean":"2024-05-09T03:12:05.854065+0000","last_became_active":"2024-05-09T03:12:05.851857+0000","last_became_peered":"2024-05-09T03:12:05.851857+0000","last_unstale":"2024-05-09T03:12:05.854065+0000","last_undegraded":"2024-05-09T03:12:05.854065+0000","last_fullsized":"2024-05-09T03:12:05.854065+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836813+0000","last_change":"2024-05-09T03:12:05.836813+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836813+0000","last_undegraded":"2024-05-09T03:12:05.836813+0000","last_fullsized":"2024-05-09T03:12:05.836813+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849695+0000","last_change":"2024-05-09T03:12:05.844269+0000","last_active":"2024-05-09T03:12:05.849695+0000","last_peered":"2024-05-09T03:12:05.849695+0000","last_clean":"2024-05-09T03:12:05.849695+0000","last_became_active":"2024-05-09T03:12:05.843939+0000","last_became_peered":"2024-05-09T03:12:05.843939+0000","last_unstale":"2024-05-09T03:12:05.849695+0000","last_undegraded":"2024-05-09T03:12:05.849695+0000","last_fullsized":"2024-05-09T03:12:05.849695+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839358+0000","last_change":"2024-05-09T03:12:05.839358+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839358+0000","last_undegraded":"2024-05-09T03:12:05.839358+0000","last_fullsized":"2024-05-09T03:12:05.839358+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851603+0000","last_change":"2024-05-09T03:12:05.847432+0000","last_active":"2024-05-09T03:12:05.851603+0000","last_peered":"2024-05-09T03:12:05.851603+0000","last_clean":"2024-05-09T03:12:05.851603+0000","last_became_active":"2024-05-09T03:12:05.847153+0000","last_became_peered":"2024-05-09T03:12:05.847153+0000","last_unstale":"2024-05-09T03:12:05.851603+0000","last_undegraded":"2024-05-09T03:12:05.851603+0000","last_fullsized":"2024-05-09T03:12:05.851603+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:02:27.480835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833512+0000","last_change":"2024-05-09T03:12:05.833512+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833512+0000","last_undegraded":"2024-05-09T03:12:05.833512+0000","last_fullsized":"2024-05-09T03:12:05.833512+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853273+0000","last_change":"2024-05-09T03:12:05.850218+0000","last_active":"2024-05-09T03:12:05.853273+0000","last_peered":"2024-05-09T03:12:05.853273+0000","last_clean":"2024-05-09T03:12:05.853273+0000","last_became_active":"2024-05-09T03:12:05.849743+0000","last_became_peered":"2024-05-09T03:12:05.849743+0000","last_unstale":"2024-05-09T03:12:05.853273+0000","last_undegraded":"2024-05-09T03:12:05.853273+0000","last_fullsized":"2024-05-09T03:12:05.853273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:40.635239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833538+0000","last_change":"2024-05-09T03:12:05.833538+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833538+0000","last_undegraded":"2024-05-09T03:12:05.833538+0000","last_fullsized":"2024-05-09T03:12:05.833538+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852904+0000","last_change":"2024-05-09T03:12:05.848391+0000","last_active":"2024-05-09T03:12:05.852904+0000","last_peered":"2024-05-09T03:12:05.852904+0000","last_clean":"2024-05-09T03:12:05.852904+0000","last_became_active":"2024-05-09T03:12:05.847525+0000","last_became_peered":"2024-05-09T03:12:05.847525+0000","last_unstale":"2024-05-09T03:12:05.852904+0000","last_undegraded":"2024-05-09T03:12:05.852904+0000","last_fullsized":"2024-05-09T03:12:05.852904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841979+0000","last_change":"2024-05-09T03:12:05.841979+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841979+0000","last_undegraded":"2024-05-09T03:12:05.841979+0000","last_fullsized":"2024-05-09T03:12:05.841979+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.847707+0000","last_change":"2024-05-09T03:12:05.846693+0000","last_active":"2024-05-09T03:12:05.847707+0000","last_peered":"2024-05-09T03:12:05.847707+0000","last_clean":"2024-05-09T03:12:05.847707+0000","last_became_active":"2024-05-09T03:12:05.846379+0000","last_became_peered":"2024-05-09T03:12:05.846379+0000","last_unstale":"2024-05-09T03:12:05.847707+0000","last_undegraded":"2024-05-09T03:12:05.847707+0000","last_fullsized":"2024-05-09T03:12:05.847707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:57.513277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840940+0000","last_change":"2024-05-09T03:12:05.840940+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840940+0000","last_undegraded":"2024-05-09T03:12:05.840940+0000","last_fullsized":"2024-05-09T03:12:05.840940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846195+0000","last_change":"2024-05-09T03:12:05.843597+0000","last_active":"2024-05-09T03:12:05.846195+0000","last_peered":"2024-05-09T03:12:05.846195+0000","last_clean":"2024-05-09T03:12:05.846195+0000","last_became_active":"2024-05-09T03:12:05.843281+0000","last_became_peered":"2024-05-09T03:12:05.843281+0000","last_unstale":"2024-05-09T03:12:05.846195+0000","last_undegraded":"2024-05-09T03:12:05.846195+0000","last_fullsized":"2024-05-09T03:12:05.846195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832473+0000","last_change":"2024-05-09T03:12:05.832473+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832473+0000","last_undegraded":"2024-05-09T03:12:05.832473+0000","last_fullsized":"2024-05-09T03:12:05.832473+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852261+0000","last_change":"2024-05-09T03:12:05.845237+0000","last_active":"2024-05-09T03:12:05.852261+0000","last_peered":"2024-05-09T03:12:05.852261+0000","last_clean":"2024-05-09T03:12:05.852261+0000","last_became_active":"2024-05-09T03:12:05.844921+0000","last_became_peered":"2024-05-09T03:12:05.844921+0000","last_unstale":"2024-05-09T03:12:05.852261+0000","last_undegraded":"2024-05-09T03:12:05.852261+0000","last_fullsized":"2024-05-09T03:12:05.852261+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:08:03.214517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835634+0000","last_change":"2024-05-09T03:12:05.835634+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835634+0000","last_undegraded":"2024-05-09T03:12:05.835634+0000","last_fullsized":"2024-05-09T03:12:05.835634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854023+0000","last_change":"2024-05-09T03:12:05.850855+0000","last_active":"2024-05-09T03:12:05.854023+0000","last_peered":"2024-05-09T03:12:05.854023+0000","last_clean":"2024-05-09T03:12:05.854023+0000","last_became_active":"2024-05-09T03:12:05.850518+0000","last_became_peered":"2024-05-09T03:12:05.850518+0000","last_unstale":"2024-05-09T03:12:05.854023+0000","last_undegraded":"2024-05-09T03:12:05.854023+0000","last_fullsized":"2024-05-09T03:12:05.854023+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834748+0000","last_change":"2024-05-09T03:12:05.834748+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834748+0000","last_undegraded":"2024-05-09T03:12:05.834748+0000","last_fullsized":"2024-05-09T03:12:05.834748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851159+0000","last_change":"2024-05-09T03:12:05.848645+0000","last_active":"2024-05-09T03:12:05.851159+0000","last_peered":"2024-05-09T03:12:05.851159+0000","last_clean":"2024-05-09T03:12:05.851159+0000","last_became_active":"2024-05-09T03:12:05.848326+0000","last_became_peered":"2024-05-09T03:12:05.848326+0000","last_unstale":"2024-05-09T03:12:05.851159+0000","last_undegraded":"2024-05-09T03:12:05.851159+0000","last_fullsized":"2024-05-09T03:12:05.851159+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:20.727622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835236+0000","last_change":"2024-05-09T03:12:05.835236+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835236+0000","last_undegraded":"2024-05-09T03:12:05.835236+0000","last_fullsized":"2024-05-09T03:12:05.835236+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853515+0000","last_change":"2024-05-09T03:12:05.843892+0000","last_active":"2024-05-09T03:12:05.853515+0000","last_peered":"2024-05-09T03:12:05.853515+0000","last_clean":"2024-05-09T03:12:05.853515+0000","last_became_active":"2024-05-09T03:12:05.843459+0000","last_became_peered":"2024-05-09T03:12:05.843459+0000","last_unstale":"2024-05-09T03:12:05.853515+0000","last_undegraded":"2024-05-09T03:12:05.853515+0000","last_fullsized":"2024-05-09T03:12:05.853515+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:48.718539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832475+0000","last_change":"2024-05-09T03:12:05.832475+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832475+0000","last_undegraded":"2024-05-09T03:12:05.832475+0000","last_fullsized":"2024-05-09T03:12:05.832475+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.845868+0000","last_change":"2024-05-09T03:12:05.844488+0000","last_active":"2024-05-09T03:12:05.845868+0000","last_peered":"2024-05-09T03:12:05.845868+0000","last_clean":"2024-05-09T03:12:05.845868+0000","last_became_active":"2024-05-09T03:12:05.844183+0000","last_became_peered":"2024-05-09T03:12:05.844183+0000","last_unstale":"2024-05-09T03:12:05.845868+0000","last_undegraded":"2024-05-09T03:12:05.845868+0000","last_fullsized":"2024-05-09T03:12:05.845868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:08:08.665854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834642+0000","last_change":"2024-05-09T03:12:05.834642+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834642+0000","last_undegraded":"2024-05-09T03:12:05.834642+0000","last_fullsized":"2024-05-09T03:12:05.834642+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852985+0000","last_change":"2024-05-09T03:12:05.849987+0000","last_active":"2024-05-09T03:12:05.852985+0000","last_peered":"2024-05-09T03:12:05.852985+0000","last_clean":"2024-05-09T03:12:05.852985+0000","last_became_active":"2024-05-09T03:12:05.848380+0000","last_became_peered":"2024-05-09T03:12:05.848380+0000","last_unstale":"2024-05-09T03:12:05.852985+0000","last_undegraded":"2024-05-09T03:12:05.852985+0000","last_fullsized":"2024-05-09T03:12:05.852985+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:09:38.070718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835177+0000","last_change":"2024-05-09T03:12:05.835177+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835177+0000","last_undegraded":"2024-05-09T03:12:05.835177+0000","last_fullsized":"2024-05-09T03:12:05.835177+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852583+0000","last_change":"2024-05-09T03:12:05.851689+0000","last_active":"2024-05-09T03:12:05.852583+0000","last_peered":"2024-05-09T03:12:05.852583+0000","last_clean":"2024-05-09T03:12:05.852583+0000","last_became_active":"2024-05-09T03:12:05.851357+0000","last_became_peered":"2024-05-09T03:12:05.851357+0000","last_unstale":"2024-05-09T03:12:05.852583+0000","last_undegraded":"2024-05-09T03:12:05.852583+0000","last_fullsized":"2024-05-09T03:12:05.852583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:49.416402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":170,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.836038+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:12:05.836038+0000","last_peered":"2024-05-09T03:12:05.836038+0000","last_clean":"2024-05-09T03:12:05.836038+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:12:05.836038+0000","last_undegraded":"2024-05-09T03:12:05.836038+0000","last_fullsized":"2024-05-09T03:12:05.836038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1228800,"data_stored":850793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":260425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":844526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:12:11.329 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:12:11.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.478+0000 7f2b164b8640 1 Processor -- start 2024-05-09T03:12:11.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.478+0000 7f2b164b8640 1 -- start start 2024-05-09T03:12:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b164b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b101613c0 0x7f2b100b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b164b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100b7d00 0x7f2b10163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b164b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b100b8110 con 0x7f2b100b5210 2024-05-09T03:12:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b164b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2b100b5620 con 0x7f2b101613c0 2024-05-09T03:12:11.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b164b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2b101609a0 con 0x7f2b100b7d00 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b0ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2b100b5210 0x7f2b100d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48622/0 (socket says 172.21.15.46:48622) 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b0ffff640 1 -- 172.21.15.46:0/1516826416 learned_addr learned my addr 172.21.15.46:0/1516826416 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b14a55640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100b7d00 0x7f2b10163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b0f7fe640 1 --2- 172.21.15.46:0/1516826416 >> v2:172.21.15.88:3300/0 conn(0x7f2b101613c0 0x7f2b100b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b0effd640 1 -- 172.21.15.46:0/1516826416 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2534758675 0 0) 0x7f2b00001110 con 0x7f2b100b5210 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.480+0000 7f2b0effd640 1 -- 172.21.15.46:0/1516826416 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2af8003520 con 0x7f2b100b5210 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.481+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100b7d00 msgr2=0x7f2b10163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.481+0000 7f2b0f7fe640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100b7d00 0x7f2b10163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.481+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/1516826416 >> v1:172.21.15.46:6789/0 conn(0x7f2b100b5210 legacy=0x7f2b100d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.481+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/1516826416 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b101617d0 con 0x7f2b101613c0 2024-05-09T03:12:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.481+0000 7f2b14a55640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100b7d00 0x7f2b10163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.481+0000 7f2b0f7fe640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b101613c0 0x7f2b100b77c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b0402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fc16b3347c61244 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b0effd640 1 -- 172.21.15.46:0/1516826416 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04037070 con 0x7f2b101613c0 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b0effd640 1 -- 172.21.15.46:0/1516826416 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b04032440 con 0x7f2b101613c0 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b0effd640 1 -- 172.21.15.46:0/1516826416 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b040327f0 con 0x7f2b101613c0 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 -- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b101613c0 msgr2=0x7f2b100b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b101613c0 0x7f2b100b77c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b0402a180 comp rx=0 tx=0).stop 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 -- 172.21.15.46:0/1516826416 shutdown_connections 2024-05-09T03:12:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100b7d00 0x7f2b10163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 --2- 172.21.15.46:0/1516826416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b101613c0 0x7f2b100b77c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 -- 172.21.15.46:0/1516826416 >> 172.21.15.46:0/1516826416 conn(0x7f2b100d29a0 msgr2=0x7f2b100d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 -- 172.21.15.46:0/1516826416 shutdown_connections 2024-05-09T03:12:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.482+0000 7f2b164b8640 1 -- 172.21.15.46:0/1516826416 wait complete. 2024-05-09T03:12:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.484+0000 7f2b164b8640 1 Processor -- start 2024-05-09T03:12:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.484+0000 7f2b164b8640 1 -- start start 2024-05-09T03:12:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b164b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100b5210 0x7f2b10152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b164b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 0x7f2b100cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b164b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b101613c0 0x7f2b100c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b164b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b10160bc0 con 0x7f2b100b7d00 2024-05-09T03:12:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b164b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b100d5560 con 0x7f2b100b5210 2024-05-09T03:12:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b164b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2b10160ea0 con 0x7f2b101613c0 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 0x7f2b100cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 0x7f2b100cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53268/0 (socket says 172.21.15.46:53268) 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/2142310142 learned_addr learned my addr 172.21.15.46:0/2142310142 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0ffff640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100b5210 0x7f2b10152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b14a55640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b101613c0 0x7f2b100c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b101613c0 msgr2=0x7f2b100c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b101613c0 0x7f2b100c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100b5210 msgr2=0x7f2b10152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100b5210 0x7f2b10152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 -- 172.21.15.46:0/2142310142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b100ca7d0 con 0x7f2b100b7d00 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b14a55640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b101613c0 0x7f2b100c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0ffff640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100b5210 0x7f2b10152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.486+0000 7f2b0f7fe640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 0x7f2b100cba40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.487+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04037050 con 0x7f2b100b7d00 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.487+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b04038040 con 0x7f2b100b7d00 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.487+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b100ca4f0 con 0x7f2b100b7d00 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.487+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04004490 con 0x7f2b100b7d00 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.487+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b100cb240 con 0x7f2b100b7d00 2024-05-09T03:12:11.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.488+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1006a270 con 0x7f2b100b7d00 2024-05-09T03:12:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.489+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2b0403c6d0 con 0x7f2b100b7d00 2024-05-09T03:12:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.489+0000 7f2b0cff9640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2afc0762d0 0x7f2afc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.489+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f2b040b7dc0 con 0x7f2b100b7d00 2024-05-09T03:12:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.490+0000 7f2b0ffff640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2afc0762d0 0x7f2afc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.490+0000 7f2b0ffff640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2afc0762d0 0x7f2afc0787e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2b00007fd0 tx=0x7f2b00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.494+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2b04081720 con 0x7f2b100b7d00 2024-05-09T03:12:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.682+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2b100d40d0 con 0x7f2afc0762d0 2024-05-09T03:12:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.689+0000 7f2b0cff9640 1 -- 172.21.15.46:0/2142310142 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7f2b100d40d0 con 0x7f2afc0762d0 2024-05-09T03:12:11.693 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:11.695 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2afc0762d0 msgr2=0x7f2afc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2afc0762d0 0x7f2afc0787e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2b00007fd0 tx=0x7f2b00009040 comp rx=0 tx=0).stop 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 msgr2=0x7f2b100cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 0x7f2b100cba40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f2b040097a0 tx=0x7f2b040046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 shutdown_connections 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b101613c0 0x7f2b100c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2afc0762d0 0x7f2afc0787e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100b7d00 0x7f2b100cba40 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 --2- 172.21.15.46:0/2142310142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100b5210 0x7f2b10152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 >> 172.21.15.46:0/2142310142 conn(0x7f2b100d29a0 msgr2=0x7f2b100d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 shutdown_connections 2024-05-09T03:12:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.692+0000 7f2b164b8640 1 -- 172.21.15.46:0/2142310142 wait complete. 2024-05-09T03:12:11.705 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-05-09T03:12:10.531012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238908,"kb_used_data":6048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735812,"statfs":{"total":773094113280,"available":772849471488,"internally_reserved":0,"allocated":6193152,"data_stored":3226534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.706220"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841057+0000","last_change":"2024-05-09T03:12:05.841057+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841057+0000","last_undegraded":"2024-05-09T03:12:05.841057+0000","last_fullsized":"2024-05-09T03:12:05.841057+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836609+0000","last_change":"2024-05-09T03:12:05.836609+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836609+0000","last_undegraded":"2024-05-09T03:12:05.836609+0000","last_fullsized":"2024-05-09T03:12:05.836609+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837976+0000","last_change":"2024-05-09T03:12:05.837976+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837976+0000","last_undegraded":"2024-05-09T03:12:05.837976+0000","last_fullsized":"2024-05-09T03:12:05.837976+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.843259+0000","last_change":"2024-05-09T03:12:05.843259+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.843259+0000","last_undegraded":"2024-05-09T03:12:05.843259+0000","last_fullsized":"2024-05-09T03:12:05.843259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839531+0000","last_change":"2024-05-09T03:12:05.839531+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839531+0000","last_undegraded":"2024-05-09T03:12:05.839531+0000","last_fullsized":"2024-05-09T03:12:05.839531+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834452+0000","last_change":"2024-05-09T03:12:05.834452+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834452+0000","last_undegraded":"2024-05-09T03:12:05.834452+0000","last_fullsized":"2024-05-09T03:12:05.834452+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836296+0000","last_change":"2024-05-09T03:12:05.836296+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836296+0000","last_undegraded":"2024-05-09T03:12:05.836296+0000","last_fullsized":"2024-05-09T03:12:05.836296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835798+0000","last_change":"2024-05-09T03:12:05.835798+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835798+0000","last_undegraded":"2024-05-09T03:12:05.835798+0000","last_fullsized":"2024-05-09T03:12:05.835798+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837939+0000","last_change":"2024-05-09T03:12:05.837939+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837939+0000","last_undegraded":"2024-05-09T03:12:05.837939+0000","last_fullsized":"2024-05-09T03:12:05.837939+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833750+0000","last_change":"2024-05-09T03:12:05.833750+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833750+0000","last_undegraded":"2024-05-09T03:12:05.833750+0000","last_fullsized":"2024-05-09T03:12:05.833750+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833154+0000","last_change":"2024-05-09T03:12:05.833154+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833154+0000","last_undegraded":"2024-05-09T03:12:05.833154+0000","last_fullsized":"2024-05-09T03:12:05.833154+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842732+0000","last_change":"2024-05-09T03:12:05.842732+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842732+0000","last_undegraded":"2024-05-09T03:12:05.842732+0000","last_fullsized":"2024-05-09T03:12:05.842732+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833533+0000","last_change":"2024-05-09T03:12:05.833533+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833533+0000","last_undegraded":"2024-05-09T03:12:05.833533+0000","last_fullsized":"2024-05-09T03:12:05.833533+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836888+0000","last_change":"2024-05-09T03:12:05.836888+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836888+0000","last_undegraded":"2024-05-09T03:12:05.836888+0000","last_fullsized":"2024-05-09T03:12:05.836888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835706+0000","last_change":"2024-05-09T03:12:05.835706+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835706+0000","last_undegraded":"2024-05-09T03:12:05.835706+0000","last_fullsized":"2024-05-09T03:12:05.835706+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833981+0000","last_change":"2024-05-09T03:12:05.833981+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833981+0000","last_undegraded":"2024-05-09T03:12:05.833981+0000","last_fullsized":"2024-05-09T03:12:05.833981+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840026+0000","last_change":"2024-05-09T03:12:05.840026+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840026+0000","last_undegraded":"2024-05-09T03:12:05.840026+0000","last_fullsized":"2024-05-09T03:12:05.840026+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838628+0000","last_change":"2024-05-09T03:12:05.838628+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838628+0000","last_undegraded":"2024-05-09T03:12:05.838628+0000","last_fullsized":"2024-05-09T03:12:05.838628+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839714+0000","last_change":"2024-05-09T03:12:05.839714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839714+0000","last_undegraded":"2024-05-09T03:12:05.839714+0000","last_fullsized":"2024-05-09T03:12:05.839714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837245+0000","last_change":"2024-05-09T03:12:05.837245+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837245+0000","last_undegraded":"2024-05-09T03:12:05.837245+0000","last_fullsized":"2024-05-09T03:12:05.837245+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838405+0000","last_change":"2024-05-09T03:12:05.838405+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838405+0000","last_undegraded":"2024-05-09T03:12:05.838405+0000","last_fullsized":"2024-05-09T03:12:05.838405+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837199+0000","last_change":"2024-05-09T03:12:05.837199+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837199+0000","last_undegraded":"2024-05-09T03:12:05.837199+0000","last_fullsized":"2024-05-09T03:12:05.837199+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836251+0000","last_change":"2024-05-09T03:12:05.836251+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836251+0000","last_undegraded":"2024-05-09T03:12:05.836251+0000","last_fullsized":"2024-05-09T03:12:05.836251+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834399+0000","last_change":"2024-05-09T03:12:05.834399+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834399+0000","last_undegraded":"2024-05-09T03:12:05.834399+0000","last_fullsized":"2024-05-09T03:12:05.834399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840127+0000","last_change":"2024-05-09T03:12:05.840127+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840127+0000","last_undegraded":"2024-05-09T03:12:05.840127+0000","last_fullsized":"2024-05-09T03:12:05.840127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.844753+0000","last_change":"2024-05-09T03:12:05.844753+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.844753+0000","last_undegraded":"2024-05-09T03:12:05.844753+0000","last_fullsized":"2024-05-09T03:12:05.844753+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836714+0000","last_change":"2024-05-09T03:12:05.836714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836714+0000","last_undegraded":"2024-05-09T03:12:05.836714+0000","last_fullsized":"2024-05-09T03:12:05.836714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842327+0000","last_change":"2024-05-09T03:12:05.842327+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842327+0000","last_undegraded":"2024-05-09T03:12:05.842327+0000","last_fullsized":"2024-05-09T03:12:05.842327+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837936+0000","last_change":"2024-05-09T03:12:05.837936+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837936+0000","last_undegraded":"2024-05-09T03:12:05.837936+0000","last_fullsized":"2024-05-09T03:12:05.837936+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838187+0000","last_change":"2024-05-09T03:12:05.838187+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838187+0000","last_undegraded":"2024-05-09T03:12:05.838187+0000","last_fullsized":"2024-05-09T03:12:05.838187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840098+0000","last_change":"2024-05-09T03:12:05.840098+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840098+0000","last_undegraded":"2024-05-09T03:12:05.840098+0000","last_fullsized":"2024-05-09T03:12:05.840098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832938+0000","last_change":"2024-05-09T03:12:05.832938+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832938+0000","last_undegraded":"2024-05-09T03:12:05.832938+0000","last_fullsized":"2024-05-09T03:12:05.832938+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837077+0000","last_change":"2024-05-09T03:12:05.837077+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837077+0000","last_undegraded":"2024-05-09T03:12:05.837077+0000","last_fullsized":"2024-05-09T03:12:05.837077+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852354+0000","last_change":"2024-05-09T03:12:05.849076+0000","last_active":"2024-05-09T03:12:05.852354+0000","last_peered":"2024-05-09T03:12:05.852354+0000","last_clean":"2024-05-09T03:12:05.852354+0000","last_became_active":"2024-05-09T03:12:05.848768+0000","last_became_peered":"2024-05-09T03:12:05.848768+0000","last_unstale":"2024-05-09T03:12:05.852354+0000","last_undegraded":"2024-05-09T03:12:05.852354+0000","last_fullsized":"2024-05-09T03:12:05.852354+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:05.791451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832809+0000","last_change":"2024-05-09T03:12:05.832809+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832809+0000","last_undegraded":"2024-05-09T03:12:05.832809+0000","last_fullsized":"2024-05-09T03:12:05.832809+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851128+0000","last_change":"2024-05-09T03:12:05.846715+0000","last_active":"2024-05-09T03:12:05.851128+0000","last_peered":"2024-05-09T03:12:05.851128+0000","last_clean":"2024-05-09T03:12:05.851128+0000","last_became_active":"2024-05-09T03:12:05.846448+0000","last_became_peered":"2024-05-09T03:12:05.846448+0000","last_unstale":"2024-05-09T03:12:05.851128+0000","last_undegraded":"2024-05-09T03:12:05.851128+0000","last_fullsized":"2024-05-09T03:12:05.851128+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:53:45.376505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833746+0000","last_change":"2024-05-09T03:12:05.833746+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833746+0000","last_undegraded":"2024-05-09T03:12:05.833746+0000","last_fullsized":"2024-05-09T03:12:05.833746+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849635+0000","last_change":"2024-05-09T03:12:05.845815+0000","last_active":"2024-05-09T03:12:05.849635+0000","last_peered":"2024-05-09T03:12:05.849635+0000","last_clean":"2024-05-09T03:12:05.849635+0000","last_became_active":"2024-05-09T03:12:05.845509+0000","last_became_peered":"2024-05-09T03:12:05.845509+0000","last_unstale":"2024-05-09T03:12:05.849635+0000","last_undegraded":"2024-05-09T03:12:05.849635+0000","last_fullsized":"2024-05-09T03:12:05.849635+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:07.083261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836173+0000","last_change":"2024-05-09T03:12:05.836173+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836173+0000","last_undegraded":"2024-05-09T03:12:05.836173+0000","last_fullsized":"2024-05-09T03:12:05.836173+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853894+0000","last_change":"2024-05-09T03:12:05.849601+0000","last_active":"2024-05-09T03:12:05.853894+0000","last_peered":"2024-05-09T03:12:05.853894+0000","last_clean":"2024-05-09T03:12:05.853894+0000","last_became_active":"2024-05-09T03:12:05.849230+0000","last_became_peered":"2024-05-09T03:12:05.849230+0000","last_unstale":"2024-05-09T03:12:05.853894+0000","last_undegraded":"2024-05-09T03:12:05.853894+0000","last_fullsized":"2024-05-09T03:12:05.853894+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:43.363860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835661+0000","last_change":"2024-05-09T03:12:05.835661+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835661+0000","last_undegraded":"2024-05-09T03:12:05.835661+0000","last_fullsized":"2024-05-09T03:12:05.835661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.856188+0000","last_change":"2024-05-09T03:12:05.853186+0000","last_active":"2024-05-09T03:12:05.856188+0000","last_peered":"2024-05-09T03:12:05.856188+0000","last_clean":"2024-05-09T03:12:05.856188+0000","last_became_active":"2024-05-09T03:12:05.852815+0000","last_became_peered":"2024-05-09T03:12:05.852815+0000","last_unstale":"2024-05-09T03:12:05.856188+0000","last_undegraded":"2024-05-09T03:12:05.856188+0000","last_fullsized":"2024-05-09T03:12:05.856188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838497+0000","last_change":"2024-05-09T03:12:05.838497+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838497+0000","last_undegraded":"2024-05-09T03:12:05.838497+0000","last_fullsized":"2024-05-09T03:12:05.838497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849662+0000","last_change":"2024-05-09T03:12:05.844443+0000","last_active":"2024-05-09T03:12:05.849662+0000","last_peered":"2024-05-09T03:12:05.849662+0000","last_clean":"2024-05-09T03:12:05.849662+0000","last_became_active":"2024-05-09T03:12:05.844119+0000","last_became_peered":"2024-05-09T03:12:05.844119+0000","last_unstale":"2024-05-09T03:12:05.849662+0000","last_undegraded":"2024-05-09T03:12:05.849662+0000","last_fullsized":"2024-05-09T03:12:05.849662+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833532+0000","last_change":"2024-05-09T03:12:05.833532+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833532+0000","last_undegraded":"2024-05-09T03:12:05.833532+0000","last_fullsized":"2024-05-09T03:12:05.833532+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852899+0000","last_change":"2024-05-09T03:12:05.847498+0000","last_active":"2024-05-09T03:12:05.852899+0000","last_peered":"2024-05-09T03:12:05.852899+0000","last_clean":"2024-05-09T03:12:05.852899+0000","last_became_active":"2024-05-09T03:12:05.847170+0000","last_became_peered":"2024-05-09T03:12:05.847170+0000","last_unstale":"2024-05-09T03:12:05.852899+0000","last_undegraded":"2024-05-09T03:12:05.852899+0000","last_fullsized":"2024-05-09T03:12:05.852899+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835683+0000","last_change":"2024-05-09T03:12:05.835683+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835683+0000","last_undegraded":"2024-05-09T03:12:05.835683+0000","last_fullsized":"2024-05-09T03:12:05.835683+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853539+0000","last_change":"2024-05-09T03:12:05.847308+0000","last_active":"2024-05-09T03:12:05.853539+0000","last_peered":"2024-05-09T03:12:05.853539+0000","last_clean":"2024-05-09T03:12:05.853539+0000","last_became_active":"2024-05-09T03:12:05.846406+0000","last_became_peered":"2024-05-09T03:12:05.846406+0000","last_unstale":"2024-05-09T03:12:05.853539+0000","last_undegraded":"2024-05-09T03:12:05.853539+0000","last_fullsized":"2024-05-09T03:12:05.853539+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838513+0000","last_change":"2024-05-09T03:12:05.838513+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838513+0000","last_undegraded":"2024-05-09T03:12:05.838513+0000","last_fullsized":"2024-05-09T03:12:05.838513+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851570+0000","last_change":"2024-05-09T03:12:05.847459+0000","last_active":"2024-05-09T03:12:05.851570+0000","last_peered":"2024-05-09T03:12:05.851570+0000","last_clean":"2024-05-09T03:12:05.851570+0000","last_became_active":"2024-05-09T03:12:05.847106+0000","last_became_peered":"2024-05-09T03:12:05.847106+0000","last_unstale":"2024-05-09T03:12:05.851570+0000","last_undegraded":"2024-05-09T03:12:05.851570+0000","last_fullsized":"2024-05-09T03:12:05.851570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835774+0000","last_change":"2024-05-09T03:12:05.835774+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835774+0000","last_undegraded":"2024-05-09T03:12:05.835774+0000","last_fullsized":"2024-05-09T03:12:05.835774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852846+0000","last_change":"2024-05-09T03:12:05.849037+0000","last_active":"2024-05-09T03:12:05.852846+0000","last_peered":"2024-05-09T03:12:05.852846+0000","last_clean":"2024-05-09T03:12:05.852846+0000","last_became_active":"2024-05-09T03:12:05.848283+0000","last_became_peered":"2024-05-09T03:12:05.848283+0000","last_unstale":"2024-05-09T03:12:05.852846+0000","last_undegraded":"2024-05-09T03:12:05.852846+0000","last_fullsized":"2024-05-09T03:12:05.852846+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:50.152051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833184+0000","last_change":"2024-05-09T03:12:05.833184+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833184+0000","last_undegraded":"2024-05-09T03:12:05.833184+0000","last_fullsized":"2024-05-09T03:12:05.833184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851612+0000","last_change":"2024-05-09T03:12:05.847630+0000","last_active":"2024-05-09T03:12:05.851612+0000","last_peered":"2024-05-09T03:12:05.851612+0000","last_clean":"2024-05-09T03:12:05.851612+0000","last_became_active":"2024-05-09T03:12:05.847302+0000","last_became_peered":"2024-05-09T03:12:05.847302+0000","last_unstale":"2024-05-09T03:12:05.851612+0000","last_undegraded":"2024-05-09T03:12:05.851612+0000","last_fullsized":"2024-05-09T03:12:05.851612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837431+0000","last_change":"2024-05-09T03:12:05.837431+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837431+0000","last_undegraded":"2024-05-09T03:12:05.837431+0000","last_fullsized":"2024-05-09T03:12:05.837431+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853524+0000","last_change":"2024-05-09T03:12:05.845847+0000","last_active":"2024-05-09T03:12:05.853524+0000","last_peered":"2024-05-09T03:12:05.853524+0000","last_clean":"2024-05-09T03:12:05.853524+0000","last_became_active":"2024-05-09T03:12:05.845474+0000","last_became_peered":"2024-05-09T03:12:05.845474+0000","last_unstale":"2024-05-09T03:12:05.853524+0000","last_undegraded":"2024-05-09T03:12:05.853524+0000","last_fullsized":"2024-05-09T03:12:05.853524+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:56.893788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839951+0000","last_change":"2024-05-09T03:12:05.839951+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839951+0000","last_undegraded":"2024-05-09T03:12:05.839951+0000","last_fullsized":"2024-05-09T03:12:05.839951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854903+0000","last_change":"2024-05-09T03:12:05.854030+0000","last_active":"2024-05-09T03:12:05.854903+0000","last_peered":"2024-05-09T03:12:05.854903+0000","last_clean":"2024-05-09T03:12:05.854903+0000","last_became_active":"2024-05-09T03:12:05.853677+0000","last_became_peered":"2024-05-09T03:12:05.853677+0000","last_unstale":"2024-05-09T03:12:05.854903+0000","last_undegraded":"2024-05-09T03:12:05.854903+0000","last_fullsized":"2024-05-09T03:12:05.854903+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:25.968267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837833+0000","last_change":"2024-05-09T03:12:05.837833+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837833+0000","last_undegraded":"2024-05-09T03:12:05.837833+0000","last_fullsized":"2024-05-09T03:12:05.837833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846449+0000","last_change":"2024-05-09T03:12:05.843325+0000","last_active":"2024-05-09T03:12:05.846449+0000","last_peered":"2024-05-09T03:12:05.846449+0000","last_clean":"2024-05-09T03:12:05.846449+0000","last_became_active":"2024-05-09T03:12:05.842923+0000","last_became_peered":"2024-05-09T03:12:05.842923+0000","last_unstale":"2024-05-09T03:12:05.846449+0000","last_undegraded":"2024-05-09T03:12:05.846449+0000","last_fullsized":"2024-05-09T03:12:05.846449+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:43:02.488792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837016+0000","last_change":"2024-05-09T03:12:05.837016+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837016+0000","last_undegraded":"2024-05-09T03:12:05.837016+0000","last_fullsized":"2024-05-09T03:12:05.837016+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853796+0000","last_change":"2024-05-09T03:12:05.848661+0000","last_active":"2024-05-09T03:12:05.853796+0000","last_peered":"2024-05-09T03:12:05.853796+0000","last_clean":"2024-05-09T03:12:05.853796+0000","last_became_active":"2024-05-09T03:12:05.848010+0000","last_became_peered":"2024-05-09T03:12:05.848010+0000","last_unstale":"2024-05-09T03:12:05.853796+0000","last_undegraded":"2024-05-09T03:12:05.853796+0000","last_fullsized":"2024-05-09T03:12:05.853796+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:20.873828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835668+0000","last_change":"2024-05-09T03:12:05.835668+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835668+0000","last_undegraded":"2024-05-09T03:12:05.835668+0000","last_fullsized":"2024-05-09T03:12:05.835668+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852646+0000","last_change":"2024-05-09T03:12:05.849792+0000","last_active":"2024-05-09T03:12:05.852646+0000","last_peered":"2024-05-09T03:12:05.852646+0000","last_clean":"2024-05-09T03:12:05.852646+0000","last_became_active":"2024-05-09T03:12:05.849439+0000","last_became_peered":"2024-05-09T03:12:05.849439+0000","last_unstale":"2024-05-09T03:12:05.852646+0000","last_undegraded":"2024-05-09T03:12:05.852646+0000","last_fullsized":"2024-05-09T03:12:05.852646+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836209+0000","last_change":"2024-05-09T03:12:05.836209+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836209+0000","last_undegraded":"2024-05-09T03:12:05.836209+0000","last_fullsized":"2024-05-09T03:12:05.836209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854627+0000","last_change":"2024-05-09T03:12:05.853093+0000","last_active":"2024-05-09T03:12:05.854627+0000","last_peered":"2024-05-09T03:12:05.854627+0000","last_clean":"2024-05-09T03:12:05.854627+0000","last_became_active":"2024-05-09T03:12:05.852692+0000","last_became_peered":"2024-05-09T03:12:05.852692+0000","last_unstale":"2024-05-09T03:12:05.854627+0000","last_undegraded":"2024-05-09T03:12:05.854627+0000","last_fullsized":"2024-05-09T03:12:05.854627+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:43.389393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841432+0000","last_change":"2024-05-09T03:12:05.841432+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841432+0000","last_undegraded":"2024-05-09T03:12:05.841432+0000","last_fullsized":"2024-05-09T03:12:05.841432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852440+0000","last_change":"2024-05-09T03:12:05.849439+0000","last_active":"2024-05-09T03:12:05.852440+0000","last_peered":"2024-05-09T03:12:05.852440+0000","last_clean":"2024-05-09T03:12:05.852440+0000","last_became_active":"2024-05-09T03:12:05.849134+0000","last_became_peered":"2024-05-09T03:12:05.849134+0000","last_unstale":"2024-05-09T03:12:05.852440+0000","last_undegraded":"2024-05-09T03:12:05.852440+0000","last_fullsized":"2024-05-09T03:12:05.852440+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:43:46.511932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835494+0000","last_change":"2024-05-09T03:12:05.835494+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835494+0000","last_undegraded":"2024-05-09T03:12:05.835494+0000","last_fullsized":"2024-05-09T03:12:05.835494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854065+0000","last_change":"2024-05-09T03:12:05.852504+0000","last_active":"2024-05-09T03:12:05.854065+0000","last_peered":"2024-05-09T03:12:05.854065+0000","last_clean":"2024-05-09T03:12:05.854065+0000","last_became_active":"2024-05-09T03:12:05.851857+0000","last_became_peered":"2024-05-09T03:12:05.851857+0000","last_unstale":"2024-05-09T03:12:05.854065+0000","last_undegraded":"2024-05-09T03:12:05.854065+0000","last_fullsized":"2024-05-09T03:12:05.854065+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836813+0000","last_change":"2024-05-09T03:12:05.836813+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836813+0000","last_undegraded":"2024-05-09T03:12:05.836813+0000","last_fullsized":"2024-05-09T03:12:05.836813+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849695+0000","last_change":"2024-05-09T03:12:05.844269+0000","last_active":"2024-05-09T03:12:05.849695+0000","last_peered":"2024-05-09T03:12:05.849695+0000","last_clean":"2024-05-09T03:12:05.849695+0000","last_became_active":"2024-05-09T03:12:05.843939+0000","last_became_peered":"2024-05-09T03:12:05.843939+0000","last_unstale":"2024-05-09T03:12:05.849695+0000","last_undegraded":"2024-05-09T03:12:05.849695+0000","last_fullsized":"2024-05-09T03:12:05.849695+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839358+0000","last_change":"2024-05-09T03:12:05.839358+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839358+0000","last_undegraded":"2024-05-09T03:12:05.839358+0000","last_fullsized":"2024-05-09T03:12:05.839358+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851603+0000","last_change":"2024-05-09T03:12:05.847432+0000","last_active":"2024-05-09T03:12:05.851603+0000","last_peered":"2024-05-09T03:12:05.851603+0000","last_clean":"2024-05-09T03:12:05.851603+0000","last_became_active":"2024-05-09T03:12:05.847153+0000","last_became_peered":"2024-05-09T03:12:05.847153+0000","last_unstale":"2024-05-09T03:12:05.851603+0000","last_undegraded":"2024-05-09T03:12:05.851603+0000","last_fullsized":"2024-05-09T03:12:05.851603+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:02:27.480835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833512+0000","last_change":"2024-05-09T03:12:05.833512+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833512+0000","last_undegraded":"2024-05-09T03:12:05.833512+0000","last_fullsized":"2024-05-09T03:12:05.833512+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853273+0000","last_change":"2024-05-09T03:12:05.850218+0000","last_active":"2024-05-09T03:12:05.853273+0000","last_peered":"2024-05-09T03:12:05.853273+0000","last_clean":"2024-05-09T03:12:05.853273+0000","last_became_active":"2024-05-09T03:12:05.849743+0000","last_became_peered":"2024-05-09T03:12:05.849743+0000","last_unstale":"2024-05-09T03:12:05.853273+0000","last_undegraded":"2024-05-09T03:12:05.853273+0000","last_fullsized":"2024-05-09T03:12:05.853273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:40.635239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833538+0000","last_change":"2024-05-09T03:12:05.833538+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833538+0000","last_undegraded":"2024-05-09T03:12:05.833538+0000","last_fullsized":"2024-05-09T03:12:05.833538+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852904+0000","last_change":"2024-05-09T03:12:05.848391+0000","last_active":"2024-05-09T03:12:05.852904+0000","last_peered":"2024-05-09T03:12:05.852904+0000","last_clean":"2024-05-09T03:12:05.852904+0000","last_became_active":"2024-05-09T03:12:05.847525+0000","last_became_peered":"2024-05-09T03:12:05.847525+0000","last_unstale":"2024-05-09T03:12:05.852904+0000","last_undegraded":"2024-05-09T03:12:05.852904+0000","last_fullsized":"2024-05-09T03:12:05.852904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841979+0000","last_change":"2024-05-09T03:12:05.841979+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841979+0000","last_undegraded":"2024-05-09T03:12:05.841979+0000","last_fullsized":"2024-05-09T03:12:05.841979+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.847707+0000","last_change":"2024-05-09T03:12:05.846693+0000","last_active":"2024-05-09T03:12:05.847707+0000","last_peered":"2024-05-09T03:12:05.847707+0000","last_clean":"2024-05-09T03:12:05.847707+0000","last_became_active":"2024-05-09T03:12:05.846379+0000","last_became_peered":"2024-05-09T03:12:05.846379+0000","last_unstale":"2024-05-09T03:12:05.847707+0000","last_undegraded":"2024-05-09T03:12:05.847707+0000","last_fullsized":"2024-05-09T03:12:05.847707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:57.513277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840940+0000","last_change":"2024-05-09T03:12:05.840940+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840940+0000","last_undegraded":"2024-05-09T03:12:05.840940+0000","last_fullsized":"2024-05-09T03:12:05.840940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846195+0000","last_change":"2024-05-09T03:12:05.843597+0000","last_active":"2024-05-09T03:12:05.846195+0000","last_peered":"2024-05-09T03:12:05.846195+0000","last_clean":"2024-05-09T03:12:05.846195+0000","last_became_active":"2024-05-09T03:12:05.843281+0000","last_became_peered":"2024-05-09T03:12:05.843281+0000","last_unstale":"2024-05-09T03:12:05.846195+0000","last_undegraded":"2024-05-09T03:12:05.846195+0000","last_fullsized":"2024-05-09T03:12:05.846195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832473+0000","last_change":"2024-05-09T03:12:05.832473+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832473+0000","last_undegraded":"2024-05-09T03:12:05.832473+0000","last_fullsized":"2024-05-09T03:12:05.832473+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852261+0000","last_change":"2024-05-09T03:12:05.845237+0000","last_active":"2024-05-09T03:12:05.852261+0000","last_peered":"2024-05-09T03:12:05.852261+0000","last_clean":"2024-05-09T03:12:05.852261+0000","last_became_active":"2024-05-09T03:12:05.844921+0000","last_became_peered":"2024-05-09T03:12:05.844921+0000","last_unstale":"2024-05-09T03:12:05.852261+0000","last_undegraded":"2024-05-09T03:12:05.852261+0000","last_fullsized":"2024-05-09T03:12:05.852261+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:08:03.214517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835634+0000","last_change":"2024-05-09T03:12:05.835634+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835634+0000","last_undegraded":"2024-05-09T03:12:05.835634+0000","last_fullsized":"2024-05-09T03:12:05.835634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854023+0000","last_change":"2024-05-09T03:12:05.850855+0000","last_active":"2024-05-09T03:12:05.854023+0000","last_peered":"2024-05-09T03:12:05.854023+0000","last_clean":"2024-05-09T03:12:05.854023+0000","last_became_active":"2024-05-09T03:12:05.850518+0000","last_became_peered":"2024-05-09T03:12:05.850518+0000","last_unstale":"2024-05-09T03:12:05.854023+0000","last_undegraded":"2024-05-09T03:12:05.854023+0000","last_fullsized":"2024-05-09T03:12:05.854023+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834748+0000","last_change":"2024-05-09T03:12:05.834748+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834748+0000","last_undegraded":"2024-05-09T03:12:05.834748+0000","last_fullsized":"2024-05-09T03:12:05.834748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851159+0000","last_change":"2024-05-09T03:12:05.848645+0000","last_active":"2024-05-09T03:12:05.851159+0000","last_peered":"2024-05-09T03:12:05.851159+0000","last_clean":"2024-05-09T03:12:05.851159+0000","last_became_active":"2024-05-09T03:12:05.848326+0000","last_became_peered":"2024-05-09T03:12:05.848326+0000","last_unstale":"2024-05-09T03:12:05.851159+0000","last_undegraded":"2024-05-09T03:12:05.851159+0000","last_fullsized":"2024-05-09T03:12:05.851159+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:20.727622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835236+0000","last_change":"2024-05-09T03:12:05.835236+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835236+0000","last_undegraded":"2024-05-09T03:12:05.835236+0000","last_fullsized":"2024-05-09T03:12:05.835236+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853515+0000","last_change":"2024-05-09T03:12:05.843892+0000","last_active":"2024-05-09T03:12:05.853515+0000","last_peered":"2024-05-09T03:12:05.853515+0000","last_clean":"2024-05-09T03:12:05.853515+0000","last_became_active":"2024-05-09T03:12:05.843459+0000","last_became_peered":"2024-05-09T03:12:05.843459+0000","last_unstale":"2024-05-09T03:12:05.853515+0000","last_undegraded":"2024-05-09T03:12:05.853515+0000","last_fullsized":"2024-05-09T03:12:05.853515+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:48.718539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832475+0000","last_change":"2024-05-09T03:12:05.832475+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832475+0000","last_undegraded":"2024-05-09T03:12:05.832475+0000","last_fullsized":"2024-05-09T03:12:05.832475+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.845868+0000","last_change":"2024-05-09T03:12:05.844488+0000","last_active":"2024-05-09T03:12:05.845868+0000","last_peered":"2024-05-09T03:12:05.845868+0000","last_clean":"2024-05-09T03:12:05.845868+0000","last_became_active":"2024-05-09T03:12:05.844183+0000","last_became_peered":"2024-05-09T03:12:05.844183+0000","last_unstale":"2024-05-09T03:12:05.845868+0000","last_undegraded":"2024-05-09T03:12:05.845868+0000","last_fullsized":"2024-05-09T03:12:05.845868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:08:08.665854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834642+0000","last_change":"2024-05-09T03:12:05.834642+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834642+0000","last_undegraded":"2024-05-09T03:12:05.834642+0000","last_fullsized":"2024-05-09T03:12:05.834642+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852985+0000","last_change":"2024-05-09T03:12:05.849987+0000","last_active":"2024-05-09T03:12:05.852985+0000","last_peered":"2024-05-09T03:12:05.852985+0000","last_clean":"2024-05-09T03:12:05.852985+0000","last_became_active":"2024-05-09T03:12:05.848380+0000","last_became_peered":"2024-05-09T03:12:05.848380+0000","last_unstale":"2024-05-09T03:12:05.852985+0000","last_undegraded":"2024-05-09T03:12:05.852985+0000","last_fullsized":"2024-05-09T03:12:05.852985+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:09:38.070718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835177+0000","last_change":"2024-05-09T03:12:05.835177+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835177+0000","last_undegraded":"2024-05-09T03:12:05.835177+0000","last_fullsized":"2024-05-09T03:12:05.835177+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852583+0000","last_change":"2024-05-09T03:12:05.851689+0000","last_active":"2024-05-09T03:12:05.852583+0000","last_peered":"2024-05-09T03:12:05.852583+0000","last_clean":"2024-05-09T03:12:05.852583+0000","last_became_active":"2024-05-09T03:12:05.851357+0000","last_became_peered":"2024-05-09T03:12:05.851357+0000","last_unstale":"2024-05-09T03:12:05.852583+0000","last_undegraded":"2024-05-09T03:12:05.852583+0000","last_fullsized":"2024-05-09T03:12:05.852583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:49.416402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":170,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.836038+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:12:05.836038+0000","last_peered":"2024-05-09T03:12:05.836038+0000","last_clean":"2024-05-09T03:12:05.836038+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:12:05.836038+0000","last_undegraded":"2024-05-09T03:12:05.836038+0000","last_fullsized":"2024-05-09T03:12:05.836038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1228800,"data_stored":850793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":260425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":844526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:12:11.709 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3f is not active+clean 2024-05-09T03:12:11.709 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.841057+0000', 'last_change': '2024-05-09T03:12:05.841057+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.841057+0000', 'last_undegraded': '2024-05-09T03:12:05.841057+0000', 'last_fullsized': '2024-05-09T03:12:05.841057+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:12:11.709 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3e is not active+clean 2024-05-09T03:12:11.709 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836609+0000', 'last_change': '2024-05-09T03:12:05.836609+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836609+0000', 'last_undegraded': '2024-05-09T03:12:05.836609+0000', 'last_fullsized': '2024-05-09T03:12:05.836609+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.709 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3d is not active+clean 2024-05-09T03:12:11.709 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837976+0000', 'last_change': '2024-05-09T03:12:05.837976+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837976+0000', 'last_undegraded': '2024-05-09T03:12:05.837976+0000', 'last_fullsized': '2024-05-09T03:12:05.837976+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3c is not active+clean 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.843259+0000', 'last_change': '2024-05-09T03:12:05.843259+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.843259+0000', 'last_undegraded': '2024-05-09T03:12:05.843259+0000', 'last_fullsized': '2024-05-09T03:12:05.843259+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3b is not active+clean 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.839531+0000', 'last_change': '2024-05-09T03:12:05.839531+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.839531+0000', 'last_undegraded': '2024-05-09T03:12:05.839531+0000', 'last_fullsized': '2024-05-09T03:12:05.839531+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3a is not active+clean 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.834452+0000', 'last_change': '2024-05-09T03:12:05.834452+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.834452+0000', 'last_undegraded': '2024-05-09T03:12:05.834452+0000', 'last_fullsized': '2024-05-09T03:12:05.834452+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:PG 9.39 is not active+clean 2024-05-09T03:12:11.710 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836296+0000', 'last_change': '2024-05-09T03:12:05.836296+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836296+0000', 'last_undegraded': '2024-05-09T03:12:05.836296+0000', 'last_fullsized': '2024-05-09T03:12:05.836296+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:PG 9.38 is not active+clean 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835798+0000', 'last_change': '2024-05-09T03:12:05.835798+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835798+0000', 'last_undegraded': '2024-05-09T03:12:05.835798+0000', 'last_fullsized': '2024-05-09T03:12:05.835798+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:PG 9.37 is not active+clean 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837939+0000', 'last_change': '2024-05-09T03:12:05.837939+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837939+0000', 'last_undegraded': '2024-05-09T03:12:05.837939+0000', 'last_fullsized': '2024-05-09T03:12:05.837939+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:PG 9.36 is not active+clean 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833750+0000', 'last_change': '2024-05-09T03:12:05.833750+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833750+0000', 'last_undegraded': '2024-05-09T03:12:05.833750+0000', 'last_fullsized': '2024-05-09T03:12:05.833750+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:PG 9.35 is not active+clean 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833154+0000', 'last_change': '2024-05-09T03:12:05.833154+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833154+0000', 'last_undegraded': '2024-05-09T03:12:05.833154+0000', 'last_fullsized': '2024-05-09T03:12:05.833154+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:PG 9.34 is not active+clean 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.842732+0000', 'last_change': '2024-05-09T03:12:05.842732+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.842732+0000', 'last_undegraded': '2024-05-09T03:12:05.842732+0000', 'last_fullsized': '2024-05-09T03:12:05.842732+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:PG 9.33 is not active+clean 2024-05-09T03:12:11.711 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833533+0000', 'last_change': '2024-05-09T03:12:05.833533+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833533+0000', 'last_undegraded': '2024-05-09T03:12:05.833533+0000', 'last_fullsized': '2024-05-09T03:12:05.833533+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:PG 9.32 is not active+clean 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836888+0000', 'last_change': '2024-05-09T03:12:05.836888+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836888+0000', 'last_undegraded': '2024-05-09T03:12:05.836888+0000', 'last_fullsized': '2024-05-09T03:12:05.836888+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:PG 9.31 is not active+clean 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835706+0000', 'last_change': '2024-05-09T03:12:05.835706+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835706+0000', 'last_undegraded': '2024-05-09T03:12:05.835706+0000', 'last_fullsized': '2024-05-09T03:12:05.835706+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:PG 9.30 is not active+clean 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833981+0000', 'last_change': '2024-05-09T03:12:05.833981+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833981+0000', 'last_undegraded': '2024-05-09T03:12:05.833981+0000', 'last_fullsized': '2024-05-09T03:12:05.833981+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2f is not active+clean 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.840026+0000', 'last_change': '2024-05-09T03:12:05.840026+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.840026+0000', 'last_undegraded': '2024-05-09T03:12:05.840026+0000', 'last_fullsized': '2024-05-09T03:12:05.840026+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2e is not active+clean 2024-05-09T03:12:11.712 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.838628+0000', 'last_change': '2024-05-09T03:12:05.838628+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.838628+0000', 'last_undegraded': '2024-05-09T03:12:05.838628+0000', 'last_fullsized': '2024-05-09T03:12:05.838628+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2d is not active+clean 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.839714+0000', 'last_change': '2024-05-09T03:12:05.839714+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.839714+0000', 'last_undegraded': '2024-05-09T03:12:05.839714+0000', 'last_fullsized': '2024-05-09T03:12:05.839714+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2c is not active+clean 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837245+0000', 'last_change': '2024-05-09T03:12:05.837245+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837245+0000', 'last_undegraded': '2024-05-09T03:12:05.837245+0000', 'last_fullsized': '2024-05-09T03:12:05.837245+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2b is not active+clean 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.838405+0000', 'last_change': '2024-05-09T03:12:05.838405+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.838405+0000', 'last_undegraded': '2024-05-09T03:12:05.838405+0000', 'last_fullsized': '2024-05-09T03:12:05.838405+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2a is not active+clean 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837199+0000', 'last_change': '2024-05-09T03:12:05.837199+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837199+0000', 'last_undegraded': '2024-05-09T03:12:05.837199+0000', 'last_fullsized': '2024-05-09T03:12:05.837199+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:PG 9.29 is not active+clean 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836251+0000', 'last_change': '2024-05-09T03:12:05.836251+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836251+0000', 'last_undegraded': '2024-05-09T03:12:05.836251+0000', 'last_fullsized': '2024-05-09T03:12:05.836251+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:PG 9.28 is not active+clean 2024-05-09T03:12:11.713 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.834399+0000', 'last_change': '2024-05-09T03:12:05.834399+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.834399+0000', 'last_undegraded': '2024-05-09T03:12:05.834399+0000', 'last_fullsized': '2024-05-09T03:12:05.834399+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:PG 9.27 is not active+clean 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.840127+0000', 'last_change': '2024-05-09T03:12:05.840127+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.840127+0000', 'last_undegraded': '2024-05-09T03:12:05.840127+0000', 'last_fullsized': '2024-05-09T03:12:05.840127+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:PG 9.26 is not active+clean 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.844753+0000', 'last_change': '2024-05-09T03:12:05.844753+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.844753+0000', 'last_undegraded': '2024-05-09T03:12:05.844753+0000', 'last_fullsized': '2024-05-09T03:12:05.844753+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:PG 9.25 is not active+clean 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836714+0000', 'last_change': '2024-05-09T03:12:05.836714+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836714+0000', 'last_undegraded': '2024-05-09T03:12:05.836714+0000', 'last_fullsized': '2024-05-09T03:12:05.836714+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:PG 9.24 is not active+clean 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.842327+0000', 'last_change': '2024-05-09T03:12:05.842327+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.842327+0000', 'last_undegraded': '2024-05-09T03:12:05.842327+0000', 'last_fullsized': '2024-05-09T03:12:05.842327+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:PG 9.23 is not active+clean 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837936+0000', 'last_change': '2024-05-09T03:12:05.837936+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837936+0000', 'last_undegraded': '2024-05-09T03:12:05.837936+0000', 'last_fullsized': '2024-05-09T03:12:05.837936+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.714 INFO:tasks.ceph.ceph_manager.ceph:PG 9.22 is not active+clean 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.838187+0000', 'last_change': '2024-05-09T03:12:05.838187+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.838187+0000', 'last_undegraded': '2024-05-09T03:12:05.838187+0000', 'last_fullsized': '2024-05-09T03:12:05.838187+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:PG 9.21 is not active+clean 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.840098+0000', 'last_change': '2024-05-09T03:12:05.840098+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.840098+0000', 'last_undegraded': '2024-05-09T03:12:05.840098+0000', 'last_fullsized': '2024-05-09T03:12:05.840098+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:PG 9.20 is not active+clean 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.832938+0000', 'last_change': '2024-05-09T03:12:05.832938+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.832938+0000', 'last_undegraded': '2024-05-09T03:12:05.832938+0000', 'last_fullsized': '2024-05-09T03:12:05.832938+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1e is not active+clean 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837077+0000', 'last_change': '2024-05-09T03:12:05.837077+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837077+0000', 'last_undegraded': '2024-05-09T03:12:05.837077+0000', 'last_fullsized': '2024-05-09T03:12:05.837077+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1f is not active+clean 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.832809+0000', 'last_change': '2024-05-09T03:12:05.832809+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.832809+0000', 'last_undegraded': '2024-05-09T03:12:05.832809+0000', 'last_fullsized': '2024-05-09T03:12:05.832809+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:12:11.715 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1c is not active+clean 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833746+0000', 'last_change': '2024-05-09T03:12:05.833746+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833746+0000', 'last_undegraded': '2024-05-09T03:12:05.833746+0000', 'last_fullsized': '2024-05-09T03:12:05.833746+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1d is not active+clean 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836173+0000', 'last_change': '2024-05-09T03:12:05.836173+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836173+0000', 'last_undegraded': '2024-05-09T03:12:05.836173+0000', 'last_fullsized': '2024-05-09T03:12:05.836173+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1a is not active+clean 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835661+0000', 'last_change': '2024-05-09T03:12:05.835661+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835661+0000', 'last_undegraded': '2024-05-09T03:12:05.835661+0000', 'last_fullsized': '2024-05-09T03:12:05.835661+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1b is not active+clean 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.838497+0000', 'last_change': '2024-05-09T03:12:05.838497+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.838497+0000', 'last_undegraded': '2024-05-09T03:12:05.838497+0000', 'last_fullsized': '2024-05-09T03:12:05.838497+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:PG 9.18 is not active+clean 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833532+0000', 'last_change': '2024-05-09T03:12:05.833532+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833532+0000', 'last_undegraded': '2024-05-09T03:12:05.833532+0000', 'last_fullsized': '2024-05-09T03:12:05.833532+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:PG 9.19 is not active+clean 2024-05-09T03:12:11.716 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835683+0000', 'last_change': '2024-05-09T03:12:05.835683+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835683+0000', 'last_undegraded': '2024-05-09T03:12:05.835683+0000', 'last_fullsized': '2024-05-09T03:12:05.835683+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:PG 9.16 is not active+clean 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.838513+0000', 'last_change': '2024-05-09T03:12:05.838513+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.838513+0000', 'last_undegraded': '2024-05-09T03:12:05.838513+0000', 'last_fullsized': '2024-05-09T03:12:05.838513+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:PG 9.17 is not active+clean 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835774+0000', 'last_change': '2024-05-09T03:12:05.835774+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835774+0000', 'last_undegraded': '2024-05-09T03:12:05.835774+0000', 'last_fullsized': '2024-05-09T03:12:05.835774+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:PG 9.14 is not active+clean 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833184+0000', 'last_change': '2024-05-09T03:12:05.833184+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833184+0000', 'last_undegraded': '2024-05-09T03:12:05.833184+0000', 'last_fullsized': '2024-05-09T03:12:05.833184+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:PG 9.15 is not active+clean 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837431+0000', 'last_change': '2024-05-09T03:12:05.837431+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837431+0000', 'last_undegraded': '2024-05-09T03:12:05.837431+0000', 'last_fullsized': '2024-05-09T03:12:05.837431+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:PG 9.12 is not active+clean 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.839951+0000', 'last_change': '2024-05-09T03:12:05.839951+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.839951+0000', 'last_undegraded': '2024-05-09T03:12:05.839951+0000', 'last_fullsized': '2024-05-09T03:12:05.839951+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:PG 9.13 is not active+clean 2024-05-09T03:12:11.717 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837833+0000', 'last_change': '2024-05-09T03:12:05.837833+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837833+0000', 'last_undegraded': '2024-05-09T03:12:05.837833+0000', 'last_fullsized': '2024-05-09T03:12:05.837833+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:PG 9.10 is not active+clean 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.837016+0000', 'last_change': '2024-05-09T03:12:05.837016+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.837016+0000', 'last_undegraded': '2024-05-09T03:12:05.837016+0000', 'last_fullsized': '2024-05-09T03:12:05.837016+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:PG 9.11 is not active+clean 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835668+0000', 'last_change': '2024-05-09T03:12:05.835668+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835668+0000', 'last_undegraded': '2024-05-09T03:12:05.835668+0000', 'last_fullsized': '2024-05-09T03:12:05.835668+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:PG 9.e is not active+clean 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836209+0000', 'last_change': '2024-05-09T03:12:05.836209+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836209+0000', 'last_undegraded': '2024-05-09T03:12:05.836209+0000', 'last_fullsized': '2024-05-09T03:12:05.836209+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:PG 9.f is not active+clean 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.841432+0000', 'last_change': '2024-05-09T03:12:05.841432+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.841432+0000', 'last_undegraded': '2024-05-09T03:12:05.841432+0000', 'last_fullsized': '2024-05-09T03:12:05.841432+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.718 INFO:tasks.ceph.ceph_manager.ceph:PG 9.c is not active+clean 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835494+0000', 'last_change': '2024-05-09T03:12:05.835494+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835494+0000', 'last_undegraded': '2024-05-09T03:12:05.835494+0000', 'last_fullsized': '2024-05-09T03:12:05.835494+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:PG 9.d is not active+clean 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.836813+0000', 'last_change': '2024-05-09T03:12:05.836813+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.836813+0000', 'last_undegraded': '2024-05-09T03:12:05.836813+0000', 'last_fullsized': '2024-05-09T03:12:05.836813+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:PG 9.a is not active+clean 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.839358+0000', 'last_change': '2024-05-09T03:12:05.839358+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.839358+0000', 'last_undegraded': '2024-05-09T03:12:05.839358+0000', 'last_fullsized': '2024-05-09T03:12:05.839358+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:PG 9.b is not active+clean 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833512+0000', 'last_change': '2024-05-09T03:12:05.833512+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833512+0000', 'last_undegraded': '2024-05-09T03:12:05.833512+0000', 'last_fullsized': '2024-05-09T03:12:05.833512+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:PG 9.9 is not active+clean 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.833538+0000', 'last_change': '2024-05-09T03:12:05.833538+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.833538+0000', 'last_undegraded': '2024-05-09T03:12:05.833538+0000', 'last_fullsized': '2024-05-09T03:12:05.833538+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:PG 9.6 is not active+clean 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.841979+0000', 'last_change': '2024-05-09T03:12:05.841979+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.841979+0000', 'last_undegraded': '2024-05-09T03:12:05.841979+0000', 'last_fullsized': '2024-05-09T03:12:05.841979+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.719 INFO:tasks.ceph.ceph_manager.ceph:PG 9.7 is not active+clean 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.840940+0000', 'last_change': '2024-05-09T03:12:05.840940+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.840940+0000', 'last_undegraded': '2024-05-09T03:12:05.840940+0000', 'last_fullsized': '2024-05-09T03:12:05.840940+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:PG 9.4 is not active+clean 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.832473+0000', 'last_change': '2024-05-09T03:12:05.832473+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.832473+0000', 'last_undegraded': '2024-05-09T03:12:05.832473+0000', 'last_fullsized': '2024-05-09T03:12:05.832473+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:PG 9.5 is not active+clean 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835634+0000', 'last_change': '2024-05-09T03:12:05.835634+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835634+0000', 'last_undegraded': '2024-05-09T03:12:05.835634+0000', 'last_fullsized': '2024-05-09T03:12:05.835634+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:PG 9.2 is not active+clean 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.834748+0000', 'last_change': '2024-05-09T03:12:05.834748+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.834748+0000', 'last_undegraded': '2024-05-09T03:12:05.834748+0000', 'last_fullsized': '2024-05-09T03:12:05.834748+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:PG 9.3 is not active+clean 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835236+0000', 'last_change': '2024-05-09T03:12:05.835236+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835236+0000', 'last_undegraded': '2024-05-09T03:12:05.835236+0000', 'last_fullsized': '2024-05-09T03:12:05.835236+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:PG 9.0 is not active+clean 2024-05-09T03:12:11.720 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.832475+0000', 'last_change': '2024-05-09T03:12:05.832475+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.832475+0000', 'last_undegraded': '2024-05-09T03:12:05.832475+0000', 'last_fullsized': '2024-05-09T03:12:05.832475+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-05-09T03:12:11.721 INFO:tasks.ceph.ceph_manager.ceph:PG 9.1 is not active+clean 2024-05-09T03:12:11.721 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.834642+0000', 'last_change': '2024-05-09T03:12:05.834642+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.834642+0000', 'last_undegraded': '2024-05-09T03:12:05.834642+0000', 'last_fullsized': '2024-05-09T03:12:05.834642+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.721 INFO:tasks.ceph.ceph_manager.ceph:PG 9.8 is not active+clean 2024-05-09T03:12:11.721 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 53, 'state': 'creating+peering', 'last_fresh': '2024-05-09T03:12:05.835177+0000', 'last_change': '2024-05-09T03:12:05.835177+0000', 'last_active': '2024-05-09T03:12:05.823852+0000', 'last_peered': '2024-05-09T03:12:05.823852+0000', 'last_clean': '2024-05-09T03:12:05.823852+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-05-09T03:12:05.835177+0000', 'last_undegraded': '2024-05-09T03:12:05.835177+0000', 'last_fullsized': '2024-05-09T03:12:05.835177+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 53, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'last_clean_scrub_stamp': '2024-05-09T03:12:05.823852+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-05-09T03:12:11.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-05-09T03:12:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.857+0000 7fef96c9a640 1 Processor -- start 2024-05-09T03:12:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.858+0000 7fef96c9a640 1 -- start start 2024-05-09T03:12:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef96c9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef900b1210 0x7fef900b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef96c9a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef90070d80 0x7fef900aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef96c9a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9015d980 con 0x7fef900aeb80 2024-05-09T03:12:11.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef96c9a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fef9015dc60 con 0x7fef90070d80 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef96c9a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fef9015df40 con 0x7fef900b1210 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef95237640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef90070d80 0x7fef900aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef95237640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef90070d80 0x7fef900aef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53280/0 (socket says 172.21.15.46:53280) 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef95237640 1 -- 172.21.15.46:0/3127047292 learned_addr learned my addr 172.21.15.46:0/3127047292 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.859+0000 7fef94a36640 1 --2- 172.21.15.46:0/3127047292 >> v2:172.21.15.88:3300/0 conn(0x7fef900b1210 0x7fef900b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef95237640 1 -- 172.21.15.46:0/3127047292 >> v2:172.21.15.88:3300/0 conn(0x7fef900b1210 msgr2=0x7fef900b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef95237640 1 --2- 172.21.15.46:0/3127047292 >> v2:172.21.15.88:3300/0 conn(0x7fef900b1210 0x7fef900b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef95237640 1 -- 172.21.15.46:0/3127047292 >> v1:172.21.15.88:6789/0 conn(0x7fef900aeb80 legacy=0x7fef90070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef95237640 1 -- 172.21.15.46:0/3127047292 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9015e220 con 0x7fef90070d80 2024-05-09T03:12:11.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef95237640 1 --2- 172.21.15.46:0/3127047292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef90070d80 0x7fef900aef90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fef900713a0 tx=0x7fef8802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e6987316ab9157 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef8f7fe640 1 -- 172.21.15.46:0/3127047292 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88036070 con 0x7fef90070d80 2024-05-09T03:12:11.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.860+0000 7fef8f7fe640 1 -- 172.21.15.46:0/3127047292 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef88031600 con 0x7fef90070d80 2024-05-09T03:12:11.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef8f7fe640 1 -- 172.21.15.46:0/3127047292 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef880319d0 con 0x7fef90070d80 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 -- 172.21.15.46:0/3127047292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef90070d80 msgr2=0x7fef900aef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 --2- 172.21.15.46:0/3127047292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef90070d80 0x7fef900aef90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fef900713a0 tx=0x7fef8802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 -- 172.21.15.46:0/3127047292 shutdown_connections 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 --2- 172.21.15.46:0/3127047292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef90070d80 0x7fef900aef90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 --2- 172.21.15.46:0/3127047292 >> v2:172.21.15.88:3300/0 conn(0x7fef900b1210 0x7fef900b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 -- 172.21.15.46:0/3127047292 >> 172.21.15.46:0/3127047292 conn(0x7fef900b3910 msgr2=0x7fef900b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 -- 172.21.15.46:0/3127047292 shutdown_connections 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.861+0000 7fef96c9a640 1 -- 172.21.15.46:0/3127047292 wait complete. 2024-05-09T03:12:11.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.863+0000 7fef96c9a640 1 Processor -- start 2024-05-09T03:12:11.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.863+0000 7fef96c9a640 1 -- start start 2024-05-09T03:12:11.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.864+0000 7fef96c9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 0x7fef9013d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.864+0000 7fef96c9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef900b1210 0x7fef90157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.864+0000 7fef96c9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900b2900 0x7fef900b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef96c9a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fef900739f0 con 0x7fef900aeb80 2024-05-09T03:12:11.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef96c9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fef90073870 con 0x7fef900b2900 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef96c9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fef90073b70 con 0x7fef900b1210 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 0x7fef9013d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 0x7fef9013d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53288/0 (socket says 172.21.15.46:53288) 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 -- 172.21.15.46:0/695677720 learned_addr learned my addr 172.21.15.46:0/695677720 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef95237640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900b2900 0x7fef900b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 -- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef900b1210 msgr2=0x7fef90157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef900b1210 0x7fef90157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 -- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900b2900 msgr2=0x7fef900b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900b2900 0x7fef900b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 -- 172.21.15.46:0/695677720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef900b3250 con 0x7fef900aeb80 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef95237640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900b2900 0x7fef900b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef94a36640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 0x7fef9013d540 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fef8400e840 tx=0x7fef8400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84017070 con 0x7fef900aeb80 2024-05-09T03:12:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef8400c940 con 0x7fef900aeb80 2024-05-09T03:12:11.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef90141120 con 0x7fef900aeb80 2024-05-09T03:12:11.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8400cd10 con 0x7fef900aeb80 2024-05-09T03:12:11.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.865+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9015b130 con 0x7fef900aeb80 2024-05-09T03:12:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.866+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9006a270 con 0x7fef900aeb80 2024-05-09T03:12:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.867+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fef8401a070 con 0x7fef900aeb80 2024-05-09T03:12:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.868+0000 7fef8d7fa640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef74076300 0x7fef74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.868+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fef840974f0 con 0x7fef900aeb80 2024-05-09T03:12:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.868+0000 7fef8ffff640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef74076300 0x7fef74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.868+0000 7fef8ffff640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef74076300 0x7fef74078810 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fef7c000f60 tx=0x7fef7c0079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:11.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:11.872+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fef900b3250 con 0x7fef900aeb80 2024-05-09T03:12:12.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.138+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fef900d1760 con 0x7fef900aeb80 2024-05-09T03:12:12.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.139+0000 7fef8d7fa640 1 -- 172.21.15.46:0/695677720 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1453 (secure 0 0 0) 0x7fef84060db0 con 0x7fef900aeb80 2024-05-09T03:12:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:12.141 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":182,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1715224161,"num_in_osds":8,"osd_in_since":1715224152,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":244641792,"bytes_avail":772849471488,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":46,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"a","status":"up:active","gid":5874}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-05-09T03:11:54.527069+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef74076300 msgr2=0x7fef74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef74076300 0x7fef74078810 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fef7c000f60 tx=0x7fef7c0079b0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 msgr2=0x7fef9013d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 0x7fef9013d540 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fef8400e840 tx=0x7fef8400ed00 comp rx=0 tx=0).stop 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 shutdown_connections 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900b2900 0x7fef900b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef900b1210 0x7fef90157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef74076300 0x7fef74078810 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 --2- 172.21.15.46:0/695677720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900aeb80 0x7fef9013d540 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 >> 172.21.15.46:0/695677720 conn(0x7fef900b3910 msgr2=0x7fef900b5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:12.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 shutdown_connections 2024-05-09T03:12:12.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.141+0000 7fef96c9a640 1 -- 172.21.15.46:0/695677720 wait complete. 2024-05-09T03:12:12.154 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-05-09T03:12:12.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:12:12.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.267+0000 7fbbbd766640 1 Processor -- start 2024-05-09T03:12:12.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.267+0000 7fbbbd766640 1 -- start start 2024-05-09T03:12:12.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.268+0000 7fbbbd766640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:12.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbbd766640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbb80b7d20 0x7fbbb80b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbbd766640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb80b8130 con 0x7fbbb80d57a0 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbbd766640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb80d5bb0 con 0x7fbbb80b7d20 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbbd766640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb80d49e0 con 0x7fbbb8160610 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbb80b7d20 0x7fbbb80b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbb80b7d20 0x7fbbb80b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53302/0 (socket says 172.21.15.46:53302) 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 -- 172.21.15.46:0/1567603278 learned_addr learned my addr 172.21.15.46:0/1567603278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb67fc640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 -- 172.21.15.46:0/1567603278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 msgr2=0x7fbbb80b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 -- 172.21.15.46:0/1567603278 >> v1:172.21.15.88:6789/0 conn(0x7fbbb80d57a0 legacy=0x7fbbb80b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb77fe640 1 -- 172.21.15.46:0/1567603278 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80d4cc0 con 0x7fbbb80b7d20 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.269+0000 7fbbb67fc640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.270+0000 7fbbb77fe640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80b9450 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009870 tx=0x7fbbb002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05cab65e6817f94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:12.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.270+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/1567603278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0037070 con 0x7fbbb80b7d20 2024-05-09T03:12:12.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.270+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/1567603278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb0032440 con 0x7fbbb80b7d20 2024-05-09T03:12:12.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.270+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/1567603278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0032810 con 0x7fbbb80b7d20 2024-05-09T03:12:12.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 -- 172.21.15.46:0/1567603278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80b7d20 msgr2=0x7fbbb80b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80b9450 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009870 tx=0x7fbbb002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 -- 172.21.15.46:0/1567603278 shutdown_connections 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80b9450 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 --2- 172.21.15.46:0/1567603278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 -- 172.21.15.46:0/1567603278 >> 172.21.15.46:0/1567603278 conn(0x7fbbb80743e0 msgr2=0x7fbbb80d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 -- 172.21.15.46:0/1567603278 shutdown_connections 2024-05-09T03:12:12.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.271+0000 7fbbbd766640 1 -- 172.21.15.46:0/1567603278 wait complete. 2024-05-09T03:12:12.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.273+0000 7fbbbd766640 1 Processor -- start 2024-05-09T03:12:12.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.273+0000 7fbbbd766640 1 -- start start 2024-05-09T03:12:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbbd766640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbbd766640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 0x7fbbb80cff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbbd766640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbbd766640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb8160a20 con 0x7fbbb80d57a0 2024-05-09T03:12:12.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbbd766640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80d5280 con 0x7fbbb80b7d20 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbbd766640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb80d15b0 con 0x7fbbb8160610 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 0x7fbbb80cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 0x7fbbb80cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53314/0 (socket says 172.21.15.46:53314) 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbb67fc640 1 -- 172.21.15.46:0/386587849 learned_addr learned my addr 172.21.15.46:0/386587849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbb6ffd640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.274+0000 7fbbb77fe640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb67fc640 1 -- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 msgr2=0x7fbbb80cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb67fc640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb67fc640 1 -- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb80b7d20 msgr2=0x7fbbb80cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb67fc640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb67fc640 1 -- 172.21.15.46:0/386587849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80ce9b0 con 0x7fbbb80d57a0 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb6ffd640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:12.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb77fe640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbb67fc640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 0x7fbbb80cff50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00c820 tx=0x7fbbac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac015070 con 0x7fbbb80d57a0 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac007a30 con 0x7fbbb80d57a0 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac007de0 con 0x7fbbb80d57a0 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb80cec50 con 0x7fbbb80d57a0 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.275+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb80cf210 con 0x7fbbb80d57a0 2024-05-09T03:12:12.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.276+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb806a270 con 0x7fbbb80d57a0 2024-05-09T03:12:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.277+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbbac018070 con 0x7fbbb80d57a0 2024-05-09T03:12:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.278+0000 7fbb9f7fe640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.278+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fbbac097990 con 0x7fbbb80d57a0 2024-05-09T03:12:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.278+0000 7fbbb6ffd640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:12.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.279+0000 7fbbb6ffd640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbbb806d3b0 tx=0x7fbba8009550 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:12.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.282+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbbac061240 con 0x7fbbb80d57a0 2024-05-09T03:12:12.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.468+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbb80cd680 con 0x7fbba4076300 2024-05-09T03:12:12.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.474+0000 7fbb9f7fe640 1 -- 172.21.15.46:0/386587849 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246625 (secure 0 0 0) 0x7fbbb80cd680 con 0x7fbba4076300 2024-05-09T03:12:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:12.480 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:12:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 msgr2=0x7fbba4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbbb806d3b0 tx=0x7fbba8009550 comp rx=0 tx=0).stop 2024-05-09T03:12:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 msgr2=0x7fbbb80cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 0x7fbbb80cff50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00c820 tx=0x7fbbac00cce0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 shutdown_connections 2024-05-09T03:12:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb8160610 0x7fbbb80cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d57a0 0x7fbbb80cff50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 --2- 172.21.15.46:0/386587849 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb80b7d20 0x7fbbb80cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 >> 172.21.15.46:0/386587849 conn(0x7fbbb80743e0 msgr2=0x7fbbb80b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 shutdown_connections 2024-05-09T03:12:12.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:12.477+0000 7fbbbd766640 1 -- 172.21.15.46:0/386587849 wait complete. 2024-05-09T03:12:12.490 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-05-09T03:12:10.531012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238908,"kb_used_data":6048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735812,"statfs":{"total":773094113280,"available":772849471488,"internally_reserved":0,"allocated":6193152,"data_stored":3226534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.706220"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841057+0000","last_change":"2024-05-09T03:12:05.841057+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841057+0000","last_undegraded":"2024-05-09T03:12:05.841057+0000","last_fullsized":"2024-05-09T03:12:05.841057+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836609+0000","last_change":"2024-05-09T03:12:05.836609+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836609+0000","last_undegraded":"2024-05-09T03:12:05.836609+0000","last_fullsized":"2024-05-09T03:12:05.836609+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837976+0000","last_change":"2024-05-09T03:12:05.837976+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837976+0000","last_undegraded":"2024-05-09T03:12:05.837976+0000","last_fullsized":"2024-05-09T03:12:05.837976+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.843259+0000","last_change":"2024-05-09T03:12:05.843259+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.843259+0000","last_undegraded":"2024-05-09T03:12:05.843259+0000","last_fullsized":"2024-05-09T03:12:05.843259+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839531+0000","last_change":"2024-05-09T03:12:05.839531+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839531+0000","last_undegraded":"2024-05-09T03:12:05.839531+0000","last_fullsized":"2024-05-09T03:12:05.839531+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834452+0000","last_change":"2024-05-09T03:12:05.834452+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834452+0000","last_undegraded":"2024-05-09T03:12:05.834452+0000","last_fullsized":"2024-05-09T03:12:05.834452+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836296+0000","last_change":"2024-05-09T03:12:05.836296+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836296+0000","last_undegraded":"2024-05-09T03:12:05.836296+0000","last_fullsized":"2024-05-09T03:12:05.836296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835798+0000","last_change":"2024-05-09T03:12:05.835798+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835798+0000","last_undegraded":"2024-05-09T03:12:05.835798+0000","last_fullsized":"2024-05-09T03:12:05.835798+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837939+0000","last_change":"2024-05-09T03:12:05.837939+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837939+0000","last_undegraded":"2024-05-09T03:12:05.837939+0000","last_fullsized":"2024-05-09T03:12:05.837939+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833750+0000","last_change":"2024-05-09T03:12:05.833750+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833750+0000","last_undegraded":"2024-05-09T03:12:05.833750+0000","last_fullsized":"2024-05-09T03:12:05.833750+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833154+0000","last_change":"2024-05-09T03:12:05.833154+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833154+0000","last_undegraded":"2024-05-09T03:12:05.833154+0000","last_fullsized":"2024-05-09T03:12:05.833154+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842732+0000","last_change":"2024-05-09T03:12:05.842732+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842732+0000","last_undegraded":"2024-05-09T03:12:05.842732+0000","last_fullsized":"2024-05-09T03:12:05.842732+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833533+0000","last_change":"2024-05-09T03:12:05.833533+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833533+0000","last_undegraded":"2024-05-09T03:12:05.833533+0000","last_fullsized":"2024-05-09T03:12:05.833533+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836888+0000","last_change":"2024-05-09T03:12:05.836888+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836888+0000","last_undegraded":"2024-05-09T03:12:05.836888+0000","last_fullsized":"2024-05-09T03:12:05.836888+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835706+0000","last_change":"2024-05-09T03:12:05.835706+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835706+0000","last_undegraded":"2024-05-09T03:12:05.835706+0000","last_fullsized":"2024-05-09T03:12:05.835706+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833981+0000","last_change":"2024-05-09T03:12:05.833981+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833981+0000","last_undegraded":"2024-05-09T03:12:05.833981+0000","last_fullsized":"2024-05-09T03:12:05.833981+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840026+0000","last_change":"2024-05-09T03:12:05.840026+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840026+0000","last_undegraded":"2024-05-09T03:12:05.840026+0000","last_fullsized":"2024-05-09T03:12:05.840026+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838628+0000","last_change":"2024-05-09T03:12:05.838628+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838628+0000","last_undegraded":"2024-05-09T03:12:05.838628+0000","last_fullsized":"2024-05-09T03:12:05.838628+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839714+0000","last_change":"2024-05-09T03:12:05.839714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839714+0000","last_undegraded":"2024-05-09T03:12:05.839714+0000","last_fullsized":"2024-05-09T03:12:05.839714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837245+0000","last_change":"2024-05-09T03:12:05.837245+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837245+0000","last_undegraded":"2024-05-09T03:12:05.837245+0000","last_fullsized":"2024-05-09T03:12:05.837245+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838405+0000","last_change":"2024-05-09T03:12:05.838405+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838405+0000","last_undegraded":"2024-05-09T03:12:05.838405+0000","last_fullsized":"2024-05-09T03:12:05.838405+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837199+0000","last_change":"2024-05-09T03:12:05.837199+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837199+0000","last_undegraded":"2024-05-09T03:12:05.837199+0000","last_fullsized":"2024-05-09T03:12:05.837199+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836251+0000","last_change":"2024-05-09T03:12:05.836251+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836251+0000","last_undegraded":"2024-05-09T03:12:05.836251+0000","last_fullsized":"2024-05-09T03:12:05.836251+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834399+0000","last_change":"2024-05-09T03:12:05.834399+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834399+0000","last_undegraded":"2024-05-09T03:12:05.834399+0000","last_fullsized":"2024-05-09T03:12:05.834399+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840127+0000","last_change":"2024-05-09T03:12:05.840127+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840127+0000","last_undegraded":"2024-05-09T03:12:05.840127+0000","last_fullsized":"2024-05-09T03:12:05.840127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.844753+0000","last_change":"2024-05-09T03:12:05.844753+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.844753+0000","last_undegraded":"2024-05-09T03:12:05.844753+0000","last_fullsized":"2024-05-09T03:12:05.844753+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836714+0000","last_change":"2024-05-09T03:12:05.836714+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836714+0000","last_undegraded":"2024-05-09T03:12:05.836714+0000","last_fullsized":"2024-05-09T03:12:05.836714+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.842327+0000","last_change":"2024-05-09T03:12:05.842327+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.842327+0000","last_undegraded":"2024-05-09T03:12:05.842327+0000","last_fullsized":"2024-05-09T03:12:05.842327+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837936+0000","last_change":"2024-05-09T03:12:05.837936+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837936+0000","last_undegraded":"2024-05-09T03:12:05.837936+0000","last_fullsized":"2024-05-09T03:12:05.837936+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838187+0000","last_change":"2024-05-09T03:12:05.838187+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838187+0000","last_undegraded":"2024-05-09T03:12:05.838187+0000","last_fullsized":"2024-05-09T03:12:05.838187+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840098+0000","last_change":"2024-05-09T03:12:05.840098+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840098+0000","last_undegraded":"2024-05-09T03:12:05.840098+0000","last_fullsized":"2024-05-09T03:12:05.840098+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832938+0000","last_change":"2024-05-09T03:12:05.832938+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832938+0000","last_undegraded":"2024-05-09T03:12:05.832938+0000","last_fullsized":"2024-05-09T03:12:05.832938+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837077+0000","last_change":"2024-05-09T03:12:05.837077+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837077+0000","last_undegraded":"2024-05-09T03:12:05.837077+0000","last_fullsized":"2024-05-09T03:12:05.837077+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852354+0000","last_change":"2024-05-09T03:12:05.849076+0000","last_active":"2024-05-09T03:12:05.852354+0000","last_peered":"2024-05-09T03:12:05.852354+0000","last_clean":"2024-05-09T03:12:05.852354+0000","last_became_active":"2024-05-09T03:12:05.848768+0000","last_became_peered":"2024-05-09T03:12:05.848768+0000","last_unstale":"2024-05-09T03:12:05.852354+0000","last_undegraded":"2024-05-09T03:12:05.852354+0000","last_fullsized":"2024-05-09T03:12:05.852354+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:05.791451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832809+0000","last_change":"2024-05-09T03:12:05.832809+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832809+0000","last_undegraded":"2024-05-09T03:12:05.832809+0000","last_fullsized":"2024-05-09T03:12:05.832809+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851128+0000","last_change":"2024-05-09T03:12:05.846715+0000","last_active":"2024-05-09T03:12:05.851128+0000","last_peered":"2024-05-09T03:12:05.851128+0000","last_clean":"2024-05-09T03:12:05.851128+0000","last_became_active":"2024-05-09T03:12:05.846448+0000","last_became_peered":"2024-05-09T03:12:05.846448+0000","last_unstale":"2024-05-09T03:12:05.851128+0000","last_undegraded":"2024-05-09T03:12:05.851128+0000","last_fullsized":"2024-05-09T03:12:05.851128+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:53:45.376505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833746+0000","last_change":"2024-05-09T03:12:05.833746+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833746+0000","last_undegraded":"2024-05-09T03:12:05.833746+0000","last_fullsized":"2024-05-09T03:12:05.833746+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849635+0000","last_change":"2024-05-09T03:12:05.845815+0000","last_active":"2024-05-09T03:12:05.849635+0000","last_peered":"2024-05-09T03:12:05.849635+0000","last_clean":"2024-05-09T03:12:05.849635+0000","last_became_active":"2024-05-09T03:12:05.845509+0000","last_became_peered":"2024-05-09T03:12:05.845509+0000","last_unstale":"2024-05-09T03:12:05.849635+0000","last_undegraded":"2024-05-09T03:12:05.849635+0000","last_fullsized":"2024-05-09T03:12:05.849635+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:07.083261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836173+0000","last_change":"2024-05-09T03:12:05.836173+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836173+0000","last_undegraded":"2024-05-09T03:12:05.836173+0000","last_fullsized":"2024-05-09T03:12:05.836173+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853894+0000","last_change":"2024-05-09T03:12:05.849601+0000","last_active":"2024-05-09T03:12:05.853894+0000","last_peered":"2024-05-09T03:12:05.853894+0000","last_clean":"2024-05-09T03:12:05.853894+0000","last_became_active":"2024-05-09T03:12:05.849230+0000","last_became_peered":"2024-05-09T03:12:05.849230+0000","last_unstale":"2024-05-09T03:12:05.853894+0000","last_undegraded":"2024-05-09T03:12:05.853894+0000","last_fullsized":"2024-05-09T03:12:05.853894+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:43.363860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835661+0000","last_change":"2024-05-09T03:12:05.835661+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835661+0000","last_undegraded":"2024-05-09T03:12:05.835661+0000","last_fullsized":"2024-05-09T03:12:05.835661+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.856188+0000","last_change":"2024-05-09T03:12:05.853186+0000","last_active":"2024-05-09T03:12:05.856188+0000","last_peered":"2024-05-09T03:12:05.856188+0000","last_clean":"2024-05-09T03:12:05.856188+0000","last_became_active":"2024-05-09T03:12:05.852815+0000","last_became_peered":"2024-05-09T03:12:05.852815+0000","last_unstale":"2024-05-09T03:12:05.856188+0000","last_undegraded":"2024-05-09T03:12:05.856188+0000","last_fullsized":"2024-05-09T03:12:05.856188+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838497+0000","last_change":"2024-05-09T03:12:05.838497+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838497+0000","last_undegraded":"2024-05-09T03:12:05.838497+0000","last_fullsized":"2024-05-09T03:12:05.838497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849662+0000","last_change":"2024-05-09T03:12:05.844443+0000","last_active":"2024-05-09T03:12:05.849662+0000","last_peered":"2024-05-09T03:12:05.849662+0000","last_clean":"2024-05-09T03:12:05.849662+0000","last_became_active":"2024-05-09T03:12:05.844119+0000","last_became_peered":"2024-05-09T03:12:05.844119+0000","last_unstale":"2024-05-09T03:12:05.849662+0000","last_undegraded":"2024-05-09T03:12:05.849662+0000","last_fullsized":"2024-05-09T03:12:05.849662+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833532+0000","last_change":"2024-05-09T03:12:05.833532+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833532+0000","last_undegraded":"2024-05-09T03:12:05.833532+0000","last_fullsized":"2024-05-09T03:12:05.833532+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852899+0000","last_change":"2024-05-09T03:12:05.847498+0000","last_active":"2024-05-09T03:12:05.852899+0000","last_peered":"2024-05-09T03:12:05.852899+0000","last_clean":"2024-05-09T03:12:05.852899+0000","last_became_active":"2024-05-09T03:12:05.847170+0000","last_became_peered":"2024-05-09T03:12:05.847170+0000","last_unstale":"2024-05-09T03:12:05.852899+0000","last_undegraded":"2024-05-09T03:12:05.852899+0000","last_fullsized":"2024-05-09T03:12:05.852899+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835683+0000","last_change":"2024-05-09T03:12:05.835683+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835683+0000","last_undegraded":"2024-05-09T03:12:05.835683+0000","last_fullsized":"2024-05-09T03:12:05.835683+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853539+0000","last_change":"2024-05-09T03:12:05.847308+0000","last_active":"2024-05-09T03:12:05.853539+0000","last_peered":"2024-05-09T03:12:05.853539+0000","last_clean":"2024-05-09T03:12:05.853539+0000","last_became_active":"2024-05-09T03:12:05.846406+0000","last_became_peered":"2024-05-09T03:12:05.846406+0000","last_unstale":"2024-05-09T03:12:05.853539+0000","last_undegraded":"2024-05-09T03:12:05.853539+0000","last_fullsized":"2024-05-09T03:12:05.853539+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:40:28.024912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.838513+0000","last_change":"2024-05-09T03:12:05.838513+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.838513+0000","last_undegraded":"2024-05-09T03:12:05.838513+0000","last_fullsized":"2024-05-09T03:12:05.838513+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851570+0000","last_change":"2024-05-09T03:12:05.847459+0000","last_active":"2024-05-09T03:12:05.851570+0000","last_peered":"2024-05-09T03:12:05.851570+0000","last_clean":"2024-05-09T03:12:05.851570+0000","last_became_active":"2024-05-09T03:12:05.847106+0000","last_became_peered":"2024-05-09T03:12:05.847106+0000","last_unstale":"2024-05-09T03:12:05.851570+0000","last_undegraded":"2024-05-09T03:12:05.851570+0000","last_fullsized":"2024-05-09T03:12:05.851570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835774+0000","last_change":"2024-05-09T03:12:05.835774+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835774+0000","last_undegraded":"2024-05-09T03:12:05.835774+0000","last_fullsized":"2024-05-09T03:12:05.835774+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852846+0000","last_change":"2024-05-09T03:12:05.849037+0000","last_active":"2024-05-09T03:12:05.852846+0000","last_peered":"2024-05-09T03:12:05.852846+0000","last_clean":"2024-05-09T03:12:05.852846+0000","last_became_active":"2024-05-09T03:12:05.848283+0000","last_became_peered":"2024-05-09T03:12:05.848283+0000","last_unstale":"2024-05-09T03:12:05.852846+0000","last_undegraded":"2024-05-09T03:12:05.852846+0000","last_fullsized":"2024-05-09T03:12:05.852846+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:43:50.152051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833184+0000","last_change":"2024-05-09T03:12:05.833184+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833184+0000","last_undegraded":"2024-05-09T03:12:05.833184+0000","last_fullsized":"2024-05-09T03:12:05.833184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851612+0000","last_change":"2024-05-09T03:12:05.847630+0000","last_active":"2024-05-09T03:12:05.851612+0000","last_peered":"2024-05-09T03:12:05.851612+0000","last_clean":"2024-05-09T03:12:05.851612+0000","last_became_active":"2024-05-09T03:12:05.847302+0000","last_became_peered":"2024-05-09T03:12:05.847302+0000","last_unstale":"2024-05-09T03:12:05.851612+0000","last_undegraded":"2024-05-09T03:12:05.851612+0000","last_fullsized":"2024-05-09T03:12:05.851612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837431+0000","last_change":"2024-05-09T03:12:05.837431+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837431+0000","last_undegraded":"2024-05-09T03:12:05.837431+0000","last_fullsized":"2024-05-09T03:12:05.837431+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853524+0000","last_change":"2024-05-09T03:12:05.845847+0000","last_active":"2024-05-09T03:12:05.853524+0000","last_peered":"2024-05-09T03:12:05.853524+0000","last_clean":"2024-05-09T03:12:05.853524+0000","last_became_active":"2024-05-09T03:12:05.845474+0000","last_became_peered":"2024-05-09T03:12:05.845474+0000","last_unstale":"2024-05-09T03:12:05.853524+0000","last_undegraded":"2024-05-09T03:12:05.853524+0000","last_fullsized":"2024-05-09T03:12:05.853524+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:56.893788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839951+0000","last_change":"2024-05-09T03:12:05.839951+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839951+0000","last_undegraded":"2024-05-09T03:12:05.839951+0000","last_fullsized":"2024-05-09T03:12:05.839951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854903+0000","last_change":"2024-05-09T03:12:05.854030+0000","last_active":"2024-05-09T03:12:05.854903+0000","last_peered":"2024-05-09T03:12:05.854903+0000","last_clean":"2024-05-09T03:12:05.854903+0000","last_became_active":"2024-05-09T03:12:05.853677+0000","last_became_peered":"2024-05-09T03:12:05.853677+0000","last_unstale":"2024-05-09T03:12:05.854903+0000","last_undegraded":"2024-05-09T03:12:05.854903+0000","last_fullsized":"2024-05-09T03:12:05.854903+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:25.968267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837833+0000","last_change":"2024-05-09T03:12:05.837833+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837833+0000","last_undegraded":"2024-05-09T03:12:05.837833+0000","last_fullsized":"2024-05-09T03:12:05.837833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846449+0000","last_change":"2024-05-09T03:12:05.843325+0000","last_active":"2024-05-09T03:12:05.846449+0000","last_peered":"2024-05-09T03:12:05.846449+0000","last_clean":"2024-05-09T03:12:05.846449+0000","last_became_active":"2024-05-09T03:12:05.842923+0000","last_became_peered":"2024-05-09T03:12:05.842923+0000","last_unstale":"2024-05-09T03:12:05.846449+0000","last_undegraded":"2024-05-09T03:12:05.846449+0000","last_fullsized":"2024-05-09T03:12:05.846449+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:43:02.488792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.837016+0000","last_change":"2024-05-09T03:12:05.837016+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.837016+0000","last_undegraded":"2024-05-09T03:12:05.837016+0000","last_fullsized":"2024-05-09T03:12:05.837016+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853796+0000","last_change":"2024-05-09T03:12:05.848661+0000","last_active":"2024-05-09T03:12:05.853796+0000","last_peered":"2024-05-09T03:12:05.853796+0000","last_clean":"2024-05-09T03:12:05.853796+0000","last_became_active":"2024-05-09T03:12:05.848010+0000","last_became_peered":"2024-05-09T03:12:05.848010+0000","last_unstale":"2024-05-09T03:12:05.853796+0000","last_undegraded":"2024-05-09T03:12:05.853796+0000","last_fullsized":"2024-05-09T03:12:05.853796+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:20.873828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835668+0000","last_change":"2024-05-09T03:12:05.835668+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835668+0000","last_undegraded":"2024-05-09T03:12:05.835668+0000","last_fullsized":"2024-05-09T03:12:05.835668+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852646+0000","last_change":"2024-05-09T03:12:05.849792+0000","last_active":"2024-05-09T03:12:05.852646+0000","last_peered":"2024-05-09T03:12:05.852646+0000","last_clean":"2024-05-09T03:12:05.852646+0000","last_became_active":"2024-05-09T03:12:05.849439+0000","last_became_peered":"2024-05-09T03:12:05.849439+0000","last_unstale":"2024-05-09T03:12:05.852646+0000","last_undegraded":"2024-05-09T03:12:05.852646+0000","last_fullsized":"2024-05-09T03:12:05.852646+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:00.752576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836209+0000","last_change":"2024-05-09T03:12:05.836209+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836209+0000","last_undegraded":"2024-05-09T03:12:05.836209+0000","last_fullsized":"2024-05-09T03:12:05.836209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854627+0000","last_change":"2024-05-09T03:12:05.853093+0000","last_active":"2024-05-09T03:12:05.854627+0000","last_peered":"2024-05-09T03:12:05.854627+0000","last_clean":"2024-05-09T03:12:05.854627+0000","last_became_active":"2024-05-09T03:12:05.852692+0000","last_became_peered":"2024-05-09T03:12:05.852692+0000","last_unstale":"2024-05-09T03:12:05.854627+0000","last_undegraded":"2024-05-09T03:12:05.854627+0000","last_fullsized":"2024-05-09T03:12:05.854627+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:43.389393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841432+0000","last_change":"2024-05-09T03:12:05.841432+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841432+0000","last_undegraded":"2024-05-09T03:12:05.841432+0000","last_fullsized":"2024-05-09T03:12:05.841432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852440+0000","last_change":"2024-05-09T03:12:05.849439+0000","last_active":"2024-05-09T03:12:05.852440+0000","last_peered":"2024-05-09T03:12:05.852440+0000","last_clean":"2024-05-09T03:12:05.852440+0000","last_became_active":"2024-05-09T03:12:05.849134+0000","last_became_peered":"2024-05-09T03:12:05.849134+0000","last_unstale":"2024-05-09T03:12:05.852440+0000","last_undegraded":"2024-05-09T03:12:05.852440+0000","last_fullsized":"2024-05-09T03:12:05.852440+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:43:46.511932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835494+0000","last_change":"2024-05-09T03:12:05.835494+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835494+0000","last_undegraded":"2024-05-09T03:12:05.835494+0000","last_fullsized":"2024-05-09T03:12:05.835494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854065+0000","last_change":"2024-05-09T03:12:05.852504+0000","last_active":"2024-05-09T03:12:05.854065+0000","last_peered":"2024-05-09T03:12:05.854065+0000","last_clean":"2024-05-09T03:12:05.854065+0000","last_became_active":"2024-05-09T03:12:05.851857+0000","last_became_peered":"2024-05-09T03:12:05.851857+0000","last_unstale":"2024-05-09T03:12:05.854065+0000","last_undegraded":"2024-05-09T03:12:05.854065+0000","last_fullsized":"2024-05-09T03:12:05.854065+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.836813+0000","last_change":"2024-05-09T03:12:05.836813+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.836813+0000","last_undegraded":"2024-05-09T03:12:05.836813+0000","last_fullsized":"2024-05-09T03:12:05.836813+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.849695+0000","last_change":"2024-05-09T03:12:05.844269+0000","last_active":"2024-05-09T03:12:05.849695+0000","last_peered":"2024-05-09T03:12:05.849695+0000","last_clean":"2024-05-09T03:12:05.849695+0000","last_became_active":"2024-05-09T03:12:05.843939+0000","last_became_peered":"2024-05-09T03:12:05.843939+0000","last_unstale":"2024-05-09T03:12:05.849695+0000","last_undegraded":"2024-05-09T03:12:05.849695+0000","last_fullsized":"2024-05-09T03:12:05.849695+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.839358+0000","last_change":"2024-05-09T03:12:05.839358+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.839358+0000","last_undegraded":"2024-05-09T03:12:05.839358+0000","last_fullsized":"2024-05-09T03:12:05.839358+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851603+0000","last_change":"2024-05-09T03:12:05.847432+0000","last_active":"2024-05-09T03:12:05.851603+0000","last_peered":"2024-05-09T03:12:05.851603+0000","last_clean":"2024-05-09T03:12:05.851603+0000","last_became_active":"2024-05-09T03:12:05.847153+0000","last_became_peered":"2024-05-09T03:12:05.847153+0000","last_unstale":"2024-05-09T03:12:05.851603+0000","last_undegraded":"2024-05-09T03:12:05.851603+0000","last_fullsized":"2024-05-09T03:12:05.851603+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:02:27.480835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833512+0000","last_change":"2024-05-09T03:12:05.833512+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833512+0000","last_undegraded":"2024-05-09T03:12:05.833512+0000","last_fullsized":"2024-05-09T03:12:05.833512+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853273+0000","last_change":"2024-05-09T03:12:05.850218+0000","last_active":"2024-05-09T03:12:05.853273+0000","last_peered":"2024-05-09T03:12:05.853273+0000","last_clean":"2024-05-09T03:12:05.853273+0000","last_became_active":"2024-05-09T03:12:05.849743+0000","last_became_peered":"2024-05-09T03:12:05.849743+0000","last_unstale":"2024-05-09T03:12:05.853273+0000","last_undegraded":"2024-05-09T03:12:05.853273+0000","last_fullsized":"2024-05-09T03:12:05.853273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:07:40.635239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.833538+0000","last_change":"2024-05-09T03:12:05.833538+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.833538+0000","last_undegraded":"2024-05-09T03:12:05.833538+0000","last_fullsized":"2024-05-09T03:12:05.833538+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852904+0000","last_change":"2024-05-09T03:12:05.848391+0000","last_active":"2024-05-09T03:12:05.852904+0000","last_peered":"2024-05-09T03:12:05.852904+0000","last_clean":"2024-05-09T03:12:05.852904+0000","last_became_active":"2024-05-09T03:12:05.847525+0000","last_became_peered":"2024-05-09T03:12:05.847525+0000","last_unstale":"2024-05-09T03:12:05.852904+0000","last_undegraded":"2024-05-09T03:12:05.852904+0000","last_fullsized":"2024-05-09T03:12:05.852904+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:44:35.804811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.841979+0000","last_change":"2024-05-09T03:12:05.841979+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.841979+0000","last_undegraded":"2024-05-09T03:12:05.841979+0000","last_fullsized":"2024-05-09T03:12:05.841979+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.847707+0000","last_change":"2024-05-09T03:12:05.846693+0000","last_active":"2024-05-09T03:12:05.847707+0000","last_peered":"2024-05-09T03:12:05.847707+0000","last_clean":"2024-05-09T03:12:05.847707+0000","last_became_active":"2024-05-09T03:12:05.846379+0000","last_became_peered":"2024-05-09T03:12:05.846379+0000","last_unstale":"2024-05-09T03:12:05.847707+0000","last_undegraded":"2024-05-09T03:12:05.847707+0000","last_fullsized":"2024-05-09T03:12:05.847707+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:27:57.513277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.840940+0000","last_change":"2024-05-09T03:12:05.840940+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.840940+0000","last_undegraded":"2024-05-09T03:12:05.840940+0000","last_fullsized":"2024-05-09T03:12:05.840940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.846195+0000","last_change":"2024-05-09T03:12:05.843597+0000","last_active":"2024-05-09T03:12:05.846195+0000","last_peered":"2024-05-09T03:12:05.846195+0000","last_clean":"2024-05-09T03:12:05.846195+0000","last_became_active":"2024-05-09T03:12:05.843281+0000","last_became_peered":"2024-05-09T03:12:05.843281+0000","last_unstale":"2024-05-09T03:12:05.846195+0000","last_undegraded":"2024-05-09T03:12:05.846195+0000","last_fullsized":"2024-05-09T03:12:05.846195+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832473+0000","last_change":"2024-05-09T03:12:05.832473+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832473+0000","last_undegraded":"2024-05-09T03:12:05.832473+0000","last_fullsized":"2024-05-09T03:12:05.832473+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852261+0000","last_change":"2024-05-09T03:12:05.845237+0000","last_active":"2024-05-09T03:12:05.852261+0000","last_peered":"2024-05-09T03:12:05.852261+0000","last_clean":"2024-05-09T03:12:05.852261+0000","last_became_active":"2024-05-09T03:12:05.844921+0000","last_became_peered":"2024-05-09T03:12:05.844921+0000","last_unstale":"2024-05-09T03:12:05.852261+0000","last_undegraded":"2024-05-09T03:12:05.852261+0000","last_fullsized":"2024-05-09T03:12:05.852261+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:08:03.214517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835634+0000","last_change":"2024-05-09T03:12:05.835634+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835634+0000","last_undegraded":"2024-05-09T03:12:05.835634+0000","last_fullsized":"2024-05-09T03:12:05.835634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.854023+0000","last_change":"2024-05-09T03:12:05.850855+0000","last_active":"2024-05-09T03:12:05.854023+0000","last_peered":"2024-05-09T03:12:05.854023+0000","last_clean":"2024-05-09T03:12:05.854023+0000","last_became_active":"2024-05-09T03:12:05.850518+0000","last_became_peered":"2024-05-09T03:12:05.850518+0000","last_unstale":"2024-05-09T03:12:05.854023+0000","last_undegraded":"2024-05-09T03:12:05.854023+0000","last_fullsized":"2024-05-09T03:12:05.854023+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834748+0000","last_change":"2024-05-09T03:12:05.834748+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834748+0000","last_undegraded":"2024-05-09T03:12:05.834748+0000","last_fullsized":"2024-05-09T03:12:05.834748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.851159+0000","last_change":"2024-05-09T03:12:05.848645+0000","last_active":"2024-05-09T03:12:05.851159+0000","last_peered":"2024-05-09T03:12:05.851159+0000","last_clean":"2024-05-09T03:12:05.851159+0000","last_became_active":"2024-05-09T03:12:05.848326+0000","last_became_peered":"2024-05-09T03:12:05.848326+0000","last_unstale":"2024-05-09T03:12:05.851159+0000","last_undegraded":"2024-05-09T03:12:05.851159+0000","last_fullsized":"2024-05-09T03:12:05.851159+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:00:20.727622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835236+0000","last_change":"2024-05-09T03:12:05.835236+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835236+0000","last_undegraded":"2024-05-09T03:12:05.835236+0000","last_fullsized":"2024-05-09T03:12:05.835236+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.853515+0000","last_change":"2024-05-09T03:12:05.843892+0000","last_active":"2024-05-09T03:12:05.853515+0000","last_peered":"2024-05-09T03:12:05.853515+0000","last_clean":"2024-05-09T03:12:05.853515+0000","last_became_active":"2024-05-09T03:12:05.843459+0000","last_became_peered":"2024-05-09T03:12:05.843459+0000","last_unstale":"2024-05-09T03:12:05.853515+0000","last_undegraded":"2024-05-09T03:12:05.853515+0000","last_fullsized":"2024-05-09T03:12:05.853515+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:48.718539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.832475+0000","last_change":"2024-05-09T03:12:05.832475+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.832475+0000","last_undegraded":"2024-05-09T03:12:05.832475+0000","last_fullsized":"2024-05-09T03:12:05.832475+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.845868+0000","last_change":"2024-05-09T03:12:05.844488+0000","last_active":"2024-05-09T03:12:05.845868+0000","last_peered":"2024-05-09T03:12:05.845868+0000","last_clean":"2024-05-09T03:12:05.845868+0000","last_became_active":"2024-05-09T03:12:05.844183+0000","last_became_peered":"2024-05-09T03:12:05.844183+0000","last_unstale":"2024-05-09T03:12:05.845868+0000","last_undegraded":"2024-05-09T03:12:05.845868+0000","last_fullsized":"2024-05-09T03:12:05.845868+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:08:08.665854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.834642+0000","last_change":"2024-05-09T03:12:05.834642+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.834642+0000","last_undegraded":"2024-05-09T03:12:05.834642+0000","last_fullsized":"2024-05-09T03:12:05.834642+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852985+0000","last_change":"2024-05-09T03:12:05.849987+0000","last_active":"2024-05-09T03:12:05.852985+0000","last_peered":"2024-05-09T03:12:05.852985+0000","last_clean":"2024-05-09T03:12:05.852985+0000","last_became_active":"2024-05-09T03:12:05.848380+0000","last_became_peered":"2024-05-09T03:12:05.848380+0000","last_unstale":"2024-05-09T03:12:05.852985+0000","last_undegraded":"2024-05-09T03:12:05.852985+0000","last_fullsized":"2024-05-09T03:12:05.852985+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:09:38.070718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":53,"state":"creating+peering","last_fresh":"2024-05-09T03:12:05.835177+0000","last_change":"2024-05-09T03:12:05.835177+0000","last_active":"2024-05-09T03:12:05.823852+0000","last_peered":"2024-05-09T03:12:05.823852+0000","last_clean":"2024-05-09T03:12:05.823852+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-09T03:12:05.835177+0000","last_undegraded":"2024-05-09T03:12:05.835177+0000","last_fullsized":"2024-05-09T03:12:05.835177+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.852583+0000","last_change":"2024-05-09T03:12:05.851689+0000","last_active":"2024-05-09T03:12:05.852583+0000","last_peered":"2024-05-09T03:12:05.852583+0000","last_clean":"2024-05-09T03:12:05.852583+0000","last_became_active":"2024-05-09T03:12:05.851357+0000","last_became_peered":"2024-05-09T03:12:05.851357+0000","last_unstale":"2024-05-09T03:12:05.852583+0000","last_undegraded":"2024-05-09T03:12:05.852583+0000","last_fullsized":"2024-05-09T03:12:05.852583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:49.416402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":170,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-05-09T03:12:05.836038+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:12:05.836038+0000","last_peered":"2024-05-09T03:12:05.836038+0000","last_clean":"2024-05-09T03:12:05.836038+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:12:05.836038+0000","last_undegraded":"2024-05-09T03:12:05.836038+0000","last_fullsized":"2024-05-09T03:12:05.836038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1228800,"data_stored":850793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":634880,"data_stored":260425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":622592,"data_stored":254158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":844526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:12:15.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:12:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.624+0000 7fe15537d640 1 Processor -- start 2024-05-09T03:12:15.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.624+0000 7fe15537d640 1 -- start start 2024-05-09T03:12:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe15537d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe150160610 0x7fe1500b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe15537d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500b7d20 0x7fe1500b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe15537d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1500b8130 con 0x7fe1500d57a0 2024-05-09T03:12:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe15537d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe1500d5bb0 con 0x7fe150160610 2024-05-09T03:12:15.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe15537d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1500d49e0 con 0x7fe1500b7d20 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe14e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe150160610 0x7fe1500b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe14f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500b7d20 0x7fe1500b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe14effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe1500d57a0 0x7fe1500b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:33146/0 (socket says 172.21.15.46:33146) 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.625+0000 7fe14effd640 1 -- 172.21.15.46:0/135774168 learned_addr learned my addr 172.21.15.46:0/135774168 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.626+0000 7fe14f7fe640 1 -- 172.21.15.46:0/135774168 >> v2:172.21.15.88:3300/0 conn(0x7fe150160610 msgr2=0x7fe1500b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.626+0000 7fe14f7fe640 1 --2- 172.21.15.46:0/135774168 >> v2:172.21.15.88:3300/0 conn(0x7fe150160610 0x7fe1500b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.626+0000 7fe14f7fe640 1 -- 172.21.15.46:0/135774168 >> v1:172.21.15.88:6789/0 conn(0x7fe1500d57a0 legacy=0x7fe1500b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:15.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.626+0000 7fe14f7fe640 1 -- 172.21.15.46:0/135774168 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1500d4cc0 con 0x7fe1500b7d20 2024-05-09T03:12:15.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.627+0000 7fe14f7fe640 1 --2- 172.21.15.46:0/135774168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500b7d20 0x7fe1500b9450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe148009d90 tx=0x7fe14802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d09b2ca2b4b574f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:15.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.627+0000 7fe14dffb640 1 -- 172.21.15.46:0/135774168 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe148007740 con 0x7fe1500b7d20 2024-05-09T03:12:15.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.627+0000 7fe14dffb640 1 -- 172.21.15.46:0/135774168 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1480078e0 con 0x7fe1500b7d20 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.627+0000 7fe14dffb640 1 -- 172.21.15.46:0/135774168 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe148007cb0 con 0x7fe1500b7d20 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.627+0000 7fe15537d640 1 -- 172.21.15.46:0/135774168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500b7d20 msgr2=0x7fe1500b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.627+0000 7fe15537d640 1 --2- 172.21.15.46:0/135774168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500b7d20 0x7fe1500b9450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe148009d90 tx=0x7fe14802a660 comp rx=0 tx=0).stop 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.628+0000 7fe15537d640 1 -- 172.21.15.46:0/135774168 shutdown_connections 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.628+0000 7fe15537d640 1 --2- 172.21.15.46:0/135774168 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500b7d20 0x7fe1500b9450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.628+0000 7fe15537d640 1 --2- 172.21.15.46:0/135774168 >> v2:172.21.15.88:3300/0 conn(0x7fe150160610 0x7fe1500b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.628+0000 7fe15537d640 1 -- 172.21.15.46:0/135774168 >> 172.21.15.46:0/135774168 conn(0x7fe1500743e0 msgr2=0x7fe1500d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.628+0000 7fe15537d640 1 -- 172.21.15.46:0/135774168 shutdown_connections 2024-05-09T03:12:15.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.628+0000 7fe15537d640 1 -- 172.21.15.46:0/135774168 wait complete. 2024-05-09T03:12:15.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.630+0000 7fe15537d640 1 Processor -- start 2024-05-09T03:12:15.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.630+0000 7fe15537d640 1 -- start start 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe15537d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1500b7d20 0x7fe1500d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe15537d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 0x7fe1500ac750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe15537d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 0x7fe1500adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe15537d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe1500d15b0 con 0x7fe150160610 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe15537d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe150160a20 con 0x7fe1500b7d20 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe15537d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1500d1920 con 0x7fe1500d57a0 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 0x7fe1500adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1500b7d20 0x7fe1500d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 0x7fe1500ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 0x7fe1500adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53316/0 (socket says 172.21.15.46:53316) 2024-05-09T03:12:15.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1500b7d20 0x7fe1500d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:41856/0 (socket says 172.21.15.46:41856) 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 0x7fe1500ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:40464/0 (socket says 172.21.15.46:40464) 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14f7fe640 1 -- 172.21.15.46:0/385206945 learned_addr learned my addr 172.21.15.46:0/385206945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 -- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1500b7d20 msgr2=0x7fe1500d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1500b7d20 0x7fe1500d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 -- 172.21.15.46:0/385206945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 msgr2=0x7fe1500adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 0x7fe1500adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14e7fc640 1 -- 172.21.15.46:0/385206945 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1500acc90 con 0x7fe1500d57a0 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.631+0000 7fe14f7fe640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 0x7fe1500adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.632+0000 7fe14e7fc640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 0x7fe1500ac750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe14400e8a0 tx=0x7fe14400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.632+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144017070 con 0x7fe1500d57a0 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.632+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe14400c940 con 0x7fe1500d57a0 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.632+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144009e60 con 0x7fe1500d57a0 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.632+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1500ad1a0 con 0x7fe1500d57a0 2024-05-09T03:12:15.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.632+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1500ad760 con 0x7fe1500d57a0 2024-05-09T03:12:15.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.633+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1500acf10 con 0x7fe1500d57a0 2024-05-09T03:12:15.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.634+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe14401a070 con 0x7fe1500d57a0 2024-05-09T03:12:15.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.635+0000 7fe1377fe640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe13c0762d0 0x7fe13c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:15.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.635+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fe144097a30 con 0x7fe1500d57a0 2024-05-09T03:12:15.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.635+0000 7fe14effd640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe13c0762d0 0x7fe13c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:15.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.635+0000 7fe14effd640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe13c0762d0 0x7fe13c0787e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe140000f60 tx=0x7fe140009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:15.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.639+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe144060db0 con 0x7fe1500d57a0 2024-05-09T03:12:15.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.827+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe15006a270 con 0x7fe13c0762d0 2024-05-09T03:12:15.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.833+0000 7fe1377fe640 1 -- 172.21.15.46:0/385206945 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7fe15006a270 con 0x7fe13c0762d0 2024-05-09T03:12:15.837 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:15.838 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:12:15.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe13c0762d0 msgr2=0x7fe13c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:15.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe13c0762d0 0x7fe13c0787e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe140000f60 tx=0x7fe140009040 comp rx=0 tx=0).stop 2024-05-09T03:12:15.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 msgr2=0x7fe1500ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:15.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 0x7fe1500ac750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe14400e8a0 tx=0x7fe14400ed60 comp rx=0 tx=0).stop 2024-05-09T03:12:15.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 shutdown_connections 2024-05-09T03:12:15.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe13c0762d0 0x7fe13c0787e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe150160610 0x7fe1500adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1500d57a0 0x7fe1500ac750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 --2- 172.21.15.46:0/385206945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1500b7d20 0x7fe1500d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 >> 172.21.15.46:0/385206945 conn(0x7fe1500743e0 msgr2=0x7fe1500d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.836+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 shutdown_connections 2024-05-09T03:12:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:15.837+0000 7fe15537d640 1 -- 172.21.15.46:0/385206945 wait complete. 2024-05-09T03:12:15.849 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-05-09T03:12:14.532188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239208,"kb_used_data":6376,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735512,"statfs":{"total":773094113280,"available":772849164288,"internally_reserved":0,"allocated":6529024,"data_stored":3373556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002207"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635631+0000","last_change":"2024-05-09T03:12:06.875061+0000","last_active":"2024-05-09T03:12:08.635631+0000","last_peered":"2024-05-09T03:12:08.635631+0000","last_clean":"2024-05-09T03:12:08.635631+0000","last_became_active":"2024-05-09T03:12:06.874731+0000","last_became_peered":"2024-05-09T03:12:06.874731+0000","last_unstale":"2024-05-09T03:12:08.635631+0000","last_undegraded":"2024-05-09T03:12:08.635631+0000","last_fullsized":"2024-05-09T03:12:08.635631+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:57:10.890405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848321+0000","last_change":"2024-05-09T03:12:06.882099+0000","last_active":"2024-05-09T03:12:07.848321+0000","last_peered":"2024-05-09T03:12:07.848321+0000","last_clean":"2024-05-09T03:12:07.848321+0000","last_became_active":"2024-05-09T03:12:06.881586+0000","last_became_peered":"2024-05-09T03:12:06.881586+0000","last_unstale":"2024-05-09T03:12:07.848321+0000","last_undegraded":"2024-05-09T03:12:07.848321+0000","last_fullsized":"2024-05-09T03:12:07.848321+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:38:30.148607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851956+0000","last_change":"2024-05-09T03:12:06.867678+0000","last_active":"2024-05-09T03:12:07.851956+0000","last_peered":"2024-05-09T03:12:07.851956+0000","last_clean":"2024-05-09T03:12:07.851956+0000","last_became_active":"2024-05-09T03:12:06.867377+0000","last_became_peered":"2024-05-09T03:12:06.867377+0000","last_unstale":"2024-05-09T03:12:07.851956+0000","last_undegraded":"2024-05-09T03:12:07.851956+0000","last_fullsized":"2024-05-09T03:12:07.851956+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:46:42.789654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851209+0000","last_change":"2024-05-09T03:12:06.862298+0000","last_active":"2024-05-09T03:12:07.851209+0000","last_peered":"2024-05-09T03:12:07.851209+0000","last_clean":"2024-05-09T03:12:07.851209+0000","last_became_active":"2024-05-09T03:12:06.861971+0000","last_became_peered":"2024-05-09T03:12:06.861971+0000","last_unstale":"2024-05-09T03:12:07.851209+0000","last_undegraded":"2024-05-09T03:12:07.851209+0000","last_fullsized":"2024-05-09T03:12:07.851209+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:36.745274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635566+0000","last_change":"2024-05-09T03:12:06.859928+0000","last_active":"2024-05-09T03:12:08.635566+0000","last_peered":"2024-05-09T03:12:08.635566+0000","last_clean":"2024-05-09T03:12:08.635566+0000","last_became_active":"2024-05-09T03:12:06.859202+0000","last_became_peered":"2024-05-09T03:12:06.859202+0000","last_unstale":"2024-05-09T03:12:08.635566+0000","last_undegraded":"2024-05-09T03:12:08.635566+0000","last_fullsized":"2024-05-09T03:12:08.635566+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:01:23.212568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.861981+0000","last_change":"2024-05-09T03:12:06.870932+0000","last_active":"2024-05-09T03:12:07.861981+0000","last_peered":"2024-05-09T03:12:07.861981+0000","last_clean":"2024-05-09T03:12:07.861981+0000","last_became_active":"2024-05-09T03:12:06.870613+0000","last_became_peered":"2024-05-09T03:12:06.870613+0000","last_unstale":"2024-05-09T03:12:07.861981+0000","last_undegraded":"2024-05-09T03:12:07.861981+0000","last_fullsized":"2024-05-09T03:12:07.861981+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:42.575954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635576+0000","last_change":"2024-05-09T03:12:06.857946+0000","last_active":"2024-05-09T03:12:08.635576+0000","last_peered":"2024-05-09T03:12:08.635576+0000","last_clean":"2024-05-09T03:12:08.635576+0000","last_became_active":"2024-05-09T03:12:06.857581+0000","last_became_peered":"2024-05-09T03:12:06.857581+0000","last_unstale":"2024-05-09T03:12:08.635576+0000","last_undegraded":"2024-05-09T03:12:08.635576+0000","last_fullsized":"2024-05-09T03:12:08.635576+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:50.182626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847526+0000","last_change":"2024-05-09T03:12:06.875867+0000","last_active":"2024-05-09T03:12:07.847526+0000","last_peered":"2024-05-09T03:12:07.847526+0000","last_clean":"2024-05-09T03:12:07.847526+0000","last_became_active":"2024-05-09T03:12:06.875260+0000","last_became_peered":"2024-05-09T03:12:06.875260+0000","last_unstale":"2024-05-09T03:12:07.847526+0000","last_undegraded":"2024-05-09T03:12:07.847526+0000","last_fullsized":"2024-05-09T03:12:07.847526+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:23:55.877585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.636015+0000","last_change":"2024-05-09T03:12:06.865565+0000","last_active":"2024-05-09T03:12:08.636015+0000","last_peered":"2024-05-09T03:12:08.636015+0000","last_clean":"2024-05-09T03:12:08.636015+0000","last_became_active":"2024-05-09T03:12:06.864907+0000","last_became_peered":"2024-05-09T03:12:06.864907+0000","last_unstale":"2024-05-09T03:12:08.636015+0000","last_undegraded":"2024-05-09T03:12:08.636015+0000","last_fullsized":"2024-05-09T03:12:08.636015+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:47:47.310049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134729+0000","last_change":"2024-05-09T03:12:06.860069+0000","last_active":"2024-05-09T03:12:08.134729+0000","last_peered":"2024-05-09T03:12:08.134729+0000","last_clean":"2024-05-09T03:12:08.134729+0000","last_became_active":"2024-05-09T03:12:06.859753+0000","last_became_peered":"2024-05-09T03:12:06.859753+0000","last_unstale":"2024-05-09T03:12:08.134729+0000","last_undegraded":"2024-05-09T03:12:08.134729+0000","last_fullsized":"2024-05-09T03:12:08.134729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:08.088069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134674+0000","last_change":"2024-05-09T03:12:06.877781+0000","last_active":"2024-05-09T03:12:08.134674+0000","last_peered":"2024-05-09T03:12:08.134674+0000","last_clean":"2024-05-09T03:12:08.134674+0000","last_became_active":"2024-05-09T03:12:06.877259+0000","last_became_peered":"2024-05-09T03:12:06.877259+0000","last_unstale":"2024-05-09T03:12:08.134674+0000","last_undegraded":"2024-05-09T03:12:08.134674+0000","last_fullsized":"2024-05-09T03:12:08.134674+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:01.729685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851389+0000","last_change":"2024-05-09T03:12:06.863520+0000","last_active":"2024-05-09T03:12:07.851389+0000","last_peered":"2024-05-09T03:12:07.851389+0000","last_clean":"2024-05-09T03:12:07.851389+0000","last_became_active":"2024-05-09T03:12:06.863242+0000","last_became_peered":"2024-05-09T03:12:06.863242+0000","last_unstale":"2024-05-09T03:12:07.851389+0000","last_undegraded":"2024-05-09T03:12:07.851389+0000","last_fullsized":"2024-05-09T03:12:07.851389+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:11.570412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.865092+0000","last_change":"2024-05-09T03:12:06.874999+0000","last_active":"2024-05-09T03:12:07.865092+0000","last_peered":"2024-05-09T03:12:07.865092+0000","last_clean":"2024-05-09T03:12:07.865092+0000","last_became_active":"2024-05-09T03:12:06.874677+0000","last_became_peered":"2024-05-09T03:12:06.874677+0000","last_unstale":"2024-05-09T03:12:07.865092+0000","last_undegraded":"2024-05-09T03:12:07.865092+0000","last_fullsized":"2024-05-09T03:12:07.865092+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:40.407189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635560+0000","last_change":"2024-05-09T03:12:06.860673+0000","last_active":"2024-05-09T03:12:08.635560+0000","last_peered":"2024-05-09T03:12:08.635560+0000","last_clean":"2024-05-09T03:12:08.635560+0000","last_became_active":"2024-05-09T03:12:06.860345+0000","last_became_peered":"2024-05-09T03:12:06.860345+0000","last_unstale":"2024-05-09T03:12:08.635560+0000","last_undegraded":"2024-05-09T03:12:08.635560+0000","last_fullsized":"2024-05-09T03:12:08.635560+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:01:17.463773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635726+0000","last_change":"2024-05-09T03:12:06.854981+0000","last_active":"2024-05-09T03:12:08.635726+0000","last_peered":"2024-05-09T03:12:08.635726+0000","last_clean":"2024-05-09T03:12:08.635726+0000","last_became_active":"2024-05-09T03:12:06.854612+0000","last_became_peered":"2024-05-09T03:12:06.854612+0000","last_unstale":"2024-05-09T03:12:08.635726+0000","last_undegraded":"2024-05-09T03:12:08.635726+0000","last_fullsized":"2024-05-09T03:12:08.635726+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:02.652281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.861876+0000","last_change":"2024-05-09T03:12:06.861055+0000","last_active":"2024-05-09T03:12:07.861876+0000","last_peered":"2024-05-09T03:12:07.861876+0000","last_clean":"2024-05-09T03:12:07.861876+0000","last_became_active":"2024-05-09T03:12:06.860680+0000","last_became_peered":"2024-05-09T03:12:06.860680+0000","last_unstale":"2024-05-09T03:12:07.861876+0000","last_undegraded":"2024-05-09T03:12:07.861876+0000","last_fullsized":"2024-05-09T03:12:07.861876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:42:50.501210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848260+0000","last_change":"2024-05-09T03:12:06.881110+0000","last_active":"2024-05-09T03:12:07.848260+0000","last_peered":"2024-05-09T03:12:07.848260+0000","last_clean":"2024-05-09T03:12:07.848260+0000","last_became_active":"2024-05-09T03:12:06.880773+0000","last_became_peered":"2024-05-09T03:12:06.880773+0000","last_unstale":"2024-05-09T03:12:07.848260+0000","last_undegraded":"2024-05-09T03:12:07.848260+0000","last_fullsized":"2024-05-09T03:12:07.848260+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:29:53.938095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852715+0000","last_change":"2024-05-09T03:12:06.876465+0000","last_active":"2024-05-09T03:12:07.852715+0000","last_peered":"2024-05-09T03:12:07.852715+0000","last_clean":"2024-05-09T03:12:07.852715+0000","last_became_active":"2024-05-09T03:12:06.876186+0000","last_became_peered":"2024-05-09T03:12:06.876186+0000","last_unstale":"2024-05-09T03:12:07.852715+0000","last_undegraded":"2024-05-09T03:12:07.852715+0000","last_fullsized":"2024-05-09T03:12:07.852715+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:23.900460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635605+0000","last_change":"2024-05-09T03:12:06.878403+0000","last_active":"2024-05-09T03:12:08.635605+0000","last_peered":"2024-05-09T03:12:08.635605+0000","last_clean":"2024-05-09T03:12:08.635605+0000","last_became_active":"2024-05-09T03:12:06.878057+0000","last_became_peered":"2024-05-09T03:12:06.878057+0000","last_unstale":"2024-05-09T03:12:08.635605+0000","last_undegraded":"2024-05-09T03:12:08.635605+0000","last_fullsized":"2024-05-09T03:12:08.635605+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:55:33.504752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848145+0000","last_change":"2024-05-09T03:12:06.876387+0000","last_active":"2024-05-09T03:12:07.848145+0000","last_peered":"2024-05-09T03:12:07.848145+0000","last_clean":"2024-05-09T03:12:07.848145+0000","last_became_active":"2024-05-09T03:12:06.876058+0000","last_became_peered":"2024-05-09T03:12:06.876058+0000","last_unstale":"2024-05-09T03:12:07.848145+0000","last_undegraded":"2024-05-09T03:12:07.848145+0000","last_fullsized":"2024-05-09T03:12:07.848145+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:11:08.999384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852195+0000","last_change":"2024-05-09T03:12:06.865690+0000","last_active":"2024-05-09T03:12:07.852195+0000","last_peered":"2024-05-09T03:12:07.852195+0000","last_clean":"2024-05-09T03:12:07.852195+0000","last_became_active":"2024-05-09T03:12:06.864776+0000","last_became_peered":"2024-05-09T03:12:06.864776+0000","last_unstale":"2024-05-09T03:12:07.852195+0000","last_undegraded":"2024-05-09T03:12:07.852195+0000","last_fullsized":"2024-05-09T03:12:07.852195+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:13:22.149775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847817+0000","last_change":"2024-05-09T03:12:06.871459+0000","last_active":"2024-05-09T03:12:07.847817+0000","last_peered":"2024-05-09T03:12:07.847817+0000","last_clean":"2024-05-09T03:12:07.847817+0000","last_became_active":"2024-05-09T03:12:06.871123+0000","last_became_peered":"2024-05-09T03:12:06.871123+0000","last_unstale":"2024-05-09T03:12:07.847817+0000","last_undegraded":"2024-05-09T03:12:07.847817+0000","last_fullsized":"2024-05-09T03:12:07.847817+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:52.705994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851444+0000","last_change":"2024-05-09T03:12:06.874942+0000","last_active":"2024-05-09T03:12:07.851444+0000","last_peered":"2024-05-09T03:12:07.851444+0000","last_clean":"2024-05-09T03:12:07.851444+0000","last_became_active":"2024-05-09T03:12:06.874618+0000","last_became_peered":"2024-05-09T03:12:06.874618+0000","last_unstale":"2024-05-09T03:12:07.851444+0000","last_undegraded":"2024-05-09T03:12:07.851444+0000","last_fullsized":"2024-05-09T03:12:07.851444+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:45.355664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.864838+0000","last_change":"2024-05-09T03:12:06.872238+0000","last_active":"2024-05-09T03:12:07.864838+0000","last_peered":"2024-05-09T03:12:07.864838+0000","last_clean":"2024-05-09T03:12:07.864838+0000","last_became_active":"2024-05-09T03:12:06.871862+0000","last_became_peered":"2024-05-09T03:12:06.871862+0000","last_unstale":"2024-05-09T03:12:07.864838+0000","last_undegraded":"2024-05-09T03:12:07.864838+0000","last_fullsized":"2024-05-09T03:12:07.864838+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:21.878636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635638+0000","last_change":"2024-05-09T03:12:06.860637+0000","last_active":"2024-05-09T03:12:08.635638+0000","last_peered":"2024-05-09T03:12:08.635638+0000","last_clean":"2024-05-09T03:12:08.635638+0000","last_became_active":"2024-05-09T03:12:06.859694+0000","last_became_peered":"2024-05-09T03:12:06.859694+0000","last_unstale":"2024-05-09T03:12:08.635638+0000","last_undegraded":"2024-05-09T03:12:08.635638+0000","last_fullsized":"2024-05-09T03:12:08.635638+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:32:51.583207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851289+0000","last_change":"2024-05-09T03:12:06.859862+0000","last_active":"2024-05-09T03:12:07.851289+0000","last_peered":"2024-05-09T03:12:07.851289+0000","last_clean":"2024-05-09T03:12:07.851289+0000","last_became_active":"2024-05-09T03:12:06.858952+0000","last_became_peered":"2024-05-09T03:12:06.858952+0000","last_unstale":"2024-05-09T03:12:07.851289+0000","last_undegraded":"2024-05-09T03:12:07.851289+0000","last_fullsized":"2024-05-09T03:12:07.851289+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:46:42.789654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862487+0000","last_change":"2024-05-09T03:12:06.870999+0000","last_active":"2024-05-09T03:12:07.862487+0000","last_peered":"2024-05-09T03:12:07.862487+0000","last_clean":"2024-05-09T03:12:07.862487+0000","last_became_active":"2024-05-09T03:12:06.870689+0000","last_became_peered":"2024-05-09T03:12:06.870689+0000","last_unstale":"2024-05-09T03:12:07.862487+0000","last_undegraded":"2024-05-09T03:12:07.862487+0000","last_fullsized":"2024-05-09T03:12:07.862487+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:23.900460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635428+0000","last_change":"2024-05-09T03:12:06.864169+0000","last_active":"2024-05-09T03:12:08.635428+0000","last_peered":"2024-05-09T03:12:08.635428+0000","last_clean":"2024-05-09T03:12:08.635428+0000","last_became_active":"2024-05-09T03:12:06.863829+0000","last_became_peered":"2024-05-09T03:12:06.863829+0000","last_unstale":"2024-05-09T03:12:08.635428+0000","last_undegraded":"2024-05-09T03:12:08.635428+0000","last_fullsized":"2024-05-09T03:12:08.635428+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:45.674921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848539+0000","last_change":"2024-05-09T03:12:06.876255+0000","last_active":"2024-05-09T03:12:07.848539+0000","last_peered":"2024-05-09T03:12:07.848539+0000","last_clean":"2024-05-09T03:12:07.848539+0000","last_became_active":"2024-05-09T03:12:06.875932+0000","last_became_peered":"2024-05-09T03:12:06.875932+0000","last_unstale":"2024-05-09T03:12:07.848539+0000","last_undegraded":"2024-05-09T03:12:07.848539+0000","last_fullsized":"2024-05-09T03:12:07.848539+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:52.165822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852699+0000","last_change":"2024-05-09T03:12:06.867560+0000","last_active":"2024-05-09T03:12:07.852699+0000","last_peered":"2024-05-09T03:12:07.852699+0000","last_clean":"2024-05-09T03:12:07.852699+0000","last_became_active":"2024-05-09T03:12:06.867294+0000","last_became_peered":"2024-05-09T03:12:06.867294+0000","last_unstale":"2024-05-09T03:12:07.852699+0000","last_undegraded":"2024-05-09T03:12:07.852699+0000","last_fullsized":"2024-05-09T03:12:07.852699+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:03:58.630759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635182+0000","last_change":"2024-05-09T03:12:06.859574+0000","last_active":"2024-05-09T03:12:08.635182+0000","last_peered":"2024-05-09T03:12:08.635182+0000","last_clean":"2024-05-09T03:12:08.635182+0000","last_became_active":"2024-05-09T03:12:06.859213+0000","last_became_peered":"2024-05-09T03:12:06.859213+0000","last_unstale":"2024-05-09T03:12:08.635182+0000","last_undegraded":"2024-05-09T03:12:08.635182+0000","last_fullsized":"2024-05-09T03:12:08.635182+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:03:09.421442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.861947+0000","last_change":"2024-05-09T03:12:06.871371+0000","last_active":"2024-05-09T03:12:07.861947+0000","last_peered":"2024-05-09T03:12:07.861947+0000","last_clean":"2024-05-09T03:12:07.861947+0000","last_became_active":"2024-05-09T03:12:06.870967+0000","last_became_peered":"2024-05-09T03:12:06.870967+0000","last_unstale":"2024-05-09T03:12:07.861947+0000","last_undegraded":"2024-05-09T03:12:07.861947+0000","last_fullsized":"2024-05-09T03:12:07.861947+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:13:22.149775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862133+0000","last_change":"2024-05-09T03:12:06.864731+0000","last_active":"2024-05-09T03:12:07.862133+0000","last_peered":"2024-05-09T03:12:07.862133+0000","last_clean":"2024-05-09T03:12:07.862133+0000","last_became_active":"2024-05-09T03:12:06.864410+0000","last_became_peered":"2024-05-09T03:12:06.864410+0000","last_unstale":"2024-05-09T03:12:07.862133+0000","last_undegraded":"2024-05-09T03:12:07.862133+0000","last_fullsized":"2024-05-09T03:12:07.862133+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:45.355664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"54'4","reported_seq":18,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862212+0000","last_change":"2024-05-09T03:12:05.849076+0000","last_active":"2024-05-09T03:12:07.862212+0000","last_peered":"2024-05-09T03:12:07.862212+0000","last_clean":"2024-05-09T03:12:07.862212+0000","last_became_active":"2024-05-09T03:12:05.848768+0000","last_became_peered":"2024-05-09T03:12:05.848768+0000","last_unstale":"2024-05-09T03:12:07.862212+0000","last_undegraded":"2024-05-09T03:12:07.862212+0000","last_fullsized":"2024-05-09T03:12:07.862212+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:56:50.381916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134732+0000","last_change":"2024-05-09T03:12:06.866413+0000","last_active":"2024-05-09T03:12:08.134732+0000","last_peered":"2024-05-09T03:12:08.134732+0000","last_clean":"2024-05-09T03:12:08.134732+0000","last_became_active":"2024-05-09T03:12:06.866047+0000","last_became_peered":"2024-05-09T03:12:06.866047+0000","last_unstale":"2024-05-09T03:12:08.134732+0000","last_undegraded":"2024-05-09T03:12:08.134732+0000","last_fullsized":"2024-05-09T03:12:08.134732+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:23:01.757384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848303+0000","last_change":"2024-05-09T03:12:05.846715+0000","last_active":"2024-05-09T03:12:07.848303+0000","last_peered":"2024-05-09T03:12:07.848303+0000","last_clean":"2024-05-09T03:12:07.848303+0000","last_became_active":"2024-05-09T03:12:05.846448+0000","last_became_peered":"2024-05-09T03:12:05.846448+0000","last_unstale":"2024-05-09T03:12:07.848303+0000","last_undegraded":"2024-05-09T03:12:07.848303+0000","last_fullsized":"2024-05-09T03:12:07.848303+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:46:41.784846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862322+0000","last_change":"2024-05-09T03:12:06.861982+0000","last_active":"2024-05-09T03:12:07.862322+0000","last_peered":"2024-05-09T03:12:07.862322+0000","last_clean":"2024-05-09T03:12:07.862322+0000","last_became_active":"2024-05-09T03:12:06.861646+0000","last_became_peered":"2024-05-09T03:12:06.861646+0000","last_unstale":"2024-05-09T03:12:07.862322+0000","last_undegraded":"2024-05-09T03:12:07.862322+0000","last_fullsized":"2024-05-09T03:12:07.862322+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:03:58.630759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862609+0000","last_change":"2024-05-09T03:12:05.845815+0000","last_active":"2024-05-09T03:12:07.862609+0000","last_peered":"2024-05-09T03:12:07.862609+0000","last_clean":"2024-05-09T03:12:07.862609+0000","last_became_active":"2024-05-09T03:12:05.845509+0000","last_became_peered":"2024-05-09T03:12:05.845509+0000","last_unstale":"2024-05-09T03:12:07.862609+0000","last_undegraded":"2024-05-09T03:12:07.862609+0000","last_fullsized":"2024-05-09T03:12:07.862609+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:01.647474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862498+0000","last_change":"2024-05-09T03:12:06.869850+0000","last_active":"2024-05-09T03:12:07.862498+0000","last_peered":"2024-05-09T03:12:07.862498+0000","last_clean":"2024-05-09T03:12:07.862498+0000","last_became_active":"2024-05-09T03:12:06.862630+0000","last_became_peered":"2024-05-09T03:12:06.862630+0000","last_unstale":"2024-05-09T03:12:07.862498+0000","last_undegraded":"2024-05-09T03:12:07.862498+0000","last_fullsized":"2024-05-09T03:12:07.862498+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:09.994335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851508+0000","last_change":"2024-05-09T03:12:05.849601+0000","last_active":"2024-05-09T03:12:07.851508+0000","last_peered":"2024-05-09T03:12:07.851508+0000","last_clean":"2024-05-09T03:12:07.851508+0000","last_became_active":"2024-05-09T03:12:05.849230+0000","last_became_peered":"2024-05-09T03:12:05.849230+0000","last_unstale":"2024-05-09T03:12:07.851508+0000","last_undegraded":"2024-05-09T03:12:07.851508+0000","last_fullsized":"2024-05-09T03:12:07.851508+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:05.791451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847877+0000","last_change":"2024-05-09T03:12:06.870158+0000","last_active":"2024-05-09T03:12:07.847877+0000","last_peered":"2024-05-09T03:12:07.847877+0000","last_clean":"2024-05-09T03:12:07.847877+0000","last_became_active":"2024-05-09T03:12:06.869312+0000","last_became_peered":"2024-05-09T03:12:06.869312+0000","last_unstale":"2024-05-09T03:12:07.847877+0000","last_undegraded":"2024-05-09T03:12:07.847877+0000","last_fullsized":"2024-05-09T03:12:07.847877+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:39:23.666377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635610+0000","last_change":"2024-05-09T03:12:05.853186+0000","last_active":"2024-05-09T03:12:08.635610+0000","last_peered":"2024-05-09T03:12:08.635610+0000","last_clean":"2024-05-09T03:12:08.635610+0000","last_became_active":"2024-05-09T03:12:05.852815+0000","last_became_peered":"2024-05-09T03:12:05.852815+0000","last_unstale":"2024-05-09T03:12:08.635610+0000","last_undegraded":"2024-05-09T03:12:08.635610+0000","last_fullsized":"2024-05-09T03:12:08.635610+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:12:20.873828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848012+0000","last_change":"2024-05-09T03:12:06.885225+0000","last_active":"2024-05-09T03:12:07.848012+0000","last_peered":"2024-05-09T03:12:07.848012+0000","last_clean":"2024-05-09T03:12:07.848012+0000","last_became_active":"2024-05-09T03:12:06.884844+0000","last_became_peered":"2024-05-09T03:12:06.884844+0000","last_unstale":"2024-05-09T03:12:07.848012+0000","last_undegraded":"2024-05-09T03:12:07.848012+0000","last_fullsized":"2024-05-09T03:12:07.848012+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:51:15.367766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.861998+0000","last_change":"2024-05-09T03:12:05.844443+0000","last_active":"2024-05-09T03:12:07.861998+0000","last_peered":"2024-05-09T03:12:07.861998+0000","last_clean":"2024-05-09T03:12:07.861998+0000","last_became_active":"2024-05-09T03:12:05.844119+0000","last_became_peered":"2024-05-09T03:12:05.844119+0000","last_unstale":"2024-05-09T03:12:07.861998+0000","last_undegraded":"2024-05-09T03:12:07.861998+0000","last_fullsized":"2024-05-09T03:12:07.861998+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:05:49.177819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.864976+0000","last_change":"2024-05-09T03:12:06.862772+0000","last_active":"2024-05-09T03:12:07.864976+0000","last_peered":"2024-05-09T03:12:07.864976+0000","last_clean":"2024-05-09T03:12:07.864976+0000","last_became_active":"2024-05-09T03:12:06.862391+0000","last_became_peered":"2024-05-09T03:12:06.862391+0000","last_unstale":"2024-05-09T03:12:07.864976+0000","last_undegraded":"2024-05-09T03:12:07.864976+0000","last_fullsized":"2024-05-09T03:12:07.864976+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:44.368668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.850818+0000","last_change":"2024-05-09T03:12:05.847498+0000","last_active":"2024-05-09T03:12:07.850818+0000","last_peered":"2024-05-09T03:12:07.850818+0000","last_clean":"2024-05-09T03:12:07.850818+0000","last_became_active":"2024-05-09T03:12:05.847170+0000","last_became_peered":"2024-05-09T03:12:05.847170+0000","last_unstale":"2024-05-09T03:12:07.850818+0000","last_undegraded":"2024-05-09T03:12:07.850818+0000","last_fullsized":"2024-05-09T03:12:07.850818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:56:50.381916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847602+0000","last_change":"2024-05-09T03:12:06.884253+0000","last_active":"2024-05-09T03:12:07.847602+0000","last_peered":"2024-05-09T03:12:07.847602+0000","last_clean":"2024-05-09T03:12:07.847602+0000","last_became_active":"2024-05-09T03:12:06.880835+0000","last_became_peered":"2024-05-09T03:12:06.880835+0000","last_unstale":"2024-05-09T03:12:07.847602+0000","last_undegraded":"2024-05-09T03:12:07.847602+0000","last_fullsized":"2024-05-09T03:12:07.847602+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:42.180179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635544+0000","last_change":"2024-05-09T03:12:05.847308+0000","last_active":"2024-05-09T03:12:08.635544+0000","last_peered":"2024-05-09T03:12:08.635544+0000","last_clean":"2024-05-09T03:12:08.635544+0000","last_became_active":"2024-05-09T03:12:05.846406+0000","last_became_peered":"2024-05-09T03:12:05.846406+0000","last_unstale":"2024-05-09T03:12:08.635544+0000","last_undegraded":"2024-05-09T03:12:08.635544+0000","last_fullsized":"2024-05-09T03:12:08.635544+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:43.363860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.636017+0000","last_change":"2024-05-09T03:12:06.854884+0000","last_active":"2024-05-09T03:12:08.636017+0000","last_peered":"2024-05-09T03:12:08.636017+0000","last_clean":"2024-05-09T03:12:08.636017+0000","last_became_active":"2024-05-09T03:12:06.854548+0000","last_became_peered":"2024-05-09T03:12:06.854548+0000","last_unstale":"2024-05-09T03:12:08.636017+0000","last_undegraded":"2024-05-09T03:12:08.636017+0000","last_fullsized":"2024-05-09T03:12:08.636017+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:53:46.381313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852472+0000","last_change":"2024-05-09T03:12:05.847459+0000","last_active":"2024-05-09T03:12:07.852472+0000","last_peered":"2024-05-09T03:12:07.852472+0000","last_clean":"2024-05-09T03:12:07.852472+0000","last_became_active":"2024-05-09T03:12:05.847106+0000","last_became_peered":"2024-05-09T03:12:05.847106+0000","last_unstale":"2024-05-09T03:12:07.852472+0000","last_undegraded":"2024-05-09T03:12:07.852472+0000","last_fullsized":"2024-05-09T03:12:07.852472+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:43:46.511932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852156+0000","last_change":"2024-05-09T03:12:06.871202+0000","last_active":"2024-05-09T03:12:07.852156+0000","last_peered":"2024-05-09T03:12:07.852156+0000","last_clean":"2024-05-09T03:12:07.852156+0000","last_became_active":"2024-05-09T03:12:06.870113+0000","last_became_peered":"2024-05-09T03:12:06.870113+0000","last_unstale":"2024-05-09T03:12:07.852156+0000","last_undegraded":"2024-05-09T03:12:07.852156+0000","last_fullsized":"2024-05-09T03:12:07.852156+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:11.570412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848379+0000","last_change":"2024-05-09T03:12:05.849037+0000","last_active":"2024-05-09T03:12:07.848379+0000","last_peered":"2024-05-09T03:12:07.848379+0000","last_clean":"2024-05-09T03:12:07.848379+0000","last_became_active":"2024-05-09T03:12:05.848283+0000","last_became_peered":"2024-05-09T03:12:05.848283+0000","last_unstale":"2024-05-09T03:12:07.848379+0000","last_undegraded":"2024-05-09T03:12:07.848379+0000","last_fullsized":"2024-05-09T03:12:07.848379+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:46:10.565605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134565+0000","last_change":"2024-05-09T03:12:06.854859+0000","last_active":"2024-05-09T03:12:08.134565+0000","last_peered":"2024-05-09T03:12:08.134565+0000","last_clean":"2024-05-09T03:12:08.134565+0000","last_became_active":"2024-05-09T03:12:06.854495+0000","last_became_peered":"2024-05-09T03:12:06.854495+0000","last_unstale":"2024-05-09T03:12:08.134565+0000","last_undegraded":"2024-05-09T03:12:08.134565+0000","last_fullsized":"2024-05-09T03:12:08.134565+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:40.407189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"54'3","reported_seq":17,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852302+0000","last_change":"2024-05-09T03:12:05.847630+0000","last_active":"2024-05-09T03:12:07.852302+0000","last_peered":"2024-05-09T03:12:07.852302+0000","last_clean":"2024-05-09T03:12:07.852302+0000","last_became_active":"2024-05-09T03:12:05.847302+0000","last_became_peered":"2024-05-09T03:12:05.847302+0000","last_unstale":"2024-05-09T03:12:07.852302+0000","last_undegraded":"2024-05-09T03:12:07.852302+0000","last_fullsized":"2024-05-09T03:12:07.852302+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:56:50.381916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852141+0000","last_change":"2024-05-09T03:12:06.869195+0000","last_active":"2024-05-09T03:12:07.852141+0000","last_peered":"2024-05-09T03:12:07.852141+0000","last_clean":"2024-05-09T03:12:07.852141+0000","last_became_active":"2024-05-09T03:12:06.868850+0000","last_became_peered":"2024-05-09T03:12:06.868850+0000","last_unstale":"2024-05-09T03:12:07.852141+0000","last_undegraded":"2024-05-09T03:12:07.852141+0000","last_fullsized":"2024-05-09T03:12:07.852141+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:42:50.501210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"54'6","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635937+0000","last_change":"2024-05-09T03:12:05.845847+0000","last_active":"2024-05-09T03:12:08.635937+0000","last_peered":"2024-05-09T03:12:08.635937+0000","last_clean":"2024-05-09T03:12:08.635937+0000","last_became_active":"2024-05-09T03:12:05.845474+0000","last_became_peered":"2024-05-09T03:12:05.845474+0000","last_unstale":"2024-05-09T03:12:08.635937+0000","last_undegraded":"2024-05-09T03:12:08.635937+0000","last_fullsized":"2024-05-09T03:12:08.635937+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:39.402381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.850879+0000","last_change":"2024-05-09T03:12:06.863980+0000","last_active":"2024-05-09T03:12:07.850879+0000","last_peered":"2024-05-09T03:12:07.850879+0000","last_clean":"2024-05-09T03:12:07.850879+0000","last_became_active":"2024-05-09T03:12:06.863652+0000","last_became_peered":"2024-05-09T03:12:06.863652+0000","last_unstale":"2024-05-09T03:12:07.850879+0000","last_undegraded":"2024-05-09T03:12:07.850879+0000","last_fullsized":"2024-05-09T03:12:07.850879+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:39:26.973074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848131+0000","last_change":"2024-05-09T03:12:05.854030+0000","last_active":"2024-05-09T03:12:07.848131+0000","last_peered":"2024-05-09T03:12:07.848131+0000","last_clean":"2024-05-09T03:12:07.848131+0000","last_became_active":"2024-05-09T03:12:05.853677+0000","last_became_peered":"2024-05-09T03:12:05.853677+0000","last_unstale":"2024-05-09T03:12:07.848131+0000","last_undegraded":"2024-05-09T03:12:07.848131+0000","last_fullsized":"2024-05-09T03:12:07.848131+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:42:49.496402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852573+0000","last_change":"2024-05-09T03:12:06.864844+0000","last_active":"2024-05-09T03:12:07.852573+0000","last_peered":"2024-05-09T03:12:07.852573+0000","last_clean":"2024-05-09T03:12:07.852573+0000","last_became_active":"2024-05-09T03:12:06.864527+0000","last_became_peered":"2024-05-09T03:12:06.864527+0000","last_unstale":"2024-05-09T03:12:07.852573+0000","last_undegraded":"2024-05-09T03:12:07.852573+0000","last_fullsized":"2024-05-09T03:12:07.852573+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:03:42.575954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.864712+0000","last_change":"2024-05-09T03:12:05.843325+0000","last_active":"2024-05-09T03:12:07.864712+0000","last_peered":"2024-05-09T03:12:07.864712+0000","last_clean":"2024-05-09T03:12:07.864712+0000","last_became_active":"2024-05-09T03:12:05.842923+0000","last_became_peered":"2024-05-09T03:12:05.842923+0000","last_unstale":"2024-05-09T03:12:07.864712+0000","last_undegraded":"2024-05-09T03:12:07.864712+0000","last_fullsized":"2024-05-09T03:12:07.864712+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:03:48.718539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635527+0000","last_change":"2024-05-09T03:12:06.863731+0000","last_active":"2024-05-09T03:12:08.635527+0000","last_peered":"2024-05-09T03:12:08.635527+0000","last_clean":"2024-05-09T03:12:08.635527+0000","last_became_active":"2024-05-09T03:12:06.863404+0000","last_became_peered":"2024-05-09T03:12:06.863404+0000","last_unstale":"2024-05-09T03:12:08.635527+0000","last_undegraded":"2024-05-09T03:12:08.635527+0000","last_fullsized":"2024-05-09T03:12:08.635527+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:50:00.174555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851040+0000","last_change":"2024-05-09T03:12:05.848661+0000","last_active":"2024-05-09T03:12:07.851040+0000","last_peered":"2024-05-09T03:12:07.851040+0000","last_clean":"2024-05-09T03:12:07.851040+0000","last_became_active":"2024-05-09T03:12:05.848010+0000","last_became_peered":"2024-05-09T03:12:05.848010+0000","last_unstale":"2024-05-09T03:12:07.851040+0000","last_undegraded":"2024-05-09T03:12:07.851040+0000","last_fullsized":"2024-05-09T03:12:07.851040+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:05.105468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852184+0000","last_change":"2024-05-09T03:12:06.880957+0000","last_active":"2024-05-09T03:12:07.852184+0000","last_peered":"2024-05-09T03:12:07.852184+0000","last_clean":"2024-05-09T03:12:07.852184+0000","last_became_active":"2024-05-09T03:12:06.880606+0000","last_became_peered":"2024-05-09T03:12:06.880606+0000","last_unstale":"2024-05-09T03:12:07.852184+0000","last_undegraded":"2024-05-09T03:12:07.852184+0000","last_fullsized":"2024-05-09T03:12:07.852184+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:04.425230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"54'8","reported_seq":22,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851358+0000","last_change":"2024-05-09T03:12:05.849792+0000","last_active":"2024-05-09T03:12:07.851358+0000","last_peered":"2024-05-09T03:12:07.851358+0000","last_clean":"2024-05-09T03:12:07.851358+0000","last_became_active":"2024-05-09T03:12:05.849439+0000","last_became_peered":"2024-05-09T03:12:05.849439+0000","last_unstale":"2024-05-09T03:12:07.851358+0000","last_undegraded":"2024-05-09T03:12:07.851358+0000","last_fullsized":"2024-05-09T03:12:07.851358+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:05.791451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852871+0000","last_change":"2024-05-09T03:12:06.868296+0000","last_active":"2024-05-09T03:12:07.852871+0000","last_peered":"2024-05-09T03:12:07.852871+0000","last_clean":"2024-05-09T03:12:07.852871+0000","last_became_active":"2024-05-09T03:12:06.867962+0000","last_became_peered":"2024-05-09T03:12:06.867962+0000","last_unstale":"2024-05-09T03:12:07.852871+0000","last_undegraded":"2024-05-09T03:12:07.852871+0000","last_fullsized":"2024-05-09T03:12:07.852871+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:09.994335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848396+0000","last_change":"2024-05-09T03:12:05.853093+0000","last_active":"2024-05-09T03:12:07.848396+0000","last_peered":"2024-05-09T03:12:07.848396+0000","last_clean":"2024-05-09T03:12:07.848396+0000","last_became_active":"2024-05-09T03:12:05.852692+0000","last_became_peered":"2024-05-09T03:12:05.852692+0000","last_unstale":"2024-05-09T03:12:07.848396+0000","last_undegraded":"2024-05-09T03:12:07.848396+0000","last_fullsized":"2024-05-09T03:12:07.848396+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:30:19.520927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851760+0000","last_change":"2024-05-09T03:12:06.865094+0000","last_active":"2024-05-09T03:12:07.851760+0000","last_peered":"2024-05-09T03:12:07.851760+0000","last_clean":"2024-05-09T03:12:07.851760+0000","last_became_active":"2024-05-09T03:12:06.864684+0000","last_became_peered":"2024-05-09T03:12:06.864684+0000","last_unstale":"2024-05-09T03:12:07.851760+0000","last_undegraded":"2024-05-09T03:12:07.851760+0000","last_fullsized":"2024-05-09T03:12:07.851760+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:42:50.501210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862251+0000","last_change":"2024-05-09T03:12:05.849439+0000","last_active":"2024-05-09T03:12:07.862251+0000","last_peered":"2024-05-09T03:12:07.862251+0000","last_clean":"2024-05-09T03:12:07.862251+0000","last_became_active":"2024-05-09T03:12:05.849134+0000","last_became_peered":"2024-05-09T03:12:05.849134+0000","last_unstale":"2024-05-09T03:12:07.862251+0000","last_undegraded":"2024-05-09T03:12:07.862251+0000","last_fullsized":"2024-05-09T03:12:07.862251+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:05.105468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848224+0000","last_change":"2024-05-09T03:12:06.872585+0000","last_active":"2024-05-09T03:12:07.848224+0000","last_peered":"2024-05-09T03:12:07.848224+0000","last_clean":"2024-05-09T03:12:07.848224+0000","last_became_active":"2024-05-09T03:12:06.872200+0000","last_became_peered":"2024-05-09T03:12:06.872200+0000","last_unstale":"2024-05-09T03:12:07.848224+0000","last_undegraded":"2024-05-09T03:12:07.848224+0000","last_fullsized":"2024-05-09T03:12:07.848224+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:01:21.244420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"54'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851141+0000","last_change":"2024-05-09T03:12:05.852504+0000","last_active":"2024-05-09T03:12:07.851141+0000","last_peered":"2024-05-09T03:12:07.851141+0000","last_clean":"2024-05-09T03:12:07.851141+0000","last_became_active":"2024-05-09T03:12:05.851857+0000","last_became_peered":"2024-05-09T03:12:05.851857+0000","last_unstale":"2024-05-09T03:12:07.851141+0000","last_undegraded":"2024-05-09T03:12:07.851141+0000","last_fullsized":"2024-05-09T03:12:07.851141+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:43:46.511932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852163+0000","last_change":"2024-05-09T03:12:06.870469+0000","last_active":"2024-05-09T03:12:07.852163+0000","last_peered":"2024-05-09T03:12:07.852163+0000","last_clean":"2024-05-09T03:12:07.852163+0000","last_became_active":"2024-05-09T03:12:06.870046+0000","last_became_peered":"2024-05-09T03:12:06.870046+0000","last_unstale":"2024-05-09T03:12:07.852163+0000","last_undegraded":"2024-05-09T03:12:07.852163+0000","last_fullsized":"2024-05-09T03:12:07.852163+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:30:20.525735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862266+0000","last_change":"2024-05-09T03:12:05.844269+0000","last_active":"2024-05-09T03:12:07.862266+0000","last_peered":"2024-05-09T03:12:07.862266+0000","last_clean":"2024-05-09T03:12:07.862266+0000","last_became_active":"2024-05-09T03:12:05.843939+0000","last_became_peered":"2024-05-09T03:12:05.843939+0000","last_unstale":"2024-05-09T03:12:07.862266+0000","last_undegraded":"2024-05-09T03:12:07.862266+0000","last_fullsized":"2024-05-09T03:12:07.862266+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:01:16.458965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.850857+0000","last_change":"2024-05-09T03:12:06.858635+0000","last_active":"2024-05-09T03:12:07.850857+0000","last_peered":"2024-05-09T03:12:07.850857+0000","last_clean":"2024-05-09T03:12:07.850857+0000","last_became_active":"2024-05-09T03:12:06.858335+0000","last_became_peered":"2024-05-09T03:12:06.858335+0000","last_unstale":"2024-05-09T03:12:07.850857+0000","last_undegraded":"2024-05-09T03:12:07.850857+0000","last_fullsized":"2024-05-09T03:12:07.850857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:50:44.394201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.848456+0000","last_change":"2024-05-09T03:12:05.847432+0000","last_active":"2024-05-09T03:12:07.848456+0000","last_peered":"2024-05-09T03:12:07.848456+0000","last_clean":"2024-05-09T03:12:07.848456+0000","last_became_active":"2024-05-09T03:12:05.847153+0000","last_became_peered":"2024-05-09T03:12:05.847153+0000","last_unstale":"2024-05-09T03:12:07.848456+0000","last_undegraded":"2024-05-09T03:12:07.848456+0000","last_fullsized":"2024-05-09T03:12:07.848456+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:13:21.144968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862279+0000","last_change":"2024-05-09T03:12:06.872195+0000","last_active":"2024-05-09T03:12:07.862279+0000","last_peered":"2024-05-09T03:12:07.862279+0000","last_clean":"2024-05-09T03:12:07.862279+0000","last_became_active":"2024-05-09T03:12:06.869847+0000","last_became_peered":"2024-05-09T03:12:06.869847+0000","last_unstale":"2024-05-09T03:12:07.862279+0000","last_undegraded":"2024-05-09T03:12:07.862279+0000","last_fullsized":"2024-05-09T03:12:07.862279+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:54:35.410978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134427+0000","last_change":"2024-05-09T03:12:05.850218+0000","last_active":"2024-05-09T03:12:08.134427+0000","last_peered":"2024-05-09T03:12:08.134427+0000","last_clean":"2024-05-09T03:12:08.134427+0000","last_became_active":"2024-05-09T03:12:05.849743+0000","last_became_peered":"2024-05-09T03:12:05.849743+0000","last_unstale":"2024-05-09T03:12:08.134427+0000","last_undegraded":"2024-05-09T03:12:08.134427+0000","last_fullsized":"2024-05-09T03:12:08.134427+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:43:02.488792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.861907+0000","last_change":"2024-05-09T03:12:06.863134+0000","last_active":"2024-05-09T03:12:07.861907+0000","last_peered":"2024-05-09T03:12:07.861907+0000","last_clean":"2024-05-09T03:12:07.861907+0000","last_became_active":"2024-05-09T03:12:06.862803+0000","last_became_peered":"2024-05-09T03:12:06.862803+0000","last_unstale":"2024-05-09T03:12:07.861907+0000","last_undegraded":"2024-05-09T03:12:07.861907+0000","last_fullsized":"2024-05-09T03:12:07.861907+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:22:04.425230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.850793+0000","last_change":"2024-05-09T03:12:05.848391+0000","last_active":"2024-05-09T03:12:07.850793+0000","last_peered":"2024-05-09T03:12:07.850793+0000","last_clean":"2024-05-09T03:12:07.850793+0000","last_became_active":"2024-05-09T03:12:05.847525+0000","last_became_peered":"2024-05-09T03:12:05.847525+0000","last_unstale":"2024-05-09T03:12:07.850793+0000","last_undegraded":"2024-05-09T03:12:07.850793+0000","last_fullsized":"2024-05-09T03:12:07.850793+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:19:07.083261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851472+0000","last_change":"2024-05-09T03:12:06.865546+0000","last_active":"2024-05-09T03:12:07.851472+0000","last_peered":"2024-05-09T03:12:07.851472+0000","last_clean":"2024-05-09T03:12:07.851472+0000","last_became_active":"2024-05-09T03:12:06.864710+0000","last_became_peered":"2024-05-09T03:12:06.864710+0000","last_unstale":"2024-05-09T03:12:07.851472+0000","last_undegraded":"2024-05-09T03:12:07.851472+0000","last_fullsized":"2024-05-09T03:12:07.851472+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:30:20.525735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"54'2","reported_seq":16,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134747+0000","last_change":"2024-05-09T03:12:05.846693+0000","last_active":"2024-05-09T03:12:08.134747+0000","last_peered":"2024-05-09T03:12:08.134747+0000","last_clean":"2024-05-09T03:12:08.134747+0000","last_became_active":"2024-05-09T03:12:05.846379+0000","last_became_peered":"2024-05-09T03:12:05.846379+0000","last_unstale":"2024-05-09T03:12:08.134747+0000","last_undegraded":"2024-05-09T03:12:08.134747+0000","last_fullsized":"2024-05-09T03:12:08.134747+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:48.599525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851878+0000","last_change":"2024-05-09T03:12:06.862556+0000","last_active":"2024-05-09T03:12:07.851878+0000","last_peered":"2024-05-09T03:12:07.851878+0000","last_clean":"2024-05-09T03:12:07.851878+0000","last_became_active":"2024-05-09T03:12:06.862080+0000","last_became_peered":"2024-05-09T03:12:06.862080+0000","last_unstale":"2024-05-09T03:12:07.851878+0000","last_undegraded":"2024-05-09T03:12:07.851878+0000","last_fullsized":"2024-05-09T03:12:07.851878+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:13:22.149775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"55'1","reported_seq":15,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862359+0000","last_change":"2024-05-09T03:12:05.843597+0000","last_active":"2024-05-09T03:12:07.862359+0000","last_peered":"2024-05-09T03:12:07.862359+0000","last_clean":"2024-05-09T03:12:07.862359+0000","last_became_active":"2024-05-09T03:12:05.843281+0000","last_became_peered":"2024-05-09T03:12:05.843281+0000","last_unstale":"2024-05-09T03:12:07.862359+0000","last_undegraded":"2024-05-09T03:12:07.862359+0000","last_fullsized":"2024-05-09T03:12:07.862359+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:59.169748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134762+0000","last_change":"2024-05-09T03:12:06.851165+0000","last_active":"2024-05-09T03:12:08.134762+0000","last_peered":"2024-05-09T03:12:08.134762+0000","last_clean":"2024-05-09T03:12:08.134762+0000","last_became_active":"2024-05-09T03:12:06.850752+0000","last_became_peered":"2024-05-09T03:12:06.850752+0000","last_unstale":"2024-05-09T03:12:08.134762+0000","last_undegraded":"2024-05-09T03:12:08.134762+0000","last_fullsized":"2024-05-09T03:12:08.134762+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:06.796259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.864936+0000","last_change":"2024-05-09T03:12:05.845237+0000","last_active":"2024-05-09T03:12:07.864936+0000","last_peered":"2024-05-09T03:12:07.864936+0000","last_clean":"2024-05-09T03:12:07.864936+0000","last_became_active":"2024-05-09T03:12:05.844921+0000","last_became_peered":"2024-05-09T03:12:05.844921+0000","last_unstale":"2024-05-09T03:12:07.864936+0000","last_undegraded":"2024-05-09T03:12:07.864936+0000","last_fullsized":"2024-05-09T03:12:07.864936+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:15.398427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.862674+0000","last_change":"2024-05-09T03:12:06.870289+0000","last_active":"2024-05-09T03:12:07.862674+0000","last_peered":"2024-05-09T03:12:07.862674+0000","last_clean":"2024-05-09T03:12:07.862674+0000","last_became_active":"2024-05-09T03:12:06.863066+0000","last_became_peered":"2024-05-09T03:12:06.863066+0000","last_unstale":"2024-05-09T03:12:07.862674+0000","last_undegraded":"2024-05-09T03:12:07.862674+0000","last_fullsized":"2024-05-09T03:12:07.862674+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:57:54.536572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852316+0000","last_change":"2024-05-09T03:12:05.850855+0000","last_active":"2024-05-09T03:12:07.852316+0000","last_peered":"2024-05-09T03:12:07.852316+0000","last_clean":"2024-05-09T03:12:07.852316+0000","last_became_active":"2024-05-09T03:12:05.850518+0000","last_became_peered":"2024-05-09T03:12:05.850518+0000","last_unstale":"2024-05-09T03:12:07.852316+0000","last_undegraded":"2024-05-09T03:12:07.852316+0000","last_fullsized":"2024-05-09T03:12:07.852316+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:49:59.169748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847995+0000","last_change":"2024-05-09T03:12:06.884114+0000","last_active":"2024-05-09T03:12:07.847995+0000","last_peered":"2024-05-09T03:12:07.847995+0000","last_clean":"2024-05-09T03:12:07.847995+0000","last_became_active":"2024-05-09T03:12:06.881550+0000","last_became_peered":"2024-05-09T03:12:06.881550+0000","last_unstale":"2024-05-09T03:12:07.847995+0000","last_undegraded":"2024-05-09T03:12:07.847995+0000","last_fullsized":"2024-05-09T03:12:07.847995+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:47.992766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134352+0000","last_change":"2024-05-09T03:12:05.848645+0000","last_active":"2024-05-09T03:12:08.134352+0000","last_peered":"2024-05-09T03:12:08.134352+0000","last_clean":"2024-05-09T03:12:08.134352+0000","last_became_active":"2024-05-09T03:12:05.848326+0000","last_became_peered":"2024-05-09T03:12:05.848326+0000","last_unstale":"2024-05-09T03:12:08.134352+0000","last_undegraded":"2024-05-09T03:12:08.134352+0000","last_fullsized":"2024-05-09T03:12:08.134352+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:49.416402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.852656+0000","last_change":"2024-05-09T03:12:06.884948+0000","last_active":"2024-05-09T03:12:07.852656+0000","last_peered":"2024-05-09T03:12:07.852656+0000","last_clean":"2024-05-09T03:12:07.852656+0000","last_became_active":"2024-05-09T03:12:06.884647+0000","last_became_peered":"2024-05-09T03:12:06.884647+0000","last_unstale":"2024-05-09T03:12:07.852656+0000","last_undegraded":"2024-05-09T03:12:07.852656+0000","last_fullsized":"2024-05-09T03:12:07.852656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:54:35.410978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.635672+0000","last_change":"2024-05-09T03:12:05.843892+0000","last_active":"2024-05-09T03:12:08.635672+0000","last_peered":"2024-05-09T03:12:08.635672+0000","last_clean":"2024-05-09T03:12:08.635672+0000","last_became_active":"2024-05-09T03:12:05.843459+0000","last_became_peered":"2024-05-09T03:12:05.843459+0000","last_unstale":"2024-05-09T03:12:08.635672+0000","last_undegraded":"2024-05-09T03:12:08.635672+0000","last_fullsized":"2024-05-09T03:12:08.635672+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:06:00.724878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134299+0000","last_change":"2024-05-09T03:12:06.851429+0000","last_active":"2024-05-09T03:12:08.134299+0000","last_peered":"2024-05-09T03:12:08.134299+0000","last_clean":"2024-05-09T03:12:08.134299+0000","last_became_active":"2024-05-09T03:12:06.851034+0000","last_became_peered":"2024-05-09T03:12:06.851034+0000","last_unstale":"2024-05-09T03:12:08.134299+0000","last_undegraded":"2024-05-09T03:12:08.134299+0000","last_fullsized":"2024-05-09T03:12:08.134299+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:33:44.368668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:08.134307+0000","last_change":"2024-05-09T03:12:05.844488+0000","last_active":"2024-05-09T03:12:08.134307+0000","last_peered":"2024-05-09T03:12:08.134307+0000","last_clean":"2024-05-09T03:12:08.134307+0000","last_became_active":"2024-05-09T03:12:05.844183+0000","last_became_peered":"2024-05-09T03:12:05.844183+0000","last_unstale":"2024-05-09T03:12:08.134307+0000","last_undegraded":"2024-05-09T03:12:08.134307+0000","last_fullsized":"2024-05-09T03:12:08.134307+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:08:03.214517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847815+0000","last_change":"2024-05-09T03:12:06.884215+0000","last_active":"2024-05-09T03:12:07.847815+0000","last_peered":"2024-05-09T03:12:07.847815+0000","last_clean":"2024-05-09T03:12:07.847815+0000","last_became_active":"2024-05-09T03:12:06.880234+0000","last_became_peered":"2024-05-09T03:12:06.880234+0000","last_unstale":"2024-05-09T03:12:07.847815+0000","last_undegraded":"2024-05-09T03:12:07.847815+0000","last_fullsized":"2024-05-09T03:12:07.847815+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:22:07.435324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847591+0000","last_change":"2024-05-09T03:12:05.849987+0000","last_active":"2024-05-09T03:12:07.847591+0000","last_peered":"2024-05-09T03:12:07.847591+0000","last_clean":"2024-05-09T03:12:07.847591+0000","last_became_active":"2024-05-09T03:12:05.848380+0000","last_became_peered":"2024-05-09T03:12:05.848380+0000","last_unstale":"2024-05-09T03:12:07.847591+0000","last_undegraded":"2024-05-09T03:12:07.847591+0000","last_fullsized":"2024-05-09T03:12:07.847591+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:35.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.847525+0000","last_change":"2024-05-09T03:12:06.883462+0000","last_active":"2024-05-09T03:12:07.847525+0000","last_peered":"2024-05-09T03:12:07.847525+0000","last_clean":"2024-05-09T03:12:07.847525+0000","last_became_active":"2024-05-09T03:12:06.883138+0000","last_became_peered":"2024-05-09T03:12:06.883138+0000","last_unstale":"2024-05-09T03:12:07.847525+0000","last_undegraded":"2024-05-09T03:12:07.847525+0000","last_fullsized":"2024-05-09T03:12:07.847525+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:05.823852+0000","last_clean_scrub_stamp":"2024-05-09T03:12:05.823852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:32:55.795383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.865177+0000","last_change":"2024-05-09T03:12:05.851689+0000","last_active":"2024-05-09T03:12:07.865177+0000","last_peered":"2024-05-09T03:12:07.865177+0000","last_clean":"2024-05-09T03:12:07.865177+0000","last_became_active":"2024-05-09T03:12:05.851357+0000","last_became_peered":"2024-05-09T03:12:05.851357+0000","last_unstale":"2024-05-09T03:12:07.865177+0000","last_undegraded":"2024-05-09T03:12:07.865177+0000","last_fullsized":"2024-05-09T03:12:07.865177+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:12:04.819044+0000","last_clean_scrub_stamp":"2024-05-09T03:12:04.819044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:08:12.530600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":174,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-05-09T03:12:07.851514+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:12:07.851514+0000","last_peered":"2024-05-09T03:12:07.851514+0000","last_clean":"2024-05-09T03:12:07.851514+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:12:07.851514+0000","last_undegraded":"2024-05-09T03:12:07.851514+0000","last_fullsized":"2024-05-09T03:12:07.851514+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1261568,"data_stored":864028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":659456,"data_stored":273500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":275112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31840,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340000,"statfs":{"total":96636764160,"available":96604160000,"internally_reserved":0,"allocated":667648,"data_stored":273556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":667648,"data_stored":273556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31836,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340004,"statfs":{"total":96636764160,"available":96604164096,"internally_reserved":0,"allocated":663552,"data_stored":273590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":866062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:12:15.853 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:12:15.853 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T03:12:16.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.009+0000 7f32ebeb1640 1 Processor -- start 2024-05-09T03:12:16.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.009+0000 7f32ebeb1640 1 -- start start 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ebeb1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ebeb1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32e40cfb90 0x7f32e406e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ebeb1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32e406d790 con 0x7f32e408cd60 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ebeb1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f32e406dda0 con 0x7f32e40cfb90 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ebeb1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32e40d23b0 con 0x7f32e4160910 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32e40cfb90 0x7f32e406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32e40cfb90 0x7f32e406e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43864/0 (socket says 172.21.15.46:43864) 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 -- 172.21.15.46:0/3772930523 learned_addr learned my addr 172.21.15.46:0/3772930523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:16.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32e944c640 1 --2- 172.21.15.46:0/3772930523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 -- 172.21.15.46:0/3772930523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 msgr2=0x7f32e40d1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:16.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 --2- 172.21.15.46:0/3772930523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40d1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 -- 172.21.15.46:0/3772930523 >> v1:172.21.15.88:6789/0 conn(0x7f32e408cd60 legacy=0x7f32e40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.011+0000 7f32ea44e640 1 -- 172.21.15.46:0/3772930523 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e406da30 con 0x7f32e40cfb90 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.012+0000 7f32ea44e640 1 --2- 172.21.15.46:0/3772930523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e40cfb90 0x7f32e406e230 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f32dc009870 tx=0x7f32dc02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95ca8c57f29ab82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.012+0000 7f32e8c4b640 1 -- 172.21.15.46:0/3772930523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc037070 con 0x7f32e40cfb90 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.012+0000 7f32e8c4b640 1 -- 172.21.15.46:0/3772930523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32dc032440 con 0x7f32e40cfb90 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.012+0000 7f32e8c4b640 1 -- 172.21.15.46:0/3772930523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32dc032810 con 0x7f32e40cfb90 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3772930523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e40cfb90 msgr2=0x7f32e406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3772930523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e40cfb90 0x7f32e406e230 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f32dc009870 tx=0x7f32dc02a130 comp rx=0 tx=0).stop 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3772930523 shutdown_connections 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3772930523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e40cfb90 0x7f32e406e230 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3772930523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40d1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3772930523 >> 172.21.15.46:0/3772930523 conn(0x7f32e40d04e0 msgr2=0x7f32e40d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:16.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3772930523 shutdown_connections 2024-05-09T03:12:16.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.013+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3772930523 wait complete. 2024-05-09T03:12:16.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.015+0000 7f32ebeb1640 1 Processor -- start 2024-05-09T03:12:16.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.015+0000 7f32ebeb1640 1 -- start start 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32ebeb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 0x7f32e4153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32ebeb1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32e40cfb90 0x7f32e40cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32ebeb1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32ebeb1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f32e406e770 con 0x7f32e408cd60 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32ebeb1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32e40d1210 con 0x7f32e40cfb90 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32ebeb1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32e406e8f0 con 0x7f32e4160910 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32e9c4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 0x7f32e4153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32e9c4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 0x7f32e4153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43878/0 (socket says 172.21.15.46:43878) 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.016+0000 7f32e9c4d640 1 -- 172.21.15.46:0/3019464335 learned_addr learned my addr 172.21.15.46:0/3019464335 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e944c640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32e40cfb90 0x7f32e40cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32ea44e640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e9c4d640 1 -- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 msgr2=0x7f32e40cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e9c4d640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e9c4d640 1 -- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32e40cfb90 msgr2=0x7f32e40cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e9c4d640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32e40cfb90 0x7f32e40cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e9c4d640 1 -- 172.21.15.46:0/3019464335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e4153d60 con 0x7f32e408cd60 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e944c640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32e40cfb90 0x7f32e40cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32ea44e640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32e9c4d640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 0x7f32e4153820 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f32e000c760 tx=0x7f32e000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0003b70 con 0x7f32e408cd60 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e0003d30 con 0x7f32e408cd60 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e40cd6d0 con 0x7f32e408cd60 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e000aa30 con 0x7f32e408cd60 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.017+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e40cb300 con 0x7f32e408cd60 2024-05-09T03:12:16.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.018+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e40b64a0 con 0x7f32e408cd60 2024-05-09T03:12:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.019+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f32e000abd0 con 0x7f32e408cd60 2024-05-09T03:12:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.019+0000 7f32da7fc640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32c80762d0 0x7f32c80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.020+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f32e0097510 con 0x7f32e408cd60 2024-05-09T03:12:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.020+0000 7f32e944c640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32c80762d0 0x7f32c80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.020+0000 7f32e944c640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32c80762d0 0x7f32c80787e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f32e41f9d20 tx=0x7f32d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.024+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f32e40b64a0 con 0x7f32e408cd60 2024-05-09T03:12:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.287+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f32e406a270 con 0x7f32e408cd60 2024-05-09T03:12:16.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.290+0000 7f32da7fc640 1 -- 172.21.15.46:0/3019464335 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f32e0060de0 con 0x7f32e408cd60 2024-05-09T03:12:16.291 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:12:16.292 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:12:16.292 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-05-09T03:12:16.292 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-05-09T03:12:16.292 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T03:12:16.292 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32c80762d0 msgr2=0x7f32c80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32c80762d0 0x7f32c80787e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f32e41f9d20 tx=0x7f32d4008040 comp rx=0 tx=0).stop 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 msgr2=0x7f32e4153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 0x7f32e4153820 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f32e000c760 tx=0x7f32e000cc20 comp rx=0 tx=0).stop 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 shutdown_connections 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f32e4160910 0x7f32e40cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32e40cfb90 0x7f32e40cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32c80762d0 0x7f32c80787e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 --2- 172.21.15.46:0/3019464335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32e408cd60 0x7f32e4153820 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 >> 172.21.15.46:0/3019464335 conn(0x7f32e40d04e0 msgr2=0x7f32e4071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:16.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 shutdown_connections 2024-05-09T03:12:16.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.292+0000 7f32ebeb1640 1 -- 172.21.15.46:0/3019464335 wait complete. 2024-05-09T03:12:16.305 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:16.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.417+0000 7f1ad80b1640 1 Processor -- start 2024-05-09T03:12:16.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.418+0000 7f1ad80b1640 1 -- start start 2024-05-09T03:12:16.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad80b1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ad00d57a0 0x7f1ad0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad80b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ad00b4af0 0x7f1ad00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad80b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ad00d49e0 con 0x7f1ad00b77e0 2024-05-09T03:12:16.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad80b1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad00d5bb0 con 0x7f1ad00b4af0 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad80b1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1ad00d4cc0 con 0x7f1ad00d57a0 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad564c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ad00b4af0 0x7f1ad00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad664e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1ad00b77e0 0x7f1ad00b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56564/0 (socket says 172.21.15.46:56564) 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad664e640 1 -- 172.21.15.46:0/637573336 learned_addr learned my addr 172.21.15.46:0/637573336 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.419+0000 7f1ad5e4d640 1 --2- 172.21.15.46:0/637573336 >> v2:172.21.15.88:3300/0 conn(0x7f1ad00d57a0 0x7f1ad0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad564c640 1 -- 172.21.15.46:0/637573336 >> v2:172.21.15.88:3300/0 conn(0x7f1ad00d57a0 msgr2=0x7f1ad0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad564c640 1 --2- 172.21.15.46:0/637573336 >> v2:172.21.15.88:3300/0 conn(0x7f1ad00d57a0 0x7f1ad0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad564c640 1 -- 172.21.15.46:0/637573336 >> v1:172.21.15.46:6789/0 conn(0x7f1ad00b77e0 legacy=0x7f1ad00b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad4e4b640 1 -- 172.21.15.46:0/637573336 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914661715 0 0) 0x7f1ad00d49e0 con 0x7f1ad00b77e0 2024-05-09T03:12:16.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad564c640 1 -- 172.21.15.46:0/637573336 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad00d4fa0 con 0x7f1ad00b4af0 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad564c640 1 --2- 172.21.15.46:0/637573336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00b4f00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1ad008ed20 tx=0x7f1acc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a1b27a971610e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.420+0000 7f1ad4e4b640 1 -- 172.21.15.46:0/637573336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc037070 con 0x7f1ad00b4af0 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad4e4b640 1 -- 172.21.15.46:0/637573336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1acc032440 con 0x7f1ad00b4af0 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad4e4b640 1 -- 172.21.15.46:0/637573336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc032810 con 0x7f1ad00b4af0 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 -- 172.21.15.46:0/637573336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00b4af0 msgr2=0x7f1ad00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/637573336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00b4f00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1ad008ed20 tx=0x7f1acc02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 -- 172.21.15.46:0/637573336 shutdown_connections 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/637573336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00b4f00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/637573336 >> v2:172.21.15.88:3300/0 conn(0x7f1ad00d57a0 0x7f1ad0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 -- 172.21.15.46:0/637573336 >> 172.21.15.46:0/637573336 conn(0x7f1ad00743e0 msgr2=0x7f1ad00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 -- 172.21.15.46:0/637573336 shutdown_connections 2024-05-09T03:12:16.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.421+0000 7f1ad80b1640 1 -- 172.21.15.46:0/637573336 wait complete. 2024-05-09T03:12:16.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.423+0000 7f1ad80b1640 1 Processor -- start 2024-05-09T03:12:16.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.423+0000 7f1ad80b1640 1 -- start start 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.424+0000 7f1ad80b1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.424+0000 7f1ad80b1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ad00b77e0 0x7f1ad0163820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.424+0000 7f1ad80b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 0x7f1ad0161790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.424+0000 7f1ad80b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad00d1920 con 0x7f1ad00d57a0 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.424+0000 7f1ad80b1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad00d15b0 con 0x7f1ad00b4af0 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad80b1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad00ca300 con 0x7f1ad00b77e0 2024-05-09T03:12:16.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 0x7f1ad0161790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 0x7f1ad0161790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43896/0 (socket says 172.21.15.46:43896) 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 -- 172.21.15.46:0/939760853 learned_addr learned my addr 172.21.15.46:0/939760853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 -- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ad00b77e0 msgr2=0x7f1ad0163820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad5e4d640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad564c640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ad00b77e0 0x7f1ad0163820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ad00b77e0 0x7f1ad0163820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 -- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad00b4af0 msgr2=0x7f1ad00c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 -- 172.21.15.46:0/939760853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad01f6370 con 0x7f1ad00d57a0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad664e640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 0x7f1ad0161790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800c6b0 tx=0x7f1ac800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8018070 con 0x7f1ad00d57a0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac8004da0 con 0x7f1ad00d57a0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800a880 con 0x7f1ad00d57a0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad0162300 con 0x7f1ad00d57a0 2024-05-09T03:12:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.425+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad0163020 con 0x7f1ad00d57a0 2024-05-09T03:12:16.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.426+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad006a270 con 0x7f1ad00d57a0 2024-05-09T03:12:16.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.427+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ac801b070 con 0x7f1ad00d57a0 2024-05-09T03:12:16.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.427+0000 7f1ac27fc640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ab40762d0 0x7f1ab40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:16.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.428+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f1ac8096c00 con 0x7f1ad00d57a0 2024-05-09T03:12:16.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.428+0000 7f1ad5e4d640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ab40762d0 0x7f1ab40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:16.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.428+0000 7f1ad5e4d640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ab40762d0 0x7f1ab40787e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1ad006d3b0 tx=0x7f1ac4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:16.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.432+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1ac809d080 con 0x7f1ad00d57a0 2024-05-09T03:12:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.662+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ad00d3350 con 0x7f1ad00d57a0 2024-05-09T03:12:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.663+0000 7f1ac27fc640 1 -- 172.21.15.46:0/939760853 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5181 (secure 0 0 0) 0x7f1ac8060560 con 0x7f1ad00d57a0 2024-05-09T03:12:16.665 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:16.665 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:09.251952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5874":{"gid":5874,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/4263404372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":4263404372},{"type":"v1","addr":"172.21.15.46:6835","nonce":4263404372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5874,"qdb_cluster":[5874]},"id":4}]} 2024-05-09T03:12:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 46 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ab40762d0 msgr2=0x7f1ab40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ab40762d0 0x7f1ab40787e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1ad006d3b0 tx=0x7f1ac4008040 comp rx=0 tx=0).stop 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 msgr2=0x7f1ad0161790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 0x7f1ad0161790 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800c6b0 tx=0x7f1ac800cb70 comp rx=0 tx=0).stop 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 shutdown_connections 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ab40762d0 0x7f1ab40787e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ad00d57a0 0x7f1ad0161790 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ad00b77e0 0x7f1ad0163820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 --2- 172.21.15.46:0/939760853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ad00b4af0 0x7f1ad00c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 >> 172.21.15.46:0/939760853 conn(0x7f1ad00743e0 msgr2=0x7f1ad00b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 shutdown_connections 2024-05-09T03:12:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:16.665+0000 7f1ad80b1640 1 -- 172.21.15.46:0/939760853 wait complete. 2024-05-09T03:12:17.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:09.251952+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5874': {'gid': 5874, 'name': 'a', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/4263404372', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 4263404372}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 4263404372}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5874, 'qdb_cluster': [5874]} 2024-05-09T03:12:17.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:12:17.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T03:12:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.196+0000 7f81dacd9640 1 Processor -- start 2024-05-09T03:12:17.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.196+0000 7f81dacd9640 1 -- start start 2024-05-09T03:12:17.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81dacd9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81d40b5140 0x7f81d40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81dacd9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f81d40b7ca0 0x7f81d40d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81dacd9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d40b80b0 con 0x7f81d40d3df0 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81dacd9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f81d40d3480 con 0x7f81d40b5140 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81dacd9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f81d40b5550 con 0x7f81d40b7ca0 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d8a75640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81d40b5140 0x7f81d40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d9276640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81d40d3df0 0x7f81d4163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56570/0 (socket says 172.21.15.46:56570) 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d9276640 1 -- 172.21.15.46:0/4200625844 learned_addr learned my addr 172.21.15.46:0/4200625844 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d3fff640 1 --2- 172.21.15.46:0/4200625844 >> v2:172.21.15.88:3300/0 conn(0x7f81d40b7ca0 0x7f81d40d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d8a75640 1 -- 172.21.15.46:0/4200625844 >> v2:172.21.15.88:3300/0 conn(0x7f81d40b7ca0 msgr2=0x7f81d40d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d8a75640 1 --2- 172.21.15.46:0/4200625844 >> v2:172.21.15.88:3300/0 conn(0x7f81d40b7ca0 0x7f81d40d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d8a75640 1 -- 172.21.15.46:0/4200625844 >> v1:172.21.15.46:6789/0 conn(0x7f81d40d3df0 legacy=0x7f81d4163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d8a75640 1 -- 172.21.15.46:0/4200625844 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d40d44e0 con 0x7f81d40b5140 2024-05-09T03:12:17.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d37fe640 1 -- 172.21.15.46:0/4200625844 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989825000 0 0) 0x7f81cc001490 con 0x7f81d40d3df0 2024-05-09T03:12:17.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.198+0000 7f81d3fff640 1 --2- 172.21.15.46:0/4200625844 >> v2:172.21.15.88:3300/0 conn(0x7f81d40b7ca0 0x7f81d40d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:17.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.199+0000 7f81d8a75640 1 --2- 172.21.15.46:0/4200625844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d40b7760 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f81d40d0290 tx=0x7f81c402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f671277d253e62 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.199+0000 7f81d37fe640 1 -- 172.21.15.46:0/4200625844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4037070 con 0x7f81d40b5140 2024-05-09T03:12:17.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.199+0000 7f81d37fe640 1 -- 172.21.15.46:0/4200625844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c4032440 con 0x7f81d40b5140 2024-05-09T03:12:17.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.199+0000 7f81d37fe640 1 -- 172.21.15.46:0/4200625844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4032810 con 0x7f81d40b5140 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 -- 172.21.15.46:0/4200625844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 msgr2=0x7f81d40b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 --2- 172.21.15.46:0/4200625844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d40b7760 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f81d40d0290 tx=0x7f81c402a230 comp rx=0 tx=0).stop 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 -- 172.21.15.46:0/4200625844 shutdown_connections 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 --2- 172.21.15.46:0/4200625844 >> v2:172.21.15.88:3300/0 conn(0x7f81d40b7ca0 0x7f81d40d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 --2- 172.21.15.46:0/4200625844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d40b7760 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 -- 172.21.15.46:0/4200625844 >> 172.21.15.46:0/4200625844 conn(0x7f81d408d3f0 msgr2=0x7f81d408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 -- 172.21.15.46:0/4200625844 shutdown_connections 2024-05-09T03:12:17.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.200+0000 7f81dacd9640 1 -- 172.21.15.46:0/4200625844 wait complete. 2024-05-09T03:12:17.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.202+0000 7f81dacd9640 1 Processor -- start 2024-05-09T03:12:17.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.202+0000 7f81dacd9640 1 -- start start 2024-05-09T03:12:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81dacd9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d41f7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81dacd9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81d40b7ca0 0x7f81d415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81dacd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81d40d3df0 0x7f81d40cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81dacd9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81d40d4d80 con 0x7f81d40b5140 2024-05-09T03:12:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81dacd9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81d40d4aa0 con 0x7f81d40d3df0 2024-05-09T03:12:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81dacd9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81d40d4200 con 0x7f81d40b7ca0 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81d8a75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d41f7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81d8a75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d41f7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43926/0 (socket says 172.21.15.46:43926) 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.203+0000 7f81d8a75640 1 -- 172.21.15.46:0/2119923130 learned_addr learned my addr 172.21.15.46:0/2119923130 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d9276640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81d40d3df0 0x7f81d40cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d3fff640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81d40b7ca0 0x7f81d415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d8a75640 1 -- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81d40b7ca0 msgr2=0x7f81d415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d8a75640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81d40b7ca0 0x7f81d415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d8a75640 1 -- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81d40d3df0 msgr2=0x7f81d40cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d8a75640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81d40d3df0 0x7f81d40cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d8a75640 1 -- 172.21.15.46:0/2119923130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d40cc910 con 0x7f81d40b5140 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d8a75640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d41f7300 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d17fa640 1 -- 172.21.15.46:0/2119923130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4037050 con 0x7f81d40b5140 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d17fa640 1 -- 172.21.15.46:0/2119923130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c4038040 con 0x7f81d40b5140 2024-05-09T03:12:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d40ccbf0 con 0x7f81d40b5140 2024-05-09T03:12:17.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81d17fa640 1 -- 172.21.15.46:0/2119923130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4004470 con 0x7f81d40b5140 2024-05-09T03:12:17.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.204+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d40cd1b0 con 0x7f81d40b5140 2024-05-09T03:12:17.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.206+0000 7f81d17fa640 1 -- 172.21.15.46:0/2119923130 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81c4030900 con 0x7f81d40b5140 2024-05-09T03:12:17.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.207+0000 7f81d17fa640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81c00762d0 0x7f81c00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.207+0000 7f81d17fa640 1 -- 172.21.15.46:0/2119923130 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f81c40b8100 con 0x7f81d40b5140 2024-05-09T03:12:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.207+0000 7f81d3fff640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81c00762d0 0x7f81c00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.207+0000 7f81e1b91740 1 Processor -- start 2024-05-09T03:12:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.207+0000 7f81e1b91740 1 -- start start 2024-05-09T03:12:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.207+0000 7f81d3fff640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81c00762d0 0x7f81c00787e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f81d40b74d0 tx=0x7f81c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a13eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5a2ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a2f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81e1b91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652a594c4e0 con 0x5652a5623590 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5652a58e7ff0 con 0x5652a561b3b0 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5652a59484a0 con 0x5652a5644270 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a2f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a2f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43940/0 (socket says 172.21.15.46:43940) 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 -- 172.21.15.46:0/2950429909 learned_addr learned my addr 172.21.15.46:0/2950429909 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d3fff640 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5a2ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d8a75640 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a13eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 -- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 msgr2=0x5652a5a13eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a13eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 -- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 msgr2=0x5652a5a2ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5a2ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 -- 172.21.15.46:0/2950429909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652a58ce720 con 0x5652a5623590 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81d9276640 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a2f120 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f81cc001c80 tx=0x7f81cc002140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cffc149e1d87571 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.209+0000 7f81ba7fc640 1 -- 172.21.15.46:0/2950429909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00d8b0 con 0x5652a5623590 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81ba7fc640 1 -- 172.21.15.46:0/2950429909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc00da50 con 0x5652a5623590 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81ba7fc640 1 -- 172.21.15.46:0/2950429909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc00de00 con 0x5652a5623590 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 -- 172.21.15.46:0/2950429909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 msgr2=0x5652a5a2f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a2f120 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f81cc001c80 tx=0x7f81cc002140 comp rx=0 tx=0).stop 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 -- 172.21.15.46:0/2950429909 shutdown_connections 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a2f120 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5a2ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 --2- 172.21.15.46:0/2950429909 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a13eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 -- 172.21.15.46:0/2950429909 >> 172.21.15.46:0/2950429909 conn(0x5652a56383c0 msgr2=0x5652a59e9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 -- 172.21.15.46:0/2950429909 shutdown_connections 2024-05-09T03:12:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.210+0000 7f81e1b91740 1 -- 172.21.15.46:0/2950429909 wait complete. 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.211+0000 7f81e1b91740 1 -- start start 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.212+0000 7f81e1b91740 1 Processor -- start 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a01260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a017a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a3e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652a58d3760 con 0x5652a5623590 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5652a5a2ecf0 con 0x5652a5644270 2024-05-09T03:12:17.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5652a5a168c0 con 0x5652a561b3b0 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81e1b91740 10 client.0 _ll_register_callbacks cb 0x5652a58f64b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a017a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a017a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43942/0 (socket says 172.21.15.46:43942) 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 -- 172.21.15.46:0/1438229354 learned_addr learned my addr 172.21.15.46:0/1438229354 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a01260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d9276640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a3e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 -- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 msgr2=0x5652a5a01260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a01260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 -- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 msgr2=0x5652a5a3e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a3e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d3fff640 1 -- 172.21.15.46:0/1438229354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652a5914bd0 con 0x5652a5623590 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.213+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a01260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81d3fff640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a017a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f81c8015d80 tx=0x7f81c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71e0b9dcea3feb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81b37fe640 1 -- 172.21.15.46:0/1438229354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8025020 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81b37fe640 1 -- 172.21.15.46:0/1438229354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f81c800ebf0 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81cc00da50 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81b37fe640 1 -- 172.21.15.46:0/1438229354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f81c802f050 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 10 client.6021 fetch_fsmap learned FSMap version 46 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5652a58fa280 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81b37fe640 1 -- 172.21.15.46:0/1438229354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc00da50 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81b37fe640 1 -- 172.21.15.46:0/1438229354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 46) ==== 3494+0+0 (secure 0 0 0) 0x7f81c8032020 con 0x5652a5623590 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 10 client.6021 fetch_fsmap finished waiting for FSMap version 46 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 10 client.6021 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 20 client.6021 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5602e40 0x5652a5a41af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 4 client.6021 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.214+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 --> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652a595a0f0 con 0x5652a5602e40 2024-05-09T03:12:17.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.215+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5602e40 0x5652a5a41af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.215+0000 7f81b37fe640 10 client.6021 ms_handle_connect on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.215+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5602e40 0x5652a5a41af0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7e8d587ae95cfaf server_cookie=e6e17afc1f203fce in_seq=0 out_seq=0 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81b37fe640 1 -- 172.21.15.46:0/1438229354 <== mds.0 v2:172.21.15.46:6834/4263404372 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5652a595a0f0 con 0x5652a5602e40 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81b37fe640 10 client.6021 handle_command_reply: tid=0 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 client.6021 shutdown 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 msgr2=0x5652a5a017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a017a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f81c8015d80 tx=0x7f81c80046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 shutdown_connections 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5602e40 0x5652a5a41af0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a3e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a017a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81b37fe640 0 client.6021 ms_handle_reset on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:17.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1438229354 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a01260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 >> 172.21.15.46:0/1438229354 conn(0x5652a56383c0 msgr2=0x5652a59e79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 shutdown_connections 2024-05-09T03:12:17.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 1 -- 172.21.15.46:0/1438229354 wait complete. 2024-05-09T03:12:17.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.216+0000 7f81e1b91740 20 client.6021 trim_cache size 0 max 16384 2024-05-09T03:12:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.235+0000 7f81e1b91740 1 Processor -- start 2024-05-09T03:12:17.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.235+0000 7f81e1b91740 1 -- start start 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a2c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a30760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a51ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81e1b91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81d4201ef0 con 0x5652a5623590 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81d4201280 con 0x5652a5644270 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81d42018a0 con 0x5652a561b3b0 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a30760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a30760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43952/0 (socket says 172.21.15.46:43952) 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 -- 172.21.15.46:0/4098072811 learned_addr learned my addr 172.21.15.46:0/4098072811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 -- 172.21.15.46:0/4098072811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 msgr2=0x5652a5a2c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a2c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 -- 172.21.15.46:0/4098072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 msgr2=0x5652a5a51ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a51ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 -- 172.21.15.46:0/4098072811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652a5a3e5c0 con 0x5652a5623590 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81d3fff640 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a30760 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f81c800e8b0 tx=0x7f81c8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b456b7d48146388 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81b9ffb640 1 -- 172.21.15.46:0/4098072811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8038070 con 0x5652a5623590 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81b9ffb640 1 -- 172.21.15.46:0/4098072811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c8021630 con 0x5652a5623590 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.237+0000 7f81b9ffb640 1 -- 172.21.15.46:0/4098072811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8025070 con 0x5652a5623590 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 -- 172.21.15.46:0/4098072811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 msgr2=0x5652a5a30760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a30760 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f81c800e8b0 tx=0x7f81c8021000 comp rx=0 tx=0).stop 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 -- 172.21.15.46:0/4098072811 shutdown_connections 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a5644270 0x5652a5a51ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a5a30760 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 --2- 172.21.15.46:0/4098072811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a561b3b0 0x5652a5a2c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 -- 172.21.15.46:0/4098072811 >> 172.21.15.46:0/4098072811 conn(0x5652a56383c0 msgr2=0x5652a5a28670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 -- 172.21.15.46:0/4098072811 shutdown_connections 2024-05-09T03:12:17.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.238+0000 7f81e1b91740 1 -- 172.21.15.46:0/4098072811 wait complete. 2024-05-09T03:12:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.239+0000 7f81e1b91740 1 -- start start 2024-05-09T03:12:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.239+0000 7f81e1b91740 1 Processor -- start 2024-05-09T03:12:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.240+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5aacc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.240+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a59e9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.240+0000 7f81e1b91740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a106e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.240+0000 7f81e1b91740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652a5a2dce0 con 0x5652a5623590 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.240+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5652a5a163a0 con 0x5652a561b3b0 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.240+0000 7f81e1b91740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5652a5a2e330 con 0x5652a5644270 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81e1b91740 10 client.0 _ll_register_callbacks cb 0x7f81c8015560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a59e9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a59e9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43960/0 (socket says 172.21.15.46:43960) 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 -- 172.21.15.46:0/1797842891 learned_addr learned my addr 172.21.15.46:0/1797842891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5aacc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d9276640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 -- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 msgr2=0x5652a5a106e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a106e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 -- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 msgr2=0x5652a5aacc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5aacc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 -- 172.21.15.46:0/1797842891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652a59e7560 con 0x5652a5623590 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5aacc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:17.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d9276640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81d3fff640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a59e9200 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f81c8009f70 tx=0x7f81c8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d490e3259635862 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81b37fe640 1 -- 172.21.15.46:0/1797842891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8042020 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81b37fe640 1 -- 172.21.15.46:0/1797842891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f81c800ebf0 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.241+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81c8021630 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81b37fe640 1 -- 172.21.15.46:0/1797842891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f81c804c050 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 10 client.6027 fetch_fsmap learned FSMap version 46 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5652a56af850 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81b37fe640 1 -- 172.21.15.46:0/1797842891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c8021630 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81b37fe640 1 -- 172.21.15.46:0/1797842891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 46) ==== 3494+0+0 (secure 0 0 0) 0x7f81c804f020 con 0x5652a5623590 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 10 client.6027 fetch_fsmap finished waiting for FSMap version 46 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 10 client.6027 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 20 client.6027 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5aafe20 0x5652a59ffb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 4 client.6027 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:12:17.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 --> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] -- command(tid 0: {"prefix": "status"}) -- 0x5652a5953f60 con 0x5652a5aafe20 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5aafe20 0x5652a59ffb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.242+0000 7f81b37fe640 10 client.6027 ms_handle_connect on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81d8a75640 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5aafe20 0x5652a59ffb30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ade28b9b30fb373 server_cookie=d9b97fe7dc1620e2 in_seq=0 out_seq=0 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81b37fe640 1 -- 172.21.15.46:0/1797842891 <== mds.0 v2:172.21.15.46:6834/4263404372 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5652a5953f60 con 0x5652a5aafe20 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81b37fe640 10 client.6027 handle_command_reply: tid=0 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 client.6027 shutdown 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 msgr2=0x5652a59e9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a59e9200 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f81c8009f70 tx=0x7f81c8015000 comp rx=0 tx=0).stop 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 shutdown_connections 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x5652a5aafe20 0x5652a59ffb30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652a5644270 0x5652a5a106e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652a5623590 0x5652a59e9200 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81b37fe640 0 client.6027 ms_handle_reset on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:17.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 --2- 172.21.15.46:0/1797842891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652a561b3b0 0x5652a5aacc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 >> 172.21.15.46:0/1797842891 conn(0x5652a56383c0 msgr2=0x5652a5a30760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 shutdown_connections 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 1 -- 172.21.15.46:0/1797842891 wait complete. 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.243+0000 7f81e1b91740 20 client.6027 trim_cache size 0 max 16384 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5874, 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:12:17.250 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:12:17.251 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.391892970000001, 2024-05-09T03:12:17.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 46, 2024-05-09T03:12:17.251 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 55, 2024-05-09T03:12:17.251 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 55, 2024-05-09T03:12:17.251 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15.343869012000001 2024-05-09T03:12:17.251 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81c00762d0 msgr2=0x7f81c00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81c00762d0 0x7f81c00787e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f81d40b74d0 tx=0x7f81c8008040 comp rx=0 tx=0).stop 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 msgr2=0x7f81d41f7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d41f7300 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f81c4000c00 tx=0x7f81c40046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 shutdown_connections 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81d40d3df0 0x7f81d40cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81d40b7ca0 0x7f81d415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81c00762d0 0x7f81c00787e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 --2- 172.21.15.46:0/2119923130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81d40b5140 0x7f81d41f7300 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 >> 172.21.15.46:0/2119923130 conn(0x7f81d408d3f0 msgr2=0x7f81d41601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 shutdown_connections 2024-05-09T03:12:17.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.244+0000 7f81dacd9640 1 -- 172.21.15.46:0/2119923130 wait complete. 2024-05-09T03:12:17.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:12:17.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.369+0000 7f826cca7640 1 Processor -- start 2024-05-09T03:12:17.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.369+0000 7f826cca7640 1 -- start start 2024-05-09T03:12:17.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826cca7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82681615f0 0x7f82680b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826cca7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82680d0b80 0x7f82680d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826cca7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82680d15b0 con 0x7f82680d25e0 2024-05-09T03:12:17.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826cca7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8268160b10 con 0x7f82681615f0 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826cca7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8268160df0 con 0x7f82680d0b80 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82681615f0 0x7f82680b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82681615f0 0x7f82680b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43970/0 (socket says 172.21.15.46:43970) 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 -- 172.21.15.46:0/1498168733 learned_addr learned my addr 172.21.15.46:0/1498168733 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f8265d9b640 1 --2- 172.21.15.46:0/1498168733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82680d0b80 0x7f82680d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 -- 172.21.15.46:0/1498168733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82680d0b80 msgr2=0x7f82680d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 --2- 172.21.15.46:0/1498168733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82680d0b80 0x7f82680d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 -- 172.21.15.46:0/1498168733 >> v1:172.21.15.88:6789/0 conn(0x7f82680d25e0 legacy=0x7f82680d29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:17.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.371+0000 7f826659c640 1 -- 172.21.15.46:0/1498168733 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82681610d0 con 0x7f82681615f0 2024-05-09T03:12:17.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.372+0000 7f826659c640 1 --2- 172.21.15.46:0/1498168733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82681615f0 0x7f82680b81b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8258009870 tx=0x7f825802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7030d8ff709f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.372+0000 7f826559a640 1 -- 172.21.15.46:0/1498168733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258037070 con 0x7f82681615f0 2024-05-09T03:12:17.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.372+0000 7f826559a640 1 -- 172.21.15.46:0/1498168733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8258032440 con 0x7f82681615f0 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.372+0000 7f826559a640 1 -- 172.21.15.46:0/1498168733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258032810 con 0x7f82681615f0 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 -- 172.21.15.46:0/1498168733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82681615f0 msgr2=0x7f82680b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 --2- 172.21.15.46:0/1498168733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82681615f0 0x7f82680b81b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8258009870 tx=0x7f825802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 -- 172.21.15.46:0/1498168733 shutdown_connections 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 --2- 172.21.15.46:0/1498168733 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82680d0b80 0x7f82680d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 --2- 172.21.15.46:0/1498168733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82681615f0 0x7f82680b81b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 -- 172.21.15.46:0/1498168733 >> 172.21.15.46:0/1498168733 conn(0x7f826806e190 msgr2=0x7f826806e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 -- 172.21.15.46:0/1498168733 shutdown_connections 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.373+0000 7f826cca7640 1 -- 172.21.15.46:0/1498168733 wait complete. 2024-05-09T03:12:17.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.375+0000 7f826cca7640 1 Processor -- start 2024-05-09T03:12:17.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.375+0000 7f826cca7640 1 -- start start 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.376+0000 7f826cca7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82680d0b80 0x7f82680c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.376+0000 7f826cca7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 0x7f826814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.376+0000 7f826cca7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82681615f0 0x7f8268162cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f826cca7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82681640c0 con 0x7f82680d25e0 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f826cca7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8268163f40 con 0x7f82680d0b80 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f826cca7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8268164240 con 0x7f82681615f0 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 0x7f826814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 0x7f826814e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43976/0 (socket says 172.21.15.46:43976) 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 -- 172.21.15.46:0/2006122842 learned_addr learned my addr 172.21.15.46:0/2006122842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f826659c640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82680d0b80 0x7f82680c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8266d9d640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82681615f0 0x7f8268162cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 -- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82681615f0 msgr2=0x7f8268162cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82681615f0 0x7f8268162cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 -- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82680d0b80 msgr2=0x7f82680c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82680d0b80 0x7f82680c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 -- 172.21.15.46:0/2006122842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8268151560 con 0x7f82680d25e0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8266d9d640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82681615f0 0x7f8268162cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8265d9b640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 0x7f826814e040 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f825c00d620 tx=0x7f825c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c00b620 con 0x7f82680d25e0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f825c019070 con 0x7f82680d25e0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826814f740 con 0x7f82680d25e0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c00ba00 con 0x7f82680d25e0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.377+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826814fce0 con 0x7f82680d25e0 2024-05-09T03:12:17.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.378+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f826806a270 con 0x7f82680d25e0 2024-05-09T03:12:17.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.379+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f825c00bba0 con 0x7f82680d25e0 2024-05-09T03:12:17.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.380+0000 7f8256ffd640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82480762d0 0x7f82480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:17.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.380+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f825c097ab0 con 0x7f82680d25e0 2024-05-09T03:12:17.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.380+0000 7f826659c640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82480762d0 0x7f82480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:17.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.380+0000 7f826659c640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82480762d0 0x7f82480787e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8258034000 tx=0x7f825802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:17.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.384+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f825c061370 con 0x7f82680d25e0 2024-05-09T03:12:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.613+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f826815c9d0 con 0x7f82680d25e0 2024-05-09T03:12:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.614+0000 7f8256ffd640 1 -- 172.21.15.46:0/2006122842 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) ==== 53+0+73 (secure 0 0 0) 0x7f825c067070 con 0x7f82680d25e0 2024-05-09T03:12:17.615 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82480762d0 msgr2=0x7f82480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82480762d0 0x7f82480787e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8258034000 tx=0x7f825802e000 comp rx=0 tx=0).stop 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 msgr2=0x7f826814e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 0x7f826814e040 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f825c00d620 tx=0x7f825c00dae0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 shutdown_connections 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82681615f0 0x7f8268162cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82480762d0 0x7f82480787e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82680d25e0 0x7f826814e040 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 --2- 172.21.15.46:0/2006122842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82680d0b80 0x7f82680c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 >> 172.21.15.46:0/2006122842 conn(0x7f826806e190 msgr2=0x7f826815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 shutdown_connections 2024-05-09T03:12:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17.616+0000 7f826cca7640 1 -- 172.21.15.46:0/2006122842 wait complete. 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:12:17.629 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:12:17.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:17.630 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:12:17.646 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:12:17.646 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:12:17.646 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:12:17.646 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.646 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:12:17.646 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42380sec preferred_lft 42380sec 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:12:17.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:12:17.648 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:17.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:17.649 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:12:17.710 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:12:17.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:17.712 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:12:17.769 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:12:17.769 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:12:17.769 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:12:17.769 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:12:17.769 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:12:17.769 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:12:17.770 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:12:17.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:12:17.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:12:17.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:12:17.770 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:12:17.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:12:17.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:12:17.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:17.899 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:12:17.915 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:12:17.915 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:12:17.915 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:12:17.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:17.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:12:17.981 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:12:17.981 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:12:17.981 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:18.037 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:12:18.037 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:12:18.038 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:18.093 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:12:18.159 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:12:18.216 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:12:18.216 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:12:18.216 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:12:18.217 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:12:18.218 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:12:18.219 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:18.219 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:18.219 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:18.219 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:18.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:18.219 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:12:18.272 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:12:18.272 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:12:18.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:12:18.340 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:12:18.400 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:12:18.401 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:18.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:18.402 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:12:18.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:12:18.409+0000 7fcc9fc2f480 -1 init, newargv = 0x7fcc84056020 newargc=15 2024-05-09T03:12:18.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54391]: starting ceph client 2024-05-09T03:12:18.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54391]: starting fuse 2024-05-09T03:12:18.427 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:12:18.427 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:12:18.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:12:18.427 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:12:18.427 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:12:18.427 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:12:18.427 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:12:18.428 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:12:18.429 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:12:18.429 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:12:18.429 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:12:18.429 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:18.429 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:12:18.429 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:12:18.549 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.54391.asok 2024-05-09T03:12:18.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:12:18.560 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54391.asok 2024-05-09T03:12:18.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:18.560 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54391.asok status 2024-05-09T03:12:18.708 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "54391", 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6039, 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:12:18.709 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6039 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3880330589 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3880330589 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6039 192.168.0.1:0/3880330589", 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3880330589", 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 46, 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 55, 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:12:18.711 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:18.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:18.717 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:12:18.718 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:18.734 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:12:18.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:18.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:18.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:18.801 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/{b,c}/d; sync .') 2024-05-09T03:12:18.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:12:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:12:18.915 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:12:18.916 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:12:18.932 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:12:19.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.087+0000 7f2551157640 1 Processor -- start 2024-05-09T03:12:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.087+0000 7f2551157640 1 -- start start 2024-05-09T03:12:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f2551157640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f254c0d0020 0x7f254c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f2551157640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f254c0b3800 0x7f254c0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f2551157640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254c0b92c0 con 0x7f254c0d1110 2024-05-09T03:12:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f2551157640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f254c0d0430 con 0x7f254c0b3800 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f2551157640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f254c0b2f10 con 0x7f254c0d0020 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f254b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f254c0b3800 0x7f254c0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f254b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f254c0b3800 0x7f254c0b3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43980/0 (socket says 172.21.15.46:43980) 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f254b59e640 1 -- 172.21.15.46:0/1606517654 learned_addr learned my addr 172.21.15.46:0/1606517654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.088+0000 7f254a59c640 1 --2- 172.21.15.46:0/1606517654 >> v2:172.21.15.88:3300/0 conn(0x7f254c0d0020 0x7f254c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.089+0000 7f254b59e640 1 -- 172.21.15.46:0/1606517654 >> v2:172.21.15.88:3300/0 conn(0x7f254c0d0020 msgr2=0x7f254c0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.089+0000 7f254b59e640 1 --2- 172.21.15.46:0/1606517654 >> v2:172.21.15.88:3300/0 conn(0x7f254c0d0020 0x7f254c0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.089+0000 7f254b59e640 1 -- 172.21.15.46:0/1606517654 >> v1:172.21.15.88:6789/0 conn(0x7f254c0d1110 legacy=0x7f254c0d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.089+0000 7f254b59e640 1 -- 172.21.15.46:0/1606517654 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c0b9560 con 0x7f254c0b3800 2024-05-09T03:12:19.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.089+0000 7f254b59e640 1 --2- 172.21.15.46:0/1606517654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c0b3c10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f254c08cef0 tx=0x7f253c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcebe93094b925 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.089+0000 7f2549d9b640 1 -- 172.21.15.46:0/1606517654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c037070 con 0x7f254c0b3800 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2549d9b640 1 -- 172.21.15.46:0/1606517654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253c032440 con 0x7f254c0b3800 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2549d9b640 1 -- 172.21.15.46:0/1606517654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c032810 con 0x7f254c0b3800 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 -- 172.21.15.46:0/1606517654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 msgr2=0x7f254c0b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 --2- 172.21.15.46:0/1606517654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c0b3c10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f254c08cef0 tx=0x7f253c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 -- 172.21.15.46:0/1606517654 shutdown_connections 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 --2- 172.21.15.46:0/1606517654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c0b3c10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 --2- 172.21.15.46:0/1606517654 >> v2:172.21.15.88:3300/0 conn(0x7f254c0d0020 0x7f254c0b32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 -- 172.21.15.46:0/1606517654 >> 172.21.15.46:0/1606517654 conn(0x7f254c0d05f0 msgr2=0x7f254c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 -- 172.21.15.46:0/1606517654 shutdown_connections 2024-05-09T03:12:19.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.090+0000 7f2551157640 1 -- 172.21.15.46:0/1606517654 wait complete. 2024-05-09T03:12:19.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.092+0000 7f2551157640 1 Processor -- start 2024-05-09T03:12:19.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.092+0000 7f2551157640 1 -- start start 2024-05-09T03:12:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f2551157640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c1501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f2551157640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254c0d0020 0x7f254c14faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f2551157640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f254c1e9540 0x7f254c15f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f2551157640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f254c09ceb0 con 0x7f254c0b3800 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f2551157640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f254c0d2030 con 0x7f254c0d0020 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f2551157640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f254c09d030 con 0x7f254c1e9540 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c1501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c1501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43996/0 (socket says 172.21.15.46:43996) 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 -- 172.21.15.46:0/2782844487 learned_addr learned my addr 172.21.15.46:0/2782844487 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254a59c640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254c0d0020 0x7f254c14faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254b59e640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f254c1e9540 0x7f254c15f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 -- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f254c1e9540 msgr2=0x7f254c15f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f254c1e9540 0x7f254c15f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 -- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254c0d0020 msgr2=0x7f254c14faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254c0d0020 0x7f254c14faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 -- 172.21.15.46:0/2782844487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c1e87e0 con 0x7f254c0b3800 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254b59e640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f254c1e9540 0x7f254c15f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254a59c640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254c0d0020 0x7f254c14faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.094+0000 7f254ad9d640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c1501e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f254400c690 tx=0x7f254400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.095+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544016020 con 0x7f254c0b3800 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.095+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2544006d70 con 0x7f254c0b3800 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.095+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254c161fb0 con 0x7f254c0b3800 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.095+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544003bb0 con 0x7f254c0b3800 2024-05-09T03:12:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.095+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254c1e8d40 con 0x7f254c0b3800 2024-05-09T03:12:19.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.095+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c06a270 con 0x7f254c0b3800 2024-05-09T03:12:19.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.096+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25440050d0 con 0x7f254c0b3800 2024-05-09T03:12:19.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.097+0000 7f25437fe640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f252c0762d0 0x7f252c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.097+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7f25440972c0 con 0x7f254c0b3800 2024-05-09T03:12:19.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.097+0000 7f254a59c640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f252c0762d0 0x7f252c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.098+0000 7f254a59c640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f252c0762d0 0x7f252c0787e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f254c1f74f0 tx=0x7f2534008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.101+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2544020020 con 0x7f254c0b3800 2024-05-09T03:12:19.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.308+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f254c0b5050 con 0x7f254c0b3800 2024-05-09T03:12:19.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.308+0000 7f25437fe640 1 -- 172.21.15.46:0/2782844487 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7f254c1e87e0 con 0x7f254c0b3800 2024-05-09T03:12:19.310 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:12:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f252c0762d0 msgr2=0x7f252c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f252c0762d0 0x7f252c0787e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f254c1f74f0 tx=0x7f2534008040 comp rx=0 tx=0).stop 2024-05-09T03:12:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 msgr2=0x7f254c1501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c1501e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f254400c690 tx=0x7f254400cb50 comp rx=0 tx=0).stop 2024-05-09T03:12:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 shutdown_connections 2024-05-09T03:12:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f254c1e9540 0x7f254c15f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254c0d0020 0x7f254c14faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f252c0762d0 0x7f252c0787e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 --2- 172.21.15.46:0/2782844487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f254c0b3800 0x7f254c1501e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 >> 172.21.15.46:0/2782844487 conn(0x7f254c0d05f0 msgr2=0x7f254c0b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 shutdown_connections 2024-05-09T03:12:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.310+0000 7f2551157640 1 -- 172.21.15.46:0/2782844487 wait complete. 2024-05-09T03:12:19.323 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:12:19.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:19.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:19.347 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54391]: fuse finished with error 0 and tester_r 0 2024-05-09T03:12:19.349 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:12:19.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:19.349 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:12:19.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-05-09T03:12:19.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.599+0000 7fc16c861640 1 Processor -- start 2024-05-09T03:12:19.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.599+0000 7fc16c861640 1 -- start start 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.600+0000 7fc16c861640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1680bb770 0x7fc1680d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.600+0000 7fc16c861640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1680d3180 0x7fc1681642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.600+0000 7fc16c861640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc168164900 con 0x7fc168163eb0 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.600+0000 7fc16c861640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc1680d4bb0 con 0x7fc1680bb770 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16c861640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc168164be0 con 0x7fc1680d3180 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1680bb770 0x7fc1680d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1680bb770 0x7fc1680d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44002/0 (socket says 172.21.15.46:44002) 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 -- 172.21.15.46:0/3415980418 learned_addr learned my addr 172.21.15.46:0/3415980418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc166d9d640 1 --2- 172.21.15.46:0/3415980418 >> v2:172.21.15.88:3300/0 conn(0x7fc1680d3180 0x7fc1681642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 -- 172.21.15.46:0/3415980418 >> v2:172.21.15.88:3300/0 conn(0x7fc1680d3180 msgr2=0x7fc1681642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 --2- 172.21.15.46:0/3415980418 >> v2:172.21.15.88:3300/0 conn(0x7fc1680d3180 0x7fc1681642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 -- 172.21.15.46:0/3415980418 >> v1:172.21.15.88:6789/0 conn(0x7fc168163eb0 legacy=0x7fc1680d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc16659c640 1 -- 172.21.15.46:0/3415980418 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1680d4840 con 0x7fc1680bb770 2024-05-09T03:12:19.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.601+0000 7fc166d9d640 1 --2- 172.21.15.46:0/3415980418 >> v2:172.21.15.88:3300/0 conn(0x7fc1680d3180 0x7fc1681642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:19.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.602+0000 7fc16659c640 1 --2- 172.21.15.46:0/3415980418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680d5040 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc16808e340 tx=0x7fc15802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4226ed418aaa63e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.602+0000 7fc165d9b640 1 -- 172.21.15.46:0/3415980418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158037070 con 0x7fc1680bb770 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.602+0000 7fc165d9b640 1 -- 172.21.15.46:0/3415980418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc158032440 con 0x7fc1680bb770 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.602+0000 7fc165d9b640 1 -- 172.21.15.46:0/3415980418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158032810 con 0x7fc1680bb770 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 -- 172.21.15.46:0/3415980418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 msgr2=0x7fc1680d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 --2- 172.21.15.46:0/3415980418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680d5040 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fc16808e340 tx=0x7fc15802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 -- 172.21.15.46:0/3415980418 shutdown_connections 2024-05-09T03:12:19.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 --2- 172.21.15.46:0/3415980418 >> v2:172.21.15.88:3300/0 conn(0x7fc1680d3180 0x7fc1681642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 --2- 172.21.15.46:0/3415980418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680d5040 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 -- 172.21.15.46:0/3415980418 >> 172.21.15.46:0/3415980418 conn(0x7fc16806e370 msgr2=0x7fc16806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 -- 172.21.15.46:0/3415980418 shutdown_connections 2024-05-09T03:12:19.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.603+0000 7fc16c861640 1 -- 172.21.15.46:0/3415980418 wait complete. 2024-05-09T03:12:19.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.605+0000 7fc16c861640 1 Processor -- start 2024-05-09T03:12:19.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.605+0000 7fc16c861640 1 -- start start 2024-05-09T03:12:19.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16c861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16c861640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168163eb0 0x7fc1680cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16c861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680cee10 0x7fc1680cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16c861640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc1680d0bc0 con 0x7fc1680bb770 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16c861640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1680bbb80 con 0x7fc168163eb0 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16c861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1680d0d40 con 0x7fc1680cee10 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44016/0 (socket says 172.21.15.46:44016) 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 -- 172.21.15.46:0/2288309251 learned_addr learned my addr 172.21.15.46:0/2288309251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168163eb0 0x7fc1680cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc166d9d640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680cee10 0x7fc1680cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 -- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680cee10 msgr2=0x7fc1680cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680cee10 0x7fc1680cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 -- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168163eb0 msgr2=0x7fc1680cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168163eb0 0x7fc1680cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc16659c640 1 -- 172.21.15.46:0/2288309251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1680cf760 con 0x7fc1680bb770 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc166d9d640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680cee10 0x7fc1680cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.606+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168163eb0 0x7fc1680cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.607+0000 7fc16659c640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680b42e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc158000c00 tx=0x7fc1580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.607+0000 7fc15effd640 1 -- 172.21.15.46:0/2288309251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158037040 con 0x7fc1680bb770 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.607+0000 7fc15effd640 1 -- 172.21.15.46:0/2288309251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc158042070 con 0x7fc1680bb770 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.607+0000 7fc15effd640 1 -- 172.21.15.46:0/2288309251 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc158030b90 con 0x7fc1680bb770 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.607+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1680b5fc0 con 0x7fc1680bb770 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.607+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1680cdbf0 con 0x7fc1680bb770 2024-05-09T03:12:19.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.609+0000 7fc15effd640 1 -- 172.21.15.46:0/2288309251 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc158047460 con 0x7fc1680bb770 2024-05-09T03:12:19.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.610+0000 7fc15effd640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1480762d0 0x7fc1480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.610+0000 7fc15effd640 1 -- 172.21.15.46:0/2288309251 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fc1580b8250 con 0x7fc1680bb770 2024-05-09T03:12:19.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.610+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1480762d0 0x7fc1480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.610+0000 7fc1737ef740 1 Processor -- start 2024-05-09T03:12:19.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.610+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1480762d0 0x7fc1480787e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc168161de0 tx=0x7fc150008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.611+0000 7fc1737ef740 1 -- start start 2024-05-09T03:12:19.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x562423420510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x56242343b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc1737ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5624233591a0 con 0x5624230306b0 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562423312170 con 0x5624230284d0 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562423355160 con 0x562423051390 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44030/0 (socket says 172.21.15.46:44030) 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 -- 172.21.15.46:0/2424812877 learned_addr learned my addr 172.21.15.46:0/2424812877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc16659c640 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x562423420510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x56242343b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 -- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 msgr2=0x562423420510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x562423420510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 -- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 msgr2=0x56242343b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x56242343b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.612+0000 7fc166d9d640 1 -- 172.21.15.46:0/2424812877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562423324390 con 0x5624230306b0 2024-05-09T03:12:19.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.613+0000 7fc166d9d640 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343b780 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc160007fb0 tx=0x7fc16000d800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c04c724eb674b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.613+0000 7fc147fff640 1 -- 172.21.15.46:0/2424812877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16000f920 con 0x5624230306b0 2024-05-09T03:12:19.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.613+0000 7fc147fff640 1 -- 172.21.15.46:0/2424812877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16000fac0 con 0x5624230306b0 2024-05-09T03:12:19.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.613+0000 7fc147fff640 1 -- 172.21.15.46:0/2424812877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc160018480 con 0x5624230306b0 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.613+0000 7fc1737ef740 1 -- 172.21.15.46:0/2424812877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 msgr2=0x56242343b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.613+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343b780 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fc160007fb0 tx=0x7fc16000d800 comp rx=0 tx=0).stop 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 -- 172.21.15.46:0/2424812877 shutdown_connections 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343b780 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x56242343b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2424812877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x562423420510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 -- 172.21.15.46:0/2424812877 >> 172.21.15.46:0/2424812877 conn(0x5624230454e0 msgr2=0x5624233f5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 -- 172.21.15.46:0/2424812877 shutdown_connections 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 -- 172.21.15.46:0/2424812877 wait complete. 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.614+0000 7fc1737ef740 1 -- start start 2024-05-09T03:12:19.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.615+0000 7fc1737ef740 1 Processor -- start 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x562423423470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x56242343f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5624232df8c0 con 0x5624230306b0 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562423422410 con 0x5624230284d0 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562423423b20 con 0x562423051390 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc1737ef740 10 client.0 _ll_register_callbacks cb 0x5624232fb860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc15ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc15ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44036/0 (socket says 172.21.15.46:44036) 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc15ffff640 1 -- 172.21.15.46:0/1930616197 learned_addr learned my addr 172.21.15.46:0/1930616197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc166d9d640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x56242343f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.616+0000 7fc16659c640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x562423423470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc15ffff640 1 -- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 msgr2=0x56242343f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc15ffff640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x56242343f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc15ffff640 1 -- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 msgr2=0x562423423470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc15ffff640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x562423423470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc15ffff640 1 -- 172.21.15.46:0/1930616197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624231337d0 con 0x5624230306b0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc15ffff640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343f1a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc15000cef0 tx=0x7fc1500046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c898b5f42149771 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1457fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1457fa640 1 -- 172.21.15.46:0/1930616197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150023030 con 0x5624230306b0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1457fa640 1 -- 172.21.15.46:0/1930616197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fc15001c040 con 0x5624230306b0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc16000fac0 con 0x5624230306b0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1457fa640 1 -- 172.21.15.46:0/1930616197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc150004020 con 0x5624230306b0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 10 client.6057 fetch_fsmap learned FSMap version 46 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x5624230bc970 con 0x5624230306b0 2024-05-09T03:12:19.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1457fa640 1 -- 172.21.15.46:0/1930616197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16000fac0 con 0x5624230306b0 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1457fa640 1 -- 172.21.15.46:0/1930616197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 46) ==== 3494+0+0 (secure 0 0 0) 0x7fc1500041c0 con 0x5624230306b0 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 10 client.6057 fetch_fsmap finished waiting for FSMap version 46 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 10 client.6057 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 20 client.6057 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.617+0000 7fc1737ef740 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x56242300ff60 0x562423421420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.618+0000 7fc1737ef740 4 client.6057 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.618+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 --> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562423366db0 con 0x56242300ff60 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.618+0000 7fc16659c640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x56242300ff60 0x562423421420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.618+0000 7fc1457fa640 10 client.6057 ms_handle_connect on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.618+0000 7fc16659c640 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x56242300ff60 0x562423421420 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c96e0a3d89129f9 server_cookie=6b15fff73659187c in_seq=0 out_seq=0 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1457fa640 1 -- 172.21.15.46:0/1930616197 <== mds.0 v2:172.21.15.46:6834/4263404372 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562423366db0 con 0x56242300ff60 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1457fa640 10 client.6057 handle_command_reply: tid=0 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 client.6057 shutdown 2024-05-09T03:12:19.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 msgr2=0x56242343f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343f1a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc15000cef0 tx=0x7fc1500046f0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 shutdown_connections 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x56242300ff60 0x562423421420 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562423051390 0x56242343f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230306b0 0x56242343f1a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1457fa640 0 client.6057 ms_handle_reset on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 --2- 172.21.15.46:0/1930616197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230284d0 0x562423423470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 >> 172.21.15.46:0/1930616197 conn(0x5624230454e0 msgr2=0x5624233f5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 shutdown_connections 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 1 -- 172.21.15.46:0/1930616197 wait complete. 2024-05-09T03:12:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.619+0000 7fc1737ef740 20 client.6057 trim_cache size 0 max 16384 2024-05-09T03:12:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.635+0000 7fc1737ef740 1 Processor -- start 2024-05-09T03:12:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.635+0000 7fc1737ef740 1 -- start start 2024-05-09T03:12:19.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56242300ff60 0x56242343b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 0x562423420510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230306b0 0x562423420a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc1737ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc168202e10 con 0x5624230284d0 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc168201e80 con 0x56242300ff60 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1681f7260 con 0x5624230306b0 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 0x562423420510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 0x562423420510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44042/0 (socket says 172.21.15.46:44042) 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 -- 172.21.15.46:0/2175576002 learned_addr learned my addr 172.21.15.46:0/2175576002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc16659c640 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56242300ff60 0x56242343b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc166d9d640 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230306b0 0x562423420a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 -- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230306b0 msgr2=0x562423420a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230306b0 0x562423420a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 -- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56242300ff60 msgr2=0x56242343b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56242300ff60 0x56242343b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.636+0000 7fc15ffff640 1 -- 172.21.15.46:0/2175576002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562423001f60 con 0x5624230284d0 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.637+0000 7fc15ffff640 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 0x562423420510 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc15000e010 tx=0x7fc15001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917ac46d600330ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.637+0000 7fc1477fe640 1 -- 172.21.15.46:0/2175576002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150035070 con 0x5624230284d0 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.637+0000 7fc1477fe640 1 -- 172.21.15.46:0/2175576002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1500041c0 con 0x5624230284d0 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.637+0000 7fc1477fe640 1 -- 172.21.15.46:0/2175576002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150023040 con 0x5624230284d0 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 -- 172.21.15.46:0/2175576002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 msgr2=0x562423420510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 0x562423420510 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc15000e010 tx=0x7fc15001f000 comp rx=0 tx=0).stop 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 -- 172.21.15.46:0/2175576002 shutdown_connections 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230306b0 0x562423420a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624230284d0 0x562423420510 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 --2- 172.21.15.46:0/2175576002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56242300ff60 0x56242343b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 -- 172.21.15.46:0/2175576002 >> 172.21.15.46:0/2175576002 conn(0x5624230454e0 msgr2=0x562423434cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 -- 172.21.15.46:0/2175576002 shutdown_connections 2024-05-09T03:12:19.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 -- 172.21.15.46:0/2175576002 wait complete. 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.638+0000 7fc1737ef740 1 -- start start 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.639+0000 7fc1737ef740 1 Processor -- start 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 0x5624233ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230284d0 0x562423436e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230306b0 0x56242341ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562423437db0 con 0x56242300ff60 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1681f78b0 con 0x5624230306b0 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562423420c60 con 0x5624230284d0 2024-05-09T03:12:19.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc1737ef740 10 client.0 _ll_register_callbacks cb 0x7fc15000a540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 0x5624233ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 0x5624233ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44058/0 (socket says 172.21.15.46:44058) 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 -- 172.21.15.46:0/3043758123 learned_addr learned my addr 172.21.15.46:0/3043758123 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc166d9d640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230306b0 0x56242341ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc15ffff640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230284d0 0x562423436e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 -- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230284d0 msgr2=0x562423436e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230284d0 0x562423436e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 -- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230306b0 msgr2=0x56242341ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230306b0 0x56242341ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.640+0000 7fc16659c640 1 -- 172.21.15.46:0/3043758123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624233efce0 con 0x56242300ff60 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc16659c640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 0x5624233ef7a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc15802a7d0 tx=0x7fc158049040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4b118c63af4aeb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1457fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1457fa640 1 -- 172.21.15.46:0/3043758123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15808a260 con 0x56242300ff60 2024-05-09T03:12:19.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1457fa640 1 -- 172.21.15.46:0/3043758123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fc1580816f0 con 0x56242300ff60 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1500041c0 con 0x56242300ff60 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1457fa640 1 -- 172.21.15.46:0/3043758123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc1580818f0 con 0x56242300ff60 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 10 client.6063 fetch_fsmap learned FSMap version 46 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) -- 0x562423410040 con 0x56242300ff60 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1457fa640 1 -- 172.21.15.46:0/3043758123 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1500041c0 con 0x56242300ff60 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1457fa640 1 -- 172.21.15.46:0/3043758123 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 46) ==== 3494+0+0 (secure 0 0 0) 0x7fc15803a030 con 0x56242300ff60 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 10 client.6063 fetch_fsmap finished waiting for FSMap version 46 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 10 client.6063 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 20 client.6063 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x562422fe53a0 0x56242345d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 4 client.6063 mds_command: new command op to 5874 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 --> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624234381f0 con 0x562422fe53a0 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.641+0000 7fc15ffff640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x562422fe53a0 0x56242345d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.642+0000 7fc1457fa640 10 client.6063 ms_handle_connect on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:19.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.642+0000 7fc15ffff640 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x562422fe53a0 0x56242345d6c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b257a81069a0c1 server_cookie=e9e64680f2a82659 in_seq=0 out_seq=0 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1457fa640 1 -- 172.21.15.46:0/3043758123 <== mds.0 v2:172.21.15.46:6834/4263404372 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x5624234381f0 con 0x562422fe53a0 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1457fa640 10 client.6063 handle_command_reply: tid=0 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1737ef740 1 client.6063 shutdown 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 msgr2=0x5624233ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1737ef740 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 0x5624233ef7a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc15802a7d0 tx=0x7fc158049040 comp rx=0 tx=0).stop 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 shutdown_connections 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.661+0000 7fc1737ef740 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:6834/4263404372,v1:172.21.15.46:6835/4263404372] conn(0x562422fe53a0 0x56242345d6c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624230306b0 0x56242341ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5624230284d0 0x562423436e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1457fa640 0 client.6063 ms_handle_reset on v2:172.21.15.46:6834/4263404372 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 1 --2- 172.21.15.46:0/3043758123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56242300ff60 0x5624233ef7a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 >> 172.21.15.46:0/3043758123 conn(0x5624230454e0 msgr2=0x562423434cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 shutdown_connections 2024-05-09T03:12:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 1 -- 172.21.15.46:0/3043758123 wait complete. 2024-05-09T03:12:19.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc1737ef740 20 client.6063 trim_cache size 0 max 16384 2024-05-09T03:12:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:19.665 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0 2024-05-09T03:12:19.665 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1480762d0 msgr2=0x7fc1480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1480762d0 0x7fc1480787e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc168161de0 tx=0x7fc150008040 comp rx=0 tx=0).stop 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 msgr2=0x7fc1680b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680b42e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc158000c00 tx=0x7fc1580046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 shutdown_connections 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1680cee10 0x7fc1680cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc168163eb0 0x7fc1680cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1480762d0 0x7fc1480787e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 --2- 172.21.15.46:0/2288309251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1680bb770 0x7fc1680b42e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.662+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 >> 172.21.15.46:0/2288309251 conn(0x7fc16806e370 msgr2=0x7fc1680b9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.663+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 shutdown_connections 2024-05-09T03:12:19.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.663+0000 7fc16c861640 1 -- 172.21.15.46:0/2288309251 wait complete. 2024-05-09T03:12:19.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:12:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.788+0000 7faaf4c15640 1 Processor -- start 2024-05-09T03:12:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.788+0000 7faaf4c15640 1 -- start start 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaf4c15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaf01613c0 0x7faaf00b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaf4c15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b7d00 0x7faaf0163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaf4c15640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaf00b8110 con 0x7faaf00b5210 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaf4c15640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faaf00b5620 con 0x7faaf01613c0 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaf4c15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faaf01609a0 con 0x7faaf00b7d00 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faae5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faaf01613c0 0x7faaf00b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaee59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faaf00b5210 0x7faaf00d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56586/0 (socket says 172.21.15.46:56586) 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.789+0000 7faaee59c640 1 -- 172.21.15.46:0/1606038476 learned_addr learned my addr 172.21.15.46:0/1606038476 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.790+0000 7faaeed9d640 1 --2- 172.21.15.46:0/1606038476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b7d00 0x7faaf0163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.790+0000 7faae5d9b640 1 -- 172.21.15.46:0/1606038476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b7d00 msgr2=0x7faaf0163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.790+0000 7faae5d9b640 1 --2- 172.21.15.46:0/1606038476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b7d00 0x7faaf0163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.790+0000 7faae5d9b640 1 -- 172.21.15.46:0/1606038476 >> v1:172.21.15.46:6789/0 conn(0x7faaf00b5210 legacy=0x7faaf00d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.790+0000 7faaedd9b640 1 -- 172.21.15.46:0/1606038476 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4271697438 0 0) 0x7faae0001110 con 0x7faaf00b5210 2024-05-09T03:12:19.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.790+0000 7faae5d9b640 1 -- 172.21.15.46:0/1606038476 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf01617d0 con 0x7faaf01613c0 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faae5d9b640 1 --2- 172.21.15.46:0/1606038476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00b77c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7faad80097a0 tx=0x7faad802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee101d570fe2ce9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaedd9b640 1 -- 172.21.15.46:0/1606038476 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8037070 con 0x7faaf01613c0 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaedd9b640 1 -- 172.21.15.46:0/1606038476 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad8032440 con 0x7faaf01613c0 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaedd9b640 1 -- 172.21.15.46:0/1606038476 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8032810 con 0x7faaf01613c0 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 -- 172.21.15.46:0/1606038476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 msgr2=0x7faaf00b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 --2- 172.21.15.46:0/1606038476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00b77c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7faad80097a0 tx=0x7faad802a180 comp rx=0 tx=0).stop 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 -- 172.21.15.46:0/1606038476 shutdown_connections 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 --2- 172.21.15.46:0/1606038476 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b7d00 0x7faaf0163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 --2- 172.21.15.46:0/1606038476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00b77c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 -- 172.21.15.46:0/1606038476 >> 172.21.15.46:0/1606038476 conn(0x7faaf00d29a0 msgr2=0x7faaf00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.791+0000 7faaf4c15640 1 -- 172.21.15.46:0/1606038476 shutdown_connections 2024-05-09T03:12:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.792+0000 7faaf4c15640 1 -- 172.21.15.46:0/1606038476 wait complete. 2024-05-09T03:12:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.794+0000 7faaf4c15640 1 Processor -- start 2024-05-09T03:12:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.794+0000 7faaf4c15640 1 -- start start 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaf4c15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b5210 0x7faaf01fadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaf4c15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf00b7d00 0x7faaf014f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaf4c15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaf4c15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faaf0160bc0 con 0x7faaf01613c0 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaf4c15640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faaf00d5560 con 0x7faaf00b7d00 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaf4c15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faaf0160ea0 con 0x7faaf00b5210 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44086/0 (socket says 172.21.15.46:44086) 2024-05-09T03:12:19.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 -- 172.21.15.46:0/3922084429 learned_addr learned my addr 172.21.15.46:0/3922084429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaee59c640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b5210 0x7faaf01fadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faae5d9b640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf00b7d00 0x7faaf014f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 -- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b5210 msgr2=0x7faaf01fadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b5210 0x7faaf01fadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 -- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf00b7d00 msgr2=0x7faaf014f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf00b7d00 0x7faaf014f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.795+0000 7faaeed9d640 1 -- 172.21.15.46:0/3922084429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf01fb2f0 con 0x7faaf01613c0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.796+0000 7faaeed9d640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00cb880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7faae8007d80 tx=0x7faae800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.796+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8014420 con 0x7faaf01613c0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.796+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae8019070 con 0x7faaf01613c0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.796+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf014f9d0 con 0x7faaf01613c0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.796+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8014800 con 0x7faaf01613c0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.796+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf015cdb0 con 0x7faaf01613c0 2024-05-09T03:12:19.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.797+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf00c9cf0 con 0x7faaf01613c0 2024-05-09T03:12:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.798+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faae80149a0 con 0x7faaf01613c0 2024-05-09T03:12:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.798+0000 7faae77fe640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad0076300 0x7faad0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.798+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7faae8097c70 con 0x7faaf01613c0 2024-05-09T03:12:19.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.798+0000 7faaee59c640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad0076300 0x7faad0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:19.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.799+0000 7faaee59c640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad0076300 0x7faad0078810 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faae0007fd0 tx=0x7faae0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:19.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:19.803+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faaf01fb2f0 con 0x7faaf01613c0 2024-05-09T03:12:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.034+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faaf006a270 con 0x7faaf01613c0 2024-05-09T03:12:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.034+0000 7faae77fe640 1 -- 172.21.15.46:0/3922084429 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) ==== 53+0+73 (secure 0 0 0) 0x7faae8060f80 con 0x7faaf01613c0 2024-05-09T03:12:20.036 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:12:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad0076300 msgr2=0x7faad0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad0076300 0x7faad0078810 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7faae0007fd0 tx=0x7faae0007de0 comp rx=0 tx=0).stop 2024-05-09T03:12:20.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 msgr2=0x7faaf00cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00cb880 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7faae8007d80 tx=0x7faae800e4b0 comp rx=0 tx=0).stop 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 shutdown_connections 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faad0076300 0x7faad0078810 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaf01613c0 0x7faaf00cb880 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faaf00b7d00 0x7faaf014f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 --2- 172.21.15.46:0/3922084429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faaf00b5210 0x7faaf01fadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 >> 172.21.15.46:0/3922084429 conn(0x7faaf00d29a0 msgr2=0x7faaf0163df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.036+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 shutdown_connections 2024-05-09T03:12:20.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20.037+0000 7faaf4c15640 1 -- 172.21.15.46:0/3922084429 wait complete. 2024-05-09T03:12:20.049 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:12:20.050 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:12:20.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.050 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:12:20.067 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42378sec preferred_lft 42378sec 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:12:20.068 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:12:20.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.070 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:12:20.131 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:12:20.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.133 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:12:20.190 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:12:20.191 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:12:20.191 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:12:20.191 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:12:20.191 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:12:20.191 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:12:20.191 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:12:20.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:12:20.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:12:20.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:12:20.191 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:12:20.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:12:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:12:20.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.321 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:12:20.337 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:12:20.337 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:12:20.337 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:12:20.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:12:20.403 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:12:20.404 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:12:20.404 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:20.459 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:12:20.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:12:20.460 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:20.515 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:12:20.581 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:12:20.638 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:12:20.639 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:12:20.640 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:20.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.641 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:12:20.694 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:12:20.694 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:12:20.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:12:20.762 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:12:20.823 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:12:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:12:20.825 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:12:20.826 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:12:20.827 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:12:20.827 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:20.827 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:12:20.827 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:20.827 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:12:20.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.827 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:12:20.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:12:20.831+0000 7fe506c88480 -1 init, newargv = 0x7fe4e8001f00 newargc=15 2024-05-09T03:12:20.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54987]: starting ceph client 2024-05-09T03:12:20.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54987]: starting fuse 2024-05-09T03:12:20.848 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:12:20.848 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:12:20.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:12:20.848 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:12:20.849 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:12:20.850 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:12:20.970 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.54987.asok 2024-05-09T03:12:20.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:12:20.981 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54987.asok 2024-05-09T03:12:20.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:20.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54987.asok status 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:12:21.129 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "54987", 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6078, 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6078 2024-05-09T03:12:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2774924560 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2774924560 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6078 192.168.0.1:0/2774924560", 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2774924560", 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:12:21.131 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 46, 2024-05-09T03:12:21.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 55, 2024-05-09T03:12:21.132 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:12:21.132 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:12:21.132 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:12:21.132 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:21.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:21.138 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:12:21.138 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:21.154 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:12:21.154 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:21.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:21.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:21.219 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'chmod 711 a/b/d; sync .') 2024-05-09T03:12:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:12:21.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:12:21.323 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:12:21.323 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:12:21.340 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:12:21.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.494+0000 7ff27ba2b640 1 Processor -- start 2024-05-09T03:12:21.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.494+0000 7ff27ba2b640 1 -- start start 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.495+0000 7ff27ba2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2740bb770 0x7ff2740d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.495+0000 7ff27ba2b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2740d3180 0x7ff2741642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.495+0000 7ff27ba2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff274164900 con 0x7ff274163eb0 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.495+0000 7ff27ba2b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff2740d4bb0 con 0x7ff2740d3180 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.495+0000 7ff27ba2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff274164be0 con 0x7ff2740bb770 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2740d3180 0x7ff2741642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2740d3180 0x7ff2741642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44096/0 (socket says 172.21.15.46:44096) 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff278fc6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff274163eb0 0x7ff2740d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56590/0 (socket says 172.21.15.46:56590) 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 -- 172.21.15.46:0/644468057 learned_addr learned my addr 172.21.15.46:0/644468057 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff2797c7640 1 --2- 172.21.15.46:0/644468057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2740bb770 0x7ff2740d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 -- 172.21.15.46:0/644468057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2740bb770 msgr2=0x7ff2740d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 --2- 172.21.15.46:0/644468057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2740bb770 0x7ff2740d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 -- 172.21.15.46:0/644468057 >> v1:172.21.15.46:6789/0 conn(0x7ff274163eb0 legacy=0x7ff2740d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff279fc8640 1 -- 172.21.15.46:0/644468057 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2740d4840 con 0x7ff2740d3180 2024-05-09T03:12:21.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.496+0000 7ff26bfff640 1 -- 172.21.15.46:0/644468057 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1262172471 0 0) 0x7ff274164900 con 0x7ff274163eb0 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.497+0000 7ff279fc8640 1 --2- 172.21.15.46:0/644468057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740d3180 0x7ff2741642c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f5e1c3d39aeaf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.497+0000 7ff26bfff640 1 -- 172.21.15.46:0/644468057 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c036070 con 0x7ff2740d3180 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.497+0000 7ff26bfff640 1 -- 172.21.15.46:0/644468057 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff26c031600 con 0x7ff2740d3180 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.497+0000 7ff26bfff640 1 -- 172.21.15.46:0/644468057 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0319d0 con 0x7ff2740d3180 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.497+0000 7ff27ba2b640 1 -- 172.21.15.46:0/644468057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740d3180 msgr2=0x7ff2741642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.497+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/644468057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740d3180 0x7ff2741642c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c02a710 comp rx=0 tx=0).stop 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.498+0000 7ff27ba2b640 1 -- 172.21.15.46:0/644468057 shutdown_connections 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.498+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/644468057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740d3180 0x7ff2741642c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.498+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/644468057 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2740bb770 0x7ff2740d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.498+0000 7ff27ba2b640 1 -- 172.21.15.46:0/644468057 >> 172.21.15.46:0/644468057 conn(0x7ff27406e370 msgr2=0x7ff27406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.498+0000 7ff27ba2b640 1 -- 172.21.15.46:0/644468057 shutdown_connections 2024-05-09T03:12:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.498+0000 7ff27ba2b640 1 -- 172.21.15.46:0/644468057 wait complete. 2024-05-09T03:12:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.499+0000 7ff27ba2b640 1 Processor -- start 2024-05-09T03:12:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.499+0000 7ff27ba2b640 1 -- start start 2024-05-09T03:12:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff27ba2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2740bb770 0x7ff2741f6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff27ba2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff274163eb0 0x7ff2740ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff27ba2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 0x7ff2740ce940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff27ba2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2740d0bc0 con 0x7ff2740cc810 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff27ba2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff2740bbb80 con 0x7ff2740bb770 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff27ba2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff2740d0d40 con 0x7ff274163eb0 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 0x7ff2740ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 0x7ff2740ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44100/0 (socket says 172.21.15.46:44100) 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 -- 172.21.15.46:0/114770429 learned_addr learned my addr 172.21.15.46:0/114770429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff2797c7640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2740bb770 0x7ff2741f6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff278fc6640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff274163eb0 0x7ff2740ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 -- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff274163eb0 msgr2=0x7ff2740ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff274163eb0 0x7ff2740ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 -- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2740bb770 msgr2=0x7ff2741f6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2740bb770 0x7ff2741f6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 -- 172.21.15.46:0/114770429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2741f7330 con 0x7ff2740cc810 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff2797c7640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2740bb770 0x7ff2741f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff279fc8640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 0x7ff2740ce940 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.501+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c036050 con 0x7ff2740cc810 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.502+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff26c042070 con 0x7ff2740cc810 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.502+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2740cceb0 con 0x7ff2740cc810 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.502+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c009510 con 0x7ff2740cc810 2024-05-09T03:12:21.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.502+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2740cd470 con 0x7ff2740cc810 2024-05-09T03:12:21.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.503+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27406a270 con 0x7ff2740cc810 2024-05-09T03:12:21.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.503+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff26c007bf0 con 0x7ff2740cc810 2024-05-09T03:12:21.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.504+0000 7ff269ffb640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2580762d0 0x7ff2580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.504+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7ff26c0b84b0 con 0x7ff2740cc810 2024-05-09T03:12:21.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.504+0000 7ff2797c7640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2580762d0 0x7ff2580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.505+0000 7ff2797c7640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2580762d0 0x7ff2580787e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff2740b8880 tx=0x7ff270008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:21.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.509+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff26c081d70 con 0x7ff2740cc810 2024-05-09T03:12:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.712+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff2740ccc40 con 0x7ff2740cc810 2024-05-09T03:12:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.713+0000 7ff269ffb640 1 -- 172.21.15.46:0/114770429 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) ==== 80+0+0 (secure 0 0 0) 0x7ff26c081670 con 0x7ff2740cc810 2024-05-09T03:12:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:12:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2580762d0 msgr2=0x7ff2580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2580762d0 0x7ff2580787e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff2740b8880 tx=0x7ff270008040 comp rx=0 tx=0).stop 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 msgr2=0x7ff2740ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 0x7ff2740ce940 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c004a40 comp rx=0 tx=0).stop 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 shutdown_connections 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2580762d0 0x7ff2580787e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2740cc810 0x7ff2740ce940 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff274163eb0 0x7ff2740ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 --2- 172.21.15.46:0/114770429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2740bb770 0x7ff2741f6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 >> 172.21.15.46:0/114770429 conn(0x7ff27406e370 msgr2=0x7ff2740b5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 shutdown_connections 2024-05-09T03:12:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.715+0000 7ff27ba2b640 1 -- 172.21.15.46:0/114770429 wait complete. 2024-05-09T03:12:21.728 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:12:21.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:21.728 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:12:21.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54987]: fuse finished with error 0 and tester_r 0 2024-05-09T03:12:21.754 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:12:21.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:12:21.754 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:12:21.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:12:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.965+0000 7fd21d053640 1 Processor -- start 2024-05-09T03:12:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.965+0000 7fd21d053640 1 -- start start 2024-05-09T03:12:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd21d053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2180b7e70 0x7fd2180d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd21d053640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2180d3df0 0x7fd2180b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd21d053640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2180b5550 con 0x7fd2180b5140 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd21d053640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd2180d3480 con 0x7fd2180d3df0 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd21d053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd2180d44e0 con 0x7fd2180b7e70 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd216d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2180b5140 0x7fd2180b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56600/0 (socket says 172.21.15.46:56600) 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd21759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2180d3df0 0x7fd2180b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.966+0000 7fd216d9d640 1 -- 172.21.15.46:0/167713030 learned_addr learned my addr 172.21.15.46:0/167713030 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.967+0000 7fd21659c640 1 --2- 172.21.15.46:0/167713030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2180b7e70 0x7fd2180d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.967+0000 7fd21759e640 1 -- 172.21.15.46:0/167713030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2180b7e70 msgr2=0x7fd2180d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.967+0000 7fd21759e640 1 --2- 172.21.15.46:0/167713030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2180b7e70 0x7fd2180d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.967+0000 7fd21759e640 1 -- 172.21.15.46:0/167713030 >> v1:172.21.15.46:6789/0 conn(0x7fd2180b5140 legacy=0x7fd2180b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.967+0000 7fd21759e640 1 -- 172.21.15.46:0/167713030 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2180d47c0 con 0x7fd2180d3df0 2024-05-09T03:12:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.967+0000 7fd215d9b640 1 -- 172.21.15.46:0/167713030 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1909333505 0 0) 0x7fd2180b5550 con 0x7fd2180b5140 2024-05-09T03:12:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21759e640 1 --2- 172.21.15.46:0/167713030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180d3df0 0x7fd2180b9380 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd210009d90 tx=0x7fd21002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e25211076a8c9e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd215d9b640 1 -- 172.21.15.46:0/167713030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210007740 con 0x7fd2180d3df0 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd215d9b640 1 -- 172.21.15.46:0/167713030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2100078e0 con 0x7fd2180d3df0 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd215d9b640 1 -- 172.21.15.46:0/167713030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd210007cb0 con 0x7fd2180d3df0 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 -- 172.21.15.46:0/167713030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180d3df0 msgr2=0x7fd2180b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 --2- 172.21.15.46:0/167713030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180d3df0 0x7fd2180b9380 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd210009d90 tx=0x7fd21002a6f0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 -- 172.21.15.46:0/167713030 shutdown_connections 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 --2- 172.21.15.46:0/167713030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180d3df0 0x7fd2180b9380 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 --2- 172.21.15.46:0/167713030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd2180b7e70 0x7fd2180d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 -- 172.21.15.46:0/167713030 >> 172.21.15.46:0/167713030 conn(0x7fd21808d3f0 msgr2=0x7fd21808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 -- 172.21.15.46:0/167713030 shutdown_connections 2024-05-09T03:12:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.968+0000 7fd21d053640 1 -- 172.21.15.46:0/167713030 wait complete. 2024-05-09T03:12:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.971+0000 7fd21d053640 1 Processor -- start 2024-05-09T03:12:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.971+0000 7fd21d053640 1 -- start start 2024-05-09T03:12:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21d053640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 0x7fd2180c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21d053640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2180d3df0 0x7fd218145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21d053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd218148580 0x7fd218148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21d053640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2180d4200 con 0x7fd2180b5140 2024-05-09T03:12:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21d053640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd2180d4d80 con 0x7fd2180d3df0 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21d053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd2180d5580 con 0x7fd218148580 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 0x7fd2180c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 0x7fd2180c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44116/0 (socket says 172.21.15.46:44116) 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 -- 172.21.15.46:0/673987644 learned_addr learned my addr 172.21.15.46:0/673987644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21659c640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2180d3df0 0x7fd218145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd21759e640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd218148580 0x7fd218148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 -- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd218148580 msgr2=0x7fd218148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd218148580 0x7fd218148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 -- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2180d3df0 msgr2=0x7fd218145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2180d3df0 0x7fd218145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.972+0000 7fd216d9d640 1 -- 172.21.15.46:0/673987644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd218148ed0 con 0x7fd2180b5140 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.973+0000 7fd216d9d640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 0x7fd2180c9810 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c410 tx=0x7fd20800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.973+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2080074e0 con 0x7fd2180b5140 2024-05-09T03:12:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.973+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2080047e0 con 0x7fd2180b5140 2024-05-09T03:12:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.973+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208004bd0 con 0x7fd2180b5140 2024-05-09T03:12:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.973+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2180c9d50 con 0x7fd2180b5140 2024-05-09T03:12:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.973+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd218146870 con 0x7fd2180b5140 2024-05-09T03:12:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.974+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21806a270 con 0x7fd2180b5140 2024-05-09T03:12:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.975+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd208011070 con 0x7fd2180b5140 2024-05-09T03:12:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.975+0000 7fd1ff7fe640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd204076300 0x7fd204078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.975+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fd208097100 con 0x7fd2180b5140 2024-05-09T03:12:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.976+0000 7fd21659c640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd204076300 0x7fd204078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.976+0000 7fd21659c640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd204076300 0x7fd204078810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd2181f9810 tx=0x7fd20c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:21.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:21.980+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd218148ed0 con 0x7fd2180b5140 2024-05-09T03:12:22.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.210+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd2180ca010 con 0x7fd2180b5140 2024-05-09T03:12:22.212 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:12:22.211+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:12:22.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.212+0000 7fd1ff7fe640 1 -- 172.21.15.46:0/673987644 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) ==== 161+0+0 (secure 0 0 0) 0x7fd208060370 con 0x7fd2180b5140 2024-05-09T03:12:22.214 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd204076300 msgr2=0x7fd204078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd204076300 0x7fd204078810 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd2181f9810 tx=0x7fd20c009190 comp rx=0 tx=0).stop 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 msgr2=0x7fd2180c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 0x7fd2180c9810 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd20800c410 tx=0x7fd20800c8d0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 shutdown_connections 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd218148580 0x7fd218148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2180d3df0 0x7fd218145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd204076300 0x7fd204078810 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 --2- 172.21.15.46:0/673987644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2180b5140 0x7fd2180c9810 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 >> 172.21.15.46:0/673987644 conn(0x7fd21808d3f0 msgr2=0x7fd218155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 shutdown_connections 2024-05-09T03:12:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.216+0000 7fd21d053640 1 -- 172.21.15.46:0/673987644 wait complete. 2024-05-09T03:12:22.219 INFO:tasks.ceph.mds.a.smithi046.stderr: -4300> 2024-05-09T03:12:08.820+0000 7fe170391640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:12:22.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_journal_corrupt_dentry_first 1.0 2024-05-09T03:12:22.279 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:12:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.348+0000 7f8bb04ee640 1 Processor -- start 2024-05-09T03:12:22.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.348+0000 7f8bb04ee640 1 -- start start 2024-05-09T03:12:22.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bb04ee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8ba80c78a0 0x7f8ba80b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:22.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bb04ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba8163e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:22.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bb04ee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba80c8720 con 0x7f8ba8164390 2024-05-09T03:12:22.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bb04ee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba8145d10 con 0x7f8ba80c78a0 2024-05-09T03:12:22.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bb04ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba80c7e80 con 0x7f8ba8072500 2024-05-09T03:12:22.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bae28a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8ba80c78a0 0x7f8ba80b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8baea8b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8ba8164390 0x7f8ba81461a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56620/0 (socket says 172.21.15.46:56620) 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8baea8b640 1 -- 172.21.15.46:0/3164366475 learned_addr learned my addr 172.21.15.46:0/3164366475 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bada89640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba8163e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bae28a640 1 -- 172.21.15.46:0/3164366475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 msgr2=0x7f8ba8163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bae28a640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba8163e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bae28a640 1 -- 172.21.15.46:0/3164366475 >> v1:172.21.15.46:6789/0 conn(0x7f8ba8164390 legacy=0x7f8ba81461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bae28a640 1 -- 172.21.15.46:0/3164366475 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba80c8160 con 0x7f8ba80c78a0 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bad288640 1 -- 172.21.15.46:0/3164366475 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1304690041 0 0) 0x7f8ba80c8720 con 0x7f8ba8164390 2024-05-09T03:12:22.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.350+0000 7f8bada89640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba8163e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.351+0000 7f8bae28a640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80b1dd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0097a0 tx=0x7f8b9c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283964d6ff31d7b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.351+0000 7f8bad288640 1 -- 172.21.15.46:0/3164366475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c037070 con 0x7f8ba80c78a0 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.351+0000 7f8bad288640 1 -- 172.21.15.46:0/3164366475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b9c032440 con 0x7f8ba80c78a0 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.351+0000 7f8bad288640 1 -- 172.21.15.46:0/3164366475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c032810 con 0x7f8ba80c78a0 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3164366475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 msgr2=0x7f8ba80b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80b1dd0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0097a0 tx=0x7f8b9c02a180 comp rx=0 tx=0).stop 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3164366475 shutdown_connections 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba8163e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3164366475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80b1dd0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3164366475 >> 172.21.15.46:0/3164366475 conn(0x7f8ba80705c0 msgr2=0x7f8ba80709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3164366475 shutdown_connections 2024-05-09T03:12:22.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.352+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3164366475 wait complete. 2024-05-09T03:12:22.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.354+0000 7f8bb04ee640 1 Processor -- start 2024-05-09T03:12:22.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.354+0000 7f8bb04ee640 1 -- start start 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bb04ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba80cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bb04ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bb04ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba8164390 0x7f8ba8144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bb04ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba81468b0 con 0x7f8ba80c78a0 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bb04ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba8072910 con 0x7f8ba8164390 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bb04ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba8146a30 con 0x7f8ba8072500 2024-05-09T03:12:22.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bada89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bada89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44176/0 (socket says 172.21.15.46:44176) 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.355+0000 7f8bada89640 1 -- 172.21.15.46:0/3904710877 learned_addr learned my addr 172.21.15.46:0/3904710877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8baea8b640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba8164390 0x7f8ba8144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bae28a640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba80cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bada89640 1 -- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 msgr2=0x7f8ba80cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bada89640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba80cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bada89640 1 -- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba8164390 msgr2=0x7f8ba8144b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bada89640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba8164390 0x7f8ba8144b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bada89640 1 -- 172.21.15.46:0/3904710877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba8162a20 con 0x7f8ba80c78a0 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bae28a640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba80cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:22.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bada89640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80ca2b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009fd0 tx=0x7f8ba400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:22.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba401a070 con 0x7f8ba80c78a0 2024-05-09T03:12:22.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba4003bb0 con 0x7f8ba80c78a0 2024-05-09T03:12:22.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba8162f20 con 0x7f8ba80c78a0 2024-05-09T03:12:22.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4004030 con 0x7f8ba80c78a0 2024-05-09T03:12:22.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.356+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba81634e0 con 0x7f8ba80c78a0 2024-05-09T03:12:22.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.357+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba806a270 con 0x7f8ba80c78a0 2024-05-09T03:12:22.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.359+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8ba401d070 con 0x7f8ba80c78a0 2024-05-09T03:12:22.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.359+0000 7f8b9affd640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b8c076300 0x7f8b8c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:22.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.359+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f8ba4097fe0 con 0x7f8ba80c78a0 2024-05-09T03:12:22.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.359+0000 7f8bae28a640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b8c076300 0x7f8b8c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:22.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.360+0000 7f8bae28a640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b8c076300 0x7f8b8c078810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c034000 tx=0x7f8b9c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:22.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.363+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8ba4010070 con 0x7f8ba80c78a0 2024-05-09T03:12:22.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.559+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}] v 0) -- 0x7f8ba80d0290 con 0x7f8ba80c78a0 2024-05-09T03:12:22.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.562+0000 7f8b9affd640 1 -- 172.21.15.46:0/3904710877 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}]=0 v19)=0 v19) ==== 139+0+0 (secure 0 0 0) 0x7f8ba40612e0 con 0x7f8ba80c78a0 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b8c076300 msgr2=0x7f8b8c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b8c076300 0x7f8b8c078810 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c034000 tx=0x7f8b9c02e000 comp rx=0 tx=0).stop 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 msgr2=0x7f8ba80ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80ca2b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009fd0 tx=0x7f8ba400d560 comp rx=0 tx=0).stop 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 shutdown_connections 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba8164390 0x7f8ba8144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b8c076300 0x7f8b8c078810 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba80c78a0 0x7f8ba80ca2b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 --2- 172.21.15.46:0/3904710877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8ba8072500 0x7f8ba80cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 >> 172.21.15.46:0/3904710877 conn(0x7f8ba80705c0 msgr2=0x7f8ba8071b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 shutdown_connections 2024-05-09T03:12:22.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:22.565+0000 7f8bb04ee640 1 -- 172.21.15.46:0/3904710877 wait complete. 2024-05-09T03:12:27.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:12:27.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.709+0000 7f95e0a27640 1 Processor -- start 2024-05-09T03:12:27.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.709+0000 7f95e0a27640 1 -- start start 2024-05-09T03:12:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95e0a27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95dc0bb770 0x7f95dc0d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95e0a27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95dc163eb0 0x7f95dc1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95e0a27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0d2a70 0x7f95dc0d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95e0a27640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95dc0d4bb0 con 0x7f95dc0bb770 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95e0a27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95dc0d33c0 con 0x7f95dc163eb0 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95e0a27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95dc164900 con 0x7f95dc0d2a70 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95da59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95dc0bb770 0x7f95dc0d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95da59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95dc0bb770 0x7f95dc0d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46774/0 (socket says 172.21.15.46:46774) 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95da59c640 1 -- 172.21.15.46:0/1056901109 learned_addr learned my addr 172.21.15.46:0/1056901109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95d9d9b640 1 --2- 172.21.15.46:0/1056901109 >> v2:172.21.15.88:3300/0 conn(0x7f95dc163eb0 0x7f95dc1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.711+0000 7f95dad9d640 1 --2- 172.21.15.46:0/1056901109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0d2a70 0x7f95dc0d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.712+0000 7f95da59c640 1 -- 172.21.15.46:0/1056901109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0d2a70 msgr2=0x7f95dc0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:27.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.712+0000 7f95da59c640 1 --2- 172.21.15.46:0/1056901109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0d2a70 0x7f95dc0d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.712+0000 7f95da59c640 1 -- 172.21.15.46:0/1056901109 >> v2:172.21.15.88:3300/0 conn(0x7f95dc163eb0 msgr2=0x7f95dc1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:27.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.712+0000 7f95da59c640 1 --2- 172.21.15.46:0/1056901109 >> v2:172.21.15.88:3300/0 conn(0x7f95dc163eb0 0x7f95dc1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.712+0000 7f95da59c640 1 -- 172.21.15.46:0/1056901109 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc164be0 con 0x7f95dc0bb770 2024-05-09T03:12:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.712+0000 7f95da59c640 1 --2- 172.21.15.46:0/1056901109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0bb770 0x7f95dc0d5040 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0d6d00 tx=0x7f95d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63da00586b32d01a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95d959a640 1 -- 172.21.15.46:0/1056901109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4037070 con 0x7f95dc0bb770 2024-05-09T03:12:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95d959a640 1 -- 172.21.15.46:0/1056901109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d4032440 con 0x7f95dc0bb770 2024-05-09T03:12:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95d959a640 1 -- 172.21.15.46:0/1056901109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4032810 con 0x7f95dc0bb770 2024-05-09T03:12:27.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 -- 172.21.15.46:0/1056901109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0bb770 msgr2=0x7f95dc0d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 --2- 172.21.15.46:0/1056901109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0bb770 0x7f95dc0d5040 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0d6d00 tx=0x7f95d402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 -- 172.21.15.46:0/1056901109 shutdown_connections 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 --2- 172.21.15.46:0/1056901109 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0d2a70 0x7f95dc0d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 --2- 172.21.15.46:0/1056901109 >> v2:172.21.15.88:3300/0 conn(0x7f95dc163eb0 0x7f95dc1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 --2- 172.21.15.46:0/1056901109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0bb770 0x7f95dc0d5040 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 -- 172.21.15.46:0/1056901109 >> 172.21.15.46:0/1056901109 conn(0x7f95dc06e370 msgr2=0x7f95dc06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 -- 172.21.15.46:0/1056901109 shutdown_connections 2024-05-09T03:12:27.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.713+0000 7f95e0a27640 1 -- 172.21.15.46:0/1056901109 wait complete. 2024-05-09T03:12:27.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.715+0000 7f95e0a27640 1 Processor -- start 2024-05-09T03:12:27.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.715+0000 7f95e0a27640 1 -- start start 2024-05-09T03:12:27.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95e0a27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0bb770 0x7f95dc15ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95e0a27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 0x7f95dc14f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95e0a27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95dc163eb0 0x7f95dc1ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95e0a27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc0bbb80 con 0x7f95dc0d2a70 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95e0a27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc0d5580 con 0x7f95dc163eb0 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95e0a27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95dc0d0bc0 con 0x7f95dc0bb770 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 0x7f95dc14f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 0x7f95dc14f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46790/0 (socket says 172.21.15.46:46790) 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 -- 172.21.15.46:0/368000842 learned_addr learned my addr 172.21.15.46:0/368000842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95dad9d640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95dc163eb0 0x7f95dc1ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95da59c640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0bb770 0x7f95dc15ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 -- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0bb770 msgr2=0x7f95dc15ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0bb770 0x7f95dc15ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 -- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95dc163eb0 msgr2=0x7f95dc1ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95dc163eb0 0x7f95dc1ebe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 -- 172.21.15.46:0/368000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc14cca0 con 0x7f95dc0d2a70 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.717+0000 7f95d9d9b640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 0x7f95dc14f600 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f95d000d5f0 tx=0x7f95d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.718+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d00125f0 con 0x7f95dc0d2a70 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.718+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d0019070 con 0x7f95dc0d2a70 2024-05-09T03:12:27.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.718+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d00129d0 con 0x7f95dc0d2a70 2024-05-09T03:12:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.718+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95dc15d370 con 0x7f95dc0d2a70 2024-05-09T03:12:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.718+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95dc15d930 con 0x7f95dc0d2a70 2024-05-09T03:12:27.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.719+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95dc0d1c80 con 0x7f95dc0d2a70 2024-05-09T03:12:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.720+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95d0012b70 con 0x7f95dc0d2a70 2024-05-09T03:12:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.720+0000 7f95caffd640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95bc0762d0 0x7f95bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.720+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f95d0097910 con 0x7f95dc0d2a70 2024-05-09T03:12:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.720+0000 7f95da59c640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95bc0762d0 0x7f95bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:27.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.721+0000 7f95da59c640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95bc0762d0 0x7f95bc0787e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f95d4034000 tx=0x7f95d402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.725+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95dc0d1c80 con 0x7f95dc0d2a70 2024-05-09T03:12:27.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:27.955+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f95dc15d100 con 0x7f95dc0d2a70 2024-05-09T03:12:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.912+0000 7f95caffd640 1 -- 172.21.15.46:0/368000842 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v49) ==== 162+0+0 (secure 0 0 0) 0x7f95d0069030 con 0x7f95dc0d2a70 2024-05-09T03:12:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95bc0762d0 msgr2=0x7f95bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95bc0762d0 0x7f95bc0787e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f95d4034000 tx=0x7f95d402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 msgr2=0x7f95dc14f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 0x7f95dc14f600 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f95d000d5f0 tx=0x7f95d000dab0 comp rx=0 tx=0).stop 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 shutdown_connections 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95dc163eb0 0x7f95dc1ebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95bc0762d0 0x7f95bc0787e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95dc0d2a70 0x7f95dc14f600 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:28.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 --2- 172.21.15.46:0/368000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95dc0bb770 0x7f95dc15ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:28.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 >> 172.21.15.46:0/368000842 conn(0x7f95dc06e370 msgr2=0x7f95dc0b9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:28.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 shutdown_connections 2024-05-09T03:12:28.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:28.915+0000 7f95e0a27640 1 -- 172.21.15.46:0/368000842 wait complete. 2024-05-09T03:12:28.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.053+0000 7f73570a1640 1 Processor -- start 2024-05-09T03:12:29.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.053+0000 7f73570a1640 1 -- start start 2024-05-09T03:12:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.054+0000 7f73570a1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7350071270 0x7f7350070c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.054+0000 7f73570a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73500ce6d0 0x7f73500ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:29.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.054+0000 7f73570a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73500cff00 con 0x7f735006f790 2024-05-09T03:12:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.054+0000 7f73570a1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f73500aed40 con 0x7f7350071270 2024-05-09T03:12:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.054+0000 7f73570a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f73500cfcb0 con 0x7f73500ce6d0 2024-05-09T03:12:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f734ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73500ce6d0 0x7f73500ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f7354e3d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7350071270 0x7f7350070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f735563e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f735006f790 0x7f735006fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:42044/0 (socket says 172.21.15.46:42044) 2024-05-09T03:12:29.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f735563e640 1 -- 172.21.15.46:0/3313536967 learned_addr learned my addr 172.21.15.46:0/3313536967 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f734ffff640 1 -- 172.21.15.46:0/3313536967 >> v2:172.21.15.88:3300/0 conn(0x7f7350071270 msgr2=0x7f7350070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f734ffff640 1 --2- 172.21.15.46:0/3313536967 >> v2:172.21.15.88:3300/0 conn(0x7f7350071270 0x7f7350070c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f734ffff640 1 -- 172.21.15.46:0/3313536967 >> v1:172.21.15.88:6789/0 conn(0x7f735006f790 legacy=0x7f735006fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f734ffff640 1 -- 172.21.15.46:0/3313536967 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73500d01e0 con 0x7f73500ce6d0 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.055+0000 7f7354e3d640 1 --2- 172.21.15.46:0/3313536967 >> v2:172.21.15.88:3300/0 conn(0x7f7350071270 0x7f7350070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.056+0000 7f734ffff640 1 --2- 172.21.15.46:0/3313536967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73500ce6d0 0x7f73500ceae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa938462cb662d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.056+0000 7f734f7fe640 1 -- 172.21.15.46:0/3313536967 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c037070 con 0x7f73500ce6d0 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.056+0000 7f734f7fe640 1 -- 172.21.15.46:0/3313536967 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733c02ae50 con 0x7f73500ce6d0 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.056+0000 7f734f7fe640 1 -- 172.21.15.46:0/3313536967 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c032660 con 0x7f73500ce6d0 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 -- 172.21.15.46:0/3313536967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73500ce6d0 msgr2=0x7f73500ceae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 --2- 172.21.15.46:0/3313536967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73500ce6d0 0x7f73500ceae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f733c0097a0 tx=0x7f733c02a100 comp rx=0 tx=0).stop 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 -- 172.21.15.46:0/3313536967 shutdown_connections 2024-05-09T03:12:29.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 --2- 172.21.15.46:0/3313536967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73500ce6d0 0x7f73500ceae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 --2- 172.21.15.46:0/3313536967 >> v2:172.21.15.88:3300/0 conn(0x7f7350071270 0x7f7350070c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 -- 172.21.15.46:0/3313536967 >> 172.21.15.46:0/3313536967 conn(0x7f73500b3c40 msgr2=0x7f73500b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 -- 172.21.15.46:0/3313536967 shutdown_connections 2024-05-09T03:12:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.057+0000 7f73570a1640 1 -- 172.21.15.46:0/3313536967 wait complete. 2024-05-09T03:12:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.059+0000 7f73570a1640 1 Processor -- start 2024-05-09T03:12:29.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.059+0000 7f73570a1640 1 -- start start 2024-05-09T03:12:29.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f73570a1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f735006f790 0x7f735015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f73570a1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7350071270 0x7f7350155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f73570a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 0x7f73501563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f73570a1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73500d0a80 con 0x7f73500ce6d0 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f73570a1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73500d07a0 con 0x7f735006f790 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f73570a1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f73500cf020 con 0x7f7350071270 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.060+0000 7f735563e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 0x7f73501563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 0x7f73501563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46792/0 (socket says 172.21.15.46:46792) 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 -- 172.21.15.46:0/94441052 learned_addr learned my addr 172.21.15.46:0/94441052 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f7354e3d640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f735006f790 0x7f735015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f734ffff640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7350071270 0x7f7350155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 -- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7350071270 msgr2=0x7f7350155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:29.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7350071270 0x7f7350155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 -- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f735006f790 msgr2=0x7f735015b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f735006f790 0x7f735015b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 -- 172.21.15.46:0/94441052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7350159ec0 con 0x7f73500ce6d0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f735563e640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 0x7f73501563f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f734800ed20 tx=0x7f734800c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348015070 con 0x7f73500ce6d0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7348011670 con 0x7f73500ce6d0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73501553d0 con 0x7f73500ce6d0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7348011a20 con 0x7f73500ce6d0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.061+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7350155990 con 0x7f73500ce6d0 2024-05-09T03:12:29.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.062+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7350159c10 con 0x7f73500ce6d0 2024-05-09T03:12:29.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.063+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7348018070 con 0x7f73500ce6d0 2024-05-09T03:12:29.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.064+0000 7f734d7fa640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73340762d0 0x7f73340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:29.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.064+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f734809a120 con 0x7f73500ce6d0 2024-05-09T03:12:29.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.064+0000 7f7354e3d640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73340762d0 0x7f73340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:29.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.065+0000 7f7354e3d640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73340762d0 0x7f73340787e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7344009560 tx=0x7f7344009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:29.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.069+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f73480633d0 con 0x7f73500ce6d0 2024-05-09T03:12:29.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.298+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f735006a270 con 0x7f73500ce6d0 2024-05-09T03:12:29.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.299+0000 7f734d7fa640 1 -- 172.21.15.46:0/94441052 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5172 (secure 0 0 0) 0x7f73480676d0 con 0x7f73500ce6d0 2024-05-09T03:12:29.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:29.301 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:28.916447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-05-09T03:12:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 50 2024-05-09T03:12:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73340762d0 msgr2=0x7f73340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73340762d0 0x7f73340787e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7344009560 tx=0x7f7344009190 comp rx=0 tx=0).stop 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 msgr2=0x7f73501563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 0x7f73501563f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f734800ed20 tx=0x7f734800c640 comp rx=0 tx=0).stop 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 shutdown_connections 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f73340762d0 0x7f73340787e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73500ce6d0 0x7f73501563f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7350071270 0x7f7350155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 --2- 172.21.15.46:0/94441052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f735006f790 0x7f735015b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 >> 172.21.15.46:0/94441052 conn(0x7f73500b3c40 msgr2=0x7f73500aeed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 shutdown_connections 2024-05-09T03:12:29.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:29.301+0000 7f73570a1640 1 -- 172.21.15.46:0/94441052 wait complete. 2024-05-09T03:12:29.717 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:28.916447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6102}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6102': {'gid': 6102, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/103238544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 103238544}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 103238544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:12:29.717 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6102:up:replay 2024-05-09T03:12:30.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.846+0000 7fb85bb2d640 1 Processor -- start 2024-05-09T03:12:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.846+0000 7fb85bb2d640 1 -- start start 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85bb2d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8540b6180 0x7fb8540b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85bb2d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb85408d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85bb2d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb85408d910 0x7fb8540d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85bb2d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb8540b90d0 con 0x7fb8540b6180 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85bb2d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb8540cfb70 con 0x7fb85408d910 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85bb2d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb8540cf2d0 con 0x7fb85408cfc0 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8540b6180 0x7fb8540b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8540b6180 0x7fb8540b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46804/0 (socket says 172.21.15.46:46804) 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 -- 172.21.15.46:0/398341085 learned_addr learned my addr 172.21.15.46:0/398341085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8590c8640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb85408d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb85a0ca640 1 --2- 172.21.15.46:0/398341085 >> v2:172.21.15.88:3300/0 conn(0x7fb85408d910 0x7fb8540d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 -- 172.21.15.46:0/398341085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 msgr2=0x7fb85408d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb85408d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 -- 172.21.15.46:0/398341085 >> v2:172.21.15.88:3300/0 conn(0x7fb85408d910 msgr2=0x7fb8540d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 --2- 172.21.15.46:0/398341085 >> v2:172.21.15.88:3300/0 conn(0x7fb85408d910 0x7fb8540d1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8598c9640 1 -- 172.21.15.46:0/398341085 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8540cf5b0 con 0x7fb8540b6180 2024-05-09T03:12:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.848+0000 7fb8590c8640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb85408d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:30.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.849+0000 7fb8598c9640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb8540b8b90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb85406faa0 tx=0x7fb85002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c29467a4308a8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:30.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.849+0000 7fb8588c7640 1 -- 172.21.15.46:0/398341085 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850037070 con 0x7fb8540b6180 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.849+0000 7fb8588c7640 1 -- 172.21.15.46:0/398341085 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb850032440 con 0x7fb8540b6180 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.849+0000 7fb8588c7640 1 -- 172.21.15.46:0/398341085 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb850032810 con 0x7fb8540b6180 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 -- 172.21.15.46:0/398341085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 msgr2=0x7fb8540b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb8540b8b90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb85406faa0 tx=0x7fb85002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 -- 172.21.15.46:0/398341085 shutdown_connections 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/398341085 >> v2:172.21.15.88:3300/0 conn(0x7fb85408d910 0x7fb8540d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb85408d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/398341085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb8540b8b90 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 -- 172.21.15.46:0/398341085 >> 172.21.15.46:0/398341085 conn(0x7fb8540b5660 msgr2=0x7fb8540b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 -- 172.21.15.46:0/398341085 shutdown_connections 2024-05-09T03:12:30.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.850+0000 7fb85bb2d640 1 -- 172.21.15.46:0/398341085 wait complete. 2024-05-09T03:12:30.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.852+0000 7fb85bb2d640 1 Processor -- start 2024-05-09T03:12:30.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.852+0000 7fb85bb2d640 1 -- start start 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85bb2d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb8541e9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85bb2d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb85408d910 0x7fb854152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85bb2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb85406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85bb2d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb85408dd20 con 0x7fb8540b6180 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85bb2d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb8540b6a60 con 0x7fb85408d910 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85bb2d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb8540d21c0 con 0x7fb85408cfc0 2024-05-09T03:12:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.853+0000 7fb85a0ca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb85406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb85406ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46816/0 (socket says 172.21.15.46:46816) 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb8590c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb85408d910 0x7fb854152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb8598c9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb8541e9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 -- 172.21.15.46:0/732962337 learned_addr learned my addr 172.21.15.46:0/732962337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 -- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 msgr2=0x7fb8541e9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb8541e9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 -- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb85408d910 msgr2=0x7fb854152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb85408d910 0x7fb854152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 -- 172.21.15.46:0/732962337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8541e91a0 con 0x7fb8540b6180 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85a0ca640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb85406ee70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009fd0 tx=0x7fb84c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c014420 con 0x7fb8540b6180 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb84c019070 con 0x7fb8540b6180 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb84c014800 con 0x7fb8540b6180 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8541e8100 con 0x7fb8540b6180 2024-05-09T03:12:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.854+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8541e86c0 con 0x7fb8540b6180 2024-05-09T03:12:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.855+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8541e7e70 con 0x7fb8540b6180 2024-05-09T03:12:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.856+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb84c0149a0 con 0x7fb8540b6180 2024-05-09T03:12:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.857+0000 7fb84a7fc640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8380762d0 0x7fb8380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.857+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7fb84c097d50 con 0x7fb8540b6180 2024-05-09T03:12:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.857+0000 7fb8598c9640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8380762d0 0x7fb8380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.858+0000 7fb8598c9640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8380762d0 0x7fb8380787e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb850034000 tx=0x7fb85002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:30.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:30.862+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb84c060f60 con 0x7fb8540b6180 2024-05-09T03:12:31.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.096+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb854161480 con 0x7fb8540b6180 2024-05-09T03:12:31.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.097+0000 7fb84a7fc640 1 -- 172.21.15.46:0/732962337 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5172 (secure 0 0 0) 0x7fb84c065260 con 0x7fb8540b6180 2024-05-09T03:12:31.098 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:31.099 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":52,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:29.921329+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-05-09T03:12:31.099 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 52 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8380762d0 msgr2=0x7fb8380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8380762d0 0x7fb8380787e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb850034000 tx=0x7fb85002e000 comp rx=0 tx=0).stop 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 msgr2=0x7fb85406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb85406ee70 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009fd0 tx=0x7fb84c00e4b0 comp rx=0 tx=0).stop 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 shutdown_connections 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8380762d0 0x7fb8380787e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8540b6180 0x7fb85406ee70 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb85408d910 0x7fb854152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 --2- 172.21.15.46:0/732962337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb85408cfc0 0x7fb8541e9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:31.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 >> 172.21.15.46:0/732962337 conn(0x7fb8540b5660 msgr2=0x7fb854072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:31.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 shutdown_connections 2024-05-09T03:12:31.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:31.099+0000 7fb85bb2d640 1 -- 172.21.15.46:0/732962337 wait complete. 2024-05-09T03:12:31.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:29.921329+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6102}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6102': {'gid': 6102, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.46:6835/103238544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 103238544}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 103238544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:12:31.514 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6102:up:rejoin 2024-05-09T03:12:32.515 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.644+0000 7f24c3172640 1 Processor -- start 2024-05-09T03:12:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.644+0000 7f24c3172640 1 -- start start 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c3172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 0x7f24bc0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c3172640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f24bc0d2090 0x7f24bc0d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c3172640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24bc0d4900 con 0x7f24bc0d1c80 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c3172640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f24bc0b5340 con 0x7f24bc0d2090 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c3172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f24bc0b4fd0 con 0x7f24bc08cd60 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24bbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f24bc0d1c80 0x7f24bc0cf2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40202/0 (socket says 172.21.15.46:40202) 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24bbfff640 1 -- 172.21.15.46:0/2309317647 learned_addr learned my addr 172.21.15.46:0/2309317647 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c0f0e640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 0x7f24bc0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.645+0000 7f24c170f640 1 --2- 172.21.15.46:0/2309317647 >> v2:172.21.15.88:3300/0 conn(0x7f24bc0d2090 0x7f24bc0d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270652670 0 0) 0x7f24bc0d4900 con 0x7f24bc0d1c80 2024-05-09T03:12:32.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24a8003520 con 0x7f24bc0d1c80 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 812680433 0 0) 0x7f24a8003520 con 0x7f24bc0d1c80 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 msgr2=0x7f24bc0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 0x7f24bc0d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 >> v2:172.21.15.88:3300/0 conn(0x7f24bc0d2090 msgr2=0x7f24bc0d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 --2- 172.21.15.46:0/2309317647 >> v2:172.21.15.88:3300/0 conn(0x7f24bc0d2090 0x7f24bc0d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc0d2c00 con 0x7f24bc0d1c80 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f24ac001a70 con 0x7f24bc0d1c80 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24c170f640 1 --2- 172.21.15.46:0/2309317647 >> v2:172.21.15.88:3300/0 conn(0x7f24bc0d2090 0x7f24bc0d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 >> v1:172.21.15.46:6789/0 conn(0x7f24bc0d1c80 legacy=0x7f24bc0cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a8003a00 0x7f24a8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 0x7f24a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24a8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0b5340 con 0x7f24a8004d40 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0b4fd0 con 0x7f24a8003a00 2024-05-09T03:12:32.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f24a800ba70 con 0x7f24bc08cd60 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24c170f640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24a8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24c0f0e640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a8003a00 0x7f24a8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.646+0000 7f24bbfff640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 0x7f24a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24c170f640 1 -- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 msgr2=0x7f24a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24c170f640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc08cd60 0x7f24a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24c170f640 1 -- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a8003a00 msgr2=0x7f24a8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24c170f640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a8003a00 0x7f24a8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24c170f640 1 -- 172.21.15.46:0/2309317647 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a8003520 con 0x7f24a8004d40 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24c170f640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24a8005150 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009c60 tx=0x7f24a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e764702ac871eb45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4009860 con 0x7f24a8004d40 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a4004020 con 0x7f24a8004d40 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.647+0000 7f24bb7fe640 1 -- 172.21.15.46:0/2309317647 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a40043f0 con 0x7f24a8004d40 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c3172640 1 -- 172.21.15.46:0/2309317647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 msgr2=0x7f24a8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c3172640 1 --2- 172.21.15.46:0/2309317647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24a8005150 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009c60 tx=0x7f24a8005830 comp rx=0 tx=0).stop 2024-05-09T03:12:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c170f640 1 -- 172.21.15.46:0/2309317647 reap_dead start 2024-05-09T03:12:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c3172640 1 -- 172.21.15.46:0/2309317647 shutdown_connections 2024-05-09T03:12:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c3172640 1 -- 172.21.15.46:0/2309317647 >> 172.21.15.46:0/2309317647 conn(0x7f24bc0d3c00 msgr2=0x7f24bc0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c3172640 1 -- 172.21.15.46:0/2309317647 shutdown_connections 2024-05-09T03:12:32.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.648+0000 7f24c3172640 1 -- 172.21.15.46:0/2309317647 wait complete. 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.650+0000 7f24c3172640 1 Processor -- start 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.650+0000 7f24c3172640 1 -- start start 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c3172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24bc1520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c3172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc160480 0x7f24bc15f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c3172640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc160bc0 0x7f24bc0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c3172640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0b92b0 con 0x7f24a8004d40 2024-05-09T03:12:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c3172640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0b9130 con 0x7f24bc160bc0 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c3172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f24bc0cf9c0 con 0x7f24bc160480 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24bc1520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24bc1520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46832/0 (socket says 172.21.15.46:46832) 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 -- 172.21.15.46:0/908114469 learned_addr learned my addr 172.21.15.46:0/908114469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c170f640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc160bc0 0x7f24bc0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24bbfff640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc160480 0x7f24bc15f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 -- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc160480 msgr2=0x7f24bc15f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc160480 0x7f24bc15f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 -- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc160bc0 msgr2=0x7f24bc0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc160bc0 0x7f24bc0cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 -- 172.21.15.46:0/908114469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc0b3c40 con 0x7f24a8004d40 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c170f640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc160bc0 0x7f24bc0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24bbfff640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc160480 0x7f24bc15f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24c0f0e640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24bc1520d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f24b4011930 tx=0x7f24b4011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.652+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b400fa30 con 0x7f24a8004d40 2024-05-09T03:12:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.653+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b400fbd0 con 0x7f24a8004d40 2024-05-09T03:12:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.653+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24bc0b3f20 con 0x7f24a8004d40 2024-05-09T03:12:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.653+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4014440 con 0x7f24a8004d40 2024-05-09T03:12:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.653+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24bc15f150 con 0x7f24a8004d40 2024-05-09T03:12:32.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.653+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24bc06a270 con 0x7f24a8004d40 2024-05-09T03:12:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.654+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f24b4012040 con 0x7f24a8004d40 2024-05-09T03:12:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.655+0000 7f24b97fa640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f249c0762d0 0x7f249c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.655+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f24b4097c80 con 0x7f24a8004d40 2024-05-09T03:12:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.655+0000 7f24bbfff640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f249c0762d0 0x7f249c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:32.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.655+0000 7f24bbfff640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f249c0762d0 0x7f249c0787e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f24ac001390 tx=0x7f24ac006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:32.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.660+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f24b4019030 con 0x7f24a8004d40 2024-05-09T03:12:32.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.889+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24bc0b40f0 con 0x7f24a8004d40 2024-05-09T03:12:32.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.890+0000 7f24b97fa640 1 -- 172.21.15.46:0/908114469 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5179 (secure 0 0 0) 0x7f24b40614f0 con 0x7f24a8004d40 2024-05-09T03:12:32.891 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:32.891 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":53,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:31.924314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":4}]} 2024-05-09T03:12:32.892 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 53 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f249c0762d0 msgr2=0x7f249c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f249c0762d0 0x7f249c0787e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f24ac001390 tx=0x7f24ac006040 comp rx=0 tx=0).stop 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 msgr2=0x7f24bc1520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24bc1520d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f24b4011930 tx=0x7f24b4011df0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 shutdown_connections 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc160bc0 0x7f24bc0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f24bc160480 0x7f24bc15f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f249c0762d0 0x7f249c0787e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 --2- 172.21.15.46:0/908114469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24a8004d40 0x7f24bc1520d0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 >> 172.21.15.46:0/908114469 conn(0x7f24bc0d3c00 msgr2=0x7f24bc15d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 shutdown_connections 2024-05-09T03:12:32.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:32.892+0000 7f24c3172640 1 -- 172.21.15.46:0/908114469 wait complete. 2024-05-09T03:12:33.225 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:12:33.224+0000 7fda4624e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:12:33.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:31.924314+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 56, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6102}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6102': {'gid': 6102, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.46:6835/103238544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 103238544}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 103238544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6102, 'qdb_cluster': [6102]} 2024-05-09T03:12:33.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:12:33.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T03:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.427+0000 7f5a5f59e640 1 Processor -- start 2024-05-09T03:12:33.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.428+0000 7f5a5f59e640 1 -- start start 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a600d4650 0x7f5a600b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a600d2930 0x7f5a600d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5f59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a600d3280 con 0x7f5a600b75c0 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5f59e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5a600d5760 con 0x7f5a600d2930 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5f59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5a600d4b50 con 0x7f5a600d4650 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a600d2930 0x7f5a600d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a600d2930 0x7f5a600d2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46836/0 (socket says 172.21.15.46:46836) 2024-05-09T03:12:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5a600b75c0 0x7f5a600b79d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40214/0 (socket says 172.21.15.46:40214) 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/1124417487 learned_addr learned my addr 172.21.15.46:0/1124417487 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.429+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/1124417487 >> v2:172.21.15.88:3300/0 conn(0x7f5a600d4650 0x7f5a600b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.430+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/1124417487 >> v2:172.21.15.88:3300/0 conn(0x7f5a600d4650 msgr2=0x7f5a600b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.430+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/1124417487 >> v2:172.21.15.88:3300/0 conn(0x7f5a600d4650 0x7f5a600b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.430+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/1124417487 >> v1:172.21.15.46:6789/0 conn(0x7f5a600b75c0 legacy=0x7f5a600b79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.430+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/1124417487 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a600d4e30 con 0x7f5a600d2930 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.430+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/1124417487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d2930 0x7f5a600d2d40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5a6008eab0 tx=0x7f5a5402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b631ad08160cc4a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5d59a640 1 -- 172.21.15.46:0/1124417487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54037070 con 0x7f5a600d2930 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5d59a640 1 -- 172.21.15.46:0/1124417487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a54032440 con 0x7f5a600d2930 2024-05-09T03:12:33.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5d59a640 1 -- 172.21.15.46:0/1124417487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54032810 con 0x7f5a600d2930 2024-05-09T03:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 -- 172.21.15.46:0/1124417487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d2930 msgr2=0x7f5a600d2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/1124417487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d2930 0x7f5a600d2d40 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5a6008eab0 tx=0x7f5a5402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 -- 172.21.15.46:0/1124417487 shutdown_connections 2024-05-09T03:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/1124417487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d2930 0x7f5a600d2d40 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/1124417487 >> v2:172.21.15.88:3300/0 conn(0x7f5a600d4650 0x7f5a600b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 -- 172.21.15.46:0/1124417487 >> 172.21.15.46:0/1124417487 conn(0x7f5a6006e110 msgr2=0x7f5a6006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 -- 172.21.15.46:0/1124417487 shutdown_connections 2024-05-09T03:12:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.431+0000 7f5a5f59e640 1 -- 172.21.15.46:0/1124417487 wait complete. 2024-05-09T03:12:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.433+0000 7f5a5f59e640 1 Processor -- start 2024-05-09T03:12:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.433+0000 7f5a5f59e640 1 -- start start 2024-05-09T03:12:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5f59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a600b75c0 0x7f5a6014cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5f59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a600d2930 0x7f5a600cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 0x7f5a601e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5f59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60160540 con 0x7f5a600d4650 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5f59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5a600d53f0 con 0x7f5a600d2930 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5f59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5a601606c0 con 0x7f5a600b75c0 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 0x7f5a601e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 0x7f5a601e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46850/0 (socket says 172.21.15.46:46850) 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/3258920332 learned_addr learned my addr 172.21.15.46:0/3258920332 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a600d2930 0x7f5a600cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a600b75c0 0x7f5a6014cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a600b75c0 msgr2=0x7f5a6014cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a600b75c0 0x7f5a6014cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a600d2930 msgr2=0x7f5a600cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a600d2930 0x7f5a600cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/3258920332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6014d470 con 0x7f5a600d4650 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.435+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 0x7f5a601e8a60 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800c430 tx=0x7f5a5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.436+0000 7f5a4effd640 1 -- 172.21.15.46:0/3258920332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580074e0 con 0x7f5a600d4650 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.436+0000 7f5a4effd640 1 -- 172.21.15.46:0/3258920332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a58007680 con 0x7f5a600d4650 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.436+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a600cbe20 con 0x7f5a600d4650 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.436+0000 7f5a4effd640 1 -- 172.21.15.46:0/3258920332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580049e0 con 0x7f5a600d4650 2024-05-09T03:12:33.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.436+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a600cdd90 con 0x7f5a600d4650 2024-05-09T03:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.438+0000 7f5a4effd640 1 -- 172.21.15.46:0/3258920332 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5a5801a020 con 0x7f5a600d4650 2024-05-09T03:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.438+0000 7f5a4effd640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a400762d0 0x7f5a400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.438+0000 7f5a4effd640 1 -- 172.21.15.46:0/3258920332 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f5a58096bf0 con 0x7f5a600d4650 2024-05-09T03:12:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.438+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a400762d0 0x7f5a400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.439+0000 7f5a6bd8b740 1 Processor -- start 2024-05-09T03:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.439+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a400762d0 0x7f5a400787e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5a6008ee00 tx=0x7f5a5000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.439+0000 7f5a6bd8b740 1 -- start start 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bcf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13be9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137de590 0x55fa13bea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fa13b074e0 con 0x55fa137ff270 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fa13aa2ff0 con 0x55fa137de590 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fa13b034a0 con 0x55fa137d63b0 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a5e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bcf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a5e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bcf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46866/0 (socket says 172.21.15.46:46866) 2024-05-09T03:12:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.440+0000 7f5a5e59c640 1 -- 172.21.15.46:0/2191950447 learned_addr learned my addr 172.21.15.46:0/2191950447 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13be9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137de590 0x55fa13bea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5e59c640 1 -- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 msgr2=0x55fa13be9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13be9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5e59c640 1 -- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137de590 msgr2=0x55fa13bea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137de590 0x55fa13bea2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5e59c640 1 -- 172.21.15.46:0/2191950447 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa13a89720 con 0x55fa137ff270 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bcf060 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5a50017770 tx=0x7f5a50014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca24656f0ea9f03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a3ffff640 1 -- 172.21.15.46:0/2191950447 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5001b070 con 0x55fa137ff270 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a3ffff640 1 -- 172.21.15.46:0/2191950447 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a50027070 con 0x55fa137ff270 2024-05-09T03:12:33.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.441+0000 7f5a3ffff640 1 -- 172.21.15.46:0/2191950447 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a50015c90 con 0x55fa137ff270 2024-05-09T03:12:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2191950447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 msgr2=0x55fa13bcf060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bcf060 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5a50017770 tx=0x7f5a50014000 comp rx=0 tx=0).stop 2024-05-09T03:12:33.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2191950447 shutdown_connections 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137de590 0x55fa13bea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13be9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2191950447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bcf060 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2191950447 >> 172.21.15.46:0/2191950447 conn(0x55fa137f33c0 msgr2=0x55fa13ba46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2191950447 shutdown_connections 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.442+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2191950447 wait complete. 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.443+0000 7f5a6bd8b740 1 -- start start 2024-05-09T03:12:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.443+0000 7f5a6bd8b740 1 Processor -- start 2024-05-09T03:12:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13b9e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137de590 0x55fa13bd1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bfc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fa13a8e760 con 0x55fa137ff270 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fa13bfd3a0 con 0x55fa137d63b0 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fa13b9ed20 con 0x55fa137de590 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 10 client.0 _ll_register_callbacks cb 0x55fa13ab14b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bfc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bfc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46878/0 (socket says 172.21.15.46:46878) 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/873815975 learned_addr learned my addr 172.21.15.46:0/873815975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13b9e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137de590 0x55fa13bd1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137de590 msgr2=0x55fa13bd1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137de590 0x55fa13bd1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 msgr2=0x55fa13b9e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13b9e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 -- 172.21.15.46:0/873815975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa13acfbd0 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137de590 0x55fa13bd1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bfc760 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5a5805bfb0 tx=0x7f5a58060070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3377cbb32e6286 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/873815975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580689c0 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/873815975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f5a58004b80 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.445+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a50027070 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/873815975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5a58004d80 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 10 client.6147 fetch_fsmap learned FSMap version 53 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55fa13ab5280 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/873815975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a50027070 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/873815975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 53) ==== 3494+0+0 (secure 0 0 0) 0x7f5a5808f070 con 0x55fa137ff270 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 10 client.6147 fetch_fsmap finished waiting for FSMap version 53 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 10 client.6147 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:12:33.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 20 client.6147 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa137bde40 0x55fa13be1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 4 client.6147 mds_command: new command op to 6102 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 --> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa13b150f0 con 0x55fa137bde40 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa137bde40 0x55fa13be1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a3d7fa640 10 client.6147 ms_handle_connect on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.446+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa137bde40 0x55fa13be1180 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc08eba6eb2ff1e server_cookie=650760f87ac2e135 in_seq=0 out_seq=0 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/873815975 <== mds.0 v2:172.21.15.46:6834/103238544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fa13b150f0 con 0x55fa137bde40 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a3d7fa640 10 client.6147 handle_command_reply: tid=0 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 client.6147 shutdown 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 msgr2=0x55fa13bfc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bfc760 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5a5805bfb0 tx=0x7f5a58060070 comp rx=0 tx=0).stop 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 shutdown_connections 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa137bde40 0x55fa13be1180 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137ff270 0x55fa13bfc760 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a3d7fa640 0 client.6147 ms_handle_reset on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137de590 0x55fa13bd1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/873815975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13b9e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 >> 172.21.15.46:0/873815975 conn(0x55fa137f33c0 msgr2=0x55fa13ba46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.447+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 shutdown_connections 2024-05-09T03:12:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.448+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/873815975 wait complete. 2024-05-09T03:12:33.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.448+0000 7f5a6bd8b740 20 client.6147 trim_cache size 0 max 16384 2024-05-09T03:12:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.467+0000 7f5a6bd8b740 1 Processor -- start 2024-05-09T03:12:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.467+0000 7f5a6bd8b740 1 -- start start 2024-05-09T03:12:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13c0cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13bfbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137ff270 0x55fa13bfc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60201d30 con 0x55fa137de590 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60201090 con 0x55fa137ff270 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5a602016e0 con 0x55fa137d63b0 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13bfbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13bfbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46880/0 (socket says 172.21.15.46:46880) 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2281113252 learned_addr learned my addr 172.21.15.46:0/2281113252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137ff270 0x55fa13bfc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13c0cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 msgr2=0x55fa13c0cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13c0cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137ff270 msgr2=0x55fa13bfc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137ff270 0x55fa13bfc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2281113252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1386a850 con 0x55fa137de590 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137ff270 0x55fa13bfc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.468+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13c0cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.469+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13bfbb70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a54035040 tx=0x7f5a5402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f147fbe794ba110c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.469+0000 7f5a3f7fe640 1 -- 172.21.15.46:0/2281113252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54042070 con 0x55fa137de590 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.469+0000 7f5a3f7fe640 1 -- 172.21.15.46:0/2281113252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a540328b0 con 0x55fa137de590 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.469+0000 7f5a3f7fe640 1 -- 172.21.15.46:0/2281113252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54037040 con 0x55fa137de590 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2281113252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 msgr2=0x55fa13bfbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13bfbb70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a54035040 tx=0x7f5a5402c000 comp rx=0 tx=0).stop 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2281113252 shutdown_connections 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137ff270 0x55fa13bfc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13bfbb70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2281113252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137d63b0 0x55fa13c0cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2281113252 >> 172.21.15.46:0/2281113252 conn(0x55fa137f33c0 msgr2=0x55fa13bcb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2281113252 shutdown_connections 2024-05-09T03:12:33.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2281113252 wait complete. 2024-05-09T03:12:33.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.470+0000 7f5a6bd8b740 1 -- start start 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.471+0000 7f5a6bd8b740 1 Processor -- start 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13ba4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13ba5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137ff270 0x55fa13bc6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fa13bfcca0 con 0x55fa137de590 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fa13bfd0e0 con 0x55fa137d63b0 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fa13bca7a0 con 0x55fa137ff270 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a6bd8b740 10 client.0 _ll_register_callbacks cb 0x7f5a5805f910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13ba5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13ba5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46888/0 (socket says 172.21.15.46:46888) 2024-05-09T03:12:33.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2640566049 learned_addr learned my addr 172.21.15.46:0/2640566049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13ba4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137ff270 0x55fa13bc6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137ff270 msgr2=0x55fa13bc6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.472+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137ff270 0x55fa13bc6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 msgr2=0x55fa13ba4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13ba4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a5dd9b640 1 -- 172.21.15.46:0/2640566049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa138e16b0 con 0x55fa137de590 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13ba4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a5ed9d640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137ff270 0x55fa13bc6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a5dd9b640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13ba5220 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009930 tx=0x7f5a54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f092d634ae9aff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:33.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/2640566049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54042040 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/2640566049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f5a54004450 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a540328b0 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/2640566049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f5a54058050 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 10 client.6153 fetch_fsmap learned FSMap version 53 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=2+}) -- 0x55fa13ac3630 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/2640566049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a540328b0 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/2640566049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 53) ==== 3494+0+0 (secure 0 0 0) 0x7f5a5405b020 con 0x55fa137de590 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 10 client.6153 fetch_fsmap finished waiting for FSMap version 53 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 10 client.6153 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 20 client.6153 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:33.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa13bc7430 0x55fa13be93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 4 client.6153 mds_command: new command op to 6102 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.473+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 --> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] -- command(tid 0: {"prefix": "status"}) -- 0x55fa13b0ef60 con 0x55fa13bc7430 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa13bc7430 0x55fa13be93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a3d7fa640 10 client.6153 ms_handle_connect on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a5e59c640 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa13bc7430 0x55fa13be93d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ace8b369127aab server_cookie=582d5a76b2686c72 in_seq=0 out_seq=0 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a3d7fa640 1 -- 172.21.15.46:0/2640566049 <== mds.0 v2:172.21.15.46:6834/103238544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55fa13b0ef60 con 0x55fa13bc7430 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a3d7fa640 10 client.6153 handle_command_reply: tid=0 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a6bd8b740 1 client.6153 shutdown 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 msgr2=0x55fa13ba5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.474+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13ba5220 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009930 tx=0x7f5a54004730 comp rx=0 tx=0).stop 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 shutdown_connections 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x55fa13bc7430 0x55fa13be93d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fa137ff270 0x55fa13bc6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fa137de590 0x55fa13ba5220 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 --2- 172.21.15.46:0/2640566049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fa137d63b0 0x55fa13ba4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a3d7fa640 0 client.6153 ms_handle_reset on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 >> 172.21.15.46:0/2640566049 conn(0x55fa137f33c0 msgr2=0x55fa13be6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 shutdown_connections 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 1 -- 172.21.15.46:0/2640566049 wait complete. 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a6bd8b740 20 client.6153 trim_cache size 0 max 16384 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:12:33.483 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6102, 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.5582217329999999, 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 53, 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 56, 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 56, 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.195188872999999 2024-05-09T03:12:33.484 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a400762d0 msgr2=0x7f5a400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a400762d0 0x7f5a400787e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5a6008ee00 tx=0x7f5a5000b040 comp rx=0 tx=0).stop 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 msgr2=0x7f5a601e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 0x7f5a601e8a60 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800c430 tx=0x7f5a5800c8f0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 shutdown_connections 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a400762d0 0x7f5a400787e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a600d4650 0x7f5a601e8a60 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a600d2930 0x7f5a600cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 --2- 172.21.15.46:0/3258920332 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a600b75c0 0x7f5a6014cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 >> 172.21.15.46:0/3258920332 conn(0x7f5a6006e110 msgr2=0x7f5a60162850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 shutdown_connections 2024-05-09T03:12:33.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.475+0000 7f5a5f59e640 1 -- 172.21.15.46:0/3258920332 wait complete. 2024-05-09T03:12:33.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:33.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.600+0000 7f1ea8d1f640 1 Processor -- start 2024-05-09T03:12:33.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.600+0000 7f1ea8d1f640 1 -- start start 2024-05-09T03:12:33.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea8d1f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ea40d57a0 0x7f1ea4160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea8d1f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ea40b4af0 0x7f1ea40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea8d1f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40b77e0 0x7f1ea40b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea8d1f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea40d5bb0 con 0x7f1ea40b4af0 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea8d1f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea40b8130 con 0x7f1ea40d57a0 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea8d1f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea40d49e0 con 0x7f1ea40b77e0 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ea40b4af0 0x7f1ea40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ea40b4af0 0x7f1ea40b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46892/0 (socket says 172.21.15.46:46892) 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2019765504 learned_addr learned my addr 172.21.15.46:0/2019765504 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea259c640 1 --2- 172.21.15.46:0/2019765504 >> v2:172.21.15.88:3300/0 conn(0x7f1ea40d57a0 0x7f1ea4160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea2d9d640 1 --2- 172.21.15.46:0/2019765504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40b77e0 0x7f1ea40b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2019765504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40b77e0 msgr2=0x7f1ea40b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea1d9b640 1 --2- 172.21.15.46:0/2019765504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40b77e0 0x7f1ea40b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.602+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2019765504 >> v2:172.21.15.88:3300/0 conn(0x7f1ea40d57a0 msgr2=0x7f1ea4160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.603+0000 7f1ea1d9b640 1 --2- 172.21.15.46:0/2019765504 >> v2:172.21.15.88:3300/0 conn(0x7f1ea40d57a0 0x7f1ea4160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.603+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2019765504 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea40d4cc0 con 0x7f1ea40b4af0 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.603+0000 7f1ea1d9b640 1 --2- 172.21.15.46:0/2019765504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b4af0 0x7f1ea40b4f00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1ea408ed20 tx=0x7f1e8c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535ac3b48a634bc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.603+0000 7f1ea159a640 1 -- 172.21.15.46:0/2019765504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c037070 con 0x7f1ea40b4af0 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.603+0000 7f1ea159a640 1 -- 172.21.15.46:0/2019765504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c032440 con 0x7f1ea40b4af0 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.603+0000 7f1ea159a640 1 -- 172.21.15.46:0/2019765504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c032810 con 0x7f1ea40b4af0 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2019765504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b4af0 msgr2=0x7f1ea40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2019765504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b4af0 0x7f1ea40b4f00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1ea408ed20 tx=0x7f1e8c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2019765504 shutdown_connections 2024-05-09T03:12:33.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2019765504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40b77e0 0x7f1ea40b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2019765504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b4af0 0x7f1ea40b4f00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2019765504 >> v2:172.21.15.88:3300/0 conn(0x7f1ea40d57a0 0x7f1ea4160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2019765504 >> 172.21.15.46:0/2019765504 conn(0x7f1ea40743e0 msgr2=0x7f1ea40d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2019765504 shutdown_connections 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.604+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2019765504 wait complete. 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.606+0000 7f1ea8d1f640 1 Processor -- start 2024-05-09T03:12:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.606+0000 7f1ea8d1f640 1 -- start start 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea8d1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea40b4af0 0x7f1ea414e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea8d1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 0x7f1ea414bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea8d1f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40d57a0 0x7f1ea415c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea8d1f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea40d15b0 con 0x7f1ea40b77e0 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea8d1f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea40d5280 con 0x7f1ea40b4af0 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea8d1f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea40d1920 con 0x7f1ea40d57a0 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 0x7f1ea414bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 0x7f1ea414bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46900/0 (socket says 172.21.15.46:46900) 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2355091918 learned_addr learned my addr 172.21.15.46:0/2355091918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea2d9d640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40d57a0 0x7f1ea415c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea259c640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea40b4af0 0x7f1ea414e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40d57a0 msgr2=0x7f1ea415c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40d57a0 0x7f1ea415c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea40b4af0 msgr2=0x7f1ea414e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea40b4af0 0x7f1ea414e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 -- 172.21.15.46:0/2355091918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea414c660 con 0x7f1ea40b77e0 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea259c640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea40b4af0 0x7f1ea414e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1ea1d9b640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 0x7f1ea414bda0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c02a220 tx=0x7f1e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.608+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c041020 con 0x7f1ea40b77e0 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.609+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8c0306c0 con 0x7f1ea40b77e0 2024-05-09T03:12:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.609+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e8c037040 con 0x7f1ea40b77e0 2024-05-09T03:12:33.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.609+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea414ec30 con 0x7f1ea40b77e0 2024-05-09T03:12:33.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.609+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea414f420 con 0x7f1ea40b77e0 2024-05-09T03:12:33.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.610+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea406a270 con 0x7f1ea40b77e0 2024-05-09T03:12:33.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.610+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1e8c044020 con 0x7f1ea40b77e0 2024-05-09T03:12:33.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.611+0000 7f1e9affd640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e840762d0 0x7f1e840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.611+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f1e8c0b7f50 con 0x7f1ea40b77e0 2024-05-09T03:12:33.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.611+0000 7f1ea259c640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e840762d0 0x7f1e840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.612+0000 7f1ea259c640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e840762d0 0x7f1e840787e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1e94006fd0 tx=0x7f1e94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.616+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1e8c04b030 con 0x7f1ea40b77e0 2024-05-09T03:12:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.846+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ea414eed0 con 0x7f1ea40b77e0 2024-05-09T03:12:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.846+0000 7f1e9affd640 1 -- 172.21.15.46:0/2355091918 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5179 (secure 0 0 0) 0x7f1e8c0817c0 con 0x7f1ea40b77e0 2024-05-09T03:12:33.848 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:33.848 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":53,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:31.924314+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":4}]} 2024-05-09T03:12:33.848 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 53 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.848+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e840762d0 msgr2=0x7f1e840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.848+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e840762d0 0x7f1e840787e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1e94006fd0 tx=0x7f1e94008040 comp rx=0 tx=0).stop 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.848+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 msgr2=0x7f1ea414bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.848+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 0x7f1ea414bda0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c02a220 tx=0x7f1e8c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 shutdown_connections 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ea40d57a0 0x7f1ea415c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e840762d0 0x7f1e840787e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ea40b77e0 0x7f1ea414bda0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 --2- 172.21.15.46:0/2355091918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ea40b4af0 0x7f1ea414e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 >> 172.21.15.46:0/2355091918 conn(0x7f1ea40743e0 msgr2=0x7f1ea40b9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 shutdown_connections 2024-05-09T03:12:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.849+0000 7f1ea8d1f640 1 -- 172.21.15.46:0/2355091918 wait complete. 2024-05-09T03:12:33.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 true 2024-05-09T03:12:33.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.977+0000 7fa7d7ce0640 1 Processor -- start 2024-05-09T03:12:33.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.978+0000 7fa7d7ce0640 1 -- start start 2024-05-09T03:12:33.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d7ce0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa7d0164bf0 0x7fa7d00d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d7ce0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7d00b7e50 0x7fa7d00b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d7ce0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d00d17a0 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d7ce0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d0164760 con 0x7fa7d00b7e50 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d7ce0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d00d14c0 con 0x7fa7d0164bf0 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7d00b7e50 0x7fa7d00b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa7d00b7e50 0x7fa7d00b9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46912/0 (socket says 172.21.15.46:46912) 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 -- 172.21.15.46:0/297741071 learned_addr learned my addr 172.21.15.46:0/297741071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d5a7c640 1 --2- 172.21.15.46:0/297741071 >> v2:172.21.15.88:3300/0 conn(0x7fa7d0164bf0 0x7fa7d00d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 -- 172.21.15.46:0/297741071 >> v2:172.21.15.88:3300/0 conn(0x7fa7d0164bf0 msgr2=0x7fa7d00d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 --2- 172.21.15.46:0/297741071 >> v2:172.21.15.88:3300/0 conn(0x7fa7d0164bf0 0x7fa7d00d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 -- 172.21.15.46:0/297741071 >> v1:172.21.15.88:6789/0 conn(0x7fa7d00d0fb0 legacy=0x7fa7d00b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d627d640 1 -- 172.21.15.46:0/297741071 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d0164110 con 0x7fa7d00b7e50 2024-05-09T03:12:33.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.979+0000 7fa7d4a7a640 1 -- 172.21.15.46:0/297741071 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1062262022 0 0) 0x7fa7d00d17a0 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.980+0000 7fa7d627d640 1 --2- 172.21.15.46:0/297741071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00b7e50 0x7fa7d00b9190 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa7d008e330 tx=0x7fa7c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f41030a8d5a465d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.980+0000 7fa7d4a7a640 1 -- 172.21.15.46:0/297741071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8037070 con 0x7fa7d00b7e50 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.980+0000 7fa7d4a7a640 1 -- 172.21.15.46:0/297741071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c8032440 con 0x7fa7d00b7e50 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.980+0000 7fa7d4a7a640 1 -- 172.21.15.46:0/297741071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c8032810 con 0x7fa7d00b7e50 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/297741071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00b7e50 msgr2=0x7fa7d00b9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/297741071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00b7e50 0x7fa7d00b9190 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fa7d008e330 tx=0x7fa7c802a270 comp rx=0 tx=0).stop 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/297741071 shutdown_connections 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/297741071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00b7e50 0x7fa7d00b9190 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/297741071 >> v2:172.21.15.88:3300/0 conn(0x7fa7d0164bf0 0x7fa7d00d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/297741071 >> 172.21.15.46:0/297741071 conn(0x7fa7d00d2920 msgr2=0x7fa7d00d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:33.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/297741071 shutdown_connections 2024-05-09T03:12:33.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.981+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/297741071 wait complete. 2024-05-09T03:12:33.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.983+0000 7fa7d7ce0640 1 Processor -- start 2024-05-09T03:12:33.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.983+0000 7fa7d7ce0640 1 -- start start 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d7ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 0x7fa7d0155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d7ce0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7d0164bf0 0x7fa7d00cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d7ce0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7d00cedc0 0x7fa7d015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d7ce0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d006ecb0 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d7ce0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d006eb30 con 0x7fa7d00cedc0 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d7ce0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d00b52b0 con 0x7fa7d0164bf0 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d5a7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 0x7fa7d0155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d5a7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 0x7fa7d0155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46924/0 (socket says 172.21.15.46:46924) 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.984+0000 7fa7d5a7c640 1 -- 172.21.15.46:0/799805850 learned_addr learned my addr 172.21.15.46:0/799805850 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d627d640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7d00cedc0 0x7fa7d015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d527b640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7d0164bf0 0x7fa7d00cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d5a7c640 1 -- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7d0164bf0 msgr2=0x7fa7d00cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d5a7c640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7d0164bf0 0x7fa7d00cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d5a7c640 1 -- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7d00cedc0 msgr2=0x7fa7d015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:33.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d5a7c640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7d00cedc0 0x7fa7d015fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d5a7c640 1 -- 172.21.15.46:0/799805850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d0156030 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d627d640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7d00cedc0 0x7fa7d015fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d5a7c640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 0x7fa7d0155af0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc009d10 tx=0x7fa7cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc018020 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7cc014480 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d00cf370 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc014830 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.985+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d00cf930 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.986+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d006a270 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.987+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa7cc005d10 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.988+0000 7fa7c67fc640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa7b4076300 0x7fa7b4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:33.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.988+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7fa7cc098110 con 0x7fa7d00d0fb0 2024-05-09T03:12:33.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.988+0000 7fa7d527b640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa7b4076300 0x7fa7b4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:33.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.989+0000 7fa7d527b640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa7b4076300 0x7fa7b4078810 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00c9910 tx=0x7fa7c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:33.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:33.993+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa7cc022070 con 0x7fa7d00d0fb0 2024-05-09T03:12:34.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.258+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"} v 0) -- 0x7fa7d00d5760 con 0x7fa7d00d0fb0 2024-05-09T03:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.939+0000 7fa7c67fc640 1 -- 172.21.15.46:0/799805850 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"}]=0 freezing mds.6102 v54) ==== 112+0+0 (secure 0 0 0) 0x7fa7cc010040 con 0x7fa7d00d0fb0 2024-05-09T03:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:freezing mds.6102 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa7b4076300 msgr2=0x7fa7b4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa7b4076300 0x7fa7b4078810 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa7d00c9910 tx=0x7fa7c0009040 comp rx=0 tx=0).stop 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 msgr2=0x7fa7d0155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 0x7fa7d0155af0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc009d10 tx=0x7fa7cc00c490 comp rx=0 tx=0).stop 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 shutdown_connections 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7d00cedc0 0x7fa7d015fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa7d0164bf0 0x7fa7d00cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa7b4076300 0x7fa7b4078810 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 --2- 172.21.15.46:0/799805850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7d00d0fb0 0x7fa7d0155af0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.942+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 >> 172.21.15.46:0/799805850 conn(0x7fa7d00d2920 msgr2=0x7fa7d00b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:34.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.943+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 shutdown_connections 2024-05-09T03:12:34.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:34.943+0000 7fa7d7ce0640 1 -- 172.21.15.46:0/799805850 wait complete. 2024-05-09T03:12:34.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo daemon-helper kill ceph --cluster ceph -w 2024-05-09T03:12:34.960 DEBUG:teuthology.orchestra.run.smithi046:> timeout 30 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --connect-timeout=60 tell mds.4:0 flush journal 2024-05-09T03:12:34.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-05-09T03:12:35.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.086+0000 7fe69b89f640 1 Processor -- start 2024-05-09T03:12:35.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.086+0000 7fe69b89f640 1 -- start start 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69b89f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6940d33b0 0x7fe6940cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69b89f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69406f910 con 0x7fe6940b77a0 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69b89f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6940d3880 con 0x7fe6940d0c40 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69b89f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe69406f370 con 0x7fe6940d33b0 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe698e3a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe6940b77a0 0x7fe6940b7bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40228/0 (socket says 172.21.15.46:40228) 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe698e3a640 1 -- 172.21.15.46:0/3689748404 learned_addr learned my addr 172.21.15.46:0/3689748404 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69963b640 1 --2- 172.21.15.46:0/3689748404 >> v2:172.21.15.88:3300/0 conn(0x7fe6940d33b0 0x7fe6940cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe693fff640 1 -- 172.21.15.46:0/3689748404 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4241415889 0 0) 0x7fe69406f910 con 0x7fe6940b77a0 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe693fff640 1 -- 172.21.15.46:0/3689748404 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe67c003520 con 0x7fe6940b77a0 2024-05-09T03:12:35.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69963b640 1 -- 172.21.15.46:0/3689748404 >> v1:172.21.15.88:6789/0 conn(0x7fe6940d0c40 legacy=0x7fe6940d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69963b640 1 -- 172.21.15.46:0/3689748404 >> v1:172.21.15.46:6789/0 conn(0x7fe6940b77a0 legacy=0x7fe6940b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.088+0000 7fe69963b640 1 -- 172.21.15.46:0/3689748404 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69406f630 con 0x7fe6940d33b0 2024-05-09T03:12:35.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.089+0000 7fe69963b640 1 --2- 172.21.15.46:0/3689748404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d33b0 0x7fe6940cf950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe684009d90 tx=0x7fe68402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=898d03dcebbf8d85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.089+0000 7fe693fff640 1 -- 172.21.15.46:0/3689748404 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684007740 con 0x7fe6940d33b0 2024-05-09T03:12:35.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe693fff640 1 -- 172.21.15.46:0/3689748404 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6840078e0 con 0x7fe6940d33b0 2024-05-09T03:12:35.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe693fff640 1 -- 172.21.15.46:0/3689748404 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684007c90 con 0x7fe6940d33b0 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe69b89f640 1 -- 172.21.15.46:0/3689748404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d33b0 msgr2=0x7fe6940cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe69b89f640 1 --2- 172.21.15.46:0/3689748404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d33b0 0x7fe6940cf950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe684009d90 tx=0x7fe68402a6f0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe69b89f640 1 -- 172.21.15.46:0/3689748404 shutdown_connections 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe69b89f640 1 --2- 172.21.15.46:0/3689748404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d33b0 0x7fe6940cf950 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.090+0000 7fe69b89f640 1 -- 172.21.15.46:0/3689748404 >> 172.21.15.46:0/3689748404 conn(0x7fe69409b7d0 msgr2=0x7fe69409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.091+0000 7fe69b89f640 1 -- 172.21.15.46:0/3689748404 shutdown_connections 2024-05-09T03:12:35.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.091+0000 7fe69b89f640 1 -- 172.21.15.46:0/3689748404 wait complete. 2024-05-09T03:12:35.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.092+0000 7fe69b89f640 1 Processor -- start 2024-05-09T03:12:35.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.092+0000 7fe69b89f640 1 -- start start 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69b89f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6940b77a0 0x7fe6940a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69b89f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d0c40 0x7fe69414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69b89f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6940d33b0 0x7fe6940b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69b89f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940b43b0 con 0x7fe6940b77a0 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69b89f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940b4230 con 0x7fe6940d0c40 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69b89f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe69415d3d0 con 0x7fe6940d33b0 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6940b77a0 0x7fe6940a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6940b77a0 0x7fe6940a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46938/0 (socket says 172.21.15.46:46938) 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 -- 172.21.15.46:0/3389130884 learned_addr learned my addr 172.21.15.46:0/3389130884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe698e3a640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d0c40 0x7fe69414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe699e3c640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6940d33b0 0x7fe6940b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 -- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6940d33b0 msgr2=0x7fe6940b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6940d33b0 0x7fe6940b1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 -- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d0c40 msgr2=0x7fe69414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d0c40 0x7fe69414bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe69963b640 1 -- 172.21.15.46:0/3389130884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69409c9d0 con 0x7fe6940b77a0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.094+0000 7fe698e3a640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6940d0c40 0x7fe69414bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.095+0000 7fe69963b640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6940b77a0 0x7fe6940a0570 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe684009d90 tx=0x7fe684004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.095+0000 7fe691ffb640 1 -- 172.21.15.46:0/3389130884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684040070 con 0x7fe6940b77a0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.095+0000 7fe691ffb640 1 -- 172.21.15.46:0/3389130884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe684031c40 con 0x7fe6940b77a0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.095+0000 7fe691ffb640 1 -- 172.21.15.46:0/3389130884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684009450 con 0x7fe6940b77a0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.095+0000 7fe69b89f640 1 -- 172.21.15.46:0/3389130884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6941602a0 con 0x7fe6940b77a0 2024-05-09T03:12:35.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.095+0000 7fe69b89f640 1 -- 172.21.15.46:0/3389130884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69409cc90 con 0x7fe6940b77a0 2024-05-09T03:12:35.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.098+0000 7fe691ffb640 1 -- 172.21.15.46:0/3389130884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe684039040 con 0x7fe6940b77a0 2024-05-09T03:12:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.098+0000 7fe6a282b740 1 Processor -- start 2024-05-09T03:12:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.098+0000 7fe6a282b740 1 -- start start 2024-05-09T03:12:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.099+0000 7fe691ffb640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6800762d0 0x7fe6800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.099+0000 7fe691ffb640 1 -- 172.21.15.46:0/3389130884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7fe6840b8830 con 0x7fe6940b77a0 2024-05-09T03:12:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.099+0000 7fe699e3c640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6800762d0 0x7fe6800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe699e3c640 1 --2- 172.21.15.46:0/3389130884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6800762d0 0x7fe6800787e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe69409e4a0 tx=0x7fe68c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e363e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e55b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a498e0 0x556887e560b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe6a282b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556887d72f50 con 0x556887a498e0 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556887d2c3e0 con 0x556887a41700 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556887d6f070 con 0x556887a6a5c0 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe699e3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a498e0 0x556887e560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe698e3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e55b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe699e3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a498e0 0x556887e560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46944/0 (socket says 172.21.15.46:46944) 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe699e3c640 1 -- 172.21.15.46:0/3636976020 learned_addr learned my addr 172.21.15.46:0/3636976020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe69963b640 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe698e3a640 1 -- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 msgr2=0x556887e363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe698e3a640 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e363e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe698e3a640 1 -- 172.21.15.46:0/3636976020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a498e0 msgr2=0x556887e560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe698e3a640 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a498e0 0x556887e560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.100+0000 7fe698e3a640 1 -- 172.21.15.46:0/3636976020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556887b4ca00 con 0x556887a41700 2024-05-09T03:12:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.101+0000 7fe698e3a640 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e55b70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe6940710e0 tx=0x7fe688007cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a838dcb9bc4b55 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.101+0000 7fe67affd640 1 -- 172.21.15.46:0/3636976020 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68800fe20 con 0x556887a41700 2024-05-09T03:12:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.101+0000 7fe67affd640 1 -- 172.21.15.46:0/3636976020 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe688018070 con 0x556887a41700 2024-05-09T03:12:35.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.101+0000 7fe67affd640 1 -- 172.21.15.46:0/3636976020 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe688005af0 con 0x556887a41700 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 -- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 msgr2=0x556887e55b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e55b70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe6940710e0 tx=0x7fe688007cd0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 -- 172.21.15.46:0/3636976020 shutdown_connections 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a498e0 0x556887e560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e55b70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3636976020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 -- 172.21.15.46:0/3636976020 >> 172.21.15.46:0/3636976020 conn(0x556887a5e710 msgr2=0x556887e0f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 -- 172.21.15.46:0/3636976020 shutdown_connections 2024-05-09T03:12:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.102+0000 7fe6a282b740 1 -- 172.21.15.46:0/3636976020 wait complete. 2024-05-09T03:12:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.103+0000 7fe6a282b740 1 -- start start 2024-05-09T03:12:35.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.103+0000 7fe6a282b740 1 Processor -- start 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e35050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a498e0 0x556887e508e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e50e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556887d0c4d0 con 0x556887a41700 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556887e699f0 con 0x556887a498e0 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556887e358a0 con 0x556887a6a5c0 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe6a282b740 10 client.0 _ll_register_callbacks cb 0x556887d390d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe69963b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e35050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe69963b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e35050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46950/0 (socket says 172.21.15.46:46950) 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe69963b640 1 -- 172.21.15.46:0/206268096 learned_addr learned my addr 172.21.15.46:0/206268096 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.104+0000 7fe698e3a640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a498e0 0x556887e508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe699e3c640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e50e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe69963b640 1 -- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 msgr2=0x556887e50e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe69963b640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e50e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe69963b640 1 -- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a498e0 msgr2=0x556887e508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe69963b640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a498e0 0x556887e508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe69963b640 1 -- 172.21.15.46:0/206268096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556887a1b190 con 0x556887a41700 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe699e3c640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e50e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe69963b640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e35050 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe68402ace0 tx=0x7fe684081200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1eb1ef5ded1eadd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe66ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe66ffff640 1 -- 172.21.15.46:0/206268096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684049030 con 0x556887a41700 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe66ffff640 1 -- 172.21.15.46:0/206268096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7fe68408a4f0 con 0x556887a41700 2024-05-09T03:12:35.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe688018070 con 0x556887a41700 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe66ffff640 1 -- 172.21.15.46:0/206268096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fe6840819d0 con 0x556887a41700 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe6a282b740 10 client.6177 fetch_fsmap learned FSMap version 54 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) -- 0x556887b94770 con 0x556887a41700 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.105+0000 7fe66ffff640 1 -- 172.21.15.46:0/206268096 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe688018070 con 0x556887a41700 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe66ffff640 1 -- 172.21.15.46:0/206268096 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 3494+0+0 (secure 0 0 0) 0x7fe684080c30 con 0x556887a41700 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe6a282b740 10 client.6177 fetch_fsmap finished waiting for FSMap version 54 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe6a282b740 10 client.6177 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe6a282b740 20 client.6177 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe6a282b740 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x556887a29190 0x556887e68b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe6a282b740 4 client.6177 mds_command: new command op to 6102 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 --> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556887d80b60 con 0x556887a29190 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe698e3a640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x556887a29190 0x556887e68b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.106+0000 7fe66ffff640 10 client.6177 ms_handle_connect on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.107+0000 7fe698e3a640 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x556887a29190 0x556887e68b80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c406778bf914305 server_cookie=65a8855a9ef7162 in_seq=0 out_seq=0 2024-05-09T03:12:35.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.107+0000 7fe66ffff640 1 -- 172.21.15.46:0/206268096 <== mds.0 v2:172.21.15.46:6834/103238544 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556887d80b60 con 0x556887a29190 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.107+0000 7fe66ffff640 10 client.6177 handle_command_reply: tid=0 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.107+0000 7fe6a282b740 1 client.6177 shutdown 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 msgr2=0x556887e35050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e35050 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fe68402ace0 tx=0x7fe684081200 comp rx=0 tx=0).stop 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 shutdown_connections 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x556887a29190 0x556887e68b80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe66ffff640 0 client.6177 ms_handle_reset on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a6a5c0 0x556887e50e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a498e0 0x556887e508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 --2- 172.21.15.46:0/206268096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e35050 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 >> 172.21.15.46:0/206268096 conn(0x556887a5e710 msgr2=0x556887e09620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 shutdown_connections 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 1 -- 172.21.15.46:0/206268096 wait complete. 2024-05-09T03:12:35.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.108+0000 7fe6a282b740 20 client.6177 trim_cache size 0 max 16384 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.110+0000 7f5f50138640 1 Processor -- start 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.110+0000 7f5f50138640 1 -- start start 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f50138640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5f480b36b0 0x7f5f4809b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f50138640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4809c120 con 0x7f5f480d3010 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f50138640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f480d3420 con 0x7f5f4809bd10 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f50138640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5f480b2980 con 0x7f5f480b36b0 2024-05-09T03:12:35.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f4e6d5640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f480d3010 0x7f5f4806e460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40234/0 (socket says 172.21.15.46:40234) 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f4e6d5640 1 -- 172.21.15.46:0/3062958203 learned_addr learned my addr 172.21.15.46:0/3062958203 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.111+0000 7f5f4ded4640 1 --2- 172.21.15.46:0/3062958203 >> v2:172.21.15.88:3300/0 conn(0x7f5f480b36b0 0x7f5f4809b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2809604521 0 0) 0x7f5f40001490 con 0x7f5f480d3010 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f34003520 con 0x7f5f480d3010 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2849704339 0 0) 0x7f5f38001110 con 0x7f5f4809bd10 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f40001490 con 0x7f5f4809bd10 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ded4640 1 -- 172.21.15.46:0/3062958203 >> v1:172.21.15.88:6789/0 conn(0x7f5f4809bd10 legacy=0x7f5f480d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ded4640 1 -- 172.21.15.46:0/3062958203 >> v1:172.21.15.46:6789/0 conn(0x7f5f480d3010 legacy=0x7f5f4806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.112+0000 7f5f4ded4640 1 -- 172.21.15.46:0/3062958203 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f480d36a0 con 0x7f5f480b36b0 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.113+0000 7f5f4ded4640 1 --2- 172.21.15.46:0/3062958203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 0x7f5f4809b7d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f440097a0 tx=0x7f5f4402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72b71a44db60e69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.113+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f44037070 con 0x7f5f480b36b0 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.113+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f44032440 con 0x7f5f480b36b0 2024-05-09T03:12:35.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.113+0000 7f5f4ced2640 1 -- 172.21.15.46:0/3062958203 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f44032810 con 0x7f5f480b36b0 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 -- 172.21.15.46:0/3062958203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 msgr2=0x7f5f4809b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 --2- 172.21.15.46:0/3062958203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 0x7f5f4809b7d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f440097a0 tx=0x7f5f4402a230 comp rx=0 tx=0).stop 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 -- 172.21.15.46:0/3062958203 shutdown_connections 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 --2- 172.21.15.46:0/3062958203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 0x7f5f4809b7d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 -- 172.21.15.46:0/3062958203 >> 172.21.15.46:0/3062958203 conn(0x7f5f480b7760 msgr2=0x7f5f480b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 -- 172.21.15.46:0/3062958203 shutdown_connections 2024-05-09T03:12:35.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.114+0000 7f5f50138640 1 -- 172.21.15.46:0/3062958203 wait complete. 2024-05-09T03:12:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.116+0000 7f5f50138640 1 Processor -- start 2024-05-09T03:12:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.116+0000 7f5f50138640 1 -- start start 2024-05-09T03:12:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f50138640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f4809bd10 0x7f5f4814fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f50138640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 0x7f5f4815ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f50138640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f480d3010 0x7f5f481f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f50138640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5f480b2bd0 con 0x7f5f480d3010 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f50138640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5f480b3220 con 0x7f5f480b36b0 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f50138640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5f480b2eb0 con 0x7f5f4809bd10 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f4e6d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f480d3010 0x7f5f481f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f4e6d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f480d3010 0x7f5f481f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46954/0 (socket says 172.21.15.46:46954) 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f4e6d5640 1 -- 172.21.15.46:0/626381830 learned_addr learned my addr 172.21.15.46:0/626381830 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f4d6d3640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 0x7f5f4815ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.117+0000 7f5f4ded4640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f4809bd10 0x7f5f4814fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f4e6d5640 1 -- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f4809bd10 msgr2=0x7f5f4814fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f4e6d5640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5f4809bd10 0x7f5f4814fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f4e6d5640 1 -- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 msgr2=0x7f5f4815ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f4e6d5640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f480b36b0 0x7f5f4815ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f4e6d5640 1 -- 172.21.15.46:0/626381830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f48150950 con 0x7f5f480d3010 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f4e6d5640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f480d3010 0x7f5f481f23b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5f40001ed0 tx=0x7f5f4000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40012400 con 0x7f5f480d3010 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f400125a0 con 0x7f5f480d3010 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f50138640 1 -- 172.21.15.46:0/626381830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4815ba70 con 0x7f5f480d3010 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40012990 con 0x7f5f480d3010 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.118+0000 7f5f50138640 1 -- 172.21.15.46:0/626381830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f480d0780 con 0x7f5f480d3010 2024-05-09T03:12:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.119+0000 7f5f50138640 1 -- 172.21.15.46:0/626381830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f5f4815b7e0 con 0x7f5f480d3010 2024-05-09T03:12:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.120+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5f40012b30 con 0x7f5f480d3010 2024-05-09T03:12:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f3e7fc640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f2c0762d0 0x7f5f2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f5f40097da0 con 0x7f5f480d3010 2024-05-09T03:12:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f4ded4640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f2c0762d0 0x7f5f2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+513 (secure 0 0 0) 0x7f5f4809c120 con 0x7f5f480d3010 2024-05-09T03:12:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f50138640 1 -- 172.21.15.46:0/626381830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f5f4815e600 con 0x7f5f480d3010 2024-05-09T03:12:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 7 ==== log(1 entries from seq 554 at 2024-05-09T03:12:34.260647+0000) ==== 308+0+0 (secure 0 0 0) 0x7f5f400962e0 con 0x7f5f480d3010 2024-05-09T03:12:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.121+0000 7f5f4ded4640 1 --2- 172.21.15.46:0/626381830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5f2c0762d0 0x7f5f2c0787e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5f44034000 tx=0x7f5f4402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.141+0000 7fe6a282b740 1 Processor -- start 2024-05-09T03:12:35.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.141+0000 7fe6a282b740 1 -- start start 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.142+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 0x556887e68b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.142+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.142+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887e3d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.142+0000 7fe6a282b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe6941feb30 con 0x556887a29190 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940b0de0 con 0x556887a41700 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe69409dfd0 con 0x556887a498e0 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 0x556887e68b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 0x556887e68b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46960/0 (socket says 172.21.15.46:46960) 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 -- 172.21.15.46:0/3369130435 learned_addr learned my addr 172.21.15.46:0/3369130435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe699e3c640 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887e3d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe698e3a640 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 -- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 msgr2=0x556887e3d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887e3d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 -- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 msgr2=0x556887e690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e690c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 -- 172.21.15.46:0/3369130435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556887a4e3c0 con 0x556887a29190 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe69963b640 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 0x556887e68b80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe684000f80 tx=0x7fe6840a3050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1901a4cb11f51c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe67a7fc640 1 -- 172.21.15.46:0/3369130435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe684049070 con 0x556887a29190 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe67a7fc640 1 -- 172.21.15.46:0/3369130435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6840b6c40 con 0x556887a29190 2024-05-09T03:12:35.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.143+0000 7fe67a7fc640 1 -- 172.21.15.46:0/3369130435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68408a780 con 0x556887a29190 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 -- 172.21.15.46:0/3369130435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 msgr2=0x556887e68b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 0x556887e68b80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe684000f80 tx=0x7fe6840a3050 comp rx=0 tx=0).stop 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 -- 172.21.15.46:0/3369130435 shutdown_connections 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887e3d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a41700 0x556887e690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 --2- 172.21.15.46:0/3369130435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a29190 0x556887e68b80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 -- 172.21.15.46:0/3369130435 >> 172.21.15.46:0/3369130435 conn(0x556887a5e710 msgr2=0x556887e35a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 -- 172.21.15.46:0/3369130435 shutdown_connections 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.144+0000 7fe6a282b740 1 -- 172.21.15.46:0/3369130435 wait complete. 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.145+0000 7fe6a282b740 1 -- start start 2024-05-09T03:12:35.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.145+0000 7fe6a282b740 1 Processor -- start 2024-05-09T03:12:35.149 INFO:teuthology.orchestra.run.smithi046.stdout:{"mds_beacon_grace":"15.000000"} 2024-05-09T03:12:35.150 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-05-09T03:12:35.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a29190 0x556887e57200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e7a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887ed56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556887e71c80 con 0x556887a41700 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe69415f480 con 0x556887a29190 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556887ed6170 con 0x556887a498e0 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe6a282b740 10 client.0 _ll_register_callbacks cb 0x7fe68407d8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e7a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e7a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46966/0 (socket says 172.21.15.46:46966) 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 -- 172.21.15.46:0/2886340016 learned_addr learned my addr 172.21.15.46:0/2886340016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a29190 0x556887e57200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe699e3c640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887ed56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 msgr2=0x556887ed56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556887a498e0 0x556887ed56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a29190 msgr2=0x556887e57200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556887a29190 0x556887e57200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.147+0000 7fe698e3a640 1 -- 172.21.15.46:0/2886340016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556887e0e190 con 0x556887a41700 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe698e3a640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556887a41700 0x556887e7a690 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe688011010 tx=0x7fe688000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3e3b67406645d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe66ffff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe66ffff640 1 -- 172.21.15.46:0/2886340016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe688005c70 con 0x556887a41700 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe66ffff640 1 -- 172.21.15.46:0/2886340016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7fe6880070d0 con 0x556887a41700 2024-05-09T03:12:35.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 1 -- 172.21.15.46:0/2886340016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6840b6c40 con 0x556887a41700 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe66ffff640 1 -- 172.21.15.46:0/2886340016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fe688038050 con 0x556887a41700 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 10 client.6189 fetch_fsmap learned FSMap version 54 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 1 -- 172.21.15.46:0/2886340016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) -- 0x556887e7abd0 con 0x556887a41700 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe66ffff640 1 -- 172.21.15.46:0/2886340016 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6840b6c40 con 0x556887a41700 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe66ffff640 1 -- 172.21.15.46:0/2886340016 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 54) ==== 3494+0+0 (secure 0 0 0) 0x7fe68803b020 con 0x556887a41700 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 10 client.6189 fetch_fsmap finished waiting for FSMap version 54 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 10 client.6189 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 20 client.6189 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 4 client.6189 mds_command: new command op to 6102 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.148+0000 7fe6a282b740 1 -- 172.21.15.46:0/2886340016 --> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556887d7a9d0 con 0x5568879fe5d0 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.149+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.149+0000 7fe66ffff640 10 client.6189 ms_handle_connect on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.149+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=138ed299bd0e043b server_cookie=709e5d2697735c64 in_seq=0 out_seq=0 2024-05-09T03:12:35.156 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:12:35.152+0000 7fda42246640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:12:35.152+0000 7fda42246640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fda42246640 time 2024-05-09T03:12:35.153692+0000 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fda4e64acfc] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.157 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.158 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:12:35.153+0000 7fda42246640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fda42246640 time 2024-05-09T03:12:35.153692+0000 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fda4e64acfc] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.159 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.160 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr:*** Caught signal (Aborted) ** 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7fda42246640 thread_name:MR_Finisher 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fda4dc3e6f0] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fda4dc8b94c] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fda4e64ae6e] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.161 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.162 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:12:35.154+0000 7fda42246640 -1 *** Caught signal (Aborted) ** 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7fda42246640 thread_name:MR_Finisher 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.163 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fda4dc3e6f0] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fda4dc8b94c] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fda4e64ae6e] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.164 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.165 INFO:tasks.ceph.mds.a.smithi046.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.166 INFO:tasks.ceph.mds.a.smithi046.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T03:12:35.166 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.167 INFO:tasks.ceph.mds.a.smithi046.stderr: -238> 2024-05-09T03:12:33.224+0000 7fda4624e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:12:35.167 INFO:tasks.ceph.mds.a.smithi046.stderr: -11> 2024-05-09T03:12:35.152+0000 7fda42246640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:35.167 INFO:tasks.ceph.mds.a.smithi046.stderr: -9> 2024-05-09T03:12:35.152+0000 7fda42246640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:35.167 INFO:tasks.ceph.mds.a.smithi046.stderr: -1> 2024-05-09T03:12:35.153+0000 7fda42246640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fda42246640 time 2024-05-09T03:12:35.153692+0000 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fda4e64acfc] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.168 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.169 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 0> 2024-05-09T03:12:35.154+0000 7fda42246640 -1 *** Caught signal (Aborted) ** 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7fda42246640 thread_name:MR_Finisher 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fda4dc3e6f0] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fda4dc8b94c] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fda4e64ae6e] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.170 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.171 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: -238> 2024-05-09T03:12:33.224+0000 7fda4624e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: -11> 2024-05-09T03:12:35.152+0000 7fda42246640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: -9> 2024-05-09T03:12:35.152+0000 7fda42246640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr: -1> 2024-05-09T03:12:35.153+0000 7fda42246640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fda42246640 time 2024-05-09T03:12:35.153692+0000 2024-05-09T03:12:35.172 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fda4e64acfc] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.173 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.174 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 0> 2024-05-09T03:12:35.154+0000 7fda42246640 -1 *** Caught signal (Aborted) ** 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7fda42246640 thread_name:MR_Finisher 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fda4dc3e6f0] 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7fda4dc8b94c] 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7fda4e64ae6e] 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x555c7cd0290b] 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x555c7cfc8640] 2024-05-09T03:12:35.175 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x3f08ed) [0x555c7cfd98ed] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (CDir::_omap_commit(int)+0x7ef) [0x555c7cfda695] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (CDir::_commit(unsigned long, int)+0x5ba) [0x555c7cfdb2a6] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x11f) [0x555c7cfdb569] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x10bf) [0x555c7cd6ce39] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDLog::try_expire(LogSegment*, int)+0x6f) [0x555c7d0902d7] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDLog::trim_all()+0x85e) [0x555c7d091026] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: ceph-mds(+0x11ebe5) [0x555c7cd07be5] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: ceph-mds(+0x11ed43) [0x555c7cd07d43] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: ceph-mds(+0x15fd0d) [0x555c7cd48d0d] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: ceph-mds(+0xf9525) [0x555c7cce2525] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (MDSInternalContextWrapper::finish(int)+0x12) [0x555c7d07e2ba] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: ceph-mds(+0x49529e) [0x555c7d07e29e] 2024-05-09T03:12:35.176 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (MDSContext::complete(int)+0x7c) [0x555c7d084a14] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (MDSIOContextBase::complete(int)+0x501) [0x555c7d08be11] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: (C_IO_Wrapper::complete(int)+0x3c) [0x555c7d08c0c0] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: (Finisher::finisher_thread_entry()+0x495) [0x7fda4e5d1a59] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 26: ceph-mds(+0x1111cd) [0x555c7ccfa1cd] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 27: (Thread::entry_wrapper()+0x43) [0x7fda4e624ad7] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 28: (Thread::_entry_func(void*)+0xd) [0x7fda4e624af3] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 29: /lib64/libc.so.6(+0x89c02) [0x7fda4dc89c02] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 30: /lib64/libc.so.6(+0x10ec40) [0x7fda4dd0ec40] 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T03:12:35.177 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T03:12:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.225+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.272+0000 7f4021c2b640 1 Processor -- start 2024-05-09T03:12:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.272+0000 7f4021c2b640 1 -- start start 2024-05-09T03:12:35.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f4021c2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c15e280 0x7f401c0b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f4021c2b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 0x7f401c0d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f4021c2b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f401c0d2b20 con 0x7f401c068ad0 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f4021c2b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f401c06ddd0 con 0x7f401c0d21d0 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f4021c2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f401c06d150 con 0x7f401c15e280 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f401affd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c15e280 0x7f401c0b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f401bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 0x7f401c0d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f401b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f401c068ad0 0x7f401c0af1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:42092/0 (socket says 172.21.15.46:42092) 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f401b7fe640 1 -- 172.21.15.46:0/3985612827 learned_addr learned my addr 172.21.15.46:0/3985612827 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.273+0000 7f401bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 0x7f401c0d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:55398/0 (socket says 172.21.15.46:55398) 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401a7fc640 1 -- 172.21.15.46:0/3985612827 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387718274 0 0) 0x7f400c001110 con 0x7f401c068ad0 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401a7fc640 1 -- 172.21.15.46:0/3985612827 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4004003520 con 0x7f401c068ad0 2024-05-09T03:12:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401affd640 1 -- 172.21.15.46:0/3985612827 >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 msgr2=0x7f401c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:35.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401affd640 1 --2- 172.21.15.46:0/3985612827 >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 0x7f401c0d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401affd640 1 -- 172.21.15.46:0/3985612827 >> v1:172.21.15.88:6789/0 conn(0x7f401c068ad0 legacy=0x7f401c0af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401affd640 1 -- 172.21.15.46:0/3985612827 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c06d430 con 0x7f401c15e280 2024-05-09T03:12:35.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.274+0000 7f401bfff640 1 --2- 172.21.15.46:0/3985612827 >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 0x7f401c0d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.275+0000 7f401affd640 1 --2- 172.21.15.46:0/3985612827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c15e280 0x7f401c0b78d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f40100097a0 tx=0x7f401002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=813366b6266d22ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.275+0000 7f401a7fc640 1 -- 172.21.15.46:0/3985612827 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010037070 con 0x7f401c15e280 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.275+0000 7f401a7fc640 1 -- 172.21.15.46:0/3985612827 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4010032440 con 0x7f401c15e280 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.275+0000 7f401a7fc640 1 -- 172.21.15.46:0/3985612827 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010032810 con 0x7f401c15e280 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.275+0000 7f4021c2b640 1 -- 172.21.15.46:0/3985612827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c15e280 msgr2=0x7f401c0b78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.275+0000 7f4021c2b640 1 --2- 172.21.15.46:0/3985612827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c15e280 0x7f401c0b78d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f40100097a0 tx=0x7f401002a180 comp rx=0 tx=0).stop 2024-05-09T03:12:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.276+0000 7f4021c2b640 1 -- 172.21.15.46:0/3985612827 shutdown_connections 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.276+0000 7f4021c2b640 1 --2- 172.21.15.46:0/3985612827 >> v2:172.21.15.88:3300/0 conn(0x7f401c0d21d0 0x7f401c0d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.276+0000 7f4021c2b640 1 --2- 172.21.15.46:0/3985612827 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c15e280 0x7f401c0b78d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.276+0000 7f4021c2b640 1 -- 172.21.15.46:0/3985612827 >> 172.21.15.46:0/3985612827 conn(0x7f401c09b7d0 msgr2=0x7f401c0b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.276+0000 7f4021c2b640 1 -- 172.21.15.46:0/3985612827 shutdown_connections 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.276+0000 7f4021c2b640 1 -- 172.21.15.46:0/3985612827 wait complete. 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.277+0000 7f4021c2b640 1 Processor -- start 2024-05-09T03:12:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.278+0000 7f4021c2b640 1 -- start start 2024-05-09T03:12:35.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f4021c2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c068ad0 0x7f401c154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f4021c2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 0x7f401c155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f4021c2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c15e280 0x7f401c1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f4021c2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f401c0b7e10 con 0x7f401c0d21d0 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f4021c2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f401c06d9f0 con 0x7f401c15e280 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f4021c2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f401c15e690 con 0x7f401c068ad0 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 0x7f401c155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c068ad0 0x7f401c154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 0x7f401c155200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46982/0 (socket says 172.21.15.46:46982) 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 -- 172.21.15.46:0/574604214 learned_addr learned my addr 172.21.15.46:0/574604214 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401bfff640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c15e280 0x7f401c1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 -- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c068ad0 msgr2=0x7f401c154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c068ad0 0x7f401c154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 -- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c15e280 msgr2=0x7f401c1587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c15e280 0x7f401c1587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.279+0000 7f401affd640 1 -- 172.21.15.46:0/574604214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c0b05e0 con 0x7f401c0d21d0 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.280+0000 7f401affd640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 0x7f401c155200 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40100098c0 tx=0x7f40100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.280+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010041020 con 0x7f401c0d21d0 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.280+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f401003d640 con 0x7f401c0d21d0 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.280+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401c0b08c0 con 0x7f401c0d21d0 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.280+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010037040 con 0x7f401c0d21d0 2024-05-09T03:12:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.280+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f401c0b0e80 con 0x7f401c0d21d0 2024-05-09T03:12:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.281+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401c06a270 con 0x7f401c0d21d0 2024-05-09T03:12:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.282+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4010044020 con 0x7f401c0d21d0 2024-05-09T03:12:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.283+0000 7f4003fff640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40080762d0 0x7f40080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.283+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f40100b7c80 con 0x7f401c0d21d0 2024-05-09T03:12:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.283+0000 7f401b7fe640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40080762d0 0x7f40080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:35.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.283+0000 7f401b7fe640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40080762d0 0x7f40080787e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f400c005880 tx=0x7f400c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:35.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.288+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f401004b030 con 0x7f401c0d21d0 2024-05-09T03:12:35.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.426+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.426+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T03:12:35.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.426+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.520+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f401c0b0360 con 0x7f401c0d21d0 2024-05-09T03:12:35.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.520+0000 7f4003fff640 1 -- 172.21.15.46:0/574604214 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5179 (secure 0 0 0) 0x7f401c0b0360 con 0x7f401c0d21d0 2024-05-09T03:12:35.522 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:35.522 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:34.260736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":4}]} 2024-05-09T03:12:35.523 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40080762d0 msgr2=0x7f40080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40080762d0 0x7f40080787e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f400c005880 tx=0x7f400c009040 comp rx=0 tx=0).stop 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 msgr2=0x7f401c155200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 0x7f401c155200 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40100098c0 tx=0x7f40100046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 shutdown_connections 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c15e280 0x7f401c1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40080762d0 0x7f40080787e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f401c0d21d0 0x7f401c155200 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 --2- 172.21.15.46:0/574604214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f401c068ad0 0x7f401c154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 >> 172.21.15.46:0/574604214 conn(0x7f401c09b7d0 msgr2=0x7f401c0cd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:35.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 shutdown_connections 2024-05-09T03:12:35.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.523+0000 7f4021c2b640 1 -- 172.21.15.46:0/574604214 wait complete. 2024-05-09T03:12:35.538 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:12:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.827+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.827+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T03:12:35.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.827+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:35.910 INFO:tasks.ceph.mds.a.smithi046.stderr:daemon-helper: command crashed with signal 6 2024-05-09T03:12:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:35.941+0000 7f5f3e7fc640 1 -- 172.21.15.46:0/626381830 <== mon.0 v2:172.21.15.46:3300/0 8 ==== log(2 entries from seq 555 at 2024-05-09T03:12:34.941035+0000) ==== 634+0+0 (secure 0 0 0) 0x7f5f4005d1b0 con 0x7f5f480d3010 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: cluster: 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: id: 36d71dd3-2926-4712-824e-80cfbafd76c2 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: health: HEALTH_OK 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: services: 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: mgr: x(active, since 3m), standbys: y 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: mds: 1/1 daemons up, 3 standby 2024-05-09T03:12:35.946 INFO:teuthology.orchestra.run.smithi046.stdout: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: data: 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: volumes: 1/1 healthy 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: pools: 3 pools, 97 pgs 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: objects: 29 objects, 600 KiB 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: usage: 234 MiB used, 720 GiB / 720 GiB avail 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: pgs: 97 active+clean 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: io: 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: client: 1.7 KiB/s rd, 426 B/s wr, 1 op/s rd, 0 op/s wr 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:35.947 INFO:teuthology.orchestra.run.smithi046.stdout:2024-05-09T03:12:35.153674+0000 mds.a [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x555c838b9180] 2024-05-09T03:12:36.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:12:36.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:36.628+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:36.628+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T03:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:36.628+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:38.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:38.229+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:38.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:38.229+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T03:12:38.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:38.229+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:40.539 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.668+0000 7fd59f3b2640 1 Processor -- start 2024-05-09T03:12:40.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.668+0000 7fd59f3b2640 1 -- start start 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59f3b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 0x7fd5981613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59f3b2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5980d5280 con 0x7fd5980b7ed0 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59f3b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5980d5560 con 0x7fd5980d4800 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59f3b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd5980b5620 con 0x7fd5980b5210 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59d94f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd5980b7ed0 0x7fd598163e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59916/0 (socket says 172.21.15.46:59916) 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59d94f640 1 -- 172.21.15.46:0/2647586542 learned_addr learned my addr 172.21.15.46:0/2647586542 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920601849 0 0) 0x7fd5980d5280 con 0x7fd5980b7ed0 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 0x7fd5981613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.670+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd584003520 con 0x7fd5980b7ed0 2024-05-09T03:12:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583194921 0 0) 0x7fd5980d5560 con 0x7fd5980d4800 2024-05-09T03:12:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5980d5280 con 0x7fd5980d4800 2024-05-09T03:12:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 316471795 0 0) 0x7fd584003520 con 0x7fd5980b7ed0 2024-05-09T03:12:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 msgr2=0x7fd5981613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 0x7fd5981613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 >> v1:172.21.15.88:6789/0 conn(0x7fd5980d4800 legacy=0x7fd5980b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd598160bc0 con 0x7fd5980b7ed0 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fd590001670 con 0x7fd5980b7ed0 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 >> v1:172.21.15.46:6789/0 conn(0x7fd5980b7ed0 legacy=0x7fd598163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd584003c60 0x7fd5840040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd584004800 0x7fd584004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 0x7fd58400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5980b5620 con 0x7fd584003c60 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd584005520 con 0x7fd584004800 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd58400bf00 con 0x7fd5980b5210 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd584003c60 0x7fd5840040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59c94d640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd584004800 0x7fd584004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d94f640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 0x7fd58400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d14e640 1 -- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 msgr2=0x7fd58400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:40.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5980b5210 0x7fd58400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d14e640 1 -- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd584004800 msgr2=0x7fd584004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd584004800 0x7fd584004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.671+0000 7fd59d14e640 1 -- 172.21.15.46:0/2647586542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd584003520 con 0x7fd584003c60 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.672+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd584003c60 0x7fd5840040b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd594009fc0 tx=0x7fd584005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ab24bb6544e373 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.672+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594034020 con 0x7fd584003c60 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.672+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5940305f0 con 0x7fd584003c60 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.672+0000 7fd58ffff640 1 -- 172.21.15.46:0/2647586542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd594009c10 con 0x7fd584003c60 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2647586542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd584003c60 msgr2=0x7fd5840040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2647586542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd584003c60 0x7fd5840040b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd594009fc0 tx=0x7fd584005830 comp rx=0 tx=0).stop 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59d94f640 1 -- 172.21.15.46:0/2647586542 reap_dead start 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2647586542 shutdown_connections 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2647586542 >> 172.21.15.46:0/2647586542 conn(0x7fd5980d29a0 msgr2=0x7fd5980d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2647586542 shutdown_connections 2024-05-09T03:12:40.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.673+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2647586542 wait complete. 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.675+0000 7fd59f3b2640 1 Processor -- start 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.675+0000 7fd59f3b2640 1 -- start start 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59f3b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd584003c60 0x7fd5981f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59f3b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 0x7fd5981f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59f3b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5981f7960 0x7fd5980b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59f3b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5980d0ae0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59f3b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd5981609a0 con 0x7fd5981f7960 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59f3b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd5980d4c10 con 0x7fd584003c60 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59c94d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 0x7fd5981f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59c94d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 0x7fd5981f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43904/0 (socket says 172.21.15.46:43904) 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59c94d640 1 -- 172.21.15.46:0/2279784692 learned_addr learned my addr 172.21.15.46:0/2279784692 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd584003c60 0x7fd5981f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.676+0000 7fd59d94f640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5981f7960 0x7fd5980b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59c94d640 1 -- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd584003c60 msgr2=0x7fd5981f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59c94d640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd584003c60 0x7fd5981f6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59c94d640 1 -- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5981f7960 msgr2=0x7fd5980b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59c94d640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5981f7960 0x7fd5980b6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59c94d640 1 -- 172.21.15.46:0/2279784692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5981f7d70 con 0x7fd5980ce3e0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59c94d640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 0x7fd5981f7420 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd58800f7f0 tx=0x7fd58800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58800fea0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd588018070 con 0x7fd5980ce3e0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58800d730 con 0x7fd5980ce3e0 2024-05-09T03:12:40.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5980cfbf0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.677+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5980cf910 con 0x7fd5980ce3e0 2024-05-09T03:12:40.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.678+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd59806a270 con 0x7fd5980ce3e0 2024-05-09T03:12:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.679+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd58800d8d0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.679+0000 7fd58dffb640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd57c076300 0x7fd57c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.680+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7fd5880973e0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.680+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd57c076300 0x7fd57c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:40.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.680+0000 7fd59d14e640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd57c076300 0x7fd57c078810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd594035000 tx=0x7fd59402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:40.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.684+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd5880605e0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.916+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5980d40d0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.917+0000 7fd58dffb640 1 -- 172.21.15.46:0/2279784692 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5179 (secure 0 0 0) 0x7fd5880648e0 con 0x7fd5980ce3e0 2024-05-09T03:12:40.919 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:40.919 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:34.260736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":4}]} 2024-05-09T03:12:40.919 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-05-09T03:12:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd57c076300 msgr2=0x7fd57c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd57c076300 0x7fd57c078810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd594035000 tx=0x7fd59402f000 comp rx=0 tx=0).stop 2024-05-09T03:12:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 msgr2=0x7fd5981f7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 0x7fd5981f7420 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd58800f7f0 tx=0x7fd58800fcb0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 shutdown_connections 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5981f7960 0x7fd5980b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd57c076300 0x7fd57c078810 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5980ce3e0 0x7fd5981f7420 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 --2- 172.21.15.46:0/2279784692 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd584003c60 0x7fd5981f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 >> 172.21.15.46:0/2279784692 conn(0x7fd5980d29a0 msgr2=0x7fd598163df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 shutdown_connections 2024-05-09T03:12:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:40.919+0000 7fd59f3b2640 1 -- 172.21.15.46:0/2279784692 wait complete. 2024-05-09T03:12:40.933 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:12:41.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-05-09T03:12:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:41.433+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:41.433+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-05-09T03:12:41.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:41.433+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:45.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.063+0000 7f535055e640 1 Processor -- start 2024-05-09T03:12:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.063+0000 7f535055e640 1 -- start start 2024-05-09T03:12:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f535055e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53480d57a0 0x7f5348160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f535055e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53480b7ef0 0x7f53480b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f535055e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53480d49e0 con 0x7f53480b4af0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f535055e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f53480d5bb0 con 0x7f53480d57a0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f535055e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53480d4cc0 con 0x7f53480b7ef0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f534e2fa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53480d57a0 0x7f5348160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f534daf9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53480b4af0 0x7f53480b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37502/0 (socket says 172.21.15.46:37502) 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f534daf9640 1 -- 172.21.15.46:0/4010059972 learned_addr learned my addr 172.21.15.46:0/4010059972 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f534eafb640 1 --2- 172.21.15.46:0/4010059972 >> v2:172.21.15.88:3300/0 conn(0x7f53480b7ef0 0x7f53480b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.065+0000 7f534d2f8640 1 -- 172.21.15.46:0/4010059972 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474341501 0 0) 0x7f53480d49e0 con 0x7f53480b4af0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534d2f8640 1 -- 172.21.15.46:0/4010059972 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5334003520 con 0x7f53480b4af0 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534e2fa640 1 -- 172.21.15.46:0/4010059972 >> v2:172.21.15.88:3300/0 conn(0x7f53480b7ef0 msgr2=0x7f53480b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534e2fa640 1 --2- 172.21.15.46:0/4010059972 >> v2:172.21.15.88:3300/0 conn(0x7f53480b7ef0 0x7f53480b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534e2fa640 1 -- 172.21.15.46:0/4010059972 >> v1:172.21.15.46:6789/0 conn(0x7f53480b4af0 legacy=0x7f53480b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534e2fa640 1 -- 172.21.15.46:0/4010059972 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53480d4fa0 con 0x7f53480d57a0 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534e2fa640 1 --2- 172.21.15.46:0/4010059972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f5348160610 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f53481640f0 tx=0x7f534402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce674b37055c471b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.066+0000 7f534d2f8640 1 -- 172.21.15.46:0/4010059972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344037070 con 0x7f53480d57a0 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f534d2f8640 1 -- 172.21.15.46:0/4010059972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5344032440 con 0x7f53480d57a0 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f534d2f8640 1 -- 172.21.15.46:0/4010059972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440327f0 con 0x7f53480d57a0 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 -- 172.21.15.46:0/4010059972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 msgr2=0x7f5348160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 --2- 172.21.15.46:0/4010059972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f5348160610 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f53481640f0 tx=0x7f534402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 -- 172.21.15.46:0/4010059972 shutdown_connections 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 --2- 172.21.15.46:0/4010059972 >> v2:172.21.15.88:3300/0 conn(0x7f53480b7ef0 0x7f53480b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 --2- 172.21.15.46:0/4010059972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f5348160610 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 -- 172.21.15.46:0/4010059972 >> 172.21.15.46:0/4010059972 conn(0x7f53480743e0 msgr2=0x7f53480d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 -- 172.21.15.46:0/4010059972 shutdown_connections 2024-05-09T03:12:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.067+0000 7f535055e640 1 -- 172.21.15.46:0/4010059972 wait complete. 2024-05-09T03:12:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.069+0000 7f535055e640 1 Processor -- start 2024-05-09T03:12:46.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.069+0000 7f535055e640 1 -- start start 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f535055e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53480b4af0 0x7f53481e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f535055e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f534815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f535055e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5348159ab0 0x7f534815cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f535055e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53480d1920 con 0x7f53480d57a0 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f535055e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53480d15b0 con 0x7f53480b4af0 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f535055e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53481e8f10 con 0x7f5348159ab0 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f534815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f534815a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53966/0 (socket says 172.21.15.46:53966) 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 -- 172.21.15.46:0/3978873558 learned_addr learned my addr 172.21.15.46:0/3978873558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534e2fa640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53480b4af0 0x7f53481e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534eafb640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5348159ab0 0x7f534815cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 -- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5348159ab0 msgr2=0x7f534815cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5348159ab0 0x7f534815cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 -- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53480b4af0 msgr2=0x7f53481e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53480b4af0 0x7f53481e8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 -- 172.21.15.46:0/3978873558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534815c120 con 0x7f53480d57a0 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534e2fa640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53480b4af0 0x7f53481e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f534daf9640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f534815a3f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f533800c410 tx=0x7f533800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.071+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338017070 con 0x7f53480d57a0 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.072+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5338005ce0 con 0x7f53480d57a0 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.072+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534815c400 con 0x7f53480d57a0 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.072+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380072d0 con 0x7f53480d57a0 2024-05-09T03:12:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.072+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534815c9c0 con 0x7f53480d57a0 2024-05-09T03:12:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.073+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534806a270 con 0x7f53480d57a0 2024-05-09T03:12:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.073+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f533801a070 con 0x7f53480d57a0 2024-05-09T03:12:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.074+0000 7f533effd640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f532c0762d0 0x7f532c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.074+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f53380969a0 con 0x7f53480d57a0 2024-05-09T03:12:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.074+0000 7f534e2fa640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f532c0762d0 0x7f532c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.075+0000 7f534e2fa640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f532c0762d0 0x7f532c0787e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5344034000 tx=0x7f534402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.079+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f533805fbc0 con 0x7f53480d57a0 2024-05-09T03:12:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.309+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f534815ef50 con 0x7f53480d57a0 2024-05-09T03:12:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.310+0000 7f533effd640 1 -- 172.21.15.46:0/3978873558 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5179 (secure 0 0 0) 0x7f5338063ec0 con 0x7f53480d57a0 2024-05-09T03:12:46.310 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:46.311 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":54,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:34.260736+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":4}]} 2024-05-09T03:12:46.311 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 54 2024-05-09T03:12:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f532c0762d0 msgr2=0x7f532c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f532c0762d0 0x7f532c0787e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5344034000 tx=0x7f534402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 msgr2=0x7f534815a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f534815a3f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f533800c410 tx=0x7f533800c8d0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 shutdown_connections 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5348159ab0 0x7f534815cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f532c0762d0 0x7f532c0787e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53480d57a0 0x7f534815a3f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 --2- 172.21.15.46:0/3978873558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53480b4af0 0x7f53481e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 >> 172.21.15.46:0/3978873558 conn(0x7f53480743e0 msgr2=0x7f53480b79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 shutdown_connections 2024-05-09T03:12:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:46.312+0000 7f535055e640 1 -- 172.21.15.46:0/3978873558 wait complete. 2024-05-09T03:12:46.324 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:12:47.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-05-09T03:12:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:47.840+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.46:6834/103238544 2024-05-09T03:12:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:47.840+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-05-09T03:12:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:47.840+0000 7fe66ffff640 1 client.6189 ms_handle_refused on v2:172.21.15.46:6834/103238544 2024-05-09T03:12:51.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:51.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.454+0000 7f95f3bb2640 1 Processor -- start 2024-05-09T03:12:51.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.455+0000 7f95f3bb2640 1 -- start start 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f3bb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95ec1615e0 0x7f95ec15dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f3bb2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ec160b20 con 0x7f95ec0d2480 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f3bb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ec0d15f0 con 0x7f95ec164c70 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f3bb2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95ec15e520 con 0x7f95ec1615e0 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f114d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95ec1615e0 0x7f95ec15dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f214f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95ec0d2480 0x7f95ec0d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37508/0 (socket says 172.21.15.46:37508) 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f214f640 1 -- 172.21.15.46:0/3590221177 learned_addr learned my addr 172.21.15.46:0/3590221177 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:51.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.456+0000 7f95f114d640 1 -- 172.21.15.46:0/3590221177 >> v1:172.21.15.88:6789/0 conn(0x7f95ec164c70 legacy=0x7f95ec0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:51.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.457+0000 7f95f114d640 1 -- 172.21.15.46:0/3590221177 >> v1:172.21.15.46:6789/0 conn(0x7f95ec0d2480 legacy=0x7f95ec0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.457+0000 7f95f114d640 1 -- 172.21.15.46:0/3590221177 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec0d18d0 con 0x7f95ec1615e0 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.457+0000 7f95f114d640 1 --2- 172.21.15.46:0/3590221177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec15dfe0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0097a0 tx=0x7f95dc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ae48624b6ba7c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.457+0000 7f95f094c640 1 -- 172.21.15.46:0/3590221177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc037070 con 0x7f95ec1615e0 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.457+0000 7f95f094c640 1 -- 172.21.15.46:0/3590221177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95dc02ae50 con 0x7f95ec1615e0 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.457+0000 7f95f094c640 1 -- 172.21.15.46:0/3590221177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc032660 con 0x7f95ec1615e0 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 -- 172.21.15.46:0/3590221177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 msgr2=0x7f95ec15dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/3590221177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec15dfe0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0097a0 tx=0x7f95dc02a100 comp rx=0 tx=0).stop 2024-05-09T03:12:51.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 -- 172.21.15.46:0/3590221177 shutdown_connections 2024-05-09T03:12:51.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/3590221177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec15dfe0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 -- 172.21.15.46:0/3590221177 >> 172.21.15.46:0/3590221177 conn(0x7f95ec073db0 msgr2=0x7f95ec0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:51.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 -- 172.21.15.46:0/3590221177 shutdown_connections 2024-05-09T03:12:51.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.458+0000 7f95f3bb2640 1 -- 172.21.15.46:0/3590221177 wait complete. 2024-05-09T03:12:51.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.460+0000 7f95f3bb2640 1 Processor -- start 2024-05-09T03:12:51.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.460+0000 7f95f3bb2640 1 -- start start 2024-05-09T03:12:51.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.461+0000 7f95f3bb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d2480 0x7f95ec153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.461+0000 7f95f3bb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.461+0000 7f95f3bb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec164c70 0x7f95ec1f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.461+0000 7f95f3bb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec1619f0 con 0x7f95ec1615e0 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.461+0000 7f95f3bb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec161150 con 0x7f95ec0d2480 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.461+0000 7f95f3bb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec0d45c0 con 0x7f95ec164c70 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec144890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53982/0 (socket says 172.21.15.46:53982) 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 -- 172.21.15.46:0/4003847081 learned_addr learned my addr 172.21.15.46:0/4003847081 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f194e640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d2480 0x7f95ec153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f214f640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec164c70 0x7f95ec1f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 -- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec164c70 msgr2=0x7f95ec1f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec164c70 0x7f95ec1f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 -- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d2480 msgr2=0x7f95ec153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d2480 0x7f95ec153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 -- 172.21.15.46:0/4003847081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec1455c0 con 0x7f95ec1615e0 2024-05-09T03:12:51.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f214f640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec164c70 0x7f95ec1f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:51.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f114d640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec144890 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f95dc000c00 tx=0x7f95dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:51.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc042070 con 0x7f95ec1615e0 2024-05-09T03:12:51.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95dc03e670 con 0x7f95ec1615e0 2024-05-09T03:12:51.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ec161d60 con 0x7f95ec1615e0 2024-05-09T03:12:51.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc037040 con 0x7f95ec1615e0 2024-05-09T03:12:51.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.462+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ec162320 con 0x7f95ec1615e0 2024-05-09T03:12:51.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.463+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ec08cfd0 con 0x7f95ec1615e0 2024-05-09T03:12:51.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.464+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95dc045070 con 0x7f95ec1615e0 2024-05-09T03:12:51.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.465+0000 7f95e27fc640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d00762d0 0x7f95d00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.465+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f95dc0b7b70 con 0x7f95ec1615e0 2024-05-09T03:12:51.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.465+0000 7f95f194e640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d00762d0 0x7f95d00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.466+0000 7f95f194e640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d00762d0 0x7f95d00787e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f95e8006fd0 tx=0x7f95e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:51.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.470+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95ec08cfd0 con 0x7f95ec1615e0 2024-05-09T03:12:51.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.707+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95ec06a270 con 0x7f95ec1615e0 2024-05-09T03:12:51.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.707+0000 7f95e27fc640 1 -- 172.21.15.46:0/4003847081 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5227 (secure 0 0 0) 0x7f95dc0813e0 con 0x7f95ec1615e0 2024-05-09T03:12:51.708 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:51.708 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5865,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:49.933827+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6835/103238544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":103238544},{"type":"v1","addr":"172.21.15.46:6835","nonce":103238544}]},"join_fscid":-1,"laggy_since":"2024-05-09T03:12:49.933827+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6102,"qdb_cluster":[6102]},"id":4}]} 2024-05-09T03:12:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 55 2024-05-09T03:12:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d00762d0 msgr2=0x7f95d00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d00762d0 0x7f95d00787e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f95e8006fd0 tx=0x7f95e8008040 comp rx=0 tx=0).stop 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 msgr2=0x7f95ec144890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec144890 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f95dc000c00 tx=0x7f95dc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 shutdown_connections 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec164c70 0x7f95ec1f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d00762d0 0x7f95d00787e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec1615e0 0x7f95ec144890 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 --2- 172.21.15.46:0/4003847081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d2480 0x7f95ec153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 >> 172.21.15.46:0/4003847081 conn(0x7f95ec073db0 msgr2=0x7f95ec0baad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 shutdown_connections 2024-05-09T03:12:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.710+0000 7f95f3bb2640 1 -- 172.21.15.46:0/4003847081 wait complete. 2024-05-09T03:12:51.722 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-05-09T03:12:51.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-05-09T03:12:51.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.940+0000 7f8f7f166640 1 Processor -- start 2024-05-09T03:12:51.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.940+0000 7f8f7f166640 1 -- start start 2024-05-09T03:12:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7f166640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f780d21a0 0x7f8f780d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7f166640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8f780d2af0 0x7f8f781647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7f166640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f780b90f0 con 0x7f8f78164ce0 2024-05-09T03:12:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7f166640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8f780d0360 con 0x7f8f780d2af0 2024-05-09T03:12:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7f166640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8f780b93a0 con 0x7f8f780d21a0 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8f780d2af0 0x7f8f781647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7d703640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8f78164ce0 0x7f8f7806e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:37518/0 (socket says 172.21.15.46:37518) 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7d703640 1 -- 172.21.15.46:0/1014308482 learned_addr learned my addr 172.21.15.46:0/1014308482 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f7cf02640 1 --2- 172.21.15.46:0/1014308482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f780d21a0 0x7f8f780d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f77fff640 1 -- 172.21.15.46:0/1014308482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f780d21a0 msgr2=0x7f8f780d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f77fff640 1 --2- 172.21.15.46:0/1014308482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f780d21a0 0x7f8f780d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f77fff640 1 -- 172.21.15.46:0/1014308482 >> v1:172.21.15.46:6789/0 conn(0x7f8f78164ce0 legacy=0x7f8f7806e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f77fff640 1 -- 172.21.15.46:0/1014308482 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f780d10b0 con 0x7f8f780d2af0 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.942+0000 7f8f777fe640 1 -- 172.21.15.46:0/1014308482 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2191336961 0 0) 0x7f8f780b90f0 con 0x7f8f78164ce0 2024-05-09T03:12:51.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.943+0000 7f8f77fff640 1 --2- 172.21.15.46:0/1014308482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f781647a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0097a0 tx=0x7f8f6c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8860adcbe4e3f867 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.943+0000 7f8f777fe640 1 -- 172.21.15.46:0/1014308482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c037070 con 0x7f8f780d2af0 2024-05-09T03:12:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.943+0000 7f8f777fe640 1 -- 172.21.15.46:0/1014308482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f6c02ae50 con 0x7f8f780d2af0 2024-05-09T03:12:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.943+0000 7f8f777fe640 1 -- 172.21.15.46:0/1014308482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c032660 con 0x7f8f780d2af0 2024-05-09T03:12:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 -- 172.21.15.46:0/1014308482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 msgr2=0x7f8f781647a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 --2- 172.21.15.46:0/1014308482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f781647a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c0097a0 tx=0x7f8f6c02a100 comp rx=0 tx=0).stop 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 -- 172.21.15.46:0/1014308482 shutdown_connections 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 --2- 172.21.15.46:0/1014308482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f781647a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 --2- 172.21.15.46:0/1014308482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f780d21a0 0x7f8f780d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 -- 172.21.15.46:0/1014308482 >> 172.21.15.46:0/1014308482 conn(0x7f8f780d04f0 msgr2=0x7f8f780d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 -- 172.21.15.46:0/1014308482 shutdown_connections 2024-05-09T03:12:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.944+0000 7f8f7f166640 1 -- 172.21.15.46:0/1014308482 wait complete. 2024-05-09T03:12:51.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.946+0000 7f8f7f166640 1 Processor -- start 2024-05-09T03:12:51.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.946+0000 7f8f7f166640 1 -- start start 2024-05-09T03:12:51.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f7f166640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f780d21a0 0x7f8f7815f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f7f166640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f780ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f7f166640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f78164ce0 0x7f8f7806e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f7f166640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780d1bf0 con 0x7f8f780d2af0 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f7f166640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780d1a70 con 0x7f8f780d21a0 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f7f166640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8f780cf8c0 con 0x7f8f78164ce0 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f780ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f780ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53990/0 (socket says 172.21.15.46:53990) 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.947+0000 7f8f77fff640 1 -- 172.21.15.46:0/4248649002 learned_addr learned my addr 172.21.15.46:0/4248649002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f7cf02640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f780d21a0 0x7f8f7815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f7d703640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f78164ce0 0x7f8f7806e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f77fff640 1 -- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f78164ce0 msgr2=0x7f8f7806e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f77fff640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f78164ce0 0x7f8f7806e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f77fff640 1 -- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f780d21a0 msgr2=0x7f8f7815f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f77fff640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f780d21a0 0x7f8f7815f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f77fff640 1 -- 172.21.15.46:0/4248649002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f780c77f0 con 0x7f8f780d2af0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f7cf02640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f780d21a0 0x7f8f7815f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f77fff640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f780ccef0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c000c00 tx=0x7f8f6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c041070 con 0x7f8f780d2af0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f6c03c460 con 0x7f8f780d2af0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f780c7ad0 con 0x7f8f780d2af0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c037040 con 0x7f8f780d2af0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.948+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f780c8090 con 0x7f8f780d2af0 2024-05-09T03:12:51.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.949+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f7806a270 con 0x7f8f780d2af0 2024-05-09T03:12:51.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.950+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8f6c044070 con 0x7f8f780d2af0 2024-05-09T03:12:51.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.950+0000 7f8f757fa640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f640762d0 0x7f8f640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:51.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.951+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f8f6c0b7bc0 con 0x7f8f780d2af0 2024-05-09T03:12:51.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.951+0000 7f8f7cf02640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f640762d0 0x7f8f640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:51.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.951+0000 7f8f7cf02640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f640762d0 0x7f8f640787e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8f78074180 tx=0x7f8f68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:51.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:51.955+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8f6c081390 con 0x7f8f780d2af0 2024-05-09T03:12:52.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.149+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) -- 0x7f8f780d26b0 con 0x7f8f780d2af0 2024-05-09T03:12:52.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.152+0000 7f8f757fa640 1 -- 172.21.15.46:0/4248649002 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7f8f6c080c90 con 0x7f8f780d2af0 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.154+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f640762d0 msgr2=0x7f8f640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.154+0000 7f8f7f166640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f640762d0 0x7f8f640787e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8f78074180 tx=0x7f8f68008040 comp rx=0 tx=0).stop 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 msgr2=0x7f8f780ccef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f780ccef0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c000c00 tx=0x7f8f6c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 shutdown_connections 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f78164ce0 0x7f8f7806e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f640762d0 0x7f8f640787e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f780d2af0 0x7f8f780ccef0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 --2- 172.21.15.46:0/4248649002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f780d21a0 0x7f8f7815f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 >> 172.21.15.46:0/4248649002 conn(0x7f8f780d04f0 msgr2=0x7f8f780d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 shutdown_connections 2024-05-09T03:12:52.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.155+0000 7f8f7f166640 1 -- 172.21.15.46:0/4248649002 wait complete. 2024-05-09T03:12:52.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 false 2024-05-09T03:12:52.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.282+0000 7f538c6a8640 1 Processor -- start 2024-05-09T03:12:52.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.282+0000 7f538c6a8640 1 -- start start 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538c6a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53840d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538c6a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53840d11f0 0x7f53840b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538c6a8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53840b8050 0x7f5384164020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538c6a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f538406e760 con 0x7f53840d11f0 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538c6a8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53840b8460 con 0x7f53840b8050 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538c6a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53840d5740 con 0x7f53840b50b0 2024-05-09T03:12:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53840d11f0 0x7f53840b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53840d11f0 0x7f53840b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54006/0 (socket says 172.21.15.46:54006) 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 -- 172.21.15.46:0/2399990383 learned_addr learned my addr 172.21.15.46:0/2399990383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538ac45640 1 --2- 172.21.15.46:0/2399990383 >> v2:172.21.15.88:3300/0 conn(0x7f53840b8050 0x7f5384164020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f538a444640 1 --2- 172.21.15.46:0/2399990383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53840d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 -- 172.21.15.46:0/2399990383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 msgr2=0x7f53840d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 --2- 172.21.15.46:0/2399990383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53840d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 -- 172.21.15.46:0/2399990383 >> v2:172.21.15.88:3300/0 conn(0x7f53840b8050 msgr2=0x7f5384164020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 --2- 172.21.15.46:0/2399990383 >> v2:172.21.15.88:3300/0 conn(0x7f53840b8050 0x7f5384164020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.284+0000 7f5389c43640 1 -- 172.21.15.46:0/2399990383 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538406de30 con 0x7f53840d11f0 2024-05-09T03:12:52.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.285+0000 7f5389c43640 1 --2- 172.21.15.46:0/2399990383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f53840b7b10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5380009870 tx=0x7f538002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d909108e602ddc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:52.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.285+0000 7f5389442640 1 -- 172.21.15.46:0/2399990383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380037070 con 0x7f53840d11f0 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.285+0000 7f5389442640 1 -- 172.21.15.46:0/2399990383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5380032440 con 0x7f53840d11f0 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.285+0000 7f5389442640 1 -- 172.21.15.46:0/2399990383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380032810 con 0x7f53840d11f0 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 -- 172.21.15.46:0/2399990383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 msgr2=0x7f53840b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2399990383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f53840b7b10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f5380009870 tx=0x7f538002a210 comp rx=0 tx=0).stop 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 -- 172.21.15.46:0/2399990383 shutdown_connections 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2399990383 >> v2:172.21.15.88:3300/0 conn(0x7f53840b8050 0x7f5384164020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2399990383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f53840b7b10 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2399990383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53840d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 -- 172.21.15.46:0/2399990383 >> 172.21.15.46:0/2399990383 conn(0x7f53840d2b80 msgr2=0x7f53840d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 -- 172.21.15.46:0/2399990383 shutdown_connections 2024-05-09T03:12:52.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.286+0000 7f538c6a8640 1 -- 172.21.15.46:0/2399990383 wait complete. 2024-05-09T03:12:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.288+0000 7f538c6a8640 1 Processor -- start 2024-05-09T03:12:52.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.288+0000 7f538c6a8640 1 -- start start 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538c6a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53841f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538c6a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53840b8050 0x7f53840cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538c6a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f5384157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538c6a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53840b54c0 con 0x7f53840d11f0 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538c6a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f538406e3f0 con 0x7f53840b8050 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538c6a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f538406ebf0 con 0x7f53840b50b0 2024-05-09T03:12:52.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f5384157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f5384157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54022/0 (socket says 172.21.15.46:54022) 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 -- 172.21.15.46:0/2743660254 learned_addr learned my addr 172.21.15.46:0/2743660254 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f5389c43640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53840b8050 0x7f53840cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538a444640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53841f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 -- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 msgr2=0x7f53841f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53841f9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 -- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53840b8050 msgr2=0x7f53840cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:52.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53840b8050 0x7f53840cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538ac45640 1 -- 172.21.15.46:0/2743660254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53841fa0b0 con 0x7f53840d11f0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.289+0000 7f538a444640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53841f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.290+0000 7f538ac45640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f5384157eb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f537c007d80 tx=0x7f537c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.290+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c014420 con 0x7f53840d11f0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.290+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537c019070 con 0x7f53840d11f0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.290+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538415dd50 con 0x7f53840d11f0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.290+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c014800 con 0x7f53840d11f0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.290+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538415e310 con 0x7f53840d11f0 2024-05-09T03:12:52.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.291+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53840d0750 con 0x7f53840d11f0 2024-05-09T03:12:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.292+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f537c0149a0 con 0x7f53840d11f0 2024-05-09T03:12:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.292+0000 7f5376ffd640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5368076300 0x7f5368078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.292+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5795+0+0 (secure 0 0 0) 0x7f537c097d80 con 0x7f53840d11f0 2024-05-09T03:12:52.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.293+0000 7f538a444640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5368076300 0x7f5368078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:52.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.293+0000 7f538a444640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5368076300 0x7f5368078810 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f537800ba50 tx=0x7f5378008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:52.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.297+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f537c061680 con 0x7f53840d11f0 2024-05-09T03:12:52.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:52.564+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"} v 0) -- 0x7f538406a270 con 0x7f53840d11f0 2024-05-09T03:12:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.155+0000 7f5376ffd640 1 -- 172.21.15.46:0/2743660254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"}]=0 unfreezing mds.6102 v56) ==== 115+0+0 (secure 0 0 0) 0x7f537c060f80 con 0x7f53840d11f0 2024-05-09T03:12:53.156 INFO:teuthology.orchestra.run.smithi046.stderr:unfreezing mds.6102 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5368076300 msgr2=0x7f5368078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5368076300 0x7f5368078810 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f537800ba50 tx=0x7f5378008040 comp rx=0 tx=0).stop 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 msgr2=0x7f5384157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f5384157eb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f537c007d80 tx=0x7f537c00e4b0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 shutdown_connections 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5368076300 0x7f5368078810 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53840d11f0 0x7f5384157eb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53840b8050 0x7f53840cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 --2- 172.21.15.46:0/2743660254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53840b50b0 0x7f53841f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 >> 172.21.15.46:0/2743660254 conn(0x7f53840d2b80 msgr2=0x7f5384163d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 shutdown_connections 2024-05-09T03:12:53.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.158+0000 7f538c6a8640 1 -- 172.21.15.46:0/2743660254 wait complete. 2024-05-09T03:12:53.177 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -n kernel.core_pattern 2024-05-09T03:12:53.202 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T03:12:53.204 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:12:53.204 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-05-09T03:12:53.275 INFO:teuthology.orchestra.run.smithi046.stdout:1715224355.53152.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/proc/self/exe', platform: 'x86_64' 2024-05-09T03:12:53.277 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1715224355.53152.core'] 2024-05-09T03:12:53.277 INFO:tasks.cephfs.cephfs_test_case:Found core file 1715224355.53152.core, deleting it 2024-05-09T03:12:53.277 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1715224355.53152.core 2024-05-09T03:12:53.353 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:12:53.353 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:12:53.353 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:12:53.353 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:12:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:12:53.354 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-05-09T03:12:53.355 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:12:53.355 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:12:53.396 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:12:53.396 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:53.486 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:12:53.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.514+0000 7f9d51576640 1 Processor -- start 2024-05-09T03:12:53.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.514+0000 7f9d51576640 1 -- start start 2024-05-09T03:12:53.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d51576640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d51576640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d4c0d4800 0x7f9d4c0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d51576640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d4c0b77c0 0x7f9d4c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d51576640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4c0b5620 con 0x7f9d4c0b77c0 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d51576640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4c0b8110 con 0x7f9d4c0d4800 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d51576640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4c0d5280 con 0x7f9d4c0b5210 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d4c0b77c0 0x7f9d4c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9d4c0b77c0 0x7f9d4c0b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54038/0 (socket says 172.21.15.46:54038) 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/3905465378 learned_addr learned my addr 172.21.15.46:0/3905465378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4affd640 1 --2- 172.21.15.46:0/3905465378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4a7fc640 1 --2- 172.21.15.46:0/3905465378 >> v2:172.21.15.88:3300/0 conn(0x7f9d4c0d4800 0x7f9d4c0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/3905465378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 msgr2=0x7f9d4c1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 --2- 172.21.15.46:0/3905465378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/3905465378 >> v2:172.21.15.88:3300/0 conn(0x7f9d4c0d4800 msgr2=0x7f9d4c0d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 --2- 172.21.15.46:0/3905465378 >> v2:172.21.15.88:3300/0 conn(0x7f9d4c0d4800 0x7f9d4c0d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/3905465378 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c0d5560 con 0x7f9d4c0b77c0 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.516+0000 7f9d4a7fc640 1 --2- 172.21.15.46:0/3905465378 >> v2:172.21.15.88:3300/0 conn(0x7f9d4c0d4800 0x7f9d4c0d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.517+0000 7f9d4b7fe640 1 --2- 172.21.15.46:0/3905465378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0b7bd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c06da50 tx=0x7f9d4402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc769e3c6d46259e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.517+0000 7f9d49ffb640 1 -- 172.21.15.46:0/3905465378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44036070 con 0x7f9d4c0b77c0 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.517+0000 7f9d49ffb640 1 -- 172.21.15.46:0/3905465378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d44031600 con 0x7f9d4c0b77c0 2024-05-09T03:12:53.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.517+0000 7f9d49ffb640 1 -- 172.21.15.46:0/3905465378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d440319d0 con 0x7f9d4c0b77c0 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 -- 172.21.15.46:0/3905465378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0b77c0 msgr2=0x7f9d4c0b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 --2- 172.21.15.46:0/3905465378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0b7bd0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c06da50 tx=0x7f9d4402a610 comp rx=0 tx=0).stop 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 -- 172.21.15.46:0/3905465378 shutdown_connections 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 --2- 172.21.15.46:0/3905465378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0b7bd0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 --2- 172.21.15.46:0/3905465378 >> v2:172.21.15.88:3300/0 conn(0x7f9d4c0d4800 0x7f9d4c0d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 --2- 172.21.15.46:0/3905465378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 -- 172.21.15.46:0/3905465378 >> 172.21.15.46:0/3905465378 conn(0x7f9d4c0d29a0 msgr2=0x7f9d4c0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 -- 172.21.15.46:0/3905465378 shutdown_connections 2024-05-09T03:12:53.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.518+0000 7f9d51576640 1 -- 172.21.15.46:0/3905465378 wait complete. 2024-05-09T03:12:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.520+0000 7f9d51576640 1 Processor -- start 2024-05-09T03:12:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.520+0000 7f9d51576640 1 -- start start 2024-05-09T03:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d51576640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d51576640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d51576640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 0x7f9d4c154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d51576640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c0d0ae0 con 0x7f9d4c0d4800 2024-05-09T03:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d51576640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c160ea0 con 0x7f9d4c0b77c0 2024-05-09T03:12:53.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d51576640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4c163e60 con 0x7f9d4c0b5210 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 0x7f9d4c154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 0x7f9d4c154fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54054/0 (socket says 172.21.15.46:54054) 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.521+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/4195278289 learned_addr learned my addr 172.21.15.46:0/4195278289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4a7fc640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4affd640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 msgr2=0x7f9d4c15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4b7fe640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d4c0b77c0 msgr2=0x7f9d4c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4b7fe640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4b7fe640 1 -- 172.21.15.46:0/4195278289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c1607f0 con 0x7f9d4c0d4800 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d4b7fe640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 0x7f9d4c154fe0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009f40 tx=0x7f9d44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44041020 con 0x7f9d4c0d4800 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d44007780 con 0x7f9d4c0d4800 2024-05-09T03:12:53.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c160510 con 0x7f9d4c0d4800 2024-05-09T03:12:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44036040 con 0x7f9d4c0d4800 2024-05-09T03:12:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.522+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c0ce390 con 0x7f9d4c0d4800 2024-05-09T03:12:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.523+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c0b4e50 con 0x7f9d4c0d4800 2024-05-09T03:12:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.524+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9d44044020 con 0x7f9d4c0d4800 2024-05-09T03:12:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.525+0000 7f9d337fe640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d380762d0 0x7f9d380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.525+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f9d440b83e0 con 0x7f9d4c0d4800 2024-05-09T03:12:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.525+0000 7f9d4affd640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d380762d0 0x7f9d380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.526+0000 7f9d4affd640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d380762d0 0x7f9d380787e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c006fd0 tx=0x7f9d3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:53.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.530+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9d4c0b4e50 con 0x7f9d4c0d4800 2024-05-09T03:12:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.762+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d4c06a270 con 0x7f9d4c0d4800 2024-05-09T03:12:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.763+0000 7f9d337fe640 1 -- 172.21.15.46:0/4195278289 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+4350 (secure 0 0 0) 0x7f9d4404b030 con 0x7f9d4c0d4800 2024-05-09T03:12:53.763 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:53.763 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:53.158425+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-05-09T03:12:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 57 2024-05-09T03:12:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d380762d0 msgr2=0x7f9d380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d380762d0 0x7f9d380787e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c006fd0 tx=0x7f9d3c008040 comp rx=0 tx=0).stop 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 msgr2=0x7f9d4c154fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 0x7f9d4c154fe0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009f40 tx=0x7f9d44004a40 comp rx=0 tx=0).stop 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 shutdown_connections 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d380762d0 0x7f9d380787e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d4c0d4800 0x7f9d4c154fe0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d4c0b77c0 0x7f9d4c0ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 --2- 172.21.15.46:0/4195278289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d4c0b5210 0x7f9d4c15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 >> 172.21.15.46:0/4195278289 conn(0x7f9d4c0d29a0 msgr2=0x7f9d4c0b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 shutdown_connections 2024-05-09T03:12:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:53.765+0000 7f9d51576640 1 -- 172.21.15.46:0/4195278289 wait complete. 2024-05-09T03:12:54.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:53.158425+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5865}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5865': {'gid': 5865, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6833/1536534537', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1536534537}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 1536534537}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:12:54.180 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5865:up:replay 2024-05-09T03:12:54.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-05-09T03:12:55.182 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.311+0000 7f0692c14640 1 Processor -- start 2024-05-09T03:12:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.311+0000 7f0692c14640 1 -- start start 2024-05-09T03:12:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f0692c14640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f068c160610 0x7f068c0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f0692c14640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f068c0b7d20 0x7f068c0b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f0692c14640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c0b8130 con 0x7f068c0d57a0 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f0692c14640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f068c0d5bb0 con 0x7f068c0b7d20 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f0692c14640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f068c0d49e0 con 0x7f068c160610 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f06911b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f068c0b7d20 0x7f068c0b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f06911b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f068c0b7d20 0x7f068c0b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54066/0 (socket says 172.21.15.46:54066) 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f06911b1640 1 -- 172.21.15.46:0/332988403 learned_addr learned my addr 172.21.15.46:0/332988403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f068bfff640 1 --2- 172.21.15.46:0/332988403 >> v2:172.21.15.88:3300/0 conn(0x7f068c160610 0x7f068c0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.312+0000 7f06911b1640 1 -- 172.21.15.46:0/332988403 >> v2:172.21.15.88:3300/0 conn(0x7f068c160610 msgr2=0x7f068c0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.313+0000 7f06911b1640 1 --2- 172.21.15.46:0/332988403 >> v2:172.21.15.88:3300/0 conn(0x7f068c160610 0x7f068c0b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.313+0000 7f06911b1640 1 -- 172.21.15.46:0/332988403 >> v1:172.21.15.88:6789/0 conn(0x7f068c0d57a0 legacy=0x7f068c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:55.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.313+0000 7f06911b1640 1 -- 172.21.15.46:0/332988403 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c0d4cc0 con 0x7f068c0b7d20 2024-05-09T03:12:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.313+0000 7f068bfff640 1 --2- 172.21.15.46:0/332988403 >> v2:172.21.15.88:3300/0 conn(0x7f068c160610 0x7f068c0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.313+0000 7f06911b1640 1 --2- 172.21.15.46:0/332988403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c0b9450 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f068c0d2f80 tx=0x7f068402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9178b40e5109d874 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.313+0000 7f068b7fe640 1 -- 172.21.15.46:0/332988403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684036070 con 0x7f068c0b7d20 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f068b7fe640 1 -- 172.21.15.46:0/332988403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0684031600 con 0x7f068c0b7d20 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f068b7fe640 1 -- 172.21.15.46:0/332988403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06840319d0 con 0x7f068c0b7d20 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 -- 172.21.15.46:0/332988403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 msgr2=0x7f068c0b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 --2- 172.21.15.46:0/332988403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c0b9450 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f068c0d2f80 tx=0x7f068402a6a0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 -- 172.21.15.46:0/332988403 shutdown_connections 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 --2- 172.21.15.46:0/332988403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c0b9450 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 --2- 172.21.15.46:0/332988403 >> v2:172.21.15.88:3300/0 conn(0x7f068c160610 0x7f068c0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 -- 172.21.15.46:0/332988403 >> 172.21.15.46:0/332988403 conn(0x7f068c0743e0 msgr2=0x7f068c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 -- 172.21.15.46:0/332988403 shutdown_connections 2024-05-09T03:12:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.314+0000 7f0692c14640 1 -- 172.21.15.46:0/332988403 wait complete. 2024-05-09T03:12:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.316+0000 7f0692c14640 1 Processor -- start 2024-05-09T03:12:55.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.316+0000 7f0692c14640 1 -- start start 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f0692c14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f0692c14640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068c0d57a0 0x7f068c1ea330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f0692c14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068c160610 0x7f068c155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f0692c14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f068c160a20 con 0x7f068c0b7d20 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f0692c14640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f068c0d5280 con 0x7f068c160610 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f0692c14640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f068c0d15b0 con 0x7f068c0d57a0 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c148a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54074/0 (socket says 172.21.15.46:54074) 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 -- 172.21.15.46:0/2104472448 learned_addr learned my addr 172.21.15.46:0/2104472448 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06911b1640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068c160610 0x7f068c155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f068bfff640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068c0d57a0 0x7f068c1ea330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 -- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068c0d57a0 msgr2=0x7f068c1ea330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068c0d57a0 0x7f068c1ea330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 -- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068c160610 msgr2=0x7f068c155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068c160610 0x7f068c155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 -- 172.21.15.46:0/2104472448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1572b0 con 0x7f068c0b7d20 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06911b1640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068c160610 0x7f068c155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.318+0000 7f06909b0640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c148a70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f067c00b3c0 tx=0x7f067c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.319+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c011830 con 0x7f068c0b7d20 2024-05-09T03:12:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.319+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067c016070 con 0x7f068c0b7d20 2024-05-09T03:12:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.319+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c157590 con 0x7f068c0b7d20 2024-05-09T03:12:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.319+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c011c50 con 0x7f068c0b7d20 2024-05-09T03:12:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.319+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1f6d90 con 0x7f068c0b7d20 2024-05-09T03:12:55.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.320+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c06a270 con 0x7f068c0b7d20 2024-05-09T03:12:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.320+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f067c01b460 con 0x7f068c0b7d20 2024-05-09T03:12:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.321+0000 7f06897fa640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0678076300 0x7f0678078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.321+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f067c0973d0 con 0x7f068c0b7d20 2024-05-09T03:12:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.321+0000 7f068bfff640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0678076300 0x7f0678078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.322+0000 7f068bfff640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0678076300 0x7f0678078810 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f068c160070 tx=0x7f0680008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:55.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.326+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f067c060bf0 con 0x7f068c0b7d20 2024-05-09T03:12:55.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.553+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f068c1e7720 con 0x7f068c0b7d20 2024-05-09T03:12:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.554+0000 7f06897fa640 1 -- 172.21.15.46:0/2104472448 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5176 (secure 0 0 0) 0x7f067c0604f0 con 0x7f068c0b7d20 2024-05-09T03:12:55.555 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:55.555 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":59,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:54.164844+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-05-09T03:12:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 59 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0678076300 msgr2=0x7f0678078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0678076300 0x7f0678078810 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f068c160070 tx=0x7f0680008040 comp rx=0 tx=0).stop 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 msgr2=0x7f068c148a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c148a70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f067c00b3c0 tx=0x7f067c00b880 comp rx=0 tx=0).stop 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 shutdown_connections 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068c160610 0x7f068c155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f068c0d57a0 0x7f068c1ea330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0678076300 0x7f0678078810 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 --2- 172.21.15.46:0/2104472448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0b7d20 0x7f068c148a70 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 >> 172.21.15.46:0/2104472448 conn(0x7f068c0743e0 msgr2=0x7f068c1e6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 shutdown_connections 2024-05-09T03:12:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:55.556+0000 7f0692c14640 1 -- 172.21.15.46:0/2104472448 wait complete. 2024-05-09T03:12:55.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:54.164844+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5865}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5865': {'gid': 5865, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.46:6833/1536534537', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1536534537}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 1536534537}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:12:55.972 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5865:up:rejoin 2024-05-09T03:12:56.972 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:12:57.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.101+0000 7f127b242640 1 Processor -- start 2024-05-09T03:12:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.101+0000 7f127b242640 1 -- start start 2024-05-09T03:12:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f127b242640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12741615e0 0x7f127415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f127b242640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12740d2480 0x7f12740d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f127b242640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1274160b20 con 0x7f1274164c70 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f127b242640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f127415e520 con 0x7f12741615e0 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f127b242640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1274161150 con 0x7f12740d2480 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f1273fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12741615e0 0x7f127415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f1278fde640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1274164c70 0x7f12740d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56466/0 (socket says 172.21.15.46:56466) 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f1273fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f12741615e0 0x7f127415dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41042/0 (socket says 172.21.15.46:41042) 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.102+0000 7f1278fde640 1 -- 172.21.15.46:0/2783851037 learned_addr learned my addr 172.21.15.46:0/2783851037 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.103+0000 7f12797df640 1 --2- 172.21.15.46:0/2783851037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12740d2480 0x7f12740d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.103+0000 7f1273fff640 1 -- 172.21.15.46:0/2783851037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12740d2480 msgr2=0x7f12740d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.103+0000 7f1273fff640 1 --2- 172.21.15.46:0/2783851037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12740d2480 0x7f12740d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.103+0000 7f1273fff640 1 -- 172.21.15.46:0/2783851037 >> v1:172.21.15.46:6789/0 conn(0x7f1274164c70 legacy=0x7f12740d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.103+0000 7f1273fff640 1 -- 172.21.15.46:0/2783851037 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12740d15f0 con 0x7f12741615e0 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.104+0000 7f1273fff640 1 --2- 172.21.15.46:0/2783851037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12741615e0 0x7f127415dfe0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f125c009d90 tx=0x7f125c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2af57fa00b7b474 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.104+0000 7f12737fe640 1 -- 172.21.15.46:0/2783851037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c007740 con 0x7f12741615e0 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.104+0000 7f12737fe640 1 -- 172.21.15.46:0/2783851037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f125c0078e0 con 0x7f12741615e0 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.104+0000 7f12737fe640 1 -- 172.21.15.46:0/2783851037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c007cb0 con 0x7f12741615e0 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 -- 172.21.15.46:0/2783851037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12741615e0 msgr2=0x7f127415dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 --2- 172.21.15.46:0/2783851037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12741615e0 0x7f127415dfe0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f125c009d90 tx=0x7f125c02a660 comp rx=0 tx=0).stop 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 -- 172.21.15.46:0/2783851037 shutdown_connections 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 --2- 172.21.15.46:0/2783851037 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12740d2480 0x7f12740d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 --2- 172.21.15.46:0/2783851037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12741615e0 0x7f127415dfe0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 -- 172.21.15.46:0/2783851037 >> 172.21.15.46:0/2783851037 conn(0x7f1274073db0 msgr2=0x7f12740741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 -- 172.21.15.46:0/2783851037 shutdown_connections 2024-05-09T03:12:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.105+0000 7f127b242640 1 -- 172.21.15.46:0/2783851037 wait complete. 2024-05-09T03:12:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.106+0000 7f127b242640 1 Processor -- start 2024-05-09T03:12:57.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.107+0000 7f127b242640 1 -- start start 2024-05-09T03:12:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f127b242640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12740d2480 0x7f12740cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f127b242640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12741615e0 0x7f12741eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f127b242640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 0x7f127415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f127b242640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12741619f0 con 0x7f1274164c70 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f127b242640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1274160de0 con 0x7f12740d2480 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f127b242640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12740d2dd0 con 0x7f12741615e0 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f12797df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 0x7f127415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f12797df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 0x7f127415c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41054/0 (socket says 172.21.15.46:41054) 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f12797df640 1 -- 172.21.15.46:0/2194341984 learned_addr learned my addr 172.21.15.46:0/2194341984 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f1278fde640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12740d2480 0x7f12740cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.108+0000 7f1273fff640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12741615e0 0x7f12741eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12797df640 1 -- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12741615e0 msgr2=0x7f12741eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12797df640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12741615e0 0x7f12741eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12797df640 1 -- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12740d2480 msgr2=0x7f12740cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12797df640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12740d2480 0x7f12740cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12797df640 1 -- 172.21.15.46:0/2194341984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12741e9f30 con 0x7f1274164c70 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12797df640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 0x7f127415c740 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f126c00d5d0 tx=0x7f126c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c0125f0 con 0x7f1274164c70 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126c019070 con 0x7f1274164c70 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c0129f0 con 0x7f1274164c70 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12741ea430 con 0x7f1274164c70 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.109+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12741ea9f0 con 0x7f1274164c70 2024-05-09T03:12:57.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.110+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12741ea1a0 con 0x7f1274164c70 2024-05-09T03:12:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.111+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f126c012b90 con 0x7f1274164c70 2024-05-09T03:12:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.111+0000 7f12717fa640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12540762d0 0x7f12540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.112+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f126c097a00 con 0x7f1274164c70 2024-05-09T03:12:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.112+0000 7f1278fde640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12540762d0 0x7f12540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.112+0000 7f1278fde640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12540762d0 0x7f12540787e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1264000f60 tx=0x7f1264009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.116+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f126c060bd0 con 0x7f1274164c70 2024-05-09T03:12:57.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.348+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f127406a270 con 0x7f1274164c70 2024-05-09T03:12:57.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.349+0000 7f12717fa640 1 -- 172.21.15.46:0/2194341984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5183 (secure 0 0 0) 0x7f126c064ed0 con 0x7f1274164c70 2024-05-09T03:12:57.350 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:12:57.350 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:56.169641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":16,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-05-09T03:12:57.350 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2024-05-09T03:12:57.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12540762d0 msgr2=0x7f12540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12540762d0 0x7f12540787e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1264000f60 tx=0x7f1264009040 comp rx=0 tx=0).stop 2024-05-09T03:12:57.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 msgr2=0x7f127415c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 0x7f127415c740 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f126c00d5d0 tx=0x7f126c00da90 comp rx=0 tx=0).stop 2024-05-09T03:12:57.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 shutdown_connections 2024-05-09T03:12:57.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12540762d0 0x7f12540787e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1274164c70 0x7f127415c740 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12741615e0 0x7f12741eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 --2- 172.21.15.46:0/2194341984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12740d2480 0x7f12740cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 >> 172.21.15.46:0/2194341984 conn(0x7f1274073db0 msgr2=0x7f127415fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 shutdown_connections 2024-05-09T03:12:57.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.351+0000 7f127b242640 1 -- 172.21.15.46:0/2194341984 wait complete. 2024-05-09T03:12:57.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 60, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:12:06.230074+0000', 'modified': '2024-05-09T03:12:56.169641+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 57, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5865}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5865': {'gid': 5865, 'name': 'c', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.46:6833/1536534537', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1536534537}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 1536534537}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5865, 'qdb_cluster': [5865]} 2024-05-09T03:12:57.766 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:12:57.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:12:57.824 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:12:57.823+0000 7f2681073640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:12:57.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.886+0000 7f4b58386640 1 Processor -- start 2024-05-09T03:12:57.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.886+0000 7f4b58386640 1 -- start start 2024-05-09T03:12:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b58386640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4b500d2e90 0x7f4b500b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b58386640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b500b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b58386640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b500d3400 con 0x7f4b500b4fc0 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b58386640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4b500d4a60 con 0x7f4b500d2e90 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b58386640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4b500d4d40 con 0x7f4b501605b0 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b56923640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b500b4fc0 0x7f4b500d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56482/0 (socket says 172.21.15.46:56482) 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b56923640 1 -- 172.21.15.46:0/960518402 learned_addr learned my addr 172.21.15.46:0/960518402 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b56122640 1 --2- 172.21.15.46:0/960518402 >> v2:172.21.15.88:3300/0 conn(0x7f4b500d2e90 0x7f4b500b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b55921640 1 --2- 172.21.15.46:0/960518402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b500b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b55120640 1 -- 172.21.15.46:0/960518402 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1246042491 0 0) 0x7f4b500d3400 con 0x7f4b500b4fc0 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.888+0000 7f4b55120640 1 -- 172.21.15.46:0/960518402 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b38003520 con 0x7f4b500b4fc0 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.889+0000 7f4b55921640 1 -- 172.21.15.46:0/960518402 >> v2:172.21.15.88:3300/0 conn(0x7f4b500d2e90 msgr2=0x7f4b500b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.889+0000 7f4b55921640 1 --2- 172.21.15.46:0/960518402 >> v2:172.21.15.88:3300/0 conn(0x7f4b500d2e90 0x7f4b500b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.889+0000 7f4b55921640 1 -- 172.21.15.46:0/960518402 >> v1:172.21.15.46:6789/0 conn(0x7f4b500b4fc0 legacy=0x7f4b500d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.889+0000 7f4b55921640 1 -- 172.21.15.46:0/960518402 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b500d5020 con 0x7f4b501605b0 2024-05-09T03:12:57.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.889+0000 7f4b56122640 1 --2- 172.21.15.46:0/960518402 >> v2:172.21.15.88:3300/0 conn(0x7f4b500d2e90 0x7f4b500b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.890+0000 7f4b55921640 1 --2- 172.21.15.46:0/960518402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b500b7ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b440097d0 tx=0x7f4b4402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ec14c26cdf3c9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.890+0000 7f4b55120640 1 -- 172.21.15.46:0/960518402 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44037070 con 0x7f4b501605b0 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.890+0000 7f4b55120640 1 -- 172.21.15.46:0/960518402 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b44032440 con 0x7f4b501605b0 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.890+0000 7f4b55120640 1 -- 172.21.15.46:0/960518402 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b440327f0 con 0x7f4b501605b0 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.890+0000 7f4b58386640 1 -- 172.21.15.46:0/960518402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 msgr2=0x7f4b500b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.890+0000 7f4b58386640 1 --2- 172.21.15.46:0/960518402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b500b7ca0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4b440097d0 tx=0x7f4b4402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.891+0000 7f4b58386640 1 -- 172.21.15.46:0/960518402 shutdown_connections 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.891+0000 7f4b58386640 1 --2- 172.21.15.46:0/960518402 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b500b7ca0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.891+0000 7f4b58386640 1 --2- 172.21.15.46:0/960518402 >> v2:172.21.15.88:3300/0 conn(0x7f4b500d2e90 0x7f4b500b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.891+0000 7f4b58386640 1 -- 172.21.15.46:0/960518402 >> 172.21.15.46:0/960518402 conn(0x7f4b5008d390 msgr2=0x7f4b5008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.891+0000 7f4b58386640 1 -- 172.21.15.46:0/960518402 shutdown_connections 2024-05-09T03:12:57.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.891+0000 7f4b58386640 1 -- 172.21.15.46:0/960518402 wait complete. 2024-05-09T03:12:57.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.892+0000 7f4b58386640 1 Processor -- start 2024-05-09T03:12:57.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.892+0000 7f4b58386640 1 -- start start 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b58386640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 0x7f4b500cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b58386640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b500d2e90 0x7f4b50159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b58386640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b5015a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b58386640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b500d5540 con 0x7f4b500b4fc0 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b58386640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b500b4c00 con 0x7f4b500d2e90 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b58386640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4b50155a20 con 0x7f4b501605b0 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 0x7f4b500cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 0x7f4b500cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41064/0 (socket says 172.21.15.46:41064) 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 -- 172.21.15.46:0/2852354777 learned_addr learned my addr 172.21.15.46:0/2852354777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b55921640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b500d2e90 0x7f4b50159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56923640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b5015a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 -- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 msgr2=0x7f4b5015a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b5015a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 -- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b500d2e90 msgr2=0x7f4b50159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b500d2e90 0x7f4b50159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.894+0000 7f4b56122640 1 -- 172.21.15.46:0/2852354777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b501fab00 con 0x7f4b500b4fc0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.895+0000 7f4b56122640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 0x7f4b500cf6c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009e60 tx=0x7f4b4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.895+0000 7f4b4effd640 1 -- 172.21.15.46:0/2852354777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40018020 con 0x7f4b500b4fc0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.895+0000 7f4b4effd640 1 -- 172.21.15.46:0/2852354777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b40014480 con 0x7f4b500b4fc0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.895+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b501f9ef0 con 0x7f4b500b4fc0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.895+0000 7f4b4effd640 1 -- 172.21.15.46:0/2852354777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40014850 con 0x7f4b500b4fc0 2024-05-09T03:12:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.895+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b501fa4b0 con 0x7f4b500b4fc0 2024-05-09T03:12:57.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.897+0000 7f4b4effd640 1 -- 172.21.15.46:0/2852354777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4b4001b020 con 0x7f4b500b4fc0 2024-05-09T03:12:57.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.897+0000 7f4b4effd640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b3c0762d0 0x7f4b3c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.897+0000 7f4b4effd640 1 -- 172.21.15.46:0/2852354777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f4b400983b0 con 0x7f4b500b4fc0 2024-05-09T03:12:57.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.898+0000 7f4b55921640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b3c0762d0 0x7f4b3c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.898+0000 7f4b5f243740 1 Processor -- start 2024-05-09T03:12:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.898+0000 7f4b55921640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b3c0762d0 0x7f4b3c0787e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4b44034000 tx=0x7f4b4402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.898+0000 7f4b5f243740 1 -- start start 2024-05-09T03:12:57.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8be5270 0x562be8fb4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8fcfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bc4590 0x562be8fd0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b5f243740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562be8eed4e0 con 0x562be8bbc3b0 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562be8e88ff0 con 0x562be8bc4590 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562be8ee94a0 con 0x562be8be5270 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8fcfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8fcfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41080/0 (socket says 172.21.15.46:41080) 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 -- 172.21.15.46:0/2079850580 learned_addr learned my addr 172.21.15.46:0/2079850580 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b56122640 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8be5270 0x562be8fb4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b56923640 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bc4590 0x562be8fd0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 -- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8be5270 msgr2=0x562be8fb4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8be5270 0x562be8fb4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 -- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bc4590 msgr2=0x562be8fd0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bc4590 0x562be8fd0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 -- 172.21.15.46:0/2079850580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be8e6f720 con 0x562be8bbc3b0 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b55921640 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8fcfbe0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4b440098c0 tx=0x7f4b4402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8c75c7aae33a07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.900+0000 7f4b37fff640 1 -- 172.21.15.46:0/2079850580 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4404f070 con 0x562be8bbc3b0 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b37fff640 1 -- 172.21.15.46:0/2079850580 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b44038040 con 0x562be8bbc3b0 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b37fff640 1 -- 172.21.15.46:0/2079850580 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44037040 con 0x562be8bbc3b0 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 -- 172.21.15.46:0/2079850580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 msgr2=0x562be8fcfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8fcfbe0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f4b440098c0 tx=0x7f4b4402c000 comp rx=0 tx=0).stop 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 -- 172.21.15.46:0/2079850580 shutdown_connections 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bc4590 0x562be8fd0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8fcfbe0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2079850580 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8be5270 0x562be8fb4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 -- 172.21.15.46:0/2079850580 >> 172.21.15.46:0/2079850580 conn(0x562be8bd93c0 msgr2=0x562be8f8a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.901+0000 7f4b5f243740 1 -- 172.21.15.46:0/2079850580 shutdown_connections 2024-05-09T03:12:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.902+0000 7f4b5f243740 1 -- 172.21.15.46:0/2079850580 wait complete. 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.902+0000 7f4b5f243740 1 -- start start 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.902+0000 7f4b5f243740 1 Processor -- start 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bbc3b0 0x562be8fb6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fc8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 0x562be8fc8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562be8e74760 con 0x562be8be5270 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562be8fcd2c0 con 0x562be8bbc3b0 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562be8fb91d0 con 0x562be8bc4590 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b5f243740 10 client.0 _ll_register_callbacks cb 0x562be8e974b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 0x562be8fc8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 0x562be8fc8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41096/0 (socket says 172.21.15.46:41096) 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 -- 172.21.15.46:0/2233280393 learned_addr learned my addr 172.21.15.46:0/2233280393 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56122640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bbc3b0 0x562be8fb6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b55921640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fc8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 -- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 msgr2=0x562be8fc8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fc8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 -- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bbc3b0 msgr2=0x562be8fb6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bbc3b0 0x562be8fb6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56923640 1 -- 172.21.15.46:0/2233280393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be8eb5bd0 con 0x562be8be5270 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.904+0000 7f4b56122640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bbc3b0 0x562be8fb6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b56923640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 0x562be8fc8bc0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4b48007700 tx=0x7f4b48001620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bde5af8d6ef025 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b357fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b357fa640 1 -- 172.21.15.46:0/2233280393 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b48019020 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b357fa640 1 -- 172.21.15.46:0/2233280393 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f4b48015440 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b44038040 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b357fa640 1 -- 172.21.15.46:0/2233280393 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f4b48015640 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 10 client.6273 fetch_fsmap learned FSMap version 60 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x562be8e9b280 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b357fa640 1 -- 172.21.15.46:0/2233280393 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b44038040 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b357fa640 1 -- 172.21.15.46:0/2233280393 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 60) ==== 3494+0+0 (secure 0 0 0) 0x7f4b48026080 con 0x562be8be5270 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 10 client.6273 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 10 client.6273 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 20 client.6273 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8ba3e40 0x562be8fcc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 4 client.6273 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:12:57.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.905+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 --> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562be8efb0f0 con 0x562be8ba3e40 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.906+0000 7f4b56122640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8ba3e40 0x562be8fcc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.906+0000 7f4b357fa640 10 client.6273 ms_handle_connect on v2:172.21.15.46:6832/1536534537 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.906+0000 7f4b56122640 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8ba3e40 0x562be8fcc500 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af005ee3c6c36bb server_cookie=b617011b683a2c6b in_seq=0 out_seq=0 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b357fa640 1 -- 172.21.15.46:0/2233280393 <== mds.0 v2:172.21.15.46:6832/1536534537 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562be8efb0f0 con 0x562be8ba3e40 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b357fa640 10 client.6273 handle_command_reply: tid=0 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 client.6273 shutdown 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 msgr2=0x562be8fc8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 0x562be8fc8bc0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4b48007700 tx=0x7f4b48001620 comp rx=0 tx=0).stop 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 shutdown_connections 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8ba3e40 0x562be8fcc500 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8be5270 0x562be8fc8bc0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fc8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b357fa640 0 client.6273 ms_handle_reset on v2:172.21.15.46:6832/1536534537 2024-05-09T03:12:57.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 --2- 172.21.15.46:0/2233280393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8bbc3b0 0x562be8fb6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 >> 172.21.15.46:0/2233280393 conn(0x562be8bd93c0 msgr2=0x562be8fb1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 shutdown_connections 2024-05-09T03:12:57.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 1 -- 172.21.15.46:0/2233280393 wait complete. 2024-05-09T03:12:57.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.907+0000 7f4b5f243740 20 client.6273 trim_cache size 0 max 16384 2024-05-09T03:12:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.926+0000 7f4b5f243740 1 Processor -- start 2024-05-09T03:12:57.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.926+0000 7f4b5f243740 1 -- start start 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.927+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8f88e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fcc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fcca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b5f243740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50153e50 con 0x562be8bbc3b0 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b50152aa0 con 0x562be8be5270 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4b501530f0 con 0x562be8bc4590 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8f88e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8f88e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41106/0 (socket says 172.21.15.46:41106) 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 -- 172.21.15.46:0/1321514578 learned_addr learned my addr 172.21.15.46:0/1321514578 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56923640 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fcca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 -- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 msgr2=0x562be8fcc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fcc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 -- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 msgr2=0x562be8fcca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fcca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 -- 172.21.15.46:0/1321514578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be8c50850 con 0x562be8bbc3b0 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56923640 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fcca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b56122640 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8f88e50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4b4005d600 tx=0x7f4b40098040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a0ac336092ecfc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b377fe640 1 -- 172.21.15.46:0/1321514578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b40094070 con 0x562be8bbc3b0 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b377fe640 1 -- 172.21.15.46:0/1321514578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b40065880 con 0x562be8bbc3b0 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.928+0000 7f4b377fe640 1 -- 172.21.15.46:0/1321514578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4006a330 con 0x562be8bbc3b0 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 -- 172.21.15.46:0/1321514578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 msgr2=0x562be8f88e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8f88e50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4b4005d600 tx=0x7f4b40098040 comp rx=0 tx=0).stop 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 -- 172.21.15.46:0/1321514578 shutdown_connections 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fcca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bc4590 0x562be8fcc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 --2- 172.21.15.46:0/1321514578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bbc3b0 0x562be8f88e50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 -- 172.21.15.46:0/1321514578 >> 172.21.15.46:0/1321514578 conn(0x562be8bd93c0 msgr2=0x562be8f8abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 -- 172.21.15.46:0/1321514578 shutdown_connections 2024-05-09T03:12:57.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.929+0000 7f4b5f243740 1 -- 172.21.15.46:0/1321514578 wait complete. 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.930+0000 7f4b5f243740 1 -- start start 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.930+0000 7f4b5f243740 1 Processor -- start 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bbc3b0 0x562be8f85380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 0x562be8fc7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fb1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562be8faf810 con 0x562be8bc4590 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562be8faf250 con 0x562be8be5270 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562be8fcaf60 con 0x562be8bbc3b0 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.931+0000 7f4b5f243740 10 client.0 _ll_register_callbacks cb 0x7f4b48006a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 0x562be8fc7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 0x562be8fc7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41122/0 (socket says 172.21.15.46:41122) 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 -- 172.21.15.46:0/4094480788 learned_addr learned my addr 172.21.15.46:0/4094480788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b56923640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fb1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b56122640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bbc3b0 0x562be8f85380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 -- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bbc3b0 msgr2=0x562be8f85380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bbc3b0 0x562be8f85380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 -- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 msgr2=0x562be8fb1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fb1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 -- 172.21.15.46:0/4094480788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be8cc76b0 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b56122640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bbc3b0 0x562be8f85380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b55921640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 0x562be8fc7660 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4b440045b0 tx=0x7f4b440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03a1ecb0e927fd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b357fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b357fa640 1 -- 172.21.15.46:0/4094480788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44037070 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b357fa640 1 -- 172.21.15.46:0/4094480788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f4b44003d60 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b40065880 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b357fa640 1 -- 172.21.15.46:0/4094480788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f4b4404f070 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b5f243740 10 client.6279 fetch_fsmap learned FSMap version 60 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.932+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x562be8ea9630 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b357fa640 1 -- 172.21.15.46:0/4094480788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b40065880 con 0x562be8bc4590 2024-05-09T03:12:57.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b357fa640 1 -- 172.21.15.46:0/4094480788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 60) ==== 3494+0+0 (secure 0 0 0) 0x7f4b44032510 con 0x562be8bc4590 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 10 client.6279 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 10 client.6279 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 20 client.6279 populate_metadata read hostname 'smithi046' 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8fb1c20 0x562be8fb5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 4 client.6279 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 --> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] -- command(tid 0: {"prefix": "status"}) -- 0x562be8fb8650 con 0x562be8fb1c20 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b56122640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8fb1c20 0x562be8fb5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b357fa640 10 client.6279 ms_handle_connect on v2:172.21.15.46:6832/1536534537 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b56122640 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8fb1c20 0x562be8fb5dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4d6a490bc44ddaa server_cookie=b55c294e15c9a1c4 in_seq=0 out_seq=0 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b357fa640 1 -- 172.21.15.46:0/4094480788 <== mds.0 v2:172.21.15.46:6832/1536534537 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x562be8fb8650 con 0x562be8fb1c20 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b357fa640 10 client.6279 handle_command_reply: tid=0 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.933+0000 7f4b5f243740 1 client.6279 shutdown 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 msgr2=0x562be8fc7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 0x562be8fc7660 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4b440045b0 tx=0x7f4b440046c0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 shutdown_connections 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x562be8fb1c20 0x562be8fb5dc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562be8be5270 0x562be8fb1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562be8bc4590 0x562be8fc7660 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b357fa640 0 client.6279 ms_handle_reset on v2:172.21.15.46:6832/1536534537 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 --2- 172.21.15.46:0/4094480788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562be8bbc3b0 0x562be8f85380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 >> 172.21.15.46:0/4094480788 conn(0x562be8bd93c0 msgr2=0x562be8f88e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 shutdown_connections 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 1 -- 172.21.15.46:0/4094480788 wait complete. 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b5f243740 20 client.6279 trim_cache size 0 max 16384 2024-05-09T03:12:57.940 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5865, 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.7742255450000002, 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 60, 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 57, 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 57, 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 56.034662617000002 2024-05-09T03:12:57.941 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:12:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b3c0762d0 msgr2=0x7f4b3c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b3c0762d0 0x7f4b3c0787e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4b44034000 tx=0x7f4b4402e000 comp rx=0 tx=0).stop 2024-05-09T03:12:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 msgr2=0x7f4b500cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:12:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 0x7f4b500cf6c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4b40009e60 tx=0x7f4b4000c490 comp rx=0 tx=0).stop 2024-05-09T03:12:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 shutdown_connections 2024-05-09T03:12:57.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b501605b0 0x7f4b5015a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b500d2e90 0x7f4b50159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b3c0762d0 0x7f4b3c0787e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 --2- 172.21.15.46:0/2852354777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b500b4fc0 0x7f4b500cf6c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:12:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.934+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 >> 172.21.15.46:0/2852354777 conn(0x7f4b5008d390 msgr2=0x7f4b5015ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:12:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.935+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 shutdown_connections 2024-05-09T03:12:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:12:57.935+0000 7f4b58386640 1 -- 172.21.15.46:0/2852354777 wait complete. 2024-05-09T03:13:00.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:13:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-05-09T03:13:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-05-09T03:13:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe66ffff640 0 client.6189 ms_handle_remote_reset on v2:172.21.15.46:6834/103238544 2024-05-09T03:13:00.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:13:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:13:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:13:00.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:00.643+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:13:06.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:13:06.244 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:13:15.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:15.651+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:13:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:15.652+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:13:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:15.652+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:13:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:15.652+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:13:15.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:15.652+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:13:30.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:30.655+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:13:30.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:30.656+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:13:30.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:30.656+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:13:30.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:30.656+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:13:30.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:30.656+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:13:36.385 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:13:36.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:13:45.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:45.656+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:13:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:45.656+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:13:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:45.656+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:13:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:45.656+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:13:45.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:13:45.656+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:14:00.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:00.659+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:00.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:00.659+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:14:00.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:00.659+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:14:00.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:00.659+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:14:00.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:00.659+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:14:06.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:14:06.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:14:15.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:15.667+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:15.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:15.667+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:14:15.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:15.667+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:14:15.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:15.667+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:14:15.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:15.667+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:14:30.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:30.670+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:30.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:30.671+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-05-09T03:14:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:30.671+0000 7fe69963b640 1 -- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 msgr2=0x556887ed5be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T03:14:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:30.671+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T03:14:30.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:30.671+0000 7fe69963b640 1 --2- 172.21.15.46:0/2886340016 >> [v2:172.21.15.46:6834/103238544,v1:172.21.15.46:6835/103238544] conn(0x5568879fe5d0 0x556887ed5be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-05-09T03:14:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-05-09T03:14:34.995 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:14:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.124+0000 7f0d4f814640 1 Processor -- start 2024-05-09T03:14:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.124+0000 7f0d4f814640 1 -- start start 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4f814640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d481615e0 0x7f0d4815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4f814640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d480d2480 0x7f0d480d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4f814640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d48160b20 con 0x7f0d48164c70 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4f814640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0d4815e520 con 0x7f0d480d2480 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4f814640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0d48161150 con 0x7f0d481615e0 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4d5b0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0d48164c70 0x7f0d480d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40826/0 (socket says 172.21.15.46:40826) 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.125+0000 7f0d4d5b0640 1 -- 172.21.15.46:0/545782982 learned_addr learned my addr 172.21.15.46:0/545782982 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4cdaf640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d481615e0 0x7f0d4815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4ddb1640 1 --2- 172.21.15.46:0/545782982 >> v2:172.21.15.88:3300/0 conn(0x7f0d480d2480 0x7f0d480d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d47fff640 1 -- 172.21.15.46:0/545782982 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762708935 0 0) 0x7f0d48160b20 con 0x7f0d48164c70 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d47fff640 1 -- 172.21.15.46:0/545782982 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d30003520 con 0x7f0d48164c70 2024-05-09T03:14:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4ddb1640 1 -- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d481615e0 msgr2=0x7f0d4815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4ddb1640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d481615e0 0x7f0d4815dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4ddb1640 1 -- 172.21.15.46:0/545782982 >> v1:172.21.15.46:6789/0 conn(0x7f0d48164c70 legacy=0x7f0d480d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4ddb1640 1 -- 172.21.15.46:0/545782982 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d480d15f0 con 0x7f0d480d2480 2024-05-09T03:14:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.126+0000 7f0d4cdaf640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d481615e0 0x7f0d4815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.127+0000 7f0d4ddb1640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d480d2480 0x7f0d480d2890 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009d90 tx=0x7f0d4002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf535810942b4b6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.127+0000 7f0d47fff640 1 -- 172.21.15.46:0/545782982 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d40007740 con 0x7f0d480d2480 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.127+0000 7f0d47fff640 1 -- 172.21.15.46:0/545782982 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d400078e0 con 0x7f0d480d2480 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.127+0000 7f0d47fff640 1 -- 172.21.15.46:0/545782982 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d40007c90 con 0x7f0d480d2480 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 -- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d480d2480 msgr2=0x7f0d480d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d480d2480 0x7f0d480d2890 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009d90 tx=0x7f0d4002a660 comp rx=0 tx=0).stop 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 -- 172.21.15.46:0/545782982 shutdown_connections 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d480d2480 0x7f0d480d2890 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 --2- 172.21.15.46:0/545782982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d481615e0 0x7f0d4815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 -- 172.21.15.46:0/545782982 >> 172.21.15.46:0/545782982 conn(0x7f0d48073db0 msgr2=0x7f0d480741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 -- 172.21.15.46:0/545782982 shutdown_connections 2024-05-09T03:14:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.128+0000 7f0d4f814640 1 -- 172.21.15.46:0/545782982 wait complete. 2024-05-09T03:14:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.130+0000 7f0d4f814640 1 Processor -- start 2024-05-09T03:14:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.130+0000 7f0d4f814640 1 -- start start 2024-05-09T03:14:35.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4f814640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d480d2480 0x7f0d480cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:35.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4f814640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 0x7f0d48162bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:35.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4f814640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d48164c70 0x7f0d48163100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:35.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4f814640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d481619f0 con 0x7f0d481615e0 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4f814640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d48160de0 con 0x7f0d48164c70 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4f814640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0d480d2dd0 con 0x7f0d480d2480 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.131+0000 7f0d4cdaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 0x7f0d48162bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 0x7f0d48162bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55328/0 (socket says 172.21.15.46:55328) 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 -- 172.21.15.46:0/3545533043 learned_addr learned my addr 172.21.15.46:0/3545533043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4d5b0640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d480d2480 0x7f0d480cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4ddb1640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d48164c70 0x7f0d48163100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 -- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d480d2480 msgr2=0x7f0d480cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d480d2480 0x7f0d480cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 -- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d48164c70 msgr2=0x7f0d48163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d48164c70 0x7f0d48163100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 -- 172.21.15.46:0/3545533043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d480ca920 con 0x7f0d481615e0 2024-05-09T03:14:35.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4cdaf640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 0x7f0d48162bc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00e990 tx=0x7f0d3c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c017070 con 0x7f0d481615e0 2024-05-09T03:14:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c00c9e0 con 0x7f0d481615e0 2024-05-09T03:14:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00cdb0 con 0x7f0d481615e0 2024-05-09T03:14:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.132+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d480cac00 con 0x7f0d481615e0 2024-05-09T03:14:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.133+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d480cb1c0 con 0x7f0d481615e0 2024-05-09T03:14:35.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.133+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4808cfd0 con 0x7f0d481615e0 2024-05-09T03:14:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.134+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0d3c01a070 con 0x7f0d481615e0 2024-05-09T03:14:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.135+0000 7f0d45ffb640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d340762d0 0x7f0d340787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.135+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f0d3c097910 con 0x7f0d481615e0 2024-05-09T03:14:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.135+0000 7f0d4d5b0640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d340762d0 0x7f0d340787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:35.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.136+0000 7f0d4d5b0640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d340762d0 0x7f0d340787e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005880 tx=0x7f0d38006aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:35.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.140+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0d4808cfd0 con 0x7f0d481615e0 2024-05-09T03:14:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.373+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0d4806a270 con 0x7f0d481615e0 2024-05-09T03:14:35.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.373+0000 7f0d45ffb640 1 -- 172.21.15.46:0/3545533043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v60) ==== 53+0+73 (secure 0 0 0) 0x7f0d3c060ae0 con 0x7f0d481615e0 2024-05-09T03:14:35.374 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:14:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d340762d0 msgr2=0x7f0d340787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d340762d0 0x7f0d340787e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005880 tx=0x7f0d38006aa0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 msgr2=0x7f0d48162bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 0x7f0d48162bc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c00e990 tx=0x7f0d3c00ee50 comp rx=0 tx=0).stop 2024-05-09T03:14:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 shutdown_connections 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d48164c70 0x7f0d48163100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d340762d0 0x7f0d340787e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d481615e0 0x7f0d48162bc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 --2- 172.21.15.46:0/3545533043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d480d2480 0x7f0d480cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 >> 172.21.15.46:0/3545533043 conn(0x7f0d48073db0 msgr2=0x7f0d4815d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.375+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 shutdown_connections 2024-05-09T03:14:35.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35.376+0000 7f0d4f814640 1 -- 172.21.15.46:0/3545533043 wait complete. 2024-05-09T03:14:35.387 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:14:35.388 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:14:35.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:35.388 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:35.405 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42242sec preferred_lft 42242sec 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:14:35.406 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:35.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:35.408 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:14:35.469 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:14:35.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:35.471 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:14:35.528 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:14:35.528 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:14:35.528 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:14:35.528 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:14:35.528 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:14:35.529 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:14:35.529 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:14:35.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:14:35.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:14:35.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:14:35.529 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:14:35.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:14:35.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:14:35.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:35.658 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:14:35.674 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:14:35.675 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:14:35.675 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:14:35.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:35.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:14:35.741 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:14:35.741 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:14:35.742 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:35.797 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:14:35.797 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:14:35.798 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:35.853 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:14:35.920 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:14:35.977 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:14:35.977 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:14:35.977 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:14:35.978 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:14:35.979 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:14:35.980 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:14:35.980 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:14:35.980 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:14:35.980 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:14:35.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:35.980 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:14:36.033 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:14:36.033 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:14:36.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:14:36.101 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:14:36.159 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:14:36.159 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:14:36.160 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:14:36.161 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:14:36.162 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:14:36.162 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:14:36.162 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:14:36.162 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:14:36.162 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:14:36.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:36.162 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:14:36.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:14:36.170+0000 7f20c40e8480 -1 init, newargv = 0x7f20a8001080 newargc=15 2024-05-09T03:14:36.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[56721]: starting ceph client 2024-05-09T03:14:36.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[56721]: starting fuse 2024-05-09T03:14:36.187 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:14:36.188 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:14:36.188 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:14:36.189 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:14:36.190 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:14:36.190 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:14:36.190 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:14:36.309 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.56721.asok 2024-05-09T03:14:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:14:36.321 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56721.asok 2024-05-09T03:14:36.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:36.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56721.asok status 2024-05-09T03:14:36.469 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:14:36.469 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:14:36.469 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:14:36.469 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:14:36.469 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:14:36.469 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "56721", 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6291, 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6291 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:14:36.470 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1864888927 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1864888927 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6291 192.168.0.1:0/1864888927", 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1864888927", 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 60, 2024-05-09T03:14:36.471 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 57, 2024-05-09T03:14:36.472 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:14:36.472 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:14:36.472 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:14:36.472 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:14:36.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:36.478 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:14:36.478 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:36.495 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:14:36.495 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:36.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:36.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:36.561 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat a/ && find a/') 2024-05-09T03:14:36.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:14:36.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:14:36.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:14:36.660 INFO:teuthology.orchestra.run.smithi046.stdout: File: a/ 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 2ah/42d Inode: 1099511627776 Links: 4 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1316/ ubuntu) 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-05-09 03:12:18.899355075 +0000 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-05-09 03:12:18.905908763 +0000 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-05-09 03:12:18.905908763 +0000 2024-05-09T03:14:36.661 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-05-09T03:14:36.674 INFO:teuthology.orchestra.run.smithi046.stdout:a/ 2024-05-09T03:14:36.674 INFO:teuthology.orchestra.run.smithi046.stdout:a/c 2024-05-09T03:14:36.674 INFO:teuthology.orchestra.run.smithi046.stdout:a/c/d 2024-05-09T03:14:36.674 INFO:teuthology.orchestra.run.smithi046.stdout:a/b 2024-05-09T03:14:36.675 INFO:teuthology.orchestra.run.smithi046.stdout:a/b/d 2024-05-09T03:14:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:14:36.680 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-05-09T03:14:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.837+0000 7f46c7c89640 1 Processor -- start 2024-05-09T03:14:36.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.837+0000 7f46c7c89640 1 -- start start 2024-05-09T03:14:36.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c7c89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46c00cf6a0 0x7f46c00b7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c7c89640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c00b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c7c89640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46c0160830 con 0x7f46c00b7c90 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c7c89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f46c00b80a0 con 0x7f46c00cf6a0 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c7c89640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46c01640a0 con 0x7f46c00d23f0 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c5a25640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46c00cf6a0 0x7f46c00b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c5a25640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46c00cf6a0 0x7f46c00b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41328/0 (socket says 172.21.15.46:41328) 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.838+0000 7f46c5a25640 1 -- 172.21.15.46:0/1351815343 learned_addr learned my addr 172.21.15.46:0/1351815343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.839+0000 7f46c6226640 1 --2- 172.21.15.46:0/1351815343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c00b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.839+0000 7f46c5a25640 1 -- 172.21.15.46:0/1351815343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 msgr2=0x7f46c00b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.839+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1351815343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c00b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.839+0000 7f46c5a25640 1 -- 172.21.15.46:0/1351815343 >> v1:172.21.15.88:6789/0 conn(0x7f46c00b7c90 legacy=0x7f46c00d1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.839+0000 7f46c5a25640 1 -- 172.21.15.46:0/1351815343 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c0164380 con 0x7f46c00cf6a0 2024-05-09T03:14:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.839+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1351815343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c00b7750 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009870 tx=0x7f46bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3474cc8cee9d230 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c4a23640 1 -- 172.21.15.46:0/1351815343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc037070 con 0x7f46c00cf6a0 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c4a23640 1 -- 172.21.15.46:0/1351815343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46bc032440 con 0x7f46c00cf6a0 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c4a23640 1 -- 172.21.15.46:0/1351815343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc032810 con 0x7f46c00cf6a0 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c7c89640 1 -- 172.21.15.46:0/1351815343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 msgr2=0x7f46c00b7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c7c89640 1 --2- 172.21.15.46:0/1351815343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c00b7750 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009870 tx=0x7f46bc02a270 comp rx=0 tx=0).stop 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c7c89640 1 -- 172.21.15.46:0/1351815343 shutdown_connections 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c7c89640 1 --2- 172.21.15.46:0/1351815343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c00b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c7c89640 1 --2- 172.21.15.46:0/1351815343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c00b7750 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.840+0000 7f46c7c89640 1 -- 172.21.15.46:0/1351815343 >> 172.21.15.46:0/1351815343 conn(0x7f46c009b7d0 msgr2=0x7f46c009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.841+0000 7f46c7c89640 1 -- 172.21.15.46:0/1351815343 shutdown_connections 2024-05-09T03:14:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.841+0000 7f46c7c89640 1 -- 172.21.15.46:0/1351815343 wait complete. 2024-05-09T03:14:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.842+0000 7f46c7c89640 1 Processor -- start 2024-05-09T03:14:36.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.842+0000 7f46c7c89640 1 -- start start 2024-05-09T03:14:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c7c89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c00b7c90 0x7f46c0151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c7c89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c01f99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c7c89640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c01f9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c7c89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46c0164c20 con 0x7f46c00cf6a0 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c7c89640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f46c0164940 con 0x7f46c00b7c90 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c7c89640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46c00b2100 con 0x7f46c00d23f0 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c01f99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c01f99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41330/0 (socket says 172.21.15.46:41330) 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 -- 172.21.15.46:0/2089147626 learned_addr learned my addr 172.21.15.46:0/2089147626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5a25640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c00b7c90 0x7f46c0151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c6226640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c01f9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 -- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 msgr2=0x7f46c01f9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c01f9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 -- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c00b7c90 msgr2=0x7f46c0151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c00b7c90 0x7f46c0151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 -- 172.21.15.46:0/2089147626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c00c85c0 con 0x7f46c00cf6a0 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5a25640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c00b7c90 0x7f46c0151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46c5224640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c01f99f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c3a0 tx=0x7f46b000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.844+0000 7f46b67fc640 1 -- 172.21.15.46:0/2089147626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0017070 con 0x7f46c00cf6a0 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.845+0000 7f46b67fc640 1 -- 172.21.15.46:0/2089147626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b00052d0 con 0x7f46c00cf6a0 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.845+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46c00c88a0 con 0x7f46c00cf6a0 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.845+0000 7f46b67fc640 1 -- 172.21.15.46:0/2089147626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b0003830 con 0x7f46c00cf6a0 2024-05-09T03:14:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.845+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46c0160aa0 con 0x7f46c00cf6a0 2024-05-09T03:14:36.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.846+0000 7f46b67fc640 1 -- 172.21.15.46:0/2089147626 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f46b001a070 con 0x7f46c00cf6a0 2024-05-09T03:14:36.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.847+0000 7f46b67fc640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46a40762d0 0x7f46a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.847+0000 7f46b67fc640 1 -- 172.21.15.46:0/2089147626 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f46b0096aa0 con 0x7f46c00cf6a0 2024-05-09T03:14:36.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.847+0000 7f46c5a25640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46a40762d0 0x7f46a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.848+0000 7f46c5a25640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46a40762d0 0x7f46a40787e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46bc034000 tx=0x7f46bc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.848+0000 7f46cec1a740 1 Processor -- start 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.848+0000 7f46cec1a740 1 -- start start 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f02a390 0x55de0f3f9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f414240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f0096b0 0x55de0f414780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46cec1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55de0f3321a0 con 0x55de0f0014d0 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55de0f2eb170 con 0x55de0f02a390 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55de0f32e160 con 0x55de0f0096b0 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.849+0000 7f46c5224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f414240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f414240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41332/0 (socket says 172.21.15.46:41332) 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 -- 172.21.15.46:0/2583528553 learned_addr learned my addr 172.21.15.46:0/2583528553 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5a25640 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f02a390 0x55de0f3f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c6226640 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f0096b0 0x55de0f414780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 -- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f0096b0 msgr2=0x55de0f414780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f0096b0 0x55de0f414780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 -- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f02a390 msgr2=0x55de0f3f9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f02a390 0x55de0f3f9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 -- 172.21.15.46:0/2583528553 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0f2fd390 con 0x55de0f0014d0 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46c5224640 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f414240 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f46b0094e90 tx=0x7f46b00684a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410414af61c4d0df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46a37fe640 1 -- 172.21.15.46:0/2583528553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b00602c0 con 0x55de0f0014d0 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46a37fe640 1 -- 172.21.15.46:0/2583528553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b005fbc0 con 0x55de0f0014d0 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.850+0000 7f46a37fe640 1 -- 172.21.15.46:0/2583528553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b005f040 con 0x55de0f0014d0 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 -- 172.21.15.46:0/2583528553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 msgr2=0x55de0f414240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f414240 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f46b0094e90 tx=0x7f46b00684a0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 -- 172.21.15.46:0/2583528553 shutdown_connections 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f0096b0 0x55de0f414780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f414240 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 --2- 172.21.15.46:0/2583528553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f02a390 0x55de0f3f9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 -- 172.21.15.46:0/2583528553 >> 172.21.15.46:0/2583528553 conn(0x55de0f01e4e0 msgr2=0x55de0f3ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 -- 172.21.15.46:0/2583528553 shutdown_connections 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.851+0000 7f46cec1a740 1 -- 172.21.15.46:0/2583528553 wait complete. 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.852+0000 7f46cec1a740 1 -- start start 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.852+0000 7f46cec1a740 1 Processor -- start 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.853+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f3fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f412e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55de0f2b88c0 con 0x55de0f0014d0 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55de0f3e9ae0 con 0x55de0f0096b0 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55de0f411b30 con 0x55de0f02a390 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 10 client.0 _ll_register_callbacks cb 0x55de0f2d4860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41338/0 (socket says 172.21.15.46:41338) 2024-05-09T03:14:36.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 -- 172.21.15.46:0/4221438424 learned_addr learned my addr 172.21.15.46:0/4221438424 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5224640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f3fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c6226640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f412e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 -- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 msgr2=0x55de0f412e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f412e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 -- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 msgr2=0x55de0f3fcb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f3fcb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 -- 172.21.15.46:0/4221438424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0f10c7d0 con 0x55de0f0014d0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46c5a25640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3fc630 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009990 tx=0x7f46bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1223fc95a8da1b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46a0ff9640 1 -- 172.21.15.46:0/4221438424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc037040 con 0x55de0f0014d0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46b005fbc0 con 0x55de0f0014d0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.854+0000 7f46a0ff9640 1 -- 172.21.15.46:0/4221438424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f46bc03ce00 con 0x55de0f0014d0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46a0ff9640 1 -- 172.21.15.46:0/4221438424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f46bc056050 con 0x55de0f0014d0 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 10 client.6303 fetch_fsmap learned FSMap version 60 2024-05-09T03:14:36.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x55de0f095970 con 0x55de0f0014d0 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46a0ff9640 1 -- 172.21.15.46:0/4221438424 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b005fbc0 con 0x55de0f0014d0 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46a0ff9640 1 -- 172.21.15.46:0/4221438424 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 60) ==== 3494+0+0 (secure 0 0 0) 0x7f46bc059020 con 0x55de0f0014d0 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 10 client.6303 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 10 client.6303 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 20 client.6303 populate_metadata read hostname 'smithi046' 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0efe8f60 0x55de0f413360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 4 client.6303 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 --> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de0f33fdb0 con 0x55de0efe8f60 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46c5224640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0efe8f60 0x55de0f413360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.855+0000 7f46a0ff9640 10 client.6303 ms_handle_connect on v2:172.21.15.46:6832/1536534537 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46c5224640 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0efe8f60 0x55de0f413360 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b31441f321a3d90 server_cookie=d194a6a6a642c718 in_seq=0 out_seq=0 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46a0ff9640 1 -- 172.21.15.46:0/4221438424 <== mds.0 v2:172.21.15.46:6832/1536534537 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55de0f33fdb0 con 0x55de0efe8f60 2024-05-09T03:14:36.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46a0ff9640 10 client.6303 handle_command_reply: tid=0 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 client.6303 shutdown 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 msgr2=0x55de0f3fc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3fc630 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009990 tx=0x7f46bc004730 comp rx=0 tx=0).stop 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 shutdown_connections 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0efe8f60 0x55de0f413360 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f412e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f3fcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 --2- 172.21.15.46:0/4221438424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3fc630 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46a0ff9640 0 client.6303 ms_handle_reset on v2:172.21.15.46:6832/1536534537 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.856+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 >> 172.21.15.46:0/4221438424 conn(0x55de0f01e4e0 msgr2=0x55de0f3ceb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.857+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 shutdown_connections 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.857+0000 7f46cec1a740 1 -- 172.21.15.46:0/4221438424 wait complete. 2024-05-09T03:14:36.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.857+0000 7f46cec1a740 20 client.6303 trim_cache size 0 max 16384 2024-05-09T03:14:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.872+0000 7f46cec1a740 1 Processor -- start 2024-05-09T03:14:36.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.872+0000 7f46cec1a740 1 -- start start 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3c8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f412e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f413340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46cec1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46c0203060 con 0x55de0f0014d0 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f46c02020a0 con 0x55de0f0096b0 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46c00c71f0 con 0x55de0f02a390 2024-05-09T03:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41348/0 (socket says 172.21.15.46:41348) 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 -- 172.21.15.46:0/1010588636 learned_addr learned my addr 172.21.15.46:0/1010588636 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c6226640 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f413340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5224640 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f412e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 -- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 msgr2=0x55de0f413340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f413340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 -- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 msgr2=0x55de0f412e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f412e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 -- 172.21.15.46:0/1010588636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0efdaf60 con 0x55de0f0014d0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3c8850 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0496b0 tx=0x7f46bc03f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be351a96a7f7c4f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46a2ffd640 1 -- 172.21.15.46:0/1010588636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc037020 con 0x55de0f0014d0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46a2ffd640 1 -- 172.21.15.46:0/1010588636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46bc04e070 con 0x55de0f0014d0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.874+0000 7f46a2ffd640 1 -- 172.21.15.46:0/1010588636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc004450 con 0x55de0f0014d0 2024-05-09T03:14:36.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 -- 172.21.15.46:0/1010588636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 msgr2=0x55de0f3c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3c8850 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0496b0 tx=0x7f46bc03f070 comp rx=0 tx=0).stop 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 -- 172.21.15.46:0/1010588636 shutdown_connections 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f413340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f412e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1010588636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3c8850 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 -- 172.21.15.46:0/1010588636 >> 172.21.15.46:0/1010588636 conn(0x55de0f01e4e0 msgr2=0x55de0f4139c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 -- 172.21.15.46:0/1010588636 shutdown_connections 2024-05-09T03:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.875+0000 7f46cec1a740 1 -- 172.21.15.46:0/1010588636 wait complete. 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.876+0000 7f46cec1a740 1 -- start start 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.876+0000 7f46cec1a740 1 Processor -- start 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f493f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f48fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55de0f3d00d0 con 0x55de0f0014d0 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f46c00c7840 con 0x55de0f0096b0 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55de0f3c8d90 con 0x55de0f02a390 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 10 client.0 _ll_register_callbacks cb 0x7f46bc0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41356/0 (socket says 172.21.15.46:41356) 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 -- 172.21.15.46:0/1239826754 learned_addr learned my addr 172.21.15.46:0/1239826754 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5224640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f493f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c6226640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f48fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 -- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 msgr2=0x55de0f48fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f48fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 -- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 msgr2=0x55de0f493f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f493f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 -- 172.21.15.46:0/1239826754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0f416770 con 0x55de0f0014d0 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c6226640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f48fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5224640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f493f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46c5a25640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3cdc10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0496e0 tx=0x7f46bc009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b666e0805ea16eef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46a0ff9640 1 -- 172.21.15.46:0/1239826754 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc037070 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46a0ff9640 1 -- 172.21.15.46:0/1239826754 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f46bc049b20 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46bc04e070 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46a0ff9640 1 -- 172.21.15.46:0/1239826754 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=60) ==== 24+0+0 (secure 0 0 0) 0x7f46bc066050 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 10 client.6309 fetch_fsmap learned FSMap version 60 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.878+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=60,monmap=2+}) -- 0x55de0f4133f0 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46a0ff9640 1 -- 172.21.15.46:0/1239826754 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46bc04e070 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46a0ff9640 1 -- 172.21.15.46:0/1239826754 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 60) ==== 3494+0+0 (secure 0 0 0) 0x7f46bc069020 con 0x55de0f0014d0 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46cec1a740 10 client.6309 fetch_fsmap finished waiting for FSMap version 60 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46cec1a740 10 client.6309 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46cec1a740 20 client.6309 populate_metadata read hostname 'smithi046' 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0f48ff70 0x55de0f3fa420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46cec1a740 4 client.6309 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 --> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de0f339c20 con 0x55de0f48ff70 2024-05-09T03:14:36.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46c5224640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0f48ff70 0x55de0f3fa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:36.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46a0ff9640 10 client.6309 ms_handle_connect on v2:172.21.15.46:6832/1536534537 2024-05-09T03:14:36.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.879+0000 7f46c5224640 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0f48ff70 0x55de0f3fa420 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0b2cfa9f3f7bdd server_cookie=99f13be5f1646bef in_seq=0 out_seq=0 2024-05-09T03:14:36.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46a0ff9640 1 -- 172.21.15.46:0/1239826754 <== mds.0 v2:172.21.15.46:6832/1536534537 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55de0f339c20 con 0x55de0f48ff70 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46a0ff9640 10 client.6309 handle_command_reply: tid=0 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 client.6309 shutdown 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 msgr2=0x55de0f3cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3cdc10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0496e0 tx=0x7f46bc009240 comp rx=0 tx=0).stop 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 shutdown_connections 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:6832/1536534537,v1:172.21.15.46:6833/1536534537] conn(0x55de0f48ff70 0x55de0f3fa420 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55de0f02a390 0x55de0f48fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46a0ff9640 0 client.6309 ms_handle_reset on v2:172.21.15.46:6832/1536534537 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55de0f0096b0 0x55de0f493f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 --2- 172.21.15.46:0/1239826754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55de0f0014d0 0x55de0f3cdc10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 >> 172.21.15.46:0/1239826754 conn(0x55de0f01e4e0 msgr2=0x55de0f412be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 shutdown_connections 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.893+0000 7f46cec1a740 1 -- 172.21.15.46:0/1239826754 wait complete. 2024-05-09T03:14:36.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46cec1a740 20 client.6309 trim_cache size 0 max 16384 2024-05-09T03:14:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:14:36.896 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0 2024-05-09T03:14:36.896 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46a40762d0 msgr2=0x7f46a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46a40762d0 0x7f46a40787e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f46bc034000 tx=0x7f46bc02e000 comp rx=0 tx=0).stop 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 msgr2=0x7f46c01f99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c01f99f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f46b000c3a0 tx=0x7f46b000c860 comp rx=0 tx=0).stop 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 shutdown_connections 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46c00d23f0 0x7f46c01f9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46a40762d0 0x7f46a40787e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46c00cf6a0 0x7f46c01f99f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 --2- 172.21.15.46:0/2089147626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46c00b7c90 0x7f46c0151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 >> 172.21.15.46:0/2089147626 conn(0x7f46c009b7d0 msgr2=0x7f46c015ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:36.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 shutdown_connections 2024-05-09T03:14:36.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:36.894+0000 7f46c7c89640 1 -- 172.21.15.46:0/2089147626 wait complete. 2024-05-09T03:14:36.906 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:14:36.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:14:36.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:14:36.996 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:14:36.996 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:14:36.996 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:14:36.996 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:37.013 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:14:37.013 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:37.070 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:14:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.224+0000 7fea7e254640 1 Processor -- start 2024-05-09T03:14:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.224+0000 7fea7e254640 1 -- start start 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea7e254640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fea780b7e70 0x7fea780b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea7e254640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea781605b0 0x7fea780b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea7e254640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea780d3400 con 0x7fea780d2e90 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea7e254640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fea780d4a60 con 0x7fea781605b0 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea7e254640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fea780d4d40 con 0x7fea780b7e70 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea781605b0 0x7fea780b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea781605b0 0x7fea780b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41360/0 (socket says 172.21.15.46:41360) 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 -- 172.21.15.46:0/1874530836 learned_addr learned my addr 172.21.15.46:0/1874530836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea76ffd640 1 --2- 172.21.15.46:0/1874530836 >> v2:172.21.15.88:3300/0 conn(0x7fea780b7e70 0x7fea780b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 -- 172.21.15.46:0/1874530836 >> v2:172.21.15.88:3300/0 conn(0x7fea780b7e70 msgr2=0x7fea780b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 --2- 172.21.15.46:0/1874530836 >> v2:172.21.15.88:3300/0 conn(0x7fea780b7e70 0x7fea780b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 -- 172.21.15.46:0/1874530836 >> v1:172.21.15.88:6789/0 conn(0x7fea780d2e90 legacy=0x7fea780b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.226+0000 7fea77fff640 1 -- 172.21.15.46:0/1874530836 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea780d5020 con 0x7fea781605b0 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.227+0000 7fea77fff640 1 --2- 172.21.15.46:0/1874530836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea781605b0 0x7fea780b97e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fea7808e880 tx=0x7fea7002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1aa60486a88666c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.227+0000 7fea767fc640 1 -- 172.21.15.46:0/1874530836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70036070 con 0x7fea781605b0 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.227+0000 7fea767fc640 1 -- 172.21.15.46:0/1874530836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea70031600 con 0x7fea781605b0 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.227+0000 7fea767fc640 1 -- 172.21.15.46:0/1874530836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea700319d0 con 0x7fea781605b0 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 -- 172.21.15.46:0/1874530836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea781605b0 msgr2=0x7fea780b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 --2- 172.21.15.46:0/1874530836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea781605b0 0x7fea780b97e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fea7808e880 tx=0x7fea7002a6a0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 -- 172.21.15.46:0/1874530836 shutdown_connections 2024-05-09T03:14:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 --2- 172.21.15.46:0/1874530836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea781605b0 0x7fea780b97e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 --2- 172.21.15.46:0/1874530836 >> v2:172.21.15.88:3300/0 conn(0x7fea780b7e70 0x7fea780b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 -- 172.21.15.46:0/1874530836 >> 172.21.15.46:0/1874530836 conn(0x7fea7808d390 msgr2=0x7fea7808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 -- 172.21.15.46:0/1874530836 shutdown_connections 2024-05-09T03:14:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.228+0000 7fea7e254640 1 -- 172.21.15.46:0/1874530836 wait complete. 2024-05-09T03:14:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.230+0000 7fea7e254640 1 Processor -- start 2024-05-09T03:14:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.230+0000 7fea7e254640 1 -- start start 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea7e254640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea780d2e90 0x7fea780cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea7e254640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea781605b0 0x7fea78159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea7e254640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 0x7fea78153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea7e254640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fea780b4d80 con 0x7fea78153850 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea7e254640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fea780b4c00 con 0x7fea781605b0 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea7e254640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fea780b4fc0 con 0x7fea780d2e90 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 0x7fea78153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 0x7fea78153c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41374/0 (socket says 172.21.15.46:41374) 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 -- 172.21.15.46:0/1694567217 learned_addr learned my addr 172.21.15.46:0/1694567217 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea76ffd640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea781605b0 0x7fea78159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea777fe640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea780d2e90 0x7fea780cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 -- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea780d2e90 msgr2=0x7fea780cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea780d2e90 0x7fea780cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 -- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea781605b0 msgr2=0x7fea78159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea781605b0 0x7fea78159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.231+0000 7fea77fff640 1 -- 172.21.15.46:0/1694567217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea780cdb10 con 0x7fea78153850 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.232+0000 7fea77fff640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 0x7fea78153c60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fea70009f40 tx=0x7fea70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.232+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70036050 con 0x7fea78153850 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.232+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea70042070 con 0x7fea78153850 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.232+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70009510 con 0x7fea78153850 2024-05-09T03:14:37.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.232+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea780cddf0 con 0x7fea78153850 2024-05-09T03:14:37.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.232+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea780ce3b0 con 0x7fea78153850 2024-05-09T03:14:37.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.233+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea7806a270 con 0x7fea78153850 2024-05-09T03:14:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.234+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fea70007bf0 con 0x7fea78153850 2024-05-09T03:14:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.234+0000 7fea5ffff640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea64076300 0x7fea64078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.234+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7fea700b84b0 con 0x7fea78153850 2024-05-09T03:14:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.235+0000 7fea777fe640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea64076300 0x7fea64078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.235+0000 7fea777fe640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea64076300 0x7fea64078810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fea7806f130 tx=0x7fea68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:37.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.239+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fea70081600 con 0x7fea78153850 2024-05-09T03:14:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.443+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fea780d00a0 con 0x7fea78153850 2024-05-09T03:14:37.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.443+0000 7fea5ffff640 1 -- 172.21.15.46:0/1694567217 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v57) ==== 80+0+242 (secure 0 0 0) 0x7fea70085900 con 0x7fea78153850 2024-05-09T03:14:37.444 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/103238544 2024-05-10T03:12:53.158166+0000 2024-05-09T03:14:37.444 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/103238544 2024-05-10T03:12:53.158166+0000 2024-05-09T03:14:37.444 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/4263404372 2024-05-10T03:12:22.212819+0000 2024-05-09T03:14:37.445 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/4263404372 2024-05-10T03:12:22.212819+0000 2024-05-09T03:14:37.445 INFO:teuthology.orchestra.run.smithi046.stderr:listed 4 entries 2024-05-09T03:14:37.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.445+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea64076300 msgr2=0x7fea64078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.445+0000 7fea7e254640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea64076300 0x7fea64078810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fea7806f130 tx=0x7fea68009040 comp rx=0 tx=0).stop 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.445+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 msgr2=0x7fea78153c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.445+0000 7fea7e254640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 0x7fea78153c60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fea70009f40 tx=0x7fea70004a40 comp rx=0 tx=0).stop 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 shutdown_connections 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea64076300 0x7fea64078810 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea78153850 0x7fea78153c60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea781605b0 0x7fea78159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 --2- 172.21.15.46:0/1694567217 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea780d2e90 0x7fea780cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 >> 172.21.15.46:0/1694567217 conn(0x7fea7808d390 msgr2=0x7fea780d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 shutdown_connections 2024-05-09T03:14:37.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.446+0000 7fea7e254640 1 -- 172.21.15.46:0/1694567217 wait complete. 2024-05-09T03:14:37.457 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:14:37.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:37.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:37.482 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[56721]: fuse finished with error 0 and tester_r 0 2024-05-09T03:14:37.484 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:14:37.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:37.484 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:14:37.549 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:14:37.549 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:14:37.549 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:37.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:14:37.566 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:14:37.567 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:14:37.567 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:37.605 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:14:37.606 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:14:37.606 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:14:37.606 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:37.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:14:37.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.760+0000 7f2f763b2640 1 Processor -- start 2024-05-09T03:14:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.760+0000 7f2f763b2640 1 -- start start 2024-05-09T03:14:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f763b2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f70160610 0x7f2f700b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f763b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700b7d20 0x7f2f700b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f763b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f700b8130 con 0x7f2f700d57a0 2024-05-09T03:14:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f763b2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2f700d5bb0 con 0x7f2f70160610 2024-05-09T03:14:37.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f763b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2f700d49e0 con 0x7f2f700b7d20 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f7494f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700b7d20 0x7f2f700b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f6f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2f70160610 0x7f2f700b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f6ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2f700d57a0 0x7f2f700b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:48556/0 (socket says 172.21.15.46:48556) 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f6ffff640 1 -- 172.21.15.46:0/2744592524 learned_addr learned my addr 172.21.15.46:0/2744592524 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f6effd640 1 -- 172.21.15.46:0/2744592524 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186219189 0 0) 0x7f2f60001110 con 0x7f2f700d57a0 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f6effd640 1 -- 172.21.15.46:0/2744592524 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f58003520 con 0x7f2f700d57a0 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f7494f640 1 -- 172.21.15.46:0/2744592524 >> v2:172.21.15.88:3300/0 conn(0x7f2f70160610 msgr2=0x7f2f700b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f7494f640 1 --2- 172.21.15.46:0/2744592524 >> v2:172.21.15.88:3300/0 conn(0x7f2f70160610 0x7f2f700b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f7494f640 1 -- 172.21.15.46:0/2744592524 >> v1:172.21.15.88:6789/0 conn(0x7f2f700d57a0 legacy=0x7f2f700b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f7494f640 1 -- 172.21.15.46:0/2744592524 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f700d4cc0 con 0x7f2f700b7d20 2024-05-09T03:14:37.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.762+0000 7f2f6f7fe640 1 --2- 172.21.15.46:0/2744592524 >> v2:172.21.15.88:3300/0 conn(0x7f2f70160610 0x7f2f700b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:14:37.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.763+0000 7f2f7494f640 1 --2- 172.21.15.46:0/2744592524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700b7d20 0x7f2f700b9450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009d90 tx=0x7f2f6802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba895f7e37962299 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.763+0000 7f2f6effd640 1 -- 172.21.15.46:0/2744592524 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68007740 con 0x7f2f700b7d20 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.763+0000 7f2f6effd640 1 -- 172.21.15.46:0/2744592524 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f680078e0 con 0x7f2f700b7d20 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.763+0000 7f2f6effd640 1 -- 172.21.15.46:0/2744592524 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f68007c90 con 0x7f2f700b7d20 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 -- 172.21.15.46:0/2744592524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700b7d20 msgr2=0x7f2f700b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 --2- 172.21.15.46:0/2744592524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700b7d20 0x7f2f700b9450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f68009d90 tx=0x7f2f6802a660 comp rx=0 tx=0).stop 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 -- 172.21.15.46:0/2744592524 shutdown_connections 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 --2- 172.21.15.46:0/2744592524 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700b7d20 0x7f2f700b9450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 --2- 172.21.15.46:0/2744592524 >> v2:172.21.15.88:3300/0 conn(0x7f2f70160610 0x7f2f700b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 -- 172.21.15.46:0/2744592524 >> 172.21.15.46:0/2744592524 conn(0x7f2f700743e0 msgr2=0x7f2f700d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 -- 172.21.15.46:0/2744592524 shutdown_connections 2024-05-09T03:14:37.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.764+0000 7f2f763b2640 1 -- 172.21.15.46:0/2744592524 wait complete. 2024-05-09T03:14:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.766+0000 7f2f763b2640 1 Processor -- start 2024-05-09T03:14:37.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.766+0000 7f2f763b2640 1 -- start start 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f763b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f700b7d20 0x7f2f70156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f763b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700d57a0 0x7f2f70152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f763b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70160610 0x7f2f70157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f763b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f700d15b0 con 0x7f2f70160610 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f763b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2f70160a20 con 0x7f2f700b7d20 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f763b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2f700d1920 con 0x7f2f700d57a0 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f7494f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70160610 0x7f2f70157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f6ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f700b7d20 0x7f2f70156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f7494f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70160610 0x7f2f70157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41376/0 (socket says 172.21.15.46:41376) 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.767+0000 7f2f6f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700d57a0 0x7f2f70152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f7494f640 1 -- 172.21.15.46:0/3988881166 learned_addr learned my addr 172.21.15.46:0/3988881166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6ffff640 1 -- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700d57a0 msgr2=0x7f2f70152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6ffff640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700d57a0 0x7f2f70152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6ffff640 1 -- 172.21.15.46:0/3988881166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70160610 msgr2=0x7f2f70157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6ffff640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70160610 0x7f2f70157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6ffff640 1 -- 172.21.15.46:0/3988881166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f70153180 con 0x7f2f700b7d20 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6ffff640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f700b7d20 0x7f2f70156020 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000c390 tx=0x7f2f60001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60017070 con 0x7f2f700b7d20 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f60005ce0 con 0x7f2f700b7d20 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f600072d0 con 0x7f2f700b7d20 2024-05-09T03:14:37.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f70154e00 con 0x7f2f700b7d20 2024-05-09T03:14:37.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.768+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f701553c0 con 0x7f2f700b7d20 2024-05-09T03:14:37.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.769+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7006a270 con 0x7f2f700b7d20 2024-05-09T03:14:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.770+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2f6001a070 con 0x7f2f700b7d20 2024-05-09T03:14:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.770+0000 7f2f6cff9640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f5c0762d0 0x7f2f5c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.770+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f2f60096840 con 0x7f2f700b7d20 2024-05-09T03:14:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.771+0000 7f2f6f7fe640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f5c0762d0 0x7f2f5c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.771+0000 7f2f6f7fe640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f5c0762d0 0x7f2f5c0787e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f70155fb0 tx=0x7f2f64009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:37.776+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2f700b8130 con 0x7f2f700b7d20 2024-05-09T03:14:38.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.005+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f701e78e0 con 0x7f2f700b7d20 2024-05-09T03:14:38.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.006+0000 7f2f6cff9640 1 -- 172.21.15.46:0/3988881166 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5183 (secure 0 0 0) 0x7f2f6005faa0 con 0x7f2f700b7d20 2024-05-09T03:14:38.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:38.007 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:56.169641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":16,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-05-09T03:14:38.007 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f5c0762d0 msgr2=0x7f2f5c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f5c0762d0 0x7f2f5c0787e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f70155fb0 tx=0x7f2f64009190 comp rx=0 tx=0).stop 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f700b7d20 msgr2=0x7f2f70156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f700b7d20 0x7f2f70156020 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000c390 tx=0x7f2f60001e70 comp rx=0 tx=0).stop 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 shutdown_connections 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f5c0762d0 0x7f2f5c0787e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f70160610 0x7f2f70157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f700d57a0 0x7f2f70152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 --2- 172.21.15.46:0/3988881166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f700b7d20 0x7f2f70156020 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 >> 172.21.15.46:0/3988881166 conn(0x7f2f700743e0 msgr2=0x7f2f700bb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 shutdown_connections 2024-05-09T03:14:38.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.008+0000 7f2f763b2640 1 -- 172.21.15.46:0/3988881166 wait complete. 2024-05-09T03:14:38.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:38.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.134+0000 7f982717e640 1 Processor -- start 2024-05-09T03:14:38.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.134+0000 7f982717e640 1 -- start start 2024-05-09T03:14:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982717e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f98200b5210 0x7f98201613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982717e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982717e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98200b77c0 0x7f98200b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982717e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f98200b5620 con 0x7f98200b77c0 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982717e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f98200b8110 con 0x7f98200b5210 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982717e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f98200d5280 con 0x7f98200d4800 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982571b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98200b77c0 0x7f98200b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982571b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98200b77c0 0x7f98200b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41382/0 (socket says 172.21.15.46:41382) 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f982571b640 1 -- 172.21.15.46:0/2792826265 learned_addr learned my addr 172.21.15.46:0/2792826265 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.135+0000 7f9824f1a640 1 --2- 172.21.15.46:0/2792826265 >> v2:172.21.15.88:3300/0 conn(0x7f98200b5210 0x7f98201613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f981ffff640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f982571b640 1 -- 172.21.15.46:0/2792826265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 msgr2=0x7f98200d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f982571b640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f982571b640 1 -- 172.21.15.46:0/2792826265 >> v2:172.21.15.88:3300/0 conn(0x7f98200b5210 msgr2=0x7f98201613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f982571b640 1 --2- 172.21.15.46:0/2792826265 >> v2:172.21.15.88:3300/0 conn(0x7f98200b5210 0x7f98201613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f982571b640 1 -- 172.21.15.46:0/2792826265 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98200d5560 con 0x7f98200b77c0 2024-05-09T03:14:38.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f9824f1a640 1 --2- 172.21.15.46:0/2792826265 >> v2:172.21.15.88:3300/0 conn(0x7f98200b5210 0x7f98201613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:38.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f981ffff640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:38.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.136+0000 7f982571b640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b77c0 0x7f98200b7bd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f982008ed70 tx=0x7f981802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41900e0d2ad6009b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f981f7fe640 1 -- 172.21.15.46:0/2792826265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9818036070 con 0x7f98200b77c0 2024-05-09T03:14:38.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f981f7fe640 1 -- 172.21.15.46:0/2792826265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9818031600 con 0x7f98200b77c0 2024-05-09T03:14:38.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f981f7fe640 1 -- 172.21.15.46:0/2792826265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98180319d0 con 0x7f98200b77c0 2024-05-09T03:14:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 -- 172.21.15.46:0/2792826265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b77c0 msgr2=0x7f98200b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b77c0 0x7f98200b7bd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f982008ed70 tx=0x7f981802a610 comp rx=0 tx=0).stop 2024-05-09T03:14:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 -- 172.21.15.46:0/2792826265 shutdown_connections 2024-05-09T03:14:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b77c0 0x7f98200b7bd0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 --2- 172.21.15.46:0/2792826265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 --2- 172.21.15.46:0/2792826265 >> v2:172.21.15.88:3300/0 conn(0x7f98200b5210 0x7f98201613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 -- 172.21.15.46:0/2792826265 >> 172.21.15.46:0/2792826265 conn(0x7f98200d29a0 msgr2=0x7f98200d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 -- 172.21.15.46:0/2792826265 shutdown_connections 2024-05-09T03:14:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.137+0000 7f982717e640 1 -- 172.21.15.46:0/2792826265 wait complete. 2024-05-09T03:14:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.139+0000 7f982717e640 1 Processor -- start 2024-05-09T03:14:38.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.139+0000 7f982717e640 1 -- start start 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982717e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 0x7f982015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982717e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200b77c0 0x7f98200ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982717e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982717e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98200d0ae0 con 0x7f98200b5210 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982717e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9820160ea0 con 0x7f98200b77c0 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982717e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9820163e60 con 0x7f98200d4800 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 0x7f982015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 0x7f982015a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41392/0 (socket says 172.21.15.46:41392) 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 -- 172.21.15.46:0/4114057313 learned_addr learned my addr 172.21.15.46:0/4114057313 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f981ffff640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200b77c0 0x7f98200ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f982571b640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 -- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 msgr2=0x7f98200cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 -- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200b77c0 msgr2=0x7f98200ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200b77c0 0x7f98200ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 -- 172.21.15.46:0/4114057313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98200ca840 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f9824f1a640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 0x7f982015a5e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f981400c7f0 tx=0x7f981400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814015070 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9814007820 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.141+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9814007c10 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.142+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98200cab20 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.142+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98200cb0e0 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.142+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982006a270 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.143+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9814018070 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.144+0000 7f981d7fa640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98040762d0 0x7f98040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.144+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f9814097b40 con 0x7f98200b5210 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.144+0000 7f981ffff640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98040762d0 0x7f98040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.144+0000 7f981ffff640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98040762d0 0x7f98040787e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98201ea170 tx=0x7f980c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.148+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9814060e30 con 0x7f98200b5210 2024-05-09T03:14:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.383+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98200d40d0 con 0x7f98200b5210 2024-05-09T03:14:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.384+0000 7f981d7fa640 1 -- 172.21.15.46:0/4114057313 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5183 (secure 0 0 0) 0x7f9814065130 con 0x7f98200b5210 2024-05-09T03:14:38.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:38.385 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:56.169641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":16,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-05-09T03:14:38.385 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98040762d0 msgr2=0x7f98040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98040762d0 0x7f98040787e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98201ea170 tx=0x7f980c008040 comp rx=0 tx=0).stop 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 msgr2=0x7f982015a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 0x7f982015a5e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f981400c7f0 tx=0x7f981400ccb0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 shutdown_connections 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98200d4800 0x7f98200cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200b77c0 0x7f98200ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98040762d0 0x7f98040787e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 --2- 172.21.15.46:0/4114057313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98200b5210 0x7f982015a5e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 >> 172.21.15.46:0/4114057313 conn(0x7f98200d29a0 msgr2=0x7f98200b6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 shutdown_connections 2024-05-09T03:14:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.386+0000 7f982717e640 1 -- 172.21.15.46:0/4114057313 wait complete. 2024-05-09T03:14:38.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.512+0000 7f801e4bd640 1 Processor -- start 2024-05-09T03:14:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.512+0000 7f801e4bd640 1 -- start start 2024-05-09T03:14:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f801e4bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b5210 0x7f80181613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f801e4bd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f80180d4800 0x7f80180d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f801e4bd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80180d5280 con 0x7f80180b77c0 2024-05-09T03:14:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f801e4bd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f80180b5620 con 0x7f80180d4800 2024-05-09T03:14:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f801e4bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f80181609a0 con 0x7f80180b5210 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f80180d4800 0x7f80180d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f80180d4800 0x7f80180d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41394/0 (socket says 172.21.15.46:41394) 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 -- 172.21.15.46:0/502665516 learned_addr learned my addr 172.21.15.46:0/502665516 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f8017fff640 1 --2- 172.21.15.46:0/502665516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b5210 0x7f80181613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 -- 172.21.15.46:0/502665516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b5210 msgr2=0x7f80181613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 --2- 172.21.15.46:0/502665516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b5210 0x7f80181613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 -- 172.21.15.46:0/502665516 >> v1:172.21.15.88:6789/0 conn(0x7f80180b77c0 legacy=0x7f80180b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.513+0000 7f80177fe640 1 -- 172.21.15.46:0/502665516 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80180d5560 con 0x7f80180d4800 2024-05-09T03:14:38.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.514+0000 7f80177fe640 1 --2- 172.21.15.46:0/502665516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180d4800 0x7f80180d4c10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f800c009870 tx=0x7f800c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716db6076dde3143 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.514+0000 7f8016ffd640 1 -- 172.21.15.46:0/502665516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800c037070 con 0x7f80180d4800 2024-05-09T03:14:38.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f8016ffd640 1 -- 172.21.15.46:0/502665516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f800c032440 con 0x7f80180d4800 2024-05-09T03:14:38.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f8016ffd640 1 -- 172.21.15.46:0/502665516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800c032810 con 0x7f80180d4800 2024-05-09T03:14:38.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 -- 172.21.15.46:0/502665516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180d4800 msgr2=0x7f80180d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 --2- 172.21.15.46:0/502665516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180d4800 0x7f80180d4c10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f800c009870 tx=0x7f800c02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 -- 172.21.15.46:0/502665516 shutdown_connections 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 --2- 172.21.15.46:0/502665516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180d4800 0x7f80180d4c10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 --2- 172.21.15.46:0/502665516 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b5210 0x7f80181613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 -- 172.21.15.46:0/502665516 >> 172.21.15.46:0/502665516 conn(0x7f80180d29a0 msgr2=0x7f80180d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 -- 172.21.15.46:0/502665516 shutdown_connections 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.515+0000 7f801e4bd640 1 -- 172.21.15.46:0/502665516 wait complete. 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.517+0000 7f801e4bd640 1 Processor -- start 2024-05-09T03:14:38.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.517+0000 7f801e4bd640 1 -- start start 2024-05-09T03:14:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.518+0000 7f801e4bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 0x7f801814cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.518+0000 7f801e4bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b77c0 0x7f80180ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.518+0000 7f801e4bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80180d4800 0x7f801814c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.518+0000 7f801e4bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80180d0ae0 con 0x7f80180b5210 2024-05-09T03:14:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.518+0000 7f801e4bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8018160ea0 con 0x7f80180d4800 2024-05-09T03:14:38.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.518+0000 7f801e4bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8018163e60 con 0x7f80180b77c0 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 0x7f801814cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 0x7f801814cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41404/0 (socket says 172.21.15.46:41404) 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 -- 172.21.15.46:0/3798716705 learned_addr learned my addr 172.21.15.46:0/3798716705 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f80177fe640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b77c0 0x7f80180ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f801ca5a640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80180d4800 0x7f801814c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 -- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b77c0 msgr2=0x7f80180ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b77c0 0x7f80180ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 -- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80180d4800 msgr2=0x7f801814c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80180d4800 0x7f801814c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 -- 172.21.15.46:0/3798716705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80180caa90 con 0x7f80180b5210 2024-05-09T03:14:38.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8017fff640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 0x7f801814cb90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f800800d600 tx=0x7f800800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80080125f0 con 0x7f80180b5210 2024-05-09T03:14:38.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8008019070 con 0x7f80180b5210 2024-05-09T03:14:38.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80180cad70 con 0x7f80180b5210 2024-05-09T03:14:38.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80080129f0 con 0x7f80180b5210 2024-05-09T03:14:38.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.519+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80180cb330 con 0x7f80180b5210 2024-05-09T03:14:38.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.520+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f801806a270 con 0x7f80180b5210 2024-05-09T03:14:38.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.521+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8008012b90 con 0x7f80180b5210 2024-05-09T03:14:38.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.522+0000 7f8014ff9640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80040762d0 0x7f80040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.522+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f8008097ab0 con 0x7f80180b5210 2024-05-09T03:14:38.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.522+0000 7f80177fe640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80040762d0 0x7f80040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.522+0000 7f80177fe640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80040762d0 0x7f80040787e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f800c034000 tx=0x7f800c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.526+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8008060c00 con 0x7f80180b5210 2024-05-09T03:14:38.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.719+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80180d40d0 con 0x7f80180b5210 2024-05-09T03:14:38.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.720+0000 7f8014ff9640 1 -- 172.21.15.46:0/3798716705 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16850 (secure 0 0 0) 0x7f8008064f00 con 0x7f80180b5210 2024-05-09T03:14:38.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:38.721 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":57,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:12:53.158229+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:12:03.923965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T03:12:05.185724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/103238544":"2024-05-10T03:12:53.158166+0000","172.21.15.46:6834/103238544":"2024-05-10T03:12:53.158166+0000","172.21.15.46:6835/4263404372":"2024-05-10T03:12:22.212819+0000","172.21.15.46:6834/4263404372":"2024-05-10T03:12:22.212819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80040762d0 msgr2=0x7f80040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80040762d0 0x7f80040787e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f800c034000 tx=0x7f800c02e000 comp rx=0 tx=0).stop 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 msgr2=0x7f801814cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 0x7f801814cb90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f800800d600 tx=0x7f800800dac0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 shutdown_connections 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80180d4800 0x7f801814c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80180b77c0 0x7f80180ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80040762d0 0x7f80040787e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 --2- 172.21.15.46:0/3798716705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80180b5210 0x7f801814cb90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 >> 172.21.15.46:0/3798716705 conn(0x7f80180d29a0 msgr2=0x7f80180b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:38.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 shutdown_connections 2024-05-09T03:14:38.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.722+0000 7f801e4bd640 1 -- 172.21.15.46:0/3798716705 wait complete. 2024-05-09T03:14:38.734 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:14:38.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.849+0000 7fc1b662b640 1 Processor -- start 2024-05-09T03:14:38.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.849+0000 7fc1b662b640 1 -- start start 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b662b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00d57a0 0x7fc1b0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b662b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1b00b4af0 0x7fc1b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b662b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1b00b77e0 0x7fc1b00b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b662b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc1b00d5bb0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b662b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc1b00b8130 con 0x7fc1b00b77e0 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b662b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b00d49e0 con 0x7fc1b00d57a0 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1b00b4af0 0x7fc1b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1b00b4af0 0x7fc1b00b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41408/0 (socket says 172.21.15.46:41408) 2024-05-09T03:14:38.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 -- 172.21.15.46:0/2811367640 learned_addr learned my addr 172.21.15.46:0/2811367640 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1affff640 1 --2- 172.21.15.46:0/2811367640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00d57a0 0x7fc1b0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1b4bc8640 1 --2- 172.21.15.46:0/2811367640 >> v2:172.21.15.88:3300/0 conn(0x7fc1b00b77e0 0x7fc1b00b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 -- 172.21.15.46:0/2811367640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00d57a0 msgr2=0x7fc1b0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 --2- 172.21.15.46:0/2811367640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00d57a0 0x7fc1b0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 -- 172.21.15.46:0/2811367640 >> v2:172.21.15.88:3300/0 conn(0x7fc1b00b77e0 msgr2=0x7fc1b00b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 --2- 172.21.15.46:0/2811367640 >> v2:172.21.15.88:3300/0 conn(0x7fc1b00b77e0 0x7fc1b00b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.851+0000 7fc1a7fff640 1 -- 172.21.15.46:0/2811367640 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b00d4cc0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.852+0000 7fc1a7fff640 1 --2- 172.21.15.46:0/2811367640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b00b4f00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc1b008f200 tx=0x7fc19802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7457eccc1f98968d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.852+0000 7fc1af7fe640 1 -- 172.21.15.46:0/2811367640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc198037070 con 0x7fc1b00b4af0 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.852+0000 7fc1af7fe640 1 -- 172.21.15.46:0/2811367640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc198032440 con 0x7fc1b00b4af0 2024-05-09T03:14:38.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.852+0000 7fc1af7fe640 1 -- 172.21.15.46:0/2811367640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc198032810 con 0x7fc1b00b4af0 2024-05-09T03:14:38.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 -- 172.21.15.46:0/2811367640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 msgr2=0x7fc1b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 --2- 172.21.15.46:0/2811367640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b00b4f00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc1b008f200 tx=0x7fc19802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 -- 172.21.15.46:0/2811367640 shutdown_connections 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 --2- 172.21.15.46:0/2811367640 >> v2:172.21.15.88:3300/0 conn(0x7fc1b00b77e0 0x7fc1b00b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 --2- 172.21.15.46:0/2811367640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b00b4f00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 --2- 172.21.15.46:0/2811367640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00d57a0 0x7fc1b0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 -- 172.21.15.46:0/2811367640 >> 172.21.15.46:0/2811367640 conn(0x7fc1b00743e0 msgr2=0x7fc1b00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 -- 172.21.15.46:0/2811367640 shutdown_connections 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.853+0000 7fc1b662b640 1 -- 172.21.15.46:0/2811367640 wait complete. 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.855+0000 7fc1b662b640 1 Processor -- start 2024-05-09T03:14:38.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.855+0000 7fc1b662b640 1 -- start start 2024-05-09T03:14:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b662b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b014ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b662b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00b77e0 0x7fc1b014c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b662b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1b00d57a0 0x7fc1b00cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b662b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b00d15b0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b662b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b00d5280 con 0x7fc1b00d57a0 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b662b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1b00d1920 con 0x7fc1b00b77e0 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b014ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b014ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41416/0 (socket says 172.21.15.46:41416) 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1affff640 1 -- 172.21.15.46:0/829030927 learned_addr learned my addr 172.21.15.46:0/829030927 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1a7fff640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00b77e0 0x7fc1b014c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.856+0000 7fc1b4bc8640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1b00d57a0 0x7fc1b00cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1affff640 1 -- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00b77e0 msgr2=0x7fc1b014c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1affff640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00b77e0 0x7fc1b014c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1affff640 1 -- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1b00d57a0 msgr2=0x7fc1b00cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1affff640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1b00d57a0 0x7fc1b00cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1affff640 1 -- 172.21.15.46:0/829030927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b014f340 con 0x7fc1b00b4af0 2024-05-09T03:14:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1affff640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b014ee00 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000d5d0 tx=0x7fc1a000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0004240 con 0x7fc1b00b4af0 2024-05-09T03:14:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1a00043e0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b014cec0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0003830 con 0x7fc1b00b4af0 2024-05-09T03:14:38.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.857+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b00c9df0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.858+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b006a270 con 0x7fc1b00b4af0 2024-05-09T03:14:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.859+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc1a00039d0 con 0x7fc1b00b4af0 2024-05-09T03:14:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.860+0000 7fc1ad7fa640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1900762d0 0x7fc1900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.860+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7fc1a0097810 con 0x7fc1b00b4af0 2024-05-09T03:14:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.860+0000 7fc1a7fff640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1900762d0 0x7fc1900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.860+0000 7fc1a7fff640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1900762d0 0x7fc1900787e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc198034000 tx=0x7fc19802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:38.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:38.864+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1a0060a70 con 0x7fc1b00b4af0 2024-05-09T03:14:39.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.095+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1b00b5ed0 con 0x7fc1b00b4af0 2024-05-09T03:14:39.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.095+0000 7fc1ad7fa640 1 -- 172.21.15.46:0/829030927 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5183 (secure 0 0 0) 0x7fc1a0064d70 con 0x7fc1b00b4af0 2024-05-09T03:14:39.096 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:39.096 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:56.169641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":16,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-05-09T03:14:39.096 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2024-05-09T03:14:39.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1900762d0 msgr2=0x7fc1900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1900762d0 0x7fc1900787e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc198034000 tx=0x7fc19802e000 comp rx=0 tx=0).stop 2024-05-09T03:14:39.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 msgr2=0x7fc1b014ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b014ee00 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000d5d0 tx=0x7fc1a000da90 comp rx=0 tx=0).stop 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 shutdown_connections 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1b00d57a0 0x7fc1b00cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1b00b77e0 0x7fc1b014c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1900762d0 0x7fc1900787e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 --2- 172.21.15.46:0/829030927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1b00b4af0 0x7fc1b014ee00 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 >> 172.21.15.46:0/829030927 conn(0x7fc1b00743e0 msgr2=0x7fc1b00d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 shutdown_connections 2024-05-09T03:14:39.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.097+0000 7fc1b662b640 1 -- 172.21.15.46:0/829030927 wait complete. 2024-05-09T03:14:39.109 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.223+0000 7f206720c640 1 Processor -- start 2024-05-09T03:14:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.223+0000 7f206720c640 1 -- start start 2024-05-09T03:14:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.224+0000 7f206720c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600d57a0 0x7f2060160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f206720c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20600b7ef0 0x7f20600b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f206720c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20600d49e0 con 0x7f20600b4af0 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f206720c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f20600d5bb0 con 0x7f20600b7ef0 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f206720c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20600d4cc0 con 0x7f20600d57a0 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20600b7ef0 0x7f20600b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20600b7ef0 0x7f20600b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41420/0 (socket says 172.21.15.46:41420) 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 -- 172.21.15.46:0/1388817067 learned_addr learned my addr 172.21.15.46:0/1388817067 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f2064fa8640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600d57a0 0x7f2060160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 -- 172.21.15.46:0/1388817067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600d57a0 msgr2=0x7f2060160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600d57a0 0x7f2060160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 -- 172.21.15.46:0/1388817067 >> v1:172.21.15.88:6789/0 conn(0x7f20600b4af0 legacy=0x7f20600b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f20657a9640 1 -- 172.21.15.46:0/1388817067 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20600d4fa0 con 0x7f20600b7ef0 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.225+0000 7f2064fa8640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600d57a0 0x7f2060160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.226+0000 7f20657a9640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20600b7ef0 0x7f20600b4f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2058009e50 tx=0x7f205802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a96d52ddeee538 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.226+0000 7f205f7fe640 1 -- 172.21.15.46:0/1388817067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058036070 con 0x7f20600b7ef0 2024-05-09T03:14:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.226+0000 7f205f7fe640 1 -- 172.21.15.46:0/1388817067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2058031600 con 0x7f20600b7ef0 2024-05-09T03:14:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.226+0000 7f205f7fe640 1 -- 172.21.15.46:0/1388817067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20580319d0 con 0x7f20600b7ef0 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 -- 172.21.15.46:0/1388817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20600b7ef0 msgr2=0x7f20600b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20600b7ef0 0x7f20600b4f00 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2058009e50 tx=0x7f205802a710 comp rx=0 tx=0).stop 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 -- 172.21.15.46:0/1388817067 shutdown_connections 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20600b7ef0 0x7f20600b4f00 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 --2- 172.21.15.46:0/1388817067 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600d57a0 0x7f2060160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 -- 172.21.15.46:0/1388817067 >> 172.21.15.46:0/1388817067 conn(0x7f20600743e0 msgr2=0x7f20600d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 -- 172.21.15.46:0/1388817067 shutdown_connections 2024-05-09T03:14:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.227+0000 7f206720c640 1 -- 172.21.15.46:0/1388817067 wait complete. 2024-05-09T03:14:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.229+0000 7f206720c640 1 Processor -- start 2024-05-09T03:14:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.229+0000 7f206720c640 1 -- start start 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f206720c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600b4af0 0x7f20601e9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f206720c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20600d57a0 0x7f2060150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f206720c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 0x7f20601f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f206720c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20600d1920 con 0x7f20601f6210 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f206720c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20600d15b0 con 0x7f20600d57a0 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f206720c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20600cb8c0 con 0x7f20600b4af0 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f20657a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 0x7f20601f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f20657a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 0x7f20601f6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41436/0 (socket says 172.21.15.46:41436) 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f20657a9640 1 -- 172.21.15.46:0/244126119 learned_addr learned my addr 172.21.15.46:0/244126119 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f205ffff640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20600d57a0 0x7f2060150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.230+0000 7f2064fa8640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600b4af0 0x7f20601e9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f20657a9640 1 -- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600b4af0 msgr2=0x7f20601e9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f20657a9640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600b4af0 0x7f20601e9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f20657a9640 1 -- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20600d57a0 msgr2=0x7f2060150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f20657a9640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20600d57a0 0x7f2060150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f20657a9640 1 -- 172.21.15.46:0/244126119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20600ca820 con 0x7f20601f6210 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f20657a9640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 0x7f20601f6620 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2058009f70 tx=0x7f2058004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205803f020 con 0x7f20601f6210 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2058037040 con 0x7f20601f6210 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058036070 con 0x7f20601f6210 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20600cab00 con 0x7f20601f6210 2024-05-09T03:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.231+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20600cb0c0 con 0x7f20601f6210 2024-05-09T03:14:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.232+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20600ca590 con 0x7f20601f6210 2024-05-09T03:14:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.233+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2058042020 con 0x7f20601f6210 2024-05-09T03:14:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.234+0000 7f205d7fa640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f204c076300 0x7f204c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.234+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f20580b7fb0 con 0x7f20601f6210 2024-05-09T03:14:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.234+0000 7f2064fa8640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f204c076300 0x7f204c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.235+0000 7f2064fa8640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f204c076300 0x7f204c078810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f206006d3b0 tx=0x7f2050008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.238+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2058049030 con 0x7f20601f6210 2024-05-09T03:14:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.467+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f206006a270 con 0x7f20601f6210 2024-05-09T03:14:39.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.467+0000 7f205d7fa640 1 -- 172.21.15.46:0/244126119 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5183 (secure 0 0 0) 0x7f20580818b0 con 0x7f20601f6210 2024-05-09T03:14:39.468 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:39.468 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:12:06.230074+0000","modified":"2024-05-09T03:12:56.169641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":57,"state":"up:active","state_seq":16,"addr":"172.21.15.46:6833/1536534537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1536534537},{"type":"v1","addr":"172.21.15.46:6833","nonce":1536534537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":4}]} 2024-05-09T03:14:39.468 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 60 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f204c076300 msgr2=0x7f204c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f204c076300 0x7f204c078810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f206006d3b0 tx=0x7f2050008040 comp rx=0 tx=0).stop 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 msgr2=0x7f20601f6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 0x7f20601f6620 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2058009f70 tx=0x7f2058004a40 comp rx=0 tx=0).stop 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 shutdown_connections 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f204c076300 0x7f204c078810 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20601f6210 0x7f20601f6620 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20600d57a0 0x7f2060150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 --2- 172.21.15.46:0/244126119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20600b4af0 0x7f20601e9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 >> 172.21.15.46:0/244126119 conn(0x7f20600743e0 msgr2=0x7f206015da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 shutdown_connections 2024-05-09T03:14:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.470+0000 7f206720c640 1 -- 172.21.15.46:0/244126119 wait complete. 2024-05-09T03:14:39.482 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.596+0000 7f8426d4d640 1 Processor -- start 2024-05-09T03:14:39.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.596+0000 7f8426d4d640 1 -- start start 2024-05-09T03:14:39.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.597+0000 7f8426d4d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84200cf590 0x7f84200d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.597+0000 7f8426d4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842006ebe0 con 0x7f84200b9310 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.597+0000 7f8426d4d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f842006eec0 con 0x7f842008d470 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.597+0000 7f8426d4d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f842006f760 con 0x7f84200cf590 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f8424ae9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f84200b9310 0x7f842008cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34558/0 (socket says 172.21.15.46:34558) 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f84252ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84200cf590 0x7f84200d2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f8424ae9640 1 -- 172.21.15.46:0/2273707600 learned_addr learned my addr 172.21.15.46:0/2273707600 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f84252ea640 1 -- 172.21.15.46:0/2273707600 >> v1:172.21.15.88:6789/0 conn(0x7f842008d470 legacy=0x7f84200d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f84252ea640 1 -- 172.21.15.46:0/2273707600 >> v1:172.21.15.46:6789/0 conn(0x7f84200b9310 legacy=0x7f842008cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f841f7fe640 1 -- 172.21.15.46:0/2273707600 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496502735 0 0) 0x7f842006ebe0 con 0x7f84200b9310 2024-05-09T03:14:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.598+0000 7f84252ea640 1 -- 172.21.15.46:0/2273707600 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842006f1a0 con 0x7f84200cf590 2024-05-09T03:14:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.599+0000 7f84252ea640 1 --2- 172.21.15.46:0/2273707600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200cf590 0x7f84200d2670 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8418009d90 tx=0x7f841802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6b895b6a614cc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.599+0000 7f841f7fe640 1 -- 172.21.15.46:0/2273707600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8418007740 con 0x7f84200cf590 2024-05-09T03:14:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.599+0000 7f841f7fe640 1 -- 172.21.15.46:0/2273707600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84180078e0 con 0x7f84200cf590 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.599+0000 7f841f7fe640 1 -- 172.21.15.46:0/2273707600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8418007cb0 con 0x7f84200cf590 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 -- 172.21.15.46:0/2273707600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200cf590 msgr2=0x7f84200d2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 --2- 172.21.15.46:0/2273707600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200cf590 0x7f84200d2670 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8418009d90 tx=0x7f841802a6f0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 -- 172.21.15.46:0/2273707600 shutdown_connections 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 --2- 172.21.15.46:0/2273707600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200cf590 0x7f84200d2670 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 -- 172.21.15.46:0/2273707600 >> 172.21.15.46:0/2273707600 conn(0x7f84200b5660 msgr2=0x7f84200b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 -- 172.21.15.46:0/2273707600 shutdown_connections 2024-05-09T03:14:39.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.600+0000 7f8426d4d640 1 -- 172.21.15.46:0/2273707600 wait complete. 2024-05-09T03:14:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.601+0000 7f8426d4d640 1 Processor -- start 2024-05-09T03:14:39.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.602+0000 7f8426d4d640 1 -- start start 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8426d4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f842008d470 0x7f84201e1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8426d4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 0x7f84201f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8426d4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84200cf590 0x7f84201e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8426d4d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84200d2d30 con 0x7f84200b9310 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8426d4d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f84200d2bb0 con 0x7f84200cf590 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8426d4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f842008d880 con 0x7f842008d470 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 0x7f84201f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 0x7f84201f94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41464/0 (socket says 172.21.15.46:41464) 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 -- 172.21.15.46:0/12203789 learned_addr learned my addr 172.21.15.46:0/12203789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f84252ea640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84200cf590 0x7f84201e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8424ae9640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f842008d470 0x7f84201e1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 -- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f842008d470 msgr2=0x7f84201e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f842008d470 0x7f84201e1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 -- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84200cf590 msgr2=0x7f84201e2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84200cf590 0x7f84201e2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f841ffff640 1 -- 172.21.15.46:0/12203789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84201e2580 con 0x7f84200b9310 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f8424ae9640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f842008d470 0x7f84201e1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.603+0000 7f84252ea640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84200cf590 0x7f84201e2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.604+0000 7f841ffff640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 0x7f84201f94a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f841400c400 tx=0x7f841400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.604+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414017070 con 0x7f84200b9310 2024-05-09T03:14:39.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.604+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84140052d0 con 0x7f84200b9310 2024-05-09T03:14:39.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.604+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84201e22a0 con 0x7f84200b9310 2024-05-09T03:14:39.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.604+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84140037f0 con 0x7f84200b9310 2024-05-09T03:14:39.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.604+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84201e8160 con 0x7f84200b9310 2024-05-09T03:14:39.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.605+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842006a270 con 0x7f84200b9310 2024-05-09T03:14:39.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.606+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f841401a070 con 0x7f84200b9310 2024-05-09T03:14:39.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.606+0000 7f841d7fa640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f840c076300 0x7f840c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.606+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7f8414096920 con 0x7f84200b9310 2024-05-09T03:14:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.606+0000 7f8424ae9640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f840c076300 0x7f840c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.607+0000 7f8424ae9640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f840c076300 0x7f840c078810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8410007fd0 tx=0x7f8410007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.611+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f841405faf0 con 0x7f84200b9310 2024-05-09T03:14:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.803+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84200d2070 con 0x7f84200b9310 2024-05-09T03:14:39.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.804+0000 7f841d7fa640 1 -- 172.21.15.46:0/12203789 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16850 (secure 0 0 0) 0x7f84140681c0 con 0x7f84200b9310 2024-05-09T03:14:39.805 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:39.805 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":57,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:12:53.158229+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:12:03.923965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-05-09T03:12:05.185724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/103238544":"2024-05-10T03:12:53.158166+0000","172.21.15.46:6834/103238544":"2024-05-10T03:12:53.158166+0000","172.21.15.46:6835/4263404372":"2024-05-10T03:12:22.212819+0000","172.21.15.46:6834/4263404372":"2024-05-10T03:12:22.212819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f840c076300 msgr2=0x7f840c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f840c076300 0x7f840c078810 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8410007fd0 tx=0x7f8410007dc0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 msgr2=0x7f84201f94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 0x7f84201f94a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f841400c400 tx=0x7f841400c8c0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 shutdown_connections 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84200cf590 0x7f84201e2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f840c076300 0x7f840c078810 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84200b9310 0x7f84201f94a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 --2- 172.21.15.46:0/12203789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f842008d470 0x7f84201e1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 >> 172.21.15.46:0/12203789 conn(0x7f84200b5660 msgr2=0x7f8420071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 shutdown_connections 2024-05-09T03:14:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.806+0000 7f8426d4d640 1 -- 172.21.15.46:0/12203789 wait complete. 2024-05-09T03:14:39.819 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:14:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.934+0000 7fb247261640 1 Processor -- start 2024-05-09T03:14:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.935+0000 7fb247261640 1 -- start start 2024-05-09T03:14:39.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb247261640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 0x7fb2400b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb247261640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb2400cfcf0 0x7fb24009cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb247261640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2400b9690 con 0x7fb2400d1110 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb247261640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb2400d2b00 con 0x7fb2400cfcf0 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb247261640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb24008cd60 con 0x7fb2400d1740 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb23ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2400d1110 0x7fb2400d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34570/0 (socket says 172.21.15.46:34570) 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb23ffff640 1 -- 172.21.15.46:0/44976231 learned_addr learned my addr 172.21.15.46:0/44976231 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb2457fe640 1 --2- 172.21.15.46:0/44976231 >> v2:172.21.15.88:3300/0 conn(0x7fb2400cfcf0 0x7fb24009cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb244ffd640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 0x7fb2400b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2466122277 0 0) 0x7fb2400b9690 con 0x7fb2400d1110 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.936+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb228003520 con 0x7fb2400d1110 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2863422949 0 0) 0x7fb228003520 con 0x7fb2400d1110 2024-05-09T03:14:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 msgr2=0x7fb2400b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 0x7fb2400b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 >> v2:172.21.15.88:3300/0 conn(0x7fb2400cfcf0 msgr2=0x7fb24009cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 --2- 172.21.15.46:0/44976231 >> v2:172.21.15.88:3300/0 conn(0x7fb2400cfcf0 0x7fb24009cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2400ca9e0 con 0x7fb2400d1110 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fb2340016f0 con 0x7fb2400d1110 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb2457fe640 1 --2- 172.21.15.46:0/44976231 >> v2:172.21.15.88:3300/0 conn(0x7fb2400cfcf0 0x7fb24009cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 >> v1:172.21.15.46:6789/0 conn(0x7fb2400d1110 legacy=0x7fb2400d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb228003a00 0x7fb228003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb228004800 0x7fb228004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 0x7fb228005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2400d2b00 con 0x7fb228003a00 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb24008cd60 con 0x7fb228004800 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb22800ba70 con 0x7fb2400d1740 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb244ffd640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb228003a00 0x7fb228003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb2457fe640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 0x7fb228005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23ffff640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb228004800 0x7fb228004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb244ffd640 1 -- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 msgr2=0x7fb228005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb244ffd640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400d1740 0x7fb228005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb244ffd640 1 -- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb228004800 msgr2=0x7fb228004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb244ffd640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb228004800 0x7fb228004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb244ffd640 1 -- 172.21.15.46:0/44976231 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb228003520 con 0x7fb228003a00 2024-05-09T03:14:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.937+0000 7fb23ffff640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb228004800 0x7fb228004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.938+0000 7fb244ffd640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb228003a00 0x7fb228003e70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb230009ac0 tx=0x7fb228005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172c0217b7cb3c5c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.938+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230036070 con 0x7fb228003a00 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.938+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb230031400 con 0x7fb228003a00 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.938+0000 7fb23f7fe640 1 -- 172.21.15.46:0/44976231 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230009690 con 0x7fb228003a00 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb247261640 1 -- 172.21.15.46:0/44976231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb228003a00 msgr2=0x7fb228003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb247261640 1 --2- 172.21.15.46:0/44976231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb228003a00 0x7fb228003e70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb230009ac0 tx=0x7fb228005830 comp rx=0 tx=0).stop 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb2457fe640 1 -- 172.21.15.46:0/44976231 reap_dead start 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb247261640 1 -- 172.21.15.46:0/44976231 shutdown_connections 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb247261640 1 -- 172.21.15.46:0/44976231 >> 172.21.15.46:0/44976231 conn(0x7fb2400d05f0 msgr2=0x7fb2400d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb247261640 1 -- 172.21.15.46:0/44976231 shutdown_connections 2024-05-09T03:14:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.939+0000 7fb247261640 1 -- 172.21.15.46:0/44976231 wait complete. 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.941+0000 7fb247261640 1 Processor -- start 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.941+0000 7fb247261640 1 -- start start 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb247261640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 0x7fb24006ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb247261640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400cd740 0x7fb2400cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb247261640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb240156aa0 0x7fb240156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb247261640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2400b93b0 con 0x7fb2400cb6e0 2024-05-09T03:14:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb247261640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb2400d1ca0 con 0x7fb240156aa0 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb247261640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb24009d1e0 con 0x7fb2400cd740 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb244ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 0x7fb24006ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb244ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 0x7fb24006ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41488/0 (socket says 172.21.15.46:41488) 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.942+0000 7fb244ffd640 1 -- 172.21.15.46:0/3683847880 learned_addr learned my addr 172.21.15.46:0/3683847880 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb23ffff640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400cd740 0x7fb2400cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb2457fe640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb240156aa0 0x7fb240156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb244ffd640 1 -- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400cd740 msgr2=0x7fb2400cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb244ffd640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400cd740 0x7fb2400cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb244ffd640 1 -- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb240156aa0 msgr2=0x7fb240156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb244ffd640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb240156aa0 0x7fb240156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb244ffd640 1 -- 172.21.15.46:0/3683847880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2401573f0 con 0x7fb2400cb6e0 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb2457fe640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb240156aa0 0x7fb240156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb244ffd640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 0x7fb24006ebf0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb230009be0 tx=0x7fb230004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230036050 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb230042070 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb230009690 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2400cbca0 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.943+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2400cc260 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.945+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2400b65b0 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.945+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb23003d450 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.946+0000 7fb23d7fa640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb22c0762d0 0x7fb22c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.946+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5881+0+0 (secure 0 0 0) 0x7fb2300b79b0 con 0x7fb2400cb6e0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.946+0000 7fb23ffff640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb22c0762d0 0x7fb22c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.946+0000 7fb23ffff640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb22c0762d0 0x7fb22c0787e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb234004840 tx=0x7fb234009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:39.950+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb2400b65b0 con 0x7fb2400cb6e0 2024-05-09T03:14:40.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.179+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb24006a270 con 0x7fb2400cb6e0 2024-05-09T03:14:40.180 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:14:40.179+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:14:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.181+0000 7fb23d7fa640 1 -- 172.21.15.46:0/3683847880 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v61) ==== 161+0+0 (secure 0 0 0) 0x7fb23004b020 con 0x7fb2400cb6e0 2024-05-09T03:14:40.181 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:14:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb22c0762d0 msgr2=0x7fb22c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb22c0762d0 0x7fb22c0787e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb234004840 tx=0x7fb234009040 comp rx=0 tx=0).stop 2024-05-09T03:14:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 msgr2=0x7fb24006ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 0x7fb24006ebf0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb230009be0 tx=0x7fb230004730 comp rx=0 tx=0).stop 2024-05-09T03:14:40.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 shutdown_connections 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb240156aa0 0x7fb240156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2400cd740 0x7fb2400cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb22c0762d0 0x7fb22c0787e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 --2- 172.21.15.46:0/3683847880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2400cb6e0 0x7fb24006ebf0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.184+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 >> 172.21.15.46:0/3683847880 conn(0x7fb2400d05f0 msgr2=0x7fb2400b6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.185+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 shutdown_connections 2024-05-09T03:14:40.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.185+0000 7fb247261640 1 -- 172.21.15.46:0/3683847880 wait complete. 2024-05-09T03:14:40.188 INFO:tasks.ceph.mds.c.smithi046.stderr: -4795> 2024-05-09T03:12:57.823+0000 7f2681073640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:14:40.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:14:40.247 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:14:40.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.317+0000 7f63eded1640 1 Processor -- start 2024-05-09T03:14:40.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.317+0000 7f63eded1640 1 -- start start 2024-05-09T03:14:40.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63eded1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f63e8165c30 0x7f63e8168050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63eded1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e806f9d0 con 0x7f63e80b3e40 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63eded1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e80cd010 con 0x7f63e80b7720 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63eded1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f63e806e530 con 0x7f63e8165c30 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63e77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f63e80b3e40 0x7f63e80b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34584/0 (socket says 172.21.15.46:34584) 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63e77fe640 1 -- 172.21.15.46:0/3669630225 learned_addr learned my addr 172.21.15.46:0/3669630225 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.318+0000 7f63e7fff640 1 --2- 172.21.15.46:0/3669630225 >> v2:172.21.15.46:3300/0 conn(0x7f63e8165c30 0x7f63e8168050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.319+0000 7f63e67fc640 1 -- 172.21.15.46:0/3669630225 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155678277 0 0) 0x7f63e806f9d0 con 0x7f63e80b3e40 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.319+0000 7f63e67fc640 1 -- 172.21.15.46:0/3669630225 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63d4003520 con 0x7f63e80b3e40 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.319+0000 7f63e7fff640 1 -- 172.21.15.46:0/3669630225 >> v1:172.21.15.88:6789/0 conn(0x7f63e80b7720 legacy=0x7f63e80b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.319+0000 7f63e7fff640 1 -- 172.21.15.46:0/3669630225 >> v1:172.21.15.46:6789/0 conn(0x7f63e80b3e40 legacy=0x7f63e80b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.319+0000 7f63e7fff640 1 -- 172.21.15.46:0/3669630225 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e80b2990 con 0x7f63e8165c30 2024-05-09T03:14:40.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63e7fff640 1 --2- 172.21.15.46:0/3669630225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e8165c30 0x7f63e8168050 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f63e00097a0 tx=0x7f63e002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccadb07c0bbebb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:40.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63e67fc640 1 -- 172.21.15.46:0/3669630225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0037070 con 0x7f63e8165c30 2024-05-09T03:14:40.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63e67fc640 1 -- 172.21.15.46:0/3669630225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e0032440 con 0x7f63e8165c30 2024-05-09T03:14:40.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63e67fc640 1 -- 172.21.15.46:0/3669630225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e00327f0 con 0x7f63e8165c30 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 -- 172.21.15.46:0/3669630225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e8165c30 msgr2=0x7f63e8168050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 --2- 172.21.15.46:0/3669630225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e8165c30 0x7f63e8168050 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f63e00097a0 tx=0x7f63e002a230 comp rx=0 tx=0).stop 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 -- 172.21.15.46:0/3669630225 shutdown_connections 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 --2- 172.21.15.46:0/3669630225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e8165c30 0x7f63e8168050 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 -- 172.21.15.46:0/3669630225 >> 172.21.15.46:0/3669630225 conn(0x7f63e809b7d0 msgr2=0x7f63e809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 -- 172.21.15.46:0/3669630225 shutdown_connections 2024-05-09T03:14:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.320+0000 7f63eded1640 1 -- 172.21.15.46:0/3669630225 wait complete. 2024-05-09T03:14:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.322+0000 7f63eded1640 1 Processor -- start 2024-05-09T03:14:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.322+0000 7f63eded1640 1 -- start start 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63eded1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f63e80b3e40 0x7f63e814ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63eded1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 0x7f63e80ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63eded1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e8165c30 0x7f63e80ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63eded1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63e806e1c0 con 0x7f63e80b7720 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63eded1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f63e80b26b0 con 0x7f63e8165c30 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63eded1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f63e80b4250 con 0x7f63e80b3e40 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 0x7f63e80ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 0x7f63e80ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41526/0 (socket says 172.21.15.46:41526) 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 -- 172.21.15.46:0/3168259908 learned_addr learned my addr 172.21.15.46:0/3168259908 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e7fff640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e8165c30 0x7f63e80ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e77fe640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f63e80b3e40 0x7f63e814ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 -- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f63e80b3e40 msgr2=0x7f63e814ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f63e80b3e40 0x7f63e814ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 -- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e8165c30 msgr2=0x7f63e80ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e8165c30 0x7f63e80ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 -- 172.21.15.46:0/3168259908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e8152220 con 0x7f63e80b7720 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.324+0000 7f63e6ffd640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 0x7f63e80ce760 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f63d000e6a0 tx=0x7f63d000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.325+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d00052d0 con 0x7f63e80b7720 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.325+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d0005470 con 0x7f63e80b7720 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.325+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e8153140 con 0x7f63e80b7720 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.325+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d000ca90 con 0x7f63e80b7720 2024-05-09T03:14:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.325+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e8153700 con 0x7f63e80b7720 2024-05-09T03:14:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.326+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63e806a270 con 0x7f63e80b7720 2024-05-09T03:14:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.326+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f63d000cc30 con 0x7f63e80b7720 2024-05-09T03:14:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.327+0000 7f63dffff640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f63c8076300 0x7f63c8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.327+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5967+0+0 (secure 0 0 0) 0x7f63d0099960 con 0x7f63e80b7720 2024-05-09T03:14:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.327+0000 7f63e77fe640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f63c8076300 0x7f63c8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.328+0000 7f63e77fe640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f63c8076300 0x7f63c8078810 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f63d8005c00 tx=0x7f63d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.332+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f63d0016020 con 0x7f63e80b7720 2024-05-09T03:14:40.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:40.568+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f63e80d1130 con 0x7f63e80b7720 2024-05-09T03:14:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.186+0000 7f63dffff640 1 -- 172.21.15.46:0/3168259908 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v62) ==== 104+0+0 (secure 0 0 0) 0x7f63d0062b60 con 0x7f63e80b7720 2024-05-09T03:14:41.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f63c8076300 msgr2=0x7f63c8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f63c8076300 0x7f63c8078810 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f63d8005c00 tx=0x7f63d8009040 comp rx=0 tx=0).stop 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 msgr2=0x7f63e80ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 0x7f63e80ce760 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f63d000e6a0 tx=0x7f63d000eb60 comp rx=0 tx=0).stop 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 shutdown_connections 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e8165c30 0x7f63e80ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f63c8076300 0x7f63c8078810 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e80b7720 0x7f63e80ce760 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 --2- 172.21.15.46:0/3168259908 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f63e80b3e40 0x7f63e814ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.189+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 >> 172.21.15.46:0/3168259908 conn(0x7f63e809b7d0 msgr2=0x7f63e80d15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.190+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 shutdown_connections 2024-05-09T03:14:41.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.190+0000 7f63eded1640 1 -- 172.21.15.46:0/3168259908 wait complete. 2024-05-09T03:14:41.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:14:41.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.329+0000 7fdab9172640 1 Processor -- start 2024-05-09T03:14:41.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.329+0000 7fdab9172640 1 -- start start 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab9172640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdab406de80 0x7fdab40d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab9172640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdab40d0630 0x7fdab40d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab9172640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab40d2620 con 0x7fdab40b27b0 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab9172640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdab40d36c0 con 0x7fdab40d0630 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab9172640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdab40d3230 con 0x7fdab406de80 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdab40d0630 0x7fdab40d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdab40d0630 0x7fdab40d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41536/0 (socket says 172.21.15.46:41536) 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.330+0000 7fdab259c640 1 -- 172.21.15.46:0/1954737011 learned_addr learned my addr 172.21.15.46:0/1954737011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab259c640 1 -- 172.21.15.46:0/1954737011 >> v2:172.21.15.88:3300/0 conn(0x7fdab406de80 msgr2=0x7fdab40d1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab2d9d640 1 --2- 172.21.15.46:0/1954737011 >> v2:172.21.15.88:3300/0 conn(0x7fdab406de80 0x7fdab40d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab259c640 1 --2- 172.21.15.46:0/1954737011 >> v2:172.21.15.88:3300/0 conn(0x7fdab406de80 0x7fdab40d1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab259c640 1 -- 172.21.15.46:0/1954737011 >> v1:172.21.15.88:6789/0 conn(0x7fdab40b27b0 legacy=0x7fdab40b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab259c640 1 -- 172.21.15.46:0/1954737011 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab40d2900 con 0x7fdab40d0630 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab2d9d640 1 --2- 172.21.15.46:0/1954737011 >> v2:172.21.15.88:3300/0 conn(0x7fdab406de80 0x7fdab40d1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.331+0000 7fdab259c640 1 --2- 172.21.15.46:0/1954737011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40d0630 0x7fdab40d0a40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdab4074610 tx=0x7fdaa802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a9a9b4d2e4c366 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab1d9b640 1 -- 172.21.15.46:0/1954737011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8037070 con 0x7fdab40d0630 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab1d9b640 1 -- 172.21.15.46:0/1954737011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8032440 con 0x7fdab40d0630 2024-05-09T03:14:41.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab1d9b640 1 -- 172.21.15.46:0/1954737011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8032810 con 0x7fdab40d0630 2024-05-09T03:14:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 -- 172.21.15.46:0/1954737011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40d0630 msgr2=0x7fdab40d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 --2- 172.21.15.46:0/1954737011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40d0630 0x7fdab40d0a40 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdab4074610 tx=0x7fdaa802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 -- 172.21.15.46:0/1954737011 shutdown_connections 2024-05-09T03:14:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 --2- 172.21.15.46:0/1954737011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40d0630 0x7fdab40d0a40 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 --2- 172.21.15.46:0/1954737011 >> v2:172.21.15.88:3300/0 conn(0x7fdab406de80 0x7fdab40d1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 -- 172.21.15.46:0/1954737011 >> 172.21.15.46:0/1954737011 conn(0x7fdab40d3bd0 msgr2=0x7fdab40d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 -- 172.21.15.46:0/1954737011 shutdown_connections 2024-05-09T03:14:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.332+0000 7fdab9172640 1 -- 172.21.15.46:0/1954737011 wait complete. 2024-05-09T03:14:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.334+0000 7fdab9172640 1 Processor -- start 2024-05-09T03:14:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.334+0000 7fdab9172640 1 -- start start 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab9172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdab406de80 0x7fdab41f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab9172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 0x7fdab41f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab9172640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab40d0630 0x7fdab409d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab9172640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdab40d4810 con 0x7fdab40b27b0 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab9172640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdab40d2ec0 con 0x7fdab40d0630 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab9172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdab40d4990 con 0x7fdab406de80 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 0x7fdab41f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 0x7fdab41f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41546/0 (socket says 172.21.15.46:41546) 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 -- 172.21.15.46:0/485937918 learned_addr learned my addr 172.21.15.46:0/485937918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:41.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab359e640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab40d0630 0x7fdab409d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab2d9d640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdab406de80 0x7fdab41f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 -- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdab406de80 msgr2=0x7fdab41f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdab406de80 0x7fdab41f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 -- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab40d0630 msgr2=0x7fdab409d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab40d0630 0x7fdab409d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 -- 172.21.15.46:0/485937918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab41f8430 con 0x7fdab40b27b0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.336+0000 7fdab259c640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 0x7fdab41f7ba0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdaa802a220 tx=0x7fdaa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.337+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8037040 con 0x7fdab40b27b0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.337+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8042070 con 0x7fdab40b27b0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.337+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8030bb0 con 0x7fdab40b27b0 2024-05-09T03:14:41.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.337+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab41f8710 con 0x7fdab40b27b0 2024-05-09T03:14:41.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.337+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab41f8cd0 con 0x7fdab40b27b0 2024-05-09T03:14:41.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.338+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab406a270 con 0x7fdab40b27b0 2024-05-09T03:14:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.339+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdaa8047460 con 0x7fdab40b27b0 2024-05-09T03:14:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.339+0000 7fda9b7fe640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdaa0076300 0x7fdaa0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.339+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5967+0+0 (secure 0 0 0) 0x7fdaa80b8390 con 0x7fdab40b27b0 2024-05-09T03:14:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.339+0000 7fdab2d9d640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdaa0076300 0x7fdaa0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.340+0000 7fdab2d9d640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdaa0076300 0x7fdaa0078810 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdab40d1220 tx=0x7fdaa4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.344+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdaa8081be0 con 0x7fdab40b27b0 2024-05-09T03:14:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:41.541+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdab41f81c0 con 0x7fdab40b27b0 2024-05-09T03:14:42.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.188+0000 7fda9b7fe640 1 -- 172.21.15.46:0/485937918 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) ==== 181+0+0 (secure 0 0 0) 0x7fdaa80814e0 con 0x7fdab40b27b0 2024-05-09T03:14:42.189 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdaa0076300 msgr2=0x7fdaa0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdaa0076300 0x7fdaa0078810 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdab40d1220 tx=0x7fdaa4008040 comp rx=0 tx=0).stop 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 msgr2=0x7fdab41f7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 0x7fdab41f7ba0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdaa802a220 tx=0x7fdaa80046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 shutdown_connections 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab40d0630 0x7fdab409d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdaa0076300 0x7fdaa0078810 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab40b27b0 0x7fdab41f7ba0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 --2- 172.21.15.46:0/485937918 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdab406de80 0x7fdab41f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 >> 172.21.15.46:0/485937918 conn(0x7fdab40d3bd0 msgr2=0x7fdab415feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 shutdown_connections 2024-05-09T03:14:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.193+0000 7fdab9172640 1 -- 172.21.15.46:0/485937918 wait complete. 2024-05-09T03:14:42.208 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:14:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.336+0000 7fe4ac219640 1 Processor -- start 2024-05-09T03:14:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.336+0000 7fe4ac219640 1 -- start start 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.337+0000 7fe4ac219640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4a40bb770 0x7fe4a40d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.337+0000 7fe4ac219640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a4163eb0 0x7fe4a41642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4ac219640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe4a40d2a70 0x7fe4a40d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4ac219640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe4a40d4bb0 con 0x7fe4a40bb770 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4ac219640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe4a40d33c0 con 0x7fe4a40d2a70 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4ac219640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a4164900 con 0x7fe4a4163eb0 2024-05-09T03:14:42.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4a40bb770 0x7fe4a40d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4a40bb770 0x7fe4a40d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41560/0 (socket says 172.21.15.46:41560) 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/1438731441 learned_addr learned my addr 172.21.15.46:0/1438731441 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4aa7b6640 1 --2- 172.21.15.46:0/1438731441 >> v2:172.21.15.88:3300/0 conn(0x7fe4a40d2a70 0x7fe4a40d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a97b4640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a4163eb0 0x7fe4a41642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/1438731441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a4163eb0 msgr2=0x7fe4a41642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a4163eb0 0x7fe4a41642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/1438731441 >> v2:172.21.15.88:3300/0 conn(0x7fe4a40d2a70 msgr2=0x7fe4a40d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 --2- 172.21.15.46:0/1438731441 >> v2:172.21.15.88:3300/0 conn(0x7fe4a40d2a70 0x7fe4a40d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/1438731441 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a4164be0 con 0x7fe4a40bb770 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4aa7b6640 1 --2- 172.21.15.46:0/1438731441 >> v2:172.21.15.88:3300/0 conn(0x7fe4a40d2a70 0x7fe4a40d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.338+0000 7fe4a97b4640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a4163eb0 0x7fe4a41642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:42.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.339+0000 7fe4a9fb5640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a40d5040 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40b56d0 tx=0x7fe49402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196bb269756deab9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.339+0000 7fe4a8fb3640 1 -- 172.21.15.46:0/1438731441 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494037070 con 0x7fe4a40bb770 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.339+0000 7fe4a8fb3640 1 -- 172.21.15.46:0/1438731441 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe494032440 con 0x7fe4a40bb770 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.339+0000 7fe4a8fb3640 1 -- 172.21.15.46:0/1438731441 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494032810 con 0x7fe4a40bb770 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 -- 172.21.15.46:0/1438731441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 msgr2=0x7fe4a40d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a40d5040 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40b56d0 tx=0x7fe49402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 -- 172.21.15.46:0/1438731441 shutdown_connections 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 --2- 172.21.15.46:0/1438731441 >> v2:172.21.15.88:3300/0 conn(0x7fe4a40d2a70 0x7fe4a40d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a4163eb0 0x7fe4a41642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 --2- 172.21.15.46:0/1438731441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a40d5040 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 -- 172.21.15.46:0/1438731441 >> 172.21.15.46:0/1438731441 conn(0x7fe4a406e370 msgr2=0x7fe4a406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 -- 172.21.15.46:0/1438731441 shutdown_connections 2024-05-09T03:14:42.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.340+0000 7fe4ac219640 1 -- 172.21.15.46:0/1438731441 wait complete. 2024-05-09T03:14:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.342+0000 7fe4ac219640 1 Processor -- start 2024-05-09T03:14:42.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.342+0000 7fe4ac219640 1 -- start start 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4ac219640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a41f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4ac219640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a40d2a70 0x7fe4a415e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4ac219640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a4163eb0 0x7fe4a415e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4ac219640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40bbb80 con 0x7fe4a40bb770 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4ac219640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a40d5580 con 0x7fe4a4163eb0 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4ac219640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a40d0bc0 con 0x7fe4a40d2a70 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a41f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a41f5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41568/0 (socket says 172.21.15.46:41568) 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/442511029 learned_addr learned my addr 172.21.15.46:0/442511029 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a97b4640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a40d2a70 0x7fe4a415e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4aa7b6640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a4163eb0 0x7fe4a415e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a40d2a70 msgr2=0x7fe4a415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a40d2a70 0x7fe4a415e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a4163eb0 msgr2=0x7fe4a415e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a4163eb0 0x7fe4a415e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a9fb5640 1 -- 172.21.15.46:0/442511029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a414be70 con 0x7fe4a40bb770 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4a97b4640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a40d2a70 0x7fe4a415e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.343+0000 7fe4aa7b6640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a4163eb0 0x7fe4a415e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.344+0000 7fe4a9fb5640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a41f5e70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe49402a220 tx=0x7fe4940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.344+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494037040 con 0x7fe4a40bb770 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.344+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe494042070 con 0x7fe4a40bb770 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.344+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe494030bb0 con 0x7fe4a40bb770 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.344+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a415eef0 con 0x7fe4a40bb770 2024-05-09T03:14:42.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.344+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a415f4b0 con 0x7fe4a40bb770 2024-05-09T03:14:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.345+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a406a270 con 0x7fe4a40bb770 2024-05-09T03:14:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.345+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe494047460 con 0x7fe4a40bb770 2024-05-09T03:14:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.346+0000 7fe4a27fc640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4900762d0 0x7fe4900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.346+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5529+0+0 (secure 0 0 0) 0x7fe4940b8200 con 0x7fe4a40bb770 2024-05-09T03:14:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.346+0000 7fe4a97b4640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4900762d0 0x7fe4900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:42.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.347+0000 7fe4a97b4640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4900762d0 0x7fe4900787e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40ccf10 tx=0x7fe498008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:42.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.351+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe494081530 con 0x7fe4a40bb770 2024-05-09T03:14:42.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:42.540+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe4a4163be0 con 0x7fe4a40bb770 2024-05-09T03:14:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.196+0000 7fe4a27fc640 1 -- 172.21.15.46:0/442511029 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) ==== 169+0+0 (secure 0 0 0) 0x7fe494085830 con 0x7fe4a40bb770 2024-05-09T03:14:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4900762d0 msgr2=0x7fe4900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4900762d0 0x7fe4900787e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40ccf10 tx=0x7fe498008040 comp rx=0 tx=0).stop 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 msgr2=0x7fe4a41f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a41f5e70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe49402a220 tx=0x7fe4940046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 shutdown_connections 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a4163eb0 0x7fe4a415e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe4a40d2a70 0x7fe4a415e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe4900762d0 0x7fe4900787e0 secure :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40ccf10 tx=0x7fe498008040 comp rx=0 tx=0).stop 2024-05-09T03:14:43.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 --2- 172.21.15.46:0/442511029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a40bb770 0x7fe4a41f5e70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 >> 172.21.15.46:0/442511029 conn(0x7fe4a406e370 msgr2=0x7fe4a40b6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:43.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 shutdown_connections 2024-05-09T03:14:43.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.201+0000 7fe4ac219640 1 -- 172.21.15.46:0/442511029 wait complete. 2024-05-09T03:14:43.217 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-05-09T03:14:43.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.346+0000 7f95ce4cd640 1 Processor -- start 2024-05-09T03:14:43.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.346+0000 7f95ce4cd640 1 -- start start 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95ce4cd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c80d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95ce4cd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95c80d11f0 0x7f95c80b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95ce4cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95c80b8050 0x7f95c8164020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95ce4cd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95c806e760 con 0x7f95c80d11f0 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95ce4cd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95c80b8460 con 0x7f95c80b8050 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95ce4cd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95c80d5740 con 0x7f95c80b50b0 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95c80d11f0 0x7f95c80b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95c80d11f0 0x7f95c80b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41582/0 (socket says 172.21.15.46:41582) 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 -- 172.21.15.46:0/332521432 learned_addr learned my addr 172.21.15.46:0/332521432 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95cca6a640 1 --2- 172.21.15.46:0/332521432 >> v2:172.21.15.88:3300/0 conn(0x7f95c80b8050 0x7f95c8164020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 -- 172.21.15.46:0/332521432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 msgr2=0x7f95c80d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 --2- 172.21.15.46:0/332521432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c80d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 -- 172.21.15.46:0/332521432 >> v2:172.21.15.88:3300/0 conn(0x7f95c80b8050 msgr2=0x7f95c8164020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 --2- 172.21.15.46:0/332521432 >> v2:172.21.15.88:3300/0 conn(0x7f95c80b8050 0x7f95c8164020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95c77fe640 1 -- 172.21.15.46:0/332521432 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c806de30 con 0x7f95c80d11f0 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.348+0000 7f95cca6a640 1 --2- 172.21.15.46:0/332521432 >> v2:172.21.15.88:3300/0 conn(0x7f95c80b8050 0x7f95c8164020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.349+0000 7f95c77fe640 1 --2- 172.21.15.46:0/332521432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80d11f0 0x7f95c80b7b10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdac776f14fb647b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.349+0000 7f95c6ffd640 1 -- 172.21.15.46:0/332521432 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc037070 con 0x7f95c80d11f0 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.349+0000 7f95c6ffd640 1 -- 172.21.15.46:0/332521432 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc032440 con 0x7f95c80d11f0 2024-05-09T03:14:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.349+0000 7f95c6ffd640 1 -- 172.21.15.46:0/332521432 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc032810 con 0x7f95c80d11f0 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 -- 172.21.15.46:0/332521432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80d11f0 msgr2=0x7f95c80b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/332521432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80d11f0 0x7f95c80b7b10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009870 tx=0x7f95bc02a210 comp rx=0 tx=0).stop 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 -- 172.21.15.46:0/332521432 shutdown_connections 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/332521432 >> v2:172.21.15.88:3300/0 conn(0x7f95c80b8050 0x7f95c8164020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/332521432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80d11f0 0x7f95c80b7b10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/332521432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c80d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 -- 172.21.15.46:0/332521432 >> 172.21.15.46:0/332521432 conn(0x7f95c80d2b80 msgr2=0x7f95c80d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 -- 172.21.15.46:0/332521432 shutdown_connections 2024-05-09T03:14:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.350+0000 7f95ce4cd640 1 -- 172.21.15.46:0/332521432 wait complete. 2024-05-09T03:14:43.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.352+0000 7f95ce4cd640 1 Processor -- start 2024-05-09T03:14:43.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.352+0000 7f95ce4cd640 1 -- start start 2024-05-09T03:14:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95ce4cd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c81ea930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95ce4cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 0x7f95c81f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95ce4cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d11f0 0x7f95c81f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95ce4cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95c80b54c0 con 0x7f95c80b8050 2024-05-09T03:14:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95ce4cd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95c806e3f0 con 0x7f95c80d11f0 2024-05-09T03:14:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95ce4cd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95c806ebf0 con 0x7f95c80b50b0 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 0x7f95c81f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 0x7f95c81f61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41588/0 (socket says 172.21.15.46:41588) 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.353+0000 7f95c77fe640 1 -- 172.21.15.46:0/2640292276 learned_addr learned my addr 172.21.15.46:0/2640292276 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95cca6a640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d11f0 0x7f95c81f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c7fff640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c81ea930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c77fe640 1 -- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 msgr2=0x7f95c81ea930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c77fe640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c81ea930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c77fe640 1 -- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d11f0 msgr2=0x7f95c81f9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c77fe640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d11f0 0x7f95c81f9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c77fe640 1 -- 172.21.15.46:0/2640292276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c81eb3d0 con 0x7f95c80b8050 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c77fe640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 0x7f95c81f61a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f95bc02a250 tx=0x7f95bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc037040 con 0x7f95c80b8050 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc042070 con 0x7f95c80b8050 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c81eb6b0 con 0x7f95c80b8050 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc030b90 con 0x7f95c80b8050 2024-05-09T03:14:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.354+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c80ce1d0 con 0x7f95c80b8050 2024-05-09T03:14:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.355+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c806a270 con 0x7f95c80b8050 2024-05-09T03:14:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.356+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95bc047460 con 0x7f95c80b8050 2024-05-09T03:14:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.357+0000 7f95c4ff9640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b40762d0 0x7f95b40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.357+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7f95bc0b7e40 con 0x7f95c80b8050 2024-05-09T03:14:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.357+0000 7f95c7fff640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b40762d0 0x7f95b40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.358+0000 7f95c7fff640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b40762d0 0x7f95b40787e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f95b80098e0 tx=0x7f95b8009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.362+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95bc04c020 con 0x7f95c80b8050 2024-05-09T03:14:43.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.552+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) -- 0x7f95c81eb160 con 0x7f95c80b8050 2024-05-09T03:14:43.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.552+0000 7f95c4ff9640 1 -- 172.21.15.46:0/2640292276 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7f95bc081310 con 0x7f95c80b8050 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b40762d0 msgr2=0x7f95b40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b40762d0 0x7f95b40787e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f95b80098e0 tx=0x7f95b8009060 comp rx=0 tx=0).stop 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 msgr2=0x7f95c81f61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 0x7f95c81f61a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f95bc02a250 tx=0x7f95bc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 shutdown_connections 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95c80d11f0 0x7f95c81f9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95b40762d0 0x7f95b40787e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95c80b8050 0x7f95c81f61a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 --2- 172.21.15.46:0/2640292276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95c80b50b0 0x7f95c81ea930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 >> 172.21.15.46:0/2640292276 conn(0x7f95c80d2b80 msgr2=0x7f95c81601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 shutdown_connections 2024-05-09T03:14:43.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.554+0000 7f95ce4cd640 1 -- 172.21.15.46:0/2640292276 wait complete. 2024-05-09T03:14:43.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T03:14:43.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.683+0000 7fe5cba15640 1 Processor -- start 2024-05-09T03:14:43.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.683+0000 7fe5cba15640 1 -- start start 2024-05-09T03:14:43.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.684+0000 7fe5cba15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c4073e40 0x7fe5c40b60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.684+0000 7fe5cba15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5c40af500 0x7fe5c40af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.684+0000 7fe5cba15640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c40b4c80 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.684+0000 7fe5cba15640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c40b6630 con 0x7fe5c40af500 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.684+0000 7fe5cba15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c40b4f20 con 0x7fe5c4073e40 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5c40af500 0x7fe5c40af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe5c40af500 0x7fe5c40af910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41596/0 (socket says 172.21.15.46:41596) 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 -- 172.21.15.46:0/2023918283 learned_addr learned my addr 172.21.15.46:0/2023918283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c8fb0640 1 --2- 172.21.15.46:0/2023918283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c4073e40 0x7fe5c40b60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 -- 172.21.15.46:0/2023918283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c4073e40 msgr2=0x7fe5c40b60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 --2- 172.21.15.46:0/2023918283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c4073e40 0x7fe5c40b60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 -- 172.21.15.46:0/2023918283 >> v1:172.21.15.88:6789/0 conn(0x7fe5c40b5ce0 legacy=0x7fe5c4073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.685+0000 7fe5c9fb2640 1 -- 172.21.15.46:0/2023918283 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c40b5200 con 0x7fe5c40af500 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.686+0000 7fe5c9fb2640 1 --2- 172.21.15.46:0/2023918283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40af500 0x7fe5c40af910 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009e50 tx=0x7fe5bc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739edeaa37df5ba1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.686+0000 7fe5bbfff640 1 -- 172.21.15.46:0/2023918283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc036070 con 0x7fe5c40af500 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.686+0000 7fe5bbfff640 1 -- 172.21.15.46:0/2023918283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc031600 con 0x7fe5c40af500 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.686+0000 7fe5bbfff640 1 -- 172.21.15.46:0/2023918283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0319d0 con 0x7fe5c40af500 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.686+0000 7fe5cba15640 1 -- 172.21.15.46:0/2023918283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40af500 msgr2=0x7fe5c40af910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.686+0000 7fe5cba15640 1 --2- 172.21.15.46:0/2023918283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40af500 0x7fe5c40af910 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009e50 tx=0x7fe5bc02a680 comp rx=0 tx=0).stop 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.687+0000 7fe5cba15640 1 -- 172.21.15.46:0/2023918283 shutdown_connections 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.687+0000 7fe5cba15640 1 --2- 172.21.15.46:0/2023918283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40af500 0x7fe5c40af910 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.687+0000 7fe5cba15640 1 --2- 172.21.15.46:0/2023918283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c4073e40 0x7fe5c40b60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.687+0000 7fe5cba15640 1 -- 172.21.15.46:0/2023918283 >> 172.21.15.46:0/2023918283 conn(0x7fe5c406d100 msgr2=0x7fe5c40b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:43.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.687+0000 7fe5cba15640 1 -- 172.21.15.46:0/2023918283 shutdown_connections 2024-05-09T03:14:43.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.687+0000 7fe5cba15640 1 -- 172.21.15.46:0/2023918283 wait complete. 2024-05-09T03:14:43.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.688+0000 7fe5cba15640 1 Processor -- start 2024-05-09T03:14:43.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.688+0000 7fe5cba15640 1 -- start start 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5cba15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c40af500 0x7fe5c4137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5cba15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 0x7fe5c415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5cba15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c415f4a0 0x7fe5c40703f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5cba15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c4146b20 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5cba15640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c40b57c0 con 0x7fe5c40af500 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5cba15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c4146ca0 con 0x7fe5c415f4a0 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 0x7fe5c415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 0x7fe5c415ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41610/0 (socket says 172.21.15.46:41610) 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 -- 172.21.15.46:0/3890000842 learned_addr learned my addr 172.21.15.46:0/3890000842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c97b1640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c40af500 0x7fe5c4137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c9fb2640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c415f4a0 0x7fe5c40703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 -- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c415f4a0 msgr2=0x7fe5c40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c415f4a0 0x7fe5c40703f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 -- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c40af500 msgr2=0x7fe5c4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c40af500 0x7fe5c4137680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 -- 172.21.15.46:0/3890000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c415ffa0 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c9fb2640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c415f4a0 0x7fe5c40703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5c8fb0640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 0x7fe5c415ef60 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400d620 tx=0x7fe5b400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.690+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400b620 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.691+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b4019070 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.691+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c415fcc0 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.691+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400ba00 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.691+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c4144520 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.692+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c41607a0 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.692+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe5b400bba0 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.693+0000 7fe5b9ffb640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5a8076300 0x7fe5a8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.693+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7fe5b4097800 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.693+0000 7fe5c97b1640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5a8076300 0x7fe5a8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:43.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.694+0000 7fe5c97b1640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5a8076300 0x7fe5a8078810 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40d2db0 tx=0x7fe5c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:43.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.698+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe5c41607a0 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.886+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fe5c40749e0 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.890+0000 7fe5b9ffb640 1 -- 172.21.15.46:0/3890000842 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7fe5b4061320 con 0x7fe5c40b5ce0 2024-05-09T03:14:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5a8076300 msgr2=0x7fe5a8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5a8076300 0x7fe5a8078810 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40d2db0 tx=0x7fe5c0009040 comp rx=0 tx=0).stop 2024-05-09T03:14:43.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 msgr2=0x7fe5c415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 0x7fe5c415ef60 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400d620 tx=0x7fe5b400dae0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 shutdown_connections 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5c415f4a0 0x7fe5c40703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5a8076300 0x7fe5a8078810 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5c40b5ce0 0x7fe5c415ef60 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 --2- 172.21.15.46:0/3890000842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5c40af500 0x7fe5c4137680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 >> 172.21.15.46:0/3890000842 conn(0x7fe5c406d100 msgr2=0x7fe5c40af910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 shutdown_connections 2024-05-09T03:14:43.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:43.892+0000 7fe5cba15640 1 -- 172.21.15.46:0/3890000842 wait complete. 2024-05-09T03:14:43.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-05-09T03:14:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.017+0000 7f4b7b6bd640 1 Processor -- start 2024-05-09T03:14:44.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.018+0000 7f4b7b6bd640 1 -- start start 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b7b6bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740d0b90 0x7f4b740b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b7b6bd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b740cf6b0 0x7f4b7406d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b7b6bd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b740d2b80 con 0x7f4b740b92e0 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b7b6bd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4b740d2e60 con 0x7f4b740cf6b0 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b7b6bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4b740d3140 con 0x7f4b740d0b90 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79c5a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b740cf6b0 0x7f4b7406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:44.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79459640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b740b92e0 0x7f4b740d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34588/0 (socket says 172.21.15.46:34588) 2024-05-09T03:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79459640 1 -- 172.21.15.46:0/1522057572 learned_addr learned my addr 172.21.15.46:0/1522057572 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b78c58640 1 --2- 172.21.15.46:0/1522057572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740d0b90 0x7f4b740b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79c5a640 1 -- 172.21.15.46:0/1522057572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740d0b90 msgr2=0x7f4b740b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79c5a640 1 --2- 172.21.15.46:0/1522057572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740d0b90 0x7f4b740b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79c5a640 1 -- 172.21.15.46:0/1522057572 >> v1:172.21.15.46:6789/0 conn(0x7f4b740b92e0 legacy=0x7f4b740d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:44.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.019+0000 7f4b79c5a640 1 -- 172.21.15.46:0/1522057572 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b740d3420 con 0x7f4b740cf6b0 2024-05-09T03:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.020+0000 7f4b79c5a640 1 --2- 172.21.15.46:0/1522057572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b7406d790 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0097a0 tx=0x7f4b6c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dbe52e55d983b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.020+0000 7f4b73fff640 1 -- 172.21.15.46:0/1522057572 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c037070 con 0x7f4b740cf6b0 2024-05-09T03:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.020+0000 7f4b73fff640 1 -- 172.21.15.46:0/1522057572 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6c02ae50 con 0x7f4b740cf6b0 2024-05-09T03:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.020+0000 7f4b73fff640 1 -- 172.21.15.46:0/1522057572 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c032660 con 0x7f4b740cf6b0 2024-05-09T03:14:44.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/1522057572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 msgr2=0x7f4b7406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/1522057572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b7406d790 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0097a0 tx=0x7f4b6c02a100 comp rx=0 tx=0).stop 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/1522057572 shutdown_connections 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/1522057572 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b7406d790 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/1522057572 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740d0b90 0x7f4b740b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/1522057572 >> 172.21.15.46:0/1522057572 conn(0x7f4b740d3c00 msgr2=0x7f4b740d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/1522057572 shutdown_connections 2024-05-09T03:14:44.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.021+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/1522057572 wait complete. 2024-05-09T03:14:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.023+0000 7f4b7b6bd640 1 Processor -- start 2024-05-09T03:14:44.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.023+0000 7f4b7b6bd640 1 -- start start 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b7b6bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740b92e0 0x7f4b741e8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b7b6bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b741e6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b7b6bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b740cd960 0x7f4b74154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b7b6bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7406e060 con 0x7f4b740cf6b0 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b7b6bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7406dee0 con 0x7f4b740cd960 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b7b6bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7406e1e0 con 0x7f4b740b92e0 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b78c58640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b741e6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b78c58640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b741e6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41624/0 (socket says 172.21.15.46:41624) 2024-05-09T03:14:44.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.024+0000 7f4b78c58640 1 -- 172.21.15.46:0/3343241308 learned_addr learned my addr 172.21.15.46:0/3343241308 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b79459640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740b92e0 0x7f4b741e8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b79c5a640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b740cd960 0x7f4b74154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b78c58640 1 -- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740b92e0 msgr2=0x7f4b741e8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b78c58640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740b92e0 0x7f4b741e8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b78c58640 1 -- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b740cd960 msgr2=0x7f4b74154430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b78c58640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b740cd960 0x7f4b74154430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b78c58640 1 -- 172.21.15.46:0/3343241308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b74154970 con 0x7f4b740cf6b0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b78c58640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b741e6740 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800d5d0 tx=0x7f4b6800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b680125f0 con 0x7f4b740cf6b0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b68019070 con 0x7f4b740cf6b0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b740ccba0 con 0x7f4b740cf6b0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b680129b0 con 0x7f4b740cf6b0 2024-05-09T03:14:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.025+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b740cd160 con 0x7f4b740cf6b0 2024-05-09T03:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.026+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b7406a270 con 0x7f4b740cf6b0 2024-05-09T03:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.027+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4b68012b50 con 0x7f4b740cf6b0 2024-05-09T03:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.028+0000 7f4b71ffb640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b600762d0 0x7f4b600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.028+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7f4b68097890 con 0x7f4b740cf6b0 2024-05-09T03:14:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.028+0000 7f4b79459640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b600762d0 0x7f4b600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.028+0000 7f4b79459640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b600762d0 0x7f4b600787e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4b640012e0 tx=0x7f4b64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:44.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.033+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4b68061320 con 0x7f4b740cf6b0 2024-05-09T03:14:44.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:44.301+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) -- 0x7f4b7406eab0 con 0x7f4b740cf6b0 2024-05-09T03:14:45.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.204+0000 7f4b71ffb640 1 -- 172.21.15.46:0/3343241308 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v247) ==== 142+0+0 (secure 0 0 0) 0x7f4b68060c20 con 0x7f4b740cf6b0 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b600762d0 msgr2=0x7f4b600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b600762d0 0x7f4b600787e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4b640012e0 tx=0x7f4b64009040 comp rx=0 tx=0).stop 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 msgr2=0x7f4b741e6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b741e6740 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800d5d0 tx=0x7f4b6800da90 comp rx=0 tx=0).stop 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 shutdown_connections 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b740cd960 0x7f4b74154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4b600762d0 0x7f4b600787e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b740cf6b0 0x7f4b741e6740 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 --2- 172.21.15.46:0/3343241308 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4b740b92e0 0x7f4b741e8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 >> 172.21.15.46:0/3343241308 conn(0x7f4b740d3c00 msgr2=0x7f4b740d0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:45.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 shutdown_connections 2024-05-09T03:14:45.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.207+0000 7f4b7b6bd640 1 -- 172.21.15.46:0/3343241308 wait complete. 2024-05-09T03:14:45.222 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-05-09T03:14:45.222 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-05-09T03:14:45.222 INFO:tasks.cephfs_test_runner:test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) 2024-05-09T03:14:45.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.345+0000 7f6cc9e92640 1 Processor -- start 2024-05-09T03:14:45.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.345+0000 7f6cc9e92640 1 -- start start 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.346+0000 7f6cc9e92640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cc40b5210 0x7f6cc41613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.346+0000 7f6cc9e92640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40b7ed0 0x7f6cc40d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.346+0000 7f6cc9e92640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cc40d5280 con 0x7f6cc40d4800 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.346+0000 7f6cc9e92640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6cc40b5620 con 0x7f6cc40b5210 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.346+0000 7f6cc9e92640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc41609a0 con 0x7f6cc40b7ed0 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cc40b5210 0x7f6cc41613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cc40b5210 0x7f6cc41613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41640/0 (socket says 172.21.15.46:41640) 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 -- 172.21.15.46:0/587754856 learned_addr learned my addr 172.21.15.46:0/587754856 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc3fff640 1 --2- 172.21.15.46:0/587754856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40b7ed0 0x7f6cc40d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 -- 172.21.15.46:0/587754856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40b7ed0 msgr2=0x7f6cc40d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 --2- 172.21.15.46:0/587754856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40b7ed0 0x7f6cc40d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 -- 172.21.15.46:0/587754856 >> v1:172.21.15.88:6789/0 conn(0x7f6cc40d4800 legacy=0x7f6cc40b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.347+0000 7f6cc37fe640 1 -- 172.21.15.46:0/587754856 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc40d5560 con 0x7f6cc40b5210 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.348+0000 7f6cc37fe640 1 --2- 172.21.15.46:0/587754856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc41613c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4009870 tx=0x7f6cb402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a244c7ed9a3306ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.348+0000 7f6cc27fc640 1 -- 172.21.15.46:0/587754856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4037070 con 0x7f6cc40b5210 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.348+0000 7f6cc27fc640 1 -- 172.21.15.46:0/587754856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb4032440 con 0x7f6cc40b5210 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.348+0000 7f6cc27fc640 1 -- 172.21.15.46:0/587754856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4032810 con 0x7f6cc40b5210 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 -- 172.21.15.46:0/587754856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 msgr2=0x7f6cc41613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/587754856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc41613c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4009870 tx=0x7f6cb402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 -- 172.21.15.46:0/587754856 shutdown_connections 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/587754856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40b7ed0 0x7f6cc40d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/587754856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc41613c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 -- 172.21.15.46:0/587754856 >> 172.21.15.46:0/587754856 conn(0x7f6cc40d29a0 msgr2=0x7f6cc40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 -- 172.21.15.46:0/587754856 shutdown_connections 2024-05-09T03:14:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.349+0000 7f6cc9e92640 1 -- 172.21.15.46:0/587754856 wait complete. 2024-05-09T03:14:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.350+0000 7f6cc9e92640 1 Processor -- start 2024-05-09T03:14:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.351+0000 7f6cc9e92640 1 -- start start 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc9e92640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc4148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc9e92640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40d4800 0x7f6cc40caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc9e92640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cc40cafe0 0x7f6cc41f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc9e92640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc40d0ae0 con 0x7f6cc40b5210 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc9e92640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc4160ea0 con 0x7f6cc40cafe0 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc9e92640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc4163e60 con 0x7f6cc40d4800 2024-05-09T03:14:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc4148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc4148530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41656/0 (socket says 172.21.15.46:41656) 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 -- 172.21.15.46:0/3781935283 learned_addr learned my addr 172.21.15.46:0/3781935283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc3fff640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cc40cafe0 0x7f6cc41f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc2ffd640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40d4800 0x7f6cc40caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 -- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40d4800 msgr2=0x7f6cc40caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40d4800 0x7f6cc40caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 -- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cc40cafe0 msgr2=0x7f6cc41f58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cc40cafe0 0x7f6cc41f58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.352+0000 7f6cc37fe640 1 -- 172.21.15.46:0/3781935283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc40cb820 con 0x7f6cc40b5210 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.353+0000 7f6cc37fe640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc4148530 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4000c00 tx=0x7f6cb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.353+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4037040 con 0x7f6cc40b5210 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.353+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb4042070 con 0x7f6cc40b5210 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.353+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc40cbb00 con 0x7f6cc40b5210 2024-05-09T03:14:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.353+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4030bd0 con 0x7f6cc40b5210 2024-05-09T03:14:45.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.353+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc40cc0c0 con 0x7f6cc40b5210 2024-05-09T03:14:45.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.354+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc406a270 con 0x7f6cc40b5210 2024-05-09T03:14:45.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.355+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6cb402ad10 con 0x7f6cc40b5210 2024-05-09T03:14:45.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.355+0000 7f6cabfff640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cb00762d0 0x7f6cb00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.355+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7f6cb40b7c40 con 0x7f6cc40b5210 2024-05-09T03:14:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.355+0000 7f6cc2ffd640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cb00762d0 0x7f6cb00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.356+0000 7f6cc2ffd640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cb00762d0 0x7f6cb00787e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40b8b60 tx=0x7f6cb8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:45.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.360+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6cb4081110 con 0x7f6cc40b5210 2024-05-09T03:14:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.553+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6cc40d40d0 con 0x7f6cc40b5210 2024-05-09T03:14:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.554+0000 7f6cabfff640 1 -- 172.21.15.46:0/3781935283 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+13206 (secure 0 0 0) 0x7f6cb4085410 con 0x7f6cc40b5210 2024-05-09T03:14:45.554 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:45.555 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":60,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:14:43.189471+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6832/1536534537":"2024-05-10T03:14:40.180078+0000","172.21.15.46:6835/103238544":"2024-05-10T03:12:53.158166+0000","172.21.15.46:6834/103238544":"2024-05-10T03:12:53.158166+0000","172.21.15.46:6833/1536534537":"2024-05-10T03:14:40.180078+0000","172.21.15.46:6835/4263404372":"2024-05-10T03:12:22.212819+0000","172.21.15.46:6834/4263404372":"2024-05-10T03:12:22.212819+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cb00762d0 msgr2=0x7f6cb00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cb00762d0 0x7f6cb00787e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40b8b60 tx=0x7f6cb8009040 comp rx=0 tx=0).stop 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 msgr2=0x7f6cc4148530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc4148530 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6cb4000c00 tx=0x7f6cb40046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 shutdown_connections 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cc40cafe0 0x7f6cc41f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cc40d4800 0x7f6cc40caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cb00762d0 0x7f6cb00787e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 --2- 172.21.15.46:0/3781935283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cc40b5210 0x7f6cc4148530 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 >> 172.21.15.46:0/3781935283 conn(0x7f6cc40d29a0 msgr2=0x7f6cc40d5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 shutdown_connections 2024-05-09T03:14:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.556+0000 7f6cc9e92640 1 -- 172.21.15.46:0/3781935283 wait complete. 2024-05-09T03:14:45.568 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T03:09:22.506928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T03:14:45.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T03:14:45.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.684+0000 7f8dedaee640 1 Processor -- start 2024-05-09T03:14:45.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.684+0000 7f8dedaee640 1 -- start start 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8dedaee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8de80b7e70 0x7f8de80b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8dedaee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de80b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8dedaee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8de80d3400 con 0x7f8de80d2e90 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8dedaee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8de80d4a60 con 0x7f8de80b7e70 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8dedaee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8de80d4d40 con 0x7f8de81605b0 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8de80b7e70 0x7f8de80b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8de80d2e90 0x7f8de80b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34604/0 (socket says 172.21.15.46:34604) 2024-05-09T03:14:45.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de77fe640 1 -- 172.21.15.46:0/3574400399 learned_addr learned my addr 172.21.15.46:0/3574400399 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:45.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de7fff640 1 --2- 172.21.15.46:0/3574400399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de80b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de6ffd640 1 -- 172.21.15.46:0/3574400399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 msgr2=0x7f8de80b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:45.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de6ffd640 1 --2- 172.21.15.46:0/3574400399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de80b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de6ffd640 1 -- 172.21.15.46:0/3574400399 >> v1:172.21.15.46:6789/0 conn(0x7f8de80d2e90 legacy=0x7f8de80b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de6ffd640 1 -- 172.21.15.46:0/3574400399 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de80d5020 con 0x7f8de80b7e70 2024-05-09T03:14:45.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.686+0000 7f8de67fc640 1 -- 172.21.15.46:0/3574400399 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919020283 0 0) 0x7f8de80d3400 con 0x7f8de80d2e90 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.687+0000 7f8de6ffd640 1 --2- 172.21.15.46:0/3574400399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de80b7e70 0x7f8de80b92a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0097a0 tx=0x7f8ddc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4281a2208f70c9b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.687+0000 7f8de67fc640 1 -- 172.21.15.46:0/3574400399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc037070 con 0x7f8de80b7e70 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.687+0000 7f8de67fc640 1 -- 172.21.15.46:0/3574400399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ddc032440 con 0x7f8de80b7e70 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.687+0000 7f8de67fc640 1 -- 172.21.15.46:0/3574400399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc032810 con 0x7f8de80b7e70 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 -- 172.21.15.46:0/3574400399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de80b7e70 msgr2=0x7f8de80b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 --2- 172.21.15.46:0/3574400399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de80b7e70 0x7f8de80b92a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0097a0 tx=0x7f8ddc02a180 comp rx=0 tx=0).stop 2024-05-09T03:14:45.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 -- 172.21.15.46:0/3574400399 shutdown_connections 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 --2- 172.21.15.46:0/3574400399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de80b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 --2- 172.21.15.46:0/3574400399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de80b7e70 0x7f8de80b92a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 -- 172.21.15.46:0/3574400399 >> 172.21.15.46:0/3574400399 conn(0x7f8de808d390 msgr2=0x7f8de808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 -- 172.21.15.46:0/3574400399 shutdown_connections 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.688+0000 7f8dedaee640 1 -- 172.21.15.46:0/3574400399 wait complete. 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.690+0000 7f8dedaee640 1 Processor -- start 2024-05-09T03:14:45.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.690+0000 7f8dedaee640 1 -- start start 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8dedaee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de80d2e90 0x7f8de8162640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8dedaee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de8162b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8dedaee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 0x7f8de80c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8dedaee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8de80b4d80 con 0x7f8de8152f40 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8dedaee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8de80b4c00 con 0x7f8de80d2e90 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8dedaee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8de80b4fc0 con 0x7f8de81605b0 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 0x7f8de80c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 0x7f8de80c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41688/0 (socket says 172.21.15.46:41688) 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.691+0000 7f8de7fff640 1 -- 172.21.15.46:0/2686104179 learned_addr learned my addr 172.21.15.46:0/2686104179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:45.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de77fe640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de80d2e90 0x7f8de8162640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de6ffd640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de8162b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de7fff640 1 -- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 msgr2=0x7f8de8162b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de7fff640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de8162b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de7fff640 1 -- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de80d2e90 msgr2=0x7f8de8162640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de7fff640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de80d2e90 0x7f8de8162640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de7fff640 1 -- 172.21.15.46:0/2686104179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de81638d0 con 0x7f8de8152f40 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8de7fff640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 0x7f8de80c9640 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8de0007d70 tx=0x7f8de000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0014420 con 0x7f8de8152f40 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de0019070 con 0x7f8de8152f40 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de80cec80 con 0x7f8de8152f40 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0014820 con 0x7f8de8152f40 2024-05-09T03:14:45.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.692+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de81635f0 con 0x7f8de8152f40 2024-05-09T03:14:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.693+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de806a270 con 0x7f8de8152f40 2024-05-09T03:14:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.694+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8de00149c0 con 0x7f8de8152f40 2024-05-09T03:14:45.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.694+0000 7f8dcffff640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dd40762d0 0x7f8dd40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:45.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.694+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7f8de0097c00 con 0x7f8de8152f40 2024-05-09T03:14:45.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.695+0000 7f8de77fe640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dd40762d0 0x7f8dd40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:45.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.695+0000 7f8de77fe640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dd40762d0 0x7f8dd40787e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8000f60 tx=0x7f8dd8007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:45.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.699+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8de0061710 con 0x7f8de8152f40 2024-05-09T03:14:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.888+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8de80d4480 con 0x7f8de8152f40 2024-05-09T03:14:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.889+0000 7f8dcffff640 1 -- 172.21.15.46:0/2686104179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) ==== 93+0+10 (secure 0 0 0) 0x7f8de0061010 con 0x7f8de8152f40 2024-05-09T03:14:45.889 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dd40762d0 msgr2=0x7f8dd40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dd40762d0 0x7f8dd40787e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8000f60 tx=0x7f8dd8007c00 comp rx=0 tx=0).stop 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 msgr2=0x7f8de80c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 0x7f8de80c9640 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8de0007d70 tx=0x7f8de000e4b0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 shutdown_connections 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dd40762d0 0x7f8dd40787e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8de8152f40 0x7f8de80c9640 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8de81605b0 0x7f8de8162b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 --2- 172.21.15.46:0/2686104179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8de80d2e90 0x7f8de8162640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 >> 172.21.15.46:0/2686104179 conn(0x7f8de808d390 msgr2=0x7f8de80d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 shutdown_connections 2024-05-09T03:14:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:45.891+0000 7f8dedaee640 1 -- 172.21.15.46:0/2686104179 wait complete. 2024-05-09T03:14:45.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-05-09T03:14:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.017+0000 7fc08c234640 1 Processor -- start 2024-05-09T03:14:46.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.017+0000 7fc08c234640 1 -- start start 2024-05-09T03:14:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc08c234640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0840b2f70 0x7fc08408d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc08c234640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc08406de60 0x7fc08406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc08c234640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc08406e7b0 con 0x7fc0840d1f60 2024-05-09T03:14:46.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc08c234640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc0840b3380 con 0x7fc08406de60 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc08c234640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc0840d2440 con 0x7fc0840b2f70 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc089fd0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc0840d1f60 0x7fc0840d34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54826/0 (socket says 172.21.15.46:54826) 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc089fd0640 1 -- 172.21.15.46:0/2371945545 learned_addr learned my addr 172.21.15.46:0/2371945545 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc0897cf640 1 --2- 172.21.15.46:0/2371945545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0840b2f70 0x7fc08408d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.018+0000 7fc08a7d1640 1 --2- 172.21.15.46:0/2371945545 >> v2:172.21.15.88:3300/0 conn(0x7fc08406de60 0x7fc08406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.019+0000 7fc088fce640 1 -- 172.21.15.46:0/2371945545 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252995872 0 0) 0x7fc07c001110 con 0x7fc0840d1f60 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.019+0000 7fc088fce640 1 -- 172.21.15.46:0/2371945545 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc06c003520 con 0x7fc0840d1f60 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.019+0000 7fc0897cf640 1 -- 172.21.15.46:0/2371945545 >> v2:172.21.15.88:3300/0 conn(0x7fc08406de60 msgr2=0x7fc08406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.019+0000 7fc0897cf640 1 --2- 172.21.15.46:0/2371945545 >> v2:172.21.15.88:3300/0 conn(0x7fc08406de60 0x7fc08406e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.019+0000 7fc0897cf640 1 -- 172.21.15.46:0/2371945545 >> v1:172.21.15.46:6789/0 conn(0x7fc0840d1f60 legacy=0x7fc0840d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:46.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.019+0000 7fc0897cf640 1 -- 172.21.15.46:0/2371945545 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0840d2720 con 0x7fc0840b2f70 2024-05-09T03:14:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.020+0000 7fc0897cf640 1 --2- 172.21.15.46:0/2371945545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0840b2f70 0x7fc08408d770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0780097a0 tx=0x7fc07802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf4ef935f691ee3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:46.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.020+0000 7fc088fce640 1 -- 172.21.15.46:0/2371945545 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc078037070 con 0x7fc0840b2f70 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.020+0000 7fc088fce640 1 -- 172.21.15.46:0/2371945545 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc078032440 con 0x7fc0840b2f70 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.020+0000 7fc088fce640 1 -- 172.21.15.46:0/2371945545 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0780327f0 con 0x7fc0840b2f70 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 -- 172.21.15.46:0/2371945545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0840b2f70 msgr2=0x7fc08408d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 --2- 172.21.15.46:0/2371945545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0840b2f70 0x7fc08408d770 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0780097a0 tx=0x7fc07802a180 comp rx=0 tx=0).stop 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 -- 172.21.15.46:0/2371945545 shutdown_connections 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 --2- 172.21.15.46:0/2371945545 >> v2:172.21.15.88:3300/0 conn(0x7fc08406de60 0x7fc08406e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 --2- 172.21.15.46:0/2371945545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc0840b2f70 0x7fc08408d770 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 -- 172.21.15.46:0/2371945545 >> 172.21.15.46:0/2371945545 conn(0x7fc0840b56e0 msgr2=0x7fc0840b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 -- 172.21.15.46:0/2371945545 shutdown_connections 2024-05-09T03:14:46.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.021+0000 7fc08c234640 1 -- 172.21.15.46:0/2371945545 wait complete. 2024-05-09T03:14:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.022+0000 7fc08c234640 1 Processor -- start 2024-05-09T03:14:46.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.023+0000 7fc08c234640 1 -- start start 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08c234640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc08406de60 0x7fc084155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08c234640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840b2f70 0x7fc08415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08c234640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 0x7fc08415d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08c234640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc0840d2fc0 con 0x7fc0840d1f60 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08c234640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc0840d2ce0 con 0x7fc0840b2f70 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08c234640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc08408dcb0 con 0x7fc08406de60 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 0x7fc08415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 0x7fc08415d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37522/0 (socket says 172.21.15.46:37522) 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 -- 172.21.15.46:0/2449239225 learned_addr learned my addr 172.21.15.46:0/2449239225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc089fd0640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc08406de60 0x7fc084155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:46.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc0897cf640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840b2f70 0x7fc08415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 -- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc08406de60 msgr2=0x7fc084155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc08406de60 0x7fc084155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 -- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840b2f70 msgr2=0x7fc08415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840b2f70 0x7fc08415cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.024+0000 7fc08a7d1640 1 -- 172.21.15.46:0/2449239225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08415c870 con 0x7fc0840d1f60 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.025+0000 7fc08a7d1640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 0x7fc08415d260 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc074009cb0 tx=0x7fc07400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.025+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074018020 con 0x7fc0840d1f60 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.025+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074014480 con 0x7fc0840d1f60 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.025+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc084163420 con 0x7fc0840d1f60 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.025+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074014870 con 0x7fc0840d1f60 2024-05-09T03:14:46.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.025+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08415dda0 con 0x7fc0840d1f60 2024-05-09T03:14:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.026+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0840c7c00 con 0x7fc0840d1f60 2024-05-09T03:14:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.026+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc074005d10 con 0x7fc0840d1f60 2024-05-09T03:14:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.027+0000 7fc0827fc640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc070076300 0x7fc070078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.027+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7fc074097f90 con 0x7fc0840d1f60 2024-05-09T03:14:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.027+0000 7fc089fd0640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc070076300 0x7fc070078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.028+0000 7fc089fd0640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc070076300 0x7fc070078810 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc07c005880 tx=0x7fc07c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:46.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.032+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc074022030 con 0x7fc0840d1f60 2024-05-09T03:14:46.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:46.297+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) -- 0x7fc08406a270 con 0x7fc0840d1f60 2024-05-09T03:14:47.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.210+0000 7fc0827fc640 1 -- 172.21.15.46:0/2449239225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v249) ==== 144+0+0 (secure 0 0 0) 0x7fc074010040 con 0x7fc0840d1f60 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc070076300 msgr2=0x7fc070078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc070076300 0x7fc070078810 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc07c005880 tx=0x7fc07c009040 comp rx=0 tx=0).stop 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 msgr2=0x7fc08415d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 0x7fc08415d260 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc074009cb0 tx=0x7fc07400c490 comp rx=0 tx=0).stop 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 shutdown_connections 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc070076300 0x7fc070078810 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0840d1f60 0x7fc08415d260 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840b2f70 0x7fc08415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 --2- 172.21.15.46:0/2449239225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc08406de60 0x7fc084155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 >> 172.21.15.46:0/2449239225 conn(0x7fc0840b56e0 msgr2=0x7fc0840b4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 shutdown_connections 2024-05-09T03:14:47.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.214+0000 7fc08c234640 1 -- 172.21.15.46:0/2449239225 wait complete. 2024-05-09T03:14:47.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T03:14:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.352+0000 7f9733cc6640 1 Processor -- start 2024-05-09T03:14:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.352+0000 7f9733cc6640 1 -- start start 2024-05-09T03:14:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9733cc6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0d2e90 0x7f972c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9733cc6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f972c1605b0 0x7f972c0b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9733cc6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f972c0d3400 con 0x7f972c0b4fc0 2024-05-09T03:14:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9733cc6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f972c0d4a60 con 0x7f972c1605b0 2024-05-09T03:14:47.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9733cc6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f972c0d4d40 con 0x7f972c0d2e90 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f972c1605b0 0x7f972c0b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f972c1605b0 0x7f972c0b7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37528/0 (socket says 172.21.15.46:37528) 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 -- 172.21.15.46:0/3987072177 learned_addr learned my addr 172.21.15.46:0/3987072177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9732263640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f972c0b4fc0 0x7f972c0d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54836/0 (socket says 172.21.15.46:54836) 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731a62640 1 --2- 172.21.15.46:0/3987072177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0d2e90 0x7f972c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 -- 172.21.15.46:0/3987072177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0d2e90 msgr2=0x7f972c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 --2- 172.21.15.46:0/3987072177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0d2e90 0x7f972c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 -- 172.21.15.46:0/3987072177 >> v1:172.21.15.46:6789/0 conn(0x7f972c0b4fc0 legacy=0x7f972c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9730a60640 1 -- 172.21.15.46:0/3987072177 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684903837 0 0) 0x7f972c0d3400 con 0x7f972c0b4fc0 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.354+0000 7f9731261640 1 -- 172.21.15.46:0/3987072177 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c0d5020 con 0x7f972c1605b0 2024-05-09T03:14:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.355+0000 7f9731261640 1 --2- 172.21.15.46:0/3987072177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c1605b0 0x7f972c0b7ca0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f971c009870 tx=0x7f971c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921ba57de1afb1d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:47.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.355+0000 7f9730a60640 1 -- 172.21.15.46:0/3987072177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c037070 con 0x7f972c1605b0 2024-05-09T03:14:47.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.355+0000 7f9730a60640 1 -- 172.21.15.46:0/3987072177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f971c032440 con 0x7f972c1605b0 2024-05-09T03:14:47.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.355+0000 7f9730a60640 1 -- 172.21.15.46:0/3987072177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c032810 con 0x7f972c1605b0 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 -- 172.21.15.46:0/3987072177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c1605b0 msgr2=0x7f972c0b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 --2- 172.21.15.46:0/3987072177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c1605b0 0x7f972c0b7ca0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f971c009870 tx=0x7f971c02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 -- 172.21.15.46:0/3987072177 shutdown_connections 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 --2- 172.21.15.46:0/3987072177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c1605b0 0x7f972c0b7ca0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 --2- 172.21.15.46:0/3987072177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0d2e90 0x7f972c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 -- 172.21.15.46:0/3987072177 >> 172.21.15.46:0/3987072177 conn(0x7f972c08d390 msgr2=0x7f972c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 -- 172.21.15.46:0/3987072177 shutdown_connections 2024-05-09T03:14:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.356+0000 7f9733cc6640 1 -- 172.21.15.46:0/3987072177 wait complete. 2024-05-09T03:14:47.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.358+0000 7f9733cc6640 1 Processor -- start 2024-05-09T03:14:47.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.358+0000 7f9733cc6640 1 -- start start 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9733cc6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0b4fc0 0x7f972c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9733cc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 0x7f972c0cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9733cc6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c1605b0 0x7f972c0ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9733cc6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f972c0d5540 con 0x7f972c0d2e90 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9733cc6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f972c0b4c00 con 0x7f972c1605b0 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9733cc6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f972c161af0 con 0x7f972c0b4fc0 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9731261640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 0x7f972c0cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9731261640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 0x7f972c0cdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37544/0 (socket says 172.21.15.46:37544) 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9731261640 1 -- 172.21.15.46:0/1360875086 learned_addr learned my addr 172.21.15.46:0/1360875086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9731a62640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0b4fc0 0x7f972c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9732263640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c1605b0 0x7f972c0ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9731261640 1 -- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0b4fc0 msgr2=0x7f972c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.359+0000 7f9731261640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0b4fc0 0x7f972c158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f9731261640 1 -- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c1605b0 msgr2=0x7f972c0ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f9731261640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c1605b0 0x7f972c0ce0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f9731261640 1 -- 172.21.15.46:0/1360875086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c0cf9e0 con 0x7f972c0d2e90 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f9731261640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 0x7f972c0cdb60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f971c009870 tx=0x7f971c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c037040 con 0x7f972c0d2e90 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f971c004250 con 0x7f972c0d2e90 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972c0cfcc0 con 0x7f972c0d2e90 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c03c670 con 0x7f972c0d2e90 2024-05-09T03:14:47.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.360+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f972c1594c0 con 0x7f972c0d2e90 2024-05-09T03:14:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.361+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f972c06a270 con 0x7f972c0d2e90 2024-05-09T03:14:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.362+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f971c03a070 con 0x7f972c0d2e90 2024-05-09T03:14:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.363+0000 7f97227fc640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97100762d0 0x7f97100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.363+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7f971c0b7fa0 con 0x7f972c0d2e90 2024-05-09T03:14:47.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.363+0000 7f9731a62640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97100762d0 0x7f97100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.364+0000 7f9731a62640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97100762d0 0x7f97100787e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f972c06f130 tx=0x7f9728009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.367+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f971c041020 con 0x7f972c0d2e90 2024-05-09T03:14:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.558+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f972c0cad00 con 0x7f972c0d2e90 2024-05-09T03:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.561+0000 7f97227fc640 1 -- 172.21.15.46:0/1360875086 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7f971c081550 con 0x7f972c0d2e90 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97100762d0 msgr2=0x7f97100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97100762d0 0x7f97100787e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f972c06f130 tx=0x7f9728009190 comp rx=0 tx=0).stop 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 msgr2=0x7f972c0cdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 0x7f972c0cdb60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f971c009870 tx=0x7f971c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 shutdown_connections 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f972c1605b0 0x7f972c0ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97100762d0 0x7f97100787e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f972c0d2e90 0x7f972c0cdb60 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 --2- 172.21.15.46:0/1360875086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f972c0b4fc0 0x7f972c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 >> 172.21.15.46:0/1360875086 conn(0x7f972c08d390 msgr2=0x7f972c15ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 shutdown_connections 2024-05-09T03:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.564+0000 7f9733cc6640 1 -- 172.21.15.46:0/1360875086 wait complete. 2024-05-09T03:14:47.576 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:14:47.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:14:47.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:14:47.661 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:14:47.661 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:14:47.662 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:47.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:14:47.678 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:14:47.678 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:47.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:14:47.695 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:14:47.696 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:47.734 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:14:47.734 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:14:47.735 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:14:47.752 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:14:47.752 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T03:14:47.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.889+0000 7fd1a991f640 1 Processor -- start 2024-05-09T03:14:47.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.889+0000 7fd1a991f640 1 -- start start 2024-05-09T03:14:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a991f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1a4164c70 0x7fd1a41615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a991f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a40d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a991f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1a4160b20 con 0x7fd1a40d2480 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a991f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd1a4161150 con 0x7fd1a4164c70 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a991f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1a40d15f0 con 0x7fd1a415dfe0 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1a4164c70 0x7fd1a41615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd1a40d2480 0x7fd1a40d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54844/0 (socket says 172.21.15.46:54844) 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a37fe640 1 -- 172.21.15.46:0/635712768 learned_addr learned my addr 172.21.15.46:0/635712768 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.890+0000 7fd1a27fc640 1 --2- 172.21.15.46:0/635712768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a40d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/635712768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 msgr2=0x7fd1a40d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a2ffd640 1 --2- 172.21.15.46:0/635712768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a40d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/635712768 >> v1:172.21.15.46:6789/0 conn(0x7fd1a40d2480 legacy=0x7fd1a40d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a1ffb640 1 -- 172.21.15.46:0/635712768 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539803315 0 0) 0x7fd1a4160b20 con 0x7fd1a40d2480 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/635712768 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a40d18d0 con 0x7fd1a4164c70 2024-05-09T03:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a2ffd640 1 --2- 172.21.15.46:0/635712768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a4164c70 0x7fd1a41615e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd1940097a0 tx=0x7fd19402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78fdc044d2f0258 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a1ffb640 1 -- 172.21.15.46:0/635712768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194037070 con 0x7fd1a4164c70 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a1ffb640 1 -- 172.21.15.46:0/635712768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd19402ae50 con 0x7fd1a4164c70 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.891+0000 7fd1a1ffb640 1 -- 172.21.15.46:0/635712768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194032660 con 0x7fd1a4164c70 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 -- 172.21.15.46:0/635712768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a4164c70 msgr2=0x7fd1a41615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 --2- 172.21.15.46:0/635712768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a4164c70 0x7fd1a41615e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd1940097a0 tx=0x7fd19402a100 comp rx=0 tx=0).stop 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 -- 172.21.15.46:0/635712768 shutdown_connections 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 --2- 172.21.15.46:0/635712768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a40d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 --2- 172.21.15.46:0/635712768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a4164c70 0x7fd1a41615e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 -- 172.21.15.46:0/635712768 >> 172.21.15.46:0/635712768 conn(0x7fd1a4073db0 msgr2=0x7fd1a40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 -- 172.21.15.46:0/635712768 shutdown_connections 2024-05-09T03:14:47.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.892+0000 7fd1a991f640 1 -- 172.21.15.46:0/635712768 wait complete. 2024-05-09T03:14:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.894+0000 7fd1a991f640 1 Processor -- start 2024-05-09T03:14:47.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.894+0000 7fd1a991f640 1 -- start start 2024-05-09T03:14:47.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.895+0000 7fd1a991f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 0x7fd1a41f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.895+0000 7fd1a991f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a415a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.895+0000 7fd1a991f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1a4164c70 0x7fd1a415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.895+0000 7fd1a991f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a415e3f0 con 0x7fd1a40d2480 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.895+0000 7fd1a991f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a40d1300 con 0x7fd1a4164c70 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.895+0000 7fd1a991f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1a40d45c0 con 0x7fd1a415dfe0 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 0x7fd1a41f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 0x7fd1a41f6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37568/0 (socket says 172.21.15.46:37568) 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/713143219 learned_addr learned my addr 172.21.15.46:0/713143219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a27fc640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a415a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a37fe640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1a4164c70 0x7fd1a415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 msgr2=0x7fd1a415a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a415a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1a4164c70 msgr2=0x7fd1a415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1a4164c70 0x7fd1a415bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:47.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 -- 172.21.15.46:0/713143219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a41f9e10 con 0x7fd1a40d2480 2024-05-09T03:14:47.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a2ffd640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 0x7fd1a41f6ec0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd194000c00 tx=0x7fd1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:47.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194041070 con 0x7fd1a40d2480 2024-05-09T03:14:47.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd19403c460 con 0x7fd1a40d2480 2024-05-09T03:14:47.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a41fa0f0 con 0x7fd1a40d2480 2024-05-09T03:14:47.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194037040 con 0x7fd1a40d2480 2024-05-09T03:14:47.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.896+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a41fa6b0 con 0x7fd1a40d2480 2024-05-09T03:14:47.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.897+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a408cfd0 con 0x7fd1a40d2480 2024-05-09T03:14:47.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.898+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd194044070 con 0x7fd1a40d2480 2024-05-09T03:14:47.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.899+0000 7fd18bfff640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1900762d0 0x7fd1900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:47.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.899+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7fd1940b8230 con 0x7fd1a40d2480 2024-05-09T03:14:47.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.899+0000 7fd1a27fc640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1900762d0 0x7fd1900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:47.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.900+0000 7fd1a27fc640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1900762d0 0x7fd1900787e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd1a415fbd0 tx=0x7fd198009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:47.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:47.903+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd1a408cfd0 con 0x7fd1a40d2480 2024-05-09T03:14:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.132+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1a406a270 con 0x7fd1a40d2480 2024-05-09T03:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.132+0000 7fd18bfff640 1 -- 172.21.15.46:0/713143219 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+3869 (secure 0 0 0) 0x7fd1940817e0 con 0x7fd1a40d2480 2024-05-09T03:14:48.133 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:48.133 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":62,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5246,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/4025806262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4025806262},{"type":"v1","addr":"172.21.15.88:6836","nonce":4025806262}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":5515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2190348639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2190348639},{"type":"v1","addr":"172.21.15.88:6837","nonce":2190348639}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41},{"gid":6240,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3025315884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3025315884},{"type":"v1","addr":"172.21.15.46:6835","nonce":3025315884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":6363,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/3079069237","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3079069237},{"type":"v1","addr":"172.21.15.46:6833","nonce":3079069237}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[]} 2024-05-09T03:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 62 2024-05-09T03:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1900762d0 msgr2=0x7fd1900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1900762d0 0x7fd1900787e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd1a415fbd0 tx=0x7fd198009190 comp rx=0 tx=0).stop 2024-05-09T03:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 msgr2=0x7fd1a41f6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 0x7fd1a41f6ec0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd194000c00 tx=0x7fd1940046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 shutdown_connections 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1a4164c70 0x7fd1a415bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1a415dfe0 0x7fd1a415a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1900762d0 0x7fd1900787e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 --2- 172.21.15.46:0/713143219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1a40d2480 0x7fd1a41f6ec0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 >> 172.21.15.46:0/713143219 conn(0x7fd1a4073db0 msgr2=0x7fd1a40b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.134+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 shutdown_connections 2024-05-09T03:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.135+0000 7fd1a991f640 1 -- 172.21.15.46:0/713143219 wait complete. 2024-05-09T03:14:48.147 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:14:48.147 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:14:48.147 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:14:48.147 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.147 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:14:48.147 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T03:14:48.147 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:14:48.148 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.148 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:14:48.148 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T03:14:48.148 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:14:48.148 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.148 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:14:48.148 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T03:14:48.148 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:14:48.149 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:14:48.159 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:14:48.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:14:48.160 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:14:48.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:14:48.161 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:14:48.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:14:48.162 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:14:48.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:14:48.203 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:14:48.204 DEBUG:teuthology.parallel:result is None 2024-05-09T03:14:48.204 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:14:48.205 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:14:48.205 DEBUG:teuthology.parallel:result is None 2024-05-09T03:14:48.205 DEBUG:teuthology.parallel:result is None 2024-05-09T03:14:48.206 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:14:48.206 DEBUG:teuthology.parallel:result is None 2024-05-09T03:14:48.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T03:14:48.293 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:14:48.294 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:14:48.296 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:14:48.296 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:14:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.327+0000 7fbeafee0640 1 Processor -- start 2024-05-09T03:14:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.328+0000 7fbeafee0640 1 -- start start 2024-05-09T03:14:48.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbeafee0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80b1910 0x7fbea815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:48.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbeafee0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea80b0880 con 0x7fbea80d3210 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbeafee0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea80b0b60 con 0x7fbea80d2310 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbeafee0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbea80d3620 con 0x7fbea80b1910 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbead47b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbea80d3210 0x7fbea80d1c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54856/0 (socket says 172.21.15.46:54856) 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbead47b640 1 -- 172.21.15.46:0/3530045428 learned_addr learned my addr 172.21.15.46:0/3530045428 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.329+0000 7fbeadc7c640 1 --2- 172.21.15.46:0/3530045428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80b1910 0x7fbea815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663802861 0 0) 0x7fbea80b0880 con 0x7fbea80d3210 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe94003520 con 0x7fbea80d3210 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902172095 0 0) 0x7fbea80b0b60 con 0x7fbea80d2310 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea80b0880 con 0x7fbea80d2310 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3530045428 >> v1:172.21.15.88:6789/0 conn(0x7fbea80d2310 legacy=0x7fbea806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3530045428 >> v1:172.21.15.46:6789/0 conn(0x7fbea80d3210 legacy=0x7fbea80d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.330+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3530045428 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea80b0e40 con 0x7fbea80b1910 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeadc7c640 1 --2- 172.21.15.46:0/3530045428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80b1910 0x7fbea815e3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbea40097d0 tx=0x7fbea402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f37098857c9c32ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4037070 con 0x7fbea80b1910 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea4032440 con 0x7fbea80b1910 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeacc7a640 1 -- 172.21.15.46:0/3530045428 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4032810 con 0x7fbea80b1910 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeafee0640 1 -- 172.21.15.46:0/3530045428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80b1910 msgr2=0x7fbea815e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3530045428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80b1910 0x7fbea815e3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbea40097d0 tx=0x7fbea402a180 comp rx=0 tx=0).stop 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.331+0000 7fbeafee0640 1 -- 172.21.15.46:0/3530045428 shutdown_connections 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.332+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3530045428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80b1910 0x7fbea815e3c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.332+0000 7fbeafee0640 1 -- 172.21.15.46:0/3530045428 >> 172.21.15.46:0/3530045428 conn(0x7fbea809b7d0 msgr2=0x7fbea809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.332+0000 7fbeafee0640 1 -- 172.21.15.46:0/3530045428 shutdown_connections 2024-05-09T03:14:48.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.332+0000 7fbeafee0640 1 -- 172.21.15.46:0/3530045428 wait complete. 2024-05-09T03:14:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.333+0000 7fbeafee0640 1 Processor -- start 2024-05-09T03:14:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.334+0000 7fbeafee0640 1 -- start start 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeafee0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 0x7fbea815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeafee0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80d2310 0x7fbea814c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeafee0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80d3210 0x7fbea814cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeafee0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbea806d830 con 0x7fbea80b1910 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeafee0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbea806d6b0 con 0x7fbea80d2310 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 0x7fbea815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeafee0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbea806d9b0 con 0x7fbea80d3210 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 0x7fbea815a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37606/0 (socket says 172.21.15.46:37606) 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3169327934 learned_addr learned my addr 172.21.15.46:0/3169327934 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:48.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbead47b640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80d2310 0x7fbea814c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeae47d640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80d3210 0x7fbea814cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80d3210 msgr2=0x7fbea814cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80d3210 0x7fbea814cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80d2310 msgr2=0x7fbea814c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80d2310 0x7fbea814c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.335+0000 7fbeadc7c640 1 -- 172.21.15.46:0/3169327934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea80c7d30 con 0x7fbea80b1910 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.336+0000 7fbeadc7c640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 0x7fbea815a4d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbea40097d0 tx=0x7fbea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.336+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4037040 con 0x7fbea80b1910 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.336+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea4004230 con 0x7fbea80b1910 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.336+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4003610 con 0x7fbea80b1910 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.336+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea80c8010 con 0x7fbea80b1910 2024-05-09T03:14:48.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.336+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea81574c0 con 0x7fbea80b1910 2024-05-09T03:14:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.337+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea8076370 con 0x7fbea80b1910 2024-05-09T03:14:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.338+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbea40037b0 con 0x7fbea80b1910 2024-05-09T03:14:48.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.338+0000 7fbe9a7fc640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe8c0762d0 0x7fbe8c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.338+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5115+0+0 (secure 0 0 0) 0x7fbea40b8340 con 0x7fbea80b1910 2024-05-09T03:14:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.339+0000 7fbead47b640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe8c0762d0 0x7fbe8c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.339+0000 7fbead47b640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe8c0762d0 0x7fbe8c0787e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005c00 tx=0x7fbe9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:48.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.342+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbea4081980 con 0x7fbea80b1910 2024-05-09T03:14:48.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-05-09T03:14:48.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-05-09T03:14:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:48.544+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fbea8076090 con 0x7fbea80b1910 2024-05-09T03:14:49.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.223+0000 7fbe9a7fc640 1 -- 172.21.15.46:0/3169327934 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v62) ==== 97+0+0 (secure 0 0 0) 0x7fbea4085c80 con 0x7fbea80b1910 2024-05-09T03:14:49.224 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe8c0762d0 msgr2=0x7fbe8c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe8c0762d0 0x7fbe8c0787e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005c00 tx=0x7fbe9c009040 comp rx=0 tx=0).stop 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 msgr2=0x7fbea815a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 0x7fbea815a4d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbea40097d0 tx=0x7fbea40046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 shutdown_connections 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbea80d3210 0x7fbea814cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80d2310 0x7fbea814c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe8c0762d0 0x7fbe8c0787e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 --2- 172.21.15.46:0/3169327934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbea80b1910 0x7fbea815a4d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 >> 172.21.15.46:0/3169327934 conn(0x7fbea809b7d0 msgr2=0x7fbea806dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 shutdown_connections 2024-05-09T03:14:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.226+0000 7fbeafee0640 1 -- 172.21.15.46:0/3169327934 wait complete. 2024-05-09T03:14:49.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T03:14:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.363+0000 7fd6f8e05640 1 Processor -- start 2024-05-09T03:14:49.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.363+0000 7fd6f8e05640 1 -- start start 2024-05-09T03:14:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f8e05640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd6f40b5210 0x7fd6f41613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f8e05640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40d4800 0x7fd6f40d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f8e05640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6f40b77c0 0x7fd6f40b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f8e05640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f40b5620 con 0x7fd6f40b77c0 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f8e05640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd6f40b8110 con 0x7fd6f40b5210 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f8e05640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f40d5280 con 0x7fd6f40d4800 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6f40b77c0 0x7fd6f40b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6f40b77c0 0x7fd6f40b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37614/0 (socket says 172.21.15.46:37614) 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 -- 172.21.15.46:0/3133703523 learned_addr learned my addr 172.21.15.46:0/3133703523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f259c640 1 --2- 172.21.15.46:0/3133703523 >> v2:172.21.15.88:3300/0 conn(0x7fd6f40b5210 0x7fd6f41613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f1d9b640 1 --2- 172.21.15.46:0/3133703523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40d4800 0x7fd6f40d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 -- 172.21.15.46:0/3133703523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40d4800 msgr2=0x7fd6f40d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 --2- 172.21.15.46:0/3133703523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40d4800 0x7fd6f40d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 -- 172.21.15.46:0/3133703523 >> v2:172.21.15.88:3300/0 conn(0x7fd6f40b5210 msgr2=0x7fd6f41613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 --2- 172.21.15.46:0/3133703523 >> v2:172.21.15.88:3300/0 conn(0x7fd6f40b5210 0x7fd6f41613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.365+0000 7fd6f2d9d640 1 -- 172.21.15.46:0/3133703523 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f40d5560 con 0x7fd6f40b77c0 2024-05-09T03:14:49.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.366+0000 7fd6f2d9d640 1 --2- 172.21.15.46:0/3133703523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40b7bd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd6f408ed70 tx=0x7fd6ec02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39571bac2680465 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:49.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.366+0000 7fd6f159a640 1 -- 172.21.15.46:0/3133703523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec036070 con 0x7fd6f40b77c0 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.366+0000 7fd6f159a640 1 -- 172.21.15.46:0/3133703523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ec031600 con 0x7fd6f40b77c0 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.366+0000 7fd6f159a640 1 -- 172.21.15.46:0/3133703523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec0319d0 con 0x7fd6f40b77c0 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 -- 172.21.15.46:0/3133703523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 msgr2=0x7fd6f40b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/3133703523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40b7bd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd6f408ed70 tx=0x7fd6ec02a610 comp rx=0 tx=0).stop 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 -- 172.21.15.46:0/3133703523 shutdown_connections 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/3133703523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40b7bd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/3133703523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40d4800 0x7fd6f40d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/3133703523 >> v2:172.21.15.88:3300/0 conn(0x7fd6f40b5210 0x7fd6f41613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 -- 172.21.15.46:0/3133703523 >> 172.21.15.46:0/3133703523 conn(0x7fd6f40d29a0 msgr2=0x7fd6f40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 -- 172.21.15.46:0/3133703523 shutdown_connections 2024-05-09T03:14:49.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.367+0000 7fd6f8e05640 1 -- 172.21.15.46:0/3133703523 wait complete. 2024-05-09T03:14:49.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.369+0000 7fd6f8e05640 1 Processor -- start 2024-05-09T03:14:49.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.369+0000 7fd6f8e05640 1 -- start start 2024-05-09T03:14:49.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f8e05640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40b5210 0x7fd6f4163340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f8e05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f8e05640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6f40d4800 0x7fd6f4156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f8e05640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f40d0ae0 con 0x7fd6f40b77c0 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f8e05640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f4160ea0 con 0x7fd6f40d4800 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f8e05640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd6f4163e60 con 0x7fd6f40b5210 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.370+0000 7fd6f1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37620/0 (socket says 172.21.15.46:37620) 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 -- 172.21.15.46:0/1918368084 learned_addr learned my addr 172.21.15.46:0/1918368084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f259c640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40b5210 0x7fd6f4163340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f2d9d640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6f40d4800 0x7fd6f4156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 -- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40b5210 msgr2=0x7fd6f4163340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40b5210 0x7fd6f4163340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 -- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6f40d4800 msgr2=0x7fd6f4156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6f40d4800 0x7fd6f4156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 -- 172.21.15.46:0/1918368084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f40ceae0 con 0x7fd6f40b77c0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f1d9b640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40cdb20 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800e870 tx=0x7fd6e800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8017070 con 0x7fd6f40b77c0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e800c940 con 0x7fd6f40b77c0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f40cd040 con 0x7fd6f40b77c0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e800cd30 con 0x7fd6f40b77c0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.371+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f40cd600 con 0x7fd6f40b77c0 2024-05-09T03:14:49.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.372+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f406a270 con 0x7fd6f40b77c0 2024-05-09T03:14:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.373+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd6e801a070 con 0x7fd6f40b77c0 2024-05-09T03:14:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.373+0000 7fd6e2ffd640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6d40762d0 0x7fd6d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.374+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7fd6e8097570 con 0x7fd6f40b77c0 2024-05-09T03:14:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.374+0000 7fd6f259c640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6d40762d0 0x7fd6d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.374+0000 7fd6f259c640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6d40762d0 0x7fd6d40787e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009a20 tx=0x7fd6e4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.378+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd6e8060aa0 con 0x7fd6f40b77c0 2024-05-09T03:14:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.646+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fd6f40ce870 con 0x7fd6f40b77c0 2024-05-09T03:14:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.646+0000 7fd6e2ffd640 1 -- 172.21.15.46:0/1918368084 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7fd6e8064da0 con 0x7fd6f40b77c0 2024-05-09T03:14:49.647 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g==", 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.648 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw==", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw==", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.649 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw==", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.650 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw==", 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA==", 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.651 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg==", 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.652 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA==", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g==", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.653 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA==", 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-05-09T03:14:49.654 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ==", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw==", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:14:49.655 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw==", 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.656 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ==", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg==", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg==", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA1PjxmuVPDJhAAHOgxPpTSEd3pibPqobcHCg==", 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.659 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg==", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ==", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:14:49.660 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:14:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.661 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-05-09T03:14:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:14:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6d40762d0 msgr2=0x7fd6d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6d40762d0 0x7fd6d40787e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009a20 tx=0x7fd6e4009190 comp rx=0 tx=0).stop 2024-05-09T03:14:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 msgr2=0x7fd6f40cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40cdb20 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800e870 tx=0x7fd6e800ed30 comp rx=0 tx=0).stop 2024-05-09T03:14:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 shutdown_connections 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6f40d4800 0x7fd6f4156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6d40762d0 0x7fd6d40787e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6f40b77c0 0x7fd6f40cdb20 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 --2- 172.21.15.46:0/1918368084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6f40b5210 0x7fd6f4163340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 >> 172.21.15.46:0/1918368084 conn(0x7fd6f40d29a0 msgr2=0x7fd6f40b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 shutdown_connections 2024-05-09T03:14:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.649+0000 7fd6f8e05640 1 -- 172.21.15.46:0/1918368084 wait complete. 2024-05-09T03:14:49.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:14:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.778+0000 7f110219f640 1 Processor -- start 2024-05-09T03:14:49.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.778+0000 7f110219f640 1 -- start start 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.779+0000 7f110219f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10fc06c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.779+0000 7f110219f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10fc160f40 con 0x7f10fc0b7180 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.779+0000 7f110219f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10fc161220 con 0x7f10fc0d3380 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.779+0000 7f110219f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f10fc161500 con 0x7f10fc0d2f70 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.779+0000 7f110119d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f10fc0b7180 0x7f10fc160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54866/0 (socket says 172.21.15.46:54866) 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.779+0000 7f110119d640 1 -- 172.21.15.46:0/3690080065 learned_addr learned my addr 172.21.15.46:0/3690080065 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f110099c640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10fc06c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992967168 0 0) 0x7f10fc160f40 con 0x7f10fc0b7180 2024-05-09T03:14:49.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e8003520 con 0x7f10fc0b7180 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4153307632 0 0) 0x7f10fc161220 con 0x7f10fc0d3380 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10fc160f40 con 0x7f10fc0d3380 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 478798082 0 0) 0x7f10e8003520 con 0x7f10fc0b7180 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 msgr2=0x7f10fc06c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10fc06c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 >> v1:172.21.15.88:6789/0 conn(0x7f10fc0d3380 legacy=0x7f10fc145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10fc0d2a50 con 0x7f10fc0b7180 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f10f80016f0 con 0x7f10fc0b7180 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f110099c640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10fc06c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 >> v1:172.21.15.46:6789/0 conn(0x7f10fc0b7180 legacy=0x7f10fc160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10e8003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004280 0x7f10e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8004e50 0x7f10e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc161500 con 0x7f10e8004e50 2024-05-09T03:14:49.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80055e0 con 0x7f10e8004280 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f10e800bf00 con 0x7f10fc0d2f70 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f110199e640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8004e50 0x7f10e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f110099c640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004280 0x7f10e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.780+0000 7f110119d640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10e8003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f110199e640 1 -- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 msgr2=0x7f10e8003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f110199e640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc0d2f70 0x7f10e8003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f110199e640 1 -- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004280 msgr2=0x7f10e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f110199e640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004280 0x7f10e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f110199e640 1 -- 172.21.15.46:0/3690080065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8003520 con 0x7f10e8004e50 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f110199e640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8004e50 0x7f10e800b9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f10f4002000 tx=0x7f10e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e150ecba0a8b5505 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.781+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4001c20 con 0x7f10e8004e50 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f4006d70 con 0x7f10e8004e50 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f10f3fff640 1 -- 172.21.15.46:0/3690080065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4003bb0 con 0x7f10e8004e50 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110219f640 1 -- 172.21.15.46:0/3690080065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8004e50 msgr2=0x7f10e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110219f640 1 --2- 172.21.15.46:0/3690080065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10e8004e50 0x7f10e800b9c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f10f4002000 tx=0x7f10e8005830 comp rx=0 tx=0).stop 2024-05-09T03:14:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110199e640 1 -- 172.21.15.46:0/3690080065 reap_dead start 2024-05-09T03:14:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110219f640 1 -- 172.21.15.46:0/3690080065 shutdown_connections 2024-05-09T03:14:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110219f640 1 -- 172.21.15.46:0/3690080065 >> 172.21.15.46:0/3690080065 conn(0x7f10fc0b46a0 msgr2=0x7f10fc0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110219f640 1 -- 172.21.15.46:0/3690080065 shutdown_connections 2024-05-09T03:14:49.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.782+0000 7f110219f640 1 -- 172.21.15.46:0/3690080065 wait complete. 2024-05-09T03:14:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.784+0000 7f110219f640 1 Processor -- start 2024-05-09T03:14:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.784+0000 7f110219f640 1 -- start start 2024-05-09T03:14:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.785+0000 7f110219f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004e50 0x7f10fc1e95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.785+0000 7f110219f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc1fa240 0x7f10fc141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.785+0000 7f110219f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 0x7f10fc0d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.785+0000 7f110219f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc146020 con 0x7f10fc0d1290 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.785+0000 7f110219f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc145ea0 con 0x7f10e8004e50 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.785+0000 7f110219f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f10fc1461a0 con 0x7f10fc1fa240 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 0x7f10fc0d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 0x7f10fc0d16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37652/0 (socket says 172.21.15.46:37652) 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 -- 172.21.15.46:0/896301404 learned_addr learned my addr 172.21.15.46:0/896301404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110119d640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004e50 0x7f10fc1e95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110099c640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc1fa240 0x7f10fc141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 -- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc1fa240 msgr2=0x7f10fc141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc1fa240 0x7f10fc141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 -- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004e50 msgr2=0x7f10fc1e95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004e50 0x7f10fc1e95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 -- 172.21.15.46:0/896301404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10fc1418d0 con 0x7f10fc0d1290 2024-05-09T03:14:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110199e640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 0x7f10fc0d16a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f10f4002120 tx=0x7f10f4001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f403e020 con 0x7f10fc0d1290 2024-05-09T03:14:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f4003e10 con 0x7f10fc0d1290 2024-05-09T03:14:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10fc0d1be0 con 0x7f10fc0d1290 2024-05-09T03:14:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.786+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4007d90 con 0x7f10fc0d1290 2024-05-09T03:14:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.787+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10fc1e8ec0 con 0x7f10fc0d1290 2024-05-09T03:14:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.787+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10fc076c70 con 0x7f10fc0d1290 2024-05-09T03:14:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.788+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f10f4041020 con 0x7f10fc0d1290 2024-05-09T03:14:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.789+0000 7f10f1ffb640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10e00762d0 0x7f10e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.789+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f10f40b73c0 con 0x7f10fc0d1290 2024-05-09T03:14:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.789+0000 7f110119d640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10e00762d0 0x7f10e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.790+0000 7f110119d640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10e00762d0 0x7f10e00787e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f10f8001050 tx=0x7f10f80011e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.793+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f10f4048020 con 0x7f10fc0d1290 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.985+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f10fc0b4220 con 0x7f10fc0d1290 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.985+0000 7f10f1ffb640 1 -- 172.21.15.46:0/896301404 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) ==== 84+0+65 (secure 0 0 0) 0x7f10fc0d1be0 con 0x7f10fc0d1290 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:14:49.986 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:14:49.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:14:49.987 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:14:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10e00762d0 msgr2=0x7f10e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10e00762d0 0x7f10e00787e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f10f8001050 tx=0x7f10f80011e0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 msgr2=0x7f10fc0d16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 0x7f10fc0d16a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f10f4002120 tx=0x7f10f4001670 comp rx=0 tx=0).stop 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 shutdown_connections 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10e00762d0 0x7f10e00787e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10fc0d1290 0x7f10fc0d16a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10fc1fa240 0x7f10fc141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 --2- 172.21.15.46:0/896301404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10e8004e50 0x7f10fc1e95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 >> 172.21.15.46:0/896301404 conn(0x7f10fc0b46a0 msgr2=0x7f10fc0d2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.987+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 shutdown_connections 2024-05-09T03:14:49.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:49.988+0000 7f110219f640 1 -- 172.21.15.46:0/896301404 wait complete. 2024-05-09T03:14:50.000 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:14:50.000 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:14:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.118+0000 7fd8cba61640 1 Processor -- start 2024-05-09T03:14:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.118+0000 7fd8cba61640 1 -- start start 2024-05-09T03:14:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8cba61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40b51e0 0x7fd8c40b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8cba61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8c40cf990 0x7fd8c4161fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8cba61640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c40ce800 con 0x7fd8c40b8d80 2024-05-09T03:14:50.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8cba61640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c408dcd0 con 0x7fd8c40cf990 2024-05-09T03:14:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8cba61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c408dfb0 con 0x7fd8c40b51e0 2024-05-09T03:14:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c8ffc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd8c40b8d80 0x7fd8c40cf280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54870/0 (socket says 172.21.15.46:54870) 2024-05-09T03:14:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c8ffc640 1 -- 172.21.15.46:0/4146241096 learned_addr learned my addr 172.21.15.46:0/4146241096 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c9ffe640 1 --2- 172.21.15.46:0/4146241096 >> v2:172.21.15.88:3300/0 conn(0x7fd8c40cf990 0x7fd8c4161fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:50.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c97fd640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40b51e0 0x7fd8c40b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8bbfff640 1 -- 172.21.15.46:0/4146241096 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844655469 0 0) 0x7fd8b4001110 con 0x7fd8c40b8d80 2024-05-09T03:14:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8bbfff640 1 -- 172.21.15.46:0/4146241096 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8b0003520 con 0x7fd8c40b8d80 2024-05-09T03:14:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8bbfff640 1 -- 172.21.15.46:0/4146241096 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 869838276 0 0) 0x7fd8b0003520 con 0x7fd8c40b8d80 2024-05-09T03:14:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c9ffe640 1 -- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40b51e0 msgr2=0x7fd8c40b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c9ffe640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40b51e0 0x7fd8c40b69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:50.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c9ffe640 1 -- 172.21.15.46:0/4146241096 >> v1:172.21.15.46:6789/0 conn(0x7fd8c40b8d80 legacy=0x7fd8c40cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c9ffe640 1 -- 172.21.15.46:0/4146241096 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c40cea80 con 0x7fd8c40cf990 2024-05-09T03:14:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.120+0000 7fd8c97fd640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40b51e0 0x7fd8c40b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:14:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.121+0000 7fd8c9ffe640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40cf990 0x7fd8c4161fd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009d90 tx=0x7fd8bc02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d17177461ad8b38f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.121+0000 7fd8bbfff640 1 -- 172.21.15.46:0/4146241096 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40018c0 con 0x7fd8c40cf990 2024-05-09T03:14:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.121+0000 7fd8bbfff640 1 -- 172.21.15.46:0/4146241096 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8bc007540 con 0x7fd8c40cf990 2024-05-09T03:14:50.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8bbfff640 1 -- 172.21.15.46:0/4146241096 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc007910 con 0x7fd8c40cf990 2024-05-09T03:14:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 -- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40cf990 msgr2=0x7fd8c4161fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40cf990 0x7fd8c4161fd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009d90 tx=0x7fd8bc02a5f0 comp rx=0 tx=0).stop 2024-05-09T03:14:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 -- 172.21.15.46:0/4146241096 shutdown_connections 2024-05-09T03:14:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40cf990 0x7fd8c4161fd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4146241096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40b51e0 0x7fd8c40b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:50.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 -- 172.21.15.46:0/4146241096 >> 172.21.15.46:0/4146241096 conn(0x7fd8c40b5670 msgr2=0x7fd8c40b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 -- 172.21.15.46:0/4146241096 shutdown_connections 2024-05-09T03:14:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.122+0000 7fd8cba61640 1 -- 172.21.15.46:0/4146241096 wait complete. 2024-05-09T03:14:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.124+0000 7fd8cba61640 1 Processor -- start 2024-05-09T03:14:50.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.124+0000 7fd8cba61640 1 -- start start 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.125+0000 7fd8cba61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40b51e0 0x7fd8c4072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.125+0000 7fd8cba61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 0x7fd8c415fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.125+0000 7fd8cba61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40cf990 0x7fd8c415a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.125+0000 7fd8cba61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c40cfda0 con 0x7fd8c40b8d80 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.125+0000 7fd8cba61640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c40b6f10 con 0x7fd8c40b51e0 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.125+0000 7fd8cba61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c408d960 con 0x7fd8c40cf990 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 0x7fd8c415fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 0x7fd8c415fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37666/0 (socket says 172.21.15.46:37666) 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 -- 172.21.15.46:0/4192168196 learned_addr learned my addr 172.21.15.46:0/4192168196 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c97fd640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40b51e0 0x7fd8c4072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c9ffe640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40cf990 0x7fd8c415a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:50.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 -- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40cf990 msgr2=0x7fd8c415a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40cf990 0x7fd8c415a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 -- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40b51e0 msgr2=0x7fd8c4072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40b51e0 0x7fd8c4072bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 -- 172.21.15.46:0/4192168196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c415a6a0 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8c8ffc640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 0x7fd8c415fdb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c4c0 tx=0x7fd8b400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4005ce0 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b4007070 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4007460 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.126+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c40c8810 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.127+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c40c8dd0 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.127+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c406a270 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.128+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd8b4007600 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.129+0000 7fd8b9ffb640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8a80762d0 0x7fd8a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.129+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7fd8b4096e50 con 0x7fd8c40b8d80 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.129+0000 7fd8c97fd640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8a80762d0 0x7fd8a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:50.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.130+0000 7fd8c97fd640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8a80762d0 0x7fd8a80787e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009560 tx=0x7fd8c0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.134+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd8c40ce800 con 0x7fd8c40b8d80 2024-05-09T03:14:50.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:50.326+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd8c40d2c00 con 0x7fd8c40b8d80 2024-05-09T03:14:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.229+0000 7fd8b9ffb640 1 -- 172.21.15.46:0/4192168196 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) ==== 138+0+0 (secure 0 0 0) 0x7fd8b4060410 con 0x7fd8c40b8d80 2024-05-09T03:14:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8a80762d0 msgr2=0x7fd8a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8a80762d0 0x7fd8a80787e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009560 tx=0x7fd8c0009190 comp rx=0 tx=0).stop 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 msgr2=0x7fd8c415fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 0x7fd8c415fdb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c4c0 tx=0x7fd8b400c980 comp rx=0 tx=0).stop 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 shutdown_connections 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd8c40cf990 0x7fd8c415a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8a80762d0 0x7fd8a80787e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8c40b8d80 0x7fd8c415fdb0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 --2- 172.21.15.46:0/4192168196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8c40b51e0 0x7fd8c4072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 >> 172.21.15.46:0/4192168196 conn(0x7fd8c40b5670 msgr2=0x7fd8c40cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 shutdown_connections 2024-05-09T03:14:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.232+0000 7fd8cba61640 1 -- 172.21.15.46:0/4192168196 wait complete. 2024-05-09T03:14:51.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.379+0000 7ff78891f640 1 Processor -- start 2024-05-09T03:14:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.379+0000 7ff78891f640 1 -- start start 2024-05-09T03:14:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78891f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff7840b2eb0 0x7ff7840b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78891f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7840cf9c0 0x7ff7840cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78891f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7840b3470 con 0x7ff7840b3e10 2024-05-09T03:14:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78891f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff7840cf530 con 0x7ff7840b2eb0 2024-05-09T03:14:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78891f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff7840ceee0 con 0x7ff7840cf9c0 2024-05-09T03:14:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff781d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff7840b2eb0 0x7ff7840b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff7840b3e10 0x7ff7840cd0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54886/0 (socket says 172.21.15.46:54886) 2024-05-09T03:14:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff78259c640 1 -- 172.21.15.46:0/2660697619 learned_addr learned my addr 172.21.15.46:0/2660697619 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff782d9d640 1 --2- 172.21.15.46:0/2660697619 >> v2:172.21.15.88:3300/0 conn(0x7ff7840cf9c0 0x7ff7840cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff781d9b640 1 -- 172.21.15.46:0/2660697619 >> v2:172.21.15.88:3300/0 conn(0x7ff7840cf9c0 msgr2=0x7ff7840cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff781d9b640 1 --2- 172.21.15.46:0/2660697619 >> v2:172.21.15.88:3300/0 conn(0x7ff7840cf9c0 0x7ff7840cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff781d9b640 1 -- 172.21.15.46:0/2660697619 >> v1:172.21.15.46:6789/0 conn(0x7ff7840b3e10 legacy=0x7ff7840cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.381+0000 7ff781d9b640 1 -- 172.21.15.46:0/2660697619 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7840cf1c0 con 0x7ff7840b2eb0 2024-05-09T03:14:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.382+0000 7ff781d9b640 1 --2- 172.21.15.46:0/2660697619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff7840b1e80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff784074ce0 tx=0x7ff77802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749585eeb6c3c3eb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.382+0000 7ff78159a640 1 -- 172.21.15.46:0/2660697619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778037070 con 0x7ff7840b2eb0 2024-05-09T03:14:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.382+0000 7ff78159a640 1 -- 172.21.15.46:0/2660697619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff778032440 con 0x7ff7840b2eb0 2024-05-09T03:14:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.382+0000 7ff78159a640 1 -- 172.21.15.46:0/2660697619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff778032810 con 0x7ff7840b2eb0 2024-05-09T03:14:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 -- 172.21.15.46:0/2660697619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 msgr2=0x7ff7840b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 --2- 172.21.15.46:0/2660697619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff7840b1e80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff784074ce0 tx=0x7ff77802a1a0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 -- 172.21.15.46:0/2660697619 shutdown_connections 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 --2- 172.21.15.46:0/2660697619 >> v2:172.21.15.88:3300/0 conn(0x7ff7840cf9c0 0x7ff7840cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 --2- 172.21.15.46:0/2660697619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff7840b1e80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 -- 172.21.15.46:0/2660697619 >> 172.21.15.46:0/2660697619 conn(0x7ff7840b4650 msgr2=0x7ff7840b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 -- 172.21.15.46:0/2660697619 shutdown_connections 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.383+0000 7ff78891f640 1 -- 172.21.15.46:0/2660697619 wait complete. 2024-05-09T03:14:51.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.385+0000 7ff78891f640 1 Processor -- start 2024-05-09T03:14:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.385+0000 7ff78891f640 1 -- start start 2024-05-09T03:14:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.386+0000 7ff78891f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff78415f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.386+0000 7ff78891f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7840b3e10 0x7ff7840c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.386+0000 7ff78891f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7840cf9c0 0x7ff7841eba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.386+0000 7ff78891f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7840d0900 con 0x7ff7840b2eb0 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.386+0000 7ff78891f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7840d0780 con 0x7ff7840cf9c0 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.386+0000 7ff78891f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff7840d1d70 con 0x7ff7840b3e10 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff78415f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff78415f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37686/0 (socket says 172.21.15.46:37686) 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 -- 172.21.15.46:0/717201408 learned_addr learned my addr 172.21.15.46:0/717201408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff782d9d640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7840cf9c0 0x7ff7841eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff781d9b640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7840b3e10 0x7ff7840c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 -- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7840b3e10 msgr2=0x7ff7840c6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7840b3e10 0x7ff7840c6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 -- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7840cf9c0 msgr2=0x7ff7841eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7840cf9c0 0x7ff7841eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 -- 172.21.15.46:0/717201408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7840c7fe0 con 0x7ff7840b2eb0 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff781d9b640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7840b3e10 0x7ff7840c6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:51.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78259c640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff78415f0c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff77400cb10 tx=0x7ff77400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774017070 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77400ac90 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7840c7a20 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774012640 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.387+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7840cae10 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.388+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff78406a270 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.389+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff77401a070 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.390+0000 7ff772ffd640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7640762d0 0x7ff7640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.390+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5227+0+0 (secure 0 0 0) 0x7ff7740970e0 con 0x7ff7840b2eb0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.390+0000 7ff781d9b640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7640762d0 0x7ff7640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.391+0000 7ff781d9b640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7640762d0 0x7ff7640787e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff778034000 tx=0x7ff77802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:51.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.395+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff7740605d0 con 0x7ff7840b2eb0 2024-05-09T03:14:51.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:51.586+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff784160560 con 0x7ff7840b2eb0 2024-05-09T03:14:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.231+0000 7ff772ffd640 1 -- 172.21.15.46:0/717201408 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) ==== 170+0+0 (secure 0 0 0) 0x7ff7740648d0 con 0x7ff7840b2eb0 2024-05-09T03:14:52.232 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:14:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7640762d0 msgr2=0x7ff7640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7640762d0 0x7ff7640787e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff778034000 tx=0x7ff77802e000 comp rx=0 tx=0).stop 2024-05-09T03:14:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 msgr2=0x7ff78415f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff78415f0c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff77400cb10 tx=0x7ff77400a060 comp rx=0 tx=0).stop 2024-05-09T03:14:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 shutdown_connections 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7840cf9c0 0x7ff7841eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7840b3e10 0x7ff7840c6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7640762d0 0x7ff7640787e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 --2- 172.21.15.46:0/717201408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7840b2eb0 0x7ff78415f0c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 >> 172.21.15.46:0/717201408 conn(0x7ff7840b4650 msgr2=0x7ff78406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 shutdown_connections 2024-05-09T03:14:52.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.235+0000 7ff78891f640 1 -- 172.21.15.46:0/717201408 wait complete. 2024-05-09T03:14:52.263 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:14:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.386+0000 7f0226270640 1 Processor -- start 2024-05-09T03:14:52.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.386+0000 7f0226270640 1 -- start start 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.387+0000 7f0226270640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.387+0000 7f0226270640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02200b80b0 con 0x7f02200b7ca0 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.387+0000 7f0226270640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02200b5550 con 0x7f02200d3fc0 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.387+0000 7f0226270640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02200d3480 con 0x7f02200b5140 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.387+0000 7f021f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02200b7ca0 0x7f02200d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54892/0 (socket says 172.21.15.46:54892) 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021f7fe640 1 -- 172.21.15.46:0/4275393284 learned_addr learned my addr 172.21.15.46:0/4275393284 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021ffff640 1 --2- 172.21.15.46:0/4275393284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2986312521 0 0) 0x7f0214001110 con 0x7f02200b7ca0 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0208003520 con 0x7f02200b7ca0 2024-05-09T03:14:52.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2720408366 0 0) 0x7f02200b5550 con 0x7f02200d3fc0 2024-05-09T03:14:52.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0214001110 con 0x7f02200d3fc0 2024-05-09T03:14:52.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4091652932 0 0) 0x7f0208003520 con 0x7f02200b7ca0 2024-05-09T03:14:52.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021ffff640 1 -- 172.21.15.46:0/4275393284 >> v1:172.21.15.88:6789/0 conn(0x7f02200d3fc0 legacy=0x7f02200b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021ffff640 1 -- 172.21.15.46:0/4275393284 >> v1:172.21.15.46:6789/0 conn(0x7f02200b7ca0 legacy=0x7f02200d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.388+0000 7f021ffff640 1 -- 172.21.15.46:0/4275393284 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02200d44e0 con 0x7f02200b5140 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.389+0000 7f021ffff640 1 --2- 172.21.15.46:0/4275393284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200b7760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02100097d0 tx=0x7f021002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=600601fa54c12b01 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.389+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210037070 con 0x7f02200b5140 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.389+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f021002acc0 con 0x7f02200b5140 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.389+0000 7f021effd640 1 -- 172.21.15.46:0/4275393284 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02140018c0 con 0x7f02200b5140 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 -- 172.21.15.46:0/4275393284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 msgr2=0x7f02200b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 --2- 172.21.15.46:0/4275393284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200b7760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f02100097d0 tx=0x7f021002a230 comp rx=0 tx=0).stop 2024-05-09T03:14:52.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 -- 172.21.15.46:0/4275393284 shutdown_connections 2024-05-09T03:14:52.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 --2- 172.21.15.46:0/4275393284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200b7760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 -- 172.21.15.46:0/4275393284 >> 172.21.15.46:0/4275393284 conn(0x7f022008d3f0 msgr2=0x7f022008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:52.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 -- 172.21.15.46:0/4275393284 shutdown_connections 2024-05-09T03:14:52.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.390+0000 7f0226270640 1 -- 172.21.15.46:0/4275393284 wait complete. 2024-05-09T03:14:52.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.392+0000 7f0226270640 1 Processor -- start 2024-05-09T03:14:52.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.392+0000 7f0226270640 1 -- start start 2024-05-09T03:14:52.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f0226270640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f0226270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 0x7f022014bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f0226270640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02200d3fc0 0x7f02201ebc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f0226270640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02200d4d80 con 0x7f02200b7ca0 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f0226270640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f02200d4aa0 con 0x7f02200d3fc0 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f0226270640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02200d5580 con 0x7f02200b5140 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f021f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 0x7f022014bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f021f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 0x7f022014bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37702/0 (socket says 172.21.15.46:37702) 2024-05-09T03:14:52.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.393+0000 7f021f7fe640 1 -- 172.21.15.46:0/3388866518 learned_addr learned my addr 172.21.15.46:0/3388866518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f022480d640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02200d3fc0 0x7f02201ebc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021ffff640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021f7fe640 1 -- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 msgr2=0x7f02200ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021f7fe640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021f7fe640 1 -- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02200d3fc0 msgr2=0x7f02201ebc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021f7fe640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02200d3fc0 0x7f02201ebc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021f7fe640 1 -- 172.21.15.46:0/3388866518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022014c6c0 con 0x7f02200b7ca0 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021ffff640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:52.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021f7fe640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 0x7f022014bc00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f021400c700 tx=0x7f021400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214017070 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0214005ad0 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022014c9a0 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0214007070 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.394+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f022014cf60 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.395+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f022006a270 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.396+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f021401a070 con 0x7f02200b7ca0 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.396+0000 7f021cff9640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f020c0762d0 0x7f020c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:52.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.397+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5609+0+0 (secure 0 0 0) 0x7f0214097340 con 0x7f02200b7ca0 2024-05-09T03:14:52.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.397+0000 7f021ffff640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f020c0762d0 0x7f020c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:52.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.397+0000 7f021ffff640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f020c0762d0 0x7f020c0787e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0210034000 tx=0x7f021002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:52.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.401+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f02200b80b0 con 0x7f02200b7ca0 2024-05-09T03:14:52.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:52.630+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f02201604e0 con 0x7f02200b7ca0 2024-05-09T03:14:53.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.262+0000 7f021cff9640 1 -- 172.21.15.46:0/3388866518 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v65) ==== 174+0+0 (secure 0 0 0) 0x7f0214060580 con 0x7f02200b7ca0 2024-05-09T03:14:53.263 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 10 and data pool 11 2024-05-09T03:14:53.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.268+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f020c0762d0 msgr2=0x7f020c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.268+0000 7f0226270640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f020c0762d0 0x7f020c0787e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0210034000 tx=0x7f021002e000 comp rx=0 tx=0).stop 2024-05-09T03:14:53.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.268+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 msgr2=0x7f022014bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.268+0000 7f0226270640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 0x7f022014bc00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f021400c700 tx=0x7f021400cbc0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.272+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 shutdown_connections 2024-05-09T03:14:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.272+0000 7f0226270640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02200d3fc0 0x7f02201ebc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.272+0000 7f0226270640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f020c0762d0 0x7f020c0787e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.272+0000 7f0226270640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02200b7ca0 0x7f022014bc00 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.272+0000 7f0226270640 1 --2- 172.21.15.46:0/3388866518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02200b5140 0x7f02200ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.272+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 >> 172.21.15.46:0/3388866518 conn(0x7f022008d3f0 msgr2=0x7f02200d38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.274+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 shutdown_connections 2024-05-09T03:14:53.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.278+0000 7f0226270640 1 -- 172.21.15.46:0/3388866518 wait complete. 2024-05-09T03:14:53.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:53.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.416+0000 7f36f6461640 1 Processor -- start 2024-05-09T03:14:53.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.416+0000 7f36f6461640 1 -- start start 2024-05-09T03:14:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f6461640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f36f00b8d10 0x7f36f00b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f6461640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36f008cfc0 0x7f36f008d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f6461640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f008d910 con 0x7f36f00b0a00 2024-05-09T03:14:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f6461640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f36f00d2270 con 0x7f36f008cfc0 2024-05-09T03:14:53.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f6461640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f36f00d2510 con 0x7f36f00b8d10 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36f008cfc0 0x7f36f008d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f49fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36f00b0a00 0x7f36f00b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54898/0 (socket says 172.21.15.46:54898) 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36f008cfc0 0x7f36f008d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37708/0 (socket says 172.21.15.46:37708) 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36f49fe640 1 -- 172.21.15.46:0/1378117938 learned_addr learned my addr 172.21.15.46:0/1378117938 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36effff640 1 --2- 172.21.15.46:0/1378117938 >> v2:172.21.15.88:3300/0 conn(0x7f36f00b8d10 0x7f36f00b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36e7fff640 1 -- 172.21.15.46:0/1378117938 >> v2:172.21.15.88:3300/0 conn(0x7f36f00b8d10 msgr2=0x7f36f00b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36e7fff640 1 --2- 172.21.15.46:0/1378117938 >> v2:172.21.15.88:3300/0 conn(0x7f36f00b8d10 0x7f36f00b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36e7fff640 1 -- 172.21.15.46:0/1378117938 >> v1:172.21.15.46:6789/0 conn(0x7f36f00b0a00 legacy=0x7f36f00b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36ef7fe640 1 -- 172.21.15.46:0/1378117938 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553567066 0 0) 0x7f36e8001490 con 0x7f36f00b0a00 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.418+0000 7f36e7fff640 1 -- 172.21.15.46:0/1378117938 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f00d27f0 con 0x7f36f008cfc0 2024-05-09T03:14:53.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.419+0000 7f36e7fff640 1 --2- 172.21.15.46:0/1378117938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f008d3d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f36f00b64c0 tx=0x7f36d802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b52543ec6e9575c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.419+0000 7f36ef7fe640 1 -- 172.21.15.46:0/1378117938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8037070 con 0x7f36f008cfc0 2024-05-09T03:14:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.419+0000 7f36ef7fe640 1 -- 172.21.15.46:0/1378117938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d8032440 con 0x7f36f008cfc0 2024-05-09T03:14:53.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.419+0000 7f36ef7fe640 1 -- 172.21.15.46:0/1378117938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8032810 con 0x7f36f008cfc0 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 -- 172.21.15.46:0/1378117938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 msgr2=0x7f36f008d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 --2- 172.21.15.46:0/1378117938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f008d3d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f36f00b64c0 tx=0x7f36d802a1a0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 -- 172.21.15.46:0/1378117938 shutdown_connections 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 --2- 172.21.15.46:0/1378117938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f008d3d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 --2- 172.21.15.46:0/1378117938 >> v2:172.21.15.88:3300/0 conn(0x7f36f00b8d10 0x7f36f00b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 -- 172.21.15.46:0/1378117938 >> 172.21.15.46:0/1378117938 conn(0x7f36f00b56e0 msgr2=0x7f36f00b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 -- 172.21.15.46:0/1378117938 shutdown_connections 2024-05-09T03:14:53.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.420+0000 7f36f6461640 1 -- 172.21.15.46:0/1378117938 wait complete. 2024-05-09T03:14:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.422+0000 7f36f6461640 1 Processor -- start 2024-05-09T03:14:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.422+0000 7f36f6461640 1 -- start start 2024-05-09T03:14:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36f6461640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f01f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36f6461640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36f00b0a00 0x7f36f00c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36f6461640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f00b8d10 0x7f36f00c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36f6461640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f36f00b9380 con 0x7f36f008cfc0 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36f6461640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f36f00cfee0 con 0x7f36f00b8d10 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36f6461640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f36f00b9500 con 0x7f36f00b0a00 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f01f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f01f3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37724/0 (socket says 172.21.15.46:37724) 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.423+0000 7f36effff640 1 -- 172.21.15.46:0/116037533 learned_addr learned my addr 172.21.15.46:0/116037533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36f49fe640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f00b8d10 0x7f36f00c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36e7fff640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36f00b0a00 0x7f36f00c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36effff640 1 -- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36f00b0a00 msgr2=0x7f36f00c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36effff640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36f00b0a00 0x7f36f00c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36effff640 1 -- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f00b8d10 msgr2=0x7f36f00c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36effff640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f00b8d10 0x7f36f00c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36effff640 1 -- 172.21.15.46:0/116037533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f00c8410 con 0x7f36f008cfc0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36effff640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f01f3fd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f36e000d800 tx=0x7f36e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0005990 con 0x7f36f008cfc0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e0005b30 con 0x7f36f008cfc0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0005d20 con 0x7f36f008cfc0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f00c86f0 con 0x7f36f008cfc0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.424+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f00c8cb0 con 0x7f36f008cfc0 2024-05-09T03:14:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.425+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f00b9120 con 0x7f36f008cfc0 2024-05-09T03:14:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.426+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f36e001b020 con 0x7f36f008cfc0 2024-05-09T03:14:53.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.427+0000 7f36ed7fa640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36d0076300 0x7f36d0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.427+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f36e0097350 con 0x7f36f008cfc0 2024-05-09T03:14:53.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.427+0000 7f36e7fff640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36d0076300 0x7f36d0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.427+0000 7f36e7fff640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36d0076300 0x7f36d0078810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f36d8034000 tx=0x7f36d802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:53.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.432+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f36f00b9120 con 0x7f36f008cfc0 2024-05-09T03:14:53.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.625+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36f006a270 con 0x7f36f008cfc0 2024-05-09T03:14:53.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.626+0000 7f36ed7fa640 1 -- 172.21.15.46:0/116037533 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16609 (secure 0 0 0) 0x7f36e00605c0 con 0x7f36f008cfc0 2024-05-09T03:14:53.627 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:53.627 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":67,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:14:52.631163+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:14:50.331114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-09T03:14:51.590716+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:53.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36d0076300 msgr2=0x7f36d0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36d0076300 0x7f36d0078810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f36d8034000 tx=0x7f36d802e000 comp rx=0 tx=0).stop 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 msgr2=0x7f36f01f3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f01f3fd0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f36e000d800 tx=0x7f36e000dcc0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 shutdown_connections 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f00b8d10 0x7f36f00c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f36f00b0a00 0x7f36f00c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f36d0076300 0x7f36d0078810 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 --2- 172.21.15.46:0/116037533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f008cfc0 0x7f36f01f3fd0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 >> 172.21.15.46:0/116037533 conn(0x7f36f00b56e0 msgr2=0x7f36f00d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 shutdown_connections 2024-05-09T03:14:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.629+0000 7f36f6461640 1 -- 172.21.15.46:0/116037533 wait complete. 2024-05-09T03:14:53.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.755+0000 7f4860340640 1 Processor -- start 2024-05-09T03:14:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.755+0000 7f4860340640 1 -- start start 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f4860340640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580d57a0 0x7f4858160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f4860340640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48580b7ef0 0x7f48580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f4860340640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48580d49e0 con 0x7f48580b4af0 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f4860340640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f48580d5bb0 con 0x7f48580b7ef0 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f4860340640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48580d4cc0 con 0x7f48580d57a0 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48580b7ef0 0x7f48580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48580b7ef0 0x7f48580b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37736/0 (socket says 172.21.15.46:37736) 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 -- 172.21.15.46:0/2148090797 learned_addr learned my addr 172.21.15.46:0/2148090797 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e0dc640 1 --2- 172.21.15.46:0/2148090797 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580d57a0 0x7f4858160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 -- 172.21.15.46:0/2148090797 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580d57a0 msgr2=0x7f4858160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 --2- 172.21.15.46:0/2148090797 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580d57a0 0x7f4858160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 -- 172.21.15.46:0/2148090797 >> v1:172.21.15.88:6789/0 conn(0x7f48580b4af0 legacy=0x7f48580b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.757+0000 7f485e8dd640 1 -- 172.21.15.46:0/2148090797 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48580d4fa0 con 0x7f48580b7ef0 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.758+0000 7f485e8dd640 1 --2- 172.21.15.46:0/2148090797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580b7ef0 0x7f48580b4f00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4850009870 tx=0x7f485002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7f7dcd983c9293 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.758+0000 7f485d0da640 1 -- 172.21.15.46:0/2148090797 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4850037070 con 0x7f48580b7ef0 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.758+0000 7f485d0da640 1 -- 172.21.15.46:0/2148090797 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4850032440 con 0x7f48580b7ef0 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.758+0000 7f485d0da640 1 -- 172.21.15.46:0/2148090797 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4850032810 con 0x7f48580b7ef0 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 -- 172.21.15.46:0/2148090797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580b7ef0 msgr2=0x7f48580b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 --2- 172.21.15.46:0/2148090797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580b7ef0 0x7f48580b4f00 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4850009870 tx=0x7f485002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 -- 172.21.15.46:0/2148090797 shutdown_connections 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 --2- 172.21.15.46:0/2148090797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580b7ef0 0x7f48580b4f00 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 --2- 172.21.15.46:0/2148090797 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580d57a0 0x7f4858160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 -- 172.21.15.46:0/2148090797 >> 172.21.15.46:0/2148090797 conn(0x7f48580743e0 msgr2=0x7f48580d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 -- 172.21.15.46:0/2148090797 shutdown_connections 2024-05-09T03:14:53.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.759+0000 7f4860340640 1 -- 172.21.15.46:0/2148090797 wait complete. 2024-05-09T03:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.761+0000 7f4860340640 1 Processor -- start 2024-05-09T03:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.761+0000 7f4860340640 1 -- start start 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f4860340640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580b4af0 0x7f4858153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f4860340640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 0x7f48580cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f4860340640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48580d03b0 0x7f48580d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f4860340640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48580d1920 con 0x7f48580d57a0 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f4860340640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f48580d15b0 con 0x7f48580d03b0 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f4860340640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48580d0d00 con 0x7f48580b4af0 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f485d8db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 0x7f48580cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f485d8db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 0x7f48580cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37742/0 (socket says 172.21.15.46:37742) 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f485d8db640 1 -- 172.21.15.46:0/3231006719 learned_addr learned my addr 172.21.15.46:0/3231006719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f485e0dc640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580b4af0 0x7f4858153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.762+0000 7f485e8dd640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48580d03b0 0x7f48580d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f485d8db640 1 -- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580b4af0 msgr2=0x7f4858153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f485d8db640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580b4af0 0x7f4858153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f485d8db640 1 -- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48580d03b0 msgr2=0x7f48580d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f485d8db640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48580d03b0 0x7f48580d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f485d8db640 1 -- 172.21.15.46:0/3231006719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48580acb30 con 0x7f48580d57a0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f485d8db640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 0x7f48580cfe70 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f484800b420 tx=0x7f484800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848011830 con 0x7f48580d57a0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4848016070 con 0x7f48580d57a0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48580cf4e0 con 0x7f48580d57a0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848011c10 con 0x7f48580d57a0 2024-05-09T03:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.763+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48580cfaa0 con 0x7f48580d57a0 2024-05-09T03:14:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.764+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48580ac8a0 con 0x7f48580d57a0 2024-05-09T03:14:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.765+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f484801b460 con 0x7f48580d57a0 2024-05-09T03:14:53.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.765+0000 7f484effd640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f483c076300 0x7f483c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.765+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f4848096f20 con 0x7f48580d57a0 2024-05-09T03:14:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.765+0000 7f485e0dc640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f483c076300 0x7f483c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:53.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.766+0000 7f485e0dc640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f483c076300 0x7f483c078810 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f485806d3b0 tx=0x7f4854007730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:53.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.770+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4848020020 con 0x7f48580d57a0 2024-05-09T03:14:53.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.963+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f485806a270 con 0x7f48580d57a0 2024-05-09T03:14:53.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.964+0000 7f484effd640 1 -- 172.21.15.46:0/3231006719 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16609 (secure 0 0 0) 0x7f4848060230 con 0x7f48580d57a0 2024-05-09T03:14:53.965 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:53.965 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":67,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:14:52.631163+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:14:50.331114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-09T03:14:51.590716+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:53.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f483c076300 msgr2=0x7f483c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f483c076300 0x7f483c078810 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f485806d3b0 tx=0x7f4854007730 comp rx=0 tx=0).stop 2024-05-09T03:14:53.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 msgr2=0x7f48580cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:53.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 0x7f48580cfe70 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f484800b420 tx=0x7f484800b8e0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 shutdown_connections 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48580d03b0 0x7f48580d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f483c076300 0x7f483c078810 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48580d57a0 0x7f48580cfe70 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 --2- 172.21.15.46:0/3231006719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48580b4af0 0x7f4858153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 >> 172.21.15.46:0/3231006719 conn(0x7f48580743e0 msgr2=0x7f48580b72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 shutdown_connections 2024-05-09T03:14:53.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:53.966+0000 7f4860340640 1 -- 172.21.15.46:0/3231006719 wait complete. 2024-05-09T03:14:53.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:14:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.097+0000 7faa554d9640 1 Processor -- start 2024-05-09T03:14:54.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.097+0000 7faa554d9640 1 -- start start 2024-05-09T03:14:54.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa554d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b7eb0 0x7faa500b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa554d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa500b1da0 0x7faa5006e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa554d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa501617c0 con 0x7faa5015d310 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa554d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faa500b2480 con 0x7faa500b1da0 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa554d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faa500b2760 con 0x7faa500b7eb0 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa4effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faa5015d310 0x7faa500b77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54900/0 (socket says 172.21.15.46:54900) 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.098+0000 7faa4effd640 1 -- 172.21.15.46:0/254796386 learned_addr learned my addr 172.21.15.46:0/254796386 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4f7fe640 1 --2- 172.21.15.46:0/254796386 >> v2:172.21.15.88:3300/0 conn(0x7faa500b1da0 0x7faa5006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa47fff640 1 --2- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b7eb0 0x7faa500b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4e7fc640 1 -- 172.21.15.46:0/254796386 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389260244 0 0) 0x7faa501617c0 con 0x7faa5015d310 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4e7fc640 1 -- 172.21.15.46:0/254796386 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa3c003520 con 0x7faa5015d310 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4f7fe640 1 -- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b7eb0 msgr2=0x7faa500b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4f7fe640 1 --2- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b7eb0 0x7faa500b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:54.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4f7fe640 1 -- 172.21.15.46:0/254796386 >> v1:172.21.15.46:6789/0 conn(0x7faa5015d310 legacy=0x7faa500b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:54.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.099+0000 7faa4f7fe640 1 -- 172.21.15.46:0/254796386 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa500b2a40 con 0x7faa500b1da0 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.100+0000 7faa4f7fe640 1 --2- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa500b1da0 0x7faa5006e300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faa48009d90 tx=0x7faa4802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d1c3c733d1bf6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.100+0000 7faa4e7fc640 1 -- 172.21.15.46:0/254796386 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48031600 con 0x7faa500b1da0 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.100+0000 7faa4e7fc640 1 -- 172.21.15.46:0/254796386 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa480317a0 con 0x7faa500b1da0 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.100+0000 7faa4e7fc640 1 -- 172.21.15.46:0/254796386 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48031b50 con 0x7faa500b1da0 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 -- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa500b1da0 msgr2=0x7faa5006e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 --2- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa500b1da0 0x7faa5006e300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faa48009d90 tx=0x7faa4802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 -- 172.21.15.46:0/254796386 shutdown_connections 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 --2- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa500b1da0 0x7faa5006e300 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 --2- 172.21.15.46:0/254796386 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b7eb0 0x7faa500b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 -- 172.21.15.46:0/254796386 >> 172.21.15.46:0/254796386 conn(0x7faa5009b7d0 msgr2=0x7faa5009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 -- 172.21.15.46:0/254796386 shutdown_connections 2024-05-09T03:14:54.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.101+0000 7faa554d9640 1 -- 172.21.15.46:0/254796386 wait complete. 2024-05-09T03:14:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.103+0000 7faa554d9640 1 Processor -- start 2024-05-09T03:14:54.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.103+0000 7faa554d9640 1 -- start start 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa554d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b1da0 0x7faa500cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa554d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa5015d310 0x7faa5015d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa554d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 0x7faa50157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa554d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa500cfe00 con 0x7faa5015f050 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa554d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa500cfc80 con 0x7faa5015d310 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa554d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faa5006e840 con 0x7faa500b1da0 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 0x7faa50157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 0x7faa50157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37748/0 (socket says 172.21.15.46:37748) 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 -- 172.21.15.46:0/3397436676 learned_addr learned my addr 172.21.15.46:0/3397436676 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa47fff640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa5015d310 0x7faa5015d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4effd640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b1da0 0x7faa500cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 -- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b1da0 msgr2=0x7faa500cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b1da0 0x7faa500cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 -- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa5015d310 msgr2=0x7faa5015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa5015d310 0x7faa5015d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa4f7fe640 1 -- 172.21.15.46:0/3397436676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5015dfb0 con 0x7faa5015f050 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.104+0000 7faa47fff640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa5015d310 0x7faa5015d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.105+0000 7faa4f7fe640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 0x7faa50157fc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7faa4802abc0 tx=0x7faa48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.105+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48009450 con 0x7faa5015f050 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.105+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa48031b50 con 0x7faa5015f050 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.105+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa5015e290 con 0x7faa5015f050 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.105+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa48003e10 con 0x7faa5015f050 2024-05-09T03:14:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.105+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa5015e850 con 0x7faa5015f050 2024-05-09T03:14:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.106+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa5006a270 con 0x7faa5015f050 2024-05-09T03:14:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.107+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faa48033070 con 0x7faa5015f050 2024-05-09T03:14:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.107+0000 7faa477fe640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa300762d0 0x7faa300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.107+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7faa480b82d0 con 0x7faa5015f050 2024-05-09T03:14:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.107+0000 7faa4effd640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa300762d0 0x7faa300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:54.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.108+0000 7faa4effd640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa300762d0 0x7faa300787e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faa400013e0 tx=0x7faa40001370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:54.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.112+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faa4803f030 con 0x7faa5015f050 2024-05-09T03:14:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:54.345+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7faa500cf8b0 con 0x7faa5015f050 2024-05-09T03:14:55.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-05-09T03:14:55.073 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-05-09T03:14:55.227 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:14:55.226+0000 7f468a871640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:14:55.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.276+0000 7faa477fe640 1 -- 172.21.15.46:0/3397436676 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v68) ==== 118+0+0 (secure 0 0 0) 0x7faa48081610 con 0x7faa5015f050 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa300762d0 msgr2=0x7faa300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa300762d0 0x7faa300787e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faa400013e0 tx=0x7faa40001370 comp rx=0 tx=0).stop 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 msgr2=0x7faa50157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 0x7faa50157fc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7faa4802abc0 tx=0x7faa48004a40 comp rx=0 tx=0).stop 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 shutdown_connections 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa300762d0 0x7faa300787e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa5015f050 0x7faa50157fc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa5015d310 0x7faa5015d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 --2- 172.21.15.46:0/3397436676 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa500b1da0 0x7faa500cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 >> 172.21.15.46:0/3397436676 conn(0x7faa5009b7d0 msgr2=0x7faa50158b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:55.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 shutdown_connections 2024-05-09T03:14:55.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.279+0000 7faa554d9640 1 -- 172.21.15.46:0/3397436676 wait complete. 2024-05-09T03:14:55.294 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:14:55.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:14:55.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.418+0000 7f349b0e2640 1 Processor -- start 2024-05-09T03:14:55.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.418+0000 7f349b0e2640 1 -- start start 2024-05-09T03:14:55.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f349b0e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3494163eb0 0x7f34941642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:55.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f349b0e2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3494164900 con 0x7f34940bb770 2024-05-09T03:14:55.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f349b0e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3494164be0 con 0x7f34940d2a70 2024-05-09T03:14:55.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f349b0e2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f34940d4bb0 con 0x7f3494163eb0 2024-05-09T03:14:55.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f348bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3494163eb0 0x7f34941642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f3498e7e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34940bb770 0x7f34940d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54908/0 (socket says 172.21.15.46:54908) 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.419+0000 7f3498e7e640 1 -- 172.21.15.46:0/2711633212 learned_addr learned my addr 172.21.15.46:0/2711633212 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.420+0000 7f348bfff640 1 -- 172.21.15.46:0/2711633212 >> v1:172.21.15.88:6789/0 conn(0x7f34940d2a70 legacy=0x7f34940d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.420+0000 7f348bfff640 1 -- 172.21.15.46:0/2711633212 >> v1:172.21.15.46:6789/0 conn(0x7f34940bb770 legacy=0x7f34940d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.420+0000 7f348b7fe640 1 -- 172.21.15.46:0/2711633212 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065376971 0 0) 0x7f3494164900 con 0x7f34940bb770 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.420+0000 7f348bfff640 1 -- 172.21.15.46:0/2711633212 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34940d4840 con 0x7f3494163eb0 2024-05-09T03:14:55.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.421+0000 7f348bfff640 1 --2- 172.21.15.46:0/2711633212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34941642c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f34840097a0 tx=0x7f348402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208e53d8be0caac6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:55.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.421+0000 7f348b7fe640 1 -- 172.21.15.46:0/2711633212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484037070 con 0x7f3494163eb0 2024-05-09T03:14:55.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.421+0000 7f348b7fe640 1 -- 172.21.15.46:0/2711633212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3484032440 con 0x7f3494163eb0 2024-05-09T03:14:55.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.421+0000 7f348b7fe640 1 -- 172.21.15.46:0/2711633212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484032810 con 0x7f3494163eb0 2024-05-09T03:14:55.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 -- 172.21.15.46:0/2711633212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 msgr2=0x7f34941642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 --2- 172.21.15.46:0/2711633212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34941642c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f34840097a0 tx=0x7f348402a230 comp rx=0 tx=0).stop 2024-05-09T03:14:55.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 -- 172.21.15.46:0/2711633212 shutdown_connections 2024-05-09T03:14:55.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 --2- 172.21.15.46:0/2711633212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34941642c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 -- 172.21.15.46:0/2711633212 >> 172.21.15.46:0/2711633212 conn(0x7f349406e370 msgr2=0x7f349406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:55.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 -- 172.21.15.46:0/2711633212 shutdown_connections 2024-05-09T03:14:55.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.422+0000 7f349b0e2640 1 -- 172.21.15.46:0/2711633212 wait complete. 2024-05-09T03:14:55.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.423+0000 7f349b0e2640 1 Processor -- start 2024-05-09T03:14:55.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.423+0000 7f349b0e2640 1 -- start start 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349b0e2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34940bb770 0x7f3494155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349b0e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34940d2a70 0x7f34940cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349b0e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34940ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349b0e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34940d0d40 con 0x7f3494163eb0 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349b0e2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f34940d0bc0 con 0x7f34940d2a70 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349b0e2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34940b9210 con 0x7f34940bb770 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34940ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34940ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37770/0 (socket says 172.21.15.46:37770) 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 -- 172.21.15.46:0/1108862422 learned_addr learned my addr 172.21.15.46:0/1108862422 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f348bfff640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34940d2a70 0x7f34940cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f3498e7e640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34940bb770 0x7f3494155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 -- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34940bb770 msgr2=0x7f3494155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34940bb770 0x7f3494155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 -- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34940d2a70 msgr2=0x7f34940cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34940d2a70 0x7f34940cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.425+0000 7f349967f640 1 -- 172.21.15.46:0/1108862422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349415f4e0 con 0x7f3494163eb0 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.426+0000 7f349967f640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34940ced10 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f348c002230 tx=0x7f348c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.426+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c003b70 con 0x7f3494163eb0 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.426+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348c003d10 con 0x7f3494163eb0 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.426+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34940cbc00 con 0x7f3494163eb0 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.426+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c0052d0 con 0x7f3494163eb0 2024-05-09T03:14:55.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.426+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34940cc690 con 0x7f3494163eb0 2024-05-09T03:14:55.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.427+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f349406a270 con 0x7f3494163eb0 2024-05-09T03:14:55.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.427+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f348c005470 con 0x7f3494163eb0 2024-05-09T03:14:55.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.428+0000 7f34897fa640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3480076300 0x7f3480078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:55.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.428+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f348c097350 con 0x7f3494163eb0 2024-05-09T03:14:55.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.428+0000 7f3498e7e640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3480076300 0x7f3480078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:55.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.429+0000 7f3498e7e640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3480076300 0x7f3480078810 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3490000f60 tx=0x7f3490007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:55.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.433+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f349415f4e0 con 0x7f3494163eb0 2024-05-09T03:14:55.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:55.668+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f34940d4130 con 0x7f3494163eb0 2024-05-09T03:14:56.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.277+0000 7f34897fa640 1 -- 172.21.15.46:0/1108862422 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v69) ==== 122+0+0 (secure 0 0 0) 0x7f348c0605d0 con 0x7f3494163eb0 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3480076300 msgr2=0x7f3480078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3480076300 0x7f3480078810 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3490000f60 tx=0x7f3490007e40 comp rx=0 tx=0).stop 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 msgr2=0x7f34940ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34940ced10 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f348c002230 tx=0x7f348c00cb90 comp rx=0 tx=0).stop 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 shutdown_connections 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3480076300 0x7f3480078810 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3494163eb0 0x7f34940ced10 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34940d2a70 0x7f34940cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 --2- 172.21.15.46:0/1108862422 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34940bb770 0x7f3494155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 >> 172.21.15.46:0/1108862422 conn(0x7f349406e370 msgr2=0x7f34940b77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 shutdown_connections 2024-05-09T03:14:56.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.280+0000 7f349b0e2640 1 -- 172.21.15.46:0/1108862422 wait complete. 2024-05-09T03:14:56.294 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:14:56.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:56.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.414+0000 7f7657947640 1 Processor -- start 2024-05-09T03:14:56.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.414+0000 7f7657947640 1 -- start start 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f7657947640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76500b92e0 0x7f76500b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f7657947640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f76500d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f7657947640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76500d0dd0 con 0x7f76500ce6a0 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f7657947640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f76500d2b80 con 0x7f76500b92e0 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f7657947640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f76500d2e60 con 0x7f76500d0480 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76500b92e0 0x7f76500b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76500b92e0 0x7f76500b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41042/0 (socket says 172.21.15.46:41042) 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 -- 172.21.15.46:0/2101564289 learned_addr learned my addr 172.21.15.46:0/2101564289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f7654ee2640 1 --2- 172.21.15.46:0/2101564289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f76500d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 -- 172.21.15.46:0/2101564289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 msgr2=0x7f76500d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 --2- 172.21.15.46:0/2101564289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f76500d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 -- 172.21.15.46:0/2101564289 >> v1:172.21.15.88:6789/0 conn(0x7f76500ce6a0 legacy=0x7f76500ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.416+0000 7f76556e3640 1 -- 172.21.15.46:0/2101564289 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76500d3140 con 0x7f76500b92e0 2024-05-09T03:14:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.417+0000 7f76556e3640 1 --2- 172.21.15.46:0/2101564289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500b92e0 0x7f76500b51a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f764c009e50 tx=0x7f764c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6ebb67edf563 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.417+0000 7f7647fff640 1 -- 172.21.15.46:0/2101564289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c036070 con 0x7f76500b92e0 2024-05-09T03:14:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.417+0000 7f7647fff640 1 -- 172.21.15.46:0/2101564289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f764c031600 con 0x7f76500b92e0 2024-05-09T03:14:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.417+0000 7f7647fff640 1 -- 172.21.15.46:0/2101564289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c0319d0 con 0x7f76500b92e0 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 -- 172.21.15.46:0/2101564289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500b92e0 msgr2=0x7f76500b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 --2- 172.21.15.46:0/2101564289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500b92e0 0x7f76500b51a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f764c009e50 tx=0x7f764c02a680 comp rx=0 tx=0).stop 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 -- 172.21.15.46:0/2101564289 shutdown_connections 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 --2- 172.21.15.46:0/2101564289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f76500d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 --2- 172.21.15.46:0/2101564289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500b92e0 0x7f76500b51a0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 -- 172.21.15.46:0/2101564289 >> 172.21.15.46:0/2101564289 conn(0x7f76500d3c00 msgr2=0x7f76500d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 -- 172.21.15.46:0/2101564289 shutdown_connections 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.418+0000 7f7657947640 1 -- 172.21.15.46:0/2101564289 wait complete. 2024-05-09T03:14:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.420+0000 7f7657947640 1 Processor -- start 2024-05-09T03:14:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.420+0000 7f7657947640 1 -- start start 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7657947640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76500b92e0 0x7f7650159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7657947640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 0x7f765014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7657947640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f765014b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7657947640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f765006e920 con 0x7f76500ce6a0 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7657947640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f765006e7a0 con 0x7f76500b92e0 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7657947640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f765006eaa0 con 0x7f76500d0480 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7654ee2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 0x7f765014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7654ee2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 0x7f765014b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41054/0 (socket says 172.21.15.46:41054) 2024-05-09T03:14:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7654ee2640 1 -- 172.21.15.46:0/3085085654 learned_addr learned my addr 172.21.15.46:0/3085085654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f76556e3640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76500b92e0 0x7f7650159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.421+0000 7f7655ee4640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f765014b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7654ee2640 1 -- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 msgr2=0x7f765014b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7654ee2640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f765014b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7654ee2640 1 -- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76500b92e0 msgr2=0x7f7650159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7654ee2640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76500b92e0 0x7f7650159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7654ee2640 1 -- 172.21.15.46:0/3085085654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765015cbe0 con 0x7f76500ce6a0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7654ee2640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 0x7f765014b010 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f764000e8a0 tx=0x7f764000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640017070 con 0x7f76500ce6a0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f764000ee90 con 0x7f76500ce6a0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764000cb00 con 0x7f76500ce6a0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765015cec0 con 0x7f76500ce6a0 2024-05-09T03:14:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.422+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765015d480 con 0x7f76500ce6a0 2024-05-09T03:14:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.423+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f765006a270 con 0x7f76500ce6a0 2024-05-09T03:14:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.424+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f764001a070 con 0x7f76500ce6a0 2024-05-09T03:14:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.424+0000 7f7645ffb640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7634076300 0x7f7634078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.424+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f7640097670 con 0x7f76500ce6a0 2024-05-09T03:14:56.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.425+0000 7f76556e3640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7634076300 0x7f7634078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.425+0000 7f76556e3640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7634076300 0x7f7634078810 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f764c033000 tx=0x7f764c009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.429+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7640060970 con 0x7f76500ce6a0 2024-05-09T03:14:56.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.658+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7650072820 con 0x7f76500ce6a0 2024-05-09T03:14:56.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.658+0000 7f7645ffb640 1 -- 172.21.15.46:0/3085085654 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5180 (secure 0 0 0) 0x7f7640064c70 con 0x7f76500ce6a0 2024-05-09T03:14:56.659 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:56.659 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":69,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:14:55.669690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:14:56.659 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 69 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7634076300 msgr2=0x7f7634078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7634076300 0x7f7634078810 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f764c033000 tx=0x7f764c009000 comp rx=0 tx=0).stop 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 msgr2=0x7f765014b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 0x7f765014b010 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f764000e8a0 tx=0x7f764000ed60 comp rx=0 tx=0).stop 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 shutdown_connections 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76500d0480 0x7f765014b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7634076300 0x7f7634078810 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76500ce6a0 0x7f765014b010 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 --2- 172.21.15.46:0/3085085654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76500b92e0 0x7f7650159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 >> 172.21.15.46:0/3085085654 conn(0x7f76500d3c00 msgr2=0x7f76500b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:56.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 shutdown_connections 2024-05-09T03:14:56.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.660+0000 7f7657947640 1 -- 172.21.15.46:0/3085085654 wait complete. 2024-05-09T03:14:56.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:14:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.787+0000 7f40a0014640 1 Processor -- start 2024-05-09T03:14:56.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.787+0000 7f40a0014640 1 -- start start 2024-05-09T03:14:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f40a0014640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4098164c70 0x7f40981615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f40a0014640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f409815dfe0 0x7f40980d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f40a0014640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4098160b20 con 0x7f40980d2480 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f40a0014640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4098161150 con 0x7f409815dfe0 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f40a0014640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f40980d15f0 con 0x7f4098164c70 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f409d5af640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f409815dfe0 0x7f40980d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f409d5af640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f409815dfe0 0x7f40980d0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41062/0 (socket says 172.21.15.46:41062) 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.788+0000 7f409d5af640 1 -- 172.21.15.46:0/1979869500 learned_addr learned my addr 172.21.15.46:0/1979869500 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409ddb0640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4098164c70 0x7f40981615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409d5af640 1 -- 172.21.15.46:0/1979869500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4098164c70 msgr2=0x7f40981615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409d5af640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4098164c70 0x7f40981615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409d5af640 1 -- 172.21.15.46:0/1979869500 >> v1:172.21.15.88:6789/0 conn(0x7f40980d2480 legacy=0x7f40980d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409d5af640 1 -- 172.21.15.46:0/1979869500 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40980d18d0 con 0x7f409815dfe0 2024-05-09T03:14:56.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409ddb0640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4098164c70 0x7f40981615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.789+0000 7f409d5af640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f409815dfe0 0x7f40980d0dc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f408c009870 tx=0x7f408c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790e7ce4e81b0d9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f409cdae640 1 -- 172.21.15.46:0/1979869500 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c037070 con 0x7f409815dfe0 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f409cdae640 1 -- 172.21.15.46:0/1979869500 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c032440 con 0x7f409815dfe0 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f409cdae640 1 -- 172.21.15.46:0/1979869500 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c032810 con 0x7f409815dfe0 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 -- 172.21.15.46:0/1979869500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f409815dfe0 msgr2=0x7f40980d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f409815dfe0 0x7f40980d0dc0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f408c009870 tx=0x7f408c02a130 comp rx=0 tx=0).stop 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 -- 172.21.15.46:0/1979869500 shutdown_connections 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f409815dfe0 0x7f40980d0dc0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 --2- 172.21.15.46:0/1979869500 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4098164c70 0x7f40981615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 -- 172.21.15.46:0/1979869500 >> 172.21.15.46:0/1979869500 conn(0x7f4098073db0 msgr2=0x7f40980741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:56.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 -- 172.21.15.46:0/1979869500 shutdown_connections 2024-05-09T03:14:56.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.790+0000 7f40a0014640 1 -- 172.21.15.46:0/1979869500 wait complete. 2024-05-09T03:14:56.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.792+0000 7f40a0014640 1 Processor -- start 2024-05-09T03:14:56.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.792+0000 7f40a0014640 1 -- start start 2024-05-09T03:14:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.793+0000 7f40a0014640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 0x7f4098152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f409815dfe0 0x7f40981522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4098164c70 0x7f40980cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f409815e3f0 con 0x7f40980d2480 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40980d1300 con 0x7f4098164c70 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f40980d45c0 con 0x7f409815dfe0 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 0x7f4098152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 0x7f4098152de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41070/0 (socket says 172.21.15.46:41070) 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 -- 172.21.15.46:0/1621370234 learned_addr learned my addr 172.21.15.46:0/1621370234 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409d5af640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f409815dfe0 0x7f40981522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409e5b1640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4098164c70 0x7f40980cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 -- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f409815dfe0 msgr2=0x7f40981522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f409815dfe0 0x7f40981522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 -- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4098164c70 msgr2=0x7f40980cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4098164c70 0x7f40980cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:56.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 -- 172.21.15.46:0/1621370234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4098153320 con 0x7f40980d2480 2024-05-09T03:14:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f409ddb0640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 0x7f4098152de0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f409000c820 tx=0x7f409000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090015070 con 0x7f40980d2480 2024-05-09T03:14:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4090007a30 con 0x7f40980d2480 2024-05-09T03:14:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40980ce510 con 0x7f40980d2480 2024-05-09T03:14:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090007de0 con 0x7f40980d2480 2024-05-09T03:14:56.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.794+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40980cead0 con 0x7f40980d2480 2024-05-09T03:14:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.795+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409808cfd0 con 0x7f40980d2480 2024-05-09T03:14:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.796+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4090018070 con 0x7f40980d2480 2024-05-09T03:14:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.797+0000 7f40967fc640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40840762d0 0x7f40840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.797+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f40900974b0 con 0x7f40980d2480 2024-05-09T03:14:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.797+0000 7f409d5af640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40840762d0 0x7f40840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.798+0000 7f409d5af640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40840762d0 0x7f40840787e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f408c034000 tx=0x7f408c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.801+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f409808cfd0 con 0x7f40980d2480 2024-05-09T03:14:56.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.997+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40980cf1f0 con 0x7f40980d2480 2024-05-09T03:14:56.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:56.998+0000 7f40967fc640 1 -- 172.21.15.46:0/1621370234 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16596 (secure 0 0 0) 0x7f4090060e10 con 0x7f40980d2480 2024-05-09T03:14:56.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:56.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":68,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:14:54.261294+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:14:50.331114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-09T03:14:51.590716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40840762d0 msgr2=0x7f40840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40840762d0 0x7f40840787e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f408c034000 tx=0x7f408c02e000 comp rx=0 tx=0).stop 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 msgr2=0x7f4098152de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 0x7f4098152de0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f409000c820 tx=0x7f409000cce0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 shutdown_connections 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4098164c70 0x7f40980cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f409815dfe0 0x7f40981522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40840762d0 0x7f40840787e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 --2- 172.21.15.46:0/1621370234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40980d2480 0x7f4098152de0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 >> 172.21.15.46:0/1621370234 conn(0x7f4098073db0 msgr2=0x7f409815bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.000+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 shutdown_connections 2024-05-09T03:14:57.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.001+0000 7f40a0014640 1 -- 172.21.15.46:0/1621370234 wait complete. 2024-05-09T03:14:57.013 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:14:57.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:14:57.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.127+0000 7f92525b2640 1 Processor -- start 2024-05-09T03:14:57.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.127+0000 7f92525b2640 1 -- start start 2024-05-09T03:14:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f92525b2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f924c1604b0 0x7f924c06d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f92525b2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f924c0d1650 0x7f924c0ba090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f92525b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924c0b95f0 con 0x7f924c06d5d0 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f92525b2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f924c0b9c00 con 0x7f924c0d1650 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f92525b2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f924c06dac0 con 0x7f924c1604b0 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f9250b4f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f924c0d1650 0x7f924c0ba090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f9250b4f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f924c0d1650 0x7f924c0ba090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41078/0 (socket says 172.21.15.46:41078) 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.128+0000 7f9250b4f640 1 -- 172.21.15.46:0/1727893476 learned_addr learned my addr 172.21.15.46:0/1727893476 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.129+0000 7f924bfff640 1 --2- 172.21.15.46:0/1727893476 >> v2:172.21.15.88:3300/0 conn(0x7f924c1604b0 0x7f924c06d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.129+0000 7f9250b4f640 1 -- 172.21.15.46:0/1727893476 >> v2:172.21.15.88:3300/0 conn(0x7f924c1604b0 msgr2=0x7f924c06d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.129+0000 7f9250b4f640 1 --2- 172.21.15.46:0/1727893476 >> v2:172.21.15.88:3300/0 conn(0x7f924c1604b0 0x7f924c06d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.129+0000 7f9250b4f640 1 -- 172.21.15.46:0/1727893476 >> v1:172.21.15.88:6789/0 conn(0x7f924c06d5d0 legacy=0x7f924c0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.129+0000 7f9250b4f640 1 -- 172.21.15.46:0/1727893476 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c06dda0 con 0x7f924c0d1650 2024-05-09T03:14:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f9250b4f640 1 --2- 172.21.15.46:0/1727893476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c0d1650 0x7f924c0ba090 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f924c0b8990 tx=0x7f924402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5305a3f9e2c627e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f924affd640 1 -- 172.21.15.46:0/1727893476 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244036070 con 0x7f924c0d1650 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f924affd640 1 -- 172.21.15.46:0/1727893476 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9244031600 con 0x7f924c0d1650 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f924affd640 1 -- 172.21.15.46:0/1727893476 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440319d0 con 0x7f924c0d1650 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f92525b2640 1 -- 172.21.15.46:0/1727893476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c0d1650 msgr2=0x7f924c0ba090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f92525b2640 1 --2- 172.21.15.46:0/1727893476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c0d1650 0x7f924c0ba090 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f924c0b8990 tx=0x7f924402a730 comp rx=0 tx=0).stop 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f92525b2640 1 -- 172.21.15.46:0/1727893476 shutdown_connections 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f92525b2640 1 --2- 172.21.15.46:0/1727893476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c0d1650 0x7f924c0ba090 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.130+0000 7f92525b2640 1 --2- 172.21.15.46:0/1727893476 >> v2:172.21.15.88:3300/0 conn(0x7f924c1604b0 0x7f924c06d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.131+0000 7f92525b2640 1 -- 172.21.15.46:0/1727893476 >> 172.21.15.46:0/1727893476 conn(0x7f924c0b79f0 msgr2=0x7f924c0b7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.131+0000 7f92525b2640 1 -- 172.21.15.46:0/1727893476 shutdown_connections 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.131+0000 7f92525b2640 1 -- 172.21.15.46:0/1727893476 wait complete. 2024-05-09T03:14:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.132+0000 7f92525b2640 1 Processor -- start 2024-05-09T03:14:57.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.133+0000 7f92525b2640 1 -- start start 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f92525b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 0x7f924c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f92525b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f924c1604b0 0x7f924c0c89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f92525b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924c159480 0x7f924c162150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f92525b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f924c0d43f0 con 0x7f924c06d5d0 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f92525b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1608c0 con 0x7f924c159480 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f92525b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f924c0d4570 con 0x7f924c1604b0 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 0x7f924c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 0x7f924c156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41086/0 (socket says 172.21.15.46:41086) 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 -- 172.21.15.46:0/2400981173 learned_addr learned my addr 172.21.15.46:0/2400981173 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924b7fe640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f924c1604b0 0x7f924c0c89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f9250b4f640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924c159480 0x7f924c162150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 -- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f924c1604b0 msgr2=0x7f924c0c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f924c1604b0 0x7f924c0c89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 -- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924c159480 msgr2=0x7f924c162150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924c159480 0x7f924c162150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.134+0000 7f924bfff640 1 -- 172.21.15.46:0/2400981173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c0c8f10 con 0x7f924c06d5d0 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.135+0000 7f924bfff640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 0x7f924c156ac0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f923c00e870 tx=0x7f923c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.135+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c017070 con 0x7f924c06d5d0 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.135+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c00c8e0 con 0x7f924c06d5d0 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.135+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c159c60 con 0x7f924c06d5d0 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.135+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00ccd0 con 0x7f924c06d5d0 2024-05-09T03:14:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.135+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c15a220 con 0x7f924c06d5d0 2024-05-09T03:14:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.136+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c06a270 con 0x7f924c06d5d0 2024-05-09T03:14:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.137+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f923c01a070 con 0x7f924c06d5d0 2024-05-09T03:14:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.137+0000 7f9248ff9640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92380762d0 0x7f92380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.137+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f923c097b10 con 0x7f924c06d5d0 2024-05-09T03:14:57.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.137+0000 7f924b7fe640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92380762d0 0x7f92380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.138+0000 7f924b7fe640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92380762d0 0x7f92380787e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f924c0d06a0 tx=0x7f9240009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.141+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f924c0c8f10 con 0x7f924c06d5d0 2024-05-09T03:14:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.327+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f924c0d0f70 con 0x7f92380762d0 2024-05-09T03:14:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.334+0000 7f9248ff9640 1 -- 172.21.15.46:0/2400981173 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252058 (secure 0 0 0) 0x7f924c0d0f70 con 0x7f92380762d0 2024-05-09T03:14:57.337 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:57.338 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92380762d0 msgr2=0x7f92380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92380762d0 0x7f92380787e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f924c0d06a0 tx=0x7f9240009040 comp rx=0 tx=0).stop 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 msgr2=0x7f924c156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 0x7f924c156ac0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f923c00e870 tx=0x7f923c00ed30 comp rx=0 tx=0).stop 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 shutdown_connections 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924c159480 0x7f924c162150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f924c1604b0 0x7f924c0c89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f92380762d0 0x7f92380787e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 --2- 172.21.15.46:0/2400981173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c06d5d0 0x7f924c156ac0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 >> 172.21.15.46:0/2400981173 conn(0x7f924c0b79f0 msgr2=0x7f924c0d4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:57.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 shutdown_connections 2024-05-09T03:14:57.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.337+0000 7f92525b2640 1 -- 172.21.15.46:0/2400981173 wait complete. 2024-05-09T03:14:57.349 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-05-09T03:14:56.577639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248816,"kb_used_data":7688,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754725904,"statfs":{"total":773094113280,"available":772839325696,"internally_reserved":0,"allocated":7872512,"data_stored":3983764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.347209"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812553+0000","last_change":"2024-05-09T03:14:53.283651+0000","last_active":"2024-05-09T03:14:54.812553+0000","last_peered":"2024-05-09T03:14:54.812553+0000","last_clean":"2024-05-09T03:14:54.812553+0000","last_became_active":"2024-05-09T03:14:53.283185+0000","last_became_peered":"2024-05-09T03:14:53.283185+0000","last_unstale":"2024-05-09T03:14:54.812553+0000","last_undegraded":"2024-05-09T03:14:54.812553+0000","last_fullsized":"2024-05-09T03:14:54.812553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:15.513935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266460+0000","last_change":"2024-05-09T03:14:53.274869+0000","last_active":"2024-05-09T03:14:54.266460+0000","last_peered":"2024-05-09T03:14:54.266460+0000","last_clean":"2024-05-09T03:14:54.266460+0000","last_became_active":"2024-05-09T03:14:53.274487+0000","last_became_peered":"2024-05-09T03:14:53.274487+0000","last_unstale":"2024-05-09T03:14:54.266460+0000","last_undegraded":"2024-05-09T03:14:54.266460+0000","last_fullsized":"2024-05-09T03:14:54.266460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:28.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728894+0000","last_change":"2024-05-09T03:14:53.264325+0000","last_active":"2024-05-09T03:14:54.728894+0000","last_peered":"2024-05-09T03:14:54.728894+0000","last_clean":"2024-05-09T03:14:54.728894+0000","last_became_active":"2024-05-09T03:14:53.263923+0000","last_became_peered":"2024-05-09T03:14:53.263923+0000","last_unstale":"2024-05-09T03:14:54.728894+0000","last_undegraded":"2024-05-09T03:14:54.728894+0000","last_fullsized":"2024-05-09T03:14:54.728894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:26.901496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811805+0000","last_change":"2024-05-09T03:14:53.269333+0000","last_active":"2024-05-09T03:14:54.811805+0000","last_peered":"2024-05-09T03:14:54.811805+0000","last_clean":"2024-05-09T03:14:54.811805+0000","last_became_active":"2024-05-09T03:14:53.267228+0000","last_became_peered":"2024-05-09T03:14:53.267228+0000","last_unstale":"2024-05-09T03:14:54.811805+0000","last_undegraded":"2024-05-09T03:14:54.811805+0000","last_fullsized":"2024-05-09T03:14:54.811805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:01.526962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264504+0000","last_change":"2024-05-09T03:14:53.275430+0000","last_active":"2024-05-09T03:14:54.264504+0000","last_peered":"2024-05-09T03:14:54.264504+0000","last_clean":"2024-05-09T03:14:54.264504+0000","last_became_active":"2024-05-09T03:14:53.275051+0000","last_became_peered":"2024-05-09T03:14:53.275051+0000","last_unstale":"2024-05-09T03:14:54.264504+0000","last_undegraded":"2024-05-09T03:14:54.264504+0000","last_fullsized":"2024-05-09T03:14:54.264504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:04:07.650264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266776+0000","last_change":"2024-05-09T03:14:53.272505+0000","last_active":"2024-05-09T03:14:54.266776+0000","last_peered":"2024-05-09T03:14:54.266776+0000","last_clean":"2024-05-09T03:14:54.266776+0000","last_became_active":"2024-05-09T03:14:53.272187+0000","last_became_peered":"2024-05-09T03:14:53.272187+0000","last_unstale":"2024-05-09T03:14:54.266776+0000","last_undegraded":"2024-05-09T03:14:54.266776+0000","last_fullsized":"2024-05-09T03:14:54.266776+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:15.379798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811648+0000","last_change":"2024-05-09T03:14:53.275469+0000","last_active":"2024-05-09T03:14:54.811648+0000","last_peered":"2024-05-09T03:14:54.811648+0000","last_clean":"2024-05-09T03:14:54.811648+0000","last_became_active":"2024-05-09T03:14:53.274274+0000","last_became_peered":"2024-05-09T03:14:53.274274+0000","last_unstale":"2024-05-09T03:14:54.811648+0000","last_undegraded":"2024-05-09T03:14:54.811648+0000","last_fullsized":"2024-05-09T03:14:54.811648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:26:48.782599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265733+0000","last_change":"2024-05-09T03:14:53.259449+0000","last_active":"2024-05-09T03:14:54.265733+0000","last_peered":"2024-05-09T03:14:54.265733+0000","last_clean":"2024-05-09T03:14:54.265733+0000","last_became_active":"2024-05-09T03:14:53.259126+0000","last_became_peered":"2024-05-09T03:14:53.259126+0000","last_unstale":"2024-05-09T03:14:54.265733+0000","last_undegraded":"2024-05-09T03:14:54.265733+0000","last_fullsized":"2024-05-09T03:14:54.265733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:55.380010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264885+0000","last_change":"2024-05-09T03:14:53.276670+0000","last_active":"2024-05-09T03:14:54.264885+0000","last_peered":"2024-05-09T03:14:54.264885+0000","last_clean":"2024-05-09T03:14:54.264885+0000","last_became_active":"2024-05-09T03:14:53.276264+0000","last_became_peered":"2024-05-09T03:14:53.276264+0000","last_unstale":"2024-05-09T03:14:54.264885+0000","last_undegraded":"2024-05-09T03:14:54.264885+0000","last_fullsized":"2024-05-09T03:14:54.264885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:07:17.309741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266485+0000","last_change":"2024-05-09T03:14:53.271463+0000","last_active":"2024-05-09T03:14:54.266485+0000","last_peered":"2024-05-09T03:14:54.266485+0000","last_clean":"2024-05-09T03:14:54.266485+0000","last_became_active":"2024-05-09T03:14:53.271122+0000","last_became_peered":"2024-05-09T03:14:53.271122+0000","last_unstale":"2024-05-09T03:14:54.266485+0000","last_undegraded":"2024-05-09T03:14:54.266485+0000","last_fullsized":"2024-05-09T03:14:54.266485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:48:21.205930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812457+0000","last_change":"2024-05-09T03:14:53.268615+0000","last_active":"2024-05-09T03:14:54.812457+0000","last_peered":"2024-05-09T03:14:54.812457+0000","last_clean":"2024-05-09T03:14:54.812457+0000","last_became_active":"2024-05-09T03:14:53.268266+0000","last_became_peered":"2024-05-09T03:14:53.268266+0000","last_unstale":"2024-05-09T03:14:54.812457+0000","last_undegraded":"2024-05-09T03:14:54.812457+0000","last_fullsized":"2024-05-09T03:14:54.812457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:50:26.840212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811961+0000","last_change":"2024-05-09T03:14:53.274762+0000","last_active":"2024-05-09T03:14:54.811961+0000","last_peered":"2024-05-09T03:14:54.811961+0000","last_clean":"2024-05-09T03:14:54.811961+0000","last_became_active":"2024-05-09T03:14:53.274456+0000","last_became_peered":"2024-05-09T03:14:53.274456+0000","last_unstale":"2024-05-09T03:14:54.811961+0000","last_undegraded":"2024-05-09T03:14:54.811961+0000","last_fullsized":"2024-05-09T03:14:54.811961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:18.263623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774921+0000","last_change":"2024-05-09T03:14:53.271480+0000","last_active":"2024-05-09T03:14:54.774921+0000","last_peered":"2024-05-09T03:14:54.774921+0000","last_clean":"2024-05-09T03:14:54.774921+0000","last_became_active":"2024-05-09T03:14:53.271164+0000","last_became_peered":"2024-05-09T03:14:53.271164+0000","last_unstale":"2024-05-09T03:14:54.774921+0000","last_undegraded":"2024-05-09T03:14:54.774921+0000","last_fullsized":"2024-05-09T03:14:54.774921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:15:18.328807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264913+0000","last_change":"2024-05-09T03:14:53.275121+0000","last_active":"2024-05-09T03:14:54.264913+0000","last_peered":"2024-05-09T03:14:54.264913+0000","last_clean":"2024-05-09T03:14:54.264913+0000","last_became_active":"2024-05-09T03:14:53.274810+0000","last_became_peered":"2024-05-09T03:14:53.274810+0000","last_unstale":"2024-05-09T03:14:54.264913+0000","last_undegraded":"2024-05-09T03:14:54.264913+0000","last_fullsized":"2024-05-09T03:14:54.264913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:53.273549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774788+0000","last_change":"2024-05-09T03:14:53.283274+0000","last_active":"2024-05-09T03:14:54.774788+0000","last_peered":"2024-05-09T03:14:54.774788+0000","last_clean":"2024-05-09T03:14:54.774788+0000","last_became_active":"2024-05-09T03:14:53.282935+0000","last_became_peered":"2024-05-09T03:14:53.282935+0000","last_unstale":"2024-05-09T03:14:54.774788+0000","last_undegraded":"2024-05-09T03:14:54.774788+0000","last_fullsized":"2024-05-09T03:14:54.774788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:23.932367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266254+0000","last_change":"2024-05-09T03:14:53.272506+0000","last_active":"2024-05-09T03:14:54.266254+0000","last_peered":"2024-05-09T03:14:54.266254+0000","last_clean":"2024-05-09T03:14:54.266254+0000","last_became_active":"2024-05-09T03:14:53.272184+0000","last_became_peered":"2024-05-09T03:14:53.272184+0000","last_unstale":"2024-05-09T03:14:54.266254+0000","last_undegraded":"2024-05-09T03:14:54.266254+0000","last_fullsized":"2024-05-09T03:14:54.266254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:15.379798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265006+0000","last_change":"2024-05-09T03:14:53.276327+0000","last_active":"2024-05-09T03:14:54.265006+0000","last_peered":"2024-05-09T03:14:54.265006+0000","last_clean":"2024-05-09T03:14:54.265006+0000","last_became_active":"2024-05-09T03:14:53.274889+0000","last_became_peered":"2024-05-09T03:14:53.274889+0000","last_unstale":"2024-05-09T03:14:54.265006+0000","last_undegraded":"2024-05-09T03:14:54.265006+0000","last_fullsized":"2024-05-09T03:14:54.265006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:01.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811919+0000","last_change":"2024-05-09T03:14:53.270028+0000","last_active":"2024-05-09T03:14:54.811919+0000","last_peered":"2024-05-09T03:14:54.811919+0000","last_clean":"2024-05-09T03:14:54.811919+0000","last_became_active":"2024-05-09T03:14:53.269125+0000","last_became_peered":"2024-05-09T03:14:53.269125+0000","last_unstale":"2024-05-09T03:14:54.811919+0000","last_undegraded":"2024-05-09T03:14:54.811919+0000","last_fullsized":"2024-05-09T03:14:54.811919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:22.884363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265538+0000","last_change":"2024-05-09T03:14:53.267505+0000","last_active":"2024-05-09T03:14:54.265538+0000","last_peered":"2024-05-09T03:14:54.265538+0000","last_clean":"2024-05-09T03:14:54.265538+0000","last_became_active":"2024-05-09T03:14:53.267168+0000","last_became_peered":"2024-05-09T03:14:53.267168+0000","last_unstale":"2024-05-09T03:14:54.265538+0000","last_undegraded":"2024-05-09T03:14:54.265538+0000","last_fullsized":"2024-05-09T03:14:54.265538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:45:15.835188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774931+0000","last_change":"2024-05-09T03:14:53.272305+0000","last_active":"2024-05-09T03:14:54.774931+0000","last_peered":"2024-05-09T03:14:54.774931+0000","last_clean":"2024-05-09T03:14:54.774931+0000","last_became_active":"2024-05-09T03:14:53.271931+0000","last_became_peered":"2024-05-09T03:14:53.271931+0000","last_unstale":"2024-05-09T03:14:54.774931+0000","last_undegraded":"2024-05-09T03:14:54.774931+0000","last_fullsized":"2024-05-09T03:14:54.774931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:34.680505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265537+0000","last_change":"2024-05-09T03:14:53.265337+0000","last_active":"2024-05-09T03:14:54.265537+0000","last_peered":"2024-05-09T03:14:54.265537+0000","last_clean":"2024-05-09T03:14:54.265537+0000","last_became_active":"2024-05-09T03:14:53.264422+0000","last_became_peered":"2024-05-09T03:14:53.264422+0000","last_unstale":"2024-05-09T03:14:54.265537+0000","last_undegraded":"2024-05-09T03:14:54.265537+0000","last_fullsized":"2024-05-09T03:14:54.265537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:02.232428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266112+0000","last_change":"2024-05-09T03:14:53.276316+0000","last_active":"2024-05-09T03:14:54.266112+0000","last_peered":"2024-05-09T03:14:54.266112+0000","last_clean":"2024-05-09T03:14:54.266112+0000","last_became_active":"2024-05-09T03:14:53.275969+0000","last_became_peered":"2024-05-09T03:14:53.275969+0000","last_unstale":"2024-05-09T03:14:54.266112+0000","last_undegraded":"2024-05-09T03:14:54.266112+0000","last_fullsized":"2024-05-09T03:14:54.266112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:28.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811458+0000","last_change":"2024-05-09T03:14:53.272467+0000","last_active":"2024-05-09T03:14:54.811458+0000","last_peered":"2024-05-09T03:14:54.811458+0000","last_clean":"2024-05-09T03:14:54.811458+0000","last_became_active":"2024-05-09T03:14:53.271720+0000","last_became_peered":"2024-05-09T03:14:53.271720+0000","last_unstale":"2024-05-09T03:14:54.811458+0000","last_undegraded":"2024-05-09T03:14:54.811458+0000","last_fullsized":"2024-05-09T03:14:54.811458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:34.680505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811437+0000","last_change":"2024-05-09T03:14:53.270665+0000","last_active":"2024-05-09T03:14:54.811437+0000","last_peered":"2024-05-09T03:14:54.811437+0000","last_clean":"2024-05-09T03:14:54.811437+0000","last_became_active":"2024-05-09T03:14:53.270311+0000","last_became_peered":"2024-05-09T03:14:53.270311+0000","last_unstale":"2024-05-09T03:14:54.811437+0000","last_undegraded":"2024-05-09T03:14:54.811437+0000","last_fullsized":"2024-05-09T03:14:54.811437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:15:18.328807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266487+0000","last_change":"2024-05-09T03:14:53.264638+0000","last_active":"2024-05-09T03:14:54.266487+0000","last_peered":"2024-05-09T03:14:54.266487+0000","last_clean":"2024-05-09T03:14:54.266487+0000","last_became_active":"2024-05-09T03:14:53.264303+0000","last_became_peered":"2024-05-09T03:14:53.264303+0000","last_unstale":"2024-05-09T03:14:54.266487+0000","last_undegraded":"2024-05-09T03:14:54.266487+0000","last_fullsized":"2024-05-09T03:14:54.266487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:41:12.167082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811940+0000","last_change":"2024-05-09T03:14:53.277241+0000","last_active":"2024-05-09T03:14:54.811940+0000","last_peered":"2024-05-09T03:14:54.811940+0000","last_clean":"2024-05-09T03:14:54.811940+0000","last_became_active":"2024-05-09T03:14:53.276942+0000","last_became_peered":"2024-05-09T03:14:53.276942+0000","last_unstale":"2024-05-09T03:14:54.811940+0000","last_undegraded":"2024-05-09T03:14:54.811940+0000","last_fullsized":"2024-05-09T03:14:54.811940+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:28.620101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728891+0000","last_change":"2024-05-09T03:14:53.269900+0000","last_active":"2024-05-09T03:14:54.728891+0000","last_peered":"2024-05-09T03:14:54.728891+0000","last_clean":"2024-05-09T03:14:54.728891+0000","last_became_active":"2024-05-09T03:14:53.269333+0000","last_became_peered":"2024-05-09T03:14:53.269333+0000","last_unstale":"2024-05-09T03:14:54.728891+0000","last_undegraded":"2024-05-09T03:14:54.728891+0000","last_fullsized":"2024-05-09T03:14:54.728891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:42.019471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265963+0000","last_change":"2024-05-09T03:14:53.276369+0000","last_active":"2024-05-09T03:14:54.265963+0000","last_peered":"2024-05-09T03:14:54.265963+0000","last_clean":"2024-05-09T03:14:54.265963+0000","last_became_active":"2024-05-09T03:14:53.276028+0000","last_became_peered":"2024-05-09T03:14:53.276028+0000","last_unstale":"2024-05-09T03:14:54.265963+0000","last_undegraded":"2024-05-09T03:14:54.265963+0000","last_fullsized":"2024-05-09T03:14:54.265963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:41.889517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728647+0000","last_change":"2024-05-09T03:14:53.271740+0000","last_active":"2024-05-09T03:14:54.728647+0000","last_peered":"2024-05-09T03:14:54.728647+0000","last_clean":"2024-05-09T03:14:54.728647+0000","last_became_active":"2024-05-09T03:14:53.271357+0000","last_became_peered":"2024-05-09T03:14:53.271357+0000","last_unstale":"2024-05-09T03:14:54.728647+0000","last_undegraded":"2024-05-09T03:14:54.728647+0000","last_fullsized":"2024-05-09T03:14:54.728647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:21:06.964110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811634+0000","last_change":"2024-05-09T03:14:53.277697+0000","last_active":"2024-05-09T03:14:54.811634+0000","last_peered":"2024-05-09T03:14:54.811634+0000","last_clean":"2024-05-09T03:14:54.811634+0000","last_became_active":"2024-05-09T03:14:53.277261+0000","last_became_peered":"2024-05-09T03:14:53.277261+0000","last_unstale":"2024-05-09T03:14:54.811634+0000","last_undegraded":"2024-05-09T03:14:54.811634+0000","last_fullsized":"2024-05-09T03:14:54.811634+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:41:12.167082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811937+0000","last_change":"2024-05-09T03:14:53.283263+0000","last_active":"2024-05-09T03:14:54.811937+0000","last_peered":"2024-05-09T03:14:54.811937+0000","last_clean":"2024-05-09T03:14:54.811937+0000","last_became_active":"2024-05-09T03:14:53.282919+0000","last_became_peered":"2024-05-09T03:14:53.282919+0000","last_unstale":"2024-05-09T03:14:54.811937+0000","last_undegraded":"2024-05-09T03:14:54.811937+0000","last_fullsized":"2024-05-09T03:14:54.811937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:58:06.949307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266075+0000","last_change":"2024-05-09T03:14:53.272407+0000","last_active":"2024-05-09T03:14:54.266075+0000","last_peered":"2024-05-09T03:14:54.266075+0000","last_clean":"2024-05-09T03:14:54.266075+0000","last_became_active":"2024-05-09T03:14:53.270109+0000","last_became_peered":"2024-05-09T03:14:53.270109+0000","last_unstale":"2024-05-09T03:14:54.266075+0000","last_undegraded":"2024-05-09T03:14:54.266075+0000","last_fullsized":"2024-05-09T03:14:54.266075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:41.889517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812413+0000","last_change":"2024-05-09T03:14:53.277547+0000","last_active":"2024-05-09T03:14:54.812413+0000","last_peered":"2024-05-09T03:14:54.812413+0000","last_clean":"2024-05-09T03:14:54.812413+0000","last_became_active":"2024-05-09T03:14:53.277127+0000","last_became_peered":"2024-05-09T03:14:53.277127+0000","last_unstale":"2024-05-09T03:14:54.812413+0000","last_undegraded":"2024-05-09T03:14:54.812413+0000","last_fullsized":"2024-05-09T03:14:54.812413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:03.889945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812615+0000","last_change":"2024-05-09T03:14:52.255267+0000","last_active":"2024-05-09T03:14:54.812615+0000","last_peered":"2024-05-09T03:14:54.812615+0000","last_clean":"2024-05-09T03:14:54.812615+0000","last_became_active":"2024-05-09T03:14:52.254837+0000","last_became_peered":"2024-05-09T03:14:52.254837+0000","last_unstale":"2024-05-09T03:14:54.812615+0000","last_undegraded":"2024-05-09T03:14:54.812615+0000","last_fullsized":"2024-05-09T03:14:54.812615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:48:20.200933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265030+0000","last_change":"2024-05-09T03:14:53.276034+0000","last_active":"2024-05-09T03:14:54.265030+0000","last_peered":"2024-05-09T03:14:54.265030+0000","last_clean":"2024-05-09T03:14:54.265030+0000","last_became_active":"2024-05-09T03:14:53.274482+0000","last_became_peered":"2024-05-09T03:14:53.274482+0000","last_unstale":"2024-05-09T03:14:54.265030+0000","last_undegraded":"2024-05-09T03:14:54.265030+0000","last_fullsized":"2024-05-09T03:14:54.265030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:38.626936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774991+0000","last_change":"2024-05-09T03:14:52.252662+0000","last_active":"2024-05-09T03:14:54.774991+0000","last_peered":"2024-05-09T03:14:54.774991+0000","last_clean":"2024-05-09T03:14:54.774991+0000","last_became_active":"2024-05-09T03:14:52.252309+0000","last_became_peered":"2024-05-09T03:14:52.252309+0000","last_unstale":"2024-05-09T03:14:54.774991+0000","last_undegraded":"2024-05-09T03:14:54.774991+0000","last_fullsized":"2024-05-09T03:14:54.774991+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:40:48.230068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266039+0000","last_change":"2024-05-09T03:14:53.267117+0000","last_active":"2024-05-09T03:14:54.266039+0000","last_peered":"2024-05-09T03:14:54.266039+0000","last_clean":"2024-05-09T03:14:54.266039+0000","last_became_active":"2024-05-09T03:14:53.266818+0000","last_became_peered":"2024-05-09T03:14:53.266818+0000","last_unstale":"2024-05-09T03:14:54.266039+0000","last_undegraded":"2024-05-09T03:14:54.266039+0000","last_fullsized":"2024-05-09T03:14:54.266039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:46.147440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266324+0000","last_change":"2024-05-09T03:14:52.251884+0000","last_active":"2024-05-09T03:14:54.266324+0000","last_peered":"2024-05-09T03:14:54.266324+0000","last_clean":"2024-05-09T03:14:54.266324+0000","last_became_active":"2024-05-09T03:14:52.251607+0000","last_became_peered":"2024-05-09T03:14:52.251607+0000","last_unstale":"2024-05-09T03:14:54.266324+0000","last_undegraded":"2024-05-09T03:14:54.266324+0000","last_fullsized":"2024-05-09T03:14:54.266324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:21.879366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812469+0000","last_change":"2024-05-09T03:14:53.271704+0000","last_active":"2024-05-09T03:14:54.812469+0000","last_peered":"2024-05-09T03:14:54.812469+0000","last_clean":"2024-05-09T03:14:54.812469+0000","last_became_active":"2024-05-09T03:14:53.271387+0000","last_became_peered":"2024-05-09T03:14:53.271387+0000","last_unstale":"2024-05-09T03:14:54.812469+0000","last_undegraded":"2024-05-09T03:14:54.812469+0000","last_fullsized":"2024-05-09T03:14:54.812469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:14:57.424141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.267036+0000","last_change":"2024-05-09T03:14:52.254562+0000","last_active":"2024-05-09T03:14:54.267036+0000","last_peered":"2024-05-09T03:14:54.267036+0000","last_clean":"2024-05-09T03:14:54.267036+0000","last_became_active":"2024-05-09T03:14:52.254039+0000","last_became_peered":"2024-05-09T03:14:52.254039+0000","last_unstale":"2024-05-09T03:14:54.267036+0000","last_undegraded":"2024-05-09T03:14:54.267036+0000","last_fullsized":"2024-05-09T03:14:54.267036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:49.240461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811851+0000","last_change":"2024-05-09T03:14:53.268322+0000","last_active":"2024-05-09T03:14:54.811851+0000","last_peered":"2024-05-09T03:14:54.811851+0000","last_clean":"2024-05-09T03:14:54.811851+0000","last_became_active":"2024-05-09T03:14:53.268025+0000","last_became_peered":"2024-05-09T03:14:53.268025+0000","last_unstale":"2024-05-09T03:14:54.811851+0000","last_undegraded":"2024-05-09T03:14:54.811851+0000","last_fullsized":"2024-05-09T03:14:54.811851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:06:10.182844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728711+0000","last_change":"2024-05-09T03:14:52.249626+0000","last_active":"2024-05-09T03:14:54.728711+0000","last_peered":"2024-05-09T03:14:54.728711+0000","last_clean":"2024-05-09T03:14:54.728711+0000","last_became_active":"2024-05-09T03:14:52.249292+0000","last_became_peered":"2024-05-09T03:14:52.249292+0000","last_unstale":"2024-05-09T03:14:54.728711+0000","last_undegraded":"2024-05-09T03:14:54.728711+0000","last_fullsized":"2024-05-09T03:14:54.728711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:57.522390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265713+0000","last_change":"2024-05-09T03:14:53.263866+0000","last_active":"2024-05-09T03:14:54.265713+0000","last_peered":"2024-05-09T03:14:54.265713+0000","last_clean":"2024-05-09T03:14:54.265713+0000","last_became_active":"2024-05-09T03:14:53.263536+0000","last_became_peered":"2024-05-09T03:14:53.263536+0000","last_unstale":"2024-05-09T03:14:54.265713+0000","last_undegraded":"2024-05-09T03:14:54.265713+0000","last_fullsized":"2024-05-09T03:14:54.265713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:46.147440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266734+0000","last_change":"2024-05-09T03:14:52.252988+0000","last_active":"2024-05-09T03:14:54.266734+0000","last_peered":"2024-05-09T03:14:54.266734+0000","last_clean":"2024-05-09T03:14:54.266734+0000","last_became_active":"2024-05-09T03:14:52.252688+0000","last_became_peered":"2024-05-09T03:14:52.252688+0000","last_unstale":"2024-05-09T03:14:54.266734+0000","last_undegraded":"2024-05-09T03:14:54.266734+0000","last_fullsized":"2024-05-09T03:14:54.266734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:59.416101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264371+0000","last_change":"2024-05-09T03:14:53.269215+0000","last_active":"2024-05-09T03:14:54.264371+0000","last_peered":"2024-05-09T03:14:54.264371+0000","last_clean":"2024-05-09T03:14:54.264371+0000","last_became_active":"2024-05-09T03:14:53.268903+0000","last_became_peered":"2024-05-09T03:14:53.268903+0000","last_unstale":"2024-05-09T03:14:54.264371+0000","last_undegraded":"2024-05-09T03:14:54.264371+0000","last_fullsized":"2024-05-09T03:14:54.264371+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:55.405229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811472+0000","last_change":"2024-05-09T03:14:52.257332+0000","last_active":"2024-05-09T03:14:54.811472+0000","last_peered":"2024-05-09T03:14:54.811472+0000","last_clean":"2024-05-09T03:14:54.811472+0000","last_became_active":"2024-05-09T03:14:52.257003+0000","last_became_peered":"2024-05-09T03:14:52.257003+0000","last_unstale":"2024-05-09T03:14:54.811472+0000","last_undegraded":"2024-05-09T03:14:54.811472+0000","last_fullsized":"2024-05-09T03:14:54.811472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:49:24.627520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774937+0000","last_change":"2024-05-09T03:14:53.262750+0000","last_active":"2024-05-09T03:14:54.774937+0000","last_peered":"2024-05-09T03:14:54.774937+0000","last_clean":"2024-05-09T03:14:54.774937+0000","last_became_active":"2024-05-09T03:14:53.262360+0000","last_became_peered":"2024-05-09T03:14:53.262360+0000","last_unstale":"2024-05-09T03:14:54.774937+0000","last_undegraded":"2024-05-09T03:14:54.774937+0000","last_fullsized":"2024-05-09T03:14:54.774937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:15.398527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264320+0000","last_change":"2024-05-09T03:14:52.259321+0000","last_active":"2024-05-09T03:14:54.264320+0000","last_peered":"2024-05-09T03:14:54.264320+0000","last_clean":"2024-05-09T03:14:54.264320+0000","last_became_active":"2024-05-09T03:14:52.258674+0000","last_became_peered":"2024-05-09T03:14:52.258674+0000","last_unstale":"2024-05-09T03:14:54.264320+0000","last_undegraded":"2024-05-09T03:14:54.264320+0000","last_fullsized":"2024-05-09T03:14:54.264320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:17:05.234840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812094+0000","last_change":"2024-05-09T03:14:53.269779+0000","last_active":"2024-05-09T03:14:54.812094+0000","last_peered":"2024-05-09T03:14:54.812094+0000","last_clean":"2024-05-09T03:14:54.812094+0000","last_became_active":"2024-05-09T03:14:53.269452+0000","last_became_peered":"2024-05-09T03:14:53.269452+0000","last_unstale":"2024-05-09T03:14:54.812094+0000","last_undegraded":"2024-05-09T03:14:54.812094+0000","last_fullsized":"2024-05-09T03:14:54.812094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:03.901635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266482+0000","last_change":"2024-05-09T03:14:52.253645+0000","last_active":"2024-05-09T03:14:54.266482+0000","last_peered":"2024-05-09T03:14:54.266482+0000","last_clean":"2024-05-09T03:14:54.266482+0000","last_became_active":"2024-05-09T03:14:52.253286+0000","last_became_peered":"2024-05-09T03:14:52.253286+0000","last_unstale":"2024-05-09T03:14:54.266482+0000","last_undegraded":"2024-05-09T03:14:54.266482+0000","last_fullsized":"2024-05-09T03:14:54.266482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:31.423280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812441+0000","last_change":"2024-05-09T03:14:53.277762+0000","last_active":"2024-05-09T03:14:54.812441+0000","last_peered":"2024-05-09T03:14:54.812441+0000","last_clean":"2024-05-09T03:14:54.812441+0000","last_became_active":"2024-05-09T03:14:53.277448+0000","last_became_peered":"2024-05-09T03:14:53.277448+0000","last_unstale":"2024-05-09T03:14:54.812441+0000","last_undegraded":"2024-05-09T03:14:54.812441+0000","last_fullsized":"2024-05-09T03:14:54.812441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:22.285256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266609+0000","last_change":"2024-05-09T03:14:52.256256+0000","last_active":"2024-05-09T03:14:54.266609+0000","last_peered":"2024-05-09T03:14:54.266609+0000","last_clean":"2024-05-09T03:14:54.266609+0000","last_became_active":"2024-05-09T03:14:52.255933+0000","last_became_peered":"2024-05-09T03:14:52.255933+0000","last_unstale":"2024-05-09T03:14:54.266609+0000","last_undegraded":"2024-05-09T03:14:54.266609+0000","last_fullsized":"2024-05-09T03:14:54.266609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:13:44.943490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266957+0000","last_change":"2024-05-09T03:14:53.271770+0000","last_active":"2024-05-09T03:14:54.266957+0000","last_peered":"2024-05-09T03:14:54.266957+0000","last_clean":"2024-05-09T03:14:54.266957+0000","last_became_active":"2024-05-09T03:14:53.271415+0000","last_became_peered":"2024-05-09T03:14:53.271415+0000","last_unstale":"2024-05-09T03:14:54.266957+0000","last_undegraded":"2024-05-09T03:14:54.266957+0000","last_fullsized":"2024-05-09T03:14:54.266957+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:48:21.205930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728954+0000","last_change":"2024-05-09T03:14:52.250488+0000","last_active":"2024-05-09T03:14:54.728954+0000","last_peered":"2024-05-09T03:14:54.728954+0000","last_clean":"2024-05-09T03:14:54.728954+0000","last_became_active":"2024-05-09T03:14:52.250163+0000","last_became_peered":"2024-05-09T03:14:52.250163+0000","last_unstale":"2024-05-09T03:14:54.728954+0000","last_undegraded":"2024-05-09T03:14:54.728954+0000","last_fullsized":"2024-05-09T03:14:54.728954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:14.393530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264771+0000","last_change":"2024-05-09T03:14:53.272479+0000","last_active":"2024-05-09T03:14:54.264771+0000","last_peered":"2024-05-09T03:14:54.264771+0000","last_clean":"2024-05-09T03:14:54.264771+0000","last_became_active":"2024-05-09T03:14:53.271902+0000","last_became_peered":"2024-05-09T03:14:53.271902+0000","last_unstale":"2024-05-09T03:14:54.264771+0000","last_undegraded":"2024-05-09T03:14:54.264771+0000","last_fullsized":"2024-05-09T03:14:54.264771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:16.554452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812009+0000","last_change":"2024-05-09T03:14:52.257050+0000","last_active":"2024-05-09T03:14:54.812009+0000","last_peered":"2024-05-09T03:14:54.812009+0000","last_clean":"2024-05-09T03:14:54.812009+0000","last_became_active":"2024-05-09T03:14:52.256714+0000","last_became_peered":"2024-05-09T03:14:52.256714+0000","last_unstale":"2024-05-09T03:14:54.812009+0000","last_undegraded":"2024-05-09T03:14:54.812009+0000","last_fullsized":"2024-05-09T03:14:54.812009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:49.240461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812056+0000","last_change":"2024-05-09T03:14:53.275165+0000","last_active":"2024-05-09T03:14:54.812056+0000","last_peered":"2024-05-09T03:14:54.812056+0000","last_clean":"2024-05-09T03:14:54.812056+0000","last_became_active":"2024-05-09T03:14:53.274855+0000","last_became_peered":"2024-05-09T03:14:53.274855+0000","last_unstale":"2024-05-09T03:14:54.812056+0000","last_undegraded":"2024-05-09T03:14:54.812056+0000","last_fullsized":"2024-05-09T03:14:54.812056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:15.398527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811653+0000","last_change":"2024-05-09T03:14:52.249673+0000","last_active":"2024-05-09T03:14:54.811653+0000","last_peered":"2024-05-09T03:14:54.811653+0000","last_clean":"2024-05-09T03:14:54.811653+0000","last_became_active":"2024-05-09T03:14:52.249364+0000","last_became_peered":"2024-05-09T03:14:52.249364+0000","last_unstale":"2024-05-09T03:14:54.811653+0000","last_undegraded":"2024-05-09T03:14:54.811653+0000","last_fullsized":"2024-05-09T03:14:54.811653+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:35.230340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812288+0000","last_change":"2024-05-09T03:14:53.271261+0000","last_active":"2024-05-09T03:14:54.812288+0000","last_peered":"2024-05-09T03:14:54.812288+0000","last_clean":"2024-05-09T03:14:54.812288+0000","last_became_active":"2024-05-09T03:14:53.270916+0000","last_became_peered":"2024-05-09T03:14:53.270916+0000","last_unstale":"2024-05-09T03:14:54.812288+0000","last_undegraded":"2024-05-09T03:14:54.812288+0000","last_fullsized":"2024-05-09T03:14:54.812288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:00:00.455176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266880+0000","last_change":"2024-05-09T03:14:52.254481+0000","last_active":"2024-05-09T03:14:54.266880+0000","last_peered":"2024-05-09T03:14:54.266880+0000","last_clean":"2024-05-09T03:14:54.266880+0000","last_became_active":"2024-05-09T03:14:52.254210+0000","last_became_peered":"2024-05-09T03:14:52.254210+0000","last_unstale":"2024-05-09T03:14:54.266880+0000","last_undegraded":"2024-05-09T03:14:54.266880+0000","last_fullsized":"2024-05-09T03:14:54.266880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:00.521965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774714+0000","last_change":"2024-05-09T03:14:53.270787+0000","last_active":"2024-05-09T03:14:54.774714+0000","last_peered":"2024-05-09T03:14:54.774714+0000","last_clean":"2024-05-09T03:14:54.774714+0000","last_became_active":"2024-05-09T03:14:53.270416+0000","last_became_peered":"2024-05-09T03:14:53.270416+0000","last_unstale":"2024-05-09T03:14:54.774714+0000","last_undegraded":"2024-05-09T03:14:54.774714+0000","last_fullsized":"2024-05-09T03:14:54.774714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:08.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812111+0000","last_change":"2024-05-09T03:14:52.258697+0000","last_active":"2024-05-09T03:14:54.812111+0000","last_peered":"2024-05-09T03:14:54.812111+0000","last_clean":"2024-05-09T03:14:54.812111+0000","last_became_active":"2024-05-09T03:14:52.258385+0000","last_became_peered":"2024-05-09T03:14:52.258385+0000","last_unstale":"2024-05-09T03:14:54.812111+0000","last_undegraded":"2024-05-09T03:14:54.812111+0000","last_fullsized":"2024-05-09T03:14:54.812111+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:44.758894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264558+0000","last_change":"2024-05-09T03:14:53.267557+0000","last_active":"2024-05-09T03:14:54.264558+0000","last_peered":"2024-05-09T03:14:54.264558+0000","last_clean":"2024-05-09T03:14:54.264558+0000","last_became_active":"2024-05-09T03:14:53.267177+0000","last_became_peered":"2024-05-09T03:14:53.267177+0000","last_unstale":"2024-05-09T03:14:54.264558+0000","last_undegraded":"2024-05-09T03:14:54.264558+0000","last_fullsized":"2024-05-09T03:14:54.264558+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:32:40.343939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266050+0000","last_change":"2024-05-09T03:14:52.249521+0000","last_active":"2024-05-09T03:14:54.266050+0000","last_peered":"2024-05-09T03:14:54.266050+0000","last_clean":"2024-05-09T03:14:54.266050+0000","last_became_active":"2024-05-09T03:14:52.248820+0000","last_became_peered":"2024-05-09T03:14:52.248820+0000","last_unstale":"2024-05-09T03:14:54.266050+0000","last_undegraded":"2024-05-09T03:14:54.266050+0000","last_fullsized":"2024-05-09T03:14:54.266050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:42.690879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266519+0000","last_change":"2024-05-09T03:14:53.265021+0000","last_active":"2024-05-09T03:14:54.266519+0000","last_peered":"2024-05-09T03:14:54.266519+0000","last_clean":"2024-05-09T03:14:54.266519+0000","last_became_active":"2024-05-09T03:14:53.264694+0000","last_became_peered":"2024-05-09T03:14:53.264694+0000","last_unstale":"2024-05-09T03:14:54.266519+0000","last_undegraded":"2024-05-09T03:14:54.266519+0000","last_fullsized":"2024-05-09T03:14:54.266519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:26:48.782599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266159+0000","last_change":"2024-05-09T03:14:52.249953+0000","last_active":"2024-05-09T03:14:54.266159+0000","last_peered":"2024-05-09T03:14:54.266159+0000","last_clean":"2024-05-09T03:14:54.266159+0000","last_became_active":"2024-05-09T03:14:52.249621+0000","last_became_peered":"2024-05-09T03:14:52.249621+0000","last_unstale":"2024-05-09T03:14:54.266159+0000","last_undegraded":"2024-05-09T03:14:54.266159+0000","last_fullsized":"2024-05-09T03:14:54.266159+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:00:23.059773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812578+0000","last_change":"2024-05-09T03:14:53.274470+0000","last_active":"2024-05-09T03:14:54.812578+0000","last_peered":"2024-05-09T03:14:54.812578+0000","last_clean":"2024-05-09T03:14:54.812578+0000","last_became_active":"2024-05-09T03:14:53.274160+0000","last_became_peered":"2024-05-09T03:14:53.274160+0000","last_unstale":"2024-05-09T03:14:54.812578+0000","last_undegraded":"2024-05-09T03:14:54.812578+0000","last_fullsized":"2024-05-09T03:14:54.812578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:20.233102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266544+0000","last_change":"2024-05-09T03:14:52.251850+0000","last_active":"2024-05-09T03:14:54.266544+0000","last_peered":"2024-05-09T03:14:54.266544+0000","last_clean":"2024-05-09T03:14:54.266544+0000","last_became_active":"2024-05-09T03:14:52.251556+0000","last_became_peered":"2024-05-09T03:14:52.251556+0000","last_unstale":"2024-05-09T03:14:54.266544+0000","last_undegraded":"2024-05-09T03:14:54.266544+0000","last_fullsized":"2024-05-09T03:14:54.266544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:18:20.434159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811964+0000","last_change":"2024-05-09T03:14:53.270950+0000","last_active":"2024-05-09T03:14:54.811964+0000","last_peered":"2024-05-09T03:14:54.811964+0000","last_clean":"2024-05-09T03:14:54.811964+0000","last_became_active":"2024-05-09T03:14:53.270649+0000","last_became_peered":"2024-05-09T03:14:53.270649+0000","last_unstale":"2024-05-09T03:14:54.811964+0000","last_undegraded":"2024-05-09T03:14:54.811964+0000","last_fullsized":"2024-05-09T03:14:54.811964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:43.695876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265960+0000","last_change":"2024-05-09T03:14:52.255286+0000","last_active":"2024-05-09T03:14:54.265960+0000","last_peered":"2024-05-09T03:14:54.265960+0000","last_clean":"2024-05-09T03:14:54.265960+0000","last_became_active":"2024-05-09T03:14:52.255006+0000","last_became_peered":"2024-05-09T03:14:52.255006+0000","last_unstale":"2024-05-09T03:14:54.265960+0000","last_undegraded":"2024-05-09T03:14:54.265960+0000","last_fullsized":"2024-05-09T03:14:54.265960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:08.938961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264869+0000","last_change":"2024-05-09T03:14:53.272215+0000","last_active":"2024-05-09T03:14:54.264869+0000","last_peered":"2024-05-09T03:14:54.264869+0000","last_clean":"2024-05-09T03:14:54.264869+0000","last_became_active":"2024-05-09T03:14:53.271485+0000","last_became_peered":"2024-05-09T03:14:53.271485+0000","last_unstale":"2024-05-09T03:14:54.264869+0000","last_undegraded":"2024-05-09T03:14:54.264869+0000","last_fullsized":"2024-05-09T03:14:54.264869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:32:38.571666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812254+0000","last_change":"2024-05-09T03:14:52.258316+0000","last_active":"2024-05-09T03:14:54.812254+0000","last_peered":"2024-05-09T03:14:54.812254+0000","last_clean":"2024-05-09T03:14:54.812254+0000","last_became_active":"2024-05-09T03:14:52.257992+0000","last_became_peered":"2024-05-09T03:14:52.257992+0000","last_unstale":"2024-05-09T03:14:54.812254+0000","last_undegraded":"2024-05-09T03:14:54.812254+0000","last_fullsized":"2024-05-09T03:14:54.812254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:14.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728639+0000","last_change":"2024-05-09T03:14:53.281183+0000","last_active":"2024-05-09T03:14:54.728639+0000","last_peered":"2024-05-09T03:14:54.728639+0000","last_clean":"2024-05-09T03:14:54.728639+0000","last_became_active":"2024-05-09T03:14:53.280857+0000","last_became_peered":"2024-05-09T03:14:53.280857+0000","last_unstale":"2024-05-09T03:14:54.728639+0000","last_undegraded":"2024-05-09T03:14:54.728639+0000","last_fullsized":"2024-05-09T03:14:54.728639+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:13.043237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266885+0000","last_change":"2024-05-09T03:14:52.248137+0000","last_active":"2024-05-09T03:14:54.266885+0000","last_peered":"2024-05-09T03:14:54.266885+0000","last_clean":"2024-05-09T03:14:54.266885+0000","last_became_active":"2024-05-09T03:14:52.247419+0000","last_became_peered":"2024-05-09T03:14:52.247419+0000","last_unstale":"2024-05-09T03:14:54.266885+0000","last_undegraded":"2024-05-09T03:14:54.266885+0000","last_fullsized":"2024-05-09T03:14:54.266885+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:21.879366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728576+0000","last_change":"2024-05-09T03:14:53.277405+0000","last_active":"2024-05-09T03:14:54.728576+0000","last_peered":"2024-05-09T03:14:54.728576+0000","last_clean":"2024-05-09T03:14:54.728576+0000","last_became_active":"2024-05-09T03:14:53.277047+0000","last_became_peered":"2024-05-09T03:14:53.277047+0000","last_unstale":"2024-05-09T03:14:54.728576+0000","last_undegraded":"2024-05-09T03:14:54.728576+0000","last_fullsized":"2024-05-09T03:14:54.728576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:31.032365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812009+0000","last_change":"2024-05-09T03:14:52.260111+0000","last_active":"2024-05-09T03:14:54.812009+0000","last_peered":"2024-05-09T03:14:54.812009+0000","last_clean":"2024-05-09T03:14:54.812009+0000","last_became_active":"2024-05-09T03:14:52.259816+0000","last_became_peered":"2024-05-09T03:14:52.259816+0000","last_unstale":"2024-05-09T03:14:54.812009+0000","last_undegraded":"2024-05-09T03:14:54.812009+0000","last_fullsized":"2024-05-09T03:14:54.812009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:44.453397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728601+0000","last_change":"2024-05-09T03:14:53.279110+0000","last_active":"2024-05-09T03:14:54.728601+0000","last_peered":"2024-05-09T03:14:54.728601+0000","last_clean":"2024-05-09T03:14:54.728601+0000","last_became_active":"2024-05-09T03:14:53.278482+0000","last_became_peered":"2024-05-09T03:14:53.278482+0000","last_unstale":"2024-05-09T03:14:54.728601+0000","last_undegraded":"2024-05-09T03:14:54.728601+0000","last_fullsized":"2024-05-09T03:14:54.728601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:52:03.704231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774524+0000","last_change":"2024-05-09T03:14:52.252673+0000","last_active":"2024-05-09T03:14:54.774524+0000","last_peered":"2024-05-09T03:14:54.774524+0000","last_clean":"2024-05-09T03:14:54.774524+0000","last_became_active":"2024-05-09T03:14:52.252297+0000","last_became_peered":"2024-05-09T03:14:52.252297+0000","last_unstale":"2024-05-09T03:14:54.774524+0000","last_undegraded":"2024-05-09T03:14:54.774524+0000","last_fullsized":"2024-05-09T03:14:54.774524+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:41:11.162085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774955+0000","last_change":"2024-05-09T03:14:53.276368+0000","last_active":"2024-05-09T03:14:54.774955+0000","last_peered":"2024-05-09T03:14:54.774955+0000","last_clean":"2024-05-09T03:14:54.774955+0000","last_became_active":"2024-05-09T03:14:53.276066+0000","last_became_peered":"2024-05-09T03:14:53.276066+0000","last_unstale":"2024-05-09T03:14:54.774955+0000","last_undegraded":"2024-05-09T03:14:54.774955+0000","last_fullsized":"2024-05-09T03:14:54.774955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:58.527388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266332+0000","last_change":"2024-05-09T03:14:52.250637+0000","last_active":"2024-05-09T03:14:54.266332+0000","last_peered":"2024-05-09T03:14:54.266332+0000","last_clean":"2024-05-09T03:14:54.266332+0000","last_became_active":"2024-05-09T03:14:52.250335+0000","last_became_peered":"2024-05-09T03:14:52.250335+0000","last_unstale":"2024-05-09T03:14:54.266332+0000","last_undegraded":"2024-05-09T03:14:54.266332+0000","last_fullsized":"2024-05-09T03:14:54.266332+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:52:58.910513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811766+0000","last_change":"2024-05-09T03:14:53.274319+0000","last_active":"2024-05-09T03:14:54.811766+0000","last_peered":"2024-05-09T03:14:54.811766+0000","last_clean":"2024-05-09T03:14:54.811766+0000","last_became_active":"2024-05-09T03:14:53.273813+0000","last_became_peered":"2024-05-09T03:14:53.273813+0000","last_unstale":"2024-05-09T03:14:54.811766+0000","last_undegraded":"2024-05-09T03:14:54.811766+0000","last_fullsized":"2024-05-09T03:14:54.811766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:40:49.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812077+0000","last_change":"2024-05-09T03:14:52.254666+0000","last_active":"2024-05-09T03:14:54.812077+0000","last_peered":"2024-05-09T03:14:54.812077+0000","last_clean":"2024-05-09T03:14:54.812077+0000","last_became_active":"2024-05-09T03:14:52.254344+0000","last_became_peered":"2024-05-09T03:14:52.254344+0000","last_unstale":"2024-05-09T03:14:54.812077+0000","last_undegraded":"2024-05-09T03:14:54.812077+0000","last_fullsized":"2024-05-09T03:14:54.812077+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:03:44.617588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812232+0000","last_change":"2024-05-09T03:14:53.277535+0000","last_active":"2024-05-09T03:14:54.812232+0000","last_peered":"2024-05-09T03:14:54.812232+0000","last_clean":"2024-05-09T03:14:54.812232+0000","last_became_active":"2024-05-09T03:14:53.277140+0000","last_became_peered":"2024-05-09T03:14:53.277140+0000","last_unstale":"2024-05-09T03:14:54.812232+0000","last_undegraded":"2024-05-09T03:14:54.812232+0000","last_fullsized":"2024-05-09T03:14:54.812232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:58.527388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264829+0000","last_change":"2024-05-09T03:14:52.253298+0000","last_active":"2024-05-09T03:14:54.264829+0000","last_peered":"2024-05-09T03:14:54.264829+0000","last_clean":"2024-05-09T03:14:54.264829+0000","last_became_active":"2024-05-09T03:14:52.252924+0000","last_became_peered":"2024-05-09T03:14:52.252924+0000","last_unstale":"2024-05-09T03:14:54.264829+0000","last_undegraded":"2024-05-09T03:14:54.264829+0000","last_fullsized":"2024-05-09T03:14:54.264829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:57.659379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812591+0000","last_change":"2024-05-09T03:14:53.271786+0000","last_active":"2024-05-09T03:14:54.812591+0000","last_peered":"2024-05-09T03:14:54.812591+0000","last_clean":"2024-05-09T03:14:54.812591+0000","last_became_active":"2024-05-09T03:14:53.271482+0000","last_became_peered":"2024-05-09T03:14:53.271482+0000","last_unstale":"2024-05-09T03:14:54.812591+0000","last_undegraded":"2024-05-09T03:14:54.812591+0000","last_fullsized":"2024-05-09T03:14:54.812591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:21.029633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.775004+0000","last_change":"2024-05-09T03:14:52.247892+0000","last_active":"2024-05-09T03:14:54.775004+0000","last_peered":"2024-05-09T03:14:54.775004+0000","last_clean":"2024-05-09T03:14:54.775004+0000","last_became_active":"2024-05-09T03:14:52.247532+0000","last_became_peered":"2024-05-09T03:14:52.247532+0000","last_unstale":"2024-05-09T03:14:54.775004+0000","last_undegraded":"2024-05-09T03:14:54.775004+0000","last_fullsized":"2024-05-09T03:14:54.775004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:26:47.777602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264913+0000","last_change":"2024-05-09T03:14:53.267608+0000","last_active":"2024-05-09T03:14:54.264913+0000","last_peered":"2024-05-09T03:14:54.264913+0000","last_clean":"2024-05-09T03:14:54.264913+0000","last_became_active":"2024-05-09T03:14:53.267287+0000","last_became_peered":"2024-05-09T03:14:53.267287+0000","last_unstale":"2024-05-09T03:14:54.264913+0000","last_undegraded":"2024-05-09T03:14:54.264913+0000","last_fullsized":"2024-05-09T03:14:54.264913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:15:22.300992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264429+0000","last_change":"2024-05-09T03:14:52.252354+0000","last_active":"2024-05-09T03:14:54.264429+0000","last_peered":"2024-05-09T03:14:54.264429+0000","last_clean":"2024-05-09T03:14:54.264429+0000","last_became_active":"2024-05-09T03:14:52.252020+0000","last_became_peered":"2024-05-09T03:14:52.252020+0000","last_unstale":"2024-05-09T03:14:54.264429+0000","last_undegraded":"2024-05-09T03:14:54.264429+0000","last_fullsized":"2024-05-09T03:14:54.264429+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:39.937419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265681+0000","last_change":"2024-05-09T03:14:53.269169+0000","last_active":"2024-05-09T03:14:54.265681+0000","last_peered":"2024-05-09T03:14:54.265681+0000","last_clean":"2024-05-09T03:14:54.265681+0000","last_became_active":"2024-05-09T03:14:53.268657+0000","last_became_peered":"2024-05-09T03:14:53.268657+0000","last_unstale":"2024-05-09T03:14:54.265681+0000","last_undegraded":"2024-05-09T03:14:54.265681+0000","last_fullsized":"2024-05-09T03:14:54.265681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:37:46.502228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811451+0000","last_change":"2024-05-09T03:14:52.254279+0000","last_active":"2024-05-09T03:14:54.811451+0000","last_peered":"2024-05-09T03:14:54.811451+0000","last_clean":"2024-05-09T03:14:54.811451+0000","last_became_active":"2024-05-09T03:14:52.253906+0000","last_became_peered":"2024-05-09T03:14:52.253906+0000","last_unstale":"2024-05-09T03:14:54.811451+0000","last_undegraded":"2024-05-09T03:14:54.811451+0000","last_fullsized":"2024-05-09T03:14:54.811451+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:07:19.253833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811683+0000","last_change":"2024-05-09T03:14:53.269228+0000","last_active":"2024-05-09T03:14:54.811683+0000","last_peered":"2024-05-09T03:14:54.811683+0000","last_clean":"2024-05-09T03:14:54.811683+0000","last_became_active":"2024-05-09T03:14:53.268877+0000","last_became_peered":"2024-05-09T03:14:53.268877+0000","last_unstale":"2024-05-09T03:14:54.811683+0000","last_undegraded":"2024-05-09T03:14:54.811683+0000","last_fullsized":"2024-05-09T03:14:54.811683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:00.421098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265331+0000","last_change":"2024-05-09T03:14:52.250766+0000","last_active":"2024-05-09T03:14:54.265331+0000","last_peered":"2024-05-09T03:14:54.265331+0000","last_clean":"2024-05-09T03:14:54.265331+0000","last_became_active":"2024-05-09T03:14:52.250471+0000","last_became_peered":"2024-05-09T03:14:52.250471+0000","last_unstale":"2024-05-09T03:14:54.265331+0000","last_undegraded":"2024-05-09T03:14:54.265331+0000","last_fullsized":"2024-05-09T03:14:54.265331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:17.258626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265854+0000","last_change":"2024-05-09T03:14:53.267693+0000","last_active":"2024-05-09T03:14:54.265854+0000","last_peered":"2024-05-09T03:14:54.265854+0000","last_clean":"2024-05-09T03:14:54.265854+0000","last_became_active":"2024-05-09T03:14:53.267324+0000","last_became_peered":"2024-05-09T03:14:53.267324+0000","last_unstale":"2024-05-09T03:14:54.265854+0000","last_undegraded":"2024-05-09T03:14:54.265854+0000","last_fullsized":"2024-05-09T03:14:54.265854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:02.232428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265812+0000","last_change":"2024-05-09T03:14:52.248455+0000","last_active":"2024-05-09T03:14:54.265812+0000","last_peered":"2024-05-09T03:14:54.265812+0000","last_clean":"2024-05-09T03:14:54.265812+0000","last_became_active":"2024-05-09T03:14:52.248157+0000","last_became_peered":"2024-05-09T03:14:52.248157+0000","last_unstale":"2024-05-09T03:14:54.265812+0000","last_undegraded":"2024-05-09T03:14:54.265812+0000","last_fullsized":"2024-05-09T03:14:54.265812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:02:31.062201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728714+0000","last_change":"2024-05-09T03:14:53.278774+0000","last_active":"2024-05-09T03:14:54.728714+0000","last_peered":"2024-05-09T03:14:54.728714+0000","last_clean":"2024-05-09T03:14:54.728714+0000","last_became_active":"2024-05-09T03:14:53.278269+0000","last_became_peered":"2024-05-09T03:14:53.278269+0000","last_unstale":"2024-05-09T03:14:54.728714+0000","last_undegraded":"2024-05-09T03:14:54.728714+0000","last_fullsized":"2024-05-09T03:14:54.728714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:29:26.854313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265434+0000","last_change":"2024-05-09T03:14:52.249297+0000","last_active":"2024-05-09T03:14:54.265434+0000","last_peered":"2024-05-09T03:14:54.265434+0000","last_clean":"2024-05-09T03:14:54.265434+0000","last_became_active":"2024-05-09T03:14:52.248966+0000","last_became_peered":"2024-05-09T03:14:52.248966+0000","last_unstale":"2024-05-09T03:14:54.265434+0000","last_undegraded":"2024-05-09T03:14:54.265434+0000","last_fullsized":"2024-05-09T03:14:54.265434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:34.997223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":199,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811400+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:14:54.811400+0000","last_peered":"2024-05-09T03:14:54.811400+0000","last_clean":"2024-05-09T03:14:54.811400+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:14:54.811400+0000","last_undegraded":"2024-05-09T03:14:54.811400+0000","last_fullsized":"2024-05-09T03:14:54.811400+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":940760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":835584,"data_stored":349868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":349866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":349846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":349888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":839680,"data_stored":351388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":940144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:14:57.353 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:14:57.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.504+0000 7f20d05dc640 1 Processor -- start 2024-05-09T03:14:57.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.504+0000 7f20d05dc640 1 -- start start 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20d05dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20c80b9200 0x7f20c808d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20d05dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 0x7f20c80b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20d05dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20c80b4ee0 con 0x7f20c808d6e0 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20d05dc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f20c80b5460 con 0x7f20c80b9200 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20d05dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20c80cfac0 con 0x7f20c80d2270 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20cdb77640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20c808d6e0 0x7f20c80d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54154/0 (socket says 172.21.15.46:54154) 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.505+0000 7f20cdb77640 1 -- 172.21.15.46:0/1124427256 learned_addr learned my addr 172.21.15.46:0/1124427256 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20ce378640 1 --2- 172.21.15.46:0/1124427256 >> v2:172.21.15.88:3300/0 conn(0x7f20c80b9200 0x7f20c808d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20ceb79640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 0x7f20c80b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402382648 0 0) 0x7f20c80b4ee0 con 0x7f20c808d6e0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20b4003520 con 0x7f20c808d6e0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 613590805 0 0) 0x7f20b4003520 con 0x7f20c808d6e0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 msgr2=0x7f20c80b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 0x7f20c80b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 >> v2:172.21.15.88:3300/0 conn(0x7f20c80b9200 msgr2=0x7f20c808d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 --2- 172.21.15.46:0/1124427256 >> v2:172.21.15.88:3300/0 conn(0x7f20c80b9200 0x7f20c808d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c80b62e0 con 0x7f20c808d6e0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f20b80016f0 con 0x7f20c808d6e0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 >> v1:172.21.15.46:6789/0 conn(0x7f20c808d6e0 legacy=0x7f20c80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 0x7f20b4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b4003fd0 0x7f20b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20b4004d40 0x7f20b4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20c80cfac0 con 0x7f20b4003fd0 2024-05-09T03:14:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20b400ba70 con 0x7f20b4004d40 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20b400bc40 con 0x7f20c80d2270 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20cdb77640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b4003fd0 0x7f20b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20ce378640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 0x7f20b4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.506+0000 7f20ceb79640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20b4004d40 0x7f20b4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cdb77640 1 -- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 msgr2=0x7f20b4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cdb77640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20c80d2270 0x7f20b4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cdb77640 1 -- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20b4004d40 msgr2=0x7f20b4005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cdb77640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20b4004d40 0x7f20b4005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cdb77640 1 -- 172.21.15.46:0/1124427256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b4003520 con 0x7f20b4003fd0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cdb77640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b4003fd0 0x7f20b4004800 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f20c8074690 tx=0x7f20b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9812cb27136615ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8007d50 con 0x7f20b4003fd0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b8005af0 con 0x7f20b4003fd0 2024-05-09T03:14:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.507+0000 7f20cd376640 1 -- 172.21.15.46:0/1124427256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8007070 con 0x7f20b4003fd0 2024-05-09T03:14:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20d05dc640 1 -- 172.21.15.46:0/1124427256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b4003fd0 msgr2=0x7f20b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20d05dc640 1 --2- 172.21.15.46:0/1124427256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20b4003fd0 0x7f20b4004800 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f20c8074690 tx=0x7f20b4005830 comp rx=0 tx=0).stop 2024-05-09T03:14:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20ceb79640 1 -- 172.21.15.46:0/1124427256 reap_dead start 2024-05-09T03:14:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20d05dc640 1 -- 172.21.15.46:0/1124427256 shutdown_connections 2024-05-09T03:14:57.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20d05dc640 1 -- 172.21.15.46:0/1124427256 >> 172.21.15.46:0/1124427256 conn(0x7f20c80b56e0 msgr2=0x7f20c80b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:57.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20d05dc640 1 -- 172.21.15.46:0/1124427256 shutdown_connections 2024-05-09T03:14:57.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.508+0000 7f20d05dc640 1 -- 172.21.15.46:0/1124427256 wait complete. 2024-05-09T03:14:57.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.510+0000 7f20d05dc640 1 Processor -- start 2024-05-09T03:14:57.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.510+0000 7f20d05dc640 1 -- start start 2024-05-09T03:14:57.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20d05dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20b4003fd0 0x7f20c81562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20d05dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c8155430 0x7f20c81f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20d05dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 0x7f20c815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20d05dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f20c80cf750 con 0x7f20c815c3b0 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20d05dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20c80cf470 con 0x7f20c8155430 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20d05dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f20c80b4a20 con 0x7f20b4003fd0 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20ceb79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 0x7f20c815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.511+0000 7f20ceb79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 0x7f20c815c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41108/0 (socket says 172.21.15.46:41108) 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 -- 172.21.15.46:0/3218442095 learned_addr learned my addr 172.21.15.46:0/3218442095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20cdb77640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c8155430 0x7f20c81f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ce378640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20b4003fd0 0x7f20c81562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 -- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20b4003fd0 msgr2=0x7f20c81562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20b4003fd0 0x7f20c81562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 -- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c8155430 msgr2=0x7f20c81f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c8155430 0x7f20c81f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 -- 172.21.15.46:0/3218442095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c815cd00 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20cdb77640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c8155430 0x7f20c81f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20ceb79640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 0x7f20c815c7c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f20c808e3e0 tx=0x7f20c000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c000fe10 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c001d070 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c8156b30 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0015670 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.512+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c81570f0 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.513+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c81568a0 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.514+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f20c0015810 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.514+0000 7f20beffd640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20ac0762d0 0x7f20ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.515+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f20c0097ed0 con 0x7f20c815c3b0 2024-05-09T03:14:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.515+0000 7f20ce378640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20ac0762d0 0x7f20ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.515+0000 7f20ce378640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20ac0762d0 0x7f20ac0787e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009b00 tx=0x7f20c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.519+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f20c815cd00 con 0x7f20c815c3b0 2024-05-09T03:14:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.707+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f20c806a270 con 0x7f20ac0762d0 2024-05-09T03:14:57.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.714+0000 7f20beffd640 1 -- 172.21.15.46:0/3218442095 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252058 (secure 0 0 0) 0x7f20c806a270 con 0x7f20ac0762d0 2024-05-09T03:14:57.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:57.722 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:14:57.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20ac0762d0 msgr2=0x7f20ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20ac0762d0 0x7f20ac0787e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f20c4009b00 tx=0x7f20c4008040 comp rx=0 tx=0).stop 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 msgr2=0x7f20c815c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 0x7f20c815c7c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f20c808e3e0 tx=0x7f20c000f580 comp rx=0 tx=0).stop 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 shutdown_connections 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f20ac0762d0 0x7f20ac0787e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20c815c3b0 0x7f20c815c7c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c8155430 0x7f20c81f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 --2- 172.21.15.46:0/3218442095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20b4003fd0 0x7f20c81562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 >> 172.21.15.46:0/3218442095 conn(0x7f20c80b56e0 msgr2=0x7f20c815d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 shutdown_connections 2024-05-09T03:14:57.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.717+0000 7f20d05dc640 1 -- 172.21.15.46:0/3218442095 wait complete. 2024-05-09T03:14:57.729 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-05-09T03:14:56.577639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":183,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248816,"kb_used_data":7688,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754725904,"statfs":{"total":773094113280,"available":772839325696,"internally_reserved":0,"allocated":7872512,"data_stored":3983764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.347209"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812553+0000","last_change":"2024-05-09T03:14:53.283651+0000","last_active":"2024-05-09T03:14:54.812553+0000","last_peered":"2024-05-09T03:14:54.812553+0000","last_clean":"2024-05-09T03:14:54.812553+0000","last_became_active":"2024-05-09T03:14:53.283185+0000","last_became_peered":"2024-05-09T03:14:53.283185+0000","last_unstale":"2024-05-09T03:14:54.812553+0000","last_undegraded":"2024-05-09T03:14:54.812553+0000","last_fullsized":"2024-05-09T03:14:54.812553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:20:15.513935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266460+0000","last_change":"2024-05-09T03:14:53.274869+0000","last_active":"2024-05-09T03:14:54.266460+0000","last_peered":"2024-05-09T03:14:54.266460+0000","last_clean":"2024-05-09T03:14:54.266460+0000","last_became_active":"2024-05-09T03:14:53.274487+0000","last_became_peered":"2024-05-09T03:14:53.274487+0000","last_unstale":"2024-05-09T03:14:54.266460+0000","last_undegraded":"2024-05-09T03:14:54.266460+0000","last_fullsized":"2024-05-09T03:14:54.266460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:28.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728894+0000","last_change":"2024-05-09T03:14:53.264325+0000","last_active":"2024-05-09T03:14:54.728894+0000","last_peered":"2024-05-09T03:14:54.728894+0000","last_clean":"2024-05-09T03:14:54.728894+0000","last_became_active":"2024-05-09T03:14:53.263923+0000","last_became_peered":"2024-05-09T03:14:53.263923+0000","last_unstale":"2024-05-09T03:14:54.728894+0000","last_undegraded":"2024-05-09T03:14:54.728894+0000","last_fullsized":"2024-05-09T03:14:54.728894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:26.901496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811805+0000","last_change":"2024-05-09T03:14:53.269333+0000","last_active":"2024-05-09T03:14:54.811805+0000","last_peered":"2024-05-09T03:14:54.811805+0000","last_clean":"2024-05-09T03:14:54.811805+0000","last_became_active":"2024-05-09T03:14:53.267228+0000","last_became_peered":"2024-05-09T03:14:53.267228+0000","last_unstale":"2024-05-09T03:14:54.811805+0000","last_undegraded":"2024-05-09T03:14:54.811805+0000","last_fullsized":"2024-05-09T03:14:54.811805+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:01.526962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264504+0000","last_change":"2024-05-09T03:14:53.275430+0000","last_active":"2024-05-09T03:14:54.264504+0000","last_peered":"2024-05-09T03:14:54.264504+0000","last_clean":"2024-05-09T03:14:54.264504+0000","last_became_active":"2024-05-09T03:14:53.275051+0000","last_became_peered":"2024-05-09T03:14:53.275051+0000","last_unstale":"2024-05-09T03:14:54.264504+0000","last_undegraded":"2024-05-09T03:14:54.264504+0000","last_fullsized":"2024-05-09T03:14:54.264504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:04:07.650264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266776+0000","last_change":"2024-05-09T03:14:53.272505+0000","last_active":"2024-05-09T03:14:54.266776+0000","last_peered":"2024-05-09T03:14:54.266776+0000","last_clean":"2024-05-09T03:14:54.266776+0000","last_became_active":"2024-05-09T03:14:53.272187+0000","last_became_peered":"2024-05-09T03:14:53.272187+0000","last_unstale":"2024-05-09T03:14:54.266776+0000","last_undegraded":"2024-05-09T03:14:54.266776+0000","last_fullsized":"2024-05-09T03:14:54.266776+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:15.379798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811648+0000","last_change":"2024-05-09T03:14:53.275469+0000","last_active":"2024-05-09T03:14:54.811648+0000","last_peered":"2024-05-09T03:14:54.811648+0000","last_clean":"2024-05-09T03:14:54.811648+0000","last_became_active":"2024-05-09T03:14:53.274274+0000","last_became_peered":"2024-05-09T03:14:53.274274+0000","last_unstale":"2024-05-09T03:14:54.811648+0000","last_undegraded":"2024-05-09T03:14:54.811648+0000","last_fullsized":"2024-05-09T03:14:54.811648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:26:48.782599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265733+0000","last_change":"2024-05-09T03:14:53.259449+0000","last_active":"2024-05-09T03:14:54.265733+0000","last_peered":"2024-05-09T03:14:54.265733+0000","last_clean":"2024-05-09T03:14:54.265733+0000","last_became_active":"2024-05-09T03:14:53.259126+0000","last_became_peered":"2024-05-09T03:14:53.259126+0000","last_unstale":"2024-05-09T03:14:54.265733+0000","last_undegraded":"2024-05-09T03:14:54.265733+0000","last_fullsized":"2024-05-09T03:14:54.265733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:26:55.380010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264885+0000","last_change":"2024-05-09T03:14:53.276670+0000","last_active":"2024-05-09T03:14:54.264885+0000","last_peered":"2024-05-09T03:14:54.264885+0000","last_clean":"2024-05-09T03:14:54.264885+0000","last_became_active":"2024-05-09T03:14:53.276264+0000","last_became_peered":"2024-05-09T03:14:53.276264+0000","last_unstale":"2024-05-09T03:14:54.264885+0000","last_undegraded":"2024-05-09T03:14:54.264885+0000","last_fullsized":"2024-05-09T03:14:54.264885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:07:17.309741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266485+0000","last_change":"2024-05-09T03:14:53.271463+0000","last_active":"2024-05-09T03:14:54.266485+0000","last_peered":"2024-05-09T03:14:54.266485+0000","last_clean":"2024-05-09T03:14:54.266485+0000","last_became_active":"2024-05-09T03:14:53.271122+0000","last_became_peered":"2024-05-09T03:14:53.271122+0000","last_unstale":"2024-05-09T03:14:54.266485+0000","last_undegraded":"2024-05-09T03:14:54.266485+0000","last_fullsized":"2024-05-09T03:14:54.266485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:48:21.205930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812457+0000","last_change":"2024-05-09T03:14:53.268615+0000","last_active":"2024-05-09T03:14:54.812457+0000","last_peered":"2024-05-09T03:14:54.812457+0000","last_clean":"2024-05-09T03:14:54.812457+0000","last_became_active":"2024-05-09T03:14:53.268266+0000","last_became_peered":"2024-05-09T03:14:53.268266+0000","last_unstale":"2024-05-09T03:14:54.812457+0000","last_undegraded":"2024-05-09T03:14:54.812457+0000","last_fullsized":"2024-05-09T03:14:54.812457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:50:26.840212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811961+0000","last_change":"2024-05-09T03:14:53.274762+0000","last_active":"2024-05-09T03:14:54.811961+0000","last_peered":"2024-05-09T03:14:54.811961+0000","last_clean":"2024-05-09T03:14:54.811961+0000","last_became_active":"2024-05-09T03:14:53.274456+0000","last_became_peered":"2024-05-09T03:14:53.274456+0000","last_unstale":"2024-05-09T03:14:54.811961+0000","last_undegraded":"2024-05-09T03:14:54.811961+0000","last_fullsized":"2024-05-09T03:14:54.811961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:18.263623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774921+0000","last_change":"2024-05-09T03:14:53.271480+0000","last_active":"2024-05-09T03:14:54.774921+0000","last_peered":"2024-05-09T03:14:54.774921+0000","last_clean":"2024-05-09T03:14:54.774921+0000","last_became_active":"2024-05-09T03:14:53.271164+0000","last_became_peered":"2024-05-09T03:14:53.271164+0000","last_unstale":"2024-05-09T03:14:54.774921+0000","last_undegraded":"2024-05-09T03:14:54.774921+0000","last_fullsized":"2024-05-09T03:14:54.774921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:15:18.328807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264913+0000","last_change":"2024-05-09T03:14:53.275121+0000","last_active":"2024-05-09T03:14:54.264913+0000","last_peered":"2024-05-09T03:14:54.264913+0000","last_clean":"2024-05-09T03:14:54.264913+0000","last_became_active":"2024-05-09T03:14:53.274810+0000","last_became_peered":"2024-05-09T03:14:53.274810+0000","last_unstale":"2024-05-09T03:14:54.264913+0000","last_undegraded":"2024-05-09T03:14:54.264913+0000","last_fullsized":"2024-05-09T03:14:54.264913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:53.273549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774788+0000","last_change":"2024-05-09T03:14:53.283274+0000","last_active":"2024-05-09T03:14:54.774788+0000","last_peered":"2024-05-09T03:14:54.774788+0000","last_clean":"2024-05-09T03:14:54.774788+0000","last_became_active":"2024-05-09T03:14:53.282935+0000","last_became_peered":"2024-05-09T03:14:53.282935+0000","last_unstale":"2024-05-09T03:14:54.774788+0000","last_undegraded":"2024-05-09T03:14:54.774788+0000","last_fullsized":"2024-05-09T03:14:54.774788+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:24:23.932367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266254+0000","last_change":"2024-05-09T03:14:53.272506+0000","last_active":"2024-05-09T03:14:54.266254+0000","last_peered":"2024-05-09T03:14:54.266254+0000","last_clean":"2024-05-09T03:14:54.266254+0000","last_became_active":"2024-05-09T03:14:53.272184+0000","last_became_peered":"2024-05-09T03:14:53.272184+0000","last_unstale":"2024-05-09T03:14:54.266254+0000","last_undegraded":"2024-05-09T03:14:54.266254+0000","last_fullsized":"2024-05-09T03:14:54.266254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:15.379798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265006+0000","last_change":"2024-05-09T03:14:53.276327+0000","last_active":"2024-05-09T03:14:54.265006+0000","last_peered":"2024-05-09T03:14:54.265006+0000","last_clean":"2024-05-09T03:14:54.265006+0000","last_became_active":"2024-05-09T03:14:53.274889+0000","last_became_peered":"2024-05-09T03:14:53.274889+0000","last_unstale":"2024-05-09T03:14:54.265006+0000","last_undegraded":"2024-05-09T03:14:54.265006+0000","last_fullsized":"2024-05-09T03:14:54.265006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:38:01.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811919+0000","last_change":"2024-05-09T03:14:53.270028+0000","last_active":"2024-05-09T03:14:54.811919+0000","last_peered":"2024-05-09T03:14:54.811919+0000","last_clean":"2024-05-09T03:14:54.811919+0000","last_became_active":"2024-05-09T03:14:53.269125+0000","last_became_peered":"2024-05-09T03:14:53.269125+0000","last_unstale":"2024-05-09T03:14:54.811919+0000","last_undegraded":"2024-05-09T03:14:54.811919+0000","last_fullsized":"2024-05-09T03:14:54.811919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:22.884363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265538+0000","last_change":"2024-05-09T03:14:53.267505+0000","last_active":"2024-05-09T03:14:54.265538+0000","last_peered":"2024-05-09T03:14:54.265538+0000","last_clean":"2024-05-09T03:14:54.265538+0000","last_became_active":"2024-05-09T03:14:53.267168+0000","last_became_peered":"2024-05-09T03:14:53.267168+0000","last_unstale":"2024-05-09T03:14:54.265538+0000","last_undegraded":"2024-05-09T03:14:54.265538+0000","last_fullsized":"2024-05-09T03:14:54.265538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:45:15.835188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774931+0000","last_change":"2024-05-09T03:14:53.272305+0000","last_active":"2024-05-09T03:14:54.774931+0000","last_peered":"2024-05-09T03:14:54.774931+0000","last_clean":"2024-05-09T03:14:54.774931+0000","last_became_active":"2024-05-09T03:14:53.271931+0000","last_became_peered":"2024-05-09T03:14:53.271931+0000","last_unstale":"2024-05-09T03:14:54.774931+0000","last_undegraded":"2024-05-09T03:14:54.774931+0000","last_fullsized":"2024-05-09T03:14:54.774931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:34.680505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265537+0000","last_change":"2024-05-09T03:14:53.265337+0000","last_active":"2024-05-09T03:14:54.265537+0000","last_peered":"2024-05-09T03:14:54.265537+0000","last_clean":"2024-05-09T03:14:54.265537+0000","last_became_active":"2024-05-09T03:14:53.264422+0000","last_became_peered":"2024-05-09T03:14:53.264422+0000","last_unstale":"2024-05-09T03:14:54.265537+0000","last_undegraded":"2024-05-09T03:14:54.265537+0000","last_fullsized":"2024-05-09T03:14:54.265537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:02.232428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266112+0000","last_change":"2024-05-09T03:14:53.276316+0000","last_active":"2024-05-09T03:14:54.266112+0000","last_peered":"2024-05-09T03:14:54.266112+0000","last_clean":"2024-05-09T03:14:54.266112+0000","last_became_active":"2024-05-09T03:14:53.275969+0000","last_became_peered":"2024-05-09T03:14:53.275969+0000","last_unstale":"2024-05-09T03:14:54.266112+0000","last_undegraded":"2024-05-09T03:14:54.266112+0000","last_fullsized":"2024-05-09T03:14:54.266112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:28.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811458+0000","last_change":"2024-05-09T03:14:53.272467+0000","last_active":"2024-05-09T03:14:54.811458+0000","last_peered":"2024-05-09T03:14:54.811458+0000","last_clean":"2024-05-09T03:14:54.811458+0000","last_became_active":"2024-05-09T03:14:53.271720+0000","last_became_peered":"2024-05-09T03:14:53.271720+0000","last_unstale":"2024-05-09T03:14:54.811458+0000","last_undegraded":"2024-05-09T03:14:54.811458+0000","last_fullsized":"2024-05-09T03:14:54.811458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:34.680505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811437+0000","last_change":"2024-05-09T03:14:53.270665+0000","last_active":"2024-05-09T03:14:54.811437+0000","last_peered":"2024-05-09T03:14:54.811437+0000","last_clean":"2024-05-09T03:14:54.811437+0000","last_became_active":"2024-05-09T03:14:53.270311+0000","last_became_peered":"2024-05-09T03:14:53.270311+0000","last_unstale":"2024-05-09T03:14:54.811437+0000","last_undegraded":"2024-05-09T03:14:54.811437+0000","last_fullsized":"2024-05-09T03:14:54.811437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:15:18.328807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266487+0000","last_change":"2024-05-09T03:14:53.264638+0000","last_active":"2024-05-09T03:14:54.266487+0000","last_peered":"2024-05-09T03:14:54.266487+0000","last_clean":"2024-05-09T03:14:54.266487+0000","last_became_active":"2024-05-09T03:14:53.264303+0000","last_became_peered":"2024-05-09T03:14:53.264303+0000","last_unstale":"2024-05-09T03:14:54.266487+0000","last_undegraded":"2024-05-09T03:14:54.266487+0000","last_fullsized":"2024-05-09T03:14:54.266487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:41:12.167082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811940+0000","last_change":"2024-05-09T03:14:53.277241+0000","last_active":"2024-05-09T03:14:54.811940+0000","last_peered":"2024-05-09T03:14:54.811940+0000","last_clean":"2024-05-09T03:14:54.811940+0000","last_became_active":"2024-05-09T03:14:53.276942+0000","last_became_peered":"2024-05-09T03:14:53.276942+0000","last_unstale":"2024-05-09T03:14:54.811940+0000","last_undegraded":"2024-05-09T03:14:54.811940+0000","last_fullsized":"2024-05-09T03:14:54.811940+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:28.620101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728891+0000","last_change":"2024-05-09T03:14:53.269900+0000","last_active":"2024-05-09T03:14:54.728891+0000","last_peered":"2024-05-09T03:14:54.728891+0000","last_clean":"2024-05-09T03:14:54.728891+0000","last_became_active":"2024-05-09T03:14:53.269333+0000","last_became_peered":"2024-05-09T03:14:53.269333+0000","last_unstale":"2024-05-09T03:14:54.728891+0000","last_undegraded":"2024-05-09T03:14:54.728891+0000","last_fullsized":"2024-05-09T03:14:54.728891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:51:42.019471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265963+0000","last_change":"2024-05-09T03:14:53.276369+0000","last_active":"2024-05-09T03:14:54.265963+0000","last_peered":"2024-05-09T03:14:54.265963+0000","last_clean":"2024-05-09T03:14:54.265963+0000","last_became_active":"2024-05-09T03:14:53.276028+0000","last_became_peered":"2024-05-09T03:14:53.276028+0000","last_unstale":"2024-05-09T03:14:54.265963+0000","last_undegraded":"2024-05-09T03:14:54.265963+0000","last_fullsized":"2024-05-09T03:14:54.265963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:41.889517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728647+0000","last_change":"2024-05-09T03:14:53.271740+0000","last_active":"2024-05-09T03:14:54.728647+0000","last_peered":"2024-05-09T03:14:54.728647+0000","last_clean":"2024-05-09T03:14:54.728647+0000","last_became_active":"2024-05-09T03:14:53.271357+0000","last_became_peered":"2024-05-09T03:14:53.271357+0000","last_unstale":"2024-05-09T03:14:54.728647+0000","last_undegraded":"2024-05-09T03:14:54.728647+0000","last_fullsized":"2024-05-09T03:14:54.728647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:21:06.964110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811634+0000","last_change":"2024-05-09T03:14:53.277697+0000","last_active":"2024-05-09T03:14:54.811634+0000","last_peered":"2024-05-09T03:14:54.811634+0000","last_clean":"2024-05-09T03:14:54.811634+0000","last_became_active":"2024-05-09T03:14:53.277261+0000","last_became_peered":"2024-05-09T03:14:53.277261+0000","last_unstale":"2024-05-09T03:14:54.811634+0000","last_undegraded":"2024-05-09T03:14:54.811634+0000","last_fullsized":"2024-05-09T03:14:54.811634+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:41:12.167082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811937+0000","last_change":"2024-05-09T03:14:53.283263+0000","last_active":"2024-05-09T03:14:54.811937+0000","last_peered":"2024-05-09T03:14:54.811937+0000","last_clean":"2024-05-09T03:14:54.811937+0000","last_became_active":"2024-05-09T03:14:53.282919+0000","last_became_peered":"2024-05-09T03:14:53.282919+0000","last_unstale":"2024-05-09T03:14:54.811937+0000","last_undegraded":"2024-05-09T03:14:54.811937+0000","last_fullsized":"2024-05-09T03:14:54.811937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:58:06.949307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266075+0000","last_change":"2024-05-09T03:14:53.272407+0000","last_active":"2024-05-09T03:14:54.266075+0000","last_peered":"2024-05-09T03:14:54.266075+0000","last_clean":"2024-05-09T03:14:54.266075+0000","last_became_active":"2024-05-09T03:14:53.270109+0000","last_became_peered":"2024-05-09T03:14:53.270109+0000","last_unstale":"2024-05-09T03:14:54.266075+0000","last_undegraded":"2024-05-09T03:14:54.266075+0000","last_fullsized":"2024-05-09T03:14:54.266075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:41.889517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812413+0000","last_change":"2024-05-09T03:14:53.277547+0000","last_active":"2024-05-09T03:14:54.812413+0000","last_peered":"2024-05-09T03:14:54.812413+0000","last_clean":"2024-05-09T03:14:54.812413+0000","last_became_active":"2024-05-09T03:14:53.277127+0000","last_became_peered":"2024-05-09T03:14:53.277127+0000","last_unstale":"2024-05-09T03:14:54.812413+0000","last_undegraded":"2024-05-09T03:14:54.812413+0000","last_fullsized":"2024-05-09T03:14:54.812413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:03.889945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812615+0000","last_change":"2024-05-09T03:14:52.255267+0000","last_active":"2024-05-09T03:14:54.812615+0000","last_peered":"2024-05-09T03:14:54.812615+0000","last_clean":"2024-05-09T03:14:54.812615+0000","last_became_active":"2024-05-09T03:14:52.254837+0000","last_became_peered":"2024-05-09T03:14:52.254837+0000","last_unstale":"2024-05-09T03:14:54.812615+0000","last_undegraded":"2024-05-09T03:14:54.812615+0000","last_fullsized":"2024-05-09T03:14:54.812615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:48:20.200933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265030+0000","last_change":"2024-05-09T03:14:53.276034+0000","last_active":"2024-05-09T03:14:54.265030+0000","last_peered":"2024-05-09T03:14:54.265030+0000","last_clean":"2024-05-09T03:14:54.265030+0000","last_became_active":"2024-05-09T03:14:53.274482+0000","last_became_peered":"2024-05-09T03:14:53.274482+0000","last_unstale":"2024-05-09T03:14:54.265030+0000","last_undegraded":"2024-05-09T03:14:54.265030+0000","last_fullsized":"2024-05-09T03:14:54.265030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:16:38.626936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774991+0000","last_change":"2024-05-09T03:14:52.252662+0000","last_active":"2024-05-09T03:14:54.774991+0000","last_peered":"2024-05-09T03:14:54.774991+0000","last_clean":"2024-05-09T03:14:54.774991+0000","last_became_active":"2024-05-09T03:14:52.252309+0000","last_became_peered":"2024-05-09T03:14:52.252309+0000","last_unstale":"2024-05-09T03:14:54.774991+0000","last_undegraded":"2024-05-09T03:14:54.774991+0000","last_fullsized":"2024-05-09T03:14:54.774991+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:40:48.230068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266039+0000","last_change":"2024-05-09T03:14:53.267117+0000","last_active":"2024-05-09T03:14:54.266039+0000","last_peered":"2024-05-09T03:14:54.266039+0000","last_clean":"2024-05-09T03:14:54.266039+0000","last_became_active":"2024-05-09T03:14:53.266818+0000","last_became_peered":"2024-05-09T03:14:53.266818+0000","last_unstale":"2024-05-09T03:14:54.266039+0000","last_undegraded":"2024-05-09T03:14:54.266039+0000","last_fullsized":"2024-05-09T03:14:54.266039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:46.147440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266324+0000","last_change":"2024-05-09T03:14:52.251884+0000","last_active":"2024-05-09T03:14:54.266324+0000","last_peered":"2024-05-09T03:14:54.266324+0000","last_clean":"2024-05-09T03:14:54.266324+0000","last_became_active":"2024-05-09T03:14:52.251607+0000","last_became_peered":"2024-05-09T03:14:52.251607+0000","last_unstale":"2024-05-09T03:14:54.266324+0000","last_undegraded":"2024-05-09T03:14:54.266324+0000","last_fullsized":"2024-05-09T03:14:54.266324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:21.879366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812469+0000","last_change":"2024-05-09T03:14:53.271704+0000","last_active":"2024-05-09T03:14:54.812469+0000","last_peered":"2024-05-09T03:14:54.812469+0000","last_clean":"2024-05-09T03:14:54.812469+0000","last_became_active":"2024-05-09T03:14:53.271387+0000","last_became_peered":"2024-05-09T03:14:53.271387+0000","last_unstale":"2024-05-09T03:14:54.812469+0000","last_undegraded":"2024-05-09T03:14:54.812469+0000","last_fullsized":"2024-05-09T03:14:54.812469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:14:57.424141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.267036+0000","last_change":"2024-05-09T03:14:52.254562+0000","last_active":"2024-05-09T03:14:54.267036+0000","last_peered":"2024-05-09T03:14:54.267036+0000","last_clean":"2024-05-09T03:14:54.267036+0000","last_became_active":"2024-05-09T03:14:52.254039+0000","last_became_peered":"2024-05-09T03:14:52.254039+0000","last_unstale":"2024-05-09T03:14:54.267036+0000","last_undegraded":"2024-05-09T03:14:54.267036+0000","last_fullsized":"2024-05-09T03:14:54.267036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:49.240461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811851+0000","last_change":"2024-05-09T03:14:53.268322+0000","last_active":"2024-05-09T03:14:54.811851+0000","last_peered":"2024-05-09T03:14:54.811851+0000","last_clean":"2024-05-09T03:14:54.811851+0000","last_became_active":"2024-05-09T03:14:53.268025+0000","last_became_peered":"2024-05-09T03:14:53.268025+0000","last_unstale":"2024-05-09T03:14:54.811851+0000","last_undegraded":"2024-05-09T03:14:54.811851+0000","last_fullsized":"2024-05-09T03:14:54.811851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:06:10.182844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728711+0000","last_change":"2024-05-09T03:14:52.249626+0000","last_active":"2024-05-09T03:14:54.728711+0000","last_peered":"2024-05-09T03:14:54.728711+0000","last_clean":"2024-05-09T03:14:54.728711+0000","last_became_active":"2024-05-09T03:14:52.249292+0000","last_became_peered":"2024-05-09T03:14:52.249292+0000","last_unstale":"2024-05-09T03:14:54.728711+0000","last_undegraded":"2024-05-09T03:14:54.728711+0000","last_fullsized":"2024-05-09T03:14:54.728711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:57.522390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265713+0000","last_change":"2024-05-09T03:14:53.263866+0000","last_active":"2024-05-09T03:14:54.265713+0000","last_peered":"2024-05-09T03:14:54.265713+0000","last_clean":"2024-05-09T03:14:54.265713+0000","last_became_active":"2024-05-09T03:14:53.263536+0000","last_became_peered":"2024-05-09T03:14:53.263536+0000","last_unstale":"2024-05-09T03:14:54.265713+0000","last_undegraded":"2024-05-09T03:14:54.265713+0000","last_fullsized":"2024-05-09T03:14:54.265713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:23:46.147440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266734+0000","last_change":"2024-05-09T03:14:52.252988+0000","last_active":"2024-05-09T03:14:54.266734+0000","last_peered":"2024-05-09T03:14:54.266734+0000","last_clean":"2024-05-09T03:14:54.266734+0000","last_became_active":"2024-05-09T03:14:52.252688+0000","last_became_peered":"2024-05-09T03:14:52.252688+0000","last_unstale":"2024-05-09T03:14:54.266734+0000","last_undegraded":"2024-05-09T03:14:54.266734+0000","last_fullsized":"2024-05-09T03:14:54.266734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:59.416101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264371+0000","last_change":"2024-05-09T03:14:53.269215+0000","last_active":"2024-05-09T03:14:54.264371+0000","last_peered":"2024-05-09T03:14:54.264371+0000","last_clean":"2024-05-09T03:14:54.264371+0000","last_became_active":"2024-05-09T03:14:53.268903+0000","last_became_peered":"2024-05-09T03:14:53.268903+0000","last_unstale":"2024-05-09T03:14:54.264371+0000","last_undegraded":"2024-05-09T03:14:54.264371+0000","last_fullsized":"2024-05-09T03:14:54.264371+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:55.405229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811472+0000","last_change":"2024-05-09T03:14:52.257332+0000","last_active":"2024-05-09T03:14:54.811472+0000","last_peered":"2024-05-09T03:14:54.811472+0000","last_clean":"2024-05-09T03:14:54.811472+0000","last_became_active":"2024-05-09T03:14:52.257003+0000","last_became_peered":"2024-05-09T03:14:52.257003+0000","last_unstale":"2024-05-09T03:14:54.811472+0000","last_undegraded":"2024-05-09T03:14:54.811472+0000","last_fullsized":"2024-05-09T03:14:54.811472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:49:24.627520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774937+0000","last_change":"2024-05-09T03:14:53.262750+0000","last_active":"2024-05-09T03:14:54.774937+0000","last_peered":"2024-05-09T03:14:54.774937+0000","last_clean":"2024-05-09T03:14:54.774937+0000","last_became_active":"2024-05-09T03:14:53.262360+0000","last_became_peered":"2024-05-09T03:14:53.262360+0000","last_unstale":"2024-05-09T03:14:54.774937+0000","last_undegraded":"2024-05-09T03:14:54.774937+0000","last_fullsized":"2024-05-09T03:14:54.774937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:15.398527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264320+0000","last_change":"2024-05-09T03:14:52.259321+0000","last_active":"2024-05-09T03:14:54.264320+0000","last_peered":"2024-05-09T03:14:54.264320+0000","last_clean":"2024-05-09T03:14:54.264320+0000","last_became_active":"2024-05-09T03:14:52.258674+0000","last_became_peered":"2024-05-09T03:14:52.258674+0000","last_unstale":"2024-05-09T03:14:54.264320+0000","last_undegraded":"2024-05-09T03:14:54.264320+0000","last_fullsized":"2024-05-09T03:14:54.264320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:17:05.234840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812094+0000","last_change":"2024-05-09T03:14:53.269779+0000","last_active":"2024-05-09T03:14:54.812094+0000","last_peered":"2024-05-09T03:14:54.812094+0000","last_clean":"2024-05-09T03:14:54.812094+0000","last_became_active":"2024-05-09T03:14:53.269452+0000","last_became_peered":"2024-05-09T03:14:53.269452+0000","last_unstale":"2024-05-09T03:14:54.812094+0000","last_undegraded":"2024-05-09T03:14:54.812094+0000","last_fullsized":"2024-05-09T03:14:54.812094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:03.901635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266482+0000","last_change":"2024-05-09T03:14:52.253645+0000","last_active":"2024-05-09T03:14:54.266482+0000","last_peered":"2024-05-09T03:14:54.266482+0000","last_clean":"2024-05-09T03:14:54.266482+0000","last_became_active":"2024-05-09T03:14:52.253286+0000","last_became_peered":"2024-05-09T03:14:52.253286+0000","last_unstale":"2024-05-09T03:14:54.266482+0000","last_undegraded":"2024-05-09T03:14:54.266482+0000","last_fullsized":"2024-05-09T03:14:54.266482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:31.423280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812441+0000","last_change":"2024-05-09T03:14:53.277762+0000","last_active":"2024-05-09T03:14:54.812441+0000","last_peered":"2024-05-09T03:14:54.812441+0000","last_clean":"2024-05-09T03:14:54.812441+0000","last_became_active":"2024-05-09T03:14:53.277448+0000","last_became_peered":"2024-05-09T03:14:53.277448+0000","last_unstale":"2024-05-09T03:14:54.812441+0000","last_undegraded":"2024-05-09T03:14:54.812441+0000","last_fullsized":"2024-05-09T03:14:54.812441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:22.285256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266609+0000","last_change":"2024-05-09T03:14:52.256256+0000","last_active":"2024-05-09T03:14:54.266609+0000","last_peered":"2024-05-09T03:14:54.266609+0000","last_clean":"2024-05-09T03:14:54.266609+0000","last_became_active":"2024-05-09T03:14:52.255933+0000","last_became_peered":"2024-05-09T03:14:52.255933+0000","last_unstale":"2024-05-09T03:14:54.266609+0000","last_undegraded":"2024-05-09T03:14:54.266609+0000","last_fullsized":"2024-05-09T03:14:54.266609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:13:44.943490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266957+0000","last_change":"2024-05-09T03:14:53.271770+0000","last_active":"2024-05-09T03:14:54.266957+0000","last_peered":"2024-05-09T03:14:54.266957+0000","last_clean":"2024-05-09T03:14:54.266957+0000","last_became_active":"2024-05-09T03:14:53.271415+0000","last_became_peered":"2024-05-09T03:14:53.271415+0000","last_unstale":"2024-05-09T03:14:54.266957+0000","last_undegraded":"2024-05-09T03:14:54.266957+0000","last_fullsized":"2024-05-09T03:14:54.266957+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:48:21.205930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728954+0000","last_change":"2024-05-09T03:14:52.250488+0000","last_active":"2024-05-09T03:14:54.728954+0000","last_peered":"2024-05-09T03:14:54.728954+0000","last_clean":"2024-05-09T03:14:54.728954+0000","last_became_active":"2024-05-09T03:14:52.250163+0000","last_became_peered":"2024-05-09T03:14:52.250163+0000","last_unstale":"2024-05-09T03:14:54.728954+0000","last_undegraded":"2024-05-09T03:14:54.728954+0000","last_fullsized":"2024-05-09T03:14:54.728954+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:14.393530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264771+0000","last_change":"2024-05-09T03:14:53.272479+0000","last_active":"2024-05-09T03:14:54.264771+0000","last_peered":"2024-05-09T03:14:54.264771+0000","last_clean":"2024-05-09T03:14:54.264771+0000","last_became_active":"2024-05-09T03:14:53.271902+0000","last_became_peered":"2024-05-09T03:14:53.271902+0000","last_unstale":"2024-05-09T03:14:54.264771+0000","last_undegraded":"2024-05-09T03:14:54.264771+0000","last_fullsized":"2024-05-09T03:14:54.264771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:41:16.554452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812009+0000","last_change":"2024-05-09T03:14:52.257050+0000","last_active":"2024-05-09T03:14:54.812009+0000","last_peered":"2024-05-09T03:14:54.812009+0000","last_clean":"2024-05-09T03:14:54.812009+0000","last_became_active":"2024-05-09T03:14:52.256714+0000","last_became_peered":"2024-05-09T03:14:52.256714+0000","last_unstale":"2024-05-09T03:14:54.812009+0000","last_undegraded":"2024-05-09T03:14:54.812009+0000","last_fullsized":"2024-05-09T03:14:54.812009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:39:49.240461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812056+0000","last_change":"2024-05-09T03:14:53.275165+0000","last_active":"2024-05-09T03:14:54.812056+0000","last_peered":"2024-05-09T03:14:54.812056+0000","last_clean":"2024-05-09T03:14:54.812056+0000","last_became_active":"2024-05-09T03:14:53.274855+0000","last_became_peered":"2024-05-09T03:14:53.274855+0000","last_unstale":"2024-05-09T03:14:54.812056+0000","last_undegraded":"2024-05-09T03:14:54.812056+0000","last_fullsized":"2024-05-09T03:14:54.812056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:04:15.398527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811653+0000","last_change":"2024-05-09T03:14:52.249673+0000","last_active":"2024-05-09T03:14:54.811653+0000","last_peered":"2024-05-09T03:14:54.811653+0000","last_clean":"2024-05-09T03:14:54.811653+0000","last_became_active":"2024-05-09T03:14:52.249364+0000","last_became_peered":"2024-05-09T03:14:52.249364+0000","last_unstale":"2024-05-09T03:14:54.811653+0000","last_undegraded":"2024-05-09T03:14:54.811653+0000","last_fullsized":"2024-05-09T03:14:54.811653+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:37:35.230340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812288+0000","last_change":"2024-05-09T03:14:53.271261+0000","last_active":"2024-05-09T03:14:54.812288+0000","last_peered":"2024-05-09T03:14:54.812288+0000","last_clean":"2024-05-09T03:14:54.812288+0000","last_became_active":"2024-05-09T03:14:53.270916+0000","last_became_peered":"2024-05-09T03:14:53.270916+0000","last_unstale":"2024-05-09T03:14:54.812288+0000","last_undegraded":"2024-05-09T03:14:54.812288+0000","last_fullsized":"2024-05-09T03:14:54.812288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:00:00.455176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266880+0000","last_change":"2024-05-09T03:14:52.254481+0000","last_active":"2024-05-09T03:14:54.266880+0000","last_peered":"2024-05-09T03:14:54.266880+0000","last_clean":"2024-05-09T03:14:54.266880+0000","last_became_active":"2024-05-09T03:14:52.254210+0000","last_became_peered":"2024-05-09T03:14:52.254210+0000","last_unstale":"2024-05-09T03:14:54.266880+0000","last_undegraded":"2024-05-09T03:14:54.266880+0000","last_fullsized":"2024-05-09T03:14:54.266880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:03:00.521965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774714+0000","last_change":"2024-05-09T03:14:53.270787+0000","last_active":"2024-05-09T03:14:54.774714+0000","last_peered":"2024-05-09T03:14:54.774714+0000","last_clean":"2024-05-09T03:14:54.774714+0000","last_became_active":"2024-05-09T03:14:53.270416+0000","last_became_peered":"2024-05-09T03:14:53.270416+0000","last_unstale":"2024-05-09T03:14:54.774714+0000","last_undegraded":"2024-05-09T03:14:54.774714+0000","last_fullsized":"2024-05-09T03:14:54.774714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:08.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812111+0000","last_change":"2024-05-09T03:14:52.258697+0000","last_active":"2024-05-09T03:14:54.812111+0000","last_peered":"2024-05-09T03:14:54.812111+0000","last_clean":"2024-05-09T03:14:54.812111+0000","last_became_active":"2024-05-09T03:14:52.258385+0000","last_became_peered":"2024-05-09T03:14:52.258385+0000","last_unstale":"2024-05-09T03:14:54.812111+0000","last_undegraded":"2024-05-09T03:14:54.812111+0000","last_fullsized":"2024-05-09T03:14:54.812111+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:44:44.758894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264558+0000","last_change":"2024-05-09T03:14:53.267557+0000","last_active":"2024-05-09T03:14:54.264558+0000","last_peered":"2024-05-09T03:14:54.264558+0000","last_clean":"2024-05-09T03:14:54.264558+0000","last_became_active":"2024-05-09T03:14:53.267177+0000","last_became_peered":"2024-05-09T03:14:53.267177+0000","last_unstale":"2024-05-09T03:14:54.264558+0000","last_undegraded":"2024-05-09T03:14:54.264558+0000","last_fullsized":"2024-05-09T03:14:54.264558+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:32:40.343939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266050+0000","last_change":"2024-05-09T03:14:52.249521+0000","last_active":"2024-05-09T03:14:54.266050+0000","last_peered":"2024-05-09T03:14:54.266050+0000","last_clean":"2024-05-09T03:14:54.266050+0000","last_became_active":"2024-05-09T03:14:52.248820+0000","last_became_peered":"2024-05-09T03:14:52.248820+0000","last_unstale":"2024-05-09T03:14:54.266050+0000","last_undegraded":"2024-05-09T03:14:54.266050+0000","last_fullsized":"2024-05-09T03:14:54.266050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:42.690879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266519+0000","last_change":"2024-05-09T03:14:53.265021+0000","last_active":"2024-05-09T03:14:54.266519+0000","last_peered":"2024-05-09T03:14:54.266519+0000","last_clean":"2024-05-09T03:14:54.266519+0000","last_became_active":"2024-05-09T03:14:53.264694+0000","last_became_peered":"2024-05-09T03:14:53.264694+0000","last_unstale":"2024-05-09T03:14:54.266519+0000","last_undegraded":"2024-05-09T03:14:54.266519+0000","last_fullsized":"2024-05-09T03:14:54.266519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:26:48.782599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266159+0000","last_change":"2024-05-09T03:14:52.249953+0000","last_active":"2024-05-09T03:14:54.266159+0000","last_peered":"2024-05-09T03:14:54.266159+0000","last_clean":"2024-05-09T03:14:54.266159+0000","last_became_active":"2024-05-09T03:14:52.249621+0000","last_became_peered":"2024-05-09T03:14:52.249621+0000","last_unstale":"2024-05-09T03:14:54.266159+0000","last_undegraded":"2024-05-09T03:14:54.266159+0000","last_fullsized":"2024-05-09T03:14:54.266159+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:00:23.059773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812578+0000","last_change":"2024-05-09T03:14:53.274470+0000","last_active":"2024-05-09T03:14:54.812578+0000","last_peered":"2024-05-09T03:14:54.812578+0000","last_clean":"2024-05-09T03:14:54.812578+0000","last_became_active":"2024-05-09T03:14:53.274160+0000","last_became_peered":"2024-05-09T03:14:53.274160+0000","last_unstale":"2024-05-09T03:14:54.812578+0000","last_undegraded":"2024-05-09T03:14:54.812578+0000","last_fullsized":"2024-05-09T03:14:54.812578+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:20.233102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266544+0000","last_change":"2024-05-09T03:14:52.251850+0000","last_active":"2024-05-09T03:14:54.266544+0000","last_peered":"2024-05-09T03:14:54.266544+0000","last_clean":"2024-05-09T03:14:54.266544+0000","last_became_active":"2024-05-09T03:14:52.251556+0000","last_became_peered":"2024-05-09T03:14:52.251556+0000","last_unstale":"2024-05-09T03:14:54.266544+0000","last_undegraded":"2024-05-09T03:14:54.266544+0000","last_fullsized":"2024-05-09T03:14:54.266544+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:18:20.434159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811964+0000","last_change":"2024-05-09T03:14:53.270950+0000","last_active":"2024-05-09T03:14:54.811964+0000","last_peered":"2024-05-09T03:14:54.811964+0000","last_clean":"2024-05-09T03:14:54.811964+0000","last_became_active":"2024-05-09T03:14:53.270649+0000","last_became_peered":"2024-05-09T03:14:53.270649+0000","last_unstale":"2024-05-09T03:14:54.811964+0000","last_undegraded":"2024-05-09T03:14:54.811964+0000","last_fullsized":"2024-05-09T03:14:54.811964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:54:43.695876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265960+0000","last_change":"2024-05-09T03:14:52.255286+0000","last_active":"2024-05-09T03:14:54.265960+0000","last_peered":"2024-05-09T03:14:54.265960+0000","last_clean":"2024-05-09T03:14:54.265960+0000","last_became_active":"2024-05-09T03:14:52.255006+0000","last_became_peered":"2024-05-09T03:14:52.255006+0000","last_unstale":"2024-05-09T03:14:54.265960+0000","last_undegraded":"2024-05-09T03:14:54.265960+0000","last_fullsized":"2024-05-09T03:14:54.265960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:08.938961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264869+0000","last_change":"2024-05-09T03:14:53.272215+0000","last_active":"2024-05-09T03:14:54.264869+0000","last_peered":"2024-05-09T03:14:54.264869+0000","last_clean":"2024-05-09T03:14:54.264869+0000","last_became_active":"2024-05-09T03:14:53.271485+0000","last_became_peered":"2024-05-09T03:14:53.271485+0000","last_unstale":"2024-05-09T03:14:54.264869+0000","last_undegraded":"2024-05-09T03:14:54.264869+0000","last_fullsized":"2024-05-09T03:14:54.264869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:32:38.571666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812254+0000","last_change":"2024-05-09T03:14:52.258316+0000","last_active":"2024-05-09T03:14:54.812254+0000","last_peered":"2024-05-09T03:14:54.812254+0000","last_clean":"2024-05-09T03:14:54.812254+0000","last_became_active":"2024-05-09T03:14:52.257992+0000","last_became_peered":"2024-05-09T03:14:52.257992+0000","last_unstale":"2024-05-09T03:14:54.812254+0000","last_undegraded":"2024-05-09T03:14:54.812254+0000","last_fullsized":"2024-05-09T03:14:54.812254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:14.374801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728639+0000","last_change":"2024-05-09T03:14:53.281183+0000","last_active":"2024-05-09T03:14:54.728639+0000","last_peered":"2024-05-09T03:14:54.728639+0000","last_clean":"2024-05-09T03:14:54.728639+0000","last_became_active":"2024-05-09T03:14:53.280857+0000","last_became_peered":"2024-05-09T03:14:53.280857+0000","last_unstale":"2024-05-09T03:14:54.728639+0000","last_undegraded":"2024-05-09T03:14:54.728639+0000","last_fullsized":"2024-05-09T03:14:54.728639+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:18:13.043237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266885+0000","last_change":"2024-05-09T03:14:52.248137+0000","last_active":"2024-05-09T03:14:54.266885+0000","last_peered":"2024-05-09T03:14:54.266885+0000","last_clean":"2024-05-09T03:14:54.266885+0000","last_became_active":"2024-05-09T03:14:52.247419+0000","last_became_peered":"2024-05-09T03:14:52.247419+0000","last_unstale":"2024-05-09T03:14:54.266885+0000","last_undegraded":"2024-05-09T03:14:54.266885+0000","last_fullsized":"2024-05-09T03:14:54.266885+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:21.879366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728576+0000","last_change":"2024-05-09T03:14:53.277405+0000","last_active":"2024-05-09T03:14:54.728576+0000","last_peered":"2024-05-09T03:14:54.728576+0000","last_clean":"2024-05-09T03:14:54.728576+0000","last_became_active":"2024-05-09T03:14:53.277047+0000","last_became_peered":"2024-05-09T03:14:53.277047+0000","last_unstale":"2024-05-09T03:14:54.728576+0000","last_undegraded":"2024-05-09T03:14:54.728576+0000","last_fullsized":"2024-05-09T03:14:54.728576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:17:31.032365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812009+0000","last_change":"2024-05-09T03:14:52.260111+0000","last_active":"2024-05-09T03:14:54.812009+0000","last_peered":"2024-05-09T03:14:54.812009+0000","last_clean":"2024-05-09T03:14:54.812009+0000","last_became_active":"2024-05-09T03:14:52.259816+0000","last_became_peered":"2024-05-09T03:14:52.259816+0000","last_unstale":"2024-05-09T03:14:54.812009+0000","last_undegraded":"2024-05-09T03:14:54.812009+0000","last_fullsized":"2024-05-09T03:14:54.812009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:44.453397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728601+0000","last_change":"2024-05-09T03:14:53.279110+0000","last_active":"2024-05-09T03:14:54.728601+0000","last_peered":"2024-05-09T03:14:54.728601+0000","last_clean":"2024-05-09T03:14:54.728601+0000","last_became_active":"2024-05-09T03:14:53.278482+0000","last_became_peered":"2024-05-09T03:14:53.278482+0000","last_unstale":"2024-05-09T03:14:54.728601+0000","last_undegraded":"2024-05-09T03:14:54.728601+0000","last_fullsized":"2024-05-09T03:14:54.728601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:52:03.704231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774524+0000","last_change":"2024-05-09T03:14:52.252673+0000","last_active":"2024-05-09T03:14:54.774524+0000","last_peered":"2024-05-09T03:14:54.774524+0000","last_clean":"2024-05-09T03:14:54.774524+0000","last_became_active":"2024-05-09T03:14:52.252297+0000","last_became_peered":"2024-05-09T03:14:52.252297+0000","last_unstale":"2024-05-09T03:14:54.774524+0000","last_undegraded":"2024-05-09T03:14:54.774524+0000","last_fullsized":"2024-05-09T03:14:54.774524+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:41:11.162085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.774955+0000","last_change":"2024-05-09T03:14:53.276368+0000","last_active":"2024-05-09T03:14:54.774955+0000","last_peered":"2024-05-09T03:14:54.774955+0000","last_clean":"2024-05-09T03:14:54.774955+0000","last_became_active":"2024-05-09T03:14:53.276066+0000","last_became_peered":"2024-05-09T03:14:53.276066+0000","last_unstale":"2024-05-09T03:14:54.774955+0000","last_undegraded":"2024-05-09T03:14:54.774955+0000","last_fullsized":"2024-05-09T03:14:54.774955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:58.527388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.266332+0000","last_change":"2024-05-09T03:14:52.250637+0000","last_active":"2024-05-09T03:14:54.266332+0000","last_peered":"2024-05-09T03:14:54.266332+0000","last_clean":"2024-05-09T03:14:54.266332+0000","last_became_active":"2024-05-09T03:14:52.250335+0000","last_became_peered":"2024-05-09T03:14:52.250335+0000","last_unstale":"2024-05-09T03:14:54.266332+0000","last_undegraded":"2024-05-09T03:14:54.266332+0000","last_fullsized":"2024-05-09T03:14:54.266332+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:52:58.910513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811766+0000","last_change":"2024-05-09T03:14:53.274319+0000","last_active":"2024-05-09T03:14:54.811766+0000","last_peered":"2024-05-09T03:14:54.811766+0000","last_clean":"2024-05-09T03:14:54.811766+0000","last_became_active":"2024-05-09T03:14:53.273813+0000","last_became_peered":"2024-05-09T03:14:53.273813+0000","last_unstale":"2024-05-09T03:14:54.811766+0000","last_undegraded":"2024-05-09T03:14:54.811766+0000","last_fullsized":"2024-05-09T03:14:54.811766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:40:49.235065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812077+0000","last_change":"2024-05-09T03:14:52.254666+0000","last_active":"2024-05-09T03:14:54.812077+0000","last_peered":"2024-05-09T03:14:54.812077+0000","last_clean":"2024-05-09T03:14:54.812077+0000","last_became_active":"2024-05-09T03:14:52.254344+0000","last_became_peered":"2024-05-09T03:14:52.254344+0000","last_unstale":"2024-05-09T03:14:54.812077+0000","last_undegraded":"2024-05-09T03:14:54.812077+0000","last_fullsized":"2024-05-09T03:14:54.812077+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:03:44.617588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812232+0000","last_change":"2024-05-09T03:14:53.277535+0000","last_active":"2024-05-09T03:14:54.812232+0000","last_peered":"2024-05-09T03:14:54.812232+0000","last_clean":"2024-05-09T03:14:54.812232+0000","last_became_active":"2024-05-09T03:14:53.277140+0000","last_became_peered":"2024-05-09T03:14:53.277140+0000","last_unstale":"2024-05-09T03:14:54.812232+0000","last_undegraded":"2024-05-09T03:14:54.812232+0000","last_fullsized":"2024-05-09T03:14:54.812232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:53:58.527388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264829+0000","last_change":"2024-05-09T03:14:52.253298+0000","last_active":"2024-05-09T03:14:54.264829+0000","last_peered":"2024-05-09T03:14:54.264829+0000","last_clean":"2024-05-09T03:14:54.264829+0000","last_became_active":"2024-05-09T03:14:52.252924+0000","last_became_peered":"2024-05-09T03:14:52.252924+0000","last_unstale":"2024-05-09T03:14:54.264829+0000","last_undegraded":"2024-05-09T03:14:54.264829+0000","last_fullsized":"2024-05-09T03:14:54.264829+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:57.659379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.812591+0000","last_change":"2024-05-09T03:14:53.271786+0000","last_active":"2024-05-09T03:14:54.812591+0000","last_peered":"2024-05-09T03:14:54.812591+0000","last_clean":"2024-05-09T03:14:54.812591+0000","last_became_active":"2024-05-09T03:14:53.271482+0000","last_became_peered":"2024-05-09T03:14:53.271482+0000","last_unstale":"2024-05-09T03:14:54.812591+0000","last_undegraded":"2024-05-09T03:14:54.812591+0000","last_fullsized":"2024-05-09T03:14:54.812591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:21.029633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.775004+0000","last_change":"2024-05-09T03:14:52.247892+0000","last_active":"2024-05-09T03:14:54.775004+0000","last_peered":"2024-05-09T03:14:54.775004+0000","last_clean":"2024-05-09T03:14:54.775004+0000","last_became_active":"2024-05-09T03:14:52.247532+0000","last_became_peered":"2024-05-09T03:14:52.247532+0000","last_unstale":"2024-05-09T03:14:54.775004+0000","last_undegraded":"2024-05-09T03:14:54.775004+0000","last_fullsized":"2024-05-09T03:14:54.775004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:26:47.777602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264913+0000","last_change":"2024-05-09T03:14:53.267608+0000","last_active":"2024-05-09T03:14:54.264913+0000","last_peered":"2024-05-09T03:14:54.264913+0000","last_clean":"2024-05-09T03:14:54.264913+0000","last_became_active":"2024-05-09T03:14:53.267287+0000","last_became_peered":"2024-05-09T03:14:53.267287+0000","last_unstale":"2024-05-09T03:14:54.264913+0000","last_undegraded":"2024-05-09T03:14:54.264913+0000","last_fullsized":"2024-05-09T03:14:54.264913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:15:22.300992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.264429+0000","last_change":"2024-05-09T03:14:52.252354+0000","last_active":"2024-05-09T03:14:54.264429+0000","last_peered":"2024-05-09T03:14:54.264429+0000","last_clean":"2024-05-09T03:14:54.264429+0000","last_became_active":"2024-05-09T03:14:52.252020+0000","last_became_peered":"2024-05-09T03:14:52.252020+0000","last_unstale":"2024-05-09T03:14:54.264429+0000","last_undegraded":"2024-05-09T03:14:54.264429+0000","last_fullsized":"2024-05-09T03:14:54.264429+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:00:39.937419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265681+0000","last_change":"2024-05-09T03:14:53.269169+0000","last_active":"2024-05-09T03:14:54.265681+0000","last_peered":"2024-05-09T03:14:54.265681+0000","last_clean":"2024-05-09T03:14:54.265681+0000","last_became_active":"2024-05-09T03:14:53.268657+0000","last_became_peered":"2024-05-09T03:14:53.268657+0000","last_unstale":"2024-05-09T03:14:54.265681+0000","last_undegraded":"2024-05-09T03:14:54.265681+0000","last_fullsized":"2024-05-09T03:14:54.265681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:37:46.502228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811451+0000","last_change":"2024-05-09T03:14:52.254279+0000","last_active":"2024-05-09T03:14:54.811451+0000","last_peered":"2024-05-09T03:14:54.811451+0000","last_clean":"2024-05-09T03:14:54.811451+0000","last_became_active":"2024-05-09T03:14:52.253906+0000","last_became_peered":"2024-05-09T03:14:52.253906+0000","last_unstale":"2024-05-09T03:14:54.811451+0000","last_undegraded":"2024-05-09T03:14:54.811451+0000","last_fullsized":"2024-05-09T03:14:54.811451+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:07:19.253833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811683+0000","last_change":"2024-05-09T03:14:53.269228+0000","last_active":"2024-05-09T03:14:54.811683+0000","last_peered":"2024-05-09T03:14:54.811683+0000","last_clean":"2024-05-09T03:14:54.811683+0000","last_became_active":"2024-05-09T03:14:53.268877+0000","last_became_peered":"2024-05-09T03:14:53.268877+0000","last_unstale":"2024-05-09T03:14:54.811683+0000","last_undegraded":"2024-05-09T03:14:54.811683+0000","last_fullsized":"2024-05-09T03:14:54.811683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:52:00.421098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265331+0000","last_change":"2024-05-09T03:14:52.250766+0000","last_active":"2024-05-09T03:14:54.265331+0000","last_peered":"2024-05-09T03:14:54.265331+0000","last_clean":"2024-05-09T03:14:54.265331+0000","last_became_active":"2024-05-09T03:14:52.250471+0000","last_became_peered":"2024-05-09T03:14:52.250471+0000","last_unstale":"2024-05-09T03:14:54.265331+0000","last_undegraded":"2024-05-09T03:14:54.265331+0000","last_fullsized":"2024-05-09T03:14:54.265331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:17.258626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265854+0000","last_change":"2024-05-09T03:14:53.267693+0000","last_active":"2024-05-09T03:14:54.265854+0000","last_peered":"2024-05-09T03:14:54.265854+0000","last_clean":"2024-05-09T03:14:54.265854+0000","last_became_active":"2024-05-09T03:14:53.267324+0000","last_became_peered":"2024-05-09T03:14:53.267324+0000","last_unstale":"2024-05-09T03:14:54.265854+0000","last_undegraded":"2024-05-09T03:14:54.265854+0000","last_fullsized":"2024-05-09T03:14:54.265854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:02.232428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265812+0000","last_change":"2024-05-09T03:14:52.248455+0000","last_active":"2024-05-09T03:14:54.265812+0000","last_peered":"2024-05-09T03:14:54.265812+0000","last_clean":"2024-05-09T03:14:54.265812+0000","last_became_active":"2024-05-09T03:14:52.248157+0000","last_became_peered":"2024-05-09T03:14:52.248157+0000","last_unstale":"2024-05-09T03:14:54.265812+0000","last_undegraded":"2024-05-09T03:14:54.265812+0000","last_fullsized":"2024-05-09T03:14:54.265812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:02:31.062201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.728714+0000","last_change":"2024-05-09T03:14:53.278774+0000","last_active":"2024-05-09T03:14:54.728714+0000","last_peered":"2024-05-09T03:14:54.728714+0000","last_clean":"2024-05-09T03:14:54.728714+0000","last_became_active":"2024-05-09T03:14:53.278269+0000","last_became_peered":"2024-05-09T03:14:53.278269+0000","last_unstale":"2024-05-09T03:14:54.728714+0000","last_undegraded":"2024-05-09T03:14:54.728714+0000","last_fullsized":"2024-05-09T03:14:54.728714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:52.229696+0000","last_clean_scrub_stamp":"2024-05-09T03:14:52.229696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:29:26.854313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.265434+0000","last_change":"2024-05-09T03:14:52.249297+0000","last_active":"2024-05-09T03:14:54.265434+0000","last_peered":"2024-05-09T03:14:54.265434+0000","last_clean":"2024-05-09T03:14:54.265434+0000","last_became_active":"2024-05-09T03:14:52.248966+0000","last_became_peered":"2024-05-09T03:14:52.248966+0000","last_unstale":"2024-05-09T03:14:54.265434+0000","last_undegraded":"2024-05-09T03:14:54.265434+0000","last_fullsized":"2024-05-09T03:14:54.265434+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:14:51.224699+0000","last_clean_scrub_stamp":"2024-05-09T03:14:51.224699+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:25:34.997223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":199,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-05-09T03:14:54.811400+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:14:54.811400+0000","last_peered":"2024-05-09T03:14:54.811400+0000","last_clean":"2024-05-09T03:14:54.811400+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:14:54.811400+0000","last_undegraded":"2024-05-09T03:14:54.811400+0000","last_fullsized":"2024-05-09T03:14:54.811400+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":940760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":835584,"data_stored":349868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":349866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":349846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":835584,"data_stored":349888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574918,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":839680,"data_stored":351388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32008,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339832,"statfs":{"total":96636764160,"available":96603987968,"internally_reserved":0,"allocated":847872,"data_stored":352004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":940144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:14:57.733 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:14:57.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T03:14:57.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.890+0000 7fecc4176640 1 Processor -- start 2024-05-09T03:14:57.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.890+0000 7fecc4176640 1 -- start start 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc4176640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0d54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc4176640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fecbc160440 0x7fecbc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc4176640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fecbc0b7ca0 0x7fecbc0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc4176640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fecbc0b80b0 con 0x7fecbc160440 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc4176640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fecbc0d0c30 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc4176640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc0d0f10 con 0x7fecbc0b4a00 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fecbc160440 0x7fecbc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fecbc160440 0x7fecbc0b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41118/0 (socket says 172.21.15.46:41118) 2024-05-09T03:14:57.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 -- 172.21.15.46:0/547701430 learned_addr learned my addr 172.21.15.46:0/547701430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc2713640 1 --2- 172.21.15.46:0/547701430 >> v2:172.21.15.88:3300/0 conn(0x7fecbc0b7ca0 0x7fecbc0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1f12640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0d54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 -- 172.21.15.46:0/547701430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 msgr2=0x7fecbc0d54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0d54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 -- 172.21.15.46:0/547701430 >> v2:172.21.15.88:3300/0 conn(0x7fecbc0b7ca0 msgr2=0x7fecbc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 --2- 172.21.15.46:0/547701430 >> v2:172.21.15.88:3300/0 conn(0x7fecbc0b7ca0 0x7fecbc0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1711640 1 -- 172.21.15.46:0/547701430 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc0d11f0 con 0x7fecbc160440 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.892+0000 7fecc1f12640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0d54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.893+0000 7fecc1711640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc160440 0x7fecbc0b7760 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0b8830 tx=0x7fecb002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77049b35c2e9fae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.893+0000 7fecc0f10640 1 -- 172.21.15.46:0/547701430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0037070 con 0x7fecbc160440 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.893+0000 7fecc0f10640 1 -- 172.21.15.46:0/547701430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb0032440 con 0x7fecbc160440 2024-05-09T03:14:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.893+0000 7fecc0f10640 1 -- 172.21.15.46:0/547701430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0032810 con 0x7fecbc160440 2024-05-09T03:14:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 -- 172.21.15.46:0/547701430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc160440 msgr2=0x7fecbc0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc160440 0x7fecbc0b7760 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0b8830 tx=0x7fecb002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 -- 172.21.15.46:0/547701430 shutdown_connections 2024-05-09T03:14:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 --2- 172.21.15.46:0/547701430 >> v2:172.21.15.88:3300/0 conn(0x7fecbc0b7ca0 0x7fecbc0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc160440 0x7fecbc0b7760 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 --2- 172.21.15.46:0/547701430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0d54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 -- 172.21.15.46:0/547701430 >> 172.21.15.46:0/547701430 conn(0x7fecbc0743e0 msgr2=0x7fecbc0d2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 -- 172.21.15.46:0/547701430 shutdown_connections 2024-05-09T03:14:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.894+0000 7fecc4176640 1 -- 172.21.15.46:0/547701430 wait complete. 2024-05-09T03:14:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.896+0000 7fecc4176640 1 Processor -- start 2024-05-09T03:14:57.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.896+0000 7fecc4176640 1 -- start start 2024-05-09T03:14:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc4176640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc4176640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 0x7fecbc14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc4176640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecbc160440 0x7fecbc0b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc4176640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc0d19f0 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc4176640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc0b5200 con 0x7fecbc160440 2024-05-09T03:14:57.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc4176640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc150250 con 0x7fecbc0b4a00 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc1711640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 0x7fecbc14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc1711640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 0x7fecbc14c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41130/0 (socket says 172.21.15.46:41130) 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.897+0000 7fecc1711640 1 -- 172.21.15.46:0/3610741508 learned_addr learned my addr 172.21.15.46:0/3610741508 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc2713640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecbc160440 0x7fecbc0b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1f12640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1711640 1 -- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 msgr2=0x7fecbc0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1711640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1711640 1 -- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecbc160440 msgr2=0x7fecbc0b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1711640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecbc160440 0x7fecbc0b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1711640 1 -- 172.21.15.46:0/3610741508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc14fd80 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc1711640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 0x7fecbc14c050 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fecb002a220 tx=0x7fecb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0037040 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb0042070 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb00308a0 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc0c9f40 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.898+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc162960 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.899+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecbc0c9cb0 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.900+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fecb0030a40 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.901+0000 7fecba7fc640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feca80762d0 0x7feca80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:57.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.901+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fecb00b7dd0 con 0x7fecbc0b7ca0 2024-05-09T03:14:57.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.901+0000 7fecc1f12640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feca80762d0 0x7feca80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:57.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.902+0000 7fecc1f12640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feca80762d0 0x7feca80787e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fecbc08e2c0 tx=0x7fecb4004730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:57.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:57.906+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fecb00810e0 con 0x7fecbc0b7ca0 2024-05-09T03:14:58.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.169+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fecbc06a270 con 0x7fecbc0b7ca0 2024-05-09T03:14:58.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.171+0000 7fecba7fc640 1 -- 172.21.15.46:0/3610741508 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7fecb00853e0 con 0x7fecbc0b7ca0 2024-05-09T03:14:58.172 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:14:58.172 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:14:58.172 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-05-09T03:14:58.172 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-05-09T03:14:58.172 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T03:14:58.172 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-05-09T03:14:58.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feca80762d0 msgr2=0x7feca80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feca80762d0 0x7feca80787e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fecbc08e2c0 tx=0x7fecb4004730 comp rx=0 tx=0).stop 2024-05-09T03:14:58.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 msgr2=0x7fecbc14c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 0x7fecbc14c050 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fecb002a220 tx=0x7fecb00046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 shutdown_connections 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecbc160440 0x7fecbc0b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feca80762d0 0x7feca80787e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecbc0b7ca0 0x7fecbc14c050 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 --2- 172.21.15.46:0/3610741508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecbc0b4a00 0x7fecbc0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 >> 172.21.15.46:0/3610741508 conn(0x7fecbc0743e0 msgr2=0x7fecbc0b92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.173+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 shutdown_connections 2024-05-09T03:14:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.174+0000 7fecc4176640 1 -- 172.21.15.46:0/3610741508 wait complete. 2024-05-09T03:14:58.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:14:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.300+0000 7f92577ce640 1 Processor -- start 2024-05-09T03:14:58.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.301+0000 7f92577ce640 1 -- start start 2024-05-09T03:14:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f92577ce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92501615f0 0x7f92500b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:58.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f92577ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f92500d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f92577ce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92500d15b0 con 0x7f92500d0b80 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f92577ce640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9250160b10 con 0x7f92501615f0 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f92577ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9250160df0 con 0x7f92500d2cf0 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92501615f0 0x7f92500b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92501615f0 0x7f92500b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41136/0 (socket says 172.21.15.46:41136) 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 -- 172.21.15.46:0/3046936465 learned_addr learned my addr 172.21.15.46:0/3046936465 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f9255d6b640 1 --2- 172.21.15.46:0/3046936465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f92500d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 -- 172.21.15.46:0/3046936465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 msgr2=0x7f92500d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 --2- 172.21.15.46:0/3046936465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f92500d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 -- 172.21.15.46:0/3046936465 >> v1:172.21.15.88:6789/0 conn(0x7f92500d0b80 legacy=0x7f92500d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.302+0000 7f925556a640 1 -- 172.21.15.46:0/3046936465 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92501610d0 con 0x7f92501615f0 2024-05-09T03:14:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.303+0000 7f925556a640 1 --2- 172.21.15.46:0/3046936465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f92500b81b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f924c009870 tx=0x7f924c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6aebd4f9966263d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.303+0000 7f9247fff640 1 -- 172.21.15.46:0/3046936465 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c037070 con 0x7f92501615f0 2024-05-09T03:14:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.303+0000 7f9247fff640 1 -- 172.21.15.46:0/3046936465 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924c032440 con 0x7f92501615f0 2024-05-09T03:14:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.303+0000 7f9247fff640 1 -- 172.21.15.46:0/3046936465 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c032810 con 0x7f92501615f0 2024-05-09T03:14:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 -- 172.21.15.46:0/3046936465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 msgr2=0x7f92500b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 --2- 172.21.15.46:0/3046936465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f92500b81b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f924c009870 tx=0x7f924c02a270 comp rx=0 tx=0).stop 2024-05-09T03:14:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 -- 172.21.15.46:0/3046936465 shutdown_connections 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 --2- 172.21.15.46:0/3046936465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f92500d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 --2- 172.21.15.46:0/3046936465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f92500b81b0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 -- 172.21.15.46:0/3046936465 >> 172.21.15.46:0/3046936465 conn(0x7f925006e190 msgr2=0x7f925006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 -- 172.21.15.46:0/3046936465 shutdown_connections 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.304+0000 7f92577ce640 1 -- 172.21.15.46:0/3046936465 wait complete. 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.306+0000 7f92577ce640 1 Processor -- start 2024-05-09T03:14:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.306+0000 7f92577ce640 1 -- start start 2024-05-09T03:14:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f92577ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92500d0b80 0x7f925015d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:58.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f92577ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f925014a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f92577ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f925014a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f92577ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f92501640c0 con 0x7f92501615f0 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f92577ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9250163f40 con 0x7f92500d0b80 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f92577ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9250164240 con 0x7f92500d2cf0 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f9255d6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f925014a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f9255d6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f925014a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41142/0 (socket says 172.21.15.46:41142) 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.307+0000 7f9255d6b640 1 -- 172.21.15.46:0/69711708 learned_addr learned my addr 172.21.15.46:0/69711708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f925556a640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92500d0b80 0x7f925015d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9255d6b640 1 -- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 msgr2=0x7f925014a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9254d69640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f925014a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9255d6b640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f925014a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9255d6b640 1 -- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92500d0b80 msgr2=0x7f925015d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9255d6b640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92500d0b80 0x7f925015d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9255d6b640 1 -- 172.21.15.46:0/69711708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925015a6f0 con 0x7f92501615f0 2024-05-09T03:14:58.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f925556a640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92500d0b80 0x7f925015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9255d6b640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f925014a910 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9248007d40 tx=0x7f924800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248014420 con 0x7f92501615f0 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92501483e0 con 0x7f92501615f0 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9248019070 con 0x7f92501615f0 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248014820 con 0x7f92501615f0 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.308+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92501489a0 con 0x7f92501615f0 2024-05-09T03:14:58.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.309+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92501eb670 con 0x7f92501615f0 2024-05-09T03:14:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.310+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f92480149c0 con 0x7f92501615f0 2024-05-09T03:14:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.311+0000 7f9245ffb640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9234076300 0x7f9234078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.311+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f9248097ec0 con 0x7f92501615f0 2024-05-09T03:14:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.311+0000 7f925556a640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9234076300 0x7f9234078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:58.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.312+0000 7f925556a640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9234076300 0x7f9234078810 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f924c034000 tx=0x7f924c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:58.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.316+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f92480610b0 con 0x7f92501615f0 2024-05-09T03:14:58.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.545+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92500d20f0 con 0x7f92501615f0 2024-05-09T03:14:58.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.546+0000 7f9245ffb640 1 -- 172.21.15.46:0/69711708 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5180 (secure 0 0 0) 0x7f92500d20f0 con 0x7f92501615f0 2024-05-09T03:14:58.546 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:14:58.546 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":69,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:14:55.669690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:14:58.547 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 69 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9234076300 msgr2=0x7f9234078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9234076300 0x7f9234078810 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f924c034000 tx=0x7f924c02e000 comp rx=0 tx=0).stop 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 msgr2=0x7f925014a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f925014a910 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9248007d40 tx=0x7f924800e4b0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 shutdown_connections 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9234076300 0x7f9234078810 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92501615f0 0x7f925014a910 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f92500d2cf0 0x7f925014a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 --2- 172.21.15.46:0/69711708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92500d0b80 0x7f925015d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:58.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 >> 172.21.15.46:0/69711708 conn(0x7f925006e190 msgr2=0x7f92500b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:58.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 shutdown_connections 2024-05-09T03:14:58.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:58.548+0000 7f92577ce640 1 -- 172.21.15.46:0/69711708 wait complete. 2024-05-09T03:14:58.963 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:14:52.631122+0000', 'modified': '2024-05-09T03:14:55.669690+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6471}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6471': {'gid': 6471, 'name': 'a', 'rank': 0, 'incarnation': 65, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6834/131652171', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 131652171}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 131652171}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6471, 'qdb_cluster': [6471]} 2024-05-09T03:14:58.963 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:14:58.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-09T03:14:59.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.086+0000 7f17d248b640 1 Processor -- start 2024-05-09T03:14:59.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.086+0000 7f17d248b640 1 -- start start 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17d248b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f17cc0d2800 0x7f17cc0b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17d248b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17cc06ebe0 con 0x7f17cc0b4700 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17d248b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17cc06eec0 con 0x7f17cc0b3f50 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17d248b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f17cc06f1a0 con 0x7f17cc0d2800 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17cbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17cc0b4700 0x7f17cc0d20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54164/0 (socket says 172.21.15.46:54164) 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17cbfff640 1 -- 172.21.15.46:0/1846378421 learned_addr learned my addr 172.21.15.46:0/1846378421 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17c3fff640 1 --2- 172.21.15.46:0/1846378421 >> v2:172.21.15.88:3300/0 conn(0x7f17cc0d2800 0x7f17cc0b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1935936328 0 0) 0x7f17cc06ebe0 con 0x7f17cc0b4700 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.088+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17b8003520 con 0x7f17cc0b4700 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1676308329 0 0) 0x7f17b8003520 con 0x7f17cc0b4700 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 >> v2:172.21.15.88:3300/0 conn(0x7f17cc0d2800 msgr2=0x7f17cc0b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 --2- 172.21.15.46:0/1846378421 >> v2:172.21.15.88:3300/0 conn(0x7f17cc0d2800 0x7f17cc0b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 >> v1:172.21.15.88:6789/0 conn(0x7f17cc0b3f50 legacy=0x7f17cc0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc06f480 con 0x7f17cc0b4700 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f17bc0016f0 con 0x7f17cc0b4700 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 >> v1:172.21.15.46:6789/0 conn(0x7f17cc0b4700 legacy=0x7f17cc0d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17b8003a50 0x7f17b8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17b8004800 0x7f17b8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17b800ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc06f1a0 con 0x7f17b80053a0 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f17b800c330 con 0x7f17b8003a50 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f17b800c530 con 0x7f17b8004800 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17d0a28640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17b800ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17cbfff640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17b8003a50 0x7f17b8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17c3fff640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17b8004800 0x7f17b8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17d0a28640 1 -- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17b8004800 msgr2=0x7f17b8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17d0a28640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17b8004800 0x7f17b8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17d0a28640 1 -- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17b8003a50 msgr2=0x7f17b8003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17d0a28640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17b8003a50 0x7f17b8003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.089+0000 7f17d0a28640 1 -- 172.21.15.46:0/1846378421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b8003520 con 0x7f17b80053a0 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.090+0000 7f17d0a28640 1 -- 172.21.15.46:0/1846378421 reap_dead start 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.090+0000 7f17d0a28640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17b800ba50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f17c4001e90 tx=0x7f17b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38667d2cbfaef329 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.090+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c4001ab0 con 0x7f17b80053a0 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.090+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c402ae50 con 0x7f17b80053a0 2024-05-09T03:14:59.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.090+0000 7f17cb7fe640 1 -- 172.21.15.46:0/1846378421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c4003b90 con 0x7f17b80053a0 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 -- 172.21.15.46:0/1846378421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 msgr2=0x7f17b800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17b800ba50 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f17c4001e90 tx=0x7f17b8005830 comp rx=0 tx=0).stop 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 -- 172.21.15.46:0/1846378421 shutdown_connections 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 --2- 172.21.15.46:0/1846378421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17b800ba50 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 -- 172.21.15.46:0/1846378421 >> 172.21.15.46:0/1846378421 conn(0x7f17cc0b55d0 msgr2=0x7f17cc0b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 -- 172.21.15.46:0/1846378421 shutdown_connections 2024-05-09T03:14:59.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.091+0000 7f17d248b640 1 -- 172.21.15.46:0/1846378421 wait complete. 2024-05-09T03:14:59.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.093+0000 7f17d248b640 1 Processor -- start 2024-05-09T03:14:59.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.093+0000 7f17d248b640 1 -- start start 2024-05-09T03:14:59.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d248b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17cc1dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d248b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17cc1eaf50 0x7f17cc1eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d248b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17cc09fca0 0x7f17cc0a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d248b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0b51d0 con 0x7f17b80053a0 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d248b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc0b4360 con 0x7f17cc09fca0 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d248b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f17cc1f60f0 con 0x7f17cc1eaf50 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17cc1dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17cc1dbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41154/0 (socket says 172.21.15.46:41154) 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17cbfff640 1 -- 172.21.15.46:0/247362615 learned_addr learned my addr 172.21.15.46:0/247362615 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17c3fff640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17cc1eaf50 0x7f17cc1eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17d0a28640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17cc09fca0 0x7f17cc0a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.094+0000 7f17cbfff640 1 -- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17cc1eaf50 msgr2=0x7f17cc1eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17cbfff640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17cc1eaf50 0x7f17cc1eb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17cbfff640 1 -- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17cc09fca0 msgr2=0x7f17cc0a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17cbfff640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17cc09fca0 0x7f17cc0a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17cbfff640 1 -- 172.21.15.46:0/247362615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17cc1dc200 con 0x7f17b80053a0 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17d0a28640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17cc09fca0 0x7f17cc0a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17cbfff640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17cc1dbcc0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0018a0 tx=0x7f17bc007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17c97fa640 1 -- 172.21.15.46:0/247362615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00f6e0 con 0x7f17b80053a0 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17c97fa640 1 -- 172.21.15.46:0/247362615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc018070 con 0x7f17b80053a0 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17cc0a05f0 con 0x7f17b80053a0 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17c97fa640 1 -- 172.21.15.46:0/247362615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00fae0 con 0x7f17b80053a0 2024-05-09T03:14:59.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.095+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17cc09d960 con 0x7f17b80053a0 2024-05-09T03:14:59.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.097+0000 7f17c97fa640 1 -- 172.21.15.46:0/247362615 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f17bc00fc80 con 0x7f17b80053a0 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.097+0000 7f17c97fa640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17ac0762d0 0x7f17ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.097+0000 7f17c97fa640 1 -- 172.21.15.46:0/247362615 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f17bc097540 con 0x7f17b80053a0 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.098+0000 7f17c3fff640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17ac0762d0 0x7f17ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.098+0000 7f17d9413740 1 Processor -- start 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.098+0000 7f17d9413740 1 -- start start 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.098+0000 7f17c3fff640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17ac0762d0 0x7f17ac0787e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0d1a50 tx=0x7f17b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d9413740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d69ff120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d9413740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5567d691c4e0 con 0x5567d65eb3b0 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5567d68b7ff0 con 0x5567d6614270 2024-05-09T03:14:59.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5567d69184a0 con 0x5567d65f3590 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41160/0 (socket says 172.21.15.46:41160) 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17c3fff640 1 -- 172.21.15.46:0/1116936473 learned_addr learned my addr 172.21.15.46:0/1116936473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17d0a28640 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d69ff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.099+0000 7f17cbfff640 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c3fff640 1 -- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 msgr2=0x5567d69ff120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c3fff640 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d69ff120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c3fff640 1 -- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 msgr2=0x5567d69e3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c3fff640 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69e3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c3fff640 1 -- 172.21.15.46:0/1116936473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567d689e720 con 0x5567d65eb3b0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c3fff640 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69febe0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f17b4014ee0 tx=0x7f17b4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fcfae96d86c1c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c1ffb640 1 -- 172.21.15.46:0/1116936473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b401b070 con 0x5567d65eb3b0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c1ffb640 1 -- 172.21.15.46:0/1116936473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b4015e30 con 0x5567d65eb3b0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.100+0000 7f17c1ffb640 1 -- 172.21.15.46:0/1116936473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b402a630 con 0x5567d65eb3b0 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 -- 172.21.15.46:0/1116936473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 msgr2=0x5567d69febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69febe0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f17b4014ee0 tx=0x7f17b4017000 comp rx=0 tx=0).stop 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 -- 172.21.15.46:0/1116936473 shutdown_connections 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d69ff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69febe0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 --2- 172.21.15.46:0/1116936473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69e3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 -- 172.21.15.46:0/1116936473 >> 172.21.15.46:0/1116936473 conn(0x5567d66083c0 msgr2=0x5567d69b9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 -- 172.21.15.46:0/1116936473 shutdown_connections 2024-05-09T03:14:59.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.101+0000 7f17d9413740 1 -- 172.21.15.46:0/1116936473 wait complete. 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.102+0000 7f17d9413740 1 -- start start 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.102+0000 7f17d9413740 1 Processor -- start 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d65eb3b0 0x5567d69de350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d69fcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d6614270 0x5567d69fee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5567d68a3760 con 0x5567d65f3590 2024-05-09T03:14:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5567d69fb650 con 0x5567d65eb3b0 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5567d69fd550 con 0x5567d6614270 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17d9413740 10 client.0 _ll_register_callbacks cb 0x5567d68c64b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d69fcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d69fcb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41168/0 (socket says 172.21.15.46:41168) 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.103+0000 7f17c3fff640 1 -- 172.21.15.46:0/2139958465 learned_addr learned my addr 172.21.15.46:0/2139958465 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17cbfff640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d65eb3b0 0x5567d69de350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17d0a28640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d6614270 0x5567d69fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17c3fff640 1 -- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d6614270 msgr2=0x5567d69fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17c3fff640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d6614270 0x5567d69fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17c3fff640 1 -- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d65eb3b0 msgr2=0x5567d69de350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17c3fff640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d65eb3b0 0x5567d69de350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17c3fff640 1 -- 172.21.15.46:0/2139958465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567d68e4bd0 con 0x5567d65f3590 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17d0a28640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d6614270 0x5567d69fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17c3fff640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d69fcb20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009fd0 tx=0x7f17b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5147a4ca280a5a5c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17aaffd640 1 -- 172.21.15.46:0/2139958465 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4014b20 con 0x5567d65f3590 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17aaffd640 1 -- 172.21.15.46:0/2139958465 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f17b402f070 con 0x5567d65f3590 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b4015e30 con 0x5567d65f3590 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17aaffd640 1 -- 172.21.15.46:0/2139958465 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f17b4040050 con 0x5567d65f3590 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17d9413740 10 client.6591 fetch_fsmap learned FSMap version 69 2024-05-09T03:14:59.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) -- 0x5567d68ca280 con 0x5567d65f3590 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.104+0000 7f17aaffd640 1 -- 172.21.15.46:0/2139958465 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17b4015e30 con 0x5567d65f3590 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17aaffd640 1 -- 172.21.15.46:0/2139958465 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 69) ==== 3494+0+0 (secure 0 0 0) 0x7f17b4043020 con 0x5567d65f3590 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17d9413740 10 client.6591 fetch_fsmap finished waiting for FSMap version 69 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17d9413740 10 client.6591 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17d9413740 20 client.6591 populate_metadata read hostname 'smithi046' 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17d9413740 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d65d2e40 0x5567d6a11b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17d9413740 4 client.6591 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 --> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567d692a0f0 con 0x5567d65d2e40 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17cbfff640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d65d2e40 0x5567d6a11b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17aaffd640 10 client.6591 ms_handle_connect on v2:172.21.15.46:6832/131652171 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.105+0000 7f17cbfff640 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d65d2e40 0x5567d6a11b90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f15f2a467828159a server_cookie=c34f76149e67f841 in_seq=0 out_seq=0 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17aaffd640 1 -- 172.21.15.46:0/2139958465 <== mds.0 v2:172.21.15.46:6832/131652171 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5567d692a0f0 con 0x5567d65d2e40 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17aaffd640 10 client.6591 handle_command_reply: tid=0 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 client.6591 shutdown 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 msgr2=0x5567d69fcb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d69fcb20 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009fd0 tx=0x7f17b40046c0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 shutdown_connections 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d65d2e40 0x5567d6a11b90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d6614270 0x5567d69fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d69fcb20 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 --2- 172.21.15.46:0/2139958465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d65eb3b0 0x5567d69de350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 >> 172.21.15.46:0/2139958465 conn(0x5567d66083c0 msgr2=0x5567d69e0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17aaffd640 0 client.6591 ms_handle_reset on v2:172.21.15.46:6832/131652171 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 shutdown_connections 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.106+0000 7f17d9413740 1 -- 172.21.15.46:0/2139958465 wait complete. 2024-05-09T03:14:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.107+0000 7f17d9413740 20 client.6591 trim_cache size 0 max 16384 2024-05-09T03:14:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.125+0000 7f17d9413740 1 Processor -- start 2024-05-09T03:14:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.125+0000 7f17d9413740 1 -- start start 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d9413740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69fdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d6a727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d9413740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc202290 con 0x5567d65eb3b0 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f17cc09f2f0 con 0x5567d6614270 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f17cc201c70 con 0x5567d65f3590 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69fdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69fdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41176/0 (socket says 172.21.15.46:41176) 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 -- 172.21.15.46:0/254810096 learned_addr learned my addr 172.21.15.46:0/254810096 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17d0a28640 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17c3fff640 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d6a727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 -- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 msgr2=0x5567d6a727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d6a727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 -- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 msgr2=0x5567d69de160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69de160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 -- 172.21.15.46:0/254810096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567d69baf50 con 0x5567d65eb3b0 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17cbfff640 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69fdd30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f17bc05cd00 tx=0x7f17bc095940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eff81a135338bd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17c17fa640 1 -- 172.21.15.46:0/254810096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc064ac0 con 0x5567d65eb3b0 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17c17fa640 1 -- 172.21.15.46:0/254810096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc069570 con 0x5567d65eb3b0 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.127+0000 7f17c17fa640 1 -- 172.21.15.46:0/254810096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc0970c0 con 0x5567d65eb3b0 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 -- 172.21.15.46:0/254810096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 msgr2=0x5567d69fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69fdd30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f17bc05cd00 tx=0x7f17bc095940 comp rx=0 tx=0).stop 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 -- 172.21.15.46:0/254810096 shutdown_connections 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d69de160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65f3590 0x5567d6a727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 --2- 172.21.15.46:0/254810096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65eb3b0 0x5567d69fdd30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 -- 172.21.15.46:0/254810096 >> 172.21.15.46:0/254810096 conn(0x5567d66083c0 msgr2=0x5567d69af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 -- 172.21.15.46:0/254810096 shutdown_connections 2024-05-09T03:14:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.128+0000 7f17d9413740 1 -- 172.21.15.46:0/254810096 wait complete. 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.129+0000 7f17d9413740 1 -- start start 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.129+0000 7f17d9413740 1 Processor -- start 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65eb3b0 0x5567d6a17cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d6a18230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d6a5ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5567d69fd060 con 0x5567d65f3590 2024-05-09T03:14:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5567d69fd7a0 con 0x5567d6614270 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5567d69fa7e0 con 0x5567d65eb3b0 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 10 client.0 _ll_register_callbacks cb 0x7f17bc05cd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d6a18230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d6a18230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41182/0 (socket says 172.21.15.46:41182) 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 -- 172.21.15.46:0/2114574644 learned_addr learned my addr 172.21.15.46:0/2114574644 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d0a28640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d6a5ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17cbfff640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65eb3b0 0x5567d6a17cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 -- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65eb3b0 msgr2=0x5567d6a17cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65eb3b0 0x5567d6a17cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 -- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 msgr2=0x5567d6a5ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d6a5ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 -- 172.21.15.46:0/2114574644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567d667f850 con 0x5567d65f3590 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d0a28640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d6a5ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17c3fff640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d6a18230 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f17b402ab10 tx=0x7f17b401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e27db127cc37e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17aaffd640 1 -- 172.21.15.46:0/2114574644 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4014b20 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17aaffd640 1 -- 172.21.15.46:0/2114574644 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f17b4034650 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.131+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17bc069570 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17aaffd640 1 -- 172.21.15.46:0/2114574644 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) ==== 24+0+0 (secure 0 0 0) 0x7f17b4054050 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 10 client.6597 fetch_fsmap learned FSMap version 69 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) -- 0x5567d66f66b0 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17aaffd640 1 -- 172.21.15.46:0/2114574644 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17bc069570 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17aaffd640 1 -- 172.21.15.46:0/2114574644 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 69) ==== 3494+0+0 (secure 0 0 0) 0x7f17b4057020 con 0x5567d65f3590 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 10 client.6597 fetch_fsmap finished waiting for FSMap version 69 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 10 client.6597 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 20 client.6597 populate_metadata read hostname 'smithi046' 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d6a5e610 0x5567d69b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 4 client.6597 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 --> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] -- command(tid 0: {"prefix": "status"}) -- 0x5567d6923f60 con 0x5567d6a5e610 2024-05-09T03:14:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17cbfff640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d6a5e610 0x5567d69b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.132+0000 7f17aaffd640 10 client.6597 ms_handle_connect on v2:172.21.15.46:6832/131652171 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17cbfff640 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d6a5e610 0x5567d69b9440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76c68a166e0f3015 server_cookie=182998a44524317 in_seq=0 out_seq=0 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17aaffd640 1 -- 172.21.15.46:0/2114574644 <== mds.0 v2:172.21.15.46:6832/131652171 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5567d6923f60 con 0x5567d6a5e610 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17aaffd640 10 client.6597 handle_command_reply: tid=0 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 client.6597 shutdown 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 msgr2=0x5567d6a18230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d6a18230 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f17b402ab10 tx=0x7f17b401b000 comp rx=0 tx=0).stop 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 shutdown_connections 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x5567d6a5e610 0x5567d69b9440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5567d6614270 0x5567d6a5ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5567d65f3590 0x5567d6a18230 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17aaffd640 0 client.6597 ms_handle_reset on v2:172.21.15.46:6832/131652171 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 --2- 172.21.15.46:0/2114574644 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5567d65eb3b0 0x5567d6a17cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 >> 172.21.15.46:0/2114574644 conn(0x5567d66083c0 msgr2=0x5567d69af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 shutdown_connections 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 1 -- 172.21.15.46:0/2114574644 wait complete. 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.133+0000 7f17d9413740 20 client.6597 trim_cache size 0 max 16384 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6471, 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:14:59.139 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 5.8712312170000001, 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 69, 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 68, 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 68, 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.835839853 2024-05-09T03:14:59.140 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17ac0762d0 msgr2=0x7f17ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17ac0762d0 0x7f17ac0787e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0d1a50 tx=0x7f17b4009190 comp rx=0 tx=0).stop 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 msgr2=0x7f17cc1dbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17cc1dbcc0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f17bc0018a0 tx=0x7f17bc007430 comp rx=0 tx=0).stop 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 shutdown_connections 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17cc09fca0 0x7f17cc0a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f17cc1eaf50 0x7f17cc1eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f17ac0762d0 0x7f17ac0787e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 --2- 172.21.15.46:0/247362615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b80053a0 0x7f17cc1dbcc0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 >> 172.21.15.46:0/247362615 conn(0x7f17cc0b55d0 msgr2=0x7f17cc0d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 shutdown_connections 2024-05-09T03:14:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.134+0000 7f17d248b640 1 -- 172.21.15.46:0/247362615 wait complete. 2024-05-09T03:14:59.146 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:14:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.259+0000 7f045752d640 1 Processor -- start 2024-05-09T03:14:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.260+0000 7f045752d640 1 -- start start 2024-05-09T03:14:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.261+0000 7f045752d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f04500d1c80 0x7f04500cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.261+0000 7f045752d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04500d4900 con 0x7f045008cd60 2024-05-09T03:14:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.261+0000 7f045752d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04500b4fd0 con 0x7f04500d2090 2024-05-09T03:14:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.261+0000 7f045752d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f04500b5340 con 0x7f04500d1c80 2024-05-09T03:14:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.261+0000 7f04552c9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f045008cd60 0x7f04500d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54180/0 (socket says 172.21.15.46:54180) 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.261+0000 7f04552c9640 1 -- 172.21.15.46:0/4008127578 learned_addr learned my addr 172.21.15.46:0/4008127578 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f0454ac8640 1 --2- 172.21.15.46:0/4008127578 >> v2:172.21.15.88:3300/0 conn(0x7f04500d1c80 0x7f04500cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185361366 0 0) 0x7f04500d4900 con 0x7f045008cd60 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0438003520 con 0x7f045008cd60 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 877088418 0 0) 0x7f0438003520 con 0x7f045008cd60 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 >> v2:172.21.15.88:3300/0 conn(0x7f04500d1c80 msgr2=0x7f04500cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 --2- 172.21.15.46:0/4008127578 >> v2:172.21.15.88:3300/0 conn(0x7f04500d1c80 0x7f04500cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 >> v1:172.21.15.88:6789/0 conn(0x7f04500d2090 legacy=0x7f04500d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500b6a50 con 0x7f045008cd60 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f0440001a70 con 0x7f045008cd60 2024-05-09T03:14:59.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 >> v1:172.21.15.46:6789/0 conn(0x7f045008cd60 legacy=0x7f04500d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438003a80 0x7f0438003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0438004800 0x7f0438004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0438005350 0x7f043800b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04500b5340 con 0x7f0438005350 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f043800c2c0 con 0x7f0438003a80 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f043800c4c0 con 0x7f0438004800 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f0455aca640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0438005350 0x7f043800b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f04552c9640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438003a80 0x7f0438003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f0454ac8640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0438004800 0x7f0438004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.262+0000 7f0455aca640 1 -- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0438004800 msgr2=0x7f0438004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f0455aca640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0438004800 0x7f0438004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f0455aca640 1 -- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438003a80 msgr2=0x7f0438003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f0455aca640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438003a80 0x7f0438003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f0455aca640 1 -- 172.21.15.46:0/4008127578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0438003520 con 0x7f0438005350 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f0455aca640 1 -- 172.21.15.46:0/4008127578 reap_dead start 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f0455aca640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0438005350 0x7f043800b9e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f0448001790 tx=0x7f0438005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f8f1e773c1c7c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480013b0 con 0x7f0438005350 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04480052d0 con 0x7f0438005350 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.263+0000 7f044ffff640 1 -- 172.21.15.46:0/4008127578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044802aae0 con 0x7f0438005350 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 -- 172.21.15.46:0/4008127578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0438005350 msgr2=0x7f043800b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0438005350 0x7f043800b9e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f0448001790 tx=0x7f0438005830 comp rx=0 tx=0).stop 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 -- 172.21.15.46:0/4008127578 shutdown_connections 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 --2- 172.21.15.46:0/4008127578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0438005350 0x7f043800b9e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 -- 172.21.15.46:0/4008127578 >> 172.21.15.46:0/4008127578 conn(0x7f04500d3c00 msgr2=0x7f04500d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 -- 172.21.15.46:0/4008127578 shutdown_connections 2024-05-09T03:14:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.264+0000 7f045752d640 1 -- 172.21.15.46:0/4008127578 wait complete. 2024-05-09T03:14:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.266+0000 7f045752d640 1 Processor -- start 2024-05-09T03:14:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.266+0000 7f045752d640 1 -- start start 2024-05-09T03:14:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.267+0000 7f045752d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438005350 0x7f04500b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.267+0000 7f045752d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0450152320 0x7f045015ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.267+0000 7f045752d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 0x7f04500cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.267+0000 7f045752d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04500d2840 con 0x7f04500b2c80 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.267+0000 7f045752d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f04500d26c0 con 0x7f0438005350 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.267+0000 7f045752d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f04500d29c0 con 0x7f0450152320 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 0x7f04500cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 0x7f04500cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41188/0 (socket says 172.21.15.46:41188) 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 -- 172.21.15.46:0/4045436185 learned_addr learned my addr 172.21.15.46:0/4045436185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f04552c9640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438005350 0x7f04500b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0454ac8640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0450152320 0x7f045015ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 -- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0450152320 msgr2=0x7f045015ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0450152320 0x7f045015ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 -- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438005350 msgr2=0x7f04500b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438005350 0x7f04500b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 -- 172.21.15.46:0/4045436185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500b1be0 con 0x7f04500b2c80 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f0455aca640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 0x7f04500cacb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f044802a590 tx=0x7f0448001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448007a70 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448005480 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480013b0 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.268+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04500b1ec0 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.269+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04500b2480 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.270+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04500b1950 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.270+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0448004b40 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.271+0000 7f044dffb640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f043c0762d0 0x7f043c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.271+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f04480b6960 con 0x7f04500b2c80 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.271+0000 7f04552c9640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f043c0762d0 0x7f043c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:14:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.272+0000 7f04552c9640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f043c0762d0 0x7f043c0787e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0450074930 tx=0x7f0440006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:14:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.276+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0448080250 con 0x7f04500b2c80 2024-05-09T03:14:59.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.506+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f045006a270 con 0x7f04500b2c80 2024-05-09T03:14:59.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.507+0000 7f044dffb640 1 -- 172.21.15.46:0/4045436185 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v69) ==== 53+0+73 (secure 0 0 0) 0x7f044807fb50 con 0x7f04500b2c80 2024-05-09T03:14:59.508 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:14:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f043c0762d0 msgr2=0x7f043c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f043c0762d0 0x7f043c0787e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0450074930 tx=0x7f0440006040 comp rx=0 tx=0).stop 2024-05-09T03:14:59.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 msgr2=0x7f04500cacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 0x7f04500cacb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f044802a590 tx=0x7f0448001120 comp rx=0 tx=0).stop 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 shutdown_connections 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f043c0762d0 0x7f043c0787e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500b2c80 0x7f04500cacb0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0450152320 0x7f045015ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 --2- 172.21.15.46:0/4045436185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0438005350 0x7f04500b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 >> 172.21.15.46:0/4045436185 conn(0x7f04500d3c00 msgr2=0x7f045015d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 shutdown_connections 2024-05-09T03:14:59.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59.509+0000 7f045752d640 1 -- 172.21.15.46:0/4045436185 wait complete. 2024-05-09T03:14:59.521 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:14:59.521 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:14:59.521 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:14:59.522 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:14:59.522 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:59.522 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:14:59.522 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:14:59.522 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:14:59.522 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:14:59.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:59.522 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:14:59.539 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42218sec preferred_lft 42218sec 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:14:59.540 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:14:59.541 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:14:59.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:59.542 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:14:59.604 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:14:59.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:59.605 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:14:59.662 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:14:59.662 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:14:59.662 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:14:59.662 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:14:59.662 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:14:59.662 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:14:59.662 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:14:59.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:14:59.662 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:14:59.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:14:59.663 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:14:59.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:14:59.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:14:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:14:59.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:59.792 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:14:59.808 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:14:59.808 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:14:59.808 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:14:59.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:14:59.808 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:14:59.874 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:14:59.874 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:14:59.874 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:59.931 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:14:59.931 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:14:59.931 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:14:59.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:15:00.054 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:15:00.111 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:15:00.111 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:15:00.112 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:15:00.113 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:00.114 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:00.114 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:00.114 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:00.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:00.114 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:15:00.167 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:15:00.167 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:15:00.210 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:15:00.235 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:15:00.294 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:15:00.295 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:00.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:00.296 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:15:00.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:15:00.305+0000 7fe20df87480 -1 init, newargv = 0x7fe1f0001080 newargc=15 2024-05-09T03:15:00.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[59051]: starting ceph client 2024-05-09T03:15:00.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[59051]: starting fuse 2024-05-09T03:15:00.321 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:15:00.322 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:15:00.322 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:15:00.323 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:15:00.324 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:15:00.324 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:15:00.324 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:15:00.324 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:15:00.324 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:15:00.443 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.59051.asok 2024-05-09T03:15:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:15:00.454 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59051.asok 2024-05-09T03:15:00.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:00.454 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59051.asok status 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "59051", 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:15:00.602 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6612, 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6612 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3594189517 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:15:00.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3594189517 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6612 192.168.0.1:0/3594189517", 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3594189517", 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 69, 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 68, 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:15:00.604 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:15:00.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:00.611 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:15:00.611 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:00.627 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:15:00.627 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:00.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:00.627 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:00.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:00.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.848+0000 7f5cc8c9a640 1 Processor -- start 2024-05-09T03:15:00.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.848+0000 7f5cc8c9a640 1 -- start start 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc8c9a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc40d0b90 0x7f5cc40b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc8c9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5cc40cf6b0 0x7f5cc406d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc8c9a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cc40d2b80 con 0x7f5cc40b92e0 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc8c9a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40d2e60 con 0x7f5cc40d0b90 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc8c9a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40d3140 con 0x7f5cc40cf6b0 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cc40d0b90 0x7f5cc40b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5cc40b92e0 0x7f5cc40d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54182/0 (socket says 172.21.15.46:54182) 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc259c640 1 -- 172.21.15.46:0/3515673291 learned_addr learned my addr 172.21.15.46:0/3515673291 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc2d9d640 1 --2- 172.21.15.46:0/3515673291 >> v2:172.21.15.88:3300/0 conn(0x7f5cc40cf6b0 0x7f5cc406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:00.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc1d9b640 1 -- 172.21.15.46:0/3515673291 >> v2:172.21.15.88:3300/0 conn(0x7f5cc40cf6b0 msgr2=0x7f5cc406d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc1d9b640 1 --2- 172.21.15.46:0/3515673291 >> v2:172.21.15.88:3300/0 conn(0x7f5cc40cf6b0 0x7f5cc406d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc1d9b640 1 -- 172.21.15.46:0/3515673291 >> v1:172.21.15.46:6789/0 conn(0x7f5cc40b92e0 legacy=0x7f5cc40d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:00.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.850+0000 7f5cc1d9b640 1 -- 172.21.15.46:0/3515673291 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40d3420 con 0x7f5cc40d0b90 2024-05-09T03:15:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.851+0000 7f5cc1d9b640 1 --2- 172.21.15.46:0/3515673291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40d0b90 0x7f5cc40b51a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40b9720 tx=0x7f5cb802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a23b39cf33cf780 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.851+0000 7f5cc159a640 1 -- 172.21.15.46:0/3515673291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8037070 con 0x7f5cc40d0b90 2024-05-09T03:15:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.851+0000 7f5cc159a640 1 -- 172.21.15.46:0/3515673291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb8032440 con 0x7f5cc40d0b90 2024-05-09T03:15:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.851+0000 7f5cc159a640 1 -- 172.21.15.46:0/3515673291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb802ae60 con 0x7f5cc40d0b90 2024-05-09T03:15:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/3515673291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40d0b90 msgr2=0x7f5cc40b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:00.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/3515673291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40d0b90 0x7f5cc40b51a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40b9720 tx=0x7f5cb802a110 comp rx=0 tx=0).stop 2024-05-09T03:15:00.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/3515673291 shutdown_connections 2024-05-09T03:15:00.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/3515673291 >> v2:172.21.15.88:3300/0 conn(0x7f5cc40cf6b0 0x7f5cc406d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:00.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/3515673291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40d0b90 0x7f5cc40b51a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:00.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/3515673291 >> 172.21.15.46:0/3515673291 conn(0x7f5cc40d3c00 msgr2=0x7f5cc40d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:00.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/3515673291 shutdown_connections 2024-05-09T03:15:00.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.852+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/3515673291 wait complete. 2024-05-09T03:15:00.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.854+0000 7f5cc8c9a640 1 Processor -- start 2024-05-09T03:15:00.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.854+0000 7f5cc8c9a640 1 -- start start 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc8c9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 0x7f5cc40b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc8c9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc40cf6b0 0x7f5cc40b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc8c9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cc40b41e0 0x7f5cc40b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc8c9a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc406e060 con 0x7f5cc40b92e0 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc8c9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc406dee0 con 0x7f5cc40cf6b0 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc8c9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc406e1e0 con 0x7f5cc40b41e0 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 0x7f5cc40b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.855+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 0x7f5cc40b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41210/0 (socket says 172.21.15.46:41210) 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 -- 172.21.15.46:0/2210576023 learned_addr learned my addr 172.21.15.46:0/2210576023 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc1d9b640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc40cf6b0 0x7f5cc40b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc2d9d640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cc40b41e0 0x7f5cc40b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 -- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cc40b41e0 msgr2=0x7f5cc40b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cc40b41e0 0x7f5cc40b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 -- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc40cf6b0 msgr2=0x7f5cc40b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc40cf6b0 0x7f5cc40b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 -- 172.21.15.46:0/2210576023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc41e9b10 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc259c640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 0x7f5cc40b3ca0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400cb40 tx=0x7f5cb400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4017070 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb400ac60 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc41e7c20 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4012640 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.856+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc415d7e0 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.857+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc41e7a10 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.858+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5cb401a070 con 0x7f5cc40b92e0 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.858+0000 7f5cb2ffd640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ca40762d0 0x7f5ca40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:00.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.859+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f5cb40973c0 con 0x7f5cc40b92e0 2024-05-09T03:15:00.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.859+0000 7f5cc1d9b640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ca40762d0 0x7f5ca40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:00.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.859+0000 7f5cc1d9b640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ca40762d0 0x7f5ca40787e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8034000 tx=0x7f5cb802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:00.863+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5cb40605b0 con 0x7f5cc40b92e0 2024-05-09T03:15:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.094+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cc406a270 con 0x7f5cc40b92e0 2024-05-09T03:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.095+0000 7f5cb2ffd640 1 -- 172.21.15.46:0/2210576023 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5180 (secure 0 0 0) 0x7f5cb40648b0 con 0x7f5cc40b92e0 2024-05-09T03:15:01.096 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:01.096 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":69,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:14:55.669690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:15:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 69 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ca40762d0 msgr2=0x7f5ca40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ca40762d0 0x7f5ca40787e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8034000 tx=0x7f5cb802e000 comp rx=0 tx=0).stop 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 msgr2=0x7f5cc40b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 0x7f5cc40b3ca0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400cb40 tx=0x7f5cb400a060 comp rx=0 tx=0).stop 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 shutdown_connections 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5cc40b41e0 0x7f5cc40b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cc40cf6b0 0x7f5cc40b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ca40762d0 0x7f5ca40787e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 --2- 172.21.15.46:0/2210576023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cc40b92e0 0x7f5cc40b3ca0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 >> 172.21.15.46:0/2210576023 conn(0x7f5cc40d3c00 msgr2=0x7f5cc41600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 shutdown_connections 2024-05-09T03:15:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.097+0000 7f5cc8c9a640 1 -- 172.21.15.46:0/2210576023 wait complete. 2024-05-09T03:15:01.109 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 true 2024-05-09T03:15:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.225+0000 7f0f33bf3640 1 Processor -- start 2024-05-09T03:15:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.225+0000 7f0f33bf3640 1 -- start start 2024-05-09T03:15:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.226+0000 7f0f33bf3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b7e70 0x7f0f2c160320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.226+0000 7f0f33bf3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f2c163d80 0x7f0f2c164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.226+0000 7f0f33bf3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2c161700 con 0x7f0f2c0b4d80 2024-05-09T03:15:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.226+0000 7f0f33bf3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0f2c0d1230 con 0x7f0f2c163d80 2024-05-09T03:15:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.226+0000 7f0f33bf3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c0d0900 con 0x7f0f2c0b7e70 2024-05-09T03:15:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f3198f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f2c0b4d80 0x7f0f2c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54188/0 (socket says 172.21.15.46:54188) 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f3198f640 1 -- 172.21.15.46:0/1507853691 learned_addr learned my addr 172.21.15.46:0/1507853691 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f32190640 1 --2- 172.21.15.46:0/1507853691 >> v2:172.21.15.46:3300/0 conn(0x7f0f2c163d80 0x7f0f2c164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f3118e640 1 --2- 172.21.15.46:0/1507853691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b7e70 0x7f0f2c160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f3098d640 1 -- 172.21.15.46:0/1507853691 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025869300 0 0) 0x7f0f2c161700 con 0x7f0f2c0b4d80 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f3098d640 1 -- 172.21.15.46:0/1507853691 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f18003520 con 0x7f0f2c0b4d80 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f32190640 1 -- 172.21.15.46:0/1507853691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b7e70 msgr2=0x7f0f2c160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f32190640 1 --2- 172.21.15.46:0/1507853691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b7e70 0x7f0f2c160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f32190640 1 -- 172.21.15.46:0/1507853691 >> v1:172.21.15.46:6789/0 conn(0x7f0f2c0b4d80 legacy=0x7f0f2c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.227+0000 7f0f32190640 1 -- 172.21.15.46:0/1507853691 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c0d0be0 con 0x7f0f2c163d80 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.228+0000 7f0f32190640 1 --2- 172.21.15.46:0/1507853691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c163d80 0x7f0f2c164190 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009d90 tx=0x7f0f2402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae678d1e67282667 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.228+0000 7f0f3098d640 1 -- 172.21.15.46:0/1507853691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24007740 con 0x7f0f2c163d80 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.228+0000 7f0f3098d640 1 -- 172.21.15.46:0/1507853691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f240078e0 con 0x7f0f2c163d80 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.228+0000 7f0f3098d640 1 -- 172.21.15.46:0/1507853691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24007cb0 con 0x7f0f2c163d80 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.228+0000 7f0f33bf3640 1 -- 172.21.15.46:0/1507853691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c163d80 msgr2=0x7f0f2c164190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.228+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/1507853691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c163d80 0x7f0f2c164190 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009d90 tx=0x7f0f2402a660 comp rx=0 tx=0).stop 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.229+0000 7f0f33bf3640 1 -- 172.21.15.46:0/1507853691 shutdown_connections 2024-05-09T03:15:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.229+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/1507853691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c163d80 0x7f0f2c164190 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.229+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/1507853691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b7e70 0x7f0f2c160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.229+0000 7f0f33bf3640 1 -- 172.21.15.46:0/1507853691 >> 172.21.15.46:0/1507853691 conn(0x7f0f2c0d2760 msgr2=0x7f0f2c0d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.229+0000 7f0f33bf3640 1 -- 172.21.15.46:0/1507853691 shutdown_connections 2024-05-09T03:15:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.229+0000 7f0f33bf3640 1 -- 172.21.15.46:0/1507853691 wait complete. 2024-05-09T03:15:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.230+0000 7f0f33bf3640 1 Processor -- start 2024-05-09T03:15:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.231+0000 7f0f33bf3640 1 -- start start 2024-05-09T03:15:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f33bf3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b4d80 0x7f0f2c1512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f33bf3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c163d80 0x7f0f2c151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f33bf3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 0x7f0f2c151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f33bf3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c0b8390 con 0x7f0f2c0ce410 2024-05-09T03:15:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f33bf3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c0d16c0 con 0x7f0f2c163d80 2024-05-09T03:15:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f33bf3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c0b8510 con 0x7f0f2c0b4d80 2024-05-09T03:15:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 0x7f0f2c151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 0x7f0f2c151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41234/0 (socket says 172.21.15.46:41234) 2024-05-09T03:15:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 -- 172.21.15.46:0/3124528268 learned_addr learned my addr 172.21.15.46:0/3124528268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f3198f640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b4d80 0x7f0f2c1512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f3118e640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c163d80 0x7f0f2c151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 -- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b4d80 msgr2=0x7f0f2c1512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b4d80 0x7f0f2c1512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 -- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c163d80 msgr2=0x7f0f2c151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c163d80 0x7f0f2c151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 -- 172.21.15.46:0/3124528268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c0cd370 con 0x7f0f2c0ce410 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.232+0000 7f0f32190640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 0x7f0f2c151fa0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0f2402ac50 tx=0x7f0f24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.233+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2403f070 con 0x7f0f2c0ce410 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.233+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f24031bc0 con 0x7f0f2c0ce410 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.233+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f24009450 con 0x7f0f2c0ce410 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.233+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c0cd650 con 0x7f0f2c0ce410 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.233+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c0cdc10 con 0x7f0f2c0ce410 2024-05-09T03:15:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.234+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2c06a270 con 0x7f0f2c0ce410 2024-05-09T03:15:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.234+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0f2403b410 con 0x7f0f2c0ce410 2024-05-09T03:15:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.235+0000 7f0f227fc640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f100762d0 0x7f0f100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.235+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f0f240b8d20 con 0x7f0f2c0ce410 2024-05-09T03:15:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.235+0000 7f0f3198f640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f100762d0 0x7f0f100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.236+0000 7f0f3198f640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f100762d0 0x7f0f100787e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0f28005880 tx=0x7f0f28006aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:01.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.240+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0f24081f90 con 0x7f0f2c0ce410 2024-05-09T03:15:01.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:01.509+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"} v 0) -- 0x7f0f2c0d4100 con 0x7f0f2c0ce410 2024-05-09T03:15:02.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.304+0000 7f0f227fc640 1 -- 172.21.15.46:0/3124528268 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"}]=0 freezing mds.6471 v70) ==== 112+0+0 (secure 0 0 0) 0x7f0f24086290 con 0x7f0f2c0ce410 2024-05-09T03:15:02.305 INFO:teuthology.orchestra.run.smithi046.stderr:freezing mds.6471 2024-05-09T03:15:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f100762d0 msgr2=0x7f0f100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f100762d0 0x7f0f100787e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0f28005880 tx=0x7f0f28006aa0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 msgr2=0x7f0f2c151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 0x7f0f2c151fa0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0f2402ac50 tx=0x7f0f24004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 shutdown_connections 2024-05-09T03:15:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0f100762d0 0x7f0f100787e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c0ce410 0x7f0f2c151fa0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c163d80 0x7f0f2c151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 --2- 172.21.15.46:0/3124528268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0f2c0b4d80 0x7f0f2c1512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 >> 172.21.15.46:0/3124528268 conn(0x7f0f2c0d2760 msgr2=0x7f0f2c161910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 shutdown_connections 2024-05-09T03:15:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.307+0000 7f0f33bf3640 1 -- 172.21.15.46:0/3124528268 wait complete. 2024-05-09T03:15:02.322 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/{b,c}/d') 2024-05-09T03:15:02.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:15:02.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:15:02.401 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-05-09T03:15:02.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.518+0000 7f27b5cbd640 1 Processor -- start 2024-05-09T03:15:02.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.518+0000 7f27b5cbd640 1 -- start start 2024-05-09T03:15:02.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27b5cbd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 0x7f27b00aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27b5cbd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b00b6d10 con 0x7f27b0070b50 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27b5cbd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b00af1f0 con 0x7f27b00b6480 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27b5cbd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27b00af4d0 con 0x7f27b015e270 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27affff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f27b0070b50 0x7f27b00732d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54202/0 (socket says 172.21.15.46:54202) 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27affff640 1 -- 172.21.15.46:0/1103910357 learned_addr learned my addr 172.21.15.46:0/1103910357 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718835513 0 0) 0x7f27b00b6d10 con 0x7f27b0070b50 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27af7fe640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 0x7f27b00aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.520+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2798003520 con 0x7f27b0070b50 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 259023482 0 0) 0x7f2798003520 con 0x7f27b0070b50 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 msgr2=0x7f27b00aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 0x7f27b00aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 >> v1:172.21.15.88:6789/0 conn(0x7f27b00b6480 legacy=0x7f27b0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b015d790 con 0x7f27b0070b50 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f27a8001a70 con 0x7f27b0070b50 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 >> v1:172.21.15.46:6789/0 conn(0x7f27b0070b50 legacy=0x7f27b00732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 0x7f27980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2798003f90 0x7f2798004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2798004d40 0x7f2798005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27b00af4d0 con 0x7f2798003f90 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f279800bc20 con 0x7f2798004d40 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f279800be20 con 0x7f27b015e270 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2798003f90 0x7f2798004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27af7fe640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 0x7f27980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27affff640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2798004d40 0x7f2798005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27aeffd640 1 -- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 msgr2=0x7f27980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27b015e270 0x7f27980039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27aeffd640 1 -- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2798004d40 msgr2=0x7f2798005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2798004d40 0x7f2798005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.521+0000 7f27aeffd640 1 -- 172.21.15.46:0/1103910357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2798003520 con 0x7f2798003f90 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.522+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2798003f90 0x7f2798004800 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f27b0071590 tx=0x7f2798005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff9a237fc2f026b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.522+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4037070 con 0x7f2798003f90 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.522+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a402ac00 con 0x7f2798003f90 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.522+0000 7f27ae7fc640 1 -- 172.21.15.46:0/1103910357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4001430 con 0x7f2798003f90 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27b5cbd640 1 -- 172.21.15.46:0/1103910357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2798003f90 msgr2=0x7f2798004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/1103910357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2798003f90 0x7f2798004800 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f27b0071590 tx=0x7f2798005830 comp rx=0 tx=0).stop 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27affff640 1 -- 172.21.15.46:0/1103910357 reap_dead start 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27b5cbd640 1 -- 172.21.15.46:0/1103910357 shutdown_connections 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27b5cbd640 1 -- 172.21.15.46:0/1103910357 >> 172.21.15.46:0/1103910357 conn(0x7f27b00b3730 msgr2=0x7f27b00b3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27b5cbd640 1 -- 172.21.15.46:0/1103910357 shutdown_connections 2024-05-09T03:15:02.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.523+0000 7f27b5cbd640 1 -- 172.21.15.46:0/1103910357 wait complete. 2024-05-09T03:15:02.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.525+0000 7f27b5cbd640 1 Processor -- start 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.525+0000 7f27b5cbd640 1 -- start start 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27b5cbd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798003f90 0x7f27b014b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27b5cbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b014dd10 0x7f27b014e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27b5cbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 0x7f27b0150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27b5cbd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27b015d340 con 0x7f27b0150b30 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27b5cbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27b015dd50 con 0x7f27b014dd10 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27b5cbd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27b015d4c0 con 0x7f2798003f90 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 0x7f27b0150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 0x7f27b0150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41248/0 (socket says 172.21.15.46:41248) 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.526+0000 7f27affff640 1 -- 172.21.15.46:0/3524749364 learned_addr learned my addr 172.21.15.46:0/3524749364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27aeffd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b014dd10 0x7f27b014e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27af7fe640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798003f90 0x7f27b014b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27affff640 1 -- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798003f90 msgr2=0x7f27b014b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27affff640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798003f90 0x7f27b014b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27affff640 1 -- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b014dd10 msgr2=0x7f27b014e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27affff640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b014dd10 0x7f27b014e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27affff640 1 -- 172.21.15.46:0/3524749364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b014b970 con 0x7f27b0150b30 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27affff640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 0x7f27b0150f40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f27a8001f10 tx=0x7f27a8007a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f2797fff640 1 -- 172.21.15.46:0/3524749364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a800aaa0 con 0x7f27b0150b30 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f2797fff640 1 -- 172.21.15.46:0/3524749364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a8018070 con 0x7f27b0150b30 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b0151480 con 0x7f27b0150b30 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f2797fff640 1 -- 172.21.15.46:0/3524749364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a80133f0 con 0x7f27b0150b30 2024-05-09T03:15:02.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.527+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b014ed20 con 0x7f27b0150b30 2024-05-09T03:15:02.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.529+0000 7f2797fff640 1 -- 172.21.15.46:0/3524749364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f27a8013590 con 0x7f27b0150b30 2024-05-09T03:15:02.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.530+0000 7f2797fff640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f279c0762d0 0x7f279c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.530+0000 7f2797fff640 1 -- 172.21.15.46:0/3524749364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f27a80979d0 con 0x7f27b0150b30 2024-05-09T03:15:02.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.530+0000 7f27af7fe640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f279c0762d0 0x7f279c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.530+0000 7f27bcb72740 1 Processor -- start 2024-05-09T03:15:02.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.530+0000 7f27bcb72740 1 -- start start 2024-05-09T03:15:02.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.530+0000 7f27af7fe640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f279c0762d0 0x7f279c0787e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f27b00b6240 tx=0x7f27a0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85642d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b856448240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856448780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27bcb72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b8563661a0 con 0x55b8560354d0 2024-05-09T03:15:02.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b85631f170 con 0x55b85603d6b0 2024-05-09T03:15:02.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b856362160 con 0x55b85605e390 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b856448240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27af7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85642d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b856448240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41252/0 (socket says 172.21.15.46:41252) 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 -- 172.21.15.46:0/1793692400 learned_addr learned my addr 172.21.15.46:0/1793692400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27affff640 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856448780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 -- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 msgr2=0x55b85642d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85642d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 -- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 msgr2=0x55b856448780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856448780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 -- 172.21.15.46:0/1793692400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b856331390 con 0x55b8560354d0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.532+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b856448240 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f27a4000fb0 tx=0x7f27a402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8afa63924d7f1b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.533+0000 7f27957fa640 1 -- 172.21.15.46:0/1793692400 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4037040 con 0x55b8560354d0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.533+0000 7f27957fa640 1 -- 172.21.15.46:0/1793692400 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a4042070 con 0x55b8560354d0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.533+0000 7f27957fa640 1 -- 172.21.15.46:0/1793692400 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40054b0 con 0x55b8560354d0 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.533+0000 7f27bcb72740 1 -- 172.21.15.46:0/1793692400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 msgr2=0x55b856448240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.533+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b856448240 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f27a4000fb0 tx=0x7f27a402e000 comp rx=0 tx=0).stop 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 -- 172.21.15.46:0/1793692400 shutdown_connections 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856448780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b856448240 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1793692400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85642d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 -- 172.21.15.46:0/1793692400 >> 172.21.15.46:0/1793692400 conn(0x55b8560524e0 msgr2=0x55b856402b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 -- 172.21.15.46:0/1793692400 shutdown_connections 2024-05-09T03:15:02.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 -- 172.21.15.46:0/1793692400 wait complete. 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.534+0000 7f27bcb72740 1 -- start start 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.535+0000 7f27bcb72740 1 Processor -- start 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 0x55b8564576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85644a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b8562ec8c0 con 0x55b85603d6b0 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b8564302b0 con 0x55b8560354d0 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b85642ff40 con 0x55b85605e390 2024-05-09T03:15:02.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27bcb72740 10 client.0 _ll_register_callbacks cb 0x55b856308860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 0x55b8564576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 0x55b8564576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41262/0 (socket says 172.21.15.46:41262) 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 -- 172.21.15.46:0/1189633900 learned_addr learned my addr 172.21.15.46:0/1189633900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27af7fe640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27affff640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85644a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 -- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 msgr2=0x55b85644a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85644a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 -- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 msgr2=0x55b856427da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.536+0000 7f27aeffd640 1 -- 172.21.15.46:0/1189633900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8561407d0 con 0x55b85603d6b0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27aeffd640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 0x55b8564576d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f27a403f040 tx=0x7f27a4001270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32d6799872643da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f278e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f278e7fc640 1 -- 172.21.15.46:0/1189633900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a4037050 con 0x55b85603d6b0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f278e7fc640 1 -- 172.21.15.46:0/1189633900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f27a4007020 con 0x55b85603d6b0 2024-05-09T03:15:02.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27a4042070 con 0x55b85603d6b0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f278e7fc640 1 -- 172.21.15.46:0/1189633900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f27a4056050 con 0x55b85603d6b0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 10 client.6642 fetch_fsmap learned FSMap version 70 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) -- 0x55b8560c9970 con 0x55b85603d6b0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f278e7fc640 1 -- 172.21.15.46:0/1189633900 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a4042070 con 0x55b85603d6b0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f278e7fc640 1 -- 172.21.15.46:0/1189633900 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 70) ==== 3494+0+0 (secure 0 0 0) 0x7f27a4059020 con 0x55b85603d6b0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 10 client.6642 fetch_fsmap finished waiting for FSMap version 70 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 10 client.6642 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 20 client.6642 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b85601cf60 0x55b856430740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 4 client.6642 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 --> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b856373db0 con 0x55b85601cf60 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.537+0000 7f27af7fe640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b85601cf60 0x55b856430740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.538+0000 7f278e7fc640 10 client.6642 ms_handle_connect on v2:172.21.15.46:6832/131652171 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.538+0000 7f27af7fe640 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b85601cf60 0x55b856430740 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=216b163df5568bda server_cookie=c0a6a30475998141 in_seq=0 out_seq=0 2024-05-09T03:15:02.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.538+0000 7f278e7fc640 1 -- 172.21.15.46:0/1189633900 <== mds.0 v2:172.21.15.46:6832/131652171 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b856373db0 con 0x55b85601cf60 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.538+0000 7f278e7fc640 10 client.6642 handle_command_reply: tid=0 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 client.6642 shutdown 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 msgr2=0x55b8564576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 0x55b8564576d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f27a403f040 tx=0x7f27a4001270 comp rx=0 tx=0).stop 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 shutdown_connections 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b85601cf60 0x55b856430740 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b85644a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85603d6b0 0x55b8564576d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f278e7fc640 0 client.6642 ms_handle_reset on v2:172.21.15.46:6832/131652171 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 --2- 172.21.15.46:0/1189633900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 >> 172.21.15.46:0/1189633900 conn(0x55b8560524e0 msgr2=0x55b85642d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 shutdown_connections 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 1 -- 172.21.15.46:0/1189633900 wait complete. 2024-05-09T03:15:02.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.539+0000 7f27bcb72740 20 client.6642 trim_cache size 0 max 16384 2024-05-09T03:15:02.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.554+0000 7f27bcb72740 1 Processor -- start 2024-05-09T03:15:02.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.554+0000 7f27bcb72740 1 -- start start 2024-05-09T03:15:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85603d6b0 0x55b856440a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 0x55b856469000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27bcb72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27b02009f0 con 0x55b85605e390 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27b01ffa30 con 0x55b8560354d0 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27b0154450 con 0x55b85603d6b0 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 0x55b856469000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 0x55b856469000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41264/0 (socket says 172.21.15.46:41264) 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 -- 172.21.15.46:0/2659146014 learned_addr learned my addr 172.21.15.46:0/2659146014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27af7fe640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27aeffd640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85603d6b0 0x55b856440a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 -- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85603d6b0 msgr2=0x55b856440a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85603d6b0 0x55b856440a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 -- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 msgr2=0x55b856427200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27affff640 1 -- 172.21.15.46:0/2659146014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b85600ef60 con 0x55b85605e390 2024-05-09T03:15:02.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27aeffd640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85603d6b0 0x55b856440a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.556+0000 7f27af7fe640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.557+0000 7f27affff640 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 0x55b856469000 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f27a8060c50 tx=0x7f27a8015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cd27a11c98a7db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.557+0000 7f2794ff9640 1 -- 172.21.15.46:0/2659146014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a807a070 con 0x55b85605e390 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.557+0000 7f2794ff9640 1 -- 172.21.15.46:0/2659146014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a807a210 con 0x55b85605e390 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.557+0000 7f2794ff9640 1 -- 172.21.15.46:0/2659146014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a80607d0 con 0x55b85605e390 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.557+0000 7f27bcb72740 1 -- 172.21.15.46:0/2659146014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 msgr2=0x55b856469000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.557+0000 7f27bcb72740 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 0x55b856469000 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f27a8060c50 tx=0x7f27a8015040 comp rx=0 tx=0).stop 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 -- 172.21.15.46:0/2659146014 shutdown_connections 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b85605e390 0x55b856469000 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85603d6b0 0x55b856440a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 --2- 172.21.15.46:0/2659146014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8560354d0 0x55b856427200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 -- 172.21.15.46:0/2659146014 >> 172.21.15.46:0/2659146014 conn(0x55b8560524e0 msgr2=0x55b856440100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 -- 172.21.15.46:0/2659146014 shutdown_connections 2024-05-09T03:15:02.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 -- 172.21.15.46:0/2659146014 wait complete. 2024-05-09T03:15:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.558+0000 7f27bcb72740 1 -- start start 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.559+0000 7f27bcb72740 1 Processor -- start 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b8564b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856456ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b8563fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27b0154aa0 con 0x55b8560354d0 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b8563ff830 con 0x55b85603d6b0 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b856469510 con 0x55b85605e390 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27bcb72740 10 client.0 _ll_register_callbacks cb 0x7f27a800af00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b8564b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b8564b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41276/0 (socket says 172.21.15.46:41276) 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27af7fe640 1 -- 172.21.15.46:0/3494526622 learned_addr learned my addr 172.21.15.46:0/3494526622 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27aeffd640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856456ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.560+0000 7f27affff640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b8563fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27af7fe640 1 -- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 msgr2=0x55b8563fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27af7fe640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b8563fcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27af7fe640 1 -- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 msgr2=0x55b856456ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27af7fe640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856456ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27af7fe640 1 -- 172.21.15.46:0/3494526622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b856469000 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27af7fe640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b8564b1030 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x55b856441e20 tx=0x7f27a0017070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a083a597c46a49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f278e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f278e7fc640 1 -- 172.21.15.46:0/3494526622 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a001b070 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f278e7fc640 1 -- 172.21.15.46:0/3494526622 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f27a0037ab0 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27a807a210 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f278e7fc640 1 -- 172.21.15.46:0/3494526622 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f27a0037cb0 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27bcb72740 10 client.6648 fetch_fsmap learned FSMap version 70 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) -- 0x55b856469250 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f278e7fc640 1 -- 172.21.15.46:0/3494526622 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a807a210 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.561+0000 7f278e7fc640 1 -- 172.21.15.46:0/3494526622 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 70) ==== 3494+0+0 (secure 0 0 0) 0x7f27a002e590 con 0x55b8560354d0 2024-05-09T03:15:02.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27bcb72740 10 client.6648 fetch_fsmap finished waiting for FSMap version 70 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27bcb72740 10 client.6648 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27bcb72740 20 client.6648 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27bcb72740 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b856460290 0x55b85643dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27bcb72740 4 client.6648 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 --> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b85636dc20 con 0x55b856460290 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27aeffd640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b856460290 0x55b85643dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f278e7fc640 10 client.6648 ms_handle_connect on v2:172.21.15.46:6832/131652171 2024-05-09T03:15:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.562+0000 7f27aeffd640 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b856460290 0x55b85643dcb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b237f696a75f0f00 server_cookie=7fcaab48a63da96e in_seq=0 out_seq=0 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f278e7fc640 1 -- 172.21.15.46:0/3494526622 <== mds.0 v2:172.21.15.46:6832/131652171 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55b85636dc20 con 0x55b856460290 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f278e7fc640 10 client.6648 handle_command_reply: tid=0 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 client.6648 shutdown 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 msgr2=0x55b8564b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b8564b1030 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x55b856441e20 tx=0x7f27a0017070 comp rx=0 tx=0).stop 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 shutdown_connections 2024-05-09T03:15:02.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:6832/131652171,v1:172.21.15.46:6834/131652171] conn(0x55b856460290 0x55b85643dcb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b85605e390 0x55b8563fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f278e7fc640 0 client.6648 ms_handle_reset on v2:172.21.15.46:6832/131652171 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85603d6b0 0x55b856456ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 --2- 172.21.15.46:0/3494526622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b8560354d0 0x55b8564b1030 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 >> 172.21.15.46:0/3494526622 conn(0x55b8560524e0 msgr2=0x55b856429b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 shutdown_connections 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 1 -- 172.21.15.46:0/3494526622 wait complete. 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.576+0000 7f27bcb72740 20 client.6648 trim_cache size 0 max 16384 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0 2024-05-09T03:15:02.578 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:15:02.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f279c0762d0 msgr2=0x7f279c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f279c0762d0 0x7f279c0787e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f27b00b6240 tx=0x7f27a0009190 comp rx=0 tx=0).stop 2024-05-09T03:15:02.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 msgr2=0x7f27b0150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 0x7f27b0150f40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f27a8001f10 tx=0x7f27a8007a60 comp rx=0 tx=0).stop 2024-05-09T03:15:02.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 shutdown_connections 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f279c0762d0 0x7f279c0787e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27b0150b30 0x7f27b0150f40 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27b014dd10 0x7f27b014e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 --2- 172.21.15.46:0/3524749364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2798003f90 0x7f27b014b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 >> 172.21.15.46:0/3524749364 conn(0x7f27b00b3730 msgr2=0x7f27b015af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 shutdown_connections 2024-05-09T03:15:02.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.577+0000 7f27b5cbd640 1 -- 172.21.15.46:0/3524749364 wait complete. 2024-05-09T03:15:02.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-05-09T03:15:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.706+0000 7f6a2707e640 1 Processor -- start 2024-05-09T03:15:02.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.706+0000 7f6a2707e640 1 -- start start 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a2707e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a200b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a2707e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a200b7ca0 0x7f6a200d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a2707e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a200b80b0 con 0x7f6a200d3df0 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a2707e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6a200d3480 con 0x7f6a200b7ca0 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a2707e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6a200b5550 con 0x7f6a200b5140 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a200b7ca0 0x7f6a200d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a200b7ca0 0x7f6a200d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41278/0 (socket says 172.21.15.46:41278) 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 -- 172.21.15.46:0/944944258 learned_addr learned my addr 172.21.15.46:0/944944258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a24e1a640 1 --2- 172.21.15.46:0/944944258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a200b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 -- 172.21.15.46:0/944944258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 msgr2=0x7f6a200b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 --2- 172.21.15.46:0/944944258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a200b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 -- 172.21.15.46:0/944944258 >> v1:172.21.15.88:6789/0 conn(0x7f6a200d3df0 legacy=0x7f6a20163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.708+0000 7f6a1ffff640 1 -- 172.21.15.46:0/944944258 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a200d44e0 con 0x7f6a200b7ca0 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.709+0000 7f6a1ffff640 1 --2- 172.21.15.46:0/944944258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200d38b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009870 tx=0x7f6a1402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2527f77a315b2bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.709+0000 7f6a1f7fe640 1 -- 172.21.15.46:0/944944258 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14037070 con 0x7f6a200b7ca0 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.709+0000 7f6a1f7fe640 1 -- 172.21.15.46:0/944944258 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a14032440 con 0x7f6a200b7ca0 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.709+0000 7f6a1f7fe640 1 -- 172.21.15.46:0/944944258 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14032810 con 0x7f6a200b7ca0 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 -- 172.21.15.46:0/944944258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 msgr2=0x7f6a200d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 --2- 172.21.15.46:0/944944258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200d38b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009870 tx=0x7f6a1402a250 comp rx=0 tx=0).stop 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 -- 172.21.15.46:0/944944258 shutdown_connections 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 --2- 172.21.15.46:0/944944258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200d38b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 --2- 172.21.15.46:0/944944258 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a200b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 -- 172.21.15.46:0/944944258 >> 172.21.15.46:0/944944258 conn(0x7f6a2008d3f0 msgr2=0x7f6a2008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 -- 172.21.15.46:0/944944258 shutdown_connections 2024-05-09T03:15:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.710+0000 7f6a2707e640 1 -- 172.21.15.46:0/944944258 wait complete. 2024-05-09T03:15:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.712+0000 7f6a2707e640 1 Processor -- start 2024-05-09T03:15:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.712+0000 7f6a2707e640 1 -- start start 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2707e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a201586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2707e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2707e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a200d3df0 0x7f6a2015aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2707e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6a200d4d80 con 0x7f6a200b7ca0 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2707e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6a200d4aa0 con 0x7f6a200d3df0 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2707e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6a200d4200 con 0x7f6a200b5140 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41286/0 (socket says 172.21.15.46:41286) 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 -- 172.21.15.46:0/548941141 learned_addr learned my addr 172.21.15.46:0/548941141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a2561b640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a200d3df0 0x7f6a2015aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a24e1a640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a201586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 -- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 msgr2=0x7f6a201586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a201586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 -- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a200d3df0 msgr2=0x7f6a2015aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a200d3df0 0x7f6a2015aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.713+0000 7f6a1ffff640 1 -- 172.21.15.46:0/548941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a20156a60 con 0x7f6a200b7ca0 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a24e1a640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a201586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a1ffff640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200cf5c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009870 tx=0x7f6a140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14041020 con 0x7f6a200b7ca0 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a140306c0 con 0x7f6a200b7ca0 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a20156d40 con 0x7f6a200b7ca0 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14037040 con 0x7f6a200b7ca0 2024-05-09T03:15:02.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.714+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a20157300 con 0x7f6a200b7ca0 2024-05-09T03:15:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.715+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2006a270 con 0x7f6a200b7ca0 2024-05-09T03:15:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.716+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6a14044020 con 0x7f6a200b7ca0 2024-05-09T03:15:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.716+0000 7f6a1d7fa640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6a0c0762d0 0x7f6a0c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.716+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f6a140b7e40 con 0x7f6a200b7ca0 2024-05-09T03:15:02.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.717+0000 7f6a24e1a640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6a0c0762d0 0x7f6a0c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:02.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.717+0000 7f6a24e1a640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6a0c0762d0 0x7f6a0c0787e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006fd0 tx=0x7f6a10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:02.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.721+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6a1404b030 con 0x7f6a200b7ca0 2024-05-09T03:15:02.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.910+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) -- 0x7f6a2006d160 con 0x7f6a200b7ca0 2024-05-09T03:15:02.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.914+0000 7f6a1d7fa640 1 -- 172.21.15.46:0/548941141 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v23)=0 v23) ==== 138+0+0 (secure 0 0 0) 0x7f6a200b80b0 con 0x7f6a200b7ca0 2024-05-09T03:15:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6a0c0762d0 msgr2=0x7f6a0c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6a0c0762d0 0x7f6a0c0787e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006fd0 tx=0x7f6a10008040 comp rx=0 tx=0).stop 2024-05-09T03:15:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 msgr2=0x7f6a200cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200cf5c0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009870 tx=0x7f6a140046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 shutdown_connections 2024-05-09T03:15:02.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a200d3df0 0x7f6a2015aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6a0c0762d0 0x7f6a0c0787e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a200b7ca0 0x7f6a200cf5c0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 --2- 172.21.15.46:0/548941141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6a200b5140 0x7f6a201586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 >> 172.21.15.46:0/548941141 conn(0x7f6a2008d3f0 msgr2=0x7f6a2008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 shutdown_connections 2024-05-09T03:15:02.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:02.916+0000 7f6a2707e640 1 -- 172.21.15.46:0/548941141 wait complete. 2024-05-09T03:15:06.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:15:06.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:15:07.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo daemon-helper kill ceph --cluster ceph -w 2024-05-09T03:15:07.932 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'timeout 60 mv a/b a/z') 2024-05-09T03:15:07.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-05-09T03:15:08.001 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:15:08.000+0000 7f468d076640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.001 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:15:08.000+0000 7f468d076640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f468d076640 time 2024-05-09T03:15:08.001120+0000 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f4692e4acfc] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.002 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.003 INFO:tasks.ceph.mds.a.smithi046.stderr:*** Caught signal (Aborted) ** 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7f468d076640 thread_name:ms_dispatch 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:15:08.001+0000 7f468d076640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f468d076640 time 2024-05-09T03:15:08.001120+0000 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f4692e4acfc] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.004 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.005 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f469243e6f0] 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f469248b94c] 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f4692e4ae6e] 2024-05-09T03:15:08.006 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.007 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr:2024-05-09T03:15:08.002+0000 7f468d076640 -1 *** Caught signal (Aborted) ** 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7f468d076640 thread_name:ms_dispatch 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f469243e6f0] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f469248b94c] 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:15:08.008 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f4692e4ae6e] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.009 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T03:15:08.010 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.011 INFO:tasks.ceph.mds.a.smithi046.stderr: -3805> 2024-05-09T03:14:55.226+0000 7f468a871640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: -8> 2024-05-09T03:15:08.000+0000 7f468d076640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: -6> 2024-05-09T03:15:08.000+0000 7f468d076640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: -1> 2024-05-09T03:15:08.001+0000 7f468d076640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f468d076640 time 2024-05-09T03:15:08.001120+0000 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f4692e4acfc] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.013 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.014 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 0> 2024-05-09T03:15:08.002+0000 7f468d076640 -1 *** Caught signal (Aborted) ** 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7f468d076640 thread_name:ms_dispatch 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f469243e6f0] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f469248b94c] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f4692e4ae6e] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.015 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.016 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.017 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.017 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.017 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.017 INFO:tasks.ceph.mds.a.smithi046.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T03:15:08.017 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.017 INFO:tasks.ceph.mds.a.smithi046.stderr: -3805> 2024-05-09T03:14:55.226+0000 7f468a871640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: -8> 2024-05-09T03:15:08.000+0000 7f468d076640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: -6> 2024-05-09T03:15:08.000+0000 7f468d076640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: -1> 2024-05-09T03:15:08.001+0000 7f468d076640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f468d076640 time 2024-05-09T03:15:08.001120+0000 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3594-gc9da4049/rpm/el9/BUILD/ceph-19.0.0-3594-gc9da4049/src/mds/MDSRank.cc: 951: ceph_abort_msg("abort() called") 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f4692e4acfc] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.019 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.020 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 0> 2024-05-09T03:15:08.002+0000 7f468d076640 -1 *** Caught signal (Aborted) ** 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: in thread 7f468d076640 thread_name:ms_dispatch 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f469243e6f0] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x7f469248b94c] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 3: raise() 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 4: abort() 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x241) [0x7f4692e4ae6e] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x5f) [0x56106615e90b] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 7: (CDentry::check_corruption(bool)+0x530) [0x561066424640] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 8: ceph-mds(+0x17eb86) [0x5610661c3b86] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 9: ceph-mds(+0x17ef86) [0x5610661c3f86] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 10: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x249b) [0x561066266ca7] 2024-05-09T03:15:08.021 INFO:tasks.ceph.mds.a.smithi046.stderr: 11: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2fab) [0x56106626a553] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 12: (Server::dispatch_client_request(boost::intrusive_ptr const&)+0xeb1) [0x56106622b8c9] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 13: (Server::handle_client_request(boost::intrusive_ptr const&)+0xa2a) [0x56106622c5d2] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 14: (Server::dispatch(boost::intrusive_ptr const&)+0xc0) [0x561066217012] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 15: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x5ac) [0x561066168ecc] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56106616d829] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56106616e150] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2c4) [0x5610661504ae] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 19: /usr/lib64/ceph/libceph-common.so.2(+0x395f01) [0x7f4692f95f01] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 20: (DispatchQueue::entry()+0x837) [0x7f4692f96a6b] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 21: /usr/lib64/ceph/libceph-common.so.2(+0x475fc1) [0x7f4693075fc1] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 22: (Thread::entry_wrapper()+0x43) [0x7f4692e24ad7] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 23: (Thread::_entry_func(void*)+0xd) [0x7f4692e24af3] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 24: /lib64/libc.so.6(+0x89c02) [0x7f4692489c02] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: 25: /lib64/libc.so.6(+0x10ec40) [0x7f469250ec40] 2024-05-09T03:15:08.022 INFO:tasks.ceph.mds.a.smithi046.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-09T03:15:08.023 INFO:tasks.ceph.mds.a.smithi046.stderr: 2024-05-09T03:15:08.075 INFO:teuthology.orchestra.run.smithi046.stdout:{"mds_beacon_grace":"15.000000"} 2024-05-09T03:15:08.075 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-05-09T03:15:08.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:08.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.091+0000 7f9ae7a53640 1 Processor -- start 2024-05-09T03:15:08.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.091+0000 7f9ae7a53640 1 -- start start 2024-05-09T03:15:08.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.092+0000 7f9ae7a53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ae00d2200 0x7f9ae00b7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.092+0000 7f9ae7a53640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae00d2900 con 0x7f9ae00d08c0 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ae7a53640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ae00d2be0 con 0x7f9ae00ce4e0 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ae7a53640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9ae00d39a0 con 0x7f9ae00d2200 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ae4fee640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9ae00d08c0 0x7f9ae00b6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44220/0 (socket says 172.21.15.46:44220) 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ae4fee640 1 -- 172.21.15.46:0/4112947266 learned_addr learned my addr 172.21.15.46:0/4112947266 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/4112947266 >> v2:172.21.15.88:3300/0 conn(0x7f9ae00d2200 0x7f9ae00b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783643425 0 0) 0x7f9ae00d2900 con 0x7f9ae00d08c0 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9acc003520 con 0x7f9ae00d08c0 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 8369153 0 0) 0x7f9acc003520 con 0x7f9ae00d08c0 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 >> v2:172.21.15.88:3300/0 conn(0x7f9ae00d2200 msgr2=0x7f9ae00b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 --2- 172.21.15.46:0/4112947266 >> v2:172.21.15.88:3300/0 conn(0x7f9ae00d2200 0x7f9ae00b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 >> v1:172.21.15.88:6789/0 conn(0x7f9ae00ce4e0 legacy=0x7f9ae00d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae00d2ec0 con 0x7f9ae00d08c0 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f9ad00016f0 con 0x7f9ae00d08c0 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/4112947266 >> v2:172.21.15.88:3300/0 conn(0x7f9ae00d2200 0x7f9ae00b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 >> v1:172.21.15.46:6789/0 conn(0x7f9ae00d08c0 legacy=0x7f9ae00b6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9acc003a50 0x7f9acc003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9acc004800 0x7f9acc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9acc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae00d39a0 con 0x7f9acc005250 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9acc00c1e0 con 0x7f9acc004800 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.093+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9acc00c3e0 con 0x7f9acc003a50 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9acc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae4fee640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9acc004800 0x7f9acc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae57ef640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9acc003a50 0x7f9acc003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae5ff0640 1 -- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9acc003a50 msgr2=0x7f9acc003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9acc003a50 0x7f9acc003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae5ff0640 1 -- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9acc004800 msgr2=0x7f9acc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9acc004800 0x7f9acc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.094+0000 7f9ae5ff0640 1 -- 172.21.15.46:0/4112947266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9acc003520 con 0x7f9acc005250 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.095+0000 7f9ae5ff0640 1 -- 172.21.15.46:0/4112947266 reap_dead start 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.095+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9acc00b9c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800e350 tx=0x7f9acc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3148acc8ef6d82f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.095+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8009de0 con 0x7f9acc005250 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.095+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ad802e680 con 0x7f9acc005250 2024-05-09T03:15:08.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.095+0000 7f9ad7fff640 1 -- 172.21.15.46:0/4112947266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad802ea70 con 0x7f9acc005250 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 -- 172.21.15.46:0/4112947266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 msgr2=0x7f9acc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9acc00b9c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800e350 tx=0x7f9acc005830 comp rx=0 tx=0).stop 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 -- 172.21.15.46:0/4112947266 shutdown_connections 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 --2- 172.21.15.46:0/4112947266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9acc00b9c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 -- 172.21.15.46:0/4112947266 >> 172.21.15.46:0/4112947266 conn(0x7f9ae00b5670 msgr2=0x7f9ae00b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 -- 172.21.15.46:0/4112947266 shutdown_connections 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.096+0000 7f9ae7a53640 1 -- 172.21.15.46:0/4112947266 wait complete. 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.098+0000 7f9ae7a53640 1 Processor -- start 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.098+0000 7f9ae7a53640 1 -- start start 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae7a53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9ae01f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae7a53640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ae0157d10 0x7f9ae00b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae7a53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ae00b7930 0x7f9ae0156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae7a53640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae00d31a0 con 0x7f9acc005250 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae7a53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae00b3420 con 0x7f9ae00b7930 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae7a53640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9ae00d3480 con 0x7f9ae0157d10 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae57ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9ae01f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae57ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9ae01f41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37472/0 (socket says 172.21.15.46:37472) 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae57ef640 1 -- 172.21.15.46:0/3882379379 learned_addr learned my addr 172.21.15.46:0/3882379379 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae4fee640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ae0157d10 0x7f9ae00b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae5ff0640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ae00b7930 0x7f9ae0156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae57ef640 1 -- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ae0157d10 msgr2=0x7f9ae00b73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.099+0000 7f9ae57ef640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ae0157d10 0x7f9ae00b73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ae57ef640 1 -- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ae00b7930 msgr2=0x7f9ae0156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ae57ef640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ae00b7930 0x7f9ae0156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ae57ef640 1 -- 172.21.15.46:0/3882379379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae00b7d40 con 0x7f9acc005250 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ae57ef640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9acc005250 0x7f9ae01f41c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f9adc00f820 tx=0x7f9adc00fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc01a070 con 0x7f9acc005250 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc005110 con 0x7f9acc005250 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ae7a53640 1 -- 172.21.15.46:0/3882379379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae01f4700 con 0x7f9acc005250 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc007050 con 0x7f9acc005250 2024-05-09T03:15:08.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.100+0000 7f9ae7a53640 1 -- 172.21.15.46:0/3882379379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae014a140 con 0x7f9acc005250 2024-05-09T03:15:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.101+0000 7f9ae7a53640 1 -- 172.21.15.46:0/3882379379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9ae00b51c0 con 0x7f9acc005250 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.102+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9adc01d070 con 0x7f9acc005250 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.103+0000 7f9ad5ffb640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ac40762d0 0x7f9ac40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.103+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f9adc096c50 con 0x7f9acc005250 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.103+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+511 (secure 0 0 0) 0x7f9ae00b51c0 con 0x7f9acc005250 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.103+0000 7f9ae4fee640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ac40762d0 0x7f9ac40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.103+0000 7f9ae7a53640 1 -- 172.21.15.46:0/3882379379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f9ae0076150 con 0x7f9acc005250 2024-05-09T03:15:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.104+0000 7f9ae4fee640 1 --2- 172.21.15.46:0/3882379379 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ac40762d0 0x7f9ac40787e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00b8210 tx=0x7f9ad0001e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:08.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.193+0000 7fb949289640 1 Processor -- start 2024-05-09T03:15:08.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.193+0000 7fb949289640 1 -- start start 2024-05-09T03:15:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.194+0000 7fb949289640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9440cf480 0x7fb9440b7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.194+0000 7fb949289640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9440ce8c0 0x7fb9440cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.194+0000 7fb949289640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9440cf210 con 0x7fb94415d2a0 2024-05-09T03:15:08.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.194+0000 7fb949289640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb94406d400 con 0x7fb9440cf480 2024-05-09T03:15:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.194+0000 7fb949289640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb9440d1020 con 0x7fb9440ce8c0 2024-05-09T03:15:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9440cf480 0x7fb9440b7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb9437fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9440ce8c0 0x7fb9440cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9440cf480 0x7fb9440b7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:34922/0 (socket says 172.21.15.46:34922) 2024-05-09T03:15:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb93bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb94415d2a0 0x7fb9440af1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:54752/0 (socket says 172.21.15.46:54752) 2024-05-09T03:15:08.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 -- 172.21.15.46:0/1653508406 learned_addr learned my addr 172.21.15.46:0/1653508406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:08.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 -- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9440ce8c0 msgr2=0x7fb9440cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 --2- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9440ce8c0 0x7fb9440cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 -- 172.21.15.46:0/1653508406 >> v1:172.21.15.88:6789/0 conn(0x7fb94415d2a0 legacy=0x7fb9440af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb9427fc640 1 -- 172.21.15.46:0/1653508406 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2579532096 0 0) 0x7fb92c001110 con 0x7fb94415d2a0 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.195+0000 7fb942ffd640 1 -- 172.21.15.46:0/1653508406 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94406cdb0 con 0x7fb9440cf480 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.196+0000 7fb942ffd640 1 --2- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440b7720 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb93400bb10 tx=0x7fb93402c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f70993294f169608 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.196+0000 7fb9427fc640 1 -- 172.21.15.46:0/1653508406 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934036020 con 0x7fb9440cf480 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.196+0000 7fb9427fc640 1 -- 172.21.15.46:0/1653508406 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb934032820 con 0x7fb9440cf480 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.196+0000 7fb9427fc640 1 -- 172.21.15.46:0/1653508406 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934032bf0 con 0x7fb9440cf480 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 -- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 msgr2=0x7fb9440b7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 --2- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440b7720 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb93400bb10 tx=0x7fb93402c920 comp rx=0 tx=0).stop 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 -- 172.21.15.46:0/1653508406 shutdown_connections 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 --2- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9440ce8c0 0x7fb9440cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 --2- 172.21.15.46:0/1653508406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440b7720 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 -- 172.21.15.46:0/1653508406 >> 172.21.15.46:0/1653508406 conn(0x7fb944073d10 msgr2=0x7fb94409b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 -- 172.21.15.46:0/1653508406 shutdown_connections 2024-05-09T03:15:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.197+0000 7fb949289640 1 -- 172.21.15.46:0/1653508406 wait complete. 2024-05-09T03:15:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.199+0000 7fb949289640 1 Processor -- start 2024-05-09T03:15:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.199+0000 7fb949289640 1 -- start start 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb949289640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 0x7fb944156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb949289640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb949289640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb94415d2a0 0x7fb9441520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb949289640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb94415d6b0 con 0x7fb9440ce8c0 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb949289640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb94406caa0 con 0x7fb9440cf480 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb949289640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb9440d0c60 con 0x7fb94415d2a0 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb942ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 0x7fb944156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb942ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 0x7fb944156ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37480/0 (socket says 172.21.15.46:37480) 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb942ffd640 1 -- 172.21.15.46:0/2865159541 learned_addr learned my addr 172.21.15.46:0/2865159541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:08.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb93bfff640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.200+0000 7fb9437fe640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb94415d2a0 0x7fb9441520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb942ffd640 1 -- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb94415d2a0 msgr2=0x7fb9441520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb942ffd640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb94415d2a0 0x7fb9441520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb942ffd640 1 -- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 msgr2=0x7fb9440ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb942ffd640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb942ffd640 1 -- 172.21.15.46:0/2865159541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb944156130 con 0x7fb9440ce8c0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb942ffd640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 0x7fb944156ef0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb93400bb10 tx=0x7fb934004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934036040 con 0x7fb9440ce8c0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb934031cb0 con 0x7fb9440ce8c0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb934031e60 con 0x7fb9440ce8c0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb944156410 con 0x7fb9440ce8c0 2024-05-09T03:15:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.201+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9441569d0 con 0x7fb9440ce8c0 2024-05-09T03:15:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.202+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb94406a270 con 0x7fb9440ce8c0 2024-05-09T03:15:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.203+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb934039020 con 0x7fb9440ce8c0 2024-05-09T03:15:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.203+0000 7fb93b7fe640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9240762d0 0x7fb9240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.204+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fb9340b83b0 con 0x7fb9440ce8c0 2024-05-09T03:15:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.204+0000 7fb93bfff640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9240762d0 0x7fb9240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.204+0000 7fb93bfff640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9240762d0 0x7fb9240787e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb94415bf60 tx=0x7fb92c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.208+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb93402f070 con 0x7fb9440ce8c0 2024-05-09T03:15:08.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.439+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9440aff30 con 0x7fb9440ce8c0 2024-05-09T03:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.440+0000 7fb93b7fe640 1 -- 172.21.15.46:0/2865159541 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5180 (secure 0 0 0) 0x7fb934081620 con 0x7fb9440ce8c0 2024-05-09T03:15:08.441 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:08.441 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:01.510352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:15:08.441 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-05-09T03:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9240762d0 msgr2=0x7fb9240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9240762d0 0x7fb9240787e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb94415bf60 tx=0x7fb92c007de0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 msgr2=0x7fb944156ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 0x7fb944156ef0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb93400bb10 tx=0x7fb934004730 comp rx=0 tx=0).stop 2024-05-09T03:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 shutdown_connections 2024-05-09T03:15:08.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb94415d2a0 0x7fb9441520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9440cf480 0x7fb9440ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9240762d0 0x7fb9240787e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 --2- 172.21.15.46:0/2865159541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9440ce8c0 0x7fb944156ef0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 >> 172.21.15.46:0/2865159541 conn(0x7fb944073d10 msgr2=0x7fb944070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 shutdown_connections 2024-05-09T03:15:08.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.442+0000 7fb949289640 1 -- 172.21.15.46:0/2865159541 wait complete. 2024-05-09T03:15:08.454 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:15:08.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:08.641+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 7 ==== log(1 entries from seq 1 at 2024-05-09T03:15:08.001095+0000) ==== 468+0+0 (secure 0 0 0) 0x7f9adc0951f0 con 0x7f9acc005250 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: cluster: 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: id: 36d71dd3-2926-4712-824e-80cfbafd76c2 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: health: HEALTH_OK 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: services: 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: mon: 3 daemons, quorum a,b,c (age 5m) 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: mgr: x(active, since 5m), standbys: y 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: mds: 1/1 daemons up, 3 standby 2024-05-09T03:15:08.643 INFO:teuthology.orchestra.run.smithi046.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: data: 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: volumes: 1/1 healthy 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: pools: 3 pools, 97 pgs 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: objects: 29 objects, 596 KiB 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: usage: 243 MiB used, 720 GiB / 720 GiB avail 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: pgs: 97 active+clean 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: io: 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: client: 0 B/s rd, 3.8 KiB/s wr, 0 op/s rd, 6 op/s wr 2024-05-09T03:15:08.644 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:08.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:08.645 INFO:teuthology.orchestra.run.smithi046.stdout:2024-05-09T03:15:08.001095+0000 mds.a [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6612:20 nref=7 cr=0x56106cc2d880)) (dversion lock w=1 last_client=6612) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x56106cb69400] 2024-05-09T03:15:08.688 INFO:tasks.ceph.mds.a.smithi046.stderr:daemon-helper: command crashed with signal 6 2024-05-09T03:15:13.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:13.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.584+0000 7f6def38d640 1 Processor -- start 2024-05-09T03:15:13.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.584+0000 7f6def38d640 1 -- start start 2024-05-09T03:15:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6def38d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80b2000 0x7f6de815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6def38d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6de80ccfc0 0x7f6de80d2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6def38d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de80b7a50 con 0x7f6de80cf660 2024-05-09T03:15:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6def38d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6de80cf1d0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6def38d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6de80ceb80 con 0x7f6de80b2000 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6dec928640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6de80cf660 0x7f6de80cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44232/0 (socket says 172.21.15.46:44232) 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6dec928640 1 -- 172.21.15.46:0/3567270173 learned_addr learned my addr 172.21.15.46:0/3567270173 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6ded129640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80b2000 0x7f6de815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6ded92a640 1 --2- 172.21.15.46:0/3567270173 >> v2:172.21.15.88:3300/0 conn(0x7f6de80ccfc0 0x7f6de80d2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3567270173 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816181628 0 0) 0x7f6de80b7a50 con 0x7f6de80cf660 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.586+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3567270173 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6dd4003520 con 0x7f6de80cf660 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.587+0000 7f6ded92a640 1 -- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80b2000 msgr2=0x7f6de815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.587+0000 7f6ded92a640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80b2000 0x7f6de815cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.587+0000 7f6ded92a640 1 -- 172.21.15.46:0/3567270173 >> v1:172.21.15.46:6789/0 conn(0x7f6de80cf660 legacy=0x7f6de80cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.587+0000 7f6ded92a640 1 -- 172.21.15.46:0/3567270173 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de80cee60 con 0x7f6de80ccfc0 2024-05-09T03:15:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.587+0000 7f6ded129640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80b2000 0x7f6de815cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:13.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.587+0000 7f6ded92a640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80ccfc0 0x7f6de80d2190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009d90 tx=0x7f6de002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b9ff55cb6dfd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:13.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3567270173 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0007740 con 0x7f6de80ccfc0 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3567270173 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de00078e0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6ddbfff640 1 -- 172.21.15.46:0/3567270173 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de0007c90 con 0x7f6de80ccfc0 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 -- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80ccfc0 msgr2=0x7f6de80d2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80ccfc0 0x7f6de80d2190 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009d90 tx=0x7f6de002a660 comp rx=0 tx=0).stop 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 -- 172.21.15.46:0/3567270173 shutdown_connections 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80ccfc0 0x7f6de80d2190 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 --2- 172.21.15.46:0/3567270173 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80b2000 0x7f6de815cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 -- 172.21.15.46:0/3567270173 >> 172.21.15.46:0/3567270173 conn(0x7f6de809b7d0 msgr2=0x7f6de80b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 -- 172.21.15.46:0/3567270173 shutdown_connections 2024-05-09T03:15:13.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.588+0000 7f6def38d640 1 -- 172.21.15.46:0/3567270173 wait complete. 2024-05-09T03:15:13.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.590+0000 7f6def38d640 1 Processor -- start 2024-05-09T03:15:13.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.590+0000 7f6def38d640 1 -- start start 2024-05-09T03:15:13.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.591+0000 7f6def38d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80b2000 0x7f6de81f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:13.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.591+0000 7f6def38d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 0x7f6de8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.591+0000 7f6def38d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80cf660 0x7f6de815ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.591+0000 7f6def38d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6de80cfa70 con 0x7f6de80ccfc0 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6def38d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6de80b7d30 con 0x7f6de80b2000 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6def38d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6de8068a60 con 0x7f6de80cf660 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 0x7f6de8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 0x7f6de8157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37484/0 (socket says 172.21.15.46:37484) 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 -- 172.21.15.46:0/342697501 learned_addr learned my addr 172.21.15.46:0/342697501 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6ded129640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80b2000 0x7f6de81f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6ded92a640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80cf660 0x7f6de815ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 -- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80cf660 msgr2=0x7f6de815ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80cf660 0x7f6de815ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 -- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80b2000 msgr2=0x7f6de81f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80b2000 0x7f6de81f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 -- 172.21.15.46:0/342697501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de81f30c0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dec928640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 0x7f6de8157eb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00c390 tx=0x7f6ddc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc017070 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.592+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ddc005ce0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.593+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de815d3a0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.593+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc0072d0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.593+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de815d960 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.594+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de806a270 con 0x7f6de80ccfc0 2024-05-09T03:15:13.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.595+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ddc01a070 con 0x7f6de80ccfc0 2024-05-09T03:15:13.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.595+0000 7f6dd9ffb640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6dcc0762d0 0x7f6dcc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:13.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.596+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f6ddc096880 con 0x7f6de80ccfc0 2024-05-09T03:15:13.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.596+0000 7f6ded129640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6dcc0762d0 0x7f6dcc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:13.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.596+0000 7f6ded129640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6dcc0762d0 0x7f6dcc0787e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009560 tx=0x7f6de4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.600+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6de81f30c0 con 0x7f6de80ccfc0 2024-05-09T03:15:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.829+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6de815d130 con 0x7f6de80ccfc0 2024-05-09T03:15:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.830+0000 7f6dd9ffb640 1 -- 172.21.15.46:0/342697501 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5180 (secure 0 0 0) 0x7f6ddc05fb00 con 0x7f6de80ccfc0 2024-05-09T03:15:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:13.831 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:01.510352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:15:13.831 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6dcc0762d0 msgr2=0x7f6dcc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6dcc0762d0 0x7f6dcc0787e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009560 tx=0x7f6de4009190 comp rx=0 tx=0).stop 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 msgr2=0x7f6de8157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 0x7f6de8157eb0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00c390 tx=0x7f6ddc00c850 comp rx=0 tx=0).stop 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 shutdown_connections 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6de80cf660 0x7f6de815ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6dcc0762d0 0x7f6dcc0787e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6de80ccfc0 0x7f6de8157eb0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 --2- 172.21.15.46:0/342697501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6de80b2000 0x7f6de81f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 >> 172.21.15.46:0/342697501 conn(0x7f6de809b7d0 msgr2=0x7f6de8158d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:13.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 shutdown_connections 2024-05-09T03:15:13.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:13.832+0000 7f6def38d640 1 -- 172.21.15.46:0/342697501 wait complete. 2024-05-09T03:15:13.844 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:15:14.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:14.655+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 8 ==== log(4 entries from seq 118 at 2024-05-09T03:15:14.521874+0000) ==== 1122+0+0 (secure 0 0 0) 0x7f9adc05c1e0 con 0x7f9acc005250 2024-05-09T03:15:15.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-05-09T03:15:15.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:15.656+0000 7f9ad5ffb640 1 -- 172.21.15.46:0/3882379379 <== mon.0 v2:172.21.15.46:3300/0 9 ==== log(2 entries from seq 684 at 2024-05-09T03:15:14.654675+0000) ==== 559+0+0 (secure 0 0 0) 0x7f9adc0601b0 con 0x7f9acc005250 2024-05-09T03:15:18.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:18.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.972+0000 7f11dbc9f640 1 Processor -- start 2024-05-09T03:15:18.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.972+0000 7f11dbc9f640 1 -- start start 2024-05-09T03:15:18.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11dbc9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 0x7f11d40aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:18.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11dbc9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11d4070d80 0x7f11d4073870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11dbc9f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d415d980 con 0x7f11d40af0c0 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11dbc9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f11d415dc60 con 0x7f11d4070d80 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11dbc9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11d415df40 con 0x7f11d40b4ff0 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11d9a3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 0x7f11d40aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11da23c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11d4070d80 0x7f11d4073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11da23c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11d4070d80 0x7f11d4073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:43596/0 (socket says 172.21.15.46:43596) 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11d923a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f11d40af0c0 0x7f11d4070670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34334/0 (socket says 172.21.15.46:34334) 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11d9a3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 0x7f11d40aeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:42444/0 (socket says 172.21.15.46:42444) 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.974+0000 7f11da23c640 1 -- 172.21.15.46:0/1733634275 learned_addr learned my addr 172.21.15.46:0/1733634275 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.975+0000 7f11da23c640 1 -- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 msgr2=0x7f11d40aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.975+0000 7f11da23c640 1 --2- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 0x7f11d40aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.975+0000 7f11da23c640 1 -- 172.21.15.46:0/1733634275 >> v1:172.21.15.88:6789/0 conn(0x7f11d40af0c0 legacy=0x7f11d4070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:18.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.975+0000 7f11da23c640 1 -- 172.21.15.46:0/1733634275 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d415e220 con 0x7f11d4070d80 2024-05-09T03:15:18.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.975+0000 7f11d9a3b640 1 --2- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 0x7f11d40aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.976+0000 7f11da23c640 1 --2- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d4070d80 0x7f11d4073870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009870 tx=0x7f11cc02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab0f1e342efc07b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.976+0000 7f11d8a39640 1 -- 172.21.15.46:0/1733634275 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc037070 con 0x7f11d4070d80 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.976+0000 7f11d8a39640 1 -- 172.21.15.46:0/1733634275 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11cc032440 con 0x7f11d4070d80 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.976+0000 7f11d8a39640 1 -- 172.21.15.46:0/1733634275 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11cc032810 con 0x7f11d4070d80 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d4070d80 msgr2=0x7f11d4073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d4070d80 0x7f11d4073870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f11cc009870 tx=0x7f11cc02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1733634275 shutdown_connections 2024-05-09T03:15:18.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d4070d80 0x7f11d4073870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:18.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1733634275 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d40b4ff0 0x7f11d40aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:18.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1733634275 >> 172.21.15.46:0/1733634275 conn(0x7f11d40b3910 msgr2=0x7f11d40b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:18.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1733634275 shutdown_connections 2024-05-09T03:15:18.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.977+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1733634275 wait complete. 2024-05-09T03:15:18.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.978+0000 7f11dbc9f640 1 Processor -- start 2024-05-09T03:15:18.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.979+0000 7f11dbc9f640 1 -- start start 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11dbc9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 0x7f11d41f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11dbc9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d40b4ff0 0x7f11d41f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11dbc9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d409e250 0x7f11d415fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11dbc9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11d415d6c0 con 0x7f11d40af0c0 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11dbc9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11d415d540 con 0x7f11d40b4ff0 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11dbc9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11d40b1310 con 0x7f11d409e250 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 0x7f11d41f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 0x7f11d41f3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47142/0 (socket says 172.21.15.46:47142) 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 -- 172.21.15.46:0/1423592412 learned_addr learned my addr 172.21.15.46:0/1423592412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d923a640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d40b4ff0 0x7f11d41f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11da23c640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d409e250 0x7f11d415fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 -- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d409e250 msgr2=0x7f11d415fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:18.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d409e250 0x7f11d415fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 -- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d40b4ff0 msgr2=0x7f11d41f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d40b4ff0 0x7f11d41f5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.980+0000 7f11d9a3b640 1 -- 172.21.15.46:0/1423592412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d41f6320 con 0x7f11d40af0c0 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.981+0000 7f11d9a3b640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 0x7f11d41f3460 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f11d000efc0 tx=0x7f11d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.981+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0018020 con 0x7f11d40af0c0 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.981+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d0014680 con 0x7f11d40af0c0 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.981+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d41f6810 con 0x7f11d40af0c0 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.981+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0014a50 con 0x7f11d40af0c0 2024-05-09T03:15:18.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.981+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d40c6cb0 con 0x7f11d40af0c0 2024-05-09T03:15:18.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.982+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d406a270 con 0x7f11d40af0c0 2024-05-09T03:15:18.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.983+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f11d0005d10 con 0x7f11d40af0c0 2024-05-09T03:15:18.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.983+0000 7f11ca7fc640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11b80762d0 0x7f11b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:18.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.983+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f11d0098800 con 0x7f11d40af0c0 2024-05-09T03:15:18.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.984+0000 7f11d923a640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11b80762d0 0x7f11b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:18.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.984+0000 7f11d923a640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11b80762d0 0x7f11b80787e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f11c4000f60 tx=0x7f11c4000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:18.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:18.987+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11d0022030 con 0x7f11d40af0c0 2024-05-09T03:15:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.217+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11d415f5c0 con 0x7f11d40af0c0 2024-05-09T03:15:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.218+0000 7f11ca7fc640 1 -- 172.21.15.46:0/1423592412 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5180 (secure 0 0 0) 0x7f11d0010040 con 0x7f11d40af0c0 2024-05-09T03:15:19.220 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:19.220 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:01.510352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:15:19.220 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11b80762d0 msgr2=0x7f11b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11b80762d0 0x7f11b80787e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f11c4000f60 tx=0x7f11c4000bc0 comp rx=0 tx=0).stop 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 msgr2=0x7f11d41f3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 0x7f11d41f3460 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f11d000efc0 tx=0x7f11d000c490 comp rx=0 tx=0).stop 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 shutdown_connections 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d409e250 0x7f11d415fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:19.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d40b4ff0 0x7f11d41f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:19.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11b80762d0 0x7f11b80787e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:19.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 --2- 172.21.15.46:0/1423592412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d40af0c0 0x7f11d41f3460 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:19.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 >> 172.21.15.46:0/1423592412 conn(0x7f11d40b3910 msgr2=0x7f11d415e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:19.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 shutdown_connections 2024-05-09T03:15:19.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:19.220+0000 7f11dbc9f640 1 -- 172.21.15.46:0/1423592412 wait complete. 2024-05-09T03:15:19.233 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:15:21.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-05-09T03:15:24.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:24.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.363+0000 7f1a95a0c640 1 Processor -- start 2024-05-09T03:15:24.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.363+0000 7f1a95a0c640 1 -- start start 2024-05-09T03:15:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a95a0c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a901613c0 0x7f1a900b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a95a0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 0x7f1a90163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a95a0c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a900b8110 con 0x7f1a900b5210 2024-05-09T03:15:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a95a0c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1a900b5620 con 0x7f1a901613c0 2024-05-09T03:15:24.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a95a0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a901609a0 con 0x7f1a900b7d00 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 0x7f1a90163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 0x7f1a90163e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:42474/0 (socket says 172.21.15.46:42474) 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a8e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a901613c0 0x7f1a900b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/1414515597 learned_addr learned my addr 172.21.15.46:0/1414515597 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.365+0000 7f1a8effd640 1 --1- 172.21.15.46:0/1414515597 >> v1:172.21.15.88:6789/0 conn(0x7f1a900b5210 0x7f1a900d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34348/0 (socket says 172.21.15.46:34348) 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.366+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/1414515597 >> v2:172.21.15.88:3300/0 conn(0x7f1a901613c0 msgr2=0x7f1a900b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.366+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/1414515597 >> v2:172.21.15.88:3300/0 conn(0x7f1a901613c0 0x7f1a900b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.366+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/1414515597 >> v1:172.21.15.88:6789/0 conn(0x7f1a900b5210 legacy=0x7f1a900d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.366+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/1414515597 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a901617d0 con 0x7f1a900b7d00 2024-05-09T03:15:24.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.366+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/1414515597 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 0x7f1a90163e60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009e50 tx=0x7f1a8802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f732a27cddccbed5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a8dffb640 1 -- 172.21.15.46:0/1414515597 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88036070 con 0x7f1a900b7d00 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a8dffb640 1 -- 172.21.15.46:0/1414515597 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88031600 con 0x7f1a900b7d00 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a8dffb640 1 -- 172.21.15.46:0/1414515597 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a880319d0 con 0x7f1a900b7d00 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 -- 172.21.15.46:0/1414515597 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 msgr2=0x7f1a90163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/1414515597 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 0x7f1a90163e60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009e50 tx=0x7f1a8802a710 comp rx=0 tx=0).stop 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 -- 172.21.15.46:0/1414515597 shutdown_connections 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/1414515597 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b7d00 0x7f1a90163e60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/1414515597 >> v2:172.21.15.88:3300/0 conn(0x7f1a901613c0 0x7f1a900b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 -- 172.21.15.46:0/1414515597 >> 172.21.15.46:0/1414515597 conn(0x7f1a900d29a0 msgr2=0x7f1a900d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 -- 172.21.15.46:0/1414515597 shutdown_connections 2024-05-09T03:15:24.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.367+0000 7f1a95a0c640 1 -- 172.21.15.46:0/1414515597 wait complete. 2024-05-09T03:15:24.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.369+0000 7f1a95a0c640 1 Processor -- start 2024-05-09T03:15:24.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.369+0000 7f1a95a0c640 1 -- start start 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a95a0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b5210 0x7f1a901f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a95a0c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a900b7d00 0x7f1a90157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a95a0c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 0x7f1a900ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a95a0c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1a90160bc0 con 0x7f1a901613c0 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a95a0c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1a900d5560 con 0x7f1a900b7d00 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a95a0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a90160ea0 con 0x7f1a900b5210 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 0x7f1a900ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 0x7f1a900ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47154/0 (socket says 172.21.15.46:47154) 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/4053274117 learned_addr learned my addr 172.21.15.46:0/4053274117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8effd640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b5210 0x7f1a901f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:24.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8e7fc640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a900b7d00 0x7f1a90157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b5210 msgr2=0x7f1a901f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b5210 0x7f1a901f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a900b7d00 msgr2=0x7f1a90157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a900b7d00 0x7f1a90157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.371+0000 7f1a8f7fe640 1 -- 172.21.15.46:0/4053274117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a901563c0 con 0x7f1a901613c0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.372+0000 7f1a8f7fe640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 0x7f1a900ccdf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009e50 tx=0x7f1a88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.372+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88036050 con 0x7f1a901613c0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.372+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88042070 con 0x7f1a901613c0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.372+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88009510 con 0x7f1a901613c0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.372+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a90156660 con 0x7f1a901613c0 2024-05-09T03:15:24.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.372+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a900cbf00 con 0x7f1a901613c0 2024-05-09T03:15:24.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.373+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a9006a270 con 0x7f1a901613c0 2024-05-09T03:15:24.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.373+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a88007bf0 con 0x7f1a901613c0 2024-05-09T03:15:24.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.374+0000 7f1a77fff640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a7c0762d0 0x7f1a7c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:24.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.374+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f1a880b82c0 con 0x7f1a901613c0 2024-05-09T03:15:24.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.374+0000 7f1a8effd640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a7c0762d0 0x7f1a7c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:24.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.375+0000 7f1a8effd640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a7c0762d0 0x7f1a7c0787e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1a80000f60 tx=0x7f1a80000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:24.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.379+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1a88081590 con 0x7f1a901613c0 2024-05-09T03:15:24.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.611+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a901577f0 con 0x7f1a901613c0 2024-05-09T03:15:24.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.611+0000 7f1a77fff640 1 -- 172.21.15.46:0/4053274117 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5180 (secure 0 0 0) 0x7f1a880b80e0 con 0x7f1a901613c0 2024-05-09T03:15:24.613 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:24.614 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:01.510352+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:15:24.614 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 70 2024-05-09T03:15:24.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a7c0762d0 msgr2=0x7f1a7c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a7c0762d0 0x7f1a7c0787e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1a80000f60 tx=0x7f1a80000bc0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 msgr2=0x7f1a900ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 0x7f1a900ccdf0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009e50 tx=0x7f1a88004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 shutdown_connections 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a7c0762d0 0x7f1a7c0787e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a901613c0 0x7f1a900ccdf0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a900b7d00 0x7f1a90157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 --2- 172.21.15.46:0/4053274117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a900b5210 0x7f1a901f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.613+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 >> 172.21.15.46:0/4053274117 conn(0x7f1a900d29a0 msgr2=0x7f1a90164800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.614+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 shutdown_connections 2024-05-09T03:15:24.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:24.614+0000 7f1a95a0c640 1 -- 172.21.15.46:0/4053274117 wait complete. 2024-05-09T03:15:24.627 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-05-09T03:15:28.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-05-09T03:15:29.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:29.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.756+0000 7f508ebc0640 1 Processor -- start 2024-05-09T03:15:29.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.756+0000 7f508ebc0640 1 -- start start 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.757+0000 7f508ebc0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5088073500 0x7f5088073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f508ebc0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508815e6e0 con 0x7f50880af670 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f508ebc0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50880afa80 con 0x7f5088161cd0 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f508ebc0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5088073e50 con 0x7f5088073500 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f5087fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5088073500 0x7f5088073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f508c95c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f50880af670 0x7f50880d1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48786/0 (socket says 172.21.15.46:48786) 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f508c95c640 1 -- 172.21.15.46:0/806167125 learned_addr learned my addr 172.21.15.46:0/806167125 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f5087fff640 1 -- 172.21.15.46:0/806167125 >> v1:172.21.15.88:6789/0 conn(0x7f5088161cd0 legacy=0x7f50881620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f5087fff640 1 -- 172.21.15.46:0/806167125 >> v1:172.21.15.46:6789/0 conn(0x7f50880af670 legacy=0x7f50880d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:29.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.758+0000 7f5087fff640 1 -- 172.21.15.46:0/806167125 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50880d1880 con 0x7f5088073500 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.759+0000 7f5087fff640 1 --2- 172.21.15.46:0/806167125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5088073500 0x7f5088073910 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f507c0097a0 tx=0x7f507c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6673095dfb0d8081 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.759+0000 7f50877fe640 1 -- 172.21.15.46:0/806167125 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c037070 con 0x7f5088073500 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.759+0000 7f50877fe640 1 -- 172.21.15.46:0/806167125 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507c032440 con 0x7f5088073500 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.759+0000 7f50877fe640 1 -- 172.21.15.46:0/806167125 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c032810 con 0x7f5088073500 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 -- 172.21.15.46:0/806167125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5088073500 msgr2=0x7f5088073910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 --2- 172.21.15.46:0/806167125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5088073500 0x7f5088073910 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f507c0097a0 tx=0x7f507c02a180 comp rx=0 tx=0).stop 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 -- 172.21.15.46:0/806167125 shutdown_connections 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 --2- 172.21.15.46:0/806167125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5088073500 0x7f5088073910 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 -- 172.21.15.46:0/806167125 >> 172.21.15.46:0/806167125 conn(0x7f50880b3910 msgr2=0x7f50880b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:29.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 -- 172.21.15.46:0/806167125 shutdown_connections 2024-05-09T03:15:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.760+0000 7f508ebc0640 1 -- 172.21.15.46:0/806167125 wait complete. 2024-05-09T03:15:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.762+0000 7f508ebc0640 1 Processor -- start 2024-05-09T03:15:29.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.762+0000 7f508ebc0640 1 -- start start 2024-05-09T03:15:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508ebc0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5088073500 0x7f508809ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508ebc0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 0x7f50881e8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508ebc0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5088161cd0 0x7f50881486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508ebc0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f508815ef20 con 0x7f50880af670 2024-05-09T03:15:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508ebc0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f508815ec40 con 0x7f5088073500 2024-05-09T03:15:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508ebc0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f508815f200 con 0x7f5088161cd0 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f5087fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 0x7f50881e8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f5087fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 0x7f50881e8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57030/0 (socket says 172.21.15.46:57030) 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f5087fff640 1 -- 172.21.15.46:0/3591914236 learned_addr learned my addr 172.21.15.46:0/3591914236 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508c95c640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5088073500 0x7f508809ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.763+0000 7f508d15d640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5088161cd0 0x7f50881486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f5087fff640 1 -- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5088161cd0 msgr2=0x7f50881486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f5087fff640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5088161cd0 0x7f50881486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f5087fff640 1 -- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5088073500 msgr2=0x7f508809ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f5087fff640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5088073500 0x7f508809ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f5087fff640 1 -- 172.21.15.46:0/3591914236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508809bd90 con 0x7f50880af670 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f5087fff640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 0x7f50881e8590 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f507c0098c0 tx=0x7f507c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c041070 con 0x7f50880af670 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507c03c930 con 0x7f50880af670 2024-05-09T03:15:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508809c070 con 0x7f50880af670 2024-05-09T03:15:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c037040 con 0x7f50880af670 2024-05-09T03:15:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.764+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508809c630 con 0x7f50880af670 2024-05-09T03:15:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.765+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508809ca70 con 0x7f50880af670 2024-05-09T03:15:29.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.766+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f507c044070 con 0x7f50880af670 2024-05-09T03:15:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.767+0000 7f50857fa640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50740762d0 0x7f50740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.767+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f507c0b7de0 con 0x7f50880af670 2024-05-09T03:15:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.767+0000 7f508c95c640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50740762d0 0x7f50740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.768+0000 7f508c95c640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50740762d0 0x7f50740787e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5078000f60 tx=0x7f5078009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:29.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:29.772+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f50880afa80 con 0x7f50880af670 2024-05-09T03:15:30.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.007+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f508806a270 con 0x7f50880af670 2024-05-09T03:15:30.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.007+0000 7f50857fa640 1 -- 172.21.15.46:0/3591914236 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5228 (secure 0 0 0) 0x7f507c081140 con 0x7f50880af670 2024-05-09T03:15:30.009 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:30.009 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5977,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:24.958436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":65,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/131652171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":131652171},{"type":"v1","addr":"172.21.15.46:6834","nonce":131652171}]},"join_fscid":-1,"laggy_since":"2024-05-09T03:15:24.958436+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6471,"qdb_cluster":[6471]},"id":5}]} 2024-05-09T03:15:30.010 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 71 2024-05-09T03:15:30.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50740762d0 msgr2=0x7f50740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50740762d0 0x7f50740787e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5078000f60 tx=0x7f5078009040 comp rx=0 tx=0).stop 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 msgr2=0x7f50881e8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 0x7f50881e8590 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f507c0098c0 tx=0x7f507c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 shutdown_connections 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5088161cd0 0x7f50881486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50740762d0 0x7f50740787e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50880af670 0x7f50881e8590 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 --2- 172.21.15.46:0/3591914236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5088073500 0x7f508809ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 >> 172.21.15.46:0/3591914236 conn(0x7f50880b3910 msgr2=0x7f50880b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 shutdown_connections 2024-05-09T03:15:30.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.009+0000 7f508ebc0640 1 -- 172.21.15.46:0/3591914236 wait complete. 2024-05-09T03:15:30.023 DEBUG:tasks.ceph_test_case:wait_until_true: success in 20s and 0 retries 2024-05-09T03:15:30.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-05-09T03:15:30.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.239+0000 7f13fd3d5640 1 Processor -- start 2024-05-09T03:15:30.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.239+0000 7f13fd3d5640 1 -- start start 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13fd3d5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13f80b7b10 0x7f13f80b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13fd3d5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13f80b92a0 0x7f13f80b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13fd3d5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f80b8460 con 0x7f13f80b50b0 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13fd3d5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f13f806e760 con 0x7f13f80b7b10 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13fd3d5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f13f80d5740 con 0x7f13f80b92a0 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13f80b7b10 0x7f13f80b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13f80b7b10 0x7f13f80b7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57042/0 (socket says 172.21.15.46:57042) 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 -- 172.21.15.46:0/3654529207 learned_addr learned my addr 172.21.15.46:0/3654529207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f6ffd640 1 --1- 172.21.15.46:0/3654529207 >> v1:172.21.15.46:6789/0 conn(0x7f13f80b50b0 0x7f13f80d0cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48802/0 (socket says 172.21.15.46:48802) 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f77fe640 1 --2- 172.21.15.46:0/3654529207 >> v2:172.21.15.88:3300/0 conn(0x7f13f80b92a0 0x7f13f80b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 -- 172.21.15.46:0/3654529207 >> v2:172.21.15.88:3300/0 conn(0x7f13f80b92a0 msgr2=0x7f13f80b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 --2- 172.21.15.46:0/3654529207 >> v2:172.21.15.88:3300/0 conn(0x7f13f80b92a0 0x7f13f80b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 -- 172.21.15.46:0/3654529207 >> v1:172.21.15.46:6789/0 conn(0x7f13f80b50b0 legacy=0x7f13f80d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f5ffb640 1 -- 172.21.15.46:0/3654529207 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397989305 0 0) 0x7f13e8001490 con 0x7f13f80b50b0 2024-05-09T03:15:30.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.241+0000 7f13f67fc640 1 -- 172.21.15.46:0/3654529207 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f806de30 con 0x7f13f80b7b10 2024-05-09T03:15:30.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.242+0000 7f13f67fc640 1 --2- 172.21.15.46:0/3654529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b7b10 0x7f13f80b7f20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f13f80d39d0 tx=0x7f13e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db12ece102b30d3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:30.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.242+0000 7f13f5ffb640 1 -- 172.21.15.46:0/3654529207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0037070 con 0x7f13f80b7b10 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.242+0000 7f13f5ffb640 1 -- 172.21.15.46:0/3654529207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e0032440 con 0x7f13f80b7b10 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.242+0000 7f13f5ffb640 1 -- 172.21.15.46:0/3654529207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0032810 con 0x7f13f80b7b10 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 -- 172.21.15.46:0/3654529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b7b10 msgr2=0x7f13f80b7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/3654529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b7b10 0x7f13f80b7f20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f13f80d39d0 tx=0x7f13e002a270 comp rx=0 tx=0).stop 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 -- 172.21.15.46:0/3654529207 shutdown_connections 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/3654529207 >> v2:172.21.15.88:3300/0 conn(0x7f13f80b92a0 0x7f13f80b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/3654529207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b7b10 0x7f13f80b7f20 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 -- 172.21.15.46:0/3654529207 >> 172.21.15.46:0/3654529207 conn(0x7f13f80d2b80 msgr2=0x7f13f80d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 -- 172.21.15.46:0/3654529207 shutdown_connections 2024-05-09T03:15:30.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.243+0000 7f13fd3d5640 1 -- 172.21.15.46:0/3654529207 wait complete. 2024-05-09T03:15:30.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.245+0000 7f13fd3d5640 1 Processor -- start 2024-05-09T03:15:30.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.245+0000 7f13fd3d5640 1 -- start start 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13fd3d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13f80b50b0 0x7f13f81f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13fd3d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f80b7b10 0x7f13f81f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13fd3d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 0x7f13f8160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13fd3d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13f80b54c0 con 0x7f13f80b92a0 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13fd3d5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f13f806e3f0 con 0x7f13f80b7b10 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13fd3d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f13f806ebf0 con 0x7f13f80b50b0 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 0x7f13f8160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 0x7f13f8160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57054/0 (socket says 172.21.15.46:57054) 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.246+0000 7f13f77fe640 1 -- 172.21.15.46:0/1796819164 learned_addr learned my addr 172.21.15.46:0/1796819164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f67fc640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f80b7b10 0x7f13f81f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f6ffd640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13f80b50b0 0x7f13f81f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f77fe640 1 -- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13f80b50b0 msgr2=0x7f13f81f5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f77fe640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13f80b50b0 0x7f13f81f5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f77fe640 1 -- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f80b7b10 msgr2=0x7f13f81f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f77fe640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f80b7b10 0x7f13f81f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f77fe640 1 -- 172.21.15.46:0/1796819164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f815ba60 con 0x7f13f80b92a0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f6ffd640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13f80b50b0 0x7f13f81f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13f77fe640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 0x7f13f8160a20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f13f000d4e0 tx=0x7f13f000d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f00125f0 con 0x7f13f80b92a0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f0019070 con 0x7f13f80b92a0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f81e9fa0 con 0x7f13f80b92a0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f00129f0 con 0x7f13f80b92a0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.247+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f81ebb00 con 0x7f13f80b92a0 2024-05-09T03:15:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.248+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f815b7d0 con 0x7f13f80b92a0 2024-05-09T03:15:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.249+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f13f0012b90 con 0x7f13f80b92a0 2024-05-09T03:15:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.250+0000 7f13ef7fe640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13d80762d0 0x7f13d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.250+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f13f0097970 con 0x7f13f80b92a0 2024-05-09T03:15:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.250+0000 7f13f6ffd640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13d80762d0 0x7f13d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.250+0000 7f13f6ffd640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13d80762d0 0x7f13d80787e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f13e80012e0 tx=0x7f13e8005b90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:30.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.255+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f13f0061280 con 0x7f13f80b92a0 2024-05-09T03:15:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.444+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f13f81ebd60 con 0x7f13f80b92a0 2024-05-09T03:15:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.448+0000 7f13ef7fe640 1 -- 172.21.15.46:0/1796819164 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f13f0060b80 con 0x7f13f80b92a0 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13d80762d0 msgr2=0x7f13d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13d80762d0 0x7f13d80787e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f13e80012e0 tx=0x7f13e8005b90 comp rx=0 tx=0).stop 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 msgr2=0x7f13f8160a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 0x7f13f8160a20 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f13f000d4e0 tx=0x7f13f000d9a0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 shutdown_connections 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13d80762d0 0x7f13d80787e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f80b92a0 0x7f13f8160a20 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13f80b7b10 0x7f13f81f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 --2- 172.21.15.46:0/1796819164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13f80b50b0 0x7f13f81f5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 >> 172.21.15.46:0/1796819164 conn(0x7f13f80d2b80 msgr2=0x7f13f815ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 shutdown_connections 2024-05-09T03:15:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.450+0000 7f13fd3d5640 1 -- 172.21.15.46:0/1796819164 wait complete. 2024-05-09T03:15:30.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 false 2024-05-09T03:15:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.578+0000 7f39afdb2640 1 Processor -- start 2024-05-09T03:15:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.578+0000 7f39afdb2640 1 -- start start 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.579+0000 7f39afdb2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39a80cfd50 0x7f39a8169770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.579+0000 7f39afdb2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a80d1d50 con 0x7f39a80b7460 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.579+0000 7f39afdb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39a80b9340 con 0x7f39a808cd60 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.579+0000 7f39afdb2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f39a80d1760 con 0x7f39a80cfd50 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.579+0000 7f39ad34d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f39a80b7460 0x7f39a809cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48806/0 (socket says 172.21.15.46:48806) 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.579+0000 7f39ad34d640 1 -- 172.21.15.46:0/562698930 learned_addr learned my addr 172.21.15.46:0/562698930 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39ae34f640 1 --2- 172.21.15.46:0/562698930 >> v2:172.21.15.88:3300/0 conn(0x7f39a80cfd50 0x7f39a8169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39acb4c640 1 -- 172.21.15.46:0/562698930 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979436828 0 0) 0x7f39a80d1d50 con 0x7f39a80b7460 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39acb4c640 1 -- 172.21.15.46:0/562698930 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3990003520 con 0x7f39a80b7460 2024-05-09T03:15:30.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39acb4c640 1 -- 172.21.15.46:0/562698930 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1873576094 0 0) 0x7f3990003520 con 0x7f39a80b7460 2024-05-09T03:15:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39ae34f640 1 -- 172.21.15.46:0/562698930 >> v1:172.21.15.88:6789/0 conn(0x7f39a808cd60 legacy=0x7f39a80d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39ae34f640 1 -- 172.21.15.46:0/562698930 >> v1:172.21.15.46:6789/0 conn(0x7f39a80b7460 legacy=0x7f39a809cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.580+0000 7f39ae34f640 1 -- 172.21.15.46:0/562698930 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a80d2af0 con 0x7f39a80cfd50 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.581+0000 7f39ae34f640 1 --2- 172.21.15.46:0/562698930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a80cfd50 0x7f39a8169770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009d90 tx=0x7f39a002a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79d643c7dca0b508 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.581+0000 7f39acb4c640 1 -- 172.21.15.46:0/562698930 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c0016f0 con 0x7f39a80cfd50 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.581+0000 7f39acb4c640 1 -- 172.21.15.46:0/562698930 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a0007540 con 0x7f39a80cfd50 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.581+0000 7f39acb4c640 1 -- 172.21.15.46:0/562698930 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a00078f0 con 0x7f39a80cfd50 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 -- 172.21.15.46:0/562698930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a80cfd50 msgr2=0x7f39a8169770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 --2- 172.21.15.46:0/562698930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a80cfd50 0x7f39a8169770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009d90 tx=0x7f39a002a560 comp rx=0 tx=0).stop 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 -- 172.21.15.46:0/562698930 shutdown_connections 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 --2- 172.21.15.46:0/562698930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a80cfd50 0x7f39a8169770 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 -- 172.21.15.46:0/562698930 >> 172.21.15.46:0/562698930 conn(0x7f39a80d0610 msgr2=0x7f39a80d0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 -- 172.21.15.46:0/562698930 shutdown_connections 2024-05-09T03:15:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.582+0000 7f39afdb2640 1 -- 172.21.15.46:0/562698930 wait complete. 2024-05-09T03:15:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.584+0000 7f39afdb2640 1 Processor -- start 2024-05-09T03:15:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.584+0000 7f39afdb2640 1 -- start start 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39afdb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a808cd60 0x7f39a81f7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39afdb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39a80b7460 0x7f39a81f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39afdb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 0x7f39a81f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39afdb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f39a80b9720 con 0x7f39a80cfd50 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39afdb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f39a80d2030 con 0x7f39a808cd60 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39afdb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f39a809d3b0 con 0x7f39a80b7460 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 0x7f39a81f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 0x7f39a81f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57064/0 (socket says 172.21.15.46:57064) 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 -- 172.21.15.46:0/1124044145 learned_addr learned my addr 172.21.15.46:0/1124044145 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39adb4e640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a808cd60 0x7f39a81f7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ad34d640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39a80b7460 0x7f39a81f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 -- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39a80b7460 msgr2=0x7f39a81f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39a80b7460 0x7f39a81f7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 -- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a808cd60 msgr2=0x7f39a81f7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a808cd60 0x7f39a81f7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.585+0000 7f39ae34f640 1 -- 172.21.15.46:0/1124044145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a8156240 con 0x7f39a80cfd50 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.586+0000 7f39ae34f640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 0x7f39a81f47a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f39a002ab50 tx=0x7f39a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.586+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a0040070 con 0x7f39a80cfd50 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.586+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a0031c40 con 0x7f39a80cfd50 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.586+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a8156520 con 0x7f39a80cfd50 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.586+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a0009450 con 0x7f39a80cfd50 2024-05-09T03:15:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.586+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a8156ae0 con 0x7f39a80cfd50 2024-05-09T03:15:30.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.587+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a8155f90 con 0x7f39a80cfd50 2024-05-09T03:15:30.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.588+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f39a0039040 con 0x7f39a80cfd50 2024-05-09T03:15:30.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.588+0000 7f39a67fc640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3994076300 0x7f3994078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:30.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.588+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f39a00b8bc0 con 0x7f39a80cfd50 2024-05-09T03:15:30.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.589+0000 7f39adb4e640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3994076300 0x7f3994078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:30.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.589+0000 7f39adb4e640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3994076300 0x7f3994078810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3998000fb0 tx=0x7f3998009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:30.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.593+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f39a0081d60 con 0x7f39a80cfd50 2024-05-09T03:15:30.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:30.860+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"} v 0) -- 0x7f39a806a270 con 0x7f39a80cfd50 2024-05-09T03:15:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.452+0000 7f39a67fc640 1 -- 172.21.15.46:0/1124044145 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"}]=0 unfreezing mds.6471 v72) ==== 115+0+0 (secure 0 0 0) 0x7f39a0086060 con 0x7f39a80cfd50 2024-05-09T03:15:31.454 INFO:teuthology.orchestra.run.smithi046.stderr:unfreezing mds.6471 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3994076300 msgr2=0x7f3994078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3994076300 0x7f3994078810 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3998000fb0 tx=0x7f3998009040 comp rx=0 tx=0).stop 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 msgr2=0x7f39a81f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 0x7f39a81f47a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f39a002ab50 tx=0x7f39a0004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 shutdown_connections 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3994076300 0x7f3994078810 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39a80cfd50 0x7f39a81f47a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f39a80b7460 0x7f39a81f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 --2- 172.21.15.46:0/1124044145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a808cd60 0x7f39a81f7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 >> 172.21.15.46:0/1124044145 conn(0x7f39a80d0610 msgr2=0x7f39a809bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 shutdown_connections 2024-05-09T03:15:31.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.455+0000 7f39afdb2640 1 -- 172.21.15.46:0/1124044145 wait complete. 2024-05-09T03:15:31.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -n kernel.core_pattern 2024-05-09T03:15:31.502 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T03:15:31.504 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:15:31.504 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-05-09T03:15:31.575 INFO:teuthology.orchestra.run.smithi046.stdout:1715224508.57997.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-05-09T03:15:31.577 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1715224508.57997.core'] 2024-05-09T03:15:31.577 INFO:tasks.cephfs.cephfs_test_case:Found core file 1715224508.57997.core, deleting it 2024-05-09T03:15:31.578 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1715224508.57997.core 2024-05-09T03:15:31.655 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:15:31.655 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:15:31.655 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:15:31.655 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:15:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:15:31.655 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-05-09T03:15:31.656 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:15:31.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:15:31.699 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:15:31.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:31.790 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:15:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.815+0000 7fe5f563b640 1 Processor -- start 2024-05-09T03:15:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.815+0000 7fe5f563b640 1 -- start start 2024-05-09T03:15:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5f563b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5f563b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5f00b7ef0 0x7fe5f00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5f563b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f00d49e0 con 0x7fe5f00b4af0 2024-05-09T03:15:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5f563b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f00d5bb0 con 0x7fe5f00b7ef0 2024-05-09T03:15:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5f563b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f00d4cc0 con 0x7fe5f00d57a0 2024-05-09T03:15:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5f00b7ef0 0x7fe5f00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5eeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5f00b7ef0 0x7fe5f00b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:59808/0 (socket says 172.21.15.46:59808) 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/29226346 learned_addr learned my addr 172.21.15.46:0/29226346 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ee7fc640 1 --1- 172.21.15.46:0/29226346 >> v1:172.21.15.88:6789/0 conn(0x7fe5f00b4af0 0x7fe5f00b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:40438/0 (socket says 172.21.15.46:40438) 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 msgr2=0x7fe5f0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 --2- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/29226346 >> v1:172.21.15.88:6789/0 conn(0x7fe5f00b4af0 legacy=0x7fe5f00b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.817+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/29226346 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f00d4fa0 con 0x7fe5f00b7ef0 2024-05-09T03:15:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.818+0000 7fe5ef7fe640 1 --2- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b7ef0 0x7fe5f00b4f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e50 tx=0x7fe5e802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8e8a132fdbe7c9a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.818+0000 7fe5edffb640 1 -- 172.21.15.46:0/29226346 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8036070 con 0x7fe5f00b7ef0 2024-05-09T03:15:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.818+0000 7fe5edffb640 1 -- 172.21.15.46:0/29226346 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e8031600 con 0x7fe5f00b7ef0 2024-05-09T03:15:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.818+0000 7fe5edffb640 1 -- 172.21.15.46:0/29226346 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e80319d0 con 0x7fe5f00b7ef0 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 -- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b7ef0 msgr2=0x7fe5f00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 --2- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b7ef0 0x7fe5f00b4f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e50 tx=0x7fe5e802a710 comp rx=0 tx=0).stop 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 -- 172.21.15.46:0/29226346 shutdown_connections 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 --2- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b7ef0 0x7fe5f00b4f00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 --2- 172.21.15.46:0/29226346 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 -- 172.21.15.46:0/29226346 >> 172.21.15.46:0/29226346 conn(0x7fe5f00743e0 msgr2=0x7fe5f00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 -- 172.21.15.46:0/29226346 shutdown_connections 2024-05-09T03:15:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.819+0000 7fe5f563b640 1 -- 172.21.15.46:0/29226346 wait complete. 2024-05-09T03:15:31.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.821+0000 7fe5f563b640 1 Processor -- start 2024-05-09T03:15:31.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.821+0000 7fe5f563b640 1 -- start start 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5f563b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b4af0 0x7fe5f0156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5f563b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f01e9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5f563b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 0x7fe5f01e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5f563b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f00d1920 con 0x7fe5f00d0320 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5f563b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f00d15b0 con 0x7fe5f00b4af0 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5f563b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f015b0b0 con 0x7fe5f00d57a0 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 0x7fe5f01e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 0x7fe5f01e96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57092/0 (socket says 172.21.15.46:57092) 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/905097313 learned_addr learned my addr 172.21.15.46:0/905097313 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5eeffd640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b4af0 0x7fe5f0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ee7fc640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f01e9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 msgr2=0x7fe5f01e9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f01e9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b4af0 msgr2=0x7fe5f0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b4af0 0x7fe5f0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 -- 172.21.15.46:0/905097313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f015d680 con 0x7fe5f00d0320 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.823+0000 7fe5ef7fe640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 0x7fe5f01e96a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8034040 tx=0x7fe5e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.824+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8036050 con 0x7fe5f00d0320 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.824+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5e8042070 con 0x7fe5f00d0320 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.824+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8009510 con 0x7fe5f00d0320 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.824+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f015b300 con 0x7fe5f00d0320 2024-05-09T03:15:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.824+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f01586a0 con 0x7fe5f00d0320 2024-05-09T03:15:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.825+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f0160970 con 0x7fe5f00d0320 2024-05-09T03:15:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.825+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe5e8007c70 con 0x7fe5f00d0320 2024-05-09T03:15:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.826+0000 7fe5d77fe640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5dc076300 0x7fe5dc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.826+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7fe5e80b8480 con 0x7fe5f00d0320 2024-05-09T03:15:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.826+0000 7fe5eeffd640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5dc076300 0x7fe5dc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:31.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.827+0000 7fe5eeffd640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5dc076300 0x7fe5dc078810 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe5f006d3b0 tx=0x7fe5e0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:31.831+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe5f0160970 con 0x7fe5f00d0320 2024-05-09T03:15:32.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.059+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5f006a270 con 0x7fe5f00d0320 2024-05-09T03:15:32.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.060+0000 7fe5d77fe640 1 -- 172.21.15.46:0/905097313 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+4352 (secure 0 0 0) 0x7fe5e80815d0 con 0x7fe5f00d0320 2024-05-09T03:15:32.062 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:32.062 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":73,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:31.456592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-05-09T03:15:32.062 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 73 2024-05-09T03:15:32.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5dc076300 msgr2=0x7fe5dc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5dc076300 0x7fe5dc078810 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe5f006d3b0 tx=0x7fe5e0009190 comp rx=0 tx=0).stop 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 msgr2=0x7fe5f01e96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 0x7fe5f01e96a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8034040 tx=0x7fe5e8004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 shutdown_connections 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5dc076300 0x7fe5dc078810 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f00d0320 0x7fe5f01e96a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe5f00d57a0 0x7fe5f01e9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 --2- 172.21.15.46:0/905097313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5f00b4af0 0x7fe5f0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 >> 172.21.15.46:0/905097313 conn(0x7fe5f00743e0 msgr2=0x7fe5f00b96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 shutdown_connections 2024-05-09T03:15:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:32.062+0000 7fe5f563b640 1 -- 172.21.15.46:0/905097313 wait complete. 2024-05-09T03:15:32.477 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:14:52.631122+0000', 'modified': '2024-05-09T03:15:31.456592+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5977}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5977': {'gid': 5977, 'name': 'd', 'rank': 0, 'incarnation': 73, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/3673458792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3673458792}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 3673458792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:15:32.477 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5977:up:replay 2024-05-09T03:15:33.479 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:33.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.604+0000 7f5b0c2ee640 1 Processor -- start 2024-05-09T03:15:33.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.604+0000 7f5b0c2ee640 1 -- start start 2024-05-09T03:15:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0c2ee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b040b5210 0x7f5b041613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:33.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0c2ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b040b7ed0 0x7f5b040d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0c2ee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b040d5280 con 0x7f5b040d4800 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0c2ee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5b040b5620 con 0x7f5b040b5210 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0c2ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5b041609a0 con 0x7f5b040b7ed0 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0a08a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5b040b5210 0x7f5b041613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b09889640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5b040d4800 0x7f5b040b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48808/0 (socket says 172.21.15.46:48808) 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b09889640 1 -- 172.21.15.46:0/2439694844 learned_addr learned my addr 172.21.15.46:0/2439694844 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.606+0000 7f5b0a88b640 1 --2- 172.21.15.46:0/2439694844 >> v2:172.21.15.88:3300/0 conn(0x7f5b040b7ed0 0x7f5b040d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.607+0000 7f5b0a08a640 1 -- 172.21.15.46:0/2439694844 >> v2:172.21.15.88:3300/0 conn(0x7f5b040b7ed0 msgr2=0x7f5b040d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.607+0000 7f5b0a08a640 1 --2- 172.21.15.46:0/2439694844 >> v2:172.21.15.88:3300/0 conn(0x7f5b040b7ed0 0x7f5b040d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.607+0000 7f5b0a08a640 1 -- 172.21.15.46:0/2439694844 >> v1:172.21.15.46:6789/0 conn(0x7f5b040d4800 legacy=0x7f5b040b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.607+0000 7f5b0a08a640 1 -- 172.21.15.46:0/2439694844 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b040d5560 con 0x7f5b040b5210 2024-05-09T03:15:33.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.607+0000 7f5b09088640 1 -- 172.21.15.46:0/2439694844 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3588221078 0 0) 0x7f5b040d5280 con 0x7f5b040d4800 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.607+0000 7f5b0a08a640 1 --2- 172.21.15.46:0/2439694844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b041613c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406f030 tx=0x7f5b0002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9037627b876a09ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b09088640 1 -- 172.21.15.46:0/2439694844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00037070 con 0x7f5b040b5210 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b09088640 1 -- 172.21.15.46:0/2439694844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00032440 con 0x7f5b040b5210 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b09088640 1 -- 172.21.15.46:0/2439694844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0002ae60 con 0x7f5b040b5210 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/2439694844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 msgr2=0x7f5b041613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/2439694844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b041613c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406f030 tx=0x7f5b0002a110 comp rx=0 tx=0).stop 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/2439694844 shutdown_connections 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/2439694844 >> v2:172.21.15.88:3300/0 conn(0x7f5b040b7ed0 0x7f5b040d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/2439694844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b041613c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/2439694844 >> 172.21.15.46:0/2439694844 conn(0x7f5b040d29a0 msgr2=0x7f5b040d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/2439694844 shutdown_connections 2024-05-09T03:15:33.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.608+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/2439694844 wait complete. 2024-05-09T03:15:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.610+0000 7f5b0c2ee640 1 Processor -- start 2024-05-09T03:15:33.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.610+0000 7f5b0c2ee640 1 -- start start 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0c2ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b040cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0c2ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b040d4800 0x7f5b04159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0c2ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b041fedf0 0x7f5b040ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0c2ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b040d0ae0 con 0x7f5b040b5210 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0c2ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5b04160ea0 con 0x7f5b041fedf0 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0c2ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5b04163e60 con 0x7f5b040d4800 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b040cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:33.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b040cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57106/0 (socket says 172.21.15.46:57106) 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 -- 172.21.15.46:0/267467788 learned_addr learned my addr 172.21.15.46:0/267467788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a88b640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b041fedf0 0x7f5b040ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 -- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b040d4800 msgr2=0x7f5b04159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b09889640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b040d4800 0x7f5b04159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b040d4800 0x7f5b04159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 -- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b041fedf0 msgr2=0x7f5b040ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b041fedf0 0x7f5b040ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 -- 172.21.15.46:0/267467788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b040cd040 con 0x7f5b040b5210 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a88b640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b041fedf0 0x7f5b040ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.612+0000 7f5b0a08a640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b040cf730 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5b00000c00 tx=0x7f5b000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.613+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00037040 con 0x7f5b040b5210 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.613+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00042070 con 0x7f5b040b5210 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.613+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b040cd320 con 0x7f5b040b5210 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.613+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00030960 con 0x7f5b040b5210 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.613+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b040cd8e0 con 0x7f5b040b5210 2024-05-09T03:15:33.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.613+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b0406a270 con 0x7f5b040b5210 2024-05-09T03:15:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.614+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5b00030b00 con 0x7f5b040b5210 2024-05-09T03:15:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.615+0000 7f5afaffd640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae80762d0 0x7f5ae80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.615+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f5b000b8b80 con 0x7f5b040b5210 2024-05-09T03:15:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.615+0000 7f5b09889640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae80762d0 0x7f5ae80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:33.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.616+0000 7f5b09889640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae80762d0 0x7f5ae80787e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5b040d31c0 tx=0x7f5af4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:33.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.619+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5b00082530 con 0x7f5b040b5210 2024-05-09T03:15:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.848+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b040d40d0 con 0x7f5b040b5210 2024-05-09T03:15:33.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.849+0000 7f5afaffd640 1 -- 172.21.15.46:0/267467788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5178 (secure 0 0 0) 0x7f5b00081e30 con 0x7f5b040b5210 2024-05-09T03:15:33.851 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:33.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":75,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:32.463321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-05-09T03:15:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 75 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae80762d0 msgr2=0x7f5ae80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae80762d0 0x7f5ae80787e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5b040d31c0 tx=0x7f5af4009040 comp rx=0 tx=0).stop 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 msgr2=0x7f5b040cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b040cf730 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5b00000c00 tx=0x7f5b000046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 shutdown_connections 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b041fedf0 0x7f5b040ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5b040d4800 0x7f5b04159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae80762d0 0x7f5ae80787e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 --2- 172.21.15.46:0/267467788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b040b5210 0x7f5b040cf730 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 >> 172.21.15.46:0/267467788 conn(0x7f5b040d29a0 msgr2=0x7f5b040baf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 shutdown_connections 2024-05-09T03:15:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:33.851+0000 7f5b0c2ee640 1 -- 172.21.15.46:0/267467788 wait complete. 2024-05-09T03:15:34.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 75, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:14:52.631122+0000', 'modified': '2024-05-09T03:15:32.463321+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5977}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5977': {'gid': 5977, 'name': 'd', 'rank': 0, 'incarnation': 73, 'state': 'up:rejoin', 'state_seq': 13, 'addr': '172.21.15.88:6836/3673458792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3673458792}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 3673458792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:15:34.267 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5977:up:rejoin 2024-05-09T03:15:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:15:35.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-05-09T03:15:35.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.396+0000 7f1472cbd640 1 Processor -- start 2024-05-09T03:15:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.396+0000 7f1472cbd640 1 -- start start 2024-05-09T03:15:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f1472cbd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f146c0d1c80 0x7f146c0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f1472cbd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c0cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f1472cbd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146c0d4900 con 0x7f146c08cd60 2024-05-09T03:15:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f1472cbd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f146c0b5340 con 0x7f146c0d1c80 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f1472cbd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f146c0b4fd0 con 0x7f146c0d2090 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f147125a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c0cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f147125a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c0cf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43756/0 (socket says 172.21.15.46:43756) 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f147125a640 1 -- 172.21.15.46:0/745710108 learned_addr learned my addr 172.21.15.46:0/745710108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.397+0000 7f146bfff640 1 --2- 172.21.15.46:0/745710108 >> v2:172.21.15.88:3300/0 conn(0x7f146c0d1c80 0x7f146c0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.398+0000 7f147125a640 1 -- 172.21.15.46:0/745710108 >> v2:172.21.15.88:3300/0 conn(0x7f146c0d1c80 msgr2=0x7f146c0cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.398+0000 7f147125a640 1 --2- 172.21.15.46:0/745710108 >> v2:172.21.15.88:3300/0 conn(0x7f146c0d1c80 0x7f146c0cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.398+0000 7f147125a640 1 -- 172.21.15.46:0/745710108 >> v1:172.21.15.88:6789/0 conn(0x7f146c08cd60 legacy=0x7f146c0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.398+0000 7f147125a640 1 -- 172.21.15.46:0/745710108 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c0b6a50 con 0x7f146c0d2090 2024-05-09T03:15:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.398+0000 7f146bfff640 1 --2- 172.21.15.46:0/745710108 >> v2:172.21.15.88:3300/0 conn(0x7f146c0d1c80 0x7f146c0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.398+0000 7f147125a640 1 --2- 172.21.15.46:0/745710108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c0cf7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1464009e80 tx=0x7f146402a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d895767d1a3f7e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f146b7fe640 1 -- 172.21.15.46:0/745710108 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464036070 con 0x7f146c0d2090 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f146b7fe640 1 -- 172.21.15.46:0/745710108 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1464031600 con 0x7f146c0d2090 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f146b7fe640 1 -- 172.21.15.46:0/745710108 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14640319d0 con 0x7f146c0d2090 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 -- 172.21.15.46:0/745710108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 msgr2=0x7f146c0cf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 --2- 172.21.15.46:0/745710108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c0cf7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1464009e80 tx=0x7f146402a6c0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 -- 172.21.15.46:0/745710108 shutdown_connections 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 --2- 172.21.15.46:0/745710108 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c0cf7f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 --2- 172.21.15.46:0/745710108 >> v2:172.21.15.88:3300/0 conn(0x7f146c0d1c80 0x7f146c0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 -- 172.21.15.46:0/745710108 >> 172.21.15.46:0/745710108 conn(0x7f146c0d3c00 msgr2=0x7f146c0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 -- 172.21.15.46:0/745710108 shutdown_connections 2024-05-09T03:15:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.399+0000 7f1472cbd640 1 -- 172.21.15.46:0/745710108 wait complete. 2024-05-09T03:15:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.401+0000 7f1472cbd640 1 Processor -- start 2024-05-09T03:15:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.401+0000 7f1472cbd640 1 -- start start 2024-05-09T03:15:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1472cbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 0x7f146c14d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1472cbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f146c0d1c80 0x7f146c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1472cbd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c15f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1472cbd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f146c0d0440 con 0x7f146c08cd60 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1472cbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f146c0d02c0 con 0x7f146c0d1c80 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1472cbd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f146c0d05c0 con 0x7f146c0d2090 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 0x7f146c14d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 0x7f146c14d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57120/0 (socket says 172.21.15.46:57120) 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 -- 172.21.15.46:0/231443948 learned_addr learned my addr 172.21.15.46:0/231443948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f146bfff640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f146c0d1c80 0x7f146c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f147125a640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c15f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 -- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 msgr2=0x7f146c15f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c15f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 -- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f146c0d1c80 msgr2=0x7f146c150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f146c0d1c80 0x7f146c150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.403+0000 7f1470a59640 1 -- 172.21.15.46:0/231443948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146c14e760 con 0x7f146c08cd60 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.404+0000 7f1470a59640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 0x7f146c14d930 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c370 tx=0x7f145c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.404+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c017070 con 0x7f146c08cd60 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.404+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f145c0052d0 con 0x7f146c08cd60 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.404+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145c003810 con 0x7f146c08cd60 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.404+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146c14ea40 con 0x7f146c08cd60 2024-05-09T03:15:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.404+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f146c14f000 con 0x7f146c08cd60 2024-05-09T03:15:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.405+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f146c06a270 con 0x7f146c08cd60 2024-05-09T03:15:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.406+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f145c01a070 con 0x7f146c08cd60 2024-05-09T03:15:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.406+0000 7f14697fa640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14580762d0 0x7f14580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.406+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f145c096960 con 0x7f146c08cd60 2024-05-09T03:15:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.406+0000 7f146bfff640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14580762d0 0x7f14580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.407+0000 7f146bfff640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14580762d0 0x7f14580787e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1460009560 tx=0x7f1460009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:35.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.411+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f145c05fb60 con 0x7f146c08cd60 2024-05-09T03:15:35.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.640+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f146c0cf6a0 con 0x7f146c08cd60 2024-05-09T03:15:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.640+0000 7f14697fa640 1 -- 172.21.15.46:0/231443948 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5185 (secure 0 0 0) 0x7f145c0681a0 con 0x7f146c08cd60 2024-05-09T03:15:35.642 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:35.642 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:34.464748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":14,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5977,"qdb_cluster":[5977]},"id":5}]} 2024-05-09T03:15:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2024-05-09T03:15:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.642+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14580762d0 msgr2=0x7f14580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.642+0000 7f1472cbd640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14580762d0 0x7f14580787e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1460009560 tx=0x7f1460009190 comp rx=0 tx=0).stop 2024-05-09T03:15:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.642+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 msgr2=0x7f146c14d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.642+0000 7f1472cbd640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 0x7f146c14d930 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c370 tx=0x7f145c00c830 comp rx=0 tx=0).stop 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.642+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 shutdown_connections 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f146c0d2090 0x7f146c15f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f146c0d1c80 0x7f146c150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f14580762d0 0x7f14580787e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 --2- 172.21.15.46:0/231443948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f146c08cd60 0x7f146c14d930 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 >> 172.21.15.46:0/231443948 conn(0x7f146c0d3c00 msgr2=0x7f146c072c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 shutdown_connections 2024-05-09T03:15:35.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:35.643+0000 7f1472cbd640 1 -- 172.21.15.46:0/231443948 wait complete. 2024-05-09T03:15:36.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:14:52.631122+0000', 'modified': '2024-05-09T03:15:34.464748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5977}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5977': {'gid': 5977, 'name': 'd', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.88:6836/3673458792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3673458792}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 3673458792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5977, 'qdb_cluster': [5977]} 2024-05-09T03:15:36.059 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:15:36.059 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-09T03:15:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.179+0000 7f182a3c7640 1 Processor -- start 2024-05-09T03:15:36.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.179+0000 7f182a3c7640 1 -- start start 2024-05-09T03:15:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f182a3c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18240b7e70 0x7f18240d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f182a3c7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18240d3df0 0x7f18240b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f182a3c7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18240b5550 con 0x7f18240b5140 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f182a3c7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f18240d3480 con 0x7f18240d3df0 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f182a3c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f18240d44e0 con 0x7f18240b7e70 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1823fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f18240b5140 0x7f18240b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60140/0 (socket says 172.21.15.46:60140) 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1823fff640 1 -- 172.21.15.46:0/3454903764 learned_addr learned my addr 172.21.15.46:0/3454903764 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1828964640 1 --2- 172.21.15.46:0/3454903764 >> v2:172.21.15.88:3300/0 conn(0x7f18240d3df0 0x7f18240b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f18237fe640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18240b7e70 0x7f18240d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3128819644 0 0) 0x7f18240b5550 con 0x7f18240b5140 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f180c003520 con 0x7f18240b5140 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1480833060 0 0) 0x7f180c003520 con 0x7f18240b5140 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18240b7e70 msgr2=0x7f18240d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18240b7e70 0x7f18240d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 >> v2:172.21.15.88:3300/0 conn(0x7f18240d3df0 msgr2=0x7f18240b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 --2- 172.21.15.46:0/3454903764 >> v2:172.21.15.88:3300/0 conn(0x7f18240d3df0 0x7f18240b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18240d47c0 con 0x7f18240b5140 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.181+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f18140016f0 con 0x7f18240b5140 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 >> v1:172.21.15.46:6789/0 conn(0x7f18240b5140 legacy=0x7f18240b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180c003a00 0x7f180c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c004800 0x7f180c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f180c005150 0x7f180c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f18240d44e0 con 0x7f180c003a00 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f180c00bf00 con 0x7f180c004800 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f180c00c080 con 0x7f180c005150 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1823fff640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180c003a00 0x7f180c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1828964640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f180c005150 0x7f180c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f18237fe640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c004800 0x7f180c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1823fff640 1 -- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f180c005150 msgr2=0x7f180c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1823fff640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f180c005150 0x7f180c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1823fff640 1 -- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c004800 msgr2=0x7f180c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1823fff640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c004800 0x7f180c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f1823fff640 1 -- 172.21.15.46:0/3454903764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c003520 con 0x7f180c003a00 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.182+0000 7f18237fe640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c004800 0x7f180c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.183+0000 7f1823fff640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180c003a00 0x7f180c003e70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f181400c2c0 tx=0x7f180c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137eb97e7b8dafb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.183+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1814007d50 con 0x7f180c003a00 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.183+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1814005ad0 con 0x7f180c003a00 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.183+0000 7f1822ffd640 1 -- 172.21.15.46:0/3454903764 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1814007070 con 0x7f180c003a00 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.183+0000 7f182a3c7640 1 -- 172.21.15.46:0/3454903764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180c003a00 msgr2=0x7f180c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.183+0000 7f182a3c7640 1 --2- 172.21.15.46:0/3454903764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f180c003a00 0x7f180c003e70 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f181400c2c0 tx=0x7f180c005830 comp rx=0 tx=0).stop 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.184+0000 7f1828964640 1 -- 172.21.15.46:0/3454903764 reap_dead start 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.184+0000 7f182a3c7640 1 -- 172.21.15.46:0/3454903764 shutdown_connections 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.184+0000 7f182a3c7640 1 -- 172.21.15.46:0/3454903764 >> 172.21.15.46:0/3454903764 conn(0x7f182408d3f0 msgr2=0x7f182408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.184+0000 7f182a3c7640 1 -- 172.21.15.46:0/3454903764 shutdown_connections 2024-05-09T03:15:36.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.184+0000 7f182a3c7640 1 -- 172.21.15.46:0/3454903764 wait complete. 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.185+0000 7f182a3c7640 1 Processor -- start 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.186+0000 7f182a3c7640 1 -- start start 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f182a3c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c003a00 0x7f18241552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f182a3c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 0x7f18241f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f182a3c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1824162380 0x7f1824162790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f182a3c7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f18240d4200 con 0x7f1824158200 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f182a3c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18240d4d80 con 0x7f180c003a00 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f182a3c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f18240d5580 con 0x7f1824162380 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 0x7f18241f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 0x7f18241f54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57072/0 (socket says 172.21.15.46:57072) 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 -- 172.21.15.46:0/1562577295 learned_addr learned my addr 172.21.15.46:0/1562577295 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f1823fff640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c003a00 0x7f18241552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f1828964640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1824162380 0x7f1824162790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 -- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1824162380 msgr2=0x7f1824162790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1824162380 0x7f1824162790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 -- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c003a00 msgr2=0x7f18241552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c003a00 0x7f18241552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.187+0000 7f18237fe640 1 -- 172.21.15.46:0/1562577295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1824162cd0 con 0x7f1824158200 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.188+0000 7f18237fe640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 0x7f18241f54e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1818011540 tx=0x7f1818011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.188+0000 7f1820ff9640 1 -- 172.21.15.46:0/1562577295 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818015870 con 0x7f1824158200 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.188+0000 7f1820ff9640 1 -- 172.21.15.46:0/1562577295 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1818015a10 con 0x7f1824158200 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.188+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18241588d0 con 0x7f1824158200 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.188+0000 7f1820ff9640 1 -- 172.21.15.46:0/1562577295 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818015dc0 con 0x7f1824158200 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.188+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1824158e90 con 0x7f1824158200 2024-05-09T03:15:36.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.189+0000 7f1820ff9640 1 -- 172.21.15.46:0/1562577295 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f181801d400 con 0x7f1824158200 2024-05-09T03:15:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.190+0000 7f1820ff9640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18100762d0 0x7f18100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.190+0000 7f1820ff9640 1 -- 172.21.15.46:0/1562577295 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f1818097e10 con 0x7f1824158200 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.190+0000 7f1823fff640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18100762d0 0x7f18100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.191+0000 7f1831278740 1 Processor -- start 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.191+0000 7f1831278740 1 -- start start 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.191+0000 7f1823fff640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18100762d0 0x7f18100787e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1814032000 tx=0x7f181402d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64f2270 0x55f9a68c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68dcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1831278740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a68dd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1831278740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f9a67fa4e0 con 0x55f9a64d1590 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f9a6795ff0 con 0x55f9a64f2270 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f9a67f64a0 con 0x55f9a64c93b0 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a68dd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a68dd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57080/0 (socket says 172.21.15.46:57080) 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 -- 172.21.15.46:0/2870240531 learned_addr learned my addr 172.21.15.46:0/2870240531 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1823fff640 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64f2270 0x55f9a68c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f18237fe640 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68dcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 -- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 msgr2=0x55f9a68dcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68dcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 -- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64f2270 msgr2=0x55f9a68c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64f2270 0x55f9a68c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.192+0000 7f1828964640 1 -- 172.21.15.46:0/2870240531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a677c720 con 0x55f9a64d1590 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.193+0000 7f1828964640 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a68dd120 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1824165050 tx=0x7f181c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ac7806d1475164 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.193+0000 7f1809ffb640 1 -- 172.21.15.46:0/2870240531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c016810 con 0x55f9a64d1590 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.193+0000 7f1809ffb640 1 -- 172.21.15.46:0/2870240531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c01f070 con 0x55f9a64d1590 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.193+0000 7f1809ffb640 1 -- 172.21.15.46:0/2870240531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c016bf0 con 0x55f9a64d1590 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 -- 172.21.15.46:0/2870240531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 msgr2=0x55f9a68dd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a68dd120 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1824165050 tx=0x7f181c009750 comp rx=0 tx=0).stop 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 -- 172.21.15.46:0/2870240531 shutdown_connections 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a68dd120 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68dcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 --2- 172.21.15.46:0/2870240531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64f2270 0x55f9a68c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 -- 172.21.15.46:0/2870240531 >> 172.21.15.46:0/2870240531 conn(0x55f9a64e63c0 msgr2=0x55f9a6897510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 -- 172.21.15.46:0/2870240531 shutdown_connections 2024-05-09T03:15:36.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.194+0000 7f1831278740 1 -- 172.21.15.46:0/2870240531 wait complete. 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.195+0000 7f1831278740 1 -- start start 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.195+0000 7f1831278740 1 Processor -- start 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68d8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a68b1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68da4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f9a6781760 con 0x55f9a64f2270 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f9a68b2600 con 0x55f9a64d1590 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f9a68d91e0 con 0x55f9a64c93b0 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1831278740 10 client.0 _ll_register_callbacks cb 0x55f9a67a44b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1828964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68da4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1828964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68da4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57096/0 (socket says 172.21.15.46:57096) 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1828964640 1 -- 172.21.15.46:0/2391895431 learned_addr learned my addr 172.21.15.46:0/2391895431 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f18237fe640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a68b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.196+0000 7f1823fff640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68d8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1828964640 1 -- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 msgr2=0x55f9a68d8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1828964640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68d8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1828964640 1 -- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 msgr2=0x55f9a68b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1828964640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a68b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1828964640 1 -- 172.21.15.46:0/2391895431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a67c2bd0 con 0x55f9a64f2270 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1828964640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68da4d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f181c01c040 tx=0x7f181c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f95da67d019183 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1802ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1802ffd640 1 -- 172.21.15.46:0/2391895431 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c016a20 con 0x55f9a64f2270 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1802ffd640 1 -- 172.21.15.46:0/2391895431 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f181c016bc0 con 0x55f9a64f2270 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f181c01f070 con 0x55f9a64f2270 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1802ffd640 1 -- 172.21.15.46:0/2391895431 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f181c034050 con 0x55f9a64f2270 2024-05-09T03:15:36.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1831278740 10 client.6732 fetch_fsmap learned FSMap version 76 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x55f9a67a8280 con 0x55f9a64f2270 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1802ffd640 1 -- 172.21.15.46:0/2391895431 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c01f070 con 0x55f9a64f2270 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1802ffd640 1 -- 172.21.15.46:0/2391895431 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 76) ==== 3494+0+0 (secure 0 0 0) 0x7f181c037020 con 0x55f9a64f2270 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.197+0000 7f1831278740 10 client.6732 fetch_fsmap finished waiting for FSMap version 76 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1831278740 10 client.6732 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1831278740 20 client.6732 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1831278740 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a64b0e40 0x55f9a68d7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1831278740 4 client.6732 mds_command: new command op to 5977 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 --> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9a68080f0 con 0x55f9a64b0e40 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1823fff640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a64b0e40 0x55f9a68d7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1802ffd640 10 client.6732 ms_handle_connect on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.198+0000 7f1823fff640 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a64b0e40 0x55f9a68d7260 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e319ec1fe9680e2 server_cookie=5ee4bfeafc5622dd in_seq=0 out_seq=0 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1802ffd640 1 -- 172.21.15.46:0/2391895431 <== mds.0 v2:172.21.15.88:6834/3673458792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f9a68080f0 con 0x55f9a64b0e40 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1802ffd640 10 client.6732 handle_command_reply: tid=0 2024-05-09T03:15:36.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 client.6732 shutdown 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 msgr2=0x55f9a68da4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68da4d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f181c01c040 tx=0x7f181c004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 shutdown_connections 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a64b0e40 0x55f9a68d7260 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68da4d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a68b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1802ffd640 0 client.6732 ms_handle_reset on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 --2- 172.21.15.46:0/2391895431 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68d8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 >> 172.21.15.46:0/2391895431 conn(0x55f9a64e63c0 msgr2=0x55f9a68bea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 shutdown_connections 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.199+0000 7f1831278740 1 -- 172.21.15.46:0/2391895431 wait complete. 2024-05-09T03:15:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.200+0000 7f1831278740 20 client.6732 trim_cache size 0 max 16384 2024-05-09T03:15:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.218+0000 7f1831278740 1 Processor -- start 2024-05-09T03:15:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.218+0000 7f1831278740 1 -- start start 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64c93b0 0x55f9a6892570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1831278740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a6898170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64f2270 0x55f9a68c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1831278740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1824201fb0 con 0x55f9a64d1590 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18241f9fb0 con 0x55f9a64c93b0 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1824201960 con 0x55f9a64f2270 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a6898170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a6898170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57102/0 (socket says 172.21.15.46:57102) 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 -- 172.21.15.46:0/1201416122 learned_addr learned my addr 172.21.15.46:0/1201416122 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1828964640 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64f2270 0x55f9a68c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 -- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64f2270 msgr2=0x55f9a68c4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64f2270 0x55f9a68c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f1823fff640 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64c93b0 0x55f9a6892570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 -- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64c93b0 msgr2=0x55f9a6892570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64c93b0 0x55f9a6892570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 -- 172.21.15.46:0/1201416122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9a655d850 con 0x55f9a64d1590 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18237fe640 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a6898170 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1818012010 tx=0x7f18180114c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbcea0af85ffc72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18097fa640 1 -- 172.21.15.46:0/1201416122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818069920 con 0x55f9a64d1590 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18097fa640 1 -- 172.21.15.46:0/1201416122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1818069ac0 con 0x55f9a64d1590 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.220+0000 7f18097fa640 1 -- 172.21.15.46:0/1201416122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18180604a0 con 0x55f9a64d1590 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 -- 172.21.15.46:0/1201416122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 msgr2=0x55f9a6898170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a6898170 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1818012010 tx=0x7f18180114c0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 -- 172.21.15.46:0/1201416122 shutdown_connections 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64f2270 0x55f9a68c4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64d1590 0x55f9a6898170 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 --2- 172.21.15.46:0/1201416122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64c93b0 0x55f9a6892570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 -- 172.21.15.46:0/1201416122 >> 172.21.15.46:0/1201416122 conn(0x55f9a64e63c0 msgr2=0x55f9a68b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 -- 172.21.15.46:0/1201416122 shutdown_connections 2024-05-09T03:15:36.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.221+0000 7f1831278740 1 -- 172.21.15.46:0/1201416122 wait complete. 2024-05-09T03:15:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.222+0000 7f1831278740 1 -- start start 2024-05-09T03:15:36.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.222+0000 7f1831278740 1 Processor -- start 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a693f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f9a68daab0 con 0x55f9a64f2270 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f9a68bda30 con 0x55f9a64d1590 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f9a68dba40 con 0x55f9a64c93b0 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.223+0000 7f1831278740 10 client.0 _ll_register_callbacks cb 0x7f1818009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57114/0 (socket says 172.21.15.46:57114) 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 -- 172.21.15.46:0/854836209 learned_addr learned my addr 172.21.15.46:0/854836209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f18237fe640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a693f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1823fff640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 -- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 msgr2=0x55f9a68afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68afb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 -- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 msgr2=0x55f9a693f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a693f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 -- 172.21.15.46:0/854836209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9a65d46b0 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1828964640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68c4c10 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f181c00b010 tx=0x7f181c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caa75ac837fb59b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1802ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1802ffd640 1 -- 172.21.15.46:0/854836209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c0044f0 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1802ffd640 1 -- 172.21.15.46:0/854836209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f181c004690 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1818069ac0 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1802ffd640 1 -- 172.21.15.46:0/854836209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f181c040050 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1831278740 10 client.6738 fetch_fsmap learned FSMap version 76 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.224+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x55f9a67b6630 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1802ffd640 1 -- 172.21.15.46:0/854836209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1818069ac0 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1802ffd640 1 -- 172.21.15.46:0/854836209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 76) ==== 3494+0+0 (secure 0 0 0) 0x7f181c043020 con 0x55f9a64f2270 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1831278740 10 client.6738 fetch_fsmap finished waiting for FSMap version 76 2024-05-09T03:15:36.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1831278740 10 client.6738 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1831278740 20 client.6738 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1831278740 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a6896270 0x55f9a695d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1831278740 4 client.6738 mds_command: new command op to 5977 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 --> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] -- command(tid 0: {"prefix": "status"}) -- 0x55f9a6801f60 con 0x55f9a6896270 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1823fff640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a6896270 0x55f9a695d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1802ffd640 10 client.6738 ms_handle_connect on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.225+0000 7f1823fff640 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a6896270 0x55f9a695d460 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11adfbd4a428dcb6 server_cookie=984742e87bd8b873 in_seq=0 out_seq=0 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1802ffd640 1 -- 172.21.15.46:0/854836209 <== mds.0 v2:172.21.15.88:6834/3673458792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55f9a6801f60 con 0x55f9a6896270 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1802ffd640 10 client.6738 handle_command_reply: tid=0 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 client.6738 shutdown 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 msgr2=0x55f9a68c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68c4c10 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f181c00b010 tx=0x7f181c00b040 comp rx=0 tx=0).stop 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 shutdown_connections 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55f9a6896270 0x55f9a695d460 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f9a64f2270 0x55f9a68c4c10 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f9a64d1590 0x55f9a693f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1802ffd640 0 client.6738 ms_handle_reset on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 --2- 172.21.15.46:0/854836209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f9a64c93b0 0x55f9a68afb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 >> 172.21.15.46:0/854836209 conn(0x55f9a64e63c0 msgr2=0x55f9a68937e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 shutdown_connections 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 1 -- 172.21.15.46:0/854836209 wait complete. 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f1831278740 20 client.6738 trim_cache size 0 max 16384 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:15:36.233 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5977, 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.768646532, 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 76, 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 71, 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 71, 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 47.932414051000002 2024-05-09T03:15:36.234 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:15:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18100762d0 msgr2=0x7f18100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f182a3c7640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18100762d0 0x7f18100787e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1814032000 tx=0x7f181402d000 comp rx=0 tx=0).stop 2024-05-09T03:15:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 msgr2=0x7f18241f54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.226+0000 7f182a3c7640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 0x7f18241f54e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1818011540 tx=0x7f1818011a00 comp rx=0 tx=0).stop 2024-05-09T03:15:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 shutdown_connections 2024-05-09T03:15:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1824162380 0x7f1824162790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18100762d0 0x7f18100787e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1824158200 0x7f18241f54e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 --2- 172.21.15.46:0/1562577295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f180c003a00 0x7f18241552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 >> 172.21.15.46:0/1562577295 conn(0x7f182408d3f0 msgr2=0x7f18240b7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 shutdown_connections 2024-05-09T03:15:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.227+0000 7f182a3c7640 1 -- 172.21.15.46:0/1562577295 wait complete. 2024-05-09T03:15:36.240 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat a/ && find a/') 2024-05-09T03:15:36.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T03:15:36.294 INFO:teuthology.orchestra.run.smithi046.stdout: File: a/ 2024-05-09T03:15:36.294 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-05-09T03:15:36.294 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 2ah/42d Inode: 1099511627776 Links: 4 2024-05-09T03:15:36.294 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1316/ ubuntu) 2024-05-09T03:15:36.295 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-09T03:15:36.295 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-05-09 03:15:02.385394943 +0000 2024-05-09T03:15:36.295 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-05-09 03:15:07.999986251 +0000 2024-05-09T03:15:36.295 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-05-09 03:15:07.999986251 +0000 2024-05-09T03:15:36.295 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: - 2024-05-09T03:15:36.309 INFO:teuthology.orchestra.run.smithi046.stdout:a/ 2024-05-09T03:15:36.309 INFO:teuthology.orchestra.run.smithi046.stdout:a/z 2024-05-09T03:15:36.309 INFO:teuthology.orchestra.run.smithi046.stdout:a/z/d 2024-05-09T03:15:36.309 INFO:teuthology.orchestra.run.smithi046.stdout:a/c 2024-05-09T03:15:36.309 INFO:teuthology.orchestra.run.smithi046.stdout:a/c/d 2024-05-09T03:15:36.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:15:36.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-05-09T03:15:36.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.430+0000 7f26658f0640 1 Processor -- start 2024-05-09T03:15:36.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.430+0000 7f26658f0640 1 -- start start 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.431+0000 7f26658f0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f266006e580 0x7f266006e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.431+0000 7f26658f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f26600cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.431+0000 7f26658f0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2660074890 con 0x7f26600d1110 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.431+0000 7f26658f0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f266006d9f0 con 0x7f266006e580 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.431+0000 7f26658f0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f266009d440 con 0x7f26600cebd0 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.431+0000 7f265effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f26600d1110 0x7f266009cd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60144/0 (socket says 172.21.15.46:60144) 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265effd640 1 -- 172.21.15.46:0/2971014848 learned_addr learned my addr 172.21.15.46:0/2971014848 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265e7fc640 1 --2- 172.21.15.46:0/2971014848 >> v2:172.21.15.88:3300/0 conn(0x7f266006e580 0x7f266006e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265f7fe640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f26600cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265dffb640 1 -- 172.21.15.46:0/2971014848 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526891544 0 0) 0x7f2650001490 con 0x7f26600d1110 2024-05-09T03:15:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265dffb640 1 -- 172.21.15.46:0/2971014848 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f264c003520 con 0x7f26600d1110 2024-05-09T03:15:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265e7fc640 1 -- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 msgr2=0x7f26600cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265e7fc640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f26600cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265e7fc640 1 -- 172.21.15.46:0/2971014848 >> v1:172.21.15.46:6789/0 conn(0x7f26600d1110 legacy=0x7f266009cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265e7fc640 1 -- 172.21.15.46:0/2971014848 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266006e130 con 0x7f266006e580 2024-05-09T03:15:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.432+0000 7f265f7fe640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f26600cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.433+0000 7f265e7fc640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f266006e580 0x7f266006e990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26480097a0 tx=0x7f264802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=351d94067d4cd729 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.433+0000 7f265dffb640 1 -- 172.21.15.46:0/2971014848 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648037070 con 0x7f266006e580 2024-05-09T03:15:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.433+0000 7f265dffb640 1 -- 172.21.15.46:0/2971014848 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2648032440 con 0x7f266006e580 2024-05-09T03:15:36.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.433+0000 7f265dffb640 1 -- 172.21.15.46:0/2971014848 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648032810 con 0x7f266006e580 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 -- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f266006e580 msgr2=0x7f266006e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f266006e580 0x7f266006e990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f26480097a0 tx=0x7f264802a180 comp rx=0 tx=0).stop 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 -- 172.21.15.46:0/2971014848 shutdown_connections 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f26600cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 --2- 172.21.15.46:0/2971014848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f266006e580 0x7f266006e990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 -- 172.21.15.46:0/2971014848 >> 172.21.15.46:0/2971014848 conn(0x7f26600d05f0 msgr2=0x7f26600d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 -- 172.21.15.46:0/2971014848 shutdown_connections 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.434+0000 7f26658f0640 1 -- 172.21.15.46:0/2971014848 wait complete. 2024-05-09T03:15:36.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.436+0000 7f26658f0640 1 Processor -- start 2024-05-09T03:15:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.436+0000 7f26658f0640 1 -- start start 2024-05-09T03:15:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f26658f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 0x7f26601ddc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f26658f0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f266009e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f26658f0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26600d1110 0x7f266009ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f26658f0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f266006ddf0 con 0x7f266006e580 2024-05-09T03:15:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f26658f0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f26600cf520 con 0x7f26600d1110 2024-05-09T03:15:36.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f26658f0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f26600b8ec0 con 0x7f26600cebd0 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f265effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 0x7f26601ddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f265effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 0x7f26601ddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57118/0 (socket says 172.21.15.46:57118) 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.437+0000 7f265effd640 1 -- 172.21.15.46:0/2469875045 learned_addr learned my addr 172.21.15.46:0/2469875045 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265f7fe640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26600d1110 0x7f266009ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265e7fc640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f266009e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265effd640 1 -- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 msgr2=0x7f266009e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265effd640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f266009e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265effd640 1 -- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26600d1110 msgr2=0x7f266009ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265effd640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26600d1110 0x7f266009ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265effd640 1 -- 172.21.15.46:0/2469875045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26601de1b0 con 0x7f266006e580 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265f7fe640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26600d1110 0x7f266009ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f265effd640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 0x7f26601ddc70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f265000cb20 tx=0x7f265000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f2657fff640 1 -- 172.21.15.46:0/2469875045 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265000aca0 con 0x7f266006e580 2024-05-09T03:15:36.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f2657fff640 1 -- 172.21.15.46:0/2469875045 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f265000ae40 con 0x7f266006e580 2024-05-09T03:15:36.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266009f8a0 con 0x7f266006e580 2024-05-09T03:15:36.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f2657fff640 1 -- 172.21.15.46:0/2469875045 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650012600 con 0x7f266006e580 2024-05-09T03:15:36.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.438+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f266009dd90 con 0x7f266006e580 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.440+0000 7f2657fff640 1 -- 172.21.15.46:0/2469875045 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26500127a0 con 0x7f266006e580 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.440+0000 7f2657fff640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2640076300 0x7f2640078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.440+0000 7f2657fff640 1 -- 172.21.15.46:0/2469875045 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f2650097cc0 con 0x7f266006e580 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.441+0000 7f265e7fc640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2640076300 0x7f2640078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.441+0000 7f265e7fc640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2640076300 0x7f2640078810 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2648034000 tx=0x7f264802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.441+0000 7f266c7a0740 1 Processor -- start 2024-05-09T03:15:36.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.441+0000 7f266c7a0740 1 -- start start 2024-05-09T03:15:36.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0d0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c0d0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f266c7a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e28bfee1a0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e28bfa7170 con 0x55e28bce6390 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e28bfea160 con 0x55e28bcc56b0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0d0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57132/0 (socket says 172.21.15.46:57132) 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 -- 172.21.15.46:0/483487352 learned_addr learned my addr 172.21.15.46:0/483487352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265effd640 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265f7fe640 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c0d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 -- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 msgr2=0x55e28c0d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c0d0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 -- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 msgr2=0x55e28c0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 -- 172.21.15.46:0/483487352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e28bfb9390 con 0x55e28bcbd4d0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.443+0000 7f265e7fc640 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0d0240 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f264802a1c0 tx=0x7f264802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44f85967858b1bc3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.444+0000 7f26557fa640 1 -- 172.21.15.46:0/483487352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264804c070 con 0x55e28bcbd4d0 2024-05-09T03:15:36.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.444+0000 7f26557fa640 1 -- 172.21.15.46:0/483487352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2648032cf0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.444+0000 7f26557fa640 1 -- 172.21.15.46:0/483487352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648037040 con 0x55e28bcbd4d0 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.444+0000 7f266c7a0740 1 -- 172.21.15.46:0/483487352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 msgr2=0x55e28c0d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.444+0000 7f266c7a0740 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0d0240 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f264802a1c0 tx=0x7f264802c000 comp rx=0 tx=0).stop 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 -- 172.21.15.46:0/483487352 shutdown_connections 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c0d0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0d0240 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 --2- 172.21.15.46:0/483487352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 -- 172.21.15.46:0/483487352 >> 172.21.15.46:0/483487352 conn(0x55e28bcda4e0 msgr2=0x55e28c08ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 -- 172.21.15.46:0/483487352 shutdown_connections 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 -- 172.21.15.46:0/483487352 wait complete. 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.445+0000 7f266c7a0740 1 -- start start 2024-05-09T03:15:36.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.446+0000 7f266c7a0740 1 Processor -- start 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c085140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0df7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e28bf748c0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e28c0e0710 con 0x55e28bce6390 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e28c0cae80 con 0x55e28bcc56b0 2024-05-09T03:15:36.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f266c7a0740 10 client.0 _ll_register_callbacks cb 0x55e28bf90860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57146/0 (socket says 172.21.15.46:57146) 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 -- 172.21.15.46:0/1255710031 learned_addr learned my addr 172.21.15.46:0/1255710031 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265e7fc640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c085140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265f7fe640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0df7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 -- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 msgr2=0x55e28c085140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c085140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 -- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 msgr2=0x55e28c0df7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0df7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.447+0000 7f265effd640 1 -- 172.21.15.46:0/1255710031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e28bdc87d0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f265effd640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0ca7d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f265000caf0 tx=0x7f2650061350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6e98151bcc0804 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f263e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f263e7fc640 1 -- 172.21.15.46:0/1255710031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2650060e00 con 0x55e28bcbd4d0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f263e7fc640 1 -- 172.21.15.46:0/1255710031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f265005c770 con 0x55e28bcbd4d0 2024-05-09T03:15:36.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2648032cf0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f263e7fc640 1 -- 172.21.15.46:0/1255710031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f26500651b0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 10 client.6750 fetch_fsmap learned FSMap version 76 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x55e28bd51970 con 0x55e28bcbd4d0 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f263e7fc640 1 -- 172.21.15.46:0/1255710031 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2648032cf0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f263e7fc640 1 -- 172.21.15.46:0/1255710031 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 76) ==== 3494+0+0 (secure 0 0 0) 0x7f265005c970 con 0x55e28bcbd4d0 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 10 client.6750 fetch_fsmap finished waiting for FSMap version 76 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 10 client.6750 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 20 client.6750 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28bca4f60 0x55e28c0e01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 4 client.6750 mds_command: new command op to 5977 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 --> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e28bffbdb0 con 0x55e28bca4f60 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.448+0000 7f265e7fc640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28bca4f60 0x55e28c0e01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.449+0000 7f263e7fc640 10 client.6750 ms_handle_connect on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.449+0000 7f265e7fc640 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28bca4f60 0x55e28c0e01b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fbcb67f9df157be server_cookie=3f223fbfc686718f in_seq=0 out_seq=0 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f263e7fc640 1 -- 172.21.15.46:0/1255710031 <== mds.0 v2:172.21.15.88:6834/3673458792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e28bffbdb0 con 0x55e28bca4f60 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f263e7fc640 10 client.6750 handle_command_reply: tid=0 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 client.6750 shutdown 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 msgr2=0x55e28c0ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0ca7d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f265000caf0 tx=0x7f2650061350 comp rx=0 tx=0).stop 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 shutdown_connections 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28bca4f60 0x55e28c0e01b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c0df7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f263e7fc640 0 client.6750 ms_handle_reset on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c085140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 --2- 172.21.15.46:0/1255710031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcbd4d0 0x55e28c0ca7d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 >> 172.21.15.46:0/1255710031 conn(0x55e28bcda4e0 msgr2=0x55e28c08ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 shutdown_connections 2024-05-09T03:15:36.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 1 -- 172.21.15.46:0/1255710031 wait complete. 2024-05-09T03:15:36.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.450+0000 7f266c7a0740 20 client.6750 trim_cache size 0 max 16384 2024-05-09T03:15:36.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.466+0000 7f266c7a0740 1 Processor -- start 2024-05-09T03:15:36.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.466+0000 7f266c7a0740 1 -- start start 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bcbd4d0 0x55e28c08b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c08b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 0x55e28c0cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f266c7a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f26602030b0 con 0x55e28bce6390 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f26602020b0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f26600c9fa0 con 0x55e28bcc56b0 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f265f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 0x55e28c0cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f265f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 0x55e28c0cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57148/0 (socket says 172.21.15.46:57148) 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f265f7fe640 1 -- 172.21.15.46:0/479340540 learned_addr learned my addr 172.21.15.46:0/479340540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f265e7fc640 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c08b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.467+0000 7f265effd640 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bcbd4d0 0x55e28c08b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265f7fe640 1 -- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 msgr2=0x55e28c08b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265f7fe640 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c08b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265f7fe640 1 -- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bcbd4d0 msgr2=0x55e28c08b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265f7fe640 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bcbd4d0 0x55e28c08b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265f7fe640 1 -- 172.21.15.46:0/479340540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e28bc96f60 con 0x55e28bce6390 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265effd640 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bcbd4d0 0x55e28c08b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f265f7fe640 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 0x55e28c0cf9e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2658009c00 tx=0x7f2658009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770250fc961e60ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f2654ff9640 1 -- 172.21.15.46:0/479340540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265800f040 con 0x55e28bce6390 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f2654ff9640 1 -- 172.21.15.46:0/479340540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f265801b070 con 0x55e28bce6390 2024-05-09T03:15:36.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.468+0000 7f2654ff9640 1 -- 172.21.15.46:0/479340540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658020690 con 0x55e28bce6390 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 -- 172.21.15.46:0/479340540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 msgr2=0x55e28c0cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 0x55e28c0cf9e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2658009c00 tx=0x7f2658009190 comp rx=0 tx=0).stop 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 -- 172.21.15.46:0/479340540 shutdown_connections 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bce6390 0x55e28c0cf9e0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcc56b0 0x55e28c08b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 --2- 172.21.15.46:0/479340540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bcbd4d0 0x55e28c08b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 -- 172.21.15.46:0/479340540 >> 172.21.15.46:0/479340540 conn(0x55e28bcda4e0 msgr2=0x55e28c0c9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 -- 172.21.15.46:0/479340540 shutdown_connections 2024-05-09T03:15:36.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.469+0000 7f266c7a0740 1 -- 172.21.15.46:0/479340540 wait complete. 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.470+0000 7f266c7a0740 1 -- start start 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.470+0000 7f266c7a0740 1 Processor -- start 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcbd4d0 0x55e28c0f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 0x55e28c0f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c134330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e28c0caa40 con 0x55e28bcc56b0 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f26600ca5f0 con 0x55e28bce6390 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e28c1353d0 con 0x55e28bcbd4d0 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f266c7a0740 10 client.0 _ll_register_callbacks cb 0x7f2650061500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 0x55e28c0f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 0x55e28c0f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57160/0 (socket says 172.21.15.46:57160) 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 -- 172.21.15.46:0/788971133 learned_addr learned my addr 172.21.15.46:0/788971133 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265f7fe640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265effd640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcbd4d0 0x55e28c0f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 -- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcbd4d0 msgr2=0x55e28c0f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcbd4d0 0x55e28c0f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 -- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 msgr2=0x55e28c134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265e7fc640 1 -- 172.21.15.46:0/788971133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e28c0b7380 con 0x55e28bcc56b0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.471+0000 7f265f7fe640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f265e7fc640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 0x55e28c0f2de0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f26480098c0 tx=0x7f2648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9e923c60ad0209 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f263e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f263e7fc640 1 -- 172.21.15.46:0/788971133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648059060 con 0x55e28bcc56b0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f263e7fc640 1 -- 172.21.15.46:0/788971133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f2648004330 con 0x55e28bcc56b0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f265801b070 con 0x55e28bcc56b0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f263e7fc640 1 -- 172.21.15.46:0/788971133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) ==== 24+0+0 (secure 0 0 0) 0x7f264805f050 con 0x55e28bcc56b0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 10 client.6756 fetch_fsmap learned FSMap version 76 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) -- 0x55e28c1351d0 con 0x55e28bcc56b0 2024-05-09T03:15:36.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f263e7fc640 1 -- 172.21.15.46:0/788971133 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f265801b070 con 0x55e28bcc56b0 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f263e7fc640 1 -- 172.21.15.46:0/788971133 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 76) ==== 3494+0+0 (secure 0 0 0) 0x7f2648062020 con 0x55e28bcc56b0 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 10 client.6756 fetch_fsmap finished waiting for FSMap version 76 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 10 client.6756 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 20 client.6756 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28c133bb0 0x55e28c134870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 4 client.6756 mds_command: new command op to 5977 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.472+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 --> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e28bff5c20 con 0x55e28c133bb0 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.473+0000 7f265effd640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28c133bb0 0x55e28c134870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.473+0000 7f263e7fc640 10 client.6756 ms_handle_connect on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.473+0000 7f265effd640 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28c133bb0 0x55e28c134870 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3af0568505bfd133 server_cookie=6988fe9f1be6ab75 in_seq=0 out_seq=0 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.484+0000 7f263e7fc640 1 -- 172.21.15.46:0/788971133 <== mds.0 v2:172.21.15.88:6834/3673458792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+25 (crc 0 0 0) 0x55e28bff5c20 con 0x55e28c133bb0 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.484+0000 7f263e7fc640 10 client.6756 handle_command_reply: tid=0 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.484+0000 7f266c7a0740 1 client.6756 shutdown 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.484+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 msgr2=0x55e28c0f2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.484+0000 7f266c7a0740 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 0x55e28c0f2de0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f26480098c0 tx=0x7f2648004730 comp rx=0 tx=0).stop 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 shutdown_connections 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:6834/3673458792,v1:172.21.15.88:6836/3673458792] conn(0x55e28c133bb0 0x55e28c134870 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e28bce6390 0x55e28c134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e28bcc56b0 0x55e28c0f2de0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f263e7fc640 0 client.6756 ms_handle_reset on v2:172.21.15.88:6834/3673458792 2024-05-09T03:15:36.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 --2- 172.21.15.46:0/788971133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e28bcbd4d0 0x55e28c0f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 >> 172.21.15.46:0/788971133 conn(0x55e28bcda4e0 msgr2=0x55e28c0d0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 shutdown_connections 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 1 -- 172.21.15.46:0/788971133 wait complete. 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f266c7a0740 20 client.6756 trim_cache size 0 max 16384 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0 2024-05-09T03:15:36.488 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2640076300 msgr2=0x7f2640078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2640076300 0x7f2640078810 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f2648034000 tx=0x7f264802e000 comp rx=0 tx=0).stop 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 msgr2=0x7f26601ddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 0x7f26601ddc70 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f265000cb20 tx=0x7f265000a060 comp rx=0 tx=0).stop 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 shutdown_connections 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26600d1110 0x7f266009ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f26600cebd0 0x7f266009e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2640076300 0x7f2640078810 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 --2- 172.21.15.46:0/2469875045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f266006e580 0x7f26601ddc70 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.485+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 >> 172.21.15.46:0/2469875045 conn(0x7f26600d05f0 msgr2=0x7f26600d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.486+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 shutdown_connections 2024-05-09T03:15:36.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.486+0000 7f26658f0640 1 -- 172.21.15.46:0/2469875045 wait complete. 2024-05-09T03:15:36.498 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:15:36.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:15:36.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:15:36.613 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:15:36.613 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:15:36.613 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:15:36.613 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:36.629 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:15:36.630 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:36.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:15:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.840+0000 7f8d9651f640 1 Processor -- start 2024-05-09T03:15:36.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.840+0000 7f8d9651f640 1 -- start start 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d9651f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8d900c78a0 0x7f8d900b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d9651f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d90072500 0x7f8d90163e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d9651f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d901461a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d9651f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8d90145d10 con 0x7f8d90072500 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d9651f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8d900c8720 con 0x7f8d900c78a0 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d9651f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8d900c7e80 con 0x7f8d90164390 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d90072500 0x7f8d90163e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8d90072500 0x7f8d90163e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57170/0 (socket says 172.21.15.46:57170) 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 -- 172.21.15.46:0/2107819890 learned_addr learned my addr 172.21.15.46:0/2107819890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d94abc640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d901461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d8ffff640 1 --2- 172.21.15.46:0/2107819890 >> v2:172.21.15.88:3300/0 conn(0x7f8d900c78a0 0x7f8d900b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 -- 172.21.15.46:0/2107819890 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 msgr2=0x7f8d901461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d901461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 -- 172.21.15.46:0/2107819890 >> v2:172.21.15.88:3300/0 conn(0x7f8d900c78a0 msgr2=0x7f8d900b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 --2- 172.21.15.46:0/2107819890 >> v2:172.21.15.88:3300/0 conn(0x7f8d900c78a0 0x7f8d900b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d87fff640 1 -- 172.21.15.46:0/2107819890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d900c8160 con 0x7f8d90072500 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d8ffff640 1 --2- 172.21.15.46:0/2107819890 >> v2:172.21.15.88:3300/0 conn(0x7f8d900c78a0 0x7f8d900b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.842+0000 7f8d94abc640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d901461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.843+0000 7f8d87fff640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d90072500 0x7f8d90163e50 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8d900cd000 tx=0x7f8d7802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40913e08fce55d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.843+0000 7f8d8f7fe640 1 -- 172.21.15.46:0/2107819890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78037070 con 0x7f8d90072500 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.843+0000 7f8d8f7fe640 1 -- 172.21.15.46:0/2107819890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d78032440 con 0x7f8d90072500 2024-05-09T03:15:36.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.843+0000 7f8d8f7fe640 1 -- 172.21.15.46:0/2107819890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78032810 con 0x7f8d90072500 2024-05-09T03:15:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 -- 172.21.15.46:0/2107819890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d90072500 msgr2=0x7f8d90163e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d90072500 0x7f8d90163e50 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8d900cd000 tx=0x7f8d7802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 -- 172.21.15.46:0/2107819890 shutdown_connections 2024-05-09T03:15:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d901461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2107819890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d90072500 0x7f8d90163e50 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2107819890 >> v2:172.21.15.88:3300/0 conn(0x7f8d900c78a0 0x7f8d900b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 -- 172.21.15.46:0/2107819890 >> 172.21.15.46:0/2107819890 conn(0x7f8d900705c0 msgr2=0x7f8d900709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 -- 172.21.15.46:0/2107819890 shutdown_connections 2024-05-09T03:15:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.844+0000 7f8d9651f640 1 -- 172.21.15.46:0/2107819890 wait complete. 2024-05-09T03:15:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.846+0000 7f8d9651f640 1 Processor -- start 2024-05-09T03:15:36.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.846+0000 7f8d9651f640 1 -- start start 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d9651f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d90072500 0x7f8d90144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d9651f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 0x7f8d900cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d9651f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d900b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d9651f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8d901466e0 con 0x7f8d900c78a0 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d9651f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8d90072910 con 0x7f8d90072500 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d9651f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8d90146860 con 0x7f8d90164390 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 0x7f8d900cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 0x7f8d900cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57186/0 (socket says 172.21.15.46:57186) 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 -- 172.21.15.46:0/2294889342 learned_addr learned my addr 172.21.15.46:0/2294889342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d8ffff640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d90072500 0x7f8d90144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d94abc640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d900b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 -- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 msgr2=0x7f8d900b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d900b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 -- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d90072500 msgr2=0x7f8d90144900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d90072500 0x7f8d90144900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.847+0000 7f8d87fff640 1 -- 172.21.15.46:0/2294889342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d90147290 con 0x7f8d900c78a0 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.848+0000 7f8d87fff640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 0x7f8d900cc220 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8d7802a220 tx=0x7f8d780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.848+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78037040 con 0x7f8d900c78a0 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.848+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d78042070 con 0x7f8d900c78a0 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.848+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d90147570 con 0x7f8d900c78a0 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.848+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78030bb0 con 0x7f8d900c78a0 2024-05-09T03:15:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.848+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d90147b30 con 0x7f8d900c78a0 2024-05-09T03:15:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.849+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9006a270 con 0x7f8d900c78a0 2024-05-09T03:15:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.849+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8d78047460 con 0x7f8d900c78a0 2024-05-09T03:15:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.850+0000 7f8d8d7fa640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d700762d0 0x7f8d700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.850+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f8d780b8410 con 0x7f8d900c78a0 2024-05-09T03:15:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.851+0000 7f8d8ffff640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d700762d0 0x7f8d700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.851+0000 7f8d8ffff640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d700762d0 0x7f8d700787e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8d900734d0 tx=0x7f8d80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:36.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:36.855+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8d78081bb0 con 0x7f8d900c78a0 2024-05-09T03:15:36.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:15:36.878 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:15:37.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.065+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8d900d0290 con 0x7f8d900c78a0 2024-05-09T03:15:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.065+0000 7f8d8d7fa640 1 -- 172.21.15.46:0/2294889342 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v71) ==== 80+0+120 (secure 0 0 0) 0x7f8d780814b0 con 0x7f8d900c78a0 2024-05-09T03:15:37.067 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/131652171 2024-05-10T03:15:31.456349+0000 2024-05-09T03:15:37.067 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/131652171 2024-05-10T03:15:31.456349+0000 2024-05-09T03:15:37.067 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:15:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d700762d0 msgr2=0x7f8d700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d700762d0 0x7f8d700787e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8d900734d0 tx=0x7f8d80008040 comp rx=0 tx=0).stop 2024-05-09T03:15:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 msgr2=0x7f8d900cc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 0x7f8d900cc220 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8d7802a220 tx=0x7f8d780046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 shutdown_connections 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8d90164390 0x7f8d900b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d700762d0 0x7f8d700787e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d900c78a0 0x7f8d900cc220 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 --2- 172.21.15.46:0/2294889342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d90072500 0x7f8d90144900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 >> 172.21.15.46:0/2294889342 conn(0x7f8d900705c0 msgr2=0x7f8d90071b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 shutdown_connections 2024-05-09T03:15:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.067+0000 7f8d9651f640 1 -- 172.21.15.46:0/2294889342 wait complete. 2024-05-09T03:15:37.081 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:15:37.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:37.081 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:37.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[59051]: fuse finished with error 0 and tester_r 0 2024-05-09T03:15:37.107 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:15:37.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:37.108 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:15:37.190 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:15:37.190 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:15:37.190 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:37.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:15:37.210 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:15:37.210 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:15:37.210 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:37.247 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:15:37.247 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:15:37.247 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:15:37.247 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:37.268 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:15:37.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.399+0000 7f1b7d0ee640 1 Processor -- start 2024-05-09T03:15:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.399+0000 7f1b7d0ee640 1 -- start start 2024-05-09T03:15:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7d0ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 0x7f1b780b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7d0ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 0x7f1b780d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7d0ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b780d0dd0 con 0x7f1b780ce6a0 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7d0ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1b780d2b80 con 0x7f1b780b92e0 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7d0ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1b780d2e60 con 0x7f1b780d0480 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7659c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 0x7f1b780d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b76d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 0x7f1b780b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7659c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 0x7f1b780d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:40570/0 (socket says 172.21.15.46:40570) 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b76d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 0x7f1b780b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:46292/0 (socket says 172.21.15.46:46292) 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7659c640 1 -- 172.21.15.46:0/2060310102 learned_addr learned my addr 172.21.15.46:0/2060310102 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b7759e640 1 --1- 172.21.15.46:0/2060310102 >> v1:172.21.15.88:6789/0 conn(0x7f1b780ce6a0 0x7f1b780ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41132/0 (socket says 172.21.15.46:41132) 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b75d9b640 1 -- 172.21.15.46:0/2060310102 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234266724 0 0) 0x7f1b70001110 con 0x7f1b780ce6a0 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.401+0000 7f1b75d9b640 1 -- 172.21.15.46:0/2060310102 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b64003520 con 0x7f1b780ce6a0 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b7659c640 1 -- 172.21.15.46:0/2060310102 >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 msgr2=0x7f1b780b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b7659c640 1 --2- 172.21.15.46:0/2060310102 >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 0x7f1b780b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b7659c640 1 -- 172.21.15.46:0/2060310102 >> v1:172.21.15.88:6789/0 conn(0x7f1b780ce6a0 legacy=0x7f1b780ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b7659c640 1 -- 172.21.15.46:0/2060310102 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b780d3140 con 0x7f1b780d0480 2024-05-09T03:15:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b76d9d640 1 --2- 172.21.15.46:0/2060310102 >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 0x7f1b780b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b7659c640 1 --2- 172.21.15.46:0/2060310102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 0x7f1b780d0890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009870 tx=0x7f1b6002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f71caebb9bb8444f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b75d9b640 1 -- 172.21.15.46:0/2060310102 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b60037070 con 0x7f1b780d0480 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.402+0000 7f1b75d9b640 1 -- 172.21.15.46:0/2060310102 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b60032440 con 0x7f1b780d0480 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b75d9b640 1 -- 172.21.15.46:0/2060310102 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b600327f0 con 0x7f1b780d0480 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/2060310102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 msgr2=0x7f1b780d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/2060310102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 0x7f1b780d0890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009870 tx=0x7f1b6002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/2060310102 shutdown_connections 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/2060310102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780d0480 0x7f1b780d0890 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/2060310102 >> v2:172.21.15.88:3300/0 conn(0x7f1b780b92e0 0x7f1b780b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/2060310102 >> 172.21.15.46:0/2060310102 conn(0x7f1b780d3c00 msgr2=0x7f1b780d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/2060310102 shutdown_connections 2024-05-09T03:15:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.403+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/2060310102 wait complete. 2024-05-09T03:15:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.405+0000 7f1b7d0ee640 1 Processor -- start 2024-05-09T03:15:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.405+0000 7f1b7d0ee640 1 -- start start 2024-05-09T03:15:37.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7d0ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780b92e0 0x7f1b78156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7d0ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b780ce6a0 0x7f1b781625f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7d0ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 0x7f1b780cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7d0ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7806e920 con 0x7f1b780d0480 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7d0ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7806e7a0 con 0x7f1b780ce6a0 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7d0ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7806eaa0 con 0x7f1b780b92e0 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 0x7f1b780cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 0x7f1b780cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57192/0 (socket says 172.21.15.46:57192) 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 -- 172.21.15.46:0/379193660 learned_addr learned my addr 172.21.15.46:0/379193660 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7659c640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b780ce6a0 0x7f1b781625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b76d9d640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780b92e0 0x7f1b78156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 -- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780b92e0 msgr2=0x7f1b78156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780b92e0 0x7f1b78156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 -- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b780ce6a0 msgr2=0x7f1b781625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b780ce6a0 0x7f1b781625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 -- 172.21.15.46:0/379193660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b78158ec0 con 0x7f1b780d0480 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b7759e640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 0x7f1b780cd1d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000c730 tx=0x7f1b7000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.407+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70007070 con 0x7f1b780d0480 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.408+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b70007210 con 0x7f1b780d0480 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.408+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70007600 con 0x7f1b780d0480 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.408+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b781593a0 con 0x7f1b780d0480 2024-05-09T03:15:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.408+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b78159960 con 0x7f1b780d0480 2024-05-09T03:15:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.409+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b78159110 con 0x7f1b780d0480 2024-05-09T03:15:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.409+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1b7001a020 con 0x7f1b780d0480 2024-05-09T03:15:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.410+0000 7f1b6f7fe640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b58076300 0x7f1b58078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.410+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f1b70096be0 con 0x7f1b780d0480 2024-05-09T03:15:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.410+0000 7f1b76d9d640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b58076300 0x7f1b58078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.411+0000 7f1b76d9d640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b58076300 0x7f1b58078810 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009bd0 tx=0x7f1b6800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:37.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.415+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1b78158ec0 con 0x7f1b780d0480 2024-05-09T03:15:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.641+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b7806a270 con 0x7f1b780d0480 2024-05-09T03:15:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.642+0000 7f1b6f7fe640 1 -- 172.21.15.46:0/379193660 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5185 (secure 0 0 0) 0x7f1b780d0dd0 con 0x7f1b780d0480 2024-05-09T03:15:37.644 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:37.644 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:34.464748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":14,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5977,"qdb_cluster":[5977]},"id":5}]} 2024-05-09T03:15:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b58076300 msgr2=0x7f1b58078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b58076300 0x7f1b58078810 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009bd0 tx=0x7f1b6800b040 comp rx=0 tx=0).stop 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 msgr2=0x7f1b780cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 0x7f1b780cd1d0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000c730 tx=0x7f1b7000cbf0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 shutdown_connections 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b58076300 0x7f1b58078810 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b780d0480 0x7f1b780cd1d0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b780ce6a0 0x7f1b781625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 --2- 172.21.15.46:0/379193660 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1b780b92e0 0x7f1b78156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 >> 172.21.15.46:0/379193660 conn(0x7f1b780d3c00 msgr2=0x7f1b780b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 shutdown_connections 2024-05-09T03:15:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.644+0000 7f1b7d0ee640 1 -- 172.21.15.46:0/379193660 wait complete. 2024-05-09T03:15:37.658 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.770+0000 7fbbbf52d640 1 Processor -- start 2024-05-09T03:15:37.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.771+0000 7fbbbf52d640 1 -- start start 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbf52d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb808cd60 0x7fbbb80d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbf52d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbbb80d1c80 0x7fbbb80cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbf52d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbb80d2090 0x7fbbb80cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbf52d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb80b5340 con 0x7fbbb80d2090 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbf52d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb80d4900 con 0x7fbbb80d1c80 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbf52d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb80b4fd0 con 0x7fbbb808cd60 2024-05-09T03:15:37.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbdaca640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbb80d2090 0x7fbbb80cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbdaca640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbbb80d2090 0x7fbbb80cf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57196/0 (socket says 172.21.15.46:57196) 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbdaca640 1 -- 172.21.15.46:0/2893894352 learned_addr learned my addr 172.21.15.46:0/2893894352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbcac8640 1 --2- 172.21.15.46:0/2893894352 >> v2:172.21.15.88:3300/0 conn(0x7fbbb80d1c80 0x7fbbb80cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.772+0000 7fbbbd2c9640 1 --2- 172.21.15.46:0/2893894352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb808cd60 0x7fbbb80d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbdaca640 1 -- 172.21.15.46:0/2893894352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb808cd60 msgr2=0x7fbbb80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbdaca640 1 --2- 172.21.15.46:0/2893894352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb808cd60 0x7fbbb80d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbdaca640 1 -- 172.21.15.46:0/2893894352 >> v2:172.21.15.88:3300/0 conn(0x7fbbb80d1c80 msgr2=0x7fbbb80cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbdaca640 1 --2- 172.21.15.46:0/2893894352 >> v2:172.21.15.88:3300/0 conn(0x7fbbb80d1c80 0x7fbbb80cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbdaca640 1 -- 172.21.15.46:0/2893894352 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb80d2c00 con 0x7fbbb80d2090 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbcac8640 1 --2- 172.21.15.46:0/2893894352 >> v2:172.21.15.88:3300/0 conn(0x7fbbb80d1c80 0x7fbbb80cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbbdaca640 1 --2- 172.21.15.46:0/2893894352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb80cf7f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8074850 tx=0x7fbbb002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f647bd01e9dc84 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbb7fff640 1 -- 172.21.15.46:0/2893894352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0037070 con 0x7fbbb80d2090 2024-05-09T03:15:37.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbb7fff640 1 -- 172.21.15.46:0/2893894352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb0032440 con 0x7fbbb80d2090 2024-05-09T03:15:37.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.773+0000 7fbbb7fff640 1 -- 172.21.15.46:0/2893894352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0032810 con 0x7fbbb80d2090 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 -- 172.21.15.46:0/2893894352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 msgr2=0x7fbbb80cf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/2893894352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb80cf7f0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8074850 tx=0x7fbbb002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 -- 172.21.15.46:0/2893894352 shutdown_connections 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/2893894352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb80cf7f0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/2893894352 >> v2:172.21.15.88:3300/0 conn(0x7fbbb80d1c80 0x7fbbb80cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/2893894352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb808cd60 0x7fbbb80d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 -- 172.21.15.46:0/2893894352 >> 172.21.15.46:0/2893894352 conn(0x7fbbb80d3c00 msgr2=0x7fbbb80d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 -- 172.21.15.46:0/2893894352 shutdown_connections 2024-05-09T03:15:37.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.774+0000 7fbbbf52d640 1 -- 172.21.15.46:0/2893894352 wait complete. 2024-05-09T03:15:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.776+0000 7fbbbf52d640 1 Processor -- start 2024-05-09T03:15:37.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.776+0000 7fbbbf52d640 1 -- start start 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.777+0000 7fbbbf52d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb808cd60 0x7fbbb80b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.777+0000 7fbbbf52d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb80d1c80 0x7fbbb81e96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.777+0000 7fbbbf52d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb81e9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.777+0000 7fbbbf52d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80b92b0 con 0x7fbbb80d2090 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.777+0000 7fbbbf52d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb80b9130 con 0x7fbbb808cd60 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.777+0000 7fbbbf52d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb8162310 con 0x7fbbb80d1c80 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb81e9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb81e9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57206/0 (socket says 172.21.15.46:57206) 2024-05-09T03:15:37.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 -- 172.21.15.46:0/4095824837 learned_addr learned my addr 172.21.15.46:0/4095824837 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbd2c9640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb808cd60 0x7fbbb80b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbcac8640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb80d1c80 0x7fbbb81e96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 -- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb80d1c80 msgr2=0x7fbbb81e96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb80d1c80 0x7fbbb81e96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 -- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb808cd60 msgr2=0x7fbbb80b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb808cd60 0x7fbbb80b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 -- 172.21.15.46:0/4095824837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb81f46d0 con 0x7fbbb80d2090 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbcac8640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb80d1c80 0x7fbbb81e96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbdaca640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb81e9c00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fbbb002a220 tx=0x7fbbb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0041020 con 0x7fbbb80d2090 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb003d640 con 0x7fbbb80d2090 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb80b1b60 con 0x7fbbb80d2090 2024-05-09T03:15:37.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.778+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0037040 con 0x7fbbb80d2090 2024-05-09T03:15:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.779+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb80b2120 con 0x7fbbb80d2090 2024-05-09T03:15:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.779+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb806a270 con 0x7fbbb80d2090 2024-05-09T03:15:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.780+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbbb0044020 con 0x7fbbb80d2090 2024-05-09T03:15:37.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.781+0000 7fbbb5ffb640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:37.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.781+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7fbbb00b7e10 con 0x7fbbb80d2090 2024-05-09T03:15:37.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.781+0000 7fbbbd2c9640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:37.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.782+0000 7fbbbd2c9640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8074930 tx=0x7fbba800d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:37.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:37.785+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbbb00bb030 con 0x7fbbb80d2090 2024-05-09T03:15:38.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.013+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbb80b8390 con 0x7fbbb80d2090 2024-05-09T03:15:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.014+0000 7fbbb5ffb640 1 -- 172.21.15.46:0/4095824837 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5185 (secure 0 0 0) 0x7fbbb004b030 con 0x7fbbb80d2090 2024-05-09T03:15:38.016 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:38.016 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:34.464748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":14,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5977,"qdb_cluster":[5977]},"id":5}]} 2024-05-09T03:15:38.016 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2024-05-09T03:15:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 msgr2=0x7fbba4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8074930 tx=0x7fbba800d040 comp rx=0 tx=0).stop 2024-05-09T03:15:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 msgr2=0x7fbbb81e9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb81e9c00 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fbbb002a220 tx=0x7fbbb00046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 shutdown_connections 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbba4076300 0x7fbba4078810 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbb80d2090 0x7fbbb81e9c00 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbbb80d1c80 0x7fbbb81e96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 --2- 172.21.15.46:0/4095824837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbb808cd60 0x7fbbb80b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 >> 172.21.15.46:0/4095824837 conn(0x7fbbb80d3c00 msgr2=0x7fbbb80b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 shutdown_connections 2024-05-09T03:15:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.016+0000 7fbbbf52d640 1 -- 172.21.15.46:0/4095824837 wait complete. 2024-05-09T03:15:38.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:15:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.142+0000 7f2ba723b640 1 Processor -- start 2024-05-09T03:15:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.142+0000 7f2ba723b640 1 -- start start 2024-05-09T03:15:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.143+0000 7f2ba723b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba01613c0 0x7f2ba00b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba723b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 0x7f2ba0163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba723b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba00b8110 con 0x7f2ba00b5210 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba723b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2ba00b5620 con 0x7f2ba01613c0 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba723b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba01609a0 con 0x7f2ba00b7d00 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba4fd7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ba00b5210 0x7f2ba00d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60148/0 (socket says 172.21.15.46:60148) 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba4fd7640 1 -- 172.21.15.46:0/3748126818 learned_addr learned my addr 172.21.15.46:0/3748126818 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9ffff640 1 --2- 172.21.15.46:0/3748126818 >> v2:172.21.15.88:3300/0 conn(0x7f2ba01613c0 0x7f2ba00b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3577361585 0 0) 0x7f2b90001110 con 0x7f2ba00b5210 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2ba57d8640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 0x7f2ba0163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b88003520 con 0x7f2ba00b5210 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2800063615 0 0) 0x7f2b88003520 con 0x7f2ba00b5210 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 msgr2=0x7f2ba0163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 0x7f2ba0163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 >> v2:172.21.15.88:3300/0 conn(0x7f2ba01613c0 msgr2=0x7f2ba00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 --2- 172.21.15.46:0/3748126818 >> v2:172.21.15.88:3300/0 conn(0x7f2ba01613c0 0x7f2ba00b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba01617d0 con 0x7f2ba00b5210 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f2b900018c0 con 0x7f2ba00b5210 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 >> v1:172.21.15.46:6789/0 conn(0x7f2ba00b5210 legacy=0x7f2ba00d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b88003a00 0x7f2b88003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 0x7f2b88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b88004d40 0x7f2b88005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba00b5620 con 0x7f2b88004d40 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba01609a0 con 0x7f2b88003a00 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.144+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2b8800ba70 con 0x7f2ba00b7d00 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba57d8640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b88004d40 0x7f2b88005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba4fd7640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b88003a00 0x7f2b88003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2b9ffff640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 0x7f2b88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba57d8640 1 -- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 msgr2=0x7f2b88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:38.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba57d8640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ba00b7d00 0x7f2b88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba57d8640 1 -- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b88003a00 msgr2=0x7f2b88003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba57d8640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b88003a00 0x7f2b88003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.145+0000 7f2ba57d8640 1 -- 172.21.15.46:0/3748126818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b88003520 con 0x7f2b88004d40 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2ba57d8640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b88004d40 0x7f2b88005150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009f90 tx=0x7f2b88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57aede91d4726cbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98009c10 con 0x7f2b88004d40 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b98036070 con 0x7f2b88004d40 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2b9f7fe640 1 -- 172.21.15.46:0/3748126818 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98031850 con 0x7f2b88004d40 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2ba723b640 1 -- 172.21.15.46:0/3748126818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b88004d40 msgr2=0x7f2b88005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2ba723b640 1 --2- 172.21.15.46:0/3748126818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b88004d40 0x7f2b88005150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009f90 tx=0x7f2b88005830 comp rx=0 tx=0).stop 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2ba57d8640 1 -- 172.21.15.46:0/3748126818 reap_dead start 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2ba723b640 1 -- 172.21.15.46:0/3748126818 shutdown_connections 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.146+0000 7f2ba723b640 1 -- 172.21.15.46:0/3748126818 >> 172.21.15.46:0/3748126818 conn(0x7f2ba00d29a0 msgr2=0x7f2ba00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.147+0000 7f2ba723b640 1 -- 172.21.15.46:0/3748126818 shutdown_connections 2024-05-09T03:15:38.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.147+0000 7f2ba723b640 1 -- 172.21.15.46:0/3748126818 wait complete. 2024-05-09T03:15:38.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.148+0000 7f2ba723b640 1 Processor -- start 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.148+0000 7f2ba723b640 1 -- start start 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba723b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b88004d40 0x7f2ba0156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba723b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 0x7f2ba00b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba723b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba00cd090 0x7f2ba00cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba723b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba0160bc0 con 0x7f2ba00b9870 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba723b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba00d5560 con 0x7f2ba00cd090 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba723b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba0160ea0 con 0x7f2b88004d40 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 0x7f2ba00b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 0x7f2ba00b9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57226/0 (socket says 172.21.15.46:57226) 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 -- 172.21.15.46:0/1831909880 learned_addr learned my addr 172.21.15.46:0/1831909880 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba4fd7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b88004d40 0x7f2ba0156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2ba57d8640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba00cd090 0x7f2ba00cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 -- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b88004d40 msgr2=0x7f2ba0156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b88004d40 0x7f2ba0156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 -- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba00cd090 msgr2=0x7f2ba00cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba00cd090 0x7f2ba00cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 -- 172.21.15.46:0/1831909880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba00ba1c0 con 0x7f2ba00b9870 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.150+0000 7f2b9ffff640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 0x7f2ba00b9c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2b94011a70 tx=0x7f2b94011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.151+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400fb50 con 0x7f2ba00b9870 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.151+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b9400fcf0 con 0x7f2ba00b9870 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.151+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba01575c0 con 0x7f2ba00b9870 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.151+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400fea0 con 0x7f2ba00b9870 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.151+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba00ca920 con 0x7f2ba00b9870 2024-05-09T03:15:38.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.152+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba00ca3b0 con 0x7f2ba00b9870 2024-05-09T03:15:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.152+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2b94012040 con 0x7f2ba00b9870 2024-05-09T03:15:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.153+0000 7f2b9d7fa640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b8c0762d0 0x7f2b8c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.153+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f2b94097c70 con 0x7f2ba00b9870 2024-05-09T03:15:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.153+0000 7f2ba4fd7640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b8c0762d0 0x7f2b8c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.154+0000 7f2ba4fd7640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b8c0762d0 0x7f2b8c0787e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2b90001330 tx=0x7f2b90001e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.158+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2ba00ba1c0 con 0x7f2ba00b9870 2024-05-09T03:15:38.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.353+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ba006a270 con 0x7f2ba00b9870 2024-05-09T03:15:38.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.355+0000 7f2b9d7fa640 1 -- 172.21.15.46:0/1831909880 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16820 (secure 0 0 0) 0x7f2b94019030 con 0x7f2ba00b9870 2024-05-09T03:15:38.357 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:38.357 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:15:31.456404+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:14:50.331114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-09T03:14:51.590716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.17","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/131652171":"2024-05-10T03:15:31.456349+0000","172.21.15.46:6832/131652171":"2024-05-10T03:15:31.456349+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b8c0762d0 msgr2=0x7f2b8c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b8c0762d0 0x7f2b8c0787e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2b90001330 tx=0x7f2b90001e70 comp rx=0 tx=0).stop 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 msgr2=0x7f2ba00b9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 0x7f2ba00b9c80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f2b94011a70 tx=0x7f2b94011f30 comp rx=0 tx=0).stop 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 shutdown_connections 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba00cd090 0x7f2ba00cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b8c0762d0 0x7f2b8c0787e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ba00b9870 0x7f2ba00b9c80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 --2- 172.21.15.46:0/1831909880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b88004d40 0x7f2ba0156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 >> 172.21.15.46:0/1831909880 conn(0x7f2ba00d29a0 msgr2=0x7f2ba015b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:38.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 shutdown_connections 2024-05-09T03:15:38.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.357+0000 7f2ba723b640 1 -- 172.21.15.46:0/1831909880 wait complete. 2024-05-09T03:15:38.370 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:15:38.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:38.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.484+0000 7f1f4163b640 1 Processor -- start 2024-05-09T03:15:38.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.484+0000 7f1f4163b640 1 -- start start 2024-05-09T03:15:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f4163b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f3c0b6ae0 0x7f1f3c0cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f4163b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1f3c0ce200 0x7f1f3c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f4163b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f3c0b2fc0 con 0x7f1f3c160690 2024-05-09T03:15:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f4163b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1f3c0b9600 con 0x7f1f3c0b6ae0 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f4163b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1f3c0b32a0 con 0x7f1f3c0ce200 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f3affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f3c0b6ae0 0x7f1f3c0cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f3a7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1f3c160690 0x7f1f3c160aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60154/0 (socket says 172.21.15.46:60154) 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.485+0000 7f1f3a7fc640 1 -- 172.21.15.46:0/3254197255 learned_addr learned my addr 172.21.15.46:0/3254197255 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3b7fe640 1 --2- 172.21.15.46:0/3254197255 >> v2:172.21.15.88:3300/0 conn(0x7f1f3c0ce200 0x7f1f3c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3affd640 1 -- 172.21.15.46:0/3254197255 >> v2:172.21.15.88:3300/0 conn(0x7f1f3c0ce200 msgr2=0x7f1f3c0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3affd640 1 --2- 172.21.15.46:0/3254197255 >> v2:172.21.15.88:3300/0 conn(0x7f1f3c0ce200 0x7f1f3c0ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3affd640 1 -- 172.21.15.46:0/3254197255 >> v1:172.21.15.46:6789/0 conn(0x7f1f3c160690 legacy=0x7f1f3c160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3affd640 1 -- 172.21.15.46:0/3254197255 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c0b8fb0 con 0x7f1f3c0b6ae0 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f39ffb640 1 -- 172.21.15.46:0/3254197255 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578470712 0 0) 0x7f1f3c0b2fc0 con 0x7f1f3c160690 2024-05-09T03:15:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3b7fe640 1 --2- 172.21.15.46:0/3254197255 >> v2:172.21.15.88:3300/0 conn(0x7f1f3c0ce200 0x7f1f3c0ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:38.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.486+0000 7f1f3affd640 1 --2- 172.21.15.46:0/3254197255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0cff80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c06d9a0 tx=0x7f1f2c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e48dba6a98c309 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f39ffb640 1 -- 172.21.15.46:0/3254197255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c037070 con 0x7f1f3c0b6ae0 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f39ffb640 1 -- 172.21.15.46:0/3254197255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f2c032440 con 0x7f1f3c0b6ae0 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f39ffb640 1 -- 172.21.15.46:0/3254197255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c032810 con 0x7f1f3c0b6ae0 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 -- 172.21.15.46:0/3254197255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c0b6ae0 msgr2=0x7f1f3c0cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 --2- 172.21.15.46:0/3254197255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0cff80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c06d9a0 tx=0x7f1f2c02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 -- 172.21.15.46:0/3254197255 shutdown_connections 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 --2- 172.21.15.46:0/3254197255 >> v2:172.21.15.88:3300/0 conn(0x7f1f3c0ce200 0x7f1f3c0ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 --2- 172.21.15.46:0/3254197255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0cff80 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 -- 172.21.15.46:0/3254197255 >> 172.21.15.46:0/3254197255 conn(0x7f1f3c0d0630 msgr2=0x7f1f3c0d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 -- 172.21.15.46:0/3254197255 shutdown_connections 2024-05-09T03:15:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.487+0000 7f1f4163b640 1 -- 172.21.15.46:0/3254197255 wait complete. 2024-05-09T03:15:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.489+0000 7f1f4163b640 1 Processor -- start 2024-05-09T03:15:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.489+0000 7f1f4163b640 1 -- start start 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0700a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f3c0ce200 0x7f1f3c151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 0x7f1f3c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c161330 con 0x7f1f3c160690 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3c1611b0 con 0x7f1f3c0ce200 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3c0ceb50 con 0x7f1f3c0b6ae0 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 0x7f1f3c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 0x7f1f3c0c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57248/0 (socket says 172.21.15.46:57248) 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 -- 172.21.15.46:0/179640736 learned_addr learned my addr 172.21.15.46:0/179640736 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3a7fc640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f3c0ce200 0x7f1f3c151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3affd640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 -- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f3c0b6ae0 msgr2=0x7f1f3c0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 -- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f3c0ce200 msgr2=0x7f1f3c151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f3c0ce200 0x7f1f3c151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 -- 172.21.15.46:0/179640736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f3c0ccba0 con 0x7f1f3c160690 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3affd640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f3b7fe640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 0x7f1f3c0c7fa0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400ccf0 tx=0x7f1f3400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400f680 con 0x7f1f3c160690 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f3400f840 con 0x7f1f3c160690 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.491+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3c06f2e0 con 0x7f1f3c160690 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.492+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400fc30 con 0x7f1f3c160690 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.492+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f3c06f8a0 con 0x7f1f3c160690 2024-05-09T03:15:38.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.492+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f3c06a270 con 0x7f1f3c160690 2024-05-09T03:15:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.493+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1f340213f0 con 0x7f1f3c160690 2024-05-09T03:15:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.494+0000 7f1f337fe640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f1c076300 0x7f1f1c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.494+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f1f34098580 con 0x7f1f3c160690 2024-05-09T03:15:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.494+0000 7f1f3affd640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f1c076300 0x7f1f1c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.495+0000 7f1f3affd640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f1c076300 0x7f1f1c078810 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c034000 tx=0x7f1f2c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.498+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1f34014030 con 0x7f1f3c160690 2024-05-09T03:15:38.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.728+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f3c0b9b30 con 0x7f1f3c160690 2024-05-09T03:15:38.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.728+0000 7f1f337fe640 1 -- 172.21.15.46:0/179640736 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5185 (secure 0 0 0) 0x7f1f34061f30 con 0x7f1f3c160690 2024-05-09T03:15:38.730 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:38.730 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:34.464748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":14,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5977,"qdb_cluster":[5977]},"id":5}]} 2024-05-09T03:15:38.731 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2024-05-09T03:15:38.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f1c076300 msgr2=0x7f1f1c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f1c076300 0x7f1f1c078810 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c034000 tx=0x7f1f2c02e000 comp rx=0 tx=0).stop 2024-05-09T03:15:38.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 msgr2=0x7f1f3c0c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 0x7f1f3c0c7fa0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400ccf0 tx=0x7f1f3400a650 comp rx=0 tx=0).stop 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 shutdown_connections 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f1c076300 0x7f1f1c078810 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f3c160690 0x7f1f3c0c7fa0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f3c0ce200 0x7f1f3c151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 --2- 172.21.15.46:0/179640736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f3c0b6ae0 0x7f1f3c0700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 >> 172.21.15.46:0/179640736 conn(0x7f1f3c0d0630 msgr2=0x7f1f3c162960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 shutdown_connections 2024-05-09T03:15:38.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.730+0000 7f1f4163b640 1 -- 172.21.15.46:0/179640736 wait complete. 2024-05-09T03:15:38.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:38.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.857+0000 7eff01b76640 1 Processor -- start 2024-05-09T03:15:38.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.857+0000 7eff01b76640 1 -- start start 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.858+0000 7eff01b76640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc1468d0 0x7efefc06c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.858+0000 7eff01b76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efefc0c8020 0x7efefc06cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.858+0000 7eff01b76640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efefc0c8590 con 0x7efefc0b1ed0 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.858+0000 7eff01b76640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efefc146440 con 0x7efefc0c8020 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.858+0000 7eff01b76640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efefc145df0 con 0x7efefc1468d0 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efefc0b1ed0 0x7efefc0c7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60162/0 (socket says 172.21.15.46:60162) 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefaffd640 1 -- 172.21.15.46:0/750918817 learned_addr learned my addr 172.21.15.46:0/750918817 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefb7fe640 1 --2- 172.21.15.46:0/750918817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc1468d0 0x7efefc06c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefbfff640 1 --2- 172.21.15.46:0/750918817 >> v2:172.21.15.88:3300/0 conn(0x7efefc0c8020 0x7efefc06cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefa7fc640 1 -- 172.21.15.46:0/750918817 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288008221 0 0) 0x7efefc0c8590 con 0x7efefc0b1ed0 2024-05-09T03:15:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefa7fc640 1 -- 172.21.15.46:0/750918817 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efee4003520 con 0x7efefc0b1ed0 2024-05-09T03:15:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefb7fe640 1 -- 172.21.15.46:0/750918817 >> v2:172.21.15.88:3300/0 conn(0x7efefc0c8020 msgr2=0x7efefc06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefb7fe640 1 --2- 172.21.15.46:0/750918817 >> v2:172.21.15.88:3300/0 conn(0x7efefc0c8020 0x7efefc06cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefb7fe640 1 -- 172.21.15.46:0/750918817 >> v1:172.21.15.46:6789/0 conn(0x7efefc0b1ed0 legacy=0x7efefc0c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.859+0000 7efefb7fe640 1 -- 172.21.15.46:0/750918817 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc1460d0 con 0x7efefc1468d0 2024-05-09T03:15:38.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.860+0000 7efefb7fe640 1 --2- 172.21.15.46:0/750918817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc1468d0 0x7efefc06c680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efeec0097a0 tx=0x7efeec02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f396d4f99899089e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.860+0000 7efefa7fc640 1 -- 172.21.15.46:0/750918817 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec037070 con 0x7efefc1468d0 2024-05-09T03:15:38.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.860+0000 7efefa7fc640 1 -- 172.21.15.46:0/750918817 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeec032440 con 0x7efefc1468d0 2024-05-09T03:15:38.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.860+0000 7efefa7fc640 1 -- 172.21.15.46:0/750918817 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec0327f0 con 0x7efefc1468d0 2024-05-09T03:15:38.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 -- 172.21.15.46:0/750918817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc1468d0 msgr2=0x7efefc06c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 --2- 172.21.15.46:0/750918817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc1468d0 0x7efefc06c680 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efeec0097a0 tx=0x7efeec02a180 comp rx=0 tx=0).stop 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 -- 172.21.15.46:0/750918817 shutdown_connections 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 --2- 172.21.15.46:0/750918817 >> v2:172.21.15.88:3300/0 conn(0x7efefc0c8020 0x7efefc06cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 --2- 172.21.15.46:0/750918817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc1468d0 0x7efefc06c680 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 -- 172.21.15.46:0/750918817 >> 172.21.15.46:0/750918817 conn(0x7efefc0b46a0 msgr2=0x7efefc0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 -- 172.21.15.46:0/750918817 shutdown_connections 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.861+0000 7eff01b76640 1 -- 172.21.15.46:0/750918817 wait complete. 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.863+0000 7eff01b76640 1 Processor -- start 2024-05-09T03:15:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.863+0000 7eff01b76640 1 -- start start 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7eff01b76640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc0b1ed0 0x7efefc1446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7eff01b76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 0x7efefc0cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7eff01b76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efefc0b8440 0x7efefc0c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7eff01b76640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1642d0 con 0x7efefc0c8020 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7eff01b76640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efefc164150 con 0x7efefc0b8440 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7eff01b76640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efefc164450 con 0x7efefc0b1ed0 2024-05-09T03:15:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7efefaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 0x7efefc0cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7efefaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 0x7efefc0cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57260/0 (socket says 172.21.15.46:57260) 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7efefaffd640 1 -- 172.21.15.46:0/1844686569 learned_addr learned my addr 172.21.15.46:0/1844686569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7efefb7fe640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc0b1ed0 0x7efefc1446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.864+0000 7efefbfff640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efefc0b8440 0x7efefc0c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efefaffd640 1 -- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc0b1ed0 msgr2=0x7efefc1446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efefaffd640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc0b1ed0 0x7efefc1446d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efefaffd640 1 -- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efefc0b8440 msgr2=0x7efefc0c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efefaffd640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efefc0b8440 0x7efefc0c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efefaffd640 1 -- 172.21.15.46:0/1844686569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc0cee80 con 0x7efefc0c8020 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efefaffd640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 0x7efefc0cff20 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efef000c440 tx=0x7efef000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0017070 con 0x7efefc0c8020 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef0005ce0 con 0x7efefc0c8020 2024-05-09T03:15:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0007270 con 0x7efefc0c8020 2024-05-09T03:15:38.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc0cf160 con 0x7efefc0c8020 2024-05-09T03:15:38.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.865+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc0cf720 con 0x7efefc0c8020 2024-05-09T03:15:38.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.866+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efefc0cebf0 con 0x7efefc0c8020 2024-05-09T03:15:38.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.867+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efef001a070 con 0x7efefc0c8020 2024-05-09T03:15:38.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.868+0000 7efee3fff640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efee80762d0 0x7efee80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:38.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.868+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7efef00968d0 con 0x7efefc0c8020 2024-05-09T03:15:38.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.868+0000 7efefb7fe640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efee80762d0 0x7efee80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:38.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.868+0000 7efefb7fe640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efee80762d0 0x7efee80787e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efeec034000 tx=0x7efeec02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:38.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:38.872+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efef005fab0 con 0x7efefc0c8020 2024-05-09T03:15:39.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.100+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efefc06a270 con 0x7efefc0c8020 2024-05-09T03:15:39.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.101+0000 7efee3fff640 1 -- 172.21.15.46:0/1844686569 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5185 (secure 0 0 0) 0x7efef0063db0 con 0x7efefc0c8020 2024-05-09T03:15:39.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:39.103 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":76,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:14:52.631122+0000","modified":"2024-05-09T03:15:34.464748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5977},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5977":{"gid":5977,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":14,"addr":"172.21.15.88:6836/3673458792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3673458792},{"type":"v1","addr":"172.21.15.88:6836","nonce":3673458792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5977,"qdb_cluster":[5977]},"id":5}]} 2024-05-09T03:15:39.103 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 76 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efee80762d0 msgr2=0x7efee80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efee80762d0 0x7efee80787e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efeec034000 tx=0x7efeec02e000 comp rx=0 tx=0).stop 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 msgr2=0x7efefc0cff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 0x7efefc0cff20 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efef000c440 tx=0x7efef000c900 comp rx=0 tx=0).stop 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 shutdown_connections 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efefc0b8440 0x7efefc0c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efee80762d0 0x7efee80787e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efefc0c8020 0x7efefc0cff20 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 --2- 172.21.15.46:0/1844686569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efefc0b1ed0 0x7efefc1446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 >> 172.21.15.46:0/1844686569 conn(0x7efefc0b46a0 msgr2=0x7efefc145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:39.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 shutdown_connections 2024-05-09T03:15:39.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.103+0000 7eff01b76640 1 -- 172.21.15.46:0/1844686569 wait complete. 2024-05-09T03:15:39.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:15:39.228 INFO:tasks.ceph.mds.d.smithi088.stderr:2024-05-09T03:15:39.227+0000 7f965bbd9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:15:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.229+0000 7fdd93e7e640 1 Processor -- start 2024-05-09T03:15:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.229+0000 7fdd93e7e640 1 -- start start 2024-05-09T03:15:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd93e7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd93e7e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd8c0d0480 0x7fdd8c0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd93e7e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd8c0ce6a0 0x7fdd8c0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd93e7e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdd8c0d0dd0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd93e7e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdd8c0d2b80 con 0x7fdd8c0d0480 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd93e7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d2e60 con 0x7fdd8c0b92e0 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd8c0ce6a0 0x7fdd8c0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd8c0ce6a0 0x7fdd8c0ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57272/0 (socket says 172.21.15.46:57272) 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 -- 172.21.15.46:0/1909467093 learned_addr learned my addr 172.21.15.46:0/1909467093 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd91419640 1 --2- 172.21.15.46:0/1909467093 >> v2:172.21.15.88:3300/0 conn(0x7fdd8c0d0480 0x7fdd8c0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd91c1a640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 -- 172.21.15.46:0/1909467093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 msgr2=0x7fdd8c0b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c0b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 -- 172.21.15.46:0/1909467093 >> v2:172.21.15.88:3300/0 conn(0x7fdd8c0d0480 msgr2=0x7fdd8c0d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 --2- 172.21.15.46:0/1909467093 >> v2:172.21.15.88:3300/0 conn(0x7fdd8c0d0480 0x7fdd8c0d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd9241b640 1 -- 172.21.15.46:0/1909467093 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c0d3140 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.231+0000 7fdd91c1a640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c0b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.232+0000 7fdd9241b640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0ceab0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c074850 tx=0x7fdd8402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda1f3ef30a7e808 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.232+0000 7fdd90c18640 1 -- 172.21.15.46:0/1909467093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84037070 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.232+0000 7fdd90c18640 1 -- 172.21.15.46:0/1909467093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd84032440 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.232+0000 7fdd90c18640 1 -- 172.21.15.46:0/1909467093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84032810 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1909467093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 msgr2=0x7fdd8c0ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0ceab0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c074850 tx=0x7fdd8402a140 comp rx=0 tx=0).stop 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1909467093 shutdown_connections 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0ceab0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1909467093 >> v2:172.21.15.88:3300/0 conn(0x7fdd8c0d0480 0x7fdd8c0d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1909467093 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c0b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1909467093 >> 172.21.15.46:0/1909467093 conn(0x7fdd8c0d3c00 msgr2=0x7fdd8c0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1909467093 shutdown_connections 2024-05-09T03:15:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.233+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1909467093 wait complete. 2024-05-09T03:15:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.235+0000 7fdd93e7e640 1 Processor -- start 2024-05-09T03:15:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.235+0000 7fdd93e7e640 1 -- start start 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd93e7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c14d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd93e7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd93e7e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd8c0d0480 0x7fdd8c0b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd93e7e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c0cf170 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd93e7e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c0ceff0 con 0x7fdd8c0d0480 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd93e7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c0cf2f0 con 0x7fdd8c0b92e0 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57282/0 (socket says 172.21.15.46:57282) 2024-05-09T03:15:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 -- 172.21.15.46:0/1587813403 learned_addr learned my addr 172.21.15.46:0/1587813403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91c1a640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c14d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd9241b640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd8c0d0480 0x7fdd8c0b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 -- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 msgr2=0x7fdd8c14d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c14d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 -- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd8c0d0480 msgr2=0x7fdd8c0b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd8c0d0480 0x7fdd8c0b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.236+0000 7fdd91419640 1 -- 172.21.15.46:0/1587813403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c0cdb30 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.237+0000 7fdd91419640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0cd5f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000d5c0 tx=0x7fdd8000da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.237+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd800125f0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.237+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd80019070 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.237+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8c15c500 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.237+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd800129d0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.237+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8c15cac0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.238+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8c06a270 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.239+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdd80012b70 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.239+0000 7fdd8a7fc640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd780762d0 0x7fdd780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.239+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7fdd800977a0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.239+0000 7fdd91c1a640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd780762d0 0x7fdd780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.240+0000 7fdd91c1a640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd780762d0 0x7fdd780787e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c071390 tx=0x7fdd7c007740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.244+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdd80060ab0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.436+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd8c15c220 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.437+0000 7fdd8a7fc640 1 -- 172.21.15.46:0/1587813403 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16820 (secure 0 0 0) 0x7fdd80064db0 con 0x7fdd8c0ce6a0 2024-05-09T03:15:39.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:39.440 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":71,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:15:31.456404+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:14:50.331114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-05-09T03:14:51.590716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.4","mappings":[{"from":6,"to":0}]},{"pgid":"11.17","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/131652171":"2024-05-10T03:15:31.456349+0000","172.21.15.46:6832/131652171":"2024-05-10T03:15:31.456349+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd780762d0 msgr2=0x7fdd780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd780762d0 0x7fdd780787e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c071390 tx=0x7fdd7c007740 comp rx=0 tx=0).stop 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 msgr2=0x7fdd8c0cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0cd5f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fdd8000d5c0 tx=0x7fdd8000da80 comp rx=0 tx=0).stop 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 shutdown_connections 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd8c0d0480 0x7fdd8c0b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd780762d0 0x7fdd780787e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd8c0ce6a0 0x7fdd8c0cd5f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 --2- 172.21.15.46:0/1587813403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd8c0b92e0 0x7fdd8c14d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 >> 172.21.15.46:0/1587813403 conn(0x7fdd8c0d3c00 msgr2=0x7fdd8c072d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 shutdown_connections 2024-05-09T03:15:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.440+0000 7fdd93e7e640 1 -- 172.21.15.46:0/1587813403 wait complete. 2024-05-09T03:15:39.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:15:39.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.567+0000 7fa0dd8a8640 1 Processor -- start 2024-05-09T03:15:39.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.567+0000 7fa0dd8a8640 1 -- start start 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.568+0000 7fa0dd8a8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0d80d04c0 0x7fa0d80d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.568+0000 7fa0dd8a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0d80d2410 0x7fa0d80d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.568+0000 7fa0dd8a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d80d0e10 con 0x7fa0d8162880 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.568+0000 7fa0dd8a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d815f340 con 0x7fa0d80d2410 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.568+0000 7fa0dd8a8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d8161da0 con 0x7fa0d80d04c0 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0d80d2410 0x7fa0d80d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0d80d2410 0x7fa0d80d2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57292/0 (socket says 172.21.15.46:57292) 2024-05-09T03:15:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 -- 172.21.15.46:0/3713314430 learned_addr learned my addr 172.21.15.46:0/3713314430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0cffff640 1 --2- 172.21.15.46:0/3713314430 >> v2:172.21.15.88:3300/0 conn(0x7fa0d80d04c0 0x7fa0d80d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 -- 172.21.15.46:0/3713314430 >> v2:172.21.15.88:3300/0 conn(0x7fa0d80d04c0 msgr2=0x7fa0d80d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 --2- 172.21.15.46:0/3713314430 >> v2:172.21.15.88:3300/0 conn(0x7fa0d80d04c0 0x7fa0d80d08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 -- 172.21.15.46:0/3713314430 >> v1:172.21.15.88:6789/0 conn(0x7fa0d8162880 legacy=0x7fa0d80b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.569+0000 7fa0d77fe640 1 -- 172.21.15.46:0/3713314430 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d8162080 con 0x7fa0d80d2410 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.570+0000 7fa0d77fe640 1 --2- 172.21.15.46:0/3713314430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d80d2410 0x7fa0d80d2820 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80b6be0 tx=0x7fa0d002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208f866f5b484dc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.570+0000 7fa0d67fc640 1 -- 172.21.15.46:0/3713314430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0036070 con 0x7fa0d80d2410 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.570+0000 7fa0d67fc640 1 -- 172.21.15.46:0/3713314430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d0031600 con 0x7fa0d80d2410 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.570+0000 7fa0d67fc640 1 -- 172.21.15.46:0/3713314430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00319d0 con 0x7fa0d80d2410 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.570+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/3713314430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d80d2410 msgr2=0x7fa0d80d2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.570+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/3713314430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d80d2410 0x7fa0d80d2820 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80b6be0 tx=0x7fa0d002a610 comp rx=0 tx=0).stop 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.571+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/3713314430 shutdown_connections 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.571+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/3713314430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d80d2410 0x7fa0d80d2820 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.571+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/3713314430 >> v2:172.21.15.88:3300/0 conn(0x7fa0d80d04c0 0x7fa0d80d08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.571+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/3713314430 >> 172.21.15.46:0/3713314430 conn(0x7fa0d80b55f0 msgr2=0x7fa0d80b5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:39.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.571+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/3713314430 shutdown_connections 2024-05-09T03:15:39.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.571+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/3713314430 wait complete. 2024-05-09T03:15:39.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.572+0000 7fa0dd8a8640 1 Processor -- start 2024-05-09T03:15:39.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.572+0000 7fa0dd8a8640 1 -- start start 2024-05-09T03:15:39.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0dd8a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d80d04c0 0x7fa0d80cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0dd8a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0d80d2410 0x7fa0d815d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0dd8a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 0x7fa0d81f52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0dd8a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d80d2ee0 con 0x7fa0d8162880 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0dd8a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d80d2d60 con 0x7fa0d80d04c0 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0dd8a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d80d3060 con 0x7fa0d80d2410 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 0x7fa0d81f52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 0x7fa0d81f52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57306/0 (socket says 172.21.15.46:57306) 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d80d04c0 0x7fa0d80cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 -- 172.21.15.46:0/2152592310 learned_addr learned my addr 172.21.15.46:0/2152592310 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0cffff640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0d80d2410 0x7fa0d815d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 -- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0d80d2410 msgr2=0x7fa0d815d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0d80d2410 0x7fa0d815d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 -- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d80d04c0 msgr2=0x7fa0d80cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d80d04c0 0x7fa0d80cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 -- 172.21.15.46:0/2152592310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d81f5830 con 0x7fa0d8162880 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.574+0000 7fa0d77fe640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 0x7fa0d81f52f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009f40 tx=0x7fa0d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.575+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0036050 con 0x7fa0d8162880 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.575+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d0042070 con 0x7fa0d8162880 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.575+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d80cb420 con 0x7fa0d8162880 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.575+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0009510 con 0x7fa0d8162880 2024-05-09T03:15:39.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.575+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d80cb9e0 con 0x7fa0d8162880 2024-05-09T03:15:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.576+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d80cb180 con 0x7fa0d8162880 2024-05-09T03:15:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.576+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa0d0007bf0 con 0x7fa0d8162880 2024-05-09T03:15:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.577+0000 7fa0d4ff9640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0b8076300 0x7fa0b8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.577+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7fa0d00b82b0 con 0x7fa0d8162880 2024-05-09T03:15:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.577+0000 7fa0d6ffd640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0b8076300 0x7fa0b8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.578+0000 7fa0d6ffd640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0b8076300 0x7fa0b8078810 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8001720 tx=0x7fa0c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.581+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa0d80cb180 con 0x7fa0d8162880 2024-05-09T03:15:39.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.811+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa0d815e9f0 con 0x7fa0d8162880 2024-05-09T03:15:39.813 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:15:39.812+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:15:39.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.813+0000 7fa0d4ff9640 1 -- 172.21.15.46:0/2152592310 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v77) ==== 161+0+0 (secure 0 0 0) 0x7fa0d81f5830 con 0x7fa0d8162880 2024-05-09T03:15:39.815 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0b8076300 msgr2=0x7fa0b8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0b8076300 0x7fa0b8078810 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8001720 tx=0x7fa0c8009040 comp rx=0 tx=0).stop 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 msgr2=0x7fa0d81f52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 0x7fa0d81f52f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009f40 tx=0x7fa0d0004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 shutdown_connections 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0b8076300 0x7fa0b8078810 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8162880 0x7fa0d81f52f0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0d80d2410 0x7fa0d815d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 --2- 172.21.15.46:0/2152592310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d80d04c0 0x7fa0d80cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 >> 172.21.15.46:0/2152592310 conn(0x7fa0d80b55f0 msgr2=0x7fa0d80b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 shutdown_connections 2024-05-09T03:15:39.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.816+0000 7fa0dd8a8640 1 -- 172.21.15.46:0/2152592310 wait complete. 2024-05-09T03:15:39.825 INFO:tasks.ceph.mds.d.smithi088.stderr: -40> 2024-05-09T03:15:39.227+0000 7f965bbd9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:15:39.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:15:39.876 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:15:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.944+0000 7f97a8f34640 1 Processor -- start 2024-05-09T03:15:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.944+0000 7f97a8f34640 1 -- start start 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a8f34640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f97a40b52b0 0x7f97a40b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a8f34640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a40b8090 con 0x7f97a4164bf0 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a8f34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97a40d17a0 con 0x7f97a40b7c80 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a8f34640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f97a4164760 con 0x7f97a40b52b0 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97a4164bf0 0x7f97a40d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60168/0 (socket says 172.21.15.46:60168) 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a259c640 1 -- 172.21.15.46:0/876151406 learned_addr learned my addr 172.21.15.46:0/876151406 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:39.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a1d9b640 1 --2- 172.21.15.46:0/876151406 >> v2:172.21.15.88:3300/0 conn(0x7f97a40b52b0 0x7f97a40b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910415237 0 0) 0x7f9794001490 con 0x7f97a4164bf0 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f978c003520 con 0x7f97a4164bf0 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2191939739 0 0) 0x7f97a40d17a0 con 0x7f97a40b7c80 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.946+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9794001490 con 0x7f97a40b7c80 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.947+0000 7f97a1d9b640 1 -- 172.21.15.46:0/876151406 >> v1:172.21.15.88:6789/0 conn(0x7f97a40b7c80 legacy=0x7f97a40b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.947+0000 7f97a1d9b640 1 -- 172.21.15.46:0/876151406 >> v1:172.21.15.46:6789/0 conn(0x7f97a4164bf0 legacy=0x7f97a40d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.947+0000 7f97a1d9b640 1 -- 172.21.15.46:0/876151406 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a40d14c0 con 0x7f97a40b52b0 2024-05-09T03:15:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.947+0000 7f97a1d9b640 1 --2- 172.21.15.46:0/876151406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a40b7740 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f97980097a0 tx=0x7f979802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccea8301bcd7059d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798037070 con 0x7f97a40b52b0 2024-05-09T03:15:39.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9798032440 con 0x7f97a40b52b0 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a159a640 1 -- 172.21.15.46:0/876151406 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798032810 con 0x7f97a40b52b0 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 -- 172.21.15.46:0/876151406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 msgr2=0x7f97a40b7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 --2- 172.21.15.46:0/876151406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a40b7740 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f97980097a0 tx=0x7f979802a230 comp rx=0 tx=0).stop 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 -- 172.21.15.46:0/876151406 shutdown_connections 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 --2- 172.21.15.46:0/876151406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a40b7740 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 -- 172.21.15.46:0/876151406 >> 172.21.15.46:0/876151406 conn(0x7f97a40d2920 msgr2=0x7f97a40d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 -- 172.21.15.46:0/876151406 shutdown_connections 2024-05-09T03:15:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.948+0000 7f97a8f34640 1 -- 172.21.15.46:0/876151406 wait complete. 2024-05-09T03:15:39.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.950+0000 7f97a8f34640 1 Processor -- start 2024-05-09T03:15:39.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.950+0000 7f97a8f34640 1 -- start start 2024-05-09T03:15:39.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a8f34640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a41f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a8f34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 0x7f97a415ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a8f34640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97a4164bf0 0x7f97a415df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a8f34640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40d1180 con 0x7f97a40b7c80 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a8f34640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f97a41643f0 con 0x7f97a40b52b0 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a8f34640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f97a406eb30 con 0x7f97a4164bf0 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 0x7f97a415ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 0x7f97a415ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57316/0 (socket says 172.21.15.46:57316) 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 -- 172.21.15.46:0/3482817917 learned_addr learned my addr 172.21.15.46:0/3482817917 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a259c640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a41f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a2d9d640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97a4164bf0 0x7f97a415df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 -- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97a4164bf0 msgr2=0x7f97a415df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97a4164bf0 0x7f97a415df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 -- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 msgr2=0x7f97a41f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a41f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 -- 172.21.15.46:0/3482817917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a41fa320 con 0x7f97a40b7c80 2024-05-09T03:15:39.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f97a1d9b640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 0x7f97a415ad90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9798035040 tx=0x7f97980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.952+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798041030 con 0x7f97a40b7c80 2024-05-09T03:15:39.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.953+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97a41fa600 con 0x7f97a40b7c80 2024-05-09T03:15:39.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.953+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979803d4e0 con 0x7f97a40b7c80 2024-05-09T03:15:39.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.953+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9798037040 con 0x7f97a40b7c80 2024-05-09T03:15:39.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.953+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41fabc0 con 0x7f97a40b7c80 2024-05-09T03:15:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.954+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a41fb000 con 0x7f97a40b7c80 2024-05-09T03:15:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.954+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9798044070 con 0x7f97a40b7c80 2024-05-09T03:15:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.955+0000 7f9792ffd640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9784076300 0x7f9784078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.955+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5939+0+0 (secure 0 0 0) 0x7f97980b7af0 con 0x7f97a40b7c80 2024-05-09T03:15:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.955+0000 7f97a259c640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9784076300 0x7f9784078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:39.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.956+0000 7f97a259c640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9784076300 0x7f9784078810 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9794005c00 tx=0x7f9794009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:39.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:39.960+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f979804b070 con 0x7f97a40b7c80 2024-05-09T03:15:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.191+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f97a406a270 con 0x7f97a40b7c80 2024-05-09T03:15:40.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.815+0000 7f9792ffd640 1 -- 172.21.15.46:0/3482817917 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v78) ==== 104+0+0 (secure 0 0 0) 0x7f9798080c00 con 0x7f97a40b7c80 2024-05-09T03:15:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9784076300 msgr2=0x7f9784078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9784076300 0x7f9784078810 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9794005c00 tx=0x7f9794009040 comp rx=0 tx=0).stop 2024-05-09T03:15:40.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 msgr2=0x7f97a415ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 0x7f97a415ad90 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9798035040 tx=0x7f97980046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 shutdown_connections 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f97a4164bf0 0x7f97a415df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9784076300 0x7f9784078810 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a40b7c80 0x7f97a415ad90 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 --2- 172.21.15.46:0/3482817917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97a40b52b0 0x7f97a41f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.817+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 >> 172.21.15.46:0/3482817917 conn(0x7f97a40d2920 msgr2=0x7f97a40b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.818+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 shutdown_connections 2024-05-09T03:15:40.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.818+0000 7f97a8f34640 1 -- 172.21.15.46:0/3482817917 wait complete. 2024-05-09T03:15:40.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:15:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.956+0000 7f4677508640 1 Processor -- start 2024-05-09T03:15:40.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.956+0000 7f4677508640 1 -- start start 2024-05-09T03:15:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.957+0000 7f4677508640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46700b5140 0x7f46700b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4677508640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f467006e420 0x7f46700b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4677508640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46700b8480 con 0x7f46700b8070 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4677508640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f46700b9b60 con 0x7f46700b5140 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4677508640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46700b5550 con 0x7f467006e420 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f46752a4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f46700b5140 0x7f46700b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4674aa3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f46700b8070 0x7f467006dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60184/0 (socket says 172.21.15.46:60184) 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4674aa3640 1 -- 172.21.15.46:0/45590492 learned_addr learned my addr 172.21.15.46:0/45590492 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4675aa5640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f467006e420 0x7f46700b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f46752a4640 1 -- 172.21.15.46:0/45590492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f467006e420 msgr2=0x7f46700b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f46752a4640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f467006e420 0x7f46700b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f46752a4640 1 -- 172.21.15.46:0/45590492 >> v1:172.21.15.46:6789/0 conn(0x7f46700b8070 legacy=0x7f467006dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f466ffff640 1 -- 172.21.15.46:0/45590492 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724639733 0 0) 0x7f4664001110 con 0x7f46700b8070 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f46752a4640 1 -- 172.21.15.46:0/45590492 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46700b9270 con 0x7f46700b5140 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.958+0000 7f4675aa5640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f467006e420 0x7f46700b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.959+0000 7f46752a4640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b5140 0x7f46700b7b30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4668009d90 tx=0x7f466802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe3d29de4a48997 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.959+0000 7f466ffff640 1 -- 172.21.15.46:0/45590492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668007740 con 0x7f46700b5140 2024-05-09T03:15:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.959+0000 7f466ffff640 1 -- 172.21.15.46:0/45590492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46680078e0 con 0x7f46700b5140 2024-05-09T03:15:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.959+0000 7f466ffff640 1 -- 172.21.15.46:0/45590492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4668007cb0 con 0x7f46700b5140 2024-05-09T03:15:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 -- 172.21.15.46:0/45590492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b5140 msgr2=0x7f46700b7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b5140 0x7f46700b7b30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4668009d90 tx=0x7f466802a660 comp rx=0 tx=0).stop 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 -- 172.21.15.46:0/45590492 shutdown_connections 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f467006e420 0x7f46700b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 --2- 172.21.15.46:0/45590492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b5140 0x7f46700b7b30 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 -- 172.21.15.46:0/45590492 >> 172.21.15.46:0/45590492 conn(0x7f46700d5e90 msgr2=0x7f46700d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 -- 172.21.15.46:0/45590492 shutdown_connections 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.960+0000 7f4677508640 1 -- 172.21.15.46:0/45590492 wait complete. 2024-05-09T03:15:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.962+0000 7f4677508640 1 Processor -- start 2024-05-09T03:15:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.962+0000 7f4677508640 1 -- start start 2024-05-09T03:15:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4677508640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f467006e420 0x7f467015f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4677508640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46700b5140 0x7f46700b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4677508640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 0x7f46701f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4677508640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46700b97f0 con 0x7f46700b8070 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4677508640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f46700b9510 con 0x7f467006e420 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4677508640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f46700ba530 con 0x7f46700b5140 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 0x7f46701f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 0x7f46701f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57336/0 (socket says 172.21.15.46:57336) 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 -- 172.21.15.46:0/3379754650 learned_addr learned my addr 172.21.15.46:0/3379754650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f46752a4640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f467006e420 0x7f467015f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4674aa3640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46700b5140 0x7f46700b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 -- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46700b5140 msgr2=0x7f46700b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46700b5140 0x7f46700b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 -- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f467006e420 msgr2=0x7f467015f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f467006e420 0x7f467015f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.963+0000 7f4675aa5640 1 -- 172.21.15.46:0/3379754650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46700acdd0 con 0x7f46700b8070 2024-05-09T03:15:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.964+0000 7f4675aa5640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 0x7f46701f6870 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f466000d5e0 tx=0x7f466000daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.964+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660018070 con 0x7f46700b8070 2024-05-09T03:15:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.964+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4660004030 con 0x7f46700b8070 2024-05-09T03:15:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.964+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46700ad0b0 con 0x7f46700b8070 2024-05-09T03:15:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.964+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4660004400 con 0x7f46700b8070 2024-05-09T03:15:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.964+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46700ad670 con 0x7f46700b8070 2024-05-09T03:15:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.965+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46700acb40 con 0x7f46700b8070 2024-05-09T03:15:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.966+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4660014620 con 0x7f46700b8070 2024-05-09T03:15:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.966+0000 7f466dffb640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f465c0762d0 0x7f465c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.966+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5939+0+0 (secure 0 0 0) 0x7f46600984a0 con 0x7f46700b8070 2024-05-09T03:15:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.967+0000 7f46752a4640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f465c0762d0 0x7f465c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.967+0000 7f46752a4640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f465c0762d0 0x7f465c0787e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4668034000 tx=0x7f4668033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:40.970+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4660061cf0 con 0x7f46700b8070 2024-05-09T03:15:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:41.161+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f467006a270 con 0x7f46700b8070 2024-05-09T03:15:42.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.001+0000 7f466dffb640 1 -- 172.21.15.46:0/3379754650 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) ==== 181+0+0 (secure 0 0 0) 0x7f46600615f0 con 0x7f46700b8070 2024-05-09T03:15:42.003 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:15:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f465c0762d0 msgr2=0x7f465c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f465c0762d0 0x7f465c0787e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4668034000 tx=0x7f4668033040 comp rx=0 tx=0).stop 2024-05-09T03:15:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 msgr2=0x7f46701f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 0x7f46701f6870 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f466000d5e0 tx=0x7f466000daa0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 shutdown_connections 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f465c0762d0 0x7f465c0787e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46700b8070 0x7f46701f6870 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f46700b5140 0x7f46700b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 --2- 172.21.15.46:0/3379754650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f467006e420 0x7f467015f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.005+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 >> 172.21.15.46:0/3379754650 conn(0x7f46700d5e90 msgr2=0x7f4670160eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.006+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 shutdown_connections 2024-05-09T03:15:42.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.006+0000 7f4677508640 1 -- 172.21.15.46:0/3379754650 wait complete. 2024-05-09T03:15:42.023 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:15:42.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.151+0000 7f48f9d4d640 1 Processor -- start 2024-05-09T03:15:42.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.151+0000 7f48f9d4d640 1 -- start start 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f9d4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f415cfe0 0x7f48f40cf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f9d4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f40cff60 con 0x7f48f40cfb50 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f9d4d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f40d2990 con 0x7f48f40b8250 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f9d4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48f40d2c70 con 0x7f48f415cfe0 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f48f40cfb50 0x7f48f40d0640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60196/0 (socket says 172.21.15.46:60196) 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f3fff640 1 -- 172.21.15.46:0/679293607 learned_addr learned my addr 172.21.15.46:0/679293607 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:42.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.152+0000 7f48f2ffd640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f415cfe0 0x7f48f40cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625641477 0 0) 0x7f48ec001490 con 0x7f48f40cfb50 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48dc003520 con 0x7f48f40cfb50 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3014825137 0 0) 0x7f48f40d2990 con 0x7f48f40b8250 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48ec001490 con 0x7f48f40b8250 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3386568623 0 0) 0x7f48dc003520 con 0x7f48f40cfb50 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f415cfe0 msgr2=0x7f48f40cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f415cfe0 0x7f48f40cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 >> v1:172.21.15.88:6789/0 conn(0x7f48f40b8250 legacy=0x7f48f406f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f40d2f50 con 0x7f48f40cfb50 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f48ec001c40 con 0x7f48f40cfb50 2024-05-09T03:15:42.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 >> v1:172.21.15.46:6789/0 conn(0x7f48f40cfb50 legacy=0x7f48f40d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc003c10 0x7f48dc004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48dc004800 0x7f48dc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc005210 0x7f48dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48f40d2c70 con 0x7f48dc005210 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f48dc00c190 con 0x7f48dc003c10 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48dc00c390 con 0x7f48dc004800 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f2ffd640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f415cfe0 0x7f48f40cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f3fff640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc005210 0x7f48dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f37fe640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc003c10 0x7f48dc004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f2ffd640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48dc004800 0x7f48dc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f37fe640 1 -- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48dc004800 msgr2=0x7f48dc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f37fe640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48dc004800 0x7f48dc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f37fe640 1 -- 172.21.15.46:0/679293607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc005210 msgr2=0x7f48dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:42.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f37fe640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc005210 0x7f48dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f37fe640 1 -- 172.21.15.46:0/679293607 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48dc003520 con 0x7f48dc003c10 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.153+0000 7f48f3fff640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc005210 0x7f48dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.154+0000 7f48f37fe640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc003c10 0x7f48dc004060 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f48e40018d0 tx=0x7f48dc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6111251ad4f2f4f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.154+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4001520 con 0x7f48dc003c10 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.154+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e402acb0 con 0x7f48dc003c10 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f27fc640 1 -- 172.21.15.46:0/679293607 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4005310 con 0x7f48dc003c10 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f9d4d640 1 -- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc003c10 msgr2=0x7f48dc004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/679293607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48dc003c10 0x7f48dc004060 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f48e40018d0 tx=0x7f48dc005830 comp rx=0 tx=0).stop 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f3fff640 1 -- 172.21.15.46:0/679293607 reap_dead start 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f9d4d640 1 -- 172.21.15.46:0/679293607 shutdown_connections 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f9d4d640 1 -- 172.21.15.46:0/679293607 >> 172.21.15.46:0/679293607 conn(0x7f48f409b7d0 msgr2=0x7f48f409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:42.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f9d4d640 1 -- 172.21.15.46:0/679293607 shutdown_connections 2024-05-09T03:15:42.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.155+0000 7f48f9d4d640 1 -- 172.21.15.46:0/679293607 wait complete. 2024-05-09T03:15:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.157+0000 7f48f9d4d640 1 Processor -- start 2024-05-09T03:15:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.157+0000 7f48f9d4d640 1 -- start start 2024-05-09T03:15:42.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.158+0000 7f48f9d4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 0x7f48f40b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.158+0000 7f48f9d4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f41f7760 0x7f48f415ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f9d4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48f41f7b70 0x7f48f40ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f9d4d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48f415d3f0 con 0x7f48dc003c10 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f9d4d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f48f40d3510 con 0x7f48f41f7b70 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f9d4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48f415d570 con 0x7f48f41f7760 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 0x7f48f40b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 0x7f48f40b1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57362/0 (socket says 172.21.15.46:57362) 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 -- 172.21.15.46:0/533927454 learned_addr learned my addr 172.21.15.46:0/533927454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f3fff640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48f41f7b70 0x7f48f40ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f2ffd640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f41f7760 0x7f48f415ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 -- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f41f7760 msgr2=0x7f48f415ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f41f7760 0x7f48f415ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 -- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48f41f7b70 msgr2=0x7f48f40ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:42.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48f41f7b70 0x7f48f40ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 -- 172.21.15.46:0/533927454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f40b03f0 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f3fff640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48f41f7b70 0x7f48f40ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48f37fe640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 0x7f48f40b1490 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f48e40018d0 tx=0x7f48e40012f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e402ae60 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.159+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e40075e0 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.160+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f40b06d0 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.160+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e4007a10 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.160+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f40b0c90 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.160+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f415cbf0 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.161+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f48e40069d0 con 0x7f48dc003c10 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.162+0000 7f48dbfff640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48e00762d0 0x7f48e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:42.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.162+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5501+0+0 (secure 0 0 0) 0x7f48e40b71b0 con 0x7f48dc003c10 2024-05-09T03:15:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.162+0000 7f48f2ffd640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48e00762d0 0x7f48e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:42.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.163+0000 7f48f2ffd640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48e00762d0 0x7f48e00787e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f48f40d0310 tx=0x7f48e8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:42.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.166+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f48f415cbf0 con 0x7f48dc003c10 2024-05-09T03:15:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:42.356+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f48f406a270 con 0x7f48dc003c10 2024-05-09T03:15:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.006+0000 7f48dbfff640 1 -- 172.21.15.46:0/533927454 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) ==== 169+0+0 (secure 0 0 0) 0x7f48e40805e0 con 0x7f48dc003c10 2024-05-09T03:15:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48e00762d0 msgr2=0x7f48e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48e00762d0 0x7f48e00787e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f48f40d0310 tx=0x7f48e8009190 comp rx=0 tx=0).stop 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 msgr2=0x7f48f40b1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 0x7f48f40b1490 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f48e40018d0 tx=0x7f48e40012f0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 shutdown_connections 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48f41f7b70 0x7f48f40ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48f41f7760 0x7f48f415ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f48e00762d0 0x7f48e00787e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 --2- 172.21.15.46:0/533927454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48dc003c10 0x7f48f40b1490 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 >> 172.21.15.46:0/533927454 conn(0x7f48f409b7d0 msgr2=0x7f48f415c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 shutdown_connections 2024-05-09T03:15:43.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.011+0000 7f48f9d4d640 1 -- 172.21.15.46:0/533927454 wait complete. 2024-05-09T03:15:43.028 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-05-09T03:15:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.149+0000 7f035b28b640 1 Processor -- start 2024-05-09T03:15:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.149+0000 7f035b28b640 1 -- start start 2024-05-09T03:15:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f035b28b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f03540d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f035b28b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0354164eb0 0x7f03540ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f035b28b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03540b90f0 con 0x7f03540d2af0 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f035b28b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f03540d0360 con 0x7f0354164eb0 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f035b28b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f03540b93a0 con 0x7f03540d21a0 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0359828640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0354164eb0 0x7f03540ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0358826640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f03540d2af0 0x7f03541647a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60208/0 (socket says 172.21.15.46:60208) 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0358826640 1 -- 172.21.15.46:0/1355520656 learned_addr learned my addr 172.21.15.46:0/1355520656 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0359828640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0354164eb0 0x7f03540ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57374/0 (socket says 172.21.15.46:57374) 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0359828640 1 -- 172.21.15.46:0/1355520656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 msgr2=0x7f03540d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0359828640 1 --2- 172.21.15.46:0/1355520656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f03540d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0359828640 1 -- 172.21.15.46:0/1355520656 >> v1:172.21.15.46:6789/0 conn(0x7f03540d2af0 legacy=0x7f03541647a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f0359828640 1 -- 172.21.15.46:0/1355520656 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03540d10b0 con 0x7f0354164eb0 2024-05-09T03:15:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.151+0000 7f034bfff640 1 -- 172.21.15.46:0/1355520656 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242432492 0 0) 0x7f03540b90f0 con 0x7f03540d2af0 2024-05-09T03:15:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.152+0000 7f0359828640 1 --2- 172.21.15.46:0/1355520656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f03540ce8b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bec6b684d4f4d11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.152+0000 7f034bfff640 1 -- 172.21.15.46:0/1355520656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c031600 con 0x7f0354164eb0 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.152+0000 7f034bfff640 1 -- 172.21.15.46:0/1355520656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c0317a0 con 0x7f0354164eb0 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.152+0000 7f034bfff640 1 -- 172.21.15.46:0/1355520656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c031b70 con 0x7f0354164eb0 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 -- 172.21.15.46:0/1355520656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 msgr2=0x7f03540ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 --2- 172.21.15.46:0/1355520656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f03540ce8b0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c02a5d0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 -- 172.21.15.46:0/1355520656 shutdown_connections 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 --2- 172.21.15.46:0/1355520656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f03540ce8b0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 --2- 172.21.15.46:0/1355520656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f03540d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 -- 172.21.15.46:0/1355520656 >> 172.21.15.46:0/1355520656 conn(0x7f03540d04f0 msgr2=0x7f03540d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 -- 172.21.15.46:0/1355520656 shutdown_connections 2024-05-09T03:15:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.153+0000 7f035b28b640 1 -- 172.21.15.46:0/1355520656 wait complete. 2024-05-09T03:15:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.155+0000 7f035b28b640 1 Processor -- start 2024-05-09T03:15:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.155+0000 7f035b28b640 1 -- start start 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f035b28b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f0354152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f035b28b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03540d2af0 0x7f0354155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f035b28b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f0354164030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f035b28b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f035406f330 con 0x7f0354164eb0 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f035b28b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f035406f1b0 con 0x7f03540d2af0 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f035b28b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f03540d1a70 con 0x7f03540d21a0 2024-05-09T03:15:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f0359828640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f0354164030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f0359828640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f0354164030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57380/0 (socket says 172.21.15.46:57380) 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.156+0000 7f0359828640 1 -- 172.21.15.46:0/1042111257 learned_addr learned my addr 172.21.15.46:0/1042111257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359027640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f0354152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0358826640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03540d2af0 0x7f0354155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359828640 1 -- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 msgr2=0x7f0354152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359828640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f0354152180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359828640 1 -- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03540d2af0 msgr2=0x7f0354155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359828640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03540d2af0 0x7f0354155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359828640 1 -- 172.21.15.46:0/1042111257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03541526c0 con 0x7f0354164eb0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0359828640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f0354164030 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c03f070 con 0x7f0354164eb0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c030b60 con 0x7f0354164eb0 2024-05-09T03:15:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03540c75a0 con 0x7f0354164eb0 2024-05-09T03:15:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c043410 con 0x7f0354164eb0 2024-05-09T03:15:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.157+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0354155540 con 0x7f0354164eb0 2024-05-09T03:15:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.158+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03541fa420 con 0x7f0354164eb0 2024-05-09T03:15:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.159+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f034c03b410 con 0x7f0354164eb0 2024-05-09T03:15:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.160+0000 7f0349ffb640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03380762d0 0x7f03380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.160+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f034c0b8760 con 0x7f0354164eb0 2024-05-09T03:15:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.160+0000 7f0359027640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03380762d0 0x7f03380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.161+0000 7f0359027640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03380762d0 0x7f03380787e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0350006fd0 tx=0x7f0350008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.165+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f03541526c0 con 0x7f0354164eb0 2024-05-09T03:15:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.364+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f0354155ac0 con 0x7f0354164eb0 2024-05-09T03:15:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.365+0000 7f0349ffb640 1 -- 172.21.15.46:0/1042111257 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f034c081d20 con 0x7f0354164eb0 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03380762d0 msgr2=0x7f03380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03380762d0 0x7f03380787e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0350006fd0 tx=0x7f0350008040 comp rx=0 tx=0).stop 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 msgr2=0x7f0354164030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f0354164030 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 shutdown_connections 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03380762d0 0x7f03380787e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0354164eb0 0x7f0354164030 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03540d2af0 0x7f0354155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 --2- 172.21.15.46:0/1042111257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03540d21a0 0x7f0354152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 >> 172.21.15.46:0/1042111257 conn(0x7f03540d04f0 msgr2=0x7f0354071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 shutdown_connections 2024-05-09T03:15:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.367+0000 7f035b28b640 1 -- 172.21.15.46:0/1042111257 wait complete. 2024-05-09T03:15:43.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T03:15:43.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.495+0000 7f7ce35dc640 1 Processor -- start 2024-05-09T03:15:43.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.495+0000 7f7ce35dc640 1 -- start start 2024-05-09T03:15:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.496+0000 7f7ce35dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7cdc0d47a0 0x7f7cdc15de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce35dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cdc0d10b0 con 0x7f7cdc0d0860 2024-05-09T03:15:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce35dc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cdc0d1390 con 0x7f7cdc0b7760 2024-05-09T03:15:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce35dc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7cdc1618d0 con 0x7f7cdc0d47a0 2024-05-09T03:15:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce0b77640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7cdc0d0860 0x7f7cdc1611c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60220/0 (socket says 172.21.15.46:60220) 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce0b77640 1 -- 172.21.15.46:0/2478811911 learned_addr learned my addr 172.21.15.46:0/2478811911 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785819410 0 0) 0x7f7cdc0d10b0 con 0x7f7cdc0d0860 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc8003520 con 0x7f7cdc0d0860 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce1378640 1 --2- 172.21.15.46:0/2478811911 >> v2:172.21.15.88:3300/0 conn(0x7f7cdc0d47a0 0x7f7cdc15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1944380725 0 0) 0x7f7cc8003520 con 0x7f7cdc0d0860 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 >> v2:172.21.15.88:3300/0 conn(0x7f7cdc0d47a0 msgr2=0x7f7cdc15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 --2- 172.21.15.46:0/2478811911 >> v2:172.21.15.88:3300/0 conn(0x7f7cdc0d47a0 0x7f7cdc15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 >> v1:172.21.15.88:6789/0 conn(0x7f7cdc0b7760 legacy=0x7f7cdc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc0d1670 con 0x7f7cdc0d0860 2024-05-09T03:15:43.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f7ccc0016f0 con 0x7f7cdc0d0860 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 >> v1:172.21.15.46:6789/0 conn(0x7f7cdc0d0860 legacy=0x7f7cdc1611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8003a50 0x7f7cc8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8004800 0x7f7cc8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cc8005350 0x7f7cc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1618d0 con 0x7f7cc8004800 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc800c280 con 0x7f7cc8005350 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc800c480 con 0x7f7cc8003a50 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.497+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8004800 0x7f7cc8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce1b79640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cc8005350 0x7f7cc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce1378640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8003a50 0x7f7cc8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce0b77640 1 -- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8003a50 msgr2=0x7f7cc8003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8003a50 0x7f7cc8003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce0b77640 1 -- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cc8005350 msgr2=0x7f7cc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cc8005350 0x7f7cc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce0b77640 1 -- 172.21.15.46:0/2478811911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc8003520 con 0x7f7cc8004800 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce1b79640 1 -- 172.21.15.46:0/2478811911 reap_dead start 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.498+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8004800 0x7f7cc8004c10 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c240 tx=0x7f7cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3052882bb38a53f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc007ce0 con 0x7f7cc8004800 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ccc005ad0 con 0x7f7cc8004800 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7cd3fff640 1 -- 172.21.15.46:0/2478811911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc007070 con 0x7f7cc8004800 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 -- 172.21.15.46:0/2478811911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8004800 msgr2=0x7f7cc8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8004800 0x7f7cc8004c10 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c240 tx=0x7f7cc8005830 comp rx=0 tx=0).stop 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 -- 172.21.15.46:0/2478811911 shutdown_connections 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/2478811911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc8004800 0x7f7cc8004c10 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 -- 172.21.15.46:0/2478811911 >> 172.21.15.46:0/2478811911 conn(0x7f7cdc0d2760 msgr2=0x7f7cdc0d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:43.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 -- 172.21.15.46:0/2478811911 shutdown_connections 2024-05-09T03:15:43.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.499+0000 7f7ce35dc640 1 -- 172.21.15.46:0/2478811911 wait complete. 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.501+0000 7f7ce35dc640 1 Processor -- start 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.501+0000 7f7ce35dc640 1 -- start start 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.502+0000 7f7ce35dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8004800 0x7f7cdc0cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.502+0000 7f7ce35dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 0x7f7cdc0cb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce35dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc15d480 0x7f7cdc0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce35dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc0d0df0 con 0x7f7cdc0b6390 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce35dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc0d0c70 con 0x7f7cdc15d480 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce35dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7cdc15de40 con 0x7f7cc8004800 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 0x7f7cdc0cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 0x7f7cdc0cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57400/0 (socket says 172.21.15.46:57400) 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 -- 172.21.15.46:0/249081928 learned_addr learned my addr 172.21.15.46:0/249081928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce1378640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8004800 0x7f7cdc0cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce1b79640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc15d480 0x7f7cdc0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 -- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8004800 msgr2=0x7f7cdc0cb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8004800 0x7f7cdc0cb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 -- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc15d480 msgr2=0x7f7cdc0c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc15d480 0x7f7cdc0c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 -- 172.21.15.46:0/249081928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc0ba820 con 0x7f7cdc0b6390 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce0b77640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 0x7f7cdc0cb670 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c240 tx=0x7f7cdc0d1560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc037040 con 0x7f7cdc0b6390 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ccc041070 con 0x7f7cdc0b6390 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc0ceab0 con 0x7f7cdc0b6390 2024-05-09T03:15:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.503+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0049e0 con 0x7f7cdc0b6390 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.504+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc0cf070 con 0x7f7cdc0b6390 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.504+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cdc06a270 con 0x7f7cdc0b6390 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.505+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7ccc004b80 con 0x7f7cdc0b6390 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.506+0000 7f7cd1ffb640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cc0076300 0x7f7cc0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.506+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f7ccc0b6bf0 con 0x7f7cdc0b6390 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.506+0000 7f7ce1378640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cc0076300 0x7f7cc0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.507+0000 7f7ce1378640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cc0076300 0x7f7cc0078810 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0d02e0 tx=0x7f7cd800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.510+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7ccc0016f0 con 0x7f7cdc0b6390 2024-05-09T03:15:43.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.703+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f7cdc076630 con 0x7f7cdc0b6390 2024-05-09T03:15:43.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.706+0000 7f7cd1ffb640 1 -- 172.21.15.46:0/249081928 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f7ccc0801a0 con 0x7f7cdc0b6390 2024-05-09T03:15:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cc0076300 msgr2=0x7f7cc0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cc0076300 0x7f7cc0078810 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0d02e0 tx=0x7f7cd800b040 comp rx=0 tx=0).stop 2024-05-09T03:15:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 msgr2=0x7f7cdc0cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 0x7f7cdc0cb670 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c240 tx=0x7f7cdc0d1560 comp rx=0 tx=0).stop 2024-05-09T03:15:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 shutdown_connections 2024-05-09T03:15:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cdc15d480 0x7f7cdc0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cc0076300 0x7f7cc0078810 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cdc0b6390 0x7f7cdc0cb670 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 --2- 172.21.15.46:0/249081928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7cc8004800 0x7f7cdc0cb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 >> 172.21.15.46:0/249081928 conn(0x7f7cdc0d2760 msgr2=0x7f7cdc0b6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 shutdown_connections 2024-05-09T03:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.709+0000 7f7ce35dc640 1 -- 172.21.15.46:0/249081928 wait complete. 2024-05-09T03:15:43.722 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-05-09T03:15:43.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.835+0000 7f129e594640 1 Processor -- start 2024-05-09T03:15:43.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.835+0000 7f129e594640 1 -- start start 2024-05-09T03:15:43.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129e594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 0x7f1298160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129e594640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12980b4af0 0x7f12980b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129e594640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12980d49e0 con 0x7f12980b77e0 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129e594640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f12980d5bb0 con 0x7f12980b4af0 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129e594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12980d4cc0 con 0x7f12980d57a0 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129cb31640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12980b77e0 0x7f12980b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60228/0 (socket says 172.21.15.46:60228) 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f129cb31640 1 -- 172.21.15.46:0/4233094891 learned_addr learned my addr 172.21.15.46:0/4233094891 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277080190 0 0) 0x7f12980d49e0 con 0x7f12980b77e0 2024-05-09T03:15:43.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f12977fe640 1 --2- 172.21.15.46:0/4233094891 >> v2:172.21.15.88:3300/0 conn(0x7f12980b4af0 0x7f12980b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f1297fff640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 0x7f1298160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.837+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1280003520 con 0x7f12980b77e0 2024-05-09T03:15:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2038855814 0 0) 0x7f1280003520 con 0x7f12980b77e0 2024-05-09T03:15:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 msgr2=0x7f1298160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 0x7f1298160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 >> v2:172.21.15.88:3300/0 conn(0x7f12980b4af0 msgr2=0x7f12980b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 --2- 172.21.15.46:0/4233094891 >> v2:172.21.15.88:3300/0 conn(0x7f12980b4af0 0x7f12980b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12980d4fa0 con 0x7f12980b77e0 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f1290001a70 con 0x7f12980b77e0 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 >> v1:172.21.15.46:6789/0 conn(0x7f12980b77e0 legacy=0x7f12980b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f1280003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 0x7f1280004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1280004d40 0x7f1280005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12980d5bb0 con 0x7f1280003a00 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f12980d4cc0 con 0x7f1280004d40 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f128000ba70 con 0x7f12980d57a0 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1297fff640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f1280003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f129cb31640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1280004d40 0x7f1280005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f12977fe640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 0x7f1280004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1297fff640 1 -- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 msgr2=0x7f1280004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1297fff640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12980d57a0 0x7f1280004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1297fff640 1 -- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1280004d40 msgr2=0x7f1280005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1297fff640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1280004d40 0x7f1280005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.838+0000 7f1297fff640 1 -- 172.21.15.46:0/4233094891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1280003520 con 0x7f1280003a00 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.839+0000 7f1297fff640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f1280003e70 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1288009a90 tx=0x7f1280005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cd648889d2447d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.839+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288036070 con 0x7f1280003a00 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.839+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1288031400 con 0x7f1280003a00 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.839+0000 7f1296ffd640 1 -- 172.21.15.46:0/4233094891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288009690 con 0x7f1280003a00 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129e594640 1 -- 172.21.15.46:0/4233094891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 msgr2=0x7f1280003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129e594640 1 --2- 172.21.15.46:0/4233094891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f1280003e70 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1288009a90 tx=0x7f1280005830 comp rx=0 tx=0).stop 2024-05-09T03:15:43.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129cb31640 1 -- 172.21.15.46:0/4233094891 reap_dead start 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129e594640 1 -- 172.21.15.46:0/4233094891 shutdown_connections 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129e594640 1 -- 172.21.15.46:0/4233094891 >> 172.21.15.46:0/4233094891 conn(0x7f12980743e0 msgr2=0x7f12980d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129e594640 1 -- 172.21.15.46:0/4233094891 shutdown_connections 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.840+0000 7f129e594640 1 -- 172.21.15.46:0/4233094891 wait complete. 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.842+0000 7f129e594640 1 Processor -- start 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.842+0000 7f129e594640 1 -- start start 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129e594640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f12981fa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129e594640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1298150e60 0x7f12981f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129e594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f129815db00 0x7f129815df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129e594640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12980d1920 con 0x7f1280003a00 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129e594640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f12980d15b0 con 0x7f129815db00 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129e594640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f129814ade0 con 0x7f1298150e60 2024-05-09T03:15:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f1297fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f12981fa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f1297fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f12981fa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57410/0 (socket says 172.21.15.46:57410) 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f1297fff640 1 -- 172.21.15.46:0/3617533963 learned_addr learned my addr 172.21.15.46:0/3617533963 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f12977fe640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1298150e60 0x7f12981f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f129cb31640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f129815db00 0x7f129815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f1297fff640 1 -- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1298150e60 msgr2=0x7f12981f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.843+0000 7f1297fff640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1298150e60 0x7f12981f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1297fff640 1 -- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f129815db00 msgr2=0x7f129815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1297fff640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f129815db00 0x7f129815df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1297fff640 1 -- 172.21.15.46:0/3617533963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129815e450 con 0x7f1280003a00 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f12977fe640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1298150e60 0x7f12981f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f129cb31640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f129815db00 0x7f129815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1297fff640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f12981fa3f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1288009bb0 tx=0x7f1288004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288040070 con 0x7f1280003a00 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f128803b460 con 0x7f1280003a00 2024-05-09T03:15:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129814ab90 con 0x7f1280003a00 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288036040 con 0x7f1280003a00 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.844+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12981f68e0 con 0x7f1280003a00 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.845+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f129806a270 con 0x7f1280003a00 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.846+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1288043070 con 0x7f1280003a00 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.846+0000 7f1294ff9640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12840762d0 0x7f12840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.847+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f12880b7bc0 con 0x7f1280003a00 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.847+0000 7f12977fe640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12840762d0 0x7f12840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.847+0000 7f12977fe640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12840762d0 0x7f12840787e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f129814dce0 tx=0x7f128c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:43.851+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f129815e450 con 0x7f1280003a00 2024-05-09T03:15:44.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:44.118+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) -- 0x7f12980b6af0 con 0x7f1280003a00 2024-05-09T03:15:45.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.012+0000 7f1294ff9640 1 -- 172.21.15.46:0/3617533963 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v296) ==== 141+0+0 (secure 0 0 0) 0x7f1288081200 con 0x7f1280003a00 2024-05-09T03:15:45.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.015+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12840762d0 msgr2=0x7f12840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.015+0000 7f129e594640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12840762d0 0x7f12840787e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f129814dce0 tx=0x7f128c008040 comp rx=0 tx=0).stop 2024-05-09T03:15:45.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.015+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 msgr2=0x7f12981fa3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.015+0000 7f129e594640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f12981fa3f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1288009bb0 tx=0x7f1288004730 comp rx=0 tx=0).stop 2024-05-09T03:15:45.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 shutdown_connections 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f129815db00 0x7f129815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1298150e60 0x7f12981f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12840762d0 0x7f12840787e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 --2- 172.21.15.46:0/3617533963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1280003a00 0x7f12981fa3f0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 >> 172.21.15.46:0/3617533963 conn(0x7f12980743e0 msgr2=0x7f12980b5a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 shutdown_connections 2024-05-09T03:15:45.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.016+0000 7f129e594640 1 -- 172.21.15.46:0/3617533963 wait complete. 2024-05-09T03:15:45.032 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-05-09T03:15:45.032 INFO:tasks.cephfs_test_runner:Starting test: test_object_deletion (tasks.cephfs.test_damage.TestDamage) That the MDS has a clean 'damaged' response to loss of any single metadata object 2024-05-09T03:15:45.032 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-05-09T03:15:45.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:15:45.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.152+0000 7f2f4fca8640 1 Processor -- start 2024-05-09T03:15:45.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.152+0000 7f2f4fca8640 1 -- start start 2024-05-09T03:15:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4fca8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f481613c0 0x7f2f480b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4fca8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f480b8110 con 0x7f2f480b5210 2024-05-09T03:15:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4fca8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f481617d0 con 0x7f2f480b7d00 2024-05-09T03:15:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4fca8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2f480b5620 con 0x7f2f481613c0 2024-05-09T03:15:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4da44640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2f480b5210 0x7f2f480d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60234/0 (socket says 172.21.15.46:60234) 2024-05-09T03:15:45.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4da44640 1 -- 172.21.15.46:0/2662084065 learned_addr learned my addr 172.21.15.46:0/2662084065 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4d243640 1 --2- 172.21.15.46:0/2662084065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f481613c0 0x7f2f480b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292690545 0 0) 0x7f2f44001110 con 0x7f2f480b5210 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f34003520 con 0x7f2f480b5210 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218113391 0 0) 0x7f2f481617d0 con 0x7f2f480b7d00 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.154+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f44001110 con 0x7f2f480b7d00 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.155+0000 7f2f4d243640 1 -- 172.21.15.46:0/2662084065 >> v1:172.21.15.88:6789/0 conn(0x7f2f480b7d00 legacy=0x7f2f48163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.155+0000 7f2f4d243640 1 -- 172.21.15.46:0/2662084065 >> v1:172.21.15.46:6789/0 conn(0x7f2f480b5210 legacy=0x7f2f480d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.155+0000 7f2f4d243640 1 -- 172.21.15.46:0/2662084065 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f480d5280 con 0x7f2f481613c0 2024-05-09T03:15:45.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.155+0000 7f2f4d243640 1 --2- 172.21.15.46:0/2662084065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f481613c0 0x7f2f480b77c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2f380097d0 tx=0x7f2f3802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad027336c3acefe8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38037070 con 0x7f2f481613c0 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f38032440 con 0x7f2f481613c0 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4ca42640 1 -- 172.21.15.46:0/2662084065 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38032810 con 0x7f2f481613c0 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 -- 172.21.15.46:0/2662084065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f481613c0 msgr2=0x7f2f480b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/2662084065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f481613c0 0x7f2f480b77c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2f380097d0 tx=0x7f2f3802a210 comp rx=0 tx=0).stop 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 -- 172.21.15.46:0/2662084065 shutdown_connections 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/2662084065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f481613c0 0x7f2f480b77c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 -- 172.21.15.46:0/2662084065 >> 172.21.15.46:0/2662084065 conn(0x7f2f480d29a0 msgr2=0x7f2f480d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 -- 172.21.15.46:0/2662084065 shutdown_connections 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.156+0000 7f2f4fca8640 1 -- 172.21.15.46:0/2662084065 wait complete. 2024-05-09T03:15:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.158+0000 7f2f4fca8640 1 Processor -- start 2024-05-09T03:15:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.158+0000 7f2f4fca8640 1 -- start start 2024-05-09T03:15:45.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.159+0000 7f2f4fca8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 0x7f2f480cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.159+0000 7f2f4fca8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b7d00 0x7f2f480cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.159+0000 7f2f4fca8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f481613c0 0x7f2f4815d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.159+0000 7f2f4fca8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48160bc0 con 0x7f2f480b5210 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.159+0000 7f2f4fca8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481609a0 con 0x7f2f481613c0 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.159+0000 7f2f4fca8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2f48160ea0 con 0x7f2f480b7d00 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 0x7f2f480cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 0x7f2f480cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57422/0 (socket says 172.21.15.46:57422) 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 -- 172.21.15.46:0/666812884 learned_addr learned my addr 172.21.15.46:0/666812884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4d243640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b7d00 0x7f2f480cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4e245640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f481613c0 0x7f2f4815d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 -- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b7d00 msgr2=0x7f2f480cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b7d00 0x7f2f480cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 -- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f481613c0 msgr2=0x7f2f4815d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f481613c0 0x7f2f4815d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 -- 172.21.15.46:0/666812884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f480cf830 con 0x7f2f480b5210 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4e245640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f481613c0 0x7f2f4815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4da44640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 0x7f2f480cba10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400c730 tx=0x7f2f4400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44017070 con 0x7f2f480b5210 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f44005ad0 con 0x7f2f480b5210 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f480cfb10 con 0x7f2f480b5210 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44007070 con 0x7f2f480b5210 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.160+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f4815ced0 con 0x7f2f480b5210 2024-05-09T03:15:45.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.161+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4806a270 con 0x7f2f480b5210 2024-05-09T03:15:45.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.162+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2f4401a070 con 0x7f2f480b5210 2024-05-09T03:15:45.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.163+0000 7f2f3e7fc640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f2c0762d0 0x7f2f2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.163+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f2f44096ce0 con 0x7f2f480b5210 2024-05-09T03:15:45.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.163+0000 7f2f4d243640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f2c0762d0 0x7f2f2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.164+0000 7f2f4d243640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f2c0762d0 0x7f2f2c0787e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2f38034000 tx=0x7f2f3802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.167+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2f44021020 con 0x7f2f480b5210 2024-05-09T03:15:45.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.362+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f480d40d0 con 0x7f2f480b5210 2024-05-09T03:15:45.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.363+0000 7f2f3e7fc640 1 -- 172.21.15.46:0/666812884 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+13077 (secure 0 0 0) 0x7f2f480b8110 con 0x7f2f480b5210 2024-05-09T03:15:45.364 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:45.364 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":74,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:15:43.002990+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/3673458792":"2024-05-10T03:15:39.813672+0000","172.21.15.88:6834/3673458792":"2024-05-10T03:15:39.813672+0000","172.21.15.46:6834/131652171":"2024-05-10T03:15:31.456349+0000","172.21.15.46:6832/131652171":"2024-05-10T03:15:31.456349+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f2c0762d0 msgr2=0x7f2f2c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f2c0762d0 0x7f2f2c0787e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2f38034000 tx=0x7f2f3802e000 comp rx=0 tx=0).stop 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 msgr2=0x7f2f480cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 0x7f2f480cba10 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400c730 tx=0x7f2f4400cbf0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 shutdown_connections 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f481613c0 0x7f2f4815d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b7d00 0x7f2f480cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f2c0762d0 0x7f2f2c0787e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 --2- 172.21.15.46:0/666812884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b5210 0x7f2f480cba10 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 >> 172.21.15.46:0/666812884 conn(0x7f2f480d29a0 msgr2=0x7f2f4815f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:45.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 shutdown_connections 2024-05-09T03:15:45.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.365+0000 7f2f4fca8640 1 -- 172.21.15.46:0/666812884 wait complete. 2024-05-09T03:15:45.378 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T03:09:22.506928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T03:15:45.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T03:15:45.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.494+0000 7f0bca534640 1 Processor -- start 2024-05-09T03:15:45.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.494+0000 7f0bca534640 1 -- start start 2024-05-09T03:15:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.495+0000 7f0bca534640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bc40b5140 0x7f0bc40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bca534640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40d3fc0 0x7f0bc40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bca534640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bc40b80b0 con 0x7f0bc40b7ca0 2024-05-09T03:15:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bca534640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0bc40d3480 con 0x7f0bc40b5140 2024-05-09T03:15:45.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bca534640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc40b5550 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0bc40b5140 0x7f0bc40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0bc40b7ca0 0x7f0bc40d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41198/0 (socket says 172.21.15.46:41198) 2024-05-09T03:15:45.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc8ad1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40d3fc0 0x7f0bc40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc37fe640 1 -- 172.21.15.46:0/1036324084 learned_addr learned my addr 172.21.15.46:0/1036324084 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc3fff640 1 -- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40d3fc0 msgr2=0x7f0bc40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc3fff640 1 --2- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40d3fc0 0x7f0bc40b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc3fff640 1 -- 172.21.15.46:0/1036324084 >> v1:172.21.15.88:6789/0 conn(0x7f0bc40b7ca0 legacy=0x7f0bc40d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.496+0000 7f0bc3fff640 1 -- 172.21.15.46:0/1036324084 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc40d44e0 con 0x7f0bc40b5140 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.497+0000 7f0bc3fff640 1 --2- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc40b7760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40097a0 tx=0x7f0bb402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9b57459693e389 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.497+0000 7f0bc2ffd640 1 -- 172.21.15.46:0/1036324084 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4037070 con 0x7f0bc40b5140 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.497+0000 7f0bc2ffd640 1 -- 172.21.15.46:0/1036324084 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bb4032440 con 0x7f0bc40b5140 2024-05-09T03:15:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.497+0000 7f0bc2ffd640 1 -- 172.21.15.46:0/1036324084 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bb4032810 con 0x7f0bc40b5140 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 -- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 msgr2=0x7f0bc40b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 --2- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc40b7760 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40097a0 tx=0x7f0bb402a210 comp rx=0 tx=0).stop 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 -- 172.21.15.46:0/1036324084 shutdown_connections 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 --2- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40d3fc0 0x7f0bc40b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 --2- 172.21.15.46:0/1036324084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc40b7760 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 -- 172.21.15.46:0/1036324084 >> 172.21.15.46:0/1036324084 conn(0x7f0bc408d3f0 msgr2=0x7f0bc408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 -- 172.21.15.46:0/1036324084 shutdown_connections 2024-05-09T03:15:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.498+0000 7f0bca534640 1 -- 172.21.15.46:0/1036324084 wait complete. 2024-05-09T03:15:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.500+0000 7f0bca534640 1 Processor -- start 2024-05-09T03:15:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.500+0000 7f0bca534640 1 -- start start 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bca534640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc4158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bca534640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40b7ca0 0x7f0bc40c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bca534640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 0x7f0bc415adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bca534640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc40d4d80 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bca534640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc40d4aa0 con 0x7f0bc40b5140 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bca534640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc40d5580 con 0x7f0bc40b7ca0 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bc8ad1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 0x7f0bc415adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bc8ad1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 0x7f0bc415adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57434/0 (socket says 172.21.15.46:57434) 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bc8ad1640 1 -- 172.21.15.46:0/3936851758 learned_addr learned my addr 172.21.15.46:0/3936851758 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bc3fff640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc4158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.501+0000 7f0bc37fe640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40b7ca0 0x7f0bc40c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc8ad1640 1 -- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40b7ca0 msgr2=0x7f0bc40c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc8ad1640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40b7ca0 0x7f0bc40c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc8ad1640 1 -- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 msgr2=0x7f0bc4158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc8ad1640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc4158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc8ad1640 1 -- 172.21.15.46:0/3936851758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc4159c90 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc3fff640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc4158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc8ad1640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 0x7f0bc415adf0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00efc0 tx=0x7f0bbc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc018020 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bbc014480 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc4159f70 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc009e60 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.502+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc415a530 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.503+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bc406a270 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.504+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0bbc005d10 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.504+0000 7f0bc0ff9640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0bb00762d0 0x7f0bb00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.505+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f0bbc0984d0 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.505+0000 7f0bc3fff640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0bb00762d0 0x7f0bb00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.505+0000 7f0bc3fff640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0bb00762d0 0x7f0bb00787e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4034000 tx=0x7f0bb402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.508+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0bbc022030 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.701+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0bc4158ce0 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.702+0000 7f0bc0ff9640 1 -- 172.21.15.46:0/3936851758 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f0bbc010040 con 0x7f0bc40d3fc0 2024-05-09T03:15:45.703 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0bb00762d0 msgr2=0x7f0bb00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0bb00762d0 0x7f0bb00787e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4034000 tx=0x7f0bb402e000 comp rx=0 tx=0).stop 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 msgr2=0x7f0bc415adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 0x7f0bc415adf0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00efc0 tx=0x7f0bbc00c490 comp rx=0 tx=0).stop 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 shutdown_connections 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0bb00762d0 0x7f0bb00787e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bc40d3fc0 0x7f0bc415adf0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0bc40b7ca0 0x7f0bc40c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 --2- 172.21.15.46:0/3936851758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bc40b5140 0x7f0bc4158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 >> 172.21.15.46:0/3936851758 conn(0x7f0bc408d3f0 msgr2=0x7f0bc4163010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:45.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 shutdown_connections 2024-05-09T03:15:45.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.704+0000 7f0bca534640 1 -- 172.21.15.46:0/3936851758 wait complete. 2024-05-09T03:15:45.717 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-05-09T03:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.830+0000 7f36119f3640 1 Processor -- start 2024-05-09T03:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.830+0000 7f36119f3640 1 -- start start 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f36119f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f360c0d57a0 0x7f360c160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f36119f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0b7ef0 0x7f360c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f36119f3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f360c0d49e0 con 0x7f360c0b4af0 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f36119f3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f360c0d5bb0 con 0x7f360c0d57a0 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f36119f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f360c0d4cc0 con 0x7f360c0b7ef0 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f360c0d57a0 0x7f360c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f360c0d57a0 0x7f360c160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51978/0 (socket says 172.21.15.46:51978) 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 -- 172.21.15.46:0/4259568867 learned_addr learned my addr 172.21.15.46:0/4259568867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 -- 172.21.15.46:0/4259568867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0b7ef0 msgr2=0x7f360c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 --2- 172.21.15.46:0/4259568867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0b7ef0 0x7f360c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 -- 172.21.15.46:0/4259568867 >> v1:172.21.15.88:6789/0 conn(0x7f360c0b4af0 legacy=0x7f360c0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.832+0000 7f360affd640 1 -- 172.21.15.46:0/4259568867 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360c0d4fa0 con 0x7f360c0d57a0 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.833+0000 7f360affd640 1 --2- 172.21.15.46:0/4259568867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c0d57a0 0x7f360c160610 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009e50 tx=0x7f35fc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5637728ecba7a95f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.833+0000 7f3609ffb640 1 -- 172.21.15.46:0/4259568867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc036070 con 0x7f360c0d57a0 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.833+0000 7f3609ffb640 1 -- 172.21.15.46:0/4259568867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc031600 con 0x7f360c0d57a0 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.833+0000 7f3609ffb640 1 -- 172.21.15.46:0/4259568867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc0319d0 con 0x7f360c0d57a0 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 -- 172.21.15.46:0/4259568867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c0d57a0 msgr2=0x7f360c160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 --2- 172.21.15.46:0/4259568867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c0d57a0 0x7f360c160610 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009e50 tx=0x7f35fc02a680 comp rx=0 tx=0).stop 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 -- 172.21.15.46:0/4259568867 shutdown_connections 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 --2- 172.21.15.46:0/4259568867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0b7ef0 0x7f360c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 --2- 172.21.15.46:0/4259568867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c0d57a0 0x7f360c160610 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 -- 172.21.15.46:0/4259568867 >> 172.21.15.46:0/4259568867 conn(0x7f360c0743e0 msgr2=0x7f360c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 -- 172.21.15.46:0/4259568867 shutdown_connections 2024-05-09T03:15:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.834+0000 7f36119f3640 1 -- 172.21.15.46:0/4259568867 wait complete. 2024-05-09T03:15:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.836+0000 7f36119f3640 1 Processor -- start 2024-05-09T03:15:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.836+0000 7f36119f3640 1 -- start start 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f36119f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360c0b4af0 0x7f360c1631f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f36119f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0d57a0 0x7f360c0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f36119f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 0x7f360c0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f36119f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f360c0d1920 con 0x7f360c1541c0 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f36119f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f360c0d15b0 con 0x7f360c0b4af0 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f36119f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f360c0c6440 con 0x7f360c0d57a0 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f360b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 0x7f360c0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f360b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 0x7f360c0b6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51992/0 (socket says 172.21.15.46:51992) 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f360b7fe640 1 -- 172.21.15.46:0/2446083411 learned_addr learned my addr 172.21.15.46:0/2446083411 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:45.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f360affd640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360c0b4af0 0x7f360c1631f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.837+0000 7f360a7fc640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0d57a0 0x7f360c0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f360b7fe640 1 -- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0d57a0 msgr2=0x7f360c0c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f360b7fe640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0d57a0 0x7f360c0c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f360b7fe640 1 -- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360c0b4af0 msgr2=0x7f360c1631f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f360b7fe640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360c0b4af0 0x7f360c1631f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f360b7fe640 1 -- 172.21.15.46:0/2446083411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360c163730 con 0x7f360c1541c0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f360b7fe640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 0x7f360c0b6df0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f360400b440 tx=0x7f360400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604015020 con 0x7f360c1541c0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3604011850 con 0x7f360c1541c0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360c0c8030 con 0x7f360c1541c0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3604011c40 con 0x7f360c1541c0 2024-05-09T03:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.838+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f360c154a40 con 0x7f360c1541c0 2024-05-09T03:15:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.839+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f360c06a270 con 0x7f360c1541c0 2024-05-09T03:15:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.840+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3604018020 con 0x7f360c1541c0 2024-05-09T03:15:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.840+0000 7f3603fff640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f35ec0762d0 0x7f35ec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.840+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f3604097130 con 0x7f360c1541c0 2024-05-09T03:15:45.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.841+0000 7f360affd640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f35ec0762d0 0x7f35ec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.841+0000 7f360affd640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f35ec0762d0 0x7f35ec0787e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f35fc033000 tx=0x7f35fc009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:45.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:45.845+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f360401f020 con 0x7f360c1541c0 2024-05-09T03:15:46.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:46.111+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) -- 0x7f360c15f670 con 0x7f360c1541c0 2024-05-09T03:15:47.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.021+0000 7f3603fff640 1 -- 172.21.15.46:0/2446083411 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v298) ==== 138+0+0 (secure 0 0 0) 0x7f360c0c8030 con 0x7f360c1541c0 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f35ec0762d0 msgr2=0x7f35ec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f35ec0762d0 0x7f35ec0787e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f35fc033000 tx=0x7f35fc009000 comp rx=0 tx=0).stop 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 msgr2=0x7f360c0b6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 0x7f360c0b6df0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f360400b440 tx=0x7f360400b840 comp rx=0 tx=0).stop 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 shutdown_connections 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f35ec0762d0 0x7f35ec0787e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f360c1541c0 0x7f360c0b6df0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f360c0d57a0 0x7f360c0c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 --2- 172.21.15.46:0/2446083411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360c0b4af0 0x7f360c1631f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 >> 172.21.15.46:0/2446083411 conn(0x7f360c0743e0 msgr2=0x7f360c0b9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:47.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 shutdown_connections 2024-05-09T03:15:47.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.024+0000 7f36119f3640 1 -- 172.21.15.46:0/2446083411 wait complete. 2024-05-09T03:15:47.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T03:15:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.163+0000 7f0af122b640 1 Processor -- start 2024-05-09T03:15:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.163+0000 7f0af122b640 1 -- start start 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0af122b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0aec0bb770 0x7f0aec0d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0af122b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec163eb0 0x7f0aec1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0af122b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aec164900 con 0x7f0aec0d2a70 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0af122b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0aec0d4bb0 con 0x7f0aec0bb770 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0af122b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0aec164be0 con 0x7f0aec163eb0 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0aead9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0aec0bb770 0x7f0aec0d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0aead9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0aec0bb770 0x7f0aec0d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51996/0 (socket says 172.21.15.46:51996) 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0aead9d640 1 -- 172.21.15.46:0/1993533357 learned_addr learned my addr 172.21.15.46:0/1993533357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.164+0000 7f0aea59c640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec163eb0 0x7f0aec1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0aead9d640 1 -- 172.21.15.46:0/1993533357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec163eb0 msgr2=0x7f0aec1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0aead9d640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec163eb0 0x7f0aec1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0aead9d640 1 -- 172.21.15.46:0/1993533357 >> v1:172.21.15.88:6789/0 conn(0x7f0aec0d2a70 legacy=0x7f0aec0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0aead9d640 1 -- 172.21.15.46:0/1993533357 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec0d4840 con 0x7f0aec0bb770 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0aea59c640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec163eb0 0x7f0aec1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0aead9d640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec0bb770 0x7f0aec0d5040 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009e50 tx=0x7f0adc02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e24629dd528de6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.165+0000 7f0ae9d9b640 1 -- 172.21.15.46:0/1993533357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc036070 con 0x7f0aec0bb770 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0ae9d9b640 1 -- 172.21.15.46:0/1993533357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc031600 con 0x7f0aec0bb770 2024-05-09T03:15:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0ae9d9b640 1 -- 172.21.15.46:0/1993533357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc0319d0 con 0x7f0aec0bb770 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 -- 172.21.15.46:0/1993533357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec0bb770 msgr2=0x7f0aec0d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec0bb770 0x7f0aec0d5040 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009e50 tx=0x7f0adc02a710 comp rx=0 tx=0).stop 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 -- 172.21.15.46:0/1993533357 shutdown_connections 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec163eb0 0x7f0aec1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 --2- 172.21.15.46:0/1993533357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec0bb770 0x7f0aec0d5040 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 -- 172.21.15.46:0/1993533357 >> 172.21.15.46:0/1993533357 conn(0x7f0aec06e370 msgr2=0x7f0aec06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 -- 172.21.15.46:0/1993533357 shutdown_connections 2024-05-09T03:15:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.166+0000 7f0af122b640 1 -- 172.21.15.46:0/1993533357 wait complete. 2024-05-09T03:15:47.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.168+0000 7f0af122b640 1 Processor -- start 2024-05-09T03:15:47.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.168+0000 7f0af122b640 1 -- start start 2024-05-09T03:15:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0af122b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aec0bb770 0x7f0aec15bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0af122b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec0d2a70 0x7f0aec0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0af122b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 0x7f0aec0ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0af122b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec0d0bc0 con 0x7f0aec163eb0 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0af122b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec0bbb80 con 0x7f0aec0bb770 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0af122b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0aec0d0d40 con 0x7f0aec0d2a70 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 0x7f0aec0ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 0x7f0aec0ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51998/0 (socket says 172.21.15.46:51998) 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 -- 172.21.15.46:0/2648478577 learned_addr learned my addr 172.21.15.46:0/2648478577 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aead9d640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aec0bb770 0x7f0aec15bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aea59c640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec0d2a70 0x7f0aec0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 -- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec0d2a70 msgr2=0x7f0aec0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec0d2a70 0x7f0aec0ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 -- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aec0bb770 msgr2=0x7f0aec15bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aec0bb770 0x7f0aec15bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 -- 172.21.15.46:0/2648478577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec15e0f0 con 0x7f0aec163eb0 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aea59c640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec0d2a70 0x7f0aec0ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.170+0000 7f0aeb59e640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 0x7f0aec0ca760 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c370 tx=0x7f0ae400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.171+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4017070 con 0x7f0aec163eb0 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.171+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae40052f0 con 0x7f0aec163eb0 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.171+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aec1f9e30 con 0x7f0aec163eb0 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.171+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4003850 con 0x7f0aec163eb0 2024-05-09T03:15:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.171+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aec1fa3f0 con 0x7f0aec163eb0 2024-05-09T03:15:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.172+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0aec15e2c0 con 0x7f0aec163eb0 2024-05-09T03:15:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.173+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0ae401a070 con 0x7f0aec163eb0 2024-05-09T03:15:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.173+0000 7f0ad37fe640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ad80762d0 0x7f0ad80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.173+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f0ae4096760 con 0x7f0aec163eb0 2024-05-09T03:15:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.173+0000 7f0aead9d640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ad80762d0 0x7f0ad80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.174+0000 7f0aead9d640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ad80762d0 0x7f0ad80787e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0adc033000 tx=0x7f0adc009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.178+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0ae40602d0 con 0x7f0aec163eb0 2024-05-09T03:15:47.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.369+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f0aec06a270 con 0x7f0aec163eb0 2024-05-09T03:15:47.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.373+0000 7f0ad37fe640 1 -- 172.21.15.46:0/2648478577 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7f0ae4068350 con 0x7f0aec163eb0 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ad80762d0 msgr2=0x7f0ad80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ad80762d0 0x7f0ad80787e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f0adc033000 tx=0x7f0adc009000 comp rx=0 tx=0).stop 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 msgr2=0x7f0aec0ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 0x7f0aec0ca760 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c370 tx=0x7f0ae400c830 comp rx=0 tx=0).stop 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 shutdown_connections 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ad80762d0 0x7f0ad80787e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aec163eb0 0x7f0aec0ca760 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0aec0d2a70 0x7f0aec0ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 --2- 172.21.15.46:0/2648478577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0aec0bb770 0x7f0aec15bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 >> 172.21.15.46:0/2648478577 conn(0x7f0aec06e370 msgr2=0x7f0aec0b5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 shutdown_connections 2024-05-09T03:15:47.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.375+0000 7f0af122b640 1 -- 172.21.15.46:0/2648478577 wait complete. 2024-05-09T03:15:47.389 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:15:47.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:15:47.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:15:47.479 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:15:47.479 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:15:47.479 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:47.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:15:47.496 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:15:47.496 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:47.514 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:15:47.514 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:15:47.514 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:47.552 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:15:47.552 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:15:47.552 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:15:47.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:15:47.572 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T03:15:47.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:47.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.705+0000 7faffac96640 1 Processor -- start 2024-05-09T03:15:47.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.705+0000 7faffac96640 1 -- start start 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.706+0000 7faffac96640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faff408cd60 0x7faff40d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.706+0000 7faffac96640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faff40d2090 0x7faff40d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.706+0000 7faffac96640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faff40d4900 con 0x7faff40d1c80 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.706+0000 7faffac96640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faff40b5340 con 0x7faff40d2090 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.706+0000 7faffac96640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faff40b4fd0 con 0x7faff408cd60 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faff40d2090 0x7faff40d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faff40d2090 0x7faff40d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52006/0 (socket says 172.21.15.46:52006) 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 -- 172.21.15.46:0/3986587179 learned_addr learned my addr 172.21.15.46:0/3986587179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff8a32640 1 --2- 172.21.15.46:0/3986587179 >> v2:172.21.15.88:3300/0 conn(0x7faff408cd60 0x7faff40d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 -- 172.21.15.46:0/3986587179 >> v2:172.21.15.88:3300/0 conn(0x7faff408cd60 msgr2=0x7faff40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 --2- 172.21.15.46:0/3986587179 >> v2:172.21.15.88:3300/0 conn(0x7faff408cd60 0x7faff40d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 -- 172.21.15.46:0/3986587179 >> v1:172.21.15.88:6789/0 conn(0x7faff40d1c80 legacy=0x7faff40cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.707+0000 7faff9233640 1 -- 172.21.15.46:0/3986587179 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff40d2c00 con 0x7faff40d2090 2024-05-09T03:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.708+0000 7faff9233640 1 --2- 172.21.15.46:0/3986587179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff40d02c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fafec009e50 tx=0x7fafec02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45123219c6e6d551 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.708+0000 7faff37fe640 1 -- 172.21.15.46:0/3986587179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec036070 con 0x7faff40d2090 2024-05-09T03:15:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.708+0000 7faff37fe640 1 -- 172.21.15.46:0/3986587179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafec031600 con 0x7faff40d2090 2024-05-09T03:15:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.708+0000 7faff37fe640 1 -- 172.21.15.46:0/3986587179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec0319d0 con 0x7faff40d2090 2024-05-09T03:15:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 -- 172.21.15.46:0/3986587179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 msgr2=0x7faff40d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 --2- 172.21.15.46:0/3986587179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff40d02c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fafec009e50 tx=0x7fafec02a6d0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 -- 172.21.15.46:0/3986587179 shutdown_connections 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 --2- 172.21.15.46:0/3986587179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff40d02c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 --2- 172.21.15.46:0/3986587179 >> v2:172.21.15.88:3300/0 conn(0x7faff408cd60 0x7faff40d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 -- 172.21.15.46:0/3986587179 >> 172.21.15.46:0/3986587179 conn(0x7faff40d3c00 msgr2=0x7faff40d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 -- 172.21.15.46:0/3986587179 shutdown_connections 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.709+0000 7faffac96640 1 -- 172.21.15.46:0/3986587179 wait complete. 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.711+0000 7faffac96640 1 Processor -- start 2024-05-09T03:15:47.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.711+0000 7faffac96640 1 -- start start 2024-05-09T03:15:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faffac96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faff408cd60 0x7faff4154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faffac96640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faff40d1c80 0x7faff415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faffac96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff41e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faffac96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faff40b92b0 con 0x7faff40d2090 2024-05-09T03:15:47.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faffac96640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faff40b9130 con 0x7faff408cd60 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faffac96640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faff40cf9c0 con 0x7faff40d1c80 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faff9233640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff41e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faff9233640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff41e8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52018/0 (socket says 172.21.15.46:52018) 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.712+0000 7faff9233640 1 -- 172.21.15.46:0/1101237719 learned_addr learned my addr 172.21.15.46:0/1101237719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff8a32640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faff408cd60 0x7faff4154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff3fff640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faff40d1c80 0x7faff415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff9233640 1 -- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faff40d1c80 msgr2=0x7faff415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff9233640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faff40d1c80 0x7faff415ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff9233640 1 -- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faff408cd60 msgr2=0x7faff4154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff9233640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faff408cd60 0x7faff4154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff9233640 1 -- 172.21.15.46:0/1101237719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff4160050 con 0x7faff40d2090 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff3fff640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faff40d1c80 0x7faff415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff9233640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff41e8280 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fafec009e50 tx=0x7fafec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec036050 con 0x7faff40d2090 2024-05-09T03:15:47.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafec042070 con 0x7faff40d2090 2024-05-09T03:15:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff4154650 con 0x7faff40d2090 2024-05-09T03:15:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafec009510 con 0x7faff40d2090 2024-05-09T03:15:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.713+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff415fa90 con 0x7faff40d2090 2024-05-09T03:15:47.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.714+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff415f520 con 0x7faff40d2090 2024-05-09T03:15:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.715+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fafec007bd0 con 0x7faff40d2090 2024-05-09T03:15:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.716+0000 7faff17fa640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafe00762d0 0x7fafe00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.716+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fafec0b81c0 con 0x7faff40d2090 2024-05-09T03:15:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.716+0000 7faff8a32640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafe00762d0 0x7fafe00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:47.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.717+0000 7faff8a32640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafe00762d0 0x7fafe00787e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faff4074930 tx=0x7fafe4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:47.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.721+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faff4154650 con 0x7faff40d2090 2024-05-09T03:15:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.950+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faff4162080 con 0x7faff40d2090 2024-05-09T03:15:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.950+0000 7faff17fa640 1 -- 172.21.15.46:0/1101237719 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) ==== 94+0+3869 (secure 0 0 0) 0x7fafec081d20 con 0x7faff40d2090 2024-05-09T03:15:47.952 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:47.952 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":78,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5603,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1465622296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1465622296},{"type":"v1","addr":"172.21.15.88:6837","nonce":1465622296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":5612,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/1738850640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":1738850640},{"type":"v1","addr":"172.21.15.46:6835","nonce":1738850640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":64},{"gid":6301,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3412082543","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3412082543},{"type":"v1","addr":"172.21.15.88:6836","nonce":3412082543}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6705,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/4157789947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":4157789947},{"type":"v1","addr":"172.21.15.46:6834","nonce":4157789947}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":74}],"filesystems":[]} 2024-05-09T03:15:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 78 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafe00762d0 msgr2=0x7fafe00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafe00762d0 0x7fafe00787e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faff4074930 tx=0x7fafe4008040 comp rx=0 tx=0).stop 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 msgr2=0x7faff41e8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff41e8280 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fafec009e50 tx=0x7fafec004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 shutdown_connections 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafe00762d0 0x7fafe00787e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faff40d2090 0x7faff41e8280 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faff40d1c80 0x7faff415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 --2- 172.21.15.46:0/1101237719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faff408cd60 0x7faff4154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 >> 172.21.15.46:0/1101237719 conn(0x7faff40d3c00 msgr2=0x7faff40ca2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 shutdown_connections 2024-05-09T03:15:47.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:47.952+0000 7faffac96640 1 -- 172.21.15.46:0/1101237719 wait complete. 2024-05-09T03:15:47.966 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:15:47.966 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:15:47.966 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:15:47.966 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:15:47.966 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:15:47.966 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T03:15:47.967 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:15:47.967 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:15:47.967 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:15:47.967 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T03:15:47.967 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:15:47.967 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:15:47.967 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:15:47.967 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T03:15:47.968 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:15:47.968 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:15:47.976 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:15:47.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:15:47.977 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:15:47.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:15:47.979 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:15:47.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:15:47.982 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:15:47.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:15:48.019 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:15:48.019 DEBUG:teuthology.parallel:result is None 2024-05-09T03:15:48.019 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:15:48.019 DEBUG:teuthology.parallel:result is None 2024-05-09T03:15:48.023 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:15:48.024 DEBUG:teuthology.parallel:result is None 2024-05-09T03:15:48.025 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:15:48.025 DEBUG:teuthology.parallel:result is None 2024-05-09T03:15:48.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T03:15:48.110 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:15:48.110 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:15:48.119 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:15:48.119 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:15:48.119 INFO:tasks.ceph.mds.d.smithi088.stderr:2024-05-09T03:15:48.119+0000 7fe3b0886640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-05-09T03:15:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.145+0000 7fd86c69a640 1 Processor -- start 2024-05-09T03:15:48.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.145+0000 7fd86c69a640 1 -- start start 2024-05-09T03:15:48.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-05-09T03:15:48.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-05-09T03:15:48.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86c69a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8640b3ef0 0x7fd86406cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86c69a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8640d30c0 0x7fd8640c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86c69a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8640b4300 con 0x7fd86406e790 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86ac37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8640d30c0 0x7fd8640c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86ac37640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8640d30c0 0x7fd8640c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52074/0 (socket says 172.21.15.46:52074) 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86ac37640 1 -- 172.21.15.46:0/351254392 learned_addr learned my addr 172.21.15.46:0/351254392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd86406d450 con 0x7fd8640d30c0 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd86406d6f0 con 0x7fd8640b3ef0 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86a436640 1 --2- 172.21.15.46:0/351254392 >> v2:172.21.15.88:3300/0 conn(0x7fd8640b3ef0 0x7fd86406cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86ac37640 1 -- 172.21.15.46:0/351254392 >> v2:172.21.15.88:3300/0 conn(0x7fd8640b3ef0 msgr2=0x7fd86406cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.147+0000 7fd86ac37640 1 --2- 172.21.15.46:0/351254392 >> v2:172.21.15.88:3300/0 conn(0x7fd8640b3ef0 0x7fd86406cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.148+0000 7fd86ac37640 1 -- 172.21.15.46:0/351254392 >> v1:172.21.15.88:6789/0 conn(0x7fd86406e790 legacy=0x7fd8640d29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.148+0000 7fd86ac37640 1 -- 172.21.15.46:0/351254392 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86406d9d0 con 0x7fd8640d30c0 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.148+0000 7fd86ac37640 1 --2- 172.21.15.46:0/351254392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640d30c0 0x7fd8640c9af0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd8640b5570 tx=0x7fd85c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06c098ca82059db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.148+0000 7fd869434640 1 -- 172.21.15.46:0/351254392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c036070 con 0x7fd8640d30c0 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.148+0000 7fd869434640 1 -- 172.21.15.46:0/351254392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd85c031600 con 0x7fd8640d30c0 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.148+0000 7fd869434640 1 -- 172.21.15.46:0/351254392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0319d0 con 0x7fd8640d30c0 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640d30c0 msgr2=0x7fd8640c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 --2- 172.21.15.46:0/351254392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640d30c0 0x7fd8640c9af0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd8640b5570 tx=0x7fd85c02a730 comp rx=0 tx=0).stop 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 shutdown_connections 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 --2- 172.21.15.46:0/351254392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640d30c0 0x7fd8640c9af0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 --2- 172.21.15.46:0/351254392 >> v2:172.21.15.88:3300/0 conn(0x7fd8640b3ef0 0x7fd86406cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 >> 172.21.15.46:0/351254392 conn(0x7fd86409b7d0 msgr2=0x7fd86409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 shutdown_connections 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.149+0000 7fd86c69a640 1 -- 172.21.15.46:0/351254392 wait complete. 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.151+0000 7fd86c69a640 1 Processor -- start 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.151+0000 7fd86c69a640 1 -- start start 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.152+0000 7fd86c69a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 0x7fd86414cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.152+0000 7fd86c69a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 0x7fd86415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.152+0000 7fd86c69a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415f090 0x7fd8640c6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.152+0000 7fd86c69a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd86406e270 con 0x7fd8640b3ef0 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.152+0000 7fd86c69a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd86406df90 con 0x7fd86415f090 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.152+0000 7fd86c69a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd864163df0 con 0x7fd86406e790 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 0x7fd86415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86a436640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 0x7fd86414cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 0x7fd86415eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52078/0 (socket says 172.21.15.46:52078) 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86a436640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 0x7fd86414cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:45738/0 (socket says 172.21.15.46:45738) 2024-05-09T03:15:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 -- 172.21.15.46:0/1038366931 learned_addr learned my addr 172.21.15.46:0/1038366931 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86ac37640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415f090 0x7fd8640c6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 -- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 msgr2=0x7fd86414cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 0x7fd86414cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 -- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415f090 msgr2=0x7fd8640c6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415f090 0x7fd8640c6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 -- 172.21.15.46:0/1038366931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd86415dab0 con 0x7fd8640b3ef0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86a436640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 0x7fd86414cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86ac37640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415f090 0x7fd8640c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd869c35640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 0x7fd86415eb50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c400 tx=0x7fd85400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854017070 con 0x7fd8640b3ef0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8540052d0 con 0x7fd8640b3ef0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86415dd90 con 0x7fd8640b3ef0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854003830 con 0x7fd8640b3ef0 2024-05-09T03:15:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.153+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86415e350 con 0x7fd8640b3ef0 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.154+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd86406f940 con 0x7fd8640b3ef0 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.155+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd85401a070 con 0x7fd8640b3ef0 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.156+0000 7fd85affd640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8480762d0 0x7fd8480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.156+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fd854096620 con 0x7fd8640b3ef0 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.156+0000 7fd86a436640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8480762d0 0x7fd8480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.156+0000 7fd86a436640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8480762d0 0x7fd8480787e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd864073de0 tx=0x7fd860008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:48.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.160+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd85405fbd0 con 0x7fd8640b3ef0 2024-05-09T03:15:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:48.364+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd864162870 con 0x7fd8640b3ef0 2024-05-09T03:15:49.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.034+0000 7fd85affd640 1 -- 172.21.15.46:0/1038366931 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) ==== 97+0+0 (secure 0 0 0) 0x7fd854068210 con 0x7fd8640b3ef0 2024-05-09T03:15:49.036 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8480762d0 msgr2=0x7fd8480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8480762d0 0x7fd8480787e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd864073de0 tx=0x7fd860008040 comp rx=0 tx=0).stop 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 msgr2=0x7fd86415eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 0x7fd86415eb50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c400 tx=0x7fd85400c8c0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 shutdown_connections 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415f090 0x7fd8640c6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8480762d0 0x7fd8480787e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8640b3ef0 0x7fd86415eb50 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 --2- 172.21.15.46:0/1038366931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86406e790 0x7fd86414cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 >> 172.21.15.46:0/1038366931 conn(0x7fd86409b7d0 msgr2=0x7fd8640cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:49.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 shutdown_connections 2024-05-09T03:15:49.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.037+0000 7fd86c69a640 1 -- 172.21.15.46:0/1038366931 wait complete. 2024-05-09T03:15:49.052 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T03:15:49.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.169+0000 7f50886d9640 1 Processor -- start 2024-05-09T03:15:49.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.170+0000 7f50886d9640 1 -- start start 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f50886d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5080160610 0x7f50800b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f50886d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50800b7d20 0x7f50800b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f50886d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50800b8130 con 0x7f50800d57a0 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f50886d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f50800d5bb0 con 0x7f5080160610 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f50886d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f50800d49e0 con 0x7f50800b7d20 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f5085c74640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5080160610 0x7f50800b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f5086c76640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50800b7d20 0x7f50800b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f5086475640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f50800d57a0 0x7f50800b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:56356/0 (socket says 172.21.15.46:56356) 2024-05-09T03:15:49.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.171+0000 7f5086475640 1 -- 172.21.15.46:0/24060711 learned_addr learned my addr 172.21.15.46:0/24060711 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:49.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.172+0000 7f5085c74640 1 -- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50800b7d20 msgr2=0x7f50800b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:49.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.172+0000 7f5085c74640 1 --2- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50800b7d20 0x7f50800b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.172+0000 7f5085c74640 1 -- 172.21.15.46:0/24060711 >> v1:172.21.15.88:6789/0 conn(0x7f50800d57a0 legacy=0x7f50800b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.172+0000 7f5085c74640 1 -- 172.21.15.46:0/24060711 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50800d4cc0 con 0x7f5080160610 2024-05-09T03:15:49.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.173+0000 7f5085c74640 1 --2- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5080160610 0x7f50800b77e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f50740097a0 tx=0x7f507402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66312825b1c27c68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.173+0000 7f5085473640 1 -- 172.21.15.46:0/24060711 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074037070 con 0x7f5080160610 2024-05-09T03:15:49.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.173+0000 7f5085473640 1 -- 172.21.15.46:0/24060711 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5074032440 con 0x7f5080160610 2024-05-09T03:15:49.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.173+0000 7f5085473640 1 -- 172.21.15.46:0/24060711 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074032810 con 0x7f5080160610 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.173+0000 7f50886d9640 1 -- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5080160610 msgr2=0x7f50800b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.173+0000 7f50886d9640 1 --2- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5080160610 0x7f50800b77e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f50740097a0 tx=0x7f507402a180 comp rx=0 tx=0).stop 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.174+0000 7f50886d9640 1 -- 172.21.15.46:0/24060711 shutdown_connections 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.174+0000 7f50886d9640 1 --2- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50800b7d20 0x7f50800b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.174+0000 7f50886d9640 1 --2- 172.21.15.46:0/24060711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5080160610 0x7f50800b77e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.174+0000 7f50886d9640 1 -- 172.21.15.46:0/24060711 >> 172.21.15.46:0/24060711 conn(0x7f50800743e0 msgr2=0x7f50800d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.174+0000 7f50886d9640 1 -- 172.21.15.46:0/24060711 shutdown_connections 2024-05-09T03:15:49.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.174+0000 7f50886d9640 1 -- 172.21.15.46:0/24060711 wait complete. 2024-05-09T03:15:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.175+0000 7f50886d9640 1 Processor -- start 2024-05-09T03:15:49.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.175+0000 7f50886d9640 1 -- start start 2024-05-09T03:15:49.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f50886d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 0x7f5080157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f50886d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50800d57a0 0x7f50800d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f50886d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5080160610 0x7f5080158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f50886d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50800d15b0 con 0x7f50800b7d20 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f50886d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5080160a20 con 0x7f50800d57a0 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f50886d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f50800d1920 con 0x7f5080160610 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 0x7f5080157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 0x7f5080157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52088/0 (socket says 172.21.15.46:52088) 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 -- 172.21.15.46:0/3669882210 learned_addr learned my addr 172.21.15.46:0/3669882210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5085c74640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50800d57a0 0x7f50800d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086c76640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5080160610 0x7f5080158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 -- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5080160610 msgr2=0x7f5080158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5080160610 0x7f5080158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 -- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50800d57a0 msgr2=0x7f50800d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50800d57a0 0x7f50800d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 -- 172.21.15.46:0/3669882210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5080159de0 con 0x7f50800b7d20 2024-05-09T03:15:49.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.177+0000 7f5086475640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 0x7f5080157100 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f507000c3a0 tx=0x7f507000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.178+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070017070 con 0x7f50800b7d20 2024-05-09T03:15:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.178+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5070005ce0 con 0x7f50800b7d20 2024-05-09T03:15:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.178+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508015a0c0 con 0x7f50800b7d20 2024-05-09T03:15:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.178+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070007310 con 0x7f50800b7d20 2024-05-09T03:15:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.178+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5080147ae0 con 0x7f50800b7d20 2024-05-09T03:15:49.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.179+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508006a270 con 0x7f50800b7d20 2024-05-09T03:15:49.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.180+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f507001a070 con 0x7f50800b7d20 2024-05-09T03:15:49.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.180+0000 7f507effd640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f506c0762d0 0x7f506c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.180+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7f5070096650 con 0x7f50800b7d20 2024-05-09T03:15:49.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.180+0000 7f5085c74640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f506c0762d0 0x7f506c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.181+0000 7f5085c74640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f506c0762d0 0x7f506c0787e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5074034000 tx=0x7f507402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.185+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f50800b8130 con 0x7f50800b7d20 2024-05-09T03:15:49.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.454+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f50800d4360 con 0x7f50800b7d20 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.455+0000 7f507effd640 1 -- 172.21.15.46:0/3669882210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5304 (secure 0 0 0) 0x7f507005fba0 con 0x7f50800b7d20 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g==", 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.457 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw==", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.458 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw==", 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-05-09T03:15:49.459 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw==", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw==", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.460 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA==", 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.461 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg==", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA==", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.462 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g==", 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.463 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA==", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ==", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.464 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw==", 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-05-09T03:15:49.465 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw==", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ==", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:15:49.466 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg==", 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.467 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg==", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA1PjxmuVPDJhAAHOgxPpTSEd3pibPqobcHCg==", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:15:49.468 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg==", 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.469 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ==", 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-05-09T03:15:49.470 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f506c0762d0 msgr2=0x7f506c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f506c0762d0 0x7f506c0787e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5074034000 tx=0x7f507402e000 comp rx=0 tx=0).stop 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 msgr2=0x7f5080157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 0x7f5080157100 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f507000c3a0 tx=0x7f507000c860 comp rx=0 tx=0).stop 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 shutdown_connections 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5080160610 0x7f5080158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50800d57a0 0x7f50800d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f506c0762d0 0x7f506c0787e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 --2- 172.21.15.46:0/3669882210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50800b7d20 0x7f5080157100 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 >> 172.21.15.46:0/3669882210 conn(0x7f50800743e0 msgr2=0x7f508015d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 shutdown_connections 2024-05-09T03:15:49.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.457+0000 7f50886d9640 1 -- 172.21.15.46:0/3669882210 wait complete. 2024-05-09T03:15:49.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:15:49.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.586+0000 7fd6e7d57640 1 Processor -- start 2024-05-09T03:15:49.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.586+0000 7fd6e7d57640 1 -- start start 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e7d57640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6e00b7ed0 0x7fd6e0163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e7d57640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6e01617d0 con 0x7fd6e00b5210 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e7d57640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6e00d5280 con 0x7fd6e01613c0 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e7d57640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e00b5620 con 0x7fd6e00b7ed0 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e5af3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6e00b5210 0x7fd6e00d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59610/0 (socket says 172.21.15.46:59610) 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e5af3640 1 -- 172.21.15.46:0/3896920359 learned_addr learned my addr 172.21.15.46:0/3896920359 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e62f4640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6e00b7ed0 0x7fd6e0163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 707235472 0 0) 0x7fd6e01617d0 con 0x7fd6e00b5210 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6cc003520 con 0x7fd6e00b5210 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3433872097 0 0) 0x7fd6e00d5280 con 0x7fd6e01613c0 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6e01617d0 con 0x7fd6e01613c0 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4003640263 0 0) 0x7fd6cc003520 con 0x7fd6e00b5210 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6e00b7ed0 msgr2=0x7fd6e0163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6e00b7ed0 0x7fd6e0163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 >> v1:172.21.15.88:6789/0 conn(0x7fd6e01613c0 legacy=0x7fd6e00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e00d5560 con 0x7fd6e00b5210 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fd6dc0016f0 con 0x7fd6e00b5210 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e62f4640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6e00b7ed0 0x7fd6e0163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 >> v1:172.21.15.46:6789/0 conn(0x7fd6e00b5210 legacy=0x7fd6e00d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6cc003c10 0x7fd6cc004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.588+0000 7fd6e4af1640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6cc004800 0x7fd6cc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e4af1640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc005260 0x7fd6cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e00b5620 con 0x7fd6cc004800 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc00bfb0 con 0x7fd6cc003c10 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd6cc00c180 con 0x7fd6cc005260 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e52f2640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6cc004800 0x7fd6cc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e5af3640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6cc003c10 0x7fd6cc004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e62f4640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc005260 0x7fd6cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e52f2640 1 -- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc005260 msgr2=0x7fd6cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e52f2640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc005260 0x7fd6cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e52f2640 1 -- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6cc003c10 msgr2=0x7fd6cc004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e52f2640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6cc003c10 0x7fd6cc004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e52f2640 1 -- 172.21.15.46:0/3896920359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc003520 con 0x7fd6cc004800 2024-05-09T03:15:49.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.589+0000 7fd6e5af3640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6cc003c10 0x7fd6cc004060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.590+0000 7fd6e52f2640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6cc004800 0x7fd6cc004c10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0001900 tx=0x7fd6cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23a9c9a8317182b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.590+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0001520 con 0x7fd6cc004800 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.590+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6d002acd0 con 0x7fd6cc004800 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.590+0000 7fd6e4af1640 1 -- 172.21.15.46:0/3896920359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0005310 con 0x7fd6cc004800 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e7d57640 1 -- 172.21.15.46:0/3896920359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6cc004800 msgr2=0x7fd6cc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/3896920359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6cc004800 0x7fd6cc004c10 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0001900 tx=0x7fd6cc005830 comp rx=0 tx=0).stop 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e62f4640 1 -- 172.21.15.46:0/3896920359 reap_dead start 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e7d57640 1 -- 172.21.15.46:0/3896920359 shutdown_connections 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e7d57640 1 -- 172.21.15.46:0/3896920359 >> 172.21.15.46:0/3896920359 conn(0x7fd6e00d29a0 msgr2=0x7fd6e00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e7d57640 1 -- 172.21.15.46:0/3896920359 shutdown_connections 2024-05-09T03:15:49.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.591+0000 7fd6e7d57640 1 -- 172.21.15.46:0/3896920359 wait complete. 2024-05-09T03:15:49.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.593+0000 7fd6e7d57640 1 Processor -- start 2024-05-09T03:15:49.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.593+0000 7fd6e7d57640 1 -- start start 2024-05-09T03:15:49.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e7d57640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc004800 0x7fd6e00c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e7d57640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6e00ca440 0x7fd6e00ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e7d57640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 0x7fd6e00cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e7d57640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e0160ea0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e7d57640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01609a0 con 0x7fd6e00ca440 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e7d57640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd6e00d4f10 con 0x7fd6cc004800 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 0x7fd6e00cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 0x7fd6e00cba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52112/0 (socket says 172.21.15.46:52112) 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 -- 172.21.15.46:0/2402864569 learned_addr learned my addr 172.21.15.46:0/2402864569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e52f2640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6e00ca440 0x7fd6e00ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e5af3640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc004800 0x7fd6e00c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 -- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc004800 msgr2=0x7fd6e00c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc004800 0x7fd6e00c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 -- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6e00ca440 msgr2=0x7fd6e00ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6e00ca440 0x7fd6e00ca850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e62f4640 1 -- 172.21.15.46:0/2402864569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e00b77a0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.594+0000 7fd6e5af3640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc004800 0x7fd6e00c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:49.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.595+0000 7fd6e62f4640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 0x7fd6e00cba00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd6e008ea90 tx=0x7fd6d8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.595+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc007ca0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.595+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6d800fdb0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.595+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e00b7a80 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.595+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8003bd0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.595+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e00b8040 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.596+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e006a270 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.596+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd6d801f020 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.597+0000 7fd6d67fc640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6c40762d0 0x7fd6c40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.597+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7fd6d80973c0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.597+0000 7fd6e5af3640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6c40762d0 0x7fd6c40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.598+0000 7fd6e5af3640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6c40762d0 0x7fd6c40787e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc001690 tx=0x7fd6dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.602+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd6d8010040 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.796+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd6e00b7530 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.796+0000 7fd6d67fc640 1 -- 172.21.15.46:0/2402864569 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) ==== 84+0+65 (secure 0 0 0) 0x7fd6d8060fe0 con 0x7fd6e00cb5f0 2024-05-09T03:15:49.798 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:49.798 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:15:49.799 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:15:49.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:15:49.799 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:15:49.799 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:15:49.799 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6c40762d0 msgr2=0x7fd6c40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6c40762d0 0x7fd6c40787e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc001690 tx=0x7fd6dc009040 comp rx=0 tx=0).stop 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 msgr2=0x7fd6e00cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 0x7fd6e00cba00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fd6e008ea90 tx=0x7fd6d8009f90 comp rx=0 tx=0).stop 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 shutdown_connections 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd6c40762d0 0x7fd6c40787e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6e00cb5f0 0x7fd6e00cba00 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6e00ca440 0x7fd6e00ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 --2- 172.21.15.46:0/2402864569 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd6cc004800 0x7fd6e00c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 >> 172.21.15.46:0/2402864569 conn(0x7fd6e00d29a0 msgr2=0x7fd6e006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 shutdown_connections 2024-05-09T03:15:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.799+0000 7fd6e7d57640 1 -- 172.21.15.46:0/2402864569 wait complete. 2024-05-09T03:15:49.812 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:15:49.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:15:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.929+0000 7f4890ef9640 1 Processor -- start 2024-05-09T03:15:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.929+0000 7f4890ef9640 1 -- start start 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4890ef9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f488c0b5140 0x7f488c0b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4890ef9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c0b8070 0x7f488c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4890ef9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f488c06e250 0x7f488c0d48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4890ef9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f488c0b9b60 con 0x7f488c06e250 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4890ef9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f488c0b5550 con 0x7f488c0b5140 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4890ef9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f488c0b8480 con 0x7f488c0b8070 2024-05-09T03:15:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f488ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f488c06e250 0x7f488c0d48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f488ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f488c06e250 0x7f488c0d48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52118/0 (socket says 172.21.15.46:52118) 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f488ad9d640 1 -- 172.21.15.46:0/1578850065 learned_addr learned my addr 172.21.15.46:0/1578850065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f4889d9b640 1 --2- 172.21.15.46:0/1578850065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c0b8070 0x7f488c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.930+0000 7f488a59c640 1 --2- 172.21.15.46:0/1578850065 >> v2:172.21.15.88:3300/0 conn(0x7f488c0b5140 0x7f488c0b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.931+0000 7f488ad9d640 1 -- 172.21.15.46:0/1578850065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c0b8070 msgr2=0x7f488c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.931+0000 7f488ad9d640 1 --2- 172.21.15.46:0/1578850065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c0b8070 0x7f488c06dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.931+0000 7f488ad9d640 1 -- 172.21.15.46:0/1578850065 >> v2:172.21.15.88:3300/0 conn(0x7f488c0b5140 msgr2=0x7f488c0b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.931+0000 7f488ad9d640 1 --2- 172.21.15.46:0/1578850065 >> v2:172.21.15.88:3300/0 conn(0x7f488c0b5140 0x7f488c0b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.931+0000 7f488ad9d640 1 -- 172.21.15.46:0/1578850065 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c06e660 con 0x7f488c06e250 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.931+0000 7f488ad9d640 1 --2- 172.21.15.46:0/1578850065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c06e250 0x7f488c0d48d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f488c08e1e0 tx=0x7f487c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964105f8b4dc8aa5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f488959a640 1 -- 172.21.15.46:0/1578850065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c037070 con 0x7f488c06e250 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f488959a640 1 -- 172.21.15.46:0/1578850065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487c032440 con 0x7f488c06e250 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f488959a640 1 -- 172.21.15.46:0/1578850065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487c032810 con 0x7f488c06e250 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 -- 172.21.15.46:0/1578850065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c06e250 msgr2=0x7f488c0d48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1578850065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c06e250 0x7f488c0d48d0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f488c08e1e0 tx=0x7f487c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 -- 172.21.15.46:0/1578850065 shutdown_connections 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1578850065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c06e250 0x7f488c0d48d0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1578850065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c0b8070 0x7f488c06dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1578850065 >> v2:172.21.15.88:3300/0 conn(0x7f488c0b5140 0x7f488c0b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 -- 172.21.15.46:0/1578850065 >> 172.21.15.46:0/1578850065 conn(0x7f488c0d5e90 msgr2=0x7f488c0d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 -- 172.21.15.46:0/1578850065 shutdown_connections 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.932+0000 7f4890ef9640 1 -- 172.21.15.46:0/1578850065 wait complete. 2024-05-09T03:15:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.935+0000 7f4890ef9640 1 Processor -- start 2024-05-09T03:15:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.935+0000 7f4890ef9640 1 -- start start 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4890ef9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c06e250 0x7f488c1e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4890ef9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 0x7f488c1f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4890ef9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c0b8070 0x7f488c0b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4890ef9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f488c0b9510 con 0x7f488c0b5140 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4890ef9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f488c0b9270 con 0x7f488c0b8070 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4890ef9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f488c0b97f0 con 0x7f488c06e250 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 0x7f488c1f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 0x7f488c1f5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52132/0 (socket says 172.21.15.46:52132) 2024-05-09T03:15:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 -- 172.21.15.46:0/1086456941 learned_addr learned my addr 172.21.15.46:0/1086456941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f488a59c640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c06e250 0x7f488c1e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f488ad9d640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c0b8070 0x7f488c0b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 -- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c06e250 msgr2=0x7f488c1e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c06e250 0x7f488c1e8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 -- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c0b8070 msgr2=0x7f488c0b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c0b8070 0x7f488c0b56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.936+0000 7f4889d9b640 1 -- 172.21.15.46:0/1086456941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c15d0b0 con 0x7f488c0b5140 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.937+0000 7f4889d9b640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 0x7f488c1f5af0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f487400e8a0 tx=0x7f487400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.937+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874017070 con 0x7f488c0b5140 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.937+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f487400c940 con 0x7f488c0b5140 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.937+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c15d390 con 0x7f488c0b5140 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.937+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f487400cd30 con 0x7f488c0b5140 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.937+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c15d950 con 0x7f488c0b5140 2024-05-09T03:15:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.938+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f488c15ce00 con 0x7f488c0b5140 2024-05-09T03:15:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.939+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f487401a070 con 0x7f488c0b5140 2024-05-09T03:15:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.939+0000 7f4882ffd640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f486c076300 0x7f486c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:49.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.939+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7f4874097470 con 0x7f488c0b5140 2024-05-09T03:15:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.940+0000 7f488a59c640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f486c076300 0x7f486c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.940+0000 7f488a59c640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f486c076300 0x7f486c078810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4884011200 tx=0x7f488400d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:49.944+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4874060a40 con 0x7f488c0b5140 2024-05-09T03:15:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:50.138+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f488c06a270 con 0x7f488c0b5140 2024-05-09T03:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.040+0000 7f4882ffd640 1 -- 172.21.15.46:0/1086456941 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) ==== 138+0+0 (secure 0 0 0) 0x7f4874064d40 con 0x7f488c0b5140 2024-05-09T03:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f486c076300 msgr2=0x7f486c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f486c076300 0x7f486c078810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4884011200 tx=0x7f488400d040 comp rx=0 tx=0).stop 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 msgr2=0x7f488c1f5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 0x7f488c1f5af0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f487400e8a0 tx=0x7f487400ed60 comp rx=0 tx=0).stop 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 shutdown_connections 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f488c0b8070 0x7f488c0b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f486c076300 0x7f486c078810 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f488c0b5140 0x7f488c1f5af0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 --2- 172.21.15.46:0/1086456941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f488c06e250 0x7f488c1e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 >> 172.21.15.46:0/1086456941 conn(0x7f488c0d5e90 msgr2=0x7f488c1600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.043+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 shutdown_connections 2024-05-09T03:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.044+0000 7f4890ef9640 1 -- 172.21.15.46:0/1086456941 wait complete. 2024-05-09T03:15:51.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:15:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.190+0000 7f5d128d8640 1 Processor -- start 2024-05-09T03:15:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.190+0000 7f5d128d8640 1 -- start start 2024-05-09T03:15:51.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d128d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d0c0d1380 0x7f5d0c0d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:51.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d128d8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:51.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d128d8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0c0d45d0 con 0x7f5d0c0bb5b0 2024-05-09T03:15:51.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d128d8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5d0c0d56e0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d128d8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0c0d1790 con 0x7f5d0c0d52d0 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d0c0d1380 0x7f5d0c0d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d0c0d1380 0x7f5d0c0d4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52142/0 (socket says 172.21.15.46:52142) 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/3471301017 learned_addr learned my addr 172.21.15.46:0/3471301017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d10e75640 1 --2- 172.21.15.46:0/3471301017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/3471301017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 msgr2=0x7f5d0c0b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 --2- 172.21.15.46:0/3471301017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0b91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/3471301017 >> v1:172.21.15.88:6789/0 conn(0x7f5d0c0bb5b0 legacy=0x7f5d0c163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/3471301017 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c1645e0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0b7fe640 1 --2- 172.21.15.46:0/3471301017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0d4d90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009870 tx=0x7f5d0002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56be2e3864dbdba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:51.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0affd640 1 -- 172.21.15.46:0/3471301017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00037070 con 0x7f5d0c0d1380 2024-05-09T03:15:51.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0affd640 1 -- 172.21.15.46:0/3471301017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d00032440 con 0x7f5d0c0d1380 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.193+0000 7f5d0affd640 1 -- 172.21.15.46:0/3471301017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00032810 con 0x7f5d0c0d1380 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 -- 172.21.15.46:0/3471301017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 msgr2=0x7f5d0c0d4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 --2- 172.21.15.46:0/3471301017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0d4d90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009870 tx=0x7f5d0002a270 comp rx=0 tx=0).stop 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 -- 172.21.15.46:0/3471301017 shutdown_connections 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 --2- 172.21.15.46:0/3471301017 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0b91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 --2- 172.21.15.46:0/3471301017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0d4d90 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 -- 172.21.15.46:0/3471301017 >> 172.21.15.46:0/3471301017 conn(0x7f5d0c0d5e80 msgr2=0x7f5d0c0d6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 -- 172.21.15.46:0/3471301017 shutdown_connections 2024-05-09T03:15:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.194+0000 7f5d128d8640 1 -- 172.21.15.46:0/3471301017 wait complete. 2024-05-09T03:15:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.197+0000 7f5d128d8640 1 Processor -- start 2024-05-09T03:15:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.197+0000 7f5d128d8640 1 -- start start 2024-05-09T03:15:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d0c0bb5b0 0x7f5d0c1510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c0b96f0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5d0c164ba0 con 0x7f5d0c0bb5b0 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5d0c0b9870 con 0x7f5d0c0d52d0 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52158/0 (socket says 172.21.15.46:52158) 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/2832461461 learned_addr learned my addr 172.21.15.46:0/2832461461 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0bfff640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d0c0bb5b0 0x7f5d0c1510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d10e75640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 msgr2=0x7f5d0c0cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d0c0bb5b0 msgr2=0x7f5d0c1510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d0c0bb5b0 0x7f5d0c1510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 -- 172.21.15.46:0/2832461461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0c151630 con 0x7f5d0c0d1380 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d0b7fe640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0cf6d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5d00000c00 tx=0x7f5d000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00041020 con 0x7f5d0c0d1380 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d000306c0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00037040 con 0x7f5d0c0d1380 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d0c0ca7f0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.199+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d0c0cadb0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.200+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d0c0ccce0 con 0x7f5d0c0d1380 2024-05-09T03:15:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.201+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5d00044020 con 0x7f5d0c0d1380 2024-05-09T03:15:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.201+0000 7f5d08ff9640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5cf80762d0 0x7f5cf80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.201+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5227+0+0 (secure 0 0 0) 0x7f5d000b7b70 con 0x7f5d0c0d1380 2024-05-09T03:15:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.202+0000 7f5d0bfff640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5cf80762d0 0x7f5cf80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.202+0000 7f5d0bfff640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5cf80762d0 0x7f5cf80787e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c0d1e00 tx=0x7f5cfc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:51.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.207+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5d0004b030 con 0x7f5d0c0d1380 2024-05-09T03:15:51.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:51.395+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f5d0c06a270 con 0x7f5d0c0d1380 2024-05-09T03:15:52.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.051+0000 7f5d08ff9640 1 -- 172.21.15.46:0/2832461461 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) ==== 170+0+0 (secure 0 0 0) 0x7f5d0c151630 con 0x7f5d0c0d1380 2024-05-09T03:15:52.053 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5cf80762d0 msgr2=0x7f5cf80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5cf80762d0 0x7f5cf80787e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c0d1e00 tx=0x7f5cfc009040 comp rx=0 tx=0).stop 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 msgr2=0x7f5d0c0cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0cf6d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f5d00000c00 tx=0x7f5d000046c0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 shutdown_connections 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5d0c0d52d0 0x7f5d0c0cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5cf80762d0 0x7f5cf80787e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c0d1380 0x7f5d0c0cf6d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 --2- 172.21.15.46:0/2832461461 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d0c0bb5b0 0x7f5d0c1510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.055+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 >> 172.21.15.46:0/2832461461 conn(0x7f5d0c0d5e80 msgr2=0x7f5d0c0ba650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:52.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.056+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 shutdown_connections 2024-05-09T03:15:52.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.056+0000 7f5d128d8640 1 -- 172.21.15.46:0/2832461461 wait complete. 2024-05-09T03:15:52.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:15:52.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.211+0000 7fabe0e6a640 1 Processor -- start 2024-05-09T03:15:52.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.211+0000 7fabe0e6a640 1 -- start start 2024-05-09T03:15:52.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabe0e6a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc06e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabe0e6a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabdc06e7e0 0x7fabdc164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabe0e6a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabdc09d470 con 0x7fabdc164ec0 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabe0e6a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fabdc09cfe0 con 0x7fabdc06e7e0 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabe0e6a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fabdc0b94a0 con 0x7fabdc0d3730 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabdc06e7e0 0x7fabdc164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabdc06e7e0 0x7fabdc164980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52166/0 (socket says 172.21.15.46:52166) 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 -- 172.21.15.46:0/2328905620 learned_addr learned my addr 172.21.15.46:0/2328905620 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabda59c640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc06e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 -- 172.21.15.46:0/2328905620 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 msgr2=0x7fabdc06e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc06e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 -- 172.21.15.46:0/2328905620 >> v1:172.21.15.88:6789/0 conn(0x7fabdc164ec0 legacy=0x7fabdc0ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabd1d9b640 1 -- 172.21.15.46:0/2328905620 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc0d11b0 con 0x7fabdc06e7e0 2024-05-09T03:15:52.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.212+0000 7fabda59c640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc06e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabd1d9b640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc164980 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009870 tx=0x7fabc402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f440c705c0b98ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabd9d9b640 1 -- 172.21.15.46:0/2328905620 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4037070 con 0x7fabdc06e7e0 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabd9d9b640 1 -- 172.21.15.46:0/2328905620 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc4032440 con 0x7fabdc06e7e0 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabd9d9b640 1 -- 172.21.15.46:0/2328905620 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4032810 con 0x7fabdc06e7e0 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 -- 172.21.15.46:0/2328905620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 msgr2=0x7fabdc164980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc164980 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009870 tx=0x7fabc402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 -- 172.21.15.46:0/2328905620 shutdown_connections 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc164980 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/2328905620 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc06e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 -- 172.21.15.46:0/2328905620 >> 172.21.15.46:0/2328905620 conn(0x7fabdc0d05f0 msgr2=0x7fabdc0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:52.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.213+0000 7fabe0e6a640 1 -- 172.21.15.46:0/2328905620 shutdown_connections 2024-05-09T03:15:52.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.214+0000 7fabe0e6a640 1 -- 172.21.15.46:0/2328905620 wait complete. 2024-05-09T03:15:52.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.217+0000 7fabe0e6a640 1 Processor -- start 2024-05-09T03:15:52.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.217+0000 7fabe0e6a640 1 -- start start 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc15f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc0727f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc164ec0 0x7fabdc0a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc08cd60 con 0x7fabdc06e7e0 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc0b9160 con 0x7fabdc164ec0 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fabdc08cee0 con 0x7fabdc0d3730 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc15f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc15f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52176/0 (socket says 172.21.15.46:52176) 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 -- 172.21.15.46:0/1826892207 learned_addr learned my addr 172.21.15.46:0/1826892207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:52.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabd1d9b640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc0727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabdad9d640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc164ec0 0x7fabdc0a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 -- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 msgr2=0x7fabdc0727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc0727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 -- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc164ec0 msgr2=0x7fabdc0a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc164ec0 0x7fabdc0a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 -- 172.21.15.46:0/1826892207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc0cac20 con 0x7fabdc06e7e0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabda59c640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc15f250 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00d800 tx=0x7fabcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc004030 con 0x7fabdc06e7e0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabcc0041d0 con 0x7fabdc06e7e0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc009640 con 0x7fabdc06e7e0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabdc0cb120 con 0x7fabdc06e7e0 2024-05-09T03:15:52.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.218+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabdc0cb6e0 con 0x7fabdc06e7e0 2024-05-09T03:15:52.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.219+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabdc0b7730 con 0x7fabdc06e7e0 2024-05-09T03:15:52.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.221+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fabcc0097e0 con 0x7fabdc06e7e0 2024-05-09T03:15:52.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.221+0000 7fabd37fe640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc076300 0x7fabbc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:52.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.221+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5609+0+0 (secure 0 0 0) 0x7fabcc098180 con 0x7fabdc06e7e0 2024-05-09T03:15:52.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.221+0000 7fabd1d9b640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc076300 0x7fabbc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:52.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.221+0000 7fabd1d9b640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc076300 0x7fabbc078810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fabc4034000 tx=0x7fabc402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:52.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.225+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fabdc0b7730 con 0x7fabdc06e7e0 2024-05-09T03:15:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:52.453+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fabdc06a270 con 0x7fabdc06e7e0 2024-05-09T03:15:53.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.054+0000 7fabd37fe640 1 -- 172.21.15.46:0/1826892207 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v81) ==== 174+0+0 (secure 0 0 0) 0x7fabcc0614d0 con 0x7fabdc06e7e0 2024-05-09T03:15:53.055 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 12 and data pool 13 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc076300 msgr2=0x7fabbc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc076300 0x7fabbc078810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fabc4034000 tx=0x7fabc402e000 comp rx=0 tx=0).stop 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 msgr2=0x7fabdc15f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc15f250 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00d800 tx=0x7fabcc00dcc0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 shutdown_connections 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc164ec0 0x7fabdc0a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0d3730 0x7fabdc0727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc076300 0x7fabbc078810 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 --2- 172.21.15.46:0/1826892207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc06e7e0 0x7fabdc15f250 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.060+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 >> 172.21.15.46:0/1826892207 conn(0x7fabdc0d05f0 msgr2=0x7fabdc0d0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:53.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.061+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 shutdown_connections 2024-05-09T03:15:53.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.061+0000 7fabe0e6a640 1 -- 172.21.15.46:0/1826892207 wait complete. 2024-05-09T03:15:53.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:15:53.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.218+0000 7f53cbc43640 1 Processor -- start 2024-05-09T03:15:53.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.218+0000 7f53cbc43640 1 -- start start 2024-05-09T03:15:53.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53cbc43640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53c41615e0 0x7f53c415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53cbc43640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53c40d2480 0x7f53c40d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53cbc43640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53c4160b20 con 0x7f53c4164c70 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53cbc43640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f53c415e520 con 0x7f53c41615e0 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53cbc43640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53c4161150 con 0x7f53c40d2480 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53c91de640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53c41615e0 0x7f53c415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53c91de640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53c41615e0 0x7f53c415dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52188/0 (socket says 172.21.15.46:52188) 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.219+0000 7f53c91de640 1 -- 172.21.15.46:0/821347476 learned_addr learned my addr 172.21.15.46:0/821347476 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.220+0000 7f53ca1e0640 1 --2- 172.21.15.46:0/821347476 >> v2:172.21.15.88:3300/0 conn(0x7f53c40d2480 0x7f53c40d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.220+0000 7f53c91de640 1 -- 172.21.15.46:0/821347476 >> v2:172.21.15.88:3300/0 conn(0x7f53c40d2480 msgr2=0x7f53c40d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.220+0000 7f53c91de640 1 --2- 172.21.15.46:0/821347476 >> v2:172.21.15.88:3300/0 conn(0x7f53c40d2480 0x7f53c40d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.220+0000 7f53c91de640 1 -- 172.21.15.46:0/821347476 >> v1:172.21.15.88:6789/0 conn(0x7f53c4164c70 legacy=0x7f53c40d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.220+0000 7f53c91de640 1 -- 172.21.15.46:0/821347476 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c40d15f0 con 0x7f53c41615e0 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.221+0000 7f53c91de640 1 --2- 172.21.15.46:0/821347476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c41615e0 0x7f53c415dfe0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f53c40d3e30 tx=0x7f53b402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642b22ad07383c00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.221+0000 7f53c89dd640 1 -- 172.21.15.46:0/821347476 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4037070 con 0x7f53c41615e0 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.221+0000 7f53c89dd640 1 -- 172.21.15.46:0/821347476 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53b4032440 con 0x7f53c41615e0 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.221+0000 7f53c89dd640 1 -- 172.21.15.46:0/821347476 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4032810 con 0x7f53c41615e0 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.221+0000 7f53cbc43640 1 -- 172.21.15.46:0/821347476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c41615e0 msgr2=0x7f53c415dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.221+0000 7f53cbc43640 1 --2- 172.21.15.46:0/821347476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c41615e0 0x7f53c415dfe0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f53c40d3e30 tx=0x7f53b402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.222+0000 7f53cbc43640 1 -- 172.21.15.46:0/821347476 shutdown_connections 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.222+0000 7f53cbc43640 1 --2- 172.21.15.46:0/821347476 >> v2:172.21.15.88:3300/0 conn(0x7f53c40d2480 0x7f53c40d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.222+0000 7f53cbc43640 1 --2- 172.21.15.46:0/821347476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c41615e0 0x7f53c415dfe0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.222+0000 7f53cbc43640 1 -- 172.21.15.46:0/821347476 >> 172.21.15.46:0/821347476 conn(0x7f53c4073db0 msgr2=0x7f53c40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.222+0000 7f53cbc43640 1 -- 172.21.15.46:0/821347476 shutdown_connections 2024-05-09T03:15:53.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.222+0000 7f53cbc43640 1 -- 172.21.15.46:0/821347476 wait complete. 2024-05-09T03:15:53.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.223+0000 7f53cbc43640 1 Processor -- start 2024-05-09T03:15:53.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.224+0000 7f53cbc43640 1 -- start start 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53cbc43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 0x7f53c41e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53cbc43640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53c41615e0 0x7f53c40cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53cbc43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53c4164c70 0x7f53c40baad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53cbc43640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53c41619f0 con 0x7f53c40d2480 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53cbc43640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53c4160de0 con 0x7f53c4164c70 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53cbc43640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53c40d2dd0 con 0x7f53c41615e0 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 0x7f53c41e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 0x7f53c41e9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52192/0 (socket says 172.21.15.46:52192) 2024-05-09T03:15:53.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 -- 172.21.15.46:0/1414295830 learned_addr learned my addr 172.21.15.46:0/1414295830 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53ca1e0640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53c4164c70 0x7f53c40baad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c91de640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53c41615e0 0x7f53c40cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 -- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53c41615e0 msgr2=0x7f53c40cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53c41615e0 0x7f53c40cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 -- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53c4164c70 msgr2=0x7f53c40baad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53c4164c70 0x7f53c40baad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.225+0000 7f53c99df640 1 -- 172.21.15.46:0/1414295830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c41ea470 con 0x7f53c40d2480 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.226+0000 7f53c99df640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 0x7f53c41e9f30 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f53c000c3a0 tx=0x7f53c000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.226+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0017070 con 0x7f53c40d2480 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.226+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c00052d0 con 0x7f53c40d2480 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.226+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c41ea8f0 con 0x7f53c40d2480 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.226+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0003830 con 0x7f53c40d2480 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.226+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c40cd800 con 0x7f53c40d2480 2024-05-09T03:15:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.227+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c408cfd0 con 0x7f53c40d2480 2024-05-09T03:15:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.228+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f53c001a070 con 0x7f53c40d2480 2024-05-09T03:15:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.228+0000 7f53ba7fc640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53a8076300 0x7f53a8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.228+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f53c0096df0 con 0x7f53c40d2480 2024-05-09T03:15:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.228+0000 7f53c91de640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53a8076300 0x7f53a8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.229+0000 7f53c91de640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53a8076300 0x7f53a8078810 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f53b4034000 tx=0x7f53b402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.232+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f53c408cfd0 con 0x7f53c40d2480 2024-05-09T03:15:53.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.422+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53c406a270 con 0x7f53c40d2480 2024-05-09T03:15:53.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.424+0000 7f53ba7fc640 1 -- 172.21.15.46:0/1414295830 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16601 (secure 0 0 0) 0x7f53c0021020 con 0x7f53c40d2480 2024-05-09T03:15:53.425 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:53.425 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:15:52.455287+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:15:50.144137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-09T03:15:51.401672+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53a8076300 msgr2=0x7f53a8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53a8076300 0x7f53a8078810 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f53b4034000 tx=0x7f53b402e000 comp rx=0 tx=0).stop 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 msgr2=0x7f53c41e9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 0x7f53c41e9f30 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f53c000c3a0 tx=0x7f53c000c860 comp rx=0 tx=0).stop 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 shutdown_connections 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53c4164c70 0x7f53c40baad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53c41615e0 0x7f53c40cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53a8076300 0x7f53a8078810 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 --2- 172.21.15.46:0/1414295830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53c40d2480 0x7f53c41e9f30 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 >> 172.21.15.46:0/1414295830 conn(0x7f53c4073db0 msgr2=0x7f53c40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 shutdown_connections 2024-05-09T03:15:53.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.426+0000 7f53cbc43640 1 -- 172.21.15.46:0/1414295830 wait complete. 2024-05-09T03:15:53.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:15:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.552+0000 7f2de8e7e640 1 Processor -- start 2024-05-09T03:15:53.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.552+0000 7f2de8e7e640 1 -- start start 2024-05-09T03:15:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de8e7e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2de40b5210 0x7f2de41613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de8e7e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2de40d4800 0x7f2de40d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de8e7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de8e7e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2de40b5620 con 0x7f2de40d4800 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de8e7e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2de40b8110 con 0x7f2de40b5210 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de8e7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2de40d5280 con 0x7f2de40b77c0 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2de40d4800 0x7f2de40d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2de40d4800 0x7f2de40d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52202/0 (socket says 172.21.15.46:52202) 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 -- 172.21.15.46:0/1380430479 learned_addr learned my addr 172.21.15.46:0/1380430479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de259c640 1 --2- 172.21.15.46:0/1380430479 >> v2:172.21.15.88:3300/0 conn(0x7f2de40b5210 0x7f2de41613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de2d9d640 1 --2- 172.21.15.46:0/1380430479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 -- 172.21.15.46:0/1380430479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 msgr2=0x7f2de40b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 --2- 172.21.15.46:0/1380430479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 -- 172.21.15.46:0/1380430479 >> v2:172.21.15.88:3300/0 conn(0x7f2de40b5210 msgr2=0x7f2de41613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 --2- 172.21.15.46:0/1380430479 >> v2:172.21.15.88:3300/0 conn(0x7f2de40b5210 0x7f2de41613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.554+0000 7f2de1d9b640 1 -- 172.21.15.46:0/1380430479 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40d5560 con 0x7f2de40d4800 2024-05-09T03:15:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.555+0000 7f2de1d9b640 1 --2- 172.21.15.46:0/1380430479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40d4800 0x7f2de40d4c10 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2de408ed70 tx=0x7f2dd802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc56dd56e4624e87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.555+0000 7f2de159a640 1 -- 172.21.15.46:0/1380430479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8037070 con 0x7f2de40d4800 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.555+0000 7f2de159a640 1 -- 172.21.15.46:0/1380430479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd8032440 con 0x7f2de40d4800 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.555+0000 7f2de159a640 1 -- 172.21.15.46:0/1380430479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8032810 con 0x7f2de40d4800 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 -- 172.21.15.46:0/1380430479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40d4800 msgr2=0x7f2de40d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/1380430479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40d4800 0x7f2de40d4c10 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f2de408ed70 tx=0x7f2dd802a140 comp rx=0 tx=0).stop 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 -- 172.21.15.46:0/1380430479 shutdown_connections 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/1380430479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/1380430479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40d4800 0x7f2de40d4c10 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/1380430479 >> v2:172.21.15.88:3300/0 conn(0x7f2de40b5210 0x7f2de41613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 -- 172.21.15.46:0/1380430479 >> 172.21.15.46:0/1380430479 conn(0x7f2de40d29a0 msgr2=0x7f2de40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 -- 172.21.15.46:0/1380430479 shutdown_connections 2024-05-09T03:15:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.556+0000 7f2de8e7e640 1 -- 172.21.15.46:0/1380430479 wait complete. 2024-05-09T03:15:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.558+0000 7f2de8e7e640 1 Processor -- start 2024-05-09T03:15:53.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.558+0000 7f2de8e7e640 1 -- start start 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de8e7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 0x7f2de4156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de8e7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de8e7e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de40d4800 0x7f2de40cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de8e7e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40d0ae0 con 0x7f2de40b5210 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de8e7e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2de4160ea0 con 0x7f2de40d4800 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de8e7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2de4163e60 con 0x7f2de40b77c0 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 0x7f2de4156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 0x7f2de4156ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52214/0 (socket says 172.21.15.46:52214) 2024-05-09T03:15:53.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.559+0000 7f2de259c640 1 -- 172.21.15.46:0/208378345 learned_addr learned my addr 172.21.15.46:0/208378345 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de1d9b640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de2d9d640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de40d4800 0x7f2de40cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de259c640 1 -- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 msgr2=0x7f2de40cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de259c640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de259c640 1 -- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de40d4800 msgr2=0x7f2de40cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de259c640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de40d4800 0x7f2de40cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de259c640 1 -- 172.21.15.46:0/208378345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de41fabe0 con 0x7f2de40b5210 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de259c640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 0x7f2de4156ae0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400e8a0 tx=0x7f2dd400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4017070 con 0x7f2de40b5210 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd400ee90 con 0x7f2de40b5210 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de4157020 con 0x7f2de40b5210 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400cb80 con 0x7f2de40b5210 2024-05-09T03:15:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.560+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de41ea0d0 con 0x7f2de40b5210 2024-05-09T03:15:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.561+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de406a270 con 0x7f2de40b5210 2024-05-09T03:15:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.562+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2dd401a070 con 0x7f2de40b5210 2024-05-09T03:15:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.562+0000 7f2dd2ffd640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2dc40762d0 0x7f2dc40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.563+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f2dd4097410 con 0x7f2de40b5210 2024-05-09T03:15:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.563+0000 7f2de1d9b640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2dc40762d0 0x7f2dc40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.563+0000 7f2de1d9b640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2dc40762d0 0x7f2dc40787e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.567+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2dd4060d80 con 0x7f2de40b5210 2024-05-09T03:15:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.760+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2de40d40d0 con 0x7f2de40b5210 2024-05-09T03:15:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.761+0000 7f2dd2ffd640 1 -- 172.21.15.46:0/208378345 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16601 (secure 0 0 0) 0x7f2dd4060680 con 0x7f2de40b5210 2024-05-09T03:15:53.762 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:53.762 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:15:52.455287+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:15:50.144137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-09T03:15:51.401672+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2dc40762d0 msgr2=0x7f2dc40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2dc40762d0 0x7f2dc40787e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd802e000 comp rx=0 tx=0).stop 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 msgr2=0x7f2de4156ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 0x7f2de4156ae0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400e8a0 tx=0x7f2dd400ed60 comp rx=0 tx=0).stop 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 shutdown_connections 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2de40d4800 0x7f2de40cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2de40b77c0 0x7f2de40cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2dc40762d0 0x7f2dc40787e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 --2- 172.21.15.46:0/208378345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2de40b5210 0x7f2de4156ae0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 >> 172.21.15.46:0/208378345 conn(0x7f2de40d29a0 msgr2=0x7f2de40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 shutdown_connections 2024-05-09T03:15:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.763+0000 7f2de8e7e640 1 -- 172.21.15.46:0/208378345 wait complete. 2024-05-09T03:15:53.776 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:15:53.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.893+0000 7f7f94e61640 1 Processor -- start 2024-05-09T03:15:53.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.893+0000 7f7f94e61640 1 -- start start 2024-05-09T03:15:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f94e61640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f900d1110 0x7f7f900d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f94e61640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f900b92c0 con 0x7f7f900d1920 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f94e61640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f900b2f10 con 0x7f7f900d1d30 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f94e61640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7f900b9560 con 0x7f7f900d1110 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f8e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f900d1110 0x7f7f900d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f8dd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7f900d1920 0x7f7f900b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59626/0 (socket says 172.21.15.46:59626) 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.894+0000 7f7f8dd9b640 1 -- 172.21.15.46:0/3207887147 learned_addr learned my addr 172.21.15.46:0/3207887147 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.895+0000 7f7f8e59c640 1 -- 172.21.15.46:0/3207887147 >> v1:172.21.15.88:6789/0 conn(0x7f7f900d1d30 legacy=0x7f7f900b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.895+0000 7f7f8e59c640 1 -- 172.21.15.46:0/3207887147 >> v1:172.21.15.46:6789/0 conn(0x7f7f900d1920 legacy=0x7f7f900b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.895+0000 7f7f8e59c640 1 -- 172.21.15.46:0/3207887147 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f900cfb20 con 0x7f7f900d1110 2024-05-09T03:15:53.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.895+0000 7f7f8d59a640 1 -- 172.21.15.46:0/3207887147 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1791646023 0 0) 0x7f7f900b92c0 con 0x7f7f900d1920 2024-05-09T03:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.895+0000 7f7f8e59c640 1 --2- 172.21.15.46:0/3207887147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1110 0x7f7f900d0020 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009d90 tx=0x7f7f8802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2252464c05c1dc79 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f8d59a640 1 -- 172.21.15.46:0/3207887147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88031600 con 0x7f7f900d1110 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f8d59a640 1 -- 172.21.15.46:0/3207887147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f880317a0 con 0x7f7f900d1110 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f8d59a640 1 -- 172.21.15.46:0/3207887147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88031b70 con 0x7f7f900d1110 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f94e61640 1 -- 172.21.15.46:0/3207887147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1110 msgr2=0x7f7f900d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f94e61640 1 --2- 172.21.15.46:0/3207887147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1110 0x7f7f900d0020 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009d90 tx=0x7f7f8802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f94e61640 1 -- 172.21.15.46:0/3207887147 shutdown_connections 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.896+0000 7f7f94e61640 1 --2- 172.21.15.46:0/3207887147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1110 0x7f7f900d0020 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.897+0000 7f7f94e61640 1 -- 172.21.15.46:0/3207887147 >> 172.21.15.46:0/3207887147 conn(0x7f7f900d05f0 msgr2=0x7f7f900d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.897+0000 7f7f94e61640 1 -- 172.21.15.46:0/3207887147 shutdown_connections 2024-05-09T03:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.897+0000 7f7f94e61640 1 -- 172.21.15.46:0/3207887147 wait complete. 2024-05-09T03:15:53.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.899+0000 7f7f94e61640 1 Processor -- start 2024-05-09T03:15:53.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.899+0000 7f7f94e61640 1 -- start start 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f94e61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 0x7f7f9015dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f94e61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f900d1d30 0x7f7f900cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f94e61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f901f86d0 0x7f7f901e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f94e61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9009d030 con 0x7f7f900d1920 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f94e61640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9009ceb0 con 0x7f7f901f86d0 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f94e61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7f900b8e00 con 0x7f7f900d1d30 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 0x7f7f9015dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 0x7f7f9015dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52238/0 (socket says 172.21.15.46:52238) 2024-05-09T03:15:53.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 -- 172.21.15.46:0/444479832 learned_addr learned my addr 172.21.15.46:0/444479832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8dd9b640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f900d1d30 0x7f7f900cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8ed9d640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f901f86d0 0x7f7f901e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 -- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f900d1d30 msgr2=0x7f7f900cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f900d1d30 0x7f7f900cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 -- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f901f86d0 msgr2=0x7f7f901e8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f901f86d0 0x7f7f901e8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8e59c640 1 -- 172.21.15.46:0/444479832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f900cdd90 con 0x7f7f900d1920 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.900+0000 7f7f8ed9d640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f901f86d0 0x7f7f901e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.901+0000 7f7f8e59c640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 0x7f7f9015dbd0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009d90 tx=0x7f7f88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.901+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88040070 con 0x7f7f900d1920 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.901+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f88030be0 con 0x7f7f900d1920 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.901+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f88045460 con 0x7f7f900d1920 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.901+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f901f7bf0 con 0x7f7f900d1920 2024-05-09T03:15:53.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.901+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f90160ed0 con 0x7f7f900d1920 2024-05-09T03:15:53.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.902+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f90161130 con 0x7f7f900d1920 2024-05-09T03:15:53.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.902+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7f88039040 con 0x7f7f900d1920 2024-05-09T03:15:53.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.903+0000 7f7f86ffd640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f70076300 0x7f7f70078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:53.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.903+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f7f880b8e60 con 0x7f7f900d1920 2024-05-09T03:15:53.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.903+0000 7f7f8dd9b640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f70076300 0x7f7f70078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:53.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.904+0000 7f7f8dd9b640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f70076300 0x7f7f70078810 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7f900a1350 tx=0x7f7f78009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:53.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:53.908+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7f88082190 con 0x7f7f900d1920 2024-05-09T03:15:54.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:54.138+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f7f9006a270 con 0x7f7f900d1920 2024-05-09T03:15:54.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-05-09T03:15:54.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-05-09T03:15:54.913 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-05-09T03:15:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.080+0000 7f7f86ffd640 1 -- 172.21.15.46:0/444479832 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v84) ==== 118+0+0 (secure 0 0 0) 0x7f7f88086490 con 0x7f7f900d1920 2024-05-09T03:15:55.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f70076300 msgr2=0x7f7f70078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f70076300 0x7f7f70078810 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7f900a1350 tx=0x7f7f78009040 comp rx=0 tx=0).stop 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 msgr2=0x7f7f9015dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 0x7f7f9015dbd0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009d90 tx=0x7f7f88004a40 comp rx=0 tx=0).stop 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 shutdown_connections 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f901f86d0 0x7f7f901e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f900d1d30 0x7f7f900cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f70076300 0x7f7f70078810 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 --2- 172.21.15.46:0/444479832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f900d1920 0x7f7f9015dbd0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 >> 172.21.15.46:0/444479832 conn(0x7f7f900d05f0 msgr2=0x7f7f9009bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 shutdown_connections 2024-05-09T03:15:55.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.083+0000 7f7f94e61640 1 -- 172.21.15.46:0/444479832 wait complete. 2024-05-09T03:15:55.098 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:15:55.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:15:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.222+0000 7f880317a640 1 Processor -- start 2024-05-09T03:15:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.223+0000 7f880317a640 1 -- start start 2024-05-09T03:15:55.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f880317a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b5140 0x7f87fc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:55.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f880317a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87fc0b7ca0 0x7f87fc0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:55.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f880317a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87fc0b80b0 con 0x7f87fc0d3df0 2024-05-09T03:15:55.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f880317a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f87fc0d3480 con 0x7f87fc0b7ca0 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f880317a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f87fc0b5550 con 0x7f87fc0b5140 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f87fbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87fc0b7ca0 0x7f87fc0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f8801717640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f87fc0d3df0 0x7f87fc163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59640/0 (socket says 172.21.15.46:59640) 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f8801717640 1 -- 172.21.15.46:0/1683381550 learned_addr learned my addr 172.21.15.46:0/1683381550 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f8800f16640 1 --2- 172.21.15.46:0/1683381550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b5140 0x7f87fc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f87fbfff640 1 -- 172.21.15.46:0/1683381550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b5140 msgr2=0x7f87fc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f87fbfff640 1 --2- 172.21.15.46:0/1683381550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b5140 0x7f87fc0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.224+0000 7f87fbfff640 1 -- 172.21.15.46:0/1683381550 >> v1:172.21.15.46:6789/0 conn(0x7f87fc0d3df0 legacy=0x7f87fc163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.225+0000 7f87fbfff640 1 -- 172.21.15.46:0/1683381550 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87fc0d44e0 con 0x7f87fc0b7ca0 2024-05-09T03:15:55.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.225+0000 7f87fb7fe640 1 -- 172.21.15.46:0/1683381550 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3409029494 0 0) 0x7f87f4001490 con 0x7f87fc0d3df0 2024-05-09T03:15:55.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.225+0000 7f87fbfff640 1 --2- 172.21.15.46:0/1683381550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b7ca0 0x7f87fc0d38b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f87f00097a0 tx=0x7f87f002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79822754b9c1a274 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.225+0000 7f87fb7fe640 1 -- 172.21.15.46:0/1683381550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f0037070 con 0x7f87fc0b7ca0 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.225+0000 7f87fb7fe640 1 -- 172.21.15.46:0/1683381550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87f0032440 con 0x7f87fc0b7ca0 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.225+0000 7f87fb7fe640 1 -- 172.21.15.46:0/1683381550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f0032810 con 0x7f87fc0b7ca0 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 -- 172.21.15.46:0/1683381550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b7ca0 msgr2=0x7f87fc0d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 --2- 172.21.15.46:0/1683381550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b7ca0 0x7f87fc0d38b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f87f00097a0 tx=0x7f87f002a210 comp rx=0 tx=0).stop 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 -- 172.21.15.46:0/1683381550 shutdown_connections 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 --2- 172.21.15.46:0/1683381550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b7ca0 0x7f87fc0d38b0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 --2- 172.21.15.46:0/1683381550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b5140 0x7f87fc0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 -- 172.21.15.46:0/1683381550 >> 172.21.15.46:0/1683381550 conn(0x7f87fc08d3f0 msgr2=0x7f87fc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 -- 172.21.15.46:0/1683381550 shutdown_connections 2024-05-09T03:15:55.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.226+0000 7f880317a640 1 -- 172.21.15.46:0/1683381550 wait complete. 2024-05-09T03:15:55.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.228+0000 7f880317a640 1 Processor -- start 2024-05-09T03:15:55.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.228+0000 7f880317a640 1 -- start start 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.229+0000 7f880317a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 0x7f87fc158a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.229+0000 7f880317a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b7ca0 0x7f87fc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.229+0000 7f880317a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87fc0d3df0 0x7f87fc1488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.229+0000 7f880317a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f87fc0d4d80 con 0x7f87fc0b5140 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.229+0000 7f880317a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f87fc0d4aa0 con 0x7f87fc0d3df0 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.229+0000 7f880317a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f87fc0d4200 con 0x7f87fc0b7ca0 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 0x7f87fc158a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 0x7f87fc158a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52248/0 (socket says 172.21.15.46:52248) 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 -- 172.21.15.46:0/93793071 learned_addr learned my addr 172.21.15.46:0/93793071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:55.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8801717640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87fc0d3df0 0x7f87fc1488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f87fbfff640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b7ca0 0x7f87fc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 -- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b7ca0 msgr2=0x7f87fc15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b7ca0 0x7f87fc15a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 -- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87fc0d3df0 msgr2=0x7f87fc1488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87fc0d3df0 0x7f87fc1488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 -- 172.21.15.46:0/93793071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87fc15ada0 con 0x7f87fc0b5140 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8801717640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87fc0d3df0 0x7f87fc1488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f87fbfff640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b7ca0 0x7f87fc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f8800f16640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 0x7f87fc158a70 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00d620 tx=0x7f87ec00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00b620 con 0x7f87fc0b5140 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87ec019070 con 0x7f87fc0b5140 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87fc1597c0 con 0x7f87fc0b5140 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00ba00 con 0x7f87fc0b5140 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.230+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87fc159d80 con 0x7f87fc0b5140 2024-05-09T03:15:55.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.231+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87fc0b8e80 con 0x7f87fc0b5140 2024-05-09T03:15:55.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.232+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f87ec00bba0 con 0x7f87fc0b5140 2024-05-09T03:15:55.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.232+0000 7f87f97fa640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87e8076300 0x7f87e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:55.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.233+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f87ec097ad0 con 0x7f87fc0b5140 2024-05-09T03:15:55.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.233+0000 7f87fbfff640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87e8076300 0x7f87e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:55.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.233+0000 7f87fbfff640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87e8076300 0x7f87e8078810 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f87f0034000 tx=0x7f87f002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:55.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.237+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f87fc0b8e80 con 0x7f87fc0b5140 2024-05-09T03:15:55.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:55.467+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f87fc06a270 con 0x7f87fc0b5140 2024-05-09T03:15:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.081+0000 7f87f97fa640 1 -- 172.21.15.46:0/93793071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v85) ==== 122+0+0 (secure 0 0 0) 0x7f87ec0613a0 con 0x7f87fc0b5140 2024-05-09T03:15:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87e8076300 msgr2=0x7f87e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87e8076300 0x7f87e8078810 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f87f0034000 tx=0x7f87f002e000 comp rx=0 tx=0).stop 2024-05-09T03:15:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 msgr2=0x7f87fc158a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 0x7f87fc158a70 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00d620 tx=0x7f87ec00dae0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 shutdown_connections 2024-05-09T03:15:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87fc0d3df0 0x7f87fc1488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87fc0b7ca0 0x7f87fc15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87e8076300 0x7f87e8078810 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 --2- 172.21.15.46:0/93793071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87fc0b5140 0x7f87fc158a70 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 >> 172.21.15.46:0/93793071 conn(0x7f87fc08d3f0 msgr2=0x7f87fc15eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 shutdown_connections 2024-05-09T03:15:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.084+0000 7f880317a640 1 -- 172.21.15.46:0/93793071 wait complete. 2024-05-09T03:15:56.098 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:15:56.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:56.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.219+0000 7fb9ef029640 1 Processor -- start 2024-05-09T03:15:56.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.219+0000 7fb9ef029640 1 -- start start 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.220+0000 7fb9ef029640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9e80b5210 0x7fb9e81613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.220+0000 7fb9ef029640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e80d5280 con 0x7fb9e80d4800 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.220+0000 7fb9ef029640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e80d5560 con 0x7fb9e80b7ed0 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.220+0000 7fb9ef029640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb9e80b5620 con 0x7fb9e80b5210 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.220+0000 7fb9e7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb9e80d4800 0x7fb9e80b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40024/0 (socket says 172.21.15.46:40024) 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.220+0000 7fb9e7fff640 1 -- 172.21.15.46:0/2831291140 learned_addr learned my addr 172.21.15.46:0/2831291140 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/2831291140 >> v2:172.21.15.88:3300/0 conn(0x7fb9e80b5210 0x7fb9e81613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258903080 0 0) 0x7fb9e80d5280 con 0x7fb9e80d4800 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9d4003520 con 0x7fb9e80d4800 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2399190587 0 0) 0x7fb9d4003520 con 0x7fb9e80d4800 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 >> v2:172.21.15.88:3300/0 conn(0x7fb9e80b5210 msgr2=0x7fb9e81613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:56.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 --2- 172.21.15.46:0/2831291140 >> v2:172.21.15.88:3300/0 conn(0x7fb9e80b5210 0x7fb9e81613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 >> v1:172.21.15.88:6789/0 conn(0x7fb9e80b7ed0 legacy=0x7fb9e8163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e8160bc0 con 0x7fb9e80d4800 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fb9d8001a70 con 0x7fb9e80d4800 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 >> v1:172.21.15.46:6789/0 conn(0x7fb9e80d4800 legacy=0x7fb9e80b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9d4003a50 0x7fb9d4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4004800 0x7fb9d4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d4005250 0x7fb9d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e80b5620 con 0x7fb9d4003a50 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d400c1e0 con 0x7fb9d4005250 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d400c3e0 con 0x7fb9d4004800 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9d4003a50 0x7fb9d4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9e7fff640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4004800 0x7fb9d4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ed5c6640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d4005250 0x7fb9d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ecdc5640 1 -- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4004800 msgr2=0x7fb9d4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4004800 0x7fb9d4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ecdc5640 1 -- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d4005250 msgr2=0x7fb9d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.221+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d4005250 0x7fb9d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9ecdc5640 1 -- 172.21.15.46:0/2831291140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d4003520 con 0x7fb9d4003a50 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9ed5c6640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d4005250 0x7fb9d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9e7fff640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4004800 0x7fb9d4004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9ed5c6640 1 -- 172.21.15.46:0/2831291140 reap_dead start 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9d4003a50 0x7fb9d4003ec0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009ac0 tx=0x7fb9d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e0fafe3b7c4975 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0036070 con 0x7fb9d4003a50 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d0031400 con 0x7fb9d4003a50 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.222+0000 7fb9e77fe640 1 -- 172.21.15.46:0/2831291140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0009690 con 0x7fb9d4003a50 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 -- 172.21.15.46:0/2831291140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9d4003a50 msgr2=0x7fb9d4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9d4003a50 0x7fb9d4003ec0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009ac0 tx=0x7fb9d4005830 comp rx=0 tx=0).stop 2024-05-09T03:15:56.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 -- 172.21.15.46:0/2831291140 shutdown_connections 2024-05-09T03:15:56.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 --2- 172.21.15.46:0/2831291140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9d4003a50 0x7fb9d4003ec0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 -- 172.21.15.46:0/2831291140 >> 172.21.15.46:0/2831291140 conn(0x7fb9e80d29a0 msgr2=0x7fb9e80d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:56.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 -- 172.21.15.46:0/2831291140 shutdown_connections 2024-05-09T03:15:56.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.223+0000 7fb9ef029640 1 -- 172.21.15.46:0/2831291140 wait complete. 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.225+0000 7fb9ef029640 1 Processor -- start 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.225+0000 7fb9ef029640 1 -- start start 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.226+0000 7fb9ef029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4003a50 0x7fb9e814be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.226+0000 7fb9ef029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 0x7fb9e815e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.226+0000 7fb9ef029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e815e6e0 0x7fb9e8149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.226+0000 7fb9ef029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e80d0ae0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.226+0000 7fb9ef029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e81609a0 con 0x7fb9e815e6e0 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.226+0000 7fb9ef029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e80d4c10 con 0x7fb9d4003a50 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 0x7fb9e815e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 0x7fb9e815e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55312/0 (socket says 172.21.15.46:55312) 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 -- 172.21.15.46:0/3246114189 learned_addr learned my addr 172.21.15.46:0/3246114189 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4003a50 0x7fb9e814be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9ed5c6640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e815e6e0 0x7fb9e8149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 -- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4003a50 msgr2=0x7fb9e814be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4003a50 0x7fb9e814be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 -- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e815e6e0 msgr2=0x7fb9e8149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e815e6e0 0x7fb9e8149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 -- 172.21.15.46:0/3246114189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e814c3c0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e7fff640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 0x7fb9e815e1a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80079e0 tx=0x7fb9d800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8007ad0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d8018070 con 0x7fb9e81e9e50 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e815eaf0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8013620 con 0x7fb9e81e9e50 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.227+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e81fa010 con 0x7fb9e81e9e50 2024-05-09T03:15:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.228+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e806a270 con 0x7fb9e81e9e50 2024-05-09T03:15:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.229+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb9d80137c0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.230+0000 7fb9e57fa640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9c80762d0 0x7fb9c80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.230+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fb9d8097c30 con 0x7fb9e81e9e50 2024-05-09T03:15:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.230+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9c80762d0 0x7fb9c80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.230+0000 7fb9ecdc5640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9c80762d0 0x7fb9c80787e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0033000 tx=0x7fb9d0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.234+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb9e815eaf0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.460+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9e80c9610 con 0x7fb9e81e9e50 2024-05-09T03:15:56.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.461+0000 7fb9e57fa640 1 -- 172.21.15.46:0/3246114189 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5180 (secure 0 0 0) 0x7fb9e814c3c0 con 0x7fb9e81e9e50 2024-05-09T03:15:56.462 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:56.462 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:15:55.468035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6888":{"gid":6888,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/2347304023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2347304023},{"type":"v1","addr":"172.21.15.46:6834","nonce":2347304023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6888,"qdb_cluster":[6888]},"id":6}]} 2024-05-09T03:15:56.462 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 85 2024-05-09T03:15:56.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.463+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9c80762d0 msgr2=0x7fb9c80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.463+0000 7fb9ef029640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9c80762d0 0x7fb9c80787e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0033000 tx=0x7fb9d0008030 comp rx=0 tx=0).stop 2024-05-09T03:15:56.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.463+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 msgr2=0x7fb9e815e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.463+0000 7fb9ef029640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 0x7fb9e815e1a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80079e0 tx=0x7fb9d800e530 comp rx=0 tx=0).stop 2024-05-09T03:15:56.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 shutdown_connections 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e815e6e0 0x7fb9e8149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb9c80762d0 0x7fb9c80787e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9e81e9e50 0x7fb9e815e1a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 --2- 172.21.15.46:0/3246114189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb9d4003a50 0x7fb9e814be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 >> 172.21.15.46:0/3246114189 conn(0x7fb9e80d29a0 msgr2=0x7fb9e80ba550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 shutdown_connections 2024-05-09T03:15:56.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.464+0000 7fb9ef029640 1 -- 172.21.15.46:0/3246114189 wait complete. 2024-05-09T03:15:56.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:15:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.589+0000 7fa465214640 1 Processor -- start 2024-05-09T03:15:56.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.589+0000 7fa465214640 1 -- start start 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa465214640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b5210 0x7fa4601613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa465214640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4600d4800 0x7fa4600d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa465214640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa4600b77c0 0x7fa4600b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa465214640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa4600b5620 con 0x7fa4600d4800 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa465214640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa4600b8110 con 0x7fa4600b77c0 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa465214640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa4600d5280 con 0x7fa4600b5210 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4600d4800 0x7fa4600d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4600d4800 0x7fa4600d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55314/0 (socket says 172.21.15.46:55314) 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 -- 172.21.15.46:0/1773092181 learned_addr learned my addr 172.21.15.46:0/1773092181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45ed9d640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b5210 0x7fa4601613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45f59e640 1 --2- 172.21.15.46:0/1773092181 >> v2:172.21.15.88:3300/0 conn(0x7fa4600b77c0 0x7fa4600b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 -- 172.21.15.46:0/1773092181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b5210 msgr2=0x7fa4601613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b5210 0x7fa4601613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 -- 172.21.15.46:0/1773092181 >> v2:172.21.15.88:3300/0 conn(0x7fa4600b77c0 msgr2=0x7fa4600b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 --2- 172.21.15.46:0/1773092181 >> v2:172.21.15.88:3300/0 conn(0x7fa4600b77c0 0x7fa4600b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45e59c640 1 -- 172.21.15.46:0/1773092181 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4600d5560 con 0x7fa4600d4800 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45ed9d640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b5210 0x7fa4601613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.591+0000 7fa45f59e640 1 --2- 172.21.15.46:0/1773092181 >> v2:172.21.15.88:3300/0 conn(0x7fa4600b77c0 0x7fa4600b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.592+0000 7fa45e59c640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600d4c10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa4600d0660 tx=0x7fa45402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcad00059107e276 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.592+0000 7fa45dd9b640 1 -- 172.21.15.46:0/1773092181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454037070 con 0x7fa4600d4800 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.592+0000 7fa45dd9b640 1 -- 172.21.15.46:0/1773092181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa454032440 con 0x7fa4600d4800 2024-05-09T03:15:56.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.592+0000 7fa45dd9b640 1 -- 172.21.15.46:0/1773092181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454032810 con 0x7fa4600d4800 2024-05-09T03:15:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 -- 172.21.15.46:0/1773092181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 msgr2=0x7fa4600d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600d4c10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa4600d0660 tx=0x7fa45402a140 comp rx=0 tx=0).stop 2024-05-09T03:15:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 -- 172.21.15.46:0/1773092181 shutdown_connections 2024-05-09T03:15:56.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 --2- 172.21.15.46:0/1773092181 >> v2:172.21.15.88:3300/0 conn(0x7fa4600b77c0 0x7fa4600b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600d4c10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 --2- 172.21.15.46:0/1773092181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b5210 0x7fa4601613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 -- 172.21.15.46:0/1773092181 >> 172.21.15.46:0/1773092181 conn(0x7fa4600d29a0 msgr2=0x7fa4600d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 -- 172.21.15.46:0/1773092181 shutdown_connections 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.593+0000 7fa465214640 1 -- 172.21.15.46:0/1773092181 wait complete. 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.595+0000 7fa465214640 1 Processor -- start 2024-05-09T03:15:56.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.595+0000 7fa465214640 1 -- start start 2024-05-09T03:15:56.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4600b5210 0x7fa4600cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b77c0 0x7fa4600cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4600d0ae0 con 0x7fa4600d4800 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa460160ea0 con 0x7fa4600b5210 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa460163e60 con 0x7fa4600b77c0 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55326/0 (socket says 172.21.15.46:55326) 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 -- 172.21.15.46:0/273857136 learned_addr learned my addr 172.21.15.46:0/273857136 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45ed9d640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4600b5210 0x7fa4600cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45e59c640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b77c0 0x7fa4600cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 -- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b77c0 msgr2=0x7fa4600cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b77c0 0x7fa4600cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 -- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4600b5210 msgr2=0x7fa4600cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4600b5210 0x7fa4600cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 -- 172.21.15.46:0/273857136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4600ce850 con 0x7fa4600d4800 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45ed9d640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4600b5210 0x7fa4600cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa45f59e640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600cfb50 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa45800cd40 tx=0x7fa45800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458003b90 con 0x7fa4600d4800 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa458003d50 con 0x7fa4600d4800 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.597+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4600ceb30 con 0x7fa4600d4800 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.598+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458004260 con 0x7fa4600d4800 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.598+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4600d0090 con 0x7fa4600d4800 2024-05-09T03:15:56.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.598+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4600ce5d0 con 0x7fa4600d4800 2024-05-09T03:15:56.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.599+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa458004400 con 0x7fa4600d4800 2024-05-09T03:15:56.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.600+0000 7fa4477fe640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa44c0762d0 0x7fa44c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.600+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fa458097f40 con 0x7fa4600d4800 2024-05-09T03:15:56.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.600+0000 7fa45ed9d640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa44c0762d0 0x7fa44c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.601+0000 7fa45ed9d640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa44c0762d0 0x7fa44c0787e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.604+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa4600ce5d0 con 0x7fa4600d4800 2024-05-09T03:15:56.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.797+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa46006a270 con 0x7fa4600d4800 2024-05-09T03:15:56.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.798+0000 7fa4477fe640 1 -- 172.21.15.46:0/273857136 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+16588 (secure 0 0 0) 0x7fa458014030 con 0x7fa4600d4800 2024-05-09T03:15:56.799 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:56.799 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":82,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:15:54.064750+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:15:50.144137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-09T03:15:51.401672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:15:56.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa44c0762d0 msgr2=0x7fa44c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa44c0762d0 0x7fa44c0787e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450008040 comp rx=0 tx=0).stop 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 msgr2=0x7fa4600cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600cfb50 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa45800cd40 tx=0x7fa45800a650 comp rx=0 tx=0).stop 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 shutdown_connections 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa44c0762d0 0x7fa44c0787e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4600d4800 0x7fa4600cfb50 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4600b77c0 0x7fa4600cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 --2- 172.21.15.46:0/273857136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4600b5210 0x7fa4600cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 >> 172.21.15.46:0/273857136 conn(0x7fa4600d29a0 msgr2=0x7fa4600d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 shutdown_connections 2024-05-09T03:15:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.801+0000 7fa465214640 1 -- 172.21.15.46:0/273857136 wait complete. 2024-05-09T03:15:56.813 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:15:56.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:15:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.927+0000 7f5e452d1640 1 Processor -- start 2024-05-09T03:15:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.927+0000 7f5e452d1640 1 -- start start 2024-05-09T03:15:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e452d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e400b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e452d1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 0x7f5e4006d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e452d1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e400d2b80 con 0x7f5e400d0480 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e452d1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5e400d2e60 con 0x7f5e400cf6b0 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e452d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5e400d3140 con 0x7f5e400b92e0 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3effd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e400b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 0x7f5e4006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3e7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5e400d0480 0x7f5e400d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:32820/0 (socket says 172.21.15.46:32820) 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3e7fc640 1 -- 172.21.15.46:0/3013560552 learned_addr learned my addr 172.21.15.46:0/3013560552 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3effd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e400b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:42966/0 (socket says 172.21.15.46:42966) 2024-05-09T03:15:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 0x7f5e4006d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:35608/0 (socket says 172.21.15.46:35608) 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3effd640 1 -- 172.21.15.46:0/3013560552 >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 msgr2=0x7f5e4006d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3effd640 1 --2- 172.21.15.46:0/3013560552 >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 0x7f5e4006d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3effd640 1 -- 172.21.15.46:0/3013560552 >> v1:172.21.15.88:6789/0 conn(0x7f5e400d0480 legacy=0x7f5e400d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.929+0000 7f5e3effd640 1 -- 172.21.15.46:0/3013560552 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e400d3420 con 0x7f5e400b92e0 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.930+0000 7f5e3f7fe640 1 --2- 172.21.15.46:0/3013560552 >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 0x7f5e4006d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.930+0000 7f5e3effd640 1 --2- 172.21.15.46:0/3013560552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e400b51a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097a0 tx=0x7f5e3002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aab5d9a666f3d9ed server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.930+0000 7f5e3dffb640 1 -- 172.21.15.46:0/3013560552 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30037070 con 0x7f5e400b92e0 2024-05-09T03:15:56.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.930+0000 7f5e3dffb640 1 -- 172.21.15.46:0/3013560552 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3002ae50 con 0x7f5e400b92e0 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.930+0000 7f5e3dffb640 1 -- 172.21.15.46:0/3013560552 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30032660 con 0x7f5e400b92e0 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 -- 172.21.15.46:0/3013560552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 msgr2=0x7f5e400b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 --2- 172.21.15.46:0/3013560552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e400b51a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5e300097a0 tx=0x7f5e3002a100 comp rx=0 tx=0).stop 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 -- 172.21.15.46:0/3013560552 shutdown_connections 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 --2- 172.21.15.46:0/3013560552 >> v2:172.21.15.88:3300/0 conn(0x7f5e400cf6b0 0x7f5e4006d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 --2- 172.21.15.46:0/3013560552 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e400b51a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 -- 172.21.15.46:0/3013560552 >> 172.21.15.46:0/3013560552 conn(0x7f5e400d3c00 msgr2=0x7f5e400d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 -- 172.21.15.46:0/3013560552 shutdown_connections 2024-05-09T03:15:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.931+0000 7f5e452d1640 1 -- 172.21.15.46:0/3013560552 wait complete. 2024-05-09T03:15:56.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.933+0000 7f5e452d1640 1 Processor -- start 2024-05-09T03:15:56.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.933+0000 7f5e452d1640 1 -- start start 2024-05-09T03:15:56.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e452d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e401fa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e452d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e400cf6b0 0x7f5e40158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e452d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 0x7f5e40075d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e452d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4006e060 con 0x7f5e400d0480 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e452d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5e4006dee0 con 0x7f5e400cf6b0 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e452d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4006e1e0 con 0x7f5e400b92e0 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 0x7f5e40075d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 0x7f5e40075d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55330/0 (socket says 172.21.15.46:55330) 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/1104069343 learned_addr learned my addr 172.21.15.46:0/1104069343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3e7fc640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e400cf6b0 0x7f5e40158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3effd640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e401fa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 msgr2=0x7f5e401fa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e401fa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e400cf6b0 msgr2=0x7f5e40158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e400cf6b0 0x7f5e40158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 -- 172.21.15.46:0/1104069343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e400cb020 con 0x7f5e400d0480 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e3f7fe640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 0x7f5e40075d40 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800e8b0 tx=0x7f5e3800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.935+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e38017070 con 0x7f5e400d0480 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.936+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e38004020 con 0x7f5e400d0480 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.936+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e400cb300 con 0x7f5e400d0480 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.936+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e380043f0 con 0x7f5e400d0480 2024-05-09T03:15:56.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.936+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e400cb8c0 con 0x7f5e400d0480 2024-05-09T03:15:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.937+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e4006a270 con 0x7f5e400d0480 2024-05-09T03:15:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.937+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5e3800c940 con 0x7f5e400d0480 2024-05-09T03:15:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.938+0000 7f5e277fe640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5e2c076300 0x7f5e2c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.938+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f5e38097700 con 0x7f5e400d0480 2024-05-09T03:15:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.938+0000 7f5e3effd640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5e2c076300 0x7f5e2c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:56.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.939+0000 7f5e3effd640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5e2c076300 0x7f5e2c078810 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5e30034000 tx=0x7f5e3002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:56.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:56.943+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5e38060990 con 0x7f5e400d0480 2024-05-09T03:15:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.130+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e400d39c0 con 0x7f5e2c076300 2024-05-09T03:15:57.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.137+0000 7f5e277fe640 1 -- 172.21.15.46:0/1104069343 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252070 (secure 0 0 0) 0x7f5e3007f8c0 con 0x7f5e2c076300 2024-05-09T03:15:57.140 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5e2c076300 msgr2=0x7f5e2c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5e2c076300 0x7f5e2c078810 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5e30034000 tx=0x7f5e3002e000 comp rx=0 tx=0).stop 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 msgr2=0x7f5e40075d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 0x7f5e40075d40 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5e3800e8b0 tx=0x7f5e3800ed70 comp rx=0 tx=0).stop 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 shutdown_connections 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5e2c076300 0x7f5e2c078810 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e400d0480 0x7f5e40075d40 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e400cf6b0 0x7f5e40158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 --2- 172.21.15.46:0/1104069343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5e400b92e0 0x7f5e401fa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 >> 172.21.15.46:0/1104069343 conn(0x7f5e400d3c00 msgr2=0x7f5e400b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 shutdown_connections 2024-05-09T03:15:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.140+0000 7f5e452d1640 1 -- 172.21.15.46:0/1104069343 wait complete. 2024-05-09T03:15:57.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-05-09T03:15:56.593628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254384,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720336,"statfs":{"total":773094113280,"available":772833624064,"internally_reserved":0,"allocated":9314304,"data_stored":4638988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.551088"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074096+0000","last_change":"2024-05-09T03:15:53.069947+0000","last_active":"2024-05-09T03:15:54.074096+0000","last_peered":"2024-05-09T03:15:54.074096+0000","last_clean":"2024-05-09T03:15:54.074096+0000","last_became_active":"2024-05-09T03:15:53.069585+0000","last_became_peered":"2024-05-09T03:15:53.069585+0000","last_unstale":"2024-05-09T03:15:54.074096+0000","last_undegraded":"2024-05-09T03:15:54.074096+0000","last_fullsized":"2024-05-09T03:15:54.074096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:04.173563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063352+0000","last_change":"2024-05-09T03:15:53.076051+0000","last_active":"2024-05-09T03:15:55.063352+0000","last_peered":"2024-05-09T03:15:55.063352+0000","last_clean":"2024-05-09T03:15:55.063352+0000","last_became_active":"2024-05-09T03:15:53.074901+0000","last_became_peered":"2024-05-09T03:15:53.074901+0000","last_unstale":"2024-05-09T03:15:55.063352+0000","last_undegraded":"2024-05-09T03:15:55.063352+0000","last_fullsized":"2024-05-09T03:15:55.063352+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:23.219622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061484+0000","last_change":"2024-05-09T03:15:53.074723+0000","last_active":"2024-05-09T03:15:55.061484+0000","last_peered":"2024-05-09T03:15:55.061484+0000","last_clean":"2024-05-09T03:15:55.061484+0000","last_became_active":"2024-05-09T03:15:53.074398+0000","last_became_peered":"2024-05-09T03:15:53.074398+0000","last_unstale":"2024-05-09T03:15:55.061484+0000","last_undegraded":"2024-05-09T03:15:55.061484+0000","last_fullsized":"2024-05-09T03:15:55.061484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:57.947878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062918+0000","last_change":"2024-05-09T03:15:53.084927+0000","last_active":"2024-05-09T03:15:55.062918+0000","last_peered":"2024-05-09T03:15:55.062918+0000","last_clean":"2024-05-09T03:15:55.062918+0000","last_became_active":"2024-05-09T03:15:53.084593+0000","last_became_peered":"2024-05-09T03:15:53.084593+0000","last_unstale":"2024-05-09T03:15:55.062918+0000","last_undegraded":"2024-05-09T03:15:55.062918+0000","last_fullsized":"2024-05-09T03:15:55.062918+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:06:03.714042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075333+0000","last_change":"2024-05-09T03:15:53.078091+0000","last_active":"2024-05-09T03:15:54.075333+0000","last_peered":"2024-05-09T03:15:54.075333+0000","last_clean":"2024-05-09T03:15:54.075333+0000","last_became_active":"2024-05-09T03:15:53.077791+0000","last_became_peered":"2024-05-09T03:15:53.077791+0000","last_unstale":"2024-05-09T03:15:54.075333+0000","last_undegraded":"2024-05-09T03:15:54.075333+0000","last_fullsized":"2024-05-09T03:15:54.075333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:46:23.557851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061393+0000","last_change":"2024-05-09T03:15:53.074257+0000","last_active":"2024-05-09T03:15:55.061393+0000","last_peered":"2024-05-09T03:15:55.061393+0000","last_clean":"2024-05-09T03:15:55.061393+0000","last_became_active":"2024-05-09T03:15:53.073905+0000","last_became_peered":"2024-05-09T03:15:53.073905+0000","last_unstale":"2024-05-09T03:15:55.061393+0000","last_undegraded":"2024-05-09T03:15:55.061393+0000","last_fullsized":"2024-05-09T03:15:55.061393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:56.338990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061782+0000","last_change":"2024-05-09T03:15:53.074885+0000","last_active":"2024-05-09T03:15:55.061782+0000","last_peered":"2024-05-09T03:15:55.061782+0000","last_clean":"2024-05-09T03:15:55.061782+0000","last_became_active":"2024-05-09T03:15:53.074505+0000","last_became_peered":"2024-05-09T03:15:53.074505+0000","last_unstale":"2024-05-09T03:15:55.061782+0000","last_undegraded":"2024-05-09T03:15:55.061782+0000","last_fullsized":"2024-05-09T03:15:55.061782+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:15.192204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061173+0000","last_change":"2024-05-09T03:15:53.071422+0000","last_active":"2024-05-09T03:15:55.061173+0000","last_peered":"2024-05-09T03:15:55.061173+0000","last_clean":"2024-05-09T03:15:55.061173+0000","last_became_active":"2024-05-09T03:15:53.071064+0000","last_became_peered":"2024-05-09T03:15:53.071064+0000","last_unstale":"2024-05-09T03:15:55.061173+0000","last_undegraded":"2024-05-09T03:15:55.061173+0000","last_fullsized":"2024-05-09T03:15:55.061173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:59.610027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061609+0000","last_change":"2024-05-09T03:15:53.074959+0000","last_active":"2024-05-09T03:15:55.061609+0000","last_peered":"2024-05-09T03:15:55.061609+0000","last_clean":"2024-05-09T03:15:55.061609+0000","last_became_active":"2024-05-09T03:15:53.074603+0000","last_became_peered":"2024-05-09T03:15:53.074603+0000","last_unstale":"2024-05-09T03:15:55.061609+0000","last_undegraded":"2024-05-09T03:15:55.061609+0000","last_fullsized":"2024-05-09T03:15:55.061609+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:51.783454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075062+0000","last_change":"2024-05-09T03:15:53.077170+0000","last_active":"2024-05-09T03:15:54.075062+0000","last_peered":"2024-05-09T03:15:54.075062+0000","last_clean":"2024-05-09T03:15:54.075062+0000","last_became_active":"2024-05-09T03:15:53.076852+0000","last_became_peered":"2024-05-09T03:15:53.076852+0000","last_unstale":"2024-05-09T03:15:54.075062+0000","last_undegraded":"2024-05-09T03:15:54.075062+0000","last_fullsized":"2024-05-09T03:15:54.075062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:21.730334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062728+0000","last_change":"2024-05-09T03:15:53.067947+0000","last_active":"2024-05-09T03:15:55.062728+0000","last_peered":"2024-05-09T03:15:55.062728+0000","last_clean":"2024-05-09T03:15:55.062728+0000","last_became_active":"2024-05-09T03:15:53.067669+0000","last_became_peered":"2024-05-09T03:15:53.067669+0000","last_unstale":"2024-05-09T03:15:55.062728+0000","last_undegraded":"2024-05-09T03:15:55.062728+0000","last_fullsized":"2024-05-09T03:15:55.062728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:26.666719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073846+0000","last_change":"2024-05-09T03:15:53.076997+0000","last_active":"2024-05-09T03:15:54.073846+0000","last_peered":"2024-05-09T03:15:54.073846+0000","last_clean":"2024-05-09T03:15:54.073846+0000","last_became_active":"2024-05-09T03:15:53.075380+0000","last_became_peered":"2024-05-09T03:15:53.075380+0000","last_unstale":"2024-05-09T03:15:54.073846+0000","last_undegraded":"2024-05-09T03:15:54.073846+0000","last_fullsized":"2024-05-09T03:15:54.073846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:58.790752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061518+0000","last_change":"2024-05-09T03:15:53.071408+0000","last_active":"2024-05-09T03:15:55.061518+0000","last_peered":"2024-05-09T03:15:55.061518+0000","last_clean":"2024-05-09T03:15:55.061518+0000","last_became_active":"2024-05-09T03:15:53.071078+0000","last_became_peered":"2024-05-09T03:15:53.071078+0000","last_unstale":"2024-05-09T03:15:55.061518+0000","last_undegraded":"2024-05-09T03:15:55.061518+0000","last_fullsized":"2024-05-09T03:15:55.061518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:31.160437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062295+0000","last_change":"2024-05-09T03:15:53.085813+0000","last_active":"2024-05-09T03:15:55.062295+0000","last_peered":"2024-05-09T03:15:55.062295+0000","last_clean":"2024-05-09T03:15:55.062295+0000","last_became_active":"2024-05-09T03:15:53.084981+0000","last_became_peered":"2024-05-09T03:15:53.084981+0000","last_unstale":"2024-05-09T03:15:55.062295+0000","last_undegraded":"2024-05-09T03:15:55.062295+0000","last_fullsized":"2024-05-09T03:15:55.062295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:48.595005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062135+0000","last_change":"2024-05-09T03:15:53.076642+0000","last_active":"2024-05-09T03:15:55.062135+0000","last_peered":"2024-05-09T03:15:55.062135+0000","last_clean":"2024-05-09T03:15:55.062135+0000","last_became_active":"2024-05-09T03:15:53.076274+0000","last_became_peered":"2024-05-09T03:15:53.076274+0000","last_unstale":"2024-05-09T03:15:55.062135+0000","last_undegraded":"2024-05-09T03:15:55.062135+0000","last_fullsized":"2024-05-09T03:15:55.062135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:07.829466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073995+0000","last_change":"2024-05-09T03:15:53.077474+0000","last_active":"2024-05-09T03:15:54.073995+0000","last_peered":"2024-05-09T03:15:54.073995+0000","last_clean":"2024-05-09T03:15:54.073995+0000","last_became_active":"2024-05-09T03:15:53.077140+0000","last_became_peered":"2024-05-09T03:15:53.077140+0000","last_unstale":"2024-05-09T03:15:54.073995+0000","last_undegraded":"2024-05-09T03:15:54.073995+0000","last_fullsized":"2024-05-09T03:15:54.073995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:49:24.252368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074972+0000","last_change":"2024-05-09T03:15:53.075806+0000","last_active":"2024-05-09T03:15:54.074972+0000","last_peered":"2024-05-09T03:15:54.074972+0000","last_clean":"2024-05-09T03:15:54.074972+0000","last_became_active":"2024-05-09T03:15:53.075367+0000","last_became_peered":"2024-05-09T03:15:53.075367+0000","last_unstale":"2024-05-09T03:15:54.074972+0000","last_undegraded":"2024-05-09T03:15:54.074972+0000","last_fullsized":"2024-05-09T03:15:54.074972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:18.363406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062441+0000","last_change":"2024-05-09T03:15:53.077550+0000","last_active":"2024-05-09T03:15:55.062441+0000","last_peered":"2024-05-09T03:15:55.062441+0000","last_clean":"2024-05-09T03:15:55.062441+0000","last_became_active":"2024-05-09T03:15:53.077229+0000","last_became_peered":"2024-05-09T03:15:53.077229+0000","last_unstale":"2024-05-09T03:15:55.062441+0000","last_undegraded":"2024-05-09T03:15:55.062441+0000","last_fullsized":"2024-05-09T03:15:55.062441+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:49.047471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072510+0000","last_change":"2024-05-09T03:15:53.086603+0000","last_active":"2024-05-09T03:15:54.072510+0000","last_peered":"2024-05-09T03:15:54.072510+0000","last_clean":"2024-05-09T03:15:54.072510+0000","last_became_active":"2024-05-09T03:15:53.086273+0000","last_became_peered":"2024-05-09T03:15:53.086273+0000","last_unstale":"2024-05-09T03:15:54.072510+0000","last_undegraded":"2024-05-09T03:15:54.072510+0000","last_fullsized":"2024-05-09T03:15:54.072510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:52.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062477+0000","last_change":"2024-05-09T03:15:53.086167+0000","last_active":"2024-05-09T03:15:55.062477+0000","last_peered":"2024-05-09T03:15:55.062477+0000","last_clean":"2024-05-09T03:15:55.062477+0000","last_became_active":"2024-05-09T03:15:53.085841+0000","last_became_peered":"2024-05-09T03:15:53.085841+0000","last_unstale":"2024-05-09T03:15:55.062477+0000","last_undegraded":"2024-05-09T03:15:55.062477+0000","last_fullsized":"2024-05-09T03:15:55.062477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:09.995250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062838+0000","last_change":"2024-05-09T03:15:53.088267+0000","last_active":"2024-05-09T03:15:55.062838+0000","last_peered":"2024-05-09T03:15:55.062838+0000","last_clean":"2024-05-09T03:15:55.062838+0000","last_became_active":"2024-05-09T03:15:53.087948+0000","last_became_peered":"2024-05-09T03:15:53.087948+0000","last_unstale":"2024-05-09T03:15:55.062838+0000","last_undegraded":"2024-05-09T03:15:55.062838+0000","last_fullsized":"2024-05-09T03:15:55.062838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:06.761713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063082+0000","last_change":"2024-05-09T03:15:53.083400+0000","last_active":"2024-05-09T03:15:55.063082+0000","last_peered":"2024-05-09T03:15:55.063082+0000","last_clean":"2024-05-09T03:15:55.063082+0000","last_became_active":"2024-05-09T03:15:53.082697+0000","last_became_peered":"2024-05-09T03:15:53.082697+0000","last_unstale":"2024-05-09T03:15:55.063082+0000","last_undegraded":"2024-05-09T03:15:55.063082+0000","last_fullsized":"2024-05-09T03:15:55.063082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:41.831877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073504+0000","last_change":"2024-05-09T03:15:53.083463+0000","last_active":"2024-05-09T03:15:54.073504+0000","last_peered":"2024-05-09T03:15:54.073504+0000","last_clean":"2024-05-09T03:15:54.073504+0000","last_became_active":"2024-05-09T03:15:53.083009+0000","last_became_peered":"2024-05-09T03:15:53.083009+0000","last_unstale":"2024-05-09T03:15:54.073504+0000","last_undegraded":"2024-05-09T03:15:54.073504+0000","last_fullsized":"2024-05-09T03:15:54.073504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:24.180236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061931+0000","last_change":"2024-05-09T03:15:53.089084+0000","last_active":"2024-05-09T03:15:55.061931+0000","last_peered":"2024-05-09T03:15:55.061931+0000","last_clean":"2024-05-09T03:15:55.061931+0000","last_became_active":"2024-05-09T03:15:53.088793+0000","last_became_peered":"2024-05-09T03:15:53.088793+0000","last_unstale":"2024-05-09T03:15:55.061931+0000","last_undegraded":"2024-05-09T03:15:55.061931+0000","last_fullsized":"2024-05-09T03:15:55.061931+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:49:21.018337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062576+0000","last_change":"2024-05-09T03:15:53.069817+0000","last_active":"2024-05-09T03:15:55.062576+0000","last_peered":"2024-05-09T03:15:55.062576+0000","last_clean":"2024-05-09T03:15:55.062576+0000","last_became_active":"2024-05-09T03:15:53.069459+0000","last_became_peered":"2024-05-09T03:15:53.069459+0000","last_unstale":"2024-05-09T03:15:55.062576+0000","last_undegraded":"2024-05-09T03:15:55.062576+0000","last_fullsized":"2024-05-09T03:15:55.062576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:06.776516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061973+0000","last_change":"2024-05-09T03:15:53.081845+0000","last_active":"2024-05-09T03:15:55.061973+0000","last_peered":"2024-05-09T03:15:55.061973+0000","last_clean":"2024-05-09T03:15:55.061973+0000","last_became_active":"2024-05-09T03:15:53.081501+0000","last_became_peered":"2024-05-09T03:15:53.081501+0000","last_unstale":"2024-05-09T03:15:55.061973+0000","last_undegraded":"2024-05-09T03:15:55.061973+0000","last_fullsized":"2024-05-09T03:15:55.061973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:25:45.894851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074597+0000","last_change":"2024-05-09T03:15:53.077829+0000","last_active":"2024-05-09T03:15:54.074597+0000","last_peered":"2024-05-09T03:15:54.074597+0000","last_clean":"2024-05-09T03:15:54.074597+0000","last_became_active":"2024-05-09T03:15:53.077499+0000","last_became_peered":"2024-05-09T03:15:53.077499+0000","last_unstale":"2024-05-09T03:15:54.074597+0000","last_undegraded":"2024-05-09T03:15:54.074597+0000","last_fullsized":"2024-05-09T03:15:54.074597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:17.882544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073600+0000","last_change":"2024-05-09T03:15:53.069972+0000","last_active":"2024-05-09T03:15:54.073600+0000","last_peered":"2024-05-09T03:15:54.073600+0000","last_clean":"2024-05-09T03:15:54.073600+0000","last_became_active":"2024-05-09T03:15:53.069637+0000","last_became_peered":"2024-05-09T03:15:53.069637+0000","last_unstale":"2024-05-09T03:15:54.073600+0000","last_undegraded":"2024-05-09T03:15:54.073600+0000","last_fullsized":"2024-05-09T03:15:54.073600+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:40:40.889305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062296+0000","last_change":"2024-05-09T03:15:53.077663+0000","last_active":"2024-05-09T03:15:55.062296+0000","last_peered":"2024-05-09T03:15:55.062296+0000","last_clean":"2024-05-09T03:15:55.062296+0000","last_became_active":"2024-05-09T03:15:53.077336+0000","last_became_peered":"2024-05-09T03:15:53.077336+0000","last_unstale":"2024-05-09T03:15:55.062296+0000","last_undegraded":"2024-05-09T03:15:55.062296+0000","last_fullsized":"2024-05-09T03:15:55.062296+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:03.516637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063282+0000","last_change":"2024-05-09T03:15:53.084489+0000","last_active":"2024-05-09T03:15:55.063282+0000","last_peered":"2024-05-09T03:15:55.063282+0000","last_clean":"2024-05-09T03:15:55.063282+0000","last_became_active":"2024-05-09T03:15:53.084126+0000","last_became_peered":"2024-05-09T03:15:53.084126+0000","last_unstale":"2024-05-09T03:15:55.063282+0000","last_undegraded":"2024-05-09T03:15:55.063282+0000","last_fullsized":"2024-05-09T03:15:55.063282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:26.713902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074805+0000","last_change":"2024-05-09T03:15:53.082583+0000","last_active":"2024-05-09T03:15:54.074805+0000","last_peered":"2024-05-09T03:15:54.074805+0000","last_clean":"2024-05-09T03:15:54.074805+0000","last_became_active":"2024-05-09T03:15:53.081246+0000","last_became_peered":"2024-05-09T03:15:53.081246+0000","last_unstale":"2024-05-09T03:15:54.074805+0000","last_undegraded":"2024-05-09T03:15:54.074805+0000","last_fullsized":"2024-05-09T03:15:54.074805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:22:10.833765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073546+0000","last_change":"2024-05-09T03:15:53.072372+0000","last_active":"2024-05-09T03:15:54.073546+0000","last_peered":"2024-05-09T03:15:54.073546+0000","last_clean":"2024-05-09T03:15:54.073546+0000","last_became_active":"2024-05-09T03:15:53.072031+0000","last_became_peered":"2024-05-09T03:15:53.072031+0000","last_unstale":"2024-05-09T03:15:54.073546+0000","last_undegraded":"2024-05-09T03:15:54.073546+0000","last_fullsized":"2024-05-09T03:15:54.073546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:18.363406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074510+0000","last_change":"2024-05-09T03:15:53.078686+0000","last_active":"2024-05-09T03:15:54.074510+0000","last_peered":"2024-05-09T03:15:54.074510+0000","last_clean":"2024-05-09T03:15:54.074510+0000","last_became_active":"2024-05-09T03:15:53.077760+0000","last_became_peered":"2024-05-09T03:15:53.077760+0000","last_unstale":"2024-05-09T03:15:54.074510+0000","last_undegraded":"2024-05-09T03:15:54.074510+0000","last_fullsized":"2024-05-09T03:15:54.074510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074552+0000","last_change":"2024-05-09T03:15:52.068420+0000","last_active":"2024-05-09T03:15:54.074552+0000","last_peered":"2024-05-09T03:15:54.074552+0000","last_clean":"2024-05-09T03:15:54.074552+0000","last_became_active":"2024-05-09T03:15:52.068090+0000","last_became_peered":"2024-05-09T03:15:52.068090+0000","last_unstale":"2024-05-09T03:15:54.074552+0000","last_undegraded":"2024-05-09T03:15:54.074552+0000","last_fullsized":"2024-05-09T03:15:54.074552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:05.768358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074289+0000","last_change":"2024-05-09T03:15:53.083690+0000","last_active":"2024-05-09T03:15:54.074289+0000","last_peered":"2024-05-09T03:15:54.074289+0000","last_clean":"2024-05-09T03:15:54.074289+0000","last_became_active":"2024-05-09T03:15:53.083068+0000","last_became_peered":"2024-05-09T03:15:53.083068+0000","last_unstale":"2024-05-09T03:15:54.074289+0000","last_undegraded":"2024-05-09T03:15:54.074289+0000","last_fullsized":"2024-05-09T03:15:54.074289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:26.794178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074584+0000","last_change":"2024-05-09T03:15:52.078222+0000","last_active":"2024-05-09T03:15:54.074584+0000","last_peered":"2024-05-09T03:15:54.074584+0000","last_clean":"2024-05-09T03:15:54.074584+0000","last_became_active":"2024-05-09T03:15:52.077869+0000","last_became_peered":"2024-05-09T03:15:52.077869+0000","last_unstale":"2024-05-09T03:15:54.074584+0000","last_undegraded":"2024-05-09T03:15:54.074584+0000","last_fullsized":"2024-05-09T03:15:54.074584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:02.508479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075148+0000","last_change":"2024-05-09T03:15:53.071266+0000","last_active":"2024-05-09T03:15:54.075148+0000","last_peered":"2024-05-09T03:15:54.075148+0000","last_clean":"2024-05-09T03:15:54.075148+0000","last_became_active":"2024-05-09T03:15:53.070975+0000","last_became_peered":"2024-05-09T03:15:53.070975+0000","last_unstale":"2024-05-09T03:15:54.075148+0000","last_undegraded":"2024-05-09T03:15:54.075148+0000","last_fullsized":"2024-05-09T03:15:54.075148+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:57.947878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072409+0000","last_change":"2024-05-09T03:15:52.073336+0000","last_active":"2024-05-09T03:15:54.072409+0000","last_peered":"2024-05-09T03:15:54.072409+0000","last_clean":"2024-05-09T03:15:54.072409+0000","last_became_active":"2024-05-09T03:15:52.073045+0000","last_became_peered":"2024-05-09T03:15:52.073045+0000","last_unstale":"2024-05-09T03:15:54.072409+0000","last_undegraded":"2024-05-09T03:15:54.072409+0000","last_fullsized":"2024-05-09T03:15:54.072409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:49:23.244210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061576+0000","last_change":"2024-05-09T03:15:53.076120+0000","last_active":"2024-05-09T03:15:55.061576+0000","last_peered":"2024-05-09T03:15:55.061576+0000","last_clean":"2024-05-09T03:15:55.061576+0000","last_became_active":"2024-05-09T03:15:53.075785+0000","last_became_peered":"2024-05-09T03:15:53.075785+0000","last_unstale":"2024-05-09T03:15:55.061576+0000","last_undegraded":"2024-05-09T03:15:55.061576+0000","last_fullsized":"2024-05-09T03:15:55.061576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:51:33.204849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072399+0000","last_change":"2024-05-09T03:15:52.075163+0000","last_active":"2024-05-09T03:15:54.072399+0000","last_peered":"2024-05-09T03:15:54.072399+0000","last_clean":"2024-05-09T03:15:54.072399+0000","last_became_active":"2024-05-09T03:15:52.074830+0000","last_became_peered":"2024-05-09T03:15:52.074830+0000","last_unstale":"2024-05-09T03:15:54.072399+0000","last_undegraded":"2024-05-09T03:15:54.072399+0000","last_fullsized":"2024-05-09T03:15:54.072399+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:05.694999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063491+0000","last_change":"2024-05-09T03:15:53.083722+0000","last_active":"2024-05-09T03:15:55.063491+0000","last_peered":"2024-05-09T03:15:55.063491+0000","last_clean":"2024-05-09T03:15:55.063491+0000","last_became_active":"2024-05-09T03:15:53.083100+0000","last_became_peered":"2024-05-09T03:15:53.083100+0000","last_unstale":"2024-05-09T03:15:55.063491+0000","last_undegraded":"2024-05-09T03:15:55.063491+0000","last_fullsized":"2024-05-09T03:15:55.063491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:18:38.837246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075433+0000","last_change":"2024-05-09T03:15:52.059453+0000","last_active":"2024-05-09T03:15:54.075433+0000","last_peered":"2024-05-09T03:15:54.075433+0000","last_clean":"2024-05-09T03:15:54.075433+0000","last_became_active":"2024-05-09T03:15:52.058216+0000","last_became_peered":"2024-05-09T03:15:52.058216+0000","last_unstale":"2024-05-09T03:15:54.075433+0000","last_undegraded":"2024-05-09T03:15:54.075433+0000","last_fullsized":"2024-05-09T03:15:54.075433+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:05.768358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075355+0000","last_change":"2024-05-09T03:15:53.075244+0000","last_active":"2024-05-09T03:15:54.075355+0000","last_peered":"2024-05-09T03:15:54.075355+0000","last_clean":"2024-05-09T03:15:54.075355+0000","last_became_active":"2024-05-09T03:15:53.074900+0000","last_became_peered":"2024-05-09T03:15:53.074900+0000","last_unstale":"2024-05-09T03:15:54.075355+0000","last_undegraded":"2024-05-09T03:15:54.075355+0000","last_fullsized":"2024-05-09T03:15:54.075355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:04.173563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073951+0000","last_change":"2024-05-09T03:15:52.078882+0000","last_active":"2024-05-09T03:15:54.073951+0000","last_peered":"2024-05-09T03:15:54.073951+0000","last_clean":"2024-05-09T03:15:54.073951+0000","last_became_active":"2024-05-09T03:15:52.078565+0000","last_became_peered":"2024-05-09T03:15:52.078565+0000","last_unstale":"2024-05-09T03:15:54.073951+0000","last_undegraded":"2024-05-09T03:15:54.073951+0000","last_fullsized":"2024-05-09T03:15:54.073951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:18:29.836613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061169+0000","last_change":"2024-05-09T03:15:53.080971+0000","last_active":"2024-05-09T03:15:55.061169+0000","last_peered":"2024-05-09T03:15:55.061169+0000","last_clean":"2024-05-09T03:15:55.061169+0000","last_became_active":"2024-05-09T03:15:53.079680+0000","last_became_peered":"2024-05-09T03:15:53.079680+0000","last_unstale":"2024-05-09T03:15:55.061169+0000","last_undegraded":"2024-05-09T03:15:55.061169+0000","last_fullsized":"2024-05-09T03:15:55.061169+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:52.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.071939+0000","last_change":"2024-05-09T03:15:52.072230+0000","last_active":"2024-05-09T03:15:54.071939+0000","last_peered":"2024-05-09T03:15:54.071939+0000","last_clean":"2024-05-09T03:15:54.071939+0000","last_became_active":"2024-05-09T03:15:52.071810+0000","last_became_peered":"2024-05-09T03:15:52.071810+0000","last_unstale":"2024-05-09T03:15:54.071939+0000","last_undegraded":"2024-05-09T03:15:54.071939+0000","last_fullsized":"2024-05-09T03:15:54.071939+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:50.775296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062341+0000","last_change":"2024-05-09T03:15:53.076664+0000","last_active":"2024-05-09T03:15:55.062341+0000","last_peered":"2024-05-09T03:15:55.062341+0000","last_clean":"2024-05-09T03:15:55.062341+0000","last_became_active":"2024-05-09T03:15:53.076351+0000","last_became_peered":"2024-05-09T03:15:53.076351+0000","last_unstale":"2024-05-09T03:15:55.062341+0000","last_undegraded":"2024-05-09T03:15:55.062341+0000","last_fullsized":"2024-05-09T03:15:55.062341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:08.551174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062075+0000","last_change":"2024-05-09T03:15:52.075417+0000","last_active":"2024-05-09T03:15:55.062075+0000","last_peered":"2024-05-09T03:15:55.062075+0000","last_clean":"2024-05-09T03:15:55.062075+0000","last_became_active":"2024-05-09T03:15:52.074250+0000","last_became_peered":"2024-05-09T03:15:52.074250+0000","last_unstale":"2024-05-09T03:15:55.062075+0000","last_undegraded":"2024-05-09T03:15:55.062075+0000","last_fullsized":"2024-05-09T03:15:55.062075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:16:46.478388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063081+0000","last_change":"2024-05-09T03:15:53.081457+0000","last_active":"2024-05-09T03:15:55.063081+0000","last_peered":"2024-05-09T03:15:55.063081+0000","last_clean":"2024-05-09T03:15:55.063081+0000","last_became_active":"2024-05-09T03:15:53.081091+0000","last_became_peered":"2024-05-09T03:15:53.081091+0000","last_unstale":"2024-05-09T03:15:55.063081+0000","last_undegraded":"2024-05-09T03:15:55.063081+0000","last_fullsized":"2024-05-09T03:15:55.063081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:58:20.566631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074682+0000","last_change":"2024-05-09T03:15:52.078314+0000","last_active":"2024-05-09T03:15:54.074682+0000","last_peered":"2024-05-09T03:15:54.074682+0000","last_clean":"2024-05-09T03:15:54.074682+0000","last_became_active":"2024-05-09T03:15:52.077908+0000","last_became_peered":"2024-05-09T03:15:52.077908+0000","last_unstale":"2024-05-09T03:15:54.074682+0000","last_undegraded":"2024-05-09T03:15:54.074682+0000","last_fullsized":"2024-05-09T03:15:54.074682+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:40.823719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061767+0000","last_change":"2024-05-09T03:15:53.074319+0000","last_active":"2024-05-09T03:15:55.061767+0000","last_peered":"2024-05-09T03:15:55.061767+0000","last_clean":"2024-05-09T03:15:55.061767+0000","last_became_active":"2024-05-09T03:15:53.073922+0000","last_became_peered":"2024-05-09T03:15:53.073922+0000","last_unstale":"2024-05-09T03:15:55.061767+0000","last_undegraded":"2024-05-09T03:15:55.061767+0000","last_fullsized":"2024-05-09T03:15:55.061767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:49:24.252368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075028+0000","last_change":"2024-05-09T03:15:52.071302+0000","last_active":"2024-05-09T03:15:54.075028+0000","last_peered":"2024-05-09T03:15:54.075028+0000","last_clean":"2024-05-09T03:15:54.075028+0000","last_became_active":"2024-05-09T03:15:52.070945+0000","last_became_peered":"2024-05-09T03:15:52.070945+0000","last_unstale":"2024-05-09T03:15:54.075028+0000","last_undegraded":"2024-05-09T03:15:54.075028+0000","last_fullsized":"2024-05-09T03:15:54.075028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:22.211464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072386+0000","last_change":"2024-05-09T03:15:53.091071+0000","last_active":"2024-05-09T03:15:54.072386+0000","last_peered":"2024-05-09T03:15:54.072386+0000","last_clean":"2024-05-09T03:15:54.072386+0000","last_became_active":"2024-05-09T03:15:53.089761+0000","last_became_peered":"2024-05-09T03:15:53.089761+0000","last_unstale":"2024-05-09T03:15:54.072386+0000","last_undegraded":"2024-05-09T03:15:54.072386+0000","last_fullsized":"2024-05-09T03:15:54.072386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:26.794178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073830+0000","last_change":"2024-05-09T03:15:52.072974+0000","last_active":"2024-05-09T03:15:54.073830+0000","last_peered":"2024-05-09T03:15:54.073830+0000","last_clean":"2024-05-09T03:15:54.073830+0000","last_became_active":"2024-05-09T03:15:52.072673+0000","last_became_peered":"2024-05-09T03:15:52.072673+0000","last_unstale":"2024-05-09T03:15:54.073830+0000","last_undegraded":"2024-05-09T03:15:54.073830+0000","last_fullsized":"2024-05-09T03:15:54.073830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:11.847485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074819+0000","last_change":"2024-05-09T03:15:53.080769+0000","last_active":"2024-05-09T03:15:54.074819+0000","last_peered":"2024-05-09T03:15:54.074819+0000","last_clean":"2024-05-09T03:15:54.074819+0000","last_became_active":"2024-05-09T03:15:53.080150+0000","last_became_peered":"2024-05-09T03:15:53.080150+0000","last_unstale":"2024-05-09T03:15:54.074819+0000","last_undegraded":"2024-05-09T03:15:54.074819+0000","last_fullsized":"2024-05-09T03:15:54.074819+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:24.180236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061520+0000","last_change":"2024-05-09T03:15:52.068844+0000","last_active":"2024-05-09T03:15:55.061520+0000","last_peered":"2024-05-09T03:15:55.061520+0000","last_clean":"2024-05-09T03:15:55.061520+0000","last_became_active":"2024-05-09T03:15:52.068357+0000","last_became_peered":"2024-05-09T03:15:52.068357+0000","last_unstale":"2024-05-09T03:15:55.061520+0000","last_undegraded":"2024-05-09T03:15:55.061520+0000","last_fullsized":"2024-05-09T03:15:55.061520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:18:29.836613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072153+0000","last_change":"2024-05-09T03:15:53.091830+0000","last_active":"2024-05-09T03:15:54.072153+0000","last_peered":"2024-05-09T03:15:54.072153+0000","last_clean":"2024-05-09T03:15:54.072153+0000","last_became_active":"2024-05-09T03:15:53.091497+0000","last_became_peered":"2024-05-09T03:15:53.091497+0000","last_unstale":"2024-05-09T03:15:54.072153+0000","last_undegraded":"2024-05-09T03:15:54.072153+0000","last_fullsized":"2024-05-09T03:15:54.072153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:46:23.557851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062469+0000","last_change":"2024-05-09T03:15:52.069591+0000","last_active":"2024-05-09T03:15:55.062469+0000","last_peered":"2024-05-09T03:15:55.062469+0000","last_clean":"2024-05-09T03:15:55.062469+0000","last_became_active":"2024-05-09T03:15:52.069257+0000","last_became_peered":"2024-05-09T03:15:52.069257+0000","last_unstale":"2024-05-09T03:15:55.062469+0000","last_undegraded":"2024-05-09T03:15:55.062469+0000","last_fullsized":"2024-05-09T03:15:55.062469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:16:17.133055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061739+0000","last_change":"2024-05-09T03:15:53.079234+0000","last_active":"2024-05-09T03:15:55.061739+0000","last_peered":"2024-05-09T03:15:55.061739+0000","last_clean":"2024-05-09T03:15:55.061739+0000","last_became_active":"2024-05-09T03:15:53.078944+0000","last_became_peered":"2024-05-09T03:15:53.078944+0000","last_unstale":"2024-05-09T03:15:55.061739+0000","last_undegraded":"2024-05-09T03:15:55.061739+0000","last_fullsized":"2024-05-09T03:15:55.061739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:50:18.761398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062456+0000","last_change":"2024-05-09T03:15:52.078830+0000","last_active":"2024-05-09T03:15:55.062456+0000","last_peered":"2024-05-09T03:15:55.062456+0000","last_clean":"2024-05-09T03:15:55.062456+0000","last_became_active":"2024-05-09T03:15:52.078379+0000","last_became_peered":"2024-05-09T03:15:52.078379+0000","last_unstale":"2024-05-09T03:15:55.062456+0000","last_undegraded":"2024-05-09T03:15:55.062456+0000","last_fullsized":"2024-05-09T03:15:55.062456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:56.939720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062186+0000","last_change":"2024-05-09T03:15:53.076826+0000","last_active":"2024-05-09T03:15:55.062186+0000","last_peered":"2024-05-09T03:15:55.062186+0000","last_clean":"2024-05-09T03:15:55.062186+0000","last_became_active":"2024-05-09T03:15:53.074980+0000","last_became_peered":"2024-05-09T03:15:53.074980+0000","last_unstale":"2024-05-09T03:15:55.062186+0000","last_undegraded":"2024-05-09T03:15:55.062186+0000","last_fullsized":"2024-05-09T03:15:55.062186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:18:30.844771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061246+0000","last_change":"2024-05-09T03:15:52.071423+0000","last_active":"2024-05-09T03:15:55.061246+0000","last_peered":"2024-05-09T03:15:55.061246+0000","last_clean":"2024-05-09T03:15:55.061246+0000","last_became_active":"2024-05-09T03:15:52.071086+0000","last_became_peered":"2024-05-09T03:15:52.071086+0000","last_unstale":"2024-05-09T03:15:55.061246+0000","last_undegraded":"2024-05-09T03:15:55.061246+0000","last_fullsized":"2024-05-09T03:15:55.061246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:02.508479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061943+0000","last_change":"2024-05-09T03:15:53.069008+0000","last_active":"2024-05-09T03:15:55.061943+0000","last_peered":"2024-05-09T03:15:55.061943+0000","last_clean":"2024-05-09T03:15:55.061943+0000","last_became_active":"2024-05-09T03:15:53.068691+0000","last_became_peered":"2024-05-09T03:15:53.068691+0000","last_unstale":"2024-05-09T03:15:55.061943+0000","last_undegraded":"2024-05-09T03:15:55.061943+0000","last_fullsized":"2024-05-09T03:15:55.061943+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:23.195822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.071923+0000","last_change":"2024-05-09T03:15:52.070582+0000","last_active":"2024-05-09T03:15:54.071923+0000","last_peered":"2024-05-09T03:15:54.071923+0000","last_clean":"2024-05-09T03:15:54.071923+0000","last_became_active":"2024-05-09T03:15:52.070229+0000","last_became_peered":"2024-05-09T03:15:52.070229+0000","last_unstale":"2024-05-09T03:15:54.071923+0000","last_undegraded":"2024-05-09T03:15:54.071923+0000","last_fullsized":"2024-05-09T03:15:54.071923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:23:14.954104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062014+0000","last_change":"2024-05-09T03:15:53.065402+0000","last_active":"2024-05-09T03:15:55.062014+0000","last_peered":"2024-05-09T03:15:55.062014+0000","last_clean":"2024-05-09T03:15:55.062014+0000","last_became_active":"2024-05-09T03:15:53.065104+0000","last_became_peered":"2024-05-09T03:15:53.065104+0000","last_unstale":"2024-05-09T03:15:55.062014+0000","last_undegraded":"2024-05-09T03:15:55.062014+0000","last_fullsized":"2024-05-09T03:15:55.062014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:07.829466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074923+0000","last_change":"2024-05-09T03:15:52.067126+0000","last_active":"2024-05-09T03:15:54.074923+0000","last_peered":"2024-05-09T03:15:54.074923+0000","last_clean":"2024-05-09T03:15:54.074923+0000","last_became_active":"2024-05-09T03:15:52.066221+0000","last_became_peered":"2024-05-09T03:15:52.066221+0000","last_unstale":"2024-05-09T03:15:54.074923+0000","last_undegraded":"2024-05-09T03:15:54.074923+0000","last_fullsized":"2024-05-09T03:15:54.074923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:25.658561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074482+0000","last_change":"2024-05-09T03:15:53.068542+0000","last_active":"2024-05-09T03:15:54.074482+0000","last_peered":"2024-05-09T03:15:54.074482+0000","last_clean":"2024-05-09T03:15:54.074482+0000","last_became_active":"2024-05-09T03:15:53.068102+0000","last_became_peered":"2024-05-09T03:15:53.068102+0000","last_unstale":"2024-05-09T03:15:54.074482+0000","last_undegraded":"2024-05-09T03:15:54.074482+0000","last_fullsized":"2024-05-09T03:15:54.074482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:52.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072141+0000","last_change":"2024-05-09T03:15:52.072827+0000","last_active":"2024-05-09T03:15:54.072141+0000","last_peered":"2024-05-09T03:15:54.072141+0000","last_clean":"2024-05-09T03:15:54.072141+0000","last_became_active":"2024-05-09T03:15:52.072097+0000","last_became_peered":"2024-05-09T03:15:52.072097+0000","last_unstale":"2024-05-09T03:15:54.072141+0000","last_undegraded":"2024-05-09T03:15:54.072141+0000","last_fullsized":"2024-05-09T03:15:54.072141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:46.656775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075113+0000","last_change":"2024-05-09T03:15:53.078967+0000","last_active":"2024-05-09T03:15:54.075113+0000","last_peered":"2024-05-09T03:15:54.075113+0000","last_clean":"2024-05-09T03:15:54.075113+0000","last_became_active":"2024-05-09T03:15:53.078607+0000","last_became_peered":"2024-05-09T03:15:53.078607+0000","last_unstale":"2024-05-09T03:15:54.075113+0000","last_undegraded":"2024-05-09T03:15:54.075113+0000","last_fullsized":"2024-05-09T03:15:54.075113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:58.790752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072455+0000","last_change":"2024-05-09T03:15:52.067341+0000","last_active":"2024-05-09T03:15:54.072455+0000","last_peered":"2024-05-09T03:15:54.072455+0000","last_clean":"2024-05-09T03:15:54.072455+0000","last_became_active":"2024-05-09T03:15:52.066938+0000","last_became_peered":"2024-05-09T03:15:52.066938+0000","last_unstale":"2024-05-09T03:15:54.072455+0000","last_undegraded":"2024-05-09T03:15:54.072455+0000","last_fullsized":"2024-05-09T03:15:54.072455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:22:09.825607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062589+0000","last_change":"2024-05-09T03:15:53.084906+0000","last_active":"2024-05-09T03:15:55.062589+0000","last_peered":"2024-05-09T03:15:55.062589+0000","last_clean":"2024-05-09T03:15:55.062589+0000","last_became_active":"2024-05-09T03:15:53.084554+0000","last_became_peered":"2024-05-09T03:15:53.084554+0000","last_unstale":"2024-05-09T03:15:55.062589+0000","last_undegraded":"2024-05-09T03:15:55.062589+0000","last_fullsized":"2024-05-09T03:15:55.062589+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:15:28.432507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062031+0000","last_change":"2024-05-09T03:15:52.068600+0000","last_active":"2024-05-09T03:15:55.062031+0000","last_peered":"2024-05-09T03:15:55.062031+0000","last_clean":"2024-05-09T03:15:55.062031+0000","last_became_active":"2024-05-09T03:15:52.067579+0000","last_became_peered":"2024-05-09T03:15:52.067579+0000","last_unstale":"2024-05-09T03:15:55.062031+0000","last_undegraded":"2024-05-09T03:15:55.062031+0000","last_fullsized":"2024-05-09T03:15:55.062031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:20.722176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061534+0000","last_change":"2024-05-09T03:15:53.075148+0000","last_active":"2024-05-09T03:15:55.061534+0000","last_peered":"2024-05-09T03:15:55.061534+0000","last_clean":"2024-05-09T03:15:55.061534+0000","last_became_active":"2024-05-09T03:15:53.074808+0000","last_became_peered":"2024-05-09T03:15:53.074808+0000","last_unstale":"2024-05-09T03:15:55.061534+0000","last_undegraded":"2024-05-09T03:15:55.061534+0000","last_fullsized":"2024-05-09T03:15:55.061534+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:07.835507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062666+0000","last_change":"2024-05-09T03:15:52.075836+0000","last_active":"2024-05-09T03:15:55.062666+0000","last_peered":"2024-05-09T03:15:55.062666+0000","last_clean":"2024-05-09T03:15:55.062666+0000","last_became_active":"2024-05-09T03:15:52.075318+0000","last_became_peered":"2024-05-09T03:15:52.075318+0000","last_unstale":"2024-05-09T03:15:55.062666+0000","last_undegraded":"2024-05-09T03:15:55.062666+0000","last_fullsized":"2024-05-09T03:15:55.062666+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:44.568140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062135+0000","last_change":"2024-05-09T03:15:53.080707+0000","last_active":"2024-05-09T03:15:55.062135+0000","last_peered":"2024-05-09T03:15:55.062135+0000","last_clean":"2024-05-09T03:15:55.062135+0000","last_became_active":"2024-05-09T03:15:53.080346+0000","last_became_peered":"2024-05-09T03:15:53.080346+0000","last_unstale":"2024-05-09T03:15:55.062135+0000","last_undegraded":"2024-05-09T03:15:55.062135+0000","last_fullsized":"2024-05-09T03:15:55.062135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:42:11.979488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061884+0000","last_change":"2024-05-09T03:15:52.072938+0000","last_active":"2024-05-09T03:15:55.061884+0000","last_peered":"2024-05-09T03:15:55.061884+0000","last_clean":"2024-05-09T03:15:55.061884+0000","last_became_active":"2024-05-09T03:15:52.072599+0000","last_became_peered":"2024-05-09T03:15:52.072599+0000","last_unstale":"2024-05-09T03:15:55.061884+0000","last_undegraded":"2024-05-09T03:15:55.061884+0000","last_fullsized":"2024-05-09T03:15:55.061884+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:12:19.346768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061366+0000","last_change":"2024-05-09T03:15:53.079067+0000","last_active":"2024-05-09T03:15:55.061366+0000","last_peered":"2024-05-09T03:15:55.061366+0000","last_clean":"2024-05-09T03:15:55.061366+0000","last_became_active":"2024-05-09T03:15:53.078752+0000","last_became_peered":"2024-05-09T03:15:53.078752+0000","last_unstale":"2024-05-09T03:15:55.061366+0000","last_undegraded":"2024-05-09T03:15:55.061366+0000","last_fullsized":"2024-05-09T03:15:55.061366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:01:53.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061713+0000","last_change":"2024-05-09T03:15:52.068116+0000","last_active":"2024-05-09T03:15:55.061713+0000","last_peered":"2024-05-09T03:15:55.061713+0000","last_clean":"2024-05-09T03:15:55.061713+0000","last_became_active":"2024-05-09T03:15:52.067782+0000","last_became_peered":"2024-05-09T03:15:52.067782+0000","last_unstale":"2024-05-09T03:15:55.061713+0000","last_undegraded":"2024-05-09T03:15:55.061713+0000","last_fullsized":"2024-05-09T03:15:55.061713+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:51.349610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074631+0000","last_change":"2024-05-09T03:15:53.077480+0000","last_active":"2024-05-09T03:15:54.074631+0000","last_peered":"2024-05-09T03:15:54.074631+0000","last_clean":"2024-05-09T03:15:54.074631+0000","last_became_active":"2024-05-09T03:15:53.076807+0000","last_became_peered":"2024-05-09T03:15:53.076807+0000","last_unstale":"2024-05-09T03:15:54.074631+0000","last_undegraded":"2024-05-09T03:15:54.074631+0000","last_fullsized":"2024-05-09T03:15:54.074631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:06.703157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062668+0000","last_change":"2024-05-09T03:15:52.072537+0000","last_active":"2024-05-09T03:15:55.062668+0000","last_peered":"2024-05-09T03:15:55.062668+0000","last_clean":"2024-05-09T03:15:55.062668+0000","last_became_active":"2024-05-09T03:15:52.072211+0000","last_became_peered":"2024-05-09T03:15:52.072211+0000","last_unstale":"2024-05-09T03:15:55.062668+0000","last_undegraded":"2024-05-09T03:15:55.062668+0000","last_fullsized":"2024-05-09T03:15:55.062668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:22.736615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062048+0000","last_change":"2024-05-09T03:15:53.078490+0000","last_active":"2024-05-09T03:15:55.062048+0000","last_peered":"2024-05-09T03:15:55.062048+0000","last_clean":"2024-05-09T03:15:55.062048+0000","last_became_active":"2024-05-09T03:15:53.078151+0000","last_became_peered":"2024-05-09T03:15:53.078151+0000","last_unstale":"2024-05-09T03:15:55.062048+0000","last_undegraded":"2024-05-09T03:15:55.062048+0000","last_fullsized":"2024-05-09T03:15:55.062048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:08.551174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063361+0000","last_change":"2024-05-09T03:15:52.070511+0000","last_active":"2024-05-09T03:15:55.063361+0000","last_peered":"2024-05-09T03:15:55.063361+0000","last_clean":"2024-05-09T03:15:55.063361+0000","last_became_active":"2024-05-09T03:15:52.070212+0000","last_became_peered":"2024-05-09T03:15:52.070212+0000","last_unstale":"2024-05-09T03:15:55.063361+0000","last_undegraded":"2024-05-09T03:15:55.063361+0000","last_fullsized":"2024-05-09T03:15:55.063361+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:07.450826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074062+0000","last_change":"2024-05-09T03:15:53.073566+0000","last_active":"2024-05-09T03:15:54.074062+0000","last_peered":"2024-05-09T03:15:54.074062+0000","last_clean":"2024-05-09T03:15:54.074062+0000","last_became_active":"2024-05-09T03:15:53.072995+0000","last_became_peered":"2024-05-09T03:15:53.072995+0000","last_unstale":"2024-05-09T03:15:54.074062+0000","last_undegraded":"2024-05-09T03:15:54.074062+0000","last_fullsized":"2024-05-09T03:15:54.074062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:46:23.557851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073707+0000","last_change":"2024-05-09T03:15:52.068346+0000","last_active":"2024-05-09T03:15:54.073707+0000","last_peered":"2024-05-09T03:15:54.073707+0000","last_clean":"2024-05-09T03:15:54.073707+0000","last_became_active":"2024-05-09T03:15:52.068035+0000","last_became_peered":"2024-05-09T03:15:52.068035+0000","last_unstale":"2024-05-09T03:15:54.073707+0000","last_undegraded":"2024-05-09T03:15:54.073707+0000","last_fullsized":"2024-05-09T03:15:54.073707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:02.508479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075057+0000","last_change":"2024-05-09T03:15:53.082192+0000","last_active":"2024-05-09T03:15:54.075057+0000","last_peered":"2024-05-09T03:15:54.075057+0000","last_clean":"2024-05-09T03:15:54.075057+0000","last_became_active":"2024-05-09T03:15:53.081850+0000","last_became_peered":"2024-05-09T03:15:53.081850+0000","last_unstale":"2024-05-09T03:15:54.075057+0000","last_undegraded":"2024-05-09T03:15:54.075057+0000","last_fullsized":"2024-05-09T03:15:54.075057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:40:40.889305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074174+0000","last_change":"2024-05-09T03:15:52.069955+0000","last_active":"2024-05-09T03:15:54.074174+0000","last_peered":"2024-05-09T03:15:54.074174+0000","last_clean":"2024-05-09T03:15:54.074174+0000","last_became_active":"2024-05-09T03:15:52.069619+0000","last_became_peered":"2024-05-09T03:15:52.069619+0000","last_unstale":"2024-05-09T03:15:54.074174+0000","last_undegraded":"2024-05-09T03:15:54.074174+0000","last_fullsized":"2024-05-09T03:15:54.074174+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:18:37.829088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061584+0000","last_change":"2024-05-09T03:15:53.079644+0000","last_active":"2024-05-09T03:15:55.061584+0000","last_peered":"2024-05-09T03:15:55.061584+0000","last_clean":"2024-05-09T03:15:55.061584+0000","last_became_active":"2024-05-09T03:15:53.079306+0000","last_became_peered":"2024-05-09T03:15:53.079306+0000","last_unstale":"2024-05-09T03:15:55.061584+0000","last_undegraded":"2024-05-09T03:15:55.061584+0000","last_fullsized":"2024-05-09T03:15:55.061584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:23:15.962262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072368+0000","last_change":"2024-05-09T03:15:52.070867+0000","last_active":"2024-05-09T03:15:54.072368+0000","last_peered":"2024-05-09T03:15:54.072368+0000","last_clean":"2024-05-09T03:15:54.072368+0000","last_became_active":"2024-05-09T03:15:52.070163+0000","last_became_peered":"2024-05-09T03:15:52.070163+0000","last_unstale":"2024-05-09T03:15:54.072368+0000","last_undegraded":"2024-05-09T03:15:54.072368+0000","last_fullsized":"2024-05-09T03:15:54.072368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:16.874387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072212+0000","last_change":"2024-05-09T03:15:53.090489+0000","last_active":"2024-05-09T03:15:54.072212+0000","last_peered":"2024-05-09T03:15:54.072212+0000","last_clean":"2024-05-09T03:15:54.072212+0000","last_became_active":"2024-05-09T03:15:53.089158+0000","last_became_peered":"2024-05-09T03:15:53.089158+0000","last_unstale":"2024-05-09T03:15:54.072212+0000","last_undegraded":"2024-05-09T03:15:54.072212+0000","last_fullsized":"2024-05-09T03:15:54.072212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:04.173563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074079+0000","last_change":"2024-05-09T03:15:52.078197+0000","last_active":"2024-05-09T03:15:54.074079+0000","last_peered":"2024-05-09T03:15:54.074079+0000","last_clean":"2024-05-09T03:15:54.074079+0000","last_became_active":"2024-05-09T03:15:52.077834+0000","last_became_peered":"2024-05-09T03:15:52.077834+0000","last_unstale":"2024-05-09T03:15:54.074079+0000","last_undegraded":"2024-05-09T03:15:54.074079+0000","last_fullsized":"2024-05-09T03:15:54.074079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:22.187664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061241+0000","last_change":"2024-05-09T03:15:53.073487+0000","last_active":"2024-05-09T03:15:55.061241+0000","last_peered":"2024-05-09T03:15:55.061241+0000","last_clean":"2024-05-09T03:15:55.061241+0000","last_became_active":"2024-05-09T03:15:53.073184+0000","last_became_peered":"2024-05-09T03:15:53.073184+0000","last_unstale":"2024-05-09T03:15:55.061241+0000","last_undegraded":"2024-05-09T03:15:55.061241+0000","last_fullsized":"2024-05-09T03:15:55.061241+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:21.730334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073564+0000","last_change":"2024-05-09T03:15:52.069881+0000","last_active":"2024-05-09T03:15:54.073564+0000","last_peered":"2024-05-09T03:15:54.073564+0000","last_clean":"2024-05-09T03:15:54.073564+0000","last_became_active":"2024-05-09T03:15:52.069547+0000","last_became_peered":"2024-05-09T03:15:52.069547+0000","last_unstale":"2024-05-09T03:15:54.073564+0000","last_undegraded":"2024-05-09T03:15:54.073564+0000","last_fullsized":"2024-05-09T03:15:54.073564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:25.705744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073502+0000","last_change":"2024-05-09T03:15:53.077445+0000","last_active":"2024-05-09T03:15:54.073502+0000","last_peered":"2024-05-09T03:15:54.073502+0000","last_clean":"2024-05-09T03:15:54.073502+0000","last_became_active":"2024-05-09T03:15:53.077112+0000","last_became_peered":"2024-05-09T03:15:53.077112+0000","last_unstale":"2024-05-09T03:15:54.073502+0000","last_undegraded":"2024-05-09T03:15:54.073502+0000","last_fullsized":"2024-05-09T03:15:54.073502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:22:10.833765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074151+0000","last_change":"2024-05-09T03:15:52.066053+0000","last_active":"2024-05-09T03:15:54.074151+0000","last_peered":"2024-05-09T03:15:54.074151+0000","last_clean":"2024-05-09T03:15:54.074151+0000","last_became_active":"2024-05-09T03:15:52.065728+0000","last_became_peered":"2024-05-09T03:15:52.065728+0000","last_unstale":"2024-05-09T03:15:54.074151+0000","last_undegraded":"2024-05-09T03:15:54.074151+0000","last_fullsized":"2024-05-09T03:15:54.074151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:11.847485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062506+0000","last_change":"2024-05-09T03:15:53.069613+0000","last_active":"2024-05-09T03:15:55.062506+0000","last_peered":"2024-05-09T03:15:55.062506+0000","last_clean":"2024-05-09T03:15:55.062506+0000","last_became_active":"2024-05-09T03:15:53.068488+0000","last_became_peered":"2024-05-09T03:15:53.068488+0000","last_unstale":"2024-05-09T03:15:55.062506+0000","last_undegraded":"2024-05-09T03:15:55.062506+0000","last_fullsized":"2024-05-09T03:15:55.062506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:12.855643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061832+0000","last_change":"2024-05-09T03:15:52.072040+0000","last_active":"2024-05-09T03:15:55.061832+0000","last_peered":"2024-05-09T03:15:55.061832+0000","last_clean":"2024-05-09T03:15:55.061832+0000","last_became_active":"2024-05-09T03:15:52.071729+0000","last_became_peered":"2024-05-09T03:15:52.071729+0000","last_unstale":"2024-05-09T03:15:55.061832+0000","last_undegraded":"2024-05-09T03:15:55.061832+0000","last_fullsized":"2024-05-09T03:15:55.061832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:46.656775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":227,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061243+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:15:55.061243+0000","last_peered":"2024-05-09T03:15:55.061243+0000","last_clean":"2024-05-09T03:15:55.061243+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:15:55.061243+0000","last_undegraded":"2024-05-09T03:15:55.061243+0000","last_fullsized":"2024-05-09T03:15:55.061243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574930,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1022117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574930,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":432373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1024000,"data_stored":433313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":431679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":433291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574930,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":432261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":431873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574930,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1022081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:15:57.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:15:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.307+0000 7ff1e8053640 1 Processor -- start 2024-05-09T03:15:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.307+0000 7ff1e8053640 1 -- start start 2024-05-09T03:15:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.308+0000 7ff1e8053640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1e00d57a0 0x7ff1e0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.308+0000 7ff1e8053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 0x7ff1e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.308+0000 7ff1e8053640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e00d49e0 con 0x7ff1e00b77e0 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.308+0000 7ff1e8053640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff1e00d5bb0 con 0x7ff1e00d57a0 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.308+0000 7ff1e8053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e00d4cc0 con 0x7ff1e00b4af0 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e55ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 0x7ff1e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e5def640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1e00d57a0 0x7ff1e0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e65f0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff1e00b77e0 0x7ff1e00b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:32828/0 (socket says 172.21.15.46:32828) 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e55ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 0x7ff1e00b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:42984/0 (socket says 172.21.15.46:42984) 2024-05-09T03:15:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e65f0640 1 -- 172.21.15.46:0/676492514 learned_addr learned my addr 172.21.15.46:0/676492514 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e55ee640 1 -- 172.21.15.46:0/676492514 >> v2:172.21.15.88:3300/0 conn(0x7ff1e00d57a0 msgr2=0x7ff1e0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e55ee640 1 --2- 172.21.15.46:0/676492514 >> v2:172.21.15.88:3300/0 conn(0x7ff1e00d57a0 0x7ff1e0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e55ee640 1 -- 172.21.15.46:0/676492514 >> v1:172.21.15.88:6789/0 conn(0x7ff1e00b77e0 legacy=0x7ff1e00b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e55ee640 1 -- 172.21.15.46:0/676492514 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e00d4fa0 con 0x7ff1e00b4af0 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.309+0000 7ff1e5def640 1 --2- 172.21.15.46:0/676492514 >> v2:172.21.15.88:3300/0 conn(0x7ff1e00d57a0 0x7ff1e0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.310+0000 7ff1e55ee640 1 --2- 172.21.15.46:0/676492514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 0x7ff1e00b4f00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00097a0 tx=0x7ff1d002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a667b3412ada2f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.310+0000 7ff1e4ded640 1 -- 172.21.15.46:0/676492514 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0037070 con 0x7ff1e00b4af0 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.310+0000 7ff1e4ded640 1 -- 172.21.15.46:0/676492514 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d0032440 con 0x7ff1e00b4af0 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.310+0000 7ff1e4ded640 1 -- 172.21.15.46:0/676492514 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0032810 con 0x7ff1e00b4af0 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 -- 172.21.15.46:0/676492514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 msgr2=0x7ff1e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 --2- 172.21.15.46:0/676492514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 0x7ff1e00b4f00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00097a0 tx=0x7ff1d002a180 comp rx=0 tx=0).stop 2024-05-09T03:15:57.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 -- 172.21.15.46:0/676492514 shutdown_connections 2024-05-09T03:15:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 --2- 172.21.15.46:0/676492514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b4af0 0x7ff1e00b4f00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 --2- 172.21.15.46:0/676492514 >> v2:172.21.15.88:3300/0 conn(0x7ff1e00d57a0 0x7ff1e0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 -- 172.21.15.46:0/676492514 >> 172.21.15.46:0/676492514 conn(0x7ff1e00743e0 msgr2=0x7ff1e00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 -- 172.21.15.46:0/676492514 shutdown_connections 2024-05-09T03:15:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.311+0000 7ff1e8053640 1 -- 172.21.15.46:0/676492514 wait complete. 2024-05-09T03:15:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.313+0000 7ff1e8053640 1 Processor -- start 2024-05-09T03:15:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.313+0000 7ff1e8053640 1 -- start start 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e8053640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1e00b4af0 0x7ff1e014f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e8053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b77e0 0x7ff1e015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e8053640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 0x7ff1e015c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e8053640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e00d1920 con 0x7ff1e00d57a0 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e8053640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e00d15b0 con 0x7ff1e00b4af0 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e8053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e01f7b90 con 0x7ff1e00b77e0 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e65f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 0x7ff1e015c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e65f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 0x7ff1e015c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55338/0 (socket says 172.21.15.46:55338) 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e65f0640 1 -- 172.21.15.46:0/2679829177 learned_addr learned my addr 172.21.15.46:0/2679829177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e55ee640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b77e0 0x7ff1e015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.314+0000 7ff1e5def640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1e00b4af0 0x7ff1e014f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e65f0640 1 -- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b77e0 msgr2=0x7ff1e015c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e65f0640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b77e0 0x7ff1e015c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e65f0640 1 -- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1e00b4af0 msgr2=0x7ff1e014f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e65f0640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1e00b4af0 0x7ff1e014f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e65f0640 1 -- 172.21.15.46:0/2679829177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e01e9d20 con 0x7ff1e00d57a0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e65f0640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 0x7ff1e015c600 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800cab0 tx=0x7ff1d800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8017070 con 0x7ff1e00d57a0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d800ac30 con 0x7ff1e00d57a0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8012400 con 0x7ff1e00d57a0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e01ea000 con 0x7ff1e00d57a0 2024-05-09T03:15:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.315+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01ea5c0 con 0x7ff1e00d57a0 2024-05-09T03:15:57.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.316+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e01e9a80 con 0x7ff1e00d57a0 2024-05-09T03:15:57.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.317+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff1d801a070 con 0x7ff1e00d57a0 2024-05-09T03:15:57.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.318+0000 7ff1d67fc640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1c4076300 0x7ff1c4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.318+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7ff1d8097330 con 0x7ff1e00d57a0 2024-05-09T03:15:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.318+0000 7ff1e5def640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1c4076300 0x7ff1c4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.318+0000 7ff1e5def640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1c4076300 0x7ff1c4078810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff1e006d3b0 tx=0x7ff1dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:57.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.323+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff1e01e9a80 con 0x7ff1e00d57a0 2024-05-09T03:15:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.508+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1e006a270 con 0x7ff1c4076300 2024-05-09T03:15:57.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.515+0000 7ff1d67fc640 1 -- 172.21.15.46:0/2679829177 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252070 (secure 0 0 0) 0x7ff1e006a270 con 0x7ff1c4076300 2024-05-09T03:15:57.518 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:57.519 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1c4076300 msgr2=0x7ff1c4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1c4076300 0x7ff1c4078810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff1e006d3b0 tx=0x7ff1dc008040 comp rx=0 tx=0).stop 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 msgr2=0x7ff1e015c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 0x7ff1e015c600 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800cab0 tx=0x7ff1d800cf70 comp rx=0 tx=0).stop 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 shutdown_connections 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1c4076300 0x7ff1c4078810 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e00d57a0 0x7ff1e015c600 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1e00b77e0 0x7ff1e015c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 --2- 172.21.15.46:0/2679829177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1e00b4af0 0x7ff1e014f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 >> 172.21.15.46:0/2679829177 conn(0x7ff1e00743e0 msgr2=0x7ff1e00baa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:57.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 shutdown_connections 2024-05-09T03:15:57.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.518+0000 7ff1e8053640 1 -- 172.21.15.46:0/2679829177 wait complete. 2024-05-09T03:15:57.530 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-05-09T03:15:56.593628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254384,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754720336,"statfs":{"total":773094113280,"available":772833624064,"internally_reserved":0,"allocated":9314304,"data_stored":4638988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.551088"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074096+0000","last_change":"2024-05-09T03:15:53.069947+0000","last_active":"2024-05-09T03:15:54.074096+0000","last_peered":"2024-05-09T03:15:54.074096+0000","last_clean":"2024-05-09T03:15:54.074096+0000","last_became_active":"2024-05-09T03:15:53.069585+0000","last_became_peered":"2024-05-09T03:15:53.069585+0000","last_unstale":"2024-05-09T03:15:54.074096+0000","last_undegraded":"2024-05-09T03:15:54.074096+0000","last_fullsized":"2024-05-09T03:15:54.074096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:04.173563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063352+0000","last_change":"2024-05-09T03:15:53.076051+0000","last_active":"2024-05-09T03:15:55.063352+0000","last_peered":"2024-05-09T03:15:55.063352+0000","last_clean":"2024-05-09T03:15:55.063352+0000","last_became_active":"2024-05-09T03:15:53.074901+0000","last_became_peered":"2024-05-09T03:15:53.074901+0000","last_unstale":"2024-05-09T03:15:55.063352+0000","last_undegraded":"2024-05-09T03:15:55.063352+0000","last_fullsized":"2024-05-09T03:15:55.063352+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:23.219622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061484+0000","last_change":"2024-05-09T03:15:53.074723+0000","last_active":"2024-05-09T03:15:55.061484+0000","last_peered":"2024-05-09T03:15:55.061484+0000","last_clean":"2024-05-09T03:15:55.061484+0000","last_became_active":"2024-05-09T03:15:53.074398+0000","last_became_peered":"2024-05-09T03:15:53.074398+0000","last_unstale":"2024-05-09T03:15:55.061484+0000","last_undegraded":"2024-05-09T03:15:55.061484+0000","last_fullsized":"2024-05-09T03:15:55.061484+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:57.947878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062918+0000","last_change":"2024-05-09T03:15:53.084927+0000","last_active":"2024-05-09T03:15:55.062918+0000","last_peered":"2024-05-09T03:15:55.062918+0000","last_clean":"2024-05-09T03:15:55.062918+0000","last_became_active":"2024-05-09T03:15:53.084593+0000","last_became_peered":"2024-05-09T03:15:53.084593+0000","last_unstale":"2024-05-09T03:15:55.062918+0000","last_undegraded":"2024-05-09T03:15:55.062918+0000","last_fullsized":"2024-05-09T03:15:55.062918+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:06:03.714042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075333+0000","last_change":"2024-05-09T03:15:53.078091+0000","last_active":"2024-05-09T03:15:54.075333+0000","last_peered":"2024-05-09T03:15:54.075333+0000","last_clean":"2024-05-09T03:15:54.075333+0000","last_became_active":"2024-05-09T03:15:53.077791+0000","last_became_peered":"2024-05-09T03:15:53.077791+0000","last_unstale":"2024-05-09T03:15:54.075333+0000","last_undegraded":"2024-05-09T03:15:54.075333+0000","last_fullsized":"2024-05-09T03:15:54.075333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:46:23.557851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061393+0000","last_change":"2024-05-09T03:15:53.074257+0000","last_active":"2024-05-09T03:15:55.061393+0000","last_peered":"2024-05-09T03:15:55.061393+0000","last_clean":"2024-05-09T03:15:55.061393+0000","last_became_active":"2024-05-09T03:15:53.073905+0000","last_became_peered":"2024-05-09T03:15:53.073905+0000","last_unstale":"2024-05-09T03:15:55.061393+0000","last_undegraded":"2024-05-09T03:15:55.061393+0000","last_fullsized":"2024-05-09T03:15:55.061393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:29:56.338990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061782+0000","last_change":"2024-05-09T03:15:53.074885+0000","last_active":"2024-05-09T03:15:55.061782+0000","last_peered":"2024-05-09T03:15:55.061782+0000","last_clean":"2024-05-09T03:15:55.061782+0000","last_became_active":"2024-05-09T03:15:53.074505+0000","last_became_peered":"2024-05-09T03:15:53.074505+0000","last_unstale":"2024-05-09T03:15:55.061782+0000","last_undegraded":"2024-05-09T03:15:55.061782+0000","last_fullsized":"2024-05-09T03:15:55.061782+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:15.192204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061173+0000","last_change":"2024-05-09T03:15:53.071422+0000","last_active":"2024-05-09T03:15:55.061173+0000","last_peered":"2024-05-09T03:15:55.061173+0000","last_clean":"2024-05-09T03:15:55.061173+0000","last_became_active":"2024-05-09T03:15:53.071064+0000","last_became_peered":"2024-05-09T03:15:53.071064+0000","last_unstale":"2024-05-09T03:15:55.061173+0000","last_undegraded":"2024-05-09T03:15:55.061173+0000","last_fullsized":"2024-05-09T03:15:55.061173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:59.610027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061609+0000","last_change":"2024-05-09T03:15:53.074959+0000","last_active":"2024-05-09T03:15:55.061609+0000","last_peered":"2024-05-09T03:15:55.061609+0000","last_clean":"2024-05-09T03:15:55.061609+0000","last_became_active":"2024-05-09T03:15:53.074603+0000","last_became_peered":"2024-05-09T03:15:53.074603+0000","last_unstale":"2024-05-09T03:15:55.061609+0000","last_undegraded":"2024-05-09T03:15:55.061609+0000","last_fullsized":"2024-05-09T03:15:55.061609+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:51.783454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075062+0000","last_change":"2024-05-09T03:15:53.077170+0000","last_active":"2024-05-09T03:15:54.075062+0000","last_peered":"2024-05-09T03:15:54.075062+0000","last_clean":"2024-05-09T03:15:54.075062+0000","last_became_active":"2024-05-09T03:15:53.076852+0000","last_became_peered":"2024-05-09T03:15:53.076852+0000","last_unstale":"2024-05-09T03:15:54.075062+0000","last_undegraded":"2024-05-09T03:15:54.075062+0000","last_fullsized":"2024-05-09T03:15:54.075062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:21.730334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062728+0000","last_change":"2024-05-09T03:15:53.067947+0000","last_active":"2024-05-09T03:15:55.062728+0000","last_peered":"2024-05-09T03:15:55.062728+0000","last_clean":"2024-05-09T03:15:55.062728+0000","last_became_active":"2024-05-09T03:15:53.067669+0000","last_became_peered":"2024-05-09T03:15:53.067669+0000","last_unstale":"2024-05-09T03:15:55.062728+0000","last_undegraded":"2024-05-09T03:15:55.062728+0000","last_fullsized":"2024-05-09T03:15:55.062728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:26.666719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073846+0000","last_change":"2024-05-09T03:15:53.076997+0000","last_active":"2024-05-09T03:15:54.073846+0000","last_peered":"2024-05-09T03:15:54.073846+0000","last_clean":"2024-05-09T03:15:54.073846+0000","last_became_active":"2024-05-09T03:15:53.075380+0000","last_became_peered":"2024-05-09T03:15:53.075380+0000","last_unstale":"2024-05-09T03:15:54.073846+0000","last_undegraded":"2024-05-09T03:15:54.073846+0000","last_fullsized":"2024-05-09T03:15:54.073846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:58.790752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061518+0000","last_change":"2024-05-09T03:15:53.071408+0000","last_active":"2024-05-09T03:15:55.061518+0000","last_peered":"2024-05-09T03:15:55.061518+0000","last_clean":"2024-05-09T03:15:55.061518+0000","last_became_active":"2024-05-09T03:15:53.071078+0000","last_became_peered":"2024-05-09T03:15:53.071078+0000","last_unstale":"2024-05-09T03:15:55.061518+0000","last_undegraded":"2024-05-09T03:15:55.061518+0000","last_fullsized":"2024-05-09T03:15:55.061518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:19:31.160437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062295+0000","last_change":"2024-05-09T03:15:53.085813+0000","last_active":"2024-05-09T03:15:55.062295+0000","last_peered":"2024-05-09T03:15:55.062295+0000","last_clean":"2024-05-09T03:15:55.062295+0000","last_became_active":"2024-05-09T03:15:53.084981+0000","last_became_peered":"2024-05-09T03:15:53.084981+0000","last_unstale":"2024-05-09T03:15:55.062295+0000","last_undegraded":"2024-05-09T03:15:55.062295+0000","last_fullsized":"2024-05-09T03:15:55.062295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:27:48.595005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062135+0000","last_change":"2024-05-09T03:15:53.076642+0000","last_active":"2024-05-09T03:15:55.062135+0000","last_peered":"2024-05-09T03:15:55.062135+0000","last_clean":"2024-05-09T03:15:55.062135+0000","last_became_active":"2024-05-09T03:15:53.076274+0000","last_became_peered":"2024-05-09T03:15:53.076274+0000","last_unstale":"2024-05-09T03:15:55.062135+0000","last_undegraded":"2024-05-09T03:15:55.062135+0000","last_fullsized":"2024-05-09T03:15:55.062135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:07.829466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073995+0000","last_change":"2024-05-09T03:15:53.077474+0000","last_active":"2024-05-09T03:15:54.073995+0000","last_peered":"2024-05-09T03:15:54.073995+0000","last_clean":"2024-05-09T03:15:54.073995+0000","last_became_active":"2024-05-09T03:15:53.077140+0000","last_became_peered":"2024-05-09T03:15:53.077140+0000","last_unstale":"2024-05-09T03:15:54.073995+0000","last_undegraded":"2024-05-09T03:15:54.073995+0000","last_fullsized":"2024-05-09T03:15:54.073995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:49:24.252368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074972+0000","last_change":"2024-05-09T03:15:53.075806+0000","last_active":"2024-05-09T03:15:54.074972+0000","last_peered":"2024-05-09T03:15:54.074972+0000","last_clean":"2024-05-09T03:15:54.074972+0000","last_became_active":"2024-05-09T03:15:53.075367+0000","last_became_peered":"2024-05-09T03:15:53.075367+0000","last_unstale":"2024-05-09T03:15:54.074972+0000","last_undegraded":"2024-05-09T03:15:54.074972+0000","last_fullsized":"2024-05-09T03:15:54.074972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:18.363406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062441+0000","last_change":"2024-05-09T03:15:53.077550+0000","last_active":"2024-05-09T03:15:55.062441+0000","last_peered":"2024-05-09T03:15:55.062441+0000","last_clean":"2024-05-09T03:15:55.062441+0000","last_became_active":"2024-05-09T03:15:53.077229+0000","last_became_peered":"2024-05-09T03:15:53.077229+0000","last_unstale":"2024-05-09T03:15:55.062441+0000","last_undegraded":"2024-05-09T03:15:55.062441+0000","last_fullsized":"2024-05-09T03:15:55.062441+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:41:49.047471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072510+0000","last_change":"2024-05-09T03:15:53.086603+0000","last_active":"2024-05-09T03:15:54.072510+0000","last_peered":"2024-05-09T03:15:54.072510+0000","last_clean":"2024-05-09T03:15:54.072510+0000","last_became_active":"2024-05-09T03:15:53.086273+0000","last_became_peered":"2024-05-09T03:15:53.086273+0000","last_unstale":"2024-05-09T03:15:54.072510+0000","last_undegraded":"2024-05-09T03:15:54.072510+0000","last_fullsized":"2024-05-09T03:15:54.072510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:52.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062477+0000","last_change":"2024-05-09T03:15:53.086167+0000","last_active":"2024-05-09T03:15:55.062477+0000","last_peered":"2024-05-09T03:15:55.062477+0000","last_clean":"2024-05-09T03:15:55.062477+0000","last_became_active":"2024-05-09T03:15:53.085841+0000","last_became_peered":"2024-05-09T03:15:53.085841+0000","last_unstale":"2024-05-09T03:15:55.062477+0000","last_undegraded":"2024-05-09T03:15:55.062477+0000","last_fullsized":"2024-05-09T03:15:55.062477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:07:09.995250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062838+0000","last_change":"2024-05-09T03:15:53.088267+0000","last_active":"2024-05-09T03:15:55.062838+0000","last_peered":"2024-05-09T03:15:55.062838+0000","last_clean":"2024-05-09T03:15:55.062838+0000","last_became_active":"2024-05-09T03:15:53.087948+0000","last_became_peered":"2024-05-09T03:15:53.087948+0000","last_unstale":"2024-05-09T03:15:55.062838+0000","last_undegraded":"2024-05-09T03:15:55.062838+0000","last_fullsized":"2024-05-09T03:15:55.062838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:59:06.761713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063082+0000","last_change":"2024-05-09T03:15:53.083400+0000","last_active":"2024-05-09T03:15:55.063082+0000","last_peered":"2024-05-09T03:15:55.063082+0000","last_clean":"2024-05-09T03:15:55.063082+0000","last_became_active":"2024-05-09T03:15:53.082697+0000","last_became_peered":"2024-05-09T03:15:53.082697+0000","last_unstale":"2024-05-09T03:15:55.063082+0000","last_undegraded":"2024-05-09T03:15:55.063082+0000","last_fullsized":"2024-05-09T03:15:55.063082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:41.831877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073504+0000","last_change":"2024-05-09T03:15:53.083463+0000","last_active":"2024-05-09T03:15:54.073504+0000","last_peered":"2024-05-09T03:15:54.073504+0000","last_clean":"2024-05-09T03:15:54.073504+0000","last_became_active":"2024-05-09T03:15:53.083009+0000","last_became_peered":"2024-05-09T03:15:53.083009+0000","last_unstale":"2024-05-09T03:15:54.073504+0000","last_undegraded":"2024-05-09T03:15:54.073504+0000","last_fullsized":"2024-05-09T03:15:54.073504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:24.180236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061931+0000","last_change":"2024-05-09T03:15:53.089084+0000","last_active":"2024-05-09T03:15:55.061931+0000","last_peered":"2024-05-09T03:15:55.061931+0000","last_clean":"2024-05-09T03:15:55.061931+0000","last_became_active":"2024-05-09T03:15:53.088793+0000","last_became_peered":"2024-05-09T03:15:53.088793+0000","last_unstale":"2024-05-09T03:15:55.061931+0000","last_undegraded":"2024-05-09T03:15:55.061931+0000","last_fullsized":"2024-05-09T03:15:55.061931+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:49:21.018337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062576+0000","last_change":"2024-05-09T03:15:53.069817+0000","last_active":"2024-05-09T03:15:55.062576+0000","last_peered":"2024-05-09T03:15:55.062576+0000","last_clean":"2024-05-09T03:15:55.062576+0000","last_became_active":"2024-05-09T03:15:53.069459+0000","last_became_peered":"2024-05-09T03:15:53.069459+0000","last_unstale":"2024-05-09T03:15:55.062576+0000","last_undegraded":"2024-05-09T03:15:55.062576+0000","last_fullsized":"2024-05-09T03:15:55.062576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:06.776516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061973+0000","last_change":"2024-05-09T03:15:53.081845+0000","last_active":"2024-05-09T03:15:55.061973+0000","last_peered":"2024-05-09T03:15:55.061973+0000","last_clean":"2024-05-09T03:15:55.061973+0000","last_became_active":"2024-05-09T03:15:53.081501+0000","last_became_peered":"2024-05-09T03:15:53.081501+0000","last_unstale":"2024-05-09T03:15:55.061973+0000","last_undegraded":"2024-05-09T03:15:55.061973+0000","last_fullsized":"2024-05-09T03:15:55.061973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:25:45.894851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074597+0000","last_change":"2024-05-09T03:15:53.077829+0000","last_active":"2024-05-09T03:15:54.074597+0000","last_peered":"2024-05-09T03:15:54.074597+0000","last_clean":"2024-05-09T03:15:54.074597+0000","last_became_active":"2024-05-09T03:15:53.077499+0000","last_became_peered":"2024-05-09T03:15:53.077499+0000","last_unstale":"2024-05-09T03:15:54.074597+0000","last_undegraded":"2024-05-09T03:15:54.074597+0000","last_fullsized":"2024-05-09T03:15:54.074597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:17.882544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073600+0000","last_change":"2024-05-09T03:15:53.069972+0000","last_active":"2024-05-09T03:15:54.073600+0000","last_peered":"2024-05-09T03:15:54.073600+0000","last_clean":"2024-05-09T03:15:54.073600+0000","last_became_active":"2024-05-09T03:15:53.069637+0000","last_became_peered":"2024-05-09T03:15:53.069637+0000","last_unstale":"2024-05-09T03:15:54.073600+0000","last_undegraded":"2024-05-09T03:15:54.073600+0000","last_fullsized":"2024-05-09T03:15:54.073600+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:40:40.889305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062296+0000","last_change":"2024-05-09T03:15:53.077663+0000","last_active":"2024-05-09T03:15:55.062296+0000","last_peered":"2024-05-09T03:15:55.062296+0000","last_clean":"2024-05-09T03:15:55.062296+0000","last_became_active":"2024-05-09T03:15:53.077336+0000","last_became_peered":"2024-05-09T03:15:53.077336+0000","last_unstale":"2024-05-09T03:15:55.062296+0000","last_undegraded":"2024-05-09T03:15:55.062296+0000","last_fullsized":"2024-05-09T03:15:55.062296+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:03.516637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063282+0000","last_change":"2024-05-09T03:15:53.084489+0000","last_active":"2024-05-09T03:15:55.063282+0000","last_peered":"2024-05-09T03:15:55.063282+0000","last_clean":"2024-05-09T03:15:55.063282+0000","last_became_active":"2024-05-09T03:15:53.084126+0000","last_became_peered":"2024-05-09T03:15:53.084126+0000","last_unstale":"2024-05-09T03:15:55.063282+0000","last_undegraded":"2024-05-09T03:15:55.063282+0000","last_fullsized":"2024-05-09T03:15:55.063282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:26.713902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074805+0000","last_change":"2024-05-09T03:15:53.082583+0000","last_active":"2024-05-09T03:15:54.074805+0000","last_peered":"2024-05-09T03:15:54.074805+0000","last_clean":"2024-05-09T03:15:54.074805+0000","last_became_active":"2024-05-09T03:15:53.081246+0000","last_became_peered":"2024-05-09T03:15:53.081246+0000","last_unstale":"2024-05-09T03:15:54.074805+0000","last_undegraded":"2024-05-09T03:15:54.074805+0000","last_fullsized":"2024-05-09T03:15:54.074805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:22:10.833765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073546+0000","last_change":"2024-05-09T03:15:53.072372+0000","last_active":"2024-05-09T03:15:54.073546+0000","last_peered":"2024-05-09T03:15:54.073546+0000","last_clean":"2024-05-09T03:15:54.073546+0000","last_became_active":"2024-05-09T03:15:53.072031+0000","last_became_peered":"2024-05-09T03:15:53.072031+0000","last_unstale":"2024-05-09T03:15:54.073546+0000","last_undegraded":"2024-05-09T03:15:54.073546+0000","last_fullsized":"2024-05-09T03:15:54.073546+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:46:18.363406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074510+0000","last_change":"2024-05-09T03:15:53.078686+0000","last_active":"2024-05-09T03:15:54.074510+0000","last_peered":"2024-05-09T03:15:54.074510+0000","last_clean":"2024-05-09T03:15:54.074510+0000","last_became_active":"2024-05-09T03:15:53.077760+0000","last_became_peered":"2024-05-09T03:15:53.077760+0000","last_unstale":"2024-05-09T03:15:54.074510+0000","last_undegraded":"2024-05-09T03:15:54.074510+0000","last_fullsized":"2024-05-09T03:15:54.074510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:47.664933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074552+0000","last_change":"2024-05-09T03:15:52.068420+0000","last_active":"2024-05-09T03:15:54.074552+0000","last_peered":"2024-05-09T03:15:54.074552+0000","last_clean":"2024-05-09T03:15:54.074552+0000","last_became_active":"2024-05-09T03:15:52.068090+0000","last_became_peered":"2024-05-09T03:15:52.068090+0000","last_unstale":"2024-05-09T03:15:54.074552+0000","last_undegraded":"2024-05-09T03:15:54.074552+0000","last_fullsized":"2024-05-09T03:15:54.074552+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:05.768358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074289+0000","last_change":"2024-05-09T03:15:53.083690+0000","last_active":"2024-05-09T03:15:54.074289+0000","last_peered":"2024-05-09T03:15:54.074289+0000","last_clean":"2024-05-09T03:15:54.074289+0000","last_became_active":"2024-05-09T03:15:53.083068+0000","last_became_peered":"2024-05-09T03:15:53.083068+0000","last_unstale":"2024-05-09T03:15:54.074289+0000","last_undegraded":"2024-05-09T03:15:54.074289+0000","last_fullsized":"2024-05-09T03:15:54.074289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:26.794178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074584+0000","last_change":"2024-05-09T03:15:52.078222+0000","last_active":"2024-05-09T03:15:54.074584+0000","last_peered":"2024-05-09T03:15:54.074584+0000","last_clean":"2024-05-09T03:15:54.074584+0000","last_became_active":"2024-05-09T03:15:52.077869+0000","last_became_peered":"2024-05-09T03:15:52.077869+0000","last_unstale":"2024-05-09T03:15:54.074584+0000","last_undegraded":"2024-05-09T03:15:54.074584+0000","last_fullsized":"2024-05-09T03:15:54.074584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:02.508479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075148+0000","last_change":"2024-05-09T03:15:53.071266+0000","last_active":"2024-05-09T03:15:54.075148+0000","last_peered":"2024-05-09T03:15:54.075148+0000","last_clean":"2024-05-09T03:15:54.075148+0000","last_became_active":"2024-05-09T03:15:53.070975+0000","last_became_peered":"2024-05-09T03:15:53.070975+0000","last_unstale":"2024-05-09T03:15:54.075148+0000","last_undegraded":"2024-05-09T03:15:54.075148+0000","last_fullsized":"2024-05-09T03:15:54.075148+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:57.947878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072409+0000","last_change":"2024-05-09T03:15:52.073336+0000","last_active":"2024-05-09T03:15:54.072409+0000","last_peered":"2024-05-09T03:15:54.072409+0000","last_clean":"2024-05-09T03:15:54.072409+0000","last_became_active":"2024-05-09T03:15:52.073045+0000","last_became_peered":"2024-05-09T03:15:52.073045+0000","last_unstale":"2024-05-09T03:15:54.072409+0000","last_undegraded":"2024-05-09T03:15:54.072409+0000","last_fullsized":"2024-05-09T03:15:54.072409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:49:23.244210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061576+0000","last_change":"2024-05-09T03:15:53.076120+0000","last_active":"2024-05-09T03:15:55.061576+0000","last_peered":"2024-05-09T03:15:55.061576+0000","last_clean":"2024-05-09T03:15:55.061576+0000","last_became_active":"2024-05-09T03:15:53.075785+0000","last_became_peered":"2024-05-09T03:15:53.075785+0000","last_unstale":"2024-05-09T03:15:55.061576+0000","last_undegraded":"2024-05-09T03:15:55.061576+0000","last_fullsized":"2024-05-09T03:15:55.061576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:51:33.204849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072399+0000","last_change":"2024-05-09T03:15:52.075163+0000","last_active":"2024-05-09T03:15:54.072399+0000","last_peered":"2024-05-09T03:15:54.072399+0000","last_clean":"2024-05-09T03:15:54.072399+0000","last_became_active":"2024-05-09T03:15:52.074830+0000","last_became_peered":"2024-05-09T03:15:52.074830+0000","last_unstale":"2024-05-09T03:15:54.072399+0000","last_undegraded":"2024-05-09T03:15:54.072399+0000","last_fullsized":"2024-05-09T03:15:54.072399+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:05.694999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063491+0000","last_change":"2024-05-09T03:15:53.083722+0000","last_active":"2024-05-09T03:15:55.063491+0000","last_peered":"2024-05-09T03:15:55.063491+0000","last_clean":"2024-05-09T03:15:55.063491+0000","last_became_active":"2024-05-09T03:15:53.083100+0000","last_became_peered":"2024-05-09T03:15:53.083100+0000","last_unstale":"2024-05-09T03:15:55.063491+0000","last_undegraded":"2024-05-09T03:15:55.063491+0000","last_fullsized":"2024-05-09T03:15:55.063491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:18:38.837246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075433+0000","last_change":"2024-05-09T03:15:52.059453+0000","last_active":"2024-05-09T03:15:54.075433+0000","last_peered":"2024-05-09T03:15:54.075433+0000","last_clean":"2024-05-09T03:15:54.075433+0000","last_became_active":"2024-05-09T03:15:52.058216+0000","last_became_peered":"2024-05-09T03:15:52.058216+0000","last_unstale":"2024-05-09T03:15:54.075433+0000","last_undegraded":"2024-05-09T03:15:54.075433+0000","last_fullsized":"2024-05-09T03:15:54.075433+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:22:05.768358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075355+0000","last_change":"2024-05-09T03:15:53.075244+0000","last_active":"2024-05-09T03:15:54.075355+0000","last_peered":"2024-05-09T03:15:54.075355+0000","last_clean":"2024-05-09T03:15:54.075355+0000","last_became_active":"2024-05-09T03:15:53.074900+0000","last_became_peered":"2024-05-09T03:15:53.074900+0000","last_unstale":"2024-05-09T03:15:54.075355+0000","last_undegraded":"2024-05-09T03:15:54.075355+0000","last_fullsized":"2024-05-09T03:15:54.075355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:04.173563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073951+0000","last_change":"2024-05-09T03:15:52.078882+0000","last_active":"2024-05-09T03:15:54.073951+0000","last_peered":"2024-05-09T03:15:54.073951+0000","last_clean":"2024-05-09T03:15:54.073951+0000","last_became_active":"2024-05-09T03:15:52.078565+0000","last_became_peered":"2024-05-09T03:15:52.078565+0000","last_unstale":"2024-05-09T03:15:54.073951+0000","last_undegraded":"2024-05-09T03:15:54.073951+0000","last_fullsized":"2024-05-09T03:15:54.073951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:18:29.836613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061169+0000","last_change":"2024-05-09T03:15:53.080971+0000","last_active":"2024-05-09T03:15:55.061169+0000","last_peered":"2024-05-09T03:15:55.061169+0000","last_clean":"2024-05-09T03:15:55.061169+0000","last_became_active":"2024-05-09T03:15:53.079680+0000","last_became_peered":"2024-05-09T03:15:53.079680+0000","last_unstale":"2024-05-09T03:15:55.061169+0000","last_undegraded":"2024-05-09T03:15:55.061169+0000","last_fullsized":"2024-05-09T03:15:55.061169+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:52.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.071939+0000","last_change":"2024-05-09T03:15:52.072230+0000","last_active":"2024-05-09T03:15:54.071939+0000","last_peered":"2024-05-09T03:15:54.071939+0000","last_clean":"2024-05-09T03:15:54.071939+0000","last_became_active":"2024-05-09T03:15:52.071810+0000","last_became_peered":"2024-05-09T03:15:52.071810+0000","last_unstale":"2024-05-09T03:15:54.071939+0000","last_undegraded":"2024-05-09T03:15:54.071939+0000","last_fullsized":"2024-05-09T03:15:54.071939+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:30:50.775296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062341+0000","last_change":"2024-05-09T03:15:53.076664+0000","last_active":"2024-05-09T03:15:55.062341+0000","last_peered":"2024-05-09T03:15:55.062341+0000","last_clean":"2024-05-09T03:15:55.062341+0000","last_became_active":"2024-05-09T03:15:53.076351+0000","last_became_peered":"2024-05-09T03:15:53.076351+0000","last_unstale":"2024-05-09T03:15:55.062341+0000","last_undegraded":"2024-05-09T03:15:55.062341+0000","last_fullsized":"2024-05-09T03:15:55.062341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:08.551174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062075+0000","last_change":"2024-05-09T03:15:52.075417+0000","last_active":"2024-05-09T03:15:55.062075+0000","last_peered":"2024-05-09T03:15:55.062075+0000","last_clean":"2024-05-09T03:15:55.062075+0000","last_became_active":"2024-05-09T03:15:52.074250+0000","last_became_peered":"2024-05-09T03:15:52.074250+0000","last_unstale":"2024-05-09T03:15:55.062075+0000","last_undegraded":"2024-05-09T03:15:55.062075+0000","last_fullsized":"2024-05-09T03:15:55.062075+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:16:46.478388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063081+0000","last_change":"2024-05-09T03:15:53.081457+0000","last_active":"2024-05-09T03:15:55.063081+0000","last_peered":"2024-05-09T03:15:55.063081+0000","last_clean":"2024-05-09T03:15:55.063081+0000","last_became_active":"2024-05-09T03:15:53.081091+0000","last_became_peered":"2024-05-09T03:15:53.081091+0000","last_unstale":"2024-05-09T03:15:55.063081+0000","last_undegraded":"2024-05-09T03:15:55.063081+0000","last_fullsized":"2024-05-09T03:15:55.063081+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:58:20.566631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074682+0000","last_change":"2024-05-09T03:15:52.078314+0000","last_active":"2024-05-09T03:15:54.074682+0000","last_peered":"2024-05-09T03:15:54.074682+0000","last_clean":"2024-05-09T03:15:54.074682+0000","last_became_active":"2024-05-09T03:15:52.077908+0000","last_became_peered":"2024-05-09T03:15:52.077908+0000","last_unstale":"2024-05-09T03:15:54.074682+0000","last_undegraded":"2024-05-09T03:15:54.074682+0000","last_fullsized":"2024-05-09T03:15:54.074682+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:52:40.823719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061767+0000","last_change":"2024-05-09T03:15:53.074319+0000","last_active":"2024-05-09T03:15:55.061767+0000","last_peered":"2024-05-09T03:15:55.061767+0000","last_clean":"2024-05-09T03:15:55.061767+0000","last_became_active":"2024-05-09T03:15:53.073922+0000","last_became_peered":"2024-05-09T03:15:53.073922+0000","last_unstale":"2024-05-09T03:15:55.061767+0000","last_undegraded":"2024-05-09T03:15:55.061767+0000","last_fullsized":"2024-05-09T03:15:55.061767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:49:24.252368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075028+0000","last_change":"2024-05-09T03:15:52.071302+0000","last_active":"2024-05-09T03:15:54.075028+0000","last_peered":"2024-05-09T03:15:54.075028+0000","last_clean":"2024-05-09T03:15:54.075028+0000","last_became_active":"2024-05-09T03:15:52.070945+0000","last_became_peered":"2024-05-09T03:15:52.070945+0000","last_unstale":"2024-05-09T03:15:54.075028+0000","last_undegraded":"2024-05-09T03:15:54.075028+0000","last_fullsized":"2024-05-09T03:15:54.075028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:42:22.211464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072386+0000","last_change":"2024-05-09T03:15:53.091071+0000","last_active":"2024-05-09T03:15:54.072386+0000","last_peered":"2024-05-09T03:15:54.072386+0000","last_clean":"2024-05-09T03:15:54.072386+0000","last_became_active":"2024-05-09T03:15:53.089761+0000","last_became_peered":"2024-05-09T03:15:53.089761+0000","last_unstale":"2024-05-09T03:15:54.072386+0000","last_undegraded":"2024-05-09T03:15:54.072386+0000","last_fullsized":"2024-05-09T03:15:54.072386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:10:26.794178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073830+0000","last_change":"2024-05-09T03:15:52.072974+0000","last_active":"2024-05-09T03:15:54.073830+0000","last_peered":"2024-05-09T03:15:54.073830+0000","last_clean":"2024-05-09T03:15:54.073830+0000","last_became_active":"2024-05-09T03:15:52.072673+0000","last_became_peered":"2024-05-09T03:15:52.072673+0000","last_unstale":"2024-05-09T03:15:54.073830+0000","last_undegraded":"2024-05-09T03:15:54.073830+0000","last_fullsized":"2024-05-09T03:15:54.073830+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:11.847485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074819+0000","last_change":"2024-05-09T03:15:53.080769+0000","last_active":"2024-05-09T03:15:54.074819+0000","last_peered":"2024-05-09T03:15:54.074819+0000","last_clean":"2024-05-09T03:15:54.074819+0000","last_became_active":"2024-05-09T03:15:53.080150+0000","last_became_peered":"2024-05-09T03:15:53.080150+0000","last_unstale":"2024-05-09T03:15:54.074819+0000","last_undegraded":"2024-05-09T03:15:54.074819+0000","last_fullsized":"2024-05-09T03:15:54.074819+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:10:24.180236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061520+0000","last_change":"2024-05-09T03:15:52.068844+0000","last_active":"2024-05-09T03:15:55.061520+0000","last_peered":"2024-05-09T03:15:55.061520+0000","last_clean":"2024-05-09T03:15:55.061520+0000","last_became_active":"2024-05-09T03:15:52.068357+0000","last_became_peered":"2024-05-09T03:15:52.068357+0000","last_unstale":"2024-05-09T03:15:55.061520+0000","last_undegraded":"2024-05-09T03:15:55.061520+0000","last_fullsized":"2024-05-09T03:15:55.061520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:18:29.836613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072153+0000","last_change":"2024-05-09T03:15:53.091830+0000","last_active":"2024-05-09T03:15:54.072153+0000","last_peered":"2024-05-09T03:15:54.072153+0000","last_clean":"2024-05-09T03:15:54.072153+0000","last_became_active":"2024-05-09T03:15:53.091497+0000","last_became_peered":"2024-05-09T03:15:53.091497+0000","last_unstale":"2024-05-09T03:15:54.072153+0000","last_undegraded":"2024-05-09T03:15:54.072153+0000","last_fullsized":"2024-05-09T03:15:54.072153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:46:23.557851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062469+0000","last_change":"2024-05-09T03:15:52.069591+0000","last_active":"2024-05-09T03:15:55.062469+0000","last_peered":"2024-05-09T03:15:55.062469+0000","last_clean":"2024-05-09T03:15:55.062469+0000","last_became_active":"2024-05-09T03:15:52.069257+0000","last_became_peered":"2024-05-09T03:15:52.069257+0000","last_unstale":"2024-05-09T03:15:55.062469+0000","last_undegraded":"2024-05-09T03:15:55.062469+0000","last_fullsized":"2024-05-09T03:15:55.062469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:16:17.133055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061739+0000","last_change":"2024-05-09T03:15:53.079234+0000","last_active":"2024-05-09T03:15:55.061739+0000","last_peered":"2024-05-09T03:15:55.061739+0000","last_clean":"2024-05-09T03:15:55.061739+0000","last_became_active":"2024-05-09T03:15:53.078944+0000","last_became_peered":"2024-05-09T03:15:53.078944+0000","last_unstale":"2024-05-09T03:15:55.061739+0000","last_undegraded":"2024-05-09T03:15:55.061739+0000","last_fullsized":"2024-05-09T03:15:55.061739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:50:18.761398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062456+0000","last_change":"2024-05-09T03:15:52.078830+0000","last_active":"2024-05-09T03:15:55.062456+0000","last_peered":"2024-05-09T03:15:55.062456+0000","last_clean":"2024-05-09T03:15:55.062456+0000","last_became_active":"2024-05-09T03:15:52.078379+0000","last_became_peered":"2024-05-09T03:15:52.078379+0000","last_unstale":"2024-05-09T03:15:55.062456+0000","last_undegraded":"2024-05-09T03:15:55.062456+0000","last_fullsized":"2024-05-09T03:15:55.062456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:46:56.939720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062186+0000","last_change":"2024-05-09T03:15:53.076826+0000","last_active":"2024-05-09T03:15:55.062186+0000","last_peered":"2024-05-09T03:15:55.062186+0000","last_clean":"2024-05-09T03:15:55.062186+0000","last_became_active":"2024-05-09T03:15:53.074980+0000","last_became_peered":"2024-05-09T03:15:53.074980+0000","last_unstale":"2024-05-09T03:15:55.062186+0000","last_undegraded":"2024-05-09T03:15:55.062186+0000","last_fullsized":"2024-05-09T03:15:55.062186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:18:30.844771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061246+0000","last_change":"2024-05-09T03:15:52.071423+0000","last_active":"2024-05-09T03:15:55.061246+0000","last_peered":"2024-05-09T03:15:55.061246+0000","last_clean":"2024-05-09T03:15:55.061246+0000","last_became_active":"2024-05-09T03:15:52.071086+0000","last_became_peered":"2024-05-09T03:15:52.071086+0000","last_unstale":"2024-05-09T03:15:55.061246+0000","last_undegraded":"2024-05-09T03:15:55.061246+0000","last_fullsized":"2024-05-09T03:15:55.061246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:02.508479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061943+0000","last_change":"2024-05-09T03:15:53.069008+0000","last_active":"2024-05-09T03:15:55.061943+0000","last_peered":"2024-05-09T03:15:55.061943+0000","last_clean":"2024-05-09T03:15:55.061943+0000","last_became_active":"2024-05-09T03:15:53.068691+0000","last_became_peered":"2024-05-09T03:15:53.068691+0000","last_unstale":"2024-05-09T03:15:55.061943+0000","last_undegraded":"2024-05-09T03:15:55.061943+0000","last_fullsized":"2024-05-09T03:15:55.061943+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:23.195822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.071923+0000","last_change":"2024-05-09T03:15:52.070582+0000","last_active":"2024-05-09T03:15:54.071923+0000","last_peered":"2024-05-09T03:15:54.071923+0000","last_clean":"2024-05-09T03:15:54.071923+0000","last_became_active":"2024-05-09T03:15:52.070229+0000","last_became_peered":"2024-05-09T03:15:52.070229+0000","last_unstale":"2024-05-09T03:15:54.071923+0000","last_undegraded":"2024-05-09T03:15:54.071923+0000","last_fullsized":"2024-05-09T03:15:54.071923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:23:14.954104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062014+0000","last_change":"2024-05-09T03:15:53.065402+0000","last_active":"2024-05-09T03:15:55.062014+0000","last_peered":"2024-05-09T03:15:55.062014+0000","last_clean":"2024-05-09T03:15:55.062014+0000","last_became_active":"2024-05-09T03:15:53.065104+0000","last_became_peered":"2024-05-09T03:15:53.065104+0000","last_unstale":"2024-05-09T03:15:55.062014+0000","last_undegraded":"2024-05-09T03:15:55.062014+0000","last_fullsized":"2024-05-09T03:15:55.062014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:06:07.829466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074923+0000","last_change":"2024-05-09T03:15:52.067126+0000","last_active":"2024-05-09T03:15:54.074923+0000","last_peered":"2024-05-09T03:15:54.074923+0000","last_clean":"2024-05-09T03:15:54.074923+0000","last_became_active":"2024-05-09T03:15:52.066221+0000","last_became_peered":"2024-05-09T03:15:52.066221+0000","last_unstale":"2024-05-09T03:15:54.074923+0000","last_undegraded":"2024-05-09T03:15:54.074923+0000","last_fullsized":"2024-05-09T03:15:54.074923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:30:25.658561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074482+0000","last_change":"2024-05-09T03:15:53.068542+0000","last_active":"2024-05-09T03:15:54.074482+0000","last_peered":"2024-05-09T03:15:54.074482+0000","last_clean":"2024-05-09T03:15:54.074482+0000","last_became_active":"2024-05-09T03:15:53.068102+0000","last_became_peered":"2024-05-09T03:15:53.068102+0000","last_unstale":"2024-05-09T03:15:54.074482+0000","last_undegraded":"2024-05-09T03:15:54.074482+0000","last_fullsized":"2024-05-09T03:15:54.074482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:52.357768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072141+0000","last_change":"2024-05-09T03:15:52.072827+0000","last_active":"2024-05-09T03:15:54.072141+0000","last_peered":"2024-05-09T03:15:54.072141+0000","last_clean":"2024-05-09T03:15:54.072141+0000","last_became_active":"2024-05-09T03:15:52.072097+0000","last_became_peered":"2024-05-09T03:15:52.072097+0000","last_unstale":"2024-05-09T03:15:54.072141+0000","last_undegraded":"2024-05-09T03:15:54.072141+0000","last_fullsized":"2024-05-09T03:15:54.072141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:46.656775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075113+0000","last_change":"2024-05-09T03:15:53.078967+0000","last_active":"2024-05-09T03:15:54.075113+0000","last_peered":"2024-05-09T03:15:54.075113+0000","last_clean":"2024-05-09T03:15:54.075113+0000","last_became_active":"2024-05-09T03:15:53.078607+0000","last_became_peered":"2024-05-09T03:15:53.078607+0000","last_unstale":"2024-05-09T03:15:54.075113+0000","last_undegraded":"2024-05-09T03:15:54.075113+0000","last_fullsized":"2024-05-09T03:15:54.075113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:58.790752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072455+0000","last_change":"2024-05-09T03:15:52.067341+0000","last_active":"2024-05-09T03:15:54.072455+0000","last_peered":"2024-05-09T03:15:54.072455+0000","last_clean":"2024-05-09T03:15:54.072455+0000","last_became_active":"2024-05-09T03:15:52.066938+0000","last_became_peered":"2024-05-09T03:15:52.066938+0000","last_unstale":"2024-05-09T03:15:54.072455+0000","last_undegraded":"2024-05-09T03:15:54.072455+0000","last_fullsized":"2024-05-09T03:15:54.072455+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:22:09.825607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062589+0000","last_change":"2024-05-09T03:15:53.084906+0000","last_active":"2024-05-09T03:15:55.062589+0000","last_peered":"2024-05-09T03:15:55.062589+0000","last_clean":"2024-05-09T03:15:55.062589+0000","last_became_active":"2024-05-09T03:15:53.084554+0000","last_became_peered":"2024-05-09T03:15:53.084554+0000","last_unstale":"2024-05-09T03:15:55.062589+0000","last_undegraded":"2024-05-09T03:15:55.062589+0000","last_fullsized":"2024-05-09T03:15:55.062589+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:15:28.432507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062031+0000","last_change":"2024-05-09T03:15:52.068600+0000","last_active":"2024-05-09T03:15:55.062031+0000","last_peered":"2024-05-09T03:15:55.062031+0000","last_clean":"2024-05-09T03:15:55.062031+0000","last_became_active":"2024-05-09T03:15:52.067579+0000","last_became_peered":"2024-05-09T03:15:52.067579+0000","last_unstale":"2024-05-09T03:15:55.062031+0000","last_undegraded":"2024-05-09T03:15:55.062031+0000","last_fullsized":"2024-05-09T03:15:55.062031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:20.722176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061534+0000","last_change":"2024-05-09T03:15:53.075148+0000","last_active":"2024-05-09T03:15:55.061534+0000","last_peered":"2024-05-09T03:15:55.061534+0000","last_clean":"2024-05-09T03:15:55.061534+0000","last_became_active":"2024-05-09T03:15:53.074808+0000","last_became_peered":"2024-05-09T03:15:53.074808+0000","last_unstale":"2024-05-09T03:15:55.061534+0000","last_undegraded":"2024-05-09T03:15:55.061534+0000","last_fullsized":"2024-05-09T03:15:55.061534+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:07.835507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062666+0000","last_change":"2024-05-09T03:15:52.075836+0000","last_active":"2024-05-09T03:15:55.062666+0000","last_peered":"2024-05-09T03:15:55.062666+0000","last_clean":"2024-05-09T03:15:55.062666+0000","last_became_active":"2024-05-09T03:15:52.075318+0000","last_became_peered":"2024-05-09T03:15:52.075318+0000","last_unstale":"2024-05-09T03:15:55.062666+0000","last_undegraded":"2024-05-09T03:15:55.062666+0000","last_fullsized":"2024-05-09T03:15:55.062666+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:45:44.568140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062135+0000","last_change":"2024-05-09T03:15:53.080707+0000","last_active":"2024-05-09T03:15:55.062135+0000","last_peered":"2024-05-09T03:15:55.062135+0000","last_clean":"2024-05-09T03:15:55.062135+0000","last_became_active":"2024-05-09T03:15:53.080346+0000","last_became_peered":"2024-05-09T03:15:53.080346+0000","last_unstale":"2024-05-09T03:15:55.062135+0000","last_undegraded":"2024-05-09T03:15:55.062135+0000","last_fullsized":"2024-05-09T03:15:55.062135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:42:11.979488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061884+0000","last_change":"2024-05-09T03:15:52.072938+0000","last_active":"2024-05-09T03:15:55.061884+0000","last_peered":"2024-05-09T03:15:55.061884+0000","last_clean":"2024-05-09T03:15:55.061884+0000","last_became_active":"2024-05-09T03:15:52.072599+0000","last_became_peered":"2024-05-09T03:15:52.072599+0000","last_unstale":"2024-05-09T03:15:55.061884+0000","last_undegraded":"2024-05-09T03:15:55.061884+0000","last_fullsized":"2024-05-09T03:15:55.061884+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:12:19.346768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061366+0000","last_change":"2024-05-09T03:15:53.079067+0000","last_active":"2024-05-09T03:15:55.061366+0000","last_peered":"2024-05-09T03:15:55.061366+0000","last_clean":"2024-05-09T03:15:55.061366+0000","last_became_active":"2024-05-09T03:15:53.078752+0000","last_became_peered":"2024-05-09T03:15:53.078752+0000","last_unstale":"2024-05-09T03:15:55.061366+0000","last_undegraded":"2024-05-09T03:15:55.061366+0000","last_fullsized":"2024-05-09T03:15:55.061366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:01:53.556837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061713+0000","last_change":"2024-05-09T03:15:52.068116+0000","last_active":"2024-05-09T03:15:55.061713+0000","last_peered":"2024-05-09T03:15:55.061713+0000","last_clean":"2024-05-09T03:15:55.061713+0000","last_became_active":"2024-05-09T03:15:52.067782+0000","last_became_peered":"2024-05-09T03:15:52.067782+0000","last_unstale":"2024-05-09T03:15:55.061713+0000","last_undegraded":"2024-05-09T03:15:55.061713+0000","last_fullsized":"2024-05-09T03:15:55.061713+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:51.349610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074631+0000","last_change":"2024-05-09T03:15:53.077480+0000","last_active":"2024-05-09T03:15:54.074631+0000","last_peered":"2024-05-09T03:15:54.074631+0000","last_clean":"2024-05-09T03:15:54.074631+0000","last_became_active":"2024-05-09T03:15:53.076807+0000","last_became_peered":"2024-05-09T03:15:53.076807+0000","last_unstale":"2024-05-09T03:15:54.074631+0000","last_undegraded":"2024-05-09T03:15:54.074631+0000","last_fullsized":"2024-05-09T03:15:54.074631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:16:06.703157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062668+0000","last_change":"2024-05-09T03:15:52.072537+0000","last_active":"2024-05-09T03:15:55.062668+0000","last_peered":"2024-05-09T03:15:55.062668+0000","last_clean":"2024-05-09T03:15:55.062668+0000","last_became_active":"2024-05-09T03:15:52.072211+0000","last_became_peered":"2024-05-09T03:15:52.072211+0000","last_unstale":"2024-05-09T03:15:55.062668+0000","last_undegraded":"2024-05-09T03:15:55.062668+0000","last_fullsized":"2024-05-09T03:15:55.062668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:25:22.736615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062048+0000","last_change":"2024-05-09T03:15:53.078490+0000","last_active":"2024-05-09T03:15:55.062048+0000","last_peered":"2024-05-09T03:15:55.062048+0000","last_clean":"2024-05-09T03:15:55.062048+0000","last_became_active":"2024-05-09T03:15:53.078151+0000","last_became_peered":"2024-05-09T03:15:53.078151+0000","last_unstale":"2024-05-09T03:15:55.062048+0000","last_undegraded":"2024-05-09T03:15:55.062048+0000","last_fullsized":"2024-05-09T03:15:55.062048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:08.551174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.063361+0000","last_change":"2024-05-09T03:15:52.070511+0000","last_active":"2024-05-09T03:15:55.063361+0000","last_peered":"2024-05-09T03:15:55.063361+0000","last_clean":"2024-05-09T03:15:55.063361+0000","last_became_active":"2024-05-09T03:15:52.070212+0000","last_became_peered":"2024-05-09T03:15:52.070212+0000","last_unstale":"2024-05-09T03:15:55.063361+0000","last_undegraded":"2024-05-09T03:15:55.063361+0000","last_fullsized":"2024-05-09T03:15:55.063361+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:38:07.450826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074062+0000","last_change":"2024-05-09T03:15:53.073566+0000","last_active":"2024-05-09T03:15:54.074062+0000","last_peered":"2024-05-09T03:15:54.074062+0000","last_clean":"2024-05-09T03:15:54.074062+0000","last_became_active":"2024-05-09T03:15:53.072995+0000","last_became_peered":"2024-05-09T03:15:53.072995+0000","last_unstale":"2024-05-09T03:15:54.074062+0000","last_undegraded":"2024-05-09T03:15:54.074062+0000","last_fullsized":"2024-05-09T03:15:54.074062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:46:23.557851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073707+0000","last_change":"2024-05-09T03:15:52.068346+0000","last_active":"2024-05-09T03:15:54.073707+0000","last_peered":"2024-05-09T03:15:54.073707+0000","last_clean":"2024-05-09T03:15:54.073707+0000","last_became_active":"2024-05-09T03:15:52.068035+0000","last_became_peered":"2024-05-09T03:15:52.068035+0000","last_unstale":"2024-05-09T03:15:54.073707+0000","last_undegraded":"2024-05-09T03:15:54.073707+0000","last_fullsized":"2024-05-09T03:15:54.073707+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:02.508479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.075057+0000","last_change":"2024-05-09T03:15:53.082192+0000","last_active":"2024-05-09T03:15:54.075057+0000","last_peered":"2024-05-09T03:15:54.075057+0000","last_clean":"2024-05-09T03:15:54.075057+0000","last_became_active":"2024-05-09T03:15:53.081850+0000","last_became_peered":"2024-05-09T03:15:53.081850+0000","last_unstale":"2024-05-09T03:15:54.075057+0000","last_undegraded":"2024-05-09T03:15:54.075057+0000","last_fullsized":"2024-05-09T03:15:54.075057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:40:40.889305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074174+0000","last_change":"2024-05-09T03:15:52.069955+0000","last_active":"2024-05-09T03:15:54.074174+0000","last_peered":"2024-05-09T03:15:54.074174+0000","last_clean":"2024-05-09T03:15:54.074174+0000","last_became_active":"2024-05-09T03:15:52.069619+0000","last_became_peered":"2024-05-09T03:15:52.069619+0000","last_unstale":"2024-05-09T03:15:54.074174+0000","last_undegraded":"2024-05-09T03:15:54.074174+0000","last_fullsized":"2024-05-09T03:15:54.074174+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:18:37.829088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061584+0000","last_change":"2024-05-09T03:15:53.079644+0000","last_active":"2024-05-09T03:15:55.061584+0000","last_peered":"2024-05-09T03:15:55.061584+0000","last_clean":"2024-05-09T03:15:55.061584+0000","last_became_active":"2024-05-09T03:15:53.079306+0000","last_became_peered":"2024-05-09T03:15:53.079306+0000","last_unstale":"2024-05-09T03:15:55.061584+0000","last_undegraded":"2024-05-09T03:15:55.061584+0000","last_fullsized":"2024-05-09T03:15:55.061584+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:23:15.962262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072368+0000","last_change":"2024-05-09T03:15:52.070867+0000","last_active":"2024-05-09T03:15:54.072368+0000","last_peered":"2024-05-09T03:15:54.072368+0000","last_clean":"2024-05-09T03:15:54.072368+0000","last_became_active":"2024-05-09T03:15:52.070163+0000","last_became_peered":"2024-05-09T03:15:52.070163+0000","last_unstale":"2024-05-09T03:15:54.072368+0000","last_undegraded":"2024-05-09T03:15:54.072368+0000","last_fullsized":"2024-05-09T03:15:54.072368+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:31:16.874387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.072212+0000","last_change":"2024-05-09T03:15:53.090489+0000","last_active":"2024-05-09T03:15:54.072212+0000","last_peered":"2024-05-09T03:15:54.072212+0000","last_clean":"2024-05-09T03:15:54.072212+0000","last_became_active":"2024-05-09T03:15:53.089158+0000","last_became_peered":"2024-05-09T03:15:53.089158+0000","last_unstale":"2024-05-09T03:15:54.072212+0000","last_undegraded":"2024-05-09T03:15:54.072212+0000","last_fullsized":"2024-05-09T03:15:54.072212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:46:04.173563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074079+0000","last_change":"2024-05-09T03:15:52.078197+0000","last_active":"2024-05-09T03:15:54.074079+0000","last_peered":"2024-05-09T03:15:54.074079+0000","last_clean":"2024-05-09T03:15:54.074079+0000","last_became_active":"2024-05-09T03:15:52.077834+0000","last_became_peered":"2024-05-09T03:15:52.077834+0000","last_unstale":"2024-05-09T03:15:54.074079+0000","last_undegraded":"2024-05-09T03:15:54.074079+0000","last_fullsized":"2024-05-09T03:15:54.074079+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:22.187664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061241+0000","last_change":"2024-05-09T03:15:53.073487+0000","last_active":"2024-05-09T03:15:55.061241+0000","last_peered":"2024-05-09T03:15:55.061241+0000","last_clean":"2024-05-09T03:15:55.061241+0000","last_became_active":"2024-05-09T03:15:53.073184+0000","last_became_peered":"2024-05-09T03:15:53.073184+0000","last_unstale":"2024-05-09T03:15:55.061241+0000","last_undegraded":"2024-05-09T03:15:55.061241+0000","last_fullsized":"2024-05-09T03:15:55.061241+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:14:21.730334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073564+0000","last_change":"2024-05-09T03:15:52.069881+0000","last_active":"2024-05-09T03:15:54.073564+0000","last_peered":"2024-05-09T03:15:54.073564+0000","last_clean":"2024-05-09T03:15:54.073564+0000","last_became_active":"2024-05-09T03:15:52.069547+0000","last_became_peered":"2024-05-09T03:15:52.069547+0000","last_unstale":"2024-05-09T03:15:54.073564+0000","last_undegraded":"2024-05-09T03:15:54.073564+0000","last_fullsized":"2024-05-09T03:15:54.073564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:25.705744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.073502+0000","last_change":"2024-05-09T03:15:53.077445+0000","last_active":"2024-05-09T03:15:54.073502+0000","last_peered":"2024-05-09T03:15:54.073502+0000","last_clean":"2024-05-09T03:15:54.073502+0000","last_became_active":"2024-05-09T03:15:53.077112+0000","last_became_peered":"2024-05-09T03:15:53.077112+0000","last_unstale":"2024-05-09T03:15:54.073502+0000","last_undegraded":"2024-05-09T03:15:54.073502+0000","last_fullsized":"2024-05-09T03:15:54.073502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:22:10.833765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:54.074151+0000","last_change":"2024-05-09T03:15:52.066053+0000","last_active":"2024-05-09T03:15:54.074151+0000","last_peered":"2024-05-09T03:15:54.074151+0000","last_clean":"2024-05-09T03:15:54.074151+0000","last_became_active":"2024-05-09T03:15:52.065728+0000","last_became_peered":"2024-05-09T03:15:52.065728+0000","last_unstale":"2024-05-09T03:15:54.074151+0000","last_undegraded":"2024-05-09T03:15:54.074151+0000","last_fullsized":"2024-05-09T03:15:54.074151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:11.847485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.062506+0000","last_change":"2024-05-09T03:15:53.069613+0000","last_active":"2024-05-09T03:15:55.062506+0000","last_peered":"2024-05-09T03:15:55.062506+0000","last_clean":"2024-05-09T03:15:55.062506+0000","last_became_active":"2024-05-09T03:15:53.068488+0000","last_became_peered":"2024-05-09T03:15:53.068488+0000","last_unstale":"2024-05-09T03:15:55.062506+0000","last_undegraded":"2024-05-09T03:15:55.062506+0000","last_fullsized":"2024-05-09T03:15:55.062506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:52.042102+0000","last_clean_scrub_stamp":"2024-05-09T03:15:52.042102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:19:12.855643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061832+0000","last_change":"2024-05-09T03:15:52.072040+0000","last_active":"2024-05-09T03:15:55.061832+0000","last_peered":"2024-05-09T03:15:55.061832+0000","last_clean":"2024-05-09T03:15:55.061832+0000","last_became_active":"2024-05-09T03:15:52.071729+0000","last_became_peered":"2024-05-09T03:15:52.071729+0000","last_unstale":"2024-05-09T03:15:55.061832+0000","last_undegraded":"2024-05-09T03:15:55.061832+0000","last_fullsized":"2024-05-09T03:15:55.061832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:15:51.033944+0000","last_clean_scrub_stamp":"2024-05-09T03:15:51.033944+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:32:46.656775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":227,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-05-09T03:15:55.061243+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:15:55.061243+0000","last_peered":"2024-05-09T03:15:55.061243+0000","last_clean":"2024-05-09T03:15:55.061243+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:15:55.061243+0000","last_undegraded":"2024-05-09T03:15:55.061243+0000","last_fullsized":"2024-05-09T03:15:55.061243+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574930,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1022117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574930,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":432373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1024000,"data_stored":433313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":1007616,"data_stored":431679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":433291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574930,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":432261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":431873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574930,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":1605632,"data_stored":1022081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:15:57.533 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:15:57.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T03:15:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.692+0000 7fb8e6cd1640 1 Processor -- start 2024-05-09T03:15:57.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.692+0000 7fb8e6cd1640 1 -- start start 2024-05-09T03:15:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e6cd1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8e01577b0 0x7fb8e00b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e6cd1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 0x7fb8e00b0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e6cd1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e00b10f0 con 0x7fb8e00aeb10 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e6cd1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e00b6d30 con 0x7fb8e01577b0 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e6cd1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e00b0ea0 con 0x7fb8e00b5b50 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e526e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb8e00aeb10 0x7fb8e00aef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40034/0 (socket says 172.21.15.46:40034) 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e526e640 1 -- 172.21.15.46:0/2784270664 learned_addr learned my addr 172.21.15.46:0/2784270664 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8e4a6d640 1 --2- 172.21.15.46:0/2784270664 >> v2:172.21.15.88:3300/0 conn(0x7fb8e01577b0 0x7fb8e00b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8dffff640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 0x7fb8e00b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.693+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286547035 0 0) 0x7fb8e00b10f0 con 0x7fb8e00aeb10 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c8003520 con 0x7fb8e00aeb10 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2706942227 0 0) 0x7fb8c8003520 con 0x7fb8e00aeb10 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 msgr2=0x7fb8e00b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 0x7fb8e00b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 >> v2:172.21.15.88:3300/0 conn(0x7fb8e01577b0 msgr2=0x7fb8e00b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 --2- 172.21.15.46:0/2784270664 >> v2:172.21.15.88:3300/0 conn(0x7fb8e01577b0 0x7fb8e00b67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e00b5070 con 0x7fb8e00aeb10 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fb8d8001a70 con 0x7fb8e00aeb10 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 >> v1:172.21.15.46:6789/0 conn(0x7fb8e00aeb10 legacy=0x7fb8e00aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 0x7fb8c8003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8c8004d40 0x7fb8c8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00b6d30 con 0x7fb8c8004020 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00b0ea0 con 0x7fb8c8004d40 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c800ffb0 con 0x7fb8e00b5b50 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8dffff640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8e4a6d640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 0x7fb8c8003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8e526e640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8c8004d40 0x7fb8c8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8dffff640 1 -- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 msgr2=0x7fb8c8003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8dffff640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e00b5b50 0x7fb8c8003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8dffff640 1 -- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8c8004d40 msgr2=0x7fb8c8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8dffff640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8c8004d40 0x7fb8c8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.694+0000 7fb8dffff640 1 -- 172.21.15.46:0/2784270664 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c8003520 con 0x7fb8c8004020 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.695+0000 7fb8dffff640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8c8004800 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009ac0 tx=0x7fb8c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7681bd68a700a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.695+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4036070 con 0x7fb8c8004020 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.695+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d4031400 con 0x7fb8c8004020 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.695+0000 7fb8df7fe640 1 -- 172.21.15.46:0/2784270664 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4009690 con 0x7fb8c8004020 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/2784270664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 msgr2=0x7fb8c8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/2784270664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8c8004800 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009ac0 tx=0x7fb8c8005830 comp rx=0 tx=0).stop 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e526e640 1 -- 172.21.15.46:0/2784270664 reap_dead start 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/2784270664 shutdown_connections 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/2784270664 >> 172.21.15.46:0/2784270664 conn(0x7fb8e00d1980 msgr2=0x7fb8e00d1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/2784270664 shutdown_connections 2024-05-09T03:15:57.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.696+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/2784270664 wait complete. 2024-05-09T03:15:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.698+0000 7fb8e6cd1640 1 Processor -- start 2024-05-09T03:15:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.698+0000 7fb8e6cd1640 1 -- start start 2024-05-09T03:15:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e6cd1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8e01f5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e6cd1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e009e720 0x7fb8e0156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e6cd1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e015c0d0 0x7fb8e015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e6cd1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00b13d0 con 0x7fb8c8004020 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e6cd1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00b5630 con 0x7fb8e009e720 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e6cd1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e00b0960 con 0x7fb8e015c0d0 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e4a6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8e01f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e4a6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8e01f5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55348/0 (socket says 172.21.15.46:55348) 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.699+0000 7fb8e4a6d640 1 -- 172.21.15.46:0/1371573031 learned_addr learned my addr 172.21.15.46:0/1371573031 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8dffff640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e009e720 0x7fb8e0156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e526e640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e015c0d0 0x7fb8e015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e4a6d640 1 -- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e015c0d0 msgr2=0x7fb8e015c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e4a6d640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e015c0d0 0x7fb8e015c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e4a6d640 1 -- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e009e720 msgr2=0x7fb8e0156cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e4a6d640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e009e720 0x7fb8e0156cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e4a6d640 1 -- 172.21.15.46:0/1371573031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e015ca20 con 0x7fb8c8004020 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e4a6d640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8e01f5940 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0011980 tx=0x7fb8d0011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:57.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000faf0 con 0x7fb8c8004020 2024-05-09T03:15:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d000fc90 con 0x7fb8c8004020 2024-05-09T03:15:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e015aae0 con 0x7fb8c8004020 2024-05-09T03:15:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0014440 con 0x7fb8c8004020 2024-05-09T03:15:57.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.700+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e01f6470 con 0x7fb8c8004020 2024-05-09T03:15:57.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.701+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e00b39e0 con 0x7fb8c8004020 2024-05-09T03:15:57.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.702+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb8d0012040 con 0x7fb8c8004020 2024-05-09T03:15:57.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.702+0000 7fb8dd7fa640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8cc076300 0x7fb8cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:57.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.702+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fb8d0097c60 con 0x7fb8c8004020 2024-05-09T03:15:57.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.702+0000 7fb8dffff640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8cc076300 0x7fb8cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:57.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.703+0000 7fb8dffff640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8cc076300 0x7fb8cc078810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4033000 tx=0x7fb8d4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:57.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.707+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb8e00b39e0 con 0x7fb8c8004020 2024-05-09T03:15:57.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.971+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb8e006a270 con 0x7fb8c8004020 2024-05-09T03:15:57.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.974+0000 7fb8dd7fa640 1 -- 172.21.15.46:0/1371573031 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7fb8d0019030 con 0x7fb8c8004020 2024-05-09T03:15:57.974 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:15:57.975 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:15:57.975 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-05-09T03:15:57.975 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-05-09T03:15:57.975 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T03:15:57.975 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8cc076300 msgr2=0x7fb8cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8cc076300 0x7fb8cc078810 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4033000 tx=0x7fb8d4008030 comp rx=0 tx=0).stop 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 msgr2=0x7fb8e01f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8e01f5940 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0011980 tx=0x7fb8d0011e40 comp rx=0 tx=0).stop 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 shutdown_connections 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8e015c0d0 0x7fb8e015c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e009e720 0x7fb8e0156cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8cc076300 0x7fb8cc078810 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 --2- 172.21.15.46:0/1371573031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8c8004020 0x7fb8e01f5940 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:57.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 >> 172.21.15.46:0/1371573031 conn(0x7fb8e00d1980 msgr2=0x7fb8e015ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:57.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 shutdown_connections 2024-05-09T03:15:57.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:57.976+0000 7fb8e6cd1640 1 -- 172.21.15.46:0/1371573031 wait complete. 2024-05-09T03:15:57.988 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:15:58.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.103+0000 7f1c4402b640 1 Processor -- start 2024-05-09T03:15:58.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.103+0000 7f1c4402b640 1 -- start start 2024-05-09T03:15:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c4402b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c3c0d2cf0 0x7f1c3c15f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c4402b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c3c0d15b0 con 0x7f1c3c1615f0 2024-05-09T03:15:58.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c4402b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c3c160b10 con 0x7f1c3c0b81b0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c4402b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3c160df0 con 0x7f1c3c0d2cf0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c41dc7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c3c1615f0 0x7f1c3c0d0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40048/0 (socket says 172.21.15.46:40048) 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c41dc7640 1 -- 172.21.15.46:0/1324647352 learned_addr learned my addr 172.21.15.46:0/1324647352 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1324647352 >> v2:172.21.15.88:3300/0 conn(0x7f1c3c0d2cf0 0x7f1c3c15f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133335468 0 0) 0x7f1c3c0d15b0 con 0x7f1c3c1615f0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c24003520 con 0x7f1c3c1615f0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3664123393 0 0) 0x7f1c24003520 con 0x7f1c3c1615f0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.105+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 >> v2:172.21.15.88:3300/0 conn(0x7f1c3c0d2cf0 msgr2=0x7f1c3c15f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 --2- 172.21.15.46:0/1324647352 >> v2:172.21.15.88:3300/0 conn(0x7f1c3c0d2cf0 0x7f1c3c15f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 >> v1:172.21.15.88:6789/0 conn(0x7f1c3c0b81b0 legacy=0x7f1c3c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c1610d0 con 0x7f1c3c1615f0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f1c2c001a90 con 0x7f1c3c1615f0 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 >> v1:172.21.15.46:6789/0 conn(0x7f1c3c1615f0 legacy=0x7f1c3c0d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c24003a00 0x7f1c24003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c24004800 0x7f1c24004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240051c0 0x7f1c2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c160df0 con 0x7f1c240051c0 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2400c160 con 0x7f1c24003a00 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2400c360 con 0x7f1c24004800 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240051c0 0x7f1c2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c41dc7640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c24003a00 0x7f1c24003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c415c6640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c24004800 0x7f1c24004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c425c8640 1 -- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c24004800 msgr2=0x7f1c24004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c24004800 0x7f1c24004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c425c8640 1 -- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c24003a00 msgr2=0x7f1c24003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c24003a00 0x7f1c24003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.106+0000 7f1c425c8640 1 -- 172.21.15.46:0/1324647352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c24003520 con 0x7f1c240051c0 2024-05-09T03:15:58.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.107+0000 7f1c425c8640 1 -- 172.21.15.46:0/1324647352 reap_dead start 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.107+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240051c0 0x7f1c2400b9c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009a90 tx=0x7f1c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce26e3cb7de7a82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.107+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c34036070 con 0x7f1c240051c0 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.107+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c34031400 con 0x7f1c240051c0 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.107+0000 7f1c40dc5640 1 -- 172.21.15.46:0/1324647352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c34009690 con 0x7f1c240051c0 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 -- 172.21.15.46:0/1324647352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240051c0 msgr2=0x7f1c2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240051c0 0x7f1c2400b9c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009a90 tx=0x7f1c24005830 comp rx=0 tx=0).stop 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 -- 172.21.15.46:0/1324647352 shutdown_connections 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1324647352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c240051c0 0x7f1c2400b9c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 -- 172.21.15.46:0/1324647352 >> 172.21.15.46:0/1324647352 conn(0x7f1c3c06e190 msgr2=0x7f1c3c06e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 -- 172.21.15.46:0/1324647352 shutdown_connections 2024-05-09T03:15:58.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.108+0000 7f1c4402b640 1 -- 172.21.15.46:0/1324647352 wait complete. 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.110+0000 7f1c4402b640 1 Processor -- start 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.110+0000 7f1c4402b640 1 -- start start 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c4402b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c240051c0 0x7f1c3c15e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c4402b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 0x7f1c3c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c4402b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c3c0d08b0 0x7f1c3c1eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c4402b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c0d25e0 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c4402b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c0d1890 con 0x7f1c240051c0 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c4402b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3c0d2760 con 0x7f1c3c0d08b0 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 0x7f1c3c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 0x7f1c3c0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55364/0 (socket says 172.21.15.46:55364) 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 -- 172.21.15.46:0/1436936416 learned_addr learned my addr 172.21.15.46:0/1436936416 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c3c0d08b0 0x7f1c3c1eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c41dc7640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c240051c0 0x7f1c3c15e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 -- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c3c0d08b0 msgr2=0x7f1c3c1eadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c3c0d08b0 0x7f1c3c1eadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 -- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c240051c0 msgr2=0x7f1c3c15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.111+0000 7f1c415c6640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c240051c0 0x7f1c3c15e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c415c6640 1 -- 172.21.15.46:0/1436936416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c0d0cc0 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c425c8640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c3c0d08b0 0x7f1c3c1eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c41dc7640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c240051c0 0x7f1c3c15e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c415c6640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 0x7f1c3c0d0370 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000f850 tx=0x7f1c3000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300052e0 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c30018070 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3c15e980 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c3000d970 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.112+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3c0ca250 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.113+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3c0c9ce0 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.114+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1c3000db10 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.114+0000 7f1c3a7fc640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c28076300 0x7f1c28078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.114+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f1c30097510 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.115+0000 7f1c41dc7640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c28076300 0x7f1c28078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.115+0000 7f1c41dc7640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c28076300 0x7f1c28078810 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c002230 tx=0x7f1c2c006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.119+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1c3c0d0cc0 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.354+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c3c06a270 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.355+0000 7f1c3a7fc640 1 -- 172.21.15.46:0/1436936416 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5180 (secure 0 0 0) 0x7f1c30060dd0 con 0x7f1c3c0cb7f0 2024-05-09T03:15:58.355 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:15:58.356 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:15:55.468035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6888":{"gid":6888,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/2347304023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2347304023},{"type":"v1","addr":"172.21.15.46:6834","nonce":2347304023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6888,"qdb_cluster":[6888]},"id":6}]} 2024-05-09T03:15:58.356 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 85 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c28076300 msgr2=0x7f1c28078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c28076300 0x7f1c28078810 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c002230 tx=0x7f1c2c006040 comp rx=0 tx=0).stop 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 msgr2=0x7f1c3c0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 0x7f1c3c0d0370 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f1c3000f850 tx=0x7f1c3000fd10 comp rx=0 tx=0).stop 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 shutdown_connections 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c3c0d08b0 0x7f1c3c1eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c28076300 0x7f1c28078810 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c3c0cb7f0 0x7f1c3c0d0370 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 --2- 172.21.15.46:0/1436936416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c240051c0 0x7f1c3c15e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 >> 172.21.15.46:0/1436936416 conn(0x7f1c3c06e190 msgr2=0x7f1c3c160330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 shutdown_connections 2024-05-09T03:15:58.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.357+0000 7f1c4402b640 1 -- 172.21.15.46:0/1436936416 wait complete. 2024-05-09T03:15:58.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:15:55.468035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6888}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6888': {'gid': 6888, 'name': 'c', 'rank': 0, 'incarnation': 81, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6834/2347304023', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2347304023}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2347304023}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6888, 'qdb_cluster': [6888]} 2024-05-09T03:15:58.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:15:58.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:15:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.894+0000 7f1c23ffa640 1 Processor -- start 2024-05-09T03:15:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.894+0000 7f1c23ffa640 1 -- start start 2024-05-09T03:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.895+0000 7f1c23ffa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1c1c163eb0 0x7f1c1c1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.895+0000 7f1c23ffa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1c164900 con 0x7f1c1c0d2a70 2024-05-09T03:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.895+0000 7f1c23ffa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1c164be0 con 0x7f1c1c0bb770 2024-05-09T03:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.895+0000 7f1c23ffa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1c1c0d4bb0 con 0x7f1c1c163eb0 2024-05-09T03:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.895+0000 7f1c22597640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1c1c0d2a70 0x7f1c1c0d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40056/0 (socket says 172.21.15.46:40056) 2024-05-09T03:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.895+0000 7f1c22597640 1 -- 172.21.15.46:0/3715700634 learned_addr learned my addr 172.21.15.46:0/3715700634 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c21595640 1 --2- 172.21.15.46:0/3715700634 >> v2:172.21.15.88:3300/0 conn(0x7f1c1c163eb0 0x7f1c1c1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953077655 0 0) 0x7f1c1c164900 con 0x7f1c1c0d2a70 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c04003520 con 0x7f1c1c0d2a70 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4010223037 0 0) 0x7f1c1c164be0 con 0x7f1c1c0bb770 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c1c164900 con 0x7f1c1c0bb770 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 482299591 0 0) 0x7f1c04003520 con 0x7f1c1c0d2a70 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 >> v2:172.21.15.88:3300/0 conn(0x7f1c1c163eb0 msgr2=0x7f1c1c1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 --2- 172.21.15.46:0/3715700634 >> v2:172.21.15.88:3300/0 conn(0x7f1c1c163eb0 0x7f1c1c1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 >> v1:172.21.15.88:6789/0 conn(0x7f1c1c0bb770 legacy=0x7f1c1c0d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c0d4840 con 0x7f1c1c0d2a70 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f1c0c001a70 con 0x7f1c1c0d2a70 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 >> v1:172.21.15.46:6789/0 conn(0x7f1c1c0d2a70 legacy=0x7f1c1c0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c04004080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c04004800 0x7f1c04004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c04005280 0x7f1c0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c0d4bb0 con 0x7f1c04003c30 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0400c230 con 0x7f1c04004800 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0400c430 con 0x7f1c04005280 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c04004080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c21595640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c04004800 0x7f1c04004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.896+0000 7f1c22597640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c04005280 0x7f1c0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c21d96640 1 -- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c04005280 msgr2=0x7f1c0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c04005280 0x7f1c0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c21d96640 1 -- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c04004800 msgr2=0x7f1c04004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c04004800 0x7f1c04004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c22597640 1 -- 172.21.15.46:0/3715700634 reap_dead start 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c21d96640 1 -- 172.21.15.46:0/3715700634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c04003520 con 0x7f1c04003c30 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.897+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c04004080 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1c140018d0 tx=0x7f1c04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaefcd289f4fc3bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14001520 con 0x7f1c04003c30 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c140052f0 con 0x7f1c04003c30 2024-05-09T03:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c20d94640 1 -- 172.21.15.46:0/3715700634 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1402ac80 con 0x7f1c04003c30 2024-05-09T03:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 -- 172.21.15.46:0/3715700634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 msgr2=0x7f1c04004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c04004080 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f1c140018d0 tx=0x7f1c04005830 comp rx=0 tx=0).stop 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 -- 172.21.15.46:0/3715700634 shutdown_connections 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/3715700634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c04004080 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 -- 172.21.15.46:0/3715700634 >> 172.21.15.46:0/3715700634 conn(0x7f1c1c06e370 msgr2=0x7f1c1c06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 -- 172.21.15.46:0/3715700634 shutdown_connections 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.898+0000 7f1c23ffa640 1 -- 172.21.15.46:0/3715700634 wait complete. 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.900+0000 7f1c23ffa640 1 Processor -- start 2024-05-09T03:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.900+0000 7f1c23ffa640 1 -- start start 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c23ffa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c1c14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c23ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c1c151c40 0x7f1c1c14e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c23ffa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c1c150870 0x7f1c1c1510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c23ffa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c0d0d40 con 0x7f1c04003c30 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c23ffa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c0d0bc0 con 0x7f1c1c151c40 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c23ffa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c1642c0 con 0x7f1c1c150870 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c1c14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c1c14f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55376/0 (socket says 172.21.15.46:55376) 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 -- 172.21.15.46:0/858519732 learned_addr learned my addr 172.21.15.46:0/858519732 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21595640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c1c151c40 0x7f1c1c14e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c22597640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c1c150870 0x7f1c1c1510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 -- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c1c150870 msgr2=0x7f1c1c1510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c1c150870 0x7f1c1c1510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 -- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c1c151c40 msgr2=0x7f1c1c14e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c1c151c40 0x7f1c1c14e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 -- 172.21.15.46:0/858519732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1c14ed40 con 0x7f1c04003c30 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c21d96640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c1c14f010 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1c14005040 tx=0x7f1c140012f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.902+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/858519732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14042070 con 0x7f1c04003c30 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.903+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/858519732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c140054a0 con 0x7f1c04003c30 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.903+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1c14f7d0 con 0x7f1c04003c30 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.903+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/858519732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14007a10 con 0x7f1c04003c30 2024-05-09T03:15:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.903+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1c14fd90 con 0x7f1c04003c30 2024-05-09T03:15:58.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.904+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/858519732 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1c14045070 con 0x7f1c04003c30 2024-05-09T03:15:58.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.905+0000 7f1c1a7fc640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c080762d0 0x7f1c080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.906+0000 7f1c1a7fc640 1 -- 172.21.15.46:0/858519732 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f1c140b6940 con 0x7f1c04003c30 2024-05-09T03:15:58.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.906+0000 7f1c21595640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c080762d0 0x7f1c080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.906+0000 7f1c21595640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c080762d0 0x7f1c080787e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c161f10 tx=0x7f1c10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.906+0000 7f1c2af88740 1 Processor -- start 2024-05-09T03:15:58.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.906+0000 7f1c2af88740 1 -- start start 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 0x55f54c21beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c236be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be2b590 0x55f54c237120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c2af88740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f54c1544e0 con 0x55f54be4c270 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f54c0efff0 con 0x55f54be2b590 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f54c1504a0 con 0x55f54be233b0 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 0x55f54c21beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 0x55f54c21beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55392/0 (socket says 172.21.15.46:55392) 2024-05-09T03:15:58.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 -- 172.21.15.46:0/3510843169 learned_addr learned my addr 172.21.15.46:0/3510843169 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c22597640 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be2b590 0x55f54c237120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21595640 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c236be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 -- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 msgr2=0x55f54c236be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c236be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 -- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be2b590 msgr2=0x55f54c237120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be2b590 0x55f54c237120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 -- 172.21.15.46:0/3510843169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54c0d6720 con 0x55f54be4c270 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 0x55f54c21beb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1c140b4e10 tx=0x7f1c1407f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81537f031c55919 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c037fe640 1 -- 172.21.15.46:0/3510843169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1407f620 con 0x55f54be4c270 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c037fe640 1 -- 172.21.15.46:0/3510843169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c14080210 con 0x55f54be4c270 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.908+0000 7f1c037fe640 1 -- 172.21.15.46:0/3510843169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1407fb10 con 0x55f54be4c270 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 -- 172.21.15.46:0/3510843169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 msgr2=0x55f54c21beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 0x55f54c21beb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1c140b4e10 tx=0x7f1c1407f9e0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 -- 172.21.15.46:0/3510843169 shutdown_connections 2024-05-09T03:15:58.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be2b590 0x55f54c237120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c236be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3510843169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be4c270 0x55f54c21beb0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 -- 172.21.15.46:0/3510843169 >> 172.21.15.46:0/3510843169 conn(0x55f54be403c0 msgr2=0x55f54c1f1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 -- 172.21.15.46:0/3510843169 shutdown_connections 2024-05-09T03:15:58.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.909+0000 7f1c2af88740 1 -- 172.21.15.46:0/3510843169 wait complete. 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.910+0000 7f1c2af88740 1 -- start start 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.910+0000 7f1c2af88740 1 Processor -- start 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c2327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 0x55f54c22f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f54c0efff0 con 0x55f54be2b590 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f54c0db760 con 0x55f54be4c270 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f54c1544e0 con 0x55f54be233b0 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c2af88740 10 client.0 _ll_register_callbacks cb 0x55f54c0fe4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 0x55f54c22f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 0x55f54c22f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55402/0 (socket says 172.21.15.46:55402) 2024-05-09T03:15:58.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 -- 172.21.15.46:0/3482489472 learned_addr learned my addr 172.21.15.46:0/3482489472 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c22597640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c2327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 -- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 msgr2=0x55f54c2327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c2327e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 -- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 msgr2=0x55f54c22fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 -- 172.21.15.46:0/3482489472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54c11cbd0 con 0x55f54be2b590 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c21595640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 0x55f54c22f650 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1c10004610 tx=0x7f1c10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aca8607812143dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.912+0000 7f1c00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c00ff9640 1 -- 172.21.15.46:0/3482489472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10029070 con 0x55f54be2b590 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c00ff9640 1 -- 172.21.15.46:0/3482489472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f1c10016780 con 0x55f54be2b590 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c14080210 con 0x55f54be2b590 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c00ff9640 1 -- 172.21.15.46:0/3482489472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f1c10034050 con 0x55f54be2b590 2024-05-09T03:15:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 10 client.7008 fetch_fsmap learned FSMap version 85 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) -- 0x55f54c102280 con 0x55f54be2b590 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c00ff9640 1 -- 172.21.15.46:0/3482489472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c14080210 con 0x55f54be2b590 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c00ff9640 1 -- 172.21.15.46:0/3482489472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 85) ==== 3494+0+0 (secure 0 0 0) 0x7f1c10037020 con 0x55f54be2b590 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 10 client.7008 fetch_fsmap finished waiting for FSMap version 85 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 10 client.7008 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 20 client.7008 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54be0ae40 0x55f54c246080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 4 client.7008 mds_command: new command op to 6888 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 --> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f54c1620f0 con 0x55f54be0ae40 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.913+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54be0ae40 0x55f54c246080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.914+0000 7f1c00ff9640 10 client.7008 ms_handle_connect on v2:172.21.15.46:6832/2347304023 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.914+0000 7f1c21d96640 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54be0ae40 0x55f54c246080 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4c2240a58933543 server_cookie=ba91811a6322adb9 in_seq=0 out_seq=0 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c00ff9640 1 -- 172.21.15.46:0/3482489472 <== mds.0 v2:172.21.15.46:6832/2347304023 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f54c1620f0 con 0x55f54be0ae40 2024-05-09T03:15:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c00ff9640 10 client.7008 handle_command_reply: tid=0 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 client.7008 shutdown 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 msgr2=0x55f54c22f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 0x55f54c22f650 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1c10004610 tx=0x7f1c10004730 comp rx=0 tx=0).stop 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 shutdown_connections 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54be0ae40 0x55f54c246080 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be2b590 0x55f54c22f650 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c00ff9640 0 client.7008 ms_handle_reset on v2:172.21.15.46:6832/2347304023 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 --2- 172.21.15.46:0/3482489472 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be233b0 0x55f54c2327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 >> 172.21.15.46:0/3482489472 conn(0x55f54be403c0 msgr2=0x55f54c1f1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 shutdown_connections 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 1 -- 172.21.15.46:0/3482489472 wait complete. 2024-05-09T03:15:58.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.915+0000 7f1c2af88740 20 client.7008 trim_cache size 0 max 16384 2024-05-09T03:15:58.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.934+0000 7f1c2af88740 1 Processor -- start 2024-05-09T03:15:58.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.934+0000 7f1c2af88740 1 -- start start 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c218d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 0x55f54c22f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c2af88740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c202580 con 0x55f54be233b0 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1c1fa590 con 0x55f54be4c270 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1c201f60 con 0x55f54be2b590 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.935+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c218d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c218d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55404/0 (socket says 172.21.15.46:55404) 2024-05-09T03:15:58.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 -- 172.21.15.46:0/4193813589 learned_addr learned my addr 172.21.15.46:0/4193813589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 -- 172.21.15.46:0/4193813589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 msgr2=0x55f54c22f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 0x55f54c22f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 -- 172.21.15.46:0/4193813589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 msgr2=0x55f54c22fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 -- 172.21.15.46:0/4193813589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f54beb7850 con 0x55f54be233b0 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c21d96640 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c218d50 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f1c140014b0 tx=0x7f1c14048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7330f7095a9303a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c02ffd640 1 -- 172.21.15.46:0/4193813589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c140b3040 con 0x55f54be233b0 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c02ffd640 1 -- 172.21.15.46:0/4193813589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1407f620 con 0x55f54be233b0 2024-05-09T03:15:58.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.936+0000 7f1c02ffd640 1 -- 172.21.15.46:0/4193813589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14080210 con 0x55f54be233b0 2024-05-09T03:15:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 -- 172.21.15.46:0/4193813589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 msgr2=0x55f54c218d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c218d50 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f1c140014b0 tx=0x7f1c14048000 comp rx=0 tx=0).stop 2024-05-09T03:15:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 -- 172.21.15.46:0/4193813589 shutdown_connections 2024-05-09T03:15:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c22fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 0x55f54c22f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4193813589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c218d50 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 -- 172.21.15.46:0/4193813589 >> 172.21.15.46:0/4193813589 conn(0x55f54be403c0 msgr2=0x55f54c21beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 -- 172.21.15.46:0/4193813589 shutdown_connections 2024-05-09T03:15:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.937+0000 7f1c2af88740 1 -- 172.21.15.46:0/4193813589 wait complete. 2024-05-09T03:15:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.938+0000 7f1c2af88740 1 -- start start 2024-05-09T03:15:58.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.938+0000 7f1c2af88740 1 Processor -- start 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c22e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 0x55f54c22eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c2445d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f54c2183e0 con 0x55f54be233b0 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f54c247020 con 0x55f54be4c270 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f54c2315a0 con 0x55f54be2b590 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c2af88740 10 client.0 _ll_register_callbacks cb 0x7f1c10004260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c22e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c21d96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c22e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55412/0 (socket says 172.21.15.46:55412) 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.939+0000 7f1c21d96640 1 -- 172.21.15.46:0/4084676086 learned_addr learned my addr 172.21.15.46:0/4084676086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c21d96640 1 -- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 msgr2=0x55f54c22eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c21d96640 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 0x55f54c22eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c22597640 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c2445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c21d96640 1 -- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 msgr2=0x55f54c2445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c21d96640 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c2445d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c21d96640 1 -- 172.21.15.46:0/4084676086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f54bf2e6b0 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c21d96640 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c22e660 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1c140018d0 tx=0x7f1c1403f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cde411b33cf280 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c00ff9640 1 -- 172.21.15.46:0/4084676086 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c14090060 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c00ff9640 1 -- 172.21.15.46:0/4084676086 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f1c1407bda0 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c1407f620 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c00ff9640 1 -- 172.21.15.46:0/4084676086 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f1c140b3040 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c2af88740 10 client.7014 fetch_fsmap learned FSMap version 85 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=85,monmap=2+}) -- 0x55f54c110630 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c00ff9640 1 -- 172.21.15.46:0/4084676086 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c1407f620 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c00ff9640 1 -- 172.21.15.46:0/4084676086 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 85) ==== 3494+0+0 (secure 0 0 0) 0x7f1c140884e0 con 0x55f54be233b0 2024-05-09T03:15:58.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c2af88740 10 client.7014 fetch_fsmap finished waiting for FSMap version 85 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c2af88740 10 client.7014 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.940+0000 7f1c2af88740 20 client.7014 populate_metadata read hostname 'smithi046' 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54c1eac80 0x55f54c244b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c2af88740 4 client.7014 mds_command: new command op to 6888 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 --> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] -- command(tid 0: {"prefix": "status"}) -- 0x55f54c15bf60 con 0x55f54c1eac80 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c21595640 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54c1eac80 0x55f54c244b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c00ff9640 10 client.7014 ms_handle_connect on v2:172.21.15.46:6832/2347304023 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c21595640 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54c1eac80 0x55f54c244b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38221f0dafc1e0ed server_cookie=31dd49a7a6a8b75d in_seq=0 out_seq=0 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c00ff9640 1 -- 172.21.15.46:0/4084676086 <== mds.0 v2:172.21.15.46:6832/2347304023 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55f54c15bf60 con 0x55f54c1eac80 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c00ff9640 10 client.7014 handle_command_reply: tid=0 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c2af88740 1 client.7014 shutdown 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 msgr2=0x55f54c22e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.941+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c22e660 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1c140018d0 tx=0x7f1c1403f070 comp rx=0 tx=0).stop 2024-05-09T03:15:58.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 shutdown_connections 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:6832/2347304023,v1:172.21.15.46:6834/2347304023] conn(0x55f54c1eac80 0x55f54c244b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f54be4c270 0x55f54c2445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f54be2b590 0x55f54c22eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 --2- 172.21.15.46:0/4084676086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f54be233b0 0x55f54c22e660 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c00ff9640 0 client.7014 ms_handle_reset on v2:172.21.15.46:6832/2347304023 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 >> 172.21.15.46:0/4084676086 conn(0x55f54be403c0 msgr2=0x55f54c22f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 shutdown_connections 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 1 -- 172.21.15.46:0/4084676086 wait complete. 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c2af88740 20 client.7014 trim_cache size 0 max 16384 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:15:58.947 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6888, 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 5.8871528550000001, 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 85, 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 82, 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 82, 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.830332221000001 2024-05-09T03:15:58.948 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c080762d0 msgr2=0x7f1c080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c080762d0 0x7f1c080787e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c161f10 tx=0x7f1c10008040 comp rx=0 tx=0).stop 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 msgr2=0x7f1c1c14f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c1c14f010 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1c14005040 tx=0x7f1c140012f0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 shutdown_connections 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1c1c150870 0x7f1c1c1510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c1c151c40 0x7f1c1c14e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1c080762d0 0x7f1c080787e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 --2- 172.21.15.46:0/858519732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c04003c30 0x7f1c1c14f010 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 >> 172.21.15.46:0/858519732 conn(0x7f1c1c06e370 msgr2=0x7f1c1c0d4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 shutdown_connections 2024-05-09T03:15:58.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:58.942+0000 7f1c23ffa640 1 -- 172.21.15.46:0/858519732 wait complete. 2024-05-09T03:15:58.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:15:59.040 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:15:59.039+0000 7fda98af8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:15:59.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.068+0000 7f4c16a27640 1 Processor -- start 2024-05-09T03:15:59.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.068+0000 7f4c16a27640 1 -- start start 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c16a27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c100b81b0 0x7f4c100d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c16a27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c100d2b20 0x7f4c10163f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c16a27640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c100d15b0 con 0x7f4c101615f0 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c16a27640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4c10160b10 con 0x7f4c100d2b20 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c16a27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4c10160df0 con 0x7f4c100b81b0 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c14fc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c100d2b20 0x7f4c10163f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c0ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4c101615f0 0x7f4c100d0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40064/0 (socket says 172.21.15.46:40064) 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c0ffff640 1 -- 172.21.15.46:0/4277924470 learned_addr learned my addr 172.21.15.46:0/4277924470 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c0f7fe640 1 --2- 172.21.15.46:0/4277924470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c100b81b0 0x7f4c100d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:59.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c14fc4640 1 -- 172.21.15.46:0/4277924470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c100b81b0 msgr2=0x7f4c100d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:15:59.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c14fc4640 1 --2- 172.21.15.46:0/4277924470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c100b81b0 0x7f4c100d25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c14fc4640 1 -- 172.21.15.46:0/4277924470 >> v1:172.21.15.46:6789/0 conn(0x7f4c101615f0 legacy=0x7f4c100d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:59.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c0effd640 1 -- 172.21.15.46:0/4277924470 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1693157464 0 0) 0x7f4c100d15b0 con 0x7f4c101615f0 2024-05-09T03:15:59.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.070+0000 7f4c14fc4640 1 -- 172.21.15.46:0/4277924470 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c101610d0 con 0x7f4c100d2b20 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.071+0000 7f4c14fc4640 1 --2- 172.21.15.46:0/4277924470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c10163f40 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4c08009d90 tx=0x7f4c0802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d79303708b681f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.071+0000 7f4c0effd640 1 -- 172.21.15.46:0/4277924470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08007740 con 0x7f4c100d2b20 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.071+0000 7f4c0effd640 1 -- 172.21.15.46:0/4277924470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c080078e0 con 0x7f4c100d2b20 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.071+0000 7f4c0effd640 1 -- 172.21.15.46:0/4277924470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08007cb0 con 0x7f4c100d2b20 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 -- 172.21.15.46:0/4277924470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 msgr2=0x7f4c10163f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 --2- 172.21.15.46:0/4277924470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c10163f40 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4c08009d90 tx=0x7f4c0802a660 comp rx=0 tx=0).stop 2024-05-09T03:15:59.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 -- 172.21.15.46:0/4277924470 shutdown_connections 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 --2- 172.21.15.46:0/4277924470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c10163f40 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 --2- 172.21.15.46:0/4277924470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c100b81b0 0x7f4c100d25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 -- 172.21.15.46:0/4277924470 >> 172.21.15.46:0/4277924470 conn(0x7f4c1006e190 msgr2=0x7f4c1006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 -- 172.21.15.46:0/4277924470 shutdown_connections 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.072+0000 7f4c16a27640 1 -- 172.21.15.46:0/4277924470 wait complete. 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.074+0000 7f4c16a27640 1 Processor -- start 2024-05-09T03:15:59.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.074+0000 7f4c16a27640 1 -- start start 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c16a27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c100b81b0 0x7f4c101502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c16a27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c100c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c16a27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c101615f0 0x7f4c100ba980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c16a27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c100d1890 con 0x7f4c100d2b20 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c16a27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4c100b85c0 con 0x7f4c100b81b0 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c16a27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4c100d2f30 con 0x7f4c101615f0 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c0f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c100c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:59.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c0f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c100c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55436/0 (socket says 172.21.15.46:55436) 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.075+0000 7f4c0f7fe640 1 -- 172.21.15.46:0/410447477 learned_addr learned my addr 172.21.15.46:0/410447477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c14fc4640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c101615f0 0x7f4c100ba980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c100b81b0 0x7f4c101502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0f7fe640 1 -- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c101615f0 msgr2=0x7f4c100ba980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0f7fe640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c101615f0 0x7f4c100ba980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0f7fe640 1 -- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c100b81b0 msgr2=0x7f4c101502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0f7fe640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c100b81b0 0x7f4c101502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0f7fe640 1 -- 172.21.15.46:0/410447477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c100ca0a0 con 0x7f4c100d2b20 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0f7fe640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c100c9870 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400d620 tx=0x7f4c0400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0400b620 con 0x7f4c100d2b20 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c04019070 con 0x7f4c100d2b20 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c100ca380 con 0x7f4c100d2b20 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0400ba00 con 0x7f4c100d2b20 2024-05-09T03:15:59.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.076+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c100cd1e0 con 0x7f4c100d2b20 2024-05-09T03:15:59.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.077+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c1006a270 con 0x7f4c100d2b20 2024-05-09T03:15:59.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.078+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4c0400bba0 con 0x7f4c100d2b20 2024-05-09T03:15:59.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.078+0000 7f4c0cff9640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bfc0762d0 0x7f4bfc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:15:59.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.079+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f4c04097b00 con 0x7f4c100d2b20 2024-05-09T03:15:59.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.079+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bfc0762d0 0x7f4bfc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:15:59.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.079+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bfc0762d0 0x7f4bfc0787e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4c00000f60 tx=0x7f4c00007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:15:59.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.083+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4c040613d0 con 0x7f4c100d2b20 2024-05-09T03:15:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.318+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4c100b5480 con 0x7f4c100d2b20 2024-05-09T03:15:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.319+0000 7f4c0cff9640 1 -- 172.21.15.46:0/410447477 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v85) ==== 53+0+73 (secure 0 0 0) 0x7f4c04060cd0 con 0x7f4c100d2b20 2024-05-09T03:15:59.320 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:15:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bfc0762d0 msgr2=0x7f4bfc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bfc0762d0 0x7f4bfc0787e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4c00000f60 tx=0x7f4c00007c00 comp rx=0 tx=0).stop 2024-05-09T03:15:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 msgr2=0x7f4c100c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:15:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c100c9870 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400d620 tx=0x7f4c0400dae0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 shutdown_connections 2024-05-09T03:15:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4c101615f0 0x7f4c100ba980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bfc0762d0 0x7f4bfc0787e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c100d2b20 0x7f4c100c9870 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 --2- 172.21.15.46:0/410447477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c100b81b0 0x7f4c101502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:15:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 >> 172.21.15.46:0/410447477 conn(0x7f4c1006e190 msgr2=0x7f4c100b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:15:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 shutdown_connections 2024-05-09T03:15:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59.321+0000 7f4c16a27640 1 -- 172.21.15.46:0/410447477 wait complete. 2024-05-09T03:15:59.333 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:15:59.333 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:15:59.333 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:15:59.333 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:15:59.334 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:59.334 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:15:59.334 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:15:59.334 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:15:59.334 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:15:59.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:59.334 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:15:59.351 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42158sec preferred_lft 42158sec 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:15:59.352 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:15:59.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:59.353 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:15:59.416 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:15:59.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:59.418 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:15:59.474 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:15:59.474 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:15:59.474 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:15:59.474 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:15:59.475 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:15:59.475 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:15:59.475 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:15:59.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:15:59.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:15:59.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:15:59.475 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:15:59.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:15:59.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:15:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:15:59.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:59.604 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:15:59.621 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:15:59.621 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:15:59.621 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:15:59.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:59.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:15:59.688 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:15:59.688 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:15:59.688 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:59.744 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:15:59.744 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:15:59.744 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:15:59.800 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:15:59.866 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:15:59.923 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:15:59.923 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:15:59.923 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:15:59.923 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:15:59.923 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:15:59.923 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:15:59.924 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:59.925 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:15:59.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:15:59.926 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:15:59.979 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:15:59.980 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:16:00.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:16:00.047 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:16:00.108 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:16:00.108 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:16:00.108 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:16:00.108 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:16:00.108 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:16:00.108 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:16:00.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:00.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:00.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:00.111 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:16:00.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:16:00.120+0000 7f2da0f34480 -1 init, newargv = 0x7f2d84001080 newargc=15 2024-05-09T03:16:00.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[62256]: starting ceph client 2024-05-09T03:16:00.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[62256]: starting fuse 2024-05-09T03:16:00.137 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:16:00.137 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:16:00.137 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:16:00.138 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:16:00.139 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:00.139 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:16:00.139 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:16:00.258 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.62256.asok 2024-05-09T03:16:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:16:00.269 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62256.asok 2024-05-09T03:16:00.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:00.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62256.asok status 2024-05-09T03:16:00.416 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:16:00.416 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:16:00.416 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "62256", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7026, 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7026 2024-05-09T03:16:00.417 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3667732272 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3667732272 2024-05-09T03:16:00.418 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.7026 192.168.0.1:0/3667732272", 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3667732272", 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 85, 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 82, 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:16:00.419 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:16:00.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:00.425 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:16:00.425 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:00.442 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:16:00.442 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:00.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:00.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:00.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:00.509 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-05-09T03:16:00.571 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:00.571 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:00.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:00.628 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-05-09T03:16:00.736 INFO:teuthology.orchestra.run.smithi046.stderr:6+0 records in 2024-05-09T03:16:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:6+0 records out 2024-05-09T03:16:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0505052 s, 125 MB/s 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> import json 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2024-05-09T03:16:00.738 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(e.errno) 2024-05-09T03:16:00.739 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:00.739 DEBUG:teuthology.orchestra.run.smithi046:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-09T03:16:00.739 DEBUG:teuthology.orchestra.run.smithi046:> print(json.dumps( 2024-05-09T03:16:00.739 DEBUG:teuthology.orchestra.run.smithi046:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-09T03:16:00.739 DEBUG:teuthology.orchestra.run.smithi046:> indent=2)) 2024-05-09T03:16:00.739 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:16:00.808 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_mode": 33188, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_ino": 1099511627777, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_dev": 42, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_nlink": 1, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_uid": 1000, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_gid": 1316, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_size": 6291456, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_atime": 1715224560.685349, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_mtime": 1715224560.713874, 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout: "st_ctime": 1715224560.713874 2024-05-09T03:16:00.809 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:16:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:16:00.818 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:00.818 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:00.875 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:16:01.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.031+0000 7f588c5ce640 1 Processor -- start 2024-05-09T03:16:01.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.031+0000 7f588c5ce640 1 -- start start 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588c5ce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58840bb770 0x7f58840d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588c5ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d3180 0x7f58841642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588c5ce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5884164900 con 0x7f5884163eb0 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588c5ce640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f58840d4bb0 con 0x7f58840bb770 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588c5ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5884164be0 con 0x7f58840d3180 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588a36a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58840bb770 0x7f58840d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588a36a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58840bb770 0x7f58840d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55440/0 (socket says 172.21.15.46:55440) 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.032+0000 7f588a36a640 1 -- 172.21.15.46:0/1928566157 learned_addr learned my addr 172.21.15.46:0/1928566157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.033+0000 7f588ab6b640 1 --2- 172.21.15.46:0/1928566157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d3180 0x7f58841642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.033+0000 7f588a36a640 1 -- 172.21.15.46:0/1928566157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d3180 msgr2=0x7f58841642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:01.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.033+0000 7f588a36a640 1 --2- 172.21.15.46:0/1928566157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d3180 0x7f58841642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.033+0000 7f588a36a640 1 -- 172.21.15.46:0/1928566157 >> v1:172.21.15.88:6789/0 conn(0x7f5884163eb0 legacy=0x7f58840d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:01.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.033+0000 7f588a36a640 1 -- 172.21.15.46:0/1928566157 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58840d4840 con 0x7f58840bb770 2024-05-09T03:16:01.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.033+0000 7f588a36a640 1 --2- 172.21.15.46:0/1928566157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f58840d5040 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5880009e50 tx=0x7f588002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e8a07f9361562f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f5889368640 1 -- 172.21.15.46:0/1928566157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880036070 con 0x7f58840bb770 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f5889368640 1 -- 172.21.15.46:0/1928566157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5880031600 con 0x7f58840bb770 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f5889368640 1 -- 172.21.15.46:0/1928566157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800319d0 con 0x7f58840bb770 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 -- 172.21.15.46:0/1928566157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 msgr2=0x7f58840d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1928566157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f58840d5040 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5880009e50 tx=0x7f588002a680 comp rx=0 tx=0).stop 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 -- 172.21.15.46:0/1928566157 shutdown_connections 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1928566157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840d3180 0x7f58841642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1928566157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f58840d5040 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 -- 172.21.15.46:0/1928566157 >> 172.21.15.46:0/1928566157 conn(0x7f588406e370 msgr2=0x7f588406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:01.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 -- 172.21.15.46:0/1928566157 shutdown_connections 2024-05-09T03:16:01.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.034+0000 7f588c5ce640 1 -- 172.21.15.46:0/1928566157 wait complete. 2024-05-09T03:16:01.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.036+0000 7f588c5ce640 1 Processor -- start 2024-05-09T03:16:01.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.036+0000 7f588c5ce640 1 -- start start 2024-05-09T03:16:01.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.037+0000 7f588c5ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f5884159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.037+0000 7f588c5ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884163eb0 0x7f58841f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.037+0000 7f588c5ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840c9b10 0x7f588415c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.037+0000 7f588c5ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f58840d0bc0 con 0x7f58840bb770 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.037+0000 7f588c5ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f58840bbb80 con 0x7f5884163eb0 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.037+0000 7f588c5ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f58840d0d40 con 0x7f58840c9b10 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f5884159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f5884159200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55456/0 (socket says 172.21.15.46:55456) 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 -- 172.21.15.46:0/1982581375 learned_addr learned my addr 172.21.15.46:0/1982581375 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588ab6b640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840c9b10 0x7f588415c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f5889b69640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884163eb0 0x7f58841f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 -- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840c9b10 msgr2=0x7f588415c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840c9b10 0x7f588415c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 -- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884163eb0 msgr2=0x7f58841f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884163eb0 0x7f58841f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 -- 172.21.15.46:0/1982581375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588415b500 con 0x7f58840bb770 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588ab6b640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840c9b10 0x7f588415c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588a36a640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f5884159200 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5880009f70 tx=0x7f5880004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588003f020 con 0x7f58840bb770 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5880037040 con 0x7f58840bb770 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588415b7e0 con 0x7f58840bb770 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880036070 con 0x7f58840bb770 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.038+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588415bda0 con 0x7f58840bb770 2024-05-09T03:16:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.039+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f588406a270 con 0x7f58840bb770 2024-05-09T03:16:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.040+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5880042020 con 0x7f58840bb770 2024-05-09T03:16:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.041+0000 7f587affd640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5868076300 0x7f5868078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.041+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f58800b7fd0 con 0x7f58840bb770 2024-05-09T03:16:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.041+0000 7f5889b69640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5868076300 0x7f5868078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.042+0000 7f5889b69640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5868076300 0x7f5868078810 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f588406e880 tx=0x7f5874009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.045+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5880049070 con 0x7f58840bb770 2024-05-09T03:16:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.250+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f58840d4130 con 0x7f58840bb770 2024-05-09T03:16:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.251+0000 7f587affd640 1 -- 172.21.15.46:0/1982581375 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) ==== 80+0+0 (secure 0 0 0) 0x7f5880081a10 con 0x7f58840bb770 2024-05-09T03:16:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:16:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.252+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5868076300 msgr2=0x7f5868078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.252+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5868076300 0x7f5868078810 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f588406e880 tx=0x7f5874009060 comp rx=0 tx=0).stop 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.252+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 msgr2=0x7f5884159200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.252+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f5884159200 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f5880009f70 tx=0x7f5880004a40 comp rx=0 tx=0).stop 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 shutdown_connections 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58840c9b10 0x7f588415c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5884163eb0 0x7f58841f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5868076300 0x7f5868078810 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 --2- 172.21.15.46:0/1982581375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58840bb770 0x7f5884159200 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 >> 172.21.15.46:0/1982581375 conn(0x7f588406e370 msgr2=0x7f58840b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 shutdown_connections 2024-05-09T03:16:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.253+0000 7f588c5ce640 1 -- 172.21.15.46:0/1982581375 wait complete. 2024-05-09T03:16:01.264 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:01.264 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:01.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:16:01.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.435+0000 7f514187e640 1 Processor -- start 2024-05-09T03:16:01.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.435+0000 7f514187e640 1 -- start start 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.436+0000 7f514187e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0b7e70 0x7f513c0b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.436+0000 7f514187e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f513c0d3400 con 0x7f513c1605b0 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.436+0000 7f514187e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f513c0d4a60 con 0x7f513c0d2e90 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.436+0000 7f514187e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f513c0d4d40 con 0x7f513c0b7e70 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f513b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f513c1605b0 0x7f513c0b97e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40076/0 (socket says 172.21.15.46:40076) 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f513b7fe640 1 -- 172.21.15.46:0/1508875853 learned_addr learned my addr 172.21.15.46:0/1508875853 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:01.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f513a7fc640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0b7e70 0x7f513c0b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374043476 0 0) 0x7f513c0d3400 con 0x7f513c1605b0 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5124003520 con 0x7f513c1605b0 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018152326 0 0) 0x7f513c0d4a60 con 0x7f513c0d2e90 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f513c0d3400 con 0x7f513c0d2e90 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1694893609 0 0) 0x7f5124003520 con 0x7f513c1605b0 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0b7e70 msgr2=0x7f513c0b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0b7e70 0x7f513c0b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 >> v1:172.21.15.88:6789/0 conn(0x7f513c0d2e90 legacy=0x7f513c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0d5020 con 0x7f513c1605b0 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f5134001a70 con 0x7f513c1605b0 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f513a7fc640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0b7e70 0x7f513c0b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 >> v1:172.21.15.46:6789/0 conn(0x7f513c1605b0 legacy=0x7f513c0b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5124003c10 0x7f5124004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124004800 0x7f5124004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5124005290 0x7f512400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0d4d40 con 0x7f5124003c10 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f512400bfe0 con 0x7f5124004800 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f512400c1e0 con 0x7f5124005290 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.437+0000 7f513affd640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5124003c10 0x7f5124004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513a7fc640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124004800 0x7f5124004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513b7fe640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5124005290 0x7f512400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513affd640 1 -- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5124005290 msgr2=0x7f512400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513affd640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5124005290 0x7f512400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513affd640 1 -- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124004800 msgr2=0x7f5124004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513affd640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124004800 0x7f5124004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.438+0000 7f513affd640 1 -- 172.21.15.46:0/1508875853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5124003520 con 0x7f5124003c10 2024-05-09T03:16:01.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f513affd640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5124003c10 0x7f5124004060 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f512c0018d0 tx=0x7f5124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cc5ab8b77ffd55 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:01.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c001520 con 0x7f5124003c10 2024-05-09T03:16:01.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512c02acb0 con 0x7f5124003c10 2024-05-09T03:16:01.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f5139ffb640 1 -- 172.21.15.46:0/1508875853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c02ae60 con 0x7f5124003c10 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f514187e640 1 -- 172.21.15.46:0/1508875853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5124003c10 msgr2=0x7f5124004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f514187e640 1 --2- 172.21.15.46:0/1508875853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5124003c10 0x7f5124004060 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f512c0018d0 tx=0x7f5124005830 comp rx=0 tx=0).stop 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f513b7fe640 1 -- 172.21.15.46:0/1508875853 reap_dead start 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f514187e640 1 -- 172.21.15.46:0/1508875853 shutdown_connections 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f514187e640 1 -- 172.21.15.46:0/1508875853 >> 172.21.15.46:0/1508875853 conn(0x7f513c08d390 msgr2=0x7f513c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f514187e640 1 -- 172.21.15.46:0/1508875853 shutdown_connections 2024-05-09T03:16:01.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.439+0000 7f514187e640 1 -- 172.21.15.46:0/1508875853 wait complete. 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.441+0000 7f514187e640 1 Processor -- start 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.441+0000 7f514187e640 1 -- start start 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f514187e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124003c10 0x7f513c152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f514187e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 0x7f513c0cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f514187e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0cc0b0 0x7f513c0ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f514187e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0b4d80 con 0x7f513c0cb760 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f514187e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f513c0b4c00 con 0x7f5124003c10 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f514187e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f513c0b7e70 con 0x7f513c0cc0b0 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 0x7f513c0cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 0x7f513c0cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55486/0 (socket says 172.21.15.46:55486) 2024-05-09T03:16:01.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 -- 172.21.15.46:0/3702886221 learned_addr learned my addr 172.21.15.46:0/3702886221 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513affd640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124003c10 0x7f513c152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 -- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0cc0b0 msgr2=0x7f513c0ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0cc0b0 0x7f513c0ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 -- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124003c10 msgr2=0x7f513c152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124003c10 0x7f513c152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 -- 172.21.15.46:0/3702886221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f513c0b5400 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513affd640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124003c10 0x7f513c152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f513a7fc640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 0x7f513c0cbb70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f5130011a00 tx=0x7f5130011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.443+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513000fa10 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.444+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f513000fbb0 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.444+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f513c160af0 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.444+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130004030 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.444+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f513c1610b0 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.445+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f513c161790 con 0x7f513c0cb760 2024-05-09T03:16:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.445+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5130012040 con 0x7f513c0cb760 2024-05-09T03:16:01.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.446+0000 7f5123fff640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51280762d0 0x7f51280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:01.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.446+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f51300978c0 con 0x7f513c0cb760 2024-05-09T03:16:01.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.446+0000 7f513affd640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51280762d0 0x7f51280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:01.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.447+0000 7f513affd640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51280762d0 0x7f51280787e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f512c031000 tx=0x7f512c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:01.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.451+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5130019070 con 0x7f513c0cb760 2024-05-09T03:16:01.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.653+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f513c06a270 con 0x7f513c0cb760 2024-05-09T03:16:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.654+0000 7f5123fff640 1 -- 172.21.15.46:0/3702886221 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) ==== 80+0+0 (secure 0 0 0) 0x7f5130061220 con 0x7f513c0cb760 2024-05-09T03:16:01.655 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51280762d0 msgr2=0x7f51280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51280762d0 0x7f51280787e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f512c031000 tx=0x7f512c02e000 comp rx=0 tx=0).stop 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 msgr2=0x7f513c0cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 0x7f513c0cbb70 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f5130011a00 tx=0x7f5130011ec0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 shutdown_connections 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f513c0cc0b0 0x7f513c0ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f51280762d0 0x7f51280787e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f513c0cb760 0x7f513c0cbb70 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 --2- 172.21.15.46:0/3702886221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5124003c10 0x7f513c152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:01.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 >> 172.21.15.46:0/3702886221 conn(0x7f513c08d390 msgr2=0x7f513c0d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 shutdown_connections 2024-05-09T03:16:01.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:01.656+0000 7f514187e640 1 -- 172.21.15.46:0/3702886221 wait complete. 2024-05-09T03:16:01.668 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:16:01.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:01.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:01.692 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[62256]: fuse finished with error 0 and tester_r 0 2024-05-09T03:16:01.694 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:16:07.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:16:07.007 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:16:07.796 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:07.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:07.796 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:07.818 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:07.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.986+0000 7f56b7f27640 1 Processor -- start 2024-05-09T03:16:07.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.987+0000 7f56b7f27640 1 -- start start 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b7f27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56b00b5210 0x7f56b01613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b7f27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56b00b7ed0 0x7f56b00d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b7f27640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b00d5280 con 0x7f56b00d4800 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b7f27640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f56b00b5620 con 0x7f56b00b5210 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b7f27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f56b01609a0 con 0x7f56b00b7ed0 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56b00b5210 0x7f56b01613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56b00b5210 0x7f56b01613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44318/0 (socket says 172.21.15.46:44318) 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 -- 172.21.15.46:0/3398590996 learned_addr learned my addr 172.21.15.46:0/3398590996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:07.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b64c4640 1 --2- 172.21.15.46:0/3398590996 >> v2:172.21.15.88:3300/0 conn(0x7f56b00b7ed0 0x7f56b00d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 -- 172.21.15.46:0/3398590996 >> v2:172.21.15.88:3300/0 conn(0x7f56b00b7ed0 msgr2=0x7f56b00d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 --2- 172.21.15.46:0/3398590996 >> v2:172.21.15.88:3300/0 conn(0x7f56b00b7ed0 0x7f56b00d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 -- 172.21.15.46:0/3398590996 >> v1:172.21.15.88:6789/0 conn(0x7f56b00d4800 legacy=0x7f56b00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b5cc3640 1 -- 172.21.15.46:0/3398590996 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b00d5560 con 0x7f56b00b5210 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.988+0000 7f56b64c4640 1 --2- 172.21.15.46:0/3398590996 >> v2:172.21.15.88:3300/0 conn(0x7f56b00b7ed0 0x7f56b00d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.989+0000 7f56b5cc3640 1 --2- 172.21.15.46:0/3398590996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01613c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f56b006f030 tx=0x7f56ac02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7748f4a8504ab4cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.989+0000 7f56b4cc1640 1 -- 172.21.15.46:0/3398590996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac037070 con 0x7f56b00b5210 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.989+0000 7f56b4cc1640 1 -- 172.21.15.46:0/3398590996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ac032440 con 0x7f56b00b5210 2024-05-09T03:16:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.989+0000 7f56b4cc1640 1 -- 172.21.15.46:0/3398590996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac032810 con 0x7f56b00b5210 2024-05-09T03:16:07.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 -- 172.21.15.46:0/3398590996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 msgr2=0x7f56b01613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:07.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 --2- 172.21.15.46:0/3398590996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01613c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f56b006f030 tx=0x7f56ac02a140 comp rx=0 tx=0).stop 2024-05-09T03:16:07.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 -- 172.21.15.46:0/3398590996 shutdown_connections 2024-05-09T03:16:07.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 --2- 172.21.15.46:0/3398590996 >> v2:172.21.15.88:3300/0 conn(0x7f56b00b7ed0 0x7f56b00d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:07.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 --2- 172.21.15.46:0/3398590996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01613c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:07.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 -- 172.21.15.46:0/3398590996 >> 172.21.15.46:0/3398590996 conn(0x7f56b00d29a0 msgr2=0x7f56b00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 -- 172.21.15.46:0/3398590996 shutdown_connections 2024-05-09T03:16:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.990+0000 7f56b7f27640 1 -- 172.21.15.46:0/3398590996 wait complete. 2024-05-09T03:16:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.992+0000 7f56b7f27640 1 Processor -- start 2024-05-09T03:16:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.992+0000 7f56b7f27640 1 -- start start 2024-05-09T03:16:07.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b7f27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01fa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:07.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b7f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56b00d4800 0x7f56b01f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b7f27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56b00ccdf0 0x7f56b00bab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b7f27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56b00d0ae0 con 0x7f56b00b5210 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b7f27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0160ea0 con 0x7f56b00ccdf0 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b7f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f56b0163e60 con 0x7f56b00d4800 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01fa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01fa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44320/0 (socket says 172.21.15.46:44320) 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 -- 172.21.15.46:0/651593776 learned_addr learned my addr 172.21.15.46:0/651593776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b64c4640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56b00ccdf0 0x7f56b00bab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b54c2640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56b00d4800 0x7f56b01f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 -- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56b00d4800 msgr2=0x7f56b01f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56b00d4800 0x7f56b01f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 -- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56b00ccdf0 msgr2=0x7f56b00bab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:07.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56b00ccdf0 0x7f56b00bab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 -- 172.21.15.46:0/651593776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b0157140 con 0x7f56b00b5210 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b54c2640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56b00d4800 0x7f56b01f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56b5cc3640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01fa9a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f56ac02a180 tx=0x7f56ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac037040 con 0x7f56b00b5210 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.994+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ac042070 con 0x7f56b00b5210 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.995+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac030bb0 con 0x7f56b00b5210 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.995+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b0157420 con 0x7f56b00b5210 2024-05-09T03:16:07.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.995+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b01579e0 con 0x7f56b00b5210 2024-05-09T03:16:07.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.995+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56b00d2db0 con 0x7f56b00b5210 2024-05-09T03:16:07.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.996+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f56ac047460 con 0x7f56b00b5210 2024-05-09T03:16:07.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.997+0000 7f56a67fc640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56940762d0 0x7f56940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:07.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.997+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f56ac0b81b0 con 0x7f56b00b5210 2024-05-09T03:16:07.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.997+0000 7f56b54c2640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56940762d0 0x7f56940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:07.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:07.998+0000 7f56b54c2640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56940762d0 0x7f56940787e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f56a0006fd0 tx=0x7f56a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.002+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f56b00d2db0 con 0x7f56b00b5210 2024-05-09T03:16:08.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.238+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56b00d4f10 con 0x7f56b00b5210 2024-05-09T03:16:08.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.238+0000 7f56a67fc640 1 -- 172.21.15.46:0/651593776 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5180 (secure 0 0 0) 0x7f56b00d4f10 con 0x7f56b00b5210 2024-05-09T03:16:08.239 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:08.239 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:15:55.468035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6888},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6888":{"gid":6888,"name":"c","rank":0,"incarnation":81,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6834/2347304023","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2347304023},{"type":"v1","addr":"172.21.15.46:6834","nonce":2347304023}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6888,"qdb_cluster":[6888]},"id":6}]} 2024-05-09T03:16:08.240 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 85 2024-05-09T03:16:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56940762d0 msgr2=0x7f56940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56940762d0 0x7f56940787e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f56a0006fd0 tx=0x7f56a0008040 comp rx=0 tx=0).stop 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 msgr2=0x7f56b01fa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01fa9a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f56ac02a180 tx=0x7f56ac0046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 shutdown_connections 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56b00ccdf0 0x7f56b00bab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56b00d4800 0x7f56b01f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56940762d0 0x7f56940787e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 --2- 172.21.15.46:0/651593776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56b00b5210 0x7f56b01fa9a0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 >> 172.21.15.46:0/651593776 conn(0x7f56b00d29a0 msgr2=0x7f56b0163220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 shutdown_connections 2024-05-09T03:16:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.241+0000 7f56b7f27640 1 -- 172.21.15.46:0/651593776 wait complete. 2024-05-09T03:16:08.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T03:16:08.391 INFO:teuthology.orchestra.run.smithi046.stdout:{"return_code":0} 2024-05-09T03:16:08.391 DEBUG:tasks.cephfs.filesystem:_json_asok output { "return_code": 0 } 2024-05-09T03:16:08.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:08.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.507+0000 7f5581d7a640 1 Processor -- start 2024-05-09T03:16:08.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.507+0000 7f5581d7a640 1 -- start start 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.508+0000 7f5581d7a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f557c0bb770 0x7f557c0d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.508+0000 7f5581d7a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557c164900 con 0x7f557c163eb0 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.508+0000 7f5581d7a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f557c164be0 con 0x7f557c0d3180 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.508+0000 7f5581d7a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f557c0d4bb0 con 0x7f557c0bb770 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.508+0000 7f557affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f557c163eb0 0x7f557c0d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43146/0 (socket says 172.21.15.46:43146) 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557affd640 1 -- 172.21.15.46:0/3597615409 learned_addr learned my addr 172.21.15.46:0/3597615409 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557b7fe640 1 --2- 172.21.15.46:0/3597615409 >> v2:172.21.15.88:3300/0 conn(0x7f557c0bb770 0x7f557c0d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557a7fc640 1 -- 172.21.15.46:0/3597615409 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10662707 0 0) 0x7f557c164900 con 0x7f557c163eb0 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557a7fc640 1 -- 172.21.15.46:0/3597615409 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5564003520 con 0x7f557c163eb0 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557a7fc640 1 -- 172.21.15.46:0/3597615409 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 165387449 0 0) 0x7f5564003520 con 0x7f557c163eb0 2024-05-09T03:16:08.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557b7fe640 1 -- 172.21.15.46:0/3597615409 >> v1:172.21.15.88:6789/0 conn(0x7f557c0d3180 legacy=0x7f557c0b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557b7fe640 1 -- 172.21.15.46:0/3597615409 >> v1:172.21.15.46:6789/0 conn(0x7f557c163eb0 legacy=0x7f557c0d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.509+0000 7f557b7fe640 1 -- 172.21.15.46:0/3597615409 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c0d4840 con 0x7f557c0bb770 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.510+0000 7f557b7fe640 1 --2- 172.21.15.46:0/3597615409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c0d5040 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e2867b544c0f016 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.510+0000 7f557a7fc640 1 -- 172.21.15.46:0/3597615409 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c037070 con 0x7f557c0bb770 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.510+0000 7f557a7fc640 1 -- 172.21.15.46:0/3597615409 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c02ac50 con 0x7f557c0bb770 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.510+0000 7f557a7fc640 1 -- 172.21.15.46:0/3597615409 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55700016f0 con 0x7f557c0bb770 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 -- 172.21.15.46:0/3597615409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 msgr2=0x7f557c0d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 --2- 172.21.15.46:0/3597615409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c0d5040 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f556c0097a0 tx=0x7f556c02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 -- 172.21.15.46:0/3597615409 shutdown_connections 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 --2- 172.21.15.46:0/3597615409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c0d5040 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 -- 172.21.15.46:0/3597615409 >> 172.21.15.46:0/3597615409 conn(0x7f557c06e370 msgr2=0x7f557c06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 -- 172.21.15.46:0/3597615409 shutdown_connections 2024-05-09T03:16:08.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.511+0000 7f5581d7a640 1 -- 172.21.15.46:0/3597615409 wait complete. 2024-05-09T03:16:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.513+0000 7f5581d7a640 1 Processor -- start 2024-05-09T03:16:08.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.513+0000 7f5581d7a640 1 -- start start 2024-05-09T03:16:08.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f5581d7a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f5581d7a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 0x7f557c0b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f5581d7a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f557c0b7f30 0x7f557c0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f5581d7a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f557c0d0bc0 con 0x7f557c163eb0 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f5581d7a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f557c0bbb80 con 0x7f557c0bb770 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f5581d7a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f557c0d0d40 con 0x7f557c0b7f30 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f557affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 0x7f557c0b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f557affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 0x7f557c0b79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44328/0 (socket says 172.21.15.46:44328) 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.514+0000 7f557affd640 1 -- 172.21.15.46:0/4117115726 learned_addr learned my addr 172.21.15.46:0/4117115726 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557b7fe640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557bfff640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f557c0b7f30 0x7f557c0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557affd640 1 -- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f557c0b7f30 msgr2=0x7f557c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557affd640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f557c0b7f30 0x7f557c0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557affd640 1 -- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 msgr2=0x7f557c154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557affd640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557affd640 1 -- 172.21.15.46:0/4117115726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c154660 con 0x7f557c163eb0 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557bfff640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f557c0b7f30 0x7f557c0ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:08.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f557affd640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 0x7f557c0b79f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f557000c680 tx=0x7f557000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570005ad0 con 0x7f557c163eb0 2024-05-09T03:16:08.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5570005c90 con 0x7f557c163eb0 2024-05-09T03:16:08.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c0ca6a0 con 0x7f557c163eb0 2024-05-09T03:16:08.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55700072b0 con 0x7f557c163eb0 2024-05-09T03:16:08.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.515+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c0cac60 con 0x7f557c163eb0 2024-05-09T03:16:08.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.516+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f557c0b8340 con 0x7f557c163eb0 2024-05-09T03:16:08.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.517+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5570007450 con 0x7f557c163eb0 2024-05-09T03:16:08.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.517+0000 7f5563fff640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55680762d0 0x7f55680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.517+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f5570097390 con 0x7f557c163eb0 2024-05-09T03:16:08.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.518+0000 7f557b7fe640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55680762d0 0x7f55680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.518+0000 7f557b7fe640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55680762d0 0x7f55680787e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f556c034000 tx=0x7f556c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.522+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f557c0b8340 con 0x7f557c163eb0 2024-05-09T03:16:08.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.752+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f557c06a270 con 0x7f557c163eb0 2024-05-09T03:16:08.754 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:08.753+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.754+0000 7f5563fff640 1 -- 172.21.15.46:0/4117115726 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v86) ==== 161+0+0 (secure 0 0 0) 0x7f5570060c60 con 0x7f557c163eb0 2024-05-09T03:16:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:08.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.757+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55680762d0 msgr2=0x7f55680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.757+0000 7f5581d7a640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55680762d0 0x7f55680787e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f556c034000 tx=0x7f556c02e000 comp rx=0 tx=0).stop 2024-05-09T03:16:08.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.757+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 msgr2=0x7f557c0b79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.757+0000 7f5581d7a640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 0x7f557c0b79f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f557000c680 tx=0x7f557000cb40 comp rx=0 tx=0).stop 2024-05-09T03:16:08.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 shutdown_connections 2024-05-09T03:16:08.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f557c0b7f30 0x7f557c0ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55680762d0 0x7f55680787e0 secure :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f556c034000 tx=0x7f556c02e000 comp rx=0 tx=0).stop 2024-05-09T03:16:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f557c163eb0 0x7f557c0b79f0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 --2- 172.21.15.46:0/4117115726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f557c0bb770 0x7f557c154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 >> 172.21.15.46:0/4117115726 conn(0x7f557c06e370 msgr2=0x7f557c0d2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 shutdown_connections 2024-05-09T03:16:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.758+0000 7f5581d7a640 1 -- 172.21.15.46:0/4117115726 wait complete. 2024-05-09T03:16:08.761 INFO:tasks.ceph.mds.c.smithi046.stderr: -3158> 2024-05-09T03:15:59.039+0000 7fda98af8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:16:08.771 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata export - 2024-05-09T03:16:08.819 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:08.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.831+0000 7f9cf1fdbf40 1 Processor -- start 2024-05-09T03:16:08.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.831+0000 7f9cf1fdbf40 1 -- start start 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cf1fdbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cf1fdbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449db66f0 0x564449e575f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cf1fdbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449e57b30 0x564449e57f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cf1fdbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564449dd25f0 con 0x564449e54ef0 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cf1fdbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564449d2ff50 con 0x564449e57b30 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cf1fdbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564449dd28d0 con 0x564449db66f0 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44376/0 (socket says 172.21.15.46:44376) 2024-05-09T03:16:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 -- 172.21.15.46:0/3114310668 learned_addr learned my addr 172.21.15.46:0/3114310668 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449db66f0 0x564449e575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449e57b30 0x564449e57f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 -- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449db66f0 msgr2=0x564449e575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449db66f0 0x564449e575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 -- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449e57b30 msgr2=0x564449e57f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449e57b30 0x564449e57f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 -- 172.21.15.46:0/3114310668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564449e54a40 con 0x564449e54ef0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449db66f0 0x564449e575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449e57b30 0x564449e57f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.832+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db7590 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001840 tx=0x7f9cdc009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fb92c660d45a1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cea2a0640 1 -- 172.21.15.46:0/3114310668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc00e8e0 con 0x564449e54ef0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cea2a0640 1 -- 172.21.15.46:0/3114310668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cdc013030 con 0x564449e54ef0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cea2a0640 1 -- 172.21.15.46:0/3114310668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc00ecc0 con 0x564449e54ef0 2024-05-09T03:16:08.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/3114310668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 msgr2=0x564449db7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db7590 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001840 tx=0x7f9cdc009ab0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/3114310668 shutdown_connections 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449e57b30 0x564449e57f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449db66f0 0x564449e575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/3114310668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db7590 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/3114310668 >> 172.21.15.46:0/3114310668 conn(0x564449cabac0 msgr2=0x564449dd3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/3114310668 shutdown_connections 2024-05-09T03:16:08.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.833+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/3114310668 wait complete. 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.834+0000 7f9cf1fdbf40 1 Processor -- start 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.834+0000 7f9cf1fdbf40 1 -- start start 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.835+0000 7f9cf1fdbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449db66f0 0x564449eeee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.835+0000 7f9cf1fdbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.835+0000 7f9cf1fdbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449e57b30 0x564449e5aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cf1fdbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564449e5f090 con 0x564449e54ef0 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cf1fdbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564449e58480 con 0x564449db66f0 2024-05-09T03:16:08.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cf1fdbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564449e5ddf0 con 0x564449e57b30 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44382/0 (socket says 172.21.15.46:44382) 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 learned_addr learned my addr 172.21.15.46:0/2429007268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449e57b30 0x564449e5aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449db66f0 0x564449eeee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449e57b30 msgr2=0x564449e5aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564449e57b30 0x564449e5aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449db66f0 msgr2=0x564449eeee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564449db66f0 0x564449eeee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564449ef5990 con 0x564449e54ef0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db82b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00084b0 tx=0x7f9ce00085a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ce929e640 1 -- 172.21.15.46:0/2429007268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce001a070 con 0x564449e54ef0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ce929e640 1 -- 172.21.15.46:0/2429007268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce0016610 con 0x564449e54ef0 2024-05-09T03:16:08.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564449ef5fd0 con 0x564449e54ef0 2024-05-09T03:16:08.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9ce929e640 1 -- 172.21.15.46:0/2429007268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0016a20 con 0x564449e54ef0 2024-05-09T03:16:08.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.836+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564449ef7b30 con 0x564449e54ef0 2024-05-09T03:16:08.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.838+0000 7f9ce929e640 1 -- 172.21.15.46:0/2429007268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9ce001d070 con 0x564449e54ef0 2024-05-09T03:16:08.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.838+0000 7f9ce929e640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cd8074ce0 0x7f9cd80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9ce929e640 1 -- 172.21.15.46:0/2429007268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f9ce0096bd0 con 0x564449e54ef0 2024-05-09T03:16:08.841 INFO:teuthology.orchestra.run.smithi046.stdout:���� 2024-05-09T03:16:08.841 INFO:teuthology.orchestra.run.smithi046.stdout: �� 2024-05-09T03:16:08.841 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:08.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cd8074ce0 0x7f9cd80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x564449e505a0 0x564449e509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:1 12.0 12:00000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449e505a0 2024-05-09T03:16:08.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x564449e505a0 0x564449e509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cd8074ce0 0x7f9cd80771f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001840 tx=0x7f9cdc01c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.839+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x564449e505a0 0x564449e509b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.840+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.4 v2:172.21.15.88:6824/902225877 1 ==== osd_op_reply(1 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x564449e505a0 2024-05-09T03:16:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.840+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564449dc93a0 0x7f9ce40563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.840+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:08000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.840+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564449dc93a0 0x7f9ce40563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.840+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564449dc93a0 0x7f9ce40563a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.842+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(2 [pgnls start_epoch 83 out=177b] v82'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f9cdc00ed80 con 0x564449dc93a0 2024-05-09T03:16:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.842+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f9cdc019020 0x7f9cdc019430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.843+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:3 12.8 12:10000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9cdc019020 2024-05-09T03:16:08.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.843+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f9cdc019020 0x7f9cdc019430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.843+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f9cdc019020 0x7f9cdc019430 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.843+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(3 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.843+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f9ce0060020 0x7f9ce005ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.843+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:4 12.18 12:18000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9ce0060020 2024-05-09T03:16:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.844+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f9ce0060020 0x7f9ce005ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.844+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f9ce0060020 0x7f9ce005ea90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.846+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(4 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.846+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564449dd15f0 0x7f9ce4062410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.846+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:5 12.4 12:20000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.846+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564449dd15f0 0x7f9ce4062410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.847+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564449dd15f0 0x7f9ce4062410 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.850+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(5 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.850+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f9cdc00f040 0x7f9cdc0373c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.850+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:6 12.14 12:28000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9cdc00f040 2024-05-09T03:16:08.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.850+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f9cdc00f040 0x7f9cdc0373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.851+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f9cdc00f040 0x7f9cdc0373c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.854+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(6 [pgnls start_epoch 83 out=149b] v82'20 uv20 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.854+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.c 12:30000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.854+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(7 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.854+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.1c 12:38000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.854+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(8 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.854+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:9 12.2 12:40000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449e505a0 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.4 v2:172.21.15.88:6824/902225877 2 ==== osd_op_reply(9 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x564449e505a0 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:10 12.12 12:48000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9cdc019020 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(10 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9ceb2a2640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f9ce0068700 0x7f9ce005e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- osd_op(unknown.0.0:11 12.a 12:50000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9ce0068700 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f9ce0068700 0x7f9ce005e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.855+0000 7f9cec2a4640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f9ce0068700 0x7f9ce005e190 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.857+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.2 v2:172.21.15.46:6808/2404849282 1 ==== osd_op_reply(11 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0068700 2024-05-09T03:16:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.857+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:12 12.1a 12:58000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449e505a0 2024-05-09T03:16:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.858+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.4 v2:172.21.15.88:6824/902225877 3 ==== osd_op_reply(12 [pgnls start_epoch 83 out=49b] v81'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x564449e505a0 2024-05-09T03:16:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.858+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:13 12.6 12:60000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9ce0060020 2024-05-09T03:16:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.858+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(13 [pgnls start_epoch 83 out=77b] v82'4 uv4 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.858+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:14 12.16 12:68000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.859+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(14 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.859+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.e 12:70000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.859+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(15 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.859+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:16 12.1e 12:78000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449e505a0 2024-05-09T03:16:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.860+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.4 v2:172.21.15.88:6824/902225877 4 ==== osd_op_reply(16 [pgnls start_epoch 83 out=49b] v81'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x564449e505a0 2024-05-09T03:16:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.860+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:17 12.1 12:80000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.860+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(17 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.860+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:18 12.11 12:88000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9cdc00f040 2024-05-09T03:16:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.860+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(18 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.860+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- osd_op(unknown.0.0:19 12.9 12:90000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9ce0068700 2024-05-09T03:16:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.861+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.2 v2:172.21.15.46:6808/2404849282 2 ==== osd_op_reply(19 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0068700 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.861+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.19 12:98000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.861+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(20 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.861+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:21 12.5 12:a0000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.862+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(21 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.862+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:22 12.15 12:a8000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.862+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(22 [pgnls start_epoch 83 out=97b] v81'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.862+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f9cdc03a440 0x7f9cdc03a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.862+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:23 12.d 12:b0000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9cdc03a440 2024-05-09T03:16:08.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.862+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f9cdc03a440 0x7f9cdc03a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:08.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.863+0000 7f9cebaa3640 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f9cdc03a440 0x7f9cdc03a850 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:08.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.863+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(23 [pgnls start_epoch 83 out=74b] v81'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f9cdc034040 con 0x7f9cdc03a440 2024-05-09T03:16:08.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.863+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.1d 12:b8000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.863+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(24 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.863+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.3 12:c0000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dc93a0 2024-05-09T03:16:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.864+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(25 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.864+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:26 12.13 12:c8000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9ce0060020 2024-05-09T03:16:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.864+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(26 [pgnls start_epoch 83 out=98b] v82'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.864+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:27 12.b 12:d0000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9cdc019020 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.864+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(27 [pgnls start_epoch 83 out=76b] v82'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.864+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:28 12.1b 12:d8000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449e505a0 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.865+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.4 v2:172.21.15.88:6824/902225877 5 ==== osd_op_reply(28 [pgnls start_epoch 83 out=49b] v81'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ce404da90 con 0x564449e505a0 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.865+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:29 12.7 12:e0000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x7f9ce0060020 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.865+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(29 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.865+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:30 12.17 12:e8000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.866+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(30 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.866+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:31 12.f 12:f0000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.866+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(31 [pgnls start_epoch 83 out=73b] v81'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.866+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:32 12.1f 12:f8000000:%01:::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x564449dcc540 con 0x564449dd15f0 2024-05-09T03:16:08.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.866+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(32 [pgnls start_epoch 83 out=101b] v82'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.866+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:33 12.10 12:09401a81:::601.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449db87f0 con 0x564449dc93a0 2024-05-09T03:16:08.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.867+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(33 601.00000000 [read 0~0] v0'0 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.867+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:34 12.10 12:09401a81:::601.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef8260 con 0x564449dc93a0 2024-05-09T03:16:08.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.867+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(34 601.00000000 [getxattrs out=124b] v0'0 uv4 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.867+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:35 12.10 12:09401a81:::601.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef8680 con 0x564449dc93a0 2024-05-09T03:16:08.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.867+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(35 601.00000000 [omap-get-header out=274b] v0'0 uv4 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:36 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef8a70 con 0x564449dc93a0 2024-05-09T03:16:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(36 601.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:37 12.10 12:0bd6d154:::602.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef8e60 con 0x564449dc93a0 2024-05-09T03:16:08.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(37 602.00000000 [read 0~0] v0'0 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:38 12.10 12:0bd6d154:::602.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(38 602.00000000 [getxattrs out=124b] v0'0 uv6 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.868+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:39 12.10 12:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(39 602.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:40 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(40 602.00000000 [omap-get-vals out=5b] v0'0 uv6 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:41 12.10 12:0cf6ad0f:::10000000000.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(41 10000000000.00000000 [read 0~0] v0'0 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:42 12.10 12:0cf6ad0f:::10000000000.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(42 10000000000.00000000 [getxattrs out=124b] v0'0 uv10 ondisk = 0) ==== 164+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.873 INFO:teuthology.orchestra.run.smithi046.stdout: ��*$- 601.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray1 ���� �� ((88���� �� ��*$- 602.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray2 ���� �� ((88���� �� ��2,510000000000.00000000�������������������!�������������������������� ���~_layout��������_parent>8subdir 2024-05-09T03:16:08.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.869+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:43 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.870+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(43 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.870+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:44 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02400 con 0x564449dc93a0 2024-05-09T03:16:08.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.870+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(44 10000000000.00000000 [omap-get-vals out=525b] v0'0 uv10 ondisk = 0) ==== 164+0+525 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.870+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:45 12.10 12:0d82a743:::600.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.870+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(45 600.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.870+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:46 12.10 12:0d82a743:::600.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(46 600.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:47 12.10 12:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(47 600.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:48 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(48 600.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:49 12.10 12:0d89b25e:::603.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(49 603.00000000 [read 0~0] v0'0 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.871+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:50 12.10 12:0d89b25e:::603.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.872+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(50 603.00000000 [getxattrs out=124b] v0'0 uv8 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.872+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:51 12.10 12:0d89b25e:::603.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.872+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(51 603.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.872+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:52 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x564449dc93a0 2024-05-09T03:16:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.872+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(52 603.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.872+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:53 12.14 12:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ab0 con 0x7f9cdc00f040 2024-05-09T03:16:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.873+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(53 1.00000000.inode [read 0~582 out=582b] v0'0 uv17 ondisk = 0) ==== 160+0+582 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.873+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:54 12.14 12:287034c4:::1.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef6ea0 con 0x7f9cdc00f040 2024-05-09T03:16:08.880 INFO:teuthology.orchestra.run.smithi046.stdout: ���� ��  (�?8stray0 ���� �� ((88���� �� ��*$- 603.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray3 ���� �� ((88���� �� ��.(11.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)��? [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:55 12.14 12:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef72e0 con 0x7f9cdc00f040 2024-05-09T03:16:08.881 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.874+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(55 1.00000000.inode [omap-get-header] v0'0 uv17 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.874+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:56 12.14 12:287034c4:::1.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449ef76f0 con 0x7f9cdc00f040 2024-05-09T03:16:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.875+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(56 1.00000000.inode [omap-get-vals out=5b] v0'0 uv17 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.875+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:57 12.14 12:292cf221:::200.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02400 con 0x7f9cdc00f040 2024-05-09T03:16:08.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.875+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(57 200.00000000 [read 0~90 out=90b] v0'0 uv20 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.875+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:58 12.14 12:292cf221:::200.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f027a0 con 0x7f9cdc00f040 2024-05-09T03:16:08.883 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��*$- 200.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@�6@�6@/:@@@ ���� �� 2024-05-09T03:16:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.876+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(58 200.00000000 [getxattrs out=4b] v0'0 uv20 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.876+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:59 12.14 12:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02b40 con 0x7f9cdc00f040 2024-05-09T03:16:08.884 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.876+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(59 200.00000000 [omap-get-header] v0'0 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.876+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:60 12.14 12:292cf221:::200.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03ef0 con 0x7f9cdc00f040 2024-05-09T03:16:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.877+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(60 200.00000000 [omap-get-vals out=5b] v0'0 uv20 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.877+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:61 12.14 12:2e2fa760:::200.00000001:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9cdc00f040 2024-05-09T03:16:08.885 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��*$- 200.00000001�������������������!�������������������������� ��I:C:/:/:��XS&YA1e@��XS&YA1h^T�? ceph_sha1(c9da4049f638c0f84bdd9b4e33c42e86046ac554 ceph_versionYceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev) entity_id0hostname smithi046 mount_point/home/ubuntu/cephtest/mnt.0pid62256root/��? ���@��XS&YA1�? [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:62 12.14 12:2e2fa760:::200.00000001:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9cdc00f040 2024-05-09T03:16:08.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.878+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(62 200.00000001 [getxattrs out=4b] v0'0 uv16 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.888 INFO:teuthology.orchestra.run.smithi046.stdout:�? [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:63 12.14 12:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9cdc00f040 2024-05-09T03:16:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.878+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(63 200.00000001 [omap-get-header] v0'0 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.889 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.878+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:64 12.14 12:2e2fa760:::200.00000001:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03ef0 con 0x7f9cdc00f040 2024-05-09T03:16:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.878+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(64 200.00000001 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.878+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:65 12.14 12:2f55791f:::606.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9cdc00f040 2024-05-09T03:16:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.879+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(65 606.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.879+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:66 12.14 12:2f55791f:::606.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9cdc00f040 2024-05-09T03:16:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.879+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(66 606.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.879+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:67 12.14 12:2f55791f:::606.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9cdc00f040 2024-05-09T03:16:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.879+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(67 606.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.879+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:68 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03ef0 con 0x7f9cdc00f040 2024-05-09T03:16:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.880+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(68 606.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc00f040 2024-05-09T03:16:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.880+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:69 12.1c 12:3ed09add:::607.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x564449dc93a0 2024-05-09T03:16:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.880+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(69 607.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.880+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:70 12.1c 12:3ed09add:::607.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x564449dc93a0 2024-05-09T03:16:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(70 607.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:71 12.1c 12:3ed09add:::607.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x564449dc93a0 2024-05-09T03:16:08.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(71 607.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:72 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03ef0 con 0x564449dc93a0 2024-05-09T03:16:08.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(72 607.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:73 12.6 12:654134d2:::mds0_openfiles.0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f02ee0 con 0x7f9ce0060020 2024-05-09T03:16:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(73 mds0_openfiles.0 [read 0~0] v0'0 uv4 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.881+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:74 12.6 12:654134d2:::mds0_openfiles.0:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03280 con 0x7f9ce0060020 2024-05-09T03:16:08.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.882+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(74 mds0_openfiles.0 [getxattrs out=4b] v0'0 uv4 ondisk = 0) ==== 160+0+4 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.896 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��*$- 606.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray6 ���� �� ((88���� �� ��*$- 607.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray7 ���� �� ((88���� �� ��.(1mds0_openfiles.0�������������������!�������������������������� �� 2024-05-09T03:16:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.882+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:75 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03620 con 0x7f9ce0060020 2024-05-09T03:16:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.882+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(75 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv4 ondisk = 0) ==== 160+0+42 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.882+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:76 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f049d0 con 0x7f9ce0060020 2024-05-09T03:16:08.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.882+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(76 mds0_openfiles.0 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.882+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:77 12.e 12:717a0223:::608.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(77 608.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:78 12.e 12:717a0223:::608.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(78 608.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:79 12.e 12:717a0223:::608.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(79 608.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:80 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04d70 con 0x564449dc93a0 2024-05-09T03:16:08.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(80 608.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.883+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:81 12.15 12:a93a17c2:::604.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dd15f0 2024-05-09T03:16:08.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.884+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(81 604.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.884+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:82 12.15 12:a93a17c2:::604.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dd15f0 2024-05-09T03:16:08.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.884+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(82 604.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.884+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:83 12.15 12:a93a17c2:::604.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dd15f0 2024-05-09T03:16:08.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.885+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(83 604.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.885+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:84 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04d70 con 0x564449dd15f0 2024-05-09T03:16:08.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.885+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(84 604.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.885+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:85 12.15 12:aa448500:::500.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dd15f0 2024-05-09T03:16:08.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.886+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(85 500.00000000 [read 0~90 out=90b] v0'0 uv3 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.886+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:86 12.15 12:aa448500:::500.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dd15f0 2024-05-09T03:16:08.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.886+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(86 500.00000000 [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.903 INFO:teuthology.orchestra.run.smithi046.stdout:���� ��4.*ceph fs volume v011 ���� �� ��*$- 608.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray8 ���� �� ((88���� �� ��*$- 604.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray4 ���� �� ((88���� �� ��*$- 500.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@@@@@@ ���� �� 2024-05-09T03:16:08.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.886+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:87 12.15 12:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dd15f0 2024-05-09T03:16:08.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.887+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(87 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.903 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.887+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:88 12.15 12:aa448500:::500.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04d70 con 0x564449dd15f0 2024-05-09T03:16:08.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.887+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(88 500.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.887+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:89 12.d 12:b50e409b:::mds_snaptable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x7f9cdc03a440 2024-05-09T03:16:08.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.888+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(89 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x7f9cdc034040 con 0x7f9cdc03a440 2024-05-09T03:16:08.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.888+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:90 12.d 12:b50e409b:::mds_snaptable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x7f9cdc03a440 2024-05-09T03:16:08.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.888+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(90 mds_snaptable [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 157+0+4 (crc 0 0 0) 0x7f9cdc034040 con 0x7f9cdc03a440 2024-05-09T03:16:08.905 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��+%. mds_snaptable�������������������!�������������������������� ��`ZFF4���� �� 2024-05-09T03:16:08.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.888+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:91 12.d 12:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x7f9cdc03a440 2024-05-09T03:16:08.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(91 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x7f9cdc03a440 2024-05-09T03:16:08.906 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:92 12.d 12:b50e409b:::mds_snaptable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04d70 con 0x7f9cdc03a440 2024-05-09T03:16:08.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(92 mds_snaptable [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x7f9cdc03a440 2024-05-09T03:16:08.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:93 12.1d 12:b871830b:::605.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(93 605.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:94 12.1d 12:b871830b:::605.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(94 605.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.889+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:95 12.1d 12:b871830b:::605.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.890+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(95 605.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.890+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:96 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x564449dc93a0 2024-05-09T03:16:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.890+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(96 605.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dc93a0 2024-05-09T03:16:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.890+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f039c0 con 0x7f9ce0060020 2024-05-09T03:16:08.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.890+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(97 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.890+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:c91d4a1d:::mds0_inotable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f03db0 con 0x7f9ce0060020 2024-05-09T03:16:08.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(98 mds0_inotable [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 157+0+4 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.910 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��*$- 605.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray5 ���� �� ((88���� �� ��+%. mds0_inotable�������������������!�������������������������� ��<6""��������� �� 2024-05-09T03:16:08.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04150 con 0x7f9ce0060020 2024-05-09T03:16:08.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(99 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.911 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04150 con 0x7f9ce0060020 2024-05-09T03:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(100 mds0_inotable [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:101 12.13 12:cd5a64a3:::100.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04150 con 0x7f9ce0060020 2024-05-09T03:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(101 100.00000000 [read 0~0] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.891+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:102 12.13 12:cd5a64a3:::100.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04150 con 0x7f9ce0060020 2024-05-09T03:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.892+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(102 100.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.913 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��*$- 100.00000000�������������������!�������������������������� �� 2024-05-09T03:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.892+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.13 12:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04150 con 0x7f9ce0060020 2024-05-09T03:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.892+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.914 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ( 2024-05-09T03:16:08.914 INFO:teuthology.orchestra.run.smithi046.stdout:( 2024-05-09T03:16:08.914 INFO:teuthology.orchestra.run.smithi046.stdout:8 2024-05-09T03:16:08.914 INFO:teuthology.orchestra.run.smithi046.stdout:�? [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449f04150 con 0x7f9ce0060020 2024-05-09T03:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.892+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 100.00000000 [omap-get-vals out=5195b] v0'0 uv1 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.915 INFO:teuthology.orchestra.run.smithi046.stdout:�? [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:105 12.b 12:d0630e4c:::mds0_sessionmap:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9cdc019020 2024-05-09T03:16:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(105 mds0_sessionmap [read 0~0] v0'0 uv3 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:106 12.b 12:d0630e4c:::mds0_sessionmap:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9cdc019020 2024-05-09T03:16:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 5 ==== osd_op_reply(106 mds0_sessionmap [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 159+0+4 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.916 INFO:teuthology.orchestra.run.smithi046.stdout: stray0_head�i����? [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:107 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9cdc019020 2024-05-09T03:16:08.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 6 ==== osd_op_reply(107 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) ==== 159+0+14 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:108 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9cdc019020 2024-05-09T03:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9ceb2a2640 1 -- 172.21.15.46:0/2429007268 <== osd.3 v2:172.21.15.46:6816/3177037259 7 ==== osd_op_reply(108 mds0_sessionmap [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 159+0+5 (crc 0 0 0) 0x7f9ce0026070 con 0x7f9cdc019020 2024-05-09T03:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.893+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9ce0060020 2024-05-09T03:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(109 609.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9ce0060020 2024-05-09T03:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(110 609.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:111 12.7 12:e0b41d06:::609.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9ce0060020 2024-05-09T03:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(111 609.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:112 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x7f9ce0060020 2024-05-09T03:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(112 609.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9ce404da90 con 0x7f9ce0060020 2024-05-09T03:16:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.894+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.895+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(113 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.895+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.896+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(114 400.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.921 INFO:teuthology.orchestra.run.smithi046.stdout:���� ������ �� ��*$- 609.00000000�������������������!�������������������������� ���~_layout��������_parent>8 stray9 ���� �� ((88���� �� ��*$- 400.00000000�������������������!�������������������������� ��0*���� �� 2024-05-09T03:16:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.896+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.896+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(115 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.922 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.896+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.f 12:f1f69726:::400.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.896+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(116 400.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.896+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.897+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(117 100.00000000.inode [read 0~582 out=582b] v0'0 uv3 ondisk = 0) ==== 162+0+582 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.923 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��0*3100.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)��? [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.924 INFO:teuthology.orchestra.run.smithi046.stdout:8 �? [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.898+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(119 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.925 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� 2024-05-09T03:16:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.898+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.898+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(120 100.00000000.inode [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 162+0+5 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.925 INFO:teuthology.orchestra.run.smithi046.stdout:���� �� ��("+ 2024-05-09T03:16:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.898+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.898+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(121 1.00000000 [read 0~0] v0'0 uv6 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.898+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.899+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(122 1.00000000 [getxattrs out=92b] v0'0 uv6 ondisk = 0) ==== 154+0+92 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.899+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.899+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(123 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 154+0+274 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.899+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564449e5b410 con 0x564449dd15f0 2024-05-09T03:16:08.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cebaa3640 1 -- 172.21.15.46:0/2429007268 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(124 1.00000000 [omap-get-vals out=524b] v0'0 uv6 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x7f9cdc034040 con 0x564449dd15f0 2024-05-09T03:16:08.928 INFO:teuthology.orchestra.run.smithi046.stdout:1.00000000�������������������!�������������������������� ��d^_layout@@ _parent ���� ��  (�?> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564449dc93a0 msgr2=0x7f9ce40563a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564449dc93a0 0x7f9ce40563a0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f9ce0060020 msgr2=0x7f9ce005ea90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f9ce0060020 0x7f9ce005ea90 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f9ce0068700 msgr2=0x7f9ce005e190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f9ce0068700 0x7f9ce005e190 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f9cdc019020 msgr2=0x7f9cdc019430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f9cdc019020 0x7f9cdc019430 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x564449e505a0 msgr2=0x564449e509b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x564449e505a0 0x564449e509b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f9cdc00f040 msgr2=0x7f9cdc0373c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f9cdc00f040 0x7f9cdc0373c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564449dd15f0 msgr2=0x7f9ce4062410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564449dd15f0 0x7f9ce4062410 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f9cdc03a440 msgr2=0x7f9cdc03a850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 reap_dead start 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f9cdc03a440 0x7f9cdc03a850 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 reap_dead start 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 reap_dead start 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cec2a4640 1 -- 172.21.15.46:0/2429007268 reap_dead start 2024-05-09T03:16:08.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cd8074ce0 msgr2=0x7f9cd80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cd8074ce0 0x7f9cd80771f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc001840 tx=0x7f9cdc01c040 comp rx=0 tx=0).stop 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 msgr2=0x564449db82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db82b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00084b0 tx=0x7f9ce00085a0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 shutdown_connections 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cd8074ce0 0x7f9cd80771f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 --2- 172.21.15.46:0/2429007268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564449e54ef0 0x564449db82b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 >> 172.21.15.46:0/2429007268 conn(0x564449cabac0 msgr2=0x564449dd3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 shutdown_connections 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:08.900+0000 7f9cf1fdbf40 1 -- 172.21.15.46:0/2429007268 wait complete. 2024-05-09T03:16:08.932 INFO:teuthology.orchestra.run.smithi046.stdout:���������? adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata ls 2024-05-09T03:16:09.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.029+0000 7f25bff1ff40 1 Processor -- start 2024-05-09T03:16:09.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.029+0000 7f25bff1ff40 1 -- start start 2024-05-09T03:16:09.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25bff1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d46ffe390 0x556d46feb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25bff1ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 0x556d470035a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25bff1ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d47091a30 0x556d47089380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25bff1ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556d470075e0 con 0x556d470065e0 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25bff1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556d46f64f50 con 0x556d46ffe390 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25bff1ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556d470078c0 con 0x556d47091a30 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25b88bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 0x556d470035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25b88bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 0x556d470035a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44396/0 (socket says 172.21.15.46:44396) 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.030+0000 7f25b88bb640 1 -- 172.21.15.46:0/124805539 learned_addr learned my addr 172.21.15.46:0/124805539 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b90bc640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d46ffe390 0x556d46feb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b98bd640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d47091a30 0x556d47089380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b88bb640 1 -- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d47091a30 msgr2=0x556d47089380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b88bb640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d47091a30 0x556d47089380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b88bb640 1 -- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d46ffe390 msgr2=0x556d46feb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b88bb640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d46ffe390 0x556d46feb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b88bb640 1 -- 172.21.15.46:0/124805539 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d47093dc0 con 0x556d470065e0 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b90bc640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d46ffe390 0x556d46feb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b98bd640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d47091a30 0x556d47089380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25b88bb640 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 0x556d470035a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f25b0001e20 tx=0x7f25b000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca2f58f8ac4bf30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25ab7fe640 1 -- 172.21.15.46:0/124805539 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0008070 con 0x556d470065e0 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25ab7fe640 1 -- 172.21.15.46:0/124805539 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b000d500 con 0x556d470065e0 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.031+0000 7f25ab7fe640 1 -- 172.21.15.46:0/124805539 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b000d970 con 0x556d470065e0 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 -- 172.21.15.46:0/124805539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 msgr2=0x556d470035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 0x556d470035a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f25b0001e20 tx=0x7f25b000b6a0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 -- 172.21.15.46:0/124805539 shutdown_connections 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d47091a30 0x556d47089380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d470065e0 0x556d470035a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/124805539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d46ffe390 0x556d46feb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 -- 172.21.15.46:0/124805539 >> 172.21.15.46:0/124805539 conn(0x556d46ee0ac0 msgr2=0x556d470016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 -- 172.21.15.46:0/124805539 shutdown_connections 2024-05-09T03:16:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.032+0000 7f25bff1ff40 1 -- 172.21.15.46:0/124805539 wait complete. 2024-05-09T03:16:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.033+0000 7f25bff1ff40 1 Processor -- start 2024-05-09T03:16:09.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.033+0000 7f25bff1ff40 1 -- start start 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25bff1ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 0x556d4712f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25bff1ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d470065e0 0x556d470885f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25bff1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d47091a30 0x556d470035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25bff1ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556d470086d0 con 0x556d46ffe390 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25bff1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556d47122b10 con 0x556d47091a30 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25bff1ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556d46fbf830 con 0x556d470065e0 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 0x556d4712f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 0x556d4712f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44404/0 (socket says 172.21.15.46:44404) 2024-05-09T03:16:09.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 learned_addr learned my addr 172.21.15.46:0/4191386785 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d47091a30 0x556d470035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d470065e0 0x556d470885f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d470065e0 msgr2=0x556d470885f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d470065e0 0x556d470885f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d47091a30 msgr2=0x556d470035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d47091a30 0x556d470035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d4712e5e0 con 0x556d46ffe390 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d47091a30 0x556d470035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556d470065e0 0x556d470885f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.034+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 0x556d4712f080 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f25ac007e00 tx=0x7f25ac006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.035+0000 7f25aa7fc640 1 -- 172.21.15.46:0/4191386785 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac017070 con 0x556d46ffe390 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.035+0000 7f25aa7fc640 1 -- 172.21.15.46:0/4191386785 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ac010710 con 0x556d46ffe390 2024-05-09T03:16:09.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.035+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556d46fbfa80 con 0x556d46ffe390 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.035+0000 7f25aa7fc640 1 -- 172.21.15.46:0/4191386785 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ac010ac0 con 0x556d46ffe390 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.035+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d4712e880 con 0x556d46ffe390 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.036+0000 7f25aa7fc640 1 -- 172.21.15.46:0/4191386785 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25ac01a070 con 0x556d46ffe390 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25aa7fc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f259c074ce0 0x7f259c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25aa7fc640 1 -- 172.21.15.46:0/4191386785 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f25ac095850 con 0x556d46ffe390 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f259c074ce0 0x7f259c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x556d47091e40 0x556d47092250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:1 12.0 12:00000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x556d47091e40 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x556d47091e40 0x556d47092250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.037+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f259c074ce0 0x7f259c0771f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x556d47000fc0 tx=0x7f25b0009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.038+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x556d47091e40 0x556d47092250 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.038+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.4 v2:172.21.15.88:6824/902225877 1 ==== osd_op_reply(1 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x556d47091e40 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.038+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f25b4022980 0x7f25b405a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.038+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:08000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b4022980 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.038+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f25b4022980 0x7f25b405a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.039+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f25b4022980 0x7f25b405a960 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.039+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(2 [pgnls start_epoch 83 out=177b] v82'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.039+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f25ac05ed50 0x7f25ac05d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.039+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:3 12.8 12:10000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25ac05ed50 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.039+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f25ac05ed50 0x7f25ac05d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.040+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f25ac05ed50 0x7f25ac05d7c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.040+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(3 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b000dcb0 con 0x7f25ac05ed50 2024-05-09T03:16:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.040+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f25b001b440 0x7f25b001b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.040+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:4 12.18 12:18000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b001b440 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.040+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f25b001b440 0x7f25b001b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.040+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f25b001b440 0x7f25b001b850 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.041+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(4 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x7f25b001b440 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.041+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f25b401f2e0 0x7f25b406c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.041+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:5 12.4 12:20000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b401f2e0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.041+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f25b401f2e0 0x7f25b406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.041+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f25b401f2e0 0x7f25b406c420 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.042+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(5 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.042+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f25ac067380 0x7f25ac095150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.042+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:6 12.14 12:28000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25ac067380 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.042+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f25ac067380 0x7f25ac095150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.042+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f25ac067380 0x7f25ac095150 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.043+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(6 [pgnls start_epoch 83 out=149b] v82'20 uv20 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f25b0036040 con 0x7f25ac067380 2024-05-09T03:16:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.043+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.c 12:30000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b4022980 2024-05-09T03:16:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.043+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(7 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.043+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.1c 12:38000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b4022980 2024-05-09T03:16:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.043+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(8 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.043+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:9 12.2 12:40000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x556d47091e40 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.4 v2:172.21.15.88:6824/902225877 2 ==== osd_op_reply(9 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x556d47091e40 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:10 12.12 12:48000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25ac05ed50 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(10 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b0036040 con 0x7f25ac05ed50 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b88bb640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f25b000f3f0 0x7f25b000f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- osd_op(unknown.0.0:11 12.a 12:50000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b000f3f0 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f25b000f3f0 0x7f25b000f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.044+0000 7f25b98bd640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f25b000f3f0 0x7f25b000f800 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.045+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.2 v2:172.21.15.46:6808/2404849282 1 ==== osd_op_reply(11 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x7f25b000f3f0 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.045+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:12 12.1a 12:58000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x556d47091e40 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.045+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.4 v2:172.21.15.88:6824/902225877 3 ==== osd_op_reply(12 [pgnls start_epoch 83 out=49b] v81'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x556d47091e40 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.045+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:13 12.6 12:60000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b001b440 2024-05-09T03:16:09.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.045+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(13 [pgnls start_epoch 83 out=77b] v82'4 uv4 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f25b4057030 con 0x7f25b001b440 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.045+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:14 12.16 12:68000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b401f2e0 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.046+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(14 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.046+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.e 12:70000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b4022980 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.046+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(15 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.046+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:16 12.1e 12:78000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x556d47091e40 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.046+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.4 v2:172.21.15.88:6824/902225877 4 ==== osd_op_reply(16 [pgnls start_epoch 83 out=49b] v81'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x556d47091e40 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.046+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:17 12.1 12:80000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b401f2e0 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(17 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:18 12.11 12:88000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25ac067380 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(18 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b0036040 con 0x7f25ac067380 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- osd_op(unknown.0.0:19 12.9 12:90000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b000f3f0 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.2 v2:172.21.15.46:6808/2404849282 2 ==== osd_op_reply(19 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x7f25b000f3f0 2024-05-09T03:16:09.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.19 12:98000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b4022980 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.047+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(20 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:21 12.5 12:a0000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b401f2e0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(21 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:22 12.15 12:a8000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x556d47003ae0 con 0x7f25b401f2e0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(22 [pgnls start_epoch 83 out=97b] v81'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f25ac05cec0 0x7f25ac093270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:23 12.d 12:b0000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25ac05cec0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.048+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f25ac05cec0 0x7f25ac093270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.049+0000 7f25b90bc640 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f25ac05cec0 0x7f25ac093270 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.049+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(23 [pgnls start_epoch 83 out=74b] v81'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25ac05cec0 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.049+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.1d 12:b8000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b4022980 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.049+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(24 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.049+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.3 12:c0000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b4022980 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.050+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(25 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b4022980 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.050+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:26 12.13 12:c8000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b001b440 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.050+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(26 [pgnls start_epoch 83 out=98b] v82'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f25b4057030 con 0x7f25b001b440 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.050+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:27 12.b 12:d0000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25ac05ed50 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.050+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(27 [pgnls start_epoch 83 out=76b] v82'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f25b0036040 con 0x7f25ac05ed50 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.050+0000 7f25b88bb640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:28 12.1b 12:d8000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x556d47091e40 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.051+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.4 v2:172.21.15.88:6824/902225877 5 ==== osd_op_reply(28 [pgnls start_epoch 83 out=49b] v81'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25b4057030 con 0x556d47091e40 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.051+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:29 12.7 12:e0000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b001b440 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.051+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(29 [pgnls start_epoch 83 out=73b] v81'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f25b4057030 con 0x7f25b001b440 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.051+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:30 12.17 12:e8000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b401f2e0 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.051+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(30 [pgnls start_epoch 83 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.051+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:31 12.f 12:f0000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b401f2e0 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(31 [pgnls start_epoch 83 out=73b] v81'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:32 12.1f 12:f8000000::::head [pgnls start_epoch 83 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e83) -- 0x7f25ac070070 con 0x7f25b401f2e0 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b90bc640 1 -- 172.21.15.46:0/4191386785 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(32 [pgnls start_epoch 83 out=101b] v82'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f25ac05fd70 con 0x7f25b401f2e0 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stdout:601.00000000 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stdout:602.00000000 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stdout:10000000000.00000000 2024-05-09T03:16:09.059 INFO:teuthology.orchestra.run.smithi046.stdout:600.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:603.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:1.00000000.inode 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:200.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:200.00000001 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:606.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:607.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:mds0_openfiles.0 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:608.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:604.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:500.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:mds_snaptable 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:605.00000000 2024-05-09T03:16:09.060 INFO:teuthology.orchestra.run.smithi046.stdout:mds0_inotable 2024-05-09T03:16:09.061 INFO:teuthology.orchestra.run.smithi046.stdout:100.00000000 2024-05-09T03:16:09.061 INFO:teuthology.orchestra.run.smithi046.stdout:mds0_sessionmap 2024-05-09T03:16:09.061 INFO:teuthology.orchestra.run.smithi046.stdout:609.00000000 2024-05-09T03:16:09.061 INFO:teuthology.orchestra.run.smithi046.stdout:400.00000000 2024-05-09T03:16:09.061 INFO:teuthology.orchestra.run.smithi046.stdout:100.00000000.inode 2024-05-09T03:16:09.061 INFO:teuthology.orchestra.run.smithi046.stdout:1.00000000 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f25b4022980 msgr2=0x7f25b405a960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f25b4022980 0x7f25b405a960 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f25b001b440 msgr2=0x7f25b001b850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f25b001b440 0x7f25b001b850 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f25b000f3f0 msgr2=0x7f25b000f800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f25b000f3f0 0x7f25b000f800 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f25ac05ed50 msgr2=0x7f25ac05d7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7f25ac05ed50 0x7f25ac05d7c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x556d47091e40 msgr2=0x556d47092250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x556d47091e40 0x556d47092250 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f25ac067380 msgr2=0x7f25ac095150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f25ac067380 0x7f25ac095150 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 reap_dead start 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f25b401f2e0 msgr2=0x7f25b406c420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7f25b401f2e0 0x7f25b406c420 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f25ac05cec0 msgr2=0x7f25ac093270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7f25ac05cec0 0x7f25ac093270 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 reap_dead start 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 reap_dead start 2024-05-09T03:16:09.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25b98bd640 1 -- 172.21.15.46:0/4191386785 reap_dead start 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f259c074ce0 msgr2=0x7f259c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f259c074ce0 0x7f259c0771f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x556d47000fc0 tx=0x7f25b0009000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 msgr2=0x556d4712f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 0x556d4712f080 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f25ac007e00 tx=0x7f25ac006750 comp rx=0 tx=0).stop 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 shutdown_connections 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f259c074ce0 0x7f259c0771f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 --2- 172.21.15.46:0/4191386785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556d46ffe390 0x556d4712f080 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 >> 172.21.15.46:0/4191386785 conn(0x556d46ee0ac0 msgr2=0x556d470013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 shutdown_connections 2024-05-09T03:16:09.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.052+0000 7f25bff1ff40 1 -- 172.21.15.46:0/4191386785 wait complete. 2024-05-09T03:16:09.066 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 10000000000.00000000 2024-05-09T03:16:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.122+0000 7f4dd5f1ef40 1 Processor -- start 2024-05-09T03:16:09.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.122+0000 7f4dd5f1ef40 1 -- start start 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dd5f1ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d337e0 0x558d84d63a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dd5f1ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d657b0 0x558d84d77de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dd5f1ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 0x558d84d78320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dd5f1ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d84d7a4d0 con 0x558d84d7c010 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dd5f1ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d84cdcf50 con 0x558d84d337e0 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dd5f1ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558d84d7f8a0 con 0x558d84d657b0 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dcf8ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 0x558d84d78320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dcf8ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 0x558d84d78320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44408/0 (socket says 172.21.15.46:44408) 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.123+0000 7f4dcf8ce640 1 -- 172.21.15.46:0/543294911 learned_addr learned my addr 172.21.15.46:0/543294911 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf0cd640 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d337e0 0x558d84d63a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dce8cc640 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d657b0 0x558d84d77de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf8ce640 1 -- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d657b0 msgr2=0x558d84d77de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf8ce640 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d657b0 0x558d84d77de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf8ce640 1 -- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d337e0 msgr2=0x558d84d63a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf8ce640 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d337e0 0x558d84d63a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf8ce640 1 -- 172.21.15.46:0/543294911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d84d64680 con 0x558d84d7c010 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcf8ce640 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 0x558d84d78320 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8001690 tx=0x7f4dc8001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551690f9a4b6f27f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcd8ca640 1 -- 172.21.15.46:0/543294911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8017070 con 0x558d84d7c010 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcd8ca640 1 -- 172.21.15.46:0/543294911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc800f710 con 0x558d84d7c010 2024-05-09T03:16:09.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.124+0000 7f4dcd8ca640 1 -- 172.21.15.46:0/543294911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800fb00 con 0x558d84d7c010 2024-05-09T03:16:09.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/543294911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 msgr2=0x558d84d78320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 0x558d84d78320 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8001690 tx=0x7f4dc8001b50 comp rx=0 tx=0).stop 2024-05-09T03:16:09.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/543294911 shutdown_connections 2024-05-09T03:16:09.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d7c010 0x558d84d78320 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d657b0 0x558d84d77de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/543294911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d337e0 0x558d84d63a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/543294911 >> 172.21.15.46:0/543294911 conn(0x558d84c58ac0 msgr2=0x558d84d63190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/543294911 shutdown_connections 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.125+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/543294911 wait complete. 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.126+0000 7f4dd5f1ef40 1 Processor -- start 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.126+0000 7f4dd5f1ef40 1 -- start start 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dd5f1ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d337e0 0x558d84e012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dd5f1ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d657b0 0x558d84d7dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dd5f1ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d7c010 0x558d84d7e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dd5f1ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d84e023a0 con 0x558d84d657b0 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dd5f1ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d84d77950 con 0x558d84d7c010 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dd5f1ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558d84e046b0 con 0x558d84d337e0 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dce8cc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d657b0 0x558d84d7dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dce8cc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d657b0 0x558d84d7dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44420/0 (socket says 172.21.15.46:44420) 2024-05-09T03:16:09.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dce8cc640 1 -- 172.21.15.46:0/1587945985 learned_addr learned my addr 172.21.15.46:0/1587945985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dcf0cd640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d337e0 0x558d84e012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.127+0000 7f4dcf8ce640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d7c010 0x558d84d7e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dce8cc640 1 -- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d337e0 msgr2=0x558d84e012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dce8cc640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d84d337e0 0x558d84e012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dce8cc640 1 -- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d7c010 msgr2=0x558d84d7e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dce8cc640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d84d7c010 0x558d84d7e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dce8cc640 1 -- 172.21.15.46:0/1587945985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d84d37de0 con 0x558d84d657b0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dce8cc640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d657b0 0x558d84d7dff0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4000c00 tx=0x7f4dc4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dcc8c8640 1 -- 172.21.15.46:0/1587945985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4063030 con 0x558d84d657b0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dcc8c8640 1 -- 172.21.15.46:0/1587945985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc40548b0 con 0x558d84d657b0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558d84d380c0 con 0x558d84d657b0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dcc8c8640 1 -- 172.21.15.46:0/1587945985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4059040 con 0x558d84d657b0 2024-05-09T03:16:09.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.128+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d84e02b70 con 0x558d84d657b0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.130+0000 7f4dcc8c8640 1 -- 172.21.15.46:0/1587945985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4dc4066030 con 0x558d84d657b0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.130+0000 7f4dcc8c8640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4dbc074cb0 0x7f4dbc0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.130+0000 7f4dcc8c8640 1 -- 172.21.15.46:0/1587945985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f4dc40d99a0 con 0x558d84d657b0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.130+0000 7f4dcf0cd640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4dbc074cb0 0x7f4dbc0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.130+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d84e06af0 0x558d84e03880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.130+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x558d84e03dc0 con 0x558d84e06af0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dcf8ce640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d84e06af0 0x558d84e03880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dcf0cd640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4dbc074cb0 0x7f4dbc0771c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005400 tx=0x7f4dc000c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dcf8ce640 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d84e06af0 0x558d84e03880 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dcf8ce640 1 -- 172.21.15.46:0/1587945985 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x7f4dc8003030 con 0x558d84e06af0 2024-05-09T03:16:09.134 INFO:teuthology.orchestra.run.smithi046.stdout:header (274 bytes) : 2024-05-09T03:16:09.134 INFO:teuthology.orchestra.run.smithi046.stdout:00000000 04 03 0c 01 00 00 09 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.134 INFO:teuthology.orchestra.run.smithi046.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-05-09T03:16:09.134 INFO:teuthology.orchestra.run.smithi046.stdout:00000020 00 00 00 00 f0 3f 3c 66 6a 98 d9 28 01 00 00 00 |.....?> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d84e06af0 msgr2=0x558d84e03880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d84e06af0 0x558d84e03880 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4dbc074cb0 msgr2=0x7f4dbc0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4dbc074cb0 0x7f4dbc0771c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005400 tx=0x7f4dc000c040 comp rx=0 tx=0).stop 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d657b0 msgr2=0x558d84d7dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.131+0000 7f4dd5f1ef40 1 --2- 172.21.15.46:0/1587945985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d84d657b0 0x558d84d7dff0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4000c00 tx=0x7f4dc4002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.132+0000 7f4dcf8ce640 1 -- 172.21.15.46:0/1587945985 reap_dead start 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.132+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 shutdown_connections 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.132+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 >> 172.21.15.46:0/1587945985 conn(0x558d84c58ac0 msgr2=0x558d84d62f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.132+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 shutdown_connections 2024-05-09T03:16:09.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.132+0000 7f4dd5f1ef40 1 -- 172.21.15.46:0/1587945985 wait complete. 2024-05-09T03:16:09.138 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 600.00000000 2024-05-09T03:16:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.192+0000 7fdea6950f40 1 Processor -- start 2024-05-09T03:16:09.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.192+0000 7fdea6950f40 1 -- start start 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea6950f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d552ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea6950f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d54e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea6950f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d54ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea6950f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a96d56e5f0 con 0x55a96d5f9150 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea6950f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a96d4cbf50 con 0x55a96d56d5f0 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea6950f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55a96d56e8d0 con 0x55a96d5653a0 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea0c19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d54ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea0c19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d54ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44422/0 (socket says 172.21.15.46:44422) 2024-05-09T03:16:09.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fdea0c19640 1 -- 172.21.15.46:0/1768247314 learned_addr learned my addr 172.21.15.46:0/1768247314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.193+0000 7fde9fc17640 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d54e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0418640 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d552ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0c19640 1 -- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 msgr2=0x55a96d552ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0c19640 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d552ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0c19640 1 -- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 msgr2=0x55a96d54e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0c19640 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d54e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0c19640 1 -- 172.21.15.46:0/1768247314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a96d568d30 con 0x55a96d5f9150 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fde9fc17640 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d54e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fdea0c19640 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d54ef30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fde9800a670 tx=0x7fde9800ab30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f936ca149342b931 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fde9ec15640 1 -- 172.21.15.46:0/1768247314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde98016020 con 0x55a96d5f9150 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fde9ec15640 1 -- 172.21.15.46:0/1768247314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9800d7b0 con 0x55a96d5f9150 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.194+0000 7fde9ec15640 1 -- 172.21.15.46:0/1768247314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9800db60 con 0x55a96d5f9150 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 -- 172.21.15.46:0/1768247314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 msgr2=0x55a96d54ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d54ef30 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fde9800a670 tx=0x7fde9800ab30 comp rx=0 tx=0).stop 2024-05-09T03:16:09.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 -- 172.21.15.46:0/1768247314 shutdown_connections 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d54ef30 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d54e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 --2- 172.21.15.46:0/1768247314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d552ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 -- 172.21.15.46:0/1768247314 >> 172.21.15.46:0/1768247314 conn(0x55a96d447ac0 msgr2=0x55a96d550310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 -- 172.21.15.46:0/1768247314 shutdown_connections 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.195+0000 7fdea6950f40 1 -- 172.21.15.46:0/1768247314 wait complete. 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.196+0000 7fdea6950f40 1 Processor -- start 2024-05-09T03:16:09.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.196+0000 7fdea6950f40 1 -- start start 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea6950f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d690470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea6950f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d5f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea6950f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d5f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea6950f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a96d5fa580 con 0x55a96d5f9150 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea6950f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a96d569550 con 0x55a96d56d5f0 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea6950f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55a96d5f9d80 con 0x55a96d5653a0 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d5f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d5f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44430/0 (socket says 172.21.15.46:44430) 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0418640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d690470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 -- 172.21.15.46:0/174788547 learned_addr learned my addr 172.21.15.46:0/174788547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fde9fc17640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d5f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 -- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 msgr2=0x55a96d690470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a96d5653a0 0x55a96d690470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 -- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 msgr2=0x55a96d5f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a96d56d5f0 0x55a96d5f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.197+0000 7fdea0c19640 1 -- 172.21.15.46:0/174788547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a96d5f6f70 con 0x55a96d5f9150 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.198+0000 7fdea0c19640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d5f5d40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fde9800ab70 tx=0x7fde980033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.198+0000 7fde9dc13640 1 -- 172.21.15.46:0/174788547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9800a330 con 0x55a96d5f9150 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.198+0000 7fde9dc13640 1 -- 172.21.15.46:0/174788547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9801f070 con 0x55a96d5f9150 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.198+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a96d5f4f80 con 0x55a96d5f9150 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.198+0000 7fde9dc13640 1 -- 172.21.15.46:0/174788547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde980246b0 con 0x55a96d5f9150 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.198+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a96d5f5540 con 0x55a96d5f9150 2024-05-09T03:16:09.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.199+0000 7fde9dc13640 1 -- 172.21.15.46:0/174788547 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fde98019020 con 0x55a96d5f9150 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.200+0000 7fde9dc13640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde8c074e50 0x7fde8c077360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.200+0000 7fde9dc13640 1 -- 172.21.15.46:0/174788547 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fde98097b20 con 0x55a96d5f9150 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.200+0000 7fdea0418640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde8c074e50 0x7fde8c077360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.200+0000 7fdea6950f40 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55a96d6909e0 0x55a96d54fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.200+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55a96d550420 con 0x55a96d6909e0 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.200+0000 7fde9fc17640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55a96d6909e0 0x55a96d54fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea0418640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde8c074e50 0x7fde8c077360 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fde90000f60 tx=0x7fde900017f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fde9fc17640 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55a96d6909e0 0x55a96d54fee0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fde9fc17640 1 -- 172.21.15.46:0/174788547 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 600.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7fde9405c040 con 0x55a96d6909e0 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stdout:header (274 bytes) : 2024-05-09T03:16:09.203 INFO:teuthology.orchestra.run.smithi046.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:* 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:* 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:* 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:* 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000110 00 00 |..| 2024-05-09T03:16:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:00000112 2024-05-09T03:16:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55a96d6909e0 msgr2=0x55a96d54fee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55a96d6909e0 0x55a96d54fee0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde8c074e50 msgr2=0x7fde8c077360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde8c074e50 0x7fde8c077360 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fde90000f60 tx=0x7fde900017f0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 msgr2=0x55a96d5f5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 --2- 172.21.15.46:0/174788547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a96d5f9150 0x55a96d5f5d40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fde9800ab70 tx=0x7fde980033a0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea0c19640 1 -- 172.21.15.46:0/174788547 reap_dead start 2024-05-09T03:16:09.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 shutdown_connections 2024-05-09T03:16:09.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.201+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 >> 172.21.15.46:0/174788547 conn(0x55a96d447ac0 msgr2=0x55a96d5f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.202+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 shutdown_connections 2024-05-09T03:16:09.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.202+0000 7fdea6950f40 1 -- 172.21.15.46:0/174788547 wait complete. 2024-05-09T03:16:09.207 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000.inode 2024-05-09T03:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.261+0000 7f34b760ef40 1 Processor -- start 2024-05-09T03:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.261+0000 7f34b760ef40 1 -- start start 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.262+0000 7f34b760ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d897820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.262+0000 7f34b760ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8265f0 0x55c72d80bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.262+0000 7f34b760ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 0x55c72d80c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.262+0000 7f34b760ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c72d784f50 con 0x55c72d8b2150 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.262+0000 7f34b760ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c72d788f90 con 0x55c72d8265f0 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.262+0000 7f34b760ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c72d8278d0 con 0x55c72d81e3a0 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 0x55c72d80c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 0x55c72d80c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44446/0 (socket says 172.21.15.46:44446) 2024-05-09T03:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 -- 172.21.15.46:0/318067171 learned_addr learned my addr 172.21.15.46:0/318067171 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34afc85640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8265f0 0x55c72d80bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 -- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 msgr2=0x55c72d897820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0486640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d897820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d897820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 -- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8265f0 msgr2=0x55c72d80bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8265f0 0x55c72d80bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 -- 172.21.15.46:0/318067171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72d8288a0 con 0x55c72d8b2150 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34afc85640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8265f0 0x55c72d80bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0486640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d897820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34b0c87640 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 0x55c72d80c4d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f34a8001570 tx=0x7f34a8001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b9799ab0d8d656 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34aec83640 1 -- 172.21.15.46:0/318067171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003650 con 0x55c72d8b2150 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34aec83640 1 -- 172.21.15.46:0/318067171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a8003810 con 0x55c72d8b2150 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.263+0000 7f34aec83640 1 -- 172.21.15.46:0/318067171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003c00 con 0x55c72d8b2150 2024-05-09T03:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 -- 172.21.15.46:0/318067171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 msgr2=0x55c72d80c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 0x55c72d80c4d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f34a8001570 tx=0x7f34a8001a30 comp rx=0 tx=0).stop 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 -- 172.21.15.46:0/318067171 shutdown_connections 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8b2150 0x55c72d80c4d0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8265f0 0x55c72d80bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 --2- 172.21.15.46:0/318067171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d897820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 -- 172.21.15.46:0/318067171 >> 172.21.15.46:0/318067171 conn(0x55c72d700ac0 msgr2=0x55c72d80b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 -- 172.21.15.46:0/318067171 shutdown_connections 2024-05-09T03:16:09.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.264+0000 7f34b760ef40 1 -- 172.21.15.46:0/318067171 wait complete. 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.265+0000 7f34b760ef40 1 Processor -- start 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.265+0000 7f34b760ef40 1 -- start start 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b760ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d89aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b760ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8265f0 0x55c72d94b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b760ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8b2150 0x55c72d94b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b760ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c72d821d60 con 0x55c72d8265f0 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b760ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c72d821a80 con 0x55c72d8b2150 2024-05-09T03:16:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b760ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c72d94bf80 con 0x55c72d81e3a0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34afc85640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8265f0 0x55c72d94b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34afc85640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8265f0 0x55c72d94b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44458/0 (socket says 172.21.15.46:44458) 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34afc85640 1 -- 172.21.15.46:0/3579993782 learned_addr learned my addr 172.21.15.46:0/3579993782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b0c87640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8b2150 0x55c72d94b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.266+0000 7f34b0486640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d89aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34afc85640 1 -- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 msgr2=0x55c72d89aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34afc85640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c72d81e3a0 0x55c72d89aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34afc85640 1 -- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8b2150 msgr2=0x55c72d94b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34afc85640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c72d8b2150 0x55c72d94b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34afc85640 1 -- 172.21.15.46:0/3579993782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c72d89a330 con 0x55c72d8265f0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34afc85640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8265f0 0x55c72d94b1f0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3498025c40 tx=0x7f3498002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34adc81640 1 -- 172.21.15.46:0/3579993782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f349806b070 con 0x55c72d8265f0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34adc81640 1 -- 172.21.15.46:0/3579993782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3498053860 con 0x55c72d8265f0 2024-05-09T03:16:09.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c72d7df930 con 0x55c72d8265f0 2024-05-09T03:16:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34adc81640 1 -- 172.21.15.46:0/3579993782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3498053c30 con 0x55c72d8265f0 2024-05-09T03:16:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.267+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c72d896ac0 con 0x55c72d8265f0 2024-05-09T03:16:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.269+0000 7f34adc81640 1 -- 172.21.15.46:0/3579993782 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f349806e070 con 0x55c72d8265f0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.269+0000 7f34adc81640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3490074cb0 0x7f34900771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.269+0000 7f34adc81640 1 -- 172.21.15.46:0/3579993782 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f34980d9b00 con 0x55c72d8265f0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.269+0000 7f34b0486640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3490074cb0 0x7f34900771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.270+0000 7f34b760ef40 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c72d939cd0 0x55c72d898870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.270+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55c72d80caa0 con 0x55c72d939cd0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.270+0000 7f34b0c87640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c72d939cd0 0x55c72d898870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.270+0000 7f34b0486640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3490074cb0 0x7f34900771c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f34a0000c00 tx=0x7f34a0001820 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.270+0000 7f34b0c87640 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c72d939cd0 0x55c72d898870 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.270+0000 7f34b0c87640 1 -- 172.21.15.46:0/3579993782 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-header] v0'0 uv17 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f34a800f030 con 0x55c72d939cd0 2024-05-09T03:16:09.273 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.273 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c72d939cd0 msgr2=0x55c72d898870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c72d939cd0 0x55c72d898870 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3490074cb0 msgr2=0x7f34900771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3490074cb0 0x7f34900771c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f34a0000c00 tx=0x7f34a0001820 comp rx=0 tx=0).stop 2024-05-09T03:16:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8265f0 msgr2=0x55c72d94b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 --2- 172.21.15.46:0/3579993782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c72d8265f0 0x55c72d94b1f0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3498025c40 tx=0x7f3498002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b0c87640 1 -- 172.21.15.46:0/3579993782 reap_dead start 2024-05-09T03:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 shutdown_connections 2024-05-09T03:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 >> 172.21.15.46:0/3579993782 conn(0x55c72d700ac0 msgr2=0x55c72d8207e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 shutdown_connections 2024-05-09T03:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.271+0000 7f34b760ef40 1 -- 172.21.15.46:0/3579993782 wait complete. 2024-05-09T03:16:09.275 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000000 2024-05-09T03:16:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.329+0000 7f4499c0bf40 1 Processor -- start 2024-05-09T03:16:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.330+0000 7f4499c0bf40 1 -- start start 2024-05-09T03:16:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4499c0bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340457e0 0x55be34090460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4499c0bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 0x55be3411fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4499c0bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be34121620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4499c0bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55be3408c4d0 con 0x55be340777b0 2024-05-09T03:16:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4499c0bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55be33feef50 con 0x55be3411f810 2024-05-09T03:16:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4499c0bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55be340918a0 con 0x55be340457e0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 0x55be3411fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 0x55be3411fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44470/0 (socket says 172.21.15.46:44470) 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 -- 172.21.15.46:0/1247771250 learned_addr learned my addr 172.21.15.46:0/1247771250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4493248640 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be34121620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492a47640 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340457e0 0x55be34090460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 -- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340457e0 msgr2=0x55be34090460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340457e0 0x55be34090460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 -- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 msgr2=0x55be34121620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be34121620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 -- 172.21.15.46:0/1247771250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be3408bae0 con 0x55be340777b0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492a47640 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340457e0 0x55be34090460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.331+0000 7f4492246640 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 0x55be3411fc20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f448800a7a0 tx=0x7f448800a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246ea51926782ce5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4491244640 1 -- 172.21.15.46:0/1247771250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488018070 con 0x55be340777b0 2024-05-09T03:16:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4491244640 1 -- 172.21.15.46:0/1247771250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4488007600 con 0x55be340777b0 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4491244640 1 -- 172.21.15.46:0/1247771250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44880079b0 con 0x55be340777b0 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1247771250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 msgr2=0x55be3411fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 0x55be3411fc20 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f448800a7a0 tx=0x7f448800a890 comp rx=0 tx=0).stop 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1247771250 shutdown_connections 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be34121620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340777b0 0x55be3411fc20 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1247771250 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340457e0 0x55be34090460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1247771250 >> 172.21.15.46:0/1247771250 conn(0x55be33f6aac0 msgr2=0x55be3408af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1247771250 shutdown_connections 2024-05-09T03:16:09.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.332+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1247771250 wait complete. 2024-05-09T03:16:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.333+0000 7f4499c0bf40 1 Processor -- start 2024-05-09T03:16:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.333+0000 7f4499c0bf40 1 -- start start 2024-05-09T03:16:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.334+0000 7f4499c0bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340457e0 0x55be3408fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.334+0000 7f4499c0bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340777b0 0x55be340903e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.334+0000 7f4499c0bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be3406e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.334+0000 7f4499c0bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55be3408b650 con 0x55be340457e0 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.334+0000 7f4499c0bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55be340700b0 con 0x55be3411f810 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.334+0000 7f4499c0bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55be34088630 con 0x55be340777b0 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340457e0 0x55be3408fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340457e0 0x55be3408fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44478/0 (socket says 172.21.15.46:44478) 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 -- 172.21.15.46:0/1821747795 learned_addr learned my addr 172.21.15.46:0/1821747795 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492246640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340777b0 0x55be340903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4493248640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be3406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 -- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340777b0 msgr2=0x55be340903e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55be340777b0 0x55be340903e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 -- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 msgr2=0x55be3406e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be3411f810 0x55be3406e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 -- 172.21.15.46:0/1821747795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be34076000 con 0x55be340457e0 2024-05-09T03:16:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4492a47640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340457e0 0x55be3408fea0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f448400c430 tx=0x7f448400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f447bfff640 1 -- 172.21.15.46:0/1821747795 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484011610 con 0x55be340457e0 2024-05-09T03:16:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f447bfff640 1 -- 172.21.15.46:0/1821747795 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4484016070 con 0x55be340457e0 2024-05-09T03:16:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55be34118ac0 con 0x55be340457e0 2024-05-09T03:16:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f447bfff640 1 -- 172.21.15.46:0/1821747795 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44840119f0 con 0x55be340457e0 2024-05-09T03:16:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.335+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be34071640 con 0x55be340457e0 2024-05-09T03:16:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.337+0000 7f447bfff640 1 -- 172.21.15.46:0/1821747795 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4484011b90 con 0x55be340457e0 2024-05-09T03:16:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f447bfff640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4480074cb0 0x7f44800771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f447bfff640 1 -- 172.21.15.46:0/1821747795 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f4484096230 con 0x55be340457e0 2024-05-09T03:16:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f4492246640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4480074cb0 0x7f44800771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55be34118630 0x55be3406ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55be34076ed0 con 0x55be34118630 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f4493248640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55be34118630 0x55be3406ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.338+0000 7f4492246640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4480074cb0 0x7f44800771c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55be340767d0 tx=0x7f448800c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4493248640 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55be34118630 0x55be3406ed60 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4493248640 1 -- 172.21.15.46:0/1821747795 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000000 [omap-get-header] v0'0 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f448c051070 con 0x55be34118630 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.341 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55be34118630 msgr2=0x55be3406ed60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55be34118630 0x55be3406ed60 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4480074cb0 msgr2=0x7f44800771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4480074cb0 0x7f44800771c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55be340767d0 tx=0x7f448800c000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340457e0 msgr2=0x55be3408fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 --2- 172.21.15.46:0/1821747795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55be340457e0 0x55be3408fea0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f448400c430 tx=0x7f448400c8f0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4493248640 1 -- 172.21.15.46:0/1821747795 reap_dead start 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 shutdown_connections 2024-05-09T03:16:09.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 >> 172.21.15.46:0/1821747795 conn(0x55be33f6aac0 msgr2=0x55be3408af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 shutdown_connections 2024-05-09T03:16:09.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.339+0000 7f4499c0bf40 1 -- 172.21.15.46:0/1821747795 wait complete. 2024-05-09T03:16:09.344 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000001 2024-05-09T03:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.398+0000 7f96334ebf40 1 Processor -- start 2024-05-09T03:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.398+0000 7f96334ebf40 1 -- start start 2024-05-09T03:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.399+0000 7f96334ebf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 0x55b3d2b4b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.399+0000 7f96334ebf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2aa3db0 0x55b3d2b34ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.399+0000 7f96334ebf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2ac02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.399+0000 7f96334ebf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3d2ab6cb0 con 0x55b3d2a747e0 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.399+0000 7f96334ebf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3d2a1df50 con 0x55b3d2b4c780 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.399+0000 7f96334ebf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b3d2a74bf0 con 0x55b3d2aa3db0 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 0x55b3d2b4b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 0x55b3d2b4b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44500/0 (socket says 172.21.15.46:44500) 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 -- 172.21.15.46:0/953248271 learned_addr learned my addr 172.21.15.46:0/953248271 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962d7b4640 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2ac02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962c7b2640 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2aa3db0 0x55b3d2b34ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 -- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2aa3db0 msgr2=0x55b3d2b34ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2aa3db0 0x55b3d2b34ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 -- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 msgr2=0x55b3d2ac02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2ac02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 -- 172.21.15.46:0/953248271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d2b4af80 con 0x55b3d2a747e0 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962cfb3640 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 0x55b3d2b4b2c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f9624025b40 tx=0x7f962404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73221fa43e101165 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962b7b0640 1 -- 172.21.15.46:0/953248271 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9624059040 con 0x55b3d2a747e0 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962b7b0640 1 -- 172.21.15.46:0/953248271 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9624064070 con 0x55b3d2a747e0 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.400+0000 7f962b7b0640 1 -- 172.21.15.46:0/953248271 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9624054bf0 con 0x55b3d2a747e0 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 -- 172.21.15.46:0/953248271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 msgr2=0x55b3d2b4b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 0x55b3d2b4b2c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f9624025b40 tx=0x7f962404e000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 -- 172.21.15.46:0/953248271 shutdown_connections 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2ac02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2aa3db0 0x55b3d2b34ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 --2- 172.21.15.46:0/953248271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2a747e0 0x55b3d2b4b2c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 -- 172.21.15.46:0/953248271 >> 172.21.15.46:0/953248271 conn(0x55b3d2999ac0 msgr2=0x55b3d2abcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 -- 172.21.15.46:0/953248271 shutdown_connections 2024-05-09T03:16:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.401+0000 7f96334ebf40 1 -- 172.21.15.46:0/953248271 wait complete. 2024-05-09T03:16:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.402+0000 7f96334ebf40 1 Processor -- start 2024-05-09T03:16:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.402+0000 7f96334ebf40 1 -- start start 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f96334ebf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2a747e0 0x55b3d2b4bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f96334ebf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2aa3db0 0x55b3d2b4df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f96334ebf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2b4e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f96334ebf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3d2b351e0 con 0x55b3d2aa3db0 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f96334ebf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3d2abb2d0 con 0x55b3d2b4c780 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f96334ebf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b3d2abaff0 con 0x55b3d2a747e0 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f962c7b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2aa3db0 0x55b3d2b4df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f962c7b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2aa3db0 0x55b3d2b4df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44502/0 (socket says 172.21.15.46:44502) 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.403+0000 7f962c7b2640 1 -- 172.21.15.46:0/874415445 learned_addr learned my addr 172.21.15.46:0/874415445 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962d7b4640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2b4e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962cfb3640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2a747e0 0x55b3d2b4bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962c7b2640 1 -- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2a747e0 msgr2=0x55b3d2b4bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962c7b2640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3d2a747e0 0x55b3d2b4bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962c7b2640 1 -- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 msgr2=0x55b3d2b4e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962c7b2640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3d2b4c780 0x55b3d2b4e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962c7b2640 1 -- 172.21.15.46:0/874415445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3d2b30b10 con 0x55b3d2aa3db0 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962c7b2640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2aa3db0 0x55b3d2b4df80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f96140086b0 tx=0x7f96140087a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962a7ae640 1 -- 172.21.15.46:0/874415445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96140147e0 con 0x55b3d2aa3db0 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962a7ae640 1 -- 172.21.15.46:0/874415445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961401a070 con 0x55b3d2aa3db0 2024-05-09T03:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3d2b44930 con 0x55b3d2aa3db0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f962a7ae640 1 -- 172.21.15.46:0/874415445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614014bc0 con 0x55b3d2aa3db0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.404+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3d2b48800 con 0x55b3d2aa3db0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.406+0000 7f962a7ae640 1 -- 172.21.15.46:0/874415445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9614015470 con 0x55b3d2aa3db0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.406+0000 7f962a7ae640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f960c074cb0 0x7f960c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.406+0000 7f962a7ae640 1 -- 172.21.15.46:0/874415445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f9614096c30 con 0x55b3d2aa3db0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.407+0000 7f96334ebf40 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b3d2b4c2a0 0x55b3d2be7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.407+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55b3d2b31080 con 0x55b3d2b4c2a0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.407+0000 7f962cfb3640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f960c074cb0 0x7f960c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.407+0000 7f962d7b4640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b3d2b4c2a0 0x55b3d2be7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.407+0000 7f962cfb3640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f960c074cb0 0x7f960c0771c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9624063000 tx=0x7f9624051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.407+0000 7f962d7b4640 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b3d2b4c2a0 0x55b3d2be7db0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f962d7b4640 1 -- 172.21.15.46:0/874415445 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000001 [omap-get-header] v0'0 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f961c003450 con 0x55b3d2b4c2a0 2024-05-09T03:16:09.411 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b3d2b4c2a0 msgr2=0x55b3d2be7db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b3d2b4c2a0 0x55b3d2be7db0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f960c074cb0 msgr2=0x7f960c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f960c074cb0 0x7f960c0771c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9624063000 tx=0x7f9624051000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2aa3db0 msgr2=0x55b3d2b4df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 --2- 172.21.15.46:0/874415445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3d2aa3db0 0x55b3d2b4df80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f96140086b0 tx=0x7f96140087a0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f962d7b4640 1 -- 172.21.15.46:0/874415445 reap_dead start 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 shutdown_connections 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 >> 172.21.15.46:0/874415445 conn(0x55b3d2999ac0 msgr2=0x55b3d2aba8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 shutdown_connections 2024-05-09T03:16:09.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.408+0000 7f96334ebf40 1 -- 172.21.15.46:0/874415445 wait complete. 2024-05-09T03:16:09.414 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_openfiles.0 2024-05-09T03:16:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.468+0000 7fd28f0cdf40 1 Processor -- start 2024-05-09T03:16:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.468+0000 7fd28f0cdf40 1 -- start start 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd28f0cdf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1783d2a0 0x558e17913cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd28f0cdf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 0x558e1786aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd28f0cdf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e17915610 0x558e17887a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd28f0cdf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558e1786ff20 con 0x558e1790ec90 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd28f0cdf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e177e6f50 con 0x558e17915610 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd28f0cdf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558e17888850 con 0x558e1783d2a0 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 0x558e1786aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 0x558e1786aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44514/0 (socket says 172.21.15.46:44514) 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 -- 172.21.15.46:0/4098993280 learned_addr learned my addr 172.21.15.46:0/4098993280 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288b95640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1783d2a0 0x558e17913cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd289396640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e17915610 0x558e17887a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 -- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1783d2a0 msgr2=0x558e17913cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1783d2a0 0x558e17913cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 -- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e17915610 msgr2=0x558e17887a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e17915610 0x558e17887a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288394640 1 -- 172.21.15.46:0/4098993280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e17887720 con 0x558e1790ec90 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd288b95640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1783d2a0 0x558e17913cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.469+0000 7fd289396640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e17915610 0x558e17887a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd288394640 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 0x558e1786aba0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001dc0 tx=0x7fd27c00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867c247c669096fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd287392640 1 -- 172.21.15.46:0/4098993280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c012020 con 0x558e1790ec90 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd287392640 1 -- 172.21.15.46:0/4098993280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27c00e670 con 0x558e1790ec90 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd287392640 1 -- 172.21.15.46:0/4098993280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c00eae0 con 0x558e1790ec90 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/4098993280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 msgr2=0x558e1786aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 0x558e1786aba0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001dc0 tx=0x7fd27c00b6c0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/4098993280 shutdown_connections 2024-05-09T03:16:09.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e17915610 0x558e17887a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.470+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e1790ec90 0x558e1786aba0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.471+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/4098993280 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1783d2a0 0x558e17913cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.471+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/4098993280 >> 172.21.15.46:0/4098993280 conn(0x558e17762ac0 msgr2=0x558e179175f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.471+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/4098993280 shutdown_connections 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.471+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/4098993280 wait complete. 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.471+0000 7fd28f0cdf40 1 Processor -- start 2024-05-09T03:16:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.471+0000 7fd28f0cdf40 1 -- start start 2024-05-09T03:16:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd28f0cdf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e1783d2a0 0x558e17884ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd28f0cdf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1790ec90 0x558e178f9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd28f0cdf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e17915610 0x558e17908e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd28f0cdf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558e17887440 con 0x558e17915610 2024-05-09T03:16:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd28f0cdf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e17913550 con 0x558e1783d2a0 2024-05-09T03:16:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd28f0cdf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558e179137f0 con 0x558e1790ec90 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e17915610 0x558e17908e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e17915610 0x558e17908e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44518/0 (socket says 172.21.15.46:44518) 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 -- 172.21.15.46:0/1209976861 learned_addr learned my addr 172.21.15.46:0/1209976861 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd288b95640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e1783d2a0 0x558e17884ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd288394640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1790ec90 0x558e178f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 -- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1790ec90 msgr2=0x558e178f9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e1790ec90 0x558e178f9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 -- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e1783d2a0 msgr2=0x558e17884ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e1783d2a0 0x558e17884ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 -- 172.21.15.46:0/1209976861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e17881c20 con 0x558e17915610 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.473+0000 7fd289396640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e17915610 0x558e17908e30 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd280025af0 tx=0x7fd280002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.474+0000 7fd286390640 1 -- 172.21.15.46:0/1209976861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280063030 con 0x558e17915610 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.474+0000 7fd286390640 1 -- 172.21.15.46:0/1209976861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2800548b0 con 0x558e17915610 2024-05-09T03:16:09.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.474+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558e178813c0 con 0x558e17915610 2024-05-09T03:16:09.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.474+0000 7fd286390640 1 -- 172.21.15.46:0/1209976861 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280059040 con 0x558e17915610 2024-05-09T03:16:09.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.474+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e17881940 con 0x558e17915610 2024-05-09T03:16:09.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.475+0000 7fd286390640 1 -- 172.21.15.46:0/1209976861 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd280066030 con 0x558e17915610 2024-05-09T03:16:09.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd286390640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd274074ce0 0x7fd2740771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd286390640 1 -- 172.21.15.46:0/1209976861 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fd2800d98f0 con 0x558e17915610 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd288b95640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd274074ce0 0x7fd2740771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558e17886980 0x558e17885a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x558e17885f50 con 0x558e17886980 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd288394640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558e17886980 0x558e17885a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.476+0000 7fd288b95640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd274074ce0 0x7fd2740771f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x558e17889c80 tx=0x7fd278007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd288394640 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558e17886980 0x558e17885a10 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd288394640 1 -- 172.21.15.46:0/1209976861 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv4 ondisk = 0) ==== 160+0+42 (crc 0 0 0) 0x7fd27c00e820 con 0x558e17886980 2024-05-09T03:16:09.479 INFO:teuthology.orchestra.run.smithi046.stdout:header (42 bytes) : 2024-05-09T03:16:09.480 INFO:teuthology.orchestra.run.smithi046.stdout:00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| 2024-05-09T03:16:09.480 INFO:teuthology.orchestra.run.smithi046.stdout:00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 04 00 00 |me v011.........| 2024-05-09T03:16:09.480 INFO:teuthology.orchestra.run.smithi046.stdout:00000020 00 00 00 00 00 01 00 00 00 00 |..........| 2024-05-09T03:16:09.480 INFO:teuthology.orchestra.run.smithi046.stdout:0000002a 2024-05-09T03:16:09.480 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558e17886980 msgr2=0x558e17885a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558e17886980 0x558e17885a10 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd274074ce0 msgr2=0x7fd2740771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd274074ce0 0x7fd2740771f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x558e17889c80 tx=0x7fd278007040 comp rx=0 tx=0).stop 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e17915610 msgr2=0x558e17908e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 --2- 172.21.15.46:0/1209976861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e17915610 0x558e17908e30 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fd280025af0 tx=0x7fd280002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd289396640 1 -- 172.21.15.46:0/1209976861 reap_dead start 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 shutdown_connections 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 >> 172.21.15.46:0/1209976861 conn(0x558e17762ac0 msgr2=0x558e17887a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 shutdown_connections 2024-05-09T03:16:09.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.477+0000 7fd28f0cdf40 1 -- 172.21.15.46:0/1209976861 wait complete. 2024-05-09T03:16:09.482 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 500.00000000 2024-05-09T03:16:09.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.537+0000 7f9a70af6f40 1 Processor -- start 2024-05-09T03:16:09.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.537+0000 7f9a70af6f40 1 -- start start 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a70af6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb02010 0x55aa1fb89d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a70af6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 0x55aa1fb8ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a70af6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fafc8a0 0x55aa1fb8f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a70af6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55aa1fb04670 con 0x55aa1fb928c0 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a70af6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aa1fa62f50 con 0x55aa1fb02010 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a70af6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55aa1fb00950 con 0x55aa1fafc8a0 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 0x55aa1fb8ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 0x55aa1fb8ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44530/0 (socket says 172.21.15.46:44530) 2024-05-09T03:16:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 -- 172.21.15.46:0/1361754746 learned_addr learned my addr 172.21.15.46:0/1361754746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a6a5c0640 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb02010 0x55aa1fb89d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a6adc1640 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fafc8a0 0x55aa1fb8f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 -- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fafc8a0 msgr2=0x55aa1fb8f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fafc8a0 0x55aa1fb8f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 -- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb02010 msgr2=0x55aa1fb89d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb02010 0x55aa1fb89d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 -- 172.21.15.46:0/1361754746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa1faff870 con 0x55aa1fb928c0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 0x55aa1fb8ecc0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x55aa1fabed90 tx=0x7f9a5400b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28de43918cd3a9c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a68dbd640 1 -- 172.21.15.46:0/1361754746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a54012020 con 0x55aa1fb928c0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a68dbd640 1 -- 172.21.15.46:0/1361754746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a5400e670 con 0x55aa1fb928c0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.539+0000 7f9a68dbd640 1 -- 172.21.15.46:0/1361754746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400ea60 con 0x55aa1fb928c0 2024-05-09T03:16:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 -- 172.21.15.46:0/1361754746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 msgr2=0x55aa1fb8ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 0x55aa1fb8ecc0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x55aa1fabed90 tx=0x7f9a5400b780 comp rx=0 tx=0).stop 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 -- 172.21.15.46:0/1361754746 shutdown_connections 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fafc8a0 0x55aa1fb8f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fb928c0 0x55aa1fb8ecc0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/1361754746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb02010 0x55aa1fb89d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 -- 172.21.15.46:0/1361754746 >> 172.21.15.46:0/1361754746 conn(0x55aa1f9deac0 msgr2=0x55aa1fb94610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 -- 172.21.15.46:0/1361754746 shutdown_connections 2024-05-09T03:16:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.540+0000 7f9a70af6f40 1 -- 172.21.15.46:0/1361754746 wait complete. 2024-05-09T03:16:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.541+0000 7f9a70af6f40 1 Processor -- start 2024-05-09T03:16:09.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.541+0000 7f9a70af6f40 1 -- start start 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a70af6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fafc8a0 0x55aa1fb90e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a70af6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fb02010 0x55aa1fb91360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a70af6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb928c0 0x55aa1fb94310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a70af6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55aa1faff3e0 con 0x55aa1fafc8a0 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a70af6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aa1fb8c960 con 0x55aa1fb928c0 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a70af6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55aa1fb93e80 con 0x55aa1fb02010 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a6a5c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fafc8a0 0x55aa1fb90e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.542+0000 7f9a6a5c0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fafc8a0 0x55aa1fb90e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44532/0 (socket says 172.21.15.46:44532) 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 -- 172.21.15.46:0/2342335562 learned_addr learned my addr 172.21.15.46:0/2342335562 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6adc1640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb928c0 0x55aa1fb94310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fb02010 0x55aa1fb91360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 -- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fb02010 msgr2=0x55aa1fb91360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fb02010 0x55aa1fb91360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 -- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb928c0 msgr2=0x55aa1fb94310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb928c0 0x55aa1fb94310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 -- 172.21.15.46:0/2342335562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa1fb86240 con 0x55aa1fafc8a0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6adc1640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aa1fb928c0 0x55aa1fb94310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55aa1fb02010 0x55aa1fb91360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a6a5c0640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fafc8a0 0x55aa1fb90e20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c045740 tx=0x7f9a5c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a637fe640 1 -- 172.21.15.46:0/2342335562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c064070 con 0x55aa1fafc8a0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a637fe640 1 -- 172.21.15.46:0/2342335562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a5c054aa0 con 0x55aa1fafc8a0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aa1fb95650 con 0x55aa1fafc8a0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a637fe640 1 -- 172.21.15.46:0/2342335562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c059040 con 0x55aa1fafc8a0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.543+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa1fb8c6a0 con 0x55aa1fafc8a0 2024-05-09T03:16:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.545+0000 7f9a637fe640 1 -- 172.21.15.46:0/2342335562 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9a5c060430 con 0x55aa1fafc8a0 2024-05-09T03:16:09.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.545+0000 7f9a637fe640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a4c074ce0 0x7f9a4c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a637fe640 1 -- 172.21.15.46:0/2342335562 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f9a5c0da910 con 0x55aa1fafc8a0 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a4c074ce0 0x7f9a4c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55aa1fb8c180 0x55aa1fb94850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55aa1fb877a0 con 0x55aa1fb8c180 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a6adc1640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55aa1fb8c180 0x55aa1fb94850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a69dbf640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a4c074ce0 0x7f9a4c0771f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x55aa1fabed90 tx=0x7f9a54009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.546+0000 7f9a6adc1640 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55aa1fb8c180 0x55aa1fb94850 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a6adc1640 1 -- 172.21.15.46:0/2342335562 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9a6400a990 con 0x55aa1fb8c180 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.549 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55aa1fb8c180 msgr2=0x55aa1fb94850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55aa1fb8c180 0x55aa1fb94850 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a4c074ce0 msgr2=0x7f9a4c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9a4c074ce0 0x7f9a4c0771f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x55aa1fabed90 tx=0x7f9a54009000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fafc8a0 msgr2=0x55aa1fb90e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 --2- 172.21.15.46:0/2342335562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aa1fafc8a0 0x55aa1fb90e20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c045740 tx=0x7f9a5c002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a6adc1640 1 -- 172.21.15.46:0/2342335562 reap_dead start 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 shutdown_connections 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 >> 172.21.15.46:0/2342335562 conn(0x55aa1f9deac0 msgr2=0x55aa1fb8ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 shutdown_connections 2024-05-09T03:16:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.547+0000 7f9a70af6f40 1 -- 172.21.15.46:0/2342335562 wait complete. 2024-05-09T03:16:09.552 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds_snaptable 2024-05-09T03:16:09.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.606+0000 7f68bbd0ff40 1 Processor -- start 2024-05-09T03:16:09.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.606+0000 7f68bbd0ff40 1 -- start start 2024-05-09T03:16:09.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68bbd0ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd7f3d2a0 0x55edd80a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68bbd0ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd80a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68bbd0ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd800ec90 0x55edd800d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68bbd0ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55edd7f6ff20 con 0x55edd7f80630 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68bbd0ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55edd7ee6f50 con 0x55edd7f3d2a0 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68bbd0ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55edd7f41cf0 con 0x55edd800ec90 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd80a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd80a34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44538/0 (socket says 172.21.15.46:44538) 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1148245102 learned_addr learned my addr 172.21.15.46:0/1148245102 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b5fda640 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd800ec90 0x55edd800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b57d9640 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd7f3d2a0 0x55edd80a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd800ec90 msgr2=0x55edd800d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd800ec90 0x55edd800d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd7f3d2a0 msgr2=0x55edd80a2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd7f3d2a0 0x55edd80a2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1148245102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd7f86090 con 0x55edd7f80630 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b5fda640 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd800ec90 0x55edd800d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.608+0000 7f68b4fd8640 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd80a34e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f68a0025b20 tx=0x7f68a004e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fb5563175a0878 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68af7fe640 1 -- 172.21.15.46:0/1148245102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0059040 con 0x55edd7f80630 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68af7fe640 1 -- 172.21.15.46:0/1148245102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a0064070 con 0x55edd7f80630 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68af7fe640 1 -- 172.21.15.46:0/1148245102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0054bf0 con 0x55edd7f80630 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1148245102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 msgr2=0x55edd80a34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd80a34e0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f68a0025b20 tx=0x7f68a004e000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1148245102 shutdown_connections 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd800ec90 0x55edd800d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd80a34e0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1148245102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd7f3d2a0 0x55edd80a2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1148245102 >> 172.21.15.46:0/1148245102 conn(0x55edd7e62ac0 msgr2=0x55edd7f6cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1148245102 shutdown_connections 2024-05-09T03:16:09.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.609+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1148245102 wait complete. 2024-05-09T03:16:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.610+0000 7f68bbd0ff40 1 Processor -- start 2024-05-09T03:16:09.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.610+0000 7f68bbd0ff40 1 -- start start 2024-05-09T03:16:09.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.611+0000 7f68bbd0ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd7f3d2a0 0x55edd800d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.611+0000 7f68bbd0ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd7ff7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.611+0000 7f68bbd0ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd800ec90 0x55edd7ff84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.611+0000 7f68bbd0ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55edd800e120 con 0x55edd7f80630 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.611+0000 7f68bbd0ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55edd7f3edb0 con 0x55edd800ec90 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.611+0000 7f68bbd0ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55edd8008f10 con 0x55edd7f3d2a0 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd7ff7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd7ff7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44550/0 (socket says 172.21.15.46:44550) 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1025535509 learned_addr learned my addr 172.21.15.46:0/1025535509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b57d9640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd7f3d2a0 0x55edd800d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b5fda640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd800ec90 0x55edd7ff84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd7f3d2a0 msgr2=0x55edd800d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd7f3d2a0 0x55edd800d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd800ec90 msgr2=0x55edd7ff84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd800ec90 0x55edd7ff84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 -- 172.21.15.46:0/1025535509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd8006df0 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b5fda640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55edd800ec90 0x55edd7ff84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b57d9640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55edd7f3d2a0 0x55edd800d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68b4fd8640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd7ff7fa0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f68a0025b20 tx=0x7f68a0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68ae7fc640 1 -- 172.21.15.46:0/1025535509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a006e070 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68ae7fc640 1 -- 172.21.15.46:0/1025535509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a0054ab0 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55edd80070d0 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68ae7fc640 1 -- 172.21.15.46:0/1025535509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0059050 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.612+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd8007690 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.614+0000 7f68ae7fc640 1 -- 172.21.15.46:0/1025535509 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f68a0071070 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.614+0000 7f68ae7fc640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6898074ce0 0x7f68980771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.614+0000 7f68ae7fc640 1 -- 172.21.15.46:0/1025535509 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f68a00db0b0 con 0x55edd7f80630 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.615+0000 7f68b57d9640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6898074ce0 0x7f68980771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.615+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55edd7f83530 0x55edd7f83940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.615+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55edd800d930 con 0x55edd7f83530 2024-05-09T03:16:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.615+0000 7f68b5fda640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55edd7f83530 0x55edd7f83940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.615+0000 7f68b57d9640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6898074ce0 0x7f68980771f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f68b0000c00 tx=0x7f68b0001690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.615+0000 7f68b5fda640 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55edd7f83530 0x55edd7f83940 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68b5fda640 1 -- 172.21.15.46:0/1025535509 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f68a800f900 con 0x55edd7f83530 2024-05-09T03:16:09.618 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.618 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55edd7f83530 msgr2=0x55edd7f83940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55edd7f83530 0x55edd7f83940 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6898074ce0 msgr2=0x7f68980771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6898074ce0 0x7f68980771f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f68b0000c00 tx=0x7f68b0001690 comp rx=0 tx=0).stop 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 msgr2=0x55edd7ff7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 --2- 172.21.15.46:0/1025535509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55edd7f80630 0x55edd7ff7fa0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f68a0025b20 tx=0x7f68a0002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68b5fda640 1 -- 172.21.15.46:0/1025535509 reap_dead start 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 shutdown_connections 2024-05-09T03:16:09.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 >> 172.21.15.46:0/1025535509 conn(0x55edd7e62ac0 msgr2=0x55edd7f6cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 shutdown_connections 2024-05-09T03:16:09.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.616+0000 7f68bbd0ff40 1 -- 172.21.15.46:0/1025535509 wait complete. 2024-05-09T03:16:09.621 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_inotable 2024-05-09T03:16:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.676+0000 7fd292dedf40 1 Processor -- start 2024-05-09T03:16:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.676+0000 7fd292dedf40 1 -- start start 2024-05-09T03:16:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd292dedf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e694f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd292dedf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69517b0 0x5637e6965980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd292dedf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 0x5637e6950880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd292dedf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5637e68c8f50 con 0x5637e69f9810 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd292dedf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5637e68ccf90 con 0x5637e69517b0 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd292dedf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5637e69664d0 con 0x5637e691f7e0 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 0x5637e6950880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 0x5637e6950880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44560/0 (socket says 172.21.15.46:44560) 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 -- 172.21.15.46:0/415599255 learned_addr learned my addr 172.21.15.46:0/415599255 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28c0b4640 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69517b0 0x5637e6965980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28c8b5640 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e694f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 -- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 msgr2=0x5637e694f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e694f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 -- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69517b0 msgr2=0x5637e6965980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69517b0 0x5637e6965980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.677+0000 7fd28d0b6640 1 -- 172.21.15.46:0/415599255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637e69505b0 con 0x5637e69f9810 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd28d0b6640 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 0x5637e6950880 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd2840015d0 tx=0x7fd284001a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a80ab5c0109faa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd28b0b2640 1 -- 172.21.15.46:0/415599255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284016070 con 0x5637e69f9810 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd28b0b2640 1 -- 172.21.15.46:0/415599255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd28400f730 con 0x5637e69f9810 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd28b0b2640 1 -- 172.21.15.46:0/415599255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28400fb20 con 0x5637e69f9810 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 -- 172.21.15.46:0/415599255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 msgr2=0x5637e6950880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 0x5637e6950880 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd2840015d0 tx=0x7fd284001a90 comp rx=0 tx=0).stop 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 -- 172.21.15.46:0/415599255 shutdown_connections 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69f9810 0x5637e6950880 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69517b0 0x5637e6965980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 --2- 172.21.15.46:0/415599255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e694f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.678+0000 7fd292dedf40 1 -- 172.21.15.46:0/415599255 >> 172.21.15.46:0/415599255 conn(0x5637e6844ac0 msgr2=0x5637e6964f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.679+0000 7fd292dedf40 1 -- 172.21.15.46:0/415599255 shutdown_connections 2024-05-09T03:16:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.679+0000 7fd292dedf40 1 -- 172.21.15.46:0/415599255 wait complete. 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.679+0000 7fd292dedf40 1 Processor -- start 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.679+0000 7fd292dedf40 1 -- start start 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd292dedf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e69fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd292dedf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69517b0 0x5637e69472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd292dedf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69f9810 0x5637e6947800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd292dedf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5637e69f9fa0 con 0x5637e69517b0 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd292dedf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5637e69f1f30 con 0x5637e69f9810 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd292dedf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5637e69f28f0 con 0x5637e691f7e0 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69517b0 0x5637e69472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69517b0 0x5637e69472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44574/0 (socket says 172.21.15.46:44574) 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 -- 172.21.15.46:0/2577547677 learned_addr learned my addr 172.21.15.46:0/2577547677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28d0b6640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69f9810 0x5637e6947800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c8b5640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e69fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 -- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 msgr2=0x5637e69fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637e691f7e0 0x5637e69fb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 -- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69f9810 msgr2=0x5637e6947800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637e69f9810 0x5637e6947800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 -- 172.21.15.46:0/2577547677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637e6922830 con 0x5637e69517b0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.681+0000 7fd28c0b4640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69517b0 0x5637e69472c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fd28000a470 tx=0x7fd280002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.682+0000 7fd28a0b0640 1 -- 172.21.15.46:0/2577547677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280066030 con 0x5637e69517b0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.682+0000 7fd28a0b0640 1 -- 172.21.15.46:0/2577547677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd28005b040 con 0x5637e69517b0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.682+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5637e6922b10 con 0x5637e69517b0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.682+0000 7fd28a0b0640 1 -- 172.21.15.46:0/2577547677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28005a070 con 0x5637e69517b0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.682+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637e69230d0 con 0x5637e69517b0 2024-05-09T03:16:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.683+0000 7fd28a0b0640 1 -- 172.21.15.46:0/2577547677 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd280069030 con 0x5637e69517b0 2024-05-09T03:16:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd28a0b0640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd278074ce0 0x7fd2780771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd28a0b0640 1 -- 172.21.15.46:0/2577547677 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fd2800d9e40 con 0x5637e69517b0 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd28c8b5640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd278074ce0 0x7fd2780771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd292dedf40 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5637e6921950 0x5637e6921d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5637e6965630 con 0x5637e6921950 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd28d0b6640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5637e6921950 0x5637e6921d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.684+0000 7fd28c8b5640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd278074ce0 0x7fd2780771f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001ed0 tx=0x7fd27c001a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd28d0b6640 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5637e6921950 0x5637e6921d60 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd28d0b6640 1 -- 172.21.15.46:0/2577547677 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd284019040 con 0x5637e6921950 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.687 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5637e6921950 msgr2=0x5637e6921d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5637e6921950 0x5637e6921d60 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd278074ce0 msgr2=0x7fd2780771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd278074ce0 0x7fd2780771f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001ed0 tx=0x7fd27c001a00 comp rx=0 tx=0).stop 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69517b0 msgr2=0x5637e69472c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 --2- 172.21.15.46:0/2577547677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637e69517b0 0x5637e69472c0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fd28000a470 tx=0x7fd280002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd28d0b6640 1 -- 172.21.15.46:0/2577547677 reap_dead start 2024-05-09T03:16:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 shutdown_connections 2024-05-09T03:16:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 >> 172.21.15.46:0/2577547677 conn(0x5637e6844ac0 msgr2=0x5637e6950790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 shutdown_connections 2024-05-09T03:16:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.685+0000 7fd292dedf40 1 -- 172.21.15.46:0/2577547677 wait complete. 2024-05-09T03:16:09.690 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000 2024-05-09T03:16:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.744+0000 7f58b4014f40 1 Processor -- start 2024-05-09T03:16:09.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.745+0000 7f58b4014f40 1 -- start start 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58b4014f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x564799444870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58b4014f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647994020e0 0x564799568080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58b4014f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564799446060 0x5647995685c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58b4014f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564799400400 con 0x5647993ffff0 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58b4014f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5647993a5f50 con 0x5647994020e0 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58b4014f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564799449000 con 0x564799446060 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x564799444870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x564799444870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44586/0 (socket says 172.21.15.46:44586) 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 -- 172.21.15.46:0/3584339737 learned_addr learned my addr 172.21.15.46:0/3584339737 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ac66c640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647994020e0 0x564799568080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ad66e640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564799446060 0x5647995685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 -- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564799446060 msgr2=0x5647995685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564799446060 0x5647995685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 -- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647994020e0 msgr2=0x564799568080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647994020e0 0x564799568080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 -- 172.21.15.46:0/3584339737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564799446510 con 0x5647993ffff0 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ad66e640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564799446060 0x5647995685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ac66c640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647994020e0 0x564799568080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.746+0000 7f58ace6d640 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x564799444870 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f589c00a8c0 tx=0x7f589c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12abaf61405feb7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58ab66a640 1 -- 172.21.15.46:0/3584339737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c017020 con 0x5647993ffff0 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58ab66a640 1 -- 172.21.15.46:0/3584339737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c00d9c0 con 0x5647993ffff0 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58ab66a640 1 -- 172.21.15.46:0/3584339737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c00ddb0 con 0x5647993ffff0 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 -- 172.21.15.46:0/3584339737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 msgr2=0x564799444870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x564799444870 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f589c00a8c0 tx=0x7f589c005d80 comp rx=0 tx=0).stop 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 -- 172.21.15.46:0/3584339737 shutdown_connections 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564799446060 0x5647995685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647994020e0 0x564799568080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 --2- 172.21.15.46:0/3584339737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x564799444870 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 -- 172.21.15.46:0/3584339737 >> 172.21.15.46:0/3584339737 conn(0x564799321ac0 msgr2=0x56479942ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 -- 172.21.15.46:0/3584339737 shutdown_connections 2024-05-09T03:16:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.747+0000 7f58b4014f40 1 -- 172.21.15.46:0/3584339737 wait complete. 2024-05-09T03:16:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.748+0000 7f58b4014f40 1 Processor -- start 2024-05-09T03:16:09.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.748+0000 7f58b4014f40 1 -- start start 2024-05-09T03:16:09.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.749+0000 7f58b4014f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x56479943db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.749+0000 7f58b4014f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647994020e0 0x5647993fe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.749+0000 7f58b4014f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564799446060 0x564799427e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.749+0000 7f58b4014f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56479943e5c0 con 0x5647993ffff0 2024-05-09T03:16:09.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.749+0000 7f58b4014f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564799443940 con 0x564799446060 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.749+0000 7f58b4014f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5647994d4c70 con 0x5647994020e0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x56479943db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x56479943db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44588/0 (socket says 172.21.15.46:44588) 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 -- 172.21.15.46:0/1367179625 learned_addr learned my addr 172.21.15.46:0/1367179625 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ad66e640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564799446060 0x564799427e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ac66c640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647994020e0 0x5647993fe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 -- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647994020e0 msgr2=0x5647993fe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647994020e0 0x5647993fe8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 -- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564799446060 msgr2=0x564799427e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564799446060 0x564799427e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 -- 172.21.15.46:0/1367179625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56479943e0c0 con 0x5647993ffff0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58ace6d640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x56479943db80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f589c00b010 tx=0x7f589c0033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58aa668640 1 -- 172.21.15.46:0/1367179625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c017040 con 0x5647993ffff0 2024-05-09T03:16:09.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58aa668640 1 -- 172.21.15.46:0/1367179625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c012d50 con 0x5647993ffff0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58aa668640 1 -- 172.21.15.46:0/1367179625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c01c660 con 0x5647993ffff0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564799443b90 con 0x5647993ffff0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.750+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647993fe0a0 con 0x5647993ffff0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.752+0000 7f58aa668640 1 -- 172.21.15.46:0/1367179625 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f589c01a020 con 0x5647993ffff0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.752+0000 7f58aa668640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5898074cb0 0x7f58980771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58aa668640 1 -- 172.21.15.46:0/1367179625 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f589c0981b0 con 0x5647993ffff0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58ac66c640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5898074cb0 0x7f58980771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58b4014f40 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647994d3240 0x5647994d3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5647993fede0 con 0x5647994d3240 2024-05-09T03:16:09.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58ad66e640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647994d3240 0x5647994d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58ac66c640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5898074cb0 0x7f58980771c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f58a0057000 tx=0x7f58a0052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.753+0000 7f58ad66e640 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647994d3240 0x5647994d3650 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58ad66e640 1 -- 172.21.15.46:0/1367179625 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f58a400e480 con 0x5647994d3240 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stdout:header (274 bytes) : 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-05-09T03:16:09.756 INFO:teuthology.orchestra.run.smithi046.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000030 00 00 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000060 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:00000090 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:000000a0 00 00 00 00 00 00 00 00 e9 3f 3c 66 f1 b3 81 03 |.........?> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647994d3240 msgr2=0x5647994d3650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647994d3240 0x5647994d3650 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5898074cb0 msgr2=0x7f58980771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5898074cb0 0x7f58980771c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f58a0057000 tx=0x7f58a0052000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 msgr2=0x56479943db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 --2- 172.21.15.46:0/1367179625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647993ffff0 0x56479943db80 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f589c00b010 tx=0x7f589c0033a0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58ad66e640 1 -- 172.21.15.46:0/1367179625 reap_dead start 2024-05-09T03:16:09.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 shutdown_connections 2024-05-09T03:16:09.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 >> 172.21.15.46:0/1367179625 conn(0x564799321ac0 msgr2=0x56479942db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 shutdown_connections 2024-05-09T03:16:09.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.754+0000 7f58b4014f40 1 -- 172.21.15.46:0/1367179625 wait complete. 2024-05-09T03:16:09.761 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_sessionmap 2024-05-09T03:16:09.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.816+0000 7efe9d1dbf40 1 Processor -- start 2024-05-09T03:16:09.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.816+0000 7efe9d1dbf40 1 -- start start 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe9d1dbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 0x55e31b8e2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe9d1dbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e31b929790 0x55e31b929ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe9d1dbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b92a0e0 0x55e31ba4b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe9d1dbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e31b88bf50 con 0x7efe800022f0 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe9d1dbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e31b88ff90 con 0x55e31b92a0e0 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe9d1dbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e31b924cb0 con 0x55e31b929790 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe96ca5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 0x55e31b8e2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe96ca5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 0x55e31b8e2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44596/0 (socket says 172.21.15.46:44596) 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.817+0000 7efe96ca5640 1 -- 172.21.15.46:0/775966470 learned_addr learned my addr 172.21.15.46:0/775966470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe974a6640 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b92a0e0 0x55e31ba4b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe96ca5640 1 -- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e31b929790 msgr2=0x55e31b929ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe96ca5640 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e31b929790 0x55e31b929ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe96ca5640 1 -- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b92a0e0 msgr2=0x55e31ba4b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe96ca5640 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b92a0e0 0x55e31ba4b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe96ca5640 1 -- 172.21.15.46:0/775966470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e31b9b9630 con 0x7efe800022f0 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe96ca5640 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 0x55e31b8e2250 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7efe88026060 tx=0x7efe88006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32f0a28fdcf438d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe954a2640 1 -- 172.21.15.46:0/775966470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88001af0 con 0x7efe800022f0 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe954a2640 1 -- 172.21.15.46:0/775966470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe88006070 con 0x7efe800022f0 2024-05-09T03:16:09.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.818+0000 7efe954a2640 1 -- 172.21.15.46:0/775966470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8804d8c0 con 0x7efe800022f0 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/775966470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 msgr2=0x55e31b8e2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 0x55e31b8e2250 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7efe88026060 tx=0x7efe88006000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/775966470 shutdown_connections 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b92a0e0 0x55e31ba4b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e31b929790 0x55e31b929ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/775966470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe800022f0 0x55e31b8e2250 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/775966470 >> 172.21.15.46:0/775966470 conn(0x55e31b807ac0 msgr2=0x55e31b90dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/775966470 shutdown_connections 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.819+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/775966470 wait complete. 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.820+0000 7efe9d1dbf40 1 Processor -- start 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.820+0000 7efe9d1dbf40 1 -- start start 2024-05-09T03:16:09.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe9d1dbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b929790 0x55e31b99c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe9d1dbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e31b92a0e0 0x55e31b99fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe9d1dbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe800022f0 0x55e31b9a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe9d1dbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e31b9b8b00 con 0x55e31b92a0e0 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe9d1dbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e31b9b9080 con 0x55e31b929790 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe9d1dbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e31b9101e0 con 0x7efe800022f0 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e31b92a0e0 0x55e31b99fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e31b92a0e0 0x55e31b99fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44600/0 (socket says 172.21.15.46:44600) 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 -- 172.21.15.46:0/92901544 learned_addr learned my addr 172.21.15.46:0/92901544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe974a6640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe800022f0 0x55e31b9a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe96ca5640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b929790 0x55e31b99c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 -- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe800022f0 msgr2=0x55e31b9a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe800022f0 0x55e31b9a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 -- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b929790 msgr2=0x55e31b99c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e31b929790 0x55e31b99c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.821+0000 7efe964a4640 1 -- 172.21.15.46:0/92901544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e31ba502c0 con 0x55e31b92a0e0 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.822+0000 7efe964a4640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e31b92a0e0 0x55e31b99fef0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00ef30 tx=0x7efe8c00c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.822+0000 7efe7ffff640 1 -- 172.21.15.46:0/92901544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c015020 con 0x55e31b92a0e0 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.822+0000 7efe7ffff640 1 -- 172.21.15.46:0/92901544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8c011450 con 0x55e31b92a0e0 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.822+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e31b9a2c10 con 0x55e31b92a0e0 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.822+0000 7efe7ffff640 1 -- 172.21.15.46:0/92901544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c011820 con 0x55e31b92a0e0 2024-05-09T03:16:09.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.822+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e31b99cca0 con 0x55e31b92a0e0 2024-05-09T03:16:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.823+0000 7efe7ffff640 1 -- 172.21.15.46:0/92901544 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efe8c018020 con 0x55e31b92a0e0 2024-05-09T03:16:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.824+0000 7efe7ffff640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe84074cb0 0x7efe840771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.824+0000 7efe7ffff640 1 -- 172.21.15.46:0/92901544 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7efe8c0961e0 con 0x55e31b92a0e0 2024-05-09T03:16:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.824+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e31b927470 0x55e31b927880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.824+0000 7efe96ca5640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe84074cb0 0x7efe840771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.824+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55e31b927dc0 con 0x55e31b927470 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.824+0000 7efe974a6640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e31b927470 0x55e31b927880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe974a6640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e31b927470 0x55e31b927880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe96ca5640 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe84074cb0 0x7efe840771c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efe88000f80 tx=0x7efe88050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe974a6640 1 -- 172.21.15.46:0/92901544 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) ==== 159+0+14 (crc 0 0 0) 0x7efe90009590 con 0x55e31b927470 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stdout:header (14 bytes) : 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stdout:00000000 01 01 08 00 00 00 04 00 00 00 00 00 00 00 |..............| 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stdout:0000000e 2024-05-09T03:16:09.827 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e31b927470 msgr2=0x55e31b927880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e31b927470 0x55e31b927880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe84074cb0 msgr2=0x7efe840771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe84074cb0 0x7efe840771c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efe88000f80 tx=0x7efe88050000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e31b92a0e0 msgr2=0x55e31b99fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 --2- 172.21.15.46:0/92901544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e31b92a0e0 0x55e31b99fef0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00ef30 tx=0x7efe8c00c620 comp rx=0 tx=0).stop 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe974a6640 1 -- 172.21.15.46:0/92901544 reap_dead start 2024-05-09T03:16:09.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 shutdown_connections 2024-05-09T03:16:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 >> 172.21.15.46:0/92901544 conn(0x55e31b807ac0 msgr2=0x55e31b8e2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 shutdown_connections 2024-05-09T03:16:09.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.825+0000 7efe9d1dbf40 1 -- 172.21.15.46:0/92901544 wait complete. 2024-05-09T03:16:09.830 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 400.00000000 2024-05-09T03:16:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.884+0000 7f1e8265cf40 1 Processor -- start 2024-05-09T03:16:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.884+0000 7f1e8265cf40 1 -- start start 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e8265cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2c17e0 0x56118b381770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e8265cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 0x56118b2f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e8265cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b399780 0x56118b380210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e8265cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56118b2c1bf0 con 0x56118b2f0db0 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e8265cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56118b303cb0 con 0x56118b399780 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e8265cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56118b303380 con 0x56118b2c17e0 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 0x56118b2f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 0x56118b2f29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44622/0 (socket says 172.21.15.46:44622) 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 -- 172.21.15.46:0/3436157432 learned_addr learned my addr 172.21.15.46:0/3436157432 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7c124640 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2c17e0 0x56118b381770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7c925640 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b399780 0x56118b380210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 -- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2c17e0 msgr2=0x56118b381770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2c17e0 0x56118b381770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 -- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b399780 msgr2=0x56118b380210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b399780 0x56118b380210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 -- 172.21.15.46:0/3436157432 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118b22c350 con 0x56118b2f0db0 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7b923640 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 0x56118b2f29f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1e700097e0 tx=0x7f1e70005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700fefad0c428a33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.886+0000 7f1e7a921640 1 -- 172.21.15.46:0/3436157432 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70019070 con 0x56118b2f0db0 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e7a921640 1 -- 172.21.15.46:0/3436157432 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e700088b0 con 0x56118b2f0db0 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e7a921640 1 -- 172.21.15.46:0/3436157432 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70008ca0 con 0x56118b2f0db0 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 -- 172.21.15.46:0/3436157432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 msgr2=0x56118b2f29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 0x56118b2f29f0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f1e700097e0 tx=0x7f1e70005a00 comp rx=0 tx=0).stop 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 -- 172.21.15.46:0/3436157432 shutdown_connections 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b399780 0x56118b380210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b2f0db0 0x56118b2f29f0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/3436157432 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2c17e0 0x56118b381770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 -- 172.21.15.46:0/3436157432 >> 172.21.15.46:0/3436157432 conn(0x56118b1e6ac0 msgr2=0x56118b2eedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 -- 172.21.15.46:0/3436157432 shutdown_connections 2024-05-09T03:16:09.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.887+0000 7f1e8265cf40 1 -- 172.21.15.46:0/3436157432 wait complete. 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.888+0000 7f1e8265cf40 1 Processor -- start 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.888+0000 7f1e8265cf40 1 -- start start 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.889+0000 7f1e8265cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b2c17e0 0x56118b3801a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.889+0000 7f1e8265cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2f0db0 0x56118b37b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.889+0000 7f1e8265cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b399780 0x56118b398b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.889+0000 7f1e8265cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56118b2ef910 con 0x56118b399780 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.889+0000 7f1e8265cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56118b2ee350 con 0x56118b2c17e0 2024-05-09T03:16:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.889+0000 7f1e8265cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56118b2ed7b0 con 0x56118b2f0db0 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b399780 0x56118b398b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b399780 0x56118b398b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44624/0 (socket says 172.21.15.46:44624) 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 -- 172.21.15.46:0/2593214147 learned_addr learned my addr 172.21.15.46:0/2593214147 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c124640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b2c17e0 0x56118b3801a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 -- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2f0db0 msgr2=0x56118b37b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7b923640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2f0db0 0x56118b37b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2f0db0 0x56118b37b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 -- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b2c17e0 msgr2=0x56118b3801a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b2c17e0 0x56118b3801a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 -- 172.21.15.46:0/2593214147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56118b2c4ef0 con 0x56118b399780 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c124640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56118b2c17e0 0x56118b3801a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7b923640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56118b2f0db0 0x56118b37b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7c925640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b399780 0x56118b398b70 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1e74025c20 tx=0x7f1e74002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7991f640 1 -- 172.21.15.46:0/2593214147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74064070 con 0x56118b399780 2024-05-09T03:16:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7991f640 1 -- 172.21.15.46:0/2593214147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e7405a040 con 0x56118b399780 2024-05-09T03:16:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e7991f640 1 -- 172.21.15.46:0/2593214147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74059030 con 0x56118b399780 2024-05-09T03:16:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56118b2c51d0 con 0x56118b399780 2024-05-09T03:16:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.890+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56118b308f90 con 0x56118b399780 2024-05-09T03:16:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.892+0000 7f1e7991f640 1 -- 172.21.15.46:0/2593214147 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1e74067070 con 0x56118b399780 2024-05-09T03:16:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.892+0000 7f1e7991f640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e68074ce0 0x7f1e680771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e7991f640 1 -- 172.21.15.46:0/2593214147 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f1e740d9c50 con 0x56118b399780 2024-05-09T03:16:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e7c124640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e68074ce0 0x7f1e680771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56118b308420 0x56118b308830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x56118b3051d0 con 0x56118b308420 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e7b923640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56118b308420 0x56118b308830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e7c124640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e68074ce0 0x7f1e680771f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x56118b2c6440 tx=0x7f1e6c007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.893+0000 7f1e7b923640 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56118b308420 0x56118b308830 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e7b923640 1 -- 172.21.15.46:0/2593214147 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1e70008a60 con 0x56118b308420 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.896 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56118b308420 msgr2=0x56118b308830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56118b308420 0x56118b308830 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e68074ce0 msgr2=0x7f1e680771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1e68074ce0 0x7f1e680771f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x56118b2c6440 tx=0x7f1e6c007040 comp rx=0 tx=0).stop 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b399780 msgr2=0x56118b398b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 --2- 172.21.15.46:0/2593214147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56118b399780 0x56118b398b70 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1e74025c20 tx=0x7f1e74002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e7c925640 1 -- 172.21.15.46:0/2593214147 reap_dead start 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 shutdown_connections 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 >> 172.21.15.46:0/2593214147 conn(0x56118b1e6ac0 msgr2=0x56118b2eec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 shutdown_connections 2024-05-09T03:16:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.894+0000 7f1e8265cf40 1 -- 172.21.15.46:0/2593214147 wait complete. 2024-05-09T03:16:09.899 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000.inode 2024-05-09T03:16:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.953+0000 7f15b0d21f40 1 Processor -- start 2024-05-09T03:16:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.953+0000 7f15b0d21f40 1 -- start start 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15b0d21f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd1956f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15b0d21f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd1957670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15b0d21f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd19fa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15b0d21f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559fd18cff50 con 0x559fd196f010 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15b0d21f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559fd18d3f90 con 0x559fd1a00eb0 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15b0d21f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559fd196d4d0 con 0x559fd19587b0 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd1957670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd1957670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44636/0 (socket says 172.21.15.46:44636) 2024-05-09T03:16:09.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15a3fff640 1 -- 172.21.15.46:0/2242878995 learned_addr learned my addr 172.21.15.46:0/2242878995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15aafec640 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd19fa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.954+0000 7f15aa7eb640 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd1956f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a3fff640 1 -- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 msgr2=0x559fd1956f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a3fff640 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd1956f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a3fff640 1 -- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 msgr2=0x559fd19fa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a3fff640 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd19fa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a3fff640 1 -- 172.21.15.46:0/2242878995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd1957bb0 con 0x559fd196f010 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a3fff640 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd1957670 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f159400a430 tx=0x7f159400a830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846773a3b6166b1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a97e9640 1 -- 172.21.15.46:0/2242878995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159400b8c0 con 0x559fd196f010 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a97e9640 1 -- 172.21.15.46:0/2242878995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f159400f030 con 0x559fd196f010 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.955+0000 7f15a97e9640 1 -- 172.21.15.46:0/2242878995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159400bcc0 con 0x559fd196f010 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2242878995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 msgr2=0x559fd1957670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd1957670 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f159400a430 tx=0x7f159400a830 comp rx=0 tx=0).stop 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2242878995 shutdown_connections 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd19fa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd1957670 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2242878995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd1956f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2242878995 >> 172.21.15.46:0/2242878995 conn(0x559fd184bac0 msgr2=0x559fd1970a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2242878995 shutdown_connections 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.956+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2242878995 wait complete. 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.957+0000 7f15b0d21f40 1 Processor -- start 2024-05-09T03:16:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.957+0000 7f15b0d21f40 1 -- start start 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15b0d21f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd192ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15b0d21f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd196acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15b0d21f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd1968130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15b0d21f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559fd19f99c0 con 0x559fd196f010 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15b0d21f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559fd196b8e0 con 0x559fd1a00eb0 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15b0d21f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559fd19fd2f0 con 0x559fd19587b0 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd196acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd196acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44650/0 (socket says 172.21.15.46:44650) 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 -- 172.21.15.46:0/2898649331 learned_addr learned my addr 172.21.15.46:0/2898649331 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15aafec640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd1968130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15aa7eb640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd192ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 -- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 msgr2=0x559fd192ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559fd19587b0 0x559fd192ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 -- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 msgr2=0x559fd1968130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd1968130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15a3fff640 1 -- 172.21.15.46:0/2898649331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd192b170 con 0x559fd196f010 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.958+0000 7f15aafec640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559fd1a00eb0 0x559fd1968130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.959+0000 7f15a3fff640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd196acd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f159400f000 tx=0x7f1594002f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.959+0000 7f15a37fe640 1 -- 172.21.15.46:0/2898649331 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1594024070 con 0x559fd196f010 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.959+0000 7f15a37fe640 1 -- 172.21.15.46:0/2898649331 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1594014420 con 0x559fd196f010 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.959+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559fd19f05a0 con 0x559fd196f010 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.959+0000 7f15a37fe640 1 -- 172.21.15.46:0/2898649331 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15940147f0 con 0x559fd196f010 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.959+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd196b620 con 0x559fd196f010 2024-05-09T03:16:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.960+0000 7f15a37fe640 1 -- 172.21.15.46:0/2898649331 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1594027070 con 0x559fd196f010 2024-05-09T03:16:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.961+0000 7f15a37fe640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f158c074cb0 0x7f158c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.961+0000 7f15a37fe640 1 -- 172.21.15.46:0/2898649331 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f1594097230 con 0x559fd196f010 2024-05-09T03:16:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.961+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559fd1927d10 0x559fd1a9a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.961+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x559fd19f1a10 con 0x559fd1927d10 2024-05-09T03:16:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.961+0000 7f15aa7eb640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f158c074cb0 0x7f158c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.961+0000 7f15aafec640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559fd1927d10 0x559fd1a9a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15aa7eb640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f158c074cb0 0x7f158c0771c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f159c057000 tx=0x7f159c052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15aafec640 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559fd1927d10 0x559fd1a9a350 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15aafec640 1 -- 172.21.15.46:0/2898649331 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f15a4003530 con 0x559fd1927d10 2024-05-09T03:16:09.967 INFO:teuthology.orchestra.run.smithi046.stdout:header (0 bytes) : 2024-05-09T03:16:09.967 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559fd1927d10 msgr2=0x559fd1a9a350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559fd1927d10 0x559fd1a9a350 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:09.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f158c074cb0 msgr2=0x7f158c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f158c074cb0 0x7f158c0771c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f159c057000 tx=0x7f159c052000 comp rx=0 tx=0).stop 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 msgr2=0x559fd196acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 --2- 172.21.15.46:0/2898649331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fd196f010 0x559fd196acd0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f159400f000 tx=0x7f1594002f70 comp rx=0 tx=0).stop 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15aafec640 1 -- 172.21.15.46:0/2898649331 reap_dead start 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 shutdown_connections 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 >> 172.21.15.46:0/2898649331 conn(0x559fd184bac0 msgr2=0x559fd19fafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 shutdown_connections 2024-05-09T03:16:09.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:09.962+0000 7f15b0d21f40 1 -- 172.21.15.46:0/2898649331 wait complete. 2024-05-09T03:16:09.969 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000 2024-05-09T03:16:10.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.024+0000 7fb8e2cf0f40 1 Processor -- start 2024-05-09T03:16:10.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.024+0000 7fb8e2cf0f40 1 -- start start 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8e2cf0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f3712630 0x5558f36d3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8e2cf0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f37a0c90 0x5558f379f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8e2cf0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f37179b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8e2cf0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5558f3678f50 con 0x5558f379dbe0 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8e2cf0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5558f367cf90 con 0x5558f37a0c90 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8e2cf0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5558f3701f20 con 0x5558f3712630 2024-05-09T03:16:10.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8dcfb9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f37179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8dcfb9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f37179b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44662/0 (socket says 172.21.15.46:44662) 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.025+0000 7fb8dcfb9640 1 -- 172.21.15.46:0/4061272092 learned_addr learned my addr 172.21.15.46:0/4061272092 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dbfb7640 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f37a0c90 0x5558f379f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dc7b8640 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f3712630 0x5558f36d3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dcfb9640 1 -- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f3712630 msgr2=0x5558f36d3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dcfb9640 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f3712630 0x5558f36d3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dcfb9640 1 -- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f37a0c90 msgr2=0x5558f379f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dcfb9640 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f37a0c90 0x5558f379f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dcfb9640 1 -- 172.21.15.46:0/4061272092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558f36d3dd0 con 0x5558f379dbe0 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dcfb9640 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f37179b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400a520 tx=0x7fb8d404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37df6169756be19b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dafb5640 1 -- 172.21.15.46:0/4061272092 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4059040 con 0x5558f379dbe0 2024-05-09T03:16:10.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dafb5640 1 -- 172.21.15.46:0/4061272092 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d4054930 con 0x5558f379dbe0 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.026+0000 7fb8dafb5640 1 -- 172.21.15.46:0/4061272092 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d4054ce0 con 0x5558f379dbe0 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/4061272092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 msgr2=0x5558f37179b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f37179b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb8d400a520 tx=0x7fb8d404e000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/4061272092 shutdown_connections 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f37179b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f37a0c90 0x5558f379f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/4061272092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f3712630 0x5558f36d3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/4061272092 >> 172.21.15.46:0/4061272092 conn(0x5558f35f4ac0 msgr2=0x5558f36fb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/4061272092 shutdown_connections 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.027+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/4061272092 wait complete. 2024-05-09T03:16:10.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.028+0000 7fb8e2cf0f40 1 Processor -- start 2024-05-09T03:16:10.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.028+0000 7fb8e2cf0f40 1 -- start start 2024-05-09T03:16:10.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8e2cf0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f3712630 0x5558f36fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8e2cf0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f383d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8e2cf0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f37a0c90 0x5558f3715350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8e2cf0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5558f36d42f0 con 0x5558f379dbe0 2024-05-09T03:16:10.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8e2cf0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5558f36d40a0 con 0x5558f3712630 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8e2cf0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5558f383dfd0 con 0x5558f37a0c90 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f383d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f383d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44676/0 (socket says 172.21.15.46:44676) 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 -- 172.21.15.46:0/506117763 learned_addr learned my addr 172.21.15.46:0/506117763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dc7b8640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f3712630 0x5558f36fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dcfb9640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f37a0c90 0x5558f3715350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 -- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f37a0c90 msgr2=0x5558f3715350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5558f37a0c90 0x5558f3715350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 -- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f3712630 msgr2=0x5558f36fbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558f3712630 0x5558f36fbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.029+0000 7fb8dbfb7640 1 -- 172.21.15.46:0/506117763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558f36fc530 con 0x5558f379dbe0 2024-05-09T03:16:10.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.030+0000 7fb8dbfb7640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f383d810 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000dfd0 tx=0x7fb8d000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.030+0000 7fb8d9fb3640 1 -- 172.21.15.46:0/506117763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000f650 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.030+0000 7fb8d9fb3640 1 -- 172.21.15.46:0/506117763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d000f7f0 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.030+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5558f36d2c90 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.030+0000 7fb8d9fb3640 1 -- 172.21.15.46:0/506117763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d000fbe0 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.030+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558f36d1520 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.031+0000 7fb8d9fb3640 1 -- 172.21.15.46:0/506117763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb8d00203f0 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.032+0000 7fb8d9fb3640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8c8074cb0 0x7fb8c80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.032+0000 7fb8d9fb3640 1 -- 172.21.15.46:0/506117763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fb8d0096c30 con 0x5558f379dbe0 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.032+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5558f3795d80 0x5558f36d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.032+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5558f36d1f80 con 0x5558f3795d80 2024-05-09T03:16:10.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.032+0000 7fb8dc7b8640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8c8074cb0 0x7fb8c80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8dcfb9640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5558f3795d80 0x5558f36d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8dc7b8640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8c8074cb0 0x7fb8c80771c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc006fd0 tx=0x7fb8cc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8dcfb9640 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5558f3795d80 0x5558f36d1a40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8dcfb9640 1 -- 172.21.15.46:0/506117763 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 154+0+274 (crc 0 0 0) 0x7fb8d4066070 con 0x5558f3795d80 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stdout:header (274 bytes) : 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stdout:00000000 04 03 0c 01 00 00 0c 00 00 00 00 00 00 00 00 00 |................| 2024-05-09T03:16:10.037 INFO:teuthology.orchestra.run.smithi046.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-05-09T03:16:10.038 INFO:teuthology.orchestra.run.smithi046.stdout:00000020 00 00 00 00 f0 3f 3c 66 8f ef d5 21 00 00 00 00 |.....?> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5558f3795d80 msgr2=0x5558f36d1a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5558f3795d80 0x5558f36d1a40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8c8074cb0 msgr2=0x7fb8c80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8c8074cb0 0x7fb8c80771c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc006fd0 tx=0x7fb8cc008040 comp rx=0 tx=0).stop 2024-05-09T03:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 msgr2=0x5558f383d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 --2- 172.21.15.46:0/506117763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5558f379dbe0 0x5558f383d810 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000dfd0 tx=0x7fb8d000b6c0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8dcfb9640 1 -- 172.21.15.46:0/506117763 reap_dead start 2024-05-09T03:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 shutdown_connections 2024-05-09T03:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.033+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 >> 172.21.15.46:0/506117763 conn(0x5558f35f4ac0 msgr2=0x5558f36d36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.034+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 shutdown_connections 2024-05-09T03:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.034+0000 7fb8e2cf0f40 1 -- 172.21.15.46:0/506117763 wait complete. 2024-05-09T03:16:10.042 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000000.00000000 2024-05-09T03:16:10.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.098+0000 7fdd8982ef40 1 Processor -- start 2024-05-09T03:16:10.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.098+0000 7fdd8982ef40 1 -- start start 2024-05-09T03:16:10.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd8982ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 0x55f4c8f168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd8982ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e945f0 0x55f4c8f1da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd8982ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8f1df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd8982ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8df2f50 con 0x55f4c8e8c3a0 2024-05-09T03:16:10.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd8982ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8df6f90 con 0x55f4c8f1fa40 2024-05-09T03:16:10.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd8982ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f4c8e958d0 con 0x55f4c8e945f0 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 0x55f4c8f168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 0x55f4c8f168f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44686/0 (socket says 172.21.15.46:44686) 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 -- 172.21.15.46:0/917883778 learned_addr learned my addr 172.21.15.46:0/917883778 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd83af7640 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8f1df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 -- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e945f0 msgr2=0x55f4c8f1da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd82af5640 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e945f0 0x55f4c8f1da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e945f0 0x55f4c8f1da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 -- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 msgr2=0x55f4c8f1df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8f1df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 -- 172.21.15.46:0/917883778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c8e8ef40 con 0x55f4c8e8c3a0 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd83af7640 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8f1df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.100+0000 7fdd832f6640 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 0x55f4c8f168f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fdd74025840 tx=0x7fdd74051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa60b1b0baf59f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd81af3640 1 -- 172.21.15.46:0/917883778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd740538d0 con 0x55f4c8e8c3a0 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd81af3640 1 -- 172.21.15.46:0/917883778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd74053a70 con 0x55f4c8e8c3a0 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd81af3640 1 -- 172.21.15.46:0/917883778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd74053e20 con 0x55f4c8e8c3a0 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 -- 172.21.15.46:0/917883778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 msgr2=0x55f4c8f168f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 0x55f4c8f168f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fdd74025840 tx=0x7fdd74051000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 -- 172.21.15.46:0/917883778 shutdown_connections 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8f1df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e945f0 0x55f4c8f1da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/917883778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e8c3a0 0x55f4c8f168f0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 -- 172.21.15.46:0/917883778 >> 172.21.15.46:0/917883778 conn(0x55f4c8d6eac0 msgr2=0x55f4c8e964e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 -- 172.21.15.46:0/917883778 shutdown_connections 2024-05-09T03:16:10.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.101+0000 7fdd8982ef40 1 -- 172.21.15.46:0/917883778 wait complete. 2024-05-09T03:16:10.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.102+0000 7fdd8982ef40 1 Processor -- start 2024-05-09T03:16:10.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.102+0000 7fdd8982ef40 1 -- start start 2024-05-09T03:16:10.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e8c3a0 0x55f4c8f10030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e945f0 0x55f4c8f133e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8e4d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8e77410 con 0x55f4c8e945f0 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8f210f0 con 0x55f4c8f1fa40 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f4c8f10570 con 0x55f4c8e8c3a0 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e945f0 0x55f4c8f133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e945f0 0x55f4c8f133e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44698/0 (socket says 172.21.15.46:44698) 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 -- 172.21.15.46:0/504250299 learned_addr learned my addr 172.21.15.46:0/504250299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd83af7640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8e4d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd832f6640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e8c3a0 0x55f4c8f10030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 -- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e8c3a0 msgr2=0x55f4c8f10030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f4c8e8c3a0 0x55f4c8f10030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 -- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 msgr2=0x55f4c8e4d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8e4d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 -- 172.21.15.46:0/504250299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c8f12d10 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd83af7640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4c8f1fa40 0x55f4c8e4d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd82af5640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e945f0 0x55f4c8f133e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00cdc0 tx=0x7fdd6c00a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd80af1640 1 -- 172.21.15.46:0/504250299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c010470 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd80af1640 1 -- 172.21.15.46:0/504250299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd6c010630 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f4c8e75e60 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd80af1640 1 -- 172.21.15.46:0/504250299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c010a20 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.104+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c8e76420 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.106+0000 7fdd80af1640 1 -- 172.21.15.46:0/504250299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdd6c00e040 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd80af1640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd64074cb0 0x7fdd640771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd80af1640 1 -- 172.21.15.46:0/504250299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fdd6c096100 con 0x55f4c8e945f0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd832f6640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd64074cb0 0x7fdd640771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f4c8f0e980 0x55f4c8fc5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55f4c8f13920 con 0x55f4c8f0e980 2024-05-09T03:16:10.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd83af7640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f4c8f0e980 0x55f4c8fc5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd83af7640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f4c8f0e980 0x55f4c8fc5760 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.107+0000 7fdd832f6640 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd64074cb0 0x7fdd640771c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdd74055040 tx=0x7fdd74052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd83af7640 1 -- 172.21.15.46:0/504250299 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-keys out=21b] v0'0 uv10 ondisk = 0) ==== 164+0+21 (crc 0 0 0) 0x7fdd7c00e610 con 0x55f4c8f0e980 2024-05-09T03:16:10.111 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs_head 2024-05-09T03:16:10.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f4c8f0e980 msgr2=0x55f4c8fc5760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f4c8f0e980 0x55f4c8fc5760 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd64074cb0 msgr2=0x7fdd640771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd64074cb0 0x7fdd640771c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdd74055040 tx=0x7fdd74052000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e945f0 msgr2=0x55f4c8f133e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 --2- 172.21.15.46:0/504250299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4c8e945f0 0x55f4c8f133e0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c00cdc0 tx=0x7fdd6c00a5a0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd83af7640 1 -- 172.21.15.46:0/504250299 reap_dead start 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 shutdown_connections 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 >> 172.21.15.46:0/504250299 conn(0x55f4c8d6eac0 msgr2=0x55f4c8e79fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 shutdown_connections 2024-05-09T03:16:10.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.108+0000 7fdd8982ef40 1 -- 172.21.15.46:0/504250299 wait complete. 2024-05-09T03:16:10.114 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 600.00000000 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.167+0000 7fc4b9120f40 1 Processor -- start 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.168+0000 7fc4b9120f40 1 -- start start 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b9120f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1f147e0 0x559ff1febe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b9120f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1fe91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b9120f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 0x559ff1fe96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b9120f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559ff1f5f670 con 0x559ff1fed8c0 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b9120f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559ff1ebdf50 con 0x559ff1f5d010 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b9120f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559ff1f14bf0 con 0x559ff1f147e0 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 0x559ff1fe96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 0x559ff1fe96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44722/0 (socket says 172.21.15.46:44722) 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 -- 172.21.15.46:0/3323179172 learned_addr learned my addr 172.21.15.46:0/3323179172 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4abfff640 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1fe91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b22cd640 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1f147e0 0x559ff1febe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 -- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1f147e0 msgr2=0x559ff1febe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1f147e0 0x559ff1febe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 -- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 msgr2=0x559ff1fe91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1fe91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 -- 172.21.15.46:0/3323179172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff1fee420 con 0x559ff1fed8c0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.169+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 0x559ff1fe96f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac001580 tx=0x7fc4ac001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f80bf767d48b3c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b12cb640 1 -- 172.21.15.46:0/3323179172 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac003650 con 0x559ff1fed8c0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b12cb640 1 -- 172.21.15.46:0/3323179172 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4ac0037f0 con 0x559ff1fed8c0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b12cb640 1 -- 172.21.15.46:0/3323179172 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac003c00 con 0x559ff1fed8c0 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 -- 172.21.15.46:0/3323179172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 msgr2=0x559ff1fe96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 0x559ff1fe96f0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac001580 tx=0x7fc4ac001a40 comp rx=0 tx=0).stop 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 -- 172.21.15.46:0/3323179172 shutdown_connections 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1fed8c0 0x559ff1fe96f0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1fe91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/3323179172 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1f147e0 0x559ff1febe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 -- 172.21.15.46:0/3323179172 >> 172.21.15.46:0/3323179172 conn(0x559ff1e39ac0 msgr2=0x559ff1fedd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 -- 172.21.15.46:0/3323179172 shutdown_connections 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.170+0000 7fc4b9120f40 1 -- 172.21.15.46:0/3323179172 wait complete. 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.171+0000 7fc4b9120f40 1 Processor -- start 2024-05-09T03:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.171+0000 7fc4b9120f40 1 -- start start 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.172+0000 7fc4b9120f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1f147e0 0x559ff1f57d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.172+0000 7fc4b9120f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1f582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.172+0000 7fc4b9120f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1fed8c0 0x559ff1fe7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.172+0000 7fc4b9120f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559ff1fef140 con 0x559ff1f147e0 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.172+0000 7fc4b9120f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559ff1fe8900 con 0x559ff1f5d010 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.172+0000 7fc4b9120f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559ff1fe4310 con 0x559ff1fed8c0 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1f147e0 0x559ff1f57d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1f147e0 0x559ff1f57d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44730/0 (socket says 172.21.15.46:44730) 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 -- 172.21.15.46:0/1634301598 learned_addr learned my addr 172.21.15.46:0/1634301598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4abfff640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1f582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1fed8c0 0x559ff1fe7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 -- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1fed8c0 msgr2=0x559ff1fe7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1fed8c0 0x559ff1fe7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 -- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 msgr2=0x559ff1f582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559ff1f5d010 0x559ff1f582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 -- 172.21.15.46:0/1634301598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff1fe4560 con 0x559ff1f147e0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559ff1fed8c0 0x559ff1fe7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b22cd640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1f147e0 0x559ff1f57d80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4057040 tx=0x7fc4a4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4ab7fe640 1 -- 172.21.15.46:0/1634301598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4064070 con 0x559ff1f147e0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4ab7fe640 1 -- 172.21.15.46:0/1634301598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4a4054cb0 con 0x559ff1f147e0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559ff1f420b0 con 0x559ff1f147e0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4ab7fe640 1 -- 172.21.15.46:0/1634301598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4059040 con 0x559ff1f147e0 2024-05-09T03:16:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.173+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff1f59f70 con 0x559ff1f147e0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.175+0000 7fc4ab7fe640 1 -- 172.21.15.46:0/1634301598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc4a4060430 con 0x559ff1f147e0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4ab7fe640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc494074ce0 0x7fc4940771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4ab7fe640 1 -- 172.21.15.46:0/1634301598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fc4a40daae0 con 0x559ff1f147e0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4abfff640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc494074ce0 0x7fc4940771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559ff1fe8190 0x559ff20877a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x559ff2087ce0 con 0x559ff1fe8190 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559ff1fe8190 0x559ff20877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.176+0000 7fc4abfff640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc494074ce0 0x7fc4940771f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x559ff1fe5510 tx=0x7fc49c00c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b2ace640 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559ff1fe8190 0x559ff20877a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b2ace640 1 -- 172.21.15.46:0/1634301598 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 600.00000000 [omap-get-keys out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7fc4ac00f030 con 0x559ff1fe8190 2024-05-09T03:16:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559ff1fe8190 msgr2=0x559ff20877a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559ff1fe8190 0x559ff20877a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc494074ce0 msgr2=0x7fc4940771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc494074ce0 0x7fc4940771f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x559ff1fe5510 tx=0x7fc49c00c040 comp rx=0 tx=0).stop 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1f147e0 msgr2=0x559ff1f57d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 --2- 172.21.15.46:0/1634301598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559ff1f147e0 0x559ff1f57d80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4057040 tx=0x7fc4a4002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b2ace640 1 -- 172.21.15.46:0/1634301598 reap_dead start 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 shutdown_connections 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 >> 172.21.15.46:0/1634301598 conn(0x559ff1e39ac0 msgr2=0x559ff1fedd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 shutdown_connections 2024-05-09T03:16:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.177+0000 7fc4b9120f40 1 -- 172.21.15.46:0/1634301598 wait complete. 2024-05-09T03:16:10.184 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000.inode 2024-05-09T03:16:10.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.239+0000 7f364c71df40 1 Processor -- start 2024-05-09T03:16:10.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.239+0000 7f364c71df40 1 -- start start 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364c71df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3630003b60 0x555c4f6e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364c71df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 0x555c4f66bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364c71df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555c4f6f99a0 0x555c4f620060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364c71df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555c4f5c9f50 con 0x555c4f6fa3f0 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364c71df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555c4f5cdf90 con 0x7f3630003b60 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364c71df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555c4f66c5a0 con 0x555c4f6f99a0 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 0x555c4f66bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 0x555c4f66bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44742/0 (socket says 172.21.15.46:44742) 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 -- 172.21.15.46:0/1562007144 learned_addr learned my addr 172.21.15.46:0/1562007144 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f364576f640 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3630003b60 0x555c4f6e6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3645f70640 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555c4f6f99a0 0x555c4f620060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 -- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555c4f6f99a0 msgr2=0x555c4f620060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555c4f6f99a0 0x555c4f620060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 -- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3630003b60 msgr2=0x555c4f6e6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3630003b60 0x555c4f6e6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.240+0000 7f3644f6e640 1 -- 172.21.15.46:0/1562007144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4f6ee170 con 0x555c4f6fa3f0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f3644f6e640 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 0x555c4f66bab0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f363c001830 tx=0x7f363c0011e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a092baf3816d2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f36377fe640 1 -- 172.21.15.46:0/1562007144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c00f710 con 0x555c4f6fa3f0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f36377fe640 1 -- 172.21.15.46:0/1562007144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f363c013030 con 0x555c4f6fa3f0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f36377fe640 1 -- 172.21.15.46:0/1562007144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c00faf0 con 0x555c4f6fa3f0 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 -- 172.21.15.46:0/1562007144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 msgr2=0x555c4f66bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 0x555c4f66bab0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f363c001830 tx=0x7f363c0011e0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 -- 172.21.15.46:0/1562007144 shutdown_connections 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555c4f6f99a0 0x555c4f620060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6fa3f0 0x555c4f66bab0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 --2- 172.21.15.46:0/1562007144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3630003b60 0x555c4f6e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 -- 172.21.15.46:0/1562007144 >> 172.21.15.46:0/1562007144 conn(0x555c4f545ac0 msgr2=0x555c4f6eea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 -- 172.21.15.46:0/1562007144 shutdown_connections 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.241+0000 7f364c71df40 1 -- 172.21.15.46:0/1562007144 wait complete. 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.242+0000 7f364c71df40 1 Processor -- start 2024-05-09T03:16:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.242+0000 7f364c71df40 1 -- start start 2024-05-09T03:16:10.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6f99a0 0x555c4f6f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555c4f6fa3f0 0x555c4f6f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3630003b60 0x555c4f652710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555c4f624e60 con 0x555c4f6f99a0 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555c4f66a0f0 con 0x555c4f6fa3f0 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555c4f624b80 con 0x7f3630003b60 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6f99a0 0x555c4f6f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6f99a0 0x555c4f6f09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44748/0 (socket says 172.21.15.46:44748) 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 -- 172.21.15.46:0/1197132682 learned_addr learned my addr 172.21.15.46:0/1197132682 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 -- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3630003b60 msgr2=0x555c4f652710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f3645f70640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3630003b60 0x555c4f652710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f3644f6e640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555c4f6fa3f0 0x555c4f6f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3630003b60 0x555c4f652710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 -- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555c4f6fa3f0 msgr2=0x555c4f6f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555c4f6fa3f0 0x555c4f6f0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 -- 172.21.15.46:0/1197132682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c4f6239b0 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364576f640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6f99a0 0x555c4f6f09f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f363800f440 tx=0x7f363800f900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f36367fc640 1 -- 172.21.15.46:0/1197132682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638014870 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f36367fc640 1 -- 172.21.15.46:0/1197132682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3638019070 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555c4f6f13f0 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.244+0000 7f36367fc640 1 -- 172.21.15.46:0/1197132682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3638014c10 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.245+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4f623c40 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.246+0000 7f36367fc640 1 -- 172.21.15.46:0/1197132682 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3638012070 con 0x555c4f6f99a0 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f36367fc640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3628074cb0 0x7f36280771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f36367fc640 1 -- 172.21.15.46:0/1197132682 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f36380962d0 con 0x555c4f6f99a0 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f3644f6e640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3628074cb0 0x7f36280771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f364c71df40 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555c4f6ee730 0x555c4f6eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x555c4f79ca50 con 0x555c4f6ee730 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f3645f70640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555c4f6ee730 0x555c4f6eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.247+0000 7f3644f6e640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3628074cb0 0x7f36280771c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x555c4f6fad90 tx=0x7f363c01a040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f3645f70640 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555c4f6ee730 0x555c4f6eeb40 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f3645f70640 1 -- 172.21.15.46:0/1197132682 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-keys out=5b] v0'0 uv17 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f364004f8c0 con 0x555c4f6ee730 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555c4f6ee730 msgr2=0x555c4f6eeb40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555c4f6ee730 0x555c4f6eeb40 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3628074cb0 msgr2=0x7f36280771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3628074cb0 0x7f36280771c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x555c4f6fad90 tx=0x7f363c01a040 comp rx=0 tx=0).stop 2024-05-09T03:16:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6f99a0 msgr2=0x555c4f6f09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 --2- 172.21.15.46:0/1197132682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c4f6f99a0 0x555c4f6f09f0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f363800f440 tx=0x7f363800f900 comp rx=0 tx=0).stop 2024-05-09T03:16:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f3645f70640 1 -- 172.21.15.46:0/1197132682 reap_dead start 2024-05-09T03:16:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 shutdown_connections 2024-05-09T03:16:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 >> 172.21.15.46:0/1197132682 conn(0x555c4f545ac0 msgr2=0x555c4f6f6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 shutdown_connections 2024-05-09T03:16:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.248+0000 7f364c71df40 1 -- 172.21.15.46:0/1197132682 wait complete. 2024-05-09T03:16:10.254 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000000 2024-05-09T03:16:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.309+0000 7f2400a23f40 1 Processor -- start 2024-05-09T03:16:10.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.309+0000 7f2400a23f40 1 -- start start 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f2400a23f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4d72130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f2400a23f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d70f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f2400a23f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f2400a23f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cbd4cd6f50 con 0x7f23e8002500 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f2400a23f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55cbd4cdaf90 con 0x55cbd4e04be0 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f2400a23f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55cbd4d78e10 con 0x55cbd4d5b150 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4d72130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4d72130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44766/0 (socket says 172.21.15.46:44766) 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 -- 172.21.15.46:0/683222811 learned_addr learned my addr 172.21.15.46:0/683222811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23facec640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23f3fff640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d70f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 -- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 msgr2=0x55cbd4d70f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d70f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 -- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 msgr2=0x55cbd4d71470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 -- 172.21.15.46:0/683222811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbd4e04060 con 0x7f23e8002500 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23facec640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23f3fff640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d70f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23fa4eb640 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4d72130 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f23ec026340 tx=0x7f23ec050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4cc76f483fcda88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23f94e9640 1 -- 172.21.15.46:0/683222811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec0016f0 con 0x7f23e8002500 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23f94e9640 1 -- 172.21.15.46:0/683222811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23ec066070 con 0x7f23e8002500 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.311+0000 7f23f94e9640 1 -- 172.21.15.46:0/683222811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec061650 con 0x7f23e8002500 2024-05-09T03:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 -- 172.21.15.46:0/683222811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 msgr2=0x55cbd4d72130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4d72130 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f23ec026340 tx=0x7f23ec050000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 -- 172.21.15.46:0/683222811 shutdown_connections 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d70f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 --2- 172.21.15.46:0/683222811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4d72130 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 -- 172.21.15.46:0/683222811 >> 172.21.15.46:0/683222811 conn(0x55cbd4c52ac0 msgr2=0x55cbd4e038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 -- 172.21.15.46:0/683222811 shutdown_connections 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.312+0000 7f2400a23f40 1 -- 172.21.15.46:0/683222811 wait complete. 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.313+0000 7f2400a23f40 1 Processor -- start 2024-05-09T03:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.313+0000 7f2400a23f40 1 -- start start 2024-05-09T03:16:10.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f2400a23f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d31780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f2400a23f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f2400a23f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4dfe630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f2400a23f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cbd4e042d0 con 0x7f23e8002500 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f2400a23f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55cbd4d58640 con 0x55cbd4e04be0 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f2400a23f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55cbd4e05090 con 0x55cbd4d5b150 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f23facec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4dfe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f23facec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4dfe630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44774/0 (socket says 172.21.15.46:44774) 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.314+0000 7f23facec640 1 -- 172.21.15.46:0/936739653 learned_addr learned my addr 172.21.15.46:0/936739653 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23f3fff640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23fa4eb640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d31780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23facec640 1 -- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 msgr2=0x55cbd4d31780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23facec640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55cbd4d5b150 0x55cbd4d31780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23facec640 1 -- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 msgr2=0x55cbd4d71120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23facec640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23facec640 1 -- 172.21.15.46:0/936739653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbd4d31cc0 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23f3fff640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cbd4e04be0 0x55cbd4d71120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23facec640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4dfe630 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x55cbd4d77980 tx=0x7f23f4002940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23f37fe640 1 -- 172.21.15.46:0/936739653 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4018020 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23f37fe640 1 -- 172.21.15.46:0/936739653 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4014890 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cbd4e00830 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f23f37fe640 1 -- 172.21.15.46:0/936739653 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4014c60 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.315+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbd4e00df0 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.317+0000 7f23f37fe640 1 -- 172.21.15.46:0/936739653 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f23f401b020 con 0x7f23e8002500 2024-05-09T03:16:10.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.317+0000 7f23f37fe640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23dc074ce0 0x7f23dc0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.317+0000 7f23f37fe640 1 -- 172.21.15.46:0/936739653 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f23f4096ae0 con 0x7f23e8002500 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.317+0000 7f23fa4eb640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23dc074ce0 0x7f23dc0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.317+0000 7f2400a23f40 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55cbd4d71660 0x55cbd4e9e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.317+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55cbd4e015f0 con 0x55cbd4d71660 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.318+0000 7f23f3fff640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55cbd4d71660 0x55cbd4e9e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.318+0000 7f23fa4eb640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23dc074ce0 0x7f23dc0771f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f23ec026340 tx=0x7f23ec052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.318+0000 7f23f3fff640 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55cbd4d71660 0x55cbd4e9e6c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.318+0000 7f23f3fff640 1 -- 172.21.15.46:0/936739653 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000000 [omap-get-keys out=5b] v0'0 uv20 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f23e400e040 con 0x55cbd4d71660 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.318+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55cbd4d71660 msgr2=0x55cbd4e9e6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.318+0000 7f2400a23f40 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55cbd4d71660 0x55cbd4e9e6c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23dc074ce0 msgr2=0x7f23dc0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23dc074ce0 0x7f23dc0771f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f23ec026340 tx=0x7f23ec052000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 msgr2=0x55cbd4dfe630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 --2- 172.21.15.46:0/936739653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23e8002500 0x55cbd4dfe630 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x55cbd4d77980 tx=0x7f23f4002940 comp rx=0 tx=0).stop 2024-05-09T03:16:10.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f23facec640 1 -- 172.21.15.46:0/936739653 reap_dead start 2024-05-09T03:16:10.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 shutdown_connections 2024-05-09T03:16:10.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 >> 172.21.15.46:0/936739653 conn(0x55cbd4c52ac0 msgr2=0x55cbd4e038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 shutdown_connections 2024-05-09T03:16:10.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.319+0000 7f2400a23f40 1 -- 172.21.15.46:0/936739653 wait complete. 2024-05-09T03:16:10.325 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000001 2024-05-09T03:16:10.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.379+0000 7f4004c20f40 1 Processor -- start 2024-05-09T03:16:10.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.379+0000 7f4004c20f40 1 -- start start 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f4004c20f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8337b0 0x55e83e830ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f4004c20f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 0x55e83e82e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f4004c20f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e846f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f4004c20f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e83e7aaf50 con 0x55e83e8db810 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f4004c20f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e83e7aef90 con 0x55e83e8337b0 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f4004c20f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e83e8484d0 con 0x55e83e8d99a0 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 0x55e83e82e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 0x55e83e82e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44788/0 (socket says 172.21.15.46:44788) 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 -- 172.21.15.46:0/1184691545 learned_addr learned my addr 172.21.15.46:0/1184691545 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffe275640 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e846f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffda74640 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8337b0 0x55e83e830ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 -- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 msgr2=0x55e83e846f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e846f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 -- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8337b0 msgr2=0x55e83e830ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8337b0 0x55e83e830ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 -- 172.21.15.46:0/1184691545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e83e82e890 con 0x55e83e8db810 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffe275640 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e846f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ffd273640 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 0x55e83e82e350 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8001de0 tx=0x7f3fe800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba253c98dcfc686 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ff7fff640 1 -- 172.21.15.46:0/1184691545 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8012020 con 0x55e83e8db810 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ff7fff640 1 -- 172.21.15.46:0/1184691545 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fe800e670 con 0x55e83e8db810 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.381+0000 7f3ff7fff640 1 -- 172.21.15.46:0/1184691545 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800ea60 con 0x55e83e8db810 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 -- 172.21.15.46:0/1184691545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 msgr2=0x55e83e82e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 0x55e83e82e350 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8001de0 tx=0x7f3fe800b6c0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 -- 172.21.15.46:0/1184691545 shutdown_connections 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e846f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8db810 0x55e83e82e350 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 --2- 172.21.15.46:0/1184691545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8337b0 0x55e83e830ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 -- 172.21.15.46:0/1184691545 >> 172.21.15.46:0/1184691545 conn(0x55e83e726ac0 msgr2=0x55e83e801060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 -- 172.21.15.46:0/1184691545 shutdown_connections 2024-05-09T03:16:10.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.382+0000 7f4004c20f40 1 -- 172.21.15.46:0/1184691545 wait complete. 2024-05-09T03:16:10.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.383+0000 7f4004c20f40 1 Processor -- start 2024-05-09T03:16:10.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.383+0000 7f4004c20f40 1 -- start start 2024-05-09T03:16:10.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f4004c20f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8337b0 0x55e83e8cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f4004c20f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e8d5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f4004c20f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8db810 0x55e83e8d2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f4004c20f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e83e832900 con 0x55e83e8337b0 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f4004c20f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e83e8326e0 con 0x55e83e8db810 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f4004c20f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e83e8d5e40 con 0x55e83e8d99a0 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f3ffda74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8337b0 0x55e83e8cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f3ffda74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8337b0 0x55e83e8cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44792/0 (socket says 172.21.15.46:44792) 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.384+0000 7f3ffda74640 1 -- 172.21.15.46:0/4258421842 learned_addr learned my addr 172.21.15.46:0/4258421842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffe275640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8db810 0x55e83e8d2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffd273640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e8d5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffda74640 1 -- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 msgr2=0x55e83e8d5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffda74640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e83e8d99a0 0x55e83e8d5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffda74640 1 -- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8db810 msgr2=0x55e83e8d2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffda74640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8db810 0x55e83e8d2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffda74640 1 -- 172.21.15.46:0/4258421842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e83e8d8080 con 0x55e83e8337b0 2024-05-09T03:16:10.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffe275640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e83e8db810 0x55e83e8d2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ffda74640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8337b0 0x55e83e8cca80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800df40 tx=0x7f3ff800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ff6ffd640 1 -- 172.21.15.46:0/4258421842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8015020 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ff6ffd640 1 -- 172.21.15.46:0/4258421842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff80030d0 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e83e847850 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f3ff6ffd640 1 -- 172.21.15.46:0/4258421842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8011480 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.385+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e83e8d5b80 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.387+0000 7f3ff6ffd640 1 -- 172.21.15.46:0/4258421842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3ff8018020 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.387+0000 7f3ff6ffd640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fe0074cb0 0x7f3fe00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.387+0000 7f3ff6ffd640 1 -- 172.21.15.46:0/4258421842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f3ff80962c0 con 0x55e83e8337b0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.388+0000 7f3ffd273640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fe0074cb0 0x7f3fe00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.388+0000 7f4004c20f40 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e83e82f400 0x55e83e97d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.388+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55e83e97dcf0 con 0x55e83e82f400 2024-05-09T03:16:10.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.388+0000 7f3ffe275640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e83e82f400 0x55e83e97d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.388+0000 7f3ffd273640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fe0074cb0 0x7f3fe00771c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55e83e84ebf0 tx=0x7f3fe800f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.388+0000 7f3ffe275640 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e83e82f400 0x55e83e97d7b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f3ffe275640 1 -- 172.21.15.46:0/4258421842 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000001 [omap-get-keys out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f3ff0057030 con 0x55e83e82f400 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e83e82f400 msgr2=0x55e83e97d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e83e82f400 0x55e83e97d7b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fe0074cb0 msgr2=0x7f3fe00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3fe0074cb0 0x7f3fe00771c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55e83e84ebf0 tx=0x7f3fe800f000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8337b0 msgr2=0x55e83e8cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 --2- 172.21.15.46:0/4258421842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e83e8337b0 0x55e83e8cca80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800df40 tx=0x7f3ff800b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f3ffe275640 1 -- 172.21.15.46:0/4258421842 reap_dead start 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 shutdown_connections 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 >> 172.21.15.46:0/4258421842 conn(0x55e83e726ac0 msgr2=0x55e83e801060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 shutdown_connections 2024-05-09T03:16:10.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.389+0000 7f4004c20f40 1 -- 172.21.15.46:0/4258421842 wait complete. 2024-05-09T03:16:10.394 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_openfiles.0 2024-05-09T03:16:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.449+0000 7fb44771bf40 1 Processor -- start 2024-05-09T03:16:10.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.449+0000 7fb44771bf40 1 -- start start 2024-05-09T03:16:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb44771bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffbd5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb44771bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffb4e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb44771bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 0x55b3ffb35a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb44771bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3ffab1f50 con 0x55b3ffb4e760 2024-05-09T03:16:10.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb44771bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3ffab5f90 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb44771bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b3ffb4f4d0 con 0x55b3ffbe2810 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 0x55b3ffb35a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 0x55b3ffb35a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44808/0 (socket says 172.21.15.46:44808) 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 -- 172.21.15.46:0/2416005268 learned_addr learned my addr 172.21.15.46:0/2416005268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440734640 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffbd5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb437fff640 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffb4e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 -- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 msgr2=0x55b3ffb4e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffb4e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 -- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb3a7b0 msgr2=0x55b3ffbd5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffbd5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb440f35640 1 -- 172.21.15.46:0/2416005268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ffbdb7c0 con 0x55b3ffb4e760 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.450+0000 7fb437fff640 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffb4e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb440f35640 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 0x55b3ffb35a80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb43800a4a0 tx=0x7fb438051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258f078f8c3747ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb43f732640 1 -- 172.21.15.46:0/2416005268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4380539d0 con 0x55b3ffb4e760 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb43f732640 1 -- 172.21.15.46:0/2416005268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb438053b70 con 0x55b3ffb4e760 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb43f732640 1 -- 172.21.15.46:0/2416005268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4380673f0 con 0x55b3ffb4e760 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 -- 172.21.15.46:0/2416005268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 msgr2=0x55b3ffb35a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 0x55b3ffb35a80 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb43800a4a0 tx=0x7fb438051000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 -- 172.21.15.46:0/2416005268 shutdown_connections 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb4e760 0x55b3ffb35a80 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffb4e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2416005268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffbd5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 -- 172.21.15.46:0/2416005268 >> 172.21.15.46:0/2416005268 conn(0x55b3ffa2dac0 msgr2=0x55b3ffb08060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.451+0000 7fb44771bf40 1 -- 172.21.15.46:0/2416005268 shutdown_connections 2024-05-09T03:16:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.452+0000 7fb44771bf40 1 -- 172.21.15.46:0/2416005268 wait complete. 2024-05-09T03:16:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.452+0000 7fb44771bf40 1 Processor -- start 2024-05-09T03:16:10.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.452+0000 7fb44771bf40 1 -- start start 2024-05-09T03:16:10.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffb362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb4e760 0x55b3ffbd2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffbde1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3ffbd65b0 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3ffb37cb0 con 0x55b3ffb4e760 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b3ffbd6800 con 0x55b3ffbe2810 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffb362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffb362a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44810/0 (socket says 172.21.15.46:44810) 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 -- 172.21.15.46:0/2949704283 learned_addr learned my addr 172.21.15.46:0/2949704283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 -- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 msgr2=0x55b3ffbde1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b3ffbe2810 0x55b3ffbde1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb437fff640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb4e760 0x55b3ffbd2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 -- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb4e760 msgr2=0x55b3ffbd2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3ffb4e760 0x55b3ffbd2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 -- 172.21.15.46:0/2949704283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ffbd31e0 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb440734640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffb362a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb43000ef90 tx=0x7fb43000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb43e730640 1 -- 172.21.15.46:0/2949704283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4300019c0 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb43e730640 1 -- 172.21.15.46:0/2949704283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb430001b60 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3ffb367e0 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.454+0000 7fb43e730640 1 -- 172.21.15.46:0/2949704283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430001d50 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.455+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ffb35aa0 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.456+0000 7fb43e730640 1 -- 172.21.15.46:0/2949704283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb430019020 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb43e730640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb420074cb0 0x7fb4200771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb43e730640 1 -- 172.21.15.46:0/2949704283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fb430095c90 con 0x55b3ffb3a7b0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b3ffb393e0 0x55b3ffc84810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55b3ffbde6e0 con 0x55b3ffb393e0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb437fff640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb420074cb0 0x7fb4200771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb440f35640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b3ffb393e0 0x55b3ffc84810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb440f35640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b3ffb393e0 0x55b3ffc84810 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.457+0000 7fb437fff640 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb420074cb0 0x7fb4200771c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55b3ffb34800 tx=0x7fb4280011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb440f35640 1 -- 172.21.15.46:0/2949704283 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-keys out=5b] v0'0 uv4 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7fb4380537d0 con 0x55b3ffb393e0 2024-05-09T03:16:10.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b3ffb393e0 msgr2=0x55b3ffc84810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b3ffb393e0 0x55b3ffc84810 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb420074cb0 msgr2=0x7fb4200771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb420074cb0 0x7fb4200771c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55b3ffb34800 tx=0x7fb4280011a0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb3a7b0 msgr2=0x55b3ffb362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 --2- 172.21.15.46:0/2949704283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b3ffb3a7b0 0x55b3ffb362a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb43000ef90 tx=0x7fb43000c620 comp rx=0 tx=0).stop 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb440f35640 1 -- 172.21.15.46:0/2949704283 reap_dead start 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 shutdown_connections 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 >> 172.21.15.46:0/2949704283 conn(0x55b3ffa2dac0 msgr2=0x55b3ffb08060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 shutdown_connections 2024-05-09T03:16:10.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.458+0000 7fb44771bf40 1 -- 172.21.15.46:0/2949704283 wait complete. 2024-05-09T03:16:10.465 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 500.00000000 2024-05-09T03:16:10.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.519+0000 7f167d11cf40 1 Processor -- start 2024-05-09T03:16:10.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.519+0000 7f167d11cf40 1 -- start start 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167d11cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0d803a0 0x5637f0d82fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167d11cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d885f0 0x5637f0d6e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167d11cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 0x5637f0ea8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167d11cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5637f0ce6f50 con 0x5637f0e14150 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167d11cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5637f0ceaf90 con 0x5637f0d885f0 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167d11cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5637f0d898d0 con 0x5637f0d803a0 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 0x5637f0ea8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 0x5637f0ea8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44832/0 (socket says 172.21.15.46:44832) 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 -- 172.21.15.46:0/871993738 learned_addr learned my addr 172.21.15.46:0/871993738 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f167597f640 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d885f0 0x5637f0d6e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676180640 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0d803a0 0x5637f0d82fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 -- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0d803a0 msgr2=0x5637f0d82fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0d803a0 0x5637f0d82fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 -- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d885f0 msgr2=0x5637f0d6e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d885f0 0x5637f0d6e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.520+0000 7f1676981640 1 -- 172.21.15.46:0/871993738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f0d699f0 con 0x5637f0e14150 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f1676981640 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 0x5637f0ea8f60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f16700015e0 tx=0x7f16700019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df7df1400d0255c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167497d640 1 -- 172.21.15.46:0/871993738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670016070 con 0x5637f0e14150 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167497d640 1 -- 172.21.15.46:0/871993738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f167000f710 con 0x5637f0e14150 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167497d640 1 -- 172.21.15.46:0/871993738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167000fb00 con 0x5637f0e14150 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 -- 172.21.15.46:0/871993738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 msgr2=0x5637f0ea8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 0x5637f0ea8f60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f16700015e0 tx=0x7f16700019e0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 -- 172.21.15.46:0/871993738 shutdown_connections 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0e14150 0x5637f0ea8f60 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d885f0 0x5637f0d6e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 --2- 172.21.15.46:0/871993738 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0d803a0 0x5637f0d82fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.521+0000 7f167d11cf40 1 -- 172.21.15.46:0/871993738 >> 172.21.15.46:0/871993738 conn(0x5637f0c62ac0 msgr2=0x5637f0d83800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.522+0000 7f167d11cf40 1 -- 172.21.15.46:0/871993738 shutdown_connections 2024-05-09T03:16:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.522+0000 7f167d11cf40 1 -- 172.21.15.46:0/871993738 wait complete. 2024-05-09T03:16:10.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.522+0000 7f167d11cf40 1 Processor -- start 2024-05-09T03:16:10.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.522+0000 7f167d11cf40 1 -- start start 2024-05-09T03:16:10.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167d11cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d803a0 0x5637f0e06750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167d11cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0d885f0 0x5637f0eb0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167d11cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0e14150 0x5637f0eb1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167d11cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5637f0e12d60 con 0x5637f0d885f0 2024-05-09T03:16:10.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167d11cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5637f0d6ae40 con 0x5637f0d803a0 2024-05-09T03:16:10.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167d11cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5637f0d6a0e0 con 0x5637f0e14150 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0d885f0 0x5637f0eb0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0d885f0 0x5637f0eb0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44844/0 (socket says 172.21.15.46:44844) 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 -- 172.21.15.46:0/86040545 learned_addr learned my addr 172.21.15.46:0/86040545 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f1676180640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d803a0 0x5637f0e06750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f1676981640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0e14150 0x5637f0eb1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 -- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0e14150 msgr2=0x5637f0eb1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0e14150 0x5637f0eb1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 -- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d803a0 msgr2=0x5637f0e06750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5637f0d803a0 0x5637f0e06750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 -- 172.21.15.46:0/86040545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637f0e15660 con 0x5637f0d885f0 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f1676981640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5637f0e14150 0x5637f0eb1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.524+0000 7f167597f640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0d885f0 0x5637f0eb0cf0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f166c000c00 tx=0x7f166c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.525+0000 7f16677fe640 1 -- 172.21.15.46:0/86040545 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c05b9f0 con 0x5637f0d885f0 2024-05-09T03:16:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.525+0000 7f16677fe640 1 -- 172.21.15.46:0/86040545 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166c05bb90 con 0x5637f0d885f0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.525+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5637f0e08030 con 0x5637f0d885f0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.525+0000 7f16677fe640 1 -- 172.21.15.46:0/86040545 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c066480 con 0x5637f0d885f0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.525+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637f0e15900 con 0x5637f0d885f0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.526+0000 7f16677fe640 1 -- 172.21.15.46:0/86040545 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f166c06d020 con 0x5637f0d885f0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.527+0000 7f16677fe640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658074cb0 0x7f16580771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.527+0000 7f16677fe640 1 -- 172.21.15.46:0/86040545 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f166c0d9eb0 con 0x5637f0d885f0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.527+0000 7f1676180640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658074cb0 0x7f16580771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.527+0000 7f167d11cf40 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5637f0dfd740 0x5637f0dfdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.527+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5637f0eb1770 con 0x5637f0dfd740 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.527+0000 7f1676981640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5637f0dfd740 0x5637f0dfdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f1676180640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658074cb0 0x7f16580771c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1668001ed0 tx=0x7f1668001aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f1676981640 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5637f0dfd740 0x5637f0dfdb50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f1676981640 1 -- 172.21.15.46:0/86040545 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 500.00000000 [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f167000f070 con 0x5637f0dfd740 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5637f0dfd740 msgr2=0x5637f0dfdb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5637f0dfd740 0x5637f0dfdb50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658074cb0 msgr2=0x7f16580771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1658074cb0 0x7f16580771c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1668001ed0 tx=0x7f1668001aa0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0d885f0 msgr2=0x5637f0eb0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 --2- 172.21.15.46:0/86040545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5637f0d885f0 0x5637f0eb0cf0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f166c000c00 tx=0x7f166c003020 comp rx=0 tx=0).stop 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f1676981640 1 -- 172.21.15.46:0/86040545 reap_dead start 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 shutdown_connections 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 >> 172.21.15.46:0/86040545 conn(0x5637f0c62ac0 msgr2=0x5637f0d83800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.528+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 shutdown_connections 2024-05-09T03:16:10.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.529+0000 7f167d11cf40 1 -- 172.21.15.46:0/86040545 wait complete. 2024-05-09T03:16:10.534 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds_snaptable 2024-05-09T03:16:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.588+0000 7f0aaec10f40 1 Processor -- start 2024-05-09T03:16:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.588+0000 7f0aaec10f40 1 -- start start 2024-05-09T03:16:10.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aaec10f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d1de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aaec10f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 0x563430c82c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aaec10f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430d25d20 0x563430c83190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aaec10f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563430c994d0 con 0x563430c9b010 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aaec10f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563430bfbf50 con 0x563430d25d20 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aaec10f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563430c9e8a0 con 0x563430c847b0 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 0x563430c82c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 0x563430c82c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44848/0 (socket says 172.21.15.46:44848) 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 -- 172.21.15.46:0/1471878130 learned_addr learned my addr 172.21.15.46:0/1471878130 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aa8169640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430d25d20 0x563430c83190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aa7968640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d1de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 -- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 msgr2=0x563430d1de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d1de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 -- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430d25d20 msgr2=0x563430c83190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430d25d20 0x563430c83190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 -- 172.21.15.46:0/1471878130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563430d2a570 con 0x563430c9b010 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aa7968640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d1de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aa8169640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430d25d20 0x563430c83190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 0x563430c82c50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0a90001df0 tx=0x7f0a9000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d27e46dd78300d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.590+0000 7f0aa6966640 1 -- 172.21.15.46:0/1471878130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a90012020 con 0x563430c9b010 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aa6966640 1 -- 172.21.15.46:0/1471878130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9000e670 con 0x563430c9b010 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aa6966640 1 -- 172.21.15.46:0/1471878130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000ea40 con 0x563430c9b010 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1471878130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 msgr2=0x563430c82c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 0x563430c82c50 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f0a90001df0 tx=0x7f0a9000b6c0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1471878130 shutdown_connections 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430d25d20 0x563430c83190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430c9b010 0x563430c82c50 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1471878130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d1de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1471878130 >> 172.21.15.46:0/1471878130 conn(0x563430b77ac0 msgr2=0x563430c93f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1471878130 shutdown_connections 2024-05-09T03:16:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.591+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1471878130 wait complete. 2024-05-09T03:16:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.592+0000 7f0aaec10f40 1 Processor -- start 2024-05-09T03:16:10.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.592+0000 7f0aaec10f40 1 -- start start 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aaec10f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d12b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aaec10f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430c9b010 0x563430d13410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aaec10f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430d25d20 0x563430c82ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aaec10f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563430c83d50 con 0x563430d25d20 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aaec10f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563430d17930 con 0x563430c9b010 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aaec10f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563430d17b80 con 0x563430c847b0 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aa8169640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430d25d20 0x563430c82ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.593+0000 7f0aa8169640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430d25d20 0x563430c82ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44854/0 (socket says 172.21.15.46:44854) 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 -- 172.21.15.46:0/1277156708 learned_addr learned my addr 172.21.15.46:0/1277156708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa7968640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d12b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430c9b010 0x563430d13410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 -- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 msgr2=0x563430d12b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563430c847b0 0x563430d12b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 -- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430c9b010 msgr2=0x563430d13410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430c9b010 0x563430d13410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 -- 172.21.15.46:0/1277156708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563430d17e60 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563430c9b010 0x563430d13410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa8169640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430d25d20 0x563430c82ba0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000c240 tx=0x7f0aa000c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa5964640 1 -- 172.21.15.46:0/1277156708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0017030 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa5964640 1 -- 172.21.15.46:0/1277156708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aa0013670 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563430d18140 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aa5964640 1 -- 172.21.15.46:0/1277156708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0013860 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.594+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563430d1af40 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.596+0000 7f0aa5964640 1 -- 172.21.15.46:0/1277156708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0aa001a070 con 0x563430d25d20 2024-05-09T03:16:10.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.596+0000 7f0aa5964640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a88074cb0 0x7f0a880771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.596+0000 7f0aa5964640 1 -- 172.21.15.46:0/1277156708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f0aa00965d0 con 0x563430d25d20 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.597+0000 7f0aa7968640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a88074cb0 0x7f0a880771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.597+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563430d13950 0x563430d1a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.597+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x563430d1a910 con 0x563430d13950 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.597+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563430d13950 0x563430d1a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.597+0000 7f0aa7968640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a88074cb0 0x7f0a880771c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0a98056000 tx=0x7f0a98050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.597+0000 7f0a9ffff640 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563430d13950 0x563430d1a3d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0a9ffff640 1 -- 172.21.15.46:0/1277156708 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 mds_snaptable [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f0a9000e820 con 0x563430d13950 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563430d13950 msgr2=0x563430d1a3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563430d13950 0x563430d1a3d0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a88074cb0 msgr2=0x7f0a880771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a88074cb0 0x7f0a880771c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0a98056000 tx=0x7f0a98050000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430d25d20 msgr2=0x563430c82ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 --2- 172.21.15.46:0/1277156708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563430d25d20 0x563430c82ba0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000c240 tx=0x7f0aa000c700 comp rx=0 tx=0).stop 2024-05-09T03:16:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aa8169640 1 -- 172.21.15.46:0/1277156708 reap_dead start 2024-05-09T03:16:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 shutdown_connections 2024-05-09T03:16:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 >> 172.21.15.46:0/1277156708 conn(0x563430b77ac0 msgr2=0x563430c98840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 shutdown_connections 2024-05-09T03:16:10.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.598+0000 7f0aaec10f40 1 -- 172.21.15.46:0/1277156708 wait complete. 2024-05-09T03:16:10.603 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_inotable 2024-05-09T03:16:10.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.658+0000 7f0d0a908f40 1 Processor -- start 2024-05-09T03:16:10.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.658+0000 7f0d0a908f40 1 -- start start 2024-05-09T03:16:10.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.659+0000 7f0d0a908f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647648f07e0 0x564764922ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.659+0000 7f0d0a908f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 0x564764938950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.659+0000 7f0d0a908f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647649c8780 0x5647649ab860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.659+0000 7f0d0a908f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564764932cb0 con 0x56476491fdb0 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.659+0000 7f0d0a908f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564764899f50 con 0x5647648f07e0 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.659+0000 7f0d0a908f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5647648f0bf0 con 0x5647649c8780 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 0x564764938950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 0x564764938950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44860/0 (socket says 172.21.15.46:44860) 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 -- 172.21.15.46:0/4029071595 learned_addr learned my addr 172.21.15.46:0/4029071595 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d043d0640 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647648f07e0 0x564764922ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d04bd1640 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647649c8780 0x5647649ab860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 -- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647649c8780 msgr2=0x5647649ab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647649c8780 0x5647649ab860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 -- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647648f07e0 msgr2=0x564764922ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647648f07e0 0x564764922ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 -- 172.21.15.46:0/4029071595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647649be920 con 0x56476491fdb0 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d03bcf640 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 0x564764938950 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0cec045ed0 tx=0x7f0cec051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a40022e579fd01d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d02bcd640 1 -- 172.21.15.46:0/4029071595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec062070 con 0x56476491fdb0 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d02bcd640 1 -- 172.21.15.46:0/4029071595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cec054970 con 0x56476491fdb0 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.660+0000 7f0d02bcd640 1 -- 172.21.15.46:0/4029071595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cec054d20 con 0x56476491fdb0 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 -- 172.21.15.46:0/4029071595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 msgr2=0x564764938950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 0x564764938950 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f0cec045ed0 tx=0x7f0cec051000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 -- 172.21.15.46:0/4029071595 shutdown_connections 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5647649c8780 0x5647649ab860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56476491fdb0 0x564764938950 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/4029071595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647648f07e0 0x564764922ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 -- 172.21.15.46:0/4029071595 >> 172.21.15.46:0/4029071595 conn(0x564764815ac0 msgr2=0x564764921800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 -- 172.21.15.46:0/4029071595 shutdown_connections 2024-05-09T03:16:10.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.661+0000 7f0d0a908f40 1 -- 172.21.15.46:0/4029071595 wait complete. 2024-05-09T03:16:10.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.662+0000 7f0d0a908f40 1 Processor -- start 2024-05-09T03:16:10.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.662+0000 7f0d0a908f40 1 -- start start 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d0a908f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647648f07e0 0x56476493c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d0a908f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56476491fdb0 0x5647649ab600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d0a908f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647649c8780 0x5647649abb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d0a908f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564764937490 con 0x5647648f07e0 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d0a908f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5647649beb90 con 0x5647649c8780 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d0a908f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5647649c83e0 con 0x56476491fdb0 2024-05-09T03:16:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d043d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647648f07e0 0x56476493c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d043d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647648f07e0 0x56476493c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44868/0 (socket says 172.21.15.46:44868) 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.663+0000 7f0d043d0640 1 -- 172.21.15.46:0/853826993 learned_addr learned my addr 172.21.15.46:0/853826993 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d04bd1640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647649c8780 0x5647649abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d03bcf640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56476491fdb0 0x5647649ab600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d043d0640 1 -- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56476491fdb0 msgr2=0x5647649ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d043d0640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56476491fdb0 0x5647649ab600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d043d0640 1 -- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647649c8780 msgr2=0x5647649abb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d043d0640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647649c8780 0x5647649abb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d043d0640 1 -- 172.21.15.46:0/853826993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647649afd00 con 0x5647648f07e0 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d04bd1640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647649c8780 0x5647649abb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d043d0640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647648f07e0 0x56476493c3a0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4010690 tx=0x7f0cf4010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d01bcb640 1 -- 172.21.15.46:0/853826993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4010e60 con 0x5647648f07e0 2024-05-09T03:16:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d01bcb640 1 -- 172.21.15.46:0/853826993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf400e9d0 con 0x5647648f07e0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5647649affe0 con 0x5647648f07e0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d01bcb640 1 -- 172.21.15.46:0/853826993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf400eda0 con 0x5647648f07e0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.664+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647649c8070 con 0x5647648f07e0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.666+0000 7f0d01bcb640 1 -- 172.21.15.46:0/853826993 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0cf4011040 con 0x5647648f07e0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.666+0000 7f0d01bcb640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ce4074ce0 0x7f0ce40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.666+0000 7f0d01bcb640 1 -- 172.21.15.46:0/853826993 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f0cf4096920 con 0x5647648f07e0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.666+0000 7f0d03bcf640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ce4074ce0 0x7f0ce40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.666+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647649aa970 0x5647649c4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.666+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5647649c5070 con 0x5647649aa970 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d04bd1640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647649aa970 0x5647649c4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d03bcf640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ce4074ce0 0x7f0ce40771f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0cec064000 tx=0x7f0cec053000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d04bd1640 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647649aa970 0x5647649c4b30 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d04bd1640 1 -- 172.21.15.46:0/853826993 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_inotable [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f0cfc00e040 con 0x5647649aa970 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647649aa970 msgr2=0x5647649c4b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5647649aa970 0x5647649c4b30 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ce4074ce0 msgr2=0x7f0ce40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ce4074ce0 0x7f0ce40771f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0cec064000 tx=0x7f0cec053000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647648f07e0 msgr2=0x56476493c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 --2- 172.21.15.46:0/853826993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5647648f07e0 0x56476493c3a0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4010690 tx=0x7f0cf4010b50 comp rx=0 tx=0).stop 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d04bd1640 1 -- 172.21.15.46:0/853826993 reap_dead start 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 shutdown_connections 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 >> 172.21.15.46:0/853826993 conn(0x564764815ac0 msgr2=0x5647649c5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 shutdown_connections 2024-05-09T03:16:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.667+0000 7f0d0a908f40 1 -- 172.21.15.46:0/853826993 wait complete. 2024-05-09T03:16:10.675 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000 2024-05-09T03:16:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.729+0000 7f7e50db3f40 1 Processor -- start 2024-05-09T03:16:10.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.729+0000 7f7e50db3f40 1 -- start start 2024-05-09T03:16:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e50db3f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce983ff0 0x55a1cea51530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e50db3f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1ce9b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e50db3f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 0x55a1ce9b1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e50db3f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a1ce984400 con 0x55a1ce9ca060 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e50db3f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a1ce929f50 con 0x55a1ce983ff0 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e50db3f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55a1ce9cd000 con 0x55a1ce9862b0 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e4b07c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 0x55a1ce9b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e4b07c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 0x55a1ce9b1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44878/0 (socket says 172.21.15.46:44878) 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e4b07c640 1 -- 172.21.15.46:0/952782930 learned_addr learned my addr 172.21.15.46:0/952782930 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e4a87b640 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce983ff0 0x55a1cea51530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.730+0000 7f7e4a07a640 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1ce9b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4b07c640 1 -- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 msgr2=0x55a1ce9b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4b07c640 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1ce9b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4b07c640 1 -- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce983ff0 msgr2=0x55a1cea51530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4b07c640 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce983ff0 0x55a1cea51530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4b07c640 1 -- 172.21.15.46:0/952782930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1ce9c16e0 con 0x55a1ce9ca060 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4a07a640 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1ce9b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e4b07c640 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 0x55a1ce9b1210 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7e44055040 tx=0x7f7e44051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb24ed765e340b17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e49078640 1 -- 172.21.15.46:0/952782930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44056040 con 0x55a1ce9ca060 2024-05-09T03:16:10.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e49078640 1 -- 172.21.15.46:0/952782930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e440528c0 con 0x55a1ce9ca060 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.731+0000 7f7e49078640 1 -- 172.21.15.46:0/952782930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44052cb0 con 0x55a1ce9ca060 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 -- 172.21.15.46:0/952782930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 msgr2=0x55a1ce9b1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 0x55a1ce9b1210 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7e44055040 tx=0x7f7e44051000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 -- 172.21.15.46:0/952782930 shutdown_connections 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce9ca060 0x55a1ce9b1210 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1ce9b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/952782930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce983ff0 0x55a1cea51530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 -- 172.21.15.46:0/952782930 >> 172.21.15.46:0/952782930 conn(0x55a1ce8a5ac0 msgr2=0x55a1ce9c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 -- 172.21.15.46:0/952782930 shutdown_connections 2024-05-09T03:16:10.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.732+0000 7f7e50db3f40 1 -- 172.21.15.46:0/952782930 wait complete. 2024-05-09T03:16:10.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.733+0000 7f7e50db3f40 1 Processor -- start 2024-05-09T03:16:10.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.733+0000 7f7e50db3f40 1 -- start start 2024-05-09T03:16:10.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e50db3f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce983ff0 0x55a1cea509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e50db3f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1cea59420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e50db3f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce9ca060 0x55a1cea59960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e50db3f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a1ce9b2280 con 0x55a1ce983ff0 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e50db3f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a1ce9c8d70 con 0x55a1ce9ca060 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e50db3f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55a1cea52bd0 con 0x55a1ce9862b0 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce983ff0 0x55a1cea509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce983ff0 0x55a1cea509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44882/0 (socket says 172.21.15.46:44882) 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 -- 172.21.15.46:0/3157870330 learned_addr learned my addr 172.21.15.46:0/3157870330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4b07c640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce9ca060 0x55a1cea59960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a07a640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1cea59420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 -- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 msgr2=0x55a1cea59420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55a1ce9862b0 0x55a1cea59420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 -- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce9ca060 msgr2=0x55a1cea59960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a1ce9ca060 0x55a1cea59960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.734+0000 7f7e4a87b640 1 -- 172.21.15.46:0/3157870330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1cea53a40 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.735+0000 7f7e4a87b640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce983ff0 0x55a1cea509e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0087e0 tx=0x7f7e3c0088d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.735+0000 7f7e33fff640 1 -- 172.21.15.46:0/3157870330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c019020 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.735+0000 7f7e33fff640 1 -- 172.21.15.46:0/3157870330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e3c015940 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.735+0000 7f7e33fff640 1 -- 172.21.15.46:0/3157870330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c015cf0 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.735+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a1cea57960 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.735+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1ce9ad420 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.736+0000 7f7e33fff640 1 -- 172.21.15.46:0/3157870330 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7e3c01c020 con 0x55a1ce983ff0 2024-05-09T03:16:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.737+0000 7f7e33fff640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e38074ce0 0x7f7e380771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.737+0000 7f7e33fff640 1 -- 172.21.15.46:0/3157870330 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f7e3c096c60 con 0x55a1ce983ff0 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.737+0000 7f7e4a07a640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e38074ce0 0x7f7e380771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.737+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55a1ceaee350 0x55a1cea57e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.737+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55a1cea53c50 con 0x55a1ceaee350 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.737+0000 7f7e4b07c640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55a1ceaee350 0x55a1cea57e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e4a07a640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e38074ce0 0x7f7e380771f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55a1ce9c8380 tx=0x7f7e400018d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e4b07c640 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55a1ceaee350 0x55a1cea57e40 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e4b07c640 1 -- 172.21.15.46:0/3157870330 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 100.00000000 [omap-get-keys out=155b] v0'0 uv1 ondisk = 0) ==== 156+0+155 (crc 0 0 0) 0x7f7e44052b30 con 0x55a1ceaee350 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray0_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray1_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray2_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray3_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray4_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray5_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray6_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray7_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray8_head 2024-05-09T03:16:10.744 INFO:teuthology.orchestra.run.smithi046.stdout:stray9_head 2024-05-09T03:16:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55a1ceaee350 msgr2=0x55a1cea57e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55a1ceaee350 0x55a1cea57e40 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e38074ce0 msgr2=0x7f7e380771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e38074ce0 0x7f7e380771f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x55a1ce9c8380 tx=0x7f7e400018d0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce983ff0 msgr2=0x55a1cea509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 --2- 172.21.15.46:0/3157870330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a1ce983ff0 0x55a1cea509e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0087e0 tx=0x7f7e3c0088d0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e4b07c640 1 -- 172.21.15.46:0/3157870330 reap_dead start 2024-05-09T03:16:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 shutdown_connections 2024-05-09T03:16:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 >> 172.21.15.46:0/3157870330 conn(0x55a1ce8a5ac0 msgr2=0x55a1ce9c8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 shutdown_connections 2024-05-09T03:16:10.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.738+0000 7f7e50db3f40 1 -- 172.21.15.46:0/3157870330 wait complete. 2024-05-09T03:16:10.747 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_sessionmap 2024-05-09T03:16:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.802+0000 7f88c2fc9f40 1 Processor -- start 2024-05-09T03:16:10.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.802+0000 7f88c2fc9f40 1 -- start start 2024-05-09T03:16:10.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88c2fc9f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x557494657b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88c2fc9f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946563f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88c2fc9f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494655920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88c2fc9f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5574945ddcb0 con 0x557494673780 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88c2fc9f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557494544f50 con 0x5574945cadb0 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88c2fc9f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55749459bbf0 con 0x55749459b7e0 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88bd292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494655920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88bd292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494655920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44892/0 (socket says 172.21.15.46:44892) 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.803+0000 7f88bd292640 1 -- 172.21.15.46:0/1789190293 learned_addr learned my addr 172.21.15.46:0/1789190293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bc290640 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bca91640 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x557494657b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bd292640 1 -- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 msgr2=0x557494657b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bd292640 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x557494657b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bd292640 1 -- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 msgr2=0x5574946563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bd292640 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bd292640 1 -- 172.21.15.46:0/1789190293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574945e1ab0 con 0x557494673780 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bca91640 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x557494657b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bd292640 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494655920 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f88b4001ca0 tx=0x7f88b400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be9d27c784d4e3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bb28e640 1 -- 172.21.15.46:0/1789190293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b400bcc0 con 0x557494673780 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bb28e640 1 -- 172.21.15.46:0/1789190293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b400f400 con 0x557494673780 2024-05-09T03:16:10.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.804+0000 7f88bb28e640 1 -- 172.21.15.46:0/1789190293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b400be60 con 0x557494673780 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/1789190293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 msgr2=0x557494655920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494655920 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f88b4001ca0 tx=0x7f88b400b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/1789190293 shutdown_connections 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494655920 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/1789190293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x557494657b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/1789190293 >> 172.21.15.46:0/1789190293 conn(0x5574944c0ac0 msgr2=0x5574945e3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/1789190293 shutdown_connections 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.805+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/1789190293 wait complete. 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.806+0000 7f88c2fc9f40 1 Processor -- start 2024-05-09T03:16:10.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.806+0000 7f88c2fc9f40 1 -- start start 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88c2fc9f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x5574947090e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88c2fc9f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88c2fc9f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494671b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88c2fc9f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5574945c8970 con 0x557494673780 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88c2fc9f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557494709c80 con 0x5574945cadb0 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88c2fc9f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5574945cc8d0 con 0x55749459b7e0 2024-05-09T03:16:10.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494671b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494671b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44904/0 (socket says 172.21.15.46:44904) 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 -- 172.21.15.46:0/2790183123 learned_addr learned my addr 172.21.15.46:0/2790183123 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bc290640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bca91640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x5574947090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 -- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 msgr2=0x5574947090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55749459b7e0 0x5574947090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 -- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 msgr2=0x5574946598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5574945cadb0 0x5574946598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.807+0000 7f88bd292640 1 -- 172.21.15.46:0/2790183123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557494656930 con 0x557494673780 2024-05-09T03:16:10.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.808+0000 7f88bd292640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494671b70 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f88b4001c70 tx=0x7f88b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.808+0000 7f88ba28c640 1 -- 172.21.15.46:0/2790183123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4020070 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.808+0000 7f88ba28c640 1 -- 172.21.15.46:0/2790183123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88b401c640 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.808+0000 7f88ba28c640 1 -- 172.21.15.46:0/2790183123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b401ca10 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.808+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5574945ccb20 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.808+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557494658dd0 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.810+0000 7f88ba28c640 1 -- 172.21.15.46:0/2790183123 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f88b4023070 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.810+0000 7f88ba28c640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88a8074cb0 0x7f88a80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.810+0000 7f88ba28c640 1 -- 172.21.15.46:0/2790183123 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f88b4097ec0 con 0x557494673780 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.810+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5574947012d0 0x557494715700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.810+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x557494659df0 con 0x5574947012d0 2024-05-09T03:16:10.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88bca91640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88a8074cb0 0x7f88a80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88bc290640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5574947012d0 0x557494715700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88bc290640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5574947012d0 0x557494715700 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88bca91640 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88a8074cb0 0x7f88a80771c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f88ac056000 tx=0x7f88ac05c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88bc290640 1 -- 172.21.15.46:0/2790183123 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 159+0+5 (crc 0 0 0) 0x7f88b00099b0 con 0x5574947012d0 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5574947012d0 msgr2=0x557494715700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.811+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5574947012d0 0x557494715700 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88a8074cb0 msgr2=0x7f88a80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88a8074cb0 0x7f88a80771c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f88ac056000 tx=0x7f88ac05c040 comp rx=0 tx=0).stop 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 msgr2=0x557494671b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 --2- 172.21.15.46:0/2790183123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557494673780 0x557494671b70 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f88b4001c70 tx=0x7f88b4000bc0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88bd292640 1 -- 172.21.15.46:0/2790183123 reap_dead start 2024-05-09T03:16:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 shutdown_connections 2024-05-09T03:16:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 >> 172.21.15.46:0/2790183123 conn(0x5574944c0ac0 msgr2=0x5574945c7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 shutdown_connections 2024-05-09T03:16:10.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.812+0000 7f88c2fc9f40 1 -- 172.21.15.46:0/2790183123 wait complete. 2024-05-09T03:16:10.819 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 400.00000000 2024-05-09T03:16:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.874+0000 7fdb5012ff40 1 Processor -- start 2024-05-09T03:16:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.875+0000 7fdb5012ff40 1 -- start start 2024-05-09T03:16:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb5012ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 0x5649ad587940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb5012ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad5045f0 0x5649ad4e6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb5012ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad4e6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb5012ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5649ad462f50 con 0x5649ad4fc3a0 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb5012ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5649ad466f90 con 0x5649ad5045f0 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb5012ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5649ad5058d0 con 0x5649ad590150 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 0x5649ad587940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 0x5649ad587940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44912/0 (socket says 172.21.15.46:44912) 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 -- 172.21.15.46:0/2904883396 learned_addr learned my addr 172.21.15.46:0/2904883396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb4a3fa640 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad4e6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb493f8640 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad5045f0 0x5649ad4e6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 -- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 msgr2=0x5649ad4e6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad4e6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 -- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad5045f0 msgr2=0x5649ad4e6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad5045f0 0x5649ad4e6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 -- 172.21.15.46:0/2904883396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ad4e6f80 con 0x5649ad4fc3a0 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb493f8640 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad5045f0 0x5649ad4e6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb49bf9640 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 0x5649ad587940 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fdb44001220 tx=0x7fdb4400b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5079b099888cf204 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.876+0000 7fdb43fff640 1 -- 172.21.15.46:0/2904883396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44012020 con 0x5649ad4fc3a0 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb43fff640 1 -- 172.21.15.46:0/2904883396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4400e690 con 0x5649ad4fc3a0 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb43fff640 1 -- 172.21.15.46:0/2904883396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4400eac0 con 0x5649ad4fc3a0 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 -- 172.21.15.46:0/2904883396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 msgr2=0x5649ad587940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 0x5649ad587940 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fdb44001220 tx=0x7fdb4400b800 comp rx=0 tx=0).stop 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 -- 172.21.15.46:0/2904883396 shutdown_connections 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad4e6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad5045f0 0x5649ad4e6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/2904883396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad4fc3a0 0x5649ad587940 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 -- 172.21.15.46:0/2904883396 >> 172.21.15.46:0/2904883396 conn(0x5649ad3deac0 msgr2=0x5649ad4e7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 -- 172.21.15.46:0/2904883396 shutdown_connections 2024-05-09T03:16:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.877+0000 7fdb5012ff40 1 -- 172.21.15.46:0/2904883396 wait complete. 2024-05-09T03:16:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.878+0000 7fdb5012ff40 1 Processor -- start 2024-05-09T03:16:10.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.878+0000 7fdb5012ff40 1 -- start start 2024-05-09T03:16:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.879+0000 7fdb5012ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad4fc3a0 0x5649ad585bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.879+0000 7fdb5012ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad5045f0 0x5649ad58bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.879+0000 7fdb5012ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad58c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.879+0000 7fdb5012ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5649ad58af20 con 0x5649ad5045f0 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.879+0000 7fdb5012ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5649ad590560 con 0x5649ad4fc3a0 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.879+0000 7fdb5012ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5649ad586520 con 0x5649ad590150 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad5045f0 0x5649ad58bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad5045f0 0x5649ad58bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44920/0 (socket says 172.21.15.46:44920) 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 -- 172.21.15.46:0/1755630326 learned_addr learned my addr 172.21.15.46:0/1755630326 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb49bf9640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad4fc3a0 0x5649ad585bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb4a3fa640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad58c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 -- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 msgr2=0x5649ad58c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5649ad590150 0x5649ad58c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 -- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad4fc3a0 msgr2=0x5649ad585bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5649ad4fc3a0 0x5649ad585bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 -- 172.21.15.46:0/1755630326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649ad4e59f0 con 0x5649ad5045f0 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb493f8640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad5045f0 0x5649ad58bef0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fdb340259e0 tx=0x7fdb34002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb42ffd640 1 -- 172.21.15.46:0/1755630326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34065020 con 0x5649ad5045f0 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb42ffd640 1 -- 172.21.15.46:0/1755630326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb34054ae0 con 0x5649ad5045f0 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5649ad621340 con 0x5649ad5045f0 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb42ffd640 1 -- 172.21.15.46:0/1755630326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34059040 con 0x5649ad5045f0 2024-05-09T03:16:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.880+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649ad4e6c90 con 0x5649ad5045f0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.882+0000 7fdb42ffd640 1 -- 172.21.15.46:0/1755630326 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdb34068020 con 0x5649ad5045f0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.882+0000 7fdb42ffd640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdb2c074ce0 0x7fdb2c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb42ffd640 1 -- 172.21.15.46:0/1755630326 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fdb340d9e70 con 0x5649ad5045f0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5649ad4ff690 0x5649ad58ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb49bf9640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdb2c074ce0 0x7fdb2c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5649ad58b5e0 con 0x5649ad4ff690 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb4a3fa640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5649ad4ff690 0x5649ad58ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb49bf9640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdb2c074ce0 0x7fdb2c0771f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdb44001220 tx=0x7fdb44009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.883+0000 7fdb4a3fa640 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5649ad4ff690 0x5649ad58ed80 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb4a3fa640 1 -- 172.21.15.46:0/1755630326 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 400.00000000 [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7fdb3c0024e0 con 0x5649ad4ff690 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5649ad4ff690 msgr2=0x5649ad58ed80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5649ad4ff690 0x5649ad58ed80 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdb2c074ce0 msgr2=0x7fdb2c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdb2c074ce0 0x7fdb2c0771f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdb44001220 tx=0x7fdb44009000 comp rx=0 tx=0).stop 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad5045f0 msgr2=0x5649ad58bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 --2- 172.21.15.46:0/1755630326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5649ad5045f0 0x5649ad58bef0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fdb340259e0 tx=0x7fdb34002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb4a3fa640 1 -- 172.21.15.46:0/1755630326 reap_dead start 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 shutdown_connections 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 >> 172.21.15.46:0/1755630326 conn(0x5649ad3deac0 msgr2=0x5649ad5882a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 shutdown_connections 2024-05-09T03:16:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.884+0000 7fdb5012ff40 1 -- 172.21.15.46:0/1755630326 wait complete. 2024-05-09T03:16:10.893 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000.inode 2024-05-09T03:16:10.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.947+0000 7f639631df40 1 Processor -- start 2024-05-09T03:16:10.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.947+0000 7f639631df40 1 -- start start 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.948+0000 7f639631df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a0cc7b0 0x558e5a170430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.948+0000 7f639631df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0e3010 0x558e5a170970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.948+0000 7f639631df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 0x558e5a165800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.948+0000 7f639631df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558e5a0e14d0 con 0x558e5a176d10 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.948+0000 7f639631df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e5a043f50 con 0x558e5a0cc7b0 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.948+0000 7f639631df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558e5a0e68a0 con 0x558e5a0e3010 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 0x558e5a165800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 0x558e5a165800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44940/0 (socket says 172.21.15.46:44940) 2024-05-09T03:16:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 -- 172.21.15.46:0/33255176 learned_addr learned my addr 172.21.15.46:0/33255176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f638fde5640 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a0cc7b0 0x558e5a170430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f638f5e4640 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0e3010 0x558e5a170970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 -- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0e3010 msgr2=0x558e5a170970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0e3010 0x558e5a170970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 -- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a0cc7b0 msgr2=0x558e5a170430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a0cc7b0 0x558e5a170430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 -- 172.21.15.46:0/33255176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e5a166580 con 0x558e5a176d10 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f63905e6640 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 0x558e5a165800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f638800a590 tx=0x7f638800a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e9794bec91bed5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f638e5e2640 1 -- 172.21.15.46:0/33255176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388018070 con 0x558e5a176d10 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f638e5e2640 1 -- 172.21.15.46:0/33255176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638800ae50 con 0x558e5a176d10 2024-05-09T03:16:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.949+0000 7f638e5e2640 1 -- 172.21.15.46:0/33255176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388007820 con 0x558e5a176d10 2024-05-09T03:16:10.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 -- 172.21.15.46:0/33255176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 msgr2=0x558e5a165800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 0x558e5a165800 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f638800a590 tx=0x7f638800a680 comp rx=0 tx=0).stop 2024-05-09T03:16:10.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 -- 172.21.15.46:0/33255176 shutdown_connections 2024-05-09T03:16:10.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a176d10 0x558e5a165800 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0e3010 0x558e5a170970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 --2- 172.21.15.46:0/33255176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a0cc7b0 0x558e5a170430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 -- 172.21.15.46:0/33255176 >> 172.21.15.46:0/33255176 conn(0x558e59fbfac0 msgr2=0x558e5a0dbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 -- 172.21.15.46:0/33255176 shutdown_connections 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.950+0000 7f639631df40 1 -- 172.21.15.46:0/33255176 wait complete. 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.951+0000 7f639631df40 1 Processor -- start 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.951+0000 7f639631df40 1 -- start start 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f639631df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0cc7b0 0x558e5a169420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f639631df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a0e3010 0x558e5a164bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f639631df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a176d10 0x558e5a0df9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f639631df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558e5a16d450 con 0x558e5a0e3010 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f639631df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e5a16ddf0 con 0x558e5a176d10 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f639631df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558e5a0ded90 con 0x558e5a0cc7b0 2024-05-09T03:16:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f638f5e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a0e3010 0x558e5a164bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f638f5e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a0e3010 0x558e5a164bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44950/0 (socket says 172.21.15.46:44950) 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.952+0000 7f638f5e4640 1 -- 172.21.15.46:0/1693149442 learned_addr learned my addr 172.21.15.46:0/1693149442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f63905e6640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a176d10 0x558e5a0df9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638fde5640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0cc7b0 0x558e5a169420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638f5e4640 1 -- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0cc7b0 msgr2=0x558e5a169420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638f5e4640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558e5a0cc7b0 0x558e5a169420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638f5e4640 1 -- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a176d10 msgr2=0x558e5a0df9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638f5e4640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e5a176d10 0x558e5a0df9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638f5e4640 1 -- 172.21.15.46:0/1693149442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e5a166eb0 con 0x558e5a0e3010 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638f5e4640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a0e3010 0x558e5a164bb0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f638400ee40 tx=0x7f638400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638d5e0640 1 -- 172.21.15.46:0/1693149442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638400cea0 con 0x558e5a0e3010 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638d5e0640 1 -- 172.21.15.46:0/1693149442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6384013070 con 0x558e5a0e3010 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558e5a166130 con 0x558e5a0e3010 2024-05-09T03:16:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f638d5e0640 1 -- 172.21.15.46:0/1693149442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638400fa60 con 0x558e5a0e3010 2024-05-09T03:16:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.953+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e5a0e08d0 con 0x558e5a0e3010 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.955+0000 7f638d5e0640 1 -- 172.21.15.46:0/1693149442 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6384009070 con 0x558e5a0e3010 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.955+0000 7f638d5e0640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f637c074ce0 0x7f637c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.955+0000 7f638d5e0640 1 -- 172.21.15.46:0/1693149442 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f63840969e0 con 0x558e5a0e3010 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.955+0000 7f639631df40 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558e5a0de190 0x558e5a216940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.955+0000 7f638fde5640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f637c074ce0 0x7f637c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.955+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x558e5a0de5a0 con 0x558e5a0de190 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.956+0000 7f63905e6640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558e5a0de190 0x558e5a216940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.956+0000 7f638fde5640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f637c074ce0 0x7f637c0771f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6380057000 tx=0x7f638005d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.956+0000 7f63905e6640 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558e5a0de190 0x558e5a216940 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f63905e6640 1 -- 172.21.15.46:0/1693149442 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 162+0+5 (crc 0 0 0) 0x7f63880078c0 con 0x558e5a0de190 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558e5a0de190 msgr2=0x558e5a216940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558e5a0de190 0x558e5a216940 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f637c074ce0 msgr2=0x7f637c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f637c074ce0 0x7f637c0771f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6380057000 tx=0x7f638005d040 comp rx=0 tx=0).stop 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a0e3010 msgr2=0x558e5a164bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 --2- 172.21.15.46:0/1693149442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558e5a0e3010 0x558e5a164bb0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f638400ee40 tx=0x7f638400c730 comp rx=0 tx=0).stop 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f63905e6640 1 -- 172.21.15.46:0/1693149442 reap_dead start 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 shutdown_connections 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 >> 172.21.15.46:0/1693149442 conn(0x558e59fbfac0 msgr2=0x558e5a09a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 shutdown_connections 2024-05-09T03:16:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:10.957+0000 7f639631df40 1 -- 172.21.15.46:0/1693149442 wait complete. 2024-05-09T03:16:10.961 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000 2024-05-09T03:16:11.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.016+0000 7faec165cf40 1 Processor -- start 2024-05-09T03:16:11.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.016+0000 7faec165cf40 1 -- start start 2024-05-09T03:16:11.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faec165cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acb45d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faec165cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbedd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faec165cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acc81f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faec165cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5653acb5d4d0 con 0x7faea80022f0 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faec165cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5653acabff50 con 0x5653acbee2d0 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faec165cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5653acb628a0 con 0x5653acbe9900 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faebb124640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acb45d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faebb124640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acb45d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44968/0 (socket says 172.21.15.46:44968) 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.017+0000 7faebb124640 1 -- 172.21.15.46:0/3780116243 learned_addr learned my addr 172.21.15.46:0/3780116243 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb124640 1 -- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 msgr2=0x5653acbedd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb124640 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbedd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb124640 1 -- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 msgr2=0x5653acc81f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb925640 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acc81f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb124640 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acc81f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb124640 1 -- 172.21.15.46:0/3780116243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653acbec550 con 0x7faea80022f0 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faebb124640 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acb45d90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7faeac025f40 tx=0x7faeac04f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8eb420736cd657b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faeb9921640 1 -- 172.21.15.46:0/3780116243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac005d00 con 0x7faea80022f0 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faeb9921640 1 -- 172.21.15.46:0/3780116243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeac0557f0 con 0x7faea80022f0 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.018+0000 7faeb9921640 1 -- 172.21.15.46:0/3780116243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac055ba0 con 0x7faea80022f0 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 -- 172.21.15.46:0/3780116243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 msgr2=0x5653acb45d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acb45d90 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7faeac025f40 tx=0x7faeac04f000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 -- 172.21.15.46:0/3780116243 shutdown_connections 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acc81f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbedd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 --2- 172.21.15.46:0/3780116243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acb45d90 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 -- 172.21.15.46:0/3780116243 >> 172.21.15.46:0/3780116243 conn(0x5653aca3bac0 msgr2=0x5653acb61480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 -- 172.21.15.46:0/3780116243 shutdown_connections 2024-05-09T03:16:11.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.019+0000 7faec165cf40 1 -- 172.21.15.46:0/3780116243 wait complete. 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.020+0000 7faec165cf40 1 Processor -- start 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.020+0000 7faec165cf40 1 -- start start 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faec165cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbeb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faec165cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acb1ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faec165cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acbdc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faec165cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5653acbec830 con 0x7faea80022f0 2024-05-09T03:16:11.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faec165cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5653acbecb10 con 0x5653acbee2d0 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faec165cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5653acc87ed0 con 0x5653acbe9900 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acbdc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acbdc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44970/0 (socket says 172.21.15.46:44970) 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 -- 172.21.15.46:0/1416114403 learned_addr learned my addr 172.21.15.46:0/1416114403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faeba923640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acb1ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb124640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbeb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 -- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 msgr2=0x5653acbeb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbeb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 -- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 msgr2=0x5653acb1ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acb1ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb925640 1 -- 172.21.15.46:0/1416114403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653acbebe90 con 0x7faea80022f0 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faeba923640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653acbee2d0 0x5653acb1ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.021+0000 7faebb124640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653acbe9900 0x5653acbeb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:11.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.022+0000 7faebb925640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acbdc930 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7faeb4001330 tx=0x7faeb4006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.022+0000 7faeb891f640 1 -- 172.21.15.46:0/1416114403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4019070 con 0x7faea80022f0 2024-05-09T03:16:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.022+0000 7faeb891f640 1 -- 172.21.15.46:0/1416114403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb4003e30 con 0x7faea80022f0 2024-05-09T03:16:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.022+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5653acbec170 con 0x7faea80022f0 2024-05-09T03:16:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.022+0000 7faeb891f640 1 -- 172.21.15.46:0/1416114403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeb4004220 con 0x7faea80022f0 2024-05-09T03:16:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.022+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653acb1a630 con 0x7faea80022f0 2024-05-09T03:16:11.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.023+0000 7faeb891f640 1 -- 172.21.15.46:0/1416114403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faeb401c070 con 0x7faea80022f0 2024-05-09T03:16:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.024+0000 7faeb891f640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fae9c074ce0 0x7fae9c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.024+0000 7faeb891f640 1 -- 172.21.15.46:0/1416114403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7faeb4095370 con 0x7faea80022f0 2024-05-09T03:16:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.024+0000 7faebb124640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fae9c074ce0 0x7fae9c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.024+0000 7faec165cf40 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5653acb462a0 0x5653acc87060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.024+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5653acc875a0 con 0x5653acb462a0 2024-05-09T03:16:11.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.024+0000 7faeba923640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5653acb462a0 0x5653acc87060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faebb124640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fae9c074ce0 0x7fae9c0771f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7faeac025f40 tx=0x7faeac069040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faeba923640 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5653acb462a0 0x5653acc87060 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faeba923640 1 -- 172.21.15.46:0/1416114403 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 1.00000000 [omap-get-keys out=20b] v0'0 uv6 ondisk = 0) ==== 154+0+20 (crc 0 0 0) 0x7faea40099c0 con 0x5653acb462a0 2024-05-09T03:16:11.028 INFO:teuthology.orchestra.run.smithi046.stdout:subdir_head 2024-05-09T03:16:11.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5653acb462a0 msgr2=0x5653acc87060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5653acb462a0 0x5653acc87060 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fae9c074ce0 msgr2=0x7fae9c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fae9c074ce0 0x7fae9c0771f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7faeac025f40 tx=0x7faeac069040 comp rx=0 tx=0).stop 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 msgr2=0x5653acbdc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 --2- 172.21.15.46:0/1416114403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faea80022f0 0x5653acbdc930 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7faeb4001330 tx=0x7faeb4006f40 comp rx=0 tx=0).stop 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faebb925640 1 -- 172.21.15.46:0/1416114403 reap_dead start 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 shutdown_connections 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.025+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 >> 172.21.15.46:0/1416114403 conn(0x5653aca3bac0 msgr2=0x5653acb5ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.026+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 shutdown_connections 2024-05-09T03:16:11.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.026+0000 7faec165cf40 1 -- 172.21.15.46:0/1416114403 wait complete. 2024-05-09T03:16:11.031 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-05-09T03:16:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.085+0000 7ff32b91af40 1 Processor -- start 2024-05-09T03:16:11.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.085+0000 7ff32b91af40 1 -- start start 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32b91af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6bde9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32b91af40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bde7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32b91af40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6bf228d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32b91af40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bc6be055f0 con 0x55bc6be8a670 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32b91af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bc6bd62f50 con 0x7ff314003630 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32b91af40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bc6be058d0 con 0x55bc6be87690 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6bf228d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6bf228d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44996/0 (socket says 172.21.15.46:44996) 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 -- 172.21.15.46:0/929860780 learned_addr learned my addr 172.21.15.46:0/929860780 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32499e640 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6bde9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff31ffff640 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bde7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 -- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 msgr2=0x55bc6bde7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bde7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 -- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 msgr2=0x55bc6bde9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6bde9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 -- 172.21.15.46:0/929860780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc6bdea530 con 0x55bc6be8a670 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff32519f640 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6bf228d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff320025f90 tx=0x7ff32004f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613d870a97e51486 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff31effd640 1 -- 172.21.15.46:0/929860780 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32000a580 con 0x55bc6be8a670 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff31effd640 1 -- 172.21.15.46:0/929860780 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32000e000 con 0x55bc6be8a670 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.087+0000 7ff31effd640 1 -- 172.21.15.46:0/929860780 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32004dcb0 con 0x55bc6be8a670 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 -- 172.21.15.46:0/929860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 msgr2=0x55bc6bf228d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6bf228d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff320025f90 tx=0x7ff32004f000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 -- 172.21.15.46:0/929860780 shutdown_connections 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6bf228d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bde7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 --2- 172.21.15.46:0/929860780 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6bde9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 -- 172.21.15.46:0/929860780 >> 172.21.15.46:0/929860780 conn(0x55bc6bcdeac0 msgr2=0x55bc6bdeb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 -- 172.21.15.46:0/929860780 shutdown_connections 2024-05-09T03:16:11.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.088+0000 7ff32b91af40 1 -- 172.21.15.46:0/929860780 wait complete. 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.089+0000 7ff32b91af40 1 Processor -- start 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.089+0000 7ff32b91af40 1 -- start start 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff32b91af40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bf21ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff32b91af40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6be8c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff32b91af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6be8ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff32b91af40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bc6bde6bb0 con 0x55bc6be8a670 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff32b91af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bc6bf210f0 con 0x7ff314003630 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff32b91af40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bc6be8d540 con 0x55bc6be87690 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff31ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6be8c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff31ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6be8c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45006/0 (socket says 172.21.15.46:45006) 2024-05-09T03:16:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.090+0000 7ff31ffff640 1 -- 172.21.15.46:0/3773588912 learned_addr learned my addr 172.21.15.46:0/3773588912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff32519f640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6be8ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff32499e640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bf21ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31ffff640 1 -- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 msgr2=0x55bc6bf21ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31ffff640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bc6be87690 0x55bc6bf21ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31ffff640 1 -- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 msgr2=0x55bc6be8ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31ffff640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff314003630 0x55bc6be8ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31ffff640 1 -- 172.21.15.46:0/3773588912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc6be8d790 con 0x55bc6be8a670 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31ffff640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6be8c910 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff3100057f0 tx=0x7ff3100084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31dffb640 1 -- 172.21.15.46:0/3773588912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31001a070 con 0x55bc6be8a670 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31dffb640 1 -- 172.21.15.46:0/3773588912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff310016610 con 0x55bc6be8a670 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bc6be8da70 con 0x55bc6be8a670 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff31dffb640 1 -- 172.21.15.46:0/3773588912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3100169c0 con 0x55bc6be8a670 2024-05-09T03:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.091+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc6bf20e30 con 0x55bc6be8a670 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.093+0000 7ff31dffb640 1 -- 172.21.15.46:0/3773588912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff31001d070 con 0x55bc6be8a670 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.093+0000 7ff31dffb640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff308074ce0 0x7ff3080771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.093+0000 7ff31dffb640 1 -- 172.21.15.46:0/3773588912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7ff310096a80 con 0x55bc6be8a670 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.093+0000 7ff32b91af40 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55bc6be8e6d0 0x55bc6be85630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.093+0000 7ff32499e640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff308074ce0 0x7ff3080771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.093+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55bc6be85b70 con 0x55bc6be8e6d0 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32519f640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55bc6be8e6d0 0x55bc6be85630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32519f640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55bc6be8e6d0 0x55bc6be85630 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32499e640 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff308074ce0 0x7ff3080771f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff3180017d0 tx=0x7ff318009270 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32519f640 1 -- 172.21.15.46:0/3773588912 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x7ff32004da90 con 0x55bc6be8e6d0 2024-05-09T03:16:11.097 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-05-09T03:16:08.000000+0000, size 0 2024-05-09T03:16:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55bc6be8e6d0 msgr2=0x55bc6be85630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55bc6be8e6d0 0x55bc6be85630 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff308074ce0 msgr2=0x7ff3080771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff308074ce0 0x7ff3080771f0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff3180017d0 tx=0x7ff318009270 comp rx=0 tx=0).stop 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 msgr2=0x55bc6be8c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 --2- 172.21.15.46:0/3773588912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc6be8a670 0x55bc6be8c910 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff3100057f0 tx=0x7ff3100084c0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32519f640 1 -- 172.21.15.46:0/3773588912 reap_dead start 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 shutdown_connections 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 >> 172.21.15.46:0/3773588912 conn(0x55bc6bcdeac0 msgr2=0x55bc6be87aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 shutdown_connections 2024-05-09T03:16:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.094+0000 7ff32b91af40 1 -- 172.21.15.46:0/3773588912 wait complete. 2024-05-09T03:16:11.099 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 600.00000000 2024-05-09T03:16:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.154+0000 7fbb650f4f40 1 Processor -- start 2024-05-09T03:16:11.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.154+0000 7fbb650f4f40 1 -- start start 2024-05-09T03:16:11.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb650f4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38881ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb650f4f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e3887915f0 0x55e388776d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb650f4f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 0x55e388777250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb650f4f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e3886eff50 con 0x55e38881ca40 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb650f4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e3886f3f90 con 0x55e3887893a0 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb650f4f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e3887928d0 con 0x55e3887915f0 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb5f3bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 0x55e388777250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb5f3bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 0x55e388777250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45028/0 (socket says 172.21.15.46:45028) 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb5f3bd640 1 -- 172.21.15.46:0/484368642 learned_addr learned my addr 172.21.15.46:0/484368642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb5ebbc640 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38881ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.155+0000 7fbb5e3bb640 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e3887915f0 0x55e388776d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5f3bd640 1 -- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e3887915f0 msgr2=0x55e388776d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5f3bd640 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e3887915f0 0x55e388776d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5f3bd640 1 -- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 msgr2=0x55e38881ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5f3bd640 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38881ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5f3bd640 1 -- 172.21.15.46:0/484368642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38874a6b0 con 0x55e38881ca40 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5f3bd640 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 0x55e388777250 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fbb58001d00 tx=0x7fbb5800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfed18410e0c4d1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5d3b9640 1 -- 172.21.15.46:0/484368642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800bd80 con 0x55e38881ca40 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5d3b9640 1 -- 172.21.15.46:0/484368642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb58001e50 con 0x55e38881ca40 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.156+0000 7fbb5d3b9640 1 -- 172.21.15.46:0/484368642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800f640 con 0x55e38881ca40 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 -- 172.21.15.46:0/484368642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 msgr2=0x55e388777250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 0x55e388777250 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fbb58001d00 tx=0x7fbb5800b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 -- 172.21.15.46:0/484368642 shutdown_connections 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e38881ca40 0x55e388777250 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e3887915f0 0x55e388776d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/484368642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38881ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 -- 172.21.15.46:0/484368642 >> 172.21.15.46:0/484368642 conn(0x55e38866bac0 msgr2=0x55e38878c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 -- 172.21.15.46:0/484368642 shutdown_connections 2024-05-09T03:16:11.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.157+0000 7fbb650f4f40 1 -- 172.21.15.46:0/484368642 wait complete. 2024-05-09T03:16:11.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.158+0000 7fbb650f4f40 1 Processor -- start 2024-05-09T03:16:11.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.158+0000 7fbb650f4f40 1 -- start start 2024-05-09T03:16:11.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb650f4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38880ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb650f4f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3887915f0 0x55e3888127c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb650f4f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e38881ca40 0x55e38881df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb650f4f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e3888143f0 con 0x55e3887915f0 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb650f4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e3888132b0 con 0x55e3887893a0 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb650f4f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e388815840 con 0x55e38881ca40 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3887915f0 0x55e3888127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3887915f0 0x55e3888127c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45040/0 (socket says 172.21.15.46:45040) 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 -- 172.21.15.46:0/2782957226 learned_addr learned my addr 172.21.15.46:0/2782957226 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5ebbc640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38880ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5f3bd640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e38881ca40 0x55e38881df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 -- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e38881ca40 msgr2=0x55e38881df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e38881ca40 0x55e38881df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 -- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 msgr2=0x55e38880ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e3887893a0 0x55e38880ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.159+0000 7fbb5e3bb640 1 -- 172.21.15.46:0/2782957226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38881b000 con 0x55e3887915f0 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.160+0000 7fbb5e3bb640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3887915f0 0x55e3888127c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c4b0 tx=0x7fbb4800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.160+0000 7fbb57fff640 1 -- 172.21.15.46:0/2782957226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48008070 con 0x55e3887915f0 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.160+0000 7fbb57fff640 1 -- 172.21.15.46:0/2782957226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb48011680 con 0x55e3887915f0 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.160+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e38881b2e0 con 0x55e3887915f0 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.160+0000 7fbb57fff640 1 -- 172.21.15.46:0/2782957226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48011a50 con 0x55e3887915f0 2024-05-09T03:16:11.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.160+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e388812ff0 con 0x55e3887915f0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb57fff640 1 -- 172.21.15.46:0/2782957226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb48011bf0 con 0x55e3887915f0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb57fff640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb40074ce0 0x7fbb400771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb57fff640 1 -- 172.21.15.46:0/2782957226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fbb48096920 con 0x55e3887915f0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb5ebbc640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb40074ce0 0x7fbb400771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e388814f30 0x55e38880f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55e38880fb10 con 0x55e388814f30 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.162+0000 7fbb5f3bd640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e388814f30 0x55e38880f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb5f3bd640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e388814f30 0x55e38880f5d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb5ebbc640 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb40074ce0 0x7fbb400771f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbb50057000 tx=0x7fbb5005a040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb5f3bd640 1 -- 172.21.15.46:0/2782957226 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 600.00000000 [stat out=16b] v0'0 uv2 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7fbb5801d4d0 con 0x55e388814f30 2024-05-09T03:16:11.170 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/600.00000000 mtime 2024-05-09T03:15:53.000000+0000, size 0 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e388814f30 msgr2=0x55e38880f5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e388814f30 0x55e38880f5d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb40074ce0 msgr2=0x7fbb400771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb40074ce0 0x7fbb400771f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbb50057000 tx=0x7fbb5005a040 comp rx=0 tx=0).stop 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3887915f0 msgr2=0x55e3888127c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 --2- 172.21.15.46:0/2782957226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3887915f0 0x55e3888127c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800c4b0 tx=0x7fbb4800c8b0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb5f3bd640 1 -- 172.21.15.46:0/2782957226 reap_dead start 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 shutdown_connections 2024-05-09T03:16:11.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 >> 172.21.15.46:0/2782957226 conn(0x55e38866bac0 msgr2=0x55e38874ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 shutdown_connections 2024-05-09T03:16:11.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.163+0000 7fbb650f4f40 1 -- 172.21.15.46:0/2782957226 wait complete. 2024-05-09T03:16:11.173 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000.inode 2024-05-09T03:16:11.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.227+0000 7f1c07e5ef40 1 Processor -- start 2024-05-09T03:16:11.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.227+0000 7f1c07e5ef40 1 -- start start 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c07e5ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 0x560b82e42ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c07e5ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d742b0 0x560b82d9fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c07e5ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82db8060 0x560b82ed7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c07e5ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560b82d72400 con 0x560b82d71ff0 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c07e5ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560b82d17f50 con 0x560b82db8060 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c07e5ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x560b82dbb000 con 0x560b82d742b0 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 0x560b82e42ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 0x560b82e42ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45052/0 (socket says 172.21.15.46:45052) 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 -- 172.21.15.46:0/848419124 learned_addr learned my addr 172.21.15.46:0/848419124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c02127640 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82db8060 0x560b82ed7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 -- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d742b0 msgr2=0x560b82d9fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1bfbfff640 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d742b0 0x560b82d9fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d742b0 0x560b82d9fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 -- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82db8060 msgr2=0x560b82ed7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82db8060 0x560b82ed7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 -- 172.21.15.46:0/848419124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b82dbb6e0 con 0x560b82d71ff0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c02127640 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82db8060 0x560b82ed7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c01926640 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 0x560b82e42ff0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4025cc0 tx=0x7f1bf4051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1ea6f8fa5111de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.229+0000 7f1c00924640 1 -- 172.21.15.46:0/848419124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4054b10 con 0x560b82d71ff0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c00924640 1 -- 172.21.15.46:0/848419124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf4065070 con 0x560b82d71ff0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c00924640 1 -- 172.21.15.46:0/848419124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf405f420 con 0x560b82d71ff0 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/848419124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 msgr2=0x560b82e42ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 0x560b82e42ff0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4025cc0 tx=0x7f1bf4051000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/848419124 shutdown_connections 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82db8060 0x560b82ed7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d742b0 0x560b82d9fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/848419124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82d71ff0 0x560b82e42ff0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/848419124 >> 172.21.15.46:0/848419124 conn(0x560b82c93ac0 msgr2=0x560b82da0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/848419124 shutdown_connections 2024-05-09T03:16:11.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.230+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/848419124 wait complete. 2024-05-09T03:16:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.231+0000 7f1c07e5ef40 1 Processor -- start 2024-05-09T03:16:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.231+0000 7f1c07e5ef40 1 -- start start 2024-05-09T03:16:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.232+0000 7f1c07e5ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d71ff0 0x560b82e3f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.232+0000 7f1c07e5ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82d742b0 0x560b82db3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.232+0000 7f1c07e5ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82db8060 0x560b82db44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.232+0000 7f1c07e5ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560b82da19f0 con 0x560b82db8060 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.232+0000 7f1c07e5ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560b82dbb950 con 0x560b82d742b0 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.232+0000 7f1c07e5ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x560b82ed6840 con 0x560b82d71ff0 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82db8060 0x560b82db44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82db8060 0x560b82db44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45062/0 (socket says 172.21.15.46:45062) 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 -- 172.21.15.46:0/3701356996 learned_addr learned my addr 172.21.15.46:0/3701356996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1bfbfff640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82d742b0 0x560b82db3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c01926640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d71ff0 0x560b82e3f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 -- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d71ff0 msgr2=0x560b82e3f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d71ff0 0x560b82e3f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 -- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82d742b0 msgr2=0x560b82db3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b82d742b0 0x560b82db3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 -- 172.21.15.46:0/3701356996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b82d9f4b0 con 0x560b82db8060 2024-05-09T03:16:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c01926640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b82d71ff0 0x560b82e3f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c02127640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82db8060 0x560b82db44d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00a4a0 tx=0x7f1bfc00a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1bfaffd640 1 -- 172.21.15.46:0/3701356996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc015070 con 0x560b82db8060 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1bfaffd640 1 -- 172.21.15.46:0/3701356996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bfc011890 con 0x560b82db8060 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560b82ed57a0 con 0x560b82db8060 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1bfaffd640 1 -- 172.21.15.46:0/3701356996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc011c80 con 0x560b82db8060 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.233+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b82ed5d60 con 0x560b82db8060 2024-05-09T03:16:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.235+0000 7f1bfaffd640 1 -- 172.21.15.46:0/3701356996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1bfc018070 con 0x560b82db8060 2024-05-09T03:16:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.235+0000 7f1bfaffd640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1be4074ce0 0x7f1be40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1bfaffd640 1 -- 172.21.15.46:0/3701356996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f1bfc096250 con 0x560b82db8060 2024-05-09T03:16:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560b82d70500 0x560b82d70910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1c01926640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1be4074ce0 0x7f1be40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x560b82ed6a40 con 0x560b82d70500 2024-05-09T03:16:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1bfbfff640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560b82d70500 0x560b82d70910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1c01926640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1be4074ce0 0x7f1be40771f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4025cc0 tx=0x7f1bf4053000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.236+0000 7f1bfbfff640 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560b82d70500 0x560b82d70910 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1bfbfff640 1 -- 172.21.15.46:0/3701356996 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv17 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7f1bec00f5c0 con 0x560b82d70500 2024-05-09T03:16:11.239 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/1.00000000.inode mtime 2024-05-09T03:16:08.000000+0000, size 582 2024-05-09T03:16:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560b82d70500 msgr2=0x560b82d70910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560b82d70500 0x560b82d70910 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1be4074ce0 msgr2=0x7f1be40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1be4074ce0 0x7f1be40771f0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4025cc0 tx=0x7f1bf4053000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82db8060 msgr2=0x560b82db44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 --2- 172.21.15.46:0/3701356996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b82db8060 0x560b82db44d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00a4a0 tx=0x7f1bfc00a960 comp rx=0 tx=0).stop 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c02127640 1 -- 172.21.15.46:0/3701356996 reap_dead start 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 shutdown_connections 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 >> 172.21.15.46:0/3701356996 conn(0x560b82c93ac0 msgr2=0x560b82da0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 shutdown_connections 2024-05-09T03:16:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.237+0000 7f1c07e5ef40 1 -- 172.21.15.46:0/3701356996 wait complete. 2024-05-09T03:16:11.241 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000000 2024-05-09T03:16:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.295+0000 7fb1b1f33f40 1 Processor -- start 2024-05-09T03:16:11.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.295+0000 7fb1b1f33f40 1 -- start start 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1b1f33f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0913a0 0x564b5e07e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1b1f33f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 0x564b5e125be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1b1f33f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e124a40 0x564b5e052470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1b1f33f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564b5dff7f50 con 0x564b5e0995f0 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1b1f33f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564b5dffbf90 con 0x564b5e0913a0 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1b1f33f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564b5e09a8d0 con 0x564b5e124a40 2024-05-09T03:16:11.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 0x564b5e125be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 0x564b5e125be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45086/0 (socket says 172.21.15.46:45086) 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 -- 172.21.15.46:0/354682693 learned_addr learned my addr 172.21.15.46:0/354682693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab9fb640 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0913a0 0x564b5e07e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ac1fc640 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e124a40 0x564b5e052470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 -- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e124a40 msgr2=0x564b5e052470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e124a40 0x564b5e052470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 -- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0913a0 msgr2=0x564b5e07e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0913a0 0x564b5e07e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 -- 172.21.15.46:0/354682693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b5e127520 con 0x564b5e0995f0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1ab1fa640 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 0x564b5e125be0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0001e50 tx=0x7fb1a000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe39331141ae8e77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.297+0000 7fb1aa1f8640 1 -- 172.21.15.46:0/354682693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0008070 con 0x564b5e0995f0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1aa1f8640 1 -- 172.21.15.46:0/354682693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a000d500 con 0x564b5e0995f0 2024-05-09T03:16:11.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1aa1f8640 1 -- 172.21.15.46:0/354682693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a000d950 con 0x564b5e0995f0 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/354682693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 msgr2=0x564b5e125be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 0x564b5e125be0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0001e50 tx=0x7fb1a000b6a0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/354682693 shutdown_connections 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e124a40 0x564b5e052470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e0995f0 0x564b5e125be0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/354682693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0913a0 0x564b5e07e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/354682693 >> 172.21.15.46:0/354682693 conn(0x564b5df73ac0 msgr2=0x564b5e09b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/354682693 shutdown_connections 2024-05-09T03:16:11.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.298+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/354682693 wait complete. 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.299+0000 7fb1b1f33f40 1 Processor -- start 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.299+0000 7fb1b1f33f40 1 -- start start 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1b1f33f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e0913a0 0x564b5e1269c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1b1f33f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0995f0 0x564b5e091f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1b1f33f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e124a40 0x564b5e092470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1b1f33f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564b5e114530 con 0x564b5e124a40 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1b1f33f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564b5e1215a0 con 0x564b5e0995f0 2024-05-09T03:16:11.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1b1f33f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564b5e092ec0 con 0x564b5e0913a0 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1ac1fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e124a40 0x564b5e092470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.300+0000 7fb1ac1fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e124a40 0x564b5e092470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45098/0 (socket says 172.21.15.46:45098) 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 -- 172.21.15.46:0/3096637192 learned_addr learned my addr 172.21.15.46:0/3096637192 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ab1fa640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0995f0 0x564b5e091f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ab9fb640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e0913a0 0x564b5e1269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 -- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e0913a0 msgr2=0x564b5e1269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564b5e0913a0 0x564b5e1269c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 -- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0995f0 msgr2=0x564b5e091f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564b5e0995f0 0x564b5e091f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 -- 172.21.15.46:0/3096637192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b5e1254f0 con 0x564b5e124a40 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1ac1fc640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e124a40 0x564b5e092470 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400dbf0 tx=0x7fb1a400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1a91f6640 1 -- 172.21.15.46:0/3096637192 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a40030d0 con 0x564b5e124a40 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1a91f6640 1 -- 172.21.15.46:0/3096637192 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a4016070 con 0x564b5e124a40 2024-05-09T03:16:11.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564b5e0929b0 con 0x564b5e124a40 2024-05-09T03:16:11.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1a91f6640 1 -- 172.21.15.46:0/3096637192 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4011480 con 0x564b5e124a40 2024-05-09T03:16:11.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.301+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b5e1150e0 con 0x564b5e124a40 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.303+0000 7fb1a91f6640 1 -- 172.21.15.46:0/3096637192 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1a4011620 con 0x564b5e124a40 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.303+0000 7fb1a91f6640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb198074ce0 0x7fb1980771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.303+0000 7fb1a91f6640 1 -- 172.21.15.46:0/3096637192 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fb1a4096240 con 0x564b5e124a40 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.304+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564b5e126f00 0x564b5e1c2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.304+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564b5e1257d0 con 0x564b5e126f00 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.304+0000 7fb1ab9fb640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb198074ce0 0x7fb1980771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.304+0000 7fb1ab1fa640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564b5e126f00 0x564b5e1c2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.304+0000 7fb1ab1fa640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564b5e126f00 0x564b5e1c2050 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.304+0000 7fb1ab9fb640 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb198074ce0 0x7fb1980771f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb19c057000 tx=0x7fb19c05a040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1ab1fa640 1 -- 172.21.15.46:0/3096637192 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000000 [stat out=16b] v0'0 uv20 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7fb1a000d730 con 0x564b5e126f00 2024-05-09T03:16:11.307 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/200.00000000 mtime 2024-05-09T03:16:08.000000+0000, size 90 2024-05-09T03:16:11.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564b5e126f00 msgr2=0x564b5e1c2050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564b5e126f00 0x564b5e1c2050 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb198074ce0 msgr2=0x7fb1980771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb198074ce0 0x7fb1980771f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb19c057000 tx=0x7fb19c05a040 comp rx=0 tx=0).stop 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e124a40 msgr2=0x564b5e092470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 --2- 172.21.15.46:0/3096637192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b5e124a40 0x564b5e092470 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400dbf0 tx=0x7fb1a400b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1ac1fc640 1 -- 172.21.15.46:0/3096637192 reap_dead start 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 shutdown_connections 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 >> 172.21.15.46:0/3096637192 conn(0x564b5df73ac0 msgr2=0x564b5e07e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 shutdown_connections 2024-05-09T03:16:11.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.305+0000 7fb1b1f33f40 1 -- 172.21.15.46:0/3096637192 wait complete. 2024-05-09T03:16:11.309 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000001 2024-05-09T03:16:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.364+0000 7f650b413f40 1 Processor -- start 2024-05-09T03:16:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.364+0000 7f650b413f40 1 -- start start 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f650b413f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6473a0 0x55b88b6d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f650b413f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b649f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f650b413f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 0x55b88b64a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f650b413f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b88b6505f0 con 0x55b88b6db150 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f650b413f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b88b5adf50 con 0x55b88b64f5f0 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f650b413f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b88b6508d0 con 0x55b88b6473a0 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 0x55b88b64a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 0x55b88b64a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45110/0 (socket says 172.21.15.46:45110) 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 -- 172.21.15.46:0/1060708501 learned_addr learned my addr 172.21.15.46:0/1060708501 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504266640 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6473a0 0x55b88b6d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f64fbfff640 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b649f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 -- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6473a0 msgr2=0x55b88b6d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6473a0 0x55b88b6d1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 -- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 msgr2=0x55b88b649f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b649f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 -- 172.21.15.46:0/1060708501 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88b635570 con 0x55b88b6db150 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6504a67640 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 0x55b88b64a480 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00b6c0 tx=0x7f64fc00bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b74871c9105c80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.366+0000 7f6503264640 1 -- 172.21.15.46:0/1060708501 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc016020 con 0x55b88b6db150 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f6503264640 1 -- 172.21.15.46:0/1060708501 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64fc00d7b0 con 0x55b88b6db150 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f6503264640 1 -- 172.21.15.46:0/1060708501 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64fc00db80 con 0x55b88b6db150 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 -- 172.21.15.46:0/1060708501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 msgr2=0x55b88b64a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 0x55b88b64a480 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00b6c0 tx=0x7f64fc00bb80 comp rx=0 tx=0).stop 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 -- 172.21.15.46:0/1060708501 shutdown_connections 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6db150 0x55b88b64a480 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b649f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 --2- 172.21.15.46:0/1060708501 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6473a0 0x55b88b6d1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 -- 172.21.15.46:0/1060708501 >> 172.21.15.46:0/1060708501 conn(0x55b88b529ac0 msgr2=0x55b88b634730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 -- 172.21.15.46:0/1060708501 shutdown_connections 2024-05-09T03:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.367+0000 7f650b413f40 1 -- 172.21.15.46:0/1060708501 wait complete. 2024-05-09T03:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.368+0000 7f650b413f40 1 Processor -- start 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.368+0000 7f650b413f40 1 -- start start 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f650b413f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6473a0 0x55b88b6d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f650b413f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b6dbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f650b413f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6db150 0x55b88b6dc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f650b413f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b88b635d20 con 0x55b88b6473a0 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f650b413f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b88b6d4a70 con 0x55b88b64f5f0 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f650b413f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b88b64a0d0 con 0x55b88b6db150 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f6504266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6473a0 0x55b88b6d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f6504266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6473a0 0x55b88b6d0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45120/0 (socket says 172.21.15.46:45120) 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.369+0000 7f6504266640 1 -- 172.21.15.46:0/2307774439 learned_addr learned my addr 172.21.15.46:0/2307774439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b6dbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504a67640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6db150 0x55b88b6dc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504266640 1 -- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6db150 msgr2=0x55b88b6dc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504266640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6db150 0x55b88b6dc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504266640 1 -- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 msgr2=0x55b88b6dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504266640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b6dbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504266640 1 -- 172.21.15.46:0/2307774439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b88b64a320 con 0x55b88b6473a0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b88b64f5f0 0x55b88b6dbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504a67640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b88b6db150 0x55b88b6dc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6504266640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6473a0 0x55b88b6d0b40 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f64f4002d20 tx=0x7f64f400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6502262640 1 -- 172.21.15.46:0/2307774439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4015020 con 0x55b88b6473a0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6502262640 1 -- 172.21.15.46:0/2307774439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64f400cd30 con 0x55b88b6473a0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b88b6d2c50 con 0x55b88b6473a0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f6502262640 1 -- 172.21.15.46:0/2307774439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f4011670 con 0x55b88b6473a0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.370+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b88b6d47b0 con 0x55b88b6473a0 2024-05-09T03:16:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.372+0000 7f6502262640 1 -- 172.21.15.46:0/2307774439 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f64f4018020 con 0x55b88b6473a0 2024-05-09T03:16:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.372+0000 7f6502262640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64e4074ce0 0x7f64e40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f6502262640 1 -- 172.21.15.46:0/2307774439 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f64f4096470 con 0x55b88b6473a0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f650b413f40 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b88b6d40f0 0x55b88b6d6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55b88b6d6a80 con 0x55b88b6d40f0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64e4074ce0 0x7f64e40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f6504a67640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b88b6d40f0 0x55b88b6d6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f6504a67640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b88b6d40f0 0x55b88b6d6540 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.373+0000 7f64fbfff640 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64e4074ce0 0x7f64e40771f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f64ec058000 tx=0x7f64ec00a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f6504a67640 1 -- 172.21.15.46:0/2307774439 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f64fc00eb10 con 0x55b88b6d40f0 2024-05-09T03:16:11.376 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/200.00000001 mtime 2024-05-09T03:16:08.000000+0000, size 14895 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b88b6d40f0 msgr2=0x55b88b6d6540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b88b6d40f0 0x55b88b6d6540 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64e4074ce0 msgr2=0x7f64e40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64e4074ce0 0x7f64e40771f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f64ec058000 tx=0x7f64ec00a000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6473a0 msgr2=0x55b88b6d0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 --2- 172.21.15.46:0/2307774439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b88b6473a0 0x55b88b6d0b40 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f64f4002d20 tx=0x7f64f400c620 comp rx=0 tx=0).stop 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f6504a67640 1 -- 172.21.15.46:0/2307774439 reap_dead start 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 shutdown_connections 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 >> 172.21.15.46:0/2307774439 conn(0x55b88b529ac0 msgr2=0x55b88b6514e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 shutdown_connections 2024-05-09T03:16:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.374+0000 7f650b413f40 1 -- 172.21.15.46:0/2307774439 wait complete. 2024-05-09T03:16:11.379 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_openfiles.0 2024-05-09T03:16:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.432+0000 7ffa23fc4f40 1 Processor -- start 2024-05-09T03:16:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.433+0000 7ffa23fc4f40 1 -- start start 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa23fc4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff61d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa23fc4f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6129970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa23fc4f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff60a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa23fc4f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562ff60b1cb0 con 0x562ff6147780 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa23fc4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562ff6018f50 con 0x562ff609edb0 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa23fc4f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562ff60b1380 con 0x562ff60a1650 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6129970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6129970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45142/0 (socket says 172.21.15.46:45142) 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 -- 172.21.15.46:0/2324645878 learned_addr learned my addr 172.21.15.46:0/2324645878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa1da8c640 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff61d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa1e28d640 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff60a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 -- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 msgr2=0x562ff60a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff60a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 -- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 msgr2=0x562ff61d5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff61d5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 -- 172.21.15.46:0/2324645878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ff6147fc0 con 0x562ff6147780 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa17fff640 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6129970 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800a430 tx=0x7ffa0800a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d442ed081aa70687 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.434+0000 7ffa1ca8a640 1 -- 172.21.15.46:0/2324645878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0800b8c0 con 0x562ff6147780 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa1ca8a640 1 -- 172.21.15.46:0/2324645878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa0800f030 con 0x562ff6147780 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa1ca8a640 1 -- 172.21.15.46:0/2324645878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0800bca0 con 0x562ff6147780 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/2324645878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 msgr2=0x562ff6129970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6129970 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800a430 tx=0x7ffa0800a8f0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/2324645878 shutdown_connections 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff60a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6129970 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/2324645878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff61d5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/2324645878 >> 172.21.15.46:0/2324645878 conn(0x562ff5f94ac0 msgr2=0x562ff6146120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/2324645878 shutdown_connections 2024-05-09T03:16:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.435+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/2324645878 wait complete. 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.436+0000 7ffa23fc4f40 1 Processor -- start 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.436+0000 7ffa23fc4f40 1 -- start start 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa23fc4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff6143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa23fc4f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff6073870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa23fc4f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6073db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa23fc4f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562ff6148780 con 0x562ff6147780 2024-05-09T03:16:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa23fc4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562ff612c0c0 con 0x562ff609edb0 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa23fc4f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562ff6145920 con 0x562ff60a1650 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa1e28d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6073db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa1e28d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6073db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45146/0 (socket says 172.21.15.46:45146) 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.437+0000 7ffa1e28d640 1 -- 172.21.15.46:0/791841513 learned_addr learned my addr 172.21.15.46:0/791841513 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1da8c640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff6143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa17fff640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff6073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1e28d640 1 -- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 msgr2=0x562ff6073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1e28d640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562ff60a1650 0x562ff6073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1e28d640 1 -- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 msgr2=0x562ff6143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1e28d640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff6143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1e28d640 1 -- 172.21.15.46:0/791841513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ff60a1b00 con 0x562ff6147780 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1da8c640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ff609edb0 0x562ff6143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa1e28d640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6073db0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ffa18054040 tx=0x7ffa18003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa16ffd640 1 -- 172.21.15.46:0/791841513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18055040 con 0x562ff6147780 2024-05-09T03:16:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa16ffd640 1 -- 172.21.15.46:0/791841513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa1805a420 con 0x562ff6147780 2024-05-09T03:16:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa16ffd640 1 -- 172.21.15.46:0/791841513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1805a810 con 0x562ff6147780 2024-05-09T03:16:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562ff6071180 con 0x562ff6147780 2024-05-09T03:16:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.438+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ff60a1320 con 0x562ff6147780 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.440+0000 7ffa16ffd640 1 -- 172.21.15.46:0/791841513 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa18058020 con 0x562ff6147780 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.440+0000 7ffa16ffd640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa00074cb0 0x7ffa000771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa16ffd640 1 -- 172.21.15.46:0/791841513 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7ffa180d9ee0 con 0x562ff6147780 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa1da8c640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa00074cb0 0x7ffa000771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562ff6072040 0x562ff6072450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x562ff6143b40 con 0x562ff6072040 2024-05-09T03:16:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa17fff640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562ff6072040 0x562ff6072450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa1da8c640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa00074cb0 0x7ffa000771c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x562ff6074440 tx=0x7ffa1000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.441+0000 7ffa17fff640 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562ff6072040 0x562ff6072450 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa17fff640 1 -- 172.21.15.46:0/791841513 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_openfiles.0 [stat out=16b] v0'0 uv4 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7ffa0800c040 con 0x562ff6072040 2024-05-09T03:16:11.444 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/mds0_openfiles.0 mtime 2024-05-09T03:16:08.000000+0000, size 0 2024-05-09T03:16:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562ff6072040 msgr2=0x562ff6072450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562ff6072040 0x562ff6072450 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa00074cb0 msgr2=0x7ffa000771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa00074cb0 0x7ffa000771c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x562ff6074440 tx=0x7ffa1000b040 comp rx=0 tx=0).stop 2024-05-09T03:16:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 msgr2=0x562ff6073db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 --2- 172.21.15.46:0/791841513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562ff6147780 0x562ff6073db0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7ffa18054040 tx=0x7ffa18003330 comp rx=0 tx=0).stop 2024-05-09T03:16:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa1e28d640 1 -- 172.21.15.46:0/791841513 reap_dead start 2024-05-09T03:16:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 shutdown_connections 2024-05-09T03:16:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 >> 172.21.15.46:0/791841513 conn(0x562ff5f94ac0 msgr2=0x562ff60a0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 shutdown_connections 2024-05-09T03:16:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.442+0000 7ffa23fc4f40 1 -- 172.21.15.46:0/791841513 wait complete. 2024-05-09T03:16:11.446 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 500.00000000 2024-05-09T03:16:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.501+0000 7f6b837def40 1 Processor -- start 2024-05-09T03:16:11.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.501+0000 7f6b837def40 1 -- start start 2024-05-09T03:16:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b837def40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 0x563e4413f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b837def40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44157010 0x563e441dc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b837def40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e44152290 0x563e441526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b837def40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563e440b7f50 con 0x563e441407b0 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b837def40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563e440bbf90 con 0x563e44152290 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b837def40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563e441554d0 con 0x563e44157010 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7d2a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 0x563e4413f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7d2a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 0x563e4413f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45150/0 (socket says 172.21.15.46:45150) 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7d2a6640 1 -- 172.21.15.46:0/2174161947 learned_addr learned my addr 172.21.15.46:0/2174161947 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7daa7640 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e44152290 0x563e441526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7caa5640 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44157010 0x563e441dc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7d2a6640 1 -- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44157010 msgr2=0x563e441dc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7d2a6640 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44157010 0x563e441dc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.502+0000 7f6b7d2a6640 1 -- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e44152290 msgr2=0x563e441526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b7d2a6640 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e44152290 0x563e441526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b7d2a6640 1 -- 172.21.15.46:0/2174161947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e4413c850 con 0x563e441407b0 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b7d2a6640 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 0x563e4413f8b0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005fd0 tx=0x7f6b70051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e29b03c8d7470a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b6f7fe640 1 -- 172.21.15.46:0/2174161947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70069070 con 0x563e441407b0 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b6f7fe640 1 -- 172.21.15.46:0/2174161947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b70051890 con 0x563e441407b0 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b6f7fe640 1 -- 172.21.15.46:0/2174161947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70055040 con 0x563e441407b0 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b837def40 1 -- 172.21.15.46:0/2174161947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 msgr2=0x563e4413f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.503+0000 7f6b837def40 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 0x563e4413f8b0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005fd0 tx=0x7f6b70051000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 -- 172.21.15.46:0/2174161947 shutdown_connections 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e44152290 0x563e441526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44157010 0x563e441dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 --2- 172.21.15.46:0/2174161947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e441407b0 0x563e4413f8b0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 -- 172.21.15.46:0/2174161947 >> 172.21.15.46:0/2174161947 conn(0x563e44033ac0 msgr2=0x563e44158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 -- 172.21.15.46:0/2174161947 shutdown_connections 2024-05-09T03:16:11.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 -- 172.21.15.46:0/2174161947 wait complete. 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.504+0000 7f6b837def40 1 Processor -- start 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.505+0000 7f6b837def40 1 -- start start 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b837def40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e441407b0 0x563e4410f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b837def40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44152290 0x563e441d8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b837def40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e44157010 0x563e441e6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b837def40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563e441546c0 con 0x563e44157010 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b837def40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563e44153190 con 0x563e441407b0 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b837def40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563e4413c590 con 0x563e44152290 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e44157010 0x563e441e6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e44157010 0x563e441e6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45152/0 (socket says 172.21.15.46:45152) 2024-05-09T03:16:11.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 -- 172.21.15.46:0/4152124904 learned_addr learned my addr 172.21.15.46:0/4152124904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7d2a6640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e441407b0 0x563e4410f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7caa5640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44152290 0x563e441d8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 -- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44152290 msgr2=0x563e441d8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563e44152290 0x563e441d8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 -- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e441407b0 msgr2=0x563e4410f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563e441407b0 0x563e4410f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.506+0000 7f6b7daa7640 1 -- 172.21.15.46:0/4152124904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e4413ee40 con 0x563e44157010 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.507+0000 7f6b7daa7640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e44157010 0x563e441e6f60 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800dbf0 tx=0x7f6b7800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.507+0000 7f6b6e7fc640 1 -- 172.21.15.46:0/4152124904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78015020 con 0x563e44157010 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.507+0000 7f6b6e7fc640 1 -- 172.21.15.46:0/4152124904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b780030d0 con 0x563e44157010 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.507+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563e4413f120 con 0x563e44157010 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.507+0000 7f6b6e7fc640 1 -- 172.21.15.46:0/4152124904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78011480 con 0x563e44157010 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.507+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e442763d0 con 0x563e44157010 2024-05-09T03:16:11.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.508+0000 7f6b6e7fc640 1 -- 172.21.15.46:0/4152124904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6b78018020 con 0x563e44157010 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.509+0000 7f6b6e7fc640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6b60074cb0 0x7f6b600771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.509+0000 7f6b6e7fc640 1 -- 172.21.15.46:0/4152124904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f6b780961f0 con 0x563e44157010 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.509+0000 7f6b7d2a6640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6b60074cb0 0x7f6b600771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.509+0000 7f6b837def40 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563e441d9c20 0x563e441da030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.509+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x563e441e74a0 con 0x563e441d9c20 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.509+0000 7f6b7caa5640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563e441d9c20 0x563e441da030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b7d2a6640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6b60074cb0 0x7f6b600771c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6b70003000 tx=0x7f6b70052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b7caa5640 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563e441d9c20 0x563e441da030 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b7caa5640 1 -- 172.21.15.46:0/4152124904 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 500.00000000 [stat out=16b] v0'0 uv3 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f6b740106e0 con 0x563e441d9c20 2024-05-09T03:16:11.513 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/500.00000000 mtime 2024-05-09T03:15:53.000000+0000, size 90 2024-05-09T03:16:11.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563e441d9c20 msgr2=0x563e441da030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563e441d9c20 0x563e441da030 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6b60074cb0 msgr2=0x7f6b600771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6b60074cb0 0x7f6b600771c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6b70003000 tx=0x7f6b70052000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e44157010 msgr2=0x563e441e6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 --2- 172.21.15.46:0/4152124904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563e44157010 0x563e441e6f60 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800dbf0 tx=0x7f6b7800b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b7daa7640 1 -- 172.21.15.46:0/4152124904 reap_dead start 2024-05-09T03:16:11.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 shutdown_connections 2024-05-09T03:16:11.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 >> 172.21.15.46:0/4152124904 conn(0x563e44033ac0 msgr2=0x563e4413f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 shutdown_connections 2024-05-09T03:16:11.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.510+0000 7f6b837def40 1 -- 172.21.15.46:0/4152124904 wait complete. 2024-05-09T03:16:11.515 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds_snaptable 2024-05-09T03:16:11.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.570+0000 7f22acb28f40 1 Processor -- start 2024-05-09T03:16:11.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.570+0000 7f22acb28f40 1 -- start start 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22acb28f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 0x557743678160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22acb28f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55774367d010 0x5577436ffed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22acb28f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557743677c80 0x557743664fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22acb28f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5577435ddf50 con 0x5577436667b0 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22acb28f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5577435e1f90 con 0x557743677c80 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22acb28f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55774367b4d0 con 0x55774367d010 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22a65f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 0x557743678160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22a65f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 0x557743678160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45174/0 (socket says 172.21.15.46:45174) 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.571+0000 7f22a65f2640 1 -- 172.21.15.46:0/4107714759 learned_addr learned my addr 172.21.15.46:0/4107714759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a6df3640 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557743677c80 0x557743664fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a5df1640 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55774367d010 0x5577436ffed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a65f2640 1 -- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55774367d010 msgr2=0x5577436ffed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a65f2640 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55774367d010 0x5577436ffed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a65f2640 1 -- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557743677c80 msgr2=0x557743664fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a65f2640 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557743677c80 0x557743664fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a65f2640 1 -- 172.21.15.46:0/4107714759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557743700410 con 0x5577436667b0 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a65f2640 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 0x557743678160 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f22a0046020 tx=0x7f22a0051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe6a4927ee45549 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a4def640 1 -- 172.21.15.46:0/4107714759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0052c60 con 0x5577436667b0 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a4def640 1 -- 172.21.15.46:0/4107714759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a0052e00 con 0x5577436667b0 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.572+0000 7f22a4def640 1 -- 172.21.15.46:0/4107714759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0067680 con 0x5577436667b0 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 -- 172.21.15.46:0/4107714759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 msgr2=0x557743678160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 0x557743678160 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f22a0046020 tx=0x7f22a0051000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 -- 172.21.15.46:0/4107714759 shutdown_connections 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557743677c80 0x557743664fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55774367d010 0x5577436ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 --2- 172.21.15.46:0/4107714759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5577436667b0 0x557743678160 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 -- 172.21.15.46:0/4107714759 >> 172.21.15.46:0/4107714759 conn(0x557743559ac0 msgr2=0x55774367ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 -- 172.21.15.46:0/4107714759 shutdown_connections 2024-05-09T03:16:11.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.573+0000 7f22acb28f40 1 -- 172.21.15.46:0/4107714759 wait complete. 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.574+0000 7f22acb28f40 1 Processor -- start 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.574+0000 7f22acb28f40 1 -- start start 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22acb28f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5577436667b0 0x55774370f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22acb28f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557743677c80 0x55774370ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22acb28f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55774367d010 0x5577436fc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22acb28f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55774367f200 con 0x55774367d010 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22acb28f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55774367a880 con 0x5577436667b0 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22acb28f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55774367ab60 con 0x557743677c80 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22a6df3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55774367d010 0x5577436fc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22a6df3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55774367d010 0x5577436fc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45188/0 (socket says 172.21.15.46:45188) 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22a6df3640 1 -- 172.21.15.46:0/3586836734 learned_addr learned my addr 172.21.15.46:0/3586836734 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22a5df1640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557743677c80 0x55774370ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.575+0000 7f22a65f2640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5577436667b0 0x55774370f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22a6df3640 1 -- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557743677c80 msgr2=0x55774370ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22a6df3640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557743677c80 0x55774370ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22a6df3640 1 -- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5577436667b0 msgr2=0x55774370f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22a6df3640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5577436667b0 0x55774370f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22a6df3640 1 -- 172.21.15.46:0/3586836734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557743635290 con 0x55774367d010 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22a6df3640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55774367d010 0x5577436fc5a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f2298006fa0 tx=0x7f229800d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f229f7fe640 1 -- 172.21.15.46:0/3586836734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298006020 con 0x55774367d010 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f229f7fe640 1 -- 172.21.15.46:0/3586836734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22980061c0 con 0x55774367d010 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f229f7fe640 1 -- 172.21.15.46:0/3586836734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298005440 con 0x55774367d010 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5577436fd530 con 0x55774367d010 2024-05-09T03:16:11.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.576+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577436fdaf0 con 0x55774367d010 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.578+0000 7f229f7fe640 1 -- 172.21.15.46:0/3586836734 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2298019030 con 0x55774367d010 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.578+0000 7f229f7fe640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2288074cb0 0x7f22880771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.578+0000 7f229f7fe640 1 -- 172.21.15.46:0/3586836734 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f2298095510 con 0x55774367d010 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.578+0000 7f22acb28f40 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x557743635460 0x5577436fe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.578+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55774370f740 con 0x557743635460 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.579+0000 7f22a65f2640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2288074cb0 0x7f22880771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.579+0000 7f22a5df1640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x557743635460 0x5577436fe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.579+0000 7f22a65f2640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2288074cb0 0x7f22880771c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f22a0025d60 tx=0x7f22a0069040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.579+0000 7f22a5df1640 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x557743635460 0x5577436fe5d0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22a5df1640 1 -- 172.21.15.46:0/3586836734 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 mds_snaptable [stat out=16b] v0'0 uv1 ondisk = 0) ==== 157+0+16 (crc 0 0 0) 0x7f22900106b0 con 0x557743635460 2024-05-09T03:16:11.582 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/mds_snaptable mtime 2024-05-09T03:15:53.000000+0000, size 70 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x557743635460 msgr2=0x5577436fe5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x557743635460 0x5577436fe5d0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2288074cb0 msgr2=0x7f22880771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2288074cb0 0x7f22880771c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f22a0025d60 tx=0x7f22a0069040 comp rx=0 tx=0).stop 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55774367d010 msgr2=0x5577436fc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 --2- 172.21.15.46:0/3586836734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55774367d010 0x5577436fc5a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f2298006fa0 tx=0x7f229800d5f0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22a6df3640 1 -- 172.21.15.46:0/3586836734 reap_dead start 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 shutdown_connections 2024-05-09T03:16:11.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 >> 172.21.15.46:0/3586836734 conn(0x557743559ac0 msgr2=0x55774367ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 shutdown_connections 2024-05-09T03:16:11.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.580+0000 7f22acb28f40 1 -- 172.21.15.46:0/3586836734 wait complete. 2024-05-09T03:16:11.585 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_inotable 2024-05-09T03:16:11.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.639+0000 7fb959c0bf40 1 Processor -- start 2024-05-09T03:16:11.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.639+0000 7fb959c0bf40 1 -- start start 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb959c0bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e5f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb959c0bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e5f87b0 0x56109e5f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb959c0bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e60f010 0x56109e60a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb959c0bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56109e6128a0 con 0x56109e5c67e0 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb959c0bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56109e60d4d0 con 0x56109e5f87b0 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb959c0bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56109e5c6bf0 con 0x56109e60f010 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e5f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e5f5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45210/0 (socket says 172.21.15.46:45210) 2024-05-09T03:16:11.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 -- 172.21.15.46:0/346450879 learned_addr learned my addr 172.21.15.46:0/346450879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb953267640 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e60f010 0x56109e60a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952265640 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e5f87b0 0x56109e5f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 -- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e60f010 msgr2=0x56109e60a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e60f010 0x56109e60a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 -- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e5f87b0 msgr2=0x56109e5f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e5f87b0 0x56109e5f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 -- 172.21.15.46:0/346450879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109e531350 con 0x56109e5c67e0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb952a66640 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e5f5f00 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009810 tx=0x7fb94c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1c6ef6f901cb04 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb951263640 1 -- 172.21.15.46:0/346450879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c019070 con 0x56109e5c67e0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb951263640 1 -- 172.21.15.46:0/346450879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb94c008890 con 0x56109e5c67e0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.641+0000 7fb951263640 1 -- 172.21.15.46:0/346450879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c008c60 con 0x56109e5c67e0 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 -- 172.21.15.46:0/346450879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 msgr2=0x56109e5f5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e5f5f00 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009810 tx=0x7fb94c005a00 comp rx=0 tx=0).stop 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 -- 172.21.15.46:0/346450879 shutdown_connections 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e60f010 0x56109e60a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e5f87b0 0x56109e5f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/346450879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e5f5f00 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 -- 172.21.15.46:0/346450879 >> 172.21.15.46:0/346450879 conn(0x56109e4ebac0 msgr2=0x56109e5f3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 -- 172.21.15.46:0/346450879 shutdown_connections 2024-05-09T03:16:11.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.642+0000 7fb959c0bf40 1 -- 172.21.15.46:0/346450879 wait complete. 2024-05-09T03:16:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.643+0000 7fb959c0bf40 1 Processor -- start 2024-05-09T03:16:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.643+0000 7fb959c0bf40 1 -- start start 2024-05-09T03:16:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb959c0bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e739ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb959c0bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e5f87b0 0x56109e68dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb959c0bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e60f010 0x56109e68e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb959c0bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56109e5f3370 con 0x56109e5c67e0 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb959c0bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56109e5f6440 con 0x56109e60f010 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb959c0bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56109e695fa0 con 0x56109e5f87b0 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb952a66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e739ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb952a66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e739ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45218/0 (socket says 172.21.15.46:45218) 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.644+0000 7fb952a66640 1 -- 172.21.15.46:0/2638566530 learned_addr learned my addr 172.21.15.46:0/2638566530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb953267640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e60f010 0x56109e68e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952265640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e5f87b0 0x56109e68dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952a66640 1 -- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e5f87b0 msgr2=0x56109e68dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952a66640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56109e5f87b0 0x56109e68dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952a66640 1 -- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e60f010 msgr2=0x56109e68e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952a66640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e60f010 0x56109e68e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952a66640 1 -- 172.21.15.46:0/2638566530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56109e73a770 con 0x56109e5c67e0 2024-05-09T03:16:11.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb953267640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56109e60f010 0x56109e68e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb952a66640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e739ac0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fb94c000c00 tx=0x7fb94c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb94bfff640 1 -- 172.21.15.46:0/2638566530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c019040 con 0x56109e5c67e0 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb94bfff640 1 -- 172.21.15.46:0/2638566530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb94c022070 con 0x56109e5c67e0 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb94bfff640 1 -- 172.21.15.46:0/2638566530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94c0148b0 con 0x56109e5c67e0 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56109e693ea0 con 0x56109e5c67e0 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.645+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56109e694460 con 0x56109e5c67e0 2024-05-09T03:16:11.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.647+0000 7fb94bfff640 1 -- 172.21.15.46:0/2638566530 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb94c01c070 con 0x56109e5c67e0 2024-05-09T03:16:11.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.647+0000 7fb94bfff640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb934074ce0 0x7fb9340771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.647+0000 7fb94bfff640 1 -- 172.21.15.46:0/2638566530 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fb94c097490 con 0x56109e5c67e0 2024-05-09T03:16:11.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.647+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56109e73a000 0x56109e6a2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.647+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x56109e699d20 con 0x56109e73a000 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.647+0000 7fb952265640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb934074ce0 0x7fb9340771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb953267640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56109e73a000 0x56109e6a2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb953267640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56109e73a000 0x56109e6a2340 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb952265640 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb934074ce0 0x7fb9340771f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb93c057000 tx=0x7fb93c052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb953267640 1 -- 172.21.15.46:0/2638566530 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_inotable [stat out=16b] v0'0 uv3 ondisk = 0) ==== 157+0+16 (crc 0 0 0) 0x7fb944011040 con 0x56109e73a000 2024-05-09T03:16:11.651 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/mds0_inotable mtime 2024-05-09T03:16:08.000000+0000, size 34 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56109e73a000 msgr2=0x56109e6a2340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56109e73a000 0x56109e6a2340 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb934074ce0 msgr2=0x7fb9340771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb934074ce0 0x7fb9340771f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb93c057000 tx=0x7fb93c052000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 msgr2=0x56109e739ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 --2- 172.21.15.46:0/2638566530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56109e5c67e0 0x56109e739ac0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fb94c000c00 tx=0x7fb94c003020 comp rx=0 tx=0).stop 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb953267640 1 -- 172.21.15.46:0/2638566530 reap_dead start 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 shutdown_connections 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.648+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 >> 172.21.15.46:0/2638566530 conn(0x56109e4ebac0 msgr2=0x56109e60a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.649+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 shutdown_connections 2024-05-09T03:16:11.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.649+0000 7fb959c0bf40 1 -- 172.21.15.46:0/2638566530 wait complete. 2024-05-09T03:16:11.654 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000 2024-05-09T03:16:11.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.708+0000 7f80b7778f40 1 Processor -- start 2024-05-09T03:16:11.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.708+0000 7f80b7778f40 1 -- start start 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b7778f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c73a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b7778f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c77b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b7778f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c763630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b7778f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c1c6dff50 con 0x557c1c778400 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b7778f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557c1c6e3f90 con 0x557c1c80d580 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b7778f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557c1c766690 con 0x557c1c766830 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c77b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c77b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45224/0 (socket says 172.21.15.46:45224) 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 -- 172.21.15.46:0/1490437129 learned_addr learned my addr 172.21.15.46:0/1490437129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b1a43640 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c763630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b1242640 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c73a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 -- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 msgr2=0x557c1c73a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c73a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 -- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 msgr2=0x557c1c763630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c763630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 -- 172.21.15.46:0/1490437129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c1c80dd70 con 0x557c1c778400 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80b0a41640 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c77b7e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f809c001e60 tx=0x7f809c00b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312adb5fa4a5fdd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80ab7fe640 1 -- 172.21.15.46:0/1490437129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c012020 con 0x557c1c778400 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80ab7fe640 1 -- 172.21.15.46:0/1490437129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f809c00e670 con 0x557c1c778400 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.710+0000 7f80ab7fe640 1 -- 172.21.15.46:0/1490437129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c00ea60 con 0x557c1c778400 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 -- 172.21.15.46:0/1490437129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 msgr2=0x557c1c77b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c77b7e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f809c001e60 tx=0x7f809c00b780 comp rx=0 tx=0).stop 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 -- 172.21.15.46:0/1490437129 shutdown_connections 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c763630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c77b7e0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 --2- 172.21.15.46:0/1490437129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c73a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 -- 172.21.15.46:0/1490437129 >> 172.21.15.46:0/1490437129 conn(0x557c1c65bac0 msgr2=0x557c1c8115e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 -- 172.21.15.46:0/1490437129 shutdown_connections 2024-05-09T03:16:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.711+0000 7f80b7778f40 1 -- 172.21.15.46:0/1490437129 wait complete. 2024-05-09T03:16:11.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.712+0000 7f80b7778f40 1 Processor -- start 2024-05-09T03:16:11.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.712+0000 7f80b7778f40 1 -- start start 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b7778f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c77e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b7778f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c7ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b7778f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c8a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b7778f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c1c804750 con 0x557c1c778400 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b7778f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557c1c80e000 con 0x557c1c80d580 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b7778f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557c1c89f8f0 con 0x557c1c766830 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.713+0000 7f80b0a41640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c7ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c7ef600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45234/0 (socket says 172.21.15.46:45234) 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 -- 172.21.15.46:0/446966020 learned_addr learned my addr 172.21.15.46:0/446966020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b1a43640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c8a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 -- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 msgr2=0x557c1c77e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b1242640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c77e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c77e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 -- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 msgr2=0x557c1c8a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c8a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 -- 172.21.15.46:0/446966020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c1c7f0330 con 0x557c1c778400 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b1a43640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557c1c80d580 0x557c1c8a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b1242640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557c1c766830 0x557c1c77e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b0a41640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c7ef600 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f809c001ed0 tx=0x7f809c0040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80aa7fc640 1 -- 172.21.15.46:0/446966020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c018640 con 0x557c1c778400 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80aa7fc640 1 -- 172.21.15.46:0/446966020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f809c018800 con 0x557c1c778400 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557c1c7f0610 con 0x557c1c778400 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80aa7fc640 1 -- 172.21.15.46:0/446966020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c018bf0 con 0x557c1c778400 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.714+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c1c7f0bd0 con 0x557c1c778400 2024-05-09T03:16:11.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.716+0000 7f80aa7fc640 1 -- 172.21.15.46:0/446966020 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f809c015020 con 0x557c1c778400 2024-05-09T03:16:11.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.716+0000 7f80aa7fc640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8094074cb0 0x7f80940771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80aa7fc640 1 -- 172.21.15.46:0/446966020 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f809c098040 con 0x557c1c778400 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b7778f40 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557c1c77e610 0x557c1c89e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x557c1c89edd0 con 0x557c1c77e610 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b1242640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8094074cb0 0x7f80940771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b1a43640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557c1c77e610 0x557c1c89e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b1a43640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557c1c77e610 0x557c1c89e890 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b1242640 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8094074cb0 0x7f80940771c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f80ac056000 tx=0x7f80ac050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.717+0000 7f80b1a43640 1 -- 172.21.15.46:0/446966020 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 100.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f80a40014d0 con 0x557c1c77e610 2024-05-09T03:16:11.720 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/100.00000000 mtime 2024-05-09T03:15:53.000000+0000, size 0 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557c1c77e610 msgr2=0x557c1c89e890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557c1c77e610 0x557c1c89e890 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8094074cb0 msgr2=0x7f80940771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8094074cb0 0x7f80940771c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f80ac056000 tx=0x7f80ac050000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 msgr2=0x557c1c7ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 --2- 172.21.15.46:0/446966020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c1c778400 0x557c1c7ef600 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f809c001ed0 tx=0x7f809c0040e0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b1a43640 1 -- 172.21.15.46:0/446966020 reap_dead start 2024-05-09T03:16:11.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 shutdown_connections 2024-05-09T03:16:11.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 >> 172.21.15.46:0/446966020 conn(0x557c1c65bac0 msgr2=0x557c1c7821b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 shutdown_connections 2024-05-09T03:16:11.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.718+0000 7f80b7778f40 1 -- 172.21.15.46:0/446966020 wait complete. 2024-05-09T03:16:11.723 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_sessionmap 2024-05-09T03:16:11.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.778+0000 7fa8a3b26f40 1 Processor -- start 2024-05-09T03:16:11.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.778+0000 7fa8a3b26f40 1 -- start start 2024-05-09T03:16:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa8a3b26f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa8a3b26f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e1215f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa8a3b26f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e13549b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa8a3b26f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652e12355f0 con 0x5652e12c0150 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa8a3b26f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5652e1192f50 con 0x5652e122c3a0 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa8a3b26f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5652e12358d0 con 0x5652e12345f0 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89d388640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e13549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89d388640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e13549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45238/0 (socket says 172.21.15.46:45238) 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89d388640 1 -- 172.21.15.46:0/3078143377 learned_addr learned my addr 172.21.15.46:0/3078143377 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89c386640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e1215f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89cb87640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89d388640 1 -- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 msgr2=0x5652e1215f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.779+0000 7fa89d388640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e1215f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89d388640 1 -- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 msgr2=0x5652e12159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89d388640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89d388640 1 -- 172.21.15.46:0/3078143377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e1219fb0 con 0x5652e12c0150 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89c386640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e1215f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89cb87640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89d388640 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e13549b0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fa894046180 tx=0x7fa894050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82069d52f86d43e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89b384640 1 -- 172.21.15.46:0/3078143377 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89404fa20 con 0x5652e12c0150 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89b384640 1 -- 172.21.15.46:0/3078143377 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa89404fbc0 con 0x5652e12c0150 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa89b384640 1 -- 172.21.15.46:0/3078143377 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa894057400 con 0x5652e12c0150 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/3078143377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 msgr2=0x5652e13549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e13549b0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fa894046180 tx=0x7fa894050000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/3078143377 shutdown_connections 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e13549b0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e1215f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.780+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/3078143377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.781+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/3078143377 >> 172.21.15.46:0/3078143377 conn(0x5652e110eac0 msgr2=0x5652e1217150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.781+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/3078143377 shutdown_connections 2024-05-09T03:16:11.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.781+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/3078143377 wait complete. 2024-05-09T03:16:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.781+0000 7fa8a3b26f40 1 Processor -- start 2024-05-09T03:16:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.781+0000 7fa8a3b26f40 1 -- start start 2024-05-09T03:16:11.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa8a3b26f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa8a3b26f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e12313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa8a3b26f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e1231900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa8a3b26f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652e1230640 con 0x5652e12c0150 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa8a3b26f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5652e12c0d90 con 0x5652e122c3a0 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa8a3b26f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5652e121a290 con 0x5652e12345f0 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e1231900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e1231900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45240/0 (socket says 172.21.15.46:45240) 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 -- 172.21.15.46:0/157472327 learned_addr learned my addr 172.21.15.46:0/157472327 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89cb87640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89c386640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e12313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 -- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 msgr2=0x5652e12313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5652e12345f0 0x5652e12313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 -- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 msgr2=0x5652e12a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5652e122c3a0 0x5652e12a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 -- 172.21.15.46:0/157472327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652e12bd520 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.783+0000 7fa89d388640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e1231900 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fa89405a040 tx=0x7fa894003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.784+0000 7fa89a382640 1 -- 172.21.15.46:0/157472327 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa894072020 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.784+0000 7fa89a382640 1 -- 172.21.15.46:0/157472327 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa894058c10 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.784+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5652e1215300 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.784+0000 7fa89a382640 1 -- 172.21.15.46:0/157472327 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89406e470 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.784+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652e12a5ca0 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.785+0000 7fa89a382640 1 -- 172.21.15.46:0/157472327 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa894075020 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa89a382640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa888074cb0 0x7fa8880771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa89a382640 1 -- 172.21.15.46:0/157472327 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fa8940dc2f0 con 0x5652e12c0150 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa89cb87640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa888074cb0 0x7fa8880771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5652e12be9b0 0x5652e12a3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5652e12a1e90 con 0x5652e12be9b0 2024-05-09T03:16:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa89c386640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5652e12be9b0 0x5652e12a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.786+0000 7fa89cb87640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa888074cb0 0x7fa8880771c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000c00 tx=0x7fa88c0017e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa89c386640 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5652e12be9b0 0x5652e12a3000 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa89c386640 1 -- 172.21.15.46:0/157472327 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 mds0_sessionmap [stat out=16b] v0'0 uv3 ondisk = 0) ==== 159+0+16 (crc 0 0 0) 0x7fa890007730 con 0x5652e12be9b0 2024-05-09T03:16:11.790 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/mds0_sessionmap mtime 2024-05-09T03:16:08.000000+0000, size 0 2024-05-09T03:16:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5652e12be9b0 msgr2=0x5652e12a3000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5652e12be9b0 0x5652e12a3000 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa888074cb0 msgr2=0x7fa8880771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa888074cb0 0x7fa8880771c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000c00 tx=0x7fa88c0017e0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 msgr2=0x5652e1231900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 --2- 172.21.15.46:0/157472327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652e12c0150 0x5652e1231900 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fa89405a040 tx=0x7fa894003330 comp rx=0 tx=0).stop 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa89d388640 1 -- 172.21.15.46:0/157472327 reap_dead start 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 shutdown_connections 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 >> 172.21.15.46:0/157472327 conn(0x5652e110eac0 msgr2=0x5652e12bdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 shutdown_connections 2024-05-09T03:16:11.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.787+0000 7fa8a3b26f40 1 -- 172.21.15.46:0/157472327 wait complete. 2024-05-09T03:16:11.792 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 400.00000000 2024-05-09T03:16:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.846+0000 7f8535c0bf40 1 Processor -- start 2024-05-09T03:16:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.846+0000 7f8535c0bf40 1 -- start start 2024-05-09T03:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.847+0000 7f8535c0bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aa4d5f0 0x55859aad0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.847+0000 7f8535c0bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aad8a40 0x55859aa32730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.847+0000 7f8535c0bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 0x55859aa2f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f8535c0bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55859aa4e5f0 con 0x55859aad9390 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f8535c0bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55859a9abf50 con 0x55859aad8a40 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f8535c0bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55859aa4e8d0 con 0x55859aa4d5f0 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 0x55859aa2f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 0x55859aa2f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45260/0 (socket says 172.21.15.46:45260) 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 -- 172.21.15.46:0/1260343615 learned_addr learned my addr 172.21.15.46:0/1260343615 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852e265640 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aad8a40 0x55859aa32730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 -- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aa4d5f0 msgr2=0x55859aad0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852ea66640 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aa4d5f0 0x55859aad0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aa4d5f0 0x55859aad0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 -- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aad8a40 msgr2=0x55859aa32730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aad8a40 0x55859aa32730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 -- 172.21.15.46:0/1260343615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859aa307c0 con 0x55859aad9390 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852e265640 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aad8a40 0x55859aa32730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852f267640 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 0x55859aa2f380 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f85200015b0 tx=0x7f85200019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16352bfba0e2cee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852d263640 1 -- 172.21.15.46:0/1260343615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520016070 con 0x55859aad9390 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852d263640 1 -- 172.21.15.46:0/1260343615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f852000f710 con 0x55859aad9390 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.848+0000 7f852d263640 1 -- 172.21.15.46:0/1260343615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852000fb00 con 0x55859aad9390 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1260343615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 msgr2=0x55859aa2f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 0x55859aa2f380 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f85200015b0 tx=0x7f85200019b0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1260343615 shutdown_connections 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad9390 0x55859aa2f380 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aad8a40 0x55859aa32730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1260343615 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aa4d5f0 0x55859aad0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1260343615 >> 172.21.15.46:0/1260343615 conn(0x55859a927ac0 msgr2=0x55859aa4f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1260343615 shutdown_connections 2024-05-09T03:16:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.849+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1260343615 wait complete. 2024-05-09T03:16:11.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.850+0000 7f8535c0bf40 1 Processor -- start 2024-05-09T03:16:11.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.850+0000 7f8535c0bf40 1 -- start start 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f8535c0bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aa4d5f0 0x55859aad7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f8535c0bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad8a40 0x55859aad05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f8535c0bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aad9390 0x55859aa33fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f8535c0bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55859aa34e50 con 0x55859aad8a40 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f8535c0bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55859aad75d0 con 0x55859aa4d5f0 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f8535c0bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55859aad8040 con 0x55859aad9390 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f852e265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad8a40 0x55859aad05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f852e265640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad8a40 0x55859aad05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45274/0 (socket says 172.21.15.46:45274) 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.851+0000 7f852e265640 1 -- 172.21.15.46:0/1524828074 learned_addr learned my addr 172.21.15.46:0/1524828074 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852ea66640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aa4d5f0 0x55859aad7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852f267640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aad9390 0x55859aa33fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852e265640 1 -- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aad9390 msgr2=0x55859aa33fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852e265640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aad9390 0x55859aa33fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852e265640 1 -- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aa4d5f0 msgr2=0x55859aad7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852e265640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55859aa4d5f0 0x55859aad7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852e265640 1 -- 172.21.15.46:0/1524828074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55859aad8320 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852f267640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55859aad9390 0x55859aa33fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f852e265640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad8a40 0x55859aad05c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f8518009f20 tx=0x7f85180084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f8527fff640 1 -- 172.21.15.46:0/1524828074 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518008e40 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f8527fff640 1 -- 172.21.15.46:0/1524828074 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85180143f0 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55859aadb520 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f8527fff640 1 -- 172.21.15.46:0/1524828074 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85180147a0 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.852+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55859aad77a0 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.854+0000 7f8527fff640 1 -- 172.21.15.46:0/1524828074 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f851801b020 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.854+0000 7f8527fff640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8510074cb0 0x7f85100771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.854+0000 7f8527fff640 1 -- 172.21.15.46:0/1524828074 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f8518095c70 con 0x55859aad8a40 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.855+0000 7f852ea66640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8510074cb0 0x7f85100771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.855+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55859aa34a00 0x55859aad2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.855+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x55859aacdff0 con 0x55859aa34a00 2024-05-09T03:16:11.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.855+0000 7f852f267640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55859aa34a00 0x55859aad2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.855+0000 7f852ea66640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8510074cb0 0x7f85100771c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8528057000 tx=0x7f8528052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.855+0000 7f852f267640 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55859aa34a00 0x55859aad2350 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f852f267640 1 -- 172.21.15.46:0/1524828074 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 400.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f8520019040 con 0x55859aa34a00 2024-05-09T03:16:11.858 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/400.00000000 mtime 2024-05-09T03:15:53.000000+0000, size 22 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55859aa34a00 msgr2=0x55859aad2350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55859aa34a00 0x55859aad2350 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8510074cb0 msgr2=0x7f85100771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8510074cb0 0x7f85100771c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8528057000 tx=0x7f8528052000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad8a40 msgr2=0x55859aad05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 --2- 172.21.15.46:0/1524828074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55859aad8a40 0x55859aad05c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f8518009f20 tx=0x7f85180084c0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f852f267640 1 -- 172.21.15.46:0/1524828074 reap_dead start 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 shutdown_connections 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 >> 172.21.15.46:0/1524828074 conn(0x55859a927ac0 msgr2=0x55859aa4f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 shutdown_connections 2024-05-09T03:16:11.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.856+0000 7f8535c0bf40 1 -- 172.21.15.46:0/1524828074 wait complete. 2024-05-09T03:16:11.860 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000.inode 2024-05-09T03:16:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.916+0000 7f74f771ef40 1 Processor -- start 2024-05-09T03:16:11.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.916+0000 7f74f771ef40 1 -- start start 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f771ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 0x559cbdc8ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f771ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdbef9a0 0x559cbdc95040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f771ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdc908f0 0x559cbdc90d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f771ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559cbdc0a5f0 con 0x559cbdc97280 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f771ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559cbdb67f50 con 0x559cbdc908f0 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f771ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559cbdc0a8d0 con 0x559cbdbef9a0 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 0x559cbdc8ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 0x559cbdc8ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45284/0 (socket says 172.21.15.46:45284) 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 -- 172.21.15.46:0/1113440287 learned_addr learned my addr 172.21.15.46:0/1113440287 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0f19640 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdc908f0 0x559cbdc90d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74eff17640 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdbef9a0 0x559cbdc95040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 -- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdbef9a0 msgr2=0x559cbdc95040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdbef9a0 0x559cbdc95040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 -- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdc908f0 msgr2=0x559cbdc90d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdc908f0 0x559cbdc90d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.917+0000 7f74f0718640 1 -- 172.21.15.46:0/1113440287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cbdbee8d0 con 0x559cbdc97280 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f0718640 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 0x559cbdc8ca70 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009730 tx=0x7f74e0005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3cce3d93207636 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74eef15640 1 -- 172.21.15.46:0/1113440287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e00078f0 con 0x559cbdc97280 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74eef15640 1 -- 172.21.15.46:0/1113440287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74e0007a90 con 0x559cbdc97280 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74eef15640 1 -- 172.21.15.46:0/1113440287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0007e40 con 0x559cbdc97280 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 -- 172.21.15.46:0/1113440287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 msgr2=0x559cbdc8ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 0x559cbdc8ca70 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009730 tx=0x7f74e0005a00 comp rx=0 tx=0).stop 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 -- 172.21.15.46:0/1113440287 shutdown_connections 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdc908f0 0x559cbdc90d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdbef9a0 0x559cbdc95040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 --2- 172.21.15.46:0/1113440287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc97280 0x559cbdc8ca70 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 -- 172.21.15.46:0/1113440287 >> 172.21.15.46:0/1113440287 conn(0x559cbdae3ac0 msgr2=0x559cbdc047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 -- 172.21.15.46:0/1113440287 shutdown_connections 2024-05-09T03:16:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.918+0000 7f74f771ef40 1 -- 172.21.15.46:0/1113440287 wait complete. 2024-05-09T03:16:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.919+0000 7f74f771ef40 1 Processor -- start 2024-05-09T03:16:11.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.919+0000 7f74f771ef40 1 -- start start 2024-05-09T03:16:11.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdbef9a0 0x559cbdc8ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc908f0 0x559cbdc8d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdc97280 0x559cbdc8dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559cbdbefdb0 con 0x559cbdc908f0 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559cbdbef510 con 0x559cbdbef9a0 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559cbdbef1a0 con 0x559cbdc97280 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc908f0 0x559cbdc8d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc908f0 0x559cbdc8d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45286/0 (socket says 172.21.15.46:45286) 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 -- 172.21.15.46:0/2727161070 learned_addr learned my addr 172.21.15.46:0/2727161070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f0718640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdbef9a0 0x559cbdc8ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f0f19640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdc97280 0x559cbdc8dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 -- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdc97280 msgr2=0x559cbdc8dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559cbdc97280 0x559cbdc8dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 -- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdbef9a0 msgr2=0x559cbdc8ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559cbdbef9a0 0x559cbdc8ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 -- 172.21.15.46:0/2727161070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cbdc02f10 con 0x559cbdc908f0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74eff17640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc908f0 0x559cbdc8d6e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009bd0 tx=0x7f74e40067e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74edf13640 1 -- 172.21.15.46:0/2727161070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e401a420 con 0x559cbdc908f0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74edf13640 1 -- 172.21.15.46:0/2727161070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74e401a5c0 con 0x559cbdc908f0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559cbdc88a10 con 0x559cbdc908f0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.921+0000 7f74edf13640 1 -- 172.21.15.46:0/2727161070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e401a970 con 0x559cbdc908f0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.922+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cbdc88fd0 con 0x559cbdc908f0 2024-05-09T03:16:11.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.923+0000 7f74edf13640 1 -- 172.21.15.46:0/2727161070 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f74e401ab10 con 0x559cbdc908f0 2024-05-09T03:16:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74edf13640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74dc074ce0 0x7f74dc0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74edf13640 1 -- 172.21.15.46:0/2727161070 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f74e4098af0 con 0x559cbdc908f0 2024-05-09T03:16:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74f771ef40 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559cbdc03be0 0x559cbdc03ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74f0718640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74dc074ce0 0x7f74dc0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x559cbdc8f1d0 con 0x559cbdc03be0 2024-05-09T03:16:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74f0f19640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559cbdc03be0 0x559cbdc03ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.924+0000 7f74f0718640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74dc074ce0 0x7f74dc0771f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009730 tx=0x7f74e0017040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f0f19640 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559cbdc03be0 0x559cbdc03ff0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f0f19640 1 -- 172.21.15.46:0/2727161070 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x7f74e804db30 con 0x559cbdc03be0 2024-05-09T03:16:11.927 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/100.00000000.inode mtime 2024-05-09T03:15:53.000000+0000, size 582 2024-05-09T03:16:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559cbdc03be0 msgr2=0x559cbdc03ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x559cbdc03be0 0x559cbdc03ff0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74dc074ce0 msgr2=0x7f74dc0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74dc074ce0 0x7f74dc0771f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f74e0009730 tx=0x7f74e0017040 comp rx=0 tx=0).stop 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc908f0 msgr2=0x559cbdc8d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 --2- 172.21.15.46:0/2727161070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559cbdc908f0 0x559cbdc8d6e0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009bd0 tx=0x7f74e40067e0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f0f19640 1 -- 172.21.15.46:0/2727161070 reap_dead start 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 shutdown_connections 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 >> 172.21.15.46:0/2727161070 conn(0x559cbdae3ac0 msgr2=0x559cbdc95040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 shutdown_connections 2024-05-09T03:16:11.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.925+0000 7f74f771ef40 1 -- 172.21.15.46:0/2727161070 wait complete. 2024-05-09T03:16:11.929 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000 2024-05-09T03:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.984+0000 7ffa79f71f40 1 Processor -- start 2024-05-09T03:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.984+0000 7ffa79f71f40 1 -- start start 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa79f71f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb78140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa79f71f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feb58990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa79f71f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feacaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa79f71f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5582fea47f50 con 0x5582feb76780 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa79f71f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5582fea4bf90 con 0x5582feb598d0 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa79f71f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5582feae0cb0 con 0x5582feacddb0 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feb58990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feb58990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45302/0 (socket says 172.21.15.46:45302) 2024-05-09T03:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 -- 172.21.15.46:0/711768978 learned_addr learned my addr 172.21.15.46:0/711768978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7423c640 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feacaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa73a3b640 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb78140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 -- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 msgr2=0x5582feb78140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb78140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 -- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 msgr2=0x5582feacaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feacaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 -- 172.21.15.46:0/711768978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582feb59e50 con 0x5582feb76780 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa73a3b640 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb78140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.985+0000 7ffa7323a640 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feb58990 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c001db0 tx=0x7ffa5c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8376b3bcbe4bbc61 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa72238640 1 -- 172.21.15.46:0/711768978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00bde0 con 0x5582feb76780 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa72238640 1 -- 172.21.15.46:0/711768978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5c00f400 con 0x5582feb76780 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa72238640 1 -- 172.21.15.46:0/711768978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00f7f0 con 0x5582feb76780 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 -- 172.21.15.46:0/711768978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 msgr2=0x5582feb58990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feb58990 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c001db0 tx=0x7ffa5c00b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 -- 172.21.15.46:0/711768978 shutdown_connections 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feacaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feb58990 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/711768978 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb78140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 -- 172.21.15.46:0/711768978 >> 172.21.15.46:0/711768978 conn(0x5582fe9c3ac0 msgr2=0x5582feb750c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 -- 172.21.15.46:0/711768978 shutdown_connections 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.986+0000 7ffa79f71f40 1 -- 172.21.15.46:0/711768978 wait complete. 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.987+0000 7ffa79f71f40 1 Processor -- start 2024-05-09T03:16:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.987+0000 7ffa79f71f40 1 -- start start 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb58990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feb57980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feacaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5582feb58ed0 con 0x5582feb76780 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5582feb5a130 con 0x5582feb598d0 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5582feb77940 con 0x5582feacddb0 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feacaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feacaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45304/0 (socket says 172.21.15.46:45304) 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 -- 172.21.15.46:0/369566711 learned_addr learned my addr 172.21.15.46:0/369566711 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa73a3b640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb58990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7323a640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feb57980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 -- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 msgr2=0x5582feb58990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5582feacddb0 0x5582feb58990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 -- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 msgr2=0x5582feb57980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5582feb598d0 0x5582feb57980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 -- 172.21.15.46:0/369566711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582feae20d0 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa7423c640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feacaa00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ffa64002250 tx=0x7ffa6400e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa71236640 1 -- 172.21.15.46:0/369566711 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6400ee90 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa71236640 1 -- 172.21.15.46:0/369566711 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa64017070 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa71236640 1 -- 172.21.15.46:0/369566711 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa64004520 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.989+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5582feae23b0 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.990+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582feae2970 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.991+0000 7ffa71236640 1 -- 172.21.15.46:0/369566711 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa640046c0 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa71236640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa54074e50 0x7ffa54077360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa71236640 1 -- 172.21.15.46:0/369566711 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7ffa64095ee0 con 0x5582feb76780 2024-05-09T03:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa73a3b640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa54074e50 0x7ffa54077360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5582feae33a0 0x5582feae37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x5582feb74000 con 0x5582feae33a0 2024-05-09T03:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa7323a640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5582feae33a0 0x5582feae37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.992+0000 7ffa73a3b640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa54074e50 0x7ffa54077360 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c057000 tx=0x7ffa6c052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa7323a640 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5582feae33a0 0x5582feae37b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa7323a640 1 -- 172.21.15.46:0/369566711 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 1.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) ==== 154+0+16 (crc 0 0 0) 0x7ffa5c00f6b0 con 0x5582feae33a0 2024-05-09T03:16:11.996 INFO:teuthology.orchestra.run.smithi046.stdout:cephfs_metadata/1.00000000 mtime 2024-05-09T03:16:08.000000+0000, size 0 2024-05-09T03:16:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5582feae33a0 msgr2=0x5582feae37b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5582feae33a0 0x5582feae37b0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa54074e50 msgr2=0x7ffa54077360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ffa54074e50 0x7ffa54077360 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c057000 tx=0x7ffa6c052000 comp rx=0 tx=0).stop 2024-05-09T03:16:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 msgr2=0x5582feacaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:11.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 --2- 172.21.15.46:0/369566711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5582feb76780 0x5582feacaa00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ffa64002250 tx=0x7ffa6400e660 comp rx=0 tx=0).stop 2024-05-09T03:16:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa7423c640 1 -- 172.21.15.46:0/369566711 reap_dead start 2024-05-09T03:16:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 shutdown_connections 2024-05-09T03:16:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 >> 172.21.15.46:0/369566711 conn(0x5582fe9c3ac0 msgr2=0x5582feacf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 shutdown_connections 2024-05-09T03:16:11.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:11.993+0000 7ffa79f71f40 1 -- 172.21.15.46:0/369566711 wait complete. 2024-05-09T03:16:11.998 INFO:tasks.cephfs.test_damage:Expectation on rm '10000000000.00000000' will be 'no damage' 2024-05-09T03:16:11.998 INFO:tasks.cephfs.test_damage:Expectation on rm '600.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.998 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000.inode' will be 'damaged_on_start' 2024-05-09T03:16:11.998 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.998 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000001' will be 'damaged_on_start' 2024-05-09T03:16:11.998 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_openfiles.0' will be 'no damage' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm '500.00000000' will be 'no damage' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds_snaptable' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_inotable' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_sessionmap' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm '400.00000000' will be 'no damage' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000.inode' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '10000000000.00000000' will be 'no damage' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '600.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_openfiles.0' will be 'no damage' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '100.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_sessionmap' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '1.00000000' will be 'damaged_on_start' 2024-05-09T03:16:11.999 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 10000000000.00000000' 2024-05-09T03:16:12.000 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:12.000 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:12.016 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:16:12.016 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:12.016 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:12.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:12.016 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:16:12.072 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:16:12.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:12.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.226+0000 7f10d7700640 1 Processor -- start 2024-05-09T03:16:12.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.227+0000 7f10d7700640 1 -- start start 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d7700640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10d00d2800 0x7f10d00b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d7700640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10d00b3f50 0x7f10d00ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d7700640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10d006ebe0 con 0x7f10d00b4700 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d7700640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f10d006eec0 con 0x7f10d00b3f50 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d7700640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f10d006f1a0 con 0x7f10d00d2800 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10d00b3f50 0x7f10d00ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10d00b3f50 0x7f10d00ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45320/0 (socket says 172.21.15.46:45320) 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 -- 172.21.15.46:0/911991399 learned_addr learned my addr 172.21.15.46:0/911991399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d4c9b640 1 --2- 172.21.15.46:0/911991399 >> v2:172.21.15.88:3300/0 conn(0x7f10d00d2800 0x7f10d00b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 -- 172.21.15.46:0/911991399 >> v2:172.21.15.88:3300/0 conn(0x7f10d00d2800 msgr2=0x7f10d00b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:12.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 --2- 172.21.15.46:0/911991399 >> v2:172.21.15.88:3300/0 conn(0x7f10d00d2800 0x7f10d00b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:12.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 -- 172.21.15.46:0/911991399 >> v1:172.21.15.88:6789/0 conn(0x7f10d00b4700 legacy=0x7f10d00d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:12.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.228+0000 7f10d5c9d640 1 -- 172.21.15.46:0/911991399 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d006f480 con 0x7f10d00b3f50 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.229+0000 7f10d5c9d640 1 --2- 172.21.15.46:0/911991399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d00ce130 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f10d00d4b40 tx=0x7f10c802a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecfec094de585b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.229+0000 7f10cffff640 1 -- 172.21.15.46:0/911991399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8037070 con 0x7f10d00b3f50 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.229+0000 7f10cffff640 1 -- 172.21.15.46:0/911991399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c8032440 con 0x7f10d00b3f50 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.229+0000 7f10cffff640 1 -- 172.21.15.46:0/911991399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c8032810 con 0x7f10d00b3f50 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 -- 172.21.15.46:0/911991399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 msgr2=0x7f10d00ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 --2- 172.21.15.46:0/911991399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d00ce130 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f10d00d4b40 tx=0x7f10c802a150 comp rx=0 tx=0).stop 2024-05-09T03:16:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 -- 172.21.15.46:0/911991399 shutdown_connections 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 --2- 172.21.15.46:0/911991399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d00ce130 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 --2- 172.21.15.46:0/911991399 >> v2:172.21.15.88:3300/0 conn(0x7f10d00d2800 0x7f10d00b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 -- 172.21.15.46:0/911991399 >> 172.21.15.46:0/911991399 conn(0x7f10d00b55d0 msgr2=0x7f10d00b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 -- 172.21.15.46:0/911991399 shutdown_connections 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.230+0000 7f10d7700640 1 -- 172.21.15.46:0/911991399 wait complete. 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.232+0000 7f10d7700640 1 Processor -- start 2024-05-09T03:16:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.232+0000 7f10d7700640 1 -- start start 2024-05-09T03:16:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d7700640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d01dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d7700640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d00b4700 0x7f10d00cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d7700640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10d00d2800 0x7f10d00ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d7700640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10d00b51d0 con 0x7f10d00b3f50 2024-05-09T03:16:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d7700640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f10d00b4360 con 0x7f10d00b4700 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d7700640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f10d00cbfd0 con 0x7f10d00d2800 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d549c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d01dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d549c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d01dadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45332/0 (socket says 172.21.15.46:45332) 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.233+0000 7f10d549c640 1 -- 172.21.15.46:0/2892364492 learned_addr learned my addr 172.21.15.46:0/2892364492 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d5c9d640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10d00d2800 0x7f10d00ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d4c9b640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d00b4700 0x7f10d00cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d549c640 1 -- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10d00d2800 msgr2=0x7f10d00ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d549c640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10d00d2800 0x7f10d00ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d549c640 1 -- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d00b4700 msgr2=0x7f10d00cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d549c640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d00b4700 0x7f10d00cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d549c640 1 -- 172.21.15.46:0/2892364492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d009e470 con 0x7f10d00b3f50 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d549c640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d01dadb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f10c000c760 tx=0x7f10c000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0003b70 con 0x7f10d00b3f50 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c0003d30 con 0x7f10d00b3f50 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c000aa90 con 0x7f10d00b3f50 2024-05-09T03:16:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d009e710 con 0x7f10d00b3f50 2024-05-09T03:16:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.234+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d009ecd0 con 0x7f10d00b3f50 2024-05-09T03:16:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.235+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d006a270 con 0x7f10d00b3f50 2024-05-09T03:16:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.236+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f10c000ac30 con 0x7f10d00b3f50 2024-05-09T03:16:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.237+0000 7f10cdffb640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10bc0762d0 0x7f10bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.237+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f10c0097560 con 0x7f10d00b3f50 2024-05-09T03:16:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.237+0000 7f10d4c9b640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10bc0762d0 0x7f10bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.238+0000 7f10d4c9b640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10bc0762d0 0x7f10bc0787e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f10c4006fd0 tx=0x7f10c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:12.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.242+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f10c0060e60 con 0x7f10d00b3f50 2024-05-09T03:16:12.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:12.469+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f10d00b5350 con 0x7f10d00b3f50 2024-05-09T03:16:13.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.122+0000 7f10cdffb640 1 -- 172.21.15.46:0/2892364492 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) ==== 161+0+0 (secure 0 0 0) 0x7f10c0060760 con 0x7f10d00b3f50 2024-05-09T03:16:13.123 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10bc0762d0 msgr2=0x7f10bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10bc0762d0 0x7f10bc0787e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f10c4006fd0 tx=0x7f10c4008040 comp rx=0 tx=0).stop 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 msgr2=0x7f10d01dadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d01dadb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f10c000c760 tx=0x7f10c000cc20 comp rx=0 tx=0).stop 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 shutdown_connections 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f10d00d2800 0x7f10d00ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d00b4700 0x7f10d00cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10bc0762d0 0x7f10bc0787e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 --2- 172.21.15.46:0/2892364492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00b3f50 0x7f10d01dadb0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 >> 172.21.15.46:0/2892364492 conn(0x7f10d00b55d0 msgr2=0x7f10d0072db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 shutdown_connections 2024-05-09T03:16:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.125+0000 7f10d7700640 1 -- 172.21.15.46:0/2892364492 wait complete. 2024-05-09T03:16:13.139 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.262+0000 7f629fdeb640 1 Processor -- start 2024-05-09T03:16:13.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.263+0000 7f629fdeb640 1 -- start start 2024-05-09T03:16:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629fdeb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62980d3fc0 0x7f629816b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629fdeb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62980b5550 con 0x7f62980b7e70 2024-05-09T03:16:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629fdeb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62980d44e0 con 0x7f62980b5140 2024-05-09T03:16:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629fdeb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f62980d3480 con 0x7f62980d3fc0 2024-05-09T03:16:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629e388640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62980d3fc0 0x7f629816b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:13.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629e388640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f62980d3fc0 0x7f629816b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45334/0 (socket says 172.21.15.46:45334) 2024-05-09T03:16:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629d386640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f62980b7e70 0x7f62980d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43486/0 (socket says 172.21.15.46:43486) 2024-05-09T03:16:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.264+0000 7f629e388640 1 -- 172.21.15.46:0/2558478043 learned_addr learned my addr 172.21.15.46:0/2558478043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.265+0000 7f629e388640 1 -- 172.21.15.46:0/2558478043 >> v1:172.21.15.88:6789/0 conn(0x7f62980b5140 legacy=0x7f62980b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.265+0000 7f629e388640 1 -- 172.21.15.46:0/2558478043 >> v1:172.21.15.46:6789/0 conn(0x7f62980b7e70 legacy=0x7f62980d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.265+0000 7f629e388640 1 -- 172.21.15.46:0/2558478043 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62980d47c0 con 0x7f62980d3fc0 2024-05-09T03:16:13.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.265+0000 7f629cb85640 1 -- 172.21.15.46:0/2558478043 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1438710833 0 0) 0x7f62980b5550 con 0x7f62980b7e70 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.265+0000 7f629e388640 1 --2- 172.21.15.46:0/2558478043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980d3fc0 0x7f629816b0d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6290009e50 tx=0x7f629002a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97ef2a874e21fda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.265+0000 7f629cb85640 1 -- 172.21.15.46:0/2558478043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290037070 con 0x7f62980d3fc0 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629cb85640 1 -- 172.21.15.46:0/2558478043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290032630 con 0x7f62980d3fc0 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629cb85640 1 -- 172.21.15.46:0/2558478043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629002e050 con 0x7f62980d3fc0 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 -- 172.21.15.46:0/2558478043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980d3fc0 msgr2=0x7f629816b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 --2- 172.21.15.46:0/2558478043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980d3fc0 0x7f629816b0d0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6290009e50 tx=0x7f629002a840 comp rx=0 tx=0).stop 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 -- 172.21.15.46:0/2558478043 shutdown_connections 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 --2- 172.21.15.46:0/2558478043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62980d3fc0 0x7f629816b0d0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 -- 172.21.15.46:0/2558478043 >> 172.21.15.46:0/2558478043 conn(0x7f629808d3f0 msgr2=0x7f629808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:13.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 -- 172.21.15.46:0/2558478043 shutdown_connections 2024-05-09T03:16:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.266+0000 7f629fdeb640 1 -- 172.21.15.46:0/2558478043 wait complete. 2024-05-09T03:16:13.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.268+0000 7f629fdeb640 1 Processor -- start 2024-05-09T03:16:13.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.268+0000 7f629fdeb640 1 -- start start 2024-05-09T03:16:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.269+0000 7f629fdeb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62980b5140 0x7f62980c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.269+0000 7f629fdeb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62980d3fc0 0x7f629815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.269+0000 7f629fdeb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 0x7f62980cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.269+0000 7f629fdeb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62980d5580 con 0x7f6298157450 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.269+0000 7f629fdeb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f62980d4d80 con 0x7f62980b5140 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.269+0000 7f629fdeb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f62980d5700 con 0x7f62980d3fc0 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 0x7f62980cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 0x7f62980cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45350/0 (socket says 172.21.15.46:45350) 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 -- 172.21.15.46:0/1451383800 learned_addr learned my addr 172.21.15.46:0/1451383800 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629db87640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62980b5140 0x7f62980c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629d386640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62980d3fc0 0x7f629815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 -- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62980d3fc0 msgr2=0x7f629815b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62980d3fc0 0x7f629815b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 -- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62980b5140 msgr2=0x7f62980c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62980b5140 0x7f62980c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 -- 172.21.15.46:0/1451383800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6298159f80 con 0x7f6298157450 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629d386640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62980d3fc0 0x7f629815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629e388640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 0x7f62980cd070 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6290009f70 tx=0x7f6290004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290037050 con 0x7f6298157450 2024-05-09T03:16:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290043070 con 0x7f6298157450 2024-05-09T03:16:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629815a260 con 0x7f6298157450 2024-05-09T03:16:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290009510 con 0x7f6298157450 2024-05-09T03:16:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.270+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629815a820 con 0x7f6298157450 2024-05-09T03:16:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.271+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6298159cf0 con 0x7f6298157450 2024-05-09T03:16:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.272+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6290007b70 con 0x7f6298157450 2024-05-09T03:16:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.273+0000 7f628e7fc640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f627c076300 0x7f627c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.273+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f62900b80b0 con 0x7f6298157450 2024-05-09T03:16:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.273+0000 7f629db87640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f627c076300 0x7f627c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.274+0000 7f629db87640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f627c076300 0x7f627c078810 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6294000f60 tx=0x7f6294009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.278+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f62900819b0 con 0x7f6298157450 2024-05-09T03:16:13.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:13.544+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f62980b4af0 con 0x7f6298157450 2024-05-09T03:16:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.124+0000 7f628e7fc640 1 -- 172.21.15.46:0/1451383800 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v89) ==== 107+0+0 (secure 0 0 0) 0x7f62980b4af0 con 0x7f6298157450 2024-05-09T03:16:14.125 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f627c076300 msgr2=0x7f627c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f627c076300 0x7f627c078810 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6294000f60 tx=0x7f6294009040 comp rx=0 tx=0).stop 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 msgr2=0x7f62980cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 0x7f62980cd070 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6290009f70 tx=0x7f6290004a40 comp rx=0 tx=0).stop 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 shutdown_connections 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f627c076300 0x7f627c078810 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6298157450 0x7f62980cd070 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62980d3fc0 0x7f629815b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 --2- 172.21.15.46:0/1451383800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62980b5140 0x7f62980c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 >> 172.21.15.46:0/1451383800 conn(0x7f629808d3f0 msgr2=0x7f62980d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 shutdown_connections 2024-05-09T03:16:14.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.127+0000 7f629fdeb640 1 -- 172.21.15.46:0/1451383800 wait complete. 2024-05-09T03:16:14.142 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:14.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.202+0000 7fad2cb16f40 1 Processor -- start 2024-05-09T03:16:14.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.202+0000 7fad2cb16f40 1 -- start start 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad2cb16f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 0x5625012526f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad2cb16f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5625011c85f0 0x5625011a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad2cb16f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562501254150 0x5625011a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad2cb16f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5625011c95f0 con 0x5625011c03a0 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad2cb16f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562501126f50 con 0x562501254150 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad2cb16f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5625011c98d0 con 0x5625011c85f0 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 0x5625012526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 0x5625012526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45370/0 (socket says 172.21.15.46:45370) 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 -- 172.21.15.46:0/2493858177 learned_addr learned my addr 172.21.15.46:0/2493858177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad25ddd640 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5625011c85f0 0x5625011a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad26ddf640 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562501254150 0x5625011a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 -- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5625011c85f0 msgr2=0x5625011a99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5625011c85f0 0x5625011a99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 -- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562501254150 msgr2=0x5625011a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562501254150 0x5625011a9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 -- 172.21.15.46:0/2493858177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625011ab020 con 0x5625011c03a0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.204+0000 7fad265de640 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 0x5625012526f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fad18001d20 tx=0x7fad1800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb42cc11cfcb914 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad24ddb640 1 -- 172.21.15.46:0/2493858177 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1800e460 con 0x5625011c03a0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad24ddb640 1 -- 172.21.15.46:0/2493858177 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad18012070 con 0x5625011c03a0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad24ddb640 1 -- 172.21.15.46:0/2493858177 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1800e880 con 0x5625011c03a0 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 -- 172.21.15.46:0/2493858177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 msgr2=0x5625012526f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 0x5625012526f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fad18001d20 tx=0x7fad1800b5b0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 -- 172.21.15.46:0/2493858177 shutdown_connections 2024-05-09T03:16:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562501254150 0x5625011a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5625011c85f0 0x5625011a99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/2493858177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c03a0 0x5625012526f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 -- 172.21.15.46:0/2493858177 >> 172.21.15.46:0/2493858177 conn(0x5625010a2ac0 msgr2=0x5625011ad6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 -- 172.21.15.46:0/2493858177 shutdown_connections 2024-05-09T03:16:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.205+0000 7fad2cb16f40 1 -- 172.21.15.46:0/2493858177 wait complete. 2024-05-09T03:16:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.206+0000 7fad2cb16f40 1 Processor -- start 2024-05-09T03:16:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.206+0000 7fad2cb16f40 1 -- start start 2024-05-09T03:16:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad2cb16f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5625011c03a0 0x56250124be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad2cb16f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 0x56250117ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad2cb16f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562501254150 0x56250117f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad2cb16f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5625011ae5c0 con 0x5625011c85f0 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad2cb16f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562501254600 con 0x5625011c03a0 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad2cb16f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5625012ea060 con 0x562501254150 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 0x56250117ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 0x56250117ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45386/0 (socket says 172.21.15.46:45386) 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 learned_addr learned my addr 172.21.15.46:0/420693403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5625011c03a0 0x56250124be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad26ddf640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562501254150 0x56250117f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562501254150 msgr2=0x56250117f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562501254150 0x56250117f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5625011c03a0 msgr2=0x56250124be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5625011c03a0 0x56250124be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56250117f8f0 con 0x5625011c85f0 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5625011c03a0 0x56250124be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:14.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.208+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 0x56250117ebe0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fad1c045f50 tx=0x7fad1c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.209+0000 7fad177fe640 1 -- 172.21.15.46:0/420693403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c053970 con 0x5625011c85f0 2024-05-09T03:16:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.209+0000 7fad177fe640 1 -- 172.21.15.46:0/420693403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad1c053b10 con 0x5625011c85f0 2024-05-09T03:16:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.209+0000 7fad177fe640 1 -- 172.21.15.46:0/420693403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c069440 con 0x5625011c85f0 2024-05-09T03:16:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.209+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56250117fd80 con 0x5625011c85f0 2024-05-09T03:16:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.209+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562501180340 con 0x5625011c85f0 2024-05-09T03:16:14.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.210+0000 7fad177fe640 1 -- 172.21.15.46:0/420693403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fad1c05c420 con 0x5625011c85f0 2024-05-09T03:16:14.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.211+0000 7fad177fe640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fad08074ce0 0x7fad080771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.211+0000 7fad177fe640 1 -- 172.21.15.46:0/420693403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7fad1c0daad0 con 0x5625011c85f0 2024-05-09T03:16:14.213 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.211+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fad08074ce0 0x7fad080771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.211+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5625011a99f0 0x5625012e96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.211+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625011ab4a0 con 0x5625011a99f0 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.211+0000 7fad26ddf640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5625011a99f0 0x5625012e96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.212+0000 7fad26ddf640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5625011a99f0 0x5625012e96c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.212+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fad08074ce0 0x7fad080771f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fad18001d20 tx=0x7fad18019040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.214+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v83'11 uv4 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.214+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012e9c00 con 0x5625011a99f0 2024-05-09T03:16:14.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.215+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v83'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.215+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f57d0 con 0x5625011a99f0 2024-05-09T03:16:14.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.218+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v83'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.218 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:14.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.218+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6b80 con 0x5625011a99f0 2024-05-09T03:16:14.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.220+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v83'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.220+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6f20 con 0x5625011a99f0 2024-05-09T03:16:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.223+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v83'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.223+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f5b70 con 0x5625011a99f0 2024-05-09T03:16:14.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.225+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v83'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.225+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f5f60 con 0x5625011a99f0 2024-05-09T03:16:14.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.228+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v83'17 uv6 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.228+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012f6350 con 0x5625011a99f0 2024-05-09T03:16:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.230+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v83'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.230+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6350 con 0x5625011a99f0 2024-05-09T03:16:14.233 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.233+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v83'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.233+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.235+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v83'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.235+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.236+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v83'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.237+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.238+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v83'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.238+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.239+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v83'23 uv10 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.239+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.240+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v83'24 uv24 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.240+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.242 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.242+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v83'25 uv25 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.242+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.243+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v83'26 uv26 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.243+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.244+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v83'27 uv27 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.244+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f6300 con 0x5625011a99f0 2024-05-09T03:16:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.245+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v83'28 uv28 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.245+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.247+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v83'29 uv29 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.247+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.248+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v83'30 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.248+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.249+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v83'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.249+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.251 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.250+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v83'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.250+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f82d0 con 0x5625011a99f0 2024-05-09T03:16:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.251+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v83'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.251+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f82d0 con 0x5625011a99f0 2024-05-09T03:16:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.253+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v83'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.253+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.254+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v83'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.254+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.255+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v83'36 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.255+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.256+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v83'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.256+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.258 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.257+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v83'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.257+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f82d0 con 0x5625011a99f0 2024-05-09T03:16:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.259+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v83'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.259+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f82d0 con 0x5625011a99f0 2024-05-09T03:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.260+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v83'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.260+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012f72c0 con 0x5625011a99f0 2024-05-09T03:16:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.261+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v83'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.261+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5625012f77d0 0x5625012f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.261+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fa130 con 0x5625012f77d0 2024-05-09T03:16:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.261+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5625012f77d0 0x5625012f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.262+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5625012f77d0 0x5625012f9bf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.263+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v83'21 uv17 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.263+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012fa4d0 con 0x5625012f77d0 2024-05-09T03:16:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.265+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v83'22 uv22 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.265+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fa870 con 0x5625012f77d0 2024-05-09T03:16:14.266 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.266+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v83'23 uv23 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.266+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fb0c0 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.267+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v83'24 uv24 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.267+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fb460 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.268+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v83'25 uv25 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.268+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fb800 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.269+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v83'26 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.269+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.271+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v83'27 uv27 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.271+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.272+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v83'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.272 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:14.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.272+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.273+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v83'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.273+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.274+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v83'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.274+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.275+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v83'31 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.275+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.277+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v83'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.277+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.278 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:14.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.278+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v83'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.278+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff600 con 0x5625012f77d0 2024-05-09T03:16:14.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.279+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v83'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.279+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff600 con 0x5625012f77d0 2024-05-09T03:16:14.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.281+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v83'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.281+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.282+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v83'36 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.282+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.283+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v83'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.283+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.284 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.284+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v83'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.284+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fcbb0 con 0x5625012f77d0 2024-05-09T03:16:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.285+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v83'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.285+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fcbb0 con 0x5625012f77d0 2024-05-09T03:16:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.286+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v83'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.286+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625012f77d0 2024-05-09T03:16:14.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.288+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v83'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x5625012f77d0 2024-05-09T03:16:14.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.288+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625011a99f0 2024-05-09T03:16:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.289+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v83'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.289+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625011a99f0 2024-05-09T03:16:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.290+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v83'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.290+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625011a99f0 2024-05-09T03:16:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.291+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v83'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.291 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.291+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fcbb0 con 0x5625011a99f0 2024-05-09T03:16:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.292+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v83'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.292+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fcbb0 con 0x5625011a99f0 2024-05-09T03:16:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.293+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v83'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.293+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fbba0 con 0x5625011a99f0 2024-05-09T03:16:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.294+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v83'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.294+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5625012fc0b0 0x5625012fe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.294+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fea10 con 0x5625012fc0b0 2024-05-09T03:16:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.294+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5625012fc0b0 0x5625012fe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.294+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5625012fc0b0 0x5625012fe4d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.295+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v83'5 uv4 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.295+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012feeb0 con 0x5625012fc0b0 2024-05-09T03:16:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.296+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v83'6 uv6 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.296+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff3b0 con 0x5625012fc0b0 2024-05-09T03:16:14.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.298+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v83'7 uv7 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.298 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:14.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.298+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012fac90 con 0x5625012fc0b0 2024-05-09T03:16:14.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.299+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v83'8 uv8 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.299+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ffcc0 con 0x5625011a99f0 2024-05-09T03:16:14.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.300+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v83'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.300+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012ffcc0 con 0x5625011a99f0 2024-05-09T03:16:14.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.301+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v83'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.301+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ffcc0 con 0x5625011a99f0 2024-05-09T03:16:14.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.303+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v83'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.303 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:14.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.303+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501300cd0 con 0x5625011a99f0 2024-05-09T03:16:14.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.304+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v83'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.304+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501300cd0 con 0x5625011a99f0 2024-05-09T03:16:14.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.305+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v83'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.305+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ffcc0 con 0x5625011a99f0 2024-05-09T03:16:14.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.307+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v83'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.307+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562501300340 0x562501300750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.307+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.307+0000 7fad26ddf640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562501300340 0x562501300750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.307+0000 7fad26ddf640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562501300340 0x562501300750 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.308+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v83'4 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.308+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.309+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v83'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.309+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.310+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v83'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.311 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:14.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.310+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501304140 con 0x562501300340 2024-05-09T03:16:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.312+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v83'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.312+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501304100 con 0x562501300340 2024-05-09T03:16:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.313+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v83'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.313+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.314+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v83'9 uv9 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.314+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.315+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v83'10 uv3 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.315+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.316+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v83'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.316+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.317+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v83'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.318 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.317+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625013030f0 con 0x562501300340 2024-05-09T03:16:14.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.319+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v83'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.319+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501303500 con 0x562501300340 2024-05-09T03:16:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.320+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v83'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.320+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562501303a10 0x562501303e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.320+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306550 con 0x562501303a10 2024-05-09T03:16:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.320+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562501303a10 0x562501303e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.321+0000 7fad265de640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562501303a10 0x562501303e20 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.322+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v83'2 uv1 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x562501303a10 2024-05-09T03:16:14.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.322+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x562501306550 con 0x562501303a10 2024-05-09T03:16:14.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.323+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v83'3 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x562501303a10 2024-05-09T03:16:14.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.323+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x562501303a10 2024-05-09T03:16:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.324+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v83'4 uv4 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x562501303a10 2024-05-09T03:16:14.325 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:14.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.324+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x562501303a10 2024-05-09T03:16:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.325+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v83'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x562501303a10 2024-05-09T03:16:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.325+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x562501303a10 2024-05-09T03:16:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.326+0000 7fad265de640 1 -- 172.21.15.46:0/420693403 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v83'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1800e830 con 0x562501303a10 2024-05-09T03:16:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.326+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x5625011a99f0 2024-05-09T03:16:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.327+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v83'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.327+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x562501306940 con 0x5625011a99f0 2024-05-09T03:16:14.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.328+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v83'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.328+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x5625011a99f0 2024-05-09T03:16:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.329+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v83'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.330 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.329+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501307950 con 0x5625011a99f0 2024-05-09T03:16:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.330+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v83'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.330+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501307950 con 0x5625011a99f0 2024-05-09T03:16:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.331+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v83'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.331+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x5625011a99f0 2024-05-09T03:16:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.332+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v83'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x5625011a99f0 2024-05-09T03:16:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.333+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501306940 con 0x5625012fc0b0 2024-05-09T03:16:14.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.334+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v83'4 uv3 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.334+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x562501306ce0 con 0x5625012fc0b0 2024-05-09T03:16:14.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.335+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v83'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.335+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x562501307080 con 0x5625012fc0b0 2024-05-09T03:16:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.336+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v83'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.336 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.336+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff860 con 0x5625012fc0b0 2024-05-09T03:16:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.337+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v83'7 uv7 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.337+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff860 con 0x5625012fc0b0 2024-05-09T03:16:14.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.338+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v83'8 uv8 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.338+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff860 con 0x5625012fc0b0 2024-05-09T03:16:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.339+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v83'9 uv1 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.339+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x5625012ff860 con 0x5625012fc0b0 2024-05-09T03:16:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.340+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v83'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.340+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff860 con 0x5625012fc0b0 2024-05-09T03:16:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.341+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v83'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.342 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.341+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x5625012ff860 con 0x5625012fc0b0 2024-05-09T03:16:14.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.342+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v83'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.342+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a2d0 con 0x5625012fc0b0 2024-05-09T03:16:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.343+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v83'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.343+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562501307ff0 0x56250130a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.343+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a410 con 0x562501307ff0 2024-05-09T03:16:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.343+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562501307ff0 0x56250130a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.344+0000 7fad25ddd640 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562501307ff0 0x56250130a970 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.345+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v83'4 uv3 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x562501307ff0 2024-05-09T03:16:14.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.345+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x56250130a410 con 0x562501307ff0 2024-05-09T03:16:14.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.346+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v83'5 uv5 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x562501307ff0 2024-05-09T03:16:14.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.346+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a410 con 0x562501307ff0 2024-05-09T03:16:14.347 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:14.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.347+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v83'6 uv6 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x562501307ff0 2024-05-09T03:16:14.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.347+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a410 con 0x562501307ff0 2024-05-09T03:16:14.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.348+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v83'7 uv7 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x562501307ff0 2024-05-09T03:16:14.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.348+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a410 con 0x5625012fc0b0 2024-05-09T03:16:14.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.350+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v83'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.350+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x56250130a410 con 0x5625012fc0b0 2024-05-09T03:16:14.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.351+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v83'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.351+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a410 con 0x5625012fc0b0 2024-05-09T03:16:14.352 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.352+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v83'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.352+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130a410 con 0x5625012fc0b0 2024-05-09T03:16:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.353+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v83'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.353+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130bf10 con 0x5625012fc0b0 2024-05-09T03:16:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.354+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v83'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.354+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x5625012fc0b0 2024-05-09T03:16:14.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.355+0000 7fad25ddd640 1 -- 172.21.15.46:0/420693403 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v83'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad1c051070 con 0x5625012fc0b0 2024-05-09T03:16:14.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.355+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.356+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v83'2 uv1 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.356+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.358+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v83'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.358+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.361 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.360+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v83'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.360+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.363+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v83'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.363+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.366+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v83'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.366+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.368+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v83'7 uv3 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.368+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.371+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v83'8 uv8 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.371+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.374 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:14.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.374+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v83'9 uv9 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.374+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.376+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v83'10 uv10 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.376+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.379+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v83'11 uv11 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.379+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.382+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v83'12 uv6 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.382+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.383+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v83'13 uv13 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.383+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.385 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.384+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v83'14 uv14 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.384+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130bec0 con 0x562501300340 2024-05-09T03:16:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.385+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v83'15 uv15 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.385+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130bec0 con 0x562501300340 2024-05-09T03:16:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.387+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v83'16 uv16 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.387+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130aeb0 con 0x562501300340 2024-05-09T03:16:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.388+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v83'17 uv17 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.388+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x56250130b360 con 0x562501300340 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v83'18 uv18 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fad20002960 con 0x562501300340 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5625011a99f0 msgr2=0x5625012e96c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5625011a99f0 0x5625012e96c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5625012fc0b0 msgr2=0x5625012fe4d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5625012fc0b0 0x5625012fe4d0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562501307ff0 msgr2=0x56250130a970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562501307ff0 0x56250130a970 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5625012f77d0 msgr2=0x5625012f9bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5625012f77d0 0x5625012f9bf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562501300340 msgr2=0x562501300750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad26ddf640 1 -- 172.21.15.46:0/420693403 reap_dead start 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562501300340 0x562501300750 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562501303a10 msgr2=0x562501303e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562501303a10 0x562501303e20 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fad08074ce0 msgr2=0x7fad080771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fad08074ce0 0x7fad080771f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fad18001d20 tx=0x7fad18019040 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 msgr2=0x56250117ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 0x56250117ebe0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fad1c045f50 tx=0x7fad1c002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 shutdown_connections 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562501303a10 0x562501303e20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562501300340 0x562501300750 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fad08074ce0 0x7fad080771f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 --2- 172.21.15.46:0/420693403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625011c85f0 0x56250117ebe0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 >> 172.21.15.46:0/420693403 conn(0x5625010a2ac0 msgr2=0x5625012526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 shutdown_connections 2024-05-09T03:16:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.389+0000 7fad2cb16f40 1 -- 172.21.15.46:0/420693403 wait complete. 2024-05-09T03:16:14.393 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-05-09T03:16:14.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.448+0000 7f375130df40 1 Processor -- start 2024-05-09T03:16:14.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.448+0000 7f375130df40 1 -- start start 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f375130df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c98a7e0 0x55b45c9b8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f375130df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c9bc7b0 0x55b45ca5c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f375130df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45ca5e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f375130df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b45c933f50 con 0x55b45ca64810 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f375130df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b45c937f90 con 0x55b45c98a7e0 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f375130df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b45c9d14d0 con 0x55b45c9bc7b0 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45ca5e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374add5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c98a7e0 0x55b45c9b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45ca5e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45394/0 (socket says 172.21.15.46:45394) 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 -- 172.21.15.46:0/3206363230 learned_addr learned my addr 172.21.15.46:0/3206363230 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f3743fff640 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c9bc7b0 0x55b45ca5c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 -- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c9bc7b0 msgr2=0x55b45ca5c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c9bc7b0 0x55b45ca5c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 -- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c98a7e0 msgr2=0x55b45c9b8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c98a7e0 0x55b45c9b8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374b5d6640 1 -- 172.21.15.46:0/3206363230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b45ca66c30 con 0x55b45ca64810 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.449+0000 7f374add5640 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c98a7e0 0x55b45c9b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f374b5d6640 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45ca5e230 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3744001690 tx=0x7f3744001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee182db7328e686 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f3749dd3640 1 -- 172.21.15.46:0/3206363230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744017070 con 0x55b45ca64810 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f3749dd3640 1 -- 172.21.15.46:0/3206363230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374400f710 con 0x55b45ca64810 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f3749dd3640 1 -- 172.21.15.46:0/3206363230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374400fb00 con 0x55b45ca64810 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 -- 172.21.15.46:0/3206363230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 msgr2=0x55b45ca5e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45ca5e230 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3744001690 tx=0x7f3744001b50 comp rx=0 tx=0).stop 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 -- 172.21.15.46:0/3206363230 shutdown_connections 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45ca5e230 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c9bc7b0 0x55b45ca5c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 --2- 172.21.15.46:0/3206363230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c98a7e0 0x55b45c9b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 -- 172.21.15.46:0/3206363230 >> 172.21.15.46:0/3206363230 conn(0x55b45c8afac0 msgr2=0x55b45c9d4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 -- 172.21.15.46:0/3206363230 shutdown_connections 2024-05-09T03:16:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.450+0000 7f375130df40 1 -- 172.21.15.46:0/3206363230 wait complete. 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.451+0000 7f375130df40 1 Processor -- start 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.451+0000 7f375130df40 1 -- start start 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f375130df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c98a7e0 0x55b45ca5d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f375130df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c9bc7b0 0x55b45ca52dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f375130df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45c9ba440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f375130df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b45ca66ed0 con 0x55b45ca64810 2024-05-09T03:16:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f375130df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b45c9b9ba0 con 0x55b45c9bc7b0 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f375130df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b45c9b9e80 con 0x55b45c98a7e0 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45c9ba440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45c9ba440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45406/0 (socket says 172.21.15.46:45406) 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 -- 172.21.15.46:0/3126813813 learned_addr learned my addr 172.21.15.46:0/3126813813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f3743fff640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c9bc7b0 0x55b45ca52dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374add5640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c98a7e0 0x55b45ca5d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 -- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c98a7e0 msgr2=0x55b45ca5d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b45c98a7e0 0x55b45ca5d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 -- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c9bc7b0 msgr2=0x55b45ca52dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b45c9bc7b0 0x55b45ca52dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 -- 172.21.15.46:0/3126813813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b45ca63150 con 0x55b45ca64810 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.453+0000 7f374b5d6640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45c9ba440 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x55b45c9d5920 tx=0x7f3744001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.454+0000 7f3748dd1640 1 -- 172.21.15.46:0/3126813813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744022030 con 0x55b45ca64810 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.454+0000 7f3748dd1640 1 -- 172.21.15.46:0/3126813813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3744006020 con 0x55b45ca64810 2024-05-09T03:16:14.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.454+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b45ca5cb50 con 0x55b45ca64810 2024-05-09T03:16:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.454+0000 7f3748dd1640 1 -- 172.21.15.46:0/3126813813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744017030 con 0x55b45ca64810 2024-05-09T03:16:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.454+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b45ca5c110 con 0x55b45ca64810 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.455+0000 7f3748dd1640 1 -- 172.21.15.46:0/3126813813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3744025070 con 0x55b45ca64810 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f3748dd1640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f372c074ce0 0x7f372c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f3748dd1640 1 -- 172.21.15.46:0/3126813813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f3744096b30 con 0x55b45ca64810 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f374add5640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f372c074ce0 0x7f372c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f375130df40 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b45c9ce560 0x55b45c9ce970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x55b45ca5af70 con 0x55b45c9ce560 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f3743fff640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b45c9ce560 0x55b45c9ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.456+0000 7f374add5640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f372c074ce0 0x7f372c0771f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f373c004170 tx=0x7f373c001650 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.457+0000 7f3743fff640 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b45c9ce560 0x55b45c9ce970 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f3743fff640 1 -- 172.21.15.46:0/3126813813 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v83'42 uv42 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f373405f780 con 0x55b45c9ce560 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b45c9ce560 msgr2=0x55b45c9ce970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b45c9ce560 0x55b45c9ce970 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f372c074ce0 msgr2=0x7f372c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f372c074ce0 0x7f372c0771f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f373c004170 tx=0x7f373c001650 comp rx=0 tx=0).stop 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 msgr2=0x55b45c9ba440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 --2- 172.21.15.46:0/3126813813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b45ca64810 0x55b45c9ba440 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x55b45c9d5920 tx=0x7f3744001250 comp rx=0 tx=0).stop 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f374b5d6640 1 -- 172.21.15.46:0/3126813813 reap_dead start 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 shutdown_connections 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 >> 172.21.15.46:0/3126813813 conn(0x55b45c8afac0 msgr2=0x55b45c9d4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 shutdown_connections 2024-05-09T03:16:14.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.458+0000 7f375130df40 1 -- 172.21.15.46:0/3126813813 wait complete. 2024-05-09T03:16:14.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.578+0000 7f7edc9b2640 1 Processor -- start 2024-05-09T03:16:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.578+0000 7f7edc9b2640 1 -- start start 2024-05-09T03:16:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7edc9b2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ed80b5140 0x7f7ed80b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7edc9b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80d3fc0 0x7f7ed80b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7edc9b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed80b80b0 con 0x7f7ed80b7ca0 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7edc9b2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed80d3480 con 0x7f7ed80b5140 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7edc9b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed80b5550 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed6d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80d3fc0 0x7f7ed80b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ed80b5140 0x7f7ed80b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ed80b5140 0x7f7ed80b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:48816/0 (socket says 172.21.15.46:48816) 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 -- 172.21.15.46:0/404309180 learned_addr learned my addr 172.21.15.46:0/404309180 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed5d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7ed80b7ca0 0x7f7ed80d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34972/0 (socket says 172.21.15.46:34972) 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 -- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80d3fc0 msgr2=0x7f7ed80b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 --2- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80d3fc0 0x7f7ed80b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 -- 172.21.15.46:0/404309180 >> v1:172.21.15.88:6789/0 conn(0x7f7ed80b7ca0 legacy=0x7f7ed80d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed559a640 1 -- 172.21.15.46:0/404309180 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076367548 0 0) 0x7f7ec0001110 con 0x7f7ed80b7ca0 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed659c640 1 -- 172.21.15.46:0/404309180 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed80d44e0 con 0x7f7ed80b5140 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.580+0000 7f7ed6d9d640 1 --2- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80d3fc0 0x7f7ed80b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.581+0000 7f7ed659c640 1 --2- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80b5140 0x7f7ed80b7760 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009e50 tx=0x7f7ec802a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c8a1fe54e40fdf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.581+0000 7f7ed559a640 1 -- 172.21.15.46:0/404309180 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8036070 con 0x7f7ed80b5140 2024-05-09T03:16:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.581+0000 7f7ed559a640 1 -- 172.21.15.46:0/404309180 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec8031600 con 0x7f7ed80b5140 2024-05-09T03:16:14.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.581+0000 7f7ed559a640 1 -- 172.21.15.46:0/404309180 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80319d0 con 0x7f7ed80b5140 2024-05-09T03:16:14.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 -- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80b5140 msgr2=0x7f7ed80b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80b5140 0x7f7ed80b7760 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009e50 tx=0x7f7ec802a7a0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 -- 172.21.15.46:0/404309180 shutdown_connections 2024-05-09T03:16:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80d3fc0 0x7f7ed80b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/404309180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80b5140 0x7f7ed80b7760 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 -- 172.21.15.46:0/404309180 >> 172.21.15.46:0/404309180 conn(0x7f7ed808d3f0 msgr2=0x7f7ed808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 -- 172.21.15.46:0/404309180 shutdown_connections 2024-05-09T03:16:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.582+0000 7f7edc9b2640 1 -- 172.21.15.46:0/404309180 wait complete. 2024-05-09T03:16:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.584+0000 7f7edc9b2640 1 Processor -- start 2024-05-09T03:16:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.584+0000 7f7edc9b2640 1 -- start start 2024-05-09T03:16:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.585+0000 7f7edc9b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ed80b5140 0x7f7ed815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.585+0000 7f7edc9b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80b7ca0 0x7f7ed8156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.585+0000 7f7edc9b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80d3fc0 0x7f7ed8156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.585+0000 7f7edc9b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed80d4d80 con 0x7f7ed80b5140 2024-05-09T03:16:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.585+0000 7f7edc9b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed80d4aa0 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.585+0000 7f7edc9b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed80d5580 con 0x7f7ed80b7ca0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ed80b5140 0x7f7ed815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ed80b5140 0x7f7ed815bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45414/0 (socket says 172.21.15.46:45414) 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed659c640 1 -- 172.21.15.46:0/3525814202 learned_addr learned my addr 172.21.15.46:0/3525814202 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed5d9b640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80b7ca0 0x7f7ed8156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80d3fc0 0x7f7ed8156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 -- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80b7ca0 msgr2=0x7f7ed8156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80b7ca0 0x7f7ed8156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 -- 172.21.15.46:0/3525814202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ed80b5140 msgr2=0x7f7ed815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ed80b5140 0x7f7ed815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 -- 172.21.15.46:0/3525814202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed8159860 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed5d9b640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80b7ca0 0x7f7ed8156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7ed6d9d640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80d3fc0 0x7f7ed8156790 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80d09a0 tx=0x7f7ed000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0017070 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.586+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed000c910 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.587+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed8159b40 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.587+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000eea0 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.587+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed815a100 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.588+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed806a270 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.588+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7ed001a070 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.589+0000 7f7eceffd640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7eb80762d0 0x7f7eb80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.589+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f7ed0097430 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.589+0000 7f7ed659c640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7eb80762d0 0x7f7eb80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.590+0000 7f7ed659c640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7eb80762d0 0x7f7eb80787e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7ec802a7e0 tx=0x7f7ec8009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:14.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.594+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7ed00605b0 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.826+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f7ed8160670 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.970+0000 7f7eceffd640 1 -- 172.21.15.46:0/3525814202 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v90) ==== 162+0+0 (secure 0 0 0) 0x7f7ed00648b0 con 0x7f7ed80d3fc0 2024-05-09T03:16:14.971 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7eb80762d0 msgr2=0x7f7eb80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7eb80762d0 0x7f7eb80787e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7ec802a7e0 tx=0x7f7ec8009000 comp rx=0 tx=0).stop 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80d3fc0 msgr2=0x7f7ed8156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80d3fc0 0x7f7ed8156790 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80d09a0 tx=0x7f7ed000ec50 comp rx=0 tx=0).stop 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 shutdown_connections 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed80d3fc0 0x7f7ed8156790 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ed80b7ca0 0x7f7ed8156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7eb80762d0 0x7f7eb80787e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 --2- 172.21.15.46:0/3525814202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ed80b5140 0x7f7ed815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:14.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 >> 172.21.15.46:0/3525814202 conn(0x7f7ed808d3f0 msgr2=0x7f7ed81630f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:14.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 shutdown_connections 2024-05-09T03:16:14.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:14.973+0000 7f7edc9b2640 1 -- 172.21.15.46:0/3525814202 wait complete. 2024-05-09T03:16:15.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:15.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.513+0000 7f085ff00640 1 Processor -- start 2024-05-09T03:16:15.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.513+0000 7f085ff00640 1 -- start start 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085ff00640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08580d57a0 0x7f0858160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085ff00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580b4af0 0x7f08580b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085ff00640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f08580b77e0 0x7f08580b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085ff00640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f08580d5bb0 con 0x7f08580d57a0 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085ff00640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f08580b8130 con 0x7f08580b77e0 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085ff00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f08580d49e0 con 0x7f08580b4af0 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08580d57a0 0x7f0858160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08580d57a0 0x7f0858160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45418/0 (socket says 172.21.15.46:45418) 2024-05-09T03:16:15.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 -- 172.21.15.46:0/3280112886 learned_addr learned my addr 172.21.15.46:0/3280112886 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085d49b640 1 --2- 172.21.15.46:0/3280112886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580b4af0 0x7f08580b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085e49d640 1 --2- 172.21.15.46:0/3280112886 >> v2:172.21.15.88:3300/0 conn(0x7f08580b77e0 0x7f08580b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 -- 172.21.15.46:0/3280112886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580b4af0 msgr2=0x7f08580b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 --2- 172.21.15.46:0/3280112886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580b4af0 0x7f08580b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 -- 172.21.15.46:0/3280112886 >> v2:172.21.15.88:3300/0 conn(0x7f08580b77e0 msgr2=0x7f08580b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 --2- 172.21.15.46:0/3280112886 >> v2:172.21.15.88:3300/0 conn(0x7f08580b77e0 0x7f08580b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.515+0000 7f085dc9c640 1 -- 172.21.15.46:0/3280112886 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08580d4cc0 con 0x7f08580d57a0 2024-05-09T03:16:15.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.516+0000 7f085dc9c640 1 --2- 172.21.15.46:0/3280112886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580d57a0 0x7f0858160610 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f085808f200 tx=0x7f085402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3cd347bfb6bea8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:15.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.516+0000 7f085cc9a640 1 -- 172.21.15.46:0/3280112886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854037070 con 0x7f08580d57a0 2024-05-09T03:16:15.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.516+0000 7f085cc9a640 1 -- 172.21.15.46:0/3280112886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0854032440 con 0x7f08580d57a0 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.516+0000 7f085cc9a640 1 -- 172.21.15.46:0/3280112886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854032810 con 0x7f08580d57a0 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 -- 172.21.15.46:0/3280112886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580d57a0 msgr2=0x7f0858160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 --2- 172.21.15.46:0/3280112886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580d57a0 0x7f0858160610 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f085808f200 tx=0x7f085402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 -- 172.21.15.46:0/3280112886 shutdown_connections 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 --2- 172.21.15.46:0/3280112886 >> v2:172.21.15.88:3300/0 conn(0x7f08580b77e0 0x7f08580b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 --2- 172.21.15.46:0/3280112886 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580b4af0 0x7f08580b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 --2- 172.21.15.46:0/3280112886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580d57a0 0x7f0858160610 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 -- 172.21.15.46:0/3280112886 >> 172.21.15.46:0/3280112886 conn(0x7f08580743e0 msgr2=0x7f08580d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 -- 172.21.15.46:0/3280112886 shutdown_connections 2024-05-09T03:16:15.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.517+0000 7f085ff00640 1 -- 172.21.15.46:0/3280112886 wait complete. 2024-05-09T03:16:15.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.519+0000 7f085ff00640 1 Processor -- start 2024-05-09T03:16:15.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.519+0000 7f085ff00640 1 -- start start 2024-05-09T03:16:15.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085ff00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08580b4af0 0x7f08580cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085ff00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 0x7f085814d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085ff00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580d57a0 0x7f08581eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085ff00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f08580d15b0 con 0x7f08580b77e0 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085ff00640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f08580d5280 con 0x7f08580b4af0 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085ff00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f08580d1920 con 0x7f08580d57a0 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085d49b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 0x7f085814d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.520+0000 7f085d49b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 0x7f085814d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45420/0 (socket says 172.21.15.46:45420) 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 -- 172.21.15.46:0/85231503 learned_addr learned my addr 172.21.15.46:0/85231503 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085dc9c640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08580b4af0 0x7f08580cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085e49d640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580d57a0 0x7f08581eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 -- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580d57a0 msgr2=0x7f08581eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580d57a0 0x7f08581eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 -- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08580b4af0 msgr2=0x7f08580cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08580b4af0 0x7f08580cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 -- 172.21.15.46:0/85231503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f085814b6c0 con 0x7f08580b77e0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085e49d640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580d57a0 0x7f08581eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085d49b640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 0x7f085814d720 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f084800e870 tx=0x7f084800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0848017070 con 0x7f08580b77e0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f084800ee60 con 0x7f08580b77e0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085814b9a0 con 0x7f08580b77e0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084800cb80 con 0x7f08580b77e0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.521+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f085814bf60 con 0x7f08580b77e0 2024-05-09T03:16:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.522+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085806a270 con 0x7f08580b77e0 2024-05-09T03:16:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.523+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f084801a070 con 0x7f08580b77e0 2024-05-09T03:16:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.523+0000 7f084e7fc640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f083c0762d0 0x7f083c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.524+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5853+0+0 (secure 0 0 0) 0x7f08480975e0 con 0x7f08580b77e0 2024-05-09T03:16:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.524+0000 7f085dc9c640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f083c0762d0 0x7f083c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:15.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.524+0000 7f085dc9c640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f083c0762d0 0x7f083c0787e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0854034000 tx=0x7f085402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:15.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.528+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0848060d80 con 0x7f08580b77e0 2024-05-09T03:16:15.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.756+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f085815fcc0 con 0x7f08580b77e0 2024-05-09T03:16:15.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.757+0000 7f084e7fc640 1 -- 172.21.15.46:0/85231503 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) ==== 94+0+5174 (secure 0 0 0) 0x7f0848060680 con 0x7f08580b77e0 2024-05-09T03:16:15.757 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:15.758 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":90,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:14.966529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7056":{"gid":7056,"name":"c","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/1281653036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1281653036},{"type":"v1","addr":"172.21.15.46:6834","nonce":1281653036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 90 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f083c0762d0 msgr2=0x7f083c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f083c0762d0 0x7f083c0787e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0854034000 tx=0x7f085402e000 comp rx=0 tx=0).stop 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 msgr2=0x7f085814d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 0x7f085814d720 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f084800e870 tx=0x7f084800ed30 comp rx=0 tx=0).stop 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 shutdown_connections 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f08580d57a0 0x7f08581eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f083c0762d0 0x7f083c0787e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08580b77e0 0x7f085814d720 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 --2- 172.21.15.46:0/85231503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08580b4af0 0x7f08580cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 >> 172.21.15.46:0/85231503 conn(0x7f08580743e0 msgr2=0x7f08580b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 shutdown_connections 2024-05-09T03:16:15.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:15.759+0000 7f085ff00640 1 -- 172.21.15.46:0/85231503 wait complete. 2024-05-09T03:16:15.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:16:14.966529+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7056}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7056': {'gid': 7056, 'name': 'c', 'rank': 0, 'incarnation': 90, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/1281653036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1281653036}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1281653036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:16:15.771 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7056:up:replay 2024-05-09T03:16:15.771 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:16:19.763 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:19.762+0000 7f11a2d59640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:16:21.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:21.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.303+0000 7f9d5a58d640 1 Processor -- start 2024-05-09T03:16:21.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.304+0000 7f9d5a58d640 1 -- start start 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d5a58d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 0x7f9d541613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d5a58d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540b7ed0 0x7f9d540d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d5a58d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540d5280 con 0x7f9d540d4800 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d5a58d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9d540b5620 con 0x7f9d540b5210 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d5a58d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9d541609a0 con 0x7f9d540b7ed0 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d53fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 0x7f9d541613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d58b2a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540b7ed0 0x7f9d540d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d537fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9d540d4800 0x7f9d540b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:51804/0 (socket says 172.21.15.46:51804) 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d53fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 0x7f9d541613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:36210/0 (socket says 172.21.15.46:36210) 2024-05-09T03:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.305+0000 7f9d537fe640 1 -- 172.21.15.46:0/3626517879 learned_addr learned my addr 172.21.15.46:0/3626517879 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.306+0000 7f9d58b2a640 1 -- 172.21.15.46:0/3626517879 >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 msgr2=0x7f9d541613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.306+0000 7f9d58b2a640 1 --2- 172.21.15.46:0/3626517879 >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 0x7f9d541613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.306+0000 7f9d58b2a640 1 -- 172.21.15.46:0/3626517879 >> v1:172.21.15.88:6789/0 conn(0x7f9d540d4800 legacy=0x7f9d540b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.306+0000 7f9d58b2a640 1 -- 172.21.15.46:0/3626517879 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d540d5560 con 0x7f9d540b7ed0 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.306+0000 7f9d53fff640 1 --2- 172.21.15.46:0/3626517879 >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 0x7f9d541613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.307+0000 7f9d58b2a640 1 --2- 172.21.15.46:0/3626517879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540b7ed0 0x7f9d540d4c10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009d90 tx=0x7f9d4c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=471b1886eb637b8f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.307+0000 7f9d52ffd640 1 -- 172.21.15.46:0/3626517879 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c031600 con 0x7f9d540b7ed0 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.307+0000 7f9d52ffd640 1 -- 172.21.15.46:0/3626517879 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4c0317a0 con 0x7f9d540b7ed0 2024-05-09T03:16:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.307+0000 7f9d52ffd640 1 -- 172.21.15.46:0/3626517879 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c031b70 con 0x7f9d540b7ed0 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.307+0000 7f9d5a58d640 1 -- 172.21.15.46:0/3626517879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540b7ed0 msgr2=0x7f9d540d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.307+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/3626517879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540b7ed0 0x7f9d540d4c10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009d90 tx=0x7f9d4c02a5d0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.308+0000 7f9d5a58d640 1 -- 172.21.15.46:0/3626517879 shutdown_connections 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.308+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/3626517879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540b7ed0 0x7f9d540d4c10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.308+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/3626517879 >> v2:172.21.15.88:3300/0 conn(0x7f9d540b5210 0x7f9d541613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.308+0000 7f9d5a58d640 1 -- 172.21.15.46:0/3626517879 >> 172.21.15.46:0/3626517879 conn(0x7f9d540d29a0 msgr2=0x7f9d540d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.308+0000 7f9d5a58d640 1 -- 172.21.15.46:0/3626517879 shutdown_connections 2024-05-09T03:16:21.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.308+0000 7f9d5a58d640 1 -- 172.21.15.46:0/3626517879 wait complete. 2024-05-09T03:16:21.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.309+0000 7f9d5a58d640 1 Processor -- start 2024-05-09T03:16:21.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.310+0000 7f9d5a58d640 1 -- start start 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d5a58d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d540b5210 0x7f9d54156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d5a58d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540d4800 0x7f9d540cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d5a58d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 0x7f9d540b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d5a58d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d540d0ae0 con 0x7f9d54155c10 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d5a58d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54160ea0 con 0x7f9d540b5210 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d5a58d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9d54163e60 con 0x7f9d540d4800 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d58b2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 0x7f9d540b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d58b2a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 0x7f9d540b6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54540/0 (socket says 172.21.15.46:54540) 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d58b2a640 1 -- 172.21.15.46:0/254705034 learned_addr learned my addr 172.21.15.46:0/254705034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d537fe640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540d4800 0x7f9d540cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d53fff640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d540b5210 0x7f9d54156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d58b2a640 1 -- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540d4800 msgr2=0x7f9d540cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.311+0000 7f9d58b2a640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540d4800 0x7f9d540cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d58b2a640 1 -- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d540b5210 msgr2=0x7f9d54156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d58b2a640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d540b5210 0x7f9d54156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d58b2a640 1 -- 172.21.15.46:0/254705034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d540b59b0 con 0x7f9d54155c10 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d58b2a640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 0x7f9d540b6a50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c02abc0 tx=0x7f9d4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c03f070 con 0x7f9d54155c10 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4c030b60 con 0x7f9d54155c10 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d540b5c90 con 0x7f9d54155c10 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c009450 con 0x7f9d54155c10 2024-05-09T03:16:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.312+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d540b6250 con 0x7f9d54155c10 2024-05-09T03:16:21.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.313+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d5406a270 con 0x7f9d54155c10 2024-05-09T03:16:21.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.314+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9d4c03b410 con 0x7f9d54155c10 2024-05-09T03:16:21.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.314+0000 7f9d50ff9640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d400762d0 0x7f9d400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.314+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7f9d4c0b8b30 con 0x7f9d54155c10 2024-05-09T03:16:21.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.314+0000 7f9d53fff640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d400762d0 0x7f9d400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.315+0000 7f9d53fff640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d400762d0 0x7f9d400787e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009560 tx=0x7f9d44009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.319+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9d4c081d20 con 0x7f9d54155c10 2024-05-09T03:16:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.547+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d540d40d0 con 0x7f9d54155c10 2024-05-09T03:16:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.548+0000 7f9d50ff9640 1 -- 172.21.15.46:0/254705034 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+5181 (secure 0 0 0) 0x7f9d4c086020 con 0x7f9d54155c10 2024-05-09T03:16:21.548 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:21.549 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:17.971251+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7056},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7056":{"gid":7056,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6834/1281653036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1281653036},{"type":"v1","addr":"172.21.15.46:6834","nonce":1281653036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7056,"qdb_cluster":[7056]},"id":6}]} 2024-05-09T03:16:21.549 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 93 2024-05-09T03:16:21.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d400762d0 msgr2=0x7f9d400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d400762d0 0x7f9d400787e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009560 tx=0x7f9d44009190 comp rx=0 tx=0).stop 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 msgr2=0x7f9d540b6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 0x7f9d540b6a50 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c02abc0 tx=0x7f9d4c004a40 comp rx=0 tx=0).stop 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 shutdown_connections 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d400762d0 0x7f9d400787e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d54155c10 0x7f9d540b6a50 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9d540d4800 0x7f9d540cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 --2- 172.21.15.46:0/254705034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d540b5210 0x7f9d54156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 >> 172.21.15.46:0/254705034 conn(0x7f9d540d29a0 msgr2=0x7f9d54163120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 shutdown_connections 2024-05-09T03:16:21.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.550+0000 7f9d5a58d640 1 -- 172.21.15.46:0/254705034 wait complete. 2024-05-09T03:16:21.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:16:17.971251+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7056}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7056': {'gid': 7056, 'name': 'c', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.46:6834/1281653036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1281653036}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1281653036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7056, 'qdb_cluster': [7056]} 2024-05-09T03:16:21.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:16:21.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:16:21.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.677+0000 7fcd81757640 1 Processor -- start 2024-05-09T03:16:21.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.677+0000 7fcd81757640 1 -- start start 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd81757640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 0x7fcd7c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd81757640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 0x7fcd7c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd81757640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd7c0b5550 con 0x7fcd7c0b5140 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd81757640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcd7c0d3480 con 0x7fcd7c0b7e70 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd81757640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7c0d44e0 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 0x7fcd7c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7a7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 0x7fcd7c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7a7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 0x7fcd7c0d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:36218/0 (socket says 172.21.15.46:36218) 2024-05-09T03:16:21.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7affd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcd7c0b5140 0x7fcd7c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:51808/0 (socket says 172.21.15.46:51808) 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/1969555769 learned_addr learned my addr 172.21.15.46:0/1969555769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 0x7fcd7c0b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43850/0 (socket says 172.21.15.46:43850) 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7b7fe640 1 -- 172.21.15.46:0/1969555769 >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 msgr2=0x7fcd7c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/1969555769 >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 0x7fcd7c0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7b7fe640 1 -- 172.21.15.46:0/1969555769 >> v1:172.21.15.88:6789/0 conn(0x7fcd7c0b5140 legacy=0x7fcd7c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7b7fe640 1 -- 172.21.15.46:0/1969555769 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c0d47c0 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.679+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/1969555769 >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 0x7fcd7c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:21.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.680+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/1969555769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 0x7fcd7c0b9380 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009d90 tx=0x7fcd7402a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b55a4c5da29f27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.680+0000 7fcd79ffb640 1 -- 172.21.15.46:0/1969555769 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74007740 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.680+0000 7fcd79ffb640 1 -- 172.21.15.46:0/1969555769 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd740078e0 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.680+0000 7fcd79ffb640 1 -- 172.21.15.46:0/1969555769 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74007cb0 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 -- 172.21.15.46:0/1969555769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 msgr2=0x7fcd7c0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 --2- 172.21.15.46:0/1969555769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 0x7fcd7c0b9380 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcd74009d90 tx=0x7fcd7402a6f0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 -- 172.21.15.46:0/1969555769 shutdown_connections 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 --2- 172.21.15.46:0/1969555769 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0d3df0 0x7fcd7c0b9380 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 --2- 172.21.15.46:0/1969555769 >> v2:172.21.15.88:3300/0 conn(0x7fcd7c0b7e70 0x7fcd7c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 -- 172.21.15.46:0/1969555769 >> 172.21.15.46:0/1969555769 conn(0x7fcd7c08d3f0 msgr2=0x7fcd7c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 -- 172.21.15.46:0/1969555769 shutdown_connections 2024-05-09T03:16:21.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.681+0000 7fcd81757640 1 -- 172.21.15.46:0/1969555769 wait complete. 2024-05-09T03:16:21.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.683+0000 7fcd81757640 1 Processor -- start 2024-05-09T03:16:21.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.683+0000 7fcd81757640 1 -- start start 2024-05-09T03:16:21.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd81757640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0b5140 0x7fcd7c15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd81757640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 0x7fcd7c159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd81757640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd7c158430 0x7fcd7c15d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd81757640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c0d4200 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd81757640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c0d4d80 con 0x7fcd7c158430 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd81757640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7c0d5580 con 0x7fcd7c0b5140 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 0x7fcd7c159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 0x7fcd7c159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54546/0 (socket says 172.21.15.46:54546) 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2121331512 learned_addr learned my addr 172.21.15.46:0/2121331512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd7c158430 0x7fcd7c15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0b5140 0x7fcd7c15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0b5140 msgr2=0x7fcd7c15bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0b5140 0x7fcd7c15bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd7c158430 msgr2=0x7fcd7c15d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd7c158430 0x7fcd7c15d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2121331512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c157390 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd7c158430 0x7fcd7c15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:21.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.685+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 0x7fcd7c159f80 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000e9e0 tx=0x7fcd7000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.686+0000 7fcd637fe640 1 -- 172.21.15.46:0/2121331512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70017070 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.686+0000 7fcd637fe640 1 -- 172.21.15.46:0/2121331512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd7000c9d0 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.686+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c157670 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.686+0000 7fcd637fe640 1 -- 172.21.15.46:0/2121331512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7000cd80 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.686+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c157c30 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.687+0000 7fcd637fe640 1 -- 172.21.15.46:0/2121331512 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcd7001a070 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.688+0000 7fcd637fe640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd68076300 0x7fcd68078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.688+0000 7fcd637fe640 1 -- 172.21.15.46:0/2121331512 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7fcd70097900 con 0x7fcd7c0d3df0 2024-05-09T03:16:21.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.688+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd68076300 0x7fcd68078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.689+0000 7fcd88607740 1 Processor -- start 2024-05-09T03:16:21.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.689+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd68076300 0x7fcd68078810 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c000f60 tx=0x7fcd6c000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.689+0000 7fcd88607740 1 -- start start 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc0390270 0x55bcc075feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd88607740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc077abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc036f590 0x55bcc077b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd88607740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bcc06984e0 con 0x55bcc03673b0 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bcc0633ff0 con 0x55bcc0390270 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bcc06944a0 con 0x55bcc036f590 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc077abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc077abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54560/0 (socket says 172.21.15.46:54560) 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.690+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/374449238 learned_addr learned my addr 172.21.15.46:0/374449238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7affd640 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc0390270 0x55bcc075feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc036f590 0x55bcc077b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc036f590 msgr2=0x55bcc077b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc036f590 0x55bcc077b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc0390270 msgr2=0x55bcc075feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc0390270 0x55bcc075feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/374449238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcc061a720 con 0x55bcc03673b0 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc036f590 0x55bcc077b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc077abe0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009fd0 tx=0x7fcd7005ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193ff11a04dc5b11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd60ff9640 1 -- 172.21.15.46:0/374449238 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70021040 con 0x55bcc03673b0 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd60ff9640 1 -- 172.21.15.46:0/374449238 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70069150 con 0x55bcc03673b0 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.691+0000 7fcd60ff9640 1 -- 172.21.15.46:0/374449238 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7005fed0 con 0x55bcc03673b0 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 -- 172.21.15.46:0/374449238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 msgr2=0x55bcc077abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc077abe0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009fd0 tx=0x7fcd7005ca40 comp rx=0 tx=0).stop 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 -- 172.21.15.46:0/374449238 shutdown_connections 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc036f590 0x55bcc077b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc077abe0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 --2- 172.21.15.46:0/374449238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc0390270 0x55bcc075feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 -- 172.21.15.46:0/374449238 >> 172.21.15.46:0/374449238 conn(0x55bcc03843c0 msgr2=0x55bcc0735510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 -- 172.21.15.46:0/374449238 shutdown_connections 2024-05-09T03:16:21.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.692+0000 7fcd88607740 1 -- 172.21.15.46:0/374449238 wait complete. 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.693+0000 7fcd88607740 1 -- start start 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.693+0000 7fcd88607740 1 Processor -- start 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0772f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc0789d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc075a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bcc061f760 con 0x55bcc036f590 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bcc0762710 con 0x55bcc03673b0 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bcc0773590 con 0x55bcc0390270 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd88607740 10 client.0 _ll_register_callbacks cb 0x55bcc06424b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc0789d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc0789d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54576/0 (socket says 172.21.15.46:54576) 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.694+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2891904712 learned_addr learned my addr 172.21.15.46:0/2891904712 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0772f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc075a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 msgr2=0x55bcc075a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc075a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 msgr2=0x55bcc0772f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0772f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2891904712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcc0660bd0 con 0x55bcc036f590 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0772f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc075a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc0789d60 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009fd0 tx=0x7fcd7000efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c3cb21938d9fb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd59ffb640 1 -- 172.21.15.46:0/2891904712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70021040 con 0x55bcc036f590 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd59ffb640 1 -- 172.21.15.46:0/2891904712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7fcd7005c3c0 con 0x55bcc036f590 2024-05-09T03:16:21.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd70069150 con 0x55bcc036f590 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd59ffb640 1 -- 172.21.15.46:0/2891904712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fcd70060a50 con 0x55bcc036f590 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd88607740 10 client.7527 fetch_fsmap learned FSMap version 93 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) -- 0x55bcc0646280 con 0x55bcc036f590 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd59ffb640 1 -- 172.21.15.46:0/2891904712 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70069150 con 0x55bcc036f590 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd59ffb640 1 -- 172.21.15.46:0/2891904712 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 93) ==== 3494+0+0 (secure 0 0 0) 0x7fcd7005c5c0 con 0x55bcc036f590 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd88607740 10 client.7527 fetch_fsmap finished waiting for FSMap version 93 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.695+0000 7fcd88607740 10 client.7527 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd88607740 20 client.7527 populate_metadata read hostname 'smithi046' 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd88607740 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc034ee40 0x55bcc0761450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd88607740 4 client.7527 mds_command: new command op to 7056 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 --> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcc06a60f0 con 0x55bcc034ee40 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc034ee40 0x55bcc0761450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd59ffb640 10 client.7527 ms_handle_connect on v2:172.21.15.46:6832/1281653036 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.696+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc034ee40 0x55bcc0761450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6562938ec00242ee server_cookie=3d2b2bd4c5e96c9d in_seq=0 out_seq=0 2024-05-09T03:16:21.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd59ffb640 1 -- 172.21.15.46:0/2891904712 <== mds.0 v2:172.21.15.46:6832/1281653036 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bcc06a60f0 con 0x55bcc034ee40 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd59ffb640 10 client.7527 handle_command_reply: tid=0 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 client.7527 shutdown 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 msgr2=0x55bcc0789d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc0789d60 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009fd0 tx=0x7fcd7000efd0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 shutdown_connections 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc034ee40 0x55bcc0761450 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc075a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc0789d60 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd59ffb640 0 client.7527 ms_handle_reset on v2:172.21.15.46:6832/1281653036 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 --2- 172.21.15.46:0/2891904712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0772f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 >> 172.21.15.46:0/2891904712 conn(0x55bcc03843c0 msgr2=0x55bcc075feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 shutdown_connections 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 1 -- 172.21.15.46:0/2891904712 wait complete. 2024-05-09T03:16:21.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.697+0000 7fcd88607740 20 client.7527 trim_cache size 0 max 16384 2024-05-09T03:16:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.716+0000 7fcd88607740 1 Processor -- start 2024-05-09T03:16:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.716+0000 7fcd88607740 1 -- start start 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.717+0000 7fcd88607740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc079daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc036f590 0x55bcc07339c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc0733f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd88607740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c202100 con 0x55bcc03673b0 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c1559c0 con 0x55bcc036f590 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7c156010 con 0x55bcc0390270 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc079daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc079daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54588/0 (socket says 172.21.15.46:54588) 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 -- 172.21.15.46:0/2961073986 learned_addr learned my addr 172.21.15.46:0/2961073986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc036f590 0x55bcc07339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc0733f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 -- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 msgr2=0x55bcc0733f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc0733f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 -- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc036f590 msgr2=0x55bcc07339c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc036f590 0x55bcc07339c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 -- 172.21.15.46:0/2961073986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcc03fb850 con 0x55bcc03673b0 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc036f590 0x55bcc07339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc0733f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd7affd640 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc079daf0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x55bcc072b6f0 tx=0x7fcd6c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a872849b4af8e32 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd5bfff640 1 -- 172.21.15.46:0/2961073986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c01e040 con 0x55bcc03673b0 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd5bfff640 1 -- 172.21.15.46:0/2961073986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd6c02a070 con 0x55bcc03673b0 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.718+0000 7fcd5bfff640 1 -- 172.21.15.46:0/2961073986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c014ad0 con 0x55bcc03673b0 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 -- 172.21.15.46:0/2961073986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 msgr2=0x55bcc079daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc079daf0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x55bcc072b6f0 tx=0x7fcd6c012000 comp rx=0 tx=0).stop 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 -- 172.21.15.46:0/2961073986 shutdown_connections 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc0733f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc036f590 0x55bcc07339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 --2- 172.21.15.46:0/2961073986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc03673b0 0x55bcc079daf0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 -- 172.21.15.46:0/2961073986 >> 172.21.15.46:0/2961073986 conn(0x55bcc03843c0 msgr2=0x55bcc0772d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 -- 172.21.15.46:0/2961073986 shutdown_connections 2024-05-09T03:16:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.719+0000 7fcd88607740 1 -- 172.21.15.46:0/2961073986 wait complete. 2024-05-09T03:16:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.720+0000 7fcd88607740 1 -- start start 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.720+0000 7fcd88607740 1 Processor -- start 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0761270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc07617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc079c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bcc0772680 con 0x55bcc036f590 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bcc07722d0 con 0x55bcc03673b0 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bcc07632a0 con 0x55bcc0390270 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 10 client.0 _ll_register_callbacks cb 0x7fcd7005c910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc07617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc07617b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54604/0 (socket says 172.21.15.46:54604) 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/311763415 learned_addr learned my addr 172.21.15.46:0/311763415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7affd640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0761270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7b7fe640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc079c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 msgr2=0x55bcc079c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc079c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 msgr2=0x55bcc0761270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0761270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/311763415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcc04726b0 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc07617b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcd70060c10 tx=0x7fcd7000cfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbe7ff5688e6f56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd59ffb640 1 -- 172.21.15.46:0/311763415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70021040 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd59ffb640 1 -- 172.21.15.46:0/311763415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7fcd7005c3c0 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.722+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd6c02a070 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd59ffb640 1 -- 172.21.15.46:0/311763415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fcd700603a0 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 10 client.7533 fetch_fsmap learned FSMap version 93 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) -- 0x55bcc0654630 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd59ffb640 1 -- 172.21.15.46:0/311763415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd6c02a070 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd59ffb640 1 -- 172.21.15.46:0/311763415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 93) ==== 3494+0+0 (secure 0 0 0) 0x7fcd7005f720 con 0x55bcc036f590 2024-05-09T03:16:21.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 10 client.7533 fetch_fsmap finished waiting for FSMap version 93 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 10 client.7533 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 20 client.7533 populate_metadata read hostname 'smithi046' 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc079ccc0 0x55bcc079d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 4 client.7533 mds_command: new command op to 7056 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 --> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] -- command(tid 0: {"prefix": "status"}) -- 0x55bcc069ff60 con 0x55bcc079ccc0 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd7affd640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc079ccc0 0x55bcc079d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd59ffb640 10 client.7533 ms_handle_connect on v2:172.21.15.46:6832/1281653036 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.723+0000 7fcd7affd640 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc079ccc0 0x55bcc079d0d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9df09eefc8e52eed server_cookie=df3e17924e063d54 in_seq=0 out_seq=0 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd59ffb640 1 -- 172.21.15.46:0/311763415 <== mds.0 v2:172.21.15.46:6832/1281653036 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55bcc069ff60 con 0x55bcc079ccc0 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd59ffb640 10 client.7533 handle_command_reply: tid=0 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 client.7533 shutdown 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 msgr2=0x55bcc07617b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc07617b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fcd70060c10 tx=0x7fcd7000cfa0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 shutdown_connections 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:6832/1281653036,v1:172.21.15.46:6834/1281653036] conn(0x55bcc079ccc0 0x55bcc079d0d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bcc0390270 0x55bcc079c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bcc036f590 0x55bcc07617b0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd59ffb640 0 client.7533 ms_handle_reset on v2:172.21.15.46:6832/1281653036 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 --2- 172.21.15.46:0/311763415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcc03673b0 0x55bcc0761270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 >> 172.21.15.46:0/311763415 conn(0x55bcc03843c0 msgr2=0x55bcc07599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 shutdown_connections 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 1 -- 172.21.15.46:0/311763415 wait complete. 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd88607740 20 client.7533 trim_cache size 0 max 16384 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:16:21.730 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7056, 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7561420710000002, 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 93, 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 85, 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12.905262662 2024-05-09T03:16:21.731 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd68076300 msgr2=0x7fcd68078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd81757640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd68076300 0x7fcd68078810 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c000f60 tx=0x7fcd6c000bc0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 msgr2=0x7fcd7c159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd81757640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 0x7fcd7c159f80 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000e9e0 tx=0x7fcd7000eea0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.724+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 shutdown_connections 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd7c158430 0x7fcd7c15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd68076300 0x7fcd68078810 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0d3df0 0x7fcd7c159f80 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 --2- 172.21.15.46:0/2121331512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcd7c0b5140 0x7fcd7c15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 >> 172.21.15.46:0/2121331512 conn(0x7fcd7c08d3f0 msgr2=0x7fcd7c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 shutdown_connections 2024-05-09T03:16:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.725+0000 7fcd81757640 1 -- 172.21.15.46:0/2121331512 wait complete. 2024-05-09T03:16:21.736 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:16:21.736 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 10000000000.00000000', proceeding to ls 2024-05-09T03:16:21.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:16:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.850+0000 7fa16dd08640 1 Processor -- start 2024-05-09T03:16:21.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.850+0000 7fa16dd08640 1 -- start start 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa16dd08640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1681613c0 0x7fa1680b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa16dd08640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1680b8110 con 0x7fa1680b5210 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa16dd08640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1681617d0 con 0x7fa1680b7d00 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa16dd08640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa1680b5620 con 0x7fa1681613c0 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa1677fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa1680b5210 0x7fa1680d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36530/0 (socket says 172.21.15.46:36530) 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa1677fe640 1 -- 172.21.15.46:0/488474391 learned_addr learned my addr 172.21.15.46:0/488474391 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa166ffd640 1 --2- 172.21.15.46:0/488474391 >> v2:172.21.15.88:3300/0 conn(0x7fa1681613c0 0x7fa1680b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa1667fc640 1 -- 172.21.15.46:0/488474391 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602283923 0 0) 0x7fa158001110 con 0x7fa1680b5210 2024-05-09T03:16:21.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.852+0000 7fa1667fc640 1 -- 172.21.15.46:0/488474391 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa150003520 con 0x7fa1680b5210 2024-05-09T03:16:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.853+0000 7fa1667fc640 1 -- 172.21.15.46:0/488474391 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3649982130 0 0) 0x7fa150003520 con 0x7fa1680b5210 2024-05-09T03:16:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.853+0000 7fa166ffd640 1 -- 172.21.15.46:0/488474391 >> v1:172.21.15.88:6789/0 conn(0x7fa1680b7d00 legacy=0x7fa168163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.853+0000 7fa166ffd640 1 -- 172.21.15.46:0/488474391 >> v1:172.21.15.46:6789/0 conn(0x7fa1680b5210 legacy=0x7fa1680d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.853+0000 7fa166ffd640 1 -- 172.21.15.46:0/488474391 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1680d5280 con 0x7fa1681613c0 2024-05-09T03:16:21.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.853+0000 7fa166ffd640 1 --2- 172.21.15.46:0/488474391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1681613c0 0x7fa1680b77c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0097a0 tx=0x7fa15c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7d7df1f1bbff27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa1667fc640 1 -- 172.21.15.46:0/488474391 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c037070 con 0x7fa1681613c0 2024-05-09T03:16:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa1667fc640 1 -- 172.21.15.46:0/488474391 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa15c02ac50 con 0x7fa1681613c0 2024-05-09T03:16:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa1667fc640 1 -- 172.21.15.46:0/488474391 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580018c0 con 0x7fa1681613c0 2024-05-09T03:16:21.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa16dd08640 1 -- 172.21.15.46:0/488474391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1681613c0 msgr2=0x7fa1680b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa16dd08640 1 --2- 172.21.15.46:0/488474391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1681613c0 0x7fa1680b77c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0097a0 tx=0x7fa15c02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa16dd08640 1 -- 172.21.15.46:0/488474391 shutdown_connections 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa16dd08640 1 --2- 172.21.15.46:0/488474391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1681613c0 0x7fa1680b77c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa16dd08640 1 -- 172.21.15.46:0/488474391 >> 172.21.15.46:0/488474391 conn(0x7fa1680d29a0 msgr2=0x7fa1680d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.854+0000 7fa16dd08640 1 -- 172.21.15.46:0/488474391 shutdown_connections 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.855+0000 7fa16dd08640 1 -- 172.21.15.46:0/488474391 wait complete. 2024-05-09T03:16:21.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.856+0000 7fa16dd08640 1 Processor -- start 2024-05-09T03:16:21.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.856+0000 7fa16dd08640 1 -- start start 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa16dd08640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1680b5210 0x7fa1681f70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa16dd08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1680b7d00 0x7fa1680cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa16dd08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 0x7fa16815f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa16dd08640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa168160bc0 con 0x7fa1681613c0 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa16dd08640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa1681609a0 con 0x7fa1680b7d00 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa16dd08640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa168160ea0 con 0x7fa1680b5210 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 0x7fa16815f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 0x7fa16815f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54618/0 (socket says 172.21.15.46:54618) 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 -- 172.21.15.46:0/181176754 learned_addr learned my addr 172.21.15.46:0/181176754 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa166ffd640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1680b7d00 0x7fa1680cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa1677fe640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1680b5210 0x7fa1681f70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 -- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1680b5210 msgr2=0x7fa1681f70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1680b5210 0x7fa1681f70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 -- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1680b7d00 msgr2=0x7fa1680cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1680b7d00 0x7fa1680cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 -- 172.21.15.46:0/181176754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1680cba10 con 0x7fa1681613c0 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.858+0000 7fa167fff640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 0x7fa16815f370 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c6b0 tx=0x7fa16000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.859+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa160003b70 con 0x7fa1681613c0 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.859+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa160003d10 con 0x7fa1681613c0 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.859+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1600052f0 con 0x7fa1681613c0 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.859+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16815f8b0 con 0x7fa1681613c0 2024-05-09T03:16:21.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.859+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1680ce1c0 con 0x7fa1681613c0 2024-05-09T03:16:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.860+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16806a270 con 0x7fa1681613c0 2024-05-09T03:16:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.860+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa160005490 con 0x7fa1681613c0 2024-05-09T03:16:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.861+0000 7fa14ffff640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1540762d0 0x7fa1540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.861+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7fa160097380 con 0x7fa1681613c0 2024-05-09T03:16:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.861+0000 7fa1677fe640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1540762d0 0x7fa1540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:21.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.862+0000 7fa1677fe640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1540762d0 0x7fa1540787e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa158001410 tx=0x7fa158001d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:21.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:21.866+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa160060610 con 0x7fa1681613c0 2024-05-09T03:16:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.099+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa1680d40d0 con 0x7fa1681613c0 2024-05-09T03:16:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.100+0000 7fa14ffff640 1 -- 172.21.15.46:0/181176754 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v93) ==== 53+0+73 (secure 0 0 0) 0x7fa160064910 con 0x7fa1681613c0 2024-05-09T03:16:22.101 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1540762d0 msgr2=0x7fa1540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1540762d0 0x7fa1540787e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa158001410 tx=0x7fa158001d70 comp rx=0 tx=0).stop 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 msgr2=0x7fa16815f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 0x7fa16815f370 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fa16000c6b0 tx=0x7fa16000cb70 comp rx=0 tx=0).stop 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 shutdown_connections 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1540762d0 0x7fa1540787e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1681613c0 0x7fa16815f370 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1680b7d00 0x7fa1680cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 --2- 172.21.15.46:0/181176754 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1680b5210 0x7fa1681f70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 >> 172.21.15.46:0/181176754 conn(0x7fa1680d29a0 msgr2=0x7fa168160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 shutdown_connections 2024-05-09T03:16:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22.102+0000 7fa16dd08640 1 -- 172.21.15.46:0/181176754 wait complete. 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:16:22.114 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:16:22.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.115 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:16:22.131 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42135sec preferred_lft 42135sec 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:22.132 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:22.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.134 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:16:22.196 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:16:22.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.198 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:16:22.256 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:16:22.256 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:16:22.256 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:16:22.256 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:16:22.256 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:16:22.257 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:16:22.257 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:16:22.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:16:22.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:16:22.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:16:22.258 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:16:22.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:16:22.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:16:22.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.386 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:16:22.402 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:16:22.402 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:16:22.402 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:16:22.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:16:22.468 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:16:22.468 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:16:22.468 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:22.524 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:16:22.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:16:22.525 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:22.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:16:22.647 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:16:22.704 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:16:22.705 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:16:22.706 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:22.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.708 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:16:22.760 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:16:22.760 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:16:22.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:16:22.828 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:16:22.887 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:16:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:22.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:22.889 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:16:22.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:16:22.898+0000 7f26cb45f480 -1 init, newargv = 0x5584d04c3780 newargc=15 2024-05-09T03:16:22.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[65059]: starting ceph client 2024-05-09T03:16:22.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[65059]: starting fuse 2024-05-09T03:16:22.915 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:16:22.915 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:16:22.916 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:16:22.917 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:16:23.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.65059.asok 2024-05-09T03:16:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:16:23.047 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65059.asok 2024-05-09T03:16:23.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:23.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65059.asok status 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "65059", 2024-05-09T03:16:23.195 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7545, 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7545 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2327050318 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:16:23.196 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2327050318 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.7545 192.168.0.1:0/2327050318", 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2327050318", 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 93, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 85, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:16:23.197 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:16:23.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:23.204 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:16:23.204 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:23.220 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:16:23.220 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:23.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:23.220 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:23.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:23.287 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:16:23.330 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:16:23.347 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:23.346+0000 7f119ed51640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-05-09T03:16:23.348 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:16:23.348 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:16:23.348 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:23.348 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:16:23.348 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 10000000000.00000000' did not caused DAMAGED state 2024-05-09T03:16:23.348 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 600.00000000' 2024-05-09T03:16:23.348 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:23.349 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:23.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:16:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.560+0000 7f725daf9640 1 Processor -- start 2024-05-09T03:16:23.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.560+0000 7f725daf9640 1 -- start start 2024-05-09T03:16:23.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f725daf9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72580b7e70 0x7f72580b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f725daf9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72581605b0 0x7f72580b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f725daf9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72580d3400 con 0x7f72580d2e90 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f725daf9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f72580d4a60 con 0x7f72581605b0 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f725daf9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f72580d4d40 con 0x7f72580b7e70 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f7257fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72581605b0 0x7f72580b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f7257fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72581605b0 0x7f72580b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54622/0 (socket says 172.21.15.46:54622) 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.561+0000 7f7257fff640 1 -- 172.21.15.46:0/2201717662 learned_addr learned my addr 172.21.15.46:0/2201717662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7257fff640 1 -- 172.21.15.46:0/2201717662 >> v2:172.21.15.88:3300/0 conn(0x7f72580b7e70 msgr2=0x7f72580b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7256ffd640 1 --2- 172.21.15.46:0/2201717662 >> v2:172.21.15.88:3300/0 conn(0x7f72580b7e70 0x7f72580b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7257fff640 1 --2- 172.21.15.46:0/2201717662 >> v2:172.21.15.88:3300/0 conn(0x7f72580b7e70 0x7f72580b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7257fff640 1 -- 172.21.15.46:0/2201717662 >> v1:172.21.15.88:6789/0 conn(0x7f72580d2e90 legacy=0x7f72580b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7257fff640 1 -- 172.21.15.46:0/2201717662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72580d5020 con 0x7f72581605b0 2024-05-09T03:16:23.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7256ffd640 1 --2- 172.21.15.46:0/2201717662 >> v2:172.21.15.88:3300/0 conn(0x7f72580b7e70 0x7f72580b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.562+0000 7f7257fff640 1 --2- 172.21.15.46:0/2201717662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f72580b97e0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f72580c95f0 tx=0x7f725002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a971e5cac385c708 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f72567fc640 1 -- 172.21.15.46:0/2201717662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250036070 con 0x7f72581605b0 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f72567fc640 1 -- 172.21.15.46:0/2201717662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7250031600 con 0x7f72581605b0 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f72567fc640 1 -- 172.21.15.46:0/2201717662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72500319d0 con 0x7f72581605b0 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 -- 172.21.15.46:0/2201717662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 msgr2=0x7f72580b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 --2- 172.21.15.46:0/2201717662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f72580b97e0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f72580c95f0 tx=0x7f725002a5f0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 -- 172.21.15.46:0/2201717662 shutdown_connections 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 --2- 172.21.15.46:0/2201717662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f72580b97e0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 --2- 172.21.15.46:0/2201717662 >> v2:172.21.15.88:3300/0 conn(0x7f72580b7e70 0x7f72580b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 -- 172.21.15.46:0/2201717662 >> 172.21.15.46:0/2201717662 conn(0x7f725808d390 msgr2=0x7f725808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 -- 172.21.15.46:0/2201717662 shutdown_connections 2024-05-09T03:16:23.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.563+0000 7f725daf9640 1 -- 172.21.15.46:0/2201717662 wait complete. 2024-05-09T03:16:23.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.565+0000 7f725daf9640 1 Processor -- start 2024-05-09T03:16:23.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.565+0000 7f725daf9640 1 -- start start 2024-05-09T03:16:23.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.566+0000 7f725daf9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580d2e90 0x7f72580cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f725daf9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f725814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f725daf9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f725814edd0 0x7f72580adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f725daf9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72580b4d80 con 0x7f72581605b0 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f725daf9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72580b4c00 con 0x7f72580d2e90 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f725daf9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f72580b4fc0 con 0x7f725814edd0 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f725814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f725814e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54632/0 (socket says 172.21.15.46:54632) 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 -- 172.21.15.46:0/142478509 learned_addr learned my addr 172.21.15.46:0/142478509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f72577fe640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580d2e90 0x7f72580cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7257fff640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f725814edd0 0x7f72580adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 -- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f725814edd0 msgr2=0x7f72580adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f725814edd0 0x7f72580adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 -- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580d2e90 msgr2=0x7f72580cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580d2e90 0x7f72580cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 -- 172.21.15.46:0/142478509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725814f1e0 con 0x7f72581605b0 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f7256ffd640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f725814e890 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f724c00c820 tx=0x7f724c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c015070 con 0x7f72581605b0 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c007a30 con 0x7f72581605b0 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.567+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c007de0 con 0x7f72581605b0 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.568+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72580acb60 con 0x7f72581605b0 2024-05-09T03:16:23.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.568+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72580ad120 con 0x7f72581605b0 2024-05-09T03:16:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.568+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725806a270 con 0x7f72581605b0 2024-05-09T03:16:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.569+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f724c018070 con 0x7f72581605b0 2024-05-09T03:16:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.570+0000 7f723ffff640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.570+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7f724c097980 con 0x7f72581605b0 2024-05-09T03:16:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.570+0000 7f72577fe640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.571+0000 7f72577fe640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7248006fd0 tx=0x7f7248008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:23.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.574+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f725814f1e0 con 0x7f72581605b0 2024-05-09T03:16:23.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.775+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f725808d940 con 0x7f72581605b0 2024-05-09T03:16:23.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.776+0000 7f723ffff640 1 -- 172.21.15.46:0/142478509 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v85) ==== 80+0+122 (secure 0 0 0) 0x7f724c0612a0 con 0x7f72581605b0 2024-05-09T03:16:23.777 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:23.777 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:23.777 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 msgr2=0x7f72440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7248006fd0 tx=0x7f7248008040 comp rx=0 tx=0).stop 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 msgr2=0x7f725814e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f725814e890 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f724c00c820 tx=0x7f724c00cce0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 shutdown_connections 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f725814edd0 0x7f72580adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72581605b0 0x7f725814e890 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 --2- 172.21.15.46:0/142478509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580d2e90 0x7f72580cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 >> 172.21.15.46:0/142478509 conn(0x7f725808d390 msgr2=0x7f72580d0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 shutdown_connections 2024-05-09T03:16:23.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.778+0000 7f725daf9640 1 -- 172.21.15.46:0/142478509 wait complete. 2024-05-09T03:16:23.790 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:23.790 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:23.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:16:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.960+0000 7f550468b640 1 Processor -- start 2024-05-09T03:16:23.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.961+0000 7f550468b640 1 -- start start 2024-05-09T03:16:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f550468b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54fc0b5140 0x7f54fc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f550468b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 0x7f54fc0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f550468b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54fc0b80b0 con 0x7f54fc0d3df0 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f550468b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f54fc0d3480 con 0x7f54fc0b5140 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f550468b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f54fc0b5550 con 0x7f54fc0b7ca0 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f5501c26640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 0x7f54fc0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f5501c26640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 0x7f54fc0d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:43934/0 (socket says 172.21.15.46:43934) 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f5502427640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54fc0b5140 0x7f54fc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.962+0000 7f5501c26640 1 -- 172.21.15.46:0/1570402892 learned_addr learned my addr 172.21.15.46:0/1570402892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.963+0000 7f5501c26640 1 -- 172.21.15.46:0/1570402892 >> v2:172.21.15.88:3300/0 conn(0x7f54fc0b5140 msgr2=0x7f54fc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.963+0000 7f5501c26640 1 --2- 172.21.15.46:0/1570402892 >> v2:172.21.15.88:3300/0 conn(0x7f54fc0b5140 0x7f54fc0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.963+0000 7f5501c26640 1 -- 172.21.15.46:0/1570402892 >> v1:172.21.15.88:6789/0 conn(0x7f54fc0d3df0 legacy=0x7f54fc163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.963+0000 7f5501c26640 1 -- 172.21.15.46:0/1570402892 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc0d44e0 con 0x7f54fc0b7ca0 2024-05-09T03:16:23.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.963+0000 7f5502427640 1 --2- 172.21.15.46:0/1570402892 >> v2:172.21.15.88:3300/0 conn(0x7f54fc0b5140 0x7f54fc0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f5501c26640 1 --2- 172.21.15.46:0/1570402892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 0x7f54fc0d38b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009870 tx=0x7f54ec02a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7db2ef4fe630e64e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f5501425640 1 -- 172.21.15.46:0/1570402892 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ec037070 con 0x7f54fc0b7ca0 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f5501425640 1 -- 172.21.15.46:0/1570402892 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54ec02ae50 con 0x7f54fc0b7ca0 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f5501425640 1 -- 172.21.15.46:0/1570402892 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ec032860 con 0x7f54fc0b7ca0 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f550468b640 1 -- 172.21.15.46:0/1570402892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 msgr2=0x7f54fc0d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f550468b640 1 --2- 172.21.15.46:0/1570402892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 0x7f54fc0d38b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009870 tx=0x7f54ec02a300 comp rx=0 tx=0).stop 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.964+0000 7f550468b640 1 -- 172.21.15.46:0/1570402892 shutdown_connections 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.965+0000 7f550468b640 1 --2- 172.21.15.46:0/1570402892 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b7ca0 0x7f54fc0d38b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.965+0000 7f550468b640 1 --2- 172.21.15.46:0/1570402892 >> v2:172.21.15.88:3300/0 conn(0x7f54fc0b5140 0x7f54fc0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.965+0000 7f550468b640 1 -- 172.21.15.46:0/1570402892 >> 172.21.15.46:0/1570402892 conn(0x7f54fc08d3f0 msgr2=0x7f54fc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.965+0000 7f550468b640 1 -- 172.21.15.46:0/1570402892 shutdown_connections 2024-05-09T03:16:23.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.965+0000 7f550468b640 1 -- 172.21.15.46:0/1570402892 wait complete. 2024-05-09T03:16:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.966+0000 7f550468b640 1 Processor -- start 2024-05-09T03:16:23.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.966+0000 7f550468b640 1 -- start start 2024-05-09T03:16:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f550468b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b5140 0x7f54fc163520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f550468b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0b7ca0 0x7f54fc161910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f550468b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 0x7f54fc161e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f550468b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc0d4d80 con 0x7f54fc0d3df0 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f550468b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc0d4aa0 con 0x7f54fc0b7ca0 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f550468b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f54fc0d4200 con 0x7f54fc0b5140 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 0x7f54fc161e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 0x7f54fc161e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54648/0 (socket says 172.21.15.46:54648) 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 -- 172.21.15.46:0/4024413193 learned_addr learned my addr 172.21.15.46:0/4024413193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502427640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b5140 0x7f54fc163520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5501c26640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0b7ca0 0x7f54fc161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 -- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b5140 msgr2=0x7f54fc163520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b5140 0x7f54fc163520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 -- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0b7ca0 msgr2=0x7f54fc161910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0b7ca0 0x7f54fc161910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 -- 172.21.15.46:0/4024413193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc162e70 con 0x7f54fc0d3df0 2024-05-09T03:16:23.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.968+0000 7f5502c28640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 0x7f54fc161e50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f54f400c3f0 tx=0x7f54f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.969+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f4017070 con 0x7f54fc0d3df0 2024-05-09T03:16:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.969+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f40052d0 con 0x7f54fc0d3df0 2024-05-09T03:16:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.969+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54fc15b990 con 0x7f54fc0d3df0 2024-05-09T03:16:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.969+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f4003810 con 0x7f54fc0d3df0 2024-05-09T03:16:23.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.969+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54fc156230 con 0x7f54fc0d3df0 2024-05-09T03:16:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.970+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54fc0d5580 con 0x7f54fc0d3df0 2024-05-09T03:16:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.971+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f54f401a070 con 0x7f54fc0d3df0 2024-05-09T03:16:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.971+0000 7f54f2ffd640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54e0076300 0x7f54e0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.971+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7f54f4096880 con 0x7f54fc0d3df0 2024-05-09T03:16:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.972+0000 7f5502427640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54e0076300 0x7f54e0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.972+0000 7f5502427640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54e0076300 0x7f54e0078810 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009bd0 tx=0x7f54f800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:23.976+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f54fc0d5580 con 0x7f54fc0d3df0 2024-05-09T03:16:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.180+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f54fc0d38b0 con 0x7f54fc0d3df0 2024-05-09T03:16:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.180+0000 7f54f2ffd640 1 -- 172.21.15.46:0/4024413193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v85) ==== 80+0+122 (secure 0 0 0) 0x7f54fc0d38b0 con 0x7f54fc0d3df0 2024-05-09T03:16:24.181 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:24.181 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:16:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54e0076300 msgr2=0x7f54e0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54e0076300 0x7f54e0078810 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f54f8009bd0 tx=0x7f54f800b040 comp rx=0 tx=0).stop 2024-05-09T03:16:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 msgr2=0x7f54fc161e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 0x7f54fc161e50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f54f400c3f0 tx=0x7f54f400c8b0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 shutdown_connections 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54e0076300 0x7f54e0078810 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc0d3df0 0x7f54fc161e50 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0b7ca0 0x7f54fc161910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 --2- 172.21.15.46:0/4024413193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc0b5140 0x7f54fc163520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 >> 172.21.15.46:0/4024413193 conn(0x7f54fc08d3f0 msgr2=0x7f54fc0b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 shutdown_connections 2024-05-09T03:16:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.182+0000 7f550468b640 1 -- 172.21.15.46:0/4024413193 wait complete. 2024-05-09T03:16:24.194 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:16:24.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:24.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:24.220 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:16:24.220 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:24.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:24.220 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:24.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.432+0000 7fbad21a8640 1 Processor -- start 2024-05-09T03:16:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.433+0000 7fbad21a8640 1 -- start start 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbad21a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc0b7e70 0x7fbacc0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbad21a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbacc0d3df0 0x7fbacc0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbad21a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbacc0b5550 con 0x7fbacc0b5140 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbad21a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbacc0d3480 con 0x7fbacc0d3df0 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbad21a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbacc0d44e0 con 0x7fbacc0b7e70 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbacc0d3df0 0x7fbacc0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbacc0b5140 0x7fbacc0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36538/0 (socket says 172.21.15.46:36538) 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacb7fe640 1 -- 172.21.15.46:0/2214662011 learned_addr learned my addr 172.21.15.46:0/2214662011 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:24.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacbfff640 1 -- 172.21.15.46:0/2214662011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc0b7e70 msgr2=0x7fbacc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:24.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacbfff640 1 --2- 172.21.15.46:0/2214662011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc0b7e70 0x7fbacc0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacbfff640 1 -- 172.21.15.46:0/2214662011 >> v1:172.21.15.46:6789/0 conn(0x7fbacc0b5140 legacy=0x7fbacc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbaca7fc640 1 -- 172.21.15.46:0/2214662011 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267244266 0 0) 0x7fbacc0b5550 con 0x7fbacc0b5140 2024-05-09T03:16:24.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.434+0000 7fbacbfff640 1 -- 172.21.15.46:0/2214662011 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbacc0d47c0 con 0x7fbacc0d3df0 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.435+0000 7fbacbfff640 1 --2- 172.21.15.46:0/2214662011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0b9380 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009d90 tx=0x7fbac402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583e0073bc28dea5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.435+0000 7fbaca7fc640 1 -- 172.21.15.46:0/2214662011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4007740 con 0x7fbacc0d3df0 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.435+0000 7fbaca7fc640 1 -- 172.21.15.46:0/2214662011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac40078e0 con 0x7fbacc0d3df0 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.435+0000 7fbaca7fc640 1 -- 172.21.15.46:0/2214662011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4007cb0 con 0x7fbacc0d3df0 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 -- 172.21.15.46:0/2214662011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 msgr2=0x7fbacc0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 --2- 172.21.15.46:0/2214662011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0b9380 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009d90 tx=0x7fbac402a6f0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 -- 172.21.15.46:0/2214662011 shutdown_connections 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 --2- 172.21.15.46:0/2214662011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0b9380 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 --2- 172.21.15.46:0/2214662011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc0b7e70 0x7fbacc0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 -- 172.21.15.46:0/2214662011 >> 172.21.15.46:0/2214662011 conn(0x7fbacc08d3f0 msgr2=0x7fbacc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 -- 172.21.15.46:0/2214662011 shutdown_connections 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.436+0000 7fbad21a8640 1 -- 172.21.15.46:0/2214662011 wait complete. 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.438+0000 7fbad21a8640 1 Processor -- start 2024-05-09T03:16:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.438+0000 7fbad21a8640 1 -- start start 2024-05-09T03:16:24.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.439+0000 7fbad21a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbacc0b5140 0x7fbacc159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbad21a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbad21a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc1459a0 0x7fbacc145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbad21a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbacc0d4200 con 0x7fbacc0d3df0 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbad21a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbacc0d4d80 con 0x7fbacc0b5140 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbad21a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbacc0d5580 con 0x7fbacc1459a0 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54666/0 (socket says 172.21.15.46:54666) 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 -- 172.21.15.46:0/3953782157 learned_addr learned my addr 172.21.15.46:0/3953782157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacb7fe640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbacc0b5140 0x7fbacc159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacbfff640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc1459a0 0x7fbacc145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 -- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc1459a0 msgr2=0x7fbacc145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc1459a0 0x7fbacc145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 -- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbacc0b5140 msgr2=0x7fbacc159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbacc0b5140 0x7fbacc159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 -- 172.21.15.46:0/3953782157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbacc1462f0 con 0x7fbacc0d3df0 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacb7fe640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbacc0b5140 0x7fbacc159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:24.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacbfff640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc1459a0 0x7fbacc145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbacaffd640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0c8980 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fbac000b410 tx=0x7fbac000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0015020 con 0x7fbacc0d3df0 2024-05-09T03:16:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.440+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0011850 con 0x7fbacc0d3df0 2024-05-09T03:16:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.441+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbacc15b400 con 0x7fbacc0d3df0 2024-05-09T03:16:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.441+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0011c40 con 0x7fbacc0d3df0 2024-05-09T03:16:24.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.441+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbacc15a640 con 0x7fbacc0d3df0 2024-05-09T03:16:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.441+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbacc15a0d0 con 0x7fbacc0d3df0 2024-05-09T03:16:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.442+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbac0018020 con 0x7fbacc0d3df0 2024-05-09T03:16:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.443+0000 7fbab3fff640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbab80762d0 0x7fbab80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.443+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5853+0+0 (secure 0 0 0) 0x7fbac0097570 con 0x7fbacc0d3df0 2024-05-09T03:16:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.443+0000 7fbacb7fe640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbab80762d0 0x7fbab80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.444+0000 7fbacb7fe640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbab80762d0 0x7fbab80787e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbabc000f60 tx=0x7fbabc007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:24.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.447+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbac001f020 con 0x7fbacc0d3df0 2024-05-09T03:16:24.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.681+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbacc06a270 con 0x7fbacc0d3df0 2024-05-09T03:16:24.682 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:24.681+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.683+0000 7fbab3fff640 1 -- 172.21.15.46:0/3953782157 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v94) ==== 161+0+0 (secure 0 0 0) 0x7fbacc1462f0 con 0x7fbacc0d3df0 2024-05-09T03:16:24.683 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbab80762d0 msgr2=0x7fbab80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbab80762d0 0x7fbab80787e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fbabc000f60 tx=0x7fbabc007ca0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 msgr2=0x7fbacc0c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0c8980 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fbac000b410 tx=0x7fbac000b8d0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 shutdown_connections 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbacc1459a0 0x7fbacc145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbab80762d0 0x7fbab80787e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbacc0d3df0 0x7fbacc0c8980 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 --2- 172.21.15.46:0/3953782157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbacc0b5140 0x7fbacc159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 >> 172.21.15.46:0/3953782157 conn(0x7fbacc08d3f0 msgr2=0x7fbacc1605d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 shutdown_connections 2024-05-09T03:16:24.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.686+0000 7fbad21a8640 1 -- 172.21.15.46:0/3953782157 wait complete. 2024-05-09T03:16:24.688 INFO:tasks.ceph.mds.c.smithi046.stderr: -674> 2024-05-09T03:16:19.762+0000 7f11a2d59640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:16:24.688 INFO:tasks.ceph.mds.c.smithi046.stderr: -123> 2024-05-09T03:16:23.346+0000 7f119ed51640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-05-09T03:16:24.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:24.743 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:24.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.817+0000 7fa83aa42640 1 Processor -- start 2024-05-09T03:16:24.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.817+0000 7fa83aa42640 1 -- start start 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa83aa42640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8340b7e70 0x7fa8340d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa83aa42640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8340d3df0 0x7fa8340b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa83aa42640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8340b5550 con 0x7fa8340b5140 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa83aa42640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa8340d3480 con 0x7fa8340d3df0 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa83aa42640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa8340d44e0 con 0x7fa8340b7e70 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa838fdf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8340d3df0 0x7fa8340b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa833fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8340b5140 0x7fa8340b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36548/0 (socket says 172.21.15.46:36548) 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa833fff640 1 -- 172.21.15.46:0/2467735767 learned_addr learned my addr 172.21.15.46:0/2467735767 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa8337fe640 1 --2- 172.21.15.46:0/2467735767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8340b7e70 0x7fa8340d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa838fdf640 1 -- 172.21.15.46:0/2467735767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8340b7e70 msgr2=0x7fa8340d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa838fdf640 1 --2- 172.21.15.46:0/2467735767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8340b7e70 0x7fa8340d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa838fdf640 1 -- 172.21.15.46:0/2467735767 >> v1:172.21.15.46:6789/0 conn(0x7fa8340b5140 legacy=0x7fa8340b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa832ffd640 1 -- 172.21.15.46:0/2467735767 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 962140779 0 0) 0x7fa8340b5550 con 0x7fa8340b5140 2024-05-09T03:16:24.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.819+0000 7fa838fdf640 1 -- 172.21.15.46:0/2467735767 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8340d47c0 con 0x7fa8340d3df0 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.820+0000 7fa838fdf640 1 --2- 172.21.15.46:0/2467735767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340d3df0 0x7fa8340b9380 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009d90 tx=0x7fa82c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1746406b2d45fb1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.820+0000 7fa832ffd640 1 -- 172.21.15.46:0/2467735767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c007740 con 0x7fa8340d3df0 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.820+0000 7fa832ffd640 1 -- 172.21.15.46:0/2467735767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c0078e0 con 0x7fa8340d3df0 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.820+0000 7fa832ffd640 1 -- 172.21.15.46:0/2467735767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c007cb0 con 0x7fa8340d3df0 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 -- 172.21.15.46:0/2467735767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340d3df0 msgr2=0x7fa8340b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2467735767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340d3df0 0x7fa8340b9380 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fa82c009d90 tx=0x7fa82c02a6f0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 -- 172.21.15.46:0/2467735767 shutdown_connections 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2467735767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340d3df0 0x7fa8340b9380 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2467735767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8340b7e70 0x7fa8340d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 -- 172.21.15.46:0/2467735767 >> 172.21.15.46:0/2467735767 conn(0x7fa83408d3f0 msgr2=0x7fa83408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 -- 172.21.15.46:0/2467735767 shutdown_connections 2024-05-09T03:16:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.821+0000 7fa83aa42640 1 -- 172.21.15.46:0/2467735767 wait complete. 2024-05-09T03:16:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.823+0000 7fa83aa42640 1 Processor -- start 2024-05-09T03:16:24.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.823+0000 7fa83aa42640 1 -- start start 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa83aa42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 0x7fa8340c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa83aa42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8340d3df0 0x7fa834145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa83aa42640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa834148580 0x7fa834148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa83aa42640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa8340d4200 con 0x7fa8340b5140 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa83aa42640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa8340d4d80 con 0x7fa8340d3df0 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa83aa42640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa8340d5580 con 0x7fa834148580 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 0x7fa8340c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 0x7fa8340c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54706/0 (socket says 172.21.15.46:54706) 2024-05-09T03:16:24.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 -- 172.21.15.46:0/2157327508 learned_addr learned my addr 172.21.15.46:0/2157327508 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa8337fe640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8340d3df0 0x7fa834145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa838fdf640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa834148580 0x7fa834148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 -- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa834148580 msgr2=0x7fa834148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa834148580 0x7fa834148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 -- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8340d3df0 msgr2=0x7fa834145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8340d3df0 0x7fa834145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.824+0000 7fa833fff640 1 -- 172.21.15.46:0/2157327508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa834148ed0 con 0x7fa8340b5140 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.825+0000 7fa833fff640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 0x7fa8340c9810 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c410 tx=0x7fa82400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.825+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8240074e0 con 0x7fa8340b5140 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.825+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8240047e0 con 0x7fa8340b5140 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.825+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8340c9d50 con 0x7fa8340b5140 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.825+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824004bd0 con 0x7fa8340b5140 2024-05-09T03:16:24.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.825+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa834146870 con 0x7fa8340b5140 2024-05-09T03:16:24.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.826+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83406a270 con 0x7fa8340b5140 2024-05-09T03:16:24.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.827+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa824011070 con 0x7fa8340b5140 2024-05-09T03:16:24.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.827+0000 7fa830ff9640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa820076300 0x7fa820078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.827+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5939+0+0 (secure 0 0 0) 0x7fa824097150 con 0x7fa8340b5140 2024-05-09T03:16:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.827+0000 7fa8337fe640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa820076300 0x7fa820078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:24.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.828+0000 7fa8337fe640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa820076300 0x7fa820078810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa8341f9810 tx=0x7fa828009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:24.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:24.832+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa834148ed0 con 0x7fa8340b5140 2024-05-09T03:16:25.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.099+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa8340ca010 con 0x7fa8340b5140 2024-05-09T03:16:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.684+0000 7fa830ff9640 1 -- 172.21.15.46:0/2157327508 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v95) ==== 107+0+0 (secure 0 0 0) 0x7fa824060370 con 0x7fa8340b5140 2024-05-09T03:16:25.685 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa820076300 msgr2=0x7fa820078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa820076300 0x7fa820078810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa8341f9810 tx=0x7fa828009190 comp rx=0 tx=0).stop 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 msgr2=0x7fa8340c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 0x7fa8340c9810 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c410 tx=0x7fa82400c8d0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 shutdown_connections 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa834148580 0x7fa834148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8340d3df0 0x7fa834145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa820076300 0x7fa820078810 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 --2- 172.21.15.46:0/2157327508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8340b5140 0x7fa8340c9810 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 >> 172.21.15.46:0/2157327508 conn(0x7fa83408d3f0 msgr2=0x7fa834155e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:25.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 shutdown_connections 2024-05-09T03:16:25.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.688+0000 7fa83aa42640 1 -- 172.21.15.46:0/2157327508 wait complete. 2024-05-09T03:16:25.703 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:25.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.763+0000 7efde9265f40 1 Processor -- start 2024-05-09T03:16:25.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.763+0000 7efde9265f40 1 -- start start 2024-05-09T03:16:25.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde9265f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d940d47e0 0x558d94120570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde9265f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d94194940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde9265f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d941ac780 0x558d9411b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde9265f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d94116cb0 con 0x558d94191a50 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde9265f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d9407df50 con 0x558d940d47e0 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde9265f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558d940d4bf0 con 0x558d941ac780 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde252c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d94194940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde252c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d94194940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51480/0 (socket says 172.21.15.46:51480) 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde252c640 1 -- 172.21.15.46:0/1662915060 learned_addr learned my addr 172.21.15.46:0/1662915060 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.764+0000 7efde2d2d640 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d940d47e0 0x558d94120570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde352e640 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d941ac780 0x558d9411b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde252c640 1 -- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d941ac780 msgr2=0x558d9411b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde252c640 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d941ac780 0x558d9411b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde252c640 1 -- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d940d47e0 msgr2=0x558d94120570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde252c640 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d940d47e0 0x558d94120570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde252c640 1 -- 172.21.15.46:0/1662915060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d9411c540 con 0x558d94191a50 2024-05-09T03:16:25.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde252c640 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d94194940 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7efdd8025ca0 tx=0x7efdd8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a8a95a916ca8b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde152a640 1 -- 172.21.15.46:0/1662915060 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8062070 con 0x558d94191a50 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde152a640 1 -- 172.21.15.46:0/1662915060 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd8054970 con 0x558d94191a50 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.765+0000 7efde152a640 1 -- 172.21.15.46:0/1662915060 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8054d60 con 0x558d94191a50 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 -- 172.21.15.46:0/1662915060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 msgr2=0x558d94194940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d94194940 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7efdd8025ca0 tx=0x7efdd8051000 comp rx=0 tx=0).stop 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 -- 172.21.15.46:0/1662915060 shutdown_connections 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d941ac780 0x558d9411b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d94194940 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 --2- 172.21.15.46:0/1662915060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d940d47e0 0x558d94120570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 -- 172.21.15.46:0/1662915060 >> 172.21.15.46:0/1662915060 conn(0x558d93ff9ac0 msgr2=0x558d941abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 -- 172.21.15.46:0/1662915060 shutdown_connections 2024-05-09T03:16:25.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.766+0000 7efde9265f40 1 -- 172.21.15.46:0/1662915060 wait complete. 2024-05-09T03:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.767+0000 7efde9265f40 1 Processor -- start 2024-05-09T03:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.767+0000 7efde9265f40 1 -- start start 2024-05-09T03:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde9265f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d940d47e0 0x558d9418fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde9265f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d9423c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde9265f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d941ac780 0x558d9423cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde9265f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d941ac280 con 0x558d94191a50 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde9265f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d941ac4d0 con 0x558d941ac780 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde9265f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558d9411d050 con 0x558d940d47e0 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde252c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d9423c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde252c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d9423c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51490/0 (socket says 172.21.15.46:51490) 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 learned_addr learned my addr 172.21.15.46:0/344974100 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde352e640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d941ac780 0x558d9423cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d940d47e0 0x558d9418fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d940d47e0 msgr2=0x558d9418fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.768+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558d940d47e0 0x558d9418fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d941ac780 msgr2=0x558d9423cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d941ac780 0x558d9423cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d9423eef0 con 0x558d94191a50 2024-05-09T03:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d9423c860 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7efdd800a5a0 tx=0x7efdd8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efdcbfff640 1 -- 172.21.15.46:0/344974100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd806e030 con 0x558d94191a50 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efdcbfff640 1 -- 172.21.15.46:0/344974100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd8054be0 con 0x558d94191a50 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558d94193b20 con 0x558d94191a50 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efdcbfff640 1 -- 172.21.15.46:0/344974100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8062050 con 0x558d94191a50 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.769+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d941a8f10 con 0x558d94191a50 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.771+0000 7efdcbfff640 1 -- 172.21.15.46:0/344974100 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efdd8071030 con 0x558d94191a50 2024-05-09T03:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.771+0000 7efdcbfff640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efdd0074cb0 0x7efdd00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.771+0000 7efdcbfff640 1 -- 172.21.15.46:0/344974100 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5939+0+0 (secure 0 0 0) 0x7efdd80daea0 con 0x558d94191a50 2024-05-09T03:16:25.773 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.772+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efdd0074cb0 0x7efdd00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.772+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d9411aab0 0x558d9423e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.772+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d941900a0 con 0x558d9411aab0 2024-05-09T03:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.772+0000 7efde352e640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d9411aab0 0x558d9423e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.772+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efdd0074cb0 0x7efdd00771c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efdd4000c00 tx=0x7efdd40017e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.772+0000 7efde352e640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d9411aab0 0x558d9423e0e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.776+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v86'43 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.776+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d9423e620 con 0x558d9411aab0 2024-05-09T03:16:25.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.778+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v86'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.778+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9423ea40 con 0x558d9411aab0 2024-05-09T03:16:25.779 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:25.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.779+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v86'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.779+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94251c20 con 0x558d9411aab0 2024-05-09T03:16:25.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.781+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v86'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.781+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94251fc0 con 0x558d9411aab0 2024-05-09T03:16:25.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.782+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v86'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.782+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94250c10 con 0x558d9411aab0 2024-05-09T03:16:25.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.784+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v86'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.784+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94251000 con 0x558d9411aab0 2024-05-09T03:16:25.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.785+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v86'49 uv22 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.785+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942513f0 con 0x558d9411aab0 2024-05-09T03:16:25.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.787+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v86'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.787+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513f0 con 0x558d9411aab0 2024-05-09T03:16:25.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.788+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v86'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.789 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:25.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.788+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.790+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v86'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.790+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.791+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v86'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.791+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.793+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v86'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.793+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.794+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v86'55 uv55 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.795 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:25.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.794+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.796+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [setxattr (30)] v86'56 uv56 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.796+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.797+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [setxattr (62)] v86'57 uv57 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.797+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.799+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [omap-set-header] v86'58 uv58 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.799+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.800+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [omap-set-vals] v86'59 uv59 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.800+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.801+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 600.00000000 [create] v86'60 uv35 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.801+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.803+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 600.00000000 [delete] v86'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.803+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.805 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:25.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.804+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v86'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.804+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.806+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v86'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.806+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.807+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v86'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.807+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.809+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v86'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.809+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.810+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 603.00000000 [create] v86'66 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.810+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.812+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 603.00000000 [delete] v86'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.812+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.813+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v86'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.813 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:25.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.813+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.814+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v86'69 uv69 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.814+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.815+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v86'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.815+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d9411aab0 2024-05-09T03:16:25.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.817+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v86'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.817+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558d942517a0 0x558d94254880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.817+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:30 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942513a0 con 0x558d942517a0 2024-05-09T03:16:25.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.817+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558d942517a0 0x558d94254880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.817+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558d942517a0 0x558d94254880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.819+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(30 1.00000000.inode [create] v86'45 uv25 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.819+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:31 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94254dc0 con 0x558d942517a0 2024-05-09T03:16:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.820+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(31 1.00000000.inode [delete] v86'46 uv46 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.820+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94255160 con 0x558d942517a0 2024-05-09T03:16:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.821+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(32 1.00000000.inode [create] v86'47 uv47 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.822 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.821+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942559b0 con 0x558d942517a0 2024-05-09T03:16:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.823+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v86'48 uv48 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.823+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94255d50 con 0x558d942517a0 2024-05-09T03:16:25.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.824+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v86'49 uv49 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.824+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942560f0 con 0x558d942517a0 2024-05-09T03:16:25.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.825+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(35 200.00000000 [create] v86'50 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.825+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94256490 con 0x558d942517a0 2024-05-09T03:16:25.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.826+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(36 200.00000000 [delete] v86'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.826+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.827+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(37 200.00000000 [create] v86'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.828 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:25.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.827+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.829+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v86'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.829+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.830+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v86'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.830+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.831+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(40 200.00000001 [create] v86'55 uv43 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.831+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.832+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(41 200.00000001 [delete] v86'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.832+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.833+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(42 200.00000001 [create] v86'57 uv57 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.834 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:25.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.833+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425a290 con 0x558d942517a0 2024-05-09T03:16:25.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.835+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(43 200.00000001 [write 0~14895] v86'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.835+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425a290 con 0x558d942517a0 2024-05-09T03:16:25.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.836+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v86'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.836+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.837+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(45 606.00000000 [create] v86'60 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.837+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.838+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(46 606.00000000 [delete] v86'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.838+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.839+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(47 606.00000000 [create] v86'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.840 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:25.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.839+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94257840 con 0x558d942517a0 2024-05-09T03:16:25.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.840+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v86'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.840+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94257840 con 0x558d942517a0 2024-05-09T03:16:25.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.841+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v86'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.841+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d942517a0 2024-05-09T03:16:25.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.842+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v86'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d942517a0 2024-05-09T03:16:25.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.843+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d9411aab0 2024-05-09T03:16:25.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.844+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(51 607.00000000 [create] v86'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.844+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94256830 con 0x558d9411aab0 2024-05-09T03:16:25.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.845+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(52 607.00000000 [delete] v86'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.845+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d9411aab0 2024-05-09T03:16:25.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.846+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.846 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:25.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.846+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94257840 con 0x558d9411aab0 2024-05-09T03:16:25.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.847+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.847+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94257840 con 0x558d9411aab0 2024-05-09T03:16:25.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.848+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.848+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94256830 con 0x558d9411aab0 2024-05-09T03:16:25.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.849+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [omap-set-header] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.849+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558d94256eb0 0x558d942592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.849+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94259930 con 0x558d94256eb0 2024-05-09T03:16:25.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.849+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558d94256eb0 0x558d942592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.850+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558d94256eb0 0x558d942592d0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.851+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v86'11 uv10 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.851+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94259e40 con 0x558d94256eb0 2024-05-09T03:16:25.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.852+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v86'12 uv12 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.852+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425a340 con 0x558d94256eb0 2024-05-09T03:16:25.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.853+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v86'13 uv13 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.854 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.853+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942555e0 con 0x558d94256eb0 2024-05-09T03:16:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.854+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v86'14 uv14 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.854+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425ac70 con 0x558d9411aab0 2024-05-09T03:16:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.855+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(61 608.00000000 [create] v86'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.855+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d9425ac70 con 0x558d9411aab0 2024-05-09T03:16:25.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.857+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(62 608.00000000 [delete] v86'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.857+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425ac70 con 0x558d9411aab0 2024-05-09T03:16:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.858+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.858 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.858+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425bc80 con 0x558d9411aab0 2024-05-09T03:16:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.859+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.859+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425bc80 con 0x558d9411aab0 2024-05-09T03:16:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.860+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.860+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425ac70 con 0x558d9411aab0 2024-05-09T03:16:25.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.862+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [omap-set-header] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.862+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558d9425b2f0 0x558d9425b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.862+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425e060 con 0x558d9425b2f0 2024-05-09T03:16:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.862+0000 7efde352e640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558d9425b2f0 0x558d9425b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.862+0000 7efde352e640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558d9425b2f0 0x558d9425b720 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.864+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(67 604.00000000 [create] v86'15 uv9 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.864+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d9425e060 con 0x558d9425b2f0 2024-05-09T03:16:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.865+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(68 604.00000000 [delete] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.865+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425e060 con 0x558d9425b2f0 2024-05-09T03:16:25.868 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.868+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(69 604.00000000 [create] v86'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.868+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f0f0 con 0x558d9425b2f0 2024-05-09T03:16:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.870+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v86'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.870+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f0b0 con 0x558d9425b2f0 2024-05-09T03:16:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.873+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v86'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.873+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f0b0 con 0x558d9425b2f0 2024-05-09T03:16:25.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.876+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.876+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f090 con 0x558d9425b2f0 2024-05-09T03:16:25.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.878+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(73 500.00000000 [create] v86'21 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.878+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d9425f090 con 0x558d9425b2f0 2024-05-09T03:16:25.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.881+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(74 500.00000000 [delete] v86'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.881+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f090 con 0x558d9425b2f0 2024-05-09T03:16:25.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.882+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(75 500.00000000 [create] v86'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.883 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:25.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.882+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f090 con 0x558d9425b2f0 2024-05-09T03:16:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.883+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v86'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.883+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d9425f090 con 0x558d9425b2f0 2024-05-09T03:16:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.884+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v86'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.885+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558d9425e570 0x558d9425e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.885+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94261530 con 0x558d9425e570 2024-05-09T03:16:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.885+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558d9425e570 0x558d9425e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.885+0000 7efde2d2d640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558d9425e570 0x558d9425e980 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.886+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(78 mds_snaptable [create] v86'7 uv6 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d9425e570 2024-05-09T03:16:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.886+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94261530 con 0x558d9425e570 2024-05-09T03:16:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.887+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(79 mds_snaptable [delete] v86'8 uv8 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d9425e570 2024-05-09T03:16:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.887+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942615b0 con 0x558d9425e570 2024-05-09T03:16:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.888+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(80 mds_snaptable [create] v86'9 uv9 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d9425e570 2024-05-09T03:16:25.889 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.888+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d9425e570 2024-05-09T03:16:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.889+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v86'10 uv10 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d9425e570 2024-05-09T03:16:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.889+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d9425e570 2024-05-09T03:16:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.891+0000 7efde2d2d640 1 -- 172.21.15.46:0/344974100 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v86'11 uv11 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd4011910 con 0x558d9425e570 2024-05-09T03:16:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.891+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d9411aab0 2024-05-09T03:16:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.892+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(83 605.00000000 [create] v86'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.892+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d9411aab0 2024-05-09T03:16:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.893+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(84 605.00000000 [delete] v86'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.893+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d9411aab0 2024-05-09T03:16:25.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.894+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.895 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:25.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.894+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942629b0 con 0x558d9411aab0 2024-05-09T03:16:25.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.895+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.895+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942629b0 con 0x558d9411aab0 2024-05-09T03:16:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.896+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.896+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d9411aab0 2024-05-09T03:16:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.897+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [omap-set-header] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9411aab0 2024-05-09T03:16:25.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.897+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942619a0 con 0x558d94256eb0 2024-05-09T03:16:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.898+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(89 mds0_inotable [create] v86'14 uv8 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.898+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94261d40 con 0x558d94256eb0 2024-05-09T03:16:25.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.899+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(90 mds0_inotable [delete] v86'15 uv15 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.899+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942620e0 con 0x558d94256eb0 2024-05-09T03:16:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.900+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(91 mds0_inotable [create] v86'16 uv16 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.901 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.900+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94262480 con 0x558d94256eb0 2024-05-09T03:16:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.902+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v86'17 uv17 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.902+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94262480 con 0x558d94256eb0 2024-05-09T03:16:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.903+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v86'18 uv18 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.903+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94262480 con 0x558d94256eb0 2024-05-09T03:16:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.904+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(94 100.00000000 [create] v86'19 uv13 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.904+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d94262480 con 0x558d94256eb0 2024-05-09T03:16:25.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.905+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(95 100.00000000 [delete] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.905+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94262480 con 0x558d94256eb0 2024-05-09T03:16:25.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.906+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(96 100.00000000 [create] v86'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.906 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.906+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94262480 con 0x558d94256eb0 2024-05-09T03:16:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.907+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v86'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.907+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94265290 con 0x558d94256eb0 2024-05-09T03:16:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.908+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v86'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.908+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558d94262fb0 0x558d94265910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.908+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94262fb0 2024-05-09T03:16:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.908+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558d94262fb0 0x558d94265910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.908+0000 7efde252c640 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558d94262fb0 0x558d94265910 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:25.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.910+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v86'9 uv7 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94262fb0 2024-05-09T03:16:25.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.910+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94262fb0 2024-05-09T03:16:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.911+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v86'10 uv10 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94262fb0 2024-05-09T03:16:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.911+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94262fb0 2024-05-09T03:16:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.912+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v86'11 uv11 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94262fb0 2024-05-09T03:16:25.912 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.912+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94262fb0 2024-05-09T03:16:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.913+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v86'12 uv12 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94262fb0 2024-05-09T03:16:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.913+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94256eb0 2024-05-09T03:16:25.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.914+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v86'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.914+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94256eb0 2024-05-09T03:16:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.915+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v86'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.915+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94256eb0 2024-05-09T03:16:25.917 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.916+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.916+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94256eb0 2024-05-09T03:16:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.918+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.918+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94256eb0 2024-05-09T03:16:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.919+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.919+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d94256eb0 2024-05-09T03:16:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.920+0000 7efde252c640 1 -- 172.21.15.46:0/344974100 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efdd8064030 con 0x558d94256eb0 2024-05-09T03:16:25.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.920+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.921+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v86'7 uv6 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.921+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.923+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v86'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.923+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.924 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:25.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.924+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v86'9 uv9 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.924+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.925+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v86'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.925+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.928+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.928+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.930+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v86'20 uv11 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.930+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.935+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v86'21 uv21 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.935+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.937+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v86'22 uv22 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.937 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.937+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.939+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v86'23 uv23 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.939+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.942+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v86'24 uv24 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.942+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.944+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v86'25 uv18 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.944+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.947+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v86'26 uv26 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.947+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.950 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.949+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v86'27 uv27 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.949+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.952+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v86'28 uv28 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.952+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.954+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v86'29 uv29 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.954+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d942653d0 con 0x558d9425b2f0 2024-05-09T03:16:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.957+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v86'30 uv30 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.957+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x558d94265e70 con 0x558d9425b2f0 2024-05-09T03:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v86'31 uv31 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efddc00c8d0 con 0x558d9425b2f0 2024-05-09T03:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d9411aab0 msgr2=0x558d9423e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558d9411aab0 0x558d9423e0e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558d94256eb0 msgr2=0x558d942592d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558d94256eb0 0x558d942592d0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558d94262fb0 msgr2=0x558d94265910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558d94262fb0 0x558d94265910 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558d942517a0 msgr2=0x558d94254880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558d942517a0 0x558d94254880 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558d9425b2f0 msgr2=0x558d9425b720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558d9425b2f0 0x558d9425b720 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558d9425e570 msgr2=0x558d9425e980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558d9425e570 0x558d9425e980 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 reap_dead start 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.959+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 reap_dead start 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde352e640 1 -- 172.21.15.46:0/344974100 reap_dead start 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efdd0074cb0 msgr2=0x7efdd00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efdd0074cb0 0x7efdd00771c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efdd4000c00 tx=0x7efdd40017e0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 msgr2=0x558d9423c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d9423c860 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7efdd800a5a0 tx=0x7efdd8002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 shutdown_connections 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efdd0074cb0 0x7efdd00771c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 --2- 172.21.15.46:0/344974100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d94191a50 0x558d9423c860 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 >> 172.21.15.46:0/344974100 conn(0x558d93ff9ac0 msgr2=0x558d941abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 shutdown_connections 2024-05-09T03:16:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:25.960+0000 7efde9265f40 1 -- 172.21.15.46:0/344974100 wait complete. 2024-05-09T03:16:25.964 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 600.00000000 2024-05-09T03:16:26.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.018+0000 7fc3bc872f40 1 Processor -- start 2024-05-09T03:16:26.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.018+0000 7fc3bc872f40 1 -- start start 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.019+0000 7fc3bc872f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff71ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.019+0000 7fc3bc872f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff71f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.019+0000 7fc3bc872f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff71b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.019+0000 7fc3bc872f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601ff6945f0 con 0x5601ff6935f0 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.019+0000 7fc3bc872f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601ff5f1f50 con 0x5601ff68b3a0 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.019+0000 7fc3bc872f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5601ff6948d0 con 0x5601ff71ea40 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff71f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff71f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51502/0 (socket says 172.21.15.46:51502) 2024-05-09T03:16:26.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 -- 172.21.15.46:0/224882307 learned_addr learned my addr 172.21.15.46:0/224882307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b633a640 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff71ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b6b3b640 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff71b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 -- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 msgr2=0x5601ff71b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff71b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 -- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 msgr2=0x5601ff71ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff71ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 -- 172.21.15.46:0/224882307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ff7216a0 con 0x5601ff6935f0 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b5b39640 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff71f390 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00b210 tx=0x7fc3ac00b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fa144590aabe0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b4b37640 1 -- 172.21.15.46:0/224882307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac008070 con 0x5601ff6935f0 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b4b37640 1 -- 172.21.15.46:0/224882307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac00d500 con 0x5601ff6935f0 2024-05-09T03:16:26.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.020+0000 7fc3b4b37640 1 -- 172.21.15.46:0/224882307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac00d8d0 con 0x5601ff6935f0 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 -- 172.21.15.46:0/224882307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 msgr2=0x5601ff71f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff71f390 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00b210 tx=0x7fc3ac00b6d0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 -- 172.21.15.46:0/224882307 shutdown_connections 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff71b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff71f390 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/224882307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff71ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 -- 172.21.15.46:0/224882307 >> 172.21.15.46:0/224882307 conn(0x5601ff56dac0 msgr2=0x5601ff6762a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 -- 172.21.15.46:0/224882307 shutdown_connections 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.021+0000 7fc3bc872f40 1 -- 172.21.15.46:0/224882307 wait complete. 2024-05-09T03:16:26.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.022+0000 7fc3bc872f40 1 Processor -- start 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.022+0000 7fc3bc872f40 1 -- start start 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3bc872f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff702a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3bc872f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff64c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3bc872f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff64c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3bc872f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601ff71b0c0 con 0x5601ff6935f0 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3bc872f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601ff7213c0 con 0x5601ff68b3a0 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3bc872f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5601ff67a8d0 con 0x5601ff71ea40 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3b5b39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff64c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3b5b39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff64c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51512/0 (socket says 172.21.15.46:51512) 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3b5b39640 1 -- 172.21.15.46:0/1080324822 learned_addr learned my addr 172.21.15.46:0/1080324822 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.023+0000 7fc3b633a640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff702a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b6b3b640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff64c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b5b39640 1 -- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 msgr2=0x5601ff64c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b5b39640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601ff71ea40 0x5601ff64c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b5b39640 1 -- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 msgr2=0x5601ff702a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b5b39640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ff68b3a0 0x5601ff702a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b5b39640 1 -- 172.21.15.46:0/1080324822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ff6755a0 con 0x5601ff6935f0 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3b5b39640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff64c470 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x5601ff68e170 tx=0x7fc3ac000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3a77fe640 1 -- 172.21.15.46:0/1080324822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac021030 con 0x5601ff6935f0 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3a77fe640 1 -- 172.21.15.46:0/1080324822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac01c440 con 0x5601ff6935f0 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3a77fe640 1 -- 172.21.15.46:0/1080324822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac01c610 con 0x5601ff6935f0 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5601ff675880 con 0x5601ff6935f0 2024-05-09T03:16:26.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.024+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601ff68d470 con 0x5601ff6935f0 2024-05-09T03:16:26.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.026+0000 7fc3a77fe640 1 -- 172.21.15.46:0/1080324822 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc3ac024070 con 0x5601ff6935f0 2024-05-09T03:16:26.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.026+0000 7fc3a77fe640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc398074ce0 0x7fc3980771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.026+0000 7fc3a77fe640 1 -- 172.21.15.46:0/1080324822 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5939+0+0 (secure 0 0 0) 0x7fc3ac097280 con 0x5601ff6935f0 2024-05-09T03:16:26.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.026+0000 7fc3b633a640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc398074ce0 0x7fc3980771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.027+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5601ff707070 0x5601ff68cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.027+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e86) -- 0x5601ff7b8560 con 0x5601ff707070 2024-05-09T03:16:26.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.027+0000 7fc3b6b3b640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5601ff707070 0x5601ff68cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.027+0000 7fc3b633a640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc398074ce0 0x7fc3980771f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8005fd0 tx=0x7fc3a800e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.027+0000 7fc3b6b3b640 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5601ff707070 0x5601ff68cdb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.028+0000 7fc3b6b3b640 1 -- 172.21.15.46:0/1080324822 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 600.00000000 [delete] v86'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc3b00527a0 con 0x5601ff707070 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.028+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5601ff707070 msgr2=0x5601ff68cdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.028+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5601ff707070 0x5601ff68cdb0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc398074ce0 msgr2=0x7fc3980771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc398074ce0 0x7fc3980771f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8005fd0 tx=0x7fc3a800e040 comp rx=0 tx=0).stop 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 msgr2=0x5601ff64c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 --2- 172.21.15.46:0/1080324822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601ff6935f0 0x5601ff64c470 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x5601ff68e170 tx=0x7fc3ac000ed0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3b6b3b640 1 -- 172.21.15.46:0/1080324822 reap_dead start 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 shutdown_connections 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 >> 172.21.15.46:0/1080324822 conn(0x5601ff56dac0 msgr2=0x5601ff678f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:26.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 shutdown_connections 2024-05-09T03:16:26.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.029+0000 7fc3bc872f40 1 -- 172.21.15.46:0/1080324822 wait complete. 2024-05-09T03:16:26.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:26.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.148+0000 7f2945e78640 1 Processor -- start 2024-05-09T03:16:26.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.148+0000 7f2945e78640 1 -- start start 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f2945e78640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2940143410 0x7f29400b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f2945e78640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f2940161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f2945e78640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29400d1860 0x7f29400d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f2945e78640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f29400b58b0 con 0x7f29400d1860 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f2945e78640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f29400b4920 con 0x7f2940143410 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f2945e78640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f29400b4ee0 con 0x7f2940160c60 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29400d1860 0x7f29400d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29400d1860 0x7f29400d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51526/0 (socket says 172.21.15.46:51526) 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 -- 172.21.15.46:0/1090472674 learned_addr learned my addr 172.21.15.46:0/1090472674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293f7fe640 1 --2- 172.21.15.46:0/1090472674 >> v2:172.21.15.88:3300/0 conn(0x7f2940143410 0x7f29400b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293effd640 1 --2- 172.21.15.46:0/1090472674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f2940161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 -- 172.21.15.46:0/1090472674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 msgr2=0x7f2940161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 --2- 172.21.15.46:0/1090472674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f2940161070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 -- 172.21.15.46:0/1090472674 >> v2:172.21.15.88:3300/0 conn(0x7f2940143410 msgr2=0x7f29400b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 --2- 172.21.15.46:0/1090472674 >> v2:172.21.15.88:3300/0 conn(0x7f2940143410 0x7f29400b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293ffff640 1 -- 172.21.15.46:0/1090472674 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29400d21b0 con 0x7f29400d1860 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.150+0000 7f293f7fe640 1 --2- 172.21.15.46:0/1090472674 >> v2:172.21.15.88:3300/0 conn(0x7f2940143410 0x7f29400b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.151+0000 7f293ffff640 1 --2- 172.21.15.46:0/1090472674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400d1c70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f29400d05d0 tx=0x7f293802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4858e79be83917e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.151+0000 7f293e7fc640 1 -- 172.21.15.46:0/1090472674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2938036070 con 0x7f29400d1860 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.151+0000 7f293e7fc640 1 -- 172.21.15.46:0/1090472674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2938031600 con 0x7f29400d1860 2024-05-09T03:16:26.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.151+0000 7f293e7fc640 1 -- 172.21.15.46:0/1090472674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29380319d0 con 0x7f29400d1860 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 -- 172.21.15.46:0/1090472674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 msgr2=0x7f29400d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 --2- 172.21.15.46:0/1090472674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400d1c70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f29400d05d0 tx=0x7f293802a610 comp rx=0 tx=0).stop 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 -- 172.21.15.46:0/1090472674 shutdown_connections 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 --2- 172.21.15.46:0/1090472674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400d1c70 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 --2- 172.21.15.46:0/1090472674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f2940161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 --2- 172.21.15.46:0/1090472674 >> v2:172.21.15.88:3300/0 conn(0x7f2940143410 0x7f29400b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 -- 172.21.15.46:0/1090472674 >> 172.21.15.46:0/1090472674 conn(0x7f29400b3620 msgr2=0x7f29400b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 -- 172.21.15.46:0/1090472674 shutdown_connections 2024-05-09T03:16:26.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.152+0000 7f2945e78640 1 -- 172.21.15.46:0/1090472674 wait complete. 2024-05-09T03:16:26.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.154+0000 7f2945e78640 1 Processor -- start 2024-05-09T03:16:26.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.154+0000 7f2945e78640 1 -- start start 2024-05-09T03:16:26.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.155+0000 7f2945e78640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.155+0000 7f2945e78640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2940143410 0x7f2940151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.155+0000 7f2945e78640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f29400af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.155+0000 7f2945e78640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f29400b4b70 con 0x7f29400d1860 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.155+0000 7f2945e78640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2940161990 con 0x7f2940143410 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.155+0000 7f2945e78640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2940143820 con 0x7f2940160c60 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51536/0 (socket says 172.21.15.46:51536) 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 -- 172.21.15.46:0/1393754468 learned_addr learned my addr 172.21.15.46:0/1393754468 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293effd640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2940143410 0x7f2940151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293ffff640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f29400af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 -- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 msgr2=0x7f29400af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f29400af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 -- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2940143410 msgr2=0x7f2940151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2940143410 0x7f2940151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 -- 172.21.15.46:0/1393754468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2940151fd0 con 0x7f29400d1860 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f293f7fe640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400acf00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f293000e910 tx=0x7f293000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930017070 con 0x7f29400d1860 2024-05-09T03:16:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293000ca00 con 0x7f29400d1860 2024-05-09T03:16:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293000cdd0 con 0x7f29400d1860 2024-05-09T03:16:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294014f5d0 con 0x7f29400d1860 2024-05-09T03:16:26.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.156+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294014fb90 con 0x7f29400d1860 2024-05-09T03:16:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.157+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294006a270 con 0x7f29400d1860 2024-05-09T03:16:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.158+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f293001a070 con 0x7f29400d1860 2024-05-09T03:16:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.159+0000 7f2927fff640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f292c076300 0x7f292c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.159+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5939+0+0 (secure 0 0 0) 0x7f29300978e0 con 0x7f29400d1860 2024-05-09T03:16:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.159+0000 7f293effd640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f292c076300 0x7f292c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:26.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.159+0000 7f293effd640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f292c076300 0x7f292c078810 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f29400b5550 tx=0x7f2934008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:26.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.163+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2940151fd0 con 0x7f29400d1860 2024-05-09T03:16:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:26.393+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f29400d0620 con 0x7f29400d1860 2024-05-09T03:16:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.172+0000 7f2927fff640 1 -- 172.21.15.46:0/1393754468 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v96) ==== 162+0+0 (secure 0 0 0) 0x7f29300610c0 con 0x7f29400d1860 2024-05-09T03:16:27.173 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f292c076300 msgr2=0x7f292c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f292c076300 0x7f292c078810 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f29400b5550 tx=0x7f2934008040 comp rx=0 tx=0).stop 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 msgr2=0x7f29400acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400acf00 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f293000e910 tx=0x7f293000edd0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 shutdown_connections 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2940160c60 0x7f29400af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2940143410 0x7f2940151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f292c076300 0x7f292c078810 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 --2- 172.21.15.46:0/1393754468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29400d1860 0x7f29400acf00 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.175+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 >> 172.21.15.46:0/1393754468 conn(0x7f29400b3620 msgr2=0x7f2940145a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.176+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 shutdown_connections 2024-05-09T03:16:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.176+0000 7f2945e78640 1 -- 172.21.15.46:0/1393754468 wait complete. 2024-05-09T03:16:27.193 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.313+0000 7f53b309a640 1 Processor -- start 2024-05-09T03:16:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.313+0000 7f53b309a640 1 -- start start 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.314+0000 7f53b309a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53ac0b5210 0x7f53ac1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.314+0000 7f53b309a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ac0b7ed0 0x7f53ac0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.314+0000 7f53b309a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ac0d5280 con 0x7f53ac0d4800 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b309a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f53ac0b5620 con 0x7f53ac0b7ed0 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b309a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53ac1609a0 con 0x7f53ac0b5210 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ac0b7ed0 0x7f53ac0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53ac0b7ed0 0x7f53ac0d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51542/0 (socket says 172.21.15.46:51542) 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 -- 172.21.15.46:0/1408590597 learned_addr learned my addr 172.21.15.46:0/1408590597 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b0e36640 1 --2- 172.21.15.46:0/1408590597 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0b5210 0x7f53ac1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 -- 172.21.15.46:0/1408590597 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0b5210 msgr2=0x7f53ac1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 --2- 172.21.15.46:0/1408590597 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0b5210 0x7f53ac1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 -- 172.21.15.46:0/1408590597 >> v1:172.21.15.88:6789/0 conn(0x7f53ac0d4800 legacy=0x7f53ac0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.315+0000 7f53b1637640 1 -- 172.21.15.46:0/1408590597 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac0d5560 con 0x7f53ac0b7ed0 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.316+0000 7f53b1637640 1 --2- 172.21.15.46:0/1408590597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0b7ed0 0x7f53ac0d4c10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f53ac08ed70 tx=0x7f53a402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259a90bdf6bfbacb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.316+0000 7f53ab7fe640 1 -- 172.21.15.46:0/1408590597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4036070 con 0x7f53ac0b7ed0 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.316+0000 7f53ab7fe640 1 -- 172.21.15.46:0/1408590597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a4031600 con 0x7f53ac0b7ed0 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.316+0000 7f53ab7fe640 1 -- 172.21.15.46:0/1408590597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40319d0 con 0x7f53ac0b7ed0 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 -- 172.21.15.46:0/1408590597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0b7ed0 msgr2=0x7f53ac0d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 --2- 172.21.15.46:0/1408590597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0b7ed0 0x7f53ac0d4c10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f53ac08ed70 tx=0x7f53a402a610 comp rx=0 tx=0).stop 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 -- 172.21.15.46:0/1408590597 shutdown_connections 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 --2- 172.21.15.46:0/1408590597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0b7ed0 0x7f53ac0d4c10 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 --2- 172.21.15.46:0/1408590597 >> v2:172.21.15.88:3300/0 conn(0x7f53ac0b5210 0x7f53ac1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 -- 172.21.15.46:0/1408590597 >> 172.21.15.46:0/1408590597 conn(0x7f53ac0d29a0 msgr2=0x7f53ac0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 -- 172.21.15.46:0/1408590597 shutdown_connections 2024-05-09T03:16:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.317+0000 7f53b309a640 1 -- 172.21.15.46:0/1408590597 wait complete. 2024-05-09T03:16:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.318+0000 7f53b309a640 1 Processor -- start 2024-05-09T03:16:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.319+0000 7f53b309a640 1 -- start start 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b309a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0b5210 0x7f53ac154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b309a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 0x7f53ac0b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b309a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac155630 0x7f53ac0b7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b309a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac0d0ae0 con 0x7f53ac0d4800 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b309a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac160ea0 con 0x7f53ac155630 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b309a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac163e60 con 0x7f53ac0b5210 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 0x7f53ac0b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 0x7f53ac0b6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51548/0 (socket says 172.21.15.46:51548) 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 -- 172.21.15.46:0/3890906989 learned_addr learned my addr 172.21.15.46:0/3890906989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b1637640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac155630 0x7f53ac0b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:27.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53b0e36640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0b5210 0x7f53ac154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 -- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0b5210 msgr2=0x7f53ac154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0b5210 0x7f53ac154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 -- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac155630 msgr2=0x7f53ac0b7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac155630 0x7f53ac0b7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.320+0000 7f53abfff640 1 -- 172.21.15.46:0/3890906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac0c98f0 con 0x7f53ac0d4800 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.321+0000 7f53abfff640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 0x7f53ac0b6d00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f53a000b420 tx=0x7f53a000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.321+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0011830 con 0x7f53ac0d4800 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.321+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a0016070 con 0x7f53ac0d4800 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.321+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac0c9bd0 con 0x7f53ac0d4800 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.321+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0011c10 con 0x7f53ac0d4800 2024-05-09T03:16:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.321+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac0ca190 con 0x7f53ac0d4800 2024-05-09T03:16:27.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.322+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ac06a270 con 0x7f53ac0d4800 2024-05-09T03:16:27.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.323+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f53a001b460 con 0x7f53ac0d4800 2024-05-09T03:16:27.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.323+0000 7f53a97fa640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53980762d0 0x7f53980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:27.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.324+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5939+0+0 (secure 0 0 0) 0x7f53a0097270 con 0x7f53ac0d4800 2024-05-09T03:16:27.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.324+0000 7f53b0e36640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53980762d0 0x7f53980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:27.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.324+0000 7f53b0e36640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53980762d0 0x7f53980787e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f539c0079a0 tx=0x7f539c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:27.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.328+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f53a0060400 con 0x7f53ac0d4800 2024-05-09T03:16:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.556+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53ac0d40d0 con 0x7f53ac0d4800 2024-05-09T03:16:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.557+0000 7f53a97fa640 1 -- 172.21.15.46:0/3890906989 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+5174 (secure 0 0 0) 0x7f53a0064700 con 0x7f53ac0d4800 2024-05-09T03:16:27.558 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:27.558 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":97,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:27.176199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7569":{"gid":7569,"name":"c","rank":0,"incarnation":97,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/1887013209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1887013209},{"type":"v1","addr":"172.21.15.46:6834","nonce":1887013209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 97 2024-05-09T03:16:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53980762d0 msgr2=0x7f53980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53980762d0 0x7f53980787e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f539c0079a0 tx=0x7f539c008040 comp rx=0 tx=0).stop 2024-05-09T03:16:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 msgr2=0x7f53ac0b6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 0x7f53ac0b6d00 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f53a000b420 tx=0x7f53a000b8e0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 shutdown_connections 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53ac155630 0x7f53ac0b7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53980762d0 0x7f53980787e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53ac0d4800 0x7f53ac0b6d00 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 --2- 172.21.15.46:0/3890906989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53ac0b5210 0x7f53ac154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 >> 172.21.15.46:0/3890906989 conn(0x7f53ac0d29a0 msgr2=0x7f53ac15ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 shutdown_connections 2024-05-09T03:16:27.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:27.559+0000 7f53b309a640 1 -- 172.21.15.46:0/3890906989 wait complete. 2024-05-09T03:16:27.571 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:16:30.194 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:30.193+0000 7f6b5361b640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-05-09T03:16:30.194 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:30.193+0000 7f6b5361b640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-05-09T03:16:30.195 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:30.194+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:30.195 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:30.194+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:16:30.201 INFO:tasks.ceph.mds.c.smithi046.stderr: -16> 2024-05-09T03:16:30.193+0000 7f6b5361b640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-05-09T03:16:30.201 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:16:30.193+0000 7f6b5361b640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-05-09T03:16:30.262 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:32.571 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.700+0000 7f550a0d8640 1 Processor -- start 2024-05-09T03:16:32.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.701+0000 7f550a0d8640 1 -- start start 2024-05-09T03:16:32.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f550a0d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55040b9310 0x7f550408cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:32.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f550a0d8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5504165650 con 0x7f550408d2a0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f550a0d8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f550406ebe0 con 0x7f550408d6b0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f550a0d8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f550406f760 con 0x7f55040b9310 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f5502ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f550408d2a0 0x7f5504164830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54202/0 (socket says 172.21.15.46:54202) 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f5502ffd640 1 -- 172.21.15.46:0/1074428022 learned_addr learned my addr 172.21.15.46:0/1074428022 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f55037fe640 1 --2- 172.21.15.46:0/1074428022 >> v2:172.21.15.88:3300/0 conn(0x7f55040b9310 0x7f550408cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298316255 0 0) 0x7f5504165650 con 0x7f550408d2a0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54ec003520 con 0x7f550408d2a0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1375324601 0 0) 0x7f550406ebe0 con 0x7f550408d6b0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.702+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5504165650 con 0x7f550408d6b0 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.703+0000 7f55037fe640 1 -- 172.21.15.46:0/1074428022 >> v1:172.21.15.88:6789/0 conn(0x7f550408d6b0 legacy=0x7f5504164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.703+0000 7f55037fe640 1 -- 172.21.15.46:0/1074428022 >> v1:172.21.15.46:6789/0 conn(0x7f550408d2a0 legacy=0x7f5504164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.703+0000 7f55037fe640 1 -- 172.21.15.46:0/1074428022 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550406eec0 con 0x7f55040b9310 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.703+0000 7f55037fe640 1 --2- 172.21.15.46:0/1074428022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55040b9310 0x7f550408cd60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f54f40097a0 tx=0x7f54f402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=400ba30377c8f028 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f4037070 con 0x7f55040b9310 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54f4032440 con 0x7f55040b9310 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f55027fc640 1 -- 172.21.15.46:0/1074428022 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54f40327f0 con 0x7f55040b9310 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f550a0d8640 1 -- 172.21.15.46:0/1074428022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55040b9310 msgr2=0x7f550408cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f550a0d8640 1 --2- 172.21.15.46:0/1074428022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55040b9310 0x7f550408cd60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f54f40097a0 tx=0x7f54f402a230 comp rx=0 tx=0).stop 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f550a0d8640 1 -- 172.21.15.46:0/1074428022 shutdown_connections 2024-05-09T03:16:32.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f550a0d8640 1 --2- 172.21.15.46:0/1074428022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55040b9310 0x7f550408cd60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f550a0d8640 1 -- 172.21.15.46:0/1074428022 >> 172.21.15.46:0/1074428022 conn(0x7f55040b5660 msgr2=0x7f55040b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.704+0000 7f550a0d8640 1 -- 172.21.15.46:0/1074428022 shutdown_connections 2024-05-09T03:16:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.705+0000 7f550a0d8640 1 -- 172.21.15.46:0/1074428022 wait complete. 2024-05-09T03:16:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.706+0000 7f550a0d8640 1 Processor -- start 2024-05-09T03:16:32.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.707+0000 7f550a0d8640 1 -- start start 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f550a0d8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f550408d2a0 0x7f55041520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f550a0d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f550408d6b0 0x7f550415d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f550a0d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 0x7f55040b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f550a0d8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f550408dac0 con 0x7f55040b9310 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f550a0d8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f550406f480 con 0x7f550408d6b0 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f550a0d8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f55040cf590 con 0x7f550408d2a0 2024-05-09T03:16:32.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 0x7f55040b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 0x7f55040b7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51574/0 (socket says 172.21.15.46:51574) 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 -- 172.21.15.46:0/4258625278 learned_addr learned my addr 172.21.15.46:0/4258625278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5502ffd640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f550408d6b0 0x7f550415d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f55037fe640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f550408d2a0 0x7f55041520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 -- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f550408d2a0 msgr2=0x7f55041520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f550408d2a0 0x7f55041520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 -- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f550408d6b0 msgr2=0x7f550415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f550408d6b0 0x7f550415d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.708+0000 7f5503fff640 1 -- 172.21.15.46:0/4258625278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55041f7a30 con 0x7f55040b9310 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.709+0000 7f5503fff640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 0x7f55040b7a10 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00c730 tx=0x7f54fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.709+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc003b70 con 0x7f55040b9310 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.709+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54fc003d10 con 0x7f55040b9310 2024-05-09T03:16:32.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.709+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54fc00aa90 con 0x7f55040b9310 2024-05-09T03:16:32.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.709+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550415d920 con 0x7f55040b9310 2024-05-09T03:16:32.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.709+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55041f7100 con 0x7f55040b9310 2024-05-09T03:16:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.710+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f550406a270 con 0x7f55040b9310 2024-05-09T03:16:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.711+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f54fc00ac30 con 0x7f55040b9310 2024-05-09T03:16:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.711+0000 7f54ebfff640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54f0076300 0x7f54f0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.711+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f54fc0974d0 con 0x7f55040b9310 2024-05-09T03:16:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.712+0000 7f55037fe640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54f0076300 0x7f54f0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:32.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.712+0000 7f55037fe640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54f0076300 0x7f54f0078810 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f54f4034000 tx=0x7f54f402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:32.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.716+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f54fc060610 con 0x7f55040b9310 2024-05-09T03:16:32.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.952+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55040b90f0 con 0x7f55040b9310 2024-05-09T03:16:32.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.953+0000 7f54ebfff640 1 -- 172.21.15.46:0/4258625278 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5168 (secure 0 0 0) 0x7f54fc064910 con 0x7f55040b9310 2024-05-09T03:16:32.954 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:32.954 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":7620,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3297564121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3297564121},{"type":"v1","addr":"172.21.15.46:6834","nonce":3297564121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:30.195085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 102 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54f0076300 msgr2=0x7f54f0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54f0076300 0x7f54f0078810 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f54f4034000 tx=0x7f54f402e000 comp rx=0 tx=0).stop 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 msgr2=0x7f55040b7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 0x7f55040b7a10 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f54fc00c730 tx=0x7f54fc00cbf0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 shutdown_connections 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f54f0076300 0x7f54f0078810 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55040b9310 0x7f55040b7a10 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f550408d6b0 0x7f550415d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 --2- 172.21.15.46:0/4258625278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f550408d2a0 0x7f55041520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 >> 172.21.15.46:0/4258625278 conn(0x7f55040b5660 msgr2=0x7f55041604a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 shutdown_connections 2024-05-09T03:16:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:32.955+0000 7f550a0d8640 1 -- 172.21.15.46:0/4258625278 wait complete. 2024-05-09T03:16:32.967 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:16:32.967 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:33.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.082+0000 7f2b14d86640 1 Processor -- start 2024-05-09T03:16:33.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.082+0000 7f2b14d86640 1 -- start start 2024-05-09T03:16:33.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b14d86640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b100d2b10 0x7f2b100b1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b14d86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b100ce6f0 0x7f2b100ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b14d86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b100b2880 con 0x7f2b100d02e0 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b14d86640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2b100b2b60 con 0x7f2b100d2b10 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b14d86640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2b100b2e40 con 0x7f2b100ce6f0 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b0dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b100d2b10 0x7f2b100b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b0dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2b100d2b10 0x7f2b100b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51590/0 (socket says 172.21.15.46:51590) 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.083+0000 7f2b0dd9b640 1 -- 172.21.15.46:0/1243984294 learned_addr learned my addr 172.21.15.46:0/1243984294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.084+0000 7f2b0ed9d640 1 --2- 172.21.15.46:0/1243984294 >> v2:172.21.15.88:3300/0 conn(0x7f2b100ce6f0 0x7f2b100ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.084+0000 7f2b0dd9b640 1 -- 172.21.15.46:0/1243984294 >> v2:172.21.15.88:3300/0 conn(0x7f2b100ce6f0 msgr2=0x7f2b100ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.084+0000 7f2b0dd9b640 1 --2- 172.21.15.46:0/1243984294 >> v2:172.21.15.88:3300/0 conn(0x7f2b100ce6f0 0x7f2b100ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.084+0000 7f2b0dd9b640 1 -- 172.21.15.46:0/1243984294 >> v1:172.21.15.88:6789/0 conn(0x7f2b100d02e0 legacy=0x7f2b100af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.084+0000 7f2b0dd9b640 1 -- 172.21.15.46:0/1243984294 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b100b3120 con 0x7f2b100d2b10 2024-05-09T03:16:33.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.085+0000 7f2b0dd9b640 1 --2- 172.21.15.46:0/1243984294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b100b1950 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2b100cf580 tx=0x7f2b0402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b833a2e72b4c578e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.085+0000 7f2b0d59a640 1 -- 172.21.15.46:0/1243984294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04037070 con 0x7f2b100d2b10 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.085+0000 7f2b0d59a640 1 -- 172.21.15.46:0/1243984294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b04032440 con 0x7f2b100d2b10 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.085+0000 7f2b0d59a640 1 -- 172.21.15.46:0/1243984294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b04032810 con 0x7f2b100d2b10 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 -- 172.21.15.46:0/1243984294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 msgr2=0x7f2b100b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1243984294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b100b1950 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f2b100cf580 tx=0x7f2b0402a270 comp rx=0 tx=0).stop 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 -- 172.21.15.46:0/1243984294 shutdown_connections 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1243984294 >> v2:172.21.15.88:3300/0 conn(0x7f2b100ce6f0 0x7f2b100ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1243984294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b100b1950 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 -- 172.21.15.46:0/1243984294 >> 172.21.15.46:0/1243984294 conn(0x7f2b1009b7d0 msgr2=0x7f2b1009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 -- 172.21.15.46:0/1243984294 shutdown_connections 2024-05-09T03:16:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.086+0000 7f2b14d86640 1 -- 172.21.15.46:0/1243984294 wait complete. 2024-05-09T03:16:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.088+0000 7f2b14d86640 1 Processor -- start 2024-05-09T03:16:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.088+0000 7f2b14d86640 1 -- start start 2024-05-09T03:16:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b14d86640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100ce6f0 0x7f2b100ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b14d86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100d02e0 0x7f2b100cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b14d86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b1015bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b14d86640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b100b2530 con 0x7f2b100d2b10 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b14d86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b100b1e90 con 0x7f2b100d02e0 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b14d86640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2b100cf040 con 0x7f2b100ce6f0 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b0ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b1015bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.089+0000 7f2b0ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b1015bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51604/0 (socket says 172.21.15.46:51604) 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 -- 172.21.15.46:0/1588706281 learned_addr learned my addr 172.21.15.46:0/1588706281 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0dd9b640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100d02e0 0x7f2b100cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0e59c640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100ce6f0 0x7f2b100ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 -- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100ce6f0 msgr2=0x7f2b100ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100ce6f0 0x7f2b100ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 -- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100d02e0 msgr2=0x7f2b100cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100d02e0 0x7f2b100cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 -- 172.21.15.46:0/1588706281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b100c9b20 con 0x7f2b100d2b10 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b0ed9d640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b1015bb90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800efc0 tx=0x7f2b0800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08018020 con 0x7f2b100d2b10 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b08014480 con 0x7f2b100d2b10 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b100cbb40 con 0x7f2b100d2b10 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08014870 con 0x7f2b100d2b10 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.090+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b100ca0a0 con 0x7f2b100d2b10 2024-05-09T03:16:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.091+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b100cb8b0 con 0x7f2b100d2b10 2024-05-09T03:16:33.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.092+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2b08005d10 con 0x7f2b100d2b10 2024-05-09T03:16:33.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.092+0000 7f2afeffd640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2af0076300 0x7f2af0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.093+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f2b08097fd0 con 0x7f2b100d2b10 2024-05-09T03:16:33.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.093+0000 7f2b0e59c640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2af0076300 0x7f2af0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.093+0000 7f2b0e59c640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2af0076300 0x7f2af0078810 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2b00000f60 tx=0x7f2b00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.097+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2b08022070 con 0x7f2b100d2b10 2024-05-09T03:16:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.328+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b1006a270 con 0x7f2b100d2b10 2024-05-09T03:16:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.328+0000 7f2afeffd640 1 -- 172.21.15.46:0/1588706281 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5168 (secure 0 0 0) 0x7f2b08010040 con 0x7f2b100d2b10 2024-05-09T03:16:33.329 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:33.330 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":7620,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3297564121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3297564121},{"type":"v1","addr":"172.21.15.46:6834","nonce":3297564121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:30.195085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 102 2024-05-09T03:16:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2af0076300 msgr2=0x7f2af0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2af0076300 0x7f2af0078810 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2b00000f60 tx=0x7f2b00009040 comp rx=0 tx=0).stop 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 msgr2=0x7f2b1015bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b1015bb90 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800efc0 tx=0x7f2b0800c490 comp rx=0 tx=0).stop 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 shutdown_connections 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2af0076300 0x7f2af0078810 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b100d2b10 0x7f2b1015bb90 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b100d02e0 0x7f2b100cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 --2- 172.21.15.46:0/1588706281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2b100ce6f0 0x7f2b100ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 >> 172.21.15.46:0/1588706281 conn(0x7f2b1009b7d0 msgr2=0x7f2b1006ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 shutdown_connections 2024-05-09T03:16:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.331+0000 7f2b14d86640 1 -- 172.21.15.46:0/1588706281 wait complete. 2024-05-09T03:16:33.343 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:33.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.456+0000 7f25733ce640 1 Processor -- start 2024-05-09T03:16:33.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.456+0000 7f25733ce640 1 -- start start 2024-05-09T03:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f25733ce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f256c1613c0 0x7f256c0b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f25733ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256c0b8110 con 0x7f256c0b5210 2024-05-09T03:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f25733ce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256c1617d0 con 0x7f256c0b7d00 2024-05-09T03:16:33.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f25733ce640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f256c0b5620 con 0x7f256c1613c0 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f257116a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f256c0b5210 0x7f256c0d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54214/0 (socket says 172.21.15.46:54214) 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f257116a640 1 -- 172.21.15.46:0/2412959381 learned_addr learned my addr 172.21.15.46:0/2412959381 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2570969640 1 --2- 172.21.15.46:0/2412959381 >> v2:172.21.15.88:3300/0 conn(0x7f256c1613c0 0x7f256c0b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2563fff640 1 -- 172.21.15.46:0/2412959381 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3728290139 0 0) 0x7f2568001110 con 0x7f256c0b5210 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2563fff640 1 -- 172.21.15.46:0/2412959381 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2558003520 con 0x7f256c0b5210 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2563fff640 1 -- 172.21.15.46:0/2412959381 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 696008688 0 0) 0x7f2558003520 con 0x7f256c0b5210 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2570969640 1 -- 172.21.15.46:0/2412959381 >> v1:172.21.15.88:6789/0 conn(0x7f256c0b7d00 legacy=0x7f256c163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2570969640 1 -- 172.21.15.46:0/2412959381 >> v1:172.21.15.46:6789/0 conn(0x7f256c0b5210 legacy=0x7f256c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.458+0000 7f2570969640 1 -- 172.21.15.46:0/2412959381 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c0d5280 con 0x7f256c1613c0 2024-05-09T03:16:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.459+0000 7f2570969640 1 --2- 172.21.15.46:0/2412959381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c1613c0 0x7f256c0b77c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f255c0097a0 tx=0x7f255c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408cb49ca919938c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.459+0000 7f2563fff640 1 -- 172.21.15.46:0/2412959381 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c037070 con 0x7f256c1613c0 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.459+0000 7f2563fff640 1 -- 172.21.15.46:0/2412959381 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f255c02ac50 con 0x7f256c1613c0 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f2563fff640 1 -- 172.21.15.46:0/2412959381 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680018c0 con 0x7f256c1613c0 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 -- 172.21.15.46:0/2412959381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c1613c0 msgr2=0x7f256c0b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 --2- 172.21.15.46:0/2412959381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c1613c0 0x7f256c0b77c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f255c0097a0 tx=0x7f255c02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 -- 172.21.15.46:0/2412959381 shutdown_connections 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 --2- 172.21.15.46:0/2412959381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c1613c0 0x7f256c0b77c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 -- 172.21.15.46:0/2412959381 >> 172.21.15.46:0/2412959381 conn(0x7f256c0d29a0 msgr2=0x7f256c0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 -- 172.21.15.46:0/2412959381 shutdown_connections 2024-05-09T03:16:33.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.460+0000 7f25733ce640 1 -- 172.21.15.46:0/2412959381 wait complete. 2024-05-09T03:16:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.462+0000 7f25733ce640 1 Processor -- start 2024-05-09T03:16:33.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.462+0000 7f25733ce640 1 -- start start 2024-05-09T03:16:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.463+0000 7f25733ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f256c0b5210 0x7f256c0cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.463+0000 7f25733ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c0b7d00 0x7f256c0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.463+0000 7f25733ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 0x7f256c0c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.463+0000 7f25733ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f256c160bc0 con 0x7f256c1613c0 2024-05-09T03:16:33.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.463+0000 7f25733ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f256c1609a0 con 0x7f256c0b7d00 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.463+0000 7f25733ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f256c160ea0 con 0x7f256c0b5210 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 0x7f256c0c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 0x7f256c0c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51606/0 (socket says 172.21.15.46:51606) 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 -- 172.21.15.46:0/552369248 learned_addr learned my addr 172.21.15.46:0/552369248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f2570969640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c0b7d00 0x7f256c0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 -- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f256c0b5210 msgr2=0x7f256c0cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257116a640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f256c0b5210 0x7f256c0cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f256c0b5210 0x7f256c0cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 -- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c0b7d00 msgr2=0x7f256c0c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c0b7d00 0x7f256c0c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 -- 172.21.15.46:0/552369248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256c0cdfa0 con 0x7f256c1613c0 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f257196b640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 0x7f256c0c9d90 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f256400c6e0 tx=0x7f256400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564003b70 con 0x7f256c1613c0 2024-05-09T03:16:33.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2564003d10 con 0x7f256c1613c0 2024-05-09T03:16:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25640052f0 con 0x7f256c1613c0 2024-05-09T03:16:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256c0ce280 con 0x7f256c1613c0 2024-05-09T03:16:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.464+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f256c15bca0 con 0x7f256c1613c0 2024-05-09T03:16:33.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.465+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f256c0cbdb0 con 0x7f256c1613c0 2024-05-09T03:16:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.466+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2564005490 con 0x7f256c1613c0 2024-05-09T03:16:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.467+0000 7f2561ffb640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2550076300 0x7f2550078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.467+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f2564097620 con 0x7f256c1613c0 2024-05-09T03:16:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.467+0000 7f257116a640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2550076300 0x7f2550078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.467+0000 7f257116a640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2550076300 0x7f2550078810 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2568001eb0 tx=0x7f2568009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.471+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f256c0cbdb0 con 0x7f256c1613c0 2024-05-09T03:16:33.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.700+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f256c06a270 con 0x7f256c1613c0 2024-05-09T03:16:33.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.701+0000 7f2561ffb640 1 -- 172.21.15.46:0/552369248 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5168 (secure 0 0 0) 0x7f2564060d20 con 0x7f256c1613c0 2024-05-09T03:16:33.702 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:33.702 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":102,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":7620,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3297564121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3297564121},{"type":"v1","addr":"172.21.15.46:6834","nonce":3297564121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:30.195085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:33.702 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 102 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2550076300 msgr2=0x7f2550078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2550076300 0x7f2550078810 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2568001eb0 tx=0x7f2568009040 comp rx=0 tx=0).stop 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 msgr2=0x7f256c0c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 0x7f256c0c9d90 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f256400c6e0 tx=0x7f256400cba0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 shutdown_connections 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2550076300 0x7f2550078810 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f256c1613c0 0x7f256c0c9d90 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f256c0b7d00 0x7f256c0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 --2- 172.21.15.46:0/552369248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f256c0b5210 0x7f256c0cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 >> 172.21.15.46:0/552369248 conn(0x7f256c0d29a0 msgr2=0x7f256c162cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 shutdown_connections 2024-05-09T03:16:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.703+0000 7f25733ce640 1 -- 172.21.15.46:0/552369248 wait complete. 2024-05-09T03:16:33.715 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 600.00000000' led to DAMAGED state 2024-05-09T03:16:33.715 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000.inode' 2024-05-09T03:16:33.715 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:33.715 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:33.731 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:16:33.731 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:33.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:33.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:33.732 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:16:33.787 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:16:33.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.942+0000 7f11ab1b2640 1 Processor -- start 2024-05-09T03:16:33.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.942+0000 7f11ab1b2640 1 -- start start 2024-05-09T03:16:33.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11ab1b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a40b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11ab1b2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a40b9690 con 0x7f11a40d1110 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11ab1b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a40ca9e0 con 0x7f11a40d1740 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11ab1b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11a40d2b00 con 0x7f11a40cfcf0 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11a3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11a40d1110 0x7f11a409cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54222/0 (socket says 172.21.15.46:54222) 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11a3fff640 1 -- 172.21.15.46:0/3782124204 learned_addr learned my addr 172.21.15.46:0/3782124204 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11a974f640 1 --2- 172.21.15.46:0/3782124204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a40b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.943+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993002331 0 0) 0x7f11a40b9690 con 0x7f11a40d1110 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f118c003520 con 0x7f11a40d1110 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2184418796 0 0) 0x7f11a40ca9e0 con 0x7f11a40d1740 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11a40b9690 con 0x7f11a40d1740 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a974f640 1 -- 172.21.15.46:0/3782124204 >> v1:172.21.15.88:6789/0 conn(0x7f11a40d1740 legacy=0x7f11a40d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a974f640 1 -- 172.21.15.46:0/3782124204 >> v1:172.21.15.46:6789/0 conn(0x7f11a40d1110 legacy=0x7f11a409cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a974f640 1 -- 172.21.15.46:0/3782124204 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a408cfb0 con 0x7f11a40cfcf0 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.944+0000 7f11a974f640 1 --2- 172.21.15.46:0/3782124204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a40b73d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f119c009dc0 tx=0x7f119c02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8606f95acf9fb8ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c007740 con 0x7f11a40cfcf0 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f119c0078e0 con 0x7f11a40cfcf0 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11a37fe640 1 -- 172.21.15.46:0/3782124204 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119c007c90 con 0x7f11a40cfcf0 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 -- 172.21.15.46:0/3782124204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 msgr2=0x7f11a40b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/3782124204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a40b73d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f119c009dc0 tx=0x7f119c02a6d0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 -- 172.21.15.46:0/3782124204 shutdown_connections 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/3782124204 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a40b73d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 -- 172.21.15.46:0/3782124204 >> 172.21.15.46:0/3782124204 conn(0x7f11a40d05f0 msgr2=0x7f11a40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 -- 172.21.15.46:0/3782124204 shutdown_connections 2024-05-09T03:16:33.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.945+0000 7f11ab1b2640 1 -- 172.21.15.46:0/3782124204 wait complete. 2024-05-09T03:16:33.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.947+0000 7f11ab1b2640 1 Processor -- start 2024-05-09T03:16:33.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.947+0000 7f11ab1b2640 1 -- start start 2024-05-09T03:16:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11ab1b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a41616c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11ab1b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 0x7f11a4151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11ab1b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11a40d1740 0x7f11a415f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11ab1b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11a40b93b0 con 0x7f11a40d1110 2024-05-09T03:16:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11ab1b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11a408cd60 con 0x7f11a40d1740 2024-05-09T03:16:33.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11ab1b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11a409d3b0 con 0x7f11a40cfcf0 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 0x7f11a4151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 0x7f11a4151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51618/0 (socket says 172.21.15.46:51618) 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 -- 172.21.15.46:0/454024530 learned_addr learned my addr 172.21.15.46:0/454024530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a8f4e640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a41616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a974f640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11a40d1740 0x7f11a415f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 -- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 msgr2=0x7f11a41616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a41616c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 -- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11a40d1740 msgr2=0x7f11a415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11a40d1740 0x7f11a415f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 -- 172.21.15.46:0/454024530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a406f490 con 0x7f11a40d1110 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.949+0000 7f11a3fff640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 0x7f11a4151f60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f119800c3c0 tx=0x7f119800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.950+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1198017070 con 0x7f11a40d1110 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.950+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1198005ce0 con 0x7f11a40d1110 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.950+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a406f770 con 0x7f11a40d1110 2024-05-09T03:16:33.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.950+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11980072f0 con 0x7f11a40d1110 2024-05-09T03:16:33.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.950+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a40cd2b0 con 0x7f11a40d1110 2024-05-09T03:16:33.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.951+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a40b65b0 con 0x7f11a40d1110 2024-05-09T03:16:33.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.951+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f119801a070 con 0x7f11a40d1110 2024-05-09T03:16:33.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.952+0000 7f11a17fa640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1190076300 0x7f1190078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:33.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.952+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f1198096a20 con 0x7f11a40d1110 2024-05-09T03:16:33.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.952+0000 7f11a8f4e640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1190076300 0x7f1190078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:33.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.953+0000 7f11a8f4e640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1190076300 0x7f1190078810 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1194001120 tx=0x7f1194009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:33.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:33.957+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11a40b65b0 con 0x7f11a40d1110 2024-05-09T03:16:34.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.185+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f11a406a270 con 0x7f11a40d1110 2024-05-09T03:16:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.212+0000 7f11a17fa640 1 -- 172.21.15.46:0/454024530 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v103) ==== 161+0+0 (secure 0 0 0) 0x7f119805fb60 con 0x7f11a40d1110 2024-05-09T03:16:34.213 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1190076300 msgr2=0x7f1190078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1190076300 0x7f1190078810 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1194001120 tx=0x7f1194009040 comp rx=0 tx=0).stop 2024-05-09T03:16:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 msgr2=0x7f11a4151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 0x7f11a4151f60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f119800c3c0 tx=0x7f119800c880 comp rx=0 tx=0).stop 2024-05-09T03:16:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 shutdown_connections 2024-05-09T03:16:34.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11a40d1740 0x7f11a415f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1190076300 0x7f1190078810 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a40d1110 0x7f11a4151f60 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 --2- 172.21.15.46:0/454024530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11a40cfcf0 0x7f11a41616c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 >> 172.21.15.46:0/454024530 conn(0x7f11a40d05f0 msgr2=0x7f11a40b6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:34.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 shutdown_connections 2024-05-09T03:16:34.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.215+0000 7f11ab1b2640 1 -- 172.21.15.46:0/454024530 wait complete. 2024-05-09T03:16:34.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.343+0000 7f7a9637e640 1 Processor -- start 2024-05-09T03:16:34.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.343+0000 7f7a9637e640 1 -- start start 2024-05-09T03:16:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.344+0000 7f7a9637e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a900d2eb0 0x7f7a900d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.344+0000 7f7a9637e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 0x7f7a900b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.344+0000 7f7a9637e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a900b80b0 con 0x7f7a900d38b0 2024-05-09T03:16:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.344+0000 7f7a9637e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7a900d4a60 con 0x7f7a900d2eb0 2024-05-09T03:16:34.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.344+0000 7f7a9637e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7a900d4d40 con 0x7f7a900b7760 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 0x7f7a900b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a900d2eb0 0x7f7a900d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a900d2eb0 0x7f7a900d32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:44264/0 (socket says 172.21.15.46:44264) 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 0x7f7a900b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:51944/0 (socket says 172.21.15.46:51944) 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 -- 172.21.15.46:0/2812414870 learned_addr learned my addr 172.21.15.46:0/2812414870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a9491b640 1 --1- 172.21.15.46:0/2812414870 >> v1:172.21.15.88:6789/0 conn(0x7f7a900d38b0 0x7f7a900d3cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:42772/0 (socket says 172.21.15.46:42772) 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 -- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 msgr2=0x7f7a900b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 --2- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 0x7f7a900b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 -- 172.21.15.46:0/2812414870 >> v1:172.21.15.88:6789/0 conn(0x7f7a900d38b0 legacy=0x7f7a900d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8ffff640 1 -- 172.21.15.46:0/2812414870 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a900d5020 con 0x7f7a900d2eb0 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.345+0000 7f7a8f7fe640 1 --2- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 0x7f7a900b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:34.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.346+0000 7f7a8ffff640 1 --2- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900d2eb0 0x7f7a900d32c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009870 tx=0x7f7a8002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a68c5321f32e960 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.346+0000 7f7a8effd640 1 -- 172.21.15.46:0/2812414870 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80037070 con 0x7f7a900d2eb0 2024-05-09T03:16:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.346+0000 7f7a8effd640 1 -- 172.21.15.46:0/2812414870 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a80032440 con 0x7f7a900d2eb0 2024-05-09T03:16:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.346+0000 7f7a8effd640 1 -- 172.21.15.46:0/2812414870 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80032810 con 0x7f7a900d2eb0 2024-05-09T03:16:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 -- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900d2eb0 msgr2=0x7f7a900d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 --2- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900d2eb0 0x7f7a900d32c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009870 tx=0x7f7a8002a250 comp rx=0 tx=0).stop 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 -- 172.21.15.46:0/2812414870 shutdown_connections 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 --2- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900b7760 0x7f7a900b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 --2- 172.21.15.46:0/2812414870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900d2eb0 0x7f7a900d32c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 -- 172.21.15.46:0/2812414870 >> 172.21.15.46:0/2812414870 conn(0x7f7a9008d390 msgr2=0x7f7a9008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 -- 172.21.15.46:0/2812414870 shutdown_connections 2024-05-09T03:16:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.347+0000 7f7a9637e640 1 -- 172.21.15.46:0/2812414870 wait complete. 2024-05-09T03:16:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.349+0000 7f7a9637e640 1 Processor -- start 2024-05-09T03:16:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.349+0000 7f7a9637e640 1 -- start start 2024-05-09T03:16:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9637e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900b7760 0x7f7a9014c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9637e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900d2eb0 0x7f7a9014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9637e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 0x7f7a901e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9637e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7a900b51e0 con 0x7f7a900d38b0 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9637e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7a900d55e0 con 0x7f7a900b7760 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9637e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7a900b5360 con 0x7f7a900d2eb0 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9491b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 0x7f7a901e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9491b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 0x7f7a901e8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51620/0 (socket says 172.21.15.46:51620) 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.350+0000 7f7a9491b640 1 -- 172.21.15.46:0/937160294 learned_addr learned my addr 172.21.15.46:0/937160294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a8f7fe640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900d2eb0 0x7f7a9014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a8ffff640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900b7760 0x7f7a9014c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9491b640 1 -- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900d2eb0 msgr2=0x7f7a9014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9491b640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900d2eb0 0x7f7a9014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9491b640 1 -- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900b7760 msgr2=0x7f7a9014c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9491b640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900b7760 0x7f7a9014c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9491b640 1 -- 172.21.15.46:0/937160294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9014c360 con 0x7f7a900d38b0 2024-05-09T03:16:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9491b640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 0x7f7a901e8b80 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800c730 tx=0x7f7a8800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88017070 con 0x7f7a900d38b0 2024-05-09T03:16:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8800ce40 con 0x7f7a900d38b0 2024-05-09T03:16:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800a840 con 0x7f7a900d38b0 2024-05-09T03:16:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a900ca730 con 0x7f7a900d38b0 2024-05-09T03:16:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.351+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a900cacf0 con 0x7f7a900d38b0 2024-05-09T03:16:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.352+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a9006a270 con 0x7f7a900d38b0 2024-05-09T03:16:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.353+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a8801a070 con 0x7f7a900d38b0 2024-05-09T03:16:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.354+0000 7f7a8cff9640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a7c0762d0 0x7f7a7c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.354+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f7a88096be0 con 0x7f7a900d38b0 2024-05-09T03:16:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.354+0000 7f7a8ffff640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a7c0762d0 0x7f7a7c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.354+0000 7f7a8ffff640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a7c0762d0 0x7f7a7c0787e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7a80034000 tx=0x7f7a8002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:34.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.359+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7a900b80b0 con 0x7f7a900d38b0 2024-05-09T03:16:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:34.623+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f7a900d6ab0 con 0x7f7a900d38b0 2024-05-09T03:16:35.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.217+0000 7f7a8cff9640 1 -- 172.21.15.46:0/937160294 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v104) ==== 101+0+0 (secure 0 0 0) 0x7f7a880683f0 con 0x7f7a900d38b0 2024-05-09T03:16:35.218 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a7c0762d0 msgr2=0x7f7a7c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a7c0762d0 0x7f7a7c0787e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7a80034000 tx=0x7f7a8002e000 comp rx=0 tx=0).stop 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 msgr2=0x7f7a901e8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 0x7f7a901e8b80 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800c730 tx=0x7f7a8800cbf0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 shutdown_connections 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a7c0762d0 0x7f7a7c0787e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a900d38b0 0x7f7a901e8b80 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a900d2eb0 0x7f7a9014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 --2- 172.21.15.46:0/937160294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a900b7760 0x7f7a9014c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 >> 172.21.15.46:0/937160294 conn(0x7f7a9008d390 msgr2=0x7f7a900b5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:35.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 shutdown_connections 2024-05-09T03:16:35.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.220+0000 7f7a9637e640 1 -- 172.21.15.46:0/937160294 wait complete. 2024-05-09T03:16:35.234 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:35.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.295+0000 7f0bfd590f40 1 Processor -- start 2024-05-09T03:16:35.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.295+0000 7f0bfd590f40 1 -- start start 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bfd590f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0be00030f0 0x55b9ffd7eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bfd590f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 0x55b9ffd7c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bfd590f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b9ffe03d90 0x55b9ffd7ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bfd590f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b9ffce1f50 con 0x55b9ffe03980 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bfd590f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b9ffce5f90 con 0x55b9ffe03d90 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bfd590f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b9ffd7f4d0 con 0x7f0be00030f0 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bf6857640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 0x55b9ffd7c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bf6857640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 0x55b9ffd7c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51636/0 (socket says 172.21.15.46:51636) 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.296+0000 7f0bf6857640 1 -- 172.21.15.46:0/3798818950 learned_addr learned my addr 172.21.15.46:0/3798818950 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf7058640 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0be00030f0 0x55b9ffd7eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf7859640 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b9ffe03d90 0x55b9ffd7ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf6857640 1 -- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0be00030f0 msgr2=0x55b9ffd7eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf6857640 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0be00030f0 0x55b9ffd7eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf6857640 1 -- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b9ffe03d90 msgr2=0x55b9ffd7ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf6857640 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b9ffe03d90 0x55b9ffd7ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf6857640 1 -- 172.21.15.46:0/3798818950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9ffd69440 con 0x55b9ffe03980 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf7859640 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b9ffe03d90 0x55b9ffd7ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf6857640 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 0x55b9ffd7c910 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00b3f0 tx=0x7f0bec051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81ef5e1b0c99a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf5855640 1 -- 172.21.15.46:0/3798818950 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec054470 con 0x55b9ffe03980 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf5855640 1 -- 172.21.15.46:0/3798818950 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bec0549c0 con 0x55b9ffe03980 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.297+0000 7f0bf5855640 1 -- 172.21.15.46:0/3798818950 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec054d70 con 0x55b9ffe03980 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 -- 172.21.15.46:0/3798818950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 msgr2=0x55b9ffd7c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 0x55b9ffd7c910 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00b3f0 tx=0x7f0bec051000 comp rx=0 tx=0).stop 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 -- 172.21.15.46:0/3798818950 shutdown_connections 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b9ffe03d90 0x55b9ffd7ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03980 0x55b9ffd7c910 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/3798818950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0be00030f0 0x55b9ffd7eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 -- 172.21.15.46:0/3798818950 >> 172.21.15.46:0/3798818950 conn(0x55b9ffc5dac0 msgr2=0x55b9ffd836b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 -- 172.21.15.46:0/3798818950 shutdown_connections 2024-05-09T03:16:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.298+0000 7f0bfd590f40 1 -- 172.21.15.46:0/3798818950 wait complete. 2024-05-09T03:16:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.299+0000 7f0bfd590f40 1 Processor -- start 2024-05-09T03:16:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.299+0000 7f0bfd590f40 1 -- start start 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bfd590f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b9ffe03980 0x55b9ffd85550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bfd590f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03d90 0x55b9ffe02a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bfd590f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be00030f0 0x55b9ffe13390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bfd590f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b9ffe11d50 con 0x55b9ffe03d90 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bfd590f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b9ffe0d280 con 0x7f0be00030f0 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bfd590f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b9ffe032a0 con 0x55b9ffe03980 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bf6857640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03d90 0x55b9ffe02a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bf6857640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03d90 0x55b9ffe02a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51642/0 (socket says 172.21.15.46:51642) 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 learned_addr learned my addr 172.21.15.46:0/1402149053 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b9ffe03980 0x55b9ffd85550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.300+0000 7f0bf7859640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be00030f0 0x55b9ffe13390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b9ffe03980 msgr2=0x55b9ffd85550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b9ffe03980 0x55b9ffd85550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be00030f0 msgr2=0x55b9ffe13390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0be00030f0 0x55b9ffe13390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9ffe0cdb0 con 0x55b9ffe03d90 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03d90 0x55b9ffe02a30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0bec001e50 tx=0x7f0bec001270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf4853640 1 -- 172.21.15.46:0/1402149053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec054470 con 0x55b9ffe03d90 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf4853640 1 -- 172.21.15.46:0/1402149053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bec054b70 con 0x55b9ffe03d90 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b9ffe0abc0 con 0x55b9ffe03d90 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bf4853640 1 -- 172.21.15.46:0/1402149053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec068480 con 0x55b9ffe03d90 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.301+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9ffe0a430 con 0x55b9ffe03d90 2024-05-09T03:16:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.303+0000 7f0bf4853640 1 -- 172.21.15.46:0/1402149053 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0bec068620 con 0x55b9ffe03d90 2024-05-09T03:16:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.303+0000 7f0bf4853640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0be4074ce0 0x7f0be40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.303+0000 7f0bf4853640 1 -- 172.21.15.46:0/1402149053 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f0bec0db7a0 con 0x55b9ffe03d90 2024-05-09T03:16:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.304+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0be4074ce0 0x7f0be40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.307 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.304+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b9ffd7df10 0x55b9ffeb4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.304+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffd68ae0 con 0x55b9ffd7df10 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.304+0000 7f0bf7859640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b9ffd7df10 0x55b9ffeb4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.304+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0be4074ce0 0x7f0be40771f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0be8005780 tx=0x7f0be8007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.304+0000 7f0bf7859640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b9ffd7df10 0x55b9ffeb4820 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.306+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v87'73 uv48 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.306+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffeb4d60 con 0x55b9ffd7df10 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.307+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v87'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.307+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5100 con 0x55b9ffd7df10 2024-05-09T03:16:35.309 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.309+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v87'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.309+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb64b0 con 0x55b9ffd7df10 2024-05-09T03:16:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.310+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v87'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.310+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb6850 con 0x55b9ffd7df10 2024-05-09T03:16:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.311+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v87'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.311+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb54a0 con 0x55b9ffd7df10 2024-05-09T03:16:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.312+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v87'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.312+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5870 con 0x55b9ffd7df10 2024-05-09T03:16:35.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.314+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v87'79 uv54 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.314+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffeb5870 con 0x55b9ffd7df10 2024-05-09T03:16:35.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.315+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v87'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.315+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5870 con 0x55b9ffd7df10 2024-05-09T03:16:35.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.316+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v87'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.317 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:35.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.316+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.317+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v87'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.317+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.319+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v87'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.319+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.320+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v87'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.320+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.323+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v87'85 uv59 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.323+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.326+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v87'86 uv86 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.326+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.328+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v87'87 uv87 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.329 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.328+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.331+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v87'88 uv88 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.331+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.333+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v87'89 uv89 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.333+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.336+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v87'90 uv90 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.336+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.339+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v87'91 uv91 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.339+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.341 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:35.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.340+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v87'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.340+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.342+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v87'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.342+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.343+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v87'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.343+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.344+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v87'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.344+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.345+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 603.00000000 [create] v87'96 uv71 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.345+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.347+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 603.00000000 [delete] v87'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.347+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.348+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v87'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.348 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:35.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.348+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.349+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v87'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.349+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.350+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v87'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.350+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffd7df10 2024-05-09T03:16:35.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.352+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v87'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.352+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b9ffeb6bf0 0x55b9ffeb9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.352+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:30 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb5840 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.352+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b9ffeb6bf0 0x55b9ffeb9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.352+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b9ffeb6bf0 0x55b9ffeb9010 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.354+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(30 1.00000000.inode [create] v87'67 uv49 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.354+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:31 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffeb9550 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.355+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(31 1.00000000.inode [delete] v87'68 uv68 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.355+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb98f0 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.358 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.358+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(32 1.00000000.inode [create] v87'69 uv69 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.358+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeb9c90 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.360+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v87'70 uv70 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.360+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeba030 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.363+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v87'71 uv71 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.363+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffeba3d0 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.365+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(35 200.00000000 [create] v87'72 uv54 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.365+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffeba770 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.368+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(36 200.00000000 [delete] v87'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.368+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.371 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.371+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(37 200.00000000 [create] v87'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.371+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.373+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v87'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.373+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.376+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v87'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.376+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.377+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(40 200.00000001 [create] v87'77 uv59 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.377+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.379+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(41 200.00000001 [delete] v87'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.379+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.380+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(42 200.00000001 [create] v87'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.380 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:35.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.380+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe570 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.381+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(43 200.00000001 [write 0~14895] v87'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.381+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe570 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.382+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v87'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.382+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.383+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(45 606.00000000 [create] v87'82 uv65 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.383+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.384+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(46 606.00000000 [delete] v87'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.384+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.385 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:35.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.385+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(47 606.00000000 [create] v87'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.385+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebbb20 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.386+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v87'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.386+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebbb20 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.387+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v87'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.387+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.388+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v87'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffeb6bf0 2024-05-09T03:16:35.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.388+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffd7df10 2024-05-09T03:16:35.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.389+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(51 607.00000000 [create] v87'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.389+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffd7df10 2024-05-09T03:16:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.390+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(52 607.00000000 [delete] v87'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.390+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffd7df10 2024-05-09T03:16:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.391+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v87'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.392 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:35.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.391+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebbb20 con 0x55b9ffd7df10 2024-05-09T03:16:35.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.392+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v87'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.392+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebbb20 con 0x55b9ffd7df10 2024-05-09T03:16:35.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.393+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v87'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.393+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebab10 con 0x55b9ffd7df10 2024-05-09T03:16:35.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.394+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [omap-set-header] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.394+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b9ffebb020 0x55b9ffebd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.394+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebd980 con 0x55b9ffebb020 2024-05-09T03:16:35.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.395+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b9ffebb020 0x55b9ffebd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.395+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b9ffebb020 0x55b9ffebd440 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.396+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v87'15 uv14 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.396+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffebdd20 con 0x55b9ffebb020 2024-05-09T03:16:35.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.397+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v87'16 uv16 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.397+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe220 con 0x55b9ffebb020 2024-05-09T03:16:35.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.398+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v87'17 uv17 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.399 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.398+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffe139b0 con 0x55b9ffebb020 2024-05-09T03:16:35.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.399+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v87'18 uv18 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.399+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebeb50 con 0x55b9ffd7df10 2024-05-09T03:16:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.400+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(61 608.00000000 [create] v87'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.400+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffebeb50 con 0x55b9ffd7df10 2024-05-09T03:16:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.401+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(62 608.00000000 [delete] v87'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.401+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebeb50 con 0x55b9ffd7df10 2024-05-09T03:16:35.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.403+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v87'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.403 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:35.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.403+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebfb60 con 0x55b9ffd7df10 2024-05-09T03:16:35.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.404+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v87'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.404+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebfb60 con 0x55b9ffd7df10 2024-05-09T03:16:35.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.405+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v87'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.405+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebeb50 con 0x55b9ffd7df10 2024-05-09T03:16:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.406+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [omap-set-header] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.406+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b9ffebf1d0 0x55b9ffebf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.406+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.407+0000 7f0bf7859640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b9ffebf1d0 0x55b9ffebf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.407+0000 7f0bf7859640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b9ffebf1d0 0x55b9ffebf5e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.408+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(67 604.00000000 [create] v87'26 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.408+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.411+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(68 604.00000000 [delete] v87'27 uv27 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.411+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.413+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(69 604.00000000 [create] v87'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.413 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:35.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.413+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec2fd0 con 0x55b9ffebf1d0 2024-05-09T03:16:35.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.414+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v87'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.414+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec2f90 con 0x55b9ffebf1d0 2024-05-09T03:16:35.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.415+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v87'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.415+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.416+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v87'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.416+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.417+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(73 500.00000000 [create] v87'32 uv25 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.417+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.418+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(74 500.00000000 [delete] v87'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.418+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.419+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(75 500.00000000 [create] v87'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.420 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:35.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.419+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.420+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v87'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.420+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec1f80 con 0x55b9ffebf1d0 2024-05-09T03:16:35.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.421+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v87'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.421+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b9ffec2880 0x55b9ffec2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.421+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec53a0 con 0x55b9ffec2880 2024-05-09T03:16:35.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.421+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b9ffec2880 0x55b9ffec2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.422+0000 7f0bf7058640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b9ffec2880 0x55b9ffec2c90 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.423+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(78 mds_snaptable [create] v87'12 uv11 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffec2880 2024-05-09T03:16:35.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.423+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec53a0 con 0x55b9ffec2880 2024-05-09T03:16:35.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.424+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(79 mds_snaptable [delete] v87'13 uv13 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffec2880 2024-05-09T03:16:35.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.425+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffec2880 2024-05-09T03:16:35.428 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:35.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.427+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(80 mds_snaptable [create] v87'14 uv14 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffec2880 2024-05-09T03:16:35.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.427+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffec2880 2024-05-09T03:16:35.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.430+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v87'15 uv15 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffec2880 2024-05-09T03:16:35.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.430+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffec2880 2024-05-09T03:16:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.433+0000 7f0bf7058640 1 -- 172.21.15.46:0/1402149053 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v87'16 uv16 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0be8014d50 con 0x55b9ffec2880 2024-05-09T03:16:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.433+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffd7df10 2024-05-09T03:16:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.434+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(83 605.00000000 [create] v87'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.434+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffd7df10 2024-05-09T03:16:35.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.435+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(84 605.00000000 [delete] v87'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.435+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffd7df10 2024-05-09T03:16:35.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.436+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v87'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.437 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.436+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec67d0 con 0x55b9ffd7df10 2024-05-09T03:16:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.437+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v87'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.437+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec67d0 con 0x55b9ffd7df10 2024-05-09T03:16:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.438+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v87'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.438+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffd7df10 2024-05-09T03:16:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.439+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [omap-set-header] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffd7df10 2024-05-09T03:16:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.439+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec57c0 con 0x55b9ffebb020 2024-05-09T03:16:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.440+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(89 mds0_inotable [create] v87'24 uv18 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.440+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec5b60 con 0x55b9ffebb020 2024-05-09T03:16:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.441+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(90 mds0_inotable [delete] v87'25 uv25 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.441+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec5f00 con 0x55b9ffebb020 2024-05-09T03:16:35.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.442+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(91 mds0_inotable [create] v87'26 uv26 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.443 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:35.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.442+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe6d0 con 0x55b9ffebb020 2024-05-09T03:16:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.443+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v87'27 uv27 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.443+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe6d0 con 0x55b9ffebb020 2024-05-09T03:16:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.444+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v87'28 uv28 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.444+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe6d0 con 0x55b9ffebb020 2024-05-09T03:16:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.445+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(94 100.00000000 [create] v87'29 uv23 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.445+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffebe6d0 con 0x55b9ffebb020 2024-05-09T03:16:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.446+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(95 100.00000000 [delete] v87'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.446+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffebe6d0 con 0x55b9ffebb020 2024-05-09T03:16:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.448+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(96 100.00000000 [create] v87'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.448 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:35.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.448+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec67f0 con 0x55b9ffebb020 2024-05-09T03:16:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.449+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v87'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.449+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec6360 con 0x55b9ffebb020 2024-05-09T03:16:35.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.450+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v87'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.450+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b9ffec7160 0x55b9ffec98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.450+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffec7160 2024-05-09T03:16:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.450+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b9ffec7160 0x55b9ffec98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.450+0000 7f0bf6857640 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b9ffec7160 0x55b9ffec98e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.451+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v87'14 uv12 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffec7160 2024-05-09T03:16:35.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.451+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffec7160 2024-05-09T03:16:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.452+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v87'15 uv15 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffec7160 2024-05-09T03:16:35.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.452+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffec7160 2024-05-09T03:16:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.453+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v87'16 uv16 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffec7160 2024-05-09T03:16:35.454 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:35.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.453+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffec7160 2024-05-09T03:16:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.455+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v87'17 uv17 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffec7160 2024-05-09T03:16:35.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.455+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebb020 2024-05-09T03:16:35.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.456+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v87'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.456+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebb020 2024-05-09T03:16:35.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.459+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v87'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.459+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebb020 2024-05-09T03:16:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.461+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v87'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.462 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.461+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffecae30 con 0x55b9ffebb020 2024-05-09T03:16:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.464+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v87'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.464+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffecae30 con 0x55b9ffebb020 2024-05-09T03:16:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.467+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v87'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.467+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebb020 2024-05-09T03:16:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.470+0000 7f0bf6857640 1 -- 172.21.15.46:0/1402149053 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bec0a0c90 con 0x55b9ffebb020 2024-05-09T03:16:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.470+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.471+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v87'12 uv11 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.471+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.472+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v87'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.472+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.474+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v87'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.474 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.474+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.475+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v87'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.475+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.476+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v87'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.476+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.478+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v87'33 uv24 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.478+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.480+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v87'34 uv34 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.480+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.485 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.485+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v87'35 uv35 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.485+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.486+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v87'36 uv36 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.486+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.488+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v87'37 uv37 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.488+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.491+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v87'38 uv31 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.491+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.493+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v87'39 uv39 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.493+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.496 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:35.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.495+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v87'40 uv40 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.496+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffecae30 con 0x55b9ffebf1d0 2024-05-09T03:16:35.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.498+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v87'41 uv41 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.499+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffecae30 con 0x55b9ffebf1d0 2024-05-09T03:16:35.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.501+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v87'42 uv42 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.501+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.502+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v87'43 uv43 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.502+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55b9ffec9e20 con 0x55b9ffebf1d0 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v87'44 uv44 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0bf0009600 con 0x55b9ffebf1d0 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b9ffd7df10 msgr2=0x55b9ffeb4820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b9ffd7df10 0x55b9ffeb4820 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b9ffebb020 msgr2=0x55b9ffebd440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b9ffebb020 0x55b9ffebd440 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b9ffec7160 msgr2=0x55b9ffec98e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b9ffec7160 0x55b9ffec98e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b9ffeb6bf0 msgr2=0x55b9ffeb9010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 reap_dead start 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b9ffeb6bf0 0x55b9ffeb9010 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b9ffebf1d0 msgr2=0x55b9ffebf5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b9ffebf1d0 0x55b9ffebf5e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b9ffec2880 msgr2=0x55b9ffec2c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b9ffec2880 0x55b9ffec2c90 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0be4074ce0 msgr2=0x7f0be40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0be4074ce0 0x7f0be40771f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0be8005780 tx=0x7f0be8007040 comp rx=0 tx=0).stop 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03d90 msgr2=0x55b9ffe02a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 --2- 172.21.15.46:0/1402149053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b9ffe03d90 0x55b9ffe02a30 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0bec001e50 tx=0x7f0bec001270 comp rx=0 tx=0).stop 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bf7859640 1 -- 172.21.15.46:0/1402149053 reap_dead start 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 shutdown_connections 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.503+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 >> 172.21.15.46:0/1402149053 conn(0x55b9ffc5dac0 msgr2=0x55b9ffd67dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.504+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 shutdown_connections 2024-05-09T03:16:35.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.504+0000 7f0bfd590f40 1 -- 172.21.15.46:0/1402149053 wait complete. 2024-05-09T03:16:35.507 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000.inode 2024-05-09T03:16:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.602+0000 7f8dd7218f40 1 Processor -- start 2024-05-09T03:16:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.602+0000 7f8dd7218f40 1 -- start start 2024-05-09T03:16:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd7218f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdedbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd7218f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cdee3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd7218f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cdee3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd7218f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5618cde575f0 con 0x5618cdee2390 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd7218f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5618cddb4f50 con 0x5618cdee1a40 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd7218f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5618cde578d0 con 0x5618cde565f0 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cdee3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cdee3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51652/0 (socket says 172.21.15.46:51652) 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 -- 172.21.15.46:0/1997273133 learned_addr learned my addr 172.21.15.46:0/1997273133 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0071640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdedbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dcf870640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cdee3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 -- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 msgr2=0x5618cdedbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdedbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 -- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 msgr2=0x5618cdee3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cdee3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 -- 172.21.15.46:0/1997273133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618cde588b0 con 0x5618cdee2390 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0071640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdedbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dcf870640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cdee3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.603+0000 7f8dd0872640 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cdee3b50 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80015b0 tx=0x7f8dc8001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5705ead9aee4d0e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dce86e640 1 -- 172.21.15.46:0/1997273133 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003650 con 0x5618cdee2390 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dce86e640 1 -- 172.21.15.46:0/1997273133 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc80037f0 con 0x5618cdee2390 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dce86e640 1 -- 172.21.15.46:0/1997273133 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003bc0 con 0x5618cdee2390 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1997273133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 msgr2=0x5618cdee3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cdee3b50 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f8dc80015b0 tx=0x7f8dc8001a70 comp rx=0 tx=0).stop 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1997273133 shutdown_connections 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cdee3b50 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cdee3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1997273133 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdedbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1997273133 >> 172.21.15.46:0/1997273133 conn(0x5618cdd30ac0 msgr2=0x5618cde3b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1997273133 shutdown_connections 2024-05-09T03:16:35.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.604+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1997273133 wait complete. 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.605+0000 7f8dd7218f40 1 Processor -- start 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.605+0000 7f8dd7218f40 1 -- start start 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.606+0000 7f8dd7218f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdf76dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cde0b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cde0b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5618cde4e730 con 0x5618cdee2390 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5618cde4ef30 con 0x5618cdee1a40 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5618cde4f210 con 0x5618cde565f0 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cde0b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cde0b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51662/0 (socket says 172.21.15.46:51662) 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 -- 172.21.15.46:0/1418923641 learned_addr learned my addr 172.21.15.46:0/1418923641 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0071640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdf76dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dcf870640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cde0b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 -- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 msgr2=0x5618cdf76dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5618cde565f0 0x5618cdf76dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 -- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 msgr2=0x5618cde0b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cde0b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 -- 172.21.15.46:0/1418923641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618cde0df90 con 0x5618cdee2390 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dcf870640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5618cdee1a40 0x5618cde0b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd0872640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cde0b5a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8000f80 tx=0x7f8dc8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dcd86c640 1 -- 172.21.15.46:0/1418923641 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8024070 con 0x5618cdee2390 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dcd86c640 1 -- 172.21.15.46:0/1418923641 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc8019040 con 0x5618cdee2390 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5618cde0e270 con 0x5618cdee2390 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dcd86c640 1 -- 172.21.15.46:0/1418923641 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003e60 con 0x5618cdee2390 2024-05-09T03:16:35.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.607+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618cdedf3a0 con 0x5618cdee2390 2024-05-09T03:16:35.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.609+0000 7f8dcd86c640 1 -- 172.21.15.46:0/1418923641 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8dc8010070 con 0x5618cdee2390 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dcd86c640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dbc074cb0 0x7f8dbc0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dcd86c640 1 -- 172.21.15.46:0/1418923641 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f8dc8097150 con 0x5618cdee2390 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dd0071640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dbc074cb0 0x7f8dbc0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5618cde0f710 0x5618cded9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e87) -- 0x5618cde4fd60 con 0x5618cde0f710 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dcf870640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5618cde0f710 0x5618cded9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.610+0000 7f8dd0071640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dbc074cb0 0x7f8dbc0771c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0058000 tx=0x7f8dc000a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.611+0000 7f8dcf870640 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5618cde0f710 0x5618cded9300 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dcf870640 1 -- 172.21.15.46:0/1418923641 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 1.00000000.inode [delete] v87'88 uv88 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8dc4009810 con 0x5618cde0f710 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5618cde0f710 msgr2=0x5618cded9300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5618cde0f710 0x5618cded9300 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dbc074cb0 msgr2=0x7f8dbc0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8dbc074cb0 0x7f8dbc0771c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0058000 tx=0x7f8dc000a000 comp rx=0 tx=0).stop 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 msgr2=0x5618cde0b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 --2- 172.21.15.46:0/1418923641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5618cdee2390 0x5618cde0b5a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8000f80 tx=0x7f8dc8001250 comp rx=0 tx=0).stop 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd0872640 1 -- 172.21.15.46:0/1418923641 reap_dead start 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 shutdown_connections 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 >> 172.21.15.46:0/1418923641 conn(0x5618cdd30ac0 msgr2=0x5618cdee0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 shutdown_connections 2024-05-09T03:16:35.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.612+0000 7f8dd7218f40 1 -- 172.21.15.46:0/1418923641 wait complete. 2024-05-09T03:16:35.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:35.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.732+0000 7f0fe1a3b640 1 Processor -- start 2024-05-09T03:16:35.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.732+0000 7f0fe1a3b640 1 -- start start 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fe1a3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fdc0d2e90 0x7f0fdc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fe1a3b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0fdc0b4fc0 0x7f0fdc0b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fe1a3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fdc0d3400 con 0x7f0fdc1605b0 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fe1a3b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0fdc0d4a60 con 0x7f0fdc0d2e90 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fe1a3b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0fdc0d4d40 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fdc0d2e90 0x7f0fdc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fdc0d2e90 0x7f0fdc0b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51672/0 (socket says 172.21.15.46:51672) 2024-05-09T03:16:35.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 -- 172.21.15.46:0/180505948 learned_addr learned my addr 172.21.15.46:0/180505948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdb7fe640 1 --2- 172.21.15.46:0/180505948 >> v2:172.21.15.88:3300/0 conn(0x7f0fdc0b4fc0 0x7f0fdc0b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 -- 172.21.15.46:0/180505948 >> v2:172.21.15.88:3300/0 conn(0x7f0fdc0b4fc0 msgr2=0x7f0fdc0b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 --2- 172.21.15.46:0/180505948 >> v2:172.21.15.88:3300/0 conn(0x7f0fdc0b4fc0 0x7f0fdc0b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 -- 172.21.15.46:0/180505948 >> v1:172.21.15.88:6789/0 conn(0x7f0fdc1605b0 legacy=0x7f0fdc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:35.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.734+0000 7f0fdaffd640 1 -- 172.21.15.46:0/180505948 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc0d5020 con 0x7f0fdc0d2e90 2024-05-09T03:16:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.735+0000 7f0fdaffd640 1 --2- 172.21.15.46:0/180505948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0d2e90 0x7f0fdc0b7760 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0c95f0 tx=0x7f0fd402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf53255eda2dcc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.735+0000 7f0fd9ffb640 1 -- 172.21.15.46:0/180505948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4036070 con 0x7f0fdc0d2e90 2024-05-09T03:16:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.735+0000 7f0fd9ffb640 1 -- 172.21.15.46:0/180505948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fd4031600 con 0x7f0fdc0d2e90 2024-05-09T03:16:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.735+0000 7f0fd9ffb640 1 -- 172.21.15.46:0/180505948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd40319d0 con 0x7f0fdc0d2e90 2024-05-09T03:16:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/180505948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0d2e90 msgr2=0x7f0fdc0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/180505948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0d2e90 0x7f0fdc0b7760 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc0c95f0 tx=0x7f0fd402a610 comp rx=0 tx=0).stop 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/180505948 shutdown_connections 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/180505948 >> v2:172.21.15.88:3300/0 conn(0x7f0fdc0b4fc0 0x7f0fdc0b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/180505948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0d2e90 0x7f0fdc0b7760 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/180505948 >> 172.21.15.46:0/180505948 conn(0x7f0fdc08d390 msgr2=0x7f0fdc08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/180505948 shutdown_connections 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.736+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/180505948 wait complete. 2024-05-09T03:16:35.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.738+0000 7f0fe1a3b640 1 Processor -- start 2024-05-09T03:16:35.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.738+0000 7f0fe1a3b640 1 -- start start 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fe1a3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 0x7f0fdc154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fe1a3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0fdc0d2e90 0x7f0fdc149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fe1a3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fdc1605b0 0x7f0fdc1616d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fe1a3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0d5540 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fe1a3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0fdc0b4c00 con 0x7f0fdc1605b0 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fe1a3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0fdc0cc5e0 con 0x7f0fdc0d2e90 2024-05-09T03:16:35.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.739+0000 7f0fdaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 0x7f0fdc154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 0x7f0fdc154910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51682/0 (socket says 172.21.15.46:51682) 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 -- 172.21.15.46:0/946195668 learned_addr learned my addr 172.21.15.46:0/946195668 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fda7fc640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0fdc0d2e90 0x7f0fdc149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdb7fe640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fdc1605b0 0x7f0fdc1616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 -- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0fdc0d2e90 msgr2=0x7f0fdc149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0fdc0d2e90 0x7f0fdc149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 -- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fdc1605b0 msgr2=0x7f0fdc1616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:35.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fdc1605b0 0x7f0fdc1616d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 -- 172.21.15.46:0/946195668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fdc1545d0 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdb7fe640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fdc1605b0 0x7f0fdc1616d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fdaffd640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 0x7f0fdc154910 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4034040 tx=0x7f0fd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4036050 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fd4042070 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fdc14a4e0 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4009510 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.740+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fdc1542f0 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.741+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fdc06a270 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.742+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0fd4007c50 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.742+0000 7f0fd3fff640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0fbc076300 0x7f0fbc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.743+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 6025+0+0 (secure 0 0 0) 0x7f0fd40b85b0 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.743+0000 7f0fda7fc640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0fbc076300 0x7f0fbc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:35.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.743+0000 7f0fda7fc640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0fbc076300 0x7f0fbc078810 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc14ad90 tx=0x7f0fc4009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:35.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.747+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0fd4081dc0 con 0x7f0fdc0b4fc0 2024-05-09T03:16:35.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:35.981+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0fdc0d1200 con 0x7f0fdc0b4fc0 2024-05-09T03:16:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.223+0000 7f0fd3fff640 1 -- 172.21.15.46:0/946195668 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v105) ==== 162+0+0 (secure 0 0 0) 0x7f0fd40816c0 con 0x7f0fdc0b4fc0 2024-05-09T03:16:36.223 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0fbc076300 msgr2=0x7f0fbc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0fbc076300 0x7f0fbc078810 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc14ad90 tx=0x7f0fc4009060 comp rx=0 tx=0).stop 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 msgr2=0x7f0fdc154910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 0x7f0fdc154910 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4034040 tx=0x7f0fd4004a40 comp rx=0 tx=0).stop 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 shutdown_connections 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fdc1605b0 0x7f0fdc1616d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0fdc0d2e90 0x7f0fdc149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0fbc076300 0x7f0fbc078810 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 --2- 172.21.15.46:0/946195668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fdc0b4fc0 0x7f0fdc154910 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 >> 172.21.15.46:0/946195668 conn(0x7f0fdc08d390 msgr2=0x7f0fdc15f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 shutdown_connections 2024-05-09T03:16:36.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.225+0000 7f0fe1a3b640 1 -- 172.21.15.46:0/946195668 wait complete. 2024-05-09T03:16:36.237 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:36.236+0000 7f0bd9d4b640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-05-09T03:16:36.237 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:36.236+0000 7f0bd9d4b640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:16:36.237 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:36.237+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:36.237 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:36.237+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:16:36.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:36.242 INFO:tasks.ceph.mds.c.smithi046.stderr: -16> 2024-05-09T03:16:36.236+0000 7f0bd9d4b640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-05-09T03:16:36.243 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:16:36.236+0000 7f0bd9d4b640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:16:36.297 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:36.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.357+0000 7f1dc9914640 1 Processor -- start 2024-05-09T03:16:36.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.357+0000 7f1dc9914640 1 -- start start 2024-05-09T03:16:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc9914640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc9914640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1dc40b26e0 0x7f1dc40d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc9914640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc415d590 con 0x7f1dc40d2f20 2024-05-09T03:16:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc9914640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1dc406eaf0 con 0x7f1dc40b26e0 2024-05-09T03:16:36.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc9914640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc40b2af0 con 0x7f1dc415d180 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc2ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1dc40b26e0 0x7f1dc40d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1dc40d2f20 0x7f1dc4160cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41770/0 (socket says 172.21.15.46:41770) 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.358+0000 7f1dc37fe640 1 -- 172.21.15.46:0/2617073585 learned_addr learned my addr 172.21.15.46:0/2617073585 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc1ffb640 1 -- 172.21.15.46:0/2617073585 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043262481 0 0) 0x7f1dbc001490 con 0x7f1dc40d2f20 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc1ffb640 1 -- 172.21.15.46:0/2617073585 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dac003520 con 0x7f1dc40d2f20 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 msgr2=0x7f1dc40b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc27fc640 1 --2- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2617073585 >> v1:172.21.15.88:6789/0 conn(0x7f1dc40d2f20 legacy=0x7f1dc4160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2617073585 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc406e1c0 con 0x7f1dc40b26e0 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.359+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc27fc640 1 --2- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc40d29e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1db80097a0 tx=0x7f1db802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc6c5c812aac7486 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc1ffb640 1 -- 172.21.15.46:0/2617073585 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8037070 con 0x7f1dc40b26e0 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc1ffb640 1 -- 172.21.15.46:0/2617073585 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db8032440 con 0x7f1dc40b26e0 2024-05-09T03:16:36.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc1ffb640 1 -- 172.21.15.46:0/2617073585 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db80327f0 con 0x7f1dc40b26e0 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc9914640 1 -- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 msgr2=0x7f1dc40d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc40d29e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1db80097a0 tx=0x7f1db802a180 comp rx=0 tx=0).stop 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc9914640 1 -- 172.21.15.46:0/2617073585 shutdown_connections 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc40d29e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2617073585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.360+0000 7f1dc9914640 1 -- 172.21.15.46:0/2617073585 >> 172.21.15.46:0/2617073585 conn(0x7f1dc409b7d0 msgr2=0x7f1dc409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.361+0000 7f1dc9914640 1 -- 172.21.15.46:0/2617073585 shutdown_connections 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.361+0000 7f1dc9914640 1 -- 172.21.15.46:0/2617073585 wait complete. 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.362+0000 7f1dc9914640 1 Processor -- start 2024-05-09T03:16:36.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.363+0000 7f1dc9914640 1 -- start start 2024-05-09T03:16:36.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc9914640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc414f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc9914640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 0x7f1dc41546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc9914640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc9914640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc406ef80 con 0x7f1dc40d2f20 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc9914640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc406e780 con 0x7f1dc40b26e0 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc9914640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc40d3330 con 0x7f1dc415d180 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 0x7f1dc41546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 0x7f1dc41546a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59794/0 (socket says 172.21.15.46:59794) 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2437865775 learned_addr learned my addr 172.21.15.46:0/2437865775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc414f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.364+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 msgr2=0x7f1dc40b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc27fc640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 msgr2=0x7f1dc414f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc27fc640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc414f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc27fc640 1 -- 172.21.15.46:0/2437865775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc4154be0 con 0x7f1dc40d2f20 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc414f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:36.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc27fc640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 0x7f1dc41546a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1db80097a0 tx=0x7f1db80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8037050 con 0x7f1dc40d2f20 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc4150370 con 0x7f1dc40d2f20 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db8043070 con 0x7f1dc40d2f20 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8006e10 con 0x7f1dc40d2f20 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.365+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc4152760 con 0x7f1dc40d2f20 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.368+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc416a0d0 con 0x7f1dc40d2f20 2024-05-09T03:16:36.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.368+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1db8032810 con 0x7f1dc40d2f20 2024-05-09T03:16:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.369+0000 7f1dabfff640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1db0076300 0x7f1db0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.369+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7f1db80b7f50 con 0x7f1dc40d2f20 2024-05-09T03:16:36.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.369+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1db0076300 0x7f1db0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.369+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1db0076300 0x7f1db0078810 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009500 tx=0x7f1db4009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:36.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.373+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1db804d020 con 0x7f1dc40d2f20 2024-05-09T03:16:36.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.603+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dc406a270 con 0x7f1dc40d2f20 2024-05-09T03:16:36.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.603+0000 7f1dabfff640 1 -- 172.21.15.46:0/2437865775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+4342 (secure 0 0 0) 0x7f1db8080fa0 con 0x7f1dc40d2f20 2024-05-09T03:16:36.604 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:36.604 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:36.237803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:36.604 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-05-09T03:16:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1db0076300 msgr2=0x7f1db0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1db0076300 0x7f1db0078810 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1db4009500 tx=0x7f1db4009110 comp rx=0 tx=0).stop 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 msgr2=0x7f1dc41546a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 0x7f1dc41546a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f1db80097a0 tx=0x7f1db80046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 shutdown_connections 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1dc415d180 0x7f1dc40b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1db0076300 0x7f1db0078810 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d2f20 0x7f1dc41546a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 --2- 172.21.15.46:0/2437865775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dc40b26e0 0x7f1dc414f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 >> 172.21.15.46:0/2437865775 conn(0x7f1dc409b7d0 msgr2=0x7f1dc4152d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.605+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 shutdown_connections 2024-05-09T03:16:36.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.606+0000 7f1dc9914640 1 -- 172.21.15.46:0/2437865775 wait complete. 2024-05-09T03:16:36.617 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:36.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.771+0000 7fd406972640 1 Processor -- start 2024-05-09T03:16:36.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.772+0000 7fd406972640 1 -- start start 2024-05-09T03:16:36.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd406972640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4000b4ff0 0x7fd400070670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd406972640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd400070bb0 0x7fd400073870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd406972640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd400070fc0 con 0x7fd4000b1210 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd406972640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd40015d980 con 0x7fd400070bb0 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd406972640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd40015dc60 con 0x7fd4000b4ff0 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd404f0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd400070bb0 0x7fd400073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd3fffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd4000b1210 0x7fd4000aeb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46526/0 (socket says 172.21.15.46:46526) 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd404f0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd400070bb0 0x7fd400073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59802/0 (socket says 172.21.15.46:59802) 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd3fffff640 1 -- 172.21.15.46:0/1110581711 learned_addr learned my addr 172.21.15.46:0/1110581711 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd3ff7fe640 1 --2- 172.21.15.46:0/1110581711 >> v2:172.21.15.88:3300/0 conn(0x7fd4000b4ff0 0x7fd400070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd404f0f640 1 -- 172.21.15.46:0/1110581711 >> v2:172.21.15.88:3300/0 conn(0x7fd4000b4ff0 msgr2=0x7fd400070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd404f0f640 1 --2- 172.21.15.46:0/1110581711 >> v2:172.21.15.88:3300/0 conn(0x7fd4000b4ff0 0x7fd400070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd404f0f640 1 -- 172.21.15.46:0/1110581711 >> v1:172.21.15.46:6789/0 conn(0x7fd4000b1210 legacy=0x7fd4000aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.773+0000 7fd404f0f640 1 -- 172.21.15.46:0/1110581711 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40015df40 con 0x7fd400070bb0 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.774+0000 7fd404f0f640 1 --2- 172.21.15.46:0/1110581711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400073870 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b4950 tx=0x7fd3f802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988d0fd047e0abcb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.774+0000 7fd3feffd640 1 -- 172.21.15.46:0/1110581711 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8007740 con 0x7fd400070bb0 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.774+0000 7fd3feffd640 1 -- 172.21.15.46:0/1110581711 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3f80078e0 con 0x7fd400070bb0 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.774+0000 7fd3feffd640 1 -- 172.21.15.46:0/1110581711 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f8007cb0 con 0x7fd400070bb0 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 -- 172.21.15.46:0/1110581711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 msgr2=0x7fd400073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 --2- 172.21.15.46:0/1110581711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400073870 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b4950 tx=0x7fd3f802a5f0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 -- 172.21.15.46:0/1110581711 shutdown_connections 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 --2- 172.21.15.46:0/1110581711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400073870 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 --2- 172.21.15.46:0/1110581711 >> v2:172.21.15.88:3300/0 conn(0x7fd4000b4ff0 0x7fd400070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 -- 172.21.15.46:0/1110581711 >> 172.21.15.46:0/1110581711 conn(0x7fd4000b3910 msgr2=0x7fd4000b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 -- 172.21.15.46:0/1110581711 shutdown_connections 2024-05-09T03:16:36.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.775+0000 7fd406972640 1 -- 172.21.15.46:0/1110581711 wait complete. 2024-05-09T03:16:36.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.777+0000 7fd406972640 1 Processor -- start 2024-05-09T03:16:36.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.777+0000 7fd406972640 1 -- start start 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4000b1210 0x7fd40015b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4000b4ff0 0x7fd40015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd40015d5a0 con 0x7fd400070bb0 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd4000b5400 con 0x7fd4000b4ff0 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd40015d720 con 0x7fd4000b1210 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59808/0 (socket says 172.21.15.46:59808) 2024-05-09T03:16:36.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 -- 172.21.15.46:0/2260990268 learned_addr learned my addr 172.21.15.46:0/2260990268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3ff7fe640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4000b1210 0x7fd40015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd404f0f640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4000b4ff0 0x7fd40015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 -- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4000b1210 msgr2=0x7fd40015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4000b1210 0x7fd40015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 -- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4000b4ff0 msgr2=0x7fd40015b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4000b4ff0 0x7fd40015b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 -- 172.21.15.46:0/2260990268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40014f5f0 con 0x7fd400070bb0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd404f0f640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4000b4ff0 0x7fd40015b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3ff7fe640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4000b1210 0x7fd40015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fffff640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400150c30 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000c3a0 tx=0x7fd3f000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0017070 con 0x7fd400070bb0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3f0005ce0 con 0x7fd400070bb0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.779+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd400151bc0 con 0x7fd400070bb0 2024-05-09T03:16:36.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.780+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f0007310 con 0x7fd400070bb0 2024-05-09T03:16:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.780+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd400157d00 con 0x7fd400070bb0 2024-05-09T03:16:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.780+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40006a270 con 0x7fd400070bb0 2024-05-09T03:16:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.781+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd3f001a070 con 0x7fd400070bb0 2024-05-09T03:16:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.782+0000 7fd3fcff9640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3ec076300 0x7fd3ec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.782+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7fd3f0096ad0 con 0x7fd400070bb0 2024-05-09T03:16:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.782+0000 7fd3ff7fe640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3ec076300 0x7fd3ec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.783+0000 7fd3ff7fe640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3ec076300 0x7fd3ec078810 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4006fd0 tx=0x7fd3f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:36.787+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd400070fc0 con 0x7fd400070bb0 2024-05-09T03:16:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.016+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd40015af10 con 0x7fd400070bb0 2024-05-09T03:16:37.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.016+0000 7fd3fcff9640 1 -- 172.21.15.46:0/2260990268 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+4342 (secure 0 0 0) 0x7fd3f00602a0 con 0x7fd400070bb0 2024-05-09T03:16:37.017 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:37.017 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":107,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:36.237803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:37.017 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 107 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3ec076300 msgr2=0x7fd3ec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3ec076300 0x7fd3ec078810 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4006fd0 tx=0x7fd3f4008040 comp rx=0 tx=0).stop 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 msgr2=0x7fd400150c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400150c30 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fd3f000c3a0 tx=0x7fd3f000c860 comp rx=0 tx=0).stop 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 shutdown_connections 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4000b4ff0 0x7fd40015b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd4000b1210 0x7fd40015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3ec076300 0x7fd3ec078810 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 --2- 172.21.15.46:0/2260990268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd400070bb0 0x7fd400150c30 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 >> 172.21.15.46:0/2260990268 conn(0x7fd4000b3910 msgr2=0x7fd4000b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 shutdown_connections 2024-05-09T03:16:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.018+0000 7fd406972640 1 -- 172.21.15.46:0/2260990268 wait complete. 2024-05-09T03:16:37.030 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:37.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:37.133 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:16:37.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:16:37.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.145+0000 7fe1a560c640 1 Processor -- start 2024-05-09T03:16:37.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.145+0000 7fe1a560c640 1 -- start start 2024-05-09T03:16:37.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe1a560c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe1a00d33a0 0x7fe1a00cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe1a560c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00d22b0 0x7fe1a0164000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe1a560c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a0160910 con 0x7fe1a00b4ee0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe1a560c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe1a00b3e40 con 0x7fe1a00d33a0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe1a560c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a00b4120 con 0x7fe1a00d22b0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe19e7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe1a00b4ee0 0x7fe1a00d1ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46530/0 (socket says 172.21.15.46:46530) 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe19e7fc640 1 -- 172.21.15.46:0/286888540 learned_addr learned my addr 172.21.15.46:0/286888540 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe19effd640 1 --2- 172.21.15.46:0/286888540 >> v2:172.21.15.88:3300/0 conn(0x7fe1a00d33a0 0x7fe1a00cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.146+0000 7fe19f7fe640 1 --2- 172.21.15.46:0/286888540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00d22b0 0x7fe1a0164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19dffb640 1 -- 172.21.15.46:0/286888540 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 921975826 0 0) 0x7fe1a0160910 con 0x7fe1a00b4ee0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19dffb640 1 -- 172.21.15.46:0/286888540 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe18c003520 con 0x7fe1a00b4ee0 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19f7fe640 1 -- 172.21.15.46:0/286888540 >> v2:172.21.15.88:3300/0 conn(0x7fe1a00d33a0 msgr2=0x7fe1a00cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19f7fe640 1 --2- 172.21.15.46:0/286888540 >> v2:172.21.15.88:3300/0 conn(0x7fe1a00d33a0 0x7fe1a00cef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19f7fe640 1 -- 172.21.15.46:0/286888540 >> v1:172.21.15.46:6789/0 conn(0x7fe1a00b4ee0 legacy=0x7fe1a00d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19f7fe640 1 -- 172.21.15.46:0/286888540 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a00b4400 con 0x7fe1a00d22b0 2024-05-09T03:16:37.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.147+0000 7fe19effd640 1 --2- 172.21.15.46:0/286888540 >> v2:172.21.15.88:3300/0 conn(0x7fe1a00d33a0 0x7fe1a00cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.148+0000 7fe19f7fe640 1 --2- 172.21.15.46:0/286888540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00d22b0 0x7fe1a0164000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe1980097a0 tx=0x7fe19802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea196eff2f2a54f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.148+0000 7fe19dffb640 1 -- 172.21.15.46:0/286888540 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198037070 con 0x7fe1a00d22b0 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.148+0000 7fe19dffb640 1 -- 172.21.15.46:0/286888540 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19802ae50 con 0x7fe1a00d22b0 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.148+0000 7fe19dffb640 1 -- 172.21.15.46:0/286888540 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198032660 con 0x7fe1a00d22b0 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 -- 172.21.15.46:0/286888540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00d22b0 msgr2=0x7fe1a0164000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 --2- 172.21.15.46:0/286888540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00d22b0 0x7fe1a0164000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe1980097a0 tx=0x7fe19802a100 comp rx=0 tx=0).stop 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 -- 172.21.15.46:0/286888540 shutdown_connections 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 --2- 172.21.15.46:0/286888540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00d22b0 0x7fe1a0164000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 --2- 172.21.15.46:0/286888540 >> v2:172.21.15.88:3300/0 conn(0x7fe1a00d33a0 0x7fe1a00cef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 -- 172.21.15.46:0/286888540 >> 172.21.15.46:0/286888540 conn(0x7fe1a009b7d0 msgr2=0x7fe1a009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 -- 172.21.15.46:0/286888540 shutdown_connections 2024-05-09T03:16:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.149+0000 7fe1a560c640 1 -- 172.21.15.46:0/286888540 wait complete. 2024-05-09T03:16:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.151+0000 7fe1a560c640 1 Processor -- start 2024-05-09T03:16:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.151+0000 7fe1a560c640 1 -- start start 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe1a560c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00b4ee0 0x7fe1a0160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe1a560c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1a00d22b0 0x7fe1a0159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe1a560c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 0x7fe1a00cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe1a560c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a00b5470 con 0x7fe1a00d33a0 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe1a560c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a00b52f0 con 0x7fe1a00d22b0 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe1a560c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a00d26c0 con 0x7fe1a00b4ee0 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 0x7fe1a00cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 0x7fe1a00cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59810/0 (socket says 172.21.15.46:59810) 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 -- 172.21.15.46:0/2093874995 learned_addr learned my addr 172.21.15.46:0/2093874995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19e7fc640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1a00d22b0 0x7fe1a0159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19effd640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00b4ee0 0x7fe1a0160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 -- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00b4ee0 msgr2=0x7fe1a0160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00b4ee0 0x7fe1a0160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 -- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1a00d22b0 msgr2=0x7fe1a0159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1a00d22b0 0x7fe1a0159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.152+0000 7fe19f7fe640 1 -- 172.21.15.46:0/2093874995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a0161040 con 0x7fe1a00d33a0 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.153+0000 7fe19f7fe640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 0x7fe1a00cb740 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fe1980097a0 tx=0x7fe1980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.153+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198041070 con 0x7fe1a00d33a0 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.153+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19803c460 con 0x7fe1a00d33a0 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.153+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a01614c0 con 0x7fe1a00d33a0 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.153+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198037040 con 0x7fe1a00d33a0 2024-05-09T03:16:37.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.153+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a00caf40 con 0x7fe1a00d33a0 2024-05-09T03:16:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.154+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a01619e0 con 0x7fe1a00d33a0 2024-05-09T03:16:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.155+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe198044070 con 0x7fe1a00d33a0 2024-05-09T03:16:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.155+0000 7fe1977fe640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1800762d0 0x7fe1800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.155+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7fe1980b7d70 con 0x7fe1a00d33a0 2024-05-09T03:16:37.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.156+0000 7fe19effd640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1800762d0 0x7fe1800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.156+0000 7fe19effd640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1800762d0 0x7fe1800787e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe190009bd0 tx=0x7fe19000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:37.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.160+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe198080dc0 con 0x7fe1a00d33a0 2024-05-09T03:16:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.397+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1a006a270 con 0x7fe1a00d33a0 2024-05-09T03:16:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.398+0000 7fe1977fe640 1 -- 172.21.15.46:0/2093874995 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5165 (secure 0 0 0) 0x7fe1980850c0 con 0x7fe1a00d33a0 2024-05-09T03:16:37.398 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:37.399 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":108,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":7680,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/782052086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":782052086},{"type":"v1","addr":"172.21.15.46:6834","nonce":782052086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:36.237803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 108 2024-05-09T03:16:37.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.399+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1800762d0 msgr2=0x7fe1800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.399+0000 7fe1a560c640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1800762d0 0x7fe1800787e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe190009bd0 tx=0x7fe19000b040 comp rx=0 tx=0).stop 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 msgr2=0x7fe1a00cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 0x7fe1a00cb740 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7fe1980097a0 tx=0x7fe1980046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 shutdown_connections 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1800762d0 0x7fe1800787e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1a00d33a0 0x7fe1a00cb740 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1a00d22b0 0x7fe1a0159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 --2- 172.21.15.46:0/2093874995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1a00b4ee0 0x7fe1a0160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 >> 172.21.15.46:0/2093874995 conn(0x7fe1a009b7d0 msgr2=0x7fe1a00ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 shutdown_connections 2024-05-09T03:16:37.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.400+0000 7fe1a560c640 1 -- 172.21.15.46:0/2093874995 wait complete. 2024-05-09T03:16:37.412 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000.inode' led to DAMAGED state 2024-05-09T03:16:37.412 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000000' 2024-05-09T03:16:37.412 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:37.412 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:37.428 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:16:37.428 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:37.428 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:37.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:37.429 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:16:37.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.639+0000 7fa474e92640 1 Processor -- start 2024-05-09T03:16:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.640+0000 7fa474e92640 1 -- start start 2024-05-09T03:16:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa474e92640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4700d3730 0x7fa47006e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa474e92640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa470165090 0x7fa4700d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa474e92640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa47009d470 con 0x7fa47006e7e0 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa474e92640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa47009cfe0 con 0x7fa470165090 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa474e92640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa4700b94a0 con 0x7fa4700d3730 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa46ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa470165090 0x7fa4700d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa46dd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa47006e7e0 0x7fa470164980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46546/0 (socket says 172.21.15.46:46546) 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa46dd9b640 1 -- 172.21.15.46:0/1061235893 learned_addr learned my addr 172.21.15.46:0/1061235893 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.641+0000 7fa46e59c640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4700d3730 0x7fa47006e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46ed9d640 1 -- 172.21.15.46:0/1061235893 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4700d3730 msgr2=0x7fa47006e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46ed9d640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4700d3730 0x7fa47006e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46ed9d640 1 -- 172.21.15.46:0/1061235893 >> v1:172.21.15.46:6789/0 conn(0x7fa47006e7e0 legacy=0x7fa470164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46ed9d640 1 -- 172.21.15.46:0/1061235893 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4700d11b0 con 0x7fa470165090 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46e59c640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4700d3730 0x7fa47006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:37.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46ed9d640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa470165090 0x7fa4700d1f50 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fa460009d90 tx=0x7fa46002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab879b31f9b6c0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.642+0000 7fa46d59a640 1 -- 172.21.15.46:0/1061235893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460031600 con 0x7fa470165090 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa46d59a640 1 -- 172.21.15.46:0/1061235893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4600317a0 con 0x7fa470165090 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa46d59a640 1 -- 172.21.15.46:0/1061235893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460031b70 con 0x7fa470165090 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 -- 172.21.15.46:0/1061235893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa470165090 msgr2=0x7fa4700d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa470165090 0x7fa4700d1f50 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fa460009d90 tx=0x7fa46002a5d0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 -- 172.21.15.46:0/1061235893 shutdown_connections 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa470165090 0x7fa4700d1f50 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 --2- 172.21.15.46:0/1061235893 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa4700d3730 0x7fa47006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 -- 172.21.15.46:0/1061235893 >> 172.21.15.46:0/1061235893 conn(0x7fa4700d05f0 msgr2=0x7fa4700d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 -- 172.21.15.46:0/1061235893 shutdown_connections 2024-05-09T03:16:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.643+0000 7fa474e92640 1 -- 172.21.15.46:0/1061235893 wait complete. 2024-05-09T03:16:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.645+0000 7fa474e92640 1 Processor -- start 2024-05-09T03:16:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.645+0000 7fa474e92640 1 -- start start 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa474e92640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 0x7fa470162620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa474e92640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4700d3730 0x7fa4700b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa474e92640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa470165090 0x7fa4700ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa474e92640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4700cfbb0 con 0x7fa47006e7e0 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa474e92640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa4700b9160 con 0x7fa4700d3730 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa474e92640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa4700cfd30 con 0x7fa470165090 2024-05-09T03:16:37.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 0x7fa470162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 0x7fa470162620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59822/0 (socket says 172.21.15.46:59822) 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 -- 172.21.15.46:0/117576683 learned_addr learned my addr 172.21.15.46:0/117576683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46dd9b640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4700d3730 0x7fa4700b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46ed9d640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa470165090 0x7fa4700ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 -- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa470165090 msgr2=0x7fa4700ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa470165090 0x7fa4700ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 -- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4700d3730 msgr2=0x7fa4700b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4700d3730 0x7fa4700b4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 -- 172.21.15.46:0/117576683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4700c7b40 con 0x7fa47006e7e0 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46ed9d640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa470165090 0x7fa4700ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.647+0000 7fa46e59c640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 0x7fa470162620 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fa46800d7d0 tx=0x7fa46800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.648+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468004030 con 0x7fa47006e7e0 2024-05-09T03:16:37.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.648+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4680041d0 con 0x7fa47006e7e0 2024-05-09T03:16:37.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.648+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4700c7e20 con 0x7fa47006e7e0 2024-05-09T03:16:37.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.648+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4680096a0 con 0x7fa47006e7e0 2024-05-09T03:16:37.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.648+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4700c83e0 con 0x7fa47006e7e0 2024-05-09T03:16:37.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.649+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4700b65b0 con 0x7fa47006e7e0 2024-05-09T03:16:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.649+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa468009840 con 0x7fa47006e7e0 2024-05-09T03:16:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.650+0000 7fa466ffd640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa450076300 0x7fa450078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.650+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7fa468097a90 con 0x7fa47006e7e0 2024-05-09T03:16:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.650+0000 7fa46dd9b640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa450076300 0x7fa450078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:37.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.651+0000 7fa46dd9b640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa450076300 0x7fa450078810 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa4700b3540 tx=0x7fa458009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:37.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.655+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa4700b65b0 con 0x7fa47006e7e0 2024-05-09T03:16:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:37.884+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa4700c7660 con 0x7fa47006e7e0 2024-05-09T03:16:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.245+0000 7fa466ffd640 1 -- 172.21.15.46:0/117576683 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v109) ==== 161+0+0 (secure 0 0 0) 0x7fa4700c7660 con 0x7fa47006e7e0 2024-05-09T03:16:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:38.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.247+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa450076300 msgr2=0x7fa450078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:38.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.247+0000 7fa474e92640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa450076300 0x7fa450078810 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa4700b3540 tx=0x7fa458009040 comp rx=0 tx=0).stop 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.247+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 msgr2=0x7fa470162620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.247+0000 7fa474e92640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 0x7fa470162620 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fa46800d7d0 tx=0x7fa46800dc90 comp rx=0 tx=0).stop 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 shutdown_connections 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa470165090 0x7fa4700ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4700d3730 0x7fa4700b4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa450076300 0x7fa450078810 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 --2- 172.21.15.46:0/117576683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa47006e7e0 0x7fa470162620 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 >> 172.21.15.46:0/117576683 conn(0x7fa4700d05f0 msgr2=0x7fa4700b4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 shutdown_connections 2024-05-09T03:16:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.248+0000 7fa474e92640 1 -- 172.21.15.46:0/117576683 wait complete. 2024-05-09T03:16:38.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:38.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.377+0000 7fc6737ce640 1 Processor -- start 2024-05-09T03:16:38.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.377+0000 7fc6737ce640 1 -- start start 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc6737ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c0b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc6737ce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc66c0d1f70 0x7fc66c161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc6737ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc66c1616b0 con 0x7fc66c160c60 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc6737ce640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc66c0b58b0 con 0x7fc66c0d1f70 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc6737ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc66c0b4920 con 0x7fc66c143410 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc671d6b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc66c0d1f70 0x7fc66c161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc670d69640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc66c160c60 0x7fc66c0d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46560/0 (socket says 172.21.15.46:46560) 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc670d69640 1 -- 172.21.15.46:0/1327784420 learned_addr learned my addr 172.21.15.46:0/1327784420 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc67156a640 1 --2- 172.21.15.46:0/1327784420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c0b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:38.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc671d6b640 1 -- 172.21.15.46:0/1327784420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 msgr2=0x7fc66c0b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc671d6b640 1 --2- 172.21.15.46:0/1327784420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c0b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc671d6b640 1 -- 172.21.15.46:0/1327784420 >> v1:172.21.15.46:6789/0 conn(0x7fc66c160c60 legacy=0x7fc66c0d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc671d6b640 1 -- 172.21.15.46:0/1327784420 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66c161990 con 0x7fc66c0d1f70 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.379+0000 7fc66bfff640 1 -- 172.21.15.46:0/1327784420 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497838002 0 0) 0x7fc66c1616b0 con 0x7fc66c160c60 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.380+0000 7fc671d6b640 1 --2- 172.21.15.46:0/1327784420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c0d1f70 0x7fc66c161070 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009d90 tx=0x7fc65c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e35ca54e21cd5e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.380+0000 7fc66bfff640 1 -- 172.21.15.46:0/1327784420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c007740 con 0x7fc66c0d1f70 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.380+0000 7fc66bfff640 1 -- 172.21.15.46:0/1327784420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65c0078e0 con 0x7fc66c0d1f70 2024-05-09T03:16:38.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.380+0000 7fc66bfff640 1 -- 172.21.15.46:0/1327784420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c007cb0 con 0x7fc66c0d1f70 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 -- 172.21.15.46:0/1327784420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c0d1f70 msgr2=0x7fc66c161070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1327784420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c0d1f70 0x7fc66c161070 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009d90 tx=0x7fc65c02a5f0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 -- 172.21.15.46:0/1327784420 shutdown_connections 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1327784420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c0d1f70 0x7fc66c161070 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1327784420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c0b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 -- 172.21.15.46:0/1327784420 >> 172.21.15.46:0/1327784420 conn(0x7fc66c0b3620 msgr2=0x7fc66c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 -- 172.21.15.46:0/1327784420 shutdown_connections 2024-05-09T03:16:38.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.381+0000 7fc6737ce640 1 -- 172.21.15.46:0/1327784420 wait complete. 2024-05-09T03:16:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.383+0000 7fc6737ce640 1 Processor -- start 2024-05-09T03:16:38.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.383+0000 7fc6737ce640 1 -- start start 2024-05-09T03:16:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc6737ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c13b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:38.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc6737ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 0x7fc66c154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc6737ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc66c1553c0 0x7fc66c1457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc6737ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c143820 con 0x7fc66c160c60 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc6737ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc66c0b4ee0 con 0x7fc66c1553c0 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc6737ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc66c0d1210 con 0x7fc66c143410 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc670d69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 0x7fc66c154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc670d69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 0x7fc66c154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59830/0 (socket says 172.21.15.46:59830) 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.384+0000 7fc670d69640 1 -- 172.21.15.46:0/1983398943 learned_addr learned my addr 172.21.15.46:0/1983398943 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc671d6b640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc66c1553c0 0x7fc66c1457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc67156a640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c13b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc670d69640 1 -- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 msgr2=0x7fc66c13b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc670d69640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c13b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc670d69640 1 -- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc66c1553c0 msgr2=0x7fc66c1457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc670d69640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc66c1553c0 0x7fc66c1457d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc670d69640 1 -- 172.21.15.46:0/1983398943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66c13c170 con 0x7fc66c160c60 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc67156a640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c13b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc670d69640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 0x7fc66c154e80 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc66000c3c0 tx=0x7fc66000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6600074e0 con 0x7fc66c160c60 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc660007680 con 0x7fc66c160c60 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66c13be90 con 0x7fc66c160c60 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660004a00 con 0x7fc66c160c60 2024-05-09T03:16:38.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.385+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66c144fd0 con 0x7fc66c160c60 2024-05-09T03:16:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.386+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66c06a270 con 0x7fc66c160c60 2024-05-09T03:16:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.387+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc660011070 con 0x7fc66c160c60 2024-05-09T03:16:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.387+0000 7fc669ffb640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc658076300 0x7fc658078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.388+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7fc660097330 con 0x7fc66c160c60 2024-05-09T03:16:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.388+0000 7fc67156a640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc658076300 0x7fc658078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:38.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.388+0000 7fc67156a640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc658076300 0x7fc658078810 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc664009a20 tx=0x7fc664008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:38.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.392+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc660060a70 con 0x7fc66c160c60 2024-05-09T03:16:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:38.659+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc66c0d0620 con 0x7fc66c160c60 2024-05-09T03:16:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.250+0000 7fc669ffb640 1 -- 172.21.15.46:0/1983398943 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v110) ==== 101+0+0 (secure 0 0 0) 0x7fc660060370 con 0x7fc66c160c60 2024-05-09T03:16:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc658076300 msgr2=0x7fc658078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc658076300 0x7fc658078810 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc664009a20 tx=0x7fc664008040 comp rx=0 tx=0).stop 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 msgr2=0x7fc66c154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 0x7fc66c154e80 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc66000c3c0 tx=0x7fc66000c880 comp rx=0 tx=0).stop 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 shutdown_connections 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc66c1553c0 0x7fc66c1457d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc658076300 0x7fc658078810 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc66c160c60 0x7fc66c154e80 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 --2- 172.21.15.46:0/1983398943 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc66c143410 0x7fc66c13b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 >> 172.21.15.46:0/1983398943 conn(0x7fc66c0b3620 msgr2=0x7fc66c0d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 shutdown_connections 2024-05-09T03:16:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.253+0000 7fc6737ce640 1 -- 172.21.15.46:0/1983398943 wait complete. 2024-05-09T03:16:39.267 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.325+0000 7fc82c3fef40 1 Processor -- start 2024-05-09T03:16:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.325+0000 7fc82c3fef40 1 -- start start 2024-05-09T03:16:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc82c3fef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5627e0 0x56220f638d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc82c3fef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f634c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc82c3fef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 0x56220f63d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc82c3fef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56220f5a94d0 con 0x56220f5ab010 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc82c3fef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56220f50bf50 con 0x56220f5947b0 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc82c3fef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56220f5ae8a0 con 0x56220f5627e0 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc8266c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 0x56220f63d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc8266c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 0x56220f63d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59846/0 (socket says 172.21.15.46:59846) 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.326+0000 7fc8266c7640 1 -- 172.21.15.46:0/3861393812 learned_addr learned my addr 172.21.15.46:0/3861393812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8256c5640 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f634c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc825ec6640 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5627e0 0x56220f638d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8266c7640 1 -- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5627e0 msgr2=0x56220f638d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8266c7640 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5627e0 0x56220f638d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8266c7640 1 -- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 msgr2=0x56220f634c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8266c7640 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f634c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8266c7640 1 -- 172.21.15.46:0/3861393812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220f63e490 con 0x56220f5ab010 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8256c5640 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f634c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc8266c7640 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 0x56220f63d710 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc82000a4b0 tx=0x7fc82000a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e27d688fa4d755e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc817fff640 1 -- 172.21.15.46:0/3861393812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820015070 con 0x56220f5ab010 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc817fff640 1 -- 172.21.15.46:0/3861393812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc82000e730 con 0x56220f5ab010 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.327+0000 7fc817fff640 1 -- 172.21.15.46:0/3861393812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82000eb20 con 0x56220f5ab010 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 -- 172.21.15.46:0/3861393812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 msgr2=0x56220f63d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 0x56220f63d710 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc82000a4b0 tx=0x7fc82000a970 comp rx=0 tx=0).stop 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 -- 172.21.15.46:0/3861393812 shutdown_connections 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5ab010 0x56220f63d710 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f634c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/3861393812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5627e0 0x56220f638d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 -- 172.21.15.46:0/3861393812 >> 172.21.15.46:0/3861393812 conn(0x56220f487ac0 msgr2=0x56220f5aca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 -- 172.21.15.46:0/3861393812 shutdown_connections 2024-05-09T03:16:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.328+0000 7fc82c3fef40 1 -- 172.21.15.46:0/3861393812 wait complete. 2024-05-09T03:16:39.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.329+0000 7fc82c3fef40 1 Processor -- start 2024-05-09T03:16:39.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.329+0000 7fc82c3fef40 1 -- start start 2024-05-09T03:16:39.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc82c3fef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 0x56220f63b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc82c3fef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f63d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc82c3fef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5ab010 0x56220f639900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc82c3fef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56220f593960 con 0x56220f5627e0 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc82c3fef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56220f593680 con 0x56220f5947b0 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc82c3fef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56220f593c40 con 0x56220f5ab010 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc825ec6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 0x56220f63b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc825ec6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 0x56220f63b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59852/0 (socket says 172.21.15.46:59852) 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 learned_addr learned my addr 172.21.15.46:0/748360278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f63d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.330+0000 7fc8266c7640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5ab010 0x56220f639900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5ab010 msgr2=0x56220f639900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56220f5ab010 0x56220f639900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 msgr2=0x56220f63d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56220f5947b0 0x56220f63d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220f62bc40 con 0x56220f5627e0 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 0x56220f63b320 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc818025da0 tx=0x7fc818002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc816ffd640 1 -- 172.21.15.46:0/748360278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81806c070 con 0x56220f5627e0 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc816ffd640 1 -- 172.21.15.46:0/748360278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc818052890 con 0x56220f5627e0 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc816ffd640 1 -- 172.21.15.46:0/748360278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc818052c60 con 0x56220f5627e0 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56220f62bf20 con 0x56220f5627e0 2024-05-09T03:16:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.331+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56220f5a60e0 con 0x56220f5627e0 2024-05-09T03:16:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.333+0000 7fc816ffd640 1 -- 172.21.15.46:0/748360278 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc81806f070 con 0x56220f5627e0 2024-05-09T03:16:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.333+0000 7fc816ffd640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc808074cb0 0x7fc8080771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.333+0000 7fc816ffd640 1 -- 172.21.15.46:0/748360278 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7fc8180da0d0 con 0x56220f5627e0 2024-05-09T03:16:39.335 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.333+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc808074cb0 0x7fc8080771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.334+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56220f63de40 0x56220f5a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.334+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f5a6e20 con 0x56220f63de40 2024-05-09T03:16:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.334+0000 7fc8266c7640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56220f63de40 0x56220f5a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.334+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc808074cb0 0x7fc8080771c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x56220f63ec70 tx=0x7fc81c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.334+0000 7fc8266c7640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56220f63de40 0x56220f5a68e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.336+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v88'102 uv78 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.336+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f5a71c0 con 0x56220f63de40 2024-05-09T03:16:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.337+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v88'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.337+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f5a7560 con 0x56220f63de40 2024-05-09T03:16:39.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.338+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v88'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.339 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:39.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.339+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6dfe00 con 0x56220f63de40 2024-05-09T03:16:39.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.340+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v88'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.340+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f639e40 con 0x56220f63de40 2024-05-09T03:16:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.341+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v88'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.341+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6dedf0 con 0x56220f63de40 2024-05-09T03:16:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.343+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v88'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.343+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df1e0 con 0x56220f63de40 2024-05-09T03:16:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.344+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v88'108 uv84 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.344+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6df5d0 con 0x56220f63de40 2024-05-09T03:16:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.345+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v88'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.345+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df5d0 con 0x56220f63de40 2024-05-09T03:16:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.347+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v88'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.420 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.347+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.348+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v88'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.348+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.350+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v88'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.350+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.351+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v88'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.351+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.352+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v88'114 uv91 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.353+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.354+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v88'115 uv115 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.354+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.355+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v88'116 uv116 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.424 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.355+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.357+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v88'117 uv117 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.357+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.358+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v88'118 uv118 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.358+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df580 con 0x56220f63de40 2024-05-09T03:16:39.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.360+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v88'119 uv119 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.360+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df940 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.361+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v88'120 uv120 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.361+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df940 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.362+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v88'121 uv95 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.362+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6df940 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.364+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v88'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.364+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df940 con 0x56220f63de40 2024-05-09T03:16:39.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.365+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v88'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.430 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.365+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df940 con 0x56220f63de40 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.366+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v88'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.366+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.367+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v88'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.368+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.369+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v88'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.369+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.370+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v88'127 uv101 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.370+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.371+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v88'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.372+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.373+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v88'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.433 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:39.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.373+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.375+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v88'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.376+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.377+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v88'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.377+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6df7f0 con 0x56220f63de40 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.378+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v88'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.378+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56220f6df960 0x56220f6e26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.378+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e2c00 con 0x56220f6df960 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.378+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56220f6df960 0x56220f6e26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.379+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56220f6df960 0x56220f6e26c0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.380+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v88'89 uv89 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.437 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.380+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e3450 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.381+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v88'90 uv90 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.381+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e37f0 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.382+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v88'91 uv91 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.382+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e3b90 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.383+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 200.00000000 [create] v88'92 uv76 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.383+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6e3f30 con 0x56220f6df960 2024-05-09T03:16:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.384+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 200.00000000 [delete] v88'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.385+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e42d0 con 0x56220f6df960 2024-05-09T03:16:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.386+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v88'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.440 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.386+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.387+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v88'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.387+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.388+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v88'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.388+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.389+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000001 [create] v88'97 uv81 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.389+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.390+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000001 [delete] v88'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.390+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.391+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v88'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.443 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.391+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e80d0 con 0x56220f6df960 2024-05-09T03:16:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.392+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [write 0~14895] v88'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.392+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e80d0 con 0x56220f6df960 2024-05-09T03:16:39.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.394+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v88'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.394+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.395+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 606.00000000 [create] v88'102 uv87 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.395+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.396+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 606.00000000 [delete] v88'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.396+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.397+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v88'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.447 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.397+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e5680 con 0x56220f6df960 2024-05-09T03:16:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.398+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v88'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.398+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e5680 con 0x56220f6df960 2024-05-09T03:16:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.399+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v88'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.399+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f6df960 2024-05-09T03:16:39.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.400+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v88'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6df960 2024-05-09T03:16:39.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.400+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f63de40 2024-05-09T03:16:39.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.401+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(51 607.00000000 [create] v88'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.401+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f63de40 2024-05-09T03:16:39.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.402+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(52 607.00000000 [delete] v88'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.402+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f63de40 2024-05-09T03:16:39.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.403+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(53 607.00000000 [create] v88'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.452 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.403+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f63de40 2024-05-09T03:16:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.404+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v88'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.404+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f63de40 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.405+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v88'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.405+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e4670 con 0x56220f63de40 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.406+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v88'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.406+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56220f6e4cf0 0x56220f6e5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.406+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e7a90 con 0x56220f6e4cf0 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.406+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56220f6e4cf0 0x56220f6e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.407+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56220f6e4cf0 0x56220f6e5190 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.408+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v88'19 uv18 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.408+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6e7e50 con 0x56220f6e4cf0 2024-05-09T03:16:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.409+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v88'20 uv20 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.409+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e8390 con 0x56220f6e4cf0 2024-05-09T03:16:39.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.410+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v88'21 uv21 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.459 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.410+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e8830 con 0x56220f6e4cf0 2024-05-09T03:16:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.411+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v88'22 uv22 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.411+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e8c60 con 0x56220f63de40 2024-05-09T03:16:39.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.412+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(61 608.00000000 [create] v88'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.412+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6e8c60 con 0x56220f63de40 2024-05-09T03:16:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.414+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(62 608.00000000 [delete] v88'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.414+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e8c60 con 0x56220f63de40 2024-05-09T03:16:39.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.415+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(63 608.00000000 [create] v88'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.463 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.415+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e9c70 con 0x56220f63de40 2024-05-09T03:16:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.416+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v88'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.416+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e9c70 con 0x56220f63de40 2024-05-09T03:16:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.418+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v88'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.418+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6e8c60 con 0x56220f63de40 2024-05-09T03:16:39.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.419+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v88'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.419+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56220f6e9330 0x56220f6eb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.419+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ebdb0 con 0x56220f6e9330 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.419+0000 7fc8266c7640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56220f6e9330 0x56220f6eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.419+0000 7fc8266c7640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56220f6e9330 0x56220f6eb770 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.420+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(67 604.00000000 [create] v88'37 uv31 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.420+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6ebe70 con 0x56220f6e9330 2024-05-09T03:16:39.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.422+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(68 604.00000000 [delete] v88'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.422+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.423+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(69 604.00000000 [create] v88'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.468 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.423+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ed2a0 con 0x56220f6e9330 2024-05-09T03:16:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.424+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v88'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.424+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ed2a0 con 0x56220f6e9330 2024-05-09T03:16:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.425+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v88'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.425+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.426+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v88'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.426+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.427+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(73 500.00000000 [create] v88'43 uv36 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.427+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.428+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(74 500.00000000 [delete] v88'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.428+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.429+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(75 500.00000000 [create] v88'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.473 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:39.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.429+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.431+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v88'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.431+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ec290 con 0x56220f6e9330 2024-05-09T03:16:39.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.432+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v88'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.432+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56220f6ec910 0x56220f6eed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.432+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef390 con 0x56220f6ec910 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.432+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56220f6ec910 0x56220f6eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.433+0000 7fc825ec6640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56220f6ec910 0x56220f6eed30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.434+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(78 mds_snaptable [create] v88'17 uv16 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6ec910 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.434+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f6ec910 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.435+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(79 mds_snaptable [delete] v88'18 uv18 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6ec910 2024-05-09T03:16:39.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.435+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f6ec910 2024-05-09T03:16:39.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.436+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(80 mds_snaptable [create] v88'19 uv19 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6ec910 2024-05-09T03:16:39.478 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:39.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.436+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f6ec910 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.437+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v88'20 uv20 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6ec910 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.437+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f6ec910 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.438+0000 7fc825ec6640 1 -- 172.21.15.46:0/748360278 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v88'21 uv21 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81806e030 con 0x56220f6ec910 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.438+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f63de40 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.439+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(83 605.00000000 [create] v88'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.439+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f63de40 2024-05-09T03:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.440+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(84 605.00000000 [delete] v88'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.440+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f63de40 2024-05-09T03:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.441+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(85 605.00000000 [create] v88'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.482 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.441+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0770 con 0x56220f63de40 2024-05-09T03:16:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.442+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v88'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.442+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0770 con 0x56220f63de40 2024-05-09T03:16:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.443+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v88'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.443+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f63de40 2024-05-09T03:16:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.444+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v88'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f63de40 2024-05-09T03:16:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.445+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6ef760 con 0x56220f6e4cf0 2024-05-09T03:16:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.446+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(89 mds0_inotable [create] v88'34 uv28 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.446+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6efb50 con 0x56220f6e4cf0 2024-05-09T03:16:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.447+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(90 mds0_inotable [delete] v88'35 uv35 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.447+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6eff40 con 0x56220f6e4cf0 2024-05-09T03:16:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.448+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(91 mds0_inotable [create] v88'36 uv36 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.487 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.448+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0330 con 0x56220f6e4cf0 2024-05-09T03:16:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.449+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v88'37 uv37 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.449+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0770 con 0x56220f6e4cf0 2024-05-09T03:16:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.450+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v88'38 uv38 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.450+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0330 con 0x56220f6e4cf0 2024-05-09T03:16:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.451+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(94 100.00000000 [create] v88'39 uv33 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.451+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6f0330 con 0x56220f6e4cf0 2024-05-09T03:16:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.452+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(95 100.00000000 [delete] v88'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.452+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0330 con 0x56220f6e4cf0 2024-05-09T03:16:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.453+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(96 100.00000000 [create] v88'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.492 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:39.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.453+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f02e0 con 0x56220f6e4cf0 2024-05-09T03:16:39.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.454+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v88'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.454+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f0790 con 0x56220f6e4cf0 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.455+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v88'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.455+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56220f6f1130 0x56220f6f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.455+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6f1130 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.455+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56220f6f1130 0x56220f6f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.455+0000 7fc8256c5640 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56220f6f1130 0x56220f6f38a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.457+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v88'18 uv17 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6f1130 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.457+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6f1130 2024-05-09T03:16:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.458+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v88'19 uv19 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6f1130 2024-05-09T03:16:39.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.458+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6f1130 2024-05-09T03:16:39.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.459+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v88'20 uv20 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6f1130 2024-05-09T03:16:39.496 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.459+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6f1130 2024-05-09T03:16:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.460+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v88'21 uv21 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6f1130 2024-05-09T03:16:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.460+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e4cf0 2024-05-09T03:16:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.461+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v88'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.461+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e4cf0 2024-05-09T03:16:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.462+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v88'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.462+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e4cf0 2024-05-09T03:16:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.463+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v88'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.499 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:39.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.463+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f4df0 con 0x56220f6e4cf0 2024-05-09T03:16:39.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.464+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v88'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.464+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f4df0 con 0x56220f6e4cf0 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.465+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v88'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.466+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e4cf0 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.467+0000 7fc8256c5640 1 -- 172.21.15.46:0/748360278 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v88'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc81c006980 con 0x56220f6e4cf0 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.467+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.468+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v88'17 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.468+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.469+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v88'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.469+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.503 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.470+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v88'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.471+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.472+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v88'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.472+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.473+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v88'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.474+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.475+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v88'45 uv37 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.475+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.476+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v88'46 uv46 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.476+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.477+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v88'47 uv47 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.507 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.477+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.478+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v88'48 uv48 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.478+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.479+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v88'49 uv49 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.479+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.480+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v88'50 uv44 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.480+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.482+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v88'51 uv51 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.482+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.482+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v88'52 uv52 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.511 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:39.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.483+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f4df0 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.484+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v88'53 uv53 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.484+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f4df0 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.485+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v88'54 uv54 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.485+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.486+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v88'55 uv55 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.486+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x56220f6f3de0 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v88'56 uv56 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc82000e070 con 0x56220f6e9330 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56220f63de40 msgr2=0x56220f5a68e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56220f63de40 0x56220f5a68e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56220f6e4cf0 msgr2=0x56220f6e5190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56220f6e4cf0 0x56220f6e5190 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56220f6f1130 msgr2=0x56220f6f38a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56220f6f1130 0x56220f6f38a0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56220f6df960 msgr2=0x56220f6e26c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56220f6df960 0x56220f6e26c0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56220f6e9330 msgr2=0x56220f6eb770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56220f6e9330 0x56220f6eb770 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56220f6ec910 msgr2=0x56220f6eed30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56220f6ec910 0x56220f6eed30 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 reap_dead start 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.487+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 reap_dead start 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc8266c7640 1 -- 172.21.15.46:0/748360278 reap_dead start 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc808074cb0 msgr2=0x7fc8080771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc808074cb0 0x7fc8080771c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x56220f63ec70 tx=0x7fc81c00b040 comp rx=0 tx=0).stop 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 msgr2=0x56220f63b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 0x56220f63b320 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc818025da0 tx=0x7fc818002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 shutdown_connections 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc808074cb0 0x7fc8080771c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 --2- 172.21.15.46:0/748360278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56220f5627e0 0x56220f63b320 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 >> 172.21.15.46:0/748360278 conn(0x56220f487ac0 msgr2=0x56220f638d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 shutdown_connections 2024-05-09T03:16:39.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.488+0000 7fc82c3fef40 1 -- 172.21.15.46:0/748360278 wait complete. 2024-05-09T03:16:39.518 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000000 2024-05-09T03:16:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.610+0000 7f2d5f40cf40 1 Processor -- start 2024-05-09T03:16:39.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.610+0000 7f2d5f40cf40 1 -- start start 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d5f40cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285dee3a0 0x55c285e798b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d5f40cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 0x55c285ddbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d5f40cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285e82150 0x55c285ddc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d5f40cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c285df75f0 con 0x55c285df65f0 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d5f40cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c285d54f50 con 0x55c285e82150 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d5f40cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c285df78d0 con 0x55c285dee3a0 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d57967640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 0x55c285ddbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d57967640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 0x55c285ddbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59870/0 (socket says 172.21.15.46:59870) 2024-05-09T03:16:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d57967640 1 -- 172.21.15.46:0/308794344 learned_addr learned my addr 172.21.15.46:0/308794344 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d58969640 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285e82150 0x55c285ddc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.611+0000 7f2d58168640 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285dee3a0 0x55c285e798b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d57967640 1 -- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285dee3a0 msgr2=0x55c285e798b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d57967640 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285dee3a0 0x55c285e798b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d57967640 1 -- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285e82150 msgr2=0x55c285ddc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d57967640 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285e82150 0x55c285ddc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d57967640 1 -- 172.21.15.46:0/308794344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c285df0fb0 con 0x55c285df65f0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d57967640 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 0x55c285ddbcb0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f2d40025bf0 tx=0x7f2d40051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b730469428152d86 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d56965640 1 -- 172.21.15.46:0/308794344 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40053930 con 0x55c285df65f0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d56965640 1 -- 172.21.15.46:0/308794344 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d40063070 con 0x55c285df65f0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d56965640 1 -- 172.21.15.46:0/308794344 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40053d10 con 0x55c285df65f0 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/308794344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 msgr2=0x55c285ddbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.612+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 0x55c285ddbcb0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f2d40025bf0 tx=0x7f2d40051000 comp rx=0 tx=0).stop 2024-05-09T03:16:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/308794344 shutdown_connections 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285e82150 0x55c285ddc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285df65f0 0x55c285ddbcb0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/308794344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285dee3a0 0x55c285e798b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/308794344 >> 172.21.15.46:0/308794344 conn(0x55c285cd0ac0 msgr2=0x55c285dd9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/308794344 shutdown_connections 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/308794344 wait complete. 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.613+0000 7f2d5f40cf40 1 Processor -- start 2024-05-09T03:16:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.614+0000 7f2d5f40cf40 1 -- start start 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d5f40cf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285dee3a0 0x55c285e78350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d5f40cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285df65f0 0x55c285e78af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d5f40cf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285e82150 0x55c285ddb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d5f40cf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c285e833d0 con 0x55c285dee3a0 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d5f40cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c285ddb7d0 con 0x55c285df65f0 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d5f40cf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c285e7dd20 con 0x55c285e82150 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285dee3a0 0x55c285e78350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285dee3a0 0x55c285e78350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59872/0 (socket says 172.21.15.46:59872) 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 -- 172.21.15.46:0/4117900543 learned_addr learned my addr 172.21.15.46:0/4117900543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d57967640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285df65f0 0x55c285e78af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58969640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285e82150 0x55c285ddb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 -- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285e82150 msgr2=0x55c285ddb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c285e82150 0x55c285ddb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 -- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285df65f0 msgr2=0x55c285e78af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c285df65f0 0x55c285e78af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 -- 172.21.15.46:0/4117900543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c285e80000 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.615+0000 7f2d58168640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285dee3a0 0x55c285e78350 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2d50002cf0 tx=0x7f2d5000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.616+0000 7f2d55963640 1 -- 172.21.15.46:0/4117900543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50015020 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.616+0000 7f2d55963640 1 -- 172.21.15.46:0/4117900543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d50011450 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.616+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c285e802e0 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.616+0000 7f2d55963640 1 -- 172.21.15.46:0/4117900543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50011820 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.616+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c285e808a0 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.617+0000 7f2d55963640 1 -- 172.21.15.46:0/4117900543 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d50018020 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d55963640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d38074cb0 0x7f2d380771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d55963640 1 -- 172.21.15.46:0/4117900543 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7f2d500964d0 con 0x55c285dee3a0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d57967640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d38074cb0 0x7f2d380771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c285e83fc0 0x55c285f12bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e88) -- 0x55c285e843d0 con 0x55c285e83fc0 2024-05-09T03:16:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d58969640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c285e83fc0 0x55c285f12bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.618+0000 7f2d57967640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d38074cb0 0x7f2d380771c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2d40067000 tx=0x7f2d40053000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.619+0000 7f2d58969640 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c285e83fc0 0x55c285f12bb0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d58969640 1 -- 172.21.15.46:0/4117900543 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000000 [delete] v88'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2d4800d960 con 0x55c285e83fc0 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c285e83fc0 msgr2=0x55c285f12bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55c285e83fc0 0x55c285f12bb0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d38074cb0 msgr2=0x7f2d380771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2d38074cb0 0x7f2d380771c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2d40067000 tx=0x7f2d40053000 comp rx=0 tx=0).stop 2024-05-09T03:16:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285dee3a0 msgr2=0x55c285e78350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 --2- 172.21.15.46:0/4117900543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c285dee3a0 0x55c285e78350 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2d50002cf0 tx=0x7f2d5000c620 comp rx=0 tx=0).stop 2024-05-09T03:16:39.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d58969640 1 -- 172.21.15.46:0/4117900543 reap_dead start 2024-05-09T03:16:39.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 shutdown_connections 2024-05-09T03:16:39.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 >> 172.21.15.46:0/4117900543 conn(0x55c285cd0ac0 msgr2=0x55c285e798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:39.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 shutdown_connections 2024-05-09T03:16:39.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.620+0000 7f2d5f40cf40 1 -- 172.21.15.46:0/4117900543 wait complete. 2024-05-09T03:16:39.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:39.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.740+0000 7fd32f76c640 1 Processor -- start 2024-05-09T03:16:39.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.740+0000 7fd32f76c640 1 -- start start 2024-05-09T03:16:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32f76c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3280b7e70 0x7fd3280b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32f76c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3281605b0 0x7fd3280b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32f76c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3280d3400 con 0x7fd3280d2e90 2024-05-09T03:16:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32f76c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd3280d4a60 con 0x7fd3280b7e70 2024-05-09T03:16:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32f76c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3280d4d40 con 0x7fd3281605b0 2024-05-09T03:16:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32cd07640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3280b7e70 0x7fd3280b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32d508640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3280d2e90 0x7fd3280b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46566/0 (socket says 172.21.15.46:46566) 2024-05-09T03:16:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32d508640 1 -- 172.21.15.46:0/563582228 learned_addr learned my addr 172.21.15.46:0/563582228 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32dd09640 1 --2- 172.21.15.46:0/563582228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3281605b0 0x7fd3280b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32cd07640 1 -- 172.21.15.46:0/563582228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3281605b0 msgr2=0x7fd3280b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32cd07640 1 --2- 172.21.15.46:0/563582228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3281605b0 0x7fd3280b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32cd07640 1 -- 172.21.15.46:0/563582228 >> v1:172.21.15.46:6789/0 conn(0x7fd3280d2e90 legacy=0x7fd3280b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd32cd07640 1 -- 172.21.15.46:0/563582228 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3280d5020 con 0x7fd3280b7e70 2024-05-09T03:16:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.742+0000 7fd327fff640 1 -- 172.21.15.46:0/563582228 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2301914612 0 0) 0x7fd3280d3400 con 0x7fd3280d2e90 2024-05-09T03:16:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.743+0000 7fd32cd07640 1 --2- 172.21.15.46:0/563582228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280b7e70 0x7fd3280b92a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0097a0 tx=0x7fd31c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262cc40f71454570 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.743+0000 7fd327fff640 1 -- 172.21.15.46:0/563582228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c037070 con 0x7fd3280b7e70 2024-05-09T03:16:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.743+0000 7fd327fff640 1 -- 172.21.15.46:0/563582228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31c032440 con 0x7fd3280b7e70 2024-05-09T03:16:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.743+0000 7fd327fff640 1 -- 172.21.15.46:0/563582228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c032810 con 0x7fd3280b7e70 2024-05-09T03:16:39.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 -- 172.21.15.46:0/563582228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280b7e70 msgr2=0x7fd3280b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 --2- 172.21.15.46:0/563582228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280b7e70 0x7fd3280b92a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0097a0 tx=0x7fd31c02a180 comp rx=0 tx=0).stop 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 -- 172.21.15.46:0/563582228 shutdown_connections 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 --2- 172.21.15.46:0/563582228 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3281605b0 0x7fd3280b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 --2- 172.21.15.46:0/563582228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280b7e70 0x7fd3280b92a0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 -- 172.21.15.46:0/563582228 >> 172.21.15.46:0/563582228 conn(0x7fd32808d390 msgr2=0x7fd32808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 -- 172.21.15.46:0/563582228 shutdown_connections 2024-05-09T03:16:39.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.744+0000 7fd32f76c640 1 -- 172.21.15.46:0/563582228 wait complete. 2024-05-09T03:16:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.746+0000 7fd32f76c640 1 Processor -- start 2024-05-09T03:16:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.746+0000 7fd32f76c640 1 -- start start 2024-05-09T03:16:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32f76c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 0x7fd3280cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32f76c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3281605b0 0x7fd3281f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32f76c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3280ca400 0x7fd328161a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32f76c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3280b4d80 con 0x7fd3280d2e90 2024-05-09T03:16:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32f76c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd3280b4c00 con 0x7fd3281605b0 2024-05-09T03:16:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32f76c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd3280b4fc0 con 0x7fd3280ca400 2024-05-09T03:16:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 0x7fd3280cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 0x7fd3280cf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59892/0 (socket says 172.21.15.46:59892) 2024-05-09T03:16:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 -- 172.21.15.46:0/2514460212 learned_addr learned my addr 172.21.15.46:0/2514460212 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32dd09640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3280ca400 0x7fd328161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32cd07640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3281605b0 0x7fd3281f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 -- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3280ca400 msgr2=0x7fd328161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3280ca400 0x7fd328161a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 -- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3281605b0 msgr2=0x7fd3281f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3281605b0 0x7fd3281f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32d508640 1 -- 172.21.15.46:0/2514460212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3280cffb0 con 0x7fd3280d2e90 2024-05-09T03:16:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32dd09640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3280ca400 0x7fd328161a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.747+0000 7fd32cd07640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3281605b0 0x7fd3281f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:39.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.748+0000 7fd32d508640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 0x7fd3280cf4f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd31800c730 tx=0x7fd31800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.748+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318018070 con 0x7fd3280d2e90 2024-05-09T03:16:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.748+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31800ce40 con 0x7fd3280d2e90 2024-05-09T03:16:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.748+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3280c9640 con 0x7fd3280d2e90 2024-05-09T03:16:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.748+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31800a880 con 0x7fd3280d2e90 2024-05-09T03:16:39.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.748+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3280c9c00 con 0x7fd3280d2e90 2024-05-09T03:16:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.749+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd32806a270 con 0x7fd3280d2e90 2024-05-09T03:16:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.750+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd31801b070 con 0x7fd3280d2e90 2024-05-09T03:16:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.750+0000 7fd325ffb640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3140762d0 0x7fd3140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.750+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 6111+0+0 (secure 0 0 0) 0x7fd318096de0 con 0x7fd3280d2e90 2024-05-09T03:16:39.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.750+0000 7fd32cd07640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3140762d0 0x7fd3140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:39.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.751+0000 7fd32cd07640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3140762d0 0x7fd3140787e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd31c034000 tx=0x7fd31c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:39.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.755+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd318060520 con 0x7fd3280d2e90 2024-05-09T03:16:39.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:39.986+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fd3280d11e0 con 0x7fd3280d2e90 2024-05-09T03:16:40.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.252+0000 7fd325ffb640 1 -- 172.21.15.46:0/2514460212 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v111) ==== 162+0+0 (secure 0 0 0) 0x7fd31805fe20 con 0x7fd3280d2e90 2024-05-09T03:16:40.252 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3140762d0 msgr2=0x7fd3140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3140762d0 0x7fd3140787e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd31c034000 tx=0x7fd31c02e000 comp rx=0 tx=0).stop 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 msgr2=0x7fd3280cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 0x7fd3280cf4f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd31800c730 tx=0x7fd31800cbf0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 shutdown_connections 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd3280ca400 0x7fd328161a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3281605b0 0x7fd3281f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd3140762d0 0x7fd3140787e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 --2- 172.21.15.46:0/2514460212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3280d2e90 0x7fd3280cf4f0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 >> 172.21.15.46:0/2514460212 conn(0x7fd32808d390 msgr2=0x7fd3280d38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 shutdown_connections 2024-05-09T03:16:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.254+0000 7fd32f76c640 1 -- 172.21.15.46:0/2514460212 wait complete. 2024-05-09T03:16:40.263 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:40.262+0000 7f8e1cfb5640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-05-09T03:16:40.263 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:40.263+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:40.263 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:40.263+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:16:40.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:40.269 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:16:40.262+0000 7f8e1cfb5640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-05-09T03:16:40.324 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:40.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.386+0000 7fe6be5a0640 1 Processor -- start 2024-05-09T03:16:40.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.386+0000 7fe6be5a0640 1 -- start start 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.387+0000 7fe6be5a0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6b80b0db0 0x7fe6b80b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6be5a0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 0x7fe6b815d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6be5a0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b806c680 con 0x7fe6b80b1a80 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6be5a0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe6b8157900 con 0x7fe6b80b0db0 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6be5a0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b806d290 con 0x7fe6b80b1e90 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 0x7fe6b815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 0x7fe6b815d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:60662/0 (socket says 172.21.15.46:60662) 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 -- 172.21.15.46:0/1542812268 learned_addr learned my addr 172.21.15.46:0/1542812268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:40.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bd59e640 1 --2- 172.21.15.46:0/1542812268 >> v2:172.21.15.88:3300/0 conn(0x7fe6b80b0db0 0x7fe6b80b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bcd9d640 1 --1- 172.21.15.46:0/1542812268 >> v1:172.21.15.88:6789/0 conn(0x7fe6b80b1a80 0x7fe6b806d720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41798/0 (socket says 172.21.15.46:41798) 2024-05-09T03:16:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 -- 172.21.15.46:0/1542812268 >> v2:172.21.15.88:3300/0 conn(0x7fe6b80b0db0 msgr2=0x7fe6b80b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 --2- 172.21.15.46:0/1542812268 >> v2:172.21.15.88:3300/0 conn(0x7fe6b80b0db0 0x7fe6b80b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 -- 172.21.15.46:0/1542812268 >> v1:172.21.15.88:6789/0 conn(0x7fe6b80b1a80 legacy=0x7fe6b806d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.388+0000 7fe6bdd9f640 1 -- 172.21.15.46:0/1542812268 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b806c960 con 0x7fe6b80b1e90 2024-05-09T03:16:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.389+0000 7fe6bdd9f640 1 --2- 172.21.15.46:0/1542812268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 0x7fe6b815d640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009e50 tx=0x7fe6b002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=578fd8a36b71aa2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.389+0000 7fe6affff640 1 -- 172.21.15.46:0/1542812268 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0036070 con 0x7fe6b80b1e90 2024-05-09T03:16:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.389+0000 7fe6affff640 1 -- 172.21.15.46:0/1542812268 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b0031600 con 0x7fe6b80b1e90 2024-05-09T03:16:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.389+0000 7fe6affff640 1 -- 172.21.15.46:0/1542812268 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00319d0 con 0x7fe6b80b1e90 2024-05-09T03:16:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 -- 172.21.15.46:0/1542812268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 msgr2=0x7fe6b815d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/1542812268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 0x7fe6b815d640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0009e50 tx=0x7fe6b002a710 comp rx=0 tx=0).stop 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 -- 172.21.15.46:0/1542812268 shutdown_connections 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/1542812268 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1e90 0x7fe6b815d640 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/1542812268 >> v2:172.21.15.88:3300/0 conn(0x7fe6b80b0db0 0x7fe6b80b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 -- 172.21.15.46:0/1542812268 >> 172.21.15.46:0/1542812268 conn(0x7fe6b8073cd0 msgr2=0x7fe6b80b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 -- 172.21.15.46:0/1542812268 shutdown_connections 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.390+0000 7fe6be5a0640 1 -- 172.21.15.46:0/1542812268 wait complete. 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.392+0000 7fe6be5a0640 1 Processor -- start 2024-05-09T03:16:40.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.392+0000 7fe6be5a0640 1 -- start start 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6be5a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 0x7fe6b81f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6be5a0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1a80 0x7fe6b809d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6be5a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6b80b1e90 0x7fe6b80d2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6be5a0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80b22a0 con 0x7fe6b80b0db0 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6be5a0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b806cf20 con 0x7fe6b80b1e90 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6be5a0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b8068a60 con 0x7fe6b80b1a80 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6bd59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 0x7fe6b81f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6bd59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 0x7fe6b81f5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59906/0 (socket says 172.21.15.46:59906) 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6bd59e640 1 -- 172.21.15.46:0/2872000206 learned_addr learned my addr 172.21.15.46:0/2872000206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6bcd9d640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1a80 0x7fe6b809d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.393+0000 7fe6bdd9f640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6b80b1e90 0x7fe6b80d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6bd59e640 1 -- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1a80 msgr2=0x7fe6b809d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6bd59e640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1a80 0x7fe6b809d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6bd59e640 1 -- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6b80b1e90 msgr2=0x7fe6b80d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6bd59e640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6b80b1e90 0x7fe6b80d2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6bd59e640 1 -- 172.21.15.46:0/2872000206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b809eed0 con 0x7fe6b80b0db0 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6bd59e640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 0x7fe6b81f5c10 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400e8a0 tx=0x7fe6b400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b4017070 con 0x7fe6b80b0db0 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b400ee90 con 0x7fe6b80b0db0 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b814e740 con 0x7fe6b80b0db0 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b400cb00 con 0x7fe6b80b0db0 2024-05-09T03:16:40.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.394+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b81f6980 con 0x7fe6b80b0db0 2024-05-09T03:16:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.395+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b8076c70 con 0x7fe6b80b0db0 2024-05-09T03:16:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.396+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe6b401a070 con 0x7fe6b80b0db0 2024-05-09T03:16:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.397+0000 7fe6adffb640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe69c076300 0x7fe69c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.397+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7fe6b4097980 con 0x7fe6b80b0db0 2024-05-09T03:16:40.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.397+0000 7fe6bcd9d640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe69c076300 0x7fe69c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.397+0000 7fe6bcd9d640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe69c076300 0x7fe69c078810 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80d3110 tx=0x7fe6a8000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:40.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.402+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe6b4060970 con 0x7fe6b80b0db0 2024-05-09T03:16:40.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.631+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6b80ce2f0 con 0x7fe6b80b0db0 2024-05-09T03:16:40.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.632+0000 7fe6adffb640 1 -- 172.21.15.46:0/2872000206 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+4342 (secure 0 0 0) 0x7fe6b4064c70 con 0x7fe6b80b0db0 2024-05-09T03:16:40.632 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:40.633 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":113,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:40.263889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:40.633 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 113 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe69c076300 msgr2=0x7fe69c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe69c076300 0x7fe69c078810 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80d3110 tx=0x7fe6a8000bc0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 msgr2=0x7fe6b81f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 0x7fe6b81f5c10 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400e8a0 tx=0x7fe6b400ed60 comp rx=0 tx=0).stop 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 shutdown_connections 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6b80b1e90 0x7fe6b80d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe6b80b1a80 0x7fe6b809d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe69c076300 0x7fe69c078810 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 --2- 172.21.15.46:0/2872000206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6b80b0db0 0x7fe6b81f5c10 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 >> 172.21.15.46:0/2872000206 conn(0x7fe6b8073cd0 msgr2=0x7fe6b80b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 shutdown_connections 2024-05-09T03:16:40.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.634+0000 7fe6be5a0640 1 -- 172.21.15.46:0/2872000206 wait complete. 2024-05-09T03:16:40.646 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:40.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:40.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.761+0000 7fc43f619640 1 Processor -- start 2024-05-09T03:16:40.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.761+0000 7fc43f619640 1 -- start start 2024-05-09T03:16:40.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.762+0000 7fc43f619640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4380d2b90 0x7fc43815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.762+0000 7fc43f619640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc438160b20 con 0x7fc438164c70 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.762+0000 7fc43f619640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4380d15f0 con 0x7fc4381615e0 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.762+0000 7fc43f619640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc43815e520 con 0x7fc4380d2b90 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.762+0000 7fc43dbb6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4380d2b90 0x7fc43815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.763+0000 7fc43d3b5640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc438164c70 0x7fc4380d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46570/0 (socket says 172.21.15.46:46570) 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.763+0000 7fc43d3b5640 1 -- 172.21.15.46:0/1763091684 learned_addr learned my addr 172.21.15.46:0/1763091684 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.763+0000 7fc43dbb6640 1 -- 172.21.15.46:0/1763091684 >> v1:172.21.15.88:6789/0 conn(0x7fc4381615e0 legacy=0x7fc4380d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.763+0000 7fc43dbb6640 1 -- 172.21.15.46:0/1763091684 >> v1:172.21.15.46:6789/0 conn(0x7fc438164c70 legacy=0x7fc4380d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.763+0000 7fc43dbb6640 1 -- 172.21.15.46:0/1763091684 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4380d18d0 con 0x7fc4380d2b90 2024-05-09T03:16:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.763+0000 7fc42ffff640 1 -- 172.21.15.46:0/1763091684 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3077449979 0 0) 0x7fc438160b20 con 0x7fc438164c70 2024-05-09T03:16:40.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc43dbb6640 1 --2- 172.21.15.46:0/1763091684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc43815dfe0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fc430009d90 tx=0x7fc43002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e5389b583c81f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc42ffff640 1 -- 172.21.15.46:0/1763091684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430007740 con 0x7fc4380d2b90 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc42ffff640 1 -- 172.21.15.46:0/1763091684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4300078e0 con 0x7fc4380d2b90 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc42ffff640 1 -- 172.21.15.46:0/1763091684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430007cb0 con 0x7fc4380d2b90 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc43f619640 1 -- 172.21.15.46:0/1763091684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 msgr2=0x7fc43815dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc43f619640 1 --2- 172.21.15.46:0/1763091684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc43815dfe0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fc430009d90 tx=0x7fc43002a660 comp rx=0 tx=0).stop 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc43f619640 1 -- 172.21.15.46:0/1763091684 shutdown_connections 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc43f619640 1 --2- 172.21.15.46:0/1763091684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc43815dfe0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.764+0000 7fc43f619640 1 -- 172.21.15.46:0/1763091684 >> 172.21.15.46:0/1763091684 conn(0x7fc438073db0 msgr2=0x7fc4380741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.765+0000 7fc43f619640 1 -- 172.21.15.46:0/1763091684 shutdown_connections 2024-05-09T03:16:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.765+0000 7fc43f619640 1 -- 172.21.15.46:0/1763091684 wait complete. 2024-05-09T03:16:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.766+0000 7fc43f619640 1 Processor -- start 2024-05-09T03:16:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.766+0000 7fc43f619640 1 -- start start 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43f619640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc4380ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43f619640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4381615e0 0x7fc4380cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43f619640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc438164c70 0x7fc4380c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43f619640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4381619f0 con 0x7fc4380d2b90 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43f619640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc438161150 con 0x7fc4381615e0 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43f619640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc4380d45c0 con 0x7fc438164c70 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc4380ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc4380ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59916/0 (socket says 172.21.15.46:59916) 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 -- 172.21.15.46:0/1038519233 learned_addr learned my addr 172.21.15.46:0/1038519233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43cbb4640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4381615e0 0x7fc4380cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43dbb6640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc438164c70 0x7fc4380c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 -- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc438164c70 msgr2=0x7fc4380c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc438164c70 0x7fc4380c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 -- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4381615e0 msgr2=0x7fc4380cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4381615e0 0x7fc4380cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 -- 172.21.15.46:0/1038519233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc438148f40 con 0x7fc4380d2b90 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43dbb6640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc438164c70 0x7fc4380c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc43d3b5640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc4380ccd50 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c3d0 tx=0x7fc43400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.768+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340074e0 con 0x7fc4380d2b90 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.769+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4340047e0 con 0x7fc4380d2b90 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.769+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc438149220 con 0x7fc4380d2b90 2024-05-09T03:16:40.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.769+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434004c10 con 0x7fc4380d2b90 2024-05-09T03:16:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.769+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4381497e0 con 0x7fc4380d2b90 2024-05-09T03:16:40.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.770+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc43808cfd0 con 0x7fc4380d2b90 2024-05-09T03:16:40.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.770+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc434011070 con 0x7fc4380d2b90 2024-05-09T03:16:40.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.771+0000 7fc42dffb640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc41c0762d0 0x7fc41c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:40.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.771+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7fc434097290 con 0x7fc4380d2b90 2024-05-09T03:16:40.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.771+0000 7fc43cbb4640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc41c0762d0 0x7fc41c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:40.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.771+0000 7fc43cbb4640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc41c0762d0 0x7fc41c0787e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc43815ba10 tx=0x7fc428009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:40.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:40.776+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc43808cfd0 con 0x7fc4380d2b90 2024-05-09T03:16:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.009+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc43806a270 con 0x7fc4380d2b90 2024-05-09T03:16:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.010+0000 7fc42dffb640 1 -- 172.21.15.46:0/1038519233 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+4342 (secure 0 0 0) 0x7fc434060a00 con 0x7fc4380d2b90 2024-05-09T03:16:41.010 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:41.011 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":113,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:40.263889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 113 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc41c0762d0 msgr2=0x7fc41c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc41c0762d0 0x7fc41c0787e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc43815ba10 tx=0x7fc428009040 comp rx=0 tx=0).stop 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 msgr2=0x7fc4380ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc4380ccd50 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fc43400c3d0 tx=0x7fc43400c890 comp rx=0 tx=0).stop 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 shutdown_connections 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc438164c70 0x7fc4380c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4381615e0 0x7fc4380cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc41c0762d0 0x7fc41c0787e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 --2- 172.21.15.46:0/1038519233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4380d2b90 0x7fc4380ccd50 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 >> 172.21.15.46:0/1038519233 conn(0x7fc438073db0 msgr2=0x7fc4380b9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 shutdown_connections 2024-05-09T03:16:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.012+0000 7fc43f619640 1 -- 172.21.15.46:0/1038519233 wait complete. 2024-05-09T03:16:41.024 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:41.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:41.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.140+0000 7fe2e5cd9640 1 Processor -- start 2024-05-09T03:16:41.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.140+0000 7fe2e5cd9640 1 -- start start 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2e5cd9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00d57a0 0x7fe2e0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2e5cd9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2e00b4af0 0x7fe2e00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2e5cd9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2e00b77e0 0x7fe2e00b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2e5cd9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e00d5bb0 con 0x7fe2e00b4af0 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2e5cd9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e00b8130 con 0x7fe2e00b77e0 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2e5cd9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e00d49e0 con 0x7fe2e00d57a0 2024-05-09T03:16:41.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2e00b4af0 0x7fe2e00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2e00b4af0 0x7fe2e00b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59918/0 (socket says 172.21.15.46:59918) 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 -- 172.21.15.46:0/2024142512 learned_addr learned my addr 172.21.15.46:0/2024142512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2df7fe640 1 --2- 172.21.15.46:0/2024142512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00d57a0 0x7fe2e0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2dffff640 1 --2- 172.21.15.46:0/2024142512 >> v2:172.21.15.88:3300/0 conn(0x7fe2e00b77e0 0x7fe2e00b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 -- 172.21.15.46:0/2024142512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00d57a0 msgr2=0x7fe2e0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 --2- 172.21.15.46:0/2024142512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00d57a0 0x7fe2e0160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 -- 172.21.15.46:0/2024142512 >> v2:172.21.15.88:3300/0 conn(0x7fe2e00b77e0 msgr2=0x7fe2e00b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 --2- 172.21.15.46:0/2024142512 >> v2:172.21.15.88:3300/0 conn(0x7fe2e00b77e0 0x7fe2e00b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2deffd640 1 -- 172.21.15.46:0/2024142512 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e00d4cc0 con 0x7fe2e00b4af0 2024-05-09T03:16:41.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.142+0000 7fe2dffff640 1 --2- 172.21.15.46:0/2024142512 >> v2:172.21.15.88:3300/0 conn(0x7fe2e00b77e0 0x7fe2e00b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:41.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.143+0000 7fe2deffd640 1 --2- 172.21.15.46:0/2024142512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00b4af0 0x7fe2e00b4f00 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fe2e008f200 tx=0x7fe2d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea7caf7995ea952 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:41.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.143+0000 7fe2de7fc640 1 -- 172.21.15.46:0/2024142512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4037070 con 0x7fe2e00b4af0 2024-05-09T03:16:41.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.143+0000 7fe2de7fc640 1 -- 172.21.15.46:0/2024142512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d4032440 con 0x7fe2e00b4af0 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.143+0000 7fe2de7fc640 1 -- 172.21.15.46:0/2024142512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4032810 con 0x7fe2e00b4af0 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/2024142512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00b4af0 msgr2=0x7fe2e00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/2024142512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00b4af0 0x7fe2e00b4f00 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fe2e008f200 tx=0x7fe2d402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/2024142512 shutdown_connections 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/2024142512 >> v2:172.21.15.88:3300/0 conn(0x7fe2e00b77e0 0x7fe2e00b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/2024142512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00b4af0 0x7fe2e00b4f00 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/2024142512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00d57a0 0x7fe2e0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/2024142512 >> 172.21.15.46:0/2024142512 conn(0x7fe2e00743e0 msgr2=0x7fe2e00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/2024142512 shutdown_connections 2024-05-09T03:16:41.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.144+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/2024142512 wait complete. 2024-05-09T03:16:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.146+0000 7fe2e5cd9640 1 Processor -- start 2024-05-09T03:16:41.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.146+0000 7fe2e5cd9640 1 -- start start 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2e5cd9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00b4af0 0x7fe2e00d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2e5cd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e00b77e0 0x7fe2e00c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2e5cd9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 0x7fe2e014bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2e5cd9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e00d15b0 con 0x7fe2e00d57a0 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2e5cd9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e00d5280 con 0x7fe2e00b77e0 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2e5cd9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e00d1920 con 0x7fe2e00b4af0 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 0x7fe2e014bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 0x7fe2e014bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59930/0 (socket says 172.21.15.46:59930) 2024-05-09T03:16:41.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2dffff640 1 -- 172.21.15.46:0/1415592576 learned_addr learned my addr 172.21.15.46:0/1415592576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2deffd640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e00b77e0 0x7fe2e00c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.147+0000 7fe2df7fe640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00b4af0 0x7fe2e00d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2dffff640 1 -- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00b4af0 msgr2=0x7fe2e00d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2dffff640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00b4af0 0x7fe2e00d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2dffff640 1 -- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e00b77e0 msgr2=0x7fe2e00c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2dffff640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e00b77e0 0x7fe2e00c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2dffff640 1 -- 172.21.15.46:0/1415592576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e01e8030 con 0x7fe2e00d57a0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2dffff640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 0x7fe2e014bca0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800ccc0 tx=0x7fe2d800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800f680 con 0x7fe2e00d57a0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d800f820 con 0x7fe2e00d57a0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e01e8310 con 0x7fe2e00d57a0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800fc10 con 0x7fe2e00d57a0 2024-05-09T03:16:41.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.148+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e01e88d0 con 0x7fe2e00d57a0 2024-05-09T03:16:41.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.149+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e00d4800 con 0x7fe2e00d57a0 2024-05-09T03:16:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.150+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe2d80213f0 con 0x7fe2e00d57a0 2024-05-09T03:16:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.150+0000 7fe2c7fff640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2cc076300 0x7fe2cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.151+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7fe2d8098310 con 0x7fe2e00d57a0 2024-05-09T03:16:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.151+0000 7fe2df7fe640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2cc076300 0x7fe2cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.151+0000 7fe2df7fe640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2cc076300 0x7fe2cc078810 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe2e006d3b0 tx=0x7fe2d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.155+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe2e00d4800 con 0x7fe2e00d57a0 2024-05-09T03:16:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.384+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2e006a270 con 0x7fe2e00d57a0 2024-05-09T03:16:41.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.385+0000 7fe2c7fff640 1 -- 172.21.15.46:0/1415592576 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5168 (secure 0 0 0) 0x7fe2d8014030 con 0x7fe2e00d57a0 2024-05-09T03:16:41.386 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:41.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":114,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2860475391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2860475391},{"type":"v1","addr":"172.21.15.46:6834","nonce":2860475391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:40.263889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:41.386 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 114 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2cc076300 msgr2=0x7fe2cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2cc076300 0x7fe2cc078810 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fe2e006d3b0 tx=0x7fe2d0008040 comp rx=0 tx=0).stop 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 msgr2=0x7fe2e014bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 0x7fe2e014bca0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800ccc0 tx=0x7fe2d800a650 comp rx=0 tx=0).stop 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 shutdown_connections 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2cc076300 0x7fe2cc078810 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e00d57a0 0x7fe2e014bca0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e00b77e0 0x7fe2e00c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 --2- 172.21.15.46:0/1415592576 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe2e00b4af0 0x7fe2e00d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 >> 172.21.15.46:0/1415592576 conn(0x7fe2e00743e0 msgr2=0x7fe2e00d3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 shutdown_connections 2024-05-09T03:16:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.387+0000 7fe2e5cd9640 1 -- 172.21.15.46:0/1415592576 wait complete. 2024-05-09T03:16:41.399 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 200.00000000' led to DAMAGED state 2024-05-09T03:16:41.399 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000001' 2024-05-09T03:16:41.399 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:41.399 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:41.416 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:16:41.416 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:41.416 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:41.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:41.416 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:16:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:16:41.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.628+0000 7f6c2f9b7640 1 Processor -- start 2024-05-09T03:16:41.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.629+0000 7f6c2f9b7640 1 -- start start 2024-05-09T03:16:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2f9b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c280b7e70 0x7f6c280b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2f9b7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c281605b0 0x7f6c280b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2f9b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c280d3400 con 0x7f6c280d2e90 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2f9b7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6c280d4a60 con 0x7f6c280b7e70 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2f9b7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6c280d4d40 con 0x7f6c281605b0 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2cf52640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c280b7e70 0x7f6c280b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2d753640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6c280d2e90 0x7f6c280b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46578/0 (socket says 172.21.15.46:46578) 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2d753640 1 -- 172.21.15.46:0/3181975723 learned_addr learned my addr 172.21.15.46:0/3181975723 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2df54640 1 --2- 172.21.15.46:0/3181975723 >> v2:172.21.15.88:3300/0 conn(0x7f6c281605b0 0x7f6c280b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2cf52640 1 -- 172.21.15.46:0/3181975723 >> v2:172.21.15.88:3300/0 conn(0x7f6c281605b0 msgr2=0x7f6c280b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2cf52640 1 --2- 172.21.15.46:0/3181975723 >> v2:172.21.15.88:3300/0 conn(0x7f6c281605b0 0x7f6c280b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.630+0000 7f6c2cf52640 1 -- 172.21.15.46:0/3181975723 >> v1:172.21.15.46:6789/0 conn(0x7f6c280d2e90 legacy=0x7f6c280b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c27fff640 1 -- 172.21.15.46:0/3181975723 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591943240 0 0) 0x7f6c280d3400 con 0x7f6c280d2e90 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c2cf52640 1 -- 172.21.15.46:0/3181975723 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c280d5020 con 0x7f6c280b7e70 2024-05-09T03:16:41.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c2df54640 1 --2- 172.21.15.46:0/3181975723 >> v2:172.21.15.88:3300/0 conn(0x7f6c281605b0 0x7f6c280b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c2cf52640 1 --2- 172.21.15.46:0/3181975723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280b7e70 0x7f6c280b92a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f6c280c95f0 tx=0x7f6c1c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294030bf458081d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c27fff640 1 -- 172.21.15.46:0/3181975723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c037070 con 0x7f6c280b7e70 2024-05-09T03:16:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c27fff640 1 -- 172.21.15.46:0/3181975723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c1c032440 con 0x7f6c280b7e70 2024-05-09T03:16:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.631+0000 7f6c27fff640 1 -- 172.21.15.46:0/3181975723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c02ae60 con 0x7f6c280b7e70 2024-05-09T03:16:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/3181975723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280b7e70 msgr2=0x7f6c280b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/3181975723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280b7e70 0x7f6c280b92a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f6c280c95f0 tx=0x7f6c1c02a110 comp rx=0 tx=0).stop 2024-05-09T03:16:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/3181975723 shutdown_connections 2024-05-09T03:16:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/3181975723 >> v2:172.21.15.88:3300/0 conn(0x7f6c281605b0 0x7f6c280b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/3181975723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280b7e70 0x7f6c280b92a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/3181975723 >> 172.21.15.46:0/3181975723 conn(0x7f6c2808d390 msgr2=0x7f6c2808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/3181975723 shutdown_connections 2024-05-09T03:16:41.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.632+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/3181975723 wait complete. 2024-05-09T03:16:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.634+0000 7f6c2f9b7640 1 Processor -- start 2024-05-09T03:16:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.634+0000 7f6c2f9b7640 1 -- start start 2024-05-09T03:16:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2f9b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 0x7f6c280cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2f9b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c281605b0 0x7f6c281f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2f9b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c28162000 0x7f6c281f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2f9b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6c280b4d80 con 0x7f6c280d2e90 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2f9b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6c280b4c00 con 0x7f6c281605b0 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2f9b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6c280b4fc0 con 0x7f6c28162000 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.635+0000 7f6c2d753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 0x7f6c280cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 0x7f6c280cc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59952/0 (socket says 172.21.15.46:59952) 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 -- 172.21.15.46:0/2324716029 learned_addr learned my addr 172.21.15.46:0/2324716029 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2cf52640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c281605b0 0x7f6c281f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2df54640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c28162000 0x7f6c281f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 -- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c28162000 msgr2=0x7f6c281f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c28162000 0x7f6c281f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 -- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c281605b0 msgr2=0x7f6c281f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:41.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c281605b0 0x7f6c281f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:41.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 -- 172.21.15.46:0/2324716029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c281f6b40 con 0x7f6c280d2e90 2024-05-09T03:16:41.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2d753640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 0x7f6c280cc0a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800cb10 tx=0x7f6c1800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c18017070 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c1800ac70 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c28160f60 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c18012620 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.636+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c28161520 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.637+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c2806a270 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.638+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6c1801a070 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.639+0000 7f6c25ffb640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c140762d0 0x7f6c140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.639+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7f6c18097540 con 0x7f6c280d2e90 2024-05-09T03:16:41.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.639+0000 7f6c2cf52640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c140762d0 0x7f6c140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.640+0000 7f6c2cf52640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c140762d0 0x7f6c140787e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c034000 tx=0x7f6c1c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:41.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.643+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6c281f6b40 con 0x7f6c280d2e90 2024-05-09T03:16:41.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:41.877+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6c280d1050 con 0x7f6c280d2e90 2024-05-09T03:16:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.271+0000 7f6c25ffb640 1 -- 172.21.15.46:0/2324716029 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v115) ==== 161+0+0 (secure 0 0 0) 0x7f6c1800ae20 con 0x7f6c280d2e90 2024-05-09T03:16:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c140762d0 msgr2=0x7f6c140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c140762d0 0x7f6c140787e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c034000 tx=0x7f6c1c02e000 comp rx=0 tx=0).stop 2024-05-09T03:16:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 msgr2=0x7f6c280cc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:42.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 0x7f6c280cc0a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800cb10 tx=0x7f6c1800a060 comp rx=0 tx=0).stop 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 shutdown_connections 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c28162000 0x7f6c281f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c281605b0 0x7f6c281f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c140762d0 0x7f6c140787e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 --2- 172.21.15.46:0/2324716029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c280d2e90 0x7f6c280cc0a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 >> 172.21.15.46:0/2324716029 conn(0x7f6c2808d390 msgr2=0x7f6c2808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 shutdown_connections 2024-05-09T03:16:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.274+0000 7f6c2f9b7640 1 -- 172.21.15.46:0/2324716029 wait complete. 2024-05-09T03:16:42.288 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.406+0000 7f721157d640 1 Processor -- start 2024-05-09T03:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.406+0000 7f721157d640 1 -- start start 2024-05-09T03:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f721157d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f720c0ceba0 0x7f720c0cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f721157d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0cf4f0 0x7f720c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f721157d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720c09d470 con 0x7f720c0d3730 2024-05-09T03:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f721157d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f720c09cfe0 con 0x7f720c0ceba0 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f721157d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f720c0b94a0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f720b7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0cf4f0 0x7f720c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f720a7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f720c0ceba0 0x7f720c0cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f720affd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f720c0d3730 0x7f720c06e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41828/0 (socket says 172.21.15.46:41828) 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.408+0000 7f720affd640 1 -- 172.21.15.46:0/3021597922 learned_addr learned my addr 172.21.15.46:0/3021597922 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.409+0000 7f720b7fe640 1 -- 172.21.15.46:0/3021597922 >> v2:172.21.15.88:3300/0 conn(0x7f720c0ceba0 msgr2=0x7f720c0cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.409+0000 7f720b7fe640 1 --2- 172.21.15.46:0/3021597922 >> v2:172.21.15.88:3300/0 conn(0x7f720c0ceba0 0x7f720c0cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.409+0000 7f720b7fe640 1 -- 172.21.15.46:0/3021597922 >> v1:172.21.15.88:6789/0 conn(0x7f720c0d3730 legacy=0x7f720c06e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.409+0000 7f7209ffb640 1 -- 172.21.15.46:0/3021597922 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1237999969 0 0) 0x7f71fc001110 con 0x7f720c0d3730 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.409+0000 7f720b7fe640 1 -- 172.21.15.46:0/3021597922 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c0d11b0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.409+0000 7f720a7fc640 1 --2- 172.21.15.46:0/3021597922 >> v2:172.21.15.88:3300/0 conn(0x7f720c0ceba0 0x7f720c0cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f720b7fe640 1 --2- 172.21.15.46:0/3021597922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0cf4f0 0x7f720c0cf900 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7204009d90 tx=0x7f720402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f543be038fd4853a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f7209ffb640 1 -- 172.21.15.46:0/3021597922 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7204031600 con 0x7f720c0cf4f0 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f7209ffb640 1 -- 172.21.15.46:0/3021597922 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72040317a0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f7209ffb640 1 -- 172.21.15.46:0/3021597922 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7204031b70 con 0x7f720c0cf4f0 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f721157d640 1 -- 172.21.15.46:0/3021597922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0cf4f0 msgr2=0x7f720c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f721157d640 1 --2- 172.21.15.46:0/3021597922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0cf4f0 0x7f720c0cf900 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7204009d90 tx=0x7f720402a5d0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f721157d640 1 -- 172.21.15.46:0/3021597922 shutdown_connections 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f721157d640 1 --2- 172.21.15.46:0/3021597922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0cf4f0 0x7f720c0cf900 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f721157d640 1 --2- 172.21.15.46:0/3021597922 >> v2:172.21.15.88:3300/0 conn(0x7f720c0ceba0 0x7f720c0cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.410+0000 7f721157d640 1 -- 172.21.15.46:0/3021597922 >> 172.21.15.46:0/3021597922 conn(0x7f720c0d05f0 msgr2=0x7f720c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.411+0000 7f721157d640 1 -- 172.21.15.46:0/3021597922 shutdown_connections 2024-05-09T03:16:42.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.411+0000 7f721157d640 1 -- 172.21.15.46:0/3021597922 wait complete. 2024-05-09T03:16:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.412+0000 7f721157d640 1 Processor -- start 2024-05-09T03:16:42.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.412+0000 7f721157d640 1 -- start start 2024-05-09T03:16:42.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f721157d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f720c0ceba0 0x7f720c15f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f721157d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 0x7f720c0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f721157d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0d3730 0x7f720c0cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f721157d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f720c0b9160 con 0x7f720c0cf4f0 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f721157d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f720c0d1490 con 0x7f720c0ceba0 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f721157d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f720c08cd60 con 0x7f720c0d3730 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 0x7f720c0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 0x7f720c0cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59966/0 (socket says 172.21.15.46:59966) 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 -- 172.21.15.46:0/278822657 learned_addr learned my addr 172.21.15.46:0/278822657 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720b7fe640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0d3730 0x7f720c0cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720affd640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f720c0ceba0 0x7f720c15f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 -- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0d3730 msgr2=0x7f720c0cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0d3730 0x7f720c0cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 -- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f720c0ceba0 msgr2=0x7f720c15f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:42.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f720c0ceba0 0x7f720c15f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.414+0000 7f720a7fc640 1 -- 172.21.15.46:0/278822657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720c1602d0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.415+0000 7f720a7fc640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 0x7f720c0cc230 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f71f400e960 tx=0x7f71f400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.415+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f4017070 con 0x7f720c0cf4f0 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.415+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71f400ca00 con 0x7f720c0cf4f0 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.415+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720c1605b0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.415+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f400cdd0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.415+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f720c160b70 con 0x7f720c0cf4f0 2024-05-09T03:16:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.416+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720c160040 con 0x7f720c0cf4f0 2024-05-09T03:16:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.417+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71f401a070 con 0x7f720c0cf4f0 2024-05-09T03:16:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.417+0000 7f72037fe640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71ec0762d0 0x7f71ec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.417+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7f71f40979e0 con 0x7f720c0cf4f0 2024-05-09T03:16:42.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.418+0000 7f720affd640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71ec0762d0 0x7f71ec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:42.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.418+0000 7f720affd640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71ec0762d0 0x7f71ec0787e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f71fc007fd0 tx=0x7f71fc007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:42.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.422+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f71f4060a70 con 0x7f720c0cf4f0 2024-05-09T03:16:42.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:42.688+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f720c0b65b0 con 0x7f720c0cf4f0 2024-05-09T03:16:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.273+0000 7f72037fe640 1 -- 172.21.15.46:0/278822657 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v116) ==== 101+0+0 (secure 0 0 0) 0x7f720c0b65b0 con 0x7f720c0cf4f0 2024-05-09T03:16:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71ec0762d0 msgr2=0x7f71ec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71ec0762d0 0x7f71ec0787e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f71fc007fd0 tx=0x7f71fc007de0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 msgr2=0x7f720c0cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 0x7f720c0cc230 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f71f400e960 tx=0x7f71f400ee20 comp rx=0 tx=0).stop 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 shutdown_connections 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f720c0d3730 0x7f720c0cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71ec0762d0 0x7f71ec0787e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f720c0cf4f0 0x7f720c0cc230 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 --2- 172.21.15.46:0/278822657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f720c0ceba0 0x7f720c15f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 >> 172.21.15.46:0/278822657 conn(0x7f720c0d05f0 msgr2=0x7f720c162a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:43.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 shutdown_connections 2024-05-09T03:16:43.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.276+0000 7f721157d640 1 -- 172.21.15.46:0/278822657 wait complete. 2024-05-09T03:16:43.291 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:43.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.351+0000 7f3663822f40 1 Processor -- start 2024-05-09T03:16:43.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.351+0000 7f3663822f40 1 -- start start 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f3663822f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f51907a7e0 0x55f519149a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f3663822f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190ac7b0 0x55f5190c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f3663822f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 0x55f5190c0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f3663822f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f5190c68a0 con 0x55f5190c3010 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f3663822f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f5190c14d0 con 0x55f5190ac7b0 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f3663822f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f51907abf0 con 0x55f51907a7e0 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f365daed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 0x55f5190c0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f365daed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 0x55f5190c0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59990/0 (socket says 172.21.15.46:59990) 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f365daed640 1 -- 172.21.15.46:0/226005469 learned_addr learned my addr 172.21.15.46:0/226005469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f365caeb640 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190ac7b0 0x55f5190c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.352+0000 7f365d2ec640 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f51907a7e0 0x55f519149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f365daed640 1 -- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f51907a7e0 msgr2=0x55f519149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f365daed640 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f51907a7e0 0x55f519149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f365daed640 1 -- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190ac7b0 msgr2=0x55f5190c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f365daed640 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190ac7b0 0x55f5190c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f365daed640 1 -- 172.21.15.46:0/226005469 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f518fe5350 con 0x55f5190c3010 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f365daed640 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 0x55f5190c0240 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f365800a550 tx=0x7f365800a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf82132eeb90cdc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f36577fe640 1 -- 172.21.15.46:0/226005469 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3658019070 con 0x55f5190c3010 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f36577fe640 1 -- 172.21.15.46:0/226005469 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f365800ae10 con 0x55f5190c3010 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.353+0000 7f36577fe640 1 -- 172.21.15.46:0/226005469 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3658007800 con 0x55f5190c3010 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 -- 172.21.15.46:0/226005469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 msgr2=0x55f5190c0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 0x55f5190c0240 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f365800a550 tx=0x7f365800a640 comp rx=0 tx=0).stop 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 -- 172.21.15.46:0/226005469 shutdown_connections 2024-05-09T03:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5190c3010 0x55f5190c0240 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190ac7b0 0x55f5190c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 --2- 172.21.15.46:0/226005469 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f51907a7e0 0x55f519149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 -- 172.21.15.46:0/226005469 >> 172.21.15.46:0/226005469 conn(0x55f518f9fac0 msgr2=0x55f519153030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 -- 172.21.15.46:0/226005469 shutdown_connections 2024-05-09T03:16:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.354+0000 7f3663822f40 1 -- 172.21.15.46:0/226005469 wait complete. 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.355+0000 7f3663822f40 1 Processor -- start 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.355+0000 7f3663822f40 1 -- start start 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f3663822f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f51907a7e0 0x55f519149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f3663822f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f5190ac7b0 0x55f51907e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f3663822f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190c3010 0x55f5190ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f3663822f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f51914a160 con 0x55f51907a7e0 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f3663822f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f51914b690 con 0x55f5190c3010 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f3663822f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f51914b480 con 0x55f5190ac7b0 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f365d2ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f51907a7e0 0x55f519149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f365d2ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f51907a7e0 0x55f519149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60006/0 (socket says 172.21.15.46:60006) 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.356+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 learned_addr learned my addr 172.21.15.46:0/1633955762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f5190ac7b0 0x55f51907e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365daed640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190c3010 0x55f5190ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f5190ac7b0 msgr2=0x55f51907e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f5190ac7b0 0x55f51907e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190c3010 msgr2=0x55f5190ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f5190c3010 0x55f5190ab250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51907c4d0 con 0x55f51907a7e0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f51907a7e0 0x55f519149c00 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3650054040 tx=0x7f3650003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f36567fc640 1 -- 172.21.15.46:0/1633955762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650055040 con 0x55f51907a7e0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f36567fc640 1 -- 172.21.15.46:0/1633955762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36500514b0 con 0x55f51907a7e0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f36567fc640 1 -- 172.21.15.46:0/1633955762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650064650 con 0x55f51907a7e0 2024-05-09T03:16:43.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f51907c7b0 con 0x55f51907a7e0 2024-05-09T03:16:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.357+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f51907f0a0 con 0x55f51907a7e0 2024-05-09T03:16:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.359+0000 7f36567fc640 1 -- 172.21.15.46:0/1633955762 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f365006b060 con 0x55f51907a7e0 2024-05-09T03:16:43.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.359+0000 7f36567fc640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3640074cb0 0x7f36400771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.359+0000 7f36567fc640 1 -- 172.21.15.46:0/1633955762 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7f36500d9bf0 con 0x55f51907a7e0 2024-05-09T03:16:43.362 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:43.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.360+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f51907ebd0 0x55f51907b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.360+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51907bce0 con 0x55f51907ebd0 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.360+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3640074cb0 0x7f36400771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.360+0000 7f365daed640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f51907ebd0 0x55f51907b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.360+0000 7f365daed640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f51907ebd0 0x55f51907b7a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.360+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3640074cb0 0x7f36400771c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x55f5190bf7b0 tx=0x7f364800c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.362+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v89'133 uv107 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.362+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191f64f0 con 0x55f51907ebd0 2024-05-09T03:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.363+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v89'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.363+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6890 con 0x55f51907ebd0 2024-05-09T03:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.365+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v89'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.365 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.365+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f7c40 con 0x55f51907ebd0 2024-05-09T03:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.366+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v89'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.366+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f7fe0 con 0x55f51907ebd0 2024-05-09T03:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.367+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v89'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.367+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6c30 con 0x55f51907ebd0 2024-05-09T03:16:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.369+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v89'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.369+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f7040 con 0x55f51907ebd0 2024-05-09T03:16:43.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.370+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v89'139 uv113 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.370+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191f7040 con 0x55f51907ebd0 2024-05-09T03:16:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.371+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v89'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.371+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.373+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v89'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.374 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:43.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.373+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.374+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v89'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.374+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.376+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v89'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.376+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.377+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v89'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.377+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.378+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v89'145 uv120 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.378+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.380+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v89'146 uv146 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.380+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.381+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v89'147 uv147 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.382 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.381+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.382+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v89'148 uv148 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.382+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.384+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v89'149 uv149 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.384+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f6fd0 con 0x55f51907ebd0 2024-05-09T03:16:43.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.385+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v89'150 uv150 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.385+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.386+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v89'151 uv151 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.386+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.388+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v89'152 uv126 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.388+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.389+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v89'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.389+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.391+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v89'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.391 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:43.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.391+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.392+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v89'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.392+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.394+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v89'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.394+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.395+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v89'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.395+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.396+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v89'158 uv132 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.396+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.397+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v89'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.397+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.399+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v89'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.399 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:43.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.399+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f73b0 con 0x55f51907ebd0 2024-05-09T03:16:43.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.400+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v89'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.400+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f7800 con 0x55f51907ebd0 2024-05-09T03:16:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.401+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v89'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.401+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191f7800 con 0x55f51907ebd0 2024-05-09T03:16:43.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.403+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v89'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.403+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55f5191f8380 0x55f5191fa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.403+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191face0 con 0x55f5191f8380 2024-05-09T03:16:43.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.403+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55f5191f8380 0x55f5191fa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.403+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55f5191f8380 0x55f5191fa7a0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.405+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v89'109 uv91 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.405+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191fb080 con 0x55f5191f8380 2024-05-09T03:16:43.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.406+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v89'110 uv110 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.406+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fb420 con 0x55f5191f8380 2024-05-09T03:16:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.407+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v89'111 uv111 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.408 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.407+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fbc70 con 0x55f5191f8380 2024-05-09T03:16:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.409+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v89'112 uv112 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.409+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fc010 con 0x55f5191f8380 2024-05-09T03:16:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.410+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v89'113 uv113 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.410+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fc3b0 con 0x55f5191f8380 2024-05-09T03:16:43.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.411+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v89'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.412 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.411+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fc750 con 0x55f5191f8380 2024-05-09T03:16:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.412+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v89'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.412+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.413+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v89'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.413+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.414+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000001 [create] v89'117 uv101 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.415+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.416+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000001 [delete] v89'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.416+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.417+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v89'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:43.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.417+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200550 con 0x55f5191f8380 2024-05-09T03:16:43.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.418+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [write 0~14895] v89'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.418+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200550 con 0x55f5191f8380 2024-05-09T03:16:43.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.419+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v89'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.419+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.421+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 606.00000000 [create] v89'122 uv107 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.421+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.422+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 606.00000000 [delete] v89'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.422+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.423+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v89'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:43.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.423+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fdb00 con 0x55f5191f8380 2024-05-09T03:16:43.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.424+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v89'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.424+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fdb00 con 0x55f5191f8380 2024-05-09T03:16:43.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.425+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v89'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.425+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f5191f8380 2024-05-09T03:16:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.426+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v89'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5191f8380 2024-05-09T03:16:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.426+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f51907ebd0 2024-05-09T03:16:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.427+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(51 607.00000000 [create] v89'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.427+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f51907ebd0 2024-05-09T03:16:43.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.429+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(52 607.00000000 [delete] v89'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.429+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f51907ebd0 2024-05-09T03:16:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.430+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(53 607.00000000 [create] v89'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.430+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fdb00 con 0x55f51907ebd0 2024-05-09T03:16:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.431+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v89'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.431+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fdb00 con 0x55f51907ebd0 2024-05-09T03:16:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.432+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v89'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.432+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fcaf0 con 0x55f51907ebd0 2024-05-09T03:16:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.433+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v89'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.433+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f5191fd000 0x55f5191ff420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.433+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191ff960 con 0x55f5191fd000 2024-05-09T03:16:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.433+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f5191fd000 0x55f5191ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.433+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f5191fd000 0x55f5191ff420 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.434+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v89'23 uv22 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.434+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5191ffdd0 con 0x55f5191fd000 2024-05-09T03:16:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.436+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v89'24 uv24 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.436+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192002d0 con 0x55f5191fd000 2024-05-09T03:16:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.437+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v89'25 uv25 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.437 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:43.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.437+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5191fb8a0 con 0x55f5191fd000 2024-05-09T03:16:43.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.438+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v89'26 uv26 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.438+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200790 con 0x55f51907ebd0 2024-05-09T03:16:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.439+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(61 608.00000000 [create] v89'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.439+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519200790 con 0x55f51907ebd0 2024-05-09T03:16:43.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.440+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(62 608.00000000 [delete] v89'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.440+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200790 con 0x55f51907ebd0 2024-05-09T03:16:43.442 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:43.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.442+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(63 608.00000000 [create] v89'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.442+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200790 con 0x55f51907ebd0 2024-05-09T03:16:43.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.444+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v89'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.444+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200790 con 0x55f51907ebd0 2024-05-09T03:16:43.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.447+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v89'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.447+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200790 con 0x55f51907ebd0 2024-05-09T03:16:43.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.449+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v89'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.450+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f5192015f0 0x55f519203a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.450+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519203f50 con 0x55f5192015f0 2024-05-09T03:16:43.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.450+0000 7f365daed640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f5192015f0 0x55f519203a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.450+0000 7f365daed640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f5192015f0 0x55f519203a10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.451+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(67 604.00000000 [create] v89'48 uv42 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.451+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519203f50 con 0x55f5192015f0 2024-05-09T03:16:43.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.453+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(68 604.00000000 [delete] v89'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.453+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.454+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(69 604.00000000 [create] v89'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.454 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:43.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.454+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519205350 con 0x55f5192015f0 2024-05-09T03:16:43.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.455+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v89'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.455+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519205350 con 0x55f5192015f0 2024-05-09T03:16:43.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.456+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v89'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.456+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.457+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v89'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.457+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.458+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(73 500.00000000 [create] v89'54 uv47 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.458+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.460+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(74 500.00000000 [delete] v89'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.460+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.461+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(75 500.00000000 [create] v89'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.462 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:43.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.461+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.462+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v89'57 uv57 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.462+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519204340 con 0x55f5192015f0 2024-05-09T03:16:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.463+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v89'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.463+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55f5192049c0 0x55f519206de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.463+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192073f0 con 0x55f5192049c0 2024-05-09T03:16:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.463+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55f5192049c0 0x55f519206de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.464+0000 7f365d2ec640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55f5192049c0 0x55f519206de0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.465+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(78 mds_snaptable [create] v89'22 uv21 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5192049c0 2024-05-09T03:16:43.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.465+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f5192049c0 2024-05-09T03:16:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.466+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(79 mds_snaptable [delete] v89'23 uv23 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5192049c0 2024-05-09T03:16:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.466+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f5192049c0 2024-05-09T03:16:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.467+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(80 mds_snaptable [create] v89'24 uv24 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5192049c0 2024-05-09T03:16:43.468 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.467+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f5192049c0 2024-05-09T03:16:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.468+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v89'25 uv25 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5192049c0 2024-05-09T03:16:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.468+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f5192049c0 2024-05-09T03:16:43.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.469+0000 7f365d2ec640 1 -- 172.21.15.46:0/1633955762 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v89'26 uv26 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f365009ef80 con 0x55f5192049c0 2024-05-09T03:16:43.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.469+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f51907ebd0 2024-05-09T03:16:43.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.470+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(83 605.00000000 [create] v89'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.470+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f51907ebd0 2024-05-09T03:16:43.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.472+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(84 605.00000000 [delete] v89'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.472+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f51907ebd0 2024-05-09T03:16:43.473 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.473+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(85 605.00000000 [create] v89'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.473+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192087d0 con 0x55f51907ebd0 2024-05-09T03:16:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.474+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v89'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.474+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192087d0 con 0x55f51907ebd0 2024-05-09T03:16:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.475+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v89'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.475+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f51907ebd0 2024-05-09T03:16:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.476+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v89'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f51907ebd0 2024-05-09T03:16:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.476+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192077c0 con 0x55f5191fd000 2024-05-09T03:16:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.477+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(89 mds0_inotable [create] v89'44 uv38 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.477+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519207b60 con 0x55f5191fd000 2024-05-09T03:16:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.478+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(90 mds0_inotable [delete] v89'45 uv45 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.478+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519207f50 con 0x55f5191fd000 2024-05-09T03:16:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.479+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(91 mds0_inotable [create] v89'46 uv46 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.479 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.479+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519200cd0 con 0x55f5191fd000 2024-05-09T03:16:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.480+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v89'47 uv47 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.480+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192087a0 con 0x55f5191fd000 2024-05-09T03:16:43.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.481+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v89'48 uv48 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.481+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192087a0 con 0x55f5191fd000 2024-05-09T03:16:43.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.482+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(94 100.00000000 [create] v89'49 uv43 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.482+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f5192087a0 con 0x55f5191fd000 2024-05-09T03:16:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.483+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(95 100.00000000 [delete] v89'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.483+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192087a0 con 0x55f5191fd000 2024-05-09T03:16:43.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.484+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(96 100.00000000 [create] v89'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.485 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:43.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.484+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f5192087a0 con 0x55f5191fd000 2024-05-09T03:16:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.485+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v89'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.485+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920b4f0 con 0x55f5191fd000 2024-05-09T03:16:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.486+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v89'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.486+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55f5192090a0 0x55f51920b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.486+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920b4c0 con 0x55f5192090a0 2024-05-09T03:16:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.486+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55f5192090a0 0x55f51920b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.487+0000 7f365caeb640 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55f5192090a0 0x55f51920b9f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.488+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v89'22 uv21 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5192090a0 2024-05-09T03:16:43.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.488+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5192090a0 2024-05-09T03:16:43.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.489+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v89'23 uv23 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5192090a0 2024-05-09T03:16:43.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.489+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5192090a0 2024-05-09T03:16:43.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.490+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v89'24 uv24 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5192090a0 2024-05-09T03:16:43.491 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.490+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5192090a0 2024-05-09T03:16:43.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.491+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v89'25 uv25 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5192090a0 2024-05-09T03:16:43.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.491+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5191fd000 2024-05-09T03:16:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.493+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v89'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.493+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5191fd000 2024-05-09T03:16:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.494+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v89'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.494+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5191fd000 2024-05-09T03:16:43.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.495+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v89'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.495 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:43.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.495+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5191fd000 2024-05-09T03:16:43.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.496+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v89'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.496+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5191fd000 2024-05-09T03:16:43.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.497+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v89'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.497+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5191fd000 2024-05-09T03:16:43.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.498+0000 7f365caeb640 1 -- 172.21.15.46:0/1633955762 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v89'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3648020040 con 0x55f5191fd000 2024-05-09T03:16:43.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.498+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5192015f0 2024-05-09T03:16:43.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.499+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v89'22 uv21 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.499+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5192015f0 2024-05-09T03:16:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.500+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v89'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.501+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f519208430 con 0x55f5192015f0 2024-05-09T03:16:43.504 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.503+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v89'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.503+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.505+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v89'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.505+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.506+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v89'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.506+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.507+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v89'57 uv49 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.507+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.508+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v89'58 uv58 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.508+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.510 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.509+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v89'59 uv59 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.509+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.511+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v89'60 uv60 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.511+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.512+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v89'61 uv61 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.512+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.514+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v89'62 uv56 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.514+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.517+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v89'63 uv63 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.517+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.520 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:43.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.519+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v89'64 uv64 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.519+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920cf40 con 0x55f5192015f0 2024-05-09T03:16:43.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.521+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v89'65 uv65 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.521+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920cf40 con 0x55f5192015f0 2024-05-09T03:16:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.522+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v89'66 uv66 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.522+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.523+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v89'67 uv67 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.523+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x55f51920bf30 con 0x55f5192015f0 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v89'68 uv68 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f365801c040 con 0x55f5192015f0 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f51907ebd0 msgr2=0x55f51907b7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55f51907ebd0 0x55f51907b7a0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f5191fd000 msgr2=0x55f5191ff420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f5191fd000 0x55f5191ff420 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55f5192090a0 msgr2=0x55f51920b9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55f5192090a0 0x55f51920b9f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 reap_dead start 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55f5191f8380 msgr2=0x55f5191fa7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55f5191f8380 0x55f5191fa7a0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f5192015f0 msgr2=0x55f519203a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f5192015f0 0x55f519203a10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55f5192049c0 msgr2=0x55f519206de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55f5192049c0 0x55f519206de0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3640074cb0 msgr2=0x7f36400771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3640074cb0 0x7f36400771c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x55f5190bf7b0 tx=0x7f364800c040 comp rx=0 tx=0).stop 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f51907a7e0 msgr2=0x55f519149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 --2- 172.21.15.46:0/1633955762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f51907a7e0 0x55f519149c00 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3650054040 tx=0x7f3650003330 comp rx=0 tx=0).stop 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f365daed640 1 -- 172.21.15.46:0/1633955762 reap_dead start 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 shutdown_connections 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 >> 172.21.15.46:0/1633955762 conn(0x55f518f9fac0 msgr2=0x55f5190c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 shutdown_connections 2024-05-09T03:16:43.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.524+0000 7f3663822f40 1 -- 172.21.15.46:0/1633955762 wait complete. 2024-05-09T03:16:43.528 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000001 2024-05-09T03:16:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.583+0000 7f9150232f40 1 Processor -- start 2024-05-09T03:16:43.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.583+0000 7f9150232f40 1 -- start start 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9150232f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 0x55d718a16790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9150232f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718a1c010 0x55d718aaee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9150232f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d718aac8c0 0x55d718aaf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9150232f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d7189d3bf0 con 0x55d7189d37e0 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9150232f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d718a1e670 con 0x55d718a1c010 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9150232f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d718a1a950 con 0x55d718aac8c0 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 0x55d718a16790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 0x55d718a16790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60034/0 (socket says 172.21.15.46:60034) 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 -- 172.21.15.46:0/891390868 learned_addr learned my addr 172.21.15.46:0/891390868 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9143fff640 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718a1c010 0x55d718aaee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f914a4fb640 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d718aac8c0 0x55d718aaf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 -- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d718aac8c0 msgr2=0x55d718aaf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d718aac8c0 0x55d718aaf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 -- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718a1c010 msgr2=0x55d718aaee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718a1c010 0x55d718aaee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9149cfa640 1 -- 172.21.15.46:0/891390868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d71893e350 con 0x55d7189d37e0 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.584+0000 7f9143fff640 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718a1c010 0x55d718aaee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9149cfa640 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 0x55d718a16790 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f913c045e50 tx=0x7f913c04e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7c7c0f2d79e80f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9148cf8640 1 -- 172.21.15.46:0/891390868 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c065070 con 0x55d7189d37e0 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9148cf8640 1 -- 172.21.15.46:0/891390868 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f913c05a040 con 0x55d7189d37e0 2024-05-09T03:16:43.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9148cf8640 1 -- 172.21.15.46:0/891390868 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c059070 con 0x55d7189d37e0 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 -- 172.21.15.46:0/891390868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 msgr2=0x55d718a16790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 0x55d718a16790 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f913c045e50 tx=0x7f913c04e000 comp rx=0 tx=0).stop 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 -- 172.21.15.46:0/891390868 shutdown_connections 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d718aac8c0 0x55d718aaf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718a1c010 0x55d718aaee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 --2- 172.21.15.46:0/891390868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7189d37e0 0x55d718a16790 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 -- 172.21.15.46:0/891390868 >> 172.21.15.46:0/891390868 conn(0x55d7188f8ac0 msgr2=0x55d718a193a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 -- 172.21.15.46:0/891390868 shutdown_connections 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.585+0000 7f9150232f40 1 -- 172.21.15.46:0/891390868 wait complete. 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.586+0000 7f9150232f40 1 Processor -- start 2024-05-09T03:16:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.586+0000 7f9150232f40 1 -- start start 2024-05-09T03:16:43.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.587+0000 7f9150232f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d7189d37e0 0x55d718aae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.587+0000 7f9150232f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d718a1c010 0x55d718aae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.587+0000 7f9150232f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718aac8c0 0x55d718aa8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.587+0000 7f9150232f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d718aaba60 con 0x55d718a1c010 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9150232f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d718aa9da0 con 0x55d718aac8c0 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9150232f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d718b3b5f0 con 0x55d7189d37e0 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d718a1c010 0x55d718aae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d718a1c010 0x55d718aae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60042/0 (socket says 172.21.15.46:60042) 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 -- 172.21.15.46:0/561019396 learned_addr learned my addr 172.21.15.46:0/561019396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9149cfa640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d7189d37e0 0x55d718aae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f914a4fb640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718aac8c0 0x55d718aa8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 -- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d7189d37e0 msgr2=0x55d718aae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d7189d37e0 0x55d718aae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 -- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718aac8c0 msgr2=0x55d718aa8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d718aac8c0 0x55d718aa8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 -- 172.21.15.46:0/561019396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d718aaf9a0 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9143fff640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d718a1c010 0x55d718aae830 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f913400bec0 tx=0x7f91340073f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9142ffd640 1 -- 172.21.15.46:0/561019396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134007c20 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9142ffd640 1 -- 172.21.15.46:0/561019396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9134007dc0 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d718a19820 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9142ffd640 1 -- 172.21.15.46:0/561019396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9134015670 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.588+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d718aaed70 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.590+0000 7f9142ffd640 1 -- 172.21.15.46:0/561019396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9134015810 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.590+0000 7f9142ffd640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f912c074cb0 0x7f912c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f9142ffd640 1 -- 172.21.15.46:0/561019396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7f9134095a80 con 0x55d718a1c010 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f9149cfa640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f912c074cb0 0x7f912c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f9150232f40 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d718aa9880 0x55d718b46ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e89) -- 0x55d718b3e350 con 0x55d718aa9880 2024-05-09T03:16:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f914a4fb640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d718aa9880 0x55d718b46ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f9149cfa640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f912c074cb0 0x7f912c0771c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f913c062000 tx=0x7f913c051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.591+0000 7f914a4fb640 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d718aa9880 0x55d718b46ab0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f914a4fb640 1 -- 172.21.15.46:0/561019396 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000001 [delete] v89'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9144011040 con 0x55d718aa9880 2024-05-09T03:16:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d718aa9880 msgr2=0x55d718b46ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d718aa9880 0x55d718b46ab0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f912c074cb0 msgr2=0x7f912c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f912c074cb0 0x7f912c0771c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f913c062000 tx=0x7f913c051000 comp rx=0 tx=0).stop 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d718a1c010 msgr2=0x55d718aae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 --2- 172.21.15.46:0/561019396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d718a1c010 0x55d718aae830 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f913400bec0 tx=0x7f91340073f0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f914a4fb640 1 -- 172.21.15.46:0/561019396 reap_dead start 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 shutdown_connections 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 >> 172.21.15.46:0/561019396 conn(0x55d7188f8ac0 msgr2=0x55d718aaf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 shutdown_connections 2024-05-09T03:16:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.593+0000 7f9150232f40 1 -- 172.21.15.46:0/561019396 wait complete. 2024-05-09T03:16:43.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.715+0000 7f5192bc7640 1 Processor -- start 2024-05-09T03:16:43.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.715+0000 7f5192bc7640 1 -- start start 2024-05-09T03:16:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f5192bc7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f518c0b1dd0 0x7f518c072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f5192bc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c1461a0 0x7f518c1465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f5192bc7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f518c0c8720 con 0x7f518c0c78a0 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f5192bc7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f518c0b21e0 con 0x7f518c0b1dd0 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f5192bc7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f518c072a40 con 0x7f518c1461a0 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f518bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f518c0b1dd0 0x7f518c072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f518bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f518c0b1dd0 0x7f518c072500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60052/0 (socket says 172.21.15.46:60052) 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.716+0000 7f518bfff640 1 -- 172.21.15.46:0/595896925 learned_addr learned my addr 172.21.15.46:0/595896925 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.717+0000 7f5191164640 1 --2- 172.21.15.46:0/595896925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c1461a0 0x7f518c1465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.717+0000 7f518bfff640 1 -- 172.21.15.46:0/595896925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c1461a0 msgr2=0x7f518c1465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.717+0000 7f518bfff640 1 --2- 172.21.15.46:0/595896925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c1461a0 0x7f518c1465b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.717+0000 7f518bfff640 1 -- 172.21.15.46:0/595896925 >> v1:172.21.15.88:6789/0 conn(0x7f518c0c78a0 legacy=0x7f518c163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.717+0000 7f518b7fe640 1 -- 172.21.15.46:0/595896925 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1477876100 0 0) 0x7f518c0c8720 con 0x7f518c0c78a0 2024-05-09T03:16:43.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.717+0000 7f518bfff640 1 -- 172.21.15.46:0/595896925 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c0c7e80 con 0x7f518c0b1dd0 2024-05-09T03:16:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f518bfff640 1 --2- 172.21.15.46:0/595896925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0b1dd0 0x7f518c072500 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5174009870 tx=0x7f517402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a24ac9452f1ee6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f518b7fe640 1 -- 172.21.15.46:0/595896925 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174037070 con 0x7f518c0b1dd0 2024-05-09T03:16:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f518b7fe640 1 -- 172.21.15.46:0/595896925 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5174032440 con 0x7f518c0b1dd0 2024-05-09T03:16:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f518b7fe640 1 -- 172.21.15.46:0/595896925 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174032810 con 0x7f518c0b1dd0 2024-05-09T03:16:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f5192bc7640 1 -- 172.21.15.46:0/595896925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0b1dd0 msgr2=0x7f518c072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f5192bc7640 1 --2- 172.21.15.46:0/595896925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0b1dd0 0x7f518c072500 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5174009870 tx=0x7f517402a270 comp rx=0 tx=0).stop 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f5192bc7640 1 -- 172.21.15.46:0/595896925 shutdown_connections 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f5192bc7640 1 --2- 172.21.15.46:0/595896925 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c1461a0 0x7f518c1465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f5192bc7640 1 --2- 172.21.15.46:0/595896925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0b1dd0 0x7f518c072500 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.718+0000 7f5192bc7640 1 -- 172.21.15.46:0/595896925 >> 172.21.15.46:0/595896925 conn(0x7f518c0705c0 msgr2=0x7f518c0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.719+0000 7f5192bc7640 1 -- 172.21.15.46:0/595896925 shutdown_connections 2024-05-09T03:16:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.719+0000 7f5192bc7640 1 -- 172.21.15.46:0/595896925 wait complete. 2024-05-09T03:16:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.720+0000 7f5192bc7640 1 Processor -- start 2024-05-09T03:16:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.720+0000 7f5192bc7640 1 -- start start 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5192bc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c0b1dd0 0x7f518c0b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5192bc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 0x7f518c162950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5192bc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c1461a0 0x7f518c162e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5192bc7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f518c0c8440 con 0x7f518c0c78a0 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5192bc7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f518c145d10 con 0x7f518c1461a0 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5192bc7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f518c164560 con 0x7f518c0b1dd0 2024-05-09T03:16:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 0x7f518c162950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 0x7f518c162950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60060/0 (socket says 172.21.15.46:60060) 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 -- 172.21.15.46:0/1547084365 learned_addr learned my addr 172.21.15.46:0/1547084365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5190963640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c0b1dd0 0x7f518c0b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f5191164640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c1461a0 0x7f518c162e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 -- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c0b1dd0 msgr2=0x7f518c0b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c0b1dd0 0x7f518c0b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 -- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c1461a0 msgr2=0x7f518c162e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c1461a0 0x7f518c162e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 -- 172.21.15.46:0/1547084365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c0ae920 con 0x7f518c0c78a0 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.722+0000 7f518bfff640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 0x7f518c162950 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5174009870 tx=0x7f51740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.723+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174041020 con 0x7f518c0c78a0 2024-05-09T03:16:43.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.723+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5174030640 con 0x7f518c0c78a0 2024-05-09T03:16:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.723+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518c0aec00 con 0x7f518c0c78a0 2024-05-09T03:16:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.723+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5174037040 con 0x7f518c0c78a0 2024-05-09T03:16:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.723+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f518c0af1c0 con 0x7f518c0c78a0 2024-05-09T03:16:43.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.724+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f518c0ae690 con 0x7f518c0c78a0 2024-05-09T03:16:43.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.724+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5174044020 con 0x7f518c0c78a0 2024-05-09T03:16:43.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.725+0000 7f51897fa640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f516c0762d0 0x7f516c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:43.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.725+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6197+0+0 (secure 0 0 0) 0x7f51740b7f50 con 0x7f518c0c78a0 2024-05-09T03:16:43.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.725+0000 7f5190963640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f516c0762d0 0x7f516c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:43.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.726+0000 7f5190963640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f516c0762d0 0x7f516c0787e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f517c0012e0 tx=0x7f517c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:43.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.730+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f517404b030 con 0x7f518c0c78a0 2024-05-09T03:16:43.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:43.960+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f518c06a270 con 0x7f518c0c78a0 2024-05-09T03:16:44.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.284+0000 7f51897fa640 1 -- 172.21.15.46:0/1547084365 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v117) ==== 162+0+0 (secure 0 0 0) 0x7f5174080f60 con 0x7f518c0c78a0 2024-05-09T03:16:44.285 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f516c0762d0 msgr2=0x7f516c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f516c0762d0 0x7f516c0787e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f517c0012e0 tx=0x7f517c009040 comp rx=0 tx=0).stop 2024-05-09T03:16:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 msgr2=0x7f518c162950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 0x7f518c162950 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5174009870 tx=0x7f51740046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 shutdown_connections 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f518c1461a0 0x7f518c162e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f516c0762d0 0x7f516c0787e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f518c0c78a0 0x7f518c162950 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 --2- 172.21.15.46:0/1547084365 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f518c0b1dd0 0x7f518c0b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 >> 172.21.15.46:0/1547084365 conn(0x7f518c0705c0 msgr2=0x7f518c0b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 shutdown_connections 2024-05-09T03:16:44.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.287+0000 7f5192bc7640 1 -- 172.21.15.46:0/1547084365 wait complete. 2024-05-09T03:16:44.301 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:44.301+0000 7f649f06f640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-05-09T03:16:44.302 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:44.302+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:44.302 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:44.302+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:16:44.304 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:44.308 INFO:tasks.ceph.mds.c.smithi046.stderr: -13> 2024-05-09T03:16:44.301+0000 7f649f06f640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-05-09T03:16:44.364 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:44.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.423+0000 7fe94eb4d640 1 Processor -- start 2024-05-09T03:16:44.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.423+0000 7fe94eb4d640 1 -- start start 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94eb4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9481606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94eb4d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9480b1e70 0x7fe94809c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94eb4d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe948072100 con 0x7fe94806c680 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94eb4d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe9480b8770 con 0x7fe9480b1e70 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94eb4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe94809c780 con 0x7fe9480b8360 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe947fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9480b1e70 0x7fe94809c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94d0ea640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe94806c680 0x7fe94806ca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46602/0 (socket says 172.21.15.46:46602) 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94d0ea640 1 -- 172.21.15.46:0/1057192236 learned_addr learned my addr 172.21.15.46:0/1057192236 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:44.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe947fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9480b1e70 0x7fe94809c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60072/0 (socket says 172.21.15.46:60072) 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe94c8e9640 1 --2- 172.21.15.46:0/1057192236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9481606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe947fff640 1 -- 172.21.15.46:0/1057192236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 msgr2=0x7fe9481606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe947fff640 1 --2- 172.21.15.46:0/1057192236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9481606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe947fff640 1 -- 172.21.15.46:0/1057192236 >> v1:172.21.15.46:6789/0 conn(0x7fe94806c680 legacy=0x7fe94806ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.425+0000 7fe947fff640 1 -- 172.21.15.46:0/1057192236 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9480723e0 con 0x7fe9480b1e70 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.426+0000 7fe947fff640 1 --2- 172.21.15.46:0/1057192236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe94809c240 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0097a0 tx=0x7fe93c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e21a4788d080f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.426+0000 7fe9477fe640 1 -- 172.21.15.46:0/1057192236 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c037070 con 0x7fe9480b1e70 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.426+0000 7fe9477fe640 1 -- 172.21.15.46:0/1057192236 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe93c02ae50 con 0x7fe9480b1e70 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.426+0000 7fe9477fe640 1 -- 172.21.15.46:0/1057192236 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c032660 con 0x7fe9480b1e70 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1057192236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 msgr2=0x7fe94809c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1057192236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe94809c240 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0097a0 tx=0x7fe93c02a100 comp rx=0 tx=0).stop 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1057192236 shutdown_connections 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1057192236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe94809c240 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1057192236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9481606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1057192236 >> 172.21.15.46:0/1057192236 conn(0x7fe9480d2c70 msgr2=0x7fe9480d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:44.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1057192236 shutdown_connections 2024-05-09T03:16:44.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.427+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1057192236 wait complete. 2024-05-09T03:16:44.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.429+0000 7fe94eb4d640 1 Processor -- start 2024-05-09T03:16:44.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.429+0000 7fe94eb4d640 1 -- start start 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe94eb4d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe94806c680 0x7fe948153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe94eb4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe948162410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe94eb4d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9480b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe94eb4d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9480726c0 con 0x7fe9480b1e70 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe94eb4d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe94809bd20 con 0x7fe94806c680 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe94eb4d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe9480b2280 con 0x7fe9480b8360 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe947fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe948162410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe947fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe948162410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60084/0 (socket says 172.21.15.46:60084) 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.430+0000 7fe947fff640 1 -- 172.21.15.46:0/1178569921 learned_addr learned my addr 172.21.15.46:0/1178569921 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe94c8e9640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe94806c680 0x7fe948153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe94d0ea640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9480b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe947fff640 1 -- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 msgr2=0x7fe9480b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe947fff640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9480b2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe947fff640 1 -- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe94806c680 msgr2=0x7fe948153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe947fff640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe94806c680 0x7fe948153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe947fff640 1 -- 172.21.15.46:0/1178569921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe948153650 con 0x7fe9480b1e70 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe94d0ea640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9480b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe947fff640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe948162410 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0098c0 tx=0x7fe93c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c041030 con 0x7fe9480b1e70 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe93c02ac50 con 0x7fe9480b1e70 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9481fa890 con 0x7fe9480b1e70 2024-05-09T03:16:44.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c037040 con 0x7fe9480b1e70 2024-05-09T03:16:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.431+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9481f9db0 con 0x7fe9480b1e70 2024-05-09T03:16:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.432+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe94806a270 con 0x7fe9480b1e70 2024-05-09T03:16:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.433+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe93c044070 con 0x7fe9480b1e70 2024-05-09T03:16:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.434+0000 7fe9457fa640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe934076300 0x7fe934078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.434+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7fe93c02ae00 con 0x7fe9480b1e70 2024-05-09T03:16:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.434+0000 7fe94c8e9640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe934076300 0x7fe934078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.434+0000 7fe94c8e9640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe934076300 0x7fe934078810 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe9480b8b90 tx=0x7fe938008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:44.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.438+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe948153650 con 0x7fe9480b1e70 2024-05-09T03:16:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.667+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe948162eb0 con 0x7fe9480b1e70 2024-05-09T03:16:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.668+0000 7fe9457fa640 1 -- 172.21.15.46:0/1178569921 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4342 (secure 0 0 0) 0x7fe93c0811b0 con 0x7fe9480b1e70 2024-05-09T03:16:44.670 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:44.670 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":119,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:44.302572+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 119 2024-05-09T03:16:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe934076300 msgr2=0x7fe934078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe934076300 0x7fe934078810 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe9480b8b90 tx=0x7fe938008040 comp rx=0 tx=0).stop 2024-05-09T03:16:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 msgr2=0x7fe948162410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe948162410 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0098c0 tx=0x7fe93c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 shutdown_connections 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe9480b8360 0x7fe9480b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe934076300 0x7fe934078810 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9480b1e70 0x7fe948162410 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 --2- 172.21.15.46:0/1178569921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe94806c680 0x7fe948153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 >> 172.21.15.46:0/1178569921 conn(0x7fe9480d2c70 msgr2=0x7fe9480711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 shutdown_connections 2024-05-09T03:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.670+0000 7fe94eb4d640 1 -- 172.21.15.46:0/1178569921 wait complete. 2024-05-09T03:16:44.682 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:44.683 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:44.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.841+0000 7f4d3c43b640 1 Processor -- start 2024-05-09T03:16:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.841+0000 7f4d3c43b640 1 -- start start 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3c43b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 0x7f4d341615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3c43b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d3415dfe0 0x7f4d340d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3c43b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d34160b20 con 0x7f4d340d2480 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3c43b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4d34161150 con 0x7f4d3415dfe0 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3c43b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d340d15f0 con 0x7f4d34164c70 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3a1d7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 0x7f4d341615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d399d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d3415dfe0 0x7f4d340d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3a1d7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 0x7f4d341615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:60824/0 (socket says 172.21.15.46:60824) 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3a9d8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4d340d2480 0x7f4d340d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.842+0000 7f4d3a1d7640 1 -- 172.21.15.46:0/3062105325 learned_addr learned my addr 172.21.15.46:0/3062105325 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.843+0000 7f4d3a1d7640 1 -- 172.21.15.46:0/3062105325 >> v2:172.21.15.88:3300/0 conn(0x7f4d3415dfe0 msgr2=0x7f4d340d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.843+0000 7f4d3a1d7640 1 --2- 172.21.15.46:0/3062105325 >> v2:172.21.15.88:3300/0 conn(0x7f4d3415dfe0 0x7f4d340d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.843+0000 7f4d3a1d7640 1 -- 172.21.15.46:0/3062105325 >> v1:172.21.15.88:6789/0 conn(0x7f4d340d2480 legacy=0x7f4d340d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.843+0000 7f4d3a1d7640 1 -- 172.21.15.46:0/3062105325 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d340d18d0 con 0x7f4d34164c70 2024-05-09T03:16:44.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.843+0000 7f4d399d6640 1 --2- 172.21.15.46:0/3062105325 >> v2:172.21.15.88:3300/0 conn(0x7f4d3415dfe0 0x7f4d340d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.843+0000 7f4d3a1d7640 1 --2- 172.21.15.46:0/3062105325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 0x7f4d341615e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009e50 tx=0x7f4d3002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41e9a6b79ddb61c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.844+0000 7f4d391d5640 1 -- 172.21.15.46:0/3062105325 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30036070 con 0x7f4d34164c70 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.844+0000 7f4d391d5640 1 -- 172.21.15.46:0/3062105325 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d30031600 con 0x7f4d34164c70 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.844+0000 7f4d391d5640 1 -- 172.21.15.46:0/3062105325 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d300319d0 con 0x7f4d34164c70 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.844+0000 7f4d3c43b640 1 -- 172.21.15.46:0/3062105325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 msgr2=0x7f4d341615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.844+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/3062105325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 0x7f4d341615e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009e50 tx=0x7f4d3002a680 comp rx=0 tx=0).stop 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.845+0000 7f4d3c43b640 1 -- 172.21.15.46:0/3062105325 shutdown_connections 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.845+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/3062105325 >> v2:172.21.15.88:3300/0 conn(0x7f4d3415dfe0 0x7f4d340d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.845+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/3062105325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d34164c70 0x7f4d341615e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.845+0000 7f4d3c43b640 1 -- 172.21.15.46:0/3062105325 >> 172.21.15.46:0/3062105325 conn(0x7f4d34073db0 msgr2=0x7f4d340741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.845+0000 7f4d3c43b640 1 -- 172.21.15.46:0/3062105325 shutdown_connections 2024-05-09T03:16:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.845+0000 7f4d3c43b640 1 -- 172.21.15.46:0/3062105325 wait complete. 2024-05-09T03:16:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.846+0000 7f4d3c43b640 1 Processor -- start 2024-05-09T03:16:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.846+0000 7f4d3c43b640 1 -- start start 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3c43b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340d2480 0x7f4d340cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3c43b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3415dfe0 0x7f4d340cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3c43b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 0x7f4d341635f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3c43b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3415e3f0 con 0x7f4d34164c70 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3c43b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d340d1300 con 0x7f4d3415dfe0 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3c43b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d340d45c0 con 0x7f4d340d2480 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3a9d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 0x7f4d341635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3a9d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 0x7f4d341635f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60086/0 (socket says 172.21.15.46:60086) 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3415dfe0 0x7f4d340cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3a9d8640 1 -- 172.21.15.46:0/1728515986 learned_addr learned my addr 172.21.15.46:0/1728515986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3a1d7640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340d2480 0x7f4d340cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 -- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340d2480 msgr2=0x7f4d340cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340d2480 0x7f4d340cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 -- 172.21.15.46:0/1728515986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 msgr2=0x7f4d341635f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 0x7f4d341635f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 -- 172.21.15.46:0/1728515986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d340cdd10 con 0x7f4d3415dfe0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d3a9d8640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 0x7f4d341635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.848+0000 7f4d399d6640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3415dfe0 0x7f4d340cd7d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e840 tx=0x7f4d2400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.849+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24017070 con 0x7f4d3415dfe0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.849+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2400c940 con 0x7f4d3415dfe0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.849+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d341614e0 con 0x7f4d3415dfe0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.849+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2400cd30 con 0x7f4d3415dfe0 2024-05-09T03:16:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.849+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d34161aa0 con 0x7f4d3415dfe0 2024-05-09T03:16:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.850+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3408cfd0 con 0x7f4d3415dfe0 2024-05-09T03:16:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.851+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4d2401a070 con 0x7f4d3415dfe0 2024-05-09T03:16:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.851+0000 7f4d2affd640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d180762d0 0x7f4d180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.851+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f4d24097da0 con 0x7f4d3415dfe0 2024-05-09T03:16:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.851+0000 7f4d3a1d7640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d180762d0 0x7f4d180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:44.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.852+0000 7f4d3a1d7640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d180762d0 0x7f4d180787e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d30033000 tx=0x7f4d30009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:44.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:44.855+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4d3408cfd0 con 0x7f4d3415dfe0 2024-05-09T03:16:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.083+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d3406a270 con 0x7f4d3415dfe0 2024-05-09T03:16:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.084+0000 7f4d2affd640 1 -- 172.21.15.46:0/1728515986 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+4342 (secure 0 0 0) 0x7f4d24060e90 con 0x7f4d3415dfe0 2024-05-09T03:16:45.085 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:45.085 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":119,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:44.302572+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 119 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d180762d0 msgr2=0x7f4d180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d180762d0 0x7f4d180787e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4d30033000 tx=0x7f4d30009000 comp rx=0 tx=0).stop 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3415dfe0 msgr2=0x7f4d340cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3415dfe0 0x7f4d340cd7d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400e840 tx=0x7f4d2400ed00 comp rx=0 tx=0).stop 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 shutdown_connections 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d180762d0 0x7f4d180787e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d34164c70 0x7f4d341635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3415dfe0 0x7f4d340cd7d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 --2- 172.21.15.46:0/1728515986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340d2480 0x7f4d340cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 >> 172.21.15.46:0/1728515986 conn(0x7f4d34073db0 msgr2=0x7f4d340ba3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 shutdown_connections 2024-05-09T03:16:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.086+0000 7f4d3c43b640 1 -- 172.21.15.46:0/1728515986 wait complete. 2024-05-09T03:16:45.099 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:16:45.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:45.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.218+0000 7f9eec405640 1 Processor -- start 2024-05-09T03:16:45.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.218+0000 7f9eec405640 1 -- start start 2024-05-09T03:16:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9eec405640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9ee40b81b0 0x7f9ee40d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9eec405640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee4163f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9eec405640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee40d15b0 con 0x7f9ee41615f0 2024-05-09T03:16:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9eec405640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee4160b10 con 0x7f9ee40b81b0 2024-05-09T03:16:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9eec405640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee4160df0 con 0x7f9ee40d2b20 2024-05-09T03:16:45.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9ee40b81b0 0x7f9ee40d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9ee40b81b0 0x7f9ee40d25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60090/0 (socket says 172.21.15.46:60090) 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 -- 172.21.15.46:0/170046199 learned_addr learned my addr 172.21.15.46:0/170046199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9eea9a2640 1 --2- 172.21.15.46:0/170046199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee4163f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 -- 172.21.15.46:0/170046199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 msgr2=0x7f9ee4163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 --2- 172.21.15.46:0/170046199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee4163f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 -- 172.21.15.46:0/170046199 >> v1:172.21.15.88:6789/0 conn(0x7f9ee41615f0 legacy=0x7f9ee40d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.220+0000 7f9ee99a0640 1 -- 172.21.15.46:0/170046199 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee41610d0 con 0x7f9ee40b81b0 2024-05-09T03:16:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.221+0000 7f9ee99a0640 1 --2- 172.21.15.46:0/170046199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee40d25e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009870 tx=0x7f9ed802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebfa30b73d67234 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.221+0000 7f9ee919f640 1 -- 172.21.15.46:0/170046199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8037070 con 0x7f9ee40b81b0 2024-05-09T03:16:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.221+0000 7f9ee919f640 1 -- 172.21.15.46:0/170046199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed8032440 con 0x7f9ee40b81b0 2024-05-09T03:16:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.221+0000 7f9ee919f640 1 -- 172.21.15.46:0/170046199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8032810 con 0x7f9ee40b81b0 2024-05-09T03:16:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 -- 172.21.15.46:0/170046199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 msgr2=0x7f9ee40d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 --2- 172.21.15.46:0/170046199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee40d25e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009870 tx=0x7f9ed802a270 comp rx=0 tx=0).stop 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 -- 172.21.15.46:0/170046199 shutdown_connections 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 --2- 172.21.15.46:0/170046199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee4163f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 --2- 172.21.15.46:0/170046199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee40d25e0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 -- 172.21.15.46:0/170046199 >> 172.21.15.46:0/170046199 conn(0x7f9ee406e190 msgr2=0x7f9ee406e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 -- 172.21.15.46:0/170046199 shutdown_connections 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.222+0000 7f9eec405640 1 -- 172.21.15.46:0/170046199 wait complete. 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.224+0000 7f9eec405640 1 Processor -- start 2024-05-09T03:16:45.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.224+0000 7f9eec405640 1 -- start start 2024-05-09T03:16:45.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eec405640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee41eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eec405640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee415af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eec405640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ee41615f0 0x7f9ee41450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eec405640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee40d1890 con 0x7f9ee40b81b0 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eec405640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee40b85c0 con 0x7f9ee41615f0 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eec405640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee40d2f30 con 0x7f9ee40d2b20 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eea1a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee41eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eea1a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee41eb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60104/0 (socket says 172.21.15.46:60104) 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.225+0000 7f9eea1a1640 1 -- 172.21.15.46:0/2450069969 learned_addr learned my addr 172.21.15.46:0/2450069969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea9a2640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ee41615f0 0x7f9ee41450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9ee99a0640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee415af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea1a1640 1 -- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 msgr2=0x7f9ee415af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea1a1640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee415af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea1a1640 1 -- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ee41615f0 msgr2=0x7f9ee41450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea1a1640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ee41615f0 0x7f9ee41450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea1a1640 1 -- 172.21.15.46:0/2450069969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee41eb800 con 0x7f9ee40b81b0 2024-05-09T03:16:45.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea9a2640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ee41615f0 0x7f9ee41450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9ee99a0640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee415af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eea1a1640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee41eb2c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c430 tx=0x7f9edc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc017070 con 0x7f9ee40b81b0 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9edc0052d0 con 0x7f9ee40b81b0 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc003830 con 0x7f9ee40b81b0 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee415b460 con 0x7f9ee40b81b0 2024-05-09T03:16:45.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.226+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee415bc50 con 0x7f9ee40b81b0 2024-05-09T03:16:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.227+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee406a270 con 0x7f9ee40b81b0 2024-05-09T03:16:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.228+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9edc01a070 con 0x7f9ee40b81b0 2024-05-09T03:16:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.228+0000 7f9ee2ffd640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ed00762d0 0x7f9ed00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.228+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f9edc096c20 con 0x7f9ee40b81b0 2024-05-09T03:16:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.229+0000 7f9ee99a0640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ed00762d0 0x7f9ed00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.229+0000 7f9ee99a0640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ed00762d0 0x7f9ed00787e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8034000 tx=0x7f9ed802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:45.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.233+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9ee41eb800 con 0x7f9ee40b81b0 2024-05-09T03:16:45.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.469+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ee40b5480 con 0x7f9ee40b81b0 2024-05-09T03:16:45.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.470+0000 7f9ee2ffd640 1 -- 172.21.15.46:0/2450069969 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+5168 (secure 0 0 0) 0x7f9edc0602a0 con 0x7f9ee40b81b0 2024-05-09T03:16:45.471 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:45.471 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":120,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":7806,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2566829830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2566829830},{"type":"v1","addr":"172.21.15.46:6834","nonce":2566829830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:44.302572+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:45.471 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 120 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ed00762d0 msgr2=0x7f9ed00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ed00762d0 0x7f9ed00787e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8034000 tx=0x7f9ed802e000 comp rx=0 tx=0).stop 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 msgr2=0x7f9ee41eb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee41eb2c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c430 tx=0x7f9edc00c8f0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 shutdown_connections 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ee41615f0 0x7f9ee41450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9ee40d2b20 0x7f9ee415af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9ed00762d0 0x7f9ed00787e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 --2- 172.21.15.46:0/2450069969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ee40b81b0 0x7f9ee41eb2c0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 >> 172.21.15.46:0/2450069969 conn(0x7f9ee406e190 msgr2=0x7f9ee4162390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 shutdown_connections 2024-05-09T03:16:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.472+0000 7f9eec405640 1 -- 172.21.15.46:0/2450069969 wait complete. 2024-05-09T03:16:45.484 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 200.00000001' led to DAMAGED state 2024-05-09T03:16:45.484 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_openfiles.0' 2024-05-09T03:16:45.484 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:45.484 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:45.502 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:16:45.502 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:45.502 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:45.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:45.503 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:16:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:16:45.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:45.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.720+0000 7fb4be586640 1 Processor -- start 2024-05-09T03:16:45.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.720+0000 7fb4be586640 1 -- start start 2024-05-09T03:16:45.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4be586640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4b80d3180 0x7fb4b81642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4be586640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b8164900 con 0x7fb4b80bb770 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4be586640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b8164be0 con 0x7fb4b8163eb0 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4be586640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b80d4bb0 con 0x7fb4b80d3180 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4bcb23640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4b80d3180 0x7fb4b81642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4b7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb4b80bb770 0x7fb4b80d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46612/0 (socket says 172.21.15.46:46612) 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.721+0000 7fb4b7fff640 1 -- 172.21.15.46:0/1325419562 learned_addr learned my addr 172.21.15.46:0/1325419562 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1325419562 >> v1:172.21.15.88:6789/0 conn(0x7fb4b8163eb0 legacy=0x7fb4b80d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1325419562 >> v1:172.21.15.46:6789/0 conn(0x7fb4b80bb770 legacy=0x7fb4b80d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1325419562 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b80d4840 con 0x7fb4b80d3180 2024-05-09T03:16:45.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4bcb23640 1 --2- 172.21.15.46:0/1325419562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b80d3180 0x7fb4b81642c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80d4d30 tx=0x7fb4b002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937d57aac61a3b43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4b6ffd640 1 -- 172.21.15.46:0/1325419562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b002ae60 con 0x7fb4b80d3180 2024-05-09T03:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4b6ffd640 1 -- 172.21.15.46:0/1325419562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b0007740 con 0x7fb4b80d3180 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.722+0000 7fb4b6ffd640 1 -- 172.21.15.46:0/1325419562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0007b10 con 0x7fb4b80d3180 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 -- 172.21.15.46:0/1325419562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b80d3180 msgr2=0x7fb4b81642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 --2- 172.21.15.46:0/1325419562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b80d3180 0x7fb4b81642c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fb4b80d4d30 tx=0x7fb4b002a640 comp rx=0 tx=0).stop 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 -- 172.21.15.46:0/1325419562 shutdown_connections 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 --2- 172.21.15.46:0/1325419562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b80d3180 0x7fb4b81642c0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 -- 172.21.15.46:0/1325419562 >> 172.21.15.46:0/1325419562 conn(0x7fb4b806e370 msgr2=0x7fb4b806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 -- 172.21.15.46:0/1325419562 shutdown_connections 2024-05-09T03:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.723+0000 7fb4be586640 1 -- 172.21.15.46:0/1325419562 wait complete. 2024-05-09T03:16:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.725+0000 7fb4be586640 1 Processor -- start 2024-05-09T03:16:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.725+0000 7fb4be586640 1 -- start start 2024-05-09T03:16:45.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.726+0000 7fb4be586640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4b80bb770 0x7fb4b8153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.726+0000 7fb4be586640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4b8163eb0 0x7fb4b81f9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.726+0000 7fb4be586640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 0x7fb4b80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.726+0000 7fb4be586640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b80d0d40 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.726+0000 7fb4be586640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b80d0bc0 con 0x7fb4b80bb770 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.726+0000 7fb4be586640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b80b9210 con 0x7fb4b8163eb0 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 0x7fb4b80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 0x7fb4b80cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60128/0 (socket says 172.21.15.46:60128) 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1979537225 learned_addr learned my addr 172.21.15.46:0/1979537225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4b7fff640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4b80bb770 0x7fb4b8153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4b77fe640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4b8163eb0 0x7fb4b81f9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4b8163eb0 msgr2=0x7fb4b81f9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4b8163eb0 0x7fb4b81f9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4b80bb770 msgr2=0x7fb4b8153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4b80bb770 0x7fb4b8153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 -- 172.21.15.46:0/1979537225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b81fa5e0 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4bcb23640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 0x7fb4b80cd260 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0031930 tx=0x7fb4b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:45.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0040070 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4b002ae60 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b80cc4e0 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0045460 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.727+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b80cca60 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.728+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b80d1c80 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.729+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb4b0039040 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.730+0000 7fb4b4ff9640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4a40762d0 0x7fb4a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:45.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.730+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7fb4b00b8ea0 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.730+0000 7fb4b7fff640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4a40762d0 0x7fb4a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:45.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.730+0000 7fb4b7fff640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4a40762d0 0x7fb4a40787e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8000f60 tx=0x7fb4a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:45.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.734+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb4b80d1c80 con 0x7fb4b81fa1d0 2024-05-09T03:16:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:45.970+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb4b806a270 con 0x7fb4b81fa1d0 2024-05-09T03:16:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.313+0000 7fb4b4ff9640 1 -- 172.21.15.46:0/1979537225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v121) ==== 161+0+0 (secure 0 0 0) 0x7fb4b81fa5e0 con 0x7fb4b81fa1d0 2024-05-09T03:16:46.314 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4a40762d0 msgr2=0x7fb4a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4a40762d0 0x7fb4a40787e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb4a8000f60 tx=0x7fb4a8009040 comp rx=0 tx=0).stop 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 msgr2=0x7fb4b80cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 0x7fb4b80cd260 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0031930 tx=0x7fb4b0004a40 comp rx=0 tx=0).stop 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 shutdown_connections 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4a40762d0 0x7fb4a40787e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4b81fa1d0 0x7fb4b80cd260 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4b8163eb0 0x7fb4b81f9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 --2- 172.21.15.46:0/1979537225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4b80bb770 0x7fb4b8153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 >> 172.21.15.46:0/1979537225 conn(0x7fb4b806e370 msgr2=0x7fb4b8163560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 shutdown_connections 2024-05-09T03:16:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.315+0000 7fb4be586640 1 -- 172.21.15.46:0/1979537225 wait complete. 2024-05-09T03:16:46.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.444+0000 7f13b3757640 1 Processor -- start 2024-05-09T03:16:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.444+0000 7f13b3757640 1 -- start start 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.445+0000 7f13b3757640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13ac0b5140 0x7f13ac0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b3757640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ac0b80b0 con 0x7f13ac0d3fc0 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b3757640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ac0b5550 con 0x7f13ac0b7ca0 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b3757640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f13ac0d3480 con 0x7f13ac0b5140 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b14f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13ac0b5140 0x7f13ac0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b1cf4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13ac0d3fc0 0x7f13ac0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34228/0 (socket says 172.21.15.46:34228) 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b1cf4640 1 -- 172.21.15.46:0/3014085747 learned_addr learned my addr 172.21.15.46:0/3014085747 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b14f3640 1 -- 172.21.15.46:0/3014085747 >> v1:172.21.15.88:6789/0 conn(0x7f13ac0b7ca0 legacy=0x7f13ac0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b14f3640 1 -- 172.21.15.46:0/3014085747 >> v1:172.21.15.46:6789/0 conn(0x7f13ac0d3fc0 legacy=0x7f13ac0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:46.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.446+0000 7f13b14f3640 1 -- 172.21.15.46:0/3014085747 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac0d44e0 con 0x7f13ac0b5140 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.447+0000 7f13b14f3640 1 --2- 172.21.15.46:0/3014085747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac0b7760 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009d90 tx=0x7f13a802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf715e1bd9f059a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.447+0000 7f13a3fff640 1 -- 172.21.15.46:0/3014085747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8037070 con 0x7f13ac0b5140 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.447+0000 7f13a3fff640 1 -- 172.21.15.46:0/3014085747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a8007740 con 0x7f13ac0b5140 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.447+0000 7f13a3fff640 1 -- 172.21.15.46:0/3014085747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8007b10 con 0x7f13ac0b5140 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 -- 172.21.15.46:0/3014085747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 msgr2=0x7f13ac0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 --2- 172.21.15.46:0/3014085747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac0b7760 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009d90 tx=0x7f13a802a780 comp rx=0 tx=0).stop 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 -- 172.21.15.46:0/3014085747 shutdown_connections 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 --2- 172.21.15.46:0/3014085747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac0b7760 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 -- 172.21.15.46:0/3014085747 >> 172.21.15.46:0/3014085747 conn(0x7f13ac08d3f0 msgr2=0x7f13ac08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 -- 172.21.15.46:0/3014085747 shutdown_connections 2024-05-09T03:16:46.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.448+0000 7f13b3757640 1 -- 172.21.15.46:0/3014085747 wait complete. 2024-05-09T03:16:46.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.450+0000 7f13b3757640 1 Processor -- start 2024-05-09T03:16:46.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.450+0000 7f13b3757640 1 -- start start 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b3757640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac145e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b3757640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13ac0b7ca0 0x7f13ac0c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b3757640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13ac0d3fc0 0x7f13ac15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b3757640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac0d4d80 con 0x7f13ac0b5140 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b3757640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f13ac0d4aa0 con 0x7f13ac0b7ca0 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b3757640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f13ac0d5580 con 0x7f13ac0d3fc0 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac145e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38762/0 (socket says 172.21.15.46:38762) 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 -- 172.21.15.46:0/970148229 learned_addr learned my addr 172.21.15.46:0/970148229 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b0cf2640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13ac0b7ca0 0x7f13ac0c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b1cf4640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13ac0d3fc0 0x7f13ac15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 -- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13ac0d3fc0 msgr2=0x7f13ac15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13ac0d3fc0 0x7f13ac15b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 -- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13ac0b7ca0 msgr2=0x7f13ac0c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13ac0b7ca0 0x7f13ac0c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.451+0000 7f13b14f3640 1 -- 172.21.15.46:0/970148229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ac15a530 con 0x7f13ac0b5140 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.452+0000 7f13b14f3640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac145e10 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f13a8035040 tx=0x7f13a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.452+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8042020 con 0x7f13ac0b5140 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.452+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a8032a70 con 0x7f13ac0b5140 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.452+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8037040 con 0x7f13ac0b5140 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.452+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ac15a810 con 0x7f13ac0b5140 2024-05-09T03:16:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.452+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ac15add0 con 0x7f13ac0b5140 2024-05-09T03:16:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.453+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13ac06a270 con 0x7f13ac0b5140 2024-05-09T03:16:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.454+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f13a8045020 con 0x7f13ac0b5140 2024-05-09T03:16:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.454+0000 7f13a1ffb640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13900762d0 0x7f13900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.454+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f13a80b8990 con 0x7f13ac0b5140 2024-05-09T03:16:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.455+0000 7f13b0cf2640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13900762d0 0x7f13900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:46.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.455+0000 7f13b0cf2640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13900762d0 0x7f13900787e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f13ac148970 tx=0x7f139c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.459+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f13a804c030 con 0x7f13ac0b5140 2024-05-09T03:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:46.726+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f13ac1605b0 con 0x7f13ac0b5140 2024-05-09T03:16:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.318+0000 7f13a1ffb640 1 -- 172.21.15.46:0/970148229 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v122) ==== 101+0+0 (secure 0 0 0) 0x7f13a80819d0 con 0x7f13ac0b5140 2024-05-09T03:16:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:16:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13900762d0 msgr2=0x7f13900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13900762d0 0x7f13900787e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f13ac148970 tx=0x7f139c009040 comp rx=0 tx=0).stop 2024-05-09T03:16:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 msgr2=0x7f13ac145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac145e10 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f13a8035040 tx=0x7f13a8004a40 comp rx=0 tx=0).stop 2024-05-09T03:16:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 shutdown_connections 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f13ac0d3fc0 0x7f13ac15b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13ac0b7ca0 0x7f13ac0c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f13900762d0 0x7f13900787e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 --2- 172.21.15.46:0/970148229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13ac0b5140 0x7f13ac145e10 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 >> 172.21.15.46:0/970148229 conn(0x7f13ac08d3f0 msgr2=0x7f13ac164930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 shutdown_connections 2024-05-09T03:16:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.321+0000 7f13b3757640 1 -- 172.21.15.46:0/970148229 wait complete. 2024-05-09T03:16:47.336 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.397+0000 7fe00511ef40 1 Processor -- start 2024-05-09T03:16:47.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.397+0000 7fe00511ef40 1 -- start start 2024-05-09T03:16:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fe00511ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fe00511ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994c4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fe00511ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e994c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fe00511ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555e9939df50 con 0x555e993f42a0 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fe00511ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555e993a1f90 con 0x555e99437630 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fe00511ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555e99426f20 con 0x555e994c5c90 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38766/0 (socket says 172.21.15.46:38766) 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 -- 172.21.15.46:0/2417887160 learned_addr learned my addr 172.21.15.46:0/2417887160 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffdacc640 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffeace640 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e994c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 -- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 msgr2=0x555e994c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e994c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 -- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 msgr2=0x555e994c4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994c4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.398+0000 7fdffe2cd640 1 -- 172.21.15.46:0/2417887160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e99421fb0 con 0x555e993f42a0 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c2a60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fdff00016b0 tx=0x7fdff0001b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08dddd8418c886 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fdffcaca640 1 -- 172.21.15.46:0/2417887160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0010790 con 0x555e993f42a0 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fdffcaca640 1 -- 172.21.15.46:0/2417887160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff0010950 con 0x555e993f42a0 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fdffcaca640 1 -- 172.21.15.46:0/2417887160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0010d40 con 0x555e993f42a0 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fe00511ef40 1 -- 172.21.15.46:0/2417887160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 msgr2=0x555e994c2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fe00511ef40 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c2a60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fdff00016b0 tx=0x7fdff0001b70 comp rx=0 tx=0).stop 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.399+0000 7fe00511ef40 1 -- 172.21.15.46:0/2417887160 shutdown_connections 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e994c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 --2- 172.21.15.46:0/2417887160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c2a60 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 -- 172.21.15.46:0/2417887160 >> 172.21.15.46:0/2417887160 conn(0x555e99319ac0 msgr2=0x555e993e34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 -- 172.21.15.46:0/2417887160 shutdown_connections 2024-05-09T03:16:47.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 -- 172.21.15.46:0/2417887160 wait complete. 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.400+0000 7fe00511ef40 1 Processor -- start 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.401+0000 7fe00511ef40 1 -- start start 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fe00511ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fe00511ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994206a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fe00511ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e99420be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fe00511ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555e994cd230 con 0x555e993f42a0 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fe00511ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555e994ccc70 con 0x555e99437630 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fe00511ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555e994c2fa0 con 0x555e994c5c90 2024-05-09T03:16:47.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38778/0 (socket says 172.21.15.46:38778) 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 learned_addr learned my addr 172.21.15.46:0/491675033 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffeace640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e99420be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 msgr2=0x555e99420be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e99420be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 msgr2=0x555e994206a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994206a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e994c2630 con 0x555e993f42a0 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffeace640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555e994c5c90 0x555e99420be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.402+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555e99437630 0x555e994206a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.403+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c1cc0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fdff0010de0 tx=0x7fdff0001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.403+0000 7fdfef7fe640 1 -- 172.21.15.46:0/491675033 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0006230 con 0x555e993f42a0 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.403+0000 7fdfef7fe640 1 -- 172.21.15.46:0/491675033 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff00063d0 con 0x555e993f42a0 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.403+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555e9943d2a0 con 0x555e993f42a0 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.403+0000 7fdfef7fe640 1 -- 172.21.15.46:0/491675033 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0011870 con 0x555e993f42a0 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.403+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e9943bbd0 con 0x555e993f42a0 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.405+0000 7fdfef7fe640 1 -- 172.21.15.46:0/491675033 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdff0019040 con 0x555e993f42a0 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.405+0000 7fdfef7fe640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfe0074cb0 0x7fdfe00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.405+0000 7fdfef7fe640 1 -- 172.21.15.46:0/491675033 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7fdff0097a50 con 0x555e993f42a0 2024-05-09T03:16:47.408 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.405+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555e994cafe0 0x555e99570500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.405+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfe0074cb0 0x7fdfe00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.405+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995648c0 con 0x555e994cafe0 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.406+0000 7fdffeace640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555e994cafe0 0x555e99570500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.406+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfe0074cb0 0x7fdfe00771c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdff4056000 tx=0x7fdff4050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.406+0000 7fdffeace640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555e994cafe0 0x555e99570500 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.408+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v90'164 uv138 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.408+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e994c2a60 con 0x555e994cafe0 2024-05-09T03:16:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.409+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v90'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.409+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99570a40 con 0x555e994cafe0 2024-05-09T03:16:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.411+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v90'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.411 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.411+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99571df0 con 0x555e994cafe0 2024-05-09T03:16:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.412+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v90'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.412+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99572190 con 0x555e994cafe0 2024-05-09T03:16:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.413+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v90'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.413+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99570de0 con 0x555e994cafe0 2024-05-09T03:16:47.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.415+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v90'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.415+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.416+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v90'170 uv144 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.416+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.418+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v90'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.418+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.419+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v90'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.419+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.421+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v90'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.421+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.422+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v90'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.422+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.423+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v90'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.423+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.425+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v90'176 uv151 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.425+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.426+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v90'177 uv177 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.426+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.428 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:47.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.427+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v90'178 uv178 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.427+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.429+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v90'179 uv179 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.429+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.430+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v90'180 uv180 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.430+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.431+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v90'181 uv181 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.431+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.433+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v90'182 uv182 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.433+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.434+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v90'183 uv157 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.434+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.436+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v90'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.436+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.437+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v90'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.438 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:47.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.437+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.438+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v90'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.438+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.440+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v90'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.440+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.441+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v90'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.441+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.442+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v90'189 uv163 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.442+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.443+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v90'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.443+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.445+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v90'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.445 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.445+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.446+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v90'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.446+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.447+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v90'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.447+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e994cafe0 2024-05-09T03:16:47.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.448+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v90'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.448+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555e99572530 0x555e99574950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.448+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995711d0 con 0x555e99572530 2024-05-09T03:16:47.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.449+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555e99572530 0x555e99574950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.449+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555e99572530 0x555e99574950 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.451+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v90'129 uv113 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.451+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99574e90 con 0x555e99572530 2024-05-09T03:16:47.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.452+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v90'130 uv130 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.452+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99575230 con 0x555e99572530 2024-05-09T03:16:47.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.453+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v90'131 uv131 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.454 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:47.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.453+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995755d0 con 0x555e99572530 2024-05-09T03:16:47.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.455+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v90'132 uv132 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.455+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99575970 con 0x555e99572530 2024-05-09T03:16:47.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.456+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v90'133 uv133 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.456+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99575d10 con 0x555e99572530 2024-05-09T03:16:47.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.458+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v90'134 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.458+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e995760b0 con 0x555e99572530 2024-05-09T03:16:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.459+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v90'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.459+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.460+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v90'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.461 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:47.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.460+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.462+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v90'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.462+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.463+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v90'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.463+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.466 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:47.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.466+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v90'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.466+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99579eb0 con 0x555e99572530 2024-05-09T03:16:47.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.467+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [write 0~14895] v90'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.467+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99579eb0 con 0x555e99572530 2024-05-09T03:16:47.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.468+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v90'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.468+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.470+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 606.00000000 [create] v90'142 uv127 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.470+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.471+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 606.00000000 [delete] v90'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.471+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.473 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:47.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.472+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v90'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.472+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99577460 con 0x555e99572530 2024-05-09T03:16:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.474+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v90'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.474+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99577460 con 0x555e99572530 2024-05-09T03:16:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.475+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v90'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.475+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e99572530 2024-05-09T03:16:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.476+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v90'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e99572530 2024-05-09T03:16:47.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.476+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e994cafe0 2024-05-09T03:16:47.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.477+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(51 607.00000000 [create] v90'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.477+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99576450 con 0x555e994cafe0 2024-05-09T03:16:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.478+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(52 607.00000000 [delete] v90'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.478+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e994cafe0 2024-05-09T03:16:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.479+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(53 607.00000000 [create] v90'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.480 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.480+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e994cafe0 2024-05-09T03:16:47.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.481+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v90'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.481+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e994cafe0 2024-05-09T03:16:47.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.482+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v90'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.482+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99576450 con 0x555e994cafe0 2024-05-09T03:16:47.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.483+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v90'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.483+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555e99576d70 0x555e99579190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.483+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e995796d0 con 0x555e99576d70 2024-05-09T03:16:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.483+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555e99576d70 0x555e99579190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.483+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555e99576d70 0x555e99579190 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.484+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.484+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99579b70 con 0x555e99576d70 2024-05-09T03:16:47.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.485+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v90'28 uv28 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.485+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a070 con 0x555e99576d70 2024-05-09T03:16:47.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.486+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v90'29 uv29 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.487 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:47.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.486+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99563fe0 con 0x555e99576d70 2024-05-09T03:16:47.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.487+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v90'30 uv30 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.487+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a9a0 con 0x555e994cafe0 2024-05-09T03:16:47.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.489+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(61 608.00000000 [create] v90'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.489+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e9957a9a0 con 0x555e994cafe0 2024-05-09T03:16:47.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.490+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(62 608.00000000 [delete] v90'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.490+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a9a0 con 0x555e994cafe0 2024-05-09T03:16:47.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.493+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(63 608.00000000 [create] v90'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.493 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:47.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.493+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957b9f0 con 0x555e994cafe0 2024-05-09T03:16:47.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.494+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v90'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.494+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957b9b0 con 0x555e994cafe0 2024-05-09T03:16:47.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.495+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v90'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.495+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a9a0 con 0x555e994cafe0 2024-05-09T03:16:47.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.496+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v90'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.496+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x555e9957b070 0x555e9957b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.496+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957dde0 con 0x555e9957b070 2024-05-09T03:16:47.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.497+0000 7fdffeace640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x555e9957b070 0x555e9957b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.497+0000 7fdffeace640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x555e9957b070 0x555e9957b480 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.499+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(67 604.00000000 [create] v90'59 uv53 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.499+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e9957dde0 con 0x555e9957b070 2024-05-09T03:16:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.500+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(68 604.00000000 [delete] v90'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.500+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957dde0 con 0x555e9957b070 2024-05-09T03:16:47.501 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.501+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(69 604.00000000 [create] v90'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.501+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957f1e0 con 0x555e9957b070 2024-05-09T03:16:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.502+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v90'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.502+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957f1e0 con 0x555e9957b070 2024-05-09T03:16:47.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.503+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v90'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.503+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957e1d0 con 0x555e9957b070 2024-05-09T03:16:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.505+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v90'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.505+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957e1d0 con 0x555e9957b070 2024-05-09T03:16:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.506+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(73 500.00000000 [create] v90'65 uv58 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.506+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e9957e1d0 con 0x555e9957b070 2024-05-09T03:16:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.507+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(74 500.00000000 [delete] v90'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.507+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957e1d0 con 0x555e9957b070 2024-05-09T03:16:47.509 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.508+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(75 500.00000000 [create] v90'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.508+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957e1d0 con 0x555e9957b070 2024-05-09T03:16:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.510+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v90'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.510+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957e1d0 con 0x555e9957b070 2024-05-09T03:16:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.511+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v90'69 uv69 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.511+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x555e9957e6e0 0x555e99580b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.511+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581160 con 0x555e9957e6e0 2024-05-09T03:16:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.511+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x555e9957e6e0 0x555e99580b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.512+0000 7fdffe2cd640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x555e9957e6e0 0x555e99580b00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.513+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(78 mds_snaptable [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e9957e6e0 2024-05-09T03:16:47.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.513+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99581220 con 0x555e9957e6e0 2024-05-09T03:16:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.514+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(79 mds_snaptable [delete] v90'28 uv28 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e9957e6e0 2024-05-09T03:16:47.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.514+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e9957e6e0 2024-05-09T03:16:47.516 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.515+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(80 mds_snaptable [create] v90'29 uv29 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e9957e6e0 2024-05-09T03:16:47.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.515+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e9957e6e0 2024-05-09T03:16:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.516+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v90'30 uv30 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e9957e6e0 2024-05-09T03:16:47.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.517+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e9957e6e0 2024-05-09T03:16:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.518+0000 7fdffe2cd640 1 -- 172.21.15.46:0/491675033 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v90'31 uv31 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff002b070 con 0x555e9957e6e0 2024-05-09T03:16:47.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.518+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e994cafe0 2024-05-09T03:16:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.519+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(83 605.00000000 [create] v90'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.519+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99581640 con 0x555e994cafe0 2024-05-09T03:16:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.520+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(84 605.00000000 [delete] v90'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.520+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e994cafe0 2024-05-09T03:16:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.521+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(85 605.00000000 [create] v90'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.521 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.521+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99582650 con 0x555e994cafe0 2024-05-09T03:16:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.522+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v90'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.522+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99582650 con 0x555e994cafe0 2024-05-09T03:16:47.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.523+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v90'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.523+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e994cafe0 2024-05-09T03:16:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.524+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v90'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e994cafe0 2024-05-09T03:16:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.524+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581640 con 0x555e99576d70 2024-05-09T03:16:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.525+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(89 mds0_inotable [create] v90'54 uv48 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.525+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e995819e0 con 0x555e99576d70 2024-05-09T03:16:47.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.526+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(90 mds0_inotable [delete] v90'55 uv55 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.526+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99581d80 con 0x555e99576d70 2024-05-09T03:16:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.527+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(91 mds0_inotable [create] v90'56 uv56 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.528 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.527+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a520 con 0x555e99576d70 2024-05-09T03:16:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.528+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v90'57 uv57 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.528+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a520 con 0x555e99576d70 2024-05-09T03:16:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.529+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v90'58 uv58 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.529+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a520 con 0x555e99576d70 2024-05-09T03:16:47.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.530+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(94 100.00000000 [create] v90'59 uv53 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.530+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e9957a520 con 0x555e99576d70 2024-05-09T03:16:47.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.531+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(95 100.00000000 [delete] v90'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.531+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a520 con 0x555e99576d70 2024-05-09T03:16:47.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.532+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(96 100.00000000 [create] v90'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.533 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:47.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.532+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e9957a520 con 0x555e99576d70 2024-05-09T03:16:47.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.533+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v90'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.533+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99584f70 con 0x555e99576d70 2024-05-09T03:16:47.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.535+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v90'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.535+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x555e99582ae0 0x555e99584f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.535+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585630 con 0x555e99582ae0 2024-05-09T03:16:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.535+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x555e99582ae0 0x555e99584f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.535+0000 7fdffdacc640 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x555e99582ae0 0x555e99584f00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.536+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v90'26 uv25 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99582ae0 2024-05-09T03:16:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.536+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99585630 con 0x555e99582ae0 2024-05-09T03:16:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.537+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v90'27 uv27 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99582ae0 2024-05-09T03:16:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.537+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585630 con 0x555e99582ae0 2024-05-09T03:16:47.539 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.538+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v90'28 uv28 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99582ae0 2024-05-09T03:16:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.538+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e99582ae0 2024-05-09T03:16:47.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.539+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v90'29 uv29 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99582ae0 2024-05-09T03:16:47.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.539+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e99576d70 2024-05-09T03:16:47.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.541+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v90'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.541+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e99576d70 2024-05-09T03:16:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.542+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v90'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.542+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e99576d70 2024-05-09T03:16:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.543+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v90'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.544 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.543+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99586a30 con 0x555e99576d70 2024-05-09T03:16:47.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.544+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v90'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.544+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99586a30 con 0x555e99576d70 2024-05-09T03:16:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.546+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v90'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.546+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e99576d70 2024-05-09T03:16:47.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.547+0000 7fdffdacc640 1 -- 172.21.15.46:0/491675033 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v90'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff405c040 con 0x555e99576d70 2024-05-09T03:16:47.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.547+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.548+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.548+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.550+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.550+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.552 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.552+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.552+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.555+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.555+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.557+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.557+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.560+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v90'69 uv61 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.560+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.561+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v90'70 uv70 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.561+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.562 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.562+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v90'71 uv71 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.562+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.565+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v90'72 uv72 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.565+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.567+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v90'73 uv73 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.567+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.569+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v90'74 uv68 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.569+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.571+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v90'75 uv75 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.571+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.574 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.573+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v90'76 uv76 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.573+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99586a30 con 0x555e9957b070 2024-05-09T03:16:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.576+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v90'77 uv77 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.576+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99586a30 con 0x555e9957b070 2024-05-09T03:16:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.578+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v90'78 uv78 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.578+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585a20 con 0x555e9957b070 2024-05-09T03:16:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.579+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v90'79 uv79 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.579+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x555e99585ed0 con 0x555e9957b070 2024-05-09T03:16:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v90'80 uv80 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fdff8009040 con 0x555e9957b070 2024-05-09T03:16:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555e994cafe0 msgr2=0x555e99570500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555e994cafe0 0x555e99570500 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555e99576d70 msgr2=0x555e99579190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555e99576d70 0x555e99579190 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x555e99582ae0 msgr2=0x555e99584f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x555e99582ae0 0x555e99584f00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555e99572530 msgr2=0x555e99574950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 reap_dead start 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555e99572530 0x555e99574950 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x555e9957b070 msgr2=0x555e9957b480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x555e9957b070 0x555e9957b480 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x555e9957e6e0 msgr2=0x555e99580b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x555e9957e6e0 0x555e99580b00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfe0074cb0 msgr2=0x7fdfe00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdfe0074cb0 0x7fdfe00771c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdff4056000 tx=0x7fdff4050000 comp rx=0 tx=0).stop 2024-05-09T03:16:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 msgr2=0x555e994c1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 --2- 172.21.15.46:0/491675033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555e993f42a0 0x555e994c1cc0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fdff0010de0 tx=0x7fdff0001360 comp rx=0 tx=0).stop 2024-05-09T03:16:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fdffeace640 1 -- 172.21.15.46:0/491675033 reap_dead start 2024-05-09T03:16:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 shutdown_connections 2024-05-09T03:16:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 >> 172.21.15.46:0/491675033 conn(0x555e99319ac0 msgr2=0x555e99421890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 shutdown_connections 2024-05-09T03:16:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.581+0000 7fe00511ef40 1 -- 172.21.15.46:0/491675033 wait complete. 2024-05-09T03:16:47.585 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_openfiles.0 2024-05-09T03:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.680+0000 7f10ccdd0f40 1 Processor -- start 2024-05-09T03:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.680+0000 7f10ccdd0f40 1 -- start start 2024-05-09T03:16:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10ccdd0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f01a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10ccdd0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f01b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10ccdd0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1efffa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10ccdd0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557a1f015cb0 con 0x557a1efd37e0 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10ccdd0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557a1ef7cf50 con 0x557a1f090a50 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10ccdd0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557a1efd3bf0 con 0x557a1f0ab780 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10c6898640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f01a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10c6898640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f01a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38784/0 (socket says 172.21.15.46:38784) 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.681+0000 7f10c6898640 1 -- 172.21.15.46:0/719105260 learned_addr learned my addr 172.21.15.46:0/719105260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c7099640 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1efffa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6097640 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f01b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6898640 1 -- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 msgr2=0x557a1efffa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6898640 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1efffa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6898640 1 -- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 msgr2=0x557a1f01b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6898640 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f01b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6898640 1 -- 172.21.15.46:0/719105260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a1f01a490 con 0x557a1efd37e0 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c6898640 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f01a990 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f10b8001630 tx=0x7f10b8001af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c8635eccc70fe7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c5095640 1 -- 172.21.15.46:0/719105260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8003650 con 0x557a1efd37e0 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c5095640 1 -- 172.21.15.46:0/719105260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b80037f0 con 0x557a1efd37e0 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.682+0000 7f10c5095640 1 -- 172.21.15.46:0/719105260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8003ba0 con 0x557a1efd37e0 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/719105260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 msgr2=0x557a1f01a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f01a990 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f10b8001630 tx=0x7f10b8001af0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/719105260 shutdown_connections 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1efffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f01b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/719105260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f01a990 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/719105260 >> 172.21.15.46:0/719105260 conn(0x557a1eef8ac0 msgr2=0x557a1f004800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/719105260 shutdown_connections 2024-05-09T03:16:47.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.683+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/719105260 wait complete. 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.684+0000 7f10ccdd0f40 1 Processor -- start 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.684+0000 7f10ccdd0f40 1 -- start start 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10ccdd0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f08d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10ccdd0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f0ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10ccdd0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1f0ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10ccdd0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557a1f0a9670 con 0x557a1efd37e0 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10ccdd0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557a1f005c60 con 0x557a1f090a50 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10ccdd0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557a1f091250 con 0x557a1f0ab780 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10c6898640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f08d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10c6898640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f08d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38794/0 (socket says 172.21.15.46:38794) 2024-05-09T03:16:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10c6898640 1 -- 172.21.15.46:0/2353574711 learned_addr learned my addr 172.21.15.46:0/2353574711 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10c7099640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1f0ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.685+0000 7f10c6097640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f0ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c6898640 1 -- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 msgr2=0x557a1f0ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c6898640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1f0ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c6898640 1 -- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 msgr2=0x557a1f0ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c6898640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557a1f090a50 0x557a1f0ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c6898640 1 -- 172.21.15.46:0/2353574711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a1f005140 con 0x557a1efd37e0 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c7099640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557a1f0ab780 0x557a1f0ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10c6898640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f08d060 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f10b80012e0 tx=0x7f10b8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10bffff640 1 -- 172.21.15.46:0/2353574711 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8023070 con 0x557a1efd37e0 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10bffff640 1 -- 172.21.15.46:0/2353574711 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b801e690 con 0x557a1efd37e0 2024-05-09T03:16:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10bffff640 1 -- 172.21.15.46:0/2353574711 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8018040 con 0x557a1efd37e0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557a1f0acda0 con 0x557a1efd37e0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.686+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a1f0186d0 con 0x557a1efd37e0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.688+0000 7f10bffff640 1 -- 172.21.15.46:0/2353574711 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f10b8026070 con 0x557a1efd37e0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.688+0000 7f10bffff640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10a8074cb0 0x7f10a80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.688+0000 7f10bffff640 1 -- 172.21.15.46:0/2353574711 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f10b80975a0 con 0x557a1efd37e0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.688+0000 7f10c6097640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10a8074cb0 0x7f10a80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.689+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557a1f000ea0 0x557a1f0193e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.689+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e90) -- 0x557a1f090e60 con 0x557a1f000ea0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.689+0000 7f10c7099640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557a1f000ea0 0x557a1f0193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.689+0000 7f10c6097640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10a8074cb0 0x7f10a80771c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f10b0056000 tx=0x7f10b0050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.689+0000 7f10c7099640 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557a1f000ea0 0x557a1f0193e0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.690+0000 7f10c7099640 1 -- 172.21.15.46:0/2353574711 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_openfiles.0 [delete] v90'31 uv31 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f10c0009ab0 con 0x557a1f000ea0 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.690+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557a1f000ea0 msgr2=0x557a1f0193e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.690+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557a1f000ea0 0x557a1f0193e0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.690+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10a8074cb0 msgr2=0x7f10a80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.690+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f10a8074cb0 0x7f10a80771c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f10b0056000 tx=0x7f10b0050000 comp rx=0 tx=0).stop 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 msgr2=0x557a1f08d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10ccdd0f40 1 --2- 172.21.15.46:0/2353574711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a1efd37e0 0x557a1f08d060 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f10b80012e0 tx=0x7f10b8001250 comp rx=0 tx=0).stop 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10c7099640 1 -- 172.21.15.46:0/2353574711 reap_dead start 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 shutdown_connections 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 >> 172.21.15.46:0/2353574711 conn(0x557a1eef8ac0 msgr2=0x557a1f01a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 shutdown_connections 2024-05-09T03:16:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.691+0000 7f10ccdd0f40 1 -- 172.21.15.46:0/2353574711 wait complete. 2024-05-09T03:16:47.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.812+0000 7f7a26827640 1 Processor -- start 2024-05-09T03:16:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.812+0000 7f7a26827640 1 -- start start 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a26827640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7a200d3730 0x7f7a2006e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a26827640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a2006e7e0 0x7f7a20164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a26827640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a2009d470 con 0x7f7a20164ec0 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a26827640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7a2009cfe0 con 0x7f7a200d3730 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a26827640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7a200b94a0 con 0x7f7a2006e7e0 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7a200d3730 0x7f7a2006e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a24dc4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7a20164ec0 0x7f7a200ceba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34236/0 (socket says 172.21.15.46:34236) 2024-05-09T03:16:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7a200d3730 0x7f7a2006e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38810/0 (socket says 172.21.15.46:38810) 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a24dc4640 1 -- 172.21.15.46:0/152894104 learned_addr learned my addr 172.21.15.46:0/152894104 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1f7fe640 1 --2- 172.21.15.46:0/152894104 >> v2:172.21.15.88:3300/0 conn(0x7f7a2006e7e0 0x7f7a20164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1ffff640 1 -- 172.21.15.46:0/152894104 >> v2:172.21.15.88:3300/0 conn(0x7f7a2006e7e0 msgr2=0x7f7a20164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1ffff640 1 --2- 172.21.15.46:0/152894104 >> v2:172.21.15.88:3300/0 conn(0x7f7a2006e7e0 0x7f7a20164980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1ffff640 1 -- 172.21.15.46:0/152894104 >> v1:172.21.15.46:6789/0 conn(0x7f7a20164ec0 legacy=0x7f7a200ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1effd640 1 -- 172.21.15.46:0/152894104 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134577853 0 0) 0x7f7a18001490 con 0x7f7a20164ec0 2024-05-09T03:16:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.814+0000 7f7a1ffff640 1 -- 172.21.15.46:0/152894104 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a200d11b0 con 0x7f7a200d3730 2024-05-09T03:16:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.815+0000 7f7a1ffff640 1 --2- 172.21.15.46:0/152894104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200d3730 0x7f7a2006e2a0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f7a200d4c70 tx=0x7f7a1002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db163eebb6a3e23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.815+0000 7f7a1effd640 1 -- 172.21.15.46:0/152894104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10037070 con 0x7f7a200d3730 2024-05-09T03:16:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.815+0000 7f7a1effd640 1 -- 172.21.15.46:0/152894104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a10032440 con 0x7f7a200d3730 2024-05-09T03:16:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.815+0000 7f7a1effd640 1 -- 172.21.15.46:0/152894104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1002ae60 con 0x7f7a200d3730 2024-05-09T03:16:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 -- 172.21.15.46:0/152894104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200d3730 msgr2=0x7f7a2006e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 --2- 172.21.15.46:0/152894104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200d3730 0x7f7a2006e2a0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f7a200d4c70 tx=0x7f7a1002a110 comp rx=0 tx=0).stop 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 -- 172.21.15.46:0/152894104 shutdown_connections 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 --2- 172.21.15.46:0/152894104 >> v2:172.21.15.88:3300/0 conn(0x7f7a2006e7e0 0x7f7a20164980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 --2- 172.21.15.46:0/152894104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a200d3730 0x7f7a2006e2a0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 -- 172.21.15.46:0/152894104 >> 172.21.15.46:0/152894104 conn(0x7f7a200d05f0 msgr2=0x7f7a200d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 -- 172.21.15.46:0/152894104 shutdown_connections 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.816+0000 7f7a26827640 1 -- 172.21.15.46:0/152894104 wait complete. 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.818+0000 7f7a26827640 1 Processor -- start 2024-05-09T03:16:47.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.818+0000 7f7a26827640 1 -- start start 2024-05-09T03:16:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a26827640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a2006e7e0 0x7f7a200cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a26827640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a200d3730 0x7f7a20151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a26827640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 0x7f7a20162980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a26827640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2008cd60 con 0x7f7a20164ec0 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a26827640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7a200b9160 con 0x7f7a2006e7e0 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a26827640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2008cee0 con 0x7f7a200d3730 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.819+0000 7f7a24dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 0x7f7a20162980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 0x7f7a20162980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38822/0 (socket says 172.21.15.46:38822) 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 -- 172.21.15.46:0/2584346836 learned_addr learned my addr 172.21.15.46:0/2584346836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a1f7fe640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a200d3730 0x7f7a20151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a1ffff640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a2006e7e0 0x7f7a200cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 -- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a200d3730 msgr2=0x7f7a20151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a200d3730 0x7f7a20151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 -- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a2006e7e0 msgr2=0x7f7a200cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a2006e7e0 0x7f7a200cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 -- 172.21.15.46:0/2584346836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2015f800 con 0x7f7a20164ec0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a24dc4640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 0x7f7a20162980 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800c6b0 tx=0x7f7a1800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18018070 con 0x7f7a20164ec0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a18004da0 con 0x7f7a20164ec0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a200c89c0 con 0x7f7a20164ec0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1800a880 con 0x7f7a20164ec0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.820+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a200c8c60 con 0x7f7a20164ec0 2024-05-09T03:16:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.821+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a200b7730 con 0x7f7a20164ec0 2024-05-09T03:16:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.822+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a1801b070 con 0x7f7a20164ec0 2024-05-09T03:16:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.823+0000 7f7a1cff9640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a0c0762d0 0x7f7a0c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.823+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f7a18097180 con 0x7f7a20164ec0 2024-05-09T03:16:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.823+0000 7f7a1ffff640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a0c0762d0 0x7f7a0c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.823+0000 7f7a1ffff640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a0c0762d0 0x7f7a0c0787e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7a10034000 tx=0x7f7a1002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:47.826+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7a200b7730 con 0x7f7a20164ec0 2024-05-09T03:16:48.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.056+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f7a2006a270 con 0x7f7a20164ec0 2024-05-09T03:16:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.323+0000 7f7a1cff9640 1 -- 172.21.15.46:0/2584346836 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v123) ==== 162+0+0 (secure 0 0 0) 0x7f7a18022020 con 0x7f7a20164ec0 2024-05-09T03:16:48.324 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.325+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a0c0762d0 msgr2=0x7f7a0c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.325+0000 7f7a26827640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a0c0762d0 0x7f7a0c0787e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7a10034000 tx=0x7f7a1002e000 comp rx=0 tx=0).stop 2024-05-09T03:16:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.325+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 msgr2=0x7f7a20162980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.325+0000 7f7a26827640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 0x7f7a20162980 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800c6b0 tx=0x7f7a1800cb70 comp rx=0 tx=0).stop 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 shutdown_connections 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a0c0762d0 0x7f7a0c0787e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a20164ec0 0x7f7a20162980 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a200d3730 0x7f7a20151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 --2- 172.21.15.46:0/2584346836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a2006e7e0 0x7f7a200cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 >> 172.21.15.46:0/2584346836 conn(0x7f7a200d05f0 msgr2=0x7f7a20164000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 shutdown_connections 2024-05-09T03:16:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.326+0000 7f7a26827640 1 -- 172.21.15.46:0/2584346836 wait complete. 2024-05-09T03:16:48.330 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:48.330+0000 7ff0b2f38640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-05-09T03:16:48.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:48.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.867+0000 7f4dfb5f3640 1 Processor -- start 2024-05-09T03:16:48.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.868+0000 7f4dfb5f3640 1 -- start start 2024-05-09T03:16:48.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4dfb5f3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4df40b92e0 0x7f4df40b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:48.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4dfb5f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4df40cf6b0 0x7f4df406d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:48.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4dfb5f3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df40d2b80 con 0x7f4df40d0480 2024-05-09T03:16:48.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4dfb5f3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4df40d2e60 con 0x7f4df40cf6b0 2024-05-09T03:16:48.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4dfb5f3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4df40d3140 con 0x7f4df40b92e0 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df9b90640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4df40cf6b0 0x7f4df406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df9b90640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4df40cf6b0 0x7f4df406d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38826/0 (socket says 172.21.15.46:38826) 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df9b90640 1 -- 172.21.15.46:0/3828780906 learned_addr learned my addr 172.21.15.46:0/3828780906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df8b8e640 1 --1- 172.21.15.46:0/3828780906 >> v1:172.21.15.46:6789/0 conn(0x7f4df40d0480 0x7f4df40d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34248/0 (socket says 172.21.15.46:34248) 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df938f640 1 --2- 172.21.15.46:0/3828780906 >> v2:172.21.15.88:3300/0 conn(0x7f4df40b92e0 0x7f4df40b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df9b90640 1 -- 172.21.15.46:0/3828780906 >> v2:172.21.15.88:3300/0 conn(0x7f4df40b92e0 msgr2=0x7f4df40b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.869+0000 7f4df9b90640 1 --2- 172.21.15.46:0/3828780906 >> v2:172.21.15.88:3300/0 conn(0x7f4df40b92e0 0x7f4df40b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.870+0000 7f4df9b90640 1 -- 172.21.15.46:0/3828780906 >> v1:172.21.15.46:6789/0 conn(0x7f4df40d0480 legacy=0x7f4df40d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.870+0000 7f4df3fff640 1 -- 172.21.15.46:0/3828780906 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158798697 0 0) 0x7f4df40d2b80 con 0x7f4df40d0480 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.870+0000 7f4df9b90640 1 -- 172.21.15.46:0/3828780906 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df40d3420 con 0x7f4df40cf6b0 2024-05-09T03:16:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.870+0000 7f4df938f640 1 --2- 172.21.15.46:0/3828780906 >> v2:172.21.15.88:3300/0 conn(0x7f4df40b92e0 0x7f4df40b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.870+0000 7f4df9b90640 1 --2- 172.21.15.46:0/3828780906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40cf6b0 0x7f4df406d790 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4df406d120 tx=0x7f4dec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e24234f7485ad1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.870+0000 7f4df3fff640 1 -- 172.21.15.46:0/3828780906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec037070 con 0x7f4df40cf6b0 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4df3fff640 1 -- 172.21.15.46:0/3828780906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dec032440 con 0x7f4df40cf6b0 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4df3fff640 1 -- 172.21.15.46:0/3828780906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec032810 con 0x7f4df40cf6b0 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/3828780906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40cf6b0 msgr2=0x7f4df406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/3828780906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40cf6b0 0x7f4df406d790 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4df406d120 tx=0x7f4dec02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/3828780906 shutdown_connections 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/3828780906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40cf6b0 0x7f4df406d790 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/3828780906 >> v2:172.21.15.88:3300/0 conn(0x7f4df40b92e0 0x7f4df40b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/3828780906 >> 172.21.15.46:0/3828780906 conn(0x7f4df40d3c00 msgr2=0x7f4df40d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/3828780906 shutdown_connections 2024-05-09T03:16:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.871+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/3828780906 wait complete. 2024-05-09T03:16:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.873+0000 7f4dfb5f3640 1 Processor -- start 2024-05-09T03:16:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.873+0000 7f4dfb5f3640 1 -- start start 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.874+0000 7f4dfb5f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40b92e0 0x7f4df4158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.874+0000 7f4dfb5f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4df40cf6b0 0x7f4df415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.874+0000 7f4dfb5f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 0x7f4df40ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.874+0000 7f4dfb5f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4df406e060 con 0x7f4df40d0480 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4dfb5f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4df406dee0 con 0x7f4df40b92e0 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4dfb5f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4df406e1e0 con 0x7f4df40cf6b0 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 0x7f4df40ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 0x7f4df40ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38836/0 (socket says 172.21.15.46:38836) 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 -- 172.21.15.46:0/53810284 learned_addr learned my addr 172.21.15.46:0/53810284 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df938f640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40b92e0 0x7f4df4158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df8b8e640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4df40cf6b0 0x7f4df415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 -- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4df40cf6b0 msgr2=0x7f4df415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4df40cf6b0 0x7f4df415c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 -- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40b92e0 msgr2=0x7f4df4158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40b92e0 0x7f4df4158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 -- 172.21.15.46:0/53810284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df40cba10 con 0x7f4df40d0480 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df9b90640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 0x7f4df40ccaa0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4dec02a220 tx=0x7f4dec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec037040 con 0x7f4df40d0480 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dec004230 con 0x7f4df40d0480 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df40cbcf0 con 0x7f4df40d0480 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.875+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec003610 con 0x7f4df40d0480 2024-05-09T03:16:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.876+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df415a420 con 0x7f4df40d0480 2024-05-09T03:16:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.877+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df406a270 con 0x7f4df40d0480 2024-05-09T03:16:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.878+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4dec0037b0 con 0x7f4df40d0480 2024-05-09T03:16:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.878+0000 7f4df1ffb640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4de00762d0 0x7f4de00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.878+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f4dec0b8430 con 0x7f4df40d0480 2024-05-09T03:16:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.879+0000 7f4df938f640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4de00762d0 0x7f4de00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:48.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.879+0000 7f4df938f640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4de00762d0 0x7f4de00787e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4df4071390 tx=0x7f4de4009870 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:48.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:48.883+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4dec081590 con 0x7f4df40d0480 2024-05-09T03:16:49.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.114+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4df40d0ca0 con 0x7f4df40d0480 2024-05-09T03:16:49.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.114+0000 7f4df1ffb640 1 -- 172.21.15.46:0/53810284 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5177 (secure 0 0 0) 0x7f4dec085890 con 0x7f4df40d0480 2024-05-09T03:16:49.115 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:49.115 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:48.326304+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7806":{"gid":7806,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/2566829830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2566829830},{"type":"v1","addr":"172.21.15.46:6834","nonce":2566829830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:16:49.116 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 124 2024-05-09T03:16:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4de00762d0 msgr2=0x7f4de00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4de00762d0 0x7f4de00787e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4df4071390 tx=0x7f4de4009870 comp rx=0 tx=0).stop 2024-05-09T03:16:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 msgr2=0x7f4df40ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:49.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 0x7f4df40ccaa0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f4dec02a220 tx=0x7f4dec0046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 shutdown_connections 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4de00762d0 0x7f4de00787e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.116+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df40d0480 0x7f4df40ccaa0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.117+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4df40cf6b0 0x7f4df415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.117+0000 7f4dfb5f3640 1 --2- 172.21.15.46:0/53810284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40b92e0 0x7f4df4158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.117+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 >> 172.21.15.46:0/53810284 conn(0x7f4df40d3c00 msgr2=0x7f4df4072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.117+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 shutdown_connections 2024-05-09T03:16:49.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:49.117+0000 7f4dfb5f3640 1 -- 172.21.15.46:0/53810284 wait complete. 2024-05-09T03:16:49.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:16:48.326304+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 90, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7806}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7806': {'gid': 7806, 'name': 'c', 'rank': 0, 'incarnation': 124, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/2566829830', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2566829830}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2566829830}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:16:49.129 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7806:up:replay 2024-05-09T03:16:49.129 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:16:54.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:16:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.661+0000 7f0b5b015640 1 Processor -- start 2024-05-09T03:16:54.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.661+0000 7f0b5b015640 1 -- start start 2024-05-09T03:16:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b5b015640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0b5408cd60 0x7f0b540d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:54.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b5b015640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b540d2090 0x7f0b540d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b5b015640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b540d4900 con 0x7f0b540d1c80 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b5b015640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0b540b5340 con 0x7f0b540d2090 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b5b015640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0b540b4fd0 con 0x7f0b5408cd60 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b595b2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b540d2090 0x7f0b540d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b53fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0b540d1c80 0x7f0b540cf2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34252/0 (socket says 172.21.15.46:34252) 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.662+0000 7f0b53fff640 1 -- 172.21.15.46:0/3215575505 learned_addr learned my addr 172.21.15.46:0/3215575505 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.663+0000 7f0b58db1640 1 --2- 172.21.15.46:0/3215575505 >> v2:172.21.15.88:3300/0 conn(0x7f0b5408cd60 0x7f0b540d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.663+0000 7f0b595b2640 1 -- 172.21.15.46:0/3215575505 >> v2:172.21.15.88:3300/0 conn(0x7f0b5408cd60 msgr2=0x7f0b540d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.663+0000 7f0b595b2640 1 --2- 172.21.15.46:0/3215575505 >> v2:172.21.15.88:3300/0 conn(0x7f0b5408cd60 0x7f0b540d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.663+0000 7f0b595b2640 1 -- 172.21.15.46:0/3215575505 >> v1:172.21.15.46:6789/0 conn(0x7f0b540d1c80 legacy=0x7f0b540cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.663+0000 7f0b595b2640 1 -- 172.21.15.46:0/3215575505 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b540d2c00 con 0x7f0b540d2090 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.664+0000 7f0b595b2640 1 --2- 172.21.15.46:0/3215575505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d2090 0x7f0b540d02c0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009d90 tx=0x7f0b4c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63473adfa5b9112f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.664+0000 7f0b537fe640 1 -- 172.21.15.46:0/3215575505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c031600 con 0x7f0b540d2090 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.664+0000 7f0b537fe640 1 -- 172.21.15.46:0/3215575505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b4c0317a0 con 0x7f0b540d2090 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.664+0000 7f0b537fe640 1 -- 172.21.15.46:0/3215575505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c031b70 con 0x7f0b540d2090 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 -- 172.21.15.46:0/3215575505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d2090 msgr2=0x7f0b540d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 --2- 172.21.15.46:0/3215575505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d2090 0x7f0b540d02c0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009d90 tx=0x7f0b4c02a610 comp rx=0 tx=0).stop 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 -- 172.21.15.46:0/3215575505 shutdown_connections 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 --2- 172.21.15.46:0/3215575505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d2090 0x7f0b540d02c0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 --2- 172.21.15.46:0/3215575505 >> v2:172.21.15.88:3300/0 conn(0x7f0b5408cd60 0x7f0b540d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 -- 172.21.15.46:0/3215575505 >> 172.21.15.46:0/3215575505 conn(0x7f0b540d3c00 msgr2=0x7f0b540d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 -- 172.21.15.46:0/3215575505 shutdown_connections 2024-05-09T03:16:54.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.665+0000 7f0b5b015640 1 -- 172.21.15.46:0/3215575505 wait complete. 2024-05-09T03:16:54.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.667+0000 7f0b5b015640 1 Processor -- start 2024-05-09T03:16:54.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.667+0000 7f0b5b015640 1 -- start start 2024-05-09T03:16:54.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b5b015640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b5408cd60 0x7f0b54153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:54.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b5b015640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 0x7f0b540cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b5b015640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b540d2090 0x7f0b54161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b5b015640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b540b92b0 con 0x7f0b540d1c80 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b5b015640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0b540b9130 con 0x7f0b540d2090 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b5b015640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0b540cf9c0 con 0x7f0b5408cd60 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 0x7f0b540cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 0x7f0b540cbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38844/0 (socket says 172.21.15.46:38844) 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b53fff640 1 -- 172.21.15.46:0/152431088 learned_addr learned my addr 172.21.15.46:0/152431088 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b595b2640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b540d2090 0x7f0b54161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.668+0000 7f0b58db1640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b5408cd60 0x7f0b54153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b53fff640 1 -- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b5408cd60 msgr2=0x7f0b54153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b53fff640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b5408cd60 0x7f0b54153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b53fff640 1 -- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b540d2090 msgr2=0x7f0b54161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b53fff640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b540d2090 0x7f0b54161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b53fff640 1 -- 172.21.15.46:0/152431088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b541600d0 con 0x7f0b540d1c80 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b53fff640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 0x7f0b540cbb40 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000c3a0 tx=0x7f0b4000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40017070 con 0x7f0b540d1c80 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b40005ce0 con 0x7f0b540d1c80 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b541603b0 con 0x7f0b540d1c80 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b400072f0 con 0x7f0b540d1c80 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.669+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b54160970 con 0x7f0b540d1c80 2024-05-09T03:16:54.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.670+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b5415fe40 con 0x7f0b540d1c80 2024-05-09T03:16:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.671+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0b4001a070 con 0x7f0b540d1c80 2024-05-09T03:16:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.672+0000 7f0b517fa640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0b38076300 0x7f0b38078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.672+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f0b40096b20 con 0x7f0b540d1c80 2024-05-09T03:16:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.672+0000 7f0b58db1640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0b38076300 0x7f0b38078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:54.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.672+0000 7f0b58db1640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0b38076300 0x7f0b38078810 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0b54074930 tx=0x7f0b48009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:54.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.676+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0b4005fb60 con 0x7f0b540d1c80 2024-05-09T03:16:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.905+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b5406a270 con 0x7f0b540d1c80 2024-05-09T03:16:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.906+0000 7f0b517fa640 1 -- 172.21.15.46:0/152431088 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+5184 (secure 0 0 0) 0x7f0b40063e60 con 0x7f0b540d1c80 2024-05-09T03:16:54.908 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:54.908 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":127,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:51.338571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7806":{"gid":7806,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2566829830","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2566829830},{"type":"v1","addr":"172.21.15.46:6834","nonce":2566829830}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7806,"qdb_cluster":[7806]},"id":6}]} 2024-05-09T03:16:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 127 2024-05-09T03:16:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0b38076300 msgr2=0x7f0b38078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0b38076300 0x7f0b38078810 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0b54074930 tx=0x7f0b48009190 comp rx=0 tx=0).stop 2024-05-09T03:16:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 msgr2=0x7f0b540cbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 0x7f0b540cbb40 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f0b4000c3a0 tx=0x7f0b4000c860 comp rx=0 tx=0).stop 2024-05-09T03:16:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 shutdown_connections 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b540d2090 0x7f0b54161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0b38076300 0x7f0b38078810 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b540d1c80 0x7f0b540cbb40 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 --2- 172.21.15.46:0/152431088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0b5408cd60 0x7f0b54153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 >> 172.21.15.46:0/152431088 conn(0x7f0b540d3c00 msgr2=0x7f0b540cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 shutdown_connections 2024-05-09T03:16:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:54.908+0000 7f0b5b015640 1 -- 172.21.15.46:0/152431088 wait complete. 2024-05-09T03:16:54.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 127, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:16:51.338571+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 90, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7806}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7806': {'gid': 7806, 'name': 'c', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/2566829830', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2566829830}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2566829830}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7806, 'qdb_cluster': [7806]} 2024-05-09T03:16:54.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:16:54.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.036+0000 7f12f275f640 1 Processor -- start 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.036+0000 7f12f275f640 1 -- start start 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12f275f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12ec0d1000 0x7f12ec1437d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12f275f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12ec0d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12f275f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12ec143d10 con 0x7f12ec140d00 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12f275f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f12ec143fe0 con 0x7f12ec0d1000 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12f275f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec1442c0 con 0x7f12ec141110 2024-05-09T03:16:55.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12e3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12ec140d00 0x7f12ec15eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34258/0 (socket says 172.21.15.46:34258) 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12e3fff640 1 -- 172.21.15.46:0/3731379119 learned_addr learned my addr 172.21.15.46:0/3731379119 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12ebfff640 1 --2- 172.21.15.46:0/3731379119 >> v2:172.21.15.88:3300/0 conn(0x7f12ec0d1000 0x7f12ec1437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12ec0d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877209031 0 0) 0x7f12ec143d10 con 0x7f12ec140d00 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.038+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12d8003520 con 0x7f12ec140d00 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2966184067 0 0) 0x7f12d8003520 con 0x7f12ec140d00 2024-05-09T03:16:55.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 msgr2=0x7f12ec0d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12ec0d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 >> v2:172.21.15.88:3300/0 conn(0x7f12ec0d1000 msgr2=0x7f12ec1437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/3731379119 >> v2:172.21.15.88:3300/0 conn(0x7f12ec0d1000 0x7f12ec1437d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec1445a0 con 0x7f12ec140d00 2024-05-09T03:16:55.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f12d40016f0 con 0x7f12ec140d00 2024-05-09T03:16:55.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 >> v1:172.21.15.46:6789/0 conn(0x7f12ec140d00 legacy=0x7f12ec15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d8003a00 0x7f12d8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12d8004800 0x7f12d8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12d8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec143fe0 con 0x7f12d8003a00 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1442c0 con 0x7f12d8004800 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12d800ba70 con 0x7f12ec141110 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12ebfff640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d8003a00 0x7f12d8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12e3fff640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12d8004800 0x7f12d8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12d8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12ebfff640 1 -- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 msgr2=0x7f12d8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12ebfff640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12d8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12ebfff640 1 -- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12d8004800 msgr2=0x7f12d8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12ebfff640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12d8004800 0x7f12d8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12ebfff640 1 -- 172.21.15.46:0/3731379119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d8003520 con 0x7f12d8003a00 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.039+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12ec141110 0x7f12d8005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.040+0000 7f12ebfff640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d8003a00 0x7f12d8003e70 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009a90 tx=0x7f12d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e59c16573eaa1b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.040+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc036070 con 0x7f12d8003a00 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.040+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12dc031400 con 0x7f12d8003a00 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.040+0000 7f12eb7fe640 1 -- 172.21.15.46:0/3731379119 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12dc009690 con 0x7f12d8003a00 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f275f640 1 -- 172.21.15.46:0/3731379119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d8003a00 msgr2=0x7f12d8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f275f640 1 --2- 172.21.15.46:0/3731379119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12d8003a00 0x7f12d8003e70 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009a90 tx=0x7f12d8005830 comp rx=0 tx=0).stop 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f0cfc640 1 -- 172.21.15.46:0/3731379119 reap_dead start 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f275f640 1 -- 172.21.15.46:0/3731379119 shutdown_connections 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f275f640 1 -- 172.21.15.46:0/3731379119 >> 172.21.15.46:0/3731379119 conn(0x7f12ec0705c0 msgr2=0x7f12ec0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f275f640 1 -- 172.21.15.46:0/3731379119 shutdown_connections 2024-05-09T03:16:55.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.041+0000 7f12f275f640 1 -- 172.21.15.46:0/3731379119 wait complete. 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.043+0000 7f12f275f640 1 Processor -- start 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.043+0000 7f12f275f640 1 -- start start 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f275f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12d8003a00 0x7f12ec0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f275f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12ec15d150 0x7f12ec1f74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f275f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 0x7f12ec0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f275f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec15f5f0 con 0x7f12ec1f79e0 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f275f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec0d1410 con 0x7f12ec15d150 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f275f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec15e660 con 0x7f12d8003a00 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 0x7f12ec0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 0x7f12ec0c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38864/0 (socket says 172.21.15.46:38864) 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 -- 172.21.15.46:0/2907970054 learned_addr learned my addr 172.21.15.46:0/2907970054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12ec15d150 0x7f12ec1f74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12d8003a00 0x7f12ec0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 -- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12d8003a00 msgr2=0x7f12ec0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12d8003a00 0x7f12ec0cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 -- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12ec15d150 msgr2=0x7f12ec1f74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12ec15d150 0x7f12ec1f74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.044+0000 7f12f0cfc640 1 -- 172.21.15.46:0/2907970054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ec1f7df0 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.045+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 0x7f12ec0c9ab0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0b2570 tx=0x7f12e4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.045+0000 7f12e97fa640 1 -- 172.21.15.46:0/2907970054 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4010e80 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.045+0000 7f12e97fa640 1 -- 172.21.15.46:0/2907970054 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e4003990 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.045+0000 7f12e97fa640 1 -- 172.21.15.46:0/2907970054 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4003da0 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.045+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ec0b4610 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.045+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ec0b4df0 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.047+0000 7f12e97fa640 1 -- 172.21.15.46:0/2907970054 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f12e401e020 con 0x7f12ec1f79e0 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.047+0000 7f12e97fa640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12cc0762d0 0x7f12cc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.047+0000 7f12e97fa640 1 -- 172.21.15.46:0/2907970054 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f12e4098090 con 0x7f12ec1f79e0 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.047+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12cc0762d0 0x7f12cc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.048+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12cc0762d0 0x7f12cc0787e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009a90 tx=0x7f12dc008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.048+0000 7f12f960b740 1 Processor -- start 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.048+0000 7f12f960b740 1 -- start start 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486fc7270 0x55b487396eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12f960b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 0x55b4873b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12f960b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b4872cf4e0 con 0x55b486f9e3b0 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b48726aff0 con 0x55b486fc7270 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b4872cb4a0 con 0x55b486fa6590 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 0x55b4873b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 0x55b4873b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38872/0 (socket says 172.21.15.46:38872) 2024-05-09T03:16:55.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.049+0000 7f12e3fff640 1 -- 172.21.15.46:0/2597555976 learned_addr learned my addr 172.21.15.46:0/2597555976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486fc7270 0x55b487396eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e3fff640 1 -- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 msgr2=0x55b4873b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e3fff640 1 -- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486fc7270 msgr2=0x55b487396eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486fc7270 0x55b487396eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e3fff640 1 -- 172.21.15.46:0/2597555976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b487251720 con 0x55b486f9e3b0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486fc7270 0x55b487396eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 0x55b4873b1be0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f12d40079f0 tx=0x7f12d40016e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78cc02cf50a53a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e1ffb640 1 -- 172.21.15.46:0/2597555976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d401b070 con 0x55b486f9e3b0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e1ffb640 1 -- 172.21.15.46:0/2597555976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d4007180 con 0x55b486f9e3b0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.050+0000 7f12e1ffb640 1 -- 172.21.15.46:0/2597555976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4004e60 con 0x55b486f9e3b0 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 -- 172.21.15.46:0/2597555976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 msgr2=0x55b4873b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 0x55b4873b1be0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f12d40079f0 tx=0x7f12d40016e0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 -- 172.21.15.46:0/2597555976 shutdown_connections 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486f9e3b0 0x55b4873b1be0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 --2- 172.21.15.46:0/2597555976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486fc7270 0x55b487396eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 -- 172.21.15.46:0/2597555976 >> 172.21.15.46:0/2597555976 conn(0x55b486fbb3c0 msgr2=0x55b48736c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 -- 172.21.15.46:0/2597555976 shutdown_connections 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.051+0000 7f12f960b740 1 -- 172.21.15.46:0/2597555976 wait complete. 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.052+0000 7f12f960b740 1 -- start start 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.052+0000 7f12f960b740 1 Processor -- start 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873915e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b487256760 con 0x55b486fa6590 2024-05-09T03:16:55.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b4873afb40 con 0x55b486f9e3b0 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b4873aa4e0 con 0x55b486fc7270 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12f960b740 10 client.0 _ll_register_callbacks cb 0x55b4872794b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38876/0 (socket says 172.21.15.46:38876) 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.053+0000 7f12e3fff640 1 -- 172.21.15.46:0/2405764568 learned_addr learned my addr 172.21.15.46:0/2405764568 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873915e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12e3fff640 1 -- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 msgr2=0x55b4873aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12e3fff640 1 -- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 msgr2=0x55b4873915e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873915e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12e3fff640 1 -- 172.21.15.46:0/2405764568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b487297bd0 con 0x55b486fa6590 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873915e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:55.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873ab7d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f12d4014000 tx=0x7f12ec144770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8358d86a71459d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12caffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12caffd640 1 -- 172.21.15.46:0/2405764568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4024060 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12caffd640 1 -- 172.21.15.46:0/2405764568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f12d40177b0 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12d4007180 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12caffd640 1 -- 172.21.15.46:0/2405764568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=127) ==== 24+0+0 (secure 0 0 0) 0x7f12d402a050 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12f960b740 10 client.7893 fetch_fsmap learned FSMap version 127 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=127,monmap=2+}) -- 0x55b48727d280 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.054+0000 7f12caffd640 1 -- 172.21.15.46:0/2405764568 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d4007180 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12caffd640 1 -- 172.21.15.46:0/2405764568 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 127) ==== 3494+0+0 (secure 0 0 0) 0x7f12d402d020 con 0x55b486fa6590 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12f960b740 10 client.7893 fetch_fsmap finished waiting for FSMap version 127 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12f960b740 10 client.7893 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12f960b740 20 client.7893 populate_metadata read hostname 'smithi046' 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12f960b740 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b486f85e40 0x55b4873a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12f960b740 4 client.7893 mds_command: new command op to 7806 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 --> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4872dd0f0 con 0x55b486f85e40 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b486f85e40 0x55b4873a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b486f85e40 0x55b4873a8f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d77127cd2e36a3 server_cookie=68207b2d386d550f in_seq=0 out_seq=0 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.055+0000 7f12caffd640 10 client.7893 ms_handle_connect on v2:172.21.15.46:6832/2566829830 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12caffd640 1 -- 172.21.15.46:0/2405764568 <== mds.0 v2:172.21.15.46:6832/2566829830 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b4872dd0f0 con 0x55b486f85e40 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12caffd640 10 client.7893 handle_command_reply: tid=0 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 client.7893 shutdown 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 msgr2=0x55b4873ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873ab7d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f12d4014000 tx=0x7f12ec144770 comp rx=0 tx=0).stop 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 shutdown_connections 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873ab7d0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b486f85e40 0x55b4873a8f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 --2- 172.21.15.46:0/2405764568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873915e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12caffd640 0 client.7893 ms_handle_reset on v2:172.21.15.46:6832/2566829830 2024-05-09T03:16:55.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 >> 172.21.15.46:0/2405764568 conn(0x55b486fbb3c0 msgr2=0x55b487393a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 shutdown_connections 2024-05-09T03:16:55.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 1 -- 172.21.15.46:0/2405764568 wait complete. 2024-05-09T03:16:55.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.056+0000 7f12f960b740 20 client.7893 trim_cache size 0 max 16384 2024-05-09T03:16:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.075+0000 7f12f960b740 1 Processor -- start 2024-05-09T03:16:55.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.075+0000 7f12f960b740 1 -- start start 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873990c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873d4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f960b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 0x55b4873d4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f960b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec200670 con 0x55b486fc7270 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f12ec1ff9d0 con 0x55b486f9e3b0 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f12ec200020 con 0x55b486fa6590 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 0x55b4873d4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 0x55b4873d4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38880/0 (socket says 172.21.15.46:38880) 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 -- 172.21.15.46:0/418662870 learned_addr learned my addr 172.21.15.46:0/418662870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12ebfff640 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12e3fff640 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873d4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 -- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 msgr2=0x55b4873d4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873d4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 -- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 msgr2=0x55b4873990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 -- 172.21.15.46:0/418662870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b487032850 con 0x55b486fc7270 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 0x55b4873d4f50 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f12e4002230 tx=0x7f12e4060ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9dfbeda395acd1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12e17fa640 1 -- 172.21.15.46:0/418662870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e407f070 con 0x55b486fc7270 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12e17fa640 1 -- 172.21.15.46:0/418662870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e4065350 con 0x55b486fc7270 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.077+0000 7f12e17fa640 1 -- 172.21.15.46:0/418662870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4013e30 con 0x55b486fc7270 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 -- 172.21.15.46:0/418662870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 msgr2=0x55b4873d4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 0x55b4873d4f50 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f12e4002230 tx=0x7f12e4060ab0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 -- 172.21.15.46:0/418662870 shutdown_connections 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fc7270 0x55b4873d4f50 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fa6590 0x55b4873d4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 --2- 172.21.15.46:0/418662870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 -- 172.21.15.46:0/418662870 >> 172.21.15.46:0/418662870 conn(0x55b486fbb3c0 msgr2=0x55b4873915e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 -- 172.21.15.46:0/418662870 shutdown_connections 2024-05-09T03:16:55.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.078+0000 7f12f960b740 1 -- 172.21.15.46:0/418662870 wait complete. 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.079+0000 7f12f960b740 1 -- start start 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.079+0000 7f12f960b740 1 Processor -- start 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873d3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b4873af330 con 0x55b486fa6590 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b487391db0 con 0x55b486f9e3b0 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b487399760 con 0x55b486fc7270 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f960b740 10 client.0 _ll_register_callbacks cb 0x7f12d4019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873d3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38884/0 (socket says 172.21.15.46:38884) 2024-05-09T03:16:55.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 -- 172.21.15.46:0/2194388971 learned_addr learned my addr 172.21.15.46:0/2194388971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12f0cfc640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873d3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 -- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 msgr2=0x55b4873d3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873d3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 -- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 msgr2=0x55b4873a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 -- 172.21.15.46:0/2194388971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4870a96b0 con 0x55b486fa6590 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12e3fff640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873d3a00 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f12d4004bb0 tx=0x7f12d4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7482f759b2704dc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12caffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.081+0000 7f12caffd640 1 -- 172.21.15.46:0/2194388971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d4004e60 con 0x55b486fa6590 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12caffd640 1 -- 172.21.15.46:0/2194388971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f12d4015460 con 0x55b486fa6590 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e4065350 con 0x55b486fa6590 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12caffd640 1 -- 172.21.15.46:0/2194388971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=127) ==== 24+0+0 (secure 0 0 0) 0x7f12d401b040 con 0x55b486fa6590 2024-05-09T03:16:55.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 10 client.7899 fetch_fsmap learned FSMap version 127 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=127,monmap=2+}) -- 0x55b48728b630 con 0x55b486fa6590 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12caffd640 1 -- 172.21.15.46:0/2194388971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e4065350 con 0x55b486fa6590 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12caffd640 1 -- 172.21.15.46:0/2194388971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 127) ==== 3494+0+0 (secure 0 0 0) 0x7f12d402d040 con 0x55b486fa6590 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 10 client.7899 fetch_fsmap finished waiting for FSMap version 127 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 10 client.7899 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 20 client.7899 populate_metadata read hostname 'smithi046' 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b487391f80 0x55b487392390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 4 client.7899 mds_command: new command op to 7806 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 --> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] -- command(tid 0: {"prefix": "status"}) -- 0x55b4872d6f60 con 0x55b487391f80 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.082+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b487391f80 0x55b487392390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12caffd640 10 client.7899 ms_handle_connect on v2:172.21.15.46:6832/2566829830 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12ebfff640 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b487391f80 0x55b487392390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87853fd3dc300430 server_cookie=83ee9cafbd296c1a in_seq=0 out_seq=0 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12caffd640 1 -- 172.21.15.46:0/2194388971 <== mds.0 v2:172.21.15.46:6832/2566829830 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b4872d6f60 con 0x55b487391f80 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12caffd640 10 client.7899 handle_command_reply: tid=0 2024-05-09T03:16:55.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 client.7899 shutdown 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 msgr2=0x55b4873d3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873d3a00 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f12d4004bb0 tx=0x7f12d4009040 comp rx=0 tx=0).stop 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 shutdown_connections 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b486fc7270 0x55b4873d3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b486fa6590 0x55b4873d3a00 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.46:6832/2566829830,v1:172.21.15.46:6834/2566829830] conn(0x55b487391f80 0x55b487392390 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 --2- 172.21.15.46:0/2194388971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b486f9e3b0 0x55b4873a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 >> 172.21.15.46:0/2194388971 conn(0x55b486fbb3c0 msgr2=0x55b48736db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12caffd640 0 client.7899 ms_handle_reset on v2:172.21.15.46:6832/2566829830 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 shutdown_connections 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 1 -- 172.21.15.46:0/2194388971 wait complete. 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.083+0000 7f12f960b740 20 client.7899 trim_cache size 0 max 16384 2024-05-09T03:16:55.089 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7806, 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7571461419999999, 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 127, 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 90, 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 90, 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.718721073999999 2024-05-09T03:16:55.090 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:16:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12cc0762d0 msgr2=0x7f12cc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12cc0762d0 0x7f12cc0787e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009a90 tx=0x7f12dc008030 comp rx=0 tx=0).stop 2024-05-09T03:16:55.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 msgr2=0x7f12ec0c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 0x7f12ec0c9ab0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f12ec0b2570 tx=0x7f12e4009f90 comp rx=0 tx=0).stop 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 shutdown_connections 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f12cc0762d0 0x7f12cc0787e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12ec1f79e0 0x7f12ec0c9ab0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12ec15d150 0x7f12ec1f74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 --2- 172.21.15.46:0/2907970054 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12d8003a00 0x7f12ec0cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 >> 172.21.15.46:0/2907970054 conn(0x7f12ec0705c0 msgr2=0x7f12ec075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 shutdown_connections 2024-05-09T03:16:55.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.084+0000 7f12f275f640 1 -- 172.21.15.46:0/2907970054 wait complete. 2024-05-09T03:16:55.097 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:16:55.097 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete mds0_openfiles.0', proceeding to ls 2024-05-09T03:16:55.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:16:55.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.211+0000 7fe3e7d76640 1 Processor -- start 2024-05-09T03:16:55.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.211+0000 7fe3e7d76640 1 -- start start 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.212+0000 7fe3e7d76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3e0164c70 0x7fe3e01615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.212+0000 7fe3e7d76640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3e015dfe0 0x7fe3e00d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.212+0000 7fe3e7d76640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e0160b20 con 0x7fe3e00d2480 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.212+0000 7fe3e7d76640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e0161150 con 0x7fe3e015dfe0 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.212+0000 7fe3e7d76640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e00d15f0 con 0x7fe3e0164c70 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e5311640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3e015dfe0 0x7fe3e00d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e6313640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe3e00d2480 0x7fe3e00d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34260/0 (socket says 172.21.15.46:34260) 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e6313640 1 -- 172.21.15.46:0/2651095783 learned_addr learned my addr 172.21.15.46:0/2651095783 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e5b12640 1 --2- 172.21.15.46:0/2651095783 >> v2:172.21.15.88:3300/0 conn(0x7fe3e0164c70 0x7fe3e01615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e5311640 1 -- 172.21.15.46:0/2651095783 >> v2:172.21.15.88:3300/0 conn(0x7fe3e0164c70 msgr2=0x7fe3e01615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e5311640 1 --2- 172.21.15.46:0/2651095783 >> v2:172.21.15.88:3300/0 conn(0x7fe3e0164c70 0x7fe3e01615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e5311640 1 -- 172.21.15.46:0/2651095783 >> v1:172.21.15.46:6789/0 conn(0x7fe3e00d2480 legacy=0x7fe3e00d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e4b10640 1 -- 172.21.15.46:0/2651095783 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765773358 0 0) 0x7fe3e0160b20 con 0x7fe3e00d2480 2024-05-09T03:16:55.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.213+0000 7fe3e5311640 1 -- 172.21.15.46:0/2651095783 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e00d18d0 con 0x7fe3e015dfe0 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.214+0000 7fe3e5311640 1 --2- 172.21.15.46:0/2651095783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e015dfe0 0x7fe3e00d0dc0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00b8900 tx=0x7fe3d002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3eb1fdcb34da68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.214+0000 7fe3e4b10640 1 -- 172.21.15.46:0/2651095783 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0037070 con 0x7fe3e015dfe0 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.214+0000 7fe3e4b10640 1 -- 172.21.15.46:0/2651095783 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d0032440 con 0x7fe3e015dfe0 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.214+0000 7fe3e4b10640 1 -- 172.21.15.46:0/2651095783 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d002ae60 con 0x7fe3e015dfe0 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 -- 172.21.15.46:0/2651095783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e015dfe0 msgr2=0x7fe3e00d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/2651095783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e015dfe0 0x7fe3e00d0dc0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe3e00b8900 tx=0x7fe3d002a110 comp rx=0 tx=0).stop 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 -- 172.21.15.46:0/2651095783 shutdown_connections 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/2651095783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e015dfe0 0x7fe3e00d0dc0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/2651095783 >> v2:172.21.15.88:3300/0 conn(0x7fe3e0164c70 0x7fe3e01615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 -- 172.21.15.46:0/2651095783 >> 172.21.15.46:0/2651095783 conn(0x7fe3e0073db0 msgr2=0x7fe3e00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 -- 172.21.15.46:0/2651095783 shutdown_connections 2024-05-09T03:16:55.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.215+0000 7fe3e7d76640 1 -- 172.21.15.46:0/2651095783 wait complete. 2024-05-09T03:16:55.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.216+0000 7fe3e7d76640 1 Processor -- start 2024-05-09T03:16:55.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.217+0000 7fe3e7d76640 1 -- start start 2024-05-09T03:16:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e7d76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3e00d2480 0x7fe3e0152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e7d76640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3e015dfe0 0x7fe3e00cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e7d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 0x7fe3e01f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e7d76640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e015e3f0 con 0x7fe3e0164c70 2024-05-09T03:16:55.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e7d76640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e00d1300 con 0x7fe3e00d2480 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e7d76640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e00d45c0 con 0x7fe3e015dfe0 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 0x7fe3e01f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 0x7fe3e01f6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38896/0 (socket says 172.21.15.46:38896) 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 -- 172.21.15.46:0/3709897554 learned_addr learned my addr 172.21.15.46:0/3709897554 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e5311640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3e015dfe0 0x7fe3e00cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e5b12640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3e00d2480 0x7fe3e0152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 -- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3e015dfe0 msgr2=0x7fe3e00cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3e015dfe0 0x7fe3e00cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 -- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3e00d2480 msgr2=0x7fe3e0152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3e00d2480 0x7fe3e0152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 -- 172.21.15.46:0/3709897554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e00ce640 con 0x7fe3e0164c70 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e5311640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3e015dfe0 0x7fe3e00cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.218+0000 7fe3e6313640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 0x7fe3e01f6630 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800c6b0 tx=0x7fe3d800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.219+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8018070 con 0x7fe3e0164c70 2024-05-09T03:16:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.219+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d8004d80 con 0x7fe3e0164c70 2024-05-09T03:16:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.219+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e00ce8e0 con 0x7fe3e0164c70 2024-05-09T03:16:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.219+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800a820 con 0x7fe3e0164c70 2024-05-09T03:16:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.219+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e00ceea0 con 0x7fe3e0164c70 2024-05-09T03:16:55.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.220+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e008cfd0 con 0x7fe3e0164c70 2024-05-09T03:16:55.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.221+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe3d801b070 con 0x7fe3e0164c70 2024-05-09T03:16:55.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.221+0000 7fe3d67fc640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3c4076300 0x7fe3c4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:55.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.221+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7fe3d8096cd0 con 0x7fe3e0164c70 2024-05-09T03:16:55.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.221+0000 7fe3e5b12640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3c4076300 0x7fe3c4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:55.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.222+0000 7fe3e5b12640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3c4076300 0x7fe3c4078810 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009aa0 tx=0x7fe3dc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:55.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.226+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe3e008cfd0 con 0x7fe3e0164c70 2024-05-09T03:16:55.313 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:16:55.312+0000 7ff0b6f40640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:16:55.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.456+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe3e006a270 con 0x7fe3e0164c70 2024-05-09T03:16:55.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.457+0000 7fe3d67fc640 1 -- 172.21.15.46:0/3709897554 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v127) ==== 53+0+73 (secure 0 0 0) 0x7fe3d805fd10 con 0x7fe3e0164c70 2024-05-09T03:16:55.459 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3c4076300 msgr2=0x7fe3c4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3c4076300 0x7fe3c4078810 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc009aa0 tx=0x7fe3dc009190 comp rx=0 tx=0).stop 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 msgr2=0x7fe3e01f6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 0x7fe3e01f6630 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800c6b0 tx=0x7fe3d800cb70 comp rx=0 tx=0).stop 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 shutdown_connections 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe3c4076300 0x7fe3c4078810 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3e0164c70 0x7fe3e01f6630 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe3e015dfe0 0x7fe3e00cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 --2- 172.21.15.46:0/3709897554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3e00d2480 0x7fe3e0152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:55.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 >> 172.21.15.46:0/3709897554 conn(0x7fe3e0073db0 msgr2=0x7fe3e00b9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:55.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 shutdown_connections 2024-05-09T03:16:55.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55.459+0000 7fe3e7d76640 1 -- 172.21.15.46:0/3709897554 wait complete. 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:16:55.472 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:16:55.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:55.473 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:16:55.490 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42102sec preferred_lft 42102sec 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:16:55.491 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:16:55.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:55.492 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:16:55.555 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:16:55.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:55.556 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:16:55.613 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:16:55.613 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:16:55.613 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:16:55.613 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:16:55.613 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:16:55.613 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:16:55.614 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:16:55.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:16:55.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:16:55.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:16:55.614 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:16:55.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:16:55.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:16:55.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:55.744 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:16:55.760 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:16:55.760 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:16:55.761 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:16:55.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:55.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:16:55.826 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:16:55.826 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:16:55.826 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:55.883 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:16:55.883 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:16:55.883 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:55.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:16:56.006 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:16:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:16:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:16:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:16:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:16:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:16:56.063 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:16:56.064 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:56.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:56.066 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:16:56.119 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:16:56.119 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:16:56.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:16:56.187 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:16:56.245 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:16:56.246 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:56.247 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:16:56.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:56.248 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:16:56.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:16:56.255+0000 7f006a796480 -1 init, newargv = 0x7f00580057c0 newargc=15 2024-05-09T03:16:56.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[67649]: starting ceph client 2024-05-09T03:16:56.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[67649]: starting fuse 2024-05-09T03:16:56.272 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:16:56.273 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:56.273 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:16:56.274 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:16:56.393 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.67649.asok 2024-05-09T03:16:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:16:56.404 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.67649.asok 2024-05-09T03:16:56.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:56.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.67649.asok status 2024-05-09T03:16:56.551 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:16:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:16:56.551 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "67649", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7911, 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:16:56.552 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 7911 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1181068909 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1181068909 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.7911 192.168.0.1:0/1181068909", 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1181068909", 2024-05-09T03:16:56.553 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:16:56.554 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 127, 2024-05-09T03:16:56.554 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 90, 2024-05-09T03:16:56.554 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:16:56.554 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:16:56.554 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:16:56.554 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:16:56.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:56.560 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:16:56.560 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:56.576 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:16:56.576 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:56.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:56.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:56.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:56.642 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:16:56.685 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:16:56.702 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:16:56.702 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:16:56.702 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:16:56.702 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:16:56.702 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs 2024-05-09T03:16:56.703 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_openfiles.0' did not caused DAMAGED state 2024-05-09T03:16:56.703 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 500.00000000' 2024-05-09T03:16:56.703 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:56.703 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:56.760 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:16:56.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.914+0000 7f77d3861640 1 Processor -- start 2024-05-09T03:16:56.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.914+0000 7f77d3861640 1 -- start start 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d3861640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77cc0d1740 0x7f77cc0b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d3861640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77cc0b9690 con 0x7f77cc0cfcf0 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d3861640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77cc0ca9e0 con 0x7f77cc0d1110 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d3861640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f77cc0d2b00 con 0x7f77cc0d1740 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d15fd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77cc0d1740 0x7f77cc0b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d15fd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77cc0d1740 0x7f77cc0b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47926/0 (socket says 172.21.15.46:47926) 2024-05-09T03:16:56.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d15fd640 1 -- 172.21.15.46:0/107661780 learned_addr learned my addr 172.21.15.46:0/107661780 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:56.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d1dfe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f77cc0cfcf0 0x7f77cc09cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41348/0 (socket says 172.21.15.46:41348) 2024-05-09T03:16:56.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d15fd640 1 -- 172.21.15.46:0/107661780 >> v1:172.21.15.88:6789/0 conn(0x7f77cc0d1110 legacy=0x7f77cc0d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:56.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d15fd640 1 -- 172.21.15.46:0/107661780 >> v1:172.21.15.46:6789/0 conn(0x7f77cc0cfcf0 legacy=0x7f77cc09cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:56.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77cbfff640 1 -- 172.21.15.46:0/107661780 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516725228 0 0) 0x7f77cc0b9690 con 0x7f77cc0cfcf0 2024-05-09T03:16:56.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.916+0000 7f77d15fd640 1 -- 172.21.15.46:0/107661780 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc08cfb0 con 0x7f77cc0d1740 2024-05-09T03:16:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.917+0000 7f77d15fd640 1 --2- 172.21.15.46:0/107661780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1740 0x7f77cc0b73d0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009870 tx=0x7f77bc02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6369705130f598a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.917+0000 7f77cbfff640 1 -- 172.21.15.46:0/107661780 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc037070 con 0x7f77cc0d1740 2024-05-09T03:16:56.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.917+0000 7f77cbfff640 1 -- 172.21.15.46:0/107661780 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77bc02ae50 con 0x7f77cc0d1740 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.917+0000 7f77cbfff640 1 -- 172.21.15.46:0/107661780 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc032860 con 0x7f77cc0d1740 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 -- 172.21.15.46:0/107661780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1740 msgr2=0x7f77cc0b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 --2- 172.21.15.46:0/107661780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1740 0x7f77cc0b73d0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009870 tx=0x7f77bc02a300 comp rx=0 tx=0).stop 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 -- 172.21.15.46:0/107661780 shutdown_connections 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 --2- 172.21.15.46:0/107661780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1740 0x7f77cc0b73d0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 -- 172.21.15.46:0/107661780 >> 172.21.15.46:0/107661780 conn(0x7f77cc0d05f0 msgr2=0x7f77cc0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 -- 172.21.15.46:0/107661780 shutdown_connections 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.918+0000 7f77d3861640 1 -- 172.21.15.46:0/107661780 wait complete. 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.920+0000 7f77d3861640 1 Processor -- start 2024-05-09T03:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.920+0000 7f77d3861640 1 -- start start 2024-05-09T03:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d3861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77cc0cfcf0 0x7f77cc1618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d3861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 0x7f77cc154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d3861640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77cc0d1740 0x7f77cc0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d3861640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc0b93b0 con 0x7f77cc0d1110 2024-05-09T03:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d3861640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc08cd60 con 0x7f77cc0d1740 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d3861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77cc09d3b0 con 0x7f77cc0cfcf0 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d0dfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 0x7f77cc154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d0dfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 0x7f77cc154fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47938/0 (socket says 172.21.15.46:47938) 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.921+0000 7f77d0dfc640 1 -- 172.21.15.46:0/2452687453 learned_addr learned my addr 172.21.15.46:0/2452687453 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d1dfe640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77cc0d1740 0x7f77cc0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d15fd640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77cc0cfcf0 0x7f77cc1618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d0dfc640 1 -- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77cc0cfcf0 msgr2=0x7f77cc1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d0dfc640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77cc0cfcf0 0x7f77cc1618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d0dfc640 1 -- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77cc0d1740 msgr2=0x7f77cc0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d0dfc640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77cc0d1740 0x7f77cc0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d0dfc640 1 -- 172.21.15.46:0/2452687453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc0c9a00 con 0x7f77cc0d1110 2024-05-09T03:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d1dfe640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77cc0d1740 0x7f77cc0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d0dfc640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 0x7f77cc154fe0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c3d0 tx=0x7f77c000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0017070 con 0x7f77cc0d1110 2024-05-09T03:16:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c00052d0 con 0x7f77cc0d1110 2024-05-09T03:16:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77cc0c9ce0 con 0x7f77cc0d1110 2024-05-09T03:16:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c00037f0 con 0x7f77cc0d1110 2024-05-09T03:16:56.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.922+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77cc0ca2a0 con 0x7f77cc0d1110 2024-05-09T03:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.924+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77cc0b8930 con 0x7f77cc0d1110 2024-05-09T03:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.924+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f77c001a070 con 0x7f77cc0d1110 2024-05-09T03:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.925+0000 7f77c9ffb640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b80762d0 0x7f77b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.925+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f77c0096ad0 con 0x7f77cc0d1110 2024-05-09T03:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.925+0000 7f77d15fd640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b80762d0 0x7f77b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.925+0000 7f77d15fd640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b80762d0 0x7f77b80787e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f77bc034000 tx=0x7f77bc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:56.929+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f77cc0b8930 con 0x7f77cc0d1110 2024-05-09T03:16:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.137+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f77cc06d1b0 con 0x7f77cc0d1110 2024-05-09T03:16:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.138+0000 7f77c9ffb640 1 -- 172.21.15.46:0/2452687453 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v90) ==== 81+0+730 (secure 0 0 0) 0x7f77cc06d1b0 con 0x7f77cc0d1110 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:57.140 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:listed 12 entries 2024-05-09T03:16:57.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b80762d0 msgr2=0x7f77b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b80762d0 0x7f77b80787e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f77bc034000 tx=0x7f77bc02e000 comp rx=0 tx=0).stop 2024-05-09T03:16:57.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 msgr2=0x7f77cc154fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 0x7f77cc154fe0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c3d0 tx=0x7f77c000c890 comp rx=0 tx=0).stop 2024-05-09T03:16:57.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 shutdown_connections 2024-05-09T03:16:57.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77cc0d1740 0x7f77cc0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77b80762d0 0x7f77b80787e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77cc0d1110 0x7f77cc154fe0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 --2- 172.21.15.46:0/2452687453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77cc0cfcf0 0x7f77cc1618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 >> 172.21.15.46:0/2452687453 conn(0x7f77cc0d05f0 msgr2=0x7f77cc0d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 shutdown_connections 2024-05-09T03:16:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.140+0000 7f77d3861640 1 -- 172.21.15.46:0/2452687453 wait complete. 2024-05-09T03:16:57.153 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:16:57.153 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:16:57.170 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:16:57.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.323+0000 7f866cfc7640 1 Processor -- start 2024-05-09T03:16:57.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.323+0000 7f866cfc7640 1 -- start start 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.324+0000 7f866cfc7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f86680bb770 0x7f86680d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.324+0000 7f866cfc7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86680d3180 0x7f86681642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.324+0000 7f866cfc7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8668164900 con 0x7f8668163eb0 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.324+0000 7f866cfc7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f86680d4bb0 con 0x7f86680bb770 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.324+0000 7f866cfc7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8668164be0 con 0x7f86680d3180 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.324+0000 7f866659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f86680bb770 0x7f86680d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f866659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f86680bb770 0x7f86680d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47950/0 (socket says 172.21.15.46:47950) 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f866659c640 1 -- 172.21.15.46:0/875250360 learned_addr learned my addr 172.21.15.46:0/875250360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f8666d9d640 1 --2- 172.21.15.46:0/875250360 >> v2:172.21.15.88:3300/0 conn(0x7f86680d3180 0x7f86681642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f866659c640 1 -- 172.21.15.46:0/875250360 >> v2:172.21.15.88:3300/0 conn(0x7f86680d3180 msgr2=0x7f86681642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f866659c640 1 --2- 172.21.15.46:0/875250360 >> v2:172.21.15.88:3300/0 conn(0x7f86680d3180 0x7f86681642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f866659c640 1 -- 172.21.15.46:0/875250360 >> v1:172.21.15.88:6789/0 conn(0x7f8668163eb0 legacy=0x7f86680d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.325+0000 7f866659c640 1 -- 172.21.15.46:0/875250360 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86680d4840 con 0x7f86680bb770 2024-05-09T03:16:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f866659c640 1 --2- 172.21.15.46:0/875250360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680d5040 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f866808e340 tx=0x7f865802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cec433682b219a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f8665d9b640 1 -- 172.21.15.46:0/875250360 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658037070 con 0x7f86680bb770 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f8665d9b640 1 -- 172.21.15.46:0/875250360 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8658032440 con 0x7f86680bb770 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f8665d9b640 1 -- 172.21.15.46:0/875250360 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658032810 con 0x7f86680bb770 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f866cfc7640 1 -- 172.21.15.46:0/875250360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 msgr2=0x7f86680d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f866cfc7640 1 --2- 172.21.15.46:0/875250360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680d5040 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f866808e340 tx=0x7f865802a270 comp rx=0 tx=0).stop 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.326+0000 7f866cfc7640 1 -- 172.21.15.46:0/875250360 shutdown_connections 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.327+0000 7f866cfc7640 1 --2- 172.21.15.46:0/875250360 >> v2:172.21.15.88:3300/0 conn(0x7f86680d3180 0x7f86681642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.327+0000 7f866cfc7640 1 --2- 172.21.15.46:0/875250360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680d5040 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.327+0000 7f866cfc7640 1 -- 172.21.15.46:0/875250360 >> 172.21.15.46:0/875250360 conn(0x7f866806e370 msgr2=0x7f866806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.327+0000 7f866cfc7640 1 -- 172.21.15.46:0/875250360 shutdown_connections 2024-05-09T03:16:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.327+0000 7f866cfc7640 1 -- 172.21.15.46:0/875250360 wait complete. 2024-05-09T03:16:57.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.329+0000 7f866cfc7640 1 Processor -- start 2024-05-09T03:16:57.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.329+0000 7f866cfc7640 1 -- start start 2024-05-09T03:16:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866cfc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866cfc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8668163eb0 0x7f866814e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866cfc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f866814ed40 0x7f86680b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866cfc7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86680d0bc0 con 0x7f86680bb770 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866cfc7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f86680bbb80 con 0x7f8668163eb0 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866cfc7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f86680d0d40 con 0x7f866814ed40 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680ac6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47962/0 (socket says 172.21.15.46:47962) 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.330+0000 7f866659c640 1 -- 172.21.15.46:0/3202096147 learned_addr learned my addr 172.21.15.46:0/3202096147 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f8666d9d640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f866814ed40 0x7f86680b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f865dd9b640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8668163eb0 0x7f866814e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866659c640 1 -- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f866814ed40 msgr2=0x7f86680b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866659c640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f866814ed40 0x7f86680b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866659c640 1 -- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8668163eb0 msgr2=0x7f866814e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866659c640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8668163eb0 0x7f866814e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866659c640 1 -- 172.21.15.46:0/3202096147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86680cb090 con 0x7f86680bb770 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f865dd9b640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8668163eb0 0x7f866814e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866659c640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680ac6c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f865802a2b0 tx=0x7f86580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658037040 con 0x7f86680bb770 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8658042070 con 0x7f86680bb770 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86680cb370 con 0x7f86680bb770 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658030880 con 0x7f86680bb770 2024-05-09T03:16:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.331+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86680cb930 con 0x7f86680bb770 2024-05-09T03:16:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.332+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f866806a270 con 0x7f86680bb770 2024-05-09T03:16:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.333+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8658030a20 con 0x7f86680bb770 2024-05-09T03:16:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.334+0000 7f865f7fe640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8648076300 0x7f8648078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.334+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f86580b81e0 con 0x7f86680bb770 2024-05-09T03:16:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.334+0000 7f865dd9b640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8648076300 0x7f8648078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.334+0000 7f865dd9b640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8648076300 0x7f8648078810 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f86681fac90 tx=0x7f8650009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.338+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8658081860 con 0x7f86680bb770 2024-05-09T03:16:57.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.542+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f86680bb0d0 con 0x7f86680bb770 2024-05-09T03:16:57.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.542+0000 7f865f7fe640 1 -- 172.21.15.46:0/3202096147 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v90) ==== 81+0+730 (secure 0 0 0) 0x7f8658081160 con 0x7f86680bb770 2024-05-09T03:16:57.544 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:16:57.544 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:16:57.544 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:16:57.544 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:16:57.545 INFO:teuthology.orchestra.run.smithi046.stderr:listed 12 entries 2024-05-09T03:16:57.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8648076300 msgr2=0x7f8648078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8648076300 0x7f8648078810 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f86681fac90 tx=0x7f8650009040 comp rx=0 tx=0).stop 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 msgr2=0x7f86680ac6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680ac6c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f865802a2b0 tx=0x7f86580046c0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 shutdown_connections 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f866814ed40 0x7f86680b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8668163eb0 0x7f866814e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8648076300 0x7f8648078810 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 --2- 172.21.15.46:0/3202096147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86680bb770 0x7f86680ac6c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 >> 172.21.15.46:0/3202096147 conn(0x7f866806e370 msgr2=0x7f8668075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 shutdown_connections 2024-05-09T03:16:57.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.544+0000 7f866cfc7640 1 -- 172.21.15.46:0/3202096147 wait complete. 2024-05-09T03:16:57.557 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:16:57.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:57.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:16:57.583 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:16:57.583 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:16:57.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:16:57.584 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:16:57.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:16:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.795+0000 7f77f5415640 1 Processor -- start 2024-05-09T03:16:57.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.795+0000 7f77f5415640 1 -- start start 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77f5415640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 0x7f77f015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77f5415640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f015f090 con 0x7f77f00d04c0 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77f5415640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f015f340 con 0x7f77f0162880 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77f5415640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77f0161da0 con 0x7f77f015e740 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77ee7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f77f00d04c0 0x7f77f00d08d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41364/0 (socket says 172.21.15.46:41364) 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77ee7fc640 1 -- 172.21.15.46:0/964436311 learned_addr learned my addr 172.21.15.46:0/964436311 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77ef7fe640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 0x7f77f015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.796+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507587321 0 0) 0x7f77f015f090 con 0x7f77f00d04c0 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77d8003520 con 0x7f77f00d04c0 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 186147791 0 0) 0x7f77f015f340 con 0x7f77f0162880 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77f015f090 con 0x7f77f0162880 2024-05-09T03:16:57.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3868112101 0 0) 0x7f77d8003520 con 0x7f77f00d04c0 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 msgr2=0x7f77f015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 0x7f77f015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 >> v1:172.21.15.88:6789/0 conn(0x7f77f0162880 legacy=0x7f77f00b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f0162080 con 0x7f77f00d04c0 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f77e4001bf0 con 0x7f77f00d04c0 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 >> v1:172.21.15.46:6789/0 conn(0x7f77f00d04c0 legacy=0x7f77f00d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 0x7f77d8003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77d8004d90 0x7f77d8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77f0161da0 con 0x7f77d80041c0 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f77d800bcc0 con 0x7f77d8004d90 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77d800bec0 con 0x7f77f015e740 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77eeffd640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 0x7f77d8003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ef7fe640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77d8004d90 0x7f77d8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ee7fc640 1 -- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 msgr2=0x7f77d8003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f015e740 0x7f77d8003c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ee7fc640 1 -- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77d8004d90 msgr2=0x7f77d8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77d8004d90 0x7f77d8005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.797+0000 7f77ee7fc640 1 -- 172.21.15.46:0/964436311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d8003520 con 0x7f77d80041c0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.798+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77d8004800 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f77e400c490 tx=0x7f77d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0e2e3da7c58491 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.798+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4032020 con 0x7f77d80041c0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.798+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e4005d40 con 0x7f77d80041c0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.798+0000 7f77edffb640 1 -- 172.21.15.46:0/964436311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e400c0e0 con 0x7f77d80041c0 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77f5415640 1 -- 172.21.15.46:0/964436311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 msgr2=0x7f77d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77f5415640 1 --2- 172.21.15.46:0/964436311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77d8004800 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f77e400c490 tx=0x7f77d8005830 comp rx=0 tx=0).stop 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77ef7fe640 1 -- 172.21.15.46:0/964436311 reap_dead start 2024-05-09T03:16:57.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77f5415640 1 -- 172.21.15.46:0/964436311 shutdown_connections 2024-05-09T03:16:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77f5415640 1 -- 172.21.15.46:0/964436311 >> 172.21.15.46:0/964436311 conn(0x7f77f00b55f0 msgr2=0x7f77f00b5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77f5415640 1 -- 172.21.15.46:0/964436311 shutdown_connections 2024-05-09T03:16:57.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.799+0000 7f77f5415640 1 -- 172.21.15.46:0/964436311 wait complete. 2024-05-09T03:16:57.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.801+0000 7f77f5415640 1 Processor -- start 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.801+0000 7f77f5415640 1 -- start start 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77f5415640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77f0152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77f5415640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77f01612d0 0x7f77f014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77f5415640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f014fc90 0x7f77f00c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77f5415640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77f015ecd0 con 0x7f77d80041c0 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77f5415640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f77f015eb50 con 0x7f77f01612d0 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77f5415640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77f015ee50 con 0x7f77f014fc90 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77f0152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.802+0000 7f77eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77f0152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47980/0 (socket says 172.21.15.46:47980) 2024-05-09T03:16:57.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 -- 172.21.15.46:0/12711383 learned_addr learned my addr 172.21.15.46:0/12711383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77f01612d0 0x7f77f014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77ef7fe640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f014fc90 0x7f77f00c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 -- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f014fc90 msgr2=0x7f77f00c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f014fc90 0x7f77f00c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 -- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77f01612d0 msgr2=0x7f77f014f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77f01612d0 0x7f77f014f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 -- 172.21.15.46:0/12711383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f00aa850 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77eeffd640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77f0152220 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f77e000f820 tx=0x7f77e000fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e000fe70 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e0018070 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f006f1d0 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e000d730 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.803+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f006f470 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.804+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77f00d1f30 con 0x7f77d80041c0 2024-05-09T03:16:57.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.805+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f77e000d8d0 con 0x7f77d80041c0 2024-05-09T03:16:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.806+0000 7f77d77fe640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77dc076300 0x7f77dc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.806+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6283+0+0 (secure 0 0 0) 0x7f77e0098230 con 0x7f77d80041c0 2024-05-09T03:16:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.806+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77dc076300 0x7f77dc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.806+0000 7f77ee7fc640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77dc076300 0x7f77dc078810 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f77e4033000 tx=0x7f77e4009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:57.810+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f77e00619a0 con 0x7f77d80041c0 2024-05-09T03:16:58.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.038+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f77f006d1b0 con 0x7f77d80041c0 2024-05-09T03:16:58.040 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:16:58.039+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:16:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.040+0000 7f77d77fe640 1 -- 172.21.15.46:0/12711383 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v128) ==== 161+0+0 (secure 0 0 0) 0x7f77e00612a0 con 0x7f77d80041c0 2024-05-09T03:16:58.042 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:16:58.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77dc076300 msgr2=0x7f77dc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:58.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77dc076300 0x7f77dc078810 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f77e4033000 tx=0x7f77e4009030 comp rx=0 tx=0).stop 2024-05-09T03:16:58.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 msgr2=0x7f77f0152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:58.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77f0152220 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f77e000f820 tx=0x7f77e000fc20 comp rx=0 tx=0).stop 2024-05-09T03:16:58.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 shutdown_connections 2024-05-09T03:16:58.045 INFO:tasks.ceph.mds.c.smithi046.stderr: -1833> 2024-05-09T03:16:48.330+0000 7ff0b2f38640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77f014fc90 0x7f77f00c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77f01612d0 0x7f77f014f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77dc076300 0x7f77dc078810 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 --2- 172.21.15.46:0/12711383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77d80041c0 0x7f77f0152220 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 >> 172.21.15.46:0/12711383 conn(0x7f77f00b55f0 msgr2=0x7f77f00b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 shutdown_connections 2024-05-09T03:16:58.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.043+0000 7f77f5415640 1 -- 172.21.15.46:0/12711383 wait complete. 2024-05-09T03:16:58.047 INFO:tasks.ceph.mds.c.smithi046.stderr: -583> 2024-05-09T03:16:55.312+0000 7ff0b6f40640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:16:58.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:16:58.103 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:16:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.173+0000 7f7b97dc7640 1 Processor -- start 2024-05-09T03:16:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.173+0000 7f7b97dc7640 1 -- start start 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b97dc7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b900b5140 0x7f7b900b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b97dc7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b900b7ca0 0x7f7b900d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b97dc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b90163f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b97dc7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7b900d3480 con 0x7f7b900b7ca0 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b97dc7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7b900b80b0 con 0x7f7b900b5140 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b97dc7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7b900b5550 con 0x7f7b900d3df0 2024-05-09T03:16:58.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b900b7ca0 0x7f7b900d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b900b7ca0 0x7f7b900d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48010/0 (socket says 172.21.15.46:48010) 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 -- 172.21.15.46:0/1536160352 learned_addr learned my addr 172.21.15.46:0/1536160352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95b63640 1 --2- 172.21.15.46:0/1536160352 >> v2:172.21.15.88:3300/0 conn(0x7f7b900b5140 0x7f7b900b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b96364640 1 --2- 172.21.15.46:0/1536160352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b90163f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 -- 172.21.15.46:0/1536160352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 msgr2=0x7f7b90163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 --2- 172.21.15.46:0/1536160352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b90163f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 -- 172.21.15.46:0/1536160352 >> v2:172.21.15.88:3300/0 conn(0x7f7b900b5140 msgr2=0x7f7b900b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 --2- 172.21.15.46:0/1536160352 >> v2:172.21.15.88:3300/0 conn(0x7f7b900b5140 0x7f7b900b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.175+0000 7f7b95362640 1 -- 172.21.15.46:0/1536160352 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b900d44e0 con 0x7f7b900b7ca0 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.176+0000 7f7b95362640 1 --2- 172.21.15.46:0/1536160352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b7ca0 0x7f7b900d38b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f7b900b88d0 tx=0x7f7b8002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac828bf52b5d91f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.176+0000 7f7b94b61640 1 -- 172.21.15.46:0/1536160352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b80037070 con 0x7f7b900b7ca0 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.176+0000 7f7b94b61640 1 -- 172.21.15.46:0/1536160352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b80032440 con 0x7f7b900b7ca0 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.176+0000 7f7b94b61640 1 -- 172.21.15.46:0/1536160352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b80032810 con 0x7f7b900b7ca0 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 -- 172.21.15.46:0/1536160352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b7ca0 msgr2=0x7f7b900d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/1536160352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b7ca0 0x7f7b900d38b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f7b900b88d0 tx=0x7f7b8002a270 comp rx=0 tx=0).stop 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 -- 172.21.15.46:0/1536160352 shutdown_connections 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/1536160352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b90163f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/1536160352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b7ca0 0x7f7b900d38b0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/1536160352 >> v2:172.21.15.88:3300/0 conn(0x7f7b900b5140 0x7f7b900b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 -- 172.21.15.46:0/1536160352 >> 172.21.15.46:0/1536160352 conn(0x7f7b9008d3f0 msgr2=0x7f7b9008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 -- 172.21.15.46:0/1536160352 shutdown_connections 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.177+0000 7f7b97dc7640 1 -- 172.21.15.46:0/1536160352 wait complete. 2024-05-09T03:16:58.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.179+0000 7f7b97dc7640 1 Processor -- start 2024-05-09T03:16:58.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.179+0000 7f7b97dc7640 1 -- start start 2024-05-09T03:16:58.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b97dc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 0x7f7b9014b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b97dc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b900b7ca0 0x7f7b9014acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b97dc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b901f9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b97dc7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7b900d4d80 con 0x7f7b900b5140 2024-05-09T03:16:58.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b97dc7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7b900d4aa0 con 0x7f7b900b7ca0 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b97dc7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7b900d4200 con 0x7f7b900d3df0 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b95b63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 0x7f7b9014b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b95b63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 0x7f7b9014b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48024/0 (socket says 172.21.15.46:48024) 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.180+0000 7f7b95b63640 1 -- 172.21.15.46:0/2491386906 learned_addr learned my addr 172.21.15.46:0/2491386906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95362640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b900b7ca0 0x7f7b9014acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b96364640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b901f9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95b63640 1 -- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 msgr2=0x7f7b901f9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95b63640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b901f9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95b63640 1 -- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b900b7ca0 msgr2=0x7f7b9014acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95b63640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b900b7ca0 0x7f7b9014acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95b63640 1 -- 172.21.15.46:0/2491386906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9014b230 con 0x7f7b900b5140 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95362640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b900b7ca0 0x7f7b9014acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b95b63640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 0x7f7b9014b470 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00d830 tx=0x7f7b8c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c018030 con 0x7f7b900b5140 2024-05-09T03:16:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8c00be10 con 0x7f7b900b5140 2024-05-09T03:16:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b90158aa0 con 0x7f7b900b5140 2024-05-09T03:16:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0098b0 con 0x7f7b900b5140 2024-05-09T03:16:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.181+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9015a870 con 0x7f7b900b5140 2024-05-09T03:16:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.182+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b9006a270 con 0x7f7b900b5140 2024-05-09T03:16:58.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.183+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7b8c01b070 con 0x7f7b900b5140 2024-05-09T03:16:58.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.184+0000 7f7b867fc640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b74076300 0x7f7b74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:58.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.184+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7f7b8c0979a0 con 0x7f7b900b5140 2024-05-09T03:16:58.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.184+0000 7f7b95362640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b74076300 0x7f7b74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:58.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.184+0000 7f7b95362640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b74076300 0x7f7b74078810 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7b80034000 tx=0x7f7b8002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:58.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.188+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7b8c061060 con 0x7f7b900b5140 2024-05-09T03:16:58.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:58.459+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f7b901559b0 con 0x7f7b900b5140 2024-05-09T03:16:59.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.048+0000 7f7b867fc640 1 -- 172.21.15.46:0/2491386906 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v129) ==== 107+0+0 (secure 0 0 0) 0x7f7b8c060960 con 0x7f7b900b5140 2024-05-09T03:16:59.050 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:16:59.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b74076300 msgr2=0x7f7b74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b74076300 0x7f7b74078810 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7b80034000 tx=0x7f7b8002e000 comp rx=0 tx=0).stop 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 msgr2=0x7f7b9014b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 0x7f7b9014b470 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00d830 tx=0x7f7b8c00dcf0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 shutdown_connections 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b900d3df0 0x7f7b901f9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b900b7ca0 0x7f7b9014acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b74076300 0x7f7b74078810 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 --2- 172.21.15.46:0/2491386906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b900b5140 0x7f7b9014b470 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 >> 172.21.15.46:0/2491386906 conn(0x7f7b9008d3f0 msgr2=0x7f7b9008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.050+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 shutdown_connections 2024-05-09T03:16:59.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.051+0000 7f7b97dc7640 1 -- 172.21.15.46:0/2491386906 wait complete. 2024-05-09T03:16:59.066 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:16:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.124+0000 7f4354011f40 1 Processor -- start 2024-05-09T03:16:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.124+0000 7f4354011f40 1 -- start start 2024-05-09T03:16:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f4354011f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dcb63a0 0x55893dca3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f4354011f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcbe5f0 0x55893dcbb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f4354011f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 0x55893dd41390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f4354011f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55893dcbf5f0 con 0x55893dd49a40 2024-05-09T03:16:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f4354011f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55893dc1cf50 con 0x55893dcbe5f0 2024-05-09T03:16:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f4354011f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55893dcbf8d0 con 0x55893dcb63a0 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 0x55893dd41390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 0x55893dd41390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48046/0 (socket says 172.21.15.46:48046) 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 -- 172.21.15.46:0/3515294072 learned_addr learned my addr 172.21.15.46:0/3515294072 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434ce66640 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dcb63a0 0x55893dca3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434c665640 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcbe5f0 0x55893dcbb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 -- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dcb63a0 msgr2=0x55893dca3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dcb63a0 0x55893dca3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 -- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcbe5f0 msgr2=0x55893dcbb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcbe5f0 0x55893dcbb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 -- 172.21.15.46:0/3515294072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55893dd4bdd0 con 0x55893dd49a40 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434ce66640 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dcb63a0 0x55893dca3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.126+0000 7f434d667640 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 0x55893dd41390 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f4344025ba0 tx=0x7f4344051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac95b4087e82856 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f434b663640 1 -- 172.21.15.46:0/3515294072 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344062070 con 0x55893dd49a40 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f434b663640 1 -- 172.21.15.46:0/3515294072 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4344054b70 con 0x55893dd49a40 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f434b663640 1 -- 172.21.15.46:0/3515294072 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4344067460 con 0x55893dd49a40 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 -- 172.21.15.46:0/3515294072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 msgr2=0x55893dd41390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 0x55893dd41390 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f4344025ba0 tx=0x7f4344051000 comp rx=0 tx=0).stop 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 -- 172.21.15.46:0/3515294072 shutdown_connections 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dd49a40 0x55893dd41390 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcbe5f0 0x55893dcbb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 --2- 172.21.15.46:0/3515294072 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dcb63a0 0x55893dca3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 -- 172.21.15.46:0/3515294072 >> 172.21.15.46:0/3515294072 conn(0x55893db98ac0 msgr2=0x55893dcb9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 -- 172.21.15.46:0/3515294072 shutdown_connections 2024-05-09T03:16:59.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.127+0000 7f4354011f40 1 -- 172.21.15.46:0/3515294072 wait complete. 2024-05-09T03:16:59.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.128+0000 7f4354011f40 1 Processor -- start 2024-05-09T03:16:59.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.128+0000 7f4354011f40 1 -- start start 2024-05-09T03:16:59.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f4354011f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcb63a0 0x55893dc778a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f4354011f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 0x55893dd3e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f4354011f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dd49a40 0x55893dd3e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f4354011f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55893dcc06e0 con 0x55893dcbe5f0 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f4354011f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55893dddb260 con 0x55893dcb63a0 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f4354011f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55893dd40d00 con 0x55893dd49a40 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 0x55893dd3e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 0x55893dd3e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48058/0 (socket says 172.21.15.46:48058) 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 learned_addr learned my addr 172.21.15.46:0/2851422585 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcb63a0 0x55893dc778a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434d667640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dd49a40 0x55893dd3e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dd49a40 msgr2=0x55893dd3e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55893dd49a40 0x55893dd3e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcb63a0 msgr2=0x55893dc778a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55893dcb63a0 0x55893dc778a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55893dd3dd40 con 0x55893dcbe5f0 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 0x55893dd3e260 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x55893dcbf240 tx=0x7f433400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.130+0000 7f434a661640 1 -- 172.21.15.46:0/2851422585 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334008070 con 0x55893dcbe5f0 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.131+0000 7f434a661640 1 -- 172.21.15.46:0/2851422585 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4334016070 con 0x55893dcbe5f0 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.131+0000 7f434a661640 1 -- 172.21.15.46:0/2851422585 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43340118a0 con 0x55893dcbe5f0 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.131+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55893dd3f5a0 con 0x55893dcbe5f0 2024-05-09T03:16:59.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.131+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55893ddd9490 con 0x55893dcbe5f0 2024-05-09T03:16:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.132+0000 7f434a661640 1 -- 172.21.15.46:0/2851422585 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4334011a40 con 0x55893dcbe5f0 2024-05-09T03:16:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.133+0000 7f434a661640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f432c074cb0 0x7f432c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.133+0000 7f434a661640 1 -- 172.21.15.46:0/2851422585 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7f4334096930 con 0x55893dcbe5f0 2024-05-09T03:16:59.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.133+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f432c074cb0 0x7f432c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.136 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:16:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.133+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55893dca1160 0x55893ddef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.133+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893dc77de0 con 0x55893dca1160 2024-05-09T03:16:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.133+0000 7f434d667640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55893dca1160 0x55893ddef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.134+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f432c074cb0 0x7f432c0771c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f433c002810 tx=0x7f433c007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.134+0000 7f434d667640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55893dca1160 0x55893ddef5d0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.138+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v91'195 uv169 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.138+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893dd3ece0 con 0x55893dca1160 2024-05-09T03:16:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.139+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v91'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.139+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddefb10 con 0x55893dca1160 2024-05-09T03:16:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.141+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v91'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.142 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:16:59.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.141+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0ec0 con 0x55893dca1160 2024-05-09T03:16:59.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.142+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v91'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.142+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1260 con 0x55893dca1160 2024-05-09T03:16:59.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.143+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v91'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.143+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddefeb0 con 0x55893dca1160 2024-05-09T03:16:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.144+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v91'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.144+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf02c0 con 0x55893dca1160 2024-05-09T03:16:59.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.146+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v91'201 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.146+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.147+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v91'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.147+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.148+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v91'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.150 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:16:59.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.148+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.149+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v91'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.150+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0ae0 con 0x55893dca1160 2024-05-09T03:16:59.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.151+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v91'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.151+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0ae0 con 0x55893dca1160 2024-05-09T03:16:59.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.152+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v91'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.152+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.153+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v91'207 uv182 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.153+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.155+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v91'208 uv208 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.155+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.156+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v91'209 uv209 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.158 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:16:59.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.156+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.157+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v91'210 uv210 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.157+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.159+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v91'211 uv211 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.159+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf0660 con 0x55893dca1160 2024-05-09T03:16:59.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.160+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v91'212 uv212 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.160+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.161+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v91'213 uv213 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.161+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.163+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v91'214 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.163+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.164+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v91'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.164+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.167 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.165+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v91'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.165+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf2610 con 0x55893dca1160 2024-05-09T03:16:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.166+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v91'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.166+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf2610 con 0x55893dca1160 2024-05-09T03:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.168+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v91'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.168+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.170+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v91'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.170+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.171+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v91'220 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.171+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.172+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v91'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.172+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.173+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v91'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.175 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:16:59.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.173+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf2610 con 0x55893dca1160 2024-05-09T03:16:59.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.175+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v91'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.175+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf2610 con 0x55893dca1160 2024-05-09T03:16:59.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.176+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v91'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.176+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf1600 con 0x55893dca1160 2024-05-09T03:16:59.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.177+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v91'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.177+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55893ddf1b10 0x55893ddf3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.177+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf4470 con 0x55893ddf1b10 2024-05-09T03:16:59.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.178+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55893ddf1b10 0x55893ddf3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.178+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55893ddf1b10 0x55893ddf3f30 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.180+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v91'151 uv133 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.180+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf4810 con 0x55893ddf1b10 2024-05-09T03:16:59.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.181+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v91'152 uv152 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.181+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf4bb0 con 0x55893ddf1b10 2024-05-09T03:16:59.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.182+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v91'153 uv153 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.184 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:16:59.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.182+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5400 con 0x55893ddf1b10 2024-05-09T03:16:59.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.184+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v91'154 uv154 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.184+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf57a0 con 0x55893ddf1b10 2024-05-09T03:16:59.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.185+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v91'155 uv155 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.185+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5b40 con 0x55893ddf1b10 2024-05-09T03:16:59.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.187+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v91'156 uv150 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.187+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.188+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v91'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.188+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.189+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v91'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.191 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:16:59.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.189+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.191+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v91'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.191+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.192+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v91'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.192+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.193+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v91'161 uv149 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.193+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.195+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v91'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.195+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.196+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v91'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.198 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:16:59.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.196+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf9940 con 0x55893ddf1b10 2024-05-09T03:16:59.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.198+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v91'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.198+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf9940 con 0x55893ddf1b10 2024-05-09T03:16:59.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.199+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v91'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.199+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.200+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v91'166 uv147 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.200+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.202+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v91'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.202+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.203+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v91'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.205 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:16:59.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.203+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf6ef0 con 0x55893ddf1b10 2024-05-09T03:16:59.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.204+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v91'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.204+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf6ef0 con 0x55893ddf1b10 2024-05-09T03:16:59.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.206+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v91'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.206+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893ddf1b10 2024-05-09T03:16:59.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.208+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v91'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddf1b10 2024-05-09T03:16:59.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.208+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893dca1160 2024-05-09T03:16:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.209+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v91'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.209+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893dca1160 2024-05-09T03:16:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.210+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v91'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.210+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893dca1160 2024-05-09T03:16:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.211+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v91'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.213 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:16:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.211+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf6ef0 con 0x55893dca1160 2024-05-09T03:16:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.213+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v91'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.213+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf6ef0 con 0x55893dca1160 2024-05-09T03:16:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.214+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v91'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.214+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5ee0 con 0x55893dca1160 2024-05-09T03:16:59.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.215+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v91'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.215+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55893ddf63f0 0x55893ddf8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.215+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf8d50 con 0x55893ddf63f0 2024-05-09T03:16:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.215+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55893ddf63f0 0x55893ddf8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.215+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55893ddf63f0 0x55893ddf8810 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.217+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v91'34 uv33 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.217+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf90f0 con 0x55893ddf63f0 2024-05-09T03:16:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.218+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v91'35 uv35 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.218+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf9490 con 0x55893ddf63f0 2024-05-09T03:16:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.219+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v91'36 uv36 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.220 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:16:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.219+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf5030 con 0x55893ddf63f0 2024-05-09T03:16:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.220+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v91'37 uv37 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.220+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf9dc0 con 0x55893dca1160 2024-05-09T03:16:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.221+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v91'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.221+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddf9dc0 con 0x55893dca1160 2024-05-09T03:16:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.223+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v91'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.223+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf9dc0 con 0x55893dca1160 2024-05-09T03:16:59.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.224+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v91'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.225 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:16:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.224+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfadd0 con 0x55893dca1160 2024-05-09T03:16:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.225+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v91'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.225+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfadd0 con 0x55893dca1160 2024-05-09T03:16:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.226+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v91'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.226+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddf9dc0 con 0x55893dca1160 2024-05-09T03:16:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.228+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v91'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.228+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55893ddfa690 0x55893ddfaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.228+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd1c0 con 0x55893ddfa690 2024-05-09T03:16:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.228+0000 7f434d667640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55893ddfa690 0x55893ddfaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.228+0000 7f434d667640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55893ddfa690 0x55893ddfaaa0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.230+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v91'70 uv64 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.230+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddfd1c0 con 0x55893ddfa690 2024-05-09T03:16:59.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.231+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v91'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.231+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd1c0 con 0x55893ddfa690 2024-05-09T03:16:59.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.232+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v91'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.234 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:16:59.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.232+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfe5c0 con 0x55893ddfa690 2024-05-09T03:16:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.233+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v91'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.233+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfe5c0 con 0x55893ddfa690 2024-05-09T03:16:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.235+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v91'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.235+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd5b0 con 0x55893ddfa690 2024-05-09T03:16:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.236+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v91'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.236+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd5b0 con 0x55893ddfa690 2024-05-09T03:16:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.237+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v91'76 uv69 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.237+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893ddfd5b0 con 0x55893ddfa690 2024-05-09T03:16:59.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.239+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v91'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.239+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd5b0 con 0x55893ddfa690 2024-05-09T03:16:59.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.240+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v91'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.241 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:16:59.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.240+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd5b0 con 0x55893ddfa690 2024-05-09T03:16:59.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.241+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v91'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.241+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893ddfd5b0 con 0x55893ddfa690 2024-05-09T03:16:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.243+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v91'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.243+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55893ddfdac0 0x55893ddffee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.243+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00540 con 0x55893ddfdac0 2024-05-09T03:16:59.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.243+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55893ddfdac0 0x55893ddffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.243+0000 7f434ce66640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55893ddfdac0 0x55893ddffee0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.245+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v91'32 uv31 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddfdac0 2024-05-09T03:16:59.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.245+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de00910 con 0x55893ddfdac0 2024-05-09T03:16:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.246+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v91'33 uv33 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddfdac0 2024-05-09T03:16:59.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.246+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893ddfdac0 2024-05-09T03:16:59.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.247+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v91'34 uv34 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddfdac0 2024-05-09T03:16:59.248 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:16:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.247+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893ddfdac0 2024-05-09T03:16:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.248+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v91'35 uv35 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddfdac0 2024-05-09T03:16:59.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.248+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893ddfdac0 2024-05-09T03:16:59.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.249+0000 7f434ce66640 1 -- 172.21.15.46:0/2851422585 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v91'36 uv36 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433c010ae0 con 0x55893ddfdac0 2024-05-09T03:16:59.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.249+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893dca1160 2024-05-09T03:16:59.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.251+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v91'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.251+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de00910 con 0x55893dca1160 2024-05-09T03:16:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.252+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v91'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.252+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893dca1160 2024-05-09T03:16:59.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.253+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v91'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.254 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:16:59.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.253+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de01920 con 0x55893dca1160 2024-05-09T03:16:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.254+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v91'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.254+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de01920 con 0x55893dca1160 2024-05-09T03:16:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.255+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v91'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.255+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893dca1160 2024-05-09T03:16:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.256+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v91'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893dca1160 2024-05-09T03:16:59.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.256+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de00910 con 0x55893ddf63f0 2024-05-09T03:16:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.257+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v91'64 uv58 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.257+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de00d00 con 0x55893ddf63f0 2024-05-09T03:16:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.258+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v91'65 uv65 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.258+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de010f0 con 0x55893ddf63f0 2024-05-09T03:16:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.259+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v91'66 uv66 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.260 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:16:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.259+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de010f0 con 0x55893ddf63f0 2024-05-09T03:16:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.260+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v91'67 uv67 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.260+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de010a0 con 0x55893ddf63f0 2024-05-09T03:16:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.261+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v91'68 uv68 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.261+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de010a0 con 0x55893ddf63f0 2024-05-09T03:16:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.262+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v91'69 uv63 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.262+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de01490 con 0x55893ddf63f0 2024-05-09T03:16:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.263+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v91'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.263+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de01880 con 0x55893ddf63f0 2024-05-09T03:16:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.264+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v91'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.266 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:16:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.264+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de01880 con 0x55893ddf63f0 2024-05-09T03:16:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.265+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v91'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.265+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de01550 con 0x55893ddf63f0 2024-05-09T03:16:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.266+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v91'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.266+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55893de016c0 0x55893de04290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.266+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893de016c0 2024-05-09T03:16:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.266+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55893de016c0 0x55893de04290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.266+0000 7f434c665640 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55893de016c0 0x55893de04290 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.267+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v91'31 uv29 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893de016c0 2024-05-09T03:16:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.267+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893de016c0 2024-05-09T03:16:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.269+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v91'32 uv32 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893de016c0 2024-05-09T03:16:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.269+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893de016c0 2024-05-09T03:16:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.270+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v91'33 uv33 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893de016c0 2024-05-09T03:16:59.271 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:16:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.270+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893de016c0 2024-05-09T03:16:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.271+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v91'34 uv34 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893de016c0 2024-05-09T03:16:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.271+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddf63f0 2024-05-09T03:16:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.272+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v91'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.272+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddf63f0 2024-05-09T03:16:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.273+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v91'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.273+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddf63f0 2024-05-09T03:16:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.274+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v91'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.276 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:16:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.274+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de057e0 con 0x55893ddf63f0 2024-05-09T03:16:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.275+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v91'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.276+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de057e0 con 0x55893ddf63f0 2024-05-09T03:16:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.277+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v91'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.277+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddf63f0 2024-05-09T03:16:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.278+0000 7f434c665640 1 -- 172.21.15.46:0/2851422585 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v91'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f433405bcc0 con 0x55893ddf63f0 2024-05-09T03:16:59.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.278+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.279+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v91'32 uv31 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.279+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.280+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v91'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.280+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.283 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:16:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.282+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.282+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.283+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.283+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.284+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.284+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.287+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v91'82 uv73 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.287+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de04810 con 0x55893ddfa690 2024-05-09T03:16:59.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.288+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v91'83 uv83 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.288+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.289+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v91'84 uv84 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.291 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:16:59.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.289+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.290+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v91'85 uv85 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.290+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.291+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v91'86 uv86 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.292+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.293+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v91'87 uv80 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.293+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.294+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v91'88 uv88 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.294+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.295+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v91'89 uv89 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.296 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:16:59.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.295+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047f0 con 0x55893ddfa690 2024-05-09T03:16:59.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.296+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v91'90 uv90 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.296+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.297+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v91'91 uv91 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.297+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de047d0 con 0x55893ddfa690 2024-05-09T03:16:59.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.298+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v91'92 uv92 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.298+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55893de04c00 con 0x55893ddfa690 2024-05-09T03:16:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v91'93 uv93 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f4344054d20 con 0x55893ddfa690 2024-05-09T03:16:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55893dca1160 msgr2=0x55893ddef5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55893dca1160 0x55893ddef5d0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55893ddf63f0 msgr2=0x55893ddf8810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55893ddf63f0 0x55893ddf8810 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55893de016c0 msgr2=0x55893de04290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55893de016c0 0x55893de04290 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55893ddf1b10 msgr2=0x55893ddf3f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55893ddf1b10 0x55893ddf3f30 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55893ddfa690 msgr2=0x55893ddfaaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55893ddfa690 0x55893ddfaaa0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55893ddfdac0 msgr2=0x55893ddffee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55893ddfdac0 0x55893ddffee0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 reap_dead start 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 reap_dead start 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f434d667640 1 -- 172.21.15.46:0/2851422585 reap_dead start 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f432c074cb0 msgr2=0x7f432c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f432c074cb0 0x7f432c0771c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f433c002810 tx=0x7f433c007040 comp rx=0 tx=0).stop 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 msgr2=0x55893dd3e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 0x55893dd3e260 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x55893dcbf240 tx=0x7f433400c880 comp rx=0 tx=0).stop 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 shutdown_connections 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f432c074cb0 0x7f432c0771c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 --2- 172.21.15.46:0/2851422585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55893dcbe5f0 0x55893dd3e260 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 >> 172.21.15.46:0/2851422585 conn(0x55893db98ac0 msgr2=0x55893dcbb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 shutdown_connections 2024-05-09T03:16:59.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.300+0000 7f4354011f40 1 -- 172.21.15.46:0/2851422585 wait complete. 2024-05-09T03:16:59.305 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 500.00000000 2024-05-09T03:16:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.399+0000 7f283d51bf40 1 Processor -- start 2024-05-09T03:16:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.399+0000 7f283d51bf40 1 -- start start 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f283d51bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f438043ff0 0x55f438111d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f283d51bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 0x55f4380722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f283d51bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f438041de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f283d51bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f438044400 con 0x55f4380462b0 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f283d51bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f437fe9f50 con 0x55f438043ff0 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f283d51bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f43808d000 con 0x55f43808a060 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f2835d33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 0x55f4380722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f2835d33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 0x55f4380722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48076/0 (socket says 172.21.15.46:48076) 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f2835d33640 1 -- 172.21.15.46:0/832343650 learned_addr learned my addr 172.21.15.46:0/832343650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f2836534640 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f438043ff0 0x55f438111d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.400+0000 7f2836d35640 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f438041de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2835d33640 1 -- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 msgr2=0x55f438041de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2835d33640 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f438041de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2835d33640 1 -- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f438043ff0 msgr2=0x55f438111d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2835d33640 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f438043ff0 0x55f438111d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2835d33640 1 -- 172.21.15.46:0/832343650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f43811a900 con 0x55f4380462b0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2836d35640 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f438041de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2835d33640 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 0x55f4380722a0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f2820001eb0 tx=0x7f282000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69400b9d32766696 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2834d31640 1 -- 172.21.15.46:0/832343650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2820008070 con 0x55f4380462b0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2834d31640 1 -- 172.21.15.46:0/832343650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f282000d500 con 0x55f4380462b0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.401+0000 7f2834d31640 1 -- 172.21.15.46:0/832343650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282000d8d0 con 0x55f4380462b0 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 -- 172.21.15.46:0/832343650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 msgr2=0x55f4380722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 0x55f4380722a0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f2820001eb0 tx=0x7f282000b6a0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 -- 172.21.15.46:0/832343650 shutdown_connections 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f438041de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f4380462b0 0x55f4380722a0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 --2- 172.21.15.46:0/832343650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f438043ff0 0x55f438111d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 -- 172.21.15.46:0/832343650 >> 172.21.15.46:0/832343650 conn(0x55f437f65ac0 msgr2=0x55f438072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 -- 172.21.15.46:0/832343650 shutdown_connections 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.402+0000 7f283d51bf40 1 -- 172.21.15.46:0/832343650 wait complete. 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.403+0000 7f283d51bf40 1 Processor -- start 2024-05-09T03:16:59.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.403+0000 7f283d51bf40 1 -- start start 2024-05-09T03:16:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f283d51bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f438043ff0 0x55f438116090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f283d51bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4380462b0 0x55f438042b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f283d51bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f4380430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f283d51bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f438071b30 con 0x55f438043ff0 2024-05-09T03:16:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f283d51bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f438087cc0 con 0x55f4380462b0 2024-05-09T03:16:59.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f283d51bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f438073350 con 0x55f43808a060 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f438043ff0 0x55f438116090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f438043ff0 0x55f438116090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48082/0 (socket says 172.21.15.46:48082) 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 -- 172.21.15.46:0/599548986 learned_addr learned my addr 172.21.15.46:0/599548986 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2835d33640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4380462b0 0x55f438042b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836d35640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f4380430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 -- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 msgr2=0x55f4380430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f43808a060 0x55f4380430c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 -- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4380462b0 msgr2=0x55f438042b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f4380462b0 0x55f438042b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.404+0000 7f2836534640 1 -- 172.21.15.46:0/599548986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4381b3850 con 0x55f438043ff0 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.405+0000 7f2836534640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f438043ff0 0x55f438116090 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f2830057040 tx=0x7f2830002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.405+0000 7f282f7fe640 1 -- 172.21.15.46:0/599548986 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830064030 con 0x55f438043ff0 2024-05-09T03:16:59.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.405+0000 7f282f7fe640 1 -- 172.21.15.46:0/599548986 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28300548b0 con 0x55f438043ff0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.405+0000 7f282f7fe640 1 -- 172.21.15.46:0/599548986 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2830059040 con 0x55f438043ff0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.405+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f4381b3b30 con 0x55f438043ff0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.405+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4381b2160 con 0x55f438043ff0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.406+0000 7f282f7fe640 1 -- 172.21.15.46:0/599548986 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2830067030 con 0x55f438043ff0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.407+0000 7f282f7fe640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2818074ce0 0x7f28180771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.407+0000 7f282f7fe640 1 -- 172.21.15.46:0/599548986 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7f28300da090 con 0x55f438043ff0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.407+0000 7f2835d33640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2818074ce0 0x7f28180771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.407+0000 7f283d51bf40 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f4381165d0 0x55f4381b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.407+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91) -- 0x55f4381af970 con 0x55f4381165d0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.407+0000 7f2836d35640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f4381165d0 0x55f4381b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.408+0000 7f2835d33640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2818074ce0 0x7f28180771f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2820001eb0 tx=0x7f2820009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.408+0000 7f2836d35640 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f4381165d0 0x55f4381b1bb0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f2836d35640 1 -- 172.21.15.46:0/599548986 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 500.00000000 [delete] v91'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f282800b9c0 con 0x55f4381165d0 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f4381165d0 msgr2=0x55f4381b1bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f4381165d0 0x55f4381b1bb0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2818074ce0 msgr2=0x7f28180771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2818074ce0 0x7f28180771f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2820001eb0 tx=0x7f2820009000 comp rx=0 tx=0).stop 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f438043ff0 msgr2=0x55f438116090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 --2- 172.21.15.46:0/599548986 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f438043ff0 0x55f438116090 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f2830057040 tx=0x7f2830002fb0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f2836d35640 1 -- 172.21.15.46:0/599548986 reap_dead start 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 shutdown_connections 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 >> 172.21.15.46:0/599548986 conn(0x55f437f65ac0 msgr2=0x55f438070ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.409+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 shutdown_connections 2024-05-09T03:16:59.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.410+0000 7f283d51bf40 1 -- 172.21.15.46:0/599548986 wait complete. 2024-05-09T03:16:59.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:16:59.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.530+0000 7f5aaea2b640 1 Processor -- start 2024-05-09T03:16:59.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.530+0000 7f5aaea2b640 1 -- start start 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.531+0000 7f5aaea2b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5aa80c78a0 0x7f5aa80b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aaea2b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aa81461a0 0x7f5aa81465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aaea2b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa80c8720 con 0x7f5aa8072500 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aaea2b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa8145d10 con 0x7f5aa81461a0 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aaea2b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa80c7e80 con 0x7f5aa80c78a0 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aa81461a0 0x7f5aa81465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aa81461a0 0x7f5aa81465b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48090/0 (socket says 172.21.15.46:48090) 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 -- 172.21.15.46:0/835520406 learned_addr learned my addr 172.21.15.46:0/835520406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aa7fff640 1 --2- 172.21.15.46:0/835520406 >> v2:172.21.15.88:3300/0 conn(0x7f5aa80c78a0 0x7f5aa80b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 -- 172.21.15.46:0/835520406 >> v2:172.21.15.88:3300/0 conn(0x7f5aa80c78a0 msgr2=0x7f5aa80b1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:16:59.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 --2- 172.21.15.46:0/835520406 >> v2:172.21.15.88:3300/0 conn(0x7f5aa80c78a0 0x7f5aa80b1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 -- 172.21.15.46:0/835520406 >> v1:172.21.15.88:6789/0 conn(0x7f5aa8072500 legacy=0x7f5aa8163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.532+0000 7f5aacfc8640 1 -- 172.21.15.46:0/835520406 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa80c8160 con 0x7f5aa81461a0 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.533+0000 7f5aacfc8640 1 --2- 172.21.15.46:0/835520406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa81461a0 0x7f5aa81465b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80cd000 tx=0x7f5aa002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46e43503454bf1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.533+0000 7f5aa6ffd640 1 -- 172.21.15.46:0/835520406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0036070 con 0x7f5aa81461a0 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.533+0000 7f5aa6ffd640 1 -- 172.21.15.46:0/835520406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa0031600 con 0x7f5aa81461a0 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.533+0000 7f5aa6ffd640 1 -- 172.21.15.46:0/835520406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa00319d0 con 0x7f5aa81461a0 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 -- 172.21.15.46:0/835520406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa81461a0 msgr2=0x7f5aa81465b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/835520406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa81461a0 0x7f5aa81465b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80cd000 tx=0x7f5aa002a730 comp rx=0 tx=0).stop 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 -- 172.21.15.46:0/835520406 shutdown_connections 2024-05-09T03:16:59.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/835520406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa81461a0 0x7f5aa81465b0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/835520406 >> v2:172.21.15.88:3300/0 conn(0x7f5aa80c78a0 0x7f5aa80b1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 -- 172.21.15.46:0/835520406 >> 172.21.15.46:0/835520406 conn(0x7f5aa80705c0 msgr2=0x7f5aa80709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 -- 172.21.15.46:0/835520406 shutdown_connections 2024-05-09T03:16:59.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.534+0000 7f5aaea2b640 1 -- 172.21.15.46:0/835520406 wait complete. 2024-05-09T03:16:59.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.535+0000 7f5aaea2b640 1 Processor -- start 2024-05-09T03:16:59.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.536+0000 7f5aaea2b640 1 -- start start 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aaea2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 0x7f5aa8135650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aaea2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5aa80c78a0 0x7f5aa80b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aaea2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aa81461a0 0x7f5aa815f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aaea2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8072910 con 0x7f5aa8072500 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aaea2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa8164560 con 0x7f5aa81461a0 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aaea2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa80d2c40 con 0x7f5aa80c78a0 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 0x7f5aa8135650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 0x7f5aa8135650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48094/0 (socket says 172.21.15.46:48094) 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 -- 172.21.15.46:0/43168193 learned_addr learned my addr 172.21.15.46:0/43168193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:16:59.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aacfc8640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aa81461a0 0x7f5aa815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa77fe640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5aa80c78a0 0x7f5aa80b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 -- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5aa80c78a0 msgr2=0x7f5aa80b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5aa80c78a0 0x7f5aa80b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 -- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aa81461a0 msgr2=0x7f5aa815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aa81461a0 0x7f5aa815f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.537+0000 7f5aa7fff640 1 -- 172.21.15.46:0/43168193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa81fa950 con 0x7f5aa8072500 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.538+0000 7f5aa7fff640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 0x7f5aa8135650 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e870 tx=0x7f5a9800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.538+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98017070 con 0x7f5aa8072500 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.538+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9800ee60 con 0x7f5aa8072500 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.538+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa81fa0b0 con 0x7f5aa8072500 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.538+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9800cb60 con 0x7f5aa8072500 2024-05-09T03:16:59.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.538+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa81fa670 con 0x7f5aa8072500 2024-05-09T03:16:59.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.539+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa8132db0 con 0x7f5aa8072500 2024-05-09T03:16:59.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.540+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5a9801a070 con 0x7f5aa8072500 2024-05-09T03:16:59.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.540+0000 7f5aa4ff9640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a940762d0 0x7f5a940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:16:59.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.540+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7f5a98097a10 con 0x7f5aa8072500 2024-05-09T03:16:59.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.540+0000 7f5aa77fe640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a940762d0 0x7f5a940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:16:59.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.541+0000 7f5aa77fe640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a940762d0 0x7f5a940787e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80cf2e0 tx=0x7f5a9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:16:59.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.545+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5a980609e0 con 0x7f5aa8072500 2024-05-09T03:16:59.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.775+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f5aa806a270 con 0x7f5aa8072500 2024-05-09T03:16:59.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.974+0000 7f5aa4ff9640 1 -- 172.21.15.46:0/43168193 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v130) ==== 162+0+0 (secure 0 0 0) 0x7f5a98064ce0 con 0x7f5aa8072500 2024-05-09T03:16:59.976 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a940762d0 msgr2=0x7f5a940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a940762d0 0x7f5a940787e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80cf2e0 tx=0x7f5a9c009040 comp rx=0 tx=0).stop 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 msgr2=0x7f5aa8135650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 0x7f5aa8135650 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e870 tx=0x7f5a9800ed30 comp rx=0 tx=0).stop 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 shutdown_connections 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5aa81461a0 0x7f5aa815f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5aa80c78a0 0x7f5aa80b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a940762d0 0x7f5a940787e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 --2- 172.21.15.46:0/43168193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aa8072500 0x7f5aa8135650 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:16:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 >> 172.21.15.46:0/43168193 conn(0x7f5aa80705c0 msgr2=0x7f5aa80b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:16:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 shutdown_connections 2024-05-09T03:16:59.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:16:59.976+0000 7f5aaea2b640 1 -- 172.21.15.46:0/43168193 wait complete. 2024-05-09T03:17:00.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:00.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.519+0000 7ff24635e640 1 Processor -- start 2024-05-09T03:17:00.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.519+0000 7ff24635e640 1 -- start start 2024-05-09T03:17:00.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff24635e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff24015e6a0 0x7ff240070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff24635e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2401620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff24635e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff24015b7e0 0x7ff24015bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff24635e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff24015eab0 con 0x7ff24015e6a0 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff24635e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff240162630 con 0x7ff24015b7e0 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff24635e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff24015c130 con 0x7ff240161ce0 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff24015e6a0 0x7ff240070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff24015e6a0 0x7ff240070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48102/0 (socket says 172.21.15.46:48102) 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 -- 172.21.15.46:0/1790580910 learned_addr learned my addr 172.21.15.46:0/1790580910 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23f7fe640 1 --2- 172.21.15.46:0/1790580910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2401620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff2448fb640 1 --2- 172.21.15.46:0/1790580910 >> v2:172.21.15.88:3300/0 conn(0x7ff24015b7e0 0x7ff24015bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 -- 172.21.15.46:0/1790580910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 msgr2=0x7ff2401620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 --2- 172.21.15.46:0/1790580910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2401620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 -- 172.21.15.46:0/1790580910 >> v2:172.21.15.88:3300/0 conn(0x7ff24015b7e0 msgr2=0x7ff24015bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 --2- 172.21.15.46:0/1790580910 >> v2:172.21.15.88:3300/0 conn(0x7ff24015b7e0 0x7ff24015bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.521+0000 7ff23ffff640 1 -- 172.21.15.46:0/1790580910 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff240070a60 con 0x7ff24015e6a0 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.522+0000 7ff23ffff640 1 --2- 172.21.15.46:0/1790580910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015e6a0 0x7ff240070230 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ff240074610 tx=0x7ff23002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9ab0dc6b3a335 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.522+0000 7ff23effd640 1 -- 172.21.15.46:0/1790580910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230037070 con 0x7ff24015e6a0 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.522+0000 7ff23effd640 1 -- 172.21.15.46:0/1790580910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff230032440 con 0x7ff24015e6a0 2024-05-09T03:17:00.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.522+0000 7ff23effd640 1 -- 172.21.15.46:0/1790580910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230032810 con 0x7ff24015e6a0 2024-05-09T03:17:00.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 -- 172.21.15.46:0/1790580910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015e6a0 msgr2=0x7ff240070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:00.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 --2- 172.21.15.46:0/1790580910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015e6a0 0x7ff240070230 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7ff240074610 tx=0x7ff23002a140 comp rx=0 tx=0).stop 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 -- 172.21.15.46:0/1790580910 shutdown_connections 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 --2- 172.21.15.46:0/1790580910 >> v2:172.21.15.88:3300/0 conn(0x7ff24015b7e0 0x7ff24015bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 --2- 172.21.15.46:0/1790580910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2401620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 --2- 172.21.15.46:0/1790580910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015e6a0 0x7ff240070230 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 -- 172.21.15.46:0/1790580910 >> 172.21.15.46:0/1790580910 conn(0x7ff2400d0630 msgr2=0x7ff2400d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 -- 172.21.15.46:0/1790580910 shutdown_connections 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.523+0000 7ff24635e640 1 -- 172.21.15.46:0/1790580910 wait complete. 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.525+0000 7ff24635e640 1 Processor -- start 2024-05-09T03:17:00.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.525+0000 7ff24635e640 1 -- start start 2024-05-09T03:17:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff24635e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 0x7ff2401e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff24635e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff24015e6a0 0x7ff240148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff24635e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2400c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff24635e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400d1900 con 0x7ff24015b7e0 2024-05-09T03:17:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff24635e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff240071020 con 0x7ff24015e6a0 2024-05-09T03:17:00.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff24635e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff240070770 con 0x7ff240161ce0 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 0x7ff2401e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 0x7ff2401e7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48116/0 (socket says 172.21.15.46:48116) 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.526+0000 7ff23ffff640 1 -- 172.21.15.46:0/2469627307 learned_addr learned my addr 172.21.15.46:0/2469627307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23f7fe640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff24015e6a0 0x7ff240148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff2448fb640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2400c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23ffff640 1 -- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 msgr2=0x7ff2400c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23ffff640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2400c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23ffff640 1 -- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff24015e6a0 msgr2=0x7ff240148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23ffff640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff24015e6a0 0x7ff240148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23ffff640 1 -- 172.21.15.46:0/2469627307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2400c8220 con 0x7ff24015b7e0 2024-05-09T03:17:00.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23f7fe640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff24015e6a0 0x7ff240148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff2448fb640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2400c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23ffff640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 0x7ff2401e7df0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff23002a180 tx=0x7ff2300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230041020 con 0x7ff24015b7e0 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2300306c0 con 0x7ff24015b7e0 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2400c8500 con 0x7ff24015b7e0 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff230037040 con 0x7ff24015b7e0 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.527+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2401e78d0 con 0x7ff24015b7e0 2024-05-09T03:17:00.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.528+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff24006a270 con 0x7ff24015b7e0 2024-05-09T03:17:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.529+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff230044020 con 0x7ff24015b7e0 2024-05-09T03:17:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.529+0000 7ff23cff9640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff22c076300 0x7ff22c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:00.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.530+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7ff2300b8150 con 0x7ff24015b7e0 2024-05-09T03:17:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.530+0000 7ff23f7fe640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff22c076300 0x7ff22c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:00.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.530+0000 7ff23f7fe640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff22c076300 0x7ff22c078810 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff234006fd0 tx=0x7ff234008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:00.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.534+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff23004b030 con 0x7ff24015b7e0 2024-05-09T03:17:00.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.766+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2400cdae0 con 0x7ff24015b7e0 2024-05-09T03:17:00.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.766+0000 7ff23cff9640 1 -- 172.21.15.46:0/2469627307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) ==== 95+0+5177 (secure 0 0 0) 0x7ff230081770 con 0x7ff24015b7e0 2024-05-09T03:17:00.768 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:00.768 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":130,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:16:59.974213+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7938},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7938":{"gid":7938,"name":"c","rank":0,"incarnation":130,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/3044304069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3044304069},{"type":"v1","addr":"172.21.15.46:6834","nonce":3044304069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:00.769 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 130 2024-05-09T03:17:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff22c076300 msgr2=0x7ff22c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff22c076300 0x7ff22c078810 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff234006fd0 tx=0x7ff234008040 comp rx=0 tx=0).stop 2024-05-09T03:17:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 msgr2=0x7ff2401e7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 0x7ff2401e7df0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7ff23002a180 tx=0x7ff2300046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 shutdown_connections 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff240161ce0 0x7ff2400c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff24015e6a0 0x7ff240148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff22c076300 0x7ff22c078810 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 --2- 172.21.15.46:0/2469627307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff24015b7e0 0x7ff2401e7df0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.768+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 >> 172.21.15.46:0/2469627307 conn(0x7ff2400d0630 msgr2=0x7ff2400b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.769+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 shutdown_connections 2024-05-09T03:17:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:00.769+0000 7ff24635e640 1 -- 172.21.15.46:0/2469627307 wait complete. 2024-05-09T03:17:00.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:16:59.974213+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 91, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7938}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7938': {'gid': 7938, 'name': 'c', 'rank': 0, 'incarnation': 130, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/3044304069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 3044304069}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 3044304069}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:17:00.781 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7938:up:replay 2024-05-09T03:17:00.782 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:17:04.056 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:04.055+0000 7f43a7321640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:17:06.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:06.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.313+0000 7fc9412f9640 1 Processor -- start 2024-05-09T03:17:06.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.314+0000 7fc9412f9640 1 -- start start 2024-05-09T03:17:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc9412f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc93c0b5210 0x7fc93c1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc9412f9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc93c0d4800 0x7fc93c0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc9412f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b77c0 0x7fc93c0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc9412f9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc93c0b5620 con 0x7fc93c0d4800 2024-05-09T03:17:06.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc9412f9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc93c0b8110 con 0x7fc93c0b5210 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc9412f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc93c0d5280 con 0x7fc93c0b77c0 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc93c0d4800 0x7fc93c0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc93c0d4800 0x7fc93c0d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40368/0 (socket says 172.21.15.46:40368) 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 -- 172.21.15.46:0/807274697 learned_addr learned my addr 172.21.15.46:0/807274697 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93affd640 1 --2- 172.21.15.46:0/807274697 >> v2:172.21.15.88:3300/0 conn(0x7fc93c0b5210 0x7fc93c1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93b7fe640 1 --2- 172.21.15.46:0/807274697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b77c0 0x7fc93c0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 -- 172.21.15.46:0/807274697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b77c0 msgr2=0x7fc93c0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 --2- 172.21.15.46:0/807274697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b77c0 0x7fc93c0b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 -- 172.21.15.46:0/807274697 >> v2:172.21.15.88:3300/0 conn(0x7fc93c0b5210 msgr2=0x7fc93c1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.315+0000 7fc93a7fc640 1 --2- 172.21.15.46:0/807274697 >> v2:172.21.15.88:3300/0 conn(0x7fc93c0b5210 0x7fc93c1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.316+0000 7fc93a7fc640 1 -- 172.21.15.46:0/807274697 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc93c0d5560 con 0x7fc93c0d4800 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.316+0000 7fc93a7fc640 1 --2- 172.21.15.46:0/807274697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0d4800 0x7fc93c0d4c10 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc93c08ed70 tx=0x7fc93002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93eac86be6a114b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.316+0000 7fc939ffb640 1 -- 172.21.15.46:0/807274697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930037070 con 0x7fc93c0d4800 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc939ffb640 1 -- 172.21.15.46:0/807274697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc930032440 con 0x7fc93c0d4800 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc939ffb640 1 -- 172.21.15.46:0/807274697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930032810 con 0x7fc93c0d4800 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 -- 172.21.15.46:0/807274697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0d4800 msgr2=0x7fc93c0d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 --2- 172.21.15.46:0/807274697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0d4800 0x7fc93c0d4c10 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc93c08ed70 tx=0x7fc93002a140 comp rx=0 tx=0).stop 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 -- 172.21.15.46:0/807274697 shutdown_connections 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 --2- 172.21.15.46:0/807274697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b77c0 0x7fc93c0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 --2- 172.21.15.46:0/807274697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0d4800 0x7fc93c0d4c10 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 --2- 172.21.15.46:0/807274697 >> v2:172.21.15.88:3300/0 conn(0x7fc93c0b5210 0x7fc93c1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 -- 172.21.15.46:0/807274697 >> 172.21.15.46:0/807274697 conn(0x7fc93c0d29a0 msgr2=0x7fc93c0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 -- 172.21.15.46:0/807274697 shutdown_connections 2024-05-09T03:17:06.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.317+0000 7fc9412f9640 1 -- 172.21.15.46:0/807274697 wait complete. 2024-05-09T03:17:06.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.319+0000 7fc9412f9640 1 Processor -- start 2024-05-09T03:17:06.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.319+0000 7fc9412f9640 1 -- start start 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9412f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b5210 0x7fc93c156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9412f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 0x7fc93c0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9412f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc93c0d4800 0x7fc93c0cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9412f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c0d0ae0 con 0x7fc93c0b77c0 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9412f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc93c160ea0 con 0x7fc93c0d4800 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9412f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc93c163e60 con 0x7fc93c0b5210 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 0x7fc93c0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 0x7fc93c0cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40382/0 (socket says 172.21.15.46:40382) 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 -- 172.21.15.46:0/4202265907 learned_addr learned my addr 172.21.15.46:0/4202265907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93b7fe640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc93c0d4800 0x7fc93c0cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93affd640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b5210 0x7fc93c156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 -- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b5210 msgr2=0x7fc93c156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b5210 0x7fc93c156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 -- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc93c0d4800 msgr2=0x7fc93c0cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc93c0d4800 0x7fc93c0cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 -- 172.21.15.46:0/4202265907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc93c1fabe0 con 0x7fc93c0b77c0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93affd640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b5210 0x7fc93c156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc93a7fc640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 0x7fc93c0cb990 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc93002a180 tx=0x7fc9300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.321+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930037040 con 0x7fc93c0b77c0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.322+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc930042070 con 0x7fc93c0b77c0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.322+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc93c157020 con 0x7fc93c0b77c0 2024-05-09T03:17:06.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.322+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930030880 con 0x7fc93c0b77c0 2024-05-09T03:17:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.322+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc93c1ea0d0 con 0x7fc93c0b77c0 2024-05-09T03:17:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.322+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc93c06a270 con 0x7fc93c0b77c0 2024-05-09T03:17:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.323+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc930030a20 con 0x7fc93c0b77c0 2024-05-09T03:17:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.324+0000 7fc9237fe640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9280762d0 0x7fc9280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.324+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7fc9300b8040 con 0x7fc93c0b77c0 2024-05-09T03:17:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.324+0000 7fc93affd640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9280762d0 0x7fc9280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.325+0000 7fc93affd640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9280762d0 0x7fc9280787e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009fd0 tx=0x7fc92c00d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.329+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc930081790 con 0x7fc93c0b77c0 2024-05-09T03:17:06.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.558+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc93c0cc4a0 con 0x7fc93c0b77c0 2024-05-09T03:17:06.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.558+0000 7fc9237fe640 1 -- 172.21.15.46:0/4202265907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+5184 (secure 0 0 0) 0x7fc930081090 con 0x7fc93c0b77c0 2024-05-09T03:17:06.560 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:06.560 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":133,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:02.979321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7938},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7938":{"gid":7938,"name":"c","rank":0,"incarnation":130,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/3044304069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3044304069},{"type":"v1","addr":"172.21.15.46:6834","nonce":3044304069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7938,"qdb_cluster":[7938]},"id":6}]} 2024-05-09T03:17:06.561 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 133 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9280762d0 msgr2=0x7fc9280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9280762d0 0x7fc9280787e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009fd0 tx=0x7fc92c00d040 comp rx=0 tx=0).stop 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 msgr2=0x7fc93c0cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 0x7fc93c0cb990 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc93002a180 tx=0x7fc9300046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 shutdown_connections 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc93c0d4800 0x7fc93c0cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9280762d0 0x7fc9280787e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc93c0b77c0 0x7fc93c0cb990 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 --2- 172.21.15.46:0/4202265907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc93c0b5210 0x7fc93c156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 >> 172.21.15.46:0/4202265907 conn(0x7fc93c0d29a0 msgr2=0x7fc93c0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 shutdown_connections 2024-05-09T03:17:06.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.561+0000 7fc9412f9640 1 -- 172.21.15.46:0/4202265907 wait complete. 2024-05-09T03:17:06.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 133, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:17:02.979321+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 91, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7938}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7938': {'gid': 7938, 'name': 'c', 'rank': 0, 'incarnation': 130, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/3044304069', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 3044304069}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 3044304069}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7938, 'qdb_cluster': [7938]} 2024-05-09T03:17:06.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:17:06.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:17:06.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.688+0000 7fcbd9ebd640 1 Processor -- start 2024-05-09T03:17:06.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.688+0000 7fcbd9ebd640 1 -- start start 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.689+0000 7fcbd9ebd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcbd40d3730 0x7fcbd406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.689+0000 7fcbd9ebd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcbd4165090 0x7fcbd40d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.689+0000 7fcbd9ebd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbd409d470 con 0x7fcbd406e7e0 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.689+0000 7fcbd9ebd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd409cfe0 con 0x7fcbd4165090 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.689+0000 7fcbd9ebd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd40b94a0 con 0x7fcbd40d3730 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcbd4165090 0x7fcbd40d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcbd4165090 0x7fcbd40d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40396/0 (socket says 172.21.15.46:40396) 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 -- 172.21.15.46:0/956085751 learned_addr learned my addr 172.21.15.46:0/956085751 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/956085751 >> v2:172.21.15.88:3300/0 conn(0x7fcbd40d3730 0x7fcbd406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 -- 172.21.15.46:0/956085751 >> v2:172.21.15.88:3300/0 conn(0x7fcbd40d3730 msgr2=0x7fcbd406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/956085751 >> v2:172.21.15.88:3300/0 conn(0x7fcbd40d3730 0x7fcbd406e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 -- 172.21.15.46:0/956085751 >> v1:172.21.15.88:6789/0 conn(0x7fcbd406e7e0 legacy=0x7fcbd4164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.690+0000 7fcbd3fff640 1 -- 172.21.15.46:0/956085751 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd40d11b0 con 0x7fcbd4165090 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.691+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/956085751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4165090 0x7fcbd40d1f50 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcbd406d990 tx=0x7fcbcc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ded92c078c5aff7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.691+0000 7fcbd27fc640 1 -- 172.21.15.46:0/956085751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc036070 con 0x7fcbd4165090 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.691+0000 7fcbd27fc640 1 -- 172.21.15.46:0/956085751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbcc031600 con 0x7fcbd4165090 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.691+0000 7fcbd27fc640 1 -- 172.21.15.46:0/956085751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0319d0 con 0x7fcbd4165090 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/956085751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4165090 msgr2=0x7fcbd40d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/956085751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4165090 0x7fcbd40d1f50 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fcbd406d990 tx=0x7fcbcc02a6a0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/956085751 shutdown_connections 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/956085751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd4165090 0x7fcbd40d1f50 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/956085751 >> v2:172.21.15.88:3300/0 conn(0x7fcbd40d3730 0x7fcbd406e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/956085751 >> 172.21.15.46:0/956085751 conn(0x7fcbd40d05f0 msgr2=0x7fcbd40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/956085751 shutdown_connections 2024-05-09T03:17:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.692+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/956085751 wait complete. 2024-05-09T03:17:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.693+0000 7fcbd9ebd640 1 Processor -- start 2024-05-09T03:17:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.694+0000 7fcbd9ebd640 1 -- start start 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd9ebd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbd406e7e0 0x7fcbd40b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd9ebd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 0x7fcbd40ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd9ebd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbd4165090 0x7fcbd41f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd9ebd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd40cfbb0 con 0x7fcbd40d3730 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd9ebd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd40b9160 con 0x7fcbd406e7e0 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd9ebd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd40cfd30 con 0x7fcbd4165090 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 0x7fcbd40ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 0x7fcbd40ccd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40406/0 (socket says 172.21.15.46:40406) 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/279145854 learned_addr learned my addr 172.21.15.46:0/279145854 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbd406e7e0 0x7fcbd40b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.695+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbd4165090 0x7fcbd41f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbd4165090 msgr2=0x7fcbd41f9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbd4165090 0x7fcbd41f9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbd406e7e0 msgr2=0x7fcbd40b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbd406e7e0 0x7fcbd40b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/279145854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd40a1390 con 0x7fcbd40d3730 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 0x7fcbd40ccd50 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800c3d0 tx=0x7fcbc800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbbbfff640 1 -- 172.21.15.46:0/279145854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8017070 con 0x7fcbd40d3730 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbbbfff640 1 -- 172.21.15.46:0/279145854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc80052d0 con 0x7fcbd40d3730 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd40a1670 con 0x7fcbd40d3730 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbbbfff640 1 -- 172.21.15.46:0/279145854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8003830 con 0x7fcbd40d3730 2024-05-09T03:17:06.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.696+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd409dce0 con 0x7fcbd40d3730 2024-05-09T03:17:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.698+0000 7fcbbbfff640 1 -- 172.21.15.46:0/279145854 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcbc801a070 con 0x7fcbd40d3730 2024-05-09T03:17:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.698+0000 7fcbbbfff640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcbc0076300 0x7fcbc0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.698+0000 7fcbbbfff640 1 -- 172.21.15.46:0/279145854 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7fcbc8096c60 con 0x7fcbd40d3730 2024-05-09T03:17:06.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.699+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcbc0076300 0x7fcbc0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.699+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcbc0076300 0x7fcbc0078810 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009aa0 tx=0x7fcbc4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.699+0000 7fcbe0d72740 1 Processor -- start 2024-05-09T03:17:06.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.699+0000 7fcbe0d72740 1 -- start start 2024-05-09T03:17:06.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d3deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d58be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e794d590 0x5604e7d59120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5604e7c764e0 con 0x5604e79453b0 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5604e7c11ff0 con 0x5604e796e270 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5604e7c724a0 con 0x5604e794d590 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d58be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d58be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40408/0 (socket says 172.21.15.46:40408) 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/2556723021 learned_addr learned my addr 172.21.15.46:0/2556723021 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d3deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e794d590 0x5604e7d59120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e794d590 msgr2=0x5604e7d59120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e794d590 0x5604e7d59120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 msgr2=0x5604e7d3deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d3deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/2556723021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e7bf8720 con 0x5604e79453b0 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d58be0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fcbc805c190 tx=0x7fcbc805f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a04bd5fb248a328 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.701+0000 7fcbb97fa640 1 -- 172.21.15.46:0/2556723021 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8079070 con 0x5604e79453b0 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbb97fa640 1 -- 172.21.15.46:0/2556723021 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc805fba0 con 0x5604e79453b0 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbb97fa640 1 -- 172.21.15.46:0/2556723021 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc805f000 con 0x5604e79453b0 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2556723021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 msgr2=0x5604e7d58be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d58be0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fcbc805c190 tx=0x7fcbc805f3e0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2556723021 shutdown_connections 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e794d590 0x5604e7d59120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d58be0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2556723021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d3deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2556723021 >> 172.21.15.46:0/2556723021 conn(0x5604e79623c0 msgr2=0x5604e7d13510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2556723021 shutdown_connections 2024-05-09T03:17:06.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.702+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2556723021 wait complete. 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.703+0000 7fcbe0d72740 1 -- start start 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.703+0000 7fcbe0d72740 1 Processor -- start 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e79453b0 0x5604e7d53db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7d388c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d56b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5604e7bfd760 con 0x5604e794d590 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5604e7d55c70 con 0x5604e796e270 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5604e7d573a0 con 0x5604e79453b0 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbe0d72740 10 client.0 _ll_register_callbacks cb 0x5604e7c204b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7d388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7d388c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40418/0 (socket says 172.21.15.46:40418) 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/1100653610 learned_addr learned my addr 172.21.15.46:0/1100653610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d56b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e79453b0 0x5604e7d53db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e79453b0 msgr2=0x5604e7d53db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e79453b0 0x5604e7d53db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 msgr2=0x5604e7d56b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d56b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/1100653610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e7c3ebd0 con 0x5604e794d590 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e79453b0 0x5604e7d53db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7d388c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800a040 tx=0x7fcbc8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4e6376bf90c144 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.705+0000 7fcbb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbb27fc640 1 -- 172.21.15.46:0/1100653610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8079040 con 0x5604e794d590 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbb27fc640 1 -- 172.21.15.46:0/1100653610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7fcbc805f010 con 0x5604e794d590 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbc805fba0 con 0x5604e794d590 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbb27fc640 1 -- 172.21.15.46:0/1100653610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7fcbc8063ea0 con 0x5604e794d590 2024-05-09T03:17:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 10 client.7998 fetch_fsmap learned FSMap version 133 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) -- 0x5604e7c24280 con 0x5604e794d590 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbb27fc640 1 -- 172.21.15.46:0/1100653610 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc805fba0 con 0x5604e794d590 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbb27fc640 1 -- 172.21.15.46:0/1100653610 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 133) ==== 3494+0+0 (secure 0 0 0) 0x7fcbc805b510 con 0x5604e794d590 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 10 client.7998 fetch_fsmap finished waiting for FSMap version 133 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 10 client.7998 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 20 client.7998 populate_metadata read hostname 'smithi046' 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e792ce40 0x5604e7d6ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 4 client.7998 mds_command: new command op to 7938 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 --> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604e7c840f0 con 0x5604e792ce40 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.706+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e792ce40 0x5604e7d6ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.707+0000 7fcbb27fc640 10 client.7998 ms_handle_connect on v2:172.21.15.46:6832/3044304069 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.707+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e792ce40 0x5604e7d6ba60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d17698093c183698 server_cookie=151aa5aebc7cc8e3 in_seq=0 out_seq=0 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.707+0000 7fcbb27fc640 1 -- 172.21.15.46:0/1100653610 <== mds.0 v2:172.21.15.46:6832/3044304069 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5604e7c840f0 con 0x5604e792ce40 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.707+0000 7fcbb27fc640 10 client.7998 handle_command_reply: tid=0 2024-05-09T03:17:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.707+0000 7fcbe0d72740 1 client.7998 shutdown 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 msgr2=0x5604e7d388c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7d388c0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800a040 tx=0x7fcbc8000bc0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 shutdown_connections 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e792ce40 0x5604e7d6ba60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e796e270 0x5604e7d56b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7d388c0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbb27fc640 0 client.7998 ms_handle_reset on v2:172.21.15.46:6832/3044304069 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/1100653610 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e79453b0 0x5604e7d53db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 >> 172.21.15.46:0/1100653610 conn(0x5604e79623c0 msgr2=0x5604e7d3deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 shutdown_connections 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 1 -- 172.21.15.46:0/1100653610 wait complete. 2024-05-09T03:17:06.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.708+0000 7fcbe0d72740 20 client.7998 trim_cache size 0 max 16384 2024-05-09T03:17:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.727+0000 7fcbe0d72740 1 Processor -- start 2024-05-09T03:17:06.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.727+0000 7fcbe0d72740 1 -- start start 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d53d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e794d590 0x5604e7d11bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d132c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd4202530 con 0x5604e79453b0 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd41f7030 con 0x5604e794d590 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd4201f10 con 0x5604e796e270 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d53d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d53d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40434/0 (socket says 172.21.15.46:40434) 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 -- 172.21.15.46:0/2213415557 learned_addr learned my addr 172.21.15.46:0/2213415557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 -- 172.21.15.46:0/2213415557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 msgr2=0x5604e7d132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 -- 172.21.15.46:0/2213415557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e794d590 msgr2=0x5604e7d11bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e794d590 0x5604e7d11bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 -- 172.21.15.46:0/2213415557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e79d9850 con 0x5604e79453b0 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d53d90 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x5604e7d59950 tx=0x7fcbc4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36574d2a61b9c8d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbb8ff9640 1 -- 172.21.15.46:0/2213415557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4031070 con 0x5604e79453b0 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbb8ff9640 1 -- 172.21.15.46:0/2213415557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc40036d0 con 0x5604e79453b0 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.729+0000 7fcbb8ff9640 1 -- 172.21.15.46:0/2213415557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4003aa0 con 0x5604e79453b0 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2213415557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 msgr2=0x5604e7d53d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d53d90 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x5604e7d59950 tx=0x7fcbc4018000 comp rx=0 tx=0).stop 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2213415557 shutdown_connections 2024-05-09T03:17:06.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e794d590 0x5604e7d11bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/2213415557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e79453b0 0x5604e7d53d90 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2213415557 >> 172.21.15.46:0/2213415557 conn(0x5604e79623c0 msgr2=0x5604e7d3deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2213415557 shutdown_connections 2024-05-09T03:17:06.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.730+0000 7fcbe0d72740 1 -- 172.21.15.46:0/2213415557 wait complete. 2024-05-09T03:17:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.731+0000 7fcbe0d72740 1 -- start start 2024-05-09T03:17:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.731+0000 7fcbe0d72740 1 Processor -- start 2024-05-09T03:17:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e79453b0 0x5604e7d386d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7dd5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d56010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5604e7d54800 con 0x5604e794d590 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5604e7d57130 con 0x5604e79453b0 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5604e7d55780 con 0x5604e796e270 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 10 client.0 _ll_register_callbacks cb 0x7fcbc8077070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7dd5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7dd5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40442/0 (socket says 172.21.15.46:40442) 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/3905493473 learned_addr learned my addr 172.21.15.46:0/3905493473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e79453b0 0x5604e7d386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d56010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 msgr2=0x5604e7d56010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d56010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e79453b0 msgr2=0x5604e7d386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e79453b0 0x5604e7d386d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 -- 172.21.15.46:0/3905493473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e7a506b0 con 0x5604e794d590 2024-05-09T03:17:06.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd3fff640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d56010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbd2ffd640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7dd5df0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8064060 tx=0x7fcbc8094d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f85ff9dae96b07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbb27fc640 1 -- 172.21.15.46:0/3905493473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc8079040 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbb27fc640 1 -- 172.21.15.46:0/3905493473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7fcbc805b510 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.733+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbc40036d0 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbb27fc640 1 -- 172.21.15.46:0/3905493473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) ==== 24+0+0 (secure 0 0 0) 0x7fcbc805fba0 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 10 client.8004 fetch_fsmap learned FSMap version 133 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) -- 0x5604e7c32630 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbb27fc640 1 -- 172.21.15.46:0/3905493473 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc40036d0 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbb27fc640 1 -- 172.21.15.46:0/3905493473 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 133) ==== 3494+0+0 (secure 0 0 0) 0x7fcbc8068ba0 con 0x5604e794d590 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 10 client.8004 fetch_fsmap finished waiting for FSMap version 133 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 10 client.8004 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 20 client.8004 populate_metadata read hostname 'smithi046' 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e7d56610 0x5604e7de0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 4 client.8004 mds_command: new command op to 7938 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 --> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] -- command(tid 0: {"prefix": "status"}) -- 0x5604e7c7df60 con 0x5604e7d56610 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e7d56610 0x5604e7de0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbb27fc640 10 client.8004 ms_handle_connect on v2:172.21.15.46:6832/3044304069 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.734+0000 7fcbd37fe640 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e7d56610 0x5604e7de0b70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d656cf7c650773 server_cookie=9b4f209ae29128a8 in_seq=0 out_seq=0 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbb27fc640 1 -- 172.21.15.46:0/3905493473 <== mds.0 v2:172.21.15.46:6832/3044304069 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5604e7c7df60 con 0x5604e7d56610 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbb27fc640 10 client.8004 handle_command_reply: tid=0 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 client.8004 shutdown 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 msgr2=0x5604e7dd5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7dd5df0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8064060 tx=0x7fcbc8094d20 comp rx=0 tx=0).stop 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 shutdown_connections 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:6832/3044304069,v1:172.21.15.46:6834/3044304069] conn(0x5604e7d56610 0x5604e7de0b70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5604e796e270 0x5604e7d56010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5604e794d590 0x5604e7dd5df0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbb27fc640 0 client.8004 ms_handle_reset on v2:172.21.15.46:6832/3044304069 2024-05-09T03:17:06.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 --2- 172.21.15.46:0/3905493473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5604e79453b0 0x5604e7d386d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 >> 172.21.15.46:0/3905493473 conn(0x5604e79623c0 msgr2=0x5604e7d37c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 shutdown_connections 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 1 -- 172.21.15.46:0/3905493473 wait complete. 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.735+0000 7fcbe0d72740 20 client.8004 trim_cache size 0 max 16384 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 7938, 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:17:06.742 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:17:06.743 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7605207180000004, 2024-05-09T03:17:06.743 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 133, 2024-05-09T03:17:06.743 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 91, 2024-05-09T03:17:06.743 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 91, 2024-05-09T03:17:06.743 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 8.6321714309999997 2024-05-09T03:17:06.743 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcbc0076300 msgr2=0x7fcbc0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcbc0076300 0x7fcbc0078810 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009aa0 tx=0x7fcbc4009190 comp rx=0 tx=0).stop 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 msgr2=0x7fcbd40ccd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 0x7fcbd40ccd50 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800c3d0 tx=0x7fcbc800c890 comp rx=0 tx=0).stop 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 shutdown_connections 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcbd4165090 0x7fcbd41f9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcbc0076300 0x7fcbc0078810 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbd40d3730 0x7fcbd40ccd50 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 --2- 172.21.15.46:0/279145854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbd406e7e0 0x7fcbd40b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 >> 172.21.15.46:0/279145854 conn(0x7fcbd40d05f0 msgr2=0x7fcbd40cf370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 shutdown_connections 2024-05-09T03:17:06.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.736+0000 7fcbd9ebd640 1 -- 172.21.15.46:0/279145854 wait complete. 2024-05-09T03:17:06.748 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:17:06.749 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 500.00000000', proceeding to ls 2024-05-09T03:17:06.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:17:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.862+0000 7f0218ee0640 1 Processor -- start 2024-05-09T03:17:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.862+0000 7f0218ee0640 1 -- start start 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.863+0000 7f0218ee0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f021408cd60 0x7f02140d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.863+0000 7f0218ee0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02140d1c80 0x7f02140cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.863+0000 7f0218ee0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02140d2090 0x7f02140cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.863+0000 7f0218ee0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f02140b5340 con 0x7f02140d1c80 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.863+0000 7f0218ee0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f02140d4900 con 0x7f02140d2090 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.863+0000 7f0218ee0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02140b4fd0 con 0x7f021408cd60 2024-05-09T03:17:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02140d1c80 0x7f02140cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02140d1c80 0x7f02140cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40454/0 (socket says 172.21.15.46:40454) 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 -- 172.21.15.46:0/2171744049 learned_addr learned my addr 172.21.15.46:0/2171744049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0212d9d640 1 --2- 172.21.15.46:0/2171744049 >> v2:172.21.15.88:3300/0 conn(0x7f02140d2090 0x7f02140cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f021259c640 1 --2- 172.21.15.46:0/2171744049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f021408cd60 0x7f02140d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 -- 172.21.15.46:0/2171744049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f021408cd60 msgr2=0x7f02140d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 --2- 172.21.15.46:0/2171744049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f021408cd60 0x7f02140d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 -- 172.21.15.46:0/2171744049 >> v2:172.21.15.88:3300/0 conn(0x7f02140d2090 msgr2=0x7f02140cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 --2- 172.21.15.46:0/2171744049 >> v2:172.21.15.88:3300/0 conn(0x7f02140d2090 0x7f02140cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.864+0000 7f0211d9b640 1 -- 172.21.15.46:0/2171744049 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02140d2c00 con 0x7f02140d1c80 2024-05-09T03:17:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.865+0000 7f0211d9b640 1 --2- 172.21.15.46:0/2171744049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d1c80 0x7f02140cf2b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f02140d3940 tx=0x7f020802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554374be2a3e56fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.865+0000 7f021159a640 1 -- 172.21.15.46:0/2171744049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208037070 con 0x7f02140d1c80 2024-05-09T03:17:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.865+0000 7f021159a640 1 -- 172.21.15.46:0/2171744049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0208032440 con 0x7f02140d1c80 2024-05-09T03:17:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.865+0000 7f021159a640 1 -- 172.21.15.46:0/2171744049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0208032810 con 0x7f02140d1c80 2024-05-09T03:17:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 -- 172.21.15.46:0/2171744049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d1c80 msgr2=0x7f02140cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 --2- 172.21.15.46:0/2171744049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d1c80 0x7f02140cf2b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f02140d3940 tx=0x7f020802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 -- 172.21.15.46:0/2171744049 shutdown_connections 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 --2- 172.21.15.46:0/2171744049 >> v2:172.21.15.88:3300/0 conn(0x7f02140d2090 0x7f02140cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 --2- 172.21.15.46:0/2171744049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d1c80 0x7f02140cf2b0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 --2- 172.21.15.46:0/2171744049 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f021408cd60 0x7f02140d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 -- 172.21.15.46:0/2171744049 >> 172.21.15.46:0/2171744049 conn(0x7f02140d3c00 msgr2=0x7f02140d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 -- 172.21.15.46:0/2171744049 shutdown_connections 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.866+0000 7f0218ee0640 1 -- 172.21.15.46:0/2171744049 wait complete. 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.868+0000 7f0218ee0640 1 Processor -- start 2024-05-09T03:17:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.868+0000 7f0218ee0640 1 -- start start 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0218ee0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 0x7f02141e8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0218ee0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02140d1c80 0x7f02141f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0218ee0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d2090 0x7f02140ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0218ee0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02140b92b0 con 0x7f021408cd60 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0218ee0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f02140b9130 con 0x7f02140d1c80 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0218ee0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0214162310 con 0x7f02140d2090 2024-05-09T03:17:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f021259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 0x7f02141e8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f021259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 0x7f02141e8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40456/0 (socket says 172.21.15.46:40456) 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f021259c640 1 -- 172.21.15.46:0/3297914693 learned_addr learned my addr 172.21.15.46:0/3297914693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0211d9b640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02140d1c80 0x7f02141f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.869+0000 7f0212d9d640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d2090 0x7f02140ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f021259c640 1 -- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d2090 msgr2=0x7f02140ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f021259c640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d2090 0x7f02140ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f021259c640 1 -- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02140d1c80 msgr2=0x7f02141f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f021259c640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02140d1c80 0x7f02141f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f021259c640 1 -- 172.21.15.46:0/3297914693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02141f89e0 con 0x7f021408cd60 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f021259c640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 0x7f02141e8b50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f020400d5f0 tx=0x7f020400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204004240 con 0x7f021408cd60 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0204019070 con 0x7f021408cd60 2024-05-09T03:17:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204003610 con 0x7f021408cd60 2024-05-09T03:17:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02140b2b40 con 0x7f021408cd60 2024-05-09T03:17:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.870+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02141e9090 con 0x7f021408cd60 2024-05-09T03:17:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.871+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02140b2930 con 0x7f021408cd60 2024-05-09T03:17:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.872+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f02040037b0 con 0x7f021408cd60 2024-05-09T03:17:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.872+0000 7f0202ffd640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f40762d0 0x7f01f40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.873+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7f0204098550 con 0x7f021408cd60 2024-05-09T03:17:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.873+0000 7f0211d9b640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f40762d0 0x7f01f40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.873+0000 7f0211d9b640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f40762d0 0x7f01f40787e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0208034000 tx=0x7f020802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:06.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:06.876+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0204061510 con 0x7f021408cd60 2024-05-09T03:17:07.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.110+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f021406a270 con 0x7f021408cd60 2024-05-09T03:17:07.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.110+0000 7f0202ffd640 1 -- 172.21.15.46:0/3297914693 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v133) ==== 53+0+73 (secure 0 0 0) 0x7f0204065810 con 0x7f021408cd60 2024-05-09T03:17:07.112 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f40762d0 msgr2=0x7f01f40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f40762d0 0x7f01f40787e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0208034000 tx=0x7f020802e000 comp rx=0 tx=0).stop 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 msgr2=0x7f02141e8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 0x7f02141e8b50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f020400d5f0 tx=0x7f020400dab0 comp rx=0 tx=0).stop 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 shutdown_connections 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d2090 0x7f02140ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:07.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02140d1c80 0x7f02141f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:07.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f40762d0 0x7f01f40787e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:07.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 --2- 172.21.15.46:0/3297914693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f021408cd60 0x7f02141e8b50 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:07.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 >> 172.21.15.46:0/3297914693 conn(0x7f02140d3c00 msgr2=0x7f02140b6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:07.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 shutdown_connections 2024-05-09T03:17:07.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07.113+0000 7f0218ee0640 1 -- 172.21.15.46:0/3297914693 wait complete. 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:17:07.126 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:17:07.127 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:17:07.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.127 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:07.144 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42090sec preferred_lft 42090sec 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.145 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:07.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.147 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:17:07.209 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:17:07.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.211 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:17:07.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:17:07.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:17:07.271 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:17:07.271 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:17:07.271 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:17:07.271 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:17:07.271 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:17:07.271 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:17:07.271 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:17:07.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:17:07.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:17:07.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:17:07.272 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:17:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:17:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:17:07.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.430 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:17:07.445 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:17:07.446 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:17:07.446 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:17:07.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:17:07.512 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:17:07.513 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:17:07.513 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:07.569 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:17:07.570 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:17:07.570 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:07.625 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:17:07.692 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:17:07.749 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:17:07.750 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:17:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:07.752 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:07.752 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:07.752 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:07.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.753 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:17:07.810 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:17:07.810 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:17:07.853 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:17:07.878 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:17:07.938 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:17:07.939 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:07.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:07.940 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:17:07.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:17:07.947+0000 7f74b89da480 -1 init, newargv = 0x7f74a80057c0 newargc=15 2024-05-09T03:17:07.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[68637]: starting ceph client 2024-05-09T03:17:07.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[68637]: starting fuse 2024-05-09T03:17:07.965 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:17:07.965 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:17:07.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:07.966 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:17:07.967 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:17:08.089 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.68637.asok 2024-05-09T03:17:08.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:17:08.097 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68637.asok 2024-05-09T03:17:08.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:08.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68637.asok status 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "68637", 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:08.245 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8016, 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 8016 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1469279690 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:08.246 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1469279690 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.8016 192.168.0.1:0/1469279690", 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1469279690", 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 133, 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 91, 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:17:08.247 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:17:08.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:08.254 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:17:08.254 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:08.270 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:17:08.270 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:08.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:08.270 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:08.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:08.336 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:17:08.379 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:17:08.397 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:17:08.397 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:17:08.397 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:08.397 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:17:08.397 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs 2024-05-09T03:17:08.397 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 500.00000000' did not caused DAMAGED state 2024-05-09T03:17:08.397 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds_snaptable' 2024-05-09T03:17:08.397 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:08.397 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:08.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:17:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.611+0000 7efef1813640 1 Processor -- start 2024-05-09T03:17:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.611+0000 7efef1813640 1 -- start start 2024-05-09T03:17:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efef1813640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 0x7efeec0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efef1813640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0b92c0 con 0x7efeec0b3800 2024-05-09T03:17:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efef1813640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeec0b2f10 con 0x7efeec0d1110 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efef1813640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efeec0d0430 con 0x7efeec0d0020 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efeeb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efeec0b3800 0x7efeec0ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48522/0 (socket says 172.21.15.46:48522) 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efeeb7fe640 1 -- 172.21.15.46:0/1700284703 learned_addr learned my addr 172.21.15.46:0/1700284703 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efeea7fc640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 0x7efeec0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.612+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1253232252 0 0) 0x7efeec0b92c0 con 0x7efeec0b3800 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed8003520 con 0x7efeec0b3800 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3129894354 0 0) 0x7efeec0b2f10 con 0x7efeec0d1110 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec0b92c0 con 0x7efeec0d1110 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3771582026 0 0) 0x7efed8003520 con 0x7efeec0b3800 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 msgr2=0x7efeec0b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 0x7efeec0b32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 >> v1:172.21.15.88:6789/0 conn(0x7efeec0d1110 legacy=0x7efeec0d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec0b9560 con 0x7efeec0b3800 2024-05-09T03:17:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7efee4001710 con 0x7efeec0b3800 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 >> v1:172.21.15.46:6789/0 conn(0x7efeec0b3800 legacy=0x7efeec0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efed8003c10 0x7efed8004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efed8004800 0x7efed8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 0x7efed800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efeec0d0430 con 0x7efed8003c10 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efed8005590 con 0x7efed8004800 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efed800bf00 con 0x7efeec0d0020 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeaffd640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efed8003c10 0x7efed8004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeea7fc640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efed8004800 0x7efed8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeb7fe640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 0x7efed800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeaffd640 1 -- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 msgr2=0x7efed800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeaffd640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec0d0020 0x7efed800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeaffd640 1 -- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efed8004800 msgr2=0x7efed8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeaffd640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efed8004800 0x7efed8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.613+0000 7efeeaffd640 1 -- 172.21.15.46:0/1700284703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed8003520 con 0x7efed8003c10 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.614+0000 7efeeaffd640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efed8003c10 0x7efed8004060 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7efeec0743b0 tx=0x7efed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd5727da54babc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.614+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc001c20 con 0x7efed8003c10 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.614+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efedc006d70 con 0x7efed8003c10 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.614+0000 7efee9ffb640 1 -- 172.21.15.46:0/1700284703 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc003bb0 con 0x7efed8003c10 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efef1813640 1 -- 172.21.15.46:0/1700284703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efed8003c10 msgr2=0x7efed8004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efef1813640 1 --2- 172.21.15.46:0/1700284703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efed8003c10 0x7efed8004060 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7efeec0743b0 tx=0x7efed8005830 comp rx=0 tx=0).stop 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efeeb7fe640 1 -- 172.21.15.46:0/1700284703 reap_dead start 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efef1813640 1 -- 172.21.15.46:0/1700284703 shutdown_connections 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efef1813640 1 -- 172.21.15.46:0/1700284703 >> 172.21.15.46:0/1700284703 conn(0x7efeec0d05f0 msgr2=0x7efeec0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efef1813640 1 -- 172.21.15.46:0/1700284703 shutdown_connections 2024-05-09T03:17:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.615+0000 7efef1813640 1 -- 172.21.15.46:0/1700284703 wait complete. 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.617+0000 7efef1813640 1 Processor -- start 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.617+0000 7efef1813640 1 -- start start 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efef1813640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efed8003c10 0x7efeec1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efef1813640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 0x7efeec1613f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efef1813640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0b7410 0x7efeec0b7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efef1813640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efeec09ceb0 con 0x7efeec160fe0 2024-05-09T03:17:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efef1813640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efeec0d2030 con 0x7efeec0b7410 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efef1813640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efeec09d030 con 0x7efed8003c10 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efeea7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 0x7efeec1613f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efeea7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 0x7efeec1613f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40480/0 (socket says 172.21.15.46:40480) 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.618+0000 7efeea7fc640 1 -- 172.21.15.46:0/2743513789 learned_addr learned my addr 172.21.15.46:0/2743513789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeeaffd640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efed8003c10 0x7efeec1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeeb7fe640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0b7410 0x7efeec0b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeea7fc640 1 -- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efed8003c10 msgr2=0x7efeec1564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeea7fc640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efed8003c10 0x7efeec1564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeea7fc640 1 -- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0b7410 msgr2=0x7efeec0b7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeea7fc640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0b7410 0x7efeec0b7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeea7fc640 1 -- 172.21.15.46:0/2743513789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec0b7d60 con 0x7efeec160fe0 2024-05-09T03:17:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efeea7fc640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 0x7efeec1613f0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7efed4011960 tx=0x7efed4011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4019070 con 0x7efeec160fe0 2024-05-09T03:17:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed4004240 con 0x7efeec160fe0 2024-05-09T03:17:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeec156a20 con 0x7efeec160fe0 2024-05-09T03:17:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed400f7c0 con 0x7efeec160fe0 2024-05-09T03:17:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.619+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeec156cf0 con 0x7efeec160fe0 2024-05-09T03:17:08.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.620+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeec0ce050 con 0x7efeec160fe0 2024-05-09T03:17:08.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.621+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efed401c070 con 0x7efeec160fe0 2024-05-09T03:17:08.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.622+0000 7efee3fff640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efecc0762d0 0x7efecc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:08.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.622+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7efed4098090 con 0x7efeec160fe0 2024-05-09T03:17:08.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.622+0000 7efeeaffd640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efecc0762d0 0x7efecc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:08.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.622+0000 7efeeaffd640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efecc0762d0 0x7efecc0787e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efedc031000 tx=0x7efedc033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:08.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.626+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efeec0ce050 con 0x7efeec160fe0 2024-05-09T03:17:08.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.828+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efeec160560 con 0x7efeec160fe0 2024-05-09T03:17:08.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.829+0000 7efee3fff640 1 -- 172.21.15.46:0/2743513789 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v91) ==== 81+0+852 (secure 0 0 0) 0x7efeec0b7d60 con 0x7efeec160fe0 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:08.831 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:08.832 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:08.832 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:08.832 INFO:teuthology.orchestra.run.smithi046.stderr:listed 14 entries 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efecc0762d0 msgr2=0x7efecc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efecc0762d0 0x7efecc0787e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efedc031000 tx=0x7efedc033040 comp rx=0 tx=0).stop 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 msgr2=0x7efeec1613f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 0x7efeec1613f0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7efed4011960 tx=0x7efed4011e20 comp rx=0 tx=0).stop 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 shutdown_connections 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0b7410 0x7efeec0b7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efecc0762d0 0x7efecc0787e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec160fe0 0x7efeec1613f0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 --2- 172.21.15.46:0/2743513789 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efed8003c10 0x7efeec1564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 >> 172.21.15.46:0/2743513789 conn(0x7efeec0d05f0 msgr2=0x7efeec153e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 shutdown_connections 2024-05-09T03:17:08.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:08.831+0000 7efef1813640 1 -- 172.21.15.46:0/2743513789 wait complete. 2024-05-09T03:17:08.844 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:08.844 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:08.861 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:17:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.015+0000 7f8c280d8640 1 Processor -- start 2024-05-09T03:17:09.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.015+0000 7f8c280d8640 1 -- start start 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c280d8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c280d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c2008cd60 0x7f8c2008d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c280d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c2008d6b0 0x7f8c200d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c280d8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8c200b5440 con 0x7f8c2008cd60 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c280d8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8c200cf4e0 con 0x7f8c2008d6b0 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c280d8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8c200cf7c0 con 0x7f8c200b6160 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c2008cd60 0x7f8c2008d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c2008cd60 0x7f8c2008d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40492/0 (socket says 172.21.15.46:40492) 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 -- 172.21.15.46:0/2571237746 learned_addr learned my addr 172.21.15.46:0/2571237746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c26675640 1 --2- 172.21.15.46:0/2571237746 >> v2:172.21.15.88:3300/0 conn(0x7f8c2008d6b0 0x7f8c200d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25e74640 1 --2- 172.21.15.46:0/2571237746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 -- 172.21.15.46:0/2571237746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 msgr2=0x7f8c200b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:09.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 --2- 172.21.15.46:0/2571237746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 -- 172.21.15.46:0/2571237746 >> v2:172.21.15.88:3300/0 conn(0x7f8c2008d6b0 msgr2=0x7f8c200d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:09.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 --2- 172.21.15.46:0/2571237746 >> v2:172.21.15.88:3300/0 conn(0x7f8c2008d6b0 0x7f8c200d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.017+0000 7f8c25673640 1 -- 172.21.15.46:0/2571237746 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c200cfaa0 con 0x7f8c2008cd60 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.018+0000 7f8c25673640 1 --2- 172.21.15.46:0/2571237746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008cd60 0x7f8c2008d170 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f8c200b9a10 tx=0x7f8c1002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bd5f695a92a51a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.018+0000 7f8c24e72640 1 -- 172.21.15.46:0/2571237746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10037070 con 0x7f8c2008cd60 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.018+0000 7f8c24e72640 1 -- 172.21.15.46:0/2571237746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c10032440 con 0x7f8c2008cd60 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.018+0000 7f8c24e72640 1 -- 172.21.15.46:0/2571237746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10032810 con 0x7f8c2008cd60 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 -- 172.21.15.46:0/2571237746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008cd60 msgr2=0x7f8c2008d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 --2- 172.21.15.46:0/2571237746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008cd60 0x7f8c2008d170 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f8c200b9a10 tx=0x7f8c1002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 -- 172.21.15.46:0/2571237746 shutdown_connections 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 --2- 172.21.15.46:0/2571237746 >> v2:172.21.15.88:3300/0 conn(0x7f8c2008d6b0 0x7f8c200d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 --2- 172.21.15.46:0/2571237746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008cd60 0x7f8c2008d170 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 --2- 172.21.15.46:0/2571237746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 -- 172.21.15.46:0/2571237746 >> 172.21.15.46:0/2571237746 conn(0x7f8c200b55d0 msgr2=0x7f8c200b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 -- 172.21.15.46:0/2571237746 shutdown_connections 2024-05-09T03:17:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.019+0000 7f8c280d8640 1 -- 172.21.15.46:0/2571237746 wait complete. 2024-05-09T03:17:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.021+0000 7f8c280d8640 1 Processor -- start 2024-05-09T03:17:09.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.021+0000 7f8c280d8640 1 -- start start 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c280d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2008cd60 0x7f8c20153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c280d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 0x7f8c200c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c280d8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c280d8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c200b8ea0 con 0x7f8c2008d6b0 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c280d8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2008dac0 con 0x7f8c2008cd60 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c280d8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8c200b9020 con 0x7f8c200b6160 2024-05-09T03:17:09.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c25673640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 0x7f8c200c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c25673640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 0x7f8c200c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40498/0 (socket says 172.21.15.46:40498) 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.022+0000 7f8c25673640 1 -- 172.21.15.46:0/640263087 learned_addr learned my addr 172.21.15.46:0/640263087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c26675640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25e74640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2008cd60 0x7f8c20153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25673640 1 -- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 msgr2=0x7f8c200c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25673640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25673640 1 -- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2008cd60 msgr2=0x7f8c20153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25673640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2008cd60 0x7f8c20153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25673640 1 -- 172.21.15.46:0/640263087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c200cb700 con 0x7f8c2008d6b0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c25673640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 0x7f8c200c97b0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f8c1002a220 tx=0x7f8c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10041020 con 0x7f8c2008d6b0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c1003d620 con 0x7f8c2008d6b0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c10037040 con 0x7f8c2008d6b0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2015c990 con 0x7f8c2008d6b0 2024-05-09T03:17:09.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.023+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c200cb9a0 con 0x7f8c2008d6b0 2024-05-09T03:17:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.024+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2006a270 con 0x7f8c2008d6b0 2024-05-09T03:17:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.025+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c10044020 con 0x7f8c2008d6b0 2024-05-09T03:17:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.026+0000 7f8c167fc640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c040762d0 0x7f8c040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.026+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7f8c100b8330 con 0x7f8c2008d6b0 2024-05-09T03:17:09.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.026+0000 7f8c25e74640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c040762d0 0x7f8c040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.026+0000 7f8c25e74640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c040762d0 0x7f8c040787e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c200747c0 tx=0x7f8c1c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.030+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8c1004b030 con 0x7f8c2008d6b0 2024-05-09T03:17:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.234+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8c200b3df0 con 0x7f8c2008d6b0 2024-05-09T03:17:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.235+0000 7f8c167fc640 1 -- 172.21.15.46:0/640263087 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v91) ==== 81+0+852 (secure 0 0 0) 0x7f8c10081a40 con 0x7f8c2008d6b0 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:09.237 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:09.238 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:09.238 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:09.238 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:09.238 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:09.238 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:listed 14 entries 2024-05-09T03:17:09.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c040762d0 msgr2=0x7f8c040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c040762d0 0x7f8c040787e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8c200747c0 tx=0x7f8c1c00b040 comp rx=0 tx=0).stop 2024-05-09T03:17:09.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 msgr2=0x7f8c200c97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 0x7f8c200c97b0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f8c1002a220 tx=0x7f8c100046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 shutdown_connections 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c200b6160 0x7f8c200c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c040762d0 0x7f8c040787e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c2008d6b0 0x7f8c200c97b0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 --2- 172.21.15.46:0/640263087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c2008cd60 0x7f8c20153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 >> 172.21.15.46:0/640263087 conn(0x7f8c200b55d0 msgr2=0x7f8c20071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 shutdown_connections 2024-05-09T03:17:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.237+0000 7f8c280d8640 1 -- 172.21.15.46:0/640263087 wait complete. 2024-05-09T03:17:09.250 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:17:09.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:09.250 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:09.276 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:17:09.276 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:09.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:09.277 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:09.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:09.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.486+0000 7fceed08b640 1 Processor -- start 2024-05-09T03:17:09.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.486+0000 7fceed08b640 1 -- start start 2024-05-09T03:17:09.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fceed08b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcee80bb770 0x7fcee80d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fceed08b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcee8163eb0 0x7fcee81642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fceed08b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80d2a70 0x7fcee80d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fceed08b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcee80d4bb0 con 0x7fcee80bb770 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fceed08b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcee80d33c0 con 0x7fcee8163eb0 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fceed08b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcee8164900 con 0x7fcee80d2a70 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fcee6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcee80bb770 0x7fcee80d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fcee6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcee80bb770 0x7fcee80d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40500/0 (socket says 172.21.15.46:40500) 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.487+0000 7fcee6d9d640 1 -- 172.21.15.46:0/275886653 learned_addr learned my addr 172.21.15.46:0/275886653 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee759e640 1 --2- 172.21.15.46:0/275886653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80d2a70 0x7fcee80d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee659c640 1 --2- 172.21.15.46:0/275886653 >> v2:172.21.15.88:3300/0 conn(0x7fcee8163eb0 0x7fcee81642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee6d9d640 1 -- 172.21.15.46:0/275886653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80d2a70 msgr2=0x7fcee80d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/275886653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80d2a70 0x7fcee80d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee6d9d640 1 -- 172.21.15.46:0/275886653 >> v2:172.21.15.88:3300/0 conn(0x7fcee8163eb0 msgr2=0x7fcee81642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/275886653 >> v2:172.21.15.88:3300/0 conn(0x7fcee8163eb0 0x7fcee81642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee6d9d640 1 -- 172.21.15.46:0/275886653 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee8164be0 con 0x7fcee80bb770 2024-05-09T03:17:09.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee659c640 1 --2- 172.21.15.46:0/275886653 >> v2:172.21.15.88:3300/0 conn(0x7fcee8163eb0 0x7fcee81642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.488+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/275886653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee80bb770 0x7fcee80d5040 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fcee80d6d00 tx=0x7fced802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e149ed09f3e810 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fcee5d9b640 1 -- 172.21.15.46:0/275886653 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8037070 con 0x7fcee80bb770 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fcee5d9b640 1 -- 172.21.15.46:0/275886653 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fced8032440 con 0x7fcee80bb770 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fcee5d9b640 1 -- 172.21.15.46:0/275886653 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8032810 con 0x7fcee80bb770 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 -- 172.21.15.46:0/275886653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee80bb770 msgr2=0x7fcee80d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 --2- 172.21.15.46:0/275886653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee80bb770 0x7fcee80d5040 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fcee80d6d00 tx=0x7fced802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 -- 172.21.15.46:0/275886653 shutdown_connections 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 --2- 172.21.15.46:0/275886653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80d2a70 0x7fcee80d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 --2- 172.21.15.46:0/275886653 >> v2:172.21.15.88:3300/0 conn(0x7fcee8163eb0 0x7fcee81642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 --2- 172.21.15.46:0/275886653 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee80bb770 0x7fcee80d5040 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 -- 172.21.15.46:0/275886653 >> 172.21.15.46:0/275886653 conn(0x7fcee806e370 msgr2=0x7fcee806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:09.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 -- 172.21.15.46:0/275886653 shutdown_connections 2024-05-09T03:17:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.489+0000 7fceed08b640 1 -- 172.21.15.46:0/275886653 wait complete. 2024-05-09T03:17:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.491+0000 7fceed08b640 1 Processor -- start 2024-05-09T03:17:09.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.492+0000 7fceed08b640 1 -- start start 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fceed08b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80bb770 0x7fcee81eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fceed08b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcee80d2a70 0x7fcee81f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fceed08b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 0x7fcee80cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fceed08b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcee80bbb80 con 0x7fcee8163eb0 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fceed08b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcee80d5580 con 0x7fcee80d2a70 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fceed08b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcee80d0bc0 con 0x7fcee80bb770 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 0x7fcee80cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 0x7fcee80cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40514/0 (socket says 172.21.15.46:40514) 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 -- 172.21.15.46:0/3283344517 learned_addr learned my addr 172.21.15.46:0/3283344517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80bb770 0x7fcee81eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee659c640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcee80d2a70 0x7fcee81f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 -- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80bb770 msgr2=0x7fcee81eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80bb770 0x7fcee81eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 -- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcee80d2a70 msgr2=0x7fcee81f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcee80d2a70 0x7fcee81f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee759e640 1 -- 172.21.15.46:0/3283344517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee81ec080 con 0x7fcee8163eb0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.493+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80bb770 0x7fcee81eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.494+0000 7fcee759e640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 0x7fcee80cbc20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fcee0007d40 tx=0x7fcee000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.494+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0014420 con 0x7fcee8163eb0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.494+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee0019070 con 0x7fcee8163eb0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.494+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0014800 con 0x7fcee8163eb0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.494+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee81ec360 con 0x7fcee8163eb0 2024-05-09T03:17:09.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.494+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee81ebb20 con 0x7fcee8163eb0 2024-05-09T03:17:09.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.495+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcee806a270 con 0x7fcee8163eb0 2024-05-09T03:17:09.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.496+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcee00149a0 con 0x7fcee8163eb0 2024-05-09T03:17:09.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.496+0000 7fcedf7fe640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcec8076300 0x7fcec8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.496+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6369+0+0 (secure 0 0 0) 0x7fcee0097fa0 con 0x7fcee8163eb0 2024-05-09T03:17:09.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.497+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcec8076300 0x7fcec8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.497+0000 7fcee6d9d640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcec8076300 0x7fcec8078810 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fced8034000 tx=0x7fced802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.501+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcee0061010 con 0x7fcee8163eb0 2024-05-09T03:17:09.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.733+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcee80d4130 con 0x7fcee8163eb0 2024-05-09T03:17:09.735 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:09.734+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:09.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.735+0000 7fcedf7fe640 1 -- 172.21.15.46:0/3283344517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v134) ==== 161+0+0 (secure 0 0 0) 0x7fcee0065310 con 0x7fcee8163eb0 2024-05-09T03:17:09.737 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcec8076300 msgr2=0x7fcec8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcec8076300 0x7fcec8078810 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fced8034000 tx=0x7fced802e000 comp rx=0 tx=0).stop 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 msgr2=0x7fcee80cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 0x7fcee80cbc20 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fcee0007d40 tx=0x7fcee000e4b0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 shutdown_connections 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcec8076300 0x7fcec8078810 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee8163eb0 0x7fcee80cbc20 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcee80d2a70 0x7fcee81f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 --2- 172.21.15.46:0/3283344517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcee80bb770 0x7fcee81eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 >> 172.21.15.46:0/3283344517 conn(0x7fcee806e370 msgr2=0x7fcee80cf700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:09.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 shutdown_connections 2024-05-09T03:17:09.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.738+0000 7fceed08b640 1 -- 172.21.15.46:0/3283344517 wait complete. 2024-05-09T03:17:09.741 INFO:tasks.ceph.mds.c.smithi046.stderr: -692> 2024-05-09T03:17:04.055+0000 7f43a7321640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:17:09.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:09.797 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.868+0000 7fa2e5496640 1 Processor -- start 2024-05-09T03:17:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.868+0000 7fa2e5496640 1 -- start start 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2e5496640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2e00d2e90 0x7fa2e00b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2e5496640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e00d3400 con 0x7fa2e00b4fc0 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2e5496640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e00d4a60 con 0x7fa2e01605b0 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2e5496640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e00d4d40 con 0x7fa2e00d2e90 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2deffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2e00d2e90 0x7fa2e00b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2df7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa2e00b4fc0 0x7fa2e00d3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48526/0 (socket says 172.21.15.46:48526) 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2df7fe640 1 -- 172.21.15.46:0/2311351802 learned_addr learned my addr 172.21.15.46:0/2311351802 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2deffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2e00d2e90 0x7fa2e00b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40554/0 (socket says 172.21.15.46:40554) 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2deffd640 1 -- 172.21.15.46:0/2311351802 >> v1:172.21.15.88:6789/0 conn(0x7fa2e01605b0 legacy=0x7fa2e00d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2deffd640 1 -- 172.21.15.46:0/2311351802 >> v1:172.21.15.46:6789/0 conn(0x7fa2e00b4fc0 legacy=0x7fa2e00d3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2ddffb640 1 -- 172.21.15.46:0/2311351802 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170437087 0 0) 0x7fa2e00d3400 con 0x7fa2e00b4fc0 2024-05-09T03:17:09.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.870+0000 7fa2deffd640 1 -- 172.21.15.46:0/2311351802 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e00d5020 con 0x7fa2e00d2e90 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.871+0000 7fa2deffd640 1 --2- 172.21.15.46:0/2311351802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e00d2e90 0x7fa2e00b7760 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00097a0 tx=0x7fa2d002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4534c950047cc5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.871+0000 7fa2ddffb640 1 -- 172.21.15.46:0/2311351802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0037070 con 0x7fa2e00d2e90 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.871+0000 7fa2ddffb640 1 -- 172.21.15.46:0/2311351802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d0032440 con 0x7fa2e00d2e90 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.871+0000 7fa2ddffb640 1 -- 172.21.15.46:0/2311351802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0032810 con 0x7fa2e00d2e90 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 -- 172.21.15.46:0/2311351802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e00d2e90 msgr2=0x7fa2e00b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 --2- 172.21.15.46:0/2311351802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e00d2e90 0x7fa2e00b7760 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00097a0 tx=0x7fa2d002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 -- 172.21.15.46:0/2311351802 shutdown_connections 2024-05-09T03:17:09.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 --2- 172.21.15.46:0/2311351802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e00d2e90 0x7fa2e00b7760 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 -- 172.21.15.46:0/2311351802 >> 172.21.15.46:0/2311351802 conn(0x7fa2e008d390 msgr2=0x7fa2e008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 -- 172.21.15.46:0/2311351802 shutdown_connections 2024-05-09T03:17:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.872+0000 7fa2e5496640 1 -- 172.21.15.46:0/2311351802 wait complete. 2024-05-09T03:17:09.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.874+0000 7fa2e5496640 1 Processor -- start 2024-05-09T03:17:09.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.874+0000 7fa2e5496640 1 -- start start 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2e5496640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2e00b4fc0 0x7fa2e00c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2e5496640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2e00d2e90 0x7fa2e0153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2e5496640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 0x7fa2e01f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2e5496640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e00d5540 con 0x7fa2e01605b0 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2e5496640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e00d46d0 con 0x7fa2e00b4fc0 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2e5496640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e01625c0 con 0x7fa2e00d2e90 2024-05-09T03:17:09.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2df7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 0x7fa2e01f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2df7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 0x7fa2e01f6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40568/0 (socket says 172.21.15.46:40568) 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.875+0000 7fa2df7fe640 1 -- 172.21.15.46:0/1094805383 learned_addr learned my addr 172.21.15.46:0/1094805383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2deffd640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2e00b4fc0 0x7fa2e00c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2de7fc640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2e00d2e90 0x7fa2e0153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2df7fe640 1 -- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2e00d2e90 msgr2=0x7fa2e0153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2df7fe640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2e00d2e90 0x7fa2e0153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2df7fe640 1 -- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2e00b4fc0 msgr2=0x7fa2e00c9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2df7fe640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2e00b4fc0 0x7fa2e00c9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2df7fe640 1 -- 172.21.15.46:0/1094805383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e0162af0 con 0x7fa2e01605b0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2de7fc640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2e00d2e90 0x7fa2e0153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2df7fe640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 0x7fa2e01f6ed0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800caa0 tx=0x7fa2d800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800aba0 con 0x7fa2e01605b0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d800ad40 con 0x7fa2e01605b0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e0162810 con 0x7fa2e01605b0 2024-05-09T03:17:09.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8012640 con 0x7fa2e01605b0 2024-05-09T03:17:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.876+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e015ade0 con 0x7fa2e01605b0 2024-05-09T03:17:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.877+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e006a270 con 0x7fa2e01605b0 2024-05-09T03:17:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.878+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa2d80127e0 con 0x7fa2e01605b0 2024-05-09T03:17:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.879+0000 7fa2c77fe640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2cc076300 0x7fa2cc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.879+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6455+0+0 (secure 0 0 0) 0x7fa2d8097d90 con 0x7fa2e01605b0 2024-05-09T03:17:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.879+0000 7fa2deffd640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2cc076300 0x7fa2cc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:09.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.879+0000 7fa2deffd640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2cc076300 0x7fa2cc078810 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0034000 tx=0x7fa2d002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:09.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:09.883+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa2e0162af0 con 0x7fa2e01605b0 2024-05-09T03:17:10.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.160+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa2e00b6a20 con 0x7fa2e01605b0 2024-05-09T03:17:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.737+0000 7fa2c77fe640 1 -- 172.21.15.46:0/1094805383 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v135) ==== 107+0+0 (secure 0 0 0) 0x7fa2d8061490 con 0x7fa2e01605b0 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2cc076300 msgr2=0x7fa2cc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2cc076300 0x7fa2cc078810 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0034000 tx=0x7fa2d002e000 comp rx=0 tx=0).stop 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 msgr2=0x7fa2e01f6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 0x7fa2e01f6ed0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa2d800caa0 tx=0x7fa2d800cf60 comp rx=0 tx=0).stop 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 shutdown_connections 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2cc076300 0x7fa2cc078810 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e01605b0 0x7fa2e01f6ed0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2e00d2e90 0x7fa2e0153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 --2- 172.21.15.46:0/1094805383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2e00b4fc0 0x7fa2e00c9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.740+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 >> 172.21.15.46:0/1094805383 conn(0x7fa2e008d390 msgr2=0x7fa2e015ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.741+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 shutdown_connections 2024-05-09T03:17:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.741+0000 7fa2e5496640 1 -- 172.21.15.46:0/1094805383 wait complete. 2024-05-09T03:17:10.757 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.815+0000 7f87336cff40 1 Processor -- start 2024-05-09T03:17:10.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.815+0000 7f87336cff40 1 -- start start 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.816+0000 7f87336cff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 0x55830994ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.816+0000 7f87336cff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583098af300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.816+0000 7f87336cff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55830995a3a0 0x5583098af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.816+0000 7f87336cff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5583098c64d0 con 0x5583098b17b0 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.816+0000 7f87336cff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558309828f50 con 0x55830995a3a0 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.816+0000 7f87336cff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5583098cb8a0 con 0x5583098c8010 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 0x55830994ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 0x55830994ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40576/0 (socket says 172.21.15.46:40576) 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 -- 172.21.15.46:0/3165268368 learned_addr learned my addr 172.21.15.46:0/3165268368 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d998640 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55830995a3a0 0x5583098af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872c996640 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583098af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 -- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 msgr2=0x5583098af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583098af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 -- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55830995a3a0 msgr2=0x5583098af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55830995a3a0 0x5583098af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 -- 172.21.15.46:0/3165268368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55830995a920 con 0x5583098b17b0 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f872d197640 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 0x55830994ae00 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8720025ad0 tx=0x7f8720050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b249652b1805ae6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f871f7fe640 1 -- 172.21.15.46:0/3165268368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720054040 con 0x5583098b17b0 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f871f7fe640 1 -- 172.21.15.46:0/3165268368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87200509b0 con 0x5583098b17b0 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.817+0000 7f871f7fe640 1 -- 172.21.15.46:0/3165268368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720050da0 con 0x5583098b17b0 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 -- 172.21.15.46:0/3165268368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 msgr2=0x55830994ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 0x55830994ae00 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8720025ad0 tx=0x7f8720050000 comp rx=0 tx=0).stop 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 -- 172.21.15.46:0/3165268368 shutdown_connections 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55830995a3a0 0x5583098af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583098af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 --2- 172.21.15.46:0/3165268368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583098b17b0 0x55830994ae00 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 -- 172.21.15.46:0/3165268368 >> 172.21.15.46:0/3165268368 conn(0x5583097a4ac0 msgr2=0x5583098c0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 -- 172.21.15.46:0/3165268368 shutdown_connections 2024-05-09T03:17:10.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.818+0000 7f87336cff40 1 -- 172.21.15.46:0/3165268368 wait complete. 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.819+0000 7f87336cff40 1 Processor -- start 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.819+0000 7f87336cff40 1 -- start start 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f87336cff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583098b17b0 0x5583098c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f87336cff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583099485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f87336cff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55830995a3a0 0x558309948b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f87336cff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55830995abc0 con 0x55830995a3a0 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f87336cff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558309955420 con 0x5583098b17b0 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f87336cff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5583099494b0 con 0x5583098c8010 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.820+0000 7f872d998640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55830995a3a0 0x558309948b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55830995a3a0 0x558309948b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40588/0 (socket says 172.21.15.46:40588) 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 learned_addr learned my addr 172.21.15.46:0/29991152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:10.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583099485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583098b17b0 0x5583098c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 msgr2=0x5583099485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583099485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583098b17b0 msgr2=0x5583098c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583098b17b0 0x5583098c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583098af9b0 con 0x55830995a3a0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583098c8010 0x5583099485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583098b17b0 0x5583098c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f872d998640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55830995a3a0 0x558309948b10 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f8728009f50 tx=0x7f87280084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f871e7fc640 1 -- 172.21.15.46:0/29991152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728015460 con 0x55830995a3a0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f871e7fc640 1 -- 172.21.15.46:0/29991152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f872801a070 con 0x55830995a3a0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5583098841c0 con 0x55830995a3a0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f871e7fc640 1 -- 172.21.15.46:0/29991152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728015820 con 0x55830995a3a0 2024-05-09T03:17:10.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.821+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583098836e0 con 0x55830995a3a0 2024-05-09T03:17:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.823+0000 7f871e7fc640 1 -- 172.21.15.46:0/29991152 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8728013030 con 0x55830995a3a0 2024-05-09T03:17:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.823+0000 7f871e7fc640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8710074ce0 0x7f87100771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f871e7fc640 1 -- 172.21.15.46:0/29991152 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6455+0+0 (secure 0 0 0) 0x7f8728096550 con 0x55830995a3a0 2024-05-09T03:17:10.826 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558309954820 0x558309954c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583098c53c0 con 0x558309954820 2024-05-09T03:17:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8710074ce0 0x7f87100771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558309954820 0x558309954c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558309954820 0x558309954c30 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.824+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8710074ce0 0x7f87100771f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8720064000 tx=0x7f8720051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.826+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v92'226 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.826+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5583099fb500 con 0x558309954820 2024-05-09T03:17:10.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.828+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v92'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.828+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fb8a0 con 0x558309954820 2024-05-09T03:17:10.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.829+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v92'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.831 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:10.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.829+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fcc50 con 0x558309954820 2024-05-09T03:17:10.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.831+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v92'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.831+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fcff0 con 0x558309954820 2024-05-09T03:17:10.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.832+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v92'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.832+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fbc40 con 0x558309954820 2024-05-09T03:17:10.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.833+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v92'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.834+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc030 con 0x558309954820 2024-05-09T03:17:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.835+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v92'232 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.835+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5583099fc420 con 0x558309954820 2024-05-09T03:17:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.836+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v92'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.836+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc420 con 0x558309954820 2024-05-09T03:17:10.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.838+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v92'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.840 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:10.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.838+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.839+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v92'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.839+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.841+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v92'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.841+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.842+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v92'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.842+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.844+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v92'238 uv213 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.844+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.845+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v92'239 uv239 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.845+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.846+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v92'240 uv240 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.848 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:10.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.846+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.848+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v92'241 uv241 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.848+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.849+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v92'242 uv242 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.849+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.851+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v92'243 uv243 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.851+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.852+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v92'244 uv244 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.852+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.853+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v92'245 uv219 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.853+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.854+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v92'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.854+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.856+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v92'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.857 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:10.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.856+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.857+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v92'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.857+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.859+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v92'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.859+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.860+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v92'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.860+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.862+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v92'251 uv225 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.862+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.863+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v92'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.863+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.864+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v92'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.866 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:10.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.864+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.865+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v92'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.866+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.867+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v92'255 uv255 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.867+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x558309954820 2024-05-09T03:17:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.868+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v92'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.868+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5583099fc790 0x5583099ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.868+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x5583099fc3d0 con 0x5583099fc790 2024-05-09T03:17:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.868+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5583099fc790 0x5583099ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.869+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5583099fc790 0x5583099ff8b0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.870+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v92'175 uv155 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.870+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5583099ffdf0 con 0x5583099fc790 2024-05-09T03:17:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.872+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v92'176 uv176 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.872+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00190 con 0x5583099fc790 2024-05-09T03:17:10.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.873+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v92'177 uv177 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.874 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:10.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.873+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a009e0 con 0x5583099fc790 2024-05-09T03:17:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.874+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v92'178 uv178 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.874+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00d80 con 0x5583099fc790 2024-05-09T03:17:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.875+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v92'179 uv179 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.875+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a01120 con 0x5583099fc790 2024-05-09T03:17:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.876+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v92'180 uv174 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.876+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a014c0 con 0x5583099fc790 2024-05-09T03:17:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.878+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v92'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.878+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a01860 con 0x5583099fc790 2024-05-09T03:17:10.880 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.879+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v92'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.879+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.880+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v92'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.880+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.884+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v92'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.884+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.885+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v92'185 uv173 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.885+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.887+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v92'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.887+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.889+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v92'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.890 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.889+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.891+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v92'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.891+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.892+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v92'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.892+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.894+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v92'190 uv171 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.894+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.895+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v92'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.895+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.896+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v92'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.898 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:10.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.896+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.897+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v92'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.897+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.898+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v92'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.898+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x5583099fc790 2024-05-09T03:17:10.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.899+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v92'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8720056030 con 0x5583099fc790 2024-05-09T03:17:10.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.899+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x558309954820 2024-05-09T03:17:10.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.901+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v92'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.901+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a00610 con 0x558309954820 2024-05-09T03:17:10.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.902+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v92'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.902+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x558309954820 2024-05-09T03:17:10.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.903+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v92'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.904 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:10.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.903+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x558309954820 2024-05-09T03:17:10.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.904+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v92'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.904+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x558309954820 2024-05-09T03:17:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.905+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v92'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.905+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a00610 con 0x558309954820 2024-05-09T03:17:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.906+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v92'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.906+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558309a02220 0x558309a02630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.906+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a04d70 con 0x558309a02220 2024-05-09T03:17:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.906+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558309a02220 0x558309a02630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.906+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558309a02220 0x558309a02630 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.908+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v92'40 uv39 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.908+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a04d70 con 0x558309a02220 2024-05-09T03:17:10.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.909+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v92'41 uv41 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.909+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a04d90 con 0x558309a02220 2024-05-09T03:17:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.910+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v92'42 uv42 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.911 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.910+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a051d0 con 0x558309a02220 2024-05-09T03:17:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.911+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v92'43 uv43 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.911+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a051d0 con 0x558309954820 2024-05-09T03:17:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.912+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v92'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.912+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a051d0 con 0x558309954820 2024-05-09T03:17:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.914+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v92'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.914+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a051d0 con 0x558309954820 2024-05-09T03:17:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.915+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v92'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.916 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.915+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a061e0 con 0x558309954820 2024-05-09T03:17:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.916+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v92'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.916+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a061e0 con 0x558309954820 2024-05-09T03:17:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.917+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v92'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.917+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a051d0 con 0x558309954820 2024-05-09T03:17:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.918+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v92'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.918+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558309a056e0 0x558309a07b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.918+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a08110 con 0x558309a056e0 2024-05-09T03:17:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.918+0000 7f872d998640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558309a056e0 0x558309a07b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.919+0000 7f872d998640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558309a056e0 0x558309a07b00 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.920+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v92'83 uv75 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.920+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a085e0 con 0x558309a056e0 2024-05-09T03:17:10.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.921+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v92'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.921+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a08ac0 con 0x558309a056e0 2024-05-09T03:17:10.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.923+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v92'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.924 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:10.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.923+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0a370 con 0x558309a056e0 2024-05-09T03:17:10.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.924+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v92'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.924+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0a710 con 0x558309a056e0 2024-05-09T03:17:10.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.925+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v92'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.925+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a09360 con 0x558309a056e0 2024-05-09T03:17:10.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.926+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v92'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.926+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a09750 con 0x558309a056e0 2024-05-09T03:17:10.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.927+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v92'89 uv82 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.927+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a09b40 con 0x558309a056e0 2024-05-09T03:17:10.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.928+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v92'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.928+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a09b40 con 0x558309a056e0 2024-05-09T03:17:10.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.929+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v92'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.931 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:10.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.929+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a09af0 con 0x558309a056e0 2024-05-09T03:17:10.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.930+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v92'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.931+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a09af0 con 0x558309a056e0 2024-05-09T03:17:10.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.932+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v92'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.932+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558309a09eb0 0x558309a0cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.932+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d510 con 0x558309a09eb0 2024-05-09T03:17:10.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.932+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558309a09eb0 0x558309a0cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.932+0000 7f872d197640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558309a09eb0 0x558309a0cfd0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.934+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v92'37 uv36 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8720056030 con 0x558309a09eb0 2024-05-09T03:17:10.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.934+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a0d510 con 0x558309a09eb0 2024-05-09T03:17:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.935+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v92'38 uv38 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8720056030 con 0x558309a09eb0 2024-05-09T03:17:10.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.935+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d550 con 0x558309a09eb0 2024-05-09T03:17:10.938 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:10.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.936+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v92'39 uv39 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8720056030 con 0x558309a09eb0 2024-05-09T03:17:10.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.936+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a09eb0 2024-05-09T03:17:10.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.938+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v92'40 uv40 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8720056030 con 0x558309a09eb0 2024-05-09T03:17:10.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.938+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a09eb0 2024-05-09T03:17:10.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.939+0000 7f872d197640 1 -- 172.21.15.46:0/29991152 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v92'41 uv41 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8720056030 con 0x558309a09eb0 2024-05-09T03:17:10.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.939+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309954820 2024-05-09T03:17:10.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.940+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v92'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.940+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309954820 2024-05-09T03:17:10.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.941+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v92'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.941+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309954820 2024-05-09T03:17:10.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.942+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v92'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.944 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:10.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.943+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0e980 con 0x558309954820 2024-05-09T03:17:10.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.944+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v92'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.944+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0e980 con 0x558309954820 2024-05-09T03:17:10.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.945+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v92'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.945+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309954820 2024-05-09T03:17:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.946+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v92'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309954820 2024-05-09T03:17:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.946+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.947+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v92'74 uv68 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.947+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.948+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v92'75 uv75 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.948+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.949+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v92'76 uv76 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.951 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.949+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.950+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v92'77 uv77 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.950+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.951+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v92'78 uv78 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.951+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.952+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v92'79 uv73 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.952+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.953+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v92'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.953+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.954+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v92'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.956 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.954+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a0d970 con 0x558309a02220 2024-05-09T03:17:10.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.955+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v92'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.955+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a10780 con 0x558309a02220 2024-05-09T03:17:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.956+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v92'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.956+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558309a0e4a0 0x558309a10d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.956+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a0e4a0 2024-05-09T03:17:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.956+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558309a0e4a0 0x558309a10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:10.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.957+0000 7f872c996640 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558309a0e4a0 0x558309a10d40 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.958+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v92'36 uv34 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a0e4a0 2024-05-09T03:17:10.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.958+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a0e4a0 2024-05-09T03:17:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.959+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v92'37 uv37 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a0e4a0 2024-05-09T03:17:10.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.959+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a0e4a0 2024-05-09T03:17:10.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.960+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v92'38 uv38 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a0e4a0 2024-05-09T03:17:10.962 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:10.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.960+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a0e4a0 2024-05-09T03:17:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.961+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v92'39 uv39 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a0e4a0 2024-05-09T03:17:10.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.961+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a02220 2024-05-09T03:17:10.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.962+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v92'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.962+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a02220 2024-05-09T03:17:10.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.964+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v92'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.964+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a02220 2024-05-09T03:17:10.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.965+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v92'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.966 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:10.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.965+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a02220 2024-05-09T03:17:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.966+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v92'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.966+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a02220 2024-05-09T03:17:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.967+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v92'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.968+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a02220 2024-05-09T03:17:10.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.969+0000 7f872c996640 1 -- 172.21.15.46:0/29991152 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v92'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872400e040 con 0x558309a02220 2024-05-09T03:17:10.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.969+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a056e0 2024-05-09T03:17:10.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.970+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v92'37 uv36 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.970+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a056e0 2024-05-09T03:17:10.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.971+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v92'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.971+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a108c0 con 0x558309a056e0 2024-05-09T03:17:10.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.973+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v92'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.974 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:10.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.973+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.974+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.974+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.975+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.975+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.977+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v92'95 uv86 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.977+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.978+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v92'96 uv96 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.978+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.980 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.979+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v92'97 uv97 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.979+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.980+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v92'98 uv98 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.980+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.981+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v92'99 uv99 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.981+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.982+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v92'100 uv93 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.982+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.983+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v92'101 uv101 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.983+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.984+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v92'102 uv102 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.986 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:10.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.985+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a12290 con 0x558309a056e0 2024-05-09T03:17:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.986+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v92'103 uv103 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.986+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a12290 con 0x558309a056e0 2024-05-09T03:17:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.987+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v92'104 uv104 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.987+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.988+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v92'105 uv105 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.988+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x558309a11280 con 0x558309a056e0 2024-05-09T03:17:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v92'106 uv106 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f872805b7e0 con 0x558309a056e0 2024-05-09T03:17:10.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558309954820 msgr2=0x558309954c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558309954820 0x558309954c30 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558309a02220 msgr2=0x558309a02630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558309a02220 0x558309a02630 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558309a0e4a0 msgr2=0x558309a10d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558309a0e4a0 0x558309a10d40 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5583099fc790 msgr2=0x5583099ff8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 reap_dead start 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5583099fc790 0x5583099ff8b0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558309a056e0 msgr2=0x558309a07b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558309a056e0 0x558309a07b00 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558309a09eb0 msgr2=0x558309a0cfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558309a09eb0 0x558309a0cfd0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8710074ce0 msgr2=0x7f87100771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8710074ce0 0x7f87100771f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8720064000 tx=0x7f8720051000 comp rx=0 tx=0).stop 2024-05-09T03:17:10.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55830995a3a0 msgr2=0x558309948b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 --2- 172.21.15.46:0/29991152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55830995a3a0 0x558309948b10 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f8728009f50 tx=0x7f87280084c0 comp rx=0 tx=0).stop 2024-05-09T03:17:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f872d998640 1 -- 172.21.15.46:0/29991152 reap_dead start 2024-05-09T03:17:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 shutdown_connections 2024-05-09T03:17:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 >> 172.21.15.46:0/29991152 conn(0x5583097a4ac0 msgr2=0x5583098b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 shutdown_connections 2024-05-09T03:17:10.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:10.989+0000 7f87336cff40 1 -- 172.21.15.46:0/29991152 wait complete. 2024-05-09T03:17:10.994 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds_snaptable 2024-05-09T03:17:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.089+0000 7f4c1ca0ef40 1 Processor -- start 2024-05-09T03:17:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.089+0000 7f4c1ca0ef40 1 -- start start 2024-05-09T03:17:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1ca0ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 0x5587053d4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1ca0ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5587053bcdb0 0x5587054f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1ca0ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705448c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1ca0ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587053cfcb0 con 0x55870538d7e0 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1ca0ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558705336f50 con 0x558705465780 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1ca0ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55870538dbf0 con 0x5587053bcdb0 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 0x5587053d4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 0x5587053d4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40600/0 (socket says 172.21.15.46:40600) 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 -- 172.21.15.46:0/4090851887 learned_addr learned my addr 172.21.15.46:0/4090851887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c16010640 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705448c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5587053bcdb0 0x5587054f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 -- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5587053bcdb0 msgr2=0x5587054f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5587053bcdb0 0x5587054f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 -- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 msgr2=0x558705448c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705448c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.090+0000 7f4c1580f640 1 -- 172.21.15.46:0/4090851887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558705467980 con 0x55870538d7e0 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1580f640 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 0x5587053d4fa0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f4c08045ed0 tx=0x7f4c08051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9794e1bb00cba21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1480d640 1 -- 172.21.15.46:0/4090851887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08062070 con 0x55870538d7e0 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1480d640 1 -- 172.21.15.46:0/4090851887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c08054970 con 0x55870538d7e0 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1480d640 1 -- 172.21.15.46:0/4090851887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08054d20 con 0x55870538d7e0 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/4090851887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 msgr2=0x5587053d4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 0x5587053d4fa0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f4c08045ed0 tx=0x7f4c08051000 comp rx=0 tx=0).stop 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/4090851887 shutdown_connections 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705448c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5587053bcdb0 0x5587054f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/4090851887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55870538d7e0 0x5587053d4fa0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.091+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/4090851887 >> 172.21.15.46:0/4090851887 conn(0x5587052b2ac0 msgr2=0x558705449800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.092+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/4090851887 shutdown_connections 2024-05-09T03:17:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.092+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/4090851887 wait complete. 2024-05-09T03:17:11.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.092+0000 7f4c1ca0ef40 1 Processor -- start 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.093+0000 7f4c1ca0ef40 1 -- start start 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1ca0ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55870538d7e0 0x558705463940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1ca0ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587053bcdb0 0x558705449790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1ca0ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705449cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1ca0ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558705467e80 con 0x5587053bcdb0 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1ca0ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558705465de0 con 0x558705465780 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1ca0ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55870544db40 con 0x55870538d7e0 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587053bcdb0 0x558705449790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587053bcdb0 0x558705449790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40614/0 (socket says 172.21.15.46:40614) 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 -- 172.21.15.46:0/1653516867 learned_addr learned my addr 172.21.15.46:0/1653516867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c1580f640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55870538d7e0 0x558705463940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c16010640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705449cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 -- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55870538d7e0 msgr2=0x558705463940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55870538d7e0 0x558705463940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 -- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 msgr2=0x558705449cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558705465780 0x558705449cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 -- 172.21.15.46:0/1653516867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587053d1900 con 0x5587053bcdb0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.094+0000 7f4c0ffff640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587053bcdb0 0x558705449790 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4c000087e0 tx=0x7f4c000088d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.095+0000 7f4c0effd640 1 -- 172.21.15.46:0/1653516867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00019020 con 0x5587053bcdb0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.095+0000 7f4c0effd640 1 -- 172.21.15.46:0/1653516867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c00015880 con 0x5587053bcdb0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.095+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5587053d3610 con 0x5587053bcdb0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.095+0000 7f4c0effd640 1 -- 172.21.15.46:0/1653516867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c00015c50 con 0x5587053bcdb0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.095+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587053d88a0 con 0x5587053bcdb0 2024-05-09T03:17:11.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.096+0000 7f4c0effd640 1 -- 172.21.15.46:0/1653516867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4c0001c020 con 0x5587053bcdb0 2024-05-09T03:17:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c0effd640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bf8074ce0 0x7f4bf80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c0effd640 1 -- 172.21.15.46:0/1653516867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6455+0+0 (secure 0 0 0) 0x7f4c00096ec0 con 0x5587053bcdb0 2024-05-09T03:17:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c1580f640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bf8074ce0 0x7f4bf80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5587053d4a30 0x5587053d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e92) -- 0x5587053d84a0 con 0x5587053d4a30 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c16010640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5587053d4a30 0x5587053d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.097+0000 7f4c1580f640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bf8074ce0 0x7f4bf80771f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4c08064000 tx=0x7f4c08053000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.098+0000 7f4c16010640 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5587053d4a30 0x5587053d7f60 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c16010640 1 -- 172.21.15.46:0/1653516867 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 mds_snaptable [delete] v92'42 uv42 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4c10003450 con 0x5587053d4a30 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5587053d4a30 msgr2=0x5587053d7f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5587053d4a30 0x5587053d7f60 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bf8074ce0 msgr2=0x7f4bf80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4bf8074ce0 0x7f4bf80771f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4c08064000 tx=0x7f4c08053000 comp rx=0 tx=0).stop 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587053bcdb0 msgr2=0x558705449790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 --2- 172.21.15.46:0/1653516867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587053bcdb0 0x558705449790 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f4c000087e0 tx=0x7f4c000088d0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c16010640 1 -- 172.21.15.46:0/1653516867 reap_dead start 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 shutdown_connections 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 >> 172.21.15.46:0/1653516867 conn(0x5587052b2ac0 msgr2=0x5587053be800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:11.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 shutdown_connections 2024-05-09T03:17:11.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.099+0000 7f4c1ca0ef40 1 -- 172.21.15.46:0/1653516867 wait complete. 2024-05-09T03:17:11.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.219+0000 7f9356fb2640 1 Processor -- start 2024-05-09T03:17:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.219+0000 7f9356fb2640 1 -- start start 2024-05-09T03:17:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9356fb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500b5140 0x7f93500b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9356fb2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93500b80b0 con 0x7f93500b7ca0 2024-05-09T03:17:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9356fb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93500b5550 con 0x7f93500d3fc0 2024-05-09T03:17:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9356fb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93500d3480 con 0x7f93500b5140 2024-05-09T03:17:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9347fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f93500b7ca0 0x7f93500d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48534/0 (socket says 172.21.15.46:48534) 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9347fff640 1 -- 172.21.15.46:0/4252400814 learned_addr learned my addr 172.21.15.46:0/4252400814 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9354d4e640 1 --2- 172.21.15.46:0/4252400814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500b5140 0x7f93500b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192931037 0 0) 0x7f9340001110 con 0x7f93500b7ca0 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9338003520 con 0x7f93500b7ca0 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3646053443 0 0) 0x7f93500b5550 con 0x7f93500d3fc0 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9340001110 con 0x7f93500d3fc0 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9354d4e640 1 -- 172.21.15.46:0/4252400814 >> v1:172.21.15.88:6789/0 conn(0x7f93500d3fc0 legacy=0x7f93500b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9354d4e640 1 -- 172.21.15.46:0/4252400814 >> v1:172.21.15.46:6789/0 conn(0x7f93500b7ca0 legacy=0x7f93500d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.221+0000 7f9354d4e640 1 -- 172.21.15.46:0/4252400814 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93500d44e0 con 0x7f93500b5140 2024-05-09T03:17:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.222+0000 7f9354d4e640 1 --2- 172.21.15.46:0/4252400814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500b5140 0x7f93500b7760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f934c0097d0 tx=0x7f934c02a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11069eb34f221ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.222+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c037070 con 0x7f93500b5140 2024-05-09T03:17:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.222+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934c032440 con 0x7f93500b5140 2024-05-09T03:17:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.222+0000 7f93477fe640 1 -- 172.21.15.46:0/4252400814 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c0327f0 con 0x7f93500b5140 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 -- 172.21.15.46:0/4252400814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500b5140 msgr2=0x7f93500b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4252400814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500b5140 0x7f93500b7760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f934c0097d0 tx=0x7f934c02a290 comp rx=0 tx=0).stop 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 -- 172.21.15.46:0/4252400814 shutdown_connections 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4252400814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500b5140 0x7f93500b7760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 -- 172.21.15.46:0/4252400814 >> 172.21.15.46:0/4252400814 conn(0x7f935008d3f0 msgr2=0x7f935008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 -- 172.21.15.46:0/4252400814 shutdown_connections 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.223+0000 7f9356fb2640 1 -- 172.21.15.46:0/4252400814 wait complete. 2024-05-09T03:17:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.225+0000 7f9356fb2640 1 Processor -- start 2024-05-09T03:17:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.225+0000 7f9356fb2640 1 -- start start 2024-05-09T03:17:11.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.226+0000 7f9356fb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 0x7f93501570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9356fb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93500b7ca0 0x7f93501e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9356fb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500d3fc0 0x7f93501ea150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9356fb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f93500d4d80 con 0x7f93500b5140 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9356fb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f93500d4aa0 con 0x7f93500b7ca0 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9356fb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f93500d5580 con 0x7f93500d3fc0 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 0x7f93501570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 0x7f93501570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40618/0 (socket says 172.21.15.46:40618) 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 -- 172.21.15.46:0/4027936967 learned_addr learned my addr 172.21.15.46:0/4027936967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9347fff640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93500b7ca0 0x7f93501e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f935554f640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500d3fc0 0x7f93501ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 -- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500d3fc0 msgr2=0x7f93501ea150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500d3fc0 0x7f93501ea150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 -- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93500b7ca0 msgr2=0x7f93501e9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93500b7ca0 0x7f93501e9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 -- 172.21.15.46:0/4027936967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9350158810 con 0x7f93500b5140 2024-05-09T03:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9354d4e640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 0x7f93501570e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f934c0097d0 tx=0x7f934c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c037040 con 0x7f93500b5140 2024-05-09T03:17:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f934c004230 con 0x7f93500b5140 2024-05-09T03:17:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9350158af0 con 0x7f93500b5140 2024-05-09T03:17:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.227+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934c006e10 con 0x7f93500b5140 2024-05-09T03:17:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.228+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93501590b0 con 0x7f93500b5140 2024-05-09T03:17:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.228+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f935006a270 con 0x7f93500b5140 2024-05-09T03:17:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.229+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f934c03a070 con 0x7f93500b5140 2024-05-09T03:17:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.230+0000 7f93457fa640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f933c0762d0 0x7f933c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.230+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6455+0+0 (secure 0 0 0) 0x7f934c0b7c50 con 0x7f93500b5140 2024-05-09T03:17:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.230+0000 7f9347fff640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f933c0762d0 0x7f933c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:11.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.231+0000 7f9347fff640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f933c0762d0 0x7f933c0787e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9340005880 tx=0x7f9340009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.235+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f934c0bd020 con 0x7f93500b5140 2024-05-09T03:17:11.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:11.467+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f935006d160 con 0x7f93500b5140 2024-05-09T03:17:12.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.089+0000 7f93457fa640 1 -- 172.21.15.46:0/4027936967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v136) ==== 162+0+0 (secure 0 0 0) 0x7f934c041020 con 0x7f93500b5140 2024-05-09T03:17:12.091 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f933c0762d0 msgr2=0x7f933c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f933c0762d0 0x7f933c0787e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9340005880 tx=0x7f9340009040 comp rx=0 tx=0).stop 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 msgr2=0x7f93501570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 0x7f93501570e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f934c0097d0 tx=0x7f934c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 shutdown_connections 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f93500d3fc0 0x7f93501ea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93500b7ca0 0x7f93501e9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f933c0762d0 0x7f933c0787e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 --2- 172.21.15.46:0/4027936967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93500b5140 0x7f93501570e0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 >> 172.21.15.46:0/4027936967 conn(0x7f935008d3f0 msgr2=0x7f93500b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:12.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 shutdown_connections 2024-05-09T03:17:12.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.092+0000 7f9356fb2640 1 -- 172.21.15.46:0/4027936967 wait complete. 2024-05-09T03:17:12.099 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:12.098+0000 7fbd9f869640 -1 mds.0.snaptable: load_2 could not read table: -2 2024-05-09T03:17:12.099 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:12.098+0000 7fbd9f869640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds_snaptable' -2 ((2) No such file or directory) 2024-05-09T03:17:12.100 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:12.098+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:12.100 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:12.098+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:12.106 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:17:12.098+0000 7fbd9f869640 -1 mds.0.snaptable: load_2 could not read table: -2 2024-05-09T03:17:12.106 INFO:tasks.ceph.mds.c.smithi046.stderr: -23> 2024-05-09T03:17:12.098+0000 7fbd9f869640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds_snaptable' -2 ((2) No such file or directory) 2024-05-09T03:17:12.110 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:12.164 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:12.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.227+0000 7fb815284640 1 Processor -- start 2024-05-09T03:17:12.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.227+0000 7fb815284640 1 -- start start 2024-05-09T03:17:12.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb815284640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8100b1210 0x7fb8100b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb815284640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb8100aeb80 0x7fb8100aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb815284640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb81015d980 con 0x7fb810070670 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb815284640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb81015dc60 con 0x7fb8100b1210 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb815284640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb81015df40 con 0x7fb8100aeb80 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb80effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8100b1210 0x7fb8100b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb80effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb8100b1210 0x7fb8100b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40638/0 (socket says 172.21.15.46:40638) 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb80effd640 1 -- 172.21.15.46:0/3544711945 learned_addr learned my addr 172.21.15.46:0/3544711945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.228+0000 7fb80e7fc640 1 --2- 172.21.15.46:0/3544711945 >> v2:172.21.15.88:3300/0 conn(0x7fb8100aeb80 0x7fb8100aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.229+0000 7fb80effd640 1 -- 172.21.15.46:0/3544711945 >> v2:172.21.15.88:3300/0 conn(0x7fb8100aeb80 msgr2=0x7fb8100aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.229+0000 7fb80effd640 1 --2- 172.21.15.46:0/3544711945 >> v2:172.21.15.88:3300/0 conn(0x7fb8100aeb80 0x7fb8100aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.229+0000 7fb80effd640 1 -- 172.21.15.46:0/3544711945 >> v1:172.21.15.88:6789/0 conn(0x7fb810070670 legacy=0x7fb810070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.229+0000 7fb80effd640 1 -- 172.21.15.46:0/3544711945 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81015e220 con 0x7fb8100b1210 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb80effd640 1 --2- 172.21.15.46:0/3544711945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb8100b4ff0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fb8100b4950 tx=0x7fb80002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6850537d41f2cc60 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb80dffb640 1 -- 172.21.15.46:0/3544711945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb800036070 con 0x7fb8100b1210 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb80dffb640 1 -- 172.21.15.46:0/3544711945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb800031600 con 0x7fb8100b1210 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb80dffb640 1 -- 172.21.15.46:0/3544711945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8000319d0 con 0x7fb8100b1210 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb815284640 1 -- 172.21.15.46:0/3544711945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 msgr2=0x7fb8100b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb815284640 1 --2- 172.21.15.46:0/3544711945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb8100b4ff0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fb8100b4950 tx=0x7fb80002a6a0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb815284640 1 -- 172.21.15.46:0/3544711945 shutdown_connections 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb815284640 1 --2- 172.21.15.46:0/3544711945 >> v2:172.21.15.88:3300/0 conn(0x7fb8100aeb80 0x7fb8100aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb815284640 1 --2- 172.21.15.46:0/3544711945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb8100b4ff0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.230+0000 7fb815284640 1 -- 172.21.15.46:0/3544711945 >> 172.21.15.46:0/3544711945 conn(0x7fb8100b3910 msgr2=0x7fb8100b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:12.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.231+0000 7fb815284640 1 -- 172.21.15.46:0/3544711945 shutdown_connections 2024-05-09T03:17:12.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.231+0000 7fb815284640 1 -- 172.21.15.46:0/3544711945 wait complete. 2024-05-09T03:17:12.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.232+0000 7fb815284640 1 Processor -- start 2024-05-09T03:17:12.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.232+0000 7fb815284640 1 -- start start 2024-05-09T03:17:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb815284640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb810070670 0x7fb8101446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb815284640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8100aeb80 0x7fb8100d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb815284640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb810145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb815284640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb8100739f0 con 0x7fb8100b1210 2024-05-09T03:17:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb815284640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb810073870 con 0x7fb8100aeb80 2024-05-09T03:17:12.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb815284640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb810073b70 con 0x7fb810070670 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb810145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb810145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40644/0 (socket says 172.21.15.46:40644) 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 -- 172.21.15.46:0/1990111070 learned_addr learned my addr 172.21.15.46:0/1990111070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80effd640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb810070670 0x7fb8101446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80e7fc640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8100aeb80 0x7fb8100d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 -- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb810070670 msgr2=0x7fb8101446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb810070670 0x7fb8101446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 -- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8100aeb80 msgr2=0x7fb8100d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8100aeb80 0x7fb8100d07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 -- 172.21.15.46:0/1990111070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81009e0b0 con 0x7fb8100b1210 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.234+0000 7fb80f7fe640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb810145800 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c3f0 tx=0x7fb80800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.235+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808017070 con 0x7fb8100b1210 2024-05-09T03:17:12.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.235+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8080052d0 con 0x7fb8100b1210 2024-05-09T03:17:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.235+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb810144bf0 con 0x7fb8100b1210 2024-05-09T03:17:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.235+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808003810 con 0x7fb8100b1210 2024-05-09T03:17:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.235+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8101f6cc0 con 0x7fb8100b1210 2024-05-09T03:17:12.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.236+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb81006a270 con 0x7fb8100b1210 2024-05-09T03:17:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.237+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb80801a070 con 0x7fb8100b1210 2024-05-09T03:17:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.237+0000 7fb8077fe640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7f00762d0 0x7fb7f00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.237+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7fb808096bf0 con 0x7fb8100b1210 2024-05-09T03:17:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.237+0000 7fb80effd640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7f00762d0 0x7fb7f00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.238+0000 7fb80effd640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7f00762d0 0x7fb7f00787e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb800033000 tx=0x7fb800009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.242+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb810144bf0 con 0x7fb8100b1210 2024-05-09T03:17:12.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.476+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8101452e0 con 0x7fb8100b1210 2024-05-09T03:17:12.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.476+0000 7fb8077fe640 1 -- 172.21.15.46:0/1990111070 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) ==== 95+0+4342 (secure 0 0 0) 0x7fb808068020 con 0x7fb8100b1210 2024-05-09T03:17:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:12.478 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:12.100114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:12.479 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-05-09T03:17:12.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7f00762d0 msgr2=0x7fb7f00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7f00762d0 0x7fb7f00787e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb800033000 tx=0x7fb800009000 comp rx=0 tx=0).stop 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 msgr2=0x7fb810145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb810145800 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c3f0 tx=0x7fb80800c8b0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 shutdown_connections 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb7f00762d0 0x7fb7f00787e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb8100b1210 0x7fb810145800 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8100aeb80 0x7fb8100d07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 --2- 172.21.15.46:0/1990111070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb810070670 0x7fb8101446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 >> 172.21.15.46:0/1990111070 conn(0x7fb8100b3910 msgr2=0x7fb810074190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 shutdown_connections 2024-05-09T03:17:12.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.479+0000 7fb815284640 1 -- 172.21.15.46:0/1990111070 wait complete. 2024-05-09T03:17:12.492 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:12.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:12.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.604+0000 7f1159e3b640 1 Processor -- start 2024-05-09T03:17:12.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.604+0000 7f1159e3b640 1 -- start start 2024-05-09T03:17:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f1159e3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11540b92e0 0x7f11540b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f1159e3b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11540d2b80 con 0x7f11540d0480 2024-05-09T03:17:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f1159e3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11540d2e60 con 0x7f11540cf6b0 2024-05-09T03:17:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f1159e3b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f11540d3140 con 0x7f11540b92e0 2024-05-09T03:17:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11537fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11540b92e0 0x7f11540b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11537fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11540b92e0 0x7f11540b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40646/0 (socket says 172.21.15.46:40646) 2024-05-09T03:17:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f1152ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11540d0480 0x7f11540d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48540/0 (socket says 172.21.15.46:48540) 2024-05-09T03:17:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11537fe640 1 -- 172.21.15.46:0/3954600866 learned_addr learned my addr 172.21.15.46:0/3954600866 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11537fe640 1 -- 172.21.15.46:0/3954600866 >> v1:172.21.15.88:6789/0 conn(0x7f11540cf6b0 legacy=0x7f115406d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11537fe640 1 -- 172.21.15.46:0/3954600866 >> v1:172.21.15.46:6789/0 conn(0x7f11540d0480 legacy=0x7f11540d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11537fe640 1 -- 172.21.15.46:0/3954600866 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11540d3420 con 0x7f11540b92e0 2024-05-09T03:17:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.606+0000 7f11527fc640 1 -- 172.21.15.46:0/3954600866 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882702614 0 0) 0x7f11540d2b80 con 0x7f11540d0480 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.607+0000 7f11537fe640 1 --2- 172.21.15.46:0/3954600866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540b92e0 0x7f11540b51a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1144009870 tx=0x7f114402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf64692154d5ed68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.607+0000 7f11527fc640 1 -- 172.21.15.46:0/3954600866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144037070 con 0x7f11540b92e0 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.607+0000 7f11527fc640 1 -- 172.21.15.46:0/3954600866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1144032440 con 0x7f11540b92e0 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.607+0000 7f11527fc640 1 -- 172.21.15.46:0/3954600866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144032810 con 0x7f11540b92e0 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 -- 172.21.15.46:0/3954600866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540b92e0 msgr2=0x7f11540b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 --2- 172.21.15.46:0/3954600866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540b92e0 0x7f11540b51a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1144009870 tx=0x7f114402a270 comp rx=0 tx=0).stop 2024-05-09T03:17:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 -- 172.21.15.46:0/3954600866 shutdown_connections 2024-05-09T03:17:12.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 --2- 172.21.15.46:0/3954600866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540b92e0 0x7f11540b51a0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 -- 172.21.15.46:0/3954600866 >> 172.21.15.46:0/3954600866 conn(0x7f11540d3c00 msgr2=0x7f11540d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:12.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 -- 172.21.15.46:0/3954600866 shutdown_connections 2024-05-09T03:17:12.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.608+0000 7f1159e3b640 1 -- 172.21.15.46:0/3954600866 wait complete. 2024-05-09T03:17:12.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.610+0000 7f1159e3b640 1 Processor -- start 2024-05-09T03:17:12.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.610+0000 7f1159e3b640 1 -- start start 2024-05-09T03:17:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1159e3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540b92e0 0x7f1154162a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1159e3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 0x7f1154159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1159e3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11540d0480 0x7f11540cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1159e3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f115416b3d0 con 0x7f11540cf6b0 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1159e3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f115416b250 con 0x7f11540b92e0 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1159e3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f115416b550 con 0x7f11540d0480 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1152ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 0x7f1154159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1152ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 0x7f1154159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40648/0 (socket says 172.21.15.46:40648) 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.611+0000 7f1152ffd640 1 -- 172.21.15.46:0/114643746 learned_addr learned my addr 172.21.15.46:0/114643746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f11537fe640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540b92e0 0x7f1154162a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1153fff640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11540d0480 0x7f11540cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1152ffd640 1 -- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11540d0480 msgr2=0x7f11540cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1152ffd640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11540d0480 0x7f11540cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1152ffd640 1 -- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540b92e0 msgr2=0x7f1154162a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1152ffd640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540b92e0 0x7f1154162a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1152ffd640 1 -- 172.21.15.46:0/114643746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1154163250 con 0x7f11540cf6b0 2024-05-09T03:17:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1152ffd640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 0x7f1154159410 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f114800c370 tx=0x7f114800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11480074e0 con 0x7f11540cf6b0 2024-05-09T03:17:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11480047e0 con 0x7f11540cf6b0 2024-05-09T03:17:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11541f9b70 con 0x7f11540cf6b0 2024-05-09T03:17:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148004bd0 con 0x7f11540cf6b0 2024-05-09T03:17:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.612+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11541fa130 con 0x7f11540cf6b0 2024-05-09T03:17:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.613+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f115406a270 con 0x7f11540cf6b0 2024-05-09T03:17:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.614+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1148011070 con 0x7f11540cf6b0 2024-05-09T03:17:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.614+0000 7f113bfff640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11400762d0 0x7f11400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.615+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7f1148097430 con 0x7f11540cf6b0 2024-05-09T03:17:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.615+0000 7f11537fe640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11400762d0 0x7f11400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.615+0000 7f11537fe640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11400762d0 0x7f11400787e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1144034000 tx=0x7f114402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.619+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1148060ae0 con 0x7f11540cf6b0 2024-05-09T03:17:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.848+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11540cff60 con 0x7f11540cf6b0 2024-05-09T03:17:12.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.848+0000 7f113bfff640 1 -- 172.21.15.46:0/114643746 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) ==== 95+0+4342 (secure 0 0 0) 0x7f11480603e0 con 0x7f11540cf6b0 2024-05-09T03:17:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:12.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":138,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:12.100114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:12.851 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 138 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11400762d0 msgr2=0x7f11400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11400762d0 0x7f11400787e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1144034000 tx=0x7f114402e000 comp rx=0 tx=0).stop 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 msgr2=0x7f1154159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 0x7f1154159410 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f114800c370 tx=0x7f114800c830 comp rx=0 tx=0).stop 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 shutdown_connections 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11540d0480 0x7f11540cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11400762d0 0x7f11400787e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11540cf6b0 0x7f1154159410 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 --2- 172.21.15.46:0/114643746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11540b92e0 0x7f1154162a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 >> 172.21.15.46:0/114643746 conn(0x7f11540d3c00 msgr2=0x7f11540b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 shutdown_connections 2024-05-09T03:17:12.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.851+0000 7f1159e3b640 1 -- 172.21.15.46:0/114643746 wait complete. 2024-05-09T03:17:12.864 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:12.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.978+0000 7f2266b9f640 1 Processor -- start 2024-05-09T03:17:12.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.978+0000 7f2266b9f640 1 -- start start 2024-05-09T03:17:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.979+0000 7f2266b9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f22600d1670 0x7f226009cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.979+0000 7f2266b9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22600d1b10 0x7f22600d2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.979+0000 7f2266b9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f22600b7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2266b9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f22600d1040 con 0x7f22600d1670 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2266b9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f22600cfe90 con 0x7f22600d1b10 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2266b9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f22600b8d80 con 0x7f22600d27a0 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f22600d1670 0x7f226009cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f22600d1670 0x7f226009cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40656/0 (socket says 172.21.15.46:40656) 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 -- 172.21.15.46:0/3405707409 learned_addr learned my addr 172.21.15.46:0/3405707409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f226539c640 1 --2- 172.21.15.46:0/3405707409 >> v2:172.21.15.88:3300/0 conn(0x7f22600d1b10 0x7f22600d2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f226639e640 1 --2- 172.21.15.46:0/3405707409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f22600b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 -- 172.21.15.46:0/3405707409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 msgr2=0x7f22600b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 --2- 172.21.15.46:0/3405707409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f22600b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 -- 172.21.15.46:0/3405707409 >> v2:172.21.15.88:3300/0 conn(0x7f22600d1b10 msgr2=0x7f22600d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 --2- 172.21.15.46:0/3405707409 >> v2:172.21.15.88:3300/0 conn(0x7f22600d1b10 0x7f22600d2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f2265b9d640 1 -- 172.21.15.46:0/3405707409 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22600d1290 con 0x7f22600d1670 2024-05-09T03:17:12.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.980+0000 7f226539c640 1 --2- 172.21.15.46:0/3405707409 >> v2:172.21.15.88:3300/0 conn(0x7f22600d1b10 0x7f22600d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:12.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.981+0000 7f2265b9d640 1 --2- 172.21.15.46:0/3405707409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1670 0x7f226009cb90 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f22600d01e0 tx=0x7f225802a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c10af1de66fb995 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.981+0000 7f2264b9b640 1 -- 172.21.15.46:0/3405707409 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2258036070 con 0x7f22600d1670 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.981+0000 7f2264b9b640 1 -- 172.21.15.46:0/3405707409 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2258031600 con 0x7f22600d1670 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.981+0000 7f2264b9b640 1 -- 172.21.15.46:0/3405707409 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22580319d0 con 0x7f22600d1670 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 -- 172.21.15.46:0/3405707409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1670 msgr2=0x7f226009cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 --2- 172.21.15.46:0/3405707409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1670 0x7f226009cb90 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f22600d01e0 tx=0x7f225802a600 comp rx=0 tx=0).stop 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 -- 172.21.15.46:0/3405707409 shutdown_connections 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 --2- 172.21.15.46:0/3405707409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f22600b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 --2- 172.21.15.46:0/3405707409 >> v2:172.21.15.88:3300/0 conn(0x7f22600d1b10 0x7f22600d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 --2- 172.21.15.46:0/3405707409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1670 0x7f226009cb90 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 -- 172.21.15.46:0/3405707409 >> 172.21.15.46:0/3405707409 conn(0x7f22600d0520 msgr2=0x7f22600d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 -- 172.21.15.46:0/3405707409 shutdown_connections 2024-05-09T03:17:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.982+0000 7f2266b9f640 1 -- 172.21.15.46:0/3405707409 wait complete. 2024-05-09T03:17:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.984+0000 7f2266b9f640 1 Processor -- start 2024-05-09T03:17:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.984+0000 7f2266b9f640 1 -- start start 2024-05-09T03:17:12.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.985+0000 7f2266b9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22600d1670 0x7f22600b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.985+0000 7f2266b9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 0x7f22601f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.985+0000 7f2266b9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f226006e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.985+0000 7f2266b9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f226009d0d0 con 0x7f22600d1b10 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.985+0000 7f2266b9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f22600b91e0 con 0x7f22600d1670 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.985+0000 7f2266b9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f226009d250 con 0x7f22600d27a0 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 0x7f22601f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 0x7f22601f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40672/0 (socket says 172.21.15.46:40672) 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 -- 172.21.15.46:0/1745132921 learned_addr learned my addr 172.21.15.46:0/1745132921 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:12.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f2265b9d640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22600d1670 0x7f22600b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226639e640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f226006e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 -- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 msgr2=0x7f226006e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f226006e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 -- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22600d1670 msgr2=0x7f22600b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22600d1670 0x7f22600b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 -- 172.21.15.46:0/1745132921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22600b0ba0 con 0x7f22600d1b10 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f2265b9d640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22600d1670 0x7f22600b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f226539c640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 0x7f22601f6ff0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f225400e8e0 tx=0x7f225400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254017070 con 0x7f22600d1b10 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225400c8e0 con 0x7f22600d1b10 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22600cc310 con 0x7f22600d1b10 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225400cc90 con 0x7f22600d1b10 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.986+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22600cc890 con 0x7f22600d1b10 2024-05-09T03:17:12.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.987+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2260076c70 con 0x7f22600d1b10 2024-05-09T03:17:12.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.988+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f225401a070 con 0x7f22600d1b10 2024-05-09T03:17:12.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.989+0000 7f225e7fc640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f224c076300 0x7f224c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:12.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.989+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7f22540977e0 con 0x7f22600d1b10 2024-05-09T03:17:12.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.989+0000 7f2265b9d640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f224c076300 0x7f224c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:12.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.989+0000 7f2265b9d640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f224c076300 0x7f224c078810 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2258033000 tx=0x7f2258009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:12.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:12.994+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2254060e00 con 0x7f22600d1b10 2024-05-09T03:17:13.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.228+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22601e63b0 con 0x7f22600d1b10 2024-05-09T03:17:13.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.229+0000 7f225e7fc640 1 -- 172.21.15.46:0/1745132921 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5168 (secure 0 0 0) 0x7f2254060700 con 0x7f22600d1b10 2024-05-09T03:17:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:13.231 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8073,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3968811793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3968811793},{"type":"v1","addr":"172.21.15.46:6834","nonce":3968811793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:12.100114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":93,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:13.231 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 139 2024-05-09T03:17:13.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f224c076300 msgr2=0x7f224c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:13.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f224c076300 0x7f224c078810 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2258033000 tx=0x7f2258009000 comp rx=0 tx=0).stop 2024-05-09T03:17:13.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 msgr2=0x7f22601f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:13.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 0x7f22601f6ff0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f225400e8e0 tx=0x7f225400eda0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 shutdown_connections 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22600d27a0 0x7f226006e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f224c076300 0x7f224c078810 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22600d1b10 0x7f22601f6ff0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 --2- 172.21.15.46:0/1745132921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f22600d1670 0x7f22600b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 >> 172.21.15.46:0/1745132921 conn(0x7f22600d0520 msgr2=0x7f22600723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 shutdown_connections 2024-05-09T03:17:13.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.231+0000 7f2266b9f640 1 -- 172.21.15.46:0/1745132921 wait complete. 2024-05-09T03:17:13.244 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds_snaptable' led to DAMAGED state 2024-05-09T03:17:13.244 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_inotable' 2024-05-09T03:17:13.244 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:13.244 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:13.261 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:13.261 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:13.261 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:13.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:13.261 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:13.317 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:13.317 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.471+0000 7fcefbb08640 1 Processor -- start 2024-05-09T03:17:13.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.471+0000 7fcefbb08640 1 -- start start 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.472+0000 7fcefbb08640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcef415e360 0x7fcef40aec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.472+0000 7fcefbb08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcef40744d0 0x7fcef4073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.472+0000 7fcefbb08640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4073b40 0x7fcef406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.472+0000 7fcefbb08640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcef4070bb0 con 0x7fcef40744d0 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.472+0000 7fcefbb08640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcef4073f50 con 0x7fcef415e360 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.472+0000 7fcefbb08640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcef40748e0 con 0x7fcef4073b40 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcef40744d0 0x7fcef4073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcef40744d0 0x7fcef4073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40684/0 (socket says 172.21.15.46:40684) 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 -- 172.21.15.46:0/382582993 learned_addr learned my addr 172.21.15.46:0/382582993 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef98a4640 1 --2- 172.21.15.46:0/382582993 >> v2:172.21.15.88:3300/0 conn(0x7fcef415e360 0x7fcef40aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcefa0a5640 1 --2- 172.21.15.46:0/382582993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4073b40 0x7fcef406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 -- 172.21.15.46:0/382582993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4073b40 msgr2=0x7fcef406f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 --2- 172.21.15.46:0/382582993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4073b40 0x7fcef406f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 -- 172.21.15.46:0/382582993 >> v2:172.21.15.88:3300/0 conn(0x7fcef415e360 msgr2=0x7fcef40aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 --2- 172.21.15.46:0/382582993 >> v2:172.21.15.88:3300/0 conn(0x7fcef415e360 0x7fcef40aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.473+0000 7fcef90a3640 1 -- 172.21.15.46:0/382582993 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef40cec80 con 0x7fcef40744d0 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.474+0000 7fcef90a3640 1 --2- 172.21.15.46:0/382582993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef40744d0 0x7fcef4073600 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fcef40cee50 tx=0x7fcee402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d520151306b13a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.474+0000 7fcef88a2640 1 -- 172.21.15.46:0/382582993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4037070 con 0x7fcef40744d0 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.474+0000 7fcef88a2640 1 -- 172.21.15.46:0/382582993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee4032440 con 0x7fcef40744d0 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.474+0000 7fcef88a2640 1 -- 172.21.15.46:0/382582993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4032810 con 0x7fcef40744d0 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.474+0000 7fcefbb08640 1 -- 172.21.15.46:0/382582993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef40744d0 msgr2=0x7fcef4073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.474+0000 7fcefbb08640 1 --2- 172.21.15.46:0/382582993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef40744d0 0x7fcef4073600 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fcef40cee50 tx=0x7fcee402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 -- 172.21.15.46:0/382582993 shutdown_connections 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 --2- 172.21.15.46:0/382582993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef4073b40 0x7fcef406f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 --2- 172.21.15.46:0/382582993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef40744d0 0x7fcef4073600 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 --2- 172.21.15.46:0/382582993 >> v2:172.21.15.88:3300/0 conn(0x7fcef415e360 0x7fcef40aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 -- 172.21.15.46:0/382582993 >> 172.21.15.46:0/382582993 conn(0x7fcef40b3980 msgr2=0x7fcef40b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 -- 172.21.15.46:0/382582993 shutdown_connections 2024-05-09T03:17:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.475+0000 7fcefbb08640 1 -- 172.21.15.46:0/382582993 wait complete. 2024-05-09T03:17:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.476+0000 7fcefbb08640 1 Processor -- start 2024-05-09T03:17:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.476+0000 7fcefbb08640 1 -- start start 2024-05-09T03:17:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefbb08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef4073b40 0x7fcef4143e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefbb08640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef40744d0 0x7fcef415e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefbb08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 0x7fcef40d1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefbb08640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40af870 con 0x7fcef415e360 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefbb08640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcef40af590 con 0x7fcef4073b40 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefbb08640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcef40afb50 con 0x7fcef40744d0 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 0x7fcef40d1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 0x7fcef40d1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40686/0 (socket says 172.21.15.46:40686) 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 -- 172.21.15.46:0/913301554 learned_addr learned my addr 172.21.15.46:0/913301554 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcef98a4640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef4073b40 0x7fcef4143e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcef90a3640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef40744d0 0x7fcef415e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 -- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef40744d0 msgr2=0x7fcef415e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef40744d0 0x7fcef415e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 -- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef4073b40 msgr2=0x7fcef4143e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef4073b40 0x7fcef4143e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 -- 172.21.15.46:0/913301554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef415f430 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcef90a3640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef40744d0 0x7fcef415e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcef98a4640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef4073b40 0x7fcef4143e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fcefa0a5640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 0x7fcef40d1cf0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fceec007d40 tx=0x7fceec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec014420 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.478+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceec019070 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.479+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef4146930 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.479+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec014800 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.479+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef4146ef0 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.479+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef406a270 con 0x7fcef415e360 2024-05-09T03:17:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.480+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fceec0149a0 con 0x7fcef415e360 2024-05-09T03:17:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.481+0000 7fceea7fc640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fced80762d0 0x7fced80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.481+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7fceec098290 con 0x7fcef415e360 2024-05-09T03:17:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.481+0000 7fcef98a4640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fced80762d0 0x7fced80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.482+0000 7fcef98a4640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fced80762d0 0x7fced80787e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcef0006fd0 tx=0x7fcef0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.486+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fceec061130 con 0x7fcef415e360 2024-05-09T03:17:13.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:13.723+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcef41443a0 con 0x7fcef415e360 2024-05-09T03:17:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.109+0000 7fceea7fc640 1 -- 172.21.15.46:0/913301554 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v140) ==== 161+0+0 (secure 0 0 0) 0x7fceec065430 con 0x7fcef415e360 2024-05-09T03:17:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fced80762d0 msgr2=0x7fced80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fced80762d0 0x7fced80787e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcef0006fd0 tx=0x7fcef0008040 comp rx=0 tx=0).stop 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 msgr2=0x7fcef40d1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 0x7fcef40d1cf0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fceec007d40 tx=0x7fceec00e4b0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 shutdown_connections 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fced80762d0 0x7fced80787e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef415e360 0x7fcef40d1cf0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcef40744d0 0x7fcef415e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 --2- 172.21.15.46:0/913301554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcef4073b40 0x7fcef4143e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 >> 172.21.15.46:0/913301554 conn(0x7fcef40b3980 msgr2=0x7fcef40b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 shutdown_connections 2024-05-09T03:17:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.112+0000 7fcefbb08640 1 -- 172.21.15.46:0/913301554 wait complete. 2024-05-09T03:17:14.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:14.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.242+0000 7fbfdaeb1640 1 Processor -- start 2024-05-09T03:17:14.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.243+0000 7fbfdaeb1640 1 -- start start 2024-05-09T03:17:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfdaeb1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfd40d3730 0x7fbfd406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfdaeb1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd4164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfdaeb1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd409d470 con 0x7fbfd4164ec0 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfdaeb1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd409cfe0 con 0x7fbfd40d3730 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfdaeb1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd40b94a0 con 0x7fbfd406e7e0 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfd40d3730 0x7fbfd406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbfd40d3730 0x7fbfd406e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40690/0 (socket says 172.21.15.46:40690) 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 -- 172.21.15.46:0/619461742 learned_addr learned my addr 172.21.15.46:0/619461742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd3fff640 1 --2- 172.21.15.46:0/619461742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd4164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 -- 172.21.15.46:0/619461742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 msgr2=0x7fbfd4164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 --2- 172.21.15.46:0/619461742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd4164980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 -- 172.21.15.46:0/619461742 >> v1:172.21.15.88:6789/0 conn(0x7fbfd4164ec0 legacy=0x7fbfd40ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.244+0000 7fbfd8c4d640 1 -- 172.21.15.46:0/619461742 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd40d11b0 con 0x7fbfd40d3730 2024-05-09T03:17:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.245+0000 7fbfd8c4d640 1 --2- 172.21.15.46:0/619461742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd406e2a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009870 tx=0x7fbfcc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b02a93b2b8847 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.245+0000 7fbfd37fe640 1 -- 172.21.15.46:0/619461742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc037070 con 0x7fbfd40d3730 2024-05-09T03:17:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.245+0000 7fbfd37fe640 1 -- 172.21.15.46:0/619461742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfcc032440 con 0x7fbfd40d3730 2024-05-09T03:17:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.245+0000 7fbfd37fe640 1 -- 172.21.15.46:0/619461742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc032810 con 0x7fbfd40d3730 2024-05-09T03:17:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/619461742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 msgr2=0x7fbfd406e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/619461742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd406e2a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009870 tx=0x7fbfcc02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/619461742 shutdown_connections 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/619461742 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd4164980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/619461742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd406e2a0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/619461742 >> 172.21.15.46:0/619461742 conn(0x7fbfd40d05f0 msgr2=0x7fbfd40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/619461742 shutdown_connections 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.246+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/619461742 wait complete. 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.248+0000 7fbfdaeb1640 1 Processor -- start 2024-05-09T03:17:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.248+0000 7fbfdaeb1640 1 -- start start 2024-05-09T03:17:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.249+0000 7fbfdaeb1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd40ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.249+0000 7fbfdaeb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd4151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.249+0000 7fbfdaeb1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfd4164ec0 0x7fbfd41f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.249+0000 7fbfdaeb1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd408cd60 con 0x7fbfd40d3730 2024-05-09T03:17:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.249+0000 7fbfdaeb1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd40b9160 con 0x7fbfd4164ec0 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.249+0000 7fbfdaeb1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd408cee0 con 0x7fbfd406e7e0 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd4151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd4151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40706/0 (socket says 172.21.15.46:40706) 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 -- 172.21.15.46:0/849976659 learned_addr learned my addr 172.21.15.46:0/849976659 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd944e640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfd4164ec0 0x7fbfd41f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd8c4d640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd40ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 -- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 msgr2=0x7fbfd40ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd40ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 -- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfd4164ec0 msgr2=0x7fbfd41f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfd4164ec0 0x7fbfd41f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 -- 172.21.15.46:0/849976659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd40ccbf0 con 0x7fbfd40d3730 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd3fff640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd4151f60 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4007e70 tx=0x7fbfc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4014620 con 0x7fbfd40d3730 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc4019070 con 0x7fbfd40d3730 2024-05-09T03:17:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4014a00 con 0x7fbfd40d3730 2024-05-09T03:17:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.250+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd40cced0 con 0x7fbfd40d3730 2024-05-09T03:17:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.251+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd40cae70 con 0x7fbfd40d3730 2024-05-09T03:17:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.251+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd40cc960 con 0x7fbfd40d3730 2024-05-09T03:17:14.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.252+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbfc4014ba0 con 0x7fbfd40d3730 2024-05-09T03:17:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.253+0000 7fbfd17fa640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfb4076300 0x7fbfb4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.253+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7fbfc4098020 con 0x7fbfd40d3730 2024-05-09T03:17:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.253+0000 7fbfd8c4d640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfb4076300 0x7fbfb4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:14.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.253+0000 7fbfd8c4d640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfb4076300 0x7fbfb4078810 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc034000 tx=0x7fbfcc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.257+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbfc40616d0 con 0x7fbfd40d3730 2024-05-09T03:17:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:14.523+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fbfd40b7730 con 0x7fbfd40d3730 2024-05-09T03:17:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.114+0000 7fbfd17fa640 1 -- 172.21.15.46:0/849976659 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v141) ==== 101+0+0 (secure 0 0 0) 0x7fbfd40b7730 con 0x7fbfd40d3730 2024-05-09T03:17:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfb4076300 msgr2=0x7fbfb4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfb4076300 0x7fbfb4078810 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc034000 tx=0x7fbfcc02e000 comp rx=0 tx=0).stop 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 msgr2=0x7fbfd4151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd4151f60 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbfc4007e70 tx=0x7fbfc400e4b0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 shutdown_connections 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfd4164ec0 0x7fbfd41f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfb4076300 0x7fbfb4078810 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd40d3730 0x7fbfd4151f60 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 --2- 172.21.15.46:0/849976659 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfd406e7e0 0x7fbfd40ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 >> 172.21.15.46:0/849976659 conn(0x7fbfd40d05f0 msgr2=0x7fbfd40b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 shutdown_connections 2024-05-09T03:17:15.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.117+0000 7fbfdaeb1640 1 -- 172.21.15.46:0/849976659 wait complete. 2024-05-09T03:17:15.132 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:15.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.192+0000 7fb75c5e0f40 1 Processor -- start 2024-05-09T03:17:15.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.192+0000 7fb75c5e0f40 1 -- start start 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb75c5e0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55556069b3a0 0x555560726680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb75c5e0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 0x55556072ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb75c5e0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556072ea40 0x55556072f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb75c5e0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5555606a45f0 con 0x5555606a35f0 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb75c5e0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555560601f50 con 0x55556069b3a0 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb75c5e0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5555606a48d0 con 0x55556072ea40 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb7558a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 0x55556072ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb7560a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55556069b3a0 0x555560726680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb7558a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 0x55556072ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40716/0 (socket says 172.21.15.46:40716) 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb7558a7640 1 -- 172.21.15.46:0/4144838547 learned_addr learned my addr 172.21.15.46:0/4144838547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:15.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.193+0000 7fb7568a9640 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556072ea40 0x55556072f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7558a7640 1 -- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556072ea40 msgr2=0x55556072f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7558a7640 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556072ea40 0x55556072f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7558a7640 1 -- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55556069b3a0 msgr2=0x555560726680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7558a7640 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55556069b3a0 0x555560726680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7558a7640 1 -- 172.21.15.46:0/4144838547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555606863d0 con 0x5555606a35f0 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7560a8640 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55556069b3a0 0x555560726680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7558a7640 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 0x55556072ee50 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fb74c001e60 tx=0x7fb74c00b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11cd77cf49e03ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7548a5640 1 -- 172.21.15.46:0/4144838547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74c008070 con 0x5555606a35f0 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7548a5640 1 -- 172.21.15.46:0/4144838547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb74c00d500 con 0x5555606a35f0 2024-05-09T03:17:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.194+0000 7fb7548a5640 1 -- 172.21.15.46:0/4144838547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74c00d910 con 0x5555606a35f0 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/4144838547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 msgr2=0x55556072ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 0x55556072ee50 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fb74c001e60 tx=0x7fb74c00b6a0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/4144838547 shutdown_connections 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556072ea40 0x55556072f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555606a35f0 0x55556072ee50 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/4144838547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55556069b3a0 0x555560726680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/4144838547 >> 172.21.15.46:0/4144838547 conn(0x55556057dac0 msgr2=0x55556069e740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/4144838547 shutdown_connections 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.195+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/4144838547 wait complete. 2024-05-09T03:17:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.196+0000 7fb75c5e0f40 1 Processor -- start 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.196+0000 7fb75c5e0f40 1 -- start start 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb75c5e0f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556069b3a0 0x5555607208c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb75c5e0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5555606a35f0 0x555560720140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb75c5e0f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 0x55556065c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb75c5e0f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555560725550 con 0x55556072ea40 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb75c5e0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555560720e00 con 0x5555606a35f0 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb75c5e0f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555560721050 con 0x55556069b3a0 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 0x55556065c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 0x55556065c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40730/0 (socket says 172.21.15.46:40730) 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 learned_addr learned my addr 172.21.15.46:0/898181903 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556069b3a0 0x5555607208c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5555606a35f0 0x555560720140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556069b3a0 msgr2=0x5555607208c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55556069b3a0 0x5555607208c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5555606a35f0 msgr2=0x555560720140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5555606a35f0 0x555560720140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.197+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555560720680 con 0x55556072ea40 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.198+0000 7fb7568a9640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 0x55556065c6b0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fb7500106b0 tx=0x7fb750010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.198+0000 7fb7477fe640 1 -- 172.21.15.46:0/898181903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb750019070 con 0x55556072ea40 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.198+0000 7fb7477fe640 1 -- 172.21.15.46:0/898181903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb750010e80 con 0x55556072ea40 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.198+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5555607be690 con 0x55556072ea40 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.198+0000 7fb7477fe640 1 -- 172.21.15.46:0/898181903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75000ebb0 con 0x55556072ea40 2024-05-09T03:17:15.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.198+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555607bec50 con 0x55556072ea40 2024-05-09T03:17:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.200+0000 7fb7477fe640 1 -- 172.21.15.46:0/898181903 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb750011040 con 0x55556072ea40 2024-05-09T03:17:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.200+0000 7fb7477fe640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb738074ce0 0x7fb7380771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.200+0000 7fb7477fe640 1 -- 172.21.15.46:0/898181903 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7fb7500969e0 con 0x55556072ea40 2024-05-09T03:17:15.203 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.200+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555560724ad0 0x5555607d4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.200+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555606a0aa0 con 0x555560724ad0 2024-05-09T03:17:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.201+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb738074ce0 0x7fb7380771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.201+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555560724ad0 0x5555607d4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.201+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555560724ad0 0x5555607d4590 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.201+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb738074ce0 0x7fb7380771f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb748056000 tx=0x7fb748050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.203+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v93'257 uv231 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.203+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607bf730 con 0x555560724ad0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.204+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v93'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.204+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d4ad0 con 0x555560724ad0 2024-05-09T03:17:15.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.205+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v93'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.207 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:15.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.205+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d5e80 con 0x555560724ad0 2024-05-09T03:17:15.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.207+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v93'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.207+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x555560724ee0 con 0x555560724ad0 2024-05-09T03:17:15.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.208+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v93'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.208+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d4e70 con 0x555560724ad0 2024-05-09T03:17:15.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.209+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v93'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.209+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d5280 con 0x555560724ad0 2024-05-09T03:17:15.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.210+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v93'263 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.210+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607d5620 con 0x555560724ad0 2024-05-09T03:17:15.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.212+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v93'264 uv264 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.212+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d5a10 con 0x555560724ad0 2024-05-09T03:17:15.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.213+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v93'265 uv265 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.215 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:15.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.213+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d5a10 con 0x555560724ad0 2024-05-09T03:17:15.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.214+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v93'266 uv266 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.214+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.216+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v93'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.216+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.217+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v93'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.217+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.218+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v93'269 uv244 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.218+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.221+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v93'270 uv270 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.221+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.223 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:15.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.222+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v93'271 uv271 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.222+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.223+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v93'272 uv272 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.223+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.225+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v93'273 uv273 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.225+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d59c0 con 0x555560724ad0 2024-05-09T03:17:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.226+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v93'274 uv274 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.226+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.227+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v93'275 uv275 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.228+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.229+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v93'276 uv250 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.229+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.230+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v93'277 uv277 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.230+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.231+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v93'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.233 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:15.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.231+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d7230 con 0x555560724ad0 2024-05-09T03:17:15.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.233+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v93'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.233+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d7230 con 0x555560724ad0 2024-05-09T03:17:15.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.234+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v93'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.234+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.235+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v93'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.235+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.236+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v93'282 uv256 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.237+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.238+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v93'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.238+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.239+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v93'284 uv284 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.241 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:15.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.239+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d7230 con 0x555560724ad0 2024-05-09T03:17:15.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.240+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v93'285 uv285 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.241+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d7230 con 0x555560724ad0 2024-05-09T03:17:15.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.242+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v93'286 uv286 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.242+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d6220 con 0x555560724ad0 2024-05-09T03:17:15.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.243+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v93'287 uv287 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.243+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5555607d6730 0x5555607d8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.243+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d9090 con 0x5555607d6730 2024-05-09T03:17:15.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.243+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5555607d6730 0x5555607d8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.244+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5555607d6730 0x5555607d8b50 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.245+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v93'196 uv179 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.245+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607d9430 con 0x5555607d6730 2024-05-09T03:17:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.247+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v93'197 uv197 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.247+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607d97d0 con 0x5555607d6730 2024-05-09T03:17:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.248+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v93'198 uv198 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.249 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.248+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607da020 con 0x5555607d6730 2024-05-09T03:17:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.249+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v93'199 uv199 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.249+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607da3c0 con 0x5555607d6730 2024-05-09T03:17:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.252+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v93'200 uv200 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.252+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607da760 con 0x5555607d6730 2024-05-09T03:17:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.253+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v93'201 uv184 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.253+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607dab00 con 0x5555607d6730 2024-05-09T03:17:15.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.255+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v93'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.255+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.256+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v93'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.258 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.256+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.259+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v93'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.259+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.261+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v93'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.261+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.262+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v93'206 uv189 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.262+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.264+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v93'207 uv207 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.264+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.265+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v93'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.267 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.265+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de900 con 0x5555607d6730 2024-05-09T03:17:15.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.267+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v93'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.267+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de900 con 0x5555607d6730 2024-05-09T03:17:15.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.268+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v93'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.268+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.269+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v93'211 uv195 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.269+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.271+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v93'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.271+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.272+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v93'213 uv213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.274 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.272+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607dbeb0 con 0x5555607d6730 2024-05-09T03:17:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.273+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v93'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.273+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607dbeb0 con 0x5555607d6730 2024-05-09T03:17:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.275+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v93'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.275+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x5555607d6730 2024-05-09T03:17:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.276+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v93'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607d6730 2024-05-09T03:17:15.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.276+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x555560724ad0 2024-05-09T03:17:15.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.277+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v93'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.277+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607daea0 con 0x555560724ad0 2024-05-09T03:17:15.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.278+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v93'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.278+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x555560724ad0 2024-05-09T03:17:15.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.279+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v93'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.281 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:15.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.279+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607dbeb0 con 0x555560724ad0 2024-05-09T03:17:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.280+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v93'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.280+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607dbeb0 con 0x555560724ad0 2024-05-09T03:17:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.281+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v93'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.281+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607daea0 con 0x555560724ad0 2024-05-09T03:17:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.283+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v93'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.283+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5555607db7a0 0x5555607ddbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.283+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de100 con 0x5555607db7a0 2024-05-09T03:17:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.283+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5555607db7a0 0x5555607ddbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.283+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5555607db7a0 0x5555607ddbc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.284+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v93'44 uv43 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.284+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607de180 con 0x5555607db7a0 2024-05-09T03:17:15.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.285+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v93'45 uv45 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.285+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de5a0 con 0x5555607db7a0 2024-05-09T03:17:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.286+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v93'46 uv46 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.288 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:15.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.286+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de5a0 con 0x5555607db7a0 2024-05-09T03:17:15.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.287+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v93'47 uv47 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.287+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de5a0 con 0x555560724ad0 2024-05-09T03:17:15.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.289+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v93'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.289+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607de5a0 con 0x555560724ad0 2024-05-09T03:17:15.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.290+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v93'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.290+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de5a0 con 0x555560724ad0 2024-05-09T03:17:15.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.291+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v93'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.293 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:15.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.291+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607df5b0 con 0x555560724ad0 2024-05-09T03:17:15.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.292+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v93'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.292+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607df5b0 con 0x555560724ad0 2024-05-09T03:17:15.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.293+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v93'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.293+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607de5a0 con 0x555560724ad0 2024-05-09T03:17:15.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.294+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v93'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.294+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5555607dec20 0x5555607e1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.294+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e1650 con 0x5555607dec20 2024-05-09T03:17:15.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.295+0000 7fb7568a9640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5555607dec20 0x5555607e1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.295+0000 7fb7568a9640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5555607dec20 0x5555607e1040 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.296+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v93'94 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.296+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607e1b40 con 0x5555607dec20 2024-05-09T03:17:15.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.297+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v93'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.297+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e2020 con 0x5555607dec20 2024-05-09T03:17:15.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.298+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v93'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.300 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:15.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.298+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e38d0 con 0x5555607dec20 2024-05-09T03:17:15.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.299+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v93'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.300+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e3c70 con 0x5555607dec20 2024-05-09T03:17:15.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.301+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v93'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.301+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e28c0 con 0x5555607dec20 2024-05-09T03:17:15.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.302+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v93'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.302+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e2cb0 con 0x5555607dec20 2024-05-09T03:17:15.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.303+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v93'100 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.303+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607e30a0 con 0x5555607dec20 2024-05-09T03:17:15.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.304+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v93'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.304+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e30a0 con 0x5555607dec20 2024-05-09T03:17:15.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.305+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v93'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.307 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:15.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.305+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e3050 con 0x5555607dec20 2024-05-09T03:17:15.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.307+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v93'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.307+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e3050 con 0x5555607dec20 2024-05-09T03:17:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.308+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v93'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.308+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5555607e3410 0x5555607e6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.308+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6a70 con 0x5555607e3410 2024-05-09T03:17:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.308+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5555607e3410 0x5555607e6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.309+0000 7fb7560a8640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5555607e3410 0x5555607e6530 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.310+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v93'43 uv43 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607e3410 2024-05-09T03:17:15.311 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:15.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.310+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6a70 con 0x5555607e3410 2024-05-09T03:17:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.311+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [write 0~70] v93'44 uv44 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607e3410 2024-05-09T03:17:15.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.311+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e2490 con 0x5555607e3410 2024-05-09T03:17:15.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.312+0000 7fb7560a8640 1 -- 172.21.15.46:0/898181903 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v93'45 uv45 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb748064070 con 0x5555607e3410 2024-05-09T03:17:15.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.312+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e2490 con 0x555560724ad0 2024-05-09T03:17:15.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.313+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(83 605.00000000 [create] v93'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.313+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607e2490 con 0x555560724ad0 2024-05-09T03:17:15.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.314+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(84 605.00000000 [delete] v93'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.314+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e2490 con 0x555560724ad0 2024-05-09T03:17:15.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.315+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(85 605.00000000 [create] v93'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.317 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:15.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.315+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e7fd0 con 0x555560724ad0 2024-05-09T03:17:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.316+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v93'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.316+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e7fd0 con 0x555560724ad0 2024-05-09T03:17:15.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.317+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v93'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.317+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f70 con 0x555560724ad0 2024-05-09T03:17:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.318+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v93'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x555560724ad0 2024-05-09T03:17:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.318+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.320+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(89 mds0_inotable [create] v93'84 uv78 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.320+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.321+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(90 mds0_inotable [delete] v93'85 uv85 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.321+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.322+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(91 mds0_inotable [create] v93'86 uv86 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.323 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.322+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.323+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v93'87 uv87 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.323+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.324+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v93'88 uv88 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.324+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.325+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(94 100.00000000 [create] v93'89 uv83 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.325+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.326+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(95 100.00000000 [delete] v93'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.326+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.327+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(96 100.00000000 [create] v93'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.329 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:15.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.327+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e6f20 con 0x5555607db7a0 2024-05-09T03:17:15.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.328+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v93'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.328+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607e9b30 con 0x5555607db7a0 2024-05-09T03:17:15.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.330+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v93'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c00d730 con 0x5555607db7a0 2024-05-09T03:17:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.330+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5555607e6a70 0x5555607e8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.330+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea1f0 con 0x5555607e6a70 2024-05-09T03:17:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.330+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5555607e6a70 0x5555607e8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.330+0000 7fb7558a7640 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5555607e6a70 0x5555607e8e90 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.331+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v93'40 uv39 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607e6a70 2024-05-09T03:17:15.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.331+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607ea1f0 con 0x5555607e6a70 2024-05-09T03:17:15.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.332+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v93'41 uv41 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607e6a70 2024-05-09T03:17:15.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.333+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea1f0 con 0x5555607e6a70 2024-05-09T03:17:15.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.334+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v93'42 uv42 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607e6a70 2024-05-09T03:17:15.335 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.334+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607e6a70 2024-05-09T03:17:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.335+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v93'43 uv43 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607e6a70 2024-05-09T03:17:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.335+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607db7a0 2024-05-09T03:17:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.336+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v93'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607db7a0 2024-05-09T03:17:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.336+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607db7a0 2024-05-09T03:17:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.337+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v93'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607db7a0 2024-05-09T03:17:15.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.337+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607db7a0 2024-05-09T03:17:15.340 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.338+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v93'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607db7a0 2024-05-09T03:17:15.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.338+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607eb5f0 con 0x5555607db7a0 2024-05-09T03:17:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.339+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v93'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607db7a0 2024-05-09T03:17:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.339+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607eb5f0 con 0x5555607db7a0 2024-05-09T03:17:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.341+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v93'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607db7a0 2024-05-09T03:17:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.341+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607db7a0 2024-05-09T03:17:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.342+0000 7fb7558a7640 1 -- 172.21.15.46:0/898181903 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v93'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb74c02a040 con 0x5555607db7a0 2024-05-09T03:17:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.342+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.343+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v93'42 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.343+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.344+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v93'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.344+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.347 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.345+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v93'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.345+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.347+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v93'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.347+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.348+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v93'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.348+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.349+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v93'107 uv99 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.349+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.350+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v93'108 uv108 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.351+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.353 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:15.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.352+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v93'109 uv109 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.352+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.353+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v93'110 uv110 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.353+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.354+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v93'111 uv111 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.354+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.355+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v93'112 uv106 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.355+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.356+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v93'113 uv113 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.356+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.357+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v93'114 uv114 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.359 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.357+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607eb5f0 con 0x5555607dec20 2024-05-09T03:17:15.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.358+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v93'115 uv115 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.358+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607eb5f0 con 0x5555607dec20 2024-05-09T03:17:15.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.360+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v93'116 uv116 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.360+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ea5e0 con 0x5555607dec20 2024-05-09T03:17:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.361+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v93'117 uv117 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.361+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93) -- 0x5555607ead00 con 0x5555607dec20 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v93'118 uv118 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb750024070 con 0x5555607dec20 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555560724ad0 msgr2=0x5555607d4590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x555560724ad0 0x5555607d4590 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5555607db7a0 msgr2=0x5555607ddbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5555607db7a0 0x5555607ddbc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5555607e6a70 msgr2=0x5555607e8e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5555607e6a70 0x5555607e8e90 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5555607d6730 msgr2=0x5555607d8b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5555607d6730 0x5555607d8b50 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 reap_dead start 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5555607dec20 msgr2=0x5555607e1040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb7568a9640 1 -- 172.21.15.46:0/898181903 reap_dead start 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5555607dec20 0x5555607e1040 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5555607e3410 msgr2=0x5555607e6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5555607e3410 0x5555607e6530 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb738074ce0 msgr2=0x7fb7380771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb738074ce0 0x7fb7380771f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb748056000 tx=0x7fb748050000 comp rx=0 tx=0).stop 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 msgr2=0x55556065c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 0x55556065c6b0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fb7500106b0 tx=0x7fb750010b70 comp rx=0 tx=0).stop 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 shutdown_connections 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5555607e3410 0x5555607e6530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5555607dec20 0x5555607e1040 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb738074ce0 0x7fb7380771f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 --2- 172.21.15.46:0/898181903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55556072ea40 0x55556065c6b0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 >> 172.21.15.46:0/898181903 conn(0x55556057dac0 msgr2=0x55556072ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 shutdown_connections 2024-05-09T03:17:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.363+0000 7fb75c5e0f40 1 -- 172.21.15.46:0/898181903 wait complete. 2024-05-09T03:17:15.369 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_inotable 2024-05-09T03:17:15.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.463+0000 7f986b904f40 1 Processor -- start 2024-05-09T03:17:15.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.463+0000 7f986b904f40 1 -- start start 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f986b904f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514117e0 0x560b514553f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f986b904f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514437b0 0x560b514ec630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f986b904f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f986b904f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560b513baf50 con 0x560b514eb810 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f986b904f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560b513bef90 con 0x560b514117e0 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f986b904f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x560b514584d0 con 0x560b514437b0 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f9865bcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f9865bcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514ecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40752/0 (socket says 172.21.15.46:40752) 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f9865bcd640 1 -- 172.21.15.46:0/1699118232 learned_addr learned my addr 172.21.15.46:0/1699118232 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.464+0000 7f98653cc640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514117e0 0x560b514553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9864bcb640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514437b0 0x560b514ec630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9865bcd640 1 -- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514437b0 msgr2=0x560b514ec630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9865bcd640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514437b0 0x560b514ec630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9865bcd640 1 -- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514117e0 msgr2=0x560b514553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9865bcd640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514117e0 0x560b514553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9865bcd640 1 -- 172.21.15.46:0/1699118232 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b5143dcd0 con 0x560b514eb810 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9864bcb640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514437b0 0x560b514ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f98653cc640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514117e0 0x560b514553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f9865bcd640 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514ecb70 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f9858001680 tx=0x7f9858001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67126f983aaaf22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f985f7fe640 1 -- 172.21.15.46:0/1699118232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858004520 con 0x560b514eb810 2024-05-09T03:17:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f985f7fe640 1 -- 172.21.15.46:0/1699118232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98580046c0 con 0x560b514eb810 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f985f7fe640 1 -- 172.21.15.46:0/1699118232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858004a90 con 0x560b514eb810 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f986b904f40 1 -- 172.21.15.46:0/1699118232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 msgr2=0x560b514ecb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.465+0000 7f986b904f40 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514ecb70 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f9858001680 tx=0x7f9858001b40 comp rx=0 tx=0).stop 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 -- 172.21.15.46:0/1699118232 shutdown_connections 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514ecb70 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514437b0 0x560b514ec630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 --2- 172.21.15.46:0/1699118232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514117e0 0x560b514553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 -- 172.21.15.46:0/1699118232 >> 172.21.15.46:0/1699118232 conn(0x560b51336ac0 msgr2=0x560b5145ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 -- 172.21.15.46:0/1699118232 shutdown_connections 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 -- 172.21.15.46:0/1699118232 wait complete. 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.466+0000 7f986b904f40 1 Processor -- start 2024-05-09T03:17:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.467+0000 7f986b904f40 1 -- start start 2024-05-09T03:17:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f986b904f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514117e0 0x560b51456ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f986b904f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514437b0 0x560b514deef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f986b904f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f986b904f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560b514ed0b0 con 0x560b514eb810 2024-05-09T03:17:15.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f986b904f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560b514e4800 con 0x560b514437b0 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f986b904f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x560b5143dfb0 con 0x560b514117e0 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f98653cc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514117e0 0x560b51456ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9864bcb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514437b0 0x560b514deef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514e5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40762/0 (socket says 172.21.15.46:40762) 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9864bcb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514437b0 0x560b514deef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:50560/0 (socket says 172.21.15.46:50560) 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 -- 172.21.15.46:0/1297473730 learned_addr learned my addr 172.21.15.46:0/1297473730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f98653cc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514117e0 0x560b51456ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:51350/0 (socket says 172.21.15.46:51350) 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 -- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514117e0 msgr2=0x560b51456ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560b514117e0 0x560b51456ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 -- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514437b0 msgr2=0x560b514deef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560b514437b0 0x560b514deef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.468+0000 7f9865bcd640 1 -- 172.21.15.46:0/1297473730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b51455e40 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.469+0000 7f9865bcd640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514e5b80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x560b51441120 tx=0x7f9858001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.469+0000 7f985e7fc640 1 -- 172.21.15.46:0/1297473730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858021030 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.469+0000 7f985e7fc640 1 -- 172.21.15.46:0/1297473730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98580134e0 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.469+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560b514560e0 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.469+0000 7f985e7fc640 1 -- 172.21.15.46:0/1297473730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858018040 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.469+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b514566a0 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.470+0000 7f985e7fc640 1 -- 172.21.15.46:0/1297473730 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9858024070 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.471+0000 7f985e7fc640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9848074ce0 0x7f98480771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.471+0000 7f985e7fc640 1 -- 172.21.15.46:0/1297473730 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7f9858097860 con 0x560b514eb810 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.471+0000 7f986b904f40 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560b5143e9d0 0x560b514e20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.471+0000 7f98653cc640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9848074ce0 0x7f98480771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.471+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e93) -- 0x560b514e25e0 con 0x560b5143e9d0 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.471+0000 7f9864bcb640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560b5143e9d0 0x560b514e20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.472+0000 7f98653cc640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9848074ce0 0x7f98480771f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x560b514e8790 tx=0x7f9860001800 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.472+0000 7f9864bcb640 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560b5143e9d0 0x560b514e20a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f9864bcb640 1 -- 172.21.15.46:0/1297473730 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_inotable [delete] v93'94 uv94 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9850052a70 con 0x560b5143e9d0 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560b5143e9d0 msgr2=0x560b514e20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560b5143e9d0 0x560b514e20a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9848074ce0 msgr2=0x7f98480771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9848074ce0 0x7f98480771f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x560b514e8790 tx=0x7f9860001800 comp rx=0 tx=0).stop 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 msgr2=0x560b514e5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 --2- 172.21.15.46:0/1297473730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560b514eb810 0x560b514e5b80 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x560b51441120 tx=0x7f9858001250 comp rx=0 tx=0).stop 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f9865bcd640 1 -- 172.21.15.46:0/1297473730 reap_dead start 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 shutdown_connections 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.473+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 >> 172.21.15.46:0/1297473730 conn(0x560b51336ac0 msgr2=0x560b514ead50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.474+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 shutdown_connections 2024-05-09T03:17:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.474+0000 7f986b904f40 1 -- 172.21.15.46:0/1297473730 wait complete. 2024-05-09T03:17:15.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:15.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.596+0000 7f644a4d9640 1 Processor -- start 2024-05-09T03:17:15.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.596+0000 7f644a4d9640 1 -- start start 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f644a4d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64440d1740 0x7f64440b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f644a4d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f644409cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f644a4d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64440b9690 con 0x7f64440d1110 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f644a4d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f64440d2b00 con 0x7f64440d1740 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f644a4d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f644408cd60 con 0x7f64440cfcf0 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f6443fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64440d1740 0x7f64440b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f6443fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f64440d1740 0x7f64440b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40774/0 (socket says 172.21.15.46:40774) 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f6443fff640 1 -- 172.21.15.46:0/2142179103 learned_addr learned my addr 172.21.15.46:0/2142179103 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f64437fe640 1 --1- 172.21.15.46:0/2142179103 >> v1:172.21.15.46:6789/0 conn(0x7f64440d1110 0x7f64440d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48560/0 (socket says 172.21.15.46:48560) 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.597+0000 7f6448a76640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f644409cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6443fff640 1 -- 172.21.15.46:0/2142179103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 msgr2=0x7f644409cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6443fff640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f644409cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6443fff640 1 -- 172.21.15.46:0/2142179103 >> v1:172.21.15.46:6789/0 conn(0x7f64440d1110 legacy=0x7f64440d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6443fff640 1 -- 172.21.15.46:0/2142179103 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64440ca9e0 con 0x7f64440d1740 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6442ffd640 1 -- 172.21.15.46:0/2142179103 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101791911 0 0) 0x7f64440b9690 con 0x7f64440d1110 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6448a76640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f644409cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6443fff640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f64440b73d0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f6434009870 tx=0x7f643402a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f518ea2e190d08d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.598+0000 7f6442ffd640 1 -- 172.21.15.46:0/2142179103 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434037070 con 0x7f64440d1740 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f6442ffd640 1 -- 172.21.15.46:0/2142179103 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6434032440 con 0x7f64440d1740 2024-05-09T03:17:15.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f6442ffd640 1 -- 172.21.15.46:0/2142179103 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434032810 con 0x7f64440d1740 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 -- 172.21.15.46:0/2142179103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 msgr2=0x7f64440b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f64440b73d0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f6434009870 tx=0x7f643402a240 comp rx=0 tx=0).stop 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 -- 172.21.15.46:0/2142179103 shutdown_connections 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f644409cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 --2- 172.21.15.46:0/2142179103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f64440b73d0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 -- 172.21.15.46:0/2142179103 >> 172.21.15.46:0/2142179103 conn(0x7f64440d05f0 msgr2=0x7f64440d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 -- 172.21.15.46:0/2142179103 shutdown_connections 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.599+0000 7f644a4d9640 1 -- 172.21.15.46:0/2142179103 wait complete. 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.601+0000 7f644a4d9640 1 Processor -- start 2024-05-09T03:17:15.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.601+0000 7f644a4d9640 1 -- start start 2024-05-09T03:17:15.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f644a4d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f6444156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f644a4d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64440d1110 0x7f644415f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f644a4d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f644409f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f644a4d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64440b93b0 con 0x7f64440d1740 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f644a4d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f64440d1ca0 con 0x7f64440d1110 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f644a4d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f644409d1e0 con 0x7f64440cfcf0 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f644409f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f644409f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40778/0 (socket says 172.21.15.46:40778) 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 -- 172.21.15.46:0/1644390609 learned_addr learned my addr 172.21.15.46:0/1644390609 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6443fff640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f6444156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f64437fe640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64440d1110 0x7f644415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 -- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 msgr2=0x7f6444156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f6444156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 -- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64440d1110 msgr2=0x7f644415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64440d1110 0x7f644415f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 -- 172.21.15.46:0/1644390609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64440ca000 con 0x7f64440d1740 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.603+0000 7f6448a76640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f644409f070 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f643c007df0 tx=0x7f643c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.604+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c01a070 con 0x7f64440d1740 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.604+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f643c003990 con 0x7f64440d1740 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.604+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c003d80 con 0x7f64440d1740 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.604+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6444160310 con 0x7f64440d1740 2024-05-09T03:17:15.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.604+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64441608d0 con 0x7f64440d1740 2024-05-09T03:17:15.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.605+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6444160080 con 0x7f64440d1740 2024-05-09T03:17:15.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.605+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f643c005d10 con 0x7f64440d1740 2024-05-09T03:17:15.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.606+0000 7f6440ff9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64300762d0 0x7f64300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:15.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.606+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 6541+0+0 (secure 0 0 0) 0x7f643c097fb0 con 0x7f64440d1740 2024-05-09T03:17:15.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.606+0000 7f6443fff640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64300762d0 0x7f64300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:15.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.607+0000 7f6443fff640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64300762d0 0x7f64300787e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6434034000 tx=0x7f643402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.610+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f643c024080 con 0x7f64440d1740 2024-05-09T03:17:15.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:15.839+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f64440b65b0 con 0x7f64440d1740 2024-05-09T03:17:16.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.120+0000 7f6440ff9640 1 -- 172.21.15.46:0/1644390609 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v142) ==== 162+0+0 (secure 0 0 0) 0x7f64440b65b0 con 0x7f64440d1740 2024-05-09T03:17:16.122 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64300762d0 msgr2=0x7f64300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64300762d0 0x7f64300787e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6434034000 tx=0x7f643402e000 comp rx=0 tx=0).stop 2024-05-09T03:17:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 msgr2=0x7f644409f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f644409f070 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f643c007df0 tx=0x7f643c00d4a0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 shutdown_connections 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f64300762d0 0x7f64300787e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64440d1740 0x7f644409f070 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64440d1110 0x7f644415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 --2- 172.21.15.46:0/1644390609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f64440cfcf0 0x7f6444156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 >> 172.21.15.46:0/1644390609 conn(0x7f64440d05f0 msgr2=0x7f6444072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.122+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 shutdown_connections 2024-05-09T03:17:16.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.123+0000 7f644a4d9640 1 -- 172.21.15.46:0/1644390609 wait complete. 2024-05-09T03:17:16.129 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:16.127+0000 7f5a4ae52640 -1 mds.0.inotable: load_2 could not read table: -2 2024-05-09T03:17:16.129 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:16.127+0000 7f5a4ae52640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds0_inotable' -2 ((2) No such file or directory) 2024-05-09T03:17:16.129 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:16.128+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:16.130 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:16.128+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:16.135 INFO:tasks.ceph.mds.c.smithi046.stderr: -27> 2024-05-09T03:17:16.127+0000 7f5a4ae52640 -1 mds.0.inotable: load_2 could not read table: -2 2024-05-09T03:17:16.135 INFO:tasks.ceph.mds.c.smithi046.stderr: -26> 2024-05-09T03:17:16.127+0000 7f5a4ae52640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds0_inotable' -2 ((2) No such file or directory) 2024-05-09T03:17:16.138 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:16.192 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.255+0000 7f2c6bd7d640 1 Processor -- start 2024-05-09T03:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.255+0000 7f2c6bd7d640 1 -- start start 2024-05-09T03:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6bd7d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b5210 0x7f2c641613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6bd7d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c640b7ed0 0x7f2c640d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6bd7d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c640d5280 con 0x7f2c640d4800 2024-05-09T03:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6bd7d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2c640b5620 con 0x7f2c640b7ed0 2024-05-09T03:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6bd7d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c641609a0 con 0x7f2c640b5210 2024-05-09T03:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6a31a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c640b7ed0 0x7f2c640d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c69318640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2c640d4800 0x7f2c640b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38096/0 (socket says 172.21.15.46:38096) 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c69318640 1 -- 172.21.15.46:0/2906162922 learned_addr learned my addr 172.21.15.46:0/2906162922 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c69b19640 1 --2- 172.21.15.46:0/2906162922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b5210 0x7f2c641613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2906162922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b5210 msgr2=0x7f2c641613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6a31a640 1 --2- 172.21.15.46:0/2906162922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b5210 0x7f2c641613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2906162922 >> v1:172.21.15.46:6789/0 conn(0x7f2c640d4800 legacy=0x7f2c640b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2906162922 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c640d5560 con 0x7f2c640b7ed0 2024-05-09T03:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.257+0000 7f2c68b17640 1 -- 172.21.15.46:0/2906162922 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1284111741 0 0) 0x7f2c640d5280 con 0x7f2c640d4800 2024-05-09T03:17:16.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.258+0000 7f2c6a31a640 1 --2- 172.21.15.46:0/2906162922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640b7ed0 0x7f2c640d4c10 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009d90 tx=0x7f2c5c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d523dbe6f5ea3e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:16.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.258+0000 7f2c68b17640 1 -- 172.21.15.46:0/2906162922 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c007740 con 0x7f2c640b7ed0 2024-05-09T03:17:16.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.258+0000 7f2c68b17640 1 -- 172.21.15.46:0/2906162922 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5c0078e0 con 0x7f2c640b7ed0 2024-05-09T03:17:16.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.258+0000 7f2c68b17640 1 -- 172.21.15.46:0/2906162922 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c007cb0 con 0x7f2c640b7ed0 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2906162922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640b7ed0 msgr2=0x7f2c640d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2906162922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640b7ed0 0x7f2c640d4c10 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009d90 tx=0x7f2c5c02a5f0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2906162922 shutdown_connections 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2906162922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640b7ed0 0x7f2c640d4c10 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2906162922 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640b5210 0x7f2c641613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2906162922 >> 172.21.15.46:0/2906162922 conn(0x7f2c640d29a0 msgr2=0x7f2c640d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2906162922 shutdown_connections 2024-05-09T03:17:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.259+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2906162922 wait complete. 2024-05-09T03:17:16.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.261+0000 7f2c6bd7d640 1 Processor -- start 2024-05-09T03:17:16.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.261+0000 7f2c6bd7d640 1 -- start start 2024-05-09T03:17:16.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6bd7d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640b5210 0x7f2c640ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6bd7d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640d4800 0x7f2c640caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6bd7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 0x7f2c640cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6bd7d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c640d0ae0 con 0x7f2c640cbf60 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6bd7d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2c64160ea0 con 0x7f2c640b5210 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6bd7d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c64163e60 con 0x7f2c640d4800 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6a31a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 0x7f2c640cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6a31a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 0x7f2c640cb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44920/0 (socket says 172.21.15.46:44920) 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2017311913 learned_addr learned my addr 172.21.15.46:0/2017311913 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c69b19640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640b5210 0x7f2c640ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.262+0000 7f2c69318640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640d4800 0x7f2c640caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640d4800 msgr2=0x7f2c640caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6a31a640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640d4800 0x7f2c640caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640b5210 msgr2=0x7f2c640ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6a31a640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640b5210 0x7f2c640ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6a31a640 1 -- 172.21.15.46:0/2017311913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c640cf360 con 0x7f2c640cbf60 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c69318640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640d4800 0x7f2c640caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c69b19640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640b5210 0x7f2c640ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6a31a640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 0x7f2c640cb410 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c02abe0 tx=0x7f2c5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c03f070 con 0x7f2c640cbf60 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5c031bc0 con 0x7f2c640cbf60 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c009410 con 0x7f2c640cbf60 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c640cf640 con 0x7f2c640cbf60 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.263+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c640cfc00 con 0x7f2c640cbf60 2024-05-09T03:17:16.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.264+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c6406a270 con 0x7f2c640cbf60 2024-05-09T03:17:16.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.265+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2c5c03b410 con 0x7f2c640cbf60 2024-05-09T03:17:16.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.265+0000 7f2c5a7fc640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c48076300 0x7f2c48078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.266+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f2c5c0b9360 con 0x7f2c640cbf60 2024-05-09T03:17:16.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.266+0000 7f2c69b19640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c48076300 0x7f2c48078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.266+0000 7f2c69b19640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c48076300 0x7f2c48078810 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009af0 tx=0x7f2c60008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:16.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.269+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2c5c0822b0 con 0x7f2c640cbf60 2024-05-09T03:17:16.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.499+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c64162480 con 0x7f2c640cbf60 2024-05-09T03:17:16.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.499+0000 7f2c5a7fc640 1 -- 172.21.15.46:0/2017311913 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+4342 (secure 0 0 0) 0x7f2c5c0865b0 con 0x7f2c640cbf60 2024-05-09T03:17:16.501 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:16.501 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:16.129915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:16.501 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2024-05-09T03:17:16.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c48076300 msgr2=0x7f2c48078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c48076300 0x7f2c48078810 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2c60009af0 tx=0x7f2c60008040 comp rx=0 tx=0).stop 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 msgr2=0x7f2c640cb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 0x7f2c640cb410 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c02abe0 tx=0x7f2c5c004a40 comp rx=0 tx=0).stop 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 shutdown_connections 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c48076300 0x7f2c48078810 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c640cbf60 0x7f2c640cb410 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c640d4800 0x7f2c640caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 --2- 172.21.15.46:0/2017311913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c640b5210 0x7f2c640ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.501+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 >> 172.21.15.46:0/2017311913 conn(0x7f2c640d29a0 msgr2=0x7f2c64160240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.502+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 shutdown_connections 2024-05-09T03:17:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.502+0000 7f2c6bd7d640 1 -- 172.21.15.46:0/2017311913 wait complete. 2024-05-09T03:17:16.515 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:16.515 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.627+0000 7f4c04654640 1 Processor -- start 2024-05-09T03:17:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.628+0000 7f4c04654640 1 -- start start 2024-05-09T03:17:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c04654640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bfc1613c0 0x7f4bfc0b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c04654640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c04654640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc0b8110 con 0x7f4bfc0b5210 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c04654640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4bfc0b5620 con 0x7f4bfc1613c0 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c04654640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc1609a0 con 0x7f4bfc0b7d00 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c01bef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bfc1613c0 0x7f4bfc0b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c01bef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4bfc1613c0 0x7f4bfc0b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44930/0 (socket says 172.21.15.46:44930) 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c01bef640 1 -- 172.21.15.46:0/3434582211 learned_addr learned my addr 172.21.15.46:0/3434582211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.629+0000 7f4c02bf1640 1 --2- 172.21.15.46:0/3434582211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.630+0000 7f4c01bef640 1 -- 172.21.15.46:0/3434582211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 msgr2=0x7f4bfc163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.630+0000 7f4c01bef640 1 --2- 172.21.15.46:0/3434582211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.630+0000 7f4c01bef640 1 -- 172.21.15.46:0/3434582211 >> v1:172.21.15.88:6789/0 conn(0x7f4bfc0b5210 legacy=0x7f4bfc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.630+0000 7f4c01bef640 1 -- 172.21.15.46:0/3434582211 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc1617d0 con 0x7f4bfc1613c0 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.630+0000 7f4c01bef640 1 --2- 172.21.15.46:0/3434582211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0b77c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009870 tx=0x7f4bec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a303f78a2bc4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.630+0000 7f4c013ee640 1 -- 172.21.15.46:0/3434582211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec037070 con 0x7f4bfc1613c0 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c013ee640 1 -- 172.21.15.46:0/3434582211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bec032440 con 0x7f4bfc1613c0 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c013ee640 1 -- 172.21.15.46:0/3434582211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec032810 con 0x7f4bfc1613c0 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 -- 172.21.15.46:0/3434582211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc1613c0 msgr2=0x7f4bfc0b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 --2- 172.21.15.46:0/3434582211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0b77c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009870 tx=0x7f4bec02a270 comp rx=0 tx=0).stop 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 -- 172.21.15.46:0/3434582211 shutdown_connections 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 --2- 172.21.15.46:0/3434582211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 --2- 172.21.15.46:0/3434582211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0b77c0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 -- 172.21.15.46:0/3434582211 >> 172.21.15.46:0/3434582211 conn(0x7f4bfc0d29a0 msgr2=0x7f4bfc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:16.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 -- 172.21.15.46:0/3434582211 shutdown_connections 2024-05-09T03:17:16.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.631+0000 7f4c04654640 1 -- 172.21.15.46:0/3434582211 wait complete. 2024-05-09T03:17:16.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.633+0000 7f4c04654640 1 Processor -- start 2024-05-09T03:17:16.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.633+0000 7f4c04654640 1 -- start start 2024-05-09T03:17:16.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 0x7f4bfc0cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc160bc0 con 0x7f4bfc0b5210 2024-05-09T03:17:16.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc0d5560 con 0x7f4bfc1613c0 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc160ea0 con 0x7f4bfc0b7d00 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 0x7f4bfc0cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 0x7f4bfc0cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44934/0 (socket says 172.21.15.46:44934) 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 -- 172.21.15.46:0/1844145870 learned_addr learned my addr 172.21.15.46:0/1844145870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c02bf1640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c01bef640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 -- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 msgr2=0x7f4bfc152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 -- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc1613c0 msgr2=0x7f4bfc0cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 -- 172.21.15.46:0/1844145870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc0ca3b0 con 0x7f4bfc0b5210 2024-05-09T03:17:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c023f0640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 0x7f4bfc0cf890 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800c3d0 tx=0x7f4bf800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8017070 con 0x7f4bfc0b5210 2024-05-09T03:17:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf80052d0 con 0x7f4bfc0b5210 2024-05-09T03:17:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc0cf610 con 0x7f4bfc0b5210 2024-05-09T03:17:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.635+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8003830 con 0x7f4bfc0b5210 2024-05-09T03:17:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.636+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc0cdd90 con 0x7f4bfc0b5210 2024-05-09T03:17:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.636+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bfc0cd8a0 con 0x7f4bfc0b5210 2024-05-09T03:17:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.637+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4bf801a070 con 0x7f4bfc0b5210 2024-05-09T03:17:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.638+0000 7f4bf2ffd640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4be00762d0 0x7f4be00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.638+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f4bf8097120 con 0x7f4bfc0b5210 2024-05-09T03:17:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.638+0000 7f4c01bef640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4be00762d0 0x7f4be00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.639+0000 7f4c01bef640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4be00762d0 0x7f4be00787e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4bec034000 tx=0x7f4bec02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.642+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4bf8021020 con 0x7f4bfc0b5210 2024-05-09T03:17:16.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.872+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bfc06a270 con 0x7f4bfc0b5210 2024-05-09T03:17:16.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.872+0000 7f4bf2ffd640 1 -- 172.21.15.46:0/1844145870 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+4342 (secure 0 0 0) 0x7f4bfc0b8110 con 0x7f4bfc0b5210 2024-05-09T03:17:16.875 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:16.875 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":144,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:16.129915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:16.875 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 144 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4be00762d0 msgr2=0x7f4be00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4be00762d0 0x7f4be00787e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4bec034000 tx=0x7f4bec02e000 comp rx=0 tx=0).stop 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 msgr2=0x7f4bfc0cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 0x7f4bfc0cf890 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4bf800c3d0 tx=0x7f4bf800c890 comp rx=0 tx=0).stop 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 shutdown_connections 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc1613c0 0x7f4bfc0cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4bfc0b7d00 0x7f4bfc152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4be00762d0 0x7f4be00787e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 --2- 172.21.15.46:0/1844145870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bfc0b5210 0x7f4bfc0cf890 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:16.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 >> 172.21.15.46:0/1844145870 conn(0x7f4bfc0d29a0 msgr2=0x7f4bfc161a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:16.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 shutdown_connections 2024-05-09T03:17:16.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:16.875+0000 7f4c04654640 1 -- 172.21.15.46:0/1844145870 wait complete. 2024-05-09T03:17:16.888 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:16.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:17.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.001+0000 7f80a56a8640 1 Processor -- start 2024-05-09T03:17:17.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.001+0000 7f80a56a8640 1 -- start start 2024-05-09T03:17:17.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f80a56a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00d57a0 0x7f80a0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f80a56a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f80a00b7ef0 0x7f80a00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f80a56a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80a00d49e0 con 0x7f80a00b4af0 2024-05-09T03:17:17.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f80a56a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f80a00d5bb0 con 0x7f80a00b7ef0 2024-05-09T03:17:17.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f80a56a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f80a00d4cc0 con 0x7f80a00d57a0 2024-05-09T03:17:17.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f809f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f80a00b7ef0 0x7f80a00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f809e7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f80a00b4af0 0x7f80a00b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38098/0 (socket says 172.21.15.46:38098) 2024-05-09T03:17:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f809f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f80a00b7ef0 0x7f80a00b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44938/0 (socket says 172.21.15.46:44938) 2024-05-09T03:17:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.002+0000 7f809e7fc640 1 -- 172.21.15.46:0/2119435276 learned_addr learned my addr 172.21.15.46:0/2119435276 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809effd640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00d57a0 0x7f80a0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809f7fe640 1 -- 172.21.15.46:0/2119435276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00d57a0 msgr2=0x7f80a0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:17.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809f7fe640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00d57a0 0x7f80a0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809f7fe640 1 -- 172.21.15.46:0/2119435276 >> v1:172.21.15.46:6789/0 conn(0x7f80a00b4af0 legacy=0x7f80a00b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809dffb640 1 -- 172.21.15.46:0/2119435276 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378937951 0 0) 0x7f80a00d49e0 con 0x7f80a00b4af0 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809f7fe640 1 -- 172.21.15.46:0/2119435276 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a00d4fa0 con 0x7f80a00b7ef0 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.003+0000 7f809effd640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00d57a0 0x7f80a0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.004+0000 7f809f7fe640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a00b7ef0 0x7f80a00b4f00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f8098009d90 tx=0x7f809802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b418e697cd3535dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.004+0000 7f809dffb640 1 -- 172.21.15.46:0/2119435276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098007740 con 0x7f80a00b7ef0 2024-05-09T03:17:17.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.004+0000 7f809dffb640 1 -- 172.21.15.46:0/2119435276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80980078e0 con 0x7f80a00b7ef0 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.004+0000 7f809dffb640 1 -- 172.21.15.46:0/2119435276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098007cb0 con 0x7f80a00b7ef0 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.004+0000 7f80a56a8640 1 -- 172.21.15.46:0/2119435276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a00b7ef0 msgr2=0x7f80a00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.004+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a00b7ef0 0x7f80a00b4f00 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f8098009d90 tx=0x7f809802a660 comp rx=0 tx=0).stop 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.005+0000 7f80a56a8640 1 -- 172.21.15.46:0/2119435276 shutdown_connections 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.005+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a00b7ef0 0x7f80a00b4f00 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.005+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2119435276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00d57a0 0x7f80a0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.005+0000 7f80a56a8640 1 -- 172.21.15.46:0/2119435276 >> 172.21.15.46:0/2119435276 conn(0x7f80a00743e0 msgr2=0x7f80a00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:17.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.005+0000 7f80a56a8640 1 -- 172.21.15.46:0/2119435276 shutdown_connections 2024-05-09T03:17:17.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.005+0000 7f80a56a8640 1 -- 172.21.15.46:0/2119435276 wait complete. 2024-05-09T03:17:17.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.006+0000 7f80a56a8640 1 Processor -- start 2024-05-09T03:17:17.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.007+0000 7f80a56a8640 1 -- start start 2024-05-09T03:17:17.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f80a56a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00b4af0 0x7f80a0162730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f80a56a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80a00d57a0 0x7f80a01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f80a56a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 0x7f80a0158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f80a56a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80a00d1920 con 0x7f80a0153d70 2024-05-09T03:17:17.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f80a56a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f80a00d15b0 con 0x7f80a00d57a0 2024-05-09T03:17:17.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f80a56a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f80a0154f10 con 0x7f80a00b4af0 2024-05-09T03:17:17.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 0x7f80a0158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 0x7f80a0158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44940/0 (socket says 172.21.15.46:44940) 2024-05-09T03:17:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 -- 172.21.15.46:0/2227898226 learned_addr learned my addr 172.21.15.46:0/2227898226 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809effd640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00b4af0 0x7f80a0162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809e7fc640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80a00d57a0 0x7f80a01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 -- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00b4af0 msgr2=0x7f80a0162730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00b4af0 0x7f80a0162730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 -- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80a00d57a0 msgr2=0x7f80a01541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80a00d57a0 0x7f80a01541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 -- 172.21.15.46:0/2227898226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a0161690 con 0x7f80a0153d70 2024-05-09T03:17:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.008+0000 7f809f7fe640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 0x7f80a0158bd0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f8098009eb0 tx=0x7f8098004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.009+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809803f070 con 0x7f80a0153d70 2024-05-09T03:17:17.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.009+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8098031c40 con 0x7f80a0153d70 2024-05-09T03:17:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.009+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098009450 con 0x7f80a0153d70 2024-05-09T03:17:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.009+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a0161970 con 0x7f80a0153d70 2024-05-09T03:17:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.009+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a0161f30 con 0x7f80a0153d70 2024-05-09T03:17:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.010+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a006a270 con 0x7f80a0153d70 2024-05-09T03:17:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.011+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8098003990 con 0x7f80a0153d70 2024-05-09T03:17:17.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.011+0000 7f80877fe640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f808c0762d0 0x7f808c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.011+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f80980b87b0 con 0x7f80a0153d70 2024-05-09T03:17:17.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.011+0000 7f809effd640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f808c0762d0 0x7f808c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.012+0000 7f809effd640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f808c0762d0 0x7f808c0787e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8090006fd0 tx=0x7f8090008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:17.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.016+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8098081690 con 0x7f80a0153d70 2024-05-09T03:17:17.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.256+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80a01598e0 con 0x7f80a0153d70 2024-05-09T03:17:17.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.257+0000 7f80877fe640 1 -- 172.21.15.46:0/2227898226 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) ==== 95+0+5168 (secure 0 0 0) 0x7f8098085990 con 0x7f80a0153d70 2024-05-09T03:17:17.259 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:17.259 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":145,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8139,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2032086476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2032086476},{"type":"v1","addr":"172.21.15.46:6834","nonce":2032086476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:16.129915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:17.259 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 145 2024-05-09T03:17:17.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f808c0762d0 msgr2=0x7f808c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f808c0762d0 0x7f808c0787e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8090006fd0 tx=0x7f8090008040 comp rx=0 tx=0).stop 2024-05-09T03:17:17.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 msgr2=0x7f80a0158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 0x7f80a0158bd0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f8098009eb0 tx=0x7f8098004a40 comp rx=0 tx=0).stop 2024-05-09T03:17:17.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 shutdown_connections 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f808c0762d0 0x7f808c0787e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80a0153d70 0x7f80a0158bd0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80a00d57a0 0x7f80a01541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 --2- 172.21.15.46:0/2227898226 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80a00b4af0 0x7f80a0162730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 >> 172.21.15.46:0/2227898226 conn(0x7f80a00743e0 msgr2=0x7f80a00b7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 shutdown_connections 2024-05-09T03:17:17.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.259+0000 7f80a56a8640 1 -- 172.21.15.46:0/2227898226 wait complete. 2024-05-09T03:17:17.272 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_inotable' led to DAMAGED state 2024-05-09T03:17:17.272 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 100.00000000' 2024-05-09T03:17:17.272 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:17.273 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:17.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:17.289 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:17.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:17.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:17.289 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:17.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:17.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:17.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.499+0000 7f4d40914640 1 Processor -- start 2024-05-09T03:17:17.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.499+0000 7f4d40914640 1 -- start start 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.500+0000 7f4d40914640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0b7e70 0x7f4d3c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.500+0000 7f4d40914640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c0b5550 con 0x7f4d3c0d3df0 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.500+0000 7f4d40914640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c0d44e0 con 0x7f4d3c0b5140 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.500+0000 7f4d40914640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c0d3480 con 0x7f4d3c0b7e70 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.500+0000 7f4d3ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4d3c0d3df0 0x7f4d3c0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38102/0 (socket says 172.21.15.46:38102) 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.500+0000 7f4d3ad9d640 1 -- 172.21.15.46:0/1233130393 learned_addr learned my addr 172.21.15.46:0/1233130393 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d31d9b640 1 --2- 172.21.15.46:0/1233130393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0b7e70 0x7f4d3c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606137133 0 0) 0x7f4d3c0b5550 con 0x7f4d3c0d3df0 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d28003520 con 0x7f4d3c0d3df0 2024-05-09T03:17:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775774282 0 0) 0x7f4d3c0d44e0 con 0x7f4d3c0b5140 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d3c0b5550 con 0x7f4d3c0b5140 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4128986448 0 0) 0x7f4d28003520 con 0x7f4d3c0d3df0 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1233130393 >> v1:172.21.15.88:6789/0 conn(0x7f4d3c0b5140 legacy=0x7f4d3c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1233130393 >> v1:172.21.15.46:6789/0 conn(0x7f4d3c0d3df0 legacy=0x7f4d3c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.501+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1233130393 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c0d47c0 con 0x7f4d3c0b7e70 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.502+0000 7f4d31d9b640 1 --2- 172.21.15.46:0/1233130393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0b7e70 0x7f4d3c0d38b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d240097d0 tx=0x7f4d2402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d820e46ace68361d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.502+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24037070 con 0x7f4d3c0b7e70 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.502+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2402acc0 con 0x7f4d3c0b7e70 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.502+0000 7f4d39d9b640 1 -- 172.21.15.46:0/1233130393 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34001a70 con 0x7f4d3c0b7e70 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 -- 172.21.15.46:0/1233130393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0b7e70 msgr2=0x7f4d3c0d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 --2- 172.21.15.46:0/1233130393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0b7e70 0x7f4d3c0d38b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4d240097d0 tx=0x7f4d2402a230 comp rx=0 tx=0).stop 2024-05-09T03:17:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 -- 172.21.15.46:0/1233130393 shutdown_connections 2024-05-09T03:17:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 --2- 172.21.15.46:0/1233130393 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0b7e70 0x7f4d3c0d38b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 -- 172.21.15.46:0/1233130393 >> 172.21.15.46:0/1233130393 conn(0x7f4d3c08d3f0 msgr2=0x7f4d3c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 -- 172.21.15.46:0/1233130393 shutdown_connections 2024-05-09T03:17:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.503+0000 7f4d40914640 1 -- 172.21.15.46:0/1233130393 wait complete. 2024-05-09T03:17:17.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.505+0000 7f4d40914640 1 Processor -- start 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.505+0000 7f4d40914640 1 -- start start 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d40914640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 0x7f4d3c163320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d40914640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0d3df0 0x7f4d3c160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d40914640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3c1611e0 0x7f4d3c15a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d40914640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c0d4200 con 0x7f4d3c0b5140 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d40914640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c0d4d80 con 0x7f4d3c1611e0 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d40914640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c0d5580 con 0x7f4d3c0d3df0 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d3a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 0x7f4d3c163320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d3a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 0x7f4d3c163320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44956/0 (socket says 172.21.15.46:44956) 2024-05-09T03:17:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.506+0000 7f4d3a59c640 1 -- 172.21.15.46:0/835051616 learned_addr learned my addr 172.21.15.46:0/835051616 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3ad9d640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3c1611e0 0x7f4d3c15a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d31d9b640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0d3df0 0x7f4d3c160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3a59c640 1 -- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0d3df0 msgr2=0x7f4d3c160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3a59c640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0d3df0 0x7f4d3c160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3a59c640 1 -- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3c1611e0 msgr2=0x7f4d3c15a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3a59c640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3c1611e0 0x7f4d3c15a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3a59c640 1 -- 172.21.15.46:0/835051616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c161a10 con 0x7f4d3c0b5140 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3ad9d640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3c1611e0 0x7f4d3c15a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d3a59c640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 0x7f4d3c163320 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00c460 tx=0x7f4d2c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c0052d0 con 0x7f4d3c0b5140 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c005490 con 0x7f4d3c0b5140 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c161cf0 con 0x7f4d3c0b5140 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c012870 con 0x7f4d3c0b5140 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.507+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c1622b0 con 0x7f4d3c0b5140 2024-05-09T03:17:17.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.508+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3c06a270 con 0x7f4d3c0b5140 2024-05-09T03:17:17.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.509+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4d2c012a10 con 0x7f4d3c0b5140 2024-05-09T03:17:17.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.509+0000 7f4d337fe640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d1c076300 0x7f4d1c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:17.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.510+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f4d2c097540 con 0x7f4d3c0b5140 2024-05-09T03:17:17.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.510+0000 7f4d31d9b640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d1c076300 0x7f4d1c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:17.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.510+0000 7f4d31d9b640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d1c076300 0x7f4d1c078810 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4d24034000 tx=0x7f4d2402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:17.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.514+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4d2c060b00 con 0x7f4d3c0b5140 2024-05-09T03:17:17.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:17.745+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4d3c15ec30 con 0x7f4d3c0b5140 2024-05-09T03:17:18.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.138+0000 7f4d337fe640 1 -- 172.21.15.46:0/835051616 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v146) ==== 161+0+0 (secure 0 0 0) 0x7f4d2c060400 con 0x7f4d3c0b5140 2024-05-09T03:17:18.140 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:18.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d1c076300 msgr2=0x7f4d1c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d1c076300 0x7f4d1c078810 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4d24034000 tx=0x7f4d2402e000 comp rx=0 tx=0).stop 2024-05-09T03:17:18.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 msgr2=0x7f4d3c163320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 0x7f4d3c163320 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00c460 tx=0x7f4d2c00c920 comp rx=0 tx=0).stop 2024-05-09T03:17:18.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 shutdown_connections 2024-05-09T03:17:18.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d3c1611e0 0x7f4d3c15a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d3c0d3df0 0x7f4d3c160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d1c076300 0x7f4d1c078810 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 --2- 172.21.15.46:0/835051616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3c0b5140 0x7f4d3c163320 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 >> 172.21.15.46:0/835051616 conn(0x7f4d3c08d3f0 msgr2=0x7f4d3c0bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:18.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.140+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 shutdown_connections 2024-05-09T03:17:18.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.141+0000 7f4d40914640 1 -- 172.21.15.46:0/835051616 wait complete. 2024-05-09T03:17:18.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:18.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.269+0000 7f577a30d640 1 Processor -- start 2024-05-09T03:17:18.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.270+0000 7f577a30d640 1 -- start start 2024-05-09T03:17:18.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f577a30d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57740cbc20 0x7f577406c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f577a30d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5774068c70 con 0x7f5774164620 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f577a30d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57740b7a40 con 0x7f57740cc350 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f577a30d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f57740b77f0 con 0x7f57740cbc20 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f5773fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57740cbc20 0x7f577406c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f5773fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f57740cbc20 0x7f577406c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44960/0 (socket says 172.21.15.46:44960) 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.271+0000 7f5773fff640 1 -- 172.21.15.46:0/3939166414 learned_addr learned my addr 172.21.15.46:0/3939166414 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5773fff640 1 -- 172.21.15.46:0/3939166414 >> v1:172.21.15.88:6789/0 conn(0x7f57740cc350 legacy=0x7f5774163f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5773fff640 1 -- 172.21.15.46:0/3939166414 >> v1:172.21.15.46:6789/0 conn(0x7f5774164620 legacy=0x7f57740cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5772ffd640 1 -- 172.21.15.46:0/3939166414 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048865979 0 0) 0x7f5774068c70 con 0x7f5774164620 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5773fff640 1 -- 172.21.15.46:0/3939166414 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5774163ba0 con 0x7f57740cbc20 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5773fff640 1 --2- 172.21.15.46:0/3939166414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57740cbc20 0x7f577406c680 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f5764009870 tx=0x7f576402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d3fa53b461bb76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:18.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5772ffd640 1 -- 172.21.15.46:0/3939166414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764037070 con 0x7f57740cbc20 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.272+0000 7f5772ffd640 1 -- 172.21.15.46:0/3939166414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5764032440 con 0x7f57740cbc20 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f5772ffd640 1 -- 172.21.15.46:0/3939166414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764032810 con 0x7f57740cbc20 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 -- 172.21.15.46:0/3939166414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57740cbc20 msgr2=0x7f577406c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 --2- 172.21.15.46:0/3939166414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57740cbc20 0x7f577406c680 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f5764009870 tx=0x7f576402a270 comp rx=0 tx=0).stop 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 -- 172.21.15.46:0/3939166414 shutdown_connections 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 --2- 172.21.15.46:0/3939166414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57740cbc20 0x7f577406c680 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 -- 172.21.15.46:0/3939166414 >> 172.21.15.46:0/3939166414 conn(0x7f577409b7d0 msgr2=0x7f57740b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 -- 172.21.15.46:0/3939166414 shutdown_connections 2024-05-09T03:17:18.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.273+0000 7f577a30d640 1 -- 172.21.15.46:0/3939166414 wait complete. 2024-05-09T03:17:18.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.275+0000 7f577a30d640 1 Processor -- start 2024-05-09T03:17:18.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.275+0000 7f577a30d640 1 -- start start 2024-05-09T03:17:18.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.276+0000 7f577a30d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57740cbc20 0x7f57741f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:18.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740cc350 0x7f5774158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 0x7f57741f9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57740cd930 con 0x7f5774164620 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5774164c80 con 0x7f57740cc350 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f57740cdab0 con 0x7f57740cbc20 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 0x7f57741f9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 0x7f57741f9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44974/0 (socket says 172.21.15.46:44974) 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 -- 172.21.15.46:0/2402315027 learned_addr learned my addr 172.21.15.46:0/2402315027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f5773fff640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57740cbc20 0x7f57741f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57737fe640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740cc350 0x7f5774158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 -- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57740cbc20 msgr2=0x7f57741f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57740cbc20 0x7f57741f69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 -- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740cc350 msgr2=0x7f5774158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740cc350 0x7f5774158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 -- 172.21.15.46:0/2402315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5774158570 con 0x7f5774164620 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57737fe640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740cc350 0x7f5774158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f57788aa640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 0x7f57741f9fd0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c730 tx=0x7f576c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c003b70 con 0x7f5774164620 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576c003d30 con 0x7f5774164620 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0052b0 con 0x7f5774164620 2024-05-09T03:17:18.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57741fa510 con 0x7f5774164620 2024-05-09T03:17:18.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.277+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57741fad00 con 0x7f5774164620 2024-05-09T03:17:18.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.278+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f577406a270 con 0x7f5774164620 2024-05-09T03:17:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.279+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f576c005660 con 0x7f5774164620 2024-05-09T03:17:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.280+0000 7f5770ff9640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57600762d0 0x7f57600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.280+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f576c097a20 con 0x7f5774164620 2024-05-09T03:17:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.280+0000 7f5773fff640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57600762d0 0x7f57600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.281+0000 7f5773fff640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57600762d0 0x7f57600787e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5764034000 tx=0x7f576402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:18.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.284+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5774158570 con 0x7f5774164620 2024-05-09T03:17:18.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:18.549+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f57740743f0 con 0x7f5774164620 2024-05-09T03:17:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.149+0000 7f5770ff9640 1 -- 172.21.15.46:0/2402315027 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v147) ==== 101+0+0 (secure 0 0 0) 0x7f576c065e20 con 0x7f5774164620 2024-05-09T03:17:19.151 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:17:19.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.152+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57600762d0 msgr2=0x7f57600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.152+0000 7f577a30d640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57600762d0 0x7f57600787e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5764034000 tx=0x7f576402e000 comp rx=0 tx=0).stop 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.152+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 msgr2=0x7f57741f9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.152+0000 7f577a30d640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 0x7f57741f9fd0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c730 tx=0x7f576c00cbf0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 shutdown_connections 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57600762d0 0x7f57600787e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5774164620 0x7f57741f9fd0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740cc350 0x7f5774158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 --2- 172.21.15.46:0/2402315027 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f57740cbc20 0x7f57741f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 >> 172.21.15.46:0/2402315027 conn(0x7f577409b7d0 msgr2=0x7f577415e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 shutdown_connections 2024-05-09T03:17:19.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.153+0000 7f577a30d640 1 -- 172.21.15.46:0/2402315027 wait complete. 2024-05-09T03:17:19.168 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.228+0000 7f01f58b1f40 1 Processor -- start 2024-05-09T03:17:19.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.228+0000 7f01f58b1f40 1 -- start start 2024-05-09T03:17:19.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01f58b1f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21cd37e0 0x55fb21dad690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01f58b1f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21d90a50 0x55fb21d8ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01f58b1f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d93610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01f58b1f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fb21d15cb0 con 0x55fb21dab780 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01f58b1f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fb21c7cf50 con 0x55fb21d90a50 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01f58b1f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fb21cd3bf0 con 0x55fb21cd37e0 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01efb7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d93610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01efb7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d93610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44984/0 (socket says 172.21.15.46:44984) 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.229+0000 7f01efb7c640 1 -- 172.21.15.46:0/4239919376 learned_addr learned my addr 172.21.15.46:0/4239919376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21d90a50 0x55fb21d8ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01ef37b640 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21cd37e0 0x55fb21dad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01efb7c640 1 -- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21cd37e0 msgr2=0x55fb21dad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01efb7c640 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21cd37e0 0x55fb21dad690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01efb7c640 1 -- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21d90a50 msgr2=0x55fb21d8ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:19.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01efb7c640 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21d90a50 0x55fb21d8ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01efb7c640 1 -- 172.21.15.46:0/4239919376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb21da9580 con 0x55fb21dab780 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01efb7c640 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d93610 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f01e800c440 tx=0x7f01e800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e3747fb85d851b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01edb78640 1 -- 172.21.15.46:0/4239919376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e80116b0 con 0x55fb21dab780 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01edb78640 1 -- 172.21.15.46:0/4239919376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e8017070 con 0x55fb21dab780 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.230+0000 7f01edb78640 1 -- 172.21.15.46:0/4239919376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8011ab0 con 0x55fb21dab780 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 -- 172.21.15.46:0/4239919376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 msgr2=0x55fb21d93610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d93610 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f01e800c440 tx=0x7f01e800c900 comp rx=0 tx=0).stop 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 -- 172.21.15.46:0/4239919376 shutdown_connections 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d93610 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21d90a50 0x55fb21d8ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/4239919376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21cd37e0 0x55fb21dad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 -- 172.21.15.46:0/4239919376 >> 172.21.15.46:0/4239919376 conn(0x55fb21bf8ac0 msgr2=0x55fb21d04840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 -- 172.21.15.46:0/4239919376 shutdown_connections 2024-05-09T03:17:19.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.231+0000 7f01f58b1f40 1 -- 172.21.15.46:0/4239919376 wait complete. 2024-05-09T03:17:19.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.232+0000 7f01f58b1f40 1 Processor -- start 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.232+0000 7f01f58b1f40 1 -- start start 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01f58b1f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21cd37e0 0x55fb21d837b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01f58b1f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21d90a50 0x55fb21da9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01f58b1f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d00ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01f58b1f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fb21da92a0 con 0x55fb21dab780 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01f58b1f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fb21d050f0 con 0x55fb21cd37e0 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01f58b1f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fb21da8d20 con 0x55fb21d90a50 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01efb7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d00ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01efb7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d00ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44988/0 (socket says 172.21.15.46:44988) 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 learned_addr learned my addr 172.21.15.46:0/201346087 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.233+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21cd37e0 0x55fb21d837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21d90a50 0x55fb21da9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21d90a50 msgr2=0x55fb21da9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01efb7c640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fb21d90a50 0x55fb21da9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21cd37e0 msgr2=0x55fb21d837b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01efb7c640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fb21cd37e0 0x55fb21d837b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb21d7e660 con 0x55fb21dab780 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01efb7c640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d00ac0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f01e80126b0 tx=0x7f01e80033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01ecb76640 1 -- 172.21.15.46:0/201346087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8012c20 con 0x55fb21dab780 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01ecb76640 1 -- 172.21.15.46:0/201346087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01e8012dc0 con 0x55fb21dab780 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01ecb76640 1 -- 172.21.15.46:0/201346087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e8026650 con 0x55fb21dab780 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fb21d7e940 con 0x55fb21dab780 2024-05-09T03:17:19.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.234+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb21d7ef00 con 0x55fb21dab780 2024-05-09T03:17:19.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.236+0000 7f01ecb76640 1 -- 172.21.15.46:0/201346087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f01e8024020 con 0x55fb21dab780 2024-05-09T03:17:19.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.236+0000 7f01ecb76640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01dc074ce0 0x7f01dc0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.236+0000 7f01ecb76640 1 -- 172.21.15.46:0/201346087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f01e800e020 con 0x55fb21dab780 2024-05-09T03:17:19.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.237+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01dc074ce0 0x7f01dc0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.239 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:19.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.237+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fb21d81900 0x55fb21d7f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.237+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21d81d10 con 0x55fb21d81900 2024-05-09T03:17:19.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.237+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fb21d81900 0x55fb21d7f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.237+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01dc074ce0 0x7f01dc0771f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f01e0000c00 tx=0x7f01e00015a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.237+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fb21d81900 0x55fb21d7f700 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.239+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v94'288 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.239+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21d7fc40 con 0x55fb21d81900 2024-05-09T03:17:19.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.241+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v94'289 uv289 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.241+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21d7ffe0 con 0x55fb21d81900 2024-05-09T03:17:19.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.242+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v94'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.244 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:19.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.242+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e3fff0 con 0x55fb21d81900 2024-05-09T03:17:19.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.243+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v94'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.243+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e403f0 con 0x55fb21d81900 2024-05-09T03:17:19.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.244+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v94'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.244+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e407f0 con 0x55fb21d81900 2024-05-09T03:17:19.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.246+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v94'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.246+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e40b90 con 0x55fb21d81900 2024-05-09T03:17:19.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.247+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v94'294 uv268 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.247+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.248+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v94'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.248+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.250+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v94'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.251 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:19.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.250+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4f4d0 con 0x55fb21d81900 2024-05-09T03:17:19.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.251+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v94'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.251+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4f4d0 con 0x55fb21d81900 2024-05-09T03:17:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.252+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v94'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.252+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.254+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v94'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.254+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.255+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v94'300 uv275 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.255+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.256+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v94'301 uv301 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.256+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.258+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v94'302 uv302 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.259 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:19.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.258+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4f4d0 con 0x55fb21d81900 2024-05-09T03:17:19.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.259+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v94'303 uv303 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.259+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4f4d0 con 0x55fb21d81900 2024-05-09T03:17:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.260+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v94'304 uv304 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.260+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4e4c0 con 0x55fb21d81900 2024-05-09T03:17:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.262+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v94'305 uv305 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.262+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.263+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v94'306 uv306 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.263+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.265+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v94'307 uv281 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.265+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.266+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v94'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.266+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.267+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v94'309 uv309 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.269 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:19.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.267+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4fbf0 con 0x55fb21d81900 2024-05-09T03:17:19.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.269+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v94'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.269+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4fbf0 con 0x55fb21d81900 2024-05-09T03:17:19.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.270+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v94'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.270+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.271+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v94'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.271+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.273+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v94'313 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.273+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.274+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v94'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.274+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.275+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v94'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.277 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:19.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.275+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4fbf0 con 0x55fb21d81900 2024-05-09T03:17:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.276+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v94'316 uv316 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.276+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4fbf0 con 0x55fb21d81900 2024-05-09T03:17:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.278+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v94'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.278+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e4ebe0 con 0x55fb21d81900 2024-05-09T03:17:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.279+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v94'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.279+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fb21e4f0f0 0x55fb21e51510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.279+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e51a50 con 0x55fb21e4f0f0 2024-05-09T03:17:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.279+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fb21e4f0f0 0x55fb21e51510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.280+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fb21e4f0f0 0x55fb21e51510 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.281+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v94'217 uv200 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.281+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e51df0 con 0x55fb21e4f0f0 2024-05-09T03:17:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.283+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v94'218 uv218 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.283+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e52190 con 0x55fb21e4f0f0 2024-05-09T03:17:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.284+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v94'219 uv219 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.285 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.284+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e529e0 con 0x55fb21e4f0f0 2024-05-09T03:17:19.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.285+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v94'220 uv220 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.285+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e52d80 con 0x55fb21e4f0f0 2024-05-09T03:17:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.286+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v94'221 uv221 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.287+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53120 con 0x55fb21e4f0f0 2024-05-09T03:17:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.288+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v94'222 uv205 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.288+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e534c0 con 0x55fb21e4f0f0 2024-05-09T03:17:19.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.289+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v94'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.289+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.290+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v94'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.292 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.290+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.292+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v94'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.292+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.293+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v94'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.293+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.294+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v94'227 uv210 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.294+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.296+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v94'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.296+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.297+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v94'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.299 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:19.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.298+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e572c0 con 0x55fb21e4f0f0 2024-05-09T03:17:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.299+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v94'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.299+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e572c0 con 0x55fb21e4f0f0 2024-05-09T03:17:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.300+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v94'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.300+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.302+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v94'232 uv216 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.302+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.303+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v94'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.303+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.304+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v94'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.306 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:19.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.304+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e54870 con 0x55fb21e4f0f0 2024-05-09T03:17:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.305+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v94'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.305+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e54870 con 0x55fb21e4f0f0 2024-05-09T03:17:19.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.306+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v94'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.307+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21e4f0f0 2024-05-09T03:17:19.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.308+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v94'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e4f0f0 2024-05-09T03:17:19.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.308+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21d81900 2024-05-09T03:17:19.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.309+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v94'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.309+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21d81900 2024-05-09T03:17:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.310+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v94'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.310+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21d81900 2024-05-09T03:17:19.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.311+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v94'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.313 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:19.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.311+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e54870 con 0x55fb21d81900 2024-05-09T03:17:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.312+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v94'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.312+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e54870 con 0x55fb21d81900 2024-05-09T03:17:19.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.313+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v94'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.313+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e53860 con 0x55fb21d81900 2024-05-09T03:17:19.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.314+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v94'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.315+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fb21e53ee0 0x55fb21e56300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.315+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e568f0 con 0x55fb21e53ee0 2024-05-09T03:17:19.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.315+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fb21e53ee0 0x55fb21e56300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.315+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fb21e53ee0 0x55fb21e56300 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.316+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v94'48 uv47 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.316+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e56ce0 con 0x55fb21e53ee0 2024-05-09T03:17:19.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.317+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v94'49 uv49 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.317+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56ce0 con 0x55fb21e53ee0 2024-05-09T03:17:19.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.318+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v94'50 uv50 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.320 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:19.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.318+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56ce0 con 0x55fb21e53ee0 2024-05-09T03:17:19.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.319+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v94'51 uv51 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.319+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56c10 con 0x55fb21d81900 2024-05-09T03:17:19.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.320+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v94'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.320+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e56c10 con 0x55fb21d81900 2024-05-09T03:17:19.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.321+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v94'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.322+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56c10 con 0x55fb21d81900 2024-05-09T03:17:19.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.323+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v94'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.326 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:19.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.323+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56c10 con 0x55fb21d81900 2024-05-09T03:17:19.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.324+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v94'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.324+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56c10 con 0x55fb21d81900 2024-05-09T03:17:19.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.325+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v94'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.325+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e56c10 con 0x55fb21d81900 2024-05-09T03:17:19.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.326+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v94'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.326+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fb21e57680 0x55fb21e57a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.326+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5a080 con 0x55fb21e57680 2024-05-09T03:17:19.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.326+0000 7f01efb7c640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fb21e57680 0x55fb21e57a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.327+0000 7f01efb7c640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fb21e57680 0x55fb21e57a90 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.328+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v94'105 uv99 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.328+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e5a420 con 0x55fb21e57680 2024-05-09T03:17:19.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.329+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v94'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.329+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5a800 con 0x55fb21e57680 2024-05-09T03:17:19.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.330+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v94'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.332 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:19.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.330+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5c0b0 con 0x55fb21e57680 2024-05-09T03:17:19.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.331+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v94'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.331+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5c450 con 0x55fb21e57680 2024-05-09T03:17:19.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.332+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v94'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.332+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5b0a0 con 0x55fb21e57680 2024-05-09T03:17:19.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.333+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v94'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.333+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e57680 2024-05-09T03:17:19.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.334+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v94'111 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.334+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e57680 2024-05-09T03:17:19.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.335+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v94'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.335+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e57680 2024-05-09T03:17:19.338 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:19.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.336+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v94'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.336+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e57680 2024-05-09T03:17:19.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.338+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v94'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.338+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e57680 2024-05-09T03:17:19.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.339+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v94'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.339+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fb21e5c7f0 0x55fb21e5ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.339+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e5c7f0 2024-05-09T03:17:19.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.339+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fb21e5c7f0 0x55fb21e5ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.339+0000 7f01ef37b640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fb21e5c7f0 0x55fb21e5ec10 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.340+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v94'46 uv45 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e5c7f0 2024-05-09T03:17:19.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.340+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e5b4b0 con 0x55fb21e5c7f0 2024-05-09T03:17:19.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.341+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v94'47 uv47 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e5c7f0 2024-05-09T03:17:19.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.341+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21e5c7f0 2024-05-09T03:17:19.344 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:19.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.342+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v94'48 uv48 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e5c7f0 2024-05-09T03:17:19.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.342+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21e5c7f0 2024-05-09T03:17:19.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.344+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v94'49 uv49 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e5c7f0 2024-05-09T03:17:19.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.344+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21e5c7f0 2024-05-09T03:17:19.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.345+0000 7f01ef37b640 1 -- 172.21.15.46:0/201346087 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v94'50 uv50 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e001a040 con 0x55fb21e5c7f0 2024-05-09T03:17:19.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.345+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21d81900 2024-05-09T03:17:19.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.346+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v94'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.346+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21d81900 2024-05-09T03:17:19.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.347+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v94'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.347+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21d81900 2024-05-09T03:17:19.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.348+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v94'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.350 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:19.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.348+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e601e0 con 0x55fb21d81900 2024-05-09T03:17:19.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.349+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v94'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.349+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e601e0 con 0x55fb21d81900 2024-05-09T03:17:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.350+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v94'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.350+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21d81900 2024-05-09T03:17:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.351+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v94'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21d81900 2024-05-09T03:17:19.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.351+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5f1d0 con 0x55fb21e53ee0 2024-05-09T03:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.352+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v94'95 uv95 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.354 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.352+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5ac90 con 0x55fb21e53ee0 2024-05-09T03:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.353+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [write 0~34] v94'96 uv96 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.354+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5ac90 con 0x55fb21e53ee0 2024-05-09T03:17:19.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.355+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v94'97 uv97 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.355+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5ac90 con 0x55fb21e53ee0 2024-05-09T03:17:19.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.356+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 100.00000000 [create] v94'98 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.356+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e5ac90 con 0x55fb21e53ee0 2024-05-09T03:17:19.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.357+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 100.00000000 [delete] v94'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.357+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5ac90 con 0x55fb21e53ee0 2024-05-09T03:17:19.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.358+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v94'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.359 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.358+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e5ac90 con 0x55fb21e53ee0 2024-05-09T03:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.359+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [omap-set-header] v94'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.359+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e61fe0 con 0x55fb21e53ee0 2024-05-09T03:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.360+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v94'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.360+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fb21e5fd00 0x55fb21e62640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.360+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e5fd00 2024-05-09T03:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.360+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fb21e5fd00 0x55fb21e62640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.360+0000 7f01eeb7a640 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fb21e5fd00 0x55fb21e62640 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.361+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v94'44 uv43 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e5fd00 2024-05-09T03:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.361+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e5fd00 2024-05-09T03:17:19.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.363+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v94'45 uv45 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e5fd00 2024-05-09T03:17:19.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.363+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e5fd00 2024-05-09T03:17:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.364+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v94'46 uv46 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e5fd00 2024-05-09T03:17:19.365 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.364+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e5fd00 2024-05-09T03:17:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.365+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v94'47 uv47 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e5fd00 2024-05-09T03:17:19.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.365+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e53ee0 2024-05-09T03:17:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.366+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(103 609.00000000 [create] v94'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.366+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e53ee0 2024-05-09T03:17:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.367+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(104 609.00000000 [delete] v94'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.367+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e53ee0 2024-05-09T03:17:19.370 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.368+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v94'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.368+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e53ee0 2024-05-09T03:17:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.369+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v94'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.370+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e53ee0 2024-05-09T03:17:19.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.370+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v94'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.370+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e53ee0 2024-05-09T03:17:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.371+0000 7f01eeb7a640 1 -- 172.21.15.46:0/201346087 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [omap-set-header] v94'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e4054ae0 con 0x55fb21e53ee0 2024-05-09T03:17:19.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.371+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.373+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v94'47 uv46 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.373+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.374+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v94'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.374+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.377 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.375+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v94'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.375+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.377+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v94'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.377+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.378+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v94'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.378+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.379+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v94'119 uv111 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.379+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.381+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v94'120 uv120 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.381+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.384 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:19.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.382+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v94'121 uv121 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.382+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.383+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v94'122 uv122 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.383+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.385+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v94'123 uv123 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.385+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.386+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v94'124 uv118 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.386+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.387+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v94'125 uv125 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.387+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.390 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.388+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v94'126 uv126 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.388+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.389+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v94'127 uv127 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.389+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.391+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v94'128 uv128 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.391+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62120 con 0x55fb21e57680 2024-05-09T03:17:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.392+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v94'129 uv129 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.392+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94) -- 0x55fb21e62b80 con 0x55fb21e57680 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v94'130 uv130 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f01e805d7d0 con 0x55fb21e57680 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fb21d81900 msgr2=0x55fb21d7f700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fb21d81900 0x55fb21d7f700 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fb21e53ee0 msgr2=0x55fb21e56300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fb21e53ee0 0x55fb21e56300 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fb21e5fd00 msgr2=0x55fb21e62640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fb21e5fd00 0x55fb21e62640 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fb21e4f0f0 msgr2=0x55fb21e51510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fb21e4f0f0 0x55fb21e51510 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 reap_dead start 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fb21e57680 msgr2=0x55fb21e57a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fb21e57680 0x55fb21e57a90 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fb21e5c7f0 msgr2=0x55fb21e5ec10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fb21e5c7f0 0x55fb21e5ec10 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01dc074ce0 msgr2=0x7f01dc0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01dc074ce0 0x7f01dc0771f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f01e0000c00 tx=0x7f01e00015a0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 msgr2=0x55fb21d00ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 --2- 172.21.15.46:0/201346087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb21dab780 0x55fb21d00ac0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f01e80126b0 tx=0x7f01e80033a0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01efb7c640 1 -- 172.21.15.46:0/201346087 reap_dead start 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 shutdown_connections 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 >> 172.21.15.46:0/201346087 conn(0x55fb21bf8ac0 msgr2=0x55fb21d8eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 shutdown_connections 2024-05-09T03:17:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.393+0000 7f01f58b1f40 1 -- 172.21.15.46:0/201346087 wait complete. 2024-05-09T03:17:19.398 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100.00000000 2024-05-09T03:17:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.493+0000 7f8920b1df40 1 Processor -- start 2024-05-09T03:17:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.493+0000 7f8920b1df40 1 -- start start 2024-05-09T03:17:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.494+0000 7f8920b1df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 0x55f10367e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.494+0000 7f8920b1df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f10368ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.494+0000 7f8920b1df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e39c0 0x55f10368b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8920b1df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f10355bf50 con 0x55f1035e47b0 2024-05-09T03:17:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8920b1df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f10355ff90 con 0x55f1035fb010 2024-05-09T03:17:19.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8920b1df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f1035f94d0 con 0x55f1035e39c0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 0x55f10367e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 0x55f10367e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44998/0 (socket says 172.21.15.46:44998) 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 -- 172.21.15.46:0/2209681767 learned_addr learned my addr 172.21.15.46:0/2209681767 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919335640 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f10368ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f891a337640 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e39c0 0x55f10368b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 -- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e39c0 msgr2=0x55f10368b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e39c0 0x55f10368b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 -- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 msgr2=0x55f10368ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f10368ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 -- 172.21.15.46:0/2209681767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f10371a1e0 con 0x55f1035e47b0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8919b36640 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 0x55f10367e2a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8914025d60 tx=0x7f8914051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59fdce2163da1d17 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8913fff640 1 -- 172.21.15.46:0/2209681767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914056030 con 0x55f1035e47b0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8913fff640 1 -- 172.21.15.46:0/2209681767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8914052c60 con 0x55f1035e47b0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.495+0000 7f8913fff640 1 -- 172.21.15.46:0/2209681767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891405b670 con 0x55f1035e47b0 2024-05-09T03:17:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 -- 172.21.15.46:0/2209681767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 msgr2=0x55f10367e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 0x55f10367e2a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8914025d60 tx=0x7f8914051000 comp rx=0 tx=0).stop 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 -- 172.21.15.46:0/2209681767 shutdown_connections 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e39c0 0x55f10368b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f10368ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2209681767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e47b0 0x55f10367e2a0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 -- 172.21.15.46:0/2209681767 >> 172.21.15.46:0/2209681767 conn(0x55f1034d7ac0 msgr2=0x55f1035fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 -- 172.21.15.46:0/2209681767 shutdown_connections 2024-05-09T03:17:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.496+0000 7f8920b1df40 1 -- 172.21.15.46:0/2209681767 wait complete. 2024-05-09T03:17:19.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.497+0000 7f8920b1df40 1 Processor -- start 2024-05-09T03:17:19.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.497+0000 7f8920b1df40 1 -- start start 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8920b1df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e39c0 0x55f1036830e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8920b1df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e47b0 0x55f103679c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8920b1df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f103686350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8920b1df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f10368cec0 con 0x55f1035e39c0 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8920b1df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f1035e2420 con 0x55f1035fb010 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8920b1df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f103686f90 con 0x55f1035e47b0 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8919b36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e39c0 0x55f1036830e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8919b36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e39c0 0x55f1036830e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45000/0 (socket says 172.21.15.46:45000) 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8919b36640 1 -- 172.21.15.46:0/2374879119 learned_addr learned my addr 172.21.15.46:0/2374879119 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f891a337640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f103686350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.498+0000 7f8919335640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e47b0 0x55f103679c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8919b36640 1 -- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e47b0 msgr2=0x55f103679c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8919b36640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1035e47b0 0x55f103679c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8919b36640 1 -- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 msgr2=0x55f103686350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8919b36640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1035fb010 0x55f103686350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8919b36640 1 -- 172.21.15.46:0/2374879119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f10371c9f0 con 0x55f1035e39c0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8919b36640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e39c0 0x55f1036830e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8914025d30 tx=0x7f8914002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8912ffd640 1 -- 172.21.15.46:0/2374879119 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8914056050 con 0x55f1035e39c0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8912ffd640 1 -- 172.21.15.46:0/2374879119 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8914052a60 con 0x55f1035e39c0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f10367e340 con 0x55f1035e39c0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8912ffd640 1 -- 172.21.15.46:0/2374879119 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891406f480 con 0x55f1035e39c0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.499+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f103686cd0 con 0x55f1035e39c0 2024-05-09T03:17:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.501+0000 7f8912ffd640 1 -- 172.21.15.46:0/2374879119 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8914076030 con 0x55f1035e39c0 2024-05-09T03:17:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.501+0000 7f8912ffd640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88fc074ce0 0x7f88fc0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.501+0000 7f8912ffd640 1 -- 172.21.15.46:0/2374879119 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f89140dba70 con 0x55f1035e39c0 2024-05-09T03:17:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.501+0000 7f8919335640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88fc074ce0 0x7f88fc0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.502+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f103686890 0x55f103681da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.502+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e94) -- 0x55f1035fd3c0 con 0x55f103686890 2024-05-09T03:17:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.502+0000 7f891a337640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f103686890 0x55f103681da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.502+0000 7f8919335640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88fc074ce0 0x7f88fc0771f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8904003f10 tx=0x7f89040038b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.502+0000 7f891a337640 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f103686890 0x55f103681da0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f891a337640 1 -- 172.21.15.46:0/2374879119 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 100.00000000 [delete] v94'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f890c010610 con 0x55f103686890 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f103686890 msgr2=0x55f103681da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f103686890 0x55f103681da0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88fc074ce0 msgr2=0x7f88fc0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88fc074ce0 0x7f88fc0771f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f8904003f10 tx=0x7f89040038b0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e39c0 msgr2=0x55f1036830e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 --2- 172.21.15.46:0/2374879119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1035e39c0 0x55f1036830e0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f8914025d30 tx=0x7f8914002fb0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f891a337640 1 -- 172.21.15.46:0/2374879119 reap_dead start 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 shutdown_connections 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 >> 172.21.15.46:0/2374879119 conn(0x55f1034d7ac0 msgr2=0x55f10371cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 shutdown_connections 2024-05-09T03:17:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.504+0000 7f8920b1df40 1 -- 172.21.15.46:0/2374879119 wait complete. 2024-05-09T03:17:19.508 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.623+0000 7f89edabd640 1 Processor -- start 2024-05-09T03:17:19.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.623+0000 7f89edabd640 1 -- start start 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89edabd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f89e80d2eb0 0x7f89e80d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89edabd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e80d4a60 con 0x7f89e80b7760 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89edabd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89e80d4d40 con 0x7f89e80b95a0 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89edabd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f89e80d5020 con 0x7f89e80d2eb0 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89e77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f89e80d2eb0 0x7f89e80d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89e6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f89e80b7760 0x7f89e80b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38130/0 (socket says 172.21.15.46:38130) 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89e6ffd640 1 -- 172.21.15.46:0/676040668 learned_addr learned my addr 172.21.15.46:0/676040668 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89e77fe640 1 -- 172.21.15.46:0/676040668 >> v1:172.21.15.88:6789/0 conn(0x7f89e80b95a0 legacy=0x7f89e80b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89e77fe640 1 -- 172.21.15.46:0/676040668 >> v1:172.21.15.46:6789/0 conn(0x7f89e80b7760 legacy=0x7f89e80b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.625+0000 7f89e77fe640 1 -- 172.21.15.46:0/676040668 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e80d5300 con 0x7f89e80d2eb0 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.626+0000 7f89e77fe640 1 --2- 172.21.15.46:0/676040668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80d2eb0 0x7f89e80d32c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f89d80097a0 tx=0x7f89d802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f300e97835bff1bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.626+0000 7f89e67fc640 1 -- 172.21.15.46:0/676040668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8037070 con 0x7f89e80d2eb0 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.626+0000 7f89e67fc640 1 -- 172.21.15.46:0/676040668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d8032440 con 0x7f89e80d2eb0 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.626+0000 7f89e67fc640 1 -- 172.21.15.46:0/676040668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8032810 con 0x7f89e80d2eb0 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 -- 172.21.15.46:0/676040668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80d2eb0 msgr2=0x7f89e80d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 --2- 172.21.15.46:0/676040668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80d2eb0 0x7f89e80d32c0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f89d80097a0 tx=0x7f89d802a230 comp rx=0 tx=0).stop 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 -- 172.21.15.46:0/676040668 shutdown_connections 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 --2- 172.21.15.46:0/676040668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80d2eb0 0x7f89e80d32c0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 -- 172.21.15.46:0/676040668 >> 172.21.15.46:0/676040668 conn(0x7f89e808d390 msgr2=0x7f89e808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 -- 172.21.15.46:0/676040668 shutdown_connections 2024-05-09T03:17:19.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.627+0000 7f89edabd640 1 -- 172.21.15.46:0/676040668 wait complete. 2024-05-09T03:17:19.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.629+0000 7f89edabd640 1 Processor -- start 2024-05-09T03:17:19.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.629+0000 7f89edabd640 1 -- start start 2024-05-09T03:17:19.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89edabd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 0x7f89e81521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89edabd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e80b95a0 0x7f89e81e9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89edabd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89e80d2eb0 0x7f89e80cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89edabd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f89e80b5360 con 0x7f89e80b7760 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89edabd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f89e80b51e0 con 0x7f89e80b95a0 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89edabd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f89e8163bd0 con 0x7f89e80d2eb0 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 0x7f89e81521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89e77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 0x7f89e81521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45012/0 (socket says 172.21.15.46:45012) 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.630+0000 7f89e77fe640 1 -- 172.21.15.46:0/392154092 learned_addr learned my addr 172.21.15.46:0/392154092 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e6ffd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e80b95a0 0x7f89e81e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e7fff640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89e80d2eb0 0x7f89e80cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e77fe640 1 -- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89e80d2eb0 msgr2=0x7f89e80cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e77fe640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89e80d2eb0 0x7f89e80cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e77fe640 1 -- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e80b95a0 msgr2=0x7f89e81e9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e77fe640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e80b95a0 0x7f89e81e9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e77fe640 1 -- 172.21.15.46:0/392154092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e80ce790 con 0x7f89e80b7760 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89e77fe640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 0x7f89e81521b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f89d80098c0 tx=0x7f89d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8037040 con 0x7f89e80b7760 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d8030770 con 0x7f89e80b7760 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e80cecc0 con 0x7f89e80b7760 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8030b40 con 0x7f89e80b7760 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.631+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e80cf280 con 0x7f89e80b7760 2024-05-09T03:17:19.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.632+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e80b8900 con 0x7f89e80b7760 2024-05-09T03:17:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.634+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f89d8044020 con 0x7f89e80b7760 2024-05-09T03:17:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.634+0000 7f89cffff640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89d40762d0 0x7f89d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.634+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f89d80b8a40 con 0x7f89e80b7760 2024-05-09T03:17:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.634+0000 7f89e6ffd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89d40762d0 0x7f89d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:19.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.635+0000 7f89e6ffd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89d40762d0 0x7f89d40787e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f89e8162360 tx=0x7f89dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:19.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.638+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f89e80b8900 con 0x7f89e80b7760 2024-05-09T03:17:19.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.867+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f89e806a270 con 0x7f89e80b7760 2024-05-09T03:17:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.978+0000 7f89cffff640 1 -- 172.21.15.46:0/392154092 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v148) ==== 162+0+0 (secure 0 0 0) 0x7f89d8032e20 con 0x7f89e80b7760 2024-05-09T03:17:19.980 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:19.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.981+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89d40762d0 msgr2=0x7f89d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.981+0000 7f89edabd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89d40762d0 0x7f89d40787e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f89e8162360 tx=0x7f89dc009040 comp rx=0 tx=0).stop 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.981+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 msgr2=0x7f89e81521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.981+0000 7f89edabd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 0x7f89e81521b0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f89d80098c0 tx=0x7f89d80046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 shutdown_connections 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f89e80d2eb0 0x7f89e80cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e80b95a0 0x7f89e81e9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f89d40762d0 0x7f89d40787e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 --2- 172.21.15.46:0/392154092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89e80b7760 0x7f89e81521b0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 >> 172.21.15.46:0/392154092 conn(0x7f89e808d390 msgr2=0x7f89e80b7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 shutdown_connections 2024-05-09T03:17:19.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:19.982+0000 7f89edabd640 1 -- 172.21.15.46:0/392154092 wait complete. 2024-05-09T03:17:19.996 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.110+0000 7f1893c29640 1 Processor -- start 2024-05-09T03:17:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.110+0000 7f1893c29640 1 -- start start 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f1893c29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f188c0b92e0 0x7f188c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f1893c29640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f188c0d0480 0x7f188c0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f1893c29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0ce6a0 0x7f188c0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f1893c29640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f188c0d0dd0 con 0x7f188c0b92e0 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f1893c29640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f188c0d2b80 con 0x7f188c0d0480 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f1893c29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f188c0d2e60 con 0x7f188c0ce6a0 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f188c0b92e0 0x7f188c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f188c0b92e0 0x7f188c0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45020/0 (socket says 172.21.15.46:45020) 2024-05-09T03:17:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 -- 172.21.15.46:0/3367360265 learned_addr learned my addr 172.21.15.46:0/3367360265 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18911c4640 1 --2- 172.21.15.46:0/3367360265 >> v2:172.21.15.88:3300/0 conn(0x7f188c0d0480 0x7f188c0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18921c6640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0ce6a0 0x7f188c0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 -- 172.21.15.46:0/3367360265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0ce6a0 msgr2=0x7f188c0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0ce6a0 0x7f188c0ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 -- 172.21.15.46:0/3367360265 >> v2:172.21.15.88:3300/0 conn(0x7f188c0d0480 msgr2=0x7f188c0d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 --2- 172.21.15.46:0/3367360265 >> v2:172.21.15.88:3300/0 conn(0x7f188c0d0480 0x7f188c0d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18919c5640 1 -- 172.21.15.46:0/3367360265 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188c0d3140 con 0x7f188c0b92e0 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.112+0000 7f18921c6640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0ce6a0 0x7f188c0ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.113+0000 7f18919c5640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b92e0 0x7f188c0b51a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f188c06d150 tx=0x7f187c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1110671a53b583f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.113+0000 7f18909c3640 1 -- 172.21.15.46:0/3367360265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c036070 con 0x7f188c0b92e0 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.113+0000 7f18909c3640 1 -- 172.21.15.46:0/3367360265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f187c031600 con 0x7f188c0b92e0 2024-05-09T03:17:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.113+0000 7f18909c3640 1 -- 172.21.15.46:0/3367360265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c0319d0 con 0x7f188c0b92e0 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 -- 172.21.15.46:0/3367360265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b92e0 msgr2=0x7f188c0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b92e0 0x7f188c0b51a0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f188c06d150 tx=0x7f187c02a610 comp rx=0 tx=0).stop 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 -- 172.21.15.46:0/3367360265 shutdown_connections 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0ce6a0 0x7f188c0ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 --2- 172.21.15.46:0/3367360265 >> v2:172.21.15.88:3300/0 conn(0x7f188c0d0480 0x7f188c0d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 --2- 172.21.15.46:0/3367360265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b92e0 0x7f188c0b51a0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 -- 172.21.15.46:0/3367360265 >> 172.21.15.46:0/3367360265 conn(0x7f188c0d3c00 msgr2=0x7f188c0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 -- 172.21.15.46:0/3367360265 shutdown_connections 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.114+0000 7f1893c29640 1 -- 172.21.15.46:0/3367360265 wait complete. 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.116+0000 7f1893c29640 1 Processor -- start 2024-05-09T03:17:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.116+0000 7f1893c29640 1 -- start start 2024-05-09T03:17:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f1893c29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0b92e0 0x7f188c159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f1893c29640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0ce6a0 0x7f188c0cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f1893c29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 0x7f188c0cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f1893c29640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f188c0cf170 con 0x7f188c0d0480 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f1893c29640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f188c0ceff0 con 0x7f188c0ce6a0 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f1893c29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f188c0cf2f0 con 0x7f188c0b92e0 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f18921c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 0x7f188c0cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f18921c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 0x7f188c0cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45024/0 (socket says 172.21.15.46:45024) 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.117+0000 7f18921c6640 1 -- 172.21.15.46:0/4112030904 learned_addr learned my addr 172.21.15.46:0/4112030904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18911c4640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0ce6a0 0x7f188c0cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18919c5640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0b92e0 0x7f188c159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18921c6640 1 -- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0b92e0 msgr2=0x7f188c159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18921c6640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0b92e0 0x7f188c159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18921c6640 1 -- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0ce6a0 msgr2=0x7f188c0cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18921c6640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0ce6a0 0x7f188c0cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18921c6640 1 -- 172.21.15.46:0/4112030904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188c0cd970 con 0x7f188c0d0480 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f18921c6640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 0x7f188c0cc9b0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f188400d520 tx=0x7f188400d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1884018020 con 0x7f188c0d0480 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1884014620 con 0x7f188c0d0480 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18840149f0 con 0x7f188c0d0480 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188c0cdc50 con 0x7f188c0d0480 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.118+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188c159670 con 0x7f188c0d0480 2024-05-09T03:17:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.119+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f188c0cd6e0 con 0x7f188c0d0480 2024-05-09T03:17:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.120+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f188401b020 con 0x7f188c0d0480 2024-05-09T03:17:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.120+0000 7f188a7fc640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18780762d0 0x7f18780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.121+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6627+0+0 (secure 0 0 0) 0x7f1884097e40 con 0x7f188c0d0480 2024-05-09T03:17:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.121+0000 7f18919c5640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18780762d0 0x7f18780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.121+0000 7f18919c5640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18780762d0 0x7f18780787e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f187c033000 tx=0x7f187c009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:20.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.125+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1884022020 con 0x7f188c0d0480 2024-05-09T03:17:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.355+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f188c06a270 con 0x7f188c0d0480 2024-05-09T03:17:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.355+0000 7f188a7fc640 1 -- 172.21.15.46:0/4112030904 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5177 (secure 0 0 0) 0x7f1884061370 con 0x7f188c0d0480 2024-05-09T03:17:20.357 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:20.357 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":148,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:19.978982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"c","rank":0,"incarnation":148,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/2032086476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2032086476},{"type":"v1","addr":"172.21.15.46:6834","nonce":2032086476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 148 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18780762d0 msgr2=0x7f18780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18780762d0 0x7f18780787e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f187c033000 tx=0x7f187c009000 comp rx=0 tx=0).stop 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 msgr2=0x7f188c0cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 0x7f188c0cc9b0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f188400d520 tx=0x7f188400d9e0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 shutdown_connections 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18780762d0 0x7f18780787e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0d0480 0x7f188c0cc9b0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0ce6a0 0x7f188c0cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 --2- 172.21.15.46:0/4112030904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0b92e0 0x7f188c159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 >> 172.21.15.46:0/4112030904 conn(0x7f188c0d3c00 msgr2=0x7f188c0b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 shutdown_connections 2024-05-09T03:17:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:20.357+0000 7f1893c29640 1 -- 172.21.15.46:0/4112030904 wait complete. 2024-05-09T03:17:20.370 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:17:22.986 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:22.984+0000 7f6b31472640 -1 log_channel(cluster) log [ERR] : dir 0x100 object missing on disk; some files may be lost (~mds0) 2024-05-09T03:17:22.989 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:22.987+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:22.989 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:22.987+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:22.995 INFO:tasks.ceph.mds.c.smithi046.stderr: -21> 2024-05-09T03:17:22.984+0000 7f6b31472640 -1 log_channel(cluster) log [ERR] : dir 0x100 object missing on disk; some files may be lost (~mds0) 2024-05-09T03:17:23.057 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:25.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:25.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.500+0000 7fadf4e30640 1 Processor -- start 2024-05-09T03:17:25.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.500+0000 7fadf4e30640 1 -- start start 2024-05-09T03:17:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.500+0000 7fadf4e30640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00d0b90 0x7fadf00b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.500+0000 7fadf4e30640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadf00d2b80 con 0x7fadf00b92e0 2024-05-09T03:17:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.500+0000 7fadf4e30640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadf00d2e60 con 0x7fadf00cf6b0 2024-05-09T03:17:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.500+0000 7fadf4e30640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fadf00d3140 con 0x7fadf00d0b90 2024-05-09T03:17:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7fadee59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fadf00b92e0 0x7fadf00d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38148/0 (socket says 172.21.15.46:38148) 2024-05-09T03:17:25.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7fadee59c640 1 -- 172.21.15.46:0/552177196 learned_addr learned my addr 172.21.15.46:0/552177196 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415565026 0 0) 0x7fadf00d2b80 con 0x7fadf00b92e0 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7fadedd9b640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00d0b90 0x7fadf00b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadd8003520 con 0x7fadf00b92e0 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 700481841 0 0) 0x7fadd8003520 con 0x7fadf00b92e0 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00d0b90 msgr2=0x7fadf00b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00d0b90 0x7fadf00b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 >> v1:172.21.15.88:6789/0 conn(0x7fadf00cf6b0 legacy=0x7fadf006d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf00d3420 con 0x7fadf00b92e0 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fade00016f0 con 0x7fadf00b92e0 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 >> v1:172.21.15.46:6789/0 conn(0x7fadf00b92e0 legacy=0x7fadf00d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80039e0 0x7fadd8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadd8004800 0x7fadd8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd8005150 0x7fadd800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fadf00d3140 con 0x7fadd8005150 2024-05-09T03:17:25.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fadd800bfe0 con 0x7fadd80039e0 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fadd800c1e0 con 0x7fadd8004800 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7fadee59c640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80039e0 0x7fadd8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.501+0000 7fadeed9d640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd8005150 0x7fadd800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.502+0000 7fadedd9b640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadd8004800 0x7fadd8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.502+0000 7fadeed9d640 1 -- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadd8004800 msgr2=0x7fadd8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.502+0000 7fadeed9d640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadd8004800 0x7fadd8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.502+0000 7fadeed9d640 1 -- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80039e0 msgr2=0x7fadd8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.502+0000 7fadeed9d640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80039e0 0x7fadd8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.502+0000 7fadeed9d640 1 -- 172.21.15.46:0/552177196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd8003520 con 0x7fadd8005150 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7fadeed9d640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd8005150 0x7fadd800b9c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fade8001f10 tx=0x7fadd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a22e5d389b25828 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8036070 con 0x7fadd8005150 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade8031600 con 0x7fadd8005150 2024-05-09T03:17:25.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7faded59a640 1 -- 172.21.15.46:0/552177196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8001b30 con 0x7fadd8005150 2024-05-09T03:17:25.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7fadf4e30640 1 -- 172.21.15.46:0/552177196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd8005150 msgr2=0x7fadd800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7fadf4e30640 1 --2- 172.21.15.46:0/552177196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadd8005150 0x7fadd800b9c0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fade8001f10 tx=0x7fadd8005830 comp rx=0 tx=0).stop 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.503+0000 7fadeed9d640 1 -- 172.21.15.46:0/552177196 reap_dead start 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.504+0000 7fadf4e30640 1 -- 172.21.15.46:0/552177196 shutdown_connections 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.504+0000 7fadf4e30640 1 -- 172.21.15.46:0/552177196 >> 172.21.15.46:0/552177196 conn(0x7fadf00d3c00 msgr2=0x7fadf00d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.504+0000 7fadf4e30640 1 -- 172.21.15.46:0/552177196 shutdown_connections 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.504+0000 7fadf4e30640 1 -- 172.21.15.46:0/552177196 wait complete. 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.507+0000 7fadf4e30640 1 Processor -- start 2024-05-09T03:17:25.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.507+0000 7fadf4e30640 1 -- start start 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd8005150 0x7fadf01520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00b4220 0x7fadf009f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 0x7fadf00b7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fadf00b51a0 con 0x7fadf009f710 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fadf00d0da0 con 0x7fadd8005150 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fadf00b5320 con 0x7fadf00b4220 2024-05-09T03:17:25.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 0x7fadf00b7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 0x7fadf00b7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45070/0 (socket says 172.21.15.46:45070) 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 -- 172.21.15.46:0/3872112213 learned_addr learned my addr 172.21.15.46:0/3872112213 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadee59c640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd8005150 0x7fadf01520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadedd9b640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00b4220 0x7fadf009f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 -- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00b4220 msgr2=0x7fadf009f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00b4220 0x7fadf009f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 -- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd8005150 msgr2=0x7fadf01520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd8005150 0x7fadf01520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 -- 172.21.15.46:0/3872112213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf00b2920 con 0x7fadf009f710 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadeed9d640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 0x7fadf00b7650 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fade802a9d0 tx=0x7fade8001490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8036040 con 0x7fadf009f710 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade8031400 con 0x7fadf009f710 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf00b2c00 con 0x7fadf009f710 2024-05-09T03:17:25.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade8001b30 con 0x7fadf009f710 2024-05-09T03:17:25.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.508+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf00b31c0 con 0x7fadf009f710 2024-05-09T03:17:25.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.509+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf006a270 con 0x7fadf009f710 2024-05-09T03:17:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.510+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fade8043020 con 0x7fadf009f710 2024-05-09T03:17:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.510+0000 7faddeffd640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fadd00762d0 0x7fadd00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.510+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7fade80b7430 con 0x7fadf009f710 2024-05-09T03:17:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.511+0000 7fadee59c640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fadd00762d0 0x7fadd00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.511+0000 7fadee59c640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fadd00762d0 0x7fadd00787e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fade00018a0 tx=0x7fade0001630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:25.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.515+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fade8080350 con 0x7fadf009f710 2024-05-09T03:17:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.749+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadf006e5b0 con 0x7fadf009f710 2024-05-09T03:17:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.750+0000 7faddeffd640 1 -- 172.21.15.46:0/3872112213 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) ==== 95+0+5168 (secure 0 0 0) 0x7fade8084650 con 0x7fadf009f710 2024-05-09T03:17:25.752 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:25.752 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":153,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1270051447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1270051447},{"type":"v1","addr":"172.21.15.46:6834","nonce":1270051447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:22.988964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:25.752 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 153 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fadd00762d0 msgr2=0x7fadd00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fadd00762d0 0x7fadd00787e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fade00018a0 tx=0x7fade0001630 comp rx=0 tx=0).stop 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 msgr2=0x7fadf00b7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 0x7fadf00b7650 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fade802a9d0 tx=0x7fade8001490 comp rx=0 tx=0).stop 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 shutdown_connections 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fadd00762d0 0x7fadd00787e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadf009f710 0x7fadf00b7650 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fadf00b4220 0x7fadf009f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 --2- 172.21.15.46:0/3872112213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd8005150 0x7fadf01520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 >> 172.21.15.46:0/3872112213 conn(0x7fadf00d3c00 msgr2=0x7fadf006d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:25.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 shutdown_connections 2024-05-09T03:17:25.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.753+0000 7fadf4e30640 1 -- 172.21.15.46:0/3872112213 wait complete. 2024-05-09T03:17:25.765 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:17:25.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:25.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.882+0000 7f4d9be9f640 1 Processor -- start 2024-05-09T03:17:25.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.882+0000 7f4d9be9f640 1 -- start start 2024-05-09T03:17:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9be9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d941615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9be9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d9415dfe0 0x7f4d940d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9be9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d94160b20 con 0x7f4d940d2480 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9be9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4d94161150 con 0x7f4d9415dfe0 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9be9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d940d15f0 con 0x7f4d94164c70 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9943a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d9415dfe0 0x7f4d940d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9a43c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4d940d2480 0x7f4d940d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40976/0 (socket says 172.21.15.46:40976) 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.883+0000 7f4d9a43c640 1 -- 172.21.15.46:0/219034802 learned_addr learned my addr 172.21.15.46:0/219034802 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d99c3b640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d941615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d9943a640 1 -- 172.21.15.46:0/219034802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 msgr2=0x7f4d941615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d9943a640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d941615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d9943a640 1 -- 172.21.15.46:0/219034802 >> v1:172.21.15.46:6789/0 conn(0x7f4d940d2480 legacy=0x7f4d940d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d9943a640 1 -- 172.21.15.46:0/219034802 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d940d18d0 con 0x7f4d9415dfe0 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d98c39640 1 -- 172.21.15.46:0/219034802 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719516605 0 0) 0x7f4d94160b20 con 0x7f4d940d2480 2024-05-09T03:17:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d99c3b640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d941615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.884+0000 7f4d9943a640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d9415dfe0 0x7f4d940d0dc0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4d840097a0 tx=0x7f4d8402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90c3af17fb60d6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d98c39640 1 -- 172.21.15.46:0/219034802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84037070 con 0x7f4d9415dfe0 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d98c39640 1 -- 172.21.15.46:0/219034802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8402ae50 con 0x7f4d9415dfe0 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d98c39640 1 -- 172.21.15.46:0/219034802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84032660 con 0x7f4d9415dfe0 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 -- 172.21.15.46:0/219034802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d9415dfe0 msgr2=0x7f4d940d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d9415dfe0 0x7f4d940d0dc0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4d840097a0 tx=0x7f4d8402a100 comp rx=0 tx=0).stop 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 -- 172.21.15.46:0/219034802 shutdown_connections 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d9415dfe0 0x7f4d940d0dc0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/219034802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d941615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 -- 172.21.15.46:0/219034802 >> 172.21.15.46:0/219034802 conn(0x7f4d94073db0 msgr2=0x7f4d940741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 -- 172.21.15.46:0/219034802 shutdown_connections 2024-05-09T03:17:25.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.885+0000 7f4d9be9f640 1 -- 172.21.15.46:0/219034802 wait complete. 2024-05-09T03:17:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.887+0000 7f4d9be9f640 1 Processor -- start 2024-05-09T03:17:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.887+0000 7f4d9be9f640 1 -- start start 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 0x7f4d941523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d9415dfe0 0x7f4d94152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d94158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9415e3f0 con 0x7f4d940d2480 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d940d1300 con 0x7f4d9415dfe0 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d940d45c0 con 0x7f4d94164c70 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 0x7f4d941523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 0x7f4d941523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55196/0 (socket says 172.21.15.46:55196) 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 -- 172.21.15.46:0/3872032159 learned_addr learned my addr 172.21.15.46:0/3872032159 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9943a640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d9415dfe0 0x7f4d94152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9a43c640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d94158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 -- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 msgr2=0x7f4d94158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d94158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 -- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d9415dfe0 msgr2=0x7f4d94152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d9415dfe0 0x7f4d94152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 -- 172.21.15.46:0/3872032159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d94152920 con 0x7f4d940d2480 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9a43c640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d94158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d99c3b640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 0x7f4d941523e0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000c820 tx=0x7f4d9000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90015070 con 0x7f4d940d2480 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d90007a30 con 0x7f4d940d2480 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d90007de0 con 0x7f4d940d2480 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d94153060 con 0x7f4d940d2480 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.888+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d94158290 con 0x7f4d940d2480 2024-05-09T03:17:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.890+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d9408cfd0 con 0x7f4d940d2480 2024-05-09T03:17:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.891+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4d90018070 con 0x7f4d940d2480 2024-05-09T03:17:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.892+0000 7f4d8a7fc640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d780762d0 0x7f4d780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.892+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7f4d90097810 con 0x7f4d940d2480 2024-05-09T03:17:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.892+0000 7f4d9943a640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d780762d0 0x7f4d780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.892+0000 7f4d9943a640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d780762d0 0x7f4d780787e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4d84034000 tx=0x7f4d8402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:25.896+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4d9408cfd0 con 0x7f4d940d2480 2024-05-09T03:17:26.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.128+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d9406a270 con 0x7f4d940d2480 2024-05-09T03:17:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.128+0000 7f4d8a7fc640 1 -- 172.21.15.46:0/3872032159 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) ==== 95+0+5168 (secure 0 0 0) 0x7f4d94152920 con 0x7f4d940d2480 2024-05-09T03:17:26.129 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:26.129 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":153,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1270051447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1270051447},{"type":"v1","addr":"172.21.15.46:6834","nonce":1270051447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:22.988964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:26.129 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 153 2024-05-09T03:17:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d780762d0 msgr2=0x7f4d780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d780762d0 0x7f4d780787e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4d84034000 tx=0x7f4d8402e000 comp rx=0 tx=0).stop 2024-05-09T03:17:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 msgr2=0x7f4d941523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 0x7f4d941523e0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f4d9000c820 tx=0x7f4d9000cce0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 shutdown_connections 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d94164c70 0x7f4d94158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d9415dfe0 0x7f4d94152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d780762d0 0x7f4d780787e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 --2- 172.21.15.46:0/3872032159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d940d2480 0x7f4d941523e0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 >> 172.21.15.46:0/3872032159 conn(0x7f4d94073db0 msgr2=0x7f4d940d49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 shutdown_connections 2024-05-09T03:17:26.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.130+0000 7f4d9be9f640 1 -- 172.21.15.46:0/3872032159 wait complete. 2024-05-09T03:17:26.142 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:26.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.256+0000 7f0215527640 1 Processor -- start 2024-05-09T03:17:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.256+0000 7f0215527640 1 -- start start 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f0215527640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 0x7f0210160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f0215527640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02100b7ef0 0x7f02100b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f0215527640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02100d49e0 con 0x7f02100b4af0 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f0215527640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f02100d5bb0 con 0x7f02100b7ef0 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f0215527640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02100d4cc0 con 0x7f02100d57a0 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 0x7f0210160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02100b7ef0 0x7f02100b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 0x7f0210160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:50314/0 (socket says 172.21.15.46:50314) 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 -- 172.21.15.46:0/2199778710 learned_addr learned my addr 172.21.15.46:0/2199778710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 -- 172.21.15.46:0/2199778710 >> v2:172.21.15.88:3300/0 conn(0x7f02100b7ef0 msgr2=0x7f02100b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 --2- 172.21.15.46:0/2199778710 >> v2:172.21.15.88:3300/0 conn(0x7f02100b7ef0 0x7f02100b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 -- 172.21.15.46:0/2199778710 >> v1:172.21.15.88:6789/0 conn(0x7f02100b4af0 legacy=0x7f02100b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020effd640 1 -- 172.21.15.46:0/2199778710 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02100d4fa0 con 0x7f02100d57a0 2024-05-09T03:17:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.258+0000 7f020f7fe640 1 --2- 172.21.15.46:0/2199778710 >> v2:172.21.15.88:3300/0 conn(0x7f02100b7ef0 0x7f02100b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.259+0000 7f020effd640 1 --2- 172.21.15.46:0/2199778710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 0x7f0210160610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0200009e50 tx=0x7f020002a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92f86e399c6d787 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.259+0000 7f020dffb640 1 -- 172.21.15.46:0/2199778710 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200036070 con 0x7f02100d57a0 2024-05-09T03:17:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.259+0000 7f020dffb640 1 -- 172.21.15.46:0/2199778710 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0200031600 con 0x7f02100d57a0 2024-05-09T03:17:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.259+0000 7f020dffb640 1 -- 172.21.15.46:0/2199778710 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02000319d0 con 0x7f02100d57a0 2024-05-09T03:17:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 -- 172.21.15.46:0/2199778710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 msgr2=0x7f0210160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 --2- 172.21.15.46:0/2199778710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 0x7f0210160610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0200009e50 tx=0x7f020002a730 comp rx=0 tx=0).stop 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 -- 172.21.15.46:0/2199778710 shutdown_connections 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 --2- 172.21.15.46:0/2199778710 >> v2:172.21.15.88:3300/0 conn(0x7f02100b7ef0 0x7f02100b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 --2- 172.21.15.46:0/2199778710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02100d57a0 0x7f0210160610 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 -- 172.21.15.46:0/2199778710 >> 172.21.15.46:0/2199778710 conn(0x7f02100743e0 msgr2=0x7f02100d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 -- 172.21.15.46:0/2199778710 shutdown_connections 2024-05-09T03:17:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.260+0000 7f0215527640 1 -- 172.21.15.46:0/2199778710 wait complete. 2024-05-09T03:17:26.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.262+0000 7f0215527640 1 Processor -- start 2024-05-09T03:17:26.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.262+0000 7f0215527640 1 -- start start 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.263+0000 7f0215527640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 0x7f02100ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.263+0000 7f0215527640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02100d57a0 0x7f0210159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.263+0000 7f0215527640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02101ea0e0 0x7f02101ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f0215527640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02100d1920 con 0x7f02100b4af0 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f0215527640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f02100d15b0 con 0x7f02100d57a0 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f0215527640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02100cc2e0 con 0x7f02101ea0e0 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 0x7f02100ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 0x7f02100ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55200/0 (socket says 172.21.15.46:55200) 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 -- 172.21.15.46:0/2944782787 learned_addr learned my addr 172.21.15.46:0/2944782787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020f7fe640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02101ea0e0 0x7f02101ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020e7fc640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02100d57a0 0x7f0210159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 -- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02101ea0e0 msgr2=0x7f02101ea4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02101ea0e0 0x7f02101ea4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 -- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02100d57a0 msgr2=0x7f0210159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02100d57a0 0x7f0210159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 -- 172.21.15.46:0/2944782787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02100c9830 con 0x7f02100b4af0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020f7fe640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02101ea0e0 0x7f02101ea4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f020effd640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 0x7f02100ca1c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f0200009f70 tx=0x7f0200004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020003f020 con 0x7f02100b4af0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0200037040 con 0x7f02100b4af0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.264+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02100c9d30 con 0x7f02100b4af0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.265+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0200036070 con 0x7f02100b4af0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.265+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02101e98e0 con 0x7f02100b4af0 2024-05-09T03:17:26.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.265+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02101e93f0 con 0x7f02100b4af0 2024-05-09T03:17:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.266+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0200042020 con 0x7f02100b4af0 2024-05-09T03:17:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.267+0000 7f01f77fe640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01fc076300 0x7f01fc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.267+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7f02000b8290 con 0x7f02100b4af0 2024-05-09T03:17:26.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.267+0000 7f020e7fc640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01fc076300 0x7f01fc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.268+0000 7f020e7fc640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01fc076300 0x7f01fc078810 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f021015acf0 tx=0x7f0204009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:26.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.271+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0200049030 con 0x7f02100b4af0 2024-05-09T03:17:26.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.506+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f021006a270 con 0x7f02100b4af0 2024-05-09T03:17:26.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.507+0000 7f01f77fe640 1 -- 172.21.15.46:0/2944782787 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) ==== 95+0+5168 (secure 0 0 0) 0x7f0200081770 con 0x7f02100b4af0 2024-05-09T03:17:26.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:26.508 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":153,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1270051447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1270051447},{"type":"v1","addr":"172.21.15.46:6834","nonce":1270051447}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:22.988964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:26.508 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 153 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01fc076300 msgr2=0x7f01fc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01fc076300 0x7f01fc078810 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f021015acf0 tx=0x7f0204009040 comp rx=0 tx=0).stop 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 msgr2=0x7f02100ca1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 0x7f02100ca1c0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f0200009f70 tx=0x7f0200004a40 comp rx=0 tx=0).stop 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 shutdown_connections 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02101ea0e0 0x7f02101ea4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02100d57a0 0x7f0210159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01fc076300 0x7f01fc078810 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 --2- 172.21.15.46:0/2944782787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02100b4af0 0x7f02100ca1c0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 >> 172.21.15.46:0/2944782787 conn(0x7f02100743e0 msgr2=0x7f02100b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 shutdown_connections 2024-05-09T03:17:26.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.509+0000 7f0215527640 1 -- 172.21.15.46:0/2944782787 wait complete. 2024-05-09T03:17:26.521 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 100.00000000' led to DAMAGED state 2024-05-09T03:17:26.521 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_sessionmap' 2024-05-09T03:17:26.521 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:26.521 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:26.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:26.538 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:26.538 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:26.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:26.538 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:26.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:26.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.751+0000 7fed098d1640 1 Processor -- start 2024-05-09T03:17:26.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.751+0000 7fed098d1640 1 -- start start 2024-05-09T03:17:26.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed098d1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed040d1c40 0x7fed040d2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed098d1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed040b0280 con 0x7fed0406f390 2024-05-09T03:17:26.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed098d1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed040b0520 con 0x7fed0406cf10 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed098d1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fed040d0450 con 0x7fed040d1c40 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed02ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fed0406f390 0x7fed040b1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40986/0 (socket says 172.21.15.46:40986) 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed02ffd640 1 -- 172.21.15.46:0/255159295 learned_addr learned my addr 172.21.15.46:0/255159295 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.752+0000 7fed037fe640 1 --2- 172.21.15.46:0/255159295 >> v2:172.21.15.88:3300/0 conn(0x7fed040d1c40 0x7fed040d2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.753+0000 7fed01ffb640 1 -- 172.21.15.46:0/255159295 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979562158 0 0) 0x7fed040b0280 con 0x7fed0406f390 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.753+0000 7fed01ffb640 1 -- 172.21.15.46:0/255159295 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecec003520 con 0x7fed0406f390 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.753+0000 7fed01ffb640 1 -- 172.21.15.46:0/255159295 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607408615 0 0) 0x7fecec003520 con 0x7fed0406f390 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.753+0000 7fed037fe640 1 -- 172.21.15.46:0/255159295 >> v1:172.21.15.88:6789/0 conn(0x7fed0406cf10 legacy=0x7fed0406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.753+0000 7fed037fe640 1 -- 172.21.15.46:0/255159295 >> v1:172.21.15.46:6789/0 conn(0x7fed0406f390 legacy=0x7fed040b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.753+0000 7fed037fe640 1 -- 172.21.15.46:0/255159295 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed040b0800 con 0x7fed040d1c40 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.754+0000 7fed037fe640 1 --2- 172.21.15.46:0/255159295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed040d2050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009d90 tx=0x7fecfc02a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1cee0c862d6a919 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.754+0000 7fed01ffb640 1 -- 172.21.15.46:0/255159295 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc035020 con 0x7fed040d1c40 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.754+0000 7fed01ffb640 1 -- 172.21.15.46:0/255159295 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecfc007540 con 0x7fed040d1c40 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.754+0000 7fed01ffb640 1 -- 172.21.15.46:0/255159295 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4001710 con 0x7fed040d1c40 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.754+0000 7fed098d1640 1 -- 172.21.15.46:0/255159295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 msgr2=0x7fed040d2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.754+0000 7fed098d1640 1 --2- 172.21.15.46:0/255159295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed040d2050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009d90 tx=0x7fecfc02a5b0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.755+0000 7fed098d1640 1 -- 172.21.15.46:0/255159295 shutdown_connections 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.755+0000 7fed098d1640 1 --2- 172.21.15.46:0/255159295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed040d2050 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.755+0000 7fed098d1640 1 -- 172.21.15.46:0/255159295 >> 172.21.15.46:0/255159295 conn(0x7fed0409b7d0 msgr2=0x7fed0409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.755+0000 7fed098d1640 1 -- 172.21.15.46:0/255159295 shutdown_connections 2024-05-09T03:17:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.755+0000 7fed098d1640 1 -- 172.21.15.46:0/255159295 wait complete. 2024-05-09T03:17:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.756+0000 7fed098d1640 1 Processor -- start 2024-05-09T03:17:26.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.757+0000 7fed098d1640 1 -- start start 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed098d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed0406cf10 0x7fed040c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed098d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 0x7fed040c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed098d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed041f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed098d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed040b1710 con 0x7fed0406f390 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed098d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fed040d3600 con 0x7fed040d1c40 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed098d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fed040d2590 con 0x7fed0406cf10 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 0x7fed040c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 0x7fed040c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55216/0 (socket says 172.21.15.46:55216) 2024-05-09T03:17:26.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 -- 172.21.15.46:0/1937634953 learned_addr learned my addr 172.21.15.46:0/1937634953 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed037fe640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed041f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed02ffd640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed0406cf10 0x7fed040c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 -- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed0406cf10 msgr2=0x7fed040c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed0406cf10 0x7fed040c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 -- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 msgr2=0x7fed041f65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed041f65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed027fc640 1 -- 172.21.15.46:0/1937634953 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed04161390 con 0x7fed0406f390 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.758+0000 7fed037fe640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed041f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.759+0000 7fed027fc640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 0x7fed040c8c40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fecf800c440 tx=0x7fecf800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.759+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8017070 con 0x7fed0406f390 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.759+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecf8005480 con 0x7fed0406f390 2024-05-09T03:17:26.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.759+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed0415ea70 con 0x7fed0406f390 2024-05-09T03:17:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.759+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf8003830 con 0x7fed0406f390 2024-05-09T03:17:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.759+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed0415f030 con 0x7fed0406f390 2024-05-09T03:17:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.760+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0415d0a0 con 0x7fed0406f390 2024-05-09T03:17:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.760+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fecf801a070 con 0x7fed0406f390 2024-05-09T03:17:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.761+0000 7fecebfff640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecf00762d0 0x7fecf00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.761+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7fecf8096c80 con 0x7fed0406f390 2024-05-09T03:17:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.761+0000 7fed02ffd640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecf00762d0 0x7fecf00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.762+0000 7fed02ffd640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecf00762d0 0x7fecf00787e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fecf4001240 tx=0x7fecf40012d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:26.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.766+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fed0415d0a0 con 0x7fed0406f390 2024-05-09T03:17:26.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:26.997+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fed04070640 con 0x7fed0406f390 2024-05-09T03:17:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.176+0000 7fecebfff640 1 -- 172.21.15.46:0/1937634953 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v154) ==== 161+0+0 (secure 0 0 0) 0x7fecf80681e0 con 0x7fed0406f390 2024-05-09T03:17:27.177 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:27.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecf00762d0 msgr2=0x7fecf00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecf00762d0 0x7fecf00787e0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fecf4001240 tx=0x7fecf40012d0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 msgr2=0x7fed040c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 0x7fed040c8c40 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fecf800c440 tx=0x7fecf800c900 comp rx=0 tx=0).stop 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 shutdown_connections 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed040d1c40 0x7fed041f65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecf00762d0 0x7fecf00787e0 secure :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fecf4001240 tx=0x7fecf40012d0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed0406f390 0x7fed040c8c40 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 --2- 172.21.15.46:0/1937634953 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fed0406cf10 0x7fed040c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 >> 172.21.15.46:0/1937634953 conn(0x7fed0409b7d0 msgr2=0x7fed0415de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.178+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 shutdown_connections 2024-05-09T03:17:27.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.179+0000 7fed098d1640 1 -- 172.21.15.46:0/1937634953 wait complete. 2024-05-09T03:17:27.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.306+0000 7fedc3a29640 1 Processor -- start 2024-05-09T03:17:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.306+0000 7fedc3a29640 1 -- start start 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc3a29640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fedbc0b6660 0x7fedbc0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc3a29640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc0b55c0 con 0x7fedbc0d1f00 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc3a29640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedbc0b58a0 con 0x7fedbc147370 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc3a29640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fedbc0b5b80 con 0x7fedbc0b6660 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc1fc6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fedbc0d1f00 0x7fedbc144d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40988/0 (socket says 172.21.15.46:40988) 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc1fc6640 1 -- 172.21.15.46:0/709164169 learned_addr learned my addr 172.21.15.46:0/709164169 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedc17c5640 1 --2- 172.21.15.46:0/709164169 >> v2:172.21.15.88:3300/0 conn(0x7fedbc0b6660 0x7fedbc0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400443036 0 0) 0x7fedbc0b55c0 con 0x7fedbc0d1f00 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feda4003520 con 0x7fedbc0d1f00 2024-05-09T03:17:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3711159128 0 0) 0x7fedbc0b58a0 con 0x7fedbc147370 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedbc0b55c0 con 0x7fedbc147370 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.308+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4215574925 0 0) 0x7feda4003520 con 0x7fedbc0d1f00 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 >> v2:172.21.15.88:3300/0 conn(0x7fedbc0b6660 msgr2=0x7fedbc0b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 --2- 172.21.15.46:0/709164169 >> v2:172.21.15.88:3300/0 conn(0x7fedbc0b6660 0x7fedbc0b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 >> v1:172.21.15.88:6789/0 conn(0x7fedbc147370 legacy=0x7fedbc0d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc0b5e60 con 0x7fedbc0d1f00 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fedac0019f0 con 0x7fedbc0d1f00 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 >> v1:172.21.15.46:6789/0 conn(0x7fedbc0d1f00 legacy=0x7fedbc144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feda4003c80 0x7feda40040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4004800 0x7feda4004c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda40052e0 0x7feda400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc0b5b80 con 0x7feda4004800 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7feda400c280 con 0x7feda4003c80 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7feda400c480 con 0x7feda40052e0 2024-05-09T03:17:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc0fc4640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4004800 0x7feda4004c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc17c5640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feda4003c80 0x7feda40040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc1fc6640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda40052e0 0x7feda400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc17c5640 1 -- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda40052e0 msgr2=0x7feda400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc17c5640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda40052e0 0x7feda400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc17c5640 1 -- 172.21.15.46:0/709164169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4004800 msgr2=0x7feda4004c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc17c5640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4004800 0x7feda4004c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc17c5640 1 -- 172.21.15.46:0/709164169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda4003520 con 0x7feda4003c80 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc0fc4640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4004800 0x7feda4004c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.309+0000 7fedc1fc6640 1 -- 172.21.15.46:0/709164169 reap_dead start 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.310+0000 7fedc17c5640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feda4003c80 0x7feda40040d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009ac0 tx=0x7feda4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5584954090b48476 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.310+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb4036070 con 0x7feda4003c80 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.310+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb4031400 con 0x7feda4003c80 2024-05-09T03:17:27.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.310+0000 7fedbbfff640 1 -- 172.21.15.46:0/709164169 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb4009690 con 0x7feda4003c80 2024-05-09T03:17:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 -- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feda4003c80 msgr2=0x7feda40040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feda4003c80 0x7feda40040d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009ac0 tx=0x7feda4005830 comp rx=0 tx=0).stop 2024-05-09T03:17:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 -- 172.21.15.46:0/709164169 shutdown_connections 2024-05-09T03:17:27.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 --2- 172.21.15.46:0/709164169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feda4003c80 0x7feda40040d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 -- 172.21.15.46:0/709164169 >> 172.21.15.46:0/709164169 conn(0x7fedbc0b3620 msgr2=0x7fedbc0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 -- 172.21.15.46:0/709164169 shutdown_connections 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.311+0000 7fedc3a29640 1 -- 172.21.15.46:0/709164169 wait complete. 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.313+0000 7fedc3a29640 1 Processor -- start 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.313+0000 7fedc3a29640 1 -- start start 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc3a29640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda4003c80 0x7fedbc1412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc3a29640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedbc0b0970 0x7fedbc0b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc3a29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 0x7fedbc1e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc3a29640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc0b4ee0 con 0x7fedbc1e4b30 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc3a29640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fedbc1447e0 con 0x7fedbc0b0970 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc3a29640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fedbc0b5060 con 0x7feda4003c80 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc1fc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 0x7fedbc1e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc1fc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 0x7fedbc1e4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55240/0 (socket says 172.21.15.46:55240) 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.314+0000 7fedc1fc6640 1 -- 172.21.15.46:0/3973628105 learned_addr learned my addr 172.21.15.46:0/3973628105 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc0fc4640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedbc0b0970 0x7fedbc0b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc17c5640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda4003c80 0x7fedbc1412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc1fc6640 1 -- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda4003c80 msgr2=0x7fedbc1412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc1fc6640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda4003c80 0x7fedbc1412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc1fc6640 1 -- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedbc0b0970 msgr2=0x7fedbc0b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc1fc6640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedbc0b0970 0x7fedbc0b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc1fc6640 1 -- 172.21.15.46:0/3973628105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedbc1e5480 con 0x7fedbc1e4b30 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc0fc4640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedbc0b0970 0x7fedbc0b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc1fc6640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 0x7fedbc1e4f40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0d2610 tx=0x7fedac007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac00aa90 con 0x7fedbc1e4b30 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedac019070 con 0x7fedbc1e4b30 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedbc146c20 con 0x7fedbc1e4b30 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedac0133f0 con 0x7fedbc1e4b30 2024-05-09T03:17:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.315+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedbc144de0 con 0x7fedbc1e4b30 2024-05-09T03:17:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.316+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedbc06a270 con 0x7fedbc1e4b30 2024-05-09T03:17:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.317+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fedac013590 con 0x7fedbc1e4b30 2024-05-09T03:17:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.318+0000 7fedb9ffb640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda80762d0 0x7feda80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.318+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7fedac097fe0 con 0x7fedbc1e4b30 2024-05-09T03:17:27.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.318+0000 7fedc17c5640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda80762d0 0x7feda80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:27.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.319+0000 7fedc17c5640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda80762d0 0x7feda80787e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fedb4033000 tx=0x7fedb4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:27.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.323+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fedbc1e5480 con 0x7fedbc1e4b30 2024-05-09T03:17:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:27.590+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fedbc0715f0 con 0x7fedbc1e4b30 2024-05-09T03:17:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.177+0000 7fedb9ffb640 1 -- 172.21.15.46:0/3973628105 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v155) ==== 101+0+0 (secure 0 0 0) 0x7fedac0614c0 con 0x7fedbc1e4b30 2024-05-09T03:17:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda80762d0 msgr2=0x7feda80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda80762d0 0x7feda80787e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fedb4033000 tx=0x7fedb4008030 comp rx=0 tx=0).stop 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 msgr2=0x7fedbc1e4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 0x7fedbc1e4f40 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fedbc0d2610 tx=0x7fedac007b20 comp rx=0 tx=0).stop 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 shutdown_connections 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda80762d0 0x7feda80787e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedbc1e4b30 0x7fedbc1e4f40 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedbc0b0970 0x7fedbc0b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 --2- 172.21.15.46:0/3973628105 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7feda4003c80 0x7fedbc1412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 >> 172.21.15.46:0/3973628105 conn(0x7fedbc0b3620 msgr2=0x7fedbc1e65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 shutdown_connections 2024-05-09T03:17:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.180+0000 7fedc3a29640 1 -- 172.21.15.46:0/3973628105 wait complete. 2024-05-09T03:17:28.194 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:28.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.253+0000 7f394c366f40 1 Processor -- start 2024-05-09T03:17:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.253+0000 7f394c366f40 1 -- start start 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394c366f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb597e0 0x5590dfb8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394c366f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfb8b7b0 0x5590dfc33c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394c366f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 0x5590dfc34160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394c366f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5590dfba04d0 con 0x5590dfc33810 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394c366f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5590dfb02f50 con 0x5590dfb597e0 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394c366f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5590dfba58a0 con 0x5590dfb8b7b0 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f3946631640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 0x5590dfc34160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f3946631640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 0x5590dfc34160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55268/0 (socket says 172.21.15.46:55268) 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f3946631640 1 -- 172.21.15.46:0/3872390407 learned_addr learned my addr 172.21.15.46:0/3872390407 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f3945e30640 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb597e0 0x5590dfb8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.254+0000 7f394562f640 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfb8b7b0 0x5590dfc33c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3946631640 1 -- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfb8b7b0 msgr2=0x5590dfc33c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3946631640 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfb8b7b0 0x5590dfc33c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3946631640 1 -- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb597e0 msgr2=0x5590dfb8a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3946631640 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb597e0 0x5590dfb8a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3946631640 1 -- 172.21.15.46:0/3872390407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590dfb89130 con 0x5590dfc33810 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f394562f640 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfb8b7b0 0x5590dfc33c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3946631640 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 0x5590dfc34160 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f394000a550 tx=0x7f394000a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23237a454fd34e49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3937fff640 1 -- 172.21.15.46:0/3872390407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940019070 con 0x5590dfc33810 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3937fff640 1 -- 172.21.15.46:0/3872390407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3940007600 con 0x5590dfc33810 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.255+0000 7f3937fff640 1 -- 172.21.15.46:0/3872390407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39400079d0 con 0x5590dfc33810 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 -- 172.21.15.46:0/3872390407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 msgr2=0x5590dfc34160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 0x5590dfc34160 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f394000a550 tx=0x7f394000a640 comp rx=0 tx=0).stop 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 -- 172.21.15.46:0/3872390407 shutdown_connections 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfc33810 0x5590dfc34160 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfb8b7b0 0x5590dfc33c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 --2- 172.21.15.46:0/3872390407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb597e0 0x5590dfb8a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 -- 172.21.15.46:0/3872390407 >> 172.21.15.46:0/3872390407 conn(0x5590dfa7eac0 msgr2=0x5590dfba3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 -- 172.21.15.46:0/3872390407 shutdown_connections 2024-05-09T03:17:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.256+0000 7f394c366f40 1 -- 172.21.15.46:0/3872390407 wait complete. 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.257+0000 7f394c366f40 1 Processor -- start 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.257+0000 7f394c366f40 1 -- start start 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394c366f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfb597e0 0x5590dfc1c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394c366f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb8b7b0 0x5590dfc32da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394c366f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfc33810 0x5590dfc2bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394c366f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5590dfb9dcb0 con 0x5590dfb597e0 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394c366f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5590dfcca5c0 con 0x5590dfb8b7b0 2024-05-09T03:17:28.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394c366f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5590dfc2d610 con 0x5590dfc33810 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfb597e0 0x5590dfc1c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfb597e0 0x5590dfc1c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55280/0 (socket says 172.21.15.46:55280) 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 learned_addr learned my addr 172.21.15.46:0/1267310921 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3946631640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfc33810 0x5590dfc2bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb8b7b0 0x5590dfc32da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfc33810 msgr2=0x5590dfc2bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dfc33810 0x5590dfc2bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb8b7b0 msgr2=0x5590dfc32da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dfb8b7b0 0x5590dfc32da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.258+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590dfc34cd0 con 0x5590dfb597e0 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.259+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfb597e0 0x5590dfc1c760 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f3938006f20 tx=0x7f3938001850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.259+0000 7f3936ffd640 1 -- 172.21.15.46:0/1267310921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3938018070 con 0x5590dfb597e0 2024-05-09T03:17:28.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.259+0000 7f3936ffd640 1 -- 172.21.15.46:0/1267310921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39380060d0 con 0x5590dfb597e0 2024-05-09T03:17:28.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.259+0000 7f3936ffd640 1 -- 172.21.15.46:0/1267310921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39380044c0 con 0x5590dfb597e0 2024-05-09T03:17:28.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.259+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5590dfc2d8f0 con 0x5590dfb597e0 2024-05-09T03:17:28.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.259+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590dfc2deb0 con 0x5590dfb597e0 2024-05-09T03:17:28.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.261+0000 7f3936ffd640 1 -- 172.21.15.46:0/1267310921 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f393801b070 con 0x5590dfb597e0 2024-05-09T03:17:28.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.261+0000 7f3936ffd640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3928074cb0 0x7f39280771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.261+0000 7f3936ffd640 1 -- 172.21.15.46:0/1267310921 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7f3938095b80 con 0x5590dfb597e0 2024-05-09T03:17:28.264 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.261+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5590dfcc9460 0x5590dfcc9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.261+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcc9db0 con 0x5590dfcc9460 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.261+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3928074cb0 0x7f39280771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.262+0000 7f3946631640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5590dfcc9460 0x5590dfcc9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.262+0000 7f3946631640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5590dfcc9460 0x5590dfcc9870 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.262+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3928074cb0 0x7f39280771c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f393c056000 tx=0x7f393c050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.264+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v95'319 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.264+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcd5a70 con 0x5590dfcc9460 2024-05-09T03:17:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.265+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v95'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.265+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd5e10 con 0x5590dfcc9460 2024-05-09T03:17:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.267+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v95'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.267 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:28.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.267+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd71c0 con 0x5590dfcc9460 2024-05-09T03:17:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.268+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v95'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.268+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd7560 con 0x5590dfcc9460 2024-05-09T03:17:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.269+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v95'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.269+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd61b0 con 0x5590dfcc9460 2024-05-09T03:17:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.271+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v95'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.271+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd65c0 con 0x5590dfcc9460 2024-05-09T03:17:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.272+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v95'325 uv299 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.272+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcd65c0 con 0x5590dfcc9460 2024-05-09T03:17:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.273+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v95'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.273+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.275+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v95'327 uv327 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.275 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:28.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.275+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.276+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v95'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.276+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.278+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v95'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.278+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.279+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v95'330 uv330 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.279+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.280+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v95'331 uv306 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.280+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.282+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v95'332 uv332 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.282+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.283+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v95'333 uv333 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.283 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:28.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.283+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.284+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v95'334 uv334 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.284+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.285+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v95'335 uv335 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.285+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.287+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v95'336 uv336 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.287+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.288+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v95'337 uv337 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.288+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.289+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v95'338 uv312 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.289+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.290+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v95'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.290+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.292 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:28.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.291+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v95'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.291+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.292+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v95'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.293+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.294+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v95'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.294+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.295+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v95'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.295+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.296+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v95'344 uv318 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.296+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.297+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v95'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.297+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.299 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:28.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.298+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v95'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.298+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.300+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v95'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.300+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.301+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v95'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.301+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcc9460 2024-05-09T03:17:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.302+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v95'349 uv349 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.302+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5590dfcd7900 0x5590dfcd9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.302+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcd6550 con 0x5590dfcd7900 2024-05-09T03:17:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.302+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5590dfcd7900 0x5590dfcd9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.303+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5590dfcd7900 0x5590dfcd9d20 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.304+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v95'239 uv221 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.304+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcd6e20 con 0x5590dfcd7900 2024-05-09T03:17:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.305+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v95'240 uv240 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.305+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcda260 con 0x5590dfcd7900 2024-05-09T03:17:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.307+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v95'241 uv241 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.307 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:28.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.307+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdaab0 con 0x5590dfcd7900 2024-05-09T03:17:28.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.308+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v95'242 uv242 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.308+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdae50 con 0x5590dfcd7900 2024-05-09T03:17:28.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.309+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v95'243 uv243 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.309+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb1f0 con 0x5590dfcd7900 2024-05-09T03:17:28.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.310+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v95'244 uv226 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.310+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcdb590 con 0x5590dfcd7900 2024-05-09T03:17:28.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.311+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v95'245 uv245 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.311+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.312+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v95'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.313 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:28.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.312+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.314+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v95'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.314+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.315+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v95'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.315+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.316+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v95'249 uv231 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.316+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.317+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v95'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.317+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.318+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v95'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.322 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.318+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdf390 con 0x5590dfcd7900 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.319+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v95'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.320+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdf390 con 0x5590dfcd7900 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.321+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v95'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.321+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.322+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v95'254 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.322+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.323+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v95'255 uv255 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.323+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.324+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v95'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.325 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:28.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.324+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdc940 con 0x5590dfcd7900 2024-05-09T03:17:28.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.325+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v95'257 uv257 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.325+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdc940 con 0x5590dfcd7900 2024-05-09T03:17:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.326+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v95'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.326+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcd7900 2024-05-09T03:17:28.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.327+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v95'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfcd7900 2024-05-09T03:17:28.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.327+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcc9460 2024-05-09T03:17:28.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.328+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v95'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.328+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcc9460 2024-05-09T03:17:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.329+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v95'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.329+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcc9460 2024-05-09T03:17:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.330+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v95'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.331 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:28.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.330+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdc940 con 0x5590dfcc9460 2024-05-09T03:17:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.331+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v95'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.331+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdc940 con 0x5590dfcc9460 2024-05-09T03:17:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.332+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v95'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.333+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdb930 con 0x5590dfcc9460 2024-05-09T03:17:28.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.334+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v95'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.334+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5590dfcdc200 0x5590dfcde620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.334+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdeb60 con 0x5590dfcdc200 2024-05-09T03:17:28.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.334+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5590dfcdc200 0x5590dfcde620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.334+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5590dfcdc200 0x5590dfcde620 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.335+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v95'52 uv51 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.335+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcdef80 con 0x5590dfcdc200 2024-05-09T03:17:28.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.336+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v95'53 uv53 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.336+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdf480 con 0x5590dfcdc200 2024-05-09T03:17:28.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.337+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v95'54 uv54 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.338 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:28.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.337+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcda6e0 con 0x5590dfcdc200 2024-05-09T03:17:28.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.339+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v95'55 uv55 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.339+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdfdb0 con 0x5590dfcc9460 2024-05-09T03:17:28.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.340+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v95'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.340+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcdfdb0 con 0x5590dfcc9460 2024-05-09T03:17:28.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.341+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v95'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.341+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdfdb0 con 0x5590dfcc9460 2024-05-09T03:17:28.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.345+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v95'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.345 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:28.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.345+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce0dc0 con 0x5590dfcc9460 2024-05-09T03:17:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.347+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v95'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.347+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce0dc0 con 0x5590dfcc9460 2024-05-09T03:17:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.349+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v95'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.349+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcdfdb0 con 0x5590dfcc9460 2024-05-09T03:17:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.350+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v95'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.350+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dfce0680 0x5590dfce0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.350+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.350+0000 7f3946631640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dfce0680 0x5590dfce0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.351+0000 7f3946631640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dfce0680 0x5590dfce0a90 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.352+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v95'116 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.352+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.353+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v95'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.354+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.355 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:28.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.355+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v95'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.355+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce4290 con 0x5590dfce0680 2024-05-09T03:17:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.356+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v95'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.356+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce4290 con 0x5590dfce0680 2024-05-09T03:17:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.357+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v95'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.357+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.358+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v95'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.358+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.360+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v95'122 uv115 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.360+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.361+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v95'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.361+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.362 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.362+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v95'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.362+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.363+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v95'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.363+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce3280 con 0x5590dfce0680 2024-05-09T03:17:28.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.364+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v95'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.364+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5590dfce3bb0 0x5590dfce3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.364+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6660 con 0x5590dfce3bb0 2024-05-09T03:17:28.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.365+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5590dfce3bb0 0x5590dfce3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.365+0000 7f3945e30640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5590dfce3bb0 0x5590dfce3fc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.366+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v95'51 uv50 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfce3bb0 2024-05-09T03:17:28.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.366+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfce6660 con 0x5590dfce3bb0 2024-05-09T03:17:28.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.367+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v95'52 uv52 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfce3bb0 2024-05-09T03:17:28.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.367+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfce3bb0 2024-05-09T03:17:28.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.368+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v95'53 uv53 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfce3bb0 2024-05-09T03:17:28.369 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:28.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.368+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfce3bb0 2024-05-09T03:17:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.370+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v95'54 uv54 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfce3bb0 2024-05-09T03:17:28.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.370+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfce3bb0 2024-05-09T03:17:28.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.371+0000 7f3945e30640 1 -- 172.21.15.46:0/1267310921 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v95'55 uv55 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393801a030 con 0x5590dfce3bb0 2024-05-09T03:17:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.371+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfcc9460 2024-05-09T03:17:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.372+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v95'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.372+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfcc9460 2024-05-09T03:17:28.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.373+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v95'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.373+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfcc9460 2024-05-09T03:17:28.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.374+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v95'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.375 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:28.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.374+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7a60 con 0x5590dfcc9460 2024-05-09T03:17:28.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.375+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v95'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.375+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7a60 con 0x5590dfcc9460 2024-05-09T03:17:28.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.376+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v95'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.376+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfcc9460 2024-05-09T03:17:28.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.377+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v95'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfcc9460 2024-05-09T03:17:28.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.377+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce6a50 con 0x5590dfcdc200 2024-05-09T03:17:28.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.378+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v95'104 uv97 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.378+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfce6df0 con 0x5590dfcdc200 2024-05-09T03:17:28.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.380+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v95'105 uv105 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.380+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7190 con 0x5590dfcdc200 2024-05-09T03:17:28.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.381+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v95'106 uv106 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.381 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.381+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7530 con 0x5590dfcdc200 2024-05-09T03:17:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.382+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v95'107 uv107 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.382+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7530 con 0x5590dfcdc200 2024-05-09T03:17:28.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.383+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v95'108 uv108 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.383+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7530 con 0x5590dfcdc200 2024-05-09T03:17:28.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.384+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v95'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.385 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:28.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.384+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfce7530 con 0x5590dfcdc200 2024-05-09T03:17:28.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.385+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [omap-set-header] v95'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.385+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea340 con 0x5590dfcdc200 2024-05-09T03:17:28.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.386+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v95'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.386+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5590dfce8060 0x5590dfcea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.386+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce8060 2024-05-09T03:17:28.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.387+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5590dfce8060 0x5590dfcea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.387+0000 7f394562f640 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5590dfce8060 0x5590dfcea9e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.388+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(99 mds0_sessionmap [create] v95'49 uv47 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfce8060 2024-05-09T03:17:28.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.388+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce8060 2024-05-09T03:17:28.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.389+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v95'50 uv50 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfce8060 2024-05-09T03:17:28.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.389+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce8060 2024-05-09T03:17:28.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.390+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(101 mds0_sessionmap [create] v95'51 uv51 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfce8060 2024-05-09T03:17:28.391 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:28.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.390+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce8060 2024-05-09T03:17:28.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.391+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v95'52 uv52 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfce8060 2024-05-09T03:17:28.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.391+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfcdc200 2024-05-09T03:17:28.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.393+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(103 609.00000000 [create] v95'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.393+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfcdc200 2024-05-09T03:17:28.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.394+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(104 609.00000000 [delete] v95'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.394+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfcdc200 2024-05-09T03:17:28.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.395+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v95'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.396 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:28.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.395+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfcdc200 2024-05-09T03:17:28.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.396+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v95'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.396+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfcdc200 2024-05-09T03:17:28.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.397+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v95'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.397+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfcdc200 2024-05-09T03:17:28.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.399+0000 7f394562f640 1 -- 172.21.15.46:0/1267310921 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [omap-set-header] v95'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f393c05c040 con 0x5590dfcdc200 2024-05-09T03:17:28.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.399+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce0680 2024-05-09T03:17:28.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.400+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v95'52 uv51 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.400+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce0680 2024-05-09T03:17:28.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.401+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v95'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.401+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcea480 con 0x5590dfce0680 2024-05-09T03:17:28.403 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:28.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.403+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v95'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.403+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.404+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v95'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.404+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.405+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v95'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.405+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.406+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v95'132 uv123 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.407+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.408+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v95'133 uv133 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.408+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.409+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v95'134 uv134 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.409 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:28.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.409+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.410+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v95'135 uv135 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.410+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.411+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v95'136 uv136 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.411+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.412+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v95'137 uv130 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.412+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.413+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v95'138 uv138 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.413+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.414+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v95'139 uv139 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.415 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:28.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.414+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcebf30 con 0x5590dfce0680 2024-05-09T03:17:28.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.415+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v95'140 uv140 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.415+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfcebf30 con 0x5590dfce0680 2024-05-09T03:17:28.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.417+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v95'141 uv141 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.417+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceaf20 con 0x5590dfce0680 2024-05-09T03:17:28.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.418+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v95'142 uv142 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.418+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x5590dfceb3d0 con 0x5590dfce0680 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v95'143 uv143 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f39400078b0 con 0x5590dfce0680 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5590dfcc9460 msgr2=0x5590dfcc9870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5590dfcc9460 0x5590dfcc9870 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5590dfcdc200 msgr2=0x5590dfcde620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5590dfcdc200 0x5590dfcde620 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5590dfce8060 msgr2=0x5590dfcea9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5590dfce8060 0x5590dfcea9e0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5590dfcd7900 msgr2=0x5590dfcd9d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 reap_dead start 2024-05-09T03:17:28.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5590dfcd7900 0x5590dfcd9d20 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dfce0680 msgr2=0x5590dfce0a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dfce0680 0x5590dfce0a90 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5590dfce3bb0 msgr2=0x5590dfce3fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5590dfce3bb0 0x5590dfce3fc0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3928074cb0 msgr2=0x7f39280771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3928074cb0 0x7f39280771c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f393c056000 tx=0x7f393c050000 comp rx=0 tx=0).stop 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfb597e0 msgr2=0x5590dfc1c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 --2- 172.21.15.46:0/1267310921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dfb597e0 0x5590dfc1c760 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f3938006f20 tx=0x7f3938001850 comp rx=0 tx=0).stop 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f3946631640 1 -- 172.21.15.46:0/1267310921 reap_dead start 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 shutdown_connections 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 >> 172.21.15.46:0/1267310921 conn(0x5590dfa7eac0 msgr2=0x5590dfba4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:28.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 shutdown_connections 2024-05-09T03:17:28.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.420+0000 7f394c366f40 1 -- 172.21.15.46:0/1267310921 wait complete. 2024-05-09T03:17:28.425 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_sessionmap 2024-05-09T03:17:28.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.519+0000 7fac6a01ef40 1 Processor -- start 2024-05-09T03:17:28.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.519+0000 7fac6a01ef40 1 -- start start 2024-05-09T03:17:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac6a01ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 0x561b8b53ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac6a01ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b53e780 0x561b8b53f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac6a01ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b498100 0x561b8b492a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac6a01ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561b8b4a8cb0 con 0x561b8b495db0 2024-05-09T03:17:28.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac6a01ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561b8b40ff50 con 0x561b8b498100 2024-05-09T03:17:28.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac6a01ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561b8b4a8380 con 0x561b8b53e780 2024-05-09T03:17:28.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.520+0000 7fac62e74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 0x561b8b53ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 0x561b8b53ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55312/0 (socket says 172.21.15.46:55312) 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 -- 172.21.15.46:0/1730261171 learned_addr learned my addr 172.21.15.46:0/1730261171 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 -- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b53e780 msgr2=0x561b8b53f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac5bfff640 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b53e780 0x561b8b53f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac63675640 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b498100 0x561b8b492a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b53e780 0x561b8b53f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 -- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b498100 msgr2=0x561b8b492a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b498100 0x561b8b492a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 -- 172.21.15.46:0/1730261171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b8b4a9af0 con 0x561b8b495db0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac63675640 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b498100 0x561b8b492a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac62e74640 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 0x561b8b53ba00 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fac54009720 tx=0x7fac54005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9aecdbe5fd36458 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac61e72640 1 -- 172.21.15.46:0/1730261171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54019070 con 0x561b8b495db0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac61e72640 1 -- 172.21.15.46:0/1730261171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac54008890 con 0x561b8b495db0 2024-05-09T03:17:28.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.521+0000 7fac61e72640 1 -- 172.21.15.46:0/1730261171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54008c60 con 0x561b8b495db0 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 -- 172.21.15.46:0/1730261171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 msgr2=0x561b8b53ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 0x561b8b53ba00 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fac54009720 tx=0x7fac54005a00 comp rx=0 tx=0).stop 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 -- 172.21.15.46:0/1730261171 shutdown_connections 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b498100 0x561b8b492a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b53e780 0x561b8b53f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/1730261171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b495db0 0x561b8b53ba00 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 -- 172.21.15.46:0/1730261171 >> 172.21.15.46:0/1730261171 conn(0x561b8b38bac0 msgr2=0x561b8b497800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 -- 172.21.15.46:0/1730261171 shutdown_connections 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.522+0000 7fac6a01ef40 1 -- 172.21.15.46:0/1730261171 wait complete. 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.523+0000 7fac6a01ef40 1 Processor -- start 2024-05-09T03:17:28.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.523+0000 7fac6a01ef40 1 -- start start 2024-05-09T03:17:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac6a01ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b495db0 0x561b8b520d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac6a01ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b498100 0x561b8b4b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac6a01ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b53e780 0x561b8b4aba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac6a01ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561b8b53d160 con 0x561b8b498100 2024-05-09T03:17:28.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac6a01ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561b8b498ae0 con 0x561b8b53e780 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac6a01ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561b8b4b2ac0 con 0x561b8b495db0 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b498100 0x561b8b4b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b498100 0x561b8b4b2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55328/0 (socket says 172.21.15.46:55328) 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac5bfff640 1 -- 172.21.15.46:0/179958794 learned_addr learned my addr 172.21.15.46:0/179958794 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac62e74640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b495db0 0x561b8b520d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.524+0000 7fac63675640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b53e780 0x561b8b4aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac5bfff640 1 -- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b495db0 msgr2=0x561b8b520d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac5bfff640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561b8b495db0 0x561b8b520d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac5bfff640 1 -- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b53e780 msgr2=0x561b8b4aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac5bfff640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561b8b53e780 0x561b8b4aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac5bfff640 1 -- 172.21.15.46:0/179958794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b8b4b1520 con 0x561b8b498100 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac5bfff640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b498100 0x561b8b4b2580 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x561b8b46b690 tx=0x7fac4c00a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac60e70640 1 -- 172.21.15.46:0/179958794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c008040 con 0x561b8b498100 2024-05-09T03:17:28.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac60e70640 1 -- 172.21.15.46:0/179958794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c014020 con 0x561b8b498100 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561b8b4b17c0 con 0x561b8b498100 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac60e70640 1 -- 172.21.15.46:0/179958794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00f400 con 0x561b8b498100 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.525+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b8b4b1d80 con 0x561b8b498100 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.527+0000 7fac60e70640 1 -- 172.21.15.46:0/179958794 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fac4c00f5a0 con 0x561b8b498100 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.527+0000 7fac60e70640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fac44074ce0 0x7fac440771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.527+0000 7fac60e70640 1 -- 172.21.15.46:0/179958794 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7fac4c096380 con 0x561b8b498100 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.527+0000 7fac62e74640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fac44074ce0 0x7fac440771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.528+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x561b8b4ab5c0 0x561b8b538800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.528+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e95) -- 0x561b8b538d40 con 0x561b8b4ab5c0 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.528+0000 7fac63675640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x561b8b4ab5c0 0x561b8b538800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.528+0000 7fac62e74640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fac44074ce0 0x7fac440771f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fac54000c00 tx=0x7fac54008000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.528+0000 7fac63675640 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x561b8b4ab5c0 0x561b8b538800 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.529+0000 7fac63675640 1 -- 172.21.15.46:0/179958794 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 mds0_sessionmap [delete] v95'53 uv53 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fac5c053800 con 0x561b8b4ab5c0 2024-05-09T03:17:28.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.529+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x561b8b4ab5c0 msgr2=0x561b8b538800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.529+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x561b8b4ab5c0 0x561b8b538800 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fac44074ce0 msgr2=0x7fac440771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fac44074ce0 0x7fac440771f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fac54000c00 tx=0x7fac54008000 comp rx=0 tx=0).stop 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b498100 msgr2=0x561b8b4b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 --2- 172.21.15.46:0/179958794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b8b498100 0x561b8b4b2580 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x561b8b46b690 tx=0x7fac4c00a5d0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac63675640 1 -- 172.21.15.46:0/179958794 reap_dead start 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 shutdown_connections 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 >> 172.21.15.46:0/179958794 conn(0x561b8b38bac0 msgr2=0x561b8b493ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 shutdown_connections 2024-05-09T03:17:28.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.530+0000 7fac6a01ef40 1 -- 172.21.15.46:0/179958794 wait complete. 2024-05-09T03:17:28.535 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.652+0000 7f7e86e15640 1 Processor -- start 2024-05-09T03:17:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.652+0000 7f7e86e15640 1 -- start start 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.653+0000 7f7e86e15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e800b7e70 0x7f7e800b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.653+0000 7f7e86e15640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e801605b0 0x7f7e800b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.653+0000 7f7e86e15640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e800d3400 con 0x7f7e800d2e90 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.653+0000 7f7e86e15640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7e800d4a60 con 0x7f7e800b7e70 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.653+0000 7f7e86e15640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7e800d4d40 con 0x7f7e801605b0 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e800b7e70 0x7f7e800b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e800b7e70 0x7f7e800b92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55334/0 (socket says 172.21.15.46:55334) 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 -- 172.21.15.46:0/2042439070 learned_addr learned my addr 172.21.15.46:0/2042439070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e84bb1640 1 --1- 172.21.15.46:0/2042439070 >> v1:172.21.15.46:6789/0 conn(0x7f7e800d2e90 0x7f7e800b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41022/0 (socket says 172.21.15.46:41022) 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e853b2640 1 --2- 172.21.15.46:0/2042439070 >> v2:172.21.15.88:3300/0 conn(0x7f7e801605b0 0x7f7e800b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 -- 172.21.15.46:0/2042439070 >> v2:172.21.15.88:3300/0 conn(0x7f7e801605b0 msgr2=0x7f7e800b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 --2- 172.21.15.46:0/2042439070 >> v2:172.21.15.88:3300/0 conn(0x7f7e801605b0 0x7f7e800b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 -- 172.21.15.46:0/2042439070 >> v1:172.21.15.46:6789/0 conn(0x7f7e800d2e90 legacy=0x7f7e800b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7ffff640 1 -- 172.21.15.46:0/2042439070 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e800d5020 con 0x7f7e800b7e70 2024-05-09T03:17:28.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.654+0000 7f7e7f7fe640 1 -- 172.21.15.46:0/2042439070 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316680097 0 0) 0x7f7e800d3400 con 0x7f7e800d2e90 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.655+0000 7f7e7ffff640 1 --2- 172.21.15.46:0/2042439070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800b7e70 0x7f7e800b92a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7e800c95f0 tx=0x7f7e7402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464a4138d0d81287 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.655+0000 7f7e7f7fe640 1 -- 172.21.15.46:0/2042439070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74037070 con 0x7f7e800b7e70 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.655+0000 7f7e7f7fe640 1 -- 172.21.15.46:0/2042439070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e74032440 con 0x7f7e800b7e70 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.655+0000 7f7e7f7fe640 1 -- 172.21.15.46:0/2042439070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74032810 con 0x7f7e800b7e70 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 -- 172.21.15.46:0/2042439070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800b7e70 msgr2=0x7f7e800b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 --2- 172.21.15.46:0/2042439070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800b7e70 0x7f7e800b92a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7e800c95f0 tx=0x7f7e7402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 -- 172.21.15.46:0/2042439070 shutdown_connections 2024-05-09T03:17:28.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 --2- 172.21.15.46:0/2042439070 >> v2:172.21.15.88:3300/0 conn(0x7f7e801605b0 0x7f7e800b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 --2- 172.21.15.46:0/2042439070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800b7e70 0x7f7e800b92a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 -- 172.21.15.46:0/2042439070 >> 172.21.15.46:0/2042439070 conn(0x7f7e8008d390 msgr2=0x7f7e8008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 -- 172.21.15.46:0/2042439070 shutdown_connections 2024-05-09T03:17:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.656+0000 7f7e86e15640 1 -- 172.21.15.46:0/2042439070 wait complete. 2024-05-09T03:17:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.657+0000 7f7e86e15640 1 Processor -- start 2024-05-09T03:17:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.657+0000 7f7e86e15640 1 -- start start 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e86e15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 0x7f7e801494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e86e15640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e801605b0 0x7f7e80156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e86e15640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e80153730 0x7f7e80161460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e86e15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e800b4d80 con 0x7f7e800d2e90 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e86e15640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e800b4c00 con 0x7f7e801605b0 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e86e15640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7e800b4fc0 con 0x7f7e80153730 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 0x7f7e801494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 0x7f7e801494c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55342/0 (socket says 172.21.15.46:55342) 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 -- 172.21.15.46:0/4236125207 learned_addr learned my addr 172.21.15.46:0/4236125207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e7ffff640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e801605b0 0x7f7e80156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e853b2640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e80153730 0x7f7e80161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 -- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e80153730 msgr2=0x7f7e80161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e80153730 0x7f7e80161460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 -- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e801605b0 msgr2=0x7f7e80156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e801605b0 0x7f7e80156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e84bb1640 1 -- 172.21.15.46:0/4236125207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e801468a0 con 0x7f7e800d2e90 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.659+0000 7f7e7ffff640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e801605b0 0x7f7e80156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.660+0000 7f7e84bb1640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 0x7f7e801494c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c390 tx=0x7f7e7000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.660+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e700074e0 con 0x7f7e800d2e90 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.660+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e70007680 con 0x7f7e800d2e90 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.660+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e801576d0 con 0x7f7e800d2e90 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.660+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e70004a20 con 0x7f7e800d2e90 2024-05-09T03:17:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.660+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e80157c90 con 0x7f7e800d2e90 2024-05-09T03:17:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.661+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e80146a70 con 0x7f7e800d2e90 2024-05-09T03:17:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.662+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7e7001a020 con 0x7f7e800d2e90 2024-05-09T03:17:28.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.662+0000 7f7e7d7fa640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e6c0762d0 0x7f7e6c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:28.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.662+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6713+0+0 (secure 0 0 0) 0x7f7e700970b0 con 0x7f7e800d2e90 2024-05-09T03:17:28.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.662+0000 7f7e7ffff640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e6c0762d0 0x7f7e6c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:28.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.663+0000 7f7e7ffff640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e6c0762d0 0x7f7e6c0787e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7e74034000 tx=0x7f7e7402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:28.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.667+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7e7005ff40 con 0x7f7e800d2e90 2024-05-09T03:17:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:28.899+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f7e8006a270 con 0x7f7e800d2e90 2024-05-09T03:17:29.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.189+0000 7f7e7d7fa640 1 -- 172.21.15.46:0/4236125207 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v156) ==== 162+0+0 (secure 0 0 0) 0x7f7e70064240 con 0x7f7e800d2e90 2024-05-09T03:17:29.190 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:29.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e6c0762d0 msgr2=0x7f7e6c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e6c0762d0 0x7f7e6c0787e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7e74034000 tx=0x7f7e7402e000 comp rx=0 tx=0).stop 2024-05-09T03:17:29.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 msgr2=0x7f7e801494c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 0x7f7e801494c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7e7000c390 tx=0x7f7e7000c850 comp rx=0 tx=0).stop 2024-05-09T03:17:29.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 shutdown_connections 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e80153730 0x7f7e80161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e801605b0 0x7f7e80156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e6c0762d0 0x7f7e6c0787e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 --2- 172.21.15.46:0/4236125207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e800d2e90 0x7f7e801494c0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.191+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 >> 172.21.15.46:0/4236125207 conn(0x7f7e8008d390 msgr2=0x7f7e800b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.192+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 shutdown_connections 2024-05-09T03:17:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.192+0000 7f7e86e15640 1 -- 172.21.15.46:0/4236125207 wait complete. 2024-05-09T03:17:29.197 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:29.197+0000 7f73f3938640 -1 mds.0.sessionmap _load_finish got (2) No such file or directory 2024-05-09T03:17:29.198 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:29.197+0000 7f73f3938640 -1 log_channel(cluster) log [ERR] : error reading sessionmap 'mds0_sessionmap' -2 ((2) No such file or directory) 2024-05-09T03:17:29.198 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:29.197+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:29.198 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:29.197+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:29.204 INFO:tasks.ceph.mds.c.smithi046.stderr: -28> 2024-05-09T03:17:29.197+0000 7f73f3938640 -1 mds.0.sessionmap _load_finish got (2) No such file or directory 2024-05-09T03:17:29.204 INFO:tasks.ceph.mds.c.smithi046.stderr: -27> 2024-05-09T03:17:29.197+0000 7f73f3938640 -1 log_channel(cluster) log [ERR] : error reading sessionmap 'mds0_sessionmap' -2 ((2) No such file or directory) 2024-05-09T03:17:29.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:29.260 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:29.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.325+0000 7fbe88ab8640 1 Processor -- start 2024-05-09T03:17:29.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.325+0000 7fbe88ab8640 1 -- start start 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe88ab8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840d1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe88ab8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe8408d880 0x7fbe840d24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe88ab8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe8406ebe0 con 0x7fbe840b9310 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe88ab8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbe8406f760 con 0x7fbe8408d880 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe88ab8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbe8406eec0 con 0x7fbe8408d470 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe82d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbe8408d880 0x7fbe840d24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe8259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe840b9310 0x7fbe8408cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41034/0 (socket says 172.21.15.46:41034) 2024-05-09T03:17:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe8259c640 1 -- 172.21.15.46:0/1939229695 learned_addr learned my addr 172.21.15.46:0/1939229695 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe81d9b640 1 --2- 172.21.15.46:0/1939229695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840d1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe82d9d640 1 -- 172.21.15.46:0/1939229695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 msgr2=0x7fbe840d1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe82d9d640 1 --2- 172.21.15.46:0/1939229695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840d1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe82d9d640 1 -- 172.21.15.46:0/1939229695 >> v1:172.21.15.46:6789/0 conn(0x7fbe840b9310 legacy=0x7fbe8408cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.327+0000 7fbe82d9d640 1 -- 172.21.15.46:0/1939229695 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8406f1a0 con 0x7fbe8408d880 2024-05-09T03:17:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.328+0000 7fbe82d9d640 1 --2- 172.21.15.46:0/1939229695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840d24a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009d90 tx=0x7fbe7c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8074159f08a0ec44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.328+0000 7fbe8159a640 1 -- 172.21.15.46:0/1939229695 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c007740 con 0x7fbe8408d880 2024-05-09T03:17:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.328+0000 7fbe8159a640 1 -- 172.21.15.46:0/1939229695 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7c0078e0 con 0x7fbe8408d880 2024-05-09T03:17:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.328+0000 7fbe8159a640 1 -- 172.21.15.46:0/1939229695 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c007cb0 con 0x7fbe8408d880 2024-05-09T03:17:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1939229695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 msgr2=0x7fbe840d24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1939229695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840d24a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009d90 tx=0x7fbe7c02a660 comp rx=0 tx=0).stop 2024-05-09T03:17:29.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1939229695 shutdown_connections 2024-05-09T03:17:29.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1939229695 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840d24a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1939229695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840d1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1939229695 >> 172.21.15.46:0/1939229695 conn(0x7fbe840b5660 msgr2=0x7fbe840b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:29.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1939229695 shutdown_connections 2024-05-09T03:17:29.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.329+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1939229695 wait complete. 2024-05-09T03:17:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.331+0000 7fbe88ab8640 1 Processor -- start 2024-05-09T03:17:29.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.331+0000 7fbe88ab8640 1 -- start start 2024-05-09T03:17:29.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe88ab8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840b7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe88ab8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe88ab8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe840b9310 0x7fbe840b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe88ab8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe840d2b60 con 0x7fbe8408d880 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe88ab8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbe840d29e0 con 0x7fbe840b9310 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe88ab8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbe840d2ce0 con 0x7fbe8408d470 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840b8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55370/0 (socket says 172.21.15.46:55370) 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 -- 172.21.15.46:0/1455525391 learned_addr learned my addr 172.21.15.46:0/1455525391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe8259c640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840b7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe82d9d640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe840b9310 0x7fbe840b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 -- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 msgr2=0x7fbe840b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840b7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 -- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe840b9310 msgr2=0x7fbe840b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe840b9310 0x7fbe840b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 -- 172.21.15.46:0/1455525391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe841e1a70 con 0x7fbe8408d880 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe81d9b640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840b8080 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e8a0 tx=0x7fbe7800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.333+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78017070 con 0x7fbe8408d880 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.334+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7800ee90 con 0x7fbe8408d880 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.334+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe841e1d50 con 0x7fbe8408d880 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.334+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800cb40 con 0x7fbe8408d880 2024-05-09T03:17:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.334+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe841e2310 con 0x7fbe8408d880 2024-05-09T03:17:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.335+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe841e17e0 con 0x7fbe8408d880 2024-05-09T03:17:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.335+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbe7801a070 con 0x7fbe8408d880 2024-05-09T03:17:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.336+0000 7fbe72ffd640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe64076300 0x7fbe64078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.336+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fbe78097720 con 0x7fbe8408d880 2024-05-09T03:17:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.336+0000 7fbe8259c640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe64076300 0x7fbe64078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.337+0000 7fbe8259c640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe64076300 0x7fbe64078810 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000f60 tx=0x7fbe74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.341+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbe78060680 con 0x7fbe8408d880 2024-05-09T03:17:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.570+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe8406a270 con 0x7fbe8408d880 2024-05-09T03:17:29.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.571+0000 7fbe72ffd640 1 -- 172.21.15.46:0/1455525391 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+4342 (secure 0 0 0) 0x7fbe78064980 con 0x7fbe8408d880 2024-05-09T03:17:29.572 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:29.572 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:29.198274+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe64076300 msgr2=0x7fbe64078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe64076300 0x7fbe64078810 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fbe74000f60 tx=0x7fbe74009040 comp rx=0 tx=0).stop 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 msgr2=0x7fbe840b8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840b8080 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800e8a0 tx=0x7fbe7800ed60 comp rx=0 tx=0).stop 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 shutdown_connections 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe840b9310 0x7fbe840b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe64076300 0x7fbe64078810 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe8408d880 0x7fbe840b8080 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 --2- 172.21.15.46:0/1455525391 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe8408d470 0x7fbe840b7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 >> 172.21.15.46:0/1455525391 conn(0x7fbe840b5660 msgr2=0x7fbe84072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 shutdown_connections 2024-05-09T03:17:29.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.573+0000 7fbe88ab8640 1 -- 172.21.15.46:0/1455525391 wait complete. 2024-05-09T03:17:29.585 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:29.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.698+0000 7f34b8a13640 1 Processor -- start 2024-05-09T03:17:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.698+0000 7f34b8a13640 1 -- start start 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b8a13640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b41613c0 0x7f34b40b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b8a13640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b40b8110 con 0x7f34b40b5210 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b8a13640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b41617d0 con 0x7f34b40b7d00 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b8a13640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34b40b5620 con 0x7f34b41613c0 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34b40b5210 0x7f34b40d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41050/0 (socket says 172.21.15.46:41050) 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b259c640 1 -- 172.21.15.46:0/2255704068 learned_addr learned my addr 172.21.15.46:0/2255704068 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b1d9b640 1 --2- 172.21.15.46:0/2255704068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b41613c0 0x7f34b40b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4206448978 0 0) 0x7f34a4001110 con 0x7f34b40b5210 2024-05-09T03:17:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f349c003520 con 0x7f34b40b5210 2024-05-09T03:17:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709476937 0 0) 0x7f34b41617d0 con 0x7f34b40b7d00 2024-05-09T03:17:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34a4001110 con 0x7f34b40b7d00 2024-05-09T03:17:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b1d9b640 1 -- 172.21.15.46:0/2255704068 >> v1:172.21.15.88:6789/0 conn(0x7f34b40b7d00 legacy=0x7f34b4163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b1d9b640 1 -- 172.21.15.46:0/2255704068 >> v1:172.21.15.46:6789/0 conn(0x7f34b40b5210 legacy=0x7f34b40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.700+0000 7f34b1d9b640 1 -- 172.21.15.46:0/2255704068 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b40d5280 con 0x7f34b41613c0 2024-05-09T03:17:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.701+0000 7f34b1d9b640 1 --2- 172.21.15.46:0/2255704068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b41613c0 0x7f34b40b77c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f34a80097d0 tx=0x7f34a802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a915ae2d78685c48 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.701+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8037070 con 0x7f34b41613c0 2024-05-09T03:17:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.701+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a8032440 con 0x7f34b41613c0 2024-05-09T03:17:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.701+0000 7f34b159a640 1 -- 172.21.15.46:0/2255704068 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8032810 con 0x7f34b41613c0 2024-05-09T03:17:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 -- 172.21.15.46:0/2255704068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b41613c0 msgr2=0x7f34b40b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 --2- 172.21.15.46:0/2255704068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b41613c0 0x7f34b40b77c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f34a80097d0 tx=0x7f34a802a210 comp rx=0 tx=0).stop 2024-05-09T03:17:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 -- 172.21.15.46:0/2255704068 shutdown_connections 2024-05-09T03:17:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 --2- 172.21.15.46:0/2255704068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b41613c0 0x7f34b40b77c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 -- 172.21.15.46:0/2255704068 >> 172.21.15.46:0/2255704068 conn(0x7f34b40d29a0 msgr2=0x7f34b40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 -- 172.21.15.46:0/2255704068 shutdown_connections 2024-05-09T03:17:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.702+0000 7f34b8a13640 1 -- 172.21.15.46:0/2255704068 wait complete. 2024-05-09T03:17:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.704+0000 7f34b8a13640 1 Processor -- start 2024-05-09T03:17:29.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.704+0000 7f34b8a13640 1 -- start start 2024-05-09T03:17:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.705+0000 7f34b8a13640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 0x7f34b40cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.705+0000 7f34b8a13640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b40b7d00 0x7f34b40cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b8a13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b41613c0 0x7f34b415d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b8a13640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34b4160bc0 con 0x7f34b40b5210 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b8a13640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f34b41609a0 con 0x7f34b41613c0 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b8a13640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34b4160ea0 con 0x7f34b40b7d00 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 0x7f34b40cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 0x7f34b40cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55378/0 (socket says 172.21.15.46:55378) 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 -- 172.21.15.46:0/1002543884 learned_addr learned my addr 172.21.15.46:0/1002543884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b1d9b640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b40b7d00 0x7f34b40cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b2d9d640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b41613c0 0x7f34b415d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 -- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b40b7d00 msgr2=0x7f34b40cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b40b7d00 0x7f34b40cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 -- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b41613c0 msgr2=0x7f34b415d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b41613c0 0x7f34b415d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 -- 172.21.15.46:0/1002543884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b40cf830 con 0x7f34b40b5210 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b259c640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 0x7f34b40cba10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c730 tx=0x7f34a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4017070 con 0x7f34b40b5210 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a4005ab0 con 0x7f34b40b5210 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b40cfb10 con 0x7f34b40b5210 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4007070 con 0x7f34b40b5210 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.706+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b415ced0 con 0x7f34b40b5210 2024-05-09T03:17:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.707+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b406a270 con 0x7f34b40b5210 2024-05-09T03:17:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.708+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34a401a070 con 0x7f34b40b5210 2024-05-09T03:17:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.709+0000 7f34a2ffd640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34940762d0 0x7f34940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.709+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f34a4097570 con 0x7f34b40b5210 2024-05-09T03:17:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.709+0000 7f34b1d9b640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34940762d0 0x7f34940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.709+0000 7f34b1d9b640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34940762d0 0x7f34940787e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f34a8034000 tx=0x7f34a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.713+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34a4021020 con 0x7f34b40b5210 2024-05-09T03:17:29.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.941+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34b40d40d0 con 0x7f34b40b5210 2024-05-09T03:17:29.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.942+0000 7f34a2ffd640 1 -- 172.21.15.46:0/1002543884 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+4342 (secure 0 0 0) 0x7f34b40b8110 con 0x7f34b40b5210 2024-05-09T03:17:29.942 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:29.943 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":158,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:29.198274+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:29.943 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 158 2024-05-09T03:17:29.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34940762d0 msgr2=0x7f34940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34940762d0 0x7f34940787e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f34a8034000 tx=0x7f34a802e000 comp rx=0 tx=0).stop 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 msgr2=0x7f34b40cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 0x7f34b40cba10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c730 tx=0x7f34a400cbf0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 shutdown_connections 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b41613c0 0x7f34b415d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34b40b7d00 0x7f34b40cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34940762d0 0x7f34940787e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 --2- 172.21.15.46:0/1002543884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34b40b5210 0x7f34b40cba10 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 >> 172.21.15.46:0/1002543884 conn(0x7f34b40d29a0 msgr2=0x7f34b415f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 shutdown_connections 2024-05-09T03:17:29.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:29.944+0000 7f34b8a13640 1 -- 172.21.15.46:0/1002543884 wait complete. 2024-05-09T03:17:29.956 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:29.956 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:30.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.070+0000 7f71ce669640 1 Processor -- start 2024-05-09T03:17:30.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.070+0000 7f71ce669640 1 -- start start 2024-05-09T03:17:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71ce669640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c8068a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71ce669640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c80b30a0 0x7f71c80d2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71ce669640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c8160a40 con 0x7f71c80b2c90 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71ce669640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f71c8068fa0 con 0x7f71c80b30a0 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71ce669640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80b34b0 con 0x7f71c8160630 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71ccc06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c80b30a0 0x7f71c80d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71bffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f71c80b2c90 0x7f71c80cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41058/0 (socket says 172.21.15.46:41058) 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.071+0000 7f71bffff640 1 -- 172.21.15.46:0/2900078553 learned_addr learned my addr 172.21.15.46:0/2900078553 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71c7fff640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c8068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71ccc06640 1 -- 172.21.15.46:0/2900078553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 msgr2=0x7f71c8068a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71ccc06640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c8068a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71ccc06640 1 -- 172.21.15.46:0/2900078553 >> v1:172.21.15.46:6789/0 conn(0x7f71c80b2c90 legacy=0x7f71c80cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71c77fe640 1 -- 172.21.15.46:0/2900078553 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635156512 0 0) 0x7f71b0001110 con 0x7f71c80b2c90 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71ccc06640 1 -- 172.21.15.46:0/2900078553 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c80b27b0 con 0x7f71c80b30a0 2024-05-09T03:17:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71c7fff640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c8068a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.072+0000 7f71ccc06640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c80d2060 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009d90 tx=0x7f71c002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31eb3be3cddd99d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71c77fe640 1 -- 172.21.15.46:0/2900078553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0007740 con 0x7f71c80b30a0 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71c77fe640 1 -- 172.21.15.46:0/2900078553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c00078e0 con 0x7f71c80b30a0 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71c77fe640 1 -- 172.21.15.46:0/2900078553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0007cb0 con 0x7f71c80b30a0 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 -- 172.21.15.46:0/2900078553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 msgr2=0x7f71c80d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c80d2060 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f71c0009d90 tx=0x7f71c002a5f0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 -- 172.21.15.46:0/2900078553 shutdown_connections 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c80d2060 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 --2- 172.21.15.46:0/2900078553 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c8068a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 -- 172.21.15.46:0/2900078553 >> 172.21.15.46:0/2900078553 conn(0x7f71c809b7d0 msgr2=0x7f71c80b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 -- 172.21.15.46:0/2900078553 shutdown_connections 2024-05-09T03:17:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.073+0000 7f71ce669640 1 -- 172.21.15.46:0/2900078553 wait complete. 2024-05-09T03:17:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.075+0000 7f71ce669640 1 Processor -- start 2024-05-09T03:17:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.075+0000 7f71ce669640 1 -- start start 2024-05-09T03:17:30.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.076+0000 7f71ce669640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71c80b2c90 0x7f71c8162e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c814cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c815d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f71c806cb00 con 0x7f71c80b30a0 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71c806c820 con 0x7f71c80b2c90 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80b78d0 con 0x7f71c8160630 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c814cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c814cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55396/0 (socket says 172.21.15.46:55396) 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 -- 172.21.15.46:0/1829647984 learned_addr learned my addr 172.21.15.46:0/1829647984 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71c7fff640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71c80b2c90 0x7f71c8162e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ccc06640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c815d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 -- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 msgr2=0x7f71c815d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c815d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 -- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71c80b2c90 msgr2=0x7f71c8162e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71c80b2c90 0x7f71c8162e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 -- 172.21.15.46:0/1829647984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c8163390 con 0x7f71c80b30a0 2024-05-09T03:17:30.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71bffff640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c814cc00 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f71c80b1730 tx=0x7f71b000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:30.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00074e0 con 0x7f71c80b30a0 2024-05-09T03:17:30.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b0007680 con 0x7f71c80b30a0 2024-05-09T03:17:30.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c8163660 con 0x7f71c80b30a0 2024-05-09T03:17:30.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0004a00 con 0x7f71c80b30a0 2024-05-09T03:17:30.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.077+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c8160c20 con 0x7f71c80b30a0 2024-05-09T03:17:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.078+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71c806a270 con 0x7f71c80b30a0 2024-05-09T03:17:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.079+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71b0011070 con 0x7f71c80b30a0 2024-05-09T03:17:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.080+0000 7f71c57fa640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71a8076300 0x7f71a8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.080+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f71b00975b0 con 0x7f71c80b30a0 2024-05-09T03:17:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.080+0000 7f71c7fff640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71a8076300 0x7f71a8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.081+0000 7f71c7fff640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71a8076300 0x7f71a8078810 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f71c80726a0 tx=0x7f71b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:30.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.085+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f71c8160a40 con 0x7f71c80b30a0 2024-05-09T03:17:30.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.314+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71c806ebf0 con 0x7f71c80b30a0 2024-05-09T03:17:30.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.314+0000 7f71c57fa640 1 -- 172.21.15.46:0/1829647984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+5168 (secure 0 0 0) 0x7f71b0060a40 con 0x7f71c80b30a0 2024-05-09T03:17:30.315 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:30.315 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":159,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8292,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486786164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486786164},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486786164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:29.198274+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:30.315 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 159 2024-05-09T03:17:30.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71a8076300 msgr2=0x7f71a8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71a8076300 0x7f71a8078810 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f71c80726a0 tx=0x7f71b8008040 comp rx=0 tx=0).stop 2024-05-09T03:17:30.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 msgr2=0x7f71c814cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c814cc00 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f71c80b1730 tx=0x7f71b000c820 comp rx=0 tx=0).stop 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 shutdown_connections 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71c8160630 0x7f71c815d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71a8076300 0x7f71a8078810 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80b30a0 0x7f71c814cc00 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 --2- 172.21.15.46:0/1829647984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71c80b2c90 0x7f71c8162e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 >> 172.21.15.46:0/1829647984 conn(0x7f71c809b7d0 msgr2=0x7f71c80d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 shutdown_connections 2024-05-09T03:17:30.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.316+0000 7f71ce669640 1 -- 172.21.15.46:0/1829647984 wait complete. 2024-05-09T03:17:30.328 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_sessionmap' led to DAMAGED state 2024-05-09T03:17:30.328 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 400.00000000' 2024-05-09T03:17:30.329 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:30.329 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:30.345 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:30.345 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:30.345 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:30.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:30.345 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:30.402 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:30.402 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:30.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.557+0000 7fddd818d640 1 Processor -- start 2024-05-09T03:17:30.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.557+0000 7fddd818d640 1 -- start start 2024-05-09T03:17:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.558+0000 7fddd818d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fddd00d2310 0x7fddd015e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.558+0000 7fddd818d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd00b0880 con 0x7fddd00b1910 2024-05-09T03:17:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.558+0000 7fddd818d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddd00b0b60 con 0x7fddd00d3210 2024-05-09T03:17:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.558+0000 7fddd818d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fddd00d3620 con 0x7fddd00d2310 2024-05-09T03:17:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.558+0000 7fddd5f29640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fddd00b1910 0x7fddd00d1c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41064/0 (socket says 172.21.15.46:41064) 2024-05-09T03:17:30.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.558+0000 7fddd5f29640 1 -- 172.21.15.46:0/940711213 learned_addr learned my addr 172.21.15.46:0/940711213 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd5728640 1 --2- 172.21.15.46:0/940711213 >> v2:172.21.15.88:3300/0 conn(0x7fddd00d2310 0x7fddd015e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199429234 0 0) 0x7fddd00b0880 con 0x7fddd00b1910 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddb8003520 con 0x7fddd00b1910 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1594296210 0 0) 0x7fddb8003520 con 0x7fddd00b1910 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 >> v2:172.21.15.88:3300/0 conn(0x7fddd00d2310 msgr2=0x7fddd015e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 --2- 172.21.15.46:0/940711213 >> v2:172.21.15.88:3300/0 conn(0x7fddd00d2310 0x7fddd015e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 >> v1:172.21.15.88:6789/0 conn(0x7fddd00d3210 legacy=0x7fddd006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd00b0e40 con 0x7fddd00b1910 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fddc0001a70 con 0x7fddd00b1910 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 >> v1:172.21.15.46:6789/0 conn(0x7fddd00b1910 legacy=0x7fddd00d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb8003a00 0x7fddb8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb8004800 0x7fddb8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8005270 0x7fddb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fddd00d3620 con 0x7fddb8004800 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fddb800c200 con 0x7fddb8005270 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fddb800c400 con 0x7fddb8003a00 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd5728640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb8004800 0x7fddb8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd5f29640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb8003a00 0x7fddb8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.559+0000 7fddd672a640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8005270 0x7fddb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd5728640 1 -- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb8003a00 msgr2=0x7fddb8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd5728640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb8003a00 0x7fddb8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd5728640 1 -- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8005270 msgr2=0x7fddb800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:30.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd5728640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8005270 0x7fddb800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd5728640 1 -- 172.21.15.46:0/940711213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb8003520 con 0x7fddb8004800 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd672a640 1 -- 172.21.15.46:0/940711213 reap_dead start 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.560+0000 7fddd5728640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb8004800 0x7fddb8004c10 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009a90 tx=0x7fddb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf791df5d35a5cb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4036070 con 0x7fddb8004800 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc4031400 con 0x7fddb8004800 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd4f27640 1 -- 172.21.15.46:0/940711213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4009690 con 0x7fddb8004800 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 -- 172.21.15.46:0/940711213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb8004800 msgr2=0x7fddb8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb8004800 0x7fddb8004c10 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009a90 tx=0x7fddb8005830 comp rx=0 tx=0).stop 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 -- 172.21.15.46:0/940711213 shutdown_connections 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 --2- 172.21.15.46:0/940711213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb8004800 0x7fddb8004c10 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 -- 172.21.15.46:0/940711213 >> 172.21.15.46:0/940711213 conn(0x7fddd009b7d0 msgr2=0x7fddd009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 -- 172.21.15.46:0/940711213 shutdown_connections 2024-05-09T03:17:30.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.561+0000 7fddd818d640 1 -- 172.21.15.46:0/940711213 wait complete. 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.563+0000 7fddd818d640 1 Processor -- start 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.563+0000 7fddd818d640 1 -- start start 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.564+0000 7fddd818d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8004800 0x7fddd015f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.564+0000 7fddd818d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddd0148b60 0x7fddd00cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.564+0000 7fddd818d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 0x7fddd0070240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.564+0000 7fddd818d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fddd015e540 con 0x7fddd0146b10 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.564+0000 7fddd818d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fddd015e3c0 con 0x7fddb8004800 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.564+0000 7fddd818d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fddd015e6c0 con 0x7fddd0148b60 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 0x7fddd0070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 0x7fddd0070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55416/0 (socket says 172.21.15.46:55416) 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 -- 172.21.15.46:0/1937262781 learned_addr learned my addr 172.21.15.46:0/1937262781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:30.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd5f29640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8004800 0x7fddd015f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd5728640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddd0148b60 0x7fddd00cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 -- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddd0148b60 msgr2=0x7fddd00cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddd0148b60 0x7fddd00cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 -- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8004800 msgr2=0x7fddd015f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8004800 0x7fddd015f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 -- 172.21.15.46:0/1937262781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd0149340 con 0x7fddd0146b10 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd5728640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddd0148b60 0x7fddd00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd672a640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 0x7fddd0070240 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fddd015e920 tx=0x7fddc800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc8019070 con 0x7fddd0146b10 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddc80050d0 con 0x7fddd0146b10 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd0149620 con 0x7fddd0146b10 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc80067d0 con 0x7fddd0146b10 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.565+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd0146f20 con 0x7fddd0146b10 2024-05-09T03:17:30.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.566+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd0070ea0 con 0x7fddd0146b10 2024-05-09T03:17:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.567+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fddc801c070 con 0x7fddd0146b10 2024-05-09T03:17:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.568+0000 7fddce7fc640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddbc0762d0 0x7fddbc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.568+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fddc8096980 con 0x7fddd0146b10 2024-05-09T03:17:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.568+0000 7fddd5f29640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddbc0762d0 0x7fddbc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.568+0000 7fddd5f29640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddbc0762d0 0x7fddbc0787e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fddc00015c0 tx=0x7fddc0006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:30.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.572+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fddc805f8e0 con 0x7fddd0146b10 2024-05-09T03:17:30.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:30.803+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fddd01490d0 con 0x7fddd0146b10 2024-05-09T03:17:31.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.206+0000 7fddce7fc640 1 -- 172.21.15.46:0/1937262781 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v160) ==== 161+0+0 (secure 0 0 0) 0x7fddc8063be0 con 0x7fddd0146b10 2024-05-09T03:17:31.208 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:31.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddbc0762d0 msgr2=0x7fddbc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddbc0762d0 0x7fddbc0787e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fddc00015c0 tx=0x7fddc0006040 comp rx=0 tx=0).stop 2024-05-09T03:17:31.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 msgr2=0x7fddd0070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 0x7fddd0070240 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fddd015e920 tx=0x7fddc800fc00 comp rx=0 tx=0).stop 2024-05-09T03:17:31.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 shutdown_connections 2024-05-09T03:17:31.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddbc0762d0 0x7fddbc0787e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddd0146b10 0x7fddd0070240 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddd0148b60 0x7fddd00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 --2- 172.21.15.46:0/1937262781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb8004800 0x7fddd015f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.208+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 >> 172.21.15.46:0/1937262781 conn(0x7fddd009b7d0 msgr2=0x7fddd0071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:31.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.209+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 shutdown_connections 2024-05-09T03:17:31.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.209+0000 7fddd818d640 1 -- 172.21.15.46:0/1937262781 wait complete. 2024-05-09T03:17:31.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.339+0000 7f6eb1201640 1 Processor -- start 2024-05-09T03:17:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.339+0000 7f6eb1201640 1 -- start start 2024-05-09T03:17:31.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eb1201640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6eac0b4ee0 0x7f6eac0d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:31.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eb1201640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6eac0d0b10 con 0x7f6eac0b8060 2024-05-09T03:17:31.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eb1201640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6eac1612e0 con 0x7f6eac164820 2024-05-09T03:17:31.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eb1201640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6eac1615c0 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eaa59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6eac0b8060 0x7f6eac0d43f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41078/0 (socket says 172.21.15.46:41078) 2024-05-09T03:17:31.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eaa59c640 1 -- 172.21.15.46:0/727034139 learned_addr learned my addr 172.21.15.46:0/727034139 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:31.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eab59e640 1 --2- 172.21.15.46:0/727034139 >> v2:172.21.15.88:3300/0 conn(0x7f6eac0b4ee0 0x7f6eac0d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:31.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6ea9d9b640 1 -- 172.21.15.46:0/727034139 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2953445606 0 0) 0x7f6e94001110 con 0x7f6eac0b8060 2024-05-09T03:17:31.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6ea9d9b640 1 -- 172.21.15.46:0/727034139 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e98003520 con 0x7f6eac0b8060 2024-05-09T03:17:31.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eab59e640 1 -- 172.21.15.46:0/727034139 >> v1:172.21.15.88:6789/0 conn(0x7f6eac164820 legacy=0x7f6eac0b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6ea9d9b640 1 -- 172.21.15.46:0/727034139 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3804861635 0 0) 0x7f6e98003520 con 0x7f6eac0b8060 2024-05-09T03:17:31.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eab59e640 1 -- 172.21.15.46:0/727034139 >> v1:172.21.15.46:6789/0 conn(0x7f6eac0b8060 legacy=0x7f6eac0d43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.341+0000 7f6eab59e640 1 -- 172.21.15.46:0/727034139 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac163d40 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.342+0000 7f6eab59e640 1 --2- 172.21.15.46:0/727034139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0d4b00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009d90 tx=0x7f6e9c02a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44071b5b4d0ec5ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:31.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.342+0000 7f6ea9d9b640 1 -- 172.21.15.46:0/727034139 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e940018c0 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.342+0000 7f6ea9d9b640 1 -- 172.21.15.46:0/727034139 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e9c032440 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.342+0000 7f6ea9d9b640 1 -- 172.21.15.46:0/727034139 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0327f0 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 -- 172.21.15.46:0/727034139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac0b4ee0 msgr2=0x7f6eac0d4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 --2- 172.21.15.46:0/727034139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0d4b00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009d90 tx=0x7f6e9c02a480 comp rx=0 tx=0).stop 2024-05-09T03:17:31.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 -- 172.21.15.46:0/727034139 shutdown_connections 2024-05-09T03:17:31.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 --2- 172.21.15.46:0/727034139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0d4b00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 -- 172.21.15.46:0/727034139 >> 172.21.15.46:0/727034139 conn(0x7f6eac0d29d0 msgr2=0x7f6eac0d2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:31.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 -- 172.21.15.46:0/727034139 shutdown_connections 2024-05-09T03:17:31.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.343+0000 7f6eb1201640 1 -- 172.21.15.46:0/727034139 wait complete. 2024-05-09T03:17:31.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.345+0000 7f6eb1201640 1 Processor -- start 2024-05-09T03:17:31.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.345+0000 7f6eb1201640 1 -- start start 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.346+0000 7f6eb1201640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.346+0000 7f6eb1201640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac164820 0x7f6eac158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.346+0000 7f6eb1201640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6eac1e9c80 0x7f6eac159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.346+0000 7f6eb1201640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac0d5040 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.346+0000 7f6eb1201640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac0b52f0 con 0x7f6eac164820 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.346+0000 7f6eb1201640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6eac0d51c0 con 0x7f6eac1e9c80 2024-05-09T03:17:31.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:31.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55426/0 (socket says 172.21.15.46:55426) 2024-05-09T03:17:31.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 -- 172.21.15.46:0/2341354715 learned_addr learned my addr 172.21.15.46:0/2341354715 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:31.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaa59c640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac164820 0x7f6eac158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:31.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eab59e640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6eac1e9c80 0x7f6eac159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:31.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 -- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6eac1e9c80 msgr2=0x7f6eac159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6eac1e9c80 0x7f6eac159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 -- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac164820 msgr2=0x7f6eac158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac164820 0x7f6eac158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 -- 172.21.15.46:0/2341354715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac0cab00 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eab59e640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6eac1e9c80 0x7f6eac159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eaad9d640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0cbba0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400c450 tx=0x7f6ea400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4017070 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea4005480 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac0cade0 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4003830 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.347+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac0cb3a0 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.348+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eac076c00 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.349+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ea401a070 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.350+0000 7f6ea37fe640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6e8c0762d0 0x7f6e8c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:31.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.350+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f6ea4096f00 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.350+0000 7f6eaa59c640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6e8c0762d0 0x7f6e8c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:31.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.351+0000 7f6eaa59c640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6e8c0762d0 0x7f6e8c0787e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6eac163370 tx=0x7f6e94001e20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:31.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.354+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6ea4060310 con 0x7f6eac0b4ee0 2024-05-09T03:17:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:31.620+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f6eac0d3090 con 0x7f6eac0b4ee0 2024-05-09T03:17:32.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.207+0000 7f6ea37fe640 1 -- 172.21.15.46:0/2341354715 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v161) ==== 101+0+0 (secure 0 0 0) 0x7f6ea4068390 con 0x7f6eac0b4ee0 2024-05-09T03:17:32.208 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:17:32.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6e8c0762d0 msgr2=0x7f6e8c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6e8c0762d0 0x7f6e8c0787e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6eac163370 tx=0x7f6e94001e20 comp rx=0 tx=0).stop 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 msgr2=0x7f6eac0cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0cbba0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400c450 tx=0x7f6ea400c910 comp rx=0 tx=0).stop 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 shutdown_connections 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6eac1e9c80 0x7f6eac159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6eac164820 0x7f6eac158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6e8c0762d0 0x7f6e8c0787e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 --2- 172.21.15.46:0/2341354715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eac0b4ee0 0x7f6eac0cbba0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 >> 172.21.15.46:0/2341354715 conn(0x7f6eac0d29d0 msgr2=0x7f6eac0d5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 shutdown_connections 2024-05-09T03:17:32.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.211+0000 7f6eb1201640 1 -- 172.21.15.46:0/2341354715 wait complete. 2024-05-09T03:17:32.225 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:32.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.281+0000 7fc337521f40 1 Processor -- start 2024-05-09T03:17:32.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.281+0000 7fc337521f40 1 -- start start 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc337521f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e34de3a0 0x55d8e35698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc337521f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e349f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc337521f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3572150 0x55d8e349fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc337521f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d8e3444f50 con 0x55d8e34e65f0 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc337521f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d8e3448f90 con 0x55d8e3572150 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc337521f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d8e34e78d0 con 0x55d8e34de3a0 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e349f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e349f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55448/0 (socket says 172.21.15.46:55448) 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 -- 172.21.15.46:0/3845895194 learned_addr learned my addr 172.21.15.46:0/3845895194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc3317ea640 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3572150 0x55d8e349fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc330fe9640 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e34de3a0 0x55d8e35698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 -- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e34de3a0 msgr2=0x55d8e35698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e34de3a0 0x55d8e35698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 -- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3572150 msgr2=0x55d8e349fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3572150 0x55d8e349fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 -- 172.21.15.46:0/3845895194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8e34e8710 con 0x55d8e34e65f0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32bfff640 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e349f6b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fc31c025c10 tx=0x7fc31c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113fb1d31468b356 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.283+0000 7fc32affd640 1 -- 172.21.15.46:0/3845895194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c0538a0 con 0x55d8e34e65f0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc32affd640 1 -- 172.21.15.46:0/3845895194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31c053a40 con 0x55d8e34e65f0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc32affd640 1 -- 172.21.15.46:0/3845895194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c053df0 con 0x55d8e34e65f0 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 -- 172.21.15.46:0/3845895194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 msgr2=0x55d8e349f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e349f6b0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fc31c025c10 tx=0x7fc31c051000 comp rx=0 tx=0).stop 2024-05-09T03:17:32.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 -- 172.21.15.46:0/3845895194 shutdown_connections 2024-05-09T03:17:32.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e3572150 0x55d8e349fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e349f6b0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 --2- 172.21.15.46:0/3845895194 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e34de3a0 0x55d8e35698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 -- 172.21.15.46:0/3845895194 >> 172.21.15.46:0/3845895194 conn(0x55d8e33c0ac0 msgr2=0x55d8e34c9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:32.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 -- 172.21.15.46:0/3845895194 shutdown_connections 2024-05-09T03:17:32.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.284+0000 7fc337521f40 1 -- 172.21.15.46:0/3845895194 wait complete. 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.285+0000 7fc337521f40 1 Processor -- start 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.285+0000 7fc337521f40 1 -- start start 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc337521f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e34de3a0 0x55d8e349f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc337521f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e34e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc337521f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e3572150 0x55d8e34e05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc337521f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d8e34c8db0 con 0x55d8e34e65f0 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc337521f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d8e34cc1f0 con 0x55d8e34de3a0 2024-05-09T03:17:32.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc337521f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d8e356b040 con 0x55d8e3572150 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.286+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e34e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e34e00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55460/0 (socket says 172.21.15.46:55460) 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 learned_addr learned my addr 172.21.15.46:0/2508859043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e34de3a0 0x55d8e349f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc3317ea640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e3572150 0x55d8e34e05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e3572150 msgr2=0x55d8e34e05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d8e3572150 0x55d8e34e05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e34de3a0 msgr2=0x55d8e349f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e34de3a0 0x55d8e349f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8e3604560 con 0x55d8e34e65f0 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d8e34de3a0 0x55d8e349f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:32.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e34e00b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00a4f0 tx=0x7fc31c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc329ffb640 1 -- 172.21.15.46:0/2508859043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c073480 con 0x55d8e34e65f0 2024-05-09T03:17:32.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc329ffb640 1 -- 172.21.15.46:0/2508859043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc31c073640 con 0x55d8e34e65f0 2024-05-09T03:17:32.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d8e3604840 con 0x55d8e34e65f0 2024-05-09T03:17:32.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc329ffb640 1 -- 172.21.15.46:0/2508859043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31c073a30 con 0x55d8e34e65f0 2024-05-09T03:17:32.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.287+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8e3604e00 con 0x55d8e34e65f0 2024-05-09T03:17:32.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.289+0000 7fc329ffb640 1 -- 172.21.15.46:0/2508859043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc31c067480 con 0x55d8e34e65f0 2024-05-09T03:17:32.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.289+0000 7fc329ffb640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc314074cb0 0x7fc3140771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc329ffb640 1 -- 172.21.15.46:0/2508859043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fc31c0dc2c0 con 0x55d8e34e65f0 2024-05-09T03:17:32.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc314074cb0 0x7fc3140771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.292 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d8e349fde0 0x55d8e3569d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e34e1130 con 0x55d8e349fde0 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc3317ea640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d8e349fde0 0x55d8e3569d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc314074cb0 0x7fc3140771c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc324000f60 tx=0x7fc3240015e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.290+0000 7fc3317ea640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d8e349fde0 0x55d8e3569d00 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.292+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v96'350 uv324 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.292+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e356a240 con 0x55d8e349fde0 2024-05-09T03:17:32.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.293+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v96'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.293+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e356a610 con 0x55d8e349fde0 2024-05-09T03:17:32.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.294+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v96'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.295 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:32.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.294+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3618f90 con 0x55d8e349fde0 2024-05-09T03:17:32.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.296+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v96'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.296+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3619330 con 0x55d8e349fde0 2024-05-09T03:17:32.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.297+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v96'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.297+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3617f80 con 0x55d8e349fde0 2024-05-09T03:17:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.298+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v96'355 uv355 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.298+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3618320 con 0x55d8e349fde0 2024-05-09T03:17:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.299+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v96'356 uv330 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.299+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.301+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v96'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.301+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.302+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v96'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.303 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:32.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.302+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.303+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v96'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.303+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.304+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v96'360 uv360 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.304+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.306+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v96'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.306+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.307+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v96'362 uv337 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.307+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.308+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v96'363 uv363 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.308+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.310 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:32.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.309+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v96'364 uv364 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.309+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.311+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v96'365 uv365 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.311+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.312+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v96'366 uv366 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.312+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.313+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v96'367 uv367 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.313+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.314+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v96'368 uv368 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.315+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.316+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v96'369 uv343 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.316+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.317+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v96'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.317+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.319 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.318+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v96'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.318+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.319+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v96'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.319+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.320+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v96'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.321+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.322+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v96'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.322+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.323+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v96'375 uv349 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.323+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.324+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v96'376 uv376 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.324+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.325+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v96'377 uv377 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.326 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.325+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.326+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v96'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.326+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.327+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v96'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.327+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e349fde0 2024-05-09T03:17:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.329+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v96'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.329+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d8e3618af0 0x55d8e361bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.329+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36186c0 con 0x55d8e3618af0 2024-05-09T03:17:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.329+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d8e3618af0 0x55d8e361bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.329+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d8e3618af0 0x55d8e361bbf0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.331+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v96'260 uv243 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.331+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e361c130 con 0x55d8e3618af0 2024-05-09T03:17:32.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.332+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v96'261 uv261 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.332+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361c4d0 con 0x55d8e3618af0 2024-05-09T03:17:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.333+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v96'262 uv262 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.334 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:32.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.333+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361cd20 con 0x55d8e3618af0 2024-05-09T03:17:32.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.334+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v96'263 uv263 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.334+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d0c0 con 0x55d8e3618af0 2024-05-09T03:17:32.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.336+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v96'264 uv264 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.336+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d460 con 0x55d8e3618af0 2024-05-09T03:17:32.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.337+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v96'265 uv248 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.337+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.338+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v96'266 uv266 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.338+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.339+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v96'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.340 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:32.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.339+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.340+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v96'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.341+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.342+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v96'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.342+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.343+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v96'270 uv253 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.343+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.344+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v96'271 uv271 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.344+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.345+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v96'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.346 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:32.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.345+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621260 con 0x55d8e3618af0 2024-05-09T03:17:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.346+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v96'273 uv273 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.346+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621260 con 0x55d8e3618af0 2024-05-09T03:17:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.348+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v96'274 uv274 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.348+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.349+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v96'275 uv259 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.349+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.350+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v96'276 uv276 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.350+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.351+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v96'277 uv277 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.352 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:32.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.351+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361e810 con 0x55d8e3618af0 2024-05-09T03:17:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.353+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v96'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.353+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361e810 con 0x55d8e3618af0 2024-05-09T03:17:32.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.354+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v96'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.354+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e3618af0 2024-05-09T03:17:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.355+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v96'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3618af0 2024-05-09T03:17:32.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.355+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e349fde0 2024-05-09T03:17:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.356+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v96'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.357+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e349fde0 2024-05-09T03:17:32.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.358+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v96'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.358+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e349fde0 2024-05-09T03:17:32.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.359+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v96'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.360 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.359+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361e810 con 0x55d8e349fde0 2024-05-09T03:17:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.360+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v96'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.360+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361e810 con 0x55d8e349fde0 2024-05-09T03:17:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.361+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v96'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.361+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361d800 con 0x55d8e349fde0 2024-05-09T03:17:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.362+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v96'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.362+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d8e361ded0 0x55d8e361e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.362+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3620c20 con 0x55d8e361ded0 2024-05-09T03:17:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.362+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d8e361ded0 0x55d8e361e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.363+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d8e361ded0 0x55d8e361e340 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.364+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v96'56 uv55 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.364+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3620fc0 con 0x55d8e361ded0 2024-05-09T03:17:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.365+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v96'57 uv57 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.365+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621470 con 0x55d8e361ded0 2024-05-09T03:17:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.366+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v96'58 uv58 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.366 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:32.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.366+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e361c950 con 0x55d8e361ded0 2024-05-09T03:17:32.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.367+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v96'59 uv59 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.367+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621da0 con 0x55d8e349fde0 2024-05-09T03:17:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.368+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v96'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.368+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3621da0 con 0x55d8e349fde0 2024-05-09T03:17:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.369+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v96'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.370+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621da0 con 0x55d8e349fde0 2024-05-09T03:17:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.371+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v96'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.371 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.371+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3622db0 con 0x55d8e349fde0 2024-05-09T03:17:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.372+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v96'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.373+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3622db0 con 0x55d8e349fde0 2024-05-09T03:17:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.374+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v96'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.374+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621da0 con 0x55d8e349fde0 2024-05-09T03:17:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.375+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v96'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.375+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d8e3622670 0x55d8e3624a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.375+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3624fd0 con 0x55d8e3622670 2024-05-09T03:17:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.376+0000 7fc3317ea640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d8e3622670 0x55d8e3624a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.376+0000 7fc3317ea640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d8e3622670 0x55d8e3624a90 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.377+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v96'127 uv121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.377+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3625010 con 0x55d8e3622670 2024-05-09T03:17:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.378+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v96'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.378+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.380+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v96'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.380 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.380+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3626410 con 0x55d8e3622670 2024-05-09T03:17:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.381+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v96'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.381+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3626410 con 0x55d8e3622670 2024-05-09T03:17:32.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.382+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v96'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.382+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.383+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v96'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.383+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.384+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v96'133 uv126 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.384+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.385+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v96'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.385+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.386 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:32.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.386+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v96'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.386+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.387+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v96'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.387+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3625400 con 0x55d8e3622670 2024-05-09T03:17:32.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.388+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v96'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.388+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d8e3625a80 0x55d8e3627ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.388+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36285a0 con 0x55d8e3625a80 2024-05-09T03:17:32.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.389+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d8e3625a80 0x55d8e3627ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.389+0000 7fc330fe9640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d8e3625a80 0x55d8e3627ea0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.390+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v96'56 uv55 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3625a80 2024-05-09T03:17:32.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.390+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3628660 con 0x55d8e3625a80 2024-05-09T03:17:32.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.392+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v96'57 uv57 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3625a80 2024-05-09T03:17:32.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.392+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e3625a80 2024-05-09T03:17:32.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.393+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v96'58 uv58 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3625a80 2024-05-09T03:17:32.393 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:32.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.393+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e3625a80 2024-05-09T03:17:32.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.394+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v96'59 uv59 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3625a80 2024-05-09T03:17:32.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.394+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e3625a80 2024-05-09T03:17:32.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.395+0000 7fc330fe9640 1 -- 172.21.15.46:0/2508859043 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v96'60 uv60 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc324011800 con 0x55d8e3625a80 2024-05-09T03:17:32.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.395+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e349fde0 2024-05-09T03:17:32.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.396+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v96'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.396+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e349fde0 2024-05-09T03:17:32.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.397+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v96'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.397+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e349fde0 2024-05-09T03:17:32.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.398+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v96'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.399 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:32.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.398+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3629a90 con 0x55d8e349fde0 2024-05-09T03:17:32.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.400+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v96'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.400+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3629a90 con 0x55d8e349fde0 2024-05-09T03:17:32.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.401+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v96'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.401+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e349fde0 2024-05-09T03:17:32.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.402+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v96'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e349fde0 2024-05-09T03:17:32.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.402+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3628a80 con 0x55d8e361ded0 2024-05-09T03:17:32.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.403+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v96'112 uv108 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.403+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3628e20 con 0x55d8e361ded0 2024-05-09T03:17:32.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.404+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v96'113 uv113 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.404+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e36291c0 con 0x55d8e361ded0 2024-05-09T03:17:32.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.405+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v96'114 uv114 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.406 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:32.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.405+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621920 con 0x55d8e361ded0 2024-05-09T03:17:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.406+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v96'115 uv115 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.406+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621920 con 0x55d8e361ded0 2024-05-09T03:17:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.407+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v96'116 uv116 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.407+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621920 con 0x55d8e361ded0 2024-05-09T03:17:32.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.408+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v96'117 uv111 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.408+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e3621920 con 0x55d8e361ded0 2024-05-09T03:17:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.409+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v96'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.409+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621920 con 0x55d8e361ded0 2024-05-09T03:17:32.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.410+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v96'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.415 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.410+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e3621920 con 0x55d8e361ded0 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.411+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v96'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.412+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362c460 con 0x55d8e361ded0 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.413+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v96'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.413+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d8e3629560 0x55d8e362b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.413+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cb20 con 0x55d8e3629560 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.413+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d8e3629560 0x55d8e362b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.413+0000 7fc32bfff640 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d8e3629560 0x55d8e362b980 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.416 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.414+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v96'54 uv54 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e3629560 2024-05-09T03:17:32.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.415+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cb20 con 0x55d8e3629560 2024-05-09T03:17:32.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.416+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v96'55 uv55 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e3629560 2024-05-09T03:17:32.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.416+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf90 con 0x55d8e361ded0 2024-05-09T03:17:32.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.417+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(103 609.00000000 [create] v96'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.417+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e362bf90 con 0x55d8e361ded0 2024-05-09T03:17:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.418+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(104 609.00000000 [delete] v96'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.418+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf90 con 0x55d8e361ded0 2024-05-09T03:17:32.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.419+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(105 609.00000000 [create] v96'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.420 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:32.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.419+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf90 con 0x55d8e361ded0 2024-05-09T03:17:32.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.421+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v96'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.421+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf10 con 0x55d8e361ded0 2024-05-09T03:17:32.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.421+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v96'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.422+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf10 con 0x55d8e361ded0 2024-05-09T03:17:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.423+0000 7fc32bfff640 1 -- 172.21.15.46:0/2508859043 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v96'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc31c0a1400 con 0x55d8e361ded0 2024-05-09T03:17:32.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.423+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf10 con 0x55d8e3622670 2024-05-09T03:17:32.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.424+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(109 400.00000000 [create] v96'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.424+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e362bf10 con 0x55d8e3622670 2024-05-09T03:17:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.425+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(110 400.00000000 [delete] v96'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.425+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362bf10 con 0x55d8e3622670 2024-05-09T03:17:32.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.426+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(111 400.00000000 [create] v96'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.427 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:32.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.426+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.428+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v96'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.428+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.429+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v96'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.429+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.430+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(114 100.00000000.inode [create] v96'144 uv136 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.430+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.431+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(115 100.00000000.inode [delete] v96'145 uv145 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.431+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.433 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:32.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.432+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(116 100.00000000.inode [create] v96'146 uv146 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.432+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.434+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v96'147 uv147 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.434+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.435+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v96'148 uv148 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.435+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.436+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(119 1.00000000 [create] v96'149 uv143 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.436+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.437+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(120 1.00000000 [delete] v96'150 uv150 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.437+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.439+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(121 1.00000000 [create] v96'151 uv151 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.439 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:32.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.439+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362dfe0 con 0x55d8e3622670 2024-05-09T03:17:32.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.440+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v96'152 uv152 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.440+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362dfe0 con 0x55d8e3622670 2024-05-09T03:17:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.441+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v96'153 uv153 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.441+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.442+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v96'154 uv154 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.442+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55d8e362cfd0 con 0x55d8e3622670 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v96'155 uv155 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc32c006d50 con 0x55d8e3622670 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d8e349fde0 msgr2=0x55d8e3569d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d8e349fde0 0x55d8e3569d00 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d8e361ded0 msgr2=0x55d8e361e340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d8e361ded0 0x55d8e361e340 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d8e3629560 msgr2=0x55d8e362b980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d8e3629560 0x55d8e362b980 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d8e3618af0 msgr2=0x55d8e361bbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d8e3618af0 0x55d8e361bbf0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d8e3622670 msgr2=0x55d8e3624a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d8e3622670 0x55d8e3624a90 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 reap_dead start 2024-05-09T03:17:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d8e3625a80 msgr2=0x55d8e3627ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d8e3625a80 0x55d8e3627ea0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.443+0000 7fc3317ea640 1 -- 172.21.15.46:0/2508859043 reap_dead start 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc314074cb0 msgr2=0x7fc3140771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc314074cb0 0x7fc3140771c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc324000f60 tx=0x7fc3240015e0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 msgr2=0x55d8e34e00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e34e00b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fc31c00a4f0 tx=0x7fc31c002fb0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 shutdown_connections 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc314074cb0 0x7fc3140771c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 --2- 172.21.15.46:0/2508859043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d8e34e65f0 0x55d8e34e00b0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 >> 172.21.15.46:0/2508859043 conn(0x55d8e33c0ac0 msgr2=0x55d8e34c9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 shutdown_connections 2024-05-09T03:17:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.444+0000 7fc337521f40 1 -- 172.21.15.46:0/2508859043 wait complete. 2024-05-09T03:17:32.447 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 400.00000000 2024-05-09T03:17:32.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.542+0000 7f0cc6ce6f40 1 Processor -- start 2024-05-09T03:17:32.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.542+0000 7f0cc6ce6f40 1 -- start start 2024-05-09T03:17:32.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cc6ce6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dbf795f0 0x5590dbffc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cc6ce6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 0x5590dbf5b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cc6ce6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cc6ce6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5590dbf7a8d0 con 0x5590dc004a40 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cc6ce6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5590dbf7a5f0 con 0x5590dc005390 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cc6ce6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5590dbf75bb0 con 0x5590dbf795f0 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cbffad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 0x5590dbf5b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cbffad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 0x5590dbf5b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55476/0 (socket says 172.21.15.46:55476) 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.543+0000 7f0cbffad640 1 -- 172.21.15.46:0/3373987215 learned_addr learned my addr 172.21.15.46:0/3373987215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbffad640 1 -- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dbf795f0 msgr2=0x5590dbffc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbffad640 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dbf795f0 0x5590dbffc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cc0faf640 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbffad640 1 -- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 msgr2=0x5590dbf5b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbffad640 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbffad640 1 -- 172.21.15.46:0/3373987215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590dbe99350 con 0x5590dc004a40 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cc0faf640 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbffad640 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 0x5590dbf5b450 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40097b0 tx=0x7f0cb4005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5894e0cd4a13855d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbefab640 1 -- 172.21.15.46:0/3373987215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4019070 con 0x5590dc004a40 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbefab640 1 -- 172.21.15.46:0/3373987215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb4008890 con 0x5590dc004a40 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.544+0000 7f0cbefab640 1 -- 172.21.15.46:0/3373987215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4008c80 con 0x5590dc004a40 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/3373987215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 msgr2=0x5590dbf5b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 0x5590dbf5b450 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40097b0 tx=0x7f0cb4005a00 comp rx=0 tx=0).stop 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/3373987215 shutdown_connections 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dc004a40 0x5590dbf5b450 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/3373987215 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dbf795f0 0x5590dbffc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/3373987215 >> 172.21.15.46:0/3373987215 conn(0x5590dbe53ac0 msgr2=0x5590dc002570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:32.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/3373987215 shutdown_connections 2024-05-09T03:17:32.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/3373987215 wait complete. 2024-05-09T03:17:32.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.545+0000 7f0cc6ce6f40 1 Processor -- start 2024-05-09T03:17:32.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.546+0000 7f0cc6ce6f40 1 -- start start 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc6ce6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dbf795f0 0x5590dbff0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc6ce6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dc004a40 0x5590dbfef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc6ce6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc6ce6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5590dc001890 con 0x5590dbf795f0 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc6ce6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5590dbf7b7f0 con 0x5590dc005390 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc6ce6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5590dbf732a0 con 0x5590dc004a40 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dbf795f0 0x5590dbff0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dbf795f0 0x5590dbff0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55480/0 (socket says 172.21.15.46:55480) 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 -- 172.21.15.46:0/725673418 learned_addr learned my addr 172.21.15.46:0/725673418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc0faf640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cbffad640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dc004a40 0x5590dbfef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 -- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dc004a40 msgr2=0x5590dbfef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5590dc004a40 0x5590dbfef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 -- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 msgr2=0x5590dbf5bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5590dc005390 0x5590dbf5bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.547+0000 7f0cc07ae640 1 -- 172.21.15.46:0/725673418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590dbff0650 con 0x5590dbf795f0 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.548+0000 7f0cc07ae640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dbf795f0 0x5590dbff0110 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0054040 tx=0x7f0cb0005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.548+0000 7f0cbdfa9640 1 -- 172.21.15.46:0/725673418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0069070 con 0x5590dbf795f0 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.548+0000 7f0cbdfa9640 1 -- 172.21.15.46:0/725673418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb004e040 con 0x5590dbf795f0 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.548+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5590dbffd990 con 0x5590dbf795f0 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.548+0000 7f0cbdfa9640 1 -- 172.21.15.46:0/725673418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0051810 con 0x5590dbf795f0 2024-05-09T03:17:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.548+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590dbf72be0 con 0x5590dbf795f0 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.549+0000 7f0cbdfa9640 1 -- 172.21.15.46:0/725673418 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0cb006c070 con 0x5590dbf795f0 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.550+0000 7f0cbdfa9640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cac074cb0 0x7f0cac0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.550+0000 7f0cbdfa9640 1 -- 172.21.15.46:0/725673418 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f0cb00d9700 con 0x5590dbf795f0 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.550+0000 7f0cbffad640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cac074cb0 0x7f0cac0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.550+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dbff0820 0x5590dbf72420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.550+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e96) -- 0x5590dbffc380 con 0x5590dbff0820 2024-05-09T03:17:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.550+0000 7f0cc0faf640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dbff0820 0x5590dbf72420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.551+0000 7f0cbffad640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cac074cb0 0x7f0cac0771c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x5590dbf605b0 tx=0x7f0cb4008000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.551+0000 7f0cc0faf640 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dbff0820 0x5590dbf72420 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc0faf640 1 -- 172.21.15.46:0/725673418 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 400.00000000 [delete] v96'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0cb8009890 con 0x5590dbff0820 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dbff0820 msgr2=0x5590dbf72420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5590dbff0820 0x5590dbf72420 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cac074cb0 msgr2=0x7f0cac0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cac074cb0 0x7f0cac0771c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x5590dbf605b0 tx=0x7f0cb4008000 comp rx=0 tx=0).stop 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dbf795f0 msgr2=0x5590dbff0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 --2- 172.21.15.46:0/725673418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5590dbf795f0 0x5590dbff0110 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0054040 tx=0x7f0cb0005b80 comp rx=0 tx=0).stop 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc0faf640 1 -- 172.21.15.46:0/725673418 reap_dead start 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 shutdown_connections 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 >> 172.21.15.46:0/725673418 conn(0x5590dbe53ac0 msgr2=0x5590dc001a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:32.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 shutdown_connections 2024-05-09T03:17:32.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.552+0000 7f0cc6ce6f40 1 -- 172.21.15.46:0/725673418 wait complete. 2024-05-09T03:17:32.555 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:32.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.673+0000 7fe1b9dc7640 1 Processor -- start 2024-05-09T03:17:32.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.673+0000 7fe1b9dc7640 1 -- start start 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b9dc7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b41605b0 0x7fe1b40b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b9dc7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b40d3400 con 0x7fe1b40b7e70 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b9dc7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b40d4a60 con 0x7fe1b40d2e90 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b9dc7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe1b40d4d40 con 0x7fe1b41605b0 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1b41605b0 0x7fe1b40b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe1b40b7e70 0x7fe1b40b92a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41094/0 (socket says 172.21.15.46:41094) 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b2ffd640 1 -- 172.21.15.46:0/2906105696 learned_addr learned my addr 172.21.15.46:0/2906105696 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b3fff640 1 -- 172.21.15.46:0/2906105696 >> v1:172.21.15.88:6789/0 conn(0x7fe1b40d2e90 legacy=0x7fe1b40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b3fff640 1 -- 172.21.15.46:0/2906105696 >> v1:172.21.15.46:6789/0 conn(0x7fe1b40b7e70 legacy=0x7fe1b40b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.674+0000 7fe1b3fff640 1 -- 172.21.15.46:0/2906105696 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b40d5020 con 0x7fe1b41605b0 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.675+0000 7fe1b3fff640 1 --2- 172.21.15.46:0/2906105696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b41605b0 0x7fe1b40b99b0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009d90 tx=0x7fe1ac02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06ab66ffa367c77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.675+0000 7fe1b27fc640 1 -- 172.21.15.46:0/2906105696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac007740 con 0x7fe1b41605b0 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.675+0000 7fe1b27fc640 1 -- 172.21.15.46:0/2906105696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1ac0078e0 con 0x7fe1b41605b0 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.675+0000 7fe1b27fc640 1 -- 172.21.15.46:0/2906105696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac007cb0 con 0x7fe1b41605b0 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/2906105696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b41605b0 msgr2=0x7fe1b40b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/2906105696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b41605b0 0x7fe1b40b99b0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009d90 tx=0x7fe1ac02a660 comp rx=0 tx=0).stop 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/2906105696 shutdown_connections 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/2906105696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b41605b0 0x7fe1b40b99b0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/2906105696 >> 172.21.15.46:0/2906105696 conn(0x7fe1b408d390 msgr2=0x7fe1b408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/2906105696 shutdown_connections 2024-05-09T03:17:32.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.676+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/2906105696 wait complete. 2024-05-09T03:17:32.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.678+0000 7fe1b9dc7640 1 Processor -- start 2024-05-09T03:17:32.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.678+0000 7fe1b9dc7640 1 -- start start 2024-05-09T03:17:32.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b9dc7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 0x7fe1b41f9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b9dc7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1b41605b0 0x7fe1b40cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b9dc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b415dd20 0x7fe1b415e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b9dc7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b40b4d80 con 0x7fe1b40d2e90 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b9dc7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b40b4c00 con 0x7fe1b415dd20 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b9dc7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b40b4fc0 con 0x7fe1b41605b0 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.679+0000 7fe1b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 0x7fe1b41f9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 0x7fe1b41f9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55488/0 (socket says 172.21.15.46:55488) 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 -- 172.21.15.46:0/912297122 learned_addr learned my addr 172.21.15.46:0/912297122 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b2ffd640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1b41605b0 0x7fe1b40cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b3fff640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b415dd20 0x7fe1b415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 -- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1b41605b0 msgr2=0x7fe1b40cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1b41605b0 0x7fe1b40cbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 -- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b415dd20 msgr2=0x7fe1b415e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b415dd20 0x7fe1b415e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 -- 172.21.15.46:0/912297122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b415e670 con 0x7fe1b40d2e90 2024-05-09T03:17:32.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b37fe640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 0x7fe1b41f9b10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400c370 tx=0x7fe1a400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4017070 con 0x7fe1b40d2e90 2024-05-09T03:17:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a40052d0 con 0x7fe1b40d2e90 2024-05-09T03:17:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4003830 con 0x7fe1b40d2e90 2024-05-09T03:17:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b41fa480 con 0x7fe1b40d2e90 2024-05-09T03:17:32.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.680+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b4157da0 con 0x7fe1b40d2e90 2024-05-09T03:17:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.681+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b406a270 con 0x7fe1b40d2e90 2024-05-09T03:17:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.682+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe1a401a070 con 0x7fe1b40d2e90 2024-05-09T03:17:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.683+0000 7fe19bfff640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1a0076300 0x7fe1a0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.683+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fe1a4096d30 con 0x7fe1b40d2e90 2024-05-09T03:17:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.683+0000 7fe1b2ffd640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1a0076300 0x7fe1a0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:32.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.683+0000 7fe1b2ffd640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1a0076300 0x7fe1a0078810 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8005880 tx=0x7fe1a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.687+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe1b415e670 con 0x7fe1b40d2e90 2024-05-09T03:17:32.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:32.918+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe1b415db00 con 0x7fe1b40d2e90 2024-05-09T03:17:33.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.217+0000 7fe19bfff640 1 -- 172.21.15.46:0/912297122 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v162) ==== 162+0+0 (secure 0 0 0) 0x7fe1a40681a0 con 0x7fe1b40d2e90 2024-05-09T03:17:33.218 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:33.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1a0076300 msgr2=0x7fe1a0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:33.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1a0076300 0x7fe1a0078810 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8005880 tx=0x7fe1a8009040 comp rx=0 tx=0).stop 2024-05-09T03:17:33.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 msgr2=0x7fe1b41f9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:33.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 0x7fe1b41f9b10 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400c370 tx=0x7fe1a400c830 comp rx=0 tx=0).stop 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 shutdown_connections 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1b415dd20 0x7fe1b415e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1b41605b0 0x7fe1b40cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1a0076300 0x7fe1a0078810 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 --2- 172.21.15.46:0/912297122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1b40d2e90 0x7fe1b41f9b10 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 >> 172.21.15.46:0/912297122 conn(0x7fe1b408d390 msgr2=0x7fe1b40d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 shutdown_connections 2024-05-09T03:17:33.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.220+0000 7fe1b9dc7640 1 -- 172.21.15.46:0/912297122 wait complete. 2024-05-09T03:17:33.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:33.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.755+0000 7f091e6b1640 1 Processor -- start 2024-05-09T03:17:33.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.755+0000 7f091e6b1640 1 -- start start 2024-05-09T03:17:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f091e6b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f09180d30f0 0x7f0918167dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:33.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f091e6b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f091815d590 con 0x7f091815d180 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f091e6b1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f091806e1c0 con 0x7f09180b28b0 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f091e6b1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f091806eaf0 con 0x7f09180d30f0 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f091cc4e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f09180d30f0 0x7f0918167dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f0917fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f091815d180 0x7f09180b21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41110/0 (socket says 172.21.15.46:41110) 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.756+0000 7f0917fff640 1 -- 172.21.15.46:0/1534476985 learned_addr learned my addr 172.21.15.46:0/1534476985 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.757+0000 7f091cc4e640 1 -- 172.21.15.46:0/1534476985 >> v1:172.21.15.88:6789/0 conn(0x7f09180b28b0 legacy=0x7f09180d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.757+0000 7f091cc4e640 1 -- 172.21.15.46:0/1534476985 >> v1:172.21.15.46:6789/0 conn(0x7f091815d180 legacy=0x7f09180b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.757+0000 7f0916ffd640 1 -- 172.21.15.46:0/1534476985 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3013017768 0 0) 0x7f0908001290 con 0x7f091815d180 2024-05-09T03:17:33.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.757+0000 7f091cc4e640 1 -- 172.21.15.46:0/1534476985 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091806e4a0 con 0x7f09180d30f0 2024-05-09T03:17:33.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.757+0000 7f091cc4e640 1 --2- 172.21.15.46:0/1534476985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09180d30f0 0x7f0918167dc0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f0910009d90 tx=0x7f091002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b9de415c7834de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f0916ffd640 1 -- 172.21.15.46:0/1534476985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910007740 con 0x7f09180d30f0 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f0916ffd640 1 -- 172.21.15.46:0/1534476985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09100078e0 con 0x7f09180d30f0 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f0916ffd640 1 -- 172.21.15.46:0/1534476985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910007cb0 con 0x7f09180d30f0 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 -- 172.21.15.46:0/1534476985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09180d30f0 msgr2=0x7f0918167dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 --2- 172.21.15.46:0/1534476985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09180d30f0 0x7f0918167dc0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f0910009d90 tx=0x7f091002a6f0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 -- 172.21.15.46:0/1534476985 shutdown_connections 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 --2- 172.21.15.46:0/1534476985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09180d30f0 0x7f0918167dc0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 -- 172.21.15.46:0/1534476985 >> 172.21.15.46:0/1534476985 conn(0x7f091809b7d0 msgr2=0x7f091809bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 -- 172.21.15.46:0/1534476985 shutdown_connections 2024-05-09T03:17:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.758+0000 7f091e6b1640 1 -- 172.21.15.46:0/1534476985 wait complete. 2024-05-09T03:17:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.760+0000 7f091e6b1640 1 Processor -- start 2024-05-09T03:17:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.760+0000 7f091e6b1640 1 -- start start 2024-05-09T03:17:33.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09180d30f0 0x7f091814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f091815d180 0x7f09180c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 0x7f09180d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09180b5040 con 0x7f091814d250 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f091806ef80 con 0x7f09180d30f0 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f09180b51c0 con 0x7f091815d180 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 0x7f09180d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 0x7f09180d1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55504/0 (socket says 172.21.15.46:55504) 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 -- 172.21.15.46:0/826721517 learned_addr learned my addr 172.21.15.46:0/826721517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f0917fff640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09180d30f0 0x7f091814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f09177fe640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f091815d180 0x7f09180c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 -- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f091815d180 msgr2=0x7f09180c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f091815d180 0x7f09180c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 -- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09180d30f0 msgr2=0x7f091814cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09180d30f0 0x7f091814cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 -- 172.21.15.46:0/826721517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09180c9120 con 0x7f091814d250 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091cc4e640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 0x7f09180d1ac0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f0910009eb0 tx=0x7f0910004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910040070 con 0x7f091814d250 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0910031c90 con 0x7f091814d250 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.762+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09180c9400 con 0x7f091814d250 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.763+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0910045680 con 0x7f091814d250 2024-05-09T03:17:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.763+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09180ca840 con 0x7f091814d250 2024-05-09T03:17:33.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.763+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09180b1a30 con 0x7f091814d250 2024-05-09T03:17:33.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.764+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0910039040 con 0x7f091814d250 2024-05-09T03:17:33.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.765+0000 7f0914ff9640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0904076300 0x7f0904078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:33.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.765+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f09100b94c0 con 0x7f091814d250 2024-05-09T03:17:33.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.765+0000 7f0917fff640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0904076300 0x7f0904078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:33.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.766+0000 7f0917fff640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0904076300 0x7f0904078810 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0908007a50 tx=0x7f0908009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:33.769+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f09180b1a30 con 0x7f091814d250 2024-05-09T03:17:34.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.000+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f091806a270 con 0x7f091814d250 2024-05-09T03:17:34.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.000+0000 7f0914ff9640 1 -- 172.21.15.46:0/826721517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) ==== 95+0+5177 (secure 0 0 0) 0x7f0910082c10 con 0x7f091814d250 2024-05-09T03:17:34.001 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:34.001 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":163,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:33.221236+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8292":{"gid":8292,"name":"c","rank":0,"incarnation":163,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/1486786164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486786164},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486786164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:34.001 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 163 2024-05-09T03:17:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0904076300 msgr2=0x7f0904078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0904076300 0x7f0904078810 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0908007a50 tx=0x7f0908009040 comp rx=0 tx=0).stop 2024-05-09T03:17:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 msgr2=0x7f09180d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 0x7f09180d1ac0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f0910009eb0 tx=0x7f0910004a40 comp rx=0 tx=0).stop 2024-05-09T03:17:34.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 shutdown_connections 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0904076300 0x7f0904078810 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f091814d250 0x7f09180d1ac0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f091815d180 0x7f09180c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 --2- 172.21.15.46:0/826721517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09180d30f0 0x7f091814cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 >> 172.21.15.46:0/826721517 conn(0x7f091809b7d0 msgr2=0x7f0918070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 shutdown_connections 2024-05-09T03:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:34.002+0000 7f091e6b1640 1 -- 172.21.15.46:0/826721517 wait complete. 2024-05-09T03:17:34.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 163, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:17:33.221236+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8292}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8292': {'gid': 8292, 'name': 'c', 'rank': 0, 'incarnation': 163, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/1486786164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1486786164}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1486786164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:17:34.014 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8292:up:replay 2024-05-09T03:17:34.014 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:17:37.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:17:37.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:17:39.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.547+0000 7f53ac5c7640 1 Processor -- start 2024-05-09T03:17:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.547+0000 7f53ac5c7640 1 -- start start 2024-05-09T03:17:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53ac5c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53a40b5210 0x7f53a41613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53ac5c7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53a40b7ed0 0x7f53a40d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53ac5c7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53a40d5280 con 0x7f53a40d4800 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53ac5c7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f53a40b5620 con 0x7f53a40b5210 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53ac5c7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53a41609a0 con 0x7f53a40b7ed0 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53aa363640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53a40b5210 0x7f53a41613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53aa363640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f53a40b5210 0x7f53a41613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38690/0 (socket says 172.21.15.46:38690) 2024-05-09T03:17:39.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53aa363640 1 -- 172.21.15.46:0/1040653662 learned_addr learned my addr 172.21.15.46:0/1040653662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.549+0000 7f53aab64640 1 --2- 172.21.15.46:0/1040653662 >> v2:172.21.15.88:3300/0 conn(0x7f53a40b7ed0 0x7f53a40d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.550+0000 7f53aa363640 1 -- 172.21.15.46:0/1040653662 >> v2:172.21.15.88:3300/0 conn(0x7f53a40b7ed0 msgr2=0x7f53a40d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.550+0000 7f53aa363640 1 --2- 172.21.15.46:0/1040653662 >> v2:172.21.15.88:3300/0 conn(0x7f53a40b7ed0 0x7f53a40d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.550+0000 7f53aa363640 1 -- 172.21.15.46:0/1040653662 >> v1:172.21.15.46:6789/0 conn(0x7f53a40d4800 legacy=0x7f53a40b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.550+0000 7f53aa363640 1 -- 172.21.15.46:0/1040653662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a40d5560 con 0x7f53a40b5210 2024-05-09T03:17:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.550+0000 7f53a9361640 1 -- 172.21.15.46:0/1040653662 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643073209 0 0) 0x7f53a40d5280 con 0x7f53a40d4800 2024-05-09T03:17:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.550+0000 7f53aa363640 1 --2- 172.21.15.46:0/1040653662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40b5210 0x7f53a41613c0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f53a406f030 tx=0x7f53a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8b4b18c7b34839 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53a9361640 1 -- 172.21.15.46:0/1040653662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0037070 con 0x7f53a40b5210 2024-05-09T03:17:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53a9361640 1 -- 172.21.15.46:0/1040653662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a0032440 con 0x7f53a40b5210 2024-05-09T03:17:39.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53a9361640 1 -- 172.21.15.46:0/1040653662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0032810 con 0x7f53a40b5210 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53ac5c7640 1 -- 172.21.15.46:0/1040653662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40b5210 msgr2=0x7f53a41613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/1040653662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40b5210 0x7f53a41613c0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f53a406f030 tx=0x7f53a002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53ac5c7640 1 -- 172.21.15.46:0/1040653662 shutdown_connections 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/1040653662 >> v2:172.21.15.88:3300/0 conn(0x7f53a40b7ed0 0x7f53a40d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/1040653662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40b5210 0x7f53a41613c0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.551+0000 7f53ac5c7640 1 -- 172.21.15.46:0/1040653662 >> 172.21.15.46:0/1040653662 conn(0x7f53a40d29a0 msgr2=0x7f53a40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.552+0000 7f53ac5c7640 1 -- 172.21.15.46:0/1040653662 shutdown_connections 2024-05-09T03:17:39.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.552+0000 7f53ac5c7640 1 -- 172.21.15.46:0/1040653662 wait complete. 2024-05-09T03:17:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.553+0000 7f53ac5c7640 1 Processor -- start 2024-05-09T03:17:39.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.553+0000 7f53ac5c7640 1 -- start start 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53ac5c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53a40b5210 0x7f53a40cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53ac5c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 0x7f53a41580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53ac5c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a41fa8c0 0x7f53a40ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53ac5c7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53a40d0ae0 con 0x7f53a40d4800 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53ac5c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53a4160ea0 con 0x7f53a41fa8c0 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53ac5c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53a4163e60 con 0x7f53a40b5210 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 0x7f53a41580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 0x7f53a41580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38692/0 (socket says 172.21.15.46:38692) 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 -- 172.21.15.46:0/648830493 learned_addr learned my addr 172.21.15.46:0/648830493 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53aab64640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a41fa8c0 0x7f53a40ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53aa363640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53a40b5210 0x7f53a40cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 -- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53a40b5210 msgr2=0x7f53a40cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53a40b5210 0x7f53a40cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 -- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a41fa8c0 msgr2=0x7f53a40ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a41fa8c0 0x7f53a40ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 -- 172.21.15.46:0/648830493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a40cfc20 con 0x7f53a40d4800 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53aa363640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53a40b5210 0x7f53a40cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.555+0000 7f53a9b62640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 0x7f53a41580e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f539400c3f0 tx=0x7f539400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.556+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5394017070 con 0x7f53a40d4800 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.556+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53940035f0 con 0x7f53a40d4800 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.556+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a40cbef0 con 0x7f53a40d4800 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.556+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53940039e0 con 0x7f53a40d4800 2024-05-09T03:17:39.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.556+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a40c9aa0 con 0x7f53a40d4800 2024-05-09T03:17:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.556+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a406a270 con 0x7f53a40d4800 2024-05-09T03:17:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.558+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f539401a070 con 0x7f53a40d4800 2024-05-09T03:17:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.558+0000 7f539affd640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53880762d0 0x7f53880787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.558+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f5394097120 con 0x7f53a40d4800 2024-05-09T03:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.558+0000 7f53aa363640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53880762d0 0x7f53880787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.559+0000 7f53aa363640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53880762d0 0x7f53880787e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f53a0034000 tx=0x7f53a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.562+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5394021020 con 0x7f53a40d4800 2024-05-09T03:17:39.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.791+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53a40d40d0 con 0x7f53a40d4800 2024-05-09T03:17:39.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.791+0000 7f539affd640 1 -- 172.21.15.46:0/648830493 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+5184 (secure 0 0 0) 0x7f53940600d0 con 0x7f53a40d4800 2024-05-09T03:17:39.792 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:39.792 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":166,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:36.231162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8292":{"gid":8292,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/1486786164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486786164},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486786164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8292,"qdb_cluster":[8292]},"id":6}]} 2024-05-09T03:17:39.792 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 166 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53880762d0 msgr2=0x7f53880787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53880762d0 0x7f53880787e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f53a0034000 tx=0x7f53a002e000 comp rx=0 tx=0).stop 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 msgr2=0x7f53a41580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 0x7f53a41580e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f539400c3f0 tx=0x7f539400c8b0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 shutdown_connections 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53a41fa8c0 0x7f53a40ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53880762d0 0x7f53880787e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53a40d4800 0x7f53a41580e0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 --2- 172.21.15.46:0/648830493 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53a40b5210 0x7f53a40cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 >> 172.21.15.46:0/648830493 conn(0x7f53a40d29a0 msgr2=0x7f53a40baf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 shutdown_connections 2024-05-09T03:17:39.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.793+0000 7f53ac5c7640 1 -- 172.21.15.46:0/648830493 wait complete. 2024-05-09T03:17:39.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 166, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:17:36.231162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8292}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8292': {'gid': 8292, 'name': 'c', 'rank': 0, 'incarnation': 163, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/1486786164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1486786164}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1486786164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8292, 'qdb_cluster': [8292]} 2024-05-09T03:17:39.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:17:39.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:17:39.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.920+0000 7fcc2782b640 1 Processor -- start 2024-05-09T03:17:39.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.920+0000 7fcc2782b640 1 -- start start 2024-05-09T03:17:39.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc2782b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc200d1740 0x7fcc200b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc2782b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcc200d1110 0x7fcc200d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc2782b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc200b9690 con 0x7fcc200cfcf0 2024-05-09T03:17:39.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc2782b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcc2008cd60 con 0x7fcc200d1740 2024-05-09T03:17:39.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc2782b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcc200ca9e0 con 0x7fcc200d1110 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc255c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc200d1740 0x7fcc200b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc25dc8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcc200cfcf0 0x7fcc2009cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54024/0 (socket says 172.21.15.46:54024) 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc25dc8640 1 -- 172.21.15.46:0/1235394845 learned_addr learned my addr 172.21.15.46:0/1235394845 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc255c7640 1 -- 172.21.15.46:0/1235394845 >> v2:172.21.15.88:3300/0 conn(0x7fcc200d1110 msgr2=0x7fcc200d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/1235394845 >> v2:172.21.15.88:3300/0 conn(0x7fcc200d1110 0x7fcc200d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc17fff640 1 -- 172.21.15.46:0/1235394845 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577212327 0 0) 0x7fcc200b9690 con 0x7fcc200cfcf0 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc255c7640 1 --2- 172.21.15.46:0/1235394845 >> v2:172.21.15.88:3300/0 conn(0x7fcc200d1110 0x7fcc200d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc255c7640 1 -- 172.21.15.46:0/1235394845 >> v1:172.21.15.46:6789/0 conn(0x7fcc200cfcf0 legacy=0x7fcc2009cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc255c7640 1 -- 172.21.15.46:0/1235394845 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2008cfb0 con 0x7fcc200d1740 2024-05-09T03:17:39.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.922+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/1235394845 >> v2:172.21.15.88:3300/0 conn(0x7fcc200d1110 0x7fcc200d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.923+0000 7fcc255c7640 1 --2- 172.21.15.46:0/1235394845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1740 0x7fcc200b73d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fcc200d1b80 tx=0x7fcc1c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41b134f2ab86be4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.923+0000 7fcc17fff640 1 -- 172.21.15.46:0/1235394845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c007740 con 0x7fcc200d1740 2024-05-09T03:17:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.923+0000 7fcc17fff640 1 -- 172.21.15.46:0/1235394845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc1c0078e0 con 0x7fcc200d1740 2024-05-09T03:17:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.923+0000 7fcc17fff640 1 -- 172.21.15.46:0/1235394845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c007cb0 con 0x7fcc200d1740 2024-05-09T03:17:39.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 -- 172.21.15.46:0/1235394845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1740 msgr2=0x7fcc200b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1235394845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1740 0x7fcc200b73d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fcc200d1b80 tx=0x7fcc1c02a680 comp rx=0 tx=0).stop 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 -- 172.21.15.46:0/1235394845 shutdown_connections 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1235394845 >> v2:172.21.15.88:3300/0 conn(0x7fcc200d1110 0x7fcc200d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1235394845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1740 0x7fcc200b73d0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 -- 172.21.15.46:0/1235394845 >> 172.21.15.46:0/1235394845 conn(0x7fcc200d05f0 msgr2=0x7fcc200d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 -- 172.21.15.46:0/1235394845 shutdown_connections 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.924+0000 7fcc2782b640 1 -- 172.21.15.46:0/1235394845 wait complete. 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.926+0000 7fcc2782b640 1 Processor -- start 2024-05-09T03:17:39.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.926+0000 7fcc2782b640 1 -- start start 2024-05-09T03:17:39.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc2782b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc200cfcf0 0x7fcc2014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc2782b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 0x7fcc201587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc2782b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc200d1740 0x7fcc20158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc2782b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2009d3b0 con 0x7fcc200d1110 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc2782b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200b93b0 con 0x7fcc200cfcf0 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc2782b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcc200d2930 con 0x7fcc200d1740 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc24dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 0x7fcc201587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.927+0000 7fcc24dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 0x7fcc201587c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38704/0 (socket says 172.21.15.46:38704) 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 -- 172.21.15.46:0/1203081124 learned_addr learned my addr 172.21.15.46:0/1203081124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc255c7640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc200cfcf0 0x7fcc2014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc200d1740 0x7fcc20158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 -- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc200d1740 msgr2=0x7fcc20158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc200d1740 0x7fcc20158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 -- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc200cfcf0 msgr2=0x7fcc2014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc200cfcf0 0x7fcc2014b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 -- 172.21.15.46:0/1203081124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc201f4150 con 0x7fcc200d1110 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc255c7640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc200cfcf0 0x7fcc2014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 0x7fcc201587c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c7c0 tx=0x7fcc1000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc15ffb640 1 -- 172.21.15.46:0/1203081124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10015070 con 0x7fcc200d1110 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc15ffb640 1 -- 172.21.15.46:0/1203081124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10007820 con 0x7fcc200d1110 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc15ffb640 1 -- 172.21.15.46:0/1203081124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10007bf0 con 0x7fcc200d1110 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc20157a00 con 0x7fcc200d1110 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.928+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc20157fc0 con 0x7fcc200d1110 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.930+0000 7fcc15ffb640 1 -- 172.21.15.46:0/1203081124 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcc10018070 con 0x7fcc200d1110 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.931+0000 7fcc15ffb640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcc040762d0 0x7fcc040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.931+0000 7fcc15ffb640 1 -- 172.21.15.46:0/1203081124 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fcc10097880 con 0x7fcc200d1110 2024-05-09T03:17:39.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.931+0000 7fcc255c7640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcc040762d0 0x7fcc040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.931+0000 7fcc255c7640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcc040762d0 0x7fcc040787e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c034000 tx=0x7fcc1c033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.931+0000 7fcc2e7c0740 1 Processor -- start 2024-05-09T03:17:39.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.932+0000 7fcc2e7c0740 1 -- start start 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad2219eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad2234be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad2235120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558ad21524e0 con 0x558ad1e213b0 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558ad20edff0 con 0x558ad1e4a270 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558ad214e4a0 con 0x558ad1e29590 2024-05-09T03:17:39.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad2234be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad2234be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38708/0 (socket says 172.21.15.46:38708) 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 -- 172.21.15.46:0/569661837 learned_addr learned my addr 172.21.15.46:0/569661837 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc255c7640 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad2219eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad2235120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 -- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 msgr2=0x558ad2235120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad2235120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 -- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 msgr2=0x558ad2219eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad2219eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 -- 172.21.15.46:0/569661837 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ad20d4720 con 0x558ad1e213b0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad2235120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad2234be0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fcc100047b0 tx=0x7fcc100602f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe11766860aeb8a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.933+0000 7fcc02ffd640 1 -- 172.21.15.46:0/569661837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1001f040 con 0x558ad1e213b0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc02ffd640 1 -- 172.21.15.46:0/569661837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10064a50 con 0x558ad1e213b0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc02ffd640 1 -- 172.21.15.46:0/569661837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10069470 con 0x558ad1e213b0 2024-05-09T03:17:39.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/569661837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 msgr2=0x558ad2234be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad2234be0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fcc100047b0 tx=0x7fcc100602f0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/569661837 shutdown_connections 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad2235120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad2234be0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/569661837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad2219eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/569661837 >> 172.21.15.46:0/569661837 conn(0x558ad1e3e3c0 msgr2=0x558ad21ef510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/569661837 shutdown_connections 2024-05-09T03:17:39.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.934+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/569661837 wait complete. 2024-05-09T03:17:39.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.935+0000 7fcc2e7c0740 1 -- start start 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.935+0000 7fcc2e7c0740 1 Processor -- start 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad221da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad222dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad222bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558ad20d9760 con 0x558ad1e213b0 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558ad22314f0 con 0x558ad1e4a270 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558ad221bb60 con 0x558ad1e29590 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc2e7c0740 10 client.0 _ll_register_callbacks cb 0x558ad20fc4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad221da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad221da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38712/0 (socket says 172.21.15.46:38712) 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 -- 172.21.15.46:0/714137708 learned_addr learned my addr 172.21.15.46:0/714137708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad222bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad222dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 -- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 msgr2=0x558ad222dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad222dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 -- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 msgr2=0x558ad222bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad222bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 -- 172.21.15.46:0/714137708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ad211abd0 con 0x558ad1e213b0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad222bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad222dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcc255c7640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad221da80 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009e80 tx=0x7fcc1c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1605e9a864471c73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.937+0000 7fcbfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcbfbfff640 1 -- 172.21.15.46:0/714137708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c04c030 con 0x558ad1e213b0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc10064a50 con 0x558ad1e213b0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcbfbfff640 1 -- 172.21.15.46:0/714137708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fcc1c03cd70 con 0x558ad1e213b0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcbfbfff640 1 -- 172.21.15.46:0/714137708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=166) ==== 24+0+0 (secure 0 0 0) 0x7fcc1c058050 con 0x558ad1e213b0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 10 client.8391 fetch_fsmap learned FSMap version 166 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=166,monmap=2+}) -- 0x558ad2100280 con 0x558ad1e213b0 2024-05-09T03:17:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcbfbfff640 1 -- 172.21.15.46:0/714137708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10064a50 con 0x558ad1e213b0 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcbfbfff640 1 -- 172.21.15.46:0/714137708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 166) ==== 3494+0+0 (secure 0 0 0) 0x7fcc1c05b020 con 0x558ad1e213b0 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 10 client.8391 fetch_fsmap finished waiting for FSMap version 166 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 10 client.8391 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 20 client.8391 populate_metadata read hostname 'smithi046' 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad1e08e40 0x558ad22301b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 4 client.8391 mds_command: new command op to 8292 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 --> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ad21600f0 con 0x558ad1e08e40 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.938+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad1e08e40 0x558ad22301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.939+0000 7fcbfbfff640 10 client.8391 ms_handle_connect on v2:172.21.15.46:6832/1486786164 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.939+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad1e08e40 0x558ad22301b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b066bc779235e3 server_cookie=15ba9ad7452d60bc in_seq=0 out_seq=0 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.939+0000 7fcbfbfff640 1 -- 172.21.15.46:0/714137708 <== mds.0 v2:172.21.15.46:6832/1486786164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558ad21600f0 con 0x558ad1e08e40 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.939+0000 7fcbfbfff640 10 client.8391 handle_command_reply: tid=0 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.939+0000 7fcc2e7c0740 1 client.8391 shutdown 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 msgr2=0x558ad221da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad221da80 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009e80 tx=0x7fcc1c004ab0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 shutdown_connections 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad1e08e40 0x558ad22301b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e4a270 0x558ad222bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e29590 0x558ad222dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcbfbfff640 0 client.8391 ms_handle_reset on v2:172.21.15.46:6832/1486786164 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/714137708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e213b0 0x558ad221da80 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 >> 172.21.15.46:0/714137708 conn(0x558ad1e3e3c0 msgr2=0x558ad22166e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 shutdown_connections 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/714137708 wait complete. 2024-05-09T03:17:39.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.940+0000 7fcc2e7c0740 20 client.8391 trim_cache size 0 max 16384 2024-05-09T03:17:39.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.959+0000 7fcc2e7c0740 1 Processor -- start 2024-05-09T03:17:39.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.959+0000 7fcc2e7c0740 1 -- start start 2024-05-09T03:17:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e213b0 0x558ad222ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e29590 0x558ad222d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 0x558ad2216660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcc20200450 con 0x558ad1e4a270 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcc201f8640 con 0x558ad1e29590 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcc201ffe90 con 0x558ad1e213b0 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc25dc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 0x558ad2216660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc25dc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 0x558ad2216660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38722/0 (socket says 172.21.15.46:38722) 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc25dc8640 1 -- 172.21.15.46:0/2409677163 learned_addr learned my addr 172.21.15.46:0/2409677163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e29590 0x558ad222d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc255c7640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e213b0 0x558ad222ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.960+0000 7fcc25dc8640 1 -- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e213b0 msgr2=0x558ad222ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e213b0 0x558ad222ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc25dc8640 1 -- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e29590 msgr2=0x558ad222d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e29590 0x558ad222d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc25dc8640 1 -- 172.21.15.46:0/2409677163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ad1eb5850 con 0x558ad1e4a270 2024-05-09T03:17:39.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc255c7640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e213b0 0x558ad222ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e29590 0x558ad222d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 0x558ad2216660 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fcc18007570 tx=0x7fcc18005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f5f136a98a6aa8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc027fc640 1 -- 172.21.15.46:0/2409677163 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18010040 con 0x558ad1e4a270 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc027fc640 1 -- 172.21.15.46:0/2409677163 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc1801a070 con 0x558ad1e4a270 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.961+0000 7fcc027fc640 1 -- 172.21.15.46:0/2409677163 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18001970 con 0x558ad1e4a270 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/2409677163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 msgr2=0x558ad2216660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 0x558ad2216660 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fcc18007570 tx=0x7fcc18005bc0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/2409677163 shutdown_connections 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e4a270 0x558ad2216660 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e29590 0x558ad222d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/2409677163 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e213b0 0x558ad222ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/2409677163 >> 172.21.15.46:0/2409677163 conn(0x558ad1e3e3c0 msgr2=0x558ad22159b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/2409677163 shutdown_connections 2024-05-09T03:17:39.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.962+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/2409677163 wait complete. 2024-05-09T03:17:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.963+0000 7fcc2e7c0740 1 -- start start 2024-05-09T03:17:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.963+0000 7fcc2e7c0740 1 Processor -- start 2024-05-09T03:17:39.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e213b0 0x558ad21edbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 0x558ad2247100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e4a270 0x558ad21efc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558ad222c910 con 0x558ad1e29590 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558ad221b7d0 con 0x558ad1e213b0 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558ad2247a90 con 0x558ad1e4a270 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc2e7c0740 10 client.0 _ll_register_callbacks cb 0x7fcc1c009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 0x558ad2247100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 0x558ad2247100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38736/0 (socket says 172.21.15.46:38736) 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 -- 172.21.15.46:0/3486263521 learned_addr learned my addr 172.21.15.46:0/3486263521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc255c7640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e213b0 0x558ad21edbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e4a270 0x558ad21efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 -- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e4a270 msgr2=0x558ad21efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e4a270 0x558ad21efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 -- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e213b0 msgr2=0x558ad21edbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.964+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e213b0 0x558ad21edbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc24dc6640 1 -- 172.21.15.46:0/3486263521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ad1f2c6b0 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc25dc8640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e4a270 0x558ad21efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc24dc6640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 0x558ad2247100 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fcc100959c0 tx=0x7fcc100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f9d34f05593e8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcbfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3486263521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1001f040 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3486263521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fcc10065be0 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc1801a070 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3486263521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=166) ==== 24+0+0 (secure 0 0 0) 0x7fcc10064a50 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 10 client.8397 fetch_fsmap learned FSMap version 166 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=166,monmap=2+}) -- 0x558ad210e630 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3486263521 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc1801a070 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3486263521 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 166) ==== 3494+0+0 (secure 0 0 0) 0x7fcc1005f900 con 0x558ad1e29590 2024-05-09T03:17:39.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 10 client.8397 fetch_fsmap finished waiting for FSMap version 166 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 10 client.8397 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 20 client.8397 populate_metadata read hostname 'smithi046' 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad22b1e20 0x558ad22b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 4 client.8397 mds_command: new command op to 8292 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.965+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 --> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] -- command(tid 0: {"prefix": "status"}) -- 0x558ad2159f60 con 0x558ad22b1e20 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc255c7640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad22b1e20 0x558ad22b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcbfbfff640 10 client.8397 ms_handle_connect on v2:172.21.15.46:6832/1486786164 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc255c7640 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad22b1e20 0x558ad22b2230 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a83527f2d2d15d53 server_cookie=cc09835458f795b5 in_seq=0 out_seq=0 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcbfbfff640 1 -- 172.21.15.46:0/3486263521 <== mds.0 v2:172.21.15.46:6832/1486786164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x558ad2159f60 con 0x558ad22b1e20 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcbfbfff640 10 client.8397 handle_command_reply: tid=0 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 client.8397 shutdown 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 msgr2=0x558ad2247100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 0x558ad2247100 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fcc100959c0 tx=0x7fcc100046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 shutdown_connections 2024-05-09T03:17:39.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:6832/1486786164,v1:172.21.15.46:6834/1486786164] conn(0x558ad22b1e20 0x558ad22b2230 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558ad1e4a270 0x558ad21efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558ad1e29590 0x558ad2247100 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcbfbfff640 0 client.8397 ms_handle_reset on v2:172.21.15.46:6832/1486786164 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 --2- 172.21.15.46:0/3486263521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558ad1e213b0 0x558ad21edbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.966+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 >> 172.21.15.46:0/3486263521 conn(0x558ad1e3e3c0 msgr2=0x558ad222ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 shutdown_connections 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2e7c0740 1 -- 172.21.15.46:0/3486263521 wait complete. 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2e7c0740 20 client.8397 trim_cache size 0 max 16384 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8292, 2024-05-09T03:17:39.972 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7443322820000002, 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 166, 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 96, 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 96, 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.704500648 2024-05-09T03:17:39.973 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcc040762d0 msgr2=0x7fcc040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcc040762d0 0x7fcc040787e0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c034000 tx=0x7fcc1c033040 comp rx=0 tx=0).stop 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 msgr2=0x7fcc201587c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 0x7fcc201587c0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c7c0 tx=0x7fcc1000cc80 comp rx=0 tx=0).stop 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 shutdown_connections 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcc200d1740 0x7fcc20158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcc040762d0 0x7fcc040787e0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc200d1110 0x7fcc201587c0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 --2- 172.21.15.46:0/1203081124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc200cfcf0 0x7fcc2014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 >> 172.21.15.46:0/1203081124 conn(0x7fcc200d05f0 msgr2=0x7fcc2015e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 shutdown_connections 2024-05-09T03:17:39.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:39.967+0000 7fcc2782b640 1 -- 172.21.15.46:0/1203081124 wait complete. 2024-05-09T03:17:39.979 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:17:39.979 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 400.00000000', proceeding to ls 2024-05-09T03:17:39.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:17:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.093+0000 7f02874a8640 1 Processor -- start 2024-05-09T03:17:40.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.093+0000 7f02874a8640 1 -- start start 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f02874a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02800b7ef0 0x7f028016b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f02874a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02800d49e0 con 0x7f02800d57a0 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f02874a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02800d4cc0 con 0x7f0280160610 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f02874a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02800d5bb0 con 0x7f02800b7ef0 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f0285244640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02800d57a0 0x7f02800b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54028/0 (socket says 172.21.15.46:54028) 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f0285244640 1 -- 172.21.15.46:0/3903056599 learned_addr learned my addr 172.21.15.46:0/3903056599 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f0285a45640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02800b7ef0 0x7f028016b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2345969681 0 0) 0x7f02800d49e0 con 0x7f02800d57a0 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0268003520 con 0x7f02800d57a0 2024-05-09T03:17:40.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2713393508 0 0) 0x7f0268003520 con 0x7f02800d57a0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02800b7ef0 msgr2=0x7f028016b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02800b7ef0 0x7f028016b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 >> v1:172.21.15.88:6789/0 conn(0x7f0280160610 legacy=0x7f02800b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800d4fa0 con 0x7f02800d57a0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.095+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f0270001a70 con 0x7f02800d57a0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 >> v1:172.21.15.46:6789/0 conn(0x7f02800d57a0 legacy=0x7f02800b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02680039e0 0x7f0268003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680043a0 0x7f02680048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02680050c0 0x7f026800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02800d5bb0 con 0x7f02680050c0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f026800bf80 con 0x7f02680043a0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f026800c180 con 0x7f02680039e0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285a45640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02680050c0 0x7f026800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285244640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02680039e0 0x7f0268003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0284a43640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680043a0 0x7f02680048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285a45640 1 -- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02680039e0 msgr2=0x7f0268003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285a45640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02680039e0 0x7f0268003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285a45640 1 -- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680043a0 msgr2=0x7f02680048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285a45640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680043a0 0x7f02680048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.096+0000 7f0285a45640 1 -- 172.21.15.46:0/3903056599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0268003520 con 0x7f02680050c0 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.097+0000 7f0285a45640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02680050c0 0x7f026800b9c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0278009a90 tx=0x7f0268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e919abee046a3e09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.097+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278036070 con 0x7f02680050c0 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.097+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0278031400 con 0x7f02680050c0 2024-05-09T03:17:40.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.097+0000 7f027ffff640 1 -- 172.21.15.46:0/3903056599 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278009690 con 0x7f02680050c0 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f02874a8640 1 -- 172.21.15.46:0/3903056599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02680050c0 msgr2=0x7f026800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f02874a8640 1 --2- 172.21.15.46:0/3903056599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02680050c0 0x7f026800b9c0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f0278009a90 tx=0x7f0268005830 comp rx=0 tx=0).stop 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f0285a45640 1 -- 172.21.15.46:0/3903056599 reap_dead start 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f02874a8640 1 -- 172.21.15.46:0/3903056599 shutdown_connections 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f02874a8640 1 -- 172.21.15.46:0/3903056599 >> 172.21.15.46:0/3903056599 conn(0x7f02800743e0 msgr2=0x7f02800d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f02874a8640 1 -- 172.21.15.46:0/3903056599 shutdown_connections 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.098+0000 7f02874a8640 1 -- 172.21.15.46:0/3903056599 wait complete. 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.100+0000 7f02874a8640 1 Processor -- start 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.100+0000 7f02874a8640 1 -- start start 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f02874a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680050c0 0x7f02801f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f02874a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02801f5890 0x7f028015ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f02874a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 0x7f0280155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f02874a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02800d15b0 con 0x7f028015bfb0 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f02874a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f02800d52e0 con 0x7f02680050c0 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f02874a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02800d1920 con 0x7f02801f5890 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 0x7f0280155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 0x7f0280155dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38748/0 (socket says 172.21.15.46:38748) 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 -- 172.21.15.46:0/3744370415 learned_addr learned my addr 172.21.15.46:0/3744370415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285244640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680050c0 0x7f02801f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0284a43640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02801f5890 0x7f028015ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 -- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02801f5890 msgr2=0x7f028015ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02801f5890 0x7f028015ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 -- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680050c0 msgr2=0x7f02801f9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680050c0 0x7f02801f9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.101+0000 7f0285a45640 1 -- 172.21.15.46:0/3744370415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800b67d0 con 0x7f028015bfb0 2024-05-09T03:17:40.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.102+0000 7f0285a45640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 0x7f0280155dd0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f0278034040 tx=0x7f0278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.102+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278036070 con 0x7f028015bfb0 2024-05-09T03:17:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.102+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0278041070 con 0x7f028015bfb0 2024-05-09T03:17:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.102+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028015c3c0 con 0x7f028015bfb0 2024-05-09T03:17:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.102+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027803b860 con 0x7f028015bfb0 2024-05-09T03:17:40.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.102+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0280157fe0 con 0x7f028015bfb0 2024-05-09T03:17:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.103+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f028006a270 con 0x7f028015bfb0 2024-05-09T03:17:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.104+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f027803ba00 con 0x7f028015bfb0 2024-05-09T03:17:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.104+0000 7f027dffb640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f026c076300 0x7f026c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.105+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f02780b7d40 con 0x7f028015bfb0 2024-05-09T03:17:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.105+0000 7f0285244640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f026c076300 0x7f026c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:40.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.105+0000 7f0285244640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f026c076300 0x7f026c078810 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0270002230 tx=0x7f0270006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.109+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f028015c3c0 con 0x7f028015bfb0 2024-05-09T03:17:40.206 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:40.206+0000 7f2ef63e5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:17:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.338+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0280162690 con 0x7f028015bfb0 2024-05-09T03:17:40.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.338+0000 7f027dffb640 1 -- 172.21.15.46:0/3744370415 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v166) ==== 53+0+73 (secure 0 0 0) 0x7f02780b7070 con 0x7f028015bfb0 2024-05-09T03:17:40.339 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:17:40.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.340+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f026c076300 msgr2=0x7f026c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.340+0000 7f02874a8640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f026c076300 0x7f026c078810 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0270002230 tx=0x7f0270006040 comp rx=0 tx=0).stop 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.340+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 msgr2=0x7f0280155dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.340+0000 7f02874a8640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 0x7f0280155dd0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f0278034040 tx=0x7f0278004730 comp rx=0 tx=0).stop 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.340+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 shutdown_connections 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f026c076300 0x7f026c078810 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f028015bfb0 0x7f0280155dd0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02801f5890 0x7f028015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 --2- 172.21.15.46:0/3744370415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02680050c0 0x7f02801f9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 >> 172.21.15.46:0/3744370415 conn(0x7f02800743e0 msgr2=0x7f0280161620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 shutdown_connections 2024-05-09T03:17:40.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40.341+0000 7f02874a8640 1 -- 172.21.15.46:0/3744370415 wait complete. 2024-05-09T03:17:40.352 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:17:40.352 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:17:40.352 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:17:40.353 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:17:40.353 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:40.353 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:17:40.353 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:17:40.353 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:17:40.353 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:17:40.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:40.353 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:17:40.369 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:17:40.369 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:17:40.369 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:17:40.369 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:17:40.370 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42057sec preferred_lft 42057sec 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.371 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:17:40.372 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:40.372 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:17:40.372 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:17:40.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:40.372 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:17:40.434 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:17:40.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:40.435 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:17:40.492 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:17:40.492 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:17:40.492 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:17:40.492 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:17:40.492 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:17:40.492 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:17:40.493 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:17:40.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:17:40.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:17:40.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:17:40.493 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:17:40.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:17:40.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:17:40.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:40.623 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:17:40.638 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:17:40.638 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:17:40.639 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:17:40.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:40.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:17:40.706 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:17:40.706 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:17:40.706 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:40.762 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:17:40.763 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:17:40.763 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:40.818 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:17:40.884 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:17:40.941 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:17:40.941 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:17:40.941 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:17:40.941 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:17:40.941 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:17:40.942 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:40.943 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:40.944 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:40.944 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:40.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:40.944 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:17:40.997 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:17:40.997 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:17:41.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:17:41.127 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:17:41.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:17:41.134+0000 7f85153ea480 -1 init, newargv = 0x7f8504001250 newargc=15 2024-05-09T03:17:41.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[71208]: starting ceph client 2024-05-09T03:17:41.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[71208]: starting fuse 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:17:41.152 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:17:41.153 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:41.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:41.154 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:17:41.208 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:17:41.208 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:41.209 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:17:41.210 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:17:41.329 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.71208.asok 2024-05-09T03:17:41.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:17:41.339 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.71208.asok 2024-05-09T03:17:41.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:41.339 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.71208.asok status 2024-05-09T03:17:41.487 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "71208", 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:17:41.488 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8409, 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 8409 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1489343860 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:17:41.489 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1489343860 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.8409 192.168.0.1:0/1489343860", 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1489343860", 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 166, 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 96, 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:17:41.490 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:17:41.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:41.496 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:17:41.496 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:41.513 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:17:41.513 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:41.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:41.513 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:41.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:41.579 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:17:41.624 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:17:41.642 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:17:41.642 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:17:41.642 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:41.643 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:17:41.643 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs 2024-05-09T03:17:41.643 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 400.00000000' did not caused DAMAGED state 2024-05-09T03:17:41.643 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 100.00000000.inode' 2024-05-09T03:17:41.643 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:41.643 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:41.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:17:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.854+0000 7fa6ed005640 1 Processor -- start 2024-05-09T03:17:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.854+0000 7fa6ed005640 1 -- start start 2024-05-09T03:17:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6ed005640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80d2e90 0x7fa6e80b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6ed005640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 0x7fa6e80b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6ed005640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6e80d3400 con 0x7fa6e80b4fc0 2024-05-09T03:17:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6ed005640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa6e80d4a60 con 0x7fa6e81605b0 2024-05-09T03:17:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6ed005640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa6e80d4d40 con 0x7fa6e80d2e90 2024-05-09T03:17:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 0x7fa6e80b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e6d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80d2e90 0x7fa6e80b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 0x7fa6e80b7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:48778/0 (socket says 172.21.15.46:48778) 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e759e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa6e80b4fc0 0x7fa6e80d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:44552/0 (socket says 172.21.15.46:44552) 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e659c640 1 -- 172.21.15.46:0/329356968 learned_addr learned my addr 172.21.15.46:0/329356968 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e6d9d640 1 -- 172.21.15.46:0/329356968 >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 msgr2=0x7fa6e80b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e6d9d640 1 --2- 172.21.15.46:0/329356968 >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 0x7fa6e80b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e6d9d640 1 -- 172.21.15.46:0/329356968 >> v1:172.21.15.88:6789/0 conn(0x7fa6e80b4fc0 legacy=0x7fa6e80d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e6d9d640 1 -- 172.21.15.46:0/329356968 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e80d5020 con 0x7fa6e80d2e90 2024-05-09T03:17:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.856+0000 7fa6e659c640 1 --2- 172.21.15.46:0/329356968 >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 0x7fa6e80b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.857+0000 7fa6e6d9d640 1 --2- 172.21.15.46:0/329356968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80d2e90 0x7fa6e80b7760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009d90 tx=0x7fa6d802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54950b44b7daee4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.857+0000 7fa6e5d9b640 1 -- 172.21.15.46:0/329356968 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8031600 con 0x7fa6e80d2e90 2024-05-09T03:17:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.857+0000 7fa6e5d9b640 1 -- 172.21.15.46:0/329356968 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6d80317a0 con 0x7fa6e80d2e90 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.857+0000 7fa6e5d9b640 1 -- 172.21.15.46:0/329356968 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8031b70 con 0x7fa6e80d2e90 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 -- 172.21.15.46:0/329356968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80d2e90 msgr2=0x7fa6e80b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 --2- 172.21.15.46:0/329356968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80d2e90 0x7fa6e80b7760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009d90 tx=0x7fa6d802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 -- 172.21.15.46:0/329356968 shutdown_connections 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 --2- 172.21.15.46:0/329356968 >> v2:172.21.15.88:3300/0 conn(0x7fa6e81605b0 0x7fa6e80b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 --2- 172.21.15.46:0/329356968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80d2e90 0x7fa6e80b7760 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 -- 172.21.15.46:0/329356968 >> 172.21.15.46:0/329356968 conn(0x7fa6e808d390 msgr2=0x7fa6e808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 -- 172.21.15.46:0/329356968 shutdown_connections 2024-05-09T03:17:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.858+0000 7fa6ed005640 1 -- 172.21.15.46:0/329356968 wait complete. 2024-05-09T03:17:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.860+0000 7fa6ed005640 1 Processor -- start 2024-05-09T03:17:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.860+0000 7fa6ed005640 1 -- start start 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.861+0000 7fa6ed005640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80b4fc0 0x7fa6e81f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.861+0000 7fa6ed005640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e80d2e90 0x7fa6e80c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6ed005640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 0x7fa6e81f9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6ed005640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e80d5540 con 0x7fa6e81605b0 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6ed005640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e80b4c00 con 0x7fa6e80d2e90 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6ed005640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa6e80ce7d0 con 0x7fa6e80b4fc0 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 0x7fa6e81f9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 0x7fa6e81f9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38764/0 (socket says 172.21.15.46:38764) 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 -- 172.21.15.46:0/236399678 learned_addr learned my addr 172.21.15.46:0/236399678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e6d9d640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80b4fc0 0x7fa6e81f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e659c640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e80d2e90 0x7fa6e80c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 -- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80b4fc0 msgr2=0x7fa6e81f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:41.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80b4fc0 0x7fa6e81f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 -- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e80d2e90 msgr2=0x7fa6e80c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e80d2e90 0x7fa6e80c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 -- 172.21.15.46:0/236399678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e8162840 con 0x7fa6e81605b0 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6e759e640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 0x7fa6e81f9f20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000c390 tx=0x7fa6e000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00074e0 con 0x7fa6e81605b0 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e0007680 con 0x7fa6e81605b0 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.862+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e8162b20 con 0x7fa6e81605b0 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.863+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00049e0 con 0x7fa6e81605b0 2024-05-09T03:17:41.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.863+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e81630e0 con 0x7fa6e81605b0 2024-05-09T03:17:41.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.863+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e806a270 con 0x7fa6e81605b0 2024-05-09T03:17:41.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.864+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa6e001a020 con 0x7fa6e81605b0 2024-05-09T03:17:41.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.865+0000 7fa6df7fe640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6c80762d0 0x7fa6c80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:41.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.865+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7fa6e00970d0 con 0x7fa6e81605b0 2024-05-09T03:17:41.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.865+0000 7fa6e6d9d640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6c80762d0 0x7fa6c80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:41.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.866+0000 7fa6e6d9d640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6c80762d0 0x7fa6c80787e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8034000 tx=0x7fa6d802f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:41.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:41.870+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa6e005ff10 con 0x7fa6e81605b0 2024-05-09T03:17:42.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.075+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa6e81625d0 con 0x7fa6e81605b0 2024-05-09T03:17:42.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.076+0000 7fa6df7fe640 1 -- 172.21.15.46:0/236399678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v96) ==== 81+0+1462 (secure 0 0 0) 0x7fa6e0064210 con 0x7fa6e81605b0 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:17:42.077 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:42.078 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:42.079 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:42.079 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:listed 24 entries 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6c80762d0 msgr2=0x7fa6c80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6c80762d0 0x7fa6c80787e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8034000 tx=0x7fa6d802f000 comp rx=0 tx=0).stop 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 msgr2=0x7fa6e81f9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 0x7fa6e81f9f20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000c390 tx=0x7fa6e000c850 comp rx=0 tx=0).stop 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 shutdown_connections 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6c80762d0 0x7fa6c80787e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6e81605b0 0x7fa6e81f9f20 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6e80d2e90 0x7fa6e80c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 --2- 172.21.15.46:0/236399678 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6e80b4fc0 0x7fa6e81f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 >> 172.21.15.46:0/236399678 conn(0x7fa6e808d390 msgr2=0x7fa6e80b6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 shutdown_connections 2024-05-09T03:17:42.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.078+0000 7fa6ed005640 1 -- 172.21.15.46:0/236399678 wait complete. 2024-05-09T03:17:42.090 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:42.090 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:42.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:17:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.261+0000 7f65fd584640 1 Processor -- start 2024-05-09T03:17:42.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.261+0000 7f65fd584640 1 -- start start 2024-05-09T03:17:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65fd584640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65f80b86f0 0x7f65f8169190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65fd584640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f80b7220 con 0x7f65f8070230 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65fd584640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f80d1bf0 con 0x7f65f815e460 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65fd584640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f65f80d1ed0 con 0x7f65f80b86f0 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65f77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65f80b86f0 0x7f65f8169190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65effff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f65f8070230 0x7f65f8070640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54040/0 (socket says 172.21.15.46:54040) 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.262+0000 7f65effff640 1 -- 172.21.15.46:0/1754107954 learned_addr learned my addr 172.21.15.46:0/1754107954 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.263+0000 7f65f77fe640 1 -- 172.21.15.46:0/1754107954 >> v1:172.21.15.88:6789/0 conn(0x7f65f815e460 legacy=0x7f65f80b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.263+0000 7f65f77fe640 1 -- 172.21.15.46:0/1754107954 >> v1:172.21.15.46:6789/0 conn(0x7f65f8070230 legacy=0x7f65f8070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.263+0000 7f65f77fe640 1 -- 172.21.15.46:0/1754107954 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f80d21b0 con 0x7f65f80b86f0 2024-05-09T03:17:42.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.264+0000 7f65f77fe640 1 --2- 172.21.15.46:0/1754107954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f80b86f0 0x7f65f8169190 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009d90 tx=0x7f65f002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fcf6ac195ba4a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.264+0000 7f65f67fc640 1 -- 172.21.15.46:0/1754107954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0007740 con 0x7f65f80b86f0 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.264+0000 7f65f67fc640 1 -- 172.21.15.46:0/1754107954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f00078e0 con 0x7f65f80b86f0 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.264+0000 7f65f67fc640 1 -- 172.21.15.46:0/1754107954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0007cb0 con 0x7f65f80b86f0 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.264+0000 7f65fd584640 1 -- 172.21.15.46:0/1754107954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f80b86f0 msgr2=0x7f65f8169190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.264+0000 7f65fd584640 1 --2- 172.21.15.46:0/1754107954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f80b86f0 0x7f65f8169190 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009d90 tx=0x7f65f002a660 comp rx=0 tx=0).stop 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.265+0000 7f65fd584640 1 -- 172.21.15.46:0/1754107954 shutdown_connections 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.265+0000 7f65fd584640 1 --2- 172.21.15.46:0/1754107954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f80b86f0 0x7f65f8169190 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.265+0000 7f65fd584640 1 -- 172.21.15.46:0/1754107954 >> 172.21.15.46:0/1754107954 conn(0x7f65f80d3b90 msgr2=0x7f65f80d3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.265+0000 7f65fd584640 1 -- 172.21.15.46:0/1754107954 shutdown_connections 2024-05-09T03:17:42.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.265+0000 7f65fd584640 1 -- 172.21.15.46:0/1754107954 wait complete. 2024-05-09T03:17:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.266+0000 7f65fd584640 1 Processor -- start 2024-05-09T03:17:42.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.266+0000 7f65fd584640 1 -- start start 2024-05-09T03:17:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65fd584640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65f8070230 0x7f65f80c13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65fd584640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f80b86f0 0x7f65f8155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65fd584640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 0x7f65f81613e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65fd584640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f65f815e870 con 0x7f65f815e460 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65fd584640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f65f80b3400 con 0x7f65f80b86f0 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65fd584640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f65f815e9f0 con 0x7f65f8070230 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 0x7f65f81613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 0x7f65f81613e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38770/0 (socket says 172.21.15.46:38770) 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 -- 172.21.15.46:0/3034309979 learned_addr learned my addr 172.21.15.46:0/3034309979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f6ffd640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65f8070230 0x7f65f80c13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65effff640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f80b86f0 0x7f65f8155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 -- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65f8070230 msgr2=0x7f65f80c13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65f8070230 0x7f65f80c13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 -- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f80b86f0 msgr2=0x7f65f8155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f80b86f0 0x7f65f8155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 -- 172.21.15.46:0/3034309979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f81e9e50 con 0x7f65f815e460 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f6ffd640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65f8070230 0x7f65f80c13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.268+0000 7f65f77fe640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 0x7f65f81613e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009eb0 tx=0x7f65f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.269+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0040070 con 0x7f65f815e460 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.269+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f0031c90 con 0x7f65f815e460 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.269+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f003c640 con 0x7f65f815e460 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.269+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f815f740 con 0x7f65f815e460 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.269+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f8160be0 con 0x7f65f815e460 2024-05-09T03:17:42.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.270+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f806a270 con 0x7f65f815e460 2024-05-09T03:17:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.270+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f65f0039040 con 0x7f65f815e460 2024-05-09T03:17:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.271+0000 7f65ef7fe640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65d8076300 0x7f65d8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.271+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f65f00b8f10 con 0x7f65f815e460 2024-05-09T03:17:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.271+0000 7f65f6ffd640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65d8076300 0x7f65d8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.272+0000 7f65f6ffd640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65d8076300 0x7f65d8078810 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f65f808d150 tx=0x7f65e8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:42.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.276+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f65f0082580 con 0x7f65f815e460 2024-05-09T03:17:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.480+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f65f81e7a70 con 0x7f65f815e460 2024-05-09T03:17:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.481+0000 7f65ef7fe640 1 -- 172.21.15.46:0/3034309979 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v96) ==== 81+0+1462 (secure 0 0 0) 0x7f65f0081e80 con 0x7f65f815e460 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:42.482 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:17:42.483 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:17:42.484 INFO:teuthology.orchestra.run.smithi046.stderr:listed 24 entries 2024-05-09T03:17:42.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65d8076300 msgr2=0x7f65d8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65d8076300 0x7f65d8078810 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f65f808d150 tx=0x7f65e8008040 comp rx=0 tx=0).stop 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 msgr2=0x7f65f81613e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 0x7f65f81613e0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009eb0 tx=0x7f65f0004a40 comp rx=0 tx=0).stop 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 shutdown_connections 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65d8076300 0x7f65d8078810 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f815e460 0x7f65f81613e0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65f80b86f0 0x7f65f8155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 --2- 172.21.15.46:0/3034309979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65f8070230 0x7f65f80c13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 >> 172.21.15.46:0/3034309979 conn(0x7f65f80d3b90 msgr2=0x7f65f80ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 shutdown_connections 2024-05-09T03:17:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.483+0000 7f65fd584640 1 -- 172.21.15.46:0/3034309979 wait complete. 2024-05-09T03:17:42.495 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:17:42.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:42.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:42.521 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:17:42.521 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:42.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:42.521 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:42.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.733+0000 7f885efa8640 1 Processor -- start 2024-05-09T03:17:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.733+0000 7f885efa8640 1 -- start start 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885efa8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88581605b0 0x7f88580b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885efa8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88580d3400 con 0x7f88580d2e90 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885efa8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88580d4a60 con 0x7f88580b7e70 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885efa8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f88580d4d40 con 0x7f88581605b0 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885d545640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88581605b0 0x7f88580b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885cd44640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88580d2e90 0x7f88580b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54046/0 (socket says 172.21.15.46:54046) 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885cd44640 1 -- 172.21.15.46:0/3209711117 learned_addr learned my addr 172.21.15.46:0/3209711117 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885d545640 1 -- 172.21.15.46:0/3209711117 >> v1:172.21.15.88:6789/0 conn(0x7f88580b7e70 legacy=0x7f88580b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885d545640 1 -- 172.21.15.46:0/3209711117 >> v1:172.21.15.46:6789/0 conn(0x7f88580d2e90 legacy=0x7f88580b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f88577fe640 1 -- 172.21.15.46:0/3209711117 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957548827 0 0) 0x7f88580d3400 con 0x7f88580d2e90 2024-05-09T03:17:42.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.735+0000 7f885d545640 1 -- 172.21.15.46:0/3209711117 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88580d5020 con 0x7f88581605b0 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.736+0000 7f885d545640 1 --2- 172.21.15.46:0/3209711117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f88580b99b0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f88480097a0 tx=0x7f884802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9dd7c04107eb0ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.736+0000 7f88577fe640 1 -- 172.21.15.46:0/3209711117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848037070 con 0x7f88581605b0 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.736+0000 7f88577fe640 1 -- 172.21.15.46:0/3209711117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8848032440 con 0x7f88581605b0 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.736+0000 7f88577fe640 1 -- 172.21.15.46:0/3209711117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848032810 con 0x7f88581605b0 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 -- 172.21.15.46:0/3209711117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 msgr2=0x7f88580b99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 --2- 172.21.15.46:0/3209711117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f88580b99b0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f88480097a0 tx=0x7f884802a1a0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 -- 172.21.15.46:0/3209711117 shutdown_connections 2024-05-09T03:17:42.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 --2- 172.21.15.46:0/3209711117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f88580b99b0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 -- 172.21.15.46:0/3209711117 >> 172.21.15.46:0/3209711117 conn(0x7f885808d390 msgr2=0x7f885808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 -- 172.21.15.46:0/3209711117 shutdown_connections 2024-05-09T03:17:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.737+0000 7f885efa8640 1 -- 172.21.15.46:0/3209711117 wait complete. 2024-05-09T03:17:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.739+0000 7f885efa8640 1 Processor -- start 2024-05-09T03:17:42.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.739+0000 7f885efa8640 1 -- start start 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.740+0000 7f885efa8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88580d2e90 0x7f885814cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.740+0000 7f885efa8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f885814fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.740+0000 7f885efa8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88581f9340 0x7f885814e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.740+0000 7f885efa8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f88580b4d80 con 0x7f88581605b0 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.740+0000 7f885efa8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f88580b4c00 con 0x7f88580d2e90 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.740+0000 7f885efa8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f88580b4fc0 con 0x7f88581f9340 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f885814fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f885814fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38780/0 (socket says 172.21.15.46:38780) 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 -- 172.21.15.46:0/1484649303 learned_addr learned my addr 172.21.15.46:0/1484649303 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f885cd44640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88580d2e90 0x7f885814cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f885d545640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88581f9340 0x7f885814e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 -- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88581f9340 msgr2=0x7f885814e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88581f9340 0x7f885814e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 -- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88580d2e90 msgr2=0x7f885814cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88580d2e90 0x7f885814cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 -- 172.21.15.46:0/1484649303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885814d540 con 0x7f88581605b0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f885cd44640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88580d2e90 0x7f885814cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f8857fff640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f885814fa10 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f884000c7b0 tx=0x7f884000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840003b70 con 0x7f88581605b0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8840003d10 con 0x7f88581605b0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885814d820 con 0x7f88581605b0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884000aa70 con 0x7f88581605b0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.741+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885814dde0 con 0x7f88581605b0 2024-05-09T03:17:42.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.742+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885806a270 con 0x7f88581605b0 2024-05-09T03:17:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.743+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f884000ac10 con 0x7f88581605b0 2024-05-09T03:17:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.744+0000 7f88557fa640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8838076300 0x7f8838078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.744+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6799+0+0 (secure 0 0 0) 0x7f8840097a00 con 0x7f88581605b0 2024-05-09T03:17:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.744+0000 7f885cd44640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8838076300 0x7f8838078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:42.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.744+0000 7f885cd44640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8838076300 0x7f8838078810 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8850000f60 tx=0x7f8850009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:42.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.748+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8840060e70 con 0x7f88581605b0 2024-05-09T03:17:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.978+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f885814d2d0 con 0x7f88581605b0 2024-05-09T03:17:42.979 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:42.979+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:42.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.980+0000 7f88557fa640 1 -- 172.21.15.46:0/1484649303 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v167) ==== 161+0+0 (secure 0 0 0) 0x7f8840060770 con 0x7f88581605b0 2024-05-09T03:17:42.981 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8838076300 msgr2=0x7f8838078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8838076300 0x7f8838078810 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8850000f60 tx=0x7f8850009040 comp rx=0 tx=0).stop 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 msgr2=0x7f885814fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f885814fa10 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f884000c7b0 tx=0x7f884000cc70 comp rx=0 tx=0).stop 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 shutdown_connections 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88581f9340 0x7f885814e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8838076300 0x7f8838078810 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88581605b0 0x7f885814fa10 secure :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f884000c7b0 tx=0x7f884000cc70 comp rx=0 tx=0).stop 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 --2- 172.21.15.46:0/1484649303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88580d2e90 0x7f885814cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:42.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.983+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 >> 172.21.15.46:0/1484649303 conn(0x7f885808d390 msgr2=0x7f885808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.984+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 shutdown_connections 2024-05-09T03:17:42.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:42.984+0000 7f885efa8640 1 -- 172.21.15.46:0/1484649303 wait complete. 2024-05-09T03:17:42.985 INFO:tasks.ceph.mds.c.smithi046.stderr: -583> 2024-05-09T03:17:40.206+0000 7f2ef63e5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:17:42.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:43.046 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.117+0000 7f1edc5f3640 1 Processor -- start 2024-05-09T03:17:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.117+0000 7f1edc5f3640 1 -- start start 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edc5f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ed415cf80 0x7f1ed40b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edc5f3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ed40b7c70 0x7f1ed40d2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edc5f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed40d2e60 0x7f1ed40d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edc5f3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed40d37b0 con 0x7f1ed415cf80 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edc5f3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed40b8080 con 0x7f1ed40b7c70 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edc5f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed40b42c0 con 0x7f1ed40d2e60 2024-05-09T03:17:43.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ed415cf80 0x7f1ed40b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ed415cf80 0x7f1ed40b7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38788/0 (socket says 172.21.15.46:38788) 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 -- 172.21.15.46:0/963082613 learned_addr learned my addr 172.21.15.46:0/963082613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1edab90640 1 --2- 172.21.15.46:0/963082613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed40d2e60 0x7f1ed40d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1ed9b8e640 1 --2- 172.21.15.46:0/963082613 >> v2:172.21.15.88:3300/0 conn(0x7f1ed40b7c70 0x7f1ed40d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 -- 172.21.15.46:0/963082613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed40d2e60 msgr2=0x7f1ed40d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 --2- 172.21.15.46:0/963082613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed40d2e60 0x7f1ed40d3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 -- 172.21.15.46:0/963082613 >> v2:172.21.15.88:3300/0 conn(0x7f1ed40b7c70 msgr2=0x7f1ed40d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 --2- 172.21.15.46:0/963082613 >> v2:172.21.15.88:3300/0 conn(0x7f1ed40b7c70 0x7f1ed40d2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1eda38f640 1 -- 172.21.15.46:0/963082613 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed40cfbb0 con 0x7f1ed415cf80 2024-05-09T03:17:43.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.119+0000 7f1ed9b8e640 1 --2- 172.21.15.46:0/963082613 >> v2:172.21.15.88:3300/0 conn(0x7f1ed40b7c70 0x7f1ed40d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.120+0000 7f1eda38f640 1 --2- 172.21.15.46:0/963082613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed415cf80 0x7f1ed40b7730 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40a0b30 tx=0x7f1ec402c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a67a3f20adb8cdf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.120+0000 7f1ed938d640 1 -- 172.21.15.46:0/963082613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4032790 con 0x7f1ed415cf80 2024-05-09T03:17:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.120+0000 7f1ed938d640 1 -- 172.21.15.46:0/963082613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec4038070 con 0x7f1ed415cf80 2024-05-09T03:17:43.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.120+0000 7f1ed938d640 1 -- 172.21.15.46:0/963082613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4032b50 con 0x7f1ed415cf80 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 -- 172.21.15.46:0/963082613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed415cf80 msgr2=0x7f1ed40b7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/963082613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed415cf80 0x7f1ed40b7730 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40a0b30 tx=0x7f1ec402c890 comp rx=0 tx=0).stop 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 -- 172.21.15.46:0/963082613 shutdown_connections 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/963082613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed40d2e60 0x7f1ed40d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/963082613 >> v2:172.21.15.88:3300/0 conn(0x7f1ed40b7c70 0x7f1ed40d2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/963082613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed415cf80 0x7f1ed40b7730 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 -- 172.21.15.46:0/963082613 >> 172.21.15.46:0/963082613 conn(0x7f1ed409b7d0 msgr2=0x7f1ed409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 -- 172.21.15.46:0/963082613 shutdown_connections 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.121+0000 7f1edc5f3640 1 -- 172.21.15.46:0/963082613 wait complete. 2024-05-09T03:17:43.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.123+0000 7f1edc5f3640 1 Processor -- start 2024-05-09T03:17:43.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.123+0000 7f1edc5f3640 1 -- start start 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1edc5f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 0x7f1ed41f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1edc5f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ed40d2e60 0x7f1ed415e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1edc5f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed415cf80 0x7f1ed415abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1edc5f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed415d390 con 0x7f1ed40b7c70 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1edc5f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed40b27b0 con 0x7f1ed40d2e60 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1edc5f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed415d510 con 0x7f1ed415cf80 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.124+0000 7f1eda38f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 0x7f1ed41f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 0x7f1ed41f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38802/0 (socket says 172.21.15.46:38802) 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 -- 172.21.15.46:0/3880636626 learned_addr learned my addr 172.21.15.46:0/3880636626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1edab90640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed415cf80 0x7f1ed415abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1ed9b8e640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ed40d2e60 0x7f1ed415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 -- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed415cf80 msgr2=0x7f1ed415abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:43.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed415cf80 0x7f1ed415abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 -- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ed40d2e60 msgr2=0x7f1ed415e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ed40d2e60 0x7f1ed415e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 -- 172.21.15.46:0/3880636626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed4159890 con 0x7f1ed40b7c70 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1edab90640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed415cf80 0x7f1ed415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1ed9b8e640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ed40d2e60 0x7f1ed415e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1eda38f640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 0x7f1ed41f1a20 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4000c00 tx=0x7f1ec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4033620 con 0x7f1ed40b7c70 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec40337c0 con 0x7f1ed40b7c70 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed4159b30 con 0x7f1ed40b7c70 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4033b90 con 0x7f1ed40b7c70 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.125+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed4159e10 con 0x7f1ed40b7c70 2024-05-09T03:17:43.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.126+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed406a270 con 0x7f1ed40b7c70 2024-05-09T03:17:43.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.127+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ec404d060 con 0x7f1ed40b7c70 2024-05-09T03:17:43.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.128+0000 7f1ed2ffd640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ec00762d0 0x7f1ec00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:43.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.128+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6885+0+0 (secure 0 0 0) 0x7f1ec40ba6a0 con 0x7f1ed40b7c70 2024-05-09T03:17:43.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.128+0000 7f1ed9b8e640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ec00762d0 0x7f1ec00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:43.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.129+0000 7f1ed9b8e640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ec00762d0 0x7f1ec00787e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8006fd0 tx=0x7f1ec8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:43.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.133+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1ed4159890 con 0x7f1ed40b7c70 2024-05-09T03:17:43.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.400+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f1ed40d00c0 con 0x7f1ed40b7c70 2024-05-09T03:17:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.982+0000 7f1ed2ffd640 1 -- 172.21.15.46:0/3880636626 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v168) ==== 107+0+0 (secure 0 0 0) 0x7f1ec4041070 con 0x7f1ed40b7c70 2024-05-09T03:17:43.983 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ec00762d0 msgr2=0x7f1ec00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ec00762d0 0x7f1ec00787e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8006fd0 tx=0x7f1ec8008040 comp rx=0 tx=0).stop 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 msgr2=0x7f1ed41f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 0x7f1ed41f1a20 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4000c00 tx=0x7f1ec40046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 shutdown_connections 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ed415cf80 0x7f1ed415abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ed40d2e60 0x7f1ed415e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ec00762d0 0x7f1ec00787e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 --2- 172.21.15.46:0/3880636626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ed40b7c70 0x7f1ed41f1a20 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:43.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 >> 172.21.15.46:0/3880636626 conn(0x7f1ed409b7d0 msgr2=0x7f1ed41609b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:43.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 shutdown_connections 2024-05-09T03:17:43.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:43.985+0000 7f1edc5f3640 1 -- 172.21.15.46:0/3880636626 wait complete. 2024-05-09T03:17:44.000 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:44.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.061+0000 7f2c76814f40 1 Processor -- start 2024-05-09T03:17:44.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.061+0000 7f2c76814f40 1 -- start start 2024-05-09T03:17:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.062+0000 7f2c76814f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634fea983a0 0x5634feb237a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.062+0000 7f2c76814f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634fea596b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.062+0000 7f2c76814f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634feb2c150 0x5634fea59bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.062+0000 7f2c76814f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634feaa15f0 con 0x5634feaa05f0 2024-05-09T03:17:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.062+0000 7f2c76814f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5634fe9fef50 con 0x5634feb2c150 2024-05-09T03:17:44.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.062+0000 7f2c76814f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5634feaa18d0 con 0x5634fea983a0 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634fea596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634fea596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38820/0 (socket says 172.21.15.46:38820) 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/798454966 learned_addr learned my addr 172.21.15.46:0/798454966 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634feb2c150 0x5634fea59bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634fea983a0 0x5634feb237a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634fea983a0 msgr2=0x5634feb237a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634fea983a0 0x5634feb237a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634feb2c150 msgr2=0x5634fea59bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634feb2c150 0x5634fea59bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/798454966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634fea85860 con 0x5634feaa05f0 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634fea983a0 0x5634feb237a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634fea596b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f2c64025aa0 tx=0x7f2c6404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940fffc3fc839e13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6de6a640 1 -- 172.21.15.46:0/798454966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64065070 con 0x5634feaa05f0 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6de6a640 1 -- 172.21.15.46:0/798454966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c640549f0 con 0x5634feaa05f0 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.063+0000 7f2c6de6a640 1 -- 172.21.15.46:0/798454966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64059040 con 0x5634feaa05f0 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 -- 172.21.15.46:0/798454966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 msgr2=0x5634fea596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634fea596b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f2c64025aa0 tx=0x7f2c6404e000 comp rx=0 tx=0).stop 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 -- 172.21.15.46:0/798454966 shutdown_connections 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634feb2c150 0x5634fea59bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634fea596b0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 --2- 172.21.15.46:0/798454966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634fea983a0 0x5634feb237a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 -- 172.21.15.46:0/798454966 >> 172.21.15.46:0/798454966 conn(0x5634fe97aac0 msgr2=0x5634fea83150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 -- 172.21.15.46:0/798454966 shutdown_connections 2024-05-09T03:17:44.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.064+0000 7f2c76814f40 1 -- 172.21.15.46:0/798454966 wait complete. 2024-05-09T03:17:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.065+0000 7f2c76814f40 1 Processor -- start 2024-05-09T03:17:44.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.065+0000 7f2c76814f40 1 -- start start 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c76814f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634fea983a0 0x5634feb1dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c76814f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634feb2cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c76814f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634feb2c150 0x5634fea596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c76814f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5634feb2af30 con 0x5634feaa05f0 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c76814f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5634fea86040 con 0x5634fea983a0 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c76814f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5634febc5520 con 0x5634feb2c150 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c6ee6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634feb2cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c6ee6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634feb2cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38832/0 (socket says 172.21.15.46:38832) 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.066+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 learned_addr learned my addr 172.21.15.46:0/1090649870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634fea983a0 0x5634feb1dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634feb2c150 0x5634fea596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634feb2c150 msgr2=0x5634fea596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634feb2c150 0x5634fea596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634fea983a0 msgr2=0x5634feb1dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634fea983a0 0x5634feb1dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634feb26ae0 con 0x5634feaa05f0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5634fea983a0 0x5634feb1dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5634feb2c150 0x5634fea596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634feb2cbe0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2c64025aa0 tx=0x7f2c64002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ce68640 1 -- 172.21.15.46:0/1090649870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64065040 con 0x5634feaa05f0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ce68640 1 -- 172.21.15.46:0/1090649870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c64054da0 con 0x5634feaa05f0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5634feb26dc0 con 0x5634feaa05f0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c6ce68640 1 -- 172.21.15.46:0/1090649870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64060620 con 0x5634feaa05f0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.067+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634feb28e80 con 0x5634feaa05f0 2024-05-09T03:17:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.069+0000 7f2c6ce68640 1 -- 172.21.15.46:0/1090649870 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2c640607c0 con 0x5634feaa05f0 2024-05-09T03:17:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.070+0000 7f2c6ce68640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c5c074ce0 0x7f2c5c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.070+0000 7f2c6ce68640 1 -- 172.21.15.46:0/1090649870 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6885+0+0 (secure 0 0 0) 0x7f2c640db660 con 0x5634feaa05f0 2024-05-09T03:17:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.070+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c5c074ce0 0x7f2c5c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.073 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:44.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.070+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5634feb27c20 0x5634feb28030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.070+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634feb28570 con 0x5634feb27c20 2024-05-09T03:17:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.070+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5634feb27c20 0x5634feb28030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.071+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c5c074ce0 0x7f2c5c0771f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2c60000c00 tx=0x7f2c60001680 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.071+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5634feb27c20 0x5634feb28030 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.072+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v97'381 uv355 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.072+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634feb28940 con 0x5634feb27c20 2024-05-09T03:17:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.074+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v97'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.074+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd1720 con 0x5634feb27c20 2024-05-09T03:17:44.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.076+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v97'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.076 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.076+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2ad0 con 0x5634feb27c20 2024-05-09T03:17:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.077+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v97'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.077+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2e70 con 0x5634feb27c20 2024-05-09T03:17:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.078+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v97'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.078+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd1ac0 con 0x5634feb27c20 2024-05-09T03:17:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.079+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v97'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.079+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd1e60 con 0x5634feb27c20 2024-05-09T03:17:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.081+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v97'387 uv361 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.081+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.082+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v97'388 uv388 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.082+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.083+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v97'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.084 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.083+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.085+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v97'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.085+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.086+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v97'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.086+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.087+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v97'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.087+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.088+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v97'393 uv368 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.089+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.090+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v97'394 uv394 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.090+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.091+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v97'395 uv395 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.092 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:44.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.091+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.092+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v97'396 uv396 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.092+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.094+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v97'397 uv397 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.094+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2200 con 0x5634feb27c20 2024-05-09T03:17:44.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.095+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v97'398 uv398 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.095+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634feb22ae0 con 0x5634feb27c20 2024-05-09T03:17:44.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.097+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v97'399 uv399 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.097+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634feb22ae0 con 0x5634feb27c20 2024-05-09T03:17:44.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.098+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v97'400 uv374 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.098+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634feb22ae0 con 0x5634feb27c20 2024-05-09T03:17:44.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.100+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v97'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.100+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.101+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v97'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.102 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:44.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.101+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.103+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v97'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.103+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.104+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v97'404 uv404 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.104+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.105+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v97'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.105+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.107+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v97'406 uv380 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.107+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.108+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v97'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.108+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.110+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v97'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.110 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:44.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.110+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.111+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v97'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.111+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.112+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v97'410 uv410 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.112+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634feb27c20 2024-05-09T03:17:44.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.114+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v97'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.114+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5634febd3720 0x5634febd5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.114+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd2470 con 0x5634febd3720 2024-05-09T03:17:44.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.114+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5634febd3720 0x5634febd5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.114+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5634febd3720 0x5634febd5b40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.116+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v97'284 uv264 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.116+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd6080 con 0x5634febd3720 2024-05-09T03:17:44.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.117+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v97'285 uv285 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.117+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd6420 con 0x5634febd3720 2024-05-09T03:17:44.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.119+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v97'286 uv286 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.120 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:44.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.119+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd6c70 con 0x5634febd3720 2024-05-09T03:17:44.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.120+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v97'287 uv287 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.120+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd7010 con 0x5634febd3720 2024-05-09T03:17:44.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.122+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v97'288 uv288 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.122+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd73b0 con 0x5634febd3720 2024-05-09T03:17:44.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.123+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v97'289 uv283 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.123+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd7750 con 0x5634febd3720 2024-05-09T03:17:44.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.125+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v97'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.125+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd7af0 con 0x5634febd3720 2024-05-09T03:17:44.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.126+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v97'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.127 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:44.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.126+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.127+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v97'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.127+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.129+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v97'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.129+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.130+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v97'294 uv282 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.130+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.131+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v97'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.131+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.132+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v97'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.133 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:44.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.132+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.134+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v97'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.134+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.135+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v97'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.135+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.136+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v97'299 uv280 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.136+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.137+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v97'300 uv300 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.137+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.138+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v97'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.139 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:44.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.139+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.140+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v97'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.140+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.141+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v97'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.141+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634febd3720 2024-05-09T03:17:44.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.142+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v97'304 uv304 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febd3720 2024-05-09T03:17:44.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.142+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634feb27c20 2024-05-09T03:17:44.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.143+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v97'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.143+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634feb27c20 2024-05-09T03:17:44.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.145+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v97'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.145+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634feb27c20 2024-05-09T03:17:44.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.146+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v97'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.147 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:44.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.146+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634feb27c20 2024-05-09T03:17:44.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.147+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v97'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.147+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634feb27c20 2024-05-09T03:17:44.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.148+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v97'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.148+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd68a0 con 0x5634feb27c20 2024-05-09T03:17:44.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.149+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v97'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.149+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5634febd84b0 0x5634febd88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.149+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdb030 con 0x5634febd84b0 2024-05-09T03:17:44.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.150+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5634febd84b0 0x5634febd88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.150+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5634febd84b0 0x5634febd88c0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.153+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v97'62 uv61 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.153+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febdb3d0 con 0x5634febd84b0 2024-05-09T03:17:44.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.154+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v97'63 uv63 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.154+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdb860 con 0x5634febd84b0 2024-05-09T03:17:44.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.155+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v97'64 uv64 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.155 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:44.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.155+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febd7b90 con 0x5634febd84b0 2024-05-09T03:17:44.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.156+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v97'65 uv65 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.156+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdc170 con 0x5634feb27c20 2024-05-09T03:17:44.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.157+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v97'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.157+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febdc170 con 0x5634feb27c20 2024-05-09T03:17:44.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.158+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v97'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.158+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdc170 con 0x5634feb27c20 2024-05-09T03:17:44.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.160+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v97'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.160 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:44.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.160+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdd180 con 0x5634feb27c20 2024-05-09T03:17:44.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.161+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v97'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.161+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdd180 con 0x5634feb27c20 2024-05-09T03:17:44.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.162+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v97'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.162+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdc170 con 0x5634feb27c20 2024-05-09T03:17:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.163+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v97'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.163+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5634febdc7f0 0x5634febdcc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.163+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf5d0 con 0x5634febdc7f0 2024-05-09T03:17:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.163+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5634febdc7f0 0x5634febdcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.164+0000 7f2c6fe6e640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5634febdc7f0 0x5634febdcc00 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.165+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v97'138 uv132 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.165+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febdf5d0 con 0x5634febdc7f0 2024-05-09T03:17:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.166+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v97'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.166+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf5d0 con 0x5634febdc7f0 2024-05-09T03:17:44.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.167+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v97'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.168 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.167+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe05e0 con 0x5634febdc7f0 2024-05-09T03:17:44.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.168+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v97'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.168+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe05e0 con 0x5634febdc7f0 2024-05-09T03:17:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.169+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v97'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.169+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf5d0 con 0x5634febdc7f0 2024-05-09T03:17:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.170+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v97'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.170+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf9c0 con 0x5634febdc7f0 2024-05-09T03:17:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.171+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v97'144 uv137 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.171+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febdf9c0 con 0x5634febdc7f0 2024-05-09T03:17:44.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.172+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v97'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.172+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf690 con 0x5634febdc7f0 2024-05-09T03:17:44.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.173+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v97'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.173 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.173+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf690 con 0x5634febdc7f0 2024-05-09T03:17:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.174+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v97'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.174+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf650 con 0x5634febdc7f0 2024-05-09T03:17:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.175+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v97'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.175+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5634febe02a0 0x5634febe29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.175+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febdf650 con 0x5634febe02a0 2024-05-09T03:17:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.175+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5634febe02a0 0x5634febe29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.176+0000 7f2c6f66d640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5634febe02a0 0x5634febe29d0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.177+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v97'61 uv60 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febe02a0 2024-05-09T03:17:44.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.177+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febdf650 con 0x5634febe02a0 2024-05-09T03:17:44.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.178+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v97'62 uv62 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febe02a0 2024-05-09T03:17:44.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.178+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634febe02a0 2024-05-09T03:17:44.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.179+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v97'63 uv63 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febe02a0 2024-05-09T03:17:44.179 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.179+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634febe02a0 2024-05-09T03:17:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.180+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v97'64 uv64 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febe02a0 2024-05-09T03:17:44.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.180+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634febe02a0 2024-05-09T03:17:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.181+0000 7f2c6f66d640 1 -- 172.21.15.46:0/1090649870 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v97'65 uv65 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c6001a040 con 0x5634febe02a0 2024-05-09T03:17:44.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.181+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634feb27c20 2024-05-09T03:17:44.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.182+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v97'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.182+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634feb27c20 2024-05-09T03:17:44.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.183+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v97'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.183+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634feb27c20 2024-05-09T03:17:44.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.184+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v97'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.185 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:44.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.184+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe3f20 con 0x5634feb27c20 2024-05-09T03:17:44.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.185+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v97'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.185+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe3f20 con 0x5634feb27c20 2024-05-09T03:17:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.187+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v97'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.187+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634feb27c20 2024-05-09T03:17:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.188+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v97'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634feb27c20 2024-05-09T03:17:44.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.188+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe2f10 con 0x5634febd84b0 2024-05-09T03:17:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.189+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v97'122 uv116 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.189+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.190+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v97'123 uv123 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.190+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.191+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v97'124 uv124 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.192 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.191+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.192+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v97'125 uv125 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.192+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.193+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v97'126 uv126 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.193+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.194+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v97'127 uv121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.195+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.196+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v97'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.196+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.197+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v97'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.197 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.197+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe32b0 con 0x5634febd84b0 2024-05-09T03:17:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.198+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v97'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.198+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6080 con 0x5634febd84b0 2024-05-09T03:17:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.199+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v97'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.199+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5634febe36a0 0x5634febe5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.199+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6740 con 0x5634febe36a0 2024-05-09T03:17:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.199+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5634febe36a0 0x5634febe5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.199+0000 7f2c6ee6c640 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5634febe36a0 0x5634febe5ac0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.201+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v97'57 uv55 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febe36a0 2024-05-09T03:17:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.201+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe6740 con 0x5634febe36a0 2024-05-09T03:17:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.202+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v97'58 uv58 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febe36a0 2024-05-09T03:17:44.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.202+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6740 con 0x5634febe36a0 2024-05-09T03:17:44.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.203+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v97'59 uv59 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febe36a0 2024-05-09T03:17:44.204 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:44.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.203+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6b30 con 0x5634febe36a0 2024-05-09T03:17:44.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.204+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v97'60 uv60 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febe36a0 2024-05-09T03:17:44.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.204+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6b30 con 0x5634febd84b0 2024-05-09T03:17:44.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.205+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v97'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.205+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe6b30 con 0x5634febd84b0 2024-05-09T03:17:44.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.206+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v97'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.206+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6740 con 0x5634febd84b0 2024-05-09T03:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.207+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v97'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.208 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.207+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6740 con 0x5634febd84b0 2024-05-09T03:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.209+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v97'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.209+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febd84b0 2024-05-09T03:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.210+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v97'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.210+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febd84b0 2024-05-09T03:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.211+0000 7f2c6ee6c640 1 -- 172.21.15.46:0/1090649870 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v97'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c64078070 con 0x5634febd84b0 2024-05-09T03:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.211+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.212+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v97'64 uv63 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.212+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.213+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v97'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.213+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.214 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.214+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v97'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.214+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.215+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v97'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.215+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.217+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v97'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.217+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.218+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v97'157 uv148 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.218+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.219+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v97'158 uv158 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.219+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.220+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v97'159 uv159 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.221 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:44.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.220+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.222+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v97'160 uv160 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.222+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.223+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v97'161 uv161 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.223+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.224+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v97'162 uv155 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.224+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.225+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v97'163 uv163 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.225+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.226+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v97'164 uv164 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.227 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:44.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.226+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.227+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v97'165 uv165 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.227+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.229+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v97'166 uv166 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.229+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe66f0 con 0x5634febdc7f0 2024-05-09T03:17:44.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.230+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v97'167 uv167 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.230+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x5634febe6c60 con 0x5634febdc7f0 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v97'168 uv168 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2c6800ea20 con 0x5634febdc7f0 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5634feb27c20 msgr2=0x5634feb28030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5634feb27c20 0x5634feb28030 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5634febd84b0 msgr2=0x5634febd88c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5634febd84b0 0x5634febd88c0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5634febe36a0 msgr2=0x5634febe5ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5634febe36a0 0x5634febe5ac0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 reap_dead start 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5634febd3720 msgr2=0x5634febd5b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5634febd3720 0x5634febd5b40 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5634febdc7f0 msgr2=0x5634febdcc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5634febdc7f0 0x5634febdcc00 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5634febe02a0 msgr2=0x5634febe29d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5634febe02a0 0x5634febe29d0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c5c074ce0 msgr2=0x7f2c5c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c5c074ce0 0x7f2c5c0771f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2c60000c00 tx=0x7f2c60001680 comp rx=0 tx=0).stop 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 msgr2=0x5634feb2cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 --2- 172.21.15.46:0/1090649870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5634feaa05f0 0x5634feb2cbe0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f2c64025aa0 tx=0x7f2c64002fb0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c6fe6e640 1 -- 172.21.15.46:0/1090649870 reap_dead start 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 shutdown_connections 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 >> 172.21.15.46:0/1090649870 conn(0x5634fe97aac0 msgr2=0x5634feb237a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 shutdown_connections 2024-05-09T03:17:44.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.231+0000 7f2c76814f40 1 -- 172.21.15.46:0/1090649870 wait complete. 2024-05-09T03:17:44.235 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100.00000000.inode 2024-05-09T03:17:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.330+0000 7f61ce70df40 1 Processor -- start 2024-05-09T03:17:44.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.330+0000 7f61ce70df40 1 -- start start 2024-05-09T03:17:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61ce70df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25767e0 0x5594f2631ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61ce70df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25a5db0 0x5594f25a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61ce70df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f2632820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61ce70df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5594f25b8cb0 con 0x5594f264e780 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61ce70df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5594f251ff50 con 0x5594f25767e0 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61ce70df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5594f2576bf0 con 0x5594f25a5db0 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f2632820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f2632820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38846/0 (socket says 172.21.15.46:38846) 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 -- 172.21.15.46:0/2615958312 learned_addr learned my addr 172.21.15.46:0/2615958312 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c79d4640 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25a5db0 0x5594f25a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c81d5640 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25767e0 0x5594f2631ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 -- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25a5db0 msgr2=0x5594f25a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25a5db0 0x5594f25a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 -- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25767e0 msgr2=0x5594f2631ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25767e0 0x5594f2631ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 -- 172.21.15.46:0/2615958312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594f2631870 con 0x5594f264e780 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c89d6640 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f2632820 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f61c0001dd0 tx=0x7f61c000b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659281f9d4ddef0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.332+0000 7f61c69d2640 1 -- 172.21.15.46:0/2615958312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c0008070 con 0x5594f264e780 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61c69d2640 1 -- 172.21.15.46:0/2615958312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c000be10 con 0x5594f264e780 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61c69d2640 1 -- 172.21.15.46:0/2615958312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c000d720 con 0x5594f264e780 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 -- 172.21.15.46:0/2615958312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 msgr2=0x5594f2632820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f2632820 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f61c0001dd0 tx=0x7f61c000b5e0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 -- 172.21.15.46:0/2615958312 shutdown_connections 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f2632820 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25a5db0 0x5594f25a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 --2- 172.21.15.46:0/2615958312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25767e0 0x5594f2631ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 -- 172.21.15.46:0/2615958312 >> 172.21.15.46:0/2615958312 conn(0x5594f249bac0 msgr2=0x5594f25a7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 -- 172.21.15.46:0/2615958312 shutdown_connections 2024-05-09T03:17:44.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.333+0000 7f61ce70df40 1 -- 172.21.15.46:0/2615958312 wait complete. 2024-05-09T03:17:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.334+0000 7f61ce70df40 1 Processor -- start 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.334+0000 7f61ce70df40 1 -- start start 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61ce70df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25767e0 0x5594f26302c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61ce70df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25a5db0 0x5594f264b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61ce70df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f264db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61ce70df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5594f2631ae0 con 0x5594f264e780 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61ce70df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5594f26e3bc0 con 0x5594f25a5db0 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61ce70df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5594f25ba410 con 0x5594f25767e0 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61c89d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f264db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61c89d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f264db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38862/0 (socket says 172.21.15.46:38862) 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.335+0000 7f61c89d6640 1 -- 172.21.15.46:0/1056181095 learned_addr learned my addr 172.21.15.46:0/1056181095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c79d4640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25a5db0 0x5594f264b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c81d5640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25767e0 0x5594f26302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c89d6640 1 -- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25767e0 msgr2=0x5594f26302c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c89d6640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25767e0 0x5594f26302c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c89d6640 1 -- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25a5db0 msgr2=0x5594f264b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c89d6640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594f25a5db0 0x5594f264b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c89d6640 1 -- 172.21.15.46:0/1056181095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594f25ba660 con 0x5594f264e780 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c81d5640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5594f25767e0 0x5594f26302c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c89d6640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f264db70 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000fb0 tx=0x7f61c0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c59d0640 1 -- 172.21.15.46:0/1056181095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c001b640 con 0x5594f264e780 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c59d0640 1 -- 172.21.15.46:0/1056181095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c0020030 con 0x5594f264e780 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5594f25ba940 con 0x5594f264e780 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61c59d0640 1 -- 172.21.15.46:0/1056181095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c001ba40 con 0x5594f264e780 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.336+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594f25baf00 con 0x5594f264e780 2024-05-09T03:17:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.338+0000 7f61c59d0640 1 -- 172.21.15.46:0/1056181095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f61c0017040 con 0x5594f264e780 2024-05-09T03:17:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61c59d0640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f61a8074cb0 0x7f61a80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61c59d0640 1 -- 172.21.15.46:0/1056181095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6885+0+0 (secure 0 0 0) 0x7f61c0098350 con 0x5594f264e780 2024-05-09T03:17:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61c81d5640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f61a8074cb0 0x7f61a80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61ce70df40 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5594f264bed0 0x5594f264c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e97) -- 0x5594f264c820 con 0x5594f264bed0 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61c79d4640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5594f264bed0 0x5594f264c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61c81d5640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f61a8074cb0 0x7f61a80771c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f61b8057000 tx=0x7f61b8007030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.339+0000 7f61c79d4640 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5594f264bed0 0x5594f264c2e0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61c79d4640 1 -- 172.21.15.46:0/1056181095 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 100.00000000.inode [delete] v97'169 uv169 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f61b0009ac0 con 0x5594f264bed0 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5594f264bed0 msgr2=0x5594f264c2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5594f264bed0 0x5594f264c2e0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f61a8074cb0 msgr2=0x7f61a80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f61a8074cb0 0x7f61a80771c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f61b8057000 tx=0x7f61b8007030 comp rx=0 tx=0).stop 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 msgr2=0x5594f264db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 --2- 172.21.15.46:0/1056181095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5594f264e780 0x5594f264db70 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f61c0000fb0 tx=0x7f61c0000ed0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61c89d6640 1 -- 172.21.15.46:0/1056181095 reap_dead start 2024-05-09T03:17:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 shutdown_connections 2024-05-09T03:17:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 >> 172.21.15.46:0/1056181095 conn(0x5594f249bac0 msgr2=0x5594f2631c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 shutdown_connections 2024-05-09T03:17:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.341+0000 7f61ce70df40 1 -- 172.21.15.46:0/1056181095 wait complete. 2024-05-09T03:17:44.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.462+0000 7f840c95e640 1 Processor -- start 2024-05-09T03:17:44.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.462+0000 7f840c95e640 1 -- start start 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840c95e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840c95e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8408163eb0 0x7f84081642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840c95e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84080d2a70 0x7f84080d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840c95e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f84080d4bb0 con 0x7f84080d2a70 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840c95e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f84080d33c0 con 0x7f8408163eb0 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840c95e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8408164900 con 0x7f84080bb770 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f8406d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84080d2a70 0x7f84080d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f8406d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84080d2a70 0x7f84080d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38878/0 (socket says 172.21.15.46:38878) 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f8406d9d640 1 -- 172.21.15.46:0/440937608 learned_addr learned my addr 172.21.15.46:0/440937608 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f8405d9b640 1 --2- 172.21.15.46:0/440937608 >> v2:172.21.15.88:3300/0 conn(0x7f8408163eb0 0x7f84081642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.463+0000 7f840659c640 1 --2- 172.21.15.46:0/440937608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8406d9d640 1 -- 172.21.15.46:0/440937608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 msgr2=0x7f84080d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:44.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8406d9d640 1 --2- 172.21.15.46:0/440937608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8406d9d640 1 -- 172.21.15.46:0/440937608 >> v2:172.21.15.88:3300/0 conn(0x7f8408163eb0 msgr2=0x7f84081642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8406d9d640 1 --2- 172.21.15.46:0/440937608 >> v2:172.21.15.88:3300/0 conn(0x7f8408163eb0 0x7f84081642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8406d9d640 1 -- 172.21.15.46:0/440937608 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8408164be0 con 0x7f84080d2a70 2024-05-09T03:17:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8405d9b640 1 --2- 172.21.15.46:0/440937608 >> v2:172.21.15.88:3300/0 conn(0x7f8408163eb0 0x7f84081642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:44.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.464+0000 7f8406d9d640 1 --2- 172.21.15.46:0/440937608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84080d2e80 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f84080b88b0 tx=0x7f83f802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db26f0a6467df3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840559a640 1 -- 172.21.15.46:0/440937608 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8037070 con 0x7f84080d2a70 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840559a640 1 -- 172.21.15.46:0/440937608 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f8032440 con 0x7f84080d2a70 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840559a640 1 -- 172.21.15.46:0/440937608 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8032810 con 0x7f84080d2a70 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 -- 172.21.15.46:0/440937608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 msgr2=0x7f84080d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 --2- 172.21.15.46:0/440937608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84080d2e80 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f84080b88b0 tx=0x7f83f802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 -- 172.21.15.46:0/440937608 shutdown_connections 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 --2- 172.21.15.46:0/440937608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84080d2e80 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 --2- 172.21.15.46:0/440937608 >> v2:172.21.15.88:3300/0 conn(0x7f8408163eb0 0x7f84081642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 --2- 172.21.15.46:0/440937608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 -- 172.21.15.46:0/440937608 >> 172.21.15.46:0/440937608 conn(0x7f840806e370 msgr2=0x7f840806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 -- 172.21.15.46:0/440937608 shutdown_connections 2024-05-09T03:17:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.465+0000 7f840c95e640 1 -- 172.21.15.46:0/440937608 wait complete. 2024-05-09T03:17:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.467+0000 7f840c95e640 1 Processor -- start 2024-05-09T03:17:44.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.467+0000 7f840c95e640 1 -- start start 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840c95e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840c95e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84081f9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840c95e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8408163eb0 0x7f84080cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840c95e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84080bbb80 con 0x7f84080d2a70 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840c95e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f84080d5580 con 0x7f8408163eb0 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840c95e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f84080d0bc0 con 0x7f84080bb770 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84081f9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84081f9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38886/0 (socket says 172.21.15.46:38886) 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 -- 172.21.15.46:0/3398866721 learned_addr learned my addr 172.21.15.46:0/3398866721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f840659c640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8406d9d640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8408163eb0 0x7f84080cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 -- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 msgr2=0x7f84080cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 -- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8408163eb0 msgr2=0x7f84080cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8408163eb0 0x7f84080cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 -- 172.21.15.46:0/3398866721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84080cf8a0 con 0x7f84080d2a70 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f8405d9b640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84081f9550 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007e70 tx=0x7f83fc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.469+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc01a070 con 0x7f84080d2a70 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.470+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83fc003b90 con 0x7f84080d2a70 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.470+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc004030 con 0x7f84080d2a70 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.470+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84081ec280 con 0x7f84080d2a70 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.470+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84081ec840 con 0x7f84080d2a70 2024-05-09T03:17:44.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.470+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84081ebff0 con 0x7f84080d2a70 2024-05-09T03:17:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.471+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f83fc005d10 con 0x7f84080d2a70 2024-05-09T03:17:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.472+0000 7f83f6ffd640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83e80762d0 0x7f83e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.472+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6885+0+0 (secure 0 0 0) 0x7f83fc098300 con 0x7f84080d2a70 2024-05-09T03:17:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.472+0000 7f840659c640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83e80762d0 0x7f83e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:44.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.473+0000 7f840659c640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83e80762d0 0x7f83e80787e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f84080b8880 tx=0x7f8400008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:44.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.476+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f83fc010070 con 0x7f84080d2a70 2024-05-09T03:17:44.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.706+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f84080ca620 con 0x7f84080d2a70 2024-05-09T03:17:44.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.983+0000 7f83f6ffd640 1 -- 172.21.15.46:0/3398866721 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v169) ==== 162+0+0 (secure 0 0 0) 0x7f83fc0611c0 con 0x7f84080d2a70 2024-05-09T03:17:44.984 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83e80762d0 msgr2=0x7f83e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83e80762d0 0x7f83e80787e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f84080b8880 tx=0x7f8400008040 comp rx=0 tx=0).stop 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 msgr2=0x7f84081f9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84081f9550 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f83fc007e70 tx=0x7f83fc00d4a0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 shutdown_connections 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8408163eb0 0x7f84080cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f83e80762d0 0x7f83e80787e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84080d2a70 0x7f84081f9550 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 --2- 172.21.15.46:0/3398866721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f84080bb770 0x7f84080cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 >> 172.21.15.46:0/3398866721 conn(0x7f840806e370 msgr2=0x7f84081623d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 shutdown_connections 2024-05-09T03:17:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:44.986+0000 7f840c95e640 1 -- 172.21.15.46:0/3398866721 wait complete. 2024-05-09T03:17:44.993 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:44.992+0000 7fb839f38640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-05-09T03:17:44.993 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:44.992+0000 7fb839f38640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:17:44.993 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:44.993+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:44.993 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:44.993+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:44.999 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:17:44.992+0000 7fb839f38640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-05-09T03:17:45.000 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:17:44.992+0000 7fb839f38640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:17:45.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:45.056 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.122+0000 7efd119d5640 1 Processor -- start 2024-05-09T03:17:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.122+0000 7efd119d5640 1 -- start start 2024-05-09T03:17:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd119d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd119d5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd0c161ce0 0x7efd0c1620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd119d5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0c162630 con 0x7efd0c15b7e0 2024-05-09T03:17:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd119d5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efd0c15eab0 con 0x7efd0c161ce0 2024-05-09T03:17:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd119d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efd0c070a60 con 0x7efd0c15e6a0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efd0c15b7e0 0x7efd0c15bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54062/0 (socket says 172.21.15.46:54062) 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0b7fe640 1 -- 172.21.15.46:0/402055537 learned_addr learned my addr 172.21.15.46:0/402055537 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0a7fc640 1 --2- 172.21.15.46:0/402055537 >> v2:172.21.15.88:3300/0 conn(0x7efd0c161ce0 0x7efd0c1620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0affd640 1 --2- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd09ffb640 1 -- 172.21.15.46:0/402055537 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 642849402 0 0) 0x7efd04001490 con 0x7efd0c15b7e0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd09ffb640 1 -- 172.21.15.46:0/402055537 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcf4003520 con 0x7efd0c15b7e0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0a7fc640 1 -- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 msgr2=0x7efd0c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0a7fc640 1 --2- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c070230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.124+0000 7efd0a7fc640 1 -- 172.21.15.46:0/402055537 >> v1:172.21.15.46:6789/0 conn(0x7efd0c15b7e0 legacy=0x7efd0c15bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.125+0000 7efd0a7fc640 1 -- 172.21.15.46:0/402055537 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c070d40 con 0x7efd0c161ce0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.125+0000 7efd0a7fc640 1 --2- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1620f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efd000097a0 tx=0x7efd0002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60be09e53600b6ac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.125+0000 7efd09ffb640 1 -- 172.21.15.46:0/402055537 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00037070 con 0x7efd0c161ce0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd09ffb640 1 -- 172.21.15.46:0/402055537 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd0002ae50 con 0x7efd0c161ce0 2024-05-09T03:17:45.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd09ffb640 1 -- 172.21.15.46:0/402055537 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00032660 con 0x7efd0c161ce0 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 -- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 msgr2=0x7efd0c1620f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 --2- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1620f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efd000097a0 tx=0x7efd0002a100 comp rx=0 tx=0).stop 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 -- 172.21.15.46:0/402055537 shutdown_connections 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 --2- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1620f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 --2- 172.21.15.46:0/402055537 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c070230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 -- 172.21.15.46:0/402055537 >> 172.21.15.46:0/402055537 conn(0x7efd0c0d0630 msgr2=0x7efd0c0d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 -- 172.21.15.46:0/402055537 shutdown_connections 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.126+0000 7efd119d5640 1 -- 172.21.15.46:0/402055537 wait complete. 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.128+0000 7efd119d5640 1 Processor -- start 2024-05-09T03:17:45.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.128+0000 7efd119d5640 1 -- start start 2024-05-09T03:17:45.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.129+0000 7efd119d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 0x7efd0c0b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.129+0000 7efd119d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c0b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.129+0000 7efd119d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd119d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c070770 con 0x7efd0c15b7e0 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd119d5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c0d1900 con 0x7efd0c161ce0 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd119d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efd0c0b3160 con 0x7efd0c15e6a0 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 0x7efd0c0b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 0x7efd0c0b7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38912/0 (socket says 172.21.15.46:38912) 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 -- 172.21.15.46:0/2284861322 learned_addr learned my addr 172.21.15.46:0/2284861322 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0b7fe640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0a7fc640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c0b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 -- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 msgr2=0x7efd0c0b8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c0b8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 -- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 msgr2=0x7efd0c1546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 -- 172.21.15.46:0/2284861322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c0b71e0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd0affd640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 0x7efd0c0b7c80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e870 tx=0x7efcfc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc017070 con 0x7efd0c15b7e0 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcfc00c8e0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0c0b90e0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00ccd0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.130+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0c154be0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.131+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0c06a270 con 0x7efd0c15b7e0 2024-05-09T03:17:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.132+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efcfc01a070 con 0x7efd0c15b7e0 2024-05-09T03:17:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.133+0000 7efcf3fff640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcf80762d0 0x7efcf80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.133+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7efcfc0979f0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.133+0000 7efd0a7fc640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcf80762d0 0x7efcf80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.134+0000 7efd0a7fc640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcf80762d0 0x7efcf80787e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7efd00034000 tx=0x7efd0002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.138+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efcfc060d50 con 0x7efd0c15b7e0 2024-05-09T03:17:45.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.368+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd0c0b97b0 con 0x7efd0c15b7e0 2024-05-09T03:17:45.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.368+0000 7efcf3fff640 1 -- 172.21.15.46:0/2284861322 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+4342 (secure 0 0 0) 0x7efcfc060650 con 0x7efd0c15b7e0 2024-05-09T03:17:45.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:45.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":170,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:44.993974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:45.370 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 170 2024-05-09T03:17:45.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcf80762d0 msgr2=0x7efcf80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcf80762d0 0x7efcf80787e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7efd00034000 tx=0x7efd0002e000 comp rx=0 tx=0).stop 2024-05-09T03:17:45.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 msgr2=0x7efd0c0b7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 0x7efd0c0b7c80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00e870 tx=0x7efcfc00ed30 comp rx=0 tx=0).stop 2024-05-09T03:17:45.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 shutdown_connections 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd0c161ce0 0x7efd0c1546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efd0c15e6a0 0x7efd0c0b8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efcf80762d0 0x7efcf80787e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 --2- 172.21.15.46:0/2284861322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c15b7e0 0x7efd0c0b7c80 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 >> 172.21.15.46:0/2284861322 conn(0x7efd0c0d0630 msgr2=0x7efd0c0b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 shutdown_connections 2024-05-09T03:17:45.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.370+0000 7efd119d5640 1 -- 172.21.15.46:0/2284861322 wait complete. 2024-05-09T03:17:45.382 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:45.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:45.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.495+0000 7fb887c8b640 1 Processor -- start 2024-05-09T03:17:45.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.496+0000 7fb887c8b640 1 -- start start 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb887c8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880070670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb887c8b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb880070fc0 con 0x7fb880070bb0 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb887c8b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb88015d980 con 0x7fb8800b1210 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb887c8b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb88015dc60 con 0x7fb8800b4ff0 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb886228640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb880070bb0 0x7fb880073870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54078/0 (socket says 172.21.15.46:54078) 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb886228640 1 -- 172.21.15.46:0/3144565632 learned_addr learned my addr 172.21.15.46:0/3144565632 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.497+0000 7fb885226640 1 --2- 172.21.15.46:0/3144565632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267451227 0 0) 0x7fb87c001490 con 0x7fb880070bb0 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb86c005730 con 0x7fb880070bb0 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249949814 0 0) 0x7fb88015d980 con 0x7fb8800b1210 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb87c001490 con 0x7fb8800b1210 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 998386464 0 0) 0x7fb86c005730 con 0x7fb880070bb0 2024-05-09T03:17:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb885226640 1 -- 172.21.15.46:0/3144565632 >> v1:172.21.15.88:6789/0 conn(0x7fb8800b1210 legacy=0x7fb8800aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb885226640 1 -- 172.21.15.46:0/3144565632 >> v1:172.21.15.46:6789/0 conn(0x7fb880070bb0 legacy=0x7fb880073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.498+0000 7fb885226640 1 -- 172.21.15.46:0/3144565632 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88015df40 con 0x7fb8800b4ff0 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.499+0000 7fb885226640 1 --2- 172.21.15.46:0/3144565632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880070670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb8700097d0 tx=0x7fb87002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5632e518f61cee5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.499+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb870037070 con 0x7fb8800b4ff0 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.499+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87002ab90 con 0x7fb8800b4ff0 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.499+0000 7fb884a25640 1 -- 172.21.15.46:0/3144565632 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c001c40 con 0x7fb8800b4ff0 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 -- 172.21.15.46:0/3144565632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 msgr2=0x7fb880070670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 --2- 172.21.15.46:0/3144565632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880070670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb8700097d0 tx=0x7fb87002a100 comp rx=0 tx=0).stop 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 -- 172.21.15.46:0/3144565632 shutdown_connections 2024-05-09T03:17:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 --2- 172.21.15.46:0/3144565632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880070670 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 -- 172.21.15.46:0/3144565632 >> 172.21.15.46:0/3144565632 conn(0x7fb8800b3910 msgr2=0x7fb8800b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 -- 172.21.15.46:0/3144565632 shutdown_connections 2024-05-09T03:17:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.500+0000 7fb887c8b640 1 -- 172.21.15.46:0/3144565632 wait complete. 2024-05-09T03:17:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.502+0000 7fb887c8b640 1 Processor -- start 2024-05-09T03:17:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.502+0000 7fb887c8b640 1 -- start start 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb887c8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 0x7fb880144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb887c8b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8800b1210 0x7fb8800b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb887c8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb887c8b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb88015d5a0 con 0x7fb880070bb0 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb887c8b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb8800b5400 con 0x7fb8800b1210 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb887c8b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb88015d720 con 0x7fb8800b4ff0 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 0x7fb880144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 0x7fb880144540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38922/0 (socket says 172.21.15.46:38922) 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 -- 172.21.15.46:0/470577550 learned_addr learned my addr 172.21.15.46:0/470577550 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885226640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8800b1210 0x7fb8800b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb886228640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 -- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 msgr2=0x7fb880155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 -- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8800b1210 msgr2=0x7fb8800b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8800b1210 0x7fb8800b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 -- 172.21.15.46:0/470577550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8801f5cc0 con 0x7fb880070bb0 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.503+0000 7fb885a27640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 0x7fb880144540 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb87800c3c0 tx=0x7fb878001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.504+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878017070 con 0x7fb880070bb0 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.504+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8780052f0 con 0x7fb880070bb0 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.504+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8801f5fa0 con 0x7fb880070bb0 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.504+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878003830 con 0x7fb880070bb0 2024-05-09T03:17:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.504+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88015fc40 con 0x7fb880070bb0 2024-05-09T03:17:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.505+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb88006a270 con 0x7fb880070bb0 2024-05-09T03:17:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.506+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb87801a070 con 0x7fb880070bb0 2024-05-09T03:17:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.506+0000 7fb8767fc640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8640762d0 0x7fb8640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.506+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7fb878097210 con 0x7fb880070bb0 2024-05-09T03:17:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.507+0000 7fb885226640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8640762d0 0x7fb8640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.507+0000 7fb885226640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8640762d0 0x7fb8640787e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb870034000 tx=0x7fb87002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.510+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb878021020 con 0x7fb880070bb0 2024-05-09T03:17:45.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.739+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8800b0b50 con 0x7fb880070bb0 2024-05-09T03:17:45.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.739+0000 7fb8767fc640 1 -- 172.21.15.46:0/470577550 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+4342 (secure 0 0 0) 0x7fb878060140 con 0x7fb880070bb0 2024-05-09T03:17:45.740 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:45.740 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":170,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:44.993974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:45.740 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 170 2024-05-09T03:17:45.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8640762d0 msgr2=0x7fb8640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8640762d0 0x7fb8640787e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb870034000 tx=0x7fb87002e000 comp rx=0 tx=0).stop 2024-05-09T03:17:45.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 msgr2=0x7fb880144540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 0x7fb880144540 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb87800c3c0 tx=0x7fb878001e70 comp rx=0 tx=0).stop 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 shutdown_connections 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb8800b4ff0 0x7fb880155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8800b1210 0x7fb8800b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb8640762d0 0x7fb8640787e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 --2- 172.21.15.46:0/470577550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb880070bb0 0x7fb880144540 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 >> 172.21.15.46:0/470577550 conn(0x7fb8800b3910 msgr2=0x7fb8801613b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 shutdown_connections 2024-05-09T03:17:45.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.741+0000 7fb887c8b640 1 -- 172.21.15.46:0/470577550 wait complete. 2024-05-09T03:17:45.753 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:45.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.867+0000 7fd93114d640 1 Processor -- start 2024-05-09T03:17:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.867+0000 7fd93114d640 1 -- start start 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.868+0000 7fd93114d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c070670 0x7fd92c070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.868+0000 7fd93114d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd92c073870 0x7fd92c073c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.868+0000 7fd93114d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd92c070fc0 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.868+0000 7fd93114d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd92c15d980 con 0x7fd92c073870 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.868+0000 7fd93114d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd92c15dc60 con 0x7fd92c070670 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd92c073870 0x7fd92c073c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd92c073870 0x7fd92c073c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59552/0 (socket says 172.21.15.46:59552) 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 -- 172.21.15.46:0/2365847513 learned_addr learned my addr 172.21.15.46:0/2365847513 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd92c0b4ff0 0x7fd92c0aeb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58010/0 (socket says 172.21.15.46:58010) 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92a59c640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c070670 0x7fd92c070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 -- 172.21.15.46:0/2365847513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c070670 msgr2=0x7fd92c070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c070670 0x7fd92c070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 -- 172.21.15.46:0/2365847513 >> v1:172.21.15.46:6789/0 conn(0x7fd92c0b4ff0 legacy=0x7fd92c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92b59e640 1 -- 172.21.15.46:0/2365847513 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92c15df40 con 0x7fd92c073870 2024-05-09T03:17:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd929d9b640 1 -- 172.21.15.46:0/2365847513 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2762250840 0 0) 0x7fd91c001110 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.869+0000 7fd92a59c640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c070670 0x7fd92c070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.870+0000 7fd92b59e640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c073870 0x7fd92c073c80 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fd924009e50 tx=0x7fd92402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82a781c77c017d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.870+0000 7fd929d9b640 1 -- 172.21.15.46:0/2365847513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924036070 con 0x7fd92c073870 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.870+0000 7fd929d9b640 1 -- 172.21.15.46:0/2365847513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd924031600 con 0x7fd92c073870 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.870+0000 7fd929d9b640 1 -- 172.21.15.46:0/2365847513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9240319d0 con 0x7fd92c073870 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.870+0000 7fd93114d640 1 -- 172.21.15.46:0/2365847513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c073870 msgr2=0x7fd92c073c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.870+0000 7fd93114d640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c073870 0x7fd92c073c80 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fd924009e50 tx=0x7fd92402a680 comp rx=0 tx=0).stop 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.871+0000 7fd93114d640 1 -- 172.21.15.46:0/2365847513 shutdown_connections 2024-05-09T03:17:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.871+0000 7fd93114d640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c073870 0x7fd92c073c80 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.871+0000 7fd93114d640 1 --2- 172.21.15.46:0/2365847513 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c070670 0x7fd92c070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.871+0000 7fd93114d640 1 -- 172.21.15.46:0/2365847513 >> 172.21.15.46:0/2365847513 conn(0x7fd92c0b3910 msgr2=0x7fd92c0b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.871+0000 7fd93114d640 1 -- 172.21.15.46:0/2365847513 shutdown_connections 2024-05-09T03:17:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.871+0000 7fd93114d640 1 -- 172.21.15.46:0/2365847513 wait complete. 2024-05-09T03:17:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.873+0000 7fd93114d640 1 Processor -- start 2024-05-09T03:17:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.873+0000 7fd93114d640 1 -- start start 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd93114d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd92c070670 0x7fd92c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd93114d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c073870 0x7fd92c15c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd93114d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 0x7fd92c143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd93114d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c15d540 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd93114d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c0b5400 con 0x7fd92c070670 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd93114d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd92c15d6c0 con 0x7fd92c073870 2024-05-09T03:17:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd92b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 0x7fd92c143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd92b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 0x7fd92c143300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59562/0 (socket says 172.21.15.46:59562) 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd92b59e640 1 -- 172.21.15.46:0/1698769376 learned_addr learned my addr 172.21.15.46:0/1698769376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.874+0000 7fd92ad9d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd92c070670 0x7fd92c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92a59c640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c073870 0x7fd92c15c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92b59e640 1 -- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c073870 msgr2=0x7fd92c15c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92b59e640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c073870 0x7fd92c15c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92b59e640 1 -- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd92c070670 msgr2=0x7fd92c15bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92b59e640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd92c070670 0x7fd92c15bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92b59e640 1 -- 172.21.15.46:0/1698769376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92c1f73c0 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd92b59e640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 0x7fd92c143300 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd924009e50 tx=0x7fd924004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924041020 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd924037040 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd92c1f6840 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924036050 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.875+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd92c1f6e00 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.876+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92c06a270 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.877+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd924007c30 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.878+0000 7fd9137fe640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd9180762d0 0x7fd9180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.878+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7fd9240b8140 con 0x7fd92c0b4ff0 2024-05-09T03:17:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.878+0000 7fd92ad9d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd9180762d0 0x7fd9180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.879+0000 7fd92ad9d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd9180762d0 0x7fd9180787e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd91c007fd0 tx=0x7fd91c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:45.883+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd92404b060 con 0x7fd92c0b4ff0 2024-05-09T03:17:46.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.116+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd92c0b67a0 con 0x7fd92c0b4ff0 2024-05-09T03:17:46.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.116+0000 7fd9137fe640 1 -- 172.21.15.46:0/1698769376 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+5168 (secure 0 0 0) 0x7fd924080ff0 con 0x7fd92c0b4ff0 2024-05-09T03:17:46.117 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:46.117 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":171,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3450505404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3450505404},{"type":"v1","addr":"172.21.15.46:6834","nonce":3450505404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":171}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:44.993974+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:46.117 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 171 2024-05-09T03:17:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd9180762d0 msgr2=0x7fd9180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd9180762d0 0x7fd9180787e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd91c007fd0 tx=0x7fd91c007de0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 msgr2=0x7fd92c143300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 0x7fd92c143300 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fd924009e50 tx=0x7fd924004a40 comp rx=0 tx=0).stop 2024-05-09T03:17:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 shutdown_connections 2024-05-09T03:17:46.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd9180762d0 0x7fd9180787e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd92c0b4ff0 0x7fd92c143300 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd92c073870 0x7fd92c15c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 --2- 172.21.15.46:0/1698769376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd92c070670 0x7fd92c15bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 >> 172.21.15.46:0/1698769376 conn(0x7fd92c0b3910 msgr2=0x7fd92c0b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 shutdown_connections 2024-05-09T03:17:46.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.118+0000 7fd93114d640 1 -- 172.21.15.46:0/1698769376 wait complete. 2024-05-09T03:17:46.130 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 100.00000000.inode' led to DAMAGED state 2024-05-09T03:17:46.130 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000' 2024-05-09T03:17:46.131 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:46.131 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:46.147 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:46.147 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:46.147 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:46.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:46.148 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:46.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.358+0000 7f3ac0566640 1 Processor -- start 2024-05-09T03:17:46.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.358+0000 7f3ac0566640 1 -- start start 2024-05-09T03:17:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3ac0566640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ab80d1110 0x7f3ab80d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3ac0566640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ab80b92c0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3ac0566640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ab80b2f10 con 0x7f3ab80d1920 2024-05-09T03:17:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3ac0566640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3ab80b9560 con 0x7f3ab80d1110 2024-05-09T03:17:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3abe302640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ab80d1110 0x7f3ab80d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:46.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3abeb03640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ab80d1d30 0x7f3ab80b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58026/0 (socket says 172.21.15.46:58026) 2024-05-09T03:17:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.359+0000 7f3abeb03640 1 -- 172.21.15.46:0/786720728 learned_addr learned my addr 172.21.15.46:0/786720728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.360+0000 7f3abe302640 1 -- 172.21.15.46:0/786720728 >> v1:172.21.15.88:6789/0 conn(0x7f3ab80d1920 legacy=0x7f3ab80b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.360+0000 7f3abe302640 1 -- 172.21.15.46:0/786720728 >> v1:172.21.15.46:6789/0 conn(0x7f3ab80d1d30 legacy=0x7f3ab80b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.360+0000 7f3abe302640 1 -- 172.21.15.46:0/786720728 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab80cfb20 con 0x7f3ab80d1110 2024-05-09T03:17:46.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.360+0000 7f3abd300640 1 -- 172.21.15.46:0/786720728 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950949986 0 0) 0x7f3ab80b92c0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.360+0000 7f3abe302640 1 --2- 172.21.15.46:0/786720728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1110 0x7f3ab80d0020 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00097a0 tx=0x7f3ab002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b67f0bddc75ba3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3abd300640 1 -- 172.21.15.46:0/786720728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0037070 con 0x7f3ab80d1110 2024-05-09T03:17:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3abd300640 1 -- 172.21.15.46:0/786720728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab0032440 con 0x7f3ab80d1110 2024-05-09T03:17:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3abd300640 1 -- 172.21.15.46:0/786720728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab002ae60 con 0x7f3ab80d1110 2024-05-09T03:17:46.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 -- 172.21.15.46:0/786720728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1110 msgr2=0x7f3ab80d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 --2- 172.21.15.46:0/786720728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1110 0x7f3ab80d0020 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f3ab00097a0 tx=0x7f3ab002a110 comp rx=0 tx=0).stop 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 -- 172.21.15.46:0/786720728 shutdown_connections 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 --2- 172.21.15.46:0/786720728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1110 0x7f3ab80d0020 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 -- 172.21.15.46:0/786720728 >> 172.21.15.46:0/786720728 conn(0x7f3ab80d05f0 msgr2=0x7f3ab80d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 -- 172.21.15.46:0/786720728 shutdown_connections 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.361+0000 7f3ac0566640 1 -- 172.21.15.46:0/786720728 wait complete. 2024-05-09T03:17:46.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.363+0000 7f3ac0566640 1 Processor -- start 2024-05-09T03:17:46.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.363+0000 7f3ac0566640 1 -- start start 2024-05-09T03:17:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.364+0000 7f3ac0566640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab80d1920 0x7f3ab81f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.364+0000 7f3ac0566640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 0x7f3ab806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.364+0000 7f3ac0566640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ab80a0990 0x7f3ab80a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:46.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.364+0000 7f3ac0566640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab809d030 con 0x7f3ab80d1d30 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.364+0000 7f3ac0566640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab809ceb0 con 0x7f3ab80d1920 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.364+0000 7f3ac0566640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab80b8e00 con 0x7f3ab80a0990 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 0x7f3ab806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 0x7f3ab806d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59574/0 (socket says 172.21.15.46:59574) 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 -- 172.21.15.46:0/3536348207 learned_addr learned my addr 172.21.15.46:0/3536348207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abeb03640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ab80a0990 0x7f3ab80a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abe302640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab80d1920 0x7f3ab81f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 -- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ab80a0990 msgr2=0x7f3ab80a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ab80a0990 0x7f3ab80a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 -- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab80d1920 msgr2=0x7f3ab81f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab80d1920 0x7f3ab81f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 -- 172.21.15.46:0/3536348207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab80a12e0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3abdb01640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 0x7f3ab806d7c0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00c3d0 tx=0x7f3aac00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:46.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac017070 con 0x7f3ab80d1d30 2024-05-09T03:17:46.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aac0052d0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab81f64a0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac0037f0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.365+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab806e5f0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.366+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab806a270 con 0x7f3ab80d1d30 2024-05-09T03:17:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.367+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3aac01a070 con 0x7f3ab80d1d30 2024-05-09T03:17:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.368+0000 7f3ab6ffd640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3aa4076300 0x7f3aa4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.368+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7f3aac096cf0 con 0x7f3ab80d1d30 2024-05-09T03:17:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.368+0000 7f3abe302640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3aa4076300 0x7f3aa4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:46.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.368+0000 7f3abe302640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3aa4076300 0x7f3aa4078810 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0034000 tx=0x7f3ab002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:46.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.372+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3aac09b070 con 0x7f3ab80d1d30 2024-05-09T03:17:46.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:46.603+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3ab80b3c20 con 0x7f3ab80d1d30 2024-05-09T03:17:47.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.000+0000 7f3ab6ffd640 1 -- 172.21.15.46:0/3536348207 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v172) ==== 161+0+0 (secure 0 0 0) 0x7f3ab80a12e0 con 0x7f3ab80d1d30 2024-05-09T03:17:47.001 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3aa4076300 msgr2=0x7f3aa4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3aa4076300 0x7f3aa4078810 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0034000 tx=0x7f3ab002e000 comp rx=0 tx=0).stop 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 msgr2=0x7f3ab806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 0x7f3ab806d7c0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f3aac00c3d0 tx=0x7f3aac00c890 comp rx=0 tx=0).stop 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 shutdown_connections 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ab80a0990 0x7f3ab80a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3aa4076300 0x7f3aa4078810 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ab80d1d30 0x7f3ab806d7c0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 --2- 172.21.15.46:0/3536348207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab80d1920 0x7f3ab81f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 >> 172.21.15.46:0/3536348207 conn(0x7f3ab80d05f0 msgr2=0x7f3ab80b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:47.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.002+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 shutdown_connections 2024-05-09T03:17:47.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.003+0000 7f3ac0566640 1 -- 172.21.15.46:0/3536348207 wait complete. 2024-05-09T03:17:47.015 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:47.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.133+0000 7f7702a19640 1 Processor -- start 2024-05-09T03:17:47.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.133+0000 7f7702a19640 1 -- start start 2024-05-09T03:17:47.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f7702a19640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76fc0d3fc0 0x7f76fc16b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:47.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f7702a19640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76fc0b5550 con 0x7f76fc0b5140 2024-05-09T03:17:47.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f7702a19640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76fc0d44e0 con 0x7f76fc0b7e70 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f7702a19640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f76fc0d3480 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f7700fb6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f76fc0d3fc0 0x7f76fc16b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f76fbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f76fc0b5140 0x7f76fc0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58036/0 (socket says 172.21.15.46:58036) 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.134+0000 7f76fbfff640 1 -- 172.21.15.46:0/2786660850 learned_addr learned my addr 172.21.15.46:0/2786660850 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.135+0000 7f7700fb6640 1 -- 172.21.15.46:0/2786660850 >> v1:172.21.15.88:6789/0 conn(0x7f76fc0b7e70 legacy=0x7f76fc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.135+0000 7f7700fb6640 1 -- 172.21.15.46:0/2786660850 >> v1:172.21.15.46:6789/0 conn(0x7f76fc0b5140 legacy=0x7f76fc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.135+0000 7f76faffd640 1 -- 172.21.15.46:0/2786660850 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267853831 0 0) 0x7f76fc0b5550 con 0x7f76fc0b5140 2024-05-09T03:17:47.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.135+0000 7f7700fb6640 1 -- 172.21.15.46:0/2786660850 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc0d47c0 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.135+0000 7f7700fb6640 1 --2- 172.21.15.46:0/2786660850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc16b0d0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009d90 tx=0x7f76f402a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0f4c61e48390ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f76faffd640 1 -- 172.21.15.46:0/2786660850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4037070 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f76faffd640 1 -- 172.21.15.46:0/2786660850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f4007740 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f76faffd640 1 -- 172.21.15.46:0/2786660850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4007b10 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 -- 172.21.15.46:0/2786660850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 msgr2=0x7f76fc16b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 --2- 172.21.15.46:0/2786660850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc16b0d0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009d90 tx=0x7f76f402a780 comp rx=0 tx=0).stop 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 -- 172.21.15.46:0/2786660850 shutdown_connections 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 --2- 172.21.15.46:0/2786660850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc16b0d0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 -- 172.21.15.46:0/2786660850 >> 172.21.15.46:0/2786660850 conn(0x7f76fc08d3f0 msgr2=0x7f76fc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:47.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 -- 172.21.15.46:0/2786660850 shutdown_connections 2024-05-09T03:17:47.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.136+0000 7f7702a19640 1 -- 172.21.15.46:0/2786660850 wait complete. 2024-05-09T03:17:47.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.138+0000 7f7702a19640 1 Processor -- start 2024-05-09T03:17:47.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.138+0000 7f7702a19640 1 -- start start 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc0b5140 0x7f76fc153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc0b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc0b6f70 0x7f76fc15c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc0d5580 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc0d4d80 con 0x7f76fc0b6f70 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f76fc0d5700 con 0x7f76fc0b5140 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc0b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc0b6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59594/0 (socket says 172.21.15.46:59594) 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 -- 172.21.15.46:0/264780236 learned_addr learned my addr 172.21.15.46:0/264780236 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:47.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7700fb6640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc0b6f70 0x7f76fc15c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fbfff640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc0b5140 0x7f76fc153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 -- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc0b5140 msgr2=0x7f76fc153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc0b5140 0x7f76fc153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 -- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc0b6f70 msgr2=0x7f76fc15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc0b6f70 0x7f76fc15c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 -- 172.21.15.46:0/264780236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc1568a0 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7700fb6640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc0b6f70 0x7f76fc15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fbfff640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc0b5140 0x7f76fc153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc0b6a30 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c400 tx=0x7f76f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0017070 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f00052d0 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76fc156b80 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0003830 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.140+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76fc157140 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.141+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc06a270 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.142+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f76f001a070 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.143+0000 7f76f8ff9640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76e8076300 0x7f76e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:47.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.143+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7f76f0096ed0 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.143+0000 7f76fbfff640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76e8076300 0x7f76e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:47.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.144+0000 7f76fbfff640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76e8076300 0x7f76e8078810 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f76ec007fd0 tx=0x7f76ec007dc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:47.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.148+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f76f00602b0 con 0x7f76fc0d3fc0 2024-05-09T03:17:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:47.414+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f76fc160c10 con 0x7f76fc0d3fc0 2024-05-09T03:17:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.006+0000 7f76f8ff9640 1 -- 172.21.15.46:0/264780236 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v173) ==== 101+0+0 (secure 0 0 0) 0x7f76f0068330 con 0x7f76fc0d3fc0 2024-05-09T03:17:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76e8076300 msgr2=0x7f76e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76e8076300 0x7f76e8078810 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f76ec007fd0 tx=0x7f76ec007dc0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 msgr2=0x7f76fc0b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc0b6a30 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c400 tx=0x7f76f000c8c0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 shutdown_connections 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc0b6f70 0x7f76fc15c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76e8076300 0x7f76e8078810 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76fc0d3fc0 0x7f76fc0b6a30 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 --2- 172.21.15.46:0/264780236 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc0b5140 0x7f76fc153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 >> 172.21.15.46:0/264780236 conn(0x7f76fc08d3f0 msgr2=0x7f76fc1e8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:48.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 shutdown_connections 2024-05-09T03:17:48.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.010+0000 7f7702a19640 1 -- 172.21.15.46:0/264780236 wait complete. 2024-05-09T03:17:48.025 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:48.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.085+0000 7f646e51ef40 1 Processor -- start 2024-05-09T03:17:48.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.085+0000 7f646e51ef40 1 -- start start 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f646e51ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 0x561373540a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f646e51ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734977b0 0x561373530e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f646e51ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x56137353fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f646e51ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5613734ac4d0 con 0x5613734657e0 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f646e51ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56137340ef50 con 0x5613734ae010 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f646e51ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5613734b18a0 con 0x5613734977b0 2024-05-09T03:17:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f6467518640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 0x561373540a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f6467518640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 0x561373540a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59622/0 (socket says 172.21.15.46:59622) 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.086+0000 7f6467518640 1 -- 172.21.15.46:0/2260936681 learned_addr learned my addr 172.21.15.46:0/2260936681 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467d19640 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x56137353fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6466d17640 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734977b0 0x561373530e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467518640 1 -- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734977b0 msgr2=0x561373530e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467518640 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734977b0 0x561373530e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467518640 1 -- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 msgr2=0x56137353fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467518640 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x56137353fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467518640 1 -- 172.21.15.46:0/2260936681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561373496cd0 con 0x5613734657e0 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467d19640 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x56137353fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6467518640 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 0x561373540a10 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f64580097e0 tx=0x7f6458005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb636b348ac0958 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6465d15640 1 -- 172.21.15.46:0/2260936681 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458019070 con 0x5613734657e0 2024-05-09T03:17:48.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6465d15640 1 -- 172.21.15.46:0/2260936681 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6458008890 con 0x5613734657e0 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.087+0000 7f6465d15640 1 -- 172.21.15.46:0/2260936681 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458008c60 con 0x5613734657e0 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 -- 172.21.15.46:0/2260936681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 msgr2=0x561373540a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 0x561373540a10 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f64580097e0 tx=0x7f6458005a00 comp rx=0 tx=0).stop 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 -- 172.21.15.46:0/2260936681 shutdown_connections 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x56137353fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734977b0 0x561373530e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 --2- 172.21.15.46:0/2260936681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734657e0 0x561373540a10 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 -- 172.21.15.46:0/2260936681 >> 172.21.15.46:0/2260936681 conn(0x56137338aac0 msgr2=0x5613734afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 -- 172.21.15.46:0/2260936681 shutdown_connections 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.088+0000 7f646e51ef40 1 -- 172.21.15.46:0/2260936681 wait complete. 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.089+0000 7f646e51ef40 1 Processor -- start 2024-05-09T03:17:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.089+0000 7f646e51ef40 1 -- start start 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f646e51ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734657e0 0x56137353ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f646e51ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 0x5613735d8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f646e51ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x5613735d93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f646e51ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561373540f50 con 0x5613734977b0 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f646e51ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56137353bcb0 con 0x5613734ae010 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f646e51ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56137352ed50 con 0x5613734657e0 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f6466d17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 0x5613735d8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f6466d17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 0x5613735d8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59626/0 (socket says 172.21.15.46:59626) 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 learned_addr learned my addr 172.21.15.46:0/3765657325 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734657e0 0x56137353ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.090+0000 7f6467d19640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x5613735d93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734657e0 msgr2=0x56137353ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5613734657e0 0x56137353ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 msgr2=0x5613735d93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5613734ae010 0x5613735d93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56137352d5b0 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 0x5613735d8e80 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f645c025be0 tx=0x7f645c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6464d13640 1 -- 172.21.15.46:0/3765657325 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c064070 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6464d13640 1 -- 172.21.15.46:0/3765657325 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645c054b30 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56137352efa0 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f6464d13640 1 -- 172.21.15.46:0/3765657325 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c059040 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.091+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56137353c140 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.093+0000 7f6464d13640 1 -- 172.21.15.46:0/3765657325 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f645c067070 con 0x5613734977b0 2024-05-09T03:17:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.093+0000 7f6464d13640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454074cb0 0x7f64540771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.093+0000 7f6464d13640 1 -- 172.21.15.46:0/3765657325 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7f645c0da2d0 con 0x5613734977b0 2024-05-09T03:17:48.096 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.094+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56137353a8a0 0x5613735e2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.094+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454074cb0 0x7f64540771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.094+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x561373540510 con 0x56137353a8a0 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.094+0000 7f6467d19640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56137353a8a0 0x5613735e2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.094+0000 7f6467d19640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56137353a8a0 0x5613735e2150 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.094+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454074cb0 0x7f64540771c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6458006010 tx=0x7f6458008000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.096+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v98'412 uv386 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.096+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613734b0280 con 0x56137353a8a0 2024-05-09T03:17:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.097+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v98'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.098+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735d9900 con 0x56137353a8a0 2024-05-09T03:17:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.099+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v98'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.099 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.099+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e36a0 con 0x56137353a8a0 2024-05-09T03:17:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.100+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v98'415 uv415 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.100+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e3a40 con 0x56137353a8a0 2024-05-09T03:17:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.101+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v98'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.101+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2690 con 0x56137353a8a0 2024-05-09T03:17:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.103+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v98'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.103+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2a80 con 0x56137353a8a0 2024-05-09T03:17:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.104+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v98'418 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.104+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e2e70 con 0x56137353a8a0 2024-05-09T03:17:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.106+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v98'419 uv419 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.106+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e70 con 0x56137353a8a0 2024-05-09T03:17:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.107+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v98'420 uv420 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.107 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.107+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.108+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v98'421 uv421 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.108+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.110+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v98'422 uv422 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.110+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.111+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v98'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.111+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.112+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v98'424 uv399 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.112+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.114+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v98'425 uv425 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.114+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.115+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v98'426 uv426 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.116 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.115+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.117+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v98'427 uv427 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.117+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.118+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v98'428 uv428 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.118+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.119+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v98'429 uv429 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.119+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.120+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v98'430 uv430 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.120+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.122+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v98'431 uv405 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.122+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.123+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v98'432 uv432 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.123+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.124+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v98'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.125 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:48.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.124+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.125+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v98'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.125+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.127+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v98'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.127+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.128+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v98'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.128+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.129+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v98'437 uv411 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.130+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.131+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v98'438 uv438 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.131+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.132+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v98'439 uv439 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.133 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.132+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.133+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v98'440 uv440 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.133+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.135+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v98'441 uv441 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.135+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x56137353a8a0 2024-05-09T03:17:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.136+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v98'442 uv442 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.136+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5613735e3220 0x5613735e6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.136+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e2e20 con 0x5613735e3220 2024-05-09T03:17:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.136+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5613735e3220 0x5613735e6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.137+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5613735e3220 0x5613735e6300 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.138+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v98'305 uv288 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.138+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e6840 con 0x5613735e3220 2024-05-09T03:17:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.140+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v98'306 uv306 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.140+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e6be0 con 0x5613735e3220 2024-05-09T03:17:48.142 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:48.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.141+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v98'307 uv307 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.141+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e6f80 con 0x5613735e3220 2024-05-09T03:17:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.143+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v98'308 uv308 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.143+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7320 con 0x5613735e3220 2024-05-09T03:17:48.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.144+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v98'309 uv309 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.144+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e76c0 con 0x5613735e3220 2024-05-09T03:17:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.146+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v98'310 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.146+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e7a60 con 0x5613735e3220 2024-05-09T03:17:48.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.147+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v98'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.147+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.149 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:48.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.148+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v98'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.148+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.150+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v98'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.150+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.151+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v98'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.151+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.153+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v98'315 uv298 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.153+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.154+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v98'316 uv316 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.154+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.156 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:48.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.155+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v98'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.155+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735eb860 con 0x5613735e3220 2024-05-09T03:17:48.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.157+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v98'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.157+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735eb860 con 0x5613735e3220 2024-05-09T03:17:48.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.158+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v98'319 uv319 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.158+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.160+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v98'320 uv304 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.160+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.161+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v98'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.161+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.163 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.162+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v98'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.162+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e8e10 con 0x5613735e3220 2024-05-09T03:17:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.163+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v98'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.163+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e8e10 con 0x5613735e3220 2024-05-09T03:17:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.165+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v98'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.165+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x5613735e3220 2024-05-09T03:17:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.166+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v98'325 uv325 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735e3220 2024-05-09T03:17:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.166+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x56137353a8a0 2024-05-09T03:17:48.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.168+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v98'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.168+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735e7e00 con 0x56137353a8a0 2024-05-09T03:17:48.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.169+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v98'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.169+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x56137353a8a0 2024-05-09T03:17:48.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.170+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v98'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.170 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.170+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e8e10 con 0x56137353a8a0 2024-05-09T03:17:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.171+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v98'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.171+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e8e10 con 0x56137353a8a0 2024-05-09T03:17:48.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.172+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v98'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.172+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735e7e00 con 0x56137353a8a0 2024-05-09T03:17:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.173+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v98'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.173+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5613735e8310 0x5613735ea730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.173+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735eac70 con 0x5613735e8310 2024-05-09T03:17:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.173+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5613735e8310 0x5613735ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.173+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5613735e8310 0x5613735ea730 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.174+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v98'66 uv65 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.174+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735eb0f0 con 0x5613735e8310 2024-05-09T03:17:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.175+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v98'67 uv67 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.175+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735eb5f0 con 0x5613735e8310 2024-05-09T03:17:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.176+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v98'68 uv68 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.177 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.176+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x56137353ad90 con 0x5613735e8310 2024-05-09T03:17:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.177+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v98'69 uv69 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.178+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ebf20 con 0x56137353a8a0 2024-05-09T03:17:48.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.179+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v98'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.179+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735ebf20 con 0x56137353a8a0 2024-05-09T03:17:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.180+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v98'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.180+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ebf20 con 0x56137353a8a0 2024-05-09T03:17:48.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.182+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v98'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.182 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:48.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.182+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ecf30 con 0x56137353a8a0 2024-05-09T03:17:48.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.183+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v98'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.183+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ecf30 con 0x56137353a8a0 2024-05-09T03:17:48.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.184+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v98'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.184+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ec3d0 con 0x56137353a8a0 2024-05-09T03:17:48.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.185+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v98'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.185+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5613735ecca0 0x5613735ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.185+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ec3d0 con 0x5613735ecca0 2024-05-09T03:17:48.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.185+0000 7f6467d19640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5613735ecca0 0x5613735ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.186+0000 7f6467d19640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5613735ecca0 0x5613735ef0c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.187+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v98'149 uv143 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.187+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.188+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v98'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.188+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.189+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v98'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.190 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:48.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.190+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f0610 con 0x5613735ecca0 2024-05-09T03:17:48.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.191+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v98'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.191+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f0610 con 0x5613735ecca0 2024-05-09T03:17:48.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.192+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v98'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.192+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.194+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v98'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.194+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.195+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v98'155 uv148 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.195+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.197+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v98'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.197+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.198 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:48.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.198+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v98'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.198+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.199+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v98'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.199+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ef600 con 0x5613735ecca0 2024-05-09T03:17:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.201+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v98'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.201+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5613735efc80 0x5613735f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.201+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f27d0 con 0x5613735efc80 2024-05-09T03:17:48.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.201+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5613735efc80 0x5613735f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.201+0000 7f6467518640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5613735efc80 0x5613735f20a0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.203+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v98'66 uv65 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735efc80 2024-05-09T03:17:48.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.203+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x5613735efc80 2024-05-09T03:17:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.204+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v98'67 uv67 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735efc80 2024-05-09T03:17:48.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.204+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x5613735efc80 2024-05-09T03:17:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.205+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v98'68 uv68 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735efc80 2024-05-09T03:17:48.205 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.205+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x5613735efc80 2024-05-09T03:17:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.206+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v98'69 uv69 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735efc80 2024-05-09T03:17:48.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.206+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x5613735efc80 2024-05-09T03:17:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.207+0000 7f6467518640 1 -- 172.21.15.46:0/3765657325 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v98'70 uv70 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6458027040 con 0x5613735efc80 2024-05-09T03:17:48.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.207+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x56137353a8a0 2024-05-09T03:17:48.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.208+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v98'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.208+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x56137353a8a0 2024-05-09T03:17:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.209+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v98'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.209+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x56137353a8a0 2024-05-09T03:17:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.210+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v98'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.211 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:48.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.210+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f3bf0 con 0x56137353a8a0 2024-05-09T03:17:48.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.211+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v98'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.211+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f3bb0 con 0x56137353a8a0 2024-05-09T03:17:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.212+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v98'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.212+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x56137353a8a0 2024-05-09T03:17:48.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.213+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v98'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x56137353a8a0 2024-05-09T03:17:48.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.214+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f2ba0 con 0x5613735e8310 2024-05-09T03:17:48.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.215+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v98'132 uv126 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.215+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f2f40 con 0x5613735e8310 2024-05-09T03:17:48.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.216+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v98'133 uv133 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.216+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f32e0 con 0x5613735e8310 2024-05-09T03:17:48.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.217+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v98'134 uv134 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.217 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:48.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.217+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ebfc0 con 0x5613735e8310 2024-05-09T03:17:48.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.218+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v98'135 uv135 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.218+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ebfc0 con 0x5613735e8310 2024-05-09T03:17:48.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.219+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v98'136 uv136 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.219+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ebfc0 con 0x5613735e8310 2024-05-09T03:17:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.220+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v98'137 uv131 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.220+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f3bd0 con 0x5613735e8310 2024-05-09T03:17:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.221+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v98'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.221+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ec010 con 0x5613735e8310 2024-05-09T03:17:48.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.222+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v98'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.223 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:48.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.222+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735ec010 con 0x5613735e8310 2024-05-09T03:17:48.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.223+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v98'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.223+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6980 con 0x5613735e8310 2024-05-09T03:17:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.224+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v98'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.225+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5613735f4530 0x5613735f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.225+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735f4530 2024-05-09T03:17:48.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.225+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5613735f4530 0x5613735f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.225+0000 7f6466d17640 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5613735f4530 0x5613735f6950 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.226+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v98'61 uv60 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735f4530 2024-05-09T03:17:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.226+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735f4530 2024-05-09T03:17:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.227+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v98'62 uv62 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735f4530 2024-05-09T03:17:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.227+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735f4530 2024-05-09T03:17:48.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.228+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v98'63 uv63 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735f4530 2024-05-09T03:17:48.229 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:48.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.228+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735f4530 2024-05-09T03:17:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.229+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v98'64 uv64 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735f4530 2024-05-09T03:17:48.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.229+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735e8310 2024-05-09T03:17:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.230+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v98'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.230+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735e8310 2024-05-09T03:17:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.232+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v98'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.232+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735e8310 2024-05-09T03:17:48.233 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.233+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v98'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.233+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f7ea0 con 0x5613735e8310 2024-05-09T03:17:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.234+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v98'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.234+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f7ea0 con 0x5613735e8310 2024-05-09T03:17:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.235+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v98'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.235+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735e8310 2024-05-09T03:17:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.236+0000 7f6466d17640 1 -- 172.21.15.46:0/3765657325 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v98'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f645c070070 con 0x5613735e8310 2024-05-09T03:17:48.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.236+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.237+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v98'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.237+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.239+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v98'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.239+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.240+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v98'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.240 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.240+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.241+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v98'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.241+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.242+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v98'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.242+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.243+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v98'170 uv170 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.244 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:48.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.244+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.245+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v98'171 uv171 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.245+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.246+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v98'172 uv172 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.246+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.247+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 1.00000000 [create] v98'173 uv168 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.247+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.248+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 1.00000000 [delete] v98'174 uv174 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.248+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.249+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v98'175 uv175 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.249 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:48.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.249+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.250+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v98'176 uv176 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.250+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.251+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v98'177 uv177 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.251+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f6e90 con 0x5613735ecca0 2024-05-09T03:17:48.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.252+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [omap-set-header] v98'178 uv178 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.252+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x5613735f72a0 con 0x5613735ecca0 2024-05-09T03:17:48.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v98'179 uv179 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6460003450 con 0x5613735ecca0 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56137353a8a0 msgr2=0x5613735e2150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56137353a8a0 0x5613735e2150 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5613735e8310 msgr2=0x5613735ea730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5613735e8310 0x5613735ea730 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5613735f4530 msgr2=0x5613735f6950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5613735f4530 0x5613735f6950 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5613735e3220 msgr2=0x5613735e6300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5613735e3220 0x5613735e6300 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5613735ecca0 msgr2=0x5613735ef0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5613735ecca0 0x5613735ef0c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5613735efc80 msgr2=0x5613735f20a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 reap_dead start 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5613735efc80 0x5613735f20a0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f6467d19640 1 -- 172.21.15.46:0/3765657325 reap_dead start 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454074cb0 msgr2=0x7f64540771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454074cb0 0x7f64540771c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6458006010 tx=0x7f6458008000 comp rx=0 tx=0).stop 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 msgr2=0x5613735d8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 0x5613735d8e80 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f645c025be0 tx=0x7f645c002fb0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 shutdown_connections 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5613735efc80 0x5613735f20a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6454074cb0 0x7f64540771c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 --2- 172.21.15.46:0/3765657325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613734977b0 0x5613735d8e80 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 >> 172.21.15.46:0/3765657325 conn(0x56137338aac0 msgr2=0x5613734afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 shutdown_connections 2024-05-09T03:17:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.253+0000 7f646e51ef40 1 -- 172.21.15.46:0/3765657325 wait complete. 2024-05-09T03:17:48.257 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000 2024-05-09T03:17:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.352+0000 7f9bc620ef40 1 Processor -- start 2024-05-09T03:17:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.352+0000 7f9bc620ef40 1 -- start start 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bc620ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d457750 0x55825d4564b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bc620ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d4569f0 0x55825d4e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bc620ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bc620ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55825d451cb0 con 0x55825d4e4e10 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bc620ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55825d3b8f50 con 0x55825d457750 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bc620ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55825d40fbf0 con 0x55825d4569f0 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59652/0 (socket says 172.21.15.46:59652) 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 -- 172.21.15.46:0/290964483 learned_addr learned my addr 172.21.15.46:0/290964483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbef68640 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d457750 0x55825d4564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbe767640 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d4569f0 0x55825d4e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 -- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d4569f0 msgr2=0x55825d4e48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d4569f0 0x55825d4e48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 -- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d457750 msgr2=0x55825d4564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d457750 0x55825d4564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 -- 172.21.15.46:0/290964483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825d453390 con 0x55825d4e4e10 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbf769640 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e5220 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8005fd0 tx=0x7f9bb8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447abef79b3526e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbd765640 1 -- 172.21.15.46:0/290964483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8017030 con 0x55825d4e4e10 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.354+0000 7f9bbd765640 1 -- 172.21.15.46:0/290964483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bb80109b0 con 0x55825d4e4e10 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bbd765640 1 -- 172.21.15.46:0/290964483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb800a3b0 con 0x55825d4e4e10 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 -- 172.21.15.46:0/290964483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 msgr2=0x55825d4e5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e5220 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8005fd0 tx=0x7f9bb8005d80 comp rx=0 tx=0).stop 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 -- 172.21.15.46:0/290964483 shutdown_connections 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e5220 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d4569f0 0x55825d4e48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/290964483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d457750 0x55825d4564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 -- 172.21.15.46:0/290964483 >> 172.21.15.46:0/290964483 conn(0x55825d334ac0 msgr2=0x55825d440800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 -- 172.21.15.46:0/290964483 shutdown_connections 2024-05-09T03:17:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.355+0000 7f9bc620ef40 1 -- 172.21.15.46:0/290964483 wait complete. 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.356+0000 7f9bc620ef40 1 Processor -- start 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.356+0000 7f9bc620ef40 1 -- start start 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.357+0000 7f9bc620ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d4569f0 0x55825d57d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.357+0000 7f9bc620ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d457750 0x55825d4dd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.357+0000 7f9bc620ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.357+0000 7f9bc620ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55825d453600 con 0x55825d4e4e10 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.357+0000 7f9bc620ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55825d4e5760 con 0x55825d4569f0 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.357+0000 7f9bc620ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55825d4e6380 con 0x55825d457750 2024-05-09T03:17:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59668/0 (socket says 172.21.15.46:59668) 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 -- 172.21.15.46:0/839439203 learned_addr learned my addr 172.21.15.46:0/839439203 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbef68640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d4569f0 0x55825d57d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbe767640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d457750 0x55825d4dd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 -- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d457750 msgr2=0x55825d4dd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55825d457750 0x55825d4dd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 -- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d4569f0 msgr2=0x55825d57d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55825d4569f0 0x55825d57d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 -- 172.21.15.46:0/839439203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825d4e21c0 con 0x55825d4e4e10 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bbf769640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e1540 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x55825d4e7cd0 tx=0x7f9bb8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9ba7fff640 1 -- 172.21.15.46:0/839439203 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb802a070 con 0x55825d4e4e10 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9ba7fff640 1 -- 172.21.15.46:0/839439203 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bb800a370 con 0x55825d4e4e10 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55825d4e24a0 con 0x55825d4e4e10 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9ba7fff640 1 -- 172.21.15.46:0/839439203 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8025410 con 0x55825d4e4e10 2024-05-09T03:17:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.358+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55825d57dde0 con 0x55825d4e4e10 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.360+0000 7f9ba7fff640 1 -- 172.21.15.46:0/839439203 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9bb802d070 con 0x55825d4e4e10 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9ba7fff640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bac074ce0 0x7f9bac0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9ba7fff640 1 -- 172.21.15.46:0/839439203 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7f9bb8097f10 con 0x55825d4e4e10 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9bbef68640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bac074ce0 0x7f9bac0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55825d43d610 0x55825d4e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e98) -- 0x55825d4e07e0 con 0x55825d43d610 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9bbe767640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55825d43d610 0x55825d4e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9bbef68640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bac074ce0 0x7f9bac0771f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0056000 tx=0x7f9bb0051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.361+0000 7f9bbe767640 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55825d43d610 0x55825d4e02a0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bbe767640 1 -- 172.21.15.46:0/839439203 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 1.00000000 [delete] v98'180 uv180 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9bb40098d0 con 0x55825d43d610 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55825d43d610 msgr2=0x55825d4e02a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55825d43d610 0x55825d4e02a0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bac074ce0 msgr2=0x7f9bac0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bac074ce0 0x7f9bac0771f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0056000 tx=0x7f9bb0051000 comp rx=0 tx=0).stop 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 msgr2=0x55825d4e1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 --2- 172.21.15.46:0/839439203 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55825d4e4e10 0x55825d4e1540 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x55825d4e7cd0 tx=0x7f9bb8003330 comp rx=0 tx=0).stop 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bbf769640 1 -- 172.21.15.46:0/839439203 reap_dead start 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 shutdown_connections 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 >> 172.21.15.46:0/839439203 conn(0x55825d334ac0 msgr2=0x55825d440800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 shutdown_connections 2024-05-09T03:17:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.363+0000 7f9bc620ef40 1 -- 172.21.15.46:0/839439203 wait complete. 2024-05-09T03:17:48.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.483+0000 7fecec0a7640 1 Processor -- start 2024-05-09T03:17:48.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.483+0000 7fecec0a7640 1 -- start start 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.484+0000 7fecec0a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fece40bb770 0x7fece40d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.484+0000 7fecec0a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece4164900 con 0x7fece4163eb0 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.484+0000 7fecec0a7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece4164be0 con 0x7fece40d3180 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.484+0000 7fecec0a7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fece40d4bb0 con 0x7fece40bb770 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece9642640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fece4163eb0 0x7fece40d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58056/0 (socket says 172.21.15.46:58056) 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece9642640 1 -- 172.21.15.46:0/471612746 learned_addr learned my addr 172.21.15.46:0/471612746 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece9e43640 1 --2- 172.21.15.46:0/471612746 >> v2:172.21.15.88:3300/0 conn(0x7fece40bb770 0x7fece40d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece8e41640 1 -- 172.21.15.46:0/471612746 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120588395 0 0) 0x7fece4164900 con 0x7fece4163eb0 2024-05-09T03:17:48.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece8e41640 1 -- 172.21.15.46:0/471612746 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecd0003520 con 0x7fece4163eb0 2024-05-09T03:17:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece8e41640 1 -- 172.21.15.46:0/471612746 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 250105965 0 0) 0x7fece4164be0 con 0x7fece40d3180 2024-05-09T03:17:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece9e43640 1 -- 172.21.15.46:0/471612746 >> v1:172.21.15.88:6789/0 conn(0x7fece40d3180 legacy=0x7fece40b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece9e43640 1 -- 172.21.15.46:0/471612746 >> v1:172.21.15.46:6789/0 conn(0x7fece4163eb0 legacy=0x7fece40d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.485+0000 7fece9e43640 1 -- 172.21.15.46:0/471612746 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece40d4840 con 0x7fece40bb770 2024-05-09T03:17:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.486+0000 7fece9e43640 1 --2- 172.21.15.46:0/471612746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece40bb770 0x7fece40d5040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fece00097a0 tx=0x7fece002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d4025ce07e52044 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.486+0000 7fece8e41640 1 -- 172.21.15.46:0/471612746 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0037070 con 0x7fece40bb770 2024-05-09T03:17:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.486+0000 7fece8e41640 1 -- 172.21.15.46:0/471612746 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece0032440 con 0x7fece40bb770 2024-05-09T03:17:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.486+0000 7fece8e41640 1 -- 172.21.15.46:0/471612746 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0032810 con 0x7fece40bb770 2024-05-09T03:17:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 -- 172.21.15.46:0/471612746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece40bb770 msgr2=0x7fece40d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 --2- 172.21.15.46:0/471612746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece40bb770 0x7fece40d5040 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fece00097a0 tx=0x7fece002a230 comp rx=0 tx=0).stop 2024-05-09T03:17:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 -- 172.21.15.46:0/471612746 shutdown_connections 2024-05-09T03:17:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 --2- 172.21.15.46:0/471612746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece40bb770 0x7fece40d5040 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 -- 172.21.15.46:0/471612746 >> 172.21.15.46:0/471612746 conn(0x7fece406e370 msgr2=0x7fece406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 -- 172.21.15.46:0/471612746 shutdown_connections 2024-05-09T03:17:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.487+0000 7fecec0a7640 1 -- 172.21.15.46:0/471612746 wait complete. 2024-05-09T03:17:48.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.489+0000 7fecec0a7640 1 Processor -- start 2024-05-09T03:17:48.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.489+0000 7fecec0a7640 1 -- start start 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecec0a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 0x7fece40cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecec0a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fece4163eb0 0x7fece40cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecec0a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece4155fa0 0x7fece41f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecec0a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fece40d0bc0 con 0x7fece40bb770 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecec0a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fece40bbb80 con 0x7fece4155fa0 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecec0a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fece40d0d40 con 0x7fece4163eb0 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 0x7fece40cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 0x7fece40cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59678/0 (socket says 172.21.15.46:59678) 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 -- 172.21.15.46:0/2918392566 learned_addr learned my addr 172.21.15.46:0/2918392566 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fecea644640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece4155fa0 0x7fece41f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9642640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fece4163eb0 0x7fece40cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 -- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fece4163eb0 msgr2=0x7fece40cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fece4163eb0 0x7fece40cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 -- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece4155fa0 msgr2=0x7fece41f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece4155fa0 0x7fece41f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.490+0000 7fece9e43640 1 -- 172.21.15.46:0/2918392566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece40ce020 con 0x7fece40bb770 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.491+0000 7fece9e43640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 0x7fece40cf0c0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fece00098c0 tx=0x7fece00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.491+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0041020 con 0x7fece40bb770 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.491+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece003d6e0 con 0x7fece40bb770 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.491+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece40ce300 con 0x7fece40bb770 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.491+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0037040 con 0x7fece40bb770 2024-05-09T03:17:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.491+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece40ce8c0 con 0x7fece40bb770 2024-05-09T03:17:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.492+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece406a270 con 0x7fece40bb770 2024-05-09T03:17:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.493+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fece0044020 con 0x7fece40bb770 2024-05-09T03:17:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.493+0000 7fecda7fc640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecc80762d0 0x7fecc80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.493+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7fece00b7fd0 con 0x7fece40bb770 2024-05-09T03:17:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.494+0000 7fece9642640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecc80762d0 0x7fecc80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.494+0000 7fece9642640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecc80762d0 0x7fecc80787e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fecd4005880 tx=0x7fecd4006aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.498+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fece004b070 con 0x7fece40bb770 2024-05-09T03:17:48.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:48.730+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fece4076490 con 0x7fece40bb770 2024-05-09T03:17:49.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.010+0000 7fecda7fc640 1 -- 172.21.15.46:0/2918392566 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v174) ==== 162+0+0 (secure 0 0 0) 0x7fece0080d60 con 0x7fece40bb770 2024-05-09T03:17:49.011 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.012+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecc80762d0 msgr2=0x7fecc80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.012+0000 7fecec0a7640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecc80762d0 0x7fecc80787e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fecd4005880 tx=0x7fecd4006aa0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.012+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 msgr2=0x7fece40cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.012+0000 7fecec0a7640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 0x7fece40cf0c0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fece00098c0 tx=0x7fece00046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.012+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 shutdown_connections 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fece4155fa0 0x7fece41f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fece4163eb0 0x7fece40cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecc80762d0 0x7fecc80787e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 --2- 172.21.15.46:0/2918392566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fece40bb770 0x7fece40cf0c0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 >> 172.21.15.46:0/2918392566 conn(0x7fece406e370 msgr2=0x7fece40d2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 shutdown_connections 2024-05-09T03:17:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.013+0000 7fecec0a7640 1 -- 172.21.15.46:0/2918392566 wait complete. 2024-05-09T03:17:49.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.140+0000 7f2080629640 1 Processor -- start 2024-05-09T03:17:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.141+0000 7f2080629640 1 -- start start 2024-05-09T03:17:49.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f2080629640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20780b92e0 0x7f20780b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:49.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f2080629640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20780d0480 0x7f20780d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:49.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f2080629640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20780d0dd0 con 0x7f20780ce6a0 2024-05-09T03:17:49.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f2080629640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f20780d2b80 con 0x7f20780d0480 2024-05-09T03:17:49.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f2080629640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f20780d2e60 con 0x7f20780b92e0 2024-05-09T03:17:49.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f207dbc4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f20780d0480 0x7f20780d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:49.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f207ebc6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f20780ce6a0 0x7f20780ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58060/0 (socket says 172.21.15.46:58060) 2024-05-09T03:17:49.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f207ebc6640 1 -- 172.21.15.46:0/2916583091 learned_addr learned my addr 172.21.15.46:0/2916583091 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:49.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.142+0000 7f207e3c5640 1 --2- 172.21.15.46:0/2916583091 >> v2:172.21.15.88:3300/0 conn(0x7f20780b92e0 0x7f20780b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:49.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.143+0000 7f207dbc4640 1 -- 172.21.15.46:0/2916583091 >> v2:172.21.15.88:3300/0 conn(0x7f20780b92e0 msgr2=0x7f20780b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:49.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.143+0000 7f207dbc4640 1 --2- 172.21.15.46:0/2916583091 >> v2:172.21.15.88:3300/0 conn(0x7f20780b92e0 0x7f20780b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.143+0000 7f207dbc4640 1 -- 172.21.15.46:0/2916583091 >> v1:172.21.15.46:6789/0 conn(0x7f20780ce6a0 legacy=0x7f20780ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.143+0000 7f207dbc4640 1 -- 172.21.15.46:0/2916583091 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20780d3140 con 0x7f20780d0480 2024-05-09T03:17:49.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.143+0000 7f207d3c3640 1 -- 172.21.15.46:0/2916583091 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219513396 0 0) 0x7f2070001490 con 0x7f20780ce6a0 2024-05-09T03:17:49.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.143+0000 7f207dbc4640 1 --2- 172.21.15.46:0/2916583091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780d0480 0x7f20780d0890 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f207806d120 tx=0x7f206802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9512d9e3bcf50c5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:49.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f207d3c3640 1 -- 172.21.15.46:0/2916583091 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068037070 con 0x7f20780d0480 2024-05-09T03:17:49.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f207d3c3640 1 -- 172.21.15.46:0/2916583091 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2068032440 con 0x7f20780d0480 2024-05-09T03:17:49.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f207d3c3640 1 -- 172.21.15.46:0/2916583091 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206802ae60 con 0x7f20780d0480 2024-05-09T03:17:49.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 -- 172.21.15.46:0/2916583091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780d0480 msgr2=0x7f20780d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 --2- 172.21.15.46:0/2916583091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780d0480 0x7f20780d0890 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f207806d120 tx=0x7f206802a110 comp rx=0 tx=0).stop 2024-05-09T03:17:49.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 -- 172.21.15.46:0/2916583091 shutdown_connections 2024-05-09T03:17:49.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 --2- 172.21.15.46:0/2916583091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780d0480 0x7f20780d0890 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 --2- 172.21.15.46:0/2916583091 >> v2:172.21.15.88:3300/0 conn(0x7f20780b92e0 0x7f20780b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 -- 172.21.15.46:0/2916583091 >> 172.21.15.46:0/2916583091 conn(0x7f20780d3c00 msgr2=0x7f20780d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 -- 172.21.15.46:0/2916583091 shutdown_connections 2024-05-09T03:17:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.144+0000 7f2080629640 1 -- 172.21.15.46:0/2916583091 wait complete. 2024-05-09T03:17:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.146+0000 7f2080629640 1 Processor -- start 2024-05-09T03:17:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.146+0000 7f2080629640 1 -- start start 2024-05-09T03:17:49.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.147+0000 7f2080629640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20780b92e0 0x7f20780cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:49.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f2080629640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 0x7f20781fa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:49.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f2080629640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20780d0480 0x7f20781f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:49.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f2080629640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f207806e920 con 0x7f20780ce6a0 2024-05-09T03:17:49.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f2080629640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f207806e7a0 con 0x7f20780d0480 2024-05-09T03:17:49.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f2080629640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f207806eaa0 con 0x7f20780b92e0 2024-05-09T03:17:49.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 0x7f20781fa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 0x7f20781fa300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59690/0 (socket says 172.21.15.46:59690) 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 -- 172.21.15.46:0/1920717245 learned_addr learned my addr 172.21.15.46:0/1920717245 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207e3c5640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20780b92e0 0x7f20780cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 -- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20780b92e0 msgr2=0x7f20780cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20780b92e0 0x7f20780cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207ebc6640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20780d0480 0x7f20781f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:49.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 -- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20780d0480 msgr2=0x7f20781f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20780d0480 0x7f20781f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 -- 172.21.15.46:0/1920717245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20780ccac0 con 0x7f20780ce6a0 2024-05-09T03:17:49.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207e3c5640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20780b92e0 0x7f20780cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:49.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207ebc6640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20780d0480 0x7f20781f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:49.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f207dbc4640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 0x7f20781fa300 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f2068009890 tx=0x7f20680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:49.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068037040 con 0x7f20780ce6a0 2024-05-09T03:17:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2068042070 con 0x7f20780ce6a0 2024-05-09T03:17:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20780ccda0 con 0x7f20780ce6a0 2024-05-09T03:17:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.148+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068030960 con 0x7f20780ce6a0 2024-05-09T03:17:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.149+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f207815b4d0 con 0x7f20780ce6a0 2024-05-09T03:17:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.149+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20780cc820 con 0x7f20780ce6a0 2024-05-09T03:17:49.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.150+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2068030b00 con 0x7f20780ce6a0 2024-05-09T03:17:49.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.151+0000 7f206effd640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f205c076300 0x7f205c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:49.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.151+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6971+0+0 (secure 0 0 0) 0x7f20680b82d0 con 0x7f20780ce6a0 2024-05-09T03:17:49.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.151+0000 7f207e3c5640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f205c076300 0x7f205c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:49.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.152+0000 7f207e3c5640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f205c076300 0x7f205c078810 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2078071390 tx=0x7f2074008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:49.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.156+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f20780cc820 con 0x7f20780ce6a0 2024-05-09T03:17:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.385+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f207806a270 con 0x7f20780ce6a0 2024-05-09T03:17:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.385+0000 7f206effd640 1 -- 172.21.15.46:0/1920717245 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+5177 (secure 0 0 0) 0x7f20680816b0 con 0x7f20780ce6a0 2024-05-09T03:17:49.386 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:49.386 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":175,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:49.013336+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8472":{"gid":8472,"name":"c","rank":0,"incarnation":175,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/3450505404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3450505404},{"type":"v1","addr":"172.21.15.46:6834","nonce":3450505404}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 175 2024-05-09T03:17:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f205c076300 msgr2=0x7f205c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f205c076300 0x7f205c078810 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2078071390 tx=0x7f2074008040 comp rx=0 tx=0).stop 2024-05-09T03:17:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 msgr2=0x7f20781fa300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 0x7f20781fa300 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f2068009890 tx=0x7f20680046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 shutdown_connections 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20780d0480 0x7f20781f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f205c076300 0x7f205c078810 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f20780ce6a0 0x7f20781fa300 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 --2- 172.21.15.46:0/1920717245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f20780b92e0 0x7f20780cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 >> 172.21.15.46:0/1920717245 conn(0x7f20780d3c00 msgr2=0x7f20780b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 shutdown_connections 2024-05-09T03:17:49.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:49.387+0000 7f2080629640 1 -- 172.21.15.46:0/1920717245 wait complete. 2024-05-09T03:17:49.399 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:17:52.032 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:52.031+0000 7fc296672640 -1 log_channel(cluster) log [ERR] : dir 0x1 object missing on disk; some files may be lost () 2024-05-09T03:17:52.032 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:52.031+0000 7fc296672640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-05-09T03:17:52.032 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:52.032+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:52.032 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:52.032+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:52.040 INFO:tasks.ceph.mds.c.smithi046.stderr: -19> 2024-05-09T03:17:52.031+0000 7fc296672640 -1 log_channel(cluster) log [ERR] : dir 0x1 object missing on disk; some files may be lost () 2024-05-09T03:17:52.040 INFO:tasks.ceph.mds.c.smithi046.stderr: -17> 2024-05-09T03:17:52.031+0000 7fc296672640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-05-09T03:17:52.103 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:54.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:54.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.530+0000 7fc8b44ee640 1 Processor -- start 2024-05-09T03:17:54.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.530+0000 7fc8b44ee640 1 -- start start 2024-05-09T03:17:54.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.531+0000 7fc8b44ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 0x7fc8ac1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.531+0000 7fc8b44ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 0x7fc8ac0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.531+0000 7fc8b44ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac0d5280 con 0x7fc8ac0d4800 2024-05-09T03:17:54.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.531+0000 7fc8b44ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc8ac0b5620 con 0x7fc8ac0b5210 2024-05-09T03:17:54.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.531+0000 7fc8b44ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc8ac1609a0 con 0x7fc8ac0b7ed0 2024-05-09T03:17:54.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b2a8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 0x7fc8ac0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b228a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 0x7fc8ac1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b228a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 0x7fc8ac1613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:59354/0 (socket says 172.21.15.46:59354) 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b2a8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 0x7fc8ac0d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:36248/0 (socket says 172.21.15.46:36248) 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b228a640 1 -- 172.21.15.46:0/1691468376 learned_addr learned my addr 172.21.15.46:0/1691468376 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b1a89640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc8ac0d4800 0x7fc8ac0b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:38696/0 (socket says 172.21.15.46:38696) 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b2a8b640 1 -- 172.21.15.46:0/1691468376 >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 msgr2=0x7fc8ac1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b2a8b640 1 --2- 172.21.15.46:0/1691468376 >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 0x7fc8ac1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b2a8b640 1 -- 172.21.15.46:0/1691468376 >> v1:172.21.15.88:6789/0 conn(0x7fc8ac0d4800 legacy=0x7fc8ac0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b2a8b640 1 -- 172.21.15.46:0/1691468376 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac0d5560 con 0x7fc8ac0b7ed0 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b1288640 1 -- 172.21.15.46:0/1691468376 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197726031 0 0) 0x7fc8ac0d5280 con 0x7fc8ac0d4800 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.532+0000 7fc8b228a640 1 --2- 172.21.15.46:0/1691468376 >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 0x7fc8ac1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:54.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.533+0000 7fc8b2a8b640 1 --2- 172.21.15.46:0/1691468376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 0x7fc8ac0d4c10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009d90 tx=0x7fc8a402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a0b840ecc093a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:54.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.533+0000 7fc8b1288640 1 -- 172.21.15.46:0/1691468376 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4031600 con 0x7fc8ac0b7ed0 2024-05-09T03:17:54.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.533+0000 7fc8b1288640 1 -- 172.21.15.46:0/1691468376 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a40317a0 con 0x7fc8ac0b7ed0 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.533+0000 7fc8b1288640 1 -- 172.21.15.46:0/1691468376 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4031b70 con 0x7fc8ac0b7ed0 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1691468376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 msgr2=0x7fc8ac0d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1691468376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 0x7fc8ac0d4c10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009d90 tx=0x7fc8a402a5d0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1691468376 shutdown_connections 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1691468376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0b7ed0 0x7fc8ac0d4c10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1691468376 >> v2:172.21.15.88:3300/0 conn(0x7fc8ac0b5210 0x7fc8ac1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1691468376 >> 172.21.15.46:0/1691468376 conn(0x7fc8ac0d29a0 msgr2=0x7fc8ac0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1691468376 shutdown_connections 2024-05-09T03:17:54.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.534+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1691468376 wait complete. 2024-05-09T03:17:54.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.536+0000 7fc8b44ee640 1 Processor -- start 2024-05-09T03:17:54.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.536+0000 7fc8b44ee640 1 -- start start 2024-05-09T03:17:54.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b44ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 0x7fc8ac15b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b44ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0d4800 0x7fc8ac15b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b44ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8ac1593a0 0x7fc8ac15be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b44ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac0d0ae0 con 0x7fc8ac0b5210 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b44ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac160ea0 con 0x7fc8ac1593a0 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b44ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc8ac163e60 con 0x7fc8ac0d4800 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b228a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 0x7fc8ac15b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b228a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 0x7fc8ac15b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59710/0 (socket says 172.21.15.46:59710) 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.537+0000 7fc8b228a640 1 -- 172.21.15.46:0/1447696875 learned_addr learned my addr 172.21.15.46:0/1447696875 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b2a8b640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8ac1593a0 0x7fc8ac15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b1a89640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0d4800 0x7fc8ac15b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b228a640 1 -- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0d4800 msgr2=0x7fc8ac15b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b228a640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0d4800 0x7fc8ac15b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b228a640 1 -- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8ac1593a0 msgr2=0x7fc8ac15be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b228a640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8ac1593a0 0x7fc8ac15be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b228a640 1 -- 172.21.15.46:0/1447696875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac0ca500 con 0x7fc8ac0b5210 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b228a640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 0x7fc8ac15b410 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800e9e0 tx=0x7fc8a800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8017070 con 0x7fc8ac0b5210 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a800c9d0 con 0x7fc8ac0b5210 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ac0ca7e0 con 0x7fc8ac0b5210 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a800cd80 con 0x7fc8ac0b5210 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.538+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ac0cada0 con 0x7fc8ac0b5210 2024-05-09T03:17:54.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.539+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ac06a270 con 0x7fc8ac0b5210 2024-05-09T03:17:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.540+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc8a801a070 con 0x7fc8ac0b5210 2024-05-09T03:17:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.541+0000 7fc8a2ffd640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8900762d0 0x7fc8900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.541+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7fc8a8097d50 con 0x7fc8ac0b5210 2024-05-09T03:17:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.541+0000 7fc8b1a89640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8900762d0 0x7fc8900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.541+0000 7fc8b1a89640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8900762d0 0x7fc8900787e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac15c6c0 tx=0x7fc89c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.545+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc8a809b030 con 0x7fc8ac0b5210 2024-05-09T03:17:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.776+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8ac155760 con 0x7fc8ac0b5210 2024-05-09T03:17:54.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.777+0000 7fc8a2ffd640 1 -- 172.21.15.46:0/1447696875 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5168 (secure 0 0 0) 0x7fc8a8060a80 con 0x7fc8ac0b5210 2024-05-09T03:17:54.778 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:54.778 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":180,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1234333269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1234333269},{"type":"v1","addr":"172.21.15.46:6834","nonce":1234333269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:52.032968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:54.778 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 180 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8900762d0 msgr2=0x7fc8900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8900762d0 0x7fc8900787e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac15c6c0 tx=0x7fc89c007de0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 msgr2=0x7fc8ac15b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 0x7fc8ac15b410 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800e9e0 tx=0x7fc8a800eea0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 shutdown_connections 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8ac1593a0 0x7fc8ac15be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8ac0d4800 0x7fc8ac15b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc8900762d0 0x7fc8900787e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 --2- 172.21.15.46:0/1447696875 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8ac0b5210 0x7fc8ac15b410 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 >> 172.21.15.46:0/1447696875 conn(0x7fc8ac0d29a0 msgr2=0x7fc8ac0b91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 shutdown_connections 2024-05-09T03:17:54.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.779+0000 7fc8b44ee640 1 -- 172.21.15.46:0/1447696875 wait complete. 2024-05-09T03:17:54.792 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:17:54.792 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:54.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.906+0000 7fe653f76640 1 Processor -- start 2024-05-09T03:17:54.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.906+0000 7fe653f76640 1 -- start start 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.907+0000 7fe653f76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe64c0af670 0x7fe64c0d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.907+0000 7fe653f76640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe64c073500 0x7fe64c073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.907+0000 7fe653f76640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64c15e6e0 con 0x7fe64c06f560 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.907+0000 7fe653f76640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe64c0afa80 con 0x7fe64c073500 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.907+0000 7fe653f76640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe64c073e50 con 0x7fe64c0af670 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe64c073500 0x7fe64c073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe64c073500 0x7fe64c073910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59722/0 (socket says 172.21.15.46:59722) 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 -- 172.21.15.46:0/1258112929 learned_addr learned my addr 172.21.15.46:0/1258112929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651d12640 1 --2- 172.21.15.46:0/1258112929 >> v2:172.21.15.88:3300/0 conn(0x7fe64c0af670 0x7fe64c0d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 -- 172.21.15.46:0/1258112929 >> v2:172.21.15.88:3300/0 conn(0x7fe64c0af670 msgr2=0x7fe64c0d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 --2- 172.21.15.46:0/1258112929 >> v2:172.21.15.88:3300/0 conn(0x7fe64c0af670 0x7fe64c0d1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 -- 172.21.15.46:0/1258112929 >> v1:172.21.15.88:6789/0 conn(0x7fe64c06f560 legacy=0x7fe64c06f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651511640 1 -- 172.21.15.46:0/1258112929 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c15e960 con 0x7fe64c073500 2024-05-09T03:17:54.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.908+0000 7fe651d12640 1 --2- 172.21.15.46:0/1258112929 >> v2:172.21.15.88:3300/0 conn(0x7fe64c0af670 0x7fe64c0d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.909+0000 7fe651511640 1 --2- 172.21.15.46:0/1258112929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c073500 0x7fe64c073910 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0729f0 tx=0x7fe63c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c586a6347ffc43c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.909+0000 7fe650d10640 1 -- 172.21.15.46:0/1258112929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c037070 con 0x7fe64c073500 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.909+0000 7fe650d10640 1 -- 172.21.15.46:0/1258112929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c032440 con 0x7fe64c073500 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.909+0000 7fe650d10640 1 -- 172.21.15.46:0/1258112929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c032810 con 0x7fe64c073500 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 -- 172.21.15.46:0/1258112929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c073500 msgr2=0x7fe64c073910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 --2- 172.21.15.46:0/1258112929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c073500 0x7fe64c073910 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0729f0 tx=0x7fe63c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 -- 172.21.15.46:0/1258112929 shutdown_connections 2024-05-09T03:17:54.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 --2- 172.21.15.46:0/1258112929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c073500 0x7fe64c073910 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 --2- 172.21.15.46:0/1258112929 >> v2:172.21.15.88:3300/0 conn(0x7fe64c0af670 0x7fe64c0d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 -- 172.21.15.46:0/1258112929 >> 172.21.15.46:0/1258112929 conn(0x7fe64c0b3910 msgr2=0x7fe64c0b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 -- 172.21.15.46:0/1258112929 shutdown_connections 2024-05-09T03:17:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.910+0000 7fe653f76640 1 -- 172.21.15.46:0/1258112929 wait complete. 2024-05-09T03:17:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.912+0000 7fe653f76640 1 Processor -- start 2024-05-09T03:17:54.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.912+0000 7fe653f76640 1 -- start start 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe653f76640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe64c06f560 0x7fe64c1e7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe653f76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe64c073500 0x7fe64c14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe653f76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 0x7fe64c1617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe653f76640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c15f200 con 0x7fe64c0af670 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe653f76640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe64c15ef20 con 0x7fe64c073500 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe653f76640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe64c0aeb80 con 0x7fe64c06f560 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 0x7fe64c1617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 0x7fe64c1617f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59726/0 (socket says 172.21.15.46:59726) 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 -- 172.21.15.46:0/1735093770 learned_addr learned my addr 172.21.15.46:0/1735093770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:54.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe651d12640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe64c06f560 0x7fe64c1e7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe651511640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe64c073500 0x7fe64c14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 -- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe64c06f560 msgr2=0x7fe64c1e7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe64c06f560 0x7fe64c1e7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 -- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe64c073500 msgr2=0x7fe64c14d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe64c073500 0x7fe64c14d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.913+0000 7fe652513640 1 -- 172.21.15.46:0/1735093770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64c0d05b0 con 0x7fe64c0af670 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.914+0000 7fe652513640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 0x7fe64c1617f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b970 tx=0x7fe64400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.914+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe644019070 con 0x7fe64c0af670 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.914+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64400cb40 con 0x7fe64c0af670 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.914+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64c0d0890 con 0x7fe64c0af670 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.914+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe644014440 con 0x7fe64c0af670 2024-05-09T03:17:54.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.914+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64c0d0e50 con 0x7fe64c0af670 2024-05-09T03:17:54.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.915+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe64c06a270 con 0x7fe64c0af670 2024-05-09T03:17:54.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.916+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe64401c070 con 0x7fe64c0af670 2024-05-09T03:17:54.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.916+0000 7fe6427fc640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6300762d0 0x7fe6300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:54.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.916+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7fe644097880 con 0x7fe64c0af670 2024-05-09T03:17:54.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.917+0000 7fe651d12640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6300762d0 0x7fe6300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:54.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.917+0000 7fe651d12640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6300762d0 0x7fe6300787e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe64c09fed0 tx=0x7fe648008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:54.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:54.921+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe6440605b0 con 0x7fe64c0af670 2024-05-09T03:17:55.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.150+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe64c0b2ac0 con 0x7fe64c0af670 2024-05-09T03:17:55.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.151+0000 7fe6427fc640 1 -- 172.21.15.46:0/1735093770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5168 (secure 0 0 0) 0x7fe6440648b0 con 0x7fe64c0af670 2024-05-09T03:17:55.152 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:55.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":180,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1234333269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1234333269},{"type":"v1","addr":"172.21.15.46:6834","nonce":1234333269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:52.032968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:55.152 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 180 2024-05-09T03:17:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6300762d0 msgr2=0x7fe6300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6300762d0 0x7fe6300787e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe64c09fed0 tx=0x7fe648008040 comp rx=0 tx=0).stop 2024-05-09T03:17:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 msgr2=0x7fe64c1617f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 0x7fe64c1617f0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fe64400b970 tx=0x7fe64400be30 comp rx=0 tx=0).stop 2024-05-09T03:17:55.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 shutdown_connections 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe6300762d0 0x7fe6300787e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe64c0af670 0x7fe64c1617f0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe64c073500 0x7fe64c14d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 --2- 172.21.15.46:0/1735093770 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe64c06f560 0x7fe64c1e7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 >> 172.21.15.46:0/1735093770 conn(0x7fe64c0b3910 msgr2=0x7fe64c0b5860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 shutdown_connections 2024-05-09T03:17:55.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.153+0000 7fe653f76640 1 -- 172.21.15.46:0/1735093770 wait complete. 2024-05-09T03:17:55.165 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:55.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:55.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.280+0000 7f8b76249640 1 Processor -- start 2024-05-09T03:17:55.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.280+0000 7f8b76249640 1 -- start start 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b76249640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8b700d0b90 0x7f8b700b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b76249640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8b700cf6b0 0x7f8b7006d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b76249640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b700d2b80 con 0x7f8b700b92e0 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b76249640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8b700d2e60 con 0x7f8b700cf6b0 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b76249640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8b700d3140 con 0x7f8b700d0b90 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8b700cf6b0 0x7f8b7006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8b700cf6b0 0x7f8b7006d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59742/0 (socket says 172.21.15.46:59742) 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 -- 172.21.15.46:0/2034966883 learned_addr learned my addr 172.21.15.46:0/2034966883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6effd640 1 --2- 172.21.15.46:0/2034966883 >> v2:172.21.15.88:3300/0 conn(0x7f8b700d0b90 0x7f8b700b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 -- 172.21.15.46:0/2034966883 >> v2:172.21.15.88:3300/0 conn(0x7f8b700d0b90 msgr2=0x7f8b700b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:55.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 --2- 172.21.15.46:0/2034966883 >> v2:172.21.15.88:3300/0 conn(0x7f8b700d0b90 0x7f8b700b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 -- 172.21.15.46:0/2034966883 >> v1:172.21.15.88:6789/0 conn(0x7f8b700b92e0 legacy=0x7f8b700d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:55.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.282+0000 7f8b6ffff640 1 -- 172.21.15.46:0/2034966883 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b700d3420 con 0x7f8b700cf6b0 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.283+0000 7f8b6ffff640 1 --2- 172.21.15.46:0/2034966883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700cf6b0 0x7f8b7006d790 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8b7006d150 tx=0x7f8b6802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afb62230fe503c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.283+0000 7f8b6e7fc640 1 -- 172.21.15.46:0/2034966883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68036070 con 0x7f8b700cf6b0 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.283+0000 7f8b6e7fc640 1 -- 172.21.15.46:0/2034966883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b68031600 con 0x7f8b700cf6b0 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.283+0000 7f8b6e7fc640 1 -- 172.21.15.46:0/2034966883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b680319d0 con 0x7f8b700cf6b0 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 -- 172.21.15.46:0/2034966883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700cf6b0 msgr2=0x7f8b7006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 --2- 172.21.15.46:0/2034966883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700cf6b0 0x7f8b7006d790 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f8b7006d150 tx=0x7f8b6802a610 comp rx=0 tx=0).stop 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 -- 172.21.15.46:0/2034966883 shutdown_connections 2024-05-09T03:17:55.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 --2- 172.21.15.46:0/2034966883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700cf6b0 0x7f8b7006d790 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 --2- 172.21.15.46:0/2034966883 >> v2:172.21.15.88:3300/0 conn(0x7f8b700d0b90 0x7f8b700b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 -- 172.21.15.46:0/2034966883 >> 172.21.15.46:0/2034966883 conn(0x7f8b700d3c00 msgr2=0x7f8b700d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:55.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 -- 172.21.15.46:0/2034966883 shutdown_connections 2024-05-09T03:17:55.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.284+0000 7f8b76249640 1 -- 172.21.15.46:0/2034966883 wait complete. 2024-05-09T03:17:55.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.286+0000 7f8b76249640 1 Processor -- start 2024-05-09T03:17:55.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.286+0000 7f8b76249640 1 -- start start 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b76249640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 0x7f8b701591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b76249640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b700cf6b0 0x7f8b7015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b76249640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b700cac90 0x7f8b700cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b76249640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7006e060 con 0x7f8b700b92e0 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b76249640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7006dee0 con 0x7f8b700cac90 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b76249640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8b7006e1e0 con 0x7f8b700cf6b0 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 0x7f8b701591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b6f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 0x7f8b701591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59756/0 (socket says 172.21.15.46:59756) 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b6f7fe640 1 -- 172.21.15.46:0/2091174353 learned_addr learned my addr 172.21.15.46:0/2091174353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b6effd640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b700cf6b0 0x7f8b7015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.287+0000 7f8b6ffff640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b700cac90 0x7f8b700cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b6f7fe640 1 -- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b700cf6b0 msgr2=0x7f8b7015bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b6f7fe640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b700cf6b0 0x7f8b7015bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b6f7fe640 1 -- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b700cac90 msgr2=0x7f8b700cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b6f7fe640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b700cac90 0x7f8b700cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b6f7fe640 1 -- 172.21.15.46:0/2091174353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b700cd760 con 0x7f8b700b92e0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b6f7fe640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 0x7f8b701591f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000b3c0 tx=0x7f8b6000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60015020 con 0x7f8b700b92e0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b60011830 con 0x7f8b700b92e0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7015b090 con 0x7f8b700b92e0 2024-05-09T03:17:55.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60011c60 con 0x7f8b700b92e0 2024-05-09T03:17:55.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.288+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7015b650 con 0x7f8b700b92e0 2024-05-09T03:17:55.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.289+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7006a270 con 0x7f8b700b92e0 2024-05-09T03:17:55.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.290+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8b60018020 con 0x7f8b700b92e0 2024-05-09T03:17:55.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.290+0000 7f8b57fff640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b5c076300 0x7f8b5c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.291+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7f8b60097570 con 0x7f8b700b92e0 2024-05-09T03:17:55.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.291+0000 7f8b6effd640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b5c076300 0x7f8b5c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.291+0000 7f8b6effd640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b5c076300 0x7f8b5c078810 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8b700d0cd0 tx=0x7f8b64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:55.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.295+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8b6001f030 con 0x7f8b700b92e0 2024-05-09T03:17:55.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.526+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b70159780 con 0x7f8b700b92e0 2024-05-09T03:17:55.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.527+0000 7f8b57fff640 1 -- 172.21.15.46:0/2091174353 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5168 (secure 0 0 0) 0x7f8b600603c0 con 0x7f8b700b92e0 2024-05-09T03:17:55.528 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:55.528 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":180,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1234333269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1234333269},{"type":"v1","addr":"172.21.15.46:6834","nonce":1234333269}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:52.032968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:55.528 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 180 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b5c076300 msgr2=0x7f8b5c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b5c076300 0x7f8b5c078810 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8b700d0cd0 tx=0x7f8b64008040 comp rx=0 tx=0).stop 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 msgr2=0x7f8b701591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 0x7f8b701591f0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000b3c0 tx=0x7f8b6000b880 comp rx=0 tx=0).stop 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 shutdown_connections 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b700cac90 0x7f8b700cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8b700cf6b0 0x7f8b7015bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8b5c076300 0x7f8b5c078810 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 --2- 172.21.15.46:0/2091174353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b700b92e0 0x7f8b701591f0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 >> 172.21.15.46:0/2091174353 conn(0x7f8b700d3c00 msgr2=0x7f8b700b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:55.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 shutdown_connections 2024-05-09T03:17:55.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.529+0000 7f8b76249640 1 -- 172.21.15.46:0/2091174353 wait complete. 2024-05-09T03:17:55.542 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000' led to DAMAGED state 2024-05-09T03:17:55.542 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 1.00000000.inode' 2024-05-09T03:17:55.542 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:55.542 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:55.558 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:55.559 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:55.559 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:55.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:55.559 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:55.615 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:55.615 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:55.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.770+0000 7f07fbc9f640 1 Processor -- start 2024-05-09T03:17:55.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.770+0000 7f07fbc9f640 1 -- start start 2024-05-09T03:17:55.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.771+0000 7f07fbc9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f07f4163eb0 0x7f07f41642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.771+0000 7f07fbc9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40d2a70 0x7f07f40d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.771+0000 7f07fbc9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07f40d33c0 con 0x7f07f40bb770 2024-05-09T03:17:55.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.771+0000 7f07fbc9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f07f40d4bb0 con 0x7f07f4163eb0 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.771+0000 7f07fbc9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f07f4164900 con 0x7f07f40d2a70 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f923a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f07f4163eb0 0x7f07f41642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f9a3b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f07f40bb770 0x7f07f40d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49376/0 (socket says 172.21.15.46:49376) 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f9a3b640 1 -- 172.21.15.46:0/2075081364 learned_addr learned my addr 172.21.15.46:0/2075081364 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07fa23c640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40d2a70 0x7f07f40d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f923a640 1 -- 172.21.15.46:0/2075081364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40d2a70 msgr2=0x7f07f40d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f923a640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40d2a70 0x7f07f40d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f923a640 1 -- 172.21.15.46:0/2075081364 >> v1:172.21.15.46:6789/0 conn(0x7f07f40bb770 legacy=0x7f07f40d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f8a39640 1 -- 172.21.15.46:0/2075081364 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821462786 0 0) 0x7f07f0001110 con 0x7f07f40bb770 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07f923a640 1 -- 172.21.15.46:0/2075081364 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f4164be0 con 0x7f07f4163eb0 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.772+0000 7f07fa23c640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40d2a70 0x7f07f40d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.773+0000 7f07f923a640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f41642c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f07e40097a0 tx=0x7f07e402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecd06d3b1c38b10 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.773+0000 7f07f8a39640 1 -- 172.21.15.46:0/2075081364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4037070 con 0x7f07f4163eb0 2024-05-09T03:17:55.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.773+0000 7f07f8a39640 1 -- 172.21.15.46:0/2075081364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07e4032440 con 0x7f07f4163eb0 2024-05-09T03:17:55.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.773+0000 7f07f8a39640 1 -- 172.21.15.46:0/2075081364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e4032810 con 0x7f07f4163eb0 2024-05-09T03:17:55.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 -- 172.21.15.46:0/2075081364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 msgr2=0x7f07f41642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f41642c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f07e40097a0 tx=0x7f07e402a180 comp rx=0 tx=0).stop 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 -- 172.21.15.46:0/2075081364 shutdown_connections 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40d2a70 0x7f07f40d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/2075081364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f41642c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 -- 172.21.15.46:0/2075081364 >> 172.21.15.46:0/2075081364 conn(0x7f07f406e370 msgr2=0x7f07f406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 -- 172.21.15.46:0/2075081364 shutdown_connections 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.774+0000 7f07fbc9f640 1 -- 172.21.15.46:0/2075081364 wait complete. 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.775+0000 7f07fbc9f640 1 Processor -- start 2024-05-09T03:17:55.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.776+0000 7f07fbc9f640 1 -- start start 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fbc9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40bb770 0x7f07f414eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fbc9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f40d2a70 0x7f07f40caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fbc9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f40cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fbc9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f07f40d0bc0 con 0x7f07f4163eb0 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fbc9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f07f40bbb80 con 0x7f07f40d2a70 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fbc9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f07f40d0d40 con 0x7f07f40bb770 2024-05-09T03:17:55.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f40cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f40cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36330/0 (socket says 172.21.15.46:36330) 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 -- 172.21.15.46:0/3958491671 learned_addr learned my addr 172.21.15.46:0/3958491671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07f923a640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f40d2a70 0x7f07f40caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07f9a3b640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40bb770 0x7f07f414eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 -- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40bb770 msgr2=0x7f07f414eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40bb770 0x7f07f414eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 -- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f40d2a70 msgr2=0x7f07f40caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f40d2a70 0x7f07f40caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.777+0000 7f07fa23c640 1 -- 172.21.15.46:0/3958491671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f40cfe00 con 0x7f07f4163eb0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.778+0000 7f07fa23c640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f40cafc0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f07ec007df0 tx=0x7f07ec00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.778+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec01a070 con 0x7f07f4163eb0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.778+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec003990 con 0x7f07f4163eb0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.778+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec003d80 con 0x7f07f4163eb0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.778+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f40cd2a0 con 0x7f07f4163eb0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.778+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f4150a30 con 0x7f07f4163eb0 2024-05-09T03:17:55.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.779+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f4150c90 con 0x7f07f4163eb0 2024-05-09T03:17:55.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.779+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f07ec01d070 con 0x7f07f4163eb0 2024-05-09T03:17:55.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.780+0000 7f07ea7fc640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f07d80762d0 0x7f07d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:55.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.780+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7f07ec097d50 con 0x7f07f4163eb0 2024-05-09T03:17:55.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.780+0000 7f07f9a3b640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f07d80762d0 0x7f07d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:55.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.781+0000 7f07f9a3b640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f07d80762d0 0x7f07d80787e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f07f0007fd0 tx=0x7f07f0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:55.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:55.785+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f07ec09b030 con 0x7f07f4163eb0 2024-05-09T03:17:56.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.012+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f07f406a270 con 0x7f07f4163eb0 2024-05-09T03:17:56.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.047+0000 7f07ea7fc640 1 -- 172.21.15.46:0/3958491671 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v181) ==== 161+0+0 (secure 0 0 0) 0x7f07ec010070 con 0x7f07f4163eb0 2024-05-09T03:17:56.048 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f07d80762d0 msgr2=0x7f07d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f07d80762d0 0x7f07d80787e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f07f0007fd0 tx=0x7f07f0007de0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 msgr2=0x7f07f40cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f40cafc0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f07ec007df0 tx=0x7f07ec00d4a0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 shutdown_connections 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f07d80762d0 0x7f07d80787e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07f4163eb0 0x7f07f40cafc0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07f40d2a70 0x7f07f40caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 --2- 172.21.15.46:0/3958491671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f07f40bb770 0x7f07f414eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.049+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 >> 172.21.15.46:0/3958491671 conn(0x7f07f406e370 msgr2=0x7f07f40ba0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.050+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 shutdown_connections 2024-05-09T03:17:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.050+0000 7f07fbc9f640 1 -- 172.21.15.46:0/3958491671 wait complete. 2024-05-09T03:17:56.062 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:17:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.178+0000 7f3e75e8b640 1 Processor -- start 2024-05-09T03:17:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.178+0000 7f3e75e8b640 1 -- start start 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.179+0000 7f3e75e8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e75e8b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e700d3180 0x7f3e701642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e75e8b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e70164900 con 0x7f3e70163eb0 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e75e8b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3e700d4bb0 con 0x7f3e700d3180 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e75e8b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3e70164be0 con 0x7f3e700bb770 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e700d3180 0x7f3e701642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e700d3180 0x7f3e701642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36332/0 (socket says 172.21.15.46:36332) 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 -- 172.21.15.46:0/2323150819 learned_addr learned my addr 172.21.15.46:0/2323150819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6f7fe640 1 --2- 172.21.15.46:0/2323150819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 -- 172.21.15.46:0/2323150819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 msgr2=0x7f3e700d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 --2- 172.21.15.46:0/2323150819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 -- 172.21.15.46:0/2323150819 >> v1:172.21.15.88:6789/0 conn(0x7f3e70163eb0 legacy=0x7f3e700d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.180+0000 7f3e6ffff640 1 -- 172.21.15.46:0/2323150819 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e700d4840 con 0x7f3e700d3180 2024-05-09T03:17:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.181+0000 7f3e6ffff640 1 --2- 172.21.15.46:0/2323150819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e700d3180 0x7f3e701642c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f3e68009870 tx=0x7f3e6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28e3b9d25a1fad9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.181+0000 7f3e6e7fc640 1 -- 172.21.15.46:0/2323150819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68037070 con 0x7f3e700d3180 2024-05-09T03:17:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.181+0000 7f3e6e7fc640 1 -- 172.21.15.46:0/2323150819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e68032440 con 0x7f3e700d3180 2024-05-09T03:17:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.181+0000 7f3e6e7fc640 1 -- 172.21.15.46:0/2323150819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68032810 con 0x7f3e700d3180 2024-05-09T03:17:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 -- 172.21.15.46:0/2323150819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e700d3180 msgr2=0x7f3e701642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/2323150819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e700d3180 0x7f3e701642c0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f3e68009870 tx=0x7f3e6802a270 comp rx=0 tx=0).stop 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 -- 172.21.15.46:0/2323150819 shutdown_connections 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/2323150819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e700d3180 0x7f3e701642c0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/2323150819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 -- 172.21.15.46:0/2323150819 >> 172.21.15.46:0/2323150819 conn(0x7f3e7006e370 msgr2=0x7f3e7006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 -- 172.21.15.46:0/2323150819 shutdown_connections 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.182+0000 7f3e75e8b640 1 -- 172.21.15.46:0/2323150819 wait complete. 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.183+0000 7f3e75e8b640 1 Processor -- start 2024-05-09T03:17:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.184+0000 7f3e75e8b640 1 -- start start 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e75e8b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e75e8b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e70163eb0 0x7f3e70161e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e75e8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 0x7f3e70162380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e75e8b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700d0bc0 con 0x7f3e70151380 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e75e8b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700bbb80 con 0x7f3e70163eb0 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e75e8b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3e700d0d40 con 0x7f3e700bb770 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 0x7f3e70162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 0x7f3e70162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36340/0 (socket says 172.21.15.46:36340) 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6f7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 -- 172.21.15.46:0/3407927011 learned_addr learned my addr 172.21.15.46:0/3407927011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6effd640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e70163eb0 0x7f3e70161e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 -- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 msgr2=0x7f3e700c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 -- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e70163eb0 msgr2=0x7f3e70161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e70163eb0 0x7f3e70161e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.185+0000 7f3e6ffff640 1 -- 172.21.15.46:0/3407927011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e700ca690 con 0x7f3e70151380 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.186+0000 7f3e6ffff640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 0x7f3e70162380 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f3e68009870 tx=0x7f3e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.186+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68041020 con 0x7f3e70151380 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.186+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6803d640 con 0x7f3e70151380 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.186+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e700ca970 con 0x7f3e70151380 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.186+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68037040 con 0x7f3e70151380 2024-05-09T03:17:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.186+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e700caf30 con 0x7f3e70151380 2024-05-09T03:17:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.187+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7006a270 con 0x7f3e70151380 2024-05-09T03:17:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.188+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e68044020 con 0x7f3e70151380 2024-05-09T03:17:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.188+0000 7f3e57fff640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.188+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7f3e680b8220 con 0x7f3e70151380 2024-05-09T03:17:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.189+0000 7f3e6f7fe640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.189+0000 7f3e6f7fe640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3e700b8880 tx=0x7f3e6000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:56.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.193+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3e6804b030 con 0x7f3e70151380 2024-05-09T03:17:56.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:56.456+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f3e7006e820 con 0x7f3e70151380 2024-05-09T03:17:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.057+0000 7f3e57fff640 1 -- 172.21.15.46:0/3407927011 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v182) ==== 101+0+0 (secure 0 0 0) 0x7f3e68080f60 con 0x7f3e70151380 2024-05-09T03:17:57.058 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3e5c0762d0 msgr2=0x7f3e5c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3e700b8880 tx=0x7f3e6000b040 comp rx=0 tx=0).stop 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 msgr2=0x7f3e70162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 0x7f3e70162380 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f3e68009870 tx=0x7f3e680046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 shutdown_connections 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e70151380 0x7f3e70162380 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e70163eb0 0x7f3e70161e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 --2- 172.21.15.46:0/3407927011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3e700bb770 0x7f3e700c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 >> 172.21.15.46:0/3407927011 conn(0x7f3e7006e370 msgr2=0x7f3e700b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 shutdown_connections 2024-05-09T03:17:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.061+0000 7f3e75e8b640 1 -- 172.21.15.46:0/3407927011 wait complete. 2024-05-09T03:17:57.076 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:17:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.135+0000 7fa692fb5f40 1 Processor -- start 2024-05-09T03:17:57.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.135+0000 7fa692fb5f40 1 -- start start 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa692fb5f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f8357e0 0x56549f9083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa692fb5f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f90c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa692fb5f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f90f810 0x56549f90ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa692fb5f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56549f7def50 con 0x56549f8677b0 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa692fb5f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56549f7e2f90 con 0x56549f90f810 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa692fb5f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56549f87c4d0 con 0x56549f8357e0 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f90c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f90c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36362/0 (socket says 172.21.15.46:36362) 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 -- 172.21.15.46:0/704398444 learned_addr learned my addr 172.21.15.46:0/704398444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f8357e0 0x56549f9083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68d27e640 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f90f810 0x56549f90ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 -- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f8357e0 msgr2=0x56549f9083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f8357e0 0x56549f9083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 -- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f90f810 msgr2=0x56549f90ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f90f810 0x56549f90ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 -- 172.21.15.46:0/704398444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549f8668d0 con 0x56549f8677b0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68d27e640 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f90f810 0x56549f90ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.137+0000 7fa68c27c640 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f90c900 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fa680025d60 tx=0x7fa680051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3097ca83b9e532 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa68b27a640 1 -- 172.21.15.46:0/704398444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68006b070 con 0x56549f8677b0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa68b27a640 1 -- 172.21.15.46:0/704398444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa680052a60 con 0x56549f8677b0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa68b27a640 1 -- 172.21.15.46:0/704398444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680052e50 con 0x56549f8677b0 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 -- 172.21.15.46:0/704398444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 msgr2=0x56549f90c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f90c900 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fa680025d60 tx=0x7fa680051000 comp rx=0 tx=0).stop 2024-05-09T03:17:57.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 -- 172.21.15.46:0/704398444 shutdown_connections 2024-05-09T03:17:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f90f810 0x56549f90ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f90c900 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/704398444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f8357e0 0x56549f9083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 -- 172.21.15.46:0/704398444 >> 172.21.15.46:0/704398444 conn(0x56549f75aac0 msgr2=0x56549f87fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 -- 172.21.15.46:0/704398444 shutdown_connections 2024-05-09T03:17:57.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.138+0000 7fa692fb5f40 1 -- 172.21.15.46:0/704398444 wait complete. 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.139+0000 7fa692fb5f40 1 Processor -- start 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.139+0000 7fa692fb5f40 1 -- start start 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.140+0000 7fa692fb5f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f8357e0 0x56549f87fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.140+0000 7fa692fb5f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f905900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.140+0000 7fa692fb5f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f90f810 0x56549f905e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.140+0000 7fa692fb5f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56549f90d380 con 0x56549f8677b0 2024-05-09T03:17:57.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.140+0000 7fa692fb5f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56549f8665f0 con 0x56549f8357e0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.140+0000 7fa692fb5f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56549f99fa90 con 0x56549f90f810 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f905900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f905900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36374/0 (socket says 172.21.15.46:36374) 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 learned_addr learned my addr 172.21.15.46:0/2025211982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f8357e0 0x56549f87fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68d27e640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f90f810 0x56549f905e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f90f810 msgr2=0x56549f905e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56549f90f810 0x56549f905e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f8357e0 msgr2=0x56549f87fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56549f8357e0 0x56549f87fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56549f9066e0 con 0x56549f8677b0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f905900 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fa68000a4f0 tx=0x7fa680002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68a278640 1 -- 172.21.15.46:0/2025211982 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa680075070 con 0x56549f8677b0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68a278640 1 -- 172.21.15.46:0/2025211982 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa680052cd0 con 0x56549f8677b0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56549f9046b0 con 0x56549f8677b0 2024-05-09T03:17:57.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.141+0000 7fa68a278640 1 -- 172.21.15.46:0/2025211982 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68006b070 con 0x56549f8677b0 2024-05-09T03:17:57.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.142+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56549f8798e0 con 0x56549f8677b0 2024-05-09T03:17:57.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.143+0000 7fa68a278640 1 -- 172.21.15.46:0/2025211982 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa68006f070 con 0x56549f8677b0 2024-05-09T03:17:57.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa68a278640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa678074cb0 0x7fa6780771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa68a278640 1 -- 172.21.15.46:0/2025211982 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7fa6800db840 con 0x56549f8677b0 2024-05-09T03:17:57.145 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:17:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa678074cb0 0x7fa6780771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56549f880180 0x56549f9b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f905020 con 0x56549f880180 2024-05-09T03:17:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa68d27e640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56549f880180 0x56549f9b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.144+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa678074cb0 0x7fa6780771c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa67c001ed0 tx=0x7fa67c001aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.145+0000 7fa68d27e640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56549f880180 0x56549f9b17a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.146+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v99'443 uv417 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.146+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f99e200 con 0x56549f880180 2024-05-09T03:17:57.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.148+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v99'444 uv444 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.148+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b1ce0 con 0x56549f880180 2024-05-09T03:17:57.150 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:17:57.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.150+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v99'445 uv445 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.150+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b3090 con 0x56549f880180 2024-05-09T03:17:57.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.151+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v99'446 uv446 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.151+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b3430 con 0x56549f880180 2024-05-09T03:17:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.153+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v99'447 uv447 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.153+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b2080 con 0x56549f880180 2024-05-09T03:17:57.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.154+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v99'448 uv448 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.154+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b2450 con 0x56549f880180 2024-05-09T03:17:57.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.155+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v99'449 uv423 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.155+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.157+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v99'450 uv450 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.157+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.159 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:17:57.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.158+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v99'451 uv451 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.158+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.160+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v99'452 uv452 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.160+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.161+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v99'453 uv453 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.161+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.163+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v99'454 uv454 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.163+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.164+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v99'455 uv430 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.164+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.166+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v99'456 uv456 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.166+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.168 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:17:57.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.167+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v99'457 uv457 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.167+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.169+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v99'458 uv458 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.169+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.170+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v99'459 uv459 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.170+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.172+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v99'460 uv460 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.172+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.173+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v99'461 uv461 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.173+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.175+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v99'462 uv436 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.175+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.176+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v99'463 uv463 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.176+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.178 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:17:57.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.178+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v99'464 uv464 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.178+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.179+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v99'465 uv465 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.179+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.181+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v99'466 uv466 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.181+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.182+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v99'467 uv467 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.183+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.184+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v99'468 uv442 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.184+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.185+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v99'469 uv469 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.185+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.186+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v99'470 uv470 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.187 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:17:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.186+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.188+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v99'471 uv471 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.188+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.189+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v99'472 uv472 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.189+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f880180 2024-05-09T03:17:57.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.190+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v99'473 uv473 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.190+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56549f9b2bd0 0x56549f9b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.190+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b27f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.191+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56549f9b2bd0 0x56549f9b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.191+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56549f9b2bd0 0x56549f9b5cf0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.192+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v99'327 uv309 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.192+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b6230 con 0x56549f9b2bd0 2024-05-09T03:17:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.194+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v99'328 uv328 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.194+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b65d0 con 0x56549f9b2bd0 2024-05-09T03:17:57.196 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:17:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.195+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v99'329 uv329 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.195+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b6970 con 0x56549f9b2bd0 2024-05-09T03:17:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.197+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v99'330 uv330 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.197+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b6d10 con 0x56549f9b2bd0 2024-05-09T03:17:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.198+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v99'331 uv331 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.198+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b70b0 con 0x56549f9b2bd0 2024-05-09T03:17:57.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.200+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v99'332 uv314 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.200+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b7450 con 0x56549f9b2bd0 2024-05-09T03:17:57.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.201+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v99'333 uv333 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.201+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b7450 con 0x56549f9b2bd0 2024-05-09T03:17:57.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.202+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v99'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.203 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:17:57.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.202+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.203+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v99'335 uv335 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.204+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.205+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v99'336 uv336 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.205+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.206+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v99'337 uv319 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.206+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.207+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v99'338 uv338 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.207+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.209+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v99'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.209 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:17:57.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.209+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.210+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v99'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.210+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.211+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v99'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.212+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.213+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v99'342 uv325 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.213+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.214+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v99'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.214+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.215+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v99'344 uv344 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.216 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:17:57.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.215+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.217+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v99'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.217+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9aeeb0 con 0x56549f9b2bd0 2024-05-09T03:17:57.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.218+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v99'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.218+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b7900 con 0x56549f9b2bd0 2024-05-09T03:17:57.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.219+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v99'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9b2bd0 2024-05-09T03:17:57.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.219+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b7900 con 0x56549f880180 2024-05-09T03:17:57.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.220+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v99'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.220+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9b7900 con 0x56549f880180 2024-05-09T03:17:57.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.221+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v99'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.221+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b7900 con 0x56549f880180 2024-05-09T03:17:57.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.222+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v99'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.223 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:17:57.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.222+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b8910 con 0x56549f880180 2024-05-09T03:17:57.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.223+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v99'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.224+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b8910 con 0x56549f880180 2024-05-09T03:17:57.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.225+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v99'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.225+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b7900 con 0x56549f880180 2024-05-09T03:17:57.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.226+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v99'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.226+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56549f9b7e10 0x56549f9ba230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.226+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9ba900 con 0x56549f9b7e10 2024-05-09T03:17:57.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.226+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56549f9b7e10 0x56549f9ba230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.226+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56549f9b7e10 0x56549f9ba230 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.227+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v99'70 uv69 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.227+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9bae10 con 0x56549f9b7e10 2024-05-09T03:17:57.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.228+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v99'71 uv71 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.228+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bb310 con 0x56549f9b7e10 2024-05-09T03:17:57.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.229+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v99'72 uv72 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.230 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:17:57.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.229+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9b7530 con 0x56549f9b7e10 2024-05-09T03:17:57.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.230+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v99'73 uv73 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.230+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bbc40 con 0x56549f880180 2024-05-09T03:17:57.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.232+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v99'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.232+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9bbc40 con 0x56549f880180 2024-05-09T03:17:57.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.233+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v99'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.233+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bbc40 con 0x56549f880180 2024-05-09T03:17:57.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.234+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v99'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.235 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:17:57.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.234+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bbc40 con 0x56549f880180 2024-05-09T03:17:57.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.235+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v99'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.235+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bbc40 con 0x56549f880180 2024-05-09T03:17:57.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.236+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v99'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.236+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bbc40 con 0x56549f880180 2024-05-09T03:17:57.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.237+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v99'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.238+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56549f9bc6b0 0x56549f9bcac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.238+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.238+0000 7fa68d27e640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56549f9bc6b0 0x56549f9bcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.238+0000 7fa68d27e640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56549f9bc6b0 0x56549f9bcac0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.239+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v99'160 uv154 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.239+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.241+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v99'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.241+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.242+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v99'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.242 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:17:57.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.242+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c0140 con 0x56549f9bc6b0 2024-05-09T03:17:57.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.243+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v99'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.243+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c0100 con 0x56549f9bc6b0 2024-05-09T03:17:57.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.244+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v99'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.244+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.245+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v99'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.245+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.246+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v99'166 uv159 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.246+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.247+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v99'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.247+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.249+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v99'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.249 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:17:57.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.249+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.250+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v99'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.250+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9bf0f0 con 0x56549f9bc6b0 2024-05-09T03:17:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.251+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v99'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.251+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56549f9bfa10 0x56549f9bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.251+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c2580 con 0x56549f9bfa10 2024-05-09T03:17:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.251+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56549f9bfa10 0x56549f9bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.252+0000 7fa68ca7d640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56549f9bfa10 0x56549f9bfe20 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.252+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v99'71 uv70 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9bfa10 2024-05-09T03:17:57.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.253+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c2580 con 0x56549f9bfa10 2024-05-09T03:17:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.254+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v99'72 uv72 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9bfa10 2024-05-09T03:17:57.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.254+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c2600 con 0x56549f9bfa10 2024-05-09T03:17:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.255+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v99'73 uv73 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa67c00f9f0 con 0x56549f9bfa10 2024-05-09T03:17:57.255 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:17:57.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.255+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f9bfa10 2024-05-09T03:17:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.256+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v99'74 uv74 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa67c017040 con 0x56549f9bfa10 2024-05-09T03:17:57.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.256+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f9bfa10 2024-05-09T03:17:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.257+0000 7fa68ca7d640 1 -- 172.21.15.46:0/2025211982 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v99'75 uv75 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa67c017040 con 0x56549f9bfa10 2024-05-09T03:17:57.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.257+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f880180 2024-05-09T03:17:57.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.258+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v99'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.258+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f880180 2024-05-09T03:17:57.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.259+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v99'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.259+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f880180 2024-05-09T03:17:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.260+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v99'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.261 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:17:57.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.260+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3a00 con 0x56549f880180 2024-05-09T03:17:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.261+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v99'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.261+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3a00 con 0x56549f880180 2024-05-09T03:17:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.262+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v99'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.262+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f880180 2024-05-09T03:17:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.263+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v99'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f880180 2024-05-09T03:17:57.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.263+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c29f0 con 0x56549f9b7e10 2024-05-09T03:17:57.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.264+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v99'142 uv136 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.264+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c2d90 con 0x56549f9b7e10 2024-05-09T03:17:57.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.265+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v99'143 uv143 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.265+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.266+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v99'144 uv144 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.267 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:17:57.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.266+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.267+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v99'145 uv145 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.267+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.268+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v99'146 uv146 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.268+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.269+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v99'147 uv141 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.269+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.270+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v99'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.270+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.271+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v99'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.272 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:17:57.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.271+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c3130 con 0x56549f9b7e10 2024-05-09T03:17:57.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.272+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v99'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.272+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c5f40 con 0x56549f9b7e10 2024-05-09T03:17:57.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.274+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v99'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.274+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56549f9c3c60 0x56549f9c6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.274+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9c3c60 2024-05-09T03:17:57.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.274+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56549f9c3c60 0x56549f9c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.274+0000 7fa68c27c640 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56549f9c3c60 0x56549f9c6500 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.275+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v99'66 uv64 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9c3c60 2024-05-09T03:17:57.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.275+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9c3c60 2024-05-09T03:17:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.276+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v99'67 uv67 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9c3c60 2024-05-09T03:17:57.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.276+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9c3c60 2024-05-09T03:17:57.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.278+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v99'68 uv68 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9c3c60 2024-05-09T03:17:57.278 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:17:57.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.278+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9c3c60 2024-05-09T03:17:57.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.279+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v99'69 uv69 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9c3c60 2024-05-09T03:17:57.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.279+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9b7e10 2024-05-09T03:17:57.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.280+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v99'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.280+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9b7e10 2024-05-09T03:17:57.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.281+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v99'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.281+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9b7e10 2024-05-09T03:17:57.283 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:17:57.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.282+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v99'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.282+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9b7e10 2024-05-09T03:17:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.284+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v99'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.284+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9b7e10 2024-05-09T03:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.285+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v99'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.285+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9b7e10 2024-05-09T03:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.286+0000 7fa68c27c640 1 -- 172.21.15.46:0/2025211982 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v99'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa6800a0880 con 0x56549f9b7e10 2024-05-09T03:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.286+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.287+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v99'74 uv73 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.287+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.288+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v99'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.289+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.290+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v99'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.291 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:17:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.290+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.292+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v99'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.293+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.294+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v99'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.294+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.295+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v99'182 uv172 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.295+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.296+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v99'183 uv183 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.296+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.297+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v99'184 uv184 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.298 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:17:57.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.297+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.298+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v99'185 uv185 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.298+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.299+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v99'186 uv186 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.299+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.300+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v99'187 uv187 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.301 INFO:teuthology.orchestra.run.smithi046.stdout:Write #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:17:57.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.300+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.301+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v99'188 uv188 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.301+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.302+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v99'189 uv189 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.302+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6080 con 0x56549f9bc6b0 2024-05-09T03:17:57.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.303+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [omap-set-header] v99'190 uv190 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.303+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56549f9c6a40 con 0x56549f9bc6b0 2024-05-09T03:17:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v99'191 uv191 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa684003880 con 0x56549f9bc6b0 2024-05-09T03:17:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56549f880180 msgr2=0x56549f9b17a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56549f880180 0x56549f9b17a0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56549f9b7e10 msgr2=0x56549f9ba230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56549f9b7e10 0x56549f9ba230 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56549f9c3c60 msgr2=0x56549f9c6500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56549f9c3c60 0x56549f9c6500 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56549f9b2bd0 msgr2=0x56549f9b5cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56549f9b2bd0 0x56549f9b5cf0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56549f9bc6b0 msgr2=0x56549f9bcac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa68d27e640 1 -- 172.21.15.46:0/2025211982 reap_dead start 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.304+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56549f9bc6b0 0x56549f9bcac0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56549f9bfa10 msgr2=0x56549f9bfe20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56549f9bfa10 0x56549f9bfe20 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa678074cb0 msgr2=0x7fa6780771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa678074cb0 0x7fa6780771c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa67c001ed0 tx=0x7fa67c001aa0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 msgr2=0x56549f905900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f905900 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fa68000a4f0 tx=0x7fa680002fb0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 shutdown_connections 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56549f9bfa10 0x56549f9bfe20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56549f9bc6b0 0x56549f9bcac0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa678074cb0 0x7fa6780771c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 --2- 172.21.15.46:0/2025211982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56549f8677b0 0x56549f905900 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 >> 172.21.15.46:0/2025211982 conn(0x56549f75aac0 msgr2=0x56549f90ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 shutdown_connections 2024-05-09T03:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.305+0000 7fa692fb5f40 1 -- 172.21.15.46:0/2025211982 wait complete. 2024-05-09T03:17:57.309 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 1.00000000.inode - 2024-05-09T03:17:57.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.406+0000 7fd3b2db3f40 1 Processor -- start 2024-05-09T03:17:57.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.406+0000 7fd3b2db3f40 1 -- start start 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.407+0000 7fd3b2db3f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x56120617e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.407+0000 7fd3b2db3f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 0x5612060db720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.407+0000 7fd3b2db3f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561206185810 0x5612060dbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.407+0000 7fd3b2db3f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5612060f24d0 con 0x5612060dd7b0 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.407+0000 7fd3b2db3f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561206054f50 con 0x5612060ab7e0 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.407+0000 7fd3b2db3f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5612060f78a0 con 0x561206185810 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 0x5612060db720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 0x5612060db720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36396/0 (socket says 172.21.15.46:36396) 2024-05-09T03:17:57.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 -- 172.21.15.46:0/104237863 learned_addr learned my addr 172.21.15.46:0/104237863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac87b640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x56120617e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ad07c640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561206185810 0x5612060dbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 -- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561206185810 msgr2=0x5612060dbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561206185810 0x5612060dbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 -- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 msgr2=0x56120617e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x56120617e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 -- 172.21.15.46:0/104237863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56120617eb90 con 0x5612060dd7b0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac87b640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x56120617e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ad07c640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561206185810 0x5612060dbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ac07a640 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 0x5612060db720 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0045e10 tx=0x7fd3a004e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596e0ffaabff71e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ab078640 1 -- 172.21.15.46:0/104237863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0065070 con 0x5612060dd7b0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ab078640 1 -- 172.21.15.46:0/104237863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a00549f0 con 0x5612060dd7b0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.408+0000 7fd3ab078640 1 -- 172.21.15.46:0/104237863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0059040 con 0x5612060dd7b0 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/104237863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 msgr2=0x5612060db720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 0x5612060db720 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0045e10 tx=0x7fd3a004e070 comp rx=0 tx=0).stop 2024-05-09T03:17:57.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/104237863 shutdown_connections 2024-05-09T03:17:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561206185810 0x5612060dbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612060dd7b0 0x5612060db720 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/104237863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x56120617e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/104237863 >> 172.21.15.46:0/104237863 conn(0x561205fd0ac0 msgr2=0x5612060f5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/104237863 shutdown_connections 2024-05-09T03:17:57.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.409+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/104237863 wait complete. 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.410+0000 7fd3b2db3f40 1 Processor -- start 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.410+0000 7fd3b2db3f40 1 -- start start 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3b2db3f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x561206175890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3b2db3f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5612060dd7b0 0x56120621beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3b2db3f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561206185810 0x56120621c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3b2db3f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561206187640 con 0x561206185810 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3b2db3f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56120621d1f0 con 0x5612060ab7e0 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3b2db3f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56120621b170 con 0x5612060dd7b0 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3ad07c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561206185810 0x56120621c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3ad07c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561206185810 0x56120621c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36404/0 (socket says 172.21.15.46:36404) 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3ad07c640 1 -- 172.21.15.46:0/3175188200 learned_addr learned my addr 172.21.15.46:0/3175188200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3ac07a640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5612060dd7b0 0x56120621beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.411+0000 7fd3ac87b640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x561206175890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3ad07c640 1 -- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5612060dd7b0 msgr2=0x56120621beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3ad07c640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5612060dd7b0 0x56120621beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3ad07c640 1 -- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 msgr2=0x561206175890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3ad07c640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5612060ab7e0 0x561206175890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3ad07c640 1 -- 172.21.15.46:0/3175188200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612060aff50 con 0x561206185810 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3ad07c640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561206185810 0x56120621c3f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009e90 tx=0x7fd3a4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3aa076640 1 -- 172.21.15.46:0/3175188200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40143f0 con 0x561206185810 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3aa076640 1 -- 172.21.15.46:0/3175188200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a4019070 con 0x561206185810 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56120621b3d0 con 0x561206185810 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3aa076640 1 -- 172.21.15.46:0/3175188200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a40147d0 con 0x561206185810 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.412+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56120621b990 con 0x561206185810 2024-05-09T03:17:57.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.414+0000 7fd3aa076640 1 -- 172.21.15.46:0/3175188200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd3a4014970 con 0x561206185810 2024-05-09T03:17:57.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.414+0000 7fd3aa076640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd398074ce0 0x7fd3980771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.414+0000 7fd3aa076640 1 -- 172.21.15.46:0/3175188200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7fd3a40964e0 con 0x561206185810 2024-05-09T03:17:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.414+0000 7fd3ac87b640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd398074ce0 0x7fd3980771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.414+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x561206186da0 0x5612060ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.414+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x56120621ce40 con 0x561206186da0 2024-05-09T03:17:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.415+0000 7fd3ac07a640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x561206186da0 0x5612060ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.415+0000 7fd3ac87b640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd398074ce0 0x7fd3980771f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0012e0 tx=0x7fd39c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.415+0000 7fd3ac07a640 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x561206186da0 0x5612060ad760 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.416+0000 7fd3ac07a640 1 -- 172.21.15.46:0/3175188200 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 1.00000000.inode [writefull 0~80] v99'348 uv348 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd3a005c040 con 0x561206186da0 2024-05-09T03:17:57.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x561206186da0 msgr2=0x5612060ad760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x561206186da0 0x5612060ad760 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd398074ce0 msgr2=0x7fd3980771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd398074ce0 0x7fd3980771f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0012e0 tx=0x7fd39c008040 comp rx=0 tx=0).stop 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561206185810 msgr2=0x56120621c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 --2- 172.21.15.46:0/3175188200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561206185810 0x56120621c3f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009e90 tx=0x7fd3a4009f80 comp rx=0 tx=0).stop 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3ad07c640 1 -- 172.21.15.46:0/3175188200 reap_dead start 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 shutdown_connections 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 >> 172.21.15.46:0/3175188200 conn(0x561205fd0ac0 msgr2=0x5612060f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 shutdown_connections 2024-05-09T03:17:57.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.417+0000 7fd3b2db3f40 1 -- 172.21.15.46:0/3175188200 wait complete. 2024-05-09T03:17:57.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:17:57.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.578+0000 7f050e0e0640 1 Processor -- start 2024-05-09T03:17:57.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.578+0000 7f050e0e0640 1 -- start start 2024-05-09T03:17:57.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f050e0e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05080b4ee0 0x7f0508144420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f050e0e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080d1840 0x7f05080d1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f050e0e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05080d2190 con 0x7f05080b65d0 2024-05-09T03:17:57.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f050e0e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f05080b52f0 con 0x7f05080b4ee0 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f050e0e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f05080b6140 con 0x7f05080d1840 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f0506ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05080b4ee0 0x7f0508144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f05077fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f05080b65d0 0x7f050806f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49416/0 (socket says 172.21.15.46:49416) 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f05077fe640 1 -- 172.21.15.46:0/869210903 learned_addr learned my addr 172.21.15.46:0/869210903 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.579+0000 7f0507fff640 1 --2- 172.21.15.46:0/869210903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080d1840 0x7f05080d1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.580+0000 7f0506ffd640 1 -- 172.21.15.46:0/869210903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080d1840 msgr2=0x7f05080d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.580+0000 7f0506ffd640 1 --2- 172.21.15.46:0/869210903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080d1840 0x7f05080d1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.580+0000 7f0506ffd640 1 -- 172.21.15.46:0/869210903 >> v1:172.21.15.46:6789/0 conn(0x7f05080b65d0 legacy=0x7f050806f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.580+0000 7f0506ffd640 1 -- 172.21.15.46:0/869210903 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05080b5530 con 0x7f05080b4ee0 2024-05-09T03:17:57.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.580+0000 7f05067fc640 1 -- 172.21.15.46:0/869210903 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3861907407 0 0) 0x7f04f8001110 con 0x7f05080b65d0 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.580+0000 7f0506ffd640 1 --2- 172.21.15.46:0/869210903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f0508144420 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0097a0 tx=0x7f04fc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e786aad8da3dd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f05067fc640 1 -- 172.21.15.46:0/869210903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc037070 con 0x7f05080b4ee0 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f05067fc640 1 -- 172.21.15.46:0/869210903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04fc032440 con 0x7f05080b4ee0 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f05067fc640 1 -- 172.21.15.46:0/869210903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc032810 con 0x7f05080b4ee0 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 -- 172.21.15.46:0/869210903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 msgr2=0x7f0508144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 --2- 172.21.15.46:0/869210903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f0508144420 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0097a0 tx=0x7f04fc02a180 comp rx=0 tx=0).stop 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 -- 172.21.15.46:0/869210903 shutdown_connections 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 --2- 172.21.15.46:0/869210903 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080d1840 0x7f05080d1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 --2- 172.21.15.46:0/869210903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f0508144420 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 -- 172.21.15.46:0/869210903 >> 172.21.15.46:0/869210903 conn(0x7f05080b3620 msgr2=0x7f05080b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:57.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.581+0000 7f050e0e0640 1 -- 172.21.15.46:0/869210903 shutdown_connections 2024-05-09T03:17:57.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.582+0000 7f050e0e0640 1 -- 172.21.15.46:0/869210903 wait complete. 2024-05-09T03:17:57.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.583+0000 7f050e0e0640 1 Processor -- start 2024-05-09T03:17:57.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.583+0000 7f050e0e0640 1 -- start start 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f050e0e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f050813e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f050e0e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080b65d0 0x7f050813fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f050e0e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05080d1840 0x7f0508140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f050e0e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05080b5dd0 con 0x7f05080b4ee0 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f050e0e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f05080b5af0 con 0x7f05080d1840 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f050e0e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0508144d50 con 0x7f05080b65d0 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f050813e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f050813e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36410/0 (socket says 172.21.15.46:36410) 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 -- 172.21.15.46:0/3519133344 learned_addr learned my addr 172.21.15.46:0/3519133344 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f0507fff640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05080d1840 0x7f0508140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f0506ffd640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080b65d0 0x7f050813fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 -- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080b65d0 msgr2=0x7f050813fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080b65d0 0x7f050813fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 -- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05080d1840 msgr2=0x7f0508140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05080d1840 0x7f0508140390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 -- 172.21.15.46:0/3519133344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05080d0470 con 0x7f05080b4ee0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f05077fe640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f050813e2d0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c410 tx=0x7f04f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.585+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f80074e0 con 0x7f05080b4ee0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.586+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f80047e0 con 0x7f05080b4ee0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.586+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05080d0750 con 0x7f05080b4ee0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.586+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8004bd0 con 0x7f05080b4ee0 2024-05-09T03:17:57.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.586+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05080d0d10 con 0x7f05080b4ee0 2024-05-09T03:17:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.586+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05080d0f50 con 0x7f05080b4ee0 2024-05-09T03:17:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.587+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f04f8011070 con 0x7f05080b4ee0 2024-05-09T03:17:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.588+0000 7f04effff640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04f4076300 0x7f04f4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.588+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 7057+0+0 (secure 0 0 0) 0x7f04f8097b70 con 0x7f05080b4ee0 2024-05-09T03:17:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.588+0000 7f0506ffd640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04f4076300 0x7f04f4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.589+0000 7f0506ffd640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04f4076300 0x7f04f4078810 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f04fc034000 tx=0x7f04fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:57.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.592+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f05080d0f50 con 0x7f05080b4ee0 2024-05-09T03:17:57.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:57.828+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f050806a270 con 0x7f05080b4ee0 2024-05-09T03:17:58.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.058+0000 7f04effff640 1 -- 172.21.15.46:0/3519133344 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v183) ==== 162+0+0 (secure 0 0 0) 0x7f05080d2190 con 0x7f05080b4ee0 2024-05-09T03:17:58.059 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.060+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04f4076300 msgr2=0x7f04f4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.060+0000 7f050e0e0640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04f4076300 0x7f04f4078810 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f04fc034000 tx=0x7f04fc02e000 comp rx=0 tx=0).stop 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.060+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 msgr2=0x7f050813e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.060+0000 7f050e0e0640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f050813e2d0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c410 tx=0x7f04f800c8d0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 shutdown_connections 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05080d1840 0x7f0508140390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f05080b65d0 0x7f050813fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04f4076300 0x7f04f4078810 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 --2- 172.21.15.46:0/3519133344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05080b4ee0 0x7f050813e2d0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 >> 172.21.15.46:0/3519133344 conn(0x7f05080b3620 msgr2=0x7f050815da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:58.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 shutdown_connections 2024-05-09T03:17:58.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.061+0000 7f050e0e0640 1 -- 172.21.15.46:0/3519133344 wait complete. 2024-05-09T03:17:58.070 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:58.070+0000 7f0c8c3b4640 -1 mds.0.cache.ino(0x1) Corrupt inode 0x1: End of buffer [buffer:2] 2024-05-09T03:17:58.071 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:17:58.070+0000 7f0c8c3b4640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:17:58.071 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:58.071+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:17:58.071 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:17:58.071+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:17:58.075 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:58.077 INFO:tasks.ceph.mds.c.smithi046.stderr: -16> 2024-05-09T03:17:58.070+0000 7f0c8c3b4640 -1 mds.0.cache.ino(0x1) Corrupt inode 0x1: End of buffer [buffer:2] 2024-05-09T03:17:58.077 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:17:58.070+0000 7f0c8c3b4640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:17:58.132 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:17:58.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.193+0000 7faa04d34640 1 Processor -- start 2024-05-09T03:17:58.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.193+0000 7faa04d34640 1 -- start start 2024-05-09T03:17:58.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7faa04d34640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 0x7faa000b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7faa04d34640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000cf6b0 0x7faa0006d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7faa04d34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000d2b80 con 0x7faa000b92e0 2024-05-09T03:17:58.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7faa04d34640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faa000d2e60 con 0x7faa000d0b90 2024-05-09T03:17:58.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7faa04d34640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faa000d3140 con 0x7faa000cf6b0 2024-05-09T03:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fdd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 0x7faa000b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fed9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000cf6b0 0x7faa0006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fdd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 0x7faa000b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:41552/0 (socket says 172.21.15.46:41552) 2024-05-09T03:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fdd9b640 1 -- 172.21.15.46:0/3989621323 learned_addr learned my addr 172.21.15.46:0/3989621323 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fe59c640 1 --1- 172.21.15.46:0/3989621323 >> v1:172.21.15.88:6789/0 conn(0x7faa000b92e0 0x7faa000d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:33224/0 (socket says 172.21.15.46:33224) 2024-05-09T03:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fed9d640 1 -- 172.21.15.46:0/3989621323 >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 msgr2=0x7faa000b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fed9d640 1 --2- 172.21.15.46:0/3989621323 >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 0x7faa000b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fed9d640 1 -- 172.21.15.46:0/3989621323 >> v1:172.21.15.88:6789/0 conn(0x7faa000b92e0 legacy=0x7faa000d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fed9d640 1 -- 172.21.15.46:0/3989621323 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa000d3420 con 0x7faa000cf6b0 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.195+0000 7fa9fdd9b640 1 --2- 172.21.15.46:0/3989621323 >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 0x7faa000b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.196+0000 7fa9fed9d640 1 --2- 172.21.15.46:0/3989621323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000cf6b0 0x7faa0006d790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009d90 tx=0x7fa9f802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff686b0a4335adb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.196+0000 7fa9fd59a640 1 -- 172.21.15.46:0/3989621323 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8031600 con 0x7faa000cf6b0 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.196+0000 7fa9fd59a640 1 -- 172.21.15.46:0/3989621323 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f80317a0 con 0x7faa000cf6b0 2024-05-09T03:17:58.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.196+0000 7fa9fd59a640 1 -- 172.21.15.46:0/3989621323 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8031b70 con 0x7faa000cf6b0 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 -- 172.21.15.46:0/3989621323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000cf6b0 msgr2=0x7faa0006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 --2- 172.21.15.46:0/3989621323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000cf6b0 0x7faa0006d790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009d90 tx=0x7fa9f802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 -- 172.21.15.46:0/3989621323 shutdown_connections 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 --2- 172.21.15.46:0/3989621323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000cf6b0 0x7faa0006d790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 --2- 172.21.15.46:0/3989621323 >> v2:172.21.15.88:3300/0 conn(0x7faa000d0b90 0x7faa000b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 -- 172.21.15.46:0/3989621323 >> 172.21.15.46:0/3989621323 conn(0x7faa000d3c00 msgr2=0x7faa000d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:58.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 -- 172.21.15.46:0/3989621323 shutdown_connections 2024-05-09T03:17:58.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.197+0000 7faa04d34640 1 -- 172.21.15.46:0/3989621323 wait complete. 2024-05-09T03:17:58.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.199+0000 7faa04d34640 1 Processor -- start 2024-05-09T03:17:58.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.199+0000 7faa04d34640 1 -- start start 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7faa04d34640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b92e0 0x7faa000cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7faa04d34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 0x7faa000cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7faa04d34640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa00158ed0 0x7faa0014efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7faa04d34640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa0006e060 con 0x7faa000cf6b0 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7faa04d34640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa0006dee0 con 0x7faa00158ed0 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7faa04d34640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faa0006e1e0 con 0x7faa000b92e0 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 0x7faa000cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 0x7faa000cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36448/0 (socket says 172.21.15.46:36448) 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b92e0 0x7faa000cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fdd9b640 1 -- 172.21.15.46:0/4154895024 learned_addr learned my addr 172.21.15.46:0/4154895024 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fed9d640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa00158ed0 0x7faa0014efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 -- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa00158ed0 msgr2=0x7faa0014efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa00158ed0 0x7faa0014efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 -- 172.21.15.46:0/4154895024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 msgr2=0x7faa000cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 0x7faa000cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 -- 172.21.15.46:0/4154895024 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa000cb450 con 0x7faa000b92e0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fdd9b640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 0x7faa000cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9fe59c640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b92e0 0x7faa000cc4f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c390 tx=0x7fa9f000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.201+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00074e0 con 0x7faa000b92e0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.202+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f0007680 con 0x7faa000b92e0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.202+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa000cb730 con 0x7faa000b92e0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.202+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00049e0 con 0x7faa000b92e0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.202+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa000cbcf0 con 0x7faa000b92e0 2024-05-09T03:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.202+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0006a270 con 0x7faa000b92e0 2024-05-09T03:17:58.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.203+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa9f001a020 con 0x7faa000b92e0 2024-05-09T03:17:58.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.204+0000 7fa9eeffd640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9e00762d0 0x7fa9e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.204+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7fa9f0097810 con 0x7faa000b92e0 2024-05-09T03:17:58.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.204+0000 7fa9fdd9b640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9e00762d0 0x7fa9e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.205+0000 7fa9fdd9b640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9e00762d0 0x7fa9e00787e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7faa001f9ae0 tx=0x7fa9f4009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.209+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa9f0021070 con 0x7faa000b92e0 2024-05-09T03:17:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.440+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa001e9960 con 0x7faa000b92e0 2024-05-09T03:17:58.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.441+0000 7fa9eeffd640 1 -- 172.21.15.46:0/4154895024 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+4343 (secure 0 0 0) 0x7fa9f0060660 con 0x7faa000b92e0 2024-05-09T03:17:58.441 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:58.442 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":185,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:58.071501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:58.442 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 185 2024-05-09T03:17:58.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9e00762d0 msgr2=0x7fa9e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9e00762d0 0x7fa9e00787e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7faa001f9ae0 tx=0x7fa9f4009240 comp rx=0 tx=0).stop 2024-05-09T03:17:58.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b92e0 msgr2=0x7faa000cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b92e0 0x7faa000cc4f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa9f000c390 tx=0x7fa9f000c850 comp rx=0 tx=0).stop 2024-05-09T03:17:58.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 shutdown_connections 2024-05-09T03:17:58.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa00158ed0 0x7faa0014efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa9e00762d0 0x7fa9e00787e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa000cf6b0 0x7faa000cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 --2- 172.21.15.46:0/4154895024 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa000b92e0 0x7faa000cc4f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 >> 172.21.15.46:0/4154895024 conn(0x7faa000d3c00 msgr2=0x7faa000d0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:58.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 shutdown_connections 2024-05-09T03:17:58.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.443+0000 7faa04d34640 1 -- 172.21.15.46:0/4154895024 wait complete. 2024-05-09T03:17:58.455 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:58.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:58.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.569+0000 7f2ad7300640 1 Processor -- start 2024-05-09T03:17:58.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.569+0000 7f2ad7300640 1 -- start start 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.570+0000 7f2ad7300640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ad00b5210 0x7f2ad01613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.570+0000 7f2ad7300640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00b7ed0 0x7f2ad00d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.570+0000 7f2ad7300640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ad00d5280 con 0x7f2ad00d4800 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.570+0000 7f2ad7300640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2ad00b5620 con 0x7f2ad00b5210 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.570+0000 7f2ad7300640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad01609a0 con 0x7f2ad00b7ed0 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.570+0000 7f2ad509c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ad00b5210 0x7f2ad01613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad509c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ad00b5210 0x7f2ad01613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36454/0 (socket says 172.21.15.46:36454) 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad509c640 1 -- 172.21.15.46:0/387864517 learned_addr learned my addr 172.21.15.46:0/387864517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad489b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ad00d4800 0x7f2ad00b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49428/0 (socket says 172.21.15.46:49428) 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad509c640 1 -- 172.21.15.46:0/387864517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00b7ed0 msgr2=0x7f2ad00d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad509c640 1 --2- 172.21.15.46:0/387864517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00b7ed0 0x7f2ad00d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad509c640 1 -- 172.21.15.46:0/387864517 >> v1:172.21.15.46:6789/0 conn(0x7f2ad00d4800 legacy=0x7f2ad00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ac7fff640 1 -- 172.21.15.46:0/387864517 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847409259 0 0) 0x7f2ad00d5280 con 0x7f2ad00d4800 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.571+0000 7f2ad509c640 1 -- 172.21.15.46:0/387864517 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad00d5560 con 0x7f2ad00b5210 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad509c640 1 --2- 172.21.15.46:0/387864517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad01613c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009870 tx=0x7f2acc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9728ca8911710d32 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ac7fff640 1 -- 172.21.15.46:0/387864517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc037070 con 0x7f2ad00b5210 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ac7fff640 1 -- 172.21.15.46:0/387864517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc032440 con 0x7f2ad00b5210 2024-05-09T03:17:58.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ac7fff640 1 -- 172.21.15.46:0/387864517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc032810 con 0x7f2ad00b5210 2024-05-09T03:17:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad7300640 1 -- 172.21.15.46:0/387864517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 msgr2=0x7f2ad01613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad7300640 1 --2- 172.21.15.46:0/387864517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad01613c0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009870 tx=0x7f2acc02a1c0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad7300640 1 -- 172.21.15.46:0/387864517 shutdown_connections 2024-05-09T03:17:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad7300640 1 --2- 172.21.15.46:0/387864517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00b7ed0 0x7f2ad00d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad7300640 1 --2- 172.21.15.46:0/387864517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad01613c0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.572+0000 7f2ad7300640 1 -- 172.21.15.46:0/387864517 >> 172.21.15.46:0/387864517 conn(0x7f2ad00d29a0 msgr2=0x7f2ad00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.573+0000 7f2ad7300640 1 -- 172.21.15.46:0/387864517 shutdown_connections 2024-05-09T03:17:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.573+0000 7f2ad7300640 1 -- 172.21.15.46:0/387864517 wait complete. 2024-05-09T03:17:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.574+0000 7f2ad7300640 1 Processor -- start 2024-05-09T03:17:58.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.574+0000 7f2ad7300640 1 -- start start 2024-05-09T03:17:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad7300640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad7300640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad00d4800 0x7f2ad01fa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad7300640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00cb860 0x7f2ad00cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad7300640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad00d0ae0 con 0x7f2ad00b5210 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad7300640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad0160ea0 con 0x7f2ad00d4800 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad7300640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad0163e60 con 0x7f2ad00cb860 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad015dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36466/0 (socket says 172.21.15.46:36466) 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 -- 172.21.15.46:0/1626356871 learned_addr learned my addr 172.21.15.46:0/1626356871 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad489b640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad00d4800 0x7f2ad01fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad589d640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00cb860 0x7f2ad00cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 -- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00cb860 msgr2=0x7f2ad00cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00cb860 0x7f2ad00cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 -- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad00d4800 msgr2=0x7f2ad01fa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad00d4800 0x7f2ad01fa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 -- 172.21.15.46:0/1626356871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad00cc1b0 con 0x7f2ad00b5210 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad589d640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00cb860 0x7f2ad00cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ad509c640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad015dea0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009870 tx=0x7f2acc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.576+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc037040 con 0x7f2ad00b5210 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.577+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc004250 con 0x7f2ad00b5210 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.577+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad00cda10 con 0x7f2ad00b5210 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.577+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc03c670 con 0x7f2ad00b5210 2024-05-09T03:17:58.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.577+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad00cdfd0 con 0x7f2ad00b5210 2024-05-09T03:17:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.578+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad006a270 con 0x7f2ad00b5210 2024-05-09T03:17:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.579+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2acc03a070 con 0x7f2ad00b5210 2024-05-09T03:17:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.579+0000 7f2ac5ffb640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ab40762d0 0x7f2ab40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.579+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7f2acc0b8540 con 0x7f2ad00b5210 2024-05-09T03:17:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.579+0000 7f2ad489b640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ab40762d0 0x7f2ab40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.580+0000 7f2ad489b640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ab40762d0 0x7f2ab40787e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00b8550 tx=0x7f2ac0007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.584+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2ad00cc1b0 con 0x7f2ad00b5210 2024-05-09T03:17:58.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.820+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ad015ef50 con 0x7f2ad00b5210 2024-05-09T03:17:58.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.820+0000 7f2ac5ffb640 1 -- 172.21.15.46:0/1626356871 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+4343 (secure 0 0 0) 0x7f2acc041020 con 0x7f2ad00b5210 2024-05-09T03:17:58.821 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:58.821 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":185,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:58.071501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:58.821 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 185 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.822+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ab40762d0 msgr2=0x7f2ab40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.822+0000 7f2ad7300640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ab40762d0 0x7f2ab40787e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2ad00b8550 tx=0x7f2ac0007c00 comp rx=0 tx=0).stop 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.822+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 msgr2=0x7f2ad015dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.822+0000 7f2ad7300640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad015dea0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009870 tx=0x7f2acc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 shutdown_connections 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2ad00cb860 0x7f2ad00cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ad00d4800 0x7f2ad01fa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ab40762d0 0x7f2ab40787e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 --2- 172.21.15.46:0/1626356871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ad00b5210 0x7f2ad015dea0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 >> 172.21.15.46:0/1626356871 conn(0x7f2ad00d29a0 msgr2=0x7f2ad00b48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 shutdown_connections 2024-05-09T03:17:58.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.823+0000 7f2ad7300640 1 -- 172.21.15.46:0/1626356871 wait complete. 2024-05-09T03:17:58.834 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:17:58.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:17:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.948+0000 7f429485b640 1 Processor -- start 2024-05-09T03:17:58.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.948+0000 7f429485b640 1 -- start start 2024-05-09T03:17:58.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f429485b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 0x7f429008cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f429485b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f429008d6b0 0x7f4290164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f429485b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f429006ebe0 con 0x7f429008d2a0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f429485b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f429006f760 con 0x7f429008d6b0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f429485b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f429006eec0 con 0x7f42900b9310 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f4287fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f429008d2a0 0x7f4290164830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49430/0 (socket says 172.21.15.46:49430) 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f4287fff640 1 -- 172.21.15.46:0/1730476369 learned_addr learned my addr 172.21.15.46:0/1730476369 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f428ed9d640 1 --2- 172.21.15.46:0/1730476369 >> v2:172.21.15.88:3300/0 conn(0x7f429008d6b0 0x7f4290164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f428e59c640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 0x7f429008cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1475669356 0 0) 0x7f429006ebe0 con 0x7f429008d2a0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f427c003520 con 0x7f429008d2a0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.950+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2207420668 0 0) 0x7f427c003520 con 0x7f429008d2a0 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 msgr2=0x7f429008cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 0x7f429008cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 >> v2:172.21.15.88:3300/0 conn(0x7f429008d6b0 msgr2=0x7f4290164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:58.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 --2- 172.21.15.46:0/1730476369 >> v2:172.21.15.88:3300/0 conn(0x7f429008d6b0 0x7f4290164f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429006f1a0 con 0x7f429008d2a0 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f42780016f0 con 0x7f429008d2a0 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 >> v1:172.21.15.46:6789/0 conn(0x7f429008d2a0 legacy=0x7f4290164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f427c003a50 0x7f427c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f427c004800 0x7f427c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 0x7f427c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f429006f760 con 0x7f427c003a50 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f429006eec0 con 0x7f427c004800 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f427c00baa0 con 0x7f42900b9310 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428e59c640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f427c003a50 0x7f427c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f4287fff640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f427c004800 0x7f427c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428ed9d640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 0x7f427c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428e59c640 1 -- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 msgr2=0x7f427c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428e59c640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42900b9310 0x7f427c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428e59c640 1 -- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f427c004800 msgr2=0x7f427c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428e59c640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f427c004800 0x7f427c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.951+0000 7f428e59c640 1 -- 172.21.15.46:0/1730476369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c003520 con 0x7f427c003a50 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.952+0000 7f428e59c640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f427c003a50 0x7f427c003ec0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f4280009a90 tx=0x7f427c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b294db314b34752e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.952+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280036070 con 0x7f427c003a50 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.952+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4280031400 con 0x7f427c003a50 2024-05-09T03:17:58.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.952+0000 7f428dd9b640 1 -- 172.21.15.46:0/1730476369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280009690 con 0x7f427c003a50 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f429485b640 1 -- 172.21.15.46:0/1730476369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f427c003a50 msgr2=0x7f427c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f429485b640 1 --2- 172.21.15.46:0/1730476369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f427c003a50 0x7f427c003ec0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f4280009a90 tx=0x7f427c005830 comp rx=0 tx=0).stop 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f428ed9d640 1 -- 172.21.15.46:0/1730476369 reap_dead start 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f429485b640 1 -- 172.21.15.46:0/1730476369 shutdown_connections 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f429485b640 1 -- 172.21.15.46:0/1730476369 >> 172.21.15.46:0/1730476369 conn(0x7f42900b5660 msgr2=0x7f42900b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f429485b640 1 -- 172.21.15.46:0/1730476369 shutdown_connections 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.953+0000 7f429485b640 1 -- 172.21.15.46:0/1730476369 wait complete. 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.955+0000 7f429485b640 1 Processor -- start 2024-05-09T03:17:58.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.955+0000 7f429485b640 1 -- start start 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f429485b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f427c003a50 0x7f42901f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f429485b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42900c8b70 0x7f42900b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f429485b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 0x7f42900b8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f429485b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4290165480 con 0x7f42900c8f80 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f429485b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f429008dac0 con 0x7f42900c8b70 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f429485b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4290165600 con 0x7f427c003a50 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 0x7f42900b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 0x7f42900b8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36486/0 (socket says 172.21.15.46:36486) 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 -- 172.21.15.46:0/1751282111 learned_addr learned my addr 172.21.15.46:0/1751282111 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f4287fff640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42900c8b70 0x7f42900b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428e59c640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f427c003a50 0x7f42901f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 -- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f427c003a50 msgr2=0x7f42901f5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f427c003a50 0x7f42901f5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 -- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42900c8b70 msgr2=0x7f42900b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42900c8b70 0x7f42900b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.956+0000 7f428ed9d640 1 -- 172.21.15.46:0/1751282111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42900caf30 con 0x7f42900c8f80 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.957+0000 7f428ed9d640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 0x7f42900b8020 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f427c004780 tx=0x7f428800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.957+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288011e60 con 0x7f42900c8f80 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.957+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f428801d070 con 0x7f42900c8f80 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.957+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42900c9750 con 0x7f42900c8f80 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.957+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4288015670 con 0x7f42900c8f80 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.957+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42900c9d10 con 0x7f42900c8f80 2024-05-09T03:17:58.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.958+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429006a270 con 0x7f42900c8f80 2024-05-09T03:17:58.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.959+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4288015810 con 0x7f42900c8f80 2024-05-09T03:17:58.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.959+0000 7f4286ffd640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42700762d0 0x7f42700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:58.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.959+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7f42880983a0 con 0x7f42900c8f80 2024-05-09T03:17:58.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.959+0000 7f428e59c640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42700762d0 0x7f42700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:58.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.960+0000 7f428e59c640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42700762d0 0x7f42700787e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4280033000 tx=0x7f4280008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:58.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:58.964+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f42900caf30 con 0x7f42900c8f80 2024-05-09T03:17:59.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.195+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4290072f70 con 0x7f42900c8f80 2024-05-09T03:17:59.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.196+0000 7f4286ffd640 1 -- 172.21.15.46:0/1751282111 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+5166 (secure 0 0 0) 0x7f4288010040 con 0x7f42900c8f80 2024-05-09T03:17:59.197 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:17:59.197 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":186,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8625,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/271467610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":271467610},{"type":"v1","addr":"172.21.15.46:6834","nonce":271467610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":186}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:17:58.071501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:17:59.197 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 186 2024-05-09T03:17:59.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42700762d0 msgr2=0x7f42700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42700762d0 0x7f42700787e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4280033000 tx=0x7f4280008030 comp rx=0 tx=0).stop 2024-05-09T03:17:59.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 msgr2=0x7f42900b8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 0x7f42900b8020 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f427c004780 tx=0x7f428800f580 comp rx=0 tx=0).stop 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 shutdown_connections 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42700762d0 0x7f42700787e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42900c8f80 0x7f42900b8020 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42900c8b70 0x7f42900b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 --2- 172.21.15.46:0/1751282111 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f427c003a50 0x7f42901f5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.198+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 >> 172.21.15.46:0/1751282111 conn(0x7f42900b5660 msgr2=0x7f42900b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.199+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 shutdown_connections 2024-05-09T03:17:59.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.199+0000 7f429485b640 1 -- 172.21.15.46:0/1751282111 wait complete. 2024-05-09T03:17:59.211 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 1.00000000.inode' led to DAMAGED state 2024-05-09T03:17:59.211 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 200.00000000' 2024-05-09T03:17:59.211 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:17:59.211 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:17:59.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:17:59.227 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:17:59.228 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:17:59.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:17:59.228 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:17:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:17:59.283 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:17:59.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:17:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.438+0000 7fb12e013640 1 Processor -- start 2024-05-09T03:17:59.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.438+0000 7fb12e013640 1 -- start start 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.439+0000 7fb12e013640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1280d3730 0x7fb12806e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.439+0000 7fb12e013640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb12809d470 con 0x7fb12806e7e0 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.439+0000 7fb12e013640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1280d11b0 con 0x7fb128165090 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.439+0000 7fb12e013640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb12809cfe0 con 0x7fb1280d3730 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb11ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb12806e7e0 0x7fb128164980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49440/0 (socket says 172.21.15.46:49440) 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb11ffff640 1 -- 172.21.15.46:0/495460655 learned_addr learned my addr 172.21.15.46:0/495460655 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb1277fe640 1 --2- 172.21.15.46:0/495460655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1280d3730 0x7fb12806e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:59.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1011684597 0 0) 0x7fb110001110 con 0x7fb12806e7e0 2024-05-09T03:17:59.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb114003520 con 0x7fb12806e7e0 2024-05-09T03:17:59.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941681640 0 0) 0x7fb1280d11b0 con 0x7fb128165090 2024-05-09T03:17:59.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb110001110 con 0x7fb128165090 2024-05-09T03:17:59.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb1277fe640 1 -- 172.21.15.46:0/495460655 >> v1:172.21.15.88:6789/0 conn(0x7fb128165090 legacy=0x7fb1280d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb1277fe640 1 -- 172.21.15.46:0/495460655 >> v1:172.21.15.46:6789/0 conn(0x7fb12806e7e0 legacy=0x7fb128164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.440+0000 7fb1277fe640 1 -- 172.21.15.46:0/495460655 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1280d1490 con 0x7fb1280d3730 2024-05-09T03:17:59.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.441+0000 7fb1277fe640 1 --2- 172.21.15.46:0/495460655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1280d3730 0x7fb12806e2a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb1180097d0 tx=0x7fb11802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=985bd46f8842de8e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:59.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.441+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118037070 con 0x7fb1280d3730 2024-05-09T03:17:59.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.441+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb118032440 con 0x7fb1280d3730 2024-05-09T03:17:59.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.441+0000 7fb126ffd640 1 -- 172.21.15.46:0/495460655 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1180327f0 con 0x7fb1280d3730 2024-05-09T03:17:59.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 -- 172.21.15.46:0/495460655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1280d3730 msgr2=0x7fb12806e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 --2- 172.21.15.46:0/495460655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1280d3730 0x7fb12806e2a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb1180097d0 tx=0x7fb11802a180 comp rx=0 tx=0).stop 2024-05-09T03:17:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 -- 172.21.15.46:0/495460655 shutdown_connections 2024-05-09T03:17:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 --2- 172.21.15.46:0/495460655 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1280d3730 0x7fb12806e2a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 -- 172.21.15.46:0/495460655 >> 172.21.15.46:0/495460655 conn(0x7fb1280d05f0 msgr2=0x7fb1280d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:17:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 -- 172.21.15.46:0/495460655 shutdown_connections 2024-05-09T03:17:59.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.442+0000 7fb12e013640 1 -- 172.21.15.46:0/495460655 wait complete. 2024-05-09T03:17:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.444+0000 7fb12e013640 1 Processor -- start 2024-05-09T03:17:59.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.444+0000 7fb12e013640 1 -- start start 2024-05-09T03:17:59.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb12e013640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb12806e7e0 0x7fb12809fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb12e013640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 0x7fb12815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb12e013640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb128165090 0x7fb1280cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb12e013640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1280cfbb0 con 0x7fb1280d3730 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb12e013640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1280b94a0 con 0x7fb12806e7e0 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb12e013640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb1280cfd30 con 0x7fb128165090 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb11ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 0x7fb12815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb11ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 0x7fb12815fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36498/0 (socket says 172.21.15.46:36498) 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb11ffff640 1 -- 172.21.15.46:0/4191957973 learned_addr learned my addr 172.21.15.46:0/4191957973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb1277fe640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb12806e7e0 0x7fb12809fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.445+0000 7fb127fff640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb128165090 0x7fb1280cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb11ffff640 1 -- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb128165090 msgr2=0x7fb1280cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb11ffff640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb128165090 0x7fb1280cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb11ffff640 1 -- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb12806e7e0 msgr2=0x7fb12809fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:17:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb11ffff640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb12806e7e0 0x7fb12809fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb11ffff640 1 -- 172.21.15.46:0/4191957973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12809f8b0 con 0x7fb1280d3730 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb11ffff640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 0x7fb12815fe10 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c730 tx=0x7fb11000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110017070 con 0x7fb1280d3730 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb110005ad0 con 0x7fb1280d3730 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1280a0a90 con 0x7fb1280d3730 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110007070 con 0x7fb1280d3730 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.446+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1280ccc10 con 0x7fb1280d3730 2024-05-09T03:17:59.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.447+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12809ebe0 con 0x7fb1280d3730 2024-05-09T03:17:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.448+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb11001a070 con 0x7fb1280d3730 2024-05-09T03:17:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.448+0000 7fb124ff9640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1080762d0 0x7fb1080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:17:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.449+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7fb110096fa0 con 0x7fb1280d3730 2024-05-09T03:17:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.449+0000 7fb1277fe640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1080762d0 0x7fb1080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:17:59.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.449+0000 7fb1277fe640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1080762d0 0x7fb1080787e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb118034000 tx=0x7fb11802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:17:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.453+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb12809d470 con 0x7fb1280d3730 2024-05-09T03:17:59.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:17:59.683+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb1280a1820 con 0x7fb1280d3730 2024-05-09T03:18:00.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.086+0000 7fb124ff9640 1 -- 172.21.15.46:0/4191957973 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v187) ==== 161+0+0 (secure 0 0 0) 0x7fb11005fc90 con 0x7fb1280d3730 2024-05-09T03:18:00.087 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:00.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1080762d0 msgr2=0x7fb1080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:00.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1080762d0 0x7fb1080787e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb118034000 tx=0x7fb11802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:00.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 msgr2=0x7fb12815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:00.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 0x7fb12815fe10 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c730 tx=0x7fb11000cbf0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 shutdown_connections 2024-05-09T03:18:00.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb128165090 0x7fb1280cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1080762d0 0x7fb1080787e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1280d3730 0x7fb12815fe10 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 --2- 172.21.15.46:0/4191957973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb12806e7e0 0x7fb12809fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 >> 172.21.15.46:0/4191957973 conn(0x7fb1280d05f0 msgr2=0x7fb128072280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:00.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 shutdown_connections 2024-05-09T03:18:00.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.088+0000 7fb12e013640 1 -- 172.21.15.46:0/4191957973 wait complete. 2024-05-09T03:18:00.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:00.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.218+0000 7f789c043640 1 Processor -- start 2024-05-09T03:18:00.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.218+0000 7f789c043640 1 -- start start 2024-05-09T03:18:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f789c043640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78940b7e70 0x7f78940b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:00.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f789c043640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78940d3400 con 0x7f78940d2e90 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f789c043640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78940d4a60 con 0x7f78941605b0 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f789c043640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f78940d4d40 con 0x7f78940b7e70 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f78995de640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78940b7e70 0x7f78940b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f7899ddf640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f78940d2e90 0x7f78940b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49454/0 (socket says 172.21.15.46:49454) 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f78995de640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f78940b7e70 0x7f78940b92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36502/0 (socket says 172.21.15.46:36502) 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.219+0000 7f7899ddf640 1 -- 172.21.15.46:0/2191694866 learned_addr learned my addr 172.21.15.46:0/2191694866 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.220+0000 7f78995de640 1 -- 172.21.15.46:0/2191694866 >> v1:172.21.15.88:6789/0 conn(0x7f78941605b0 legacy=0x7f78940b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.220+0000 7f78995de640 1 -- 172.21.15.46:0/2191694866 >> v1:172.21.15.46:6789/0 conn(0x7f78940d2e90 legacy=0x7f78940b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.220+0000 7f7898ddd640 1 -- 172.21.15.46:0/2191694866 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3661166292 0 0) 0x7f78940d3400 con 0x7f78940d2e90 2024-05-09T03:18:00.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.220+0000 7f78995de640 1 -- 172.21.15.46:0/2191694866 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78940d5020 con 0x7f78940b7e70 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.220+0000 7f78995de640 1 --2- 172.21.15.46:0/2191694866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7e70 0x7f78940b92a0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f78880097a0 tx=0x7f788802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434d938aa6edd12b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f7898ddd640 1 -- 172.21.15.46:0/2191694866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888037070 con 0x7f78940b7e70 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f7898ddd640 1 -- 172.21.15.46:0/2191694866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788802ae50 con 0x7f78940b7e70 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f7898ddd640 1 -- 172.21.15.46:0/2191694866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888032660 con 0x7f78940b7e70 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 -- 172.21.15.46:0/2191694866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7e70 msgr2=0x7f78940b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 --2- 172.21.15.46:0/2191694866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7e70 0x7f78940b92a0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f78880097a0 tx=0x7f788802a100 comp rx=0 tx=0).stop 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 -- 172.21.15.46:0/2191694866 shutdown_connections 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 --2- 172.21.15.46:0/2191694866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f78940b7e70 0x7f78940b92a0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 -- 172.21.15.46:0/2191694866 >> 172.21.15.46:0/2191694866 conn(0x7f789408d390 msgr2=0x7f789408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 -- 172.21.15.46:0/2191694866 shutdown_connections 2024-05-09T03:18:00.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.221+0000 7f789c043640 1 -- 172.21.15.46:0/2191694866 wait complete. 2024-05-09T03:18:00.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.223+0000 7f789c043640 1 Processor -- start 2024-05-09T03:18:00.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.223+0000 7f789c043640 1 -- start start 2024-05-09T03:18:00.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.224+0000 7f789c043640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d2e90 0x7f78940cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.224+0000 7f789c043640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78941605b0 0x7f789414e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.224+0000 7f789c043640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 0x7f789414d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.224+0000 7f789c043640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78940b4d80 con 0x7f789414eed0 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.224+0000 7f789c043640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f78940b4c00 con 0x7f78941605b0 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.224+0000 7f789c043640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f78940b4fc0 con 0x7f78940d2e90 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 0x7f789414d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 0x7f789414d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36512/0 (socket says 172.21.15.46:36512) 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 -- 172.21.15.46:0/3862583635 learned_addr learned my addr 172.21.15.46:0/3862583635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f7899ddf640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d2e90 0x7f78940cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f78995de640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78941605b0 0x7f789414e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 -- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d2e90 msgr2=0x7f78940cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d2e90 0x7f78940cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 -- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78941605b0 msgr2=0x7f789414e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78941605b0 0x7f789414e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 -- 172.21.15.46:0/3862583635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789414f2e0 con 0x7f789414eed0 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789a5e0640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 0x7f789414d760 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b440 tx=0x7f788c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c015020 con 0x7f789414eed0 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788c011830 con 0x7f789414eed0 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78940cf7d0 con 0x7f789414eed0 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.225+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c011c20 con 0x7f789414eed0 2024-05-09T03:18:00.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.226+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f789414cc80 con 0x7f789414eed0 2024-05-09T03:18:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.226+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78940cf540 con 0x7f789414eed0 2024-05-09T03:18:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.227+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f788c018020 con 0x7f789414eed0 2024-05-09T03:18:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.228+0000 7f78927fc640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7880076300 0x7f7880078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.228+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7f788c097760 con 0x7f789414eed0 2024-05-09T03:18:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.228+0000 7f7899ddf640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7880076300 0x7f7880078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:00.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.229+0000 7f7899ddf640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7880076300 0x7f7880078810 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f78840012e0 tx=0x7f7884009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:00.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.233+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f788c01f030 con 0x7f789414eed0 2024-05-09T03:18:00.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:00.503+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f78940d4360 con 0x7f789414eed0 2024-05-09T03:18:01.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.090+0000 7f78927fc640 1 -- 172.21.15.46:0/3862583635 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v188) ==== 101+0+0 (secure 0 0 0) 0x7f789414f2e0 con 0x7f789414eed0 2024-05-09T03:18:01.091 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7880076300 msgr2=0x7f7880078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7880076300 0x7f7880078810 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f78840012e0 tx=0x7f7884009040 comp rx=0 tx=0).stop 2024-05-09T03:18:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 msgr2=0x7f789414d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 0x7f789414d760 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b440 tx=0x7f788c00b900 comp rx=0 tx=0).stop 2024-05-09T03:18:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 shutdown_connections 2024-05-09T03:18:01.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7880076300 0x7f7880078810 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f789414eed0 0x7f789414d760 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78941605b0 0x7f789414e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 --2- 172.21.15.46:0/3862583635 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f78940d2e90 0x7f78940cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 >> 172.21.15.46:0/3862583635 conn(0x7f789408d390 msgr2=0x7f78940d0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 shutdown_connections 2024-05-09T03:18:01.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.094+0000 7f789c043640 1 -- 172.21.15.46:0/3862583635 wait complete. 2024-05-09T03:18:01.107 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:01.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.162+0000 7f95c7647f40 1 Processor -- start 2024-05-09T03:18:01.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.163+0000 7f95c7647f40 1 -- start start 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c7647f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc917a3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c7647f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c7647f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c7647f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dc917265f0 con 0x55dc9171d3a0 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c7647f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dc91683f50 con 0x55dc917b1150 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c7647f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55dc917268d0 con 0x55dc917255f0 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc917a3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc917a3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36526/0 (socket says 172.21.15.46:36526) 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 -- 172.21.15.46:0/2860673575 learned_addr learned my addr 172.21.15.46:0/2860673575 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:01.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c1910640 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c090e640 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 -- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917255f0 msgr2=0x55dc917ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 -- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917b1150 msgr2=0x55dc917aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 -- 172.21.15.46:0/2860673575 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc917a8de0 con 0x55dc9171d3a0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c1910640 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.164+0000 7f95c110f640 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc917a3ca0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00b320 tx=0x7f95bc00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0de18bcc70f598 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95bb7fe640 1 -- 172.21.15.46:0/2860673575 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc012020 con 0x55dc9171d3a0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95bb7fe640 1 -- 172.21.15.46:0/2860673575 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc00d6b0 con 0x55dc9171d3a0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95bb7fe640 1 -- 172.21.15.46:0/2860673575 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00daa0 con 0x55dc9171d3a0 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 -- 172.21.15.46:0/2860673575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 msgr2=0x55dc917a3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc917a3ca0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00b320 tx=0x7f95bc00b7e0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 -- 172.21.15.46:0/2860673575 shutdown_connections 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2860673575 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc917a3ca0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 -- 172.21.15.46:0/2860673575 >> 172.21.15.46:0/2860673575 conn(0x55dc915ffac0 msgr2=0x55dc91720820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 -- 172.21.15.46:0/2860673575 shutdown_connections 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.165+0000 7f95c7647f40 1 -- 172.21.15.46:0/2860673575 wait complete. 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.166+0000 7f95c7647f40 1 Processor -- start 2024-05-09T03:18:01.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.166+0000 7f95c7647f40 1 -- start start 2024-05-09T03:18:01.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.167+0000 7f95c7647f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc9179f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.167+0000 7f95c7647f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c7647f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c7647f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dc917af480 con 0x55dc9171d3a0 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c7647f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dc917a2910 con 0x55dc917255f0 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c7647f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55dc917a89b0 con 0x55dc917b1150 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc9179f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc9179f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36540/0 (socket says 172.21.15.46:36540) 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 learned_addr learned my addr 172.21.15.46:0/2192653836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c1910640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917b1150 msgr2=0x55dc917aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917255f0 msgr2=0x55dc917ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc917255f0 0x55dc917ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc917a8c00 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c1910640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc917b1150 0x55dc917aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc9179f760 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00bce0 tx=0x7f95bc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95ba7fc640 1 -- 172.21.15.46:0/2192653836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00e680 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95ba7fc640 1 -- 172.21.15.46:0/2192653836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95bc00e820 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dc9179fca0 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95ba7fc640 1 -- 172.21.15.46:0/2192653836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00ec10 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.168+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc917a90e0 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.170+0000 7f95ba7fc640 1 -- 172.21.15.46:0/2192653836 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95bc015020 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.170+0000 7f95ba7fc640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95a4074cb0 0x7f95a40771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95ba7fc640 1 -- 172.21.15.46:0/2192653836 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7f95bc098830 con 0x55dc9171d3a0 2024-05-09T03:18:01.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95a4074cb0 0x7f95a40771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.174 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55dc917a2300 0x55dc917a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc917aa090 con 0x55dc917a2300 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95c1910640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55dc917a2300 0x55dc917a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95a4074cb0 0x7f95a40771c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x55dc917a10a0 tx=0x7f95ac005a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.171+0000 7f95c1910640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55dc917a2300 0x55dc917a9b50 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.173+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v100'474 uv448 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.173+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91854590 con 0x55dc917a2300 2024-05-09T03:18:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.175+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v100'475 uv475 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.175+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91854930 con 0x55dc917a2300 2024-05-09T03:18:01.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.176+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v100'476 uv476 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.176 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:01.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.176+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855ce0 con 0x55dc917a2300 2024-05-09T03:18:01.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.177+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v100'477 uv477 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.177+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91856080 con 0x55dc917a2300 2024-05-09T03:18:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.178+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v100'478 uv478 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.178+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91854cd0 con 0x55dc917a2300 2024-05-09T03:18:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.179+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v100'479 uv479 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.179+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918550c0 con 0x55dc917a2300 2024-05-09T03:18:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.181+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v100'480 uv454 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.181+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc918554b0 con 0x55dc917a2300 2024-05-09T03:18:01.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.182+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v100'481 uv481 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.182+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918554b0 con 0x55dc917a2300 2024-05-09T03:18:01.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.183+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v100'482 uv482 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.184 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:01.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.183+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.185+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v100'483 uv483 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.185+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.186+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v100'484 uv484 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.186+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.187+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v100'485 uv485 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.187+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.189+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v100'486 uv461 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.189+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.190+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v100'487 uv487 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.190+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.191+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v100'488 uv488 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.192 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:01.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.191+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.193+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v100'489 uv489 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.193+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.194+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v100'490 uv490 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.194+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855460 con 0x55dc917a2300 2024-05-09T03:18:01.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.195+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v100'491 uv491 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.195+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855890 con 0x55dc917a2300 2024-05-09T03:18:01.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.196+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v100'492 uv492 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.196+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855840 con 0x55dc917a2300 2024-05-09T03:18:01.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.198+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v100'493 uv467 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.198+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91855840 con 0x55dc917a2300 2024-05-09T03:18:01.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.199+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v100'494 uv494 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.199+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855840 con 0x55dc917a2300 2024-05-09T03:18:01.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.200+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v100'495 uv495 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.201 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:01.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.200+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855840 con 0x55dc917a2300 2024-05-09T03:18:01.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.201+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v100'496 uv496 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.202+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855840 con 0x55dc917a2300 2024-05-09T03:18:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.203+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v100'497 uv497 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.203+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91855840 con 0x55dc917a2300 2024-05-09T03:18:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.204+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v100'498 uv498 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.204+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc917a2300 2024-05-09T03:18:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.205+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v100'499 uv473 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.205+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc917a2300 2024-05-09T03:18:01.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.206+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v100'500 uv500 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.207+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc917a2300 2024-05-09T03:18:01.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.208+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v100'501 uv501 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.208 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.208+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc917a2300 2024-05-09T03:18:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.209+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v100'502 uv502 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.209+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc917a2300 2024-05-09T03:18:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.210+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v100'503 uv503 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.210+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc917a2300 2024-05-09T03:18:01.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.211+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v100'504 uv504 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.211+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55dc91856930 0x55dc91858d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.211+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918556d0 con 0x55dc91856930 2024-05-09T03:18:01.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.212+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55dc91856930 0x55dc91858d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.212+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55dc91856930 0x55dc91858d50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.213+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v100'349 uv348 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.213+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91859290 con 0x55dc91856930 2024-05-09T03:18:01.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.215+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v100'350 uv350 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.215+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91859630 con 0x55dc91856930 2024-05-09T03:18:01.216 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:01.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.216+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v100'351 uv351 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.216+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91859e80 con 0x55dc91856930 2024-05-09T03:18:01.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.217+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v100'352 uv352 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.217+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a220 con 0x55dc91856930 2024-05-09T03:18:01.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.218+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v100'353 uv353 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.218+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a5c0 con 0x55dc91856930 2024-05-09T03:18:01.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.220+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v100'354 uv336 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.220+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.221+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v100'355 uv355 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.221+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.222+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v100'356 uv356 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.223 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.222+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.223+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v100'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.223+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.224+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v100'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.224+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.225+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v100'359 uv341 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.225+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.227+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v100'360 uv360 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.227+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.228+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v100'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.228 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:01.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.228+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185e3c0 con 0x55dc91856930 2024-05-09T03:18:01.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.229+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v100'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.229+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185e3c0 con 0x55dc91856930 2024-05-09T03:18:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.230+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v100'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.230+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.231+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v100'364 uv347 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.231+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.232+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v100'365 uv365 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.232+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.234 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:01.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.234+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v100'366 uv366 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.234+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185b970 con 0x55dc91856930 2024-05-09T03:18:01.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.235+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v100'367 uv367 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.235+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185b970 con 0x55dc91856930 2024-05-09T03:18:01.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.236+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v100'368 uv368 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.236+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc91856930 2024-05-09T03:18:01.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.237+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v100'369 uv369 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc91856930 2024-05-09T03:18:01.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.237+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc917a2300 2024-05-09T03:18:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.238+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v100'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.238+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc917a2300 2024-05-09T03:18:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.239+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v100'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.239+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc917a2300 2024-05-09T03:18:01.240 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.240+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v100'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.240+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185b970 con 0x55dc917a2300 2024-05-09T03:18:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.241+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v100'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.241+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185b970 con 0x55dc917a2300 2024-05-09T03:18:01.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.242+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v100'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.242+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185a960 con 0x55dc917a2300 2024-05-09T03:18:01.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.243+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v100'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.243+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55dc9185ae70 0x55dc9185d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.243+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185d7f0 con 0x55dc9185ae70 2024-05-09T03:18:01.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.244+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55dc9185ae70 0x55dc9185d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.244+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55dc9185ae70 0x55dc9185d290 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.245+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v100'74 uv73 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.245+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185e0e0 con 0x55dc9185ae70 2024-05-09T03:18:01.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.246+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v100'75 uv75 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.246+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185e480 con 0x55dc9185ae70 2024-05-09T03:18:01.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.247+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v100'76 uv76 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.248 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:01.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.247+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185e820 con 0x55dc9185ae70 2024-05-09T03:18:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.248+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v100'77 uv77 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.248+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185ec30 con 0x55dc917a2300 2024-05-09T03:18:01.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.250+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v100'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.250+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185ec30 con 0x55dc917a2300 2024-05-09T03:18:01.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.251+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v100'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.251+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185ec30 con 0x55dc917a2300 2024-05-09T03:18:01.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.252+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v100'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.253 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:01.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.252+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185fc40 con 0x55dc917a2300 2024-05-09T03:18:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.254+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v100'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.254+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185fc40 con 0x55dc917a2300 2024-05-09T03:18:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.255+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v100'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.255+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185ec30 con 0x55dc917a2300 2024-05-09T03:18:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.256+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v100'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.256+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc9185f500 0x55dc91861920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.256+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91861e60 con 0x55dc9185f500 2024-05-09T03:18:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.257+0000 7f95c1910640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc9185f500 0x55dc91861920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.257+0000 7f95c1910640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc9185f500 0x55dc91861920 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.258+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v100'171 uv165 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.258+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91861e60 con 0x55dc9185f500 2024-05-09T03:18:01.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.259+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v100'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.259+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91862280 con 0x55dc9185f500 2024-05-09T03:18:01.261 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:01.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.261+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v100'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.261+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91863260 con 0x55dc9185f500 2024-05-09T03:18:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.262+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v100'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.262+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91863260 con 0x55dc9185f500 2024-05-09T03:18:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.263+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v100'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.263+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91862250 con 0x55dc9185f500 2024-05-09T03:18:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.264+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v100'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.264+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91862250 con 0x55dc9185f500 2024-05-09T03:18:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.265+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v100'177 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.265+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91862250 con 0x55dc9185f500 2024-05-09T03:18:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.267+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v100'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.267+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91862250 con 0x55dc9185f500 2024-05-09T03:18:01.269 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.268+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v100'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.268+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91862250 con 0x55dc9185f500 2024-05-09T03:18:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.270+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v100'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.270+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91862250 con 0x55dc9185f500 2024-05-09T03:18:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.271+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v100'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.271+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55dc918628d0 0x55dc91864cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.271+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91865310 con 0x55dc918628d0 2024-05-09T03:18:01.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.271+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55dc918628d0 0x55dc91864cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.272+0000 7f95c110f640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55dc918628d0 0x55dc91864cf0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.273+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v100'76 uv75 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc918628d0 2024-05-09T03:18:01.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.273+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc918628d0 2024-05-09T03:18:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.274+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v100'77 uv77 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc918628d0 2024-05-09T03:18:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.274+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc918628d0 2024-05-09T03:18:01.275 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.275+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v100'78 uv78 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc918628d0 2024-05-09T03:18:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.275+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc918628d0 2024-05-09T03:18:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.276+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v100'79 uv79 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc918628d0 2024-05-09T03:18:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.276+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc918628d0 2024-05-09T03:18:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.277+0000 7f95c110f640 1 -- 172.21.15.46:0/2192653836 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v100'80 uv80 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95bc02a020 con 0x55dc918628d0 2024-05-09T03:18:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.277+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc917a2300 2024-05-09T03:18:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.279+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v100'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.279+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc917a2300 2024-05-09T03:18:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.280+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v100'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.280+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc917a2300 2024-05-09T03:18:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.281+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v100'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.281 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.281+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918666f0 con 0x55dc917a2300 2024-05-09T03:18:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.282+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v100'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.282+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918666f0 con 0x55dc917a2300 2024-05-09T03:18:01.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.283+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v100'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.283+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc917a2300 2024-05-09T03:18:01.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.284+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v100'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc917a2300 2024-05-09T03:18:01.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.284+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc918656e0 con 0x55dc9185ae70 2024-05-09T03:18:01.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.285+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v100'152 uv146 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.285+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91865a80 con 0x55dc9185ae70 2024-05-09T03:18:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.286+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v100'153 uv153 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.286+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91865e60 con 0x55dc9185ae70 2024-05-09T03:18:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.287+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v100'154 uv154 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.288 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:01.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.287+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185dcd0 con 0x55dc9185ae70 2024-05-09T03:18:01.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.288+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v100'155 uv155 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.288+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185dcd0 con 0x55dc9185ae70 2024-05-09T03:18:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.289+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v100'156 uv156 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.289+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185dcd0 con 0x55dc9185ae70 2024-05-09T03:18:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.290+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v100'157 uv151 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.290+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc9185dcd0 con 0x55dc9185ae70 2024-05-09T03:18:01.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.291+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v100'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.291+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185dcd0 con 0x55dc9185ae70 2024-05-09T03:18:01.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.292+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v100'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.293 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:01.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.292+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9185dcd0 con 0x55dc9185ae70 2024-05-09T03:18:01.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.293+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v100'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.293+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869090 con 0x55dc9185ae70 2024-05-09T03:18:01.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.294+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v100'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.295+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55dc91866710 0x55dc91869750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.295+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc91866710 2024-05-09T03:18:01.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.295+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55dc91866710 0x55dc91869750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.295+0000 7f95c090e640 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55dc91866710 0x55dc91869750 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.296+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v100'70 uv69 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc91866710 2024-05-09T03:18:01.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.296+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc91866710 2024-05-09T03:18:01.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.297+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v100'71 uv71 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc91866710 2024-05-09T03:18:01.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.297+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc91866710 2024-05-09T03:18:01.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.298+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v100'72 uv72 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc91866710 2024-05-09T03:18:01.299 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:01.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.298+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc91866710 2024-05-09T03:18:01.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.299+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v100'73 uv73 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc91866710 2024-05-09T03:18:01.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.299+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185ae70 2024-05-09T03:18:01.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.300+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v100'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.301+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185ae70 2024-05-09T03:18:01.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.302+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v100'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.302+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185ae70 2024-05-09T03:18:01.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.303+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v100'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.304 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:01.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.303+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185ae70 2024-05-09T03:18:01.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.304+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v100'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.304+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185ae70 2024-05-09T03:18:01.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.306+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v100'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.306+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185ae70 2024-05-09T03:18:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.307+0000 7f95c090e640 1 -- 172.21.15.46:0/2192653836 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v100'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95ac007910 con 0x55dc9185ae70 2024-05-09T03:18:01.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.307+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185f500 2024-05-09T03:18:01.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.308+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v100'79 uv78 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.308+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185f500 2024-05-09T03:18:01.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.310+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v100'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.310+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91868b30 con 0x55dc9185f500 2024-05-09T03:18:01.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.311+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v100'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.311 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.311+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.312+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v100'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.312+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.313+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v100'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.313+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.315+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v100'192 uv186 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.315+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.316+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v100'193 uv193 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.316+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.317+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v100'194 uv194 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.317 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.317+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.318+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v100'195 uv195 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.318+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.319+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v100'196 uv196 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.320+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.320+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v100'197 uv191 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.321+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.322+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v100'198 uv198 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.322+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.323+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v100'199 uv199 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.323 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.323+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9186aca0 con 0x55dc9185f500 2024-05-09T03:18:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.324+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v100'200 uv200 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.324+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc9186aca0 con 0x55dc9185f500 2024-05-09T03:18:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.325+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v100'201 uv201 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.325+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.326+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v100'202 uv202 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.326+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x55dc91869c90 con 0x55dc9185f500 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v100'203 uv203 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f95b40548e0 con 0x55dc9185f500 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55dc917a2300 msgr2=0x55dc917a9b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55dc917a2300 0x55dc917a9b50 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55dc9185ae70 msgr2=0x55dc9185d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55dc9185ae70 0x55dc9185d290 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55dc91866710 msgr2=0x55dc91869750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55dc91866710 0x55dc91869750 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55dc91856930 msgr2=0x55dc91858d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 reap_dead start 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55dc91856930 0x55dc91858d50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc9185f500 msgr2=0x55dc91861920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc9185f500 0x55dc91861920 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55dc918628d0 msgr2=0x55dc91864cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55dc918628d0 0x55dc91864cf0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95a4074cb0 msgr2=0x7f95a40771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95a4074cb0 0x7f95a40771c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x55dc917a10a0 tx=0x7f95ac005a00 comp rx=0 tx=0).stop 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 msgr2=0x55dc9179f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 --2- 172.21.15.46:0/2192653836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc9171d3a0 0x55dc9179f760 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00bce0 tx=0x7f95bc000ed0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c1910640 1 -- 172.21.15.46:0/2192653836 reap_dead start 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 shutdown_connections 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 >> 172.21.15.46:0/2192653836 conn(0x55dc915ffac0 msgr2=0x55dc91722450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 shutdown_connections 2024-05-09T03:18:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.327+0000 7f95c7647f40 1 -- 172.21.15.46:0/2192653836 wait complete. 2024-05-09T03:18:01.331 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 200.00000000 - 2024-05-09T03:18:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.427+0000 7fa02ed66f40 1 Processor -- start 2024-05-09T03:18:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.427+0000 7fa02ed66f40 1 -- start start 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02ed66f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e89a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02ed66f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e00010 0x5575f7e924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02ed66f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e90080 0x5575f7e929e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02ed66f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5575f7d60f50 con 0x5575f7de97b0 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02ed66f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5575f7d64f90 con 0x5575f7e00010 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02ed66f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5575f7dfe4d0 con 0x5575f7e90080 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa028830640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e89a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa028830640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e89a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36554/0 (socket says 172.21.15.46:36554) 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa028830640 1 -- 172.21.15.46:0/4152397032 learned_addr learned my addr 172.21.15.46:0/4152397032 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa02802f640 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e00010 0x5575f7e924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.428+0000 7fa029031640 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e90080 0x5575f7e929e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa028830640 1 -- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e90080 msgr2=0x5575f7e929e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa028830640 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e90080 0x5575f7e929e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa028830640 1 -- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e00010 msgr2=0x5575f7e924a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa028830640 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e00010 0x5575f7e924a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa028830640 1 -- 172.21.15.46:0/4152397032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f7e93a10 con 0x5575f7de97b0 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa028830640 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e89a80 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fa018025bb0 tx=0x7fa018050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9939341490674c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa02702d640 1 -- 172.21.15.46:0/4152397032 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018067070 con 0x5575f7de97b0 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa02702d640 1 -- 172.21.15.46:0/4152397032 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa01804fc50 con 0x5575f7de97b0 2024-05-09T03:18:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa02702d640 1 -- 172.21.15.46:0/4152397032 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018062400 con 0x5575f7de97b0 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.429+0000 7fa02ed66f40 1 -- 172.21.15.46:0/4152397032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 msgr2=0x5575f7e89a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e89a80 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fa018025bb0 tx=0x7fa018050000 comp rx=0 tx=0).stop 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 -- 172.21.15.46:0/4152397032 shutdown_connections 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e90080 0x5575f7e929e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e00010 0x5575f7e924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/4152397032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e89a80 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 -- 172.21.15.46:0/4152397032 >> 172.21.15.46:0/4152397032 conn(0x5575f7cdcac0 msgr2=0x5575f7e01a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 -- 172.21.15.46:0/4152397032 shutdown_connections 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 -- 172.21.15.46:0/4152397032 wait complete. 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.430+0000 7fa02ed66f40 1 Processor -- start 2024-05-09T03:18:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.431+0000 7fa02ed66f40 1 -- start start 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02ed66f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e88c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02ed66f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e00010 0x5575f7db8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02ed66f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e90080 0x5575f7f22230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02ed66f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5575f7de8400 con 0x5575f7de97b0 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02ed66f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5575f7e8a550 con 0x5575f7e90080 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02ed66f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5575f7e93450 con 0x5575f7e00010 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e88c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e88c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36570/0 (socket says 172.21.15.46:36570) 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 -- 172.21.15.46:0/335811774 learned_addr learned my addr 172.21.15.46:0/335811774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa029031640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e90080 0x5575f7f22230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa02802f640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e00010 0x5575f7db8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 -- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e00010 msgr2=0x5575f7db8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5575f7e00010 0x5575f7db8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 -- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e90080 msgr2=0x5575f7f22230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e90080 0x5575f7f22230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 -- 172.21.15.46:0/335811774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f7e92b80 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.432+0000 7fa028830640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e88c00 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fa018025bb0 tx=0x7fa018003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.433+0000 7fa02602b640 1 -- 172.21.15.46:0/335811774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018067040 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.433+0000 7fa02602b640 1 -- 172.21.15.46:0/335811774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa01804fa50 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.433+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5575f7f20740 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.433+0000 7fa02602b640 1 -- 172.21.15.46:0/335811774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018062be0 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.433+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f7e89140 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.434+0000 7fa02602b640 1 -- 172.21.15.46:0/335811774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa01806a070 con 0x5575f7de97b0 2024-05-09T03:18:01.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa02602b640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa008074cb0 0x7fa0080771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa02602b640 1 -- 172.21.15.46:0/335811774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7fa0180db3a0 con 0x5575f7de97b0 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa02802f640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa008074cb0 0x7fa0080771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5575f7db87d0 0x5575f7dfd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x5575f7f22770 con 0x5575f7db87d0 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa029031640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5575f7db87d0 0x5575f7dfd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.435+0000 7fa02802f640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa008074cb0 0x7fa0080771c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x5575f7de7250 tx=0x7fa010001730 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.436+0000 7fa029031640 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5575f7db87d0 0x5575f7dfd330 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa029031640 1 -- 172.21.15.46:0/335811774 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000000 [writefull 0~80] v100'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa02000c720 con 0x5575f7db87d0 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5575f7db87d0 msgr2=0x5575f7dfd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5575f7db87d0 0x5575f7dfd330 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa008074cb0 msgr2=0x7fa0080771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa008074cb0 0x7fa0080771c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x5575f7de7250 tx=0x7fa010001730 comp rx=0 tx=0).stop 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 msgr2=0x5575f7e88c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 --2- 172.21.15.46:0/335811774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575f7de97b0 0x5575f7e88c00 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fa018025bb0 tx=0x7fa018003330 comp rx=0 tx=0).stop 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa029031640 1 -- 172.21.15.46:0/335811774 reap_dead start 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 shutdown_connections 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.437+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 >> 172.21.15.46:0/335811774 conn(0x5575f7cdcac0 msgr2=0x5575f7e922f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:01.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.438+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 shutdown_connections 2024-05-09T03:18:01.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.438+0000 7fa02ed66f40 1 -- 172.21.15.46:0/335811774 wait complete. 2024-05-09T03:18:01.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:01.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.599+0000 7f8f23100640 1 Processor -- start 2024-05-09T03:18:01.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.599+0000 7f8f23100640 1 -- start start 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f23100640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8f1c15d310 0x7f8f1c0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f23100640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b1da0 0x7f8f1c0b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f23100640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f1c0b80f0 con 0x7f8f1c0b7ce0 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f23100640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1c1617c0 con 0x7f8f1c15d310 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f23100640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1c0b2480 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f2169d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8f1c0b7ce0 0x7f8f1c0d29d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49484/0 (socket says 172.21.15.46:49484) 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.600+0000 7f8f2169d640 1 -- 172.21.15.46:0/1891386749 learned_addr learned my addr 172.21.15.46:0/1891386749 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f20e9c640 1 --2- 172.21.15.46:0/1891386749 >> v2:172.21.15.88:3300/0 conn(0x7f8f1c15d310 0x7f8f1c0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f13fff640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b1da0 0x7f8f1c0b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f137fe640 1 -- 172.21.15.46:0/1891386749 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3993477124 0 0) 0x7f8f14001490 con 0x7f8f1c0b7ce0 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f137fe640 1 -- 172.21.15.46:0/1891386749 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f04003520 con 0x7f8f1c0b7ce0 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f20e9c640 1 -- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b1da0 msgr2=0x7f8f1c0b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f20e9c640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b1da0 0x7f8f1c0b77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f20e9c640 1 -- 172.21.15.46:0/1891386749 >> v1:172.21.15.46:6789/0 conn(0x7f8f1c0b7ce0 legacy=0x7f8f1c0d29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f137fe640 1 -- 172.21.15.46:0/1891386749 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 265559839 0 0) 0x7f8f04003520 con 0x7f8f1c0b7ce0 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f20e9c640 1 -- 172.21.15.46:0/1891386749 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c0b2760 con 0x7f8f1c15d310 2024-05-09T03:18:01.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.601+0000 7f8f13fff640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b1da0 0x7f8f1c0b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:01.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.602+0000 7f8f20e9c640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c0b3f70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8f180097a0 tx=0x7f8f1802a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f23b536a60fa39f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.602+0000 7f8f137fe640 1 -- 172.21.15.46:0/1891386749 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18037070 con 0x7f8f1c15d310 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.602+0000 7f8f137fe640 1 -- 172.21.15.46:0/1891386749 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1802ab10 con 0x7f8f1c15d310 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.602+0000 7f8f137fe640 1 -- 172.21.15.46:0/1891386749 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14001c40 con 0x7f8f1c15d310 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 -- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 msgr2=0x7f8f1c0b3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c0b3f70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8f180097a0 tx=0x7f8f1802a080 comp rx=0 tx=0).stop 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 -- 172.21.15.46:0/1891386749 shutdown_connections 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b1da0 0x7f8f1c0b77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 --2- 172.21.15.46:0/1891386749 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c0b3f70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 -- 172.21.15.46:0/1891386749 >> 172.21.15.46:0/1891386749 conn(0x7f8f1c09b7d0 msgr2=0x7f8f1c09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 -- 172.21.15.46:0/1891386749 shutdown_connections 2024-05-09T03:18:01.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.603+0000 7f8f23100640 1 -- 172.21.15.46:0/1891386749 wait complete. 2024-05-09T03:18:01.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.604+0000 7f8f23100640 1 Processor -- start 2024-05-09T03:18:01.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.605+0000 7f8f23100640 1 -- start start 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f23100640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 0x7f8f1c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f23100640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b7ce0 0x7f8f1c147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f23100640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f23100640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c0cfc80 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f23100640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c161aa0 con 0x7f8f1c15d310 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f23100640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1c0cfe00 con 0x7f8f1c0b7ce0 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 0x7f8f1c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 0x7f8f1c0c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36578/0 (socket says 172.21.15.46:36578) 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 -- 172.21.15.46:0/366004368 learned_addr learned my addr 172.21.15.46:0/366004368 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f2169d640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f13fff640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b7ce0 0x7f8f1c147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 -- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b7ce0 msgr2=0x7f8f1c147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b7ce0 0x7f8f1c147e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 -- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 msgr2=0x7f8f1c148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.606+0000 7f8f20e9c640 1 -- 172.21.15.46:0/366004368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c146860 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.607+0000 7f8f20e9c640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 0x7f8f1c0c6920 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8f18035040 tx=0x7f8f180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.607+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18041020 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.607+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1803d620 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.607+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18037040 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.607+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f1c146d60 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.607+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f1c147320 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.608+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f1c0b2fd0 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.609+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8f18044020 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.609+0000 7f8f117fa640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f080762d0 0x7f8f080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.609+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 7143+0+0 (secure 0 0 0) 0x7f8f180b7f40 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.609+0000 7f8f13fff640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f080762d0 0x7f8f080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.610+0000 7f8f13fff640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f080762d0 0x7f8f080787e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c146730 tx=0x7f8f0c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:01.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.614+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8f1c0b2fd0 con 0x7f8f1c0b1da0 2024-05-09T03:18:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:01.848+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f8f1c06a270 con 0x7f8f1c0b1da0 2024-05-09T03:18:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.097+0000 7f8f117fa640 1 -- 172.21.15.46:0/366004368 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v189) ==== 162+0+0 (secure 0 0 0) 0x7f8f1804b030 con 0x7f8f1c0b1da0 2024-05-09T03:18:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f080762d0 msgr2=0x7f8f080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f080762d0 0x7f8f080787e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c146730 tx=0x7f8f0c008040 comp rx=0 tx=0).stop 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 msgr2=0x7f8f1c0c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 0x7f8f1c0c6920 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8f18035040 tx=0x7f8f180046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 shutdown_connections 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f1c15d310 0x7f8f1c148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8f1c0b7ce0 0x7f8f1c147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8f080762d0 0x7f8f080787e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 --2- 172.21.15.46:0/366004368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f1c0b1da0 0x7f8f1c0c6920 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 >> 172.21.15.46:0/366004368 conn(0x7f8f1c09b7d0 msgr2=0x7f8f1c0b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 shutdown_connections 2024-05-09T03:18:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.100+0000 7f8f23100640 1 -- 172.21.15.46:0/366004368 wait complete. 2024-05-09T03:18:02.111 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:02.111+0000 7f943b16b640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (22) Invalid argument 2024-05-09T03:18:02.111 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:02.111+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:02.112 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:02.111+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:02.113 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:02.116 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:18:02.111+0000 7f943b16b640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (22) Invalid argument 2024-05-09T03:18:02.171 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.230+0000 7ff39d95e640 1 Processor -- start 2024-05-09T03:18:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.230+0000 7ff39d95e640 1 -- start start 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.231+0000 7ff39d95e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3980d57a0 0x7ff398160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.231+0000 7ff39d95e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3980b4af0 0x7ff3980b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.231+0000 7ff39d95e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3980d49e0 con 0x7ff3980b77e0 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.231+0000 7ff39d95e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff3980d5bb0 con 0x7ff3980d57a0 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.231+0000 7ff39d95e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff3980d4cc0 con 0x7ff3980b4af0 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3980d57a0 0x7ff398160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3980d57a0 0x7ff398160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36602/0 (socket says 172.21.15.46:36602) 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 -- 172.21.15.46:0/3015787924 learned_addr learned my addr 172.21.15.46:0/3015787924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff38ffff640 1 --2- 172.21.15.46:0/3015787924 >> v2:172.21.15.88:3300/0 conn(0x7ff3980b4af0 0x7ff3980b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 -- 172.21.15.46:0/3015787924 >> v2:172.21.15.88:3300/0 conn(0x7ff3980b4af0 msgr2=0x7ff3980b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 --2- 172.21.15.46:0/3015787924 >> v2:172.21.15.88:3300/0 conn(0x7ff3980b4af0 0x7ff3980b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 -- 172.21.15.46:0/3015787924 >> v1:172.21.15.88:6789/0 conn(0x7ff3980b77e0 legacy=0x7ff3980b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.232+0000 7ff396ffd640 1 -- 172.21.15.46:0/3015787924 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3980d4fa0 con 0x7ff3980d57a0 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.233+0000 7ff396ffd640 1 --2- 172.21.15.46:0/3015787924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980d57a0 0x7ff398160610 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff3980d2f80 tx=0x7ff38802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d3a0c8e27509e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.233+0000 7ff3967fc640 1 -- 172.21.15.46:0/3015787924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388037070 con 0x7ff3980d57a0 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.233+0000 7ff3967fc640 1 -- 172.21.15.46:0/3015787924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff388032440 con 0x7ff3980d57a0 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.233+0000 7ff3967fc640 1 -- 172.21.15.46:0/3015787924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388032810 con 0x7ff3980d57a0 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 -- 172.21.15.46:0/3015787924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980d57a0 msgr2=0x7ff398160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 --2- 172.21.15.46:0/3015787924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980d57a0 0x7ff398160610 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7ff3980d2f80 tx=0x7ff38802a270 comp rx=0 tx=0).stop 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 -- 172.21.15.46:0/3015787924 shutdown_connections 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 --2- 172.21.15.46:0/3015787924 >> v2:172.21.15.88:3300/0 conn(0x7ff3980b4af0 0x7ff3980b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 --2- 172.21.15.46:0/3015787924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980d57a0 0x7ff398160610 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 -- 172.21.15.46:0/3015787924 >> 172.21.15.46:0/3015787924 conn(0x7ff3980743e0 msgr2=0x7ff3980d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:02.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 -- 172.21.15.46:0/3015787924 shutdown_connections 2024-05-09T03:18:02.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.234+0000 7ff39d95e640 1 -- 172.21.15.46:0/3015787924 wait complete. 2024-05-09T03:18:02.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.235+0000 7ff39d95e640 1 Processor -- start 2024-05-09T03:18:02.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.235+0000 7ff39d95e640 1 -- start start 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff39d95e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3980b4af0 0x7ff3980ced20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff39d95e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 0x7ff39815b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff39d95e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3980d57a0 0x7ff3980d0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff39d95e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff3980d1920 con 0x7ff3980b77e0 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff39d95e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff3980d15b0 con 0x7ff3980d57a0 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff39d95e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff3980cd920 con 0x7ff3980b4af0 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 0x7ff39815b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 0x7ff39815b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36606/0 (socket says 172.21.15.46:36606) 2024-05-09T03:18:02.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 -- 172.21.15.46:0/2020927717 learned_addr learned my addr 172.21.15.46:0/2020927717 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff396ffd640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3980b4af0 0x7ff3980ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff3977fe640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3980d57a0 0x7ff3980d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 -- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3980b4af0 msgr2=0x7ff3980ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3980b4af0 0x7ff3980ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 -- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3980d57a0 msgr2=0x7ff3980d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3980d57a0 0x7ff3980d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 -- 172.21.15.46:0/2020927717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3980cb150 con 0x7ff3980b77e0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff3977fe640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3980d57a0 0x7ff3980d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.237+0000 7ff38ffff640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 0x7ff39815b550 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e8a0 tx=0x7ff38000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.238+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380017070 con 0x7ff3980b77e0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.238+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38000c940 con 0x7ff3980b77e0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.238+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38000cd30 con 0x7ff3980b77e0 2024-05-09T03:18:02.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.238+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3980ca8b0 con 0x7ff3980b77e0 2024-05-09T03:18:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.238+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3980cae70 con 0x7ff3980b77e0 2024-05-09T03:18:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.239+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff39806a270 con 0x7ff3980b77e0 2024-05-09T03:18:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.240+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff38001a070 con 0x7ff3980b77e0 2024-05-09T03:18:02.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.240+0000 7ff394ff9640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3780762d0 0x7ff3780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.240+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7ff380097a20 con 0x7ff3980b77e0 2024-05-09T03:18:02.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.241+0000 7ff396ffd640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3780762d0 0x7ff3780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.241+0000 7ff396ffd640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3780762d0 0x7ff3780787e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff388034000 tx=0x7ff38802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:02.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.245+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff380060740 con 0x7ff3980b77e0 2024-05-09T03:18:02.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.476+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3980b7ea0 con 0x7ff3980b77e0 2024-05-09T03:18:02.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.476+0000 7ff394ff9640 1 -- 172.21.15.46:0/2020927717 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+4343 (secure 0 0 0) 0x7ff380064a40 con 0x7ff3980b77e0 2024-05-09T03:18:02.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:02.477 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":191,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:02.112140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:02.478 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 191 2024-05-09T03:18:02.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3780762d0 msgr2=0x7ff3780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3780762d0 0x7ff3780787e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff388034000 tx=0x7ff38802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:02.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 msgr2=0x7ff39815b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 0x7ff39815b550 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7ff38000e8a0 tx=0x7ff38000ed60 comp rx=0 tx=0).stop 2024-05-09T03:18:02.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 shutdown_connections 2024-05-09T03:18:02.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3980d57a0 0x7ff3980d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3780762d0 0x7ff3780787e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3980b77e0 0x7ff39815b550 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 --2- 172.21.15.46:0/2020927717 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3980b4af0 0x7ff3980ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 >> 172.21.15.46:0/2020927717 conn(0x7ff3980743e0 msgr2=0x7ff39815f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:02.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 shutdown_connections 2024-05-09T03:18:02.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.479+0000 7ff39d95e640 1 -- 172.21.15.46:0/2020927717 wait complete. 2024-05-09T03:18:02.491 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:02.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.645+0000 7f00ae029640 1 Processor -- start 2024-05-09T03:18:02.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.645+0000 7f00ae029640 1 -- start start 2024-05-09T03:18:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.646+0000 7f00ae029640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00a80b5210 0x7f00a81613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.646+0000 7f00ae029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 0x7f00a80d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.646+0000 7f00ae029640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a80d5280 con 0x7f00a80b77c0 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.646+0000 7f00ae029640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f00a80b5620 con 0x7f00a80b5210 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.646+0000 7f00ae029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00a81609a0 con 0x7f00a80d4800 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f00a80b77c0 0x7f00a80b7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49510/0 (socket says 172.21.15.46:49510) 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a7fff640 1 -- 172.21.15.46:0/705239661 learned_addr learned my addr 172.21.15.46:0/705239661 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234032980 0 0) 0x7f00a80d5280 con 0x7f00a80b77c0 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 0x7f00a80d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a77fe640 1 --2- 172.21.15.46:0/705239661 >> v2:172.21.15.88:3300/0 conn(0x7f00a80b5210 0x7f00a81613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0090003520 con 0x7f00a80b77c0 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2258717776 0 0) 0x7f0090003520 con 0x7f00a80b77c0 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 msgr2=0x7f00a80d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 0x7f00a80d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 >> v2:172.21.15.88:3300/0 conn(0x7f00a80b5210 msgr2=0x7f00a81613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 --2- 172.21.15.46:0/705239661 >> v2:172.21.15.88:3300/0 conn(0x7f00a80b5210 0x7f00a81613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a80d5560 con 0x7f00a80b77c0 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f00a0001a70 con 0x7f00a80b77c0 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 >> v1:172.21.15.46:6789/0 conn(0x7f00a80b77c0 legacy=0x7f00a80b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0090003a00 0x7f0090003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0090004800 0x7f0090004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 0x7f0090005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00a80b5620 con 0x7f0090004800 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00a81609a0 con 0x7f0090003a00 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f009000baf0 con 0x7f00a80d4800 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0090004800 0x7f0090004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a77fe640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0090003a00 0x7f0090003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.647+0000 7f00a7fff640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 0x7f0090005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.648+0000 7f00a6ffd640 1 -- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 msgr2=0x7f0090005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.648+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a80d4800 0x7f0090005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.648+0000 7f00a6ffd640 1 -- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0090003a00 msgr2=0x7f0090003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.648+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0090003a00 0x7f0090003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.648+0000 7f00a6ffd640 1 -- 172.21.15.46:0/705239661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0090003520 con 0x7f0090004800 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.648+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0090004800 0x7f0090004c10 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f009c009ac0 tx=0x7f0090005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57d7ba42c8db268 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c036070 con 0x7f0090004800 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c031400 con 0x7f0090004800 2024-05-09T03:18:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00a67fc640 1 -- 172.21.15.46:0/705239661 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c009690 con 0x7f0090004800 2024-05-09T03:18:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00ae029640 1 -- 172.21.15.46:0/705239661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0090004800 msgr2=0x7f0090004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00ae029640 1 --2- 172.21.15.46:0/705239661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0090004800 0x7f0090004c10 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f009c009ac0 tx=0x7f0090005830 comp rx=0 tx=0).stop 2024-05-09T03:18:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00a7fff640 1 -- 172.21.15.46:0/705239661 reap_dead start 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00ae029640 1 -- 172.21.15.46:0/705239661 shutdown_connections 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00ae029640 1 -- 172.21.15.46:0/705239661 >> 172.21.15.46:0/705239661 conn(0x7f00a80d29a0 msgr2=0x7f00a80d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00ae029640 1 -- 172.21.15.46:0/705239661 shutdown_connections 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.649+0000 7f00ae029640 1 -- 172.21.15.46:0/705239661 wait complete. 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.651+0000 7f00ae029640 1 Processor -- start 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.651+0000 7f00ae029640 1 -- start start 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00ae029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a81fa3b0 0x7f00a80cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00ae029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a814b850 0x7f00a80ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00ae029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 0x7f00a814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00ae029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00a80d0ae0 con 0x7f00a80cd330 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00ae029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00a8160ea0 con 0x7f00a81fa3b0 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00ae029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00a80d4c10 con 0x7f00a814b850 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 0x7f00a814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 0x7f00a814e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36630/0 (socket says 172.21.15.46:36630) 2024-05-09T03:18:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 -- 172.21.15.46:0/1543112573 learned_addr learned my addr 172.21.15.46:0/1543112573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a77fe640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a81fa3b0 0x7f00a80cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a814b850 0x7f00a80ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 -- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a814b850 msgr2=0x7f00a80ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a814b850 0x7f00a80ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 -- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a81fa3b0 msgr2=0x7f00a80cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a81fa3b0 0x7f00a80cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 -- 172.21.15.46:0/1543112573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a80cd740 con 0x7f00a80cd330 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a77fe640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a81fa3b0 0x7f00a80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a6ffd640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a814b850 0x7f00a80ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.653+0000 7f00a7fff640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 0x7f00a814e1f0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f00a806f050 tx=0x7f00a000a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.654+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a00133f0 con 0x7f00a80cd330 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.654+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a0018070 con 0x7f00a80cd330 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.654+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a80cf590 con 0x7f00a80cd330 2024-05-09T03:18:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.654+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0013810 con 0x7f00a80cd330 2024-05-09T03:18:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.654+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a80b92e0 con 0x7f00a80cd330 2024-05-09T03:18:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.655+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a806a270 con 0x7f00a80cd330 2024-05-09T03:18:02.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.655+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00a00139b0 con 0x7f00a80cd330 2024-05-09T03:18:02.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.656+0000 7f008ffff640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00940762d0 0x7f00940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:02.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.656+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f00a0098510 con 0x7f00a80cd330 2024-05-09T03:18:02.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.656+0000 7f00a77fe640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00940762d0 0x7f00940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:02.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.657+0000 7f00a77fe640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00940762d0 0x7f00940787e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f00a8164930 tx=0x7f0098008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:02.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.661+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f00a80cd740 con 0x7f00a80cd330 2024-05-09T03:18:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.889+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00a80d40d0 con 0x7f00a80cd330 2024-05-09T03:18:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.890+0000 7f008ffff640 1 -- 172.21.15.46:0/1543112573 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+4343 (secure 0 0 0) 0x7f00a00617e0 con 0x7f00a80cd330 2024-05-09T03:18:02.891 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:02.891 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":191,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:02.112140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:02.891 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 191 2024-05-09T03:18:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00940762d0 msgr2=0x7f00940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00940762d0 0x7f00940787e0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f00a8164930 tx=0x7f0098008040 comp rx=0 tx=0).stop 2024-05-09T03:18:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 msgr2=0x7f00a814e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 0x7f00a814e1f0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f00a806f050 tx=0x7f00a000a730 comp rx=0 tx=0).stop 2024-05-09T03:18:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 shutdown_connections 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00940762d0 0x7f00940787e0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00a80cd330 0x7f00a814e1f0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00a814b850 0x7f00a80ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 --2- 172.21.15.46:0/1543112573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a81fa3b0 0x7f00a80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 >> 172.21.15.46:0/1543112573 conn(0x7f00a80d29a0 msgr2=0x7f00a8163df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 shutdown_connections 2024-05-09T03:18:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:02.892+0000 7f00ae029640 1 -- 172.21.15.46:0/1543112573 wait complete. 2024-05-09T03:18:02.904 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:02.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.018+0000 7f22b4d72640 1 Processor -- start 2024-05-09T03:18:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.018+0000 7f22b4d72640 1 -- start start 2024-05-09T03:18:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22b4d72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22b00b5210 0x7f22b01613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22b4d72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b00b7ed0 0x7f22b00d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22b4d72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b00d5280 con 0x7f22b00d4800 2024-05-09T03:18:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22b4d72640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f22b00b5620 con 0x7f22b00b5210 2024-05-09T03:18:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22b4d72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f22b01609a0 con 0x7f22b00b7ed0 2024-05-09T03:18:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22add9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f22b00d4800 0x7f22b00b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49520/0 (socket says 172.21.15.46:49520) 2024-05-09T03:18:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22add9b640 1 -- 172.21.15.46:0/3641491453 learned_addr learned my addr 172.21.15.46:0/3641491453 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22aed9d640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b00b7ed0 0x7f22b00d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3641491453 >> v2:172.21.15.88:3300/0 conn(0x7f22b00b5210 0x7f22b01613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566505299 0 0) 0x7f22b00d5280 con 0x7f22b00d4800 2024-05-09T03:18:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2298003520 con 0x7f22b00d4800 2024-05-09T03:18:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3799127294 0 0) 0x7f2298003520 con 0x7f22b00d4800 2024-05-09T03:18:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b00b7ed0 msgr2=0x7f22b00d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b00b7ed0 0x7f22b00d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 >> v2:172.21.15.88:3300/0 conn(0x7f22b00b5210 msgr2=0x7f22b01613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 --2- 172.21.15.46:0/3641491453 >> v2:172.21.15.88:3300/0 conn(0x7f22b00b5210 0x7f22b01613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b00d5560 con 0x7f22b00d4800 2024-05-09T03:18:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f22a40016f0 con 0x7f22b00d4800 2024-05-09T03:18:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3641491453 >> v2:172.21.15.88:3300/0 conn(0x7f22b00b5210 0x7f22b01613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 >> v1:172.21.15.46:6789/0 conn(0x7f22b00d4800 legacy=0x7f22b00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2298003a00 0x7f2298003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.020+0000 7f22ad59a640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2298004800 0x7f2298004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22ad59a640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298005150 0x7f229800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22b00b5620 con 0x7f2298004800 2024-05-09T03:18:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f22b01609a0 con 0x7f2298005150 2024-05-09T03:18:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f229800bf50 con 0x7f2298003a00 2024-05-09T03:18:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22add9b640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2298004800 0x7f2298004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2298003a00 0x7f2298003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22aed9d640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298005150 0x7f229800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22add9b640 1 -- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2298003a00 msgr2=0x7f2298003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22add9b640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2298003a00 0x7f2298003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22add9b640 1 -- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298005150 msgr2=0x7f229800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22add9b640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298005150 0x7f229800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22add9b640 1 -- 172.21.15.46:0/3641491453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2298003520 con 0x7f2298004800 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22aed9d640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298005150 0x7f229800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.021+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2298003a00 0x7f2298003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.022+0000 7f22add9b640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2298004800 0x7f2298004c10 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f22a400c290 tx=0x7f2298005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caaa3b5a31979663 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.022+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4007d50 con 0x7f2298004800 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.022+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a4005af0 con 0x7f2298004800 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.022+0000 7f22ad59a640 1 -- 172.21.15.46:0/3641491453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4007070 con 0x7f2298004800 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22b4d72640 1 -- 172.21.15.46:0/3641491453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2298004800 msgr2=0x7f2298004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3641491453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2298004800 0x7f2298004c10 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f22a400c290 tx=0x7f2298005830 comp rx=0 tx=0).stop 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22aed9d640 1 -- 172.21.15.46:0/3641491453 reap_dead start 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22b4d72640 1 -- 172.21.15.46:0/3641491453 shutdown_connections 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22b4d72640 1 -- 172.21.15.46:0/3641491453 >> 172.21.15.46:0/3641491453 conn(0x7f22b00d29a0 msgr2=0x7f22b00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22b4d72640 1 -- 172.21.15.46:0/3641491453 shutdown_connections 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.023+0000 7f22b4d72640 1 -- 172.21.15.46:0/3641491453 wait complete. 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.025+0000 7f22b4d72640 1 Processor -- start 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.025+0000 7f22b4d72640 1 -- start start 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22b4d72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298004800 0x7f22b00b9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22b4d72640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b0157d20 0x7f22b00cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22b4d72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 0x7f22b00ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22b4d72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22b00d0ae0 con 0x7f22b0158130 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22b4d72640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0160ea0 con 0x7f2298004800 2024-05-09T03:18:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22b4d72640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f22b00b7ed0 con 0x7f22b0157d20 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 0x7f22b00ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 0x7f22b00ca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36648/0 (socket says 172.21.15.46:36648) 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 -- 172.21.15.46:0/3419016043 learned_addr learned my addr 172.21.15.46:0/3419016043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298004800 0x7f22b00b9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22add9b640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b0157d20 0x7f22b00cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 -- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b0157d20 msgr2=0x7f22b00cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b0157d20 0x7f22b00cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 -- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298004800 msgr2=0x7f22b00b9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298004800 0x7f22b00b9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.026+0000 7f22aed9d640 1 -- 172.21.15.46:0/3419016043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b00bacb0 con 0x7f22b0158130 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f22add9b640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b0157d20 0x7f22b00cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f22aed9d640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 0x7f22b00ca440 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f22b00bb9f0 tx=0x7f22a800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0009860 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a8003990 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b00bb6e0 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8009c10 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.027+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b00bb400 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.028+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b00bae80 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.029+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f22a8015430 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.029+0000 7f229effd640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2290076300 0x7f2290078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.029+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f22a8098350 con 0x7f22b0158130 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.029+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2290076300 0x7f2290078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.030+0000 7f22ae59c640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2290076300 0x7f2290078810 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009ef0 tx=0x7f22a0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.034+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f22b00bae80 con 0x7f22b0158130 2024-05-09T03:18:03.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.265+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22b006a270 con 0x7f22b0158130 2024-05-09T03:18:03.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.265+0000 7f229effd640 1 -- 172.21.15.46:0/3419016043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+5166 (secure 0 0 0) 0x7f22a8019030 con 0x7f22b0158130 2024-05-09T03:18:03.266 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:03.266 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":192,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/436596417","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":436596417},{"type":"v1","addr":"172.21.15.46:6834","nonce":436596417}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:02.112140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:03.267 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 192 2024-05-09T03:18:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2290076300 msgr2=0x7f2290078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2290076300 0x7f2290078810 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f22a0009ef0 tx=0x7f22a0009190 comp rx=0 tx=0).stop 2024-05-09T03:18:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 msgr2=0x7f22b00ca440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 0x7f22b00ca440 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f22b00bb9f0 tx=0x7f22a800f580 comp rx=0 tx=0).stop 2024-05-09T03:18:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 shutdown_connections 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2290076300 0x7f2290078810 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22b0158130 0x7f22b00ca440 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f22b0157d20 0x7f22b00cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 --2- 172.21.15.46:0/3419016043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2298004800 0x7f22b00b9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 >> 172.21.15.46:0/3419016043 conn(0x7f22b00d29a0 msgr2=0x7f22b015ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 shutdown_connections 2024-05-09T03:18:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.268+0000 7f22b4d72640 1 -- 172.21.15.46:0/3419016043 wait complete. 2024-05-09T03:18:03.280 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 200.00000000' led to DAMAGED state 2024-05-09T03:18:03.280 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 200.00000001' 2024-05-09T03:18:03.280 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:03.280 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:03.296 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:03.297 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:03.297 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:03.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:03.297 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:03.352 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:03.353 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.507+0000 7f426a1dc640 1 Processor -- start 2024-05-09T03:18:03.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.507+0000 7f426a1dc640 1 -- start start 2024-05-09T03:18:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f426a1dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f42641642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f426a1dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f42640d2a70 0x7f42640d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f426a1dc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42640d33c0 con 0x7f42640bb770 2024-05-09T03:18:03.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f426a1dc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f42640d4bb0 con 0x7f42640d2a70 2024-05-09T03:18:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f426a1dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4264164900 con 0x7f4264163eb0 2024-05-09T03:18:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f4262ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f42641642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f4263fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f42640d2a70 0x7f42640d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f4262ffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f42641642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:41080/0 (socket says 172.21.15.46:41080) 2024-05-09T03:18:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f4262ffd640 1 -- 172.21.15.46:0/1106408731 learned_addr learned my addr 172.21.15.46:0/1106408731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:03.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.508+0000 7f42637fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f42640bb770 0x7f42640d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:33280/0 (socket says 172.21.15.46:33280) 2024-05-09T03:18:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.509+0000 7f4262ffd640 1 -- 172.21.15.46:0/1106408731 >> v2:172.21.15.88:3300/0 conn(0x7f42640d2a70 msgr2=0x7f42640d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.509+0000 7f4262ffd640 1 --2- 172.21.15.46:0/1106408731 >> v2:172.21.15.88:3300/0 conn(0x7f42640d2a70 0x7f42640d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.509+0000 7f4262ffd640 1 -- 172.21.15.46:0/1106408731 >> v1:172.21.15.88:6789/0 conn(0x7f42640bb770 legacy=0x7f42640d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.509+0000 7f4262ffd640 1 -- 172.21.15.46:0/1106408731 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4264164be0 con 0x7f4264163eb0 2024-05-09T03:18:03.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.510+0000 7f4262ffd640 1 --2- 172.21.15.46:0/1106408731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f42641642c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4258009870 tx=0x7f425802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c78c1f0c6282a96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.510+0000 7f42627fc640 1 -- 172.21.15.46:0/1106408731 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4258037070 con 0x7f4264163eb0 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.510+0000 7f42627fc640 1 -- 172.21.15.46:0/1106408731 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4258032440 con 0x7f4264163eb0 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.510+0000 7f42627fc640 1 -- 172.21.15.46:0/1106408731 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4258032810 con 0x7f4264163eb0 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.510+0000 7f426a1dc640 1 -- 172.21.15.46:0/1106408731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 msgr2=0x7f42641642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.510+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1106408731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f42641642c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4258009870 tx=0x7f425802a270 comp rx=0 tx=0).stop 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.511+0000 7f426a1dc640 1 -- 172.21.15.46:0/1106408731 shutdown_connections 2024-05-09T03:18:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.511+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1106408731 >> v2:172.21.15.88:3300/0 conn(0x7f42640d2a70 0x7f42640d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.511+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1106408731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f42641642c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.511+0000 7f426a1dc640 1 -- 172.21.15.46:0/1106408731 >> 172.21.15.46:0/1106408731 conn(0x7f426406e370 msgr2=0x7f426406e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.511+0000 7f426a1dc640 1 -- 172.21.15.46:0/1106408731 shutdown_connections 2024-05-09T03:18:03.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.511+0000 7f426a1dc640 1 -- 172.21.15.46:0/1106408731 wait complete. 2024-05-09T03:18:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.512+0000 7f426a1dc640 1 Processor -- start 2024-05-09T03:18:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.513+0000 7f426a1dc640 1 -- start start 2024-05-09T03:18:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f426a1dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42640bb770 0x7f426415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f426a1dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 0x7f4264148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f426a1dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f426414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f426a1dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42640d0bc0 con 0x7f42640d2a70 2024-05-09T03:18:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f426a1dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f42640bbb80 con 0x7f42640bb770 2024-05-09T03:18:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f426a1dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f42640d0d40 con 0x7f4264163eb0 2024-05-09T03:18:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f4262ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 0x7f4264148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f4262ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 0x7f4264148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36656/0 (socket says 172.21.15.46:36656) 2024-05-09T03:18:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f4262ffd640 1 -- 172.21.15.46:0/1812441671 learned_addr learned my addr 172.21.15.46:0/1812441671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f4263fff640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f426414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f42637fe640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42640bb770 0x7f426415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.514+0000 7f4262ffd640 1 -- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 msgr2=0x7f426414b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f4262ffd640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f426414b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f4262ffd640 1 -- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42640bb770 msgr2=0x7f426415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:03.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f4262ffd640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42640bb770 0x7f426415d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f4262ffd640 1 -- 172.21.15.46:0/1812441671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42641ec830 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f4263fff640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f426414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f4262ffd640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 0x7f4264148d40 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f425802a2b0 tx=0x7f42580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4258037040 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4258042070 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42641497c0 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4258030880 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.515+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f426414b320 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.516+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42641ec5a0 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.517+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4258030a20 con 0x7f42640d2a70 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.517+0000 7f424bfff640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42500762d0 0x7f42500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:03.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.517+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f42580b8450 con 0x7f42640d2a70 2024-05-09T03:18:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.518+0000 7f42637fe640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42500762d0 0x7f42500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.518+0000 7f42637fe640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42500762d0 0x7f42500787e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4254000f60 tx=0x7f4254000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:03.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.522+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4258081060 con 0x7f42640d2a70 2024-05-09T03:18:03.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:03.750+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f426406a270 con 0x7f42640d2a70 2024-05-09T03:18:04.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.122+0000 7f424bfff640 1 -- 172.21.15.46:0/1812441671 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v193) ==== 161+0+0 (secure 0 0 0) 0x7f4258085360 con 0x7f42640d2a70 2024-05-09T03:18:04.124 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42500762d0 msgr2=0x7f42500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42500762d0 0x7f42500787e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4254000f60 tx=0x7f4254000bc0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 msgr2=0x7f4264148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 0x7f4264148d40 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f425802a2b0 tx=0x7f42580046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 shutdown_connections 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4264163eb0 0x7f426414b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42500762d0 0x7f42500787e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42640d2a70 0x7f4264148d40 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 --2- 172.21.15.46:0/1812441671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42640bb770 0x7f426415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 >> 172.21.15.46:0/1812441671 conn(0x7f426406e370 msgr2=0x7f42640b6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:04.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 shutdown_connections 2024-05-09T03:18:04.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.125+0000 7f426a1dc640 1 -- 172.21.15.46:0/1812441671 wait complete. 2024-05-09T03:18:04.138 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.254+0000 7f38c512d640 1 Processor -- start 2024-05-09T03:18:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.254+0000 7f38c512d640 1 -- start start 2024-05-09T03:18:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38c512d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c00d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38c512d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38c0163eb0 0x7f38c01642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38c512d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38c0164900 con 0x7f38c00d2a70 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38c512d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f38c00d4bb0 con 0x7f38c0163eb0 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38c512d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38c0164be0 con 0x7f38c00bb770 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38bf59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f38c00d2a70 0x7f38c00d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49534/0 (socket says 172.21.15.46:49534) 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38bf59e640 1 -- 172.21.15.46:0/3622830570 learned_addr learned my addr 172.21.15.46:0/3622830570 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.255+0000 7f38be59c640 1 --2- 172.21.15.46:0/3622830570 >> v2:172.21.15.88:3300/0 conn(0x7f38c0163eb0 0x7f38c01642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bed9d640 1 --2- 172.21.15.46:0/3622830570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c00d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bdd9b640 1 -- 172.21.15.46:0/3622830570 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066849364 0 0) 0x7f38c0164900 con 0x7f38c00d2a70 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bdd9b640 1 -- 172.21.15.46:0/3622830570 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38ac003520 con 0x7f38c00d2a70 2024-05-09T03:18:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bed9d640 1 -- 172.21.15.46:0/3622830570 >> v2:172.21.15.88:3300/0 conn(0x7f38c0163eb0 msgr2=0x7f38c01642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bed9d640 1 --2- 172.21.15.46:0/3622830570 >> v2:172.21.15.88:3300/0 conn(0x7f38c0163eb0 0x7f38c01642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bed9d640 1 -- 172.21.15.46:0/3622830570 >> v1:172.21.15.46:6789/0 conn(0x7f38c00d2a70 legacy=0x7f38c00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38bed9d640 1 -- 172.21.15.46:0/3622830570 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c00d4840 con 0x7f38c00bb770 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.256+0000 7f38be59c640 1 --2- 172.21.15.46:0/3622830570 >> v2:172.21.15.88:3300/0 conn(0x7f38c0163eb0 0x7f38c01642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.257+0000 7f38bed9d640 1 --2- 172.21.15.46:0/3622830570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c00d5040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009d90 tx=0x7f38b002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e545469e392d00fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.257+0000 7f38bdd9b640 1 -- 172.21.15.46:0/3622830570 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0007740 con 0x7f38c00bb770 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.257+0000 7f38bdd9b640 1 -- 172.21.15.46:0/3622830570 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b00078e0 con 0x7f38c00bb770 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.257+0000 7f38bdd9b640 1 -- 172.21.15.46:0/3622830570 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0007c90 con 0x7f38c00bb770 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 -- 172.21.15.46:0/3622830570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 msgr2=0x7f38c00d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 --2- 172.21.15.46:0/3622830570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c00d5040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009d90 tx=0x7f38b002a660 comp rx=0 tx=0).stop 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 -- 172.21.15.46:0/3622830570 shutdown_connections 2024-05-09T03:18:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 --2- 172.21.15.46:0/3622830570 >> v2:172.21.15.88:3300/0 conn(0x7f38c0163eb0 0x7f38c01642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 --2- 172.21.15.46:0/3622830570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c00d5040 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 -- 172.21.15.46:0/3622830570 >> 172.21.15.46:0/3622830570 conn(0x7f38c006e370 msgr2=0x7f38c006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 -- 172.21.15.46:0/3622830570 shutdown_connections 2024-05-09T03:18:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.258+0000 7f38c512d640 1 -- 172.21.15.46:0/3622830570 wait complete. 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.260+0000 7f38c512d640 1 Processor -- start 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.260+0000 7f38c512d640 1 -- start start 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38c512d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38c512d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 0x7f38c00ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38c512d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0163eb0 0x7f38c00cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38c512d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f38c00d0bc0 con 0x7f38c00d2a70 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38c512d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f38c00bbb80 con 0x7f38c0163eb0 2024-05-09T03:18:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38c512d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38c00d0d40 con 0x7f38c00bb770 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 0x7f38c00ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 0x7f38c00ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36662/0 (socket says 172.21.15.46:36662) 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38be59c640 1 -- 172.21.15.46:0/2223938392 learned_addr learned my addr 172.21.15.46:0/2223938392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38bed9d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.261+0000 7f38bf59e640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0163eb0 0x7f38c00cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38be59c640 1 -- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 msgr2=0x7f38c0152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38be59c640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c0152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38be59c640 1 -- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0163eb0 msgr2=0x7f38c00cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38be59c640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0163eb0 0x7f38c00cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38be59c640 1 -- 172.21.15.46:0/2223938392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c00cfee0 con 0x7f38c00d2a70 2024-05-09T03:18:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38be59c640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 0x7f38c00ca800 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e900 tx=0x7f38a800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8017070 con 0x7f38c00d2a70 2024-05-09T03:18:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a800ca00 con 0x7f38c00d2a70 2024-05-09T03:18:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c00cd580 con 0x7f38c00d2a70 2024-05-09T03:18:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cdf0 con 0x7f38c00d2a70 2024-05-09T03:18:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.262+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c01525d0 con 0x7f38c00d2a70 2024-05-09T03:18:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.263+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38c00cfc50 con 0x7f38c00d2a70 2024-05-09T03:18:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.264+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f38a801a070 con 0x7f38c00d2a70 2024-05-09T03:18:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.264+0000 7f38b77fe640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38a00762d0 0x7f38a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.265+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f38a8097b80 con 0x7f38c00d2a70 2024-05-09T03:18:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.265+0000 7f38bed9d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38a00762d0 0x7f38a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.265+0000 7f38bed9d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38a00762d0 0x7f38a00787e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f38b0034000 tx=0x7f38b002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.269+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f38a8060e50 con 0x7f38c00d2a70 2024-05-09T03:18:04.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:04.538+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f38c006a270 con 0x7f38c00d2a70 2024-05-09T03:18:05.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.128+0000 7f38b77fe640 1 -- 172.21.15.46:0/2223938392 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v194) ==== 101+0+0 (secure 0 0 0) 0x7f38a8060750 con 0x7f38c00d2a70 2024-05-09T03:18:05.130 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38a00762d0 msgr2=0x7f38a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38a00762d0 0x7f38a00787e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f38b0034000 tx=0x7f38b002f000 comp rx=0 tx=0).stop 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 msgr2=0x7f38c00ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 0x7f38c00ca800 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e900 tx=0x7f38a800edc0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 shutdown_connections 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38c0163eb0 0x7f38c00cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38a00762d0 0x7f38a00787e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38c00d2a70 0x7f38c00ca800 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 --2- 172.21.15.46:0/2223938392 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38c00bb770 0x7f38c0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.131+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 >> 172.21.15.46:0/2223938392 conn(0x7f38c006e370 msgr2=0x7f38c00d2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.132+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 shutdown_connections 2024-05-09T03:18:05.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.132+0000 7f38c512d640 1 -- 172.21.15.46:0/2223938392 wait complete. 2024-05-09T03:18:05.146 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:05.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.206+0000 7f23e9139f40 1 Processor -- start 2024-05-09T03:18:05.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.206+0000 7f23e9139f40 1 -- start start 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e9139f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 0x560490b447e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e9139f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b42030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e9139f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490b43810 0x560490b43e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e9139f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560490ab04d0 con 0x560490a697e0 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e9139f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560490a12f50 con 0x560490a9b7b0 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e9139f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x560490ab58a0 con 0x560490b43810 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 0x560490b447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 0x560490b447e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36688/0 (socket says 172.21.15.46:36688) 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 -- 172.21.15.46:0/2028295841 learned_addr learned my addr 172.21.15.46:0/2028295841 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e3402640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490b43810 0x560490b43e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b42030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 -- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490b43810 msgr2=0x560490b43e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490b43810 0x560490b43e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 -- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 msgr2=0x560490b42030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b42030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 -- 172.21.15.46:0/2028295841 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560490a98dd0 con 0x560490a697e0 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b42030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e3402640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490b43810 0x560490b43e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 0x560490b447e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f23d4001e90 tx=0x7f23d400b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b0908e1f03d18a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.208+0000 7f23e1bff640 1 -- 172.21.15.46:0/2028295841 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4008070 con 0x560490a697e0 2024-05-09T03:18:05.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e1bff640 1 -- 172.21.15.46:0/2028295841 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d400d500 con 0x560490a697e0 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e1bff640 1 -- 172.21.15.46:0/2028295841 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d400d8d0 con 0x560490a697e0 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 -- 172.21.15.46:0/2028295841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 msgr2=0x560490b447e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 0x560490b447e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f23d4001e90 tx=0x7f23d400b6a0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 -- 172.21.15.46:0/2028295841 shutdown_connections 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490b43810 0x560490b43e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b42030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2028295841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490a697e0 0x560490b447e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 -- 172.21.15.46:0/2028295841 >> 172.21.15.46:0/2028295841 conn(0x56049098eac0 msgr2=0x560490ab3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 -- 172.21.15.46:0/2028295841 shutdown_connections 2024-05-09T03:18:05.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.209+0000 7f23e9139f40 1 -- 172.21.15.46:0/2028295841 wait complete. 2024-05-09T03:18:05.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.210+0000 7f23e9139f40 1 Processor -- start 2024-05-09T03:18:05.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.210+0000 7f23e9139f40 1 -- start start 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.211+0000 7f23e9139f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490a697e0 0x560490b39a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.211+0000 7f23e9139f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b3bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.211+0000 7f23e9139f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 0x560490b28850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.211+0000 7f23e9139f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560490b45b30 con 0x560490b43810 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.211+0000 7f23e9139f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560490ab4810 con 0x560490a9b7b0 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.211+0000 7f23e9139f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x560490b42570 con 0x560490a697e0 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 0x560490b28850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 0x560490b28850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36690/0 (socket says 172.21.15.46:36690) 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 learned_addr learned my addr 172.21.15.46:0/2435230016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490a697e0 0x560490b39a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b3bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490a697e0 msgr2=0x560490b39a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x560490a697e0 0x560490b39a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 msgr2=0x560490b3bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560490a9b7b0 0x560490b3bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560490b39f70 con 0x560490b43810 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e3402640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 0x560490b28850 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00f4d0 tx=0x7f23dc00f990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e0bfd640 1 -- 172.21.15.46:0/2435230016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc019020 con 0x560490b43810 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e0bfd640 1 -- 172.21.15.46:0/2435230016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23dc015690 con 0x560490b43810 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560490ab53c0 con 0x560490b43810 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e0bfd640 1 -- 172.21.15.46:0/2435230016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc015a60 con 0x560490b43810 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.212+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560490b32010 con 0x560490b43810 2024-05-09T03:18:05.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.214+0000 7f23e0bfd640 1 -- 172.21.15.46:0/2435230016 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f23dc01c020 con 0x560490b43810 2024-05-09T03:18:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23e0bfd640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23c4074cb0 0x7f23c40771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23e0bfd640 1 -- 172.21.15.46:0/2435230016 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f23dc096e90 con 0x560490b43810 2024-05-09T03:18:05.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23c4074cb0 0x7f23c40771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.217 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:05.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x560490a99ac0 0x560490b34bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be5710 con 0x560490a99ac0 2024-05-09T03:18:05.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x560490a99ac0 0x560490b34bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23c4074cb0 0x7f23c40771c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f23d4001e90 tx=0x7f23d4008000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.215+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x560490a99ac0 0x560490b34bf0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.217+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v101'505 uv479 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.217+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490be5ab0 con 0x560490a99ac0 2024-05-09T03:18:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.219+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v101'506 uv506 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.219+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be5e50 con 0x560490a99ac0 2024-05-09T03:18:05.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.220+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v101'507 uv507 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.221 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.220+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be7200 con 0x560490a99ac0 2024-05-09T03:18:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.221+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v101'508 uv508 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.221+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490b3a160 con 0x560490a99ac0 2024-05-09T03:18:05.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.223+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v101'509 uv509 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.223+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be61f0 con 0x560490a99ac0 2024-05-09T03:18:05.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.224+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v101'510 uv510 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.224+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6590 con 0x560490a99ac0 2024-05-09T03:18:05.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.225+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v101'511 uv485 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.225+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490be6980 con 0x560490a99ac0 2024-05-09T03:18:05.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.227+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v101'512 uv512 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.227+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6980 con 0x560490a99ac0 2024-05-09T03:18:05.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.228+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v101'513 uv513 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.229 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:05.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.228+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.230+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v101'514 uv514 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.230+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.231+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v101'515 uv515 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.231+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.233+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v101'516 uv516 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.233+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.234+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v101'517 uv492 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.234+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.236+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v101'518 uv518 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.236+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.237+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v101'519 uv519 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.238 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:05.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.237+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.238+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v101'520 uv520 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.238+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.240+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v101'521 uv521 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.240+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.241+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v101'522 uv522 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.241+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.243+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v101'523 uv523 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.243+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.244+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v101'524 uv498 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.244+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.246+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v101'525 uv525 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.246+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.247+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v101'526 uv526 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.248 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.247+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.249+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v101'527 uv527 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.249+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.250+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v101'528 uv528 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.250+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.252+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v101'529 uv529 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.252+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.253+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v101'530 uv504 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.254+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.255+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v101'531 uv531 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.255+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.256+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v101'532 uv532 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.257 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:05.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.256+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.258+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v101'533 uv533 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.258+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.259+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v101'534 uv534 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.259+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490a99ac0 2024-05-09T03:18:05.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.261+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v101'535 uv535 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.261+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560490be6d20 0x560490be9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.261+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490be6930 con 0x560490be6d20 2024-05-09T03:18:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.261+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560490be6d20 0x560490be9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.262+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560490be6d20 0x560490be9ac0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.263+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v101'371 uv353 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.263+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bea000 con 0x560490be6d20 2024-05-09T03:18:05.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.264+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v101'372 uv372 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.264+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bea3a0 con 0x560490be6d20 2024-05-09T03:18:05.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.266+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v101'373 uv373 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.266 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.266+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beabf0 con 0x560490be6d20 2024-05-09T03:18:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.267+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v101'374 uv374 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.267+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beaf90 con 0x560490be6d20 2024-05-09T03:18:05.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.268+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v101'375 uv375 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.268+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beb330 con 0x560490be6d20 2024-05-09T03:18:05.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.269+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v101'376 uv370 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.269+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490beb6d0 con 0x560490be6d20 2024-05-09T03:18:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.271+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v101'377 uv377 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.271+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.272+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v101'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.273 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.272+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.273+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v101'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.273+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.275+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v101'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.275+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.276+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v101'381 uv363 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.276+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.277+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v101'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.277+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.278+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v101'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.279 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.278+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bef4d0 con 0x560490be6d20 2024-05-09T03:18:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.279+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v101'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.279+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bef4d0 con 0x560490be6d20 2024-05-09T03:18:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.280+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v101'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.280+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.282+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v101'386 uv369 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.282+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.283+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v101'387 uv387 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.283+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.284+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v101'388 uv388 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.285 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.284+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beca80 con 0x560490be6d20 2024-05-09T03:18:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.285+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v101'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.285+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beca80 con 0x560490be6d20 2024-05-09T03:18:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.286+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v101'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.286+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490be6d20 2024-05-09T03:18:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.287+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v101'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490be6d20 2024-05-09T03:18:05.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.287+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490a99ac0 2024-05-09T03:18:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.289+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v101'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.289+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490beba70 con 0x560490a99ac0 2024-05-09T03:18:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.290+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v101'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.290+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490a99ac0 2024-05-09T03:18:05.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.291+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v101'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.291 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.291+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beca80 con 0x560490a99ac0 2024-05-09T03:18:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.292+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v101'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.292+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beca80 con 0x560490a99ac0 2024-05-09T03:18:05.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.293+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v101'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.293+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beba70 con 0x560490a99ac0 2024-05-09T03:18:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.294+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v101'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.294+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560490bebf80 0x560490bee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.294+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beeab0 con 0x560490bebf80 2024-05-09T03:18:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.294+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560490bebf80 0x560490bee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.295+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560490bebf80 0x560490bee3e0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.296+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v101'78 uv77 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.296+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490beeea0 con 0x560490bebf80 2024-05-09T03:18:05.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.297+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v101'79 uv79 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.297+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beeea0 con 0x560490bebf80 2024-05-09T03:18:05.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.298+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v101'80 uv80 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.299 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:05.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.298+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beeea0 con 0x560490bebf80 2024-05-09T03:18:05.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.299+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v101'81 uv81 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.299+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beed60 con 0x560490a99ac0 2024-05-09T03:18:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.301+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v101'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.301+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490beed60 con 0x560490a99ac0 2024-05-09T03:18:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.302+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v101'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.302+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beed60 con 0x560490a99ac0 2024-05-09T03:18:05.305 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.305+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v101'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.305+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beed60 con 0x560490a99ac0 2024-05-09T03:18:05.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.308+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v101'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.308+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beed60 con 0x560490a99ac0 2024-05-09T03:18:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.310+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v101'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.311+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490beed60 con 0x560490a99ac0 2024-05-09T03:18:05.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.312+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v101'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.312+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x560490bef430 0x560490bef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.312+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf21e0 con 0x560490bef430 2024-05-09T03:18:05.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.312+0000 7f23e3402640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x560490bef430 0x560490bef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.312+0000 7f23e3402640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x560490bef430 0x560490bef880 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.314+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v101'182 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.314+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bf2580 con 0x560490bef430 2024-05-09T03:18:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.315+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v101'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.315+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf2960 con 0x560490bef430 2024-05-09T03:18:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.316+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v101'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.316 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.316+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bea840 con 0x560490bef430 2024-05-09T03:18:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.317+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v101'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.317+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf42f0 con 0x560490bef430 2024-05-09T03:18:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.318+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v101'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.318+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf3200 con 0x560490bef430 2024-05-09T03:18:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.319+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v101'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.319+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf35f0 con 0x560490bef430 2024-05-09T03:18:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.320+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v101'188 uv181 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.320+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bf3990 con 0x560490bef430 2024-05-09T03:18:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.321+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v101'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.321+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf3d80 con 0x560490bef430 2024-05-09T03:18:05.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.322+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v101'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.323 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:05.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.322+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf3d80 con 0x560490bef430 2024-05-09T03:18:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.323+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v101'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.323+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf3d30 con 0x560490bef430 2024-05-09T03:18:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.324+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v101'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.324+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x560490bf4c10 0x560490bf7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.324+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf7570 con 0x560490bf4c10 2024-05-09T03:18:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.325+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x560490bf4c10 0x560490bf7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.325+0000 7f23e2c01640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x560490bf4c10 0x560490bf7030 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.326+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v101'81 uv80 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490bf4c10 2024-05-09T03:18:05.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.326+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bf75f0 con 0x560490bf4c10 2024-05-09T03:18:05.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.327+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v101'82 uv82 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490bf4c10 2024-05-09T03:18:05.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.327+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490bf4c10 2024-05-09T03:18:05.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.328+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v101'83 uv83 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490bf4c10 2024-05-09T03:18:05.329 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:05.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.328+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490bf4c10 2024-05-09T03:18:05.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.329+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v101'84 uv84 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490bf4c10 2024-05-09T03:18:05.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.329+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490bf4c10 2024-05-09T03:18:05.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.330+0000 7f23e2c01640 1 -- 172.21.15.46:0/2435230016 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v101'85 uv85 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23d400db30 con 0x560490bf4c10 2024-05-09T03:18:05.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.330+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490a99ac0 2024-05-09T03:18:05.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.331+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v101'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.331+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490a99ac0 2024-05-09T03:18:05.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.332+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v101'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.332+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490a99ac0 2024-05-09T03:18:05.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.333+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v101'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.334 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:05.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.333+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf89f0 con 0x560490a99ac0 2024-05-09T03:18:05.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.335+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v101'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.335+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf89f0 con 0x560490a99ac0 2024-05-09T03:18:05.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.335+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v101'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.336+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490a99ac0 2024-05-09T03:18:05.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.336+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v101'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490a99ac0 2024-05-09T03:18:05.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.337+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490bebf80 2024-05-09T03:18:05.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.338+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v101'162 uv156 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.338+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490bebf80 2024-05-09T03:18:05.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.339+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v101'163 uv163 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.339+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf79e0 con 0x560490bebf80 2024-05-09T03:18:05.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.340+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v101'164 uv164 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.340 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:05.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.340+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf2df0 con 0x560490bebf80 2024-05-09T03:18:05.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.341+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v101'165 uv165 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.341+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf2df0 con 0x560490bebf80 2024-05-09T03:18:05.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.342+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v101'166 uv166 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.342+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf2df0 con 0x560490bebf80 2024-05-09T03:18:05.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.343+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v101'167 uv161 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.343+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bf2df0 con 0x560490bebf80 2024-05-09T03:18:05.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.344+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v101'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.344+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf2df0 con 0x560490bebf80 2024-05-09T03:18:05.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.345+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v101'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.346 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:05.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.345+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bf2df0 con 0x560490bebf80 2024-05-09T03:18:05.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.346+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v101'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.346+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfa830 con 0x560490bebf80 2024-05-09T03:18:05.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.347+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v101'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.348+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x560490bf8510 0x560490bfa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.348+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfaef0 con 0x560490bf8510 2024-05-09T03:18:05.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.348+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x560490bf8510 0x560490bfa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.348+0000 7f23dbfff640 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x560490bf8510 0x560490bfa930 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.349+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v101'74 uv73 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bf8510 2024-05-09T03:18:05.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.349+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bfaef0 con 0x560490bf8510 2024-05-09T03:18:05.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.350+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v101'75 uv75 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bf8510 2024-05-09T03:18:05.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.350+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfaef0 con 0x560490bf8510 2024-05-09T03:18:05.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.351+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v101'76 uv76 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bf8510 2024-05-09T03:18:05.352 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:05.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.351+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfaef0 con 0x560490bf8510 2024-05-09T03:18:05.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.352+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v101'77 uv77 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bf8510 2024-05-09T03:18:05.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.352+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfaef0 con 0x560490bebf80 2024-05-09T03:18:05.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.354+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v101'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.354+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bfaea0 con 0x560490bebf80 2024-05-09T03:18:05.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.355+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v101'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.355+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfaea0 con 0x560490bebf80 2024-05-09T03:18:05.357 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:05.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.356+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v101'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.356+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfaea0 con 0x560490bebf80 2024-05-09T03:18:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.357+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v101'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.357+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfae70 con 0x560490bebf80 2024-05-09T03:18:05.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.358+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v101'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.358+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfae70 con 0x560490bebf80 2024-05-09T03:18:05.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.359+0000 7f23dbfff640 1 -- 172.21.15.46:0/2435230016 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v101'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23cc051070 con 0x560490bebf80 2024-05-09T03:18:05.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.360+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfae70 con 0x560490bef430 2024-05-09T03:18:05.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.361+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v101'84 uv83 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.361+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bfae70 con 0x560490bef430 2024-05-09T03:18:05.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.362+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v101'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.362+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfae70 con 0x560490bef430 2024-05-09T03:18:05.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.363+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v101'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.364 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:05.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.363+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.364+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v101'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.364+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.366+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v101'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.366+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.367+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v101'204 uv196 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.367+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.368+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v101'205 uv205 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.368+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.369+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v101'206 uv206 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.370 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:05.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.369+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.371+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v101'207 uv207 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.371+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.372+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v101'208 uv208 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.372+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.373+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v101'209 uv203 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.373+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.374+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v101'210 uv210 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.374+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.375+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v101'211 uv211 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.376 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:05.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.375+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfc2f0 con 0x560490bef430 2024-05-09T03:18:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.377+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v101'212 uv212 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.377+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfc2f0 con 0x560490bef430 2024-05-09T03:18:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.378+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v101'213 uv213 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.378+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb2e0 con 0x560490bef430 2024-05-09T03:18:05.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.379+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v101'214 uv214 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.379+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x560490bfb790 con 0x560490bef430 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.380+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v101'215 uv215 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f23dc05bd80 con 0x560490bef430 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x560490a99ac0 msgr2=0x560490b34bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x560490a99ac0 0x560490b34bf0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560490bebf80 msgr2=0x560490bee3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x560490bebf80 0x560490bee3e0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x560490bf8510 msgr2=0x560490bfa930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x560490bf8510 0x560490bfa930 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560490be6d20 msgr2=0x560490be9ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x560490be6d20 0x560490be9ac0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 reap_dead start 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x560490bef430 msgr2=0x560490bef880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x560490bef430 0x560490bef880 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e3402640 1 -- 172.21.15.46:0/2435230016 reap_dead start 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x560490bf4c10 msgr2=0x560490bf7030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x560490bf4c10 0x560490bf7030 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23c4074cb0 msgr2=0x7f23c40771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23c4074cb0 0x7f23c40771c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f23d4001e90 tx=0x7f23d4008000 comp rx=0 tx=0).stop 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 msgr2=0x560490b28850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 0x560490b28850 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00f4d0 tx=0x7f23dc00f990 comp rx=0 tx=0).stop 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 shutdown_connections 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x560490bf4c10 0x560490bf7030 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f23c4074cb0 0x7f23c40771c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 --2- 172.21.15.46:0/2435230016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560490b43810 0x560490b28850 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 >> 172.21.15.46:0/2435230016 conn(0x56049098eac0 msgr2=0x560490b43e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:05.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 shutdown_connections 2024-05-09T03:18:05.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.381+0000 7f23e9139f40 1 -- 172.21.15.46:0/2435230016 wait complete. 2024-05-09T03:18:05.385 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 200.00000001 - 2024-05-09T03:18:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.481+0000 7f99c3f86f40 1 Processor -- start 2024-05-09T03:18:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.481+0000 7f99c3f86f40 1 -- start start 2024-05-09T03:18:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99c3f86f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 0x55e51da1ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99c3f86f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da1dc90 0x55e51d979cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99c3f86f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51da24610 0x55e51d97a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99c3f86f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e51d97ef20 con 0x55e51d94c2a0 2024-05-09T03:18:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99c3f86f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e51d8f5f50 con 0x55e51da24610 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99c3f86f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e51d997bd0 con 0x55e51da1dc90 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99bda4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 0x55e51da1ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99bda4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 0x55e51da1ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36718/0 (socket says 172.21.15.46:36718) 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.482+0000 7f99bda4e640 1 -- 172.21.15.46:0/2052766599 learned_addr learned my addr 172.21.15.46:0/2052766599 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99be24f640 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51da24610 0x55e51d97a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bd24d640 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da1dc90 0x55e51d979cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bda4e640 1 -- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da1dc90 msgr2=0x55e51d979cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bda4e640 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da1dc90 0x55e51d979cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bda4e640 1 -- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51da24610 msgr2=0x55e51d97a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bda4e640 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51da24610 0x55e51d97a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bda4e640 1 -- 172.21.15.46:0/2052766599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51d9797f0 con 0x55e51d94c2a0 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99be24f640 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51da24610 0x55e51d97a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:05.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99bda4e640 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 0x55e51da1ab60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f99b0025a20 tx=0x7f99b0050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7888b0bec9f69818 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99b7fff640 1 -- 172.21.15.46:0/2052766599 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0051a60 con 0x55e51d94c2a0 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99b7fff640 1 -- 172.21.15.46:0/2052766599 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b0051c00 con 0x55e51d94c2a0 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.483+0000 7f99b7fff640 1 -- 172.21.15.46:0/2052766599 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0065430 con 0x55e51d94c2a0 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2052766599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 msgr2=0x55e51da1ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 0x55e51da1ab60 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f99b0025a20 tx=0x7f99b0050000 comp rx=0 tx=0).stop 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2052766599 shutdown_connections 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51da24610 0x55e51d97a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da1dc90 0x55e51d979cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2052766599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51d94c2a0 0x55e51da1ab60 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2052766599 >> 172.21.15.46:0/2052766599 conn(0x55e51d871ac0 msgr2=0x55e51d97bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2052766599 shutdown_connections 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.484+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2052766599 wait complete. 2024-05-09T03:18:05.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.485+0000 7f99c3f86f40 1 Processor -- start 2024-05-09T03:18:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.485+0000 7f99c3f86f40 1 -- start start 2024-05-09T03:18:05.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99c3f86f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51d94c2a0 0x55e51da15dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99c3f86f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51da1dc90 0x55e51da1b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99c3f86f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da24610 0x55e51da1b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99c3f86f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e51d9795a0 con 0x55e51da1dc90 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99c3f86f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e51da1c310 con 0x55e51d94c2a0 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99c3f86f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e51da1a5b0 con 0x55e51da24610 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51da1dc90 0x55e51da1b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51da1dc90 0x55e51da1b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36734/0 (socket says 172.21.15.46:36734) 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 -- 172.21.15.46:0/2814019982 learned_addr learned my addr 172.21.15.46:0/2814019982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bda4e640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51d94c2a0 0x55e51da15dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99be24f640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da24610 0x55e51da1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 -- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da24610 msgr2=0x55e51da1b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e51da24610 0x55e51da1b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 -- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51d94c2a0 msgr2=0x55e51da15dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e51d94c2a0 0x55e51da15dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.486+0000 7f99bd24d640 1 -- 172.21.15.46:0/2814019982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51d98fca0 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.487+0000 7f99bd24d640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51da1dc90 0x55e51da1b130 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f99a800cc60 tx=0x7f99a800a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.487+0000 7f99b6ffd640 1 -- 172.21.15.46:0/2814019982 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8010450 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.487+0000 7f99b6ffd640 1 -- 172.21.15.46:0/2814019982 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a80105f0 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.487+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e51da16310 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.487+0000 7f99b6ffd640 1 -- 172.21.15.46:0/2814019982 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a80109a0 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.487+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e51d990480 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.489+0000 7f99b6ffd640 1 -- 172.21.15.46:0/2814019982 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f99a8010b40 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.489+0000 7f99b6ffd640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99a0074cb0 0x7f99a00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.489+0000 7f99b6ffd640 1 -- 172.21.15.46:0/2814019982 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f99a8096790 con 0x55e51da1dc90 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.489+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e51dabf9a0 0x55e51dabfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.489+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x55e51da19270 con 0x55e51dabf9a0 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.490+0000 7f99bda4e640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99a0074cb0 0x7f99a00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.490+0000 7f99be24f640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e51dabf9a0 0x55e51dabfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.490+0000 7f99bda4e640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99a0074cb0 0x7f99a00771c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f99b0063000 tx=0x7f99b0051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.490+0000 7f99be24f640 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e51dabf9a0 0x55e51dabfdb0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99be24f640 1 -- 172.21.15.46:0/2814019982 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000001 [writefull 0~80] v101'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f99b8009820 con 0x55e51dabf9a0 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e51dabf9a0 msgr2=0x55e51dabfdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e51dabf9a0 0x55e51dabfdb0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99a0074cb0 msgr2=0x7f99a00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f99a0074cb0 0x7f99a00771c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f99b0063000 tx=0x7f99b0051000 comp rx=0 tx=0).stop 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51da1dc90 msgr2=0x55e51da1b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 --2- 172.21.15.46:0/2814019982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e51da1dc90 0x55e51da1b130 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f99a800cc60 tx=0x7f99a800a580 comp rx=0 tx=0).stop 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99be24f640 1 -- 172.21.15.46:0/2814019982 reap_dead start 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 shutdown_connections 2024-05-09T03:18:05.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 >> 172.21.15.46:0/2814019982 conn(0x55e51d871ac0 msgr2=0x55e51da23970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 shutdown_connections 2024-05-09T03:18:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.492+0000 7f99c3f86f40 1 -- 172.21.15.46:0/2814019982 wait complete. 2024-05-09T03:18:05.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.652+0000 7f056f49a640 1 Processor -- start 2024-05-09T03:18:05.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.652+0000 7f056f49a640 1 -- start start 2024-05-09T03:18:05.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.653+0000 7f056f49a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05680bb770 0x7f05680d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.653+0000 7f056f49a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f05681642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056f49a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f05680d2a70 0x7f05680d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056f49a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f05680d4bb0 con 0x7f05680bb770 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056f49a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f05680d33c0 con 0x7f05680d2a70 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056f49a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0568164900 con 0x7f0568163eb0 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05680bb770 0x7f05680d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05680bb770 0x7f05680d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36746/0 (socket says 172.21.15.46:36746) 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 -- 172.21.15.46:0/4134602545 learned_addr learned my addr 172.21.15.46:0/4134602545 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056ca35640 1 --2- 172.21.15.46:0/4134602545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f05681642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056da37640 1 --2- 172.21.15.46:0/4134602545 >> v2:172.21.15.88:3300/0 conn(0x7f05680d2a70 0x7f05680d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 -- 172.21.15.46:0/4134602545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 msgr2=0x7f05681642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 --2- 172.21.15.46:0/4134602545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f05681642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 -- 172.21.15.46:0/4134602545 >> v2:172.21.15.88:3300/0 conn(0x7f05680d2a70 msgr2=0x7f05680d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:05.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 --2- 172.21.15.46:0/4134602545 >> v2:172.21.15.88:3300/0 conn(0x7f05680d2a70 0x7f05680d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.654+0000 7f056d236640 1 -- 172.21.15.46:0/4134602545 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0568164be0 con 0x7f05680bb770 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.655+0000 7f056d236640 1 --2- 172.21.15.46:0/4134602545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680bb770 0x7f05680d5040 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f05680b56d0 tx=0x7f055802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a773f672c21466 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.655+0000 7f0567fff640 1 -- 172.21.15.46:0/4134602545 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558036070 con 0x7f05680bb770 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.655+0000 7f0567fff640 1 -- 172.21.15.46:0/4134602545 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0558031600 con 0x7f05680bb770 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.655+0000 7f0567fff640 1 -- 172.21.15.46:0/4134602545 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580319d0 con 0x7f05680bb770 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 -- 172.21.15.46:0/4134602545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680bb770 msgr2=0x7f05680d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 --2- 172.21.15.46:0/4134602545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680bb770 0x7f05680d5040 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f05680b56d0 tx=0x7f055802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 -- 172.21.15.46:0/4134602545 shutdown_connections 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 --2- 172.21.15.46:0/4134602545 >> v2:172.21.15.88:3300/0 conn(0x7f05680d2a70 0x7f05680d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 --2- 172.21.15.46:0/4134602545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f05681642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 --2- 172.21.15.46:0/4134602545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680bb770 0x7f05680d5040 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 -- 172.21.15.46:0/4134602545 >> 172.21.15.46:0/4134602545 conn(0x7f056806e370 msgr2=0x7f056806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 -- 172.21.15.46:0/4134602545 shutdown_connections 2024-05-09T03:18:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.656+0000 7f056f49a640 1 -- 172.21.15.46:0/4134602545 wait complete. 2024-05-09T03:18:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.657+0000 7f056f49a640 1 Processor -- start 2024-05-09T03:18:05.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.658+0000 7f056f49a640 1 -- start start 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056f49a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05680bb770 0x7f05681502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056f49a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 0x7f056814e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056f49a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f056814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056f49a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05680bbb80 con 0x7f05680d2a70 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056f49a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f05680d5580 con 0x7f05680bb770 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056f49a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f05680d0bc0 con 0x7f0568163eb0 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 0x7f056814e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 0x7f056814e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36750/0 (socket says 172.21.15.46:36750) 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 -- 172.21.15.46:0/302168304 learned_addr learned my addr 172.21.15.46:0/302168304 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056d236640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05680bb770 0x7f05681502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056da37640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f056814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 -- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 msgr2=0x7f056814c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f056814c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 -- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05680bb770 msgr2=0x7f05681502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05680bb770 0x7f05681502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 -- 172.21.15.46:0/302168304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056814d530 con 0x7f05680d2a70 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.659+0000 7f056ca35640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 0x7f056814e5d0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f055c00d5f0 tx=0x7f055c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.660+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0125f0 con 0x7f05680d2a70 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.660+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f055c019070 con 0x7f05680d2a70 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.660+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056814d810 con 0x7f05680d2a70 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.660+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c0129d0 con 0x7f05680d2a70 2024-05-09T03:18:05.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.660+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056814ddd0 con 0x7f05680d2a70 2024-05-09T03:18:05.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.661+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f056814d2a0 con 0x7f05680d2a70 2024-05-09T03:18:05.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.661+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f055c012b70 con 0x7f05680d2a70 2024-05-09T03:18:05.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.662+0000 7f0565ffb640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f05540762d0 0x7f05540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:05.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.662+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7229+0+0 (secure 0 0 0) 0x7f055c097f30 con 0x7f05680d2a70 2024-05-09T03:18:05.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.662+0000 7f056d236640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f05540762d0 0x7f05540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:05.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.663+0000 7f056d236640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f05540762d0 0x7f05540787e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0558033000 tx=0x7f0558009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:05.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.667+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f055c060bc0 con 0x7f05680d2a70 2024-05-09T03:18:05.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:05.898+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f05681642c0 con 0x7f05680d2a70 2024-05-09T03:18:06.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.130+0000 7f0565ffb640 1 -- 172.21.15.46:0/302168304 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v195) ==== 162+0+0 (secure 0 0 0) 0x7f05681642c0 con 0x7f05680d2a70 2024-05-09T03:18:06.130 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f05540762d0 msgr2=0x7f05540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f05540762d0 0x7f05540787e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f0558033000 tx=0x7f0558009000 comp rx=0 tx=0).stop 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 msgr2=0x7f056814e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 0x7f056814e5d0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f055c00d5f0 tx=0x7f055c00dab0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 shutdown_connections 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0568163eb0 0x7f056814c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f05540762d0 0x7f05540787e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05680d2a70 0x7f056814e5d0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 --2- 172.21.15.46:0/302168304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05680bb770 0x7f05681502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 >> 172.21.15.46:0/302168304 conn(0x7f056806e370 msgr2=0x7f05680b6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:06.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 shutdown_connections 2024-05-09T03:18:06.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.132+0000 7f056f49a640 1 -- 172.21.15.46:0/302168304 wait complete. 2024-05-09T03:18:06.146 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:06.147 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:06.146+0000 7f6c49f35640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:18:06.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:06.146+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:06.147 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:06.146+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:06.153 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:18:06.146+0000 7f6c49f35640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:18:06.208 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:06.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.264+0000 7f96e5aee640 1 Processor -- start 2024-05-09T03:18:06.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.265+0000 7f96e5aee640 1 -- start start 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96e5aee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 0x7f96e00b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96e5aee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96e00d34e0 0x7f96e008d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96e5aee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e00d2440 con 0x7f96e006de60 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96e5aee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f96e00d2720 con 0x7f96e00d34e0 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96e5aee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96e00d2a00 con 0x7f96e00d1f60 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96dffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96e006de60 0x7f96e006e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54412/0 (socket says 172.21.15.46:54412) 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96dffff640 1 -- 172.21.15.46:0/3141263064 learned_addr learned my addr 172.21.15.46:0/3141263064 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96deffd640 1 --2- 172.21.15.46:0/3141263064 >> v2:172.21.15.88:3300/0 conn(0x7f96e00d34e0 0x7f96e008d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96df7fe640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 0x7f96e00b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414016084 0 0) 0x7f96e00d2440 con 0x7f96e006de60 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.266+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96c8003520 con 0x7f96e006de60 2024-05-09T03:18:06.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3380520615 0 0) 0x7f96c8003520 con 0x7f96e006de60 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 msgr2=0x7f96e00b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 0x7f96e00b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 >> v2:172.21.15.88:3300/0 conn(0x7f96e00d34e0 msgr2=0x7f96e008d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 --2- 172.21.15.46:0/3141263064 >> v2:172.21.15.88:3300/0 conn(0x7f96e00d34e0 0x7f96e008d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e00d2ce0 con 0x7f96e006de60 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f96d8001a70 con 0x7f96e006de60 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96deffd640 1 --2- 172.21.15.46:0/3141263064 >> v2:172.21.15.88:3300/0 conn(0x7f96e00d34e0 0x7f96e008d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 >> v1:172.21.15.46:6789/0 conn(0x7f96e006de60 legacy=0x7f96e006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8003a00 0x7f96c8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 0x7f96c8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8004d40 0x7f96c8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00d2720 con 0x7f96c8003a00 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00d2a00 con 0x7f96c8004d40 2024-05-09T03:18:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96c800ba70 con 0x7f96e00d1f60 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96df7fe640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8003a00 0x7f96c8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96deffd640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 0x7f96c8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96dffff640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8004d40 0x7f96c8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96df7fe640 1 -- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 msgr2=0x7f96c8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96df7fe640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e00d1f60 0x7f96c8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96df7fe640 1 -- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8004d40 msgr2=0x7f96c8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96df7fe640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8004d40 0x7f96c8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.267+0000 7f96df7fe640 1 -- 172.21.15.46:0/3141263064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c8003520 con 0x7f96c8003a00 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.268+0000 7f96df7fe640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8003a00 0x7f96c8003e70 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f96e0165220 tx=0x7f96c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d68432e3ab1253 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.268+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0036070 con 0x7f96c8003a00 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.268+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d0031400 con 0x7f96c8003a00 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.268+0000 7f96de7fc640 1 -- 172.21.15.46:0/3141263064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0009690 con 0x7f96c8003a00 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96e5aee640 1 -- 172.21.15.46:0/3141263064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8003a00 msgr2=0x7f96c8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96e5aee640 1 --2- 172.21.15.46:0/3141263064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8003a00 0x7f96c8003e70 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f96e0165220 tx=0x7f96c8005830 comp rx=0 tx=0).stop 2024-05-09T03:18:06.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96dffff640 1 -- 172.21.15.46:0/3141263064 reap_dead start 2024-05-09T03:18:06.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96e5aee640 1 -- 172.21.15.46:0/3141263064 shutdown_connections 2024-05-09T03:18:06.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96e5aee640 1 -- 172.21.15.46:0/3141263064 >> 172.21.15.46:0/3141263064 conn(0x7f96e00b56e0 msgr2=0x7f96e00b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:06.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96e5aee640 1 -- 172.21.15.46:0/3141263064 shutdown_connections 2024-05-09T03:18:06.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.269+0000 7f96e5aee640 1 -- 172.21.15.46:0/3141263064 wait complete. 2024-05-09T03:18:06.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.271+0000 7f96e5aee640 1 Processor -- start 2024-05-09T03:18:06.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.271+0000 7f96e5aee640 1 -- start start 2024-05-09T03:18:06.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96e5aee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8003a00 0x7f96e0158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96e5aee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e015dc80 0x7f96e00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96e5aee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 0x7f96e00b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96e5aee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00d3a70 con 0x7f96e00cf790 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96e5aee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00d38f0 con 0x7f96c8003a00 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96e5aee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96e008dcb0 con 0x7f96e015dc80 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.272+0000 7f96dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 0x7f96e00b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 0x7f96e00b25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41422/0 (socket says 172.21.15.46:41422) 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 -- 172.21.15.46:0/2715479636 learned_addr learned my addr 172.21.15.46:0/2715479636 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96df7fe640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8003a00 0x7f96e0158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96deffd640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e015dc80 0x7f96e00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 -- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e015dc80 msgr2=0x7f96e00cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e015dc80 0x7f96e00cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 -- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8003a00 msgr2=0x7f96e0158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8003a00 0x7f96e0158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 -- 172.21.15.46:0/2715479636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e00cfba0 con 0x7f96e00cf790 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96dffff640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 0x7f96e00b25d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f96e006fac0 tx=0x7f96d8007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d800ac20 con 0x7f96e00cf790 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d8018070 con 0x7f96e00cf790 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e00caca0 con 0x7f96e00cf790 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d80133f0 con 0x7f96e00cf790 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.273+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e015ef50 con 0x7f96e00cf790 2024-05-09T03:18:06.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.274+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e00cae70 con 0x7f96e00cf790 2024-05-09T03:18:06.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.275+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f96d8013590 con 0x7f96e00cf790 2024-05-09T03:18:06.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.276+0000 7f96c7fff640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96cc0762d0 0x7f96cc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.276+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f96d80982c0 con 0x7f96e00cf790 2024-05-09T03:18:06.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.276+0000 7f96df7fe640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96cc0762d0 0x7f96cc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.277+0000 7f96df7fe640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96cc0762d0 0x7f96cc0787e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f96e0165220 tx=0x7f96d0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:06.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.282+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f96e00cfba0 con 0x7f96e00cf790 2024-05-09T03:18:06.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.510+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96e015ea00 con 0x7f96e00cf790 2024-05-09T03:18:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.510+0000 7f96c7fff640 1 -- 172.21.15.46:0/2715479636 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+4343 (secure 0 0 0) 0x7f96d8060ef0 con 0x7f96e00cf790 2024-05-09T03:18:06.511 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:06.511 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":197,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:06.147270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:06.511 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 197 2024-05-09T03:18:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96cc0762d0 msgr2=0x7f96cc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96cc0762d0 0x7f96cc0787e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f96e0165220 tx=0x7f96d0008030 comp rx=0 tx=0).stop 2024-05-09T03:18:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 msgr2=0x7f96e00b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 0x7f96e00b25d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f96e006fac0 tx=0x7f96d8007bd0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 shutdown_connections 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f96cc0762d0 0x7f96cc0787e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96e00cf790 0x7f96e00b25d0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96e015dc80 0x7f96e00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 --2- 172.21.15.46:0/2715479636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96c8003a00 0x7f96e0158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 >> 172.21.15.46:0/2715479636 conn(0x7f96e00b56e0 msgr2=0x7f96e01642b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 shutdown_connections 2024-05-09T03:18:06.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.512+0000 7f96e5aee640 1 -- 172.21.15.46:0/2715479636 wait complete. 2024-05-09T03:18:06.525 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:06.525 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:06.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.680+0000 7f822f78d640 1 Processor -- start 2024-05-09T03:18:06.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.680+0000 7f822f78d640 1 -- start start 2024-05-09T03:18:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822f78d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82280d2b90 0x7f822815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822f78d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8228160b20 con 0x7f8228164c70 2024-05-09T03:18:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822f78d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82280d15f0 con 0x7f82281615e0 2024-05-09T03:18:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822f78d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f822815e520 con 0x7f82280d2b90 2024-05-09T03:18:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822dd2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82280d2b90 0x7f822815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822d529640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8228164c70 0x7f82280d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54420/0 (socket says 172.21.15.46:54420) 2024-05-09T03:18:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.681+0000 7f822d529640 1 -- 172.21.15.46:0/1079215645 learned_addr learned my addr 172.21.15.46:0/1079215645 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.682+0000 7f822dd2a640 1 -- 172.21.15.46:0/1079215645 >> v1:172.21.15.88:6789/0 conn(0x7f82281615e0 legacy=0x7f82280d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.682+0000 7f822dd2a640 1 -- 172.21.15.46:0/1079215645 >> v1:172.21.15.46:6789/0 conn(0x7f8228164c70 legacy=0x7f82280d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.682+0000 7f821ffff640 1 -- 172.21.15.46:0/1079215645 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454718127 0 0) 0x7f8228160b20 con 0x7f8228164c70 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.682+0000 7f822dd2a640 1 -- 172.21.15.46:0/1079215645 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82280d18d0 con 0x7f82280d2b90 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.682+0000 7f822dd2a640 1 --2- 172.21.15.46:0/1079215645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82280d2b90 0x7f822815dfe0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f8220009d90 tx=0x7f822002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1feaa20c4b05e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f821ffff640 1 -- 172.21.15.46:0/1079215645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220007740 con 0x7f82280d2b90 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f821ffff640 1 -- 172.21.15.46:0/1079215645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82200078e0 con 0x7f82280d2b90 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f821ffff640 1 -- 172.21.15.46:0/1079215645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220007cb0 con 0x7f82280d2b90 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 -- 172.21.15.46:0/1079215645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82280d2b90 msgr2=0x7f822815dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 --2- 172.21.15.46:0/1079215645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82280d2b90 0x7f822815dfe0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f8220009d90 tx=0x7f822002a660 comp rx=0 tx=0).stop 2024-05-09T03:18:06.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 -- 172.21.15.46:0/1079215645 shutdown_connections 2024-05-09T03:18:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 --2- 172.21.15.46:0/1079215645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82280d2b90 0x7f822815dfe0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 -- 172.21.15.46:0/1079215645 >> 172.21.15.46:0/1079215645 conn(0x7f8228073db0 msgr2=0x7f82280741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 -- 172.21.15.46:0/1079215645 shutdown_connections 2024-05-09T03:18:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.683+0000 7f822f78d640 1 -- 172.21.15.46:0/1079215645 wait complete. 2024-05-09T03:18:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.685+0000 7f822f78d640 1 Processor -- start 2024-05-09T03:18:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.685+0000 7f822f78d640 1 -- start start 2024-05-09T03:18:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.686+0000 7f822f78d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d2b90 0x7f82280ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.686+0000 7f822f78d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 0x7f82280cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.686+0000 7f822f78d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8228164c70 0x7f82280c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.686+0000 7f822f78d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82281619f0 con 0x7f82281615e0 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.686+0000 7f822f78d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8228161150 con 0x7f82280d2b90 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.686+0000 7f822f78d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f82280d45c0 con 0x7f8228164c70 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 0x7f82280cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 0x7f82280cf620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41438/0 (socket says 172.21.15.46:41438) 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 -- 172.21.15.46:0/62860444 learned_addr learned my addr 172.21.15.46:0/62860444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822d529640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d2b90 0x7f82280ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822dd2a640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8228164c70 0x7f82280c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 -- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8228164c70 msgr2=0x7f82280c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8228164c70 0x7f82280c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 -- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d2b90 msgr2=0x7f82280ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d2b90 0x7f82280ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 -- 172.21.15.46:0/62860444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8228148f40 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822cd28640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 0x7f82280cf620 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f821800c430 tx=0x7f821800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218017070 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82180052d0 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8228149220 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.687+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218003830 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.688+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82281497e0 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.688+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8228148cb0 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.689+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f821801a070 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.690+0000 7f821dffb640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f820c0762d0 0x7f820c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.690+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f82180972d0 con 0x7f82281615e0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.690+0000 7f822d529640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f820c0762d0 0x7f820c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:06.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.691+0000 7f822d529640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f820c0762d0 0x7f820c0787e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8224000f60 tx=0x7f8224007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:06.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.694+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8218021020 con 0x7f82281615e0 2024-05-09T03:18:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.921+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f822808cfd0 con 0x7f82281615e0 2024-05-09T03:18:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.922+0000 7f821dffb640 1 -- 172.21.15.46:0/62860444 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+4343 (secure 0 0 0) 0x7f822808cfd0 con 0x7f82281615e0 2024-05-09T03:18:06.922 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:06.923 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":197,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:06.147270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 197 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f820c0762d0 msgr2=0x7f820c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f820c0762d0 0x7f820c0787e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8224000f60 tx=0x7f8224007e40 comp rx=0 tx=0).stop 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 msgr2=0x7f82280cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 0x7f82280cf620 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f821800c430 tx=0x7f821800c8f0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 shutdown_connections 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8228164c70 0x7f82280c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f820c0762d0 0x7f820c0787e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82281615e0 0x7f82280cf620 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 --2- 172.21.15.46:0/62860444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d2b90 0x7f82280ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 >> 172.21.15.46:0/62860444 conn(0x7f8228073db0 msgr2=0x7f82280b9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 shutdown_connections 2024-05-09T03:18:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:06.924+0000 7f822f78d640 1 -- 172.21.15.46:0/62860444 wait complete. 2024-05-09T03:18:06.936 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:06.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.049+0000 7f6c8211f640 1 Processor -- start 2024-05-09T03:18:07.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.049+0000 7f6c8211f640 1 -- start start 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c8211f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c7c0d57a0 0x7f6c7c160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c8211f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b4af0 0x7f6c7c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c8211f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c7c0b77e0 0x7f6c7c0b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c8211f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6c7c0d5bb0 con 0x7f6c7c0d57a0 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c8211f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6c7c0b8130 con 0x7f6c7c0b77e0 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c8211f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7c0d49e0 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c7c0d57a0 0x7f6c7c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6c7c0d57a0 0x7f6c7c160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41440/0 (socket says 172.21.15.46:41440) 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/2392387199 learned_addr learned my addr 172.21.15.46:0/2392387199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7affd640 1 --2- 172.21.15.46:0/2392387199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b4af0 0x7f6c7c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7bfff640 1 --2- 172.21.15.46:0/2392387199 >> v2:172.21.15.88:3300/0 conn(0x7f6c7c0b77e0 0x7f6c7c0b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/2392387199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b4af0 msgr2=0x7f6c7c0b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 --2- 172.21.15.46:0/2392387199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b4af0 0x7f6c7c0b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/2392387199 >> v2:172.21.15.88:3300/0 conn(0x7f6c7c0b77e0 msgr2=0x7f6c7c0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 --2- 172.21.15.46:0/2392387199 >> v2:172.21.15.88:3300/0 conn(0x7f6c7c0b77e0 0x7f6c7c0b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/2392387199 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7c0d4cc0 con 0x7f6c7c0d57a0 2024-05-09T03:18:07.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.051+0000 7f6c7bfff640 1 --2- 172.21.15.46:0/2392387199 >> v2:172.21.15.88:3300/0 conn(0x7f6c7c0b77e0 0x7f6c7c0b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.052+0000 7f6c7b7fe640 1 --2- 172.21.15.46:0/2392387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c160610 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c08f200 tx=0x7f6c6c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3d964d8db9764c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.052+0000 7f6c7a7fc640 1 -- 172.21.15.46:0/2392387199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c037070 con 0x7f6c7c0d57a0 2024-05-09T03:18:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.052+0000 7f6c7a7fc640 1 -- 172.21.15.46:0/2392387199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6c032440 con 0x7f6c7c0d57a0 2024-05-09T03:18:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.052+0000 7f6c7a7fc640 1 -- 172.21.15.46:0/2392387199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c032810 con 0x7f6c7c0d57a0 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 -- 172.21.15.46:0/2392387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0d57a0 msgr2=0x7f6c7c160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 --2- 172.21.15.46:0/2392387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c160610 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c08f200 tx=0x7f6c6c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 -- 172.21.15.46:0/2392387199 shutdown_connections 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 --2- 172.21.15.46:0/2392387199 >> v2:172.21.15.88:3300/0 conn(0x7f6c7c0b77e0 0x7f6c7c0b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 --2- 172.21.15.46:0/2392387199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b4af0 0x7f6c7c0b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 --2- 172.21.15.46:0/2392387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c160610 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 -- 172.21.15.46:0/2392387199 >> 172.21.15.46:0/2392387199 conn(0x7f6c7c0743e0 msgr2=0x7f6c7c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 -- 172.21.15.46:0/2392387199 shutdown_connections 2024-05-09T03:18:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.053+0000 7f6c8211f640 1 -- 172.21.15.46:0/2392387199 wait complete. 2024-05-09T03:18:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.055+0000 7f6c8211f640 1 Processor -- start 2024-05-09T03:18:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.055+0000 7f6c8211f640 1 -- start start 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c8211f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 0x7f6c7c0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c8211f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b77e0 0x7f6c7c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c8211f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c1eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c8211f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c0d15b0 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c8211f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c0d5280 con 0x7f6c7c0d57a0 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c8211f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7c0d1920 con 0x7f6c7c0b77e0 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 0x7f6c7c0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 0x7f6c7c0cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41444/0 (socket says 172.21.15.46:41444) 2024-05-09T03:18:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/307000586 learned_addr learned my addr 172.21.15.46:0/307000586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7bfff640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c1eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7affd640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b77e0 0x7f6c7c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b77e0 msgr2=0x7f6c7c14d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b77e0 0x7f6c7c14d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c7c0d57a0 msgr2=0x7f6c7c1eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c1eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7b7fe640 1 -- 172.21.15.46:0/307000586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7c14b6c0 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.056+0000 7f6c7bfff640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c1eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.057+0000 7f6c7b7fe640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 0x7f6c7c0cec00 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c02a220 tx=0x7f6c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.057+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c037040 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.057+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c6c042070 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.057+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7c14b9a0 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.057+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c030880 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.057+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7c14bf60 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.058+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c7c0cf190 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.059+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6c6c030a20 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.059+0000 7f6c63fff640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c680762d0 0x7f6c680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.059+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f6c6c0b8590 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.059+0000 7f6c7affd640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c680762d0 0x7f6c680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.060+0000 7f6c7affd640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c680762d0 0x7f6c680787e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0bb020 tx=0x7f6c70008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:07.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.064+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6c6c081810 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.299+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c7c06a270 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.299+0000 7f6c63fff640 1 -- 172.21.15.46:0/307000586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+5169 (secure 0 0 0) 0x7f6c6c081110 con 0x7f6c7c0b4af0 2024-05-09T03:18:07.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:07.300 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":198,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3106969477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3106969477},{"type":"v1","addr":"172.21.15.46:6834","nonce":3106969477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":198}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:06.147270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:07.300 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 198 2024-05-09T03:18:07.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c680762d0 msgr2=0x7f6c680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c680762d0 0x7f6c680787e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c0bb020 tx=0x7f6c70008040 comp rx=0 tx=0).stop 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 msgr2=0x7f6c7c0cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 0x7f6c7c0cec00 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c02a220 tx=0x7f6c6c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 shutdown_connections 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c7c0d57a0 0x7f6c7c1eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6c7c0b77e0 0x7f6c7c14d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6c680762d0 0x7f6c680787e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 --2- 172.21.15.46:0/307000586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c7c0b4af0 0x7f6c7c0cec00 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.301+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 >> 172.21.15.46:0/307000586 conn(0x7f6c7c0743e0 msgr2=0x7f6c7c0b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.302+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 shutdown_connections 2024-05-09T03:18:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.302+0000 7f6c8211f640 1 -- 172.21.15.46:0/307000586 wait complete. 2024-05-09T03:18:07.313 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 200.00000001' led to DAMAGED state 2024-05-09T03:18:07.314 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 500.00000000' 2024-05-09T03:18:07.314 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:07.314 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:07.330 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:07.330 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:07.330 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:07.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:07.331 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:07.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:07.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:18:07.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:18:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.543+0000 7f723f59e640 1 Processor -- start 2024-05-09T03:18:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.543+0000 7f723f59e640 1 -- start start 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72400d3040 0x7f72400d4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72400d4b70 0x7f72400b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723f59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72400d5260 con 0x7f72400d11e0 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723f59e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f72400d5540 con 0x7f72400d3040 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723f59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f72400b9230 con 0x7f72400d4b70 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72400d3040 0x7f72400d4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72400d3040 0x7f72400d4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41454/0 (socket says 172.21.15.46:41454) 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 -- 172.21.15.46:0/1507347983 learned_addr learned my addr 172.21.15.46:0/1507347983 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723ed9d640 1 --2- 172.21.15.46:0/1507347983 >> v2:172.21.15.88:3300/0 conn(0x7f72400d4b70 0x7f72400b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 -- 172.21.15.46:0/1507347983 >> v2:172.21.15.88:3300/0 conn(0x7f72400d4b70 msgr2=0x7f72400b9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:07.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 --2- 172.21.15.46:0/1507347983 >> v2:172.21.15.88:3300/0 conn(0x7f72400d4b70 0x7f72400b9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 -- 172.21.15.46:0/1507347983 >> v1:172.21.15.88:6789/0 conn(0x7f72400d11e0 legacy=0x7f72400d2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:07.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723dd9b640 1 -- 172.21.15.46:0/1507347983 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72400b9510 con 0x7f72400d3040 2024-05-09T03:18:07.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.545+0000 7f723ed9d640 1 --2- 172.21.15.46:0/1507347983 >> v2:172.21.15.88:3300/0 conn(0x7f72400d4b70 0x7f72400b9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:07.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.546+0000 7f723dd9b640 1 --2- 172.21.15.46:0/1507347983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d3040 0x7f72400d4630 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f72400d38a0 tx=0x7f723402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd3b888caa33df7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:07.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.546+0000 7f723d59a640 1 -- 172.21.15.46:0/1507347983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234037070 con 0x7f72400d3040 2024-05-09T03:18:07.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.546+0000 7f723d59a640 1 -- 172.21.15.46:0/1507347983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7234032440 con 0x7f72400d3040 2024-05-09T03:18:07.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.546+0000 7f723d59a640 1 -- 172.21.15.46:0/1507347983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234032810 con 0x7f72400d3040 2024-05-09T03:18:07.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 -- 172.21.15.46:0/1507347983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d3040 msgr2=0x7f72400d4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 --2- 172.21.15.46:0/1507347983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d3040 0x7f72400d4630 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f72400d38a0 tx=0x7f723402a140 comp rx=0 tx=0).stop 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 -- 172.21.15.46:0/1507347983 shutdown_connections 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 --2- 172.21.15.46:0/1507347983 >> v2:172.21.15.88:3300/0 conn(0x7f72400d4b70 0x7f72400b9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 --2- 172.21.15.46:0/1507347983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d3040 0x7f72400d4630 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 -- 172.21.15.46:0/1507347983 >> 172.21.15.46:0/1507347983 conn(0x7f724006e110 msgr2=0x7f724006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 -- 172.21.15.46:0/1507347983 shutdown_connections 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.547+0000 7f723f59e640 1 -- 172.21.15.46:0/1507347983 wait complete. 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.549+0000 7f723f59e640 1 Processor -- start 2024-05-09T03:18:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.549+0000 7f723f59e640 1 -- start start 2024-05-09T03:18:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723f59e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72400d11e0 0x7f72401530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 0x7f72400ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723f59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72401e8f40 0x7f72401e9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723f59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72400d15f0 con 0x7f72400d4b70 2024-05-09T03:18:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723f59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72400d4f80 con 0x7f72401e8f40 2024-05-09T03:18:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723f59e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f72400ba250 con 0x7f72400d11e0 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 0x7f72400ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 0x7f72400ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41460/0 (socket says 172.21.15.46:41460) 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.550+0000 7f723dd9b640 1 -- 172.21.15.46:0/92534912 learned_addr learned my addr 172.21.15.46:0/92534912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723ed9d640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72401e8f40 0x7f72401e9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723e59c640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72400d11e0 0x7f72401530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723dd9b640 1 -- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72400d11e0 msgr2=0x7f72401530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723dd9b640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72400d11e0 0x7f72401530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723dd9b640 1 -- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72401e8f40 msgr2=0x7f72401e9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723dd9b640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72401e8f40 0x7f72401e9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723dd9b640 1 -- 172.21.15.46:0/92534912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72401e9890 con 0x7f72400d4b70 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723ed9d640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72401e8f40 0x7f72401e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723dd9b640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 0x7f72400ccb80 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f723402a180 tx=0x7f72340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234037040 con 0x7f72400d4b70 2024-05-09T03:18:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7234042070 con 0x7f72400d4b70 2024-05-09T03:18:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72400cd600 con 0x7f72400d4b70 2024-05-09T03:18:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234030bb0 con 0x7f72400d4b70 2024-05-09T03:18:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.551+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72400cdf70 con 0x7f72400d4b70 2024-05-09T03:18:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.552+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72400cd370 con 0x7f72400d4b70 2024-05-09T03:18:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.553+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7234047460 con 0x7f72400d4b70 2024-05-09T03:18:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.554+0000 7f722effd640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7220076300 0x7f7220078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.554+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f72340b8990 con 0x7f72400d4b70 2024-05-09T03:18:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.554+0000 7f723e59c640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7220076300 0x7f7220078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.555+0000 7f723e59c640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7220076300 0x7f7220078810 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f723000a750 tx=0x7f7230008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:07.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.559+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f72401e9890 con 0x7f72400d4b70 2024-05-09T03:18:07.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:07.787+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7240076c00 con 0x7f72400d4b70 2024-05-09T03:18:08.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.159+0000 7f722effd640 1 -- 172.21.15.46:0/92534912 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v199) ==== 161+0+0 (secure 0 0 0) 0x7f7234081c10 con 0x7f72400d4b70 2024-05-09T03:18:08.160 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7220076300 msgr2=0x7f7220078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7220076300 0x7f7220078810 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f723000a750 tx=0x7f7230008040 comp rx=0 tx=0).stop 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 msgr2=0x7f72400ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 0x7f72400ccb80 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f723402a180 tx=0x7f72340046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 shutdown_connections 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72401e8f40 0x7f72401e9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7220076300 0x7f7220078810 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72400d4b70 0x7f72400ccb80 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 --2- 172.21.15.46:0/92534912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72400d11e0 0x7f72401530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 >> 172.21.15.46:0/92534912 conn(0x7f724006e110 msgr2=0x7f7240161e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:08.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 shutdown_connections 2024-05-09T03:18:08.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.161+0000 7f723f59e640 1 -- 172.21.15.46:0/92534912 wait complete. 2024-05-09T03:18:08.174 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:08.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.290+0000 7f9c06938640 1 Processor -- start 2024-05-09T03:18:08.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.290+0000 7f9c06938640 1 -- start start 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.291+0000 7f9c06938640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000b7e70 0x7f9c000b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.291+0000 7f9c06938640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c001605b0 0x7f9c000b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.291+0000 7f9c06938640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c000d3400 con 0x7f9c000d2e90 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.291+0000 7f9c06938640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9c000d4a60 con 0x7f9c001605b0 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.291+0000 7f9c06938640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9c000d4d40 con 0x7f9c000b7e70 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9bfffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9c000d2e90 0x7f9c000b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54422/0 (socket says 172.21.15.46:54422) 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9bfffff640 1 -- 172.21.15.46:0/911887015 learned_addr learned my addr 172.21.15.46:0/911887015 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9c04ed5640 1 --2- 172.21.15.46:0/911887015 >> v2:172.21.15.46:3300/0 conn(0x7f9c001605b0 0x7f9c000b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9bff7fe640 1 --2- 172.21.15.46:0/911887015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000b7e70 0x7f9c000b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9c04ed5640 1 -- 172.21.15.46:0/911887015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000b7e70 msgr2=0x7f9c000b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:08.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9c04ed5640 1 --2- 172.21.15.46:0/911887015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000b7e70 0x7f9c000b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9c04ed5640 1 -- 172.21.15.46:0/911887015 >> v1:172.21.15.46:6789/0 conn(0x7f9c000d2e90 legacy=0x7f9c000b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:08.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9c04ed5640 1 -- 172.21.15.46:0/911887015 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c000d5020 con 0x7f9c001605b0 2024-05-09T03:18:08.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.292+0000 7f9bfeffd640 1 -- 172.21.15.46:0/911887015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798051486 0 0) 0x7f9c000d3400 con 0x7f9c000d2e90 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.293+0000 7f9c04ed5640 1 --2- 172.21.15.46:0/911887015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000b97e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009dc0 tx=0x7f9bf802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10e85d7fcb3f16d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.293+0000 7f9bfeffd640 1 -- 172.21.15.46:0/911887015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf802ae60 con 0x7f9c001605b0 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.293+0000 7f9bfeffd640 1 -- 172.21.15.46:0/911887015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf8007740 con 0x7f9c001605b0 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.293+0000 7f9bfeffd640 1 -- 172.21.15.46:0/911887015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8007b10 con 0x7f9c001605b0 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 -- 172.21.15.46:0/911887015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 msgr2=0x7f9c000b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 --2- 172.21.15.46:0/911887015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000b97e0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8009dc0 tx=0x7f9bf802a680 comp rx=0 tx=0).stop 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 -- 172.21.15.46:0/911887015 shutdown_connections 2024-05-09T03:18:08.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 --2- 172.21.15.46:0/911887015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000b97e0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 --2- 172.21.15.46:0/911887015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000b7e70 0x7f9c000b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 -- 172.21.15.46:0/911887015 >> 172.21.15.46:0/911887015 conn(0x7f9c0008d390 msgr2=0x7f9c0008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 -- 172.21.15.46:0/911887015 shutdown_connections 2024-05-09T03:18:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.294+0000 7f9c06938640 1 -- 172.21.15.46:0/911887015 wait complete. 2024-05-09T03:18:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.295+0000 7f9c06938640 1 Processor -- start 2024-05-09T03:18:08.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.296+0000 7f9c06938640 1 -- start start 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c06938640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000d2e90 0x7f9c0015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c06938640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000cf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c06938640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000ccd20 0x7f9c0014a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c06938640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000b4d80 con 0x7f9c001605b0 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c06938640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000b4c00 con 0x7f9c000d2e90 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c06938640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9c000b4fc0 con 0x7f9c000ccd20 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000cf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000cf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41480/0 (socket says 172.21.15.46:41480) 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 -- 172.21.15.46:0/3093369167 learned_addr learned my addr 172.21.15.46:0/3093369167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bfffff640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000d2e90 0x7f9c0015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9c04ed5640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000ccd20 0x7f9c0014a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 -- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000ccd20 msgr2=0x7f9c0014a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:08.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000ccd20 0x7f9c0014a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 -- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000d2e90 msgr2=0x7f9c0015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000d2e90 0x7f9c0015c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.297+0000 7f9bff7fe640 1 -- 172.21.15.46:0/3093369167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c000ce950 con 0x7f9c001605b0 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.298+0000 7f9bff7fe640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000cf9f0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400d5f0 tx=0x7f9bf400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.298+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf40125f0 con 0x7f9c001605b0 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.298+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf4019070 con 0x7f9c001605b0 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.298+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c000cec30 con 0x7f9c001605b0 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.298+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf40129d0 con 0x7f9c001605b0 2024-05-09T03:18:08.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.298+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c000cf1f0 con 0x7f9c001605b0 2024-05-09T03:18:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.299+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c0006a270 con 0x7f9c001605b0 2024-05-09T03:18:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.300+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9bf4012b70 con 0x7f9c001605b0 2024-05-09T03:18:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.300+0000 7f9bfcff9640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bec0762d0 0x7f9bec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.300+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f9bf4097f00 con 0x7f9c001605b0 2024-05-09T03:18:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.300+0000 7f9bfffff640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bec0762d0 0x7f9bec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:08.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.301+0000 7f9bfffff640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bec0762d0 0x7f9bec0787e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0007fd0 tx=0x7f9bf0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:08.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.305+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9bf4060bc0 con 0x7f9c001605b0 2024-05-09T03:18:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:08.568+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f9c0015f840 con 0x7f9c001605b0 2024-05-09T03:18:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.164+0000 7f9bfcff9640 1 -- 172.21.15.46:0/3093369167 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v200) ==== 101+0+0 (secure 0 0 0) 0x7f9bf4069030 con 0x7f9c001605b0 2024-05-09T03:18:09.165 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bec0762d0 msgr2=0x7f9bec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bec0762d0 0x7f9bec0787e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0007fd0 tx=0x7f9bf0007de0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 msgr2=0x7f9c000cf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000cf9f0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f9bf400d5f0 tx=0x7f9bf400dab0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 shutdown_connections 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9c000ccd20 0x7f9c0014a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bec0762d0 0x7f9bec0787e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c001605b0 0x7f9c000cf9f0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 --2- 172.21.15.46:0/3093369167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000d2e90 0x7f9c0015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 >> 172.21.15.46:0/3093369167 conn(0x7f9c0008d390 msgr2=0x7f9c0015d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 shutdown_connections 2024-05-09T03:18:09.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.167+0000 7f9c06938640 1 -- 172.21.15.46:0/3093369167 wait complete. 2024-05-09T03:18:09.180 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.238+0000 7f00c152ff40 1 Processor -- start 2024-05-09T03:18:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.238+0000 7f00c152ff40 1 -- start start 2024-05-09T03:18:09.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00c152ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac09ab5f0 0x563ac0a2bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00c152ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 0x563ac0ac30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00c152ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a37390 0x563ac0ac3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00c152ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ac09ac8d0 con 0x563ac0a36a40 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00c152ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ac09ac5f0 con 0x563ac09ab5f0 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00c152ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563ac09a7bb0 con 0x563ac0a37390 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 0x563ac0ac30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 0x563ac0ac30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41490/0 (socket says 172.21.15.46:41490) 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 -- 172.21.15.46:0/1151991902 learned_addr learned my addr 172.21.15.46:0/1151991902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00baff9640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac09ab5f0 0x563ac0a2bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a37390 0x563ac0ac3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 -- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a37390 msgr2=0x563ac0ac3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a37390 0x563ac0ac3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 -- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac09ab5f0 msgr2=0x563ac0a2bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac09ab5f0 0x563ac0a2bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00b3fff640 1 -- 172.21.15.46:0/1151991902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac08cb350 con 0x563ac0a36a40 2024-05-09T03:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00baff9640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac09ab5f0 0x563ac0a2bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.239+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a37390 0x563ac0ac3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 0x563ac0ac30e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f00a40094f0 tx=0x7f00a40099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb0aae9f289f461 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00b9ff7640 1 -- 172.21.15.46:0/1151991902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4017070 con 0x563ac0a36a40 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00b9ff7640 1 -- 172.21.15.46:0/1151991902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a4006520 con 0x563ac0a36a40 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00b9ff7640 1 -- 172.21.15.46:0/1151991902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a40068f0 con 0x563ac0a36a40 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 -- 172.21.15.46:0/1151991902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 msgr2=0x563ac0ac30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 0x563ac0ac30e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f00a40094f0 tx=0x7f00a40099b0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 -- 172.21.15.46:0/1151991902 shutdown_connections 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a37390 0x563ac0ac3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac0a36a40 0x563ac0ac30e0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1151991902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac09ab5f0 0x563ac0a2bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 -- 172.21.15.46:0/1151991902 >> 172.21.15.46:0/1151991902 conn(0x563ac0885ac0 msgr2=0x563ac0992a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 -- 172.21.15.46:0/1151991902 shutdown_connections 2024-05-09T03:18:09.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.240+0000 7f00c152ff40 1 -- 172.21.15.46:0/1151991902 wait complete. 2024-05-09T03:18:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.241+0000 7f00c152ff40 1 Processor -- start 2024-05-09T03:18:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.241+0000 7f00c152ff40 1 -- start start 2024-05-09T03:18:09.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00c152ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 0x563ac0a2eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00c152ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a36a40 0x563ac0a28790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00c152ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac0a37390 0x563ac09a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00c152ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ac0a34800 con 0x563ac09ab5f0 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00c152ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ac098e450 con 0x563ac0a37390 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00c152ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563ac0a34a60 con 0x563ac0a36a40 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 0x563ac0a2eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 0x563ac0a2eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41506/0 (socket says 172.21.15.46:41506) 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 learned_addr learned my addr 172.21.15.46:0/1595896485 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac0a37390 0x563ac09a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a36a40 0x563ac0a28790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a36a40 msgr2=0x563ac0a28790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ac0a36a40 0x563ac0a28790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac0a37390 msgr2=0x563ac09a4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac0a37390 0x563ac09a4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac0a35180 con 0x563ac09ab5f0 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ac0a37390 0x563ac09a4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 0x563ac0a2eff0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00be60 tx=0x7f00ac00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.243+0000 7f00b8ff5640 1 -- 172.21.15.46:0/1595896485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac007bb0 con 0x563ac09ab5f0 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.244+0000 7f00b8ff5640 1 -- 172.21.15.46:0/1595896485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00ac007d50 con 0x563ac09ab5f0 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.244+0000 7f00b8ff5640 1 -- 172.21.15.46:0/1595896485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac015660 con 0x563ac09ab5f0 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.244+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563ac0a2e230 con 0x563ac09ab5f0 2024-05-09T03:18:09.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.244+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac0a2e7f0 con 0x563ac09ab5f0 2024-05-09T03:18:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.245+0000 7f00b8ff5640 1 -- 172.21.15.46:0/1595896485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00ac01e440 con 0x563ac09ab5f0 2024-05-09T03:18:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.246+0000 7f00b8ff5640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f009c074ce0 0x7f009c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.246+0000 7f00b8ff5640 1 -- 172.21.15.46:0/1595896485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f00ac095ea0 con 0x563ac09ab5f0 2024-05-09T03:18:09.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.246+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f009c074ce0 0x7f009c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.248 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.246+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x563ac0a35880 0x563ac0a38a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.246+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0a353c0 con 0x563ac0a35880 2024-05-09T03:18:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.246+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x563ac0a35880 0x563ac0a38a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.247+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f009c074ce0 0x7f009c0771f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f00a40094f0 tx=0x7f00a400f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.247+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x563ac0a35880 0x563ac0a38a70 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.249+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v102'536 uv510 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.249+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac09a5950 con 0x563ac0a35880 2024-05-09T03:18:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.250+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v102'537 uv537 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.250+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0adca20 con 0x563ac0a35880 2024-05-09T03:18:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.251+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v102'538 uv538 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.252 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.251+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0adddd0 con 0x563ac0a35880 2024-05-09T03:18:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.253+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v102'539 uv539 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.253+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ade170 con 0x563ac0a35880 2024-05-09T03:18:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.255+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v102'540 uv540 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.255+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0adcdc0 con 0x563ac0a35880 2024-05-09T03:18:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.256+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v102'541 uv541 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.256+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add1b0 con 0x563ac0a35880 2024-05-09T03:18:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.258+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v102'542 uv516 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.258+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0add1b0 con 0x563ac0a35880 2024-05-09T03:18:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.259+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v102'543 uv543 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.259+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add1b0 con 0x563ac0a35880 2024-05-09T03:18:09.261 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.261+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v102'544 uv544 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.261+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add1b0 con 0x563ac0a35880 2024-05-09T03:18:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.263+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v102'545 uv545 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.263+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.264+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v102'546 uv546 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.264+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.266+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v102'547 uv547 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.266+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.267+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v102'548 uv523 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.267+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.269+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v102'549 uv549 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.269+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.270+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v102'550 uv550 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.271 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:09.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.270+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.272+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v102'551 uv551 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.272+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.273+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v102'552 uv552 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.273+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add160 con 0x563ac0a35880 2024-05-09T03:18:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.274+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v102'553 uv553 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.274+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add5c0 con 0x563ac0a35880 2024-05-09T03:18:09.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.276+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v102'554 uv554 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.276+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0adda30 con 0x563ac0a35880 2024-05-09T03:18:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.277+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v102'555 uv529 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.278+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0adda30 con 0x563ac0a35880 2024-05-09T03:18:09.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.279+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v102'556 uv556 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.279+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.280+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v102'557 uv557 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.281 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.280+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.282+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v102'558 uv558 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.282+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.283+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v102'559 uv559 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.283+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.285+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v102'560 uv560 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.285+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.286+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v102'561 uv535 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.286+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.288+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v102'562 uv562 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.288+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.289+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v102'563 uv563 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.290 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:09.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.289+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.291+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v102'564 uv564 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.291+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.292+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v102'565 uv565 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.292+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0a35880 2024-05-09T03:18:09.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.294+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v102'566 uv566 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.294+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563ac0add960 0x563ac0ae0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.294+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0add550 con 0x563ac0add960 2024-05-09T03:18:09.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.294+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563ac0add960 0x563ac0ae0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.295+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563ac0add960 0x563ac0ae0a30 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.296+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v102'393 uv375 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.296+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae0f70 con 0x563ac0add960 2024-05-09T03:18:09.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.297+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v102'394 uv394 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.297+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae1310 con 0x563ac0add960 2024-05-09T03:18:09.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.298+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v102'395 uv395 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.299 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:09.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.298+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae1b60 con 0x563ac0add960 2024-05-09T03:18:09.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.300+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v102'396 uv396 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.300+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae1f00 con 0x563ac0add960 2024-05-09T03:18:09.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.301+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v102'397 uv397 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.301+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae22a0 con 0x563ac0add960 2024-05-09T03:18:09.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.302+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v102'398 uv380 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.302+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.303+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v102'399 uv399 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.303+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.304+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v102'400 uv400 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.305 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:09.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.304+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.306+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v102'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.306+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.307+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v102'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.307+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.308+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v102'403 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.308+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.309+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v102'404 uv404 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.309+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.310+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v102'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.311 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:09.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.310+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae60a0 con 0x563ac0add960 2024-05-09T03:18:09.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.312+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v102'406 uv406 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.312+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae60a0 con 0x563ac0add960 2024-05-09T03:18:09.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.313+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v102'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.313+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.314+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v102'408 uv391 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.314+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.315+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v102'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.315+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.316+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v102'410 uv410 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.317 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:09.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.316+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae3650 con 0x563ac0add960 2024-05-09T03:18:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.318+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v102'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.318+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae3650 con 0x563ac0add960 2024-05-09T03:18:09.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.319+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v102'412 uv412 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.319+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0add960 2024-05-09T03:18:09.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.320+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v102'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0add960 2024-05-09T03:18:09.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.320+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0a35880 2024-05-09T03:18:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.321+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v102'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.321+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0a35880 2024-05-09T03:18:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.322+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v102'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.322+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0a35880 2024-05-09T03:18:09.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.323+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v102'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.324 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:09.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.323+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae3650 con 0x563ac0a35880 2024-05-09T03:18:09.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.324+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v102'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.324+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae3650 con 0x563ac0a35880 2024-05-09T03:18:09.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.325+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v102'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.325+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae2640 con 0x563ac0a35880 2024-05-09T03:18:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.326+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v102'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.326+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563ac0ae2cc0 0x563ac0ae50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.326+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae5660 con 0x563ac0ae2cc0 2024-05-09T03:18:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.326+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563ac0ae2cc0 0x563ac0ae50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.327+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563ac0ae2cc0 0x563ac0ae50e0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.328+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v102'82 uv81 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.328+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae5b70 con 0x563ac0ae2cc0 2024-05-09T03:18:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.329+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v102'83 uv83 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.329+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae6070 con 0x563ac0ae2cc0 2024-05-09T03:18:09.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.330+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v102'84 uv84 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.330 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.330+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae1790 con 0x563ac0ae2cc0 2024-05-09T03:18:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.331+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v102'85 uv85 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.331+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae69a0 con 0x563ac0a35880 2024-05-09T03:18:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.332+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v102'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.332+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae69a0 con 0x563ac0a35880 2024-05-09T03:18:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.334+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v102'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.334+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae69a0 con 0x563ac0a35880 2024-05-09T03:18:09.335 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.335+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v102'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.335+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae79b0 con 0x563ac0a35880 2024-05-09T03:18:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.336+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v102'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.336+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae79b0 con 0x563ac0a35880 2024-05-09T03:18:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.337+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v102'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.337+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae69a0 con 0x563ac0a35880 2024-05-09T03:18:09.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.339+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v102'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.339+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563ac0ae7270 0x563ac0ae9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.339+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9bd0 con 0x563ac0ae7270 2024-05-09T03:18:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.339+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563ac0ae7270 0x563ac0ae9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.339+0000 7f00bb7fa640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563ac0ae7270 0x563ac0ae9690 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.341+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v102'193 uv187 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.341+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae9bd0 con 0x563ac0ae7270 2024-05-09T03:18:09.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.342+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v102'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.342+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.345 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:09.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.343+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v102'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.343+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aeb000 con 0x563ac0ae7270 2024-05-09T03:18:09.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.344+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v102'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.344+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aeb000 con 0x563ac0ae7270 2024-05-09T03:18:09.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.345+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v102'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.345+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.346+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v102'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.346+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.347+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v102'199 uv192 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.347+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.348+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v102'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.349+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.350 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:09.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.350+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v102'201 uv201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.350+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.351+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v102'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.351+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0ae9ff0 con 0x563ac0ae7270 2024-05-09T03:18:09.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.352+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v102'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.352+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563ac0aea500 0x563ac0aec960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.352+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed0b0 con 0x563ac0aea500 2024-05-09T03:18:09.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.352+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563ac0aea500 0x563ac0aec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.353+0000 7f00baff9640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563ac0aea500 0x563ac0aec960 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.354+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v102'86 uv85 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0aea500 2024-05-09T03:18:09.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.354+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0aea500 2024-05-09T03:18:09.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.355+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v102'87 uv87 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0aea500 2024-05-09T03:18:09.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.355+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0aea500 2024-05-09T03:18:09.356 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:09.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.356+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v102'88 uv88 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0aea500 2024-05-09T03:18:09.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.356+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0aea500 2024-05-09T03:18:09.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.357+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v102'89 uv89 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0aea500 2024-05-09T03:18:09.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.357+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0aea500 2024-05-09T03:18:09.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.358+0000 7f00baff9640 1 -- 172.21.15.46:0/1595896485 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v102'90 uv90 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00ac01b020 con 0x563ac0aea500 2024-05-09T03:18:09.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.358+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0a35880 2024-05-09T03:18:09.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.359+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v102'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.359+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0a35880 2024-05-09T03:18:09.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.360+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v102'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.360+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0a35880 2024-05-09T03:18:09.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.361+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v102'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.362 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:09.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.362+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aee490 con 0x563ac0a35880 2024-05-09T03:18:09.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.362+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v102'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.362+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aee490 con 0x563ac0a35880 2024-05-09T03:18:09.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.363+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v102'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.363+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0a35880 2024-05-09T03:18:09.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.364+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v102'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0a35880 2024-05-09T03:18:09.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.364+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aed480 con 0x563ac0ae2cc0 2024-05-09T03:18:09.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.366+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v102'172 uv166 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.366+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0aed890 con 0x563ac0ae2cc0 2024-05-09T03:18:09.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.367+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v102'173 uv173 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.367+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aedc80 con 0x563ac0ae2cc0 2024-05-09T03:18:09.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.368+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v102'174 uv174 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.368 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:09.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.368+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aedc80 con 0x563ac0ae2cc0 2024-05-09T03:18:09.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.369+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v102'175 uv175 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.369+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aedc80 con 0x563ac0ae2cc0 2024-05-09T03:18:09.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.370+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v102'176 uv176 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.370+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aedc80 con 0x563ac0ae2cc0 2024-05-09T03:18:09.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.371+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v102'177 uv171 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.371+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0aedc80 con 0x563ac0ae2cc0 2024-05-09T03:18:09.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.372+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v102'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.372+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aedc80 con 0x563ac0ae2cc0 2024-05-09T03:18:09.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.373+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v102'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.374 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:09.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.373+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0aedcc0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.374+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v102'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.374+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af0f70 con 0x563ac0ae2cc0 2024-05-09T03:18:09.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.375+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v102'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.375+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563ac0aeec90 0x563ac0af15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.375+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0aeec90 2024-05-09T03:18:09.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.375+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563ac0aeec90 0x563ac0af15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.376+0000 7f00b3fff640 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563ac0aeec90 0x563ac0af15d0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.377+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v102'78 uv77 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0aeec90 2024-05-09T03:18:09.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.377+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0aeec90 2024-05-09T03:18:09.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.378+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v102'79 uv79 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0aeec90 2024-05-09T03:18:09.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.378+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0aeec90 2024-05-09T03:18:09.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.379+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v102'80 uv80 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0aeec90 2024-05-09T03:18:09.380 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:09.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.379+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0aeec90 2024-05-09T03:18:09.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.380+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v102'81 uv81 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0aeec90 2024-05-09T03:18:09.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.380+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.382+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v102'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.382+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.383+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v102'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.383+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.384+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v102'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.385 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:09.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.384+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.385+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v102'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.385+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.386+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v102'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.386+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae2cc0 2024-05-09T03:18:09.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.388+0000 7f00b3fff640 1 -- 172.21.15.46:0/1595896485 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v102'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00a401a040 con 0x563ac0ae2cc0 2024-05-09T03:18:09.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.388+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.389+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v102'89 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.389+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.390+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v102'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.390+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.392 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.391+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v102'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.391+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.392+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v102'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.392+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.394+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v102'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.394+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.395+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v102'216 uv208 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.395+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.396+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v102'217 uv217 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.396+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.397+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v102'218 uv218 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.398 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.397+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.399+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v102'219 uv219 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.399+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.400+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v102'220 uv220 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.400+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.401+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v102'221 uv215 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.401+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.402+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v102'222 uv222 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.402+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.403+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v102'223 uv223 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.404 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:09.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.403+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.404+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v102'224 uv224 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.404+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.405+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v102'225 uv225 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.405+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af10b0 con 0x563ac0ae7270 2024-05-09T03:18:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.406+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v102'226 uv226 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.407+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ac0af1b10 con 0x563ac0ae7270 2024-05-09T03:18:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v102'227 uv227 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f00b4051660 con 0x563ac0ae7270 2024-05-09T03:18:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x563ac0a35880 msgr2=0x563ac0a38a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x563ac0a35880 0x563ac0a38a70 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563ac0ae2cc0 msgr2=0x563ac0ae50e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563ac0ae2cc0 0x563ac0ae50e0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563ac0aeec90 msgr2=0x563ac0af15d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563ac0aeec90 0x563ac0af15d0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563ac0add960 msgr2=0x563ac0ae0a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563ac0add960 0x563ac0ae0a30 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563ac0ae7270 msgr2=0x563ac0ae9690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563ac0ae7270 0x563ac0ae9690 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 reap_dead start 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563ac0aea500 msgr2=0x563ac0aec960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563ac0aea500 0x563ac0aec960 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00bb7fa640 1 -- 172.21.15.46:0/1595896485 reap_dead start 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f009c074ce0 msgr2=0x7f009c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f009c074ce0 0x7f009c0771f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f00a40094f0 tx=0x7f00a400f000 comp rx=0 tx=0).stop 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 msgr2=0x563ac0a2eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 0x563ac0a2eff0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00be60 tx=0x7f00ac00bf50 comp rx=0 tx=0).stop 2024-05-09T03:18:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 shutdown_connections 2024-05-09T03:18:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563ac0aea500 0x563ac0aec960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f009c074ce0 0x7f009c0771f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 --2- 172.21.15.46:0/1595896485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ac09ab5f0 0x563ac0a2eff0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 >> 172.21.15.46:0/1595896485 conn(0x563ac0885ac0 msgr2=0x563ac0a2bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 shutdown_connections 2024-05-09T03:18:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.408+0000 7f00c152ff40 1 -- 172.21.15.46:0/1595896485 wait complete. 2024-05-09T03:18:09.412 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 500.00000000 - 2024-05-09T03:18:09.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.507+0000 7fd7e867ef40 1 Processor -- start 2024-05-09T03:18:09.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.507+0000 7fd7e867ef40 1 -- start start 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e867ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 0x55d88b0eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e867ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0ef7b0 0x55d88b0edec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e867ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b106010 0x55d88b0ee400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e867ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d88b1044d0 con 0x55d88b0bd7e0 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e867ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d88b066f50 con 0x55d88b106010 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e867ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d88b1098a0 con 0x55d88b0ef7b0 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 0x55d88b0eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 0x55d88b0eafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41526/0 (socket says 172.21.15.46:41526) 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 -- 172.21.15.46:0/3102061836 learned_addr learned my addr 172.21.15.46:0/3102061836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2947640 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b106010 0x55d88b0ee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7dbfff640 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0ef7b0 0x55d88b0edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 -- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0ef7b0 msgr2=0x55d88b0edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0ef7b0 0x55d88b0edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 -- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b106010 msgr2=0x55d88b0ee400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b106010 0x55d88b0ee400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 -- 172.21.15.46:0/3102061836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d88b194630 con 0x55d88b0bd7e0 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.509+0000 7fd7e2146640 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 0x55d88b0eafc0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400a770 tx=0x7fd7d400a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73462d5a3bb9182 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e1144640 1 -- 172.21.15.46:0/3102061836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4019070 con 0x55d88b0bd7e0 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e1144640 1 -- 172.21.15.46:0/3102061836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d4007600 con 0x55d88b0bd7e0 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e1144640 1 -- 172.21.15.46:0/3102061836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40079f0 con 0x55d88b0bd7e0 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 -- 172.21.15.46:0/3102061836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 msgr2=0x55d88b0eafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 0x55d88b0eafc0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400a770 tx=0x7fd7d400a860 comp rx=0 tx=0).stop 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 -- 172.21.15.46:0/3102061836 shutdown_connections 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b106010 0x55d88b0ee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0ef7b0 0x55d88b0edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/3102061836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b0bd7e0 0x55d88b0eafc0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 -- 172.21.15.46:0/3102061836 >> 172.21.15.46:0/3102061836 conn(0x55d88afe2ac0 msgr2=0x55d88b107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 -- 172.21.15.46:0/3102061836 shutdown_connections 2024-05-09T03:18:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.510+0000 7fd7e867ef40 1 -- 172.21.15.46:0/3102061836 wait complete. 2024-05-09T03:18:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.511+0000 7fd7e867ef40 1 Processor -- start 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.511+0000 7fd7e867ef40 1 -- start start 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.512+0000 7fd7e867ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0bd7e0 0x55d88b0c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.512+0000 7fd7e867ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b0ef7b0 0x55d88b187d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.512+0000 7fd7e867ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b106010 0x55d88b1923d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.512+0000 7fd7e867ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d88b1895e0 con 0x55d88b106010 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.512+0000 7fd7e867ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d88b0ecdd0 con 0x55d88b0ef7b0 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.512+0000 7fd7e867ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d88b108350 con 0x55d88b0bd7e0 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b106010 0x55d88b1923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b106010 0x55d88b1923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41536/0 (socket says 172.21.15.46:41536) 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 -- 172.21.15.46:0/1558249556 learned_addr learned my addr 172.21.15.46:0/1558249556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7dbfff640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b0ef7b0 0x55d88b187d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2146640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0bd7e0 0x55d88b0c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 -- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0bd7e0 msgr2=0x55d88b0c1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0bd7e0 0x55d88b0c1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 -- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b0ef7b0 msgr2=0x55d88b187d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b0ef7b0 0x55d88b187d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 -- 172.21.15.46:0/1558249556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d88b0eec60 con 0x55d88b106010 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7dbfff640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d88b0ef7b0 0x55d88b187d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2146640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d88b0bd7e0 0x55d88b0c1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e2947640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b106010 0x55d88b1923d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x55d88b19a230 tx=0x7fd7dc001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7db7fe640 1 -- 172.21.15.46:0/1558249556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc0020d0 con 0x55d88b106010 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7db7fe640 1 -- 172.21.15.46:0/1558249556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7dc017070 con 0x55d88b106010 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7db7fe640 1 -- 172.21.15.46:0/1558249556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc012610 con 0x55d88b106010 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d88b0eef40 con 0x55d88b106010 2024-05-09T03:18:09.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.513+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d88b0c1970 con 0x55d88b106010 2024-05-09T03:18:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.515+0000 7fd7db7fe640 1 -- 172.21.15.46:0/1558249556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd7dc0127b0 con 0x55d88b106010 2024-05-09T03:18:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.515+0000 7fd7db7fe640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7c4074e50 0x7fd7c4077360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7db7fe640 1 -- 172.21.15.46:0/1558249556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7fd7dc096690 con 0x55d88b106010 2024-05-09T03:18:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7e2146640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7c4074e50 0x7fd7c4077360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d88b1885f0 0x55d88b197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x55d88b192e20 con 0x55d88b1885f0 2024-05-09T03:18:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7dbfff640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d88b1885f0 0x55d88b197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7e2146640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7c4074e50 0x7fd7c4077360 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4000c00 tx=0x7fd7d400c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.516+0000 7fd7dbfff640 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d88b1885f0 0x55d88b197de0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7dbfff640 1 -- 172.21.15.46:0/1558249556 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 500.00000000 [writefull 0~80] v102'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd7cc05c040 con 0x55d88b1885f0 2024-05-09T03:18:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d88b1885f0 msgr2=0x55d88b197de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d88b1885f0 0x55d88b197de0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7c4074e50 msgr2=0x7fd7c4077360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7c4074e50 0x7fd7c4077360 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4000c00 tx=0x7fd7d400c000 comp rx=0 tx=0).stop 2024-05-09T03:18:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b106010 msgr2=0x55d88b1923d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 --2- 172.21.15.46:0/1558249556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d88b106010 0x55d88b1923d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x55d88b19a230 tx=0x7fd7dc001830 comp rx=0 tx=0).stop 2024-05-09T03:18:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e2947640 1 -- 172.21.15.46:0/1558249556 reap_dead start 2024-05-09T03:18:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 shutdown_connections 2024-05-09T03:18:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 >> 172.21.15.46:0/1558249556 conn(0x55d88afe2ac0 msgr2=0x55d88b1909f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 shutdown_connections 2024-05-09T03:18:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.518+0000 7fd7e867ef40 1 -- 172.21.15.46:0/1558249556 wait complete. 2024-05-09T03:18:09.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.678+0000 7f3fac62b640 1 Processor -- start 2024-05-09T03:18:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.678+0000 7f3fac62b640 1 -- start start 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fac62b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3fa40d3730 0x7f3fa406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fac62b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fa406e7e0 0x7f3fa4164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fac62b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa4164ec0 0x7f3fa41652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fac62b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3fa4165810 con 0x7f3fa406e7e0 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fac62b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3fa409cfe0 con 0x7f3fa40d3730 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fac62b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa40b94a0 con 0x7f3fa4164ec0 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fa9bc6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fa406e7e0 0x7f3fa4164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fa9bc6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3fa406e7e0 0x7f3fa4164980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41538/0 (socket says 172.21.15.46:41538) 2024-05-09T03:18:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.679+0000 7f3fa9bc6640 1 -- 172.21.15.46:0/4060782674 learned_addr learned my addr 172.21.15.46:0/4060782674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3faabc8640 1 --2- 172.21.15.46:0/4060782674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa4164ec0 0x7f3fa41652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3faa3c7640 1 --2- 172.21.15.46:0/4060782674 >> v2:172.21.15.88:3300/0 conn(0x7f3fa40d3730 0x7f3fa406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3fa9bc6640 1 -- 172.21.15.46:0/4060782674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa4164ec0 msgr2=0x7f3fa41652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3fa9bc6640 1 --2- 172.21.15.46:0/4060782674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa4164ec0 0x7f3fa41652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3fa9bc6640 1 -- 172.21.15.46:0/4060782674 >> v2:172.21.15.88:3300/0 conn(0x7f3fa40d3730 msgr2=0x7f3fa406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3fa9bc6640 1 --2- 172.21.15.46:0/4060782674 >> v2:172.21.15.88:3300/0 conn(0x7f3fa40d3730 0x7f3fa406e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3fa9bc6640 1 -- 172.21.15.46:0/4060782674 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa409d470 con 0x7f3fa406e7e0 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.680+0000 7f3fa9bc6640 1 --2- 172.21.15.46:0/4060782674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa406e7e0 0x7f3fa4164980 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3fa406d990 tx=0x7f3f9402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152142202073d493 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fa93c5640 1 -- 172.21.15.46:0/4060782674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94037070 con 0x7f3fa406e7e0 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fa93c5640 1 -- 172.21.15.46:0/4060782674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f94032440 con 0x7f3fa406e7e0 2024-05-09T03:18:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fa93c5640 1 -- 172.21.15.46:0/4060782674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94032810 con 0x7f3fa406e7e0 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 -- 172.21.15.46:0/4060782674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa406e7e0 msgr2=0x7f3fa4164980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 --2- 172.21.15.46:0/4060782674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa406e7e0 0x7f3fa4164980 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f3fa406d990 tx=0x7f3f9402a140 comp rx=0 tx=0).stop 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 -- 172.21.15.46:0/4060782674 shutdown_connections 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 --2- 172.21.15.46:0/4060782674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa4164ec0 0x7f3fa41652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 --2- 172.21.15.46:0/4060782674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa406e7e0 0x7f3fa4164980 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 --2- 172.21.15.46:0/4060782674 >> v2:172.21.15.88:3300/0 conn(0x7f3fa40d3730 0x7f3fa406e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 -- 172.21.15.46:0/4060782674 >> 172.21.15.46:0/4060782674 conn(0x7f3fa40d05f0 msgr2=0x7f3fa40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.681+0000 7f3fac62b640 1 -- 172.21.15.46:0/4060782674 shutdown_connections 2024-05-09T03:18:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.682+0000 7f3fac62b640 1 -- 172.21.15.46:0/4060782674 wait complete. 2024-05-09T03:18:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.683+0000 7f3fac62b640 1 Processor -- start 2024-05-09T03:18:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.683+0000 7f3fac62b640 1 -- start start 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fac62b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fa406e7e0 0x7f3fa415f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fac62b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa40d3730 0x7f3fa41f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fac62b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 0x7f3fa40cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fac62b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40b9160 con 0x7f3fa4164ec0 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fac62b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40d1490 con 0x7f3fa406e7e0 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fac62b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa408cd60 con 0x7f3fa40d3730 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 0x7f3fa40cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 0x7f3fa40cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41544/0 (socket says 172.21.15.46:41544) 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 -- 172.21.15.46:0/993299940 learned_addr learned my addr 172.21.15.46:0/993299940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faa3c7640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fa406e7e0 0x7f3fa415f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3fa9bc6640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa40d3730 0x7f3fa41f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 -- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa40d3730 msgr2=0x7f3fa41f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa40d3730 0x7f3fa41f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 -- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fa406e7e0 msgr2=0x7f3fa415f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fa406e7e0 0x7f3fa415f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 -- 172.21.15.46:0/993299940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa40cc090 con 0x7f3fa4164ec0 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.685+0000 7f3faabc8640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 0x7f3fa40cd520 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009fd0 tx=0x7f3f9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.686+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c014420 con 0x7f3fa4164ec0 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.686+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f9c019070 con 0x7f3fa4164ec0 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.686+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c014800 con 0x7f3fa4164ec0 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.686+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa40cc760 con 0x7f3fa4164ec0 2024-05-09T03:18:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.686+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa40ccd20 con 0x7f3fa4164ec0 2024-05-09T03:18:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.687+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa40b7730 con 0x7f3fa4164ec0 2024-05-09T03:18:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.687+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3f9c0149a0 con 0x7f3fa4164ec0 2024-05-09T03:18:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.688+0000 7f3f9affd640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3f880762d0 0x7f3f880787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.688+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f3f9c0982c0 con 0x7f3fa4164ec0 2024-05-09T03:18:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.688+0000 7f3faa3c7640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3f880762d0 0x7f3f880787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.689+0000 7f3faa3c7640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3f880762d0 0x7f3f880787e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0003610 tx=0x7f3fa0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.693+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3fa40b7730 con 0x7f3fa4164ec0 2024-05-09T03:18:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.925+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f3fa40d1810 con 0x7f3fa4164ec0 2024-05-09T03:18:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.990+0000 7f3f9affd640 1 -- 172.21.15.46:0/993299940 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v201) ==== 162+0+0 (secure 0 0 0) 0x7f3f9c061010 con 0x7f3fa4164ec0 2024-05-09T03:18:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3f880762d0 msgr2=0x7f3f880787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3f880762d0 0x7f3f880787e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0003610 tx=0x7f3fa0008040 comp rx=0 tx=0).stop 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 msgr2=0x7f3fa40cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 0x7f3fa40cd520 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009fd0 tx=0x7f3f9c00e4b0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 shutdown_connections 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3f880762d0 0x7f3f880787e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fa4164ec0 0x7f3fa40cd520 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3fa40d3730 0x7f3fa41f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 --2- 172.21.15.46:0/993299940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fa406e7e0 0x7f3fa415f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:09.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 >> 172.21.15.46:0/993299940 conn(0x7f3fa40d05f0 msgr2=0x7f3fa4072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:09.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 shutdown_connections 2024-05-09T03:18:09.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:09.993+0000 7f3fac62b640 1 -- 172.21.15.46:0/993299940 wait complete. 2024-05-09T03:18:09.995 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:09.995+0000 7f2421472640 -1 mds.0.purge_queue operator(): Error -22 loading Journaler 2024-05-09T03:18:09.995 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:09.995+0000 7f2421472640 -1 mds.0.201 unhandled write error (22) Invalid argument, force readonly... 2024-05-09T03:18:10.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:10.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.120+0000 7fb38ff3f640 1 Processor -- start 2024-05-09T03:18:10.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.121+0000 7fb38ff3f640 1 -- start start 2024-05-09T03:18:10.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38ff3f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 0x7fb3880b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38ff3f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3880b8110 con 0x7fb3880b5210 2024-05-09T03:18:10.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38ff3f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3881617d0 con 0x7fb3880b7d00 2024-05-09T03:18:10.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38ff3f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb3880b5620 con 0x7fb3881613c0 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38dcdb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb3880b5210 0x7fb3880d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54430/0 (socket says 172.21.15.46:54430) 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38dcdb640 1 -- 172.21.15.46:0/2630780473 learned_addr learned my addr 172.21.15.46:0/2630780473 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361128663 0 0) 0x7fb380001110 con 0x7fb3880b5210 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.122+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb370003520 con 0x7fb3880b5210 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38d4da640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 0x7fb3880b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2295804747 0 0) 0x7fb370003520 con 0x7fb3880b5210 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 msgr2=0x7fb3880b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 0x7fb3880b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 >> v1:172.21.15.88:6789/0 conn(0x7fb3880b7d00 legacy=0x7fb388163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3880d5280 con 0x7fb3880b5210 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fb3800018c0 con 0x7fb3880b5210 2024-05-09T03:18:10.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 >> v1:172.21.15.46:6789/0 conn(0x7fb3880b5210 legacy=0x7fb3880d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003a30 0x7fb370003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 0x7fb370004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb370004d40 0x7fb3700051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3880b5620 con 0x7fb370004d40 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb37000bb80 con 0x7fb370003a30 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb37000bd80 con 0x7fb3881613c0 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb370004d40 0x7fb3700051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38dcdb640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003a30 0x7fb370003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38d4da640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 0x7fb370004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38e4dc640 1 -- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 msgr2=0x7fb370004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:10.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3881613c0 0x7fb370004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38e4dc640 1 -- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003a30 msgr2=0x7fb370003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb370003a30 0x7fb370003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.123+0000 7fb38e4dc640 1 -- 172.21.15.46:0/2630780473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb370003520 con 0x7fb370004d40 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.124+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb370004d40 0x7fb3700051f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fb38808ea90 tx=0x7fb370005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636f484a302a71a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.124+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378001ab0 con 0x7fb370004d40 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.124+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb37802adf0 con 0x7fb370004d40 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.124+0000 7fb38ccd9640 1 -- 172.21.15.46:0/2630780473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378003b70 con 0x7fb370004d40 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38ff3f640 1 -- 172.21.15.46:0/2630780473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb370004d40 msgr2=0x7fb3700051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/2630780473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb370004d40 0x7fb3700051f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fb38808ea90 tx=0x7fb370005830 comp rx=0 tx=0).stop 2024-05-09T03:18:10.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38e4dc640 1 -- 172.21.15.46:0/2630780473 reap_dead start 2024-05-09T03:18:10.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38ff3f640 1 -- 172.21.15.46:0/2630780473 shutdown_connections 2024-05-09T03:18:10.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38ff3f640 1 -- 172.21.15.46:0/2630780473 >> 172.21.15.46:0/2630780473 conn(0x7fb3880d29a0 msgr2=0x7fb3880d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:10.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38ff3f640 1 -- 172.21.15.46:0/2630780473 shutdown_connections 2024-05-09T03:18:10.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.125+0000 7fb38ff3f640 1 -- 172.21.15.46:0/2630780473 wait complete. 2024-05-09T03:18:10.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.127+0000 7fb38ff3f640 1 Processor -- start 2024-05-09T03:18:10.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.127+0000 7fb38ff3f640 1 -- start start 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38ff3f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb370004d40 0x7fb3881523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38ff3f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f7040 0x7fb3880cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38ff3f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 0x7fb3880cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38ff3f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb388160bc0 con 0x7fb3880cf1d0 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38ff3f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3881609a0 con 0x7fb3881f7040 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38ff3f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb388160ea0 con 0x7fb370004d40 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38e4dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 0x7fb3880cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38e4dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 0x7fb3880cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41572/0 (socket says 172.21.15.46:41572) 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.128+0000 7fb38e4dc640 1 -- 172.21.15.46:0/3417692490 learned_addr learned my addr 172.21.15.46:0/3417692490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38d4da640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f7040 0x7fb3880cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38dcdb640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb370004d40 0x7fb3881523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38e4dc640 1 -- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb370004d40 msgr2=0x7fb3881523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb370004d40 0x7fb3881523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38e4dc640 1 -- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f7040 msgr2=0x7fb3880cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f7040 0x7fb3880cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38e4dc640 1 -- 172.21.15.46:0/3417692490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3880ce130 con 0x7fb3880cf1d0 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38dcdb640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb370004d40 0x7fb3881523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38e4dc640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 0x7fb3880cf5e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fb37802a8a0 tx=0x7fb3780014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378037040 con 0x7fb3880cf1d0 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb378040070 con 0x7fb3880cf1d0 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3880ce410 con 0x7fb3880cf1d0 2024-05-09T03:18:10.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378007df0 con 0x7fb3880cf1d0 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.129+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3880ce9d0 con 0x7fb3880cf1d0 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.130+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb38806a270 con 0x7fb3880cf1d0 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.131+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb3780052f0 con 0x7fb3880cf1d0 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.131+0000 7fb3867fc640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3740762d0 0x7fb3740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.131+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7fb3780b7880 con 0x7fb3880cf1d0 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.132+0000 7fb38dcdb640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3740762d0 0x7fb3740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.132+0000 7fb38dcdb640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3740762d0 0x7fb3740787e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb380009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:10.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.136+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb378080c20 con 0x7fb3880cf1d0 2024-05-09T03:18:10.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.365+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3880d40d0 con 0x7fb3880cf1d0 2024-05-09T03:18:10.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.366+0000 7fb3867fc640 1 -- 172.21.15.46:0/3417692490 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) ==== 95+0+5178 (secure 0 0 0) 0x7fb378080520 con 0x7fb3880cf1d0 2024-05-09T03:18:10.367 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:10.367 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":201,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:09.989745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8751":{"gid":8751,"name":"c","rank":0,"incarnation":201,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/3106969477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3106969477},{"type":"v1","addr":"172.21.15.46:6834","nonce":3106969477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:10.367 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 201 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3740762d0 msgr2=0x7fb3740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3740762d0 0x7fb3740787e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb380009040 comp rx=0 tx=0).stop 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 msgr2=0x7fb3880cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 0x7fb3880cf5e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fb37802a8a0 tx=0x7fb3780014a0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 shutdown_connections 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3740762d0 0x7fb3740787e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3880cf1d0 0x7fb3880cf5e0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3881f7040 0x7fb3880cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 --2- 172.21.15.46:0/3417692490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb370004d40 0x7fb3881523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 >> 172.21.15.46:0/3417692490 conn(0x7fb3880d29a0 msgr2=0x7fb38815f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 shutdown_connections 2024-05-09T03:18:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.368+0000 7fb38ff3f640 1 -- 172.21.15.46:0/3417692490 wait complete. 2024-05-09T03:18:10.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:10.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.903+0000 7f1babc61640 1 Processor -- start 2024-05-09T03:18:10.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.903+0000 7f1babc61640 1 -- start start 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.904+0000 7f1babc61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ba40b8d10 0x7f1ba40b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.904+0000 7f1babc61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 0x7f1ba408d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1babc61640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ba408d910 con 0x7f1ba40b0a00 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1babc61640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba40d2270 con 0x7f1ba40b8d10 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1babc61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba40d2510 con 0x7f1ba408cfc0 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 0x7f1ba408d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba99fd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ba40b8d10 0x7f1ba40b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 0x7f1ba408d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:54714/0 (socket says 172.21.15.46:54714) 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 -- 172.21.15.46:0/2748384426 learned_addr learned my addr 172.21.15.46:0/2748384426 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1baa1fe640 1 --1- 172.21.15.46:0/2748384426 >> v1:172.21.15.88:6789/0 conn(0x7f1ba40b0a00 0x7f1ba40b0e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:43118/0 (socket says 172.21.15.46:43118) 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 -- 172.21.15.46:0/2748384426 >> v2:172.21.15.88:3300/0 conn(0x7f1ba40b8d10 msgr2=0x7f1ba40b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 --2- 172.21.15.46:0/2748384426 >> v2:172.21.15.88:3300/0 conn(0x7f1ba40b8d10 0x7f1ba40b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 -- 172.21.15.46:0/2748384426 >> v1:172.21.15.88:6789/0 conn(0x7f1ba40b0a00 legacy=0x7f1ba40b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba91fc640 1 -- 172.21.15.46:0/2748384426 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba40d27f0 con 0x7f1ba408cfc0 2024-05-09T03:18:10.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.905+0000 7f1ba99fd640 1 --2- 172.21.15.46:0/2748384426 >> v2:172.21.15.88:3300/0 conn(0x7f1ba40b8d10 0x7f1ba40b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.906+0000 7f1ba91fc640 1 --2- 172.21.15.46:0/2748384426 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 0x7f1ba408d3d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009870 tx=0x7f1b9402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3109f7e246bb4493 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.906+0000 7f1ba89fb640 1 -- 172.21.15.46:0/2748384426 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b94037070 con 0x7f1ba408cfc0 2024-05-09T03:18:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.906+0000 7f1ba89fb640 1 -- 172.21.15.46:0/2748384426 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b94032440 con 0x7f1ba408cfc0 2024-05-09T03:18:10.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.906+0000 7f1ba89fb640 1 -- 172.21.15.46:0/2748384426 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b94032810 con 0x7f1ba408cfc0 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 -- 172.21.15.46:0/2748384426 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 msgr2=0x7f1ba408d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 --2- 172.21.15.46:0/2748384426 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 0x7f1ba408d3d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009870 tx=0x7f1b9402a270 comp rx=0 tx=0).stop 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 -- 172.21.15.46:0/2748384426 shutdown_connections 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 --2- 172.21.15.46:0/2748384426 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba408cfc0 0x7f1ba408d3d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 --2- 172.21.15.46:0/2748384426 >> v2:172.21.15.88:3300/0 conn(0x7f1ba40b8d10 0x7f1ba40b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 -- 172.21.15.46:0/2748384426 >> 172.21.15.46:0/2748384426 conn(0x7f1ba40b56e0 msgr2=0x7f1ba40b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 -- 172.21.15.46:0/2748384426 shutdown_connections 2024-05-09T03:18:10.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.907+0000 7f1babc61640 1 -- 172.21.15.46:0/2748384426 wait complete. 2024-05-09T03:18:10.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.909+0000 7f1babc61640 1 Processor -- start 2024-05-09T03:18:10.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.909+0000 7f1babc61640 1 -- start start 2024-05-09T03:18:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1babc61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 0x7f1ba415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1babc61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba40b0a00 0x7f1ba41540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1babc61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ba40b8d10 0x7f1ba40cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1babc61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba40b9380 con 0x7f1ba408cfc0 2024-05-09T03:18:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1babc61640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba40cfee0 con 0x7f1ba40b8d10 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1babc61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba40b9500 con 0x7f1ba40b0a00 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1ba99fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 0x7f1ba415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1ba99fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 0x7f1ba415eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41574/0 (socket says 172.21.15.46:41574) 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.910+0000 7f1ba99fd640 1 -- 172.21.15.46:0/836362474 learned_addr learned my addr 172.21.15.46:0/836362474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba91fc640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba40b0a00 0x7f1ba41540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1baa1fe640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ba40b8d10 0x7f1ba40cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba99fd640 1 -- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba40b0a00 msgr2=0x7f1ba41540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba99fd640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba40b0a00 0x7f1ba41540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba99fd640 1 -- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ba40b8d10 msgr2=0x7f1ba40cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba99fd640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ba40b8d10 0x7f1ba40cc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba99fd640 1 -- 172.21.15.46:0/836362474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba415eff0 con 0x7f1ba408cfc0 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1baa1fe640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ba40b8d10 0x7f1ba40cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1ba99fd640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 0x7f1ba415eab0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000e900 tx=0x7f1ba000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000ca00 con 0x7f1ba408cfc0 2024-05-09T03:18:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba000cba0 con 0x7f1ba408cfc0 2024-05-09T03:18:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba4154620 con 0x7f1ba408cfc0 2024-05-09T03:18:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0005990 con 0x7f1ba408cfc0 2024-05-09T03:18:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.911+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba415f7e0 con 0x7f1ba408cfc0 2024-05-09T03:18:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.912+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ba40b81f0 con 0x7f1ba408cfc0 2024-05-09T03:18:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.913+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ba0005b30 con 0x7f1ba408cfc0 2024-05-09T03:18:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.914+0000 7f1b9a7fc640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b88076300 0x7f1b88078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.914+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f1ba0097f60 con 0x7f1ba408cfc0 2024-05-09T03:18:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.914+0000 7f1ba91fc640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b88076300 0x7f1b88078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.914+0000 7f1ba91fc640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b88076300 0x7f1b88078810 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1b94034000 tx=0x7f1b9402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:10.918+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1ba40b81f0 con 0x7f1ba408cfc0 2024-05-09T03:18:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.152+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ba406a270 con 0x7f1ba408cfc0 2024-05-09T03:18:11.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.153+0000 7f1b9a7fc640 1 -- 172.21.15.46:0/836362474 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+5181 (secure 0 0 0) 0x7f1ba0017020 con 0x7f1ba408cfc0 2024-05-09T03:18:11.154 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:11.154 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":202,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:10.000383+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8751":{"gid":8751,"name":"c","rank":0,"incarnation":201,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.46:6834/3106969477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3106969477},{"type":"v1","addr":"172.21.15.46:6834","nonce":3106969477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:11.154 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 202 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b88076300 msgr2=0x7f1b88078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b88076300 0x7f1b88078810 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1b94034000 tx=0x7f1b9402e000 comp rx=0 tx=0).stop 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 msgr2=0x7f1ba415eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 0x7f1ba415eab0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f1ba000e900 tx=0x7f1ba000edc0 comp rx=0 tx=0).stop 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 shutdown_connections 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ba40b8d10 0x7f1ba40cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1ba40b0a00 0x7f1ba41540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1b88076300 0x7f1b88078810 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 --2- 172.21.15.46:0/836362474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba408cfc0 0x7f1ba415eab0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 >> 172.21.15.46:0/836362474 conn(0x7f1ba40b56e0 msgr2=0x7f1ba40c95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 shutdown_connections 2024-05-09T03:18:11.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:11.155+0000 7f1babc61640 1 -- 172.21.15.46:0/836362474 wait complete. 2024-05-09T03:18:11.167 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 202, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:18:10.000383+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 102, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8751}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8751': {'gid': 8751, 'name': 'c', 'rank': 0, 'incarnation': 201, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.46:6834/3106969477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 3106969477}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 3106969477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:18:11.167 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8751:up:reconnect 2024-05-09T03:18:11.167 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:18:16.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:16.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.298+0000 7f21b71c7640 1 Processor -- start 2024-05-09T03:18:16.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.298+0000 7f21b71c7640 1 -- start start 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.299+0000 7f21b71c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00d57a0 0x7f21b0160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.299+0000 7f21b71c7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f21b00b4af0 0x7f21b00b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.299+0000 7f21b71c7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b00d49e0 con 0x7f21b00b77e0 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.299+0000 7f21b71c7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f21b00d5bb0 con 0x7f21b00b4af0 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.299+0000 7f21b71c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f21b00d4cc0 con 0x7f21b00d57a0 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21b5764640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f21b00b77e0 0x7f21b00b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48458/0 (socket says 172.21.15.46:48458) 2024-05-09T03:18:16.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21b5764640 1 -- 172.21.15.46:0/3737904015 learned_addr learned my addr 172.21.15.46:0/3737904015 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a7fff640 1 --2- 172.21.15.46:0/3737904015 >> v2:172.21.15.88:3300/0 conn(0x7f21b00b4af0 0x7f21b00b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21b4f63640 1 --2- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00d57a0 0x7f21b0160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a77fe640 1 -- 172.21.15.46:0/3737904015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114971120 0 0) 0x7f21b00d49e0 con 0x7f21b00b77e0 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a77fe640 1 -- 172.21.15.46:0/3737904015 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2198003520 con 0x7f21b00b77e0 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a7fff640 1 -- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00d57a0 msgr2=0x7f21b0160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a7fff640 1 --2- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00d57a0 0x7f21b0160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a7fff640 1 -- 172.21.15.46:0/3737904015 >> v1:172.21.15.46:6789/0 conn(0x7f21b00b77e0 legacy=0x7f21b00b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:16.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.300+0000 7f21a7fff640 1 -- 172.21.15.46:0/3737904015 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b00d4fa0 con 0x7f21b00b4af0 2024-05-09T03:18:16.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.301+0000 7f21a7fff640 1 --2- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00b4af0 0x7f21b00b4f00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f21a00097a0 tx=0x7f21a002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa1ecb2a0ee10ad6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:16.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.301+0000 7f21a77fe640 1 -- 172.21.15.46:0/3737904015 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0037070 con 0x7f21b00b4af0 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.301+0000 7f21a77fe640 1 -- 172.21.15.46:0/3737904015 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a0032440 con 0x7f21b00b4af0 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.301+0000 7f21a77fe640 1 -- 172.21.15.46:0/3737904015 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0032810 con 0x7f21b00b4af0 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 -- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00b4af0 msgr2=0x7f21b00b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00b4af0 0x7f21b00b4f00 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f21a00097a0 tx=0x7f21a002a180 comp rx=0 tx=0).stop 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 -- 172.21.15.46:0/3737904015 shutdown_connections 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00b4af0 0x7f21b00b4f00 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3737904015 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00d57a0 0x7f21b0160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 -- 172.21.15.46:0/3737904015 >> 172.21.15.46:0/3737904015 conn(0x7f21b00743e0 msgr2=0x7f21b00d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 -- 172.21.15.46:0/3737904015 shutdown_connections 2024-05-09T03:18:16.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.302+0000 7f21b71c7640 1 -- 172.21.15.46:0/3737904015 wait complete. 2024-05-09T03:18:16.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.304+0000 7f21b71c7640 1 Processor -- start 2024-05-09T03:18:16.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.304+0000 7f21b71c7640 1 -- start start 2024-05-09T03:18:16.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b71c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 0x7f21b0156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b71c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00b77e0 0x7f21b01528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b71c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00d57a0 0x7f21b0157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b71c7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f21b00d1920 con 0x7f21b00b4af0 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b71c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f21b00d15b0 con 0x7f21b00d57a0 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b71c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f21b01f7140 con 0x7f21b00b77e0 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.305+0000 7f21b4f63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 0x7f21b0156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 0x7f21b0156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50458/0 (socket says 172.21.15.46:50458) 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 -- 172.21.15.46:0/3266149652 learned_addr learned my addr 172.21.15.46:0/3266149652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b5764640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00d57a0 0x7f21b0157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21a7fff640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00b77e0 0x7f21b01528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 -- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00b77e0 msgr2=0x7f21b01528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00b77e0 0x7f21b01528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 -- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00d57a0 msgr2=0x7f21b0157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:16.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00d57a0 0x7f21b0157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 -- 172.21.15.46:0/3266149652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b0159030 con 0x7f21b00b4af0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b4f63640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 0x7f21b0156a30 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00e870 tx=0x7f21ac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac017070 con 0x7f21b00b4af0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21ac00c940 con 0x7f21b00b4af0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b0159310 con 0x7f21b00b4af0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00cd10 con 0x7f21b00b4af0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.306+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b0157430 con 0x7f21b00b4af0 2024-05-09T03:18:16.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.307+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b006a270 con 0x7f21b00b4af0 2024-05-09T03:18:16.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.308+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f21ac01a070 con 0x7f21b00b4af0 2024-05-09T03:18:16.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.309+0000 7f21a57fa640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f219c0762d0 0x7f219c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:16.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.309+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f21ac097dd0 con 0x7f21b00b4af0 2024-05-09T03:18:16.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.309+0000 7f21a7fff640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f219c0762d0 0x7f219c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:16.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.309+0000 7f21a7fff640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f219c0762d0 0x7f219c0787e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f21a0034000 tx=0x7f21a002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:16.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.313+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f21ac060a80 con 0x7f21b00b4af0 2024-05-09T03:18:16.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.541+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21b00ba8a0 con 0x7f21b00b4af0 2024-05-09T03:18:16.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.541+0000 7f21a57fa640 1 -- 172.21.15.46:0/3266149652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+5185 (secure 0 0 0) 0x7f21ac064d80 con 0x7f21b00b4af0 2024-05-09T03:18:16.542 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:16.542 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:12.993836+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8751":{"gid":8751,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/3106969477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3106969477},{"type":"v1","addr":"172.21.15.46:6834","nonce":3106969477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8751,"qdb_cluster":[8751]},"id":6}]} 2024-05-09T03:18:16.543 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 204 2024-05-09T03:18:16.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.543+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f219c0762d0 msgr2=0x7f219c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:16.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.543+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f219c0762d0 0x7f219c0787e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f21a0034000 tx=0x7f21a002e000 comp rx=0 tx=0).stop 2024-05-09T03:18:16.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.543+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 msgr2=0x7f21b0156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:16.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.543+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 0x7f21b0156a30 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00e870 tx=0x7f21ac00ed30 comp rx=0 tx=0).stop 2024-05-09T03:18:16.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.543+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 shutdown_connections 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f21b00d57a0 0x7f21b0157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f21b00b77e0 0x7f21b01528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f219c0762d0 0x7f219c0787e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 --2- 172.21.15.46:0/3266149652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21b00b4af0 0x7f21b0156a30 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 >> 172.21.15.46:0/3266149652 conn(0x7f21b00743e0 msgr2=0x7f21b00badc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 shutdown_connections 2024-05-09T03:18:16.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:16.544+0000 7f21b71c7640 1 -- 172.21.15.46:0/3266149652 wait complete. 2024-05-09T03:18:16.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.080+0000 7fe16bebd640 1 Processor -- start 2024-05-09T03:18:17.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.081+0000 7fe16bebd640 1 -- start start 2024-05-09T03:18:17.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe16bebd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe164164c70 0x7fe1641615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe16bebd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1640d2480 0x7fe1640d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe16bebd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe164160b20 con 0x7fe16415dfe0 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe16bebd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe164161150 con 0x7fe164164c70 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe16bebd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1640d15f0 con 0x7fe1640d2480 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe169c59640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe164164c70 0x7fe1641615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe169458640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe16415dfe0 0x7fe1640d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48470/0 (socket says 172.21.15.46:48470) 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe169458640 1 -- 172.21.15.46:0/3652929639 learned_addr learned my addr 172.21.15.46:0/3652929639 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.082+0000 7fe16a45a640 1 --2- 172.21.15.46:0/3652929639 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1640d2480 0x7fe1640d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.083+0000 7fe169c59640 1 -- 172.21.15.46:0/3652929639 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1640d2480 msgr2=0x7fe1640d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.083+0000 7fe169c59640 1 --2- 172.21.15.46:0/3652929639 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1640d2480 0x7fe1640d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.083+0000 7fe169c59640 1 -- 172.21.15.46:0/3652929639 >> v1:172.21.15.46:6789/0 conn(0x7fe16415dfe0 legacy=0x7fe1640d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.083+0000 7fe168c57640 1 -- 172.21.15.46:0/3652929639 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1167991734 0 0) 0x7fe164160b20 con 0x7fe16415dfe0 2024-05-09T03:18:17.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.083+0000 7fe169c59640 1 -- 172.21.15.46:0/3652929639 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1640d18d0 con 0x7fe164164c70 2024-05-09T03:18:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.083+0000 7fe169c59640 1 --2- 172.21.15.46:0/3652929639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe164164c70 0x7fe1641615e0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097a0 tx=0x7fe16002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3953ff5c8f6ea1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.084+0000 7fe168c57640 1 -- 172.21.15.46:0/3652929639 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160037070 con 0x7fe164164c70 2024-05-09T03:18:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.084+0000 7fe168c57640 1 -- 172.21.15.46:0/3652929639 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe160032440 con 0x7fe164164c70 2024-05-09T03:18:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.084+0000 7fe168c57640 1 -- 172.21.15.46:0/3652929639 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160032810 con 0x7fe164164c70 2024-05-09T03:18:17.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.084+0000 7fe16bebd640 1 -- 172.21.15.46:0/3652929639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe164164c70 msgr2=0x7fe1641615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.084+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3652929639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe164164c70 0x7fe1641615e0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe1600097a0 tx=0x7fe16002a180 comp rx=0 tx=0).stop 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.085+0000 7fe16bebd640 1 -- 172.21.15.46:0/3652929639 shutdown_connections 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.085+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3652929639 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe1640d2480 0x7fe1640d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.085+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3652929639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe164164c70 0x7fe1641615e0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.085+0000 7fe16bebd640 1 -- 172.21.15.46:0/3652929639 >> 172.21.15.46:0/3652929639 conn(0x7fe164073db0 msgr2=0x7fe1640741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.085+0000 7fe16bebd640 1 -- 172.21.15.46:0/3652929639 shutdown_connections 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.085+0000 7fe16bebd640 1 -- 172.21.15.46:0/3652929639 wait complete. 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.087+0000 7fe16bebd640 1 Processor -- start 2024-05-09T03:18:17.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.087+0000 7fe16bebd640 1 -- start start 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16bebd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 0x7fe164150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16bebd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe16415dfe0 0x7fe1640b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16bebd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe164164c70 0x7fe1640b8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16bebd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe1640d2dd0 con 0x7fe1640d2480 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16bebd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe16415e3f0 con 0x7fe164164c70 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16bebd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe1640d45c0 con 0x7fe16415dfe0 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 0x7fe164150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 0x7fe164150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50474/0 (socket says 172.21.15.46:50474) 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 -- 172.21.15.46:0/3217578154 learned_addr learned my addr 172.21.15.46:0/3217578154 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe16a45a640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe164164c70 0x7fe1640b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169458640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe16415dfe0 0x7fe1640b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 -- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe16415dfe0 msgr2=0x7fe1640b7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe16415dfe0 0x7fe1640b7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 -- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe164164c70 msgr2=0x7fe1640b8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.088+0000 7fe169c59640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe164164c70 0x7fe1640b8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe169c59640 1 -- 172.21.15.46:0/3217578154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe164150d50 con 0x7fe1640d2480 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe169458640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe16415dfe0 0x7fe1640b7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe16a45a640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe164164c70 0x7fe1640b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe169c59640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 0x7fe164150810 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fe160035040 tx=0x7fe1600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160037040 con 0x7fe1640d2480 2024-05-09T03:18:17.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe160030500 con 0x7fe1640d2480 2024-05-09T03:18:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1640cf130 con 0x7fe1640d2480 2024-05-09T03:18:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160030910 con 0x7fe1640d2480 2024-05-09T03:18:17.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.089+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1640cf6f0 con 0x7fe1640d2480 2024-05-09T03:18:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.090+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe16408cfd0 con 0x7fe1640d2480 2024-05-09T03:18:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.091+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe160044020 con 0x7fe1640d2480 2024-05-09T03:18:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.091+0000 7fe15a7fc640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1480762d0 0x7fe1480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.091+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7fe1600b7fd0 con 0x7fe1640d2480 2024-05-09T03:18:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.092+0000 7fe169458640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1480762d0 0x7fe1480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.092+0000 7fe169458640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1480762d0 0x7fe1480787e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe16415af70 tx=0x7fe154007cc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.096+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe16408cfd0 con 0x7fe1640d2480 2024-05-09T03:18:17.159 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:17.159+0000 7f2424478640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:18:17.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.326+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe16414aef0 con 0x7fe1640d2480 2024-05-09T03:18:17.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.326+0000 7fe15a7fc640 1 -- 172.21.15.46:0/3217578154 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+5185 (secure 0 0 0) 0x7fe160081250 con 0x7fe1640d2480 2024-05-09T03:18:17.327 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:17.327 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:12.993836+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8751":{"gid":8751,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/3106969477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3106969477},{"type":"v1","addr":"172.21.15.46:6834","nonce":3106969477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8751,"qdb_cluster":[8751]},"id":6}]} 2024-05-09T03:18:17.328 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 204 2024-05-09T03:18:17.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1480762d0 msgr2=0x7fe1480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1480762d0 0x7fe1480787e0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe16415af70 tx=0x7fe154007cc0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 msgr2=0x7fe164150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 0x7fe164150810 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fe160035040 tx=0x7fe1600046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 shutdown_connections 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe164164c70 0x7fe1640b8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe16415dfe0 0x7fe1640b7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe1480762d0 0x7fe1480787e0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 --2- 172.21.15.46:0/3217578154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1640d2480 0x7fe164150810 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.328+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 >> 172.21.15.46:0/3217578154 conn(0x7fe164073db0 msgr2=0x7fe16415d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.329+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 shutdown_connections 2024-05-09T03:18:17.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.329+0000 7fe16bebd640 1 -- 172.21.15.46:0/3217578154 wait complete. 2024-05-09T03:18:17.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 204, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:18:12.993836+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 102, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8751}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8751': {'gid': 8751, 'name': 'c', 'rank': 0, 'incarnation': 201, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/3106969477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 3106969477}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 3106969477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8751, 'qdb_cluster': [8751]} 2024-05-09T03:18:17.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:18:17.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:18:17.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.455+0000 7f9139f0d640 1 Processor -- start 2024-05-09T03:18:17.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.455+0000 7f9139f0d640 1 -- start start 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9139f0d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91340b7e70 0x7f91340b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9139f0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9139f0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91340d3400 con 0x7f91340d2e90 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9139f0d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f91340d4a60 con 0x7f91340b7e70 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9139f0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f91340d4d40 con 0x7f91341605b0 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91340b7e70 0x7f91340b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f91340b7e70 0x7f91340b92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50476/0 (socket says 172.21.15.46:50476) 2024-05-09T03:18:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 -- 172.21.15.46:0/4005370997 learned_addr learned my addr 172.21.15.46:0/4005370997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9133fff640 1 --2- 172.21.15.46:0/4005370997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 -- 172.21.15.46:0/4005370997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 msgr2=0x7f91340b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 --2- 172.21.15.46:0/4005370997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 -- 172.21.15.46:0/4005370997 >> v1:172.21.15.88:6789/0 conn(0x7f91340d2e90 legacy=0x7f91340b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.457+0000 7f9132ffd640 1 -- 172.21.15.46:0/4005370997 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91340d5020 con 0x7f91340b7e70 2024-05-09T03:18:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.458+0000 7f9132ffd640 1 --2- 172.21.15.46:0/4005370997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340b7e70 0x7f91340b92a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f9128009870 tx=0x7f912802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e01d93c9ebb563 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.458+0000 7f91327fc640 1 -- 172.21.15.46:0/4005370997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128037070 con 0x7f91340b7e70 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.458+0000 7f91327fc640 1 -- 172.21.15.46:0/4005370997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9128032440 con 0x7f91340b7e70 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.458+0000 7f91327fc640 1 -- 172.21.15.46:0/4005370997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128032810 con 0x7f91340b7e70 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 -- 172.21.15.46:0/4005370997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340b7e70 msgr2=0x7f91340b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 --2- 172.21.15.46:0/4005370997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340b7e70 0x7f91340b92a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f9128009870 tx=0x7f912802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 -- 172.21.15.46:0/4005370997 shutdown_connections 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 --2- 172.21.15.46:0/4005370997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 --2- 172.21.15.46:0/4005370997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340b7e70 0x7f91340b92a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 -- 172.21.15.46:0/4005370997 >> 172.21.15.46:0/4005370997 conn(0x7f913408d390 msgr2=0x7f913408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 -- 172.21.15.46:0/4005370997 shutdown_connections 2024-05-09T03:18:17.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.459+0000 7f9139f0d640 1 -- 172.21.15.46:0/4005370997 wait complete. 2024-05-09T03:18:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.461+0000 7f9139f0d640 1 Processor -- start 2024-05-09T03:18:17.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.461+0000 7f9139f0d640 1 -- start start 2024-05-09T03:18:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f9139f0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 0x7f913414f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f9139f0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f9139f0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340caf60 0x7f913414cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f9139f0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91340b4d80 con 0x7f91340d2e90 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f9139f0d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91340b4c00 con 0x7f91340caf60 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f9139f0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f91340b4fc0 con 0x7f91341605b0 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 0x7f913414f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 0x7f913414f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50488/0 (socket says 172.21.15.46:50488) 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.462+0000 7f91337fe640 1 -- 172.21.15.46:0/927735489 learned_addr learned my addr 172.21.15.46:0/927735489 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f9132ffd640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f9133fff640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340caf60 0x7f913414cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f91337fe640 1 -- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 msgr2=0x7f91340caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f91337fe640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f91337fe640 1 -- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340caf60 msgr2=0x7f913414cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f91337fe640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340caf60 0x7f913414cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f91337fe640 1 -- 172.21.15.46:0/927735489 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91340cb5e0 con 0x7f91340d2e90 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f9133fff640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340caf60 0x7f913414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f91337fe640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 0x7f913414f330 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3f0 tx=0x7f912400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f911bfff640 1 -- 172.21.15.46:0/927735489 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124015020 con 0x7f91340d2e90 2024-05-09T03:18:17.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f911bfff640 1 -- 172.21.15.46:0/927735489 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124011830 con 0x7f91340d2e90 2024-05-09T03:18:17.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f911bfff640 1 -- 172.21.15.46:0/927735489 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124011c20 con 0x7f91340d2e90 2024-05-09T03:18:17.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91340cb8c0 con 0x7f91340d2e90 2024-05-09T03:18:17.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.463+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91340cbe80 con 0x7f91340d2e90 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.465+0000 7f911bfff640 1 -- 172.21.15.46:0/927735489 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9124018020 con 0x7f91340d2e90 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.466+0000 7f911bfff640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9120076300 0x7f9120078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.466+0000 7f911bfff640 1 -- 172.21.15.46:0/927735489 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f9124097960 con 0x7f91340d2e90 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.466+0000 7f9132ffd640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9120076300 0x7f9120078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.466+0000 7f9140dc3740 1 Processor -- start 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.467+0000 7f9132ffd640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9120076300 0x7f9120078810 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9128034000 tx=0x7f912802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.467+0000 7f9140dc3740 1 -- start start 2024-05-09T03:18:17.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa2658eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa22603b0 0x564fa2673be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 0x564fa2674120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9140dc3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564fa25914e0 con 0x564fa2268590 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564fa252cff0 con 0x564fa22603b0 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564fa258d4a0 con 0x564fa2289270 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 0x564fa2674120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 0x564fa2674120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50490/0 (socket says 172.21.15.46:50490) 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 -- 172.21.15.46:0/2627643786 learned_addr learned my addr 172.21.15.46:0/2627643786 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa22603b0 0x564fa2673be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f91337fe640 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa2658eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 -- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 msgr2=0x564fa2658eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa2658eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 -- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa22603b0 msgr2=0x564fa2673be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa22603b0 0x564fa2673be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 -- 172.21.15.46:0/2627643786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fa2513720 con 0x564fa2268590 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.468+0000 7f9133fff640 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 0x564fa2674120 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f912c009fa0 tx=0x7f912c0077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8677ff92c67a2d52 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9119ffb640 1 -- 172.21.15.46:0/2627643786 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c01a070 con 0x564fa2268590 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9119ffb640 1 -- 172.21.15.46:0/2627643786 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c003da0 con 0x564fa2268590 2024-05-09T03:18:17.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9119ffb640 1 -- 172.21.15.46:0/2627643786 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c004230 con 0x564fa2268590 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 -- 172.21.15.46:0/2627643786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 msgr2=0x564fa2674120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 0x564fa2674120 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f912c009fa0 tx=0x7f912c0077c0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 -- 172.21.15.46:0/2627643786 shutdown_connections 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa2268590 0x564fa2674120 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa22603b0 0x564fa2673be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2627643786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa2658eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.469+0000 7f9140dc3740 1 -- 172.21.15.46:0/2627643786 >> 172.21.15.46:0/2627643786 conn(0x564fa227d3c0 msgr2=0x564fa262e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.470+0000 7f9140dc3740 1 -- 172.21.15.46:0/2627643786 shutdown_connections 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.470+0000 7f9140dc3740 1 -- 172.21.15.46:0/2627643786 wait complete. 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.470+0000 7f9140dc3740 1 -- start start 2024-05-09T03:18:17.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.471+0000 7f9140dc3740 1 Processor -- start 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2687720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa2644c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2627c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564fa2518760 con 0x564fa22603b0 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564fa2644790 con 0x564fa2289270 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564fa2645510 con 0x564fa2268590 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9140dc3740 10 client.0 _ll_register_callbacks cb 0x564fa253b4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2687720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2687720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50500/0 (socket says 172.21.15.46:50500) 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 -- 172.21.15.46:0/2625617442 learned_addr learned my addr 172.21.15.46:0/2625617442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9133fff640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2627c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa2644c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 -- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 msgr2=0x564fa2644c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa2644c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 -- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 msgr2=0x564fa2627c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2627c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 -- 172.21.15.46:0/2625617442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fa2559bd0 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91337fe640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2687720 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3c0 tx=0x7f9124060500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1fa9a6e8f7dfd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.472+0000 7f91127fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f91127fc640 1 -- 172.21.15.46:0/2625617442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124093060 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f91127fc640 1 -- 172.21.15.46:0/2625617442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f9124067d70 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f912c003da0 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f91127fc640 1 -- 172.21.15.46:0/2625617442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=204) ==== 24+0+0 (secure 0 0 0) 0x7f912407e050 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 10 client.8850 fetch_fsmap learned FSMap version 204 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=204,monmap=2+}) -- 0x564fa253f280 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f91127fc640 1 -- 172.21.15.46:0/2625617442 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c003da0 con 0x564fa22603b0 2024-05-09T03:18:17.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f91127fc640 1 -- 172.21.15.46:0/2625617442 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 204) ==== 3494+0+0 (secure 0 0 0) 0x7f91240952b0 con 0x564fa22603b0 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 10 client.8850 fetch_fsmap finished waiting for FSMap version 204 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 10 client.8850 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 20 client.8850 populate_metadata read hostname 'smithi046' 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa2247e40 0x564fa265b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 4 client.8850 mds_command: new command op to 8751 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 --> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fa259f0f0 con 0x564fa2247e40 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.473+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa2247e40 0x564fa265b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.474+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa2247e40 0x564fa265b560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faa53bb18152e90b server_cookie=c91d891abfb58ff1 in_seq=0 out_seq=0 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.474+0000 7f91127fc640 10 client.8850 ms_handle_connect on v2:172.21.15.46:6832/3106969477 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.474+0000 7f91127fc640 1 -- 172.21.15.46:0/2625617442 <== mds.0 v2:172.21.15.46:6832/3106969477 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564fa259f0f0 con 0x564fa2247e40 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.474+0000 7f91127fc640 10 client.8850 handle_command_reply: tid=0 2024-05-09T03:18:17.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.474+0000 7f9140dc3740 1 client.8850 shutdown 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 msgr2=0x564fa2687720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2687720 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3c0 tx=0x7f9124060500 comp rx=0 tx=0).stop 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 shutdown_connections 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa2247e40 0x564fa265b560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2627c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa2644c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2625617442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2687720 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f91127fc640 0 client.8850 ms_handle_reset on v2:172.21.15.46:6832/3106969477 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 >> 172.21.15.46:0/2625617442 conn(0x564fa227d3c0 msgr2=0x564fa262e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 shutdown_connections 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 1 -- 172.21.15.46:0/2625617442 wait complete. 2024-05-09T03:18:17.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.475+0000 7f9140dc3740 20 client.8850 trim_cache size 0 max 16384 2024-05-09T03:18:17.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.494+0000 7f9140dc3740 1 Processor -- start 2024-05-09T03:18:17.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.494+0000 7f9140dc3740 1 -- start start 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa26981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2698720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9140dc3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9134202d90 con 0x564fa22603b0 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91342021c0 con 0x564fa2289270 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9134202740 con 0x564fa2268590 2024-05-09T03:18:17.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50508/0 (socket says 172.21.15.46:50508) 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 -- 172.21.15.46:0/4153387380 learned_addr learned my addr 172.21.15.46:0/4153387380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9133fff640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2698720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9132ffd640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa26981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 -- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 msgr2=0x564fa26981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa26981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 -- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 msgr2=0x564fa2698720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2698720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 -- 172.21.15.46:0/4153387380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fa22f4850 con 0x564fa22603b0 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9133fff640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2698720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f9132ffd640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa26981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.495+0000 7f91337fe640 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f912405c900 tx=0x7f9124097070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7622aeac656a84c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f91197fa640 1 -- 172.21.15.46:0/4153387380 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124079070 con 0x564fa22603b0 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f91197fa640 1 -- 172.21.15.46:0/4153387380 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912405f7c0 con 0x564fa22603b0 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f91197fa640 1 -- 172.21.15.46:0/4153387380 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124067b70 con 0x564fa22603b0 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 -- 172.21.15.46:0/4153387380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 msgr2=0x564fa2675760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f912405c900 tx=0x7f9124097070 comp rx=0 tx=0).stop 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 -- 172.21.15.46:0/4153387380 shutdown_connections 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2289270 0x564fa2698720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2268590 0x564fa26981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 --2- 172.21.15.46:0/4153387380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 -- 172.21.15.46:0/4153387380 >> 172.21.15.46:0/4153387380 conn(0x564fa227d3c0 msgr2=0x564fa2644920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 -- 172.21.15.46:0/4153387380 shutdown_connections 2024-05-09T03:18:17.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.496+0000 7f9140dc3740 1 -- 172.21.15.46:0/4153387380 wait complete. 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.497+0000 7f9140dc3740 1 -- start start 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.497+0000 7f9140dc3740 1 Processor -- start 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2268590 0x564fa2675ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa26f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564fa2628610 con 0x564fa22603b0 2024-05-09T03:18:17.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564fa26281c0 con 0x564fa2268590 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564fa2670560 con 0x564fa2289270 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 10 client.0 _ll_register_callbacks cb 0x7f9124060b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50516/0 (socket says 172.21.15.46:50516) 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 -- 172.21.15.46:0/2526402095 learned_addr learned my addr 172.21.15.46:0/2526402095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2268590 0x564fa2675ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9133fff640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa26f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 -- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 msgr2=0x564fa26f0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa26f0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 -- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2268590 msgr2=0x564fa2675ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2268590 0x564fa2675ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 -- 172.21.15.46:0/2526402095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fa236b6b0 con 0x564fa22603b0 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2268590 0x564fa2675ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91337fe640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3c0 tx=0x7f912400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d168733efecc37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91127fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91127fc640 1 -- 172.21.15.46:0/2526402095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912406f020 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91127fc640 1 -- 172.21.15.46:0/2526402095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f912405bdb0 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f912405f7c0 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f91127fc640 1 -- 172.21.15.46:0/2526402095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=204) ==== 24+0+0 (secure 0 0 0) 0x7f9124093030 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.499+0000 7f9140dc3740 10 client.8856 fetch_fsmap learned FSMap version 204 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=204,monmap=2+}) -- 0x564fa254d630 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f91127fc640 1 -- 172.21.15.46:0/2526402095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912405f7c0 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f91127fc640 1 -- 172.21.15.46:0/2526402095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 204) ==== 3494+0+0 (secure 0 0 0) 0x7f912405bfb0 con 0x564fa22603b0 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 10 client.8856 fetch_fsmap finished waiting for FSMap version 204 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 10 client.8856 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 20 client.8856 populate_metadata read hostname 'smithi046' 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa26f14d0 0x564fa2659dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 4 client.8856 mds_command: new command op to 8751 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:18:17.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 --> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] -- command(tid 0: {"prefix": "status"}) -- 0x564fa2671fa0 con 0x564fa26f14d0 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa26f14d0 0x564fa2659dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f91127fc640 10 client.8856 ms_handle_connect on v2:172.21.15.46:6832/3106969477 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9132ffd640 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa26f14d0 0x564fa2659dc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87632997e0f4e2b5 server_cookie=463274f91c728994 in_seq=0 out_seq=0 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f91127fc640 1 -- 172.21.15.46:0/2526402095 <== mds.0 v2:172.21.15.46:6832/3106969477 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x564fa2671fa0 con 0x564fa26f14d0 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f91127fc640 10 client.8856 handle_command_reply: tid=0 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.500+0000 7f9140dc3740 1 client.8856 shutdown 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 msgr2=0x564fa2675760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3c0 tx=0x7f912400e000 comp rx=0 tx=0).stop 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 shutdown_connections 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:6832/3106969477,v1:172.21.15.46:6834/3106969477] conn(0x564fa26f14d0 0x564fa2659dc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fa2289270 0x564fa26f0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fa2268590 0x564fa2675ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 --2- 172.21.15.46:0/2526402095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fa22603b0 0x564fa2675760 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 >> 172.21.15.46:0/2526402095 conn(0x564fa227d3c0 msgr2=0x564fa26446f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f91127fc640 0 client.8856 ms_handle_reset on v2:172.21.15.46:6832/3106969477 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 shutdown_connections 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 1 -- 172.21.15.46:0/2526402095 wait complete. 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9140dc3740 20 client.8856 trim_cache size 0 max 16384 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8751, 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:18:17.508 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:18:17.509 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 7.5099107780000001, 2024-05-09T03:18:17.509 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 204, 2024-05-09T03:18:17.509 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 102, 2024-05-09T03:18:17.509 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 102, 2024-05-09T03:18:17.509 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.290790561 2024-05-09T03:18:17.509 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9120076300 msgr2=0x7f9120078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9120076300 0x7f9120078810 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9128034000 tx=0x7f912802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 msgr2=0x7f913414f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 0x7f913414f330 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f912400b3f0 tx=0x7f912400b8b0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 shutdown_connections 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340caf60 0x7f913414cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f91341605b0 0x7f91340caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9120076300 0x7f9120078810 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 --2- 172.21.15.46:0/927735489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91340d2e90 0x7f913414f330 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.501+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 >> 172.21.15.46:0/927735489 conn(0x7f913408d390 msgr2=0x7f91340b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.502+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 shutdown_connections 2024-05-09T03:18:17.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.502+0000 7f9139f0d640 1 -- 172.21.15.46:0/927735489 wait complete. 2024-05-09T03:18:17.513 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:18:17.513 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:17.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.627+0000 7f540c0bd640 1 Processor -- start 2024-05-09T03:18:17.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.627+0000 7f540c0bd640 1 -- start start 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f540c0bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54040b7ef0 0x7f540416b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f540c0bd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54040d49e0 con 0x7f54040d57a0 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f540c0bd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54040d4cc0 con 0x7f5404160610 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f540c0bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f54040d5bb0 con 0x7f54040b7ef0 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f5409e59640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f54040d57a0 0x7f54040b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48482/0 (socket says 172.21.15.46:48482) 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f5409e59640 1 -- 172.21.15.46:0/211606550 learned_addr learned my addr 172.21.15.46:0/211606550 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f540a65a640 1 --2- 172.21.15.46:0/211606550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54040b7ef0 0x7f540416b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692290373 0 0) 0x7f54040d49e0 con 0x7f54040d57a0 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f0003520 con 0x7f54040d57a0 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.629+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098071216 0 0) 0x7f54040d4cc0 con 0x7f5404160610 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.630+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54040d49e0 con 0x7f5404160610 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.630+0000 7f540a65a640 1 -- 172.21.15.46:0/211606550 >> v1:172.21.15.88:6789/0 conn(0x7f5404160610 legacy=0x7f54040b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.630+0000 7f540a65a640 1 -- 172.21.15.46:0/211606550 >> v1:172.21.15.46:6789/0 conn(0x7f54040d57a0 legacy=0x7f54040b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.630+0000 7f540a65a640 1 -- 172.21.15.46:0/211606550 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54040d4fa0 con 0x7f54040b7ef0 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.630+0000 7f540a65a640 1 --2- 172.21.15.46:0/211606550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54040b7ef0 0x7f540416b1f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009dc0 tx=0x7f53fc02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=299c4bdf0201b971 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc007740 con 0x7f54040b7ef0 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53fc0078e0 con 0x7f54040b7ef0 2024-05-09T03:18:17.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f5408e57640 1 -- 172.21.15.46:0/211606550 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc007cb0 con 0x7f54040b7ef0 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 -- 172.21.15.46:0/211606550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54040b7ef0 msgr2=0x7f540416b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 --2- 172.21.15.46:0/211606550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54040b7ef0 0x7f540416b1f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009dc0 tx=0x7f53fc02a6d0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 -- 172.21.15.46:0/211606550 shutdown_connections 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 --2- 172.21.15.46:0/211606550 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54040b7ef0 0x7f540416b1f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 -- 172.21.15.46:0/211606550 >> 172.21.15.46:0/211606550 conn(0x7f54040743e0 msgr2=0x7f54040d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 -- 172.21.15.46:0/211606550 shutdown_connections 2024-05-09T03:18:17.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.631+0000 7f540c0bd640 1 -- 172.21.15.46:0/211606550 wait complete. 2024-05-09T03:18:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.633+0000 7f540c0bd640 1 Processor -- start 2024-05-09T03:18:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.633+0000 7f540c0bd640 1 -- start start 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.634+0000 7f540c0bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 0x7f54041e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f540c0bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5404160610 0x7f54040ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f540c0bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f540415b240 0x7f54041500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f540c0bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54040d1920 con 0x7f54040d57a0 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f540c0bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54040d15b0 con 0x7f5404160610 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f540c0bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f54040ca8d0 con 0x7f540415b240 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 0x7f54041e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 0x7f54041e9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50530/0 (socket says 172.21.15.46:50530) 2024-05-09T03:18:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 -- 172.21.15.46:0/1793741966 learned_addr learned my addr 172.21.15.46:0/1793741966 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409658640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5404160610 0x7f54040ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f540a65a640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f540415b240 0x7f54041500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 -- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f540415b240 msgr2=0x7f54041500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f540415b240 0x7f54041500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 -- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5404160610 msgr2=0x7f54040ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5404160610 0x7f54040ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 -- 172.21.15.46:0/1793741966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54041ea500 con 0x7f54040d57a0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f5409e59640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 0x7f54041e9cb0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f540000c410 tx=0x7f540000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.635+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400017070 con 0x7f54040d57a0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.636+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5400005ce0 con 0x7f54040d57a0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.636+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54040c98b0 con 0x7f54040d57a0 2024-05-09T03:18:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.636+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54000072d0 con 0x7f54040d57a0 2024-05-09T03:18:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.636+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54040c9e70 con 0x7f54040d57a0 2024-05-09T03:18:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.636+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f540406a270 con 0x7f54040d57a0 2024-05-09T03:18:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.637+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f540001a070 con 0x7f54040d57a0 2024-05-09T03:18:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.638+0000 7f53fa7fc640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53e80762d0 0x7f53e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:17.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.638+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f54000973a0 con 0x7f54040d57a0 2024-05-09T03:18:17.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.638+0000 7f5409658640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53e80762d0 0x7f53e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:17.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.639+0000 7f5409658640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53e80762d0 0x7f53e80787e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f540415fa90 tx=0x7f53f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:17.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.642+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f54000600b0 con 0x7f54040d57a0 2024-05-09T03:18:17.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.877+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f540415fda0 con 0x7f54040d57a0 2024-05-09T03:18:17.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.877+0000 7f53fa7fc640 1 -- 172.21.15.46:0/1793741966 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+5185 (secure 0 0 0) 0x7f54000643b0 con 0x7f54040d57a0 2024-05-09T03:18:17.878 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:17.878 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":204,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:12.993836+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8751":{"gid":8751,"name":"c","rank":0,"incarnation":201,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/3106969477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3106969477},{"type":"v1","addr":"172.21.15.46:6834","nonce":3106969477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8751,"qdb_cluster":[8751]},"id":6}]} 2024-05-09T03:18:17.879 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 204 2024-05-09T03:18:17.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53e80762d0 msgr2=0x7f53e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53e80762d0 0x7f53e80787e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f540415fa90 tx=0x7f53f4009040 comp rx=0 tx=0).stop 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 msgr2=0x7f54041e9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 0x7f54041e9cb0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f540000c410 tx=0x7f540000c8d0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 shutdown_connections 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f540415b240 0x7f54041500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5404160610 0x7f54040ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53e80762d0 0x7f53e80787e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 --2- 172.21.15.46:0/1793741966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54040d57a0 0x7f54041e9cb0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.879+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 >> 172.21.15.46:0/1793741966 conn(0x7f54040743e0 msgr2=0x7f540415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.880+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 shutdown_connections 2024-05-09T03:18:17.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:17.880+0000 7f540c0bd640 1 -- 172.21.15.46:0/1793741966 wait complete. 2024-05-09T03:18:17.891 INFO:tasks.cephfs.test_damage:Mutation 'Corrupt 500.00000000' did not prevent MDS startup, attempting ls... 2024-05-09T03:18:17.891 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:18:18.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.005+0000 7fbcecbd5640 1 Processor -- start 2024-05-09T03:18:18.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.005+0000 7fbcecbd5640 1 -- start start 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbcecbd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 0x7fbce81698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbcecbd5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce80b4ee0 con 0x7fbce8162610 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbcecbd5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce80b62e0 con 0x7fbce80b9200 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbcecbd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbce80b5460 con 0x7fbce80b7a50 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbcddd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbce8162610 0x7fbce8162a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48496/0 (socket says 172.21.15.46:48496) 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbcddd9b640 1 -- 172.21.15.46:0/1548835772 learned_addr learned my addr 172.21.15.46:0/1548835772 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.006+0000 7fbce6d9d640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 0x7fbce81698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605154559 0 0) 0x7fbce80b4ee0 con 0x7fbce8162610 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd4003520 con 0x7fbce8162610 2024-05-09T03:18:18.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3221654580 0 0) 0x7fbcd4003520 con 0x7fbce8162610 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 msgr2=0x7fbce81698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 0x7fbce81698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 >> v1:172.21.15.88:6789/0 conn(0x7fbce80b9200 legacy=0x7fbce808d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce80b5180 con 0x7fbce8162610 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fbcd00016f0 con 0x7fbce8162610 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 >> v1:172.21.15.46:6789/0 conn(0x7fbce8162610 legacy=0x7fbce8162a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd40039e0 0x7fbcd4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd4004800 0x7fbcd4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 0x7fbcd4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd400baf0 con 0x7fbcd40039e0 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd400bca0 con 0x7fbcd4004800 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd400bea0 con 0x7fbce80b7a50 2024-05-09T03:18:18.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce659c640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd40039e0 0x7fbcd4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbcddd9b640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd4004800 0x7fbcd4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce6d9d640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 0x7fbcd4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce659c640 1 -- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 msgr2=0x7fbcd4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce659c640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce80b7a50 0x7fbcd4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce659c640 1 -- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd4004800 msgr2=0x7fbcd4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce659c640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd4004800 0x7fbcd4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.007+0000 7fbce659c640 1 -- 172.21.15.46:0/1548835772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd4003520 con 0x7fbcd40039e0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.008+0000 7fbce659c640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd40039e0 0x7fbcd4003e10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8001810 tx=0x7fbcd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35aafa788198caae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.008+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8037070 con 0x7fbcd40039e0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.008+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd802ac80 con 0x7fbcd40039e0 2024-05-09T03:18:18.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.008+0000 7fbce5d9b640 1 -- 172.21.15.46:0/1548835772 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8001430 con 0x7fbcd40039e0 2024-05-09T03:18:18.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbcecbd5640 1 -- 172.21.15.46:0/1548835772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd40039e0 msgr2=0x7fbcd4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/1548835772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd40039e0 0x7fbcd4003e10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fbcd8001810 tx=0x7fbcd4005830 comp rx=0 tx=0).stop 2024-05-09T03:18:18.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbce6d9d640 1 -- 172.21.15.46:0/1548835772 reap_dead start 2024-05-09T03:18:18.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbcecbd5640 1 -- 172.21.15.46:0/1548835772 shutdown_connections 2024-05-09T03:18:18.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbcecbd5640 1 -- 172.21.15.46:0/1548835772 >> 172.21.15.46:0/1548835772 conn(0x7fbce80b56e0 msgr2=0x7fbce80b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:18.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbcecbd5640 1 -- 172.21.15.46:0/1548835772 shutdown_connections 2024-05-09T03:18:18.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.009+0000 7fbcecbd5640 1 -- 172.21.15.46:0/1548835772 wait complete. 2024-05-09T03:18:18.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.011+0000 7fbcecbd5640 1 Processor -- start 2024-05-09T03:18:18.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.011+0000 7fbcecbd5640 1 -- start start 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcecbd5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd40039e0 0x7fbce806f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcecbd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce81595e0 0x7fbce8160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcecbd5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 0x7fbce8160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcecbd5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbce80cf750 con 0x7fbce81580f0 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcecbd5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbce80cfac0 con 0x7fbcd40039e0 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcecbd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbce808d8b0 con 0x7fbce81595e0 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 0x7fbce8160ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 0x7fbce8160ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50540/0 (socket says 172.21.15.46:50540) 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 -- 172.21.15.46:0/2614764235 learned_addr learned my addr 172.21.15.46:0/2614764235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcddd9b640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce81595e0 0x7fbce8160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce659c640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd40039e0 0x7fbce806f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 -- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce81595e0 msgr2=0x7fbce8160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce81595e0 0x7fbce8160ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 -- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd40039e0 msgr2=0x7fbce806f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd40039e0 0x7fbce806f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 -- 172.21.15.46:0/2614764235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce806dac0 con 0x7fbce81580f0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbce6d9d640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 0x7fbce8160ff0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fbce80b98f0 tx=0x7fbce000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce0015430 con 0x7fbce81580f0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.013+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce001d070 con 0x7fbce81580f0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.014+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce815ba60 con 0x7fbce81580f0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.014+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce0015830 con 0x7fbce81580f0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.014+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce80b4480 con 0x7fbce81580f0 2024-05-09T03:18:18.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.014+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce8159220 con 0x7fbce81580f0 2024-05-09T03:18:18.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.015+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbce00159d0 con 0x7fbce81580f0 2024-05-09T03:18:18.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.016+0000 7fbcdf7fe640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbcc80762d0 0x7fbcc80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:18.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.016+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7fbce0098430 con 0x7fbce81580f0 2024-05-09T03:18:18.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.016+0000 7fbce659c640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbcc80762d0 0x7fbcc80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:18.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.017+0000 7fbce659c640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbcc80762d0 0x7fbcc80787e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80017e0 tx=0x7fbcd8005040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:18.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.021+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbce806dac0 con 0x7fbce81580f0 2024-05-09T03:18:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.253+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbce806a270 con 0x7fbce81580f0 2024-05-09T03:18:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.254+0000 7fbcdf7fe640 1 -- 172.21.15.46:0/2614764235 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v204) ==== 53+0+73 (secure 0 0 0) 0x7fbce0010040 con 0x7fbce81580f0 2024-05-09T03:18:18.255 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbcc80762d0 msgr2=0x7fbcc80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbcc80762d0 0x7fbcc80787e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80017e0 tx=0x7fbcd8005040 comp rx=0 tx=0).stop 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 msgr2=0x7fbce8160ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 0x7fbce8160ff0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fbce80b98f0 tx=0x7fbce000f580 comp rx=0 tx=0).stop 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 shutdown_connections 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbcc80762d0 0x7fbcc80787e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbce81580f0 0x7fbce8160ff0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbce81595e0 0x7fbce8160ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 --2- 172.21.15.46:0/2614764235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcd40039e0 0x7fbce806f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:18.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 >> 172.21.15.46:0/2614764235 conn(0x7fbce80b56e0 msgr2=0x7fbce80727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:18.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 shutdown_connections 2024-05-09T03:18:18.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18.256+0000 7fbcecbd5640 1 -- 172.21.15.46:0/2614764235 wait complete. 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:18:18.270 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:18:18.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:18.271 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:18:18.287 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:18:18.287 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:18:18.287 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:18:18.288 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42019sec preferred_lft 42019sec 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:18:18.289 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:18.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:18.290 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:18:18.352 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:18:18.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:18.353 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:18:18.410 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:18:18.410 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:18:18.410 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:18:18.410 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:18:18.410 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:18:18.411 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:18:18.411 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:18:18.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:18:18.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:18:18.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:18:18.411 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:18:18.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:18:18.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:18:18.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:18.541 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:18:18.556 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:18:18.556 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:18:18.557 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:18:18.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:18.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:18:18.622 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:18:18.622 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:18:18.622 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:18.678 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:18:18.678 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:18:18.678 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:18.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:18:18.801 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:18:18.858 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:18:18.858 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:18:18.859 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:18:18.859 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:18:18.859 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:18:18.859 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:18:18.859 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:18:18.859 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:18:18.860 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:18:18.860 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:18:18.860 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:18:18.860 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:18:18.860 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:18:18.861 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:18.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:18.862 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:18:18.914 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:18:18.915 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:18:18.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:18:18.983 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:18:19.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:18:19.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:19.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:19.044 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:19.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:19.045 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:18:19.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:18:19.053+0000 7f1c0071c480 -1 init, newargv = 0x5623710dd780 newargc=15 2024-05-09T03:18:19.054 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[74327]: starting ceph client 2024-05-09T03:18:19.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[74327]: starting fuse 2024-05-09T03:18:19.070 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:18:19.070 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:18:19.070 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:18:19.070 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:18:19.070 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:18:19.070 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:18:19.071 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:18:19.072 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:18:19.192 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.74327.asok 2024-05-09T03:18:19.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:18:19.204 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74327.asok 2024-05-09T03:18:19.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:19.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74327.asok status 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:18:19.352 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "74327", 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 8874, 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 8874 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:18:19.353 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 457974064 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 457974064 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.8874 192.168.0.1:0/457974064", 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/457974064", 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:18:19.354 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 204, 2024-05-09T03:18:19.355 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 102, 2024-05-09T03:18:19.355 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:18:19.355 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:18:19.355 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:18:19.355 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:18:19.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:19.361 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:18:19.361 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:19.378 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:18:19.378 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:19.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:19.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:19.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:19.445 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:18:19.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:19.488 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir foo) 2024-05-09T03:18:19.505 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:18:19.505 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:18:19.505 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:19.505 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:18:19.505 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs 2024-05-09T03:18:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:19.521 INFO:teuthology.orchestra.run.smithi046.stderr:mkdir: cannot create directory ‘foo’: Read-only file system 2024-05-09T03:18:19.521 INFO:tasks.cephfs.test_damage:mkdir: cannot create directory ‘foo’: Read-only file system 2024-05-09T03:18:19.521 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt mds_snaptable' 2024-05-09T03:18:19.521 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:19.521 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:19.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:18:19.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.734+0000 7f42c7c68640 1 Processor -- start 2024-05-09T03:18:19.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.734+0000 7f42c7c68640 1 -- start start 2024-05-09T03:18:19.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.735+0000 7f42c7c68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f42c00b5370 0x7f42c00d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.735+0000 7f42c7c68640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42c00d21b0 con 0x7f42c01434a0 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.735+0000 7f42c7c68640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42c01438b0 con 0x7f42c00d1da0 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.735+0000 7f42c7c68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f42c00b4920 con 0x7f42c00b5370 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c5a04640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42c01434a0 0x7f42c0160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48510/0 (socket says 172.21.15.46:48510) 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c5a04640 1 -- 172.21.15.46:0/2480648491 learned_addr learned my addr 172.21.15.46:0/2480648491 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c5203640 1 --2- 172.21.15.46:0/2480648491 >> v2:172.21.15.88:3300/0 conn(0x7f42c00b5370 0x7f42c00d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2344882121 0 0) 0x7f42bc001110 con 0x7f42c01434a0 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42ac003520 con 0x7f42c01434a0 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 984148784 0 0) 0x7f42ac003520 con 0x7f42c01434a0 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 >> v2:172.21.15.88:3300/0 conn(0x7f42c00b5370 msgr2=0x7f42c00d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 --2- 172.21.15.46:0/2480648491 >> v2:172.21.15.88:3300/0 conn(0x7f42c00b5370 0x7f42c00d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 >> v1:172.21.15.88:6789/0 conn(0x7f42c00d1da0 legacy=0x7f42c00cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c01616b0 con 0x7f42c01434a0 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f42bc0018c0 con 0x7f42c01434a0 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.736+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 >> v1:172.21.15.46:6789/0 conn(0x7f42c01434a0 legacy=0x7f42c0160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c4a02640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42ac003a50 0x7f42ac003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c4a02640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42ac004800 0x7f42ac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c4a02640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00b4920 con 0x7f42ac005350 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f42ac00c280 con 0x7f42ac003a50 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f42ac00c480 con 0x7f42ac004800 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c6205640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c5a04640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42ac003a50 0x7f42ac003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c5203640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42ac004800 0x7f42ac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c6205640 1 -- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42ac004800 msgr2=0x7f42ac004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c6205640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42ac004800 0x7f42ac004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c6205640 1 -- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42ac003a50 msgr2=0x7f42ac003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c6205640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42ac003a50 0x7f42ac003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.737+0000 7f42c6205640 1 -- 172.21.15.46:0/2480648491 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac003520 con 0x7f42ac005350 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.738+0000 7f42c6205640 1 -- 172.21.15.46:0/2480648491 reap_dead start 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.738+0000 7f42c6205640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42ac00b9c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f42b8001e60 tx=0x7f42ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604da1cf103f5ddd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.738+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8001ab0 con 0x7f42ac005350 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.738+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42b802ae40 con 0x7f42ac005350 2024-05-09T03:18:19.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.738+0000 7f42c4a02640 1 -- 172.21.15.46:0/2480648491 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b8003b90 con 0x7f42ac005350 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 -- 172.21.15.46:0/2480648491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 msgr2=0x7f42ac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42ac00b9c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f42b8001e60 tx=0x7f42ac005830 comp rx=0 tx=0).stop 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 -- 172.21.15.46:0/2480648491 shutdown_connections 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 --2- 172.21.15.46:0/2480648491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42ac00b9c0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 -- 172.21.15.46:0/2480648491 >> 172.21.15.46:0/2480648491 conn(0x7f42c00b3620 msgr2=0x7f42c00b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 -- 172.21.15.46:0/2480648491 shutdown_connections 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.739+0000 7f42c7c68640 1 -- 172.21.15.46:0/2480648491 wait complete. 2024-05-09T03:18:19.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.741+0000 7f42c7c68640 1 Processor -- start 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.741+0000 7f42c7c68640 1 -- start start 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c7c68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42c014e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c7c68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42c0144fd0 0x7f42c0141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c7c68640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42c0145cb0 0x7f42c01460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c7c68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00b4b70 con 0x7f42ac005350 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c7c68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f42c00b4ee0 con 0x7f42c0144fd0 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c7c68640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f42c00b5780 con 0x7f42c0145cb0 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c5a04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42c014e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c5a04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42c014e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50562/0 (socket says 172.21.15.46:50562) 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.742+0000 7f42c5a04640 1 -- 172.21.15.46:0/1020488201 learned_addr learned my addr 172.21.15.46:0/1020488201 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5203640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42c0144fd0 0x7f42c0141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5a04640 1 -- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42c0145cb0 msgr2=0x7f42c01460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5a04640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42c0145cb0 0x7f42c01460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5a04640 1 -- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42c0144fd0 msgr2=0x7f42c0141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5a04640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42c0144fd0 0x7f42c0141480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5a04640 1 -- 172.21.15.46:0/1020488201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c0146600 con 0x7f42ac005350 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c5a04640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42c014e070 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f42bc007310 tx=0x7f42bc0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc01a070 con 0x7f42ac005350 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42bc00f8e0 con 0x7f42ac005350 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc00fcd0 con 0x7f42ac005350 2024-05-09T03:18:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c015df60 con 0x7f42ac005350 2024-05-09T03:18:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.743+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c0144210 con 0x7f42ac005350 2024-05-09T03:18:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.744+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c006a270 con 0x7f42ac005350 2024-05-09T03:18:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.745+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f42bc01d360 con 0x7f42ac005350 2024-05-09T03:18:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.745+0000 7f42b67fc640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42a40762d0 0x7f42a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.745+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f42bc097540 con 0x7f42ac005350 2024-05-09T03:18:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.746+0000 7f42c5203640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42a40762d0 0x7f42a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.746+0000 7f42c5203640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42a40762d0 0x7f42a40787e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f42c00d0ee0 tx=0x7f42b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.750+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f42c0146600 con 0x7f42ac005350 2024-05-09T03:18:19.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.959+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f42c00d0620 con 0x7f42ac005350 2024-05-09T03:18:19.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.960+0000 7f42b67fc640 1 -- 172.21.15.46:0/1020488201 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 36 entries v102) ==== 81+0+2190 (secure 0 0 0) 0x7f42c00d21b0 con 0x7f42ac005350 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:18:19.961 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:18:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:18:19.963 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:18:19.964 INFO:teuthology.orchestra.run.smithi046.stderr:listed 36 entries 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42a40762d0 msgr2=0x7f42a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42a40762d0 0x7f42a40787e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f42c00d0ee0 tx=0x7f42b0008040 comp rx=0 tx=0).stop 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 msgr2=0x7f42c014e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42c014e070 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f42bc007310 tx=0x7f42bc0077d0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 shutdown_connections 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42c0145cb0 0x7f42c01460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42c0144fd0 0x7f42c0141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f42a40762d0 0x7f42a40787e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 --2- 172.21.15.46:0/1020488201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac005350 0x7f42c014e070 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 >> 172.21.15.46:0/1020488201 conn(0x7f42c00b3620 msgr2=0x7f42c00b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 shutdown_connections 2024-05-09T03:18:19.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:19.962+0000 7f42c7c68640 1 -- 172.21.15.46:0/1020488201 wait complete. 2024-05-09T03:18:19.974 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:19.974 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:19.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:18:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.146+0000 7fb03e019640 1 Processor -- start 2024-05-09T03:18:20.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.146+0000 7fb03e019640 1 -- start start 2024-05-09T03:18:20.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb03e019640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb0380b7e70 0x7fb0380d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb03e019640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0380b5550 con 0x7fb0380d3df0 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb03e019640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0380d44e0 con 0x7fb0380b5140 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb03e019640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb0380d3480 con 0x7fb0380b7e70 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb037fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb0380d3df0 0x7fb0380b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48524/0 (socket says 172.21.15.46:48524) 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb037fff640 1 -- 172.21.15.46:0/2668992603 learned_addr learned my addr 172.21.15.46:0/2668992603 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb02effd640 1 --2- 172.21.15.46:0/2668992603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb0380b7e70 0x7fb0380d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722556173 0 0) 0x7fb0380b5550 con 0x7fb0380d3df0 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb024003520 con 0x7fb0380d3df0 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2299228458 0 0) 0x7fb0380d44e0 con 0x7fb0380b5140 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0380b5550 con 0x7fb0380b5140 2024-05-09T03:18:20.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb02effd640 1 -- 172.21.15.46:0/2668992603 >> v1:172.21.15.88:6789/0 conn(0x7fb0380b5140 legacy=0x7fb0380b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb02effd640 1 -- 172.21.15.46:0/2668992603 >> v1:172.21.15.46:6789/0 conn(0x7fb0380d3df0 legacy=0x7fb0380b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.148+0000 7fb02effd640 1 -- 172.21.15.46:0/2668992603 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0380d47c0 con 0x7fb0380b7e70 2024-05-09T03:18:20.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.149+0000 7fb02effd640 1 --2- 172.21.15.46:0/2668992603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb0380b7e70 0x7fb0380d38b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0200097d0 tx=0x7fb02002a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b173ac017c88f1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.149+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb020037070 con 0x7fb0380b7e70 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.149+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb020032440 con 0x7fb0380b7e70 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.149+0000 7fb036ffd640 1 -- 172.21.15.46:0/2668992603 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0200327f0 con 0x7fb0380b7e70 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 -- 172.21.15.46:0/2668992603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb0380b7e70 msgr2=0x7fb0380d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 --2- 172.21.15.46:0/2668992603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb0380b7e70 0x7fb0380d38b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0200097d0 tx=0x7fb02002a290 comp rx=0 tx=0).stop 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 -- 172.21.15.46:0/2668992603 shutdown_connections 2024-05-09T03:18:20.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 --2- 172.21.15.46:0/2668992603 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb0380b7e70 0x7fb0380d38b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 -- 172.21.15.46:0/2668992603 >> 172.21.15.46:0/2668992603 conn(0x7fb03808d3f0 msgr2=0x7fb03808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 -- 172.21.15.46:0/2668992603 shutdown_connections 2024-05-09T03:18:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.150+0000 7fb03e019640 1 -- 172.21.15.46:0/2668992603 wait complete. 2024-05-09T03:18:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.152+0000 7fb03e019640 1 Processor -- start 2024-05-09T03:18:20.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.152+0000 7fb03e019640 1 -- start start 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb03e019640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 0x7fb03814e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb03e019640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0380d3df0 0x7fb038160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb03e019640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb038160fb0 0x7fb038162c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb03e019640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb0380d4200 con 0x7fb0380b5140 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb03e019640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb0380d4d80 con 0x7fb0380d3df0 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb03e019640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb0380d5580 con 0x7fb038160fb0 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.153+0000 7fb0377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 0x7fb03814e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 0x7fb03814e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50576/0 (socket says 172.21.15.46:50576) 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 -- 172.21.15.46:0/3085371699 learned_addr learned my addr 172.21.15.46:0/3085371699 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:20.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb037fff640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb038160fb0 0x7fb038162c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb02effd640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0380d3df0 0x7fb038160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 -- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb038160fb0 msgr2=0x7fb038162c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb038160fb0 0x7fb038162c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 -- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0380d3df0 msgr2=0x7fb038160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0380d3df0 0x7fb038160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 -- 172.21.15.46:0/3085371699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb038162590 con 0x7fb0380b5140 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb0377fe640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 0x7fb03814e300 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fb02800c460 tx=0x7fb02800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0280052d0 con 0x7fb0380b5140 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb028005490 con 0x7fb0380b5140 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb028012870 con 0x7fb0380b5140 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0381614f0 con 0x7fb0380b5140 2024-05-09T03:18:20.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.154+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb038161ab0 con 0x7fb0380b5140 2024-05-09T03:18:20.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.155+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03806a270 con 0x7fb0380b5140 2024-05-09T03:18:20.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.156+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb028012a10 con 0x7fb0380b5140 2024-05-09T03:18:20.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.157+0000 7fb034ff9640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb0180762d0 0x7fb0180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.157+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7fb0280977e0 con 0x7fb0380b5140 2024-05-09T03:18:20.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.157+0000 7fb02effd640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb0180762d0 0x7fb0180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.157+0000 7fb02effd640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb0180762d0 0x7fb0180787e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb020034000 tx=0x7fb02002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:20.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.161+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb028060370 con 0x7fb0380b5140 2024-05-09T03:18:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.364+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb0380b4750 con 0x7fb0380b5140 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.364+0000 7fb034ff9640 1 -- 172.21.15.46:0/3085371699 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 36 entries v102) ==== 81+0+2190 (secure 0 0 0) 0x7fb028064670 con 0x7fb0380b5140 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:18:20.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:18:20.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:18:20.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:18:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:18:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:18:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:18:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:18:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:18:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:18:20.369 INFO:teuthology.orchestra.run.smithi046.stderr:listed 36 entries 2024-05-09T03:18:20.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.366+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb0180762d0 msgr2=0x7fb0180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.366+0000 7fb03e019640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb0180762d0 0x7fb0180787e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb020034000 tx=0x7fb02002e000 comp rx=0 tx=0).stop 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.366+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 msgr2=0x7fb03814e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.366+0000 7fb03e019640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 0x7fb03814e300 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fb02800c460 tx=0x7fb02800c920 comp rx=0 tx=0).stop 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 shutdown_connections 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb038160fb0 0x7fb038162c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0380d3df0 0x7fb038160a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb0180762d0 0x7fb0180787e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 --2- 172.21.15.46:0/3085371699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0380b5140 0x7fb03814e300 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 >> 172.21.15.46:0/3085371699 conn(0x7fb03808d3f0 msgr2=0x7fb03815f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 shutdown_connections 2024-05-09T03:18:20.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.367+0000 7fb03e019640 1 -- 172.21.15.46:0/3085371699 wait complete. 2024-05-09T03:18:20.379 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:18:20.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:20.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:20.405 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:18:20.405 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:20.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:20.406 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:20.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.618+0000 7f4792bf3640 1 Processor -- start 2024-05-09T03:18:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.618+0000 7f4792bf3640 1 -- start start 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.619+0000 7f4792bf3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 0x7f478c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.619+0000 7f4792bf3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f478c0d3df0 0x7f478c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.619+0000 7f4792bf3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f478c0b5550 con 0x7f478c0b5140 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.619+0000 7f4792bf3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f478c0d3480 con 0x7f478c0d3df0 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.619+0000 7f4792bf3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f478c0d44e0 con 0x7f478c0b7e70 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4783fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 0x7f478c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4791190640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f478c0d3df0 0x7f478c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f479098f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f478c0b5140 0x7f478c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:44132/0 (socket says 172.21.15.46:44132) 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f479098f640 1 -- 172.21.15.46:0/1454068456 learned_addr learned my addr 172.21.15.46:0/1454068456 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4783fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 0x7f478c0d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:33522/0 (socket says 172.21.15.46:33522) 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4791190640 1 -- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 msgr2=0x7f478c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4791190640 1 --2- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 0x7f478c0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4791190640 1 -- 172.21.15.46:0/1454068456 >> v1:172.21.15.88:6789/0 conn(0x7f478c0b5140 legacy=0x7f478c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f47837fe640 1 -- 172.21.15.46:0/1454068456 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 160148570 0 0) 0x7f478c0b5550 con 0x7f478c0b5140 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4791190640 1 -- 172.21.15.46:0/1454068456 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478c0d47c0 con 0x7f478c0d3df0 2024-05-09T03:18:20.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.620+0000 7f4783fff640 1 --2- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 0x7f478c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:20.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.621+0000 7f4791190640 1 --2- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0b9380 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4784009d90 tx=0x7f478402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6abd42ebb55253 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:20.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.621+0000 7f47837fe640 1 -- 172.21.15.46:0/1454068456 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784007740 con 0x7f478c0d3df0 2024-05-09T03:18:20.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.621+0000 7f47837fe640 1 -- 172.21.15.46:0/1454068456 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47840078e0 con 0x7f478c0d3df0 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.621+0000 7f47837fe640 1 -- 172.21.15.46:0/1454068456 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784007cb0 con 0x7f478c0d3df0 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 -- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 msgr2=0x7f478c0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 --2- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0b9380 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4784009d90 tx=0x7f478402a6f0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 -- 172.21.15.46:0/1454068456 shutdown_connections 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 --2- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0b9380 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 --2- 172.21.15.46:0/1454068456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b7e70 0x7f478c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 -- 172.21.15.46:0/1454068456 >> 172.21.15.46:0/1454068456 conn(0x7f478c08d3f0 msgr2=0x7f478c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 -- 172.21.15.46:0/1454068456 shutdown_connections 2024-05-09T03:18:20.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.622+0000 7f4792bf3640 1 -- 172.21.15.46:0/1454068456 wait complete. 2024-05-09T03:18:20.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.624+0000 7f4792bf3640 1 Processor -- start 2024-05-09T03:18:20.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.624+0000 7f4792bf3640 1 -- start start 2024-05-09T03:18:20.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4792bf3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b5140 0x7f478c0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4792bf3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4792bf3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 0x7f478c154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4792bf3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f478c0d4200 con 0x7f478c158270 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4792bf3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f478c0d4d80 con 0x7f478c0d3df0 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4792bf3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f478c0d5580 con 0x7f478c0b5140 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4791190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 0x7f478c154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4791190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 0x7f478c154b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50588/0 (socket says 172.21.15.46:50588) 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.625+0000 7f4791190640 1 -- 172.21.15.46:0/4128963652 learned_addr learned my addr 172.21.15.46:0/4128963652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4783fff640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f479098f640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b5140 0x7f478c0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4791190640 1 -- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b5140 msgr2=0x7f478c0c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4791190640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b5140 0x7f478c0c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4791190640 1 -- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 msgr2=0x7f478c0c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4791190640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4791190640 1 -- 172.21.15.46:0/4128963652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f478c1589d0 con 0x7f478c158270 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f479098f640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b5140 0x7f478c0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4783fff640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4791190640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 0x7f478c154b30 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f4784009eb0 tx=0x7f4784004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784040070 con 0x7f478c158270 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4784031c40 con 0x7f478c158270 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f478c158cb0 con 0x7f478c158270 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784009450 con 0x7f478c158270 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.626+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f478c159270 con 0x7f478c158270 2024-05-09T03:18:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.627+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f478c158740 con 0x7f478c158270 2024-05-09T03:18:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.628+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4784003990 con 0x7f478c158270 2024-05-09T03:18:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.629+0000 7f47817fa640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47780762d0 0x7f47780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.629+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7315+0+0 (secure 0 0 0) 0x7f47840b8a60 con 0x7f478c158270 2024-05-09T03:18:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.629+0000 7f479098f640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47780762d0 0x7f47780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:20.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.629+0000 7f479098f640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47780762d0 0x7f47780787e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4788000f60 tx=0x7f4788007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.633+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4784081bc0 con 0x7f478c158270 2024-05-09T03:18:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.865+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f478c163370 con 0x7f478c158270 2024-05-09T03:18:20.866 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:20.866+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.867+0000 7f47817fa640 1 -- 172.21.15.46:0/4128963652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v205) ==== 161+0+0 (secure 0 0 0) 0x7f47840814c0 con 0x7f478c158270 2024-05-09T03:18:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:20.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.870+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47780762d0 msgr2=0x7f47780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.870+0000 7f4792bf3640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47780762d0 0x7f47780787e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4788000f60 tx=0x7f4788007ca0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.870+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 msgr2=0x7f478c154b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:20.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.870+0000 7f4792bf3640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 0x7f478c154b30 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f4784009eb0 tx=0x7f4784004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:20.871 INFO:tasks.ceph.mds.c.smithi046.stderr: -1806> 2024-05-09T03:18:09.995+0000 7f2421472640 -1 mds.0.purge_queue operator(): Error -22 loading Journaler 2024-05-09T03:18:20.872 INFO:tasks.ceph.mds.c.smithi046.stderr: -1803> 2024-05-09T03:18:09.995+0000 7f2421472640 -1 mds.0.201 unhandled write error (22) Invalid argument, force readonly... 2024-05-09T03:18:20.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 shutdown_connections 2024-05-09T03:18:20.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47780762d0 0x7f47780787e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f478c158270 0x7f478c154b30 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f478c0d3df0 0x7f478c0c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 --2- 172.21.15.46:0/4128963652 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f478c0b5140 0x7f478c0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:20.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 >> 172.21.15.46:0/4128963652 conn(0x7f478c08d3f0 msgr2=0x7f478c161180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:20.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 shutdown_connections 2024-05-09T03:18:20.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:20.871+0000 7f4792bf3640 1 -- 172.21.15.46:0/4128963652 wait complete. 2024-05-09T03:18:20.873 INFO:tasks.ceph.mds.c.smithi046.stderr: -622> 2024-05-09T03:18:17.159+0000 7f2424478640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:18:20.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:20.929 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:21.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.000+0000 7f95f5205640 1 Processor -- start 2024-05-09T03:18:21.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.000+0000 7f95f5205640 1 -- start start 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.001+0000 7f95f5205640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95f00d1740 0x7f95f00b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.001+0000 7f95f5205640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95f00d1110 0x7f95f00d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.001+0000 7f95f5205640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f00b9690 con 0x7f95f00cfcf0 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.001+0000 7f95f5205640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95f008cd60 con 0x7f95f00d1740 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.001+0000 7f95f5205640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95f00ca9e0 con 0x7f95f00d1110 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95f00d1740 0x7f95f00b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95f00d1740 0x7f95f00b73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50612/0 (socket says 172.21.15.46:50612) 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 -- 172.21.15.46:0/3615910949 learned_addr learned my addr 172.21.15.46:0/3615910949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 -- 172.21.15.46:0/3615910949 >> v2:172.21.15.88:3300/0 conn(0x7f95f00d1110 msgr2=0x7f95f00d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95ee59c640 1 --2- 172.21.15.46:0/3615910949 >> v2:172.21.15.88:3300/0 conn(0x7f95f00d1110 0x7f95f00d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 --2- 172.21.15.46:0/3615910949 >> v2:172.21.15.88:3300/0 conn(0x7f95f00d1110 0x7f95f00d23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 -- 172.21.15.46:0/3615910949 >> v1:172.21.15.88:6789/0 conn(0x7f95f00cfcf0 legacy=0x7f95f009cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:21.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.002+0000 7f95eed9d640 1 -- 172.21.15.46:0/3615910949 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f008cfb0 con 0x7f95f00d1740 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.003+0000 7f95eed9d640 1 --2- 172.21.15.46:0/3615910949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f00b73d0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f95f00d1b80 tx=0x7f95e002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd0007e2261425a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.003+0000 7f95edd9b640 1 -- 172.21.15.46:0/3615910949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0037070 con 0x7f95f00d1740 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.003+0000 7f95edd9b640 1 -- 172.21.15.46:0/3615910949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e0032440 con 0x7f95f00d1740 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.003+0000 7f95edd9b640 1 -- 172.21.15.46:0/3615910949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0032810 con 0x7f95f00d1740 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.003+0000 7f95f5205640 1 -- 172.21.15.46:0/3615910949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 msgr2=0x7f95f00b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.003+0000 7f95f5205640 1 --2- 172.21.15.46:0/3615910949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f00b73d0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f95f00d1b80 tx=0x7f95e002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.004+0000 7f95f5205640 1 -- 172.21.15.46:0/3615910949 shutdown_connections 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.004+0000 7f95f5205640 1 --2- 172.21.15.46:0/3615910949 >> v2:172.21.15.88:3300/0 conn(0x7f95f00d1110 0x7f95f00d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.004+0000 7f95f5205640 1 --2- 172.21.15.46:0/3615910949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f00b73d0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.004+0000 7f95f5205640 1 -- 172.21.15.46:0/3615910949 >> 172.21.15.46:0/3615910949 conn(0x7f95f00d05f0 msgr2=0x7f95f00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:21.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.004+0000 7f95f5205640 1 -- 172.21.15.46:0/3615910949 shutdown_connections 2024-05-09T03:18:21.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.004+0000 7f95f5205640 1 -- 172.21.15.46:0/3615910949 wait complete. 2024-05-09T03:18:21.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.006+0000 7f95f5205640 1 Processor -- start 2024-05-09T03:18:21.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.006+0000 7f95f5205640 1 -- start start 2024-05-09T03:18:21.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95f5205640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95f00cfcf0 0x7f95f00a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95f5205640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95f00d1110 0x7f95f01f1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95f5205640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f01614d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95f5205640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95f009d3b0 con 0x7f95f00d1740 2024-05-09T03:18:21.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95f5205640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95f00b93b0 con 0x7f95f00cfcf0 2024-05-09T03:18:21.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95f5205640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95f00d2930 con 0x7f95f00d1110 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95ef59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f01614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95ef59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f01614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50622/0 (socket says 172.21.15.46:50622) 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95ef59e640 1 -- 172.21.15.46:0/817927979 learned_addr learned my addr 172.21.15.46:0/817927979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95eed9d640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95f00cfcf0 0x7f95f00a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.007+0000 7f95ee59c640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95f00d1110 0x7f95f01f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95ef59e640 1 -- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95f00d1110 msgr2=0x7f95f01f1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95ef59e640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95f00d1110 0x7f95f01f1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95ef59e640 1 -- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95f00cfcf0 msgr2=0x7f95f00a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95ef59e640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95f00cfcf0 0x7f95f00a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95ef59e640 1 -- 172.21.15.46:0/817927979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f01f4c50 con 0x7f95f00d1740 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95eed9d640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95f00cfcf0 0x7f95f00a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95ef59e640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f01614d0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f95e800b9e0 tx=0x7f95e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8003b70 con 0x7f95f00d1740 2024-05-09T03:18:21.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e8003d10 con 0x7f95f00d1740 2024-05-09T03:18:21.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f01f4f30 con 0x7f95f00d1740 2024-05-09T03:18:21.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e800cb60 con 0x7f95f00d1740 2024-05-09T03:18:21.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.008+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f01f54f0 con 0x7f95f00d1740 2024-05-09T03:18:21.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.009+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f00b65b0 con 0x7f95f00d1740 2024-05-09T03:18:21.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.010+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95e800cd00 con 0x7f95f00d1740 2024-05-09T03:18:21.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.010+0000 7f95d77fe640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95dc076300 0x7f95dc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.010+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7401+0+0 (secure 0 0 0) 0x7f95e8097a80 con 0x7f95f00d1740 2024-05-09T03:18:21.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.011+0000 7f95eed9d640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95dc076300 0x7f95dc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.011+0000 7f95eed9d640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95dc076300 0x7f95dc078810 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f95e0034000 tx=0x7f95e002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.015+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95f00b65b0 con 0x7f95f00d1740 2024-05-09T03:18:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.284+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f95f006a270 con 0x7f95f00d1740 2024-05-09T03:18:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.869+0000 7f95d77fe640 1 -- 172.21.15.46:0/817927979 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v206) ==== 107+0+0 (secure 0 0 0) 0x7f95e8060d40 con 0x7f95f00d1740 2024-05-09T03:18:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95dc076300 msgr2=0x7f95dc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95dc076300 0x7f95dc078810 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f95e0034000 tx=0x7f95e002e000 comp rx=0 tx=0).stop 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 msgr2=0x7f95f01614d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f01614d0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f95e800b9e0 tx=0x7f95e800bea0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 shutdown_connections 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95dc076300 0x7f95dc078810 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95f00d1740 0x7f95f01614d0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95f00d1110 0x7f95f01f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 --2- 172.21.15.46:0/817927979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95f00cfcf0 0x7f95f00a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 >> 172.21.15.46:0/817927979 conn(0x7f95f00d05f0 msgr2=0x7f95f009bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:21.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 shutdown_connections 2024-05-09T03:18:21.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.872+0000 7f95f5205640 1 -- 172.21.15.46:0/817927979 wait complete. 2024-05-09T03:18:21.888 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.948+0000 7feefba1ef40 1 Processor -- start 2024-05-09T03:18:21.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.948+0000 7feefba1ef40 1 -- start start 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feefba1ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abe34940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feefba1ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abeb83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feefba1ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abe309a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feefba1ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5598abd92f50 con 0x5598abde92a0 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feefba1ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5598abe2cd40 con 0x5598abebac90 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feefba1ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5598abe34e80 con 0x5598abec1610 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feef54e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abe34940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feef54e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abe34940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50640/0 (socket says 172.21.15.46:50640) 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.949+0000 7feef54e6640 1 -- 172.21.15.46:0/4154098595 learned_addr learned my addr 172.21.15.46:0/4154098595 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feeeffff640 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abeb83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef5ce7640 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abe309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef54e6640 1 -- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 msgr2=0x5598abe309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef54e6640 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abe309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef54e6640 1 -- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 msgr2=0x5598abeb83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef54e6640 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abeb83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef54e6640 1 -- 172.21.15.46:0/4154098595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598abe19020 con 0x5598abde92a0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef5ce7640 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abe309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feef54e6640 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abe34940 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7feee8009510 tx=0x7feee80099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7c765fea8f125a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feeef7fe640 1 -- 172.21.15.46:0/4154098595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8017070 con 0x5598abde92a0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feeef7fe640 1 -- 172.21.15.46:0/4154098595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8006580 con 0x5598abde92a0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.950+0000 7feeef7fe640 1 -- 172.21.15.46:0/4154098595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8006950 con 0x5598abde92a0 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 -- 172.21.15.46:0/4154098595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 msgr2=0x5598abe34940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abe34940 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7feee8009510 tx=0x7feee80099d0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 -- 172.21.15.46:0/4154098595 shutdown_connections 2024-05-09T03:18:21.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abe309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abeb83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 --2- 172.21.15.46:0/4154098595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abe34940 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 -- 172.21.15.46:0/4154098595 >> 172.21.15.46:0/4154098595 conn(0x5598abd0eac0 msgr2=0x5598abe171a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 -- 172.21.15.46:0/4154098595 shutdown_connections 2024-05-09T03:18:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.951+0000 7feefba1ef40 1 -- 172.21.15.46:0/4154098595 wait complete. 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.952+0000 7feefba1ef40 1 Processor -- start 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.952+0000 7feefba1ef40 1 -- start start 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feefba1ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abeb3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feefba1ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abe2f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feefba1ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abf5cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feefba1ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5598abe18dd0 con 0x5598abde92a0 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feefba1ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5598abeb2700 con 0x5598abebac90 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feefba1ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5598abeb8cb0 con 0x5598abec1610 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feef54e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abeb3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feef54e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abeb3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50648/0 (socket says 172.21.15.46:50648) 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 learned_addr learned my addr 172.21.15.46:0/2657824515 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abe2f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.953+0000 7feef5ce7640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abf5cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 msgr2=0x5598abf5cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5598abec1610 0x5598abf5cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 msgr2=0x5598abe2f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598abebac90 0x5598abe2f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598abeb8a00 con 0x5598abde92a0 2024-05-09T03:18:21.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abeb3440 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7feee8015040 tx=0x7feee8003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feeee7fc640 1 -- 172.21.15.46:0/2657824515 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8020020 con 0x5598abde92a0 2024-05-09T03:18:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feeee7fc640 1 -- 172.21.15.46:0/2657824515 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee8018040 con 0x5598abde92a0 2024-05-09T03:18:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5598abeb03d0 con 0x5598abde92a0 2024-05-09T03:18:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feeee7fc640 1 -- 172.21.15.46:0/2657824515 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8017070 con 0x5598abde92a0 2024-05-09T03:18:21.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.954+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598abe1b840 con 0x5598abde92a0 2024-05-09T03:18:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.956+0000 7feeee7fc640 1 -- 172.21.15.46:0/2657824515 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feee8023020 con 0x5598abde92a0 2024-05-09T03:18:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.956+0000 7feeee7fc640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed8074ce0 0x7feed80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.956+0000 7feeee7fc640 1 -- 172.21.15.46:0/2657824515 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7401+0+0 (secure 0 0 0) 0x7feee8097280 con 0x5598abde92a0 2024-05-09T03:18:21.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.956+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed8074ce0 0x7feed80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.959 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.957+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5598abf5d1d0 0x5598abf5d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.957+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abdd84c0 con 0x5598abf5d1d0 2024-05-09T03:18:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.957+0000 7feef5ce7640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5598abf5d1d0 0x5598abf5d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.957+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed8074ce0 0x7feed80771f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7feee0057000 tx=0x7feee0052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.957+0000 7feef5ce7640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5598abf5d1d0 0x5598abf5d5e0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.959+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v103'567 uv541 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.959+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abe2f8c0 con 0x5598abf5d1d0 2024-05-09T03:18:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.961+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v103'568 uv568 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.961+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf65f60 con 0x5598abf5d1d0 2024-05-09T03:18:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.962+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v103'569 uv569 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.963 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:21.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.962+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf67310 con 0x5598abf5d1d0 2024-05-09T03:18:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.964+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v103'570 uv570 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.964+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abeb2120 con 0x5598abf5d1d0 2024-05-09T03:18:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.965+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v103'571 uv571 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.965+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66300 con 0x5598abf5d1d0 2024-05-09T03:18:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.967+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v103'572 uv572 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.967+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf666f0 con 0x5598abf5d1d0 2024-05-09T03:18:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.969+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v103'573 uv547 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.969+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf66ae0 con 0x5598abf5d1d0 2024-05-09T03:18:21.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.970+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v103'574 uv574 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.970+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66ae0 con 0x5598abf5d1d0 2024-05-09T03:18:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.971+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v103'575 uv575 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.972 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:21.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.971+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.973+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v103'576 uv576 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.973+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.974+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v103'577 uv577 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.974+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.976+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v103'578 uv578 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.976+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.977+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v103'579 uv554 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.977+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.978+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v103'580 uv580 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.978+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.980+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v103'581 uv581 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.980 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:21.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.980+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.981+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v103'582 uv582 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.981+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.982+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v103'583 uv583 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.982+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66a90 con 0x5598abf5d1d0 2024-05-09T03:18:21.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.984+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v103'584 uv584 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.984+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66e50 con 0x5598abf5d1d0 2024-05-09T03:18:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.985+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v103'585 uv585 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.985+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66e50 con 0x5598abf5d1d0 2024-05-09T03:18:21.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.986+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v103'586 uv560 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.986+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.988+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v103'587 uv587 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.988+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.989+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v103'588 uv588 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.990 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:21.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.989+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.991+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v103'589 uv589 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.991+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.992+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v103'590 uv590 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.992+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.993+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v103'591 uv591 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.993+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.995+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v103'592 uv566 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.995+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.996+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v103'593 uv593 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.996+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.997+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v103'594 uv594 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.998 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:21.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.997+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:21.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.998+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v103'595 uv595 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:21.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:21.998+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.000+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v103'596 uv596 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.000+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf5d1d0 2024-05-09T03:18:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.001+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v103'597 uv597 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.001+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5598abf67bc0 0x5598abf69fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.001+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf66d00 con 0x5598abf67bc0 2024-05-09T03:18:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.001+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5598abf67bc0 0x5598abf69fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.002+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5598abf67bc0 0x5598abf69fe0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.003+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v103'417 uv397 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.003+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6a520 con 0x5598abf67bc0 2024-05-09T03:18:22.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.005+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v103'418 uv418 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.005+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6a8c0 con 0x5598abf67bc0 2024-05-09T03:18:22.007 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:22.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.006+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v103'419 uv419 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.007+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6b110 con 0x5598abf67bc0 2024-05-09T03:18:22.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.008+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v103'420 uv420 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.008+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6b4b0 con 0x5598abf67bc0 2024-05-09T03:18:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.010+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v103'421 uv421 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.010+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6b850 con 0x5598abf67bc0 2024-05-09T03:18:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.011+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v103'422 uv416 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.011+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.012+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v103'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.012+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.014+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v103'424 uv424 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.015 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:22.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.014+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.015+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v103'425 uv425 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.015+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.017+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v103'426 uv426 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.017+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.019+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v103'427 uv415 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.019+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.020+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v103'428 uv428 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.020+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.022+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v103'429 uv429 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.023 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:22.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.022+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f650 con 0x5598abf67bc0 2024-05-09T03:18:22.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.023+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v103'430 uv430 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.023+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f650 con 0x5598abf67bc0 2024-05-09T03:18:22.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.025+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v103'431 uv431 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.025+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.026+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v103'432 uv413 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.026+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.027+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v103'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.027+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.029+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v103'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.029 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:22.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.029+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6cc00 con 0x5598abf67bc0 2024-05-09T03:18:22.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.030+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v103'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.030+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6cc00 con 0x5598abf67bc0 2024-05-09T03:18:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.032+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v103'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.032+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf67bc0 2024-05-09T03:18:22.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.033+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v103'437 uv437 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf67bc0 2024-05-09T03:18:22.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.033+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf5d1d0 2024-05-09T03:18:22.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.034+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v103'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.034+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf5d1d0 2024-05-09T03:18:22.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.035+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v103'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.035+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf5d1d0 2024-05-09T03:18:22.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.036+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v103'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.037 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:22.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.036+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6cc00 con 0x5598abf5d1d0 2024-05-09T03:18:22.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.037+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v103'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.037+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6cc00 con 0x5598abf5d1d0 2024-05-09T03:18:22.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.038+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v103'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.038+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6bbf0 con 0x5598abf5d1d0 2024-05-09T03:18:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.039+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v103'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.039+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5598abf6c100 0x5598abf6e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.039+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6eaa0 con 0x5598abf6c100 2024-05-09T03:18:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.039+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5598abf6c100 0x5598abf6e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.040+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5598abf6c100 0x5598abf6e520 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.041+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v103'86 uv85 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.041+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6efb0 con 0x5598abf6c100 2024-05-09T03:18:22.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.042+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v103'87 uv87 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.042+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f4b0 con 0x5598abf6c100 2024-05-09T03:18:22.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.043+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v103'88 uv88 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.044 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:22.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.043+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6ad40 con 0x5598abf6c100 2024-05-09T03:18:22.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.044+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v103'89 uv89 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.044+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6fde0 con 0x5598abf5d1d0 2024-05-09T03:18:22.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.045+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v103'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.046+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6fde0 con 0x5598abf5d1d0 2024-05-09T03:18:22.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.047+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v103'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.047+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6fde0 con 0x5598abf5d1d0 2024-05-09T03:18:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.048+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v103'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.049 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:22.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.048+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf70df0 con 0x5598abf5d1d0 2024-05-09T03:18:22.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.049+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v103'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.049+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf70df0 con 0x5598abf5d1d0 2024-05-09T03:18:22.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.050+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v103'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.050+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6fde0 con 0x5598abf5d1d0 2024-05-09T03:18:22.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.052+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v103'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.052+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5598abf706b0 0x5598abf72ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.052+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73010 con 0x5598abf706b0 2024-05-09T03:18:22.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.052+0000 7feef5ce7640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5598abf706b0 0x5598abf72ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.052+0000 7feef5ce7640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5598abf706b0 0x5598abf72ad0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.054+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v103'205 uv198 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.054+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf73050 con 0x5598abf706b0 2024-05-09T03:18:22.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.055+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v103'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.055+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.057 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:22.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.056+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v103'207 uv207 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.056+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf74450 con 0x5598abf706b0 2024-05-09T03:18:22.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.057+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v103'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.058+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf74450 con 0x5598abf706b0 2024-05-09T03:18:22.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.059+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v103'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.059+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.060+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v103'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.060+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.061+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v103'211 uv204 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.062+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.063+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v103'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.063+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.065 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:22.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.064+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v103'213 uv213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.064+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.065+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v103'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.065+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf73440 con 0x5598abf706b0 2024-05-09T03:18:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.067+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v103'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.067+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5598abf73950 0x5598abf75db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.067+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf76500 con 0x5598abf73950 2024-05-09T03:18:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.067+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5598abf73950 0x5598abf75db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.067+0000 7feef54e6640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5598abf73950 0x5598abf75db0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.068+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v103'91 uv90 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf73950 2024-05-09T03:18:22.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.068+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf73950 2024-05-09T03:18:22.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.069+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v103'92 uv92 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf73950 2024-05-09T03:18:22.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.070+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf73950 2024-05-09T03:18:22.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.071+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v103'93 uv93 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf73950 2024-05-09T03:18:22.071 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:22.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.071+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf73950 2024-05-09T03:18:22.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.072+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v103'94 uv94 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf73950 2024-05-09T03:18:22.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.072+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf73950 2024-05-09T03:18:22.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.073+0000 7feef54e6640 1 -- 172.21.15.46:0/2657824515 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v103'95 uv95 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee801a070 con 0x5598abf73950 2024-05-09T03:18:22.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.073+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf5d1d0 2024-05-09T03:18:22.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.074+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v103'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.074+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf5d1d0 2024-05-09T03:18:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.075+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v103'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.075+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf5d1d0 2024-05-09T03:18:22.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.076+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v103'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.077 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:22.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.076+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf77920 con 0x5598abf5d1d0 2024-05-09T03:18:22.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.077+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v103'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.077+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf778e0 con 0x5598abf5d1d0 2024-05-09T03:18:22.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.078+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v103'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.078+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf5d1d0 2024-05-09T03:18:22.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.079+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v103'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf5d1d0 2024-05-09T03:18:22.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.079+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf768d0 con 0x5598abf6c100 2024-05-09T03:18:22.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.080+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v103'182 uv176 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.081+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf76c70 con 0x5598abf6c100 2024-05-09T03:18:22.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.082+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v103'183 uv183 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.082+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf77060 con 0x5598abf6c100 2024-05-09T03:18:22.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.083+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v103'184 uv184 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.083 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:22.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.083+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f960 con 0x5598abf6c100 2024-05-09T03:18:22.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.084+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v103'185 uv185 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.084+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f960 con 0x5598abf6c100 2024-05-09T03:18:22.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.085+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v103'186 uv186 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.085+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f960 con 0x5598abf6c100 2024-05-09T03:18:22.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.086+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v103'187 uv181 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.086+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf6f960 con 0x5598abf6c100 2024-05-09T03:18:22.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.087+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v103'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.087+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f960 con 0x5598abf6c100 2024-05-09T03:18:22.088 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:22.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.088+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v103'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.088+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf6f960 con 0x5598abf6c100 2024-05-09T03:18:22.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.089+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v103'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.089+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a660 con 0x5598abf6c100 2024-05-09T03:18:22.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.090+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v103'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.090+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5598abf78210 0x5598abf7aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.090+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf78210 2024-05-09T03:18:22.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.090+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5598abf78210 0x5598abf7aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.091+0000 7feeeffff640 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5598abf78210 0x5598abf7aa50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.092+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v103'83 uv81 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf78210 2024-05-09T03:18:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.092+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf78210 2024-05-09T03:18:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.093+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v103'84 uv84 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf78210 2024-05-09T03:18:22.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.093+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf78210 2024-05-09T03:18:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.094+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v103'85 uv85 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf78210 2024-05-09T03:18:22.095 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.094+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf78210 2024-05-09T03:18:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.095+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v103'86 uv86 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf78210 2024-05-09T03:18:22.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.095+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf6c100 2024-05-09T03:18:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.096+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v103'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.096+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf6c100 2024-05-09T03:18:22.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.097+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v103'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.097+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf6c100 2024-05-09T03:18:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.099+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v103'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.099 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:22.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.099+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf6c100 2024-05-09T03:18:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.100+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v103'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.100+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf6c100 2024-05-09T03:18:22.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.101+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v103'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.101+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf6c100 2024-05-09T03:18:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.102+0000 7feeeffff640 1 -- 172.21.15.46:0/2657824515 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v103'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feee0051070 con 0x5598abf6c100 2024-05-09T03:18:22.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.102+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.104+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v103'94 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.104+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.105+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v103'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.105+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.106+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v103'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.107 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:22.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.107+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.108+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v103'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.108+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.109+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v103'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.109+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.110+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v103'229 uv220 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.110+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.112+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v103'230 uv230 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.112+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.113+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v103'231 uv231 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.114 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:22.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.113+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.115+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v103'232 uv232 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.115+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.116+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v103'233 uv233 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.116+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.117+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v103'234 uv227 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.117+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.118+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v103'235 uv235 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.118+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.119+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v103'236 uv236 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.120 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:22.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.119+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.120+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v103'237 uv237 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.120+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.121+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v103'238 uv238 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.121+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7a630 con 0x5598abf706b0 2024-05-09T03:18:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.122+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v103'239 uv239 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.123+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5598abf7af90 con 0x5598abf706b0 2024-05-09T03:18:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v103'240 uv240 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7feef000d670 con 0x5598abf706b0 2024-05-09T03:18:22.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5598abf5d1d0 msgr2=0x5598abf5d5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5598abf5d1d0 0x5598abf5d5e0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5598abf6c100 msgr2=0x5598abf6e520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5598abf6c100 0x5598abf6e520 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5598abf78210 msgr2=0x5598abf7aa50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5598abf78210 0x5598abf7aa50 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5598abf67bc0 msgr2=0x5598abf69fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5598abf67bc0 0x5598abf69fe0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 reap_dead start 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5598abf706b0 msgr2=0x5598abf72ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5598abf706b0 0x5598abf72ad0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5598abf73950 msgr2=0x5598abf75db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5598abf73950 0x5598abf75db0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed8074ce0 msgr2=0x7feed80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feed8074ce0 0x7feed80771f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7feee0057000 tx=0x7feee0052000 comp rx=0 tx=0).stop 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 msgr2=0x5598abeb3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 --2- 172.21.15.46:0/2657824515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5598abde92a0 0x5598abeb3440 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7feee8015040 tx=0x7feee8003020 comp rx=0 tx=0).stop 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feef5ce7640 1 -- 172.21.15.46:0/2657824515 reap_dead start 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 shutdown_connections 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 >> 172.21.15.46:0/2657824515 conn(0x5598abd0eac0 msgr2=0x5598abe309a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 shutdown_connections 2024-05-09T03:18:22.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.124+0000 7feefba1ef40 1 -- 172.21.15.46:0/2657824515 wait complete. 2024-05-09T03:18:22.128 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds_snaptable - 2024-05-09T03:18:22.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.224+0000 7fd2fa3edf40 1 Processor -- start 2024-05-09T03:18:22.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.225+0000 7fd2fa3edf40 1 -- start start 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2fa3edf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653368b63a0 0x565336946dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2fa3edf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 0x5653368a4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2fa3edf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56533694a150 0x5653368a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2fa3edf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56533681cf50 con 0x5653368be5f0 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2fa3edf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x565336820f90 con 0x56533694a150 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2fa3edf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5653368bf8d0 con 0x5653368b63a0 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 0x5653368a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 0x5653368a4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50662/0 (socket says 172.21.15.46:50662) 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 -- 172.21.15.46:0/3679296164 learned_addr learned my addr 172.21.15.46:0/3679296164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2f3eb7640 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653368b63a0 0x565336946dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2f46b8640 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56533694a150 0x5653368a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 -- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653368b63a0 msgr2=0x565336946dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653368b63a0 0x565336946dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 -- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56533694a150 msgr2=0x5653368a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56533694a150 0x5653368a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 -- 172.21.15.46:0/3679296164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653368a0d10 con 0x5653368be5f0 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.226+0000 7fd2ebfff640 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 0x5653368a4180 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc025a00 tx=0x7fd2dc04e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0bbd81455e207f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2f2eb5640 1 -- 172.21.15.46:0/3679296164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc064070 con 0x5653368be5f0 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2f2eb5640 1 -- 172.21.15.46:0/3679296164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2dc0548b0 con 0x5653368be5f0 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2f2eb5640 1 -- 172.21.15.46:0/3679296164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc059040 con 0x5653368be5f0 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/3679296164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 msgr2=0x5653368a4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 0x5653368a4180 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc025a00 tx=0x7fd2dc04e000 comp rx=0 tx=0).stop 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/3679296164 shutdown_connections 2024-05-09T03:18:22.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56533694a150 0x5653368a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368be5f0 0x5653368a4180 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/3679296164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5653368b63a0 0x565336946dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/3679296164 >> 172.21.15.46:0/3679296164 conn(0x565336798ac0 msgr2=0x5653368a36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/3679296164 shutdown_connections 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.227+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/3679296164 wait complete. 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.228+0000 7fd2fa3edf40 1 Processor -- start 2024-05-09T03:18:22.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.228+0000 7fd2fa3edf40 1 -- start start 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368b63a0 0x565336934d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653368be5f0 0x5653368a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56533694a150 0x5653369e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5653368a0ff0 con 0x5653368b63a0 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x565336948230 con 0x5653368be5f0 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x565336947be0 con 0x56533694a150 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368b63a0 0x565336934d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368b63a0 0x565336934d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50678/0 (socket says 172.21.15.46:50678) 2024-05-09T03:18:22.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 -- 172.21.15.46:0/2541233035 learned_addr learned my addr 172.21.15.46:0/2541233035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2ebfff640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653368be5f0 0x5653368a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f46b8640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56533694a150 0x5653369e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 -- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56533694a150 msgr2=0x5653369e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56533694a150 0x5653369e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 -- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653368be5f0 msgr2=0x5653368a42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5653368be5f0 0x5653368a42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 -- 172.21.15.46:0/2541233035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565336935290 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f46b8640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56533694a150 0x5653369e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f3eb7640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368b63a0 0x565336934d50 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400ee90 tx=0x7fd2e400c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f1eb3640 1 -- 172.21.15.46:0/2541233035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e400f860 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f1eb3640 1 -- 172.21.15.46:0/2541233035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e4013030 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2f1eb3640 1 -- 172.21.15.46:0/2541233035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e400fc00 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.230+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5653368a06f0 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.231+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653369e1600 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.232+0000 7fd2f1eb3640 1 -- 172.21.15.46:0/2541233035 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd2e40173d0 con 0x5653368b63a0 2024-05-09T03:18:22.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2f1eb3640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2d4074cb0 0x7fd2d40771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2f1eb3640 1 -- 172.21.15.46:0/2541233035 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7401+0+0 (secure 0 0 0) 0x7fd2e4097260 con 0x5653368b63a0 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2ebfff640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2d4074cb0 0x7fd2d40771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5653369e2340 0x565336944140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5653368a47e0 con 0x5653369e2340 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2f46b8640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5653369e2340 0x565336944140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.233+0000 7fd2ebfff640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2d4074cb0 0x7fd2d40771c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00a4f0 tx=0x7fd2dc051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.234+0000 7fd2f46b8640 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5653369e2340 0x565336944140 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2f46b8640 1 -- 172.21.15.46:0/2541233035 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 mds_snaptable [writefull 0~80] v103'96 uv96 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2ec002660 con 0x5653369e2340 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5653369e2340 msgr2=0x565336944140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5653369e2340 0x565336944140 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2d4074cb0 msgr2=0x7fd2d40771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2d4074cb0 0x7fd2d40771c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00a4f0 tx=0x7fd2dc051000 comp rx=0 tx=0).stop 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368b63a0 msgr2=0x565336934d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2fa3edf40 1 --2- 172.21.15.46:0/2541233035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653368b63a0 0x565336934d50 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400ee90 tx=0x7fd2e400c730 comp rx=0 tx=0).stop 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.235+0000 7fd2f46b8640 1 -- 172.21.15.46:0/2541233035 reap_dead start 2024-05-09T03:18:22.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.236+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 shutdown_connections 2024-05-09T03:18:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.236+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 >> 172.21.15.46:0/2541233035 conn(0x565336798ac0 msgr2=0x5653368a36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.236+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 shutdown_connections 2024-05-09T03:18:22.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.236+0000 7fd2fa3edf40 1 -- 172.21.15.46:0/2541233035 wait complete. 2024-05-09T03:18:22.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:22.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.397+0000 7f96b7727640 1 Processor -- start 2024-05-09T03:18:22.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.397+0000 7f96b7727640 1 -- start start 2024-05-09T03:18:22.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b7727640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96b00d16b0 0x7f96b0071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b7727640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b00d2b40 con 0x7f96b00b9300 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b7727640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b00d2e20 con 0x7f96b00d25a0 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b7727640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f96b00b70d0 con 0x7f96b00d16b0 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b4cc2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96b00b9300 0x7f96b015d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48554/0 (socket says 172.21.15.46:48554) 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b4cc2640 1 -- 172.21.15.46:0/2115797596 learned_addr learned my addr 172.21.15.46:0/2115797596 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.398+0000 7f96b54c3640 1 --2- 172.21.15.46:0/2115797596 >> v2:172.21.15.88:3300/0 conn(0x7f96b00d16b0 0x7f96b0071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915603998 0 0) 0x7f96b00d2b40 con 0x7f96b00b9300 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9698003520 con 0x7f96b00b9300 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939606934 0 0) 0x7f96b00d2e20 con 0x7f96b00d25a0 2024-05-09T03:18:22.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96b00d2b40 con 0x7f96b00d25a0 2024-05-09T03:18:22.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96b54c3640 1 -- 172.21.15.46:0/2115797596 >> v1:172.21.15.88:6789/0 conn(0x7f96b00d25a0 legacy=0x7f96b015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96b54c3640 1 -- 172.21.15.46:0/2115797596 >> v1:172.21.15.46:6789/0 conn(0x7f96b00b9300 legacy=0x7f96b015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.399+0000 7f96b54c3640 1 -- 172.21.15.46:0/2115797596 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b00d3100 con 0x7f96b00d16b0 2024-05-09T03:18:22.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96b54c3640 1 --2- 172.21.15.46:0/2115797596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0071570 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f96b00b7250 tx=0x7f96a802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b07d968148548829 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8037070 con 0x7f96b00d16b0 2024-05-09T03:18:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a8032440 con 0x7f96b00d16b0 2024-05-09T03:18:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96affff640 1 -- 172.21.15.46:0/2115797596 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8032810 con 0x7f96b00d16b0 2024-05-09T03:18:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96b7727640 1 -- 172.21.15.46:0/2115797596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 msgr2=0x7f96b0071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96b7727640 1 --2- 172.21.15.46:0/2115797596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0071570 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f96b00b7250 tx=0x7f96a802a210 comp rx=0 tx=0).stop 2024-05-09T03:18:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96b7727640 1 -- 172.21.15.46:0/2115797596 shutdown_connections 2024-05-09T03:18:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96b7727640 1 --2- 172.21.15.46:0/2115797596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0071570 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.400+0000 7f96b7727640 1 -- 172.21.15.46:0/2115797596 >> 172.21.15.46:0/2115797596 conn(0x7f96b00d3ae0 msgr2=0x7f96b00d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:22.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.401+0000 7f96b7727640 1 -- 172.21.15.46:0/2115797596 shutdown_connections 2024-05-09T03:18:22.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.401+0000 7f96b7727640 1 -- 172.21.15.46:0/2115797596 wait complete. 2024-05-09T03:18:22.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.402+0000 7f96b7727640 1 Processor -- start 2024-05-09T03:18:22.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.402+0000 7f96b7727640 1 -- start start 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b7727640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96b00b9300 0x7f96b00c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b7727640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b7727640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 0x7f96b0147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b7727640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96b00b31c0 con 0x7f96b00d25a0 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b7727640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96b015e550 con 0x7f96b00d16b0 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b7727640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f96b00b3340 con 0x7f96b00b9300 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 0x7f96b0147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 0x7f96b0147970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50690/0 (socket says 172.21.15.46:50690) 2024-05-09T03:18:22.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 -- 172.21.15.46:0/1388524777 learned_addr learned my addr 172.21.15.46:0/1388524777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b4cc2640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b54c3640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96b00b9300 0x7f96b00c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 -- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96b00b9300 msgr2=0x7f96b00c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96b00b9300 0x7f96b00c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 -- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 msgr2=0x7f96b0145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 -- 172.21.15.46:0/1388524777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b0144b20 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96b5cc4640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 0x7f96b0147970 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f96a000c3c0 tx=0x7f96a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.404+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0017070 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.405+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a00052d0 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.405+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b0144e00 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.405+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0003830 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.405+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b01453c0 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.406+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b006a270 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.407+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f96a001a070 con 0x7f96b00d25a0 2024-05-09T03:18:22.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.407+0000 7f96adffb640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f969c0762d0 0x7f969c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:22.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.407+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7401+0+0 (secure 0 0 0) 0x7f96a0097340 con 0x7f96b00d25a0 2024-05-09T03:18:22.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.407+0000 7f96b54c3640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f969c0762d0 0x7f969c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:22.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.408+0000 7f96b54c3640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f969c0762d0 0x7f969c0787e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f96a8034000 tx=0x7f96a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:22.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.411+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f96a0021020 con 0x7f96b00d25a0 2024-05-09T03:18:22.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:22.639+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f96b00717c0 con 0x7f96b00d25a0 2024-05-09T03:18:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.210+0000 7f96adffb640 1 -- 172.21.15.46:0/1388524777 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v207) ==== 162+0+0 (secure 0 0 0) 0x7f96a0060000 con 0x7f96b00d25a0 2024-05-09T03:18:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f969c0762d0 msgr2=0x7f969c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f969c0762d0 0x7f969c0787e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f96a8034000 tx=0x7f96a802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 msgr2=0x7f96b0147970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 0x7f96b0147970 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f96a000c3c0 tx=0x7f96a000c880 comp rx=0 tx=0).stop 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 shutdown_connections 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f969c0762d0 0x7f969c0787e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96b00d25a0 0x7f96b0147970 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96b00d16b0 0x7f96b0145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 --2- 172.21.15.46:0/1388524777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f96b00b9300 0x7f96b00c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 >> 172.21.15.46:0/1388524777 conn(0x7f96b00d3ae0 msgr2=0x7f96b01607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 shutdown_connections 2024-05-09T03:18:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.213+0000 7f96b7727640 1 -- 172.21.15.46:0/1388524777 wait complete. 2024-05-09T03:18:23.220 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:23.219+0000 7febc6360640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': virtual void SnapServer::decode_server_state(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 5 < 101: Malformed input [buffer:3] 2024-05-09T03:18:23.220 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:23.220+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:23.220 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:23.220+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:23.226 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:18:23.219+0000 7febc6360640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': virtual void SnapServer::decode_server_state(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 5 < 101: Malformed input [buffer:3] 2024-05-09T03:18:23.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:23.289 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:23.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.357+0000 7faf68415640 1 Processor -- start 2024-05-09T03:18:23.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.357+0000 7faf68415640 1 -- start start 2024-05-09T03:18:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf68415640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf600b5210 0x7faf601613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf68415640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf600d4800 0x7faf600d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf68415640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf600d5280 con 0x7faf600b77c0 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf68415640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faf600b5620 con 0x7faf600b5210 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf68415640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faf601609a0 con 0x7faf600d4800 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf661b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf600b5210 0x7faf601613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf669b2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faf600b77c0 0x7faf600b7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48586/0 (socket says 172.21.15.46:48586) 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf669b2640 1 -- 172.21.15.46:0/1420072763 learned_addr learned my addr 172.21.15.46:0/1420072763 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf661b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf600b5210 0x7faf601613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50722/0 (socket says 172.21.15.46:50722) 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf659b0640 1 --2- 172.21.15.46:0/1420072763 >> v2:172.21.15.88:3300/0 conn(0x7faf600d4800 0x7faf600d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf661b1640 1 -- 172.21.15.46:0/1420072763 >> v2:172.21.15.88:3300/0 conn(0x7faf600d4800 msgr2=0x7faf600d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf661b1640 1 --2- 172.21.15.46:0/1420072763 >> v2:172.21.15.88:3300/0 conn(0x7faf600d4800 0x7faf600d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.359+0000 7faf661b1640 1 -- 172.21.15.46:0/1420072763 >> v1:172.21.15.46:6789/0 conn(0x7faf600b77c0 legacy=0x7faf600b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.360+0000 7faf651af640 1 -- 172.21.15.46:0/1420072763 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1094078538 0 0) 0x7faf600d5280 con 0x7faf600b77c0 2024-05-09T03:18:23.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.360+0000 7faf661b1640 1 -- 172.21.15.46:0/1420072763 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf600d5560 con 0x7faf600b5210 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.360+0000 7faf661b1640 1 --2- 172.21.15.46:0/1420072763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b5210 0x7faf601613c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7faf6006ee20 tx=0x7faf5c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56184b29d17d970d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.360+0000 7faf651af640 1 -- 172.21.15.46:0/1420072763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c037070 con 0x7faf600b5210 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.360+0000 7faf651af640 1 -- 172.21.15.46:0/1420072763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf5c032440 con 0x7faf600b5210 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.360+0000 7faf651af640 1 -- 172.21.15.46:0/1420072763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c02ae60 con 0x7faf600b5210 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 -- 172.21.15.46:0/1420072763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b5210 msgr2=0x7faf601613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 --2- 172.21.15.46:0/1420072763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b5210 0x7faf601613c0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7faf6006ee20 tx=0x7faf5c02a110 comp rx=0 tx=0).stop 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 -- 172.21.15.46:0/1420072763 shutdown_connections 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 --2- 172.21.15.46:0/1420072763 >> v2:172.21.15.88:3300/0 conn(0x7faf600d4800 0x7faf600d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 --2- 172.21.15.46:0/1420072763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b5210 0x7faf601613c0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 -- 172.21.15.46:0/1420072763 >> 172.21.15.46:0/1420072763 conn(0x7faf600d29a0 msgr2=0x7faf600d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 -- 172.21.15.46:0/1420072763 shutdown_connections 2024-05-09T03:18:23.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.361+0000 7faf68415640 1 -- 172.21.15.46:0/1420072763 wait complete. 2024-05-09T03:18:23.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.363+0000 7faf68415640 1 Processor -- start 2024-05-09T03:18:23.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.363+0000 7faf68415640 1 -- start start 2024-05-09T03:18:23.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.364+0000 7faf68415640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf600b5210 0x7faf600ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.364+0000 7faf68415640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 0x7faf60151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.364+0000 7faf68415640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf600d4800 0x7faf600cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.364+0000 7faf68415640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf600d0ae0 con 0x7faf600b77c0 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.364+0000 7faf68415640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faf60160ea0 con 0x7faf600d4800 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.364+0000 7faf68415640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faf60163e60 con 0x7faf600b5210 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 0x7faf60151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 0x7faf60151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50732/0 (socket says 172.21.15.46:50732) 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 -- 172.21.15.46:0/248668775 learned_addr learned my addr 172.21.15.46:0/248668775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf669b2640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf600d4800 0x7faf600cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf661b1640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf600b5210 0x7faf600ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 -- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf600b5210 msgr2=0x7faf600ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf600b5210 0x7faf600ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 -- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf600d4800 msgr2=0x7faf600cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf600d4800 0x7faf600cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 -- 172.21.15.46:0/248668775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf600cda10 con 0x7faf600b77c0 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf659b0640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 0x7faf60151fc0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7faf5000d620 tx=0x7faf5000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5000b620 con 0x7faf600b77c0 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf50019070 con 0x7faf600b77c0 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf600c9dc0 con 0x7faf600b77c0 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5000ba00 con 0x7faf600b77c0 2024-05-09T03:18:23.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.365+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf600cdf90 con 0x7faf600b77c0 2024-05-09T03:18:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.366+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf6006a270 con 0x7faf600b77c0 2024-05-09T03:18:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.367+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faf5000bba0 con 0x7faf600b77c0 2024-05-09T03:18:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.368+0000 7faf56ffd640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf44076300 0x7faf44078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.368+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7faf500980b0 con 0x7faf600b77c0 2024-05-09T03:18:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.368+0000 7faf661b1640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf44076300 0x7faf44078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.369+0000 7faf661b1640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf44076300 0x7faf44078810 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7faf5c034000 tx=0x7faf5c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:23.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.372+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faf50060bc0 con 0x7faf600b77c0 2024-05-09T03:18:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.601+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf600c9610 con 0x7faf600b77c0 2024-05-09T03:18:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.601+0000 7faf56ffd640 1 -- 172.21.15.46:0/248668775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+4343 (secure 0 0 0) 0x7faf50064ec0 con 0x7faf600b77c0 2024-05-09T03:18:23.603 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:23.603 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:23.220659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf44076300 msgr2=0x7faf44078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf44076300 0x7faf44078810 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7faf5c034000 tx=0x7faf5c02e000 comp rx=0 tx=0).stop 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 msgr2=0x7faf60151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 0x7faf60151fc0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7faf5000d620 tx=0x7faf5000dae0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 shutdown_connections 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf600d4800 0x7faf600cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf44076300 0x7faf44078810 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf600b77c0 0x7faf60151fc0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 --2- 172.21.15.46:0/248668775 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf600b5210 0x7faf600ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 >> 172.21.15.46:0/248668775 conn(0x7faf600d29a0 msgr2=0x7faf6006ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 shutdown_connections 2024-05-09T03:18:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.604+0000 7faf68415640 1 -- 172.21.15.46:0/248668775 wait complete. 2024-05-09T03:18:23.616 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:23.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.729+0000 7f34f7d5e640 1 Processor -- start 2024-05-09T03:18:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.730+0000 7f34f7d5e640 1 -- start start 2024-05-09T03:18:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f7d5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f34f00b5210 0x7f34f01613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f7d5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b7ed0 0x7f34f00d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f7d5e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f00d5280 con 0x7f34f00d4800 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f7d5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f34f00b5620 con 0x7f34f00b5210 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f7d5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34f01609a0 con 0x7f34f00b7ed0 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f52f9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34f00d4800 0x7f34f00b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48592/0 (socket says 172.21.15.46:48592) 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f52f9640 1 -- 172.21.15.46:0/478537337 learned_addr learned my addr 172.21.15.46:0/478537337 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f5afa640 1 --2- 172.21.15.46:0/478537337 >> v2:172.21.15.88:3300/0 conn(0x7f34f00b5210 0x7f34f01613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f62fb640 1 --2- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b7ed0 0x7f34f00d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.731+0000 7f34f4af8640 1 -- 172.21.15.46:0/478537337 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541391493 0 0) 0x7f34f00d5280 con 0x7f34f00d4800 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.732+0000 7f34f4af8640 1 -- 172.21.15.46:0/478537337 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34dc003520 con 0x7f34f00d4800 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.732+0000 7f34f5afa640 1 -- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b7ed0 msgr2=0x7f34f00d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.732+0000 7f34f5afa640 1 --2- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b7ed0 0x7f34f00d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.732+0000 7f34f5afa640 1 -- 172.21.15.46:0/478537337 >> v1:172.21.15.46:6789/0 conn(0x7f34f00d4800 legacy=0x7f34f00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.732+0000 7f34f5afa640 1 -- 172.21.15.46:0/478537337 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f00d5560 con 0x7f34f00b5210 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f5afa640 1 --2- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f00b5210 0x7f34f01613c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009d90 tx=0x7f34ec02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b3432f2b32a87ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f4af8640 1 -- 172.21.15.46:0/478537337 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec031600 con 0x7f34f00b5210 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f4af8640 1 -- 172.21.15.46:0/478537337 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ec0317a0 con 0x7f34f00b5210 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f4af8640 1 -- 172.21.15.46:0/478537337 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec031b50 con 0x7f34f00b5210 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f7d5e640 1 -- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f00b5210 msgr2=0x7f34f01613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f00b5210 0x7f34f01613c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f34ec009d90 tx=0x7f34ec02a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f7d5e640 1 -- 172.21.15.46:0/478537337 shutdown_connections 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b7ed0 0x7f34f00d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/478537337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f00b5210 0x7f34f01613c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.733+0000 7f34f7d5e640 1 -- 172.21.15.46:0/478537337 >> 172.21.15.46:0/478537337 conn(0x7f34f00d29a0 msgr2=0x7f34f00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.734+0000 7f34f7d5e640 1 -- 172.21.15.46:0/478537337 shutdown_connections 2024-05-09T03:18:23.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.734+0000 7f34f7d5e640 1 -- 172.21.15.46:0/478537337 wait complete. 2024-05-09T03:18:23.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.735+0000 7f34f7d5e640 1 Processor -- start 2024-05-09T03:18:23.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.735+0000 7f34f7d5e640 1 -- start start 2024-05-09T03:18:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f7d5e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b5210 0x7f34f00ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f7d5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 0x7f34f00b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f7d5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f01f6610 0x7f34f01f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f7d5e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34f00d0ae0 con 0x7f34f00d4800 2024-05-09T03:18:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f7d5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f34f0160ea0 con 0x7f34f01f6610 2024-05-09T03:18:23.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f7d5e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f34f0163e60 con 0x7f34f00b5210 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 0x7f34f00b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 0x7f34f00b71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50744/0 (socket says 172.21.15.46:50744) 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 -- 172.21.15.46:0/1993319200 learned_addr learned my addr 172.21.15.46:0/1993319200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f62fb640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f01f6610 0x7f34f01f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f5afa640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b5210 0x7f34f00ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 -- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b5210 msgr2=0x7f34f00ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b5210 0x7f34f00ac9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 -- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f01f6610 msgr2=0x7f34f01f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f01f6610 0x7f34f01f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 -- 172.21.15.46:0/1993319200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f00b7a30 con 0x7f34f00d4800 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f5afa640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b5210 0x7f34f00ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34f52f9640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 0x7f34f00b71d0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f34e000c440 tx=0x7f34e000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:23.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0017070 con 0x7f34f00d4800 2024-05-09T03:18:23.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.737+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e0005ce0 con 0x7f34f00d4800 2024-05-09T03:18:23.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.738+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f00b7d10 con 0x7f34f00d4800 2024-05-09T03:18:23.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.738+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e0007270 con 0x7f34f00d4800 2024-05-09T03:18:23.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.738+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f014f370 con 0x7f34f00d4800 2024-05-09T03:18:23.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.738+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f006a270 con 0x7f34f00d4800 2024-05-09T03:18:23.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.739+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34e001a070 con 0x7f34f00d4800 2024-05-09T03:18:23.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.740+0000 7f34e67fc640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34d40762d0 0x7f34d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:23.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.740+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7f34e0097230 con 0x7f34f00d4800 2024-05-09T03:18:23.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.740+0000 7f34f5afa640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34d40762d0 0x7f34d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:23.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.741+0000 7f34f5afa640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34d40762d0 0x7f34d40787e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f34ec034000 tx=0x7f34ec033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:23.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.744+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34e005ffc0 con 0x7f34f00d4800 2024-05-09T03:18:23.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.971+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34f00b77c0 con 0x7f34f00d4800 2024-05-09T03:18:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.971+0000 7f34e67fc640 1 -- 172.21.15.46:0/1993319200 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) ==== 95+0+4343 (secure 0 0 0) 0x7f34e00642c0 con 0x7f34f00d4800 2024-05-09T03:18:23.973 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:23.973 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":209,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:23.220659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:23.973 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 209 2024-05-09T03:18:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34d40762d0 msgr2=0x7f34d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34d40762d0 0x7f34d40787e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f34ec034000 tx=0x7f34ec033040 comp rx=0 tx=0).stop 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 msgr2=0x7f34f00b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 0x7f34f00b71d0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f34e000c440 tx=0x7f34e000c900 comp rx=0 tx=0).stop 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 shutdown_connections 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34f01f6610 0x7f34f01f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f34d40762d0 0x7f34d40787e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34f00d4800 0x7f34f00b71d0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 --2- 172.21.15.46:0/1993319200 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f34f00b5210 0x7f34f00ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 >> 172.21.15.46:0/1993319200 conn(0x7f34f00d29a0 msgr2=0x7f34f0164220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 shutdown_connections 2024-05-09T03:18:23.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:23.973+0000 7f34f7d5e640 1 -- 172.21.15.46:0/1993319200 wait complete. 2024-05-09T03:18:23.986 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:23.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:24.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.101+0000 7fbfcbb38640 1 Processor -- start 2024-05-09T03:18:24.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.101+0000 7fbfcbb38640 1 -- start start 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.102+0000 7fbfcbb38640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc4164c70 0x7fbfc41615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.102+0000 7fbfcbb38640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc4160b20 con 0x7fbfc415dfe0 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.102+0000 7fbfcbb38640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc40d15f0 con 0x7fbfc40d2480 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.102+0000 7fbfcbb38640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc4161150 con 0x7fbfc4164c70 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.102+0000 7fbfc90d3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbfc415dfe0 0x7fbfc40d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48604/0 (socket says 172.21.15.46:48604) 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.102+0000 7fbfc90d3640 1 -- 172.21.15.46:0/3359213735 learned_addr learned my addr 172.21.15.46:0/3359213735 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc98d4640 1 --2- 172.21.15.46:0/3359213735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc4164c70 0x7fbfc41615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319295005 0 0) 0x7fbfc4160b20 con 0x7fbfc415dfe0 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfb0003520 con 0x7fbfc415dfe0 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718375249 0 0) 0x7fbfc40d15f0 con 0x7fbfc40d2480 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfc4160b20 con 0x7fbfc40d2480 2024-05-09T03:18:24.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc98d4640 1 -- 172.21.15.46:0/3359213735 >> v1:172.21.15.88:6789/0 conn(0x7fbfc40d2480 legacy=0x7fbfc40d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc98d4640 1 -- 172.21.15.46:0/3359213735 >> v1:172.21.15.46:6789/0 conn(0x7fbfc415dfe0 legacy=0x7fbfc40d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.103+0000 7fbfc98d4640 1 -- 172.21.15.46:0/3359213735 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc40d18d0 con 0x7fbfc4164c70 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.104+0000 7fbfc98d4640 1 --2- 172.21.15.46:0/3359213735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc4164c70 0x7fbfc41615e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00097d0 tx=0x7fbfc002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad4a20ed067522b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.104+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0037070 con 0x7fbfc4164c70 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.104+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc0032440 con 0x7fbfc4164c70 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.104+0000 7fbfc88d2640 1 -- 172.21.15.46:0/3359213735 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc00327f0 con 0x7fbfc4164c70 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 -- 172.21.15.46:0/3359213735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc4164c70 msgr2=0x7fbfc41615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/3359213735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc4164c70 0x7fbfc41615e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00097d0 tx=0x7fbfc002a210 comp rx=0 tx=0).stop 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 -- 172.21.15.46:0/3359213735 shutdown_connections 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/3359213735 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc4164c70 0x7fbfc41615e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 -- 172.21.15.46:0/3359213735 >> 172.21.15.46:0/3359213735 conn(0x7fbfc4073db0 msgr2=0x7fbfc40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:24.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 -- 172.21.15.46:0/3359213735 shutdown_connections 2024-05-09T03:18:24.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.105+0000 7fbfcbb38640 1 -- 172.21.15.46:0/3359213735 wait complete. 2024-05-09T03:18:24.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.107+0000 7fbfcbb38640 1 Processor -- start 2024-05-09T03:18:24.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.107+0000 7fbfcbb38640 1 -- start start 2024-05-09T03:18:24.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfcbb38640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 0x7fbfc40ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfcbb38640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc415dfe0 0x7fbfc40cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfcbb38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfc4164c70 0x7fbfc41f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfcbb38640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc40d45c0 con 0x7fbfc40d2480 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfcbb38640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc415e3f0 con 0x7fbfc4164c70 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfcbb38640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc40d4740 con 0x7fbfc415dfe0 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 0x7fbfc40ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 0x7fbfc40ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50760/0 (socket says 172.21.15.46:50760) 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 -- 172.21.15.46:0/574277317 learned_addr learned my addr 172.21.15.46:0/574277317 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc90d3640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc415dfe0 0x7fbfc40cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfca0d5640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfc4164c70 0x7fbfc41f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 -- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc415dfe0 msgr2=0x7fbfc40cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc415dfe0 0x7fbfc40cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 -- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfc4164c70 msgr2=0x7fbfc41f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfc4164c70 0x7fbfc41f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc98d4640 1 -- 172.21.15.46:0/574277317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc41f7a70 con 0x7fbfc40d2480 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfca0d5640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfc4164c70 0x7fbfc41f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.108+0000 7fbfc90d3640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc415dfe0 0x7fbfc40cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.109+0000 7fbfc98d4640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 0x7fbfc40ce4b0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0035040 tx=0x7fbfc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.109+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0037050 con 0x7fbfc40d2480 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.109+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc41f7d50 con 0x7fbfc40d2480 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.109+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc0043070 con 0x7fbfc40d2480 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.109+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc003ca10 con 0x7fbfc40d2480 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.109+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc41f74f0 con 0x7fbfc40d2480 2024-05-09T03:18:24.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.110+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc408cfd0 con 0x7fbfc40d2480 2024-05-09T03:18:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.111+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbfc003cbb0 con 0x7fbfc40d2480 2024-05-09T03:18:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.111+0000 7fbfba7fc640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfa8076300 0x7fbfa8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.111+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7fbfc00b95f0 con 0x7fbfc40d2480 2024-05-09T03:18:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.111+0000 7fbfc90d3640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfa8076300 0x7fbfa8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.112+0000 7fbfc90d3640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfa8076300 0x7fbfa8078810 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4007a60 tx=0x7fbfb4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:24.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.115+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbfc408cfd0 con 0x7fbfc40d2480 2024-05-09T03:18:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.349+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfc406a270 con 0x7fbfc40d2480 2024-05-09T03:18:24.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.350+0000 7fbfba7fc640 1 -- 172.21.15.46:0/574277317 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+5169 (secure 0 0 0) 0x7fbfc0082870 con 0x7fbfc40d2480 2024-05-09T03:18:24.351 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:24.352 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":210,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":8934,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2546691383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2546691383},{"type":"v1","addr":"172.21.15.46:6834","nonce":2546691383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:23.220659+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:24.352 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 210 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfa8076300 msgr2=0x7fbfa8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfa8076300 0x7fbfa8078810 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4007a60 tx=0x7fbfb4009040 comp rx=0 tx=0).stop 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 msgr2=0x7fbfc40ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 0x7fbfc40ce4b0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0035040 tx=0x7fbfc00046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 shutdown_connections 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbfc4164c70 0x7fbfc41f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbfc415dfe0 0x7fbfc40cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbfa8076300 0x7fbfa8078810 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 --2- 172.21.15.46:0/574277317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc40d2480 0x7fbfc40ce4b0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 >> 172.21.15.46:0/574277317 conn(0x7fbfc4073db0 msgr2=0x7fbfc415d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:24.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 shutdown_connections 2024-05-09T03:18:24.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.352+0000 7fbfcbb38640 1 -- 172.21.15.46:0/574277317 wait complete. 2024-05-09T03:18:24.365 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt mds_snaptable' led to DAMAGED state 2024-05-09T03:18:24.365 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt mds0_inotable' 2024-05-09T03:18:24.365 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:24.365 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:24.382 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:24.382 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:24.382 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:24.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:24.382 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:24.438 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:24.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:24.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.592+0000 7efe7ddd5640 1 Processor -- start 2024-05-09T03:18:24.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.592+0000 7efe7ddd5640 1 -- start start 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.593+0000 7efe7ddd5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efe78143410 0x7efe780b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.593+0000 7efe7ddd5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe78160c60 0x7efe78161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe7ddd5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe781616b0 con 0x7efe780d1860 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe7ddd5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7efe780b58b0 con 0x7efe78160c60 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe7ddd5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efe780b4920 con 0x7efe78143410 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe78160c60 0x7efe78161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efe78160c60 0x7efe78161070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50770/0 (socket says 172.21.15.46:50770) 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 -- 172.21.15.46:0/39236451 learned_addr learned my addr 172.21.15.46:0/39236451 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe777fe640 1 --2- 172.21.15.46:0/39236451 >> v2:172.21.15.88:3300/0 conn(0x7efe78143410 0x7efe780b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 -- 172.21.15.46:0/39236451 >> v2:172.21.15.88:3300/0 conn(0x7efe78143410 msgr2=0x7efe780b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 --2- 172.21.15.46:0/39236451 >> v2:172.21.15.88:3300/0 conn(0x7efe78143410 0x7efe780b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 -- 172.21.15.46:0/39236451 >> v1:172.21.15.88:6789/0 conn(0x7efe780d1860 legacy=0x7efe780d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe76ffd640 1 -- 172.21.15.46:0/39236451 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe78161990 con 0x7efe78160c60 2024-05-09T03:18:24.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.594+0000 7efe777fe640 1 --2- 172.21.15.46:0/39236451 >> v2:172.21.15.88:3300/0 conn(0x7efe78143410 0x7efe780b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.595+0000 7efe76ffd640 1 --2- 172.21.15.46:0/39236451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78160c60 0x7efe78161070 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7efe780d05d0 tx=0x7efe6c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86550f1c5b54492 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.595+0000 7efe767fc640 1 -- 172.21.15.46:0/39236451 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c037070 con 0x7efe78160c60 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.595+0000 7efe767fc640 1 -- 172.21.15.46:0/39236451 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6c032440 con 0x7efe78160c60 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.595+0000 7efe767fc640 1 -- 172.21.15.46:0/39236451 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c032810 con 0x7efe78160c60 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 -- 172.21.15.46:0/39236451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78160c60 msgr2=0x7efe78161070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/39236451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78160c60 0x7efe78161070 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7efe780d05d0 tx=0x7efe6c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 -- 172.21.15.46:0/39236451 shutdown_connections 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/39236451 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78160c60 0x7efe78161070 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/39236451 >> v2:172.21.15.88:3300/0 conn(0x7efe78143410 0x7efe780b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 -- 172.21.15.46:0/39236451 >> 172.21.15.46:0/39236451 conn(0x7efe780b3620 msgr2=0x7efe780b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:24.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 -- 172.21.15.46:0/39236451 shutdown_connections 2024-05-09T03:18:24.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.596+0000 7efe7ddd5640 1 -- 172.21.15.46:0/39236451 wait complete. 2024-05-09T03:18:24.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.597+0000 7efe7ddd5640 1 Processor -- start 2024-05-09T03:18:24.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.598+0000 7efe7ddd5640 1 -- start start 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe7ddd5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe780d1860 0x7efe78158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe7ddd5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 0x7efe781f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe7ddd5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe78160c60 0x7efe78137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe7ddd5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efe78143820 con 0x7efe78143410 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe7ddd5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efe780b4ee0 con 0x7efe780d1860 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe7ddd5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efe780d1210 con 0x7efe78160c60 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 0x7efe781f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 0x7efe781f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50784/0 (socket says 172.21.15.46:50784) 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 -- 172.21.15.46:0/3004752119 learned_addr learned my addr 172.21.15.46:0/3004752119 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe777fe640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe780d1860 0x7efe78158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe77fff640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe78160c60 0x7efe78137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 -- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe78160c60 msgr2=0x7efe78137760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe78160c60 0x7efe78137760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 -- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe780d1860 msgr2=0x7efe78158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe780d1860 0x7efe78158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.599+0000 7efe76ffd640 1 -- 172.21.15.46:0/3004752119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe78159160 con 0x7efe78143410 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.600+0000 7efe76ffd640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 0x7efe781f6e50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7efe6c000c00 tx=0x7efe6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.600+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c041020 con 0x7efe78143410 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.600+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6c0306c0 con 0x7efe78143410 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.600+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe781382d0 con 0x7efe78143410 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.600+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c037040 con 0x7efe78143410 2024-05-09T03:18:24.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.600+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe781f6650 con 0x7efe78143410 2024-05-09T03:18:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.601+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe78138040 con 0x7efe78143410 2024-05-09T03:18:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.602+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efe6c044020 con 0x7efe78143410 2024-05-09T03:18:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.602+0000 7efe5ffff640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe640762d0 0x7efe640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.602+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7efe6c0b8490 con 0x7efe78143410 2024-05-09T03:18:24.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.602+0000 7efe777fe640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe640762d0 0x7efe640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:24.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.603+0000 7efe777fe640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe640762d0 0x7efe640787e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efe68006fd0 tx=0x7efe68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:24.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.607+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efe6c04b030 con 0x7efe78143410 2024-05-09T03:18:24.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:24.838+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efe7806a270 con 0x7efe78143410 2024-05-09T03:18:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.227+0000 7efe5ffff640 1 -- 172.21.15.46:0/3004752119 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v211) ==== 161+0+0 (secure 0 0 0) 0x7efe6c081020 con 0x7efe78143410 2024-05-09T03:18:25.229 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.230+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe640762d0 msgr2=0x7efe640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe640762d0 0x7efe640787e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efe68006fd0 tx=0x7efe68008040 comp rx=0 tx=0).stop 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 msgr2=0x7efe781f6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 0x7efe781f6e50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7efe6c000c00 tx=0x7efe6c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 shutdown_connections 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efe78160c60 0x7efe78137760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efe640762d0 0x7efe640787e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe78143410 0x7efe781f6e50 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 --2- 172.21.15.46:0/3004752119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe780d1860 0x7efe78158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 >> 172.21.15.46:0/3004752119 conn(0x7efe780b3620 msgr2=0x7efe780b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 shutdown_connections 2024-05-09T03:18:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.231+0000 7efe7ddd5640 1 -- 172.21.15.46:0/3004752119 wait complete. 2024-05-09T03:18:25.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.359+0000 7fa542f8d640 1 Processor -- start 2024-05-09T03:18:25.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.359+0000 7fa542f8d640 1 -- start start 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa542f8d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa53c0b5140 0x7fa53c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa542f8d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa542f8d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa53c0b80b0 con 0x7fa53c0d3df0 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa542f8d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa53c0d3480 con 0x7fa53c0b5140 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa542f8d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa53c0b5550 con 0x7fa53c0b7ca0 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa540d29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa53c0b5140 0x7fa53c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa54152a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa53c0d3df0 0x7fa53c163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48614/0 (socket says 172.21.15.46:48614) 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa54152a640 1 -- 172.21.15.46:0/3071063055 learned_addr learned my addr 172.21.15.46:0/3071063055 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:25.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa540d29640 1 --2- 172.21.15.46:0/3071063055 >> v2:172.21.15.46:3300/0 conn(0x7fa53c0b5140 0x7fa53c0b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50788/0 (socket says 172.21.15.46:50788) 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.360+0000 7fa53bfff640 1 --2- 172.21.15.46:0/3071063055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa540d29640 1 -- 172.21.15.46:0/3071063055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 msgr2=0x7fa53c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa540d29640 1 --2- 172.21.15.46:0/3071063055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa540d29640 1 -- 172.21.15.46:0/3071063055 >> v1:172.21.15.46:6789/0 conn(0x7fa53c0d3df0 legacy=0x7fa53c163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa53b7fe640 1 -- 172.21.15.46:0/3071063055 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1465027995 0 0) 0x7fa534001110 con 0x7fa53c0d3df0 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa540d29640 1 -- 172.21.15.46:0/3071063055 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c0d44e0 con 0x7fa53c0b5140 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa540d29640 1 --2- 172.21.15.46:0/3071063055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c0b7760 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0097a0 tx=0x7fa52c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887708715bbc9b2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa53b7fe640 1 -- 172.21.15.46:0/3071063055 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c037070 con 0x7fa53c0b5140 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.361+0000 7fa53b7fe640 1 -- 172.21.15.46:0/3071063055 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c032440 con 0x7fa53c0b5140 2024-05-09T03:18:25.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa53b7fe640 1 -- 172.21.15.46:0/3071063055 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c032810 con 0x7fa53c0b5140 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 -- 172.21.15.46:0/3071063055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 msgr2=0x7fa53c0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 --2- 172.21.15.46:0/3071063055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c0b7760 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0097a0 tx=0x7fa52c02a210 comp rx=0 tx=0).stop 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 -- 172.21.15.46:0/3071063055 shutdown_connections 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 --2- 172.21.15.46:0/3071063055 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 --2- 172.21.15.46:0/3071063055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c0b7760 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 -- 172.21.15.46:0/3071063055 >> 172.21.15.46:0/3071063055 conn(0x7fa53c08d3f0 msgr2=0x7fa53c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 -- 172.21.15.46:0/3071063055 shutdown_connections 2024-05-09T03:18:25.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.362+0000 7fa542f8d640 1 -- 172.21.15.46:0/3071063055 wait complete. 2024-05-09T03:18:25.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.364+0000 7fa542f8d640 1 Processor -- start 2024-05-09T03:18:25.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.364+0000 7fa542f8d640 1 -- start start 2024-05-09T03:18:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.365+0000 7fa542f8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:25.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.365+0000 7fa542f8d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.365+0000 7fa542f8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53c0d3df0 0x7fa53c157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.365+0000 7fa542f8d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c0d4d80 con 0x7fa53c0b5140 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.365+0000 7fa542f8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa53c0d4aa0 con 0x7fa53c0d3df0 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.365+0000 7fa542f8d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa53c0d4200 con 0x7fa53c0b7ca0 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50804/0 (socket says 172.21.15.46:50804) 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 -- 172.21.15.46:0/1761920604 learned_addr learned my addr 172.21.15.46:0/1761920604 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa53bfff640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa54152a640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53c0d3df0 0x7fa53c157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 -- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 msgr2=0x7fa53c157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 -- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53c0d3df0 msgr2=0x7fa53c157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53c0d3df0 0x7fa53c157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:25.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 -- 172.21.15.46:0/1761920604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c15be70 con 0x7fa53c0b5140 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa54152a640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53c0d3df0 0x7fa53c157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa540d29640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c15b290 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa52c000c00 tx=0x7fa52c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c037040 con 0x7fa53c0b5140 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c042070 con 0x7fa53c0b5140 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53c158410 con 0x7fa53c0b5140 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa52c030bb0 con 0x7fa53c0b5140 2024-05-09T03:18:25.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.366+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53c156eb0 con 0x7fa53c0b5140 2024-05-09T03:18:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.367+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa53c06a270 con 0x7fa53c0b5140 2024-05-09T03:18:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.368+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa52c047460 con 0x7fa53c0b5140 2024-05-09T03:18:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.369+0000 7fa5397fa640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa51c0762d0 0x7fa51c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.369+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7fa52c0b86a0 con 0x7fa53c0b5140 2024-05-09T03:18:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.369+0000 7fa53bfff640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa51c0762d0 0x7fa51c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:25.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.370+0000 7fa53bfff640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa51c0762d0 0x7fa51c0787e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa53c163670 tx=0x7fa524009830 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:25.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.374+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa52c032e20 con 0x7fa53c0b5140 2024-05-09T03:18:25.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:25.640+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa53c1604e0 con 0x7fa53c0b5140 2024-05-09T03:18:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.229+0000 7fa5397fa640 1 -- 172.21.15.46:0/1761920604 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v212) ==== 101+0+0 (secure 0 0 0) 0x7fa52c081910 con 0x7fa53c0b5140 2024-05-09T03:18:26.231 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa51c0762d0 msgr2=0x7fa51c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa51c0762d0 0x7fa51c0787e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa53c163670 tx=0x7fa524009830 comp rx=0 tx=0).stop 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 msgr2=0x7fa53c15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c15b290 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa52c000c00 tx=0x7fa52c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 shutdown_connections 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa53c0d3df0 0x7fa53c157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa53c0b7ca0 0x7fa53c157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa51c0762d0 0x7fa51c0787e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 --2- 172.21.15.46:0/1761920604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa53c0b5140 0x7fa53c15b290 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 >> 172.21.15.46:0/1761920604 conn(0x7fa53c08d3f0 msgr2=0x7fa53c0ba2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 shutdown_connections 2024-05-09T03:18:26.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.233+0000 7fa542f8d640 1 -- 172.21.15.46:0/1761920604 wait complete. 2024-05-09T03:18:26.249 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.308+0000 7f29f590ff40 1 Processor -- start 2024-05-09T03:18:26.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.309+0000 7f29f590ff40 1 -- start start 2024-05-09T03:18:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29f590ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e96657b0 0x5608e97050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29f590ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9700f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29f590ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e9664210 0x5608e9664620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29f590ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5608e967a4d0 con 0x5608e967c010 2024-05-09T03:18:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29f590ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5608e95dcf50 con 0x5608e9664210 2024-05-09T03:18:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29f590ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5608e967f8a0 con 0x5608e96657b0 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9700f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9700f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41044/0 (socket says 172.21.15.46:41044) 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 -- 172.21.15.46:0/3856565041 learned_addr learned my addr 172.21.15.46:0/3856565041 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e96657b0 0x5608e97050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29efbd8640 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e9664210 0x5608e9664620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 -- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e96657b0 msgr2=0x5608e97050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e96657b0 0x5608e97050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 -- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e9664210 msgr2=0x5608e9664620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e9664210 0x5608e9664620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 -- 172.21.15.46:0/3856565041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608e9664b60 con 0x5608e967c010 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.310+0000 7f29eebd6640 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9700f10 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f29e400b210 tx=0x7f29e400b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03a36023a796488 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29edbd4640 1 -- 172.21.15.46:0/3856565041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e4008070 con 0x5608e967c010 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29edbd4640 1 -- 172.21.15.46:0/3856565041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e400d500 con 0x5608e967c010 2024-05-09T03:18:26.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29edbd4640 1 -- 172.21.15.46:0/3856565041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e400d8d0 con 0x5608e967c010 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 -- 172.21.15.46:0/3856565041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 msgr2=0x5608e9700f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9700f10 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f29e400b210 tx=0x7f29e400b6d0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 -- 172.21.15.46:0/3856565041 shutdown_connections 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e9664210 0x5608e9664620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9700f10 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 --2- 172.21.15.46:0/3856565041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e96657b0 0x5608e97050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 -- 172.21.15.46:0/3856565041 >> 172.21.15.46:0/3856565041 conn(0x5608e9558ac0 msgr2=0x5608e9674f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 -- 172.21.15.46:0/3856565041 shutdown_connections 2024-05-09T03:18:26.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.311+0000 7f29f590ff40 1 -- 172.21.15.46:0/3856565041 wait complete. 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.312+0000 7f29f590ff40 1 Processor -- start 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.312+0000 7f29f590ff40 1 -- start start 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29f590ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e9664210 0x5608e96fc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29f590ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e96657b0 0x5608e9705b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29f590ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9706050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29f590ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5608e965f710 con 0x5608e967c010 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29f590ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5608e965fc50 con 0x5608e96657b0 2024-05-09T03:18:26.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29f590ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5608e965ff30 con 0x5608e9664210 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9706050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9706050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41050/0 (socket says 172.21.15.46:41050) 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 learned_addr learned my addr 172.21.15.46:0/223532779 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e9664210 0x5608e96fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e96657b0 0x5608e9705b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e9664210 msgr2=0x5608e96fc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5608e9664210 0x5608e96fc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e96657b0 msgr2=0x5608e9705b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5608e96657b0 0x5608e9705b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608e9634850 con 0x5608e967c010 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.314+0000 7f29efbd8640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9706050 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f29e8057040 tx=0x7f29e8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.315+0000 7f29ecbd2640 1 -- 172.21.15.46:0/223532779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8065070 con 0x5608e967c010 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.315+0000 7f29ecbd2640 1 -- 172.21.15.46:0/223532779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e8054b80 con 0x5608e967c010 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.315+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5608e96fcea0 con 0x5608e967c010 2024-05-09T03:18:26.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.315+0000 7f29ecbd2640 1 -- 172.21.15.46:0/223532779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8059040 con 0x5608e967c010 2024-05-09T03:18:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.315+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608e965efd0 con 0x5608e967c010 2024-05-09T03:18:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.316+0000 7f29ecbd2640 1 -- 172.21.15.46:0/223532779 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f29e8068070 con 0x5608e967c010 2024-05-09T03:18:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.317+0000 7f29ecbd2640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29dc074cb0 0x7f29dc0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.317+0000 7f29ecbd2640 1 -- 172.21.15.46:0/223532779 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7f29e80da340 con 0x5608e967c010 2024-05-09T03:18:26.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.317+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29dc074cb0 0x7f29dc0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.320 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:26.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.317+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5608e96fda60 0x5608e96fde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.317+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97af890 con 0x5608e96fda60 2024-05-09T03:18:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.317+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5608e96fda60 0x5608e96fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.318+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5608e96fda60 0x5608e96fde70 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.318+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29dc074cb0 0x7f29dc0771c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x5608e9676f00 tx=0x7f29e0005d80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.320+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v104'598 uv572 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.320+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97afc30 con 0x5608e96fda60 2024-05-09T03:18:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.321+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v104'599 uv599 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.321+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97affd0 con 0x5608e96fda60 2024-05-09T03:18:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.323+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v104'600 uv600 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.324 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:26.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.323+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b1380 con 0x5608e96fda60 2024-05-09T03:18:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.324+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v104'601 uv601 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.324+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b1720 con 0x5608e96fda60 2024-05-09T03:18:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.325+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v104'602 uv602 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.325+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0370 con 0x5608e96fda60 2024-05-09T03:18:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.326+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v104'603 uv603 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.326+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0780 con 0x5608e96fda60 2024-05-09T03:18:26.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.328+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v104'604 uv578 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.328+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.329+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v104'605 uv605 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.329+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.332 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.330+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v104'606 uv606 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.330+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.332+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v104'607 uv607 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.332+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.333+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v104'608 uv608 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.333+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.335+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v104'609 uv609 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.335+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.336+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v104'610 uv585 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.336+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.337+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v104'611 uv611 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.337+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.339+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v104'612 uv612 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.341 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:26.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.339+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.340+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v104'613 uv613 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.340+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.342+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v104'614 uv614 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.342+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.343+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v104'615 uv615 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.343+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.344+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v104'616 uv616 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.344+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.346+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v104'617 uv591 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.346+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.347+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v104'618 uv618 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.347+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.350 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:26.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.349+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v104'619 uv619 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.349+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.350+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v104'620 uv620 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.350+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.351+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v104'621 uv621 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.351+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.353+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v104'622 uv622 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.353+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.354+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v104'623 uv597 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.354+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.355+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v104'624 uv624 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.355+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.358 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.357+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v104'625 uv625 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.357+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.358+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v104'626 uv626 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.358+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.359+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v104'627 uv627 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.359+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e96fda60 2024-05-09T03:18:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.360+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v104'628 uv628 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.360+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5608e97b0f10 0x5608e97b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.360+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b0b20 con 0x5608e97b0f10 2024-05-09T03:18:26.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.361+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5608e97b0f10 0x5608e97b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.361+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5608e97b0f10 0x5608e97b3fe0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.363+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v104'438 uv421 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.363+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b4520 con 0x5608e97b0f10 2024-05-09T03:18:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.364+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v104'439 uv439 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.364+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b48c0 con 0x5608e97b0f10 2024-05-09T03:18:26.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.365+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v104'440 uv440 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.367 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:26.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.365+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b4c60 con 0x5608e97b0f10 2024-05-09T03:18:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.366+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v104'441 uv441 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.366+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5000 con 0x5608e97b0f10 2024-05-09T03:18:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.367+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v104'442 uv442 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.367+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b53a0 con 0x5608e97b0f10 2024-05-09T03:18:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.369+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v104'443 uv426 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.369+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b5740 con 0x5608e97b0f10 2024-05-09T03:18:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.370+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v104'444 uv444 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.370+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.373 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.371+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v104'445 uv445 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.371+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.372+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v104'446 uv446 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.372+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.374+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v104'447 uv447 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.374+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.375+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v104'448 uv431 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.375+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.376+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v104'449 uv449 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.376+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.379 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:26.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.377+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v104'450 uv450 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.377+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b9540 con 0x5608e97b0f10 2024-05-09T03:18:26.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.379+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v104'451 uv451 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.379+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b9540 con 0x5608e97b0f10 2024-05-09T03:18:26.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.380+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v104'452 uv452 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.380+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.381+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v104'453 uv437 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.381+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.383+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v104'454 uv454 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.383+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.385 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:26.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.384+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v104'455 uv455 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.384+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b6af0 con 0x5608e97b0f10 2024-05-09T03:18:26.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.385+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v104'456 uv456 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.385+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b6af0 con 0x5608e97b0f10 2024-05-09T03:18:26.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.386+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v104'457 uv457 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.386+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e97b0f10 2024-05-09T03:18:26.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.387+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v104'458 uv458 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97b0f10 2024-05-09T03:18:26.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.387+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e96fda60 2024-05-09T03:18:26.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.388+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v104'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.388+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e96fda60 2024-05-09T03:18:26.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.389+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v104'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.389+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e96fda60 2024-05-09T03:18:26.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.390+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v104'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.392 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:26.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.390+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b6af0 con 0x5608e96fda60 2024-05-09T03:18:26.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.392+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v104'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.392+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b6af0 con 0x5608e96fda60 2024-05-09T03:18:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.393+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v104'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.393+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b5ae0 con 0x5608e96fda60 2024-05-09T03:18:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.394+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v104'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.394+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5608e97b6160 0x5608e97b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.394+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8b90 con 0x5608e97b6160 2024-05-09T03:18:26.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.394+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5608e97b6160 0x5608e97b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.394+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5608e97b6160 0x5608e97b8580 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.395+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v104'90 uv89 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.395+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b8f80 con 0x5608e97b6160 2024-05-09T03:18:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.396+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v104'91 uv91 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.396+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8f80 con 0x5608e97b6160 2024-05-09T03:18:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.397+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v104'92 uv92 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.399 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.397+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8f80 con 0x5608e97b6160 2024-05-09T03:18:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.398+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v104'93 uv93 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.398+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8ea0 con 0x5608e96fda60 2024-05-09T03:18:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.400+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v104'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.400+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97b8ea0 con 0x5608e96fda60 2024-05-09T03:18:26.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.401+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v104'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.401+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8e20 con 0x5608e96fda60 2024-05-09T03:18:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.403+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v104'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.405 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.403+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8e20 con 0x5608e96fda60 2024-05-09T03:18:26.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.406+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v104'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.406+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8e20 con 0x5608e96fda60 2024-05-09T03:18:26.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.408+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v104'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.408+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97b8e20 con 0x5608e96fda60 2024-05-09T03:18:26.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.409+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v104'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.410+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5608e97b9740 0x5608e97b9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.410+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bc320 con 0x5608e97b9740 2024-05-09T03:18:26.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.410+0000 7f29efbd8640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5608e97b9740 0x5608e97b9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.410+0000 7f29efbd8640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5608e97b9740 0x5608e97b9b50 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.412+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v104'216 uv210 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e8071070 con 0x5608e97b9740 2024-05-09T03:18:26.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.412+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97bc6c0 con 0x5608e97b9740 2024-05-09T03:18:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.413+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v104'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e8071070 con 0x5608e97b9740 2024-05-09T03:18:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.413+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bca60 con 0x5608e97b9740 2024-05-09T03:18:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.414+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v104'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e8071070 con 0x5608e97b9740 2024-05-09T03:18:26.416 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:26.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.414+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e96fd1a0 con 0x5608e97b9740 2024-05-09T03:18:26.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.415+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v104'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.415+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97be3f0 con 0x5608e97b9740 2024-05-09T03:18:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.416+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v104'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.416+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bd340 con 0x5608e97b9740 2024-05-09T03:18:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.417+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v104'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.417+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bd750 con 0x5608e97b9740 2024-05-09T03:18:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.419+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v104'222 uv215 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.419+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97bdb40 con 0x5608e97b9740 2024-05-09T03:18:26.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.420+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v104'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.420+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bdf30 con 0x5608e97b9740 2024-05-09T03:18:26.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.421+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v104'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.423 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:26.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.421+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bdf30 con 0x5608e97b9740 2024-05-09T03:18:26.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.422+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v104'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.423+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97bdee0 con 0x5608e97b9740 2024-05-09T03:18:26.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.424+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v104'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.424+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5608e97becf0 0x5608e97c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.424+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1650 con 0x5608e97becf0 2024-05-09T03:18:26.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.424+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5608e97becf0 0x5608e97c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.424+0000 7f29ef3d7640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5608e97becf0 0x5608e97c1110 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.426+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v104'97 uv96 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97becf0 2024-05-09T03:18:26.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.426+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c16d0 con 0x5608e97becf0 2024-05-09T03:18:26.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.427+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v104'98 uv98 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97becf0 2024-05-09T03:18:26.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.427+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e97becf0 2024-05-09T03:18:26.429 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.428+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v104'99 uv99 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97becf0 2024-05-09T03:18:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.428+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e97becf0 2024-05-09T03:18:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.429+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v104'100 uv100 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97becf0 2024-05-09T03:18:26.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.429+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e97becf0 2024-05-09T03:18:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.430+0000 7f29ef3d7640 1 -- 172.21.15.46:0/223532779 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v104'101 uv101 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e0021420 con 0x5608e97becf0 2024-05-09T03:18:26.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.430+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e96fda60 2024-05-09T03:18:26.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.431+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v104'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.431+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e96fda60 2024-05-09T03:18:26.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.432+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v104'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.433+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e96fda60 2024-05-09T03:18:26.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.434+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v104'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.435 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.434+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c2b00 con 0x5608e96fda60 2024-05-09T03:18:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.435+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v104'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.435+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c2b00 con 0x5608e96fda60 2024-05-09T03:18:26.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.436+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v104'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.436+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e96fda60 2024-05-09T03:18:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.437+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v104'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e96fda60 2024-05-09T03:18:26.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.437+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e97b6160 2024-05-09T03:18:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.438+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v104'192 uv186 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.438+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e97b6160 2024-05-09T03:18:26.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.439+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v104'193 uv193 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.439+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1af0 con 0x5608e97b6160 2024-05-09T03:18:26.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.440+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v104'194 uv194 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.442 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:26.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.440+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1b90 con 0x5608e97b6160 2024-05-09T03:18:26.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.441+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v104'195 uv195 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.441+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1b90 con 0x5608e97b6160 2024-05-09T03:18:26.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.443+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v104'196 uv196 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.443+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1b90 con 0x5608e97b6160 2024-05-09T03:18:26.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.443+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v104'197 uv191 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.443+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c1b90 con 0x5608e97b6160 2024-05-09T03:18:26.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.444+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v104'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.444+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1b90 con 0x5608e97b6160 2024-05-09T03:18:26.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.446+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v104'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.447 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:26.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.446+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c1b90 con 0x5608e97b6160 2024-05-09T03:18:26.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.447+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v104'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.447+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c49a0 con 0x5608e97b6160 2024-05-09T03:18:26.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.448+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v104'201 uv201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e400d730 con 0x5608e97b6160 2024-05-09T03:18:26.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.448+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5608e97c26c0 0x5608e97c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.448+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97c26c0 2024-05-09T03:18:26.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.448+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5608e97c26c0 0x5608e97c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.448+0000 7f29eebd6640 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5608e97c26c0 0x5608e97c5020 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.449+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v104'87 uv86 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97c26c0 2024-05-09T03:18:26.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.449+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97c26c0 2024-05-09T03:18:26.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.451+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v104'88 uv88 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97c26c0 2024-05-09T03:18:26.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.451+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97c26c0 2024-05-09T03:18:26.453 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:26.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.452+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v104'89 uv89 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97c26c0 2024-05-09T03:18:26.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.452+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97c26c0 2024-05-09T03:18:26.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.453+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v104'90 uv90 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97c26c0 2024-05-09T03:18:26.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.453+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b6160 2024-05-09T03:18:26.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.454+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v104'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97b6160 2024-05-09T03:18:26.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.454+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b6160 2024-05-09T03:18:26.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.455+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v104'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97b6160 2024-05-09T03:18:26.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.455+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b6160 2024-05-09T03:18:26.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.456+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v104'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97b6160 2024-05-09T03:18:26.458 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:26.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.456+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4b20 con 0x5608e97b6160 2024-05-09T03:18:26.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.457+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v104'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97b6160 2024-05-09T03:18:26.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.458+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b6160 2024-05-09T03:18:26.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.459+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v104'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97b6160 2024-05-09T03:18:26.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.459+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b6160 2024-05-09T03:18:26.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.459+0000 7f29eebd6640 1 -- 172.21.15.46:0/223532779 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v104'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e402a040 con 0x5608e97b6160 2024-05-09T03:18:26.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.460+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b9740 2024-05-09T03:18:26.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.461+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v104'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.461+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b9740 2024-05-09T03:18:26.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.462+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v104'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.462+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c4ae0 con 0x5608e97b9740 2024-05-09T03:18:26.465 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:26.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.463+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v104'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.464+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.465+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v104'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.465+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.466+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v104'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.466+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.467+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v104'241 uv233 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.467+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.469+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v104'242 uv242 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.469+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.470+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v104'243 uv243 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.471 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:26.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.470+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.471+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v104'244 uv244 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.471+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.472+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v104'245 uv245 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.472+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.473+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v104'246 uv240 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.473+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.474+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v104'247 uv247 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.474+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.475+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v104'248 uv248 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.477 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.475+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c6570 con 0x5608e97b9740 2024-05-09T03:18:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.476+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v104'249 uv249 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.476+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c6570 con 0x5608e97b9740 2024-05-09T03:18:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.477+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v104'250 uv250 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.477+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.478+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v104'251 uv251 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.478+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x5608e97c5560 con 0x5608e97b9740 2024-05-09T03:18:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v104'252 uv252 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f29e80b9040 con 0x5608e97b9740 2024-05-09T03:18:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5608e96fda60 msgr2=0x5608e96fde70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5608e96fda60 0x5608e96fde70 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5608e97b6160 msgr2=0x5608e97b8580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5608e97b6160 0x5608e97b8580 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5608e97c26c0 msgr2=0x5608e97c5020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5608e97c26c0 0x5608e97c5020 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5608e97b0f10 msgr2=0x5608e97b3fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 reap_dead start 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5608e97b0f10 0x5608e97b3fe0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5608e97b9740 msgr2=0x5608e97b9b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5608e97b9740 0x5608e97b9b50 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5608e97becf0 msgr2=0x5608e97c1110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5608e97becf0 0x5608e97c1110 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29dc074cb0 msgr2=0x7f29dc0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29dc074cb0 0x7f29dc0771c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x5608e9676f00 tx=0x7f29e0005d80 comp rx=0 tx=0).stop 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 msgr2=0x5608e9706050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 --2- 172.21.15.46:0/223532779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608e967c010 0x5608e9706050 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f29e8057040 tx=0x7f29e8002fb0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29efbd8640 1 -- 172.21.15.46:0/223532779 reap_dead start 2024-05-09T03:18:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 shutdown_connections 2024-05-09T03:18:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 >> 172.21.15.46:0/223532779 conn(0x5608e9558ac0 msgr2=0x5608e9700350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 shutdown_connections 2024-05-09T03:18:26.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.480+0000 7f29f590ff40 1 -- 172.21.15.46:0/223532779 wait complete. 2024-05-09T03:18:26.485 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-05-09T03:18:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.580+0000 7ff1bad1bf40 1 Processor -- start 2024-05-09T03:18:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.580+0000 7ff1bad1bf40 1 -- start start 2024-05-09T03:18:26.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.581+0000 7ff1bad1bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adf63420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.581+0000 7ff1bad1bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff3a40 0x55f1adff61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.581+0000 7ff1bad1bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 0x55f1adf4a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.581+0000 7ff1bad1bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f1adf698d0 con 0x55f1adff4390 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.581+0000 7ff1bad1bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f1adf695f0 con 0x55f1adf685f0 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.581+0000 7ff1bad1bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f1adf64bb0 con 0x55f1adff3a40 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 0x55f1adf4a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 0x55f1adf4a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41062/0 (socket says 172.21.15.46:41062) 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 -- 172.21.15.46:0/539798680 learned_addr learned my addr 172.21.15.46:0/539798680 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b47e3640 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adf63420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b3fe2640 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff3a40 0x55f1adff61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 -- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff3a40 msgr2=0x55f1adff61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff3a40 0x55f1adff61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 -- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 msgr2=0x55f1adf63420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adf63420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 -- 172.21.15.46:0/539798680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ade88350 con 0x55f1adff4390 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b4fe4640 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 0x55f1adf4a640 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00a560 tx=0x7ff1ac00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5732a1721d4160fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b2fe0640 1 -- 172.21.15.46:0/539798680 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac019070 con 0x55f1adff4390 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b2fe0640 1 -- 172.21.15.46:0/539798680 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac007600 con 0x55f1adff4390 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.582+0000 7ff1b2fe0640 1 -- 172.21.15.46:0/539798680 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac0079f0 con 0x55f1adff4390 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/539798680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 msgr2=0x55f1adf4a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 0x55f1adf4a640 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00a560 tx=0x7ff1ac00a650 comp rx=0 tx=0).stop 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/539798680 shutdown_connections 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff4390 0x55f1adf4a640 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff3a40 0x55f1adff61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/539798680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adf63420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/539798680 >> 172.21.15.46:0/539798680 conn(0x55f1ade42ac0 msgr2=0x55f1adf4b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:26.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/539798680 shutdown_connections 2024-05-09T03:18:26.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.583+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/539798680 wait complete. 2024-05-09T03:18:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.584+0000 7ff1bad1bf40 1 Processor -- start 2024-05-09T03:18:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.584+0000 7ff1bad1bf40 1 -- start start 2024-05-09T03:18:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1bad1bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adfe9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1bad1bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff3a40 0x55f1adff5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1bad1bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff4390 0x55f1adff6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1bad1bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f1adf68df0 con 0x55f1adff3a40 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1bad1bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f1adf62350 con 0x55f1adf685f0 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1bad1bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f1adf22080 con 0x55f1adff4390 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff3a40 0x55f1adff5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff3a40 0x55f1adff5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41074/0 (socket says 172.21.15.46:41074) 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 -- 172.21.15.46:0/2122189955 learned_addr learned my addr 172.21.15.46:0/2122189955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b47e3640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adfe9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b4fe4640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff4390 0x55f1adff6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 -- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff4390 msgr2=0x55f1adff6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f1adff4390 0x55f1adff6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 -- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 msgr2=0x55f1adfe9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1adf685f0 0x55f1adfe9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.585+0000 7ff1b3fe2640 1 -- 172.21.15.46:0/2122189955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1adf222d0 con 0x55f1adff3a40 2024-05-09T03:18:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.586+0000 7ff1b3fe2640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff3a40 0x55f1adff5d10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009ec0 tx=0x7ff19c008400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.586+0000 7ff1b1fde640 1 -- 172.21.15.46:0/2122189955 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c015460 con 0x55f1adff3a40 2024-05-09T03:18:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.586+0000 7ff1b1fde640 1 -- 172.21.15.46:0/2122189955 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19c01a070 con 0x55f1adff3a40 2024-05-09T03:18:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.586+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f1adfea790 con 0x55f1adff3a40 2024-05-09T03:18:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.586+0000 7ff1b1fde640 1 -- 172.21.15.46:0/2122189955 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c015800 con 0x55f1adff3a40 2024-05-09T03:18:26.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.586+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1adfead50 con 0x55f1adff3a40 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.587+0000 7ff1b1fde640 1 -- 172.21.15.46:0/2122189955 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff19c013030 con 0x55f1adff3a40 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.588+0000 7ff1b1fde640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff194074cb0 0x7ff1940771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.588+0000 7ff1b1fde640 1 -- 172.21.15.46:0/2122189955 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7ff19c0967c0 con 0x55f1adff3a40 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.588+0000 7ff1b47e3640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff194074cb0 0x7ff1940771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.588+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f1adfedb60 0x55f1adfedf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.588+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x55f1adfee4b0 con 0x55f1adfedb60 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.588+0000 7ff1b4fe4640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f1adfedb60 0x55f1adfedf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.589+0000 7ff1b4fe4640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f1adfedb60 0x55f1adfedf70 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.589+0000 7ff1b47e3640 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff194074cb0 0x7ff1940771c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4056000 tx=0x7ff1a405e040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1b4fe4640 1 -- 172.21.15.46:0/2122189955 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~80] v104'202 uv202 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1ac01c040 con 0x55f1adfedb60 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f1adfedb60 msgr2=0x55f1adfedf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f1adfedb60 0x55f1adfedf70 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff194074cb0 msgr2=0x7ff1940771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff194074cb0 0x7ff1940771c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4056000 tx=0x7ff1a405e040 comp rx=0 tx=0).stop 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff3a40 msgr2=0x55f1adff5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 --2- 172.21.15.46:0/2122189955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f1adff3a40 0x55f1adff5d10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7ff19c009ec0 tx=0x7ff19c008400 comp rx=0 tx=0).stop 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1b4fe4640 1 -- 172.21.15.46:0/2122189955 reap_dead start 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 shutdown_connections 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 >> 172.21.15.46:0/2122189955 conn(0x55f1ade42ac0 msgr2=0x55f1adf4b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 shutdown_connections 2024-05-09T03:18:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.590+0000 7ff1bad1bf40 1 -- 172.21.15.46:0/2122189955 wait complete. 2024-05-09T03:18:26.595 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.752+0000 7f6ce382b640 1 Processor -- start 2024-05-09T03:18:26.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.752+0000 7f6ce382b640 1 -- start start 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.753+0000 7f6ce382b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cdc0d2e90 0x7f6cdc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.753+0000 7f6ce382b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6cdc1605b0 0x7f6cdc0b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.753+0000 7f6ce382b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cdc0d3400 con 0x7f6cdc0b4fc0 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.753+0000 7f6ce382b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc0d4a60 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.753+0000 7f6ce382b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc0d4d40 con 0x7f6cdc1605b0 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cdc0d2e90 0x7f6cdc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cdc0d2e90 0x7f6cdc0b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41088/0 (socket says 172.21.15.46:41088) 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 -- 172.21.15.46:0/492702866 learned_addr learned my addr 172.21.15.46:0/492702866 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce0dc6640 1 --2- 172.21.15.46:0/492702866 >> v2:172.21.15.88:3300/0 conn(0x7f6cdc1605b0 0x7f6cdc0b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 -- 172.21.15.46:0/492702866 >> v2:172.21.15.88:3300/0 conn(0x7f6cdc1605b0 msgr2=0x7f6cdc0b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 --2- 172.21.15.46:0/492702866 >> v2:172.21.15.88:3300/0 conn(0x7f6cdc1605b0 0x7f6cdc0b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 -- 172.21.15.46:0/492702866 >> v1:172.21.15.88:6789/0 conn(0x7f6cdc0b4fc0 legacy=0x7f6cdc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.754+0000 7f6ce15c7640 1 -- 172.21.15.46:0/492702866 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc0d5020 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.755+0000 7f6ce15c7640 1 --2- 172.21.15.46:0/492702866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc0b7760 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc08e880 tx=0x7f6cd802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28555c6d5736b6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.755+0000 7f6ccffff640 1 -- 172.21.15.46:0/492702866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8037070 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.755+0000 7f6ccffff640 1 -- 172.21.15.46:0/492702866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd8032440 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.755+0000 7f6ccffff640 1 -- 172.21.15.46:0/492702866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8032810 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 -- 172.21.15.46:0/492702866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 msgr2=0x7f6cdc0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 --2- 172.21.15.46:0/492702866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc0b7760 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc08e880 tx=0x7f6cd802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 -- 172.21.15.46:0/492702866 shutdown_connections 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 --2- 172.21.15.46:0/492702866 >> v2:172.21.15.88:3300/0 conn(0x7f6cdc1605b0 0x7f6cdc0b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 --2- 172.21.15.46:0/492702866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc0b7760 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 -- 172.21.15.46:0/492702866 >> 172.21.15.46:0/492702866 conn(0x7f6cdc08d390 msgr2=0x7f6cdc08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 -- 172.21.15.46:0/492702866 shutdown_connections 2024-05-09T03:18:26.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.756+0000 7f6ce382b640 1 -- 172.21.15.46:0/492702866 wait complete. 2024-05-09T03:18:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.758+0000 7f6ce382b640 1 Processor -- start 2024-05-09T03:18:26.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.758+0000 7f6ce382b640 1 -- start start 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce382b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cdc0b4fc0 0x7f6cdc1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce382b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc1609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce382b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cdc1605b0 0x7f6cdc160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce382b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc0d5540 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce382b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc0b4c00 con 0x7f6cdc0b4fc0 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce382b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc0cee50 con 0x7f6cdc1605b0 2024-05-09T03:18:26.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc1609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc1609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41094/0 (socket says 172.21.15.46:41094) 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 -- 172.21.15.46:0/642388904 learned_addr learned my addr 172.21.15.46:0/642388904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce15c7640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cdc0b4fc0 0x7f6cdc1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce1dc8640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cdc1605b0 0x7f6cdc160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 -- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cdc1605b0 msgr2=0x7f6cdc160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cdc1605b0 0x7f6cdc160f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 -- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cdc0b4fc0 msgr2=0x7f6cdc1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cdc0b4fc0 0x7f6cdc1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce0dc6640 1 -- 172.21.15.46:0/642388904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc1586f0 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.759+0000 7f6ce1dc8640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cdc1605b0 0x7f6cdc160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.760+0000 7f6ce0dc6640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc1609c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000e8a0 tx=0x7f6cd000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.760+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0017070 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.760+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cd000c940 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.760+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc158990 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.760+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd000cd30 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.760+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc158f50 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.761+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc0cebc0 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.762+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6cd001a070 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.762+0000 7f6ccdffb640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cc0076300 0x7f6cc0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.762+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7487+0+0 (secure 0 0 0) 0x7f6cd0097aa0 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.762+0000 7f6ce15c7640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cc0076300 0x7f6cc0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:26.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.763+0000 7f6ce15c7640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cc0076300 0x7f6cc0078810 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8034000 tx=0x7f6cd802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:26.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.767+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6cd00606c0 con 0x7f6cdc0d2e90 2024-05-09T03:18:26.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:26.997+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f6cdc06a270 con 0x7f6cdc0d2e90 2024-05-09T03:18:27.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.242+0000 7f6ccdffb640 1 -- 172.21.15.46:0/642388904 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v213) ==== 162+0+0 (secure 0 0 0) 0x7f6cd00649c0 con 0x7f6cdc0d2e90 2024-05-09T03:18:27.244 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cc0076300 msgr2=0x7f6cc0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cc0076300 0x7f6cc0078810 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8034000 tx=0x7f6cd802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 msgr2=0x7f6cdc1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc1609c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000e8a0 tx=0x7f6cd000ed60 comp rx=0 tx=0).stop 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 shutdown_connections 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cdc1605b0 0x7f6cdc160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6cc0076300 0x7f6cc0078810 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cdc0d2e90 0x7f6cdc1609c0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 --2- 172.21.15.46:0/642388904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cdc0b4fc0 0x7f6cdc1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 >> 172.21.15.46:0/642388904 conn(0x7f6cdc08d390 msgr2=0x7f6cdc0b69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 shutdown_connections 2024-05-09T03:18:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.246+0000 7f6ce382b640 1 -- 172.21.15.46:0/642388904 wait complete. 2024-05-09T03:18:27.253 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:27.252+0000 7f28fccfb640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': virtual void InoTable::decode_state(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 2 < 101: Malformed input [buffer:3] 2024-05-09T03:18:27.254 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:27.253+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:27.254 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:27.253+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:27.260 INFO:tasks.ceph.mds.c.smithi046.stderr: -25> 2024-05-09T03:18:27.252+0000 7f28fccfb640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': virtual void InoTable::decode_state(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 2 < 101: Malformed input [buffer:3] 2024-05-09T03:18:27.262 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:27.318 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.382+0000 7f98cf19d640 1 Processor -- start 2024-05-09T03:18:27.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.382+0000 7f98cf19d640 1 -- start start 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cf19d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c8161590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cf19d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98c80b2c30 0x7f98c80cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cf19d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98c80cea00 con 0x7f98c80b3040 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cf19d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f98c80cece0 con 0x7f98c80b2c30 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cf19d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f98c80cefc0 con 0x7f98c80ce4f0 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cd99a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98c80b2c30 0x7f98c80cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cd99a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98c80b2c30 0x7f98c80cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41134/0 (socket says 172.21.15.46:41134) 2024-05-09T03:18:27.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98cd99a640 1 -- 172.21.15.46:0/4193844522 learned_addr learned my addr 172.21.15.46:0/4193844522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.383+0000 7f98ce19b640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c8161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.384+0000 7f98cd99a640 1 -- 172.21.15.46:0/4193844522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 msgr2=0x7f98c8161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.384+0000 7f98cd99a640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c8161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.384+0000 7f98cd99a640 1 -- 172.21.15.46:0/4193844522 >> v1:172.21.15.88:6789/0 conn(0x7f98c80b3040 legacy=0x7f98c80b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.384+0000 7f98cd99a640 1 -- 172.21.15.46:0/4193844522 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c80cf2a0 con 0x7f98c80b2c30 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.384+0000 7f98ce19b640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c8161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.384+0000 7f98cd99a640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c80cfa80 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009870 tx=0x7f98b802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31e8afa67f520d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cd199640 1 -- 172.21.15.46:0/4193844522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8037070 con 0x7f98c80b2c30 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cd199640 1 -- 172.21.15.46:0/4193844522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b8032440 con 0x7f98c80b2c30 2024-05-09T03:18:27.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cd199640 1 -- 172.21.15.46:0/4193844522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b8032810 con 0x7f98c80b2c30 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 -- 172.21.15.46:0/4193844522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 msgr2=0x7f98c80cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c80cfa80 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009870 tx=0x7f98b802a130 comp rx=0 tx=0).stop 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 -- 172.21.15.46:0/4193844522 shutdown_connections 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c80cfa80 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 --2- 172.21.15.46:0/4193844522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c8161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 -- 172.21.15.46:0/4193844522 >> 172.21.15.46:0/4193844522 conn(0x7f98c80b4400 msgr2=0x7f98c80b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 -- 172.21.15.46:0/4193844522 shutdown_connections 2024-05-09T03:18:27.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.385+0000 7f98cf19d640 1 -- 172.21.15.46:0/4193844522 wait complete. 2024-05-09T03:18:27.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.387+0000 7f98cf19d640 1 Processor -- start 2024-05-09T03:18:27.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.387+0000 7f98cf19d640 1 -- start start 2024-05-09T03:18:27.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c81e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98c80b3040 0x7f98c81ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c80b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f98c80d0140 con 0x7f98c80b2c30 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f98c80cffc0 con 0x7f98c80b3040 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f98c80b3e30 con 0x7f98c80ce4f0 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c81e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c81e59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41140/0 (socket says 172.21.15.46:41140) 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 -- 172.21.15.46:0/460466131 learned_addr learned my addr 172.21.15.46:0/460466131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cd99a640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98c80b3040 0x7f98c81ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce99c640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c80b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 -- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 msgr2=0x7f98c80b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c80b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 -- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98c80b3040 msgr2=0x7f98c81ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98c80b3040 0x7f98c81ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 -- 172.21.15.46:0/460466131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c80cb790 con 0x7f98c80b2c30 2024-05-09T03:18:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98ce19b640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c81e59f0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f98c400d7d0 tx=0x7f98c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c40096a0 con 0x7f98c80b2c30 2024-05-09T03:18:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98c4019070 con 0x7f98c80b2c30 2024-05-09T03:18:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c80ca9d0 con 0x7f98c80b2c30 2024-05-09T03:18:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.389+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4009a80 con 0x7f98c80b2c30 2024-05-09T03:18:27.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.390+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c80caf90 con 0x7f98c80b2c30 2024-05-09T03:18:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.391+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c8076c70 con 0x7f98c80b2c30 2024-05-09T03:18:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.391+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f98c4009c20 con 0x7f98c80b2c30 2024-05-09T03:18:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.392+0000 7f98beffd640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98ac0762d0 0x7f98ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.392+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7f98c4098020 con 0x7f98c80b2c30 2024-05-09T03:18:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.392+0000 7f98cd99a640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98ac0762d0 0x7f98ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.393+0000 7f98cd99a640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98ac0762d0 0x7f98ac0787e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f98b8034000 tx=0x7f98b802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:27.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.396+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f98c4060be0 con 0x7f98c80b2c30 2024-05-09T03:18:27.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.627+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98c8071810 con 0x7f98c80b2c30 2024-05-09T03:18:27.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.628+0000 7f98beffd640 1 -- 172.21.15.46:0/460466131 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) ==== 95+0+4343 (secure 0 0 0) 0x7f98c4064ee0 con 0x7f98c80b2c30 2024-05-09T03:18:27.629 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:27.630 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":215,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:27.254785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 215 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98ac0762d0 msgr2=0x7f98ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98ac0762d0 0x7f98ac0787e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f98b8034000 tx=0x7f98b802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 msgr2=0x7f98c81e59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c81e59f0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f98c400d7d0 tx=0x7f98c400dc90 comp rx=0 tx=0).stop 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 shutdown_connections 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f98c80ce4f0 0x7f98c80b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98c80b3040 0x7f98c81ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f98ac0762d0 0x7f98ac0787e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 --2- 172.21.15.46:0/460466131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98c80b2c30 0x7f98c81e59f0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 >> 172.21.15.46:0/460466131 conn(0x7f98c80b4400 msgr2=0x7f98c80b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 shutdown_connections 2024-05-09T03:18:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.630+0000 7f98cf19d640 1 -- 172.21.15.46:0/460466131 wait complete. 2024-05-09T03:18:27.643 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:27.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:27.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.756+0000 7fdc8fc54640 1 Processor -- start 2024-05-09T03:18:27.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.756+0000 7fdc8fc54640 1 -- start start 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.757+0000 7fdc8fc54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc88164c70 0x7fdc881615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.757+0000 7fdc8fc54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc88160b20 con 0x7fdc8815dfe0 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.757+0000 7fdc8fc54640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880d15f0 con 0x7fdc880d2480 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.757+0000 7fdc8fc54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdc88161150 con 0x7fdc88164c70 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8d1ef640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc8815dfe0 0x7fdc880d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54612/0 (socket says 172.21.15.46:54612) 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8d1ef640 1 -- 172.21.15.46:0/3065182695 learned_addr learned my addr 172.21.15.46:0/3065182695 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8d9f0640 1 --2- 172.21.15.46:0/3065182695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc88164c70 0x7fdc881615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541378480 0 0) 0x7fdc88160b20 con 0x7fdc8815dfe0 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc74003520 con 0x7fdc8815dfe0 2024-05-09T03:18:27.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 754753322 0 0) 0x7fdc880d15f0 con 0x7fdc880d2480 2024-05-09T03:18:27.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc88160b20 con 0x7fdc880d2480 2024-05-09T03:18:27.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8d9f0640 1 -- 172.21.15.46:0/3065182695 >> v1:172.21.15.88:6789/0 conn(0x7fdc880d2480 legacy=0x7fdc880d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2708984856 0 0) 0x7fdc74003520 con 0x7fdc8815dfe0 2024-05-09T03:18:27.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8d9f0640 1 -- 172.21.15.46:0/3065182695 >> v1:172.21.15.46:6789/0 conn(0x7fdc8815dfe0 legacy=0x7fdc880d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.758+0000 7fdc8d9f0640 1 -- 172.21.15.46:0/3065182695 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc880d18d0 con 0x7fdc88164c70 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.759+0000 7fdc8d9f0640 1 --2- 172.21.15.46:0/3065182695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc88164c70 0x7fdc881615e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdc840097d0 tx=0x7fdc8402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b834f6c659a3b35 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.759+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84037070 con 0x7fdc88164c70 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.759+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8402ac30 con 0x7fdc88164c70 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.759+0000 7fdc8c9ee640 1 -- 172.21.15.46:0/3065182695 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc780016f0 con 0x7fdc88164c70 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 -- 172.21.15.46:0/3065182695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc88164c70 msgr2=0x7fdc881615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/3065182695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc88164c70 0x7fdc881615e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdc840097d0 tx=0x7fdc8402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 -- 172.21.15.46:0/3065182695 shutdown_connections 2024-05-09T03:18:27.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/3065182695 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc88164c70 0x7fdc881615e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 -- 172.21.15.46:0/3065182695 >> 172.21.15.46:0/3065182695 conn(0x7fdc88073db0 msgr2=0x7fdc880741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:27.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 -- 172.21.15.46:0/3065182695 shutdown_connections 2024-05-09T03:18:27.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.760+0000 7fdc8fc54640 1 -- 172.21.15.46:0/3065182695 wait complete. 2024-05-09T03:18:27.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.762+0000 7fdc8fc54640 1 Processor -- start 2024-05-09T03:18:27.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.762+0000 7fdc8fc54640 1 -- start start 2024-05-09T03:18:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8fc54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc880d2480 0x7fdc8814c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8fc54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc8815dfe0 0x7fdc880caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8fc54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 0x7fdc880cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8fc54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc880d45c0 con 0x7fdc88164c70 2024-05-09T03:18:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8fc54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8815e3f0 con 0x7fdc880d2480 2024-05-09T03:18:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8fc54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdc880d4740 con 0x7fdc8815dfe0 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8e1f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 0x7fdc880cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8e1f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 0x7fdc880cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41142/0 (socket says 172.21.15.46:41142) 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8e1f1640 1 -- 172.21.15.46:0/1187296481 learned_addr learned my addr 172.21.15.46:0/1187296481 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8d1ef640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc8815dfe0 0x7fdc880caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.763+0000 7fdc8d9f0640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc880d2480 0x7fdc8814c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8e1f1640 1 -- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc8815dfe0 msgr2=0x7fdc880caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8e1f1640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc8815dfe0 0x7fdc880caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8e1f1640 1 -- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc880d2480 msgr2=0x7fdc8814c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:27.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8e1f1640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc880d2480 0x7fdc8814c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8e1f1640 1 -- 172.21.15.46:0/1187296481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc880cd610 con 0x7fdc88164c70 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8d9f0640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc880d2480 0x7fdc8814c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8d1ef640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc8815dfe0 0x7fdc880caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8e1f1640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 0x7fdc880cb3e0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c760 tx=0x7fdc8000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80003b70 con 0x7fdc88164c70 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc880cd8f0 con 0x7fdc88164c70 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc80003d10 con 0x7fdc88164c70 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000aa90 con 0x7fdc88164c70 2024-05-09T03:18:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.764+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc8814ca80 con 0x7fdc88164c70 2024-05-09T03:18:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.765+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc880cd380 con 0x7fdc88164c70 2024-05-09T03:18:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.766+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc8000ac30 con 0x7fdc88164c70 2024-05-09T03:18:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.766+0000 7fdc7e7fc640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc6c0762d0 0x7fdc6c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.766+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7fdc80097c50 con 0x7fdc88164c70 2024-05-09T03:18:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.767+0000 7fdc8d9f0640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc6c0762d0 0x7fdc6c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.767+0000 7fdc8d9f0640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc6c0762d0 0x7fdc6c0787e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdc84034000 tx=0x7fdc8402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:27.771+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdc8009d020 con 0x7fdc88164c70 2024-05-09T03:18:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.002+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc8808cfd0 con 0x7fdc88164c70 2024-05-09T03:18:28.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.003+0000 7fdc7e7fc640 1 -- 172.21.15.46:0/1187296481 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) ==== 95+0+4343 (secure 0 0 0) 0x7fdc8808cfd0 con 0x7fdc88164c70 2024-05-09T03:18:28.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:28.005 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":215,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:27.254785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:28.005 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 215 2024-05-09T03:18:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc6c0762d0 msgr2=0x7fdc6c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc6c0762d0 0x7fdc6c0787e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdc84034000 tx=0x7fdc8402e000 comp rx=0 tx=0).stop 2024-05-09T03:18:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 msgr2=0x7fdc880cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 0x7fdc880cb3e0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000c760 tx=0x7fdc8000cc20 comp rx=0 tx=0).stop 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 shutdown_connections 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc6c0762d0 0x7fdc6c0787e0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc88164c70 0x7fdc880cb3e0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc8815dfe0 0x7fdc880caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 --2- 172.21.15.46:0/1187296481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc880d2480 0x7fdc8814c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 >> 172.21.15.46:0/1187296481 conn(0x7fdc88073db0 msgr2=0x7fdc8815fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 shutdown_connections 2024-05-09T03:18:28.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.005+0000 7fdc8fc54640 1 -- 172.21.15.46:0/1187296481 wait complete. 2024-05-09T03:18:28.018 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:28.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:28.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.131+0000 7fddc0061640 1 Processor -- start 2024-05-09T03:18:28.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.131+0000 7fddc0061640 1 -- start start 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddc0061640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 0x7fddb80d2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddc0061640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fddb80cfb20 0x7fddb80b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddc0061640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddb80d2030 con 0x7fddb80d2790 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddc0061640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fddb80d0fa0 con 0x7fddb80b5030 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddc0061640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fddb80cfff0 con 0x7fddb80cfb20 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbddfd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 0x7fddb80d2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbddfd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 0x7fddb80d2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41152/0 (socket says 172.21.15.46:41152) 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbe5fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fddb80cfb20 0x7fddb80b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbddfd640 1 -- 172.21.15.46:0/3425770306 learned_addr learned my addr 172.21.15.46:0/3425770306 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbe5fe640 1 -- 172.21.15.46:0/3425770306 >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 msgr2=0x7fddb80d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbe5fe640 1 --2- 172.21.15.46:0/3425770306 >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 0x7fddb80d2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbe5fe640 1 -- 172.21.15.46:0/3425770306 >> v1:172.21.15.88:6789/0 conn(0x7fddb80d2790 legacy=0x7fddb80cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbe5fe640 1 -- 172.21.15.46:0/3425770306 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb80d2ba0 con 0x7fddb80cfb20 2024-05-09T03:18:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbcdfb640 1 -- 172.21.15.46:0/3425770306 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4005133950 0 0) 0x7fddb80d2030 con 0x7fddb80d2790 2024-05-09T03:18:28.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.133+0000 7fddbddfd640 1 --2- 172.21.15.46:0/3425770306 >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 0x7fddb80d2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.134+0000 7fddbe5fe640 1 --2- 172.21.15.46:0/3425770306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80b6b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fddb80cab30 tx=0x7fddb002a550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6678832149f17be4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.134+0000 7fddbcdfb640 1 -- 172.21.15.46:0/3425770306 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb0007740 con 0x7fddb80cfb20 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.134+0000 7fddbcdfb640 1 -- 172.21.15.46:0/3425770306 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb00078e0 con 0x7fddb80cfb20 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.134+0000 7fddbcdfb640 1 -- 172.21.15.46:0/3425770306 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb0007cb0 con 0x7fddb80cfb20 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 -- 172.21.15.46:0/3425770306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 msgr2=0x7fddb80b6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 --2- 172.21.15.46:0/3425770306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80b6b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fddb80cab30 tx=0x7fddb002a550 comp rx=0 tx=0).stop 2024-05-09T03:18:28.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 -- 172.21.15.46:0/3425770306 shutdown_connections 2024-05-09T03:18:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 --2- 172.21.15.46:0/3425770306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80b6b20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 --2- 172.21.15.46:0/3425770306 >> v2:172.21.15.46:3300/0 conn(0x7fddb80b5030 0x7fddb80d2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 -- 172.21.15.46:0/3425770306 >> 172.21.15.46:0/3425770306 conn(0x7fddb80d3bd0 msgr2=0x7fddb80d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 -- 172.21.15.46:0/3425770306 shutdown_connections 2024-05-09T03:18:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.135+0000 7fddc0061640 1 -- 172.21.15.46:0/3425770306 wait complete. 2024-05-09T03:18:28.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.137+0000 7fddc0061640 1 Processor -- start 2024-05-09T03:18:28.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.137+0000 7fddc0061640 1 -- start start 2024-05-09T03:18:28.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddc0061640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 0x7fddb8153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddc0061640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddc0061640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb80d2790 0x7fddb80c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddc0061640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fddb80d0950 con 0x7fddb80b5030 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddc0061640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fddb815e9c0 con 0x7fddb80cfb20 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddc0061640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fddb80d0c30 con 0x7fddb80d2790 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.138+0000 7fddbddfd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 0x7fddb8153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 0x7fddb8153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41154/0 (socket says 172.21.15.46:41154) 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 -- 172.21.15.46:0/542954872 learned_addr learned my addr 172.21.15.46:0/542954872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbd5fc640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbe5fe640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb80d2790 0x7fddb80c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 -- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb80d2790 msgr2=0x7fddb80c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb80d2790 0x7fddb80c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 -- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 msgr2=0x7fddb80c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 -- 172.21.15.46:0/542954872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb8157a40 con 0x7fddb80b5030 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbe5fe640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb80d2790 0x7fddb80c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddbddfd640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 0x7fddb8153040 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fddb400e9e0 tx=0x7fddb400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4005990 con 0x7fddb80b5030 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb4005b30 con 0x7fddb80b5030 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb400c9d0 con 0x7fddb80b5030 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb8157760 con 0x7fddb80b5030 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.139+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb8153d20 con 0x7fddb80b5030 2024-05-09T03:18:28.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.140+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddb806a270 con 0x7fddb80b5030 2024-05-09T03:18:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.141+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fddb400cb70 con 0x7fddb80b5030 2024-05-09T03:18:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.141+0000 7fddae7fc640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd9c076300 0x7fdd9c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.142+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7fddb40980b0 con 0x7fddb80b5030 2024-05-09T03:18:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.142+0000 7fddbd5fc640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd9c076300 0x7fdd9c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.142+0000 7fddbd5fc640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd9c076300 0x7fdd9c078810 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fddb80c7690 tx=0x7fdda8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:28.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.146+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fddb8157a40 con 0x7fddb80b5030 2024-05-09T03:18:28.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.377+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddb80d1330 con 0x7fddb80b5030 2024-05-09T03:18:28.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.377+0000 7fddae7fc640 1 -- 172.21.15.46:0/542954872 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) ==== 95+0+5166 (secure 0 0 0) 0x7fddb4017020 con 0x7fddb80b5030 2024-05-09T03:18:28.379 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:28.379 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":216,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9006,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/202666403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":202666403},{"type":"v1","addr":"172.21.15.46:6834","nonce":202666403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":216}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:27.254785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:28.379 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 216 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd9c076300 msgr2=0x7fdd9c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd9c076300 0x7fdd9c078810 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fddb80c7690 tx=0x7fdda8009040 comp rx=0 tx=0).stop 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 msgr2=0x7fddb8153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 0x7fddb8153040 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fddb400e9e0 tx=0x7fddb400eea0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 shutdown_connections 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fddb80d2790 0x7fddb80c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddb80cfb20 0x7fddb80c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdd9c076300 0x7fdd9c078810 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 --2- 172.21.15.46:0/542954872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddb80b5030 0x7fddb8153040 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 >> 172.21.15.46:0/542954872 conn(0x7fddb80d3bd0 msgr2=0x7fddb815dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 shutdown_connections 2024-05-09T03:18:28.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.380+0000 7fddc0061640 1 -- 172.21.15.46:0/542954872 wait complete. 2024-05-09T03:18:28.393 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt mds0_inotable' led to DAMAGED state 2024-05-09T03:18:28.393 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 400.00000000' 2024-05-09T03:18:28.393 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:28.393 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:28.410 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:28.410 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:28.410 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:28.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:28.410 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:28.466 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:28.466 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:28.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.622+0000 7f91aa88b640 1 Processor -- start 2024-05-09T03:18:28.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.622+0000 7f91aa88b640 1 -- start start 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91aa88b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91a40b5370 0x7f91a40d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91aa88b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a40d21b0 con 0x7f91a41434a0 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91aa88b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a41438b0 con 0x7f91a40d1da0 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91aa88b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f91a40b4920 con 0x7f91a40b5370 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f91a41434a0 0x7f91a4160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54622/0 (socket says 172.21.15.46:54622) 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a3fff640 1 -- 172.21.15.46:0/4208173665 learned_addr learned my addr 172.21.15.46:0/4208173665 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f919bfff640 1 --2- 172.21.15.46:0/4208173665 >> v2:172.21.15.88:3300/0 conn(0x7f91a40b5370 0x7f91a40d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946169258 0 0) 0x7f9194001110 con 0x7f91a41434a0 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9190003520 con 0x7f91a41434a0 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2361439872 0 0) 0x7f91a41438b0 con 0x7f91a40d1da0 2024-05-09T03:18:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9194001110 con 0x7f91a40d1da0 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2472496503 0 0) 0x7f9190003520 con 0x7f91a41434a0 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 >> v2:172.21.15.88:3300/0 conn(0x7f91a40b5370 msgr2=0x7f91a40d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 --2- 172.21.15.46:0/4208173665 >> v2:172.21.15.88:3300/0 conn(0x7f91a40b5370 0x7f91a40d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 >> v1:172.21.15.88:6789/0 conn(0x7f91a40d1da0 legacy=0x7f91a40cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a41616b0 con 0x7f91a41434a0 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f91940018c0 con 0x7f91a41434a0 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f919bfff640 1 --2- 172.21.15.46:0/4208173665 >> v2:172.21.15.88:3300/0 conn(0x7f91a40b5370 0x7f91a40d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.624+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 >> v1:172.21.15.46:6789/0 conn(0x7f91a41434a0 legacy=0x7f91a4160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a37fe640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190003c30 0x7f9190004080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a37fe640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9190004800 0x7f9190004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a37fe640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9190005320 0x7f919000b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40b4920 con 0x7f9190005320 2024-05-09T03:18:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f919000c2c0 con 0x7f9190004800 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f919000c4c0 con 0x7f9190003c30 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a8e28640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9190005320 0x7f919000b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f919bfff640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9190004800 0x7f9190004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a3fff640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190003c30 0x7f9190004080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a8e28640 1 -- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190003c30 msgr2=0x7f9190004080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a8e28640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190003c30 0x7f9190004080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a8e28640 1 -- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9190004800 msgr2=0x7f9190004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a8e28640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9190004800 0x7f9190004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.625+0000 7f91a8e28640 1 -- 172.21.15.46:0/4208173665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9190003520 con 0x7f9190005320 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.626+0000 7f91a8e28640 1 -- 172.21.15.46:0/4208173665 reap_dead start 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.626+0000 7f91a8e28640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9190005320 0x7f919000b9e0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f919c001fd0 tx=0x7f9190005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30b608313b0ba4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.626+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c001c20 con 0x7f9190005320 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.626+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919c003b70 con 0x7f9190005320 2024-05-09T03:18:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.626+0000 7f91a37fe640 1 -- 172.21.15.46:0/4208173665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c0050d0 con 0x7f9190005320 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 -- 172.21.15.46:0/4208173665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9190005320 msgr2=0x7f919000b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9190005320 0x7f919000b9e0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f919c001fd0 tx=0x7f9190005830 comp rx=0 tx=0).stop 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 -- 172.21.15.46:0/4208173665 shutdown_connections 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 --2- 172.21.15.46:0/4208173665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9190005320 0x7f919000b9e0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 -- 172.21.15.46:0/4208173665 >> 172.21.15.46:0/4208173665 conn(0x7f91a40b3620 msgr2=0x7f91a40b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 -- 172.21.15.46:0/4208173665 shutdown_connections 2024-05-09T03:18:28.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.627+0000 7f91aa88b640 1 -- 172.21.15.46:0/4208173665 wait complete. 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.628+0000 7f91aa88b640 1 Processor -- start 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.629+0000 7f91aa88b640 1 -- start start 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91aa88b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190005320 0x7f91a415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91aa88b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a4144bf0 0x7f91a4145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91aa88b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 0x7f91a40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91aa88b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40b4b70 con 0x7f91a40732d0 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91aa88b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40b4ee0 con 0x7f91a4144bf0 2024-05-09T03:18:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91aa88b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f91a40b5780 con 0x7f9190005320 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 0x7f91a40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 0x7f91a40736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41170/0 (socket says 172.21.15.46:41170) 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 -- 172.21.15.46:0/1979173156 learned_addr learned my addr 172.21.15.46:0/1979173156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a3fff640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190005320 0x7f91a415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f919bfff640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a4144bf0 0x7f91a4145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 -- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190005320 msgr2=0x7f91a415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190005320 0x7f91a415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 -- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a4144bf0 msgr2=0x7f91a4145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a4144bf0 0x7f91a4145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.630+0000 7f91a8e28640 1 -- 172.21.15.46:0/1979173156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a4073c20 con 0x7f91a40732d0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.631+0000 7f91a8e28640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 0x7f91a40736e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f919c001fd0 tx=0x7f919c001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.631+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c031400 con 0x7f91a40732d0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.631+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919c0315a0 con 0x7f91a40732d0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.631+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a4145960 con 0x7f91a40732d0 2024-05-09T03:18:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.631+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c031970 con 0x7f91a40732d0 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.631+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a40aea50 con 0x7f91a40732d0 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.632+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a406a270 con 0x7f91a40732d0 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.632+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f919c031b10 con 0x7f91a40732d0 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.633+0000 7f91a17fa640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91840762d0 0x7f91840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.633+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7f919c0b77a0 con 0x7f91a40732d0 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.633+0000 7f91a3fff640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91840762d0 0x7f91840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.634+0000 7f91a3fff640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91840762d0 0x7f91840787e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9194001860 tx=0x7f91940079f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:28.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.638+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f91a4073c20 con 0x7f91a40732d0 2024-05-09T03:18:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:28.868+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f91a41456f0 con 0x7f91a40732d0 2024-05-09T03:18:29.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.265+0000 7f91a17fa640 1 -- 172.21.15.46:0/1979173156 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v217) ==== 161+0+0 (secure 0 0 0) 0x7f919c038070 con 0x7f91a40732d0 2024-05-09T03:18:29.268 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91840762d0 msgr2=0x7f91840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91840762d0 0x7f91840787e0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f9194001860 tx=0x7f91940079f0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 msgr2=0x7f91a40736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 0x7f91a40736e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f919c001fd0 tx=0x7f919c001670 comp rx=0 tx=0).stop 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 shutdown_connections 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91840762d0 0x7f91840787e0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91a40732d0 0x7f91a40736e0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a4144bf0 0x7f91a4145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 --2- 172.21.15.46:0/1979173156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9190005320 0x7f91a415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 >> 172.21.15.46:0/1979173156 conn(0x7f91a40b3620 msgr2=0x7f91a40cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:29.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 shutdown_connections 2024-05-09T03:18:29.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.268+0000 7f91aa88b640 1 -- 172.21.15.46:0/1979173156 wait complete. 2024-05-09T03:18:29.283 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:29.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.401+0000 7f6ffdde3640 1 Processor -- start 2024-05-09T03:18:29.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.401+0000 7f6ffdde3640 1 -- start start 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ffdde3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ff80b5140 0x7f6ff80b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ffdde3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ff80b7ca0 0x7f6ff80d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ffdde3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff80b80b0 con 0x7f6ff80d3df0 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ffdde3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff80d3480 con 0x7f6ff80b5140 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ffdde3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff80b5550 con 0x7f6ff80b7ca0 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ff80b5140 0x7f6ff80b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:29.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ff80b5140 0x7f6ff80b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41172/0 (socket says 172.21.15.46:41172) 2024-05-09T03:18:29.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 -- 172.21.15.46:0/3643131070 learned_addr learned my addr 172.21.15.46:0/3643131070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:29.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff6ffd640 1 --2- 172.21.15.46:0/3643131070 >> v2:172.21.15.88:3300/0 conn(0x7f6ff80b7ca0 0x7f6ff80d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:29.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 -- 172.21.15.46:0/3643131070 >> v2:172.21.15.88:3300/0 conn(0x7f6ff80b7ca0 msgr2=0x7f6ff80d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:29.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 --2- 172.21.15.46:0/3643131070 >> v2:172.21.15.88:3300/0 conn(0x7f6ff80b7ca0 0x7f6ff80d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 -- 172.21.15.46:0/3643131070 >> v1:172.21.15.88:6789/0 conn(0x7f6ff80d3df0 legacy=0x7f6ff8163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:29.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.403+0000 7f6ff77fe640 1 -- 172.21.15.46:0/3643131070 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff80d44e0 con 0x7f6ff80b5140 2024-05-09T03:18:29.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.404+0000 7f6ff77fe640 1 --2- 172.21.15.46:0/3643131070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff80b7760 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80d0290 tx=0x7f6fe802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2906bf4e50b0d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:29.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.404+0000 7f6ff67fc640 1 -- 172.21.15.46:0/3643131070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8037070 con 0x7f6ff80b5140 2024-05-09T03:18:29.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.404+0000 7f6ff67fc640 1 -- 172.21.15.46:0/3643131070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe802ae50 con 0x7f6ff80b5140 2024-05-09T03:18:29.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.404+0000 7f6ff67fc640 1 -- 172.21.15.46:0/3643131070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8032860 con 0x7f6ff80b5140 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 -- 172.21.15.46:0/3643131070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 msgr2=0x7f6ff80b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/3643131070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff80b7760 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80d0290 tx=0x7f6fe802a300 comp rx=0 tx=0).stop 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 -- 172.21.15.46:0/3643131070 shutdown_connections 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/3643131070 >> v2:172.21.15.88:3300/0 conn(0x7f6ff80b7ca0 0x7f6ff80d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/3643131070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff80b7760 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 -- 172.21.15.46:0/3643131070 >> 172.21.15.46:0/3643131070 conn(0x7f6ff808d3f0 msgr2=0x7f6ff808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 -- 172.21.15.46:0/3643131070 shutdown_connections 2024-05-09T03:18:29.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.405+0000 7f6ffdde3640 1 -- 172.21.15.46:0/3643131070 wait complete. 2024-05-09T03:18:29.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.407+0000 7f6ffdde3640 1 Processor -- start 2024-05-09T03:18:29.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.407+0000 7f6ffdde3640 1 -- start start 2024-05-09T03:18:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ffdde3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff814ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ffdde3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ff80b7ca0 0x7f6ff815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ffdde3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ff80d3df0 0x7f6ff8158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ffdde3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff80d4d80 con 0x7f6ff80b5140 2024-05-09T03:18:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ffdde3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff80d4aa0 con 0x7f6ff80b7ca0 2024-05-09T03:18:29.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ffdde3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff80d4200 con 0x7f6ff80d3df0 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff814ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff814ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41188/0 (socket says 172.21.15.46:41188) 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.408+0000 7f6ff77fe640 1 -- 172.21.15.46:0/1410572260 learned_addr learned my addr 172.21.15.46:0/1410572260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff6ffd640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ff80b7ca0 0x7f6ff815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff7fff640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ff80d3df0 0x7f6ff8158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff77fe640 1 -- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ff80d3df0 msgr2=0x7f6ff8158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff77fe640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ff80d3df0 0x7f6ff8158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff77fe640 1 -- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ff80b7ca0 msgr2=0x7f6ff815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff77fe640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ff80b7ca0 0x7f6ff815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff77fe640 1 -- 172.21.15.46:0/1410572260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff814b650 con 0x7f6ff80b5140 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff6ffd640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ff80b7ca0 0x7f6ff815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff7fff640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ff80d3df0 0x7f6ff8158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ff77fe640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff814ac80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f6fe802a340 tx=0x7f6fe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8041020 con 0x7f6ff80b5140 2024-05-09T03:18:29.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe803c890 con 0x7f6ff80b5140 2024-05-09T03:18:29.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff8159280 con 0x7f6ff80b5140 2024-05-09T03:18:29.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8037040 con 0x7f6ff80b5140 2024-05-09T03:18:29.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.409+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff8159840 con 0x7f6ff80b5140 2024-05-09T03:18:29.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.410+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff806a270 con 0x7f6ff80b5140 2024-05-09T03:18:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.411+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6fe8044020 con 0x7f6ff80b5140 2024-05-09T03:18:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.411+0000 7f6fdffff640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fe4076300 0x7f6fe4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.412+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7f6fe80b85b0 con 0x7f6ff80b5140 2024-05-09T03:18:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.412+0000 7f6ff6ffd640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fe4076300 0x7f6fe4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.412+0000 7f6ff6ffd640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fe4076300 0x7f6fe4078810 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6ff815eee0 tx=0x7f6fec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:29.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.416+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6fe804b070 con 0x7f6ff80b5140 2024-05-09T03:18:29.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:29.681+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f6ff815f4d0 con 0x7f6ff80b5140 2024-05-09T03:18:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.267+0000 7f6fdffff640 1 -- 172.21.15.46:0/1410572260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v218) ==== 101+0+0 (secure 0 0 0) 0x7f6ff814b650 con 0x7f6ff80b5140 2024-05-09T03:18:30.269 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fe4076300 msgr2=0x7f6fe4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fe4076300 0x7f6fe4078810 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6ff815eee0 tx=0x7f6fec008040 comp rx=0 tx=0).stop 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 msgr2=0x7f6ff814ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff814ac80 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f6fe802a340 tx=0x7f6fe80046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 shutdown_connections 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ff80d3df0 0x7f6ff8158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ff80b7ca0 0x7f6ff815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fe4076300 0x7f6fe4078810 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 --2- 172.21.15.46:0/1410572260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ff80b5140 0x7f6ff814ac80 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 >> 172.21.15.46:0/1410572260 conn(0x7f6ff808d3f0 msgr2=0x7f6ff8164670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 shutdown_connections 2024-05-09T03:18:30.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.270+0000 7f6ffdde3640 1 -- 172.21.15.46:0/1410572260 wait complete. 2024-05-09T03:18:30.286 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:30.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.346+0000 7f800dbc9f40 1 Processor -- start 2024-05-09T03:18:30.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.346+0000 7f800dbc9f40 1 -- start start 2024-05-09T03:18:30.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.347+0000 7f800dbc9f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f800dbc9f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c6b5820 0x564f2c6b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f800dbc9f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c7577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f800dbc9f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564f2c62ef50 con 0x564f2c6b6170 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f800dbc9f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564f2c632f90 con 0x564f2c75d410 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f800dbc9f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564f2c6cc4d0 con 0x564f2c6b5820 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c7577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c7577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41206/0 (socket says 172.21.15.46:41206) 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 -- 172.21.15.46:0/2102327590 learned_addr learned my addr 172.21.15.46:0/2102327590 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007691640 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8006e90640 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c6b5820 0x564f2c6b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 -- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c6b5820 msgr2=0x564f2c6b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c6b5820 0x564f2c6b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 -- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c75d410 msgr2=0x564f2c75dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 -- 172.21.15.46:0/2102327590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2c758d30 con 0x564f2c6b6170 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007691640 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8007e92640 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c7577d0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8000025db0 tx=0x7f8000008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9d70a60c29ed26 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8005e8e640 1 -- 172.21.15.46:0/2102327590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8000001a70 con 0x564f2c6b6170 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8005e8e640 1 -- 172.21.15.46:0/2102327590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8000006070 con 0x564f2c6b6170 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.348+0000 7f8005e8e640 1 -- 172.21.15.46:0/2102327590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800004d640 con 0x564f2c6b6170 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 -- 172.21.15.46:0/2102327590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 msgr2=0x564f2c7577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c7577d0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f8000025db0 tx=0x7f8000008030 comp rx=0 tx=0).stop 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 -- 172.21.15.46:0/2102327590 shutdown_connections 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c7577d0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c6b5820 0x564f2c6b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/2102327590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 -- 172.21.15.46:0/2102327590 >> 172.21.15.46:0/2102327590 conn(0x564f2c5aaac0 msgr2=0x564f2c6cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 -- 172.21.15.46:0/2102327590 shutdown_connections 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.349+0000 7f800dbc9f40 1 -- 172.21.15.46:0/2102327590 wait complete. 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.350+0000 7f800dbc9f40 1 Processor -- start 2024-05-09T03:18:30.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.350+0000 7f800dbc9f40 1 -- start start 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.351+0000 7f800dbc9f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c6b5820 0x564f2c74ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.351+0000 7f800dbc9f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c75a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.351+0000 7f800dbc9f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.351+0000 7f800dbc9f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564f2c6b6580 con 0x564f2c6b6170 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.351+0000 7f800dbc9f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564f2c75d820 con 0x564f2c6b5820 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.351+0000 7f800dbc9f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564f2c758880 con 0x564f2c75d410 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c75a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c75a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41222/0 (socket says 172.21.15.46:41222) 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 learned_addr learned my addr 172.21.15.46:0/1481332453 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c6b5820 0x564f2c74ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8007e92640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c75d410 msgr2=0x564f2c75dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564f2c75d410 0x564f2c75dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c6b5820 msgr2=0x564f2c74ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c6b5820 0x564f2c74ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f2c6ca120 con 0x564f2c6b6170 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f2c6b5820 0x564f2c74ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c75a590 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009f80 tx=0x7f7ffc005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8004e8c640 1 -- 172.21.15.46:0/1481332453 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc01a070 con 0x564f2c6b6170 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8004e8c640 1 -- 172.21.15.46:0/1481332453 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ffc016410 con 0x564f2c6b6170 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564f2c6ca400 con 0x564f2c6b6170 2024-05-09T03:18:30.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f8004e8c640 1 -- 172.21.15.46:0/1481332453 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0167e0 con 0x564f2c6b6170 2024-05-09T03:18:30.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.352+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f2c6d1140 con 0x564f2c6b6170 2024-05-09T03:18:30.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.354+0000 7f8004e8c640 1 -- 172.21.15.46:0/1481332453 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7ffc01d070 con 0x564f2c6b6170 2024-05-09T03:18:30.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f8004e8c640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ff4074ce0 0x7f7ff40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f8004e8c640 1 -- 172.21.15.46:0/1481332453 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7f7ffc096f80 con 0x564f2c6b6170 2024-05-09T03:18:30.358 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:30.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564f2c7f63f0 0x564f2c7f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8024c0 con 0x564f2c7f63f0 2024-05-09T03:18:30.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ff4074ce0 0x7f7ff40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f8007e92640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564f2c7f63f0 0x564f2c7f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f8007e92640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564f2c7f63f0 0x564f2c7f6800 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.355+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ff4074ce0 0x7f7ff40771f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x564f2c6d2780 tx=0x7f7ff80018e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.357+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v105'629 uv603 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.357+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c802860 con 0x564f2c7f63f0 2024-05-09T03:18:30.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.359+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v105'630 uv630 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.359+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c802c00 con 0x564f2c7f63f0 2024-05-09T03:18:30.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.360+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v105'631 uv631 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.362 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:30.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.360+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803fb0 con 0x564f2c7f63f0 2024-05-09T03:18:30.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.361+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v105'632 uv632 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.361+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c804350 con 0x564f2c7f63f0 2024-05-09T03:18:30.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.363+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v105'633 uv633 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.363+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c802fa0 con 0x564f2c7f63f0 2024-05-09T03:18:30.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.364+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v105'634 uv634 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.364+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.365+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v105'635 uv609 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.365+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.367+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v105'636 uv636 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.367+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.369 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:30.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.368+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v105'637 uv637 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.368+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.369+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v105'638 uv638 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.369+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.371+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v105'639 uv639 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.371+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.372+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v105'640 uv640 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.372+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.373+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v105'641 uv616 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.373+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.375+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v105'642 uv642 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.375+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.376+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v105'643 uv643 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.378 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:30.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.376+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.378+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v105'644 uv644 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.378+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.379+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v105'645 uv645 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.379+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c803340 con 0x564f2c7f63f0 2024-05-09T03:18:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.380+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v105'646 uv646 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.380+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.381+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v105'647 uv647 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.382+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.383+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v105'648 uv622 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.383+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.384+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v105'649 uv649 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.384+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.387 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:30.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.386+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v105'650 uv650 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.386+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.387+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v105'651 uv651 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.387+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.388+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v105'652 uv652 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.388+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.389+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v105'653 uv653 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.389+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.391+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v105'654 uv628 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.391+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.392+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v105'655 uv655 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.392+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.393+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v105'656 uv656 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.395 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.393+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.395+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v105'657 uv657 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.395+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.396+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v105'658 uv658 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.396+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c7f63f0 2024-05-09T03:18:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.397+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v105'659 uv659 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.397+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564f2c8046f0 0x564f2c806b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.397+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c75e370 con 0x564f2c8046f0 2024-05-09T03:18:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.397+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564f2c8046f0 0x564f2c806b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.398+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564f2c8046f0 0x564f2c806b10 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.400+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v105'459 uv442 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.400+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c807050 con 0x564f2c8046f0 2024-05-09T03:18:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.401+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v105'460 uv460 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.401+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8073f0 con 0x564f2c8046f0 2024-05-09T03:18:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.402+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v105'461 uv461 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.405 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:30.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.402+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c807c40 con 0x564f2c8046f0 2024-05-09T03:18:30.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.404+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v105'462 uv462 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.404+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c807fe0 con 0x564f2c8046f0 2024-05-09T03:18:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.405+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v105'463 uv463 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.405+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808380 con 0x564f2c8046f0 2024-05-09T03:18:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.406+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v105'464 uv447 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.406+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c808720 con 0x564f2c8046f0 2024-05-09T03:18:30.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.407+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v105'465 uv465 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.407+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.408+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v105'466 uv466 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.410 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:30.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.408+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.410+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v105'467 uv467 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.410+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.411+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v105'468 uv468 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.411+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.412+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v105'469 uv452 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.412+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.413+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v105'470 uv470 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.413+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.415+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v105'471 uv471 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.417 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:30.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.415+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c520 con 0x564f2c8046f0 2024-05-09T03:18:30.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.416+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v105'472 uv472 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.416+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c520 con 0x564f2c8046f0 2024-05-09T03:18:30.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.417+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v105'473 uv473 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.418+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.419+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v105'474 uv458 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.419+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.420+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v105'475 uv475 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.420+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.421+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v105'476 uv476 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.423 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:30.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.421+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c809ad0 con 0x564f2c8046f0 2024-05-09T03:18:30.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.423+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v105'477 uv477 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.423+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c809ad0 con 0x564f2c8046f0 2024-05-09T03:18:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.424+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v105'478 uv478 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.424+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c8046f0 2024-05-09T03:18:30.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.425+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v105'479 uv479 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c8046f0 2024-05-09T03:18:30.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.425+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c7f63f0 2024-05-09T03:18:30.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.426+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v105'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.426+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c7f63f0 2024-05-09T03:18:30.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.427+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v105'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.427+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c7f63f0 2024-05-09T03:18:30.430 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:30.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.428+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v105'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.428+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c809ad0 con 0x564f2c7f63f0 2024-05-09T03:18:30.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.429+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v105'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.429+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c809ad0 con 0x564f2c7f63f0 2024-05-09T03:18:30.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.430+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v105'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.430+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c808ac0 con 0x564f2c7f63f0 2024-05-09T03:18:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.431+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v105'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.431+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564f2c808fd0 0x564f2c80b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.431+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80b930 con 0x564f2c808fd0 2024-05-09T03:18:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.431+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564f2c808fd0 0x564f2c80b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.432+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564f2c808fd0 0x564f2c80b3f0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.433+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v105'94 uv93 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.433+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c80bcd0 con 0x564f2c808fd0 2024-05-09T03:18:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.434+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v105'95 uv95 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.434+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c180 con 0x564f2c808fd0 2024-05-09T03:18:30.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.435+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v105'96 uv96 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.436 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.435+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c807870 con 0x564f2c808fd0 2024-05-09T03:18:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.436+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v105'97 uv97 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.436+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80cab0 con 0x564f2c7f63f0 2024-05-09T03:18:30.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.437+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v105'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.437+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c80cab0 con 0x564f2c7f63f0 2024-05-09T03:18:30.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.439+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v105'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.439+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80cab0 con 0x564f2c7f63f0 2024-05-09T03:18:30.441 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:30.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.440+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v105'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.440+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80dac0 con 0x564f2c7f63f0 2024-05-09T03:18:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.441+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v105'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.441+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80dac0 con 0x564f2c7f63f0 2024-05-09T03:18:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.442+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v105'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.443+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80cab0 con 0x564f2c7f63f0 2024-05-09T03:18:30.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.444+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v105'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.444+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564f2c80d380 0x564f2c80d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.444+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80feb0 con 0x564f2c80d380 2024-05-09T03:18:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.444+0000 7f8007e92640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564f2c80d380 0x564f2c80d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.445+0000 7f8007e92640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564f2c80d380 0x564f2c80d790 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.446+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v105'227 uv221 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.446+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c80feb0 con 0x564f2c80d380 2024-05-09T03:18:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.447+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v105'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.447+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80feb0 con 0x564f2c80d380 2024-05-09T03:18:30.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.448+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v105'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.450 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:30.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.448+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c600 con 0x564f2c80d380 2024-05-09T03:18:30.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.449+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v105'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.449+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c600 con 0x564f2c80d380 2024-05-09T03:18:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.450+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v105'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.451+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c600 con 0x564f2c80d380 2024-05-09T03:18:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.452+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v105'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.452+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c5e0 con 0x564f2c80d380 2024-05-09T03:18:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.453+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v105'233 uv226 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.453+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c80c5e0 con 0x564f2c80d380 2024-05-09T03:18:30.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.455+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v105'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.455+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c5e0 con 0x564f2c80d380 2024-05-09T03:18:30.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.456+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v105'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.457 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:30.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.456+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c5e0 con 0x564f2c80d380 2024-05-09T03:18:30.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.457+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v105'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.457+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c80c5e0 con 0x564f2c80d380 2024-05-09T03:18:30.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.458+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v105'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.458+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x564f2c810890 0x564f2c812cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.458+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8131f0 con 0x564f2c810890 2024-05-09T03:18:30.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.459+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x564f2c810890 0x564f2c812cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.459+0000 7f8007691640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x564f2c810890 0x564f2c812cb0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.460+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v105'102 uv101 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c810890 2024-05-09T03:18:30.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.460+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c810890 2024-05-09T03:18:30.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.461+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v105'103 uv103 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c810890 2024-05-09T03:18:30.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.461+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c810890 2024-05-09T03:18:30.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.462+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v105'104 uv104 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c810890 2024-05-09T03:18:30.464 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:30.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.462+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c813600 con 0x564f2c810890 2024-05-09T03:18:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.464+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v105'105 uv105 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c810890 2024-05-09T03:18:30.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.464+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c810890 2024-05-09T03:18:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.465+0000 7f8007691640 1 -- 172.21.15.46:0/1481332453 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v105'106 uv106 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ff8012d00 con 0x564f2c810890 2024-05-09T03:18:30.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.465+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c7f63f0 2024-05-09T03:18:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.466+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v105'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.466+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c7f63f0 2024-05-09T03:18:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.467+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v105'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.467+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c7f63f0 2024-05-09T03:18:30.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.468+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v105'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.469 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.468+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c813600 con 0x564f2c7f63f0 2024-05-09T03:18:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.469+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v105'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.469+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c7f63f0 2024-05-09T03:18:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.470+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v105'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.470+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c7f63f0 2024-05-09T03:18:30.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.471+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v105'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c7f63f0 2024-05-09T03:18:30.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.471+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8135c0 con 0x564f2c808fd0 2024-05-09T03:18:30.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.472+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v105'203 uv202 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.472+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c813a90 con 0x564f2c808fd0 2024-05-09T03:18:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.473+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v105'204 uv204 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.473+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c813e80 con 0x564f2c808fd0 2024-05-09T03:18:30.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.474+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v105'205 uv205 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.476 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.474+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c814720 con 0x564f2c808fd0 2024-05-09T03:18:30.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.475+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v105'206 uv206 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.475+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c814720 con 0x564f2c808fd0 2024-05-09T03:18:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.476+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v105'207 uv207 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.476+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8146d0 con 0x564f2c808fd0 2024-05-09T03:18:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.477+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v105'208 uv201 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.477+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c8146d0 con 0x564f2c808fd0 2024-05-09T03:18:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.478+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v105'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.478+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8146d0 con 0x564f2c808fd0 2024-05-09T03:18:30.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.479+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v105'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.481 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:30.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.479+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c8146d0 con 0x564f2c808fd0 2024-05-09T03:18:30.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.480+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v105'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.480+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817460 con 0x564f2c808fd0 2024-05-09T03:18:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.481+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v105'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.482+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x564f2c815010 0x564f2c817850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.482+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c815010 2024-05-09T03:18:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.482+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x564f2c815010 0x564f2c817850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.482+0000 7f8006e90640 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x564f2c815010 0x564f2c817850 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.483+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v105'91 uv90 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c815010 2024-05-09T03:18:30.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.483+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c815010 2024-05-09T03:18:30.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.484+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v105'92 uv92 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c815010 2024-05-09T03:18:30.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.484+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c815010 2024-05-09T03:18:30.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.485+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v105'93 uv93 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c815010 2024-05-09T03:18:30.487 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:30.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.485+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c815010 2024-05-09T03:18:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.486+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v105'94 uv94 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c815010 2024-05-09T03:18:30.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.486+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c808fd0 2024-05-09T03:18:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.488+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v105'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.488+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c808fd0 2024-05-09T03:18:30.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.489+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v105'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.489+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c808fd0 2024-05-09T03:18:30.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.490+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v105'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.492 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.490+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c808fd0 2024-05-09T03:18:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.491+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v105'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.491+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c808fd0 2024-05-09T03:18:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.492+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v105'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.492+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c808fd0 2024-05-09T03:18:30.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.493+0000 7f8006e90640 1 -- 172.21.15.46:0/1481332453 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v105'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7ffc026070 con 0x564f2c808fd0 2024-05-09T03:18:30.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.493+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.494+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v105'104 uv103 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.494+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.496+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v105'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.496+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.497+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v105'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.499 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:30.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.497+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.499+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v105'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.499+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.500+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v105'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.500+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.502+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v105'253 uv245 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.502+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.503+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v105'254 uv254 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.503+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.504+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v105'255 uv255 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.506 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:30.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.504+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.505+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v105'256 uv256 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.505+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.506+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v105'257 uv257 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.506+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.508+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v105'258 uv252 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.508+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.509+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v105'259 uv259 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.509+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.510+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v105'260 uv260 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.512 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:30.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.511+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.512+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v105'261 uv261 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.512+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.513+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v105'262 uv262 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.513+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817430 con 0x564f2c80d380 2024-05-09T03:18:30.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.514+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v105'263 uv263 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.514+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x564f2c817d90 con 0x564f2c80d380 2024-05-09T03:18:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v105'264 uv264 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8000006220 con 0x564f2c80d380 2024-05-09T03:18:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564f2c7f63f0 msgr2=0x564f2c7f6800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x564f2c7f63f0 0x564f2c7f6800 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564f2c808fd0 msgr2=0x564f2c80b3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564f2c808fd0 0x564f2c80b3f0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x564f2c815010 msgr2=0x564f2c817850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x564f2c815010 0x564f2c817850 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564f2c8046f0 msgr2=0x564f2c806b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x564f2c8046f0 0x564f2c806b10 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 reap_dead start 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564f2c80d380 msgr2=0x564f2c80d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564f2c80d380 0x564f2c80d790 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x564f2c810890 msgr2=0x564f2c812cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.515+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x564f2c810890 0x564f2c812cb0 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ff4074ce0 msgr2=0x7f7ff40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ff4074ce0 0x7f7ff40771f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x564f2c6d2780 tx=0x7f7ff80018e0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 msgr2=0x564f2c75a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 --2- 172.21.15.46:0/1481332453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f2c6b6170 0x564f2c75a590 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009f80 tx=0x7f7ffc005ec0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f8007e92640 1 -- 172.21.15.46:0/1481332453 reap_dead start 2024-05-09T03:18:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 shutdown_connections 2024-05-09T03:18:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 >> 172.21.15.46:0/1481332453 conn(0x564f2c5aaac0 msgr2=0x564f2c6cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 shutdown_connections 2024-05-09T03:18:30.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.516+0000 7f800dbc9f40 1 -- 172.21.15.46:0/1481332453 wait complete. 2024-05-09T03:18:30.521 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 400.00000000 - 2024-05-09T03:18:30.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.616+0000 7fbeb6083f40 1 Processor -- start 2024-05-09T03:18:30.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.616+0000 7fbeb6083f40 1 -- start start 2024-05-09T03:18:30.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb6083f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3959b5f0 0x55fd39629360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb6083f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39626a40 0x55fd39596240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb6083f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 0x55fd39596780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb6083f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fd3959c8d0 con 0x55fd39627390 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb6083f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fd3959c5f0 con 0x55fd3959b5f0 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb6083f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fd39597bb0 con 0x55fd39626a40 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb034c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 0x55fd39596780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb034c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 0x55fd39596780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41246/0 (socket says 172.21.15.46:41246) 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeb034c640 1 -- 172.21.15.46:0/465094471 learned_addr learned my addr 172.21.15.46:0/465094471 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeafb4b640 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3959b5f0 0x55fd39629360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.617+0000 7fbeaf34a640 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39626a40 0x55fd39596240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeb034c640 1 -- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39626a40 msgr2=0x55fd39596240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeb034c640 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39626a40 0x55fd39596240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeb034c640 1 -- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3959b5f0 msgr2=0x55fd39629360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeb034c640 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3959b5f0 0x55fd39629360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeb034c640 1 -- 172.21.15.46:0/465094471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd394bb350 con 0x55fd39627390 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeafb4b640 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3959b5f0 0x55fd39629360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeb034c640 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 0x55fd39596780 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fbea8025b80 tx=0x7fbea8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec58d717c7e65b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeae348640 1 -- 172.21.15.46:0/465094471 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea8062070 con 0x55fd39627390 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeae348640 1 -- 172.21.15.46:0/465094471 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea8054950 con 0x55fd39627390 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.618+0000 7fbeae348640 1 -- 172.21.15.46:0/465094471 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea8054d00 con 0x55fd39627390 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 -- 172.21.15.46:0/465094471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 msgr2=0x55fd39596780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 0x55fd39596780 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fbea8025b80 tx=0x7fbea8051000 comp rx=0 tx=0).stop 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 -- 172.21.15.46:0/465094471 shutdown_connections 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd39627390 0x55fd39596780 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39626a40 0x55fd39596240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/465094471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd3959b5f0 0x55fd39629360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 -- 172.21.15.46:0/465094471 >> 172.21.15.46:0/465094471 conn(0x55fd39475ac0 msgr2=0x55fd39624570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 -- 172.21.15.46:0/465094471 shutdown_connections 2024-05-09T03:18:30.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.619+0000 7fbeb6083f40 1 -- 172.21.15.46:0/465094471 wait complete. 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.620+0000 7fbeb6083f40 1 Processor -- start 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.620+0000 7fbeb6083f40 1 -- start start 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb6083f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3959b5f0 0x55fd3957cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb6083f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd39626a40 0x55fd39619210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb6083f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39627390 0x55fd39619750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb6083f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fd395939f0 con 0x55fd3959b5f0 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb6083f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fd39624ff0 con 0x55fd39626a40 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb6083f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fd3961a2e0 con 0x55fd39627390 2024-05-09T03:18:30.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3959b5f0 0x55fd3957cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeaf34a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd39626a40 0x55fd39619210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3959b5f0 0x55fd3957cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41250/0 (socket says 172.21.15.46:41250) 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 -- 172.21.15.46:0/3888310430 learned_addr learned my addr 172.21.15.46:0/3888310430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb034c640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39627390 0x55fd39619750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 -- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39627390 msgr2=0x55fd39619750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39627390 0x55fd39619750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 -- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd39626a40 msgr2=0x55fd39619210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd39626a40 0x55fd39619210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeafb4b640 1 -- 172.21.15.46:0/3888310430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd3961dca0 con 0x55fd3959b5f0 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeb034c640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fd39627390 0x55fd39619750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.621+0000 7fbeaf34a640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fd39626a40 0x55fd39619210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.622+0000 7fbeafb4b640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3959b5f0 0x55fd3957cc10 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fbea000f580 tx=0x7fbea000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.622+0000 7fbead346640 1 -- 172.21.15.46:0/3888310430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0019030 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.622+0000 7fbead346640 1 -- 172.21.15.46:0/3888310430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea0015690 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.622+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd3961df80 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.622+0000 7fbead346640 1 -- 172.21.15.46:0/3888310430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0015860 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.622+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd3961fee0 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.623+0000 7fbead346640 1 -- 172.21.15.46:0/3888310430 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbea001c070 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.624+0000 7fbead346640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe90074ce0 0x7fbe900771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.624+0000 7fbead346640 1 -- 172.21.15.46:0/3888310430 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7fbea0096fe0 con 0x55fd3959b5f0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.624+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fd396208a0 0x55fd39620cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.624+0000 7fbeaf34a640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe90074ce0 0x7fbe900771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.624+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55fd3961ca40 con 0x55fd396208a0 2024-05-09T03:18:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.624+0000 7fbeb034c640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fd396208a0 0x55fd39620cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.625+0000 7fbeaf34a640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe90074ce0 0x7fbe900771f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x55fd3957e310 tx=0x7fbe9800c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.625+0000 7fbeb034c640 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fd396208a0 0x55fd39620cb0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb034c640 1 -- 172.21.15.46:0/3888310430 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 400.00000000 [writefull 0~80] v105'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbea806f040 con 0x55fd396208a0 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fd396208a0 msgr2=0x55fd39620cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fd396208a0 0x55fd39620cb0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe90074ce0 msgr2=0x7fbe900771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe90074ce0 0x7fbe900771f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x55fd3957e310 tx=0x7fbe9800c040 comp rx=0 tx=0).stop 2024-05-09T03:18:30.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3959b5f0 msgr2=0x55fd3957cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 --2- 172.21.15.46:0/3888310430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fd3959b5f0 0x55fd3957cc10 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fbea000f580 tx=0x7fbea000fa40 comp rx=0 tx=0).stop 2024-05-09T03:18:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb034c640 1 -- 172.21.15.46:0/3888310430 reap_dead start 2024-05-09T03:18:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 shutdown_connections 2024-05-09T03:18:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 >> 172.21.15.46:0/3888310430 conn(0x55fd39475ac0 msgr2=0x55fd396237e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 shutdown_connections 2024-05-09T03:18:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.626+0000 7fbeb6083f40 1 -- 172.21.15.46:0/3888310430 wait complete. 2024-05-09T03:18:30.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:30.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.788+0000 7f11beea8640 1 Processor -- start 2024-05-09T03:18:30.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.788+0000 7f11beea8640 1 -- start start 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11beea8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11b80d2e90 0x7f11b80b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11beea8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b81605b0 0x7f11b80b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11beea8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11b80b4fc0 0x7f11b80d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11beea8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f11b80d3400 con 0x7f11b80b4fc0 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11beea8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f11b80d4a60 con 0x7f11b80d2e90 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11beea8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11b80d4d40 con 0x7f11b81605b0 2024-05-09T03:18:30.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11b80b4fc0 0x7f11b80d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f11b80b4fc0 0x7f11b80d38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41264/0 (socket says 172.21.15.46:41264) 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 -- 172.21.15.46:0/3945685257 learned_addr learned my addr 172.21.15.46:0/3945685257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bcc44640 1 --2- 172.21.15.46:0/3945685257 >> v2:172.21.15.88:3300/0 conn(0x7f11b80d2e90 0x7f11b80b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11b7fff640 1 --2- 172.21.15.46:0/3945685257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b81605b0 0x7f11b80b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 -- 172.21.15.46:0/3945685257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b81605b0 msgr2=0x7f11b80b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 --2- 172.21.15.46:0/3945685257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b81605b0 0x7f11b80b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 -- 172.21.15.46:0/3945685257 >> v2:172.21.15.88:3300/0 conn(0x7f11b80d2e90 msgr2=0x7f11b80b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 --2- 172.21.15.46:0/3945685257 >> v2:172.21.15.88:3300/0 conn(0x7f11b80d2e90 0x7f11b80b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.790+0000 7f11bd445640 1 -- 172.21.15.46:0/3945685257 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b80d5020 con 0x7f11b80b4fc0 2024-05-09T03:18:30.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.791+0000 7f11bd445640 1 --2- 172.21.15.46:0/3945685257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b80b4fc0 0x7f11b80d38b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009e50 tx=0x7f11b002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9296bfe5848485 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.791+0000 7f11b77fe640 1 -- 172.21.15.46:0/3945685257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b0036070 con 0x7f11b80b4fc0 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.791+0000 7f11b77fe640 1 -- 172.21.15.46:0/3945685257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b0031600 con 0x7f11b80b4fc0 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.791+0000 7f11b77fe640 1 -- 172.21.15.46:0/3945685257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00319d0 con 0x7f11b80b4fc0 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 -- 172.21.15.46:0/3945685257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b80b4fc0 msgr2=0x7f11b80d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 --2- 172.21.15.46:0/3945685257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b80b4fc0 0x7f11b80d38b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009e50 tx=0x7f11b002a6c0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 -- 172.21.15.46:0/3945685257 shutdown_connections 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 --2- 172.21.15.46:0/3945685257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b80b4fc0 0x7f11b80d38b0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 --2- 172.21.15.46:0/3945685257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b81605b0 0x7f11b80b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 --2- 172.21.15.46:0/3945685257 >> v2:172.21.15.88:3300/0 conn(0x7f11b80d2e90 0x7f11b80b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 -- 172.21.15.46:0/3945685257 >> 172.21.15.46:0/3945685257 conn(0x7f11b808d390 msgr2=0x7f11b808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 -- 172.21.15.46:0/3945685257 shutdown_connections 2024-05-09T03:18:30.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.792+0000 7f11beea8640 1 -- 172.21.15.46:0/3945685257 wait complete. 2024-05-09T03:18:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.794+0000 7f11beea8640 1 Processor -- start 2024-05-09T03:18:30.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.794+0000 7f11beea8640 1 -- start start 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11beea8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b80b4fc0 0x7f11b81f9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11beea8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80d2e90 0x7f11b80ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11beea8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 0x7f11b81e89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11beea8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11b80b4d80 con 0x7f11b81605b0 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11beea8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11b80b4c00 con 0x7f11b80d2e90 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11beea8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11b80d3df0 con 0x7f11b80b4fc0 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11bd445640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 0x7f11b81e89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11bd445640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 0x7f11b81e89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41270/0 (socket says 172.21.15.46:41270) 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.795+0000 7f11bd445640 1 -- 172.21.15.46:0/3878339002 learned_addr learned my addr 172.21.15.46:0/3878339002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bcc44640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b80b4fc0 0x7f11b81f9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11b7fff640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80d2e90 0x7f11b80ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bd445640 1 -- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b80b4fc0 msgr2=0x7f11b81f9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bd445640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b80b4fc0 0x7f11b81f9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bd445640 1 -- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80d2e90 msgr2=0x7f11b80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bd445640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80d2e90 0x7f11b80ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bd445640 1 -- 172.21.15.46:0/3878339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b80cb220 con 0x7f11b81605b0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11bd445640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 0x7f11b81e89e0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f11b0034040 tx=0x7f11b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b0036050 con 0x7f11b81605b0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b0042070 con 0x7f11b81605b0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b0009510 con 0x7f11b81605b0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b80cb500 con 0x7f11b81605b0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.796+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b80cbac0 con 0x7f11b81605b0 2024-05-09T03:18:30.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.797+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b80caf90 con 0x7f11b81605b0 2024-05-09T03:18:30.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.798+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f11b0007c70 con 0x7f11b81605b0 2024-05-09T03:18:30.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.799+0000 7f11b57fa640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11980762d0 0x7f11980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:30.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.799+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7573+0+0 (secure 0 0 0) 0x7f11b00b8be0 con 0x7f11b81605b0 2024-05-09T03:18:30.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.799+0000 7f11bcc44640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11980762d0 0x7f11980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:30.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.799+0000 7f11bcc44640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11980762d0 0x7f11980787e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f11b806f130 tx=0x7f11a8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:30.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:30.804+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11b00815f0 con 0x7f11b81605b0 2024-05-09T03:18:31.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.036+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f11b806a270 con 0x7f11b81605b0 2024-05-09T03:18:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.275+0000 7f11b57fa640 1 -- 172.21.15.46:0/3878339002 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v219) ==== 162+0+0 (secure 0 0 0) 0x7f11b00858f0 con 0x7f11b81605b0 2024-05-09T03:18:31.277 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11980762d0 msgr2=0x7f11980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11980762d0 0x7f11980787e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f11b806f130 tx=0x7f11a8009190 comp rx=0 tx=0).stop 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 msgr2=0x7f11b81e89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 0x7f11b81e89e0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f11b0034040 tx=0x7f11b0004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 shutdown_connections 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11980762d0 0x7f11980787e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11b81605b0 0x7f11b81e89e0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11b80d2e90 0x7f11b80ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 --2- 172.21.15.46:0/3878339002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11b80b4fc0 0x7f11b81f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 >> 172.21.15.46:0/3878339002 conn(0x7f11b808d390 msgr2=0x7f11b80b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:31.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 shutdown_connections 2024-05-09T03:18:31.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.278+0000 7f11beea8640 1 -- 172.21.15.46:0/3878339002 wait complete. 2024-05-09T03:18:31.289 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:31.288+0000 7fa6852bc640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-05-09T03:18:31.290 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:31.289+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:31.290 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:31.289+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:31.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:31.297 INFO:tasks.ceph.mds.c.smithi046.stderr: -25> 2024-05-09T03:18:31.288+0000 7fa6852bc640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-05-09T03:18:31.352 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:31.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.408+0000 7ff103e7e640 1 Processor -- start 2024-05-09T03:18:31.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.408+0000 7ff103e7e640 1 -- start start 2024-05-09T03:18:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff103e7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0b5210 0x7ff0fc1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff103e7e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff0fc0b7ed0 0x7ff0fc0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff103e7e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0fc0d5280 con 0x7ff0fc0d4800 2024-05-09T03:18:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff103e7e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff0fc0b5620 con 0x7ff0fc0b7ed0 2024-05-09T03:18:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff103e7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc1609a0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff101419640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0fc0d4800 0x7ff0fc0b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54652/0 (socket says 172.21.15.46:54652) 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff101419640 1 -- 172.21.15.46:0/116061926 learned_addr learned my addr 172.21.15.46:0/116061926 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff10241b640 1 --2- 172.21.15.46:0/116061926 >> v2:172.21.15.88:3300/0 conn(0x7ff0fc0b7ed0 0x7ff0fc0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff101c1a640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0b5210 0x7ff0fc1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff100c18640 1 -- 172.21.15.46:0/116061926 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091256480 0 0) 0x7ff0fc0d5280 con 0x7ff0fc0d4800 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff100c18640 1 -- 172.21.15.46:0/116061926 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0e8003520 con 0x7ff0fc0d4800 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff10241b640 1 -- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0b5210 msgr2=0x7ff0fc1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff10241b640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0b5210 0x7ff0fc1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff10241b640 1 -- 172.21.15.46:0/116061926 >> v1:172.21.15.46:6789/0 conn(0x7ff0fc0d4800 legacy=0x7ff0fc0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff10241b640 1 -- 172.21.15.46:0/116061926 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc0d5560 con 0x7ff0fc0b7ed0 2024-05-09T03:18:31.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.410+0000 7ff101c1a640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0b5210 0x7ff0fc1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:31.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.411+0000 7ff10241b640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0b7ed0 0x7ff0fc0d4c10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009d90 tx=0x7ff0f402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9d4da5703456423 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:31.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.411+0000 7ff100c18640 1 -- 172.21.15.46:0/116061926 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4031600 con 0x7ff0fc0b7ed0 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.411+0000 7ff100c18640 1 -- 172.21.15.46:0/116061926 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f40317a0 con 0x7ff0fc0b7ed0 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.411+0000 7ff100c18640 1 -- 172.21.15.46:0/116061926 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4031b50 con 0x7ff0fc0b7ed0 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 -- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0b7ed0 msgr2=0x7ff0fc0d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0b7ed0 0x7ff0fc0d4c10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009d90 tx=0x7ff0f402a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 -- 172.21.15.46:0/116061926 shutdown_connections 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0b7ed0 0x7ff0fc0d4c10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 --2- 172.21.15.46:0/116061926 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0b5210 0x7ff0fc1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 -- 172.21.15.46:0/116061926 >> 172.21.15.46:0/116061926 conn(0x7ff0fc0d29a0 msgr2=0x7ff0fc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 -- 172.21.15.46:0/116061926 shutdown_connections 2024-05-09T03:18:31.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.412+0000 7ff103e7e640 1 -- 172.21.15.46:0/116061926 wait complete. 2024-05-09T03:18:31.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.414+0000 7ff103e7e640 1 Processor -- start 2024-05-09T03:18:31.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.414+0000 7ff103e7e640 1 -- start start 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.415+0000 7ff103e7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 0x7ff0fc14ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.415+0000 7ff103e7e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0d4800 0x7ff0fc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.415+0000 7ff103e7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0cccf0 0x7ff0fc151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.415+0000 7ff103e7e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc0d0ae0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.415+0000 7ff103e7e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc160ea0 con 0x7ff0fc0d4800 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.415+0000 7ff103e7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff0fc163e60 con 0x7ff0fc0cccf0 2024-05-09T03:18:31.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 0x7ff0fc14ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 0x7ff0fc14ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41304/0 (socket says 172.21.15.46:41304) 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 -- 172.21.15.46:0/979626581 learned_addr learned my addr 172.21.15.46:0/979626581 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff10241b640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0cccf0 0x7ff0fc151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101419640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0d4800 0x7ff0fc0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 -- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0cccf0 msgr2=0x7ff0fc151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0cccf0 0x7ff0fc151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 -- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0d4800 msgr2=0x7ff0fc0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0d4800 0x7ff0fc0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 -- 172.21.15.46:0/979626581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc14f1d0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff101c1a640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 0x7ff0fc14ec90 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800e990 tx=0x7ff0f800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8017070 con 0x7ff0fc0b5210 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc0cfbd0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f800c9e0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f800cdb0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.416+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc14f8e0 con 0x7ff0fc0b5210 2024-05-09T03:18:31.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.417+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc0cd160 con 0x7ff0fc0b5210 2024-05-09T03:18:31.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.419+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff0f801a070 con 0x7ff0fc0b5210 2024-05-09T03:18:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.419+0000 7ff0f27fc640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff0e00762d0 0x7ff0e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.419+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7ff0f8098490 con 0x7ff0fc0b5210 2024-05-09T03:18:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.419+0000 7ff101419640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff0e00762d0 0x7ff0e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.420+0000 7ff101419640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff0e00762d0 0x7ff0e00787e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0ac850 tx=0x7ff0ec006aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.423+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff0f8060f70 con 0x7ff0fc0b5210 2024-05-09T03:18:31.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.652+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0fc06a270 con 0x7ff0fc0b5210 2024-05-09T03:18:31.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.653+0000 7ff0f27fc640 1 -- 172.21.15.46:0/979626581 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) ==== 95+0+4343 (secure 0 0 0) 0x7ff0f8065270 con 0x7ff0fc0b5210 2024-05-09T03:18:31.655 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:31.655 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":221,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:31.290538+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:31.655 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 221 2024-05-09T03:18:31.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff0e00762d0 msgr2=0x7ff0e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff0e00762d0 0x7ff0e00787e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0ac850 tx=0x7ff0ec006aa0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 msgr2=0x7ff0fc14ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 0x7ff0fc14ec90 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800e990 tx=0x7ff0f800ee50 comp rx=0 tx=0).stop 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 shutdown_connections 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0fc0cccf0 0x7ff0fc151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0fc0d4800 0x7ff0fc0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff0e00762d0 0x7ff0e00787e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 --2- 172.21.15.46:0/979626581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0fc0b5210 0x7ff0fc14ec90 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 >> 172.21.15.46:0/979626581 conn(0x7ff0fc0d29a0 msgr2=0x7ff0fc160220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 shutdown_connections 2024-05-09T03:18:31.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.655+0000 7ff103e7e640 1 -- 172.21.15.46:0/979626581 wait complete. 2024-05-09T03:18:31.668 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:31.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.820+0000 7f4593dce640 1 Processor -- start 2024-05-09T03:18:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.820+0000 7f4593dce640 1 -- start start 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4593dce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f458c160610 0x7f458c0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4593dce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f458c0b7d20 0x7f458c0b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4593dce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f458c0b8130 con 0x7f458c0d57a0 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4593dce640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f458c0d5bb0 con 0x7f458c160610 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4593dce640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f458c0d49e0 con 0x7f458c0b7d20 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f458c160610 0x7f458c0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f458c160610 0x7f458c0b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41308/0 (socket says 172.21.15.46:41308) 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 -- 172.21.15.46:0/445175053 learned_addr learned my addr 172.21.15.46:0/445175053 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f459236b640 1 --2- 172.21.15.46:0/445175053 >> v2:172.21.15.88:3300/0 conn(0x7f458c0b7d20 0x7f458c0b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 -- 172.21.15.46:0/445175053 >> v2:172.21.15.88:3300/0 conn(0x7f458c0b7d20 msgr2=0x7f458c0b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 --2- 172.21.15.46:0/445175053 >> v2:172.21.15.88:3300/0 conn(0x7f458c0b7d20 0x7f458c0b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 -- 172.21.15.46:0/445175053 >> v1:172.21.15.88:6789/0 conn(0x7f458c0d57a0 legacy=0x7f458c0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f4591369640 1 -- 172.21.15.46:0/445175053 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c0d4cc0 con 0x7f458c160610 2024-05-09T03:18:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.822+0000 7f459236b640 1 --2- 172.21.15.46:0/445175053 >> v2:172.21.15.88:3300/0 conn(0x7f458c0b7d20 0x7f458c0b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.823+0000 7f4591369640 1 --2- 172.21.15.46:0/445175053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c160610 0x7f458c0b77e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f458c1640f0 tx=0x7f458002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c4f9ec82d7ad93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.823+0000 7f4590b68640 1 -- 172.21.15.46:0/445175053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580037070 con 0x7f458c160610 2024-05-09T03:18:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.823+0000 7f4590b68640 1 -- 172.21.15.46:0/445175053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4580032440 con 0x7f458c160610 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.823+0000 7f4590b68640 1 -- 172.21.15.46:0/445175053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580032810 con 0x7f458c160610 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 -- 172.21.15.46:0/445175053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c160610 msgr2=0x7f458c0b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 --2- 172.21.15.46:0/445175053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c160610 0x7f458c0b77e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f458c1640f0 tx=0x7f458002a270 comp rx=0 tx=0).stop 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 -- 172.21.15.46:0/445175053 shutdown_connections 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 --2- 172.21.15.46:0/445175053 >> v2:172.21.15.88:3300/0 conn(0x7f458c0b7d20 0x7f458c0b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 --2- 172.21.15.46:0/445175053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c160610 0x7f458c0b77e0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 -- 172.21.15.46:0/445175053 >> 172.21.15.46:0/445175053 conn(0x7f458c0743e0 msgr2=0x7f458c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 -- 172.21.15.46:0/445175053 shutdown_connections 2024-05-09T03:18:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.824+0000 7f4593dce640 1 -- 172.21.15.46:0/445175053 wait complete. 2024-05-09T03:18:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.826+0000 7f4593dce640 1 Processor -- start 2024-05-09T03:18:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.826+0000 7f4593dce640 1 -- start start 2024-05-09T03:18:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4593dce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 0x7f458c15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4593dce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f458c0d57a0 0x7f458c1e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4593dce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f458c160610 0x7f458c1ea390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4593dce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f458c160a20 con 0x7f458c0b7d20 2024-05-09T03:18:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4593dce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f458c0d5280 con 0x7f458c0d57a0 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4593dce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f458c0d15b0 con 0x7f458c160610 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4591b6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 0x7f458c15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4591b6a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 0x7f458c15c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41324/0 (socket says 172.21.15.46:41324) 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4591b6a640 1 -- 172.21.15.46:0/1812573599 learned_addr learned my addr 172.21.15.46:0/1812573599 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f4591369640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f458c0d57a0 0x7f458c1e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.827+0000 7f459236b640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f458c160610 0x7f458c1ea390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4591b6a640 1 -- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f458c160610 msgr2=0x7f458c1ea390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4591b6a640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f458c160610 0x7f458c1ea390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4591b6a640 1 -- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f458c0d57a0 msgr2=0x7f458c1e9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4591b6a640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f458c0d57a0 0x7f458c1e9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4591b6a640 1 -- 172.21.15.46:0/1812573599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c15dce0 con 0x7f458c0b7d20 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4591b6a640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 0x7f458c15c130 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f457c00c730 tx=0x7f457c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c003b70 con 0x7f458c0b7d20 2024-05-09T03:18:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457c003d10 con 0x7f458c0b7d20 2024-05-09T03:18:31.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c00a9f0 con 0x7f458c0b7d20 2024-05-09T03:18:31.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458c15dfc0 con 0x7f458c0b7d20 2024-05-09T03:18:31.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.828+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458c15e580 con 0x7f458c0b7d20 2024-05-09T03:18:31.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.829+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f458c15e960 con 0x7f458c0b7d20 2024-05-09T03:18:31.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.830+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f457c00ab90 con 0x7f458c0b7d20 2024-05-09T03:18:31.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.830+0000 7f458a7fc640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4578076300 0x7f4578078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:31.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.830+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7f457c097b30 con 0x7f458c0b7d20 2024-05-09T03:18:31.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.831+0000 7f4591369640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4578076300 0x7f4578078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:31.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.831+0000 7f4591369640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4578076300 0x7f4578078810 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4580034000 tx=0x7f458002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:31.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:31.835+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f458c15e960 con 0x7f458c0b7d20 2024-05-09T03:18:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.062+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f458c06a270 con 0x7f458c0b7d20 2024-05-09T03:18:32.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.063+0000 7f458a7fc640 1 -- 172.21.15.46:0/1812573599 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) ==== 95+0+4343 (secure 0 0 0) 0x7f458c0b8130 con 0x7f458c0b7d20 2024-05-09T03:18:32.065 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:32.065 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":221,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:31.290538+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:32.065 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 221 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4578076300 msgr2=0x7f4578078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4578076300 0x7f4578078810 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4580034000 tx=0x7f458002e000 comp rx=0 tx=0).stop 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 msgr2=0x7f458c15c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 0x7f458c15c130 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f457c00c730 tx=0x7f457c00cbf0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 shutdown_connections 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f458c160610 0x7f458c1ea390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f458c0d57a0 0x7f458c1e9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4578076300 0x7f4578078810 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 --2- 172.21.15.46:0/1812573599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f458c0b7d20 0x7f458c15c130 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 >> 172.21.15.46:0/1812573599 conn(0x7f458c0743e0 msgr2=0x7f458c1e5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 shutdown_connections 2024-05-09T03:18:32.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.065+0000 7f4593dce640 1 -- 172.21.15.46:0/1812573599 wait complete. 2024-05-09T03:18:32.078 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:32.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:32.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.192+0000 7f0cf49b7640 1 Processor -- start 2024-05-09T03:18:32.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.192+0000 7f0cf49b7640 1 -- start start 2024-05-09T03:18:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.193+0000 7f0cf49b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.193+0000 7f0cf49b7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0cf00cf6b0 0x7f0cf006d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.193+0000 7f0cf49b7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cf00d2b80 con 0x7f0cf00d0480 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.193+0000 7f0cf49b7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0cf00d2e60 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.193+0000 7f0cf49b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf00d3140 con 0x7f0cf00b92e0 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0ceed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0cf00cf6b0 0x7f0cf006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0cee59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0cedd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0cf00d0480 0x7f0cf00d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:47354/0 (socket says 172.21.15.46:47354) 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0cedd9b640 1 -- 172.21.15.46:0/1474295248 learned_addr learned my addr 172.21.15.46:0/1474295248 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0ceed9d640 1 -- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 msgr2=0x7f0cf00b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0ceed9d640 1 --2- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0ceed9d640 1 -- 172.21.15.46:0/1474295248 >> v1:172.21.15.88:6789/0 conn(0x7f0cf00d0480 legacy=0x7f0cf00d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.194+0000 7f0ceed9d640 1 -- 172.21.15.46:0/1474295248 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf00d3420 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.195+0000 7f0ceed9d640 1 --2- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf006d790 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009d90 tx=0x7f0ce802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b419ef99d51dd672 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.195+0000 7f0ced59a640 1 -- 172.21.15.46:0/1474295248 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031600 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.195+0000 7f0ced59a640 1 -- 172.21.15.46:0/1474295248 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce80317a0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.195+0000 7f0ced59a640 1 -- 172.21.15.46:0/1474295248 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8031b70 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 -- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 msgr2=0x7f0cf006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf006d790 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009d90 tx=0x7f0ce802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 -- 172.21.15.46:0/1474295248 shutdown_connections 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf006d790 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/1474295248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 -- 172.21.15.46:0/1474295248 >> 172.21.15.46:0/1474295248 conn(0x7f0cf00d3c00 msgr2=0x7f0cf00d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:32.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 -- 172.21.15.46:0/1474295248 shutdown_connections 2024-05-09T03:18:32.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.196+0000 7f0cf49b7640 1 -- 172.21.15.46:0/1474295248 wait complete. 2024-05-09T03:18:32.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.198+0000 7f0cf49b7640 1 Processor -- start 2024-05-09T03:18:32.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.198+0000 7f0cf49b7640 1 -- start start 2024-05-09T03:18:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cf49b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cf49b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf01ea8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cf49b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 0x7f0cf00b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cf49b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf006e060 con 0x7f0cf00d0480 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cf49b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf006dee0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cf49b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf006e1e0 con 0x7f0cf00b92e0 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0ceed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 0x7f0cf00b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0ceed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 0x7f0cf00b3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41340/0 (socket says 172.21.15.46:41340) 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0ceed9d640 1 -- 172.21.15.46:0/3406787732 learned_addr learned my addr 172.21.15.46:0/3406787732 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cedd9b640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf01ea8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.199+0000 7f0cee59c640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cedd9b640 1 -- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 msgr2=0x7f0cf00ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cedd9b640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cedd9b640 1 -- 172.21.15.46:0/3406787732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 msgr2=0x7f0cf00b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cedd9b640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 0x7f0cf00b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cedd9b640 1 -- 172.21.15.46:0/3406787732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf01f66c0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cee59c640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0ceed9d640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 0x7f0cf00b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cedd9b640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf01ea8e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0074da0 tx=0x7f0ce400c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4017070 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce4005ce0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf01f6c60 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4007310 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.200+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf01f7220 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.201+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf006efe0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.202+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0ce401a070 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.203+0000 7f0cdeffd640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cd00762d0 0x7f0cd00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.203+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7f0ce4097460 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.203+0000 7f0cee59c640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cd00762d0 0x7f0cd00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.203+0000 7f0cee59c640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cd00762d0 0x7f0cd00787e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0071390 tx=0x7f0ce0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:32.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.207+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0cf006efe0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.443+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cf01f69f0 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.444+0000 7f0cdeffd640 1 -- 172.21.15.46:0/3406787732 <== mon.1 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+5169 (secure 0 0 0) 0x7f0ce4021020 con 0x7f0cf00cf6b0 2024-05-09T03:18:32.446 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:32.446 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":222,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9069,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2789195040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2789195040},{"type":"v1","addr":"172.21.15.46:6834","nonce":2789195040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:31.290538+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":106,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:32.446 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 222 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cd00762d0 msgr2=0x7f0cd00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cd00762d0 0x7f0cd00787e0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0071390 tx=0x7f0ce0009190 comp rx=0 tx=0).stop 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 msgr2=0x7f0cf01ea8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf01ea8e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0074da0 tx=0x7f0ce400c7a0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 shutdown_connections 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0cd00762d0 0x7f0cd00787e0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cf00d0480 0x7f0cf00b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cf00cf6b0 0x7f0cf01ea8e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 --2- 172.21.15.46:0/3406787732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0cf00b92e0 0x7f0cf00ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 >> 172.21.15.46:0/3406787732 conn(0x7f0cf00d3c00 msgr2=0x7f0cf00b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:32.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 shutdown_connections 2024-05-09T03:18:32.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.446+0000 7f0cf49b7640 1 -- 172.21.15.46:0/3406787732 wait complete. 2024-05-09T03:18:32.459 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 400.00000000' led to DAMAGED state 2024-05-09T03:18:32.459 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 100.00000000.inode' 2024-05-09T03:18:32.459 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:32.459 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:32.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:32.476 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:32.476 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:32.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:32.476 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:32.532 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:32.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.686+0000 7fc45995e640 1 Processor -- start 2024-05-09T03:18:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.686+0000 7fc45995e640 1 -- start start 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.687+0000 7fc45995e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4540d3730 0x7fc45406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.687+0000 7fc45995e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc45406e7e0 0x7fc454164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.687+0000 7fc45995e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc45409d470 con 0x7fc454164ec0 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.687+0000 7fc45995e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc45409cfe0 con 0x7fc4540d3730 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.687+0000 7fc45995e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc4540b94a0 con 0x7fc45406e7e0 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc452ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4540d3730 0x7fc45406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc4537fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc454164ec0 0x7fc4540ceba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54658/0 (socket says 172.21.15.46:54658) 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc4537fe640 1 -- 172.21.15.46:0/1255422732 learned_addr learned my addr 172.21.15.46:0/1255422732 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc4527fc640 1 --2- 172.21.15.46:0/1255422732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc45406e7e0 0x7fc454164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc452ffd640 1 -- 172.21.15.46:0/1255422732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc45406e7e0 msgr2=0x7fc454164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc452ffd640 1 --2- 172.21.15.46:0/1255422732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc45406e7e0 0x7fc454164980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc452ffd640 1 -- 172.21.15.46:0/1255422732 >> v1:172.21.15.46:6789/0 conn(0x7fc454164ec0 legacy=0x7fc4540ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc452ffd640 1 -- 172.21.15.46:0/1255422732 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540d11b0 con 0x7fc4540d3730 2024-05-09T03:18:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.688+0000 7fc451ffb640 1 -- 172.21.15.46:0/1255422732 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20296631 0 0) 0x7fc44c001110 con 0x7fc454164ec0 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc452ffd640 1 --2- 172.21.15.46:0/1255422732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4540d3730 0x7fc45406e2a0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fc444009d90 tx=0x7fc44402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e1a9f76d3678cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc451ffb640 1 -- 172.21.15.46:0/1255422732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444031600 con 0x7fc4540d3730 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc451ffb640 1 -- 172.21.15.46:0/1255422732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4440317a0 con 0x7fc4540d3730 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc451ffb640 1 -- 172.21.15.46:0/1255422732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444031b70 con 0x7fc4540d3730 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc45995e640 1 -- 172.21.15.46:0/1255422732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4540d3730 msgr2=0x7fc45406e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc45995e640 1 --2- 172.21.15.46:0/1255422732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4540d3730 0x7fc45406e2a0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fc444009d90 tx=0x7fc44402a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc45995e640 1 -- 172.21.15.46:0/1255422732 shutdown_connections 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc45995e640 1 --2- 172.21.15.46:0/1255422732 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc45406e7e0 0x7fc454164980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc45995e640 1 --2- 172.21.15.46:0/1255422732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4540d3730 0x7fc45406e2a0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.689+0000 7fc45995e640 1 -- 172.21.15.46:0/1255422732 >> 172.21.15.46:0/1255422732 conn(0x7fc4540d05f0 msgr2=0x7fc4540d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.690+0000 7fc45995e640 1 -- 172.21.15.46:0/1255422732 shutdown_connections 2024-05-09T03:18:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.690+0000 7fc45995e640 1 -- 172.21.15.46:0/1255422732 wait complete. 2024-05-09T03:18:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.691+0000 7fc45995e640 1 Processor -- start 2024-05-09T03:18:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.692+0000 7fc45995e640 1 -- start start 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc45995e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 0x7fc4540c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc45995e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4540d3730 0x7fc4540b7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc45995e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc454164ec0 0x7fc4540b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc45995e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc45408cd60 con 0x7fc45406e7e0 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc45995e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc4540b9160 con 0x7fc4540d3730 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc45995e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc45408cee0 con 0x7fc454164ec0 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 0x7fc4540c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 0x7fc4540c90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41364/0 (socket says 172.21.15.46:41364) 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 -- 172.21.15.46:0/3419159711 learned_addr learned my addr 172.21.15.46:0/3419159711 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc4527fc640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4540d3730 0x7fc4540b7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc4537fe640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc454164ec0 0x7fc4540b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 -- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc454164ec0 msgr2=0x7fc4540b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc454164ec0 0x7fc4540b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 -- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4540d3730 msgr2=0x7fc4540b7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4540d3730 0x7fc4540b7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc452ffd640 1 -- 172.21.15.46:0/3419159711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540c99f0 con 0x7fc45406e7e0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.693+0000 7fc4537fe640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc454164ec0 0x7fc4540b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.694+0000 7fc452ffd640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 0x7fc4540c90a0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fc444000f80 tx=0x7fc444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.694+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44403f070 con 0x7fc45406e7e0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.694+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc444030be0 con 0x7fc45406e7e0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.694+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4540c9cd0 con 0x7fc45406e7e0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.694+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444009450 con 0x7fc45406e7e0 2024-05-09T03:18:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.694+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4540ca290 con 0x7fc45406e7e0 2024-05-09T03:18:32.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.695+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4540b65b0 con 0x7fc45406e7e0 2024-05-09T03:18:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.696+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc444031650 con 0x7fc45406e7e0 2024-05-09T03:18:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.696+0000 7fc43bfff640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc440076300 0x7fc440078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.696+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7fc4440b8e90 con 0x7fc45406e7e0 2024-05-09T03:18:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.696+0000 7fc4527fc640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc440076300 0x7fc440078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:32.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.697+0000 7fc4527fc640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc440076300 0x7fc440078810 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc448006fd0 tx=0x7fc448008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:32.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.701+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc4540b65b0 con 0x7fc45406e7e0 2024-05-09T03:18:32.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:32.930+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc45406a270 con 0x7fc45406e7e0 2024-05-09T03:18:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.298+0000 7fc43bfff640 1 -- 172.21.15.46:0/3419159711 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v223) ==== 161+0+0 (secure 0 0 0) 0x7fc444082050 con 0x7fc45406e7e0 2024-05-09T03:18:33.300 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc440076300 msgr2=0x7fc440078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc440076300 0x7fc440078810 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc448006fd0 tx=0x7fc448008040 comp rx=0 tx=0).stop 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 msgr2=0x7fc4540c90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 0x7fc4540c90a0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fc444000f80 tx=0x7fc444004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 shutdown_connections 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc454164ec0 0x7fc4540b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4540d3730 0x7fc4540b7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc440076300 0x7fc440078810 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 --2- 172.21.15.46:0/3419159711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc45406e7e0 0x7fc4540c90a0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 >> 172.21.15.46:0/3419159711 conn(0x7fc4540d05f0 msgr2=0x7fc4540d2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 shutdown_connections 2024-05-09T03:18:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.301+0000 7fc45995e640 1 -- 172.21.15.46:0/3419159711 wait complete. 2024-05-09T03:18:33.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.429+0000 7f95f2de3640 1 Processor -- start 2024-05-09T03:18:33.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.429+0000 7f95f2de3640 1 -- start start 2024-05-09T03:18:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f2de3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f2de3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95ec144420 0x7f95ec06f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f2de3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ec144830 con 0x7f95ec0d1840 2024-05-09T03:18:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f2de3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95ec0b6140 con 0x7f95ec144420 2024-05-09T03:18:33.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f2de3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec0b5530 con 0x7f95ec0b65d0 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f0b7f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95ebfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95ec144420 0x7f95ec06f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f1380640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f95ec0d1840 0x7f95ec0d1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:47362/0 (socket says 172.21.15.46:47362) 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f1380640 1 -- 172.21.15.46:0/2527169246 learned_addr learned my addr 172.21.15.46:0/2527169246 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f0b7f640 1 -- 172.21.15.46:0/2527169246 >> v2:172.21.15.88:3300/0 conn(0x7f95ec144420 msgr2=0x7f95ec06f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f0b7f640 1 --2- 172.21.15.46:0/2527169246 >> v2:172.21.15.88:3300/0 conn(0x7f95ec144420 0x7f95ec06f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f0b7f640 1 -- 172.21.15.46:0/2527169246 >> v1:172.21.15.88:6789/0 conn(0x7f95ec0d1840 legacy=0x7f95ec0d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95eb7fe640 1 -- 172.21.15.46:0/2527169246 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199008595 0 0) 0x7f95ec144830 con 0x7f95ec0d1840 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95f0b7f640 1 -- 172.21.15.46:0/2527169246 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec0b5810 con 0x7f95ec0b65d0 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.431+0000 7f95ebfff640 1 --2- 172.21.15.46:0/2527169246 >> v2:172.21.15.88:3300/0 conn(0x7f95ec144420 0x7f95ec06f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.432+0000 7f95f0b7f640 1 --2- 172.21.15.46:0/2527169246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0b4ee0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0097a0 tx=0x7f95dc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4874f3a2ca7be564 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.432+0000 7f95eb7fe640 1 -- 172.21.15.46:0/2527169246 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc037070 con 0x7f95ec0b65d0 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.432+0000 7f95eb7fe640 1 -- 172.21.15.46:0/2527169246 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95dc032440 con 0x7f95ec0b65d0 2024-05-09T03:18:33.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.432+0000 7f95eb7fe640 1 -- 172.21.15.46:0/2527169246 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95dc032810 con 0x7f95ec0b65d0 2024-05-09T03:18:33.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 -- 172.21.15.46:0/2527169246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 msgr2=0x7f95ec0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2527169246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0b4ee0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0097a0 tx=0x7f95dc02a180 comp rx=0 tx=0).stop 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 -- 172.21.15.46:0/2527169246 shutdown_connections 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2527169246 >> v2:172.21.15.88:3300/0 conn(0x7f95ec144420 0x7f95ec06f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2527169246 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0b4ee0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 -- 172.21.15.46:0/2527169246 >> 172.21.15.46:0/2527169246 conn(0x7f95ec0b3620 msgr2=0x7f95ec0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 -- 172.21.15.46:0/2527169246 shutdown_connections 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.433+0000 7f95f2de3640 1 -- 172.21.15.46:0/2527169246 wait complete. 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.435+0000 7f95f2de3640 1 Processor -- start 2024-05-09T03:18:33.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.435+0000 7f95f2de3640 1 -- start start 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.436+0000 7f95f2de3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.436+0000 7f95f2de3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d1840 0x7f95ec15d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.436+0000 7f95f2de3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 0x7f95ec1e9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.436+0000 7f95f2de3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec06fe40 con 0x7f95ec144420 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.436+0000 7f95f2de3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95ec0b5dd0 con 0x7f95ec0d1840 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.436+0000 7f95f2de3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95ec144d50 con 0x7f95ec0b65d0 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 0x7f95ec1e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 0x7f95ec1e9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41380/0 (socket says 172.21.15.46:41380) 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 -- 172.21.15.46:0/2739167011 learned_addr learned my addr 172.21.15.46:0/2739167011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:33.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95ebfff640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d1840 0x7f95ec15d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f0b7f640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 -- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 msgr2=0x7f95ec0aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 -- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d1840 msgr2=0x7f95ec15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d1840 0x7f95ec15d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 -- 172.21.15.46:0/2739167011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ec133b40 con 0x7f95ec144420 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95ebfff640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d1840 0x7f95ec15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f1380640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 0x7f95ec1e9500 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f95e400c680 tx=0x7f95e400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4018070 con 0x7f95ec144420 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e4004d60 con 0x7f95ec144420 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ec0ac4b0 con 0x7f95ec144420 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e400a880 con 0x7f95ec144420 2024-05-09T03:18:33.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.437+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ec0aca70 con 0x7f95ec144420 2024-05-09T03:18:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.438+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ec133d10 con 0x7f95ec144420 2024-05-09T03:18:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.439+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95e400aa20 con 0x7f95ec144420 2024-05-09T03:18:33.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.440+0000 7f95e97fa640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d80762d0 0x7f95d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.440+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7f95e4097330 con 0x7f95ec144420 2024-05-09T03:18:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.440+0000 7f95f0b7f640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d80762d0 0x7f95d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:33.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.440+0000 7f95f0b7f640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d80762d0 0x7f95d80787e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f95dc034000 tx=0x7f95dc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:33.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.444+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95e405fd90 con 0x7f95ec144420 2024-05-09T03:18:33.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:33.709+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f95ec0ac220 con 0x7f95ec144420 2024-05-09T03:18:34.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.305+0000 7f95e97fa640 1 -- 172.21.15.46:0/2739167011 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v224) ==== 101+0+0 (secure 0 0 0) 0x7f95e4064090 con 0x7f95ec144420 2024-05-09T03:18:34.307 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d80762d0 msgr2=0x7f95d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d80762d0 0x7f95d80787e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f95dc034000 tx=0x7f95dc02e000 comp rx=0 tx=0).stop 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 msgr2=0x7f95ec1e9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 0x7f95ec1e9500 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f95e400c680 tx=0x7f95e400cb40 comp rx=0 tx=0).stop 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 shutdown_connections 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d80762d0 0x7f95d80787e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95ec144420 0x7f95ec1e9500 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95ec0d1840 0x7f95ec15d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 --2- 172.21.15.46:0/2739167011 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95ec0b65d0 0x7f95ec0aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 >> 172.21.15.46:0/2739167011 conn(0x7f95ec0b3620 msgr2=0x7f95ec1606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 shutdown_connections 2024-05-09T03:18:34.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.308+0000 7f95f2de3640 1 -- 172.21.15.46:0/2739167011 wait complete. 2024-05-09T03:18:34.323 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.382+0000 7fc717f66f40 1 Processor -- start 2024-05-09T03:18:34.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.382+0000 7fc717f66f40 1 -- start start 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.383+0000 7fc717f66f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.383+0000 7fc717f66f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563570924010 0x56357091cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.383+0000 7fc717f66f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5635709a96b0 0x56357091d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.383+0000 7fc717f66f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5635709224d0 con 0x56357090d7b0 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.383+0000 7fc717f66f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563570884f50 con 0x563570924010 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.383+0000 7fc717f66f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5635709278a0 con 0x5635709a96b0 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709a89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41386/0 (socket says 172.21.15.46:41386) 2024-05-09T03:18:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 -- 172.21.15.46:0/3019926863 learned_addr learned my addr 172.21.15.46:0/3019926863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563570924010 0x56357091cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc712231640 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5635709a96b0 0x56357091d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 -- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5635709a96b0 msgr2=0x56357091d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5635709a96b0 0x56357091d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 -- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563570924010 msgr2=0x56357091cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563570924010 0x56357091cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 -- 172.21.15.46:0/3019926863 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56357090c620 con 0x56357090d7b0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563570924010 0x56357091cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc711a30640 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709a89a0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fc704057040 tx=0x7fc70404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ad08b69b7c3add server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc710a2e640 1 -- 172.21.15.46:0/3019926863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704059040 con 0x56357090d7b0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc710a2e640 1 -- 172.21.15.46:0/3019926863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc704064070 con 0x56357090d7b0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.384+0000 7fc710a2e640 1 -- 172.21.15.46:0/3019926863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704054b30 con 0x56357090d7b0 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 -- 172.21.15.46:0/3019926863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 msgr2=0x5635709a89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709a89a0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fc704057040 tx=0x7fc70404e000 comp rx=0 tx=0).stop 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 -- 172.21.15.46:0/3019926863 shutdown_connections 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5635709a96b0 0x56357091d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563570924010 0x56357091cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3019926863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709a89a0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 -- 172.21.15.46:0/3019926863 >> 172.21.15.46:0/3019926863 conn(0x563570800ac0 msgr2=0x5635709212a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 -- 172.21.15.46:0/3019926863 shutdown_connections 2024-05-09T03:18:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.385+0000 7fc717f66f40 1 -- 172.21.15.46:0/3019926863 wait complete. 2024-05-09T03:18:34.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.386+0000 7fc717f66f40 1 Processor -- start 2024-05-09T03:18:34.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.386+0000 7fc717f66f40 1 -- start start 2024-05-09T03:18:34.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc717f66f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc717f66f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563570924010 0x5635708dfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc717f66f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635709a96b0 0x5635709a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc717f66f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56357090c900 con 0x56357090d7b0 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc717f66f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56357090cbe0 con 0x5635709a96b0 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc717f66f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56357091f8c0 con 0x563570924010 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc711a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc711a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41390/0 (socket says 172.21.15.46:41390) 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.387+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 learned_addr learned my addr 172.21.15.46:0/3889823728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc712231640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635709a96b0 0x5635709a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563570924010 0x5635708dfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563570924010 msgr2=0x5635708dfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563570924010 0x5635708dfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635709a96b0 msgr2=0x5635709a89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5635709a96b0 0x5635709a89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635708e0000 con 0x56357090d7b0 2024-05-09T03:18:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709b3950 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc704061040 tx=0x7fc704002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc70affd640 1 -- 172.21.15.46:0/3889823728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70406e070 con 0x56357090d7b0 2024-05-09T03:18:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc70affd640 1 -- 172.21.15.46:0/3889823728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc704054ae0 con 0x56357090d7b0 2024-05-09T03:18:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5635709ac4e0 con 0x56357090d7b0 2024-05-09T03:18:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc70affd640 1 -- 172.21.15.46:0/3889823728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704059050 con 0x56357090d7b0 2024-05-09T03:18:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.388+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635709acaa0 con 0x56357090d7b0 2024-05-09T03:18:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.390+0000 7fc70affd640 1 -- 172.21.15.46:0/3889823728 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc704071070 con 0x56357090d7b0 2024-05-09T03:18:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc70affd640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc6f4074cb0 0x7fc6f40771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc70affd640 1 -- 172.21.15.46:0/3889823728 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7fc7040db600 con 0x56357090d7b0 2024-05-09T03:18:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc6f4074cb0 0x7fc6f40771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.394 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56357091f0e0 0x563570a4b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x56357091f4f0 con 0x56357091f0e0 2024-05-09T03:18:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc712231640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56357091f0e0 0x563570a4b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.391+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc6f4074cb0 0x7fc6f40771c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc001ed0 tx=0x7fc6fc001d00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.392+0000 7fc712231640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56357091f0e0 0x563570a4b280 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.394+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v106'660 uv634 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.394+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a4b7c0 con 0x56357091f0e0 2024-05-09T03:18:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.395+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v106'661 uv661 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.395+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57290 con 0x56357091f0e0 2024-05-09T03:18:34.398 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.396+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v106'662 uv662 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.397+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a58640 con 0x56357091f0e0 2024-05-09T03:18:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.398+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v106'663 uv663 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.398+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a589e0 con 0x56357091f0e0 2024-05-09T03:18:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.399+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v106'664 uv664 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.399+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57630 con 0x56357091f0e0 2024-05-09T03:18:34.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.400+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v106'665 uv665 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.400+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57a20 con 0x56357091f0e0 2024-05-09T03:18:34.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.402+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v106'666 uv640 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.402+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a57e10 con 0x56357091f0e0 2024-05-09T03:18:34.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.403+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v106'667 uv667 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.403+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57e10 con 0x56357091f0e0 2024-05-09T03:18:34.405 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:34.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.404+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v106'668 uv668 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.404+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.405+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v106'669 uv669 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.405+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.407+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v106'670 uv670 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.407+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.408+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v106'671 uv671 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.408+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.410+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v106'672 uv647 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.410+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.411+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v106'673 uv673 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.411+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.412+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v106'674 uv674 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.414 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.412+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.414+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v106'675 uv675 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.414+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.415+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v106'676 uv676 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.415+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.417+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v106'677 uv677 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.417+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.418+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v106'678 uv678 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.418+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a58230 con 0x56357091f0e0 2024-05-09T03:18:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.419+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v106'679 uv653 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.419+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a58230 con 0x56357091f0e0 2024-05-09T03:18:34.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.421+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v106'680 uv680 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.421+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.422+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v106'681 uv681 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.424 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:34.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.422+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.423+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v106'682 uv682 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.423+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.425+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v106'683 uv683 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.425+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.426+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v106'684 uv684 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.426+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.427+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v106'685 uv659 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.427+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.429+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v106'686 uv686 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.429+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.430+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v106'687 uv687 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.432 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:34.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.430+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.431+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v106'688 uv688 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.431+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.433+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v106'689 uv689 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.433+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a57dc0 con 0x56357091f0e0 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.434+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v106'690 uv690 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.434+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563570a581b0 0x563570a5b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.434+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5b7e0 con 0x563570a581b0 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.434+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563570a581b0 0x563570a5b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.435+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563570a581b0 0x563570a5b2a0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.436+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v106'480 uv463 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.436+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a5bb80 con 0x563570a581b0 2024-05-09T03:18:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.438+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v106'481 uv481 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.438+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5bf20 con 0x563570a581b0 2024-05-09T03:18:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.439+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v106'482 uv482 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.441 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:34.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.439+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c7e0 con 0x563570a581b0 2024-05-09T03:18:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.441+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v106'483 uv483 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.441+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5cb80 con 0x563570a581b0 2024-05-09T03:18:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.442+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v106'484 uv484 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.442+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5cf20 con 0x563570a581b0 2024-05-09T03:18:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.443+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v106'485 uv468 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.443+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a5d2c0 con 0x563570a581b0 2024-05-09T03:18:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.445+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v106'486 uv486 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.445+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5d2c0 con 0x563570a581b0 2024-05-09T03:18:34.447 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.446+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v106'487 uv487 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.446+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.447+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v106'488 uv488 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.447+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.448+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v106'489 uv489 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.448+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.450+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v106'490 uv473 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.450+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.451+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v106'491 uv491 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.451+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.452+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v106'492 uv492 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.454 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:34.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.452+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.454+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v106'493 uv493 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.454+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.455+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v106'494 uv494 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.455+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.456+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v106'495 uv479 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.456+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.457+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v106'496 uv496 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.458+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5c410 con 0x563570a581b0 2024-05-09T03:18:34.460 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.459+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v106'497 uv497 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.459+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5e7d0 con 0x563570a581b0 2024-05-09T03:18:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.460+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v106'498 uv498 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.460+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5e780 con 0x563570a581b0 2024-05-09T03:18:34.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.461+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v106'499 uv499 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.461+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5d770 con 0x563570a581b0 2024-05-09T03:18:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.462+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v106'500 uv500 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a581b0 2024-05-09T03:18:34.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.462+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5d770 con 0x56357091f0e0 2024-05-09T03:18:34.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.464+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v106'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.464+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a5d770 con 0x56357091f0e0 2024-05-09T03:18:34.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.465+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v106'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.465+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5d770 con 0x56357091f0e0 2024-05-09T03:18:34.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.466+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v106'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.467 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:34.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.466+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5e780 con 0x56357091f0e0 2024-05-09T03:18:34.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.467+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v106'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.467+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5e780 con 0x56357091f0e0 2024-05-09T03:18:34.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.468+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v106'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.468+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a5d770 con 0x56357091f0e0 2024-05-09T03:18:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.469+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v106'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.469+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563570a5dc80 0x563570a600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.469+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a60600 con 0x563570a5dc80 2024-05-09T03:18:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.469+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563570a5dc80 0x563570a600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.469+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563570a5dc80 0x563570a600a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.470+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v106'98 uv97 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.470+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a60ea0 con 0x563570a5dc80 2024-05-09T03:18:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.471+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v106'99 uv99 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.471+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a61240 con 0x563570a5dc80 2024-05-09T03:18:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.472+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v106'100 uv100 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.474 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:34.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.472+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a615e0 con 0x563570a5dc80 2024-05-09T03:18:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.473+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v106'101 uv101 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.474+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a619d0 con 0x56357091f0e0 2024-05-09T03:18:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.475+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v106'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.475+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a619d0 con 0x56357091f0e0 2024-05-09T03:18:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.476+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v106'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.476+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a619d0 con 0x56357091f0e0 2024-05-09T03:18:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.477+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v106'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.479 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:34.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.477+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a629e0 con 0x56357091f0e0 2024-05-09T03:18:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.479+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v106'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.479+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a629e0 con 0x56357091f0e0 2024-05-09T03:18:34.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.480+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v106'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.480+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a619d0 con 0x56357091f0e0 2024-05-09T03:18:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.481+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v106'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.481+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563570a62050 0x563570a62460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.481+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.481+0000 7fc712231640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563570a62050 0x563570a62460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.482+0000 7fc712231640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563570a62050 0x563570a62460 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.483+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v106'238 uv232 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.483+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.484+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v106'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.484+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.487 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:34.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.485+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v106'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.485+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a65e80 con 0x563570a62050 2024-05-09T03:18:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.486+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v106'241 uv241 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.486+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a65e40 con 0x563570a62050 2024-05-09T03:18:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.487+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v106'242 uv242 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.487+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.489+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v106'243 uv243 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.489+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.490+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v106'244 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.490+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.491+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v106'245 uv245 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.491+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.492+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v106'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.494 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.492+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.493+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v106'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.494+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a64e30 con 0x563570a62050 2024-05-09T03:18:34.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.495+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v106'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.495+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563570a65730 0x563570a65b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.495+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a682a0 con 0x563570a65730 2024-05-09T03:18:34.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.495+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563570a65730 0x563570a65b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.495+0000 7fc711a30640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563570a65730 0x563570a65b40 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.497+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v106'107 uv106 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a65730 2024-05-09T03:18:34.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.497+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a682a0 con 0x563570a65730 2024-05-09T03:18:34.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.498+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v106'108 uv108 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a65730 2024-05-09T03:18:34.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.498+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68360 con 0x563570a65730 2024-05-09T03:18:34.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.499+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v106'109 uv109 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a65730 2024-05-09T03:18:34.501 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:34.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.499+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68750 con 0x563570a65730 2024-05-09T03:18:34.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.501+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v106'110 uv110 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a65730 2024-05-09T03:18:34.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.501+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68750 con 0x563570a65730 2024-05-09T03:18:34.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.502+0000 7fc711a30640 1 -- 172.21.15.46:0/3889823728 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v106'111 uv111 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc704070030 con 0x563570a65730 2024-05-09T03:18:34.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.502+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68750 con 0x56357091f0e0 2024-05-09T03:18:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.503+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v106'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.503+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a68750 con 0x56357091f0e0 2024-05-09T03:18:34.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.504+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v106'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.504+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68750 con 0x56357091f0e0 2024-05-09T03:18:34.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.505+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v106'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.507 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:34.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.505+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a69760 con 0x56357091f0e0 2024-05-09T03:18:34.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.506+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v106'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.506+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a69760 con 0x56357091f0e0 2024-05-09T03:18:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.507+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v106'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.507+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68750 con 0x56357091f0e0 2024-05-09T03:18:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.508+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v106'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x56357091f0e0 2024-05-09T03:18:34.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.508+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68750 con 0x563570a5dc80 2024-05-09T03:18:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.509+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v106'213 uv207 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.509+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a68af0 con 0x563570a5dc80 2024-05-09T03:18:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.510+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v106'214 uv214 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.511+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.512+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v106'215 uv215 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.513 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:34.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.512+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.513+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v106'216 uv216 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.513+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.514+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v106'217 uv217 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.514+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.515+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v106'218 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.515+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.516+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v106'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.516+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.517+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v106'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.518 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.517+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a68e90 con 0x563570a5dc80 2024-05-09T03:18:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.518+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v106'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.518+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bca0 con 0x563570a5dc80 2024-05-09T03:18:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.519+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v106'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.519+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563570a699c0 0x563570a6c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.519+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a699c0 2024-05-09T03:18:34.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.520+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563570a699c0 0x563570a6c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.520+0000 7fc70bfff640 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563570a699c0 0x563570a6c280 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.521+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v106'95 uv94 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a699c0 2024-05-09T03:18:34.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.521+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a699c0 2024-05-09T03:18:34.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.522+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v106'96 uv96 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a699c0 2024-05-09T03:18:34.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.522+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a699c0 2024-05-09T03:18:34.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.523+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v106'97 uv97 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a699c0 2024-05-09T03:18:34.525 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:34.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.523+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a699c0 2024-05-09T03:18:34.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.524+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v106'98 uv98 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a699c0 2024-05-09T03:18:34.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.524+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a5dc80 2024-05-09T03:18:34.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.526+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v106'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.526+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a5dc80 2024-05-09T03:18:34.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.527+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v106'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.527+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a5dc80 2024-05-09T03:18:34.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.528+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v106'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.530 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:34.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.528+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a5dc80 2024-05-09T03:18:34.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.529+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v106'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.529+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a5dc80 2024-05-09T03:18:34.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.530+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v106'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.530+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a5dc80 2024-05-09T03:18:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.531+0000 7fc70bfff640 1 -- 172.21.15.46:0/3889823728 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v106'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc6fc00d950 con 0x563570a5dc80 2024-05-09T03:18:34.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.531+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a62050 2024-05-09T03:18:34.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.533+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v106'110 uv109 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.533+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a62050 2024-05-09T03:18:34.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.534+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v106'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.534+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6bde0 con 0x563570a62050 2024-05-09T03:18:34.536 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:34.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.535+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v106'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.535+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.536+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v106'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.536+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.537+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v106'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.537+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.539+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v106'265 uv257 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.539+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.540+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v106'266 uv266 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.540+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.541+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v106'267 uv267 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.543 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:34.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.541+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.542+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v106'268 uv268 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.542+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.544+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v106'269 uv269 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.544+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.545+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v106'270 uv264 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.545+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.546+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v106'271 uv271 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.546+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.548 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:34.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.547+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v106'272 uv272 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.547+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6d7d0 con 0x563570a62050 2024-05-09T03:18:34.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.548+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v106'273 uv273 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.548+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6d7d0 con 0x563570a62050 2024-05-09T03:18:34.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.549+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v106'274 uv274 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.549+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6c7c0 con 0x563570a62050 2024-05-09T03:18:34.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.550+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v106'275 uv275 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.550+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x563570a6cee0 con 0x563570a62050 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v106'276 uv276 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc70c00aa60 con 0x563570a62050 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56357091f0e0 msgr2=0x563570a4b280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56357091f0e0 0x563570a4b280 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563570a5dc80 msgr2=0x563570a600a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x563570a5dc80 0x563570a600a0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563570a699c0 msgr2=0x563570a6c280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563570a699c0 0x563570a6c280 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563570a581b0 msgr2=0x563570a5b2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563570a581b0 0x563570a5b2a0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 reap_dead start 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563570a62050 msgr2=0x563570a62460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x563570a62050 0x563570a62460 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc712231640 1 -- 172.21.15.46:0/3889823728 reap_dead start 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563570a65730 msgr2=0x563570a65b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563570a65730 0x563570a65b40 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc6f4074cb0 msgr2=0x7fc6f40771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.551+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc6f4074cb0 0x7fc6f40771c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc001ed0 tx=0x7fc6fc001d00 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 msgr2=0x5635709b3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709b3950 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc704061040 tx=0x7fc704002fb0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 shutdown_connections 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563570a65730 0x563570a65b40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc6f4074cb0 0x7fc6f40771c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 --2- 172.21.15.46:0/3889823728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56357090d7b0 0x5635709b3950 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 >> 172.21.15.46:0/3889823728 conn(0x563570800ac0 msgr2=0x56357090b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:34.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 shutdown_connections 2024-05-09T03:18:34.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.552+0000 7fc717f66f40 1 -- 172.21.15.46:0/3889823728 wait complete. 2024-05-09T03:18:34.556 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 100.00000000.inode - 2024-05-09T03:18:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.651+0000 7f227e786f40 1 Processor -- start 2024-05-09T03:18:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.651+0000 7f227e786f40 1 -- start start 2024-05-09T03:18:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f227e786f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f227e786f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec64090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f227e786f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ec645d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f227e786f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dc2ebdbf50 con 0x55dc2ec753a0 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f227e786f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dc2ebdff90 con 0x55dc2ed09150 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f227e786f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55dc2ec7e8d0 con 0x55dc2ec7d5f0 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41402/0 (socket says 172.21.15.46:41402) 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 -- 172.21.15.46:0/800173581 learned_addr learned my addr 172.21.15.46:0/800173581 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2277a4f640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec64090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278a51640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ec645d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 -- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 msgr2=0x55dc2ec64090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec64090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 -- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 msgr2=0x55dc2ec645d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ec645d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.652+0000 7f2278250640 1 -- 172.21.15.46:0/800173581 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2ec371b0 con 0x55dc2ec753a0 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f2277a4f640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f2278a51640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ec645d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f2278250640 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfcbc0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f2268001680 tx=0x7f2268001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eef6dc49661fbbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f2276a4d640 1 -- 172.21.15.46:0/800173581 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268003650 con 0x55dc2ec753a0 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f2276a4d640 1 -- 172.21.15.46:0/800173581 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2268003810 con 0x55dc2ec753a0 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f2276a4d640 1 -- 172.21.15.46:0/800173581 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268003c00 con 0x55dc2ec753a0 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f227e786f40 1 -- 172.21.15.46:0/800173581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 msgr2=0x55dc2ecfcbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.653+0000 7f227e786f40 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfcbc0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f2268001680 tx=0x7f2268001b40 comp rx=0 tx=0).stop 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 -- 172.21.15.46:0/800173581 shutdown_connections 2024-05-09T03:18:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ec645d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 --2- 172.21.15.46:0/800173581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfcbc0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 -- 172.21.15.46:0/800173581 >> 172.21.15.46:0/800173581 conn(0x55dc2eb57ac0 msgr2=0x55dc2ec626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 -- 172.21.15.46:0/800173581 shutdown_connections 2024-05-09T03:18:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 -- 172.21.15.46:0/800173581 wait complete. 2024-05-09T03:18:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.654+0000 7f227e786f40 1 Processor -- start 2024-05-09T03:18:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.655+0000 7f227e786f40 1 -- start start 2024-05-09T03:18:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f227e786f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f227e786f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec63740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f227e786f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ed040a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f227e786f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55dc2ec75fd0 con 0x55dc2ec753a0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f227e786f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dc2ec762b0 con 0x55dc2ed09150 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f227e786f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55dc2ecfe1e0 con 0x55dc2ec7d5f0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfa9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41412/0 (socket says 172.21.15.46:41412) 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 -- 172.21.15.46:0/4187220721 learned_addr learned my addr 172.21.15.46:0/4187220721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2277a4f640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec63740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278a51640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ed040a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 -- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 msgr2=0x55dc2ec63740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55dc2ec7d5f0 0x55dc2ec63740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 -- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 msgr2=0x55dc2ed040a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dc2ed09150 0x55dc2ed040a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 -- 172.21.15.46:0/4187220721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc2ecfaf20 con 0x55dc2ec753a0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.656+0000 7f2278250640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfa9e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f2268001330 tx=0x7f2268001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.657+0000 7f2275a4b640 1 -- 172.21.15.46:0/4187220721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268025070 con 0x55dc2ec753a0 2024-05-09T03:18:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.657+0000 7f2275a4b640 1 -- 172.21.15.46:0/4187220721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2268019040 con 0x55dc2ec753a0 2024-05-09T03:18:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.657+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dc2ed07260 con 0x55dc2ec753a0 2024-05-09T03:18:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.657+0000 7f2275a4b640 1 -- 172.21.15.46:0/4187220721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22680119f0 con 0x55dc2ec753a0 2024-05-09T03:18:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.657+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc2ed07820 con 0x55dc2ec753a0 2024-05-09T03:18:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.659+0000 7f2275a4b640 1 -- 172.21.15.46:0/4187220721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2268028070 con 0x55dc2ec753a0 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.659+0000 7f2275a4b640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2264074ce0 0x7f22640771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.659+0000 7f2275a4b640 1 -- 172.21.15.46:0/4187220721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7f2268098070 con 0x55dc2ec753a0 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.659+0000 7f2277a4f640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2264074ce0 0x7f22640771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.659+0000 7f227e786f40 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc2eda3a30 0x55dc2eda3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.660+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55dc2ecfcea0 con 0x55dc2eda3a30 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.660+0000 7f2278a51640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc2eda3a30 0x55dc2eda3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.660+0000 7f2277a4f640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2264074ce0 0x7f22640771f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f226c057000 tx=0x7f226c060040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.660+0000 7f2278a51640 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc2eda3a30 0x55dc2eda3e40 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f2278a51640 1 -- 172.21.15.46:0/4187220721 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 100.00000000.inode [writefull 0~80] v106'277 uv277 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f227000f520 con 0x55dc2eda3a30 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc2eda3a30 msgr2=0x55dc2eda3e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55dc2eda3a30 0x55dc2eda3e40 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2264074ce0 msgr2=0x7f22640771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2264074ce0 0x7f22640771f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f226c057000 tx=0x7f226c060040 comp rx=0 tx=0).stop 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 msgr2=0x55dc2ecfa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 --2- 172.21.15.46:0/4187220721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55dc2ec753a0 0x55dc2ecfa9e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f2268001330 tx=0x7f2268001250 comp rx=0 tx=0).stop 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f2278a51640 1 -- 172.21.15.46:0/4187220721 reap_dead start 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 shutdown_connections 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 >> 172.21.15.46:0/4187220721 conn(0x55dc2eb57ac0 msgr2=0x55dc2ec78600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 shutdown_connections 2024-05-09T03:18:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.662+0000 7f227e786f40 1 -- 172.21.15.46:0/4187220721 wait complete. 2024-05-09T03:18:34.666 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:34.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.823+0000 7f0a8d83b640 1 Processor -- start 2024-05-09T03:18:34.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.823+0000 7f0a8d83b640 1 -- start start 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a8d83b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0a880d2e90 0x7f0a880b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a8d83b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880b4fc0 0x7f0a880b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a8d83b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a880d3400 con 0x7f0a881605b0 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a8d83b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0a880d4a60 con 0x7f0a880d2e90 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a8d83b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0a880d4d40 con 0x7f0a880b4fc0 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a867fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0a881605b0 0x7f0a880d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:54686/0 (socket says 172.21.15.46:54686) 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a867fc640 1 -- 172.21.15.46:0/2709960252 learned_addr learned my addr 172.21.15.46:0/2709960252 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a86ffd640 1 --2- 172.21.15.46:0/2709960252 >> v2:172.21.15.88:3300/0 conn(0x7f0a880d2e90 0x7f0a880b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a877fe640 1 --2- 172.21.15.46:0/2709960252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880b4fc0 0x7f0a880b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a85ffb640 1 -- 172.21.15.46:0/2709960252 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110510043 0 0) 0x7f0a880d3400 con 0x7f0a881605b0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a85ffb640 1 -- 172.21.15.46:0/2709960252 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a70003520 con 0x7f0a881605b0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a877fe640 1 -- 172.21.15.46:0/2709960252 >> v2:172.21.15.88:3300/0 conn(0x7f0a880d2e90 msgr2=0x7f0a880b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a877fe640 1 --2- 172.21.15.46:0/2709960252 >> v2:172.21.15.88:3300/0 conn(0x7f0a880d2e90 0x7f0a880b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a877fe640 1 -- 172.21.15.46:0/2709960252 >> v1:172.21.15.46:6789/0 conn(0x7f0a881605b0 legacy=0x7f0a880d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a877fe640 1 -- 172.21.15.46:0/2709960252 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a880d5020 con 0x7f0a880b4fc0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.825+0000 7f0a86ffd640 1 --2- 172.21.15.46:0/2709960252 >> v2:172.21.15.88:3300/0 conn(0x7f0a880d2e90 0x7f0a880b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.826+0000 7f0a877fe640 1 --2- 172.21.15.46:0/2709960252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880b4fc0 0x7f0a880b7ca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0a800097a0 tx=0x7f0a8002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b999d6b15f7eeba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.826+0000 7f0a85ffb640 1 -- 172.21.15.46:0/2709960252 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80037070 con 0x7f0a880b4fc0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.826+0000 7f0a85ffb640 1 -- 172.21.15.46:0/2709960252 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a80032440 con 0x7f0a880b4fc0 2024-05-09T03:18:34.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.826+0000 7f0a85ffb640 1 -- 172.21.15.46:0/2709960252 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a800327f0 con 0x7f0a880b4fc0 2024-05-09T03:18:34.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2709960252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880b4fc0 msgr2=0x7f0a880b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2709960252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880b4fc0 0x7f0a880b7ca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0a800097a0 tx=0x7f0a8002a180 comp rx=0 tx=0).stop 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2709960252 shutdown_connections 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2709960252 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880b4fc0 0x7f0a880b7ca0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2709960252 >> v2:172.21.15.88:3300/0 conn(0x7f0a880d2e90 0x7f0a880b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2709960252 >> 172.21.15.46:0/2709960252 conn(0x7f0a8808d390 msgr2=0x7f0a8808d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2709960252 shutdown_connections 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.827+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2709960252 wait complete. 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.829+0000 7f0a8d83b640 1 Processor -- start 2024-05-09T03:18:34.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.829+0000 7f0a8d83b640 1 -- start start 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.830+0000 7f0a8d83b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 0x7f0a880cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.830+0000 7f0a8d83b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880d2e90 0x7f0a88158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.830+0000 7f0a8d83b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a881605b0 0x7f0a881587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.830+0000 7f0a8d83b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a880d5540 con 0x7f0a880b4fc0 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.830+0000 7f0a8d83b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0a880b4c00 con 0x7f0a881605b0 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.830+0000 7f0a8d83b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0a88156f10 con 0x7f0a880d2e90 2024-05-09T03:18:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 0x7f0a880cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 0x7f0a880cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41416/0 (socket says 172.21.15.46:41416) 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 -- 172.21.15.46:0/2274091090 learned_addr learned my addr 172.21.15.46:0/2274091090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a877fe640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a881605b0 0x7f0a881587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a867fc640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880d2e90 0x7f0a88158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 -- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880d2e90 msgr2=0x7f0a88158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880d2e90 0x7f0a88158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 -- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a881605b0 msgr2=0x7f0a881587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a881605b0 0x7f0a881587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 -- 172.21.15.46:0/2274091090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a88158f40 con 0x7f0a880b4fc0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a86ffd640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 0x7f0a880cc360 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800efc0 tx=0x7f0a7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78018020 con 0x7f0a880b4fc0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a88159200 con 0x7f0a880b4fc0 2024-05-09T03:18:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a78014680 con 0x7f0a880b4fc0 2024-05-09T03:18:34.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78014a50 con 0x7f0a880b4fc0 2024-05-09T03:18:34.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.831+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a881597c0 con 0x7f0a880b4fc0 2024-05-09T03:18:34.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.832+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a880cc8a0 con 0x7f0a880b4fc0 2024-05-09T03:18:34.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.833+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a78005d10 con 0x7f0a880b4fc0 2024-05-09T03:18:34.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.834+0000 7f0a6ffff640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a74076300 0x7f0a74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:34.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.834+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7659+0+0 (secure 0 0 0) 0x7f0a78098a80 con 0x7f0a880b4fc0 2024-05-09T03:18:34.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.834+0000 7f0a867fc640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a74076300 0x7f0a74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:34.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.834+0000 7f0a867fc640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a74076300 0x7f0a74078810 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c005880 tx=0x7f0a7c006aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:34.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:34.838+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0a78022030 con 0x7f0a880b4fc0 2024-05-09T03:18:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:35.068+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0a8806a270 con 0x7f0a880b4fc0 2024-05-09T03:18:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:35.996+0000 7f0a6ffff640 1 -- 172.21.15.46:0/2274091090 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v225) ==== 162+0+0 (secure 0 0 0) 0x7f0a78010040 con 0x7f0a880b4fc0 2024-05-09T03:18:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a74076300 msgr2=0x7f0a74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a74076300 0x7f0a74078810 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c005880 tx=0x7f0a7c006aa0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 msgr2=0x7f0a880cc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 0x7f0a880cc360 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0a7800efc0 tx=0x7f0a7800c490 comp rx=0 tx=0).stop 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 shutdown_connections 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a881605b0 0x7f0a881587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0a880d2e90 0x7f0a88158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0a74076300 0x7f0a74078810 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 --2- 172.21.15.46:0/2274091090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a880b4fc0 0x7f0a880cc360 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 >> 172.21.15.46:0/2274091090 conn(0x7f0a8808d390 msgr2=0x7f0a8815ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 shutdown_connections 2024-05-09T03:18:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.000+0000 7f0a8d83b640 1 -- 172.21.15.46:0/2274091090 wait complete. 2024-05-09T03:18:36.011 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:36.010+0000 7efd2617e640 -1 mds.0.cache.ino(0x100) Corrupt inode 0x100: End of buffer [buffer:2] 2024-05-09T03:18:36.011 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:36.010+0000 7efd2617e640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:18:36.012 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:36.011+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:36.012 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:36.011+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:36.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:36.019 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:18:36.010+0000 7efd2617e640 -1 mds.0.cache.ino(0x100) Corrupt inode 0x100: End of buffer [buffer:2] 2024-05-09T03:18:36.019 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:18:36.010+0000 7efd2617e640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:18:36.078 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:36.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.140+0000 7faa2cfcf640 1 Processor -- start 2024-05-09T03:18:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.140+0000 7faa2cfcf640 1 -- start start 2024-05-09T03:18:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa2cfcf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa280d57a0 0x7faa28160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa2cfcf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa280b7ef0 0x7faa280b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa2cfcf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa280d49e0 con 0x7faa280b4af0 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa2cfcf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faa280d5bb0 con 0x7faa280d57a0 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa2cfcf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faa280d4cc0 con 0x7faa280b7ef0 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa2659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faa280d57a0 0x7faa28160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa1dd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faa280b4af0 0x7faa280b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46348/0 (socket says 172.21.15.46:46348) 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.141+0000 7faa1dd9b640 1 -- 172.21.15.46:0/718259897 learned_addr learned my addr 172.21.15.46:0/718259897 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa26d9d640 1 --2- 172.21.15.46:0/718259897 >> v2:172.21.15.88:3300/0 conn(0x7faa280b7ef0 0x7faa280b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa2659c640 1 -- 172.21.15.46:0/718259897 >> v2:172.21.15.88:3300/0 conn(0x7faa280b7ef0 msgr2=0x7faa280b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa2659c640 1 --2- 172.21.15.46:0/718259897 >> v2:172.21.15.88:3300/0 conn(0x7faa280b7ef0 0x7faa280b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa2659c640 1 -- 172.21.15.46:0/718259897 >> v1:172.21.15.46:6789/0 conn(0x7faa280b4af0 legacy=0x7faa280b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa2659c640 1 -- 172.21.15.46:0/718259897 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa280d4fa0 con 0x7faa280d57a0 2024-05-09T03:18:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa25d9b640 1 -- 172.21.15.46:0/718259897 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807515634 0 0) 0x7faa280d49e0 con 0x7faa280b4af0 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.142+0000 7faa2659c640 1 --2- 172.21.15.46:0/718259897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa280d57a0 0x7faa28160610 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7faa281640f0 tx=0x7faa1802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75c3795d11ceeb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa25d9b640 1 -- 172.21.15.46:0/718259897 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18037070 con 0x7faa280d57a0 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa25d9b640 1 -- 172.21.15.46:0/718259897 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa18032440 con 0x7faa280d57a0 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa25d9b640 1 -- 172.21.15.46:0/718259897 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18032810 con 0x7faa280d57a0 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 -- 172.21.15.46:0/718259897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa280d57a0 msgr2=0x7faa28160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/718259897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa280d57a0 0x7faa28160610 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7faa281640f0 tx=0x7faa1802a1a0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 -- 172.21.15.46:0/718259897 shutdown_connections 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/718259897 >> v2:172.21.15.88:3300/0 conn(0x7faa280b7ef0 0x7faa280b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/718259897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa280d57a0 0x7faa28160610 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 -- 172.21.15.46:0/718259897 >> 172.21.15.46:0/718259897 conn(0x7faa280743e0 msgr2=0x7faa280d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 -- 172.21.15.46:0/718259897 shutdown_connections 2024-05-09T03:18:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.143+0000 7faa2cfcf640 1 -- 172.21.15.46:0/718259897 wait complete. 2024-05-09T03:18:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.145+0000 7faa2cfcf640 1 Processor -- start 2024-05-09T03:18:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.146+0000 7faa2cfcf640 1 -- start start 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2cfcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa280b4af0 0x7faa280c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2cfcf640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa280d57a0 0x7faa281e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2cfcf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 0x7faa2814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2cfcf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa280d1920 con 0x7faa281e83b0 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2cfcf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa280d15b0 con 0x7faa280b4af0 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2cfcf640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faa2814da60 con 0x7faa280d57a0 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 0x7faa2814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 0x7faa2814c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58724/0 (socket says 172.21.15.46:58724) 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 -- 172.21.15.46:0/3394737508 learned_addr learned my addr 172.21.15.46:0/3394737508 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2659c640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa280b4af0 0x7faa280c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa1dd9b640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa280d57a0 0x7faa281e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 -- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa280d57a0 msgr2=0x7faa281e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa280d57a0 0x7faa281e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 -- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa280b4af0 msgr2=0x7faa280c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa280b4af0 0x7faa280c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa26d9d640 1 -- 172.21.15.46:0/3394737508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa280c9ff0 con 0x7faa281e83b0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.147+0000 7faa2659c640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa280b4af0 0x7faa280c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.148+0000 7faa26d9d640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 0x7faa2814c260 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7faa20007e60 tx=0x7faa2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.148+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20014420 con 0x7faa281e83b0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.148+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa20019070 con 0x7faa281e83b0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.148+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa281e78d0 con 0x7faa281e83b0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.148+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20014820 con 0x7faa281e83b0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.148+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2814d7a0 con 0x7faa281e83b0 2024-05-09T03:18:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.149+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa2806a270 con 0x7faa281e83b0 2024-05-09T03:18:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.150+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faa200149c0 con 0x7faa281e83b0 2024-05-09T03:18:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.150+0000 7faa1f7fe640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa080762d0 0x7faa080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.151+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7faa20098640 con 0x7faa281e83b0 2024-05-09T03:18:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.151+0000 7faa2659c640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa080762d0 0x7faa080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.151+0000 7faa2659c640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa080762d0 0x7faa080787e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7faa18034000 tx=0x7faa1802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.155+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faa200617a0 con 0x7faa281e83b0 2024-05-09T03:18:36.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.382+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa280c9d80 con 0x7faa281e83b0 2024-05-09T03:18:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.382+0000 7faa1f7fe640 1 -- 172.21.15.46:0/3394737508 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+4343 (secure 0 0 0) 0x7faa200610a0 con 0x7faa281e83b0 2024-05-09T03:18:36.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:36.384 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":227,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:36.012476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:36.384 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 227 2024-05-09T03:18:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa080762d0 msgr2=0x7faa080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa080762d0 0x7faa080787e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7faa18034000 tx=0x7faa1802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 msgr2=0x7faa2814c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 0x7faa2814c260 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7faa20007e60 tx=0x7faa2000e4b0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 shutdown_connections 2024-05-09T03:18:36.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa080762d0 0x7faa080787e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa281e83b0 0x7faa2814c260 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faa280d57a0 0x7faa281e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 --2- 172.21.15.46:0/3394737508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa280b4af0 0x7faa280c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 >> 172.21.15.46:0/3394737508 conn(0x7faa280743e0 msgr2=0x7faa281e7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 shutdown_connections 2024-05-09T03:18:36.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.384+0000 7faa2cfcf640 1 -- 172.21.15.46:0/3394737508 wait complete. 2024-05-09T03:18:36.397 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:36.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.510+0000 7f1a238b8640 1 Processor -- start 2024-05-09T03:18:36.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.510+0000 7f1a238b8640 1 -- start start 2024-05-09T03:18:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a238b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c164c70 0x7f1a1c1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a238b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a1c0d2480 0x7f1a1c0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a238b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1c160b20 con 0x7f1a1c15dfe0 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a238b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1a1c161150 con 0x7f1a1c0d2480 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a238b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1c0d15f0 con 0x7f1a1c164c70 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a20e53640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1a1c15dfe0 0x7f1a1c0d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46362/0 (socket says 172.21.15.46:46362) 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a20e53640 1 -- 172.21.15.46:0/1242689687 learned_addr learned my addr 172.21.15.46:0/1242689687 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21654640 1 --2- 172.21.15.46:0/1242689687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c164c70 0x7f1a1c1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21e55640 1 --2- 172.21.15.46:0/1242689687 >> v2:172.21.15.88:3300/0 conn(0x7f1a1c0d2480 0x7f1a1c0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a1bfff640 1 -- 172.21.15.46:0/1242689687 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868164411 0 0) 0x7f1a1c160b20 con 0x7f1a1c15dfe0 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a1bfff640 1 -- 172.21.15.46:0/1242689687 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a04003520 con 0x7f1a1c15dfe0 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21654640 1 -- 172.21.15.46:0/1242689687 >> v2:172.21.15.88:3300/0 conn(0x7f1a1c0d2480 msgr2=0x7f1a1c0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21654640 1 --2- 172.21.15.46:0/1242689687 >> v2:172.21.15.88:3300/0 conn(0x7f1a1c0d2480 0x7f1a1c0d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21654640 1 -- 172.21.15.46:0/1242689687 >> v1:172.21.15.46:6789/0 conn(0x7f1a1c15dfe0 legacy=0x7f1a1c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21654640 1 -- 172.21.15.46:0/1242689687 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c0d18d0 con 0x7f1a1c164c70 2024-05-09T03:18:36.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.512+0000 7f1a21e55640 1 --2- 172.21.15.46:0/1242689687 >> v2:172.21.15.88:3300/0 conn(0x7f1a1c0d2480 0x7f1a1c0d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.513+0000 7f1a21654640 1 --2- 172.21.15.46:0/1242689687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c164c70 0x7f1a1c1615e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009d90 tx=0x7f1a0c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4ae544dcf8abd03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a1bfff640 1 -- 172.21.15.46:0/1242689687 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c007740 con 0x7f1a1c164c70 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a1bfff640 1 -- 172.21.15.46:0/1242689687 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c0078e0 con 0x7f1a1c164c70 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a1bfff640 1 -- 172.21.15.46:0/1242689687 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c007cb0 con 0x7f1a1c164c70 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 -- 172.21.15.46:0/1242689687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c164c70 msgr2=0x7f1a1c1615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1242689687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c164c70 0x7f1a1c1615e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009d90 tx=0x7f1a0c02a660 comp rx=0 tx=0).stop 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 -- 172.21.15.46:0/1242689687 shutdown_connections 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1242689687 >> v2:172.21.15.88:3300/0 conn(0x7f1a1c0d2480 0x7f1a1c0d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1242689687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c164c70 0x7f1a1c1615e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 -- 172.21.15.46:0/1242689687 >> 172.21.15.46:0/1242689687 conn(0x7f1a1c073db0 msgr2=0x7f1a1c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:36.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 -- 172.21.15.46:0/1242689687 shutdown_connections 2024-05-09T03:18:36.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.514+0000 7f1a238b8640 1 -- 172.21.15.46:0/1242689687 wait complete. 2024-05-09T03:18:36.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.516+0000 7f1a238b8640 1 Processor -- start 2024-05-09T03:18:36.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.516+0000 7f1a238b8640 1 -- start start 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a238b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 0x7f1a1c14ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a238b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c15dfe0 0x7f1a1c163fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a238b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a1c164c70 0x7f1a1c1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a238b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c0d2dd0 con 0x7f1a1c0d2480 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a238b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c15e3f0 con 0x7f1a1c164c70 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a238b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1c0d45c0 con 0x7f1a1c15dfe0 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 0x7f1a1c14ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 0x7f1a1c14ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58736/0 (socket says 172.21.15.46:58736) 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 -- 172.21.15.46:0/1145966199 learned_addr learned my addr 172.21.15.46:0/1145966199 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21e55640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a1c164c70 0x7f1a1c1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a20e53640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c15dfe0 0x7f1a1c163fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 -- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c15dfe0 msgr2=0x7f1a1c163fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c15dfe0 0x7f1a1c163fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 -- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a1c164c70 msgr2=0x7f1a1c1f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a1c164c70 0x7f1a1c1f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 -- 172.21.15.46:0/1145966199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c164750 con 0x7f1a1c0d2480 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a20e53640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c15dfe0 0x7f1a1c163fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a21654640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 0x7f1a1c14ecd0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009d90 tx=0x7f1a0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c03f070 con 0x7f1a1c0d2480 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.518+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c031c40 con 0x7f1a1c0d2480 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.519+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c009450 con 0x7f1a1c0d2480 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.519+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1c0ce3e0 con 0x7f1a1c0d2480 2024-05-09T03:18:36.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.519+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a1c0ce9a0 con 0x7f1a1c0d2480 2024-05-09T03:18:36.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.520+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a1c08cfd0 con 0x7f1a1c0d2480 2024-05-09T03:18:36.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.521+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a0c043410 con 0x7f1a1c0d2480 2024-05-09T03:18:36.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.521+0000 7f1a19ffb640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a080762d0 0x7f1a080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.522+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7f1a0c0b9490 con 0x7f1a1c0d2480 2024-05-09T03:18:36.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.522+0000 7f1a20e53640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a080762d0 0x7f1a080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.522+0000 7f1a20e53640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a080762d0 0x7f1a080787e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c15c590 tx=0x7f1a10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.525+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1a1c08cfd0 con 0x7f1a1c0d2480 2024-05-09T03:18:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.757+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a1c06a270 con 0x7f1a1c0d2480 2024-05-09T03:18:36.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.758+0000 7f1a19ffb640 1 -- 172.21.15.46:0/1145966199 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+4343 (secure 0 0 0) 0x7f1a0c082080 con 0x7f1a1c0d2480 2024-05-09T03:18:36.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:36.760 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":227,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:36.012476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 227 2024-05-09T03:18:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a080762d0 msgr2=0x7f1a080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a080762d0 0x7f1a080787e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c15c590 tx=0x7f1a10009040 comp rx=0 tx=0).stop 2024-05-09T03:18:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 msgr2=0x7f1a1c14ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 0x7f1a1c14ecd0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009d90 tx=0x7f1a0c004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 shutdown_connections 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a1c164c70 0x7f1a1c1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1a1c15dfe0 0x7f1a1c163fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1a080762d0 0x7f1a080787e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 --2- 172.21.15.46:0/1145966199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a1c0d2480 0x7f1a1c14ecd0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 >> 172.21.15.46:0/1145966199 conn(0x7f1a1c073db0 msgr2=0x7f1a1c0cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 shutdown_connections 2024-05-09T03:18:36.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.760+0000 7f1a238b8640 1 -- 172.21.15.46:0/1145966199 wait complete. 2024-05-09T03:18:36.773 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:36.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:36.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.886+0000 7ff0a77dc640 1 Processor -- start 2024-05-09T03:18:36.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.886+0000 7ff0a77dc640 1 -- start start 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.887+0000 7ff0a77dc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff0a00b6180 0x7ff0a00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.887+0000 7ff0a77dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0a008dae0 0x7ff0a00b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.887+0000 7ff0a77dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a00b90d0 con 0x7ff0a008cfc0 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.887+0000 7ff0a77dc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a00cfb70 con 0x7ff0a008dae0 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.887+0000 7ff0a77dc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a00cf2d0 con 0x7ff0a00b6180 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a5d79640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0a008dae0 0x7ff0a00b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a4d77640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0a008cfc0 0x7ff0a008d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46366/0 (socket says 172.21.15.46:46366) 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a4d77640 1 -- 172.21.15.46:0/3180360014 learned_addr learned my addr 172.21.15.46:0/3180360014 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a5578640 1 --2- 172.21.15.46:0/3180360014 >> v2:172.21.15.88:3300/0 conn(0x7ff0a00b6180 0x7ff0a00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a5d79640 1 -- 172.21.15.46:0/3180360014 >> v2:172.21.15.88:3300/0 conn(0x7ff0a00b6180 msgr2=0x7ff0a00b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a5d79640 1 --2- 172.21.15.46:0/3180360014 >> v2:172.21.15.88:3300/0 conn(0x7ff0a00b6180 0x7ff0a00b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a5d79640 1 -- 172.21.15.46:0/3180360014 >> v1:172.21.15.46:6789/0 conn(0x7ff0a008cfc0 legacy=0x7ff0a008d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff0a5d79640 1 -- 172.21.15.46:0/3180360014 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a00cf5b0 con 0x7ff0a008dae0 2024-05-09T03:18:36.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.888+0000 7ff097fff640 1 -- 172.21.15.46:0/3180360014 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511587720 0 0) 0x7ff0a00b90d0 con 0x7ff0a008cfc0 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.889+0000 7ff0a5d79640 1 --2- 172.21.15.46:0/3180360014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008dae0 0x7ff0a00b6f80 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00746e0 tx=0x7ff09802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d451c5753b1f6a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.889+0000 7ff097fff640 1 -- 172.21.15.46:0/3180360014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098007740 con 0x7ff0a008dae0 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.889+0000 7ff097fff640 1 -- 172.21.15.46:0/3180360014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0980078e0 con 0x7ff0a008dae0 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.889+0000 7ff097fff640 1 -- 172.21.15.46:0/3180360014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098007cb0 con 0x7ff0a008dae0 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.889+0000 7ff0a77dc640 1 -- 172.21.15.46:0/3180360014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008dae0 msgr2=0x7ff0a00b6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.889+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/3180360014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008dae0 0x7ff0a00b6f80 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff0a00746e0 tx=0x7ff09802a680 comp rx=0 tx=0).stop 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.890+0000 7ff0a77dc640 1 -- 172.21.15.46:0/3180360014 shutdown_connections 2024-05-09T03:18:36.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.890+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/3180360014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008dae0 0x7ff0a00b6f80 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.890+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/3180360014 >> v2:172.21.15.88:3300/0 conn(0x7ff0a00b6180 0x7ff0a00b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.890+0000 7ff0a77dc640 1 -- 172.21.15.46:0/3180360014 >> 172.21.15.46:0/3180360014 conn(0x7ff0a00b5660 msgr2=0x7ff0a00b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.890+0000 7ff0a77dc640 1 -- 172.21.15.46:0/3180360014 shutdown_connections 2024-05-09T03:18:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.890+0000 7ff0a77dc640 1 -- 172.21.15.46:0/3180360014 wait complete. 2024-05-09T03:18:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.891+0000 7ff0a77dc640 1 Processor -- start 2024-05-09T03:18:36.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.891+0000 7ff0a77dc640 1 -- start start 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a77dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 0x7ff0a01e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a77dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0a008dae0 0x7ff0a01520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a77dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0a00b6180 0x7ff0a01607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a77dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a00b74c0 con 0x7ff0a008cfc0 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a77dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a00b6a60 con 0x7ff0a00b6180 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a77dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a00b7640 con 0x7ff0a008dae0 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 0x7ff0a01e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 0x7ff0a01e6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58744/0 (socket says 172.21.15.46:58744) 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 -- 172.21.15.46:0/112204081 learned_addr learned my addr 172.21.15.46:0/112204081 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5d79640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0a00b6180 0x7ff0a01607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a4d77640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0a008dae0 0x7ff0a01520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 -- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0a008dae0 msgr2=0x7ff0a01520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0a008dae0 0x7ff0a01520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 -- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0a00b6180 msgr2=0x7ff0a01607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0a00b6180 0x7ff0a01607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 -- 172.21.15.46:0/112204081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a01e7f90 con 0x7ff0a008cfc0 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.893+0000 7ff0a5578640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 0x7ff0a01e6aa0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00d5d0 tx=0x7ff09c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.894+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c004240 con 0x7ff0a008cfc0 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.894+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09c0043e0 con 0x7ff0a008cfc0 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.894+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a01e7410 con 0x7ff0a008cfc0 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.894+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09c003830 con 0x7ff0a008cfc0 2024-05-09T03:18:36.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.894+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a01e79d0 con 0x7ff0a008cfc0 2024-05-09T03:18:36.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.895+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a006a270 con 0x7ff0a008cfc0 2024-05-09T03:18:36.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.895+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff09c0039d0 con 0x7ff0a008cfc0 2024-05-09T03:18:36.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.896+0000 7ff095ffb640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff084076300 0x7ff084078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:36.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.896+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7ff09c098130 con 0x7ff0a008cfc0 2024-05-09T03:18:36.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.896+0000 7ff0a4d77640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff084076300 0x7ff084078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:36.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.897+0000 7ff0a4d77640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff084076300 0x7ff084078810 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff0a01f9d80 tx=0x7ff090007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:36.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:36.901+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff09c061160 con 0x7ff0a008cfc0 2024-05-09T03:18:37.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.145+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0a0072810 con 0x7ff0a008cfc0 2024-05-09T03:18:37.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.146+0000 7ff095ffb640 1 -- 172.21.15.46:0/112204081 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 228 v228) ==== 95+0+5166 (secure 0 0 0) 0x7ff09c060a60 con 0x7ff0a008cfc0 2024-05-09T03:18:37.148 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:37.148 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":228,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9132,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/935847027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":935847027},{"type":"v1","addr":"172.21.15.46:6834","nonce":935847027}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:36.012476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:37.148 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 228 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff084076300 msgr2=0x7ff084078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff084076300 0x7ff084078810 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff0a01f9d80 tx=0x7ff090007c00 comp rx=0 tx=0).stop 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 msgr2=0x7ff0a01e6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 0x7ff0a01e6aa0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00d5d0 tx=0x7ff09c00da90 comp rx=0 tx=0).stop 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 shutdown_connections 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0a00b6180 0x7ff0a01607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff0a008dae0 0x7ff0a01520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff084076300 0x7ff084078810 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 --2- 172.21.15.46:0/112204081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0a008cfc0 0x7ff0a01e6aa0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 >> 172.21.15.46:0/112204081 conn(0x7ff0a00b5660 msgr2=0x7ff0a00d0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 shutdown_connections 2024-05-09T03:18:37.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.148+0000 7ff0a77dc640 1 -- 172.21.15.46:0/112204081 wait complete. 2024-05-09T03:18:37.161 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 100.00000000.inode' led to DAMAGED state 2024-05-09T03:18:37.161 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 1.00000000.inode' 2024-05-09T03:18:37.161 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:37.161 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:37.177 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:37.177 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:37.177 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:37.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:37.178 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:37.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:37.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:37.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.387+0000 7f3ce701e640 1 Processor -- start 2024-05-09T03:18:37.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.387+0000 7f3ce701e640 1 -- start start 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.388+0000 7f3ce701e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce00cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.388+0000 7f3ce701e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ce00cf4f0 0x7f3ce00cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.388+0000 7f3ce701e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce009d470 con 0x7f3ce00d3730 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.388+0000 7f3ce701e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce009cfe0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.388+0000 7f3ce701e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00b94a0 con 0x7f3ce00ceba0 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ce00cf4f0 0x7f3ce00cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ce00cf4f0 0x7f3ce00cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58758/0 (socket says 172.21.15.46:58758) 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 -- 172.21.15.46:0/2695176573 learned_addr learned my addr 172.21.15.46:0/2695176573 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3cdffff640 1 --2- 172.21.15.46:0/2695176573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce00cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 -- 172.21.15.46:0/2695176573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 msgr2=0x7f3ce00cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:37.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 --2- 172.21.15.46:0/2695176573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce00cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 -- 172.21.15.46:0/2695176573 >> v1:172.21.15.88:6789/0 conn(0x7f3ce00d3730 legacy=0x7f3ce006e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:37.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.389+0000 7f3ce55bb640 1 -- 172.21.15.46:0/2695176573 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00d11b0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.390+0000 7f3ce55bb640 1 --2- 172.21.15.46:0/2695176573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce00cf900 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009e50 tx=0x7f3cd802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d3ecf799868093 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.390+0000 7f3cdf7fe640 1 -- 172.21.15.46:0/2695176573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8036070 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.390+0000 7f3cdf7fe640 1 -- 172.21.15.46:0/2695176573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd8031600 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.390+0000 7f3cdf7fe640 1 -- 172.21.15.46:0/2695176573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80319d0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.390+0000 7f3ce701e640 1 -- 172.21.15.46:0/2695176573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 msgr2=0x7f3ce00cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.390+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2695176573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce00cf900 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8009e50 tx=0x7f3cd802a680 comp rx=0 tx=0).stop 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.391+0000 7f3ce701e640 1 -- 172.21.15.46:0/2695176573 shutdown_connections 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.391+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2695176573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce00cf900 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.391+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2695176573 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce00cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.391+0000 7f3ce701e640 1 -- 172.21.15.46:0/2695176573 >> 172.21.15.46:0/2695176573 conn(0x7f3ce00d05f0 msgr2=0x7f3ce00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.391+0000 7f3ce701e640 1 -- 172.21.15.46:0/2695176573 shutdown_connections 2024-05-09T03:18:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.391+0000 7f3ce701e640 1 -- 172.21.15.46:0/2695176573 wait complete. 2024-05-09T03:18:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.392+0000 7f3ce701e640 1 Processor -- start 2024-05-09T03:18:37.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.392+0000 7f3ce701e640 1 -- start start 2024-05-09T03:18:37.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce701e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce009d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce701e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce01506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce701e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ce00d3730 0x7f3ce01530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce701e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00b9160 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce701e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00d1490 con 0x7f3ce00d3730 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce701e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce008cd60 con 0x7f3ce00ceba0 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce01506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce01506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58770/0 (socket says 172.21.15.46:58770) 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 -- 172.21.15.46:0/2737543473 learned_addr learned my addr 172.21.15.46:0/2737543473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce55bb640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ce00d3730 0x7f3ce01530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3ce4dba640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce009d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 -- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 msgr2=0x7f3ce009d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:37.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce009d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 -- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ce00d3730 msgr2=0x7f3ce01530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ce00d3730 0x7f3ce01530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.394+0000 7f3cdffff640 1 -- 172.21.15.46:0/2737543473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce01511c0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.395+0000 7f3cdffff640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce01506f0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00d620 tx=0x7f3ccc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.395+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00b620 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.395+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc019070 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.395+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce01514a0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.395+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00ba00 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.395+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce009f2f0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.396+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce00b65b0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.396+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3ccc00bba0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.397+0000 7f3cdd7fa640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3cc40762d0 0x7f3cc40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.397+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7f3ccc0980f0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.397+0000 7f3ce4dba640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3cc40762d0 0x7f3cc40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:37.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.398+0000 7f3ce4dba640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3cc40762d0 0x7f3cc40787e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4001720 tx=0x7f3cd4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.402+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3ce00b65b0 con 0x7f3ce00cf4f0 2024-05-09T03:18:37.489 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:18:37.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:18:37.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:37.632+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3ce006a270 con 0x7f3ce00cf4f0 2024-05-09T03:18:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.027+0000 7f3cdd7fa640 1 -- 172.21.15.46:0/2737543473 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v229) ==== 161+0+0 (secure 0 0 0) 0x7f3ccc060c00 con 0x7f3ce00cf4f0 2024-05-09T03:18:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:38.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3cc40762d0 msgr2=0x7f3cc40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:38.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3cc40762d0 0x7f3cc40787e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4001720 tx=0x7f3cd4009040 comp rx=0 tx=0).stop 2024-05-09T03:18:38.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 msgr2=0x7f3ce01506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce01506f0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00d620 tx=0x7f3ccc00dae0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 shutdown_connections 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ce00d3730 0x7f3ce01530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3cc40762d0 0x7f3cc40787e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ce00cf4f0 0x7f3ce01506f0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 --2- 172.21.15.46:0/2737543473 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3ce00ceba0 0x7f3ce009d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 >> 172.21.15.46:0/2737543473 conn(0x7f3ce00d05f0 msgr2=0x7f3ce0072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 shutdown_connections 2024-05-09T03:18:38.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.029+0000 7f3ce701e640 1 -- 172.21.15.46:0/2737543473 wait complete. 2024-05-09T03:18:38.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.163+0000 7ff2593e0640 1 Processor -- start 2024-05-09T03:18:38.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.163+0000 7ff2593e0640 1 -- start start 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2593e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2540ceba0 0x7ff2540cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2593e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2540cf4f0 0x7ff2540cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2593e0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff25409d470 con 0x7ff2540d3730 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2593e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff25409cfe0 con 0x7ff2540ceba0 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2593e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff2540b94a0 con 0x7ff2540cf4f0 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2540ceba0 0x7ff2540cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff2540ceba0 0x7ff2540cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58774/0 (socket says 172.21.15.46:58774) 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 -- 172.21.15.46:0/400824818 learned_addr learned my addr 172.21.15.46:0/400824818 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 -- 172.21.15.46:0/400824818 >> v2:172.21.15.88:3300/0 conn(0x7ff2540cf4f0 msgr2=0x7ff2540cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 --2- 172.21.15.46:0/400824818 >> v2:172.21.15.88:3300/0 conn(0x7ff2540cf4f0 0x7ff2540cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 -- 172.21.15.46:0/400824818 >> v1:172.21.15.88:6789/0 conn(0x7ff2540d3730 legacy=0x7ff25406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:38.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.165+0000 7ff2527fc640 1 -- 172.21.15.46:0/400824818 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2540d11b0 con 0x7ff2540ceba0 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.166+0000 7ff2527fc640 1 --2- 172.21.15.46:0/400824818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540ceba0 0x7ff2540cefb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff254074a20 tx=0x7ff24802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f2ea5248214f08 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.166+0000 7ff251ffb640 1 -- 172.21.15.46:0/400824818 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248037070 con 0x7ff2540ceba0 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.166+0000 7ff251ffb640 1 -- 172.21.15.46:0/400824818 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff248032440 con 0x7ff2540ceba0 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.166+0000 7ff251ffb640 1 -- 172.21.15.46:0/400824818 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248032810 con 0x7ff2540ceba0 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 -- 172.21.15.46:0/400824818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540ceba0 msgr2=0x7ff2540cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 --2- 172.21.15.46:0/400824818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540ceba0 0x7ff2540cefb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7ff254074a20 tx=0x7ff24802a140 comp rx=0 tx=0).stop 2024-05-09T03:18:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 -- 172.21.15.46:0/400824818 shutdown_connections 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 --2- 172.21.15.46:0/400824818 >> v2:172.21.15.88:3300/0 conn(0x7ff2540cf4f0 0x7ff2540cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 --2- 172.21.15.46:0/400824818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540ceba0 0x7ff2540cefb0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 -- 172.21.15.46:0/400824818 >> 172.21.15.46:0/400824818 conn(0x7ff2540d05f0 msgr2=0x7ff2540d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 -- 172.21.15.46:0/400824818 shutdown_connections 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.167+0000 7ff2593e0640 1 -- 172.21.15.46:0/400824818 wait complete. 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.169+0000 7ff2593e0640 1 Processor -- start 2024-05-09T03:18:38.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.169+0000 7ff2593e0640 1 -- start start 2024-05-09T03:18:38.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2593e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2540ceba0 0x7ff2540c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:38.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2593e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 0x7ff2540cb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2593e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2540d3730 0x7ff25415e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2593e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2540b9160 con 0x7ff2540cf4f0 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2593e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff2540d1490 con 0x7ff2540d3730 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2593e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff25408cd60 con 0x7ff2540ceba0 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 0x7ff2540cb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 0x7ff2540cb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58784/0 (socket says 172.21.15.46:58784) 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2527fc640 1 -- 172.21.15.46:0/3709487090 learned_addr learned my addr 172.21.15.46:0/3709487090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff2537fe640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2540d3730 0x7ff25415e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.170+0000 7ff252ffd640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2540ceba0 0x7ff2540c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2527fc640 1 -- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2540ceba0 msgr2=0x7ff2540c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2527fc640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2540ceba0 0x7ff2540c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2527fc640 1 -- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2540d3730 msgr2=0x7ff25415e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2527fc640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2540d3730 0x7ff25415e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2527fc640 1 -- 172.21.15.46:0/3709487090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2540cbc40 con 0x7ff2540cf4f0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2527fc640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 0x7ff2540cb700 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff24802a180 tx=0x7ff2480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248041070 con 0x7ff2540cf4f0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff24803c870 con 0x7ff2540cf4f0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2540ca120 con 0x7ff2540cf4f0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248037040 con 0x7ff2540cf4f0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.171+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2540cac20 con 0x7ff2540cf4f0 2024-05-09T03:18:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.172+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2540c9e90 con 0x7ff2540cf4f0 2024-05-09T03:18:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.173+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff248044070 con 0x7ff2540cf4f0 2024-05-09T03:18:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.173+0000 7ff23b7fe640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2400762d0 0x7ff2400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.174+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7ff2480b89f0 con 0x7ff2540cf4f0 2024-05-09T03:18:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.174+0000 7ff252ffd640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2400762d0 0x7ff2400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.174+0000 7ff252ffd640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2400762d0 0x7ff2400787e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff244009f60 tx=0x7ff2440094f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:38.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.178+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff25409d470 con 0x7ff2540cf4f0 2024-05-09T03:18:38.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:38.445+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7ff2540b65b0 con 0x7ff2540cf4f0 2024-05-09T03:18:39.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.034+0000 7ff23b7fe640 1 -- 172.21.15.46:0/3709487090 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v230) ==== 101+0+0 (secure 0 0 0) 0x7ff2540b65b0 con 0x7ff2540cf4f0 2024-05-09T03:18:39.036 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:39.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2400762d0 msgr2=0x7ff2400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2400762d0 0x7ff2400787e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff244009f60 tx=0x7ff2440094f0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 msgr2=0x7ff2540cb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 0x7ff2540cb700 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7ff24802a180 tx=0x7ff2480046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 shutdown_connections 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2540d3730 0x7ff25415e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2400762d0 0x7ff2400787e0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2540cf4f0 0x7ff2540cb700 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 --2- 172.21.15.46:0/3709487090 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2540ceba0 0x7ff2540c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 >> 172.21.15.46:0/3709487090 conn(0x7ff2540d05f0 msgr2=0x7ff2540d1f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 shutdown_connections 2024-05-09T03:18:39.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.037+0000 7ff2593e0640 1 -- 172.21.15.46:0/3709487090 wait complete. 2024-05-09T03:18:39.052 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:39.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.112+0000 7f181f174f40 1 Processor -- start 2024-05-09T03:18:39.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.112+0000 7f181f174f40 1 -- start start 2024-05-09T03:18:39.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181f174f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 0x558a72e1a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181f174f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e17a40 0x558a72e138f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181f174f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72e18390 0x558a72d72370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181f174f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558a72d8d5f0 con 0x558a72d8c5f0 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181f174f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558a72ceaf50 con 0x558a72e18390 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181f174f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558a72d8d8d0 con 0x558a72e17a40 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f1818c3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 0x558a72e1a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f1818c3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 0x558a72e1a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58794/0 (socket says 172.21.15.46:58794) 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f1818c3c640 1 -- 172.21.15.46:0/1548981428 learned_addr learned my addr 172.21.15.46:0/1548981428 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181843b640 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e17a40 0x558a72e138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.113+0000 7f181943d640 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72e18390 0x558a72d72370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1818c3c640 1 -- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e17a40 msgr2=0x558a72e138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:39.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1818c3c640 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e17a40 0x558a72e138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1818c3c640 1 -- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72e18390 msgr2=0x558a72d72370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1818c3c640 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72e18390 0x558a72d72370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1818c3c640 1 -- 172.21.15.46:0/1548981428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a72d843b0 con 0x558a72d8c5f0 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1818c3c640 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 0x558a72e1a590 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1808055040 tx=0x7f1808051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea63baa0c957d674 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1817439640 1 -- 172.21.15.46:0/1548981428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808056040 con 0x558a72d8c5f0 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1817439640 1 -- 172.21.15.46:0/1548981428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1808052980 con 0x558a72d8c5f0 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.114+0000 7f1817439640 1 -- 172.21.15.46:0/1548981428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808052d50 con 0x558a72d8c5f0 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 -- 172.21.15.46:0/1548981428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 msgr2=0x558a72e1a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 0x558a72e1a590 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f1808055040 tx=0x7f1808051000 comp rx=0 tx=0).stop 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 -- 172.21.15.46:0/1548981428 shutdown_connections 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72e18390 0x558a72d72370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e17a40 0x558a72e138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 --2- 172.21.15.46:0/1548981428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72d8c5f0 0x558a72e1a590 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 -- 172.21.15.46:0/1548981428 >> 172.21.15.46:0/1548981428 conn(0x558a72c66ac0 msgr2=0x558a72d716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:39.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 -- 172.21.15.46:0/1548981428 shutdown_connections 2024-05-09T03:18:39.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.115+0000 7f181f174f40 1 -- 172.21.15.46:0/1548981428 wait complete. 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.116+0000 7f181f174f40 1 Processor -- start 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.116+0000 7f181f174f40 1 -- start start 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181f174f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72d8c5f0 0x558a72e06460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181f174f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 0x558a72e14620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181f174f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e18390 0x558a72e14b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181f174f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558a72e0f620 con 0x558a72e17a40 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181f174f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558a72d73d10 con 0x558a72d8c5f0 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181f174f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558a72e0f070 con 0x558a72e18390 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181843b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 0x558a72e14620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181843b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 0x558a72e14620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58810/0 (socket says 172.21.15.46:58810) 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 learned_addr learned my addr 172.21.15.46:0/3053742036 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.117+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72d8c5f0 0x558a72e06460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181943d640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e18390 0x558a72e14b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e18390 msgr2=0x558a72e14b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558a72e18390 0x558a72e14b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72d8c5f0 msgr2=0x558a72e06460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558a72d8c5f0 0x558a72e06460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a72e0cc90 con 0x558a72e17a40 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 0x558a72e14620 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f180c0086d0 tx=0x7f180c0087c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f1816437640 1 -- 172.21.15.46:0/3053742036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c0138c0 con 0x558a72e17a40 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f1816437640 1 -- 172.21.15.46:0/3053742036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180c01a070 con 0x558a72e17a40 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f1816437640 1 -- 172.21.15.46:0/3053742036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c013c80 con 0x558a72e17a40 2024-05-09T03:18:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558a72e0cf70 con 0x558a72e17a40 2024-05-09T03:18:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.118+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a72e0d530 con 0x558a72e17a40 2024-05-09T03:18:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.120+0000 7f1816437640 1 -- 172.21.15.46:0/3053742036 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f180c015470 con 0x558a72e17a40 2024-05-09T03:18:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.120+0000 7f1816437640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1804074ce0 0x7f18040771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.120+0000 7f1816437640 1 -- 172.21.15.46:0/3053742036 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7f180c0974a0 con 0x558a72e17a40 2024-05-09T03:18:39.124 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.121+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1804074ce0 0x7f18040771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.121+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558a72e0a720 0x558a72e0ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.121+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72e0b070 con 0x558a72e0a720 2024-05-09T03:18:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.121+0000 7f181943d640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558a72e0a720 0x558a72e0ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.121+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1804074ce0 0x7f18040771f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1808005fd0 tx=0x7f1808052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.121+0000 7f181943d640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558a72e0a720 0x558a72e0ab30 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.123+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v107'691 uv665 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.123+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ebd7b0 con 0x558a72e0a720 2024-05-09T03:18:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.125+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v107'692 uv692 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.125+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebdb50 con 0x558a72e0a720 2024-05-09T03:18:39.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.126+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v107'693 uv693 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.128 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:39.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.126+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebef00 con 0x558a72e0a720 2024-05-09T03:18:39.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.128+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v107'694 uv694 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.128+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebf2a0 con 0x558a72e0a720 2024-05-09T03:18:39.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.129+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v107'695 uv695 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.129+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebdef0 con 0x558a72e0a720 2024-05-09T03:18:39.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.131+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v107'696 uv696 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.131+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe2e0 con 0x558a72e0a720 2024-05-09T03:18:39.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.132+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v107'697 uv671 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.132+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ebe6d0 con 0x558a72e0a720 2024-05-09T03:18:39.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.134+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v107'698 uv698 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.134+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe6d0 con 0x558a72e0a720 2024-05-09T03:18:39.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.135+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v107'699 uv699 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.137 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:39.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.135+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.136+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v107'700 uv700 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.136+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.138+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v107'701 uv701 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.138+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.139+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v107'702 uv702 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.139+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.140+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v107'703 uv678 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.141+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.142+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v107'704 uv704 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.142+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.143+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v107'705 uv705 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.145 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.143+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.145+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v107'706 uv706 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.145+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.146+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v107'707 uv707 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.146+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.147+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v107'708 uv708 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.147+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.149+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v107'709 uv709 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.149+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.150+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v107'710 uv684 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.150+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.152+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v107'711 uv711 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.152+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.153+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v107'712 uv712 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.155 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:39.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.153+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.155+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v107'713 uv713 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.155+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.156+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v107'714 uv714 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.156+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.158+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v107'715 uv715 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.158+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.159+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v107'716 uv690 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.159+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.161+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v107'717 uv717 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.161+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.164 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:39.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.162+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v107'718 uv718 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.162+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.163+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v107'719 uv719 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.163+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.165+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v107'720 uv720 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.165+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72e0a720 2024-05-09T03:18:39.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.166+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v107'721 uv721 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.166+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558a72ebea90 0x558a72ec1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.166+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ebe680 con 0x558a72ebea90 2024-05-09T03:18:39.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.166+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558a72ebea90 0x558a72ec1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.167+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558a72ebea90 0x558a72ec1b60 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.169+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v107'501 uv484 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.169+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec20a0 con 0x558a72ebea90 2024-05-09T03:18:39.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.170+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v107'502 uv502 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.170+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec2440 con 0x558a72ebea90 2024-05-09T03:18:39.173 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.171+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v107'503 uv503 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.171+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec2c90 con 0x558a72ebea90 2024-05-09T03:18:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.173+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v107'504 uv504 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.173+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec3030 con 0x558a72ebea90 2024-05-09T03:18:39.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.174+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v107'505 uv505 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.174+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec33d0 con 0x558a72ebea90 2024-05-09T03:18:39.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.175+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v107'506 uv489 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.175+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec37c0 con 0x558a72ebea90 2024-05-09T03:18:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.177+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v107'507 uv507 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.177+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec37c0 con 0x558a72ebea90 2024-05-09T03:18:39.179 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:39.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.178+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v107'508 uv508 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.178+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec3c00 con 0x558a72ebea90 2024-05-09T03:18:39.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.179+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v107'509 uv509 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.179+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec3c00 con 0x558a72ebea90 2024-05-09T03:18:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.181+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v107'510 uv510 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.181+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec3940 con 0x558a72ebea90 2024-05-09T03:18:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.182+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v107'511 uv494 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.182+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec3940 con 0x558a72ebea90 2024-05-09T03:18:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.183+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v107'512 uv512 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.183+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38f0 con 0x558a72ebea90 2024-05-09T03:18:39.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.184+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v107'513 uv513 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.186 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:39.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.184+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38f0 con 0x558a72ebea90 2024-05-09T03:18:39.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.186+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v107'514 uv514 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.186+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.187+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v107'515 uv515 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.187+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.188+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v107'516 uv500 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.188+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.189+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v107'517 uv517 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.189+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.192 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.191+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v107'518 uv518 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.191+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.192+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v107'519 uv519 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.192+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.193+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v107'520 uv520 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.193+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72ebea90 2024-05-09T03:18:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.194+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v107'521 uv521 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ebea90 2024-05-09T03:18:39.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.194+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72e0a720 2024-05-09T03:18:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.196+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v107'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.196+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72e0a720 2024-05-09T03:18:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.197+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v107'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.197+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec38a0 con 0x558a72e0a720 2024-05-09T03:18:39.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.198+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v107'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.200 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.198+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec4ca0 con 0x558a72e0a720 2024-05-09T03:18:39.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.199+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v107'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.199+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec4ca0 con 0x558a72e0a720 2024-05-09T03:18:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.200+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v107'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.200+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec3c90 con 0x558a72e0a720 2024-05-09T03:18:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.201+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v107'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.201+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558a72ec41a0 0x558a72ec65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.201+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec6bd0 con 0x558a72ec41a0 2024-05-09T03:18:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.201+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558a72ec41a0 0x558a72ec65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.201+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558a72ec41a0 0x558a72ec65c0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.204+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v107'102 uv101 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.204+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec70d0 con 0x558a72ec41a0 2024-05-09T03:18:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.205+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v107'103 uv103 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.205+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec75d0 con 0x558a72ec41a0 2024-05-09T03:18:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.206+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v107'104 uv104 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:39.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.206+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec28c0 con 0x558a72ec41a0 2024-05-09T03:18:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.207+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v107'105 uv105 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.207+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7f00 con 0x558a72e0a720 2024-05-09T03:18:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.209+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v107'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.209+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec7f00 con 0x558a72e0a720 2024-05-09T03:18:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.210+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v107'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.210+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7f00 con 0x558a72e0a720 2024-05-09T03:18:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.211+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v107'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.213 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:39.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.211+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec8f10 con 0x558a72e0a720 2024-05-09T03:18:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.212+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v107'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.212+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec8f10 con 0x558a72e0a720 2024-05-09T03:18:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.213+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v107'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.213+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7f00 con 0x558a72e0a720 2024-05-09T03:18:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.214+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v107'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.214+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558a72ec85d0 0x558a72eca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.215+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb050 con 0x558a72ec85d0 2024-05-09T03:18:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.215+0000 7f181943d640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558a72ec85d0 0x558a72eca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.215+0000 7f181943d640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558a72ec85d0 0x558a72eca9f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.217+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v107'249 uv243 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.217+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.218+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v107'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.218+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.220 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.219+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v107'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.219+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecc430 con 0x558a72ec85d0 2024-05-09T03:18:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.220+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v107'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.220+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecc430 con 0x558a72ec85d0 2024-05-09T03:18:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.221+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v107'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.221+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.222+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v107'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.222+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.223+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v107'255 uv248 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.223+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.224+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v107'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.224+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.225+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v107'257 uv257 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.227 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.225+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.227+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v107'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.227+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecb420 con 0x558a72ec85d0 2024-05-09T03:18:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.228+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v107'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.228+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558a72ecbaa0 0x558a72ecdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.228+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece5f0 con 0x558a72ecbaa0 2024-05-09T03:18:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.228+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558a72ecbaa0 0x558a72ecdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.228+0000 7f1818c3c640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558a72ecbaa0 0x558a72ecdec0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.230+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v107'112 uv111 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ecbaa0 2024-05-09T03:18:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.230+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72ecbaa0 2024-05-09T03:18:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.231+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v107'113 uv113 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ecbaa0 2024-05-09T03:18:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.231+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72ecbaa0 2024-05-09T03:18:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.232+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v107'114 uv114 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ecbaa0 2024-05-09T03:18:39.234 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.232+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72ecbaa0 2024-05-09T03:18:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.233+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v107'115 uv115 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ecbaa0 2024-05-09T03:18:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.233+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72ecbaa0 2024-05-09T03:18:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.234+0000 7f1818c3c640 1 -- 172.21.15.46:0/3053742036 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v107'116 uv116 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f18080705e0 con 0x558a72ecbaa0 2024-05-09T03:18:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.234+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72e0a720 2024-05-09T03:18:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.236+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v107'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.236+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72e0a720 2024-05-09T03:18:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.237+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v107'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.237+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72e0a720 2024-05-09T03:18:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.238+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v107'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.239 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.238+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecfa10 con 0x558a72e0a720 2024-05-09T03:18:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.239+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v107'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.239+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecf9d0 con 0x558a72e0a720 2024-05-09T03:18:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.240+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v107'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.240+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72e0a720 2024-05-09T03:18:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.241+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v107'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72e0a720 2024-05-09T03:18:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.241+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ece9c0 con 0x558a72ec41a0 2024-05-09T03:18:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.242+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v107'223 uv217 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.242+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72eced80 con 0x558a72ec41a0 2024-05-09T03:18:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.243+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v107'224 uv224 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.243+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ecf1b0 con 0x558a72ec41a0 2024-05-09T03:18:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.244+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v107'225 uv225 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.246 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.244+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7a80 con 0x558a72ec41a0 2024-05-09T03:18:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.246+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v107'226 uv226 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.246+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7a80 con 0x558a72ec41a0 2024-05-09T03:18:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.246+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v107'227 uv227 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.247+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7a80 con 0x558a72ec41a0 2024-05-09T03:18:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.247+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v107'228 uv222 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.248+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ec7a80 con 0x558a72ec41a0 2024-05-09T03:18:39.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.249+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v107'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.249+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7a80 con 0x558a72ec41a0 2024-05-09T03:18:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.250+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v107'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.251 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.250+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ec7a80 con 0x558a72ec41a0 2024-05-09T03:18:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.251+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v107'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.251+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed27a0 con 0x558a72ec41a0 2024-05-09T03:18:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.252+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v107'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.252+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558a72ed0350 0x558a72ed2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.252+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ed0350 2024-05-09T03:18:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.252+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558a72ed0350 0x558a72ed2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.252+0000 7f181843b640 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558a72ed0350 0x558a72ed2bc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.253+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v107'99 uv98 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ed0350 2024-05-09T03:18:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.253+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ed0350 2024-05-09T03:18:39.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.254+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v107'100 uv100 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ed0350 2024-05-09T03:18:39.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.254+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ed0350 2024-05-09T03:18:39.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.256+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v107'101 uv101 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ed0350 2024-05-09T03:18:39.257 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:39.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.256+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ed0350 2024-05-09T03:18:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.257+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v107'102 uv102 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ed0350 2024-05-09T03:18:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.257+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec41a0 2024-05-09T03:18:39.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.258+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v107'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.258+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec41a0 2024-05-09T03:18:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.259+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v107'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.259+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec41a0 2024-05-09T03:18:39.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.260+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v107'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.262 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:39.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.260+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec41a0 2024-05-09T03:18:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.262+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v107'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.262+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec41a0 2024-05-09T03:18:39.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.263+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v107'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.263+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec41a0 2024-05-09T03:18:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.264+0000 7f181843b640 1 -- 172.21.15.46:0/3053742036 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v107'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f180c05c220 con 0x558a72ec41a0 2024-05-09T03:18:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.264+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.265+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v107'115 uv114 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.265+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.267+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v107'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.267+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.270 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.268+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v107'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.268+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.270+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v107'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.270+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.271+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v107'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.271+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.273+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v107'278 uv277 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.273+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.274+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v107'279 uv279 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.274+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.275+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v107'280 uv280 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.277 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.275+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.276+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v107'281 uv281 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.276+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.278+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v107'282 uv282 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.278+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.279+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v107'283 uv276 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.279+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.280+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v107'284 uv284 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.280+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.281+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v107'285 uv285 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.283 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:39.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.281+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.282+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v107'286 uv286 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.283+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.284+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v107'287 uv287 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.284+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed2770 con 0x558a72ec85d0 2024-05-09T03:18:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.285+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v107'288 uv288 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.285+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x558a72ed3100 con 0x558a72ec85d0 2024-05-09T03:18:39.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v107'289 uv289 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1810009040 con 0x558a72ec85d0 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558a72e0a720 msgr2=0x558a72e0ab30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558a72e0a720 0x558a72e0ab30 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558a72ec41a0 msgr2=0x558a72ec65c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558a72ec41a0 0x558a72ec65c0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558a72ed0350 msgr2=0x558a72ed2bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558a72ed0350 0x558a72ed2bc0 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558a72ebea90 msgr2=0x558a72ec1b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558a72ebea90 0x558a72ec1b60 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558a72ec85d0 msgr2=0x558a72eca9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558a72ec85d0 0x558a72eca9f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181943d640 1 -- 172.21.15.46:0/3053742036 reap_dead start 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558a72ecbaa0 msgr2=0x558a72ecdec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558a72ecbaa0 0x558a72ecdec0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1804074ce0 msgr2=0x7f18040771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1804074ce0 0x7f18040771f0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1808005fd0 tx=0x7f1808052000 comp rx=0 tx=0).stop 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 msgr2=0x558a72e14620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 0x558a72e14620 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f180c0086d0 tx=0x7f180c0087c0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 shutdown_connections 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558a72ecbaa0 0x558a72ecdec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558a72ec85d0 0x558a72eca9f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1804074ce0 0x7f18040771f0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 --2- 172.21.15.46:0/3053742036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558a72e17a40 0x558a72e14620 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 >> 172.21.15.46:0/3053742036 conn(0x558a72c66ac0 msgr2=0x558a72e19c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 shutdown_connections 2024-05-09T03:18:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.286+0000 7f181f174f40 1 -- 172.21.15.46:0/3053742036 wait complete. 2024-05-09T03:18:39.291 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 1.00000000.inode 0 2024-05-09T03:18:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.346+0000 7f96acc75f40 1 Processor -- start 2024-05-09T03:18:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.346+0000 7f96acc75f40 1 -- start start 2024-05-09T03:18:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96acc75f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9dd4e7e0 0x562d9de0a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96acc75f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9de0ba50 0x562d9de08920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96acc75f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 0x562d9dd92390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96acc75f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562d9dd90cb0 con 0x562d9de26780 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96acc75f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562d9dcf7f50 con 0x562d9de0ba50 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96acc75f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562d9dd4ebf0 con 0x562d9dd4e7e0 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 0x562d9dd92390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a5f3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9de0ba50 0x562d9de08920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a673d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9dd4e7e0 0x562d9de0a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 0x562d9dd92390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58826/0 (socket says 172.21.15.46:58826) 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 -- 172.21.15.46:0/728238861 learned_addr learned my addr 172.21.15.46:0/728238861 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 -- 172.21.15.46:0/728238861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9dd4e7e0 msgr2=0x562d9de0a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9dd4e7e0 0x562d9de0a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 -- 172.21.15.46:0/728238861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9de0ba50 msgr2=0x562d9de08920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9de0ba50 0x562d9de08920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.347+0000 7f96a6f3e640 1 -- 172.21.15.46:0/728238861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d9de0d280 con 0x562d9de26780 2024-05-09T03:18:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96a6f3e640 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 0x562d9dd92390 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f96a000c440 tx=0x7f96a000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27d1c791dbeea40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96a4f3a640 1 -- 172.21.15.46:0/728238861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a00116b0 con 0x562d9de26780 2024-05-09T03:18:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96a4f3a640 1 -- 172.21.15.46:0/728238861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a0017070 con 0x562d9de26780 2024-05-09T03:18:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96a4f3a640 1 -- 172.21.15.46:0/728238861 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0011ad0 con 0x562d9de26780 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 -- 172.21.15.46:0/728238861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 msgr2=0x562d9dd92390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 0x562d9dd92390 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f96a000c440 tx=0x7f96a000c900 comp rx=0 tx=0).stop 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 -- 172.21.15.46:0/728238861 shutdown_connections 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de26780 0x562d9dd92390 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9de0ba50 0x562d9de08920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 --2- 172.21.15.46:0/728238861 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9dd4e7e0 0x562d9de0a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 -- 172.21.15.46:0/728238861 >> 172.21.15.46:0/728238861 conn(0x562d9dc73ac0 msgr2=0x562d9de25b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 -- 172.21.15.46:0/728238861 shutdown_connections 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.348+0000 7f96acc75f40 1 -- 172.21.15.46:0/728238861 wait complete. 2024-05-09T03:18:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.349+0000 7f96acc75f40 1 Processor -- start 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.349+0000 7f96acc75f40 1 -- start start 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9dd4e7e0 0x562d9dd52e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de0ba50 0x562d9de23e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9de26780 0x562d9dd80750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562d9de0a310 con 0x562d9de0ba50 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562d9de249b0 con 0x562d9dd4e7e0 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562d9de25f00 con 0x562d9de26780 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de0ba50 0x562d9de23e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de0ba50 0x562d9de23e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58832/0 (socket says 172.21.15.46:58832) 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 -- 172.21.15.46:0/2331351696 learned_addr learned my addr 172.21.15.46:0/2331351696 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a673d640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9dd4e7e0 0x562d9dd52e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a6f3e640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9de26780 0x562d9dd80750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 -- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9de26780 msgr2=0x562d9dd80750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562d9de26780 0x562d9dd80750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 -- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9dd4e7e0 msgr2=0x562d9dd52e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d9dd4e7e0 0x562d9dd52e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 -- 172.21.15.46:0/2331351696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d9dd96300 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96a5f3c640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de0ba50 0x562d9de23e80 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f969c00a4f0 tx=0x7f969c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96977fe640 1 -- 172.21.15.46:0/2331351696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c063030 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96977fe640 1 -- 172.21.15.46:0/2331351696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969c054c90 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562d9de26150 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.351+0000 7f96977fe640 1 -- 172.21.15.46:0/2331351696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c059040 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.352+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d9de0e530 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.353+0000 7f96977fe640 1 -- 172.21.15.46:0/2331351696 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f969c066030 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96977fe640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9688074cb0 0x7f96880771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96977fe640 1 -- 172.21.15.46:0/2331351696 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7f969c0da290 con 0x562d9de0ba50 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96a673d640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9688074cb0 0x7f96880771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96acc75f40 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562d9debc340 0x562d9debf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x562d9debc750 con 0x562d9debc340 2024-05-09T03:18:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96a6f3e640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562d9debc340 0x562d9debf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96a673d640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9688074cb0 0x7f96880771c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9698000c00 tx=0x7f9698001720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.354+0000 7f96a6f3e640 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562d9debc340 0x562d9debf420 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96a6f3e640 1 -- 172.21.15.46:0/2331351696 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 1.00000000.inode [truncate 0] v107'522 uv522 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f96a0011850 con 0x562d9debc340 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562d9debc340 msgr2=0x562d9debf420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562d9debc340 0x562d9debf420 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9688074cb0 msgr2=0x7f96880771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9688074cb0 0x7f96880771c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f9698000c00 tx=0x7f9698001720 comp rx=0 tx=0).stop 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de0ba50 msgr2=0x562d9de23e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 --2- 172.21.15.46:0/2331351696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562d9de0ba50 0x562d9de23e80 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f969c00a4f0 tx=0x7f969c002fb0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96a6f3e640 1 -- 172.21.15.46:0/2331351696 reap_dead start 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 shutdown_connections 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 >> 172.21.15.46:0/2331351696 conn(0x562d9dc73ac0 msgr2=0x562d9dd966e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 shutdown_connections 2024-05-09T03:18:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.356+0000 7f96acc75f40 1 -- 172.21.15.46:0/2331351696 wait complete. 2024-05-09T03:18:39.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:39.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.478+0000 7fe807253640 1 Processor -- start 2024-05-09T03:18:39.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.478+0000 7fe807253640 1 -- start start 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.479+0000 7fe807253640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8000b5140 0x7fe8000b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.479+0000 7fe807253640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe8000d3fc0 0x7fe8000b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.479+0000 7fe807253640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8000b80b0 con 0x7fe8000b7ca0 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe807253640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe8000d3480 con 0x7fe8000b5140 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe807253640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe8000b5550 con 0x7fe8000d3fc0 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8000b5140 0x7fe8000b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8000b5140 0x7fe8000b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58842/0 (socket says 172.21.15.46:58842) 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 -- 172.21.15.46:0/3381949037 learned_addr learned my addr 172.21.15.46:0/3381949037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe8057f0640 1 --2- 172.21.15.46:0/3381949037 >> v2:172.21.15.88:3300/0 conn(0x7fe8000d3fc0 0x7fe8000b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 -- 172.21.15.46:0/3381949037 >> v2:172.21.15.88:3300/0 conn(0x7fe8000d3fc0 msgr2=0x7fe8000b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 --2- 172.21.15.46:0/3381949037 >> v2:172.21.15.88:3300/0 conn(0x7fe8000d3fc0 0x7fe8000b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 -- 172.21.15.46:0/3381949037 >> v1:172.21.15.88:6789/0 conn(0x7fe8000b7ca0 legacy=0x7fe8000d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.480+0000 7fe804fef640 1 -- 172.21.15.46:0/3381949037 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8000d44e0 con 0x7fe8000b5140 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.481+0000 7fe804fef640 1 --2- 172.21.15.46:0/3381949037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b5140 0x7fe8000b7760 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe80008f180 tx=0x7fe7fc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9253ea943d4e2e11 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.481+0000 7fe7f77fe640 1 -- 172.21.15.46:0/3381949037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc037070 con 0x7fe8000b5140 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.481+0000 7fe7f77fe640 1 -- 172.21.15.46:0/3381949037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7fc032440 con 0x7fe8000b5140 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.481+0000 7fe7f77fe640 1 -- 172.21.15.46:0/3381949037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc032810 con 0x7fe8000b5140 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 -- 172.21.15.46:0/3381949037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b5140 msgr2=0x7fe8000b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 --2- 172.21.15.46:0/3381949037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b5140 0x7fe8000b7760 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fe80008f180 tx=0x7fe7fc02a270 comp rx=0 tx=0).stop 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 -- 172.21.15.46:0/3381949037 shutdown_connections 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 --2- 172.21.15.46:0/3381949037 >> v2:172.21.15.88:3300/0 conn(0x7fe8000d3fc0 0x7fe8000b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 --2- 172.21.15.46:0/3381949037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b5140 0x7fe8000b7760 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 -- 172.21.15.46:0/3381949037 >> 172.21.15.46:0/3381949037 conn(0x7fe80008d3f0 msgr2=0x7fe80008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 -- 172.21.15.46:0/3381949037 shutdown_connections 2024-05-09T03:18:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.482+0000 7fe807253640 1 -- 172.21.15.46:0/3381949037 wait complete. 2024-05-09T03:18:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.484+0000 7fe807253640 1 Processor -- start 2024-05-09T03:18:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.484+0000 7fe807253640 1 -- start start 2024-05-09T03:18:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe807253640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000b5140 0x7fe8001f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe807253640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 0x7fe800155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe807253640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000d3fc0 0x7fe8001f99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe807253640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000d4d80 con 0x7fe8000b7ca0 2024-05-09T03:18:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe807253640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe8000d4aa0 con 0x7fe8000d3fc0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe807253640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe8000d5580 con 0x7fe8000b5140 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 0x7fe800155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 0x7fe800155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58850/0 (socket says 172.21.15.46:58850) 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 -- 172.21.15.46:0/4161996290 learned_addr learned my addr 172.21.15.46:0/4161996290 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe8057f0640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000d3fc0 0x7fe8001f99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe804fef640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000b5140 0x7fe8001f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 -- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000b5140 msgr2=0x7fe8001f62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000b5140 0x7fe8001f62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 -- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000d3fc0 msgr2=0x7fe8001f99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000d3fc0 0x7fe8001f99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.485+0000 7fe7f7fff640 1 -- 172.21.15.46:0/4161996290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8001fa730 con 0x7fe8000b7ca0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.486+0000 7fe7f7fff640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 0x7fe800155150 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000b3f0 tx=0x7fe7f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.486+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0015020 con 0x7fe8000b7ca0 2024-05-09T03:18:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.486+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f0011830 con 0x7fe8000b7ca0 2024-05-09T03:18:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.486+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8001590d0 con 0x7fe8000b7ca0 2024-05-09T03:18:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.486+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0011c20 con 0x7fe8000b7ca0 2024-05-09T03:18:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.486+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe800159690 con 0x7fe8000b7ca0 2024-05-09T03:18:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.487+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe80006a270 con 0x7fe8000b7ca0 2024-05-09T03:18:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.488+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe7f0018020 con 0x7fe8000b7ca0 2024-05-09T03:18:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.488+0000 7fe7f57fa640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ec0762d0 0x7fe7ec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.489+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7745+0+0 (secure 0 0 0) 0x7fe7f00978d0 con 0x7fe8000b7ca0 2024-05-09T03:18:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.489+0000 7fe804fef640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ec0762d0 0x7fe7ec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.489+0000 7fe804fef640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ec0762d0 0x7fe7ec0787e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc034000 tx=0x7fe7fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:39.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.493+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe7f001f030 con 0x7fe8000b7ca0 2024-05-09T03:18:39.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.728+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe8001606a0 con 0x7fe8000b7ca0 2024-05-09T03:18:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.995+0000 7fe7f57fa640 1 -- 172.21.15.46:0/4161996290 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v231) ==== 162+0+0 (secure 0 0 0) 0x7fe7f00603e0 con 0x7fe8000b7ca0 2024-05-09T03:18:39.997 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:39.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ec0762d0 msgr2=0x7fe7ec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ec0762d0 0x7fe7ec0787e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc034000 tx=0x7fe7fc02e000 comp rx=0 tx=0).stop 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 msgr2=0x7fe800155150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 0x7fe800155150 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fe7f000b3f0 tx=0x7fe7f000b8b0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 shutdown_connections 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8000d3fc0 0x7fe8001f99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ec0762d0 0x7fe7ec0787e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8000b7ca0 0x7fe800155150 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 --2- 172.21.15.46:0/4161996290 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe8000b5140 0x7fe8001f62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 >> 172.21.15.46:0/4161996290 conn(0x7fe80008d3f0 msgr2=0x7fe800162d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 shutdown_connections 2024-05-09T03:18:40.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:39.998+0000 7fe807253640 1 -- 172.21.15.46:0/4161996290 wait complete. 2024-05-09T03:18:40.004 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:40.003+0000 7f7b9499a640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-05-09T03:18:40.004 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:40.003+0000 7f7b9499a640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:18:40.005 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:40.004+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:40.005 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:40.004+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:40.011 INFO:tasks.ceph.mds.c.smithi046.stderr: -16> 2024-05-09T03:18:40.003+0000 7f7b9499a640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-05-09T03:18:40.011 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:18:40.003+0000 7f7b9499a640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:18:40.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:40.067 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.131+0000 7f3914061640 1 Processor -- start 2024-05-09T03:18:40.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.132+0000 7f3914061640 1 -- start start 2024-05-09T03:18:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3914061640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f390c0d2960 0x7f390c0cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3914061640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c0cf950 0x7f390c0b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3914061640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390c0d2030 con 0x7f390c0b5030 2024-05-09T03:18:40.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3914061640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f390c0d0fa0 con 0x7f390c0d2960 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3914061640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f390c0cfff0 con 0x7f390c0cf950 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3911dfd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f390c0b5030 0x7f390c0d2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46400/0 (socket says 172.21.15.46:46400) 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3911dfd640 1 -- 172.21.15.46:0/2152740868 learned_addr learned my addr 172.21.15.46:0/2152740868 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f39125fe640 1 --2- 172.21.15.46:0/2152740868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c0cf950 0x7f390c0b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f39115fc640 1 --2- 172.21.15.46:0/2152740868 >> v2:172.21.15.88:3300/0 conn(0x7f390c0d2960 0x7f390c0cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3910dfb640 1 -- 172.21.15.46:0/2152740868 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520961803 0 0) 0x7f390c0d2030 con 0x7f390c0b5030 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.133+0000 7f3910dfb640 1 -- 172.21.15.46:0/2152740868 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38f8003520 con 0x7f390c0b5030 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.134+0000 7f39125fe640 1 -- 172.21.15.46:0/2152740868 >> v2:172.21.15.88:3300/0 conn(0x7f390c0d2960 msgr2=0x7f390c0cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.134+0000 7f39125fe640 1 --2- 172.21.15.46:0/2152740868 >> v2:172.21.15.88:3300/0 conn(0x7f390c0d2960 0x7f390c0cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.134+0000 7f39125fe640 1 -- 172.21.15.46:0/2152740868 >> v1:172.21.15.46:6789/0 conn(0x7f390c0b5030 legacy=0x7f390c0d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.134+0000 7f39125fe640 1 -- 172.21.15.46:0/2152740868 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c15e9c0 con 0x7f390c0cf950 2024-05-09T03:18:40.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.134+0000 7f39115fc640 1 --2- 172.21.15.46:0/2152740868 >> v2:172.21.15.88:3300/0 conn(0x7f390c0d2960 0x7f390c0cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.134+0000 7f39125fe640 1 --2- 172.21.15.46:0/2152740868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c0cf950 0x7f390c0b6b20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3904009d90 tx=0x7f390402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9613e69710c25222 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3910dfb640 1 -- 172.21.15.46:0/2152740868 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904031600 con 0x7f390c0cf950 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3910dfb640 1 -- 172.21.15.46:0/2152740868 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39040317a0 con 0x7f390c0cf950 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3910dfb640 1 -- 172.21.15.46:0/2152740868 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904031b50 con 0x7f390c0cf950 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 -- 172.21.15.46:0/2152740868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c0cf950 msgr2=0x7f390c0b6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 --2- 172.21.15.46:0/2152740868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c0cf950 0x7f390c0b6b20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3904009d90 tx=0x7f390402a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 -- 172.21.15.46:0/2152740868 shutdown_connections 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 --2- 172.21.15.46:0/2152740868 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c0cf950 0x7f390c0b6b20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 --2- 172.21.15.46:0/2152740868 >> v2:172.21.15.88:3300/0 conn(0x7f390c0d2960 0x7f390c0cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 -- 172.21.15.46:0/2152740868 >> 172.21.15.46:0/2152740868 conn(0x7f390c0d3bd0 msgr2=0x7f390c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:40.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 -- 172.21.15.46:0/2152740868 shutdown_connections 2024-05-09T03:18:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.135+0000 7f3914061640 1 -- 172.21.15.46:0/2152740868 wait complete. 2024-05-09T03:18:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.137+0000 7f3914061640 1 Processor -- start 2024-05-09T03:18:40.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.137+0000 7f3914061640 1 -- start start 2024-05-09T03:18:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.138+0000 7f3914061640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c0b5030 0x7f390c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.138+0000 7f3914061640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 0x7f390c1f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.138+0000 7f3914061640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c158b70 0x7f390c0c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.138+0000 7f3914061640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f390c0d0950 con 0x7f390c0cf950 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.138+0000 7f3914061640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f390c0cfd60 con 0x7f390c0b5030 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.138+0000 7f3914061640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f390c0d0c30 con 0x7f390c158b70 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 0x7f390c1f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 0x7f390c1f35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58886/0 (socket says 172.21.15.46:58886) 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 -- 172.21.15.46:0/4033657711 learned_addr learned my addr 172.21.15.46:0/4033657711 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39125fe640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c158b70 0x7f390c0c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f3911dfd640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c0b5030 0x7f390c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 -- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c158b70 msgr2=0x7f390c0c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c158b70 0x7f390c0c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 -- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c0b5030 msgr2=0x7f390c0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c0b5030 0x7f390c0ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 -- 172.21.15.46:0/4033657711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390c1592e0 con 0x7f390c0cf950 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f3911dfd640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c0b5030 0x7f390c0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39115fc640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 0x7f390c1f35e0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e8d0 tx=0x7f38fc00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc017070 con 0x7f390c0cf950 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc00c940 con 0x7f390c0cf950 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390c1595c0 con 0x7f390c0cf950 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00cd30 con 0x7f390c0cf950 2024-05-09T03:18:40.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.139+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f390c159b80 con 0x7f390c0cf950 2024-05-09T03:18:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.140+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f390c159050 con 0x7f390c0cf950 2024-05-09T03:18:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.141+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f38fc01a070 con 0x7f390c0cf950 2024-05-09T03:18:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.142+0000 7f39027fc640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38f0076300 0x7f38f0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.142+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7f38fc0978c0 con 0x7f390c0cf950 2024-05-09T03:18:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.142+0000 7f3911dfd640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38f0076300 0x7f38f0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.143+0000 7f3911dfd640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38f0076300 0x7f38f0078810 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f39080010e0 tx=0x7f3908009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.147+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f38fc0602f0 con 0x7f390c0cf950 2024-05-09T03:18:40.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.378+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f390c15a0a0 con 0x7f390c0cf950 2024-05-09T03:18:40.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.379+0000 7f39027fc640 1 -- 172.21.15.46:0/4033657711 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+4343 (secure 0 0 0) 0x7f38fc0645f0 con 0x7f390c0cf950 2024-05-09T03:18:40.380 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:40.381 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:40.005512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:40.381 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2024-05-09T03:18:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38f0076300 msgr2=0x7f38f0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38f0076300 0x7f38f0078810 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f39080010e0 tx=0x7f3908009040 comp rx=0 tx=0).stop 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 msgr2=0x7f390c1f35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 0x7f390c1f35e0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e8d0 tx=0x7f38fc00ed90 comp rx=0 tx=0).stop 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 shutdown_connections 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f390c158b70 0x7f390c0c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38f0076300 0x7f38f0078810 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f390c0cf950 0x7f390c1f35e0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 --2- 172.21.15.46:0/4033657711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390c0b5030 0x7f390c0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 >> 172.21.15.46:0/4033657711 conn(0x7f390c0d3bd0 msgr2=0x7f390c0b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 shutdown_connections 2024-05-09T03:18:40.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.381+0000 7f3914061640 1 -- 172.21.15.46:0/4033657711 wait complete. 2024-05-09T03:18:40.394 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:40.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:40.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.507+0000 7fb33e7eb640 1 Processor -- start 2024-05-09T03:18:40.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.507+0000 7fb33e7eb640 1 -- start start 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33e7eb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb33815e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33e7eb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3380cfc20 0x7fb3380d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33e7eb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3380ceb80 con 0x7fb33809bfe0 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33e7eb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb3380d0570 con 0x7fb3380cfc20 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33e7eb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb3380cf790 con 0x7fb3380ce5e0 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33cd88640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb3380cfc20 0x7fb3380d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb3377fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb33809bfe0 0x7fb3380afa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46412/0 (socket says 172.21.15.46:46412) 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb3377fe640 1 -- 172.21.15.46:0/448562965 learned_addr learned my addr 172.21.15.46:0/448562965 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb337fff640 1 --2- 172.21.15.46:0/448562965 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb33815e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33cd88640 1 -- 172.21.15.46:0/448562965 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 msgr2=0x7fb33815e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.508+0000 7fb33cd88640 1 --2- 172.21.15.46:0/448562965 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb33815e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb33cd88640 1 -- 172.21.15.46:0/448562965 >> v1:172.21.15.46:6789/0 conn(0x7fb33809bfe0 legacy=0x7fb3380afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb33cd88640 1 -- 172.21.15.46:0/448562965 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3380cee60 con 0x7fb3380cfc20 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb336ffd640 1 -- 172.21.15.46:0/448562965 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205578717 0 0) 0x7fb3380ceb80 con 0x7fb33809bfe0 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb33cd88640 1 --2- 172.21.15.46:0/448562965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3380cfc20 0x7fb3380d0030 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb330009d90 tx=0x7fb33002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b771d3b4463f5909 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb336ffd640 1 -- 172.21.15.46:0/448562965 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330031600 con 0x7fb3380cfc20 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb336ffd640 1 -- 172.21.15.46:0/448562965 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3300317a0 con 0x7fb3380cfc20 2024-05-09T03:18:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.509+0000 7fb336ffd640 1 -- 172.21.15.46:0/448562965 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330031b70 con 0x7fb3380cfc20 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 -- 172.21.15.46:0/448562965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3380cfc20 msgr2=0x7fb3380d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/448562965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3380cfc20 0x7fb3380d0030 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fb330009d90 tx=0x7fb33002a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 -- 172.21.15.46:0/448562965 shutdown_connections 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/448562965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3380cfc20 0x7fb3380d0030 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/448562965 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb33815e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 -- 172.21.15.46:0/448562965 >> 172.21.15.46:0/448562965 conn(0x7fb3380d2c70 msgr2=0x7fb3380d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 -- 172.21.15.46:0/448562965 shutdown_connections 2024-05-09T03:18:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.510+0000 7fb33e7eb640 1 -- 172.21.15.46:0/448562965 wait complete. 2024-05-09T03:18:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.512+0000 7fb33e7eb640 1 Processor -- start 2024-05-09T03:18:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.512+0000 7fb33e7eb640 1 -- start start 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.513+0000 7fb33e7eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 0x7fb33815a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.513+0000 7fb33e7eb640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb3381e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.513+0000 7fb33e7eb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3380cfc20 0x7fb3380cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.513+0000 7fb33e7eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3380b0130 con 0x7fb33809bfe0 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.513+0000 7fb33e7eb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3380cf420 con 0x7fb3380cfc20 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.513+0000 7fb33e7eb640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb3381f5d50 con 0x7fb3380ce5e0 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 0x7fb33815a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 0x7fb33815a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58904/0 (socket says 172.21.15.46:58904) 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 -- 172.21.15.46:0/2597854420 learned_addr learned my addr 172.21.15.46:0/2597854420 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb33cd88640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3380cfc20 0x7fb3380cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb3377fe640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb3381e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 -- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 msgr2=0x7fb3381e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb3381e6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 -- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3380cfc20 msgr2=0x7fb3380cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3380cfc20 0x7fb3380cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 -- 172.21.15.46:0/2597854420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33815a7b0 con 0x7fb33809bfe0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb33cd88640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3380cfc20 0x7fb3380cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb3377fe640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb3381e6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb337fff640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 0x7fb33815a270 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d7d0 tx=0x7fb32800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328018020 con 0x7fb33809bfe0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3280096c0 con 0x7fb33809bfe0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb328009ab0 con 0x7fb33809bfe0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.514+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3380c9160 con 0x7fb33809bfe0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.515+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3380c85e0 con 0x7fb33809bfe0 2024-05-09T03:18:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.515+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb33806a270 con 0x7fb33809bfe0 2024-05-09T03:18:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.516+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb32801b020 con 0x7fb33809bfe0 2024-05-09T03:18:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.517+0000 7fb334ff9640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3240762d0 0x7fb3240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.517+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7fb328098510 con 0x7fb33809bfe0 2024-05-09T03:18:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.517+0000 7fb3377fe640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3240762d0 0x7fb3240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.518+0000 7fb3377fe640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3240762d0 0x7fb3240787e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb32c000f60 tx=0x7fb32c007cc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.521+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb328022020 con 0x7fb33809bfe0 2024-05-09T03:18:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.749+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb33815e660 con 0x7fb33809bfe0 2024-05-09T03:18:40.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.750+0000 7fb334ff9640 1 -- 172.21.15.46:0/2597854420 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+4343 (secure 0 0 0) 0x7fb3280614f0 con 0x7fb33809bfe0 2024-05-09T03:18:40.752 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:40.752 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":232,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:40.005512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:40.752 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 232 2024-05-09T03:18:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3240762d0 msgr2=0x7fb3240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3240762d0 0x7fb3240787e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb32c000f60 tx=0x7fb32c007cc0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 msgr2=0x7fb33815a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 0x7fb33815a270 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fb32800d7d0 tx=0x7fb32800dc90 comp rx=0 tx=0).stop 2024-05-09T03:18:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 shutdown_connections 2024-05-09T03:18:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3380cfc20 0x7fb3380cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb3380ce5e0 0x7fb3381e6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb3240762d0 0x7fb3240787e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 --2- 172.21.15.46:0/2597854420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb33809bfe0 0x7fb33815a270 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 >> 172.21.15.46:0/2597854420 conn(0x7fb3380d2c70 msgr2=0x7fb3380b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 shutdown_connections 2024-05-09T03:18:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.752+0000 7fb33e7eb640 1 -- 172.21.15.46:0/2597854420 wait complete. 2024-05-09T03:18:40.765 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:40.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.878+0000 7f407f11f640 1 Processor -- start 2024-05-09T03:18:40.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.878+0000 7f407f11f640 1 -- start start 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407f11f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407f11f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40780d1c80 0x7f40780cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407f11f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40780d2090 0x7f40780cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407f11f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f40780b5340 con 0x7f40780d1c80 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407f11f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f40780d4900 con 0x7f40780d2090 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407f11f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f40780b4fd0 con 0x7f407808cd60 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40780d1c80 0x7f40780cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40780d1c80 0x7f40780cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58906/0 (socket says 172.21.15.46:58906) 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 -- 172.21.15.46:0/930703562 learned_addr learned my addr 172.21.15.46:0/930703562 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407cebb640 1 --2- 172.21.15.46:0/930703562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f407d6bc640 1 --2- 172.21.15.46:0/930703562 >> v2:172.21.15.88:3300/0 conn(0x7f40780d2090 0x7f40780cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 -- 172.21.15.46:0/930703562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 msgr2=0x7f40780d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 --2- 172.21.15.46:0/930703562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 -- 172.21.15.46:0/930703562 >> v2:172.21.15.88:3300/0 conn(0x7f40780d2090 msgr2=0x7f40780cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 --2- 172.21.15.46:0/930703562 >> v2:172.21.15.88:3300/0 conn(0x7f40780d2090 0x7f40780cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.880+0000 7f4077fff640 1 -- 172.21.15.46:0/930703562 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40780d2c00 con 0x7f40780d1c80 2024-05-09T03:18:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.881+0000 7f4077fff640 1 --2- 172.21.15.46:0/930703562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f40780cf2b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f40780d3940 tx=0x7f406c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b4f2a2607a6109 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.881+0000 7f40777fe640 1 -- 172.21.15.46:0/930703562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c037070 con 0x7f40780d1c80 2024-05-09T03:18:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.881+0000 7f40777fe640 1 -- 172.21.15.46:0/930703562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c032440 con 0x7f40780d1c80 2024-05-09T03:18:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.881+0000 7f40777fe640 1 -- 172.21.15.46:0/930703562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c032810 con 0x7f40780d1c80 2024-05-09T03:18:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 -- 172.21.15.46:0/930703562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 msgr2=0x7f40780cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 --2- 172.21.15.46:0/930703562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f40780cf2b0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f40780d3940 tx=0x7f406c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 -- 172.21.15.46:0/930703562 shutdown_connections 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 --2- 172.21.15.46:0/930703562 >> v2:172.21.15.88:3300/0 conn(0x7f40780d2090 0x7f40780cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 --2- 172.21.15.46:0/930703562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f40780cf2b0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 --2- 172.21.15.46:0/930703562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 -- 172.21.15.46:0/930703562 >> 172.21.15.46:0/930703562 conn(0x7f40780d3c00 msgr2=0x7f40780d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 -- 172.21.15.46:0/930703562 shutdown_connections 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.882+0000 7f407f11f640 1 -- 172.21.15.46:0/930703562 wait complete. 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.884+0000 7f407f11f640 1 Processor -- start 2024-05-09T03:18:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.884+0000 7f407f11f640 1 -- start start 2024-05-09T03:18:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407f11f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407f11f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f4078160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407f11f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780d2090 0x7f40781f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407f11f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40780b92b0 con 0x7f40780d1c80 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407f11f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40780b9130 con 0x7f40780d2090 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407f11f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4078162310 con 0x7f407808cd60 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f4077fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f4078160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f4077fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f4078160640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58916/0 (socket says 172.21.15.46:58916) 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f4077fff640 1 -- 172.21.15.46:0/691198632 learned_addr learned my addr 172.21.15.46:0/691198632 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407cebb640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.885+0000 7f407d6bc640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780d2090 0x7f40781f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f4077fff640 1 -- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 msgr2=0x7f40780b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f4077fff640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f4077fff640 1 -- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780d2090 msgr2=0x7f40781f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f4077fff640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780d2090 0x7f40781f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f4077fff640 1 -- 172.21.15.46:0/691198632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407815c6e0 con 0x7f40780d1c80 2024-05-09T03:18:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f4077fff640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f4078160640 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f406c000c00 tx=0x7f406c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c041020 con 0x7f40780d1c80 2024-05-09T03:18:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c03d620 con 0x7f40780d1c80 2024-05-09T03:18:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c037040 con 0x7f40780d1c80 2024-05-09T03:18:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40780b2a60 con 0x7f40780d1c80 2024-05-09T03:18:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.886+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40780b11b0 con 0x7f40780d1c80 2024-05-09T03:18:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.887+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f407806a270 con 0x7f40780d1c80 2024-05-09T03:18:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.888+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f406c044020 con 0x7f40780d1c80 2024-05-09T03:18:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.888+0000 7f40757fa640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40640762d0 0x7f40640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.889+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7f406c0b88b0 con 0x7f40780d1c80 2024-05-09T03:18:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.889+0000 7f407cebb640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40640762d0 0x7f40640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.889+0000 7f407cebb640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40640762d0 0x7f40640787e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4078074930 tx=0x7f4068009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:40.892+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f406c04b030 con 0x7f40780d1c80 2024-05-09T03:18:41.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.127+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40780caa90 con 0x7f40780d1c80 2024-05-09T03:18:41.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.128+0000 7f40757fa640 1 -- 172.21.15.46:0/691198632 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) ==== 95+0+5169 (secure 0 0 0) 0x7f406c0812c0 con 0x7f40780d1c80 2024-05-09T03:18:41.129 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:41.130 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":233,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9201,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2490064583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2490064583},{"type":"v1","addr":"172.21.15.46:6834","nonce":2490064583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":233}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:40.005512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:41.130 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 233 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40640762d0 msgr2=0x7f40640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40640762d0 0x7f40640787e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f4078074930 tx=0x7f4068009190 comp rx=0 tx=0).stop 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 msgr2=0x7f4078160640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f4078160640 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f406c000c00 tx=0x7f406c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 shutdown_connections 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40780d2090 0x7f40781f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40640762d0 0x7f40640787e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40780d1c80 0x7f4078160640 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 --2- 172.21.15.46:0/691198632 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f407808cd60 0x7f40780b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 >> 172.21.15.46:0/691198632 conn(0x7f40780d3c00 msgr2=0x7f407815e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:41.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 shutdown_connections 2024-05-09T03:18:41.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.130+0000 7f407f11f640 1 -- 172.21.15.46:0/691198632 wait complete. 2024-05-09T03:18:41.143 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 1.00000000.inode' led to DAMAGED state 2024-05-09T03:18:41.143 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 200.00000000' 2024-05-09T03:18:41.143 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:41.143 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:41.160 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:41.160 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:41.160 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:41.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:41.160 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:41.216 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.370+0000 7f0efe934640 1 Processor -- start 2024-05-09T03:18:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.370+0000 7f0efe934640 1 -- start start 2024-05-09T03:18:41.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0efe934640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b7e70 0x7f0ef80d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0efe934640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef80b5550 con 0x7f0ef80b5140 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0efe934640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef80d44e0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0efe934640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef80d3480 con 0x7f0ef80b7e70 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0ef80b5140 0x7f0ef80b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46424/0 (socket says 172.21.15.46:46424) 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef7fff640 1 -- 172.21.15.46:0/4166625713 learned_addr learned my addr 172.21.15.46:0/4166625713 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef77fe640 1 --2- 172.21.15.46:0/4166625713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b7e70 0x7f0ef80d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2195424526 0 0) 0x7f0ef80b5550 con 0x7f0ef80b5140 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ee0003520 con 0x7f0ef80b5140 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 99953747 0 0) 0x7f0ef80d44e0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.372+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef80b5550 con 0x7f0ef80d3df0 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.373+0000 7f0ef77fe640 1 -- 172.21.15.46:0/4166625713 >> v1:172.21.15.88:6789/0 conn(0x7f0ef80d3df0 legacy=0x7f0ef80b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.373+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3202696357 0 0) 0x7f0ee0003520 con 0x7f0ef80b5140 2024-05-09T03:18:41.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.373+0000 7f0ef77fe640 1 -- 172.21.15.46:0/4166625713 >> v1:172.21.15.46:6789/0 conn(0x7f0ef80b5140 legacy=0x7f0ef80b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.373+0000 7f0ef77fe640 1 -- 172.21.15.46:0/4166625713 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef80d47c0 con 0x7f0ef80b7e70 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.373+0000 7f0ef77fe640 1 --2- 172.21.15.46:0/4166625713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b7e70 0x7f0ef80d38b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0097d0 tx=0x7f0eec02a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ec728956f78870 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec037070 con 0x7f0ef80b7e70 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec02acc0 con 0x7f0ef80b7e70 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0ef6ffd640 1 -- 172.21.15.46:0/4166625713 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80016f0 con 0x7f0ef80b7e70 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 -- 172.21.15.46:0/4166625713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b7e70 msgr2=0x7f0ef80d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 --2- 172.21.15.46:0/4166625713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b7e70 0x7f0ef80d38b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0097d0 tx=0x7f0eec02a230 comp rx=0 tx=0).stop 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 -- 172.21.15.46:0/4166625713 shutdown_connections 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 --2- 172.21.15.46:0/4166625713 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b7e70 0x7f0ef80d38b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 -- 172.21.15.46:0/4166625713 >> 172.21.15.46:0/4166625713 conn(0x7f0ef808d3f0 msgr2=0x7f0ef808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 -- 172.21.15.46:0/4166625713 shutdown_connections 2024-05-09T03:18:41.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.374+0000 7f0efe934640 1 -- 172.21.15.46:0/4166625713 wait complete. 2024-05-09T03:18:41.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.376+0000 7f0efe934640 1 Processor -- start 2024-05-09T03:18:41.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.376+0000 7f0efe934640 1 -- start start 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efe934640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b5140 0x7f0ef81501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efe934640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 0x7f0ef80ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efe934640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef80cdbc0 0x7f0ef80cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efe934640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef80d4200 con 0x7f0ef80d3df0 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efe934640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef80d4d80 con 0x7f0ef80cdbc0 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efe934640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef80d5580 con 0x7f0ef80b5140 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 0x7f0ef80ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 0x7f0ef80ac6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58922/0 (socket says 172.21.15.46:58922) 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 -- 172.21.15.46:0/580274098 learned_addr learned my addr 172.21.15.46:0/580274098 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef7fff640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b5140 0x7f0ef81501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0efced1640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef80cdbc0 0x7f0ef80cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 -- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b5140 msgr2=0x7f0ef81501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b5140 0x7f0ef81501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 -- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef80cdbc0 msgr2=0x7f0ef80cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef80cdbc0 0x7f0ef80cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 -- 172.21.15.46:0/580274098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef80caf10 con 0x7f0ef80d3df0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.378+0000 7f0ef77fe640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 0x7f0ef80ac6d0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f0eec035040 tx=0x7f0eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.379+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec037040 con 0x7f0ef80d3df0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.379+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec004230 con 0x7f0ef80d3df0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.379+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef80cb1f0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.379+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec006e10 con 0x7f0ef80d3df0 2024-05-09T03:18:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.379+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef80cb7b0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.380+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef80d17c0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.380+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0eec003610 con 0x7f0ef80d3df0 2024-05-09T03:18:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.381+0000 7f0ef4ff9640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ee40762d0 0x7f0ee40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.381+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7f0eec0b85c0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.382+0000 7f0ef7fff640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ee40762d0 0x7f0ee40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.382+0000 7f0ef7fff640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ee40762d0 0x7f0ee40787e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80013c0 tx=0x7f0ee80016a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.386+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0ef80d17c0 con 0x7f0ef80d3df0 2024-05-09T03:18:41.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:41.615+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0ef806a270 con 0x7f0ef80d3df0 2024-05-09T03:18:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.046+0000 7f0ef4ff9640 1 -- 172.21.15.46:0/580274098 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v234) ==== 161+0+0 (secure 0 0 0) 0x7f0eec04d020 con 0x7f0ef80d3df0 2024-05-09T03:18:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:42.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ee40762d0 msgr2=0x7f0ee40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:42.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ee40762d0 0x7f0ee40787e0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80013c0 tx=0x7f0ee80016a0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 msgr2=0x7f0ef80ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:42.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 0x7f0ef80ac6d0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f0eec035040 tx=0x7f0eec0046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 shutdown_connections 2024-05-09T03:18:42.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef80cdbc0 0x7f0ef80cbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ee40762d0 0x7f0ee40787e0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ef80d3df0 0x7f0ef80ac6d0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 --2- 172.21.15.46:0/580274098 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ef80b5140 0x7f0ef81501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 >> 172.21.15.46:0/580274098 conn(0x7f0ef808d3f0 msgr2=0x7f0ef80d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 shutdown_connections 2024-05-09T03:18:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.049+0000 7f0efe934640 1 -- 172.21.15.46:0/580274098 wait complete. 2024-05-09T03:18:42.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:42.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.181+0000 7fd7c6861640 1 Processor -- start 2024-05-09T03:18:42.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.181+0000 7fd7c6861640 1 -- start start 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.182+0000 7fd7c6861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 0x7fd7c006c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.182+0000 7fd7c6861640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd7c006cbc0 0x7fd7c00b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.182+0000 7fd7c6861640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c00b5d30 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.182+0000 7fd7c6861640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c0070450 con 0x7fd7c006cbc0 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.182+0000 7fd7c6861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c00b5ae0 con 0x7fd7c00b12d0 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 0x7fd7c006c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 0x7fd7c006c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:49204/0 (socket says 172.21.15.46:49204) 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7c4dfe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd7c006cbc0 0x7fd7c00b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 -- 172.21.15.46:0/3116783974 learned_addr learned my addr 172.21.15.46:0/3116783974 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 -- 172.21.15.46:0/3116783974 >> v2:172.21.15.88:3300/0 conn(0x7fd7c006cbc0 msgr2=0x7fd7c00b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:42.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/3116783974 >> v2:172.21.15.88:3300/0 conn(0x7fd7c006cbc0 0x7fd7c00b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 -- 172.21.15.46:0/3116783974 >> v1:172.21.15.88:6789/0 conn(0x7fd7c00ce3f0 legacy=0x7fd7c006fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:42.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7beffd640 1 -- 172.21.15.46:0/3116783974 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494804066 0 0) 0x7fd7c00b5d30 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7bf7fe640 1 -- 172.21.15.46:0/3116783974 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c00b6010 con 0x7fd7c00b12d0 2024-05-09T03:18:42.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.183+0000 7fd7c4dfe640 1 --2- 172.21.15.46:0/3116783974 >> v2:172.21.15.88:3300/0 conn(0x7fd7c006cbc0 0x7fd7c00b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.184+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/3116783974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 0x7fd7c006c680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009870 tx=0x7fd7b402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9abc403b8f4b6df6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.184+0000 7fd7beffd640 1 -- 172.21.15.46:0/3116783974 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4037070 con 0x7fd7c00b12d0 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.184+0000 7fd7beffd640 1 -- 172.21.15.46:0/3116783974 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b4032440 con 0x7fd7c00b12d0 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.184+0000 7fd7beffd640 1 -- 172.21.15.46:0/3116783974 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4032810 con 0x7fd7c00b12d0 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 -- 172.21.15.46:0/3116783974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 msgr2=0x7fd7c006c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 --2- 172.21.15.46:0/3116783974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 0x7fd7c006c680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009870 tx=0x7fd7b402a270 comp rx=0 tx=0).stop 2024-05-09T03:18:42.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 -- 172.21.15.46:0/3116783974 shutdown_connections 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 --2- 172.21.15.46:0/3116783974 >> v2:172.21.15.88:3300/0 conn(0x7fd7c006cbc0 0x7fd7c00b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 --2- 172.21.15.46:0/3116783974 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c00b12d0 0x7fd7c006c680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 -- 172.21.15.46:0/3116783974 >> 172.21.15.46:0/3116783974 conn(0x7fd7c00b4400 msgr2=0x7fd7c006f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 -- 172.21.15.46:0/3116783974 shutdown_connections 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.185+0000 7fd7c6861640 1 -- 172.21.15.46:0/3116783974 wait complete. 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.187+0000 7fd7c6861640 1 Processor -- start 2024-05-09T03:18:42.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.187+0000 7fd7c6861640 1 -- start start 2024-05-09T03:18:42.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c6861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c006cbc0 0x7fd7c01f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:42.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c6861640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c00b12d0 0x7fd7c0157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:42.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c6861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 0x7fd7c01f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c6861640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c015c8e0 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c6861640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c00b65d0 con 0x7fd7c00b12d0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c6861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c015ca60 con 0x7fd7c006cbc0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c4dfe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 0x7fd7c01f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.188+0000 7fd7c4dfe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 0x7fd7c01f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58932/0 (socket says 172.21.15.46:58932) 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 -- 172.21.15.46:0/378739593 learned_addr learned my addr 172.21.15.46:0/378739593 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7bf7fe640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c00b12d0 0x7fd7c0157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7bffff640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c006cbc0 0x7fd7c01f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 -- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c006cbc0 msgr2=0x7fd7c01f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c006cbc0 0x7fd7c01f4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 -- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c00b12d0 msgr2=0x7fd7c0157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c00b12d0 0x7fd7c0157c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 -- 172.21.15.46:0/378739593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c01f5880 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c4dfe640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 0x7fd7c01f38b0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800efc0 tx=0x7fd7b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:42.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8018020 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b8014480 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c01f5b60 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8014870 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.189+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c01f6120 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.190+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c006a270 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.191+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd7b8005d10 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.192+0000 7fd7bcff9640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7ac0762d0 0x7fd7ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.192+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7fd7b8098c40 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.192+0000 7fd7bffff640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7ac0762d0 0x7fd7ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.192+0000 7fd7bffff640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7ac0762d0 0x7fd7ac0787e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0007fd0 tx=0x7fd7b0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:42.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.196+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd7b8022030 con 0x7fd7c00ce3f0 2024-05-09T03:18:42.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:42.462+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fd7c01f5610 con 0x7fd7c00ce3f0 2024-05-09T03:18:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.052+0000 7fd7bcff9640 1 -- 172.21.15.46:0/378739593 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v235) ==== 101+0+0 (secure 0 0 0) 0x7fd7b8010040 con 0x7fd7c00ce3f0 2024-05-09T03:18:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7ac0762d0 msgr2=0x7fd7ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7ac0762d0 0x7fd7ac0787e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0007fd0 tx=0x7fd7b0009040 comp rx=0 tx=0).stop 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 msgr2=0x7fd7c01f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 0x7fd7c01f38b0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800efc0 tx=0x7fd7b800c490 comp rx=0 tx=0).stop 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 shutdown_connections 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd7ac0762d0 0x7fd7ac0787e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c00ce3f0 0x7fd7c01f38b0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7c00b12d0 0x7fd7c0157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 --2- 172.21.15.46:0/378739593 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7c006cbc0 0x7fd7c01f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 >> 172.21.15.46:0/378739593 conn(0x7fd7c00b4400 msgr2=0x7fd7c0160550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 shutdown_connections 2024-05-09T03:18:43.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.055+0000 7fd7c6861640 1 -- 172.21.15.46:0/378739593 wait complete. 2024-05-09T03:18:43.071 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:43.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.130+0000 7ff1c0de6f40 1 Processor -- start 2024-05-09T03:18:43.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.130+0000 7ff1c0de6f40 1 -- start start 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1c0de6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc329250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1c0de6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc492700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1c0de6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc493050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1c0de6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558bdc2d2f50 con 0x558bdc401780 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1c0de6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558bdc2d6f90 con 0x558bdc3e6a50 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1c0de6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558bdc36bcb0 con 0x558bdc492c40 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.131+0000 7ff1ba0ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc492700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc492700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58958/0 (socket says 172.21.15.46:58958) 2024-05-09T03:18:43.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/545944293 learned_addr learned my addr 172.21.15.46:0/545944293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc329250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc493050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 msgr2=0x558bdc493050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc493050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 msgr2=0x558bdc329250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc329250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/545944293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bdc3e5b40 con 0x558bdc401780 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc329250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc493050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc492700 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0001d80 tx=0x7ff1b0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcba799a7a30541 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1b90ab640 1 -- 172.21.15.46:0/545944293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000bd70 con 0x558bdc401780 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1b90ab640 1 -- 172.21.15.46:0/545944293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b000f400 con 0x558bdc401780 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.132+0000 7ff1b90ab640 1 -- 172.21.15.46:0/545944293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000f7f0 con 0x558bdc401780 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/545944293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 msgr2=0x558bdc492700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc492700 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0001d80 tx=0x7ff1b0001e70 comp rx=0 tx=0).stop 2024-05-09T03:18:43.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/545944293 shutdown_connections 2024-05-09T03:18:43.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc493050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc492700 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/545944293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc329250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/545944293 >> 172.21.15.46:0/545944293 conn(0x558bdc24eac0 msgr2=0x558bdc3fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:43.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/545944293 shutdown_connections 2024-05-09T03:18:43.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.133+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/545944293 wait complete. 2024-05-09T03:18:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.134+0000 7ff1c0de6f40 1 Processor -- start 2024-05-09T03:18:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.134+0000 7ff1c0de6f40 1 -- start start 2024-05-09T03:18:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1c0de6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc400d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1c0de6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc402f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1c0de6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc403450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1c0de6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558bdc493590 con 0x558bdc401780 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1c0de6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558bdc32a9d0 con 0x558bdc3e6a50 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1c0de6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x558bdc401c20 con 0x558bdc492c40 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1ba0ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc402f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1ba0ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc402f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58966/0 (socket says 172.21.15.46:58966) 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.135+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 learned_addr learned my addr 172.21.15.46:0/493454470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc400d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc403450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 msgr2=0x558bdc403450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x558bdc492c40 0x558bdc403450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 msgr2=0x558bdc400d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc400d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bdc3557d0 con 0x558bdc401780 2024-05-09T03:18:43.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558bdc3e6a50 0x558bdc400d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc402f10 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0000f60 tx=0x7ff1b0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1a3fff640 1 -- 172.21.15.46:0/493454470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000be70 con 0x558bdc401780 2024-05-09T03:18:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1a3fff640 1 -- 172.21.15.46:0/493454470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b0021020 con 0x558bdc401780 2024-05-09T03:18:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558bdc401e70 con 0x558bdc401780 2024-05-09T03:18:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1a3fff640 1 -- 172.21.15.46:0/493454470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b001c650 con 0x558bdc401780 2024-05-09T03:18:43.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.136+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bdc402430 con 0x558bdc401780 2024-05-09T03:18:43.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.138+0000 7ff1a3fff640 1 -- 172.21.15.46:0/493454470 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff1b0017070 con 0x558bdc401780 2024-05-09T03:18:43.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1a3fff640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1a8074cb0 0x7ff1a80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1a3fff640 1 -- 172.21.15.46:0/493454470 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7ff1b00982b0 con 0x558bdc401780 2024-05-09T03:18:43.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1a8074cb0 0x7ff1a80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.142 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558bdc32dd60 0x558bdc49b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc32a340 con 0x558bdc32dd60 2024-05-09T03:18:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558bdc32dd60 0x558bdc49b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.139+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1a8074cb0 0x7ff1a80771c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0015a0 tx=0x7ff1ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.140+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558bdc32dd60 0x558bdc49b220 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.142+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v108'722 uv696 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.142+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4012a0 con 0x558bdc32dd60 2024-05-09T03:18:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.143+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v108'723 uv723 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.143+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc3711c0 con 0x558bdc32dd60 2024-05-09T03:18:43.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.145+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v108'724 uv724 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.146 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:43.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.145+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a4b40 con 0x558bdc32dd60 2024-05-09T03:18:43.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.146+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v108'725 uv725 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.146+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a4ee0 con 0x558bdc32dd60 2024-05-09T03:18:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.148+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v108'726 uv726 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.148+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a3b30 con 0x558bdc32dd60 2024-05-09T03:18:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.149+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v108'727 uv727 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.149+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a3f20 con 0x558bdc32dd60 2024-05-09T03:18:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.150+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v108'728 uv702 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.151+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a4310 con 0x558bdc32dd60 2024-05-09T03:18:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.152+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v108'729 uv729 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.152+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a4700 con 0x558bdc32dd60 2024-05-09T03:18:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.153+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v108'730 uv730 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.155 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.153+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a4700 con 0x558bdc32dd60 2024-05-09T03:18:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.155+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v108'731 uv731 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.155+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a4700 con 0x558bdc32dd60 2024-05-09T03:18:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.156+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v108'732 uv732 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.156+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a4700 con 0x558bdc32dd60 2024-05-09T03:18:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.158+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v108'733 uv733 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.158+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.159+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v108'734 uv709 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.159+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.161+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v108'735 uv735 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.161+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.163 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.162+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v108'736 uv736 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.162+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.164+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v108'737 uv737 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.164+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.165+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v108'738 uv738 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.165+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.166+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v108'739 uv739 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.166+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.168+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v108'740 uv740 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.168+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.169+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v108'741 uv715 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.169+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.171+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v108'742 uv742 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.171+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.172+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v108'743 uv743 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.174 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.172+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.173+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v108'744 uv744 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.173+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.175+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v108'745 uv745 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.175+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.176+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v108'746 uv746 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.176+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.178+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v108'747 uv721 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.178+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.179+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v108'748 uv748 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.179+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.180+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v108'749 uv749 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.182 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.180+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.182+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v108'750 uv750 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.182+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.183+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v108'751 uv751 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.183+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc32dd60 2024-05-09T03:18:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.185+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v108'752 uv752 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.185+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558bdc4a5790 0x558bdc4a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.185+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a46b0 con 0x558bdc4a5790 2024-05-09T03:18:43.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.185+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558bdc4a5790 0x558bdc4a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.185+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558bdc4a5790 0x558bdc4a7bb0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.187+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v108'523 uv522 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.187+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a80f0 con 0x558bdc4a5790 2024-05-09T03:18:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.189+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v108'524 uv524 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.189+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a8530 con 0x558bdc4a5790 2024-05-09T03:18:43.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.190+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v108'525 uv525 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.191 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:43.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.190+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a8df0 con 0x558bdc4a5790 2024-05-09T03:18:43.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.191+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v108'526 uv526 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.191+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9190 con 0x558bdc4a5790 2024-05-09T03:18:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.192+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v108'527 uv527 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.192+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9530 con 0x558bdc4a5790 2024-05-09T03:18:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.194+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v108'528 uv510 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.194+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a9920 con 0x558bdc4a5790 2024-05-09T03:18:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.195+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v108'529 uv529 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.195+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9d10 con 0x558bdc4a5790 2024-05-09T03:18:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.196+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v108'530 uv530 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.198 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.196+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4aa100 con 0x558bdc4a5790 2024-05-09T03:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.197+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v108'531 uv531 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.197+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4aa100 con 0x558bdc4a5790 2024-05-09T03:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.198+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v108'532 uv532 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.198+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.200+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v108'533 uv515 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.200+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.201+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v108'534 uv534 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.201+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.203+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v108'535 uv535 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.204 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.203+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ad850 con 0x558bdc4a5790 2024-05-09T03:18:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.204+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v108'536 uv536 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.204+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ad850 con 0x558bdc4a5790 2024-05-09T03:18:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.206+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v108'537 uv537 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.206+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.207+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v108'538 uv521 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.207+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.208+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v108'539 uv539 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.208+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.211 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.209+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v108'540 uv540 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.209+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4aae00 con 0x558bdc4a5790 2024-05-09T03:18:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.211+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v108'541 uv541 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.211+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4aae00 con 0x558bdc4a5790 2024-05-09T03:18:43.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.212+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v108'542 uv542 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.212+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc4a5790 2024-05-09T03:18:43.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.213+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v108'543 uv543 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4a5790 2024-05-09T03:18:43.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.213+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc32dd60 2024-05-09T03:18:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.214+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v108'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.214+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc32dd60 2024-05-09T03:18:43.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.215+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v108'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.215+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc32dd60 2024-05-09T03:18:43.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.216+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v108'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.218 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:43.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.216+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4aae00 con 0x558bdc32dd60 2024-05-09T03:18:43.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.217+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v108'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.217+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4aae00 con 0x558bdc32dd60 2024-05-09T03:18:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.219+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v108'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.219+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a9df0 con 0x558bdc32dd60 2024-05-09T03:18:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.220+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v108'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.220+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558bdc4aa500 0x558bdc4aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.220+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ad1d0 con 0x558bdc4aa500 2024-05-09T03:18:43.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.220+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558bdc4aa500 0x558bdc4aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.220+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558bdc4aa500 0x558bdc4aa980 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.221+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v108'106 uv105 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.221+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4ad590 con 0x558bdc4aa500 2024-05-09T03:18:43.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.222+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v108'107 uv107 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.222+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ada90 con 0x558bdc4aa500 2024-05-09T03:18:43.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.223+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v108'108 uv108 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.225 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:43.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.223+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4a8a20 con 0x558bdc4aa500 2024-05-09T03:18:43.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.224+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v108'109 uv109 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.224+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ae3c0 con 0x558bdc32dd60 2024-05-09T03:18:43.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.226+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v108'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.226+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4ae3c0 con 0x558bdc32dd60 2024-05-09T03:18:43.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.227+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v108'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.227+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ae3c0 con 0x558bdc32dd60 2024-05-09T03:18:43.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.228+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v108'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.230 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:43.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.228+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4af3d0 con 0x558bdc32dd60 2024-05-09T03:18:43.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.229+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v108'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.229+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4af3d0 con 0x558bdc32dd60 2024-05-09T03:18:43.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.230+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v108'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.230+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ae3c0 con 0x558bdc32dd60 2024-05-09T03:18:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.232+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v108'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.232+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558bdc4aec90 0x558bdc4af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.232+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.232+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558bdc4aec90 0x558bdc4af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.233+0000 7ff1bb0af640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558bdc4aec90 0x558bdc4af0a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.234+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v108'260 uv254 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.234+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.235+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v108'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.235+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.236+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v108'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.238 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:43.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.236+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.237+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v108'263 uv263 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.237+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.238+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v108'264 uv264 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.238+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.239+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v108'265 uv265 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.239+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.241+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v108'266 uv259 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.241+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.242+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v108'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.242+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.243+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v108'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.246 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:43.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.243+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.244+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v108'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.244+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b17a0 con 0x558bdc4aec90 2024-05-09T03:18:43.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.246+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v108'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.246+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558bdc4b2140 0x558bdc4b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.246+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b4c10 con 0x558bdc4b2140 2024-05-09T03:18:43.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.246+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558bdc4b2140 0x558bdc4b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.246+0000 7ff1ba8ae640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558bdc4b2140 0x558bdc4b2550 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.248+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v108'117 uv116 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4b2140 2024-05-09T03:18:43.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.248+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b4c10 con 0x558bdc4b2140 2024-05-09T03:18:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.249+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v108'118 uv118 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4b2140 2024-05-09T03:18:43.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.249+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b4c90 con 0x558bdc4b2140 2024-05-09T03:18:43.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.250+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v108'119 uv119 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4b2140 2024-05-09T03:18:43.252 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:43.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.250+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc4b2140 2024-05-09T03:18:43.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.251+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v108'120 uv120 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4b2140 2024-05-09T03:18:43.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.251+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc4b2140 2024-05-09T03:18:43.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.252+0000 7ff1ba8ae640 1 -- 172.21.15.46:0/493454470 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v108'121 uv121 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1ac0045a0 con 0x558bdc4b2140 2024-05-09T03:18:43.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.252+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc32dd60 2024-05-09T03:18:43.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.253+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v108'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.253+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc32dd60 2024-05-09T03:18:43.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.254+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v108'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.254+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc32dd60 2024-05-09T03:18:43.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.255+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v108'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.257 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:43.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.255+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b60c0 con 0x558bdc32dd60 2024-05-09T03:18:43.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.256+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v108'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.256+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b60c0 con 0x558bdc32dd60 2024-05-09T03:18:43.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.257+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v108'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.257+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc32dd60 2024-05-09T03:18:43.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.258+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v108'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc32dd60 2024-05-09T03:18:43.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.258+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b50b0 con 0x558bdc4aa500 2024-05-09T03:18:43.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.260+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v108'233 uv227 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.260+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b5450 con 0x558bdc4aa500 2024-05-09T03:18:43.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.261+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v108'234 uv234 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.261+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b57f0 con 0x558bdc4aa500 2024-05-09T03:18:43.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.262+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v108'235 uv235 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.263 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.262+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b5b90 con 0x558bdc4aa500 2024-05-09T03:18:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.263+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v108'236 uv236 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.263+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b5b90 con 0x558bdc4aa500 2024-05-09T03:18:43.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.264+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v108'237 uv237 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.264+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b5b90 con 0x558bdc4aa500 2024-05-09T03:18:43.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.265+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v108'238 uv232 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.265+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b5b90 con 0x558bdc4aa500 2024-05-09T03:18:43.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.266+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v108'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.266+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b5b90 con 0x558bdc4aa500 2024-05-09T03:18:43.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.267+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v108'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.269 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.267+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b5b90 con 0x558bdc4aa500 2024-05-09T03:18:43.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.268+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v108'241 uv241 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.268+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b89a0 con 0x558bdc4aa500 2024-05-09T03:18:43.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.269+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v108'242 uv242 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.269+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558bdc4b66c0 0x558bdc4b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.269+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4b66c0 2024-05-09T03:18:43.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.269+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558bdc4b66c0 0x558bdc4b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.270+0000 7ff1ba0ad640 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558bdc4b66c0 0x558bdc4b8ae0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.271+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v108'103 uv102 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4b66c0 2024-05-09T03:18:43.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.271+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4b66c0 2024-05-09T03:18:43.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.272+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v108'104 uv104 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4b66c0 2024-05-09T03:18:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.272+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4b66c0 2024-05-09T03:18:43.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.273+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v108'105 uv105 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4b66c0 2024-05-09T03:18:43.275 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.273+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4b66c0 2024-05-09T03:18:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.274+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v108'106 uv106 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4b66c0 2024-05-09T03:18:43.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.274+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aa500 2024-05-09T03:18:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.276+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v108'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.276+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aa500 2024-05-09T03:18:43.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.277+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v108'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.277+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aa500 2024-05-09T03:18:43.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.278+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v108'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.280 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:43.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.278+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ba030 con 0x558bdc4aa500 2024-05-09T03:18:43.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.279+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v108'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.279+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ba030 con 0x558bdc4aa500 2024-05-09T03:18:43.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.281+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v108'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.281+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aa500 2024-05-09T03:18:43.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.282+0000 7ff1ba0ad640 1 -- 172.21.15.46:0/493454470 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v108'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b005cfa0 con 0x558bdc4aa500 2024-05-09T03:18:43.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.282+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.283+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v108'120 uv119 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.283+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.284+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v108'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.284+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.285+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v108'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.287 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:43.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.286+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.287+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v108'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.287+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.288+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v108'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.288+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.290+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v108'290 uv282 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.290+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.291+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v108'291 uv291 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.291+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.294 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:43.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.292+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v108'292 uv292 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.292+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.293+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v108'293 uv293 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.293+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.295+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v108'294 uv294 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.295+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.296+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v108'295 uv289 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.296+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.297+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v108'296 uv296 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.297+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.298+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v108'297 uv297 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.300 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:43.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.298+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ba030 con 0x558bdc4aec90 2024-05-09T03:18:43.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.299+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v108'298 uv298 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.299+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4ba030 con 0x558bdc4aec90 2024-05-09T03:18:43.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.300+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v108'299 uv299 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.300+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b9020 con 0x558bdc4aec90 2024-05-09T03:18:43.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.301+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v108'300 uv300 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.301+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x558bdc4b93e0 con 0x558bdc4aec90 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v108'301 uv301 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff1b40549d0 con 0x558bdc4aec90 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558bdc32dd60 msgr2=0x558bdc49b220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x558bdc32dd60 0x558bdc49b220 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558bdc4aa500 msgr2=0x558bdc4aa980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x558bdc4aa500 0x558bdc4aa980 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558bdc4b66c0 msgr2=0x558bdc4b8ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x558bdc4b66c0 0x558bdc4b8ae0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558bdc4a5790 msgr2=0x558bdc4a7bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x558bdc4a5790 0x558bdc4a7bb0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558bdc4aec90 msgr2=0x558bdc4af0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x558bdc4aec90 0x558bdc4af0a0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558bdc4b2140 msgr2=0x558bdc4b2550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x558bdc4b2140 0x558bdc4b2550 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 reap_dead start 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 reap_dead start 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1bb0af640 1 -- 172.21.15.46:0/493454470 reap_dead start 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1a8074cb0 msgr2=0x7ff1a80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1a8074cb0 0x7ff1a80771c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac0015a0 tx=0x7ff1ac008040 comp rx=0 tx=0).stop 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 msgr2=0x558bdc402f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.302+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc402f10 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7ff1b0000f60 tx=0x7ff1b0000ed0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.303+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 shutdown_connections 2024-05-09T03:18:43.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.303+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1a8074cb0 0x7ff1a80771c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.303+0000 7ff1c0de6f40 1 --2- 172.21.15.46:0/493454470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558bdc401780 0x558bdc402f10 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.303+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 >> 172.21.15.46:0/493454470 conn(0x558bdc24eac0 msgr2=0x558bdc329250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:43.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.303+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 shutdown_connections 2024-05-09T03:18:43.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.303+0000 7ff1c0de6f40 1 -- 172.21.15.46:0/493454470 wait complete. 2024-05-09T03:18:43.308 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 200.00000000 0 2024-05-09T03:18:43.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.403+0000 7f9d1340df40 1 Processor -- start 2024-05-09T03:18:43.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.403+0000 7f9d1340df40 1 -- start start 2024-05-09T03:18:43.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d1340df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578121a9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d1340df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120ef060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d1340df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578120ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d1340df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557812098f50 con 0x557812137fd0 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d1340df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55781209cf90 con 0x5578121aca50 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d1340df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557812131cb0 con 0x5578121c7780 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d0ca67640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578120ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d0ca67640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578120ef5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58978/0 (socket says 172.21.15.46:58978) 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d0ca67640 1 -- 172.21.15.46:0/3033064430 learned_addr learned my addr 172.21.15.46:0/3033064430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d0c266640 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578121a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.404+0000 7f9d0ba65640 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0ca67640 1 -- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 msgr2=0x5578120ef060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0ca67640 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120ef060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0ca67640 1 -- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 msgr2=0x5578121a9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0ca67640 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578121a9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0ca67640 1 -- 172.21.15.46:0/3033064430 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578121c5cb0 con 0x557812137fd0 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0ca67640 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578120ef5a0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f9d04025a00 tx=0x7f9d04051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae00cd3cb5649b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0aa63640 1 -- 172.21.15.46:0/3033064430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04056040 con 0x557812137fd0 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0aa63640 1 -- 172.21.15.46:0/3033064430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d04052890 con 0x557812137fd0 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.405+0000 7f9d0aa63640 1 -- 172.21.15.46:0/3033064430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04052c40 con 0x557812137fd0 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 -- 172.21.15.46:0/3033064430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 msgr2=0x5578120ef5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578120ef5a0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f9d04025a00 tx=0x7f9d04051000 comp rx=0 tx=0).stop 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 -- 172.21.15.46:0/3033064430 shutdown_connections 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578120ef5a0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 --2- 172.21.15.46:0/3033064430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578121a9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 -- 172.21.15.46:0/3033064430 >> 172.21.15.46:0/3033064430 conn(0x557812014ac0 msgr2=0x5578121c5fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 -- 172.21.15.46:0/3033064430 shutdown_connections 2024-05-09T03:18:43.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.406+0000 7f9d1340df40 1 -- 172.21.15.46:0/3033064430 wait complete. 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.407+0000 7f9d1340df40 1 Processor -- start 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.407+0000 7f9d1340df40 1 -- start start 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d1340df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578121a8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d1340df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578120f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d1340df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d1340df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5578121acf00 con 0x557812137fd0 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d1340df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5578121ad1e0 con 0x5578121aca50 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d1340df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5578121ab800 con 0x5578121c7780 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578121a8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578121a8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58988/0 (socket says 172.21.15.46:58988) 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 -- 172.21.15.46:0/767154525 learned_addr learned my addr 172.21.15.46:0/767154525 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0ba65640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578120f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0ca67640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 -- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 msgr2=0x5578120f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5578121c7780 0x5578120f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 -- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 msgr2=0x5578120f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5578121aca50 0x5578120f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.408+0000 7f9d0c266640 1 -- 172.21.15.46:0/767154525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578121a8ef0 con 0x557812137fd0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.409+0000 7f9d0c266640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578121a8760 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00ef30 tx=0x7f9cfc00c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.409+0000 7f9d09a61640 1 -- 172.21.15.46:0/767154525 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc015020 con 0x557812137fd0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.409+0000 7f9d09a61640 1 -- 172.21.15.46:0/767154525 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cfc011450 con 0x557812137fd0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.409+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5578121c72c0 con 0x557812137fd0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.409+0000 7f9d09a61640 1 -- 172.21.15.46:0/767154525 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc011840 con 0x557812137fd0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.409+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578121c5a00 con 0x557812137fd0 2024-05-09T03:18:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.410+0000 7f9d09a61640 1 -- 172.21.15.46:0/767154525 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9cfc018020 con 0x557812137fd0 2024-05-09T03:18:43.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.411+0000 7f9d09a61640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cf8074ce0 0x7f9cf80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.411+0000 7f9d09a61640 1 -- 172.21.15.46:0/767154525 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7f9cfc096aa0 con 0x557812137fd0 2024-05-09T03:18:43.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.411+0000 7f9d0ba65640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cf8074ce0 0x7f9cf80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.411+0000 7f9d1340df40 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5578121afaf0 0x5578121c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.411+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x5578121af5a0 con 0x5578121afaf0 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.411+0000 7f9d0ca67640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5578121afaf0 0x5578121c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.412+0000 7f9d0ba65640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cf8074ce0 0x7f9cf80771f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x5578121ac1d0 tx=0x7f9d00001810 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.412+0000 7f9d0ca67640 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5578121afaf0 0x5578121c9df0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d0ca67640 1 -- 172.21.15.46:0/767154525 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000000 [truncate 0] v108'544 uv544 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d04052ac0 con 0x5578121afaf0 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5578121afaf0 msgr2=0x5578121c9df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5578121afaf0 0x5578121c9df0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cf8074ce0 msgr2=0x7f9cf80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9cf8074ce0 0x7f9cf80771f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x5578121ac1d0 tx=0x7f9d00001810 comp rx=0 tx=0).stop 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 msgr2=0x5578121a8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 --2- 172.21.15.46:0/767154525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557812137fd0 0x5578121a8760 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00ef30 tx=0x7f9cfc00c620 comp rx=0 tx=0).stop 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d0ca67640 1 -- 172.21.15.46:0/767154525 reap_dead start 2024-05-09T03:18:43.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 shutdown_connections 2024-05-09T03:18:43.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 >> 172.21.15.46:0/767154525 conn(0x557812014ac0 msgr2=0x5578121aec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:43.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 shutdown_connections 2024-05-09T03:18:43.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.413+0000 7f9d1340df40 1 -- 172.21.15.46:0/767154525 wait complete. 2024-05-09T03:18:43.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.534+0000 7f7765786640 1 Processor -- start 2024-05-09T03:18:43.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.534+0000 7f7765786640 1 -- start start 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.535+0000 7f7765786640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600b7e70 0x7f77600b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.535+0000 7f7765786640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77600d3400 con 0x7f77600d2e90 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.535+0000 7f7765786640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77600d4a60 con 0x7f77601605b0 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.535+0000 7f7765786640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77600d4d40 con 0x7f77600b7e70 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f77600d2e90 0x7f77600b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46456/0 (socket says 172.21.15.46:46456) 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775effd640 1 -- 172.21.15.46:0/1643704084 learned_addr learned my addr 172.21.15.46:0/1643704084 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:43.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775e7fc640 1 --2- 172.21.15.46:0/1643704084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600b7e70 0x7f77600b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 129092361 0 0) 0x7f77600d3400 con 0x7f77600d2e90 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7748003520 con 0x7f77600d2e90 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036568044 0 0) 0x7f77600d4a60 con 0x7f77601605b0 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77600d3400 con 0x7f77601605b0 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775e7fc640 1 -- 172.21.15.46:0/1643704084 >> v1:172.21.15.88:6789/0 conn(0x7f77601605b0 legacy=0x7f77600b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775e7fc640 1 -- 172.21.15.46:0/1643704084 >> v1:172.21.15.46:6789/0 conn(0x7f77600d2e90 legacy=0x7f77600b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.536+0000 7f775e7fc640 1 -- 172.21.15.46:0/1643704084 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77600d5020 con 0x7f77600b7e70 2024-05-09T03:18:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.537+0000 7f775e7fc640 1 --2- 172.21.15.46:0/1643704084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600b7e70 0x7f77600b92a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7754009800 tx=0x7f775402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ffb522b79b4833 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.537+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754037070 con 0x7f77600b7e70 2024-05-09T03:18:43.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.537+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7754032440 con 0x7f77600b7e70 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.537+0000 7f775dffb640 1 -- 172.21.15.46:0/1643704084 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754032810 con 0x7f77600b7e70 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 -- 172.21.15.46:0/1643704084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600b7e70 msgr2=0x7f77600b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 --2- 172.21.15.46:0/1643704084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600b7e70 0x7f77600b92a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7754009800 tx=0x7f775402a230 comp rx=0 tx=0).stop 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 -- 172.21.15.46:0/1643704084 shutdown_connections 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 --2- 172.21.15.46:0/1643704084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600b7e70 0x7f77600b92a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 -- 172.21.15.46:0/1643704084 >> 172.21.15.46:0/1643704084 conn(0x7f776008d390 msgr2=0x7f776008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 -- 172.21.15.46:0/1643704084 shutdown_connections 2024-05-09T03:18:43.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.538+0000 7f7765786640 1 -- 172.21.15.46:0/1643704084 wait complete. 2024-05-09T03:18:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.540+0000 7f7765786640 1 Processor -- start 2024-05-09T03:18:43.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.540+0000 7f7765786640 1 -- start start 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f7765786640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600d2e90 0x7f7760152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f7765786640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77601605b0 0x7f7760153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f7765786640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 0x7f776015cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f7765786640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77600b4d80 con 0x7f776015c930 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f7765786640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f77600b4c00 con 0x7f77601605b0 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f7765786640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77600b4fc0 con 0x7f77600d2e90 2024-05-09T03:18:43.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 0x7f776015cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 0x7f776015cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58992/0 (socket says 172.21.15.46:58992) 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 -- 172.21.15.46:0/2774428517 learned_addr learned my addr 172.21.15.46:0/2774428517 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775e7fc640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77601605b0 0x7f7760153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775effd640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600d2e90 0x7f7760152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 -- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600d2e90 msgr2=0x7f7760152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600d2e90 0x7f7760152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 -- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77601605b0 msgr2=0x7f7760153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.541+0000 7f775f7fe640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77601605b0 0x7f7760153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f775f7fe640 1 -- 172.21.15.46:0/2774428517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7760156a60 con 0x7f776015c930 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f775e7fc640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77601605b0 0x7f7760153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f775f7fe640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 0x7f776015cd40 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f775800c730 tx=0x7f775800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758003b70 con 0x7f776015c930 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7758003d10 con 0x7f776015c930 2024-05-09T03:18:43.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776015be50 con 0x7f776015c930 2024-05-09T03:18:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775800aa70 con 0x7f776015c930 2024-05-09T03:18:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.542+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77601542f0 con 0x7f776015c930 2024-05-09T03:18:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.543+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77601567d0 con 0x7f776015c930 2024-05-09T03:18:43.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.544+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f775800ac10 con 0x7f776015c930 2024-05-09T03:18:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.544+0000 7f77477fe640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f774c0762d0 0x7f774c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.544+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7831+0+0 (secure 0 0 0) 0x7f7758097bd0 con 0x7f776015c930 2024-05-09T03:18:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.544+0000 7f775effd640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f774c0762d0 0x7f774c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:43.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.545+0000 7f775effd640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f774c0762d0 0x7f774c0787e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7750007f50 tx=0x7f7750007ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:43.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.549+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7758060610 con 0x7f776015c930 2024-05-09T03:18:43.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:43.780+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f776006a270 con 0x7f776015c930 2024-05-09T03:18:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.053+0000 7f77477fe640 1 -- 172.21.15.46:0/2774428517 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v236) ==== 162+0+0 (secure 0 0 0) 0x7f7758064910 con 0x7f776015c930 2024-05-09T03:18:44.055 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:44.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f774c0762d0 msgr2=0x7f774c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f774c0762d0 0x7f774c0787e0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7750007f50 tx=0x7f7750007ee0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 msgr2=0x7f776015cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 0x7f776015cd40 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f775800c730 tx=0x7f775800cbf0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 shutdown_connections 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f774c0762d0 0x7f774c0787e0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f776015c930 0x7f776015cd40 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77601605b0 0x7f7760153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.055+0000 7f7765786640 1 --2- 172.21.15.46:0/2774428517 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77600d2e90 0x7f7760152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.056+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 >> 172.21.15.46:0/2774428517 conn(0x7f776008d390 msgr2=0x7f776008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.056+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 shutdown_connections 2024-05-09T03:18:44.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.056+0000 7f7765786640 1 -- 172.21.15.46:0/2774428517 wait complete. 2024-05-09T03:18:44.069 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:44.065+0000 7f0488866640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:18:44.070 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:44.065+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:44.071 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:44.065+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:44.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:44.072 INFO:tasks.ceph.mds.c.smithi046.stderr: -17> 2024-05-09T03:18:44.065+0000 7f0488866640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-05-09T03:18:44.128 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.188+0000 7f56faa0c640 1 Processor -- start 2024-05-09T03:18:44.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.188+0000 7f56faa0c640 1 -- start start 2024-05-09T03:18:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56faa0c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 0x7f56f4160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56faa0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b7ef0 0x7f56f40b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56faa0c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f40d49e0 con 0x7f56f40b4af0 2024-05-09T03:18:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56faa0c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f56f40d5bb0 con 0x7f56f40d57a0 2024-05-09T03:18:44.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56faa0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40d4cc0 con 0x7f56f40b7ef0 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 0x7f56f4160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 0x7f56f4160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58996/0 (socket says 172.21.15.46:58996) 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f3fff640 1 -- 172.21.15.46:0/1848806219 learned_addr learned my addr 172.21.15.46:0/1848806219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f8fa9640 1 --2- 172.21.15.46:0/1848806219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b7ef0 0x7f56f40b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f37fe640 1 -- 172.21.15.46:0/1848806219 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3760394541 0 0) 0x7f56f40d49e0 con 0x7f56f40b4af0 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f37fe640 1 -- 172.21.15.46:0/1848806219 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e0003520 con 0x7f56f40b4af0 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f8fa9640 1 -- 172.21.15.46:0/1848806219 >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 msgr2=0x7f56f4160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f8fa9640 1 --2- 172.21.15.46:0/1848806219 >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 0x7f56f4160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f8fa9640 1 -- 172.21.15.46:0/1848806219 >> v1:172.21.15.88:6789/0 conn(0x7f56f40b4af0 legacy=0x7f56f40b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f8fa9640 1 -- 172.21.15.46:0/1848806219 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40d4fa0 con 0x7f56f40b7ef0 2024-05-09T03:18:44.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.190+0000 7f56f3fff640 1 --2- 172.21.15.46:0/1848806219 >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 0x7f56f4160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.191+0000 7f56f8fa9640 1 --2- 172.21.15.46:0/1848806219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b7ef0 0x7f56f40b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009d90 tx=0x7f56ec02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=773c78b6e9424c73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.191+0000 7f56f37fe640 1 -- 172.21.15.46:0/1848806219 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec007740 con 0x7f56f40b7ef0 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.191+0000 7f56f37fe640 1 -- 172.21.15.46:0/1848806219 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec0078e0 con 0x7f56f40b7ef0 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.191+0000 7f56f37fe640 1 -- 172.21.15.46:0/1848806219 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec007c90 con 0x7f56f40b7ef0 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 -- 172.21.15.46:0/1848806219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b7ef0 msgr2=0x7f56f40b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 --2- 172.21.15.46:0/1848806219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b7ef0 0x7f56f40b4f00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009d90 tx=0x7f56ec02a660 comp rx=0 tx=0).stop 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 -- 172.21.15.46:0/1848806219 shutdown_connections 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 --2- 172.21.15.46:0/1848806219 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b7ef0 0x7f56f40b4f00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 --2- 172.21.15.46:0/1848806219 >> v2:172.21.15.46:3300/0 conn(0x7f56f40d57a0 0x7f56f4160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 -- 172.21.15.46:0/1848806219 >> 172.21.15.46:0/1848806219 conn(0x7f56f40743e0 msgr2=0x7f56f40d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:44.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 -- 172.21.15.46:0/1848806219 shutdown_connections 2024-05-09T03:18:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.192+0000 7f56faa0c640 1 -- 172.21.15.46:0/1848806219 wait complete. 2024-05-09T03:18:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.194+0000 7f56faa0c640 1 Processor -- start 2024-05-09T03:18:44.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.194+0000 7f56faa0c640 1 -- start start 2024-05-09T03:18:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56faa0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b4af0 0x7f56f41e6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56faa0c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f40d57a0 0x7f56f40d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56faa0c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 0x7f56f4158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56faa0c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56f40d1920 con 0x7f56f415bea0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56faa0c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f56f40d15b0 con 0x7f56f40d57a0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56faa0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f56f41f77a0 con 0x7f56f40b4af0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56f8fa9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 0x7f56f4158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56f8fa9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 0x7f56f4158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59006/0 (socket says 172.21.15.46:59006) 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.195+0000 7f56f8fa9640 1 -- 172.21.15.46:0/585115249 learned_addr learned my addr 172.21.15.46:0/585115249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56ebfff640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f40d57a0 0x7f56f40d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f3fff640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b4af0 0x7f56f41e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f8fa9640 1 -- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b4af0 msgr2=0x7f56f41e6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f8fa9640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b4af0 0x7f56f41e6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f8fa9640 1 -- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f40d57a0 msgr2=0x7f56f40d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f8fa9640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f40d57a0 0x7f56f40d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f8fa9640 1 -- 172.21.15.46:0/585115249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f415c5d0 con 0x7f56f415bea0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f8fa9640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 0x7f56f4158560 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009d90 tx=0x7f56ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec03f070 con 0x7f56f415bea0 2024-05-09T03:18:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f415c8b0 con 0x7f56f415bea0 2024-05-09T03:18:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec031c40 con 0x7f56f415bea0 2024-05-09T03:18:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec009450 con 0x7f56f415bea0 2024-05-09T03:18:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.196+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f415ce70 con 0x7f56f415bea0 2024-05-09T03:18:44.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.197+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f406a270 con 0x7f56f415bea0 2024-05-09T03:18:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.198+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f56ec003990 con 0x7f56f415bea0 2024-05-09T03:18:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.198+0000 7f56f17fa640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56d40762d0 0x7f56d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.198+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7f56ec0b94a0 con 0x7f56f415bea0 2024-05-09T03:18:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.199+0000 7f56f3fff640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56d40762d0 0x7f56d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.199+0000 7f56f3fff640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56d40762d0 0x7f56d40787e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f56f406d3b0 tx=0x7f56e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.202+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f56ec0824c0 con 0x7f56f415bea0 2024-05-09T03:18:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.429+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56f40ba1c0 con 0x7f56f415bea0 2024-05-09T03:18:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.430+0000 7f56f17fa640 1 -- 172.21.15.46:0/585115249 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) ==== 95+0+4343 (secure 0 0 0) 0x7f56ec081dc0 con 0x7f56f415bea0 2024-05-09T03:18:44.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:44.432 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:44.067407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56d40762d0 msgr2=0x7f56d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56d40762d0 0x7f56d40787e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f56f406d3b0 tx=0x7f56e4008040 comp rx=0 tx=0).stop 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 msgr2=0x7f56f4158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 0x7f56f4158560 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009d90 tx=0x7f56ec004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 shutdown_connections 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56d40762d0 0x7f56d40787e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56f415bea0 0x7f56f4158560 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56f40d57a0 0x7f56f40d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 --2- 172.21.15.46:0/585115249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f56f40b4af0 0x7f56f41e6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 >> 172.21.15.46:0/585115249 conn(0x7f56f40743e0 msgr2=0x7f56f40ba7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 shutdown_connections 2024-05-09T03:18:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.432+0000 7f56faa0c640 1 -- 172.21.15.46:0/585115249 wait complete. 2024-05-09T03:18:44.445 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:44.446 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:44.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.559+0000 7f670e0bd640 1 Processor -- start 2024-05-09T03:18:44.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.559+0000 7f670e0bd640 1 -- start start 2024-05-09T03:18:44.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f670e0bd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6708164c70 0x7f67081615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f670e0bd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67080d2480 0x7f67080d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f670e0bd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6708160b20 con 0x7f670815dfe0 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f670e0bd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6708161150 con 0x7f6708164c70 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f670e0bd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f67080d15f0 con 0x7f67080d2480 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f67077fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6708164c70 0x7f67081615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f6706ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f670815dfe0 0x7f67080d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46464/0 (socket says 172.21.15.46:46464) 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f6706ffd640 1 -- 172.21.15.46:0/923517187 learned_addr learned my addr 172.21.15.46:0/923517187 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.560+0000 7f6707fff640 1 --2- 172.21.15.46:0/923517187 >> v2:172.21.15.88:3300/0 conn(0x7f67080d2480 0x7f67080d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.561+0000 7f67077fe640 1 -- 172.21.15.46:0/923517187 >> v2:172.21.15.88:3300/0 conn(0x7f67080d2480 msgr2=0x7f67080d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.561+0000 7f67077fe640 1 --2- 172.21.15.46:0/923517187 >> v2:172.21.15.88:3300/0 conn(0x7f67080d2480 0x7f67080d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.561+0000 7f67077fe640 1 -- 172.21.15.46:0/923517187 >> v1:172.21.15.46:6789/0 conn(0x7f670815dfe0 legacy=0x7f67080d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.561+0000 7f67077fe640 1 -- 172.21.15.46:0/923517187 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67080d18d0 con 0x7f6708164c70 2024-05-09T03:18:44.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.561+0000 7f67067fc640 1 -- 172.21.15.46:0/923517187 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070201182 0 0) 0x7f6708160b20 con 0x7f670815dfe0 2024-05-09T03:18:44.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.561+0000 7f67077fe640 1 --2- 172.21.15.46:0/923517187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6708164c70 0x7f67081615e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f67080d3e30 tx=0x7f66f802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124eaade0c2d529a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f67067fc640 1 -- 172.21.15.46:0/923517187 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8037070 con 0x7f6708164c70 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f67067fc640 1 -- 172.21.15.46:0/923517187 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f8032440 con 0x7f6708164c70 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f67067fc640 1 -- 172.21.15.46:0/923517187 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f802ae60 con 0x7f6708164c70 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 -- 172.21.15.46:0/923517187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6708164c70 msgr2=0x7f67081615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 --2- 172.21.15.46:0/923517187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6708164c70 0x7f67081615e0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f67080d3e30 tx=0x7f66f802a110 comp rx=0 tx=0).stop 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 -- 172.21.15.46:0/923517187 shutdown_connections 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 --2- 172.21.15.46:0/923517187 >> v2:172.21.15.88:3300/0 conn(0x7f67080d2480 0x7f67080d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 --2- 172.21.15.46:0/923517187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6708164c70 0x7f67081615e0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 -- 172.21.15.46:0/923517187 >> 172.21.15.46:0/923517187 conn(0x7f6708073db0 msgr2=0x7f67080741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 -- 172.21.15.46:0/923517187 shutdown_connections 2024-05-09T03:18:44.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.562+0000 7f670e0bd640 1 -- 172.21.15.46:0/923517187 wait complete. 2024-05-09T03:18:44.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.564+0000 7f670e0bd640 1 Processor -- start 2024-05-09T03:18:44.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.564+0000 7f670e0bd640 1 -- start start 2024-05-09T03:18:44.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 0x7f67081622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f670815dfe0 0x7f6708152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6708164c70 0x7f67080cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67080d2dd0 con 0x7f67080d2480 2024-05-09T03:18:44.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f670815e3f0 con 0x7f6708164c70 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f67080d45c0 con 0x7f670815dfe0 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 0x7f67081622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 0x7f67081622b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59018/0 (socket says 172.21.15.46:59018) 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 -- 172.21.15.46:0/2065678176 learned_addr learned my addr 172.21.15.46:0/2065678176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f6707fff640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6708164c70 0x7f67080cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 -- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f670815dfe0 msgr2=0x7f6708152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f670815dfe0 0x7f6708152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 -- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6708164c70 msgr2=0x7f67080cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6708164c70 0x7f67080cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 -- 172.21.15.46:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67080ce8d0 con 0x7f67080d2480 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f6707fff640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6708164c70 0x7f67080cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f67077fe640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 0x7f67081622b0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f66f8035040 tx=0x7f66f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8037050 con 0x7f67080d2480 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f8038040 con 0x7f67080d2480 2024-05-09T03:18:44.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8004450 con 0x7f67080d2480 2024-05-09T03:18:44.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67080cebb0 con 0x7f67080d2480 2024-05-09T03:18:44.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.566+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6708153510 con 0x7f67080d2480 2024-05-09T03:18:44.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.567+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670808cfd0 con 0x7f67080d2480 2024-05-09T03:18:44.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.568+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f66f803c6d0 con 0x7f67080d2480 2024-05-09T03:18:44.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.569+0000 7f66effff640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66f40762d0 0x7f66f40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.569+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7f66f80b8920 con 0x7f67080d2480 2024-05-09T03:18:44.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.569+0000 7f6706ffd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66f40762d0 0x7f66f40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.569+0000 7f6706ffd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66f40762d0 0x7f66f40787e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f66fc007fd0 tx=0x7f66fc007960 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.573+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f670808cfd0 con 0x7f67080d2480 2024-05-09T03:18:44.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.807+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f670806a270 con 0x7f67080d2480 2024-05-09T03:18:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.807+0000 7f66effff640 1 -- 172.21.15.46:0/2065678176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) ==== 95+0+4343 (secure 0 0 0) 0x7f66f80819e0 con 0x7f67080d2480 2024-05-09T03:18:44.809 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:44.809 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":238,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:44.067407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 238 2024-05-09T03:18:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.809+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66f40762d0 msgr2=0x7f66f40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.809+0000 7f670e0bd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66f40762d0 0x7f66f40787e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f66fc007fd0 tx=0x7f66fc007960 comp rx=0 tx=0).stop 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.809+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 msgr2=0x7f67081622b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.809+0000 7f670e0bd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 0x7f67081622b0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f66f8035040 tx=0x7f66f80046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 shutdown_connections 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6708164c70 0x7f67080cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f670815dfe0 0x7f6708152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f66f40762d0 0x7f66f40787e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 --2- 172.21.15.46:0/2065678176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67080d2480 0x7f67081622b0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 >> 172.21.15.46:0/2065678176 conn(0x7f6708073db0 msgr2=0x7f670815bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 shutdown_connections 2024-05-09T03:18:44.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.810+0000 7f670e0bd640 1 -- 172.21.15.46:0/2065678176 wait complete. 2024-05-09T03:18:44.823 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:44.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.935+0000 7f0ec8fcf640 1 Processor -- start 2024-05-09T03:18:44.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.936+0000 7f0ec8fcf640 1 -- start start 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec8fcf640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 0x7f0ec40b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec8fcf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec40d2b80 con 0x7f0ec40cf6b0 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec8fcf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec40d2e60 con 0x7f0ec40d0480 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec8fcf640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec40d3140 con 0x7f0ec40b92e0 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0ec40cf6b0 0x7f0ec406d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46472/0 (socket says 172.21.15.46:46472) 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec2d9d640 1 -- 172.21.15.46:0/3583802546 learned_addr learned my addr 172.21.15.46:0/3583802546 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec259c640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 0x7f0ec40b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3335289929 0 0) 0x7f0ec40d2b80 con 0x7f0ec40cf6b0 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.937+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb0003520 con 0x7f0ec40cf6b0 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2929428046 0 0) 0x7f0eb0003520 con 0x7f0ec40cf6b0 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 msgr2=0x7f0ec40b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 0x7f0ec40b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 >> v1:172.21.15.88:6789/0 conn(0x7f0ec40d0480 legacy=0x7f0ec40d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec40d3420 con 0x7f0ec40cf6b0 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f0ebc0016f0 con 0x7f0ec40cf6b0 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 >> v1:172.21.15.46:6789/0 conn(0x7f0ec40cf6b0 legacy=0x7f0ec406d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0003a30 0x7f0eb0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 0x7f0eb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0eb0004d40 0x7f0eb00051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40d3140 con 0x7f0eb0004d40 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb000bc20 con 0x7f0eb0003a30 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb000be20 con 0x7f0ec40b92e0 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec2d9d640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0eb0004d40 0x7f0eb00051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec259c640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0003a30 0x7f0eb0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec1d9b640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 0x7f0eb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec2d9d640 1 -- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 msgr2=0x7f0eb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec2d9d640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec40b92e0 0x7f0eb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec2d9d640 1 -- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0003a30 msgr2=0x7f0eb0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec2d9d640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0003a30 0x7f0eb0003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.938+0000 7f0ec2d9d640 1 -- 172.21.15.46:0/3583802546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb0003520 con 0x7f0eb0004d40 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.939+0000 7f0ec2d9d640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0eb0004d40 0x7f0eb00051f0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00c210 tx=0x7f0eb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27009270911a9153 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.939+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc007d10 con 0x7f0eb0004d40 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.939+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ebc005ad0 con 0x7f0eb0004d40 2024-05-09T03:18:44.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.939+0000 7f0ec159a640 1 -- 172.21.15.46:0/3583802546 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc007070 con 0x7f0eb0004d40 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/3583802546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0eb0004d40 msgr2=0x7f0eb00051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/3583802546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0eb0004d40 0x7f0eb00051f0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00c210 tx=0x7f0eb0005830 comp rx=0 tx=0).stop 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec2d9d640 1 -- 172.21.15.46:0/3583802546 reap_dead start 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/3583802546 shutdown_connections 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/3583802546 >> 172.21.15.46:0/3583802546 conn(0x7f0ec40d3c00 msgr2=0x7f0ec40d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/3583802546 shutdown_connections 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.940+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/3583802546 wait complete. 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.942+0000 7f0ec8fcf640 1 Processor -- start 2024-05-09T03:18:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.942+0000 7f0ec8fcf640 1 -- start start 2024-05-09T03:18:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec8fcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0004d40 0x7f0ec41f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec8fcf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 0x7f0ec41643b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec8fcf640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec4158c50 0x7f0ec409e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec8fcf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40b5530 con 0x7f0ec40a1780 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec8fcf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec40b53b0 con 0x7f0eb0004d40 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec8fcf640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec4165010 con 0x7f0ec4158c50 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 0x7f0ec41643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 0x7f0ec41643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59030/0 (socket says 172.21.15.46:59030) 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec1d9b640 1 -- 172.21.15.46:0/4171486450 learned_addr learned my addr 172.21.15.46:0/4171486450 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec259c640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0004d40 0x7f0ec41f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.943+0000 7f0ec2d9d640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec4158c50 0x7f0ec409e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec1d9b640 1 -- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec4158c50 msgr2=0x7f0ec409e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec1d9b640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec4158c50 0x7f0ec409e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec1d9b640 1 -- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0004d40 msgr2=0x7f0ec41f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec1d9b640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0004d40 0x7f0ec41f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec1d9b640 1 -- 172.21.15.46:0/4171486450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec41f73b0 con 0x7f0ec40a1780 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec1d9b640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 0x7f0ec41643b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00f850 tx=0x7f0eac00fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac01a070 con 0x7f0ec40a1780 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eac001d00 con 0x7f0ec40a1780 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec41648f0 con 0x7f0ec40a1780 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac007050 con 0x7f0ec40a1780 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.944+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec41f70d0 con 0x7f0ec40a1780 2024-05-09T03:18:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.945+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec406a270 con 0x7f0ec40a1780 2024-05-09T03:18:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.946+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0eac01d070 con 0x7f0ec40a1780 2024-05-09T03:18:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.946+0000 7f0ebaffd640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ea40762d0 0x7f0ea40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.947+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7f0eac097680 con 0x7f0ec40a1780 2024-05-09T03:18:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.947+0000 7f0ec259c640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ea40762d0 0x7f0ea40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.947+0000 7f0ec259c640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ea40762d0 0x7f0ea40787e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4071390 tx=0x7f0eb400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:44.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:44.951+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0eac060060 con 0x7f0ec40a1780 2024-05-09T03:18:45.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.184+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ec40725e0 con 0x7f0ec40a1780 2024-05-09T03:18:45.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.185+0000 7f0ebaffd640 1 -- 172.21.15.46:0/4171486450 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) ==== 95+0+5169 (secure 0 0 0) 0x7f0eac064360 con 0x7f0ec40a1780 2024-05-09T03:18:45.187 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:45.187 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":239,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9261,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2638610471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2638610471},{"type":"v1","addr":"172.21.15.46:6834","nonce":2638610471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":239}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:44.067407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:45.187 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 239 2024-05-09T03:18:45.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ea40762d0 msgr2=0x7f0ea40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ea40762d0 0x7f0ea40787e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4071390 tx=0x7f0eb400b040 comp rx=0 tx=0).stop 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 msgr2=0x7f0ec41643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 0x7f0ec41643b0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0eac00f850 tx=0x7f0eac00fd10 comp rx=0 tx=0).stop 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 shutdown_connections 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ec4158c50 0x7f0ec409e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ea40762d0 0x7f0ea40787e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ec40a1780 0x7f0ec41643b0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 --2- 172.21.15.46:0/4171486450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb0004d40 0x7f0ec41f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 >> 172.21.15.46:0/4171486450 conn(0x7f0ec40d3c00 msgr2=0x7f0ec40cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 shutdown_connections 2024-05-09T03:18:45.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.187+0000 7f0ec8fcf640 1 -- 172.21.15.46:0/4171486450 wait complete. 2024-05-09T03:18:45.200 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 200.00000000' led to DAMAGED state 2024-05-09T03:18:45.201 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 200.00000001' 2024-05-09T03:18:45.201 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:45.201 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:45.217 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:45.217 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:45.217 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:45.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:45.217 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:45.273 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:45.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:45.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.428+0000 7f0d5415f640 1 Processor -- start 2024-05-09T03:18:45.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.429+0000 7f0d5415f640 1 -- start start 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d5415f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d4c0b7e70 0x7f0d4c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d5415f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d4c0d3df0 0x7f0d4c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d5415f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d4c0b5550 con 0x7f0d4c0b5140 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d5415f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0d4c0d3480 con 0x7f0d4c0d3df0 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d5415f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0d4c0d44e0 con 0x7f0d4c0b7e70 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d4c0d3df0 0x7f0d4c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:45.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d4c0d3df0 0x7f0d4c0b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59034/0 (socket says 172.21.15.46:59034) 2024-05-09T03:18:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 -- 172.21.15.46:0/2844581326 learned_addr learned my addr 172.21.15.46:0/2844581326 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d516fa640 1 --2- 172.21.15.46:0/2844581326 >> v2:172.21.15.88:3300/0 conn(0x7f0d4c0b7e70 0x7f0d4c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 -- 172.21.15.46:0/2844581326 >> v2:172.21.15.88:3300/0 conn(0x7f0d4c0b7e70 msgr2=0x7f0d4c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 --2- 172.21.15.46:0/2844581326 >> v2:172.21.15.88:3300/0 conn(0x7f0d4c0b7e70 0x7f0d4c0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 -- 172.21.15.46:0/2844581326 >> v1:172.21.15.88:6789/0 conn(0x7f0d4c0b5140 legacy=0x7f0d4c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.430+0000 7f0d526fc640 1 -- 172.21.15.46:0/2844581326 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c0d47c0 con 0x7f0d4c0d3df0 2024-05-09T03:18:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.431+0000 7f0d526fc640 1 --2- 172.21.15.46:0/2844581326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c0d3df0 0x7f0d4c0b9380 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0d0290 tx=0x7f0d4402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae01696e055e8379 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.431+0000 7f0d50ef9640 1 -- 172.21.15.46:0/2844581326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44036070 con 0x7f0d4c0d3df0 2024-05-09T03:18:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.431+0000 7f0d50ef9640 1 -- 172.21.15.46:0/2844581326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d44031600 con 0x7f0d4c0d3df0 2024-05-09T03:18:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.431+0000 7f0d50ef9640 1 -- 172.21.15.46:0/2844581326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d440319f0 con 0x7f0d4c0d3df0 2024-05-09T03:18:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 -- 172.21.15.46:0/2844581326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c0d3df0 msgr2=0x7f0d4c0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:45.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 --2- 172.21.15.46:0/2844581326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c0d3df0 0x7f0d4c0b9380 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0d0290 tx=0x7f0d4402a7c0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 -- 172.21.15.46:0/2844581326 shutdown_connections 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 --2- 172.21.15.46:0/2844581326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c0d3df0 0x7f0d4c0b9380 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 --2- 172.21.15.46:0/2844581326 >> v2:172.21.15.88:3300/0 conn(0x7f0d4c0b7e70 0x7f0d4c0d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 -- 172.21.15.46:0/2844581326 >> 172.21.15.46:0/2844581326 conn(0x7f0d4c08d3f0 msgr2=0x7f0d4c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 -- 172.21.15.46:0/2844581326 shutdown_connections 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.432+0000 7f0d5415f640 1 -- 172.21.15.46:0/2844581326 wait complete. 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.434+0000 7f0d5415f640 1 Processor -- start 2024-05-09T03:18:45.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.434+0000 7f0d5415f640 1 -- start start 2024-05-09T03:18:45.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d5415f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d4c0b5140 0x7f0d4c0c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d5415f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d4c0d3df0 0x7f0d4c156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d5415f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 0x7f0d4c162790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d5415f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c0d4200 con 0x7f0d4c162380 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d5415f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4c0d4d80 con 0x7f0d4c0b5140 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d5415f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4c0d5580 con 0x7f0d4c0d3df0 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d526fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 0x7f0d4c162790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d526fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 0x7f0d4c162790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59042/0 (socket says 172.21.15.46:59042) 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.435+0000 7f0d526fc640 1 -- 172.21.15.46:0/3883880766 learned_addr learned my addr 172.21.15.46:0/3883880766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d51efb640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d4c0b5140 0x7f0d4c0c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d516fa640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d4c0d3df0 0x7f0d4c156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d526fc640 1 -- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d4c0d3df0 msgr2=0x7f0d4c156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d526fc640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d4c0d3df0 0x7f0d4c156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d526fc640 1 -- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d4c0b5140 msgr2=0x7f0d4c0c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d526fc640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d4c0b5140 0x7f0d4c0c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d526fc640 1 -- 172.21.15.46:0/3883880766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4c162cd0 con 0x7f0d4c162380 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d51efb640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d4c0b5140 0x7f0d4c0c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d526fc640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 0x7f0d4c162790 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f0d4402a800 tx=0x7f0d44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44036050 con 0x7f0d4c162380 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d44042070 con 0x7f0d4c162380 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d4c1f6300 con 0x7f0d4c162380 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44009510 con 0x7f0d4c162380 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.436+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d4c15b8b0 con 0x7f0d4c162380 2024-05-09T03:18:45.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.437+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d4c1f64d0 con 0x7f0d4c162380 2024-05-09T03:18:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.438+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0d44004470 con 0x7f0d4c162380 2024-05-09T03:18:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.439+0000 7f0d4a7fc640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d380762d0 0x7f0d380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.439+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7f0d440b8db0 con 0x7f0d4c162380 2024-05-09T03:18:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.439+0000 7f0d51efb640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d380762d0 0x7f0d380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:45.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.439+0000 7f0d51efb640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d380762d0 0x7f0d380787e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0016a0 tx=0x7f0d3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.443+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0d4c162cd0 con 0x7f0d4c162380 2024-05-09T03:18:45.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:45.675+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0d4c06a270 con 0x7f0d4c162380 2024-05-09T03:18:46.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.076+0000 7f0d4a7fc640 1 -- 172.21.15.46:0/3883880766 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v240) ==== 161+0+0 (secure 0 0 0) 0x7f0d44081de0 con 0x7f0d4c162380 2024-05-09T03:18:46.079 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d380762d0 msgr2=0x7f0d380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d380762d0 0x7f0d380787e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0016a0 tx=0x7f0d3c009040 comp rx=0 tx=0).stop 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 msgr2=0x7f0d4c162790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 0x7f0d4c162790 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f0d4402a800 tx=0x7f0d44004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 shutdown_connections 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d380762d0 0x7f0d380787e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d4c162380 0x7f0d4c162790 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0d4c0d3df0 0x7f0d4c156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 --2- 172.21.15.46:0/3883880766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d4c0b5140 0x7f0d4c0c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 >> 172.21.15.46:0/3883880766 conn(0x7f0d4c08d3f0 msgr2=0x7f0d4c0b7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:46.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 shutdown_connections 2024-05-09T03:18:46.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.079+0000 7f0d5415f640 1 -- 172.21.15.46:0/3883880766 wait complete. 2024-05-09T03:18:46.093 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:46.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.209+0000 7f81847ce640 1 Processor -- start 2024-05-09T03:18:46.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.209+0000 7f81847ce640 1 -- start start 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f81847ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 0x7f817c0cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f81847ce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f817c0cf4f0 0x7f817c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f81847ce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c09d470 con 0x7f817c0d3730 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f81847ce640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f817c09cfe0 con 0x7f817c0cf4f0 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f81847ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f817c0b94a0 con 0x7f817c0ceba0 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f8181d69640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 0x7f817c0cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f8182d6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f817c0cf4f0 0x7f817c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f8181d69640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 0x7f817c0cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:41320/0 (socket says 172.21.15.46:41320) 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f818256a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f817c0d3730 0x7f817c06e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:37332/0 (socket says 172.21.15.46:37332) 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f8182d6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f817c0cf4f0 0x7f817c0cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:47164/0 (socket says 172.21.15.46:47164) 2024-05-09T03:18:46.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.211+0000 7f8181d69640 1 -- 172.21.15.46:0/3146951689 learned_addr learned my addr 172.21.15.46:0/3146951689 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:46.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.212+0000 7f8182d6b640 1 -- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 msgr2=0x7f817c0cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:46.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.212+0000 7f8182d6b640 1 --2- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 0x7f817c0cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.212+0000 7f8182d6b640 1 -- 172.21.15.46:0/3146951689 >> v1:172.21.15.88:6789/0 conn(0x7f817c0d3730 legacy=0x7f817c06e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:46.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.212+0000 7f8182d6b640 1 -- 172.21.15.46:0/3146951689 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0d11b0 con 0x7f817c0cf4f0 2024-05-09T03:18:46.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.212+0000 7f8181d69640 1 --2- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 0x7f817c0cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.212+0000 7f8182d6b640 1 --2- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c0cf900 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f81740097a0 tx=0x7f817402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d48763ceb74ba81 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f8181568640 1 -- 172.21.15.46:0/3146951689 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174037070 con 0x7f817c0cf4f0 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f8181568640 1 -- 172.21.15.46:0/3146951689 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817402ae50 con 0x7f817c0cf4f0 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f8181568640 1 -- 172.21.15.46:0/3146951689 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174032660 con 0x7f817c0cf4f0 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 -- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 msgr2=0x7f817c0cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 --2- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c0cf900 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f81740097a0 tx=0x7f817402a100 comp rx=0 tx=0).stop 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 -- 172.21.15.46:0/3146951689 shutdown_connections 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 --2- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c0cf900 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 --2- 172.21.15.46:0/3146951689 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0ceba0 0x7f817c0cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 -- 172.21.15.46:0/3146951689 >> 172.21.15.46:0/3146951689 conn(0x7f817c0d05f0 msgr2=0x7f817c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 -- 172.21.15.46:0/3146951689 shutdown_connections 2024-05-09T03:18:46.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.213+0000 7f81847ce640 1 -- 172.21.15.46:0/3146951689 wait complete. 2024-05-09T03:18:46.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.215+0000 7f81847ce640 1 Processor -- start 2024-05-09T03:18:46.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.215+0000 7f81847ce640 1 -- start start 2024-05-09T03:18:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f81847ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 0x7f817c160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f81847ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f81847ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0d3730 0x7f817c161050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f81847ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0b9160 con 0x7f817c0ceba0 2024-05-09T03:18:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f81847ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f817c0d1490 con 0x7f817c0cf4f0 2024-05-09T03:18:46.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f81847ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f817c08cd60 con 0x7f817c0d3730 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 0x7f817c160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 0x7f817c160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45602/0 (socket says 172.21.15.46:45602) 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 -- 172.21.15.46:0/1309560763 learned_addr learned my addr 172.21.15.46:0/1309560763 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f8182d6b640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0d3730 0x7f817c161050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f8181d69640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 -- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0d3730 msgr2=0x7f817c161050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0d3730 0x7f817c161050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 -- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 msgr2=0x7f817c15b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c15b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 -- 172.21.15.46:0/1309560763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0c7cf0 con 0x7f817c0ceba0 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f818256a640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 0x7f817c160520 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f816c00c7d0 tx=0x7f816c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:46.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.217+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c017070 con 0x7f817c0ceba0 2024-05-09T03:18:46.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.218+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817c0c7fd0 con 0x7f817c0ceba0 2024-05-09T03:18:46.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.218+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816c00a5e0 con 0x7f817c0ceba0 2024-05-09T03:18:46.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.218+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c00a990 con 0x7f817c0ceba0 2024-05-09T03:18:46.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.218+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817c0c8590 con 0x7f817c0ceba0 2024-05-09T03:18:46.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.218+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f817c0b65b0 con 0x7f817c0ceba0 2024-05-09T03:18:46.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.219+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f816c01a070 con 0x7f817c0ceba0 2024-05-09T03:18:46.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.220+0000 7f817affd640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81680762d0 0x7f81680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:46.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.220+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7f816c097c60 con 0x7f817c0ceba0 2024-05-09T03:18:46.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.220+0000 7f8181d69640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81680762d0 0x7f81680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:46.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.221+0000 7f8181d69640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81680762d0 0x7f81680787e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8170009630 tx=0x7f8170009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:46.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.224+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f817c0b65b0 con 0x7f817c0ceba0 2024-05-09T03:18:46.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:46.489+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f817c06a270 con 0x7f817c0ceba0 2024-05-09T03:18:47.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.082+0000 7f817affd640 1 -- 172.21.15.46:0/1309560763 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v241) ==== 101+0+0 (secure 0 0 0) 0x7f817c09d470 con 0x7f817c0ceba0 2024-05-09T03:18:47.083 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:47.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81680762d0 msgr2=0x7f81680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81680762d0 0x7f81680787e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f8170009630 tx=0x7f8170009240 comp rx=0 tx=0).stop 2024-05-09T03:18:47.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 msgr2=0x7f817c160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 0x7f817c160520 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f816c00c7d0 tx=0x7f816c00cc90 comp rx=0 tx=0).stop 2024-05-09T03:18:47.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 shutdown_connections 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f817c0d3730 0x7f817c161050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f817c0cf4f0 0x7f817c15b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81680762d0 0x7f81680787e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 --2- 172.21.15.46:0/1309560763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f817c0ceba0 0x7f817c160520 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 >> 172.21.15.46:0/1309560763 conn(0x7f817c0d05f0 msgr2=0x7f817c0d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 shutdown_connections 2024-05-09T03:18:47.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.084+0000 7f81847ce640 1 -- 172.21.15.46:0/1309560763 wait complete. 2024-05-09T03:18:47.099 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:47.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.158+0000 7fcad9e3bf40 1 Processor -- start 2024-05-09T03:18:47.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.158+0000 7fcad9e3bf40 1 -- start start 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad9e3bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e92a7e0 0x55980e957c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad9e3bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 0x55980e95a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad9e3bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e973010 0x55980ea95ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad9e3bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55980e9714d0 con 0x55980e95c7b0 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad9e3bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55980e8d3f50 con 0x55980e973010 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad9e3bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55980e9768a0 con 0x55980e92a7e0 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 0x55980e95a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 0x55980e95a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45612/0 (socket says 172.21.15.46:45612) 2024-05-09T03:18:47.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 -- 172.21.15.46:0/2183097819 learned_addr learned my addr 172.21.15.46:0/2183097819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3905640 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e92a7e0 0x55980e957c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad4106640 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e973010 0x55980ea95ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 -- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e92a7e0 msgr2=0x55980e957c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e92a7e0 0x55980e957c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 -- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e973010 msgr2=0x55980ea95ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e973010 0x55980ea95ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 -- 172.21.15.46:0/2183097819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55980e9f65f0 con 0x55980e95c7b0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad4106640 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e973010 0x55980ea95ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad3104640 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 0x55980e95a710 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fcac8000c00 tx=0x7fcac8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5893eac54ed151 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.160+0000 7fcad2102640 1 -- 172.21.15.46:0/2183097819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8052a90 con 0x55980e95c7b0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad2102640 1 -- 172.21.15.46:0/2183097819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac8052c50 con 0x55980e95c7b0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad2102640 1 -- 172.21.15.46:0/2183097819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8067480 con 0x55980e95c7b0 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2183097819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 msgr2=0x55980e95a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 0x55980e95a710 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fcac8000c00 tx=0x7fcac8051000 comp rx=0 tx=0).stop 2024-05-09T03:18:47.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2183097819 shutdown_connections 2024-05-09T03:18:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e973010 0x55980ea95ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e95c7b0 0x55980e95a710 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2183097819 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e92a7e0 0x55980e957c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2183097819 >> 172.21.15.46:0/2183097819 conn(0x55980e84fac0 msgr2=0x55980e974a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2183097819 shutdown_connections 2024-05-09T03:18:47.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.161+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2183097819 wait complete. 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.162+0000 7fcad9e3bf40 1 Processor -- start 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.162+0000 7fcad9e3bf40 1 -- start start 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.163+0000 7fcad9e3bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e92a7e0 0x55980e9fcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.163+0000 7fcad9e3bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e95c7b0 0x55980e957a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.163+0000 7fcad9e3bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e973010 0x55980e957fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.163+0000 7fcad9e3bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55980e975570 con 0x55980e92a7e0 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.163+0000 7fcad9e3bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55980e970a10 con 0x55980e95c7b0 2024-05-09T03:18:47.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.163+0000 7fcad9e3bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55980e9fc050 con 0x55980e973010 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e92a7e0 0x55980e9fcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e92a7e0 0x55980e9fcb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45622/0 (socket says 172.21.15.46:45622) 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 learned_addr learned my addr 172.21.15.46:0/2856786388 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e95c7b0 0x55980e957a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad4106640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e973010 0x55980e957fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e973010 msgr2=0x55980e957fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55980e973010 0x55980e957fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e95c7b0 msgr2=0x55980e957a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55980e95c7b0 0x55980e957a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55980e9f9890 con 0x55980e92a7e0 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e92a7e0 0x55980e9fcb50 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fcac400b480 tx=0x7fcac400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad1100640 1 -- 172.21.15.46:0/2856786388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4017020 con 0x55980e92a7e0 2024-05-09T03:18:47.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad1100640 1 -- 172.21.15.46:0/2856786388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac4012660 con 0x55980e92a7e0 2024-05-09T03:18:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55980e9fd800 con 0x55980e92a7e0 2024-05-09T03:18:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.164+0000 7fcad1100640 1 -- 172.21.15.46:0/2856786388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac4012a30 con 0x55980e92a7e0 2024-05-09T03:18:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.165+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55980e9fa230 con 0x55980e92a7e0 2024-05-09T03:18:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.166+0000 7fcad1100640 1 -- 172.21.15.46:0/2856786388 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcac401a020 con 0x55980e92a7e0 2024-05-09T03:18:47.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad1100640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcac0074ce0 0x7fcac00771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad1100640 1 -- 172.21.15.46:0/2856786388 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7fcac40970b0 con 0x55980e92a7e0 2024-05-09T03:18:47.170 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcac0074ce0 0x7fcac00771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55980e9fb220 0x55980e9fb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980e9f8ef0 con 0x55980e9fb220 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad4106640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55980e9fb220 0x55980e9fb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.167+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcac0074ce0 0x7fcac00771f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcac8066000 tx=0x7fcac8053000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.168+0000 7fcad4106640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55980e9fb220 0x55980e9fb630 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.170+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v109'753 uv727 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.170+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaa6940 con 0x55980e9fb220 2024-05-09T03:18:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.171+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v109'754 uv754 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.171+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980e9fbb70 con 0x55980e9fb220 2024-05-09T03:18:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.173+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v109'755 uv755 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.175 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.173+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7cf0 con 0x55980e9fb220 2024-05-09T03:18:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.174+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v109'756 uv756 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.174+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa8090 con 0x55980e9fb220 2024-05-09T03:18:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.176+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v109'757 uv757 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.176+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa6ce0 con 0x55980e9fb220 2024-05-09T03:18:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.177+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v109'758 uv758 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.177+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.179+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v109'759 uv733 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.179+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.180+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v109'760 uv760 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.180+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.184 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.182+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v109'761 uv761 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.182+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.184+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v109'762 uv762 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.184+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.185+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v109'763 uv763 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.185+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.187+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v109'764 uv764 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.187+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.188+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v109'765 uv740 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.188+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.190+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v109'766 uv766 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.190+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.191+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v109'767 uv767 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.193 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:47.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.191+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.193+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v109'768 uv768 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.193+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.194+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v109'769 uv769 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.194+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa70d0 con 0x55980e9fb220 2024-05-09T03:18:47.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.196+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v109'770 uv770 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.196+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa8430 con 0x55980e9fb220 2024-05-09T03:18:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.197+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v109'771 uv771 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.197+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.199+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v109'772 uv746 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.199+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.200+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v109'773 uv773 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.200+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.201+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v109'774 uv774 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.203 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.201+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.203+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v109'775 uv775 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.203+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.204+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v109'776 uv776 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.204+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.206+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v109'777 uv777 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.206+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.207+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v109'778 uv752 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.207+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.209+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v109'779 uv779 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.209+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.210+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v109'780 uv780 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.212 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:47.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.210+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.212+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v109'781 uv781 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.212+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.213+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v109'782 uv782 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.213+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980e9fb220 2024-05-09T03:18:47.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.214+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v109'783 uv783 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.214+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55980eaa8940 0x55980eaaad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.214+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaa7800 con 0x55980eaa8940 2024-05-09T03:18:47.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.215+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55980eaa8940 0x55980eaaad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.215+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55980eaa8940 0x55980eaaad60 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.217+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v109'545 uv527 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.217+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaab2a0 con 0x55980eaa8940 2024-05-09T03:18:47.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.218+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v109'546 uv546 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.218+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaab640 con 0x55980eaa8940 2024-05-09T03:18:47.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.220+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v109'547 uv547 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.222 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:47.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.220+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaabe90 con 0x55980eaa8940 2024-05-09T03:18:47.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.221+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v109'548 uv548 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.221+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaac230 con 0x55980eaa8940 2024-05-09T03:18:47.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.222+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v109'549 uv549 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.222+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaac5d0 con 0x55980eaa8940 2024-05-09T03:18:47.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.224+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v109'550 uv544 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.224+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaac970 con 0x55980eaa8940 2024-05-09T03:18:47.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.225+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v109'551 uv551 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.225+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.227+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v109'552 uv552 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.228 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:47.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.227+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.228+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v109'553 uv553 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.228+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.229+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v109'554 uv554 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.229+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.231+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v109'555 uv537 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.231+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.232+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v109'556 uv556 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.232+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.234+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v109'557 uv557 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.235 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.234+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab0770 con 0x55980eaa8940 2024-05-09T03:18:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.235+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v109'558 uv558 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.235+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab0770 con 0x55980eaa8940 2024-05-09T03:18:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.237+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v109'559 uv559 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.237+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.239+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v109'560 uv543 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.239+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.241+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v109'561 uv561 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.241+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.242+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v109'562 uv562 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.243 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:47.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.242+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaadd20 con 0x55980eaa8940 2024-05-09T03:18:47.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.243+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v109'563 uv563 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.243+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaadd20 con 0x55980eaa8940 2024-05-09T03:18:47.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.245+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v109'564 uv564 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.245+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980eaa8940 2024-05-09T03:18:47.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.246+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v109'565 uv565 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eaa8940 2024-05-09T03:18:47.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.246+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980e9fb220 2024-05-09T03:18:47.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.247+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v109'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.247+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980e9fb220 2024-05-09T03:18:47.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.248+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v109'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.248+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980e9fb220 2024-05-09T03:18:47.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.250+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v109'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.251 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:47.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.250+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaadd20 con 0x55980e9fb220 2024-05-09T03:18:47.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.251+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v109'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.251+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaadd20 con 0x55980e9fb220 2024-05-09T03:18:47.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.252+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v109'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.252+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaacd10 con 0x55980e9fb220 2024-05-09T03:18:47.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.253+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v109'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.253+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55980eaad220 0x55980eaaf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.253+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaafcf0 con 0x55980eaad220 2024-05-09T03:18:47.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.253+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55980eaad220 0x55980eaaf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.253+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55980eaad220 0x55980eaaf6c0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.254+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v109'110 uv109 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.254+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab0200 con 0x55980eaad220 2024-05-09T03:18:47.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.255+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v109'111 uv111 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.255+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab06c0 con 0x55980eaad220 2024-05-09T03:18:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.256+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v109'112 uv112 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.258 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.257+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eaabac0 con 0x55980eaad220 2024-05-09T03:18:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.258+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v109'113 uv113 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.258+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab1020 con 0x55980e9fb220 2024-05-09T03:18:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.259+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v109'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.259+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab1020 con 0x55980e9fb220 2024-05-09T03:18:47.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.261+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v109'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.261+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab1020 con 0x55980e9fb220 2024-05-09T03:18:47.263 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:47.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.262+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v109'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.262+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab1020 con 0x55980e9fb220 2024-05-09T03:18:47.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.263+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v109'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.263+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab1020 con 0x55980e9fb220 2024-05-09T03:18:47.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.265+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v109'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.265+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab1020 con 0x55980e9fb220 2024-05-09T03:18:47.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.266+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v109'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.266+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55980eab1920 0x55980eab3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.266+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab4280 con 0x55980eab1920 2024-05-09T03:18:47.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.266+0000 7fcad4106640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55980eab1920 0x55980eab3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.266+0000 7fcad4106640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55980eab1920 0x55980eab3d40 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.268+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v109'271 uv265 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.268+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab42c0 con 0x55980eab1920 2024-05-09T03:18:47.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.269+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v109'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.269+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.271 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:47.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.270+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v109'273 uv273 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.270+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab56f0 con 0x55980eab1920 2024-05-09T03:18:47.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.271+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v109'274 uv274 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.271+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab56f0 con 0x55980eab1920 2024-05-09T03:18:47.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.272+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v109'275 uv275 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.272+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.273+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v109'276 uv276 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.273+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.274+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v109'277 uv270 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.274+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.275+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v109'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.275+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.278 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:47.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.276+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v109'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.276+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.277+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v109'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.277+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab46e0 con 0x55980eab1920 2024-05-09T03:18:47.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.279+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v109'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.279+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55980eab4d60 0x55980eab7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.279+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab77e0 con 0x55980eab4d60 2024-05-09T03:18:47.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.279+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55980eab4d60 0x55980eab7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.279+0000 7fcad3905640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55980eab4d60 0x55980eab7180 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.280+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v109'122 uv121 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eab4d60 2024-05-09T03:18:47.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.280+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980eab4d60 2024-05-09T03:18:47.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.281+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v109'123 uv123 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eab4d60 2024-05-09T03:18:47.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.281+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980eab4d60 2024-05-09T03:18:47.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.283+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v109'124 uv124 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eab4d60 2024-05-09T03:18:47.284 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:47.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.283+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980eab4d60 2024-05-09T03:18:47.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.284+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v109'125 uv125 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eab4d60 2024-05-09T03:18:47.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.284+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980eab4d60 2024-05-09T03:18:47.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.285+0000 7fcad3905640 1 -- 172.21.15.46:0/2856786388 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v109'126 uv126 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac4023070 con 0x55980eab4d60 2024-05-09T03:18:47.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.285+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980e9fb220 2024-05-09T03:18:47.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.286+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v109'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.286+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980e9fb220 2024-05-09T03:18:47.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.287+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v109'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.287+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980e9fb220 2024-05-09T03:18:47.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.288+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v109'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.290 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:47.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.288+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8bc0 con 0x55980e9fb220 2024-05-09T03:18:47.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.289+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v109'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.289+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8bc0 con 0x55980e9fb220 2024-05-09T03:18:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.290+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v109'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.290+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980e9fb220 2024-05-09T03:18:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.291+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v109'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980e9fb220 2024-05-09T03:18:47.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.291+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab7bb0 con 0x55980eaad220 2024-05-09T03:18:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.292+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v109'243 uv237 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.292+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab7f70 con 0x55980eaad220 2024-05-09T03:18:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.293+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v109'244 uv244 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.293+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8360 con 0x55980eaad220 2024-05-09T03:18:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.295+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v109'245 uv245 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.296 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.295+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8750 con 0x55980eaad220 2024-05-09T03:18:47.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.296+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v109'246 uv246 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.296+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8750 con 0x55980eaad220 2024-05-09T03:18:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.297+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v109'247 uv247 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.297+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab84a0 con 0x55980eaad220 2024-05-09T03:18:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.298+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v109'248 uv242 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.298+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eab84a0 con 0x55980eaad220 2024-05-09T03:18:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.299+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v109'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.299+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8450 con 0x55980eaad220 2024-05-09T03:18:47.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.300+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v109'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.302 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:47.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.300+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eab8450 con 0x55980eaad220 2024-05-09T03:18:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.301+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v109'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.301+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabb230 con 0x55980eaad220 2024-05-09T03:18:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.302+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v109'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.303+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55980eabb3a0 0x55980eabb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.303+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eabb3a0 2024-05-09T03:18:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.303+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55980eabb3a0 0x55980eabb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.303+0000 7fcad3104640 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55980eabb3a0 0x55980eabb8f0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.304+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v109'107 uv106 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eabb3a0 2024-05-09T03:18:47.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.304+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eabb3a0 2024-05-09T03:18:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.305+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v109'108 uv108 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eabb3a0 2024-05-09T03:18:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.305+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eabb3a0 2024-05-09T03:18:47.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.307+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v109'109 uv109 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eabb3a0 2024-05-09T03:18:47.308 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:47.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.307+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eabb3a0 2024-05-09T03:18:47.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.308+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v109'110 uv110 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eabb3a0 2024-05-09T03:18:47.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.308+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eaad220 2024-05-09T03:18:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.309+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v109'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.309+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eaad220 2024-05-09T03:18:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.312+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v109'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.312+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eaad220 2024-05-09T03:18:47.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.313+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v109'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.315 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:47.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.313+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eaad220 2024-05-09T03:18:47.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.314+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v109'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.314+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eaad220 2024-05-09T03:18:47.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.315+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v109'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.316+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eaad220 2024-05-09T03:18:47.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.317+0000 7fcad3104640 1 -- 172.21.15.46:0/2856786388 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v109'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcac8050050 con 0x55980eaad220 2024-05-09T03:18:47.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.317+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.318+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v109'125 uv124 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.318+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.319+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v109'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.319+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.320+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v109'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.322 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:47.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.320+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.322+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v109'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.322+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.323+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v109'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.323+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.325+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v109'302 uv294 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.325+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.326+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v109'303 uv303 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.326+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.327+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v109'304 uv304 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.329 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:47.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.327+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabbe30 con 0x55980eab1920 2024-05-09T03:18:47.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.328+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v109'305 uv305 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.328+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabbe30 con 0x55980eab1920 2024-05-09T03:18:47.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.329+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v109'306 uv306 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.329+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabbe30 con 0x55980eab1920 2024-05-09T03:18:47.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.331+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v109'307 uv301 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.331+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.332+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v109'308 uv308 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.332+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.333+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v109'309 uv309 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.335 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.333+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.334+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v109'310 uv310 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.334+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.335+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v109'311 uv311 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.335+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabadb0 con 0x55980eab1920 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.336+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v109'312 uv312 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.336+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55980eabbe30 con 0x55980eab1920 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v109'313 uv313 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcacc00b4f0 con 0x55980eab1920 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55980e9fb220 msgr2=0x55980e9fb630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55980e9fb220 0x55980e9fb630 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55980eaad220 msgr2=0x55980eaaf6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55980eaad220 0x55980eaaf6c0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55980eabb3a0 msgr2=0x55980eabb8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55980eabb3a0 0x55980eabb8f0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55980eaa8940 msgr2=0x55980eaaad60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55980eaa8940 0x55980eaaad60 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 reap_dead start 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55980eab1920 msgr2=0x55980eab3d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55980eab1920 0x55980eab3d40 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55980eab4d60 msgr2=0x55980eab7180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.337+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55980eab4d60 0x55980eab7180 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcac0074ce0 msgr2=0x7fcac00771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcac0074ce0 0x7fcac00771f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fcac8066000 tx=0x7fcac8053000 comp rx=0 tx=0).stop 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e92a7e0 msgr2=0x55980e9fcb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 --2- 172.21.15.46:0/2856786388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55980e92a7e0 0x55980e9fcb50 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fcac400b480 tx=0x7fcac400b940 comp rx=0 tx=0).stop 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad4106640 1 -- 172.21.15.46:0/2856786388 reap_dead start 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 shutdown_connections 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 >> 172.21.15.46:0/2856786388 conn(0x55980e84fac0 msgr2=0x55980e974a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 shutdown_connections 2024-05-09T03:18:47.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.338+0000 7fcad9e3bf40 1 -- 172.21.15.46:0/2856786388 wait complete. 2024-05-09T03:18:47.343 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 200.00000001 0 2024-05-09T03:18:47.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.438+0000 7ff5d431df40 1 Processor -- start 2024-05-09T03:18:47.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.438+0000 7ff5d431df40 1 -- start start 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5d431df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d513a0 0x5557b6ddc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5d431df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 0x5557b6d563c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5d431df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d56900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5d431df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5557b6d5a5f0 con 0x5557b6d595f0 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5d431df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5557b6cb7f50 con 0x5557b6de4a40 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5d431df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5557b6d5a8d0 con 0x5557b6d513a0 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 0x5557b6d563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 0x5557b6d563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45634/0 (socket says 172.21.15.46:45634) 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 -- 172.21.15.46:0/557431281 learned_addr learned my addr 172.21.15.46:0/557431281 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5cdb37640 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d56900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5cd336640 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d513a0 0x5557b6ddc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 -- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d513a0 msgr2=0x5557b6ddc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d513a0 0x5557b6ddc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 -- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 msgr2=0x5557b6d56900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d56900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.439+0000 7ff5ccb35640 1 -- 172.21.15.46:0/557431281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557b6de7440 con 0x5557b6d595f0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 0x5557b6d563c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4001e50 tx=0x7ff5c400b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7530b9ea4f3a7dbd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5bf7fe640 1 -- 172.21.15.46:0/557431281 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4008070 con 0x5557b6d595f0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5bf7fe640 1 -- 172.21.15.46:0/557431281 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c400d500 con 0x5557b6d595f0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5bf7fe640 1 -- 172.21.15.46:0/557431281 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400d910 con 0x5557b6d595f0 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 -- 172.21.15.46:0/557431281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 msgr2=0x5557b6d563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 0x5557b6d563c0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4001e50 tx=0x7ff5c400b6a0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 -- 172.21.15.46:0/557431281 shutdown_connections 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d56900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d595f0 0x5557b6d563c0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 --2- 172.21.15.46:0/557431281 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d513a0 0x5557b6ddc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 -- 172.21.15.46:0/557431281 >> 172.21.15.46:0/557431281 conn(0x5557b6c33ac0 msgr2=0x5557b6d3c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 -- 172.21.15.46:0/557431281 shutdown_connections 2024-05-09T03:18:47.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.440+0000 7ff5d431df40 1 -- 172.21.15.46:0/557431281 wait complete. 2024-05-09T03:18:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.441+0000 7ff5d431df40 1 Processor -- start 2024-05-09T03:18:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.441+0000 7ff5d431df40 1 -- start start 2024-05-09T03:18:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d513a0 0x5557b6dd2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d595f0 0x5557b6d3ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d3b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5557b6d3e730 con 0x5557b6d513a0 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5557b6d5b710 con 0x5557b6de4a40 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5557b6e7f480 con 0x5557b6d595f0 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d513a0 0x5557b6dd2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d513a0 0x5557b6dd2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45650/0 (socket says 172.21.15.46:45650) 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 -- 172.21.15.46:0/1502213189 learned_addr learned my addr 172.21.15.46:0/1502213189 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d595f0 0x5557b6d3ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cdb37640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d3b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 -- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d595f0 msgr2=0x5557b6d3ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d595f0 0x5557b6d3ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 -- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 msgr2=0x5557b6d3b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d3b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 -- 172.21.15.46:0/1502213189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557b6dd32c0 con 0x5557b6d513a0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5557b6d595f0 0x5557b6d3ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cdb37640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5557b6de4a40 0x5557b6d3b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5cd336640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d513a0 0x5557b6dd2d80 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009fa0 tx=0x7ff5c00084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5be7fc640 1 -- 172.21.15.46:0/1502213189 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0015660 con 0x5557b6d513a0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5be7fc640 1 -- 172.21.15.46:0/1502213189 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c001a070 con 0x5557b6d513a0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5557b6d12fc0 con 0x5557b6d513a0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5be7fc640 1 -- 172.21.15.46:0/1502213189 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0015a40 con 0x5557b6d513a0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.443+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557b6ddd2b0 con 0x5557b6d513a0 2024-05-09T03:18:47.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.445+0000 7ff5be7fc640 1 -- 172.21.15.46:0/1502213189 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff5c0013030 con 0x5557b6d513a0 2024-05-09T03:18:47.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.445+0000 7ff5be7fc640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5b0074ce0 0x7ff5b00771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5be7fc640 1 -- 172.21.15.46:0/1502213189 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7ff5c0096b60 con 0x5557b6d513a0 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5b0074ce0 0x7ff5b00771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5d431df40 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5557b6dddd90 0x5557b6e7e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5557b6e7eaf0 con 0x5557b6dddd90 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5cdb37640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5557b6dddd90 0x5557b6e7e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5ccb35640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5b0074ce0 0x7ff5b00771f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x5557b6de6c60 tx=0x7ff5c4009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.446+0000 7ff5cdb37640 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5557b6dddd90 0x5557b6e7e5b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5cdb37640 1 -- 172.21.15.46:0/1502213189 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 200.00000001 [truncate 0] v109'566 uv566 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff5c8052990 con 0x5557b6dddd90 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5557b6dddd90 msgr2=0x5557b6e7e5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5557b6dddd90 0x5557b6e7e5b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5b0074ce0 msgr2=0x7ff5b00771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5b0074ce0 0x7ff5b00771f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x5557b6de6c60 tx=0x7ff5c4009000 comp rx=0 tx=0).stop 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d513a0 msgr2=0x5557b6dd2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 --2- 172.21.15.46:0/1502213189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b6d513a0 0x5557b6dd2d80 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009fa0 tx=0x7ff5c00084c0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5cdb37640 1 -- 172.21.15.46:0/1502213189 reap_dead start 2024-05-09T03:18:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 shutdown_connections 2024-05-09T03:18:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 >> 172.21.15.46:0/1502213189 conn(0x5557b6c33ac0 msgr2=0x5557b6de5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 shutdown_connections 2024-05-09T03:18:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.448+0000 7ff5d431df40 1 -- 172.21.15.46:0/1502213189 wait complete. 2024-05-09T03:18:47.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.569+0000 7f0dc5eee640 1 Processor -- start 2024-05-09T03:18:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.569+0000 7f0dc5eee640 1 -- start start 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.570+0000 7f0dc5eee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00bb770 0x7f0dc00d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.570+0000 7f0dc5eee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc0164900 con 0x7f0dc00d3180 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.570+0000 7f0dc5eee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc0164be0 con 0x7f0dc0163eb0 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.570+0000 7f0dc5eee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc00d4bb0 con 0x7f0dc00bb770 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0dc00d3180 0x7f0dc00b9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46038/0 (socket says 172.21.15.46:46038) 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbffff640 1 -- 172.21.15.46:0/3340978148 learned_addr learned my addr 172.21.15.46:0/3340978148 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbf7fe640 1 --2- 172.21.15.46:0/3340978148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00bb770 0x7f0dc00d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031211358 0 0) 0x7f0dc0164900 con 0x7f0dc00d3180 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dac003520 con 0x7f0dc00d3180 2024-05-09T03:18:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2356993703 0 0) 0x7f0dc0164be0 con 0x7f0dc0163eb0 2024-05-09T03:18:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dc0164900 con 0x7f0dc0163eb0 2024-05-09T03:18:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbf7fe640 1 -- 172.21.15.46:0/3340978148 >> v1:172.21.15.88:6789/0 conn(0x7f0dc0163eb0 legacy=0x7f0dc00d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbf7fe640 1 -- 172.21.15.46:0/3340978148 >> v1:172.21.15.46:6789/0 conn(0x7f0dc00d3180 legacy=0x7f0dc00b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.571+0000 7f0dbf7fe640 1 -- 172.21.15.46:0/3340978148 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc00d4840 con 0x7f0dc00bb770 2024-05-09T03:18:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.572+0000 7f0dbf7fe640 1 --2- 172.21.15.46:0/3340978148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00bb770 0x7f0dc00d5040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097d0 tx=0x7f0db002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a17e03d62a6650b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.572+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0037070 con 0x7f0dc00bb770 2024-05-09T03:18:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.572+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db0032440 con 0x7f0dc00bb770 2024-05-09T03:18:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.572+0000 7f0dbeffd640 1 -- 172.21.15.46:0/3340978148 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0032810 con 0x7f0dc00bb770 2024-05-09T03:18:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 -- 172.21.15.46:0/3340978148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00bb770 msgr2=0x7f0dc00d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/3340978148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00bb770 0x7f0dc00d5040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097d0 tx=0x7f0db002a210 comp rx=0 tx=0).stop 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 -- 172.21.15.46:0/3340978148 shutdown_connections 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/3340978148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc00bb770 0x7f0dc00d5040 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 -- 172.21.15.46:0/3340978148 >> 172.21.15.46:0/3340978148 conn(0x7f0dc006e370 msgr2=0x7f0dc006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 -- 172.21.15.46:0/3340978148 shutdown_connections 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.573+0000 7f0dc5eee640 1 -- 172.21.15.46:0/3340978148 wait complete. 2024-05-09T03:18:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.575+0000 7f0dc5eee640 1 Processor -- start 2024-05-09T03:18:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.575+0000 7f0dc5eee640 1 -- start start 2024-05-09T03:18:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dc5eee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc00bb770 0x7f0dc015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dc5eee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 0x7f0dc0151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dc5eee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc015eac0 0x7f0dc0162520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dc5eee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc00d0bc0 con 0x7f0dc0163eb0 2024-05-09T03:18:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dc5eee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc00bbb80 con 0x7f0dc00bb770 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dc5eee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc00d0d40 con 0x7f0dc015eac0 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 0x7f0dc0151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 0x7f0dc0151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45658/0 (socket says 172.21.15.46:45658) 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 -- 172.21.15.46:0/1357264594 learned_addr learned my addr 172.21.15.46:0/1357264594 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dbf7fe640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc00bb770 0x7f0dc015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dbffff640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc015eac0 0x7f0dc0162520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 -- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc015eac0 msgr2=0x7f0dc0162520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc015eac0 0x7f0dc0162520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 -- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc00bb770 msgr2=0x7f0dc015ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc00bb770 0x7f0dc015ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 -- 172.21.15.46:0/1357264594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc015d3b0 con 0x7f0dc0163eb0 2024-05-09T03:18:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0dbf7fe640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc00bb770 0x7f0dc015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.576+0000 7f0db7fff640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 0x7f0dc0151010 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c430 tx=0x7f0da800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.577+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8017070 con 0x7f0dc0163eb0 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.577+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da80052d0 con 0x7f0dc0163eb0 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.577+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc015d690 con 0x7f0dc0163eb0 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.577+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8003830 con 0x7f0dc0163eb0 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.577+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc00c9870 con 0x7f0dc0163eb0 2024-05-09T03:18:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.578+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc006a270 con 0x7f0dc0163eb0 2024-05-09T03:18:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.578+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0da801a070 con 0x7f0dc0163eb0 2024-05-09T03:18:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.579+0000 7f0dbcff9640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0da0076300 0x7f0da0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.579+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7917+0+0 (secure 0 0 0) 0x7f0da8097220 con 0x7f0dc0163eb0 2024-05-09T03:18:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.579+0000 7f0dbf7fe640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0da0076300 0x7f0da0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.580+0000 7f0dbf7fe640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0da0076300 0x7f0da0078810 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0db0034000 tx=0x7f0db002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.584+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0da8060250 con 0x7f0dc0163eb0 2024-05-09T03:18:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:47.822+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0dc00d4130 con 0x7f0dc0163eb0 2024-05-09T03:18:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.087+0000 7f0dbcff9640 1 -- 172.21.15.46:0/1357264594 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v242) ==== 162+0+0 (secure 0 0 0) 0x7f0da80682d0 con 0x7f0dc0163eb0 2024-05-09T03:18:48.089 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:48.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0da0076300 msgr2=0x7f0da0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0da0076300 0x7f0da0078810 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0db0034000 tx=0x7f0db002e000 comp rx=0 tx=0).stop 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 msgr2=0x7f0dc0151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 0x7f0dc0151010 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c430 tx=0x7f0da800c8f0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 shutdown_connections 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0dc015eac0 0x7f0dc0162520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0da0076300 0x7f0da0078810 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0dc0163eb0 0x7f0dc0151010 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 --2- 172.21.15.46:0/1357264594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0dc00bb770 0x7f0dc015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 >> 172.21.15.46:0/1357264594 conn(0x7f0dc006e370 msgr2=0x7f0dc00b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 shutdown_connections 2024-05-09T03:18:48.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.089+0000 7f0dc5eee640 1 -- 172.21.15.46:0/1357264594 wait complete. 2024-05-09T03:18:48.102 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:48.100+0000 7f421cc82640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-05-09T03:18:48.103 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:48.101+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:18:48.103 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:18:48.101+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:18:48.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:48.108 INFO:tasks.ceph.mds.c.smithi046.stderr: -15> 2024-05-09T03:18:48.100+0000 7f421cc82640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-05-09T03:18:48.163 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:18:48.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.221+0000 7fc56e2d9640 1 Processor -- start 2024-05-09T03:18:48.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.221+0000 7fc56e2d9640 1 -- start start 2024-05-09T03:18:48.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.222+0000 7fc56e2d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc5681615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.222+0000 7fc56e2d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5680d2480 0x7fc5680d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.222+0000 7fc56e2d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc568160b20 con 0x7fc56815dfe0 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.222+0000 7fc56e2d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc568161150 con 0x7fc5680d2480 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.222+0000 7fc56e2d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc5680d15f0 con 0x7fc568164c70 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.222+0000 7fc5677fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc56815dfe0 0x7fc5680d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46052/0 (socket says 172.21.15.46:46052) 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc5677fe640 1 -- 172.21.15.46:0/1666156323 learned_addr learned my addr 172.21.15.46:0/1666156323 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc567fff640 1 --2- 172.21.15.46:0/1666156323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc5681615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc56c876640 1 --2- 172.21.15.46:0/1666156323 >> v2:172.21.15.88:3300/0 conn(0x7fc5680d2480 0x7fc5680d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc566ffd640 1 -- 172.21.15.46:0/1666156323 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1669116840 0 0) 0x7fc568160b20 con 0x7fc56815dfe0 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc566ffd640 1 -- 172.21.15.46:0/1666156323 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc550003520 con 0x7fc56815dfe0 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc566ffd640 1 -- 172.21.15.46:0/1666156323 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 148545419 0 0) 0x7fc550003520 con 0x7fc56815dfe0 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc567fff640 1 -- 172.21.15.46:0/1666156323 >> v2:172.21.15.88:3300/0 conn(0x7fc5680d2480 msgr2=0x7fc5680d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc567fff640 1 --2- 172.21.15.46:0/1666156323 >> v2:172.21.15.88:3300/0 conn(0x7fc5680d2480 0x7fc5680d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc567fff640 1 -- 172.21.15.46:0/1666156323 >> v1:172.21.15.46:6789/0 conn(0x7fc56815dfe0 legacy=0x7fc5680d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.223+0000 7fc567fff640 1 -- 172.21.15.46:0/1666156323 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5680d18d0 con 0x7fc568164c70 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.224+0000 7fc567fff640 1 --2- 172.21.15.46:0/1666156323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc5681615e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc55802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7aa83dcb8c66ad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.224+0000 7fc566ffd640 1 -- 172.21.15.46:0/1666156323 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558037070 con 0x7fc568164c70 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.224+0000 7fc566ffd640 1 -- 172.21.15.46:0/1666156323 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55802aba0 con 0x7fc568164c70 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.224+0000 7fc566ffd640 1 -- 172.21.15.46:0/1666156323 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c0016f0 con 0x7fc568164c70 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 -- 172.21.15.46:0/1666156323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 msgr2=0x7fc5681615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/1666156323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc5681615e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc5580097a0 tx=0x7fc55802a110 comp rx=0 tx=0).stop 2024-05-09T03:18:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 -- 172.21.15.46:0/1666156323 shutdown_connections 2024-05-09T03:18:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/1666156323 >> v2:172.21.15.88:3300/0 conn(0x7fc5680d2480 0x7fc5680d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/1666156323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc5681615e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 -- 172.21.15.46:0/1666156323 >> 172.21.15.46:0/1666156323 conn(0x7fc568073db0 msgr2=0x7fc5680741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 -- 172.21.15.46:0/1666156323 shutdown_connections 2024-05-09T03:18:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.225+0000 7fc56e2d9640 1 -- 172.21.15.46:0/1666156323 wait complete. 2024-05-09T03:18:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.227+0000 7fc56e2d9640 1 Processor -- start 2024-05-09T03:18:48.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.227+0000 7fc56e2d9640 1 -- start start 2024-05-09T03:18:48.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.228+0000 7fc56e2d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5680d2480 0x7fc5680c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.228+0000 7fc56e2d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 0x7fc5680ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56e2d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc56815d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56e2d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc5680d2dd0 con 0x7fc56815dfe0 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56e2d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc56815e3f0 con 0x7fc5680d2480 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56e2d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc5680d45c0 con 0x7fc568164c70 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 0x7fc5680ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 0x7fc5680ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45698/0 (socket says 172.21.15.46:45698) 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 -- 172.21.15.46:0/4248731106 learned_addr learned my addr 172.21.15.46:0/4248731106 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56c876640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc56815d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc567fff640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5680d2480 0x7fc5680c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 -- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 msgr2=0x7fc56815d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc56815d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 -- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5680d2480 msgr2=0x7fc5680c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5680d2480 0x7fc5680c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 -- 172.21.15.46:0/4248731106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5680cfe40 con 0x7fc56815dfe0 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56c876640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc56815d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc567fff640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5680d2480 0x7fc5680c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc5677fe640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 0x7fc5680ceb10 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c500 tx=0x7fc55c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c005ce0 con 0x7fc56815dfe0 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55c007070 con 0x7fc56815dfe0 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c007460 con 0x7fc56815dfe0 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5680cf880 con 0x7fc56815dfe0 2024-05-09T03:18:48.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.229+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc568148200 con 0x7fc56815dfe0 2024-05-09T03:18:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.231+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5680cf5f0 con 0x7fc56815dfe0 2024-05-09T03:18:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.231+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc55c007600 con 0x7fc56815dfe0 2024-05-09T03:18:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.232+0000 7fc564ff9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5540762d0 0x7fc5540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.232+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7fc55c097bc0 con 0x7fc56815dfe0 2024-05-09T03:18:48.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.232+0000 7fc567fff640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5540762d0 0x7fc5540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.232+0000 7fc567fff640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5540762d0 0x7fc5540787e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc558034000 tx=0x7fc55802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:48.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.236+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc55c060c20 con 0x7fc56815dfe0 2024-05-09T03:18:48.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.468+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc56808cfd0 con 0x7fc56815dfe0 2024-05-09T03:18:48.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.469+0000 7fc564ff9640 1 -- 172.21.15.46:0/4248731106 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) ==== 95+0+4343 (secure 0 0 0) 0x7fc56808cfd0 con 0x7fc56815dfe0 2024-05-09T03:18:48.471 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:48.471 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":244,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:48.103030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:48.472 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 244 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5540762d0 msgr2=0x7fc5540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5540762d0 0x7fc5540787e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc558034000 tx=0x7fc55802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 msgr2=0x7fc5680ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 0x7fc5680ceb10 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00c500 tx=0x7fc55c00c9c0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 shutdown_connections 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc568164c70 0x7fc56815d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5540762d0 0x7fc5540787e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc56815dfe0 0x7fc5680ceb10 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 --2- 172.21.15.46:0/4248731106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5680d2480 0x7fc5680c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.471+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 >> 172.21.15.46:0/4248731106 conn(0x7fc568073db0 msgr2=0x7fc5680b9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.472+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 shutdown_connections 2024-05-09T03:18:48.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.472+0000 7fc56e2d9640 1 -- 172.21.15.46:0/4248731106 wait complete. 2024-05-09T03:18:48.485 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:48.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:48.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.637+0000 7fd06e772640 1 Processor -- start 2024-05-09T03:18:48.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.637+0000 7fd06e772640 1 -- start start 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.638+0000 7fd06e772640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0680d1c80 0x7fd0680cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.638+0000 7fd06e772640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd0680d2090 0x7fd0680cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd06e772640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0680d4900 con 0x7fd06808cd60 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd06e772640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd0680b5340 con 0x7fd0680d1c80 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd06e772640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd0680b4fd0 con 0x7fd0680d2090 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd0677fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0680d1c80 0x7fd0680cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd067fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd06808cd60 0x7fd0680d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46062/0 (socket says 172.21.15.46:46062) 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd067fff640 1 -- 172.21.15.46:0/839035951 learned_addr learned my addr 172.21.15.46:0/839035951 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd0677fe640 1 -- 172.21.15.46:0/839035951 >> v2:172.21.15.88:3300/0 conn(0x7fd0680d2090 msgr2=0x7fd0680cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd0677fe640 1 --2- 172.21.15.46:0/839035951 >> v2:172.21.15.88:3300/0 conn(0x7fd0680d2090 0x7fd0680cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd0677fe640 1 -- 172.21.15.46:0/839035951 >> v1:172.21.15.46:6789/0 conn(0x7fd06808cd60 legacy=0x7fd0680d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd066ffd640 1 -- 172.21.15.46:0/839035951 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588105625 0 0) 0x7fd0680d4900 con 0x7fd06808cd60 2024-05-09T03:18:48.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.639+0000 7fd0677fe640 1 -- 172.21.15.46:0/839035951 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0680b6a50 con 0x7fd0680d1c80 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.640+0000 7fd0677fe640 1 --2- 172.21.15.46:0/839035951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680cf2b0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0097a0 tx=0x7fd05c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd30fd2a61b950c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.640+0000 7fd066ffd640 1 -- 172.21.15.46:0/839035951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c037070 con 0x7fd0680d1c80 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.640+0000 7fd066ffd640 1 -- 172.21.15.46:0/839035951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c02ae50 con 0x7fd0680d1c80 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.640+0000 7fd066ffd640 1 -- 172.21.15.46:0/839035951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c032660 con 0x7fd0680d1c80 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 -- 172.21.15.46:0/839035951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 msgr2=0x7fd0680cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 --2- 172.21.15.46:0/839035951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680cf2b0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0097a0 tx=0x7fd05c02a140 comp rx=0 tx=0).stop 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 -- 172.21.15.46:0/839035951 shutdown_connections 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 --2- 172.21.15.46:0/839035951 >> v2:172.21.15.88:3300/0 conn(0x7fd0680d2090 0x7fd0680cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 --2- 172.21.15.46:0/839035951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680cf2b0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 -- 172.21.15.46:0/839035951 >> 172.21.15.46:0/839035951 conn(0x7fd0680d3c00 msgr2=0x7fd0680d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 -- 172.21.15.46:0/839035951 shutdown_connections 2024-05-09T03:18:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.641+0000 7fd06e772640 1 -- 172.21.15.46:0/839035951 wait complete. 2024-05-09T03:18:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.642+0000 7fd06e772640 1 Processor -- start 2024-05-09T03:18:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.643+0000 7fd06e772640 1 -- start start 2024-05-09T03:18:48.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06e772640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06808cd60 0x7fd0680b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06e772640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06e772640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680d2090 0x7fd06814ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06e772640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd0680d0440 con 0x7fd0680d1c80 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06e772640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0680d02c0 con 0x7fd06808cd60 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06e772640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd0680d05c0 con 0x7fd0680d2090 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45720/0 (socket says 172.21.15.46:45720) 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 -- 172.21.15.46:0/2673661941 learned_addr learned my addr 172.21.15.46:0/2673661941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd067fff640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06808cd60 0x7fd0680b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd06cd0f640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680d2090 0x7fd06814ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 -- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680d2090 msgr2=0x7fd06814ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680d2090 0x7fd06814ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 -- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06808cd60 msgr2=0x7fd0680b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06808cd60 0x7fd0680b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.644+0000 7fd0677fe640 1 -- 172.21.15.46:0/2673661941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0680caec0 con 0x7fd0680d1c80 2024-05-09T03:18:48.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.645+0000 7fd0677fe640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680b1300 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0098c0 tx=0x7fd05c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:48.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.645+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c042070 con 0x7fd0680d1c80 2024-05-09T03:18:48.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.645+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c03e670 con 0x7fd0680d1c80 2024-05-09T03:18:48.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.645+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0680cb1a0 con 0x7fd0680d1c80 2024-05-09T03:18:48.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.645+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c037040 con 0x7fd0680d1c80 2024-05-09T03:18:48.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.645+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0680cb760 con 0x7fd0680d1c80 2024-05-09T03:18:48.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.646+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06806a270 con 0x7fd0680d1c80 2024-05-09T03:18:48.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.647+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd05c045070 con 0x7fd0680d1c80 2024-05-09T03:18:48.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.647+0000 7fd064ff9640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0540762d0 0x7fd0540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:48.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.647+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7fd05c0b8e10 con 0x7fd0680d1c80 2024-05-09T03:18:48.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.647+0000 7fd067fff640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0540762d0 0x7fd0540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:48.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.648+0000 7fd067fff640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0540762d0 0x7fd0540787e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd058007fd0 tx=0x7fd058007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:48.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.651+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd05c081990 con 0x7fd0680d1c80 2024-05-09T03:18:48.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.882+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0680b8080 con 0x7fd0680d1c80 2024-05-09T03:18:48.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.883+0000 7fd064ff9640 1 -- 172.21.15.46:0/2673661941 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) ==== 95+0+4343 (secure 0 0 0) 0x7fd05c085c90 con 0x7fd0680d1c80 2024-05-09T03:18:48.885 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:48.885 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":244,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:48.103030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:48.885 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 244 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0540762d0 msgr2=0x7fd0540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0540762d0 0x7fd0540787e0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd058007fd0 tx=0x7fd058007de0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 msgr2=0x7fd0680b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680b1300 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0098c0 tx=0x7fd05c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 shutdown_connections 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd0680d2090 0x7fd06814ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd0540762d0 0x7fd0540787e0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0680d1c80 0x7fd0680b1300 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 --2- 172.21.15.46:0/2673661941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd06808cd60 0x7fd0680b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:48.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 >> 172.21.15.46:0/2673661941 conn(0x7fd0680d3c00 msgr2=0x7fd068072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:48.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 shutdown_connections 2024-05-09T03:18:48.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:48.885+0000 7fd06e772640 1 -- 172.21.15.46:0/2673661941 wait complete. 2024-05-09T03:18:48.898 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:18:48.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:49.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.011+0000 7f1195414640 1 Processor -- start 2024-05-09T03:18:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.011+0000 7f1195414640 1 -- start start 2024-05-09T03:18:49.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f1195414640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 0x7f11900b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f1195414640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11900b8130 con 0x7f11900d57a0 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f1195414640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11900d49e0 con 0x7f11900b7d20 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f1195414640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11900d5bb0 con 0x7f1190160610 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f118effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11900d57a0 0x7f11900b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46074/0 (socket says 172.21.15.46:46074) 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f118effd640 1 -- 172.21.15.46:0/2998998437 learned_addr learned my addr 172.21.15.46:0/2998998437 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f118e7fc640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 0x7f11900b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.012+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3664459044 0 0) 0x7f1180001110 con 0x7f11900d57a0 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1178003520 con 0x7f11900d57a0 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3590110287 0 0) 0x7f11900d49e0 con 0x7f11900b7d20 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1180001110 con 0x7f11900b7d20 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3738234904 0 0) 0x7f1178003520 con 0x7f11900d57a0 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 msgr2=0x7f11900b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:49.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 0x7f11900b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 >> v1:172.21.15.88:6789/0 conn(0x7f11900b7d20 legacy=0x7f11900b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11900d4cc0 con 0x7f11900d57a0 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f11800018c0 con 0x7f11900d57a0 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 >> v1:172.21.15.46:6789/0 conn(0x7f11900d57a0 legacy=0x7f11900b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1178003c60 0x7f11780040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1178004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 0x7f117800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11900d5bb0 con 0x7f1178004800 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1178005580 con 0x7f1178003c60 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f117800bf00 con 0x7f1190160610 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118e7fc640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1178004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118effd640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1178003c60 0x7f11780040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118f7fe640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 0x7f117800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118e7fc640 1 -- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 msgr2=0x7f117800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118e7fc640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1190160610 0x7f117800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118e7fc640 1 -- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1178003c60 msgr2=0x7f11780040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118e7fc640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1178003c60 0x7f11780040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.013+0000 7f118e7fc640 1 -- 172.21.15.46:0/2998998437 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1178003520 con 0x7f1178004800 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.014+0000 7f118e7fc640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1178004c50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1184009af0 tx=0x7f1178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe77b17ece8a861 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.014+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1184036070 con 0x7f1178004800 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.014+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1184031400 con 0x7f1178004800 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.014+0000 7f118dffb640 1 -- 172.21.15.46:0/2998998437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1184009690 con 0x7f1178004800 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f1195414640 1 -- 172.21.15.46:0/2998998437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 msgr2=0x7f1178004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f1195414640 1 --2- 172.21.15.46:0/2998998437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1178004c50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1184009af0 tx=0x7f1178005830 comp rx=0 tx=0).stop 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f118f7fe640 1 -- 172.21.15.46:0/2998998437 reap_dead start 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f1195414640 1 -- 172.21.15.46:0/2998998437 shutdown_connections 2024-05-09T03:18:49.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f1195414640 1 -- 172.21.15.46:0/2998998437 >> 172.21.15.46:0/2998998437 conn(0x7f11900743e0 msgr2=0x7f11900d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:49.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f1195414640 1 -- 172.21.15.46:0/2998998437 shutdown_connections 2024-05-09T03:18:49.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.015+0000 7f1195414640 1 -- 172.21.15.46:0/2998998437 wait complete. 2024-05-09T03:18:49.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.017+0000 7f1195414640 1 Processor -- start 2024-05-09T03:18:49.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.017+0000 7f1195414640 1 -- start start 2024-05-09T03:18:49.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f1195414640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1190153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f1195414640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11900b5650 0x7f11900b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f1195414640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11900b5fa0 0x7f1190152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f1195414640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1190160a20 con 0x7f1178004800 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f1195414640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11900d5280 con 0x7f11900b5fa0 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f1195414640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11900d15b0 con 0x7f11900b5650 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f118effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1190153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f118effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1190153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45738/0 (socket says 172.21.15.46:45738) 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.018+0000 7f118effd640 1 -- 172.21.15.46:0/3689707888 learned_addr learned my addr 172.21.15.46:0/3689707888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118f7fe640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11900b5fa0 0x7f1190152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118e7fc640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11900b5650 0x7f11900b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118effd640 1 -- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11900b5650 msgr2=0x7f11900b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118effd640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11900b5650 0x7f11900b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118effd640 1 -- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11900b5fa0 msgr2=0x7f1190152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118effd640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11900b5fa0 0x7f1190152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118effd640 1 -- 172.21.15.46:0/3689707888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1190156e70 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118f7fe640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11900b5fa0 0x7f1190152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f118effd640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1190153cf0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1180007390 tx=0x7f1180007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118000fa50 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1180018070 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1190155dd0 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118000fe50 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.019+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1190156390 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.020+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119006a270 con 0x7f1178004800 2024-05-09T03:18:49.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.021+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f118001d460 con 0x7f1178004800 2024-05-09T03:18:49.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.022+0000 7f11777fe640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f117c0762d0 0x7f117c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.022+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f11800981a0 con 0x7f1178004800 2024-05-09T03:18:49.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.022+0000 7f118e7fc640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f117c0762d0 0x7f117c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.022+0000 7f118e7fc640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f117c0762d0 0x7f117c0787e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1184033000 tx=0x7f1184008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.026+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11900b8130 con 0x7f1178004800 2024-05-09T03:18:49.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.260+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f119015d360 con 0x7f1178004800 2024-05-09T03:18:49.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.261+0000 7f11777fe640 1 -- 172.21.15.46:0/3689707888 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) ==== 95+0+5169 (secure 0 0 0) 0x7f1180061160 con 0x7f1178004800 2024-05-09T03:18:49.263 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:49.263 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":245,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9327,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2827001018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2827001018},{"type":"v1","addr":"172.21.15.46:6834","nonce":2827001018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":245}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:48.103030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:49.263 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 245 2024-05-09T03:18:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f117c0762d0 msgr2=0x7f117c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f117c0762d0 0x7f117c0787e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f1184033000 tx=0x7f1184008030 comp rx=0 tx=0).stop 2024-05-09T03:18:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 msgr2=0x7f1190153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1190153cf0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1180007390 tx=0x7f1180007850 comp rx=0 tx=0).stop 2024-05-09T03:18:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 shutdown_connections 2024-05-09T03:18:49.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11900b5fa0 0x7f1190152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11900b5650 0x7f11900b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f117c0762d0 0x7f117c0787e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 --2- 172.21.15.46:0/3689707888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1178004800 0x7f1190153cf0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 >> 172.21.15.46:0/3689707888 conn(0x7f11900743e0 msgr2=0x7f11900c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:49.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 shutdown_connections 2024-05-09T03:18:49.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.263+0000 7f1195414640 1 -- 172.21.15.46:0/3689707888 wait complete. 2024-05-09T03:18:49.276 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 200.00000001' led to DAMAGED state 2024-05-09T03:18:49.276 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 500.00000000' 2024-05-09T03:18:49.276 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:18:49.276 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:18:49.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:18:49.293 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:49.293 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:18:49.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:49.293 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:18:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:18:49.349 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:18:49.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:18:49.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.504+0000 7f7b651a9640 1 Processor -- start 2024-05-09T03:18:49.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.504+0000 7f7b651a9640 1 -- start start 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b651a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600bb770 0x7f7b600d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b651a9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b60163eb0 0x7f7b601642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b651a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b600d2a70 0x7f7b600d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b651a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7b600d4bb0 con 0x7f7b600d2a70 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b651a9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7b600d33c0 con 0x7f7b60163eb0 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b651a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7b60164900 con 0x7f7b600bb770 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b5f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b600d2a70 0x7f7b600d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b5f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b600d2a70 0x7f7b600d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45748/0 (socket says 172.21.15.46:45748) 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.505+0000 7f7b5f59e640 1 -- 172.21.15.46:0/2699724208 learned_addr learned my addr 172.21.15.46:0/2699724208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5ed9d640 1 --2- 172.21.15.46:0/2699724208 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600bb770 0x7f7b600d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5659c640 1 --2- 172.21.15.46:0/2699724208 >> v2:172.21.15.88:3300/0 conn(0x7f7b60163eb0 0x7f7b601642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5f59e640 1 -- 172.21.15.46:0/2699724208 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600bb770 msgr2=0x7f7b600d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5f59e640 1 --2- 172.21.15.46:0/2699724208 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600bb770 0x7f7b600d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5f59e640 1 -- 172.21.15.46:0/2699724208 >> v2:172.21.15.88:3300/0 conn(0x7f7b60163eb0 msgr2=0x7f7b601642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5f59e640 1 --2- 172.21.15.46:0/2699724208 >> v2:172.21.15.88:3300/0 conn(0x7f7b60163eb0 0x7f7b601642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5f59e640 1 -- 172.21.15.46:0/2699724208 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b60164be0 con 0x7f7b600d2a70 2024-05-09T03:18:49.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.506+0000 7f7b5f59e640 1 --2- 172.21.15.46:0/2699724208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b600d2a70 0x7f7b600d2e80 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f7b600b88b0 tx=0x7f7b5802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f7f19de45207a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b5e59c640 1 -- 172.21.15.46:0/2699724208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58036070 con 0x7f7b600d2a70 2024-05-09T03:18:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b5e59c640 1 -- 172.21.15.46:0/2699724208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b58031600 con 0x7f7b600d2a70 2024-05-09T03:18:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b5e59c640 1 -- 172.21.15.46:0/2699724208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580319d0 con 0x7f7b600d2a70 2024-05-09T03:18:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 -- 172.21.15.46:0/2699724208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b600d2a70 msgr2=0x7f7b600d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 --2- 172.21.15.46:0/2699724208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b600d2a70 0x7f7b600d2e80 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f7b600b88b0 tx=0x7f7b5802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 -- 172.21.15.46:0/2699724208 shutdown_connections 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 --2- 172.21.15.46:0/2699724208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b600d2a70 0x7f7b600d2e80 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 --2- 172.21.15.46:0/2699724208 >> v2:172.21.15.88:3300/0 conn(0x7f7b60163eb0 0x7f7b601642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 --2- 172.21.15.46:0/2699724208 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600bb770 0x7f7b600d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 -- 172.21.15.46:0/2699724208 >> 172.21.15.46:0/2699724208 conn(0x7f7b6006e370 msgr2=0x7f7b6006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 -- 172.21.15.46:0/2699724208 shutdown_connections 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.507+0000 7f7b651a9640 1 -- 172.21.15.46:0/2699724208 wait complete. 2024-05-09T03:18:49.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.509+0000 7f7b651a9640 1 Processor -- start 2024-05-09T03:18:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.510+0000 7f7b651a9640 1 -- start start 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b651a9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b600bb770 0x7f7b601f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b651a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600d2a70 0x7f7b60155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b651a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 0x7f7b600cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b651a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7b600bbb80 con 0x7f7b60163eb0 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b651a9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7b600d5580 con 0x7f7b600bb770 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b651a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7b600d0bc0 con 0x7f7b600d2a70 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 0x7f7b600cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 0x7f7b600cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45750/0 (socket says 172.21.15.46:45750) 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 -- 172.21.15.46:0/1576969506 learned_addr learned my addr 172.21.15.46:0/1576969506 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5ed9d640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b600bb770 0x7f7b601f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5659c640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600d2a70 0x7f7b60155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 -- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600d2a70 msgr2=0x7f7b60155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600d2a70 0x7f7b60155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 -- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b600bb770 msgr2=0x7f7b601f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b600bb770 0x7f7b601f9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.511+0000 7f7b5f59e640 1 -- 172.21.15.46:0/1576969506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b601fa940 con 0x7f7b60163eb0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.512+0000 7f7b5f59e640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 0x7f7b600cde10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f7b5802ac90 tx=0x7f7b58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.512+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5803f020 con 0x7f7b60163eb0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.512+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b58037040 con 0x7f7b60163eb0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.512+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58036070 con 0x7f7b60163eb0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.512+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b601fac20 con 0x7f7b60163eb0 2024-05-09T03:18:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.512+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b601fa3c0 con 0x7f7b60163eb0 2024-05-09T03:18:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.513+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b601fa6b0 con 0x7f7b60163eb0 2024-05-09T03:18:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.514+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7b58042020 con 0x7f7b60163eb0 2024-05-09T03:18:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.514+0000 7f7b57fff640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b40076300 0x7f7b40078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.514+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f7b580b80c0 con 0x7f7b60163eb0 2024-05-09T03:18:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.515+0000 7f7b5ed9d640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b40076300 0x7f7b40078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:49.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.515+0000 7f7b5ed9d640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b40076300 0x7f7b40078810 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7b600b8880 tx=0x7f7b50009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:49.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.519+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7b58049030 con 0x7f7b60163eb0 2024-05-09T03:18:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:49.751+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7b600d5360 con 0x7f7b60163eb0 2024-05-09T03:18:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.114+0000 7f7b57fff640 1 -- 172.21.15.46:0/1576969506 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v246) ==== 161+0+0 (secure 0 0 0) 0x7f7b58080ad0 con 0x7f7b60163eb0 2024-05-09T03:18:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b40076300 msgr2=0x7f7b40078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b40076300 0x7f7b40078810 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f7b600b8880 tx=0x7f7b50009190 comp rx=0 tx=0).stop 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 msgr2=0x7f7b600cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 0x7f7b600cde10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f7b5802ac90 tx=0x7f7b58004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 shutdown_connections 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7b40076300 0x7f7b40078810 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b60163eb0 0x7f7b600cde10 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7b600d2a70 0x7f7b60155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.116+0000 7f7b651a9640 1 --2- 172.21.15.46:0/1576969506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b600bb770 0x7f7b601f9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.117+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 >> 172.21.15.46:0/1576969506 conn(0x7f7b6006e370 msgr2=0x7f7b600b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.117+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 shutdown_connections 2024-05-09T03:18:50.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.117+0000 7f7b651a9640 1 -- 172.21.15.46:0/1576969506 wait complete. 2024-05-09T03:18:50.131 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:18:50.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.245+0000 7ff9ffdfa640 1 Processor -- start 2024-05-09T03:18:50.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.245+0000 7ff9ffdfa640 1 -- start start 2024-05-09T03:18:50.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9ffdfa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9f80ceba0 0x7ff9f80cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:50.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9ffdfa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9f80cf4f0 0x7ff9f80cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9ffdfa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9f80b91f0 con 0x7ff9f806e2f0 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9ffdfa640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff9f80d2ff0 con 0x7ff9f80cf4f0 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9ffdfa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff9f80d1320 con 0x7ff9f80ceba0 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9fe397640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9f80cf4f0 0x7ff9f80cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9fdb96640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff9f806e2f0 0x7ff9f806e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46086/0 (socket says 172.21.15.46:46086) 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.246+0000 7ff9fdb96640 1 -- 172.21.15.46:0/3563470632 learned_addr learned my addr 172.21.15.46:0/3563470632 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fd395640 1 --2- 172.21.15.46:0/3563470632 >> v2:172.21.15.88:3300/0 conn(0x7ff9f80ceba0 0x7ff9f80cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fe397640 1 -- 172.21.15.46:0/3563470632 >> v2:172.21.15.88:3300/0 conn(0x7ff9f80ceba0 msgr2=0x7ff9f80cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fe397640 1 --2- 172.21.15.46:0/3563470632 >> v2:172.21.15.88:3300/0 conn(0x7ff9f80ceba0 0x7ff9f80cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fe397640 1 -- 172.21.15.46:0/3563470632 >> v1:172.21.15.46:6789/0 conn(0x7ff9f806e2f0 legacy=0x7ff9f806e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fe397640 1 -- 172.21.15.46:0/3563470632 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f809ceb0 con 0x7ff9f80cf4f0 2024-05-09T03:18:50.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fcb94640 1 -- 172.21.15.46:0/3563470632 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853177006 0 0) 0x7ff9f80b91f0 con 0x7ff9f806e2f0 2024-05-09T03:18:50.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fd395640 1 --2- 172.21.15.46:0/3563470632 >> v2:172.21.15.88:3300/0 conn(0x7ff9f80ceba0 0x7ff9f80cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.247+0000 7ff9fe397640 1 --2- 172.21.15.46:0/3563470632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f80cf900 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80743b0 tx=0x7ff9e802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b240d886b95744 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9fcb94640 1 -- 172.21.15.46:0/3563470632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8007740 con 0x7ff9f80cf4f0 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9fcb94640 1 -- 172.21.15.46:0/3563470632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e80078e0 con 0x7ff9f80cf4f0 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9fcb94640 1 -- 172.21.15.46:0/3563470632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8007cb0 con 0x7ff9f80cf4f0 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/3563470632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80cf4f0 msgr2=0x7ff9f80cf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/3563470632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f80cf900 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ff9f80743b0 tx=0x7ff9e802a680 comp rx=0 tx=0).stop 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/3563470632 shutdown_connections 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/3563470632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f80cf900 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/3563470632 >> v2:172.21.15.88:3300/0 conn(0x7ff9f80ceba0 0x7ff9f80cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/3563470632 >> 172.21.15.46:0/3563470632 conn(0x7ff9f80d05f0 msgr2=0x7ff9f80d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/3563470632 shutdown_connections 2024-05-09T03:18:50.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.248+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/3563470632 wait complete. 2024-05-09T03:18:50.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.250+0000 7ff9ffdfa640 1 Processor -- start 2024-05-09T03:18:50.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.250+0000 7ff9ffdfa640 1 -- start start 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9ffdfa640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9f806e2f0 0x7ff9f81ea410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9ffdfa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 0x7ff9f81e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9ffdfa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f81e9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9ffdfa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f806ee10 con 0x7ff9f80ceba0 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9ffdfa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff9f80d1570 con 0x7ff9f80cf4f0 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9ffdfa640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff9f80d3220 con 0x7ff9f806e2f0 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 0x7ff9f81e8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 0x7ff9f81e8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45766/0 (socket says 172.21.15.46:45766) 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 -- 172.21.15.46:0/4183080826 learned_addr learned my addr 172.21.15.46:0/4183080826 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fdb96640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9f806e2f0 0x7ff9f81ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fe397640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f81e9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 -- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9f806e2f0 msgr2=0x7ff9f81ea410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:50.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9f806e2f0 0x7ff9f81ea410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 -- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9f80cf4f0 msgr2=0x7ff9f81e9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f81e9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 -- 172.21.15.46:0/4183080826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9f81e0df0 con 0x7ff9f80ceba0 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.252+0000 7ff9fd395640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 0x7ff9f81e8ff0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00d800 tx=0x7ff9ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.253+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec004030 con 0x7ff9f80ceba0 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.253+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ec0041d0 con 0x7ff9f80ceba0 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.253+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9f81e3730 con 0x7ff9f80ceba0 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.253+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0096a0 con 0x7ff9f80ceba0 2024-05-09T03:18:50.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.253+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9f81e1090 con 0x7ff9f80ceba0 2024-05-09T03:18:50.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.253+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9f8073190 con 0x7ff9f80ceba0 2024-05-09T03:18:50.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.255+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff9ec009840 con 0x7ff9f80ceba0 2024-05-09T03:18:50.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.255+0000 7ff9f67fc640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9e4076300 0x7ff9e4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:50.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.255+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7ff9ec098c80 con 0x7ff9f80ceba0 2024-05-09T03:18:50.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.255+0000 7ff9fdb96640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9e4076300 0x7ff9e4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:50.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.256+0000 7ff9fdb96640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9e4076300 0x7ff9e4078810 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0000f60 tx=0x7ff9f0007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:50.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.259+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff9f8073190 con 0x7ff9f80ceba0 2024-05-09T03:18:50.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:50.523+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7ff9f806f3a0 con 0x7ff9f80ceba0 2024-05-09T03:18:51.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.119+0000 7ff9f67fc640 1 -- 172.21.15.46:0/4183080826 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v247) ==== 101+0+0 (secure 0 0 0) 0x7ff9f806f3a0 con 0x7ff9f80ceba0 2024-05-09T03:18:51.121 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9e4076300 msgr2=0x7ff9e4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9e4076300 0x7ff9e4078810 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0000f60 tx=0x7ff9f0007c00 comp rx=0 tx=0).stop 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 msgr2=0x7ff9f81e8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 0x7ff9f81e8ff0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00d800 tx=0x7ff9ec00dcc0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 shutdown_connections 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9f80cf4f0 0x7ff9f81e9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9e4076300 0x7ff9e4078810 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9f80ceba0 0x7ff9f81e8ff0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 --2- 172.21.15.46:0/4183080826 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9f806e2f0 0x7ff9f81ea410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 >> 172.21.15.46:0/4183080826 conn(0x7ff9f80d05f0 msgr2=0x7ff9f81e7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 shutdown_connections 2024-05-09T03:18:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.122+0000 7ff9ffdfa640 1 -- 172.21.15.46:0/4183080826 wait complete. 2024-05-09T03:18:51.137 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:18:51.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.195+0000 7f2b5f3dbf40 1 Processor -- start 2024-05-09T03:18:51.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.195+0000 7f2b5f3dbf40 1 -- start start 2024-05-09T03:18:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b5f3dbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x5633020f9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b5f3dbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x56330219d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b5f3dbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x5633020f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b5f3dbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563302072f50 con 0x5633021a0150 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b5f3dbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563302076f90 con 0x5633021145f0 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b5f3dbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5633021158d0 con 0x56330210c3a0 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x5633020f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x5633020f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45770/0 (socket says 172.21.15.46:45770) 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 -- 172.21.15.46:0/2560635068 learned_addr learned my addr 172.21.15.46:0/2560635068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b586a4640 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x56330219d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x5633020f9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 -- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 msgr2=0x5633020f9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x5633020f9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 -- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 msgr2=0x56330219d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x56330219d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.196+0000 7f2b596a6640 1 -- 172.21.15.46:0/2560635068 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633020f9940 con 0x5633021a0150 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b596a6640 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x5633020f65b0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f2b50006fa0 tx=0x7f2b50001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e52adbcd2d622b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b576a2640 1 -- 172.21.15.46:0/2560635068 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50004520 con 0x5633021a0150 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b576a2640 1 -- 172.21.15.46:0/2560635068 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b500046e0 con 0x5633021a0150 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b576a2640 1 -- 172.21.15.46:0/2560635068 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50004b10 con 0x5633021a0150 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/2560635068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 msgr2=0x5633020f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x5633020f65b0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f2b50006fa0 tx=0x7f2b50001a80 comp rx=0 tx=0).stop 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/2560635068 shutdown_connections 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x5633020f65b0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x56330219d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/2560635068 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x5633020f9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/2560635068 >> 172.21.15.46:0/2560635068 conn(0x563301feeac0 msgr2=0x5633021164e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/2560635068 shutdown_connections 2024-05-09T03:18:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.197+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/2560635068 wait complete. 2024-05-09T03:18:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.198+0000 7f2b5f3dbf40 1 Processor -- start 2024-05-09T03:18:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.198+0000 7f2b5f3dbf40 1 -- start start 2024-05-09T03:18:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b5f3dbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x56330219dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b5f3dbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x563302198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b5f3dbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x563302198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b5f3dbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5633020fa390 con 0x5633021a0150 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b5f3dbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5633020fa670 con 0x5633021145f0 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b5f3dbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56330218f1e0 con 0x56330210c3a0 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x563302198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x563302198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45782/0 (socket says 172.21.15.46:45782) 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 learned_addr learned my addr 172.21.15.46:0/672645247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x563302198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x56330219dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 msgr2=0x56330219dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56330210c3a0 0x56330219dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 msgr2=0x563302198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5633021145f0 0x563302198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633020f6780 con 0x5633021a0150 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b596a6640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x563302198f60 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x56330210fbf0 tx=0x7f2b50004460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.200+0000 7f2b566a0640 1 -- 172.21.15.46:0/672645247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50024070 con 0x5633021a0150 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.201+0000 7f2b566a0640 1 -- 172.21.15.46:0/672645247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b50019040 con 0x5633021a0150 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.201+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56330219b290 con 0x5633021a0150 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.201+0000 7f2b566a0640 1 -- 172.21.15.46:0/672645247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5001f8a0 con 0x5633021a0150 2024-05-09T03:18:51.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.201+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633022321a0 con 0x5633021a0150 2024-05-09T03:18:51.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.202+0000 7f2b566a0640 1 -- 172.21.15.46:0/672645247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2b50027070 con 0x5633021a0150 2024-05-09T03:18:51.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.203+0000 7f2b566a0640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b44074cb0 0x7f2b440771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.203+0000 7f2b566a0640 1 -- 172.21.15.46:0/672645247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f2b50098020 con 0x5633021a0150 2024-05-09T03:18:51.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.203+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b44074cb0 0x7f2b440771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.206 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:18:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.203+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56330219e0e0 0x5633022309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.203+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633021926d0 con 0x56330219e0e0 2024-05-09T03:18:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.203+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56330219e0e0 0x5633022309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.204+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b44074cb0 0x7f2b440771c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2b48000c00 tx=0x7f2b48001790 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.204+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56330219e0e0 0x5633022309f0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.206+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v110'784 uv758 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.206+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302230f30 con 0x56330219e0e0 2024-05-09T03:18:51.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.207+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v110'785 uv785 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.208+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302231300 con 0x56330219e0e0 2024-05-09T03:18:51.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.209+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v110'786 uv786 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.210 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:18:51.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.209+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244c80 con 0x56330219e0e0 2024-05-09T03:18:51.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.210+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v110'787 uv787 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.210+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302245020 con 0x56330219e0e0 2024-05-09T03:18:51.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.211+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v110'788 uv788 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.212+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302243c70 con 0x56330219e0e0 2024-05-09T03:18:51.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.213+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v110'789 uv789 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.213+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244060 con 0x56330219e0e0 2024-05-09T03:18:51.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.214+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v110'790 uv764 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.214+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302244450 con 0x56330219e0e0 2024-05-09T03:18:51.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.216+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v110'791 uv791 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.216+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244450 con 0x56330219e0e0 2024-05-09T03:18:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.217+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v110'792 uv792 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.219 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:18:51.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.217+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.219+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v110'793 uv793 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.219+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.220+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v110'794 uv794 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.220+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.222+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v110'795 uv795 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.222+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.223+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v110'796 uv771 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.223+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.224+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v110'797 uv797 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.225+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.227 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:18:51.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.226+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v110'798 uv798 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.226+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.228+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v110'799 uv799 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.228+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.229+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v110'800 uv800 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.229+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.230+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v110'801 uv801 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.230+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.232+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v110'802 uv802 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.232+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.233+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v110'803 uv777 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.233+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.235+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v110'804 uv804 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.235+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.236+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v110'805 uv805 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.238 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:18:51.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.236+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.238+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v110'806 uv806 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.238+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.239+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v110'807 uv807 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.239+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.240+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v110'808 uv808 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.240+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.242+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v110'809 uv783 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.242+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.243+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v110'810 uv810 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.243+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.245+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v110'811 uv811 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.246 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:18:51.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.245+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.246+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v110'812 uv812 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.246+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.247+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v110'813 uv813 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.247+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x56330219e0e0 2024-05-09T03:18:51.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.249+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v110'814 uv814 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.249+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563302244810 0x5633022478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.249+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302244400 con 0x563302244810 2024-05-09T03:18:51.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.249+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563302244810 0x5633022478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.250+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563302244810 0x5633022478e0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.251+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v110'567 uv549 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.251+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302247e20 con 0x563302244810 2024-05-09T03:18:51.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.252+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v110'568 uv568 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.252+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022481c0 con 0x563302244810 2024-05-09T03:18:51.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.254+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v110'569 uv569 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.255 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:18:51.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.254+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302248a60 con 0x563302244810 2024-05-09T03:18:51.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.255+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v110'570 uv570 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.255+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302248e00 con 0x563302244810 2024-05-09T03:18:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.256+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v110'571 uv571 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.256+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022491a0 con 0x563302244810 2024-05-09T03:18:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.257+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v110'572 uv554 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.258+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302249540 con 0x563302244810 2024-05-09T03:18:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.259+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v110'573 uv573 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.259+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022498e0 con 0x563302244810 2024-05-09T03:18:51.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.260+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v110'574 uv574 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.261 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:18:51.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.260+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302249cd0 con 0x563302244810 2024-05-09T03:18:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.261+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v110'575 uv575 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.261+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302249cd0 con 0x563302244810 2024-05-09T03:18:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.262+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v110'576 uv576 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.262+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302249a10 con 0x563302244810 2024-05-09T03:18:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.263+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v110'577 uv566 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.263+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.265+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v110'578 uv578 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.265+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.266+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v110'579 uv579 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.267 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:18:51.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.266+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.267+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v110'580 uv580 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.267+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.268+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v110'581 uv581 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.268+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.269+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v110'582 uv565 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.269+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.271+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v110'583 uv583 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.271+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.272+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v110'584 uv584 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.273 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:18:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.272+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224a9d0 con 0x563302244810 2024-05-09T03:18:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.273+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v110'585 uv585 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.273+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224a9d0 con 0x563302244810 2024-05-09T03:18:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.274+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v110'586 uv586 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.274+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x563302244810 2024-05-09T03:18:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.275+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v110'587 uv587 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302244810 2024-05-09T03:18:51.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.275+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x56330219e0e0 2024-05-09T03:18:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.276+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v110'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.276+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x5633022499c0 con 0x56330219e0e0 2024-05-09T03:18:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.277+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v110'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.277+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x56330219e0e0 2024-05-09T03:18:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.278+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v110'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.280 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:18:51.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.278+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224a9d0 con 0x56330219e0e0 2024-05-09T03:18:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.279+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v110'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.279+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224a9d0 con 0x56330219e0e0 2024-05-09T03:18:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.280+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v110'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.280+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5633022499c0 con 0x56330219e0e0 2024-05-09T03:18:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.281+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v110'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.281+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56330224a040 0x56330224a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.281+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224ce70 con 0x56330224a040 2024-05-09T03:18:51.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.282+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56330224a040 0x56330224a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.282+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56330224a040 0x56330224a470 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.283+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v110'114 uv113 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.283+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x56330224ce70 con 0x56330224a040 2024-05-09T03:18:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.284+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v110'115 uv115 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.284+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224ce70 con 0x56330224a040 2024-05-09T03:18:51.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.285+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v110'116 uv116 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.287 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:18:51.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.285+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224d280 con 0x56330224a040 2024-05-09T03:18:51.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.286+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v110'117 uv117 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.287+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224d280 con 0x56330219e0e0 2024-05-09T03:18:51.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.288+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v110'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.288+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x56330224d280 con 0x56330219e0e0 2024-05-09T03:18:51.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.289+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v110'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.289+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224d280 con 0x56330219e0e0 2024-05-09T03:18:51.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.290+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v110'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.292 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:18:51.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.290+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224e290 con 0x56330219e0e0 2024-05-09T03:18:51.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.292+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v110'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.292+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224e290 con 0x56330219e0e0 2024-05-09T03:18:51.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.293+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v110'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.293+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x56330224d280 con 0x56330219e0e0 2024-05-09T03:18:51.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.294+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v110'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.294+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56330224d790 0x56330224fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.294+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302250130 con 0x56330224d790 2024-05-09T03:18:51.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.295+0000 7f2b596a6640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56330224d790 0x56330224fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.295+0000 7f2b596a6640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56330224d790 0x56330224fbb0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.297+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v110'282 uv276 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.297+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x5633022505d0 con 0x56330224d790 2024-05-09T03:18:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.298+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v110'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.298+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302250ab0 con 0x56330224d790 2024-05-09T03:18:51.301 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:18:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.299+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v110'284 uv284 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.299+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302252360 con 0x56330224d790 2024-05-09T03:18:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.300+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v110'285 uv285 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.300+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302252700 con 0x56330224d790 2024-05-09T03:18:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.301+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v110'286 uv286 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.302+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302251350 con 0x56330224d790 2024-05-09T03:18:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.303+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v110'287 uv287 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.303+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302251740 con 0x56330224d790 2024-05-09T03:18:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.304+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v110'288 uv281 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.304+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302251740 con 0x56330224d790 2024-05-09T03:18:51.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.305+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v110'289 uv289 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.305+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302251740 con 0x56330224d790 2024-05-09T03:18:51.308 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:18:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.306+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v110'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.306+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302251740 con 0x56330224d790 2024-05-09T03:18:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.308+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v110'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.308+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302251740 con 0x56330224d790 2024-05-09T03:18:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.309+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v110'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.309+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563302252aa0 0x563302254ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.309+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302251740 con 0x563302252aa0 2024-05-09T03:18:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.309+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563302252aa0 0x563302254ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.310+0000 7f2b58ea5640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563302252aa0 0x563302254ec0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.311+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v110'127 uv126 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302252aa0 2024-05-09T03:18:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.311+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302251740 con 0x563302252aa0 2024-05-09T03:18:51.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.312+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v110'128 uv128 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302252aa0 2024-05-09T03:18:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.312+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x563302252aa0 2024-05-09T03:18:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.313+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v110'129 uv129 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302252aa0 2024-05-09T03:18:51.315 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:18:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.313+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x563302252aa0 2024-05-09T03:18:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.314+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v110'130 uv130 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302252aa0 2024-05-09T03:18:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.314+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x563302252aa0 2024-05-09T03:18:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.315+0000 7f2b58ea5640 1 -- 172.21.15.46:0/672645247 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v110'131 uv131 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b480124d0 con 0x563302252aa0 2024-05-09T03:18:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.315+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330219e0e0 2024-05-09T03:18:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.316+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v110'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.316+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302255400 con 0x56330219e0e0 2024-05-09T03:18:51.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.318+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v110'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.318+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330219e0e0 2024-05-09T03:18:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.319+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v110'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.320 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:18:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.319+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302256410 con 0x56330219e0e0 2024-05-09T03:18:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.320+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v110'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.320+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302256410 con 0x56330219e0e0 2024-05-09T03:18:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.321+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v110'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.321+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330219e0e0 2024-05-09T03:18:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.322+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v110'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330219e0e0 2024-05-09T03:18:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.322+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.323+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v110'253 uv247 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.323+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.324+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v110'254 uv254 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.324+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.325+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v110'255 uv255 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.327 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:18:51.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.325+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.326+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v110'256 uv256 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.326+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.327+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v110'257 uv257 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.327+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.328+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v110'258 uv252 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.328+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.330+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v110'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.330+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.331+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v110'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.332 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:18:51.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.331+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302255400 con 0x56330224a040 2024-05-09T03:18:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.332+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v110'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.332+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302250f90 con 0x56330224a040 2024-05-09T03:18:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.333+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v110'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.333+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563302256160 0x5633022588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.333+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x563302256160 2024-05-09T03:18:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.333+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563302256160 0x5633022588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.333+0000 7f2b586a4640 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563302256160 0x5633022588d0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.334+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v110'111 uv110 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x563302256160 2024-05-09T03:18:51.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.334+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302258e10 con 0x563302256160 2024-05-09T03:18:51.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.336+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v110'112 uv112 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x563302256160 2024-05-09T03:18:51.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.336+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x563302256160 2024-05-09T03:18:51.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.337+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v110'113 uv113 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x563302256160 2024-05-09T03:18:51.338 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:18:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.337+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x563302256160 2024-05-09T03:18:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.338+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v110'114 uv114 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x563302256160 2024-05-09T03:18:51.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.338+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224a040 2024-05-09T03:18:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.339+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v110'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.339+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224a040 2024-05-09T03:18:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.340+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v110'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.340+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224a040 2024-05-09T03:18:51.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.341+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v110'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.343 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:18:51.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.341+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302259e20 con 0x56330224a040 2024-05-09T03:18:51.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.342+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v110'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.342+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302259e20 con 0x56330224a040 2024-05-09T03:18:51.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.343+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v110'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.343+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224a040 2024-05-09T03:18:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.344+0000 7f2b586a4640 1 -- 172.21.15.46:0/672645247 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v110'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b4c05d040 con 0x56330224a040 2024-05-09T03:18:51.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.344+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.346+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v110'130 uv129 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.346+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.347+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v110'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.347+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.350 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:18:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.349+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v110'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.349+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.350+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v110'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.350+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.351+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v110'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.351+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.352+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v110'314 uv306 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.352+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.354+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v110'315 uv315 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.354+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.355+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v110'316 uv316 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.356 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:18:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.355+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.356+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v110'317 uv317 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.356+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.357+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v110'318 uv318 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.357+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.359+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v110'319 uv313 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.359+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.360+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v110'320 uv320 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.360+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.361+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v110'321 uv321 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.362 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:18:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.361+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302259e20 con 0x56330224d790 2024-05-09T03:18:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.362+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v110'322 uv322 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.362+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302259e20 con 0x56330224d790 2024-05-09T03:18:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.363+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v110'323 uv323 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.363+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.364+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v110'324 uv324 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.364+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x563302258e10 con 0x56330224d790 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v110'325 uv325 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f2b50026030 con 0x56330224d790 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56330219e0e0 msgr2=0x5633022309f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56330219e0e0 0x5633022309f0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56330224a040 msgr2=0x56330224a470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56330224a040 0x56330224a470 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563302256160 msgr2=0x5633022588d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x563302256160 0x5633022588d0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563302244810 msgr2=0x5633022478e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 reap_dead start 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x563302244810 0x5633022478e0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56330224d790 msgr2=0x56330224fbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56330224d790 0x56330224fbb0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563302252aa0 msgr2=0x563302254ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x563302252aa0 0x563302254ec0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b44074cb0 msgr2=0x7f2b440771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2b44074cb0 0x7f2b440771c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2b48000c00 tx=0x7f2b48001790 comp rx=0 tx=0).stop 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 msgr2=0x563302198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 --2- 172.21.15.46:0/672645247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633021a0150 0x563302198f60 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x56330210fbf0 tx=0x7f2b50004460 comp rx=0 tx=0).stop 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b596a6640 1 -- 172.21.15.46:0/672645247 reap_dead start 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 shutdown_connections 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 >> 172.21.15.46:0/672645247 conn(0x563301feeac0 msgr2=0x5633021164e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 shutdown_connections 2024-05-09T03:18:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.365+0000 7f2b5f3dbf40 1 -- 172.21.15.46:0/672645247 wait complete. 2024-05-09T03:18:51.370 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 500.00000000 0 2024-05-09T03:18:51.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.425+0000 7f26673c4f40 1 Processor -- start 2024-05-09T03:18:51.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.425+0000 7f26673c4f40 1 -- start start 2024-05-09T03:18:51.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f26673c4f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4a797e0 0x564fc4b39e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f26673c4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4aa8db0 0x564fc4b35160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f26673c4f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4b356a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f26673c4f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564fc4abbcb0 con 0x564fc4b51780 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f26673c4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564fc4a22f50 con 0x564fc4aa8db0 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f26673c4f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564fc4a79bf0 con 0x564fc4a797e0 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4b356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4b356a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45808/0 (socket says 172.21.15.46:45808) 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 -- 172.21.15.46:0/2749676010 learned_addr learned my addr 172.21.15.46:0/2749676010 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266068b640 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4aa8db0 0x564fc4b35160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f2660e8c640 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4a797e0 0x564fc4b39e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 -- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4a797e0 msgr2=0x564fc4b39e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4a797e0 0x564fc4b39e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 -- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4aa8db0 msgr2=0x564fc4b35160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4aa8db0 0x564fc4b35160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.426+0000 7f266168d640 1 -- 172.21.15.46:0/2749676010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fc4b35ca0 con 0x564fc4b51780 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f266168d640 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4b356a0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f2658001570 tx=0x7f2658001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd4d6a97f5f8b31 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f265f689640 1 -- 172.21.15.46:0/2749676010 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26580044e0 con 0x564fc4b51780 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f265f689640 1 -- 172.21.15.46:0/2749676010 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2658004680 con 0x564fc4b51780 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f265f689640 1 -- 172.21.15.46:0/2749676010 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658004a50 con 0x564fc4b51780 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 -- 172.21.15.46:0/2749676010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 msgr2=0x564fc4b356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4b356a0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f2658001570 tx=0x7f2658001a30 comp rx=0 tx=0).stop 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 -- 172.21.15.46:0/2749676010 shutdown_connections 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4b356a0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4aa8db0 0x564fc4b35160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 --2- 172.21.15.46:0/2749676010 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4a797e0 0x564fc4b39e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.427+0000 7f26673c4f40 1 -- 172.21.15.46:0/2749676010 >> 172.21.15.46:0/2749676010 conn(0x564fc499eac0 msgr2=0x564fc4ac1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:51.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.428+0000 7f26673c4f40 1 -- 172.21.15.46:0/2749676010 shutdown_connections 2024-05-09T03:18:51.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.428+0000 7f26673c4f40 1 -- 172.21.15.46:0/2749676010 wait complete. 2024-05-09T03:18:51.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.428+0000 7f26673c4f40 1 Processor -- start 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.428+0000 7f26673c4f40 1 -- start start 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f26673c4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4a797e0 0x564fc4b533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f26673c4f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4aa8db0 0x564fc4b53930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f26673c4f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4be71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f26673c4f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564fc4b50380 con 0x564fc4b51780 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f26673c4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564fc4aaaab0 con 0x564fc4a797e0 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f26673c4f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564fc4ac00d0 con 0x564fc4aa8db0 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4be71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4be71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45824/0 (socket says 172.21.15.46:45824) 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 -- 172.21.15.46:0/3510507911 learned_addr learned my addr 172.21.15.46:0/3510507911 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f2660e8c640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4a797e0 0x564fc4b533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266068b640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4aa8db0 0x564fc4b53930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 -- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4aa8db0 msgr2=0x564fc4b53930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564fc4aa8db0 0x564fc4b53930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 -- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4a797e0 msgr2=0x564fc4b533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4a797e0 0x564fc4b533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 -- 172.21.15.46:0/3510507911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fc4b36ca0 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f2660e8c640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564fc4a797e0 0x564fc4b533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f266168d640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4be71b0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f2658001a70 tx=0x7f2658000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.430+0000 7f265e687640 1 -- 172.21.15.46:0/3510507911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658024070 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.431+0000 7f265e687640 1 -- 172.21.15.46:0/3510507911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2658019040 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.431+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564fc4b36f80 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.431+0000 7f265e687640 1 -- 172.21.15.46:0/3510507911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265801f8e0 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.431+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fc4be7f10 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.432+0000 7f265e687640 1 -- 172.21.15.46:0/3510507911 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2658027070 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f265e687640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f264c074cb0 0x7f264c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f265e687640 1 -- 172.21.15.46:0/3510507911 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f2658097f90 con 0x564fc4b51780 2024-05-09T03:18:51.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f2660e8c640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f264c074cb0 0x7f264c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f26673c4f40 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564fc4b51350 0x564fc4abda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x564fc4b33c00 con 0x564fc4b51350 2024-05-09T03:18:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f266068b640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564fc4b51350 0x564fc4abda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.433+0000 7f2660e8c640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f264c074cb0 0x7f264c0771c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x564fc4a7e440 tx=0x7f265000d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.434+0000 7f266068b640 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564fc4b51350 0x564fc4abda70 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f266068b640 1 -- 172.21.15.46:0/3510507911 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 500.00000000 [truncate 0] v110'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f265405d040 con 0x564fc4b51350 2024-05-09T03:18:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564fc4b51350 msgr2=0x564fc4abda70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f26673c4f40 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x564fc4b51350 0x564fc4abda70 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f264c074cb0 msgr2=0x7f264c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f26673c4f40 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f264c074cb0 0x7f264c0771c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x564fc4a7e440 tx=0x7f265000d040 comp rx=0 tx=0).stop 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 msgr2=0x564fc4be71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f26673c4f40 1 --2- 172.21.15.46:0/3510507911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564fc4b51780 0x564fc4be71b0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f2658001a70 tx=0x7f2658000f40 comp rx=0 tx=0).stop 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.435+0000 7f266168d640 1 -- 172.21.15.46:0/3510507911 reap_dead start 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.436+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 shutdown_connections 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.436+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 >> 172.21.15.46:0/3510507911 conn(0x564fc499eac0 msgr2=0x564fc4b50b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.436+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 shutdown_connections 2024-05-09T03:18:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.436+0000 7f26673c4f40 1 -- 172.21.15.46:0/3510507911 wait complete. 2024-05-09T03:18:51.440 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:18:51.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.556+0000 7fc1cd434640 1 Processor -- start 2024-05-09T03:18:51.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.556+0000 7fc1cd434640 1 -- start start 2024-05-09T03:18:51.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.557+0000 7fc1cd434640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1c8071100 0x7fc1c80b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.557+0000 7fc1cd434640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1c80cfd40 0x7fc1c80732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.557+0000 7fc1cd434640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c80b5930 con 0x7fc1c80d0150 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.557+0000 7fc1cd434640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c80cf8b0 con 0x7fc1c8071100 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.557+0000 7fc1cd434640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c80b5c10 con 0x7fc1c80cfd40 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1c8071100 0x7fc1c80b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1c8071100 0x7fc1c80b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45828/0 (socket says 172.21.15.46:45828) 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 -- 172.21.15.46:0/355339706 learned_addr learned my addr 172.21.15.46:0/355339706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c67fc640 1 --2- 172.21.15.46:0/355339706 >> v2:172.21.15.88:3300/0 conn(0x7fc1c80cfd40 0x7fc1c80732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 -- 172.21.15.46:0/355339706 >> v2:172.21.15.88:3300/0 conn(0x7fc1c80cfd40 msgr2=0x7fc1c80732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 --2- 172.21.15.46:0/355339706 >> v2:172.21.15.88:3300/0 conn(0x7fc1c80cfd40 0x7fc1c80732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 -- 172.21.15.46:0/355339706 >> v1:172.21.15.88:6789/0 conn(0x7fc1c80d0150 legacy=0x7fc1c806f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.558+0000 7fc1c6ffd640 1 -- 172.21.15.46:0/355339706 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c80cf540 con 0x7fc1c8071100 2024-05-09T03:18:51.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.559+0000 7fc1c6ffd640 1 --2- 172.21.15.46:0/355339706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c8071100 0x7fc1c80b52a0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8074560 tx=0x7fc1c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b72dbffb5d1f62 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.559+0000 7fc1c5ffb640 1 -- 172.21.15.46:0/355339706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0037070 con 0x7fc1c8071100 2024-05-09T03:18:51.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.559+0000 7fc1c5ffb640 1 -- 172.21.15.46:0/355339706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1c0032440 con 0x7fc1c8071100 2024-05-09T03:18:51.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.559+0000 7fc1c5ffb640 1 -- 172.21.15.46:0/355339706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0032810 con 0x7fc1c8071100 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 -- 172.21.15.46:0/355339706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c8071100 msgr2=0x7fc1c80b52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 --2- 172.21.15.46:0/355339706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c8071100 0x7fc1c80b52a0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8074560 tx=0x7fc1c002a270 comp rx=0 tx=0).stop 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 -- 172.21.15.46:0/355339706 shutdown_connections 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 --2- 172.21.15.46:0/355339706 >> v2:172.21.15.88:3300/0 conn(0x7fc1c80cfd40 0x7fc1c80732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 --2- 172.21.15.46:0/355339706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c8071100 0x7fc1c80b52a0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 -- 172.21.15.46:0/355339706 >> 172.21.15.46:0/355339706 conn(0x7fc1c80b3620 msgr2=0x7fc1c80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 -- 172.21.15.46:0/355339706 shutdown_connections 2024-05-09T03:18:51.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.560+0000 7fc1cd434640 1 -- 172.21.15.46:0/355339706 wait complete. 2024-05-09T03:18:51.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.562+0000 7fc1cd434640 1 Processor -- start 2024-05-09T03:18:51.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.562+0000 7fc1cd434640 1 -- start start 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1cd434640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1c8071100 0x7fc1c8150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1cd434640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 0x7fc1c81525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1cd434640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1c80d0150 0x7fc1c8152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1cd434640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c80b6070 con 0x7fc1c80cfd40 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1cd434640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c80b5ef0 con 0x7fc1c8071100 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1cd434640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c815c9d0 con 0x7fc1c80d0150 2024-05-09T03:18:51.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 0x7fc1c81525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 0x7fc1c81525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45838/0 (socket says 172.21.15.46:45838) 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 -- 172.21.15.46:0/3398389278 learned_addr learned my addr 172.21.15.46:0/3398389278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c6ffd640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1c8071100 0x7fc1c8150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c77fe640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1c80d0150 0x7fc1c8152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 -- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1c80d0150 msgr2=0x7fc1c8152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1c80d0150 0x7fc1c8152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 -- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1c8071100 msgr2=0x7fc1c8150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1c8071100 0x7fc1c8150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.563+0000 7fc1c67fc640 1 -- 172.21.15.46:0/3398389278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c8150f70 con 0x7fc1c80cfd40 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.564+0000 7fc1c67fc640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 0x7fc1c81525c0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000d620 tx=0x7fc1b000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.564+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b000b620 con 0x7fc1c80cfd40 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.564+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b0019070 con 0x7fc1c80cfd40 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.564+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c8156dd0 con 0x7fc1c80cfd40 2024-05-09T03:18:51.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.564+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b000ba00 con 0x7fc1c80cfd40 2024-05-09T03:18:51.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.564+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c8151dc0 con 0x7fc1c80cfd40 2024-05-09T03:18:51.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.565+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c806a270 con 0x7fc1c80cfd40 2024-05-09T03:18:51.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.566+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc1b000bba0 con 0x7fc1c80cfd40 2024-05-09T03:18:51.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.567+0000 7fc1bf7fe640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1a80762d0 0x7fc1a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:51.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.567+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7fc1b0098cc0 con 0x7fc1c80cfd40 2024-05-09T03:18:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.567+0000 7fc1c6ffd640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1a80762d0 0x7fc1a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:51.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.567+0000 7fc1c6ffd640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1a80762d0 0x7fc1a80787e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0034000 tx=0x7fc1c002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:51.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.571+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1b0061720 con 0x7fc1c80cfd40 2024-05-09T03:18:51.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:51.805+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fc1c80d0760 con 0x7fc1c80cfd40 2024-05-09T03:18:52.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.125+0000 7fc1bf7fe640 1 -- 172.21.15.46:0/3398389278 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v248) ==== 162+0+0 (secure 0 0 0) 0x7fc1b0065a20 con 0x7fc1c80cfd40 2024-05-09T03:18:52.127 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:18:52.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1a80762d0 msgr2=0x7fc1a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1a80762d0 0x7fc1a80787e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0034000 tx=0x7fc1c002e000 comp rx=0 tx=0).stop 2024-05-09T03:18:52.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 msgr2=0x7fc1c81525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 0x7fc1c81525c0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc1b000d620 tx=0x7fc1b000dae0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 shutdown_connections 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1c80d0150 0x7fc1c8152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1a80762d0 0x7fc1a80787e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c80cfd40 0x7fc1c81525c0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 --2- 172.21.15.46:0/3398389278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1c8071100 0x7fc1c8150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 >> 172.21.15.46:0/3398389278 conn(0x7fc1c80b3620 msgr2=0x7fc1c815be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 shutdown_connections 2024-05-09T03:18:52.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.128+0000 7fc1cd434640 1 -- 172.21.15.46:0/3398389278 wait complete. 2024-05-09T03:18:52.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:52.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.673+0000 7f9462596640 1 Processor -- start 2024-05-09T03:18:52.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.673+0000 7f9462596640 1 -- start start 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.674+0000 7f9462596640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f945c1615e0 0x7f945c15dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.674+0000 7f9462596640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f945c0d2480 0x7f945c0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.674+0000 7f9462596640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945c160b20 con 0x7f945c164c70 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.674+0000 7f9462596640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f945c15e520 con 0x7f945c0d2480 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.674+0000 7f9462596640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f945c161150 con 0x7f945c1615e0 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f9460b33640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f945c0d2480 0x7f945c0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f945bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f945c164c70 0x7f945c0d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46104/0 (socket says 172.21.15.46:46104) 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f945bfff640 1 -- 172.21.15.46:0/1578148730 learned_addr learned my addr 172.21.15.46:0/1578148730 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f945b7fe640 1 --2- 172.21.15.46:0/1578148730 >> v2:172.21.15.88:3300/0 conn(0x7f945c1615e0 0x7f945c15dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:52.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f9460b33640 1 -- 172.21.15.46:0/1578148730 >> v2:172.21.15.88:3300/0 conn(0x7f945c1615e0 msgr2=0x7f945c15dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f9460b33640 1 --2- 172.21.15.46:0/1578148730 >> v2:172.21.15.88:3300/0 conn(0x7f945c1615e0 0x7f945c15dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f9460b33640 1 -- 172.21.15.46:0/1578148730 >> v1:172.21.15.46:6789/0 conn(0x7f945c164c70 legacy=0x7f945c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f9460b33640 1 -- 172.21.15.46:0/1578148730 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c0d15f0 con 0x7f945c0d2480 2024-05-09T03:18:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.675+0000 7f945affd640 1 -- 172.21.15.46:0/1578148730 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1513674638 0 0) 0x7f945c160b20 con 0x7f945c164c70 2024-05-09T03:18:52.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.676+0000 7f9460b33640 1 --2- 172.21.15.46:0/1578148730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c0d2480 0x7f945c0d2890 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f945c0cce00 tx=0x7f945402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb61c8e7b44f4b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:52.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.676+0000 7f945affd640 1 -- 172.21.15.46:0/1578148730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454007740 con 0x7f945c0d2480 2024-05-09T03:18:52.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.676+0000 7f945affd640 1 -- 172.21.15.46:0/1578148730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94540078e0 con 0x7f945c0d2480 2024-05-09T03:18:52.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.676+0000 7f945affd640 1 -- 172.21.15.46:0/1578148730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454007cb0 con 0x7f945c0d2480 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 -- 172.21.15.46:0/1578148730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c0d2480 msgr2=0x7f945c0d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 --2- 172.21.15.46:0/1578148730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c0d2480 0x7f945c0d2890 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f945c0cce00 tx=0x7f945402a5f0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 -- 172.21.15.46:0/1578148730 shutdown_connections 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 --2- 172.21.15.46:0/1578148730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c0d2480 0x7f945c0d2890 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 --2- 172.21.15.46:0/1578148730 >> v2:172.21.15.88:3300/0 conn(0x7f945c1615e0 0x7f945c15dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 -- 172.21.15.46:0/1578148730 >> 172.21.15.46:0/1578148730 conn(0x7f945c073db0 msgr2=0x7f945c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 -- 172.21.15.46:0/1578148730 shutdown_connections 2024-05-09T03:18:52.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.677+0000 7f9462596640 1 -- 172.21.15.46:0/1578148730 wait complete. 2024-05-09T03:18:52.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.679+0000 7f9462596640 1 Processor -- start 2024-05-09T03:18:52.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.679+0000 7f9462596640 1 -- start start 2024-05-09T03:18:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9462596640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f945c0d2480 0x7f945c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9462596640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f945c1615e0 0x7f945c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9462596640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 0x7f945c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:52.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9462596640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f945c1619f0 con 0x7f945c164c70 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9462596640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f945c160de0 con 0x7f945c0d2480 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9462596640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f945c0d2dd0 con 0x7f945c1615e0 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9460b33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 0x7f945c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9460b33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 0x7f945c0ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45846/0 (socket says 172.21.15.46:45846) 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f9460b33640 1 -- 172.21.15.46:0/2200865782 learned_addr learned my addr 172.21.15.46:0/2200865782 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f945bfff640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f945c0d2480 0x7f945c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.680+0000 7f945b7fe640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f945c1615e0 0x7f945c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9460b33640 1 -- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f945c1615e0 msgr2=0x7f945c159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9460b33640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f945c1615e0 0x7f945c159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9460b33640 1 -- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f945c0d2480 msgr2=0x7f945c155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9460b33640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f945c0d2480 0x7f945c155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9460b33640 1 -- 172.21.15.46:0/2200865782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c0cdcb0 con 0x7f945c164c70 2024-05-09T03:18:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9460b33640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 0x7f945c0ce550 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f945402abe0 tx=0x7f9454004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945403f070 con 0x7f945c164c70 2024-05-09T03:18:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9454031c40 con 0x7f945c164c70 2024-05-09T03:18:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945c0c98b0 con 0x7f945c164c70 2024-05-09T03:18:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454009450 con 0x7f945c164c70 2024-05-09T03:18:52.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.681+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f945c0c9e70 con 0x7f945c164c70 2024-05-09T03:18:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.682+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f945c0d2890 con 0x7f945c164c70 2024-05-09T03:18:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.683+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9454003990 con 0x7f945c164c70 2024-05-09T03:18:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.683+0000 7f9458ff9640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94480762d0 0x7f94480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.684+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f94540b9090 con 0x7f945c164c70 2024-05-09T03:18:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.684+0000 7f945bfff640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94480762d0 0x7f94480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:52.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.684+0000 7f945bfff640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94480762d0 0x7f94480787e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f944c007fd0 tx=0x7f944c007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:52.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.688+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f945c0d2890 con 0x7f945c164c70 2024-05-09T03:18:52.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.918+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f945c08cfd0 con 0x7f945c164c70 2024-05-09T03:18:52.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.918+0000 7f9458ff9640 1 -- 172.21.15.46:0/2200865782 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) ==== 95+0+5178 (secure 0 0 0) 0x7f945c08cfd0 con 0x7f945c164c70 2024-05-09T03:18:52.920 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:52.920 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":249,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:52.129270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9327":{"gid":9327,"name":"c","rank":0,"incarnation":249,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/2827001018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2827001018},{"type":"v1","addr":"172.21.15.46:6834","nonce":2827001018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:18:52.921 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 249 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94480762d0 msgr2=0x7f94480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94480762d0 0x7f94480787e0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f944c007fd0 tx=0x7f944c007de0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 msgr2=0x7f945c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 0x7f945c0ce550 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f945402abe0 tx=0x7f9454004a40 comp rx=0 tx=0).stop 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 shutdown_connections 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f94480762d0 0x7f94480787e0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f945c164c70 0x7f945c0ce550 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f945c1615e0 0x7f945c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 --2- 172.21.15.46:0/2200865782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f945c0d2480 0x7f945c155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 >> 172.21.15.46:0/2200865782 conn(0x7f945c073db0 msgr2=0x7f945c15e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:52.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 shutdown_connections 2024-05-09T03:18:52.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:52.921+0000 7f9462596640 1 -- 172.21.15.46:0/2200865782 wait complete. 2024-05-09T03:18:52.934 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 249, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:18:52.129270+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9327': {'gid': 9327, 'name': 'c', 'rank': 0, 'incarnation': 249, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/2827001018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2827001018}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2827001018}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:18:52.935 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9327:up:replay 2024-05-09T03:18:52.935 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:18:58.339 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:18:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.468+0000 7f15b582b640 1 Processor -- start 2024-05-09T03:18:58.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.468+0000 7f15b582b640 1 -- start start 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15b582b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15b00b81b0 0x7f15b00d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15b582b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00d2b20 0x7f15b0163f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15b582b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b00d15b0 con 0x7f15b01615f0 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15b582b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f15b0160b10 con 0x7f15b00b81b0 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15b582b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f15b0160df0 con 0x7f15b00d2b20 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15a7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15b00b81b0 0x7f15b00d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15aeffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f15b01615f0 0x7f15b00d0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35416/0 (socket says 172.21.15.46:35416) 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15aeffd640 1 -- 172.21.15.46:0/3462597728 learned_addr learned my addr 172.21.15.46:0/3462597728 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15af7fe640 1 --2- 172.21.15.46:0/3462597728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00d2b20 0x7f15b0163f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15a7fff640 1 -- 172.21.15.46:0/3462597728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00d2b20 msgr2=0x7f15b0163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15a7fff640 1 --2- 172.21.15.46:0/3462597728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00d2b20 0x7f15b0163f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15a7fff640 1 -- 172.21.15.46:0/3462597728 >> v1:172.21.15.46:6789/0 conn(0x7f15b01615f0 legacy=0x7f15b00d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15a7fff640 1 -- 172.21.15.46:0/3462597728 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b01610d0 con 0x7f15b00b81b0 2024-05-09T03:18:58.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.469+0000 7f15ae7fc640 1 -- 172.21.15.46:0/3462597728 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209292018 0 0) 0x7f15b00d15b0 con 0x7f15b01615f0 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.470+0000 7f15a7fff640 1 --2- 172.21.15.46:0/3462597728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b00b81b0 0x7f15b00d25e0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f15980097a0 tx=0x7f159802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5f67065b4dd1c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.470+0000 7f15ae7fc640 1 -- 172.21.15.46:0/3462597728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598037070 con 0x7f15b00b81b0 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.470+0000 7f15ae7fc640 1 -- 172.21.15.46:0/3462597728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1598032440 con 0x7f15b00b81b0 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.470+0000 7f15ae7fc640 1 -- 172.21.15.46:0/3462597728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598032810 con 0x7f15b00b81b0 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 -- 172.21.15.46:0/3462597728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b00b81b0 msgr2=0x7f15b00d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 --2- 172.21.15.46:0/3462597728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b00b81b0 0x7f15b00d25e0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f15980097a0 tx=0x7f159802a180 comp rx=0 tx=0).stop 2024-05-09T03:18:58.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 -- 172.21.15.46:0/3462597728 shutdown_connections 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 --2- 172.21.15.46:0/3462597728 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00d2b20 0x7f15b0163f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 --2- 172.21.15.46:0/3462597728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b00b81b0 0x7f15b00d25e0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 -- 172.21.15.46:0/3462597728 >> 172.21.15.46:0/3462597728 conn(0x7f15b006e190 msgr2=0x7f15b006e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 -- 172.21.15.46:0/3462597728 shutdown_connections 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.471+0000 7f15b582b640 1 -- 172.21.15.46:0/3462597728 wait complete. 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.473+0000 7f15b582b640 1 Processor -- start 2024-05-09T03:18:58.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.473+0000 7f15b582b640 1 -- start start 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15b582b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00b81b0 0x7f15b00ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15b582b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15b00d2b20 0x7f15b00cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15b582b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 0x7f15b00add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15b582b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15b00d1890 con 0x7f15b01615f0 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15b582b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f15b00b85c0 con 0x7f15b00d2b20 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15b582b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f15b00d2f30 con 0x7f15b00b81b0 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 0x7f15b00add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15af7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 0x7f15b00add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59276/0 (socket says 172.21.15.46:59276) 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15af7fe640 1 -- 172.21.15.46:0/1209355836 learned_addr learned my addr 172.21.15.46:0/1209355836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15aeffd640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00b81b0 0x7f15b00ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.474+0000 7f15a7fff640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15b00d2b20 0x7f15b00cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15af7fe640 1 -- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00b81b0 msgr2=0x7f15b00ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15af7fe640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00b81b0 0x7f15b00ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15af7fe640 1 -- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15b00d2b20 msgr2=0x7f15b00cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15af7fe640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15b00d2b20 0x7f15b00cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15af7fe640 1 -- 172.21.15.46:0/1209355836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b00cf450 con 0x7f15b01615f0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15aeffd640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00b81b0 0x7f15b00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15af7fe640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 0x7f15b00add70 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f15a800ef90 tx=0x7f15a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8018020 con 0x7f15b01615f0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a8014480 con 0x7f15b01615f0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b00cbbe0 con 0x7f15b01615f0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8009e60 con 0x7f15b01615f0 2024-05-09T03:18:58.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.475+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b00cc1a0 con 0x7f15b01615f0 2024-05-09T03:18:58.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.476+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b00cc8c0 con 0x7f15b01615f0 2024-05-09T03:18:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.477+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f15a8005d10 con 0x7f15b01615f0 2024-05-09T03:18:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.477+0000 7f15acff9640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f15900762d0 0x7f15900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.478+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f15a8098a50 con 0x7f15b01615f0 2024-05-09T03:18:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.478+0000 7f15aeffd640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f15900762d0 0x7f15900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.478+0000 7f15aeffd640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f15900762d0 0x7f15900787e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f15a0000f60 tx=0x7f15a0007ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.483+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f15a8022030 con 0x7f15b01615f0 2024-05-09T03:18:58.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.717+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15b006a270 con 0x7f15b01615f0 2024-05-09T03:18:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.718+0000 7f15acff9640 1 -- 172.21.15.46:0/1209355836 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) ==== 95+0+5185 (secure 0 0 0) 0x7f15a8010040 con 0x7f15b01615f0 2024-05-09T03:18:58.720 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:18:58.720 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":252,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:18:55.143529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9327},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9327":{"gid":9327,"name":"c","rank":0,"incarnation":249,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/2827001018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2827001018},{"type":"v1","addr":"172.21.15.46:6834","nonce":2827001018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9327,"qdb_cluster":[9327]},"id":6}]} 2024-05-09T03:18:58.720 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 252 2024-05-09T03:18:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f15900762d0 msgr2=0x7f15900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f15900762d0 0x7f15900787e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f15a0000f60 tx=0x7f15a0007ca0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 msgr2=0x7f15b00add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 0x7f15b00add70 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f15a800ef90 tx=0x7f15a800c490 comp rx=0 tx=0).stop 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 shutdown_connections 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f15900762d0 0x7f15900787e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15b01615f0 0x7f15b00add70 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15b00d2b20 0x7f15b00cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 --2- 172.21.15.46:0/1209355836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f15b00b81b0 0x7f15b00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 >> 172.21.15.46:0/1209355836 conn(0x7f15b006e190 msgr2=0x7f15b0163ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 shutdown_connections 2024-05-09T03:18:58.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.720+0000 7f15b582b640 1 -- 172.21.15.46:0/1209355836 wait complete. 2024-05-09T03:18:58.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 252, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:18:55.143529+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9327}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9327': {'gid': 9327, 'name': 'c', 'rank': 0, 'incarnation': 249, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/2827001018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2827001018}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2827001018}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9327, 'qdb_cluster': [9327]} 2024-05-09T03:18:58.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:18:58.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:18:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.847+0000 7f38e4a61640 1 Processor -- start 2024-05-09T03:18:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.848+0000 7f38e4a61640 1 -- start start 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38e4a61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38e0143410 0x7f38e00b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38e4a61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e00d1f70 0x7f38e0161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38e4a61640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38e01616b0 con 0x7f38e0160c60 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38e4a61640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f38e00b58b0 con 0x7f38e0143410 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38e4a61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38e00b4920 con 0x7f38e00d1f70 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38de59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38e0143410 0x7f38e00b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e00d1f70 0x7f38e0161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38ddd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f38e0160c60 0x7f38e00d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:51278/0 (socket says 172.21.15.46:51278) 2024-05-09T03:18:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38de59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38e0143410 0x7f38e00b5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:38290/0 (socket says 172.21.15.46:38290) 2024-05-09T03:18:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38ddd9b640 1 -- 172.21.15.46:0/3449027398 learned_addr learned my addr 172.21.15.46:0/3449027398 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38de59c640 1 -- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e00d1f70 msgr2=0x7f38e0161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38de59c640 1 --2- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e00d1f70 0x7f38e0161070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38de59c640 1 -- 172.21.15.46:0/3449027398 >> v1:172.21.15.88:6789/0 conn(0x7f38e0160c60 legacy=0x7f38e00d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.850+0000 7f38de59c640 1 -- 172.21.15.46:0/3449027398 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e0161990 con 0x7f38e0143410 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.851+0000 7f38de59c640 1 --2- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e00b5370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009d90 tx=0x7f38d802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1da5277114ecdb5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.851+0000 7f38dd59a640 1 -- 172.21.15.46:0/3449027398 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d8031600 con 0x7f38e0143410 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.851+0000 7f38dd59a640 1 -- 172.21.15.46:0/3449027398 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d80317a0 con 0x7f38e0143410 2024-05-09T03:18:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.851+0000 7f38dd59a640 1 -- 172.21.15.46:0/3449027398 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d8031b70 con 0x7f38e0143410 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.851+0000 7f38e4a61640 1 -- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 msgr2=0x7f38e00b5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.851+0000 7f38e4a61640 1 --2- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e00b5370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009d90 tx=0x7f38d802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.853+0000 7f38e4a61640 1 -- 172.21.15.46:0/3449027398 shutdown_connections 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.853+0000 7f38e4a61640 1 --2- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e00d1f70 0x7f38e0161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.853+0000 7f38e4a61640 1 --2- 172.21.15.46:0/3449027398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e00b5370 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.853+0000 7f38e4a61640 1 -- 172.21.15.46:0/3449027398 >> 172.21.15.46:0/3449027398 conn(0x7f38e00b3620 msgr2=0x7f38e00b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.853+0000 7f38e4a61640 1 -- 172.21.15.46:0/3449027398 shutdown_connections 2024-05-09T03:18:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.853+0000 7f38e4a61640 1 -- 172.21.15.46:0/3449027398 wait complete. 2024-05-09T03:18:58.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 Processor -- start 2024-05-09T03:18:58.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 -- start start 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e0160c60 0x7f38e00ad480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 0x7f38e00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0143820 con 0x7f38e00732d0 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f38e00b4ee0 con 0x7f38e0143410 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38e4a61640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38e00d1210 con 0x7f38e0160c60 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 0x7f38e00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 0x7f38e00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59286/0 (socket says 172.21.15.46:59286) 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 -- 172.21.15.46:0/125565813 learned_addr learned my addr 172.21.15.46:0/125565813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e0160c60 0x7f38e00ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38de59c640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 -- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e0160c60 msgr2=0x7f38e00ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e0160c60 0x7f38e00ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 -- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 msgr2=0x7f38e015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e015dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.855+0000 7f38ded9d640 1 -- 172.21.15.46:0/125565813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e0073c20 con 0x7f38e00732d0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.856+0000 7f38ded9d640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 0x7f38e00736e0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f38d000e7c0 tx=0x7f38d000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.856+0000 7f38d6ffd640 1 -- 172.21.15.46:0/125565813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0017070 con 0x7f38e00732d0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.856+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e00b0900 con 0x7f38e00732d0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.856+0000 7f38d6ffd640 1 -- 172.21.15.46:0/125565813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d000c730 con 0x7f38e00732d0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.856+0000 7f38d6ffd640 1 -- 172.21.15.46:0/125565813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d000cb20 con 0x7f38e00732d0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.856+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e00aeee0 con 0x7f38e00732d0 2024-05-09T03:18:58.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.858+0000 7f38d6ffd640 1 -- 172.21.15.46:0/125565813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f38d001a070 con 0x7f38e00732d0 2024-05-09T03:18:58.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.859+0000 7f38d6ffd640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38c00762d0 0x7f38c00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.859+0000 7f38d6ffd640 1 -- 172.21.15.46:0/125565813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f38d0097d00 con 0x7f38e00732d0 2024-05-09T03:18:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.859+0000 7f38de59c640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38c00762d0 0x7f38c00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.859+0000 7f38eb9ef740 1 Processor -- start 2024-05-09T03:18:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.859+0000 7f38eb9ef740 1 -- start start 2024-05-09T03:18:58.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.860+0000 7f38de59c640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38c00762d0 0x7f38c00787e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38d8034000 tx=0x7f38d802f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb29eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb45120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d21ba624e0 con 0x55d21b739590 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d21b9fdff0 con 0x55d21b7313b0 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d21ba5e4a0 con 0x55d21b75a270 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb45120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38de59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb29eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ddd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb45120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59290/0 (socket says 172.21.15.46:59290) 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38de59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb29eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:37080/0 (socket says 172.21.15.46:37080) 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 -- 172.21.15.46:0/4151859656 learned_addr learned my addr 172.21.15.46:0/4151859656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 -- 172.21.15.46:0/4151859656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 msgr2=0x55d21bb29eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb29eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 -- 172.21.15.46:0/4151859656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 msgr2=0x55d21bb44be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 -- 172.21.15.46:0/4151859656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d21b9e4720 con 0x55d21b739590 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.861+0000 7f38ded9d640 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb45120 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f38d0004770 tx=0x7f38d0060260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f41ef9bc6b46bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38bffff640 1 -- 172.21.15.46:0/4151859656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0021070 con 0x55d21b739590 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38bffff640 1 -- 172.21.15.46:0/4151859656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d00693b0 con 0x55d21b739590 2024-05-09T03:18:58.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38bffff640 1 -- 172.21.15.46:0/4151859656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d005f8f0 con 0x55d21b739590 2024-05-09T03:18:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 -- 172.21.15.46:0/4151859656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 msgr2=0x55d21bb45120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb45120 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f38d0004770 tx=0x7f38d0060260 comp rx=0 tx=0).stop 2024-05-09T03:18:58.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 -- 172.21.15.46:0/4151859656 shutdown_connections 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb45120 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/4151859656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb29eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.862+0000 7f38eb9ef740 1 -- 172.21.15.46:0/4151859656 >> 172.21.15.46:0/4151859656 conn(0x55d21b74e3c0 msgr2=0x55d21baff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.863+0000 7f38eb9ef740 1 -- 172.21.15.46:0/4151859656 shutdown_connections 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.863+0000 7f38eb9ef740 1 -- 172.21.15.46:0/4151859656 wait complete. 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.863+0000 7f38eb9ef740 1 -- start start 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.864+0000 7f38eb9ef740 1 Processor -- start 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb42700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb42c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d21b9e9760 con 0x55d21b739590 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d21bb40180 con 0x55d21b7313b0 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d21bb452a0 con 0x55d21b75a270 2024-05-09T03:18:58.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38eb9ef740 10 client.0 _ll_register_callbacks cb 0x55d21ba0c4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38ddd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb42700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38ddd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb42700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59298/0 (socket says 172.21.15.46:59298) 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38ddd9b640 1 -- 172.21.15.46:0/2272077499 learned_addr learned my addr 172.21.15.46:0/2272077499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38de59c640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.865+0000 7f38ded9d640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb42c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ddd9b640 1 -- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 msgr2=0x55d21bb42c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb42c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ddd9b640 1 -- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 msgr2=0x55d21bb44780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ddd9b640 1 -- 172.21.15.46:0/2272077499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d21ba2abd0 con 0x55d21b739590 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ded9d640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb42c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb42700 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f38c8007700 tx=0x7f38c80015f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9636e3a9825ff11f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2272077499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8001ce0 con 0x55d21b739590 2024-05-09T03:18:58.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d00693b0 con 0x55d21b739590 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2272077499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f38c8005a60 con 0x55d21b739590 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2272077499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=252) ==== 24+0+0 (secure 0 0 0) 0x7f38c8005c60 con 0x55d21b739590 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 10 client.9417 fetch_fsmap learned FSMap version 252 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=252,monmap=2+}) -- 0x55d21ba10280 con 0x55d21b739590 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2272077499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d00693b0 con 0x55d21b739590 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2272077499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 252) ==== 3494+0+0 (secure 0 0 0) 0x7f38c8006050 con 0x55d21b739590 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 10 client.9417 fetch_fsmap finished waiting for FSMap version 252 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 10 client.9417 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 20 client.9417 populate_metadata read hostname 'smithi046' 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21b718e40 0x55d21bb57500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 4 client.9417 mds_command: new command op to 9327 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.866+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 --> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d21ba700f0 con 0x55d21b718e40 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.867+0000 7f38de59c640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21b718e40 0x55d21bb57500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.867+0000 7f38bd7fa640 10 client.9417 ms_handle_connect on v2:172.21.15.46:6832/2827001018 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.867+0000 7f38de59c640 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21b718e40 0x55d21bb57500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbb2bf21797cd607 server_cookie=5537fd3cc8c6c8a5 in_seq=0 out_seq=0 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2272077499 <== mds.0 v2:172.21.15.46:6832/2827001018 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d21ba700f0 con 0x55d21b718e40 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38bd7fa640 10 client.9417 handle_command_reply: tid=0 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 client.9417 shutdown 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 msgr2=0x55d21bb42700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb42700 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f38c8007700 tx=0x7f38c80015f0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 shutdown_connections 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21b718e40 0x55d21bb57500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b75a270 0x55d21bb42c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b739590 0x55d21bb42700 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38bd7fa640 0 client.9417 ms_handle_reset on v2:172.21.15.46:6832/2827001018 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2272077499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bb44780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 >> 172.21.15.46:0/2272077499 conn(0x55d21b74e3c0 msgr2=0x55d21bb29eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 shutdown_connections 2024-05-09T03:18:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2272077499 wait complete. 2024-05-09T03:18:58.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.868+0000 7f38eb9ef740 20 client.9417 trim_cache size 0 max 16384 2024-05-09T03:18:58.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.888+0000 7f38eb9ef740 1 Processor -- start 2024-05-09T03:18:58.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.888+0000 7f38eb9ef740 1 -- start start 2024-05-09T03:18:58.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 0x55d21bb67d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b75a270 0x55d21bb43af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0201040 con 0x55d21b7313b0 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0200400 con 0x55d21b75a270 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38e02009b0 con 0x55d21b739590 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 0x55d21bb67d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 0x55d21bb67d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59314/0 (socket says 172.21.15.46:59314) 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 -- 172.21.15.46:0/2470354284 learned_addr learned my addr 172.21.15.46:0/2470354284 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38ded9d640 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b75a270 0x55d21bb43af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 -- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 msgr2=0x55d21bb435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 -- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b75a270 msgr2=0x55d21bb43af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b75a270 0x55d21bb43af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 -- 172.21.15.46:0/2470354284 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d21b7c5850 con 0x55d21b7313b0 2024-05-09T03:18:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.889+0000 7f38de59c640 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 0x55d21bb67d00 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009e80 tx=0x7f38d802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6dfcc3f6262a1a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38bf7fe640 1 -- 172.21.15.46:0/2470354284 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d8055720 con 0x55d21b7313b0 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38bf7fe640 1 -- 172.21.15.46:0/2470354284 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d803ce50 con 0x55d21b7313b0 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38bf7fe640 1 -- 172.21.15.46:0/2470354284 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d8055ac0 con 0x55d21b7313b0 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2470354284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 msgr2=0x55d21bb67d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 0x55d21bb67d00 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009e80 tx=0x7f38d802e000 comp rx=0 tx=0).stop 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2470354284 shutdown_connections 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b75a270 0x55d21bb43af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2470354284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b7313b0 0x55d21bb67d00 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2470354284 >> 172.21.15.46:0/2470354284 conn(0x55d21b74e3c0 msgr2=0x55d21bb3e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2470354284 shutdown_connections 2024-05-09T03:18:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.890+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2470354284 wait complete. 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.891+0000 7f38eb9ef740 1 -- start start 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.891+0000 7f38eb9ef740 1 Processor -- start 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bbc0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 0x55d21bb43500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d21bb2c990 con 0x55d21b75a270 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d21bb1b980 con 0x55d21b7313b0 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d21bb2cfe0 con 0x55d21b739590 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38eb9ef740 10 client.0 _ll_register_callbacks cb 0x7f38c8006a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 0x55d21bb43500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 0x55d21bb43500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59322/0 (socket says 172.21.15.46:59322) 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 -- 172.21.15.46:0/2451669117 learned_addr learned my addr 172.21.15.46:0/2451669117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:58.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38de59c640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bbc0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ddd9b640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 -- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 msgr2=0x55d21bb463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 -- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 msgr2=0x55d21bbc0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bbc0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 -- 172.21.15.46:0/2451669117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d21b83c6b0 con 0x55d21b75a270 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38ded9d640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 0x55d21bb43500 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f38d0060390 tx=0x7f38d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c463930139055b39 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.893+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2451669117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d00693b0 con 0x55d21b75a270 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2451669117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f38d0065b20 con 0x55d21b75a270 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38d803ce50 con 0x55d21b75a270 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2451669117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=252) ==== 24+0+0 (secure 0 0 0) 0x7f38d0065d20 con 0x55d21b75a270 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 10 client.9423 fetch_fsmap learned FSMap version 252 2024-05-09T03:18:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=252,monmap=2+}) -- 0x55d21ba1e630 con 0x55d21b75a270 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2451669117 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d803ce50 con 0x55d21b75a270 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2451669117 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 252) ==== 3494+0+0 (secure 0 0 0) 0x7f38d0024060 con 0x55d21b75a270 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 10 client.9423 fetch_fsmap finished waiting for FSMap version 252 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 10 client.9423 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 20 client.9423 populate_metadata read hostname 'smithi046' 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21bb2c0b0 0x55d21bb43a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 4 client.9423 mds_command: new command op to 9327 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 --> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] -- command(tid 0: {"prefix": "status"}) -- 0x55d21ba69f60 con 0x55d21bb2c0b0 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.894+0000 7f38de59c640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21bb2c0b0 0x55d21bb43a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38bd7fa640 10 client.9423 ms_handle_connect on v2:172.21.15.46:6832/2827001018 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38de59c640 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21bb2c0b0 0x55d21bb43a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63291dac341345ad server_cookie=430945741f39bae4 in_seq=0 out_seq=0 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38bd7fa640 1 -- 172.21.15.46:0/2451669117 <== mds.0 v2:172.21.15.46:6832/2827001018 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55d21ba69f60 con 0x55d21bb2c0b0 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38bd7fa640 10 client.9423 handle_command_reply: tid=0 2024-05-09T03:18:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 client.9423 shutdown 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 msgr2=0x55d21bb43500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 0x55d21bb43500 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f38d0060390 tx=0x7f38d00046c0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 shutdown_connections 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:6832/2827001018,v1:172.21.15.46:6834/2827001018] conn(0x55d21bb2c0b0 0x55d21bb43a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d21b75a270 0x55d21bb43500 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d21b739590 0x55d21bb463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 --2- 172.21.15.46:0/2451669117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d21b7313b0 0x55d21bbc0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38bd7fa640 0 client.9423 ms_handle_reset on v2:172.21.15.46:6832/2827001018 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 >> 172.21.15.46:0/2451669117 conn(0x55d21b74e3c0 msgr2=0x55d21bb3f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 shutdown_connections 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 1 -- 172.21.15.46:0/2451669117 wait complete. 2024-05-09T03:18:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.895+0000 7f38eb9ef740 20 client.9423 trim_cache size 0 max 16384 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9327, 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7659660160000001, 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 252, 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 110, 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 110, 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.731599971 2024-05-09T03:18:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:18:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38c00762d0 msgr2=0x7f38c00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38c00762d0 0x7f38c00787e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38d8034000 tx=0x7f38d802f000 comp rx=0 tx=0).stop 2024-05-09T03:18:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 msgr2=0x7f38e00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 0x7f38e00736e0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f38d000e7c0 tx=0x7f38d000ec80 comp rx=0 tx=0).stop 2024-05-09T03:18:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 shutdown_connections 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f38c00762d0 0x7f38c00787e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38e00732d0 0x7f38e00736e0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f38e0160c60 0x7f38e00ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 --2- 172.21.15.46:0/125565813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38e0143410 0x7f38e015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 >> 172.21.15.46:0/125565813 conn(0x7f38e00b3620 msgr2=0x7f38e0071000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 shutdown_connections 2024-05-09T03:18:58.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:58.896+0000 7f38e4a61640 1 -- 172.21.15.46:0/125565813 wait complete. 2024-05-09T03:18:58.908 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:18:58.909 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Truncate 500.00000000', proceeding to ls 2024-05-09T03:18:58.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:18:59.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.022+0000 7f67f5489640 1 Processor -- start 2024-05-09T03:18:59.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.022+0000 7f67f5489640 1 -- start start 2024-05-09T03:18:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67f5489640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00d0280 0x7f67f00b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67f5489640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f00cef80 con 0x7f67f00ce590 2024-05-09T03:18:59.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67f5489640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f00b2540 con 0x7f67f00d31e0 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67f5489640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f67f00b2b90 con 0x7f67f00d0280 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67ee7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f67f00ce590 0x7f67f00d2ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35420/0 (socket says 172.21.15.46:35420) 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67ee7fc640 1 -- 172.21.15.46:0/2875142139 learned_addr learned my addr 172.21.15.46:0/2875142139 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67eeffd640 1 --2- 172.21.15.46:0/2875142139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00d0280 0x7f67f00b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413583093 0 0) 0x7f67f00cef80 con 0x7f67f00ce590 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67dc003520 con 0x7f67f00ce590 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1162130807 0 0) 0x7f67f00b2540 con 0x7f67f00d31e0 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67f00cef80 con 0x7f67f00d31e0 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67eeffd640 1 -- 172.21.15.46:0/2875142139 >> v1:172.21.15.88:6789/0 conn(0x7f67f00d31e0 legacy=0x7f67f00b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67eeffd640 1 -- 172.21.15.46:0/2875142139 >> v1:172.21.15.46:6789/0 conn(0x7f67f00ce590 legacy=0x7f67f00d2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.024+0000 7f67eeffd640 1 -- 172.21.15.46:0/2875142139 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f00b2820 con 0x7f67f00d0280 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67eeffd640 1 --2- 172.21.15.46:0/2875142139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00d0280 0x7f67f00b3c10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009e10 tx=0x7f67e002a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74d6661b7c6d99b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0007740 con 0x7f67f00d0280 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e00078e0 con 0x7f67f00d0280 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67edffb640 1 -- 172.21.15.46:0/2875142139 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0007cb0 con 0x7f67f00d0280 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67f5489640 1 -- 172.21.15.46:0/2875142139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00d0280 msgr2=0x7f67f00b3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67f5489640 1 --2- 172.21.15.46:0/2875142139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00d0280 0x7f67f00b3c10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009e10 tx=0x7f67e002a730 comp rx=0 tx=0).stop 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67f5489640 1 -- 172.21.15.46:0/2875142139 shutdown_connections 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67f5489640 1 --2- 172.21.15.46:0/2875142139 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00d0280 0x7f67f00b3c10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.025+0000 7f67f5489640 1 -- 172.21.15.46:0/2875142139 >> 172.21.15.46:0/2875142139 conn(0x7f67f009b7d0 msgr2=0x7f67f009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.026+0000 7f67f5489640 1 -- 172.21.15.46:0/2875142139 shutdown_connections 2024-05-09T03:18:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.026+0000 7f67f5489640 1 -- 172.21.15.46:0/2875142139 wait complete. 2024-05-09T03:18:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.027+0000 7f67f5489640 1 Processor -- start 2024-05-09T03:18:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.027+0000 7f67f5489640 1 -- start start 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67f5489640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00ce590 0x7f67f014ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67f5489640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 0x7f67f00c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67f5489640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67f00c8140 0x7f67f0071440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67f5489640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67f00b42d0 con 0x7f67f00d0280 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67f5489640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67f00b4150 con 0x7f67f00c8140 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67f5489640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f67f015e4e0 con 0x7f67f00ce590 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 0x7f67f00c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 0x7f67f00c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59328/0 (socket says 172.21.15.46:59328) 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 -- 172.21.15.46:0/2596488175 learned_addr learned my addr 172.21.15.46:0/2596488175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67eeffd640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00ce590 0x7f67f014ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:59.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ef7fe640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67f00c8140 0x7f67f0071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 -- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00ce590 msgr2=0x7f67f014ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00ce590 0x7f67f014ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 -- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67f00c8140 msgr2=0x7f67f0071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67f00c8140 0x7f67f0071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 -- 172.21.15.46:0/2596488175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f006cee0 con 0x7f67f00d0280 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.029+0000 7f67ee7fc640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 0x7f67f00c7c00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f67d800c410 tx=0x7f67d800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.030+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d8017070 con 0x7f67f00d0280 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.030+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d8005da0 con 0x7f67f00d0280 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.030+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f006d1c0 con 0x7f67f00d0280 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.030+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d80072f0 con 0x7f67f00d0280 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.030+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f006d780 con 0x7f67f00d0280 2024-05-09T03:18:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.030+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f00c86a0 con 0x7f67f00d0280 2024-05-09T03:18:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.031+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f67d801a070 con 0x7f67f00d0280 2024-05-09T03:18:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.032+0000 7f67e77fe640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d00762d0 0x7f67d00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:18:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.032+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f67d8097300 con 0x7f67f00d0280 2024-05-09T03:18:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.032+0000 7f67eeffd640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d00762d0 0x7f67d00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:18:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.033+0000 7f67eeffd640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d00762d0 0x7f67d00787e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f67e0034000 tx=0x7f67e0033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:18:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.037+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f67d8060360 con 0x7f67f00d0280 2024-05-09T03:18:59.118 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:18:59.116+0000 7f7c0e3e5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:18:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.272+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f67f006a270 con 0x7f67f00d0280 2024-05-09T03:18:59.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.272+0000 7f67e77fe640 1 -- 172.21.15.46:0/2596488175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v252) ==== 53+0+73 (secure 0 0 0) 0x7f67d805fc60 con 0x7f67f00d0280 2024-05-09T03:18:59.274 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:18:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d00762d0 msgr2=0x7f67d00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d00762d0 0x7f67d00787e0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f67e0034000 tx=0x7f67e0033040 comp rx=0 tx=0).stop 2024-05-09T03:18:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 msgr2=0x7f67f00c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:18:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 0x7f67f00c7c00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f67d800c410 tx=0x7f67d800c8d0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 shutdown_connections 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67f00c8140 0x7f67f0071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d00762d0 0x7f67d00787e0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f00d0280 0x7f67f00c7c00 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 --2- 172.21.15.46:0/2596488175 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67f00ce590 0x7f67f014ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.274+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 >> 172.21.15.46:0/2596488175 conn(0x7f67f009b7d0 msgr2=0x7f67f0070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.275+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 shutdown_connections 2024-05-09T03:18:59.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59.275+0000 7f67f5489640 1 -- 172.21.15.46:0/2596488175 wait complete. 2024-05-09T03:18:59.287 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:18:59.287 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:18:59.287 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:18:59.287 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:18:59.288 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:59.288 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:18:59.288 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:18:59.288 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:18:59.288 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:18:59.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:59.288 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:18:59.304 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:18:59.304 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:18:59.304 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41978sec preferred_lft 41978sec 2024-05-09T03:18:59.305 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.306 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:18:59.307 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:18:59.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:59.308 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:18:59.369 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:18:59.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:59.371 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:18:59.427 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:18:59.428 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:18:59.428 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:18:59.428 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:18:59.428 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:18:59.428 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:18:59.428 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:18:59.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:18:59.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:18:59.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:18:59.428 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:18:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:18:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:18:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:18:59.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:59.557 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:18:59.574 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:18:59.574 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:18:59.574 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:18:59.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:59.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:18:59.640 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:18:59.641 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:18:59.641 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:59.696 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:18:59.696 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:18:59.696 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:18:59.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:18:59.819 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:18:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:18:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:18:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:18:59.877 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:18:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:18:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:18:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:18:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:18:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:18:59.878 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:18:59.879 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:18:59.880 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:18:59.881 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:59.882 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:18:59.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:18:59.882 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:18:59.933 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:18:59.933 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:18:59.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:19:00.002 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:19:00.059 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:19:00.060 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:19:00.061 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:00.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:00.062 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:19:00.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:19:00.071+0000 7f4960b72480 -1 init, newargv = 0x7f4950005d20 newargc=15 2024-05-09T03:19:00.072 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[78050]: starting ceph client 2024-05-09T03:19:00.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[78050]: starting fuse 2024-05-09T03:19:00.088 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:19:00.088 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:19:00.089 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:19:00.090 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:19:00.211 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.78050.asok 2024-05-09T03:19:00.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:19:00.222 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78050.asok 2024-05-09T03:19:00.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:00.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78050.asok status 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:19:00.370 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "78050", 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9435, 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 9435 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1355077484 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:19:00.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1355077484 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.9435 192.168.0.1:0/1355077484", 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1355077484", 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 252, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 110, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:19:00.372 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:19:00.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:00.379 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:19:00.379 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:00.395 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:19:00.396 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:00.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:00.396 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:00.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:00.461 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:19:00.504 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:19:00.521 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:19:00.522 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:19:00.522 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:00.522 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:19:00.522 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs 2024-05-09T03:19:00.522 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 500.00000000' did not caused DAMAGED state 2024-05-09T03:19:00.522 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate mds_snaptable' 2024-05-09T03:19:00.522 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:00.522 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:00.579 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:19:00.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.733+0000 7fa069c22640 1 Processor -- start 2024-05-09T03:19:00.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.734+0000 7fa069c22640 1 -- start start 2024-05-09T03:19:00.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa069c22640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa0640b5140 0x7fa0640b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:00.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa069c22640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640d3fc0 0x7fa0640b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:00.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa069c22640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0640b80b0 con 0x7fa0640b7ca0 2024-05-09T03:19:00.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa069c22640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa0640d3480 con 0x7fa0640b5140 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa069c22640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa0640b5550 con 0x7fa0640d3fc0 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa062ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa0640b7ca0 0x7fa0640d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35434/0 (socket says 172.21.15.46:35434) 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa062ffd640 1 -- 172.21.15.46:0/1626087984 learned_addr learned my addr 172.21.15.46:0/1626087984 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa0637fe640 1 --2- 172.21.15.46:0/1626087984 >> v2:172.21.15.88:3300/0 conn(0x7fa0640b5140 0x7fa0640b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa063fff640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640d3fc0 0x7fa0640b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa0627fc640 1 -- 172.21.15.46:0/1626087984 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243904437 0 0) 0x7fa058001110 con 0x7fa0640b7ca0 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.735+0000 7fa0627fc640 1 -- 172.21.15.46:0/1626087984 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa04c003520 con 0x7fa0640b7ca0 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa0637fe640 1 -- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640d3fc0 msgr2=0x7fa0640b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa0637fe640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640d3fc0 0x7fa0640b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa0627fc640 1 -- 172.21.15.46:0/1626087984 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2926071790 0 0) 0x7fa04c003520 con 0x7fa0640b7ca0 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa0637fe640 1 -- 172.21.15.46:0/1626087984 >> v1:172.21.15.46:6789/0 conn(0x7fa0640b7ca0 legacy=0x7fa0640d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa0637fe640 1 -- 172.21.15.46:0/1626087984 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0640d44e0 con 0x7fa0640b5140 2024-05-09T03:19:00.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa063fff640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640d3fc0 0x7fa0640b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.736+0000 7fa0637fe640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b5140 0x7fa0640b7760 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0540097a0 tx=0x7fa05402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d37f9f0a7149c1f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa0627fc640 1 -- 172.21.15.46:0/1626087984 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa054037070 con 0x7fa0640b5140 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa0627fc640 1 -- 172.21.15.46:0/1626087984 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa05402ac30 con 0x7fa0640b5140 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa0627fc640 1 -- 172.21.15.46:0/1626087984 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0580018c0 con 0x7fa0640b5140 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 -- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b5140 msgr2=0x7fa0640b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b5140 0x7fa0640b7760 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0540097a0 tx=0x7fa05402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 -- 172.21.15.46:0/1626087984 shutdown_connections 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640d3fc0 0x7fa0640b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 --2- 172.21.15.46:0/1626087984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b5140 0x7fa0640b7760 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 -- 172.21.15.46:0/1626087984 >> 172.21.15.46:0/1626087984 conn(0x7fa06408d3f0 msgr2=0x7fa06408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.737+0000 7fa069c22640 1 -- 172.21.15.46:0/1626087984 shutdown_connections 2024-05-09T03:19:00.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.738+0000 7fa069c22640 1 -- 172.21.15.46:0/1626087984 wait complete. 2024-05-09T03:19:00.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.739+0000 7fa069c22640 1 Processor -- start 2024-05-09T03:19:00.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.740+0000 7fa069c22640 1 -- start start 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa069c22640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640b5140 0x7fa064154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa069c22640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b7ca0 0x7fa06415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa069c22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 0x7fa06415b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa069c22640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0640d4d80 con 0x7fa0640d3fc0 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa069c22640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa0640d4aa0 con 0x7fa0640b7ca0 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa069c22640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa0640d5580 con 0x7fa0640b5140 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 0x7fa06415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 0x7fa06415b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59330/0 (socket says 172.21.15.46:59330) 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 -- 172.21.15.46:0/3480591417 learned_addr learned my addr 172.21.15.46:0/3480591417 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa0637fe640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640b5140 0x7fa064154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa062ffd640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b7ca0 0x7fa06415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 -- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640b5140 msgr2=0x7fa064154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640b5140 0x7fa064154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 -- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b7ca0 msgr2=0x7fa06415af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b7ca0 0x7fa06415af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.741+0000 7fa063fff640 1 -- 172.21.15.46:0/3480591417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa064154fa0 con 0x7fa0640d3fc0 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.742+0000 7fa063fff640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 0x7fa06415b470 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009e00 tx=0x7fa05c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.742+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05c018020 con 0x7fa0640d3fc0 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.742+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa05c014480 con 0x7fa0640d3fc0 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.742+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa064159940 con 0x7fa0640d3fc0 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.742+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa05c014870 con 0x7fa0640d3fc0 2024-05-09T03:19:00.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.742+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa06415a150 con 0x7fa0640d3fc0 2024-05-09T03:19:00.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.743+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa064159be0 con 0x7fa0640d3fc0 2024-05-09T03:19:00.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.743+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa05c005d10 con 0x7fa0640d3fc0 2024-05-09T03:19:00.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.744+0000 7fa04bfff640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0500762d0 0x7fa0500787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:00.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.744+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7fa05c098a90 con 0x7fa0640d3fc0 2024-05-09T03:19:00.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.745+0000 7fa0637fe640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0500762d0 0x7fa0500787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:00.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.745+0000 7fa0637fe640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0500762d0 0x7fa0500787e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa054034000 tx=0x7fa05402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:00.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.749+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa05c022030 con 0x7fa0640d3fc0 2024-05-09T03:19:00.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.954+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa06406a270 con 0x7fa0640d3fc0 2024-05-09T03:19:00.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.955+0000 7fa04bfff640 1 -- 172.21.15.46:0/3480591417 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 52 entries v110) ==== 81+0+3162 (secure 0 0 0) 0x7fa05c010040 con 0x7fa0640d3fc0 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:00.956 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:00.957 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:00.958 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:00.959 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:00.960 INFO:teuthology.orchestra.run.smithi046.stderr:listed 52 entries 2024-05-09T03:19:00.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0500762d0 msgr2=0x7fa0500787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0500762d0 0x7fa0500787e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fa054034000 tx=0x7fa05402e000 comp rx=0 tx=0).stop 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 msgr2=0x7fa06415b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 0x7fa06415b470 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa05c009e00 tx=0x7fa05c00c490 comp rx=0 tx=0).stop 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 shutdown_connections 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa0500762d0 0x7fa0500787e0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0640d3fc0 0x7fa06415b470 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0640b7ca0 0x7fa06415af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 --2- 172.21.15.46:0/3480591417 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa0640b5140 0x7fa064154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 >> 172.21.15.46:0/3480591417 conn(0x7fa06408d3f0 msgr2=0x7fa06415efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 shutdown_connections 2024-05-09T03:19:00.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:00.957+0000 7fa069c22640 1 -- 172.21.15.46:0/3480591417 wait complete. 2024-05-09T03:19:00.969 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:00.969 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:00.986 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:19:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.141+0000 7fe580b57640 1 Processor -- start 2024-05-09T03:19:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.141+0000 7fe580b57640 1 -- start start 2024-05-09T03:19:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe580b57640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe57c0c78a0 0x7fe57c0b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe580b57640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe57c072500 0x7fe57c163e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe580b57640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57c0c8720 con 0x7fe57c164390 2024-05-09T03:19:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe580b57640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe57c145d10 con 0x7fe57c0c78a0 2024-05-09T03:19:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe580b57640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe57c0c7e80 con 0x7fe57c072500 2024-05-09T03:19:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe57a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe57c0c78a0 0x7fe57c0b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe57a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe57c0c78a0 0x7fe57c0b1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59342/0 (socket says 172.21.15.46:59342) 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.142+0000 7fe57a59c640 1 -- 172.21.15.46:0/1700638743 learned_addr learned my addr 172.21.15.46:0/1700638743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.143+0000 7fe573fff640 1 --2- 172.21.15.46:0/1700638743 >> v2:172.21.15.88:3300/0 conn(0x7fe57c072500 0x7fe57c163e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.143+0000 7fe57a59c640 1 -- 172.21.15.46:0/1700638743 >> v2:172.21.15.88:3300/0 conn(0x7fe57c072500 msgr2=0x7fe57c163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.143+0000 7fe57a59c640 1 --2- 172.21.15.46:0/1700638743 >> v2:172.21.15.88:3300/0 conn(0x7fe57c072500 0x7fe57c163e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.143+0000 7fe57a59c640 1 -- 172.21.15.46:0/1700638743 >> v1:172.21.15.88:6789/0 conn(0x7fe57c164390 legacy=0x7fe57c1461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.143+0000 7fe57a59c640 1 -- 172.21.15.46:0/1700638743 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c0c8160 con 0x7fe57c0c78a0 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.143+0000 7fe573fff640 1 --2- 172.21.15.46:0/1700638743 >> v2:172.21.15.88:3300/0 conn(0x7fe57c072500 0x7fe57c163e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe57a59c640 1 --2- 172.21.15.46:0/1700638743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c0c78a0 0x7fe57c0b1dd0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0740c0 tx=0x7fe56c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f489fae94b4c545 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe579d9b640 1 -- 172.21.15.46:0/1700638743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c037070 con 0x7fe57c0c78a0 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe579d9b640 1 -- 172.21.15.46:0/1700638743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56c032440 con 0x7fe57c0c78a0 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe579d9b640 1 -- 172.21.15.46:0/1700638743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c032810 con 0x7fe57c0c78a0 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe580b57640 1 -- 172.21.15.46:0/1700638743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c0c78a0 msgr2=0x7fe57c0b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe580b57640 1 --2- 172.21.15.46:0/1700638743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c0c78a0 0x7fe57c0b1dd0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0740c0 tx=0x7fe56c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe580b57640 1 -- 172.21.15.46:0/1700638743 shutdown_connections 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe580b57640 1 --2- 172.21.15.46:0/1700638743 >> v2:172.21.15.88:3300/0 conn(0x7fe57c072500 0x7fe57c163e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe580b57640 1 --2- 172.21.15.46:0/1700638743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c0c78a0 0x7fe57c0b1dd0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.144+0000 7fe580b57640 1 -- 172.21.15.46:0/1700638743 >> 172.21.15.46:0/1700638743 conn(0x7fe57c0705c0 msgr2=0x7fe57c0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.145+0000 7fe580b57640 1 -- 172.21.15.46:0/1700638743 shutdown_connections 2024-05-09T03:19:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.145+0000 7fe580b57640 1 -- 172.21.15.46:0/1700638743 wait complete. 2024-05-09T03:19:01.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.146+0000 7fe580b57640 1 Processor -- start 2024-05-09T03:19:01.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.147+0000 7fe580b57640 1 -- start start 2024-05-09T03:19:01.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe580b57640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 0x7fe57c1f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe580b57640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe57c0c78a0 0x7fe57c143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe580b57640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe57c164390 0x7fe57c0b0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe580b57640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c1468b0 con 0x7fe57c072500 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe580b57640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe57c072910 con 0x7fe57c164390 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe580b57640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe57c146a30 con 0x7fe57c0c78a0 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 0x7fe57c1f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 0x7fe57c1f67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59346/0 (socket says 172.21.15.46:59346) 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 -- 172.21.15.46:0/3835015141 learned_addr learned my addr 172.21.15.46:0/3835015141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe573fff640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe57c0c78a0 0x7fe57c143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57ad9d640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe57c164390 0x7fe57c0b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 -- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe57c0c78a0 msgr2=0x7fe57c143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe57c0c78a0 0x7fe57c143410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 -- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe57c164390 msgr2=0x7fe57c0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe57c164390 0x7fe57c0b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.148+0000 7fe57a59c640 1 -- 172.21.15.46:0/3835015141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57c1f7960 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.149+0000 7fe57a59c640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 0x7fe57c1f67a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fe56c000c00 tx=0x7fe56c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.149+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c041020 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.149+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56c030970 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.149+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57c143950 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.149+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c037040 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.149+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57c1f7340 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.150+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe57c06a270 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.151+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe56c044020 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.151+0000 7fe572ffd640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe55c0762d0 0x7fe55c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.151+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7fe56c0b8590 con 0x7fe57c072500 2024-05-09T03:19:01.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.151+0000 7fe573fff640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe55c0762d0 0x7fe55c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.152+0000 7fe573fff640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe55c0762d0 0x7fe55c0787e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0b4530 tx=0x7fe564008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:01.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.156+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe56c04b070 con 0x7fe57c072500 2024-05-09T03:19:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.364+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe57c0b8c10 con 0x7fe57c072500 2024-05-09T03:19:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.365+0000 7fe572ffd640 1 -- 172.21.15.46:0/3835015141 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 52 entries v110) ==== 81+0+3162 (secure 0 0 0) 0x7fe56c080f10 con 0x7fe57c072500 2024-05-09T03:19:01.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:01.365 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:01.366 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:01.367 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:01.369 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:listed 52 entries 2024-05-09T03:19:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe55c0762d0 msgr2=0x7fe55c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe55c0762d0 0x7fe55c0787e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0b4530 tx=0x7fe564008040 comp rx=0 tx=0).stop 2024-05-09T03:19:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 msgr2=0x7fe57c1f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 0x7fe57c1f67a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fe56c000c00 tx=0x7fe56c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 shutdown_connections 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe57c164390 0x7fe57c0b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe57c0c78a0 0x7fe57c143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe55c0762d0 0x7fe55c0787e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 --2- 172.21.15.46:0/3835015141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe57c072500 0x7fe57c1f67a0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 >> 172.21.15.46:0/3835015141 conn(0x7fe57c0705c0 msgr2=0x7fe57c0d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 shutdown_connections 2024-05-09T03:19:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.367+0000 7fe580b57640 1 -- 172.21.15.46:0/3835015141 wait complete. 2024-05-09T03:19:01.379 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:19:01.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:01.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:01.405 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:19:01.405 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:01.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:01.405 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:01.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.618+0000 7f5165d2d640 1 Processor -- start 2024-05-09T03:19:01.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.618+0000 7f5165d2d640 1 -- start start 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f5165d2d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51600d3730 0x7f516006e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f5165d2d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 0x7f51600d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f5165d2d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516009d470 con 0x7f516006e7e0 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f5165d2d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f516009cfe0 con 0x7f51600d3730 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f5165d2d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f51600b94a0 con 0x7f5160165090 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f515ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 0x7f51600d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f515f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51600d3730 0x7f516006e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f515effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f516006e7e0 0x7f5160164980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:51300/0 (socket says 172.21.15.46:51300) 2024-05-09T03:19:01.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f515effd640 1 -- 172.21.15.46:0/3592180585 learned_addr learned my addr 172.21.15.46:0/3592180585 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:01.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.619+0000 7f515ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 0x7f51600d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:37126/0 (socket says 172.21.15.46:37126) 2024-05-09T03:19:01.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.620+0000 7f515ffff640 1 -- 172.21.15.46:0/3592180585 >> v2:172.21.15.88:3300/0 conn(0x7f51600d3730 msgr2=0x7f516006e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.620+0000 7f515ffff640 1 --2- 172.21.15.46:0/3592180585 >> v2:172.21.15.88:3300/0 conn(0x7f51600d3730 0x7f516006e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.620+0000 7f515ffff640 1 -- 172.21.15.46:0/3592180585 >> v1:172.21.15.88:6789/0 conn(0x7f516006e7e0 legacy=0x7f5160164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.620+0000 7f515ffff640 1 -- 172.21.15.46:0/3592180585 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51600d11b0 con 0x7f5160165090 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.620+0000 7f515f7fe640 1 --2- 172.21.15.46:0/3592180585 >> v2:172.21.15.88:3300/0 conn(0x7f51600d3730 0x7f516006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.621+0000 7f515ffff640 1 --2- 172.21.15.46:0/3592180585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 0x7f51600d1f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5158009d90 tx=0x7f515802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af0ab9d01631e6dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.621+0000 7f515e7fc640 1 -- 172.21.15.46:0/3592180585 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158031600 con 0x7f5160165090 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.621+0000 7f515e7fc640 1 -- 172.21.15.46:0/3592180585 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51580317a0 con 0x7f5160165090 2024-05-09T03:19:01.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.621+0000 7f515e7fc640 1 -- 172.21.15.46:0/3592180585 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158031b70 con 0x7f5160165090 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.621+0000 7f5165d2d640 1 -- 172.21.15.46:0/3592180585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 msgr2=0x7f51600d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.621+0000 7f5165d2d640 1 --2- 172.21.15.46:0/3592180585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 0x7f51600d1f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5158009d90 tx=0x7f515802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.622+0000 7f5165d2d640 1 -- 172.21.15.46:0/3592180585 shutdown_connections 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.622+0000 7f5165d2d640 1 --2- 172.21.15.46:0/3592180585 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5160165090 0x7f51600d1f50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.622+0000 7f5165d2d640 1 --2- 172.21.15.46:0/3592180585 >> v2:172.21.15.88:3300/0 conn(0x7f51600d3730 0x7f516006e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.622+0000 7f5165d2d640 1 -- 172.21.15.46:0/3592180585 >> 172.21.15.46:0/3592180585 conn(0x7f51600d05f0 msgr2=0x7f51600d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.622+0000 7f5165d2d640 1 -- 172.21.15.46:0/3592180585 shutdown_connections 2024-05-09T03:19:01.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.622+0000 7f5165d2d640 1 -- 172.21.15.46:0/3592180585 wait complete. 2024-05-09T03:19:01.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.623+0000 7f5165d2d640 1 Processor -- start 2024-05-09T03:19:01.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.624+0000 7f5165d2d640 1 -- start start 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f5165d2d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f516006e7e0 0x7f51600ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f5165d2d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600d3730 0x7f51600c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f5165d2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 0x7f51600ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f5165d2d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f51600cfbb0 con 0x7f5160165090 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f5165d2d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f51600b9160 con 0x7f51600d3730 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f5165d2d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f51600cfd30 con 0x7f516006e7e0 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 0x7f51600ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 0x7f51600ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59356/0 (socket says 172.21.15.46:59356) 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600d3730 0x7f51600c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 -- 172.21.15.46:0/2720486768 learned_addr learned my addr 172.21.15.46:0/2720486768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515f7fe640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f516006e7e0 0x7f51600ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 -- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f516006e7e0 msgr2=0x7f51600ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f516006e7e0 0x7f51600ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 -- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600d3730 msgr2=0x7f51600c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600d3730 0x7f51600c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515ffff640 1 -- 172.21.15.46:0/2720486768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51600ca760 con 0x7f5160165090 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.625+0000 7f515effd640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600d3730 0x7f51600c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.626+0000 7f515ffff640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 0x7f51600ce030 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f515802abc0 tx=0x7f5158004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.626+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515803f070 con 0x7f5160165090 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.626+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5158030b60 con 0x7f5160165090 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.626+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158009450 con 0x7f5160165090 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.626+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f516015ff70 con 0x7f5160165090 2024-05-09T03:19:01.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.626+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5160160530 con 0x7f5160165090 2024-05-09T03:19:01.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.627+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516015fce0 con 0x7f5160165090 2024-05-09T03:19:01.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.628+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f515803b410 con 0x7f5160165090 2024-05-09T03:19:01.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.628+0000 7f5147fff640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f514c076300 0x7f514c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:01.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.628+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8003+0+0 (secure 0 0 0) 0x7f51580b9400 con 0x7f5160165090 2024-05-09T03:19:01.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.629+0000 7f515f7fe640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f514c076300 0x7f514c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:01.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.629+0000 7f515f7fe640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f514c076300 0x7f514c078810 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5150009560 tx=0x7f5150009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:01.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.633+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5158081d80 con 0x7f5160165090 2024-05-09T03:19:01.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.861+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f51600c7450 con 0x7f5160165090 2024-05-09T03:19:01.862 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:01.861+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.863+0000 7f5147fff640 1 -- 172.21.15.46:0/2720486768 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v253) ==== 161+0+0 (secure 0 0 0) 0x7f51600c7450 con 0x7f5160165090 2024-05-09T03:19:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f514c076300 msgr2=0x7f514c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f514c076300 0x7f514c078810 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5150009560 tx=0x7f5150009190 comp rx=0 tx=0).stop 2024-05-09T03:19:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 msgr2=0x7f51600ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 0x7f51600ce030 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f515802abc0 tx=0x7f5158004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 shutdown_connections 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f514c076300 0x7f514c078810 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5160165090 0x7f51600ce030 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51600d3730 0x7f51600c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 --2- 172.21.15.46:0/2720486768 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f516006e7e0 0x7f51600ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 >> 172.21.15.46:0/2720486768 conn(0x7f51600d05f0 msgr2=0x7f51600b6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 shutdown_connections 2024-05-09T03:19:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.866+0000 7f5165d2d640 1 -- 172.21.15.46:0/2720486768 wait complete. 2024-05-09T03:19:01.868 INFO:tasks.ceph.mds.c.smithi046.stderr: -583> 2024-05-09T03:18:59.116+0000 7f7c0e3e5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:19:01.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:01.925 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.998+0000 7f9f41a13640 1 Processor -- start 2024-05-09T03:19:01.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.998+0000 7f9f41a13640 1 -- start start 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.999+0000 7f9f41a13640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f3c0b5140 0x7f9f3c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.999+0000 7f9f41a13640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f3c0b80b0 con 0x7f9f3c0b7ca0 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.999+0000 7f9f41a13640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f3c0b5550 con 0x7f9f3c0d3fc0 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:01.999+0000 7f9f41a13640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9f3c0d3480 con 0x7f9f3c0b5140 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f3a7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9f3c0b7ca0 0x7f9f3c0d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35448/0 (socket says 172.21.15.46:35448) 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f3a7fc640 1 -- 172.21.15.46:0/679504212 learned_addr learned my addr 172.21.15.46:0/679504212 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f3affd640 1 --2- 172.21.15.46:0/679504212 >> v2:172.21.15.88:3300/0 conn(0x7f9f3c0b5140 0x7f9f3c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155605305 0 0) 0x7f9f30001110 con 0x7f9f3c0b7ca0 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f24003520 con 0x7f9f3c0b7ca0 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 446237511 0 0) 0x7f9f24003520 con 0x7f9f3c0b7ca0 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 >> v2:172.21.15.88:3300/0 conn(0x7f9f3c0b5140 msgr2=0x7f9f3c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 --2- 172.21.15.46:0/679504212 >> v2:172.21.15.88:3300/0 conn(0x7f9f3c0b5140 0x7f9f3c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 >> v1:172.21.15.88:6789/0 conn(0x7f9f3c0d3fc0 legacy=0x7f9f3c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3c0d44e0 con 0x7f9f3c0b7ca0 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f9f300018c0 con 0x7f9f3c0b7ca0 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 >> v1:172.21.15.46:6789/0 conn(0x7f9f3c0b7ca0 legacy=0x7f9f3c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f24003a50 0x7f9f24003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f24004800 0x7f9f24004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2400c1a0 con 0x7f9f24005250 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2400c350 con 0x7f9f24003a50 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9f2400c550 con 0x7f9f24004800 2024-05-09T03:19:02.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f3b7fe640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f3affd640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f24003a50 0x7f9f24003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.000+0000 7f9f3a7fc640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f24004800 0x7f9f24004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 -- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f24004800 msgr2=0x7f9f24004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f24004800 0x7f9f24004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 -- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f24003a50 msgr2=0x7f9f24003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f24003a50 0x7f9f24003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 -- 172.21.15.46:0/679504212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f24003520 con 0x7f9f24005250 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3affd640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f24003a50 0x7f9f24003ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 -- 172.21.15.46:0/679504212 reap_dead start 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.001+0000 7f9f3b7fe640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f2400b9c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f9f34001e90 tx=0x7f9f24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d968c9a177a215a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c009690 con 0x7f9f24005250 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3402ae50 con 0x7f9f24005250 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f39ffb640 1 -- 172.21.15.46:0/679504212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34001ab0 con 0x7f9f24005250 2024-05-09T03:19:02.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 -- 172.21.15.46:0/679504212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 msgr2=0x7f9f2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f2400b9c0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f9f34001e90 tx=0x7f9f24005830 comp rx=0 tx=0).stop 2024-05-09T03:19:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 -- 172.21.15.46:0/679504212 shutdown_connections 2024-05-09T03:19:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 --2- 172.21.15.46:0/679504212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f2400b9c0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 -- 172.21.15.46:0/679504212 >> 172.21.15.46:0/679504212 conn(0x7f9f3c08d3f0 msgr2=0x7f9f3c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 -- 172.21.15.46:0/679504212 shutdown_connections 2024-05-09T03:19:02.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.002+0000 7f9f41a13640 1 -- 172.21.15.46:0/679504212 wait complete. 2024-05-09T03:19:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.004+0000 7f9f41a13640 1 Processor -- start 2024-05-09T03:19:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.004+0000 7f9f41a13640 1 -- start start 2024-05-09T03:19:02.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f41a13640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f3c1f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f41a13640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f3c1f6bd0 0x7f9f3c15e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f41a13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c1f9710 0x7f9f3c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f41a13640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3c0d4d80 con 0x7f9f24005250 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f41a13640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3c0d4aa0 con 0x7f9f3c1f9710 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f41a13640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3c0d5580 con 0x7f9f3c1f6bd0 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f3c1f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f3c1f6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59388/0 (socket says 172.21.15.46:59388) 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 -- 172.21.15.46:0/2712519999 learned_addr learned my addr 172.21.15.46:0/2712519999 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3a7fc640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f3c1f6bd0 0x7f9f3c15e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3b7fe640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c1f9710 0x7f9f3c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 -- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f3c1f6bd0 msgr2=0x7f9f3c15e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f3c1f6bd0 0x7f9f3c15e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 -- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c1f9710 msgr2=0x7f9f3c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c1f9710 0x7f9f3c15ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 -- 172.21.15.46:0/2712519999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3c1556e0 con 0x7f9f24005250 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3b7fe640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c1f9710 0x7f9f3c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3a7fc640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f3c1f6bd0 0x7f9f3c15e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f3affd640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f3c1f6690 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c011930 tx=0x7f9f2c011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.006+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c00fa90 con 0x7f9f24005250 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.007+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2c00fc30 con 0x7f9f24005250 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.007+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c014440 con 0x7f9f24005250 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.007+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3c1f7300 con 0x7f9f24005250 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.007+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3c1590d0 con 0x7f9f24005250 2024-05-09T03:19:02.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.008+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f3c06a270 con 0x7f9f24005250 2024-05-09T03:19:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.008+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9f2c012040 con 0x7f9f24005250 2024-05-09T03:19:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.009+0000 7f9f23fff640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f28076300 0x7f9f28078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.009+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8089+0+0 (secure 0 0 0) 0x7f9f2c0985a0 con 0x7f9f24005250 2024-05-09T03:19:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.009+0000 7f9f3a7fc640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f28076300 0x7f9f28078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.010+0000 7f9f3a7fc640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f28076300 0x7f9f28078810 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0d0800 tx=0x7f9f30009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.014+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9f2c019030 con 0x7f9f24005250 2024-05-09T03:19:02.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.279+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f9f3c1f7820 con 0x7f9f24005250 2024-05-09T03:19:02.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.864+0000 7f9f23fff640 1 -- 172.21.15.46:0/2712519999 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v254) ==== 107+0+0 (secure 0 0 0) 0x7f9f2c061520 con 0x7f9f24005250 2024-05-09T03:19:02.865 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f28076300 msgr2=0x7f9f28078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f28076300 0x7f9f28078810 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0d0800 tx=0x7f9f30009040 comp rx=0 tx=0).stop 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 msgr2=0x7f9f3c1f6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f3c1f6690 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c011930 tx=0x7f9f2c011df0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 shutdown_connections 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f3c1f9710 0x7f9f3c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9f3c1f6bd0 0x7f9f3c15e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9f28076300 0x7f9f28078810 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 --2- 172.21.15.46:0/2712519999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f24005250 0x7f9f3c1f6690 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 >> 172.21.15.46:0/2712519999 conn(0x7f9f3c08d3f0 msgr2=0x7f9f3c162290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 shutdown_connections 2024-05-09T03:19:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.868+0000 7f9f41a13640 1 -- 172.21.15.46:0/2712519999 wait complete. 2024-05-09T03:19:02.882 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:02.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.940+0000 7ff3ce2b1f40 1 Processor -- start 2024-05-09T03:19:02.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.940+0000 7ff3ce2b1f40 1 -- start start 2024-05-09T03:19:02.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3ce2b1f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d10d57e0 0x5629d111c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3ce2b1f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d1104db0 0x5629d11a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3ce2b1f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 0x5629d1101a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3ce2b1f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5629d1117cb0 con 0x5629d11ad780 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3ce2b1f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5629d107ef50 con 0x5629d1104db0 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3ce2b1f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5629d10d5bf0 con 0x5629d10d57e0 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3c857a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 0x5629d1101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3c857a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 0x5629d1101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59392/0 (socket says 172.21.15.46:59392) 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.941+0000 7ff3c857a640 1 -- 172.21.15.46:0/260246337 learned_addr learned my addr 172.21.15.46:0/260246337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c7578640 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d1104db0 0x5629d11a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d10d57e0 0x5629d111c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c857a640 1 -- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d10d57e0 msgr2=0x5629d111c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c857a640 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d10d57e0 0x5629d111c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c857a640 1 -- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d1104db0 msgr2=0x5629d11a6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c857a640 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d1104db0 0x5629d11a6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c857a640 1 -- 172.21.15.46:0/260246337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629d1107cf0 con 0x5629d11ad780 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c857a640 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 0x5629d1101a20 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0001c40 tx=0x7ff3c000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7403c64b79646c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c6576640 1 -- 172.21.15.46:0/260246337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000bcc0 con 0x5629d11ad780 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c6576640 1 -- 172.21.15.46:0/260246337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c0001e50 con 0x5629d11ad780 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.942+0000 7ff3c6576640 1 -- 172.21.15.46:0/260246337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000f640 con 0x5629d11ad780 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/260246337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 msgr2=0x5629d1101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 0x5629d1101a20 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0001c40 tx=0x7ff3c000b5b0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/260246337 shutdown_connections 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d11ad780 0x5629d1101a20 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d1104db0 0x5629d11a6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/260246337 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d10d57e0 0x5629d111c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/260246337 >> 172.21.15.46:0/260246337 conn(0x5629d0ffaac0 msgr2=0x5629d1103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/260246337 shutdown_connections 2024-05-09T03:19:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.943+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/260246337 wait complete. 2024-05-09T03:19:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.944+0000 7ff3ce2b1f40 1 Processor -- start 2024-05-09T03:19:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.944+0000 7ff3ce2b1f40 1 -- start start 2024-05-09T03:19:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3ce2b1f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 0x5629d11ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3ce2b1f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d1104db0 0x5629d118f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3ce2b1f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d11ad780 0x5629d11917f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3ce2b1f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5629d1191070 con 0x5629d10d57e0 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3ce2b1f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5629d111c4b0 con 0x5629d11ad780 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3ce2b1f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5629d11912c0 con 0x5629d1104db0 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 0x5629d11ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 0x5629d11ab340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59396/0 (socket says 172.21.15.46:59396) 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 learned_addr learned my addr 172.21.15.46:0/1686483902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d1104db0 0x5629d118f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c857a640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d11ad780 0x5629d11917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d1104db0 msgr2=0x5629d118f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5629d1104db0 0x5629d118f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d11ad780 msgr2=0x5629d11917f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5629d11ad780 0x5629d11917f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.945+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629d11a9940 con 0x5629d10d57e0 2024-05-09T03:19:02.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.946+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 0x5629d11ab340 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8008240 tx=0x7ff3b8008700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.946+0000 7ff3c5574640 1 -- 172.21.15.46:0/1686483902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8016030 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.946+0000 7ff3c5574640 1 -- 172.21.15.46:0/1686483902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3b8012410 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.946+0000 7ff3c5574640 1 -- 172.21.15.46:0/1686483902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800e040 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.946+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5629d11ade70 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.946+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629d11b0df0 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.948+0000 7ff3c5574640 1 -- 172.21.15.46:0/1686483902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff3b8019070 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.948+0000 7ff3c5574640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3a8074ce0 0x7ff3a80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.948+0000 7ff3c5574640 1 -- 172.21.15.46:0/1686483902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8089+0+0 (secure 0 0 0) 0x7ff3b8096780 con 0x5629d10d57e0 2024-05-09T03:19:02.988 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.948+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5629d11029b0 0x5629d1247680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.948+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d118f9a0 con 0x5629d11029b0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.948+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3a8074ce0 0x7ff3a80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.949+0000 7ff3c857a640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5629d11029b0 0x5629d1247680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.949+0000 7ff3c857a640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5629d11029b0 0x5629d1247680 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.949+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3a8074ce0 0x7ff3a80771f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0057000 tx=0x7ff3b0052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.953+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v111'815 uv789 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.953+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1191d30 con 0x5629d11029b0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.955+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v111'816 uv816 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.955+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d11a6190 con 0x5629d11029b0 2024-05-09T03:19:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.957+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v111'817 uv817 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.991 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.957+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d11a9b10 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.958+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v111'818 uv818 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.958+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250fa0 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.960+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v111'819 uv819 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.960+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d124ff90 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.961+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v111'820 uv820 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.961+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250330 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.963+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v111'821 uv795 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.963+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.965+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v111'822 uv822 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.965+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.993 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:02.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.966+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v111'823 uv823 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.966+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.968+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v111'824 uv824 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.968+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.969+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v111'825 uv825 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.969+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.971+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v111'826 uv826 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.971+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.972+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v111'827 uv802 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.972+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.974+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v111'828 uv828 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.974+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.975+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v111'829 uv829 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.996 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:02.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.975+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.977+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v111'830 uv830 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.977+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.978+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v111'831 uv831 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.978+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12506d0 con 0x5629d11029b0 2024-05-09T03:19:02.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.980+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v111'832 uv832 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.980+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.981+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v111'833 uv833 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.981+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.983+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v111'834 uv808 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.983+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.984+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v111'835 uv835 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.984+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.985+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v111'836 uv836 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:02.998 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.986+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.987+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v111'837 uv837 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.987+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.988+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v111'838 uv838 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.988+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.990+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v111'839 uv839 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.990+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.991+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v111'840 uv814 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.991+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.993+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v111'841 uv841 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.993+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.994+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v111'842 uv842 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.001 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.994+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.996+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v111'843 uv843 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.996+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.997+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v111'844 uv844 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.997+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d11029b0 2024-05-09T03:19:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.999+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v111'845 uv845 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.999+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5629d1251850 0x5629d1253c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.999+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1250a90 con 0x5629d1251850 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.999+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5629d1251850 0x5629d1253c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:02.999+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5629d1251850 0x5629d1253c70 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.001+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v111'591 uv571 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.001+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d12541b0 con 0x5629d1251850 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.003+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v111'592 uv592 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.003+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1254550 con 0x5629d1251850 2024-05-09T03:19:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.004+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v111'593 uv593 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.005 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:03.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.004+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1254da0 con 0x5629d1251850 2024-05-09T03:19:03.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.006+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v111'594 uv594 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.006+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255140 con 0x5629d1251850 2024-05-09T03:19:03.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.008+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v111'595 uv595 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.008+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12554e0 con 0x5629d1251850 2024-05-09T03:19:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.009+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v111'596 uv590 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.009+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.010+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v111'597 uv597 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.010+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.011+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v111'598 uv598 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.012 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.011+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.013+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v111'599 uv599 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.013+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.014+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v111'600 uv600 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.014+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.015+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v111'601 uv589 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.015+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.016+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v111'602 uv602 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.016+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.017+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v111'603 uv603 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.018 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:03.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.017+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12592e0 con 0x5629d1251850 2024-05-09T03:19:03.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.023+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v111'604 uv604 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.023+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12592e0 con 0x5629d1251850 2024-05-09T03:19:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.025+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v111'605 uv605 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.025+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.026+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v111'606 uv587 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.026+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.027+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v111'607 uv607 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.027+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.028+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v111'608 uv608 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.029 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:03.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.028+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1256890 con 0x5629d1251850 2024-05-09T03:19:03.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.029+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v111'609 uv609 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.029+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1256890 con 0x5629d1251850 2024-05-09T03:19:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.030+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v111'610 uv610 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.030+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d1251850 2024-05-09T03:19:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.031+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v111'611 uv611 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d1251850 2024-05-09T03:19:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.031+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d11029b0 2024-05-09T03:19:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.033+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v111'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.033+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d11029b0 2024-05-09T03:19:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.034+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v111'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.034+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d11029b0 2024-05-09T03:19:03.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.035+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v111'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.035 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.035+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1256890 con 0x5629d11029b0 2024-05-09T03:19:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.036+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v111'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.036+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1256890 con 0x5629d11029b0 2024-05-09T03:19:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.037+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v111'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.037+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1255880 con 0x5629d11029b0 2024-05-09T03:19:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.038+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v111'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.038+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5629d1255d90 0x5629d12581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.038+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1258810 con 0x5629d1255d90 2024-05-09T03:19:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.038+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5629d1255d90 0x5629d12581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.038+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5629d1255d90 0x5629d12581b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.041+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v111'120 uv119 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.041+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1258d10 con 0x5629d1255d90 2024-05-09T03:19:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.042+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v111'121 uv121 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.043+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259210 con 0x5629d1255d90 2024-05-09T03:19:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.043+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v111'122 uv122 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.044 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:03.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.043+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12549d0 con 0x5629d1255d90 2024-05-09T03:19:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.044+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v111'123 uv123 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.044+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259b40 con 0x5629d11029b0 2024-05-09T03:19:03.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.046+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v111'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.046+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1259b40 con 0x5629d11029b0 2024-05-09T03:19:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.048+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v111'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.048+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259b40 con 0x5629d11029b0 2024-05-09T03:19:03.049 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.049+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v111'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.049+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d125ab50 con 0x5629d11029b0 2024-05-09T03:19:03.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.050+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v111'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.050+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d125ab50 con 0x5629d11029b0 2024-05-09T03:19:03.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.051+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v111'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.051+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259b40 con 0x5629d11029b0 2024-05-09T03:19:03.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.053+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v111'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.053+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5629d125a1c0 0x5629d125a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.053+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d125cfc0 con 0x5629d125a1c0 2024-05-09T03:19:03.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.053+0000 7ff3c857a640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5629d125a1c0 0x5629d125a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.053+0000 7ff3c857a640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5629d125a1c0 0x5629d125a5d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.055+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v111'294 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.055+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d125cfc0 con 0x5629d125a1c0 2024-05-09T03:19:03.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.056+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v111'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.056+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d125cfc0 con 0x5629d125a1c0 2024-05-09T03:19:03.058 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:03.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.057+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v111'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.057+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12596d0 con 0x5629d125a1c0 2024-05-09T03:19:03.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.058+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v111'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.058+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259690 con 0x5629d125a1c0 2024-05-09T03:19:03.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.060+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v111'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.060+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259690 con 0x5629d125a1c0 2024-05-09T03:19:03.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.061+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v111'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.061+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259670 con 0x5629d125a1c0 2024-05-09T03:19:03.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.062+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v111'300 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.062+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1259670 con 0x5629d125a1c0 2024-05-09T03:19:03.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.063+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v111'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.063+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259670 con 0x5629d125a1c0 2024-05-09T03:19:03.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.064+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v111'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.065 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:03.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.064+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259670 con 0x5629d125a1c0 2024-05-09T03:19:03.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.065+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v111'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.066+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1259670 con 0x5629d125a1c0 2024-05-09T03:19:03.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.067+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v111'304 uv304 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.067+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5629d125d980 0x5629d125fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.067+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12602e0 con 0x5629d125d980 2024-05-09T03:19:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.067+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5629d125d980 0x5629d125fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.067+0000 7ff3c7d79640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5629d125d980 0x5629d125fda0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.069+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v111'132 uv131 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d125d980 2024-05-09T03:19:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.069+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d125d980 2024-05-09T03:19:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.070+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v111'133 uv133 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d125d980 2024-05-09T03:19:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.070+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d125d980 2024-05-09T03:19:03.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.071+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v111'134 uv134 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d125d980 2024-05-09T03:19:03.072 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:03.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.071+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d125d980 2024-05-09T03:19:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.072+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v111'135 uv135 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d125d980 2024-05-09T03:19:03.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.073+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d125d980 2024-05-09T03:19:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.074+0000 7ff3c7d79640 1 -- 172.21.15.46:0/1686483902 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v111'136 uv136 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b805b3b0 con 0x5629d125d980 2024-05-09T03:19:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.074+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d11029b0 2024-05-09T03:19:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.075+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v111'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.075+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d11029b0 2024-05-09T03:19:03.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.076+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v111'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.076+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d11029b0 2024-05-09T03:19:03.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.077+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v111'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.078 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:03.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.077+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12616c0 con 0x5629d11029b0 2024-05-09T03:19:03.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.078+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v111'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.078+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12616c0 con 0x5629d11029b0 2024-05-09T03:19:03.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.079+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v111'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.079+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d11029b0 2024-05-09T03:19:03.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.080+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v111'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d11029b0 2024-05-09T03:19:03.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.080+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12606b0 con 0x5629d1255d90 2024-05-09T03:19:03.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.081+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v111'263 uv257 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.081+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1260a70 con 0x5629d1255d90 2024-05-09T03:19:03.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.082+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v111'264 uv264 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.082+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1260e60 con 0x5629d1255d90 2024-05-09T03:19:03.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.083+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v111'265 uv265 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.084 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:03.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.083+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1261250 con 0x5629d1255d90 2024-05-09T03:19:03.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.084+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v111'266 uv266 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.084+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1261250 con 0x5629d1255d90 2024-05-09T03:19:03.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.085+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v111'267 uv267 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.085+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d12616b0 con 0x5629d1255d90 2024-05-09T03:19:03.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.086+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v111'268 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.086+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1261250 con 0x5629d1255d90 2024-05-09T03:19:03.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.087+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v111'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.087+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1261250 con 0x5629d1255d90 2024-05-09T03:19:03.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.088+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v111'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.089 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:03.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.088+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1261200 con 0x5629d1255d90 2024-05-09T03:19:03.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.089+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v111'271 uv271 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.089+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264470 con 0x5629d1255d90 2024-05-09T03:19:03.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.090+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v111'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.090+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5629d1262150 0x5629d1264570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.090+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264b30 con 0x5629d1262150 2024-05-09T03:19:03.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.090+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5629d1262150 0x5629d1264570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.091+0000 7ff3c7578640 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5629d1262150 0x5629d1264570 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.092+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v111'116 uv114 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1262150 2024-05-09T03:19:03.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.092+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1264b30 con 0x5629d1262150 2024-05-09T03:19:03.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.093+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v111'117 uv117 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1262150 2024-05-09T03:19:03.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.093+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264b30 con 0x5629d1262150 2024-05-09T03:19:03.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.094+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v111'118 uv118 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1262150 2024-05-09T03:19:03.095 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:03.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.094+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d1262150 2024-05-09T03:19:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.095+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v111'119 uv119 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1262150 2024-05-09T03:19:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.095+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d1255d90 2024-05-09T03:19:03.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.097+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v111'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.097+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d1255d90 2024-05-09T03:19:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.098+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v111'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.098+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d1255d90 2024-05-09T03:19:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.099+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v111'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.100 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.099+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1265f30 con 0x5629d1255d90 2024-05-09T03:19:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.100+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v111'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.100+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1265f30 con 0x5629d1255d90 2024-05-09T03:19:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.102+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v111'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.102+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d1255d90 2024-05-09T03:19:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.103+0000 7ff3c7578640 1 -- 172.21.15.46:0/1686483902 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v111'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3b0060430 con 0x5629d1255d90 2024-05-09T03:19:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.103+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.105+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v111'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.105+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.106+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v111'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.106+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.107 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.107+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v111'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.107+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.108+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v111'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.108+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.109+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v111'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.109+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.111+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v111'327 uv318 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.111+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.113+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v111'328 uv328 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.113+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.114+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v111'329 uv329 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.115 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:03.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.114+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.115+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v111'330 uv330 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.115+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.116+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v111'331 uv331 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.116+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.117+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v111'332 uv325 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.117+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.118+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v111'333 uv333 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.118+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.119+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v111'334 uv334 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.120 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:03.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.119+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1265f30 con 0x5629d125a1c0 2024-05-09T03:19:03.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.120+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v111'335 uv335 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.120+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1265f30 con 0x5629d125a1c0 2024-05-09T03:19:03.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.121+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v111'336 uv336 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.121+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1264f20 con 0x5629d125a1c0 2024-05-09T03:19:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.122+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v111'337 uv337 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.122+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5629d1265640 con 0x5629d125a1c0 2024-05-09T03:19:03.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v111'338 uv338 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff3c001d450 con 0x5629d125a1c0 2024-05-09T03:19:03.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5629d11029b0 msgr2=0x5629d1247680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5629d11029b0 0x5629d1247680 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5629d1255d90 msgr2=0x5629d12581b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5629d1255d90 0x5629d12581b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5629d1262150 msgr2=0x5629d1264570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x5629d1262150 0x5629d1264570 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5629d1251850 msgr2=0x5629d1253c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x5629d1251850 0x5629d1253c70 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3c857a640 1 -- 172.21.15.46:0/1686483902 reap_dead start 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5629d125a1c0 msgr2=0x5629d125a5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5629d125a1c0 0x5629d125a5d0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5629d125d980 msgr2=0x5629d125fda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.123+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5629d125d980 0x5629d125fda0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3a8074ce0 msgr2=0x7ff3a80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3a8074ce0 0x7ff3a80771f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0057000 tx=0x7ff3b0052000 comp rx=0 tx=0).stop 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 msgr2=0x5629d11ab340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 0x5629d11ab340 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8008240 tx=0x7ff3b8008700 comp rx=0 tx=0).stop 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 shutdown_connections 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x5629d125d980 0x5629d125fda0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5629d125a1c0 0x5629d125a5d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3a8074ce0 0x7ff3a80771f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 --2- 172.21.15.46:0/1686483902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629d10d57e0 0x5629d11ab340 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 >> 172.21.15.46:0/1686483902 conn(0x5629d0ffaac0 msgr2=0x5629d1103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 shutdown_connections 2024-05-09T03:19:03.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.124+0000 7ff3ce2b1f40 1 -- 172.21.15.46:0/1686483902 wait complete. 2024-05-09T03:19:03.128 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate mds_snaptable 0 2024-05-09T03:19:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.224+0000 7fc7a2ecbf40 1 Processor -- start 2024-05-09T03:19:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.224+0000 7fc7a2ecbf40 1 -- start start 2024-05-09T03:19:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc7a2ecbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 0x56513df8bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc7a2ecbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de6d010 0x56513de66180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc7a2ecbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de519a0 0x56513de666c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc7a2ecbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56513de6b4d0 con 0x56513de567b0 2024-05-09T03:19:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc7a2ecbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56513ddcdf50 con 0x56513de6d010 2024-05-09T03:19:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc7a2ecbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56513de708a0 con 0x56513de519a0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.225+0000 7fc79c993640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 0x56513df8bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 0x56513df8bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59404/0 (socket says 172.21.15.46:59404) 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 -- 172.21.15.46:0/681790711 learned_addr learned my addr 172.21.15.46:0/681790711 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79d194640 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de519a0 0x56513de666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c192640 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de6d010 0x56513de66180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 -- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de519a0 msgr2=0x56513de666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de519a0 0x56513de666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 -- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de6d010 msgr2=0x56513de66180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de6d010 0x56513de66180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 -- 172.21.15.46:0/681790711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56513defcb10 con 0x56513de567b0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79c993640 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 0x56513df8bff0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b750 tx=0x7fc78c00bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b3c51a17d717bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79b190640 1 -- 172.21.15.46:0/681790711 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c016020 con 0x56513de567b0 2024-05-09T03:19:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79b190640 1 -- 172.21.15.46:0/681790711 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc78c00be60 con 0x56513de567b0 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.226+0000 7fc79b190640 1 -- 172.21.15.46:0/681790711 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c00dba0 con 0x56513de567b0 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/681790711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 msgr2=0x56513df8bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 0x56513df8bff0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b750 tx=0x7fc78c00bc10 comp rx=0 tx=0).stop 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/681790711 shutdown_connections 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de519a0 0x56513de666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de6d010 0x56513de66180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/681790711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de567b0 0x56513df8bff0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/681790711 >> 172.21.15.46:0/681790711 conn(0x56513dd49ac0 msgr2=0x56513de51080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/681790711 shutdown_connections 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.227+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/681790711 wait complete. 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.228+0000 7fc7a2ecbf40 1 Processor -- start 2024-05-09T03:19:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.228+0000 7fc7a2ecbf40 1 -- start start 2024-05-09T03:19:03.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc7a2ecbf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de519a0 0x56513def3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc7a2ecbf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de567b0 0x56513df8be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc7a2ecbf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de6d010 0x56513df8c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc7a2ecbf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56513def7fa0 con 0x56513de6d010 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc7a2ecbf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56513de507a0 con 0x56513de519a0 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc7a2ecbf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56513defc310 con 0x56513de567b0 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc79d194640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de6d010 0x56513df8c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc79d194640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de6d010 0x56513df8c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59414/0 (socket says 172.21.15.46:59414) 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc79d194640 1 -- 172.21.15.46:0/2885198279 learned_addr learned my addr 172.21.15.46:0/2885198279 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc79c993640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de519a0 0x56513def3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.229+0000 7fc79c192640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de567b0 0x56513df8be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79d194640 1 -- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de567b0 msgr2=0x56513df8be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79d194640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56513de567b0 0x56513df8be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79d194640 1 -- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de519a0 msgr2=0x56513def3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79d194640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de519a0 0x56513def3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79d194640 1 -- 172.21.15.46:0/2885198279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56513defb030 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79c993640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56513de519a0 0x56513def3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79d194640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de6d010 0x56513df8c340 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fc794025be0 tx=0x7fc794002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79a18e640 1 -- 172.21.15.46:0/2885198279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794063030 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79a18e640 1 -- 172.21.15.46:0/2885198279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7940547f0 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56513defb2a0 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc79a18e640 1 -- 172.21.15.46:0/2885198279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc794059040 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.230+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56513def8560 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.232+0000 7fc79a18e640 1 -- 172.21.15.46:0/2885198279 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc794066030 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.232+0000 7fc79a18e640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc77c074cb0 0x7fc77c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.232+0000 7fc79a18e640 1 -- 172.21.15.46:0/2885198279 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8089+0+0 (secure 0 0 0) 0x7fc7940da0c0 con 0x56513de6d010 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.232+0000 7fc79c993640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc77c074cb0 0x7fc77c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.233+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56513de69ef0 0x56513de6a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.233+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x56513de6a840 con 0x56513de69ef0 2024-05-09T03:19:03.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.233+0000 7fc79c192640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56513de69ef0 0x56513de6a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.233+0000 7fc79c993640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc77c074cb0 0x7fc77c0771c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc78c000f80 tx=0x7fc78c007030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.233+0000 7fc79c192640 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56513de69ef0 0x56513de6a300 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc79c192640 1 -- 172.21.15.46:0/2885198279 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 mds_snaptable [truncate 0] v111'137 uv137 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc78400e4e0 con 0x56513de69ef0 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56513de69ef0 msgr2=0x56513de6a300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56513de69ef0 0x56513de6a300 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc77c074cb0 msgr2=0x7fc77c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc77c074cb0 0x7fc77c0771c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc78c000f80 tx=0x7fc78c007030 comp rx=0 tx=0).stop 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de6d010 msgr2=0x56513df8c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 --2- 172.21.15.46:0/2885198279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56513de6d010 0x56513df8c340 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fc794025be0 tx=0x7fc794002fb0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc79d194640 1 -- 172.21.15.46:0/2885198279 reap_dead start 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 shutdown_connections 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 >> 172.21.15.46:0/2885198279 conn(0x56513dd49ac0 msgr2=0x56513de714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:03.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 shutdown_connections 2024-05-09T03:19:03.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.235+0000 7fc7a2ecbf40 1 -- 172.21.15.46:0/2885198279 wait complete. 2024-05-09T03:19:03.238 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:03.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.355+0000 7f9bd976c640 1 Processor -- start 2024-05-09T03:19:03.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.355+0000 7f9bd976c640 1 -- start start 2024-05-09T03:19:03.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd976c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bd40b5140 0x7f9bd40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd976c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40d3fc0 0x7f9bd40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd976c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd40b80b0 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd976c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd40d3480 con 0x7f9bd40b5140 2024-05-09T03:19:03.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd976c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd40b5550 con 0x7f9bd40d3fc0 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9bd40b7ca0 0x7f9bd40d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35468/0 (socket says 172.21.15.46:35468) 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd27fc640 1 -- 172.21.15.46:0/3394792428 learned_addr learned my addr 172.21.15.46:0/3394792428 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd2ffd640 1 --2- 172.21.15.46:0/3394792428 >> v2:172.21.15.88:3300/0 conn(0x7f9bd40b5140 0x7f9bd40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd37fe640 1 --2- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40d3fc0 0x7f9bd40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd1ffb640 1 -- 172.21.15.46:0/3394792428 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2210266603 0 0) 0x7f9bc8001110 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd1ffb640 1 -- 172.21.15.46:0/3394792428 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bbc003520 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd2ffd640 1 -- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40d3fc0 msgr2=0x7f9bd40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd2ffd640 1 --2- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40d3fc0 0x7f9bd40b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd2ffd640 1 -- 172.21.15.46:0/3394792428 >> v1:172.21.15.46:6789/0 conn(0x7f9bd40b7ca0 legacy=0x7f9bd40d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.357+0000 7f9bd2ffd640 1 -- 172.21.15.46:0/3394792428 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd40d44e0 con 0x7f9bd40b5140 2024-05-09T03:19:03.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.358+0000 7f9bd2ffd640 1 --2- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40b5140 0x7f9bd40b7760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009d90 tx=0x7f9bc402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c93627e2e29529 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.358+0000 7f9bd1ffb640 1 -- 172.21.15.46:0/3394792428 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4007740 con 0x7f9bd40b5140 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.358+0000 7f9bd1ffb640 1 -- 172.21.15.46:0/3394792428 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc40078e0 con 0x7f9bd40b5140 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.358+0000 7f9bd1ffb640 1 -- 172.21.15.46:0/3394792428 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4007c90 con 0x7f9bd40b5140 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 -- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40b5140 msgr2=0x7f9bd40b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 --2- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40b5140 0x7f9bd40b7760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009d90 tx=0x7f9bc402a6f0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 -- 172.21.15.46:0/3394792428 shutdown_connections 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 --2- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40d3fc0 0x7f9bd40b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 --2- 172.21.15.46:0/3394792428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40b5140 0x7f9bd40b7760 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 -- 172.21.15.46:0/3394792428 >> 172.21.15.46:0/3394792428 conn(0x7f9bd408d3f0 msgr2=0x7f9bd408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 -- 172.21.15.46:0/3394792428 shutdown_connections 2024-05-09T03:19:03.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.359+0000 7f9bd976c640 1 -- 172.21.15.46:0/3394792428 wait complete. 2024-05-09T03:19:03.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.361+0000 7f9bd976c640 1 Processor -- start 2024-05-09T03:19:03.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.361+0000 7f9bd976c640 1 -- start start 2024-05-09T03:19:03.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.362+0000 7f9bd976c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40b5140 0x7f9bd4154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.362+0000 7f9bd976c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 0x7f9bd415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.362+0000 7f9bd976c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40d3fc0 0x7f9bd415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.362+0000 7f9bd976c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd40d4d80 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.362+0000 7f9bd976c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd40d4aa0 con 0x7f9bd40d3fc0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.362+0000 7f9bd976c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd40d5580 con 0x7f9bd40b5140 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 0x7f9bd415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 0x7f9bd415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59420/0 (socket says 172.21.15.46:59420) 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 -- 172.21.15.46:0/2265513403 learned_addr learned my addr 172.21.15.46:0/2265513403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd37fe640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40d3fc0 0x7f9bd415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd2ffd640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40b5140 0x7f9bd4154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 -- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40b5140 msgr2=0x7f9bd4154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40b5140 0x7f9bd4154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 -- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40d3fc0 msgr2=0x7f9bd415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40d3fc0 0x7f9bd415b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 -- 172.21.15.46:0/2265513403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd415a310 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd37fe640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40d3fc0 0x7f9bd415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd27fc640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 0x7f9bd415b0a0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c730 tx=0x7f9bc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8005ab0 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc8001d30 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8005e50 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd4155d00 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.363+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd41f6f60 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.364+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd406a270 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.365+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9bc8007070 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.366+0000 7f9bbb7fe640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bc00764a0 0x7f9bc00789b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:03.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.366+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8089+0+0 (secure 0 0 0) 0x7f9bc8098420 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.366+0000 7f9bd2ffd640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bc00764a0 0x7f9bc00789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:03.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.366+0000 7f9bd2ffd640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bc00764a0 0x7f9bc00789b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4034000 tx=0x7f9bc402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:03.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.370+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9bd40b80b0 con 0x7f9bd40b7ca0 2024-05-09T03:19:03.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:03.607+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f9bd40cfa20 con 0x7f9bd40b7ca0 2024-05-09T03:19:04.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.169+0000 7f9bbb7fe640 1 -- 172.21.15.46:0/2265513403 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v255) ==== 162+0+0 (secure 0 0 0) 0x7f9bc8060e30 con 0x7f9bd40b7ca0 2024-05-09T03:19:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bc00764a0 msgr2=0x7f9bc00789b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bc00764a0 0x7f9bc00789b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4034000 tx=0x7f9bc402f000 comp rx=0 tx=0).stop 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 msgr2=0x7f9bd415b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 0x7f9bd415b0a0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f9bc800c730 tx=0x7f9bc800cbf0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 shutdown_connections 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd40d3fc0 0x7f9bd415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9bc00764a0 0x7f9bc00789b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bd40b7ca0 0x7f9bd415b0a0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 --2- 172.21.15.46:0/2265513403 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9bd40b5140 0x7f9bd4154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 >> 172.21.15.46:0/2265513403 conn(0x7f9bd408d3f0 msgr2=0x7f9bd41630d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 shutdown_connections 2024-05-09T03:19:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.172+0000 7f9bd976c640 1 -- 172.21.15.46:0/2265513403 wait complete. 2024-05-09T03:19:04.179 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:04.178+0000 7f87de8fb640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': End of buffer [buffer:2] 2024-05-09T03:19:04.180 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:04.179+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:04.180 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:04.179+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:19:04.185 INFO:tasks.ceph.mds.c.smithi046.stderr: -23> 2024-05-09T03:19:04.178+0000 7f87de8fb640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': End of buffer [buffer:2] 2024-05-09T03:19:04.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:04.246 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.312+0000 7f1f29dee640 1 Processor -- start 2024-05-09T03:19:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.312+0000 7f1f29dee640 1 -- start start 2024-05-09T03:19:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f29dee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f24164c70 0x7f1f241615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f29dee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f24160b20 con 0x7f1f2415dfe0 2024-05-09T03:19:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f29dee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f240d15f0 con 0x7f1f240d2480 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f29dee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1f24161150 con 0x7f1f24164c70 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f237fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f24164c70 0x7f1f241615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f22ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1f2415dfe0 0x7f1f240d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35484/0 (socket says 172.21.15.46:35484) 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.313+0000 7f1f22ffd640 1 -- 172.21.15.46:0/2640417518 learned_addr learned my addr 172.21.15.46:0/2640417518 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.314+0000 7f1f237fe640 1 -- 172.21.15.46:0/2640417518 >> v1:172.21.15.88:6789/0 conn(0x7f1f240d2480 legacy=0x7f1f240d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.314+0000 7f1f237fe640 1 -- 172.21.15.46:0/2640417518 >> v1:172.21.15.46:6789/0 conn(0x7f1f2415dfe0 legacy=0x7f1f240d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.314+0000 7f1f237fe640 1 -- 172.21.15.46:0/2640417518 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f240d18d0 con 0x7f1f24164c70 2024-05-09T03:19:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f237fe640 1 --2- 172.21.15.46:0/2640417518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f24164c70 0x7f1f241615e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f1f140097a0 tx=0x7f1f1402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b667ec25eb2d9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f227fc640 1 -- 172.21.15.46:0/2640417518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f14037070 con 0x7f1f24164c70 2024-05-09T03:19:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f227fc640 1 -- 172.21.15.46:0/2640417518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f14032440 con 0x7f1f24164c70 2024-05-09T03:19:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f227fc640 1 -- 172.21.15.46:0/2640417518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f14032810 con 0x7f1f24164c70 2024-05-09T03:19:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 -- 172.21.15.46:0/2640417518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f24164c70 msgr2=0x7f1f241615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 --2- 172.21.15.46:0/2640417518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f24164c70 0x7f1f241615e0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f1f140097a0 tx=0x7f1f1402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 -- 172.21.15.46:0/2640417518 shutdown_connections 2024-05-09T03:19:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 --2- 172.21.15.46:0/2640417518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f24164c70 0x7f1f241615e0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 -- 172.21.15.46:0/2640417518 >> 172.21.15.46:0/2640417518 conn(0x7f1f24073db0 msgr2=0x7f1f240741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 -- 172.21.15.46:0/2640417518 shutdown_connections 2024-05-09T03:19:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.315+0000 7f1f29dee640 1 -- 172.21.15.46:0/2640417518 wait complete. 2024-05-09T03:19:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.317+0000 7f1f29dee640 1 Processor -- start 2024-05-09T03:19:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.317+0000 7f1f29dee640 1 -- start start 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f29dee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f240d2480 0x7f1f241f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f29dee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 0x7f1f24151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f29dee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f24164c70 0x7f1f241f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f29dee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f240d45c0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f29dee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2415e3f0 con 0x7f1f240d2480 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f29dee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1f240d4740 con 0x7f1f24164c70 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 0x7f1f24151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 0x7f1f24151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59454/0 (socket says 172.21.15.46:59454) 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 -- 172.21.15.46:0/800919850 learned_addr learned my addr 172.21.15.46:0/800919850 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f237fe640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f240d2480 0x7f1f241f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f23fff640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f24164c70 0x7f1f241f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 -- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f24164c70 msgr2=0x7f1f241f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f24164c70 0x7f1f241f7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 -- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f240d2480 msgr2=0x7f1f241f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f240d2480 0x7f1f241f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 -- 172.21.15.46:0/800919850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f241f6ac0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.319+0000 7f1f22ffd640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 0x7f1f24151d90 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800c400 tx=0x7f1f1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.320+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f18017070 con 0x7f1f2415dfe0 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.320+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f18005ce0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.320+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f241f80b0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.320+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f180072d0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.320+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f241522d0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.321+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f2408cfd0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.321+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1f1801a070 con 0x7f1f2415dfe0 2024-05-09T03:19:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.322+0000 7f1f0bfff640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f100762d0 0x7f1f100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.322+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7f1f18097350 con 0x7f1f2415dfe0 2024-05-09T03:19:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.322+0000 7f1f237fe640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f100762d0 0x7f1f100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.323+0000 7f1f237fe640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f100762d0 0x7f1f100787e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1f14034000 tx=0x7f1f1402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.327+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1f2408cfd0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.554+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f2406a270 con 0x7f1f2415dfe0 2024-05-09T03:19:04.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.555+0000 7f1f0bfff640 1 -- 172.21.15.46:0/800919850 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) ==== 95+0+4343 (secure 0 0 0) 0x7f1f1805fbb0 con 0x7f1f2415dfe0 2024-05-09T03:19:04.556 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:04.556 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":257,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:04.180016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:04.556 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 257 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f100762d0 msgr2=0x7f1f100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f100762d0 0x7f1f100787e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1f14034000 tx=0x7f1f1402e000 comp rx=0 tx=0).stop 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 msgr2=0x7f1f24151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 0x7f1f24151d90 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800c400 tx=0x7f1f1800c8c0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 shutdown_connections 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1f24164c70 0x7f1f241f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1f100762d0 0x7f1f100787e0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f2415dfe0 0x7f1f24151d90 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 --2- 172.21.15.46:0/800919850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f240d2480 0x7f1f241f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 >> 172.21.15.46:0/800919850 conn(0x7f1f24073db0 msgr2=0x7f1f2415fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:04.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 shutdown_connections 2024-05-09T03:19:04.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.557+0000 7f1f29dee640 1 -- 172.21.15.46:0/800919850 wait complete. 2024-05-09T03:19:04.569 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:04.569 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.683+0000 7fc62151f640 1 Processor -- start 2024-05-09T03:19:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.683+0000 7fc62151f640 1 -- start start 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc62151f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc61c15d180 0x7fc61c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc62151f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc61c0b26e0 0x7fc61c0d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc62151f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0d2f20 0x7fc61c160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc62151f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc61c06eaf0 con 0x7fc61c0b26e0 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc62151f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc61c15d590 con 0x7fc61c15d180 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc62151f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c0b2af0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.684+0000 7fc613fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc61c0b26e0 0x7fc61c0d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc61c0b26e0 0x7fc61c0d29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59466/0 (socket says 172.21.15.46:59466) 2024-05-09T03:19:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 -- 172.21.15.46:0/1803743884 learned_addr learned my addr 172.21.15.46:0/1803743884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc61affd640 1 --2- 172.21.15.46:0/1803743884 >> v2:172.21.15.88:3300/0 conn(0x7fc61c15d180 0x7fc61c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc61b7fe640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0d2f20 0x7fc61c160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 -- 172.21.15.46:0/1803743884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0d2f20 msgr2=0x7fc61c160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0d2f20 0x7fc61c160cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 -- 172.21.15.46:0/1803743884 >> v2:172.21.15.88:3300/0 conn(0x7fc61c15d180 msgr2=0x7fc61c0b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 --2- 172.21.15.46:0/1803743884 >> v2:172.21.15.88:3300/0 conn(0x7fc61c15d180 0x7fc61c0b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc613fff640 1 -- 172.21.15.46:0/1803743884 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c06e1c0 con 0x7fc61c0b26e0 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc61affd640 1 --2- 172.21.15.46:0/1803743884 >> v2:172.21.15.88:3300/0 conn(0x7fc61c15d180 0x7fc61c0b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.685+0000 7fc61b7fe640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0d2f20 0x7fc61c160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc613fff640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0b26e0 0x7fc61c0d29e0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fc61c09c360 tx=0x7fc60402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892ece0f9afbd73d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc61a7fc640 1 -- 172.21.15.46:0/1803743884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604037070 con 0x7fc61c0b26e0 2024-05-09T03:19:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc61a7fc640 1 -- 172.21.15.46:0/1803743884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc604032440 con 0x7fc61c0b26e0 2024-05-09T03:19:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc61a7fc640 1 -- 172.21.15.46:0/1803743884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604032810 con 0x7fc61c0b26e0 2024-05-09T03:19:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc62151f640 1 -- 172.21.15.46:0/1803743884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0b26e0 msgr2=0x7fc61c0d29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc62151f640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0b26e0 0x7fc61c0d29e0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fc61c09c360 tx=0x7fc60402a1e0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc62151f640 1 -- 172.21.15.46:0/1803743884 shutdown_connections 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc62151f640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0d2f20 0x7fc61c160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.686+0000 7fc62151f640 1 --2- 172.21.15.46:0/1803743884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0b26e0 0x7fc61c0d29e0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.687+0000 7fc62151f640 1 --2- 172.21.15.46:0/1803743884 >> v2:172.21.15.88:3300/0 conn(0x7fc61c15d180 0x7fc61c0b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.687+0000 7fc62151f640 1 -- 172.21.15.46:0/1803743884 >> 172.21.15.46:0/1803743884 conn(0x7fc61c09b7d0 msgr2=0x7fc61c09bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.687+0000 7fc62151f640 1 -- 172.21.15.46:0/1803743884 shutdown_connections 2024-05-09T03:19:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.687+0000 7fc62151f640 1 -- 172.21.15.46:0/1803743884 wait complete. 2024-05-09T03:19:04.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.688+0000 7fc62151f640 1 Processor -- start 2024-05-09T03:19:04.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.688+0000 7fc62151f640 1 -- start start 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc62151f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0b26e0 0x7fc61c160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc62151f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 0x7fc61c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc62151f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc61c15d180 0x7fc61c159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc62151f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c06ef80 con 0x7fc61c0d2f20 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc62151f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c06e780 con 0x7fc61c15d180 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc62151f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c0d3330 con 0x7fc61c0b26e0 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 0x7fc61c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 0x7fc61c1599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59472/0 (socket says 172.21.15.46:59472) 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 -- 172.21.15.46:0/2776826950 learned_addr learned my addr 172.21.15.46:0/2776826950 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc61b7fe640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc61c15d180 0x7fc61c159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc61affd640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0b26e0 0x7fc61c160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 -- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0b26e0 msgr2=0x7fc61c160260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0b26e0 0x7fc61c160260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 -- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc61c15d180 msgr2=0x7fc61c159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc61c15d180 0x7fc61c159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 -- 172.21.15.46:0/2776826950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c15d910 con 0x7fc61c0d2f20 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc613fff640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 0x7fc61c1599f0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fc60402a220 tx=0x7fc6040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.690+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604037040 con 0x7fc61c0d2f20 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.691+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc604042070 con 0x7fc61c0d2f20 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.691+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c15dbf0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.691+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604030bb0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.691+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c15e1b0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.691+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c15fa10 con 0x7fc61c0d2f20 2024-05-09T03:19:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.692+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc604047460 con 0x7fc61c0d2f20 2024-05-09T03:19:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.693+0000 7fc6137fe640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5fc0762d0 0x7fc5fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.693+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7fc6040b8c40 con 0x7fc61c0d2f20 2024-05-09T03:19:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.693+0000 7fc61affd640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5fc0762d0 0x7fc5fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.693+0000 7fc61affd640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5fc0762d0 0x7fc5fc0787e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc60c006fd0 tx=0x7fc60c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.697+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc6040815b0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.934+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc61c150cd0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.935+0000 7fc6137fe640 1 -- 172.21.15.46:0/2776826950 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 257 v257) ==== 95+0+4343 (secure 0 0 0) 0x7fc6040858b0 con 0x7fc61c0d2f20 2024-05-09T03:19:04.936 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:04.936 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":257,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:04.180016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 257 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5fc0762d0 msgr2=0x7fc5fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5fc0762d0 0x7fc5fc0787e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc60c006fd0 tx=0x7fc60c008040 comp rx=0 tx=0).stop 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 msgr2=0x7fc61c1599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 0x7fc61c1599f0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fc60402a220 tx=0x7fc6040046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 shutdown_connections 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc61c15d180 0x7fc61c159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5fc0762d0 0x7fc5fc0787e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc61c0d2f20 0x7fc61c1599f0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 --2- 172.21.15.46:0/2776826950 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc61c0b26e0 0x7fc61c160260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 >> 172.21.15.46:0/2776826950 conn(0x7fc61c09b7d0 msgr2=0x7fc61c15c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 shutdown_connections 2024-05-09T03:19:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:04.938+0000 7fc62151f640 1 -- 172.21.15.46:0/2776826950 wait complete. 2024-05-09T03:19:04.950 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:04.950 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.067+0000 7f43d0314640 1 Processor -- start 2024-05-09T03:19:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.067+0000 7f43d0314640 1 -- start start 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43d0314640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f43c81615e0 0x7f43c815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43d0314640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 0x7f43c80d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43d0314640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c8160b20 con 0x7f43c8164c70 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43d0314640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f43c815e520 con 0x7f43c81615e0 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43d0314640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f43c8161150 con 0x7f43c80d2480 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43ce0b0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f43c8164c70 0x7f43c80d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35500/0 (socket says 172.21.15.46:35500) 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.068+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1264269564 learned_addr learned my addr 172.21.15.46:0/1264269564 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd8af640 1 --2- 172.21.15.46:0/1264269564 >> v2:172.21.15.88:3300/0 conn(0x7f43c81615e0 0x7f43c815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43ce8b1640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 0x7f43c80d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791527050 0 0) 0x7f43c8160b20 con 0x7f43c8164c70 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43b4003520 con 0x7f43c8164c70 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 512579022 0 0) 0x7f43b4003520 con 0x7f43c8164c70 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 msgr2=0x7f43c80d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 0x7f43c80d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 >> v2:172.21.15.88:3300/0 conn(0x7f43c81615e0 msgr2=0x7f43c815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 --2- 172.21.15.46:0/1264269564 >> v2:172.21.15.88:3300/0 conn(0x7f43c81615e0 0x7f43c815dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c80d15f0 con 0x7f43c8164c70 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f43c40016f0 con 0x7f43c8164c70 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 >> v1:172.21.15.46:6789/0 conn(0x7f43c8164c70 legacy=0x7f43c80d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 0x7f43b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43b4004d40 0x7f43b4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43c8161150 con 0x7f43b4003a00 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f43b400ba70 con 0x7f43b4004d40 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f43b400bbf0 con 0x7f43c80d2480 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43cd8af640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 0x7f43b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.069+0000 7f43ce8b1640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43b4004d40 0x7f43b4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 msgr2=0x7f43b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c80d2480 0x7f43b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43b4004d40 msgr2=0x7f43b4005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43b4004d40 0x7f43b4005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1264269564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b4003520 con 0x7f43b4003a00 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce8b1640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43b4004d40 0x7f43b4005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43b4003e70 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f43c400c210 tx=0x7f43b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4454970e09c120bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4007d50 con 0x7f43b4003a00 2024-05-09T03:19:05.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c4005ad0 con 0x7f43b4003a00 2024-05-09T03:19:05.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.070+0000 7f43cd0ae640 1 -- 172.21.15.46:0/1264269564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4007070 con 0x7f43b4003a00 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43d0314640 1 -- 172.21.15.46:0/1264269564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 msgr2=0x7f43b4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43d0314640 1 --2- 172.21.15.46:0/1264269564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43b4003e70 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f43c400c210 tx=0x7f43b4005830 comp rx=0 tx=0).stop 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43ce8b1640 1 -- 172.21.15.46:0/1264269564 reap_dead start 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43d0314640 1 -- 172.21.15.46:0/1264269564 shutdown_connections 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43d0314640 1 -- 172.21.15.46:0/1264269564 >> 172.21.15.46:0/1264269564 conn(0x7f43c8073db0 msgr2=0x7f43c80741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43d0314640 1 -- 172.21.15.46:0/1264269564 shutdown_connections 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.071+0000 7f43d0314640 1 -- 172.21.15.46:0/1264269564 wait complete. 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.073+0000 7f43d0314640 1 Processor -- start 2024-05-09T03:19:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.073+0000 7f43d0314640 1 -- start start 2024-05-09T03:19:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43c815e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c815ea20 0x7f43c80cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43c80cc3a0 0x7f43c8144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f43c81619f0 con 0x7f43b4003a00 2024-05-09T03:19:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f43c8160de0 con 0x7f43c80cc3a0 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f43c80d2890 con 0x7f43c815ea20 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43c815e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43c815e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59496/0 (socket says 172.21.15.46:59496) 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1485072902 learned_addr learned my addr 172.21.15.46:0/1485072902 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43cd8af640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c815ea20 0x7f43c80cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce8b1640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43c80cc3a0 0x7f43c8144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c815ea20 msgr2=0x7f43c80cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c815ea20 0x7f43c80cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43c80cc3a0 msgr2=0x7f43c8144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43c80cc3a0 0x7f43c8144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 -- 172.21.15.46:0/1485072902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c815f250 con 0x7f43b4003a00 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43ce0b0640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43c815e4e0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f43c400c300 tx=0x7f43c4001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4007d50 con 0x7f43b4003a00 2024-05-09T03:19:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c4042070 con 0x7f43b4003a00 2024-05-09T03:19:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4007290 con 0x7f43b4003a00 2024-05-09T03:19:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.075+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c815f530 con 0x7f43b4003a00 2024-05-09T03:19:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.076+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c815faf0 con 0x7f43b4003a00 2024-05-09T03:19:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.076+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c808cfd0 con 0x7f43b4003a00 2024-05-09T03:19:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.077+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f43c4007430 con 0x7f43b4003a00 2024-05-09T03:19:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.078+0000 7f43beffd640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43ac076300 0x7f43ac078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.078+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7f43c40b7fe0 con 0x7f43b4003a00 2024-05-09T03:19:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.078+0000 7f43cd8af640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43ac076300 0x7f43ac078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.079+0000 7f43cd8af640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43ac076300 0x7f43ac078810 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f43c81605a0 tx=0x7f43b8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.083+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f43c808cfd0 con 0x7f43b4003a00 2024-05-09T03:19:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.316+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c806a270 con 0x7f43b4003a00 2024-05-09T03:19:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.317+0000 7f43beffd640 1 -- 172.21.15.46:0/1485072902 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) ==== 95+0+5166 (secure 0 0 0) 0x7f43c4005aa0 con 0x7f43b4003a00 2024-05-09T03:19:05.318 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:05.318 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":258,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/905138228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":905138228},{"type":"v1","addr":"172.21.15.46:6834","nonce":905138228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":258}],"filesystems":[{"mdsmap":{"epoch":257,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:04.180016+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 258 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43ac076300 msgr2=0x7f43ac078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43ac076300 0x7f43ac078810 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f43c81605a0 tx=0x7f43b8009190 comp rx=0 tx=0).stop 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 msgr2=0x7f43c815e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43c815e4e0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f43c400c300 tx=0x7f43c4001a30 comp rx=0 tx=0).stop 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 shutdown_connections 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f43c80cc3a0 0x7f43c8144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f43c815ea20 0x7f43c80cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f43ac076300 0x7f43ac078810 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 --2- 172.21.15.46:0/1485072902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43b4003a00 0x7f43c815e4e0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 >> 172.21.15.46:0/1485072902 conn(0x7f43c8073db0 msgr2=0x7f43c80d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 shutdown_connections 2024-05-09T03:19:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.319+0000 7f43d0314640 1 -- 172.21.15.46:0/1485072902 wait complete. 2024-05-09T03:19:05.331 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate mds_snaptable' led to DAMAGED state 2024-05-09T03:19:05.331 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate mds0_inotable' 2024-05-09T03:19:05.331 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:05.332 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:05.348 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:19:05.348 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:05.348 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:05.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:05.348 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:19:05.404 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:19:05.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:05.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.558+0000 7ff3727b7640 1 Processor -- start 2024-05-09T03:19:05.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.558+0000 7ff3727b7640 1 -- start start 2024-05-09T03:19:05.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff3727b7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff36c0d2eb0 0x7ff36c0d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff3727b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 0x7ff36c0b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff3727b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff36c0d4a60 con 0x7ff36c0b7760 2024-05-09T03:19:05.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff3727b7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff36c0d4d40 con 0x7ff36c0d2eb0 2024-05-09T03:19:05.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff3727b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff36c0d5020 con 0x7ff36c0b95a0 2024-05-09T03:19:05.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff36b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff36c0b7760 0x7ff36c0b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35514/0 (socket says 172.21.15.46:35514) 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff36b7fe640 1 -- 172.21.15.46:0/723271687 learned_addr learned my addr 172.21.15.46:0/723271687 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff36bfff640 1 --2- 172.21.15.46:0/723271687 >> v2:172.21.15.88:3300/0 conn(0x7ff36c0d2eb0 0x7ff36c0d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff370d54640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 0x7ff36c0b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3551819885 0 0) 0x7ff36c0d4a60 con 0x7ff36c0b7760 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.560+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff354003520 con 0x7ff36c0b7760 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3120404618 0 0) 0x7ff354003520 con 0x7ff36c0b7760 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 msgr2=0x7ff36c0b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 0x7ff36c0b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 >> v2:172.21.15.88:3300/0 conn(0x7ff36c0d2eb0 msgr2=0x7ff36c0d32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 --2- 172.21.15.46:0/723271687 >> v2:172.21.15.88:3300/0 conn(0x7ff36c0d2eb0 0x7ff36c0d32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c0d5300 con 0x7ff36c0b7760 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ff3600016f0 con 0x7ff36c0b7760 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 >> v1:172.21.15.46:6789/0 conn(0x7ff36c0b7760 legacy=0x7ff36c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 0x7ff354003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff354003fd0 0x7ff354004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff354004d40 0x7ff354005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c0d4d40 con 0x7ff354004d40 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c0d5020 con 0x7ff354003fd0 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff35400bac0 con 0x7ff36c0b95a0 2024-05-09T03:19:05.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36bfff640 1 --2- 172.21.15.46:0/723271687 >> v2:172.21.15.88:3300/0 conn(0x7ff36c0d2eb0 0x7ff36c0d32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff370d54640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff354004d40 0x7ff354005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36b7fe640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff354003fd0 0x7ff354004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff36bfff640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 0x7ff354003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff370d54640 1 -- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 msgr2=0x7ff354003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff370d54640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c0b95a0 0x7ff354003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff370d54640 1 -- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff354003fd0 msgr2=0x7ff354004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff370d54640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff354003fd0 0x7ff354004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.561+0000 7ff370d54640 1 -- 172.21.15.46:0/723271687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff354003520 con 0x7ff354004d40 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.562+0000 7ff370d54640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff354004d40 0x7ff354005150 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff364009f90 tx=0x7ff354005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e8b88779b300aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.562+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364009c10 con 0x7ff354004d40 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.562+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff364036070 con 0x7ff354004d40 2024-05-09T03:19:05.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.562+0000 7ff36affd640 1 -- 172.21.15.46:0/723271687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364031810 con 0x7ff354004d40 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff3727b7640 1 -- 172.21.15.46:0/723271687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff354004d40 msgr2=0x7ff354005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff3727b7640 1 --2- 172.21.15.46:0/723271687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff354004d40 0x7ff354005150 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff364009f90 tx=0x7ff354005830 comp rx=0 tx=0).stop 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff370d54640 1 -- 172.21.15.46:0/723271687 reap_dead start 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff3727b7640 1 -- 172.21.15.46:0/723271687 shutdown_connections 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff3727b7640 1 -- 172.21.15.46:0/723271687 >> 172.21.15.46:0/723271687 conn(0x7ff36c08d390 msgr2=0x7ff36c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff3727b7640 1 -- 172.21.15.46:0/723271687 shutdown_connections 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.563+0000 7ff3727b7640 1 -- 172.21.15.46:0/723271687 wait complete. 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.565+0000 7ff3727b7640 1 Processor -- start 2024-05-09T03:19:05.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.565+0000 7ff3727b7640 1 -- start start 2024-05-09T03:19:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff3727b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 0x7ff36c0cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff3727b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff36c15cb70 0x7ff36c14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff3727b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c1f9900 0x7ff36c1f9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff3727b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c0b5360 con 0x7ff36c15c160 2024-05-09T03:19:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff3727b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c0b51e0 con 0x7ff36c15cb70 2024-05-09T03:19:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff3727b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff36c163bd0 con 0x7ff36c1f9900 2024-05-09T03:19:05.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff36bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 0x7ff36c0cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff36bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 0x7ff36c0cf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59520/0 (socket says 172.21.15.46:59520) 2024-05-09T03:19:05.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff36bfff640 1 -- 172.21.15.46:0/718494136 learned_addr learned my addr 172.21.15.46:0/718494136 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:05.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff36b7fe640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff36c15cb70 0x7ff36c14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.566+0000 7ff370d54640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c1f9900 0x7ff36c1f9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff36bfff640 1 -- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c1f9900 msgr2=0x7ff36c1f9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff36bfff640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c1f9900 0x7ff36c1f9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff36bfff640 1 -- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff36c15cb70 msgr2=0x7ff36c14bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:05.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff36bfff640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff36c15cb70 0x7ff36c14bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:05.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff36bfff640 1 -- 172.21.15.46:0/718494136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c1fa250 con 0x7ff36c15c160 2024-05-09T03:19:05.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff370d54640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c1f9900 0x7ff36c1f9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:05.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff36bfff640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 0x7ff36c0cf350 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7ff35c011930 tx=0x7ff35c011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:05.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c00fbd0 con 0x7ff36c15c160 2024-05-09T03:19:05.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff35c00fd70 con 0x7ff36c15c160 2024-05-09T03:19:05.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36c15cf80 con 0x7ff36c15c160 2024-05-09T03:19:05.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0156a0 con 0x7ff36c15c160 2024-05-09T03:19:05.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.567+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff36c1fa7b0 con 0x7ff36c15c160 2024-05-09T03:19:05.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.568+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36c0cef80 con 0x7ff36c15c160 2024-05-09T03:19:05.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.569+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff35c015840 con 0x7ff36c15c160 2024-05-09T03:19:05.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.569+0000 7ff368ff9640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3580762d0 0x7ff3580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:05.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.569+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7ff35c098ab0 con 0x7ff36c15c160 2024-05-09T03:19:05.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.569+0000 7ff36b7fe640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3580762d0 0x7ff3580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:05.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.570+0000 7ff36b7fe640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3580762d0 0x7ff3580787e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff360001d00 tx=0x7ff3600047e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:05.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.574+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff36c1fa250 con 0x7ff36c15c160 2024-05-09T03:19:05.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:05.802+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff36c06a270 con 0x7ff36c15c160 2024-05-09T03:19:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.193+0000 7ff368ff9640 1 -- 172.21.15.46:0/718494136 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v259) ==== 161+0+0 (secure 0 0 0) 0x7ff35c019030 con 0x7ff36c15c160 2024-05-09T03:19:06.194 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3580762d0 msgr2=0x7ff3580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3580762d0 0x7ff3580787e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff360001d00 tx=0x7ff3600047e0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 msgr2=0x7ff36c0cf350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 0x7ff36c0cf350 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7ff35c011930 tx=0x7ff35c011df0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 shutdown_connections 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff36c1f9900 0x7ff36c1f9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff36c15cb70 0x7ff36c14bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff3580762d0 0x7ff3580787e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 --2- 172.21.15.46:0/718494136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff36c15c160 0x7ff36c0cf350 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 >> 172.21.15.46:0/718494136 conn(0x7ff36c08d390 msgr2=0x7ff36c0b9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 shutdown_connections 2024-05-09T03:19:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.195+0000 7ff3727b7640 1 -- 172.21.15.46:0/718494136 wait complete. 2024-05-09T03:19:06.209 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.326+0000 7fce24a49640 1 Processor -- start 2024-05-09T03:19:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.326+0000 7fce24a49640 1 -- start start 2024-05-09T03:19:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce24a49640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b7e70 0x7fce200d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce24a49640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce200b5550 con 0x7fce200d3df0 2024-05-09T03:19:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce24a49640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce200d44e0 con 0x7fce200b5140 2024-05-09T03:19:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce24a49640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fce200d3480 con 0x7fce200b7e70 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fce200d3df0 0x7fce200b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34836/0 (socket says 172.21.15.46:34836) 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1ed9d640 1 -- 172.21.15.46:0/4194106303 learned_addr learned my addr 172.21.15.46:0/4194106303 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1dd9b640 1 --2- 172.21.15.46:0/4194106303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b7e70 0x7fce200d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700446994 0 0) 0x7fce200b5550 con 0x7fce200d3df0 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce0c003520 con 0x7fce200d3df0 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 767060653 0 0) 0x7fce200d44e0 con 0x7fce200b5140 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce200b5550 con 0x7fce200b5140 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1dd9b640 1 -- 172.21.15.46:0/4194106303 >> v1:172.21.15.88:6789/0 conn(0x7fce200b5140 legacy=0x7fce200b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.328+0000 7fce1dd9b640 1 -- 172.21.15.46:0/4194106303 >> v1:172.21.15.46:6789/0 conn(0x7fce200d3df0 legacy=0x7fce200b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.329+0000 7fce1dd9b640 1 -- 172.21.15.46:0/4194106303 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce200d47c0 con 0x7fce200b7e70 2024-05-09T03:19:06.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.329+0000 7fce1dd9b640 1 --2- 172.21.15.46:0/4194106303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b7e70 0x7fce200d38b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce080097d0 tx=0x7fce0802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df8071415518b72e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:06.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.329+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08037070 con 0x7fce200b7e70 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce08032440 con 0x7fce200b7e70 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce1d59a640 1 -- 172.21.15.46:0/4194106303 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce080327f0 con 0x7fce200b7e70 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 -- 172.21.15.46:0/4194106303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b7e70 msgr2=0x7fce200d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 --2- 172.21.15.46:0/4194106303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b7e70 0x7fce200d38b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce080097d0 tx=0x7fce0802a290 comp rx=0 tx=0).stop 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 -- 172.21.15.46:0/4194106303 shutdown_connections 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 --2- 172.21.15.46:0/4194106303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b7e70 0x7fce200d38b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 -- 172.21.15.46:0/4194106303 >> 172.21.15.46:0/4194106303 conn(0x7fce2008d3f0 msgr2=0x7fce2008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 -- 172.21.15.46:0/4194106303 shutdown_connections 2024-05-09T03:19:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.330+0000 7fce24a49640 1 -- 172.21.15.46:0/4194106303 wait complete. 2024-05-09T03:19:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.332+0000 7fce24a49640 1 Processor -- start 2024-05-09T03:19:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.332+0000 7fce24a49640 1 -- start start 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce24a49640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b5140 0x7fce200cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce24a49640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 0x7fce201fa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce24a49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce2015cb80 0x7fce20153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce24a49640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fce200d4200 con 0x7fce200d3df0 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce24a49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fce200d4d80 con 0x7fce2015cb80 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce24a49640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fce200d5580 con 0x7fce200b5140 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 0x7fce201fa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 0x7fce201fa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40228/0 (socket says 172.21.15.46:40228) 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 -- 172.21.15.46:0/807697305 learned_addr learned my addr 172.21.15.46:0/807697305 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1ed9d640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce2015cb80 0x7fce20153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1e59c640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b5140 0x7fce200cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 -- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b5140 msgr2=0x7fce200cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b5140 0x7fce200cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 -- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce2015cb80 msgr2=0x7fce20153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce2015cb80 0x7fce20153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 -- 172.21.15.46:0/807697305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce200cd8e0 con 0x7fce200d3df0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce1dd9b640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 0x7fce201fa3e0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fce080097d0 tx=0x7fce080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.334+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08037040 con 0x7fce200d3df0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.335+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce08004230 con 0x7fce200d3df0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.335+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce201f9900 con 0x7fce200d3df0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.335+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce08003610 con 0x7fce200d3df0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.335+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce201f9ec0 con 0x7fce200d3df0 2024-05-09T03:19:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.335+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce200cd650 con 0x7fce200d3df0 2024-05-09T03:19:06.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.336+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fce080037b0 con 0x7fce200d3df0 2024-05-09T03:19:06.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.337+0000 7fce16ffd640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce00076300 0x7fce00078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:06.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.337+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7fce080b8f70 con 0x7fce200d3df0 2024-05-09T03:19:06.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.337+0000 7fce1e59c640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce00076300 0x7fce00078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:06.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.338+0000 7fce1e59c640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce00076300 0x7fce00078810 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fce180014a0 tx=0x7fce18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.341+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fce080818e0 con 0x7fce200d3df0 2024-05-09T03:19:06.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:06.608+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fce2006a270 con 0x7fce200d3df0 2024-05-09T03:19:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.198+0000 7fce16ffd640 1 -- 172.21.15.46:0/807697305 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v260) ==== 101+0+0 (secure 0 0 0) 0x7fce08085be0 con 0x7fce200d3df0 2024-05-09T03:19:07.199 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce00076300 msgr2=0x7fce00078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce00076300 0x7fce00078810 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fce180014a0 tx=0x7fce18009040 comp rx=0 tx=0).stop 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 msgr2=0x7fce201fa3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 0x7fce201fa3e0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fce080097d0 tx=0x7fce080046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 shutdown_connections 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce2015cb80 0x7fce20153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fce00076300 0x7fce00078810 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce200d3df0 0x7fce201fa3e0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 --2- 172.21.15.46:0/807697305 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fce200b5140 0x7fce200cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 >> 172.21.15.46:0/807697305 conn(0x7fce2008d3f0 msgr2=0x7fce2015dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:07.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 shutdown_connections 2024-05-09T03:19:07.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.201+0000 7fce24a49640 1 -- 172.21.15.46:0/807697305 wait complete. 2024-05-09T03:19:07.215 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.275+0000 7faa9cd2df40 1 Processor -- start 2024-05-09T03:19:07.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.275+0000 7faa9cd2df40 1 -- start start 2024-05-09T03:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.276+0000 7faa9cd2df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf88fadb0 0x55fcf8987980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9cd2df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8987ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9cd2df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf89863f0 0x55fcf88fc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9cd2df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fcf890dcb0 con 0x55fcf89a3780 2024-05-09T03:19:07.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9cd2df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fcf8874f50 con 0x55fcf88fadb0 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9cd2df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fcf890d380 con 0x55fcf89863f0 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8987ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8987ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40250/0 (socket says 172.21.15.46:40250) 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 -- 172.21.15.46:0/305929744 learned_addr learned my addr 172.21.15.46:0/305929744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa95d8e640 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf88fadb0 0x55fcf8987980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9658f640 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf89863f0 0x55fcf88fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 -- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf89863f0 msgr2=0x55fcf88fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf89863f0 0x55fcf88fc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 -- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf88fadb0 msgr2=0x55fcf8987980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf88fadb0 0x55fcf8987980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 -- 172.21.15.46:0/305929744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcf8911ae0 con 0x55fcf89a3780 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa9558d640 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8987ec0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7faa8c001d80 tx=0x7faa8c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111af40f3caece2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa87fff640 1 -- 172.21.15.46:0/305929744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c00bd10 con 0x55fcf89a3780 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa87fff640 1 -- 172.21.15.46:0/305929744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c00f400 con 0x55fcf89a3780 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.277+0000 7faa87fff640 1 -- 172.21.15.46:0/305929744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c00f7f0 con 0x55fcf89a3780 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 -- 172.21.15.46:0/305929744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 msgr2=0x55fcf8987ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8987ec0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7faa8c001d80 tx=0x7faa8c001e70 comp rx=0 tx=0).stop 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 -- 172.21.15.46:0/305929744 shutdown_connections 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf89863f0 0x55fcf88fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8987ec0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/305929744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf88fadb0 0x55fcf8987980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 -- 172.21.15.46:0/305929744 >> 172.21.15.46:0/305929744 conn(0x55fcf87f0ac0 msgr2=0x55fcf89a20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 -- 172.21.15.46:0/305929744 shutdown_connections 2024-05-09T03:19:07.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.278+0000 7faa9cd2df40 1 -- 172.21.15.46:0/305929744 wait complete. 2024-05-09T03:19:07.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.279+0000 7faa9cd2df40 1 Processor -- start 2024-05-09T03:19:07.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.279+0000 7faa9cd2df40 1 -- start start 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.280+0000 7faa9cd2df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf88fadb0 0x55fcf8911dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.280+0000 7faa9cd2df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf89863f0 0x55fcf88cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.280+0000 7faa9cd2df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8985900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.280+0000 7faa9cd2df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fcf8986c00 con 0x55fcf89a3780 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.280+0000 7faa9cd2df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fcf898a830 con 0x55fcf89863f0 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.280+0000 7faa9cd2df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55fcf88fcd90 con 0x55fcf88fadb0 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8985900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8985900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40260/0 (socket says 172.21.15.46:40260) 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 learned_addr learned my addr 172.21.15.46:0/2064364973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf89863f0 0x55fcf88cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf88fadb0 0x55fcf8911dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf88fadb0 msgr2=0x55fcf8911dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf88fadb0 0x55fcf8911dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf89863f0 msgr2=0x55fcf88cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcf89863f0 0x55fcf88cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcf88fcfe0 con 0x55fcf89a3780 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55fcf88fadb0 0x55fcf8911dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9658f640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8985900 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7faa90009db0 tx=0x7faa90009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa86ffd640 1 -- 172.21.15.46:0/2064364973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa900143f0 con 0x55fcf89a3780 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa86ffd640 1 -- 172.21.15.46:0/2064364973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa90019070 con 0x55fcf89a3780 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fcf8910bb0 con 0x55fcf89a3780 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.281+0000 7faa86ffd640 1 -- 172.21.15.46:0/2064364973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa900147d0 con 0x55fcf89a3780 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.282+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcf898a570 con 0x55fcf89a3780 2024-05-09T03:19:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.283+0000 7faa86ffd640 1 -- 172.21.15.46:0/2064364973 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faa90014970 con 0x55fcf89a3780 2024-05-09T03:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.284+0000 7faa86ffd640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa78074cb0 0x7faa780771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.284+0000 7faa86ffd640 1 -- 172.21.15.46:0/2064364973 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7faa90096910 con 0x55fcf89a3780 2024-05-09T03:19:07.286 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.284+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fcf8989fb0 0x55fcf88cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.284+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa78074cb0 0x7faa780771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.284+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf88cb5a0 con 0x55fcf8989fb0 2024-05-09T03:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.284+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fcf8989fb0 0x55fcf88cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.285+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fcf8989fb0 0x55fcf88cb060 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.285+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa78074cb0 0x7faa780771c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7faa88056000 tx=0x7faa8805c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.287+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v112'846 uv820 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.287+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a45490 con 0x55fcf8989fb0 2024-05-09T03:19:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.288+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v112'847 uv847 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.288+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a45830 con 0x55fcf8989fb0 2024-05-09T03:19:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.290+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v112'848 uv848 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.291 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.290+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46be0 con 0x55fcf8989fb0 2024-05-09T03:19:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.291+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v112'849 uv849 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.291+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46f80 con 0x55fcf8989fb0 2024-05-09T03:19:07.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.293+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v112'850 uv850 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.293+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a45bd0 con 0x55fcf8989fb0 2024-05-09T03:19:07.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.294+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v112'851 uv851 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.294+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a45fc0 con 0x55fcf8989fb0 2024-05-09T03:19:07.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.295+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v112'852 uv826 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.295+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.297+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v112'853 uv853 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.297+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.298+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v112'854 uv854 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.299 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:07.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.298+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.300+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v112'855 uv855 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.300+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.301+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v112'856 uv856 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.301+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.303+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v112'857 uv857 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.303+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.304+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v112'858 uv833 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.305+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.306+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v112'859 uv859 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.306+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.307+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v112'860 uv860 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.308 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.307+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.309+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v112'861 uv861 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.309+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.310+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v112'862 uv862 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.310+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.312+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v112'863 uv863 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.312+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.313+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v112'864 uv864 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.313+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.315+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v112'865 uv839 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.315+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.316+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v112'866 uv866 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.316+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.318+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v112'867 uv867 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.318 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:07.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.318+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.319+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v112'868 uv868 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.319+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.321+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v112'869 uv869 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.321+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.322+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v112'870 uv870 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.322+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.324+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v112'871 uv845 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.324+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.325+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v112'872 uv872 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.325+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.326+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v112'873 uv873 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.327 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.326+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.327+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v112'874 uv874 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.328+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.329+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v112'875 uv875 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.329+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8989fb0 2024-05-09T03:19:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.330+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v112'876 uv876 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.330+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fcf8a46760 0x55fcf8a49840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.330+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a46360 con 0x55fcf8a46760 2024-05-09T03:19:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.330+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fcf8a46760 0x55fcf8a49840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.331+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fcf8a46760 0x55fcf8a49840 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.332+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v112'612 uv595 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.333+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a49d80 con 0x55fcf8a46760 2024-05-09T03:19:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.334+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v112'613 uv613 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.334+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4a120 con 0x55fcf8a46760 2024-05-09T03:19:07.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.335+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v112'614 uv614 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.336 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:07.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.335+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4a970 con 0x55fcf8a46760 2024-05-09T03:19:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.336+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v112'615 uv615 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.336+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4ad10 con 0x55fcf8a46760 2024-05-09T03:19:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.337+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v112'616 uv616 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.337+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b0b0 con 0x55fcf8a46760 2024-05-09T03:19:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.339+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v112'617 uv600 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.339+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a4b450 con 0x55fcf8a46760 2024-05-09T03:19:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.340+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v112'618 uv618 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.340+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b450 con 0x55fcf8a46760 2024-05-09T03:19:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.341+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v112'619 uv619 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.342 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:07.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.341+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b890 con 0x55fcf8a46760 2024-05-09T03:19:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.342+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v112'620 uv620 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.343+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b890 con 0x55fcf8a46760 2024-05-09T03:19:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.344+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v112'621 uv621 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.344+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b5f0 con 0x55fcf8a46760 2024-05-09T03:19:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.345+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v112'622 uv605 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.345+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a4b5f0 con 0x55fcf8a46760 2024-05-09T03:19:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.346+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v112'623 uv623 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.346+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b5a0 con 0x55fcf8a46760 2024-05-09T03:19:07.348 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.347+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v112'624 uv624 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.347+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b5a0 con 0x55fcf8a46760 2024-05-09T03:19:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.349+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v112'625 uv625 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.349+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b550 con 0x55fcf8a46760 2024-05-09T03:19:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.350+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v112'626 uv626 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.350+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b550 con 0x55fcf8a46760 2024-05-09T03:19:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.351+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v112'627 uv611 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.351+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a4b550 con 0x55fcf8a46760 2024-05-09T03:19:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.352+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v112'628 uv628 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.352+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8a46760 2024-05-09T03:19:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.353+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v112'629 uv629 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.354 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.353+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8a46760 2024-05-09T03:19:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.354+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v112'630 uv630 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.354+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8a46760 2024-05-09T03:19:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.355+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v112'631 uv631 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.355+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8a46760 2024-05-09T03:19:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.357+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v112'632 uv632 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a46760 2024-05-09T03:19:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.357+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8989fb0 2024-05-09T03:19:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.358+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v112'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.358+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8989fb0 2024-05-09T03:19:07.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.359+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v112'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.359+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8989fb0 2024-05-09T03:19:07.360 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.360+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v112'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.360+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8989fb0 2024-05-09T03:19:07.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.361+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v112'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.361+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8989fb0 2024-05-09T03:19:07.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.362+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v112'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.362+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4b530 con 0x55fcf8989fb0 2024-05-09T03:19:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.363+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v112'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.363+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fcf8a4bea0 0x55fcf8a4c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.363+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4e990 con 0x55fcf8a4bea0 2024-05-09T03:19:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.363+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fcf8a4bea0 0x55fcf8a4c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.364+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fcf8a4bea0 0x55fcf8a4c2b0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.365+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v112'124 uv123 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.365+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a4e990 con 0x55fcf8a4bea0 2024-05-09T03:19:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.366+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v112'125 uv125 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.366+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4ed80 con 0x55fcf8a4bea0 2024-05-09T03:19:07.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.367+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v112'126 uv126 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.368 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.367+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4ed80 con 0x55fcf8a4bea0 2024-05-09T03:19:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.368+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v112'127 uv127 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.368+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4ed80 con 0x55fcf8989fb0 2024-05-09T03:19:07.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.370+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v112'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.370+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a4ed80 con 0x55fcf8989fb0 2024-05-09T03:19:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.371+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v112'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.371+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4ed80 con 0x55fcf8989fb0 2024-05-09T03:19:07.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.372+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v112'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.373 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:07.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.372+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4fd90 con 0x55fcf8989fb0 2024-05-09T03:19:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.373+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v112'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.373+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4fd90 con 0x55fcf8989fb0 2024-05-09T03:19:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.375+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v112'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.375+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a4ed80 con 0x55fcf8989fb0 2024-05-09T03:19:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.376+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v112'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.376+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fcf8a4f290 0x55fcf8a516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.376+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a51d10 con 0x55fcf8a4f290 2024-05-09T03:19:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.377+0000 7faa9658f640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fcf8a4f290 0x55fcf8a516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.377+0000 7faa9658f640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fcf8a4f290 0x55fcf8a516b0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.378+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v112'305 uv299 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.378+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a521d0 con 0x55fcf8a4f290 2024-05-09T03:19:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.379+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v112'306 uv306 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.379+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a526b0 con 0x55fcf8a4f290 2024-05-09T03:19:07.381 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:07.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.380+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v112'307 uv307 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.380+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a53f60 con 0x55fcf8a4f290 2024-05-09T03:19:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.383+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v112'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.383+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a54300 con 0x55fcf8a4f290 2024-05-09T03:19:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.384+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v112'309 uv309 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.384+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a52f50 con 0x55fcf8a4f290 2024-05-09T03:19:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.385+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v112'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.385+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a53340 con 0x55fcf8a4f290 2024-05-09T03:19:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.386+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v112'311 uv304 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.386+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a53340 con 0x55fcf8a4f290 2024-05-09T03:19:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.387+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v112'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.387+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a532f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.389+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v112'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.389 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:07.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.389+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a532f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.390+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v112'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.390+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a532f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.391+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v112'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.391+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fcf8a53800 0x55fcf8a566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.391+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56bf0 con 0x55fcf8a53800 2024-05-09T03:19:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.391+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fcf8a53800 0x55fcf8a566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.392+0000 7faa95d8e640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fcf8a53800 0x55fcf8a566b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.393+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v112'138 uv137 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a53800 2024-05-09T03:19:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.393+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a56bf0 con 0x55fcf8a53800 2024-05-09T03:19:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.394+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v112'139 uv139 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a53800 2024-05-09T03:19:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.394+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a53800 2024-05-09T03:19:07.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.395+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v112'140 uv140 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a53800 2024-05-09T03:19:07.396 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:07.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.395+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a53800 2024-05-09T03:19:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.397+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v112'141 uv141 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a53800 2024-05-09T03:19:07.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.397+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a53800 2024-05-09T03:19:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.398+0000 7faa95d8e640 1 -- 172.21.15.46:0/2064364973 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v112'142 uv142 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8805a420 con 0x55fcf8a53800 2024-05-09T03:19:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.398+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8989fb0 2024-05-09T03:19:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.400+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v112'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.400+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8989fb0 2024-05-09T03:19:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.401+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v112'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.401+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8989fb0 2024-05-09T03:19:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.402+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v112'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.403 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.402+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a57ff0 con 0x55fcf8989fb0 2024-05-09T03:19:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.403+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v112'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.403+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a57ff0 con 0x55fcf8989fb0 2024-05-09T03:19:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.404+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v112'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.404+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8989fb0 2024-05-09T03:19:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.405+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v112'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8989fb0 2024-05-09T03:19:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.405+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.406+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v112'273 uv267 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.406+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.407+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v112'274 uv274 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.407+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.408+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v112'275 uv275 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.409 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:07.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.408+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.410+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v112'276 uv276 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.410+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.411+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v112'277 uv277 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.411+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.412+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v112'278 uv272 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.412+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.413+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v112'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.413+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a56fe0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.414+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v112'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.415 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.414+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a573d0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.415+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v112'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.415+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a59e70 con 0x55fcf8a4bea0 2024-05-09T03:19:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.416+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v112'282 uv282 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.416+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fcf8a57b10 0x55fcf8a59f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.416+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a57b10 2024-05-09T03:19:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.416+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fcf8a57b10 0x55fcf8a59f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.417+0000 7faa9558d640 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fcf8a57b10 0x55fcf8a59f30 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.418+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v112'120 uv119 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a57b10 2024-05-09T03:19:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.418+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a57b10 2024-05-09T03:19:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.419+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v112'121 uv121 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a57b10 2024-05-09T03:19:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.419+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a57b10 2024-05-09T03:19:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.420+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v112'122 uv122 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a57b10 2024-05-09T03:19:07.420 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.420+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a4b0 con 0x55fcf8a57b10 2024-05-09T03:19:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.421+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v112'123 uv123 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a57b10 2024-05-09T03:19:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.421+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4bea0 2024-05-09T03:19:07.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.422+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v112'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.422+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4bea0 2024-05-09T03:19:07.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.424+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v112'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.424+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4bea0 2024-05-09T03:19:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.425+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v112'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.425 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.425+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4bea0 2024-05-09T03:19:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.426+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v112'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.426+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4bea0 2024-05-09T03:19:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.427+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v112'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.427+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4bea0 2024-05-09T03:19:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.428+0000 7faa9558d640 1 -- 172.21.15.46:0/2064364973 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v112'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa8c00f5b0 con 0x55fcf8a4bea0 2024-05-09T03:19:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.428+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4f290 2024-05-09T03:19:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.429+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v112'140 uv139 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.429+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4f290 2024-05-09T03:19:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.431+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v112'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.431+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4f290 2024-05-09T03:19:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.432+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v112'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.433 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.432+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.433+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v112'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.433+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.434+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v112'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.435+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4f290 2024-05-09T03:19:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.436+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v112'339 uv331 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.436+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a5a470 con 0x55fcf8a4f290 2024-05-09T03:19:07.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.438+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v112'340 uv340 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.438+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a910 con 0x55fcf8a4f290 2024-05-09T03:19:07.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.439+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v112'341 uv341 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.439 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:07.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.439+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a910 con 0x55fcf8a4f290 2024-05-09T03:19:07.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.440+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v112'342 uv342 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.440+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.441+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v112'343 uv343 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.441+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.442+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v112'344 uv338 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.442+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.443+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v112'345 uv345 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.443+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.444+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v112'346 uv346 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.445 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:07.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.444+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.445+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v112'347 uv347 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.445+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5b880 con 0x55fcf8a4f290 2024-05-09T03:19:07.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.446+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v112'348 uv348 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.446+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.447+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v112'349 uv349 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.447+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x55fcf8a5a870 con 0x55fcf8a4f290 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.448+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v112'350 uv350 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7faa9005b4f0 con 0x55fcf8a4f290 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fcf8989fb0 msgr2=0x55fcf88cb060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55fcf8989fb0 0x55fcf88cb060 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fcf8a4bea0 msgr2=0x55fcf8a4c2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55fcf8a4bea0 0x55fcf8a4c2b0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fcf8a57b10 msgr2=0x55fcf8a59f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55fcf8a57b10 0x55fcf8a59f30 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fcf8a46760 msgr2=0x55fcf8a49840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55fcf8a46760 0x55fcf8a49840 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 reap_dead start 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fcf8a4f290 msgr2=0x55fcf8a516b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55fcf8a4f290 0x55fcf8a516b0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fcf8a53800 msgr2=0x55fcf8a566b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55fcf8a53800 0x55fcf8a566b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa78074cb0 msgr2=0x7faa780771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faa78074cb0 0x7faa780771c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7faa88056000 tx=0x7faa8805c040 comp rx=0 tx=0).stop 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 msgr2=0x55fcf8985900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 --2- 172.21.15.46:0/2064364973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf89a3780 0x55fcf8985900 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7faa90009db0 tx=0x7faa90009ea0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9658f640 1 -- 172.21.15.46:0/2064364973 reap_dead start 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 shutdown_connections 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 >> 172.21.15.46:0/2064364973 conn(0x55fcf87f0ac0 msgr2=0x55fcf89a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:07.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 shutdown_connections 2024-05-09T03:19:07.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.449+0000 7faa9cd2df40 1 -- 172.21.15.46:0/2064364973 wait complete. 2024-05-09T03:19:07.453 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate mds0_inotable 0 2024-05-09T03:19:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.548+0000 7ff5d320bf40 1 Processor -- start 2024-05-09T03:19:07.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.548+0000 7ff5d320bf40 1 -- start start 2024-05-09T03:19:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5d320bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5556663843a0 0x55566636e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5d320bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55566638c5f0 0x555666345c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5d320bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x55566640f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5d320bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5556662eaf50 con 0x55566638ca00 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5d320bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5556662eef90 con 0x55566638c5f0 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5d320bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55566638ce10 con 0x5556663843a0 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5cc7f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x55566640f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5cc7f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x55566640f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40282/0 (socket says 172.21.15.46:40282) 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.549+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/292378835 learned_addr learned my addr 172.21.15.46:0/292378835 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cb7ef640 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55566638c5f0 0x555666345c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5556663843a0 msgr2=0x55566636e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cc7f1640 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5556663843a0 0x55566636e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55566638c5f0 msgr2=0x555666345c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cc7f1640 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55566638c5f0 0x555666345c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/292378835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555666372960 con 0x55566638ca00 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5cc7f1640 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x55566640f810 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc005fd0 tx=0x7ff5bc051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a39c97c06d10b2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5ca7ed640 1 -- 172.21.15.46:0/292378835 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc067070 con 0x55566638ca00 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5ca7ed640 1 -- 172.21.15.46:0/292378835 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5bc04fb60 con 0x55566638ca00 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.550+0000 7ff5ca7ed640 1 -- 172.21.15.46:0/292378835 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc062430 con 0x55566638ca00 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 -- 172.21.15.46:0/292378835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 msgr2=0x55566640f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x55566640f810 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc005fd0 tx=0x7ff5bc051000 comp rx=0 tx=0).stop 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 -- 172.21.15.46:0/292378835 shutdown_connections 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x55566640f810 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55566638c5f0 0x555666345c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/292378835 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5556663843a0 0x55566636e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 -- 172.21.15.46:0/292378835 >> 172.21.15.46:0/292378835 conn(0x555666266ac0 msgr2=0x555666371730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 -- 172.21.15.46:0/292378835 shutdown_connections 2024-05-09T03:19:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.551+0000 7ff5d320bf40 1 -- 172.21.15.46:0/292378835 wait complete. 2024-05-09T03:19:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.552+0000 7ff5d320bf40 1 Processor -- start 2024-05-09T03:19:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.552+0000 7ff5d320bf40 1 -- start start 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5d320bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5556663843a0 0x55566640ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5d320bf40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55566638c5f0 0x5556664125b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5d320bf40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x555666412af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5d320bf40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55566641a550 con 0x55566638ca00 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5d320bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5556663723a0 con 0x5556663843a0 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5d320bf40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5556664b1560 con 0x55566638c5f0 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x555666412af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x555666412af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40294/0 (socket says 172.21.15.46:40294) 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/1874895138 learned_addr learned my addr 172.21.15.46:0/1874895138 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cb7ef640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55566638c5f0 0x5556664125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cbff0640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5556663843a0 0x55566640ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55566638c5f0 msgr2=0x5556664125b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55566638c5f0 0x5556664125b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5556663843a0 msgr2=0x55566640ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5556663843a0 0x55566640ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.553+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/1874895138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55566640ef50 con 0x55566638ca00 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.554+0000 7ff5cc7f1640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x555666412af0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc025a80 tx=0x7ff5bc003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.554+0000 7ff5c97eb640 1 -- 172.21.15.46:0/1874895138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc070070 con 0x55566638ca00 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.554+0000 7ff5c97eb640 1 -- 172.21.15.46:0/1874895138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5bc062cb0 con 0x55566638ca00 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.554+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555666413840 con 0x55566638ca00 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.554+0000 7ff5c97eb640 1 -- 172.21.15.46:0/1874895138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc067040 con 0x55566638ca00 2024-05-09T03:19:07.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.554+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555666413560 con 0x55566638ca00 2024-05-09T03:19:07.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.555+0000 7ff5c97eb640 1 -- 172.21.15.46:0/1874895138 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff5bc073070 con 0x55566638ca00 2024-05-09T03:19:07.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.556+0000 7ff5c97eb640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5ac074cb0 0x7ff5ac0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.556+0000 7ff5c97eb640 1 -- 172.21.15.46:0/1874895138 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7ff5bc0db930 con 0x55566638ca00 2024-05-09T03:19:07.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.556+0000 7ff5cbff0640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5ac074cb0 0x7ff5ac0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.556+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55566640f160 0x555666411780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.556+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x555666411cc0 con 0x55566640f160 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.556+0000 7ff5cb7ef640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55566640f160 0x555666411780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.557+0000 7ff5cbff0640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5ac074cb0 0x7ff5ac0771c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400e390 tx=0x7ff5c4001690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.557+0000 7ff5cb7ef640 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55566640f160 0x555666411780 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5cb7ef640 1 -- 172.21.15.46:0/1874895138 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_inotable [truncate 0] v112'283 uv283 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff5b400c3c0 con 0x55566640f160 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55566640f160 msgr2=0x555666411780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55566640f160 0x555666411780 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5ac074cb0 msgr2=0x7ff5ac0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff5ac074cb0 0x7ff5ac0771c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400e390 tx=0x7ff5c4001690 comp rx=0 tx=0).stop 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 msgr2=0x555666412af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 --2- 172.21.15.46:0/1874895138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55566638ca00 0x555666412af0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc025a80 tx=0x7ff5bc003330 comp rx=0 tx=0).stop 2024-05-09T03:19:07.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5cc7f1640 1 -- 172.21.15.46:0/1874895138 reap_dead start 2024-05-09T03:19:07.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 shutdown_connections 2024-05-09T03:19:07.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 >> 172.21.15.46:0/1874895138 conn(0x555666266ac0 msgr2=0x555666371730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:07.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 shutdown_connections 2024-05-09T03:19:07.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.558+0000 7ff5d320bf40 1 -- 172.21.15.46:0/1874895138 wait complete. 2024-05-09T03:19:07.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:07.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:19:07.637 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:19:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.679+0000 7fc724453640 1 Processor -- start 2024-05-09T03:19:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.679+0000 7fc724453640 1 -- start start 2024-05-09T03:19:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc724453640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c070940 0x7fc71c0b7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc724453640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc71c162360 con 0x7fc71c0b6f80 2024-05-09T03:19:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc724453640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc71c162610 con 0x7fc71c0b2e40 2024-05-09T03:19:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc724453640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc71c1628f0 con 0x7fc71c070940 2024-05-09T03:19:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7219ee640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc71c0b6f80 0x7fc71c070230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34862/0 (socket says 172.21.15.46:34862) 2024-05-09T03:19:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7219ee640 1 -- 172.21.15.46:0/3069330523 learned_addr learned my addr 172.21.15.46:0/3069330523 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7229f0640 1 --2- 172.21.15.46:0/3069330523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c070940 0x7fc71c0b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74520152 0 0) 0x7fc71c162360 con 0x7fc71c0b6f80 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc704003520 con 0x7fc71c0b6f80 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968419314 0 0) 0x7fc71c162610 con 0x7fc71c0b2e40 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc71c162360 con 0x7fc71c0b2e40 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7229f0640 1 -- 172.21.15.46:0/3069330523 >> v1:172.21.15.88:6789/0 conn(0x7fc71c0b2e40 legacy=0x7fc71c161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7229f0640 1 -- 172.21.15.46:0/3069330523 >> v1:172.21.15.46:6789/0 conn(0x7fc71c0b6f80 legacy=0x7fc71c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.681+0000 7fc7229f0640 1 -- 172.21.15.46:0/3069330523 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71c0d26b0 con 0x7fc71c070940 2024-05-09T03:19:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.682+0000 7fc7229f0640 1 --2- 172.21.15.46:0/3069330523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c070940 0x7fc71c0b7390 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc7140097d0 tx=0x7fc71402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13e440e08d2df873 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.682+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714037070 con 0x7fc71c070940 2024-05-09T03:19:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.682+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc714032440 con 0x7fc71c070940 2024-05-09T03:19:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.682+0000 7fc7211ed640 1 -- 172.21.15.46:0/3069330523 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7140327f0 con 0x7fc71c070940 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 -- 172.21.15.46:0/3069330523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c070940 msgr2=0x7fc71c0b7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 --2- 172.21.15.46:0/3069330523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c070940 0x7fc71c0b7390 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc7140097d0 tx=0x7fc71402a210 comp rx=0 tx=0).stop 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 -- 172.21.15.46:0/3069330523 shutdown_connections 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 --2- 172.21.15.46:0/3069330523 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c070940 0x7fc71c0b7390 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 -- 172.21.15.46:0/3069330523 >> 172.21.15.46:0/3069330523 conn(0x7fc71c0d05a0 msgr2=0x7fc71c0d09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 -- 172.21.15.46:0/3069330523 shutdown_connections 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.683+0000 7fc724453640 1 -- 172.21.15.46:0/3069330523 wait complete. 2024-05-09T03:19:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.685+0000 7fc724453640 1 Processor -- start 2024-05-09T03:19:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.685+0000 7fc724453640 1 -- start start 2024-05-09T03:19:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc724453640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc71c0b2e40 0x7fc71c14efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc724453640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c0b6f80 0x7fc71c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc724453640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 0x7fc71c0cc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc724453640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c0b33d0 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc724453640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c0b3250 con 0x7fc71c0b2e40 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc724453640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc71c0ce750 con 0x7fc71c0b6f80 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc7229f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 0x7fc71c0cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc7229f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 0x7fc71c0cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40304/0 (socket says 172.21.15.46:40304) 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.686+0000 7fc7229f0640 1 -- 172.21.15.46:0/950919291 learned_addr learned my addr 172.21.15.46:0/950919291 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7219ee640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c0b6f80 0x7fc71c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7221ef640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc71c0b2e40 0x7fc71c14efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7229f0640 1 -- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c0b6f80 msgr2=0x7fc71c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7229f0640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c0b6f80 0x7fc71c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7229f0640 1 -- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc71c0b2e40 msgr2=0x7fc71c14efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7229f0640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc71c0b2e40 0x7fc71c14efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7229f0640 1 -- 172.21.15.46:0/950919291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71c0cabd0 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc7229f0640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 0x7fc71c0cc170 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc71402a250 tx=0x7fc7140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714037040 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71c0cb0d0 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71403b430 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71403b7e0 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.687+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc71c0cb690 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.688+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71c0d1bf0 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.689+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc714044020 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.690+0000 7fc71affd640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7080762d0 0x7fc7080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.690+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8175+0+0 (secure 0 0 0) 0x7fc7140b8150 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.690+0000 7fc7221ef640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7080762d0 0x7fc7080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:07.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.690+0000 7fc7221ef640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7080762d0 0x7fc7080787e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc70c001120 tx=0x7fc70c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.694+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc71c0d1bf0 con 0x7fc71c0cd4c0 2024-05-09T03:19:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:07.926+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fc71c06a270 con 0x7fc71c0cd4c0 2024-05-09T03:19:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.204+0000 7fc71affd640 1 -- 172.21.15.46:0/950919291 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v261) ==== 162+0+0 (secure 0 0 0) 0x7fc714080a40 con 0x7fc71c0cd4c0 2024-05-09T03:19:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.206+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7080762d0 msgr2=0x7fc7080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.206+0000 7fc724453640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7080762d0 0x7fc7080787e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc70c001120 tx=0x7fc70c009040 comp rx=0 tx=0).stop 2024-05-09T03:19:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.206+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 msgr2=0x7fc71c0cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.206+0000 7fc724453640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 0x7fc71c0cc170 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fc71402a250 tx=0x7fc7140046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.206+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 shutdown_connections 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc7080762d0 0x7fc7080787e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc71c0cd4c0 0x7fc71c0cc170 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc71c0b6f80 0x7fc71c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 --2- 172.21.15.46:0/950919291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc71c0b2e40 0x7fc71c14efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 >> 172.21.15.46:0/950919291 conn(0x7fc71c0d05a0 msgr2=0x7fc71c15fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 shutdown_connections 2024-05-09T03:19:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.207+0000 7fc724453640 1 -- 172.21.15.46:0/950919291 wait complete. 2024-05-09T03:19:08.213 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:08.213+0000 7fcea5907640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': End of buffer [buffer:2] 2024-05-09T03:19:08.214 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:08.214+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:08.214 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:08.214+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:19:08.219 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:19:08.213+0000 7fcea5907640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': End of buffer [buffer:2] 2024-05-09T03:19:08.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:08.277 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:08.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.379+0000 7f050a7d5640 1 Processor -- start 2024-05-09T03:19:08.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.379+0000 7f050a7d5640 1 -- start start 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f050a7d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504164a40 0x7f0504160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f050a7d5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f05040710c0 0x7f0504073730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f050a7d5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0504163fa0 con 0x7f0504161380 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f050a7d5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f05040cfcb0 con 0x7f05040710c0 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f050a7d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f05040cff90 con 0x7f0504164a40 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f0503fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504164a40 0x7f0504160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f0508d72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f05040710c0 0x7f0504073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.380+0000 7f04fbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0504161380 0x7f05040709b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:42006/0 (socket says 172.21.15.46:42006) 2024-05-09T03:19:08.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.381+0000 7f04fbfff640 1 -- 172.21.15.46:0/2061357378 learned_addr learned my addr 172.21.15.46:0/2061357378 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.381+0000 7f0503fff640 1 -- 172.21.15.46:0/2061357378 >> v2:172.21.15.88:3300/0 conn(0x7f05040710c0 msgr2=0x7f0504073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.381+0000 7f0503fff640 1 --2- 172.21.15.46:0/2061357378 >> v2:172.21.15.88:3300/0 conn(0x7f05040710c0 0x7f0504073730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.381+0000 7f0503fff640 1 -- 172.21.15.46:0/2061357378 >> v1:172.21.15.88:6789/0 conn(0x7f0504161380 legacy=0x7f05040709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.381+0000 7f0503fff640 1 -- 172.21.15.46:0/2061357378 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0504164240 con 0x7f0504164a40 2024-05-09T03:19:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.381+0000 7f0508d72640 1 --2- 172.21.15.46:0/2061357378 >> v2:172.21.15.88:3300/0 conn(0x7f05040710c0 0x7f0504073730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.382+0000 7f0503fff640 1 --2- 172.21.15.46:0/2061357378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504164a40 0x7f0504160340 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f04f40097a0 tx=0x7f04f402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0a41044e3c27344 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.382+0000 7f05037fe640 1 -- 172.21.15.46:0/2061357378 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4037070 con 0x7f0504164a40 2024-05-09T03:19:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.382+0000 7f05037fe640 1 -- 172.21.15.46:0/2061357378 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f4032440 con 0x7f0504164a40 2024-05-09T03:19:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.382+0000 7f05037fe640 1 -- 172.21.15.46:0/2061357378 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4032810 con 0x7f0504164a40 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 -- 172.21.15.46:0/2061357378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504164a40 msgr2=0x7f0504160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2061357378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504164a40 0x7f0504160340 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f04f40097a0 tx=0x7f04f402a180 comp rx=0 tx=0).stop 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 -- 172.21.15.46:0/2061357378 shutdown_connections 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2061357378 >> v2:172.21.15.88:3300/0 conn(0x7f05040710c0 0x7f0504073730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2061357378 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504164a40 0x7f0504160340 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 -- 172.21.15.46:0/2061357378 >> 172.21.15.46:0/2061357378 conn(0x7f05040b3c40 msgr2=0x7f05040b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 -- 172.21.15.46:0/2061357378 shutdown_connections 2024-05-09T03:19:08.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.383+0000 7f050a7d5640 1 -- 172.21.15.46:0/2061357378 wait complete. 2024-05-09T03:19:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.384+0000 7f050a7d5640 1 Processor -- start 2024-05-09T03:19:08.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.385+0000 7f050a7d5640 1 -- start start 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f050a7d5640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504161380 0x7f05040b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f050a7d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 0x7f0504147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f050a7d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0504159350 0x7f0504146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f050a7d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05040d0270 con 0x7f0504164a40 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f050a7d5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0504161790 con 0x7f0504159350 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f050a7d5640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f05040d03f0 con 0x7f0504161380 2024-05-09T03:19:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 0x7f0504147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 0x7f0504147260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40334/0 (socket says 172.21.15.46:40334) 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 -- 172.21.15.46:0/2591466612 learned_addr learned my addr 172.21.15.46:0/2591466612 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f0508d72640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0504159350 0x7f0504146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f0503fff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504161380 0x7f05040b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 -- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504161380 msgr2=0x7f05040b6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504161380 0x7f05040b6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 -- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0504159350 msgr2=0x7f0504146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.386+0000 7f04fbfff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0504159350 0x7f0504146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f04fbfff640 1 -- 172.21.15.46:0/2591466612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0504146f50 con 0x7f0504164a40 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f0503fff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504161380 0x7f05040b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f04fbfff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 0x7f0504147260 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f05040cf7a0 tx=0x7f04ec00c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec017070 con 0x7f0504164a40 2024-05-09T03:19:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04ec005ce0 con 0x7f0504164a40 2024-05-09T03:19:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ec007310 con 0x7f0504164a40 2024-05-09T03:19:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05040b4bf0 con 0x7f0504164a40 2024-05-09T03:19:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.387+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05040b51b0 con 0x7f0504164a40 2024-05-09T03:19:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.388+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f050415d7c0 con 0x7f0504164a40 2024-05-09T03:19:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.389+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f04ec01a070 con 0x7f0504164a40 2024-05-09T03:19:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.390+0000 7f05017fa640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04e40762d0 0x7f04e40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.390+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7f04ec0970f0 con 0x7f0504164a40 2024-05-09T03:19:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.390+0000 7f0503fff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04e40762d0 0x7f04e40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.390+0000 7f0503fff640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04e40762d0 0x7f04e40787e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f04f4034000 tx=0x7f04f402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.394+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f050415d7c0 con 0x7f0504164a40 2024-05-09T03:19:08.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.630+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05040b7060 con 0x7f0504164a40 2024-05-09T03:19:08.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.630+0000 7f05017fa640 1 -- 172.21.15.46:0/2591466612 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) ==== 95+0+4343 (secure 0 0 0) 0x7f04ec05fa40 con 0x7f0504164a40 2024-05-09T03:19:08.631 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:08.632 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":263,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:08.214484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:08.632 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 263 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04e40762d0 msgr2=0x7f04e40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04e40762d0 0x7f04e40787e0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f04f4034000 tx=0x7f04f402e000 comp rx=0 tx=0).stop 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 msgr2=0x7f0504147260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 0x7f0504147260 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f05040cf7a0 tx=0x7f04ec00c6e0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 shutdown_connections 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0504159350 0x7f0504146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f04e40762d0 0x7f04e40787e0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0504164a40 0x7f0504147260 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 --2- 172.21.15.46:0/2591466612 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0504161380 0x7f05040b6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 >> 172.21.15.46:0/2591466612 conn(0x7f05040b3c40 msgr2=0x7f0504070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 shutdown_connections 2024-05-09T03:19:08.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.633+0000 7f050a7d5640 1 -- 172.21.15.46:0/2591466612 wait complete. 2024-05-09T03:19:08.645 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:08.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.799+0000 7fb4e9649640 1 Processor -- start 2024-05-09T03:19:08.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.800+0000 7fb4e9649640 1 -- start start 2024-05-09T03:19:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e9649640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4e41604a0 0x7fb4e40d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e9649640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40d1170 0x7fb4e40b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e9649640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4e40d1920 con 0x7fb4e40b7d20 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e9649640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb4e40d5130 con 0x7fb4e41604a0 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e9649640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb4e40b8130 con 0x7fb4e40d1170 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4e41604a0 0x7fb4e40d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4e41604a0 0x7fb4e40d0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40336/0 (socket says 172.21.15.46:40336) 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 -- 172.21.15.46:0/2865199099 learned_addr learned my addr 172.21.15.46:0/2865199099 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e27fc640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40d1170 0x7fb4e40b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 -- 172.21.15.46:0/2865199099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40d1170 msgr2=0x7fb4e40b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40d1170 0x7fb4e40b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 -- 172.21.15.46:0/2865199099 >> v1:172.21.15.88:6789/0 conn(0x7fb4e40b7d20 legacy=0x7fb4e40d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e2ffd640 1 -- 172.21.15.46:0/2865199099 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e40d1640 con 0x7fb4e41604a0 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.801+0000 7fb4e27fc640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40d1170 0x7fb4e40b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:08.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.802+0000 7fb4e2ffd640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e41604a0 0x7fb4e40d0c30 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009e50 tx=0x7fb4d402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9095fe0a8c95991 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:08.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.802+0000 7fb4e1ffb640 1 -- 172.21.15.46:0/2865199099 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4036070 con 0x7fb4e41604a0 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.802+0000 7fb4e1ffb640 1 -- 172.21.15.46:0/2865199099 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4d4031600 con 0x7fb4e41604a0 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.802+0000 7fb4e1ffb640 1 -- 172.21.15.46:0/2865199099 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40319d0 con 0x7fb4e41604a0 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 -- 172.21.15.46:0/2865199099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e41604a0 msgr2=0x7fb4e40d0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e41604a0 0x7fb4e40d0c30 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4009e50 tx=0x7fb4d402a660 comp rx=0 tx=0).stop 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 -- 172.21.15.46:0/2865199099 shutdown_connections 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40d1170 0x7fb4e40b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 --2- 172.21.15.46:0/2865199099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e41604a0 0x7fb4e40d0c30 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 -- 172.21.15.46:0/2865199099 >> 172.21.15.46:0/2865199099 conn(0x7fb4e40741d0 msgr2=0x7fb4e40745e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 -- 172.21.15.46:0/2865199099 shutdown_connections 2024-05-09T03:19:08.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.803+0000 7fb4e9649640 1 -- 172.21.15.46:0/2865199099 wait complete. 2024-05-09T03:19:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.805+0000 7fb4e9649640 1 Processor -- start 2024-05-09T03:19:08.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.805+0000 7fb4e9649640 1 -- start start 2024-05-09T03:19:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e9649640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40b7d20 0x7fb4e4153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e9649640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 0x7fb4e40c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e9649640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4e41604a0 0x7fb4e40c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e9649640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e41608b0 con 0x7fb4e40d1170 2024-05-09T03:19:08.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e9649640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e40d4b40 con 0x7fb4e41604a0 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e9649640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb4e40ba330 con 0x7fb4e40b7d20 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 0x7fb4e40c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.806+0000 7fb4e27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 0x7fb4e40c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40352/0 (socket says 172.21.15.46:40352) 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 -- 172.21.15.46:0/1609811149 learned_addr learned my addr 172.21.15.46:0/1609811149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e2ffd640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40b7d20 0x7fb4e4153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e37fe640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4e41604a0 0x7fb4e40c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 -- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40b7d20 msgr2=0x7fb4e4153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40b7d20 0x7fb4e4153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 -- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4e41604a0 msgr2=0x7fb4e40c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4e41604a0 0x7fb4e40c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 -- 172.21.15.46:0/1609811149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e40ca850 con 0x7fb4e40d1170 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e37fe640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4e41604a0 0x7fb4e40c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e2ffd640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40b7d20 0x7fb4e4153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:08.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e27fc640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 0x7fb4e40c9690 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00c820 tx=0x7fb4cc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc015070 con 0x7fb4e40d1170 2024-05-09T03:19:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4cc007a30 con 0x7fb4e40d1170 2024-05-09T03:19:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e40ceb80 con 0x7fb4e40d1170 2024-05-09T03:19:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc007de0 con 0x7fb4e40d1170 2024-05-09T03:19:08.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.807+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e40cf140 con 0x7fb4e40d1170 2024-05-09T03:19:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.808+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e406a270 con 0x7fb4e40d1170 2024-05-09T03:19:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.809+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb4cc018070 con 0x7fb4e40d1170 2024-05-09T03:19:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.810+0000 7fb4db7fe640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4c4076300 0x7fb4c4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.810+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7fb4cc0983c0 con 0x7fb4e40d1170 2024-05-09T03:19:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.810+0000 7fb4e2ffd640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4c4076300 0x7fb4c4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:08.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.811+0000 7fb4e2ffd640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4c4076300 0x7fb4c4078810 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4033000 tx=0x7fb4d4009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:08.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:08.814+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb4cc0612c0 con 0x7fb4e40d1170 2024-05-09T03:19:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.044+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4e40ca5a0 con 0x7fb4e40d1170 2024-05-09T03:19:09.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.045+0000 7fb4db7fe640 1 -- 172.21.15.46:0/1609811149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) ==== 95+0+4343 (secure 0 0 0) 0x7fb4cc060bc0 con 0x7fb4e40d1170 2024-05-09T03:19:09.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:09.046 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":263,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:08.214484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:09.047 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 263 2024-05-09T03:19:09.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4c4076300 msgr2=0x7fb4c4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4c4076300 0x7fb4c4078810 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb4d4033000 tx=0x7fb4d4009000 comp rx=0 tx=0).stop 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 msgr2=0x7fb4e40c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 0x7fb4e40c9690 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00c820 tx=0x7fb4cc00cce0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 shutdown_connections 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4e41604a0 0x7fb4e40c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4c4076300 0x7fb4c4078810 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4e40d1170 0x7fb4e40c9690 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 --2- 172.21.15.46:0/1609811149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb4e40b7d20 0x7fb4e4153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.047+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 >> 172.21.15.46:0/1609811149 conn(0x7fb4e40741d0 msgr2=0x7fb4e415fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.048+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 shutdown_connections 2024-05-09T03:19:09.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.048+0000 7fb4e9649640 1 -- 172.21.15.46:0/1609811149 wait complete. 2024-05-09T03:19:09.060 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:09.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:09.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.175+0000 7f7318572640 1 Processor -- start 2024-05-09T03:19:09.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.175+0000 7f7318572640 1 -- start start 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7318572640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73100b7ed0 0x7f7310163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7318572640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73101617d0 con 0x7f73101613c0 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7318572640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73100d5280 con 0x7f73100b5210 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7318572640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f73100b5620 con 0x7f73100b7ed0 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7315b0d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73101613c0 0x7f73100b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34882/0 (socket says 172.21.15.46:34882) 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7315b0d640 1 -- 172.21.15.46:0/2208836528 learned_addr learned my addr 172.21.15.46:0/2208836528 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7316b0f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73100b7ed0 0x7f7310163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7316b0f640 1 -- 172.21.15.46:0/2208836528 >> v1:172.21.15.88:6789/0 conn(0x7f73100b5210 legacy=0x7f73100d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7316b0f640 1 -- 172.21.15.46:0/2208836528 >> v1:172.21.15.46:6789/0 conn(0x7f73101613c0 legacy=0x7f73100b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f731530c640 1 -- 172.21.15.46:0/2208836528 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3049290327 0 0) 0x7f73101617d0 con 0x7f73101613c0 2024-05-09T03:19:09.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.177+0000 7f7316b0f640 1 -- 172.21.15.46:0/2208836528 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73100d5560 con 0x7f73100b7ed0 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.178+0000 7f7316b0f640 1 --2- 172.21.15.46:0/2208836528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73100b7ed0 0x7f7310163e60 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f7308009d90 tx=0x7f730802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a54fe75dac807049 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.178+0000 7f731530c640 1 -- 172.21.15.46:0/2208836528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7308007740 con 0x7f73100b7ed0 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.178+0000 7f731530c640 1 -- 172.21.15.46:0/2208836528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73080078e0 con 0x7f73100b7ed0 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.178+0000 7f731530c640 1 -- 172.21.15.46:0/2208836528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7308007cb0 con 0x7f73100b7ed0 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 -- 172.21.15.46:0/2208836528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73100b7ed0 msgr2=0x7f7310163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 --2- 172.21.15.46:0/2208836528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73100b7ed0 0x7f7310163e60 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f7308009d90 tx=0x7f730802a6f0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 -- 172.21.15.46:0/2208836528 shutdown_connections 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 --2- 172.21.15.46:0/2208836528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73100b7ed0 0x7f7310163e60 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 -- 172.21.15.46:0/2208836528 >> 172.21.15.46:0/2208836528 conn(0x7f73100d29a0 msgr2=0x7f73100d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 -- 172.21.15.46:0/2208836528 shutdown_connections 2024-05-09T03:19:09.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.179+0000 7f7318572640 1 -- 172.21.15.46:0/2208836528 wait complete. 2024-05-09T03:19:09.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.181+0000 7f7318572640 1 Processor -- start 2024-05-09T03:19:09.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.181+0000 7f7318572640 1 -- start start 2024-05-09T03:19:09.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7318572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73100b5210 0x7f731014ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7318572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73101613c0 0x7f73100cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7318572640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 0x7f731014e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7318572640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7310160ea0 con 0x7f731014e180 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7318572640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73101609a0 con 0x7f73100b5210 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7318572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f73100d4f10 con 0x7f73101613c0 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7316b0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 0x7f731014e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7316b0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 0x7f731014e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40376/0 (socket says 172.21.15.46:40376) 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7316b0f640 1 -- 172.21.15.46:0/1942158210 learned_addr learned my addr 172.21.15.46:0/1942158210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f731630e640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73100b5210 0x7f731014ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7315b0d640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73101613c0 0x7f73100cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7316b0f640 1 -- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73101613c0 msgr2=0x7f73100cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.182+0000 7f7316b0f640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73101613c0 0x7f73100cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7316b0f640 1 -- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73100b5210 msgr2=0x7f731014ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7316b0f640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73100b5210 0x7f731014ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7316b0f640 1 -- 172.21.15.46:0/1942158210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f731014ead0 con 0x7f731014e180 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7315b0d640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73101613c0 0x7f73100cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:09.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7316b0f640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 0x7f731014e590 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7308009eb0 tx=0x7f7308004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:09.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730803f070 con 0x7f731014e180 2024-05-09T03:19:09.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7308031c90 con 0x7f731014e180 2024-05-09T03:19:09.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7308044680 con 0x7f731014e180 2024-05-09T03:19:09.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73100ce290 con 0x7f731014e180 2024-05-09T03:19:09.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.183+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73100ce850 con 0x7f731014e180 2024-05-09T03:19:09.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.184+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73100cdfe0 con 0x7f731014e180 2024-05-09T03:19:09.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.185+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f730803b410 con 0x7f731014e180 2024-05-09T03:19:09.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.185+0000 7f7306ffd640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72f4076300 0x7f72f4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.186+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7f73080b9a20 con 0x7f731014e180 2024-05-09T03:19:09.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.186+0000 7f731630e640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72f4076300 0x7f72f4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.186+0000 7f731630e640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72f4076300 0x7f72f4078810 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f730c000f60 tx=0x7f730c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:09.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.190+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f731014ead0 con 0x7f731014e180 2024-05-09T03:19:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.425+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f731006a270 con 0x7f731014e180 2024-05-09T03:19:09.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.426+0000 7f7306ffd640 1 -- 172.21.15.46:0/1942158210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) ==== 95+0+5166 (secure 0 0 0) 0x7f7308082310 con 0x7f731014e180 2024-05-09T03:19:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:09.427 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":264,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9561,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/120572030","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":120572030},{"type":"v1","addr":"172.21.15.46:6834","nonce":120572030}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":264}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:08.214484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 264 2024-05-09T03:19:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72f4076300 msgr2=0x7f72f4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72f4076300 0x7f72f4078810 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f730c000f60 tx=0x7f730c009040 comp rx=0 tx=0).stop 2024-05-09T03:19:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 msgr2=0x7f731014e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 0x7f731014e590 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f7308009eb0 tx=0x7f7308004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 shutdown_connections 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72f4076300 0x7f72f4078810 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f731014e180 0x7f731014e590 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f73101613c0 0x7f73100cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 --2- 172.21.15.46:0/1942158210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73100b5210 0x7f731014ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 >> 172.21.15.46:0/1942158210 conn(0x7f73100d29a0 msgr2=0x7f73100bada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 shutdown_connections 2024-05-09T03:19:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.428+0000 7f7318572640 1 -- 172.21.15.46:0/1942158210 wait complete. 2024-05-09T03:19:09.440 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate mds0_inotable' led to DAMAGED state 2024-05-09T03:19:09.440 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 400.00000000' 2024-05-09T03:19:09.440 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:09.440 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:09.457 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:19:09.457 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:09.457 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:09.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:09.457 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:19:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:19:09.513 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.667+0000 7ff4792c6640 1 Processor -- start 2024-05-09T03:19:09.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.667+0000 7ff4792c6640 1 -- start start 2024-05-09T03:19:09.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff4792c6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740b7e70 0x7ff4740b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff4792c6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4741605b0 0x7ff4740b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff4792c6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4740d3400 con 0x7ff4740d2e90 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff4792c6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff4740d4a60 con 0x7ff4741605b0 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff4792c6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4740d4d40 con 0x7ff4740b7e70 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff472ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4740d2e90 0x7ff4740b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34884/0 (socket says 172.21.15.46:34884) 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff472ffd640 1 -- 172.21.15.46:0/3345895051 learned_addr learned my addr 172.21.15.46:0/3345895051 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.668+0000 7ff4737fe640 1 --2- 172.21.15.46:0/3345895051 >> v2:172.21.15.88:3300/0 conn(0x7ff4741605b0 0x7ff4740b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff4727fc640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740b7e70 0x7ff4740b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff471ffb640 1 -- 172.21.15.46:0/3345895051 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067429933 0 0) 0x7ff4740d3400 con 0x7ff4740d2e90 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff471ffb640 1 -- 172.21.15.46:0/3345895051 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff45c003520 con 0x7ff4740d2e90 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff471ffb640 1 -- 172.21.15.46:0/3345895051 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2895761744 0 0) 0x7ff45c003520 con 0x7ff4740d2e90 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff4737fe640 1 -- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740b7e70 msgr2=0x7ff4740b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:09.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff4737fe640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740b7e70 0x7ff4740b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff4737fe640 1 -- 172.21.15.46:0/3345895051 >> v1:172.21.15.46:6789/0 conn(0x7ff4740d2e90 legacy=0x7ff4740b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff4737fe640 1 -- 172.21.15.46:0/3345895051 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4740d5020 con 0x7ff4741605b0 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.669+0000 7ff4727fc640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740b7e70 0x7ff4740b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.670+0000 7ff4737fe640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4740b97e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009dc0 tx=0x7ff46c02a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=710012ebb3b5b6ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.670+0000 7ff471ffb640 1 -- 172.21.15.46:0/3345895051 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4640016f0 con 0x7ff4741605b0 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.670+0000 7ff471ffb640 1 -- 172.21.15.46:0/3345895051 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff46c007540 con 0x7ff4741605b0 2024-05-09T03:19:09.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.670+0000 7ff471ffb640 1 -- 172.21.15.46:0/3345895051 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c0078f0 con 0x7ff4741605b0 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 -- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 msgr2=0x7ff4740b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4740b97e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009dc0 tx=0x7ff46c02a610 comp rx=0 tx=0).stop 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 -- 172.21.15.46:0/3345895051 shutdown_connections 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4740b97e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 --2- 172.21.15.46:0/3345895051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740b7e70 0x7ff4740b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 -- 172.21.15.46:0/3345895051 >> 172.21.15.46:0/3345895051 conn(0x7ff47408d390 msgr2=0x7ff47408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:09.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 -- 172.21.15.46:0/3345895051 shutdown_connections 2024-05-09T03:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.671+0000 7ff4792c6640 1 -- 172.21.15.46:0/3345895051 wait complete. 2024-05-09T03:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.673+0000 7ff4792c6640 1 Processor -- start 2024-05-09T03:19:09.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.673+0000 7ff4792c6640 1 -- start start 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4792c6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740d2e90 0x7ff474163450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4792c6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4741f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4792c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 0x7ff474158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4792c6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4740b4d80 con 0x7ff4741f6a10 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4792c6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4740b4c00 con 0x7ff4741605b0 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4792c6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4740b4fc0 con 0x7ff4740d2e90 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 0x7ff474158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 0x7ff474158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40386/0 (socket says 172.21.15.46:40386) 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 -- 172.21.15.46:0/1615734225 learned_addr learned my addr 172.21.15.46:0/1615734225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4727fc640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4741f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff472ffd640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740d2e90 0x7ff474163450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 -- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740d2e90 msgr2=0x7ff474163450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740d2e90 0x7ff474163450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 -- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 msgr2=0x7ff4741f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4741f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.674+0000 7ff4737fe640 1 -- 172.21.15.46:0/1615734225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff474162a90 con 0x7ff4741f6a10 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.675+0000 7ff4737fe640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 0x7ff474158310 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7ff46c02ac00 tx=0x7ff46c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.675+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c03f070 con 0x7ff4741f6a10 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.675+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4741f70d0 con 0x7ff4741f6a10 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.675+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff46c031c40 con 0x7ff4741f6a10 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.675+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c009450 con 0x7ff4741f6a10 2024-05-09T03:19:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.675+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff474161620 con 0x7ff4741f6a10 2024-05-09T03:19:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.676+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff474162d00 con 0x7ff4741f6a10 2024-05-09T03:19:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.677+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff46c003990 con 0x7ff4741f6a10 2024-05-09T03:19:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.677+0000 7ff45b7fe640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff460076300 0x7ff460078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.677+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7ff46c0b93a0 con 0x7ff4741f6a10 2024-05-09T03:19:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.678+0000 7ff472ffd640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff460076300 0x7ff460078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.678+0000 7ff472ffd640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff460076300 0x7ff460078810 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff47406f130 tx=0x7ff464009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:09.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.682+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff46c082230 con 0x7ff4741f6a10 2024-05-09T03:19:09.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:09.910+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff47406a270 con 0x7ff4741f6a10 2024-05-09T03:19:10.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.221+0000 7ff45b7fe640 1 -- 172.21.15.46:0/1615734225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v265) ==== 161+0+0 (secure 0 0 0) 0x7ff46c081b30 con 0x7ff4741f6a10 2024-05-09T03:19:10.222 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:10.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff460076300 msgr2=0x7ff460078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:10.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff460076300 0x7ff460078810 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff47406f130 tx=0x7ff464009040 comp rx=0 tx=0).stop 2024-05-09T03:19:10.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 msgr2=0x7ff474158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:10.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 0x7ff474158310 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7ff46c02ac00 tx=0x7ff46c004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 shutdown_connections 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff460076300 0x7ff460078810 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4741f6a10 0x7ff474158310 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4741605b0 0x7ff4741f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 --2- 172.21.15.46:0/1615734225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4740d2e90 0x7ff474163450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 >> 172.21.15.46:0/1615734225 conn(0x7ff47408d390 msgr2=0x7ff4740b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 shutdown_connections 2024-05-09T03:19:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.223+0000 7ff4792c6640 1 -- 172.21.15.46:0/1615734225 wait complete. 2024-05-09T03:19:10.236 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:10.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.353+0000 7f8c520e0640 1 Processor -- start 2024-05-09T03:19:10.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.353+0000 7f8c520e0640 1 -- start start 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c520e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4c0ceba0 0x7f8c4c0cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c520e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8c4c0cf4f0 0x7f8c4c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c520e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c09d470 con 0x7f8c4c0d3730 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c520e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8c4c09cfe0 con 0x7f8c4c0ceba0 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c520e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8c4c0b94a0 con 0x7f8c4c0cf4f0 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4c0ceba0 0x7f8c4c0cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c4c0d3730 0x7f8c4c06e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34900/0 (socket says 172.21.15.46:34900) 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4b7fe640 1 -- 172.21.15.46:0/181045696 learned_addr learned my addr 172.21.15.46:0/181045696 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/181045696 >> v2:172.21.15.88:3300/0 conn(0x7f8c4c0cf4f0 0x7f8c4c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:10.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4affd640 1 -- 172.21.15.46:0/181045696 >> v2:172.21.15.88:3300/0 conn(0x7f8c4c0cf4f0 msgr2=0x7f8c4c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:10.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4affd640 1 --2- 172.21.15.46:0/181045696 >> v2:172.21.15.88:3300/0 conn(0x7f8c4c0cf4f0 0x7f8c4c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4affd640 1 -- 172.21.15.46:0/181045696 >> v1:172.21.15.46:6789/0 conn(0x7f8c4c0d3730 legacy=0x7f8c4c06e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:10.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.355+0000 7f8c4affd640 1 -- 172.21.15.46:0/181045696 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c0d11b0 con 0x7f8c4c0ceba0 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.356+0000 7f8c4affd640 1 --2- 172.21.15.46:0/181045696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0ceba0 0x7f8c4c0cefb0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c074a20 tx=0x7f8c4002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43edd3f0d634a14b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.356+0000 7f8c4a7fc640 1 -- 172.21.15.46:0/181045696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c40037070 con 0x7f8c4c0ceba0 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.356+0000 7f8c4a7fc640 1 -- 172.21.15.46:0/181045696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c40032440 con 0x7f8c4c0ceba0 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.356+0000 7f8c4a7fc640 1 -- 172.21.15.46:0/181045696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4002ae60 con 0x7f8c4c0ceba0 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 -- 172.21.15.46:0/181045696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0ceba0 msgr2=0x7f8c4c0cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 --2- 172.21.15.46:0/181045696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0ceba0 0x7f8c4c0cefb0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c074a20 tx=0x7f8c4002a110 comp rx=0 tx=0).stop 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 -- 172.21.15.46:0/181045696 shutdown_connections 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 --2- 172.21.15.46:0/181045696 >> v2:172.21.15.88:3300/0 conn(0x7f8c4c0cf4f0 0x7f8c4c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 --2- 172.21.15.46:0/181045696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0ceba0 0x7f8c4c0cefb0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 -- 172.21.15.46:0/181045696 >> 172.21.15.46:0/181045696 conn(0x7f8c4c0d05f0 msgr2=0x7f8c4c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:10.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 -- 172.21.15.46:0/181045696 shutdown_connections 2024-05-09T03:19:10.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.357+0000 7f8c520e0640 1 -- 172.21.15.46:0/181045696 wait complete. 2024-05-09T03:19:10.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.359+0000 7f8c520e0640 1 Processor -- start 2024-05-09T03:19:10.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.359+0000 7f8c520e0640 1 -- start start 2024-05-09T03:19:10.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c520e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c4c0ceba0 0x7f8c4c0ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:10.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c520e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c4c0cf4f0 0x7f8c4c0cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c520e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 0x7f8c4c15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c520e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c0b9160 con 0x7f8c4c0d3730 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c520e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c0d1490 con 0x7f8c4c0cf4f0 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c520e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c08cd60 con 0x7f8c4c0ceba0 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 0x7f8c4c15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 0x7f8c4c15cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40408/0 (socket says 172.21.15.46:40408) 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.360+0000 7f8c4bfff640 1 -- 172.21.15.46:0/1508604512 learned_addr learned my addr 172.21.15.46:0/1508604512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4affd640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c4c0cf4f0 0x7f8c4c0cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4b7fe640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c4c0ceba0 0x7f8c4c0ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4bfff640 1 -- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c4c0ceba0 msgr2=0x7f8c4c0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c4c0ceba0 0x7f8c4c0ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4bfff640 1 -- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c4c0cf4f0 msgr2=0x7f8c4c0cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c4c0cf4f0 0x7f8c4c0cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4bfff640 1 -- 172.21.15.46:0/1508604512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c0cc570 con 0x7f8c4c0d3730 2024-05-09T03:19:10.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 0x7f8c4c15cc30 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8c44007d80 tx=0x7f8c4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:10.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44014420 con 0x7f8c4c0d3730 2024-05-09T03:19:10.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c44019070 con 0x7f8c4c0d3730 2024-05-09T03:19:10.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c0cbcd0 con 0x7f8c4c0d3730 2024-05-09T03:19:10.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c44014800 con 0x7f8c4c0d3730 2024-05-09T03:19:10.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.361+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c0cc290 con 0x7f8c4c0d3730 2024-05-09T03:19:10.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.362+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c0b65b0 con 0x7f8c4c0d3730 2024-05-09T03:19:10.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.363+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c440149a0 con 0x7f8c4c0d3730 2024-05-09T03:19:10.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.363+0000 7f8c33fff640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c38076300 0x7f8c38078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:10.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.364+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7f8c44098820 con 0x7f8c4c0d3730 2024-05-09T03:19:10.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.364+0000 7f8c4b7fe640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c38076300 0x7f8c38078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:10.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.364+0000 7f8c4b7fe640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c38076300 0x7f8c38078810 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c000f60 tx=0x7f8c3c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:10.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.368+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8c4c0b65b0 con 0x7f8c4c0d3730 2024-05-09T03:19:10.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:10.642+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f8c4c06a270 con 0x7f8c4c0d3730 2024-05-09T03:19:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.230+0000 7f8c33fff640 1 -- 172.21.15.46:0/1508604512 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v266) ==== 101+0+0 (secure 0 0 0) 0x7f8c440610e0 con 0x7f8c4c0d3730 2024-05-09T03:19:11.231 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:19:11.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c38076300 msgr2=0x7f8c38078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c38076300 0x7f8c38078810 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c000f60 tx=0x7f8c3c009040 comp rx=0 tx=0).stop 2024-05-09T03:19:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 msgr2=0x7f8c4c15cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 0x7f8c4c15cc30 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f8c44007d80 tx=0x7f8c4400e4b0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 shutdown_connections 2024-05-09T03:19:11.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c38076300 0x7f8c38078810 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d3730 0x7f8c4c15cc30 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c4c0cf4f0 0x7f8c4c0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 --2- 172.21.15.46:0/1508604512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c4c0ceba0 0x7f8c4c0ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.233+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 >> 172.21.15.46:0/1508604512 conn(0x7f8c4c0d05f0 msgr2=0x7f8c4c0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.234+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 shutdown_connections 2024-05-09T03:19:11.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.234+0000 7f8c520e0640 1 -- 172.21.15.46:0/1508604512 wait complete. 2024-05-09T03:19:11.249 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:11.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.309+0000 7fc4acd1ff40 1 Processor -- start 2024-05-09T03:19:11.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.309+0000 7fc4acd1ff40 1 -- start start 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4acd1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a172db0 0x55602a170430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4acd1ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 0x55602a16fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4acd1ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21d0f0 0x55602a1fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4acd1ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55602a185cb0 con 0x55602a21b780 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4acd1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55602a0ecf50 con 0x55602a172db0 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4acd1ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55602a185380 con 0x55602a21d0f0 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 0x55602a16fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 0x55602a16fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40428/0 (socket says 172.21.15.46:40428) 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc49ffff640 1 -- 172.21.15.46:0/511951564 learned_addr learned my addr 172.21.15.46:0/511951564 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:11.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a172db0 0x55602a170430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.310+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21d0f0 0x55602a1fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc49ffff640 1 -- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21d0f0 msgr2=0x55602a1fd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc49ffff640 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21d0f0 0x55602a1fd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc49ffff640 1 -- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a172db0 msgr2=0x55602a170430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc49ffff640 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a172db0 0x55602a170430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc49ffff640 1 -- 172.21.15.46:0/511951564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55602a170970 con 0x55602a21b780 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc49ffff640 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 0x55602a16fcc0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc490001d20 tx=0x7fc49000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52ca4a9972d0c1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc4a57e5640 1 -- 172.21.15.46:0/511951564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49000bd20 con 0x55602a21b780 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc4a57e5640 1 -- 172.21.15.46:0/511951564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc49000f400 con 0x55602a21b780 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.311+0000 7fc4a57e5640 1 -- 172.21.15.46:0/511951564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49000f7f0 con 0x55602a21b780 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/511951564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 msgr2=0x55602a16fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 0x55602a16fcc0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc490001d20 tx=0x7fc49000b5b0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/511951564 shutdown_connections 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21d0f0 0x55602a1fd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a21b780 0x55602a16fcc0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/511951564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a172db0 0x55602a170430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/511951564 >> 172.21.15.46:0/511951564 conn(0x55602a068ac0 msgr2=0x55602a1754c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/511951564 shutdown_connections 2024-05-09T03:19:11.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.312+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/511951564 wait complete. 2024-05-09T03:19:11.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.313+0000 7fc4acd1ff40 1 Processor -- start 2024-05-09T03:19:11.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.313+0000 7fc4acd1ff40 1 -- start start 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4acd1ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 0x55602a170260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4acd1ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21b780 0x55602a1899c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4acd1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a1fc7f0 0x55602a143250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4acd1ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55602a1fe890 con 0x55602a172db0 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4acd1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55602a211700 con 0x55602a1fc7f0 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4acd1ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55602a1707a0 con 0x55602a21b780 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4a67e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 0x55602a170260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4a67e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 0x55602a170260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40434/0 (socket says 172.21.15.46:40434) 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 learned_addr learned my addr 172.21.15.46:0/3525313420 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:11.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a1fc7f0 0x55602a143250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.314+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21b780 0x55602a1899c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21b780 msgr2=0x55602a1899c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55602a21b780 0x55602a1899c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a1fc7f0 msgr2=0x55602a143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a1fc7f0 0x55602a143250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55602a187f80 con 0x55602a172db0 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55602a1fc7f0 0x55602a143250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 0x55602a170260 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc498009ef0 tx=0x7fc498008400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc49f7fe640 1 -- 172.21.15.46:0/3525313420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498016630 con 0x55602a172db0 2024-05-09T03:19:11.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc49f7fe640 1 -- 172.21.15.46:0/3525313420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4980167d0 con 0x55602a172db0 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc49f7fe640 1 -- 172.21.15.46:0/3525313420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498016ba0 con 0x55602a172db0 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55602a144df0 con 0x55602a172db0 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.315+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55602a188190 con 0x55602a172db0 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.317+0000 7fc49f7fe640 1 -- 172.21.15.46:0/3525313420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc4980153e0 con 0x55602a172db0 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.317+0000 7fc49f7fe640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc488074cb0 0x7fc4880771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.317+0000 7fc49f7fe640 1 -- 172.21.15.46:0/3525313420 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7fc498096890 con 0x55602a172db0 2024-05-09T03:19:11.321 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.318+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55602a146fa0 0x55602a1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.318+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a143790 con 0x55602a146fa0 2024-05-09T03:19:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.318+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc488074cb0 0x7fc4880771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.318+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55602a146fa0 0x55602a1473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.318+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55602a146fa0 0x55602a1473b0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.318+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc488074cb0 0x7fc4880771c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x55602a16f9d0 tx=0x7fc490009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.320+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v113'877 uv851 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.320+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2bdb20 con 0x55602a146fa0 2024-05-09T03:19:11.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.322+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v113'878 uv878 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.322+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bdec0 con 0x55602a146fa0 2024-05-09T03:19:11.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.323+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v113'879 uv879 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.325 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:11.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.323+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bf270 con 0x55602a146fa0 2024-05-09T03:19:11.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.325+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v113'880 uv880 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.325+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bf610 con 0x55602a146fa0 2024-05-09T03:19:11.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.326+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v113'881 uv881 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.326+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2be260 con 0x55602a146fa0 2024-05-09T03:19:11.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.328+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v113'882 uv882 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.328+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2be670 con 0x55602a146fa0 2024-05-09T03:19:11.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.330+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v113'883 uv857 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.330+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2bea60 con 0x55602a146fa0 2024-05-09T03:19:11.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.331+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v113'884 uv884 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.331+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea60 con 0x55602a146fa0 2024-05-09T03:19:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.333+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v113'885 uv885 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.334 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:11.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.333+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.335+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v113'886 uv886 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.335+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.336+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v113'887 uv887 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.336+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.338+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v113'888 uv888 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.338+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.340+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v113'889 uv864 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.340+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.341+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v113'890 uv890 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.341+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.343+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v113'891 uv891 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.344 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.343+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.344+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v113'892 uv892 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.344+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.346+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v113'893 uv893 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.346+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.348+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v113'894 uv894 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.348+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.349+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v113'895 uv895 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.349+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.351+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v113'896 uv870 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.351+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.353+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v113'897 uv897 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.353+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.355+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v113'898 uv898 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.355 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:11.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.355+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.356+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v113'899 uv899 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.356+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.358+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v113'900 uv900 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.358+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.359+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v113'901 uv901 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.359+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.361+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v113'902 uv876 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.361+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.362+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v113'903 uv903 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.362+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.364+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v113'904 uv904 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.365 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.364+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.365+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v113'905 uv905 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.365+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.367+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v113'906 uv906 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.367+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a146fa0 2024-05-09T03:19:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.368+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v113'907 uv907 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.368+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55602a2bee10 0x55602a2c1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.368+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2bea10 con 0x55602a2bee10 2024-05-09T03:19:11.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.369+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55602a2bee10 0x55602a2c1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.369+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55602a2bee10 0x55602a2c1ed0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.371+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v113'633 uv616 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.371+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c2410 con 0x55602a2bee10 2024-05-09T03:19:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.372+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v113'634 uv634 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.372+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c27b0 con 0x55602a2bee10 2024-05-09T03:19:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.373+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v113'635 uv635 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.374 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:11.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.373+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3000 con 0x55602a2bee10 2024-05-09T03:19:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.375+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v113'636 uv636 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.375+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c33a0 con 0x55602a2bee10 2024-05-09T03:19:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.376+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v113'637 uv637 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.376+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3740 con 0x55602a2bee10 2024-05-09T03:19:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.377+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v113'638 uv621 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.377+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c3ae0 con 0x55602a2bee10 2024-05-09T03:19:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.378+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v113'639 uv639 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.378+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.381 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.381+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v113'640 uv640 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.381+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.382+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v113'641 uv641 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.382+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.384+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v113'642 uv642 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.384+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.385+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v113'643 uv626 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.385+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.386+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v113'644 uv644 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.386+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.387+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v113'645 uv645 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.388 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.387+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c78e0 con 0x55602a2bee10 2024-05-09T03:19:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.389+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v113'646 uv646 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.389+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c78e0 con 0x55602a2bee10 2024-05-09T03:19:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.390+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v113'647 uv647 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.390+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.391+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v113'648 uv632 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.391+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.392+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v113'649 uv649 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.393+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.394+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v113'650 uv650 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.394 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:11.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.394+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c4e90 con 0x55602a2bee10 2024-05-09T03:19:11.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.395+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v113'651 uv651 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.395+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c4e90 con 0x55602a2bee10 2024-05-09T03:19:11.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.396+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v113'652 uv652 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.396+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a2bee10 2024-05-09T03:19:11.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.397+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v113'653 uv653 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2bee10 2024-05-09T03:19:11.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.397+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a146fa0 2024-05-09T03:19:11.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.399+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v113'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.399+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a146fa0 2024-05-09T03:19:11.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.400+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v113'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.400+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a146fa0 2024-05-09T03:19:11.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.401+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v113'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.402 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:11.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.401+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c4e90 con 0x55602a146fa0 2024-05-09T03:19:11.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.402+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v113'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.402+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c4e90 con 0x55602a146fa0 2024-05-09T03:19:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.403+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v113'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.403+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c3e80 con 0x55602a146fa0 2024-05-09T03:19:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.404+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v113'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.404+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55602a2c4390 0x55602a2c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.404+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c6e40 con 0x55602a2c4390 2024-05-09T03:19:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.404+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55602a2c4390 0x55602a2c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.404+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55602a2c4390 0x55602a2c67b0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.406+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v113'128 uv127 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.406+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c7350 con 0x55602a2c4390 2024-05-09T03:19:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.407+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v113'129 uv129 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.407+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c7810 con 0x55602a2c4390 2024-05-09T03:19:11.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.408+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v113'130 uv130 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.409 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:11.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.408+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c2c30 con 0x55602a2c4390 2024-05-09T03:19:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.409+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v113'131 uv131 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.409+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c8170 con 0x55602a146fa0 2024-05-09T03:19:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.410+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v113'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.410+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c8170 con 0x55602a146fa0 2024-05-09T03:19:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.412+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v113'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.412+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c8170 con 0x55602a146fa0 2024-05-09T03:19:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.413+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v113'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.414 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.413+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c9180 con 0x55602a146fa0 2024-05-09T03:19:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.414+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v113'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.414+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c9180 con 0x55602a146fa0 2024-05-09T03:19:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.415+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v113'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.415+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c8170 con 0x55602a146fa0 2024-05-09T03:19:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.416+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v113'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.416+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55602a2c8a40 0x55602a2cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.417+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb3a0 con 0x55602a2c8a40 2024-05-09T03:19:11.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.417+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55602a2c8a40 0x55602a2cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.417+0000 7fc4a6fe8640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55602a2c8a40 0x55602a2cae60 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.419+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v113'316 uv310 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.419+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2cb3a0 con 0x55602a2c8a40 2024-05-09T03:19:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.420+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v113'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.420+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.421 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.421+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v113'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.421+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cc7a0 con 0x55602a2c8a40 2024-05-09T03:19:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.422+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v113'319 uv319 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.422+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cc7a0 con 0x55602a2c8a40 2024-05-09T03:19:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.423+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v113'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.423+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.424+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v113'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.424+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.425+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v113'322 uv315 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.426+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.427+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v113'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.427+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.428 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.428+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v113'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.428+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.429+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v113'325 uv325 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.429+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cb790 con 0x55602a2c8a40 2024-05-09T03:19:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.430+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v113'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.430+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55602a2cbe10 0x55602a2ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.430+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2ce8e0 con 0x55602a2cbe10 2024-05-09T03:19:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.430+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55602a2cbe10 0x55602a2ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.431+0000 7fc4a67e7640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55602a2cbe10 0x55602a2ce230 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.432+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v113'143 uv142 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2cbe10 2024-05-09T03:19:11.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.432+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a2cbe10 2024-05-09T03:19:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.433+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v113'144 uv144 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2cbe10 2024-05-09T03:19:11.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.433+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a2cbe10 2024-05-09T03:19:11.435 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.435+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v113'145 uv145 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2cbe10 2024-05-09T03:19:11.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.435+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a2cbe10 2024-05-09T03:19:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.436+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v113'146 uv146 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2cbe10 2024-05-09T03:19:11.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.436+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a2cbe10 2024-05-09T03:19:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.437+0000 7fc4a67e7640 1 -- 172.21.15.46:0/3525313420 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v113'147 uv147 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49805b3c0 con 0x55602a2cbe10 2024-05-09T03:19:11.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.437+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a146fa0 2024-05-09T03:19:11.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.439+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v113'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.439+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a146fa0 2024-05-09T03:19:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.440+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v113'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.440+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a146fa0 2024-05-09T03:19:11.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.441+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v113'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.441 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.441+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cfcc0 con 0x55602a146fa0 2024-05-09T03:19:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.442+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v113'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.442+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cfcc0 con 0x55602a146fa0 2024-05-09T03:19:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.443+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v113'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.443+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a146fa0 2024-05-09T03:19:11.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.444+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v113'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a146fa0 2024-05-09T03:19:11.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.444+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cecb0 con 0x55602a2c4390 2024-05-09T03:19:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.445+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v113'284 uv283 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.445+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2cf0a0 con 0x55602a2c4390 2024-05-09T03:19:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.446+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v113'285 uv285 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.446+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2cf490 con 0x55602a2c4390 2024-05-09T03:19:11.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.447+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v113'286 uv286 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.448 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:11.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.447+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c7cf0 con 0x55602a2c4390 2024-05-09T03:19:11.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.448+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v113'287 uv287 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.448+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c7cf0 con 0x55602a2c4390 2024-05-09T03:19:11.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.449+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v113'288 uv288 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.449+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c7cf0 con 0x55602a2c4390 2024-05-09T03:19:11.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.450+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v113'289 uv282 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.450+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2c7cf0 con 0x55602a2c4390 2024-05-09T03:19:11.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.451+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v113'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.451+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c7cf0 con 0x55602a2c4390 2024-05-09T03:19:11.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.452+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v113'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.453 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:11.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.452+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2c7cf0 con 0x55602a2c4390 2024-05-09T03:19:11.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.453+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v113'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.453+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2ab0 con 0x55602a2c4390 2024-05-09T03:19:11.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.454+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v113'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.454+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55602a2d0660 0x55602a2d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.454+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2d0660 2024-05-09T03:19:11.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.454+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55602a2d0660 0x55602a2d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.455+0000 7fc49ffff640 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55602a2d0660 0x55602a2d2a80 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.456+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v113'124 uv123 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2d0660 2024-05-09T03:19:11.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.456+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2d0660 2024-05-09T03:19:11.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.457+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v113'125 uv125 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2d0660 2024-05-09T03:19:11.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.457+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2d0660 2024-05-09T03:19:11.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.458+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v113'126 uv126 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2d0660 2024-05-09T03:19:11.459 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:11.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.458+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2d0660 2024-05-09T03:19:11.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.459+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v113'127 uv127 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2d0660 2024-05-09T03:19:11.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.459+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c4390 2024-05-09T03:19:11.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.461+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v113'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.461+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c4390 2024-05-09T03:19:11.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.462+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v113'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.462+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c4390 2024-05-09T03:19:11.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.463+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v113'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.464 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:11.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.463+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d3fd0 con 0x55602a2c4390 2024-05-09T03:19:11.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.464+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v113'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.464+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d3fd0 con 0x55602a2c4390 2024-05-09T03:19:11.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.466+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v113'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.466+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c4390 2024-05-09T03:19:11.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.467+0000 7fc49ffff640 1 -- 172.21.15.46:0/3525313420 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v113'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc49000f990 con 0x55602a2c4390 2024-05-09T03:19:11.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.467+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.468+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v113'145 uv144 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.468+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.469+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v113'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.469+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.470+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v113'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.471 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:11.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.470+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.471+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v113'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.472+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.473+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v113'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.473+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.474+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v113'351 uv343 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.474+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.475+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v113'352 uv352 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.475+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.477 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.476+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v113'353 uv353 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.476+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.478+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v113'354 uv354 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.478+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.479+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v113'355 uv355 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.479+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.480+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v113'356 uv350 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.480+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.481+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v113'357 uv357 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.481+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.482 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.482+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v113'358 uv358 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.482+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d4010 con 0x55602a2c8a40 2024-05-09T03:19:11.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.483+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v113'359 uv359 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.483+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d3fd0 con 0x55602a2c8a40 2024-05-09T03:19:11.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.484+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v113'360 uv360 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.484+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d2fc0 con 0x55602a2c8a40 2024-05-09T03:19:11.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.485+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v113'361 uv361 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.485+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55602a2d3470 con 0x55602a2c8a40 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v113'362 uv362 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc4a00518b0 con 0x55602a2c8a40 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55602a146fa0 msgr2=0x55602a1473b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55602a146fa0 0x55602a1473b0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55602a2c4390 msgr2=0x55602a2c67b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55602a2c4390 0x55602a2c67b0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55602a2d0660 msgr2=0x55602a2d2a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55602a2d0660 0x55602a2d2a80 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55602a2bee10 msgr2=0x55602a2c1ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55602a2bee10 0x55602a2c1ed0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55602a2c8a40 msgr2=0x55602a2cae60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55602a2c8a40 0x55602a2cae60 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55602a2cbe10 msgr2=0x55602a2ce230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55602a2cbe10 0x55602a2ce230 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 reap_dead start 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 reap_dead start 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4a6fe8640 1 -- 172.21.15.46:0/3525313420 reap_dead start 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc488074cb0 msgr2=0x7fc4880771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc488074cb0 0x7fc4880771c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x55602a16f9d0 tx=0x7fc490009000 comp rx=0 tx=0).stop 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 msgr2=0x55602a170260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 0x55602a170260 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fc498009ef0 tx=0x7fc498008400 comp rx=0 tx=0).stop 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 shutdown_connections 2024-05-09T03:19:11.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc488074cb0 0x7fc4880771c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 --2- 172.21.15.46:0/3525313420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55602a172db0 0x55602a170260 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 >> 172.21.15.46:0/3525313420 conn(0x55602a068ac0 msgr2=0x55602a218f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 shutdown_connections 2024-05-09T03:19:11.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.486+0000 7fc4acd1ff40 1 -- 172.21.15.46:0/3525313420 wait complete. 2024-05-09T03:19:11.491 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 400.00000000 0 2024-05-09T03:19:11.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.547+0000 7f206392af40 1 Processor -- start 2024-05-09T03:19:11.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.547+0000 7f206392af40 1 -- start start 2024-05-09T03:19:11.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f206392af40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287aca10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f206392af40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287ac1c010 0x56287aca1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f206392af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287aca64e0 0x56287aca1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f206392af40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56287ab7cf50 con 0x56287ac057b0 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f206392af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56287ab80f90 con 0x56287aca64e0 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f206392af40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56287ac1a4d0 con 0x56287ac1c010 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287aca10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287aca10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40442/0 (socket says 172.21.15.46:40442) 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 -- 172.21.15.46:0/2191107223 learned_addr learned my addr 172.21.15.46:0/2191107223 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205d18f640 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287aca64e0 0x56287aca1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c18d640 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287ac1c010 0x56287aca1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 -- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287ac1c010 msgr2=0x56287aca1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287ac1c010 0x56287aca1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 -- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287aca64e0 msgr2=0x56287aca1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287aca64e0 0x56287aca1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205c98e640 1 -- 172.21.15.46:0/2191107223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56287ac18060 con 0x56287ac057b0 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.548+0000 7f205d18f640 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287aca64e0 0x56287aca1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f205c98e640 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287aca10d0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f204c005fd0 tx=0x7f204c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896f50d5b08cf6bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f205b18b640 1 -- 172.21.15.46:0/2191107223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c069070 con 0x56287ac057b0 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f205b18b640 1 -- 172.21.15.46:0/2191107223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204c051890 con 0x56287ac057b0 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f205b18b640 1 -- 172.21.15.46:0/2191107223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c055070 con 0x56287ac057b0 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 -- 172.21.15.46:0/2191107223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 msgr2=0x56287aca10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287aca10d0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f204c005fd0 tx=0x7f204c051000 comp rx=0 tx=0).stop 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 -- 172.21.15.46:0/2191107223 shutdown_connections 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287aca64e0 0x56287aca1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287ac1c010 0x56287aca1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 --2- 172.21.15.46:0/2191107223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287aca10d0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 -- 172.21.15.46:0/2191107223 >> 172.21.15.46:0/2191107223 conn(0x56287aaf8ac0 msgr2=0x56287ac1da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 -- 172.21.15.46:0/2191107223 shutdown_connections 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.549+0000 7f206392af40 1 -- 172.21.15.46:0/2191107223 wait complete. 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.550+0000 7f206392af40 1 Processor -- start 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.550+0000 7f206392af40 1 -- start start 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.551+0000 7f206392af40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287ac03de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287ac1c010 0x56287aca8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287aca64e0 0x56287aca9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56287ac19270 con 0x56287ac057b0 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56287abd4c20 con 0x56287ac1c010 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56287ad3de50 con 0x56287aca64e0 2024-05-09T03:19:11.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287ac03de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287ac03de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40444/0 (socket says 172.21.15.46:40444) 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 -- 172.21.15.46:0/2001376495 learned_addr learned my addr 172.21.15.46:0/2001376495 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205d18f640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287aca64e0 0x56287aca9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c18d640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287ac1c010 0x56287aca8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 -- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287aca64e0 msgr2=0x56287aca9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287aca64e0 0x56287aca9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 -- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287ac1c010 msgr2=0x56287aca8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287ac1c010 0x56287aca8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 -- 172.21.15.46:0/2001376495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56287aca9660 con 0x56287ac057b0 2024-05-09T03:19:11.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c18d640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56287ac1c010 0x56287aca8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205d18f640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56287aca64e0 0x56287aca9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205c98e640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287ac03de0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f204c025a70 tx=0x7f204c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205a189640 1 -- 172.21.15.46:0/2001376495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c055050 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205a189640 1 -- 172.21.15.46:0/2001376495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204c051bd0 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56287ad3e3b0 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f205a189640 1 -- 172.21.15.46:0/2001376495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c069040 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.552+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56287abd4960 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.554+0000 7f205a189640 1 -- 172.21.15.46:0/2001376495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f204c058070 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.554+0000 7f205a189640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2048074cb0 0x7f20480771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f205a189640 1 -- 172.21.15.46:0/2001376495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7f204c0db970 con 0x56287ac057b0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f206392af40 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56287ac04320 0x56287aca76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x56287aca7c10 con 0x56287ac04320 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f205c18d640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2048074cb0 0x7f20480771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f205d18f640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56287ac04320 0x56287aca76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f205c18d640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2048074cb0 0x7f20480771c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x56287ac1efb0 tx=0x7f2050001590 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.555+0000 7f205d18f640 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56287ac04320 0x56287aca76d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f205d18f640 1 -- 172.21.15.46:0/2001376495 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 400.00000000 [truncate 0] v113'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f205400e430 con 0x56287ac04320 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56287ac04320 msgr2=0x56287aca76d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56287ac04320 0x56287aca76d0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2048074cb0 msgr2=0x7f20480771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2048074cb0 0x7f20480771c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x56287ac1efb0 tx=0x7f2050001590 comp rx=0 tx=0).stop 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 msgr2=0x56287ac03de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 --2- 172.21.15.46:0/2001376495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56287ac057b0 0x56287ac03de0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f204c025a70 tx=0x7f204c003330 comp rx=0 tx=0).stop 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f205d18f640 1 -- 172.21.15.46:0/2001376495 reap_dead start 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 shutdown_connections 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 >> 172.21.15.46:0/2001376495 conn(0x56287aaf8ac0 msgr2=0x56287ac1da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 shutdown_connections 2024-05-09T03:19:11.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.557+0000 7f206392af40 1 -- 172.21.15.46:0/2001376495 wait complete. 2024-05-09T03:19:11.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.682+0000 7f7ea195e640 1 Processor -- start 2024-05-09T03:19:11.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.682+0000 7f7ea195e640 1 -- start start 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7ea195e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e9c0b7e70 0x7f7e9c0b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7ea195e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7ea195e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e9c0d3400 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7ea195e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7e9c0d4a60 con 0x7f7e9c0b7e70 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7ea195e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9c0d4d40 con 0x7f7e9c1605b0 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7e9c0d2e90 0x7f7e9c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34916/0 (socket says 172.21.15.46:34916) 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9affd640 1 -- 172.21.15.46:0/2466820799 learned_addr learned my addr 172.21.15.46:0/2466820799 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:11.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9a7fc640 1 --2- 172.21.15.46:0/2466820799 >> v2:172.21.15.88:3300/0 conn(0x7f7e9c0b7e70 0x7f7e9c0b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9b7fe640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e99ffb640 1 -- 172.21.15.46:0/2466820799 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200245719 0 0) 0x7f7e9c0d3400 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e99ffb640 1 -- 172.21.15.46:0/2466820799 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e88003520 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9a7fc640 1 -- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 msgr2=0x7f7e9c0b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9a7fc640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0b97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9a7fc640 1 -- 172.21.15.46:0/2466820799 >> v1:172.21.15.46:6789/0 conn(0x7f7e9c0d2e90 legacy=0x7f7e9c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9a7fc640 1 -- 172.21.15.46:0/2466820799 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9c0d5020 con 0x7f7e9c0b7e70 2024-05-09T03:19:11.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.684+0000 7f7e9b7fe640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.685+0000 7f7e9a7fc640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c0b7e70 0x7f7e9c0b92a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7e840097a0 tx=0x7f7e8402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=873ec58ab73d4fce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.685+0000 7f7e99ffb640 1 -- 172.21.15.46:0/2466820799 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84037070 con 0x7f7e9c0b7e70 2024-05-09T03:19:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.685+0000 7f7e99ffb640 1 -- 172.21.15.46:0/2466820799 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e84032440 con 0x7f7e9c0b7e70 2024-05-09T03:19:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.685+0000 7f7e99ffb640 1 -- 172.21.15.46:0/2466820799 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e84032810 con 0x7f7e9c0b7e70 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 -- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c0b7e70 msgr2=0x7f7e9c0b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c0b7e70 0x7f7e9c0b92a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7e840097a0 tx=0x7f7e8402a180 comp rx=0 tx=0).stop 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 -- 172.21.15.46:0/2466820799 shutdown_connections 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0b97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 --2- 172.21.15.46:0/2466820799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c0b7e70 0x7f7e9c0b92a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 -- 172.21.15.46:0/2466820799 >> 172.21.15.46:0/2466820799 conn(0x7f7e9c08d390 msgr2=0x7f7e9c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 -- 172.21.15.46:0/2466820799 shutdown_connections 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.686+0000 7f7ea195e640 1 -- 172.21.15.46:0/2466820799 wait complete. 2024-05-09T03:19:11.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.688+0000 7f7ea195e640 1 Processor -- start 2024-05-09T03:19:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.688+0000 7f7ea195e640 1 -- start start 2024-05-09T03:19:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7ea195e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 0x7f7e9c0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7ea195e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7ea195e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c1580f0 0x7f7e9c0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7ea195e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9c0b4d80 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7ea195e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9c0b4c00 con 0x7f7e9c1580f0 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7ea195e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9c0b4fc0 con 0x7f7e9c1605b0 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7e9affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 0x7f7e9c0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7e9affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 0x7f7e9c0cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40458/0 (socket says 172.21.15.46:40458) 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.689+0000 7f7e9affd640 1 -- 172.21.15.46:0/3317302407 learned_addr learned my addr 172.21.15.46:0/3317302407 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9b7fe640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c1580f0 0x7f7e9c0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9a7fc640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9affd640 1 -- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 msgr2=0x7f7e9c0caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9affd640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9affd640 1 -- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c1580f0 msgr2=0x7f7e9c0b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9affd640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c1580f0 0x7f7e9c0b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9affd640 1 -- 172.21.15.46:0/3317302407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9c0cc3e0 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e9affd640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 0x7f7e9c0cfb20 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00cbe0 tx=0x7f7e8c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c00aca0 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00ae40 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9c0cb340 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c012600 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.690+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e9c0cb900 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.691+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e9c06a270 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.693+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7e8c0127a0 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.693+0000 7f7e93fff640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e7c0762d0 0x7f7e7c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.693+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8261+0+0 (secure 0 0 0) 0x7f7e8c0991d0 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.693+0000 7f7e9a7fc640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e7c0762d0 0x7f7e7c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:11.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.694+0000 7f7e9a7fc640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e7c0762d0 0x7f7e7c0787e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7e84034000 tx=0x7f7e8402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.697+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7e8c062050 con 0x7f7e9c0d2e90 2024-05-09T03:19:11.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:11.928+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f7e9c0b71e0 con 0x7f7e9c0d2e90 2024-05-09T03:19:12.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.232+0000 7f7e93fff640 1 -- 172.21.15.46:0/3317302407 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v267) ==== 162+0+0 (secure 0 0 0) 0x7f7e8c061950 con 0x7f7e9c0d2e90 2024-05-09T03:19:12.233 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e7c0762d0 msgr2=0x7f7e7c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e7c0762d0 0x7f7e7c0787e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7e84034000 tx=0x7f7e8402e000 comp rx=0 tx=0).stop 2024-05-09T03:19:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 msgr2=0x7f7e9c0cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 0x7f7e9c0cfb20 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00cbe0 tx=0x7f7e8c00a060 comp rx=0 tx=0).stop 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 shutdown_connections 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9c1580f0 0x7f7e9c0b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7e9c1605b0 0x7f7e9c0caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7e7c0762d0 0x7f7e7c0787e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 --2- 172.21.15.46:0/3317302407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e9c0d2e90 0x7f7e9c0cfb20 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 >> 172.21.15.46:0/3317302407 conn(0x7f7e9c08d390 msgr2=0x7f7e9c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 shutdown_connections 2024-05-09T03:19:12.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.234+0000 7f7ea195e640 1 -- 172.21.15.46:0/3317302407 wait complete. 2024-05-09T03:19:12.241 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:12.241+0000 7f94bd296640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-05-09T03:19:12.242 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:12.242+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:12.242 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:12.242+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:19:12.248 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:19:12.241+0000 7f94bd296640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-05-09T03:19:12.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:12.304 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:12.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.366+0000 7fd86db1f640 1 Processor -- start 2024-05-09T03:19:12.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.366+0000 7fd86db1f640 1 -- start start 2024-05-09T03:19:12.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd86db1f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd868164c70 0x7fd8681615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd86db1f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd8680d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd86db1f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd868160b20 con 0x7fd8680d2480 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd86db1f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd868161150 con 0x7fd868164c70 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd86db1f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd8680d15f0 con 0x7fd86815dfe0 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd8677fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd868164c70 0x7fd8681615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd867fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd8680d2480 0x7fd8680d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34924/0 (socket says 172.21.15.46:34924) 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.367+0000 7fd867fff640 1 -- 172.21.15.46:0/3231511561 learned_addr learned my addr 172.21.15.46:0/3231511561 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd866ffd640 1 --2- 172.21.15.46:0/3231511561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd8680d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd8677fe640 1 -- 172.21.15.46:0/3231511561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 msgr2=0x7fd8680d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd8677fe640 1 --2- 172.21.15.46:0/3231511561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd8680d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd8677fe640 1 -- 172.21.15.46:0/3231511561 >> v1:172.21.15.46:6789/0 conn(0x7fd8680d2480 legacy=0x7fd8680d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd8677fe640 1 -- 172.21.15.46:0/3231511561 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8680d18d0 con 0x7fd868164c70 2024-05-09T03:19:12.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd8667fc640 1 -- 172.21.15.46:0/3231511561 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885884272 0 0) 0x7fd868160b20 con 0x7fd8680d2480 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.368+0000 7fd8677fe640 1 --2- 172.21.15.46:0/3231511561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd8681615e0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fd8580097a0 tx=0x7fd85802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3fdeb9e371587c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd8667fc640 1 -- 172.21.15.46:0/3231511561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858037070 con 0x7fd868164c70 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd8667fc640 1 -- 172.21.15.46:0/3231511561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd85802ae50 con 0x7fd868164c70 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd8667fc640 1 -- 172.21.15.46:0/3231511561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858032660 con 0x7fd868164c70 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 -- 172.21.15.46:0/3231511561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 msgr2=0x7fd8681615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 --2- 172.21.15.46:0/3231511561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd8681615e0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fd8580097a0 tx=0x7fd85802a100 comp rx=0 tx=0).stop 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 -- 172.21.15.46:0/3231511561 shutdown_connections 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 --2- 172.21.15.46:0/3231511561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd8680d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 --2- 172.21.15.46:0/3231511561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd8681615e0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 -- 172.21.15.46:0/3231511561 >> 172.21.15.46:0/3231511561 conn(0x7fd868073db0 msgr2=0x7fd8680741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 -- 172.21.15.46:0/3231511561 shutdown_connections 2024-05-09T03:19:12.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.369+0000 7fd86db1f640 1 -- 172.21.15.46:0/3231511561 wait complete. 2024-05-09T03:19:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.371+0000 7fd86db1f640 1 Processor -- start 2024-05-09T03:19:12.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.371+0000 7fd86db1f640 1 -- start start 2024-05-09T03:19:12.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd86db1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8680d2480 0x7fd8681f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd86db1f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd86815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd86db1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd86815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd86db1f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd86815e3f0 con 0x7fd868164c70 2024-05-09T03:19:12.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd86db1f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd8680d1300 con 0x7fd8680d2480 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd86db1f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd8680d45c0 con 0x7fd86815dfe0 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd86815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd86815bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40494/0 (socket says 172.21.15.46:40494) 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 -- 172.21.15.46:0/1964348878 learned_addr learned my addr 172.21.15.46:0/1964348878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd8677fe640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8680d2480 0x7fd8681f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd866ffd640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd86815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 -- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 msgr2=0x7fd86815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd86815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 -- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8680d2480 msgr2=0x7fd8681f6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8680d2480 0x7fd8681f6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 -- 172.21.15.46:0/1964348878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8681f9e10 con 0x7fd868164c70 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd8677fe640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8680d2480 0x7fd8681f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd866ffd640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd86815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:12.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.373+0000 7fd867fff640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd86815bc60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fd86000c680 tx=0x7fd86000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:12.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.374+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd860018070 con 0x7fd868164c70 2024-05-09T03:19:12.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.374+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd860004d60 con 0x7fd868164c70 2024-05-09T03:19:12.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.374+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8681fa0f0 con 0x7fd868164c70 2024-05-09T03:19:12.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.374+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86000a880 con 0x7fd868164c70 2024-05-09T03:19:12.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.374+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8681fa6b0 con 0x7fd868164c70 2024-05-09T03:19:12.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.375+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd86808cfd0 con 0x7fd868164c70 2024-05-09T03:19:12.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.375+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd86001b070 con 0x7fd868164c70 2024-05-09T03:19:12.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.376+0000 7fd84ffff640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8540762d0 0x7fd8540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.376+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7fd860097630 con 0x7fd868164c70 2024-05-09T03:19:12.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.376+0000 7fd8677fe640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8540762d0 0x7fd8540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.377+0000 7fd8677fe640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8540762d0 0x7fd8540787e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd858034000 tx=0x7fd85802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:12.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.380+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd86808cfd0 con 0x7fd868164c70 2024-05-09T03:19:12.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.608+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd86806a270 con 0x7fd868164c70 2024-05-09T03:19:12.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.609+0000 7fd84ffff640 1 -- 172.21.15.46:0/1964348878 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) ==== 95+0+4343 (secure 0 0 0) 0x7fd860060460 con 0x7fd868164c70 2024-05-09T03:19:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:12.610 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":269,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:12.242507+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:12.610 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 269 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8540762d0 msgr2=0x7fd8540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8540762d0 0x7fd8540787e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd858034000 tx=0x7fd85802e000 comp rx=0 tx=0).stop 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 msgr2=0x7fd86815bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd86815bc60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fd86000c680 tx=0x7fd86000cb40 comp rx=0 tx=0).stop 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 shutdown_connections 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd8540762d0 0x7fd8540787e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd868164c70 0x7fd86815bc60 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd86815dfe0 0x7fd86815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 --2- 172.21.15.46:0/1964348878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8680d2480 0x7fd8681f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 >> 172.21.15.46:0/1964348878 conn(0x7fd868073db0 msgr2=0x7fd8680b9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:12.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 shutdown_connections 2024-05-09T03:19:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.611+0000 7fd86db1f640 1 -- 172.21.15.46:0/1964348878 wait complete. 2024-05-09T03:19:12.623 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:12.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:12.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.737+0000 7f11364c8640 1 Processor -- start 2024-05-09T03:19:12.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.737+0000 7f11364c8640 1 -- start start 2024-05-09T03:19:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.738+0000 7f11364c8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11300b7ef0 0x7f113016b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.738+0000 7f11364c8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11300d49e0 con 0x7f11300d57a0 2024-05-09T03:19:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.738+0000 7f11364c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11300d4cc0 con 0x7f1130160610 2024-05-09T03:19:12.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.738+0000 7f11364c8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f11300d5bb0 con 0x7f11300b7ef0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.738+0000 7f112ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11300d57a0 0x7f11300b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34934/0 (socket says 172.21.15.46:34934) 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.738+0000 7f112ffff640 1 -- 172.21.15.46:0/607734697 learned_addr learned my addr 172.21.15.46:0/607734697 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2962827828 0 0) 0x7f11300d49e0 con 0x7f11300d57a0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1134a65640 1 --2- 172.21.15.46:0/607734697 >> v2:172.21.15.88:3300/0 conn(0x7f11300b7ef0 0x7f113016b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f111c003520 con 0x7f11300d57a0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2393483108 0 0) 0x7f111c003520 con 0x7f11300d57a0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 >> v2:172.21.15.88:3300/0 conn(0x7f11300b7ef0 msgr2=0x7f113016b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 --2- 172.21.15.46:0/607734697 >> v2:172.21.15.88:3300/0 conn(0x7f11300b7ef0 0x7f113016b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 >> v1:172.21.15.88:6789/0 conn(0x7f1130160610 legacy=0x7f11300b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11300d4fa0 con 0x7f11300d57a0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f11200016f0 con 0x7f11300d57a0 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 >> v1:172.21.15.46:6789/0 conn(0x7f11300d57a0 legacy=0x7f11300b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f111c003a50 0x7f111c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c004800 0x7f111c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f111c005250 0x7f111c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11300d5bb0 con 0x7f111c005250 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f111c00c1e0 con 0x7f111c003a50 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f111c00c3e0 con 0x7f111c004800 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1134a65640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f111c005250 0x7f111c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f112ffff640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f111c003a50 0x7f111c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1127fff640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c004800 0x7f111c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1134a65640 1 -- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c004800 msgr2=0x7f111c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1134a65640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c004800 0x7f111c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1134a65640 1 -- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f111c003a50 msgr2=0x7f111c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.739+0000 7f1134a65640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f111c003a50 0x7f111c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.740+0000 7f1134a65640 1 -- 172.21.15.46:0/607734697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c003520 con 0x7f111c005250 2024-05-09T03:19:12.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.740+0000 7f1134a65640 1 -- 172.21.15.46:0/607734697 reap_dead start 2024-05-09T03:19:12.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.740+0000 7f1134a65640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f111c005250 0x7f111c00b9c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1128009fc0 tx=0x7f111c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59ad7521af3eac3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:12.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.740+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128035020 con 0x7f111c005250 2024-05-09T03:19:12.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.740+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1128003bd0 con 0x7f111c005250 2024-05-09T03:19:12.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.740+0000 7f112f7fe640 1 -- 172.21.15.46:0/607734697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128009c10 con 0x7f111c005250 2024-05-09T03:19:12.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 -- 172.21.15.46:0/607734697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f111c005250 msgr2=0x7f111c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f111c005250 0x7f111c00b9c0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1128009fc0 tx=0x7f111c005830 comp rx=0 tx=0).stop 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 -- 172.21.15.46:0/607734697 shutdown_connections 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 --2- 172.21.15.46:0/607734697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f111c005250 0x7f111c00b9c0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 -- 172.21.15.46:0/607734697 >> 172.21.15.46:0/607734697 conn(0x7f11300743e0 msgr2=0x7f11300d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 -- 172.21.15.46:0/607734697 shutdown_connections 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.741+0000 7f11364c8640 1 -- 172.21.15.46:0/607734697 wait complete. 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.743+0000 7f11364c8640 1 Processor -- start 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.743+0000 7f11364c8640 1 -- start start 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f11364c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c005250 0x7f113014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f11364c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1130161ba0 0x7f1130161fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f11364c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 0x7f113014d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f11364c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11300d15b0 con 0x7f113014d510 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f11364c8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1130160a20 con 0x7f1130161ba0 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f11364c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11300d1920 con 0x7f111c005250 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.744+0000 7f1134a65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 0x7f113014d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 0x7f113014d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40512/0 (socket says 172.21.15.46:40512) 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 -- 172.21.15.46:0/1675425021 learned_addr learned my addr 172.21.15.46:0/1675425021 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1127fff640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1130161ba0 0x7f1130161fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f112ffff640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c005250 0x7f113014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 -- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c005250 msgr2=0x7f113014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c005250 0x7f113014aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 -- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1130161ba0 msgr2=0x7f1130161fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1130161ba0 0x7f1130161fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 -- 172.21.15.46:0/1675425021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113014de60 con 0x7f113014d510 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f1134a65640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 0x7f113014d920 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f112800e230 tx=0x7f1128009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128035030 con 0x7f113014d510 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1128043070 con 0x7f113014d510 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112802d070 con 0x7f113014d510 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1130162970 con 0x7f113014d510 2024-05-09T03:19:12.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.745+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1130162f30 con 0x7f113014d510 2024-05-09T03:19:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.746+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11301626e0 con 0x7f113014d510 2024-05-09T03:19:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.747+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1128005d10 con 0x7f113014d510 2024-05-09T03:19:12.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.748+0000 7f112d7fa640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11100762d0 0x7f11100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:12.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.748+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7f11280b8f20 con 0x7f113014d510 2024-05-09T03:19:12.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.748+0000 7f112ffff640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11100762d0 0x7f11100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:12.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.748+0000 7f112ffff640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11100762d0 0x7f11100787e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f113006d3b0 tx=0x7f1120001e70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:12.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.752+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1128081790 con 0x7f113014d510 2024-05-09T03:19:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.984+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f113006a270 con 0x7f113014d510 2024-05-09T03:19:12.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.985+0000 7f112d7fa640 1 -- 172.21.15.46:0/1675425021 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) ==== 95+0+4343 (secure 0 0 0) 0x7f1128085a90 con 0x7f113014d510 2024-05-09T03:19:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:12.986 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":269,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:12.242507+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:12.986 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 269 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11100762d0 msgr2=0x7f11100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11100762d0 0x7f11100787e0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f113006d3b0 tx=0x7f1120001e70 comp rx=0 tx=0).stop 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 msgr2=0x7f113014d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 0x7f113014d920 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f112800e230 tx=0x7f1128009750 comp rx=0 tx=0).stop 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 shutdown_connections 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11100762d0 0x7f11100787e0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f113014d510 0x7f113014d920 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1130161ba0 0x7f1130161fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 --2- 172.21.15.46:0/1675425021 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f111c005250 0x7f113014aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 >> 172.21.15.46:0/1675425021 conn(0x7f11300743e0 msgr2=0x7f113015ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 shutdown_connections 2024-05-09T03:19:12.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:12.987+0000 7f11364c8640 1 -- 172.21.15.46:0/1675425021 wait complete. 2024-05-09T03:19:12.999 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:12.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.116+0000 7f5a74c27640 1 Processor -- start 2024-05-09T03:19:13.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.116+0000 7f5a74c27640 1 -- start start 2024-05-09T03:19:13.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.117+0000 7f5a74c27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 0x7f5a700b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.117+0000 7f5a74c27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a700b7d20 0x7f5a700b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.117+0000 7f5a74c27640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a700b8130 con 0x7f5a700d57a0 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.117+0000 7f5a74c27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5a700d5bb0 con 0x7f5a700b7d20 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.117+0000 7f5a74c27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700d49e0 con 0x7f5a70160610 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6dd9b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 0x7f5a700b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a700b7d20 0x7f5a700b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5a700d57a0 0x7f5a700b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:42072/0 (socket says 172.21.15.46:42072) 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5a700b7d20 0x7f5a700b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:35712/0 (socket says 172.21.15.46:35712) 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6dd9b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 0x7f5a700b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:49232/0 (socket says 172.21.15.46:49232) 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6e59c640 1 -- 172.21.15.46:0/3326283850 learned_addr learned my addr 172.21.15.46:0/3326283850 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:13.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6ed9d640 1 -- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 msgr2=0x7f5a700b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6ed9d640 1 --2- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 0x7f5a700b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6ed9d640 1 -- 172.21.15.46:0/3326283850 >> v1:172.21.15.88:6789/0 conn(0x7f5a700d57a0 legacy=0x7f5a700b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6ed9d640 1 -- 172.21.15.46:0/3326283850 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700d4cc0 con 0x7f5a700b7d20 2024-05-09T03:19:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6d59a640 1 -- 172.21.15.46:0/3326283850 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1704380890 0 0) 0x7f5a60001490 con 0x7f5a700d57a0 2024-05-09T03:19:13.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.118+0000 7f5a6dd9b640 1 --2- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 0x7f5a700b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.119+0000 7f5a6ed9d640 1 --2- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a700b7d20 0x7f5a700b9450 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5a680097a0 tx=0x7f5a6802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82a37ba0eb872f70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.119+0000 7f5a6d59a640 1 -- 172.21.15.46:0/3326283850 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68037070 con 0x7f5a700b7d20 2024-05-09T03:19:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.119+0000 7f5a6d59a640 1 -- 172.21.15.46:0/3326283850 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a68032440 con 0x7f5a700b7d20 2024-05-09T03:19:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.119+0000 7f5a6d59a640 1 -- 172.21.15.46:0/3326283850 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a68032810 con 0x7f5a700b7d20 2024-05-09T03:19:13.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 -- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a700b7d20 msgr2=0x7f5a700b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 --2- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a700b7d20 0x7f5a700b9450 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5a680097a0 tx=0x7f5a6802a180 comp rx=0 tx=0).stop 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 -- 172.21.15.46:0/3326283850 shutdown_connections 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 --2- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a700b7d20 0x7f5a700b9450 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 --2- 172.21.15.46:0/3326283850 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a70160610 0x7f5a700b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 -- 172.21.15.46:0/3326283850 >> 172.21.15.46:0/3326283850 conn(0x7f5a700743e0 msgr2=0x7f5a700d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 -- 172.21.15.46:0/3326283850 shutdown_connections 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.120+0000 7f5a74c27640 1 -- 172.21.15.46:0/3326283850 wait complete. 2024-05-09T03:19:13.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.122+0000 7f5a74c27640 1 Processor -- start 2024-05-09T03:19:13.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.122+0000 7f5a74c27640 1 -- start start 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a74c27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a700b7d20 0x7f5a70154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a74c27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 0x7f5a70156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a74c27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a70160610 0x7f5a701566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a74c27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70160a20 con 0x7f5a700d57a0 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a74c27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700d5280 con 0x7f5a70160610 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a74c27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700d15b0 con 0x7f5a700b7d20 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a6dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 0x7f5a70156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a6dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 0x7f5a70156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40522/0 (socket says 172.21.15.46:40522) 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a6dd9b640 1 -- 172.21.15.46:0/1616571276 learned_addr learned my addr 172.21.15.46:0/1616571276 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a6ed9d640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a70160610 0x7f5a701566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.123+0000 7f5a6e59c640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a700b7d20 0x7f5a70154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a6dd9b640 1 -- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a700b7d20 msgr2=0x7f5a70154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a6dd9b640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a700b7d20 0x7f5a70154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a6dd9b640 1 -- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a70160610 msgr2=0x7f5a701566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a6dd9b640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a70160610 0x7f5a701566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a6dd9b640 1 -- 172.21.15.46:0/1616571276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70154bc0 con 0x7f5a700d57a0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a6dd9b640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 0x7f5a70156170 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800e960 tx=0x7f5a5800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58017070 con 0x7f5a700d57a0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a5800ca00 con 0x7f5a700d57a0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a701550d0 con 0x7f5a700d57a0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5800cdd0 con 0x7f5a700d57a0 2024-05-09T03:19:13.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.124+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a70155690 con 0x7f5a700d57a0 2024-05-09T03:19:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.125+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a70154e20 con 0x7f5a700d57a0 2024-05-09T03:19:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.126+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5a5801a070 con 0x7f5a700d57a0 2024-05-09T03:19:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.126+0000 7f5a66ffd640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a50076300 0x7f5a50078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.126+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7f5a580981c0 con 0x7f5a700d57a0 2024-05-09T03:19:13.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.127+0000 7f5a6e59c640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a50076300 0x7f5a50078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.127+0000 7f5a6e59c640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a50076300 0x7f5a50078810 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5a600012e0 tx=0x7f5a60001630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:13.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.131+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5a58060ac0 con 0x7f5a700d57a0 2024-05-09T03:19:13.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.367+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a7006a270 con 0x7f5a700d57a0 2024-05-09T03:19:13.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.368+0000 7f5a66ffd640 1 -- 172.21.15.46:0/1616571276 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 270 v270) ==== 95+0+5166 (secure 0 0 0) 0x7f5a58064dc0 con 0x7f5a700d57a0 2024-05-09T03:19:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:13.369 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":270,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9624,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/925919281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":925919281},{"type":"v1","addr":"172.21.15.46:6834","nonce":925919281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":270}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:12.242507+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:13.369 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 270 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a50076300 msgr2=0x7f5a50078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a50076300 0x7f5a50078810 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f5a600012e0 tx=0x7f5a60001630 comp rx=0 tx=0).stop 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 msgr2=0x7f5a70156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 0x7f5a70156170 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f5a5800e960 tx=0x7f5a5800ee20 comp rx=0 tx=0).stop 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 shutdown_connections 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5a70160610 0x7f5a701566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5a50076300 0x7f5a50078810 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a700d57a0 0x7f5a70156170 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 --2- 172.21.15.46:0/1616571276 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5a700b7d20 0x7f5a70154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 >> 172.21.15.46:0/1616571276 conn(0x7f5a700743e0 msgr2=0x7f5a7015eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 shutdown_connections 2024-05-09T03:19:13.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.370+0000 7f5a74c27640 1 -- 172.21.15.46:0/1616571276 wait complete. 2024-05-09T03:19:13.382 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 400.00000000' led to DAMAGED state 2024-05-09T03:19:13.382 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 100.00000000.inode' 2024-05-09T03:19:13.382 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:13.382 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:13.398 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:19:13.399 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:13.399 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:13.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:13.399 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:19:13.454 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:19:13.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:13.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.610+0000 7fa1f7249640 1 Processor -- start 2024-05-09T03:19:13.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.610+0000 7fa1f7249640 1 -- start start 2024-05-09T03:19:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f7249640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1110 0x7fa1f00d0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f7249640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1f00d1d30 0x7fa1f00b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f7249640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f00b92c0 con 0x7fa1f00d1920 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f7249640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa1f00b2f10 con 0x7fa1f00d1d30 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f7249640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f00b9560 con 0x7fa1f00d1110 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1effff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa1f00d1920 0x7fa1f00b32c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34942/0 (socket says 172.21.15.46:34942) 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1effff640 1 -- 172.21.15.46:0/3634066653 learned_addr learned my addr 172.21.15.46:0/3634066653 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f57e6640 1 --2- 172.21.15.46:0/3634066653 >> v2:172.21.15.88:3300/0 conn(0x7fa1f00d1d30 0x7fa1f00b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.611+0000 7fa1f4fe5640 1 --2- 172.21.15.46:0/3634066653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1110 0x7fa1f00d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1ef7fe640 1 -- 172.21.15.46:0/3634066653 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2081046402 0 0) 0x7fa1f00b92c0 con 0x7fa1f00d1920 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1ef7fe640 1 -- 172.21.15.46:0/3634066653 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1dc003520 con 0x7fa1f00d1920 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1f4fe5640 1 -- 172.21.15.46:0/3634066653 >> v2:172.21.15.88:3300/0 conn(0x7fa1f00d1d30 msgr2=0x7fa1f00b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1f4fe5640 1 --2- 172.21.15.46:0/3634066653 >> v2:172.21.15.88:3300/0 conn(0x7fa1f00d1d30 0x7fa1f00b39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1f4fe5640 1 -- 172.21.15.46:0/3634066653 >> v1:172.21.15.46:6789/0 conn(0x7fa1f00d1920 legacy=0x7fa1f00b32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1ef7fe640 1 -- 172.21.15.46:0/3634066653 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1951797953 0 0) 0x7fa1dc003520 con 0x7fa1f00d1920 2024-05-09T03:19:13.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.612+0000 7fa1f4fe5640 1 -- 172.21.15.46:0/3634066653 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f00cfb20 con 0x7fa1f00d1110 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.613+0000 7fa1f4fe5640 1 --2- 172.21.15.46:0/3634066653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1110 0x7fa1f00d0020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e002a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2725d0782a9c5624 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.613+0000 7fa1ef7fe640 1 -- 172.21.15.46:0/3634066653 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80016f0 con 0x7fa1f00d1110 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.613+0000 7fa1ef7fe640 1 -- 172.21.15.46:0/3634066653 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e0007540 con 0x7fa1f00d1110 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.613+0000 7fa1ef7fe640 1 -- 172.21.15.46:0/3634066653 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0007910 con 0x7fa1f00d1110 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 -- 172.21.15.46:0/3634066653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1110 msgr2=0x7fa1f00d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 --2- 172.21.15.46:0/3634066653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1110 0x7fa1f00d0020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009d90 tx=0x7fa1e002a4d0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 -- 172.21.15.46:0/3634066653 shutdown_connections 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 --2- 172.21.15.46:0/3634066653 >> v2:172.21.15.88:3300/0 conn(0x7fa1f00d1d30 0x7fa1f00b39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 --2- 172.21.15.46:0/3634066653 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1110 0x7fa1f00d0020 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 -- 172.21.15.46:0/3634066653 >> 172.21.15.46:0/3634066653 conn(0x7fa1f00d05f0 msgr2=0x7fa1f00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 -- 172.21.15.46:0/3634066653 shutdown_connections 2024-05-09T03:19:13.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.614+0000 7fa1f7249640 1 -- 172.21.15.46:0/3634066653 wait complete. 2024-05-09T03:19:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.615+0000 7fa1f7249640 1 Processor -- start 2024-05-09T03:19:13.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.616+0000 7fa1f7249640 1 -- start start 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f7249640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1920 0x7fa1f01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f7249640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 0x7fa1f0153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f7249640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1f0071ee0 0x7fa1f0154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f7249640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f009d030 con 0x7fa1f00d1d30 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f7249640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f009ceb0 con 0x7fa1f0071ee0 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f7249640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f00b8e00 con 0x7fa1f00d1920 2024-05-09T03:19:13.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 0x7fa1f0153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 0x7fa1f0153100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40524/0 (socket says 172.21.15.46:40524) 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 -- 172.21.15.46:0/4094462063 learned_addr learned my addr 172.21.15.46:0/4094462063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f4fe5640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1920 0x7fa1f01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1f57e6640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1f0071ee0 0x7fa1f0154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 -- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1920 msgr2=0x7fa1f01572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1920 0x7fa1f01572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 -- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1f0071ee0 msgr2=0x7fa1f0154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1f0071ee0 0x7fa1f0154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.617+0000 7fa1effff640 1 -- 172.21.15.46:0/4094462063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f0072610 con 0x7fa1f00d1d30 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.618+0000 7fa1effff640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 0x7fa1f0153100 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c490 tx=0x7fa1d800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.618+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8005ce0 con 0x7fa1f00d1d30 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.618+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d8007070 con 0x7fa1f00d1d30 2024-05-09T03:19:13.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.618+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f00728f0 con 0x7fa1f00d1d30 2024-05-09T03:19:13.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.618+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8007460 con 0x7fa1f00d1d30 2024-05-09T03:19:13.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.618+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f006dc40 con 0x7fa1f00d1d30 2024-05-09T03:19:13.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.619+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f0072380 con 0x7fa1f00d1d30 2024-05-09T03:19:13.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.620+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa1d8007600 con 0x7fa1f00d1d30 2024-05-09T03:19:13.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.620+0000 7fa1ed7fa640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1d00762d0 0x7fa1d00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:13.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.620+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7fa1d8097bb0 con 0x7fa1f00d1d30 2024-05-09T03:19:13.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.620+0000 7fa1f4fe5640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1d00762d0 0x7fa1d00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:13.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.621+0000 7fa1f4fe5640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1d00762d0 0x7fa1d00787e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0034000 tx=0x7fa1e002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:13.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.625+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa1d80604b0 con 0x7fa1f00d1d30 2024-05-09T03:19:13.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:13.857+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa1f006a270 con 0x7fa1f00d1d30 2024-05-09T03:19:14.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.249+0000 7fa1ed7fa640 1 -- 172.21.15.46:0/4094462063 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v271) ==== 161+0+0 (secure 0 0 0) 0x7fa1d80647b0 con 0x7fa1f00d1d30 2024-05-09T03:19:14.250 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1d00762d0 msgr2=0x7fa1d00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1d00762d0 0x7fa1d00787e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0034000 tx=0x7fa1e002f000 comp rx=0 tx=0).stop 2024-05-09T03:19:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 msgr2=0x7fa1f0153100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 0x7fa1f0153100 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c490 tx=0x7fa1d800c950 comp rx=0 tx=0).stop 2024-05-09T03:19:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 shutdown_connections 2024-05-09T03:19:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1f0071ee0 0x7fa1f0154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa1d00762d0 0x7fa1d00787e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f00d1d30 0x7fa1f0153100 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 --2- 172.21.15.46:0/4094462063 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa1f00d1920 0x7fa1f01572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 >> 172.21.15.46:0/4094462063 conn(0x7fa1f00d05f0 msgr2=0x7fa1f00b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 shutdown_connections 2024-05-09T03:19:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.251+0000 7fa1f7249640 1 -- 172.21.15.46:0/4094462063 wait complete. 2024-05-09T03:19:14.264 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.381+0000 7f8025fbe640 1 Processor -- start 2024-05-09T03:19:14.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.381+0000 7f8025fbe640 1 -- start start 2024-05-09T03:19:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.382+0000 7f8025fbe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 0x7f80200d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.382+0000 7f8025fbe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 0x7f80201642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.382+0000 7f8025fbe640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8020164900 con 0x7f8020163eb0 2024-05-09T03:19:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.382+0000 7f8025fbe640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f80200d4bb0 con 0x7f80200bb770 2024-05-09T03:19:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.382+0000 7f8025fbe640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8020164be0 con 0x7f80200d3180 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 0x7f80201642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 0x7f80200d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 0x7f80201642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:49280/0 (socket says 172.21.15.46:49280) 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 0x7f80200d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:35738/0 (socket says 172.21.15.46:35738) 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 -- 172.21.15.46:0/3370557671 learned_addr learned my addr 172.21.15.46:0/3370557671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f8017fff640 1 --1- 172.21.15.46:0/3370557671 >> v1:172.21.15.88:6789/0 conn(0x7f8020163eb0 0x7f80200d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:42084/0 (socket says 172.21.15.46:42084) 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 -- 172.21.15.46:0/3370557671 >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 msgr2=0x7f80200d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 --2- 172.21.15.46:0/3370557671 >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 0x7f80200d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 -- 172.21.15.46:0/3370557671 >> v1:172.21.15.88:6789/0 conn(0x7f8020163eb0 legacy=0x7f80200d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801ffff640 1 -- 172.21.15.46:0/3370557671 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80200d4840 con 0x7f80200d3180 2024-05-09T03:19:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801effd640 1 -- 172.21.15.46:0/3370557671 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1438108633 0 0) 0x7f8020164900 con 0x7f8020163eb0 2024-05-09T03:19:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.383+0000 7f801f7fe640 1 --2- 172.21.15.46:0/3370557671 >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 0x7f80200d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.384+0000 7f801ffff640 1 --2- 172.21.15.46:0/3370557671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 0x7f80201642c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8018009e50 tx=0x7f801802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c3c2147ee6be29d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.384+0000 7f801effd640 1 -- 172.21.15.46:0/3370557671 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018036070 con 0x7f80200d3180 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.384+0000 7f801effd640 1 -- 172.21.15.46:0/3370557671 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8018031600 con 0x7f80200d3180 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.384+0000 7f801effd640 1 -- 172.21.15.46:0/3370557671 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80180319d0 con 0x7f80200d3180 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 -- 172.21.15.46:0/3370557671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 msgr2=0x7f80201642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 --2- 172.21.15.46:0/3370557671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 0x7f80201642c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8018009e50 tx=0x7f801802a710 comp rx=0 tx=0).stop 2024-05-09T03:19:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 -- 172.21.15.46:0/3370557671 shutdown_connections 2024-05-09T03:19:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 --2- 172.21.15.46:0/3370557671 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200d3180 0x7f80201642c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 --2- 172.21.15.46:0/3370557671 >> v2:172.21.15.88:3300/0 conn(0x7f80200bb770 0x7f80200d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 -- 172.21.15.46:0/3370557671 >> 172.21.15.46:0/3370557671 conn(0x7f802006e370 msgr2=0x7f802006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 -- 172.21.15.46:0/3370557671 shutdown_connections 2024-05-09T03:19:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.385+0000 7f8025fbe640 1 -- 172.21.15.46:0/3370557671 wait complete. 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.387+0000 7f8025fbe640 1 Processor -- start 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.387+0000 7f8025fbe640 1 -- start start 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f8025fbe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200bb770 0x7f80200ce470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f8025fbe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8020163eb0 0x7f8020152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f8025fbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 0x7f80200ad160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f8025fbe640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f80200d0bc0 con 0x7f80200acbe0 2024-05-09T03:19:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f8025fbe640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f80200bbb80 con 0x7f8020163eb0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f8025fbe640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f80200d0d40 con 0x7f80200bb770 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f801ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 0x7f80200ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f801ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 0x7f80200ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40534/0 (socket says 172.21.15.46:40534) 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.388+0000 7f801ffff640 1 -- 172.21.15.46:0/1992979137 learned_addr learned my addr 172.21.15.46:0/1992979137 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801f7fe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200bb770 0x7f80200ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f8017fff640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8020163eb0 0x7f8020152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801ffff640 1 -- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200bb770 msgr2=0x7f80200ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801ffff640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200bb770 0x7f80200ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801ffff640 1 -- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8020163eb0 msgr2=0x7f8020152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801ffff640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8020163eb0 0x7f8020152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801ffff640 1 -- 172.21.15.46:0/1992979137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80200cede0 con 0x7f80200acbe0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801ffff640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 0x7f80200ad160 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f8018034040 tx=0x7f8018004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018041020 con 0x7f80200acbe0 2024-05-09T03:19:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8018037040 con 0x7f80200acbe0 2024-05-09T03:19:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80200cf0c0 con 0x7f80200acbe0 2024-05-09T03:19:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018036050 con 0x7f80200acbe0 2024-05-09T03:19:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.389+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80200cf680 con 0x7f80200acbe0 2024-05-09T03:19:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.390+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80200ceb50 con 0x7f80200acbe0 2024-05-09T03:19:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.391+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8018007c50 con 0x7f80200acbe0 2024-05-09T03:19:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.391+0000 7f801cff9640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80000762d0 0x7f80000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.392+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7f80180b8830 con 0x7f80200acbe0 2024-05-09T03:19:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.392+0000 7f801f7fe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80000762d0 0x7f80000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.392+0000 7f801f7fe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80000762d0 0x7f80000787e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f80200b8880 tx=0x7f8010008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:14.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.396+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f801804b060 con 0x7f80200acbe0 2024-05-09T03:19:14.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:14.663+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f80201634a0 con 0x7f80200acbe0 2024-05-09T03:19:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.256+0000 7f801cff9640 1 -- 172.21.15.46:0/1992979137 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v272) ==== 101+0+0 (secure 0 0 0) 0x7f80180816f0 con 0x7f80200acbe0 2024-05-09T03:19:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80000762d0 msgr2=0x7f80000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80000762d0 0x7f80000787e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f80200b8880 tx=0x7f8010008040 comp rx=0 tx=0).stop 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 msgr2=0x7f80200ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 0x7f80200ad160 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f8018034040 tx=0x7f8018004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 shutdown_connections 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f80000762d0 0x7f80000787e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80200acbe0 0x7f80200ad160 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8020163eb0 0x7f8020152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 --2- 172.21.15.46:0/1992979137 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f80200bb770 0x7f80200ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 >> 172.21.15.46:0/1992979137 conn(0x7f802006e370 msgr2=0x7f80200d2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 shutdown_connections 2024-05-09T03:19:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.259+0000 7f8025fbe640 1 -- 172.21.15.46:0/1992979137 wait complete. 2024-05-09T03:19:15.273 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.332+0000 7f9d5ad1af40 1 Processor -- start 2024-05-09T03:19:15.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.332+0000 7f9d5ad1af40 1 -- start start 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d5ad1af40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 0x55b74b859680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d5ad1af40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7ea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d5ad1af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b8738a0 0x55b74b909900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d5ad1af40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b74b747f50 con 0x55b74b874c10 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d5ad1af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b74b74bf90 con 0x55b74b8738a0 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d5ad1af40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b74b7e0cb0 con 0x55b74b7e65a0 2024-05-09T03:19:15.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 0x55b74b859680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 0x55b74b859680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40558/0 (socket says 172.21.15.46:40558) 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 -- 172.21.15.46:0/1536180745 learned_addr learned my addr 172.21.15.46:0/1536180745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d546cc640 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b8738a0 0x55b74b909900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d536ca640 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 -- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 msgr2=0x55b74b7ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 -- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b8738a0 msgr2=0x55b74b909900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b8738a0 0x55b74b909900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.333+0000 7f9d53ecb640 1 -- 172.21.15.46:0/1536180745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b74b7cc690 con 0x55b74b874c10 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 0x55b74b859680 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400c610 tx=0x7f9d4400cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a881a62b0f1760 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d526c8640 1 -- 172.21.15.46:0/1536180745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44015020 con 0x55b74b874c10 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d526c8640 1 -- 172.21.15.46:0/1536180745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4400e5d0 con 0x55b74b874c10 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d526c8640 1 -- 172.21.15.46:0/1536180745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4400e9a0 con 0x55b74b874c10 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/1536180745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 msgr2=0x55b74b859680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 0x55b74b859680 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400c610 tx=0x7f9d4400cad0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/1536180745 shutdown_connections 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b8738a0 0x55b74b909900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/1536180745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b874c10 0x55b74b859680 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.334+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/1536180745 >> 172.21.15.46:0/1536180745 conn(0x55b74b6c3ac0 msgr2=0x55b74b79e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.335+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/1536180745 shutdown_connections 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.335+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/1536180745 wait complete. 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.335+0000 7f9d5ad1af40 1 Processor -- start 2024-05-09T03:19:15.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.335+0000 7f9d5ad1af40 1 -- start start 2024-05-09T03:19:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d5ad1af40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7e2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d5ad1af40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 0x55b74b7e4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d5ad1af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b874c10 0x55b74b878080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d5ad1af40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b74b86c420 con 0x55b74b8738a0 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d5ad1af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b74b7e6d40 con 0x55b74b874c10 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d5ad1af40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b74b7e6f90 con 0x55b74b7e65a0 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 0x55b74b7e4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 0x55b74b7e4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40562/0 (socket says 172.21.15.46:40562) 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 learned_addr learned my addr 172.21.15.46:0/2666958323 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7e2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d546cc640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b874c10 0x55b74b878080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 msgr2=0x55b74b7e2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b74b7e65a0 0x55b74b7e2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b874c10 msgr2=0x55b74b878080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b874c10 0x55b74b878080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b74b79f340 con 0x55b74b8738a0 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d546cc640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b74b874c10 0x55b74b878080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:15.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.337+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 0x55b74b7e4470 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009ef0 tx=0x7f9d4800f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.338+0000 7f9d516c6640 1 -- 172.21.15.46:0/2666958323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4800e770 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.338+0000 7f9d516c6640 1 -- 172.21.15.46:0/2666958323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4800e910 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.338+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b74b7e22b0 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.338+0000 7f9d516c6640 1 -- 172.21.15.46:0/2666958323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4800ed00 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.338+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b74b877880 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.339+0000 7f9d516c6640 1 -- 172.21.15.46:0/2666958323 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9d4801a430 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.340+0000 7f9d516c6640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d40074ce0 0x7f9d400771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.340+0000 7f9d516c6640 1 -- 172.21.15.46:0/2666958323 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7f9d48097a50 con 0x55b74b8738a0 2024-05-09T03:19:15.343 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:15.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.340+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b74b878ad0 0x55b74b912990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.340+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b7e49b0 con 0x55b74b878ad0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.340+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d40074ce0 0x7f9d400771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.340+0000 7f9d546cc640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b74b878ad0 0x55b74b912990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.341+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d40074ce0 0x7f9d400771f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400c610 tx=0x7f9d44009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.341+0000 7f9d546cc640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b74b878ad0 0x55b74b912990 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.343+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v114'908 uv882 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.343+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b912ed0 con 0x55b74b878ad0 2024-05-09T03:19:15.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.345+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v114'909 uv909 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.345+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b913270 con 0x55b74b878ad0 2024-05-09T03:19:15.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.346+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v114'910 uv910 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.347 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:15.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.346+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91cc70 con 0x55b74b878ad0 2024-05-09T03:19:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.347+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v114'911 uv911 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.347+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91d010 con 0x55b74b878ad0 2024-05-09T03:19:15.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.349+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v114'912 uv912 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.349+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91bc60 con 0x55b74b878ad0 2024-05-09T03:19:15.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.350+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v114'913 uv913 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.350+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c000 con 0x55b74b878ad0 2024-05-09T03:19:15.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.352+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v114'914 uv888 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.352+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.353+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v114'915 uv915 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.353+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.354+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v114'916 uv916 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.355 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:15.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.354+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.356+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v114'917 uv917 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.356+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.357+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v114'918 uv918 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.357+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.359+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v114'919 uv919 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.359+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.361+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v114'920 uv895 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.361+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.362+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v114'921 uv921 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.362+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.363+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v114'922 uv922 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.364 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:15.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.363+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.365+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v114'923 uv923 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.365+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.366+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v114'924 uv924 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.366+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.368+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v114'925 uv925 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.368+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.370+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v114'926 uv926 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.370+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.372+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v114'927 uv901 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.372+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.374+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v114'928 uv928 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.374+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.375+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v114'929 uv929 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.376 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:15.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.375+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.377+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v114'930 uv930 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.377+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.378+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v114'931 uv931 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.378+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.380+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v114'932 uv932 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.380+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.382+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v114'933 uv907 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.382+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.383+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v114'934 uv934 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.383+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.385+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v114'935 uv935 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.385 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:15.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.385+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.386+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v114'936 uv936 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.386+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.388+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v114'937 uv937 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.388+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b878ad0 2024-05-09T03:19:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.389+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v114'938 uv938 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.389+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b74b91c7a0 0x55b74b91f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.389+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b91c3a0 con 0x55b74b91c7a0 2024-05-09T03:19:15.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.389+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b74b91c7a0 0x55b74b91f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.390+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b74b91c7a0 0x55b74b91f8d0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.392+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v114'654 uv637 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.392+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b91fe10 con 0x55b74b91c7a0 2024-05-09T03:19:15.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.393+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v114'655 uv655 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.393+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9201b0 con 0x55b74b91c7a0 2024-05-09T03:19:15.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.394+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v114'656 uv656 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.395 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.394+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b920a00 con 0x55b74b91c7a0 2024-05-09T03:19:15.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.395+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v114'657 uv657 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.395+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b920da0 con 0x55b74b91c7a0 2024-05-09T03:19:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.397+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v114'658 uv658 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.397+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921140 con 0x55b74b91c7a0 2024-05-09T03:19:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.398+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v114'659 uv642 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.398+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b9214e0 con 0x55b74b91c7a0 2024-05-09T03:19:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.399+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v114'660 uv660 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.399+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.400+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v114'661 uv661 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.401 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.400+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.402+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v114'662 uv662 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.402+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.403+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v114'663 uv663 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.403+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.404+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v114'664 uv647 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.404+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.405+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v114'665 uv665 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.405+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.407+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v114'666 uv666 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.407 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.407+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9252e0 con 0x55b74b91c7a0 2024-05-09T03:19:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.408+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v114'667 uv667 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.408+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9252e0 con 0x55b74b91c7a0 2024-05-09T03:19:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.409+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v114'668 uv668 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.409+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.411+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v114'669 uv653 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.411+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.412+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v114'670 uv670 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.412+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.413+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v114'671 uv671 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.414 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:15.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.413+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b922890 con 0x55b74b91c7a0 2024-05-09T03:19:15.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.414+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v114'672 uv672 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.414+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b922890 con 0x55b74b91c7a0 2024-05-09T03:19:15.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.416+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v114'673 uv673 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.416+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b91c7a0 2024-05-09T03:19:15.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.417+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v114'674 uv674 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b91c7a0 2024-05-09T03:19:15.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.417+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b878ad0 2024-05-09T03:19:15.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.418+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v114'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.418+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b878ad0 2024-05-09T03:19:15.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.419+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v114'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.419+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b878ad0 2024-05-09T03:19:15.421 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:15.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.420+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v114'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.420+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b922890 con 0x55b74b878ad0 2024-05-09T03:19:15.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.421+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v114'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.421+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b922890 con 0x55b74b878ad0 2024-05-09T03:19:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.422+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v114'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.422+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b921880 con 0x55b74b878ad0 2024-05-09T03:19:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.423+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v114'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.423+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b74b921f00 0x55b74b924320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.424+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9249a0 con 0x55b74b921f00 2024-05-09T03:19:15.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.424+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b74b921f00 0x55b74b924320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.424+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b74b921f00 0x55b74b924320 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.425+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v114'132 uv131 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.425+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b924eb0 con 0x55b74b921f00 2024-05-09T03:19:15.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.426+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v114'133 uv133 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.426+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9253b0 con 0x55b74b921f00 2024-05-09T03:19:15.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.427+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v114'134 uv134 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.428 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.427+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b920630 con 0x55b74b921f00 2024-05-09T03:19:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.428+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v114'135 uv135 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.428+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925ce0 con 0x55b74b878ad0 2024-05-09T03:19:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.430+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v114'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.430+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b925ce0 con 0x55b74b878ad0 2024-05-09T03:19:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.431+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v114'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.431+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925ce0 con 0x55b74b878ad0 2024-05-09T03:19:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.432+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v114'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.432 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.432+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925ce0 con 0x55b74b878ad0 2024-05-09T03:19:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.433+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v114'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.433+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925ce0 con 0x55b74b878ad0 2024-05-09T03:19:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.434+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v114'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.434+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925ce0 con 0x55b74b878ad0 2024-05-09T03:19:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.435+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v114'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.435+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b74b926770 0x55b74b926b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.435+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.435+0000 7f9d546cc640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b74b926770 0x55b74b926b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.436+0000 7f9d546cc640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b74b926770 0x55b74b926b80 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.437+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v114'327 uv321 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.437+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.438+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v114'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.438+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.439+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v114'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.440 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:15.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.439+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92a1a0 con 0x55b74b926770 2024-05-09T03:19:15.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.440+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v114'330 uv330 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.440+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92a1a0 con 0x55b74b926770 2024-05-09T03:19:15.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.441+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v114'331 uv331 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.441+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.443+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v114'332 uv332 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.443+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.444+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v114'333 uv326 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.444+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.445+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v114'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.445+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.446+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v114'335 uv335 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.446 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:15.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.446+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.447+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v114'336 uv336 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.447+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b929190 con 0x55b74b926770 2024-05-09T03:19:15.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.448+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v114'337 uv337 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.448+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b74b929860 0x55b74b92bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.448+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c410 con 0x55b74b929860 2024-05-09T03:19:15.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.448+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b74b929860 0x55b74b92bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.449+0000 7f9d53ecb640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b74b929860 0x55b74b92bcc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.450+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v114'148 uv147 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b929860 2024-05-09T03:19:15.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.450+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b929860 2024-05-09T03:19:15.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.451+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v114'149 uv149 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b929860 2024-05-09T03:19:15.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.451+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b929860 2024-05-09T03:19:15.453 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:15.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.452+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v114'150 uv150 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b929860 2024-05-09T03:19:15.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.452+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b929860 2024-05-09T03:19:15.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.453+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v114'151 uv151 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b929860 2024-05-09T03:19:15.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.453+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b929860 2024-05-09T03:19:15.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.454+0000 7f9d53ecb640 1 -- 172.21.15.46:0/2666958323 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v114'152 uv152 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d44011450 con 0x55b74b929860 2024-05-09T03:19:15.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.454+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b878ad0 2024-05-09T03:19:15.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.455+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v114'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.455+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b878ad0 2024-05-09T03:19:15.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.456+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v114'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.457+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b878ad0 2024-05-09T03:19:15.458 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:15.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.458+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v114'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.458+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b878ad0 2024-05-09T03:19:15.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.459+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v114'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.459+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b878ad0 2024-05-09T03:19:15.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.460+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v114'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.460+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b878ad0 2024-05-09T03:19:15.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.461+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v114'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b878ad0 2024-05-09T03:19:15.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.461+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92c7e0 con 0x55b74b921f00 2024-05-09T03:19:15.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.462+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v114'294 uv288 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.462+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b92cc50 con 0x55b74b921f00 2024-05-09T03:19:15.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.463+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v114'295 uv295 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.463+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b92d040 con 0x55b74b921f00 2024-05-09T03:19:15.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.464+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v114'296 uv296 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.465 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:15.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.464+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925860 con 0x55b74b921f00 2024-05-09T03:19:15.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.465+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v114'297 uv297 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.465+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925860 con 0x55b74b921f00 2024-05-09T03:19:15.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.466+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v114'298 uv298 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.466+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925860 con 0x55b74b921f00 2024-05-09T03:19:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.467+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v114'299 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.467+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b925860 con 0x55b74b921f00 2024-05-09T03:19:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.468+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v114'300 uv300 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.468+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925860 con 0x55b74b921f00 2024-05-09T03:19:15.470 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.469+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v114'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.469+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b925860 con 0x55b74b921f00 2024-05-09T03:19:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.470+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v114'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.470+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b930600 con 0x55b74b921f00 2024-05-09T03:19:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.471+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v114'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.471+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b74b92e1b0 0x55b74b930a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.471+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b92e1b0 2024-05-09T03:19:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.472+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b74b92e1b0 0x55b74b930a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.472+0000 7f9d536ca640 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b74b92e1b0 0x55b74b930a20 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.473+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v114'128 uv127 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b92e1b0 2024-05-09T03:19:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.473+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b92e1b0 2024-05-09T03:19:15.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.474+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v114'129 uv129 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b92e1b0 2024-05-09T03:19:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.474+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b92e1b0 2024-05-09T03:19:15.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.475+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v114'130 uv130 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b92e1b0 2024-05-09T03:19:15.476 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:15.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.475+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b92e1b0 2024-05-09T03:19:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.476+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v114'131 uv131 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b92e1b0 2024-05-09T03:19:15.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.476+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b921f00 2024-05-09T03:19:15.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.478+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v114'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.478+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b921f00 2024-05-09T03:19:15.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.479+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v114'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.479+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b921f00 2024-05-09T03:19:15.480 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:15.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.480+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v114'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.480+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b921f00 2024-05-09T03:19:15.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.481+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v114'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.481+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b921f00 2024-05-09T03:19:15.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.482+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v114'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.482+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b921f00 2024-05-09T03:19:15.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.483+0000 7f9d536ca640 1 -- 172.21.15.46:0/2666958323 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v114'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4805c530 con 0x55b74b921f00 2024-05-09T03:19:15.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.483+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.485+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v114'151 uv150 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.485+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.486+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v114'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.486+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.487 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:15.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.487+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v114'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.487+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.488+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v114'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.488+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.490+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v114'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.490+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.491+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v114'363 uv355 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.491+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.492+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v114'364 uv364 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.492+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.494+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v114'365 uv365 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.494 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:15.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.494+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.495+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v114'366 uv366 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.495+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.496+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v114'367 uv367 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.496+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.497+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v114'368 uv362 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.497+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.499+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v114'369 uv369 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.499+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.500+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v114'370 uv370 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.500 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:15.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.500+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.501+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v114'371 uv371 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.501+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.502+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v114'372 uv372 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.502+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b9305d0 con 0x55b74b926770 2024-05-09T03:19:15.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.503+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v114'373 uv373 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.503+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55b74b930f60 con 0x55b74b926770 2024-05-09T03:19:15.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.504+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v114'374 uv374 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d4c04da90 con 0x55b74b926770 2024-05-09T03:19:15.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.504+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b74b878ad0 msgr2=0x55b74b912990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.504+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55b74b878ad0 0x55b74b912990 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b74b921f00 msgr2=0x55b74b924320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55b74b921f00 0x55b74b924320 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b74b92e1b0 msgr2=0x55b74b930a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55b74b92e1b0 0x55b74b930a20 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b74b91c7a0 msgr2=0x55b74b91f8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55b74b91c7a0 0x55b74b91f8d0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 reap_dead start 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b74b926770 msgr2=0x55b74b926b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b74b926770 0x55b74b926b80 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d546cc640 1 -- 172.21.15.46:0/2666958323 reap_dead start 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b74b929860 msgr2=0x55b74b92bcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b74b929860 0x55b74b92bcc0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d40074ce0 msgr2=0x7f9d400771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d40074ce0 0x7f9d400771f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400c610 tx=0x7f9d44009000 comp rx=0 tx=0).stop 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 msgr2=0x55b74b7e4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 0x55b74b7e4470 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009ef0 tx=0x7f9d4800f820 comp rx=0 tx=0).stop 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 shutdown_connections 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55b74b929860 0x55b74b92bcc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b74b926770 0x55b74b926b80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9d40074ce0 0x7f9d400771f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 --2- 172.21.15.46:0/2666958323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b74b8738a0 0x55b74b7e4470 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 >> 172.21.15.46:0/2666958323 conn(0x55b74b6c3ac0 msgr2=0x55b74b79e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 shutdown_connections 2024-05-09T03:19:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.505+0000 7f9d5ad1af40 1 -- 172.21.15.46:0/2666958323 wait complete. 2024-05-09T03:19:15.509 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 100.00000000.inode 0 2024-05-09T03:19:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.605+0000 7f91e7b0ff40 1 Processor -- start 2024-05-09T03:19:15.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.605+0000 7f91e7b0ff40 1 -- start start 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.606+0000 7f91e7b0ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea6033a0 0x55f0ea68ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.606+0000 7f91e7b0ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea696e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.606+0000 7f91e7b0ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea696a40 0x55f0ea697390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.606+0000 7f91e7b0ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f0ea569f50 con 0x55f0ea60b5f0 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.606+0000 7f91e7b0ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f0ea56df90 con 0x55f0ea696a40 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.606+0000 7f91e7b0ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f0ea60c8d0 con 0x55f0ea6033a0 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea696e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea696e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40572/0 (socket says 172.21.15.46:40572) 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2374999135 learned_addr learned my addr 172.21.15.46:0/2374999135 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e1dda640 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea696a40 0x55f0ea697390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e15d9640 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea6033a0 0x55f0ea68ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea6033a0 msgr2=0x55f0ea68ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea6033a0 0x55f0ea68ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea696a40 msgr2=0x55f0ea697390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea696a40 0x55f0ea697390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2374999135 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0ea5ee7f0 con 0x55f0ea60b5f0 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91e0dd8640 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea696e50 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f91d8001e10 tx=0x7f91d800b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b587c94ef55023e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91d37fe640 1 -- 172.21.15.46:0/2374999135 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8012020 con 0x55f0ea60b5f0 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91d37fe640 1 -- 172.21.15.46:0/2374999135 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d800e690 con 0x55f0ea60b5f0 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.607+0000 7f91d37fe640 1 -- 172.21.15.46:0/2374999135 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d800ea80 con 0x55f0ea60b5f0 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2374999135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 msgr2=0x55f0ea696e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea696e50 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f91d8001e10 tx=0x7f91d800b780 comp rx=0 tx=0).stop 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2374999135 shutdown_connections 2024-05-09T03:19:15.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea696a40 0x55f0ea697390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea696e50 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2374999135 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea6033a0 0x55f0ea68ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2374999135 >> 172.21.15.46:0/2374999135 conn(0x55f0ea4e5ac0 msgr2=0x55f0ea5f0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2374999135 shutdown_connections 2024-05-09T03:19:15.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.608+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2374999135 wait complete. 2024-05-09T03:19:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.609+0000 7f91e7b0ff40 1 Processor -- start 2024-05-09T03:19:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.609+0000 7f91e7b0ff40 1 -- start start 2024-05-09T03:19:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e7b0ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea6033a0 0x55f0ea691e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e7b0ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea72d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e7b0ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea696a40 0x55f0ea72df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e7b0ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f0ea68ef80 con 0x55f0ea60b5f0 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e7b0ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f0ea5eca90 con 0x55f0ea6033a0 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e7b0ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f0ea606d10 con 0x55f0ea696a40 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e0dd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea72d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e0dd8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea72d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40588/0 (socket says 172.21.15.46:40588) 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.610+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2187712409 learned_addr learned my addr 172.21.15.46:0/2187712409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e1dda640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea696a40 0x55f0ea72df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e15d9640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea6033a0 0x55f0ea691e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea696a40 msgr2=0x55f0ea72df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e0dd8640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f0ea696a40 0x55f0ea72df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea6033a0 msgr2=0x55f0ea691e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e0dd8640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0ea6033a0 0x55f0ea691e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e0dd8640 1 -- 172.21.15.46:0/2187712409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0ea606790 con 0x55f0ea60b5f0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e0dd8640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea72d9e0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f91d8010000 tx=0x7f91d80040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91d27fc640 1 -- 172.21.15.46:0/2187712409 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d801a640 con 0x55f0ea60b5f0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91d27fc640 1 -- 172.21.15.46:0/2187712409 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d801a800 con 0x55f0ea60b5f0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f0ea694570 con 0x55f0ea60b5f0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91d27fc640 1 -- 172.21.15.46:0/2187712409 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d801abf0 con 0x55f0ea60b5f0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.611+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0ea606a30 con 0x55f0ea60b5f0 2024-05-09T03:19:15.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.613+0000 7f91d27fc640 1 -- 172.21.15.46:0/2187712409 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f91d8015020 con 0x55f0ea60b5f0 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.613+0000 7f91d27fc640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91c4074ce0 0x7f91c40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.613+0000 7f91d27fc640 1 -- 172.21.15.46:0/2187712409 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7f91d809aa50 con 0x55f0ea60b5f0 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.613+0000 7f91e15d9640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91c4074ce0 0x7f91c40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.614+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f0ea68ddf0 0x55f0ea690fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.614+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f0ea691520 con 0x55f0ea68ddf0 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.614+0000 7f91e1dda640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f0ea68ddf0 0x55f0ea690fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.614+0000 7f91e15d9640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91c4074ce0 0x7f91c40771f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55f0ea5aea60 tx=0x7f91d4050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.614+0000 7f91e1dda640 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f0ea68ddf0 0x55f0ea690fe0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e1dda640 1 -- 172.21.15.46:0/2187712409 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 100.00000000.inode [truncate 0] v114'375 uv375 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f91dc0026a0 con 0x55f0ea68ddf0 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f0ea68ddf0 msgr2=0x55f0ea690fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55f0ea68ddf0 0x55f0ea690fe0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91c4074ce0 msgr2=0x7f91c40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f91c4074ce0 0x7f91c40771f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x55f0ea5aea60 tx=0x7f91d4050000 comp rx=0 tx=0).stop 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 msgr2=0x55f0ea72d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 --2- 172.21.15.46:0/2187712409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f0ea60b5f0 0x55f0ea72d9e0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f91d8010000 tx=0x7f91d80040e0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e1dda640 1 -- 172.21.15.46:0/2187712409 reap_dead start 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 shutdown_connections 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 >> 172.21.15.46:0/2187712409 conn(0x55f0ea4e5ac0 msgr2=0x55f0ea5f12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 shutdown_connections 2024-05-09T03:19:15.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.616+0000 7f91e7b0ff40 1 -- 172.21.15.46:0/2187712409 wait complete. 2024-05-09T03:19:15.619 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:15.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.737+0000 7fecdac54640 1 Processor -- start 2024-05-09T03:19:15.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.737+0000 7fecdac54640 1 -- start start 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecdac54640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fecd40d1740 0x7fecd40b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecdac54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd40d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecdac54640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecd40b9690 con 0x7fecd40cfcf0 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecdac54640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fecd408cd60 con 0x7fecd40d1740 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecdac54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fecd40ca9e0 con 0x7fecd40d1110 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecd91f1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fecd40cfcf0 0x7fecd409cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34968/0 (socket says 172.21.15.46:34968) 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.738+0000 7fecd91f1640 1 -- 172.21.15.46:0/2397236722 learned_addr learned my addr 172.21.15.46:0/2397236722 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd89f0640 1 --2- 172.21.15.46:0/2397236722 >> v2:172.21.15.88:3300/0 conn(0x7fecd40d1740 0x7fecd40b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd3fff640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd40d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd37fe640 1 -- 172.21.15.46:0/2397236722 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2837490496 0 0) 0x7fecd40b9690 con 0x7fecd40cfcf0 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd37fe640 1 -- 172.21.15.46:0/2397236722 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecc4003520 con 0x7fecd40cfcf0 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd37fe640 1 -- 172.21.15.46:0/2397236722 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 872037830 0 0) 0x7fecc4003520 con 0x7fecd40cfcf0 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd89f0640 1 -- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 msgr2=0x7fecd40d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd89f0640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd40d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd89f0640 1 -- 172.21.15.46:0/2397236722 >> v1:172.21.15.46:6789/0 conn(0x7fecd40cfcf0 legacy=0x7fecd409cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd89f0640 1 -- 172.21.15.46:0/2397236722 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd408cfb0 con 0x7fecd40d1740 2024-05-09T03:19:15.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.739+0000 7fecd3fff640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd40d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:15.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.740+0000 7fecd89f0640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd40b73d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fecc80097a0 tx=0x7fecc802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dada6fdd0a795b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.740+0000 7fecd37fe640 1 -- 172.21.15.46:0/2397236722 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8037070 con 0x7fecd40d1740 2024-05-09T03:19:15.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.740+0000 7fecd37fe640 1 -- 172.21.15.46:0/2397236722 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc802aba0 con 0x7fecd40d1740 2024-05-09T03:19:15.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.740+0000 7fecd37fe640 1 -- 172.21.15.46:0/2397236722 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc001a70 con 0x7fecd40d1740 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 -- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 msgr2=0x7fecd40b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd40b73d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fecc80097a0 tx=0x7fecc802a110 comp rx=0 tx=0).stop 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 -- 172.21.15.46:0/2397236722 shutdown_connections 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd40d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 --2- 172.21.15.46:0/2397236722 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd40b73d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 -- 172.21.15.46:0/2397236722 >> 172.21.15.46:0/2397236722 conn(0x7fecd40d05f0 msgr2=0x7fecd40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 -- 172.21.15.46:0/2397236722 shutdown_connections 2024-05-09T03:19:15.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.741+0000 7fecdac54640 1 -- 172.21.15.46:0/2397236722 wait complete. 2024-05-09T03:19:15.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.743+0000 7fecdac54640 1 Processor -- start 2024-05-09T03:19:15.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.743+0000 7fecdac54640 1 -- start start 2024-05-09T03:19:15.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecdac54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 0x7fecd41599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecdac54640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd414e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecdac54640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd415b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecdac54640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fecd409d3b0 con 0x7fecd40cfcf0 2024-05-09T03:19:15.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecdac54640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fecd40b93b0 con 0x7fecd40d1740 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecdac54640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fecd40d2930 con 0x7fecd40d1110 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 0x7fecd41599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 0x7fecd41599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40592/0 (socket says 172.21.15.46:40592) 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 -- 172.21.15.46:0/2514581731 learned_addr learned my addr 172.21.15.46:0/2514581731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd91f1640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd415b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd3fff640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd414e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 -- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 msgr2=0x7fecd414e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd414e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 -- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 msgr2=0x7fecd415b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd415b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.744+0000 7fecd89f0640 1 -- 172.21.15.46:0/2514581731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecd414e560 con 0x7fecd40cfcf0 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.745+0000 7fecd89f0640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 0x7fecd41599e0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fecc8035040 tx=0x7fecc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.745+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8041070 con 0x7fecd40cfcf0 2024-05-09T03:19:15.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.745+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc803c660 con 0x7fecd40cfcf0 2024-05-09T03:19:15.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.745+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecd4159100 con 0x7fecd40cfcf0 2024-05-09T03:19:15.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.745+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8037040 con 0x7fecd40cfcf0 2024-05-09T03:19:15.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.745+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecd41593e0 con 0x7fecd40cfcf0 2024-05-09T03:19:15.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.746+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecd40b65b0 con 0x7fecd40cfcf0 2024-05-09T03:19:15.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.747+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fecc8044070 con 0x7fecd40cfcf0 2024-05-09T03:19:15.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.747+0000 7fecd17fa640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecb8076300 0x7fecb8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:15.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.747+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8347+0+0 (secure 0 0 0) 0x7fecc80b86e0 con 0x7fecd40cfcf0 2024-05-09T03:19:15.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.747+0000 7fecd3fff640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecb8076300 0x7fecb8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:15.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.748+0000 7fecd3fff640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecb8076300 0x7fecb8078810 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fecd415d7d0 tx=0x7fecc0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:15.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.752+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fecd40b65b0 con 0x7fecd40cfcf0 2024-05-09T03:19:15.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:15.983+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fecd406a270 con 0x7fecd40cfcf0 2024-05-09T03:19:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.258+0000 7fecd17fa640 1 -- 172.21.15.46:0/2514581731 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v273) ==== 162+0+0 (secure 0 0 0) 0x7fecc8080fe0 con 0x7fecd40cfcf0 2024-05-09T03:19:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecb8076300 msgr2=0x7fecb8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecb8076300 0x7fecb8078810 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fecd415d7d0 tx=0x7fecc0009190 comp rx=0 tx=0).stop 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 msgr2=0x7fecd41599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 0x7fecd41599e0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fecc8035040 tx=0x7fecc80046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 shutdown_connections 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fecd40d1740 0x7fecd415b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fecd40d1110 0x7fecd414e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fecb8076300 0x7fecb8078810 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 --2- 172.21.15.46:0/2514581731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecd40cfcf0 0x7fecd41599e0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 >> 172.21.15.46:0/2514581731 conn(0x7fecd40d05f0 msgr2=0x7fecd415e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:16.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 shutdown_connections 2024-05-09T03:19:16.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.261+0000 7fecdac54640 1 -- 172.21.15.46:0/2514581731 wait complete. 2024-05-09T03:19:16.272 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:16.272+0000 7f15acdea640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-05-09T03:19:16.273 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:16.272+0000 7f15acdea640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:19:16.273 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:16.273+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:16.273 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:16.273+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:19:16.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:16.279 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:19:16.272+0000 7f15acdea640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-05-09T03:19:16.279 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:19:16.272+0000 7f15acdea640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-05-09T03:19:16.335 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.391+0000 7fd92e2bd640 1 Processor -- start 2024-05-09T03:19:16.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.391+0000 7fd92e2bd640 1 -- start start 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd92e2bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 0x7fd9280d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd92e2bd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd928161cd0 0x7fd9281620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd92e2bd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd92815e6e0 con 0x7fd928073500 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd92e2bd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd9280afa80 con 0x7fd928161cd0 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd92e2bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd92815e960 con 0x7fd9280af670 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 0x7fd9280d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 0x7fd9280d1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:40898/0 (socket says 172.21.15.46:40898) 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 -- 172.21.15.46:0/1973616982 learned_addr learned my addr 172.21.15.46:0/1973616982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd92c85a640 1 --2- 172.21.15.46:0/1973616982 >> v2:172.21.15.88:3300/0 conn(0x7fd928161cd0 0x7fd9281620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 -- 172.21.15.46:0/1973616982 >> v2:172.21.15.88:3300/0 conn(0x7fd928161cd0 msgr2=0x7fd9281620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 --2- 172.21.15.46:0/1973616982 >> v2:172.21.15.88:3300/0 conn(0x7fd928161cd0 0x7fd9281620e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 -- 172.21.15.46:0/1973616982 >> v1:172.21.15.88:6789/0 conn(0x7fd928073500 legacy=0x7fd928073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd9277fe640 1 -- 172.21.15.46:0/1973616982 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1044531523 0 0) 0x7fd910001110 con 0x7fd928073500 2024-05-09T03:19:16.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.393+0000 7fd927fff640 1 -- 172.21.15.46:0/1973616982 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92815ec40 con 0x7fd9280af670 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.394+0000 7fd927fff640 1 --2- 172.21.15.46:0/1973616982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 0x7fd9280d1170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd918009870 tx=0x7fd91802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2d92de3d545b48d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.394+0000 7fd9277fe640 1 -- 172.21.15.46:0/1973616982 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918037070 con 0x7fd9280af670 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.394+0000 7fd9277fe640 1 -- 172.21.15.46:0/1973616982 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd918032440 con 0x7fd9280af670 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.394+0000 7fd9277fe640 1 -- 172.21.15.46:0/1973616982 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918032810 con 0x7fd9280af670 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 -- 172.21.15.46:0/1973616982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 msgr2=0x7fd9280d1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/1973616982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 0x7fd9280d1170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd918009870 tx=0x7fd91802a270 comp rx=0 tx=0).stop 2024-05-09T03:19:16.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 -- 172.21.15.46:0/1973616982 shutdown_connections 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/1973616982 >> v2:172.21.15.88:3300/0 conn(0x7fd928161cd0 0x7fd9281620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/1973616982 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd9280af670 0x7fd9280d1170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 -- 172.21.15.46:0/1973616982 >> 172.21.15.46:0/1973616982 conn(0x7fd9280b3910 msgr2=0x7fd9280b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 -- 172.21.15.46:0/1973616982 shutdown_connections 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.395+0000 7fd92e2bd640 1 -- 172.21.15.46:0/1973616982 wait complete. 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.397+0000 7fd92e2bd640 1 Processor -- start 2024-05-09T03:19:16.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.397+0000 7fd92e2bd640 1 -- start start 2024-05-09T03:19:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd92e2bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 0x7fd92815f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd92e2bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9280af670 0x7fd928142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd92e2bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd928161cd0 0x7fd92813f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd92e2bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd9280aeb80 con 0x7fd928073500 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd92e2bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd92815f200 con 0x7fd9280af670 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd92e2bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd9280aed00 con 0x7fd928161cd0 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd927fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 0x7fd92815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd927fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 0x7fd92815f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36536/0 (socket says 172.21.15.46:36536) 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.398+0000 7fd927fff640 1 -- 172.21.15.46:0/4284712167 learned_addr learned my addr 172.21.15.46:0/4284712167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd91ffff640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9280af670 0x7fd928142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd92c85a640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd928161cd0 0x7fd92813f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd927fff640 1 -- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd928161cd0 msgr2=0x7fd92813f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd927fff640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd928161cd0 0x7fd92813f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd927fff640 1 -- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9280af670 msgr2=0x7fd928142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd927fff640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9280af670 0x7fd928142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd927fff640 1 -- 172.21.15.46:0/4284712167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92815ffc0 con 0x7fd928073500 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd92c85a640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd928161cd0 0x7fd92813f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:16.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd927fff640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 0x7fd92815f770 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fd918009870 tx=0x7fd9180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918041020 con 0x7fd928073500 2024-05-09T03:19:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd918030660 con 0x7fd928073500 2024-05-09T03:19:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9281602a0 con 0x7fd928073500 2024-05-09T03:19:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918037040 con 0x7fd928073500 2024-05-09T03:19:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.399+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd928160860 con 0x7fd928073500 2024-05-09T03:19:16.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.400+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92806a270 con 0x7fd928073500 2024-05-09T03:19:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.401+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd918044020 con 0x7fd928073500 2024-05-09T03:19:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.401+0000 7fd9257fa640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd908076300 0x7fd908078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.401+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7fd9180b88f0 con 0x7fd928073500 2024-05-09T03:19:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.402+0000 7fd91ffff640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd908076300 0x7fd908078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.402+0000 7fd91ffff640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd908076300 0x7fd908078810 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd928150950 tx=0x7fd910009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.406+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd91804b030 con 0x7fd928073500 2024-05-09T03:19:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.635+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd92806e4c0 con 0x7fd928073500 2024-05-09T03:19:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.636+0000 7fd9257fa640 1 -- 172.21.15.46:0/4284712167 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) ==== 95+0+4343 (secure 0 0 0) 0x7fd9180816d0 con 0x7fd928073500 2024-05-09T03:19:16.637 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:16.637 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":275,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:16.273928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 275 2024-05-09T03:19:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd908076300 msgr2=0x7fd908078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd908076300 0x7fd908078810 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd928150950 tx=0x7fd910009040 comp rx=0 tx=0).stop 2024-05-09T03:19:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 msgr2=0x7fd92815f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 0x7fd92815f770 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fd918009870 tx=0x7fd9180046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 shutdown_connections 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd928161cd0 0x7fd92813f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9280af670 0x7fd928142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd908076300 0x7fd908078810 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 --2- 172.21.15.46:0/4284712167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd928073500 0x7fd92815f770 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 >> 172.21.15.46:0/4284712167 conn(0x7fd9280b3910 msgr2=0x7fd92815a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 shutdown_connections 2024-05-09T03:19:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.638+0000 7fd92e2bd640 1 -- 172.21.15.46:0/4284712167 wait complete. 2024-05-09T03:19:16.650 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:16.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:16.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.805+0000 7f4597449640 1 Processor -- start 2024-05-09T03:19:16.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.805+0000 7f4597449640 1 -- start start 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f4597449640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45900b5210 0x7f45901613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f4597449640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f45900b7ed0 0x7f45900d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f4597449640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45900d5280 con 0x7f45900d4800 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f4597449640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f45900b5620 con 0x7f45900b5210 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f4597449640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f45901609a0 con 0x7f45900b7ed0 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45900b5210 0x7f45901613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f45900b5210 0x7f45901613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36538/0 (socket says 172.21.15.46:36538) 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 -- 172.21.15.46:0/1270287690 learned_addr learned my addr 172.21.15.46:0/1270287690 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45959e6640 1 --2- 172.21.15.46:0/1270287690 >> v2:172.21.15.88:3300/0 conn(0x7f45900b7ed0 0x7f45900d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 -- 172.21.15.46:0/1270287690 >> v2:172.21.15.88:3300/0 conn(0x7f45900b7ed0 msgr2=0x7f45900d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 --2- 172.21.15.46:0/1270287690 >> v2:172.21.15.88:3300/0 conn(0x7f45900b7ed0 0x7f45900d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 -- 172.21.15.46:0/1270287690 >> v1:172.21.15.88:6789/0 conn(0x7f45900d4800 legacy=0x7f45900b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45951e5640 1 -- 172.21.15.46:0/1270287690 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45900d5560 con 0x7f45900b5210 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.807+0000 7f45959e6640 1 --2- 172.21.15.46:0/1270287690 >> v2:172.21.15.88:3300/0 conn(0x7f45900b7ed0 0x7f45900d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.808+0000 7f45951e5640 1 --2- 172.21.15.46:0/1270287690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45901613c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f459006f030 tx=0x7f458002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72785506aa827f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.808+0000 7f458ffff640 1 -- 172.21.15.46:0/1270287690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580036070 con 0x7f45900b5210 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.808+0000 7f458ffff640 1 -- 172.21.15.46:0/1270287690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4580031600 con 0x7f45900b5210 2024-05-09T03:19:16.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.808+0000 7f458ffff640 1 -- 172.21.15.46:0/1270287690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800319d0 con 0x7f45900b5210 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 -- 172.21.15.46:0/1270287690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 msgr2=0x7f45901613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 --2- 172.21.15.46:0/1270287690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45901613c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f459006f030 tx=0x7f458002a610 comp rx=0 tx=0).stop 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 -- 172.21.15.46:0/1270287690 shutdown_connections 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 --2- 172.21.15.46:0/1270287690 >> v2:172.21.15.88:3300/0 conn(0x7f45900b7ed0 0x7f45900d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 --2- 172.21.15.46:0/1270287690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45901613c0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 -- 172.21.15.46:0/1270287690 >> 172.21.15.46:0/1270287690 conn(0x7f45900d29a0 msgr2=0x7f45900d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 -- 172.21.15.46:0/1270287690 shutdown_connections 2024-05-09T03:19:16.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.809+0000 7f4597449640 1 -- 172.21.15.46:0/1270287690 wait complete. 2024-05-09T03:19:16.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.811+0000 7f4597449640 1 Processor -- start 2024-05-09T03:19:16.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.811+0000 7f4597449640 1 -- start start 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f4597449640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45900cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f4597449640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45900d4800 0x7f45901f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f4597449640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45901f75e0 0x7f45900bab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f4597449640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45900d0ae0 con 0x7f45900b5210 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f4597449640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4590160ea0 con 0x7f45900d4800 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f4597449640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4590163e60 con 0x7f45901f75e0 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f45951e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45900cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f45951e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45900cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36552/0 (socket says 172.21.15.46:36552) 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.812+0000 7f45951e5640 1 -- 172.21.15.46:0/1560750889 learned_addr learned my addr 172.21.15.46:0/1560750889 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45949e4640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45900d4800 0x7f45901f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45959e6640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45901f75e0 0x7f45900bab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45951e5640 1 -- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45901f75e0 msgr2=0x7f45900bab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45951e5640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45901f75e0 0x7f45900bab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45951e5640 1 -- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45900d4800 msgr2=0x7f45901f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45951e5640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45900d4800 0x7f45901f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45951e5640 1 -- 172.21.15.46:0/1560750889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45901f6840 con 0x7f45900b5210 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45949e4640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45900d4800 0x7f45901f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f45951e5640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45900cf5e0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f4580034040 tx=0x7f4580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580036050 con 0x7f45900b5210 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4580042070 con 0x7f45900b5210 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45901f9f70 con 0x7f45900b5210 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4580009510 con 0x7f45900b5210 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.813+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45901fa530 con 0x7f45900b5210 2024-05-09T03:19:16.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.814+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45900d2db0 con 0x7f45900b5210 2024-05-09T03:19:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.815+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4580007c30 con 0x7f45900b5210 2024-05-09T03:19:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.816+0000 7f458dffb640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f457c0762d0 0x7f457c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.816+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f45800b9c20 con 0x7f45900b5210 2024-05-09T03:19:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.816+0000 7f45949e4640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f457c0762d0 0x7f457c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:16.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.817+0000 7f45949e4640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f457c0762d0 0x7f457c0787e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f459015dfb0 tx=0x7f4584008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:16.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:16.820+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f45900d2db0 con 0x7f45900b5210 2024-05-09T03:19:17.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.053+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45900d4f10 con 0x7f45900b5210 2024-05-09T03:19:17.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.053+0000 7f458dffb640 1 -- 172.21.15.46:0/1560750889 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) ==== 95+0+4343 (secure 0 0 0) 0x7f45900d4f10 con 0x7f45900b5210 2024-05-09T03:19:17.054 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:17.054 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":275,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:16.273928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:17.054 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 275 2024-05-09T03:19:17.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.055+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f457c0762d0 msgr2=0x7f457c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.055+0000 7f4597449640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f457c0762d0 0x7f457c0787e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f459015dfb0 tx=0x7f4584008040 comp rx=0 tx=0).stop 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.055+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 msgr2=0x7f45900cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.055+0000 7f4597449640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45900cf5e0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f4580034040 tx=0x7f4580004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 shutdown_connections 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f45901f75e0 0x7f45900bab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45900d4800 0x7f45901f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f457c0762d0 0x7f457c0787e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 --2- 172.21.15.46:0/1560750889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45900b5210 0x7f45900cf5e0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 >> 172.21.15.46:0/1560750889 conn(0x7f45900d29a0 msgr2=0x7f4590163220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 shutdown_connections 2024-05-09T03:19:17.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.056+0000 7f4597449640 1 -- 172.21.15.46:0/1560750889 wait complete. 2024-05-09T03:19:17.067 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:17.067 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.181+0000 7f326803b640 1 Processor -- start 2024-05-09T03:19:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.181+0000 7f326803b640 1 -- start start 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f326803b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3260160610 0x7f32600b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f326803b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32600b7d20 0x7f32600b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f326803b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32600b8130 con 0x7f32600d57a0 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f326803b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f32600d5bb0 con 0x7f32600b7d20 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f326803b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f32600d49e0 con 0x7f3260160610 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32600b7d20 0x7f32600b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32600b7d20 0x7f32600b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36554/0 (socket says 172.21.15.46:36554) 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 -- 172.21.15.46:0/2142310437 learned_addr learned my addr 172.21.15.46:0/2142310437 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32655d6640 1 --2- 172.21.15.46:0/2142310437 >> v2:172.21.15.88:3300/0 conn(0x7f3260160610 0x7f32600b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 -- 172.21.15.46:0/2142310437 >> v2:172.21.15.88:3300/0 conn(0x7f3260160610 msgr2=0x7f32600b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 --2- 172.21.15.46:0/2142310437 >> v2:172.21.15.88:3300/0 conn(0x7f3260160610 0x7f32600b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 -- 172.21.15.46:0/2142310437 >> v1:172.21.15.88:6789/0 conn(0x7f32600d57a0 legacy=0x7f32600b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.183+0000 7f32665d8640 1 -- 172.21.15.46:0/2142310437 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32600d4cc0 con 0x7f32600b7d20 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.184+0000 7f32665d8640 1 --2- 172.21.15.46:0/2142310437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600b7d20 0x7f32600b9450 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f32600d2f80 tx=0x7f325802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24fdaffc828c009 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.184+0000 7f3264dd5640 1 -- 172.21.15.46:0/2142310437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258036070 con 0x7f32600b7d20 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.184+0000 7f3264dd5640 1 -- 172.21.15.46:0/2142310437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3258031600 con 0x7f32600b7d20 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.184+0000 7f3264dd5640 1 -- 172.21.15.46:0/2142310437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32580319d0 con 0x7f32600b7d20 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 -- 172.21.15.46:0/2142310437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600b7d20 msgr2=0x7f32600b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 --2- 172.21.15.46:0/2142310437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600b7d20 0x7f32600b9450 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f32600d2f80 tx=0x7f325802a730 comp rx=0 tx=0).stop 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 -- 172.21.15.46:0/2142310437 shutdown_connections 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 --2- 172.21.15.46:0/2142310437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600b7d20 0x7f32600b9450 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 --2- 172.21.15.46:0/2142310437 >> v2:172.21.15.88:3300/0 conn(0x7f3260160610 0x7f32600b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 -- 172.21.15.46:0/2142310437 >> 172.21.15.46:0/2142310437 conn(0x7f32600743e0 msgr2=0x7f32600d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:17.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 -- 172.21.15.46:0/2142310437 shutdown_connections 2024-05-09T03:19:17.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.185+0000 7f326803b640 1 -- 172.21.15.46:0/2142310437 wait complete. 2024-05-09T03:19:17.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.187+0000 7f326803b640 1 Processor -- start 2024-05-09T03:19:17.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.187+0000 7f326803b640 1 -- start start 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f326803b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32600b7d20 0x7f32601e9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f326803b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 0x7f326015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f326803b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260160610 0x7f3260159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f326803b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3260160a20 con 0x7f32600d57a0 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f326803b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32600d5280 con 0x7f32600b7d20 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f326803b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f32600d15b0 con 0x7f3260160610 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f32655d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 0x7f326015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f32655d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 0x7f326015e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36564/0 (socket says 172.21.15.46:36564) 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f32655d6640 1 -- 172.21.15.46:0/1507383586 learned_addr learned my addr 172.21.15.46:0/1507383586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f3265dd7640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32600b7d20 0x7f32601e9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f32665d8640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260160610 0x7f3260159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f32655d6640 1 -- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260160610 msgr2=0x7f3260159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.188+0000 7f32655d6640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260160610 0x7f3260159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32655d6640 1 -- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32600b7d20 msgr2=0x7f32601e9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32655d6640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32600b7d20 0x7f32601e9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32655d6640 1 -- 172.21.15.46:0/1507383586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32601fa3e0 con 0x7f32600d57a0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32655d6640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 0x7f326015e2d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f325000d5f0 tx=0x7f325000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500125f0 con 0x7f32600d57a0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3250019070 con 0x7f32600d57a0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32601fa6c0 con 0x7f32600d57a0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32500129d0 con 0x7f32600d57a0 2024-05-09T03:19:17.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.189+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601fac80 con 0x7f32600d57a0 2024-05-09T03:19:17.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.190+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32601fa130 con 0x7f32600d57a0 2024-05-09T03:19:17.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.191+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3250012b70 con 0x7f32600d57a0 2024-05-09T03:19:17.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.191+0000 7f32567fc640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32440762d0 0x7f32440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.191+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f3250098440 con 0x7f32600d57a0 2024-05-09T03:19:17.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.192+0000 7f3265dd7640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32440762d0 0x7f32440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.192+0000 7f3265dd7640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32440762d0 0x7f32440787e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f326006d3b0 tx=0x7f325c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:17.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.196+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3250060bc0 con 0x7f32600d57a0 2024-05-09T03:19:17.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.434+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f326006a270 con 0x7f32600d57a0 2024-05-09T03:19:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.435+0000 7f32567fc640 1 -- 172.21.15.46:0/1507383586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 276 v276) ==== 95+0+5169 (secure 0 0 0) 0x7f3250064ec0 con 0x7f32600d57a0 2024-05-09T03:19:17.436 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:17.436 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":276,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1350199591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1350199591},{"type":"v1","addr":"172.21.15.46:6834","nonce":1350199591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":276}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:16.273928+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:17.436 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 276 2024-05-09T03:19:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32440762d0 msgr2=0x7f32440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32440762d0 0x7f32440787e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f326006d3b0 tx=0x7f325c009040 comp rx=0 tx=0).stop 2024-05-09T03:19:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 msgr2=0x7f326015e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 0x7f326015e2d0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f325000d5f0 tx=0x7f325000dab0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 shutdown_connections 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3260160610 0x7f3260159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f32440762d0 0x7f32440787e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600d57a0 0x7f326015e2d0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 --2- 172.21.15.46:0/1507383586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32600b7d20 0x7f32601e9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 >> 172.21.15.46:0/1507383586 conn(0x7f32600743e0 msgr2=0x7f32600b75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 shutdown_connections 2024-05-09T03:19:17.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.437+0000 7f326803b640 1 -- 172.21.15.46:0/1507383586 wait complete. 2024-05-09T03:19:17.449 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 100.00000000.inode' led to DAMAGED state 2024-05-09T03:19:17.449 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 10000000000.00000000:sixmegs_head' 2024-05-09T03:19:17.450 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:17.450 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:17.467 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:19:17.467 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:17.467 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:17.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:17.467 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:19:17.523 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:19:17.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.677+0000 7f4ac2ad9640 1 Processor -- start 2024-05-09T03:19:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.678+0000 7f4ac2ad9640 1 -- start start 2024-05-09T03:19:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac2ad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4abc0cbc20 0x7f4abc06c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac2ad9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cc350 0x7f4abc163f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac2ad9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4abc164450 0x7f4abc164860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac2ad9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4abc0b77f0 con 0x7f4abc0cbc20 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac2ad9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4abc068c70 con 0x7f4abc164450 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac2ad9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4abc0b7a40 con 0x7f4abc0cc350 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac0875640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4abc0cbc20 0x7f4abc06c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac0875640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4abc0cbc20 0x7f4abc06c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36580/0 (socket says 172.21.15.46:36580) 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac0875640 1 -- 172.21.15.46:0/3460027231 learned_addr learned my addr 172.21.15.46:0/3460027231 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac0875640 1 -- 172.21.15.46:0/3460027231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cc350 msgr2=0x7f4abc163f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4ac1076640 1 --2- 172.21.15.46:0/3460027231 >> v2:172.21.15.88:3300/0 conn(0x7f4abc164450 0x7f4abc164860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.679+0000 7f4abbfff640 1 --2- 172.21.15.46:0/3460027231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cc350 0x7f4abc163f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3460027231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cc350 0x7f4abc163f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4ac0875640 1 -- 172.21.15.46:0/3460027231 >> v2:172.21.15.88:3300/0 conn(0x7f4abc164450 msgr2=0x7f4abc164860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3460027231 >> v2:172.21.15.88:3300/0 conn(0x7f4abc164450 0x7f4abc164860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4ac0875640 1 -- 172.21.15.46:0/3460027231 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4abc163ba0 con 0x7f4abc0cbc20 2024-05-09T03:19:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4ac1076640 1 --2- 172.21.15.46:0/3460027231 >> v2:172.21.15.88:3300/0 conn(0x7f4abc164450 0x7f4abc164860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3460027231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cbc20 0x7f4abc06c680 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f4abc069260 tx=0x7f4ab002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58e805a06e28326 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.680+0000 7f4abb7fe640 1 -- 172.21.15.46:0/3460027231 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0037070 con 0x7f4abc0cbc20 2024-05-09T03:19:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4abb7fe640 1 -- 172.21.15.46:0/3460027231 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab0032440 con 0x7f4abc0cbc20 2024-05-09T03:19:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4abb7fe640 1 -- 172.21.15.46:0/3460027231 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0032810 con 0x7f4abc0cbc20 2024-05-09T03:19:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3460027231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cbc20 msgr2=0x7f4abc06c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3460027231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cbc20 0x7f4abc06c680 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f4abc069260 tx=0x7f4ab002a140 comp rx=0 tx=0).stop 2024-05-09T03:19:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3460027231 shutdown_connections 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3460027231 >> v2:172.21.15.88:3300/0 conn(0x7f4abc164450 0x7f4abc164860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3460027231 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cc350 0x7f4abc163f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3460027231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cbc20 0x7f4abc06c680 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3460027231 >> 172.21.15.46:0/3460027231 conn(0x7f4abc09b7d0 msgr2=0x7f4abc0b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3460027231 shutdown_connections 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.681+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3460027231 wait complete. 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.683+0000 7f4ac2ad9640 1 Processor -- start 2024-05-09T03:19:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.683+0000 7f4ac2ad9640 1 -- start start 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.684+0000 7f4ac2ad9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cbc20 0x7f4abc0d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.684+0000 7f4ac2ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 0x7f4abc0d1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.684+0000 7f4ac2ad9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4abc164450 0x7f4abc1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.684+0000 7f4ac2ad9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4abc0cedd0 con 0x7f4abc0cc350 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.684+0000 7f4ac2ad9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4abc0cec50 con 0x7f4abc164450 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.684+0000 7f4ac2ad9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4abc0cef50 con 0x7f4abc0cbc20 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 0x7f4abc0d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 0x7f4abc0d1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36596/0 (socket says 172.21.15.46:36596) 2024-05-09T03:19:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 -- 172.21.15.46:0/3224182156 learned_addr learned my addr 172.21.15.46:0/3224182156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ac1076640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4abc164450 0x7f4abc1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cbc20 0x7f4abc0d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 -- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cbc20 msgr2=0x7f4abc0d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cbc20 0x7f4abc0d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 -- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4abc164450 msgr2=0x7f4abc1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4abc164450 0x7f4abc1548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 -- 172.21.15.46:0/3224182156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4abc0d1ed0 con 0x7f4abc0cc350 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cbc20 0x7f4abc0d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4abbfff640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 0x7f4abc0d1990 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800c5a0 tx=0x7f4aa800ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8015070 con 0x7f4abc0cc350 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa8004220 con 0x7f4abc0cc350 2024-05-09T03:19:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8003610 con 0x7f4abc0cc350 2024-05-09T03:19:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.685+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4abc154df0 con 0x7f4abc0cc350 2024-05-09T03:19:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.686+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4abc0cff30 con 0x7f4abc0cc350 2024-05-09T03:19:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.686+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4abc0d2670 con 0x7f4abc0cc350 2024-05-09T03:19:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.687+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4aa80037b0 con 0x7f4abc0cc350 2024-05-09T03:19:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.688+0000 7f4ab97fa640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4aa00762d0 0x7f4aa00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.688+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f4aa8097e60 con 0x7f4abc0cc350 2024-05-09T03:19:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.688+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4aa00762d0 0x7f4aa00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.689+0000 7f4ac0875640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4aa00762d0 0x7f4aa00787e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0034000 tx=0x7f4ab002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.692+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4aa8060610 con 0x7f4abc0cc350 2024-05-09T03:19:17.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:17.921+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4abc06a270 con 0x7f4abc0cc350 2024-05-09T03:19:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.280+0000 7f4ab97fa640 1 -- 172.21.15.46:0/3224182156 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v277) ==== 161+0+0 (secure 0 0 0) 0x7f4aa8064910 con 0x7f4abc0cc350 2024-05-09T03:19:18.281 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:18.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.282+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4aa00762d0 msgr2=0x7f4aa00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:18.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.282+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4aa00762d0 0x7f4aa00787e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0034000 tx=0x7f4ab002e000 comp rx=0 tx=0).stop 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.282+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 msgr2=0x7f4abc0d1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.282+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 0x7f4abc0d1990 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800c5a0 tx=0x7f4aa800ca60 comp rx=0 tx=0).stop 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 shutdown_connections 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4abc164450 0x7f4abc1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4aa00762d0 0x7f4aa00787e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4abc0cc350 0x7f4abc0d1990 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 --2- 172.21.15.46:0/3224182156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4abc0cbc20 0x7f4abc0d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 >> 172.21.15.46:0/3224182156 conn(0x7f4abc09b7d0 msgr2=0x7f4abc0cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 shutdown_connections 2024-05-09T03:19:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.283+0000 7f4ac2ad9640 1 -- 172.21.15.46:0/3224182156 wait complete. 2024-05-09T03:19:18.296 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.412+0000 7fc9efabd640 1 Processor -- start 2024-05-09T03:19:18.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.412+0000 7fc9efabd640 1 -- start start 2024-05-09T03:19:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9efabd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9e815e0c0 0x7fc9e8160930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9efabd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e80b5220 0x7fc9e8073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9efabd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9e8073b40 0x7fc9e8163f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9efabd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e80af490 con 0x7fc9e815e0c0 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9efabd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e8074880 con 0x7fc9e8073b40 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9efabd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e8073f50 con 0x7fc9e80b5220 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9e815e0c0 0x7fc9e8160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc9e815e0c0 0x7fc9e8160930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36612/0 (socket says 172.21.15.46:36612) 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 -- 172.21.15.46:0/252321812 learned_addr learned my addr 172.21.15.46:0/252321812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ee05a640 1 --2- 172.21.15.46:0/252321812 >> v2:172.21.15.88:3300/0 conn(0x7fc9e8073b40 0x7fc9e8163f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed058640 1 --2- 172.21.15.46:0/252321812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e80b5220 0x7fc9e8073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 -- 172.21.15.46:0/252321812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e80b5220 msgr2=0x7fc9e8073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 --2- 172.21.15.46:0/252321812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e80b5220 0x7fc9e8073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 -- 172.21.15.46:0/252321812 >> v2:172.21.15.88:3300/0 conn(0x7fc9e8073b40 msgr2=0x7fc9e8163f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 --2- 172.21.15.46:0/252321812 >> v2:172.21.15.88:3300/0 conn(0x7fc9e8073b40 0x7fc9e8163f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.414+0000 7fc9ed859640 1 -- 172.21.15.46:0/252321812 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e80b6530 con 0x7fc9e815e0c0 2024-05-09T03:19:18.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.415+0000 7fc9ed859640 1 --2- 172.21.15.46:0/252321812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8160930 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fc9e806ef10 tx=0x7fc9d802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48726d057edea500 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.415+0000 7fc9ec857640 1 -- 172.21.15.46:0/252321812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8036070 con 0x7fc9e815e0c0 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.415+0000 7fc9ec857640 1 -- 172.21.15.46:0/252321812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d8031600 con 0x7fc9e815e0c0 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.415+0000 7fc9ec857640 1 -- 172.21.15.46:0/252321812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d80319d0 con 0x7fc9e815e0c0 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 -- 172.21.15.46:0/252321812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 msgr2=0x7fc9e8160930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 --2- 172.21.15.46:0/252321812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8160930 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fc9e806ef10 tx=0x7fc9d802a610 comp rx=0 tx=0).stop 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 -- 172.21.15.46:0/252321812 shutdown_connections 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 --2- 172.21.15.46:0/252321812 >> v2:172.21.15.88:3300/0 conn(0x7fc9e8073b40 0x7fc9e8163f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 --2- 172.21.15.46:0/252321812 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e80b5220 0x7fc9e8073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 --2- 172.21.15.46:0/252321812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8160930 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 -- 172.21.15.46:0/252321812 >> 172.21.15.46:0/252321812 conn(0x7fc9e80b3980 msgr2=0x7fc9e80b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 -- 172.21.15.46:0/252321812 shutdown_connections 2024-05-09T03:19:18.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.416+0000 7fc9efabd640 1 -- 172.21.15.46:0/252321812 wait complete. 2024-05-09T03:19:18.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.418+0000 7fc9efabd640 1 Processor -- start 2024-05-09T03:19:18.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.418+0000 7fc9efabd640 1 -- start start 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9efabd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e8073b40 0x7fc9e8158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9efabd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9e80b5220 0x7fc9e81f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9efabd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8162a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9efabd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e80b5ee0 con 0x7fc9e815e0c0 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9efabd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e80b5c00 con 0x7fc9e80b5220 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9efabd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e80b61c0 con 0x7fc9e8073b40 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9ee05a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8162a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9ee05a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8162a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36618/0 (socket says 172.21.15.46:36618) 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9ee05a640 1 -- 172.21.15.46:0/3166722533 learned_addr learned my addr 172.21.15.46:0/3166722533 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9ed859640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e8073b40 0x7fc9e8158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.419+0000 7fc9ed058640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9e80b5220 0x7fc9e81f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9ee05a640 1 -- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e8073b40 msgr2=0x7fc9e8158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9ee05a640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e8073b40 0x7fc9e8158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9ee05a640 1 -- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9e80b5220 msgr2=0x7fc9e81f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9ee05a640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9e80b5220 0x7fc9e81f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9ee05a640 1 -- 172.21.15.46:0/3166722533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e81585b0 con 0x7fc9e815e0c0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9ee05a640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8162a20 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000e7f0 tx=0x7fc9e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0017070 con 0x7fc9e815e0c0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e000c730 con 0x7fc9e815e0c0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e8156fd0 con 0x7fc9e815e0c0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000cb40 con 0x7fc9e815e0c0 2024-05-09T03:19:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.420+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e8157590 con 0x7fc9e815e0c0 2024-05-09T03:19:18.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.421+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e806a270 con 0x7fc9e815e0c0 2024-05-09T03:19:18.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.422+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc9e001a070 con 0x7fc9e815e0c0 2024-05-09T03:19:18.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.422+0000 7fc9e67fc640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9d40762d0 0x7fc9d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:18.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.423+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7fc9e00985d0 con 0x7fc9e815e0c0 2024-05-09T03:19:18.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.423+0000 7fc9ed859640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9d40762d0 0x7fc9d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:18.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.423+0000 7fc9ed859640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9d40762d0 0x7fc9d40787e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8033000 tx=0x7fc9d8009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:18.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.427+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc9e0060e00 con 0x7fc9e815e0c0 2024-05-09T03:19:18.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:18.693+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc9e80714d0 con 0x7fc9e815e0c0 2024-05-09T03:19:19.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.289+0000 7fc9e67fc640 1 -- 172.21.15.46:0/3166722533 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v278) ==== 101+0+0 (secure 0 0 0) 0x7fc9e0065100 con 0x7fc9e815e0c0 2024-05-09T03:19:19.290 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9d40762d0 msgr2=0x7fc9d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9d40762d0 0x7fc9d40787e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8033000 tx=0x7fc9d8009000 comp rx=0 tx=0).stop 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 msgr2=0x7fc9e8162a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8162a20 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000e7f0 tx=0x7fc9e000ecb0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 shutdown_connections 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc9d40762d0 0x7fc9d40787e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9e815e0c0 0x7fc9e8162a20 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9e80b5220 0x7fc9e81f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 --2- 172.21.15.46:0/3166722533 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9e8073b40 0x7fc9e8158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 >> 172.21.15.46:0/3166722533 conn(0x7fc9e80b3980 msgr2=0x7fc9e806f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:19.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 shutdown_connections 2024-05-09T03:19:19.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.292+0000 7fc9efabd640 1 -- 172.21.15.46:0/3166722533 wait complete. 2024-05-09T03:19:19.306 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.364+0000 7fe2ec022f40 1 Processor -- start 2024-05-09T03:19:19.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.364+0000 7fe2ec022f40 1 -- start start 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.365+0000 7fe2ec022f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 0x56409730b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.365+0000 7fe2ec022f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x564097311400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.365+0000 7fe2ec022f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973b5a40 0x5640973b7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.365+0000 7fe2ec022f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564097288f50 con 0x5640973223a0 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.365+0000 7fe2ec022f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56409728cf90 con 0x56409732a5f0 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.365+0000 7fe2ec022f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56409732b8d0 con 0x5640973b5a40 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 0x56409730b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 0x56409730b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36622/0 (socket says 172.21.15.46:36622) 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 -- 172.21.15.46:0/3197866253 learned_addr learned my addr 172.21.15.46:0/3197866253 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x564097311400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e62eb640 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973b5a40 0x5640973b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 -- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973b5a40 msgr2=0x5640973b7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:19.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973b5a40 0x5640973b7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 -- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 msgr2=0x564097311400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x564097311400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 -- 172.21.15.46:0/3197866253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564097310aa0 con 0x5640973223a0 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 0x56409730b9f0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fe2d805a040 tx=0x7fe2d8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc714d3533462493 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2d7fff640 1 -- 172.21.15.46:0/3197866253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8067070 con 0x5640973223a0 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2d7fff640 1 -- 172.21.15.46:0/3197866253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d80577f0 con 0x5640973223a0 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.366+0000 7fe2d7fff640 1 -- 172.21.15.46:0/3197866253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d805c040 con 0x5640973223a0 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 -- 172.21.15.46:0/3197866253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 msgr2=0x56409730b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 0x56409730b9f0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fe2d805a040 tx=0x7fe2d8051000 comp rx=0 tx=0).stop 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 -- 172.21.15.46:0/3197866253 shutdown_connections 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973b5a40 0x5640973b7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x564097311400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/3197866253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973223a0 0x56409730b9f0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 -- 172.21.15.46:0/3197866253 >> 172.21.15.46:0/3197866253 conn(0x564097204ac0 msgr2=0x56409732c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:19.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 -- 172.21.15.46:0/3197866253 shutdown_connections 2024-05-09T03:19:19.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.367+0000 7fe2ec022f40 1 -- 172.21.15.46:0/3197866253 wait complete. 2024-05-09T03:19:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.368+0000 7fe2ec022f40 1 Processor -- start 2024-05-09T03:19:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.368+0000 7fe2ec022f40 1 -- start start 2024-05-09T03:19:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2ec022f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973223a0 0x56409739ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2ec022f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x5640973b78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2ec022f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973b5a40 0x5640973b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2ec022f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5640973107c0 con 0x5640973b5a40 2024-05-09T03:19:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2ec022f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5640973104e0 con 0x56409732a5f0 2024-05-09T03:19:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2ec022f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5640973256c0 con 0x5640973223a0 2024-05-09T03:19:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2e62eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973b5a40 0x5640973b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2e62eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973b5a40 0x5640973b7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36630/0 (socket says 172.21.15.46:36630) 2024-05-09T03:19:19.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 learned_addr learned my addr 172.21.15.46:0/360503838 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x5640973b78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.369+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973223a0 0x56409739ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973223a0 msgr2=0x56409739ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2e62eb640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5640973223a0 0x56409739ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 msgr2=0x5640973b78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2e62eb640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56409732a5f0 0x5640973b78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564097385b40 con 0x5640973b5a40 2024-05-09T03:19:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2e62eb640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973b5a40 0x5640973b7de0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000ce40 tx=0x7fe2e000a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2d6ffd640 1 -- 172.21.15.46:0/360503838 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e000ae60 con 0x5640973b5a40 2024-05-09T03:19:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2d6ffd640 1 -- 172.21.15.46:0/360503838 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e00087e0 con 0x5640973b5a40 2024-05-09T03:19:19.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640973aa4a0 con 0x5640973b5a40 2024-05-09T03:19:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2d6ffd640 1 -- 172.21.15.46:0/360503838 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0008bd0 con 0x5640973b5a40 2024-05-09T03:19:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.370+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640973aaa60 con 0x5640973b5a40 2024-05-09T03:19:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.372+0000 7fe2d6ffd640 1 -- 172.21.15.46:0/360503838 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe2e0015440 con 0x5640973b5a40 2024-05-09T03:19:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.372+0000 7fe2d6ffd640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2c8074ce0 0x7fe2c80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.372+0000 7fe2d6ffd640 1 -- 172.21.15.46:0/360503838 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7fe2e0096bf0 con 0x5640973b5a40 2024-05-09T03:19:19.379 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.373+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5640973b4620 0x564097397800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.373+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097397d40 con 0x5640973b4620 2024-05-09T03:19:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.373+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2c8074ce0 0x7fe2c80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.373+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5640973b4620 0x564097397800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.373+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5640973b4620 0x564097397800 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.373+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2c8074ce0 0x7fe2c80771f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8062000 tx=0x7fe2d8007030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.376+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v115'939 uv913 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.376+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409745ba10 con 0x5640973b4620 2024-05-09T03:19:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.377+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v115'940 uv940 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.377+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745bdb0 con 0x5640973b4620 2024-05-09T03:19:19.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.379+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v115'941 uv941 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.384 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.379+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745d160 con 0x5640973b4620 2024-05-09T03:19:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.381+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v115'942 uv942 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.381+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745d500 con 0x5640973b4620 2024-05-09T03:19:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.382+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v115'943 uv943 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.382+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c150 con 0x5640973b4620 2024-05-09T03:19:19.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.384+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v115'944 uv944 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.384+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c540 con 0x5640973b4620 2024-05-09T03:19:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.386+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v115'945 uv919 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.386+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409745c540 con 0x5640973b4620 2024-05-09T03:19:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.388+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v115'946 uv946 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.388+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.389+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v115'947 uv947 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.390 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.389+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.391+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v115'948 uv948 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.391+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.392+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v115'949 uv949 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.392+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.394+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v115'950 uv950 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.394+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.395+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v115'951 uv926 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.395+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.397+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v115'952 uv952 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.397+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.399+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v115'953 uv953 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.399 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:19.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.399+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.400+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v115'954 uv954 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.400+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.402+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v115'955 uv955 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.402+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c4f0 con 0x5640973b4620 2024-05-09T03:19:19.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.403+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v115'956 uv956 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.403+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c8b0 con 0x5640973b4620 2024-05-09T03:19:19.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.405+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v115'957 uv957 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.405+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.406+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v115'958 uv932 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.406+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.408+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v115'959 uv959 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.408+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.409+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v115'960 uv960 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.410 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:19.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.409+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.411+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v115'961 uv961 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.411+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.412+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v115'962 uv962 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.412+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.414+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v115'963 uv963 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.414+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.415+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v115'964 uv938 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.415+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.416+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v115'965 uv965 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.417+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.418+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v115'966 uv966 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.418 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.418+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.419+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v115'967 uv967 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.420+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.421+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v115'968 uv968 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.421+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x5640973b4620 2024-05-09T03:19:19.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.422+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v115'969 uv969 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.422+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56409745d8a0 0x56409745fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.422+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409745c760 con 0x56409745d8a0 2024-05-09T03:19:19.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.423+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56409745d8a0 0x56409745fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.423+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56409745d8a0 0x56409745fcc0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.425+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v115'675 uv658 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.425+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097460200 con 0x56409745d8a0 2024-05-09T03:19:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.426+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v115'676 uv676 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.426+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974605a0 con 0x56409745d8a0 2024-05-09T03:19:19.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.428+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v115'677 uv677 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.428 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:19.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.428+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097460df0 con 0x56409745d8a0 2024-05-09T03:19:19.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.429+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v115'678 uv678 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.429+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461190 con 0x56409745d8a0 2024-05-09T03:19:19.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.430+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v115'679 uv679 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.430+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461530 con 0x56409745d8a0 2024-05-09T03:19:19.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.432+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v115'680 uv663 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.432+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x5640974618d0 con 0x56409745d8a0 2024-05-09T03:19:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.433+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v115'681 uv681 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.433+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.434+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v115'682 uv682 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.435 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:19.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.434+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.436+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v115'683 uv683 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.436+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.437+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v115'684 uv684 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.437+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.438+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v115'685 uv668 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.438+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.440+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v115'686 uv686 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.440+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.441+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v115'687 uv687 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.442 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:19.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.441+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974656d0 con 0x56409745d8a0 2024-05-09T03:19:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.443+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v115'688 uv688 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.443+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974656d0 con 0x56409745d8a0 2024-05-09T03:19:19.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.444+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v115'689 uv689 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.444+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.445+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v115'690 uv674 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.445+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.447+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v115'691 uv691 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.447+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.448+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v115'692 uv692 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.449 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:19.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.448+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097462c80 con 0x56409745d8a0 2024-05-09T03:19:19.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.449+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v115'693 uv693 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.449+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097462c80 con 0x56409745d8a0 2024-05-09T03:19:19.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.451+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v115'694 uv694 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.451+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x56409745d8a0 2024-05-09T03:19:19.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.452+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v115'695 uv695 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409745d8a0 2024-05-09T03:19:19.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.452+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x5640973b4620 2024-05-09T03:19:19.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.453+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v115'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.454+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097461c70 con 0x5640973b4620 2024-05-09T03:19:19.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.455+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v115'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.455+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x5640973b4620 2024-05-09T03:19:19.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.455+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v115'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.456 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:19.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.456+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097462c80 con 0x5640973b4620 2024-05-09T03:19:19.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.457+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v115'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.457+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097462c80 con 0x5640973b4620 2024-05-09T03:19:19.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.458+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v115'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.458+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097461c70 con 0x5640973b4620 2024-05-09T03:19:19.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.458+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v115'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.459+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564097462180 0x5640974645a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.459+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097464b70 con 0x564097462180 2024-05-09T03:19:19.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.459+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564097462180 0x5640974645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.459+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564097462180 0x5640974645a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.460+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v115'136 uv135 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.460+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097464f60 con 0x564097462180 2024-05-09T03:19:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.461+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v115'137 uv137 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.461+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097464f60 con 0x564097462180 2024-05-09T03:19:19.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.462+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v115'138 uv138 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.463 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:19.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.462+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097464f60 con 0x564097462180 2024-05-09T03:19:19.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.463+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v115'139 uv139 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.464+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097464f60 con 0x5640973b4620 2024-05-09T03:19:19.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.465+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v115'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.465+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097464f60 con 0x5640973b4620 2024-05-09T03:19:19.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.467+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v115'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.467+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097464f60 con 0x5640973b4620 2024-05-09T03:19:19.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.468+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v115'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.468 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:19.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.468+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097465f70 con 0x5640973b4620 2024-05-09T03:19:19.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.469+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v115'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.469+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097465f70 con 0x5640973b4620 2024-05-09T03:19:19.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.470+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v115'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.470+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097464f60 con 0x5640973b4620 2024-05-09T03:19:19.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.471+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v115'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.471+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5640974655e0 0x564097467a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.471+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097468060 con 0x5640974655e0 2024-05-09T03:19:19.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.471+0000 7fe2e62eb640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5640974655e0 0x564097467a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.472+0000 7fe2e62eb640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5640974655e0 0x564097467a00 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.473+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v115'338 uv332 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.473+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097468500 con 0x5640974655e0 2024-05-09T03:19:19.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.474+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v115'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.474+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974689e0 con 0x5640974655e0 2024-05-09T03:19:19.476 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:19.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.475+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v115'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.475+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746a290 con 0x5640974655e0 2024-05-09T03:19:19.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.477+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v115'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.477+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746a630 con 0x5640974655e0 2024-05-09T03:19:19.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.478+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v115'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.478+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097469280 con 0x5640974655e0 2024-05-09T03:19:19.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.479+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v115'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.479+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097469670 con 0x5640974655e0 2024-05-09T03:19:19.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.480+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v115'344 uv337 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.480+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097469a60 con 0x5640974655e0 2024-05-09T03:19:19.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.481+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v115'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.481+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097469a60 con 0x5640974655e0 2024-05-09T03:19:19.482 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:19.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.482+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v115'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.482+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097469a10 con 0x5640974655e0 2024-05-09T03:19:19.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.483+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v115'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.483+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097469a10 con 0x5640974655e0 2024-05-09T03:19:19.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.484+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v115'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.484+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56409746a9d0 0x56409746cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.484+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d330 con 0x56409746a9d0 2024-05-09T03:19:19.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.484+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56409746a9d0 0x56409746cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.485+0000 7fe2e5aea640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56409746a9d0 0x56409746cdf0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.486+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v115'153 uv152 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409746a9d0 2024-05-09T03:19:19.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.486+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409746d330 con 0x56409746a9d0 2024-05-09T03:19:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.487+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v115'154 uv154 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409746a9d0 2024-05-09T03:19:19.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.487+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x56409746a9d0 2024-05-09T03:19:19.488 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:19.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.488+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v115'155 uv155 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409746a9d0 2024-05-09T03:19:19.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.488+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x56409746a9d0 2024-05-09T03:19:19.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.489+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v115'156 uv156 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409746a9d0 2024-05-09T03:19:19.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.489+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x56409746a9d0 2024-05-09T03:19:19.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.490+0000 7fe2e5aea640 1 -- 172.21.15.46:0/360503838 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v115'157 uv157 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2d806f460 con 0x56409746a9d0 2024-05-09T03:19:19.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.490+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x5640973b4620 2024-05-09T03:19:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.492+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v115'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.492+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409746d720 con 0x5640973b4620 2024-05-09T03:19:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.493+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v115'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.493+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x5640973b4620 2024-05-09T03:19:19.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.494+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v115'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.494 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.494+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746e730 con 0x5640973b4620 2024-05-09T03:19:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.495+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v115'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.495+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746e730 con 0x5640973b4620 2024-05-09T03:19:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.496+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v115'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.496+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x5640973b4620 2024-05-09T03:19:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.497+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v115'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x5640973b4620 2024-05-09T03:19:19.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.497+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.498+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v115'304 uv298 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.498+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.499+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v115'305 uv305 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.499+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.501 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:19.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.500+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v115'306 uv306 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.500+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.502+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v115'307 uv307 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.502+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.503+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v115'308 uv308 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.503+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.504+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v115'309 uv303 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.504+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.505+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v115'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.505+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.506 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:19.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.506+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v115'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.506+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56409746d720 con 0x564097462180 2024-05-09T03:19:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.507+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v115'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.507+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097470530 con 0x564097462180 2024-05-09T03:19:19.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.508+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v115'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc010640 con 0x564097462180 2024-05-09T03:19:19.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.508+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56409746e250 0x564097470b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.508+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097470670 con 0x56409746e250 2024-05-09T03:19:19.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.508+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56409746e250 0x564097470b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.509+0000 7fe2e52e9640 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56409746e250 0x564097470b50 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.510+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v115'132 uv131 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x56409746e250 2024-05-09T03:19:19.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.510+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097470670 con 0x56409746e250 2024-05-09T03:19:19.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.511+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v115'133 uv133 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x56409746e250 2024-05-09T03:19:19.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.511+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097470670 con 0x56409746e250 2024-05-09T03:19:19.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.512+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v115'134 uv134 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x56409746e250 2024-05-09T03:19:19.512 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:19.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.512+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x56409746e250 2024-05-09T03:19:19.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.513+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v115'135 uv135 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x56409746e250 2024-05-09T03:19:19.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.513+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x564097462180 2024-05-09T03:19:19.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.514+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v115'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x564097462180 2024-05-09T03:19:19.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.514+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097471090 con 0x564097462180 2024-05-09T03:19:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.515+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v115'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x564097462180 2024-05-09T03:19:19.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.515+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x564097462180 2024-05-09T03:19:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.516+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v115'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x564097462180 2024-05-09T03:19:19.517 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:19.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.517+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974720a0 con 0x564097462180 2024-05-09T03:19:19.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.517+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v115'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x564097462180 2024-05-09T03:19:19.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.517+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974720a0 con 0x564097462180 2024-05-09T03:19:19.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.518+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v115'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x564097462180 2024-05-09T03:19:19.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.518+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x564097462180 2024-05-09T03:19:19.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.520+0000 7fe2e52e9640 1 -- 172.21.15.46:0/360503838 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v115'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2dc027040 con 0x564097462180 2024-05-09T03:19:19.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.520+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.521+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v115'156 uv155 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.521+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.522+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v115'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.522+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.524 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:19.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.524+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v115'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.524+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.525+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v115'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.525+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.526+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v115'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.526+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.527+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v115'376 uv375 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.527+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.529+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v115'377 uv377 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.529+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.530+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v115'378 uv378 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.531 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:19.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.531+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.532+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v115'379 uv379 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.532+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.533+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v115'380 uv380 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.533+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.534+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v115'381 uv374 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.534+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.535+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v115'382 uv382 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.535+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.537+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v115'383 uv383 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.537 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:19.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.537+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974720a0 con 0x5640974655e0 2024-05-09T03:19:19.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.538+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v115'384 uv384 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.538+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x5640974720a0 con 0x5640974655e0 2024-05-09T03:19:19.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.539+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v115'385 uv385 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.539+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471090 con 0x5640974655e0 2024-05-09T03:19:19.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.541+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v115'386 uv386 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.541+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x564097471540 con 0x5640974655e0 2024-05-09T03:19:19.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v115'387 uv387 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe2e0008d70 con 0x5640974655e0 2024-05-09T03:19:19.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5640973b4620 msgr2=0x564097397800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x5640973b4620 0x564097397800 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564097462180 msgr2=0x5640974645a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x564097462180 0x5640974645a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56409746e250 msgr2=0x564097470b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56409746e250 0x564097470b50 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56409745d8a0 msgr2=0x56409745fcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56409745d8a0 0x56409745fcc0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 reap_dead start 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5640974655e0 msgr2=0x564097467a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5640974655e0 0x564097467a00 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56409746a9d0 msgr2=0x56409746cdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56409746a9d0 0x56409746cdf0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2c8074ce0 msgr2=0x7fe2c80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe2c8074ce0 0x7fe2c80771f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8062000 tx=0x7fe2d8007030 comp rx=0 tx=0).stop 2024-05-09T03:19:19.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973b5a40 msgr2=0x5640973b7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 --2- 172.21.15.46:0/360503838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5640973b5a40 0x5640973b7de0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fe2e000ce40 tx=0x7fe2e000a750 comp rx=0 tx=0).stop 2024-05-09T03:19:19.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2e62eb640 1 -- 172.21.15.46:0/360503838 reap_dead start 2024-05-09T03:19:19.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 shutdown_connections 2024-05-09T03:19:19.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 >> 172.21.15.46:0/360503838 conn(0x564097204ac0 msgr2=0x564097326b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:19.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 shutdown_connections 2024-05-09T03:19:19.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.542+0000 7fe2ec022f40 1 -- 172.21.15.46:0/360503838 wait complete. 2024-05-09T03:19:19.549 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 sixmegs_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:19:19.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.648+0000 7f30af4cff40 1 Processor -- start 2024-05-09T03:19:19.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.648+0000 7f30af4cff40 1 -- start start 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30af4cff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc437f0 0x55c78bc73690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30af4cff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc757c0 0x55c78bd15fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30af4cff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bd16510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30af4cff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c78bc8a4e0 con 0x55c78bc8c020 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30af4cff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c78bbecf50 con 0x55c78bc757c0 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30af4cff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c78bc8f8b0 con 0x55c78bc437f0 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30a9798640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bd16510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30a9798640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bd16510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36638/0 (socket says 172.21.15.46:36638) 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30a9798640 1 -- 172.21.15.46:0/1930277997 learned_addr learned my addr 172.21.15.46:0/1930277997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30a8796640 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc757c0 0x55c78bd15fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.649+0000 7f30a8f97640 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc437f0 0x55c78bc73690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a9798640 1 -- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc437f0 msgr2=0x55c78bc73690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a9798640 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc437f0 0x55c78bc73690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a9798640 1 -- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc757c0 msgr2=0x55c78bd15fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a9798640 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc757c0 0x55c78bd15fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a9798640 1 -- 172.21.15.46:0/1930277997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78bc74b40 con 0x55c78bc8c020 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a8f97640 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc437f0 0x55c78bc73690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a9798640 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bd16510 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3098001b20 tx=0x7f3098006930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6b840427a5227 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a7794640 1 -- 172.21.15.46:0/1930277997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098016070 con 0x55c78bc8c020 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a7794640 1 -- 172.21.15.46:0/1930277997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3098004540 con 0x55c78bc8c020 2024-05-09T03:19:19.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.650+0000 7f30a7794640 1 -- 172.21.15.46:0/1930277997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098004930 con 0x55c78bc8c020 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 -- 172.21.15.46:0/1930277997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 msgr2=0x55c78bd16510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bd16510 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3098001b20 tx=0x7f3098006930 comp rx=0 tx=0).stop 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 -- 172.21.15.46:0/1930277997 shutdown_connections 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bd16510 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc757c0 0x55c78bd15fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 --2- 172.21.15.46:0/1930277997 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc437f0 0x55c78bc73690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 -- 172.21.15.46:0/1930277997 >> 172.21.15.46:0/1930277997 conn(0x55c78bb68ac0 msgr2=0x55c78bd1acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 -- 172.21.15.46:0/1930277997 shutdown_connections 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.651+0000 7f30af4cff40 1 -- 172.21.15.46:0/1930277997 wait complete. 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.652+0000 7f30af4cff40 1 Processor -- start 2024-05-09T03:19:19.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.652+0000 7f30af4cff40 1 -- start start 2024-05-09T03:19:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30af4cff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc437f0 0x55c78bd17790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30af4cff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc757c0 0x55c78bdad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30af4cff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bdad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30af4cff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c78bc72d90 con 0x55c78bc8c020 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30af4cff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c78bc74590 con 0x55c78bc437f0 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30af4cff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55c78bdadf70 con 0x55c78bc757c0 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bdad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bdad600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36646/0 (socket says 172.21.15.46:36646) 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 -- 172.21.15.46:0/581310670 learned_addr learned my addr 172.21.15.46:0/581310670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a8f97640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc437f0 0x55c78bd17790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a8796640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc757c0 0x55c78bdad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 -- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc757c0 msgr2=0x55c78bdad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55c78bc757c0 0x55c78bdad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 -- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc437f0 msgr2=0x55c78bd17790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc437f0 0x55c78bd17790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a9798640 1 -- 172.21.15.46:0/581310670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c78bd1c720 con 0x55c78bc8c020 2024-05-09T03:19:19.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.653+0000 7f30a8f97640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c78bc437f0 0x55c78bd17790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.654+0000 7f30a9798640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bdad600 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3098006e30 tx=0x7f3098001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.654+0000 7f30a6792640 1 -- 172.21.15.46:0/581310670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098021020 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.654+0000 7f30a6792640 1 -- 172.21.15.46:0/581310670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f309801d430 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.654+0000 7f30a6792640 1 -- 172.21.15.46:0/581310670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098016030 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.654+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c78bd1c440 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.654+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c78bd18450 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.655+0000 7f30a6792640 1 -- 172.21.15.46:0/581310670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3098024020 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.656+0000 7f30a6792640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3088074ce0 0x7f30880771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.656+0000 7f30a6792640 1 -- 172.21.15.46:0/581310670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f3098097d50 con 0x55c78bc8c020 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.656+0000 7f30a8f97640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3088074ce0 0x7f30880771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.656+0000 7f30af4cff40 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55c78bdadb40 0x55c78bdb6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.656+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=104b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x55c78bd17cd0 con 0x55c78bdadb40 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.656+0000 7f30a8796640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55c78bdadb40 0x55c78bdb6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.657+0000 7f30a8f97640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3088074ce0 0x7f30880771f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f30a0005fd0 tx=0x7f30a0005a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.657+0000 7f30a8796640 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55c78bdadb40 0x55c78bdb6e70 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.658+0000 7f30a8796640 1 -- 172.21.15.46:0/581310670 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v115'970 uv970 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3090054990 con 0x55c78bdadb40 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.658+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55c78bdadb40 msgr2=0x55c78bdb6e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.658+0000 7f30af4cff40 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55c78bdadb40 0x55c78bdb6e70 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3088074ce0 msgr2=0x7f30880771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3088074ce0 0x7f30880771f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f30a0005fd0 tx=0x7f30a0005a00 comp rx=0 tx=0).stop 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 msgr2=0x55c78bdad600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 --2- 172.21.15.46:0/581310670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c78bc8c020 0x55c78bdad600 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3098006e30 tx=0x7f3098001250 comp rx=0 tx=0).stop 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30a9798640 1 -- 172.21.15.46:0/581310670 reap_dead start 2024-05-09T03:19:19.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 shutdown_connections 2024-05-09T03:19:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 >> 172.21.15.46:0/581310670 conn(0x55c78bb68ac0 msgr2=0x55c78bc73690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 shutdown_connections 2024-05-09T03:19:19.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.659+0000 7f30af4cff40 1 -- 172.21.15.46:0/581310670 wait complete. 2024-05-09T03:19:19.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:19.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.779+0000 7f4cb41c7640 1 Processor -- start 2024-05-09T03:19:19.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.779+0000 7f4cb41c7640 1 -- start start 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb41c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0b5140 0x7f4cac0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb41c7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cac0b80b0 con 0x7f4cac0b7ca0 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb41c7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cac0b5550 con 0x7f4cac0d3fc0 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb41c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cac0d3480 con 0x7f4cac0b5140 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb1762640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4cac0b7ca0 0x7f4cac0d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:46860/0 (socket says 172.21.15.46:46860) 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb1762640 1 -- 172.21.15.46:0/842128762 learned_addr learned my addr 172.21.15.46:0/842128762 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb1f63640 1 --2- 172.21.15.46:0/842128762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0b5140 0x7f4cac0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3013616363 0 0) 0x7f4c9c001110 con 0x7f4cac0b7ca0 2024-05-09T03:19:19.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c98003520 con 0x7f4cac0b7ca0 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 870906708 0 0) 0x7f4cac0b5550 con 0x7f4cac0d3fc0 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c9c001110 con 0x7f4cac0d3fc0 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb1f63640 1 -- 172.21.15.46:0/842128762 >> v1:172.21.15.88:6789/0 conn(0x7f4cac0d3fc0 legacy=0x7f4cac0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb1f63640 1 -- 172.21.15.46:0/842128762 >> v1:172.21.15.46:6789/0 conn(0x7f4cac0b7ca0 legacy=0x7f4cac0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.781+0000 7f4cb1f63640 1 -- 172.21.15.46:0/842128762 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac0d44e0 con 0x7f4cac0b5140 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.782+0000 7f4cb1f63640 1 --2- 172.21.15.46:0/842128762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0b5140 0x7f4cac0b7760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80097d0 tx=0x7f4ca802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623ad516000d875d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.782+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8037070 con 0x7f4cac0b5140 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.782+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca8032440 con 0x7f4cac0b5140 2024-05-09T03:19:19.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.782+0000 7f4cb0f61640 1 -- 172.21.15.46:0/842128762 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca80327f0 con 0x7f4cac0b5140 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 -- 172.21.15.46:0/842128762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0b5140 msgr2=0x7f4cac0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/842128762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0b5140 0x7f4cac0b7760 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80097d0 tx=0x7f4ca802a290 comp rx=0 tx=0).stop 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 -- 172.21.15.46:0/842128762 shutdown_connections 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/842128762 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0b5140 0x7f4cac0b7760 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 -- 172.21.15.46:0/842128762 >> 172.21.15.46:0/842128762 conn(0x7f4cac08d3f0 msgr2=0x7f4cac08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 -- 172.21.15.46:0/842128762 shutdown_connections 2024-05-09T03:19:19.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.783+0000 7f4cb41c7640 1 -- 172.21.15.46:0/842128762 wait complete. 2024-05-09T03:19:19.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.785+0000 7f4cb41c7640 1 Processor -- start 2024-05-09T03:19:19.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.785+0000 7f4cb41c7640 1 -- start start 2024-05-09T03:19:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb41c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 0x7f4cac0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb41c7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cac0b7ca0 0x7f4cac1f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb41c7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0d3fc0 0x7f4cac1f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb41c7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac0d4d80 con 0x7f4cac0b5140 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb41c7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac0d4aa0 con 0x7f4cac0b7ca0 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb41c7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cac0d5580 con 0x7f4cac0d3fc0 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb1f63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 0x7f4cac0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb1f63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 0x7f4cac0ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36658/0 (socket says 172.21.15.46:36658) 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb1f63640 1 -- 172.21.15.46:0/1686009035 learned_addr learned my addr 172.21.15.46:0/1686009035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb2764640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0d3fc0 0x7f4cac1f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.786+0000 7f4cb1762640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cac0b7ca0 0x7f4cac1f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb1f63640 1 -- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0d3fc0 msgr2=0x7f4cac1f7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb1f63640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0d3fc0 0x7f4cac1f7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb1f63640 1 -- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cac0b7ca0 msgr2=0x7f4cac1f0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb1f63640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cac0b7ca0 0x7f4cac1f0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb1f63640 1 -- 172.21.15.46:0/1686009035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac1fa770 con 0x7f4cac0b5140 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb1f63640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 0x7f4cac0ca490 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8000c00 tx=0x7f4ca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8041070 con 0x7f4cac0b5140 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca803c870 con 0x7f4cac0b5140 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cac0c99b0 con 0x7f4cac0b5140 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8037040 con 0x7f4cac0b5140 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.787+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cac0c9f70 con 0x7f4cac0b5140 2024-05-09T03:19:19.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.788+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cac0d1a00 con 0x7f4cac0b5140 2024-05-09T03:19:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.789+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4ca8044070 con 0x7f4cac0b5140 2024-05-09T03:19:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.790+0000 7f4ca27fc640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c900762d0 0x7f4c900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.790+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f4ca80b8850 con 0x7f4cac0b5140 2024-05-09T03:19:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.790+0000 7f4cb1762640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c900762d0 0x7f4c900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:19.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.790+0000 7f4cb1762640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c900762d0 0x7f4c900787e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005880 tx=0x7f4c9c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:19.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:19.794+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4cac0d1a00 con 0x7f4cac0b5140 2024-05-09T03:19:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.023+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4cac06a270 con 0x7f4cac0b5140 2024-05-09T03:19:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.294+0000 7f4ca27fc640 1 -- 172.21.15.46:0/1686009035 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v279) ==== 162+0+0 (secure 0 0 0) 0x7f4ca8080fd0 con 0x7f4cac0b5140 2024-05-09T03:19:20.295 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c900762d0 msgr2=0x7f4c900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c900762d0 0x7f4c900787e0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005880 tx=0x7f4c9c009040 comp rx=0 tx=0).stop 2024-05-09T03:19:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 msgr2=0x7f4cac0ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:20.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 0x7f4cac0ca490 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8000c00 tx=0x7f4ca80046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 shutdown_connections 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cac0d3fc0 0x7f4cac1f7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cac0b7ca0 0x7f4cac1f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c900762d0 0x7f4c900787e0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 --2- 172.21.15.46:0/1686009035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cac0b5140 0x7f4cac0ca490 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 >> 172.21.15.46:0/1686009035 conn(0x7f4cac08d3f0 msgr2=0x7f4cac08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 shutdown_connections 2024-05-09T03:19:20.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.297+0000 7f4cb41c7640 1 -- 172.21.15.46:0/1686009035 wait complete. 2024-05-09T03:19:20.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:20.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.834+0000 7ff9d3357640 1 Processor -- start 2024-05-09T03:19:20.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.835+0000 7ff9d3357640 1 -- start start 2024-05-09T03:19:20.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d3357640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9cc0b5210 0x7ff9cc1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d3357640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d3357640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9cc0b77c0 0x7ff9cc0b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d3357640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff9cc0b5620 con 0x7ff9cc0b5210 2024-05-09T03:19:20.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d3357640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff9cc0b8110 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d3357640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff9cc0d5280 con 0x7ff9cc0d4800 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9cc0b5210 0x7ff9cc1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9cc0b5210 0x7ff9cc1613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36662/0 (socket says 172.21.15.46:36662) 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 -- 172.21.15.46:0/2173643144 learned_addr learned my addr 172.21.15.46:0/2173643144 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d08f2640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d18f4640 1 --2- 172.21.15.46:0/2173643144 >> v2:172.21.15.88:3300/0 conn(0x7ff9cc0b77c0 0x7ff9cc0b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 -- 172.21.15.46:0/2173643144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 msgr2=0x7ff9cc0d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc0d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 -- 172.21.15.46:0/2173643144 >> v2:172.21.15.88:3300/0 conn(0x7ff9cc0b77c0 msgr2=0x7ff9cc0b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 --2- 172.21.15.46:0/2173643144 >> v2:172.21.15.88:3300/0 conn(0x7ff9cc0b77c0 0x7ff9cc0b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d10f3640 1 -- 172.21.15.46:0/2173643144 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc0d5560 con 0x7ff9cc0b5210 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.836+0000 7ff9d08f2640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc0d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.837+0000 7ff9d18f4640 1 --2- 172.21.15.46:0/2173643144 >> v2:172.21.15.88:3300/0 conn(0x7ff9cc0b77c0 0x7ff9cc0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.837+0000 7ff9d10f3640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc1613c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0d3bd0 tx=0x7ff9c402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9aded7d0cf0a82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.837+0000 7ff9cbfff640 1 -- 172.21.15.46:0/2173643144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4037070 con 0x7ff9cc0b5210 2024-05-09T03:19:20.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.837+0000 7ff9cbfff640 1 -- 172.21.15.46:0/2173643144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c4032440 con 0x7ff9cc0b5210 2024-05-09T03:19:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.837+0000 7ff9cbfff640 1 -- 172.21.15.46:0/2173643144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4032810 con 0x7ff9cc0b5210 2024-05-09T03:19:20.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 -- 172.21.15.46:0/2173643144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b5210 msgr2=0x7ff9cc1613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc1613c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0d3bd0 tx=0x7ff9c402a140 comp rx=0 tx=0).stop 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 -- 172.21.15.46:0/2173643144 shutdown_connections 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 --2- 172.21.15.46:0/2173643144 >> v2:172.21.15.88:3300/0 conn(0x7ff9cc0b77c0 0x7ff9cc0b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc0d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 --2- 172.21.15.46:0/2173643144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc1613c0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 -- 172.21.15.46:0/2173643144 >> 172.21.15.46:0/2173643144 conn(0x7ff9cc0d29a0 msgr2=0x7ff9cc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 -- 172.21.15.46:0/2173643144 shutdown_connections 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.838+0000 7ff9d3357640 1 -- 172.21.15.46:0/2173643144 wait complete. 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.840+0000 7ff9d3357640 1 Processor -- start 2024-05-09T03:19:20.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.840+0000 7ff9d3357640 1 -- start start 2024-05-09T03:19:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d3357640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d3357640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 0x7ff9cc0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d3357640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc15f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d3357640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc0d0ae0 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d3357640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc160ea0 con 0x7ff9cc0b5210 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d3357640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff9cc163e60 con 0x7ff9cc0d4800 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.841+0000 7ff9d08f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 0x7ff9cc0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 0x7ff9cc0c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36678/0 (socket says 172.21.15.46:36678) 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 -- 172.21.15.46:0/1295129751 learned_addr learned my addr 172.21.15.46:0/1295129751 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d10f3640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d18f4640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc15f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 -- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 msgr2=0x7ff9cc15f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc15f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 -- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9cc0b5210 msgr2=0x7ff9cc14c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc14c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 -- 172.21.15.46:0/1295129751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc0cd040 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d08f2640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 0x7ff9cc0c9850 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000c7f0 tx=0x7ff9c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0015070 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c00077c0 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0007bb0 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.842+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9cc0cd320 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.843+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9cc0cd8e0 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.843+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9cc06a270 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.844+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff9c0018070 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.845+0000 7ff9c9ffb640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b80762d0 0x7ff9b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.845+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7ff9c0098130 con 0x7ff9cc0b77c0 2024-05-09T03:19:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.845+0000 7ff9d10f3640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b80762d0 0x7ff9b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:20.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.846+0000 7ff9d10f3640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b80762d0 0x7ff9b80787e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4034000 tx=0x7ff9c402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:20.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:20.850+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff9c00608e0 con 0x7ff9cc0b77c0 2024-05-09T03:19:21.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.079+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9cc0c9610 con 0x7ff9cc0b77c0 2024-05-09T03:19:21.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.080+0000 7ff9c9ffb640 1 -- 172.21.15.46:0/1295129751 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 280 v280) ==== 95+0+5178 (secure 0 0 0) 0x7ff9c0064be0 con 0x7ff9cc0b77c0 2024-05-09T03:19:21.081 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:21.081 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":280,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":280,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:20.297492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9678},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9678":{"gid":9678,"name":"c","rank":0,"incarnation":280,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/1350199591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1350199591},{"type":"v1","addr":"172.21.15.46:6834","nonce":1350199591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:21.081 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 280 2024-05-09T03:19:21.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b80762d0 msgr2=0x7ff9b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:21.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b80762d0 0x7ff9b80787e0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4034000 tx=0x7ff9c402e000 comp rx=0 tx=0).stop 2024-05-09T03:19:21.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 msgr2=0x7ff9cc0c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:21.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 0x7ff9cc0c9850 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000c7f0 tx=0x7ff9c000ccb0 comp rx=0 tx=0).stop 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 shutdown_connections 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9cc0d4800 0x7ff9cc15f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b80762d0 0x7ff9b80787e0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9cc0b77c0 0x7ff9cc0c9850 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 --2- 172.21.15.46:0/1295129751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9cc0b5210 0x7ff9cc14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 >> 172.21.15.46:0/1295129751 conn(0x7ff9cc0d29a0 msgr2=0x7ff9cc161bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 shutdown_connections 2024-05-09T03:19:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:21.082+0000 7ff9d3357640 1 -- 172.21.15.46:0/1295129751 wait complete. 2024-05-09T03:19:21.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 280, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:19:20.297492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 115, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9678}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9678': {'gid': 9678, 'name': 'c', 'rank': 0, 'incarnation': 280, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/1350199591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1350199591}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1350199591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:19:21.095 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9678:up:replay 2024-05-09T03:19:21.095 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:19:26.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.628+0000 7f00d6546640 1 Processor -- start 2024-05-09T03:19:26.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.628+0000 7f00d6546640 1 -- start start 2024-05-09T03:19:26.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d6546640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00d00b9310 0x7f00d008cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:26.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d6546640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00d008d6b0 0x7f00d0164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:26.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d6546640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d006ebe0 con 0x7f00d008d2a0 2024-05-09T03:19:26.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d6546640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f00d006f760 con 0x7f00d00b9310 2024-05-09T03:19:26.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d6546640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f00d006eec0 con 0x7f00d008d6b0 2024-05-09T03:19:26.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00d00b9310 0x7f00d008cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f00d00b9310 0x7f00d008cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47764/0 (socket says 172.21.15.46:47764) 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 -- 172.21.15.46:0/2046089379 learned_addr learned my addr 172.21.15.46:0/2046089379 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d4ae3640 1 --2- 172.21.15.46:0/2046089379 >> v2:172.21.15.88:3300/0 conn(0x7f00d008d6b0 0x7f00d0164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 -- 172.21.15.46:0/2046089379 >> v2:172.21.15.88:3300/0 conn(0x7f00d008d6b0 msgr2=0x7f00d0164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 --2- 172.21.15.46:0/2046089379 >> v2:172.21.15.88:3300/0 conn(0x7f00d008d6b0 0x7f00d0164f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 -- 172.21.15.46:0/2046089379 >> v1:172.21.15.88:6789/0 conn(0x7f00d008d2a0 legacy=0x7f00d0164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00cffff640 1 -- 172.21.15.46:0/2046089379 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d006f1a0 con 0x7f00d00b9310 2024-05-09T03:19:26.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.630+0000 7f00d4ae3640 1 --2- 172.21.15.46:0/2046089379 >> v2:172.21.15.88:3300/0 conn(0x7f00d008d6b0 0x7f00d0164f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.631+0000 7f00cffff640 1 --2- 172.21.15.46:0/2046089379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d008cd60 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f00d00d3850 tx=0x7f00c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb3d2cbfadcb675 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.631+0000 7f00ceffd640 1 -- 172.21.15.46:0/2046089379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0037070 con 0x7f00d00b9310 2024-05-09T03:19:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.631+0000 7f00ceffd640 1 -- 172.21.15.46:0/2046089379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c0032440 con 0x7f00d00b9310 2024-05-09T03:19:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.631+0000 7f00ceffd640 1 -- 172.21.15.46:0/2046089379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0032810 con 0x7f00d00b9310 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 -- 172.21.15.46:0/2046089379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 msgr2=0x7f00d008cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 --2- 172.21.15.46:0/2046089379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d008cd60 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f00d00d3850 tx=0x7f00c002a270 comp rx=0 tx=0).stop 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 -- 172.21.15.46:0/2046089379 shutdown_connections 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 --2- 172.21.15.46:0/2046089379 >> v2:172.21.15.88:3300/0 conn(0x7f00d008d6b0 0x7f00d0164f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 --2- 172.21.15.46:0/2046089379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d008cd60 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 -- 172.21.15.46:0/2046089379 >> 172.21.15.46:0/2046089379 conn(0x7f00d00b5660 msgr2=0x7f00d00b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 -- 172.21.15.46:0/2046089379 shutdown_connections 2024-05-09T03:19:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.632+0000 7f00d6546640 1 -- 172.21.15.46:0/2046089379 wait complete. 2024-05-09T03:19:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.634+0000 7f00d6546640 1 Processor -- start 2024-05-09T03:19:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.634+0000 7f00d6546640 1 -- start start 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.635+0000 7f00d6546640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d008d2a0 0x7f00d014c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.635+0000 7f00d6546640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00d008d6b0 0x7f00d00c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.635+0000 7f00d6546640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d00cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.635+0000 7f00d6546640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00d0165480 con 0x7f00d00b9310 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.635+0000 7f00d6546640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00d008dac0 con 0x7f00d008d2a0 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.635+0000 7f00d6546640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f00d0165600 con 0x7f00d008d6b0 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d00cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d00cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47770/0 (socket says 172.21.15.46:47770) 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 -- 172.21.15.46:0/316693522 learned_addr learned my addr 172.21.15.46:0/316693522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00cffff640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d008d2a0 0x7f00d014c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00cf7fe640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00d008d6b0 0x7f00d00c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:26.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 -- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00d008d6b0 msgr2=0x7f00d00c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00d008d6b0 0x7f00d00c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 -- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d008d2a0 msgr2=0x7f00d014c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d008d2a0 0x7f00d014c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 -- 172.21.15.46:0/316693522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d00c7080 con 0x7f00d00b9310 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d4ae3640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d00cec60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f00c8007df0 tx=0x7f00c800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c801a070 con 0x7f00d00b9310 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c8003990 con 0x7f00d00b9310 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d01528a0 con 0x7f00d00b9310 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8003d80 con 0x7f00d00b9310 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.636+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d015f260 con 0x7f00d00b9310 2024-05-09T03:19:26.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.637+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d006a270 con 0x7f00d00b9310 2024-05-09T03:19:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.638+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00c801d070 con 0x7f00d00b9310 2024-05-09T03:19:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.639+0000 7f00ccff9640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00bc0762d0 0x7f00bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.639+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f00c8098390 con 0x7f00d00b9310 2024-05-09T03:19:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.639+0000 7f00cffff640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00bc0762d0 0x7f00bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:26.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.639+0000 7f00cffff640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00bc0762d0 0x7f00bc0787e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f00c0034000 tx=0x7f00c002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:26.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.644+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f00c8010070 con 0x7f00d00b9310 2024-05-09T03:19:26.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.879+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00d00c7250 con 0x7f00d00b9310 2024-05-09T03:19:26.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.879+0000 7f00ccff9640 1 -- 172.21.15.46:0/316693522 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) ==== 95+0+5185 (secure 0 0 0) 0x7f00d00c7250 con 0x7f00d00b9310 2024-05-09T03:19:26.880 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:26.880 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":283,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:23.314768+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9678},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9678":{"gid":9678,"name":"c","rank":0,"incarnation":280,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/1350199591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1350199591},{"type":"v1","addr":"172.21.15.46:6834","nonce":1350199591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9678,"qdb_cluster":[9678]},"id":6}]} 2024-05-09T03:19:26.880 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 283 2024-05-09T03:19:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00bc0762d0 msgr2=0x7f00bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00bc0762d0 0x7f00bc0787e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f00c0034000 tx=0x7f00c002e000 comp rx=0 tx=0).stop 2024-05-09T03:19:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 msgr2=0x7f00d00cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:26.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d00cec60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f00c8007df0 tx=0x7f00c800d4a0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 shutdown_connections 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f00bc0762d0 0x7f00bc0787e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00d00b9310 0x7f00d00cec60 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f00d008d6b0 0x7f00d00c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 --2- 172.21.15.46:0/316693522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d008d2a0 0x7f00d014c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 >> 172.21.15.46:0/316693522 conn(0x7f00d00b5660 msgr2=0x7f00d01646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.881+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 shutdown_connections 2024-05-09T03:19:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:26.882+0000 7f00d6546640 1 -- 172.21.15.46:0/316693522 wait complete. 2024-05-09T03:19:26.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 283, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:19:23.314768+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 115, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9678}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9678': {'gid': 9678, 'name': 'c', 'rank': 0, 'incarnation': 280, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/1350199591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1350199591}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1350199591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9678, 'qdb_cluster': [9678]} 2024-05-09T03:19:26.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:19:26.894 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:19:27.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.009+0000 7f733270d640 1 Processor -- start 2024-05-09T03:19:27.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.009+0000 7f733270d640 1 -- start start 2024-05-09T03:19:27.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f733270d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f732c0d0020 0x7f732c0b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f733270d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0b3800 0x7f732c0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f733270d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732c0b92c0 con 0x7f732c0d1110 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f733270d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f732c0d0430 con 0x7f732c0d0020 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f733270d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f732c0b2f10 con 0x7f732c0b3800 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f732b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f732c0d0020 0x7f732c0b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f732bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f732c0d1110 0x7f732c0d1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35826/0 (socket says 172.21.15.46:35826) 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.010+0000 7f732bfff640 1 -- 172.21.15.46:0/2978843429 learned_addr learned my addr 172.21.15.46:0/2978843429 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f7330caa640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0b3800 0x7f732c0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f732b7fe640 1 -- 172.21.15.46:0/2978843429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0b3800 msgr2=0x7f732c0b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f732b7fe640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0b3800 0x7f732c0b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f732b7fe640 1 -- 172.21.15.46:0/2978843429 >> v1:172.21.15.46:6789/0 conn(0x7f732c0d1110 legacy=0x7f732c0d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f732b7fe640 1 -- 172.21.15.46:0/2978843429 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c0b9560 con 0x7f732c0d0020 2024-05-09T03:19:27.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f7330caa640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0b3800 0x7f732c0b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:27.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.011+0000 7f732b7fe640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0d0020 0x7f732c0b32c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f73200097a0 tx=0x7f732002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2d39515568b991 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f732affd640 1 -- 172.21.15.46:0/2978843429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320037070 con 0x7f732c0d0020 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f732affd640 1 -- 172.21.15.46:0/2978843429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f732002adb0 con 0x7f732c0d0020 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f732affd640 1 -- 172.21.15.46:0/2978843429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320032660 con 0x7f732c0d0020 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 -- 172.21.15.46:0/2978843429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0d0020 msgr2=0x7f732c0b32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0d0020 0x7f732c0b32c0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f73200097a0 tx=0x7f732002a060 comp rx=0 tx=0).stop 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 -- 172.21.15.46:0/2978843429 shutdown_connections 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0b3800 0x7f732c0b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 --2- 172.21.15.46:0/2978843429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0d0020 0x7f732c0b32c0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 -- 172.21.15.46:0/2978843429 >> 172.21.15.46:0/2978843429 conn(0x7f732c0d05f0 msgr2=0x7f732c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 -- 172.21.15.46:0/2978843429 shutdown_connections 2024-05-09T03:19:27.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.012+0000 7f733270d640 1 -- 172.21.15.46:0/2978843429 wait complete. 2024-05-09T03:19:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.014+0000 7f733270d640 1 Processor -- start 2024-05-09T03:19:27.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.014+0000 7f733270d640 1 -- start start 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f733270d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 0x7f732c1f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f733270d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0d0020 0x7f732c15a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f733270d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f732c15a970 0x7f732c158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f733270d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f732c09ceb0 con 0x7f732c0b3800 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f733270d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f732c0d2030 con 0x7f732c15a970 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f733270d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f732c09d030 con 0x7f732c0d0020 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 0x7f732c1f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 0x7f732c1f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47778/0 (socket says 172.21.15.46:47778) 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 -- 172.21.15.46:0/1345278349 learned_addr learned my addr 172.21.15.46:0/1345278349 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f7330caa640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f732c15a970 0x7f732c158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732b7fe640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0d0020 0x7f732c15a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 -- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0d0020 msgr2=0x7f732c15a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0d0020 0x7f732c15a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 -- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f732c15a970 msgr2=0x7f732c158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f732c15a970 0x7f732c158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 -- 172.21.15.46:0/1345278349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732c0b0770 con 0x7f732c0b3800 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.016+0000 7f732bfff640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 0x7f732c1f4780 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f731c00c340 tx=0x7f731c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.017+0000 7f7328ff9640 1 -- 172.21.15.46:0/1345278349 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c017070 con 0x7f732c0b3800 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.017+0000 7f7328ff9640 1 -- 172.21.15.46:0/1345278349 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731c005ce0 con 0x7f732c0b3800 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.017+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732c15b160 con 0x7f732c0b3800 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.017+0000 7f7328ff9640 1 -- 172.21.15.46:0/1345278349 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c007310 con 0x7f732c0b3800 2024-05-09T03:19:27.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.017+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732c0b0a10 con 0x7f732c0b3800 2024-05-09T03:19:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.018+0000 7f7328ff9640 1 -- 172.21.15.46:0/1345278349 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f731c01a070 con 0x7f732c0b3800 2024-05-09T03:19:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.019+0000 7f7328ff9640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7318076300 0x7f7318078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.019+0000 7f7328ff9640 1 -- 172.21.15.46:0/1345278349 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f731c097440 con 0x7f732c0b3800 2024-05-09T03:19:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.019+0000 7f732b7fe640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7318076300 0x7f7318078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.020+0000 7f73395c2740 1 Processor -- start 2024-05-09T03:19:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.020+0000 7f73395c2740 1 -- start start 2024-05-09T03:19:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.020+0000 7f732b7fe640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7318076300 0x7f7318078810 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7320034000 tx=0x7f732002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72aea6270 0x55e72b275eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b290be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f73395c2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 0x55e72b291120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f73395c2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e72b1ae4e0 con 0x55e72ae85590 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e72b149ff0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e72b1aa4a0 con 0x55e72aea6270 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 0x55e72b291120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 0x55e72b291120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47788/0 (socket says 172.21.15.46:47788) 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 -- 172.21.15.46:0/2892722957 learned_addr learned my addr 172.21.15.46:0/2892722957 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f732b7fe640 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b290be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f732bfff640 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72aea6270 0x55e72b275eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 -- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72aea6270 msgr2=0x55e72b275eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72aea6270 0x55e72b275eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 -- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 msgr2=0x55e72b290be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b290be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.021+0000 7f7330caa640 1 -- 172.21.15.46:0/2892722957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e72b130720 con 0x55e72ae85590 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.022+0000 7f7330caa640 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 0x55e72b291120 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f732400f520 tx=0x7f732400f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a520f3419878be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.022+0000 7f7311ffb640 1 -- 172.21.15.46:0/2892722957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732401c070 con 0x55e72ae85590 2024-05-09T03:19:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.022+0000 7f7311ffb640 1 -- 172.21.15.46:0/2892722957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f732400d7a0 con 0x55e72ae85590 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.022+0000 7f7311ffb640 1 -- 172.21.15.46:0/2892722957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732400db50 con 0x55e72ae85590 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.022+0000 7f73395c2740 1 -- 172.21.15.46:0/2892722957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 msgr2=0x55e72b291120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.022+0000 7f73395c2740 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 0x55e72b291120 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f732400f520 tx=0x7f732400f9e0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 -- 172.21.15.46:0/2892722957 shutdown_connections 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae85590 0x55e72b291120 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b290be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 --2- 172.21.15.46:0/2892722957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72aea6270 0x55e72b275eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 -- 172.21.15.46:0/2892722957 >> 172.21.15.46:0/2892722957 conn(0x55e72ae9a3c0 msgr2=0x55e72b24b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 -- 172.21.15.46:0/2892722957 shutdown_connections 2024-05-09T03:19:27.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 -- 172.21.15.46:0/2892722957 wait complete. 2024-05-09T03:19:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.023+0000 7f73395c2740 1 -- start start 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.024+0000 7f73395c2740 1 Processor -- start 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b29ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b28b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 0x55e72b28b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e72b135760 con 0x55e72aea6270 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e72b245340 con 0x55e72ae7d3b0 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e72b278730 con 0x55e72ae85590 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f73395c2740 10 client.0 _ll_register_callbacks cb 0x55e72b1584b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f7330caa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 0x55e72b28b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f7330caa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 0x55e72b28b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47798/0 (socket says 172.21.15.46:47798) 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f7330caa640 1 -- 172.21.15.46:0/3603774519 learned_addr learned my addr 172.21.15.46:0/3603774519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f732bfff640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b29ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f732b7fe640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b28b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.025+0000 7f7330caa640 1 -- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 msgr2=0x55e72b28b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f7330caa640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b28b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f7330caa640 1 -- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 msgr2=0x55e72b29ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f7330caa640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b29ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f7330caa640 1 -- 172.21.15.46:0/3603774519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e72b176bd0 con 0x55e72aea6270 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f732b7fe640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b28b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f7330caa640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 0x55e72b28b6b0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f732400fa20 tx=0x7f7324004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c186ce8becac6dd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f730affd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f730affd640 1 -- 172.21.15.46:0/3603774519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732401c050 con 0x55e72aea6270 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f732400d7a0 con 0x55e72aea6270 2024-05-09T03:19:27.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f730affd640 1 -- 172.21.15.46:0/3603774519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f732400dd60 con 0x55e72aea6270 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f730affd640 1 -- 172.21.15.46:0/3603774519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f7324007c90 con 0x55e72aea6270 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f73395c2740 10 client.9759 fetch_fsmap learned FSMap version 283 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=283,monmap=2+}) -- 0x55e72b15c280 con 0x55e72aea6270 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f730affd640 1 -- 172.21.15.46:0/3603774519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f732400d7a0 con 0x55e72aea6270 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f730affd640 1 -- 172.21.15.46:0/3603774519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 283) ==== 3494+0+0 (secure 0 0 0) 0x7f732400c040 con 0x55e72aea6270 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.026+0000 7f73395c2740 10 client.9759 fetch_fsmap finished waiting for FSMap version 283 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f73395c2740 10 client.9759 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f73395c2740 20 client.9759 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f73395c2740 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72ae64e40 0x55e72b26ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f73395c2740 4 client.9759 mds_command: new command op to 9678 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 --> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e72b1bc0f0 con 0x55e72ae64e40 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f732bfff640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72ae64e40 0x55e72b26ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f730affd640 10 client.9759 ms_handle_connect on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.027+0000 7f732bfff640 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72ae64e40 0x55e72b26ff40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a851cd6d83c723ff server_cookie=34c051c7297131ee in_seq=0 out_seq=0 2024-05-09T03:19:27.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f730affd640 1 -- 172.21.15.46:0/3603774519 <== mds.0 v2:172.21.15.46:6832/1350199591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e72b1bc0f0 con 0x55e72ae64e40 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f730affd640 10 client.9759 handle_command_reply: tid=0 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 client.9759 shutdown 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 msgr2=0x55e72b28b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 0x55e72b28b6b0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f732400fa20 tx=0x7f7324004ab0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 shutdown_connections 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72ae64e40 0x55e72b26ff40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72aea6270 0x55e72b28b6b0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b28b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f730affd640 0 client.9759 ms_handle_reset on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 --2- 172.21.15.46:0/3603774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72ae7d3b0 0x55e72b29ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 >> 172.21.15.46:0/3603774519 conn(0x55e72ae9a3c0 msgr2=0x55e72b24b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 shutdown_connections 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 1 -- 172.21.15.46:0/3603774519 wait complete. 2024-05-09T03:19:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.028+0000 7f73395c2740 20 client.9759 trim_cache size 0 max 16384 2024-05-09T03:19:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.047+0000 7f73395c2740 1 Processor -- start 2024-05-09T03:19:27.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.047+0000 7f73395c2740 1 -- start start 2024-05-09T03:19:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f73395c2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b2b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b2b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b2924d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f73395c2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f732c201980 con 0x55e72ae7d3b0 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f732c200f50 con 0x55e72aea6270 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f732c2013f0 con 0x55e72ae85590 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f732bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b2b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f732bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b2b3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47806/0 (socket says 172.21.15.46:47806) 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.048+0000 7f732bfff640 1 -- 172.21.15.46:0/2749702331 learned_addr learned my addr 172.21.15.46:0/2749702331 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f7330caa640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b2924d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732bfff640 1 -- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 msgr2=0x55e72b2b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732b7fe640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b2b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732bfff640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b2b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732bfff640 1 -- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 msgr2=0x55e72b2924d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732bfff640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b2924d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732bfff640 1 -- 172.21.15.46:0/2749702331 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e72af11850 con 0x55e72ae7d3b0 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f7330caa640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b2924d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732b7fe640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b2b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f732bfff640 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b2b3e50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f731c05c130 tx=0x7f731c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f102490bda24a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f73117fa640 1 -- 172.21.15.46:0/2749702331 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c063e40 con 0x55e72ae7d3b0 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f73117fa640 1 -- 172.21.15.46:0/2749702331 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731c0688f0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.049+0000 7f73117fa640 1 -- 172.21.15.46:0/2749702331 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c0810c0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 -- 172.21.15.46:0/2749702331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 msgr2=0x55e72b2b3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b2b3e50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f731c05c130 tx=0x7f731c000bc0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 -- 172.21.15.46:0/2749702331 shutdown_connections 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b2924d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b2b4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 --2- 172.21.15.46:0/2749702331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b2b3e50 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 -- 172.21.15.46:0/2749702331 >> 172.21.15.46:0/2749702331 conn(0x55e72ae9a3c0 msgr2=0x55e72b28a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 -- 172.21.15.46:0/2749702331 shutdown_connections 2024-05-09T03:19:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.050+0000 7f73395c2740 1 -- 172.21.15.46:0/2749702331 wait complete. 2024-05-09T03:19:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.051+0000 7f73395c2740 1 -- start start 2024-05-09T03:19:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.051+0000 7f73395c2740 1 Processor -- start 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b311b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b24b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b24b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e72b245ee0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e72b28bf50 con 0x55e72aea6270 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e72b244e60 con 0x55e72ae85590 2024-05-09T03:19:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f73395c2740 10 client.0 _ll_register_callbacks cb 0x7f732401a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.052+0000 7f732bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b311b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b311b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47822/0 (socket says 172.21.15.46:47822) 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 -- 172.21.15.46:0/1574680214 learned_addr learned my addr 172.21.15.46:0/1574680214 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f7330caa640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b24b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732b7fe640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b24b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 -- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 msgr2=0x55e72b24b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b24b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 -- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 msgr2=0x55e72b24b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b24b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 -- 172.21.15.46:0/1574680214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e72af886b0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f7330caa640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b24b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f732bfff640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b311b90 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f731c05fad0 tx=0x7f731c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35f8a317d58b01d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f730affd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:27.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f730affd640 1 -- 172.21.15.46:0/1574680214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c081250 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f730affd640 1 -- 172.21.15.46:0/1574680214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f731c0074b0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f731c0688f0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f730affd640 1 -- 172.21.15.46:0/1574680214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f731c063e40 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f73395c2740 10 client.9765 fetch_fsmap learned FSMap version 283 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=283,monmap=2+}) -- 0x55e72b16a630 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.053+0000 7f730affd640 1 -- 172.21.15.46:0/1574680214 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731c0688f0 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f730affd640 1 -- 172.21.15.46:0/1574680214 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 283) ==== 3494+0+0 (secure 0 0 0) 0x7f731c068b40 con 0x55e72ae7d3b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 10 client.9765 fetch_fsmap finished waiting for FSMap version 283 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 10 client.9765 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 20 client.9765 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72b2910b0 0x55e72b2914c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 4 client.9765 mds_command: new command op to 9678 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 --> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] -- command(tid 0: {"prefix": "status"}) -- 0x55e72b1b5f60 con 0x55e72b2910b0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f732b7fe640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72b2910b0 0x55e72b2914c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f730affd640 10 client.9765 ms_handle_connect on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:27.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f732b7fe640 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72b2910b0 0x55e72b2914c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16bef0ed487b8623 server_cookie=8bcba20a9cca9179 in_seq=0 out_seq=0 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f730affd640 1 -- 172.21.15.46:0/1574680214 <== mds.0 v2:172.21.15.46:6832/1350199591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55e72b1b5f60 con 0x55e72b2910b0 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f730affd640 10 client.9765 handle_command_reply: tid=0 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.054+0000 7f73395c2740 1 client.9765 shutdown 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 msgr2=0x55e72b311b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b311b90 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f731c05fad0 tx=0x7f731c009040 comp rx=0 tx=0).stop 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 shutdown_connections 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x55e72b2910b0 0x55e72b2914c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e72aea6270 0x55e72b24b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e72ae85590 0x55e72b24b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f730affd640 0 client.9765 ms_handle_reset on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 --2- 172.21.15.46:0/1574680214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e72ae7d3b0 0x55e72b311b90 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 >> 172.21.15.46:0/1574680214 conn(0x55e72ae9a3c0 msgr2=0x55e72b2924b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 shutdown_connections 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 1 -- 172.21.15.46:0/1574680214 wait complete. 2024-05-09T03:19:27.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f73395c2740 20 client.9765 trim_cache size 0 max 16384 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9678, 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7565181560000003, 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 283, 2024-05-09T03:19:27.062 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 115, 2024-05-09T03:19:27.063 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 115, 2024-05-09T03:19:27.063 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.718942439999999 2024-05-09T03:19:27.063 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7318076300 msgr2=0x7f7318078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7318076300 0x7f7318078810 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7320034000 tx=0x7f732002e000 comp rx=0 tx=0).stop 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 msgr2=0x7f732c1f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 0x7f732c1f4780 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f731c00c340 tx=0x7f731c00c800 comp rx=0 tx=0).stop 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 shutdown_connections 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f732c15a970 0x7f732c158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f732c0d0020 0x7f732c15a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7318076300 0x7f7318078810 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 --2- 172.21.15.46:0/1345278349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f732c0b3800 0x7f732c1f4780 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.055+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 >> 172.21.15.46:0/1345278349 conn(0x7f732c0d05f0 msgr2=0x7f732c0b7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.056+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 shutdown_connections 2024-05-09T03:19:27.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.056+0000 7f733270d640 1 -- 172.21.15.46:0/1345278349 wait complete. 2024-05-09T03:19:27.067 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:19:27.068 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap key 10000000000.00000000:sixmegs_head', proceeding to ls 2024-05-09T03:19:27.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:19:27.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.181+0000 7f0dac33f640 1 Processor -- start 2024-05-09T03:19:27.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.181+0000 7f0dac33f640 1 -- start start 2024-05-09T03:19:27.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.182+0000 7f0dac33f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0da40af670 0x7f0da40d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.182+0000 7f0dac33f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0da4073500 0x7f0da4073910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.182+0000 7f0dac33f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0da415e6e0 con 0x7f0da406f560 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.182+0000 7f0dac33f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0da40afa80 con 0x7f0da40af670 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.182+0000 7f0dac33f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0da4073e50 con 0x7f0da4073500 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.182+0000 7f0daa0db640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0da40af670 0x7f0da40d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0daa0db640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0da40af670 0x7f0da40d1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47838/0 (socket says 172.21.15.46:47838) 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0daa0db640 1 -- 172.21.15.46:0/1265117194 learned_addr learned my addr 172.21.15.46:0/1265117194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0da98da640 1 --2- 172.21.15.46:0/1265117194 >> v2:172.21.15.88:3300/0 conn(0x7f0da4073500 0x7f0da4073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0daa0db640 1 -- 172.21.15.46:0/1265117194 >> v2:172.21.15.88:3300/0 conn(0x7f0da4073500 msgr2=0x7f0da4073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0daa0db640 1 --2- 172.21.15.46:0/1265117194 >> v2:172.21.15.88:3300/0 conn(0x7f0da4073500 0x7f0da4073910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0daa0db640 1 -- 172.21.15.46:0/1265117194 >> v1:172.21.15.46:6789/0 conn(0x7f0da406f560 legacy=0x7f0da406f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0daa0db640 1 -- 172.21.15.46:0/1265117194 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da415e960 con 0x7f0da40af670 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0da90d9640 1 -- 172.21.15.46:0/1265117194 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586079061 0 0) 0x7f0d9c001490 con 0x7f0da406f560 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.183+0000 7f0da98da640 1 --2- 172.21.15.46:0/1265117194 >> v2:172.21.15.88:3300/0 conn(0x7f0da4073500 0x7f0da4073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.184+0000 7f0daa0db640 1 --2- 172.21.15.46:0/1265117194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da40af670 0x7f0da40d1170 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f0da40b7300 tx=0x7f0da002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3243af63aa51541e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.184+0000 7f0da90d9640 1 -- 172.21.15.46:0/1265117194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0037070 con 0x7f0da40af670 2024-05-09T03:19:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.184+0000 7f0da90d9640 1 -- 172.21.15.46:0/1265117194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da0032440 con 0x7f0da40af670 2024-05-09T03:19:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.184+0000 7f0da90d9640 1 -- 172.21.15.46:0/1265117194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0032810 con 0x7f0da40af670 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.184+0000 7f0dac33f640 1 -- 172.21.15.46:0/1265117194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da40af670 msgr2=0x7f0da40d1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.184+0000 7f0dac33f640 1 --2- 172.21.15.46:0/1265117194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da40af670 0x7f0da40d1170 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f0da40b7300 tx=0x7f0da002a270 comp rx=0 tx=0).stop 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.185+0000 7f0dac33f640 1 -- 172.21.15.46:0/1265117194 shutdown_connections 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.185+0000 7f0dac33f640 1 --2- 172.21.15.46:0/1265117194 >> v2:172.21.15.88:3300/0 conn(0x7f0da4073500 0x7f0da4073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.185+0000 7f0dac33f640 1 --2- 172.21.15.46:0/1265117194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da40af670 0x7f0da40d1170 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.185+0000 7f0dac33f640 1 -- 172.21.15.46:0/1265117194 >> 172.21.15.46:0/1265117194 conn(0x7f0da40b3910 msgr2=0x7f0da40b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.185+0000 7f0dac33f640 1 -- 172.21.15.46:0/1265117194 shutdown_connections 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.185+0000 7f0dac33f640 1 -- 172.21.15.46:0/1265117194 wait complete. 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.186+0000 7f0dac33f640 1 Processor -- start 2024-05-09T03:19:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.186+0000 7f0dac33f640 1 -- start start 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0dac33f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 0x7f0da4161e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0dac33f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0da4073500 0x7f0da41e9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0dac33f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0da40af670 0x7f0da415f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0dac33f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0da415f200 con 0x7f0da406f560 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0dac33f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0da415ef20 con 0x7f0da40af670 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0dac33f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0da40aeb80 con 0x7f0da4073500 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 0x7f0da4161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 0x7f0da4161e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47848/0 (socket says 172.21.15.46:47848) 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 -- 172.21.15.46:0/3597438441 learned_addr learned my addr 172.21.15.46:0/3597438441 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa8dc640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0da40af670 0x7f0da415f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0da98da640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0da4073500 0x7f0da41e9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 -- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0da4073500 msgr2=0x7f0da41e9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0da4073500 0x7f0da41e9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 -- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0da40af670 msgr2=0x7f0da415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0da40af670 0x7f0da415f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 -- 172.21.15.46:0/3597438441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da41e8f20 con 0x7f0da406f560 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0da98da640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0da4073500 0x7f0da41e9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.188+0000 7f0daa0db640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 0x7f0da4161e30 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f0da00090b0 tx=0x7f0da00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.189+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0037040 con 0x7f0da406f560 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.189+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da0004030 con 0x7f0da406f560 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.189+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da41e9200 con 0x7f0da406f560 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.189+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0004400 con 0x7f0da406f560 2024-05-09T03:19:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.189+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da41e97c0 con 0x7f0da406f560 2024-05-09T03:19:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.190+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0da406a270 con 0x7f0da406f560 2024-05-09T03:19:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.190+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0da003a070 con 0x7f0da406f560 2024-05-09T03:19:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.191+0000 7f0d9affd640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d880762d0 0x7f0d880787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.191+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f0da00b8750 con 0x7f0da406f560 2024-05-09T03:19:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.191+0000 7f0da98da640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d880762d0 0x7f0d880787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.192+0000 7f0da98da640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d880762d0 0x7f0d880787e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0da41f9860 tx=0x7f0d94008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.195+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0da0041020 con 0x7f0da406f560 2024-05-09T03:19:27.285 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:27.284+0000 7ffaf2aca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:19:27.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.430+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0da415dad0 con 0x7f0da406f560 2024-05-09T03:19:27.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.431+0000 7f0d9affd640 1 -- 172.21.15.46:0/3597438441 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v283) ==== 53+0+73 (secure 0 0 0) 0x7f0da0080f00 con 0x7f0da406f560 2024-05-09T03:19:27.431 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d880762d0 msgr2=0x7f0d880787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d880762d0 0x7f0d880787e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f0da41f9860 tx=0x7f0d94008040 comp rx=0 tx=0).stop 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 msgr2=0x7f0da4161e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 0x7f0da4161e30 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f0da00090b0 tx=0x7f0da00046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 shutdown_connections 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0da40af670 0x7f0da415f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0da4073500 0x7f0da41e9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0d880762d0 0x7f0d880787e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 --2- 172.21.15.46:0/3597438441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0da406f560 0x7f0da4161e30 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 >> 172.21.15.46:0/3597438441 conn(0x7f0da40b3910 msgr2=0x7f0da40b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 shutdown_connections 2024-05-09T03:19:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27.433+0000 7f0dac33f640 1 -- 172.21.15.46:0/3597438441 wait complete. 2024-05-09T03:19:27.444 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:19:27.445 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:19:27.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:27.445 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:19:27.462 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:19:27.462 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:19:27.462 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:19:27.462 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41950sec preferred_lft 41950sec 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:19:27.463 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:19:27.464 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:27.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:27.465 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:19:27.527 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:19:27.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:27.529 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:19:27.586 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:19:27.586 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:19:27.586 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:19:27.586 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:19:27.586 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:19:27.587 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:19:27.587 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:19:27.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:19:27.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:19:27.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:19:27.587 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:19:27.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:19:27.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:19:27.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:27.716 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:19:27.732 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:19:27.732 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:19:27.732 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:19:27.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:27.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:19:27.798 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:19:27.798 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:19:27.798 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:27.855 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:19:27.855 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:19:27.855 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:27.911 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:19:27.978 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:19:28.035 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:19:28.036 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:28.037 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:28.038 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:28.038 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:28.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:28.038 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:19:28.092 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:19:28.092 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:19:28.135 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:19:28.161 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:19:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:19:28.221 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:19:28.222 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:19:28.222 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:19:28.222 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:19:28.222 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:19:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:19:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:19:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:19:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:19:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:19:28.223 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:19:28.224 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:28.224 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:28.224 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:28.224 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:28.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:28.225 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:19:28.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:19:28.230+0000 7ff7cc124480 -1 init, newargv = 0x557e4b8fb780 newargc=15 2024-05-09T03:19:28.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[80590]: starting ceph client 2024-05-09T03:19:28.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[80590]: starting fuse 2024-05-09T03:19:28.248 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:19:28.248 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:28.248 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:19:28.249 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:19:28.250 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:28.250 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:19:28.250 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:19:28.369 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.80590.asok 2024-05-09T03:19:28.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:19:28.380 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.80590.asok 2024-05-09T03:19:28.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:28.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.80590.asok status 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "80590", 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:19:28.528 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9780, 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 9780 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1271969918 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:19:28.529 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1271969918 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.9780 192.168.0.1:0/1271969918", 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1271969918", 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 283, 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 115, 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:19:28.530 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:19:28.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:28.537 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:19:28.537 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:28.553 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:19:28.553 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:28.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:28.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:28.619 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:19:28.619 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> import json 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/./subdir/sixmegs'"'"') 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(e.errno) 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> print(json.dumps( 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> indent=2)) 2024-05-09T03:19:28.620 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:19:28.661 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:19:28.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-05-09T03:19:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-05-09T03:19:28.740 INFO:tasks.cephfs.test_damage:Result: EIO on client 2024-05-09T03:19:28.741 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.894+0000 7f82c9db7640 1 Processor -- start 2024-05-09T03:19:28.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.894+0000 7f82c9db7640 1 -- start start 2024-05-09T03:19:28.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c9db7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82c41613c0 0x7f82c40b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c9db7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c40b8110 con 0x7f82c40b5210 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c9db7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c41617d0 con 0x7f82c40b7d00 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c9db7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f82c40b5620 con 0x7f82c41613c0 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82c40b5210 0x7f82c40d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35854/0 (socket says 172.21.15.46:35854) 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c37fe640 1 -- 172.21.15.46:0/1360073408 learned_addr learned my addr 172.21.15.46:0/1360073408 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3170222797 0 0) 0x7f82b4001110 con 0x7f82c40b5210 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/1360073408 >> v2:172.21.15.88:3300/0 conn(0x7f82c41613c0 0x7f82c40b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82ac003520 con 0x7f82c40b5210 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 148577263 0 0) 0x7f82ac003520 con 0x7f82c40b5210 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 >> v2:172.21.15.88:3300/0 conn(0x7f82c41613c0 msgr2=0x7f82c40b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 --2- 172.21.15.46:0/1360073408 >> v2:172.21.15.88:3300/0 conn(0x7f82c41613c0 0x7f82c40b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:28.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 >> v1:172.21.15.88:6789/0 conn(0x7f82c40b7d00 legacy=0x7f82c4163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c40d5280 con 0x7f82c40b5210 2024-05-09T03:19:28.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f82b40018c0 con 0x7f82c40b5210 2024-05-09T03:19:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 >> v1:172.21.15.46:6789/0 conn(0x7f82c40b5210 legacy=0x7f82c40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82ac003a50 0x7f82ac003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82ac004800 0x7f82ac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac005250 0x7f82ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82c40b5620 con 0x7f82ac004800 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f82ac00c1e0 con 0x7f82ac003a50 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.896+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f82ac00c3e0 con 0x7f82ac005250 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82ac004800 0x7f82ac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c37fe640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82ac003a50 0x7f82ac003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c3fff640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac005250 0x7f82ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c2ffd640 1 -- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac005250 msgr2=0x7f82ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac005250 0x7f82ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c2ffd640 1 -- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82ac003a50 msgr2=0x7f82ac003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82ac003a50 0x7f82ac003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c2ffd640 1 -- 172.21.15.46:0/1360073408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82ac003520 con 0x7f82ac004800 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.897+0000 7f82c3fff640 1 -- 172.21.15.46:0/1360073408 reap_dead start 2024-05-09T03:19:28.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82ac004800 0x7f82ac004c10 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009ac0 tx=0x7f82ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545d8b5a415af4cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8036070 con 0x7f82ac004800 2024-05-09T03:19:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b8031400 con 0x7f82ac004800 2024-05-09T03:19:28.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c27fc640 1 -- 172.21.15.46:0/1360073408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8009690 con 0x7f82ac004800 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c9db7640 1 -- 172.21.15.46:0/1360073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82ac004800 msgr2=0x7f82ac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c9db7640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82ac004800 0x7f82ac004c10 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009ac0 tx=0x7f82ac005830 comp rx=0 tx=0).stop 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c9db7640 1 -- 172.21.15.46:0/1360073408 shutdown_connections 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c9db7640 1 --2- 172.21.15.46:0/1360073408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82ac004800 0x7f82ac004c10 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.898+0000 7f82c9db7640 1 -- 172.21.15.46:0/1360073408 >> 172.21.15.46:0/1360073408 conn(0x7f82c40d29a0 msgr2=0x7f82c40d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.899+0000 7f82c9db7640 1 -- 172.21.15.46:0/1360073408 shutdown_connections 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.899+0000 7f82c9db7640 1 -- 172.21.15.46:0/1360073408 wait complete. 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.900+0000 7f82c9db7640 1 Processor -- start 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.901+0000 7f82c9db7640 1 -- start start 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c9db7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac004800 0x7f82c40cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c9db7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 0x7f82c40c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c9db7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82c40c9d90 0x7f82c40cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c9db7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82c4160bc0 con 0x7f82c414c0b0 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c9db7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f82c41609a0 con 0x7f82c40c9d90 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c9db7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f82c4160ea0 con 0x7f82ac004800 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 0x7f82c40c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 0x7f82c40c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47862/0 (socket says 172.21.15.46:47862) 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 -- 172.21.15.46:0/272955813 learned_addr learned my addr 172.21.15.46:0/272955813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c3fff640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82c40c9d90 0x7f82c40cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c37fe640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac004800 0x7f82c40cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 -- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac004800 msgr2=0x7f82c40cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac004800 0x7f82c40cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 -- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82c40c9d90 msgr2=0x7f82c40cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82c40c9d90 0x7f82c40cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 -- 172.21.15.46:0/272955813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c40ca1a0 con 0x7f82c414c0b0 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.902+0000 7f82c2ffd640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 0x7f82c40c9850 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009be0 tx=0x7f82b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.903+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b803f020 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.903+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b8037040 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.903+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c415d430 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.903+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8036070 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.903+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c415d9f0 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.903+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c415d180 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.904+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f82b8042020 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.905+0000 7f82abfff640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82b00762d0 0x7f82b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.905+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f82b80b8220 con 0x7f82c414c0b0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.905+0000 7f82c37fe640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82b00762d0 0x7f82b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.906+0000 7f82c37fe640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82b00762d0 0x7f82b00787e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f82b4001eb0 tx=0x7f82b4007aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:28.910+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f82b8049030 con 0x7f82c414c0b0 2024-05-09T03:19:29.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.140+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82c406a270 con 0x7f82c414c0b0 2024-05-09T03:19:29.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.141+0000 7f82abfff640 1 -- 172.21.15.46:0/272955813 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 283 v283) ==== 95+0+5185 (secure 0 0 0) 0x7f82c40ca1a0 con 0x7f82c414c0b0 2024-05-09T03:19:29.142 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:29.142 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":283,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":283,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:23.314768+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9678},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9678":{"gid":9678,"name":"c","rank":0,"incarnation":280,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/1350199591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1350199591},{"type":"v1","addr":"172.21.15.46:6834","nonce":1350199591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9678,"qdb_cluster":[9678]},"id":6}]} 2024-05-09T03:19:29.142 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 283 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82b00762d0 msgr2=0x7f82b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82b00762d0 0x7f82b00787e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f82b4001eb0 tx=0x7f82b4007aa0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 msgr2=0x7f82c40c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 0x7f82c40c9850 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009be0 tx=0x7f82b8004730 comp rx=0 tx=0).stop 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 shutdown_connections 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82c40c9d90 0x7f82c40cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82b00762d0 0x7f82b00787e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82c414c0b0 0x7f82c40c9850 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 --2- 172.21.15.46:0/272955813 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f82ac004800 0x7f82c40cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 >> 172.21.15.46:0/272955813 conn(0x7f82c40d29a0 msgr2=0x7f82c41626e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 shutdown_connections 2024-05-09T03:19:29.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.143+0000 7f82c9db7640 1 -- 172.21.15.46:0/272955813 wait complete. 2024-05-09T03:19:29.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-05-09T03:19:29.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.271+0000 7f70afe53640 1 Processor -- start 2024-05-09T03:19:29.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.271+0000 7f70afe53640 1 -- start start 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70afe53640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70a815dfb0 0x7f70a8070420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70afe53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70a8070960 0x7f70a8160b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70afe53640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a81610b0 0x7f70a81614c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70afe53640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f70a8161a00 con 0x7f70a815dfb0 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70afe53640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f70a8070d70 con 0x7f70a8070960 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70afe53640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f70a8073650 con 0x7f70a81610b0 2024-05-09T03:19:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70a815dfb0 0x7f70a8070420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70a815dfb0 0x7f70a8070420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47874/0 (socket says 172.21.15.46:47874) 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 -- 172.21.15.46:0/2081450910 learned_addr learned my addr 172.21.15.46:0/2081450910 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/2081450910 >> v2:172.21.15.88:3300/0 conn(0x7f70a8070960 0x7f70a8160b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/2081450910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a81610b0 0x7f70a81614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 -- 172.21.15.46:0/2081450910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a81610b0 msgr2=0x7f70a81614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 --2- 172.21.15.46:0/2081450910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a81610b0 0x7f70a81614c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 -- 172.21.15.46:0/2081450910 >> v2:172.21.15.88:3300/0 conn(0x7f70a8070960 msgr2=0x7f70a8160b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 --2- 172.21.15.46:0/2081450910 >> v2:172.21.15.88:3300/0 conn(0x7f70a8070960 0x7f70a8160b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.273+0000 7f70adbef640 1 -- 172.21.15.46:0/2081450910 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a8073930 con 0x7f70a815dfb0 2024-05-09T03:19:29.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.274+0000 7f70adbef640 1 --2- 172.21.15.46:0/2081450910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a815dfb0 0x7f70a8070420 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f70a80cfe10 tx=0x7f709802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6febe31e0f697a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.274+0000 7f70acbed640 1 -- 172.21.15.46:0/2081450910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098036070 con 0x7f70a815dfb0 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.274+0000 7f70acbed640 1 -- 172.21.15.46:0/2081450910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7098031600 con 0x7f70a815dfb0 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.274+0000 7f70acbed640 1 -- 172.21.15.46:0/2081450910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70980319d0 con 0x7f70a815dfb0 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 -- 172.21.15.46:0/2081450910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a815dfb0 msgr2=0x7f70a8070420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 --2- 172.21.15.46:0/2081450910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a815dfb0 0x7f70a8070420 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f70a80cfe10 tx=0x7f709802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 -- 172.21.15.46:0/2081450910 shutdown_connections 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 --2- 172.21.15.46:0/2081450910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a81610b0 0x7f70a81614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 --2- 172.21.15.46:0/2081450910 >> v2:172.21.15.88:3300/0 conn(0x7f70a8070960 0x7f70a8160b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 --2- 172.21.15.46:0/2081450910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a815dfb0 0x7f70a8070420 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 -- 172.21.15.46:0/2081450910 >> 172.21.15.46:0/2081450910 conn(0x7f70a80b3690 msgr2=0x7f70a80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 -- 172.21.15.46:0/2081450910 shutdown_connections 2024-05-09T03:19:29.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.275+0000 7f70afe53640 1 -- 172.21.15.46:0/2081450910 wait complete. 2024-05-09T03:19:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.277+0000 7f70afe53640 1 Processor -- start 2024-05-09T03:19:29.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.277+0000 7f70afe53640 1 -- start start 2024-05-09T03:19:29.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70afe53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a8070960 0x7f70a813ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70afe53640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a815dfb0 0x7f70a8154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70afe53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 0x7f70a813db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70afe53640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70a80ce820 con 0x7f70a81610b0 2024-05-09T03:19:29.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70afe53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8073ef0 con 0x7f70a8070960 2024-05-09T03:19:29.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70afe53640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f70a80732b0 con 0x7f70a815dfb0 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 0x7f70a813db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 0x7f70a813db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47886/0 (socket says 172.21.15.46:47886) 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.278+0000 7f70ae3f0640 1 -- 172.21.15.46:0/1722439294 learned_addr learned my addr 172.21.15.46:0/1722439294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70adbef640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a8070960 0x7f70a813ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a815dfb0 0x7f70a8154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ae3f0640 1 -- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a815dfb0 msgr2=0x7f70a8154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a815dfb0 0x7f70a8154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ae3f0640 1 -- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a8070960 msgr2=0x7f70a813ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a8070960 0x7f70a813ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ae3f0640 1 -- 172.21.15.46:0/1722439294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a8139fd0 con 0x7f70a81610b0 2024-05-09T03:19:29.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70adbef640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a8070960 0x7f70a813ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:29.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 0x7f70a813db30 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f70a000d5d0 tx=0x7f70a000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70a67fc640 1 -- 172.21.15.46:0/1722439294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a00125f0 con 0x7f70a81610b0 2024-05-09T03:19:29.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70a67fc640 1 -- 172.21.15.46:0/1722439294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0019070 con 0x7f70a81610b0 2024-05-09T03:19:29.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a813a2b0 con 0x7f70a81610b0 2024-05-09T03:19:29.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70a67fc640 1 -- 172.21.15.46:0/1722439294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a00129b0 con 0x7f70a81610b0 2024-05-09T03:19:29.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.279+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a813a870 con 0x7f70a81610b0 2024-05-09T03:19:29.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.281+0000 7f70a67fc640 1 -- 172.21.15.46:0/1722439294 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f70a0012b50 con 0x7f70a81610b0 2024-05-09T03:19:29.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.282+0000 7f70a67fc640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70940762d0 0x7f70940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.282+0000 7f70a67fc640 1 -- 172.21.15.46:0/1722439294 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f70a0098570 con 0x7f70a81610b0 2024-05-09T03:19:29.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.282+0000 7f70adbef640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70940762d0 0x7f70940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.283+0000 7f70b6de8740 1 Processor -- start 2024-05-09T03:19:29.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.283+0000 7f70adbef640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70940762d0 0x7f70940787e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7098033000 tx=0x7f7098009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.283+0000 7f70b6de8740 1 -- start start 2024-05-09T03:19:29.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3dcf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dacf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dad450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70b6de8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ab3d0fd10 con 0x563ab39e6670 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ab3c97750 con 0x563ab3a07350 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563ab3d0bcd0 con 0x563ab39de490 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dad450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47892/0 (socket says 172.21.15.46:47892) 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 -- 172.21.15.46:0/2001717928 learned_addr learned my addr 172.21.15.46:0/2001717928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70adbef640 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3dcf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dacf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 -- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 msgr2=0x563ab3dacf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dacf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 -- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 msgr2=0x563ab3dcf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3dcf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.284+0000 7f70ae3f0640 1 -- 172.21.15.46:0/2001717928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ab3ae9790 con 0x563ab39e6670 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.285+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dad450 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f70a005d1e0 tx=0x7f70a0060700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2c2fcc7f3dd504 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.285+0000 7f708f7fe640 1 -- 172.21.15.46:0/2001717928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0069310 con 0x563ab39e6670 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.285+0000 7f708f7fe640 1 -- 172.21.15.46:0/2001717928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0060bf0 con 0x563ab39e6670 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.285+0000 7f708f7fe640 1 -- 172.21.15.46:0/2001717928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0064fa0 con 0x563ab39e6670 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.285+0000 7f70b6de8740 1 -- 172.21.15.46:0/2001717928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 msgr2=0x563ab3dad450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.285+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dad450 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f70a005d1e0 tx=0x7f70a0060700 comp rx=0 tx=0).stop 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 -- 172.21.15.46:0/2001717928 shutdown_connections 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dad450 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dacf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2001717928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3dcf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 -- 172.21.15.46:0/2001717928 >> 172.21.15.46:0/2001717928 conn(0x563ab39fb4a0 msgr2=0x563ab3daa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 -- 172.21.15.46:0/2001717928 shutdown_connections 2024-05-09T03:19:29.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 -- 172.21.15.46:0/2001717928 wait complete. 2024-05-09T03:19:29.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.286+0000 7f70b6de8740 1 -- start start 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.287+0000 7f70b6de8740 1 Processor -- start 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dc7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dc8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3ded7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ab3cccbd0 con 0x563ab39e6670 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ab3dc97f0 con 0x563ab3a07350 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563ab3dec4d0 con 0x563ab39de490 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70b6de8740 10 client.0 _ll_register_callbacks cb 0x563ab3cc1210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dc8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dc8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47898/0 (socket says 172.21.15.46:47898) 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 -- 172.21.15.46:0/2105972249 learned_addr learned my addr 172.21.15.46:0/2105972249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70adbef640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dc7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3ded7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 -- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 msgr2=0x563ab3dc7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dc7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 -- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 msgr2=0x563ab3ded7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3ded7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.288+0000 7f70ad3ee640 1 -- 172.21.15.46:0/2105972249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ab39b8130 con 0x563ab39e6670 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dc8c90 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f709c012330 tx=0x7f709c0127f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629481887d37f852 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f708cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f708cff9640 1 -- 172.21.15.46:0/2105972249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c019070 con 0x563ab39e6670 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70a0060bf0 con 0x563ab39e6670 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f708cff9640 1 -- 172.21.15.46:0/2105972249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f709c004240 con 0x563ab39e6670 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f708cff9640 1 -- 172.21.15.46:0/2105972249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f709c025050 con 0x563ab39e6670 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 10 client.9798 fetch_fsmap learned FSMap version 283 2024-05-09T03:19:29.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=283,monmap=2+}) -- 0x563ab3ae95c0 con 0x563ab39e6670 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f708cff9640 1 -- 172.21.15.46:0/2105972249 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0060bf0 con 0x563ab39e6670 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f708cff9640 1 -- 172.21.15.46:0/2105972249 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 283) ==== 3494+0+0 (secure 0 0 0) 0x7f709c003610 con 0x563ab39e6670 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 10 client.9798 fetch_fsmap finished waiting for FSMap version 283 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 10 client.9798 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 20 client.9798 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab39c6130 0x563ab3df0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 4 client.9798 mds_command: new command op to 9678 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.289+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 --> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ab3d1d920 con 0x563ab39c6130 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.290+0000 7f70adbef640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab39c6130 0x563ab3df0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.290+0000 7f708cff9640 10 client.9798 ms_handle_connect on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.290+0000 7f70adbef640 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab39c6130 0x563ab3df0c50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73cd670e951df4b8 server_cookie=5093ba0ee2eace96 in_seq=0 out_seq=0 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f708cff9640 1 -- 172.21.15.46:0/2105972249 <== mds.0 v2:172.21.15.46:6832/1350199591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563ab3d1d920 con 0x563ab39c6130 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f708cff9640 10 client.9798 handle_command_reply: tid=0 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 client.9798 shutdown 2024-05-09T03:19:29.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 msgr2=0x563ab3dc8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dc8c90 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f709c012330 tx=0x7f709c0127f0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 shutdown_connections 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab39c6130 0x563ab3df0c50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab3a07350 0x563ab3ded7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab39e6670 0x563ab3dc8c90 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 --2- 172.21.15.46:0/2105972249 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3dc7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f708cff9640 0 client.9798 ms_handle_reset on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 >> 172.21.15.46:0/2105972249 conn(0x563ab39fb4a0 msgr2=0x563ab3df3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 shutdown_connections 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 1 -- 172.21.15.46:0/2105972249 wait complete. 2024-05-09T03:19:29.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.291+0000 7f70b6de8740 20 client.9798 trim_cache size 0 max 16384 2024-05-09T03:19:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.305+0000 7f70b6de8740 1 Processor -- start 2024-05-09T03:19:29.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.305+0000 7f70b6de8740 1 -- start start 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3da6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39e6670 0x563ab3e6a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3deeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70b6de8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70a82017e0 con 0x563ab3a07350 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f70a8200b70 con 0x563ab39e6670 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f70a8201190 con 0x563ab39de490 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3deeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3deeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47912/0 (socket says 172.21.15.46:47912) 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 -- 172.21.15.46:0/870512369 learned_addr learned my addr 172.21.15.46:0/870512369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39e6670 0x563ab3e6a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70adbef640 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3da6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 -- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 msgr2=0x563ab3da6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3da6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 -- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39e6670 msgr2=0x563ab3e6a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39e6670 0x563ab3e6a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 -- 172.21.15.46:0/870512369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ab39eb150 con 0x563ab3a07350 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70adbef640 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3da6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3deeb70 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f70a0060b80 tx=0x7f70a0081000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61681411ea5c6db7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f708effd640 1 -- 172.21.15.46:0/870512369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a0069310 con 0x563ab3a07350 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f708effd640 1 -- 172.21.15.46:0/870512369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0060bf0 con 0x563ab3a07350 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.307+0000 7f708effd640 1 -- 172.21.15.46:0/870512369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a00699c0 con 0x563ab3a07350 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 -- 172.21.15.46:0/870512369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 msgr2=0x563ab3deeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3deeb70 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f70a0060b80 tx=0x7f70a0081000 comp rx=0 tx=0).stop 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 -- 172.21.15.46:0/870512369 shutdown_connections 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3deeb70 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39e6670 0x563ab3e6a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 --2- 172.21.15.46:0/870512369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39de490 0x563ab3da6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 -- 172.21.15.46:0/870512369 >> 172.21.15.46:0/870512369 conn(0x563ab39fb4a0 msgr2=0x563ab3ded7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 -- 172.21.15.46:0/870512369 shutdown_connections 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.308+0000 7f70b6de8740 1 -- 172.21.15.46:0/870512369 wait complete. 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.309+0000 7f70b6de8740 1 -- start start 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.309+0000 7f70b6de8740 1 Processor -- start 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39de490 0x563ab3e66720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39e6670 0x563ab3da69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3df0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ab3def8c0 con 0x563ab3a07350 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ab3dc9280 con 0x563ab39de490 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x563ab3defeb0 con 0x563ab39e6670 2024-05-09T03:19:29.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70b6de8740 10 client.0 _ll_register_callbacks cb 0x7f709c00d640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3df0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3df0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47922/0 (socket says 172.21.15.46:47922) 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 -- 172.21.15.46:0/3835815296 learned_addr learned my addr 172.21.15.46:0/3835815296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70adbef640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39de490 0x563ab3e66720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ad3ee640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39e6670 0x563ab3da69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 -- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39e6670 msgr2=0x563ab3da69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39e6670 0x563ab3da69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 -- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39de490 msgr2=0x563ab3e66720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39de490 0x563ab3e66720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 -- 172.21.15.46:0/3835815296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ab3e66c60 con 0x563ab3a07350 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f70ae3f0640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3df0bb0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f70a005cc70 tx=0x7f70a0098070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfcd922ce347000 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.311+0000 7f708cff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:29.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f708cff9640 1 -- 172.21.15.46:0/3835815296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a006e060 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f708cff9640 1 -- 172.21.15.46:0/3835815296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f70a00980e0 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70a0060bf0 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f708cff9640 1 -- 172.21.15.46:0/3835815296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=283) ==== 24+0+0 (secure 0 0 0) 0x7f70a0069310 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 10 client.9804 fetch_fsmap learned FSMap version 283 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=283,monmap=2+}) -- 0x563ab3dca3c0 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f708cff9640 1 -- 172.21.15.46:0/3835815296 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a0060bf0 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f708cff9640 1 -- 172.21.15.46:0/3835815296 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 283) ==== 3494+0+0 (secure 0 0 0) 0x7f70a0013070 con 0x563ab3a07350 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 10 client.9804 fetch_fsmap finished waiting for FSMap version 283 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 10 client.9804 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 20 client.9804 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab3e52900 0x563ab3e52d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 4 client.9804 mds_command: new command op to 9678 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-05-09T03:19:29.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 --> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x563ab3dceca0 con 0x563ab3e52900 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.312+0000 7f70adbef640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab3e52900 0x563ab3e52d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f708cff9640 10 client.9804 ms_handle_connect on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70adbef640 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab3e52900 0x563ab3e52d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81fe215f1e8b9358 server_cookie=d1356a86cef7471c in_seq=0 out_seq=0 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f708cff9640 1 -- 172.21.15.46:0/3835815296 <== mds.0 v2:172.21.15.46:6832/1350199591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+215 (crc 0 0 0) 0x563ab3dceca0 con 0x563ab3e52900 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f708cff9640 10 client.9804 handle_command_reply: tid=0 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 client.9804 shutdown 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 msgr2=0x563ab3df0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3df0bb0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f70a005cc70 tx=0x7f70a0098070 comp rx=0 tx=0).stop 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 shutdown_connections 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:6832/1350199591,v1:172.21.15.46:6834/1350199591] conn(0x563ab3e52900 0x563ab3e52d10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ab3a07350 0x563ab3df0bb0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x563ab39e6670 0x563ab3da69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f708cff9640 0 client.9804 ms_handle_reset on v2:172.21.15.46:6832/1350199591 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 --2- 172.21.15.46:0/3835815296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ab39de490 0x563ab3e66720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 >> 172.21.15.46:0/3835815296 conn(0x563ab39fb4a0 msgr2=0x563ab3dc7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 shutdown_connections 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.313+0000 7f70b6de8740 1 -- 172.21.15.46:0/3835815296 wait complete. 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70b6de8740 20 client.9804 trim_cache size 0 max 16384 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "dentry", 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1212279463, 2024-05-09T03:19:29.322 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511627776, 2024-05-09T03:19:29.323 INFO:teuthology.orchestra.run.smithi046.stdout: "frag": "*", 2024-05-09T03:19:29.323 INFO:teuthology.orchestra.run.smithi046.stdout: "dname": "sixmegs", 2024-05-09T03:19:29.323 INFO:teuthology.orchestra.run.smithi046.stdout: "snap_id": "head", 2024-05-09T03:19:29.323 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/subdir/sixmegs" 2024-05-09T03:19:29.323 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:19:29.323 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70940762d0 msgr2=0x7f70940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70940762d0 0x7f70940787e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7098033000 tx=0x7f7098009000 comp rx=0 tx=0).stop 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 msgr2=0x7f70a813db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 0x7f70a813db30 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f70a000d5d0 tx=0x7f70a000da90 comp rx=0 tx=0).stop 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 shutdown_connections 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f70940762d0 0x7f70940787e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70a81610b0 0x7f70a813db30 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70a815dfb0 0x7f70a8154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 --2- 172.21.15.46:0/1722439294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70a8070960 0x7f70a813ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 >> 172.21.15.46:0/1722439294 conn(0x7f70a80b3690 msgr2=0x7f70a80b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 shutdown_connections 2024-05-09T03:19:29.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.314+0000 7f70afe53640 1 -- 172.21.15.46:0/1722439294 wait complete. 2024-05-09T03:19:29.326 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 100.00000000:stray0_head' 2024-05-09T03:19:29.326 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:29.326 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:29.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:19:29.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.496+0000 7f2a43da8640 1 Processor -- start 2024-05-09T03:19:29.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.497+0000 7f2a43da8640 1 -- start start 2024-05-09T03:19:29.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a43da8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a3c0b5140 0x7f2a3c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a43da8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a3c0d3fc0 0x7f2a3c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a43da8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a3c0b80b0 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a43da8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2a3c0d3480 con 0x7f2a3c0d3fc0 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a43da8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2a3c0b5550 con 0x7f2a3c0b5140 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a42345640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a3c0d3fc0 0x7f2a3c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a42345640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a3c0d3fc0 0x7f2a3c0b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47926/0 (socket says 172.21.15.46:47926) 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.498+0000 7f2a42345640 1 -- 172.21.15.46:0/3009281852 learned_addr learned my addr 172.21.15.46:0/3009281852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a41b44640 1 --2- 172.21.15.46:0/3009281852 >> v2:172.21.15.88:3300/0 conn(0x7f2a3c0b5140 0x7f2a3c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a42345640 1 -- 172.21.15.46:0/3009281852 >> v2:172.21.15.88:3300/0 conn(0x7f2a3c0b5140 msgr2=0x7f2a3c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a42345640 1 --2- 172.21.15.46:0/3009281852 >> v2:172.21.15.88:3300/0 conn(0x7f2a3c0b5140 0x7f2a3c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a42345640 1 -- 172.21.15.46:0/3009281852 >> v1:172.21.15.88:6789/0 conn(0x7f2a3c0b7ca0 legacy=0x7f2a3c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a42345640 1 -- 172.21.15.46:0/3009281852 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c0d44e0 con 0x7f2a3c0d3fc0 2024-05-09T03:19:29.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a41b44640 1 --2- 172.21.15.46:0/3009281852 >> v2:172.21.15.88:3300/0 conn(0x7f2a3c0b5140 0x7f2a3c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a42345640 1 --2- 172.21.15.46:0/3009281852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0b9380 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0b88d0 tx=0x7f2a2c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee68caf88a4333b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.499+0000 7f2a40b42640 1 -- 172.21.15.46:0/3009281852 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c036070 con 0x7f2a3c0d3fc0 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a40b42640 1 -- 172.21.15.46:0/3009281852 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c031600 con 0x7f2a3c0d3fc0 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a40b42640 1 -- 172.21.15.46:0/3009281852 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c0319d0 con 0x7f2a3c0d3fc0 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 -- 172.21.15.46:0/3009281852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0d3fc0 msgr2=0x7f2a3c0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 --2- 172.21.15.46:0/3009281852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0b9380 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0b88d0 tx=0x7f2a2c02a730 comp rx=0 tx=0).stop 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 -- 172.21.15.46:0/3009281852 shutdown_connections 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 --2- 172.21.15.46:0/3009281852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0b9380 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 --2- 172.21.15.46:0/3009281852 >> v2:172.21.15.88:3300/0 conn(0x7f2a3c0b5140 0x7f2a3c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 -- 172.21.15.46:0/3009281852 >> 172.21.15.46:0/3009281852 conn(0x7f2a3c08d3f0 msgr2=0x7f2a3c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 -- 172.21.15.46:0/3009281852 shutdown_connections 2024-05-09T03:19:29.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.500+0000 7f2a43da8640 1 -- 172.21.15.46:0/3009281852 wait complete. 2024-05-09T03:19:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.502+0000 7f2a43da8640 1 Processor -- start 2024-05-09T03:19:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.502+0000 7f2a43da8640 1 -- start start 2024-05-09T03:19:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.503+0000 7f2a43da8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a3c0b5140 0x7f2a3c0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.503+0000 7f2a43da8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 0x7f2a3c14e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.503+0000 7f2a43da8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.503+0000 7f2a43da8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0d4d80 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.503+0000 7f2a43da8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0d4aa0 con 0x7f2a3c0b5140 2024-05-09T03:19:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.503+0000 7f2a43da8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3c0d5580 con 0x7f2a3c0d3fc0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 0x7f2a3c14e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 0x7f2a3c14e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47938/0 (socket says 172.21.15.46:47938) 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 -- 172.21.15.46:0/1814442545 learned_addr learned my addr 172.21.15.46:0/1814442545 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a42345640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41b44640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a3c0b5140 0x7f2a3c0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 -- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a3c0d3fc0 msgr2=0x7f2a3c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 -- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a3c0b5140 msgr2=0x7f2a3c0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a3c0b5140 0x7f2a3c0ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 -- 172.21.15.46:0/1814442545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c14e620 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a42345640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a41343640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 0x7f2a3c14e0e0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000b3f0 tx=0x7f2a3000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30015020 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a30011830 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c0cab20 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30011c20 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.504+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c0cb0e0 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.505+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c0ca890 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.506+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a30018020 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.507+0000 7f2a3a7fc640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2a28076300 0x7f2a28078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.507+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f2a30098070 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.507+0000 7f2a41b44640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2a28076300 0x7f2a28078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.508+0000 7f2a41b44640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2a28076300 0x7f2a28078810 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400a950 tx=0x7f2a34008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.512+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2a3001f020 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.715+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2a3c06a270 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.715+0000 7f2a3a7fc640 1 -- 172.21.15.46:0/1814442545 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 62 entries v115) ==== 81+0+3766 (secure 0 0 0) 0x7f2a30060e40 con 0x7f2a3c0b7ca0 2024-05-09T03:19:29.716 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:29.716 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:29.716 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:29.716 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:29.717 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:29.718 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:29.719 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:29.720 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:29.721 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:29.722 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:29.722 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:listed 62 entries 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.717+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2a28076300 msgr2=0x7f2a28078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.717+0000 7f2a43da8640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2a28076300 0x7f2a28078810 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400a950 tx=0x7f2a34008040 comp rx=0 tx=0).stop 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.717+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 msgr2=0x7f2a3c14e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.717+0000 7f2a43da8640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 0x7f2a3c14e0e0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000b3f0 tx=0x7f2a3000b8b0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 shutdown_connections 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2a3c0d3fc0 0x7f2a3c0ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2a28076300 0x7f2a28078810 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a3c0b7ca0 0x7f2a3c14e0e0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 --2- 172.21.15.46:0/1814442545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a3c0b5140 0x7f2a3c0ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 >> 172.21.15.46:0/1814442545 conn(0x7f2a3c08d3f0 msgr2=0x7f2a3c160130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 shutdown_connections 2024-05-09T03:19:29.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.718+0000 7f2a43da8640 1 -- 172.21.15.46:0/1814442545 wait complete. 2024-05-09T03:19:29.729 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:29.730 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:29.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:19:29.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.901+0000 7f8c9ef06640 1 Processor -- start 2024-05-09T03:19:29.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.901+0000 7f8c9ef06640 1 -- start start 2024-05-09T03:19:29.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9ef06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c980cf140 0x7f8c980b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9ef06640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c9806ebe0 con 0x7f8c980d2800 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9ef06640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c9806eec0 con 0x7f8c980b4700 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9ef06640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8c9806f1a0 con 0x7f8c980cf140 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9d4a3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c980cf140 0x7f8c980b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c97fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c980d2800 0x7f8c980b3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35870/0 (socket says 172.21.15.46:35870) 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c97fff640 1 -- 172.21.15.46:0/2068908437 learned_addr learned my addr 172.21.15.46:0/2068908437 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9d4a3640 1 -- 172.21.15.46:0/2068908437 >> v1:172.21.15.88:6789/0 conn(0x7f8c980b4700 legacy=0x7f8c980d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9d4a3640 1 -- 172.21.15.46:0/2068908437 >> v1:172.21.15.46:6789/0 conn(0x7f8c980d2800 legacy=0x7f8c980b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c977fe640 1 -- 172.21.15.46:0/2068908437 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469941091 0 0) 0x7f8c9806ebe0 con 0x7f8c980d2800 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.903+0000 7f8c9d4a3640 1 -- 172.21.15.46:0/2068908437 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9806f480 con 0x7f8c980cf140 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.904+0000 7f8c9d4a3640 1 --2- 172.21.15.46:0/2068908437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980cf140 0x7f8c980b4120 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009d90 tx=0x7f8c9002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4b47bb0777acf40 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.904+0000 7f8c977fe640 1 -- 172.21.15.46:0/2068908437 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90031600 con 0x7f8c980cf140 2024-05-09T03:19:29.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.904+0000 7f8c977fe640 1 -- 172.21.15.46:0/2068908437 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c900317a0 con 0x7f8c980cf140 2024-05-09T03:19:29.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.904+0000 7f8c977fe640 1 -- 172.21.15.46:0/2068908437 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90031b70 con 0x7f8c980cf140 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 -- 172.21.15.46:0/2068908437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980cf140 msgr2=0x7f8c980b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/2068908437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980cf140 0x7f8c980b4120 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009d90 tx=0x7f8c9002a5d0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 -- 172.21.15.46:0/2068908437 shutdown_connections 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/2068908437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980cf140 0x7f8c980b4120 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 -- 172.21.15.46:0/2068908437 >> 172.21.15.46:0/2068908437 conn(0x7f8c980b55d0 msgr2=0x7f8c980b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 -- 172.21.15.46:0/2068908437 shutdown_connections 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.905+0000 7f8c9ef06640 1 -- 172.21.15.46:0/2068908437 wait complete. 2024-05-09T03:19:29.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.907+0000 7f8c9ef06640 1 Processor -- start 2024-05-09T03:19:29.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.907+0000 7f8c9ef06640 1 -- start start 2024-05-09T03:19:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9ef06640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c980b4700 0x7f8c981eb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9ef06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 0x7f8c981f9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9ef06640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c981fa300 0x7f8c981fa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9ef06640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c980b51d0 con 0x7f8c980d2800 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9ef06640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c980b4c50 con 0x7f8c981fa300 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9ef06640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f8c981fac50 con 0x7f8c980b4700 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 0x7f8c981f9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c97fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 0x7f8c981f9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47954/0 (socket says 172.21.15.46:47954) 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c97fff640 1 -- 172.21.15.46:0/1187414785 learned_addr learned my addr 172.21.15.46:0/1187414785 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c9d4a3640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c981fa300 0x7f8c981fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c97fff640 1 -- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c980b4700 msgr2=0x7f8c981eb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.908+0000 7f8c97fff640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c980b4700 0x7f8c981eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c97fff640 1 -- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c981fa300 msgr2=0x7f8c981fa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c97fff640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c981fa300 0x7f8c981fa710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c97fff640 1 -- 172.21.15.46:0/1187414785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c981ebdd0 con 0x7f8c980d2800 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c97fff640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 0x7f8c981f9dc0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00c3c0 tx=0x7f8c8c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c0074e0 con 0x7f8c980d2800 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8c0047e0 con 0x7f8c980d2800 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c981ec0b0 con 0x7f8c980d2800 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c004bd0 con 0x7f8c980d2800 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.909+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c981f7990 con 0x7f8c980d2800 2024-05-09T03:19:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.910+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c9806a270 con 0x7f8c980d2800 2024-05-09T03:19:29.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.911+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c8c011070 con 0x7f8c980d2800 2024-05-09T03:19:29.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.911+0000 7f8c957fa640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c84076300 0x7f8c84078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:29.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.911+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f8c8c0977a0 con 0x7f8c980d2800 2024-05-09T03:19:29.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.912+0000 7f8c9cca2640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c84076300 0x7f8c84078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:29.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.912+0000 7f8c9cca2640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c84076300 0x7f8c84078810 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8c88006fd0 tx=0x7f8c88006bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:29.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:29.916+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8c8c060570 con 0x7f8c980d2800 2024-05-09T03:19:30.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.121+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8c9806ebd0 con 0x7f8c980d2800 2024-05-09T03:19:30.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.122+0000 7f8c957fa640 1 -- 172.21.15.46:0/1187414785 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 62 entries v115) ==== 81+0+3766 (secure 0 0 0) 0x7f8c8c05fe70 con 0x7f8c980d2800 2024-05-09T03:19:30.123 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:30.123 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:30.123 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:30.123 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:30.124 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:30.125 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:30.127 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:30.130 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:30.131 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:30.131 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:30.131 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:30.131 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:30.132 INFO:teuthology.orchestra.run.smithi046.stderr:listed 62 entries 2024-05-09T03:19:30.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c84076300 msgr2=0x7f8c84078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c84076300 0x7f8c84078810 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f8c88006fd0 tx=0x7f8c88006bc0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 msgr2=0x7f8c981f9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 0x7f8c981f9dc0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c00c3c0 tx=0x7f8c8c00c880 comp rx=0 tx=0).stop 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 shutdown_connections 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c981fa300 0x7f8c981fa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8c84076300 0x7f8c84078810 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c980d2800 0x7f8c981f9dc0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 --2- 172.21.15.46:0/1187414785 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f8c980b4700 0x7f8c981eb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 >> 172.21.15.46:0/1187414785 conn(0x7f8c980b55d0 msgr2=0x7f8c980d20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 shutdown_connections 2024-05-09T03:19:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.124+0000 7f8c9ef06640 1 -- 172.21.15.46:0/1187414785 wait complete. 2024-05-09T03:19:30.137 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:19:30.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:30.137 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:30.163 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:19:30.163 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:30.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:30.164 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:30.220 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.376+0000 7f37782d9640 1 Processor -- start 2024-05-09T03:19:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.376+0000 7f37782d9640 1 -- start start 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f37782d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700bb770 0x7f37700d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f37782d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37700d3180 0x7f37701642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f37782d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3770164900 con 0x7f3770163eb0 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f37782d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f37700d4bb0 con 0x7f37700d3180 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f37782d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3770164be0 con 0x7f37700bb770 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f3776876640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f37700d3180 0x7f37701642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f3775874640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3770163eb0 0x7f37700d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35884/0 (socket says 172.21.15.46:35884) 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.377+0000 7f3775874640 1 -- 172.21.15.46:0/2149331989 learned_addr learned my addr 172.21.15.46:0/2149331989 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3776075640 1 --2- 172.21.15.46:0/2149331989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700bb770 0x7f37700d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3776876640 1 -- 172.21.15.46:0/2149331989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700bb770 msgr2=0x7f37700d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3776876640 1 --2- 172.21.15.46:0/2149331989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700bb770 0x7f37700d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3776876640 1 -- 172.21.15.46:0/2149331989 >> v1:172.21.15.46:6789/0 conn(0x7f3770163eb0 legacy=0x7f37700d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3776876640 1 -- 172.21.15.46:0/2149331989 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37700d4840 con 0x7f37700d3180 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3775073640 1 -- 172.21.15.46:0/2149331989 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 567476671 0 0) 0x7f3770164900 con 0x7f3770163eb0 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.378+0000 7f3776876640 1 --2- 172.21.15.46:0/2149331989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700d3180 0x7f37701642c0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f3760009d90 tx=0x7f376002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31487a07d566142a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f3775073640 1 -- 172.21.15.46:0/2149331989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760007740 con 0x7f37700d3180 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f3775073640 1 -- 172.21.15.46:0/2149331989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37600078e0 con 0x7f37700d3180 2024-05-09T03:19:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f3775073640 1 -- 172.21.15.46:0/2149331989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760007cb0 con 0x7f37700d3180 2024-05-09T03:19:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 -- 172.21.15.46:0/2149331989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700d3180 msgr2=0x7f37701642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 --2- 172.21.15.46:0/2149331989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700d3180 0x7f37701642c0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f3760009d90 tx=0x7f376002a660 comp rx=0 tx=0).stop 2024-05-09T03:19:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 -- 172.21.15.46:0/2149331989 shutdown_connections 2024-05-09T03:19:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 --2- 172.21.15.46:0/2149331989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700d3180 0x7f37701642c0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 --2- 172.21.15.46:0/2149331989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700bb770 0x7f37700d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 -- 172.21.15.46:0/2149331989 >> 172.21.15.46:0/2149331989 conn(0x7f377006e370 msgr2=0x7f377006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 -- 172.21.15.46:0/2149331989 shutdown_connections 2024-05-09T03:19:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.379+0000 7f37782d9640 1 -- 172.21.15.46:0/2149331989 wait complete. 2024-05-09T03:19:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.381+0000 7f37782d9640 1 Processor -- start 2024-05-09T03:19:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.381+0000 7f37782d9640 1 -- start start 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 0x7f37700cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3770163eb0 0x7f37701f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700cd630 0x7f37701f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f37700d0bc0 con 0x7f37700bb770 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f37700bbb80 con 0x7f3770163eb0 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f37700d0d40 con 0x7f37700cd630 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 0x7f37700cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 0x7f37700cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47984/0 (socket says 172.21.15.46:47984) 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 -- 172.21.15.46:0/1162273920 learned_addr learned my addr 172.21.15.46:0/1162273920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776876640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700cd630 0x7f37701f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3775874640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3770163eb0 0x7f37701f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 -- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700cd630 msgr2=0x7f37701f8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700cd630 0x7f37701f8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 -- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3770163eb0 msgr2=0x7f37701f7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3770163eb0 0x7f37701f7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 -- 172.21.15.46:0/1162273920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37701f7600 con 0x7f37700bb770 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3775874640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3770163eb0 0x7f37701f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f3776075640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 0x7f37700cf120 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f376800d600 tx=0x7f376800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768004240 con 0x7f37700bb770 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37680043e0 con 0x7f37700bb770 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37700cde60 con 0x7f37700bb770 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768003830 con 0x7f37700bb770 2024-05-09T03:19:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.383+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37701f78a0 con 0x7f37700bb770 2024-05-09T03:19:30.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.384+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f377006a270 con 0x7f37700bb770 2024-05-09T03:19:30.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.385+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f37680039d0 con 0x7f37700bb770 2024-05-09T03:19:30.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.386+0000 7f376effd640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f375c076300 0x7f375c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.386+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8433+0+0 (secure 0 0 0) 0x7f3768098380 con 0x7f37700bb770 2024-05-09T03:19:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.386+0000 7f3775874640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f375c076300 0x7f375c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.387+0000 7f3775874640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f375c076300 0x7f375c078810 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3764000f60 tx=0x7f3764009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.391+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f3768061150 con 0x7f37700bb770 2024-05-09T03:19:30.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.621+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f37700b8b00 con 0x7f37700bb770 2024-05-09T03:19:30.622 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:30.622+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:30.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.623+0000 7f376effd640 1 -- 172.21.15.46:0/1162273920 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v284) ==== 161+0+0 (secure 0 0 0) 0x7f3768060a50 con 0x7f37700bb770 2024-05-09T03:19:30.624 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f375c076300 msgr2=0x7f375c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f375c076300 0x7f375c078810 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3764000f60 tx=0x7f3764009040 comp rx=0 tx=0).stop 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 msgr2=0x7f37700cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 0x7f37700cf120 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f376800d600 tx=0x7f376800dac0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 shutdown_connections 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f37700cd630 0x7f37701f8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3770163eb0 0x7f37701f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f375c076300 0x7f375c078810 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 --2- 172.21.15.46:0/1162273920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37700bb770 0x7f37700cf120 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 >> 172.21.15.46:0/1162273920 conn(0x7f377006e370 msgr2=0x7f37700b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:30.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 shutdown_connections 2024-05-09T03:19:30.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.627+0000 7f37782d9640 1 -- 172.21.15.46:0/1162273920 wait complete. 2024-05-09T03:19:30.629 INFO:tasks.ceph.mds.c.smithi046.stderr: -483> 2024-05-09T03:19:27.284+0000 7ffaf2aca640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:19:30.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:30.687 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.757+0000 7f58ada2b640 1 Processor -- start 2024-05-09T03:19:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.757+0000 7f58ada2b640 1 -- start start 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.758+0000 7f58ada2b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58a8162880 0x7f58a80b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.758+0000 7f58ada2b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f58a80d04c0 0x7f58a80d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.758+0000 7f58ada2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a80d0e10 con 0x7f58a80d2410 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.758+0000 7f58ada2b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f58a815f340 con 0x7f58a8162880 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.758+0000 7f58ada2b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f58a8161da0 con 0x7f58a80d04c0 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58a8162880 0x7f58a80b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58a8162880 0x7f58a80b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48008/0 (socket says 172.21.15.46:48008) 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 -- 172.21.15.46:0/339957888 learned_addr learned my addr 172.21.15.46:0/339957888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a77fe640 1 --1- 172.21.15.46:0/339957888 >> v1:172.21.15.46:6789/0 conn(0x7f58a80d2410 0x7f58a80d2820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35908/0 (socket says 172.21.15.46:35908) 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a67fc640 1 --2- 172.21.15.46:0/339957888 >> v2:172.21.15.88:3300/0 conn(0x7f58a80d04c0 0x7f58a80d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 -- 172.21.15.46:0/339957888 >> v2:172.21.15.88:3300/0 conn(0x7f58a80d04c0 msgr2=0x7f58a80d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/339957888 >> v2:172.21.15.88:3300/0 conn(0x7f58a80d04c0 0x7f58a80d08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 -- 172.21.15.46:0/339957888 >> v1:172.21.15.46:6789/0 conn(0x7f58a80d2410 legacy=0x7f58a80d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a5ffb640 1 -- 172.21.15.46:0/339957888 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 383301104 0 0) 0x7f58a80d0e10 con 0x7f58a80d2410 2024-05-09T03:19:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.759+0000 7f58a6ffd640 1 -- 172.21.15.46:0/339957888 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a8162080 con 0x7f58a8162880 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.760+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/339957888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a8162880 0x7f58a80b2d60 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f58a80b6be0 tx=0x7f589802a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1150d883605e844 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.760+0000 7f58a5ffb640 1 -- 172.21.15.46:0/339957888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898037070 con 0x7f58a8162880 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.760+0000 7f58a5ffb640 1 -- 172.21.15.46:0/339957888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5898032440 con 0x7f58a8162880 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.760+0000 7f58a5ffb640 1 -- 172.21.15.46:0/339957888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898032810 con 0x7f58a8162880 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 -- 172.21.15.46:0/339957888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a8162880 msgr2=0x7f58a80b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 --2- 172.21.15.46:0/339957888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a8162880 0x7f58a80b2d60 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f58a80b6be0 tx=0x7f589802a150 comp rx=0 tx=0).stop 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 -- 172.21.15.46:0/339957888 shutdown_connections 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 --2- 172.21.15.46:0/339957888 >> v2:172.21.15.88:3300/0 conn(0x7f58a80d04c0 0x7f58a80d08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 --2- 172.21.15.46:0/339957888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a8162880 0x7f58a80b2d60 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 -- 172.21.15.46:0/339957888 >> 172.21.15.46:0/339957888 conn(0x7f58a80b55f0 msgr2=0x7f58a80b5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 -- 172.21.15.46:0/339957888 shutdown_connections 2024-05-09T03:19:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.761+0000 7f58ada2b640 1 -- 172.21.15.46:0/339957888 wait complete. 2024-05-09T03:19:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.762+0000 7f58ada2b640 1 Processor -- start 2024-05-09T03:19:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.763+0000 7f58ada2b640 1 -- start start 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.764+0000 7f58ada2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58a80d04c0 0x7f58a80cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.764+0000 7f58ada2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 0x7f58a81f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.764+0000 7f58ada2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58a8162880 0x7f58a81e9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.764+0000 7f58a67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 0x7f58a81f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.764+0000 7f58ada2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f58a80d2f30 con 0x7f58a80d2410 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 0x7f58a81f7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48012/0 (socket says 172.21.15.46:48012) 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a67fc640 1 -- 172.21.15.46:0/4211969400 learned_addr learned my addr 172.21.15.46:0/4211969400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f58a80b32a0 con 0x7f58a8162880 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f58a80d30b0 con 0x7f58a80d04c0 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58a80d04c0 0x7f58a80cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a77fe640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58a8162880 0x7f58a81e9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a6ffd640 1 -- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58a8162880 msgr2=0x7f58a81e9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58a8162880 0x7f58a81e9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a6ffd640 1 -- 172.21.15.46:0/4211969400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 msgr2=0x7f58a81f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 0x7f58a81f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a6ffd640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a80cde50 con 0x7f58a80d04c0 2024-05-09T03:19:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.765+0000 7f58a67fc640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 0x7f58a81f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.767+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58a80d04c0 0x7f58a80cd910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5898031000 tx=0x7f58980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.767+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898037040 con 0x7f58a80d04c0 2024-05-09T03:19:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.767+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5898004250 con 0x7f58a80d04c0 2024-05-09T03:19:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.767+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5898003610 con 0x7f58a80d04c0 2024-05-09T03:19:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.767+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a81f8800 con 0x7f58a80d04c0 2024-05-09T03:19:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.768+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a81f4a40 con 0x7f58a80d04c0 2024-05-09T03:19:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.769+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a81f4550 con 0x7f58a80d04c0 2024-05-09T03:19:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.770+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f58980037b0 con 0x7f58a80d04c0 2024-05-09T03:19:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.770+0000 7f588ffff640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5894076350 0x7f5894078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.770+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 5 ==== osd_map(116..116 src has 1..116) ==== 8519+0+0 (secure 0 0 0) 0x7f58980b9570 con 0x7f58a80d04c0 2024-05-09T03:19:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.770+0000 7f58a67fc640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5894076350 0x7f5894078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.771+0000 7f58a67fc640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5894076350 0x7f5894078860 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f58a80d2c00 tx=0x7f589c009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:30.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:30.775+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5898081d20 con 0x7f58a80d04c0 2024-05-09T03:19:31.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.048+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f58a80d1f30 con 0x7f58a80d04c0 2024-05-09T03:19:31.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.634+0000 7f588ffff640 1 -- 172.21.15.46:0/4211969400 <== mon.2 v2:172.21.15.88:3301/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v285) ==== 107+0+0 (secure 0 0 0) 0x7f589808a3f0 con 0x7f58a80d04c0 2024-05-09T03:19:31.635 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5894076350 msgr2=0x7f5894078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5894076350 0x7f5894078860 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f58a80d2c00 tx=0x7f589c009110 comp rx=0 tx=0).stop 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58a80d04c0 msgr2=0x7f58a80cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58a80d04c0 0x7f58a80cd910 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5898031000 tx=0x7f58980046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 shutdown_connections 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58a8162880 0x7f58a81e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5894076350 0x7f5894078860 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a80d2410 0x7f58a81f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 --2- 172.21.15.46:0/4211969400 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f58a80d04c0 0x7f58a80cd910 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 >> 172.21.15.46:0/4211969400 conn(0x7f58a80b55f0 msgr2=0x7f58a815ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 shutdown_connections 2024-05-09T03:19:31.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.637+0000 7f58ada2b640 1 -- 172.21.15.46:0/4211969400 wait complete. 2024-05-09T03:19:31.651 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:31.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.709+0000 7f69cfb28f40 1 Processor -- start 2024-05-09T03:19:31.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.709+0000 7f69cfb28f40 1 -- start start 2024-05-09T03:19:31.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69cfb28f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361756b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69cfb28f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036247a40 0x55d03619e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69cfb28f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036248390 0x55d036245fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69cfb28f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d0361bd8d0 con 0x55d0361bc5f0 2024-05-09T03:19:31.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69cfb28f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d0361bd5f0 con 0x55d036247a40 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69cfb28f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d0361b8bb0 con 0x55d036248390 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c95f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c95f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361756b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48028/0 (socket says 172.21.15.46:48028) 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c8df1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036247a40 0x55d03619e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c95f2640 1 -- 172.21.15.46:0/2217300248 learned_addr learned my addr 172.21.15.46:0/2217300248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c9df3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036248390 0x55d036245fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c8df1640 1 -- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036248390 msgr2=0x55d036245fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c8df1640 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036248390 0x55d036245fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c8df1640 1 -- 172.21.15.46:0/2217300248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 msgr2=0x55d0361756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c8df1640 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c8df1640 1 -- 172.21.15.46:0/2217300248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0360dc350 con 0x55d036247a40 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.711+0000 7f69c95f2640 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:31.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.712+0000 7f69c8df1640 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036247a40 0x55d03619e640 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x55d0361bd240 tx=0x7f69c0005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d52315088ffd917 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.716+0000 7f69bb7fe640 1 -- 172.21.15.46:0/2217300248 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0019070 con 0x55d036247a40 2024-05-09T03:19:31.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.716+0000 7f69bb7fe640 1 -- 172.21.15.46:0/2217300248 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c0008890 con 0x55d036247a40 2024-05-09T03:19:31.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.716+0000 7f69bb7fe640 1 -- 172.21.15.46:0/2217300248 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0008c60 con 0x55d036247a40 2024-05-09T03:19:31.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 -- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036247a40 msgr2=0x55d03619e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036247a40 0x55d03619e640 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x55d0361bd240 tx=0x7f69c0005a00 comp rx=0 tx=0).stop 2024-05-09T03:19:31.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 -- 172.21.15.46:0/2217300248 shutdown_connections 2024-05-09T03:19:31.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036248390 0x55d036245fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036247a40 0x55d03619e640 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/2217300248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 -- 172.21.15.46:0/2217300248 >> 172.21.15.46:0/2217300248 conn(0x55d036096ac0 msgr2=0x55d036245570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:31.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 -- 172.21.15.46:0/2217300248 shutdown_connections 2024-05-09T03:19:31.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.717+0000 7f69cfb28f40 1 -- 172.21.15.46:0/2217300248 wait complete. 2024-05-09T03:19:31.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.718+0000 7f69cfb28f40 1 Processor -- start 2024-05-09T03:19:31.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.718+0000 7f69cfb28f40 1 -- start start 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69cfb28f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69cfb28f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036247a40 0x55d03624a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69cfb28f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036248390 0x55d03623c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69cfb28f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d03623d8e0 con 0x55d0361bc5f0 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69cfb28f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d036249db0 con 0x55d036248390 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69cfb28f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d036241eb0 con 0x55d036247a40 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69c95f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69c95f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361b7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48042/0 (socket says 172.21.15.46:48042) 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 learned_addr learned my addr 172.21.15.46:0/1582939968 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69c9df3640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036248390 0x55d03623c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.719+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036247a40 0x55d03624a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036247a40 msgr2=0x55d03624a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036247a40 0x55d03624a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036248390 msgr2=0x55d03623c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d036248390 0x55d03623c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d03623b8d0 con 0x55d0361bc5f0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d036247a40 0x55d03624a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361b7640 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009e70 tx=0x7f69bc009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69ba7fc640 1 -- 172.21.15.46:0/1582939968 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc01b070 con 0x55d0361bc5f0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69ba7fc640 1 -- 172.21.15.46:0/1582939968 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69bc016480 con 0x55d0361bc5f0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d03623bbb0 con 0x55d0361bc5f0 2024-05-09T03:19:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69ba7fc640 1 -- 172.21.15.46:0/1582939968 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc016830 con 0x55d0361bc5f0 2024-05-09T03:19:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.720+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d036241bf0 con 0x55d0361bc5f0 2024-05-09T03:19:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.722+0000 7f69ba7fc640 1 -- 172.21.15.46:0/1582939968 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f69bc01e070 con 0x55d0361bc5f0 2024-05-09T03:19:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.722+0000 7f69ba7fc640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69ac074cb0 0x7f69ac0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.722+0000 7f69ba7fc640 1 -- 172.21.15.46:0/1582939968 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8519+0+0 (secure 0 0 0) 0x7f69bc096570 con 0x55d0361bc5f0 2024-05-09T03:19:31.724 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:31.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.723+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d0362417b0 0x55d036239450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.723+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69ac074cb0 0x7f69ac0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.723+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d03624a6f0 con 0x55d0362417b0 2024-05-09T03:19:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.723+0000 7f69c9df3640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d0362417b0 0x55d036239450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.723+0000 7f69c9df3640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d0362417b0 0x55d036239450 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.723+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69ac074cb0 0x7f69ac0771c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x55d0361bd240 tx=0x7f69c0008000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.725+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v116'971 uv944 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.725+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d036239990 con 0x55d0362417b0 2024-05-09T03:19:31.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.727+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v116'972 uv972 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.727+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036239d60 con 0x55d0362417b0 2024-05-09T03:19:31.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.728+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v116'973 uv973 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.729 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:31.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.728+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d03623a100 con 0x55d0362417b0 2024-05-09T03:19:31.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.730+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v116'974 uv974 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.730+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ef190 con 0x55d0362417b0 2024-05-09T03:19:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.731+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v116'975 uv975 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.731+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee180 con 0x55d0362417b0 2024-05-09T03:19:31.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.733+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v116'976 uv976 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.733+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee570 con 0x55d0362417b0 2024-05-09T03:19:31.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.735+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v116'977 uv950 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.735+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362ee570 con 0x55d0362417b0 2024-05-09T03:19:31.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.736+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v116'978 uv978 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.736+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.737+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v116'979 uv979 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.738 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:31.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.737+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.739+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v116'980 uv980 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.739+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.740+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v116'981 uv981 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.740+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.742+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v116'982 uv982 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.742+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.743+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v116'983 uv970 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.743+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.745+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v116'984 uv984 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.745+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.746+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v116'985 uv985 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.747 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:31.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.746+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.747+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v116'986 uv986 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.747+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.749+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v116'987 uv987 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.749+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ee520 con 0x55d0362417b0 2024-05-09T03:19:31.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.750+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v116'988 uv988 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.750+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eed90 con 0x55d0362417b0 2024-05-09T03:19:31.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.752+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v116'989 uv989 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.752+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eed90 con 0x55d0362417b0 2024-05-09T03:19:31.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.753+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v116'990 uv963 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.753+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362eed90 con 0x55d0362417b0 2024-05-09T03:19:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.755+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v116'991 uv991 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.755+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eed90 con 0x55d0362417b0 2024-05-09T03:19:31.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.756+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v116'992 uv992 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.757 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:31.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.756+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eed90 con 0x55d0362417b0 2024-05-09T03:19:31.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.757+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v116'993 uv993 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.757+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eed90 con 0x55d0362417b0 2024-05-09T03:19:31.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.759+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v116'994 uv994 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.759+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.760+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v116'995 uv995 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.760+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.762+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v116'996 uv969 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.762+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.763+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v116'997 uv997 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.763+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.765+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v116'998 uv998 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.766 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:31.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.765+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.766+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v116'999 uv999 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.766+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.767+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v116'1000 uv1000 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.767+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362417b0 2024-05-09T03:19:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.769+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v116'1001 uv1001 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.769+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d0362efa40 0x55d0362f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.769+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362eec50 con 0x55d0362efa40 2024-05-09T03:19:31.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.769+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d0362efa40 0x55d0362f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.769+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d0362efa40 0x55d0362f1e60 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.771+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v116'699 uv679 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.771+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f23a0 con 0x55d0362efa40 2024-05-09T03:19:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.773+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v116'700 uv700 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.773+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f2740 con 0x55d0362efa40 2024-05-09T03:19:31.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.774+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v116'701 uv701 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.775 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.774+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f2f90 con 0x55d0362efa40 2024-05-09T03:19:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.775+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v116'702 uv702 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.775+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3330 con 0x55d0362efa40 2024-05-09T03:19:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.776+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v116'703 uv703 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.776+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f36d0 con 0x55d0362efa40 2024-05-09T03:19:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.778+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v116'704 uv698 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.778+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f3a70 con 0x55d0362efa40 2024-05-09T03:19:31.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.779+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v116'705 uv705 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.779+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.780+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v116'706 uv706 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.781 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:31.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.780+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.782+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v116'707 uv707 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.782+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.783+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v116'708 uv708 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.783+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.784+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v116'709 uv697 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.784+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.786+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v116'710 uv710 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.786+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.787+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v116'711 uv711 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.788 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:31.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.787+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f7870 con 0x55d0362efa40 2024-05-09T03:19:31.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.789+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v116'712 uv712 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.789+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f7870 con 0x55d0362efa40 2024-05-09T03:19:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.790+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v116'713 uv713 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.790+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.791+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v116'714 uv695 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.791+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.792+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v116'715 uv715 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.792+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.794+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v116'716 uv716 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.794 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.794+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f4e20 con 0x55d0362efa40 2024-05-09T03:19:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.795+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v116'717 uv717 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.795+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f4e20 con 0x55d0362efa40 2024-05-09T03:19:31.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.796+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v116'718 uv718 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.796+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362efa40 2024-05-09T03:19:31.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.797+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v116'719 uv719 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362efa40 2024-05-09T03:19:31.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.797+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362417b0 2024-05-09T03:19:31.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.799+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v116'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.799+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362417b0 2024-05-09T03:19:31.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.800+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v116'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.800+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362417b0 2024-05-09T03:19:31.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.801+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v116'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.801 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:31.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.801+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f4e20 con 0x55d0362417b0 2024-05-09T03:19:31.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.802+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v116'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.802+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f4e20 con 0x55d0362417b0 2024-05-09T03:19:31.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.803+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v116'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.803+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f3e10 con 0x55d0362417b0 2024-05-09T03:19:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.804+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v116'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.804+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d0362f4320 0x55d0362f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.804+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f6d40 con 0x55d0362f4320 2024-05-09T03:19:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.804+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d0362f4320 0x55d0362f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.805+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d0362f4320 0x55d0362f6740 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.806+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v116'142 uv141 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.806+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f7220 con 0x55d0362f4320 2024-05-09T03:19:31.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.807+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v116'143 uv143 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.807+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f7720 con 0x55d0362f4320 2024-05-09T03:19:31.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.808+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v116'144 uv144 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.809 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:31.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.808+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f2bc0 con 0x55d0362f4320 2024-05-09T03:19:31.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.809+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v116'145 uv145 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.809+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f8050 con 0x55d0362417b0 2024-05-09T03:19:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.810+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v116'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.811+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362f8050 con 0x55d0362417b0 2024-05-09T03:19:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.812+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v116'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.812+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f8050 con 0x55d0362417b0 2024-05-09T03:19:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.813+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v116'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.814 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:31.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.813+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f8050 con 0x55d0362417b0 2024-05-09T03:19:31.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.814+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v116'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.814+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f8050 con 0x55d0362417b0 2024-05-09T03:19:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.815+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v116'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.815+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362f8050 con 0x55d0362417b0 2024-05-09T03:19:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.817+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v116'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.817+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d0362f89b0 0x55d0362fadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.817+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb310 con 0x55d0362f89b0 2024-05-09T03:19:31.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.817+0000 7f69c9df3640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d0362f89b0 0x55d0362fadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.817+0000 7f69c9df3640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d0362f89b0 0x55d0362fadd0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.819+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v116'349 uv343 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.819+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362fb310 con 0x55d0362f89b0 2024-05-09T03:19:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.820+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v116'350 uv350 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.820+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.821+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v116'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.822 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.821+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fc710 con 0x55d0362f89b0 2024-05-09T03:19:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.822+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v116'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.822+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fc710 con 0x55d0362f89b0 2024-05-09T03:19:31.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.823+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v116'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.823+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.824+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v116'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.824+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.825+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v116'355 uv348 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.825+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.827+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v116'356 uv356 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.827+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.828+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v116'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.828 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:31.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.828+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.829+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v116'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.829+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fb700 con 0x55d0362f89b0 2024-05-09T03:19:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.830+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v116'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.830+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d0362fbd80 0x55d0362fe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.830+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362fe770 con 0x55d0362fbd80 2024-05-09T03:19:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.830+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d0362fbd80 0x55d0362fe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.831+0000 7f69c95f2640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d0362fbd80 0x55d0362fe1a0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.832+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v116'158 uv157 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362fbd80 2024-05-09T03:19:31.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.832+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362fbd80 2024-05-09T03:19:31.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.833+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v116'159 uv159 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362fbd80 2024-05-09T03:19:31.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.833+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362fbd80 2024-05-09T03:19:31.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.835+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v116'160 uv160 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362fbd80 2024-05-09T03:19:31.835 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:31.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.835+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362fbd80 2024-05-09T03:19:31.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.836+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v116'161 uv161 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69bc01d030 con 0x55d0362fbd80 2024-05-09T03:19:31.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.836+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362fbd80 2024-05-09T03:19:31.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.837+0000 7f69c95f2640 1 -- 172.21.15.46:0/1582939968 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v116'162 uv162 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69bc090040 con 0x55d0362fbd80 2024-05-09T03:19:31.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.837+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362417b0 2024-05-09T03:19:31.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.838+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v116'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.838+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362417b0 2024-05-09T03:19:31.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.839+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v116'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.839+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362417b0 2024-05-09T03:19:31.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.840+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v116'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.841 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:31.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.840+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ffb50 con 0x55d0362417b0 2024-05-09T03:19:31.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.841+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v116'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.841+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ffb50 con 0x55d0362417b0 2024-05-09T03:19:31.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.842+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v116'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.842+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362417b0 2024-05-09T03:19:31.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.843+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v116'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362417b0 2024-05-09T03:19:31.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.843+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362feb40 con 0x55d0362f4320 2024-05-09T03:19:31.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.845+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v116'314 uv308 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.845+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362fef50 con 0x55d0362f4320 2024-05-09T03:19:31.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.846+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v116'315 uv315 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.846+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ff340 con 0x55d0362f4320 2024-05-09T03:19:31.847 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:31.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.847+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v116'316 uv316 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.847+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ff730 con 0x55d0362f4320 2024-05-09T03:19:31.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.848+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v116'317 uv317 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.848+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ff730 con 0x55d0362f4320 2024-05-09T03:19:31.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.849+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v116'318 uv318 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.849+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ffb70 con 0x55d0362f4320 2024-05-09T03:19:31.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.850+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v116'319 uv313 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.850+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d0362ffb70 con 0x55d0362f4320 2024-05-09T03:19:31.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.851+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v116'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.851+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ff780 con 0x55d0362f4320 2024-05-09T03:19:31.853 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:31.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.852+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v116'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.853+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0362ff780 con 0x55d0362f4320 2024-05-09T03:19:31.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.854+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v116'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.854+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0363028f0 con 0x55d0362f4320 2024-05-09T03:19:31.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.855+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v116'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.855+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d0363004a0 0x55d0363028c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.855+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0363004a0 2024-05-09T03:19:31.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.855+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d0363004a0 0x55d0363028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.855+0000 7f69c8df1640 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d0363004a0 0x55d0363028c0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.857+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v116'137 uv135 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0363004a0 2024-05-09T03:19:31.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.857+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0363004a0 2024-05-09T03:19:31.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.858+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v116'138 uv138 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0363004a0 2024-05-09T03:19:31.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.858+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0363004a0 2024-05-09T03:19:31.859 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:31.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.859+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v116'139 uv139 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0363004a0 2024-05-09T03:19:31.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.859+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0363004a0 2024-05-09T03:19:31.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.860+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v116'140 uv140 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0363004a0 2024-05-09T03:19:31.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.860+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f4320 2024-05-09T03:19:31.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.861+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v116'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.861+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f4320 2024-05-09T03:19:31.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.862+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v116'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.862+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f4320 2024-05-09T03:19:31.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.863+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v116'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.864 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:31.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.863+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036303e10 con 0x55d0362f4320 2024-05-09T03:19:31.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.864+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v116'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.864+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036303e10 con 0x55d0362f4320 2024-05-09T03:19:31.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.865+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v116'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.865+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f4320 2024-05-09T03:19:31.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.866+0000 7f69c8df1640 1 -- 172.21.15.46:0/1582939968 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v116'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c0026040 con 0x55d0362f4320 2024-05-09T03:19:31.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.866+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.868+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v116'161 uv160 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.868+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.869+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v116'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.869+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.871 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.871+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v116'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.871+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.872+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v116'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.872+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.873+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v116'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.873+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.875+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v116'389 uv380 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.875+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.876+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v116'390 uv390 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.876+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.877+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v116'391 uv391 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.877 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:31.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.877+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.878+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v116'392 uv392 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.878+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.879+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v116'393 uv393 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.879+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.880+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v116'394 uv387 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.880+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.881+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v116'395 uv395 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.881+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.882+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v116'396 uv396 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.883 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:31.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.882+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036303e10 con 0x55d0362f89b0 2024-05-09T03:19:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.883+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v116'397 uv397 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.883+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036303e10 con 0x55d0362f89b0 2024-05-09T03:19:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.885+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v116'398 uv398 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.885+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d036302e00 con 0x55d0362f89b0 2024-05-09T03:19:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.886+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v116'399 uv399 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.886+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55d0363031f0 con 0x55d0362f89b0 2024-05-09T03:19:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v116'400 uv400 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f69c4051680 con 0x55d0362f89b0 2024-05-09T03:19:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d0362417b0 msgr2=0x55d036239450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d0362417b0 0x55d036239450 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d0362f4320 msgr2=0x55d0362f6740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d0362f4320 0x55d0362f6740 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d0363004a0 msgr2=0x55d0363028c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d0363004a0 0x55d0363028c0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d0362efa40 msgr2=0x55d0362f1e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d0362efa40 0x55d0362f1e60 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 reap_dead start 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d0362f89b0 msgr2=0x55d0362fadd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d0362f89b0 0x55d0362fadd0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69c9df3640 1 -- 172.21.15.46:0/1582939968 reap_dead start 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d0362fbd80 msgr2=0x55d0362fe1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d0362fbd80 0x55d0362fe1a0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69ac074cb0 msgr2=0x7f69ac0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69ac074cb0 0x7f69ac0771c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x55d0361bd240 tx=0x7f69c0008000 comp rx=0 tx=0).stop 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 msgr2=0x55d0361b7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361b7640 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009e70 tx=0x7f69bc009f60 comp rx=0 tx=0).stop 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 shutdown_connections 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d0362fbd80 0x55d0362fe1a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d0362f89b0 0x55d0362fadd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69ac074cb0 0x7f69ac0771c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 --2- 172.21.15.46:0/1582939968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d0361bc5f0 0x55d0361b7640 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.887+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 >> 172.21.15.46:0/1582939968 conn(0x55d036096ac0 msgr2=0x55d0361756b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.888+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 shutdown_connections 2024-05-09T03:19:31.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.888+0000 7f69cfb28f40 1 -- 172.21.15.46:0/1582939968 wait complete. 2024-05-09T03:19:31.892 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 100.00000000 stray0_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:19:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.948+0000 7f52e7292f40 1 Processor -- start 2024-05-09T03:19:31.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.948+0000 7f52e7292f40 1 -- start start 2024-05-09T03:19:31.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e7292f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15ea7f7f0 0x55f15eb3ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e7292f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 0x55f15eaac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e7292f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eaac980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e7292f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f15eac1cc0 con 0x55f15eaaedc0 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e7292f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f15ea28f50 con 0x55f15ea7f7f0 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e7292f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f15ea7fc00 con 0x55f15eb57790 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 0x55f15eaac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 0x55f15eaac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48054/0 (socket says 172.21.15.46:48054) 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 -- 172.21.15.46:0/3783427211 learned_addr learned my addr 172.21.15.46:0/3783427211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e0d5c640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15ea7f7f0 0x55f15eb3ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e155d640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eaac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 -- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 msgr2=0x55f15eaac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eaac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 -- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15ea7f7f0 msgr2=0x55f15eb3ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:31.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15ea7f7f0 0x55f15eb3ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 -- 172.21.15.46:0/3783427211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15eb599d0 con 0x55f15eaaedc0 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e0d5c640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15ea7f7f0 0x55f15eb3ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e155d640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eaac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52e055b640 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 0x55f15eaac440 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f52d400a770 tx=0x7f52d400a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138e58927e78e78f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52df559640 1 -- 172.21.15.46:0/3783427211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4019070 con 0x55f15eaaedc0 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52df559640 1 -- 172.21.15.46:0/3783427211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d4007600 con 0x55f15eaaedc0 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.950+0000 7f52df559640 1 -- 172.21.15.46:0/3783427211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4007a10 con 0x55f15eaaedc0 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 -- 172.21.15.46:0/3783427211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 msgr2=0x55f15eaac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 0x55f15eaac440 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f52d400a770 tx=0x7f52d400a860 comp rx=0 tx=0).stop 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 -- 172.21.15.46:0/3783427211 shutdown_connections 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eaac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15eaaedc0 0x55f15eaac440 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 --2- 172.21.15.46:0/3783427211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15ea7f7f0 0x55f15eb3ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 -- 172.21.15.46:0/3783427211 >> 172.21.15.46:0/3783427211 conn(0x55f15e9a4ac0 msgr2=0x55f15eaad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:31.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 -- 172.21.15.46:0/3783427211 shutdown_connections 2024-05-09T03:19:31.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.951+0000 7f52e7292f40 1 -- 172.21.15.46:0/3783427211 wait complete. 2024-05-09T03:19:31.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.952+0000 7f52e7292f40 1 Processor -- start 2024-05-09T03:19:31.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.952+0000 7f52e7292f40 1 -- start start 2024-05-09T03:19:31.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e7292f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15ea7f7f0 0x55f15eb39bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e7292f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15eaaedc0 0x55f15eb3cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e7292f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eac3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e7292f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f15eb55c40 con 0x55f15ea7f7f0 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e7292f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f15eb3dad0 con 0x55f15eaaedc0 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e7292f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f15eb38b30 con 0x55f15eb57790 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e0d5c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15ea7f7f0 0x55f15eb39bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e0d5c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15ea7f7f0 0x55f15eb39bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48068/0 (socket says 172.21.15.46:48068) 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.953+0000 7f52e0d5c640 1 -- 172.21.15.46:0/1030520941 learned_addr learned my addr 172.21.15.46:0/1030520941 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e055b640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15eaaedc0 0x55f15eb3cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e155d640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eac3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e0d5c640 1 -- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 msgr2=0x55f15eac3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e0d5c640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f15eb57790 0x55f15eac3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e0d5c640 1 -- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15eaaedc0 msgr2=0x55f15eb3cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e0d5c640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15eaaedc0 0x55f15eb3cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e0d5c640 1 -- 172.21.15.46:0/1030520941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15eb3a130 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e055b640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f15eaaedc0 0x55f15eb3cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e0d5c640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15ea7f7f0 0x55f15eb39bf0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f52d0054040 tx=0x7f52d0003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52de557640 1 -- 172.21.15.46:0/1030520941 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0055040 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52de557640 1 -- 172.21.15.46:0/1030520941 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d0066680 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f15eb38d80 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52de557640 1 -- 172.21.15.46:0/1030520941 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0066a30 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.954+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15eac34d0 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.956+0000 7f52de557640 1 -- 172.21.15.46:0/1030520941 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f52d0058020 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.956+0000 7f52de557640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc074ce0 0x7f52cc0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.956+0000 7f52de557640 1 -- 172.21.15.46:0/1030520941 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8519+0+0 (secure 0 0 0) 0x7f52d00daf10 con 0x55f15ea7f7f0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.957+0000 7f52e055b640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc074ce0 0x7f52cc0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.957+0000 7f52e7292f40 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f15ebf73b0 0x55f15ebf97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.957+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=103b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55f15ebf9d10 con 0x55f15ebf73b0 2024-05-09T03:19:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.957+0000 7f52e155d640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f15ebf73b0 0x55f15ebf97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.957+0000 7f52e055b640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc074ce0 0x7f52cc0771f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x55f15eb3ac20 tx=0x7f52d400c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.957+0000 7f52e155d640 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f15ebf73b0 0x55f15ebf97d0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e155d640 1 -- 172.21.15.46:0/1030520941 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 100.00000000 [omap-set-vals] v116'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f52d80103b0 con 0x55f15ebf73b0 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f15ebf73b0 msgr2=0x55f15ebf97d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55f15ebf73b0 0x55f15ebf97d0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc074ce0 msgr2=0x7f52cc0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc074ce0 0x7f52cc0771f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x55f15eb3ac20 tx=0x7f52d400c000 comp rx=0 tx=0).stop 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15ea7f7f0 msgr2=0x55f15eb39bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 --2- 172.21.15.46:0/1030520941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f15ea7f7f0 0x55f15eb39bf0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f52d0054040 tx=0x7f52d0003330 comp rx=0 tx=0).stop 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e155d640 1 -- 172.21.15.46:0/1030520941 reap_dead start 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 shutdown_connections 2024-05-09T03:19:31.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 >> 172.21.15.46:0/1030520941 conn(0x55f15e9a4ac0 msgr2=0x55f15eab1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:31.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 shutdown_connections 2024-05-09T03:19:31.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:31.959+0000 7f52e7292f40 1 -- 172.21.15.46:0/1030520941 wait complete. 2024-05-09T03:19:31.963 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.080+0000 7f9271f9f640 1 Processor -- start 2024-05-09T03:19:32.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.080+0000 7f9271f9f640 1 -- start start 2024-05-09T03:19:32.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f9271f9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c15dff0 0x7f926c161250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f9271f9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f926c0d4940 con 0x7f926c0d3060 2024-05-09T03:19:32.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f9271f9f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f926c0d4c20 con 0x7f926c0d5700 2024-05-09T03:19:32.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f9271f9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f926c0d4f00 con 0x7f926c15dff0 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f926bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f926c0d3060 0x7f926c160b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35924/0 (socket says 172.21.15.46:35924) 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f926bfff640 1 -- 172.21.15.46:0/3044640532 learned_addr learned my addr 172.21.15.46:0/3044640532 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f927179e640 1 --2- 172.21.15.46:0/3044640532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c15dff0 0x7f926c161250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810161417 0 0) 0x7f926c0d4940 con 0x7f926c0d3060 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9254003520 con 0x7f926c0d3060 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2954573887 0 0) 0x7f926c0d4c20 con 0x7f926c0d5700 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.081+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f926c0d4940 con 0x7f926c0d5700 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.082+0000 7f927179e640 1 -- 172.21.15.46:0/3044640532 >> v1:172.21.15.88:6789/0 conn(0x7f926c0d5700 legacy=0x7f926c0d2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.082+0000 7f927179e640 1 -- 172.21.15.46:0/3044640532 >> v1:172.21.15.46:6789/0 conn(0x7f926c0d3060 legacy=0x7f926c160b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.082+0000 7f927179e640 1 -- 172.21.15.46:0/3044640532 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c0d51e0 con 0x7f926c15dff0 2024-05-09T03:19:32.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.082+0000 7f927179e640 1 --2- 172.21.15.46:0/3044640532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c15dff0 0x7f926c161250 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f92640097d0 tx=0x7f926402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a55d201b745c1143 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264037070 con 0x7f926c15dff0 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9264032440 con 0x7f926c15dff0 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f926b7fe640 1 -- 172.21.15.46:0/3044640532 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92640327f0 con 0x7f926c15dff0 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 -- 172.21.15.46:0/3044640532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c15dff0 msgr2=0x7f926c161250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 --2- 172.21.15.46:0/3044640532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c15dff0 0x7f926c161250 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f92640097d0 tx=0x7f926402a210 comp rx=0 tx=0).stop 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 -- 172.21.15.46:0/3044640532 shutdown_connections 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 --2- 172.21.15.46:0/3044640532 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c15dff0 0x7f926c161250 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 -- 172.21.15.46:0/3044640532 >> 172.21.15.46:0/3044640532 conn(0x7f926c06e110 msgr2=0x7f926c06e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 -- 172.21.15.46:0/3044640532 shutdown_connections 2024-05-09T03:19:32.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.083+0000 7f9271f9f640 1 -- 172.21.15.46:0/3044640532 wait complete. 2024-05-09T03:19:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.085+0000 7f9271f9f640 1 Processor -- start 2024-05-09T03:19:32.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.085+0000 7f9271f9f640 1 -- start start 2024-05-09T03:19:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 0x7f926c15eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f926c15dff0 0x7f926c0c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c153e30 0x7f926c154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f926c16b1f0 con 0x7f926c0d5700 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f926c16b070 con 0x7f926c15dff0 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f926c16b370 con 0x7f926c153e30 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 0x7f926c15eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 0x7f926c15eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48072/0 (socket says 172.21.15.46:48072) 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 -- 172.21.15.46:0/1800147140 learned_addr learned my addr 172.21.15.46:0/1800147140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f927179e640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c153e30 0x7f926c154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f926bfff640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f926c15dff0 0x7f926c0c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 -- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c153e30 msgr2=0x7f926c154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c153e30 0x7f926c154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 -- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f926c15dff0 msgr2=0x7f926c0c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f926c15dff0 0x7f926c0c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 -- 172.21.15.46:0/1800147140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c154780 con 0x7f926c0d5700 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9270f9d640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 0x7f926c15eb70 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c3c0 tx=0x7f925c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c017070 con 0x7f926c0d5700 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925c0052d0 con 0x7f926c0d5700 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c14af20 con 0x7f926c0d5700 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.087+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c003830 con 0x7f926c0d5700 2024-05-09T03:19:32.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.088+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c14b4e0 con 0x7f926c0d5700 2024-05-09T03:19:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.088+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926c076c00 con 0x7f926c0d5700 2024-05-09T03:19:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.089+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f925c01a070 con 0x7f926c0d5700 2024-05-09T03:19:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.090+0000 7f92697fa640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9258076300 0x7f9258078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.090+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8519+0+0 (secure 0 0 0) 0x7f925c0973c0 con 0x7f926c0d5700 2024-05-09T03:19:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.090+0000 7f926bfff640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9258076300 0x7f9258078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.091+0000 7f926bfff640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9258076300 0x7f9258078810 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9260000f80 tx=0x7f9260009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:32.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.094+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f926c154780 con 0x7f926c0d5700 2024-05-09T03:19:32.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.325+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f926c0d67c0 con 0x7f926c0d5700 2024-05-09T03:19:32.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.718+0000 7f92697fa640 1 -- 172.21.15.46:0/1800147140 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v286) ==== 162+0+0 (secure 0 0 0) 0x7f925c0681c0 con 0x7f926c0d5700 2024-05-09T03:19:32.719 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9258076300 msgr2=0x7f9258078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9258076300 0x7f9258078810 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9260000f80 tx=0x7f9260009040 comp rx=0 tx=0).stop 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 msgr2=0x7f926c15eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 0x7f926c15eb70 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f925c00c3c0 tx=0x7f925c00c880 comp rx=0 tx=0).stop 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 shutdown_connections 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f926c153e30 0x7f926c154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f926c15dff0 0x7f926c0c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9258076300 0x7f9258078810 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 --2- 172.21.15.46:0/1800147140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f926c0d5700 0x7f926c15eb70 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 >> 172.21.15.46:0/1800147140 conn(0x7f926c06e110 msgr2=0x7f926c0c8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 shutdown_connections 2024-05-09T03:19:32.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.721+0000 7f9271f9f640 1 -- 172.21.15.46:0/1800147140 wait complete. 2024-05-09T03:19:32.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.850+0000 7f4f2d39d640 1 Processor -- start 2024-05-09T03:19:32.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.850+0000 7f4f2d39d640 1 -- start start 2024-05-09T03:19:32.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f2d39d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 0x7f4f280b7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f2d39d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f280d4b70 con 0x7f4f280d14b0 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f2d39d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f280d4e50 con 0x7f4f280b4e60 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f2d39d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f28163e30 con 0x7f4f28160450 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f27fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f280d14b0 0x7f4f280b9140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35938/0 (socket says 172.21.15.46:35938) 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f27fff640 1 -- 172.21.15.46:0/3681623915 learned_addr learned my addr 172.21.15.46:0/3681623915 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f277fe640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 0x7f4f280b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 98511298 0 0) 0x7f4f280d4b70 con 0x7f4f280d14b0 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f10003520 con 0x7f4f280d14b0 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 239232722 0 0) 0x7f4f10003520 con 0x7f4f280d14b0 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 msgr2=0x7f4f280b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 0x7f4f280b7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 >> v1:172.21.15.88:6789/0 conn(0x7f4f280b4e60 legacy=0x7f4f281642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f280d5130 con 0x7f4f280d14b0 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f4f180016f0 con 0x7f4f280d14b0 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.852+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 >> v1:172.21.15.46:6789/0 conn(0x7f4f280d14b0 legacy=0x7f4f280b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f26ffd640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f10003aa0 0x7f4f10003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f26ffd640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 0x7f4f10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f26ffd640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f10004d90 0x7f4f10005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f28163e30 con 0x7f4f10003aa0 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f1000bbd0 con 0x7f4f10004d90 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1000bdd0 con 0x7f4f28160450 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f27fff640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f10003aa0 0x7f4f10003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f277fe640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 0x7f4f10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f2cb9c640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f10004d90 0x7f4f10005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f27fff640 1 -- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 msgr2=0x7f4f10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f27fff640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f28160450 0x7f4f10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f27fff640 1 -- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f10004d90 msgr2=0x7f4f10005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:32.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f27fff640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f10004d90 0x7f4f10005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.853+0000 7f4f27fff640 1 -- 172.21.15.46:0/3681623915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f10003520 con 0x7f4f10003aa0 2024-05-09T03:19:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.854+0000 7f4f27fff640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f10003aa0 0x7f4f10003ed0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f4f28074d40 tx=0x7f4f10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d153eec27ad6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.854+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18007d10 con 0x7f4f10003aa0 2024-05-09T03:19:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.854+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f18005ad0 con 0x7f4f10003aa0 2024-05-09T03:19:32.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.854+0000 7f4f26ffd640 1 -- 172.21.15.46:0/3681623915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18007070 con 0x7f4f10003aa0 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2d39d640 1 -- 172.21.15.46:0/3681623915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f10003aa0 msgr2=0x7f4f10003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/3681623915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f10003aa0 0x7f4f10003ed0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f4f28074d40 tx=0x7f4f10005830 comp rx=0 tx=0).stop 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2cb9c640 1 -- 172.21.15.46:0/3681623915 reap_dead start 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2d39d640 1 -- 172.21.15.46:0/3681623915 shutdown_connections 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2d39d640 1 -- 172.21.15.46:0/3681623915 >> 172.21.15.46:0/3681623915 conn(0x7f4f280d2670 msgr2=0x7f4f280d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2d39d640 1 -- 172.21.15.46:0/3681623915 shutdown_connections 2024-05-09T03:19:32.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.855+0000 7f4f2d39d640 1 -- 172.21.15.46:0/3681623915 wait complete. 2024-05-09T03:19:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.856+0000 7f4f2d39d640 1 Processor -- start 2024-05-09T03:19:32.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.857+0000 7f4f2d39d640 1 -- start start 2024-05-09T03:19:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2d39d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10003aa0 0x7f4f280ac600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2d39d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f280b41d0 0x7f4f280b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2d39d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 0x7f4f281f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2d39d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280d4700 con 0x7f4f281f5000 2024-05-09T03:19:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2d39d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f280b9850 con 0x7f4f280b41d0 2024-05-09T03:19:32.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2d39d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f280d4880 con 0x7f4f10003aa0 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 0x7f4f281f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 0x7f4f281f5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48100/0 (socket says 172.21.15.46:48100) 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 -- 172.21.15.46:0/2588313444 learned_addr learned my addr 172.21.15.46:0/2588313444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f277fe640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f280b41d0 0x7f4f280b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f27fff640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10003aa0 0x7f4f280ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 -- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10003aa0 msgr2=0x7f4f280ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10003aa0 0x7f4f280ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 -- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f280b41d0 msgr2=0x7f4f280b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f280b41d0 0x7f4f280b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 -- 172.21.15.46:0/2588313444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f28162350 con 0x7f4f281f5000 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f27fff640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10003aa0 0x7f4f280ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f277fe640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f280b41d0 0x7f4f280b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.858+0000 7f4f2cb9c640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 0x7f4f281f5410 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f4f280d3670 tx=0x7f4f2000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:32.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.859+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20003b80 con 0x7f4f281f5000 2024-05-09T03:19:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.859+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f20018070 con 0x7f4f281f5000 2024-05-09T03:19:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.859+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000d770 con 0x7f4f281f5000 2024-05-09T03:19:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.859+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f281625f0 con 0x7f4f281f5000 2024-05-09T03:19:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.859+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f28162bb0 con 0x7f4f281f5000 2024-05-09T03:19:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.860+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f28076c70 con 0x7f4f281f5000 2024-05-09T03:19:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.860+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4f2000d910 con 0x7f4f281f5000 2024-05-09T03:19:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.861+0000 7f4f24ff9640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f14076300 0x7f4f14078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.861+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8519+0+0 (secure 0 0 0) 0x7f4f200980d0 con 0x7f4f281f5000 2024-05-09T03:19:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.861+0000 7f4f27fff640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f14076300 0x7f4f14078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:32.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.862+0000 7f4f27fff640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f14076300 0x7f4f14078810 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4f18032000 tx=0x7f4f18009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:32.866+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4f200608a0 con 0x7f4f281f5000 2024-05-09T03:19:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.095+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f28076490 con 0x7f4f281f5000 2024-05-09T03:19:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.095+0000 7f4f24ff9640 1 -- 172.21.15.46:0/2588313444 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 287 v287) ==== 95+0+5178 (secure 0 0 0) 0x7f4f20064ba0 con 0x7f4f281f5000 2024-05-09T03:19:33.096 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:33.096 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":287,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":287,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:32.721616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9834},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9834":{"gid":9834,"name":"c","rank":0,"incarnation":287,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/3872337434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3872337434},{"type":"v1","addr":"172.21.15.46:6834","nonce":3872337434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 287 2024-05-09T03:19:33.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.097+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f14076300 msgr2=0x7f4f14078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.097+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f14076300 0x7f4f14078810 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4f18032000 tx=0x7f4f18009030 comp rx=0 tx=0).stop 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.097+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 msgr2=0x7f4f281f5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.097+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 0x7f4f281f5410 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f4f280d3670 tx=0x7f4f2000ff70 comp rx=0 tx=0).stop 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 shutdown_connections 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f14076300 0x7f4f14078810 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f281f5000 0x7f4f281f5410 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f280b41d0 0x7f4f280b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 --2- 172.21.15.46:0/2588313444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f10003aa0 0x7f4f280ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 >> 172.21.15.46:0/2588313444 conn(0x7f4f280d2670 msgr2=0x7f4f280d5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 shutdown_connections 2024-05-09T03:19:33.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:33.098+0000 7f4f2d39d640 1 -- 172.21.15.46:0/2588313444 wait complete. 2024-05-09T03:19:33.110 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:19:35.737 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:35.737+0000 7ff28e1c4640 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x100is fatal because it is a system directory for this rank 2024-05-09T03:19:35.739 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:35.738+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:35.739 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:35.738+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:19:35.746 INFO:tasks.ceph.mds.c.smithi046.stderr: -17> 2024-05-09T03:19:35.737+0000 7ff28e1c4640 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x100is fatal because it is a system directory for this rank 2024-05-09T03:19:35.808 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:37.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:19:37.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:19:38.111 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.240+0000 7f5b04449640 1 Processor -- start 2024-05-09T03:19:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.240+0000 7f5b04449640 1 -- start start 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.241+0000 7f5b04449640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5afc164c70 0x7f5afc1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.241+0000 7f5b04449640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5afc0d2480 0x7f5afc0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.241+0000 7f5b04449640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5afc160b20 con 0x7f5afc15dfe0 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.241+0000 7f5b04449640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5afc161150 con 0x7f5afc164c70 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.241+0000 7f5b04449640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5afc0d15f0 con 0x7f5afc0d2480 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5afc164c70 0x7f5afc1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5afc164c70 0x7f5afc1615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43882/0 (socket says 172.21.15.46:43882) 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 -- 172.21.15.46:0/2144554628 learned_addr learned my addr 172.21.15.46:0/2144554628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 -- 172.21.15.46:0/2144554628 >> v2:172.21.15.88:3300/0 conn(0x7f5afc0d2480 msgr2=0x7f5afc0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b029e6640 1 --2- 172.21.15.46:0/2144554628 >> v2:172.21.15.88:3300/0 conn(0x7f5afc0d2480 0x7f5afc0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 --2- 172.21.15.46:0/2144554628 >> v2:172.21.15.88:3300/0 conn(0x7f5afc0d2480 0x7f5afc0d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 -- 172.21.15.46:0/2144554628 >> v1:172.21.15.88:6789/0 conn(0x7f5afc15dfe0 legacy=0x7f5afc0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:38.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b021e5640 1 -- 172.21.15.46:0/2144554628 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc0d18d0 con 0x7f5afc164c70 2024-05-09T03:19:38.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.242+0000 7f5b029e6640 1 --2- 172.21.15.46:0/2144554628 >> v2:172.21.15.88:3300/0 conn(0x7f5afc0d2480 0x7f5afc0d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.243+0000 7f5b021e5640 1 --2- 172.21.15.46:0/2144554628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc1615e0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0d3e30 tx=0x7f5af802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af61b6b532c03068 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.243+0000 7f5b011e3640 1 -- 172.21.15.46:0/2144554628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8036070 con 0x7f5afc164c70 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.243+0000 7f5b011e3640 1 -- 172.21.15.46:0/2144554628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af8031600 con 0x7f5afc164c70 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.243+0000 7f5b011e3640 1 -- 172.21.15.46:0/2144554628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af80319d0 con 0x7f5afc164c70 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 -- 172.21.15.46:0/2144554628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 msgr2=0x7f5afc1615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 --2- 172.21.15.46:0/2144554628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc1615e0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0d3e30 tx=0x7f5af802a610 comp rx=0 tx=0).stop 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 -- 172.21.15.46:0/2144554628 shutdown_connections 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 --2- 172.21.15.46:0/2144554628 >> v2:172.21.15.88:3300/0 conn(0x7f5afc0d2480 0x7f5afc0d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 --2- 172.21.15.46:0/2144554628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc1615e0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 -- 172.21.15.46:0/2144554628 >> 172.21.15.46:0/2144554628 conn(0x7f5afc073db0 msgr2=0x7f5afc0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 -- 172.21.15.46:0/2144554628 shutdown_connections 2024-05-09T03:19:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.244+0000 7f5b04449640 1 -- 172.21.15.46:0/2144554628 wait complete. 2024-05-09T03:19:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.246+0000 7f5b04449640 1 Processor -- start 2024-05-09T03:19:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.246+0000 7f5b04449640 1 -- start start 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b04449640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5afc0d2480 0x7f5afc0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b04449640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5afc15dfe0 0x7f5afc0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b04449640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b04449640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc0d2dd0 con 0x7f5afc164c70 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b04449640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5afc15e3f0 con 0x7f5afc0d2480 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b04449640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5afc0d45c0 con 0x7f5afc15dfe0 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc0ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43890/0 (socket says 172.21.15.46:43890) 2024-05-09T03:19:38.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 -- 172.21.15.46:0/2932501396 learned_addr learned my addr 172.21.15.46:0/2932501396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b021e5640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5afc0d2480 0x7f5afc0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b019e4640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5afc15dfe0 0x7f5afc0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 -- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5afc15dfe0 msgr2=0x7f5afc0cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5afc15dfe0 0x7f5afc0cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 -- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5afc0d2480 msgr2=0x7f5afc0ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5afc0d2480 0x7f5afc0ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.247+0000 7f5b029e6640 1 -- 172.21.15.46:0/2932501396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc1fa0f0 con 0x7f5afc164c70 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.248+0000 7f5b029e6640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc0ce800 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f5af400c5e0 tx=0x7f5af400caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.248+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4015070 con 0x7f5afc164c70 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.248+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af4004240 con 0x7f5afc164c70 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.248+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4003610 con 0x7f5afc164c70 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.248+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5afc1fa3d0 con 0x7f5afc164c70 2024-05-09T03:19:38.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.248+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5afc1fa990 con 0x7f5afc164c70 2024-05-09T03:19:38.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.249+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5afc08cfd0 con 0x7f5afc164c70 2024-05-09T03:19:38.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.250+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5af40037b0 con 0x7f5afc164c70 2024-05-09T03:19:38.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.250+0000 7f5af2ffd640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae00762d0 0x7f5ae00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.250+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f5af40982f0 con 0x7f5afc164c70 2024-05-09T03:19:38.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.251+0000 7f5b021e5640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae00762d0 0x7f5ae00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.251+0000 7f5b021e5640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae00762d0 0x7f5ae00787e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5af8033000 tx=0x7f5af8009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.255+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5afc08cfd0 con 0x7f5afc164c70 2024-05-09T03:19:38.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.488+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5afc06a270 con 0x7f5afc164c70 2024-05-09T03:19:38.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.488+0000 7f5af2ffd640 1 -- 172.21.15.46:0/2932501396 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) ==== 95+0+5169 (secure 0 0 0) 0x7f5af40609d0 con 0x7f5afc164c70 2024-05-09T03:19:38.489 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:38.489 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":292,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9879,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2209678569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2209678569},{"type":"v1","addr":"172.21.15.46:6834","nonce":2209678569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":292}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:35.739265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:38.490 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 292 2024-05-09T03:19:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.490+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae00762d0 msgr2=0x7f5ae00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.490+0000 7f5b04449640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae00762d0 0x7f5ae00787e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5af8033000 tx=0x7f5af8009000 comp rx=0 tx=0).stop 2024-05-09T03:19:38.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.490+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 msgr2=0x7f5afc0ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.490+0000 7f5b04449640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc0ce800 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f5af400c5e0 tx=0x7f5af400caa0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.490+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 shutdown_connections 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.490+0000 7f5b04449640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5ae00762d0 0x7f5ae00787e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.491+0000 7f5b04449640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5afc164c70 0x7f5afc0ce800 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.491+0000 7f5b04449640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5afc15dfe0 0x7f5afc0cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.491+0000 7f5b04449640 1 --2- 172.21.15.46:0/2932501396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5afc0d2480 0x7f5afc0ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.491+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 >> 172.21.15.46:0/2932501396 conn(0x7f5afc073db0 msgr2=0x7f5afc15e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.491+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 shutdown_connections 2024-05-09T03:19:38.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.491+0000 7f5b04449640 1 -- 172.21.15.46:0/2932501396 wait complete. 2024-05-09T03:19:38.502 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:19:38.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.616+0000 7f69be18d640 1 Processor -- start 2024-05-09T03:19:38.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.616+0000 7f69be18d640 1 -- start start 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69be18d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69b80b84b0 0x7f69b80d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69be18d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69b80d1700 0x7f69b809b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69be18d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b809bd10 0x7f69b815fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69be18d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f69b8072710 con 0x7f69b80b84b0 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69be18d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f69b809c120 con 0x7f69b80d1700 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69be18d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f69b80cd4a0 con 0x7f69b809bd10 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69b80b84b0 0x7f69b80d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69b80b84b0 0x7f69b80d11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43894/0 (socket says 172.21.15.46:43894) 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 -- 172.21.15.46:0/3365413462 learned_addr learned my addr 172.21.15.46:0/3365413462 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b6ffd640 1 --2- 172.21.15.46:0/3365413462 >> v2:172.21.15.88:3300/0 conn(0x7f69b80d1700 0x7f69b809b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 -- 172.21.15.46:0/3365413462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b809bd10 msgr2=0x7f69b815fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:38.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 --2- 172.21.15.46:0/3365413462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b809bd10 0x7f69b815fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 -- 172.21.15.46:0/3365413462 >> v2:172.21.15.88:3300/0 conn(0x7f69b80d1700 msgr2=0x7f69b809b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 --2- 172.21.15.46:0/3365413462 >> v2:172.21.15.88:3300/0 conn(0x7f69b80d1700 0x7f69b809b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b77fe640 1 -- 172.21.15.46:0/3365413462 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b8071b90 con 0x7f69b80b84b0 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.618+0000 7f69b6ffd640 1 --2- 172.21.15.46:0/3365413462 >> v2:172.21.15.88:3300/0 conn(0x7f69b80d1700 0x7f69b809b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.619+0000 7f69b77fe640 1 --2- 172.21.15.46:0/3365413462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80d11c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f69b80b26a0 tx=0x7f69a802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3c3021560a702b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.619+0000 7f69b67fc640 1 -- 172.21.15.46:0/3365413462 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8004030 con 0x7f69b80b84b0 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.619+0000 7f69b67fc640 1 -- 172.21.15.46:0/3365413462 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a80041d0 con 0x7f69b80b84b0 2024-05-09T03:19:38.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.619+0000 7f69b67fc640 1 -- 172.21.15.46:0/3365413462 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8003610 con 0x7f69b80b84b0 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 -- 172.21.15.46:0/3365413462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 msgr2=0x7f69b80d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 --2- 172.21.15.46:0/3365413462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80d11c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f69b80b26a0 tx=0x7f69a802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 -- 172.21.15.46:0/3365413462 shutdown_connections 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 --2- 172.21.15.46:0/3365413462 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b809bd10 0x7f69b815fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 --2- 172.21.15.46:0/3365413462 >> v2:172.21.15.88:3300/0 conn(0x7f69b80d1700 0x7f69b809b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 --2- 172.21.15.46:0/3365413462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80d11c0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 -- 172.21.15.46:0/3365413462 >> 172.21.15.46:0/3365413462 conn(0x7f69b80d2c70 msgr2=0x7f69b80d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:38.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 -- 172.21.15.46:0/3365413462 shutdown_connections 2024-05-09T03:19:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.620+0000 7f69be18d640 1 -- 172.21.15.46:0/3365413462 wait complete. 2024-05-09T03:19:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.622+0000 7f69be18d640 1 Processor -- start 2024-05-09T03:19:38.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.622+0000 7f69be18d640 1 -- start start 2024-05-09T03:19:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69be18d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69b809bd10 0x7f69b80ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69be18d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69be18d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b80d1700 0x7f69b80b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69be18d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69b8072430 con 0x7f69b80b84b0 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69be18d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f69b8072150 con 0x7f69b809bd10 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69be18d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f69b8070a10 con 0x7f69b80d1700 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69b6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69b6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43910/0 (socket says 172.21.15.46:43910) 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.623+0000 7f69b6ffd640 1 -- 172.21.15.46:0/1638103907 learned_addr learned my addr 172.21.15.46:0/1638103907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b77fe640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69b809bd10 0x7f69b80ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b7fff640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b80d1700 0x7f69b80b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b6ffd640 1 -- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b80d1700 msgr2=0x7f69b80b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b6ffd640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b80d1700 0x7f69b80b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b6ffd640 1 -- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69b809bd10 msgr2=0x7f69b80ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b6ffd640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69b809bd10 0x7f69b80ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b6ffd640 1 -- 172.21.15.46:0/1638103907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b80cab60 con 0x7f69b80b84b0 2024-05-09T03:19:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69b6ffd640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80c90d0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c7f0 tx=0x7f69ac00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac015070 con 0x7f69b80b84b0 2024-05-09T03:19:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac007820 con 0x7f69b80b84b0 2024-05-09T03:19:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b80b3220 con 0x7f69b80b84b0 2024-05-09T03:19:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac007bd0 con 0x7f69b80b84b0 2024-05-09T03:19:38.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.624+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b80b37e0 con 0x7f69b80b84b0 2024-05-09T03:19:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.625+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b806a270 con 0x7f69b80b84b0 2024-05-09T03:19:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.626+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f69ac018070 con 0x7f69b80b84b0 2024-05-09T03:19:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.626+0000 7f699ffff640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69a40762d0 0x7f69a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.627+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f69ac098210 con 0x7f69b80b84b0 2024-05-09T03:19:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.627+0000 7f69b77fe640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69a40762d0 0x7f69a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.627+0000 7f69b77fe640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69a40762d0 0x7f69a40787e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f69a8033000 tx=0x7f69a802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.631+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f69ac060920 con 0x7f69b80b84b0 2024-05-09T03:19:38.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.864+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69b80b77b0 con 0x7f69b80b84b0 2024-05-09T03:19:38.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.864+0000 7f699ffff640 1 -- 172.21.15.46:0/1638103907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) ==== 95+0+5169 (secure 0 0 0) 0x7f69ac064c20 con 0x7f69b80b84b0 2024-05-09T03:19:38.865 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:38.865 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":292,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9879,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2209678569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2209678569},{"type":"v1","addr":"172.21.15.46:6834","nonce":2209678569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":292}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:35.739265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:38.865 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 292 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69a40762d0 msgr2=0x7f69a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69a40762d0 0x7f69a40787e0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f69a8033000 tx=0x7f69a802e000 comp rx=0 tx=0).stop 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 msgr2=0x7f69b80c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80c90d0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c7f0 tx=0x7f69ac00ccb0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 shutdown_connections 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69b80d1700 0x7f69b80b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f69a40762d0 0x7f69a40787e0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69b80b84b0 0x7f69b80c90d0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 --2- 172.21.15.46:0/1638103907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69b809bd10 0x7f69b80ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 >> 172.21.15.46:0/1638103907 conn(0x7f69b80d2c70 msgr2=0x7f69b815f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 shutdown_connections 2024-05-09T03:19:38.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.866+0000 7f69be18d640 1 -- 172.21.15.46:0/1638103907 wait complete. 2024-05-09T03:19:38.878 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:19:38.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:38.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.992+0000 7f74a74a8640 1 Processor -- start 2024-05-09T03:19:38.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.992+0000 7f74a74a8640 1 -- start start 2024-05-09T03:19:38.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.993+0000 7f74a74a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74a00d2b90 0x7f74a015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:38.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.993+0000 7f74a74a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a0160b20 con 0x7f74a01615e0 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.993+0000 7f74a74a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74a00d15f0 con 0x7f74a0164c70 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.993+0000 7f74a74a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f74a015e520 con 0x7f74a00d2b90 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.994+0000 7f74a5a45640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74a00d2b90 0x7f74a015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.994+0000 7f74a4a43640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f74a01615e0 0x7f74a00d2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35094/0 (socket says 172.21.15.46:35094) 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.994+0000 7f74a4a43640 1 -- 172.21.15.46:0/71630605 learned_addr learned my addr 172.21.15.46:0/71630605 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.994+0000 7f74a5a45640 1 -- 172.21.15.46:0/71630605 >> v1:172.21.15.88:6789/0 conn(0x7f74a0164c70 legacy=0x7f74a00d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.994+0000 7f74a5a45640 1 -- 172.21.15.46:0/71630605 >> v1:172.21.15.46:6789/0 conn(0x7f74a01615e0 legacy=0x7f74a00d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.994+0000 7f74a5a45640 1 -- 172.21.15.46:0/71630605 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a00d18d0 con 0x7f74a00d2b90 2024-05-09T03:19:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f74a5a45640 1 --2- 172.21.15.46:0/71630605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a00d2b90 0x7f74a015dfe0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f7498009d90 tx=0x7f749802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528274e09b88d7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f7497fff640 1 -- 172.21.15.46:0/71630605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498031600 con 0x7f74a00d2b90 2024-05-09T03:19:38.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f7497fff640 1 -- 172.21.15.46:0/71630605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74980317a0 con 0x7f74a00d2b90 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f7497fff640 1 -- 172.21.15.46:0/71630605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498031b70 con 0x7f74a00d2b90 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f74a74a8640 1 -- 172.21.15.46:0/71630605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a00d2b90 msgr2=0x7f74a015dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f74a74a8640 1 --2- 172.21.15.46:0/71630605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a00d2b90 0x7f74a015dfe0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f7498009d90 tx=0x7f749802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f74a74a8640 1 -- 172.21.15.46:0/71630605 shutdown_connections 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f74a74a8640 1 --2- 172.21.15.46:0/71630605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a00d2b90 0x7f74a015dfe0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.995+0000 7f74a74a8640 1 -- 172.21.15.46:0/71630605 >> 172.21.15.46:0/71630605 conn(0x7f74a0073db0 msgr2=0x7f74a00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.996+0000 7f74a74a8640 1 -- 172.21.15.46:0/71630605 shutdown_connections 2024-05-09T03:19:38.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.996+0000 7f74a74a8640 1 -- 172.21.15.46:0/71630605 wait complete. 2024-05-09T03:19:38.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.997+0000 7f74a74a8640 1 Processor -- start 2024-05-09T03:19:38.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.997+0000 7f74a74a8640 1 -- start start 2024-05-09T03:19:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a74a8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74a00d2b90 0x7f74a00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a74a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 0x7f74a0162b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a74a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a0164c70 0x7f74a01522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a74a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74a01619f0 con 0x7f74a01615e0 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a74a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74a0161150 con 0x7f74a0164c70 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a74a8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f74a00d45c0 con 0x7f74a00d2b90 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 0x7f74a0162b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 0x7f74a0162b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43920/0 (socket says 172.21.15.46:43920) 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 -- 172.21.15.46:0/1318637071 learned_addr learned my addr 172.21.15.46:0/1318637071 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a5a45640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a0164c70 0x7f74a01522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a5244640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74a00d2b90 0x7f74a00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 -- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74a00d2b90 msgr2=0x7f74a00cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74a00d2b90 0x7f74a00cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 -- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a0164c70 msgr2=0x7f74a01522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a0164c70 0x7f74a01522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 -- 172.21.15.46:0/1318637071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a00cb1d0 con 0x7f74a01615e0 2024-05-09T03:19:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f74a4a43640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 0x7f74a0162b50 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f749000c400 tx=0x7f749000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490017070 con 0x7f74a01615e0 2024-05-09T03:19:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:38.999+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7490005d00 con 0x7f74a01615e0 2024-05-09T03:19:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.000+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490007310 con 0x7f74a01615e0 2024-05-09T03:19:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.000+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a00cb4b0 con 0x7f74a01615e0 2024-05-09T03:19:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.000+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a00cba70 con 0x7f74a01615e0 2024-05-09T03:19:39.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.001+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74a008cfd0 con 0x7f74a01615e0 2024-05-09T03:19:39.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.001+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f749001a070 con 0x7f74a01615e0 2024-05-09T03:19:39.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.002+0000 7f7495ffb640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74840762d0 0x7f74840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.002+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f7490097490 con 0x7f74a01615e0 2024-05-09T03:19:39.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.002+0000 7f74a5244640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74840762d0 0x7f74840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.003+0000 7f74a5244640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74840762d0 0x7f74840787e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f749c006fd0 tx=0x7f749c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.007+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f74a008cfd0 con 0x7f74a01615e0 2024-05-09T03:19:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.239+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74a006a270 con 0x7f74a01615e0 2024-05-09T03:19:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.240+0000 7f7495ffb640 1 -- 172.21.15.46:0/1318637071 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 292 v292) ==== 95+0+5169 (secure 0 0 0) 0x7f749005fba0 con 0x7f74a01615e0 2024-05-09T03:19:39.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:39.241 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":292,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":9879,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2209678569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2209678569},{"type":"v1","addr":"172.21.15.46:6834","nonce":2209678569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":292}],"filesystems":[{"mdsmap":{"epoch":291,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:35.739265+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 292 2024-05-09T03:19:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74840762d0 msgr2=0x7f74840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74840762d0 0x7f74840787e0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f749c006fd0 tx=0x7f749c009060 comp rx=0 tx=0).stop 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 msgr2=0x7f74a0162b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 0x7f74a0162b50 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f749000c400 tx=0x7f749000c8c0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 shutdown_connections 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74a0164c70 0x7f74a01522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f74840762d0 0x7f74840787e0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74a01615e0 0x7f74a0162b50 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 --2- 172.21.15.46:0/1318637071 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f74a00d2b90 0x7f74a00cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.242+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 >> 172.21.15.46:0/1318637071 conn(0x7f74a0073db0 msgr2=0x7f74a015fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.243+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 shutdown_connections 2024-05-09T03:19:39.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.243+0000 7f74a74a8640 1 -- 172.21.15.46:0/1318637071 wait complete. 2024-05-09T03:19:39.255 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap key 100.00000000:stray0_head' led to DAMAGED state 2024-05-09T03:19:39.255 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 1.00000000:subdir_head' 2024-05-09T03:19:39.255 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:39.255 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:39.272 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:19:39.272 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:39.272 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:39.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:39.272 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:19:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:19:39.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.483+0000 7f6cff029640 1 Processor -- start 2024-05-09T03:19:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.483+0000 7f6cff029640 1 -- start start 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.484+0000 7f6cff029640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cf80b3890 0x7f6cf80cc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.484+0000 7f6cff029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf815e350 0x7f6cf80b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.484+0000 7f6cff029640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf80b7920 con 0x7f6cf80d2150 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.484+0000 7f6cff029640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf80b3ca0 con 0x7f6cf80b3890 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.484+0000 7f6cff029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf80b7c00 con 0x7f6cf815e350 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cf80b3890 0x7f6cf80cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6cf80b3890 0x7f6cf80cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43928/0 (socket says 172.21.15.46:43928) 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/2389001970 learned_addr learned my addr 172.21.15.46:0/2389001970 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6ceffff640 1 --2- 172.21.15.46:0/2389001970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf815e350 0x7f6cf80b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/2389001970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf815e350 msgr2=0x7f6cf80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 --2- 172.21.15.46:0/2389001970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf815e350 0x7f6cf80b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/2389001970 >> v1:172.21.15.88:6789/0 conn(0x7f6cf80d2150 legacy=0x7f6cf80d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.485+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/2389001970 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf806c890 con 0x7f6cf80b3890 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.486+0000 7f6cfcdc5640 1 --2- 172.21.15.46:0/2389001970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf80cc810 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009e50 tx=0x7f6cf402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786ec1937d1432ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.486+0000 7f6cef7fe640 1 -- 172.21.15.46:0/2389001970 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4036070 con 0x7f6cf80b3890 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.486+0000 7f6cef7fe640 1 -- 172.21.15.46:0/2389001970 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4031600 con 0x7f6cf80b3890 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.486+0000 7f6cef7fe640 1 -- 172.21.15.46:0/2389001970 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40319d0 con 0x7f6cf80b3890 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 -- 172.21.15.46:0/2389001970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 msgr2=0x7f6cf80cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 --2- 172.21.15.46:0/2389001970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf80cc810 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009e50 tx=0x7f6cf402a5f0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 -- 172.21.15.46:0/2389001970 shutdown_connections 2024-05-09T03:19:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 --2- 172.21.15.46:0/2389001970 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf815e350 0x7f6cf80b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 --2- 172.21.15.46:0/2389001970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf80cc810 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 -- 172.21.15.46:0/2389001970 >> 172.21.15.46:0/2389001970 conn(0x7f6cf809b7d0 msgr2=0x7f6cf80b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 -- 172.21.15.46:0/2389001970 shutdown_connections 2024-05-09T03:19:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.487+0000 7f6cff029640 1 -- 172.21.15.46:0/2389001970 wait complete. 2024-05-09T03:19:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.489+0000 7f6cff029640 1 Processor -- start 2024-05-09T03:19:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.489+0000 7f6cff029640 1 -- start start 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cff029640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf814bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cff029640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf80d2150 0x7f6cf815f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cff029640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf815e350 0x7f6cf8070760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cff029640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf80b1e10 con 0x7f6cf80b3890 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cff029640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf80ccd50 con 0x7f6cf815e350 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cff029640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf815e760 con 0x7f6cf80d2150 2024-05-09T03:19:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cfcdc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf814bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cfcdc5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf814bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43942/0 (socket says 172.21.15.46:43942) 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/3624833414 learned_addr learned my addr 172.21.15.46:0/3624833414 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6ceffff640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf80d2150 0x7f6cf815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.490+0000 7f6cfd5c6640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf815e350 0x7f6cf8070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf80d2150 msgr2=0x7f6cf815f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cfcdc5640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf80d2150 0x7f6cf815f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf815e350 msgr2=0x7f6cf8070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cfcdc5640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf815e350 0x7f6cf8070760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cfcdc5640 1 -- 172.21.15.46:0/3624833414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf8160f50 con 0x7f6cf80b3890 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cfcdc5640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf814bbe0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4034040 tx=0x7f6cf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf403f020 con 0x7f6cf80b3890 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4037040 con 0x7f6cf80b3890 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4036070 con 0x7f6cf80b3890 2024-05-09T03:19:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf81603d0 con 0x7f6cf80b3890 2024-05-09T03:19:39.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.491+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf8160990 con 0x7f6cf80b3890 2024-05-09T03:19:39.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.492+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf806a270 con 0x7f6cf80b3890 2024-05-09T03:19:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.493+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6cf4042020 con 0x7f6cf80b3890 2024-05-09T03:19:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.493+0000 7f6ced7fa640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ce4076300 0x7f6ce4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.493+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f6cf40b88e0 con 0x7f6cf80b3890 2024-05-09T03:19:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.494+0000 7f6ceffff640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ce4076300 0x7f6ce4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.494+0000 7f6ceffff640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ce4076300 0x7f6ce4078810 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8161390 tx=0x7f6ce8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.497+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6cf4049030 con 0x7f6cf80b3890 2024-05-09T03:19:39.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.727+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6cf815cc80 con 0x7f6cf80b3890 2024-05-09T03:19:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.752+0000 7f6ced7fa640 1 -- 172.21.15.46:0/3624833414 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v293) ==== 161+0+0 (secure 0 0 0) 0x7f6cf4007c70 con 0x7f6cf80b3890 2024-05-09T03:19:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ce4076300 msgr2=0x7f6ce4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ce4076300 0x7f6ce4078810 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8161390 tx=0x7f6ce8009190 comp rx=0 tx=0).stop 2024-05-09T03:19:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 msgr2=0x7f6cf814bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf814bbe0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4034040 tx=0x7f6cf4004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 shutdown_connections 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf815e350 0x7f6cf8070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6cf80d2150 0x7f6cf815f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ce4076300 0x7f6ce4078810 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 --2- 172.21.15.46:0/3624833414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf80b3890 0x7f6cf814bbe0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 >> 172.21.15.46:0/3624833414 conn(0x7f6cf809b7d0 msgr2=0x7f6cf806cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.755+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 shutdown_connections 2024-05-09T03:19:39.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.756+0000 7f6cff029640 1 -- 172.21.15.46:0/3624833414 wait complete. 2024-05-09T03:19:39.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.883+0000 7f99319f3640 1 Processor -- start 2024-05-09T03:19:39.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.883+0000 7f99319f3640 1 -- start start 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f99319f3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f992c0d2eb0 0x7f992c0d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f99319f3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f992c0b7760 0x7f992c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f99319f3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f992c0b80b0 con 0x7f992c0d38b0 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f99319f3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f992c0d4a60 con 0x7f992c0d2eb0 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f99319f3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f992c0d4d40 con 0x7f992c0b7760 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f992c0d2eb0 0x7f992c0d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f992c0d38b0 0x7f992c0d3cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35108/0 (socket says 172.21.15.46:35108) 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992b7fe640 1 -- 172.21.15.46:0/3582863188 learned_addr learned my addr 172.21.15.46:0/3582863188 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992a7fc640 1 --2- 172.21.15.46:0/3582863188 >> v2:172.21.15.88:3300/0 conn(0x7f992c0b7760 0x7f992c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992affd640 1 -- 172.21.15.46:0/3582863188 >> v2:172.21.15.88:3300/0 conn(0x7f992c0b7760 msgr2=0x7f992c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992affd640 1 --2- 172.21.15.46:0/3582863188 >> v2:172.21.15.88:3300/0 conn(0x7f992c0b7760 0x7f992c0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992affd640 1 -- 172.21.15.46:0/3582863188 >> v1:172.21.15.46:6789/0 conn(0x7f992c0d38b0 legacy=0x7f992c0d3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f992affd640 1 -- 172.21.15.46:0/3582863188 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992c0d5020 con 0x7f992c0d2eb0 2024-05-09T03:19:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.885+0000 7f9929ffb640 1 -- 172.21.15.46:0/3582863188 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659014278 0 0) 0x7f9924001110 con 0x7f992c0d38b0 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.886+0000 7f992affd640 1 --2- 172.21.15.46:0/3582863188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0d2eb0 0x7f992c0d32c0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f992c0d03c0 tx=0x7f991c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c5f4856f5952f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.886+0000 7f9929ffb640 1 -- 172.21.15.46:0/3582863188 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c007740 con 0x7f992c0d2eb0 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.886+0000 7f9929ffb640 1 -- 172.21.15.46:0/3582863188 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f991c0078e0 con 0x7f992c0d2eb0 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.886+0000 7f9929ffb640 1 -- 172.21.15.46:0/3582863188 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c007cb0 con 0x7f992c0d2eb0 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 -- 172.21.15.46:0/3582863188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0d2eb0 msgr2=0x7f992c0d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 --2- 172.21.15.46:0/3582863188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0d2eb0 0x7f992c0d32c0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f992c0d03c0 tx=0x7f991c02a680 comp rx=0 tx=0).stop 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 -- 172.21.15.46:0/3582863188 shutdown_connections 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 --2- 172.21.15.46:0/3582863188 >> v2:172.21.15.88:3300/0 conn(0x7f992c0b7760 0x7f992c0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 --2- 172.21.15.46:0/3582863188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0d2eb0 0x7f992c0d32c0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 -- 172.21.15.46:0/3582863188 >> 172.21.15.46:0/3582863188 conn(0x7f992c08d390 msgr2=0x7f992c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 -- 172.21.15.46:0/3582863188 shutdown_connections 2024-05-09T03:19:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.887+0000 7f99319f3640 1 -- 172.21.15.46:0/3582863188 wait complete. 2024-05-09T03:19:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.889+0000 7f99319f3640 1 Processor -- start 2024-05-09T03:19:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.889+0000 7f99319f3640 1 -- start start 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f99319f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 0x7f992c1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f99319f3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c0d2eb0 0x7f992c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f99319f3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f992c0d38b0 0x7f992c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f99319f3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f992c0b51e0 con 0x7f992c0b7760 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f99319f3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f992c0d55e0 con 0x7f992c0d2eb0 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f99319f3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f992c0b5360 con 0x7f992c0d38b0 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f992affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 0x7f992c1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f992affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 0x7f992c1f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43968/0 (socket says 172.21.15.46:43968) 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f992affd640 1 -- 172.21.15.46:0/82148255 learned_addr learned my addr 172.21.15.46:0/82148255 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f992a7fc640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c0d2eb0 0x7f992c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.890+0000 7f992b7fe640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f992c0d38b0 0x7f992c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992affd640 1 -- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f992c0d38b0 msgr2=0x7f992c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992affd640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f992c0d38b0 0x7f992c0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992affd640 1 -- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c0d2eb0 msgr2=0x7f992c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992affd640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c0d2eb0 0x7f992c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992affd640 1 -- 172.21.15.46:0/82148255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992c0cc460 con 0x7f992c0b7760 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992b7fe640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f992c0d38b0 0x7f992c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f992affd640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 0x7f992c1f6420 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f991c02ac70 tx=0x7f991c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c03f070 con 0x7f992c0b7760 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f991c031c40 con 0x7f992c0b7760 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992c0c9940 con 0x7f992c0b7760 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c043410 con 0x7f992c0b7760 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.891+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f992c154e80 con 0x7f992c0b7760 2024-05-09T03:19:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.892+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f992c06a270 con 0x7f992c0b7760 2024-05-09T03:19:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.893+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f991c0435b0 con 0x7f992c0b7760 2024-05-09T03:19:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.893+0000 7f9913fff640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9918076300 0x7f9918078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.894+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f991c0b9680 con 0x7f992c0b7760 2024-05-09T03:19:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.894+0000 7f992a7fc640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9918076300 0x7f9918078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.894+0000 7f992a7fc640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9918076300 0x7f9918078810 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f992c0b6a80 tx=0x7f9920008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:39.898+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f991c082360 con 0x7f992c0b7760 2024-05-09T03:19:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.169+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f992c0c9420 con 0x7f992c0b7760 2024-05-09T03:19:40.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.761+0000 7f9913fff640 1 -- 172.21.15.46:0/82148255 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v294) ==== 101+0+0 (secure 0 0 0) 0x7f991c081c60 con 0x7f992c0b7760 2024-05-09T03:19:40.762 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9918076300 msgr2=0x7f9918078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9918076300 0x7f9918078810 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f992c0b6a80 tx=0x7f9920008040 comp rx=0 tx=0).stop 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 msgr2=0x7f992c1f6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 0x7f992c1f6420 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f991c02ac70 tx=0x7f991c004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 shutdown_connections 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f992c0d38b0 0x7f992c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c0d2eb0 0x7f992c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9918076300 0x7f9918078810 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 --2- 172.21.15.46:0/82148255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c0b7760 0x7f992c1f6420 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 >> 172.21.15.46:0/82148255 conn(0x7f992c08d390 msgr2=0x7f992c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 shutdown_connections 2024-05-09T03:19:40.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.765+0000 7f99319f3640 1 -- 172.21.15.46:0/82148255 wait complete. 2024-05-09T03:19:40.780 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:40.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.840+0000 7fd302dfef40 1 Processor -- start 2024-05-09T03:19:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.840+0000 7fd302dfef40 1 -- start start 2024-05-09T03:19:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd302dfef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed641d67e0 0x55ed64291b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd302dfef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed64293a50 0x55ed6421dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd302dfef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 0x55ed642079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd302dfef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ed64218cb0 con 0x55ed642ae780 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd302dfef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ed6417ff50 con 0x55ed641d67e0 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd302dfef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55ed641d6bf0 con 0x55ed64293a50 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd2fd0c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 0x55ed642079f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd2fd0c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 0x55ed642079f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43992/0 (socket says 172.21.15.46:43992) 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.841+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/1655000609 learned_addr learned my addr 172.21.15.46:0/1655000609 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed64293a50 msgr2=0x55ed6421dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed64293a50 0x55ed6421dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed641d67e0 0x55ed64291b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed641d67e0 msgr2=0x55ed64291b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed641d67e0 0x55ed64291b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/1655000609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed642acf10 con 0x55ed642ae780 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 0x55ed642079f0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400a5a0 tx=0x7fd2f4051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7a41c3e90019a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fb0c5640 1 -- 172.21.15.46:0/1655000609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4069070 con 0x55ed642ae780 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fb0c5640 1 -- 172.21.15.46:0/1655000609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f4052a90 con 0x55ed642ae780 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.842+0000 7fd2fb0c5640 1 -- 172.21.15.46:0/1655000609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4056070 con 0x55ed642ae780 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 -- 172.21.15.46:0/1655000609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 msgr2=0x55ed642079f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 0x55ed642079f0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400a5a0 tx=0x7fd2f4051000 comp rx=0 tx=0).stop 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 -- 172.21.15.46:0/1655000609 shutdown_connections 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed642ae780 0x55ed642079f0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed64293a50 0x55ed6421dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 --2- 172.21.15.46:0/1655000609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed641d67e0 0x55ed64291b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 -- 172.21.15.46:0/1655000609 >> 172.21.15.46:0/1655000609 conn(0x55ed640fbac0 msgr2=0x55ed6421e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 -- 172.21.15.46:0/1655000609 shutdown_connections 2024-05-09T03:19:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.843+0000 7fd302dfef40 1 -- 172.21.15.46:0/1655000609 wait complete. 2024-05-09T03:19:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.844+0000 7fd302dfef40 1 Processor -- start 2024-05-09T03:19:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.844+0000 7fd302dfef40 1 -- start start 2024-05-09T03:19:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd302dfef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed641d67e0 0x55ed643438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd302dfef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 0x55ed6421db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd302dfef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed642ae780 0x55ed6421e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd302dfef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ed642abf90 con 0x55ed64293a50 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd302dfef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ed64295e40 con 0x55ed642ae780 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd302dfef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55ed6421a230 con 0x55ed641d67e0 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 0x55ed6421db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 0x55ed6421db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43998/0 (socket says 172.21.15.46:43998) 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 learned_addr learned my addr 172.21.15.46:0/230175787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed642ae780 0x55ed6421e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed641d67e0 0x55ed643438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed641d67e0 msgr2=0x55ed643438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55ed641d67e0 0x55ed643438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed642ae780 msgr2=0x55ed6421e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed642ae780 0x55ed6421e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.845+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed6421a480 con 0x55ed64293a50 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.846+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 0x55ed6421db30 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009da0 tx=0x7fd2f0006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.846+0000 7fd2fa0c3640 1 -- 172.21.15.46:0/230175787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0017070 con 0x55ed64293a50 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.846+0000 7fd2fa0c3640 1 -- 172.21.15.46:0/230175787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f0012410 con 0x55ed64293a50 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.846+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ed6421a760 con 0x55ed64293a50 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.846+0000 7fd2fa0c3640 1 -- 172.21.15.46:0/230175787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0006e60 con 0x55ed64293a50 2024-05-09T03:19:40.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.846+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed6421ad20 con 0x55ed64293a50 2024-05-09T03:19:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.848+0000 7fd2fa0c3640 1 -- 172.21.15.46:0/230175787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd2f001a070 con 0x55ed64293a50 2024-05-09T03:19:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.848+0000 7fd2fa0c3640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2e8074ce0 0x7fd2e80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.848+0000 7fd2fa0c3640 1 -- 172.21.15.46:0/230175787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7fd2f0095eb0 con 0x55ed64293a50 2024-05-09T03:19:40.850 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.849+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2e8074ce0 0x7fd2e80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.849+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55ed642953c0 0x55ed64350550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.849+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6421b520 con 0x55ed642953c0 2024-05-09T03:19:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.849+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55ed642953c0 0x55ed64350550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.849+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2e8074ce0 0x7fd2e80771f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0012e0 tx=0x7fd2ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.849+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55ed642953c0 0x55ed64350550 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.852+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v117'1002 uv976 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.852+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64350a90 con 0x55ed642953c0 2024-05-09T03:19:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.853+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v117'1003 uv1003 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.853+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64350e30 con 0x55ed642953c0 2024-05-09T03:19:40.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.855+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v117'1004 uv1004 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.856 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.855+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643521e0 con 0x55ed642953c0 2024-05-09T03:19:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.856+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v117'1005 uv1005 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.857+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64352580 con 0x55ed642953c0 2024-05-09T03:19:40.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.858+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v117'1006 uv1006 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.858+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643511d0 con 0x55ed642953c0 2024-05-09T03:19:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.859+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v117'1007 uv1007 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.859+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643515c0 con 0x55ed642953c0 2024-05-09T03:19:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.861+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v117'1008 uv982 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.861+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64351960 con 0x55ed642953c0 2024-05-09T03:19:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.863+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v117'1009 uv1009 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.863+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.864+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v117'1010 uv1010 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.865 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:40.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.864+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.865+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v117'1011 uv1011 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.865+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.867+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v117'1012 uv1012 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.867+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.869+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v117'1013 uv1013 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.869+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.870+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v117'1014 uv989 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.870+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.872+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v117'1015 uv1015 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.872+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.873+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v117'1016 uv1016 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.874 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:40.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.873+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.875+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v117'1017 uv1017 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.875+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.876+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v117'1018 uv1018 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.876+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.877+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v117'1019 uv1019 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.877+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.879+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v117'1020 uv1020 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.879+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.880+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v117'1021 uv995 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.880+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.882+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v117'1022 uv1022 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.882+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.883+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v117'1023 uv1023 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.884 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:40.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.883+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.884+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v117'1024 uv1024 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.885+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.886+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v117'1025 uv1025 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.886+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.888+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v117'1026 uv1026 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.888+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.889+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v117'1027 uv1001 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.889+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.891+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v117'1028 uv1028 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.891+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.892+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v117'1029 uv1029 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.893 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:40.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.892+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.894+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v117'1030 uv1030 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.894+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.895+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v117'1031 uv1031 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.895+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64351d00 con 0x55ed642953c0 2024-05-09T03:19:40.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.896+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v117'1032 uv1032 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.896+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55ed64351d00 0x55ed64354e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.896+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64355380 con 0x55ed64351d00 2024-05-09T03:19:40.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.897+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55ed64351d00 0x55ed64354e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.897+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55ed64351d00 0x55ed64354e40 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.899+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v117'721 uv703 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.899+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64355720 con 0x55ed64351d00 2024-05-09T03:19:40.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.900+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v117'722 uv722 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.900+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64355ac0 con 0x55ed64351d00 2024-05-09T03:19:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.901+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v117'723 uv723 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.902 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:40.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.901+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64355e60 con 0x55ed64351d00 2024-05-09T03:19:40.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.903+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v117'724 uv724 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.903+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64356200 con 0x55ed64351d00 2024-05-09T03:19:40.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.904+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v117'725 uv725 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.904+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643565a0 con 0x55ed64351d00 2024-05-09T03:19:40.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.905+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v117'726 uv708 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.905+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64356940 con 0x55ed64351d00 2024-05-09T03:19:40.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.907+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v117'727 uv727 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.907+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64356ce0 con 0x55ed64351d00 2024-05-09T03:19:40.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.908+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v117'728 uv728 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.908 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:40.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.908+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.909+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v117'729 uv729 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.909+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.910+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v117'730 uv730 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.910+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.912+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v117'731 uv713 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.912+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.913+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v117'732 uv732 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.913+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.914+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v117'733 uv733 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.915 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:40.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.914+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.916+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v117'734 uv734 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.916+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.917+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v117'735 uv735 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.917+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.918+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v117'736 uv719 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.918+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.920+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v117'737 uv737 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.920+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed642ab120 con 0x55ed64351d00 2024-05-09T03:19:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.921+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v117'738 uv738 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.921 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.921+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643581a0 con 0x55ed64351d00 2024-05-09T03:19:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.922+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v117'739 uv739 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.922+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643581a0 con 0x55ed64351d00 2024-05-09T03:19:40.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.923+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v117'740 uv740 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.923+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64357190 con 0x55ed64351d00 2024-05-09T03:19:40.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.924+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v117'741 uv741 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed64351d00 2024-05-09T03:19:40.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.924+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64357190 con 0x55ed642953c0 2024-05-09T03:19:40.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.926+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v117'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.926+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64357190 con 0x55ed642953c0 2024-05-09T03:19:40.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.927+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v117'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.927+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64357190 con 0x55ed642953c0 2024-05-09T03:19:40.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.928+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v117'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.929 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:40.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.928+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643581a0 con 0x55ed642953c0 2024-05-09T03:19:40.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.929+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v117'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.929+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643581a0 con 0x55ed642953c0 2024-05-09T03:19:40.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.930+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v117'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.930+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64357190 con 0x55ed642953c0 2024-05-09T03:19:40.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.931+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v117'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.931+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55ed643576a0 0x55ed64359ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.931+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435a0d0 con 0x55ed643576a0 2024-05-09T03:19:40.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.931+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55ed643576a0 0x55ed64359ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.931+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55ed643576a0 0x55ed64359ac0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.932+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v117'146 uv145 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.933+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed6435a5e0 con 0x55ed643576a0 2024-05-09T03:19:40.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.934+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v117'147 uv147 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.934+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435aae0 con 0x55ed643576a0 2024-05-09T03:19:40.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.935+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v117'148 uv148 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.935 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:40.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.935+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64356dc0 con 0x55ed643576a0 2024-05-09T03:19:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.936+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v117'149 uv149 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.936+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435b410 con 0x55ed642953c0 2024-05-09T03:19:40.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.937+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v117'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.937+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed6435b410 con 0x55ed642953c0 2024-05-09T03:19:40.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.938+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v117'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.938+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435b410 con 0x55ed642953c0 2024-05-09T03:19:40.940 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:40.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.939+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v117'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.939+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435c420 con 0x55ed642953c0 2024-05-09T03:19:40.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.941+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v117'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.941+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435c420 con 0x55ed642953c0 2024-05-09T03:19:40.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.942+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v117'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.942+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435b410 con 0x55ed642953c0 2024-05-09T03:19:40.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.943+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v117'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.943+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55ed6435ba90 0x55ed6435def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.943+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e550 con 0x55ed6435ba90 2024-05-09T03:19:40.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.943+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55ed6435ba90 0x55ed6435def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.944+0000 7fd2fd0c9640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55ed6435ba90 0x55ed6435def0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.945+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v117'360 uv354 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.945+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.946+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v117'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.946+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.948+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v117'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.948 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:40.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.948+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435f930 con 0x55ed6435ba90 2024-05-09T03:19:40.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.949+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v117'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.949+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435f930 con 0x55ed6435ba90 2024-05-09T03:19:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.950+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v117'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.950+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.951+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v117'365 uv365 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.951+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.953+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v117'366 uv359 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.953+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.954+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v117'367 uv367 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.954+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.955+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v117'368 uv368 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.956 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.955+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.957+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v117'369 uv369 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.957+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed6435e920 con 0x55ed6435ba90 2024-05-09T03:19:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.958+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v117'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.958+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55ed6435efa0 0x55ed643613e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.958+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361b00 con 0x55ed6435efa0 2024-05-09T03:19:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.958+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55ed6435efa0 0x55ed643613e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.959+0000 7fd2fc8c8640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55ed6435efa0 0x55ed643613e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.960+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v117'163 uv162 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed6435efa0 2024-05-09T03:19:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.960+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed6435efa0 2024-05-09T03:19:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.961+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v117'164 uv164 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed6435efa0 2024-05-09T03:19:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.961+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed6435efa0 2024-05-09T03:19:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.962+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v117'165 uv165 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed6435efa0 2024-05-09T03:19:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.962+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed6435efa0 2024-05-09T03:19:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.964+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v117'166 uv166 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed6435efa0 2024-05-09T03:19:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.964+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed6435efa0 2024-05-09T03:19:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.965+0000 7fd2fc8c8640 1 -- 172.21.15.46:0/230175787 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v117'167 uv167 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2ec0136a0 con 0x55ed6435efa0 2024-05-09T03:19:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.965+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed642953c0 2024-05-09T03:19:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.966+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v117'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.966+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed642953c0 2024-05-09T03:19:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.967+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v117'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.967+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed642953c0 2024-05-09T03:19:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.968+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v117'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.969 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.968+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ee0 con 0x55ed642953c0 2024-05-09T03:19:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.969+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v117'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.969+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ee0 con 0x55ed642953c0 2024-05-09T03:19:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.970+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v117'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.970+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed642953c0 2024-05-09T03:19:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.971+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v117'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed642953c0 2024-05-09T03:19:40.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.971+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64361ed0 con 0x55ed643576a0 2024-05-09T03:19:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.972+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v117'325 uv318 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.972+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64362300 con 0x55ed643576a0 2024-05-09T03:19:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.974+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v117'326 uv326 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.974+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643626f0 con 0x55ed643576a0 2024-05-09T03:19:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.975+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v117'327 uv327 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.975 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:40.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.975+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ae0 con 0x55ed643576a0 2024-05-09T03:19:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.976+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v117'328 uv328 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.976+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ae0 con 0x55ed643576a0 2024-05-09T03:19:40.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.977+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v117'329 uv329 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.977+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ea0 con 0x55ed643576a0 2024-05-09T03:19:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.978+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v117'330 uv324 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.978+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64362ea0 con 0x55ed643576a0 2024-05-09T03:19:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.979+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v117'331 uv331 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.979+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ea0 con 0x55ed643576a0 2024-05-09T03:19:40.980 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.980+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v117'332 uv332 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.980+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64362ea0 con 0x55ed643576a0 2024-05-09T03:19:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.981+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v117'333 uv333 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.981+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643627d0 con 0x55ed643576a0 2024-05-09T03:19:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.982+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v117'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.982+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55ed64363260 0x55ed64365680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.982+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64365c50 con 0x55ed64363260 2024-05-09T03:19:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.982+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55ed64363260 0x55ed64365680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.983+0000 7fd2fc0c7640 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55ed64363260 0x55ed64365680 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:40.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.984+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v117'142 uv140 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed64363260 2024-05-09T03:19:40.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.984+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed64365c50 con 0x55ed64363260 2024-05-09T03:19:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.985+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v117'143 uv143 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed64363260 2024-05-09T03:19:40.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.985+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64365c50 con 0x55ed64363260 2024-05-09T03:19:40.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.986+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v117'144 uv144 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed64363260 2024-05-09T03:19:40.987 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:40.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.986+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64366100 con 0x55ed64363260 2024-05-09T03:19:40.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.987+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v117'145 uv145 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed64363260 2024-05-09T03:19:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.987+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64366100 con 0x55ed643576a0 2024-05-09T03:19:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.989+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v117'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.989+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed643576a0 2024-05-09T03:19:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.990+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v117'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.990+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed643576a0 2024-05-09T03:19:40.991 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.991+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v117'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.991+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643670c0 con 0x55ed643576a0 2024-05-09T03:19:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.992+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v117'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.992+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643670c0 con 0x55ed643576a0 2024-05-09T03:19:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.994+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v117'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.994+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed643576a0 2024-05-09T03:19:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.995+0000 7fd2fc0c7640 1 -- 172.21.15.46:0/230175787 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v117'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f0019030 con 0x55ed643576a0 2024-05-09T03:19:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.995+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.996+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v117'166 uv165 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.996+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.997+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v117'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:40.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:40.997+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.000+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v117'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.001 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:41.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.000+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.001+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v117'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.001+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.003+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v117'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.003+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.004+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v117'402 uv393 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.004+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.005+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v117'403 uv403 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.006+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.007+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v117'404 uv404 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.007 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.007+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.008+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v117'405 uv405 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.008+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.009+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v117'406 uv406 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.009+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.011+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v117'407 uv400 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.011+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.012+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v117'408 uv408 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.012+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.013+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v117'409 uv409 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.014 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.013+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643670c0 con 0x55ed6435ba90 2024-05-09T03:19:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.014+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v117'410 uv410 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.014+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643670c0 con 0x55ed6435ba90 2024-05-09T03:19:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.016+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v117'411 uv411 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.016+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed643660b0 con 0x55ed6435ba90 2024-05-09T03:19:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.017+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v117'412 uv412 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.017+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55ed64366320 con 0x55ed6435ba90 2024-05-09T03:19:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v117'413 uv413 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd2f4050050 con 0x55ed6435ba90 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55ed642953c0 msgr2=0x55ed64350550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55ed642953c0 0x55ed64350550 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55ed643576a0 msgr2=0x55ed64359ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55ed643576a0 0x55ed64359ac0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55ed64363260 msgr2=0x55ed64365680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55ed64363260 0x55ed64365680 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55ed64351d00 msgr2=0x55ed64354e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55ed64351d00 0x55ed64354e40 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55ed6435ba90 msgr2=0x55ed6435def0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55ed6435ba90 0x55ed6435def0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55ed6435efa0 msgr2=0x55ed643613e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55ed6435efa0 0x55ed643613e0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 reap_dead start 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 reap_dead start 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd2fd0c9640 1 -- 172.21.15.46:0/230175787 reap_dead start 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2e8074ce0 msgr2=0x7fd2e80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2e8074ce0 0x7fd2e80771f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0012e0 tx=0x7fd2ec008040 comp rx=0 tx=0).stop 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 msgr2=0x55ed6421db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 0x55ed6421db30 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009da0 tx=0x7fd2f0006750 comp rx=0 tx=0).stop 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 shutdown_connections 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd2e8074ce0 0x7fd2e80771f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 --2- 172.21.15.46:0/230175787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed64293a50 0x55ed6421db30 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 >> 172.21.15.46:0/230175787 conn(0x55ed640fbac0 msgr2=0x55ed6421e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.018+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 shutdown_connections 2024-05-09T03:19:41.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.019+0000 7fd302dfef40 1 -- 172.21.15.46:0/230175787 wait complete. 2024-05-09T03:19:41.022 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 1.00000000 subdir_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:19:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.079+0000 7f56e610ff40 1 Processor -- start 2024-05-09T03:19:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.079+0000 7f56e610ff40 1 -- start start 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e610ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc227270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e610ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc1807c0 0x5626cc17bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e610ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 0x5626cc17ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e610ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5626cc1954e0 con 0x5626cc228820 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e610ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5626cc0f7f50 con 0x5626cc14e7f0 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e610ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5626cc19a8b0 con 0x5626cc1807c0 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.080+0000 7f56e03da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 0x5626cc17ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 0x5626cc17ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44016/0 (socket says 172.21.15.46:44016) 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 -- 172.21.15.46:0/745674611 learned_addr learned my addr 172.21.15.46:0/745674611 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56dfbd9640 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc227270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56df3d8640 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc1807c0 0x5626cc17bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 -- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc1807c0 msgr2=0x5626cc17bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc1807c0 0x5626cc17bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 -- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 msgr2=0x5626cc227270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc227270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 -- 172.21.15.46:0/745674611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626cc198f70 con 0x5626cc228820 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56dfbd9640 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc227270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56e03da640 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 0x5626cc17ef00 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f56d8045fd0 tx=0x7f56d8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e53bddede6f2c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56de3d6640 1 -- 172.21.15.46:0/745674611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d8052a90 con 0x5626cc228820 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56de3d6640 1 -- 172.21.15.46:0/745674611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d8052c30 con 0x5626cc228820 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.081+0000 7f56de3d6640 1 -- 172.21.15.46:0/745674611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d8067480 con 0x5626cc228820 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 -- 172.21.15.46:0/745674611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 msgr2=0x5626cc17ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 0x5626cc17ef00 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f56d8045fd0 tx=0x7f56d8051000 comp rx=0 tx=0).stop 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 -- 172.21.15.46:0/745674611 shutdown_connections 2024-05-09T03:19:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc228820 0x5626cc17ef00 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc1807c0 0x5626cc17bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 --2- 172.21.15.46:0/745674611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc227270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 -- 172.21.15.46:0/745674611 >> 172.21.15.46:0/745674611 conn(0x5626cc073ac0 msgr2=0x5626cc17e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 -- 172.21.15.46:0/745674611 shutdown_connections 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.082+0000 7f56e610ff40 1 -- 172.21.15.46:0/745674611 wait complete. 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.083+0000 7f56e610ff40 1 Processor -- start 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.083+0000 7f56e610ff40 1 -- start start 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e610ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc2251c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e610ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc1807c0 0x5626cc215380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e610ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc228820 0x5626cc212040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e610ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5626cc2277b0 con 0x5626cc1807c0 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e610ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5626cc213d20 con 0x5626cc14e7f0 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e610ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5626cc2c16d0 con 0x5626cc228820 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc1807c0 0x5626cc215380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc1807c0 0x5626cc215380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44030/0 (socket says 172.21.15.46:44030) 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 -- 172.21.15.46:0/924481195 learned_addr learned my addr 172.21.15.46:0/924481195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56e03da640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc228820 0x5626cc212040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56dfbd9640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc2251c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 -- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc228820 msgr2=0x5626cc212040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5626cc228820 0x5626cc212040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 -- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 msgr2=0x5626cc2251c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5626cc14e7f0 0x5626cc2251c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.084+0000 7f56df3d8640 1 -- 172.21.15.46:0/924481195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626cc2c19b0 con 0x5626cc1807c0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.085+0000 7f56df3d8640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc1807c0 0x5626cc215380 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f56d4006dc0 tx=0x7f56d4005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.085+0000 7f56dd3d4640 1 -- 172.21.15.46:0/924481195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4017070 con 0x5626cc1807c0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.085+0000 7f56dd3d4640 1 -- 172.21.15.46:0/924481195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d4010710 con 0x5626cc1807c0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.085+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5626cc2c1c90 con 0x5626cc1807c0 2024-05-09T03:19:41.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.085+0000 7f56dd3d4640 1 -- 172.21.15.46:0/924481195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4010ac0 con 0x5626cc1807c0 2024-05-09T03:19:41.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.085+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626cc2c2250 con 0x5626cc1807c0 2024-05-09T03:19:41.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.087+0000 7f56dd3d4640 1 -- 172.21.15.46:0/924481195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f56d401a070 con 0x5626cc1807c0 2024-05-09T03:19:41.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.087+0000 7f56dd3d4640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56cc074cb0 0x7f56cc0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.087+0000 7f56dd3d4640 1 -- 172.21.15.46:0/924481195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f56d4096040 con 0x5626cc1807c0 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.087+0000 7f56dfbd9640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56cc074cb0 0x7f56cc0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.087+0000 7f56e610ff40 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5626cc2bd050 0x5626cc2bd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.087+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=103b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x5626cc2bd9a0 con 0x5626cc2bd050 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.088+0000 7f56e03da640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5626cc2bd050 0x5626cc2bd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.088+0000 7f56dfbd9640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56cc074cb0 0x7f56cc0771c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x5626cc191f10 tx=0x7f56d0007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.088+0000 7f56e03da640 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5626cc2bd050 0x5626cc2bd460 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.089+0000 7f56e03da640 1 -- 172.21.15.46:0/924481195 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 1.00000000 [omap-set-vals] v117'414 uv414 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f56d8050050 con 0x5626cc2bd050 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.089+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5626cc2bd050 msgr2=0x5626cc2bd460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.089+0000 7f56e610ff40 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x5626cc2bd050 0x5626cc2bd460 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56cc074cb0 msgr2=0x7f56cc0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f56cc074cb0 0x7f56cc0771c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x5626cc191f10 tx=0x7f56d0007040 comp rx=0 tx=0).stop 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc1807c0 msgr2=0x5626cc215380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 --2- 172.21.15.46:0/924481195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5626cc1807c0 0x5626cc215380 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f56d4006dc0 tx=0x7f56d4005720 comp rx=0 tx=0).stop 2024-05-09T03:19:41.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e03da640 1 -- 172.21.15.46:0/924481195 reap_dead start 2024-05-09T03:19:41.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 shutdown_connections 2024-05-09T03:19:41.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 >> 172.21.15.46:0/924481195 conn(0x5626cc073ac0 msgr2=0x5626cc1921a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:41.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 shutdown_connections 2024-05-09T03:19:41.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.090+0000 7f56e610ff40 1 -- 172.21.15.46:0/924481195 wait complete. 2024-05-09T03:19:41.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:41.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.211+0000 7f4dcae9f640 1 Processor -- start 2024-05-09T03:19:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.211+0000 7f4dcae9f640 1 -- start start 2024-05-09T03:19:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dcae9f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dc40b5140 0x7f4dc40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dcae9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc40d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dcae9f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc40b80b0 con 0x7f4dc40d3df0 2024-05-09T03:19:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dcae9f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc40d3480 con 0x7f4dc40b5140 2024-05-09T03:19:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dcae9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc40b5550 con 0x7f4dc40b7ca0 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dc8c3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4dc40b5140 0x7f4dc40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dc943c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4dc40d3df0 0x7f4dc4163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35112/0 (socket says 172.21.15.46:35112) 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.212+0000 7f4dc943c640 1 -- 172.21.15.46:0/2827328129 learned_addr learned my addr 172.21.15.46:0/2827328129 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.213+0000 7f4dbbfff640 1 --2- 172.21.15.46:0/2827328129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc40d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.213+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/2827328129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 msgr2=0x7f4dc40d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.213+0000 7f4dc8c3b640 1 --2- 172.21.15.46:0/2827328129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc40d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.213+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/2827328129 >> v1:172.21.15.46:6789/0 conn(0x7f4dc40d3df0 legacy=0x7f4dc4163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.213+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/2827328129 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc40d44e0 con 0x7f4dc40b5140 2024-05-09T03:19:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.213+0000 7f4dc8c3b640 1 --2- 172.21.15.46:0/2827328129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc40b7760 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00097a0 tx=0x7f4dc002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa28409e68b79c0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dbb7fe640 1 -- 172.21.15.46:0/2827328129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0037070 con 0x7f4dc40b5140 2024-05-09T03:19:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dbb7fe640 1 -- 172.21.15.46:0/2827328129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc0032440 con 0x7f4dc40b5140 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dbb7fe640 1 -- 172.21.15.46:0/2827328129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0032810 con 0x7f4dc40b5140 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 -- 172.21.15.46:0/2827328129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 msgr2=0x7f4dc40b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/2827328129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc40b7760 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00097a0 tx=0x7f4dc002a210 comp rx=0 tx=0).stop 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 -- 172.21.15.46:0/2827328129 shutdown_connections 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/2827328129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc40d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/2827328129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc40b7760 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 -- 172.21.15.46:0/2827328129 >> 172.21.15.46:0/2827328129 conn(0x7f4dc408d3f0 msgr2=0x7f4dc408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 -- 172.21.15.46:0/2827328129 shutdown_connections 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.214+0000 7f4dcae9f640 1 -- 172.21.15.46:0/2827328129 wait complete. 2024-05-09T03:19:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.216+0000 7f4dcae9f640 1 Processor -- start 2024-05-09T03:19:41.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.216+0000 7f4dcae9f640 1 -- start start 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dcae9f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc4154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dcae9f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc4155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dcae9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4dc40d3df0 0x7f4dc4155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dcae9f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc40d4d80 con 0x7f4dc40b5140 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dcae9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc40d4aa0 con 0x7f4dc40d3df0 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dcae9f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc40d4200 con 0x7f4dc40b7ca0 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc4154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc4154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44060/0 (socket says 172.21.15.46:44060) 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/757878825 learned_addr learned my addr 172.21.15.46:0/757878825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dbbfff640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc4155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc943c640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4dc40d3df0 0x7f4dc4155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 msgr2=0x7f4dc4155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc4155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4dc40d3df0 msgr2=0x7f4dc4155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4dc40d3df0 0x7f4dc4155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 -- 172.21.15.46:0/757878825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc41554b0 con 0x7f4dc40b5140 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4dc8c3b640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc4154690 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00098c0 tx=0x7f4dc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.218+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0037040 con 0x7f4dc40b5140 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.219+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc0030770 con 0x7f4dc40b5140 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.219+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc4163840 con 0x7f4dc40b5140 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.219+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0030b40 con 0x7f4dc40b5140 2024-05-09T03:19:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.219+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc415a170 con 0x7f4dc40b5140 2024-05-09T03:19:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.219+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc406a270 con 0x7f4dc40b5140 2024-05-09T03:19:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.220+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4dc0044020 con 0x7f4dc40b5140 2024-05-09T03:19:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.221+0000 7f4db97fa640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4db0076300 0x7f4db0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.221+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f4dc00b90a0 con 0x7f4dc40b5140 2024-05-09T03:19:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.221+0000 7f4dbbfff640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4db0076300 0x7f4db0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.222+0000 7f4dbbfff640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4db0076300 0x7f4db0078810 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4dc41558d0 tx=0x7f4db4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.225+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4dc41554b0 con 0x7f4dc40b5140 2024-05-09T03:19:41.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.453+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4dc41604e0 con 0x7f4dc40b5140 2024-05-09T03:19:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.763+0000 7f4db97fa640 1 -- 172.21.15.46:0/757878825 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v295) ==== 162+0+0 (secure 0 0 0) 0x7f4dc0032e20 con 0x7f4dc40b5140 2024-05-09T03:19:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.766+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4db0076300 msgr2=0x7f4db0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.766+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4db0076300 0x7f4db0078810 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f4dc41558d0 tx=0x7f4db4008040 comp rx=0 tx=0).stop 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.766+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 msgr2=0x7f4dc4154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.766+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc4154690 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00098c0 tx=0x7f4dc00046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 shutdown_connections 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4dc40d3df0 0x7f4dc4155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4dc40b7ca0 0x7f4dc4155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4db0076300 0x7f4db0078810 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 --2- 172.21.15.46:0/757878825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4dc40b5140 0x7f4dc4154690 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 >> 172.21.15.46:0/757878825 conn(0x7f4dc408d3f0 msgr2=0x7f4dc40ba5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 shutdown_connections 2024-05-09T03:19:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:41.767+0000 7f4dcae9f640 1 -- 172.21.15.46:0/757878825 wait complete. 2024-05-09T03:19:42.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:42.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.314+0000 7f52f3605640 1 Processor -- start 2024-05-09T03:19:42.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.314+0000 7f52f3605640 1 -- start start 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.315+0000 7f52f3605640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec160610 0x7f52ec0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.315+0000 7f52f3605640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52ec0b8130 con 0x7f52ec0b7d20 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.315+0000 7f52f3605640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52ec0d49e0 con 0x7f52ec0d57a0 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.315+0000 7f52f3605640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec0d5bb0 con 0x7f52ec160610 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52f1ba2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52ec0b7d20 0x7f52ec0b9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35120/0 (socket says 172.21.15.46:35120) 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52f1ba2640 1 -- 172.21.15.46:0/3375405287 learned_addr learned my addr 172.21.15.46:0/3375405287 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:42.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52f0ba0640 1 --2- 172.21.15.46:0/3375405287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec160610 0x7f52ec0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:42.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531899009 0 0) 0x7f52e4001490 con 0x7f52ec0b7d20 2024-05-09T03:19:42.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d8003520 con 0x7f52ec0b7d20 2024-05-09T03:19:42.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2989273202 0 0) 0x7f52ec0d49e0 con 0x7f52ec0d57a0 2024-05-09T03:19:42.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52e4001490 con 0x7f52ec0d57a0 2024-05-09T03:19:42.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52f0ba0640 1 -- 172.21.15.46:0/3375405287 >> v1:172.21.15.88:6789/0 conn(0x7f52ec0d57a0 legacy=0x7f52ec0b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52f0ba0640 1 -- 172.21.15.46:0/3375405287 >> v1:172.21.15.46:6789/0 conn(0x7f52ec0b7d20 legacy=0x7f52ec0b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.316+0000 7f52f0ba0640 1 -- 172.21.15.46:0/3375405287 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec0d4cc0 con 0x7f52ec160610 2024-05-09T03:19:42.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.317+0000 7f52f0ba0640 1 --2- 172.21.15.46:0/3375405287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec160610 0x7f52ec0b77e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0097d0 tx=0x7f52dc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b79fa6d1c692c76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:42.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.317+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc037070 con 0x7f52ec160610 2024-05-09T03:19:42.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.317+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc032440 con 0x7f52ec160610 2024-05-09T03:19:42.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.317+0000 7f52e3fff640 1 -- 172.21.15.46:0/3375405287 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc032810 con 0x7f52ec160610 2024-05-09T03:19:42.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 -- 172.21.15.46:0/3375405287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec160610 msgr2=0x7f52ec0b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 --2- 172.21.15.46:0/3375405287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec160610 0x7f52ec0b77e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0097d0 tx=0x7f52dc02a210 comp rx=0 tx=0).stop 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 -- 172.21.15.46:0/3375405287 shutdown_connections 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 --2- 172.21.15.46:0/3375405287 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec160610 0x7f52ec0b77e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 -- 172.21.15.46:0/3375405287 >> 172.21.15.46:0/3375405287 conn(0x7f52ec0743e0 msgr2=0x7f52ec0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 -- 172.21.15.46:0/3375405287 shutdown_connections 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.318+0000 7f52f3605640 1 -- 172.21.15.46:0/3375405287 wait complete. 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.320+0000 7f52f3605640 1 Processor -- start 2024-05-09T03:19:42.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.320+0000 7f52f3605640 1 -- start start 2024-05-09T03:19:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f3605640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0b7d20 0x7f52ec15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f3605640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0d57a0 0x7f52ec1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f3605640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 0x7f52ec153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f3605640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec160a20 con 0x7f52ec160610 2024-05-09T03:19:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f3605640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec0d15b0 con 0x7f52ec0d57a0 2024-05-09T03:19:42.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f3605640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec0d1920 con 0x7f52ec0b7d20 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f1ba2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 0x7f52ec153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f1ba2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 0x7f52ec153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44068/0 (socket says 172.21.15.46:44068) 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f1ba2640 1 -- 172.21.15.46:0/4056213654 learned_addr learned my addr 172.21.15.46:0/4056213654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f0ba0640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0d57a0 0x7f52ec1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.321+0000 7f52f13a1640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0b7d20 0x7f52ec15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f1ba2640 1 -- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0b7d20 msgr2=0x7f52ec15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f1ba2640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0b7d20 0x7f52ec15bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f1ba2640 1 -- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0d57a0 msgr2=0x7f52ec1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f1ba2640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0d57a0 0x7f52ec1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f1ba2640 1 -- 172.21.15.46:0/4056213654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec160c30 con 0x7f52ec160610 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f1ba2640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 0x7f52ec153610 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f52e400cfd0 tx=0x7f52e400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40020b0 con 0x7f52ec160610 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e4012400 con 0x7f52ec160610 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40127f0 con 0x7f52ec160610 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ec1fa720 con 0x7f52ec160610 2024-05-09T03:19:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.322+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ec1face0 con 0x7f52ec160610 2024-05-09T03:19:42.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.323+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ec1fa490 con 0x7f52ec160610 2024-05-09T03:19:42.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.324+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f52e4012990 con 0x7f52ec160610 2024-05-09T03:19:42.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.324+0000 7f52e1ffb640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52d00764a0 0x7f52d00789b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:42.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.325+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f52e40986f0 con 0x7f52ec160610 2024-05-09T03:19:42.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.325+0000 7f52f13a1640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52d00764a0 0x7f52d00789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:42.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.325+0000 7f52f13a1640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52d00764a0 0x7f52d00789b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f52e8001120 tx=0x7f52e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:42.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.329+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f52ec0b8130 con 0x7f52ec160610 2024-05-09T03:19:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.561+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52ec06a270 con 0x7f52ec160610 2024-05-09T03:19:42.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.562+0000 7f52e1ffb640 1 -- 172.21.15.46:0/4056213654 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 296 v296) ==== 95+0+5178 (secure 0 0 0) 0x7f52e4060ef0 con 0x7f52ec160610 2024-05-09T03:19:42.563 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:42.563 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":296,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":296,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:41.767556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9879":{"gid":9879,"name":"c","rank":0,"incarnation":296,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/2209678569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2209678569},{"type":"v1","addr":"172.21.15.46:6834","nonce":2209678569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:42.563 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 296 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52d00764a0 msgr2=0x7f52d00789b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52d00764a0 0x7f52d00789b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f52e8001120 tx=0x7f52e8009040 comp rx=0 tx=0).stop 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 msgr2=0x7f52ec153610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 0x7f52ec153610 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f52e400cfd0 tx=0x7f52e400a430 comp rx=0 tx=0).stop 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 shutdown_connections 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52d00764a0 0x7f52d00789b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ec160610 0x7f52ec153610 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec0d57a0 0x7f52ec1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 --2- 172.21.15.46:0/4056213654 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0b7d20 0x7f52ec15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 >> 172.21.15.46:0/4056213654 conn(0x7f52ec0743e0 msgr2=0x7f52ec0bae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 shutdown_connections 2024-05-09T03:19:42.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:42.564+0000 7f52f3605640 1 -- 172.21.15.46:0/4056213654 wait complete. 2024-05-09T03:19:42.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 296, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:19:41.767556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 117, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9879}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9879': {'gid': 9879, 'name': 'c', 'rank': 0, 'incarnation': 296, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/2209678569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2209678569}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2209678569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:19:42.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9879:up:replay 2024-05-09T03:19:42.576 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:19:46.752 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:46.751+0000 7ff2ab77d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:19:47.791 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:47.791+0000 7fafc24c3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T03:19:47.980 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.108+0000 7fe552239640 1 Processor -- start 2024-05-09T03:19:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.108+0000 7fe552239640 1 -- start start 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe552239640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe54c08cd60 0x7fe54c0d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe552239640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c0d1c80 0x7fe54c0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe552239640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe54c0d2090 0x7fe54c0cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe552239640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe54c0b5340 con 0x7fe54c0d2090 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe552239640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe54c0d4900 con 0x7fe54c08cd60 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe552239640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe54c0b4fd0 con 0x7fe54c0d1c80 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe54c0d2090 0x7fe54c0cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe54c0d2090 0x7fe54c0cf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60794/0 (socket says 172.21.15.46:60794) 2024-05-09T03:19:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 -- 172.21.15.46:0/1107428482 learned_addr learned my addr 172.21.15.46:0/1107428482 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54b7fe640 1 --2- 172.21.15.46:0/1107428482 >> v2:172.21.15.88:3300/0 conn(0x7fe54c08cd60 0x7fe54c0d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54affd640 1 --2- 172.21.15.46:0/1107428482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c0d1c80 0x7fe54c0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 -- 172.21.15.46:0/1107428482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c0d1c80 msgr2=0x7fe54c0cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 --2- 172.21.15.46:0/1107428482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c0d1c80 0x7fe54c0cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 -- 172.21.15.46:0/1107428482 >> v2:172.21.15.88:3300/0 conn(0x7fe54c08cd60 msgr2=0x7fe54c0d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 --2- 172.21.15.46:0/1107428482 >> v2:172.21.15.88:3300/0 conn(0x7fe54c08cd60 0x7fe54c0d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.110+0000 7fe54bfff640 1 -- 172.21.15.46:0/1107428482 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54c0d2c00 con 0x7fe54c0d2090 2024-05-09T03:19:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.111+0000 7fe54bfff640 1 --2- 172.21.15.46:0/1107428482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c0cf7f0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fe544009e50 tx=0x7fe54402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f41582598ff5f1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.111+0000 7fe54a7fc640 1 -- 172.21.15.46:0/1107428482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544036070 con 0x7fe54c0d2090 2024-05-09T03:19:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.111+0000 7fe54a7fc640 1 -- 172.21.15.46:0/1107428482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe544031600 con 0x7fe54c0d2090 2024-05-09T03:19:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.111+0000 7fe54a7fc640 1 -- 172.21.15.46:0/1107428482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5440319d0 con 0x7fe54c0d2090 2024-05-09T03:19:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 -- 172.21.15.46:0/1107428482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 msgr2=0x7fe54c0cf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 --2- 172.21.15.46:0/1107428482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c0cf7f0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fe544009e50 tx=0x7fe54402a6c0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 -- 172.21.15.46:0/1107428482 shutdown_connections 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 --2- 172.21.15.46:0/1107428482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c0cf7f0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 --2- 172.21.15.46:0/1107428482 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c0d1c80 0x7fe54c0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 --2- 172.21.15.46:0/1107428482 >> v2:172.21.15.88:3300/0 conn(0x7fe54c08cd60 0x7fe54c0d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 -- 172.21.15.46:0/1107428482 >> 172.21.15.46:0/1107428482 conn(0x7fe54c0d3c00 msgr2=0x7fe54c0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 -- 172.21.15.46:0/1107428482 shutdown_connections 2024-05-09T03:19:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.112+0000 7fe552239640 1 -- 172.21.15.46:0/1107428482 wait complete. 2024-05-09T03:19:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.114+0000 7fe552239640 1 Processor -- start 2024-05-09T03:19:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.114+0000 7fe552239640 1 -- start start 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe552239640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c08cd60 0x7fe54c15ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe552239640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe54c0d1c80 0x7fe54c1f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe552239640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c1f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe552239640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c0b92b0 con 0x7fe54c0d2090 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe552239640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe54c0b9130 con 0x7fe54c0d1c80 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe552239640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe54c162310 con 0x7fe54c08cd60 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c1f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c1f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60806/0 (socket says 172.21.15.46:60806) 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 -- 172.21.15.46:0/416528584 learned_addr learned my addr 172.21.15.46:0/416528584 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54b7fe640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c08cd60 0x7fe54c15ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54affd640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe54c0d1c80 0x7fe54c1f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 -- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c08cd60 msgr2=0x7fe54c15ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c08cd60 0x7fe54c15ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 -- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe54c0d1c80 msgr2=0x7fe54c1f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe54c0d1c80 0x7fe54c1f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 -- 172.21.15.46:0/416528584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54c15fa20 con 0x7fe54c0d2090 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.116+0000 7fe54bfff640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c1f4d00 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fe544034040 tx=0x7fe544004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.117+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54403f020 con 0x7fe54c0d2090 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.117+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe544037040 con 0x7fe54c0d2090 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.117+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54c15fd00 con 0x7fe54c0d2090 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.117+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe544036070 con 0x7fe54c0d2090 2024-05-09T03:19:48.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.117+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54c0b0d00 con 0x7fe54c0d2090 2024-05-09T03:19:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.118+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe54c06a270 con 0x7fe54c0d2090 2024-05-09T03:19:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.119+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe544042020 con 0x7fe54c0d2090 2024-05-09T03:19:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.119+0000 7fe533fff640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5380762d0 0x7fe5380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.119+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7fe5440b8710 con 0x7fe54c0d2090 2024-05-09T03:19:48.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.119+0000 7fe54b7fe640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5380762d0 0x7fe5380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.120+0000 7fe54b7fe640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5380762d0 0x7fe5380787e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe54c074930 tx=0x7fe53c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.123+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe544049030 con 0x7fe54c0d2090 2024-05-09T03:19:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.356+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe54c0caa90 con 0x7fe54c0d2090 2024-05-09T03:19:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.357+0000 7fe533fff640 1 -- 172.21.15.46:0/416528584 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 300 v300) ==== 95+0+5185 (secure 0 0 0) 0x7fe544080f90 con 0x7fe54c0d2090 2024-05-09T03:19:48.358 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:48.358 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":300,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":300,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:47.780947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9879":{"gid":9879,"name":"c","rank":0,"incarnation":296,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6834/2209678569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2209678569},{"type":"v1","addr":"172.21.15.46:6834","nonce":2209678569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9879,"qdb_cluster":[9879]},"id":6}]} 2024-05-09T03:19:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 300 2024-05-09T03:19:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5380762d0 msgr2=0x7fe5380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5380762d0 0x7fe5380787e0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe54c074930 tx=0x7fe53c009190 comp rx=0 tx=0).stop 2024-05-09T03:19:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 msgr2=0x7fe54c1f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c1f4d00 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fe544034040 tx=0x7fe544004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 shutdown_connections 2024-05-09T03:19:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe5380762d0 0x7fe5380787e0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe54c0d2090 0x7fe54c1f4d00 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe54c0d1c80 0x7fe54c1f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 --2- 172.21.15.46:0/416528584 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe54c08cd60 0x7fe54c15ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 >> 172.21.15.46:0/416528584 conn(0x7fe54c0d3c00 msgr2=0x7fe54c15cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 shutdown_connections 2024-05-09T03:19:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.359+0000 7fe552239640 1 -- 172.21.15.46:0/416528584 wait complete. 2024-05-09T03:19:48.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 300, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:19:47.780947+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 117, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9879}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9879': {'gid': 9879, 'name': 'c', 'rank': 0, 'incarnation': 296, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.46:6834/2209678569', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2209678569}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2209678569}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9879, 'qdb_cluster': [9879]} 2024-05-09T03:19:48.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:19:48.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:19:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.486+0000 7f52f10f9640 1 Processor -- start 2024-05-09T03:19:48.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.486+0000 7f52f10f9640 1 -- start start 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.487+0000 7f52f10f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 0x7f52ec0d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.487+0000 7f52f10f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52ec0d3180 0x7f52ec1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.487+0000 7f52f10f9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52ec164900 con 0x7f52ec163eb0 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.487+0000 7f52f10f9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f52ec0d4bb0 con 0x7f52ec0d3180 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.487+0000 7f52f10f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec164be0 con 0x7f52ec0bb770 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52e3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52ec163eb0 0x7f52ec0d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41026/0 (socket says 172.21.15.46:41026) 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52e3fff640 1 -- 172.21.15.46:0/2839532874 learned_addr learned my addr 172.21.15.46:0/2839532874 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:48.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52eb59e640 1 --2- 172.21.15.46:0/2839532874 >> v2:172.21.15.88:3300/0 conn(0x7f52ec0d3180 0x7f52ec1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 0x7f52ec0d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 860442193 0 0) 0x7f52ec164900 con 0x7f52ec163eb0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52d8003520 con 0x7f52ec163eb0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3010644305 0 0) 0x7f52d8003520 con 0x7f52ec163eb0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 msgr2=0x7f52ec0d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 0x7f52ec0d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 >> v2:172.21.15.88:3300/0 conn(0x7f52ec0d3180 msgr2=0x7f52ec1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 --2- 172.21.15.46:0/2839532874 >> v2:172.21.15.88:3300/0 conn(0x7f52ec0d3180 0x7f52ec1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec0d4840 con 0x7f52ec163eb0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f52d40016f0 con 0x7f52ec163eb0 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 >> v1:172.21.15.46:6789/0 conn(0x7f52ec163eb0 legacy=0x7f52ec0d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52d8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d8004800 0x7f52d8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 0x7f52d8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec0d4bb0 con 0x7f52d8003a00 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec164be0 con 0x7f52d8004800 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52d800ba70 con 0x7f52ec0bb770 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52d8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.488+0000 7f52e3fff640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d8004800 0x7f52d8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52eb59e640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 0x7f52d8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52ead9d640 1 -- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 msgr2=0x7f52d8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec0bb770 0x7f52d8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52ead9d640 1 -- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d8004800 msgr2=0x7f52d8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d8004800 0x7f52d8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52ead9d640 1 -- 172.21.15.46:0/2839532874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d8003520 con 0x7f52d8003a00 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.489+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52d8003e70 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009a90 tx=0x7f52d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9895b782ba8a54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc036070 con 0x7f52d8003a00 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc031400 con 0x7f52d8003a00 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52ea59c640 1 -- 172.21.15.46:0/2839532874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc009690 con 0x7f52d8003a00 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52f10f9640 1 -- 172.21.15.46:0/2839532874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 msgr2=0x7f52d8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52f10f9640 1 --2- 172.21.15.46:0/2839532874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52d8003e70 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009a90 tx=0x7f52d8005830 comp rx=0 tx=0).stop 2024-05-09T03:19:48.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52eb59e640 1 -- 172.21.15.46:0/2839532874 reap_dead start 2024-05-09T03:19:48.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52f10f9640 1 -- 172.21.15.46:0/2839532874 shutdown_connections 2024-05-09T03:19:48.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52f10f9640 1 -- 172.21.15.46:0/2839532874 >> 172.21.15.46:0/2839532874 conn(0x7f52ec06e370 msgr2=0x7f52ec06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52f10f9640 1 -- 172.21.15.46:0/2839532874 shutdown_connections 2024-05-09T03:19:48.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.490+0000 7f52f10f9640 1 -- 172.21.15.46:0/2839532874 wait complete. 2024-05-09T03:19:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.492+0000 7f52f10f9640 1 Processor -- start 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.492+0000 7f52f10f9640 1 -- start start 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52f10f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52ec14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52f10f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec14dd30 0x7f52ec14f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52f10f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec14e140 0x7f52ec14f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52f10f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec0d0d40 con 0x7f52d8003a00 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52f10f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec0d0bc0 con 0x7f52ec14dd30 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52f10f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec0d5040 con 0x7f52ec14e140 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52ec14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52ec14d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60818/0 (socket says 172.21.15.46:60818) 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 -- 172.21.15.46:0/1258651913 learned_addr learned my addr 172.21.15.46:0/1258651913 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52e3fff640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec14dd30 0x7f52ec14f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52eb59e640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec14e140 0x7f52ec14f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 -- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec14e140 msgr2=0x7f52ec14f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec14e140 0x7f52ec14f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 -- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec14dd30 msgr2=0x7f52ec14f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec14dd30 0x7f52ec14f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 -- 172.21.15.46:0/1258651913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec0cafe0 con 0x7f52d8003a00 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.494+0000 7f52ead9d640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52ec14d7f0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009bb0 tx=0x7f52dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.495+0000 7f52e37fe640 1 -- 172.21.15.46:0/1258651913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc040070 con 0x7f52d8003a00 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.495+0000 7f52e37fe640 1 -- 172.21.15.46:0/1258651913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc03b460 con 0x7f52d8003a00 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.495+0000 7f52e37fe640 1 -- 172.21.15.46:0/1258651913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc036040 con 0x7f52d8003a00 2024-05-09T03:19:48.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.495+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ec0cb2c0 con 0x7f52d8003a00 2024-05-09T03:19:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.495+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ec14ea50 con 0x7f52d8003a00 2024-05-09T03:19:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.497+0000 7f52e37fe640 1 -- 172.21.15.46:0/1258651913 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f52dc043070 con 0x7f52d8003a00 2024-05-09T03:19:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.497+0000 7f52e37fe640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc0762d0 0x7f52cc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.498+0000 7f52e37fe640 1 -- 172.21.15.46:0/1258651913 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f52dc0b8930 con 0x7f52d8003a00 2024-05-09T03:19:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.498+0000 7f52e3fff640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc0762d0 0x7f52cc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.498+0000 7f52f7faa740 1 Processor -- start 2024-05-09T03:19:48.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.498+0000 7f52f7faa740 1 -- start start 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.498+0000 7f52e3fff640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc0762d0 0x7f52cc0787e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f52ec15e7f0 tx=0x7f52d40078f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981e010eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e02bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc20590 0x55981e02c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52f7faa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55981df494e0 con 0x55981dc41270 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55981dee4ff0 con 0x55981dc183b0 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55981df454a0 con 0x55981dc20590 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981e010eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981e010eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60828/0 (socket says 172.21.15.46:60828) 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 -- 172.21.15.46:0/162457589 learned_addr learned my addr 172.21.15.46:0/162457589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52e3fff640 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e02bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52eb59e640 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc20590 0x55981e02c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 -- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc20590 msgr2=0x55981e02c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc20590 0x55981e02c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 -- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 msgr2=0x55981e02bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e02bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 -- 172.21.15.46:0/162457589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981decb720 con 0x55981dc41270 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52ead9d640 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981e010eb0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f52dc07d1b0 tx=0x7f52dc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adf575e6e45f037 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52e0ff9640 1 -- 172.21.15.46:0/162457589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc04a040 con 0x55981dc41270 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52e0ff9640 1 -- 172.21.15.46:0/162457589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc085430 con 0x55981dc41270 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.500+0000 7f52e0ff9640 1 -- 172.21.15.46:0/162457589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc089e50 con 0x55981dc41270 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 -- 172.21.15.46:0/162457589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 msgr2=0x55981e010eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981e010eb0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f52dc07d1b0 tx=0x7f52dc009750 comp rx=0 tx=0).stop 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 -- 172.21.15.46:0/162457589 shutdown_connections 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc20590 0x55981e02c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e02bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 --2- 172.21.15.46:0/162457589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981e010eb0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 -- 172.21.15.46:0/162457589 >> 172.21.15.46:0/162457589 conn(0x55981dc353c0 msgr2=0x55981dfe6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.501+0000 7f52f7faa740 1 -- 172.21.15.46:0/162457589 shutdown_connections 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.502+0000 7f52f7faa740 1 -- 172.21.15.46:0/162457589 wait complete. 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.502+0000 7f52f7faa740 1 -- start start 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.502+0000 7f52f7faa740 1 Processor -- start 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981dff90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981e028a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc41270 0x55981e00b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55981df454a0 con 0x55981dc20590 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55981ded0760 con 0x55981dc41270 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55981df494e0 con 0x55981dc183b0 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52f7faa740 10 client.0 _ll_register_callbacks cb 0x55981def34b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:48.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981e028a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981e028a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60838/0 (socket says 172.21.15.46:60838) 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 -- 172.21.15.46:0/2793658853 learned_addr learned my addr 172.21.15.46:0/2793658853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981dff90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52eb59e640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc41270 0x55981e00b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 -- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 msgr2=0x55981dff90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981dff90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 -- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc41270 msgr2=0x55981e00b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc41270 0x55981e00b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.504+0000 7f52e3fff640 1 -- 172.21.15.46:0/2793658853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981df11bd0 con 0x55981dc20590 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52e3fff640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981e028a10 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f52d4001eb0 tx=0x7f52ec164ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a101dab67d774e99 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52c9ffb640 1 -- 172.21.15.46:0/2793658853 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4023440 con 0x55981dc20590 2024-05-09T03:19:48.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52c9ffb640 1 -- 172.21.15.46:0/2793658853 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f52d40235e0 con 0x55981dc20590 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52dc085430 con 0x55981dc20590 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52c9ffb640 1 -- 172.21.15.46:0/2793658853 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=300) ==== 24+0+0 (secure 0 0 0) 0x7f52d4031050 con 0x55981dc20590 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 10 client.9969 fetch_fsmap learned FSMap version 300 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=300,monmap=2+}) -- 0x55981def7280 con 0x55981dc20590 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52c9ffb640 1 -- 172.21.15.46:0/2793658853 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc085430 con 0x55981dc20590 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52c9ffb640 1 -- 172.21.15.46:0/2793658853 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 300) ==== 3494+0+0 (secure 0 0 0) 0x7f52d40237e0 con 0x55981dc20590 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 10 client.9969 fetch_fsmap finished waiting for FSMap version 300 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 10 client.9969 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 20 client.9969 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dbffe40 0x55981dffb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 4 client.9969 mds_command: new command op to 9879 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 --> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55981df570f0 con 0x55981dbffe40 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.505+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dbffe40 0x55981dffb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.506+0000 7f52c9ffb640 10 client.9969 ms_handle_connect on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:48.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.506+0000 7f52ead9d640 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dbffe40 0x55981dffb150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d98c4debae71b9dc server_cookie=fcb2d4ab0e89116b in_seq=0 out_seq=0 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.506+0000 7f52c9ffb640 1 -- 172.21.15.46:0/2793658853 <== mds.0 v2:172.21.15.46:6832/2209678569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55981df570f0 con 0x55981dbffe40 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.506+0000 7f52c9ffb640 10 client.9969 handle_command_reply: tid=0 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.506+0000 7f52f7faa740 1 client.9969 shutdown 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 msgr2=0x55981e028a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981e028a10 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f52d4001eb0 tx=0x7f52ec164ad0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 shutdown_connections 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dbffe40 0x55981dffb150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc41270 0x55981e00b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52c9ffb640 0 client.9969 ms_handle_reset on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981e028a10 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 --2- 172.21.15.46:0/2793658853 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981dff90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 >> 172.21.15.46:0/2793658853 conn(0x55981dc353c0 msgr2=0x55981dfe6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 shutdown_connections 2024-05-09T03:19:48.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 1 -- 172.21.15.46:0/2793658853 wait complete. 2024-05-09T03:19:48.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.507+0000 7f52f7faa740 20 client.9969 trim_cache size 0 max 16384 2024-05-09T03:19:48.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.526+0000 7f52f7faa740 1 Processor -- start 2024-05-09T03:19:48.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.526+0000 7f52f7faa740 1 -- start start 2024-05-09T03:19:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e03a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981dffda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc41270 0x55981dffdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52f7faa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec203de0 con 0x55981dc20590 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec203140 con 0x55981dc183b0 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec203790 con 0x55981dc41270 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981dffda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52e3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981dffda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60844/0 (socket says 172.21.15.46:60844) 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52e3fff640 1 -- 172.21.15.46:0/3081284084 learned_addr learned my addr 172.21.15.46:0/3081284084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52e3fff640 1 -- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc41270 msgr2=0x55981dffdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.527+0000 7f52e3fff640 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc41270 0x55981dffdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52ead9d640 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e03a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52e3fff640 1 -- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 msgr2=0x55981e03a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52e3fff640 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e03a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52e3fff640 1 -- 172.21.15.46:0/3081284084 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981dcac850 con 0x55981dc20590 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52ead9d640 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e03a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52e3fff640 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981dffda10 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f52d40018e0 tx=0x7f52d4023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4755b9d37ce04168 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52cbfff640 1 -- 172.21.15.46:0/3081284084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4023640 con 0x55981dc20590 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52cbfff640 1 -- 172.21.15.46:0/3081284084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d4028070 con 0x55981dc20590 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.528+0000 7f52cbfff640 1 -- 172.21.15.46:0/3081284084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d402d450 con 0x55981dc20590 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 -- 172.21.15.46:0/3081284084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 msgr2=0x55981dffda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981dffda10 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f52d40018e0 tx=0x7f52d4023000 comp rx=0 tx=0).stop 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 -- 172.21.15.46:0/3081284084 shutdown_connections 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc41270 0x55981dffdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc20590 0x55981dffda10 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 --2- 172.21.15.46:0/3081284084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc183b0 0x55981e03a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 -- 172.21.15.46:0/3081284084 >> 172.21.15.46:0/3081284084 conn(0x55981dc353c0 msgr2=0x55981e00b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 -- 172.21.15.46:0/3081284084 shutdown_connections 2024-05-09T03:19:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 -- 172.21.15.46:0/3081284084 wait complete. 2024-05-09T03:19:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.529+0000 7f52f7faa740 1 -- start start 2024-05-09T03:19:48.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.530+0000 7f52f7faa740 1 Processor -- start 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981e0aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc20590 0x55981e0a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981dfe0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55981dff9870 con 0x55981dc41270 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55981dee4ff0 con 0x55981dc20590 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55981dfff5d0 con 0x55981dc183b0 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52f7faa740 10 client.0 _ll_register_callbacks cb 0x7f52d4019040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:48.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981dfe0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981dfe0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60850/0 (socket says 172.21.15.46:60850) 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 -- 172.21.15.46:0/1714619369 learned_addr learned my addr 172.21.15.46:0/1714619369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52ead9d640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981e0aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52e3fff640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc20590 0x55981e0a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 -- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 msgr2=0x55981e0aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981e0aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 -- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc20590 msgr2=0x55981e0a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc20590 0x55981e0a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.531+0000 7f52eb59e640 1 -- 172.21.15.46:0/1714619369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981dd236b0 con 0x55981dc41270 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52eb59e640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981dfe0300 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f52ec08e340 tx=0x7f52e4004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29317ecfbc338408 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:48.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52c9ffb640 1 -- 172.21.15.46:0/1714619369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e40189e0 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52c9ffb640 1 -- 172.21.15.46:0/1714619369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f52e4018b80 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52d4028070 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52c9ffb640 1 -- 172.21.15.46:0/1714619369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=300) ==== 24+0+0 (secure 0 0 0) 0x7f52e4018da0 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 10 client.9975 fetch_fsmap learned FSMap version 300 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=300,monmap=2+}) -- 0x55981df05630 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52c9ffb640 1 -- 172.21.15.46:0/1714619369 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d4028070 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52c9ffb640 1 -- 172.21.15.46:0/1714619369 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 300) ==== 3494+0+0 (secure 0 0 0) 0x7f52e4022430 con 0x55981dc41270 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 10 client.9975 fetch_fsmap finished waiting for FSMap version 300 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 10 client.9975 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 20 client.9975 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dfe4de0 0x55981dfe6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 4 client.9975 mds_command: new command op to 9879 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 --> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] -- command(tid 0: {"prefix": "status"}) -- 0x55981df50f60 con 0x55981dfe4de0 2024-05-09T03:19:48.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.532+0000 7f52ead9d640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dfe4de0 0x55981dfe6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52c9ffb640 10 client.9975 ms_handle_connect on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52ead9d640 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dfe4de0 0x55981dfe6200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf4475b8f20b1792 server_cookie=83f4f2737ce182c9 in_seq=0 out_seq=0 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52c9ffb640 1 -- 172.21.15.46:0/1714619369 <== mds.0 v2:172.21.15.46:6832/2209678569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55981df50f60 con 0x55981dfe4de0 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52c9ffb640 10 client.9975 handle_command_reply: tid=0 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 client.9975 shutdown 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 msgr2=0x55981dfe0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981dfe0300 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f52ec08e340 tx=0x7f52e4004840 comp rx=0 tx=0).stop 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 shutdown_connections 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x55981dfe4de0 0x55981dfe6200 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55981dc41270 0x55981dfe0300 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55981dc20590 0x55981e0a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52c9ffb640 0 client.9975 ms_handle_reset on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:48.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 --2- 172.21.15.46:0/1714619369 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55981dc183b0 0x55981e0aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 >> 172.21.15.46:0/1714619369 conn(0x55981dc353c0 msgr2=0x55981e02d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 shutdown_connections 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 1 -- 172.21.15.46:0/1714619369 wait complete. 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.533+0000 7f52f7faa740 20 client.9975 trim_cache size 0 max 16384 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9879, 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:19:48.541 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:19:48.542 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7645880409999997, 2024-05-09T03:19:48.542 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 300, 2024-05-09T03:19:48.542 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 117, 2024-05-09T03:19:48.542 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 117, 2024-05-09T03:19:48.542 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12.723623504000001 2024-05-09T03:19:48.542 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc0762d0 msgr2=0x7f52cc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc0762d0 0x7f52cc0787e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f52ec15e7f0 tx=0x7f52d40078f0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 msgr2=0x7f52ec14d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52ec14d7f0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009bb0 tx=0x7f52dc004730 comp rx=0 tx=0).stop 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 shutdown_connections 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52ec14e140 0x7f52ec14f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52ec14dd30 0x7f52ec14f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52cc0762d0 0x7f52cc0787e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 --2- 172.21.15.46:0/1258651913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d8003a00 0x7f52ec14d7f0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 >> 172.21.15.46:0/1258651913 conn(0x7f52ec06e370 msgr2=0x7f52ec0b7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 shutdown_connections 2024-05-09T03:19:48.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.534+0000 7f52f10f9640 1 -- 172.21.15.46:0/1258651913 wait complete. 2024-05-09T03:19:48.546 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:19:48.546 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap key 1.00000000:subdir_head', proceeding to ls 2024-05-09T03:19:48.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:19:48.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.660+0000 7f036c13f640 1 Processor -- start 2024-05-09T03:19:48.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.660+0000 7f036c13f640 1 -- start start 2024-05-09T03:19:48.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.661+0000 7f036c13f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0364160610 0x7f03640b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.661+0000 7f036c13f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03640b7d20 0x7f03640b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.661+0000 7f036c13f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03640b8130 con 0x7f03640d57a0 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.661+0000 7f036c13f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f03640d5bb0 con 0x7f0364160610 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.661+0000 7f036c13f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f03640d49e0 con 0x7f03640b7d20 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f0369edb640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f03640d57a0 0x7f03640b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41030/0 (socket says 172.21.15.46:41030) 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f0369edb640 1 -- 172.21.15.46:0/1559532148 learned_addr learned my addr 172.21.15.46:0/1559532148 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f03696da640 1 --2- 172.21.15.46:0/1559532148 >> v2:172.21.15.88:3300/0 conn(0x7f0364160610 0x7f03640b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f036a6dc640 1 --2- 172.21.15.46:0/1559532148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03640b7d20 0x7f03640b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f0368ed9640 1 -- 172.21.15.46:0/1559532148 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585198558 0 0) 0x7f0360001110 con 0x7f03640d57a0 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f0368ed9640 1 -- 172.21.15.46:0/1559532148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0350003520 con 0x7f03640d57a0 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f036a6dc640 1 -- 172.21.15.46:0/1559532148 >> v2:172.21.15.88:3300/0 conn(0x7f0364160610 msgr2=0x7f03640b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f036a6dc640 1 --2- 172.21.15.46:0/1559532148 >> v2:172.21.15.88:3300/0 conn(0x7f0364160610 0x7f03640b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f036a6dc640 1 -- 172.21.15.46:0/1559532148 >> v1:172.21.15.46:6789/0 conn(0x7f03640d57a0 legacy=0x7f03640b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f0368ed9640 1 -- 172.21.15.46:0/1559532148 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1111442971 0 0) 0x7f0350003520 con 0x7f03640d57a0 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f036a6dc640 1 -- 172.21.15.46:0/1559532148 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03640d4cc0 con 0x7f03640b7d20 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.662+0000 7f03696da640 1 --2- 172.21.15.46:0/1559532148 >> v2:172.21.15.88:3300/0 conn(0x7f0364160610 0x7f03640b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.663+0000 7f036a6dc640 1 --2- 172.21.15.46:0/1559532148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03640b7d20 0x7f03640b9450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f035c009d90 tx=0x7f035c02a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5f082a2aa62fa87 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.663+0000 7f0368ed9640 1 -- 172.21.15.46:0/1559532148 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03600018c0 con 0x7f03640b7d20 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.663+0000 7f0368ed9640 1 -- 172.21.15.46:0/1559532148 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f035c007540 con 0x7f03640b7d20 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.663+0000 7f0368ed9640 1 -- 172.21.15.46:0/1559532148 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c0078f0 con 0x7f03640b7d20 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 -- 172.21.15.46:0/1559532148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03640b7d20 msgr2=0x7f03640b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 --2- 172.21.15.46:0/1559532148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03640b7d20 0x7f03640b9450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f035c009d90 tx=0x7f035c02a5f0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 -- 172.21.15.46:0/1559532148 shutdown_connections 2024-05-09T03:19:48.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 --2- 172.21.15.46:0/1559532148 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f03640b7d20 0x7f03640b9450 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 --2- 172.21.15.46:0/1559532148 >> v2:172.21.15.88:3300/0 conn(0x7f0364160610 0x7f03640b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 -- 172.21.15.46:0/1559532148 >> 172.21.15.46:0/1559532148 conn(0x7f03640743e0 msgr2=0x7f03640d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 -- 172.21.15.46:0/1559532148 shutdown_connections 2024-05-09T03:19:48.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.664+0000 7f036c13f640 1 -- 172.21.15.46:0/1559532148 wait complete. 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.666+0000 7f036c13f640 1 Processor -- start 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.666+0000 7f036c13f640 1 -- start start 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036c13f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 0x7f03640b7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036c13f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03640d57a0 0x7f03640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036c13f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0364160610 0x7f0364144240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036c13f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f03640d15b0 con 0x7f03640b7d20 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036c13f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0364160a20 con 0x7f03640d57a0 2024-05-09T03:19:48.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036c13f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f03640d1920 con 0x7f0364160610 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 0x7f03640b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 0x7f03640b7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60854/0 (socket says 172.21.15.46:60854) 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 -- 172.21.15.46:0/1255371132 learned_addr learned my addr 172.21.15.46:0/1255371132 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f03696da640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03640d57a0 0x7f03640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f036a6dc640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0364160610 0x7f0364144240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 -- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0364160610 msgr2=0x7f0364144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0364160610 0x7f0364144240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 -- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03640d57a0 msgr2=0x7f03640b6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03640d57a0 0x7f03640b6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.667+0000 7f0369edb640 1 -- 172.21.15.46:0/1255371132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036415cb10 con 0x7f03640b7d20 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.668+0000 7f0369edb640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 0x7f03640b7070 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f036000c500 tx=0x7f036000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.668+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360005ce0 con 0x7f03640b7d20 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.668+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0360007070 con 0x7f03640b7d20 2024-05-09T03:19:48.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.668+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036415cd80 con 0x7f03640b7d20 2024-05-09T03:19:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.668+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360007480 con 0x7f03640b7d20 2024-05-09T03:19:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.668+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036415d340 con 0x7f03640b7d20 2024-05-09T03:19:48.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.669+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f036406a270 con 0x7f03640b7d20 2024-05-09T03:19:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.670+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f036001b400 con 0x7f03640b7d20 2024-05-09T03:19:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.670+0000 7f035a7fc640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03480762d0 0x7f03480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.670+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f0360097d80 con 0x7f03640b7d20 2024-05-09T03:19:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.671+0000 7f03696da640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03480762d0 0x7f03480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:48.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.671+0000 7f03696da640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03480762d0 0x7f03480787e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f03640d3d40 tx=0x7f0354009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:48.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.675+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f03640b8130 con 0x7f03640b7d20 2024-05-09T03:19:48.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.906+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f036415e620 con 0x7f03640b7d20 2024-05-09T03:19:48.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.906+0000 7f035a7fc640 1 -- 172.21.15.46:0/1255371132 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v300) ==== 53+0+73 (secure 0 0 0) 0x7f03600604f0 con 0x7f03640b7d20 2024-05-09T03:19:48.907 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.908+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03480762d0 msgr2=0x7f03480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.908+0000 7f036c13f640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03480762d0 0x7f03480787e0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f03640d3d40 tx=0x7f0354009190 comp rx=0 tx=0).stop 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 msgr2=0x7f03640b7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 0x7f03640b7070 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f036000c500 tx=0x7f036000c9c0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 shutdown_connections 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0364160610 0x7f0364144240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03640d57a0 0x7f03640b6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f03480762d0 0x7f03480787e0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 --2- 172.21.15.46:0/1255371132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03640b7d20 0x7f03640b7070 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:48.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 >> 172.21.15.46:0/1255371132 conn(0x7f03640743e0 msgr2=0x7f03640ba9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 shutdown_connections 2024-05-09T03:19:48.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:48.909+0000 7f036c13f640 1 -- 172.21.15.46:0/1255371132 wait complete. 2024-05-09T03:19:48.920 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:19:48.920 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:19:48.920 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:19:48.920 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:19:48.921 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:48.921 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:19:48.921 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:19:48.921 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:19:48.921 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:19:48.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:48.921 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:19:48.937 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:19:48.937 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:19:48.937 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:19:48.938 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41929sec preferred_lft 41929sec 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:19:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:19:48.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:48.940 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:19:49.001 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:19:49.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:49.003 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:19:49.060 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:19:49.060 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:19:49.060 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:19:49.060 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:19:49.060 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:19:49.061 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:19:49.061 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:19:49.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:19:49.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:19:49.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:19:49.061 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:19:49.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:19:49.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:19:49.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:49.191 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:19:49.208 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:19:49.209 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:19:49.209 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:19:49.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:49.209 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:19:49.274 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:19:49.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:19:49.274 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:49.330 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:19:49.331 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:19:49.331 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:49.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:19:49.457 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:19:49.514 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:19:49.515 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:19:49.516 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:49.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:49.517 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:19:49.570 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:19:49.571 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:19:49.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:19:49.639 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:19:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:19:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:19:49.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:49.700 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:19:49.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:19:49.707+0000 7f12071ff480 -1 init, newargv = 0x7f11e8001950 newargc=15 2024-05-09T03:19:49.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[82120]: starting ceph client 2024-05-09T03:19:49.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[82120]: starting fuse 2024-05-09T03:19:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:19:49.724 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:19:49.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:19:49.724 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:19:49.724 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:19:49.724 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:19:49.724 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:19:49.725 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:19:49.726 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:19:49.726 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:19:49.726 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:19:49.726 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:49.726 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:19:49.726 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:19:49.847 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.82120.asok 2024-05-09T03:19:49.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:19:49.857 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.82120.asok 2024-05-09T03:19:49.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:49.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.82120.asok status 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:19:50.004 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "82120", 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 9987, 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 9987 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3603481363 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:19:50.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3603481363 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.9987 192.168.0.1:0/3603481363", 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3603481363", 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 300, 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 117, 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:19:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:19:50.007 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:19:50.007 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:19:50.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:50.013 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:19:50.013 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:50.029 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:19:50.029 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:50.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:50.030 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> import json 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> import sys 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> try: 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/./subdir'"'"') 2024-05-09T03:19:50.096 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> sys.exit(e.errno) 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> print(json.dumps( 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> dict([(a, getattr(s, a)) for a in attrs]), 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> indent=2)) 2024-05-09T03:19:50.097 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:19:50.139 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:19:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-05-09T03:19:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-05-09T03:19:50.216 INFO:tasks.cephfs.test_damage:Result: EIO on client 2024-05-09T03:19:50.216 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.370+0000 7f4f29f27640 1 Processor -- start 2024-05-09T03:19:50.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.370+0000 7f4f29f27640 1 -- start start 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.371+0000 7f4f29f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f241613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f29f27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f240d4800 0x7f4f240d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f29f27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f240b77c0 0x7f4f240b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f29f27640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4f240b5620 con 0x7f4f240b77c0 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f29f27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4f240b8110 con 0x7f4f240d4800 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f29f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f240d5280 con 0x7f4f240b5210 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f240b77c0 0x7f4f240b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f240b77c0 0x7f4f240b7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60858/0 (socket says 172.21.15.46:60858) 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 -- 172.21.15.46:0/3103309606 learned_addr learned my addr 172.21.15.46:0/3103309606 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f237fe640 1 --2- 172.21.15.46:0/3103309606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f241613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f1affd640 1 --2- 172.21.15.46:0/3103309606 >> v2:172.21.15.88:3300/0 conn(0x7f4f240d4800 0x7f4f240d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 -- 172.21.15.46:0/3103309606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 msgr2=0x7f4f241613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 --2- 172.21.15.46:0/3103309606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f241613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 -- 172.21.15.46:0/3103309606 >> v2:172.21.15.88:3300/0 conn(0x7f4f240d4800 msgr2=0x7f4f240d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 --2- 172.21.15.46:0/3103309606 >> v2:172.21.15.88:3300/0 conn(0x7f4f240d4800 0x7f4f240d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.372+0000 7f4f23fff640 1 -- 172.21.15.46:0/3103309606 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f240d5560 con 0x7f4f240b77c0 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.373+0000 7f4f23fff640 1 --2- 172.21.15.46:0/3103309606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240b77c0 0x7f4f240b7bd0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f4f2406da50 tx=0x7f4f1c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161a05d5b497d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.373+0000 7f4f22ffd640 1 -- 172.21.15.46:0/3103309606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c036070 con 0x7f4f240b77c0 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.373+0000 7f4f22ffd640 1 -- 172.21.15.46:0/3103309606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f1c031600 con 0x7f4f240b77c0 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.373+0000 7f4f22ffd640 1 -- 172.21.15.46:0/3103309606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c0319d0 con 0x7f4f240b77c0 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 -- 172.21.15.46:0/3103309606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240b77c0 msgr2=0x7f4f240b7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 --2- 172.21.15.46:0/3103309606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240b77c0 0x7f4f240b7bd0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f4f2406da50 tx=0x7f4f1c02a610 comp rx=0 tx=0).stop 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 -- 172.21.15.46:0/3103309606 shutdown_connections 2024-05-09T03:19:50.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 --2- 172.21.15.46:0/3103309606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240b77c0 0x7f4f240b7bd0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 --2- 172.21.15.46:0/3103309606 >> v2:172.21.15.88:3300/0 conn(0x7f4f240d4800 0x7f4f240d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 --2- 172.21.15.46:0/3103309606 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f241613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 -- 172.21.15.46:0/3103309606 >> 172.21.15.46:0/3103309606 conn(0x7f4f240d29a0 msgr2=0x7f4f240d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 -- 172.21.15.46:0/3103309606 shutdown_connections 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.374+0000 7f4f29f27640 1 -- 172.21.15.46:0/3103309606 wait complete. 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.376+0000 7f4f29f27640 1 Processor -- start 2024-05-09T03:19:50.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.376+0000 7f4f29f27640 1 -- start start 2024-05-09T03:19:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f29f27640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f2415fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f29f27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f240b77c0 0x7f4f240ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f29f27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 0x7f4f24154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f29f27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f240d0ae0 con 0x7f4f240d4800 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f29f27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f24160ea0 con 0x7f4f240b77c0 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f29f27640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4f24163e60 con 0x7f4f240b5210 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 0x7f4f24154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 0x7f4f24154fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60864/0 (socket says 172.21.15.46:60864) 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f23fff640 1 -- 172.21.15.46:0/287971362 learned_addr learned my addr 172.21.15.46:0/287971362 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f237fe640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f2415fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.377+0000 7f4f1affd640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f240b77c0 0x7f4f240ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f23fff640 1 -- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 msgr2=0x7f4f2415fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f23fff640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f2415fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f23fff640 1 -- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f240b77c0 msgr2=0x7f4f240ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f23fff640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f240b77c0 0x7f4f240ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f23fff640 1 -- 172.21.15.46:0/287971362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f241607f0 con 0x7f4f240d4800 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f23fff640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 0x7f4f24154fe0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009f40 tx=0x7f4f1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c041020 con 0x7f4f240d4800 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f1c0077a0 con 0x7f4f240d4800 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f24160510 con 0x7f4f240d4800 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c036040 con 0x7f4f240d4800 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.378+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f240ce390 con 0x7f4f240d4800 2024-05-09T03:19:50.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.379+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f240b4e50 con 0x7f4f240d4800 2024-05-09T03:19:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.380+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4f1c044020 con 0x7f4f240d4800 2024-05-09T03:19:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.380+0000 7f4f20ff9640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f040762d0 0x7f4f040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.381+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f4f1c0b8c20 con 0x7f4f240d4800 2024-05-09T03:19:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.381+0000 7f4f237fe640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f040762d0 0x7f4f040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.381+0000 7f4f237fe640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f040762d0 0x7f4f040787e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009a20 tx=0x7f4f14009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.385+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4f240b4e50 con 0x7f4f240d4800 2024-05-09T03:19:50.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.613+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f2406a270 con 0x7f4f240d4800 2024-05-09T03:19:50.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.613+0000 7f4f20ff9640 1 -- 172.21.15.46:0/287971362 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 300 v300) ==== 95+0+5185 (secure 0 0 0) 0x7f4f1c04b030 con 0x7f4f240d4800 2024-05-09T03:19:50.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:50.614 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":300,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":300,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:47.780947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9879":{"gid":9879,"name":"c","rank":0,"incarnation":296,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6834/2209678569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2209678569},{"type":"v1","addr":"172.21.15.46:6834","nonce":2209678569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9879,"qdb_cluster":[9879]},"id":6}]} 2024-05-09T03:19:50.614 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 300 2024-05-09T03:19:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f040762d0 msgr2=0x7f4f040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f040762d0 0x7f4f040787e0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009a20 tx=0x7f4f14009190 comp rx=0 tx=0).stop 2024-05-09T03:19:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 msgr2=0x7f4f24154fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 0x7f4f24154fe0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009f40 tx=0x7f4f1c004a40 comp rx=0 tx=0).stop 2024-05-09T03:19:50.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 shutdown_connections 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4f040762d0 0x7f4f040787e0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f240d4800 0x7f4f24154fe0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f240b77c0 0x7f4f240ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 --2- 172.21.15.46:0/287971362 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4f240b5210 0x7f4f2415fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 >> 172.21.15.46:0/287971362 conn(0x7f4f240d29a0 msgr2=0x7f4f240b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 shutdown_connections 2024-05-09T03:19:50.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.615+0000 7f4f29f27640 1 -- 172.21.15.46:0/287971362 wait complete. 2024-05-09T03:19:50.627 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-05-09T03:19:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.744+0000 7fef95827640 1 Processor -- start 2024-05-09T03:19:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.744+0000 7fef95827640 1 -- start start 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.745+0000 7fef95827640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef900b50b0 0x7fef900d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.745+0000 7fef95827640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef900b8220 0x7fef900b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.745+0000 7fef95827640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef900d5740 con 0x7fef900d11f0 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.745+0000 7fef95827640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fef9006e760 con 0x7fef900b50b0 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.745+0000 7fef95827640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fef9006de30 con 0x7fef900b8220 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef900b50b0 0x7fef900d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fef900b50b0 0x7fef900d0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60878/0 (socket says 172.21.15.46:60878) 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 -- 172.21.15.46:0/1124801334 learned_addr learned my addr 172.21.15.46:0/1124801334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/1124801334 >> v2:172.21.15.88:3300/0 conn(0x7fef900b8220 0x7fef900b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 -- 172.21.15.46:0/1124801334 >> v2:172.21.15.88:3300/0 conn(0x7fef900b8220 msgr2=0x7fef900b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 --2- 172.21.15.46:0/1124801334 >> v2:172.21.15.88:3300/0 conn(0x7fef900b8220 0x7fef900b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 -- 172.21.15.46:0/1124801334 >> v1:172.21.15.88:6789/0 conn(0x7fef900d11f0 legacy=0x7fef900b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.746+0000 7fef8effd640 1 -- 172.21.15.46:0/1124801334 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9006e110 con 0x7fef900b50b0 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.747+0000 7fef8effd640 1 --2- 172.21.15.46:0/1124801334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef900d0cb0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fef900d39d0 tx=0x7fef8002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0685661a292447 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.747+0000 7fef8e7fc640 1 -- 172.21.15.46:0/1124801334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80037070 con 0x7fef900b50b0 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.747+0000 7fef8e7fc640 1 -- 172.21.15.46:0/1124801334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef80032440 con 0x7fef900b50b0 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.747+0000 7fef8e7fc640 1 -- 172.21.15.46:0/1124801334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80032810 con 0x7fef900b50b0 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 -- 172.21.15.46:0/1124801334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 msgr2=0x7fef900d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 --2- 172.21.15.46:0/1124801334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef900d0cb0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fef900d39d0 tx=0x7fef8002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 -- 172.21.15.46:0/1124801334 shutdown_connections 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 --2- 172.21.15.46:0/1124801334 >> v2:172.21.15.88:3300/0 conn(0x7fef900b8220 0x7fef900b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 --2- 172.21.15.46:0/1124801334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef900d0cb0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 -- 172.21.15.46:0/1124801334 >> 172.21.15.46:0/1124801334 conn(0x7fef900d2b80 msgr2=0x7fef900d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 -- 172.21.15.46:0/1124801334 shutdown_connections 2024-05-09T03:19:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.748+0000 7fef95827640 1 -- 172.21.15.46:0/1124801334 wait complete. 2024-05-09T03:19:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.749+0000 7fef95827640 1 Processor -- start 2024-05-09T03:19:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.750+0000 7fef95827640 1 -- start start 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef95827640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef9014ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef95827640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900d11f0 0x7fef90150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef95827640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef90160050 0x7fef90150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef95827640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fef9006ebf0 con 0x7fef900b50b0 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef95827640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fef900b54c0 con 0x7fef900d11f0 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef95827640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fef900d1800 con 0x7fef90160050 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef9014ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef9014ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60886/0 (socket says 172.21.15.46:60886) 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 -- 172.21.15.46:0/3874057658 learned_addr learned my addr 172.21.15.46:0/3874057658 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef87fff640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900d11f0 0x7fef90150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef90160050 0x7fef90150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 -- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef90160050 msgr2=0x7fef90150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef90160050 0x7fef90150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 -- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900d11f0 msgr2=0x7fef90150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900d11f0 0x7fef90150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.751+0000 7fef8effd640 1 -- 172.21.15.46:0/3874057658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9014f6c0 con 0x7fef900b50b0 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.752+0000 7fef8effd640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef9014ea00 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fef80000c00 tx=0x7fef800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.752+0000 7fef8cff9640 1 -- 172.21.15.46:0/3874057658 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80041070 con 0x7fef900b50b0 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.752+0000 7fef8cff9640 1 -- 172.21.15.46:0/3874057658 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef8003c870 con 0x7fef900b50b0 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.752+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9014f9a0 con 0x7fef900b50b0 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.752+0000 7fef8cff9640 1 -- 172.21.15.46:0/3874057658 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef80037040 con 0x7fef900b50b0 2024-05-09T03:19:50.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.752+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9014ff60 con 0x7fef900b50b0 2024-05-09T03:19:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.754+0000 7fef8cff9640 1 -- 172.21.15.46:0/3874057658 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fef80044070 con 0x7fef900b50b0 2024-05-09T03:19:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.754+0000 7fef8cff9640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef70076300 0x7fef70078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.754+0000 7fef8cff9640 1 -- 172.21.15.46:0/3874057658 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7fef800b8b60 con 0x7fef900b50b0 2024-05-09T03:19:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.755+0000 7fef87fff640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef70076300 0x7fef70078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.755+0000 7fef9c7bb740 1 Processor -- start 2024-05-09T03:19:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.755+0000 7fef9c7bb740 1 -- start start 2024-05-09T03:19:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.755+0000 7fef87fff640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef70076300 0x7fef70078810 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fef90162f60 tx=0x7fef78009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea4036d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea3e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea01a670 0x5605ea3e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5605ea343d10 con 0x5605ea012490 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5605ea2cb750 con 0x5605ea03b350 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5605ea33fcd0 con 0x5605ea01a670 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.756+0000 7fef87fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea3e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea3e0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60898/0 (socket says 172.21.15.46:60898) 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 -- 172.21.15.46:0/541354962 learned_addr learned my addr 172.21.15.46:0/541354962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef8effd640 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea4036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea01a670 0x5605ea3e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 -- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea01a670 msgr2=0x5605ea3e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea01a670 0x5605ea3e1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 -- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 msgr2=0x5605ea4036d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea4036d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 -- 172.21.15.46:0/541354962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ea11d790 con 0x5605ea012490 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea01a670 0x5605ea3e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef87fff640 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea3e0f10 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fef7800eb40 tx=0x7fef78011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30918e9006079abe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef857fa640 1 -- 172.21.15.46:0/541354962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7801b040 con 0x5605ea012490 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef857fa640 1 -- 172.21.15.46:0/541354962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78024070 con 0x5605ea012490 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.757+0000 7fef857fa640 1 -- 172.21.15.46:0/541354962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78015c40 con 0x5605ea012490 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 -- 172.21.15.46:0/541354962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 msgr2=0x5605ea3e0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea3e0f10 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fef7800eb40 tx=0x7fef78011000 comp rx=0 tx=0).stop 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 -- 172.21.15.46:0/541354962 shutdown_connections 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea01a670 0x5605ea3e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea3e0f10 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/541354962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea4036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 -- 172.21.15.46:0/541354962 >> 172.21.15.46:0/541354962 conn(0x5605ea02f4a0 msgr2=0x5605ea3de6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 -- 172.21.15.46:0/541354962 shutdown_connections 2024-05-09T03:19:50.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.758+0000 7fef9c7bb740 1 -- 172.21.15.46:0/541354962 wait complete. 2024-05-09T03:19:50.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.759+0000 7fef9c7bb740 1 -- start start 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.759+0000 7fef9c7bb740 1 Processor -- start 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.760+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4356d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.760+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea427300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.760+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea427840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.760+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5605ea300bd0 con 0x5605ea012490 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.760+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5605ea435f50 con 0x5605ea01a670 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.760+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5605ea3ff570 con 0x5605ea03b350 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef9c7bb740 10 client.0 _ll_register_callbacks cb 0x5605ea2f5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4356d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60900/0 (socket says 172.21.15.46:60900) 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 -- 172.21.15.46:0/1467680670 learned_addr learned my addr 172.21.15.46:0/1467680670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef87fff640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea427300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea427840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 -- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 msgr2=0x5605ea427840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea427840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 -- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 msgr2=0x5605ea427300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea427300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 -- 172.21.15.46:0/1467680670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605e9fec130 con 0x5605ea012490 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef87fff640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea427300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef8effd640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4356d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fef800b6860 tx=0x7fef800809b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdee9b4a0089228 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef6e7fc640 1 -- 172.21.15.46:0/1467680670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8004b040 con 0x5605ea012490 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef6e7fc640 1 -- 172.21.15.46:0/1467680670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7fef80085120 con 0x5605ea012490 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef78024070 con 0x5605ea012490 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef6e7fc640 1 -- 172.21.15.46:0/1467680670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=300) ==== 24+0+0 (secure 0 0 0) 0x7fef80085320 con 0x5605ea012490 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef9c7bb740 10 client.10005 fetch_fsmap learned FSMap version 300 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.761+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=300,monmap=2+}) -- 0x5605ea11d5c0 con 0x5605ea012490 2024-05-09T03:19:50.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef6e7fc640 1 -- 172.21.15.46:0/1467680670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78024070 con 0x5605ea012490 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef6e7fc640 1 -- 172.21.15.46:0/1467680670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 300) ==== 3494+0+0 (secure 0 0 0) 0x7fef800804a0 con 0x5605ea012490 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef9c7bb740 10 client.10005 fetch_fsmap finished waiting for FSMap version 300 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef9c7bb740 10 client.10005 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef9c7bb740 20 client.10005 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605e9ffa130 0x5605ea3f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef9c7bb740 4 client.10005 mds_command: new command op to 9879 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 --> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605ea351920 con 0x5605e9ffa130 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef87fff640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605e9ffa130 0x5605ea3f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef87fff640 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605e9ffa130 0x5605ea3f8f20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e753db84c79a1ba server_cookie=fbd9ffd3daf143c1 in_seq=0 out_seq=0 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.762+0000 7fef6e7fc640 10 client.10005 ms_handle_connect on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef6e7fc640 1 -- 172.21.15.46:0/1467680670 <== mds.0 v2:172.21.15.46:6832/2209678569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5605ea351920 con 0x5605e9ffa130 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef6e7fc640 10 client.10005 handle_command_reply: tid=0 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 client.10005 shutdown 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 msgr2=0x5605ea4356d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4356d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fef800b6860 tx=0x7fef800809b0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 shutdown_connections 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605e9ffa130 0x5605ea3f8f20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea427840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef6e7fc640 0 client.10005 ms_handle_reset on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea427300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/1467680670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4356d0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 >> 172.21.15.46:0/1467680670 conn(0x5605ea02f4a0 msgr2=0x5605ea3e0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 shutdown_connections 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 1 -- 172.21.15.46:0/1467680670 wait complete. 2024-05-09T03:19:50.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.763+0000 7fef9c7bb740 20 client.10005 trim_cache size 0 max 16384 2024-05-09T03:19:50.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.777+0000 7fef9c7bb740 1 Processor -- start 2024-05-09T03:19:50.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.778+0000 7fef9c7bb740 1 -- start start 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea012490 0x5605ea3fc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 0x5605ea3f7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea3fcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fef90203cd0 con 0x5605ea01a670 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fef90203030 con 0x5605ea03b350 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fef90203680 con 0x5605ea012490 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 0x5605ea3f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 0x5605ea3f7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60906/0 (socket says 172.21.15.46:60906) 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 -- 172.21.15.46:0/4066490321 learned_addr learned my addr 172.21.15.46:0/4066490321 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea3fcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef8effd640 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea012490 0x5605ea3fc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 -- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea012490 msgr2=0x5605ea3fc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea012490 0x5605ea3fc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 -- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 msgr2=0x5605ea3fcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea3fcf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef87fff640 1 -- 172.21.15.46:0/4066490321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ea01f150 con 0x5605ea01a670 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.779+0000 7fef8effd640 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea012490 0x5605ea3fc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.780+0000 7fef87fff640 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 0x5605ea3f7140 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x5605ea40f570 tx=0x7fef78013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17290995b1c4e0c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.780+0000 7fef84ff9640 1 -- 172.21.15.46:0/4066490321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78037070 con 0x5605ea01a670 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.780+0000 7fef84ff9640 1 -- 172.21.15.46:0/4066490321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78024040 con 0x5605ea01a670 2024-05-09T03:19:50.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.780+0000 7fef84ff9640 1 -- 172.21.15.46:0/4066490321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78004240 con 0x5605ea01a670 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 -- 172.21.15.46:0/4066490321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 msgr2=0x5605ea3f7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 0x5605ea3f7140 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x5605ea40f570 tx=0x7fef78013000 comp rx=0 tx=0).stop 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 -- 172.21.15.46:0/4066490321 shutdown_connections 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea03b350 0x5605ea3fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea01a670 0x5605ea3f7140 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/4066490321 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea012490 0x5605ea3fc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 -- 172.21.15.46:0/4066490321 >> 172.21.15.46:0/4066490321 conn(0x5605ea02f4a0 msgr2=0x5605ea3fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 -- 172.21.15.46:0/4066490321 shutdown_connections 2024-05-09T03:19:50.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.781+0000 7fef9c7bb740 1 -- 172.21.15.46:0/4066490321 wait complete. 2024-05-09T03:19:50.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.782+0000 7fef9c7bb740 1 -- start start 2024-05-09T03:19:50.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.782+0000 7fef9c7bb740 1 Processor -- start 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4a2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea402600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea3e0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5605ea3f8690 con 0x5605ea012490 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5605ea3ff270 con 0x5605ea01a670 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5605ea3f81f0 con 0x5605ea03b350 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef9c7bb740 10 client.0 _ll_register_callbacks cb 0x7fef88009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4a2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60908/0 (socket says 172.21.15.46:60908) 2024-05-09T03:19:50.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 -- 172.21.15.46:0/2437233367 learned_addr learned my addr 172.21.15.46:0/2437233367 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef87fff640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea402600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8f7fe640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea3e0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 -- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 msgr2=0x5605ea3e0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea3e0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 -- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 msgr2=0x5605ea402600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea402600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.783+0000 7fef8effd640 1 -- 172.21.15.46:0/2437233367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ea402b40 con 0x5605ea012490 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef8effd640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4a2820 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fef800b8e80 tx=0x7fef800b63c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b941b90e2ebdaea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef6e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef6e7fc640 1 -- 172.21.15.46:0/2437233367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef800b3060 con 0x5605ea012490 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef6e7fc640 1 -- 172.21.15.46:0/2437233367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7fef8003cc40 con 0x5605ea012490 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef78024040 con 0x5605ea012490 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef6e7fc640 1 -- 172.21.15.46:0/2437233367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=300) ==== 24+0+0 (secure 0 0 0) 0x7fef8004b040 con 0x5605ea012490 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 10 client.10011 fetch_fsmap learned FSMap version 300 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=300,monmap=2+}) -- 0x5605ea402b40 con 0x5605ea012490 2024-05-09T03:19:50.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef6e7fc640 1 -- 172.21.15.46:0/2437233367 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef78024040 con 0x5605ea012490 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef6e7fc640 1 -- 172.21.15.46:0/2437233367 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 300) ==== 3494+0+0 (secure 0 0 0) 0x7fef8007ff10 con 0x5605ea012490 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 10 client.10011 fetch_fsmap finished waiting for FSMap version 300 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 10 client.10011 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 20 client.10011 populate_metadata read hostname 'smithi046' 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605ea435340 0x5605ea4a6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 4 client.10011 mds_command: new command op to 9879 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 --> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5605ea3dc500 con 0x5605ea435340 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.784+0000 7fef87fff640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605ea435340 0x5605ea4a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef6e7fc640 10 client.10011 ms_handle_connect on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef87fff640 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605ea435340 0x5605ea4a6280 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e67517d2891279e server_cookie=66776702b83bbf92 in_seq=0 out_seq=0 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef6e7fc640 1 -- 172.21.15.46:0/2437233367 <== mds.0 v2:172.21.15.46:6832/2209678569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+193 (crc 0 0 0) 0x5605ea3dc500 con 0x5605ea435340 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef6e7fc640 10 client.10011 handle_command_reply: tid=0 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 client.10011 shutdown 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 msgr2=0x5605ea4a2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4a2820 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fef800b8e80 tx=0x7fef800b63c0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 shutdown_connections 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:6832/2209678569,v1:172.21.15.46:6834/2209678569] conn(0x5605ea435340 0x5605ea4a6280 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5605ea03b350 0x5605ea3e0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605ea01a670 0x5605ea402600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 --2- 172.21.15.46:0/2437233367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5605ea012490 0x5605ea4a2820 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef6e7fc640 0 client.10011 ms_handle_reset on v2:172.21.15.46:6832/2209678569 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 >> 172.21.15.46:0/2437233367 conn(0x5605ea02f4a0 msgr2=0x5605ea427300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 shutdown_connections 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 1 -- 172.21.15.46:0/2437233367 wait complete. 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.785+0000 7fef9c7bb740 20 client.10011 trim_cache size 0 max 16384 2024-05-09T03:19:50.791 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "dentry", 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 366175820, 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1, 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "frag": "*", 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "dname": "subdir", 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "snap_id": "head", 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/subdir" 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:19:50.792 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef70076300 msgr2=0x7fef70078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef70076300 0x7fef70078810 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fef90162f60 tx=0x7fef78009060 comp rx=0 tx=0).stop 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 msgr2=0x7fef9014ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef9014ea00 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fef80000c00 tx=0x7fef800046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 shutdown_connections 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fef90160050 0x7fef90150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef900d11f0 0x7fef90150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fef70076300 0x7fef70078810 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 --2- 172.21.15.46:0/3874057658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef900b50b0 0x7fef9014ea00 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 >> 172.21.15.46:0/3874057658 conn(0x7fef900d2b80 msgr2=0x7fef900b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 shutdown_connections 2024-05-09T03:19:50.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.786+0000 7fef95827640 1 -- 172.21.15.46:0/3874057658 wait complete. 2024-05-09T03:19:50.798 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 10000000000.00000000' 2024-05-09T03:19:50.798 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:50.798 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:50.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:19:50.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.972+0000 7f551ae3b640 1 Processor -- start 2024-05-09T03:19:50.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.972+0000 7f551ae3b640 1 -- start start 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f551ae3b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55140d2800 0x7f55140b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f551ae3b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551406ebe0 con 0x7f55140b4700 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f551ae3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551406eec0 con 0x7f55140b3f50 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f551ae3b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f551406f1a0 con 0x7f55140d2800 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f5518bd7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f55140b4700 0x7f55140d20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41044/0 (socket says 172.21.15.46:41044) 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f5518bd7640 1 -- 172.21.15.46:0/3004719817 learned_addr learned my addr 172.21.15.46:0/3004719817 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 716459458 0 0) 0x7f551406ebe0 con 0x7f55140b4700 2024-05-09T03:19:50.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54fc003520 con 0x7f55140b4700 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.973+0000 7f5513fff640 1 --2- 172.21.15.46:0/3004719817 >> v2:172.21.15.88:3300/0 conn(0x7f55140d2800 0x7f55140b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2573993745 0 0) 0x7f54fc003520 con 0x7f55140b4700 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 >> v2:172.21.15.88:3300/0 conn(0x7f55140d2800 msgr2=0x7f55140b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 --2- 172.21.15.46:0/3004719817 >> v2:172.21.15.88:3300/0 conn(0x7f55140d2800 0x7f55140b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 >> v1:172.21.15.88:6789/0 conn(0x7f55140b3f50 legacy=0x7f55140ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551406f480 con 0x7f55140b4700 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f55040016f0 con 0x7f55140b4700 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 >> v1:172.21.15.46:6789/0 conn(0x7f55140b4700 legacy=0x7f55140d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc003a50 0x7f54fc003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc004800 0x7f54fc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0053a0 0x7f54fc00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f551406f1a0 con 0x7f54fc003a50 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54fc00c330 con 0x7f54fc0053a0 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f54fc00c530 con 0x7f54fc004800 2024-05-09T03:19:50.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5518bd7640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc003a50 0x7f54fc003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55193d8640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0053a0 0x7f54fc00ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5513fff640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc004800 0x7f54fc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5518bd7640 1 -- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc004800 msgr2=0x7f54fc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5518bd7640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f54fc004800 0x7f54fc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5518bd7640 1 -- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0053a0 msgr2=0x7f54fc00ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5518bd7640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc0053a0 0x7f54fc00ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f5518bd7640 1 -- 172.21.15.46:0/3004719817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54fc003520 con 0x7f54fc003a50 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.974+0000 7f55193d8640 1 -- 172.21.15.46:0/3004719817 reap_dead start 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.975+0000 7f5518bd7640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc003a50 0x7f54fc003ec0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f550400c210 tx=0x7f54fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4501f90e7505b713 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.975+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504007d10 con 0x7f54fc003a50 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.975+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5504005ad0 con 0x7f54fc003a50 2024-05-09T03:19:50.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.975+0000 7f55137fe640 1 -- 172.21.15.46:0/3004719817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5504007070 con 0x7f54fc003a50 2024-05-09T03:19:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 -- 172.21.15.46:0/3004719817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc003a50 msgr2=0x7f54fc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc003a50 0x7f54fc003ec0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f550400c210 tx=0x7f54fc005830 comp rx=0 tx=0).stop 2024-05-09T03:19:50.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 -- 172.21.15.46:0/3004719817 shutdown_connections 2024-05-09T03:19:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 --2- 172.21.15.46:0/3004719817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54fc003a50 0x7f54fc003ec0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 -- 172.21.15.46:0/3004719817 >> 172.21.15.46:0/3004719817 conn(0x7f55140b55d0 msgr2=0x7f55140b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 -- 172.21.15.46:0/3004719817 shutdown_connections 2024-05-09T03:19:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.976+0000 7f551ae3b640 1 -- 172.21.15.46:0/3004719817 wait complete. 2024-05-09T03:19:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.978+0000 7f551ae3b640 1 Processor -- start 2024-05-09T03:19:50.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.978+0000 7f551ae3b640 1 -- start start 2024-05-09T03:19:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f551ae3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc003a50 0x7f55141e0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f551ae3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 0x7f55141d4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f551ae3b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55141e6d10 0x7f55140d0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f551ae3b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f55140b51d0 con 0x7f55141e2030 2024-05-09T03:19:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f551ae3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f55140b4360 con 0x7f54fc003a50 2024-05-09T03:19:50.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f551ae3b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f55140b2390 con 0x7f55141e6d10 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f5513fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 0x7f55141d4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f5513fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 0x7f55141d4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60914/0 (socket says 172.21.15.46:60914) 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.979+0000 7f5513fff640 1 -- 172.21.15.46:0/2878489495 learned_addr learned my addr 172.21.15.46:0/2878489495 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5518bd7640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc003a50 0x7f55141e0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f55193d8640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55141e6d10 0x7f55140d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5513fff640 1 -- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55141e6d10 msgr2=0x7f55140d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5513fff640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55141e6d10 0x7f55140d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5513fff640 1 -- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc003a50 msgr2=0x7f55141e0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5513fff640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc003a50 0x7f55141e0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5513fff640 1 -- 172.21.15.46:0/2878489495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55141e0f90 con 0x7f55141e2030 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5518bd7640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc003a50 0x7f55141e0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f55193d8640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55141e6d10 0x7f55140d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f5513fff640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 0x7f55141d4c30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f5508011630 tx=0x7f5508011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550800f8f0 con 0x7f55141e2030 2024-05-09T03:19:50.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550800fab0 con 0x7f55141e2030 2024-05-09T03:19:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55141e1270 con 0x7f55141e2030 2024-05-09T03:19:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550800fea0 con 0x7f55141e2030 2024-05-09T03:19:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.980+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55141e1830 con 0x7f55141e2030 2024-05-09T03:19:50.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.981+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551406a270 con 0x7f55141e2030 2024-05-09T03:19:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.982+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5508015460 con 0x7f55141e2030 2024-05-09T03:19:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.983+0000 7f55117fa640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55000762d0 0x7f55000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.983+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f5508098c90 con 0x7f55141e2030 2024-05-09T03:19:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.983+0000 7f5518bd7640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55000762d0 0x7f55000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:50.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.984+0000 7f5518bd7640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55000762d0 0x7f55000787e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f5504032000 tx=0x7f5504009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:50.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:50.988+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5508019030 con 0x7f55141e2030 2024-05-09T03:19:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.191+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f55141e0d20 con 0x7f55141e2030 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.192+0000 7f55117fa640 1 -- 172.21.15.46:0/2878489495 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 66 entries v117) ==== 81+0+4010 (secure 0 0 0) 0x7f55080619c0 con 0x7f55141e2030 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:51.193 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:51.194 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:51.195 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:51.196 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:51.197 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:listed 66 entries 2024-05-09T03:19:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55000762d0 msgr2=0x7f55000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55000762d0 0x7f55000787e0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f5504032000 tx=0x7f5504009030 comp rx=0 tx=0).stop 2024-05-09T03:19:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 msgr2=0x7f55141d4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 0x7f55141d4c30 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f5508011630 tx=0x7f5508011af0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 shutdown_connections 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f55141e6d10 0x7f55140d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f55000762d0 0x7f55000787e0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55141e2030 0x7f55141d4c30 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 --2- 172.21.15.46:0/2878489495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54fc003a50 0x7f55141e0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 >> 172.21.15.46:0/2878489495 conn(0x7f55140b55d0 msgr2=0x7f55140cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 shutdown_connections 2024-05-09T03:19:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.194+0000 7f551ae3b640 1 -- 172.21.15.46:0/2878489495 wait complete. 2024-05-09T03:19:51.206 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:19:51.206 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:19:51.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:19:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.377+0000 7f4d3915e640 1 Processor -- start 2024-05-09T03:19:51.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.377+0000 7f4d3915e640 1 -- start start 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.378+0000 7f4d3915e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b5140 0x7f4d340b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.378+0000 7f4d3915e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d340b80b0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.378+0000 7f4d3915e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d340b5550 con 0x7f4d340b7ca0 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.378+0000 7f4d3915e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d340d3480 con 0x7f4d340b5140 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d3359e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4d340d3fc0 0x7f4d340b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41056/0 (socket says 172.21.15.46:41056) 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d3359e640 1 -- 172.21.15.46:0/1000867203 learned_addr learned my addr 172.21.15.46:0/1000867203 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d32d9d640 1 --2- 172.21.15.46:0/1000867203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b5140 0x7f4d340b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 170628003 0 0) 0x7f4d2c001110 con 0x7f4d340d3fc0 2024-05-09T03:19:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d1c003520 con 0x7f4d340d3fc0 2024-05-09T03:19:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3913762058 0 0) 0x7f4d340b5550 con 0x7f4d340b7ca0 2024-05-09T03:19:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d2c001110 con 0x7f4d340b7ca0 2024-05-09T03:19:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d32d9d640 1 -- 172.21.15.46:0/1000867203 >> v1:172.21.15.88:6789/0 conn(0x7f4d340b7ca0 legacy=0x7f4d340d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d32d9d640 1 -- 172.21.15.46:0/1000867203 >> v1:172.21.15.46:6789/0 conn(0x7f4d340d3fc0 legacy=0x7f4d340b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.379+0000 7f4d32d9d640 1 -- 172.21.15.46:0/1000867203 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d340d44e0 con 0x7f4d340b5140 2024-05-09T03:19:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.380+0000 7f4d32d9d640 1 --2- 172.21.15.46:0/1000867203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b5140 0x7f4d340b7760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009dc0 tx=0x7f4d2402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83173e5bc9a32af5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.380+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24007740 con 0x7f4d340b5140 2024-05-09T03:19:51.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.380+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d240078e0 con 0x7f4d340b5140 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.380+0000 7f4d31d9b640 1 -- 172.21.15.46:0/1000867203 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d24007c90 con 0x7f4d340b5140 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 -- 172.21.15.46:0/1000867203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b5140 msgr2=0x7f4d340b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1000867203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b5140 0x7f4d340b7760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009dc0 tx=0x7f4d2402a760 comp rx=0 tx=0).stop 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 -- 172.21.15.46:0/1000867203 shutdown_connections 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1000867203 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b5140 0x7f4d340b7760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 -- 172.21.15.46:0/1000867203 >> 172.21.15.46:0/1000867203 conn(0x7f4d3408d3f0 msgr2=0x7f4d3408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 -- 172.21.15.46:0/1000867203 shutdown_connections 2024-05-09T03:19:51.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.381+0000 7f4d3915e640 1 -- 172.21.15.46:0/1000867203 wait complete. 2024-05-09T03:19:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.383+0000 7f4d3915e640 1 Processor -- start 2024-05-09T03:19:51.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.383+0000 7f4d3915e640 1 -- start start 2024-05-09T03:19:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3915e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d340b5140 0x7f4d3415b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3915e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b7ca0 0x7f4d34149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3915e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 0x7f4d34149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3915e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4d340d4d80 con 0x7f4d340d3fc0 2024-05-09T03:19:51.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3915e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d340d4aa0 con 0x7f4d340b5140 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3915e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4d340d5580 con 0x7f4d340b7ca0 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 0x7f4d34149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 0x7f4d34149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60926/0 (socket says 172.21.15.46:60926) 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.384+0000 7f4d3359e640 1 -- 172.21.15.46:0/1758321429 learned_addr learned my addr 172.21.15.46:0/1758321429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d32d9d640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d340b5140 0x7f4d3415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3259c640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b7ca0 0x7f4d34149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3359e640 1 -- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b7ca0 msgr2=0x7f4d34149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3359e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b7ca0 0x7f4d34149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3359e640 1 -- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d340b5140 msgr2=0x7f4d3415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3359e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d340b5140 0x7f4d3415b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3359e640 1 -- 172.21.15.46:0/1758321429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3414a450 con 0x7f4d340d3fc0 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d32d9d640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d340b5140 0x7f4d3415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3359e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 0x7f4d34149c40 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00c730 tx=0x7f4d2c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:51.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c017070 con 0x7f4d340d3fc0 2024-05-09T03:19:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c005ad0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3414a730 con 0x7f4d340d3fc0 2024-05-09T03:19:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c007070 con 0x7f4d340d3fc0 2024-05-09T03:19:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.385+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3414acf0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.386+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d340b77e0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.387+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4d2c01a070 con 0x7f4d340d3fc0 2024-05-09T03:19:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.387+0000 7f4d1b7fe640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d200762d0 0x7f4d200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.387+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7f4d2c0976e0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.388+0000 7f4d32d9d640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d200762d0 0x7f4d200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.388+0000 7f4d32d9d640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d200762d0 0x7f4d200787e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4d24034000 tx=0x7f4d2402f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:51.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.392+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4d340b77e0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.596+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4d3414a1e0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.597+0000 7f4d1b7fe640 1 -- 172.21.15.46:0/1758321429 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 66 entries v117) ==== 81+0+4010 (secure 0 0 0) 0x7f4d340b80b0 con 0x7f4d340d3fc0 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:51.598 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:51.599 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:51.600 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:19:51.601 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:19:51.602 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:19:51.603 INFO:teuthology.orchestra.run.smithi046.stderr:listed 66 entries 2024-05-09T03:19:51.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d200762d0 msgr2=0x7f4d200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d200762d0 0x7f4d200787e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4d24034000 tx=0x7f4d2402f000 comp rx=0 tx=0).stop 2024-05-09T03:19:51.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 msgr2=0x7f4d34149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 0x7f4d34149c40 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00c730 tx=0x7f4d2c00cbf0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 shutdown_connections 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4d200762d0 0x7f4d200787e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d340d3fc0 0x7f4d34149c40 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4d340b7ca0 0x7f4d34149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 --2- 172.21.15.46:0/1758321429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d340b5140 0x7f4d3415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 >> 172.21.15.46:0/1758321429 conn(0x7f4d3408d3f0 msgr2=0x7f4d340b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 shutdown_connections 2024-05-09T03:19:51.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.599+0000 7f4d3915e640 1 -- 172.21.15.46:0/1758321429 wait complete. 2024-05-09T03:19:51.611 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:19:51.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:51.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:19:51.637 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:19:51.637 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:19:51.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:19:51.637 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:19:51.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:19:51.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.848+0000 7fbe2d15e640 1 Processor -- start 2024-05-09T03:19:51.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.848+0000 7fbe2d15e640 1 -- start start 2024-05-09T03:19:51.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.849+0000 7fbe2d15e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280bb770 0x7fbe280d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.849+0000 7fbe2d15e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe28164900 con 0x7fbe280d3180 2024-05-09T03:19:51.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.849+0000 7fbe2d15e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe28164be0 con 0x7fbe28163eb0 2024-05-09T03:19:51.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.849+0000 7fbe2d15e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbe280d4bb0 con 0x7fbe280bb770 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.849+0000 7fbe2759e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe280d3180 0x7fbe280b9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41064/0 (socket says 172.21.15.46:41064) 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.849+0000 7fbe2759e640 1 -- 172.21.15.46:0/1923375411 learned_addr learned my addr 172.21.15.46:0/1923375411 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280bb770 0x7fbe280d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314717855 0 0) 0x7fbe28164900 con 0x7fbe280d3180 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe14003520 con 0x7fbe280d3180 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1104373255 0 0) 0x7fbe28164be0 con 0x7fbe28163eb0 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe28164900 con 0x7fbe28163eb0 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 998039580 0 0) 0x7fbe14003520 con 0x7fbe280d3180 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280bb770 msgr2=0x7fbe280d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280bb770 0x7fbe280d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 >> v1:172.21.15.88:6789/0 conn(0x7fbe28163eb0 legacy=0x7fbe280d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe280d4840 con 0x7fbe280d3180 2024-05-09T03:19:51.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fbe20001a70 con 0x7fbe280d3180 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 >> v1:172.21.15.46:6789/0 conn(0x7fbe280d3180 legacy=0x7fbe280b9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe14003c10 0x7fbe14004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14004800 0x7fbe14004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe14005260 0x7fbe1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe280d4bb0 con 0x7fbe14003c10 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1400c170 con 0x7fbe14004800 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbe1400c370 con 0x7fbe14005260 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280bb770 0x7fbe280d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe14003c10 0x7fbe14004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe1ffff640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14004800 0x7fbe14004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.850+0000 7fbe2759e640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe14005260 0x7fbe1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.851+0000 7fbe26d9d640 1 -- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe14005260 msgr2=0x7fbe1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.851+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe14005260 0x7fbe1400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.851+0000 7fbe26d9d640 1 -- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14004800 msgr2=0x7fbe14004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.851+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14004800 0x7fbe14004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.851+0000 7fbe26d9d640 1 -- 172.21.15.46:0/1923375411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe14003520 con 0x7fbe14003c10 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.851+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe14003c10 0x7fbe14004060 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fbe280d2a40 tx=0x7fbe14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc53ced55338011 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18036070 con 0x7fbe14003c10 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe18004030 con 0x7fbe14003c10 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2659c640 1 -- 172.21.15.46:0/1923375411 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe180097f0 con 0x7fbe14003c10 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1923375411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe14003c10 msgr2=0x7fbe14004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1923375411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe14003c10 0x7fbe14004060 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fbe280d2a40 tx=0x7fbe14005830 comp rx=0 tx=0).stop 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2759e640 1 -- 172.21.15.46:0/1923375411 reap_dead start 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1923375411 shutdown_connections 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1923375411 >> 172.21.15.46:0/1923375411 conn(0x7fbe2806e370 msgr2=0x7fbe2806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1923375411 shutdown_connections 2024-05-09T03:19:51.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.852+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1923375411 wait complete. 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.854+0000 7fbe2d15e640 1 Processor -- start 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.854+0000 7fbe2d15e640 1 -- start start 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2d15e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14003c10 0x7fbe280d5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2d15e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280d5660 0x7fbe281f9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2d15e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 0x7fbe2814dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2d15e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe280d0d40 con 0x7fbe281f9ed0 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2d15e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbe280d0bc0 con 0x7fbe14003c10 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2d15e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbe281642c0 con 0x7fbe280d5660 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 0x7fbe2814dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 0x7fbe2814dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60946/0 (socket says 172.21.15.46:60946) 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 -- 172.21.15.46:0/1758791070 learned_addr learned my addr 172.21.15.46:0/1758791070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14003c10 0x7fbe280d5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe1ffff640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280d5660 0x7fbe281f9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 -- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280d5660 msgr2=0x7fbe281f9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280d5660 0x7fbe281f9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 -- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14003c10 msgr2=0x7fbe280d5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14003c10 0x7fbe280d5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 -- 172.21.15.46:0/1758791070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe281fa6c0 con 0x7fbe281f9ed0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.856+0000 7fbe2759e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 0x7fbe2814dc40 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fbe2808f170 tx=0x7fbe20007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.857+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2000aa50 con 0x7fbe281f9ed0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.857+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe20019070 con 0x7fbe281f9ed0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.857+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe280c96f0 con 0x7fbe281f9ed0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.857+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2000ae30 con 0x7fbe281f9ed0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.857+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe280c9cb0 con 0x7fbe281f9ed0 2024-05-09T03:19:51.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.858+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe281fa430 con 0x7fbe281f9ed0 2024-05-09T03:19:51.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.858+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbe2001e460 con 0x7fbe281f9ed0 2024-05-09T03:19:51.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.859+0000 7fbe1f7fe640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe080762d0 0x7fbe080787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:51.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.859+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8605+0+0 (secure 0 0 0) 0x7fbe20098530 con 0x7fbe281f9ed0 2024-05-09T03:19:51.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.859+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe080762d0 0x7fbe080787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:51.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.860+0000 7fbe26d9d640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe080762d0 0x7fbe080787e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbe18033000 tx=0x7fbe18008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:51.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:51.864+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbe200612f0 con 0x7fbe281f9ed0 2024-05-09T03:19:52.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.096+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbe2806a270 con 0x7fbe281f9ed0 2024-05-09T03:19:52.098 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:19:52.097+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:19:52.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.099+0000 7fbe1f7fe640 1 -- 172.21.15.46:0/1758791070 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v301) ==== 161+0+0 (secure 0 0 0) 0x7fbe20060bf0 con 0x7fbe281f9ed0 2024-05-09T03:19:52.099 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe080762d0 msgr2=0x7fbe080787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe080762d0 0x7fbe080787e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbe18033000 tx=0x7fbe18008030 comp rx=0 tx=0).stop 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 msgr2=0x7fbe2814dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 0x7fbe2814dc40 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fbe2808f170 tx=0x7fbe20007b10 comp rx=0 tx=0).stop 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 shutdown_connections 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbe080762d0 0x7fbe080787e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe281f9ed0 0x7fbe2814dc40 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbe280d5660 0x7fbe281f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 --2- 172.21.15.46:0/1758791070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe14003c10 0x7fbe280d5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 >> 172.21.15.46:0/1758791070 conn(0x7fbe2806e370 msgr2=0x7fbe280b7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:52.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 shutdown_connections 2024-05-09T03:19:52.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.102+0000 7fbe2d15e640 1 -- 172.21.15.46:0/1758791070 wait complete. 2024-05-09T03:19:52.104 INFO:tasks.ceph.mds.c.smithi046.stderr: -452> 2024-05-09T03:19:46.751+0000 7ff2ab77d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:19:52.119 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:19:52.164 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:19:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.235+0000 7fbcbdc47640 1 Processor -- start 2024-05-09T03:19:52.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.235+0000 7fbcbdc47640 1 -- start start 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcbdc47640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbcb80d1740 0x7fbcb80b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcbdc47640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcb80b9690 con 0x7fbcb80d1110 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcbdc47640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcb80ca9e0 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcbdc47640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbcb80d2b00 con 0x7fbcb80d1740 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbcb80d1110 0x7fbcb80d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41072/0 (socket says 172.21.15.46:41072) 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb6ffd640 1 -- 172.21.15.46:0/3691655496 learned_addr learned my addr 172.21.15.46:0/3691655496 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:52.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb77fe640 1 --2- 172.21.15.46:0/3691655496 >> v2:172.21.15.88:3300/0 conn(0x7fbcb80d1740 0x7fbcb80b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31205319 0 0) 0x7fbcb80b9690 con 0x7fbcb80d1110 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbca0003520 con 0x7fbcb80d1110 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 997872599 0 0) 0x7fbcb80ca9e0 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb80b9690 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.237+0000 7fbcb77fe640 1 -- 172.21.15.46:0/3691655496 >> v1:172.21.15.88:6789/0 conn(0x7fbcb80cfcf0 legacy=0x7fbcb809cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.238+0000 7fbcb77fe640 1 -- 172.21.15.46:0/3691655496 >> v1:172.21.15.46:6789/0 conn(0x7fbcb80d1110 legacy=0x7fbcb80d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.238+0000 7fbcb77fe640 1 -- 172.21.15.46:0/3691655496 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb808cfb0 con 0x7fbcb80d1740 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.238+0000 7fbcb77fe640 1 --2- 172.21.15.46:0/3691655496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1740 0x7fbcb80b73d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbca80097a0 tx=0x7fbca802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=899540602518146b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8037070 con 0x7fbcb80d1740 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca8032440 con 0x7fbcb80d1740 2024-05-09T03:19:52.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcb67fc640 1 -- 172.21.15.46:0/3691655496 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8032810 con 0x7fbcb80d1740 2024-05-09T03:19:52.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 -- 172.21.15.46:0/3691655496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1740 msgr2=0x7fbcb80b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/3691655496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1740 0x7fbcb80b73d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbca80097a0 tx=0x7fbca802a230 comp rx=0 tx=0).stop 2024-05-09T03:19:52.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 -- 172.21.15.46:0/3691655496 shutdown_connections 2024-05-09T03:19:52.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/3691655496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1740 0x7fbcb80b73d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 -- 172.21.15.46:0/3691655496 >> 172.21.15.46:0/3691655496 conn(0x7fbcb80d05f0 msgr2=0x7fbcb80d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:52.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 -- 172.21.15.46:0/3691655496 shutdown_connections 2024-05-09T03:19:52.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.239+0000 7fbcbdc47640 1 -- 172.21.15.46:0/3691655496 wait complete. 2024-05-09T03:19:52.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.241+0000 7fbcbdc47640 1 Processor -- start 2024-05-09T03:19:52.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.241+0000 7fbcbdc47640 1 -- start start 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcbdc47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 0x7fbcb814fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcbdc47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1110 0x7fbcb8151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcbdc47640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcb80d1740 0x7fbcb8152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcbdc47640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb80b93b0 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcbdc47640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb808cd60 con 0x7fbcb80d1110 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcbdc47640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb809d3b0 con 0x7fbcb80d1740 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 0x7fbcb814fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:52.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 0x7fbcb814fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60974/0 (socket says 172.21.15.46:60974) 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 -- 172.21.15.46:0/1759290390 learned_addr learned my addr 172.21.15.46:0/1759290390 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb6ffd640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1110 0x7fbcb8151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb7fff640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcb80d1740 0x7fbcb8152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 -- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcb80d1740 msgr2=0x7fbcb8152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcb80d1740 0x7fbcb8152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 -- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1110 msgr2=0x7fbcb8151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1110 0x7fbcb8151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 -- 172.21.15.46:0/1759290390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb8150b40 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.243+0000 7fbcb77fe640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 0x7fbcb814fed0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fbca8000c00 tx=0x7fbca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.244+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8041020 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.244+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca803d640 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.244+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb8150e20 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.244+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8037040 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.244+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb81513e0 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.244+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb80b65b0 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.246+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbca8044020 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.246+0000 7fbc9ffff640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca4076300 0x7fbca4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:52.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.246+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fbca80b8e50 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.246+0000 7fbcb6ffd640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca4076300 0x7fbca4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:52.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.247+0000 7fbcb6ffd640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca4076300 0x7fbca4078810 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80b9280 tx=0x7fbcac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:52.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.250+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbcb80b65b0 con 0x7fbcb80cfcf0 2024-05-09T03:19:52.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:52.520+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fbcb806a270 con 0x7fbcb80cfcf0 2024-05-09T03:19:53.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.104+0000 7fbc9ffff640 1 -- 172.21.15.46:0/1759290390 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v302) ==== 107+0+0 (secure 0 0 0) 0x7fbca804b070 con 0x7fbcb80cfcf0 2024-05-09T03:19:53.105 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:19:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca4076300 msgr2=0x7fbca4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca4076300 0x7fbca4078810 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80b9280 tx=0x7fbcac009040 comp rx=0 tx=0).stop 2024-05-09T03:19:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 msgr2=0x7fbcb814fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 0x7fbcb814fed0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fbca8000c00 tx=0x7fbca80046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 shutdown_connections 2024-05-09T03:19:53.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcb80d1740 0x7fbcb8152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcb80d1110 0x7fbcb8151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca4076300 0x7fbca4078810 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 --2- 172.21.15.46:0/1759290390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcb80cfcf0 0x7fbcb814fed0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 >> 172.21.15.46:0/1759290390 conn(0x7fbcb80d05f0 msgr2=0x7fbcb809bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:53.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 shutdown_connections 2024-05-09T03:19:53.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.108+0000 7fbcbdc47640 1 -- 172.21.15.46:0/1759290390 wait complete. 2024-05-09T03:19:53.123 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:19:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.182+0000 7f7ceb528f40 1 Processor -- start 2024-05-09T03:19:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.182+0000 7f7ceb528f40 1 -- start start 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ceb528f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 0x56007a1b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ceb528f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a122c00 0x56007a10c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ceb528f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56007a1274b0 0x56007a1b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ceb528f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56007a083f50 con 0x7f7ccc003680 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ceb528f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56007a087f90 con 0x56007a122c00 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ceb528f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56007a0de400 con 0x56007a1274b0 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ce4ff2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 0x56007a1b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ce4ff2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 0x56007a1b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60996/0 (socket says 172.21.15.46:60996) 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.183+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3957413995 learned_addr learned my addr 172.21.15.46:0/3957413995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a122c00 0x56007a10c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56007a1274b0 msgr2=0x56007a1b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:53.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56007a1274b0 0x56007a1b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a122c00 msgr2=0x56007a10c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a122c00 0x56007a10c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3957413995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007a1b25d0 con 0x7f7ccc003680 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a122c00 0x56007a10c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 0x56007a1b1f30 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400a7b0 tx=0x7f7cd4005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0fb546d0511e5b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce37ef640 1 -- 172.21.15.46:0/3957413995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4017030 con 0x7f7ccc003680 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce37ef640 1 -- 172.21.15.46:0/3957413995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd4010ab0 con 0x7f7ccc003680 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.184+0000 7f7ce37ef640 1 -- 172.21.15.46:0/3957413995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4010c60 con 0x7f7ccc003680 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3957413995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 msgr2=0x56007a1b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 0x56007a1b1f30 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400a7b0 tx=0x7f7cd4005d80 comp rx=0 tx=0).stop 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3957413995 shutdown_connections 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56007a1274b0 0x56007a1b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a122c00 0x56007a10c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3957413995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc003680 0x56007a1b1f30 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3957413995 >> 172.21.15.46:0/3957413995 conn(0x560079fffac0 msgr2=0x56007a123150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3957413995 shutdown_connections 2024-05-09T03:19:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.185+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3957413995 wait complete. 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.186+0000 7f7ceb528f40 1 Processor -- start 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.186+0000 7f7ceb528f40 1 -- start start 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ceb528f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56007a122c00 0x56007a1ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ceb528f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a1274b0 0x56007a10acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ceb528f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ccc003680 0x56007a10b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ceb528f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56007a1b4b60 con 0x56007a122c00 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ceb528f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56007a11d1a0 con 0x56007a1274b0 2024-05-09T03:19:53.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ceb528f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x56007a1b4640 con 0x7f7ccc003680 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ce4ff2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56007a122c00 0x56007a1ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ce4ff2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56007a122c00 0x56007a1ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32774/0 (socket says 172.21.15.46:32774) 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 learned_addr learned my addr 172.21.15.46:0/3616029064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a1274b0 0x56007a10acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ce57f3640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ccc003680 0x56007a10b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.187+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ccc003680 msgr2=0x56007a10b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ccc003680 0x56007a10b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a1274b0 msgr2=0x56007a10acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a1274b0 0x56007a10acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007a106980 con 0x56007a122c00 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56007a1274b0 0x56007a10acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce57f3640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ccc003680 0x56007a10b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56007a122c00 0x56007a1ae5d0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4010690 tx=0x7f7cd4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce27ed640 1 -- 172.21.15.46:0/3616029064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400a420 con 0x56007a122c00 2024-05-09T03:19:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce27ed640 1 -- 172.21.15.46:0/3616029064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd4005a40 con 0x56007a122c00 2024-05-09T03:19:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56007a1b42d0 con 0x56007a122c00 2024-05-09T03:19:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ce27ed640 1 -- 172.21.15.46:0/3616029064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4026460 con 0x56007a122c00 2024-05-09T03:19:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.188+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007a11cee0 con 0x56007a122c00 2024-05-09T03:19:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.190+0000 7f7ce27ed640 1 -- 172.21.15.46:0/3616029064 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7cd401a030 con 0x56007a122c00 2024-05-09T03:19:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.190+0000 7f7ce27ed640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cd0074ce0 0x7f7cd00771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.190+0000 7f7ce27ed640 1 -- 172.21.15.46:0/3616029064 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7f7cd4098bd0 con 0x56007a122c00 2024-05-09T03:19:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.191+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cd0074ce0 0x7f7cd00771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.193 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:19:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.191+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56007a11c370 0x56007a11c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.191+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a255000 con 0x56007a11c370 2024-05-09T03:19:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.191+0000 7f7ce57f3640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56007a11c370 0x56007a11c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.191+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cd0074ce0 0x7f7cd00771f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x56007a24cd60 tx=0x7f7cd80011b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.191+0000 7f7ce57f3640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56007a11c370 0x56007a11c780 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.194+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v118'1033 uv1007 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.194+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a2553a0 con 0x56007a11c370 2024-05-09T03:19:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.196+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v118'1034 uv1034 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.196+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a255740 con 0x56007a11c370 2024-05-09T03:19:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.197+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v118'1035 uv1035 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.198 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:19:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.197+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256af0 con 0x56007a11c370 2024-05-09T03:19:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.198+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v118'1036 uv1036 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.198+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256e90 con 0x56007a11c370 2024-05-09T03:19:53.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.200+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v118'1037 uv1037 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.200+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a255ae0 con 0x56007a11c370 2024-05-09T03:19:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.201+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v118'1038 uv1038 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.201+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a255ed0 con 0x56007a11c370 2024-05-09T03:19:53.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.203+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v118'1039 uv1013 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.203+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.205+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v118'1040 uv1040 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.205+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.206+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v118'1041 uv1041 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.207 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:19:53.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.206+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.208+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v118'1042 uv1042 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.208+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.209+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v118'1043 uv1043 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.209+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.211+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v118'1044 uv1044 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.211+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.212+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v118'1045 uv1020 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.213+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.214+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v118'1046 uv1046 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.214+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.216 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:19:53.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.215+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v118'1047 uv1047 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.215+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.217+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v118'1048 uv1048 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.217+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.218+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v118'1049 uv1049 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.218+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.220+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v118'1050 uv1050 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.220+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.222+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v118'1051 uv1051 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.222+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.223+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v118'1052 uv1026 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.223+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.225+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v118'1053 uv1053 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.225+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.228+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v118'1054 uv1054 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.228 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:19:53.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.228+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.229+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v118'1055 uv1055 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.229+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.231+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v118'1056 uv1056 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.231+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.232+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v118'1057 uv1057 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.232+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.234+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v118'1058 uv1032 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.234+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.235+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v118'1059 uv1059 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.235+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.237 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:19:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.236+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v118'1060 uv1060 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.237+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.238+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v118'1061 uv1061 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.238+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.239+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v118'1062 uv1062 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.239+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a11c370 2024-05-09T03:19:53.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.241+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v118'1063 uv1063 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.241+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56007a2566a0 0x56007a259750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.241+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a256270 con 0x56007a2566a0 2024-05-09T03:19:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.241+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56007a2566a0 0x56007a259750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.241+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56007a2566a0 0x56007a259750 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.243+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v118'745 uv725 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.243+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a259c90 con 0x56007a2566a0 2024-05-09T03:19:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.245+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v118'746 uv746 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.245+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25a030 con 0x56007a2566a0 2024-05-09T03:19:53.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.246+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v118'747 uv747 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.247 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:19:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.246+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25a880 con 0x56007a2566a0 2024-05-09T03:19:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.247+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v118'748 uv748 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.247+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25ac20 con 0x56007a2566a0 2024-05-09T03:19:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.248+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v118'749 uv749 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.248+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25afc0 con 0x56007a2566a0 2024-05-09T03:19:53.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.250+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v118'750 uv744 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.250+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a25b360 con 0x56007a2566a0 2024-05-09T03:19:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.251+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v118'751 uv751 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.251+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.252+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v118'752 uv752 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.253 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:19:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.252+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.253+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v118'753 uv753 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.253+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.255+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v118'754 uv754 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.255+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.256+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v118'755 uv743 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.256+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.258+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v118'756 uv756 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.258+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.259+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v118'757 uv757 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.260 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:19:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.259+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25f160 con 0x56007a2566a0 2024-05-09T03:19:53.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.261+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v118'758 uv758 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.261+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25f160 con 0x56007a2566a0 2024-05-09T03:19:53.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.263+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v118'759 uv759 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.263+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.264+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v118'760 uv741 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.264+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.265+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v118'761 uv761 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.266+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.267+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v118'762 uv762 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.270 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:19:53.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.267+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25c710 con 0x56007a2566a0 2024-05-09T03:19:53.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.268+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v118'763 uv763 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.268+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25c710 con 0x56007a2566a0 2024-05-09T03:19:53.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.269+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v118'764 uv764 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.269+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a2566a0 2024-05-09T03:19:53.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.271+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v118'765 uv765 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a2566a0 2024-05-09T03:19:53.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.271+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a11c370 2024-05-09T03:19:53.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.272+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v118'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.272+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a11c370 2024-05-09T03:19:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.273+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v118'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.273+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a11c370 2024-05-09T03:19:53.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.274+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v118'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.275 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:19:53.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.274+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25c710 con 0x56007a11c370 2024-05-09T03:19:53.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.275+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v118'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.275+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25c710 con 0x56007a11c370 2024-05-09T03:19:53.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.276+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v118'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.276+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25b700 con 0x56007a11c370 2024-05-09T03:19:53.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.277+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v118'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.277+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56007a25bc10 0x56007a25e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.277+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25e570 con 0x56007a25bc10 2024-05-09T03:19:53.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.277+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56007a25bc10 0x56007a25e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.278+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56007a25bc10 0x56007a25e030 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.279+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v118'151 uv150 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.279+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a25ea60 con 0x56007a25bc10 2024-05-09T03:19:53.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.280+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v118'152 uv152 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.280+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25ef60 con 0x56007a25bc10 2024-05-09T03:19:53.281 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:19:53.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.281+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v118'153 uv153 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.281+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25a4b0 con 0x56007a25bc10 2024-05-09T03:19:53.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.282+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v118'154 uv154 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.282+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25f890 con 0x56007a11c370 2024-05-09T03:19:53.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.284+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v118'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.284+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a25f890 con 0x56007a11c370 2024-05-09T03:19:53.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.285+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v118'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.285+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25f890 con 0x56007a11c370 2024-05-09T03:19:53.286 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:19:53.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.286+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v118'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.286+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a2608e0 con 0x56007a11c370 2024-05-09T03:19:53.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.287+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v118'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.287+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a2608a0 con 0x56007a11c370 2024-05-09T03:19:53.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.288+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v118'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.288+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25f890 con 0x56007a11c370 2024-05-09T03:19:53.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.289+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v118'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.289+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56007a260160 0x56007a260570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.289+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.289+0000 7f7ce57f3640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56007a260160 0x56007a260570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.290+0000 7f7ce57f3640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56007a260160 0x56007a260570 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.291+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v118'371 uv365 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.291+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.292+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v118'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.292+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.294 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:19:53.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.293+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v118'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.293+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a263d80 con 0x56007a260160 2024-05-09T03:19:53.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.294+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v118'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.294+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a263d80 con 0x56007a260160 2024-05-09T03:19:53.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.296+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v118'375 uv375 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.296+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.297+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v118'376 uv376 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.297+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.298+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v118'377 uv370 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.298+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.299+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v118'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.299+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.300+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v118'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.301 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:19:53.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.300+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.302+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v118'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.302+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a262d70 con 0x56007a260160 2024-05-09T03:19:53.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.303+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v118'381 uv381 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.303+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56007a263690 0x56007a263aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.303+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266170 con 0x56007a263690 2024-05-09T03:19:53.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.303+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56007a263690 0x56007a263aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.303+0000 7f7ce4ff2640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56007a263690 0x56007a263aa0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.305+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v118'168 uv167 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a263690 2024-05-09T03:19:53.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.305+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a266170 con 0x56007a263690 2024-05-09T03:19:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.306+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v118'169 uv169 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a263690 2024-05-09T03:19:53.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.306+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a2661f0 con 0x56007a263690 2024-05-09T03:19:53.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.307+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v118'170 uv170 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a263690 2024-05-09T03:19:53.308 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:19:53.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.307+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266610 con 0x56007a263690 2024-05-09T03:19:53.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.308+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v118'171 uv171 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a263690 2024-05-09T03:19:53.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.308+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266610 con 0x56007a263690 2024-05-09T03:19:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.309+0000 7f7ce4ff2640 1 -- 172.21.15.46:0/3616029064 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v118'172 uv172 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd405d610 con 0x56007a263690 2024-05-09T03:19:53.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.309+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a2662c0 con 0x56007a11c370 2024-05-09T03:19:53.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.311+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v118'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.311+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a2662c0 con 0x56007a11c370 2024-05-09T03:19:53.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.312+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v118'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.312+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a2662c0 con 0x56007a11c370 2024-05-09T03:19:53.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.313+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v118'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.313 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:19:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.313+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266300 con 0x56007a11c370 2024-05-09T03:19:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.314+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v118'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.314+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a267820 con 0x56007a11c370 2024-05-09T03:19:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.315+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v118'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.315+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266700 con 0x56007a11c370 2024-05-09T03:19:53.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.316+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v118'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a11c370 2024-05-09T03:19:53.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.316+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266700 con 0x56007a25bc10 2024-05-09T03:19:53.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.317+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v118'335 uv329 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.317+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a266b60 con 0x56007a25bc10 2024-05-09T03:19:53.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.318+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v118'336 uv336 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.318+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a266f00 con 0x56007a25bc10 2024-05-09T03:19:53.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.319+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v118'337 uv337 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd8010a70 con 0x56007a25bc10 2024-05-09T03:19:53.320 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:19:53.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.319+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a267390 con 0x56007a25bc10 2024-05-09T03:19:53.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.320+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v118'338 uv338 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.320+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a267390 con 0x56007a25bc10 2024-05-09T03:19:53.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.321+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v118'339 uv339 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.322+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a267390 con 0x56007a25bc10 2024-05-09T03:19:53.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.323+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v118'340 uv334 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.323+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a267390 con 0x56007a25bc10 2024-05-09T03:19:53.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.324+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v118'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.324+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a267390 con 0x56007a25bc10 2024-05-09T03:19:53.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.325+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v118'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.326 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:19:53.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.325+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a267390 con 0x56007a25bc10 2024-05-09T03:19:53.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.326+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v118'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.326+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a25f480 con 0x56007a25bc10 2024-05-09T03:19:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.327+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v118'344 uv344 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.327+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56007a268160 0x56007a26a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.327+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a268160 2024-05-09T03:19:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.327+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56007a268160 0x56007a26a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.328+0000 7f7ce47f1640 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56007a268160 0x56007a26a8d0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.329+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v118'147 uv145 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a268160 2024-05-09T03:19:53.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.329+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a268160 2024-05-09T03:19:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.330+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v118'148 uv148 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a268160 2024-05-09T03:19:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.330+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a268160 2024-05-09T03:19:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.331+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v118'149 uv149 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a268160 2024-05-09T03:19:53.332 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:19:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.331+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a268160 2024-05-09T03:19:53.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.332+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v118'150 uv150 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a268160 2024-05-09T03:19:53.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.332+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a25bc10 2024-05-09T03:19:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.333+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v118'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.333+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a25bc10 2024-05-09T03:19:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.334+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v118'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.335+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a25bc10 2024-05-09T03:19:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.336+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v118'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.336 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:19:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.336+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26be20 con 0x56007a25bc10 2024-05-09T03:19:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.337+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v118'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.337+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26be20 con 0x56007a25bc10 2024-05-09T03:19:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.338+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v118'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.338+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a25bc10 2024-05-09T03:19:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.339+0000 7f7ce47f1640 1 -- 172.21.15.46:0/3616029064 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v118'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cd8016040 con 0x56007a25bc10 2024-05-09T03:19:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.339+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.340+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v118'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.340+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.342+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v118'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.342+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.343 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:19:53.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.343+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v118'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.343+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.344+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v118'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.344+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.345+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v118'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.346+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.347+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v118'416 uv406 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.347+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.348+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v118'417 uv417 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.348+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.350 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:19:53.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.349+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v118'418 uv418 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.349+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.350+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v118'419 uv419 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.350+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.352+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v118'420 uv420 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.352+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.353+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v118'421 uv414 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.353+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.354+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v118'422 uv422 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.354+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.355+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v118'423 uv423 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.355 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:19:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.355+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26be20 con 0x56007a260160 2024-05-09T03:19:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.356+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v118'424 uv424 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.356+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26be20 con 0x56007a260160 2024-05-09T03:19:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.357+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v118'425 uv425 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.357+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26ae10 con 0x56007a260160 2024-05-09T03:19:53.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.358+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v118'426 uv426 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.358+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x56007a26b2c0 con 0x56007a260160 2024-05-09T03:19:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v118'427 uv427 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7cdc0527c0 con 0x56007a260160 2024-05-09T03:19:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56007a11c370 msgr2=0x56007a11c780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x56007a11c370 0x56007a11c780 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56007a25bc10 msgr2=0x56007a25e030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x56007a25bc10 0x56007a25e030 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56007a268160 msgr2=0x56007a26a8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x56007a268160 0x56007a26a8d0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56007a2566a0 msgr2=0x56007a259750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 reap_dead start 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x56007a2566a0 0x56007a259750 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56007a260160 msgr2=0x56007a260570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x56007a260160 0x56007a260570 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56007a263690 msgr2=0x56007a263aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x56007a263690 0x56007a263aa0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cd0074ce0 msgr2=0x7f7cd00771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7cd0074ce0 0x7f7cd00771f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x56007a24cd60 tx=0x7f7cd80011b0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56007a122c00 msgr2=0x56007a1ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 --2- 172.21.15.46:0/3616029064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56007a122c00 0x56007a1ae5d0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4010690 tx=0x7f7cd4003330 comp rx=0 tx=0).stop 2024-05-09T03:19:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ce57f3640 1 -- 172.21.15.46:0/3616029064 reap_dead start 2024-05-09T03:19:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 shutdown_connections 2024-05-09T03:19:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 >> 172.21.15.46:0/3616029064 conn(0x560079fffac0 msgr2=0x56007a10c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 shutdown_connections 2024-05-09T03:19:53.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.359+0000 7f7ceb528f40 1 -- 172.21.15.46:0/3616029064 wait complete. 2024-05-09T03:19:53.363 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 10000000000.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:19:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.459+0000 7f52a9908f40 1 Processor -- start 2024-05-09T03:19:53.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.459+0000 7f52a9908f40 1 -- start start 2024-05-09T03:19:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a9908f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c177e0 0x559805c48a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a9908f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 0x559805c62c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a9908f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805cef780 0x559805c631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a9908f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559805c59cb0 con 0x559805c46db0 2024-05-09T03:19:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a9908f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559805bc0f50 con 0x559805cef780 2024-05-09T03:19:53.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a9908f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559805c17bf0 con 0x559805c177e0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 0x559805c62c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 0x559805c62c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32786/0 (socket says 172.21.15.46:32786) 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 -- 172.21.15.46:0/3527071723 learned_addr learned my addr 172.21.15.46:0/3527071723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a3bd3640 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805cef780 0x559805c631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a33d2640 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c177e0 0x559805c48a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 -- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c177e0 msgr2=0x559805c48a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c177e0 0x559805c48a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 -- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805cef780 msgr2=0x559805c631d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805cef780 0x559805c631d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 -- 172.21.15.46:0/3527071723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559805c42860 con 0x559805c46db0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a33d2640 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c177e0 0x559805c48a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a2bd1640 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 0x559805c62c90 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f528c025b40 tx=0x7f528c04e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245c98efe5076c53 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a1bcf640 1 -- 172.21.15.46:0/3527071723 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c065070 con 0x559805c46db0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a1bcf640 1 -- 172.21.15.46:0/3527071723 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528c0549f0 con 0x559805c46db0 2024-05-09T03:19:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.461+0000 7f52a1bcf640 1 -- 172.21.15.46:0/3527071723 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c059040 con 0x559805c46db0 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 -- 172.21.15.46:0/3527071723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 msgr2=0x559805c62c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 0x559805c62c90 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f528c025b40 tx=0x7f528c04e000 comp rx=0 tx=0).stop 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 -- 172.21.15.46:0/3527071723 shutdown_connections 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805cef780 0x559805c631d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805c46db0 0x559805c62c90 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 --2- 172.21.15.46:0/3527071723 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c177e0 0x559805c48a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 -- 172.21.15.46:0/3527071723 >> 172.21.15.46:0/3527071723 conn(0x559805b3cac0 msgr2=0x559805cd23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 -- 172.21.15.46:0/3527071723 shutdown_connections 2024-05-09T03:19:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.462+0000 7f52a9908f40 1 -- 172.21.15.46:0/3527071723 wait complete. 2024-05-09T03:19:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.463+0000 7f52a9908f40 1 Processor -- start 2024-05-09T03:19:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.463+0000 7f52a9908f40 1 -- start start 2024-05-09T03:19:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.464+0000 7f52a9908f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805c177e0 0x559805d86750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.464+0000 7f52a9908f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c46db0 0x559805ceeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.464+0000 7f52a9908f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805cef780 0x559805cef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.464+0000 7f52a9908f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559805c43590 con 0x559805cef780 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.464+0000 7f52a9908f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559805c456f0 con 0x559805c177e0 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.464+0000 7f52a9908f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559805c5b820 con 0x559805c46db0 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805cef780 0x559805cef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805cef780 0x559805cef0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32788/0 (socket says 172.21.15.46:32788) 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 -- 172.21.15.46:0/705075047 learned_addr learned my addr 172.21.15.46:0/705075047 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a33d2640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805c177e0 0x559805d86750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a2bd1640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c46db0 0x559805ceeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 -- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c46db0 msgr2=0x559805ceeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559805c46db0 0x559805ceeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 -- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805c177e0 msgr2=0x559805d86750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559805c177e0 0x559805d86750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 -- 172.21.15.46:0/705075047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559805cd2770 con 0x559805cef780 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a3bd3640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805cef780 0x559805cef0b0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f5294005e50 tx=0x7f52940019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a0bcd640 1 -- 172.21.15.46:0/705075047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52940123f0 con 0x559805cef780 2024-05-09T03:19:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a0bcd640 1 -- 172.21.15.46:0/705075047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5294017070 con 0x559805cef780 2024-05-09T03:19:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559805cd2490 con 0x559805cef780 2024-05-09T03:19:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a0bcd640 1 -- 172.21.15.46:0/705075047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52940127d0 con 0x559805cef780 2024-05-09T03:19:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.465+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559805c3f3f0 con 0x559805cef780 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.467+0000 7f52a0bcd640 1 -- 172.21.15.46:0/705075047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5294012970 con 0x559805cef780 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.467+0000 7f52a0bcd640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5284074cb0 0x7f52840771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a0bcd640 1 -- 172.21.15.46:0/705075047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7f5294096b00 con 0x559805cef780 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a33d2640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5284074cb0 0x7f52840771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a9908f40 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559805c5b9a0 0x559805c41d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x559805d86c90 con 0x559805c5b9a0 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a2bd1640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559805c5b9a0 0x559805c41d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a33d2640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5284074cb0 0x7f52840771c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x559805c1c440 tx=0x7f529c005aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.468+0000 7f52a2bd1640 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559805c5b9a0 0x559805c41d20 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a2bd1640 1 -- 172.21.15.46:0/705075047 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-header] v118'1064 uv1064 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f528c068040 con 0x559805c5b9a0 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559805c5b9a0 msgr2=0x559805c41d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x559805c5b9a0 0x559805c41d20 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5284074cb0 msgr2=0x7f52840771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5284074cb0 0x7f52840771c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x559805c1c440 tx=0x7f529c005aa0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805cef780 msgr2=0x559805cef0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 --2- 172.21.15.46:0/705075047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559805cef780 0x559805cef0b0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f5294005e50 tx=0x7f52940019a0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a3bd3640 1 -- 172.21.15.46:0/705075047 reap_dead start 2024-05-09T03:19:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 shutdown_connections 2024-05-09T03:19:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 >> 172.21.15.46:0/705075047 conn(0x559805b3cac0 msgr2=0x559805cd2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 shutdown_connections 2024-05-09T03:19:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.470+0000 7f52a9908f40 1 -- 172.21.15.46:0/705075047 wait complete. 2024-05-09T03:19:53.474 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:19:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.590+0000 7f1ae139d640 1 Processor -- start 2024-05-09T03:19:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.590+0000 7f1ae139d640 1 -- start start 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.591+0000 7f1ae139d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1adc0b5140 0x7f1adc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ae139d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0d3fc0 0x7f1adc0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ae139d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1adc0b80b0 con 0x7f1adc0b7ca0 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ae139d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1adc0d3480 con 0x7f1adc0b5140 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ae139d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1adc0b5550 con 0x7f1adc0d3fc0 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adaffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1adc0b5140 0x7f1adc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adb7fe640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0d3fc0 0x7f1adc0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ada7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1adc0b7ca0 0x7f1adc0d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:39764/0 (socket says 172.21.15.46:39764) 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adaffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1adc0b5140 0x7f1adc0b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:43712/0 (socket says 172.21.15.46:43712) 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ada7fc640 1 -- 172.21.15.46:0/595729477 learned_addr learned my addr 172.21.15.46:0/595729477 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:19:53.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adaffd640 1 -- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0d3fc0 msgr2=0x7f1adc0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adaffd640 1 --2- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0d3fc0 0x7f1adc0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adaffd640 1 -- 172.21.15.46:0/595729477 >> v1:172.21.15.88:6789/0 conn(0x7f1adc0b7ca0 legacy=0x7f1adc0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adaffd640 1 -- 172.21.15.46:0/595729477 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc0d44e0 con 0x7f1adc0b5140 2024-05-09T03:19:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1ad9ffb640 1 -- 172.21.15.46:0/595729477 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925331077 0 0) 0x7f1ad0001110 con 0x7f1adc0b7ca0 2024-05-09T03:19:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.592+0000 7f1adb7fe640 1 --2- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0d3fc0 0x7f1adc0b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:19:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.593+0000 7f1adaffd640 1 --2- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0b5140 0x7f1adc0b7760 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0097a0 tx=0x7f1acc02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc20da08713d151 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.593+0000 7f1ad9ffb640 1 -- 172.21.15.46:0/595729477 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc037070 con 0x7f1adc0b5140 2024-05-09T03:19:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.593+0000 7f1ad9ffb640 1 -- 172.21.15.46:0/595729477 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1acc032440 con 0x7f1adc0b5140 2024-05-09T03:19:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.593+0000 7f1ad9ffb640 1 -- 172.21.15.46:0/595729477 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc032810 con 0x7f1adc0b5140 2024-05-09T03:19:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 -- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0b5140 msgr2=0x7f1adc0b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 --2- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0b5140 0x7f1adc0b7760 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f1acc0097a0 tx=0x7f1acc02a210 comp rx=0 tx=0).stop 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 -- 172.21.15.46:0/595729477 shutdown_connections 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 --2- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0d3fc0 0x7f1adc0b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 --2- 172.21.15.46:0/595729477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0b5140 0x7f1adc0b7760 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 -- 172.21.15.46:0/595729477 >> 172.21.15.46:0/595729477 conn(0x7f1adc08d3f0 msgr2=0x7f1adc08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 -- 172.21.15.46:0/595729477 shutdown_connections 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.594+0000 7f1ae139d640 1 -- 172.21.15.46:0/595729477 wait complete. 2024-05-09T03:19:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.596+0000 7f1ae139d640 1 Processor -- start 2024-05-09T03:19:53.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.596+0000 7f1ae139d640 1 -- start start 2024-05-09T03:19:53.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1ae139d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 0x7f1adc15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1ae139d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0b7ca0 0x7f1adc156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1ae139d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0d3fc0 0x7f1adc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1ae139d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc0d4d80 con 0x7f1adc0b5140 2024-05-09T03:19:53.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1ae139d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1adc0d4aa0 con 0x7f1adc0d3fc0 2024-05-09T03:19:53.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1ae139d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1adc0d5580 con 0x7f1adc0b7ca0 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1adaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 0x7f1adc15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1adaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 0x7f1adc15bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32798/0 (socket says 172.21.15.46:32798) 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.597+0000 7f1adaffd640 1 -- 172.21.15.46:0/1376004303 learned_addr learned my addr 172.21.15.46:0/1376004303 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adb7fe640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0d3fc0 0x7f1adc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1ada7fc640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0b7ca0 0x7f1adc156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adaffd640 1 -- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0b7ca0 msgr2=0x7f1adc156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adaffd640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0b7ca0 0x7f1adc156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adaffd640 1 -- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0d3fc0 msgr2=0x7f1adc156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adaffd640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0d3fc0 0x7f1adc156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adaffd640 1 -- 172.21.15.46:0/1376004303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1adc159860 con 0x7f1adc0b5140 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1adaffd640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 0x7f1adc15bce0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f1acc000c00 tx=0x7f1acc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc037050 con 0x7f1adc0b5140 2024-05-09T03:19:53.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1acc038040 con 0x7f1adc0b5140 2024-05-09T03:19:53.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1adc159b40 con 0x7f1adc0b5140 2024-05-09T03:19:53.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1acc004470 con 0x7f1adc0b5140 2024-05-09T03:19:53.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.598+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1adc15a100 con 0x7f1adc0b5140 2024-05-09T03:19:53.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.599+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1adc06a270 con 0x7f1adc0b5140 2024-05-09T03:19:53.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.600+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1acc03c6d0 con 0x7f1adc0b5140 2024-05-09T03:19:53.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.600+0000 7f1ac37fe640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ac80762d0 0x7f1ac80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:53.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.601+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7f1acc0b8a30 con 0x7f1adc0b5140 2024-05-09T03:19:53.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.601+0000 7f1ada7fc640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ac80762d0 0x7f1ac80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:53.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.601+0000 7f1ada7fc640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ac80762d0 0x7f1ac80787e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1adc157e50 tx=0x7f1ad0007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:53.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.605+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1acc04c020 con 0x7f1adc0b5140 2024-05-09T03:19:53.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:53.835+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f1adc160670 con 0x7f1adc0b5140 2024-05-09T03:19:54.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.106+0000 7f1ac37fe640 1 -- 172.21.15.46:0/1376004303 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v303) ==== 162+0+0 (secure 0 0 0) 0x7f1acc081050 con 0x7f1adc0b5140 2024-05-09T03:19:54.107 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ac80762d0 msgr2=0x7f1ac80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ac80762d0 0x7f1ac80787e0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1adc157e50 tx=0x7f1ad0007de0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 msgr2=0x7f1adc15bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 0x7f1adc15bce0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f1acc000c00 tx=0x7f1acc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 shutdown_connections 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc0d3fc0 0x7f1adc156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1adc0b7ca0 0x7f1adc156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ac80762d0 0x7f1ac80787e0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 --2- 172.21.15.46:0/1376004303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1adc0b5140 0x7f1adc15bce0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 >> 172.21.15.46:0/1376004303 conn(0x7f1adc08d3f0 msgr2=0x7f1adc1630f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:54.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 shutdown_connections 2024-05-09T03:19:54.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.109+0000 7f1ae139d640 1 -- 172.21.15.46:0/1376004303 wait complete. 2024-05-09T03:19:54.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:19:54.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.646+0000 7fc99e186640 1 Processor -- start 2024-05-09T03:19:54.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.646+0000 7fc99e186640 1 -- start start 2024-05-09T03:19:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.647+0000 7fc99e186640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9981615e0 0x7fc99815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.647+0000 7fc99e186640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc9980d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.647+0000 7fc99e186640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc998160b20 con 0x7fc998164c70 2024-05-09T03:19:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.647+0000 7fc99e186640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc99815e520 con 0x7fc9981615e0 2024-05-09T03:19:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.647+0000 7fc99e186640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc998161150 con 0x7fc9980d2480 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc996ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9981615e0 0x7fc99815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc997fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc9980d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc996ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc9981615e0 0x7fc99815dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:43740/0 (socket says 172.21.15.46:43740) 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc997fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc9980d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:46528/0 (socket says 172.21.15.46:46528) 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc9977fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc998164c70 0x7fc9980d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:39772/0 (socket says 172.21.15.46:39772) 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc996ffd640 1 -- 172.21.15.46:0/248343570 learned_addr learned my addr 172.21.15.46:0/248343570 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc997fff640 1 -- 172.21.15.46:0/248343570 >> v2:172.21.15.88:3300/0 conn(0x7fc9981615e0 msgr2=0x7fc99815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc997fff640 1 --2- 172.21.15.46:0/248343570 >> v2:172.21.15.88:3300/0 conn(0x7fc9981615e0 0x7fc99815dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc997fff640 1 -- 172.21.15.46:0/248343570 >> v1:172.21.15.88:6789/0 conn(0x7fc998164c70 legacy=0x7fc9980d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.648+0000 7fc997fff640 1 -- 172.21.15.46:0/248343570 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9980d15f0 con 0x7fc9980d2480 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.649+0000 7fc997fff640 1 --2- 172.21.15.46:0/248343570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc9980d2890 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc990009d90 tx=0x7fc99002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f1ed5cba684cbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.649+0000 7fc9967fc640 1 -- 172.21.15.46:0/248343570 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990007740 con 0x7fc9980d2480 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.649+0000 7fc9967fc640 1 -- 172.21.15.46:0/248343570 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9900078e0 con 0x7fc9980d2480 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.649+0000 7fc9967fc640 1 -- 172.21.15.46:0/248343570 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990007cb0 con 0x7fc9980d2480 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 -- 172.21.15.46:0/248343570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 msgr2=0x7fc9980d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 --2- 172.21.15.46:0/248343570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc9980d2890 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc990009d90 tx=0x7fc99002a660 comp rx=0 tx=0).stop 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 -- 172.21.15.46:0/248343570 shutdown_connections 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 --2- 172.21.15.46:0/248343570 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc9980d2890 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 --2- 172.21.15.46:0/248343570 >> v2:172.21.15.88:3300/0 conn(0x7fc9981615e0 0x7fc99815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 -- 172.21.15.46:0/248343570 >> 172.21.15.46:0/248343570 conn(0x7fc998073db0 msgr2=0x7fc9980741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 -- 172.21.15.46:0/248343570 shutdown_connections 2024-05-09T03:19:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.650+0000 7fc99e186640 1 -- 172.21.15.46:0/248343570 wait complete. 2024-05-09T03:19:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.652+0000 7fc99e186640 1 Processor -- start 2024-05-09T03:19:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.652+0000 7fc99e186640 1 -- start start 2024-05-09T03:19:54.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc99e186640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc998152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc99e186640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 0x7fc9980cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc99e186640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc998164c70 0x7fc9980cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc99e186640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc9981619f0 con 0x7fc9981615e0 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc99e186640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc998160de0 con 0x7fc998164c70 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc99e186640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc9980d2dd0 con 0x7fc9980d2480 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc996ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 0x7fc9980cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc996ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 0x7fc9980cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:32812/0 (socket says 172.21.15.46:32812) 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc996ffd640 1 -- 172.21.15.46:0/3267122013 learned_addr learned my addr 172.21.15.46:0/3267122013 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc9977fe640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc998152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.653+0000 7fc997fff640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc998164c70 0x7fc9980cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc996ffd640 1 -- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 msgr2=0x7fc998152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc996ffd640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc998152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc996ffd640 1 -- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc998164c70 msgr2=0x7fc9980cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc996ffd640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc998164c70 0x7fc9980cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc996ffd640 1 -- 172.21.15.46:0/3267122013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9981632e0 con 0x7fc9981615e0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc996ffd640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 0x7fc9980cd3d0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00e870 tx=0x7fc98c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c017070 con 0x7fc9981615e0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c00c8e0 con 0x7fc9981615e0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9981635c0 con 0x7fc9981615e0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00ccd0 con 0x7fc9981615e0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.654+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc998163b80 con 0x7fc9981615e0 2024-05-09T03:19:54.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.655+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9980c9220 con 0x7fc9981615e0 2024-05-09T03:19:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.656+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc98c01a070 con 0x7fc9981615e0 2024-05-09T03:19:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.656+0000 7fc97ffff640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc984076300 0x7fc984078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:19:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.657+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fc98c098410 con 0x7fc9981615e0 2024-05-09T03:19:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.657+0000 7fc9977fe640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc984076300 0x7fc984078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:19:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.657+0000 7fc9977fe640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc984076300 0x7fc984078810 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc988000f60 tx=0x7fc988000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:19:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.661+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc9980c9220 con 0x7fc9981615e0 2024-05-09T03:19:54.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.890+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc998163070 con 0x7fc9981615e0 2024-05-09T03:19:54.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.891+0000 7fc97ffff640 1 -- 172.21.15.46:0/3267122013 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 304 v304) ==== 95+0+5181 (secure 0 0 0) 0x7fc98c060af0 con 0x7fc9981615e0 2024-05-09T03:19:54.892 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:19:54.892 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":304,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":304,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:54.110128+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10035":{"gid":10035,"name":"c","rank":0,"incarnation":304,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/1774919282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1774919282},{"type":"v1","addr":"172.21.15.46:6834","nonce":1774919282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:19:54.892 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 304 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc984076300 msgr2=0x7fc984078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc984076300 0x7fc984078810 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fc988000f60 tx=0x7fc988000bc0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 msgr2=0x7fc9980cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 0x7fc9980cd3d0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00e870 tx=0x7fc98c00ed30 comp rx=0 tx=0).stop 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 shutdown_connections 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc998164c70 0x7fc9980cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc984076300 0x7fc984078810 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9981615e0 0x7fc9980cd3d0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 --2- 172.21.15.46:0/3267122013 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc9980d2480 0x7fc998152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:19:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 >> 172.21.15.46:0/3267122013 conn(0x7fc998073db0 msgr2=0x7fc99815c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:19:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 shutdown_connections 2024-05-09T03:19:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:19:54.893+0000 7fc99e186640 1 -- 172.21.15.46:0/3267122013 wait complete. 2024-05-09T03:19:54.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 304, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:19:54.110128+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10035}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10035': {'gid': 10035, 'name': 'c', 'rank': 0, 'incarnation': 304, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/1774919282', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1774919282}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1774919282}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:19:54.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10035:up:replay 2024-05-09T03:19:54.906 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:19:58.112 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:19:58.111+0000 7fc3c0760640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:20:00.310 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.439+0000 7f6fe73d6640 1 Processor -- start 2024-05-09T03:20:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.439+0000 7f6fe73d6640 1 -- start start 2024-05-09T03:20:00.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe73d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6fe00b92e0 0x7f6fe00b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe73d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 0x7f6fe006d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe73d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe00d2b80 con 0x7f6fe00d0480 2024-05-09T03:20:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe73d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe00d2e60 con 0x7f6fe00b92e0 2024-05-09T03:20:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe73d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe00d3140 con 0x7f6fe00cf6b0 2024-05-09T03:20:00.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe4971640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6fe00d0480 0x7f6fe00d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49420/0 (socket says 172.21.15.46:49420) 2024-05-09T03:20:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe4971640 1 -- 172.21.15.46:0/2193733438 learned_addr learned my addr 172.21.15.46:0/2193733438 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2193733438 >> v2:172.21.15.88:3300/0 conn(0x7f6fe00b92e0 0x7f6fe00b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fe5973640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 0x7f6fe006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513324649 0 0) 0x7f6fe00d2b80 con 0x7f6fe00d0480 2024-05-09T03:20:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.441+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fcc003520 con 0x7f6fe00d0480 2024-05-09T03:20:00.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2687240925 0 0) 0x7f6fcc003520 con 0x7f6fe00d0480 2024-05-09T03:20:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 msgr2=0x7f6fe006d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 0x7f6fe006d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 >> v2:172.21.15.88:3300/0 conn(0x7f6fe00b92e0 msgr2=0x7f6fe00b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 --2- 172.21.15.46:0/2193733438 >> v2:172.21.15.88:3300/0 conn(0x7f6fe00b92e0 0x7f6fe00b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe00d3420 con 0x7f6fe00d0480 2024-05-09T03:20:00.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f6fd00016f0 con 0x7f6fe00d0480 2024-05-09T03:20:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2193733438 >> v2:172.21.15.88:3300/0 conn(0x7f6fe00b92e0 0x7f6fe00b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 >> v1:172.21.15.46:6789/0 conn(0x7f6fe00d0480 legacy=0x7f6fe00d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fcc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 0x7f6fcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fcc004d40 0x7f6fcc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00d2e60 con 0x7f6fcc003a00 2024-05-09T03:20:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00d3140 con 0x7f6fcc004d40 2024-05-09T03:20:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc00baf0 con 0x7f6fe00cf6b0 2024-05-09T03:20:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fcc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe4971640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 0x7f6fcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5973640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fcc004d40 0x7f6fcc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 -- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 msgr2=0x7f6fcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe00cf6b0 0x7f6fcc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 -- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fcc004d40 msgr2=0x7f6fcc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fcc004d40 0x7f6fcc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.442+0000 7f6fe5172640 1 -- 172.21.15.46:0/2193733438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc003520 con 0x7f6fcc003a00 2024-05-09T03:20:00.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.443+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fcc003e70 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009c60 tx=0x7f6fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28136d31067e4f92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.443+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc009860 con 0x7f6fcc003a00 2024-05-09T03:20:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.443+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc004020 con 0x7f6fcc003a00 2024-05-09T03:20:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.443+0000 7f6fd7fff640 1 -- 172.21.15.46:0/2193733438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0043f0 con 0x7f6fcc003a00 2024-05-09T03:20:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2193733438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 msgr2=0x7f6fcc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2193733438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fcc003e70 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009c60 tx=0x7f6fcc005830 comp rx=0 tx=0).stop 2024-05-09T03:20:00.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe5973640 1 -- 172.21.15.46:0/2193733438 reap_dead start 2024-05-09T03:20:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2193733438 shutdown_connections 2024-05-09T03:20:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2193733438 >> 172.21.15.46:0/2193733438 conn(0x7f6fe00d3c00 msgr2=0x7f6fe00d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2193733438 shutdown_connections 2024-05-09T03:20:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.444+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2193733438 wait complete. 2024-05-09T03:20:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.446+0000 7f6fe73d6640 1 Processor -- start 2024-05-09T03:20:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.446+0000 7f6fe73d6640 1 -- start start 2024-05-09T03:20:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe73d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fe0160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe73d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe01f6970 0x7f6fe0159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe73d6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe01626a0 0x7f6fe0162ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe73d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe006db20 con 0x7f6fcc003a00 2024-05-09T03:20:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe73d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe006d9a0 con 0x7f6fe01f6970 2024-05-09T03:20:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe73d6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe006dca0 con 0x7f6fe01626a0 2024-05-09T03:20:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe5172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fe0160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe5172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fe0160440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44574/0 (socket says 172.21.15.46:44574) 2024-05-09T03:20:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.447+0000 7f6fe5172640 1 -- 172.21.15.46:0/2020289680 learned_addr learned my addr 172.21.15.46:0/2020289680 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5973640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe01626a0 0x7f6fe0162ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe4971640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe01f6970 0x7f6fe0159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5172640 1 -- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe01626a0 msgr2=0x7f6fe0162ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe01626a0 0x7f6fe0162ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5172640 1 -- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe01f6970 msgr2=0x7f6fe0159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe01f6970 0x7f6fe0159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5172640 1 -- 172.21.15.46:0/2020289680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe0162ff0 con 0x7f6fcc003a00 2024-05-09T03:20:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe5172640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fe0160440 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc000c00 tx=0x7f6fdc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc040070 con 0x7f6fcc003a00 2024-05-09T03:20:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc003820 con 0x7f6fcc003a00 2024-05-09T03:20:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc009860 con 0x7f6fcc003a00 2024-05-09T03:20:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.448+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe0161d30 con 0x7f6fcc003a00 2024-05-09T03:20:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.449+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe0160f70 con 0x7f6fcc003a00 2024-05-09T03:20:00.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.449+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe006a270 con 0x7f6fcc003a00 2024-05-09T03:20:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.450+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6fdc043070 con 0x7f6fcc003a00 2024-05-09T03:20:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.451+0000 7f6fd5ffb640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fc40762d0 0x7f6fc40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.451+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7f6fdc0b89b0 con 0x7f6fcc003a00 2024-05-09T03:20:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.451+0000 7f6fe4971640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fc40762d0 0x7f6fc40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.452+0000 7f6fe4971640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fc40762d0 0x7f6fc40787e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0001010 tx=0x7f6fd0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.456+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6fe0162ff0 con 0x7f6fcc003a00 2024-05-09T03:20:00.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.683+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fe00d4ae0 con 0x7f6fcc003a00 2024-05-09T03:20:00.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.683+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/2020289680 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 307 v307) ==== 95+0+5190 (secure 0 0 0) 0x7f6fdc081680 con 0x7f6fcc003a00 2024-05-09T03:20:00.685 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:00.685 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":307,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":307,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:19:57.126492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10035},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10035":{"gid":10035,"name":"c","rank":0,"incarnation":304,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6834/1774919282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1774919282},{"type":"v1","addr":"172.21.15.46:6834","nonce":1774919282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10035,"qdb_cluster":[10035]},"id":6}]} 2024-05-09T03:20:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 307 2024-05-09T03:20:00.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fc40762d0 msgr2=0x7f6fc40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fc40762d0 0x7f6fc40787e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0001010 tx=0x7f6fd0009040 comp rx=0 tx=0).stop 2024-05-09T03:20:00.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 msgr2=0x7f6fe0160440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fe0160440 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc000c00 tx=0x7f6fdc004730 comp rx=0 tx=0).stop 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 shutdown_connections 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6fe01626a0 0x7f6fe0162ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fe01f6970 0x7f6fe0159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6fc40762d0 0x7f6fc40787e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 --2- 172.21.15.46:0/2020289680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fcc003a00 0x7f6fe0160440 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 >> 172.21.15.46:0/2020289680 conn(0x7f6fe00d3c00 msgr2=0x7f6fe00d0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 shutdown_connections 2024-05-09T03:20:00.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.685+0000 7f6fe73d6640 1 -- 172.21.15.46:0/2020289680 wait complete. 2024-05-09T03:20:00.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 307, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:19:57.126492+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10035}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10035': {'gid': 10035, 'name': 'c', 'rank': 0, 'incarnation': 304, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6834/1774919282', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1774919282}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 1774919282}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10035, 'qdb_cluster': [10035]} 2024-05-09T03:20:00.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:20:00.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:20:00.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.813+0000 7fefbd4d9640 1 Processor -- start 2024-05-09T03:20:00.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.813+0000 7fefbd4d9640 1 -- start start 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.814+0000 7fefbd4d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fefb80d2f00 0x7fefb80b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.814+0000 7fefbd4d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fefb8071600 0x7fefb8071a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.814+0000 7fefbd4d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefb8071f50 con 0x7fefb8161b80 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.814+0000 7fefbd4d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fefb80ceed0 con 0x7fefb80d2f00 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.814+0000 7fefbd4d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fefb80cf1b0 con 0x7fefb8071600 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fefb80d2f00 0x7fefb80b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fefb80d2f00 0x7fefb80b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44580/0 (socket says 172.21.15.46:44580) 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 -- 172.21.15.46:0/3051022075 learned_addr learned my addr 172.21.15.46:0/3051022075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb77fe640 1 --2- 172.21.15.46:0/3051022075 >> v2:172.21.15.88:3300/0 conn(0x7fefb8071600 0x7fefb8071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 -- 172.21.15.46:0/3051022075 >> v2:172.21.15.88:3300/0 conn(0x7fefb8071600 msgr2=0x7fefb8071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 --2- 172.21.15.46:0/3051022075 >> v2:172.21.15.88:3300/0 conn(0x7fefb8071600 0x7fefb8071a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 -- 172.21.15.46:0/3051022075 >> v1:172.21.15.88:6789/0 conn(0x7fefb8161b80 legacy=0x7fefb80ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.815+0000 7fefb67fc640 1 -- 172.21.15.46:0/3051022075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb80cf490 con 0x7fefb80d2f00 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.816+0000 7fefb67fc640 1 --2- 172.21.15.46:0/3051022075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb80d2f00 0x7fefb80b2f60 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fefb80cedb0 tx=0x7fefac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983d0c3e3eebed45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.816+0000 7fefb5ffb640 1 -- 172.21.15.46:0/3051022075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac037070 con 0x7fefb80d2f00 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.816+0000 7fefb5ffb640 1 -- 172.21.15.46:0/3051022075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac032440 con 0x7fefb80d2f00 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.816+0000 7fefb5ffb640 1 -- 172.21.15.46:0/3051022075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac032810 con 0x7fefb80d2f00 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 -- 172.21.15.46:0/3051022075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb80d2f00 msgr2=0x7fefb80b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/3051022075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb80d2f00 0x7fefb80b2f60 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fefb80cedb0 tx=0x7fefac02a270 comp rx=0 tx=0).stop 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 -- 172.21.15.46:0/3051022075 shutdown_connections 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/3051022075 >> v2:172.21.15.88:3300/0 conn(0x7fefb8071600 0x7fefb8071a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/3051022075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb80d2f00 0x7fefb80b2f60 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 -- 172.21.15.46:0/3051022075 >> 172.21.15.46:0/3051022075 conn(0x7fefb80d3a80 msgr2=0x7fefb80d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 -- 172.21.15.46:0/3051022075 shutdown_connections 2024-05-09T03:20:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.817+0000 7fefbd4d9640 1 -- 172.21.15.46:0/3051022075 wait complete. 2024-05-09T03:20:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.819+0000 7fefbd4d9640 1 Processor -- start 2024-05-09T03:20:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.819+0000 7fefbd4d9640 1 -- start start 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefbd4d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fefb8071600 0x7fefb8143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefbd4d9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fefb80d2f00 0x7fefb8146df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefbd4d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 0x7fefb81e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefbd4d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefb815d6d0 con 0x7fefb8161b80 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefbd4d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fefb80d19d0 con 0x7fefb8071600 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefbd4d9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fefb815d850 con 0x7fefb80d2f00 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 0x7fefb81e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 0x7fefb81e9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44596/0 (socket says 172.21.15.46:44596) 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 -- 172.21.15.46:0/963104786 learned_addr learned my addr 172.21.15.46:0/963104786 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fefb8071600 0x7fefb8143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb67fc640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fefb80d2f00 0x7fefb8146df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 -- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fefb80d2f00 msgr2=0x7fefb8146df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fefb80d2f00 0x7fefb8146df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 -- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fefb8071600 msgr2=0x7fefb8143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fefb8071600 0x7fefb8143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 -- 172.21.15.46:0/963104786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb81599c0 con 0x7fefb8161b80 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.820+0000 7fefb77fe640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 0x7fefb81e9b70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fefb0007d70 tx=0x7fefb000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.821+0000 7fef9f7fe640 1 -- 172.21.15.46:0/963104786 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0014420 con 0x7fefb8161b80 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.821+0000 7fef9f7fe640 1 -- 172.21.15.46:0/963104786 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefb0019070 con 0x7fefb8161b80 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.821+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb8159ca0 con 0x7fefb8161b80 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.821+0000 7fef9f7fe640 1 -- 172.21.15.46:0/963104786 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb0014820 con 0x7fefb8161b80 2024-05-09T03:20:00.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.821+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb815a260 con 0x7fefb8161b80 2024-05-09T03:20:00.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.823+0000 7fef9f7fe640 1 -- 172.21.15.46:0/963104786 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fefb00149c0 con 0x7fefb8161b80 2024-05-09T03:20:00.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.823+0000 7fef9f7fe640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fefa40762d0 0x7fefa40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.823+0000 7fef9f7fe640 1 -- 172.21.15.46:0/963104786 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fefb0098900 con 0x7fefb8161b80 2024-05-09T03:20:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.824+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fefa40762d0 0x7fefa40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.824+0000 7fefc4392740 1 Processor -- start 2024-05-09T03:20:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.824+0000 7fefc4392740 1 -- start start 2024-05-09T03:20:00.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.824+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fefa40762d0 0x7fefa40787e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fefa8001720 tx=0x7fefa8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a015ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefc4392740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a0177be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a0178120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefc4392740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5630a00954e0 con 0x56309fd643b0 2024-05-09T03:20:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5630a0030ff0 con 0x56309fd8d270 2024-05-09T03:20:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5630a00914a0 con 0x56309fd6c590 2024-05-09T03:20:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a0177be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a0177be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44606/0 (socket says 172.21.15.46:44606) 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefb67fc640 1 -- 172.21.15.46:0/1941821814 learned_addr learned my addr 172.21.15.46:0/1941821814 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a015ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.825+0000 7fefb77fe640 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a0178120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fefb67fc640 1 -- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 msgr2=0x5630a0178120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fefb67fc640 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a0178120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fefb67fc640 1 -- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 msgr2=0x5630a015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fefb67fc640 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a015ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fefb67fc640 1 -- 172.21.15.46:0/1941821814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630a0017720 con 0x56309fd643b0 2024-05-09T03:20:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fefb67fc640 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a0177be0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fefac009960 tx=0x7fefac02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d94b5da860cb6d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fef9cff9640 1 -- 172.21.15.46:0/1941821814 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac037040 con 0x56309fd643b0 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fef9cff9640 1 -- 172.21.15.46:0/1941821814 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac032800 con 0x56309fd643b0 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.826+0000 7fef9cff9640 1 -- 172.21.15.46:0/1941821814 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac032bd0 con 0x56309fd643b0 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 -- 172.21.15.46:0/1941821814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 msgr2=0x5630a0177be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a0177be0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fefac009960 tx=0x7fefac02c000 comp rx=0 tx=0).stop 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 -- 172.21.15.46:0/1941821814 shutdown_connections 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a0178120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a0177be0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 --2- 172.21.15.46:0/1941821814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a015ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 -- 172.21.15.46:0/1941821814 >> 172.21.15.46:0/1941821814 conn(0x56309fd813c0 msgr2=0x5630a0132510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 -- 172.21.15.46:0/1941821814 shutdown_connections 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.827+0000 7fefc4392740 1 -- 172.21.15.46:0/1941821814 wait complete. 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.828+0000 7fefc4392740 1 -- start start 2024-05-09T03:20:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.828+0000 7fefc4392740 1 Processor -- start 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd6c590 0x5630a018a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a018a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5630a001c760 con 0x56309fd643b0 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5630a015f610 con 0x56309fd6c590 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5630a015f350 con 0x56309fd8d270 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.829+0000 7fefc4392740 10 client.0 _ll_register_callbacks cb 0x5630a003f4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a015ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44618/0 (socket says 172.21.15.46:44618) 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 -- 172.21.15.46:0/2412862445 learned_addr learned my addr 172.21.15.46:0/2412862445 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb67fc640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd6c590 0x5630a018a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb77fe640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a018a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 -- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 msgr2=0x5630a018a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a018a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 -- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd6c590 msgr2=0x5630a018a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd6c590 0x5630a018a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 -- 172.21.15.46:0/2412862445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630a005dbd0 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a015ee10 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fefa8014730 tx=0x7fefa8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255cb180bba0ba44 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fef95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fef95ffb640 1 -- 172.21.15.46:0/2412862445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8025020 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fef95ffb640 1 -- 172.21.15.46:0/2412862445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fefa8013770 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefac032800 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.830+0000 7fef95ffb640 1 -- 172.21.15.46:0/2412862445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=307) ==== 24+0+0 (secure 0 0 0) 0x7fefa8030050 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 10 client.10086 fetch_fsmap learned FSMap version 307 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=307,monmap=2+}) -- 0x5630a0043280 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fef95ffb640 1 -- 172.21.15.46:0/2412862445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac032800 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fef95ffb640 1 -- 172.21.15.46:0/2412862445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 307) ==== 3494+0+0 (secure 0 0 0) 0x7fefa8020040 con 0x56309fd643b0 2024-05-09T03:20:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 10 client.10086 fetch_fsmap finished waiting for FSMap version 307 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 10 client.10086 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 20 client.10086 populate_metadata read hostname 'smithi046' 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x56309fd4be40 0x5630a0149ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 4 client.10086 mds_command: new command op to 10035 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 --> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630a00a30f0 con 0x56309fd4be40 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefb67fc640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x56309fd4be40 0x5630a0149ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fef95ffb640 10 client.10086 ms_handle_connect on v2:172.21.15.46:6832/1774919282 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.831+0000 7fefb67fc640 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x56309fd4be40 0x5630a0149ab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9da6d8bdf36107b server_cookie=f0b252c153e8b977 in_seq=0 out_seq=0 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fef95ffb640 1 -- 172.21.15.46:0/2412862445 <== mds.0 v2:172.21.15.46:6832/1774919282 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5630a00a30f0 con 0x56309fd4be40 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fef95ffb640 10 client.10086 handle_command_reply: tid=0 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 client.10086 shutdown 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 msgr2=0x5630a015ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a015ee10 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fefa8014730 tx=0x7fefa8000ed0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 shutdown_connections 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x56309fd4be40 0x5630a0149ab0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a018a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd6c590 0x5630a018a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fef95ffb640 0 client.10086 ms_handle_reset on v2:172.21.15.46:6832/1774919282 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 --2- 172.21.15.46:0/2412862445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a015ee10 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 >> 172.21.15.46:0/2412862445 conn(0x56309fd813c0 msgr2=0x5630a015ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 shutdown_connections 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.832+0000 7fefc4392740 1 -- 172.21.15.46:0/2412862445 wait complete. 2024-05-09T03:20:00.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.833+0000 7fefc4392740 20 client.10086 trim_cache size 0 max 16384 2024-05-09T03:20:00.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.851+0000 7fefc4392740 1 Processor -- start 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.851+0000 7fefc4392740 1 -- start start 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd643b0 0x5630a0179570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefc4392740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 0x5630a0179ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a014bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefc4392740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefb8202a20 con 0x56309fd6c590 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fefb8201a40 con 0x56309fd643b0 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fefb8200da0 con 0x56309fd8d270 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 0x5630a0179ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 0x5630a0179ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44622/0 (socket says 172.21.15.46:44622) 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 -- 172.21.15.46:0/1835802380 learned_addr learned my addr 172.21.15.46:0/1835802380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd643b0 0x5630a0179570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb77fe640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a014bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 -- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 msgr2=0x5630a014bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a014bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 -- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd643b0 msgr2=0x5630a0179570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd643b0 0x5630a0179570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 -- 172.21.15.46:0/1835802380 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56309fdf8850 con 0x56309fd6c590 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd643b0 0x5630a0179570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb77fe640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a014bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fefb67fc640 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 0x5630a0179ab0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fefac009930 tx=0x7fefac008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2290ce959792de78 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fef97fff640 1 -- 172.21.15.46:0/1835802380 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac04a070 con 0x56309fd6c590 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fef97fff640 1 -- 172.21.15.46:0/1835802380 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac0043c0 con 0x56309fd6c590 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.853+0000 7fef97fff640 1 -- 172.21.15.46:0/1835802380 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac037040 con 0x56309fd6c590 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 -- 172.21.15.46:0/1835802380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 msgr2=0x5630a0179ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 0x5630a0179ab0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fefac009930 tx=0x7fefac008030 comp rx=0 tx=0).stop 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 -- 172.21.15.46:0/1835802380 shutdown_connections 2024-05-09T03:20:00.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd8d270 0x5630a014bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd6c590 0x5630a0179ab0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 --2- 172.21.15.46:0/1835802380 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd643b0 0x5630a0179570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 -- 172.21.15.46:0/1835802380 >> 172.21.15.46:0/1835802380 conn(0x56309fd813c0 msgr2=0x5630a0173300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 -- 172.21.15.46:0/1835802380 shutdown_connections 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.854+0000 7fefc4392740 1 -- 172.21.15.46:0/1835802380 wait complete. 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.855+0000 7fefc4392740 1 -- start start 2024-05-09T03:20:00.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.855+0000 7fefc4392740 1 Processor -- start 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a01eed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a014c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a01fb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5630a0147ff0 con 0x56309fd643b0 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fefb82013f0 con 0x56309fd8d270 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5630a0147850 con 0x56309fd6c590 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.856+0000 7fefc4392740 10 client.0 _ll_register_callbacks cb 0x7fefa8001400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a01eed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a01eed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44634/0 (socket says 172.21.15.46:44634) 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 -- 172.21.15.46:0/580369490 learned_addr learned my addr 172.21.15.46:0/580369490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb77fe640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a01fb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb67fc640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a014c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 -- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 msgr2=0x5630a014c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a014c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 -- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 msgr2=0x5630a01fb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a01fb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 -- 172.21.15.46:0/580369490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56309fe6f6b0 con 0x56309fd643b0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefb6ffd640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a01eed60 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fefa800fc20 tx=0x7fefa8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faca3b2d7ac13709 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fef95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fef95ffb640 1 -- 172.21.15.46:0/580369490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa8025040 con 0x56309fd643b0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fef95ffb640 1 -- 172.21.15.46:0/580369490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fefa8013a40 con 0x56309fd643b0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefac0043c0 con 0x56309fd643b0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fef95ffb640 1 -- 172.21.15.46:0/580369490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=307) ==== 24+0+0 (secure 0 0 0) 0x7fefa8041050 con 0x56309fd643b0 2024-05-09T03:20:00.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefc4392740 10 client.10092 fetch_fsmap learned FSMap version 307 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.857+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=307,monmap=2+}) -- 0x5630a0051630 con 0x56309fd643b0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fef95ffb640 1 -- 172.21.15.46:0/580369490 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefac0043c0 con 0x56309fd643b0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fef95ffb640 1 -- 172.21.15.46:0/580369490 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 307) ==== 3494+0+0 (secure 0 0 0) 0x7fefa8020040 con 0x56309fd643b0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 10 client.10092 fetch_fsmap finished waiting for FSMap version 307 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 10 client.10092 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 20 client.10092 populate_metadata read hostname 'smithi046' 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x5630a018c8b0 0x5630a015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 4 client.10092 mds_command: new command op to 10035 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 --> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] -- command(tid 0: {"prefix": "status"}) -- 0x5630a009cf60 con 0x5630a018c8b0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefb67fc640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x5630a018c8b0 0x5630a015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fef95ffb640 10 client.10092 ms_handle_connect on v2:172.21.15.46:6832/1774919282 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefb67fc640 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x5630a018c8b0 0x5630a015dc00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e86f58a32a8ec8ac server_cookie=18e600bee4de4f67 in_seq=0 out_seq=0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fef95ffb640 1 -- 172.21.15.46:0/580369490 <== mds.0 v2:172.21.15.46:6832/1774919282 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x5630a009cf60 con 0x5630a018c8b0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fef95ffb640 10 client.10092 handle_command_reply: tid=0 2024-05-09T03:20:00.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.858+0000 7fefc4392740 1 client.10092 shutdown 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 msgr2=0x5630a01eed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a01eed60 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fefa800fc20 tx=0x7fefa8014000 comp rx=0 tx=0).stop 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 shutdown_connections 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:6832/1774919282,v1:172.21.15.46:6834/1774919282] conn(0x5630a018c8b0 0x5630a015dc00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56309fd8d270 0x5630a01fb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x56309fd6c590 0x5630a014c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fef95ffb640 0 client.10092 ms_handle_reset on v2:172.21.15.46:6832/1774919282 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 --2- 172.21.15.46:0/580369490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56309fd643b0 0x5630a01eed60 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 >> 172.21.15.46:0/580369490 conn(0x56309fd813c0 msgr2=0x5630a0173300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 shutdown_connections 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 1 -- 172.21.15.46:0/580369490 wait complete. 2024-05-09T03:20:00.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefc4392740 20 client.10092 trim_cache size 0 max 16384 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10035, 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7465662200000001, 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 307, 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 118, 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 118, 2024-05-09T03:20:00.867 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 8.6944671499999995 2024-05-09T03:20:00.868 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:20:00.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fefa40762d0 msgr2=0x7fefa40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fefa40762d0 0x7fefa40787e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fefa8001720 tx=0x7fefa8009040 comp rx=0 tx=0).stop 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 msgr2=0x7fefb81e9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 0x7fefb81e9b70 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7fefb0007d70 tx=0x7fefb000e4b0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 shutdown_connections 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fefa40762d0 0x7fefa40787e0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefb8161b80 0x7fefb81e9b70 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fefb80d2f00 0x7fefb8146df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 --2- 172.21.15.46:0/963104786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fefb8071600 0x7fefb8143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.859+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 >> 172.21.15.46:0/963104786 conn(0x7fefb80d3a80 msgr2=0x7fefb80d1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.860+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 shutdown_connections 2024-05-09T03:20:00.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.860+0000 7fefbd4d9640 1 -- 172.21.15.46:0/963104786 wait complete. 2024-05-09T03:20:00.872 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:20:00.872 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap header on 10000000000.00000000', proceeding to ls 2024-05-09T03:20:00.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:20:00.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.985+0000 7f72a8372640 1 Processor -- start 2024-05-09T03:20:00.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.985+0000 7f72a8372640 1 -- start start 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a8372640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 0x7f72a00cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a8372640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72a00d2090 0x7f72a00cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a8372640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72a00d4900 con 0x7f72a008cd60 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a8372640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f72a00b5340 con 0x7f72a00d2090 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a8372640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f72a00b4fd0 con 0x7f72a00d1c80 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a610e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f72a008cd60 0x7f72a00d26c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49428/0 (socket says 172.21.15.46:49428) 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a610e640 1 -- 172.21.15.46:0/165655376 learned_addr learned my addr 172.21.15.46:0/165655376 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a690f640 1 --2- 172.21.15.46:0/165655376 >> v2:172.21.15.88:3300/0 conn(0x7f72a00d2090 0x7f72a00cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a590d640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 0x7f72a00cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932283305 0 0) 0x7f72a00d4900 con 0x7f72a008cd60 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.987+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7288003520 con 0x7f72a008cd60 2024-05-09T03:20:00.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2448900045 0 0) 0x7f7288003520 con 0x7f72a008cd60 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 msgr2=0x7f72a00cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 0x7f72a00cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 >> v2:172.21.15.88:3300/0 conn(0x7f72a00d2090 msgr2=0x7f72a00cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 --2- 172.21.15.46:0/165655376 >> v2:172.21.15.88:3300/0 conn(0x7f72a00d2090 0x7f72a00cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a00b6a50 con 0x7f72a008cd60 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f72980016f0 con 0x7f72a008cd60 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 >> v1:172.21.15.46:6789/0 conn(0x7f72a008cd60 legacy=0x7f72a00d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7288003a00 0x7f7288003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 0x7f7288004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7288004d40 0x7f7288005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72a00b5340 con 0x7f7288004d40 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72a00b4fd0 con 0x7f7288003a00 2024-05-09T03:20:00.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f728800baf0 con 0x7f72a00d1c80 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a690f640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7288004d40 0x7f7288005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a610e640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7288003a00 0x7f7288003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a590d640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 0x7f7288004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a690f640 1 -- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 msgr2=0x7f7288004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a690f640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72a00d1c80 0x7f7288004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a690f640 1 -- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7288003a00 msgr2=0x7f7288003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a690f640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7288003a00 0x7f7288003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.988+0000 7f72a690f640 1 -- 172.21.15.46:0/165655376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7288003520 con 0x7f7288004d40 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.989+0000 7f72a690f640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7288004d40 0x7f7288005150 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f7290009fc0 tx=0x7f7288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edccd6988f1105d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.989+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290009c10 con 0x7f7288004d40 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.989+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7290036070 con 0x7f7288004d40 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.989+0000 7f72a510c640 1 -- 172.21.15.46:0/165655376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729002d070 con 0x7f7288004d40 2024-05-09T03:20:00.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a8372640 1 -- 172.21.15.46:0/165655376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7288004d40 msgr2=0x7f7288005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a8372640 1 --2- 172.21.15.46:0/165655376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7288004d40 0x7f7288005150 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f7290009fc0 tx=0x7f7288005830 comp rx=0 tx=0).stop 2024-05-09T03:20:00.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a690f640 1 -- 172.21.15.46:0/165655376 reap_dead start 2024-05-09T03:20:00.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a8372640 1 -- 172.21.15.46:0/165655376 shutdown_connections 2024-05-09T03:20:00.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a8372640 1 -- 172.21.15.46:0/165655376 >> 172.21.15.46:0/165655376 conn(0x7f72a00d3c00 msgr2=0x7f72a00d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:00.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a8372640 1 -- 172.21.15.46:0/165655376 shutdown_connections 2024-05-09T03:20:00.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.990+0000 7f72a8372640 1 -- 172.21.15.46:0/165655376 wait complete. 2024-05-09T03:20:00.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.992+0000 7f72a8372640 1 Processor -- start 2024-05-09T03:20:00.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.992+0000 7f72a8372640 1 -- start start 2024-05-09T03:20:00.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a8372640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7288004d40 0x7f72a00cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a8372640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72a00cacb0 0x7f72a0152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a8372640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 0x7f72a00b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a8372640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72a00cf430 con 0x7f72a006f1e0 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a8372640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72a00cf2b0 con 0x7f72a00cacb0 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a8372640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f72a00cf5b0 con 0x7f7288004d40 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a690f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 0x7f72a00b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a690f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 0x7f72a00b1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44662/0 (socket says 172.21.15.46:44662) 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a690f640 1 -- 172.21.15.46:0/953828130 learned_addr learned my addr 172.21.15.46:0/953828130 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a610e640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7288004d40 0x7f72a00cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.993+0000 7f72a590d640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72a00cacb0 0x7f72a0152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a690f640 1 -- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7288004d40 msgr2=0x7f72a00cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a690f640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7288004d40 0x7f72a00cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a690f640 1 -- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72a00cacb0 msgr2=0x7f72a0152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a690f640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72a00cacb0 0x7f72a0152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a690f640 1 -- 172.21.15.46:0/953828130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a00cca00 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a690f640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 0x7f72a00b1950 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f7290033040 tx=0x7f7290009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290009c10 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7290003990 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a00cd7a0 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290003d80 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.994+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a00cb700 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.995+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a00cb190 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.996+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7290005d10 con 0x7f72a006f1e0 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.996+0000 7f729effd640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f728c0762d0 0x7f728c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:01.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.996+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7f72900b9720 con 0x7f72a006f1e0 2024-05-09T03:20:01.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.997+0000 7f72a610e640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f728c0762d0 0x7f728c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:01.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:00.997+0000 7f72a610e640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f728c0762d0 0x7f728c0787e0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7298001d00 tx=0x7f7298000fb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.001+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f729004a060 con 0x7f72a006f1e0 2024-05-09T03:20:01.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.234+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f72a00caa90 con 0x7f72a006f1e0 2024-05-09T03:20:01.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.234+0000 7f729effd640 1 -- 172.21.15.46:0/953828130 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v307) ==== 53+0+73 (secure 0 0 0) 0x7f7290082510 con 0x7f72a006f1e0 2024-05-09T03:20:01.236 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:20:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f728c0762d0 msgr2=0x7f728c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f728c0762d0 0x7f728c0787e0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7298001d00 tx=0x7f7298000fb0 comp rx=0 tx=0).stop 2024-05-09T03:20:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 msgr2=0x7f72a00b1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:01.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 0x7f72a00b1950 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f7290033040 tx=0x7f7290009750 comp rx=0 tx=0).stop 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 shutdown_connections 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f728c0762d0 0x7f728c0787e0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72a006f1e0 0x7f72a00b1950 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72a00cacb0 0x7f72a0152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 --2- 172.21.15.46:0/953828130 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7288004d40 0x7f72a00cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 >> 172.21.15.46:0/953828130 conn(0x7f72a00d3c00 msgr2=0x7f72a015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 shutdown_connections 2024-05-09T03:20:01.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01.236+0000 7f72a8372640 1 -- 172.21.15.46:0/953828130 wait complete. 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:20:01.249 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:20:01.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:01.250 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:20:01.266 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:20:01.266 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:01.267 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41916sec preferred_lft 41916sec 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:20:01.268 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:01.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:01.269 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:20:01.331 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:20:01.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:01.333 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:20:01.389 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:20:01.389 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:20:01.389 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:20:01.389 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:20:01.389 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:20:01.389 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:20:01.389 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:20:01.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:20:01.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:20:01.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:20:01.390 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:20:01.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:20:01.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:20:01.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:01.519 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:20:01.535 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:20:01.535 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:20:01.535 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:20:01.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:01.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:20:01.601 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:20:01.601 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:20:01.602 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:01.658 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:20:01.658 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:20:01.658 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:01.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:20:01.781 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:20:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:20:01.838 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:20:01.839 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:20:01.840 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:01.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:01.841 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:20:01.894 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:20:01.894 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:20:01.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:20:01.963 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:20:02.022 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:20:02.023 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:02.024 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:02.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:02.025 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:20:02.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:20:02.031+0000 7f5822725480 -1 init, newargv = 0x55abbdaf6780 newargc=15 2024-05-09T03:20:02.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[83199]: starting ceph client 2024-05-09T03:20:02.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[83199]: starting fuse 2024-05-09T03:20:02.049 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:20:02.050 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:02.050 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:20:02.051 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:20:02.053 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:20:02.171 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.83199.asok 2024-05-09T03:20:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:20:02.181 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83199.asok 2024-05-09T03:20:02.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:02.182 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83199.asok status 2024-05-09T03:20:02.329 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:20:02.329 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:20:02.329 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:20:02.329 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:20:02.329 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:20:02.329 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "83199", 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10104, 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10104 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:02.330 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3548390559 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3548390559 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10104 192.168.0.1:0/3548390559", 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3548390559", 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 307, 2024-05-09T03:20:02.331 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 118, 2024-05-09T03:20:02.332 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:20:02.332 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:20:02.332 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:20:02.332 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:20:02.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:02.338 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:20:02.338 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:02.354 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:20:02.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:02.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:02.355 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:02.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:02.421 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:20:02.463 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:20:02.480 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:02.479+0000 7fc3bc758640 -1 mds.0.cache.dir(0x10000000000) Corrupt fnode in dirfrag 0x10000000000: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:02.481 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:20:02.481 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:20:02.481 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:02.482 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:20:02.482 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 10000000000.00000000' did not caused DAMAGED state 2024-05-09T03:20:02.482 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 600.00000000' 2024-05-09T03:20:02.482 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:02.482 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:02.538 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:20:02.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.692+0000 7fd52a4b8640 1 Processor -- start 2024-05-09T03:20:02.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.692+0000 7fd52a4b8640 1 -- start start 2024-05-09T03:20:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd52a4b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd5241437d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd52a4b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd524140d00 0x7fd52415eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd52a4b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd524143d10 con 0x7fd52415f420 2024-05-09T03:20:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd52a4b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd524143fe0 con 0x7fd524140d00 2024-05-09T03:20:02.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd52a4b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd5241442c0 con 0x7fd5240d1000 2024-05-09T03:20:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd523fff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd5241437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd5237fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd524140d00 0x7fd52415eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd528a55640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd52415f420 0x7fd5240d2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:53370/0 (socket says 172.21.15.46:53370) 2024-05-09T03:20:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.694+0000 7fd528a55640 1 -- 172.21.15.46:0/466490146 learned_addr learned my addr 172.21.15.46:0/466490146 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.695+0000 7fd523fff640 1 -- 172.21.15.46:0/466490146 >> v2:172.21.15.88:3300/0 conn(0x7fd524140d00 msgr2=0x7fd52415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:02.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.695+0000 7fd523fff640 1 --2- 172.21.15.46:0/466490146 >> v2:172.21.15.88:3300/0 conn(0x7fd524140d00 0x7fd52415eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.695+0000 7fd523fff640 1 -- 172.21.15.46:0/466490146 >> v1:172.21.15.88:6789/0 conn(0x7fd52415f420 legacy=0x7fd5240d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.695+0000 7fd523fff640 1 -- 172.21.15.46:0/466490146 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5241445a0 con 0x7fd5240d1000 2024-05-09T03:20:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.695+0000 7fd5237fe640 1 --2- 172.21.15.46:0/466490146 >> v2:172.21.15.88:3300/0 conn(0x7fd524140d00 0x7fd52415eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.695+0000 7fd523fff640 1 --2- 172.21.15.46:0/466490146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd5241437d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd5140097a0 tx=0x7fd51402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c0668b0c188114 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:02.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd522ffd640 1 -- 172.21.15.46:0/466490146 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514037070 con 0x7fd5240d1000 2024-05-09T03:20:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd522ffd640 1 -- 172.21.15.46:0/466490146 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd514032440 con 0x7fd5240d1000 2024-05-09T03:20:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd522ffd640 1 -- 172.21.15.46:0/466490146 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514032810 con 0x7fd5240d1000 2024-05-09T03:20:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 -- 172.21.15.46:0/466490146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 msgr2=0x7fd5241437d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/466490146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd5241437d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd5140097a0 tx=0x7fd51402a180 comp rx=0 tx=0).stop 2024-05-09T03:20:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 -- 172.21.15.46:0/466490146 shutdown_connections 2024-05-09T03:20:02.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/466490146 >> v2:172.21.15.88:3300/0 conn(0x7fd524140d00 0x7fd52415eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/466490146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd5241437d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 -- 172.21.15.46:0/466490146 >> 172.21.15.46:0/466490146 conn(0x7fd5240705c0 msgr2=0x7fd5240709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 -- 172.21.15.46:0/466490146 shutdown_connections 2024-05-09T03:20:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.696+0000 7fd52a4b8640 1 -- 172.21.15.46:0/466490146 wait complete. 2024-05-09T03:20:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.698+0000 7fd52a4b8640 1 Processor -- start 2024-05-09T03:20:02.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.698+0000 7fd52a4b8640 1 -- start start 2024-05-09T03:20:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd52a4b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd524157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd52a4b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 0x7fd524134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd52a4b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd52415f420 0x7fd5241f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd52a4b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd52415e660 con 0x7fd524140d00 2024-05-09T03:20:02.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd52a4b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd5240d1410 con 0x7fd52415f420 2024-05-09T03:20:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd52a4b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd5240d3350 con 0x7fd5240d1000 2024-05-09T03:20:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 0x7fd524134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 0x7fd524134d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44676/0 (socket says 172.21.15.46:44676) 2024-05-09T03:20:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 -- 172.21.15.46:0/1198140080 learned_addr learned my addr 172.21.15.46:0/1198140080 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd523fff640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd524157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:02.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd528a55640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd52415f420 0x7fd5241f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 -- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 msgr2=0x7fd524157f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd524157f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 -- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd52415f420 msgr2=0x7fd5241f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd52415f420 0x7fd5241f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 -- 172.21.15.46:0/1198140080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5240cb5b0 con 0x7fd524140d00 2024-05-09T03:20:02.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd523fff640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd524157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.700+0000 7fd5237fe640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 0x7fd524134d20 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fd51800e900 tx=0x7fd51800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.701+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518017070 con 0x7fd524140d00 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.701+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd51800ca00 con 0x7fd524140d00 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.701+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52415acc0 con 0x7fd524140d00 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.701+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51800cdf0 con 0x7fd524140d00 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.701+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52415b280 con 0x7fd524140d00 2024-05-09T03:20:02.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.701+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5240cb9b0 con 0x7fd524140d00 2024-05-09T03:20:02.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.702+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd51801a070 con 0x7fd524140d00 2024-05-09T03:20:02.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.703+0000 7fd520ff9640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd5100762d0 0x7fd5100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:02.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.703+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fd518098180 con 0x7fd524140d00 2024-05-09T03:20:02.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.703+0000 7fd523fff640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd5100762d0 0x7fd5100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:02.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.703+0000 7fd523fff640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd5100762d0 0x7fd5100787e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd514034000 tx=0x7fd51402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:02.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.707+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd518060780 con 0x7fd524140d00 2024-05-09T03:20:02.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.911+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd52406a270 con 0x7fd524140d00 2024-05-09T03:20:02.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.912+0000 7fd520ff9640 1 -- 172.21.15.46:0/1198140080 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 68 entries v118) ==== 81+0+4132 (secure 0 0 0) 0x7fd518064a80 con 0x7fd524140d00 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:02.915 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:02.916 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:02.917 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:02.918 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:02.919 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:02.919 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:02.919 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:02.919 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:listed 68 entries 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd5100762d0 msgr2=0x7fd5100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd5100762d0 0x7fd5100787e0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fd514034000 tx=0x7fd51402e000 comp rx=0 tx=0).stop 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 msgr2=0x7fd524134d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 0x7fd524134d20 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fd51800e900 tx=0x7fd51800edc0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 shutdown_connections 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd52415f420 0x7fd5241f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd5100762d0 0x7fd5100787e0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd524140d00 0x7fd524134d20 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 --2- 172.21.15.46:0/1198140080 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd5240d1000 0x7fd524157f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:02.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 >> 172.21.15.46:0/1198140080 conn(0x7fd5240705c0 msgr2=0x7fd524161060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:02.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 shutdown_connections 2024-05-09T03:20:02.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:02.914+0000 7fd52a4b8640 1 -- 172.21.15.46:0/1198140080 wait complete. 2024-05-09T03:20:02.927 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:02.927 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:02.943 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:20:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.097+0000 7f0e79172640 1 Processor -- start 2024-05-09T03:20:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.097+0000 7f0e79172640 1 -- start start 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.098+0000 7f0e79172640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e740b5140 0x7f0e740b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.098+0000 7f0e79172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b7ca0 0x7f0e740d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.098+0000 7f0e79172640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e740b80b0 con 0x7f0e740d3df0 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.098+0000 7f0e79172640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0e740d3480 con 0x7f0e740b5140 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.098+0000 7f0e79172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0e740b5550 con 0x7f0e740b7ca0 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.098+0000 7f0e7359e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0e740d3df0 0x7f0e74163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49430/0 (socket says 172.21.15.46:49430) 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e7359e640 1 -- 172.21.15.46:0/4000066435 learned_addr learned my addr 172.21.15.46:0/4000066435 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e72d9d640 1 --2- 172.21.15.46:0/4000066435 >> v2:172.21.15.88:3300/0 conn(0x7f0e740b5140 0x7f0e740b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e7259c640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b7ca0 0x7f0e740d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e71d9b640 1 -- 172.21.15.46:0/4000066435 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802787199 0 0) 0x7f0e6c001490 con 0x7f0e740d3df0 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e71d9b640 1 -- 172.21.15.46:0/4000066435 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e5c003520 con 0x7f0e740d3df0 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e72d9d640 1 -- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b7ca0 msgr2=0x7f0e740d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e72d9d640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b7ca0 0x7f0e740d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e72d9d640 1 -- 172.21.15.46:0/4000066435 >> v1:172.21.15.46:6789/0 conn(0x7f0e740d3df0 legacy=0x7f0e74163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e72d9d640 1 -- 172.21.15.46:0/4000066435 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e740d44e0 con 0x7f0e740b5140 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.099+0000 7f0e7259c640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b7ca0 0x7f0e740d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.100+0000 7f0e72d9d640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b5140 0x7f0e740b7760 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0e640097a0 tx=0x7f0e6402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c5a75eb1c7e07b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.100+0000 7f0e71d9b640 1 -- 172.21.15.46:0/4000066435 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64037070 con 0x7f0e740b5140 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.100+0000 7f0e71d9b640 1 -- 172.21.15.46:0/4000066435 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e64032440 con 0x7f0e740b5140 2024-05-09T03:20:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.100+0000 7f0e71d9b640 1 -- 172.21.15.46:0/4000066435 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e640327f0 con 0x7f0e740b5140 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 -- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b5140 msgr2=0x7f0e740b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b5140 0x7f0e740b7760 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0e640097a0 tx=0x7f0e6402a210 comp rx=0 tx=0).stop 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 -- 172.21.15.46:0/4000066435 shutdown_connections 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b7ca0 0x7f0e740d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 --2- 172.21.15.46:0/4000066435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b5140 0x7f0e740b7760 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 -- 172.21.15.46:0/4000066435 >> 172.21.15.46:0/4000066435 conn(0x7f0e7408d3f0 msgr2=0x7f0e7408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 -- 172.21.15.46:0/4000066435 shutdown_connections 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.101+0000 7f0e79172640 1 -- 172.21.15.46:0/4000066435 wait complete. 2024-05-09T03:20:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.103+0000 7f0e79172640 1 Processor -- start 2024-05-09T03:20:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.103+0000 7f0e79172640 1 -- start start 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e79172640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b5140 0x7f0e74161cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e79172640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b7ca0 0x7f0e7414abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e79172640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 0x7f0e74162a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e79172640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0e740d4d80 con 0x7f0e740d3df0 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e79172640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0e740d4aa0 con 0x7f0e740b7ca0 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e79172640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0e740d4200 con 0x7f0e740b5140 2024-05-09T03:20:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e7359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 0x7f0e74162a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e7359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 0x7f0e74162a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44682/0 (socket says 172.21.15.46:44682) 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e7359e640 1 -- 172.21.15.46:0/328343752 learned_addr learned my addr 172.21.15.46:0/328343752 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e7259c640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b7ca0 0x7f0e7414abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.104+0000 7f0e72d9d640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b5140 0x7f0e74161cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e7359e640 1 -- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b5140 msgr2=0x7f0e74161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e7359e640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b5140 0x7f0e74161cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e7359e640 1 -- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b7ca0 msgr2=0x7f0e7414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e7359e640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b7ca0 0x7f0e7414abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e7359e640 1 -- 172.21.15.46:0/328343752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7414b6a0 con 0x7f0e740d3df0 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e7359e640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 0x7f0e74162a00 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00cad0 tx=0x7f0e6c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c00ac30 con 0x7f0e740d3df0 2024-05-09T03:20:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e6c00add0 con 0x7f0e740d3df0 2024-05-09T03:20:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e741621f0 con 0x7f0e740d3df0 2024-05-09T03:20:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c012620 con 0x7f0e740d3df0 2024-05-09T03:20:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.105+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e74160c10 con 0x7f0e740d3df0 2024-05-09T03:20:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.106+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e7406a270 con 0x7f0e740d3df0 2024-05-09T03:20:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.107+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0e6c0127c0 con 0x7f0e740d3df0 2024-05-09T03:20:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.108+0000 7f0e5b7fe640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e600762d0 0x7f0e600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.108+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7f0e6c098750 con 0x7f0e740d3df0 2024-05-09T03:20:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.108+0000 7f0e72d9d640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e600762d0 0x7f0e600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.108+0000 7f0e72d9d640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e600762d0 0x7f0e600787e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0e64034000 tx=0x7f0e6402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.112+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0e7414b6a0 con 0x7f0e740d3df0 2024-05-09T03:20:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.318+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0e741606a0 con 0x7f0e740d3df0 2024-05-09T03:20:03.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.319+0000 7f0e5b7fe640 1 -- 172.21.15.46:0/328343752 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 68 entries v118) ==== 81+0+4132 (secure 0 0 0) 0x7f0e740b80b0 con 0x7f0e740d3df0 2024-05-09T03:20:03.321 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:03.321 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:03.321 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:03.322 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:03.323 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:03.324 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:03.325 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:03.326 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:03.327 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:03.328 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:03.329 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:03.330 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:03.330 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:03.330 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:03.330 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:03.330 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:03.332 INFO:teuthology.orchestra.run.smithi046.stderr:listed 68 entries 2024-05-09T03:20:03.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e600762d0 msgr2=0x7f0e600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e600762d0 0x7f0e600787e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0e64034000 tx=0x7f0e6402e000 comp rx=0 tx=0).stop 2024-05-09T03:20:03.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 msgr2=0x7f0e74162a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 0x7f0e74162a00 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c00cad0 tx=0x7f0e6c00cf90 comp rx=0 tx=0).stop 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 shutdown_connections 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0e600762d0 0x7f0e600787e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e740d3df0 0x7f0e74162a00 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e740b7ca0 0x7f0e7414abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 --2- 172.21.15.46:0/328343752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0e740b5140 0x7f0e74161cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 >> 172.21.15.46:0/328343752 conn(0x7f0e7408d3f0 msgr2=0x7f0e740bb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:03.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 shutdown_connections 2024-05-09T03:20:03.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.321+0000 7f0e79172640 1 -- 172.21.15.46:0/328343752 wait complete. 2024-05-09T03:20:03.335 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:20:03.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:03.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:03.362 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:20:03.362 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:20:03.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:03.362 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:20:03.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:20:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.572+0000 7fb714900640 1 Processor -- start 2024-05-09T03:20:03.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.572+0000 7fb714900640 1 -- start start 2024-05-09T03:20:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb714900640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb7100d2f60 0x7fb7100cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb714900640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb71009d470 con 0x7fb7100d3730 2024-05-09T03:20:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb714900640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7100d11b0 con 0x7fb7100ceba0 2024-05-09T03:20:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb714900640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb71009cfe0 con 0x7fb7100d2f60 2024-05-09T03:20:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb70e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7100d3730 0x7fb71006e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49432/0 (socket says 172.21.15.46:49432) 2024-05-09T03:20:03.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb70e59c640 1 -- 172.21.15.46:0/3504556741 learned_addr learned my addr 172.21.15.46:0/3504556741 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb70ed9d640 1 --2- 172.21.15.46:0/3504556741 >> v2:172.21.15.88:3300/0 conn(0x7fb7100d2f60 0x7fb7100cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744672126 0 0) 0x7fb708001110 con 0x7fb7100d3730 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.574+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6fc003520 con 0x7fb7100d3730 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2558978699 0 0) 0x7fb7100d11b0 con 0x7fb7100ceba0 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb708001110 con 0x7fb7100ceba0 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3401142261 0 0) 0x7fb6fc003520 con 0x7fb7100d3730 2024-05-09T03:20:03.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 >> v2:172.21.15.88:3300/0 conn(0x7fb7100d2f60 msgr2=0x7fb7100cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 --2- 172.21.15.46:0/3504556741 >> v2:172.21.15.88:3300/0 conn(0x7fb7100d2f60 0x7fb7100cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 >> v1:172.21.15.88:6789/0 conn(0x7fb7100ceba0 legacy=0x7fb7100cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71006e9b0 con 0x7fb7100d3730 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fb708001840 con 0x7fb7100d3730 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 >> v1:172.21.15.46:6789/0 conn(0x7fb7100d3730 legacy=0x7fb71006e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb6fc0040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6fc004800 0x7fb6fc004c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6fc005380 0x7fb6fc00ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc00c2f0 con 0x7fb6fc003c80 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc00c4a0 con 0x7fb6fc004800 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb6fc00c6a0 con 0x7fb6fc005380 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70e59c640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb6fc0040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70dd9b640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6fc004800 0x7fb6fc004c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70ed9d640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6fc005380 0x7fb6fc00ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70e59c640 1 -- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6fc005380 msgr2=0x7fb6fc00ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70e59c640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6fc005380 0x7fb6fc00ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70e59c640 1 -- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6fc004800 msgr2=0x7fb6fc004c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70e59c640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6fc004800 0x7fb6fc004c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70e59c640 1 -- 172.21.15.46:0/3504556741 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc003520 con 0x7fb6fc003c80 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.575+0000 7fb70ed9d640 1 -- 172.21.15.46:0/3504556741 reap_dead start 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.576+0000 7fb70e59c640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb6fc0040d0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c320 tx=0x7fb6fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124aa3197b098649 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.576+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708007d90 con 0x7fb6fc003c80 2024-05-09T03:20:03.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.576+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7080058d0 con 0x7fb6fc003c80 2024-05-09T03:20:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.576+0000 7fb70d59a640 1 -- 172.21.15.46:0/3504556741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708005d20 con 0x7fb6fc003c80 2024-05-09T03:20:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 -- 172.21.15.46:0/3504556741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 msgr2=0x7fb6fc0040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb6fc0040d0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c320 tx=0x7fb6fc005830 comp rx=0 tx=0).stop 2024-05-09T03:20:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 -- 172.21.15.46:0/3504556741 shutdown_connections 2024-05-09T03:20:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 --2- 172.21.15.46:0/3504556741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb6fc0040d0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 -- 172.21.15.46:0/3504556741 >> 172.21.15.46:0/3504556741 conn(0x7fb7100d05f0 msgr2=0x7fb7100d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 -- 172.21.15.46:0/3504556741 shutdown_connections 2024-05-09T03:20:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.577+0000 7fb714900640 1 -- 172.21.15.46:0/3504556741 wait complete. 2024-05-09T03:20:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.579+0000 7fb714900640 1 Processor -- start 2024-05-09T03:20:03.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.579+0000 7fb714900640 1 -- start start 2024-05-09T03:20:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb714900640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb71015e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb714900640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7100cbd00 0x7fb710162cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb714900640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb710163210 0x7fb7100c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb714900640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb7100d3370 con 0x7fb6fc003c80 2024-05-09T03:20:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb714900640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb7100b94a0 con 0x7fb7100cbd00 2024-05-09T03:20:03.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb714900640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb7100b9160 con 0x7fb710163210 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb70e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb71015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb70e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb71015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44692/0 (socket says 172.21.15.46:44692) 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.580+0000 7fb70e59c640 1 -- 172.21.15.46:0/2935216618 learned_addr learned my addr 172.21.15.46:0/2935216618 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70dd9b640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7100cbd00 0x7fb710162cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70ed9d640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb710163210 0x7fb7100c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70e59c640 1 -- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb710163210 msgr2=0x7fb7100c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70e59c640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb710163210 0x7fb7100c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70e59c640 1 -- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7100cbd00 msgr2=0x7fb710162cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70e59c640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7100cbd00 0x7fb710162cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70e59c640 1 -- 172.21.15.46:0/2935216618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb710163620 con 0x7fb6fc003c80 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70dd9b640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7100cbd00 0x7fb710162cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70ed9d640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb710163210 0x7fb7100c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb70e59c640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb71015e240 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c440 tx=0x7fb708001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708038040 con 0x7fb6fc003c80 2024-05-09T03:20:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7080047e0 con 0x7fb6fc003c80 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7100c8d50 con 0x7fb6fc003c80 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708004990 con 0x7fb6fc003c80 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.581+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7101f9ae0 con 0x7fb6fc003c80 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.582+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7100b65b0 con 0x7fb6fc003c80 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.583+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb708043020 con 0x7fb6fc003c80 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.583+0000 7fb706ffd640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6f00762d0 0x7fb6f00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.583+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8691+0+0 (secure 0 0 0) 0x7fb7080b7b10 con 0x7fb6fc003c80 2024-05-09T03:20:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.583+0000 7fb70dd9b640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6f00762d0 0x7fb6f00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.584+0000 7fb70dd9b640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6f00762d0 0x7fb6f00787e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb7100b4fe0 tx=0x7fb6f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.588+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb7100b65b0 con 0x7fb6fc003c80 2024-05-09T03:20:03.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.817+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb71006a270 con 0x7fb6fc003c80 2024-05-09T03:20:03.819 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:03.818+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:20:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.820+0000 7fb706ffd640 1 -- 172.21.15.46:0/2935216618 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v308) ==== 161+0+0 (secure 0 0 0) 0x7fb710163620 con 0x7fb6fc003c80 2024-05-09T03:20:03.821 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6f00762d0 msgr2=0x7fb6f00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6f00762d0 0x7fb6f00787e0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb7100b4fe0 tx=0x7fb6f8009040 comp rx=0 tx=0).stop 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 msgr2=0x7fb71015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb71015e240 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fb70800c440 tx=0x7fb708001a20 comp rx=0 tx=0).stop 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 shutdown_connections 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb710163210 0x7fb7100c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7100cbd00 0x7fb710162cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6f00762d0 0x7fb6f00787e0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 --2- 172.21.15.46:0/2935216618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc003c80 0x7fb71015e240 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 >> 172.21.15.46:0/2935216618 conn(0x7fb7100d05f0 msgr2=0x7fb7100b6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 shutdown_connections 2024-05-09T03:20:03.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.823+0000 7fb714900640 1 -- 172.21.15.46:0/2935216618 wait complete. 2024-05-09T03:20:03.826 INFO:tasks.ceph.mds.c.smithi046.stderr: -704> 2024-05-09T03:19:58.111+0000 7fc3c0760640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:20:03.826 INFO:tasks.ceph.mds.c.smithi046.stderr: -111> 2024-05-09T03:20:02.479+0000 7fc3bc758640 -1 mds.0.cache.dir(0x10000000000) Corrupt fnode in dirfrag 0x10000000000: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:03.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:20:03.883 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:03.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.953+0000 7fb1d8c22640 1 Processor -- start 2024-05-09T03:20:03.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.953+0000 7fb1d8c22640 1 -- start start 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1d8c22640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1d40ceba0 0x7fb1d40cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1d8c22640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d409d470 con 0x7fb1d40d3730 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1d8c22640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d40d11b0 con 0x7fb1d40cf4f0 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1d8c22640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb1d409cfe0 con 0x7fb1d40ceba0 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1d259c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb1d40d3730 0x7fb1d406e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49444/0 (socket says 172.21.15.46:49444) 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1d259c640 1 -- 172.21.15.46:0/1345566132 learned_addr learned my addr 172.21.15.46:0/1345566132 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:03.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.954+0000 7fb1cbfff640 1 --2- 172.21.15.46:0/1345566132 >> v2:172.21.15.46:3300/0 conn(0x7fb1d40ceba0 0x7fb1d40cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.955+0000 7fb1cbfff640 1 -- 172.21.15.46:0/1345566132 >> v1:172.21.15.88:6789/0 conn(0x7fb1d40cf4f0 legacy=0x7fb1d40d1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.955+0000 7fb1cbfff640 1 -- 172.21.15.46:0/1345566132 >> v1:172.21.15.46:6789/0 conn(0x7fb1d40d3730 legacy=0x7fb1d406e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.955+0000 7fb1cbfff640 1 -- 172.21.15.46:0/1345566132 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d406e9b0 con 0x7fb1d40ceba0 2024-05-09T03:20:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.955+0000 7fb1cbfff640 1 --2- 172.21.15.46:0/1345566132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d40cefb0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0097a0 tx=0x7fb1bc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ae1babfa7ec7c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d1d9b640 1 -- 172.21.15.46:0/1345566132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc037070 con 0x7fb1d40ceba0 2024-05-09T03:20:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d1d9b640 1 -- 172.21.15.46:0/1345566132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1bc032440 con 0x7fb1d40ceba0 2024-05-09T03:20:03.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d1d9b640 1 -- 172.21.15.46:0/1345566132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc032810 con 0x7fb1d40ceba0 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 -- 172.21.15.46:0/1345566132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 msgr2=0x7fb1d40cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/1345566132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d40cefb0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0097a0 tx=0x7fb1bc02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 -- 172.21.15.46:0/1345566132 shutdown_connections 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/1345566132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d40cefb0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 -- 172.21.15.46:0/1345566132 >> 172.21.15.46:0/1345566132 conn(0x7fb1d40d05f0 msgr2=0x7fb1d40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 -- 172.21.15.46:0/1345566132 shutdown_connections 2024-05-09T03:20:03.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.956+0000 7fb1d8c22640 1 -- 172.21.15.46:0/1345566132 wait complete. 2024-05-09T03:20:03.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.958+0000 7fb1d8c22640 1 Processor -- start 2024-05-09T03:20:03.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.958+0000 7fb1d8c22640 1 -- start start 2024-05-09T03:20:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d4152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d40cf4f0 0x7fb1d40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1d40d3730 0x7fb1d41f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d40b9160 con 0x7fb1d40ceba0 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d40b94a0 con 0x7fb1d40cf4f0 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d408cd60 con 0x7fb1d40d3730 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d4152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d4152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44706/0 (socket says 172.21.15.46:44706) 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 -- 172.21.15.46:0/843802314 learned_addr learned my addr 172.21.15.46:0/843802314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1cbfff640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d40cf4f0 0x7fb1d40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 -- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1d40d3730 msgr2=0x7fb1d41f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1d40d3730 0x7fb1d41f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 -- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d40cf4f0 msgr2=0x7fb1d40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d40cf4f0 0x7fb1d40cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:03.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 -- 172.21.15.46:0/843802314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d40ca8c0 con 0x7fb1d40ceba0 2024-05-09T03:20:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d259c640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d4152060 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c400 tx=0x7fb1c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4017070 con 0x7fb1d40ceba0 2024-05-09T03:20:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1c4005ce0 con 0x7fb1d40ceba0 2024-05-09T03:20:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d40ca340 con 0x7fb1d40ceba0 2024-05-09T03:20:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40072d0 con 0x7fb1d40ceba0 2024-05-09T03:20:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.960+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d40ccb50 con 0x7fb1d40ceba0 2024-05-09T03:20:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.961+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d40b65b0 con 0x7fb1d40ceba0 2024-05-09T03:20:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.962+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1c401a070 con 0x7fb1d40ceba0 2024-05-09T03:20:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.963+0000 7fb1caffd640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1b40762d0 0x7fb1b40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.963+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8777+0+0 (secure 0 0 0) 0x7fb1c4097580 con 0x7fb1d40ceba0 2024-05-09T03:20:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.963+0000 7fb1cbfff640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1b40762d0 0x7fb1b40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.963+0000 7fb1cbfff640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1b40762d0 0x7fb1b40787e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc034000 tx=0x7fb1bc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:03.967+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb1d40b65b0 con 0x7fb1d40ceba0 2024-05-09T03:20:04.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.234+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fb1d40ca0a0 con 0x7fb1d40ceba0 2024-05-09T03:20:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.822+0000 7fb1caffd640 1 -- 172.21.15.46:0/843802314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v309) ==== 107+0+0 (secure 0 0 0) 0x7fb1d40ca0a0 con 0x7fb1d40ceba0 2024-05-09T03:20:04.824 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:20:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1b40762d0 msgr2=0x7fb1b40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1b40762d0 0x7fb1b40787e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc034000 tx=0x7fb1bc02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 msgr2=0x7fb1d4152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d4152060 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c400 tx=0x7fb1c400c8c0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 shutdown_connections 2024-05-09T03:20:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb1d40d3730 0x7fb1d41f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1d40cf4f0 0x7fb1d40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb1b40762d0 0x7fb1b40787e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 --2- 172.21.15.46:0/843802314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d40ceba0 0x7fb1d4152060 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 >> 172.21.15.46:0/843802314 conn(0x7fb1d40d05f0 msgr2=0x7fb1d40727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 shutdown_connections 2024-05-09T03:20:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.826+0000 7fb1d8c22640 1 -- 172.21.15.46:0/843802314 wait complete. 2024-05-09T03:20:04.843 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:20:04.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.902+0000 7fe98d73ef40 1 Processor -- start 2024-05-09T03:20:04.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.902+0000 7fe98d73ef40 1 -- start start 2024-05-09T03:20:04.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe98d73ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448485f0 0x55d24482e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe98d73ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 0x55d2448d5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe98d73ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448d4390 0x55d2448d5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe98d73ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d2448495f0 con 0x55d2448d3a40 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe98d73ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d2447a6f50 con 0x55d2448485f0 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe98d73ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d2448498d0 con 0x55d2448d4390 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 0x55d2448d5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 0x55d2448d5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44714/0 (socket says 172.21.15.46:44714) 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 -- 172.21.15.46:0/1640102394 learned_addr learned my addr 172.21.15.46:0/1640102394 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe987206640 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448485f0 0x55d24482e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe987a07640 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448d4390 0x55d2448d5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 -- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448d4390 msgr2=0x55d2448d5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448d4390 0x55d2448d5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 -- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448485f0 msgr2=0x55d24482e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448485f0 0x55d24482e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe986a05640 1 -- 172.21.15.46:0/1640102394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d244843af0 con 0x55d2448d3a40 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.903+0000 7fe987206640 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448485f0 0x55d24482e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe986a05640 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 0x55d2448d5610 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009720 tx=0x7fe97c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797fc1b57f0fa3d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe985a03640 1 -- 172.21.15.46:0/1640102394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c019070 con 0x55d2448d3a40 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe985a03640 1 -- 172.21.15.46:0/1640102394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c008890 con 0x55d2448d3a40 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe985a03640 1 -- 172.21.15.46:0/1640102394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c008c80 con 0x55d2448d3a40 2024-05-09T03:20:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1640102394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 msgr2=0x55d2448d5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 0x55d2448d5610 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009720 tx=0x7fe97c005a00 comp rx=0 tx=0).stop 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1640102394 shutdown_connections 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448d4390 0x55d2448d5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d3a40 0x55d2448d5610 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1640102394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448485f0 0x55d24482e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.904+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1640102394 >> 172.21.15.46:0/1640102394 conn(0x55d244722ac0 msgr2=0x55d24482b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.905+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1640102394 shutdown_connections 2024-05-09T03:20:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.905+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1640102394 wait complete. 2024-05-09T03:20:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.905+0000 7fe98d73ef40 1 Processor -- start 2024-05-09T03:20:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.906+0000 7fe98d73ef40 1 -- start start 2024-05-09T03:20:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe98d73ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448485f0 0x55d2448c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe98d73ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448d3a40 0x55d2448c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe98d73ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 0x55d244965de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe98d73ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d244842f40 con 0x55d2448d4390 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe98d73ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d244843750 con 0x55d2448d3a40 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe98d73ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55d244843190 con 0x55d2448485f0 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 0x55d244965de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 0x55d244965de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44722/0 (socket says 172.21.15.46:44722) 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 learned_addr learned my addr 172.21.15.46:0/1254988088 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448d3a40 0x55d2448c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448485f0 0x55d2448c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448485f0 msgr2=0x55d2448c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55d2448485f0 0x55d2448c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448d3a40 msgr2=0x55d2448c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:04.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d2448d3a40 0x55d2448c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.907+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d244965930 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.908+0000 7fe987a07640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 0x55d244965de0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fe98000dcd0 tx=0x7fe98000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.908+0000 7fe984a01640 1 -- 172.21.15.46:0/1254988088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98000f650 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.908+0000 7fe984a01640 1 -- 172.21.15.46:0/1254988088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98000f810 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.908+0000 7fe984a01640 1 -- 172.21.15.46:0/1254988088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98000fc00 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.908+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d2448ce9f0 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.908+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24484a9c0 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.909+0000 7fe984a01640 1 -- 172.21.15.46:0/1254988088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe9800203f0 con 0x55d2448d4390 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.910+0000 7fe984a01640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe974074cb0 0x7fe9740771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.910+0000 7fe984a01640 1 -- 172.21.15.46:0/1254988088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8777+0+0 (secure 0 0 0) 0x7fe980097690 con 0x55d2448d4390 2024-05-09T03:20:04.915 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:20:04.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.910+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d244966320 0x55d244964860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.910+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe974074cb0 0x7fe9740771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.910+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244977c60 con 0x55d244966320 2024-05-09T03:20:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.910+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d244966320 0x55d244964860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.911+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d244966320 0x55d244964860 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:04.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.911+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe974074cb0 0x7fe9740771c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe978056000 tx=0x7fe978007030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.915+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v119'1065 uv1038 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.915+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244978000 con 0x55d244966320 2024-05-09T03:20:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.917+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v119'1066 uv1066 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.917+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449783a0 con 0x55d244966320 2024-05-09T03:20:04.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.918+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v119'1067 uv1067 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.920 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:20:04.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.918+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244979750 con 0x55d244966320 2024-05-09T03:20:04.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.920+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v119'1068 uv1068 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.920+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2448c9860 con 0x55d244966320 2024-05-09T03:20:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.921+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v119'1069 uv1069 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.921+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978740 con 0x55d244966320 2024-05-09T03:20:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.923+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v119'1070 uv1070 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.923+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978ae0 con 0x55d244966320 2024-05-09T03:20:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.924+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v119'1071 uv1044 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.924+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244978ed0 con 0x55d244966320 2024-05-09T03:20:04.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.926+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v119'1072 uv1072 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.926+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978ed0 con 0x55d244966320 2024-05-09T03:20:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.927+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v119'1073 uv1073 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.929 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:20:04.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.927+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.929+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v119'1074 uv1074 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.929+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.930+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v119'1075 uv1075 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.930+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.932+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v119'1076 uv1076 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.932+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.933+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v119'1077 uv1064 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.933+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.935+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v119'1078 uv1078 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.935+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.936+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v119'1079 uv1079 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.938 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:20:04.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.936+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.938+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v119'1080 uv1080 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.938+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.939+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v119'1081 uv1081 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.939+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.941+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v119'1082 uv1082 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.941+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.942+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v119'1083 uv1083 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.942+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.944+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v119'1084 uv1057 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.944+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.945+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v119'1085 uv1085 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.945+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.946+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v119'1086 uv1086 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.948 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:20:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.946+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.948+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v119'1087 uv1087 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.948+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.949+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v119'1088 uv1088 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.949+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.950+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v119'1089 uv1089 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.950+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.952+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v119'1090 uv1063 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.952+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.953+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v119'1091 uv1091 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.953+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.955+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v119'1092 uv1092 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.956 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:20:04.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.955+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.956+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v119'1093 uv1093 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.956+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.957+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v119'1094 uv1094 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.957+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244966320 2024-05-09T03:20:04.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.959+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v119'1095 uv1095 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.959+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d244979280 0x55d24497c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.959+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244978e80 con 0x55d244979280 2024-05-09T03:20:04.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.959+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d244979280 0x55d24497c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.960+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d244979280 0x55d24497c010 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:04.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.964+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v119'769 uv749 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.964+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24497c550 con 0x55d244979280 2024-05-09T03:20:04.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.965+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v119'770 uv770 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.965+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497c8f0 con 0x55d244979280 2024-05-09T03:20:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.966+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v119'771 uv771 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.968 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:20:04.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.966+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497d140 con 0x55d244979280 2024-05-09T03:20:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.967+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v119'772 uv772 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.967+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497d4e0 con 0x55d244979280 2024-05-09T03:20:04.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.968+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v119'773 uv773 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.968+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497d880 con 0x55d244979280 2024-05-09T03:20:04.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.970+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v119'774 uv768 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.970+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24497dc20 con 0x55d244979280 2024-05-09T03:20:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.971+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v119'775 uv775 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.971+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.972+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v119'776 uv776 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.974 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:20:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.972+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.974+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v119'777 uv777 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.974+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.975+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v119'778 uv778 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.975+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.976+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v119'779 uv767 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.976+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.978+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v119'780 uv780 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.978+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.980 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:20:04.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.979+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v119'781 uv781 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.979+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981a20 con 0x55d244979280 2024-05-09T03:20:04.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.980+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v119'782 uv782 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.980+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981a20 con 0x55d244979280 2024-05-09T03:20:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.981+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v119'783 uv783 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.981+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.983+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v119'784 uv765 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.983+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.984+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v119'785 uv785 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.984+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.985+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v119'786 uv786 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.987 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:20:04.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.985+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497efd0 con 0x55d244979280 2024-05-09T03:20:04.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.986+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v119'787 uv787 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.986+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497efd0 con 0x55d244979280 2024-05-09T03:20:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.987+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v119'788 uv788 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.987+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244979280 2024-05-09T03:20:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.988+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v119'789 uv789 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244979280 2024-05-09T03:20:04.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.988+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244966320 2024-05-09T03:20:04.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.990+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v119'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.990+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244966320 2024-05-09T03:20:04.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.991+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v119'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.991+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244966320 2024-05-09T03:20:04.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.992+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v119'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.994 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:20:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.992+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497efd0 con 0x55d244966320 2024-05-09T03:20:04.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.993+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v119'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.993+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497efd0 con 0x55d244966320 2024-05-09T03:20:04.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.994+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v119'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.994+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24497dfc0 con 0x55d244966320 2024-05-09T03:20:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.995+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v119'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.995+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d24497e4d0 0x55d2449808f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.995+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244980e30 con 0x55d24497e4d0 2024-05-09T03:20:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.995+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d24497e4d0 0x55d2449808f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:04.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.995+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d24497e4d0 0x55d2449808f0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.999+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v119'157 uv156 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:04.999+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244980ef0 con 0x55d24497e4d0 2024-05-09T03:20:05.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.000+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v119'158 uv158 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.000+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981330 con 0x55d24497e4d0 2024-05-09T03:20:05.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.001+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v119'159 uv159 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.003 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:20:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.001+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981330 con 0x55d24497e4d0 2024-05-09T03:20:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.002+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v119'160 uv160 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.002+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981330 con 0x55d244966320 2024-05-09T03:20:05.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.003+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v119'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.003+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244981330 con 0x55d244966320 2024-05-09T03:20:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.005+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v119'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.005+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981330 con 0x55d244966320 2024-05-09T03:20:05.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.006+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v119'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.008 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:20:05.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.006+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244982340 con 0x55d244966320 2024-05-09T03:20:05.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.007+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v119'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.007+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244982340 con 0x55d244966320 2024-05-09T03:20:05.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.009+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v119'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.009+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244981330 con 0x55d244966320 2024-05-09T03:20:05.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.010+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v119'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.010+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d244981840 0x55d244983c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.010+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244984270 con 0x55d244981840 2024-05-09T03:20:05.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.011+0000 7fe987a07640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d244981840 0x55d244983c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.011+0000 7fe987a07640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d244981840 0x55d244983c60 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.012+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v119'382 uv376 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.012+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244984730 con 0x55d244981840 2024-05-09T03:20:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.013+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v119'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.013+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244984c10 con 0x55d244981840 2024-05-09T03:20:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.014+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v119'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.016 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:20:05.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.014+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449864c0 con 0x55d244981840 2024-05-09T03:20:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.016+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v119'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.016+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244986860 con 0x55d244981840 2024-05-09T03:20:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.017+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v119'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.017+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244981840 2024-05-09T03:20:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.018+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v119'387 uv387 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.018+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244981840 2024-05-09T03:20:05.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.019+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v119'388 uv381 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.019+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244981840 2024-05-09T03:20:05.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.020+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v119'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.020+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244981840 2024-05-09T03:20:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.021+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v119'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.023 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:20:05.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.021+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244981840 2024-05-09T03:20:05.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.022+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v119'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.022+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244981840 2024-05-09T03:20:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.023+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v119'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.024+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d244985db0 0x55d244988c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.024+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449854b0 con 0x55d244985db0 2024-05-09T03:20:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.024+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d244985db0 0x55d244988c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.024+0000 7fe987206640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d244985db0 0x55d244988c10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.025+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v119'173 uv172 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244985db0 2024-05-09T03:20:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.025+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244985db0 2024-05-09T03:20:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.027+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v119'174 uv174 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244985db0 2024-05-09T03:20:05.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.027+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244985db0 2024-05-09T03:20:05.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.028+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v119'175 uv175 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244985db0 2024-05-09T03:20:05.029 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:20:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.028+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244985db0 2024-05-09T03:20:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.029+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v119'176 uv176 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244985db0 2024-05-09T03:20:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.029+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244985db0 2024-05-09T03:20:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.030+0000 7fe987206640 1 -- 172.21.15.46:0/1254988088 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v119'177 uv177 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe978054b50 con 0x55d244985db0 2024-05-09T03:20:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.030+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244966320 2024-05-09T03:20:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.031+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v119'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.031+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244966320 2024-05-09T03:20:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.032+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v119'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.032+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244966320 2024-05-09T03:20:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.033+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v119'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.035 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:20:05.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.034+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498a160 con 0x55d244966320 2024-05-09T03:20:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.035+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v119'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.035+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498a160 con 0x55d244966320 2024-05-09T03:20:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.036+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v119'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.036+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d244966320 2024-05-09T03:20:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.037+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v119'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244966320 2024-05-09T03:20:05.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.037+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d24497e4d0 2024-05-09T03:20:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.038+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v119'345 uv339 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.038+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244989150 con 0x55d24497e4d0 2024-05-09T03:20:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.039+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v119'346 uv346 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.039+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989150 con 0x55d24497e4d0 2024-05-09T03:20:05.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.040+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v119'347 uv347 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.042 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:20:05.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.040+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989510 con 0x55d24497e4d0 2024-05-09T03:20:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.041+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v119'348 uv348 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.041+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989510 con 0x55d24497e4d0 2024-05-09T03:20:05.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.042+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v119'349 uv349 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.042+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989510 con 0x55d24497e4d0 2024-05-09T03:20:05.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.043+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v119'350 uv344 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.043+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d244989510 con 0x55d24497e4d0 2024-05-09T03:20:05.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.045+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v119'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.045+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989510 con 0x55d24497e4d0 2024-05-09T03:20:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.046+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v119'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.047 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:20:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.046+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d244989510 con 0x55d24497e4d0 2024-05-09T03:20:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.047+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v119'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.047+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d2449898d0 con 0x55d24497e4d0 2024-05-09T03:20:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.048+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v119'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.048+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d244989f90 0x55d24498c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.048+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244989f90 2024-05-09T03:20:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.048+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d244989f90 0x55d24498c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.048+0000 7fe986a05640 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d244989f90 0x55d24498c3b0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.050+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v119'152 uv150 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244989f90 2024-05-09T03:20:05.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.050+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244989f90 2024-05-09T03:20:05.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.051+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v119'153 uv153 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244989f90 2024-05-09T03:20:05.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.051+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244989f90 2024-05-09T03:20:05.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.052+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v119'154 uv154 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244989f90 2024-05-09T03:20:05.053 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:20:05.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.052+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244989f90 2024-05-09T03:20:05.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.053+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v119'155 uv155 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d244989f90 2024-05-09T03:20:05.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.053+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d24497e4d0 2024-05-09T03:20:05.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.054+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v119'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.054+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d24497e4d0 2024-05-09T03:20:05.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.056+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v119'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.056+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d24497e4d0 2024-05-09T03:20:05.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.057+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v119'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.058 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:20:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.057+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498d900 con 0x55d24497e4d0 2024-05-09T03:20:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.058+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v119'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.058+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498d900 con 0x55d24497e4d0 2024-05-09T03:20:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.059+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v119'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.059+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d24497e4d0 2024-05-09T03:20:05.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.060+0000 7fe986a05640 1 -- 172.21.15.46:0/1254988088 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v119'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe97c01c040 con 0x55d24497e4d0 2024-05-09T03:20:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.060+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.061+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v119'176 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.062+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.063+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v119'177 uv177 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.063+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.065 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:20:05.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.064+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v119'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.064+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.065+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v119'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.065+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.067+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v119'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.067+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.068+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v119'429 uv420 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.068+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.069+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v119'430 uv430 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.069+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.070+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v119'431 uv431 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.072 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:20:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.070+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.071+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v119'432 uv432 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.071+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.073+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v119'433 uv433 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.073+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.074+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v119'434 uv427 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.074+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.075+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v119'435 uv435 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.075+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.076+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v119'436 uv436 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.078 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:20:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.076+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498d900 con 0x55d244981840 2024-05-09T03:20:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.077+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v119'437 uv437 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.077+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498d900 con 0x55d244981840 2024-05-09T03:20:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.078+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v119'438 uv438 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.079+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498c8f0 con 0x55d244981840 2024-05-09T03:20:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.080+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v119'439 uv439 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.080+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55d24498cda0 con 0x55d244981840 2024-05-09T03:20:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v119'440 uv440 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe980010db0 con 0x55d244981840 2024-05-09T03:20:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d244966320 msgr2=0x55d244964860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55d244966320 0x55d244964860 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d24497e4d0 msgr2=0x55d2449808f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55d24497e4d0 0x55d2449808f0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d244989f90 msgr2=0x55d24498c3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55d244989f90 0x55d24498c3b0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d244979280 msgr2=0x55d24497c010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55d244979280 0x55d24497c010 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 reap_dead start 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d244981840 msgr2=0x55d244983c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55d244981840 0x55d244983c60 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe987a07640 1 -- 172.21.15.46:0/1254988088 reap_dead start 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d244985db0 msgr2=0x55d244988c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d244985db0 0x55d244988c10 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe974074cb0 msgr2=0x7fe9740771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe974074cb0 0x7fe9740771c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fe978056000 tx=0x7fe978007030 comp rx=0 tx=0).stop 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 msgr2=0x55d244965de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 0x55d244965de0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fe98000dcd0 tx=0x7fe98000b6c0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 shutdown_connections 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55d244985db0 0x55d244988c10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe974074cb0 0x7fe9740771c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 --2- 172.21.15.46:0/1254988088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d2448d4390 0x55d244965de0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 >> 172.21.15.46:0/1254988088 conn(0x55d244722ac0 msgr2=0x55d24482ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 shutdown_connections 2024-05-09T03:20:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.081+0000 7fe98d73ef40 1 -- 172.21.15.46:0/1254988088 wait complete. 2024-05-09T03:20:05.086 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 600.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:20:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.182+0000 7fe53df25f40 1 Processor -- start 2024-05-09T03:20:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.182+0000 7fe53df25f40 1 -- start start 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe53df25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08c695f0 0x561c08cf3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe53df25f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 0x561c08c4bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe53df25f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf5390 0x561c08ceb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe53df25f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561c08c6a8d0 con 0x561c08cf4a40 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe53df25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561c08c6a5f0 con 0x561c08c695f0 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe53df25f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561c08c65bb0 con 0x561c08cf5390 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 0x561c08c4bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 0x561c08c4bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44738/0 (socket says 172.21.15.46:44738) 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 -- 172.21.15.46:0/448178883 learned_addr learned my addr 172.21.15.46:0/448178883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe536f1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08c695f0 0x561c08cf3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe537720640 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf5390 0x561c08ceb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 -- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf5390 msgr2=0x561c08ceb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf5390 0x561c08ceb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 -- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08c695f0 msgr2=0x561c08cf3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08c695f0 0x561c08cf3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 -- 172.21.15.46:0/448178883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c08b89350 con 0x561c08cf4a40 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe52ffff640 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 0x561c08c4bc20 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fe520045eb0 tx=0x7fe520051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0cbce5e05b1a91 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe535f1d640 1 -- 172.21.15.46:0/448178883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520062070 con 0x561c08cf4a40 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe535f1d640 1 -- 172.21.15.46:0/448178883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe520054b50 con 0x561c08cf4a40 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.184+0000 7fe535f1d640 1 -- 172.21.15.46:0/448178883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520067460 con 0x561c08cf4a40 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 -- 172.21.15.46:0/448178883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 msgr2=0x561c08c4bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 0x561c08c4bc20 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fe520045eb0 tx=0x7fe520051000 comp rx=0 tx=0).stop 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 -- 172.21.15.46:0/448178883 shutdown_connections 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf5390 0x561c08ceb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08cf4a40 0x561c08c4bc20 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 --2- 172.21.15.46:0/448178883 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08c695f0 0x561c08cf3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 -- 172.21.15.46:0/448178883 >> 172.21.15.46:0/448178883 conn(0x561c08b43ac0 msgr2=0x561c08c61210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 -- 172.21.15.46:0/448178883 shutdown_connections 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.185+0000 7fe53df25f40 1 -- 172.21.15.46:0/448178883 wait complete. 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.186+0000 7fe53df25f40 1 Processor -- start 2024-05-09T03:20:05.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.186+0000 7fe53df25f40 1 -- start start 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.187+0000 7fe53df25f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08c695f0 0x561c08cee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.187+0000 7fe53df25f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf4a40 0x561c08cf0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.187+0000 7fe53df25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08cf5390 0x561c08cf0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.187+0000 7fe53df25f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561c08cf4520 con 0x561c08c695f0 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.187+0000 7fe53df25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561c08c64720 con 0x561c08cf5390 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.187+0000 7fe53df25f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561c08cee7d0 con 0x561c08cf4a40 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08c695f0 0x561c08cee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08c695f0 0x561c08cee0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44740/0 (socket says 172.21.15.46:44740) 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 -- 172.21.15.46:0/1528472753 learned_addr learned my addr 172.21.15.46:0/1528472753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe52ffff640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf4a40 0x561c08cf0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe537720640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08cf5390 0x561c08cf0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 -- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf4a40 msgr2=0x561c08cf0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561c08cf4a40 0x561c08cf0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 -- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08cf5390 msgr2=0x561c08cf0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561c08cf5390 0x561c08cf0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 -- 172.21.15.46:0/1528472753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c08c62920 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe536f1f640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08c695f0 0x561c08cee0a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fe528009ef0 tx=0x7fe528006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe534f1b640 1 -- 172.21.15.46:0/1528472753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52801a070 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe534f1b640 1 -- 172.21.15.46:0/1528472753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe528015680 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561c08ceea20 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe534f1b640 1 -- 172.21.15.46:0/1528472753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe528015a30 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.188+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c08c62be0 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.190+0000 7fe534f1b640 1 -- 172.21.15.46:0/1528472753 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe52801d070 con 0x561c08c695f0 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.190+0000 7fe534f1b640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe518074cb0 0x7fe5180771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.190+0000 7fe534f1b640 1 -- 172.21.15.46:0/1528472753 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8777+0+0 (secure 0 0 0) 0x7fe5280969d0 con 0x561c08c695f0 2024-05-09T03:20:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.191+0000 7fe52ffff640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe518074cb0 0x7fe5180771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.191+0000 7fe53df25f40 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561c08cf1820 0x561c08d99f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.191+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x561c08cf1090 con 0x561c08cf1820 2024-05-09T03:20:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.191+0000 7fe537720640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561c08cf1820 0x561c08d99f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.191+0000 7fe537720640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561c08cf1820 0x561c08d99f80 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.191+0000 7fe52ffff640 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe518074cb0 0x7fe5180771c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe520066000 tx=0x7fe520053000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe537720640 1 -- 172.21.15.46:0/1528472753 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 600.00000000 [omap-set-header] v119'1096 uv1096 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe53000d960 con 0x561c08cf1820 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561c08cf1820 msgr2=0x561c08d99f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561c08cf1820 0x561c08d99f80 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe518074cb0 msgr2=0x7fe5180771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe518074cb0 0x7fe5180771c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fe520066000 tx=0x7fe520053000 comp rx=0 tx=0).stop 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08c695f0 msgr2=0x561c08cee0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 --2- 172.21.15.46:0/1528472753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561c08c695f0 0x561c08cee0a0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fe528009ef0 tx=0x7fe528006750 comp rx=0 tx=0).stop 2024-05-09T03:20:05.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe537720640 1 -- 172.21.15.46:0/1528472753 reap_dead start 2024-05-09T03:20:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 shutdown_connections 2024-05-09T03:20:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 >> 172.21.15.46:0/1528472753 conn(0x561c08b43ac0 msgr2=0x561c08c61210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 shutdown_connections 2024-05-09T03:20:05.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.193+0000 7fe53df25f40 1 -- 172.21.15.46:0/1528472753 wait complete. 2024-05-09T03:20:05.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:20:05.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.314+0000 7f2f4e5e3640 1 Processor -- start 2024-05-09T03:20:05.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.314+0000 7f2f4e5e3640 1 -- start start 2024-05-09T03:20:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f4e5e3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f480b4ee0 0x7f2f48144420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f4e5e3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f480b5530 con 0x7f2f480b65d0 2024-05-09T03:20:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f4e5e3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f480b5810 con 0x7f2f480d1840 2024-05-09T03:20:05.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f4e5e3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2f480b52f0 con 0x7f2f480b4ee0 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f3ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f480b4ee0 0x7f2f48144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f47fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2f480b65d0 0x7f2f4806f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49468/0 (socket says 172.21.15.46:49468) 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.315+0000 7f2f47fff640 1 -- 172.21.15.46:0/308148691 learned_addr learned my addr 172.21.15.46:0/308148691 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.316+0000 7f2f3ffff640 1 -- 172.21.15.46:0/308148691 >> v1:172.21.15.88:6789/0 conn(0x7f2f480d1840 legacy=0x7f2f480d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.316+0000 7f2f3ffff640 1 -- 172.21.15.46:0/308148691 >> v1:172.21.15.46:6789/0 conn(0x7f2f480b65d0 legacy=0x7f2f4806f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.316+0000 7f2f3ffff640 1 -- 172.21.15.46:0/308148691 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f480b5af0 con 0x7f2f480b4ee0 2024-05-09T03:20:05.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.316+0000 7f2f477fe640 1 -- 172.21.15.46:0/308148691 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 445766816 0 0) 0x7f2f480b5530 con 0x7f2f480b65d0 2024-05-09T03:20:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.316+0000 7f2f3ffff640 1 --2- 172.21.15.46:0/308148691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f48144420 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097a0 tx=0x7f2f3002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc0655dcd803d98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f477fe640 1 -- 172.21.15.46:0/308148691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30037070 con 0x7f2f480b4ee0 2024-05-09T03:20:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f477fe640 1 -- 172.21.15.46:0/308148691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f30032440 con 0x7f2f480b4ee0 2024-05-09T03:20:05.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f477fe640 1 -- 172.21.15.46:0/308148691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30032810 con 0x7f2f480b4ee0 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/308148691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 msgr2=0x7f2f48144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/308148691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f48144420 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097a0 tx=0x7f2f3002a230 comp rx=0 tx=0).stop 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/308148691 shutdown_connections 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/308148691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f48144420 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/308148691 >> 172.21.15.46:0/308148691 conn(0x7f2f480b3620 msgr2=0x7f2f480b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/308148691 shutdown_connections 2024-05-09T03:20:05.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.317+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/308148691 wait complete. 2024-05-09T03:20:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.319+0000 7f2f4e5e3640 1 Processor -- start 2024-05-09T03:20:05.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.319+0000 7f2f4e5e3640 1 -- start start 2024-05-09T03:20:05.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.320+0000 7f2f4e5e3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f481605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b65d0 0x7f2f48136c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f480d1840 0x7f2f48137160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48144d50 con 0x7f2f480b4ee0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2f480b6140 con 0x7f2f480d1840 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2f48144ed0 con 0x7f2f480b65d0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f481605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f481605d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44762/0 (socket says 172.21.15.46:44762) 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 -- 172.21.15.46:0/935920820 learned_addr learned my addr 172.21.15.46:0/935920820 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f3ffff640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b65d0 0x7f2f48136c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4cb80640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f480d1840 0x7f2f48137160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 -- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b65d0 msgr2=0x7f2f48136c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b65d0 0x7f2f48136c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 -- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f480d1840 msgr2=0x7f2f48137160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f480d1840 0x7f2f48137160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 -- 172.21.15.46:0/935920820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f480aee20 con 0x7f2f480b4ee0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f47fff640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f481605d0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800c3a0 tx=0x7f2f3800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f380074e0 con 0x7f2f480b4ee0 2024-05-09T03:20:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f380047e0 con 0x7f2f480b4ee0 2024-05-09T03:20:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f480af100 con 0x7f2f480b4ee0 2024-05-09T03:20:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38004c10 con 0x7f2f480b4ee0 2024-05-09T03:20:05.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.321+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f480af6c0 con 0x7f2f480b4ee0 2024-05-09T03:20:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.322+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4806a270 con 0x7f2f480b4ee0 2024-05-09T03:20:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.323+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2f38011070 con 0x7f2f480b4ee0 2024-05-09T03:20:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.324+0000 7f2f457fa640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f280762d0 0x7f2f280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.324+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8777+0+0 (secure 0 0 0) 0x7f2f38098370 con 0x7f2f480b4ee0 2024-05-09T03:20:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.324+0000 7f2f3ffff640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f280762d0 0x7f2f280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:05.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.324+0000 7f2f3ffff640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f280762d0 0x7f2f280787e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2f30034000 tx=0x7f2f3002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:05.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.328+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2f38060a20 con 0x7f2f480b4ee0 2024-05-09T03:20:05.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:05.557+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f2f4806f470 con 0x7f2f480b4ee0 2024-05-09T03:20:06.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.155+0000 7f2f457fa640 1 -- 172.21.15.46:0/935920820 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v310) ==== 162+0+0 (secure 0 0 0) 0x7f2f38064d20 con 0x7f2f480b4ee0 2024-05-09T03:20:06.157 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:20:06.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f280762d0 msgr2=0x7f2f280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f280762d0 0x7f2f280787e0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2f30034000 tx=0x7f2f3002e000 comp rx=0 tx=0).stop 2024-05-09T03:20:06.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 msgr2=0x7f2f481605d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f481605d0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800c3a0 tx=0x7f2f3800c860 comp rx=0 tx=0).stop 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 shutdown_connections 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2f480d1840 0x7f2f48137160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2f480b65d0 0x7f2f48136c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2f280762d0 0x7f2f280787e0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 --2- 172.21.15.46:0/935920820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f480b4ee0 0x7f2f481605d0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 >> 172.21.15.46:0/935920820 conn(0x7f2f480b3620 msgr2=0x7f2f48147560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 shutdown_connections 2024-05-09T03:20:06.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.158+0000 7f2f4e5e3640 1 -- 172.21.15.46:0/935920820 wait complete. 2024-05-09T03:20:06.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.296+0000 7f53e67ce640 1 Processor -- start 2024-05-09T03:20:06.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.296+0000 7f53e67ce640 1 -- start start 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53e67ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 0x7f53e0068a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53e67ce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f53e00b30a0 0x7f53e00d2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53e67ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e0160a40 con 0x7f53e00b2c90 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53e67ce640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f53e0068fa0 con 0x7f53e00b30a0 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53e67ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53e00b34b0 con 0x7f53e0160630 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53df7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f53e00b2c90 0x7f53e00cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43172/0 (socket says 172.21.15.46:43172) 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53df7fe640 1 -- 172.21.15.46:0/2926950494 learned_addr learned my addr 172.21.15.46:0/2926950494 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53dffff640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 0x7f53e0068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53e4d6b640 1 --2- 172.21.15.46:0/2926950494 >> v2:172.21.15.88:3300/0 conn(0x7f53e00b30a0 0x7f53e00d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892661714 0 0) 0x7f53c8001110 con 0x7f53e00b2c90 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.298+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53cc003520 con 0x7f53e00b2c90 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 729625601 0 0) 0x7f53cc003520 con 0x7f53e00b2c90 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 msgr2=0x7f53e0068a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 0x7f53e0068a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 >> v2:172.21.15.88:3300/0 conn(0x7f53e00b30a0 msgr2=0x7f53e00d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 --2- 172.21.15.46:0/2926950494 >> v2:172.21.15.88:3300/0 conn(0x7f53e00b30a0 0x7f53e00d2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e00b27b0 con 0x7f53e00b2c90 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f53c80018c0 con 0x7f53e00b2c90 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 >> v1:172.21.15.46:6789/0 conn(0x7f53e00b2c90 legacy=0x7f53e00cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53cc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 0x7f53cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53cc004d40 0x7f53cc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53e0068fa0 con 0x7f53cc003a00 2024-05-09T03:20:06.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53e00b34b0 con 0x7f53cc004d40 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53cc00baf0 con 0x7f53e0160630 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53dffff640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53cc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53df7fe640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 0x7f53cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53e4d6b640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53cc004d40 0x7f53cc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53dffff640 1 -- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 msgr2=0x7f53cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53dffff640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0160630 0x7f53cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53dffff640 1 -- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53cc004d40 msgr2=0x7f53cc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53dffff640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53cc004d40 0x7f53cc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.299+0000 7f53dffff640 1 -- 172.21.15.46:0/2926950494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc003520 con 0x7f53cc003a00 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.300+0000 7f53dffff640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53cc003e70 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009a90 tx=0x7f53cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d7eff704376ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.300+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0036070 con 0x7f53cc003a00 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.300+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d0031400 con 0x7f53cc003a00 2024-05-09T03:20:06.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.300+0000 7f53deffd640 1 -- 172.21.15.46:0/2926950494 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0009690 con 0x7f53cc003a00 2024-05-09T03:20:06.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e67ce640 1 -- 172.21.15.46:0/2926950494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 msgr2=0x7f53cc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e67ce640 1 --2- 172.21.15.46:0/2926950494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53cc003e70 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009a90 tx=0x7f53cc005830 comp rx=0 tx=0).stop 2024-05-09T03:20:06.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e4d6b640 1 -- 172.21.15.46:0/2926950494 reap_dead start 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e67ce640 1 -- 172.21.15.46:0/2926950494 shutdown_connections 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e67ce640 1 -- 172.21.15.46:0/2926950494 >> 172.21.15.46:0/2926950494 conn(0x7f53e009b7d0 msgr2=0x7f53e00b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e67ce640 1 -- 172.21.15.46:0/2926950494 shutdown_connections 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.301+0000 7f53e67ce640 1 -- 172.21.15.46:0/2926950494 wait complete. 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.303+0000 7f53e67ce640 1 Processor -- start 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.303+0000 7f53e67ce640 1 -- start start 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e67ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53e0152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e67ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53e015fbf0 0x7f53e0156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e67ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0153ed0 0x7f53e01542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e67ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53e006cb00 con 0x7f53cc003a00 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e67ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53e006c820 con 0x7f53e015fbf0 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e67ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f53e00b78d0 con 0x7f53e0153ed0 2024-05-09T03:20:06.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53e0152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53e0152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60570/0 (socket says 172.21.15.46:60570) 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53dffff640 1 -- 172.21.15.46:0/1147077829 learned_addr learned my addr 172.21.15.46:0/1147077829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53e4d6b640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0153ed0 0x7f53e01542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.304+0000 7f53df7fe640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53e015fbf0 0x7f53e0156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dffff640 1 -- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0153ed0 msgr2=0x7f53e01542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dffff640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0153ed0 0x7f53e01542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dffff640 1 -- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53e015fbf0 msgr2=0x7f53e0156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dffff640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53e015fbf0 0x7f53e0156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dffff640 1 -- 172.21.15.46:0/1147077829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e0154820 con 0x7f53cc003a00 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53e4d6b640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0153ed0 0x7f53e01542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dffff640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53e0152b90 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f53d0000c00 tx=0x7f53d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0040020 con 0x7f53cc003a00 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d003b680 con 0x7f53cc003a00 2024-05-09T03:20:06.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e015eb50 con 0x7f53cc003a00 2024-05-09T03:20:06.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0036040 con 0x7f53cc003a00 2024-05-09T03:20:06.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.305+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e015f110 con 0x7f53cc003a00 2024-05-09T03:20:06.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.306+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e006a270 con 0x7f53cc003a00 2024-05-09T03:20:06.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.307+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f53d0043020 con 0x7f53cc003a00 2024-05-09T03:20:06.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.307+0000 7f53dcff9640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53c0076300 0x7f53c0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:06.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.307+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8777+0+0 (secure 0 0 0) 0x7f53d00b89f0 con 0x7f53cc003a00 2024-05-09T03:20:06.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.308+0000 7f53df7fe640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53c0076300 0x7f53c0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:06.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.308+0000 7f53df7fe640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53c0076300 0x7f53c0078810 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f53c8001eb0 tx=0x7f53c8007b20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:06.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.312+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f53d004a030 con 0x7f53cc003a00 2024-05-09T03:20:06.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.539+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53e00af5b0 con 0x7f53cc003a00 2024-05-09T03:20:06.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.540+0000 7f53dcff9640 1 -- 172.21.15.46:0/1147077829 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 311 v311) ==== 95+0+5178 (secure 0 0 0) 0x7f53e0154820 con 0x7f53cc003a00 2024-05-09T03:20:06.541 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:06.542 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":311,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":311,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:06.160308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10125":{"gid":10125,"name":"c","rank":0,"incarnation":311,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/134300288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":134300288},{"type":"v1","addr":"172.21.15.46:6834","nonce":134300288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:06.542 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 311 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53c0076300 msgr2=0x7f53c0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53c0076300 0x7f53c0078810 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f53c8001eb0 tx=0x7f53c8007b20 comp rx=0 tx=0).stop 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 msgr2=0x7f53e0152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53e0152b90 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f53d0000c00 tx=0x7f53d0004730 comp rx=0 tx=0).stop 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 shutdown_connections 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f53e0153ed0 0x7f53e01542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53e015fbf0 0x7f53e0156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f53c0076300 0x7f53c0078810 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 --2- 172.21.15.46:0/1147077829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53cc003a00 0x7f53e0152b90 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:06.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 >> 172.21.15.46:0/1147077829 conn(0x7f53e009b7d0 msgr2=0x7f53e0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:06.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 shutdown_connections 2024-05-09T03:20:06.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:06.542+0000 7f53e67ce640 1 -- 172.21.15.46:0/1147077829 wait complete. 2024-05-09T03:20:06.556 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:20:07.850 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:20:07.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:20:09.183 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:09.181+0000 7ff43eb60640 -1 mds.0.cache.dir(0x600) Corrupt fnode in dirfrag 0x600: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:09.183 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:09.181+0000 7ff43eb60640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-05-09T03:20:09.183 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:09.182+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:20:09.183 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:09.182+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:20:09.190 INFO:tasks.ceph.mds.c.smithi046.stderr: -17> 2024-05-09T03:20:09.181+0000 7ff43eb60640 -1 mds.0.cache.dir(0x600) Corrupt fnode in dirfrag 0x600: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:09.190 INFO:tasks.ceph.mds.c.smithi046.stderr: -14> 2024-05-09T03:20:09.181+0000 7ff43eb60640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-05-09T03:20:09.253 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:11.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.685+0000 7f86b79b7640 1 Processor -- start 2024-05-09T03:20:11.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.685+0000 7f86b79b7640 1 -- start start 2024-05-09T03:20:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b79b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00b84b0 0x7f86b00d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b79b7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86b009bee0 0x7f86b00cd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b79b7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b00cd4a0 con 0x7f86b00d1700 2024-05-09T03:20:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b79b7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f86b0072710 con 0x7f86b009bee0 2024-05-09T03:20:11.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b79b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f86b0071b90 con 0x7f86b00b84b0 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b4f52640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f86b00d1700 0x7f86b009b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43198/0 (socket says 172.21.15.46:43198) 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b4f52640 1 -- 172.21.15.46:0/271467778 learned_addr learned my addr 172.21.15.46:0/271467778 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5753640 1 --2- 172.21.15.46:0/271467778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00b84b0 0x7f86b00d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5f54640 1 --2- 172.21.15.46:0/271467778 >> v2:172.21.15.88:3300/0 conn(0x7f86b009bee0 0x7f86b00cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86a7fff640 1 -- 172.21.15.46:0/271467778 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1463215348 0 0) 0x7f86b00cd4a0 con 0x7f86b00d1700 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86a7fff640 1 -- 172.21.15.46:0/271467778 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f869c003520 con 0x7f86b00d1700 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5753640 1 -- 172.21.15.46:0/271467778 >> v2:172.21.15.88:3300/0 conn(0x7f86b009bee0 msgr2=0x7f86b00cd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5753640 1 --2- 172.21.15.46:0/271467778 >> v2:172.21.15.88:3300/0 conn(0x7f86b009bee0 0x7f86b00cd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5753640 1 -- 172.21.15.46:0/271467778 >> v1:172.21.15.46:6789/0 conn(0x7f86b00d1700 legacy=0x7f86b009b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5753640 1 -- 172.21.15.46:0/271467778 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b0071e70 con 0x7f86b00b84b0 2024-05-09T03:20:11.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.687+0000 7f86b5f54640 1 --2- 172.21.15.46:0/271467778 >> v2:172.21.15.88:3300/0 conn(0x7f86b009bee0 0x7f86b00cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.688+0000 7f86b5753640 1 --2- 172.21.15.46:0/271467778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00b84b0 0x7f86b00d11c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009790 tx=0x7f86ac02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=139bba365b4e0d05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86a7fff640 1 -- 172.21.15.46:0/271467778 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac037070 con 0x7f86b00b84b0 2024-05-09T03:20:11.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86a7fff640 1 -- 172.21.15.46:0/271467778 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86ac032440 con 0x7f86b00b84b0 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86a7fff640 1 -- 172.21.15.46:0/271467778 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0327f0 con 0x7f86b00b84b0 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 -- 172.21.15.46:0/271467778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00b84b0 msgr2=0x7f86b00d11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 --2- 172.21.15.46:0/271467778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00b84b0 0x7f86b00d11c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009790 tx=0x7f86ac02a180 comp rx=0 tx=0).stop 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 -- 172.21.15.46:0/271467778 shutdown_connections 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 --2- 172.21.15.46:0/271467778 >> v2:172.21.15.88:3300/0 conn(0x7f86b009bee0 0x7f86b00cd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 --2- 172.21.15.46:0/271467778 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00b84b0 0x7f86b00d11c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 -- 172.21.15.46:0/271467778 >> 172.21.15.46:0/271467778 conn(0x7f86b00d2c70 msgr2=0x7f86b00d3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 -- 172.21.15.46:0/271467778 shutdown_connections 2024-05-09T03:20:11.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.689+0000 7f86b79b7640 1 -- 172.21.15.46:0/271467778 wait complete. 2024-05-09T03:20:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.691+0000 7f86b79b7640 1 Processor -- start 2024-05-09T03:20:11.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.691+0000 7f86b79b7640 1 -- start start 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b79b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86b00b84b0 0x7f86b0157ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b79b7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00d1700 0x7f86b00b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b79b7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 0x7f86b006c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b79b7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86b0160b20 con 0x7f86b00b3a50 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b79b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f86b0072430 con 0x7f86b00b84b0 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b79b7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f86b0070a10 con 0x7f86b00d1700 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 0x7f86b006c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 0x7f86b006c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60588/0 (socket says 172.21.15.46:60588) 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 -- 172.21.15.46:0/441320699 learned_addr learned my addr 172.21.15.46:0/441320699 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5753640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86b00b84b0 0x7f86b0157ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:11.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b4f52640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00d1700 0x7f86b00b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 -- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00d1700 msgr2=0x7f86b00b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00d1700 0x7f86b00b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 -- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86b00b84b0 msgr2=0x7f86b0157ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86b00b84b0 0x7f86b0157ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 -- 172.21.15.46:0/441320699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b00b2c90 con 0x7f86b00b3a50 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b4f52640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00d1700 0x7f86b00b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5753640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86b00b84b0 0x7f86b0157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.693+0000 7f86b5f54640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 0x7f86b006c680 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009e10 tx=0x7f86a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.694+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8018020 con 0x7f86b00b3a50 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.694+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86a8014480 con 0x7f86b00b3a50 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.694+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b00b2f70 con 0x7f86b00b3a50 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.694+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8014850 con 0x7f86b00b3a50 2024-05-09T03:20:11.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.694+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b00cc010 con 0x7f86b00b3a50 2024-05-09T03:20:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.695+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b00b5260 con 0x7f86b00b3a50 2024-05-09T03:20:11.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.696+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f86a8005d10 con 0x7f86b00b3a50 2024-05-09T03:20:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.696+0000 7f86a5ffb640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f86940762d0 0x7f86940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.696+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f86a8099040 con 0x7f86b00b3a50 2024-05-09T03:20:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.696+0000 7f86b5753640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f86940762d0 0x7f86940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:11.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.697+0000 7f86b5753640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f86940762d0 0x7f86940787e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f86ac034000 tx=0x7f86ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:11.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.701+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f86a8022030 con 0x7f86b00b3a50 2024-05-09T03:20:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.928+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86b0069c60 con 0x7f86b00b3a50 2024-05-09T03:20:11.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.929+0000 7f86a5ffb640 1 -- 172.21.15.46:0/441320699 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) ==== 95+0+5170 (secure 0 0 0) 0x7f86a8010040 con 0x7f86b00b3a50 2024-05-09T03:20:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:11.931 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":316,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10173,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2183116503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2183116503},{"type":"v1","addr":"172.21.15.46:6834","nonce":2183116503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":316}],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:09.183477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:11.931 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 316 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f86940762d0 msgr2=0x7f86940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f86940762d0 0x7f86940787e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f86ac034000 tx=0x7f86ac02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 msgr2=0x7f86b006c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 0x7f86b006c680 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009e10 tx=0x7f86a800c490 comp rx=0 tx=0).stop 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 shutdown_connections 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f86940762d0 0x7f86940787e0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86b00b3a50 0x7f86b006c680 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f86b00d1700 0x7f86b00b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 --2- 172.21.15.46:0/441320699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86b00b84b0 0x7f86b0157ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 >> 172.21.15.46:0/441320699 conn(0x7f86b00d2c70 msgr2=0x7f86b0075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 shutdown_connections 2024-05-09T03:20:11.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:11.931+0000 7f86b79b7640 1 -- 172.21.15.46:0/441320699 wait complete. 2024-05-09T03:20:11.944 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:20:11.944 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:12.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.057+0000 7faf7e82b640 1 Processor -- start 2024-05-09T03:20:12.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.057+0000 7faf7e82b640 1 -- start start 2024-05-09T03:20:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf7e82b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf780d0b90 0x7faf780b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf7e82b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780cf6b0 0x7faf7806d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf7e82b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf780d2b80 con 0x7faf780b92e0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf7e82b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faf780d2e60 con 0x7faf780d0b90 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf7e82b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faf780d3140 con 0x7faf780cf6b0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf77fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faf780b92e0 0x7faf780d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43208/0 (socket says 172.21.15.46:43208) 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf77fff640 1 -- 172.21.15.46:0/3049393458 learned_addr learned my addr 172.21.15.46:0/3049393458 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf777fe640 1 --2- 172.21.15.46:0/3049393458 >> v2:172.21.15.88:3300/0 conn(0x7faf780d0b90 0x7faf780b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.058+0000 7faf7cdc8640 1 --2- 172.21.15.46:0/3049393458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780cf6b0 0x7faf7806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf76ffd640 1 -- 172.21.15.46:0/3049393458 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240244669 0 0) 0x7faf780d2b80 con 0x7faf780b92e0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf76ffd640 1 -- 172.21.15.46:0/3049393458 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf60003520 con 0x7faf780b92e0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf7cdc8640 1 -- 172.21.15.46:0/3049393458 >> v2:172.21.15.88:3300/0 conn(0x7faf780d0b90 msgr2=0x7faf780b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf7cdc8640 1 --2- 172.21.15.46:0/3049393458 >> v2:172.21.15.88:3300/0 conn(0x7faf780d0b90 0x7faf780b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf7cdc8640 1 -- 172.21.15.46:0/3049393458 >> v1:172.21.15.46:6789/0 conn(0x7faf780b92e0 legacy=0x7faf780d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf7cdc8640 1 -- 172.21.15.46:0/3049393458 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf780d3420 con 0x7faf780cf6b0 2024-05-09T03:20:12.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf777fe640 1 --2- 172.21.15.46:0/3049393458 >> v2:172.21.15.88:3300/0 conn(0x7faf780d0b90 0x7faf780b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.059+0000 7faf7cdc8640 1 --2- 172.21.15.46:0/3049393458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780cf6b0 0x7faf7806d790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf70009d90 tx=0x7faf7002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a2ea32b4f2e449 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf76ffd640 1 -- 172.21.15.46:0/3049393458 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70031600 con 0x7faf780cf6b0 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf76ffd640 1 -- 172.21.15.46:0/3049393458 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf700317a0 con 0x7faf780cf6b0 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf76ffd640 1 -- 172.21.15.46:0/3049393458 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70031b50 con 0x7faf780cf6b0 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 -- 172.21.15.46:0/3049393458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780cf6b0 msgr2=0x7faf7806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 --2- 172.21.15.46:0/3049393458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780cf6b0 0x7faf7806d790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faf70009d90 tx=0x7faf7002a5d0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 -- 172.21.15.46:0/3049393458 shutdown_connections 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 --2- 172.21.15.46:0/3049393458 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780cf6b0 0x7faf7806d790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 --2- 172.21.15.46:0/3049393458 >> v2:172.21.15.88:3300/0 conn(0x7faf780d0b90 0x7faf780b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 -- 172.21.15.46:0/3049393458 >> 172.21.15.46:0/3049393458 conn(0x7faf780d3c00 msgr2=0x7faf780d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 -- 172.21.15.46:0/3049393458 shutdown_connections 2024-05-09T03:20:12.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.060+0000 7faf7e82b640 1 -- 172.21.15.46:0/3049393458 wait complete. 2024-05-09T03:20:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.062+0000 7faf7e82b640 1 Processor -- start 2024-05-09T03:20:12.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.062+0000 7faf7e82b640 1 -- start start 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7e82b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780b92e0 0x7faf780b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7e82b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 0x7faf78155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7e82b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf780cc4a0 0x7faf780cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7e82b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf7806e060 con 0x7faf780cf6b0 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7e82b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faf7806dee0 con 0x7faf780cc4a0 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7e82b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faf7806e1e0 con 0x7faf780b92e0 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 0x7faf78155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 0x7faf78155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60602/0 (socket says 172.21.15.46:60602) 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 -- 172.21.15.46:0/4289462640 learned_addr learned my addr 172.21.15.46:0/4289462640 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf77fff640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780b92e0 0x7faf780b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf7cdc8640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf780cc4a0 0x7faf780cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 -- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780b92e0 msgr2=0x7faf780b8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780b92e0 0x7faf780b8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 -- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf780cc4a0 msgr2=0x7faf780cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf780cc4a0 0x7faf780cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 -- 172.21.15.46:0/4289462640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf780cb400 con 0x7faf780cf6b0 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.064+0000 7faf777fe640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 0x7faf78155f80 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00e8e0 tx=0x7faf6c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.065+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c017070 con 0x7faf780cf6b0 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.065+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf6c00ca00 con 0x7faf780cf6b0 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.065+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf780cb6e0 con 0x7faf780cf6b0 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.065+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c00cdb0 con 0x7faf780cf6b0 2024-05-09T03:20:12.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.065+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf780cbca0 con 0x7faf780cf6b0 2024-05-09T03:20:12.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.066+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf780cb170 con 0x7faf780cf6b0 2024-05-09T03:20:12.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.067+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faf6c01a070 con 0x7faf780cf6b0 2024-05-09T03:20:12.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.067+0000 7faf74ff9640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf640762d0 0x7faf640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.067+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7faf6c098560 con 0x7faf780cf6b0 2024-05-09T03:20:12.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.067+0000 7faf77fff640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf640762d0 0x7faf640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.068+0000 7faf77fff640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf640762d0 0x7faf640787e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7faf68005880 tx=0x7faf68006aa0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.072+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faf6c060ac0 con 0x7faf780cf6b0 2024-05-09T03:20:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.300+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf7806a270 con 0x7faf780cf6b0 2024-05-09T03:20:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.300+0000 7faf74ff9640 1 -- 172.21.15.46:0/4289462640 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) ==== 95+0+5170 (secure 0 0 0) 0x7faf6c064dc0 con 0x7faf780cf6b0 2024-05-09T03:20:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:12.303 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":316,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10173,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2183116503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2183116503},{"type":"v1","addr":"172.21.15.46:6834","nonce":2183116503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":316}],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:09.183477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:12.303 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 316 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf640762d0 msgr2=0x7faf640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf640762d0 0x7faf640787e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7faf68005880 tx=0x7faf68006aa0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 msgr2=0x7faf78155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 0x7faf78155f80 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00e8e0 tx=0x7faf6c00eda0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 shutdown_connections 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf780cc4a0 0x7faf780cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faf640762d0 0x7faf640787e0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf780cf6b0 0x7faf78155f80 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 --2- 172.21.15.46:0/4289462640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf780b92e0 0x7faf780b8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 >> 172.21.15.46:0/4289462640 conn(0x7faf780d3c00 msgr2=0x7faf78072cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 shutdown_connections 2024-05-09T03:20:12.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.303+0000 7faf7e82b640 1 -- 172.21.15.46:0/4289462640 wait complete. 2024-05-09T03:20:12.316 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:20:12.316 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.429+0000 7fedc6f94640 1 Processor -- start 2024-05-09T03:20:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.429+0000 7fedc6f94640 1 -- start start 2024-05-09T03:20:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc6f94640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fedc00b9310 0x7fedc008cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc6f94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d6b0 0x7fedc0164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc6f94640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc006ebe0 con 0x7fedc008d2a0 2024-05-09T03:20:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc6f94640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fedc006f760 con 0x7fedc00b9310 2024-05-09T03:20:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc6f94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fedc006eec0 con 0x7fedc008d6b0 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fedc00b9310 0x7fedc008cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fedc00b9310 0x7fedc008cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60608/0 (socket says 172.21.15.46:60608) 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 -- 172.21.15.46:0/733604479 learned_addr learned my addr 172.21.15.46:0/733604479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc5531640 1 --2- 172.21.15.46:0/733604479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d6b0 0x7fedc0164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 -- 172.21.15.46:0/733604479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d6b0 msgr2=0x7fedc0164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 --2- 172.21.15.46:0/733604479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d6b0 0x7fedc0164f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 -- 172.21.15.46:0/733604479 >> v1:172.21.15.88:6789/0 conn(0x7fedc008d2a0 legacy=0x7fedc0164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.431+0000 7fedc4d30640 1 -- 172.21.15.46:0/733604479 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc006f1a0 con 0x7fedc00b9310 2024-05-09T03:20:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.432+0000 7fedc4d30640 1 --2- 172.21.15.46:0/733604479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc008cd60 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009870 tx=0x7fedb002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28cc97747cd7972 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.432+0000 7fedbf7fe640 1 -- 172.21.15.46:0/733604479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0037070 con 0x7fedc00b9310 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.432+0000 7fedbf7fe640 1 -- 172.21.15.46:0/733604479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb0032440 con 0x7fedc00b9310 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.432+0000 7fedbf7fe640 1 -- 172.21.15.46:0/733604479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb0032810 con 0x7fedc00b9310 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 -- 172.21.15.46:0/733604479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 msgr2=0x7fedc008cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 --2- 172.21.15.46:0/733604479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc008cd60 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009870 tx=0x7fedb002a270 comp rx=0 tx=0).stop 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 -- 172.21.15.46:0/733604479 shutdown_connections 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 --2- 172.21.15.46:0/733604479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d6b0 0x7fedc0164f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 --2- 172.21.15.46:0/733604479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc008cd60 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 -- 172.21.15.46:0/733604479 >> 172.21.15.46:0/733604479 conn(0x7fedc00b5660 msgr2=0x7fedc00b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 -- 172.21.15.46:0/733604479 shutdown_connections 2024-05-09T03:20:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.433+0000 7fedc6f94640 1 -- 172.21.15.46:0/733604479 wait complete. 2024-05-09T03:20:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.435+0000 7fedc6f94640 1 Processor -- start 2024-05-09T03:20:12.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.435+0000 7fedc6f94640 1 -- start start 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc6f94640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d2a0 0x7fedc015cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc6f94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedc008d6b0 0x7fedc014ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc6f94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc015d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc6f94640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fedc0165480 con 0x7fedc00b9310 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc6f94640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fedc008dac0 con 0x7fedc008d6b0 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc6f94640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fedc0165600 con 0x7fedc008d2a0 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc5531640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc015d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc5531640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc015d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60624/0 (socket says 172.21.15.46:60624) 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.436+0000 7fedc5531640 1 -- 172.21.15.46:0/873716498 learned_addr learned my addr 172.21.15.46:0/873716498 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc4d30640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d2a0 0x7fedc015cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedbffff640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedc008d6b0 0x7fedc014ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc5531640 1 -- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d2a0 msgr2=0x7fedc015cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc5531640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d2a0 0x7fedc015cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc5531640 1 -- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedc008d6b0 msgr2=0x7fedc014ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc5531640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedc008d6b0 0x7fedc014ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc5531640 1 -- 172.21.15.46:0/873716498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc015bea0 con 0x7fedc00b9310 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc5531640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc015d500 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fedb800efc0 tx=0x7fedb800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8018020 con 0x7fedc00b9310 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb8014480 con 0x7fedc00b9310 2024-05-09T03:20:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8014870 con 0x7fedc00b9310 2024-05-09T03:20:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc015c180 con 0x7fedc00b9310 2024-05-09T03:20:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.437+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc015c740 con 0x7fedc00b9310 2024-05-09T03:20:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.438+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc006a270 con 0x7fedc00b9310 2024-05-09T03:20:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.439+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fedb8005d10 con 0x7fedc00b9310 2024-05-09T03:20:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.439+0000 7fedbd7fa640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda00762d0 0x7feda00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.440+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7fedb8098fa0 con 0x7fedc00b9310 2024-05-09T03:20:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.440+0000 7fedc4d30640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda00762d0 0x7feda00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.440+0000 7fedc4d30640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda00762d0 0x7feda00787e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fedb0034000 tx=0x7fedb002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.444+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fedb8022030 con 0x7fedc00b9310 2024-05-09T03:20:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.673+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedc008cef0 con 0x7fedc00b9310 2024-05-09T03:20:12.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.674+0000 7fedbd7fa640 1 -- 172.21.15.46:0/873716498 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 316 v316) ==== 95+0+5170 (secure 0 0 0) 0x7fedb8010040 con 0x7fedc00b9310 2024-05-09T03:20:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:12.676 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":316,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10173,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2183116503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2183116503},{"type":"v1","addr":"172.21.15.46:6834","nonce":2183116503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":316}],"filesystems":[{"mdsmap":{"epoch":315,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:09.183477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:12.676 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 316 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda00762d0 msgr2=0x7feda00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda00762d0 0x7feda00787e0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fedb0034000 tx=0x7fedb002e000 comp rx=0 tx=0).stop 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 msgr2=0x7fedc015d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc015d500 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fedb800efc0 tx=0x7fedb800c490 comp rx=0 tx=0).stop 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 shutdown_connections 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7feda00762d0 0x7feda00787e0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedc00b9310 0x7fedc015d500 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fedc008d6b0 0x7fedc014ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 --2- 172.21.15.46:0/873716498 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fedc008d2a0 0x7fedc015cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 >> 172.21.15.46:0/873716498 conn(0x7fedc00b5660 msgr2=0x7fedc00cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 shutdown_connections 2024-05-09T03:20:12.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.676+0000 7fedc6f94640 1 -- 172.21.15.46:0/873716498 wait complete. 2024-05-09T03:20:12.689 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 600.00000000' led to DAMAGED state 2024-05-09T03:20:12.689 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on mds0_openfiles.0' 2024-05-09T03:20:12.689 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:12.689 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:12.706 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:12.706 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:12.706 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:20:12.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:12.706 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:20:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:20:12.763 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:20:12.763 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:20:12.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.917+0000 7f704c500640 1 Processor -- start 2024-05-09T03:20:12.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.917+0000 7f704c500640 1 -- start start 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704c500640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70440d32c0 0x7f70440b9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704c500640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70440d4a60 con 0x7f70440d2eb0 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704c500640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70440d4d40 con 0x7f70440b7e70 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704c500640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f70440d5020 con 0x7f70440d32c0 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704a29c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f70440d2eb0 0x7f70440b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43214/0 (socket says 172.21.15.46:43214) 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704a29c640 1 -- 172.21.15.46:0/1420972033 learned_addr learned my addr 172.21.15.46:0/1420972033 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704aa9d640 1 --2- 172.21.15.46:0/1420972033 >> v2:172.21.15.88:3300/0 conn(0x7f70440d32c0 0x7f70440b9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704929a640 1 -- 172.21.15.46:0/1420972033 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573912770 0 0) 0x7f70440d4a60 con 0x7f70440d2eb0 2024-05-09T03:20:12.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704929a640 1 -- 172.21.15.46:0/1420972033 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7030003520 con 0x7f70440d2eb0 2024-05-09T03:20:12.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704aa9d640 1 -- 172.21.15.46:0/1420972033 >> v1:172.21.15.88:6789/0 conn(0x7f70440b7e70 legacy=0x7f70440b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704aa9d640 1 -- 172.21.15.46:0/1420972033 >> v1:172.21.15.46:6789/0 conn(0x7f70440d2eb0 legacy=0x7f70440b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.919+0000 7f704aa9d640 1 -- 172.21.15.46:0/1420972033 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70440d5300 con 0x7f70440d32c0 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.920+0000 7f704aa9d640 1 --2- 172.21.15.46:0/1420972033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70440b9cb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f703c009d90 tx=0x7f703c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f547b88e9a38473 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.920+0000 7f704929a640 1 -- 172.21.15.46:0/1420972033 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c007740 con 0x7f70440d32c0 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.920+0000 7f704929a640 1 -- 172.21.15.46:0/1420972033 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f703c0078e0 con 0x7f70440d32c0 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.920+0000 7f704929a640 1 -- 172.21.15.46:0/1420972033 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703c007cb0 con 0x7f70440d32c0 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 -- 172.21.15.46:0/1420972033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 msgr2=0x7f70440b9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 --2- 172.21.15.46:0/1420972033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70440b9cb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f703c009d90 tx=0x7f703c02a6f0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 -- 172.21.15.46:0/1420972033 shutdown_connections 2024-05-09T03:20:12.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 --2- 172.21.15.46:0/1420972033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70440b9cb0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 -- 172.21.15.46:0/1420972033 >> 172.21.15.46:0/1420972033 conn(0x7f704408d390 msgr2=0x7f704408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:12.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 -- 172.21.15.46:0/1420972033 shutdown_connections 2024-05-09T03:20:12.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.921+0000 7f704c500640 1 -- 172.21.15.46:0/1420972033 wait complete. 2024-05-09T03:20:12.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.923+0000 7f704c500640 1 Processor -- start 2024-05-09T03:20:12.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.923+0000 7f704c500640 1 -- start start 2024-05-09T03:20:12.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704c500640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 0x7f70440cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704c500640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70441522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704c500640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70440ce5a0 0x7f70441f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704c500640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70440b5360 con 0x7f70440d2eb0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704c500640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f70440b51e0 con 0x7f70440d32c0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704c500640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7044163bd0 con 0x7f70440ce5a0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704a29c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 0x7f70440cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704a29c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 0x7f70440cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60640/0 (socket says 172.21.15.46:60640) 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.924+0000 7f704a29c640 1 -- 172.21.15.46:0/2773398804 learned_addr learned my addr 172.21.15.46:0/2773398804 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704aa9d640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70440ce5a0 0x7f70441f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f7049a9b640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70441522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704a29c640 1 -- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70440ce5a0 msgr2=0x7f70441f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704a29c640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70440ce5a0 0x7f70441f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704a29c640 1 -- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 msgr2=0x7f70441522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704a29c640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70441522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704a29c640 1 -- 172.21.15.46:0/2773398804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7044152810 con 0x7f70440d2eb0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704a29c640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 0x7f70440cccb0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f704000c3c0 tx=0x7f704000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040017070 con 0x7f70440d2eb0 2024-05-09T03:20:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7040005dc0 con 0x7f70440d2eb0 2024-05-09T03:20:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040007310 con 0x7f70440d2eb0 2024-05-09T03:20:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70440cd500 con 0x7f70440d2eb0 2024-05-09T03:20:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.925+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70440cdac0 con 0x7f70440d2eb0 2024-05-09T03:20:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.926+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f704406a270 con 0x7f70440d2eb0 2024-05-09T03:20:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.927+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f704001a070 con 0x7f70440d2eb0 2024-05-09T03:20:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.927+0000 7f703affd640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7028076300 0x7f7028078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.928+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f7040097660 con 0x7f70440d2eb0 2024-05-09T03:20:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.928+0000 7f7049a9b640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7028076300 0x7f7028078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.928+0000 7f7049a9b640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7028076300 0x7f7028078810 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f70440cc970 tx=0x7f7034009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:12.932+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f704005fc40 con 0x7f70440d2eb0 2024-05-09T03:20:13.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.162+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7044162ff0 con 0x7f70440d2eb0 2024-05-09T03:20:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.197+0000 7f703affd640 1 -- 172.21.15.46:0/2773398804 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v317) ==== 161+0+0 (secure 0 0 0) 0x7f7040063f40 con 0x7f70440d2eb0 2024-05-09T03:20:13.199 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7028076300 msgr2=0x7f7028078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7028076300 0x7f7028078810 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f70440cc970 tx=0x7f7034009040 comp rx=0 tx=0).stop 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 msgr2=0x7f70440cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 0x7f70440cccb0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f704000c3c0 tx=0x7f704000c880 comp rx=0 tx=0).stop 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 shutdown_connections 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f70440ce5a0 0x7f70441f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70440d32c0 0x7f70441522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7028076300 0x7f7028078810 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 --2- 172.21.15.46:0/2773398804 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70440d2eb0 0x7f70440cccb0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 >> 172.21.15.46:0/2773398804 conn(0x7f704408d390 msgr2=0x7f704415e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:13.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 shutdown_connections 2024-05-09T03:20:13.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.200+0000 7f704c500640 1 -- 172.21.15.46:0/2773398804 wait complete. 2024-05-09T03:20:13.213 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:20:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.327+0000 7fa6d03ce640 1 Processor -- start 2024-05-09T03:20:13.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.327+0000 7fa6d03ce640 1 -- start start 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6d03ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 0x7fa6c816b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6d03ce640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6c80b5550 con 0x7fa6c80b5140 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6d03ce640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6c80d44e0 con 0x7fa6c80b7e70 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6d03ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c80d3480 con 0x7fa6c80d3fc0 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6ce16a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6c80b5140 0x7fa6c80b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43224/0 (socket says 172.21.15.46:43224) 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6ce16a640 1 -- 172.21.15.46:0/2278704452 learned_addr learned my addr 172.21.15.46:0/2278704452 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270340619 0 0) 0x7fa6c80b5550 con 0x7fa6c80b5140 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6b4003520 con 0x7fa6c80b5140 2024-05-09T03:20:13.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.329+0000 7fa6ce96b640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 0x7fa6c816b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3522617713 0 0) 0x7fa6b4003520 con 0x7fa6c80b5140 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 msgr2=0x7fa6c816b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 0x7fa6c816b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 >> v1:172.21.15.88:6789/0 conn(0x7fa6c80b7e70 legacy=0x7fa6c80d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c80d47c0 con 0x7fa6c80b5140 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fa6c4001a70 con 0x7fa6c80b5140 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 >> v1:172.21.15.46:6789/0 conn(0x7fa6c80b5140 legacy=0x7fa6c80b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6b40039e0 0x7fa6b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 0x7fa6b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b4004d40 0x7fa6b4005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b400bb50 con 0x7fa6b40039e0 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b400bcd0 con 0x7fa6b4004d40 2024-05-09T03:20:13.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa6b400be80 con 0x7fa6c80d3fc0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6b40039e0 0x7fa6b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6cd969640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 0x7fa6b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce96b640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b4004d40 0x7fa6b4005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce16a640 1 -- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 msgr2=0x7fa6b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c80d3fc0 0x7fa6b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce16a640 1 -- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b4004d40 msgr2=0x7fa6b4005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b4004d40 0x7fa6b4005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.330+0000 7fa6ce16a640 1 -- 172.21.15.46:0/2278704452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b4003520 con 0x7fa6b40039e0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.331+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6b40039e0 0x7fa6b4003e10 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400a850 tx=0x7fa6b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66a3944a602c6cea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.331+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c400a4a0 con 0x7fa6b40039e0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.331+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c402f070 con 0x7fa6b40039e0 2024-05-09T03:20:13.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.331+0000 7fa6cd168640 1 -- 172.21.15.46:0/2278704452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c402bc30 con 0x7fa6b40039e0 2024-05-09T03:20:13.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2278704452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6b40039e0 msgr2=0x7fa6b4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2278704452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6b40039e0 0x7fa6b4003e10 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400a850 tx=0x7fa6b4005830 comp rx=0 tx=0).stop 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6ce96b640 1 -- 172.21.15.46:0/2278704452 reap_dead start 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2278704452 shutdown_connections 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2278704452 >> 172.21.15.46:0/2278704452 conn(0x7fa6c808d3f0 msgr2=0x7fa6c808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2278704452 shutdown_connections 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.332+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2278704452 wait complete. 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.334+0000 7fa6d03ce640 1 Processor -- start 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.334+0000 7fa6d03ce640 1 -- start start 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6d03ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b40039e0 0x7fa6c81f9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6d03ce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c8153da0 0x7fa6c81585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6d03ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 0x7fa6c8162640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6d03ce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c80d5580 con 0x7fa6c8162230 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6d03ce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c80d4d80 con 0x7fa6b40039e0 2024-05-09T03:20:13.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6d03ce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c80d5700 con 0x7fa6c8153da0 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6ce96b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 0x7fa6c8162640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6ce96b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 0x7fa6c8162640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60666/0 (socket says 172.21.15.46:60666) 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6ce96b640 1 -- 172.21.15.46:0/2425595641 learned_addr learned my addr 172.21.15.46:0/2425595641 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b40039e0 0x7fa6c81f9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.335+0000 7fa6cd969640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c8153da0 0x7fa6c81585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6ce96b640 1 -- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c8153da0 msgr2=0x7fa6c81585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6ce96b640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c8153da0 0x7fa6c81585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6ce96b640 1 -- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b40039e0 msgr2=0x7fa6c81f9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6ce96b640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b40039e0 0x7fa6c81f9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6ce96b640 1 -- 172.21.15.46:0/2425595641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c8162b80 con 0x7fa6c8162230 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6ce96b640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 0x7fa6c8162640 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fa6c808f1a0 tx=0x7fa6c0011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0019020 con 0x7fa6c8162230 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c0004240 con 0x7fa6c8162230 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c80cfb90 con 0x7fa6c8162230 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c000f770 con 0x7fa6c8162230 2024-05-09T03:20:13.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.336+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c8159530 con 0x7fa6c8162230 2024-05-09T03:20:13.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.337+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c8158fc0 con 0x7fa6c8162230 2024-05-09T03:20:13.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.338+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa6c001c020 con 0x7fa6c8162230 2024-05-09T03:20:13.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.338+0000 7fa6beffd640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6ac0762d0 0x7fa6ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:13.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.338+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7fa6c0098120 con 0x7fa6c8162230 2024-05-09T03:20:13.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.338+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6ac0762d0 0x7fa6ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:13.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.339+0000 7fa6ce16a640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6ac0762d0 0x7fa6ac0787e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4033000 tx=0x7fa6c4009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:13.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.343+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa6c0023020 con 0x7fa6c8162230 2024-05-09T03:20:13.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:13.615+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa6c806a270 con 0x7fa6c8162230 2024-05-09T03:20:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.204+0000 7fa6beffd640 1 -- 172.21.15.46:0/2425595641 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v318) ==== 101+0+0 (secure 0 0 0) 0x7fa6c8162b80 con 0x7fa6c8162230 2024-05-09T03:20:14.206 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:20:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6ac0762d0 msgr2=0x7fa6ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6ac0762d0 0x7fa6ac0787e0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4033000 tx=0x7fa6c4009030 comp rx=0 tx=0).stop 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 msgr2=0x7fa6c8162640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 0x7fa6c8162640 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fa6c808f1a0 tx=0x7fa6c0011cf0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 shutdown_connections 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa6ac0762d0 0x7fa6ac0787e0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6c8162230 0x7fa6c8162640 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6c8153da0 0x7fa6c81585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 --2- 172.21.15.46:0/2425595641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6b40039e0 0x7fa6c81f9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.206+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 >> 172.21.15.46:0/2425595641 conn(0x7fa6c808d3f0 msgr2=0x7fa6c80b7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.207+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 shutdown_connections 2024-05-09T03:20:14.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.207+0000 7fa6d03ce640 1 -- 172.21.15.46:0/2425595641 wait complete. 2024-05-09T03:20:14.222 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:20:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.280+0000 7fbdb5c39f40 1 Processor -- start 2024-05-09T03:20:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.280+0000 7fbdb5c39f40 1 -- start start 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdb5c39f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cfa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdb5c39f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248cf0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdb5c39f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248c69260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdb5c39f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556248bcaf50 con 0x7fbd980030f0 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdb5c39f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556248bcef90 con 0x556248c68e50 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdb5c39f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556248c53f20 con 0x556248cf0ae0 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cfa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cfa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60690/0 (socket says 172.21.15.46:60690) 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 -- 172.21.15.46:0/129873518 learned_addr learned my addr 172.21.15.46:0/129873518 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaff02640 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248c69260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 -- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 msgr2=0x556248cf0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248cf0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 -- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 msgr2=0x556248c69260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248c69260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 -- 172.21.15.46:0/129873518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556248cefee0 con 0x7fbd980030f0 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdaf701640 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cfa450 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fbda00097b0 tx=0x7fbda0005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5d4763987fe337 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdadefe640 1 -- 172.21.15.46:0/129873518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0007830 con 0x7fbd980030f0 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdadefe640 1 -- 172.21.15.46:0/129873518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda00079d0 con 0x7fbd980030f0 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.282+0000 7fbdadefe640 1 -- 172.21.15.46:0/129873518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0007dc0 con 0x7fbd980030f0 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/129873518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 msgr2=0x556248cfa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cfa450 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fbda00097b0 tx=0x7fbda0005a00 comp rx=0 tx=0).stop 2024-05-09T03:20:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/129873518 shutdown_connections 2024-05-09T03:20:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248c69260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248cf0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/129873518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cfa450 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/129873518 >> 172.21.15.46:0/129873518 conn(0x556248b46ac0 msgr2=0x556248cf8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/129873518 shutdown_connections 2024-05-09T03:20:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.283+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/129873518 wait complete. 2024-05-09T03:20:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.284+0000 7fbdb5c39f40 1 Processor -- start 2024-05-09T03:20:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.284+0000 7fbdb5c39f40 1 -- start start 2024-05-09T03:20:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdb5c39f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248cfa450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdb5c39f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248ce9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdb5c39f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cea010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdb5c39f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556248c50f20 con 0x7fbd980030f0 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdb5c39f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556248cfb500 con 0x556248c68e50 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdb5c39f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556248cefc00 con 0x556248cf0ae0 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdaff02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cea010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdaff02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cea010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60702/0 (socket says 172.21.15.46:60702) 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.285+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 learned_addr learned my addr 172.21.15.46:0/4035895975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248cfa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248ce9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 msgr2=0x556248ce9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaff02640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248ce9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 msgr2=0x556248cfa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaff02640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556248c68e50 0x556248cfa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556248ce8250 con 0x7fbd980030f0 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556248cf0ae0 0x556248ce9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdaff02640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cea010 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x556248c277e0 tx=0x7fbda800e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdacefc640 1 -- 172.21.15.46:0/4035895975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8017030 con 0x7fbd980030f0 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdacefc640 1 -- 172.21.15.46:0/4035895975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda8006020 con 0x7fbd980030f0 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdacefc640 1 -- 172.21.15.46:0/4035895975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8006d30 con 0x7fbd980030f0 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556248ce8cd0 con 0x7fbd980030f0 2024-05-09T03:20:14.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.286+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556248ce7f70 con 0x7fbd980030f0 2024-05-09T03:20:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.288+0000 7fbdacefc640 1 -- 172.21.15.46:0/4035895975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbda8013680 con 0x7fbd980030f0 2024-05-09T03:20:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdacefc640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd9c074ce0 0x7fbd9c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdacefc640 1 -- 172.21.15.46:0/4035895975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7fbda80967e0 con 0x7fbd980030f0 2024-05-09T03:20:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd9c074ce0 0x7fbd9c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.292 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:20:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556248d88ec0 0x556248d892d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9bcb0 con 0x556248d88ec0 2024-05-09T03:20:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556248d88ec0 0x556248d892d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.289+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd9c074ce0 0x7fbd9c0771f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fbda0000c00 tx=0x7fbda0007000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.290+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556248d88ec0 0x556248d892d0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.292+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v120'1097 uv1070 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.292+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248d9c050 con 0x556248d88ec0 2024-05-09T03:20:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.295+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v120'1098 uv1098 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.295+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9c3f0 con 0x556248d88ec0 2024-05-09T03:20:14.297 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:20:14.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.296+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v120'1099 uv1099 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.296+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9d7a0 con 0x556248d88ec0 2024-05-09T03:20:14.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.298+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v120'1100 uv1100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.298+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d89990 con 0x556248d88ec0 2024-05-09T03:20:14.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.299+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v120'1101 uv1101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.299+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9c790 con 0x556248d88ec0 2024-05-09T03:20:14.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.301+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v120'1102 uv1102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.301+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.303+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v120'1103 uv1076 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.303+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.304+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v120'1104 uv1104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.304+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.306+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v120'1105 uv1105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.308 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:20:14.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.306+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.307+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v120'1106 uv1106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.307+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.309+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v120'1107 uv1107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.309+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.310+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v120'1108 uv1108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.310+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.312+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v120'1109 uv1083 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.312+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.313+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v120'1110 uv1110 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.313+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.314+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v120'1111 uv1111 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.316 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:20:14.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.314+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.316+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v120'1112 uv1112 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.316+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.317+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v120'1113 uv1113 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.317+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.319+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v120'1114 uv1114 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.319+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.320+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v120'1115 uv1115 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.320+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.322+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v120'1116 uv1096 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.322+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.324+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v120'1117 uv1117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.324+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.325+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v120'1118 uv1118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.327 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:20:14.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.325+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.327+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v120'1119 uv1119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.327+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.328+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v120'1120 uv1120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.328+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.329+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v120'1121 uv1121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.329+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.331+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v120'1122 uv1095 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.331+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.333+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v120'1123 uv1123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.333+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.334+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v120'1124 uv1124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.336 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:20:14.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.334+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.336+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v120'1125 uv1125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.336+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.337+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v120'1126 uv1126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.337+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248d9cb30 con 0x556248d88ec0 2024-05-09T03:20:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.339+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v120'1127 uv1127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.339+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556248d9db40 0x556248d9ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.339+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da04a0 con 0x556248d9db40 2024-05-09T03:20:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.339+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556248d9db40 0x556248d9ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.339+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556248d9db40 0x556248d9ff60 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.341+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v120'791 uv773 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.341+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da0840 con 0x556248d9db40 2024-05-09T03:20:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.342+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v120'792 uv792 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.343+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da0be0 con 0x556248d9db40 2024-05-09T03:20:14.345 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:20:14.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.344+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v120'793 uv793 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.344+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da0f80 con 0x556248d9db40 2024-05-09T03:20:14.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.345+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v120'794 uv794 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.345+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1320 con 0x556248d9db40 2024-05-09T03:20:14.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.346+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v120'795 uv795 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.346+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da16c0 con 0x556248d9db40 2024-05-09T03:20:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.347+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v120'796 uv778 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.347+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da1a60 con 0x556248d9db40 2024-05-09T03:20:14.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.349+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v120'797 uv797 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.349+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.351 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:20:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.350+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v120'798 uv798 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.350+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.351+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v120'799 uv799 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.351+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.352+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v120'800 uv800 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.352+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.354+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v120'801 uv783 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.354+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.355+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v120'802 uv802 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.355+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.357 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:20:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.356+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v120'803 uv803 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.356+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5860 con 0x556248d9db40 2024-05-09T03:20:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.358+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v120'804 uv804 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.358+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5860 con 0x556248d9db40 2024-05-09T03:20:14.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.359+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v120'805 uv805 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.359+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.360+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v120'806 uv789 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.360+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.361+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v120'807 uv807 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.361+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.362+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v120'808 uv808 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.364 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:20:14.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.362+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da2e10 con 0x556248d9db40 2024-05-09T03:20:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.363+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v120'809 uv809 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.363+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da2e10 con 0x556248d9db40 2024-05-09T03:20:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.364+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v120'810 uv810 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.364+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d9db40 2024-05-09T03:20:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.366+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v120'811 uv811 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248d9db40 2024-05-09T03:20:14.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.366+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d88ec0 2024-05-09T03:20:14.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.367+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v120'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.367+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d88ec0 2024-05-09T03:20:14.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.368+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v120'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.368+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d88ec0 2024-05-09T03:20:14.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.369+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v120'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.371 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:20:14.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.369+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da2e10 con 0x556248d88ec0 2024-05-09T03:20:14.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.370+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v120'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.370+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da2e10 con 0x556248d88ec0 2024-05-09T03:20:14.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.371+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v120'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.371+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da1e00 con 0x556248d88ec0 2024-05-09T03:20:14.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.372+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v120'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.372+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556248da2310 0x556248da4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.372+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da4de0 con 0x556248da2310 2024-05-09T03:20:14.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.372+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556248da2310 0x556248da4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.373+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556248da2310 0x556248da4750 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.374+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v120'161 uv160 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.374+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da51d0 con 0x556248da2310 2024-05-09T03:20:14.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.375+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v120'162 uv162 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.375+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da51d0 con 0x556248da2310 2024-05-09T03:20:14.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.376+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v120'163 uv163 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.377 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:20:14.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.376+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da51d0 con 0x556248da2310 2024-05-09T03:20:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.377+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v120'164 uv164 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.377+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5090 con 0x556248d88ec0 2024-05-09T03:20:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.379+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v120'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.379+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da5090 con 0x556248d88ec0 2024-05-09T03:20:14.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.380+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v120'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.380+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5090 con 0x556248d88ec0 2024-05-09T03:20:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.381+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v120'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.383 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:20:14.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.381+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5090 con 0x556248d88ec0 2024-05-09T03:20:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.382+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v120'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.382+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5090 con 0x556248d88ec0 2024-05-09T03:20:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.383+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v120'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.383+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da5090 con 0x556248d88ec0 2024-05-09T03:20:14.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.385+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v120'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.385+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556248da59b0 0x556248da5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.385+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da85d0 con 0x556248da59b0 2024-05-09T03:20:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.385+0000 7fbdaff02640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556248da59b0 0x556248da5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.385+0000 7fbdaff02640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556248da59b0 0x556248da5dc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.387+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v120'393 uv387 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.387+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da8970 con 0x556248da59b0 2024-05-09T03:20:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.388+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v120'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.388+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da8d10 con 0x556248da59b0 2024-05-09T03:20:14.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.389+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v120'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.391 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:20:14.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.389+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248ce3010 con 0x556248da59b0 2024-05-09T03:20:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.390+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v120'396 uv396 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.390+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248daa610 con 0x556248da59b0 2024-05-09T03:20:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.391+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v120'397 uv397 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.391+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da95a0 con 0x556248da59b0 2024-05-09T03:20:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.392+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v120'398 uv398 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.392+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da99b0 con 0x556248da59b0 2024-05-09T03:20:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.393+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v120'399 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.393+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248da9d50 con 0x556248da59b0 2024-05-09T03:20:14.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.395+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v120'400 uv400 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.395+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da9d50 con 0x556248da59b0 2024-05-09T03:20:14.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.396+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v120'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.398 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:20:14.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.396+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da9d50 con 0x556248da59b0 2024-05-09T03:20:14.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.397+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v120'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.397+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248da9d50 con 0x556248da59b0 2024-05-09T03:20:14.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.398+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v120'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.398+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556248daa110 0x556248dacf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.398+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad460 con 0x556248daa110 2024-05-09T03:20:14.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.398+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556248daa110 0x556248dacf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.399+0000 7fbdaf701640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556248daa110 0x556248dacf20 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.400+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v120'178 uv177 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248daa110 2024-05-09T03:20:14.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.400+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad4a0 con 0x556248daa110 2024-05-09T03:20:14.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.401+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v120'179 uv179 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248daa110 2024-05-09T03:20:14.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.401+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248daa110 2024-05-09T03:20:14.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.402+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v120'180 uv180 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248daa110 2024-05-09T03:20:14.404 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:20:14.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.402+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248daa110 2024-05-09T03:20:14.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.403+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v120'181 uv181 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248daa110 2024-05-09T03:20:14.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.403+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248daa110 2024-05-09T03:20:14.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.404+0000 7fbdaf701640 1 -- 172.21.15.46:0/4035895975 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v120'182 uv182 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda0016440 con 0x556248daa110 2024-05-09T03:20:14.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.404+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248d88ec0 2024-05-09T03:20:14.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.405+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v120'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.405+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248d88ec0 2024-05-09T03:20:14.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.406+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v120'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.406+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248d88ec0 2024-05-09T03:20:14.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.407+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v120'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.409 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:20:14.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.407+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dae8d0 con 0x556248d88ec0 2024-05-09T03:20:14.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.408+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v120'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.408+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dae8d0 con 0x556248d88ec0 2024-05-09T03:20:14.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.409+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v120'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.409+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248d88ec0 2024-05-09T03:20:14.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.410+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v120'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248d88ec0 2024-05-09T03:20:14.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.410+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248da2310 2024-05-09T03:20:14.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.412+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v120'355 uv349 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.412+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248da2310 2024-05-09T03:20:14.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.413+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v120'356 uv356 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.413+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad8c0 con 0x556248da2310 2024-05-09T03:20:14.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.414+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v120'357 uv357 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.415 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:20:14.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.414+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da2310 2024-05-09T03:20:14.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.415+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v120'358 uv358 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.415+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da2310 2024-05-09T03:20:14.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.416+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v120'359 uv359 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.416+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da2310 2024-05-09T03:20:14.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.417+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v120'360 uv354 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.417+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da2310 2024-05-09T03:20:14.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.418+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v120'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.418+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da2310 2024-05-09T03:20:14.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.419+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v120'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.421 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:20:14.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.419+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da2310 2024-05-09T03:20:14.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.420+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v120'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.420+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db0770 con 0x556248da2310 2024-05-09T03:20:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.422+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v120'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.422+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556248dae490 0x556248db0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.422+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248dae490 2024-05-09T03:20:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.422+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556248dae490 0x556248db0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.422+0000 7fbdaef00640 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556248dae490 0x556248db0df0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.423+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v120'157 uv155 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248dae490 2024-05-09T03:20:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.423+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248dae490 2024-05-09T03:20:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.424+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v120'158 uv158 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248dae490 2024-05-09T03:20:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.424+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248dae490 2024-05-09T03:20:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.425+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v120'159 uv159 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248dae490 2024-05-09T03:20:14.427 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:20:14.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.425+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248dae490 2024-05-09T03:20:14.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.426+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v120'160 uv160 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248dae490 2024-05-09T03:20:14.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.427+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248da2310 2024-05-09T03:20:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.428+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v120'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.428+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248da2310 2024-05-09T03:20:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.429+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v120'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.429+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db08b0 con 0x556248da2310 2024-05-09T03:20:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.430+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v120'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.432 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:20:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.430+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad980 con 0x556248da2310 2024-05-09T03:20:14.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.432+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v120'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.432+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad980 con 0x556248da2310 2024-05-09T03:20:14.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.433+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v120'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.433+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad980 con 0x556248da2310 2024-05-09T03:20:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.434+0000 7fbdaef00640 1 -- 172.21.15.46:0/4035895975 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v120'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda4001430 con 0x556248da2310 2024-05-09T03:20:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.434+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.435+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v120'181 uv180 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.435+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.437+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v120'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.437+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.438+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v120'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.440 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:20:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.438+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.439+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v120'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.439+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.440+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v120'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.441+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.442+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v120'442 uv433 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.442+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.443+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v120'443 uv443 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.443+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.444+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v120'444 uv444 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.446 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:20:14.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.444+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.445+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v120'445 uv445 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.445+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.446+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v120'446 uv446 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.447+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.448+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v120'447 uv440 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.448+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.449+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v120'448 uv448 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.449+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.450+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v120'449 uv449 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.452 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:20:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.450+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.451+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v120'450 uv450 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.451+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.452+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v120'451 uv451 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.452+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248dad960 con 0x556248da59b0 2024-05-09T03:20:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.453+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v120'452 uv452 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.453+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x556248db1330 con 0x556248da59b0 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v120'453 uv453 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbda805b060 con 0x556248da59b0 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556248d88ec0 msgr2=0x556248d892d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556248d88ec0 0x556248d892d0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556248da2310 msgr2=0x556248da4750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556248da2310 0x556248da4750 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556248dae490 msgr2=0x556248db0df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556248dae490 0x556248db0df0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556248d9db40 msgr2=0x556248d9ff60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556248d9db40 0x556248d9ff60 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 reap_dead start 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556248da59b0 msgr2=0x556248da5dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556248da59b0 0x556248da5dc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdaff02640 1 -- 172.21.15.46:0/4035895975 reap_dead start 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556248daa110 msgr2=0x556248dacf20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556248daa110 0x556248dacf20 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd9c074ce0 msgr2=0x7fbd9c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd9c074ce0 0x7fbd9c0771f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fbda0000c00 tx=0x7fbda0007000 comp rx=0 tx=0).stop 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 msgr2=0x556248cea010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cea010 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x556248c277e0 tx=0x7fbda800e890 comp rx=0 tx=0).stop 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 shutdown_connections 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556248daa110 0x556248dacf20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbd9c074ce0 0x7fbd9c0771f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 --2- 172.21.15.46:0/4035895975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd980030f0 0x556248cea010 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.454+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 >> 172.21.15.46:0/4035895975 conn(0x556248b46ac0 msgr2=0x556248c24e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.455+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 shutdown_connections 2024-05-09T03:20:14.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.455+0000 7fbdb5c39f40 1 -- 172.21.15.46:0/4035895975 wait complete. 2024-05-09T03:20:14.460 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader mds0_openfiles.0 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:20:14.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.515+0000 7f27f22f6f40 1 Processor -- start 2024-05-09T03:20:14.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.515+0000 7f27f22f6f40 1 -- start start 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27f22f6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 0x557d599105e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27f22f6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557d59911800 0x557d59900d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27f22f6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d598800f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27f22f6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557d597def50 con 0x7f27d80030f0 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27f22f6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557d597e2f90 con 0x557d59901570 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27f22f6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557d5987c4d0 con 0x557d59911800 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27ebdc0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 0x557d599105e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27ebdc0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 0x557d599105e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60732/0 (socket says 172.21.15.46:60732) 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.516+0000 7f27ebdc0640 1 -- 172.21.15.46:0/499024159 learned_addr learned my addr 172.21.15.46:0/499024159 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ec5c1640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d598800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27eb5bf640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557d59911800 0x557d59900d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ebdc0640 1 -- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557d59911800 msgr2=0x557d59900d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:14.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557d59911800 0x557d59900d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ebdc0640 1 -- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 msgr2=0x557d598800f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d598800f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ebdc0640 1 -- 172.21.15.46:0/499024159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d598637f0 con 0x7f27d80030f0 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ec5c1640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d598800f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27eb5bf640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557d59911800 0x557d59900d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 0x557d599105e0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009570 tx=0x7f27e4009a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52fbf12ee974f3f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ea5bd640 1 -- 172.21.15.46:0/499024159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400e860 con 0x7f27d80030f0 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ea5bd640 1 -- 172.21.15.46:0/499024159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e4013030 con 0x7f27d80030f0 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.517+0000 7f27ea5bd640 1 -- 172.21.15.46:0/499024159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e400ec20 con 0x7f27d80030f0 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 -- 172.21.15.46:0/499024159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 msgr2=0x557d599105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 0x557d599105e0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009570 tx=0x7f27e4009a30 comp rx=0 tx=0).stop 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 -- 172.21.15.46:0/499024159 shutdown_connections 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d598800f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x557d59911800 0x557d59900d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/499024159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d80030f0 0x557d599105e0 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 -- 172.21.15.46:0/499024159 >> 172.21.15.46:0/499024159 conn(0x557d5975aac0 msgr2=0x557d59862d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:14.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 -- 172.21.15.46:0/499024159 shutdown_connections 2024-05-09T03:20:14.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.518+0000 7f27f22f6f40 1 -- 172.21.15.46:0/499024159 wait complete. 2024-05-09T03:20:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.519+0000 7f27f22f6f40 1 Processor -- start 2024-05-09T03:20:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.519+0000 7f27f22f6f40 1 -- start start 2024-05-09T03:20:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27f22f6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d59910a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27f22f6f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d59911800 0x557d59839de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27f22f6f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27d80030f0 0x557d598fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27f22f6f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557d59863510 con 0x557d59911800 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27f22f6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557d598659a0 con 0x557d59901570 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27f22f6f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x557d59865c80 con 0x7f27d80030f0 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d59911800 0x557d59839de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d59911800 0x557d59839de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60738/0 (socket says 172.21.15.46:60738) 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 -- 172.21.15.46:0/3626445273 learned_addr learned my addr 172.21.15.46:0/3626445273 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d59910a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27ec5c1640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27d80030f0 0x557d598fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 -- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27d80030f0 msgr2=0x557d598fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27d80030f0 0x557d598fd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 -- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 msgr2=0x557d59910a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d59910a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27eb5bf640 1 -- 172.21.15.46:0/3626445273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d5985e1b0 con 0x557d59911800 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.520+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557d59901570 0x557d59910a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:14.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.521+0000 7f27eb5bf640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d59911800 0x557d59839de0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f27d4046180 tx=0x7f27d4003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.521+0000 7f27e95bb640 1 -- 172.21.15.46:0/3626445273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d4068070 con 0x557d59911800 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.521+0000 7f27e95bb640 1 -- 172.21.15.46:0/3626445273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d404f880 con 0x557d59911800 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.521+0000 7f27e95bb640 1 -- 172.21.15.46:0/3626445273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d404fc50 con 0x557d59911800 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.521+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557d5985e490 con 0x557d59911800 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.521+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d59862480 con 0x557d59911800 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.522+0000 7f27e95bb640 1 -- 172.21.15.46:0/3626445273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f27d4063400 con 0x557d59911800 2024-05-09T03:20:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.523+0000 7f27e95bb640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27cc074cb0 0x7f27cc0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.523+0000 7f27e95bb640 1 -- 172.21.15.46:0/3626445273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f27d40da370 con 0x557d59911800 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.523+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27cc074cb0 0x7f27cc0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.523+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557d59866150 0x557d59866560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.523+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x557d59900400 con 0x557d59866150 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.523+0000 7f27ec5c1640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557d59866150 0x557d59866560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.524+0000 7f27ebdc0640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27cc074cb0 0x7f27cc0771c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009570 tx=0x7f27e401b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.524+0000 7f27ec5c1640 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557d59866150 0x557d59866560 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27ec5c1640 1 -- 172.21.15.46:0/3626445273 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-set-header] v120'165 uv165 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f27dc006550 con 0x557d59866150 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557d59866150 msgr2=0x557d59866560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x557d59866150 0x557d59866560 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27cc074cb0 msgr2=0x7f27cc0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27cc074cb0 0x7f27cc0771c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009570 tx=0x7f27e401b040 comp rx=0 tx=0).stop 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d59911800 msgr2=0x557d59839de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 --2- 172.21.15.46:0/3626445273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d59911800 0x557d59839de0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f27d4046180 tx=0x7f27d4003020 comp rx=0 tx=0).stop 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27ec5c1640 1 -- 172.21.15.46:0/3626445273 reap_dead start 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 shutdown_connections 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 >> 172.21.15.46:0/3626445273 conn(0x557d5975aac0 msgr2=0x557d59824520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 shutdown_connections 2024-05-09T03:20:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.525+0000 7f27f22f6f40 1 -- 172.21.15.46:0/3626445273 wait complete. 2024-05-09T03:20:14.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:20:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.646+0000 7f945636c640 1 Processor -- start 2024-05-09T03:20:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.646+0000 7f945636c640 1 -- start start 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.647+0000 7f945636c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94500bb770 0x7f94500d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.647+0000 7f945636c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9450163eb0 0x7f94501642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.647+0000 7f945636c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9450164900 con 0x7f94500d2a70 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.647+0000 7f945636c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f94500d4bb0 con 0x7f94500bb770 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.647+0000 7f945636c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9450164be0 con 0x7f9450163eb0 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94500bb770 0x7f94500d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94500bb770 0x7f94500d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60750/0 (socket says 172.21.15.46:60750) 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 -- 172.21.15.46:0/747408474 learned_addr learned my addr 172.21.15.46:0/747408474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944f7fe640 1 --2- 172.21.15.46:0/747408474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9450163eb0 0x7f94501642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 -- 172.21.15.46:0/747408474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9450163eb0 msgr2=0x7f94501642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 --2- 172.21.15.46:0/747408474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9450163eb0 0x7f94501642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 -- 172.21.15.46:0/747408474 >> v1:172.21.15.88:6789/0 conn(0x7f94500d2a70 legacy=0x7f94500d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944ffff640 1 -- 172.21.15.46:0/747408474 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94500d4840 con 0x7f94500bb770 2024-05-09T03:20:14.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.648+0000 7f944effd640 1 -- 172.21.15.46:0/747408474 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 337849592 0 0) 0x7f9450164900 con 0x7f94500d2a70 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.649+0000 7f944ffff640 1 --2- 172.21.15.46:0/747408474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500bb770 0x7f94500d5040 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f9440009870 tx=0x7f944002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54df57c4d2a72a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.649+0000 7f944effd640 1 -- 172.21.15.46:0/747408474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440037070 con 0x7f94500bb770 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.649+0000 7f944effd640 1 -- 172.21.15.46:0/747408474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9440032440 con 0x7f94500bb770 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.649+0000 7f944effd640 1 -- 172.21.15.46:0/747408474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440032810 con 0x7f94500bb770 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.649+0000 7f945636c640 1 -- 172.21.15.46:0/747408474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500bb770 msgr2=0x7f94500d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.649+0000 7f945636c640 1 --2- 172.21.15.46:0/747408474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500bb770 0x7f94500d5040 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f9440009870 tx=0x7f944002a270 comp rx=0 tx=0).stop 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.650+0000 7f945636c640 1 -- 172.21.15.46:0/747408474 shutdown_connections 2024-05-09T03:20:14.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.650+0000 7f945636c640 1 --2- 172.21.15.46:0/747408474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9450163eb0 0x7f94501642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.650+0000 7f945636c640 1 --2- 172.21.15.46:0/747408474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500bb770 0x7f94500d5040 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.650+0000 7f945636c640 1 -- 172.21.15.46:0/747408474 >> 172.21.15.46:0/747408474 conn(0x7f945006e370 msgr2=0x7f945006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:14.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.650+0000 7f945636c640 1 -- 172.21.15.46:0/747408474 shutdown_connections 2024-05-09T03:20:14.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.650+0000 7f945636c640 1 -- 172.21.15.46:0/747408474 wait complete. 2024-05-09T03:20:14.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.651+0000 7f945636c640 1 Processor -- start 2024-05-09T03:20:14.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.652+0000 7f945636c640 1 -- start start 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f945636c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94500bb770 0x7f945015c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f945636c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 0x7f94501eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f945636c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9450163eb0 0x7f94501eb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f945636c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94500d0bc0 con 0x7f94500d2a70 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f945636c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f94500bbb80 con 0x7f9450163eb0 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f945636c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f94500d0d40 con 0x7f94500bb770 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 0x7f94501eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 0x7f94501eb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60758/0 (socket says 172.21.15.46:60758) 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 -- 172.21.15.46:0/4263182237 learned_addr learned my addr 172.21.15.46:0/4263182237 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f9454909640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9450163eb0 0x7f94501eb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944ffff640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94500bb770 0x7f945015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 -- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94500bb770 msgr2=0x7f945015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94500bb770 0x7f945015c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 -- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9450163eb0 msgr2=0x7f94501eb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9450163eb0 0x7f94501eb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.653+0000 7f944f7fe640 1 -- 172.21.15.46:0/4263182237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945015e560 con 0x7f94500d2a70 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.654+0000 7f944f7fe640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 0x7f94501eb2a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f944400e870 tx=0x7f944400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.654+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9444017070 con 0x7f94500d2a70 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.654+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f944400c940 con 0x7f94500d2a70 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.654+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945015e840 con 0x7f94500d2a70 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.654+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944400cd30 con 0x7f94500d2a70 2024-05-09T03:20:14.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.654+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f945015ee00 con 0x7f94500d2a70 2024-05-09T03:20:14.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.655+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94500b51e0 con 0x7f94500d2a70 2024-05-09T03:20:14.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.656+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f944401a070 con 0x7f94500d2a70 2024-05-09T03:20:14.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.656+0000 7f944cff9640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f943c0762d0 0x7f943c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:14.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.656+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f9444098430 con 0x7f94500d2a70 2024-05-09T03:20:14.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.656+0000 7f944ffff640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f943c0762d0 0x7f943c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:14.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.657+0000 7f944ffff640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f943c0762d0 0x7f943c0787e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f9440034000 tx=0x7f944002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.661+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f94500b51e0 con 0x7f94500d2a70 2024-05-09T03:20:14.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:14.899+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f945015e2f0 con 0x7f94500d2a70 2024-05-09T03:20:15.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.009+0000 7f944cff9640 1 -- 172.21.15.46:0/4263182237 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v319) ==== 162+0+0 (secure 0 0 0) 0x7f9444060aa0 con 0x7f94500d2a70 2024-05-09T03:20:15.011 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:20:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f943c0762d0 msgr2=0x7f943c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f943c0762d0 0x7f943c0787e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f9440034000 tx=0x7f944002e000 comp rx=0 tx=0).stop 2024-05-09T03:20:15.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 msgr2=0x7f94501eb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 0x7f94501eb2a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f944400e870 tx=0x7f944400ed30 comp rx=0 tx=0).stop 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 shutdown_connections 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9450163eb0 0x7f94501eb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f943c0762d0 0x7f943c0787e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94500d2a70 0x7f94501eb2a0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 --2- 172.21.15.46:0/4263182237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f94500bb770 0x7f945015c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.011+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 >> 172.21.15.46:0/4263182237 conn(0x7f945006e370 msgr2=0x7f94500d2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.012+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 shutdown_connections 2024-05-09T03:20:15.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.012+0000 7f945636c640 1 -- 172.21.15.46:0/4263182237 wait complete. 2024-05-09T03:20:15.015 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:15.013+0000 7fcd671bf640 -1 mds.0.openfiles _load_finish: corrupted header/values: End of buffer [buffer:2] 2024-05-09T03:20:15.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.545+0000 7f0134678640 1 Processor -- start 2024-05-09T03:20:15.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.545+0000 7f0134678640 1 -- start start 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0134678640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f012c1615f0 0x7f012c0b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0134678640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f012c0d2cf0 0x7f012c0d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0134678640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f012c0d15b0 con 0x7f012c0d0b80 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0134678640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f012c160b10 con 0x7f012c0d2cf0 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0134678640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f012c160df0 con 0x7f012c1615f0 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0131c13640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f012c0d0b80 0x7f012c0d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43260/0 (socket says 172.21.15.46:43260) 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132c15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f012c0d2cf0 0x7f012c0d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0131c13640 1 -- 172.21.15.46:0/669705794 learned_addr learned my addr 172.21.15.46:0/669705794 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132414640 1 --2- 172.21.15.46:0/669705794 >> v2:172.21.15.88:3300/0 conn(0x7f012c1615f0 0x7f012c0b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0131412640 1 -- 172.21.15.46:0/669705794 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097412434 0 0) 0x7f012c0d15b0 con 0x7f012c0d0b80 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0131412640 1 -- 172.21.15.46:0/669705794 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0114003520 con 0x7f012c0d0b80 2024-05-09T03:20:15.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132c15640 1 -- 172.21.15.46:0/669705794 >> v2:172.21.15.88:3300/0 conn(0x7f012c1615f0 msgr2=0x7f012c0b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132c15640 1 --2- 172.21.15.46:0/669705794 >> v2:172.21.15.88:3300/0 conn(0x7f012c1615f0 0x7f012c0b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132c15640 1 -- 172.21.15.46:0/669705794 >> v1:172.21.15.46:6789/0 conn(0x7f012c0d0b80 legacy=0x7f012c0d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132c15640 1 -- 172.21.15.46:0/669705794 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c1610d0 con 0x7f012c0d2cf0 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.547+0000 7f0132414640 1 --2- 172.21.15.46:0/669705794 >> v2:172.21.15.88:3300/0 conn(0x7f012c1615f0 0x7f012c0b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.548+0000 7f0132c15640 1 --2- 172.21.15.46:0/669705794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c0d2cf0 0x7f012c0d0f90 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f012c08e2a0 tx=0x7f012402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ff813f5b009c23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.548+0000 7f0131412640 1 -- 172.21.15.46:0/669705794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124037070 con 0x7f012c0d2cf0 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.548+0000 7f0131412640 1 -- 172.21.15.46:0/669705794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0124032440 con 0x7f012c0d2cf0 2024-05-09T03:20:15.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.548+0000 7f0131412640 1 -- 172.21.15.46:0/669705794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124032810 con 0x7f012c0d2cf0 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 -- 172.21.15.46:0/669705794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c0d2cf0 msgr2=0x7f012c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 --2- 172.21.15.46:0/669705794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c0d2cf0 0x7f012c0d0f90 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f012c08e2a0 tx=0x7f012402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 -- 172.21.15.46:0/669705794 shutdown_connections 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 --2- 172.21.15.46:0/669705794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c0d2cf0 0x7f012c0d0f90 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 --2- 172.21.15.46:0/669705794 >> v2:172.21.15.88:3300/0 conn(0x7f012c1615f0 0x7f012c0b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 -- 172.21.15.46:0/669705794 >> 172.21.15.46:0/669705794 conn(0x7f012c06e190 msgr2=0x7f012c06e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 -- 172.21.15.46:0/669705794 shutdown_connections 2024-05-09T03:20:15.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.549+0000 7f0134678640 1 -- 172.21.15.46:0/669705794 wait complete. 2024-05-09T03:20:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.551+0000 7f0134678640 1 Processor -- start 2024-05-09T03:20:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.551+0000 7f0134678640 1 -- start start 2024-05-09T03:20:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0134678640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f012c0d0b80 0x7f012c159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0134678640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f012c0d2cf0 0x7f012c155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0134678640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 0x7f012c0ba310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0134678640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f012c1640c0 con 0x7f012c1615f0 2024-05-09T03:20:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0134678640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f012c163f40 con 0x7f012c0d2cf0 2024-05-09T03:20:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0134678640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f012c164240 con 0x7f012c0d0b80 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0132c15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 0x7f012c0ba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0132c15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 0x7f012c0ba310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60780/0 (socket says 172.21.15.46:60780) 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0132c15640 1 -- 172.21.15.46:0/3115574981 learned_addr learned my addr 172.21.15.46:0/3115574981 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.552+0000 7f0132414640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f012c0d0b80 0x7f012c159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0131c13640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f012c0d2cf0 0x7f012c155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0132c15640 1 -- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f012c0d0b80 msgr2=0x7f012c159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0132c15640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f012c0d0b80 0x7f012c159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0132c15640 1 -- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f012c0d2cf0 msgr2=0x7f012c155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0132c15640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f012c0d2cf0 0x7f012c155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0132c15640 1 -- 172.21.15.46:0/3115574981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c1577e0 con 0x7f012c1615f0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0132c15640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 0x7f012c0ba310 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f0124035040 tx=0x7f01240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124037040 con 0x7f012c1615f0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0124042070 con 0x7f012c1615f0 2024-05-09T03:20:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01240308a0 con 0x7f012c1615f0 2024-05-09T03:20:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012c157ac0 con 0x7f012c1615f0 2024-05-09T03:20:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.553+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012c158080 con 0x7f012c1615f0 2024-05-09T03:20:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.554+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f012c15a710 con 0x7f012c1615f0 2024-05-09T03:20:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.555+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0124030a40 con 0x7f012c1615f0 2024-05-09T03:20:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.556+0000 7f012affd640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0118076300 0x7f0118078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.556+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f01240b96e0 con 0x7f012c1615f0 2024-05-09T03:20:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.556+0000 7f0132414640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0118076300 0x7f0118078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.556+0000 7f0132414640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0118076300 0x7f0118078810 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f011c009fd0 tx=0x7f011c0097f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.559+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0124081ce0 con 0x7f012c1615f0 2024-05-09T03:20:15.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.786+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f012c160930 con 0x7f012c1615f0 2024-05-09T03:20:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.786+0000 7f012affd640 1 -- 172.21.15.46:0/3115574981 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) ==== 95+0+5181 (secure 0 0 0) 0x7f012c160930 con 0x7f012c1615f0 2024-05-09T03:20:15.788 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:15.788 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":319,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":319,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:15.009739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10173":{"gid":10173,"name":"c","rank":0,"incarnation":319,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/2183116503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2183116503},{"type":"v1","addr":"172.21.15.46:6834","nonce":2183116503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:15.788 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 319 2024-05-09T03:20:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0118076300 msgr2=0x7f0118078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0118076300 0x7f0118078810 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f011c009fd0 tx=0x7f011c0097f0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 msgr2=0x7f012c0ba310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:15.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 0x7f012c0ba310 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f0124035040 tx=0x7f01240046c0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 shutdown_connections 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0118076300 0x7f0118078810 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f012c1615f0 0x7f012c0ba310 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f012c0d2cf0 0x7f012c155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 --2- 172.21.15.46:0/3115574981 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f012c0d0b80 0x7f012c159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 >> 172.21.15.46:0/3115574981 conn(0x7f012c06e190 msgr2=0x7f012c0d5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 shutdown_connections 2024-05-09T03:20:15.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:15.788+0000 7f0134678640 1 -- 172.21.15.46:0/3115574981 wait complete. 2024-05-09T03:20:15.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 319, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:20:15.009739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10173}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10173': {'gid': 10173, 'name': 'c', 'rank': 0, 'incarnation': 319, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6834/2183116503', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2183116503}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2183116503}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:20:15.802 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10173:up:replay 2024-05-09T03:20:15.802 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:20:20.193 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:20.191+0000 7fcd6b1c7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:20:21.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.334+0000 7f883c0a7640 1 Processor -- start 2024-05-09T03:20:21.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.334+0000 7f883c0a7640 1 -- start start 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.335+0000 7f883c0a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 0x7f883415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.335+0000 7f883c0a7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8834160b20 con 0x7f88341615e0 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.335+0000 7f883c0a7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88340d15f0 con 0x7f8834164c70 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.335+0000 7f883c0a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f883415e520 con 0x7f88340d2b90 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.335+0000 7f8839642640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88341615e0 0x7f88340d2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34372/0 (socket says 172.21.15.46:34372) 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8839642640 1 -- 172.21.15.46:0/739691 learned_addr learned my addr 172.21.15.46:0/739691 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f883a644640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 0x7f883415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767648498 0 0) 0x7f8834160b20 con 0x7f88341615e0 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f881c003520 con 0x7f88341615e0 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 919024341 0 0) 0x7f88340d15f0 con 0x7f8834164c70 2024-05-09T03:20:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8834160b20 con 0x7f8834164c70 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1524420786 0 0) 0x7f881c003520 con 0x7f88341615e0 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 msgr2=0x7f883415dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 0x7f883415dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 >> v1:172.21.15.88:6789/0 conn(0x7f8834164c70 legacy=0x7f88340d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88340d18d0 con 0x7f88341615e0 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f8828001a70 con 0x7f88341615e0 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 >> v1:172.21.15.46:6789/0 conn(0x7f88341615e0 legacy=0x7f88340d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f881c0040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881c004800 0x7f881c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 0x7f881c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0054f0 con 0x7f881c003c60 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f881c00bf00 con 0x7f881c004800 2024-05-09T03:20:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f881c00c080 con 0x7f88340d2b90 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.336+0000 7f8839e43640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f881c0040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f883a644640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 0x7f881c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f8839642640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881c004800 0x7f881c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f8839e43640 1 -- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 msgr2=0x7f881c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f8839e43640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88340d2b90 0x7f881c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f8839e43640 1 -- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881c004800 msgr2=0x7f881c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f8839e43640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f881c004800 0x7f881c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.337+0000 7f8839e43640 1 -- 172.21.15.46:0/739691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c003520 con 0x7f881c003c60 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f8839e43640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f881c0040b0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f882c001900 tx=0x7f881c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08cd2a1f8813fa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c001520 con 0x7f881c003c60 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f882c02acd0 con 0x7f881c003c60 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f8838e41640 1 -- 172.21.15.46:0/739691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c005310 con 0x7f881c003c60 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f883c0a7640 1 -- 172.21.15.46:0/739691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 msgr2=0x7f881c0040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f883c0a7640 1 --2- 172.21.15.46:0/739691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f881c0040b0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f882c001900 tx=0x7f881c005830 comp rx=0 tx=0).stop 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f883a644640 1 -- 172.21.15.46:0/739691 reap_dead start 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f883c0a7640 1 -- 172.21.15.46:0/739691 shutdown_connections 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.338+0000 7f883c0a7640 1 -- 172.21.15.46:0/739691 >> 172.21.15.46:0/739691 conn(0x7f8834073db0 msgr2=0x7f88340741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.339+0000 7f883c0a7640 1 -- 172.21.15.46:0/739691 shutdown_connections 2024-05-09T03:20:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.339+0000 7f883c0a7640 1 -- 172.21.15.46:0/739691 wait complete. 2024-05-09T03:20:21.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.340+0000 7f883c0a7640 1 Processor -- start 2024-05-09T03:20:21.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.340+0000 7f883c0a7640 1 -- start start 2024-05-09T03:20:21.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883c0a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f88340caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883c0a7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88340cf850 0x7f88340cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883c0a7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88341520c0 0x7f88341f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883c0a7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f88341619f0 con 0x7f881c003c60 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883c0a7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8834161150 con 0x7f88340cf850 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883c0a7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f883415dfe0 con 0x7f88341520c0 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f88340caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f88340caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46304/0 (socket says 172.21.15.46:46304) 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 -- 172.21.15.46:0/4177218888 learned_addr learned my addr 172.21.15.46:0/4177218888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839642640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88340cf850 0x7f88340cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f883a644640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88341520c0 0x7f88341f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 -- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88341520c0 msgr2=0x7f88341f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88341520c0 0x7f88341f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 -- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88340cf850 msgr2=0x7f88340cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88340cf850 0x7f88340cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 -- 172.21.15.46:0/4177218888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88340cb260 con 0x7f881c003c60 2024-05-09T03:20:21.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.342+0000 7f8839e43640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f88340caa10 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f882c005040 tx=0x7f882c0012f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.343+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c007a10 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.343+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f882c0075e0 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.343+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f882c001520 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.343+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88340cb540 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.343+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88340cbb00 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.344+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f883408cfd0 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.345+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f882c0069d0 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.345+0000 7f88327fc640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8820076300 0x7f8820078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.345+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f882c0b7970 con 0x7f881c003c60 2024-05-09T03:20:21.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.346+0000 7f8839642640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8820076300 0x7f8820078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.346+0000 7f8839642640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8820076300 0x7f8820078810 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f88340c9660 tx=0x7f8828006040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.350+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f883408cfd0 con 0x7f881c003c60 2024-05-09T03:20:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.578+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f883406a270 con 0x7f881c003c60 2024-05-09T03:20:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.579+0000 7f88327fc640 1 -- 172.21.15.46:0/4177218888 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 322 v322) ==== 95+0+5190 (secure 0 0 0) 0x7f882c0806b0 con 0x7f881c003c60 2024-05-09T03:20:21.581 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:21.581 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":322,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":322,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:18.014385+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10173},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10173":{"gid":10173,"name":"c","rank":0,"incarnation":319,"state":"up:active","state_seq":5,"addr":"172.21.15.46:6834/2183116503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2183116503},{"type":"v1","addr":"172.21.15.46:6834","nonce":2183116503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10173,"qdb_cluster":[10173]},"id":6}]} 2024-05-09T03:20:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 322 2024-05-09T03:20:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8820076300 msgr2=0x7f8820078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8820076300 0x7f8820078810 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f88340c9660 tx=0x7f8828006040 comp rx=0 tx=0).stop 2024-05-09T03:20:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 msgr2=0x7f88340caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f88340caa10 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f882c005040 tx=0x7f882c0012f0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 shutdown_connections 2024-05-09T03:20:21.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88341520c0 0x7f88341f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88340cf850 0x7f88340cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8820076300 0x7f8820078810 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 --2- 172.21.15.46:0/4177218888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f881c003c60 0x7f88340caa10 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 >> 172.21.15.46:0/4177218888 conn(0x7f8834073db0 msgr2=0x7f883415f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 shutdown_connections 2024-05-09T03:20:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.581+0000 7f883c0a7640 1 -- 172.21.15.46:0/4177218888 wait complete. 2024-05-09T03:20:21.594 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 322, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:15:52.455242+0000', 'modified': '2024-05-09T03:20:18.014385+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10173}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10173': {'gid': 10173, 'name': 'c', 'rank': 0, 'incarnation': 319, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.46:6834/2183116503', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 2183116503}, {'type': 'v1', 'addr': '172.21.15.46:6834', 'nonce': 2183116503}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10173, 'qdb_cluster': [10173]} 2024-05-09T03:20:21.594 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:20:21.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:20:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.708+0000 7f4718586640 1 Processor -- start 2024-05-09T03:20:21.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.708+0000 7f4718586640 1 -- start start 2024-05-09T03:20:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.709+0000 7f4718586640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f47100d3fc0 0x7f471016b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.709+0000 7f4718586640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47100b5550 con 0x7f47100b7e70 2024-05-09T03:20:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.709+0000 7f4718586640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47100d44e0 con 0x7f47100b5140 2024-05-09T03:20:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.709+0000 7f4718586640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f47100d3480 con 0x7f47100d3fc0 2024-05-09T03:20:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715b21640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f47100b7e70 0x7f47100d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34386/0 (socket says 172.21.15.46:34386) 2024-05-09T03:20:21.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715b21640 1 -- 172.21.15.46:0/1227766126 learned_addr learned my addr 172.21.15.46:0/1227766126 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4716b23640 1 --2- 172.21.15.46:0/1227766126 >> v2:172.21.15.88:3300/0 conn(0x7f47100d3fc0 0x7f471016b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 584191945 0 0) 0x7f47100b5550 con 0x7f47100b7e70 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f8003520 con 0x7f47100b7e70 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902014041 0 0) 0x7f47100d44e0 con 0x7f47100b5140 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47100b5550 con 0x7f47100b5140 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1174642760 0 0) 0x7f46f8003520 con 0x7f47100b7e70 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4716b23640 1 -- 172.21.15.46:0/1227766126 >> v1:172.21.15.88:6789/0 conn(0x7f47100b5140 legacy=0x7f47100b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4716b23640 1 -- 172.21.15.46:0/1227766126 >> v1:172.21.15.46:6789/0 conn(0x7f47100b7e70 legacy=0x7f47100d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.710+0000 7f4716b23640 1 -- 172.21.15.46:0/1227766126 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47100d47c0 con 0x7f47100d3fc0 2024-05-09T03:20:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.711+0000 7f4716b23640 1 --2- 172.21.15.46:0/1227766126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47100d3fc0 0x7f471016b0d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f47080097a0 tx=0x7f470802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2c1a61243f83d66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.711+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708037070 con 0x7f47100d3fc0 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.711+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470802ace0 con 0x7f47100d3fc0 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.711+0000 7f4715320640 1 -- 172.21.15.46:0/1227766126 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47040016f0 con 0x7f47100d3fc0 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 -- 172.21.15.46:0/1227766126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47100d3fc0 msgr2=0x7f471016b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 --2- 172.21.15.46:0/1227766126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47100d3fc0 0x7f471016b0d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f47080097a0 tx=0x7f470802a250 comp rx=0 tx=0).stop 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 -- 172.21.15.46:0/1227766126 shutdown_connections 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 --2- 172.21.15.46:0/1227766126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47100d3fc0 0x7f471016b0d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 -- 172.21.15.46:0/1227766126 >> 172.21.15.46:0/1227766126 conn(0x7f471008d3f0 msgr2=0x7f471008d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 -- 172.21.15.46:0/1227766126 shutdown_connections 2024-05-09T03:20:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.712+0000 7f4718586640 1 -- 172.21.15.46:0/1227766126 wait complete. 2024-05-09T03:20:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.714+0000 7f4718586640 1 Processor -- start 2024-05-09T03:20:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.714+0000 7f4718586640 1 -- start start 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4718586640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47100b5140 0x7f471015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4718586640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 0x7f4710143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4718586640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4710144530 0x7f4710156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4718586640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47100d5580 con 0x7f47100d3fc0 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4718586640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f47100d4d80 con 0x7f4710144530 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4718586640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f47100d5700 con 0x7f47100b5140 2024-05-09T03:20:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 0x7f4710143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 0x7f4710143ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46308/0 (socket says 172.21.15.46:46308) 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4715b21640 1 -- 172.21.15.46:0/3661818504 learned_addr learned my addr 172.21.15.46:0/3661818504 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4716b23640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4710144530 0x7f4710156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.715+0000 7f4716322640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47100b5140 0x7f471015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4715b21640 1 -- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47100b5140 msgr2=0x7f471015b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4715b21640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47100b5140 0x7f471015b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4715b21640 1 -- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4710144530 msgr2=0x7f4710156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4715b21640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4710144530 0x7f4710156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4715b21640 1 -- 172.21.15.46:0/3661818504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4710155400 con 0x7f47100d3fc0 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4716322640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47100b5140 0x7f471015b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4716b23640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4710144530 0x7f4710156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4715b21640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 0x7f4710143ff0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f470400c4f0 tx=0x7f470400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f470effd640 1 -- 172.21.15.46:0/3661818504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704005ce0 con 0x7f47100d3fc0 2024-05-09T03:20:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f470effd640 1 -- 172.21.15.46:0/3661818504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4704007070 con 0x7f47100d3fc0 2024-05-09T03:20:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f471015a020 con 0x7f47100d3fc0 2024-05-09T03:20:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f470effd640 1 -- 172.21.15.46:0/3661818504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704007460 con 0x7f47100d3fc0 2024-05-09T03:20:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.716+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f471015a5e0 con 0x7f47100d3fc0 2024-05-09T03:20:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.718+0000 7f470effd640 1 -- 172.21.15.46:0/3661818504 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4704007600 con 0x7f47100d3fc0 2024-05-09T03:20:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.719+0000 7f470effd640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46fc076300 0x7f46fc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.719+0000 7f470effd640 1 -- 172.21.15.46:0/3661818504 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f4704097fe0 con 0x7f47100d3fc0 2024-05-09T03:20:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.719+0000 7f4716322640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46fc076300 0x7f46fc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.720+0000 7f4716322640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46fc076300 0x7f46fc078810 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f47000014a0 tx=0x7f4700009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.720+0000 7f471f443740 1 Processor -- start 2024-05-09T03:20:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.720+0000 7f471f443740 1 -- start start 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f471f443740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 0x5601fa304be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef9590 0x5601fa305120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f471f443740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601fa2224e0 con 0x5601f9ef13b0 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601fa1bdff0 con 0x5601f9f1a270 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5601fa21e4a0 con 0x5601f9ef9590 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 0x5601fa304be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 0x5601fa304be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46310/0 (socket says 172.21.15.46:46310) 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f4715b21640 1 -- 172.21.15.46:0/707766193 learned_addr learned my addr 172.21.15.46:0/707766193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.721+0000 7f4716322640 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4716b23640 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef9590 0x5601fa305120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4715b21640 1 -- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef9590 msgr2=0x5601fa305120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4715b21640 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef9590 0x5601fa305120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4715b21640 1 -- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 msgr2=0x5601fa2e9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4715b21640 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4715b21640 1 -- 172.21.15.46:0/707766193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601fa1a4720 con 0x5601f9ef13b0 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f4715b21640 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 0x5601fa304be0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f470405c620 tx=0x7f4704060020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f812be8f4cfb0ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f46f7fff640 1 -- 172.21.15.46:0/707766193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704064950 con 0x5601f9ef13b0 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f46f7fff640 1 -- 172.21.15.46:0/707766193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4704069370 con 0x5601f9ef13b0 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.722+0000 7f46f7fff640 1 -- 172.21.15.46:0/707766193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47040690e0 con 0x5601f9ef13b0 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 -- 172.21.15.46:0/707766193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 msgr2=0x5601fa304be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 0x5601fa304be0 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f470405c620 tx=0x7f4704060020 comp rx=0 tx=0).stop 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 -- 172.21.15.46:0/707766193 shutdown_connections 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef9590 0x5601fa305120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef13b0 0x5601fa304be0 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 --2- 172.21.15.46:0/707766193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 -- 172.21.15.46:0/707766193 >> 172.21.15.46:0/707766193 conn(0x5601f9f0e3c0 msgr2=0x5601fa2bf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 -- 172.21.15.46:0/707766193 shutdown_connections 2024-05-09T03:20:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.723+0000 7f471f443740 1 -- 172.21.15.46:0/707766193 wait complete. 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.724+0000 7f471f443740 1 -- start start 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.724+0000 7f471f443740 1 Processor -- start 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa301390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601fa1a9760 con 0x5601f9ef9590 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601fa303c80 con 0x5601f9f1a270 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5601fa301cb0 con 0x5601f9ef13b0 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 10 client.0 _ll_register_callbacks cb 0x5601fa1cc4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2e64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46318/0 (socket says 172.21.15.46:46318) 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 -- 172.21.15.46:0/2171706877 learned_addr learned my addr 172.21.15.46:0/2171706877 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4716b23640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4716322640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa301390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 -- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 msgr2=0x5601fa301390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa301390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 -- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 msgr2=0x5601fa2e6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 -- 172.21.15.46:0/2171706877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601fa1eabd0 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4716322640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa301390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f4715b21640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2e64f0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f470405cb90 tx=0x7f4704097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b33b04050266c2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f46f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f46f57fa640 1 -- 172.21.15.46:0/2171706877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704064950 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4704069370 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.726+0000 7f46f57fa640 1 -- 172.21.15.46:0/2171706877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f470405fa10 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f46f57fa640 1 -- 172.21.15.46:0/2171706877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=322) ==== 24+0+0 (secure 0 0 0) 0x7f4704018b10 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 10 client.10254 fetch_fsmap learned FSMap version 322 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=322,monmap=2+}) -- 0x5601fa1d0280 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f46f57fa640 1 -- 172.21.15.46:0/2171706877 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4704069370 con 0x5601f9ef9590 2024-05-09T03:20:21.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f46f57fa640 1 -- 172.21.15.46:0/2171706877 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 322) ==== 3494+0+0 (secure 0 0 0) 0x7f4704018cb0 con 0x5601f9ef9590 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 10 client.10254 fetch_fsmap finished waiting for FSMap version 322 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 10 client.10254 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 20 client.10254 populate_metadata read hostname 'smithi046' 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601f9ed8e40 0x5601fa2ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 4 client.10254 mds_command: new command op to 10173 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 --> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601fa2300f0 con 0x5601f9ed8e40 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f4716322640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601f9ed8e40 0x5601fa2ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f46f57fa640 10 client.10254 ms_handle_connect on v2:172.21.15.46:6832/2183116503 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.727+0000 7f4716322640 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601f9ed8e40 0x5601fa2ffcd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f92832e497ad7542 server_cookie=c5f4c851fb47393f in_seq=0 out_seq=0 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f46f57fa640 1 -- 172.21.15.46:0/2171706877 <== mds.0 v2:172.21.15.46:6832/2183116503 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5601fa2300f0 con 0x5601f9ed8e40 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f46f57fa640 10 client.10254 handle_command_reply: tid=0 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 client.10254 shutdown 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 msgr2=0x5601fa2e64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2e64f0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f470405cb90 tx=0x7f4704097040 comp rx=0 tx=0).stop 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 shutdown_connections 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601f9ed8e40 0x5601fa2ffcd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9f1a270 0x5601fa2e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2e64f0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 --2- 172.21.15.46:0/2171706877 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa301390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f46f57fa640 0 client.10254 ms_handle_reset on v2:172.21.15.46:6832/2183116503 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.728+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 >> 172.21.15.46:0/2171706877 conn(0x5601f9f0e3c0 msgr2=0x5601fa2bf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.729+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 shutdown_connections 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.729+0000 7f471f443740 1 -- 172.21.15.46:0/2171706877 wait complete. 2024-05-09T03:20:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.729+0000 7f471f443740 20 client.10254 trim_cache size 0 max 16384 2024-05-09T03:20:21.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.748+0000 7f471f443740 1 Processor -- start 2024-05-09T03:20:21.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.748+0000 7f471f443740 1 -- start start 2024-05-09T03:20:21.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef13b0 0x5601fa2b9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f471f443740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2ba4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9f1a270 0x5601fa304e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f471f443740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4710203b40 con 0x5601f9ef9590 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4710202ea0 con 0x5601f9ef13b0 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f47102034f0 con 0x5601f9f1a270 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2ba4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f4715b21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2ba4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46320/0 (socket says 172.21.15.46:46320) 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f4715b21640 1 -- 172.21.15.46:0/423260129 learned_addr learned my addr 172.21.15.46:0/423260129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f4716322640 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef13b0 0x5601fa2b9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.749+0000 7f4716b23640 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9f1a270 0x5601fa304e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f4715b21640 1 -- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9f1a270 msgr2=0x5601fa304e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f4715b21640 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9f1a270 0x5601fa304e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f4715b21640 1 -- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef13b0 msgr2=0x5601fa2b9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f4715b21640 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef13b0 0x5601fa2b9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f4715b21640 1 -- 172.21.15.46:0/423260129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f9f85850 con 0x5601f9ef9590 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f4715b21640 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2ba4f0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f470405cb40 tx=0x7f4704063000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5682aa655ef8912d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f46f77fe640 1 -- 172.21.15.46:0/423260129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704064950 con 0x5601f9ef9590 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f46f77fe640 1 -- 172.21.15.46:0/423260129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4704069370 con 0x5601f9ef9590 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.750+0000 7f46f77fe640 1 -- 172.21.15.46:0/423260129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704095270 con 0x5601f9ef9590 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 -- 172.21.15.46:0/423260129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 msgr2=0x5601fa2ba4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2ba4f0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f470405cb40 tx=0x7f4704063000 comp rx=0 tx=0).stop 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 -- 172.21.15.46:0/423260129 shutdown_connections 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9f1a270 0x5601fa304e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9ef9590 0x5601fa2ba4f0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 --2- 172.21.15.46:0/423260129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef13b0 0x5601fa2b9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 -- 172.21.15.46:0/423260129 >> 172.21.15.46:0/423260129 conn(0x5601f9f0e3c0 msgr2=0x5601fa2e4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 -- 172.21.15.46:0/423260129 shutdown_connections 2024-05-09T03:20:21.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.751+0000 7f471f443740 1 -- 172.21.15.46:0/423260129 wait complete. 2024-05-09T03:20:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.752+0000 7f471f443740 1 -- start start 2024-05-09T03:20:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.752+0000 7f471f443740 1 Processor -- start 2024-05-09T03:20:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa381160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef9590 0x5601fa2be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 0x5601fa2b9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5601fa2ff310 con 0x5601f9f1a270 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601fa301f60 con 0x5601f9ef9590 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5601fa2ec8c0 con 0x5601f9ef13b0 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f471f443740 10 client.0 _ll_register_callbacks cb 0x7f4700001610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f4716b23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 0x5601fa2b9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f4716b23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 0x5601fa2b9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46324/0 (socket says 172.21.15.46:46324) 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f4716b23640 1 -- 172.21.15.46:0/1520577955 learned_addr learned my addr 172.21.15.46:0/1520577955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f4715b21640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef9590 0x5601fa2be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.753+0000 7f4716322640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa381160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4716b23640 1 -- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 msgr2=0x5601fa381160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4716b23640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa381160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4716b23640 1 -- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef9590 msgr2=0x5601fa2be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4716b23640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef9590 0x5601fa2be770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4716b23640 1 -- 172.21.15.46:0/1520577955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601f9ffc6b0 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4715b21640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef9590 0x5601fa2be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f4716b23640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 0x5601fa2b9fb0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f47080097a0 tx=0x7f4708004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7694a950599ee4a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f46f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f46f57fa640 1 -- 172.21.15.46:0/1520577955 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708037040 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f46f57fa640 1 -- 172.21.15.46:0/1520577955 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f4708003810 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4704069370 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f46f57fa640 1 -- 172.21.15.46:0/1520577955 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=322) ==== 24+0+0 (secure 0 0 0) 0x7f4708004510 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f471f443740 10 client.10260 fetch_fsmap learned FSMap version 322 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.754+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=322,monmap=2+}) -- 0x5601fa1de630 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f46f57fa640 1 -- 172.21.15.46:0/1520577955 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4704069370 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f46f57fa640 1 -- 172.21.15.46:0/1520577955 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 322) ==== 3494+0+0 (secure 0 0 0) 0x7f470802ace0 con 0x5601f9f1a270 2024-05-09T03:20:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 10 client.10260 fetch_fsmap finished waiting for FSMap version 322 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 10 client.10260 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 20 client.10260 populate_metadata read hostname 'smithi046' 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601fa37b3f0 0x5601fa2bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 4 client.10260 mds_command: new command op to 10173 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 --> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] -- command(tid 0: {"prefix": "status"}) -- 0x5601fa229f60 con 0x5601fa37b3f0 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f4716322640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601fa37b3f0 0x5601fa2bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f46f57fa640 10 client.10260 ms_handle_connect on v2:172.21.15.46:6832/2183116503 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f4716322640 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601fa37b3f0 0x5601fa2bfe90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dead9b92b4cea665 server_cookie=6edceb4518420ae8 in_seq=0 out_seq=0 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f46f57fa640 1 -- 172.21.15.46:0/1520577955 <== mds.0 v2:172.21.15.46:6832/2183116503 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x5601fa229f60 con 0x5601fa37b3f0 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f46f57fa640 10 client.10260 handle_command_reply: tid=0 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.755+0000 7f471f443740 1 client.10260 shutdown 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 msgr2=0x5601fa2b9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 0x5601fa2b9fb0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f47080097a0 tx=0x7f4708004730 comp rx=0 tx=0).stop 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 shutdown_connections 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:6832/2183116503,v1:172.21.15.46:6834/2183116503] conn(0x5601fa37b3f0 0x5601fa2bfe90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601f9f1a270 0x5601fa2b9fb0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601f9ef9590 0x5601fa2be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 --2- 172.21.15.46:0/1520577955 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5601f9ef13b0 0x5601fa381160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f46f57fa640 0 client.10260 ms_handle_reset on v2:172.21.15.46:6832/2183116503 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 >> 172.21.15.46:0/1520577955 conn(0x5601f9f0e3c0 msgr2=0x5601fa2e39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 shutdown_connections 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 1 -- 172.21.15.46:0/1520577955 wait complete. 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f471f443740 20 client.10260 trim_cache size 0 max 16384 2024-05-09T03:20:21.763 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10173, 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.7458349789999996, 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 322, 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 120, 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 120, 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 12.502525811 2024-05-09T03:20:21.764 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:20:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46fc076300 msgr2=0x7f46fc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46fc076300 0x7f46fc078810 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f47000014a0 tx=0x7f4700009040 comp rx=0 tx=0).stop 2024-05-09T03:20:21.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 msgr2=0x7f4710143ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 0x7f4710143ff0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f470400c4f0 tx=0x7f470400c9b0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 shutdown_connections 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4710144530 0x7f4710156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f46fc076300 0x7f46fc078810 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47100d3fc0 0x7f4710143ff0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 --2- 172.21.15.46:0/3661818504 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47100b5140 0x7f471015b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.756+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 >> 172.21.15.46:0/3661818504 conn(0x7f471008d3f0 msgr2=0x7f4710161290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.757+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 shutdown_connections 2024-05-09T03:20:21.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.757+0000 7f4718586640 1 -- 172.21.15.46:0/3661818504 wait complete. 2024-05-09T03:20:21.769 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:20:21.769 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap header on mds0_openfiles.0', proceeding to ls 2024-05-09T03:20:21.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:20:21.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.882+0000 7f7a8fb8d640 1 Processor -- start 2024-05-09T03:20:21.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.883+0000 7f7a8fb8d640 1 -- start start 2024-05-09T03:20:21.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8fb8d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a881613c0 0x7f7a880b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8fb8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a880b7d00 0x7f7a88163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8fb8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a880b8110 con 0x7f7a880b5210 2024-05-09T03:20:21.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8fb8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7a880b5620 con 0x7f7a880b7d00 2024-05-09T03:20:21.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8fb8d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7a881609a0 con 0x7f7a881613c0 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8d128640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a881613c0 0x7f7a880b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8e12a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a880b7d00 0x7f7a88163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8d929640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7a880b5210 0x7f7a880d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:41560/0 (socket says 172.21.15.46:41560) 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.884+0000 7f7a8d929640 1 -- 172.21.15.46:0/3998489967 learned_addr learned my addr 172.21.15.46:0/3998489967 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.885+0000 7f7a8e12a640 1 -- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a881613c0 msgr2=0x7f7a880b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.885+0000 7f7a8e12a640 1 --2- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a881613c0 0x7f7a880b77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.885+0000 7f7a8e12a640 1 -- 172.21.15.46:0/3998489967 >> v1:172.21.15.88:6789/0 conn(0x7f7a880b5210 legacy=0x7f7a880d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.885+0000 7f7a8e12a640 1 -- 172.21.15.46:0/3998489967 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a881617d0 con 0x7f7a880b7d00 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.885+0000 7f7a8c927640 1 -- 172.21.15.46:0/3998489967 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635004909 0 0) 0x7f7a78001490 con 0x7f7a880b5210 2024-05-09T03:20:21.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8e12a640 1 --2- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b7d00 0x7f7a88163e60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7a800097a0 tx=0x7f7a8002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0d304bf786fce5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8c927640 1 -- 172.21.15.46:0/3998489967 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80037070 con 0x7f7a880b7d00 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8c927640 1 -- 172.21.15.46:0/3998489967 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a80032440 con 0x7f7a880b7d00 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8c927640 1 -- 172.21.15.46:0/3998489967 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80032810 con 0x7f7a880b7d00 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b7d00 msgr2=0x7f7a88163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b7d00 0x7f7a88163e60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7a800097a0 tx=0x7f7a8002a180 comp rx=0 tx=0).stop 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/3998489967 shutdown_connections 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b7d00 0x7f7a88163e60 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/3998489967 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a881613c0 0x7f7a880b77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.886+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/3998489967 >> 172.21.15.46:0/3998489967 conn(0x7f7a880d29a0 msgr2=0x7f7a880d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.887+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/3998489967 shutdown_connections 2024-05-09T03:20:21.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.887+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/3998489967 wait complete. 2024-05-09T03:20:21.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.888+0000 7f7a8fb8d640 1 Processor -- start 2024-05-09T03:20:21.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.888+0000 7f7a8fb8d640 1 -- start start 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8fb8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b5210 0x7f7a8814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8fb8d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a880b7d00 0x7f7a880b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8fb8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 0x7f7a880c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8fb8d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7a88160bc0 con 0x7f7a881613c0 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8fb8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7a880d5560 con 0x7f7a880b5210 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8fb8d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7a88160ea0 con 0x7f7a880b7d00 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 0x7f7a880c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 0x7f7a880c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46340/0 (socket says 172.21.15.46:46340) 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 -- 172.21.15.46:0/1163564216 learned_addr learned my addr 172.21.15.46:0/1163564216 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8d929640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b5210 0x7f7a8814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8d128640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a880b7d00 0x7f7a880b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 -- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a880b7d00 msgr2=0x7f7a880b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a880b7d00 0x7f7a880b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 -- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b5210 msgr2=0x7f7a8814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b5210 0x7f7a8814bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 -- 172.21.15.46:0/1163564216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a880b63f0 con 0x7f7a881613c0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.890+0000 7f7a8e12a640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 0x7f7a880c9850 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f7a800098c0 tx=0x7f7a800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.891+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80037040 con 0x7f7a881613c0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.891+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a80042070 con 0x7f7a881613c0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.891+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a800308a0 con 0x7f7a881613c0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.891+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a880b66d0 con 0x7f7a881613c0 2024-05-09T03:20:21.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.891+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a880b6c90 con 0x7f7a881613c0 2024-05-09T03:20:21.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.892+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a880b6160 con 0x7f7a881613c0 2024-05-09T03:20:21.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.893+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a80030a40 con 0x7f7a881613c0 2024-05-09T03:20:21.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.893+0000 7f7a867fc640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a74076300 0x7f7a74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:21.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.893+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f7a800b87c0 con 0x7f7a881613c0 2024-05-09T03:20:21.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.893+0000 7f7a8d929640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a74076300 0x7f7a74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:21.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.894+0000 7f7a8d929640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a74076300 0x7f7a74078810 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7a780012e0 tx=0x7f7a78001630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:21.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:21.898+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7a80080ec0 con 0x7f7a881613c0 2024-05-09T03:20:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.128+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7a8806a270 con 0x7f7a881613c0 2024-05-09T03:20:22.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.128+0000 7f7a867fc640 1 -- 172.21.15.46:0/1163564216 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v322) ==== 53+0+73 (secure 0 0 0) 0x7f7a800851c0 con 0x7f7a881613c0 2024-05-09T03:20:22.130 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:20:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.130+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a74076300 msgr2=0x7f7a74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.130+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a74076300 0x7f7a74078810 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7a780012e0 tx=0x7f7a78001630 comp rx=0 tx=0).stop 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 msgr2=0x7f7a880c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 0x7f7a880c9850 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f7a800098c0 tx=0x7f7a800046c0 comp rx=0 tx=0).stop 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 shutdown_connections 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7a74076300 0x7f7a74078810 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a881613c0 0x7f7a880c9850 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7a880b7d00 0x7f7a880b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 --2- 172.21.15.46:0/1163564216 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a880b5210 0x7f7a8814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 >> 172.21.15.46:0/1163564216 conn(0x7f7a880d29a0 msgr2=0x7f7a8815f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 shutdown_connections 2024-05-09T03:20:22.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22.131+0000 7f7a8fb8d640 1 -- 172.21.15.46:0/1163564216 wait complete. 2024-05-09T03:20:22.143 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:20:22.144 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:20:22.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.144 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:20:22.160 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:20:22.160 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:22.161 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41895sec preferred_lft 41895sec 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:20:22.162 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:20:22.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.163 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:20:22.225 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:20:22.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.226 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:20:22.283 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:20:22.283 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:20:22.283 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:20:22.283 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:20:22.283 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:20:22.284 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:20:22.284 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:20:22.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:20:22.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:20:22.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:20:22.284 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:20:22.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:20:22.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:20:22.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.413 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:20:22.430 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:20:22.430 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:20:22.430 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:20:22.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.430 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:20:22.496 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:20:22.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:20:22.496 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:22.553 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:20:22.554 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:20:22.554 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:22.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:20:22.676 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:20:22.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:20:22.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:22.735 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:22.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.736 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:20:22.790 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:20:22.790 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:20:22.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:20:22.859 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:20:22.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:20:22.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:20:22.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:22.920 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:20:22.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:20:22.927+0000 7f27fa53f480 -1 init, newargv = 0x7f27dc054070 newargc=15 2024-05-09T03:20:22.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[84622]: starting ceph client 2024-05-09T03:20:22.934 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[84622]: starting fuse 2024-05-09T03:20:22.945 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:20:22.946 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:20:22.946 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:20:22.947 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:20:22.948 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:20:23.069 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.84622.asok 2024-05-09T03:20:23.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:20:23.080 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.84622.asok 2024-05-09T03:20:23.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:23.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.84622.asok status 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:20:23.227 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "84622", 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10269, 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10269 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:23.228 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 759442829 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 759442829 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10269 192.168.0.1:0/759442829", 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/759442829", 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 322, 2024-05-09T03:20:23.229 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 120, 2024-05-09T03:20:23.230 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:20:23.230 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:20:23.230 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:20:23.230 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:20:23.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:23.236 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:20:23.236 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:23.252 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:20:23.252 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:23.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:23.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:23.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:23.318 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-05-09T03:20:23.361 INFO:teuthology.orchestra.run:waiting for 20 2024-05-09T03:20:23.379 INFO:teuthology.orchestra.run.smithi046.stdout:.: 2024-05-09T03:20:23.379 INFO:teuthology.orchestra.run.smithi046.stdout:subdir 2024-05-09T03:20:23.380 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:23.380 INFO:teuthology.orchestra.run.smithi046.stdout:./subdir: 2024-05-09T03:20:23.380 INFO:teuthology.orchestra.run.smithi046.stdout:sixmegs 2024-05-09T03:20:23.380 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on mds0_openfiles.0' did not caused DAMAGED state 2024-05-09T03:20:23.381 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 100.00000000' 2024-05-09T03:20:23.381 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:23.381 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:23.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:20:23.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.590+0000 7f696bca8640 1 Processor -- start 2024-05-09T03:20:23.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.590+0000 7f696bca8640 1 -- start start 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.591+0000 7f696bca8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b5140 0x7f69640b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696bca8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69640d3fc0 0x7f69640b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696bca8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69640b80b0 con 0x7f69640b7ca0 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696bca8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f69640d3480 con 0x7f69640d3fc0 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696bca8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f69640b5550 con 0x7f69640b5140 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696a245640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69640d3fc0 0x7f69640b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f6969243640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f69640b7ca0 0x7f69640d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34390/0 (socket says 172.21.15.46:34390) 2024-05-09T03:20:23.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f6969243640 1 -- 172.21.15.46:0/912194410 learned_addr learned my addr 172.21.15.46:0/912194410 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f6969a44640 1 --2- 172.21.15.46:0/912194410 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b5140 0x7f69640b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696a245640 1 -- 172.21.15.46:0/912194410 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b5140 msgr2=0x7f69640b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696a245640 1 --2- 172.21.15.46:0/912194410 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b5140 0x7f69640b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696a245640 1 -- 172.21.15.46:0/912194410 >> v1:172.21.15.46:6789/0 conn(0x7f69640b7ca0 legacy=0x7f69640d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.592+0000 7f696a245640 1 -- 172.21.15.46:0/912194410 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69640d44e0 con 0x7f69640d3fc0 2024-05-09T03:20:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.593+0000 7f696a245640 1 --2- 172.21.15.46:0/912194410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f69640b9380 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f695c009d90 tx=0x7f695c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11697505ff5efe4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.593+0000 7f6968a42640 1 -- 172.21.15.46:0/912194410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c007740 con 0x7f69640d3fc0 2024-05-09T03:20:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.593+0000 7f6968a42640 1 -- 172.21.15.46:0/912194410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c0078e0 con 0x7f69640d3fc0 2024-05-09T03:20:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.593+0000 7f6968a42640 1 -- 172.21.15.46:0/912194410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c007cb0 con 0x7f69640d3fc0 2024-05-09T03:20:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 -- 172.21.15.46:0/912194410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 msgr2=0x7f69640b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 --2- 172.21.15.46:0/912194410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f69640b9380 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f695c009d90 tx=0x7f695c02a6f0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 -- 172.21.15.46:0/912194410 shutdown_connections 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 --2- 172.21.15.46:0/912194410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f69640b9380 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 --2- 172.21.15.46:0/912194410 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b5140 0x7f69640b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 -- 172.21.15.46:0/912194410 >> 172.21.15.46:0/912194410 conn(0x7f696408d3f0 msgr2=0x7f696408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 -- 172.21.15.46:0/912194410 shutdown_connections 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.594+0000 7f696bca8640 1 -- 172.21.15.46:0/912194410 wait complete. 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.595+0000 7f696bca8640 1 Processor -- start 2024-05-09T03:20:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.596+0000 7f696bca8640 1 -- start start 2024-05-09T03:20:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696bca8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69640b5140 0x7f69641f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696bca8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b7ca0 0x7f69641e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696bca8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f6964159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696bca8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69640d4d80 con 0x7f69640d3fc0 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696bca8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f69640d4aa0 con 0x7f69640b5140 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696bca8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f69640d5580 con 0x7f69640b7ca0 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f6964159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f6964159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46360/0 (socket says 172.21.15.46:46360) 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 -- 172.21.15.46:0/1109186085 learned_addr learned my addr 172.21.15.46:0/1109186085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f6969243640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b7ca0 0x7f69641e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f6969a44640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69640b5140 0x7f69641f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 -- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b7ca0 msgr2=0x7f69641e9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b7ca0 0x7f69641e9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 -- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69640b5140 msgr2=0x7f69641f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69640b5140 0x7f69641f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 -- 172.21.15.46:0/1109186085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696415a8e0 con 0x7f69640d3fc0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f6969243640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b7ca0 0x7f69641e9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.597+0000 7f696a245640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f6964159080 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f695c009eb0 tx=0x7f695c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.598+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c040030 con 0x7f69640d3fc0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.598+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c031a90 con 0x7f69640d3fc0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.598+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c031e60 con 0x7f69640d3fc0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.598+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696415abc0 con 0x7f69640d3fc0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.598+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f696415b180 con 0x7f69640d3fc0 2024-05-09T03:20:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.599+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f696415a650 con 0x7f69640d3fc0 2024-05-09T03:20:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.600+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f695c03b410 con 0x7f69640d3fc0 2024-05-09T03:20:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.600+0000 7f695a7fc640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6948076300 0x7f6948078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.600+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f695c0b9660 con 0x7f69640d3fc0 2024-05-09T03:20:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.601+0000 7f6969a44640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6948076300 0x7f6948078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.601+0000 7f6969a44640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6948076300 0x7f6948078810 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6960006fd0 tx=0x7f6960008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.604+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f695c0823d0 con 0x7f69640d3fc0 2024-05-09T03:20:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.806+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f696415edb0 con 0x7f69640d3fc0 2024-05-09T03:20:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.807+0000 7f695a7fc640 1 -- 172.21.15.46:0/1109186085 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 72 entries v120) ==== 81+0+4374 (secure 0 0 0) 0x7f695c081cd0 con 0x7f69640d3fc0 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/134300288 2024-05-10T03:20:09.183188+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/134300288 2024-05-10T03:20:09.183188+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1774919282 2024-05-10T03:20:03.819873+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1774919282 2024-05-10T03:20:03.819873+0000 2024-05-09T03:20:23.809 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:23.810 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:23.811 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:23.813 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:23.814 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:23.814 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:23.814 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:23.814 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:23.814 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:listed 72 entries 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6948076300 msgr2=0x7f6948078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6948076300 0x7f6948078810 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6960006fd0 tx=0x7f6960008040 comp rx=0 tx=0).stop 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 msgr2=0x7f6964159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f6964159080 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f695c009eb0 tx=0x7f695c004a40 comp rx=0 tx=0).stop 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 shutdown_connections 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6948076300 0x7f6948078810 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69640d3fc0 0x7f6964159080 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f69640b7ca0 0x7f69641e9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 --2- 172.21.15.46:0/1109186085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69640b5140 0x7f69641f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 >> 172.21.15.46:0/1109186085 conn(0x7f696408d3f0 msgr2=0x7f6964161ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.809+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 shutdown_connections 2024-05-09T03:20:23.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.810+0000 7f696bca8640 1 -- 172.21.15.46:0/1109186085 wait complete. 2024-05-09T03:20:23.822 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:23.822 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:23.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:20:23.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.993+0000 7f625cec8640 1 Processor -- start 2024-05-09T03:20:23.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.993+0000 7f625cec8640 1 -- start start 2024-05-09T03:20:23.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.994+0000 7f625cec8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 0x7f62580b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625cec8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f62580d1f70 0x7f6258161070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625cec8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62581616b0 con 0x7f6258160c60 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625cec8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f62580b58b0 con 0x7f62580d1f70 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625cec8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f62580b4920 con 0x7f6258143410 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f6255d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6258160c60 0x7f62580d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34392/0 (socket says 172.21.15.46:34392) 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f6255d9b640 1 -- 172.21.15.46:0/1664743388 learned_addr learned my addr 172.21.15.46:0/1664743388 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f6256d9d640 1 --2- 172.21.15.46:0/1664743388 >> v2:172.21.15.88:3300/0 conn(0x7f62580d1f70 0x7f6258161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625659c640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 0x7f62580b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994871544 0 0) 0x7f62581616b0 con 0x7f6258160c60 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6244003520 con 0x7f6258160c60 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 212867723 0 0) 0x7f6244003520 con 0x7f6258160c60 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 msgr2=0x7f62580b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 0x7f62580b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:23.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 >> v2:172.21.15.88:3300/0 conn(0x7f62580d1f70 msgr2=0x7f6258161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 --2- 172.21.15.46:0/1664743388 >> v2:172.21.15.88:3300/0 conn(0x7f62580d1f70 0x7f6258161070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6258161990 con 0x7f6258160c60 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f62400016f0 con 0x7f6258160c60 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 >> v1:172.21.15.46:6789/0 conn(0x7f6258160c60 legacy=0x7f62580d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244003a00 0x7f6244003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6244004800 0x7f6244004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.995+0000 7f625559a640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 0x7f6244005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62580b58b0 con 0x7f6244004800 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f62580b4920 con 0x7f6244003a00 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f624400baf0 con 0x7f6258143410 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6255d9b640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6244004800 0x7f6244004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f625659c640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244003a00 0x7f6244003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6256d9d640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 0x7f6244005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6255d9b640 1 -- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 msgr2=0x7f6244005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6255d9b640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6258143410 0x7f6244005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6255d9b640 1 -- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244003a00 msgr2=0x7f6244003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6255d9b640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244003a00 0x7f6244003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.996+0000 7f6255d9b640 1 -- 172.21.15.46:0/1664743388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6244003520 con 0x7f6244004800 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f6255d9b640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6244004800 0x7f6244004c10 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f62580b5f00 tx=0x7f6244005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2437d8710c228a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240007d70 con 0x7f6244004800 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6240005ad0 con 0x7f6244004800 2024-05-09T03:20:24.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f625559a640 1 -- 172.21.15.46:0/1664743388 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240007070 con 0x7f6244004800 2024-05-09T03:20:24.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f625cec8640 1 -- 172.21.15.46:0/1664743388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6244004800 msgr2=0x7f6244004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f625cec8640 1 --2- 172.21.15.46:0/1664743388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6244004800 0x7f6244004c10 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f62580b5f00 tx=0x7f6244005830 comp rx=0 tx=0).stop 2024-05-09T03:20:24.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.997+0000 7f6256d9d640 1 -- 172.21.15.46:0/1664743388 reap_dead start 2024-05-09T03:20:24.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.998+0000 7f625cec8640 1 -- 172.21.15.46:0/1664743388 shutdown_connections 2024-05-09T03:20:24.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.998+0000 7f625cec8640 1 -- 172.21.15.46:0/1664743388 >> 172.21.15.46:0/1664743388 conn(0x7f62580b3620 msgr2=0x7f62580b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:24.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.998+0000 7f625cec8640 1 -- 172.21.15.46:0/1664743388 shutdown_connections 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:23.998+0000 7f625cec8640 1 -- 172.21.15.46:0/1664743388 wait complete. 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.000+0000 7f625cec8640 1 Processor -- start 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.000+0000 7f625cec8640 1 -- start start 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625cec8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244004800 0x7f625813df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625cec8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 0x7f625815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625cec8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62581f2620 0x7f62581404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625cec8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6258143820 con 0x7f62581f2210 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625cec8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f62580b4ee0 con 0x7f6244004800 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625cec8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f62580d1210 con 0x7f62581f2620 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 0x7f625815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 0x7f625815f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46392/0 (socket says 172.21.15.46:46392) 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 -- 172.21.15.46:0/3758381294 learned_addr learned my addr 172.21.15.46:0/3758381294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f625659c640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244004800 0x7f625813df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6256d9d640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62581f2620 0x7f62581404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 -- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62581f2620 msgr2=0x7f62581404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62581f2620 0x7f62581404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 -- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244004800 msgr2=0x7f625813df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244004800 0x7f625813df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.001+0000 7f6255d9b640 1 -- 172.21.15.46:0/3758381294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625813e440 con 0x7f62581f2210 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.002+0000 7f6255d9b640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 0x7f625815f920 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f62580b6df0 tx=0x7f6258161b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.002+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240037040 con 0x7f62581f2210 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.002+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6240042070 con 0x7f62581f2210 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.002+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625815e880 con 0x7f62581f2210 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.002+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62400016f0 con 0x7f62581f2210 2024-05-09T03:20:24.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.002+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f625815ee40 con 0x7f62581f2210 2024-05-09T03:20:24.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.003+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f625806a270 con 0x7f62581f2210 2024-05-09T03:20:24.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.003+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6240007070 con 0x7f62581f2210 2024-05-09T03:20:24.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.004+0000 7f624effd640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f62380761e0 0x7f62380786f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.004+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7f62400b8060 con 0x7f62581f2210 2024-05-09T03:20:24.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.004+0000 7f625659c640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f62380761e0 0x7f62380786f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.005+0000 7f625659c640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f62380761e0 0x7f62380786f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6250014430 tx=0x7f625000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.009+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6240005aa0 con 0x7f62581f2210 2024-05-09T03:20:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.216+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f62580d0620 con 0x7f62581f2210 2024-05-09T03:20:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.217+0000 7f624effd640 1 -- 172.21.15.46:0/3758381294 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 72 entries v120) ==== 81+0+4374 (secure 0 0 0) 0x7f6240080c90 con 0x7f62581f2210 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/134300288 2024-05-10T03:20:09.183188+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/134300288 2024-05-10T03:20:09.183188+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1774919282 2024-05-10T03:20:03.819873+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3872337434 2024-05-10T03:19:35.738958+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1774919282 2024-05-10T03:20:03.819873+0000 2024-05-09T03:20:24.219 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/925919281 2024-05-10T03:19:16.273718+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1270051447 2024-05-10T03:17:29.198075+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3968811793 2024-05-10T03:17:16.129644+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1350199591 2024-05-10T03:19:30.622459+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:24.220 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2388442105 2024-05-10T03:17:44.993790+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2209678569 2024-05-10T03:19:52.097765+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2032086476 2024-05-10T03:17:22.988689+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3044304069 2024-05-10T03:17:09.735805+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2566829830 2024-05-10T03:16:58.040696+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1281653036 2024-05-10T03:16:24.681981+0000 2024-05-09T03:20:24.221 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1234333269 2024-05-10T03:17:58.071311+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/782052086 2024-05-10T03:16:40.263688+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3297564121 2024-05-10T03:16:36.237612+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/271467610 2024-05-10T03:18:02.111947+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/436596417 2024-05-10T03:18:06.147085+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1486786164 2024-05-10T03:17:42.979548+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2585438921 2024-05-10T03:18:23.220415+0000 2024-05-09T03:20:24.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3423945789 2024-05-10T03:17:12.099885+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2546691383 2024-05-10T03:18:27.254542+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/3450505404 2024-05-10T03:17:52.032707+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2347304023 2024-05-10T03:16:08.753704+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/202666403 2024-05-10T03:18:31.290277+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2789195040 2024-05-10T03:18:36.012259+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/935847027 2024-05-10T03:18:40.005228+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/905138228 2024-05-10T03:19:08.214271+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2490064583 2024-05-10T03:18:44.067190+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2827001018 2024-05-10T03:19:01.862132+0000 2024-05-09T03:20:24.223 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/2860475391 2024-05-10T03:16:44.302289+0000 2024-05-09T03:20:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/3106969477 2024-05-10T03:18:20.866657+0000 2024-05-09T03:20:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/2638610471 2024-05-10T03:18:48.102832+0000 2024-05-09T03:20:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1033379302 2024-05-10T03:19:04.179713+0000 2024-05-09T03:20:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6834/1887013209 2024-05-10T03:16:30.194822+0000 2024-05-09T03:20:24.224 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6832/120572030 2024-05-10T03:19:12.242288+0000 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:listed 72 entries 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f62380761e0 msgr2=0x7f62380786f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f62380761e0 0x7f62380786f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6250014430 tx=0x7f625000b040 comp rx=0 tx=0).stop 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 msgr2=0x7f625815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 0x7f625815f920 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f62580b6df0 tx=0x7f6258161b60 comp rx=0 tx=0).stop 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 shutdown_connections 2024-05-09T03:20:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f62581f2620 0x7f62581404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f62380761e0 0x7f62380786f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62581f2210 0x7f625815f920 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 --2- 172.21.15.46:0/3758381294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6244004800 0x7f625813df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 >> 172.21.15.46:0/3758381294 conn(0x7f62580b3620 msgr2=0x7f625815b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 shutdown_connections 2024-05-09T03:20:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.219+0000 7f625cec8640 1 -- 172.21.15.46:0/3758381294 wait complete. 2024-05-09T03:20:24.232 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:20:24.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:24.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:24.259 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:20:24.259 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:20:24.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:24.259 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:20:24.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:20:24.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.469+0000 7fd98e95e640 1 Processor -- start 2024-05-09T03:20:24.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.469+0000 7fd98e95e640 1 -- start start 2024-05-09T03:20:24.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd98e95e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd9880b7e70 0x7fd9880d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd98e95e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9880b5550 con 0x7fd9880b5140 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd98e95e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9880d44e0 con 0x7fd9880d3df0 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd98e95e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd9880d3480 con 0x7fd9880b7e70 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd987fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd9880b5140 0x7fd9880b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34408/0 (socket says 172.21.15.46:34408) 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd987fff640 1 -- 172.21.15.46:0/2547252562 learned_addr learned my addr 172.21.15.46:0/2547252562 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd9877fe640 1 --2- 172.21.15.46:0/2547252562 >> v2:172.21.15.88:3300/0 conn(0x7fd9880b7e70 0x7fd9880d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2912887661 0 0) 0x7fd9880b5550 con 0x7fd9880b5140 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.471+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd970003520 con 0x7fd9880b5140 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2747039985 0 0) 0x7fd970003520 con 0x7fd9880b5140 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 >> v2:172.21.15.88:3300/0 conn(0x7fd9880b7e70 msgr2=0x7fd9880d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 --2- 172.21.15.46:0/2547252562 >> v2:172.21.15.88:3300/0 conn(0x7fd9880b7e70 0x7fd9880d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 >> v1:172.21.15.88:6789/0 conn(0x7fd9880d3df0 legacy=0x7fd9880b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9880d47c0 con 0x7fd9880b5140 2024-05-09T03:20:24.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fd9780016f0 con 0x7fd9880b5140 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 >> v1:172.21.15.46:6789/0 conn(0x7fd9880b5140 legacy=0x7fd9880b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd970003a50 0x7fd970003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970004800 0x7fd970004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd970005250 0x7fd97000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd97000c1a0 con 0x7fd970003a50 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd97000c350 con 0x7fd970004800 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd97000c550 con 0x7fd970005250 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd987fff640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd970003a50 0x7fd970003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd9877fe640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970004800 0x7fd970004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd98cefb640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd970005250 0x7fd97000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd987fff640 1 -- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd970005250 msgr2=0x7fd97000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd987fff640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd970005250 0x7fd97000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd987fff640 1 -- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970004800 msgr2=0x7fd970004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd987fff640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970004800 0x7fd970004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd987fff640 1 -- 172.21.15.46:0/2547252562 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd970003520 con 0x7fd970003a50 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.472+0000 7fd98cefb640 1 -- 172.21.15.46:0/2547252562 reap_dead start 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.473+0000 7fd987fff640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd970003a50 0x7fd970003ec0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fd978001050 tx=0x7fd970005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb1546c1f7847c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.473+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978007d10 con 0x7fd970003a50 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.473+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd978005ad0 con 0x7fd970003a50 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.473+0000 7fd986ffd640 1 -- 172.21.15.46:0/2547252562 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978007070 con 0x7fd970003a50 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 -- 172.21.15.46:0/2547252562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd970003a50 msgr2=0x7fd970003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd970003a50 0x7fd970003ec0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7fd978001050 tx=0x7fd970005830 comp rx=0 tx=0).stop 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 -- 172.21.15.46:0/2547252562 shutdown_connections 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2547252562 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd970003a50 0x7fd970003ec0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 -- 172.21.15.46:0/2547252562 >> 172.21.15.46:0/2547252562 conn(0x7fd98808d3f0 msgr2=0x7fd98808d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 -- 172.21.15.46:0/2547252562 shutdown_connections 2024-05-09T03:20:24.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.474+0000 7fd98e95e640 1 -- 172.21.15.46:0/2547252562 wait complete. 2024-05-09T03:20:24.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.476+0000 7fd98e95e640 1 Processor -- start 2024-05-09T03:20:24.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.476+0000 7fd98e95e640 1 -- start start 2024-05-09T03:20:24.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98e95e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970003a50 0x7fd9881f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98e95e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd98815abd0 0x7fd98815afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98e95e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 0x7fd988158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98e95e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880d4200 con 0x7fd988158ae0 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98e95e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880d4d80 con 0x7fd970003a50 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98e95e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880d5580 con 0x7fd98815abd0 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 0x7fd988158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 0x7fd988158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46402/0 (socket says 172.21.15.46:46402) 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 -- 172.21.15.46:0/2815186156 learned_addr learned my addr 172.21.15.46:0/2815186156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd987fff640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970003a50 0x7fd9881f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd9877fe640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd98815abd0 0x7fd98815afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 -- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd98815abd0 msgr2=0x7fd98815afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd98815abd0 0x7fd98815afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 -- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970003a50 msgr2=0x7fd9881f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970003a50 0x7fd9881f6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd98cefb640 1 -- 172.21.15.46:0/2815186156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd988159430 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.477+0000 7fd987fff640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970003a50 0x7fd9881f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.478+0000 7fd98cefb640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 0x7fd988158ef0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd988160a20 tx=0x7fd98000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.478+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980006b50 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.478+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd980006d10 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.478+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9880bb100 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.478+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980003970 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.478+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98815bba0 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.479+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98806a270 con 0x7fd988158ae0 2024-05-09T03:20:24.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.480+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd98001a020 con 0x7fd988158ae0 2024-05-09T03:20:24.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.480+0000 7fd984ff9640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd974076300 0x7fd974078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.480+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8863+0+0 (secure 0 0 0) 0x7fd980097960 con 0x7fd988158ae0 2024-05-09T03:20:24.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.480+0000 7fd987fff640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd974076300 0x7fd974078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.481+0000 7fd987fff640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd974076300 0x7fd974078810 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd978032000 tx=0x7fd978009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.485+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd988159430 con 0x7fd988158ae0 2024-05-09T03:20:24.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.713+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd9881638a0 con 0x7fd988158ae0 2024-05-09T03:20:24.715 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:24.714+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:20:24.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.715+0000 7fd984ff9640 1 -- 172.21.15.46:0/2815186156 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v323) ==== 161+0+0 (secure 0 0 0) 0x7fd980060590 con 0x7fd988158ae0 2024-05-09T03:20:24.717 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:20:24.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.718+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd974076300 msgr2=0x7fd974078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.718+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd974076300 0x7fd974078810 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd978032000 tx=0x7fd978009030 comp rx=0 tx=0).stop 2024-05-09T03:20:24.720 INFO:tasks.ceph.mds.c.smithi046.stderr: -1837> 2024-05-09T03:20:15.013+0000 7fcd671bf640 -1 mds.0.openfiles _load_finish: corrupted header/values: End of buffer [buffer:2] 2024-05-09T03:20:24.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.718+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 msgr2=0x7fd988158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 0x7fd988158ef0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fd988160a20 tx=0x7fd98000eb90 comp rx=0 tx=0).stop 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 shutdown_connections 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd974076300 0x7fd974078810 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd988158ae0 0x7fd988158ef0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd98815abd0 0x7fd98815afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 --2- 172.21.15.46:0/2815186156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd970003a50 0x7fd9881f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 >> 172.21.15.46:0/2815186156 conn(0x7fd98808d3f0 msgr2=0x7fd9881582c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 shutdown_connections 2024-05-09T03:20:24.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.719+0000 7fd98e95e640 1 -- 172.21.15.46:0/2815186156 wait complete. 2024-05-09T03:20:24.722 INFO:tasks.ceph.mds.c.smithi046.stderr: -663> 2024-05-09T03:20:20.191+0000 7fcd6b1c7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:20:24.732 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:20:24.778 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.848+0000 7f830c6b1640 1 Processor -- start 2024-05-09T03:20:24.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.848+0000 7f830c6b1640 1 -- start start 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830c6b1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83040b5140 0x7f83040b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830c6b1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f83040d3fc0 0x7f83040b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830c6b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83040b80b0 con 0x7f83040b7ca0 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830c6b1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f83040d3480 con 0x7f83040b5140 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830c6b1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f83040b5550 con 0x7f83040d3fc0 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830a44d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83040b5140 0x7f83040b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f8309c4c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f83040b7ca0 0x7f83040d38b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34430/0 (socket says 172.21.15.46:34430) 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f8309c4c640 1 -- 172.21.15.46:0/2830188164 learned_addr learned my addr 172.21.15.46:0/2830188164 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830ac4e640 1 --2- 172.21.15.46:0/2830188164 >> v2:172.21.15.88:3300/0 conn(0x7f83040d3fc0 0x7f83040b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830a44d640 1 -- 172.21.15.46:0/2830188164 >> v2:172.21.15.88:3300/0 conn(0x7f83040d3fc0 msgr2=0x7f83040b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830a44d640 1 --2- 172.21.15.46:0/2830188164 >> v2:172.21.15.88:3300/0 conn(0x7f83040d3fc0 0x7f83040b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830a44d640 1 -- 172.21.15.46:0/2830188164 >> v1:172.21.15.46:6789/0 conn(0x7f83040b7ca0 legacy=0x7f83040d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830944b640 1 -- 172.21.15.46:0/2830188164 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839225191 0 0) 0x7f82f4001110 con 0x7f83040b7ca0 2024-05-09T03:20:24.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.850+0000 7f830a44d640 1 -- 172.21.15.46:0/2830188164 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83040d44e0 con 0x7f83040b5140 2024-05-09T03:20:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.851+0000 7f830a44d640 1 --2- 172.21.15.46:0/2830188164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b5140 0x7f83040b7760 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f830408f180 tx=0x7f830002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2a6cbb37e9cf82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.851+0000 7f830944b640 1 -- 172.21.15.46:0/2830188164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300007740 con 0x7f83040b5140 2024-05-09T03:20:24.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.851+0000 7f830944b640 1 -- 172.21.15.46:0/2830188164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83000078e0 con 0x7f83040b5140 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.851+0000 7f830944b640 1 -- 172.21.15.46:0/2830188164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300007cb0 con 0x7f83040b5140 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 -- 172.21.15.46:0/2830188164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b5140 msgr2=0x7f83040b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 --2- 172.21.15.46:0/2830188164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b5140 0x7f83040b7760 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f830408f180 tx=0x7f830002a710 comp rx=0 tx=0).stop 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 -- 172.21.15.46:0/2830188164 shutdown_connections 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 --2- 172.21.15.46:0/2830188164 >> v2:172.21.15.88:3300/0 conn(0x7f83040d3fc0 0x7f83040b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 --2- 172.21.15.46:0/2830188164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b5140 0x7f83040b7760 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 -- 172.21.15.46:0/2830188164 >> 172.21.15.46:0/2830188164 conn(0x7f830408d3f0 msgr2=0x7f830408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 -- 172.21.15.46:0/2830188164 shutdown_connections 2024-05-09T03:20:24.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.852+0000 7f830c6b1640 1 -- 172.21.15.46:0/2830188164 wait complete. 2024-05-09T03:20:24.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.854+0000 7f830c6b1640 1 Processor -- start 2024-05-09T03:20:24.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.854+0000 7f830c6b1640 1 -- start start 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830c6b1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040b5140 0x7f830415bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830c6b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 0x7f83041f9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830c6b1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040d3fc0 0x7f83041f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830c6b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83040d4d80 con 0x7f83040b7ca0 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830c6b1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f83040d4aa0 con 0x7f83040b5140 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830c6b1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f83040d5580 con 0x7f83040d3fc0 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f8309c4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 0x7f83041f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f8309c4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 0x7f83041f9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46426/0 (socket says 172.21.15.46:46426) 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f8309c4c640 1 -- 172.21.15.46:0/3096895589 learned_addr learned my addr 172.21.15.46:0/3096895589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830a44d640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040b5140 0x7f830415bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.855+0000 7f830ac4e640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040d3fc0 0x7f83041f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f8309c4c640 1 -- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040d3fc0 msgr2=0x7f83041f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f8309c4c640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040d3fc0 0x7f83041f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f8309c4c640 1 -- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040b5140 msgr2=0x7f830415bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f8309c4c640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040b5140 0x7f830415bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f8309c4c640 1 -- 172.21.15.46:0/3096895589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83041f98b0 con 0x7f83040b7ca0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f830ac4e640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040d3fc0 0x7f83041f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f8309c4c640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 0x7f83041f9370 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f82f400c410 tx=0x7f82f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40074e0 con 0x7f83040b7ca0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f40047e0 con 0x7f83040b7ca0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830415ad30 con 0x7f83040b7ca0 2024-05-09T03:20:24.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4004bd0 con 0x7f83040b7ca0 2024-05-09T03:20:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.856+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f830415a490 con 0x7f83040b7ca0 2024-05-09T03:20:24.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.857+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830406a270 con 0x7f83040b7ca0 2024-05-09T03:20:24.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.858+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f82f4011070 con 0x7f83040b7ca0 2024-05-09T03:20:24.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.858+0000 7f82faffd640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82e80762d0 0x7f82e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:24.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.858+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8949+0+0 (secure 0 0 0) 0x7f82f4097e00 con 0x7f83040b7ca0 2024-05-09T03:20:24.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.858+0000 7f830a44d640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82e80762d0 0x7f82e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:24.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.859+0000 7f830a44d640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82e80762d0 0x7f82e80787e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8300034000 tx=0x7f830002f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:24.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:24.863+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f83040b80b0 con 0x7f83040b7ca0 2024-05-09T03:20:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.130+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f83040b6290 con 0x7f83040b7ca0 2024-05-09T03:20:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.717+0000 7f82faffd640 1 -- 172.21.15.46:0/3096895589 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v324) ==== 107+0+0 (secure 0 0 0) 0x7f82f4060a60 con 0x7f83040b7ca0 2024-05-09T03:20:25.719 INFO:teuthology.orchestra.run.smithi046.stderr:nothing to do: rank is not damaged 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82e80762d0 msgr2=0x7f82e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82e80762d0 0x7f82e80787e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8300034000 tx=0x7f830002f000 comp rx=0 tx=0).stop 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 msgr2=0x7f83041f9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 0x7f83041f9370 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f82f400c410 tx=0x7f82f400c8d0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 shutdown_connections 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f83040d3fc0 0x7f83041f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f82e80762d0 0x7f82e80787e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83040b7ca0 0x7f83041f9370 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 --2- 172.21.15.46:0/3096895589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83040b5140 0x7f830415bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 >> 172.21.15.46:0/3096895589 conn(0x7f830408d3f0 msgr2=0x7f8304161030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 shutdown_connections 2024-05-09T03:20:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.720+0000 7f830c6b1640 1 -- 172.21.15.46:0/3096895589 wait complete. 2024-05-09T03:20:25.737 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:20:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.797+0000 7f8db1726f40 1 Processor -- start 2024-05-09T03:20:25.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.797+0000 7f8db1726f40 1 -- start start 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.798+0000 7f8db1726f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df46b47b0 0x562df47503d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.798+0000 7f8db1726f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df46c6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.798+0000 7f8db1726f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46c5d30 0x562df46c6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.798+0000 7f8db1726f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562df462bf50 con 0x562df46cb010 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.798+0000 7f8db1726f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562df462ff90 con 0x562df46b47b0 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.798+0000 7f8db1726f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562df46c94d0 con 0x562df46c5d30 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df46c6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df46c6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53498/0 (socket says 172.21.15.46:53498) 2024-05-09T03:20:25.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 -- 172.21.15.46:0/4182572514 learned_addr learned my addr 172.21.15.46:0/4182572514 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df46b47b0 0x562df47503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8daafa8640 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46c5d30 0x562df46c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 -- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46c5d30 msgr2=0x562df46c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46c5d30 0x562df46c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 -- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df46b47b0 msgr2=0x562df47503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df46b47b0 0x562df47503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 -- 172.21.15.46:0/4182572514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562df46b0650 con 0x562df46cb010 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8daafa8640 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46c5d30 0x562df46c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df46c6320 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f8da0001de0 tx=0x7f8da000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37e7709197ac8e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da8fa4640 1 -- 172.21.15.46:0/4182572514 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0012020 con 0x562df46cb010 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da8fa4640 1 -- 172.21.15.46:0/4182572514 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da000e690 con 0x562df46cb010 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.799+0000 7f8da8fa4640 1 -- 172.21.15.46:0/4182572514 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da000ea80 con 0x562df46cb010 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 -- 172.21.15.46:0/4182572514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 msgr2=0x562df46c6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df46c6320 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f8da0001de0 tx=0x7f8da000b6c0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 -- 172.21.15.46:0/4182572514 shutdown_connections 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46c5d30 0x562df46c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df46c6320 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 --2- 172.21.15.46:0/4182572514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df46b47b0 0x562df47503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 -- 172.21.15.46:0/4182572514 >> 172.21.15.46:0/4182572514 conn(0x562df45a7ac0 msgr2=0x562df46cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 -- 172.21.15.46:0/4182572514 shutdown_connections 2024-05-09T03:20:25.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.800+0000 7f8db1726f40 1 -- 172.21.15.46:0/4182572514 wait complete. 2024-05-09T03:20:25.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.801+0000 7f8db1726f40 1 Processor -- start 2024-05-09T03:20:25.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.801+0000 7f8db1726f40 1 -- start start 2024-05-09T03:20:25.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8db1726f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46b47b0 0x562df474be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8db1726f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df47ea110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8db1726f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df47ea650 0x562df47eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8db1726f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x562df46c8750 con 0x562df46cb010 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8db1726f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562df474cde0 con 0x562df47ea650 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8db1726f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x562df474c910 con 0x562df46b47b0 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.802+0000 7f8da9fa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df47ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df47ea110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53512/0 (socket says 172.21.15.46:53512) 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 learned_addr learned my addr 172.21.15.46:0/1671266609 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46b47b0 0x562df474be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8daafa8640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df47ea650 0x562df47eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46b47b0 msgr2=0x562df474be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46b47b0 0x562df474be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df47ea650 msgr2=0x562df47eae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df47ea650 0x562df47eae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562df46c6a30 con 0x562df46cb010 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x562df46b47b0 0x562df474be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8daafa8640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562df47ea650 0x562df47eae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df47ea110 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f8da000b700 tx=0x7f8da00040e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8d9b7fe640 1 -- 172.21.15.46:0/1671266609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da001a510 con 0x562df46cb010 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8d9b7fe640 1 -- 172.21.15.46:0/1671266609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da001a6b0 con 0x562df46cb010 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562df46c6d10 con 0x562df46cb010 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8d9b7fe640 1 -- 172.21.15.46:0/1671266609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da001aaa0 con 0x562df46cb010 2024-05-09T03:20:25.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.803+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562df4759000 con 0x562df46cb010 2024-05-09T03:20:25.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.805+0000 7f8d9b7fe640 1 -- 172.21.15.46:0/1671266609 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8da0015020 con 0x562df46cb010 2024-05-09T03:20:25.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.806+0000 7f8d9b7fe640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d8c074ce0 0x7f8d8c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.806+0000 7f8d9b7fe640 1 -- 172.21.15.46:0/1671266609 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8949+0+0 (secure 0 0 0) 0x7f8da0098ec0 con 0x562df46cb010 2024-05-09T03:20:25.809 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.806+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x562df47eb820 0x562df47f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.806+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47f6050 con 0x562df47eb820 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.806+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d8c074ce0 0x7f8d8c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.806+0000 7f8daafa8640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x562df47eb820 0x562df47f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.807+0000 7f8daafa8640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x562df47eb820 0x562df47f5b10 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.807+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d8c074ce0 0x7f8d8c0771f0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c057000 tx=0x7f8d9c052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.809+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v121'1128 uv1102 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.809+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47fe860 con 0x562df47eb820 2024-05-09T03:20:25.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.811+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v121'1129 uv1129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.811+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47fec00 con 0x562df47eb820 2024-05-09T03:20:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.812+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v121'1130 uv1130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.814 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:20:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.812+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47fffb0 con 0x562df47eb820 2024-05-09T03:20:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.814+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v121'1131 uv1131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.814+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4800350 con 0x562df47eb820 2024-05-09T03:20:25.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.815+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v121'1132 uv1132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.815+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47fefa0 con 0x562df47eb820 2024-05-09T03:20:25.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.817+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v121'1133 uv1133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.817+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff340 con 0x562df47eb820 2024-05-09T03:20:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.818+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v121'1134 uv1108 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.818+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47ff730 con 0x562df47eb820 2024-05-09T03:20:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.820+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v121'1135 uv1135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.820+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff730 con 0x562df47eb820 2024-05-09T03:20:25.823 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:20:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.821+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v121'1136 uv1136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.821+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.823+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v121'1137 uv1137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.823+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.825+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v121'1138 uv1138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.825+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.826+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v121'1139 uv1139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.826+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.828+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v121'1140 uv1115 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.828+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.830+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v121'1141 uv1141 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.830+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.833 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:20:25.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.831+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v121'1142 uv1142 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.831+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.833+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v121'1143 uv1143 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.833+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.834+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v121'1144 uv1144 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.834+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.836+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v121'1145 uv1145 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.836+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.837+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v121'1146 uv1146 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.837+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.839+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v121'1147 uv1121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.839+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.841+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v121'1148 uv1148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.841+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.842+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v121'1149 uv1149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.844 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:20:25.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.842+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.844+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v121'1150 uv1150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.844+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.846+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v121'1151 uv1151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.846+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.847+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v121'1152 uv1152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.848+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.850+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v121'1153 uv1127 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.850+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.852+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v121'1154 uv1154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.852+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.855 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:20:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.853+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v121'1155 uv1155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.853+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.855+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v121'1156 uv1156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.855+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.857+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v121'1157 uv1157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.857+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df47eb820 2024-05-09T03:20:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.858+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v121'1158 uv1158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.858+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562df48006f0 0x562df4802b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.859+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47ff6e0 con 0x562df48006f0 2024-05-09T03:20:25.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.859+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562df48006f0 0x562df4802b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.859+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562df48006f0 0x562df4802b10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.861+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v121'815 uv795 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.861+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df4803050 con 0x562df48006f0 2024-05-09T03:20:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.863+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v121'816 uv816 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.863+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48033f0 con 0x562df48006f0 2024-05-09T03:20:25.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.864+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v121'817 uv817 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.866 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:20:25.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.864+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4803790 con 0x562df48006f0 2024-05-09T03:20:25.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.865+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v121'818 uv818 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.865+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4803b30 con 0x562df48006f0 2024-05-09T03:20:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.867+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v121'819 uv819 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.867+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4803ed0 con 0x562df48006f0 2024-05-09T03:20:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.868+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v121'820 uv814 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.868+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df4804270 con 0x562df48006f0 2024-05-09T03:20:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.869+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v121'821 uv821 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.869+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4804610 con 0x562df48006f0 2024-05-09T03:20:25.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.871+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v121'822 uv822 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.872 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:20:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.871+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.872+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v121'823 uv823 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.872+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.873+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v121'824 uv824 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.873+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.874+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v121'825 uv813 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.874+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.876+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v121'826 uv826 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.876+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.877+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v121'827 uv827 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.879 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:20:25.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.877+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.878+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v121'828 uv828 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.879+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.880+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v121'829 uv829 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.880+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.881+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v121'830 uv811 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.881+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.883+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v121'831 uv831 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.883+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.884+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v121'832 uv832 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.885 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:20:25.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.884+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.885+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v121'833 uv833 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.885+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.886+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v121'834 uv834 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.886+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df47eb450 con 0x562df48006f0 2024-05-09T03:20:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.887+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v121'835 uv835 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df48006f0 2024-05-09T03:20:25.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.887+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4804ac0 con 0x562df47eb820 2024-05-09T03:20:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.889+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v121'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.889+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df4804ac0 con 0x562df47eb820 2024-05-09T03:20:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.890+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v121'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.890+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4804ac0 con 0x562df47eb820 2024-05-09T03:20:25.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.891+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v121'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.892 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:20:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.891+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4805ad0 con 0x562df47eb820 2024-05-09T03:20:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.892+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v121'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.892+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4805ad0 con 0x562df47eb820 2024-05-09T03:20:25.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.893+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v121'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.893+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4804ac0 con 0x562df47eb820 2024-05-09T03:20:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.894+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v121'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.894+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562df4804fd0 0x562df48073f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.894+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4807990 con 0x562df4804fd0 2024-05-09T03:20:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.894+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562df4804fd0 0x562df48073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.895+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562df4804fd0 0x562df48073f0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.896+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v121'168 uv167 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.896+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df4807e70 con 0x562df4804fd0 2024-05-09T03:20:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.897+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v121'169 uv169 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.897+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4808370 con 0x562df4804fd0 2024-05-09T03:20:25.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.898+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v121'170 uv170 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.900 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:20:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.898+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48046f0 con 0x562df4804fd0 2024-05-09T03:20:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.899+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v121'171 uv171 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.899+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4808ca0 con 0x562df47eb820 2024-05-09T03:20:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.900+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v121'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.900+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df4808ca0 con 0x562df47eb820 2024-05-09T03:20:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.902+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v121'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.902+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4808ca0 con 0x562df47eb820 2024-05-09T03:20:25.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.903+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v121'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.905 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:20:25.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.903+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4808ca0 con 0x562df47eb820 2024-05-09T03:20:25.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.904+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v121'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.904+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4808ca0 con 0x562df47eb820 2024-05-09T03:20:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.905+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v121'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.905+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4808ce0 con 0x562df47eb820 2024-05-09T03:20:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.907+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v121'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.907+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562df48095e0 0x562df48099f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.907+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.907+0000 7f8daafa8640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562df48095e0 0x562df48099f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.907+0000 7f8daafa8640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562df48095e0 0x562df48099f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.909+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v121'404 uv398 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.909+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.910+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v121'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.910+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.911+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v121'406 uv406 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.913 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:20:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.911+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480d1c0 con 0x562df48095e0 2024-05-09T03:20:25.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.912+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v121'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.912+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480d1c0 con 0x562df48095e0 2024-05-09T03:20:25.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.913+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v121'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.913+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.914+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v121'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.914+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.915+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v121'410 uv403 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.915+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.916+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v121'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.916+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.917+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v121'412 uv412 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.919 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:20:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.918+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.919+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v121'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.919+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480c1b0 con 0x562df48095e0 2024-05-09T03:20:25.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.920+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v121'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.920+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562df480cab0 0x562df480eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.920+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480f410 con 0x562df480cab0 2024-05-09T03:20:25.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.920+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562df480cab0 0x562df480eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.921+0000 7f8daa7a7640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562df480cab0 0x562df480eed0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.922+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v121'183 uv182 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df480cab0 2024-05-09T03:20:25.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.922+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df480cab0 2024-05-09T03:20:25.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.923+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v121'184 uv184 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df480cab0 2024-05-09T03:20:25.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.923+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df480cab0 2024-05-09T03:20:25.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.924+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v121'185 uv185 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df480cab0 2024-05-09T03:20:25.926 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:20:25.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.924+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df480cab0 2024-05-09T03:20:25.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.925+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v121'186 uv186 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df480cab0 2024-05-09T03:20:25.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.925+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df480cab0 2024-05-09T03:20:25.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.926+0000 7f8daa7a7640 1 -- 172.21.15.46:0/1671266609 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v121'187 uv187 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d9c05d040 con 0x562df480cab0 2024-05-09T03:20:25.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.926+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df47eb820 2024-05-09T03:20:25.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.927+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v121'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.927+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df47eb820 2024-05-09T03:20:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.928+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v121'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.928+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480f7e0 con 0x562df47eb820 2024-05-09T03:20:25.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.930+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v121'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.931 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:20:25.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.930+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810c30 con 0x562df47eb820 2024-05-09T03:20:25.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.931+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v121'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.931+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810be0 con 0x562df47eb820 2024-05-09T03:20:25.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.932+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v121'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.932+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480fbd0 con 0x562df47eb820 2024-05-09T03:20:25.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.933+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v121'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df47eb820 2024-05-09T03:20:25.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.933+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df480fbd0 con 0x562df4804fd0 2024-05-09T03:20:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.934+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v121'365 uv359 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.934+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df480ff70 con 0x562df4804fd0 2024-05-09T03:20:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.935+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v121'366 uv366 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.935+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.936+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v121'367 uv367 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.938 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:20:25.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.936+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.937+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v121'368 uv368 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.937+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.938+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v121'369 uv369 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.938+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.939+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v121'370 uv364 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.939+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.940+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v121'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.940+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.941+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v121'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.943 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:20:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.941+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4810310 con 0x562df4804fd0 2024-05-09T03:20:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.942+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v121'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.943+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4813120 con 0x562df4804fd0 2024-05-09T03:20:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.944+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v121'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.944+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562df4810cd0 0x562df4813620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.944+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4810cd0 2024-05-09T03:20:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.944+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562df4810cd0 0x562df4813620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:25.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.944+0000 7f8da9fa6640 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562df4810cd0 0x562df4813620 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:25.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.945+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v121'162 uv160 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4810cd0 2024-05-09T03:20:25.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.945+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4810cd0 2024-05-09T03:20:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.947+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v121'163 uv163 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4810cd0 2024-05-09T03:20:25.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.947+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4810cd0 2024-05-09T03:20:25.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.948+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v121'164 uv164 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4810cd0 2024-05-09T03:20:25.949 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:20:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.948+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4810cd0 2024-05-09T03:20:25.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.949+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v121'165 uv165 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4810cd0 2024-05-09T03:20:25.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.949+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4804fd0 2024-05-09T03:20:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.951+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v121'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.951+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4804fd0 2024-05-09T03:20:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.952+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v121'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.952+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4804fd0 2024-05-09T03:20:25.954 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:20:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.953+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v121'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.953+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4804fd0 2024-05-09T03:20:25.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.954+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v121'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.954+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4804fd0 2024-05-09T03:20:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.955+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v121'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.955+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df4804fd0 2024-05-09T03:20:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.956+0000 7f8da9fa6640 1 -- 172.21.15.46:0/1671266609 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v121'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da002a020 con 0x562df4804fd0 2024-05-09T03:20:25.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.956+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.958+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v121'186 uv185 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.958+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.959+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v121'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.959+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.962 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:20:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.960+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v121'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.960+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.963+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v121'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.963+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.963+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v121'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.963+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.964+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v121'455 uv446 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.964+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.966+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v121'456 uv456 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.966+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.967+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v121'457 uv457 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.968 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:20:25.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.967+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.968+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v121'458 uv458 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.968+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.969+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v121'459 uv459 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.969+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.971+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v121'460 uv453 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.971+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.972+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v121'461 uv461 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.972+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.973+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v121'462 uv462 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.974 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:20:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.973+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.973+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v121'463 uv463 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.973+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.975+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v121'464 uv464 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.975+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df48130f0 con 0x562df48095e0 2024-05-09T03:20:25.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.976+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v121'465 uv465 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.976+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x562df4813bb0 con 0x562df48095e0 2024-05-09T03:20:25.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v121'466 uv466 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8da4003710 con 0x562df48095e0 2024-05-09T03:20:25.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x562df47eb820 msgr2=0x562df47f5b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x562df47eb820 0x562df47f5b10 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562df4804fd0 msgr2=0x562df48073f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x562df4804fd0 0x562df48073f0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562df4810cd0 msgr2=0x562df4813620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x562df4810cd0 0x562df4813620 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562df48006f0 msgr2=0x562df4802b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 reap_dead start 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x562df48006f0 0x562df4802b10 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562df48095e0 msgr2=0x562df48099f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x562df48095e0 0x562df48099f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562df480cab0 msgr2=0x562df480eed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x562df480cab0 0x562df480eed0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d8c074ce0 msgr2=0x7f8d8c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8d8c074ce0 0x7f8d8c0771f0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c057000 tx=0x7f8d9c052000 comp rx=0 tx=0).stop 2024-05-09T03:20:25.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 msgr2=0x562df47ea110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:25.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 --2- 172.21.15.46:0/1671266609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562df46cb010 0x562df47ea110 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f8da000b700 tx=0x7f8da00040e0 comp rx=0 tx=0).stop 2024-05-09T03:20:25.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8daafa8640 1 -- 172.21.15.46:0/1671266609 reap_dead start 2024-05-09T03:20:25.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 shutdown_connections 2024-05-09T03:20:25.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 >> 172.21.15.46:0/1671266609 conn(0x562df45a7ac0 msgr2=0x562df46cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:25.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 shutdown_connections 2024-05-09T03:20:25.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:25.977+0000 7f8db1726f40 1 -- 172.21.15.46:0/1671266609 wait complete. 2024-05-09T03:20:25.982 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 100.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:20:26.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.036+0000 7f581e35df40 1 Processor -- start 2024-05-09T03:20:26.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.036+0000 7f581e35df40 1 -- start start 2024-05-09T03:20:26.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.037+0000 7f581e35df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 0x555df793e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.037+0000 7f581e35df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df79257b0 0x555df7922fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.037+0000 7f581e35df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df7923510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.037+0000 7f581e35df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555df793a4d0 con 0x555df78f37e0 2024-05-09T03:20:26.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.037+0000 7f581e35df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555df789cf50 con 0x555df793c010 2024-05-09T03:20:26.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.037+0000 7f581e35df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555df793f8a0 con 0x555df79257b0 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 0x555df793e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 0x555df793e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53530/0 (socket says 172.21.15.46:53530) 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 -- 172.21.15.46:0/1152686425 learned_addr learned my addr 172.21.15.46:0/1152686425 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5818626640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df7923510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817624640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df79257b0 0x555df7922fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 -- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df79257b0 msgr2=0x555df7922fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df79257b0 0x555df7922fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 -- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 msgr2=0x555df7923510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df7923510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 -- 172.21.15.46:0/1152686425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df793e6a0 con 0x555df78f37e0 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817624640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df79257b0 0x555df7922fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5818626640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df7923510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:26.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5817e25640 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 0x555df793e160 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f5808055040 tx=0x7f5808051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5bd949b0d991 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5816622640 1 -- 172.21.15.46:0/1152686425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808056040 con 0x555df78f37e0 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5816622640 1 -- 172.21.15.46:0/1152686425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58080528c0 con 0x555df78f37e0 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.038+0000 7f5816622640 1 -- 172.21.15.46:0/1152686425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808052cb0 con 0x555df78f37e0 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 -- 172.21.15.46:0/1152686425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 msgr2=0x555df793e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 0x555df793e160 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f5808055040 tx=0x7f5808051000 comp rx=0 tx=0).stop 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 -- 172.21.15.46:0/1152686425 shutdown_connections 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df7923510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df79257b0 0x555df7922fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 --2- 172.21.15.46:0/1152686425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df78f37e0 0x555df793e160 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 -- 172.21.15.46:0/1152686425 >> 172.21.15.46:0/1152686425 conn(0x555df7818ac0 msgr2=0x555df793da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 -- 172.21.15.46:0/1152686425 shutdown_connections 2024-05-09T03:20:26.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.039+0000 7f581e35df40 1 -- 172.21.15.46:0/1152686425 wait complete. 2024-05-09T03:20:26.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.040+0000 7f581e35df40 1 Processor -- start 2024-05-09T03:20:26.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.040+0000 7f581e35df40 1 -- start start 2024-05-09T03:20:26.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f581e35df40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df78f37e0 0x555df79b9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f581e35df40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df79257b0 0x555df79c03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f581e35df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df79c0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f581e35df40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555df7937160 con 0x555df79257b0 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f581e35df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555df7936cd0 con 0x555df793c010 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f581e35df40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555df79c8230 con 0x555df78f37e0 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f5817624640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df79257b0 0x555df79c03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f5817e25640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df78f37e0 0x555df79b9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f5817624640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df79257b0 0x555df79c03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53538/0 (socket says 172.21.15.46:53538) 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f5817624640 1 -- 172.21.15.46:0/467906989 learned_addr learned my addr 172.21.15.46:0/467906989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.041+0000 7f5818626640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df79c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5817624640 1 -- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df78f37e0 msgr2=0x555df79b9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5817624640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555df78f37e0 0x555df79b9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5817624640 1 -- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 msgr2=0x555df79c0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5817624640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555df793c010 0x555df79c0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5817624640 1 -- 172.21.15.46:0/467906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555df7935610 con 0x555df79257b0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5817624640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df79257b0 0x555df79c03c0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f5800009f10 tx=0x7f5800008400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5815620640 1 -- 172.21.15.46:0/467906989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5800008de0 con 0x555df79257b0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5815620640 1 -- 172.21.15.46:0/467906989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5800015460 con 0x555df79257b0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f5815620640 1 -- 172.21.15.46:0/467906989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5800015810 con 0x555df79257b0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555df7938aa0 con 0x555df79257b0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.042+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555df7923ce0 con 0x555df79257b0 2024-05-09T03:20:26.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.044+0000 7f5815620640 1 -- 172.21.15.46:0/467906989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f580001c020 con 0x555df79257b0 2024-05-09T03:20:26.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.044+0000 7f5815620640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57f8074cb0 0x7f57f80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f5815620640 1 -- 172.21.15.46:0/467906989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8949+0+0 (secure 0 0 0) 0x7f5800096c50 con 0x555df79257b0 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f5817e25640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57f8074cb0 0x7f57f80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f581e35df40 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555df79c2d40 0x555df7a6fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x555df79be1a0 con 0x555df79c2d40 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f5818626640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555df79c2d40 0x555df7a6fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f5817e25640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57f8074cb0 0x7f57f80771c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5808066000 tx=0x7f5808052000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.045+0000 7f5818626640 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555df79c2d40 0x555df7a6fa10 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f5818626640 1 -- 172.21.15.46:0/467906989 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 100.00000000 [omap-set-header] v121'375 uv375 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5810010610 con 0x555df79c2d40 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555df79c2d40 msgr2=0x555df7a6fa10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x555df79c2d40 0x555df7a6fa10 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57f8074cb0 msgr2=0x7f57f80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f57f8074cb0 0x7f57f80771c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5808066000 tx=0x7f5808052000 comp rx=0 tx=0).stop 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df79257b0 msgr2=0x555df79c03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 --2- 172.21.15.46:0/467906989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555df79257b0 0x555df79c03c0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f5800009f10 tx=0x7f5800008400 comp rx=0 tx=0).stop 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f5818626640 1 -- 172.21.15.46:0/467906989 reap_dead start 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 shutdown_connections 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 >> 172.21.15.46:0/467906989 conn(0x555df7818ac0 msgr2=0x555df79c4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 shutdown_connections 2024-05-09T03:20:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.047+0000 7f581e35df40 1 -- 172.21.15.46:0/467906989 wait complete. 2024-05-09T03:20:26.052 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:20:26.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.168+0000 7f0abf861640 1 Processor -- start 2024-05-09T03:20:26.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.168+0000 7f0abf861640 1 -- start start 2024-05-09T03:20:26.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.169+0000 7f0abf861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 0x7f0ab8144570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.169+0000 7f0abf861640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0ab806c680 0x7f0ab806ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.169+0000 7f0abf861640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ab815fc80 con 0x7f0ab815e6b0 2024-05-09T03:20:26.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.169+0000 7f0abf861640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0ab80728f0 con 0x7f0ab806c680 2024-05-09T03:20:26.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.169+0000 7f0abf861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0ab806cfd0 con 0x7f0ab815f870 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abcdfc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 0x7f0ab8144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abddfe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0ab806c680 0x7f0ab806ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abd5fd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0ab815e6b0 0x7f0ab80721e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:45258/0 (socket says 172.21.15.46:45258) 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abcdfc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 0x7f0ab8144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:58512/0 (socket says 172.21.15.46:58512) 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abd5fd640 1 -- 172.21.15.46:0/3396540957 learned_addr learned my addr 172.21.15.46:0/3396540957 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abddfe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0ab806c680 0x7f0ab806ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:59730/0 (socket says 172.21.15.46:59730) 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abcdfc640 1 -- 172.21.15.46:0/3396540957 >> v2:172.21.15.88:3300/0 conn(0x7f0ab806c680 msgr2=0x7f0ab806ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:26.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abcdfc640 1 --2- 172.21.15.46:0/3396540957 >> v2:172.21.15.88:3300/0 conn(0x7f0ab806c680 0x7f0ab806ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abcdfc640 1 -- 172.21.15.46:0/3396540957 >> v1:172.21.15.88:6789/0 conn(0x7f0ab815e6b0 legacy=0x7f0ab80721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3396540957 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 409578233 0 0) 0x7f0aa8001490 con 0x7f0ab815e6b0 2024-05-09T03:20:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.170+0000 7f0abcdfc640 1 -- 172.21.15.46:0/3396540957 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab815ec20 con 0x7f0ab815f870 2024-05-09T03:20:26.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.171+0000 7f0abcdfc640 1 --2- 172.21.15.46:0/3396540957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 0x7f0ab8144570 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097a0 tx=0x7f0aac02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b37b0a10d7ba6135 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.171+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3396540957 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac037070 con 0x7f0ab815f870 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.171+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3396540957 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aac032440 con 0x7f0ab815f870 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.171+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3396540957 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac032810 con 0x7f0ab815f870 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 -- 172.21.15.46:0/3396540957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 msgr2=0x7f0ab8144570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 --2- 172.21.15.46:0/3396540957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 0x7f0ab8144570 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0aac0097a0 tx=0x7f0aac02a180 comp rx=0 tx=0).stop 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 -- 172.21.15.46:0/3396540957 shutdown_connections 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 --2- 172.21.15.46:0/3396540957 >> v2:172.21.15.88:3300/0 conn(0x7f0ab806c680 0x7f0ab806ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 --2- 172.21.15.46:0/3396540957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815f870 0x7f0ab8144570 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 -- 172.21.15.46:0/3396540957 >> 172.21.15.46:0/3396540957 conn(0x7f0ab806de50 msgr2=0x7f0ab80b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:26.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 -- 172.21.15.46:0/3396540957 shutdown_connections 2024-05-09T03:20:26.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.172+0000 7f0abf861640 1 -- 172.21.15.46:0/3396540957 wait complete. 2024-05-09T03:20:26.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.174+0000 7f0abf861640 1 Processor -- start 2024-05-09T03:20:26.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.174+0000 7f0abf861640 1 -- start start 2024-05-09T03:20:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abf861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 0x7f0ab80cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abf861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815e6b0 0x7f0ab815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abf861640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ab815f870 0x7f0ab8141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abf861640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab815f4c0 con 0x7f0ab806c680 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abf861640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab815f1e0 con 0x7f0ab815f870 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abf861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f0ab80d25b0 con 0x7f0ab815e6b0 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 0x7f0ab80cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 0x7f0ab80cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53548/0 (socket says 172.21.15.46:53548) 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 -- 172.21.15.46:0/252906836 learned_addr learned my addr 172.21.15.46:0/252906836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abddfe640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ab815f870 0x7f0ab8141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abcdfc640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815e6b0 0x7f0ab815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 -- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815e6b0 msgr2=0x7f0ab815bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815e6b0 0x7f0ab815bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 -- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ab815f870 msgr2=0x7f0ab8141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ab815f870 0x7f0ab8141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.175+0000 7f0abd5fd640 1 -- 172.21.15.46:0/252906836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab80cbd20 con 0x7f0ab806c680 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.176+0000 7f0abd5fd640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 0x7f0ab80cf430 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800caf0 tx=0x7f0aa800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.176+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8017070 con 0x7f0ab806c680 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.176+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0aa800aca0 con 0x7f0ab806c680 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.176+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab80b60c0 con 0x7f0ab806c680 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.176+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8012620 con 0x7f0ab806c680 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.176+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab80b6680 con 0x7f0ab806c680 2024-05-09T03:20:26.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.177+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ab815c080 con 0x7f0ab806c680 2024-05-09T03:20:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.178+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0aa801a070 con 0x7f0ab806c680 2024-05-09T03:20:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.178+0000 7f0ab5ffb640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0aa4076300 0x7f0aa4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.178+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8949+0+0 (secure 0 0 0) 0x7f0aa8097f90 con 0x7f0ab806c680 2024-05-09T03:20:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.178+0000 7f0abcdfc640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0aa4076300 0x7f0aa4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.179+0000 7f0abcdfc640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0aa4076300 0x7f0aa4078810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0aac034000 tx=0x7f0aac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.183+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0ab815fc80 con 0x7f0ab806c680 2024-05-09T03:20:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:26.413+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0ab806a270 con 0x7f0ab806c680 2024-05-09T03:20:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.045+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/252906836 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v325) ==== 162+0+0 (secure 0 0 0) 0x7f0aa80605b0 con 0x7f0ab806c680 2024-05-09T03:20:27.047 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0aa4076300 msgr2=0x7f0aa4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0aa4076300 0x7f0aa4078810 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0aac034000 tx=0x7f0aac02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 msgr2=0x7f0ab80cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 0x7f0ab80cf430 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800caf0 tx=0x7f0aa800a060 comp rx=0 tx=0).stop 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 shutdown_connections 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ab815f870 0x7f0ab8141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f0ab815e6b0 0x7f0ab815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0aa4076300 0x7f0aa4078810 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 --2- 172.21.15.46:0/252906836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c680 0x7f0ab80cf430 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 >> 172.21.15.46:0/252906836 conn(0x7f0ab806de50 msgr2=0x7f0ab8161190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 shutdown_connections 2024-05-09T03:20:27.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.048+0000 7f0abf861640 1 -- 172.21.15.46:0/252906836 wait complete. 2024-05-09T03:20:27.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:27.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.180+0000 7eff09d1c640 1 Processor -- start 2024-05-09T03:20:27.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.180+0000 7eff09d1c640 1 -- start start 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.181+0000 7eff09d1c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff041615e0 0x7eff0415dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.181+0000 7eff09d1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff04160b20 con 0x7eff04164c70 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.181+0000 7eff09d1c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff040d15f0 con 0x7eff040d2480 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.181+0000 7eff09d1c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7eff0415e520 con 0x7eff041615e0 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.181+0000 7eff037fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff04164c70 0x7eff040d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59024/0 (socket says 172.21.15.46:59024) 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.181+0000 7eff037fe640 1 -- 172.21.15.46:0/3577443578 learned_addr learned my addr 172.21.15.46:0/3577443578 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454068431 0 0) 0x7eff04160b20 con 0x7eff04164c70 2024-05-09T03:20:27.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efeec003520 con 0x7eff04164c70 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff02ffd640 1 --2- 172.21.15.46:0/3577443578 >> v2:172.21.15.88:3300/0 conn(0x7eff041615e0 0x7eff0415dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1056899850 0 0) 0x7efeec003520 con 0x7eff04164c70 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 >> v2:172.21.15.88:3300/0 conn(0x7eff041615e0 msgr2=0x7eff0415dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 --2- 172.21.15.46:0/3577443578 >> v2:172.21.15.88:3300/0 conn(0x7eff041615e0 0x7eff0415dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 >> v1:172.21.15.88:6789/0 conn(0x7eff040d2480 legacy=0x7eff040d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff040d18d0 con 0x7eff04164c70 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7efef40016f0 con 0x7eff04164c70 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 >> v1:172.21.15.46:6789/0 conn(0x7eff04164c70 legacy=0x7eff040d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7efeec003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec004800 0x7efeec004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0053a0 0x7efeec00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efeec00c300 con 0x7efeec003a50 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efeec00c4b0 con 0x7efeec0053a0 2024-05-09T03:20:27.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7efeec00c6b0 con 0x7efeec004800 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff02ffd640 1 --2- 172.21.15.46:0/3577443578 >> v2:172.21.15.88:3300/0 conn(0x7eff041615e0 0x7eff0415dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff037fe640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7efeec003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff03fff640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0053a0 0x7efeec00ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff02ffd640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec004800 0x7efeec004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff037fe640 1 -- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec004800 msgr2=0x7efeec004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff037fe640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7efeec004800 0x7efeec004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff037fe640 1 -- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0053a0 msgr2=0x7efeec00ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff037fe640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0053a0 0x7efeec00ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff037fe640 1 -- 172.21.15.46:0/3577443578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec003520 con 0x7efeec003a50 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.182+0000 7eff03fff640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeec0053a0 0x7efeec00ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.183+0000 7eff03fff640 1 -- 172.21.15.46:0/3577443578 reap_dead start 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.183+0000 7eff037fe640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7efeec003ec0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7efef400c290 tx=0x7efeec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7a786527058c5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.183+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4007d50 con 0x7efeec003a50 2024-05-09T03:20:27.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.183+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef4005af0 con 0x7efeec003a50 2024-05-09T03:20:27.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.183+0000 7eff027fc640 1 -- 172.21.15.46:0/3577443578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4007070 con 0x7efeec003a50 2024-05-09T03:20:27.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 -- 172.21.15.46:0/3577443578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 msgr2=0x7efeec003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7efeec003ec0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7efef400c290 tx=0x7efeec005830 comp rx=0 tx=0).stop 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 -- 172.21.15.46:0/3577443578 shutdown_connections 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 --2- 172.21.15.46:0/3577443578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7efeec003ec0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 -- 172.21.15.46:0/3577443578 >> 172.21.15.46:0/3577443578 conn(0x7eff04073db0 msgr2=0x7eff040741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 -- 172.21.15.46:0/3577443578 shutdown_connections 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.184+0000 7eff09d1c640 1 -- 172.21.15.46:0/3577443578 wait complete. 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.186+0000 7eff09d1c640 1 Processor -- start 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.186+0000 7eff09d1c640 1 -- start start 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff09d1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7eff041fa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff09d1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0414ad60 0x7eff0414a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff09d1c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff041f6740 0x7eff040cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff09d1c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7eff041619f0 con 0x7efeec003a50 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff09d1c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff04161150 con 0x7eff0414ad60 2024-05-09T03:20:27.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff09d1c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7eff0415dfe0 con 0x7eff041f6740 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.187+0000 7eff037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7eff041fa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7eff041fa7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53572/0 (socket says 172.21.15.46:53572) 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 -- 172.21.15.46:0/4118652820 learned_addr learned my addr 172.21.15.46:0/4118652820 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 -- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff041f6740 msgr2=0x7eff040cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff041f6740 0x7eff040cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 -- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0414ad60 msgr2=0x7eff0414a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff02ffd640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0414ad60 0x7eff0414a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0414ad60 0x7eff0414a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 -- 172.21.15.46:0/4118652820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff04161d10 con 0x7efeec003a50 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff02ffd640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0414ad60 0x7eff0414a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff037fe640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7eff041fa7d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7efef400c290 tx=0x7efef40019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4037040 con 0x7efeec003a50 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef4041070 con 0x7efeec003a50 2024-05-09T03:20:27.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff041f6ec0 con 0x7efeec003a50 2024-05-09T03:20:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef40016f0 con 0x7efeec003a50 2024-05-09T03:20:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.188+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff041f7480 con 0x7efeec003a50 2024-05-09T03:20:27.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.189+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff0408cfd0 con 0x7efeec003a50 2024-05-09T03:20:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.190+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efef40049e0 con 0x7efeec003a50 2024-05-09T03:20:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.191+0000 7efeebfff640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efef0076300 0x7efef0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.191+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8949+0+0 (secure 0 0 0) 0x7efef40b8b40 con 0x7efeec003a50 2024-05-09T03:20:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.191+0000 7eff02ffd640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efef0076300 0x7efef0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:27.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.192+0000 7eff02ffd640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efef0076300 0x7efef0078810 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7eff0414deb0 tx=0x7efef8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:27.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.195+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7eff0408cfd0 con 0x7efeec003a50 2024-05-09T03:20:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.423+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff0406a270 con 0x7efeec003a50 2024-05-09T03:20:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.423+0000 7efeebfff640 1 -- 172.21.15.46:0/4118652820 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 326 v326) ==== 95+0+5178 (secure 0 0 0) 0x7efef40817a0 con 0x7efeec003a50 2024-05-09T03:20:27.425 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:27.425 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":326,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":326,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:27.050420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10299":{"gid":10299,"name":"c","rank":0,"incarnation":326,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/574897435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":574897435},{"type":"v1","addr":"172.21.15.46:6834","nonce":574897435}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:27.425 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 326 2024-05-09T03:20:27.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efef0076300 msgr2=0x7efef0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efef0076300 0x7efef0078810 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7eff0414deb0 tx=0x7efef8008040 comp rx=0 tx=0).stop 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 msgr2=0x7eff041fa7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7eff041fa7d0 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7efef400c290 tx=0x7efef40019f0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 shutdown_connections 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7eff041f6740 0x7eff040cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff0414ad60 0x7eff0414a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7efef0076300 0x7efef0078810 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 --2- 172.21.15.46:0/4118652820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeec003a50 0x7eff041fa7d0 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 >> 172.21.15.46:0/4118652820 conn(0x7eff04073db0 msgr2=0x7eff040d4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 shutdown_connections 2024-05-09T03:20:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:27.426+0000 7eff09d1c640 1 -- 172.21.15.46:0/4118652820 wait complete. 2024-05-09T03:20:27.439 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:20:30.066 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:30.064+0000 7fa7cc92a640 -1 mds.0.cache.dir(0x100) Corrupt fnode in dirfrag 0x100: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:30.067 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:30.066+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:20:30.068 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:30.066+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:20:30.074 INFO:tasks.ceph.mds.c.smithi046.stderr: -22> 2024-05-09T03:20:30.064+0000 7fa7cc92a640 -1 mds.0.cache.dir(0x100) Corrupt fnode in dirfrag 0x100: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:30.136 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:32.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:32.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.567+0000 7f4205a69640 1 Processor -- start 2024-05-09T03:20:32.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.567+0000 7f4205a69640 1 -- start start 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.568+0000 7f4205a69640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42001615e0 0x7f420015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.568+0000 7f4205a69640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42000d2480 0x7f42000d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.568+0000 7f4205a69640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4200160b20 con 0x7f4200164c70 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.568+0000 7f4205a69640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f420015e520 con 0x7f42000d2480 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.568+0000 7f4205a69640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4200161150 con 0x7f42001615e0 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42000d2480 0x7f42000d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42000d2480 0x7f42000d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53604/0 (socket says 172.21.15.46:53604) 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 -- 172.21.15.46:0/3084230916 learned_addr learned my addr 172.21.15.46:0/3084230916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41f6ffd640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42001615e0 0x7f420015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 -- 172.21.15.46:0/3084230916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42001615e0 msgr2=0x7f420015dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42001615e0 0x7f420015dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 -- 172.21.15.46:0/3084230916 >> v1:172.21.15.88:6789/0 conn(0x7f4200164c70 legacy=0x7f42000d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41fffff640 1 -- 172.21.15.46:0/3084230916 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42000d15f0 con 0x7f42000d2480 2024-05-09T03:20:32.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.569+0000 7f41f6ffd640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42001615e0 0x7f420015dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.570+0000 7f41fffff640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42000d2480 0x7f42000d2890 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009e50 tx=0x7f41f802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6714fc1889131a90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.570+0000 7f41feffd640 1 -- 172.21.15.46:0/3084230916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8036070 con 0x7f42000d2480 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.570+0000 7f41feffd640 1 -- 172.21.15.46:0/3084230916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f8031600 con 0x7f42000d2480 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.570+0000 7f41feffd640 1 -- 172.21.15.46:0/3084230916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f80319d0 con 0x7f42000d2480 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.570+0000 7f4205a69640 1 -- 172.21.15.46:0/3084230916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42000d2480 msgr2=0x7f42000d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42000d2480 0x7f42000d2890 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009e50 tx=0x7f41f802a710 comp rx=0 tx=0).stop 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 -- 172.21.15.46:0/3084230916 shutdown_connections 2024-05-09T03:20:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42000d2480 0x7f42000d2890 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 --2- 172.21.15.46:0/3084230916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42001615e0 0x7f420015dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 -- 172.21.15.46:0/3084230916 >> 172.21.15.46:0/3084230916 conn(0x7f4200073db0 msgr2=0x7f42000741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 -- 172.21.15.46:0/3084230916 shutdown_connections 2024-05-09T03:20:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.571+0000 7f4205a69640 1 -- 172.21.15.46:0/3084230916 wait complete. 2024-05-09T03:20:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.572+0000 7f4205a69640 1 Processor -- start 2024-05-09T03:20:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.573+0000 7f4205a69640 1 -- start start 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f4205a69640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42000d2480 0x7f42001eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f4205a69640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42001615e0 0x7f4200158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f4205a69640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 0x7f42000cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f4205a69640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42001619f0 con 0x7f4200164c70 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f4205a69640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4200160de0 con 0x7f42001615e0 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f4205a69640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f42000d2dd0 con 0x7f42000d2480 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 0x7f42000cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 0x7f42000cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53608/0 (socket says 172.21.15.46:53608) 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41fffff640 1 -- 172.21.15.46:0/3070016752 learned_addr learned my addr 172.21.15.46:0/3070016752 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41ff7fe640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42000d2480 0x7f42001eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41f6ffd640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42001615e0 0x7f4200158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41fffff640 1 -- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42000d2480 msgr2=0x7f42001eb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.574+0000 7f41fffff640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42000d2480 0x7f42001eb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fffff640 1 -- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42001615e0 msgr2=0x7f4200158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fffff640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42001615e0 0x7f4200158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fffff640 1 -- 172.21.15.46:0/3070016752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42001ea680 con 0x7f4200164c70 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fffff640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 0x7f42000cf4c0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f41f8034040 tx=0x7f41f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8036050 con 0x7f4200164c70 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f8042070 con 0x7f4200164c70 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42001ea960 con 0x7f4200164c70 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8009510 con 0x7f4200164c70 2024-05-09T03:20:32.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.575+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42000cd270 con 0x7f4200164c70 2024-05-09T03:20:32.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.576+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420008cfd0 con 0x7f4200164c70 2024-05-09T03:20:32.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.577+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f41f8007c50 con 0x7f4200164c70 2024-05-09T03:20:32.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.577+0000 7f41fcff9640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41e00762d0 0x7f41e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.578+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f41f80b9140 con 0x7f4200164c70 2024-05-09T03:20:32.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.578+0000 7f41ff7fe640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41e00762d0 0x7f41e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.578+0000 7f41ff7fe640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41e00762d0 0x7f41e00787e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f41f0001720 tx=0x7f41f0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:32.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.582+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f420008cfd0 con 0x7f4200164c70 2024-05-09T03:20:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.814+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f420006a270 con 0x7f4200164c70 2024-05-09T03:20:32.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.815+0000 7f41fcff9640 1 -- 172.21.15.46:0/3070016752 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5170 (secure 0 0 0) 0x7f41f8081670 con 0x7f4200164c70 2024-05-09T03:20:32.816 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:32.817 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":331,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2034185175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2034185175},{"type":"v1","addr":"172.21.15.46:6834","nonce":2034185175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:30.067821+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:32.817 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 331 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41e00762d0 msgr2=0x7f41e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41e00762d0 0x7f41e00787e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f41f0001720 tx=0x7f41f0009040 comp rx=0 tx=0).stop 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 msgr2=0x7f42000cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 0x7f42000cf4c0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f41f8034040 tx=0x7f41f8004a40 comp rx=0 tx=0).stop 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 shutdown_connections 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f41e00762d0 0x7f41e00787e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4200164c70 0x7f42000cf4c0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42001615e0 0x7f4200158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 --2- 172.21.15.46:0/3070016752 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f42000d2480 0x7f42001eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 >> 172.21.15.46:0/3070016752 conn(0x7f4200073db0 msgr2=0x7f42000741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 shutdown_connections 2024-05-09T03:20:32.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.817+0000 7f4205a69640 1 -- 172.21.15.46:0/3070016752 wait complete. 2024-05-09T03:20:32.830 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:20:32.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.943+0000 7f7f99308640 1 Processor -- start 2024-05-09T03:20:32.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.944+0000 7f7f99308640 1 -- start start 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f99308640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f940b92e0 0x7f7f940b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f99308640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940d0480 0x7f7f940d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f99308640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f940d0dd0 con 0x7f7f940ce6a0 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f99308640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7f940d2b80 con 0x7f7f940b92e0 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f99308640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7f940d2e60 con 0x7f7f940d0480 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f940b92e0 0x7f7f940b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f940b92e0 0x7f7f940b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53620/0 (socket says 172.21.15.46:53620) 2024-05-09T03:20:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 -- 172.21.15.46:0/364038995 learned_addr learned my addr 172.21.15.46:0/364038995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f927fc640 1 --2- 172.21.15.46:0/364038995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940d0480 0x7f7f940d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 -- 172.21.15.46:0/364038995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940d0480 msgr2=0x7f7f940d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 --2- 172.21.15.46:0/364038995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940d0480 0x7f7f940d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 -- 172.21.15.46:0/364038995 >> v1:172.21.15.88:6789/0 conn(0x7f7f940ce6a0 legacy=0x7f7f940ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.945+0000 7f7f92ffd640 1 -- 172.21.15.46:0/364038995 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f940d3140 con 0x7f7f940b92e0 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.946+0000 7f7f92ffd640 1 --2- 172.21.15.46:0/364038995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940b92e0 0x7f7f940b51a0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009870 tx=0x7f7f8402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c797ebe9efa64cdd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.946+0000 7f7f91ffb640 1 -- 172.21.15.46:0/364038995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84037070 con 0x7f7f940b92e0 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.946+0000 7f7f91ffb640 1 -- 172.21.15.46:0/364038995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f84032440 con 0x7f7f940b92e0 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.946+0000 7f7f91ffb640 1 -- 172.21.15.46:0/364038995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84032810 con 0x7f7f940b92e0 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 -- 172.21.15.46:0/364038995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940b92e0 msgr2=0x7f7f940b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 --2- 172.21.15.46:0/364038995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940b92e0 0x7f7f940b51a0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f7f84009870 tx=0x7f7f8402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 -- 172.21.15.46:0/364038995 shutdown_connections 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 --2- 172.21.15.46:0/364038995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940d0480 0x7f7f940d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 --2- 172.21.15.46:0/364038995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940b92e0 0x7f7f940b51a0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 -- 172.21.15.46:0/364038995 >> 172.21.15.46:0/364038995 conn(0x7f7f940d3c00 msgr2=0x7f7f940d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 -- 172.21.15.46:0/364038995 shutdown_connections 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.947+0000 7f7f99308640 1 -- 172.21.15.46:0/364038995 wait complete. 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.949+0000 7f7f99308640 1 Processor -- start 2024-05-09T03:20:32.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.949+0000 7f7f99308640 1 -- start start 2024-05-09T03:20:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.950+0000 7f7f99308640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940b92e0 0x7f7f94159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.950+0000 7f7f99308640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 0x7f7f9414b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.950+0000 7f7f99308640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f940d0480 0x7f7f9414b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.950+0000 7f7f99308640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9406e920 con 0x7f7f940ce6a0 2024-05-09T03:20:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.950+0000 7f7f99308640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9406e7a0 con 0x7f7f940d0480 2024-05-09T03:20:32.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.950+0000 7f7f99308640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9406eaa0 con 0x7f7f940b92e0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 0x7f7f9414b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 0x7f7f9414b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53624/0 (socket says 172.21.15.46:53624) 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 -- 172.21.15.46:0/3333920846 learned_addr learned my addr 172.21.15.46:0/3333920846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f92ffd640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940b92e0 0x7f7f94159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f937fe640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f940d0480 0x7f7f9414b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 -- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940b92e0 msgr2=0x7f7f94159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940b92e0 0x7f7f94159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 -- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f940d0480 msgr2=0x7f7f9414b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f940d0480 0x7f7f9414b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 -- 172.21.15.46:0/3333920846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9415cbe0 con 0x7f7f940ce6a0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f92ffd640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940b92e0 0x7f7f94159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f937fe640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f940d0480 0x7f7f9414b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f927fc640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 0x7f7f9414b010 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800d600 tx=0x7f7f8800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f880125f0 con 0x7f7f940ce6a0 2024-05-09T03:20:32.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f88019070 con 0x7f7f940ce6a0 2024-05-09T03:20:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f880129b0 con 0x7f7f940ce6a0 2024-05-09T03:20:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9415cec0 con 0x7f7f940ce6a0 2024-05-09T03:20:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.951+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9415d480 con 0x7f7f940ce6a0 2024-05-09T03:20:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.952+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9406a270 con 0x7f7f940ce6a0 2024-05-09T03:20:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.953+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7f88012b50 con 0x7f7f940ce6a0 2024-05-09T03:20:32.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.953+0000 7f7f7b7fe640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f80076300 0x7f7f80078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.954+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f7f88098610 con 0x7f7f940ce6a0 2024-05-09T03:20:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.954+0000 7f7f92ffd640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f80076300 0x7f7f80078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:32.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.954+0000 7f7f92ffd640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f80076300 0x7f7f80078810 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7f84034000 tx=0x7f7f8402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:32.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:32.958+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7f88061230 con 0x7f7f940ce6a0 2024-05-09T03:20:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.190+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f94072820 con 0x7f7f940ce6a0 2024-05-09T03:20:33.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.191+0000 7f7f7b7fe640 1 -- 172.21.15.46:0/3333920846 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5170 (secure 0 0 0) 0x7f7f88060b30 con 0x7f7f940ce6a0 2024-05-09T03:20:33.193 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:33.193 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":331,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2034185175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2034185175},{"type":"v1","addr":"172.21.15.46:6834","nonce":2034185175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:30.067821+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:33.193 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 331 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f80076300 msgr2=0x7f7f80078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f80076300 0x7f7f80078810 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7f84034000 tx=0x7f7f8402e000 comp rx=0 tx=0).stop 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 msgr2=0x7f7f9414b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 0x7f7f9414b010 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800d600 tx=0x7f7f8800dac0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 shutdown_connections 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f940d0480 0x7f7f9414b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7f80076300 0x7f7f80078810 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f940ce6a0 0x7f7f9414b010 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 --2- 172.21.15.46:0/3333920846 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7f940b92e0 0x7f7f94159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 >> 172.21.15.46:0/3333920846 conn(0x7f7f940d3c00 msgr2=0x7f7f940b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 shutdown_connections 2024-05-09T03:20:33.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.193+0000 7f7f99308640 1 -- 172.21.15.46:0/3333920846 wait complete. 2024-05-09T03:20:33.206 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:20:33.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:33.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.319+0000 7f65db9e0640 1 Processor -- start 2024-05-09T03:20:33.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.319+0000 7f65db9e0640 1 -- start start 2024-05-09T03:20:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65db9e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65d41608f0 0x7f65d4157120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65db9e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d40b13e0 0x7f65d40b5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65db9e0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d4160e20 con 0x7f65d40b6060 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65db9e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f65d4161100 con 0x7f65d41608f0 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65db9e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f65d41613e0 con 0x7f65d40b13e0 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d977c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f65d41608f0 0x7f65d4157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d8f7b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f65d40b6060 0x7f65d40b0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59042/0 (socket says 172.21.15.46:59042) 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d8f7b640 1 -- 172.21.15.46:0/2284461880 learned_addr learned my addr 172.21.15.46:0/2284461880 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d9f7d640 1 --2- 172.21.15.46:0/2284461880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d40b13e0 0x7f65d40b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d977c640 1 -- 172.21.15.46:0/2284461880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d40b13e0 msgr2=0x7f65d40b5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d977c640 1 --2- 172.21.15.46:0/2284461880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d40b13e0 0x7f65d40b5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d977c640 1 -- 172.21.15.46:0/2284461880 >> v1:172.21.15.46:6789/0 conn(0x7f65d40b6060 legacy=0x7f65d40b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d977c640 1 -- 172.21.15.46:0/2284461880 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d41616c0 con 0x7f65d41608f0 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.321+0000 7f65d3fff640 1 -- 172.21.15.46:0/2284461880 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578777005 0 0) 0x7f65d4160e20 con 0x7f65d40b6060 2024-05-09T03:20:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.322+0000 7f65d977c640 1 --2- 172.21.15.46:0/2284461880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d41608f0 0x7f65d4157120 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009d90 tx=0x7f65c402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c121001dbabac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.322+0000 7f65d3fff640 1 -- 172.21.15.46:0/2284461880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4007740 con 0x7f65d41608f0 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.322+0000 7f65d3fff640 1 -- 172.21.15.46:0/2284461880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c40078e0 con 0x7f65d41608f0 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.322+0000 7f65d3fff640 1 -- 172.21.15.46:0/2284461880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c4007cb0 con 0x7f65d41608f0 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 -- 172.21.15.46:0/2284461880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d41608f0 msgr2=0x7f65d4157120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 --2- 172.21.15.46:0/2284461880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d41608f0 0x7f65d4157120 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009d90 tx=0x7f65c402a660 comp rx=0 tx=0).stop 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 -- 172.21.15.46:0/2284461880 shutdown_connections 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 --2- 172.21.15.46:0/2284461880 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d40b13e0 0x7f65d40b5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 --2- 172.21.15.46:0/2284461880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d41608f0 0x7f65d4157120 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 -- 172.21.15.46:0/2284461880 >> 172.21.15.46:0/2284461880 conn(0x7f65d40b3910 msgr2=0x7f65d40b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 -- 172.21.15.46:0/2284461880 shutdown_connections 2024-05-09T03:20:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.323+0000 7f65db9e0640 1 -- 172.21.15.46:0/2284461880 wait complete. 2024-05-09T03:20:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.326+0000 7f65db9e0640 1 Processor -- start 2024-05-09T03:20:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.326+0000 7f65db9e0640 1 -- start start 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d40b6060 0x7f65d414b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d41608f0 0x7f65d414c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 0x7f65d4154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f65d41577e0 con 0x7f65d4154b30 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f65d4157660 con 0x7f65d40b6060 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f65d40b6470 con 0x7f65d41608f0 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 0x7f65d4154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 0x7f65d4154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53646/0 (socket says 172.21.15.46:53646) 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 -- 172.21.15.46:0/3463580540 learned_addr learned my addr 172.21.15.46:0/3463580540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d977c640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d40b6060 0x7f65d414b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d8f7b640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d41608f0 0x7f65d414c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 -- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d41608f0 msgr2=0x7f65d414c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d41608f0 0x7f65d414c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 -- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d40b6060 msgr2=0x7f65d414b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d40b6060 0x7f65d414b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 -- 172.21.15.46:0/3463580540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d4155480 con 0x7f65d4154b30 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d8f7b640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d41608f0 0x7f65d414c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d9f7d640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 0x7f65d4154f40 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00d5a0 tx=0x7f65cc00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0125f0 con 0x7f65d4154b30 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc019070 con 0x7f65d4154b30 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d415c5c0 con 0x7f65d4154b30 2024-05-09T03:20:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc0129f0 con 0x7f65d4154b30 2024-05-09T03:20:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.327+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d415cb80 con 0x7f65d4154b30 2024-05-09T03:20:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.328+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65d406a270 con 0x7f65d4154b30 2024-05-09T03:20:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.329+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f65cc012b90 con 0x7f65d4154b30 2024-05-09T03:20:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.330+0000 7f65d1ffb640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65c00762d0 0x7f65c00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.330+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f65cc099170 con 0x7f65d4154b30 2024-05-09T03:20:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.330+0000 7f65d977c640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65c00762d0 0x7f65c00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.332+0000 7f65d977c640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65c00762d0 0x7f65c00787e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f65c4034000 tx=0x7f65c4033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.335+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f65d4155480 con 0x7f65d4154b30 2024-05-09T03:20:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.563+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65d4073f30 con 0x7f65d4154b30 2024-05-09T03:20:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.564+0000 7f65d1ffb640 1 -- 172.21.15.46:0/3463580540 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5170 (secure 0 0 0) 0x7f65cc061e10 con 0x7f65d4154b30 2024-05-09T03:20:33.566 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:33.566 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":331,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2034185175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2034185175},{"type":"v1","addr":"172.21.15.46:6834","nonce":2034185175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":331}],"filesystems":[{"mdsmap":{"epoch":330,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:30.067821+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 331 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65c00762d0 msgr2=0x7f65c00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65c00762d0 0x7f65c00787e0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f65c4034000 tx=0x7f65c4033040 comp rx=0 tx=0).stop 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 msgr2=0x7f65d4154f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 0x7f65d4154f40 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00d5a0 tx=0x7f65cc00da60 comp rx=0 tx=0).stop 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 shutdown_connections 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65c00762d0 0x7f65c00787e0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65d4154b30 0x7f65d4154f40 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f65d41608f0 0x7f65d414c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 --2- 172.21.15.46:0/3463580540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65d40b6060 0x7f65d414b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 >> 172.21.15.46:0/3463580540 conn(0x7f65d40b3910 msgr2=0x7f65d40b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 shutdown_connections 2024-05-09T03:20:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.566+0000 7f65db9e0640 1 -- 172.21.15.46:0/3463580540 wait complete. 2024-05-09T03:20:33.579 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 100.00000000' led to DAMAGED state 2024-05-09T03:20:33.579 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on mds0_sessionmap' 2024-05-09T03:20:33.580 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:33.580 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:33.596 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:33.596 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:33.596 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:20:33.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:33.596 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:20:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:20:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:20:33.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:20:33.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.807+0000 7f27a3900640 1 Processor -- start 2024-05-09T03:20:33.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.807+0000 7f27a3900640 1 -- start start 2024-05-09T03:20:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a3900640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f279c0b5140 0x7f279c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a3900640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f279c0b7ca0 0x7f279c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a3900640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279c0b80b0 con 0x7f279c0d3df0 2024-05-09T03:20:33.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a3900640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f279c0d3480 con 0x7f279c0b7ca0 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a3900640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f279c0b5550 con 0x7f279c0b5140 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a0e9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f279c0b7ca0 0x7f279c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a1e9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f279c0d3df0 0x7f279c163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59046/0 (socket says 172.21.15.46:59046) 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a1e9d640 1 -- 172.21.15.46:0/4083141739 learned_addr learned my addr 172.21.15.46:0/4083141739 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a169c640 1 --2- 172.21.15.46:0/4083141739 >> v2:172.21.15.88:3300/0 conn(0x7f279c0b5140 0x7f279c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a0e9b640 1 -- 172.21.15.46:0/4083141739 >> v2:172.21.15.88:3300/0 conn(0x7f279c0b5140 msgr2=0x7f279c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a0e9b640 1 --2- 172.21.15.46:0/4083141739 >> v2:172.21.15.88:3300/0 conn(0x7f279c0b5140 0x7f279c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a0e9b640 1 -- 172.21.15.46:0/4083141739 >> v1:172.21.15.46:6789/0 conn(0x7f279c0d3df0 legacy=0x7f279c163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f2793fff640 1 -- 172.21.15.46:0/4083141739 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031746380 0 0) 0x7f2794001110 con 0x7f279c0d3df0 2024-05-09T03:20:33.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.808+0000 7f27a0e9b640 1 -- 172.21.15.46:0/4083141739 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c0d44e0 con 0x7f279c0b7ca0 2024-05-09T03:20:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.809+0000 7f27a0e9b640 1 --2- 172.21.15.46:0/4083141739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0b7ca0 0x7f279c0d38b0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f279c0b88d0 tx=0x7f278c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505bc1582e774476 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.809+0000 7f2793fff640 1 -- 172.21.15.46:0/4083141739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c037070 con 0x7f279c0b7ca0 2024-05-09T03:20:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.809+0000 7f2793fff640 1 -- 172.21.15.46:0/4083141739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f278c032440 con 0x7f279c0b7ca0 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.809+0000 7f2793fff640 1 -- 172.21.15.46:0/4083141739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278c032810 con 0x7f279c0b7ca0 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 -- 172.21.15.46:0/4083141739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0b7ca0 msgr2=0x7f279c0d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 --2- 172.21.15.46:0/4083141739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0b7ca0 0x7f279c0d38b0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f279c0b88d0 tx=0x7f278c02a230 comp rx=0 tx=0).stop 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 -- 172.21.15.46:0/4083141739 shutdown_connections 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 --2- 172.21.15.46:0/4083141739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0b7ca0 0x7f279c0d38b0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 --2- 172.21.15.46:0/4083141739 >> v2:172.21.15.88:3300/0 conn(0x7f279c0b5140 0x7f279c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 -- 172.21.15.46:0/4083141739 >> 172.21.15.46:0/4083141739 conn(0x7f279c08d3f0 msgr2=0x7f279c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 -- 172.21.15.46:0/4083141739 shutdown_connections 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.810+0000 7f27a3900640 1 -- 172.21.15.46:0/4083141739 wait complete. 2024-05-09T03:20:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.812+0000 7f27a3900640 1 Processor -- start 2024-05-09T03:20:33.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.812+0000 7f27a3900640 1 -- start start 2024-05-09T03:20:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a3900640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f279c0b5140 0x7f279c156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a3900640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c0b7ca0 0x7f279c0c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a3900640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 0x7f279c1e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a3900640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f279c0d4d80 con 0x7f279c0d3df0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a3900640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f279c0d4aa0 con 0x7f279c0b7ca0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a3900640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f279c0d4200 con 0x7f279c0b5140 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.813+0000 7f27a1e9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 0x7f279c1e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 0x7f279c1e9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53666/0 (socket says 172.21.15.46:53666) 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 -- 172.21.15.46:0/2572297589 learned_addr learned my addr 172.21.15.46:0/2572297589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a0e9b640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c0b7ca0 0x7f279c0c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a169c640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f279c0b5140 0x7f279c156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 -- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f279c0b5140 msgr2=0x7f279c156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f279c0b5140 0x7f279c156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 -- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c0b7ca0 msgr2=0x7f279c0c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c0b7ca0 0x7f279c0c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 -- 172.21.15.46:0/2572297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279c0c92c0 con 0x7f279c0d3df0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a1e9d640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 0x7f279c1e9cf0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f2794001eb0 tx=0x7f279400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27940074e0 con 0x7f279c0d3df0 2024-05-09T03:20:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27940047e0 con 0x7f279c0d3df0 2024-05-09T03:20:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279c1fa3f0 con 0x7f279c0d3df0 2024-05-09T03:20:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.814+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794004bd0 con 0x7f279c0d3df0 2024-05-09T03:20:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.815+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279c1fa9b0 con 0x7f279c0d3df0 2024-05-09T03:20:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.817+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279c0d2ff0 con 0x7f279c0d3df0 2024-05-09T03:20:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.817+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2794011070 con 0x7f279c0d3df0 2024-05-09T03:20:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.818+0000 7f2791ffb640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27800762d0 0x7f27800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.818+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f2794097e70 con 0x7f279c0d3df0 2024-05-09T03:20:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.818+0000 7f27a169c640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27800762d0 0x7f27800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.818+0000 7f27a169c640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27800762d0 0x7f27800787e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f27980099c0 tx=0x7f2798003990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:33.822+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f279c0d2ff0 con 0x7f279c0d3df0 2024-05-09T03:20:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.053+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f279c0b6790 con 0x7f279c0d3df0 2024-05-09T03:20:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.088+0000 7f2791ffb640 1 -- 172.21.15.46:0/2572297589 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v332) ==== 161+0+0 (secure 0 0 0) 0x7f279c0c92c0 con 0x7f279c0d3df0 2024-05-09T03:20:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27800762d0 msgr2=0x7f27800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27800762d0 0x7f27800787e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f27980099c0 tx=0x7f2798003990 comp rx=0 tx=0).stop 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 msgr2=0x7f279c1e9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 0x7f279c1e9cf0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f2794001eb0 tx=0x7f279400c820 comp rx=0 tx=0).stop 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 shutdown_connections 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27800762d0 0x7f27800787e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f279c0d3df0 0x7f279c1e9cf0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f279c0b7ca0 0x7f279c0c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 --2- 172.21.15.46:0/2572297589 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f279c0b5140 0x7f279c156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 >> 172.21.15.46:0/2572297589 conn(0x7f279c08d3f0 msgr2=0x7f279c161cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:34.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 shutdown_connections 2024-05-09T03:20:34.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.090+0000 7f27a3900640 1 -- 172.21.15.46:0/2572297589 wait complete. 2024-05-09T03:20:34.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:20:34.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.217+0000 7fc181d66640 1 Processor -- start 2024-05-09T03:20:34.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.217+0000 7fc181d66640 1 -- start start 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc181d66640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b4700 0x7fc17c0d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc181d66640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc17c0d2630 0x7fc17c0b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc181d66640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c06ebe0 con 0x7fc17c0b3f50 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc181d66640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc17c06eec0 con 0x7fc17c0d2630 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc181d66640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c06f1a0 con 0x7fc17c0b4700 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc17c0b3f50 0x7fc17c08cd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59052/0 (socket says 172.21.15.46:59052) 2024-05-09T03:20:34.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17bfff640 1 -- 172.21.15.46:0/1670824776 learned_addr learned my addr 172.21.15.46:0/1670824776 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17affd640 1 --2- 172.21.15.46:0/1670824776 >> v2:172.21.15.88:3300/0 conn(0x7fc17c0d2630 0x7fc17c0b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17b7fe640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b4700 0x7fc17c0d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17a7fc640 1 -- 172.21.15.46:0/1670824776 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923769548 0 0) 0x7fc17c06ebe0 con 0x7fc17c0b3f50 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17a7fc640 1 -- 172.21.15.46:0/1670824776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc164003520 con 0x7fc17c0b3f50 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.218+0000 7fc17affd640 1 -- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b4700 msgr2=0x7fc17c0d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.219+0000 7fc17affd640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b4700 0x7fc17c0d20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.219+0000 7fc17affd640 1 -- 172.21.15.46:0/1670824776 >> v1:172.21.15.46:6789/0 conn(0x7fc17c0b3f50 legacy=0x7fc17c08cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.219+0000 7fc17affd640 1 -- 172.21.15.46:0/1670824776 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c06f480 con 0x7fc17c0d2630 2024-05-09T03:20:34.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.219+0000 7fc17b7fe640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b4700 0x7fc17c0d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:34.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.219+0000 7fc17affd640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c0b3a10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc1700097a0 tx=0x7fc17002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd2f6e7c9568378 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.219+0000 7fc17a7fc640 1 -- 172.21.15.46:0/1670824776 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170037070 con 0x7fc17c0d2630 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc17a7fc640 1 -- 172.21.15.46:0/1670824776 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc17002ae50 con 0x7fc17c0d2630 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc17a7fc640 1 -- 172.21.15.46:0/1670824776 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170032640 con 0x7fc17c0d2630 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 -- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 msgr2=0x7fc17c0b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c0b3a10 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc1700097a0 tx=0x7fc17002a100 comp rx=0 tx=0).stop 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 -- 172.21.15.46:0/1670824776 shutdown_connections 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c0b3a10 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 --2- 172.21.15.46:0/1670824776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b4700 0x7fc17c0d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 -- 172.21.15.46:0/1670824776 >> 172.21.15.46:0/1670824776 conn(0x7fc17c0b55d0 msgr2=0x7fc17c0b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 -- 172.21.15.46:0/1670824776 shutdown_connections 2024-05-09T03:20:34.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.220+0000 7fc181d66640 1 -- 172.21.15.46:0/1670824776 wait complete. 2024-05-09T03:20:34.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.222+0000 7fc181d66640 1 Processor -- start 2024-05-09T03:20:34.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.222+0000 7fc181d66640 1 -- start start 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc181d66640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b3f50 0x7fc17c0b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc181d66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 0x7fc17c1f9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc181d66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c1fa070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc181d66640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c0b51d0 con 0x7fc17c0b4700 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc181d66640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c0d2a40 con 0x7fc17c0d2630 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc181d66640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c0b2000 con 0x7fc17c0b3f50 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 0x7fc17c1f9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 0x7fc17c1f9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53674/0 (socket says 172.21.15.46:53674) 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 -- 172.21.15.46:0/2963432519 learned_addr learned my addr 172.21.15.46:0/2963432519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:34.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17b7fe640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b3f50 0x7fc17c0b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17bfff640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c1fa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 -- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b3f50 msgr2=0x7fc17c0b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b3f50 0x7fc17c0b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 -- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 msgr2=0x7fc17c1fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c1fa070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 -- 172.21.15.46:0/2963432519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c0b1d40 con 0x7fc17c0b4700 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc17affd640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 0x7fc17c1f9b30 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc1700097a0 tx=0x7fc1700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.224+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170041020 con 0x7fc17c0b4700 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.225+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1700044c0 con 0x7fc17c0b4700 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.225+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c0b2250 con 0x7fc17c0b4700 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.225+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170037040 con 0x7fc17c0b4700 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.225+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17c0b2810 con 0x7fc17c0b4700 2024-05-09T03:20:34.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.225+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17c06a270 con 0x7fc17c0b4700 2024-05-09T03:20:34.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.226+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc170044020 con 0x7fc17c0b4700 2024-05-09T03:20:34.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.227+0000 7fc163fff640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc168076300 0x7fc168078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:34.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.227+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7fc1700b8210 con 0x7fc17c0b4700 2024-05-09T03:20:34.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.227+0000 7fc17b7fe640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc168076300 0x7fc168078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:34.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.228+0000 7fc17b7fe640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc168076300 0x7fc168078810 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009560 tx=0x7fc16c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:34.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.232+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc17004b020 con 0x7fc17c0b4700 2024-05-09T03:20:34.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:34.497+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc17c0b5350 con 0x7fc17c0b4700 2024-05-09T03:20:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.096+0000 7fc163fff640 1 -- 172.21.15.46:0/2963432519 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v333) ==== 101+0+0 (secure 0 0 0) 0x7fc170080750 con 0x7fc17c0b4700 2024-05-09T03:20:35.097 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:20:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc168076300 msgr2=0x7fc168078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc168076300 0x7fc168078810 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009560 tx=0x7fc16c009190 comp rx=0 tx=0).stop 2024-05-09T03:20:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 msgr2=0x7fc17c1f9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 0x7fc17c1f9b30 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc1700097a0 tx=0x7fc1700046c0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 shutdown_connections 2024-05-09T03:20:35.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc17c0d2630 0x7fc17c1fa070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc168076300 0x7fc168078810 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc17c0b4700 0x7fc17c1f9b30 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 --2- 172.21.15.46:0/2963432519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc17c0b3f50 0x7fc17c0b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 >> 172.21.15.46:0/2963432519 conn(0x7fc17c0b55d0 msgr2=0x7fc17c08d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 shutdown_connections 2024-05-09T03:20:35.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.099+0000 7fc181d66640 1 -- 172.21.15.46:0/2963432519 wait complete. 2024-05-09T03:20:35.114 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:20:35.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.174+0000 7f90c991ef40 1 Processor -- start 2024-05-09T03:20:35.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.174+0000 7f90c991ef40 1 -- start start 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c991ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7170b67e0 0x55e7170e7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c991ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170e87b0 0x55e717190c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c991ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 0x55e717191160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c991ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e7170fd4d0 con 0x55e717190810 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c991ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e71705ff50 con 0x55e7170b67e0 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c991ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e7171028a0 con 0x55e7170e87b0 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c3119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 0x55e717191160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c3119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 0x55e717191160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53700/0 (socket says 172.21.15.46:53700) 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.175+0000 7f90c3119640 1 -- 172.21.15.46:0/994747291 learned_addr learned my addr 172.21.15.46:0/994747291 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c2918640 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7170b67e0 0x55e7170e7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90bbfff640 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170e87b0 0x55e717190c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c3119640 1 -- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170e87b0 msgr2=0x55e717190c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:35.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c3119640 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170e87b0 0x55e717190c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c3119640 1 -- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7170b67e0 msgr2=0x55e7170e7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c3119640 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7170b67e0 0x55e7170e7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c3119640 1 -- 172.21.15.46:0/994747291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7170e6130 con 0x55e717190810 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c3119640 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 0x55e717191160 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c3e0 tx=0x7f90bc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee98230dd1ee12ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c1916640 1 -- 172.21.15.46:0/994747291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0115f0 con 0x55e717190810 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c1916640 1 -- 172.21.15.46:0/994747291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc011790 con 0x55e717190810 2024-05-09T03:20:35.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.176+0000 7f90c1916640 1 -- 172.21.15.46:0/994747291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc011980 con 0x55e717190810 2024-05-09T03:20:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 -- 172.21.15.46:0/994747291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 msgr2=0x55e717191160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 0x55e717191160 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f90bc00c3e0 tx=0x7f90bc00c8a0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 -- 172.21.15.46:0/994747291 shutdown_connections 2024-05-09T03:20:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e717190810 0x55e717191160 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170e87b0 0x55e717190c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 --2- 172.21.15.46:0/994747291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7170b67e0 0x55e7170e7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 -- 172.21.15.46:0/994747291 >> 172.21.15.46:0/994747291 conn(0x55e716fdbac0 msgr2=0x55e717100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:35.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 -- 172.21.15.46:0/994747291 shutdown_connections 2024-05-09T03:20:35.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.177+0000 7f90c991ef40 1 -- 172.21.15.46:0/994747291 wait complete. 2024-05-09T03:20:35.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.178+0000 7f90c991ef40 1 Processor -- start 2024-05-09T03:20:35.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.178+0000 7f90c991ef40 1 -- start start 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c991ef40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170b67e0 0x55e7170fae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c991ef40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7170e87b0 0x55e7170fb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c991ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e717190810 0x55e7170fb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c991ef40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e7170facb0 con 0x55e7170e87b0 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c991ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e717103850 con 0x55e717190810 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c991ef40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e71718b990 con 0x55e7170b67e0 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7170e87b0 0x55e7170fb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90bbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7170e87b0 0x55e7170fb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53704/0 (socket says 172.21.15.46:53704) 2024-05-09T03:20:35.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 learned_addr learned my addr 172.21.15.46:0/1589603839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c3119640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e717190810 0x55e7170fb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.179+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170b67e0 0x55e7170fae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170b67e0 msgr2=0x55e7170fae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7170b67e0 0x55e7170fae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e717190810 msgr2=0x55e7170fb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e717190810 0x55e7170fb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e717191cd0 con 0x55e7170e87b0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7170e87b0 0x55e7170fb370 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f90ac045df0 tx=0x7f90ac002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90c0914640 1 -- 172.21.15.46:0/1589603839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac064070 con 0x55e7170e87b0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90c0914640 1 -- 172.21.15.46:0/1589603839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ac0549f0 con 0x55e7170e87b0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e7170b9b70 con 0x55e7170e87b0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90c0914640 1 -- 172.21.15.46:0/1589603839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac059040 con 0x55e7170e87b0 2024-05-09T03:20:35.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.180+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7170ba130 con 0x55e7170e87b0 2024-05-09T03:20:35.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.182+0000 7f90c0914640 1 -- 172.21.15.46:0/1589603839 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f90ac067070 con 0x55e7170e87b0 2024-05-09T03:20:35.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.182+0000 7f90c0914640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a4074ce0 0x7f90a40771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.182+0000 7f90c0914640 1 -- 172.21.15.46:0/1589603839 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f90ac0dae10 con 0x55e7170e87b0 2024-05-09T03:20:35.184 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:20:35.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.183+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a4074ce0 0x7f90a40771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.183+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e7170b8970 0x55e7170b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.183+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7170b92c0 con 0x55e7170b8970 2024-05-09T03:20:35.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.183+0000 7f90c3119640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e7170b8970 0x55e7170b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.183+0000 7f90c3119640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e7170b8970 0x55e7170b8d80 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.183+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a4074ce0 0x7f90a40771f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f90b4000f60 tx=0x7f90b4007840 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.186+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v122'1159 uv1133 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.186+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717232820 con 0x55e7170b8970 2024-05-09T03:20:35.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.187+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v122'1160 uv1160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.187+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717232bc0 con 0x55e7170b8970 2024-05-09T03:20:35.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.195+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v122'1161 uv1161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.196 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:20:35.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.195+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233f70 con 0x55e7170b8970 2024-05-09T03:20:35.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.197+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v122'1162 uv1162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.197+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717234310 con 0x55e7170b8970 2024-05-09T03:20:35.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.198+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v122'1163 uv1163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.198+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717232f60 con 0x55e7170b8970 2024-05-09T03:20:35.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.199+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v122'1164 uv1164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.200+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233350 con 0x55e7170b8970 2024-05-09T03:20:35.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.201+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v122'1165 uv1139 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.201+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.203+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v122'1166 uv1166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.203+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.204+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v122'1167 uv1167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.205 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:20:35.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.204+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.205+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v122'1168 uv1168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.205+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.207+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v122'1169 uv1169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.207+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.208+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v122'1170 uv1170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.208+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.210+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v122'1171 uv1146 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.210+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.211+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v122'1172 uv1172 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.211+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.212+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v122'1173 uv1173 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.213 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:20:35.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.213+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.214+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v122'1174 uv1174 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.214+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.216+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v122'1175 uv1175 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.216+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172336f0 con 0x55e7170b8970 2024-05-09T03:20:35.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.218+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v122'1176 uv1176 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.218+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233b20 con 0x55e7170b8970 2024-05-09T03:20:35.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.219+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v122'1177 uv1177 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.219+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233ad0 con 0x55e7170b8970 2024-05-09T03:20:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.221+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v122'1178 uv1152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.221+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717233ad0 con 0x55e7170b8970 2024-05-09T03:20:35.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.222+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v122'1179 uv1179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.222+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233ad0 con 0x55e7170b8970 2024-05-09T03:20:35.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.224+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v122'1180 uv1180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.224 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:20:35.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.224+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.225+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v122'1181 uv1181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.225+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.226+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v122'1182 uv1182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.227+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.228+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v122'1183 uv1183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.228+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.230+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v122'1184 uv1158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.230+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.231+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v122'1185 uv1185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.231+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.233+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v122'1186 uv1186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.234 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:20:35.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.233+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.234+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v122'1187 uv1187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.234+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.236+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v122'1188 uv1188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.236+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717233960 con 0x55e7170b8970 2024-05-09T03:20:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.237+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v122'1189 uv1189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.237+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e7172346b0 0x55e717236ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.237+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237010 con 0x55e7172346b0 2024-05-09T03:20:35.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.238+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e7172346b0 0x55e717236ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.238+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e7172346b0 0x55e717236ad0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.240+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v122'837 uv819 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.240+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172373b0 con 0x55e7172346b0 2024-05-09T03:20:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.242+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v122'838 uv838 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.242+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237750 con 0x55e7172346b0 2024-05-09T03:20:35.243 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:20:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.243+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v122'839 uv839 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.243+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237fa0 con 0x55e7172346b0 2024-05-09T03:20:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.244+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v122'840 uv840 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.244+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717238340 con 0x55e7172346b0 2024-05-09T03:20:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.246+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v122'841 uv841 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.246+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172386e0 con 0x55e7172346b0 2024-05-09T03:20:35.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.247+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v122'842 uv824 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.247+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717238a80 con 0x55e7172346b0 2024-05-09T03:20:35.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.248+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v122'843 uv843 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.249+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717238e20 con 0x55e7172346b0 2024-05-09T03:20:35.250 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:20:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.250+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v122'844 uv844 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.250+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.251+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v122'845 uv845 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.251+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.252+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v122'846 uv846 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.252+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.254+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v122'847 uv829 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.254+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.255+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v122'848 uv848 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.255+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.257+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v122'849 uv849 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.257 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:20:35.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.257+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.258+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v122'850 uv850 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.258+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.259+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v122'851 uv851 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.259+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.261+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v122'852 uv835 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.261+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.262+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v122'853 uv853 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.262+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.263+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v122'854 uv854 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.264 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:20:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.263+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.265+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v122'855 uv855 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.265+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717237bd0 con 0x55e7172346b0 2024-05-09T03:20:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.266+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v122'856 uv856 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.266+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7172346b0 2024-05-09T03:20:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.267+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v122'857 uv857 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e7172346b0 2024-05-09T03:20:35.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.267+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7170b8970 2024-05-09T03:20:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.269+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v122'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.269+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7170b8970 2024-05-09T03:20:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.270+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v122'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.270+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7170b8970 2024-05-09T03:20:35.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.271+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v122'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.272 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:20:35.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.271+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7170b8970 2024-05-09T03:20:35.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.272+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v122'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.272+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7170b8970 2024-05-09T03:20:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.273+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v122'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.273+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717239320 con 0x55e7170b8970 2024-05-09T03:20:35.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.274+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v122'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.274+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55e717239c20 0x55e71723c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.274+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723c580 con 0x55e717239c20 2024-05-09T03:20:35.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.274+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55e717239c20 0x55e71723c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.274+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55e717239c20 0x55e71723c040 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.275+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v122'172 uv171 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.275+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e71723c920 con 0x55e717239c20 2024-05-09T03:20:35.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.277+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v122'173 uv173 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.277+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723ccc0 con 0x55e717239c20 2024-05-09T03:20:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.277+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v122'174 uv174 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.278 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:20:35.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.278+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717238ec0 con 0x55e717239c20 2024-05-09T03:20:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.278+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v122'175 uv175 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.279+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723d560 con 0x55e7170b8970 2024-05-09T03:20:35.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.280+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v122'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.280+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e71723d560 con 0x55e7170b8970 2024-05-09T03:20:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.281+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v122'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.281+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723d560 con 0x55e7170b8970 2024-05-09T03:20:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.282+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v122'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.283 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:20:35.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.282+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723e570 con 0x55e7170b8970 2024-05-09T03:20:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.284+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v122'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.284+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723e570 con 0x55e7170b8970 2024-05-09T03:20:35.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.285+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v122'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.285+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723d560 con 0x55e7170b8970 2024-05-09T03:20:35.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.286+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v122'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.286+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55e71723dbe0 0x55e717240040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.286+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240640 con 0x55e71723dbe0 2024-05-09T03:20:35.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.286+0000 7f90c3119640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55e71723dbe0 0x55e717240040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.287+0000 7f90c3119640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55e71723dbe0 0x55e717240040 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.288+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v122'415 uv409 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.288+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.289+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v122'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.289+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.290+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v122'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.291 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:20:35.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.291+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717241a20 con 0x55e71723dbe0 2024-05-09T03:20:35.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.292+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v122'418 uv418 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.292+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717241a20 con 0x55e71723dbe0 2024-05-09T03:20:35.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.293+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v122'419 uv419 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.293+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.294+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v122'420 uv420 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.294+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.295+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v122'421 uv414 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.295+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.296+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v122'422 uv422 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.296+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.297+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v122'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.298 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:20:35.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.297+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.298+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v122'424 uv424 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.299+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717240a10 con 0x55e71723dbe0 2024-05-09T03:20:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.300+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v122'425 uv425 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.300+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55e717241310 0x55e717243730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.300+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717243c70 con 0x55e717241310 2024-05-09T03:20:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.300+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55e717241310 0x55e717243730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.300+0000 7f90c2918640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55e717241310 0x55e717243730 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.302+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v122'188 uv187 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e717241310 2024-05-09T03:20:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.302+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717243cf0 con 0x55e717241310 2024-05-09T03:20:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.303+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v122'189 uv189 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e717241310 2024-05-09T03:20:35.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.303+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e717241310 2024-05-09T03:20:35.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.304+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v122'190 uv190 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e717241310 2024-05-09T03:20:35.304 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:20:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.304+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e717241310 2024-05-09T03:20:35.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.305+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v122'191 uv191 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e717241310 2024-05-09T03:20:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.305+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e717241310 2024-05-09T03:20:35.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.306+0000 7f90c2918640 1 -- 172.21.15.46:0/1589603839 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v122'192 uv192 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90b400d910 con 0x55e717241310 2024-05-09T03:20:35.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.306+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e7170b8970 2024-05-09T03:20:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.307+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v122'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.307+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e717244110 con 0x55e7170b8970 2024-05-09T03:20:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.308+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v122'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.308+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e7170b8970 2024-05-09T03:20:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.309+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v122'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.310 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:20:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.309+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717245120 con 0x55e7170b8970 2024-05-09T03:20:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.310+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v122'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.310+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717245120 con 0x55e7170b8970 2024-05-09T03:20:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.311+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v122'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.311+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e7170b8970 2024-05-09T03:20:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.312+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v122'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e7170b8970 2024-05-09T03:20:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.312+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244110 con 0x55e717239c20 2024-05-09T03:20:35.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.313+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v122'376 uv369 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.314+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172444b0 con 0x55e717239c20 2024-05-09T03:20:35.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.315+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v122'377 uv377 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.315+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717244850 con 0x55e717239c20 2024-05-09T03:20:35.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.316+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v122'378 uv378 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.317 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:20:35.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.316+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172450f0 con 0x55e717239c20 2024-05-09T03:20:35.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.317+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v122'379 uv379 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.317+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172450f0 con 0x55e717239c20 2024-05-09T03:20:35.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.318+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v122'380 uv380 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.318+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723d100 con 0x55e717239c20 2024-05-09T03:20:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.319+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v122'381 uv375 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.319+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e71723d100 con 0x55e717239c20 2024-05-09T03:20:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.320+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v122'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.320+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723d100 con 0x55e717239c20 2024-05-09T03:20:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.321+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v122'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.322 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:20:35.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.321+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e71723d100 con 0x55e717239c20 2024-05-09T03:20:35.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.322+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v122'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.322+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e717245100 con 0x55e717239c20 2024-05-09T03:20:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.324+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v122'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.324+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e717245a20 0x55e717248170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.324+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717245a20 2024-05-09T03:20:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.324+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e717245a20 0x55e717248170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.324+0000 7f90bbfff640 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e717245a20 0x55e717248170 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.325+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v122'167 uv165 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717245a20 2024-05-09T03:20:35.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.325+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717245a20 2024-05-09T03:20:35.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.326+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v122'168 uv168 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717245a20 2024-05-09T03:20:35.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.326+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717245a20 2024-05-09T03:20:35.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.327+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v122'169 uv169 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717245a20 2024-05-09T03:20:35.328 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:20:35.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.327+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717245a20 2024-05-09T03:20:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.328+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v122'170 uv170 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717245a20 2024-05-09T03:20:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.329+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717239c20 2024-05-09T03:20:35.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.330+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v122'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.330+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717239c20 2024-05-09T03:20:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.331+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v122'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.331+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717239c20 2024-05-09T03:20:35.333 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:20:35.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.332+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v122'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.332+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172496c0 con 0x55e717239c20 2024-05-09T03:20:35.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.333+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v122'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.333+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172496c0 con 0x55e717239c20 2024-05-09T03:20:35.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.334+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v122'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.334+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e717239c20 2024-05-09T03:20:35.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.335+0000 7f90bbfff640 1 -- 172.21.15.46:0/1589603839 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v122'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90ac070070 con 0x55e717239c20 2024-05-09T03:20:35.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.335+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.337+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v122'191 uv190 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.337+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.338+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v122'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.338+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.340 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:20:35.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.339+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v122'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.339+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.340+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v122'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.341+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.343+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v122'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.343+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.344+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v122'468 uv459 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.345+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.346+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v122'469 uv469 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.346+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.347+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v122'470 uv470 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.347 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:20:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.347+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.348+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v122'471 uv471 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.348+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.349+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v122'472 uv472 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.349+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.350+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v122'473 uv466 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.350+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.351+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v122'474 uv474 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.352+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.353+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v122'475 uv475 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.353 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:20:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.353+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172496c0 con 0x55e71723dbe0 2024-05-09T03:20:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.354+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v122'476 uv476 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.354+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172496c0 con 0x55e71723dbe0 2024-05-09T03:20:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.355+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v122'477 uv477 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.355+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.356+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v122'478 uv478 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.356+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55e7172486b0 con 0x55e71723dbe0 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v122'479 uv479 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f90bc011940 con 0x55e71723dbe0 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e7170b8970 msgr2=0x55e7170b8d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x55e7170b8970 0x55e7170b8d80 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55e717239c20 msgr2=0x55e71723c040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x55e717239c20 0x55e71723c040 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e717245a20 msgr2=0x55e717248170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55e717245a20 0x55e717248170 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e7172346b0 msgr2=0x55e717236ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x55e7172346b0 0x55e717236ad0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 reap_dead start 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55e71723dbe0 msgr2=0x55e717240040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55e71723dbe0 0x55e717240040 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55e717241310 msgr2=0x55e717243730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x55e717241310 0x55e717243730 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a4074ce0 msgr2=0x7f90a40771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a4074ce0 0x7f90a40771f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f90b4000f60 tx=0x7f90b4007840 comp rx=0 tx=0).stop 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7170e87b0 msgr2=0x55e7170fb370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 --2- 172.21.15.46:0/1589603839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7170e87b0 0x55e7170fb370 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f90ac045df0 tx=0x7f90ac002fb0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c3119640 1 -- 172.21.15.46:0/1589603839 reap_dead start 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 shutdown_connections 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 >> 172.21.15.46:0/1589603839 conn(0x55e716fdbac0 msgr2=0x55e7171893b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 shutdown_connections 2024-05-09T03:20:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.357+0000 7f90c991ef40 1 -- 172.21.15.46:0/1589603839 wait complete. 2024-05-09T03:20:35.361 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader mds0_sessionmap deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:20:35.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.458+0000 7f0e1e3d9f40 1 Processor -- start 2024-05-09T03:20:35.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.458+0000 7f0e1e3d9f40 1 -- start start 2024-05-09T03:20:35.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e1e3d9f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61657e0 0x55bca6233d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e1e3d9f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61977b0 0x55bca61a9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e1e3d9f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e1e3d9f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bca61ac4d0 con 0x55bca61ae010 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e1e3d9f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bca610ef50 con 0x55bca61977b0 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e1e3d9f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bca61b18a0 con 0x55bca61657e0 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53712/0 (socket says 172.21.15.46:53712) 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 -- 172.21.15.46:0/2716957514 learned_addr learned my addr 172.21.15.46:0/2716957514 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e17ea1640 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61657e0 0x55bca6233d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e176a0640 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61977b0 0x55bca61a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 -- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61657e0 msgr2=0x55bca6233d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61657e0 0x55bca6233d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 -- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61977b0 msgr2=0x55bca61a9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61977b0 0x55bca61a9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.459+0000 7f0e186a2640 1 -- 172.21.15.46:0/2716957514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bca6238910 con 0x55bca61ae010 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e186a2640 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62d0930 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f0e08001af0 tx=0x7f0e08006930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad51ff610f6dec1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1669e640 1 -- 172.21.15.46:0/2716957514 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08016070 con 0x55bca61ae010 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1669e640 1 -- 172.21.15.46:0/2716957514 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e08004540 con 0x55bca61ae010 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1669e640 1 -- 172.21.15.46:0/2716957514 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08004970 con 0x55bca61ae010 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2716957514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 msgr2=0x55bca62d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62d0930 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f0e08001af0 tx=0x7f0e08006930 comp rx=0 tx=0).stop 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2716957514 shutdown_connections 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62d0930 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61977b0 0x55bca61a9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2716957514 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61657e0 0x55bca6233d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2716957514 >> 172.21.15.46:0/2716957514 conn(0x55bca608aac0 msgr2=0x55bca61afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2716957514 shutdown_connections 2024-05-09T03:20:35.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.460+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2716957514 wait complete. 2024-05-09T03:20:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.461+0000 7f0e1e3d9f40 1 Processor -- start 2024-05-09T03:20:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.461+0000 7f0e1e3d9f40 1 -- start start 2024-05-09T03:20:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61657e0 0x55bca6231ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61977b0 0x55bca6232000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bca61b0a40 con 0x55bca61ae010 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bca61a8cd0 con 0x55bca61657e0 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bca623a710 con 0x55bca61977b0 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53720/0 (socket says 172.21.15.46:53720) 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 -- 172.21.15.46:0/2577266604 learned_addr learned my addr 172.21.15.46:0/2577266604 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e17ea1640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61657e0 0x55bca6231ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e176a0640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61977b0 0x55bca6232000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 -- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61977b0 msgr2=0x55bca6232000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bca61977b0 0x55bca6232000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 -- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61657e0 msgr2=0x55bca6231ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bca61657e0 0x55bca6231ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 -- 172.21.15.46:0/2577266604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bca6230d00 con 0x55bca61ae010 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e186a2640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62cd5b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f0e08004f60 tx=0x7f0e08001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1569c640 1 -- 172.21.15.46:0/2577266604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08016030 con 0x55bca61ae010 2024-05-09T03:20:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1569c640 1 -- 172.21.15.46:0/2577266604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e0801d430 con 0x55bca61ae010 2024-05-09T03:20:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bca6230fe0 con 0x55bca61ae010 2024-05-09T03:20:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.463+0000 7f0e1569c640 1 -- 172.21.15.46:0/2577266604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0801d800 con 0x55bca61ae010 2024-05-09T03:20:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.464+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bca62315a0 con 0x55bca61ae010 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.465+0000 7f0e1569c640 1 -- 172.21.15.46:0/2577266604 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0e0801d9a0 con 0x55bca61ae010 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e1569c640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0df8074cb0 0x7f0df80771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e1569c640 1 -- 172.21.15.46:0/2577266604 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f0e08098560 con 0x55bca61ae010 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e17ea1640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0df8074cb0 0x7f0df80771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55bca62cdaf0 0x55bca62d8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55bca62cdf00 con 0x55bca62cdaf0 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e176a0640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55bca62cdaf0 0x55bca62d8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.466+0000 7f0e17ea1640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0df8074cb0 0x7f0df80771c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0e10056000 tx=0x7f0e10050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.467+0000 7f0e176a0640 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55bca62cdaf0 0x55bca62d8990 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e176a0640 1 -- 172.21.15.46:0/2577266604 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(1 mds0_sessionmap [omap-set-header] v122'171 uv171 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0e00009930 con 0x55bca62cdaf0 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55bca62cdaf0 msgr2=0x55bca62d8990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x55bca62cdaf0 0x55bca62d8990 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0df8074cb0 msgr2=0x7f0df80771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0df8074cb0 0x7f0df80771c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0e10056000 tx=0x7f0e10050000 comp rx=0 tx=0).stop 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 msgr2=0x55bca62cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 --2- 172.21.15.46:0/2577266604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bca61ae010 0x55bca62cd5b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f0e08004f60 tx=0x7f0e08001250 comp rx=0 tx=0).stop 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e186a2640 1 -- 172.21.15.46:0/2577266604 reap_dead start 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 shutdown_connections 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 >> 172.21.15.46:0/2577266604 conn(0x55bca608aac0 msgr2=0x55bca623eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 shutdown_connections 2024-05-09T03:20:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.468+0000 7f0e1e3d9f40 1 -- 172.21.15.46:0/2577266604 wait complete. 2024-05-09T03:20:35.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:20:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.588+0000 7f31d1572640 1 Processor -- start 2024-05-09T03:20:35.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.588+0000 7f31d1572640 1 -- start start 2024-05-09T03:20:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.589+0000 7f31d1572640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 0x7f31cc1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.589+0000 7f31d1572640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31cc0d2a70 0x7f31cc0d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.589+0000 7f31d1572640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31cc0d33c0 con 0x7f31cc0bb770 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.589+0000 7f31d1572640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f31cc0d4bb0 con 0x7f31cc163eb0 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.589+0000 7f31d1572640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f31cc164900 con 0x7f31cc0d2a70 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31ca7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 0x7f31cc1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31ca7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 0x7f31cc1642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53736/0 (socket says 172.21.15.46:53736) 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31ca7fc640 1 -- 172.21.15.46:0/976025566 learned_addr learned my addr 172.21.15.46:0/976025566 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31cb7fe640 1 --2- 172.21.15.46:0/976025566 >> v2:172.21.15.88:3300/0 conn(0x7f31cc0d2a70 0x7f31cc0d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31cb7fe640 1 -- 172.21.15.46:0/976025566 >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 msgr2=0x7f31cc1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31cb7fe640 1 --2- 172.21.15.46:0/976025566 >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 0x7f31cc1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31cb7fe640 1 -- 172.21.15.46:0/976025566 >> v1:172.21.15.88:6789/0 conn(0x7f31cc0bb770 legacy=0x7f31cc0d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31cb7fe640 1 -- 172.21.15.46:0/976025566 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31cc164be0 con 0x7f31cc0d2a70 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31c9ffb640 1 -- 172.21.15.46:0/976025566 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2100331725 0 0) 0x7f31bc001110 con 0x7f31cc0bb770 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.590+0000 7f31ca7fc640 1 --2- 172.21.15.46:0/976025566 >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 0x7f31cc1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.591+0000 7f31cb7fe640 1 --2- 172.21.15.46:0/976025566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc0d2a70 0x7f31cc0d2e80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f31cc08e340 tx=0x7f31c402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=513640bb3cd0029a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.591+0000 7f31c9ffb640 1 -- 172.21.15.46:0/976025566 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c4007740 con 0x7f31cc0d2a70 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.591+0000 7f31c9ffb640 1 -- 172.21.15.46:0/976025566 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c40078e0 con 0x7f31cc0d2a70 2024-05-09T03:20:35.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.591+0000 7f31c9ffb640 1 -- 172.21.15.46:0/976025566 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c4007cb0 con 0x7f31cc0d2a70 2024-05-09T03:20:35.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 -- 172.21.15.46:0/976025566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc0d2a70 msgr2=0x7f31cc0d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 --2- 172.21.15.46:0/976025566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc0d2a70 0x7f31cc0d2e80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f31cc08e340 tx=0x7f31c402a680 comp rx=0 tx=0).stop 2024-05-09T03:20:35.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 -- 172.21.15.46:0/976025566 shutdown_connections 2024-05-09T03:20:35.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 --2- 172.21.15.46:0/976025566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc0d2a70 0x7f31cc0d2e80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 --2- 172.21.15.46:0/976025566 >> v2:172.21.15.46:3300/0 conn(0x7f31cc163eb0 0x7f31cc1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 -- 172.21.15.46:0/976025566 >> 172.21.15.46:0/976025566 conn(0x7f31cc06e370 msgr2=0x7f31cc06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 -- 172.21.15.46:0/976025566 shutdown_connections 2024-05-09T03:20:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.592+0000 7f31d1572640 1 -- 172.21.15.46:0/976025566 wait complete. 2024-05-09T03:20:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.594+0000 7f31d1572640 1 Processor -- start 2024-05-09T03:20:35.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.594+0000 7f31d1572640 1 -- start start 2024-05-09T03:20:35.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31d1572640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 0x7f31cc155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31d1572640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31cc0d2a70 0x7f31cc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31d1572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc163eb0 0x7f31cc1ecd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31d1572640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc0d0bc0 con 0x7f31cc0bb770 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31d1572640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc0bbb80 con 0x7f31cc163eb0 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31d1572640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f31cc0d0d40 con 0x7f31cc0d2a70 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 0x7f31cc155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 0x7f31cc155e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53748/0 (socket says 172.21.15.46:53748) 2024-05-09T03:20:35.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 -- 172.21.15.46:0/1480786951 learned_addr learned my addr 172.21.15.46:0/1480786951 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31cb7fe640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc163eb0 0x7f31cc1ecd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31ca7fc640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31cc0d2a70 0x7f31cc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 -- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31cc0d2a70 msgr2=0x7f31cc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31cc0d2a70 0x7f31cc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 -- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc163eb0 msgr2=0x7f31cc1ecd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc163eb0 0x7f31cc1ecd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.595+0000 7f31caffd640 1 -- 172.21.15.46:0/1480786951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31cc1563c0 con 0x7f31cc0bb770 2024-05-09T03:20:35.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.596+0000 7f31caffd640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 0x7f31cc155e80 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c3c0 tx=0x7f31bc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.596+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc017070 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.596+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31bc0035f0 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.596+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31cc15d610 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.596+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc0039c0 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.596+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31cc1619c0 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.597+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31cc15d7e0 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.597+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f31bc01a070 con 0x7f31cc0bb770 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.598+0000 7f31b37fe640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31b80762d0 0x7f31b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:35.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.598+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9035+0+0 (secure 0 0 0) 0x7f31bc0975e0 con 0x7f31cc0bb770 2024-05-09T03:20:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.598+0000 7f31ca7fc640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31b80762d0 0x7f31b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:35.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.599+0000 7f31ca7fc640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31b80762d0 0x7f31b80787e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f31cc1f9f00 tx=0x7f31c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:35.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.603+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f31cc1563c0 con 0x7f31cc0bb770 2024-05-09T03:20:35.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:35.833+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f31cc0d09a0 con 0x7f31cc0bb770 2024-05-09T03:20:36.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.098+0000 7f31b37fe640 1 -- 172.21.15.46:0/1480786951 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v334) ==== 162+0+0 (secure 0 0 0) 0x7f31cc0d33c0 con 0x7f31cc0bb770 2024-05-09T03:20:36.099 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:20:36.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31b80762d0 msgr2=0x7f31b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31b80762d0 0x7f31b80787e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f31cc1f9f00 tx=0x7f31c0008040 comp rx=0 tx=0).stop 2024-05-09T03:20:36.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 msgr2=0x7f31cc155e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 0x7f31cc155e80 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c3c0 tx=0x7f31bc00c880 comp rx=0 tx=0).stop 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 shutdown_connections 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31cc163eb0 0x7f31cc1ecd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f31cc0d2a70 0x7f31cc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.100+0000 7f31d1572640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31b80762d0 0x7f31b80787e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.101+0000 7f31d1572640 1 --2- 172.21.15.46:0/1480786951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31cc0bb770 0x7f31cc155e80 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.101+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 >> 172.21.15.46:0/1480786951 conn(0x7f31cc06e370 msgr2=0x7f31cc0b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.101+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 shutdown_connections 2024-05-09T03:20:36.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.101+0000 7f31d1572640 1 -- 172.21.15.46:0/1480786951 wait complete. 2024-05-09T03:20:36.108 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:36.108+0000 7ff093d58640 -1 log_channel(cluster) log [ERR] : corrupt sessionmap header: virtual void SessionMapStore::decode_header(ceph::bufferlist&) no longer understand old encoding version 1 < 101: Malformed input [buffer:3] 2024-05-09T03:20:36.109 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:36.108+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:20:36.109 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:36.108+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:20:36.114 INFO:tasks.ceph.mds.c.smithi046.stderr: -24> 2024-05-09T03:20:36.108+0000 7ff093d58640 -1 log_channel(cluster) log [ERR] : corrupt sessionmap header: virtual void SessionMapStore::decode_header(ceph::bufferlist&) no longer understand old encoding version 1 < 101: Malformed input [buffer:3] 2024-05-09T03:20:36.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:36.171 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.233+0000 7f81cfb84640 1 Processor -- start 2024-05-09T03:20:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.233+0000 7f81cfb84640 1 -- start start 2024-05-09T03:20:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cfb84640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c80b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cfb84640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f81c806de60 0x7f81c806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cfb84640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c806e7b0 con 0x7f81c80d34e0 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cfb84640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f81c808de80 con 0x7f81c806de60 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cfb84640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81c80d2440 con 0x7f81c80d1f60 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cd11f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f81c80d34e0 0x7f81c808d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42406/0 (socket says 172.21.15.46:42406) 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cd11f640 1 -- 172.21.15.46:0/95107776 learned_addr learned my addr 172.21.15.46:0/95107776 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81ce121640 1 --2- 172.21.15.46:0/95107776 >> v2:172.21.15.88:3300/0 conn(0x7f81c806de60 0x7f81c806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.234+0000 7f81cd920640 1 --2- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c80b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81cc91e640 1 -- 172.21.15.46:0/95107776 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940462924 0 0) 0x7f81bc001110 con 0x7f81c80d34e0 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81cc91e640 1 -- 172.21.15.46:0/95107776 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81b4003520 con 0x7f81c80d34e0 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81ce121640 1 -- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 msgr2=0x7f81c80b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81ce121640 1 --2- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c80b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81ce121640 1 -- 172.21.15.46:0/95107776 >> v1:172.21.15.46:6789/0 conn(0x7f81c80d34e0 legacy=0x7f81c808d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81cc91e640 1 -- 172.21.15.46:0/95107776 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 99439257 0 0) 0x7f81b4003520 con 0x7f81c80d34e0 2024-05-09T03:20:36.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.235+0000 7f81ce121640 1 -- 172.21.15.46:0/95107776 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c80d2720 con 0x7f81c806de60 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.236+0000 7f81ce121640 1 --2- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c806e270 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009d90 tx=0x7f81c002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15de2913013cf093 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.236+0000 7f81cc91e640 1 -- 172.21.15.46:0/95107776 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0018c0 con 0x7f81c806de60 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.236+0000 7f81cc91e640 1 -- 172.21.15.46:0/95107776 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c0007540 con 0x7f81c806de60 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.236+0000 7f81cc91e640 1 -- 172.21.15.46:0/95107776 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0007910 con 0x7f81c806de60 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.236+0000 7f81cfb84640 1 -- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 msgr2=0x7f81c806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.236+0000 7f81cfb84640 1 --2- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c806e270 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009d90 tx=0x7f81c002a5f0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.237+0000 7f81cfb84640 1 -- 172.21.15.46:0/95107776 shutdown_connections 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.237+0000 7f81cfb84640 1 --2- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c806e270 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.237+0000 7f81cfb84640 1 --2- 172.21.15.46:0/95107776 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c80b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.237+0000 7f81cfb84640 1 -- 172.21.15.46:0/95107776 >> 172.21.15.46:0/95107776 conn(0x7f81c80b56e0 msgr2=0x7f81c80b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.237+0000 7f81cfb84640 1 -- 172.21.15.46:0/95107776 shutdown_connections 2024-05-09T03:20:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.237+0000 7f81cfb84640 1 -- 172.21.15.46:0/95107776 wait complete. 2024-05-09T03:20:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.238+0000 7f81cfb84640 1 Processor -- start 2024-05-09T03:20:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.238+0000 7f81cfb84640 1 -- start start 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cfb84640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c80cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cfb84640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c8155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cfb84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 0x7f81c815d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cfb84640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81c80d2fc0 con 0x7f81c80d34e0 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cfb84640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81c80d2ce0 con 0x7f81c806de60 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cfb84640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81c80d38f0 con 0x7f81c80d1f60 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 0x7f81c815d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 0x7f81c815d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45972/0 (socket says 172.21.15.46:45972) 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 -- 172.21.15.46:0/4138813355 learned_addr learned my addr 172.21.15.46:0/4138813355 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cd11f640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c8155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81cd920640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c80cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 -- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 msgr2=0x7f81c8155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c8155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 -- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 msgr2=0x7f81c80cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c80cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 -- 172.21.15.46:0/4138813355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c80cc520 con 0x7f81c80d34e0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.240+0000 7f81ce121640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 0x7f81c815d910 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009d90 tx=0x7f81c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.241+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0040070 con 0x7f81c80d34e0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.241+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c80cc9a0 con 0x7f81c80d34e0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.241+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c0031bc0 con 0x7f81c80d34e0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.241+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0007e90 con 0x7f81c80d34e0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.241+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c80ccf60 con 0x7f81c80d34e0 2024-05-09T03:20:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.241+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c80cc6f0 con 0x7f81c80d34e0 2024-05-09T03:20:36.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.242+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81c0039040 con 0x7f81c80d34e0 2024-05-09T03:20:36.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.243+0000 7f81ba7fc640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81ac0762d0 0x7f81ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.243+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7f81c00b9250 con 0x7f81c80d34e0 2024-05-09T03:20:36.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.243+0000 7f81cd920640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81ac0762d0 0x7f81ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.244+0000 7f81cd920640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81ac0762d0 0x7f81ac0787e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f81c80c76f0 tx=0x7f81c4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.248+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f81c00817c0 con 0x7f81c80d34e0 2024-05-09T03:20:36.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.478+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81c80b9660 con 0x7f81c80d34e0 2024-05-09T03:20:36.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.479+0000 7f81ba7fc640 1 -- 172.21.15.46:0/4138813355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 336 v336) ==== 95+0+4343 (secure 0 0 0) 0x7f81c0085ac0 con 0x7f81c80d34e0 2024-05-09T03:20:36.480 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:36.480 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":336,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":336,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:36.109178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:36.480 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 336 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81ac0762d0 msgr2=0x7f81ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81ac0762d0 0x7f81ac0787e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f81c80c76f0 tx=0x7f81c4009190 comp rx=0 tx=0).stop 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 msgr2=0x7f81c815d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 0x7f81c815d910 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f81c0009d90 tx=0x7f81c0004a40 comp rx=0 tx=0).stop 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 shutdown_connections 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f81ac0762d0 0x7f81ac0787e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81c80d34e0 0x7f81c815d910 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81c80d1f60 0x7f81c8155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 --2- 172.21.15.46:0/4138813355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81c806de60 0x7f81c80cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 >> 172.21.15.46:0/4138813355 conn(0x7f81c80b56e0 msgr2=0x7f81c8075d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:36.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 shutdown_connections 2024-05-09T03:20:36.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.481+0000 7f81cfb84640 1 -- 172.21.15.46:0/4138813355 wait complete. 2024-05-09T03:20:36.493 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:20:36.493 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.608+0000 7ff9cd79d640 1 Processor -- start 2024-05-09T03:20:36.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.608+0000 7ff9cd79d640 1 -- start start 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.609+0000 7ff9cd79d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c80d14b0 0x7ff9c80b9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.609+0000 7ff9cd79d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9c80b9680 0x7ff9c80b7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9cd79d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c80d4b70 con 0x7ff9c8160450 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9cd79d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c8163e30 con 0x7ff9c80b9680 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9cd79d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c80d4e50 con 0x7ff9c80d14b0 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9c80b9680 0x7ff9c80b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9ccf9c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff9c8160450 0x7ff9c81642c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42408/0 (socket says 172.21.15.46:42408) 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9ccf9c640 1 -- 172.21.15.46:0/1415593336 learned_addr learned my addr 172.21.15.46:0/1415593336 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c7fff640 1 --2- 172.21.15.46:0/1415593336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c80d14b0 0x7ff9c80b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c77fe640 1 -- 172.21.15.46:0/1415593336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c80d14b0 msgr2=0x7ff9c80b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:36.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c77fe640 1 --2- 172.21.15.46:0/1415593336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c80d14b0 0x7ff9c80b9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c77fe640 1 -- 172.21.15.46:0/1415593336 >> v1:172.21.15.46:6789/0 conn(0x7ff9c8160450 legacy=0x7ff9c81642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c6ffd640 1 -- 172.21.15.46:0/1415593336 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2631667177 0 0) 0x7ff9c80d4b70 con 0x7ff9c8160450 2024-05-09T03:20:36.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.610+0000 7ff9c77fe640 1 -- 172.21.15.46:0/1415593336 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c80d5130 con 0x7ff9c80b9680 2024-05-09T03:20:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.611+0000 7ff9c77fe640 1 --2- 172.21.15.46:0/1415593336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c80b7670 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc0097a0 tx=0x7ff9bc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d90aeaf1536ad36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.611+0000 7ff9c6ffd640 1 -- 172.21.15.46:0/1415593336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc037070 con 0x7ff9c80b9680 2024-05-09T03:20:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.611+0000 7ff9c6ffd640 1 -- 172.21.15.46:0/1415593336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9bc032440 con 0x7ff9c80b9680 2024-05-09T03:20:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.611+0000 7ff9c6ffd640 1 -- 172.21.15.46:0/1415593336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc032810 con 0x7ff9c80b9680 2024-05-09T03:20:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 -- 172.21.15.46:0/1415593336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 msgr2=0x7ff9c80b7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/1415593336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c80b7670 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc0097a0 tx=0x7ff9bc02a180 comp rx=0 tx=0).stop 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 -- 172.21.15.46:0/1415593336 shutdown_connections 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/1415593336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c80b7670 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/1415593336 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c80d14b0 0x7ff9c80b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 -- 172.21.15.46:0/1415593336 >> 172.21.15.46:0/1415593336 conn(0x7ff9c80d2670 msgr2=0x7ff9c80d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 -- 172.21.15.46:0/1415593336 shutdown_connections 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.612+0000 7ff9cd79d640 1 -- 172.21.15.46:0/1415593336 wait complete. 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.614+0000 7ff9cd79d640 1 Processor -- start 2024-05-09T03:20:36.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.614+0000 7ff9cd79d640 1 -- start start 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9cd79d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c8154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9cd79d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9c80d14b0 0x7ff9c815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9cd79d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c8160450 0x7ff9c8161f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9cd79d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c80d4880 con 0x7ff9c80b9680 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9cd79d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c80d4700 con 0x7ff9c80d14b0 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9cd79d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c80b4e60 con 0x7ff9c8160450 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c8154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c8154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45982/0 (socket says 172.21.15.46:45982) 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9c7fff640 1 -- 172.21.15.46:0/2506000512 learned_addr learned my addr 172.21.15.46:0/2506000512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:36.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.615+0000 7ff9c77fe640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9c80d14b0 0x7ff9c815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c7fff640 1 -- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c8160450 msgr2=0x7ff9c8161f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9ccf9c640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c8160450 0x7ff9c8161f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c7fff640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c8160450 0x7ff9c8161f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c7fff640 1 -- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9c80d14b0 msgr2=0x7ff9c815a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c7fff640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9c80d14b0 0x7ff9c815a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c7fff640 1 -- 172.21.15.46:0/2506000512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c81609b0 con 0x7ff9c80b9680 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c77fe640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9c80d14b0 0x7ff9c815a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c7fff640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c8154ff0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800d600 tx=0x7ff9b800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8004240 con 0x7ff9c80b9680 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b80043e0 con 0x7ff9c80b9680 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c81619e0 con 0x7ff9c80b9680 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8003830 con 0x7ff9c80b9680 2024-05-09T03:20:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.616+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c8160f00 con 0x7ff9c80b9680 2024-05-09T03:20:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.617+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c8076c70 con 0x7ff9c80b9680 2024-05-09T03:20:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.618+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff9b80039d0 con 0x7ff9c80b9680 2024-05-09T03:20:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.618+0000 7ff9c4ff9640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b4076300 0x7ff9b4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.618+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7ff9b80986f0 con 0x7ff9c80b9680 2024-05-09T03:20:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.618+0000 7ff9c77fe640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b4076300 0x7ff9b4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.619+0000 7ff9c77fe640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b4076300 0x7ff9b4078810 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc034000 tx=0x7ff9bc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.623+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff9b8061190 con 0x7ff9c80b9680 2024-05-09T03:20:36.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.850+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9c80d3fd0 con 0x7ff9c80b9680 2024-05-09T03:20:36.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.850+0000 7ff9c4ff9640 1 -- 172.21.15.46:0/2506000512 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 336 v336) ==== 95+0+4343 (secure 0 0 0) 0x7ff9b8060a90 con 0x7ff9c80b9680 2024-05-09T03:20:36.851 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:36.851 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":336,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":336,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:36.109178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:36.852 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 336 2024-05-09T03:20:36.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b4076300 msgr2=0x7ff9b4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b4076300 0x7ff9b4078810 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc034000 tx=0x7ff9bc02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:36.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 msgr2=0x7ff9c8154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c8154ff0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7ff9b800d600 tx=0x7ff9b800dac0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 shutdown_connections 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff9c8160450 0x7ff9c8161f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9c80d14b0 0x7ff9c815a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff9b4076300 0x7ff9b4078810 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 --2- 172.21.15.46:0/2506000512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c80b9680 0x7ff9c8154ff0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.852+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 >> 172.21.15.46:0/2506000512 conn(0x7ff9c80d2670 msgr2=0x7ff9c80d5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.853+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 shutdown_connections 2024-05-09T03:20:36.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.853+0000 7ff9cd79d640 1 -- 172.21.15.46:0/2506000512 wait complete. 2024-05-09T03:20:36.865 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:20:36.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.979+0000 7ff4e0347640 1 Processor -- start 2024-05-09T03:20:36.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.979+0000 7ff4e0347640 1 -- start start 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4e0347640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4d80b6ae0 0x7ff4d80cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4e0347640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 0x7ff4d80ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4e0347640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d80b2fc0 con 0x7ff4d8160690 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4e0347640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d80b9600 con 0x7ff4d80b6ae0 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4e0347640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d80b32a0 con 0x7ff4d80ce200 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4dd8e2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff4d8160690 0x7ff4d8160aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42410/0 (socket says 172.21.15.46:42410) 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4dd8e2640 1 -- 172.21.15.46:0/4206779373 learned_addr learned my addr 172.21.15.46:0/4206779373 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:36.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4de0e3640 1 --2- 172.21.15.46:0/4206779373 >> v2:172.21.15.88:3300/0 conn(0x7ff4d80b6ae0 0x7ff4d80cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.980+0000 7ff4de8e4640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 0x7ff4d80ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3997855415 0 0) 0x7ff4d80b2fc0 con 0x7ff4d8160690 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4c0003520 con 0x7ff4d8160690 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 609889066 0 0) 0x7ff4c0003520 con 0x7ff4d8160690 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 msgr2=0x7ff4d80ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 0x7ff4d80ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 >> v2:172.21.15.88:3300/0 conn(0x7ff4d80b6ae0 msgr2=0x7ff4d80cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 --2- 172.21.15.46:0/4206779373 >> v2:172.21.15.88:3300/0 conn(0x7ff4d80b6ae0 0x7ff4d80cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d80b8fb0 con 0x7ff4d8160690 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7ff4cc0016f0 con 0x7ff4d8160690 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 >> v1:172.21.15.46:6789/0 conn(0x7ff4d8160690 legacy=0x7ff4d8160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0003a00 0x7ff4c0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 0x7ff4c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c0004d40 0x7ff4c0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d80b9600 con 0x7ff4c0004d40 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d80b32a0 con 0x7ff4c0003a00 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c000ba70 con 0x7ff4d80ce200 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de8e4640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c0004d40 0x7ff4c0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de0e3640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0003a00 0x7ff4c0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4dd8e2640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 0x7ff4c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de8e4640 1 -- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 msgr2=0x7ff4c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de8e4640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d80ce200 0x7ff4c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de8e4640 1 -- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0003a00 msgr2=0x7ff4c0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de8e4640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0003a00 0x7ff4c0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.981+0000 7ff4de8e4640 1 -- 172.21.15.46:0/4206779373 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c0003520 con 0x7ff4c0004d40 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.982+0000 7ff4de8e4640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c0004d40 0x7ff4c0005150 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80d37c0 tx=0x7ff4c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6977113e6a3fabfe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.982+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8037070 con 0x7ff4c0004d40 2024-05-09T03:20:36.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.982+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c8032620 con 0x7ff4c0004d40 2024-05-09T03:20:36.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.982+0000 7ff4dd0e1640 1 -- 172.21.15.46:0/4206779373 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8009c10 con 0x7ff4c0004d40 2024-05-09T03:20:36.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4e0347640 1 -- 172.21.15.46:0/4206779373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c0004d40 msgr2=0x7ff4c0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4e0347640 1 --2- 172.21.15.46:0/4206779373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c0004d40 0x7ff4c0005150 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80d37c0 tx=0x7ff4c0005830 comp rx=0 tx=0).stop 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4de8e4640 1 -- 172.21.15.46:0/4206779373 reap_dead start 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4e0347640 1 -- 172.21.15.46:0/4206779373 shutdown_connections 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4e0347640 1 -- 172.21.15.46:0/4206779373 >> 172.21.15.46:0/4206779373 conn(0x7ff4d80d0630 msgr2=0x7ff4d80d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4e0347640 1 -- 172.21.15.46:0/4206779373 shutdown_connections 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.983+0000 7ff4e0347640 1 -- 172.21.15.46:0/4206779373 wait complete. 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.985+0000 7ff4e0347640 1 Processor -- start 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.985+0000 7ff4e0347640 1 -- start start 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4e0347640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0004d40 0x7ff4d80c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4e0347640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 0x7ff4d80c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4e0347640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d8150400 0x7ff4d80c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4e0347640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d8161330 con 0x7ff4d814fff0 2024-05-09T03:20:36.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4e0347640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d81611b0 con 0x7ff4c0004d40 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4e0347640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d80ce610 con 0x7ff4d8150400 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4dd8e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 0x7ff4d80c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4dd8e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 0x7ff4d80c7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46010/0 (socket says 172.21.15.46:46010) 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.986+0000 7ff4dd8e2640 1 -- 172.21.15.46:0/536543666 learned_addr learned my addr 172.21.15.46:0/536543666 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4de0e3640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0004d40 0x7ff4d80c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4de8e4640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d8150400 0x7ff4d80c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4dd8e2640 1 -- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d8150400 msgr2=0x7ff4d80c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4dd8e2640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d8150400 0x7ff4d80c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4dd8e2640 1 -- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0004d40 msgr2=0x7ff4d80c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4dd8e2640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0004d40 0x7ff4d80c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4dd8e2640 1 -- 172.21.15.46:0/536543666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d80cd1b0 con 0x7ff4d814fff0 2024-05-09T03:20:36.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4dd8e2640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 0x7ff4d80c7a60 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc007310 tx=0x7ff4cc007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00f9c0 con 0x7ff4d814fff0 2024-05-09T03:20:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4cc018070 con 0x7ff4d814fff0 2024-05-09T03:20:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d80cd490 con 0x7ff4d814fff0 2024-05-09T03:20:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc00fd80 con 0x7ff4d814fff0 2024-05-09T03:20:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.987+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d80cda50 con 0x7ff4d814fff0 2024-05-09T03:20:36.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.988+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d806a270 con 0x7ff4d814fff0 2024-05-09T03:20:36.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.989+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff4cc01d460 con 0x7ff4d814fff0 2024-05-09T03:20:36.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.989+0000 7ff4d6ffd640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff4c4076300 0x7ff4c4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:36.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.990+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7ff4cc098430 con 0x7ff4d814fff0 2024-05-09T03:20:36.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.990+0000 7ff4de0e3640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff4c4076300 0x7ff4c4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:36.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.990+0000 7ff4de0e3640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff4c4076300 0x7ff4c4078810 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009bc0 tx=0x7ff4d0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:36.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:36.994+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff4cc060f50 con 0x7ff4d814fff0 2024-05-09T03:20:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.224+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4d80ccf40 con 0x7ff4d814fff0 2024-05-09T03:20:37.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.225+0000 7ff4d6ffd640 1 -- 172.21.15.46:0/536543666 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 337 v337) ==== 95+0+5170 (secure 0 0 0) 0x7ff4cc060850 con 0x7ff4d814fff0 2024-05-09T03:20:37.226 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:37.226 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":337,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10416,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1514520479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1514520479},{"type":"v1","addr":"172.21.15.46:6834","nonce":1514520479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":337}],"filesystems":[{"mdsmap":{"epoch":336,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:36.109178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 337 2024-05-09T03:20:37.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff4c4076300 msgr2=0x7ff4c4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff4c4076300 0x7ff4c4078810 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009bc0 tx=0x7ff4d0009190 comp rx=0 tx=0).stop 2024-05-09T03:20:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 msgr2=0x7ff4d80c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 0x7ff4d80c7a60 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc007310 tx=0x7ff4cc007710 comp rx=0 tx=0).stop 2024-05-09T03:20:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 shutdown_connections 2024-05-09T03:20:37.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff4d8150400 0x7ff4d80c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff4c4076300 0x7ff4c4078810 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4d814fff0 0x7ff4d80c7a60 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 --2- 172.21.15.46:0/536543666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c0004d40 0x7ff4d80c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 >> 172.21.15.46:0/536543666 conn(0x7ff4d80d0630 msgr2=0x7ff4d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 shutdown_connections 2024-05-09T03:20:37.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.227+0000 7ff4e0347640 1 -- 172.21.15.46:0/536543666 wait complete. 2024-05-09T03:20:37.242 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on mds0_sessionmap' led to DAMAGED state 2024-05-09T03:20:37.242 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 1.00000000' 2024-05-09T03:20:37.242 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:37.242 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:37.259 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:37.259 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-05-09T03:20:37.259 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:20:37.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:20:37.259 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:20:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:20:37.317 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-09T03:20:37.317 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:20:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.473+0000 7f2fce584640 1 Processor -- start 2024-05-09T03:20:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.473+0000 7f2fce584640 1 -- start start 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fce584640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2fc80b4700 0x7f2fc80d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fce584640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fc80b4120 0x7f2fc80ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fce584640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fc806ebe0 con 0x7f2fc80d2630 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fce584640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2fc80b4530 con 0x7f2fc80b4120 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fce584640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2fc806eec0 con 0x7f2fc80b4700 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fccb21640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fc80b4120 0x7f2fc80ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fbffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2fc80d2630 0x7f2fc80b3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42426/0 (socket says 172.21.15.46:42426) 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fbffff640 1 -- 172.21.15.46:0/704180052 learned_addr learned my addr 172.21.15.46:0/704180052 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.474+0000 7f2fccb21640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fc80b4120 0x7f2fc80ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46022/0 (socket says 172.21.15.46:46022) 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.475+0000 7f2fc7fff640 1 --2- 172.21.15.46:0/704180052 >> v2:172.21.15.88:3300/0 conn(0x7f2fc80b4700 0x7f2fc80d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.475+0000 7f2fccb21640 1 -- 172.21.15.46:0/704180052 >> v2:172.21.15.88:3300/0 conn(0x7f2fc80b4700 msgr2=0x7f2fc80d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.475+0000 7f2fccb21640 1 --2- 172.21.15.46:0/704180052 >> v2:172.21.15.88:3300/0 conn(0x7f2fc80b4700 0x7f2fc80d20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.475+0000 7f2fccb21640 1 -- 172.21.15.46:0/704180052 >> v1:172.21.15.46:6789/0 conn(0x7f2fc80d2630 legacy=0x7f2fc80b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.475+0000 7f2fccb21640 1 -- 172.21.15.46:0/704180052 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc806f1a0 con 0x7f2fc80b4120 2024-05-09T03:20:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.475+0000 7f2fccb21640 1 --2- 172.21.15.46:0/704180052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80b4120 0x7f2fc80ce130 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8070130 tx=0x7f2fc002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cc5ebe29915848 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fc77fe640 1 -- 172.21.15.46:0/704180052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0007740 con 0x7f2fc80b4120 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fc77fe640 1 -- 172.21.15.46:0/704180052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc00078e0 con 0x7f2fc80b4120 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fc77fe640 1 -- 172.21.15.46:0/704180052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0007cb0 con 0x7f2fc80b4120 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 -- 172.21.15.46:0/704180052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80b4120 msgr2=0x7f2fc80ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 --2- 172.21.15.46:0/704180052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80b4120 0x7f2fc80ce130 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8070130 tx=0x7f2fc002a560 comp rx=0 tx=0).stop 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 -- 172.21.15.46:0/704180052 shutdown_connections 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 --2- 172.21.15.46:0/704180052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80b4120 0x7f2fc80ce130 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 --2- 172.21.15.46:0/704180052 >> v2:172.21.15.88:3300/0 conn(0x7f2fc80b4700 0x7f2fc80d20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 -- 172.21.15.46:0/704180052 >> 172.21.15.46:0/704180052 conn(0x7f2fc80b55d0 msgr2=0x7f2fc80b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 -- 172.21.15.46:0/704180052 shutdown_connections 2024-05-09T03:20:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.476+0000 7f2fce584640 1 -- 172.21.15.46:0/704180052 wait complete. 2024-05-09T03:20:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.478+0000 7f2fce584640 1 Processor -- start 2024-05-09T03:20:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.478+0000 7f2fce584640 1 -- start start 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fce584640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fc80b4120 0x7f2fc81d7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fce584640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fc80b4700 0x7f2fc80a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fce584640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 0x7f2fc80a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fce584640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc80d2a40 con 0x7f2fc80d2630 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fce584640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc80d2db0 con 0x7f2fc80b4120 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fce584640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc80b51d0 con 0x7f2fc80b4700 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 0x7f2fc80a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 0x7f2fc80a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46038/0 (socket says 172.21.15.46:46038) 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 -- 172.21.15.46:0/2784746505 learned_addr learned my addr 172.21.15.46:0/2784746505 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fc7fff640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fc80b4120 0x7f2fc81d7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fbffff640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fc80b4700 0x7f2fc80a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 -- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fc80b4700 msgr2=0x7f2fc80a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fc80b4700 0x7f2fc80a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 -- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fc80b4120 msgr2=0x7f2fc81d7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fc80b4120 0x7f2fc81d7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 -- 172.21.15.46:0/2784746505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc809ea40 con 0x7f2fc80d2630 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fccb21640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 0x7f2fc80a0b20 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009e50 tx=0x7f2fc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc003f070 con 0x7f2fc80d2630 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.480+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc0031bf0 con 0x7f2fc80d2630 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.481+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc809ed20 con 0x7f2fc80d2630 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.481+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0044460 con 0x7f2fc80d2630 2024-05-09T03:20:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.481+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc809f000 con 0x7f2fc80d2630 2024-05-09T03:20:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.482+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc809e7b0 con 0x7f2fc80d2630 2024-05-09T03:20:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.482+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2fc0007790 con 0x7f2fc80d2630 2024-05-09T03:20:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.483+0000 7f2fc57fa640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fa80762d0 0x7f2fa80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.483+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7f2fc00b9b50 con 0x7f2fc80d2630 2024-05-09T03:20:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.483+0000 7f2fc7fff640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fa80762d0 0x7f2fa80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.484+0000 7f2fc7fff640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fa80762d0 0x7f2fa80787e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80079a0 tx=0x7f2fb8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.488+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2fc00820b0 con 0x7f2fc80d2630 2024-05-09T03:20:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:37.720+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2fc80d13d0 con 0x7f2fc80d2630 2024-05-09T03:20:37.970 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:20:37.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:20:38.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.122+0000 7f2fc57fa640 1 -- 172.21.15.46:0/2784746505 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v338) ==== 161+0+0 (secure 0 0 0) 0x7f2fc00863b0 con 0x7f2fc80d2630 2024-05-09T03:20:38.123 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fa80762d0 msgr2=0x7f2fa80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fa80762d0 0x7f2fa80787e0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80079a0 tx=0x7f2fb8008040 comp rx=0 tx=0).stop 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 msgr2=0x7f2fc80a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 0x7f2fc80a0b20 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009e50 tx=0x7f2fc0004a40 comp rx=0 tx=0).stop 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 shutdown_connections 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2fa80762d0 0x7f2fa80787e0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fc80d2630 0x7f2fc80a0b20 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2fc80b4700 0x7f2fc80a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 --2- 172.21.15.46:0/2784746505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2fc80b4120 0x7f2fc81d7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 >> 172.21.15.46:0/2784746505 conn(0x7f2fc80b55d0 msgr2=0x7f2fc80729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:38.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 shutdown_connections 2024-05-09T03:20:38.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.125+0000 7f2fce584640 1 -- 172.21.15.46:0/2784746505 wait complete. 2024-05-09T03:20:38.138 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-05-09T03:20:38.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.254+0000 7f6ec5cee640 1 Processor -- start 2024-05-09T03:20:38.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.254+0000 7f6ec5cee640 1 -- start start 2024-05-09T03:20:38.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.255+0000 7f6ec5cee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ec006e2f0 0x7f6ec006e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:38.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.255+0000 7f6ec5cee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 0x7f6ec00d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:38.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.255+0000 7f6ec5cee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec00b91f0 con 0x7f6ec00d2930 2024-05-09T03:20:38.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.255+0000 7f6ec5cee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6ec00d1320 con 0x7f6ec006e2f0 2024-05-09T03:20:38.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.255+0000 7f6ec5cee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6ec00d1570 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 0x7f6ec00d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:38.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 0x7f6ec00d23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:58302/0 (socket says 172.21.15.46:58302) 2024-05-09T03:20:38.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6ec00d2930 0x7f6ec00ceba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34738/0 (socket says 172.21.15.46:34738) 2024-05-09T03:20:38.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebf7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ec006e2f0 0x7f6ec006e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:38.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebeffd640 1 -- 172.21.15.46:0/2641076456 learned_addr learned my addr 172.21.15.46:0/2641076456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebf7fe640 1 -- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 msgr2=0x7f6ec00d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebf7fe640 1 --2- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 0x7f6ec00d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebf7fe640 1 -- 172.21.15.46:0/2641076456 >> v1:172.21.15.88:6789/0 conn(0x7f6ec00d2930 legacy=0x7f6ec00ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebe7fc640 1 -- 172.21.15.46:0/2641076456 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1789546056 0 0) 0x7f6ec00b91f0 con 0x7f6ec00d2930 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebf7fe640 1 -- 172.21.15.46:0/2641076456 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec009ceb0 con 0x7f6ec006e2f0 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.256+0000 7f6ebeffd640 1 --2- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 0x7f6ec00d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.257+0000 7f6ebf7fe640 1 --2- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec006e2f0 0x7f6ec006e700 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009d90 tx=0x7f6eb002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2d90a42a300b98c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.257+0000 7f6ebe7fc640 1 -- 172.21.15.46:0/2641076456 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0007740 con 0x7f6ec006e2f0 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.257+0000 7f6ebe7fc640 1 -- 172.21.15.46:0/2641076456 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eb00078e0 con 0x7f6ec006e2f0 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.257+0000 7f6ebe7fc640 1 -- 172.21.15.46:0/2641076456 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0007cb0 con 0x7f6ec006e2f0 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 -- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec006e2f0 msgr2=0x7f6ec006e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec006e2f0 0x7f6ec006e700 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009d90 tx=0x7f6eb002a660 comp rx=0 tx=0).stop 2024-05-09T03:20:38.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 -- 172.21.15.46:0/2641076456 shutdown_connections 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec00d1fe0 0x7f6ec00d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/2641076456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec006e2f0 0x7f6ec006e700 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 -- 172.21.15.46:0/2641076456 >> 172.21.15.46:0/2641076456 conn(0x7f6ec00d05f0 msgr2=0x7f6ec00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 -- 172.21.15.46:0/2641076456 shutdown_connections 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.258+0000 7f6ec5cee640 1 -- 172.21.15.46:0/2641076456 wait complete. 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.260+0000 7f6ec5cee640 1 Processor -- start 2024-05-09T03:20:38.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.260+0000 7f6ec5cee640 1 -- start start 2024-05-09T03:20:38.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ec5cee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec006e2f0 0x7f6ec01e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:38.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ec5cee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 0x7f6ec01de6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ec5cee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec00d2930 0x7f6ec01e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ec5cee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00b9460 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ec5cee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00d3220 con 0x7f6ec00d2930 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ec5cee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6ec00b99a0 con 0x7f6ec006e2f0 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ebeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 0x7f6ec01de6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ebeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 0x7f6ec01de6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46040/0 (socket says 172.21.15.46:46040) 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ebeffd640 1 -- 172.21.15.46:0/4126482774 learned_addr learned my addr 172.21.15.46:0/4126482774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ebf7fe640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec006e2f0 0x7f6ec01e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.261+0000 7f6ebffff640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec00d2930 0x7f6ec01e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebeffd640 1 -- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec006e2f0 msgr2=0x7f6ec01e8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebeffd640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec006e2f0 0x7f6ec01e8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebeffd640 1 -- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec00d2930 msgr2=0x7f6ec01e1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:38.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebeffd640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec00d2930 0x7f6ec01e1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebeffd640 1 -- 172.21.15.46:0/4126482774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec01e03c0 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebf7fe640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec006e2f0 0x7f6ec01e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebffff640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec00d2930 0x7f6ec01e1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ebeffd640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 0x7f6ec01de6f0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800e9e0 tx=0x7f6ea800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8017070 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea800c9d0 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea800cd80 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec01e06a0 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.262+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec01e96b0 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.263+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec006a270 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.264+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ea801a070 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.264+0000 7f6eb7fff640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ea00762d0 0x7f6ea00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:38.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.265+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7f6ea8098680 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.265+0000 7f6ebf7fe640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ea00762d0 0x7f6ea00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:38.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.265+0000 7f6ebf7fe640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ea00762d0 0x7f6ea00787e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0034000 tx=0x7f6eb0033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:38.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.269+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6ea80611d0 con 0x7f6ec00d1fe0 2024-05-09T03:20:38.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:38.538+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f6ec00d2710 con 0x7f6ec00d1fe0 2024-05-09T03:20:39.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.129+0000 7f6eb7fff640 1 -- 172.21.15.46:0/4126482774 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v339) ==== 101+0+0 (secure 0 0 0) 0x7f6ea8060ad0 con 0x7f6ec00d1fe0 2024-05-09T03:20:39.130 INFO:teuthology.orchestra.run.smithi046.stderr:repaired: restoring rank 6:0 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ea00762d0 msgr2=0x7f6ea00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ea00762d0 0x7f6ea00787e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0034000 tx=0x7f6eb0033040 comp rx=0 tx=0).stop 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 msgr2=0x7f6ec01de6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 0x7f6ec01de6f0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800e9e0 tx=0x7f6ea800eea0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 shutdown_connections 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ec00d2930 0x7f6ec01e1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6ea00762d0 0x7f6ea00787e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec00d1fe0 0x7f6ec01de6f0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 --2- 172.21.15.46:0/4126482774 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f6ec006e2f0 0x7f6ec01e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 >> 172.21.15.46:0/4126482774 conn(0x7f6ec00d05f0 msgr2=0x7f6ec01e5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 shutdown_connections 2024-05-09T03:20:39.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.132+0000 7f6ec5cee640 1 -- 172.21.15.46:0/4126482774 wait complete. 2024-05-09T03:20:39.146 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-05-09T03:20:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.206+0000 7fcb107b7f40 1 Processor -- start 2024-05-09T03:20:39.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.206+0000 7fcb107b7f40 1 -- start start 2024-05-09T03:20:39.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb107b7f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b49ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb107b7f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 0x556978abbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb107b7f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978b66780 0x556978b4a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb107b7f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556978ad0cb0 con 0x556978abddb0 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb107b7f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556978a37f50 con 0x556978a8e7e0 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb107b7f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556978a8ebf0 con 0x556978b66780 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 0x556978abbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 0x556978abbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46072/0 (socket says 172.21.15.46:46072) 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb03fff640 1 -- 172.21.15.46:0/582558928 learned_addr learned my addr 172.21.15.46:0/582558928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb0a281640 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b49ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.207+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978b66780 0x556978b4a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb03fff640 1 -- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978b66780 msgr2=0x556978b4a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:39.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb03fff640 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978b66780 0x556978b4a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb03fff640 1 -- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 msgr2=0x556978b49ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb03fff640 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b49ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb03fff640 1 -- 172.21.15.46:0/582558928 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556978b49870 con 0x556978abddb0 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978b66780 0x556978b4a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb03fff640 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 0x556978abbb60 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009780 tx=0x7fcaf4005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed818f1ff3b48502 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb0927f640 1 -- 172.21.15.46:0/582558928 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4008870 con 0x556978abddb0 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb0927f640 1 -- 172.21.15.46:0/582558928 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf4018070 con 0x556978abddb0 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.208+0000 7fcb0927f640 1 -- 172.21.15.46:0/582558928 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4008c30 con 0x556978abddb0 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 -- 172.21.15.46:0/582558928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 msgr2=0x556978abbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 0x556978abbb60 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009780 tx=0x7fcaf4005a00 comp rx=0 tx=0).stop 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 -- 172.21.15.46:0/582558928 shutdown_connections 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978b66780 0x556978b4a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978abddb0 0x556978abbb60 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/582558928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b49ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 -- 172.21.15.46:0/582558928 >> 172.21.15.46:0/582558928 conn(0x5569789b3ac0 msgr2=0x556978abf800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 -- 172.21.15.46:0/582558928 shutdown_connections 2024-05-09T03:20:39.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.209+0000 7fcb107b7f40 1 -- 172.21.15.46:0/582558928 wait complete. 2024-05-09T03:20:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.210+0000 7fcb107b7f40 1 Processor -- start 2024-05-09T03:20:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.210+0000 7fcb107b7f40 1 -- start start 2024-05-09T03:20:39.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb107b7f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb107b7f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978abddb0 0x556978b64950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb107b7f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 0x556978b64e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb107b7f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556978b49ae0 con 0x556978b66780 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb107b7f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556978b4ed10 con 0x556978a8e7e0 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb107b7f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x556978ad91c0 con 0x556978abddb0 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb0aa82640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 0x556978b64e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb0aa82640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 0x556978b64e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46082/0 (socket says 172.21.15.46:46082) 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 learned_addr learned my addr 172.21.15.46:0/2576542100 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978abddb0 0x556978b64950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.211+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978abddb0 msgr2=0x556978b64950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x556978abddb0 0x556978b64950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 msgr2=0x556978b478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556978a8e7e0 0x556978b478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556978abfb90 con 0x556978b66780 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 0x556978b64e90 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fcb04004e20 tx=0x7fcb040018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb037fe640 1 -- 172.21.15.46:0/2576542100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04018070 con 0x556978b66780 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb037fe640 1 -- 172.21.15.46:0/2576542100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb040102e0 con 0x556978b66780 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556978b66240 con 0x556978b66780 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb037fe640 1 -- 172.21.15.46:0/2576542100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb040106b0 con 0x556978b66780 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.212+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556978b66520 con 0x556978b66780 2024-05-09T03:20:39.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.214+0000 7fcb037fe640 1 -- 172.21.15.46:0/2576542100 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcb0401b070 con 0x556978b66780 2024-05-09T03:20:39.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.214+0000 7fcb037fe640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcaec074cb0 0x7fcaec0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.214+0000 7fcb037fe640 1 -- 172.21.15.46:0/2576542100 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7fcb04096590 con 0x556978b66780 2024-05-09T03:20:39.217 INFO:teuthology.orchestra.run.smithi046.stdout:Importing pool 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.215+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcaec074cb0 0x7fcaec0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.215+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556978ad8a50 0x556978c019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.215+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c01f20 con 0x556978ad8a50 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.215+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556978ad8a50 0x556978c019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.215+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcaec074cb0 0x7fcaec0771c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fcafc056000 tx=0x7fcafc050000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.215+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556978ad8a50 0x556978c019e0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.218+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(1 601.00000000 [create] v123'1190 uv1164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.218+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c0a6c0 con 0x556978ad8a50 2024-05-09T03:20:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.219+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(2 601.00000000 [delete] v123'1191 uv1191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.219+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0aa60 con 0x556978ad8a50 2024-05-09T03:20:39.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.221+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(3 601.00000000 [create] v123'1192 uv1192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.222 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-05-09T03:20:39.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.221+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0be10 con 0x556978ad8a50 2024-05-09T03:20:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.222+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v123'1193 uv1193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.223+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0c1b0 con 0x556978ad8a50 2024-05-09T03:20:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.224+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v123'1194 uv1194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.224+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0ae00 con 0x556978ad8a50 2024-05-09T03:20:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.225+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v123'1195 uv1195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.225+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b1a0 con 0x556978ad8a50 2024-05-09T03:20:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.227+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 7 ==== osd_op_reply(7 602.00000000 [create] v123'1196 uv1170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.227+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.229+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 8 ==== osd_op_reply(8 602.00000000 [delete] v123'1197 uv1197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.229+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.230+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 9 ==== osd_op_reply(9 602.00000000 [create] v123'1198 uv1198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.231 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-05-09T03:20:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.230+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.232+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v123'1199 uv1199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.232+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.233+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v123'1200 uv1200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.233+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.235+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v123'1201 uv1201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.235+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.236+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 13 ==== osd_op_reply(13 10000000000.00000000 [create] v123'1202 uv1177 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.236+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.238+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v123'1203 uv1203 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.238+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.240+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 15 ==== osd_op_reply(15 10000000000.00000000 [create] v123'1204 uv1204 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.240 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-05-09T03:20:39.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.240+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.241+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v123'1205 uv1205 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.241+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.243+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v123'1206 uv1206 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.243+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.244+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v123'1207 uv1207 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.244+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=524b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.246+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v123'1208 uv1208 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.246+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.248+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 20 ==== osd_op_reply(20 600.00000000 [create] v123'1209 uv1183 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.248+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.249+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 21 ==== osd_op_reply(21 600.00000000 [delete] v123'1210 uv1210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.249+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.250+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 22 ==== osd_op_reply(22 600.00000000 [create] v123'1211 uv1211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.251 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-05-09T03:20:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.250+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.252+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v123'1212 uv1212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.252+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.253+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v123'1213 uv1213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.253+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.255+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v123'1214 uv1214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.255+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.256+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 26 ==== osd_op_reply(26 603.00000000 [create] v123'1215 uv1189 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.256+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.258+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 27 ==== osd_op_reply(27 603.00000000 [delete] v123'1216 uv1216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.258+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.259+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 28 ==== osd_op_reply(28 603.00000000 [create] v123'1217 uv1217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.260 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-05-09T03:20:39.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.259+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.261+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v123'1218 uv1218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.261+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.262+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v123'1219 uv1219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.262+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978ad8a50 2024-05-09T03:20:39.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.264+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v123'1220 uv1220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.264+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556978c0b940 0x556978c0ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.264+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0b540 con 0x556978c0b940 2024-05-09T03:20:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.264+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556978c0b940 0x556978c0ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.265+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556978c0b940 0x556978c0ea70 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.267+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(32 1.00000000.inode [create] v123'858 uv841 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.267+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c0efb0 con 0x556978c0b940 2024-05-09T03:20:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.268+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== osd_op_reply(33 1.00000000.inode [delete] v123'859 uv859 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.268+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0f350 con 0x556978c0b940 2024-05-09T03:20:39.269 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-05-09T03:20:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.269+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 3 ==== osd_op_reply(34 1.00000000.inode [create] v123'860 uv860 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.269+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0fba0 con 0x556978c0b940 2024-05-09T03:20:39.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.270+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v123'861 uv861 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.270+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c0ff40 con 0x556978c0b940 2024-05-09T03:20:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.272+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v123'862 uv862 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.272+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c102e0 con 0x556978c0b940 2024-05-09T03:20:39.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.273+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 6 ==== osd_op_reply(37 200.00000000 [create] v123'863 uv846 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.273+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c10680 con 0x556978c0b940 2024-05-09T03:20:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.274+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 7 ==== osd_op_reply(38 200.00000000 [delete] v123'864 uv864 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.274+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.275+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 8 ==== osd_op_reply(39 200.00000000 [create] v123'865 uv865 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.276 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-05-09T03:20:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.276+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.277+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v123'866 uv866 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.277+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.278+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v123'867 uv867 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.278+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.279+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 11 ==== osd_op_reply(42 200.00000001 [create] v123'868 uv851 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.279+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.281+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 12 ==== osd_op_reply(43 200.00000001 [delete] v123'869 uv869 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.281+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.282+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 13 ==== osd_op_reply(44 200.00000001 [create] v123'870 uv870 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.282 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-05-09T03:20:39.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.282+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~14895 in=14895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c14480 con 0x556978c0b940 2024-05-09T03:20:39.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.283+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 14 ==== osd_op_reply(45 200.00000001 [write 0~14895] v123'871 uv871 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.283+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c14480 con 0x556978c0b940 2024-05-09T03:20:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.285+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v123'872 uv872 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.285+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.286+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 16 ==== osd_op_reply(47 606.00000000 [create] v123'873 uv857 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.286+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.287+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 17 ==== osd_op_reply(48 606.00000000 [delete] v123'874 uv874 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.287+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.288+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 18 ==== osd_op_reply(49 606.00000000 [create] v123'875 uv875 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.289 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-05-09T03:20:39.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.288+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c11a30 con 0x556978c0b940 2024-05-09T03:20:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.289+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v123'876 uv876 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.289+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c11a30 con 0x556978c0b940 2024-05-09T03:20:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.291+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v123'877 uv877 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.291+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978c0b940 2024-05-09T03:20:39.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.292+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.5 v2:172.21.15.88:6800/122581025 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v123'878 uv878 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c0b940 2024-05-09T03:20:39.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.292+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978ad8a50 2024-05-09T03:20:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.293+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 32 ==== osd_op_reply(53 607.00000000 [create] v123'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.293+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978ad8a50 2024-05-09T03:20:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.294+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 33 ==== osd_op_reply(54 607.00000000 [delete] v123'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.294+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978ad8a50 2024-05-09T03:20:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.295+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 34 ==== osd_op_reply(55 607.00000000 [create] v123'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.296 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-05-09T03:20:39.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.295+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c11a30 con 0x556978ad8a50 2024-05-09T03:20:39.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.296+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v123'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.296+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c11a30 con 0x556978ad8a50 2024-05-09T03:20:39.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.297+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v123'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.297+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c10a20 con 0x556978ad8a50 2024-05-09T03:20:39.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.298+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v123'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.299+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556978c110a0 0x556978c134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.299+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c13a90 con 0x556978c110a0 2024-05-09T03:20:39.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.299+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556978c110a0 0x556978c134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.299+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556978c110a0 0x556978c134c0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.300+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v123'176 uv175 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.300+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c13e80 con 0x556978c110a0 2024-05-09T03:20:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.301+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v123'177 uv177 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.301+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c13e80 con 0x556978c110a0 2024-05-09T03:20:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.302+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v123'178 uv178 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.303 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-05-09T03:20:39.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.302+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c13e80 con 0x556978c110a0 2024-05-09T03:20:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.303+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v123'179 uv179 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.303+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c142a0 con 0x556978ad8a50 2024-05-09T03:20:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.305+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 38 ==== osd_op_reply(63 608.00000000 [create] v123'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.305+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c142a0 con 0x556978ad8a50 2024-05-09T03:20:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.306+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 39 ==== osd_op_reply(64 608.00000000 [delete] v123'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.306+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c13e30 con 0x556978ad8a50 2024-05-09T03:20:39.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.307+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 40 ==== osd_op_reply(65 608.00000000 [create] v123'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.308 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-05-09T03:20:39.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.307+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c13e30 con 0x556978ad8a50 2024-05-09T03:20:39.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.309+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v123'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.309+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c15310 con 0x556978ad8a50 2024-05-09T03:20:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.310+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v123'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.310+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c13ee0 con 0x556978ad8a50 2024-05-09T03:20:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.311+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v123'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.311+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556978c14920 0x556978c14d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.311+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c172a0 con 0x556978c14920 2024-05-09T03:20:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.311+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556978c14920 0x556978c14d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.312+0000 7fcb0aa82640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556978c14920 0x556978c14d30 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.313+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(69 604.00000000 [create] v123'426 uv420 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.313+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c17640 con 0x556978c14920 2024-05-09T03:20:39.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.314+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== osd_op_reply(70 604.00000000 [delete] v123'427 uv427 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.314+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c17a20 con 0x556978c14920 2024-05-09T03:20:39.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.315+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 3 ==== osd_op_reply(71 604.00000000 [create] v123'428 uv428 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.316 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-05-09T03:20:39.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.316+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c192d0 con 0x556978c14920 2024-05-09T03:20:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.317+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v123'429 uv429 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.317+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c19670 con 0x556978c14920 2024-05-09T03:20:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.318+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v123'430 uv430 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.318+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c182c0 con 0x556978c14920 2024-05-09T03:20:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.319+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v123'431 uv431 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.319+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c186d0 con 0x556978c14920 2024-05-09T03:20:39.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.320+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 7 ==== osd_op_reply(75 500.00000000 [create] v123'432 uv425 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.320+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c18ac0 con 0x556978c14920 2024-05-09T03:20:39.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.321+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 8 ==== osd_op_reply(76 500.00000000 [delete] v123'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.321+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c18ac0 con 0x556978c14920 2024-05-09T03:20:39.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.323+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 9 ==== osd_op_reply(77 500.00000000 [create] v123'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.323 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-05-09T03:20:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.323+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c18a70 con 0x556978c14920 2024-05-09T03:20:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.324+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v123'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.324+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c18a70 con 0x556978c14920 2024-05-09T03:20:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.325+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v123'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.325+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556978c18e60 0x556978c1bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.325+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c470 con 0x556978c18e60 2024-05-09T03:20:39.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.326+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556978c18e60 0x556978c1bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.326+0000 7fcb0a281640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556978c18e60 0x556978c1bf30 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.327+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(80 mds_snaptable [create] v123'193 uv192 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c18e60 2024-05-09T03:20:39.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.327+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1c470 con 0x556978c18e60 2024-05-09T03:20:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.328+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(81 mds_snaptable [delete] v123'194 uv194 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c18e60 2024-05-09T03:20:39.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.328+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c470 con 0x556978c18e60 2024-05-09T03:20:39.330 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-05-09T03:20:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.329+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.7 v2:172.21.15.88:6816/3368726188 3 ==== osd_op_reply(82 mds_snaptable [create] v123'195 uv195 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c18e60 2024-05-09T03:20:39.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.329+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c18e60 2024-05-09T03:20:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.331+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.7 v2:172.21.15.88:6816/3368726188 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v123'196 uv196 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c18e60 2024-05-09T03:20:39.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.331+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c18e60 2024-05-09T03:20:39.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.332+0000 7fcb0a281640 1 -- 172.21.15.46:0/2576542100 <== osd.7 v2:172.21.15.88:6816/3368726188 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v123'197 uv197 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcafc05c040 con 0x556978c18e60 2024-05-09T03:20:39.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.332+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978ad8a50 2024-05-09T03:20:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.333+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 44 ==== osd_op_reply(85 605.00000000 [create] v123'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.333+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978ad8a50 2024-05-09T03:20:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.334+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 45 ==== osd_op_reply(86 605.00000000 [delete] v123'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.334+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978ad8a50 2024-05-09T03:20:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.335+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 46 ==== osd_op_reply(87 605.00000000 [create] v123'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.336 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-05-09T03:20:39.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.335+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1d870 con 0x556978ad8a50 2024-05-09T03:20:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.336+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v123'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.336+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1d870 con 0x556978ad8a50 2024-05-09T03:20:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.337+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v123'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.337+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978ad8a50 2024-05-09T03:20:39.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.338+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.0 v2:172.21.15.46:6800/1118951116 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v123'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978ad8a50 2024-05-09T03:20:39.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.339+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.340+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 5 ==== osd_op_reply(91 mds0_inotable [create] v123'386 uv380 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.340+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.341+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 6 ==== osd_op_reply(92 mds0_inotable [delete] v123'387 uv387 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.341+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.342+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 7 ==== osd_op_reply(93 mds0_inotable [create] v123'388 uv388 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.343 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-05-09T03:20:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.342+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.343+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v123'389 uv389 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.343+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.344+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v123'390 uv390 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.344+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.345+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 10 ==== osd_op_reply(96 100.00000000 [create] v123'391 uv385 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.345+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.346+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 11 ==== osd_op_reply(97 100.00000000 [delete] v123'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.346+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.347+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 12 ==== osd_op_reply(98 100.00000000 [create] v123'393 uv393 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.348 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-05-09T03:20:39.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.347+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1c860 con 0x556978c110a0 2024-05-09T03:20:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.348+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v123'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.348+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f670 con 0x556978c110a0 2024-05-09T03:20:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.349+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v123'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.350+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556978c1d220 0x556978c1fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.350+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c1d220 2024-05-09T03:20:39.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.350+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556978c1d220 0x556978c1fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.350+0000 7fcb03fff640 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556978c1d220 0x556978c1fb90 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.351+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== osd_op_reply(101 mds0_sessionmap [create] v123'172 uv171 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c1d220 2024-05-09T03:20:39.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.351+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c1d220 2024-05-09T03:20:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.352+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v123'173 uv173 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c1d220 2024-05-09T03:20:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.352+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c1d220 2024-05-09T03:20:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.353+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.3 v2:172.21.15.46:6816/3177037259 3 ==== osd_op_reply(103 mds0_sessionmap [create] v123'174 uv174 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c1d220 2024-05-09T03:20:39.354 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-05-09T03:20:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.353+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c1d220 2024-05-09T03:20:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.354+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.3 v2:172.21.15.46:6816/3177037259 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v123'175 uv175 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c1d220 2024-05-09T03:20:39.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.354+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c110a0 2024-05-09T03:20:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.356+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 15 ==== osd_op_reply(105 609.00000000 [create] v123'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.356+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c110a0 2024-05-09T03:20:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.357+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 16 ==== osd_op_reply(106 609.00000000 [delete] v123'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.357+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c110a0 2024-05-09T03:20:39.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.358+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 17 ==== osd_op_reply(107 609.00000000 [create] v123'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.359 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-05-09T03:20:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.358+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c110a0 2024-05-09T03:20:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.359+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v123'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.360+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c110a0 2024-05-09T03:20:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.361+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v123'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.361+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c110a0 2024-05-09T03:20:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.362+0000 7fcb03fff640 1 -- 172.21.15.46:0/2576542100 <== osd.1 v2:172.21.15.46:6824/1118146182 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v123'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf4008770 con 0x556978c110a0 2024-05-09T03:20:39.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.362+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.363+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 12 ==== osd_op_reply(111 400.00000000 [create] v123'196 uv195 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.363+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.365+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 13 ==== osd_op_reply(112 400.00000000 [delete] v123'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.365+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.366 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-05-09T03:20:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.366+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 14 ==== osd_op_reply(113 400.00000000 [create] v123'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.366+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.367+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v123'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.367+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.368+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v123'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.369+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.370+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 17 ==== osd_op_reply(116 100.00000000.inode [create] v123'480 uv472 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.370+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.371+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 18 ==== osd_op_reply(117 100.00000000.inode [delete] v123'481 uv481 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.371+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.373 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-05-09T03:20:39.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.372+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 19 ==== osd_op_reply(118 100.00000000.inode [create] v123'482 uv482 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.372+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.374+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v123'483 uv483 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.374+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.375+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v123'484 uv484 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.375+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.376+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 22 ==== osd_op_reply(121 1.00000000 [create] v123'485 uv479 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.376+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.377+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 23 ==== osd_op_reply(122 1.00000000 [delete] v123'486 uv486 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.377+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.378+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 24 ==== osd_op_reply(123 1.00000000 [create] v123'487 uv487 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.379 INFO:teuthology.orchestra.run.smithi046.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-05-09T03:20:39.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.378+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.380+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v123'488 uv488 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.380+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.381+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v123'489 uv489 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.381+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c1f640 con 0x556978c14920 2024-05-09T03:20:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.382+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v123'490 uv490 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.382+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=523b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x556978c200d0 con 0x556978c14920 2024-05-09T03:20:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 <== osd.6 v2:172.21.15.88:6808/516824467 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v123'491 uv491 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fcb0401a030 con 0x556978c14920 2024-05-09T03:20:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556978ad8a50 msgr2=0x556978c019e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x556978ad8a50 0x556978c019e0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556978c110a0 msgr2=0x556978c134c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x556978c110a0 0x556978c134c0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556978c1d220 msgr2=0x556978c1fb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x556978c1d220 0x556978c1fb90 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556978c0b940 msgr2=0x556978c0ea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x556978c0b940 0x556978c0ea70 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 reap_dead start 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556978c14920 msgr2=0x556978c14d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x556978c14920 0x556978c14d30 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556978c18e60 msgr2=0x556978c1bf30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb0aa82640 1 -- 172.21.15.46:0/2576542100 reap_dead start 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556978c18e60 0x556978c1bf30 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcaec074cb0 msgr2=0x7fcaec0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcaec074cb0 0x7fcaec0771c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fcafc056000 tx=0x7fcafc050000 comp rx=0 tx=0).stop 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 msgr2=0x556978b64e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 0x556978b64e90 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fcb04004e20 tx=0x7fcb040018a0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 shutdown_connections 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x556978c18e60 0x556978c1bf30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcaec074cb0 0x7fcaec0771c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 --2- 172.21.15.46:0/2576542100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556978b66780 0x556978b64e90 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 >> 172.21.15.46:0/2576542100 conn(0x5569789b3ac0 msgr2=0x556978b49c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 shutdown_connections 2024-05-09T03:20:39.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.383+0000 7fcb107b7f40 1 -- 172.21.15.46:0/2576542100 wait complete. 2024-05-09T03:20:39.387 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 1.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-05-09T03:20:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.483+0000 7f599e316f40 1 Processor -- start 2024-05-09T03:20:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.483+0000 7f599e316f40 1 -- start start 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f599e316f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 0x55b7b8580930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f599e316f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b856c7b0 0x55b7b8585160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f599e316f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b8583010 0x55b7b857e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f599e316f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b7b85814d0 con 0x55b7b853a7e0 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f599e316f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b7b84e3f50 con 0x55b7b8583010 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f599e316f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b7b85868a0 con 0x55b7b856c7b0 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.484+0000 7f5997de0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 0x55b7b8580930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 0x55b7b8580930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46098/0 (socket says 172.21.15.46:46098) 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 -- 172.21.15.46:0/3214984582 learned_addr learned my addr 172.21.15.46:0/3214984582 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f59985e1640 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b8583010 0x55b7b857e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f59975df640 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b856c7b0 0x55b7b8585160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 -- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b856c7b0 msgr2=0x55b7b8585160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b856c7b0 0x55b7b8585160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 -- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b8583010 msgr2=0x55b7b857e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b8583010 0x55b7b857e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 -- 172.21.15.46:0/3214984582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b8608690 con 0x55b7b853a7e0 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f5997de0640 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 0x55b7b8580930 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f5988057040 tx=0x7f598804e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e7c1c92b29307b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f59965dd640 1 -- 172.21.15.46:0/3214984582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988059040 con 0x55b7b853a7e0 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f59965dd640 1 -- 172.21.15.46:0/3214984582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5988054930 con 0x55b7b853a7e0 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.485+0000 7f59965dd640 1 -- 172.21.15.46:0/3214984582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988054ce0 con 0x55b7b853a7e0 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 -- 172.21.15.46:0/3214984582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 msgr2=0x55b7b8580930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 0x55b7b8580930 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f5988057040 tx=0x7f598804e000 comp rx=0 tx=0).stop 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 -- 172.21.15.46:0/3214984582 shutdown_connections 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b8583010 0x55b7b857e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b856c7b0 0x55b7b8585160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 --2- 172.21.15.46:0/3214984582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b853a7e0 0x55b7b8580930 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 -- 172.21.15.46:0/3214984582 >> 172.21.15.46:0/3214984582 conn(0x55b7b845fac0 msgr2=0x55b7b8584a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 -- 172.21.15.46:0/3214984582 shutdown_connections 2024-05-09T03:20:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.486+0000 7f599e316f40 1 -- 172.21.15.46:0/3214984582 wait complete. 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.487+0000 7f599e316f40 1 Processor -- start 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.487+0000 7f599e316f40 1 -- start start 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f599e316f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b853a7e0 0x55b7b853e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f599e316f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b856c7b0 0x55b7b86067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f599e316f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b8583010 0x55b7b8584a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f599e316f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b7b8580e70 con 0x55b7b8583010 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f599e316f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b7b86088b0 con 0x55b7b856c7b0 2024-05-09T03:20:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f599e316f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55b7b860f6c0 con 0x55b7b853a7e0 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f59985e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b8583010 0x55b7b8584a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f59985e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b8583010 0x55b7b8584a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46114/0 (socket says 172.21.15.46:46114) 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f59985e1640 1 -- 172.21.15.46:0/1326389237 learned_addr learned my addr 172.21.15.46:0/1326389237 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f5997de0640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b853a7e0 0x55b7b853e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.488+0000 7f59975df640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b856c7b0 0x55b7b86067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59985e1640 1 -- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b853a7e0 msgr2=0x55b7b853e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59985e1640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55b7b853a7e0 0x55b7b853e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59985e1640 1 -- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b856c7b0 msgr2=0x55b7b86067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59985e1640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b7b856c7b0 0x55b7b86067d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59985e1640 1 -- 172.21.15.46:0/1326389237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b853f160 con 0x55b7b8583010 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59985e1640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b8583010 0x55b7b8584a50 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f599000e430 tx=0x7f599000e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59955db640 1 -- 172.21.15.46:0/1326389237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990016020 con 0x55b7b8583010 2024-05-09T03:20:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59955db640 1 -- 172.21.15.46:0/1326389237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5990012600 con 0x55b7b8583010 2024-05-09T03:20:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b7b8606e60 con 0x55b7b8583010 2024-05-09T03:20:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f59955db640 1 -- 172.21.15.46:0/1326389237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59900129d0 con 0x55b7b8583010 2024-05-09T03:20:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.489+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b8607420 con 0x55b7b8583010 2024-05-09T03:20:39.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.491+0000 7f59955db640 1 -- 172.21.15.46:0/1326389237 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5990019020 con 0x55b7b8583010 2024-05-09T03:20:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.491+0000 7f59955db640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5984074ce0 0x7f59840771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.491+0000 7f59955db640 1 -- 172.21.15.46:0/1326389237 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7f59900968b0 con 0x55b7b8583010 2024-05-09T03:20:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.491+0000 7f5997de0640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5984074ce0 0x7f59840771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.491+0000 7f599e316f40 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b7b86031b0 0x55b7b86035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.492+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x55b7b8603b00 con 0x55b7b86031b0 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.492+0000 7f59975df640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b7b86031b0 0x55b7b86035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.492+0000 7f5997de0640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5984074ce0 0x7f59840771f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5988065000 tx=0x7f5988051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.492+0000 7f59975df640 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b7b86031b0 0x55b7b86035c0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f59975df640 1 -- 172.21.15.46:0/1326389237 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== osd_op_reply(1 1.00000000 [omap-set-header] v123'492 uv492 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f598c009b70 con 0x55b7b86031b0 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b7b86031b0 msgr2=0x55b7b86035c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x55b7b86031b0 0x55b7b86035c0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5984074ce0 msgr2=0x7f59840771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5984074ce0 0x7f59840771f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5988065000 tx=0x7f5988051000 comp rx=0 tx=0).stop 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b8583010 msgr2=0x55b7b8584a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 --2- 172.21.15.46:0/1326389237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b8583010 0x55b7b8584a50 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7f599000e430 tx=0x7f599000e8f0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f59985e1640 1 -- 172.21.15.46:0/1326389237 reap_dead start 2024-05-09T03:20:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 shutdown_connections 2024-05-09T03:20:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 >> 172.21.15.46:0/1326389237 conn(0x55b7b845fac0 msgr2=0x55b7b856bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 shutdown_connections 2024-05-09T03:20:39.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.494+0000 7f599e316f40 1 -- 172.21.15.46:0/1326389237 wait complete. 2024-05-09T03:20:39.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:20:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.618+0000 7f88782e0640 1 Processor -- start 2024-05-09T03:20:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.618+0000 7f88782e0640 1 -- start start 2024-05-09T03:20:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f88782e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88700bb770 0x7f88700d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f88782e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8870163eb0 0x7f88701642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f88782e0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8870164900 con 0x7f88700d2a70 2024-05-09T03:20:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f88782e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f88700d4bb0 con 0x7f88700bb770 2024-05-09T03:20:39.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f88782e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8870164be0 con 0x7f8870163eb0 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f887607c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88700bb770 0x7f88700d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f887607c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88700bb770 0x7f88700d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46118/0 (socket says 172.21.15.46:46118) 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.619+0000 7f887607c640 1 -- 172.21.15.46:0/2635609731 learned_addr learned my addr 172.21.15.46:0/2635609731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.620+0000 7f887587b640 1 --2- 172.21.15.46:0/2635609731 >> v2:172.21.15.88:3300/0 conn(0x7f8870163eb0 0x7f88701642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.620+0000 7f887607c640 1 -- 172.21.15.46:0/2635609731 >> v2:172.21.15.88:3300/0 conn(0x7f8870163eb0 msgr2=0x7f88701642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.620+0000 7f887607c640 1 --2- 172.21.15.46:0/2635609731 >> v2:172.21.15.88:3300/0 conn(0x7f8870163eb0 0x7f88701642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.620+0000 7f887607c640 1 -- 172.21.15.46:0/2635609731 >> v1:172.21.15.88:6789/0 conn(0x7f88700d2a70 legacy=0x7f88700d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.620+0000 7f887607c640 1 -- 172.21.15.46:0/2635609731 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88700d4840 con 0x7f88700bb770 2024-05-09T03:20:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.620+0000 7f887607c640 1 --2- 172.21.15.46:0/2635609731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700bb770 0x7f88700d5040 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f88700d6d00 tx=0x7f886c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01eacdf1b57dde6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f887507a640 1 -- 172.21.15.46:0/2635609731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c037070 con 0x7f88700bb770 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f887507a640 1 -- 172.21.15.46:0/2635609731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f886c032440 con 0x7f88700bb770 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f887507a640 1 -- 172.21.15.46:0/2635609731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886c032810 con 0x7f88700bb770 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f88782e0640 1 -- 172.21.15.46:0/2635609731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700bb770 msgr2=0x7f88700d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f88782e0640 1 --2- 172.21.15.46:0/2635609731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700bb770 0x7f88700d5040 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f88700d6d00 tx=0x7f886c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f88782e0640 1 -- 172.21.15.46:0/2635609731 shutdown_connections 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f88782e0640 1 --2- 172.21.15.46:0/2635609731 >> v2:172.21.15.88:3300/0 conn(0x7f8870163eb0 0x7f88701642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f88782e0640 1 --2- 172.21.15.46:0/2635609731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88700bb770 0x7f88700d5040 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.621+0000 7f88782e0640 1 -- 172.21.15.46:0/2635609731 >> 172.21.15.46:0/2635609731 conn(0x7f887006e370 msgr2=0x7f887006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.622+0000 7f88782e0640 1 -- 172.21.15.46:0/2635609731 shutdown_connections 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.622+0000 7f88782e0640 1 -- 172.21.15.46:0/2635609731 wait complete. 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.623+0000 7f88782e0640 1 Processor -- start 2024-05-09T03:20:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.623+0000 7f88782e0640 1 -- start start 2024-05-09T03:20:39.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88700bb770 0x7f88700ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88700d2a70 0x7f88700cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 0x7f887014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f88700d0bc0 con 0x7f8870163eb0 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f88700bbb80 con 0x7f88700d2a70 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f88700d0d40 con 0x7f88700bb770 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 0x7f887014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 0x7f887014f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46120/0 (socket says 172.21.15.46:46120) 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 -- 172.21.15.46:0/1371254254 learned_addr learned my addr 172.21.15.46:0/1371254254 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887587b640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88700d2a70 0x7f88700cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887607c640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88700bb770 0x7f88700ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 -- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88700bb770 msgr2=0x7f88700ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88700bb770 0x7f88700ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 -- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88700d2a70 msgr2=0x7f88700cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88700d2a70 0x7f88700cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 -- 172.21.15.46:0/1371254254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88701501b0 con 0x7f8870163eb0 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f887687d640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 0x7f887014f310 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f886800b9e0 tx=0x7f886800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868003b70 con 0x7f8870163eb0 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8868003d10 con 0x7f8870163eb0 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8870150490 con 0x7f8870163eb0 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886800cb60 con 0x7f8870163eb0 2024-05-09T03:20:39.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.625+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8870150a50 con 0x7f8870163eb0 2024-05-09T03:20:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.626+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887006a270 con 0x7f8870163eb0 2024-05-09T03:20:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.627+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f886800cd00 con 0x7f8870163eb0 2024-05-09T03:20:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.628+0000 7f8866ffd640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88540762d0 0x7f88540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.628+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7f8868098130 con 0x7f8870163eb0 2024-05-09T03:20:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.628+0000 7f887607c640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88540762d0 0x7f88540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:39.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.629+0000 7f887607c640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88540762d0 0x7f88540787e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f886c034000 tx=0x7f886c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:39.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.633+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8868060690 con 0x7f8870163eb0 2024-05-09T03:20:39.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:39.860+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f887014ff40 con 0x7f8870163eb0 2024-05-09T03:20:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.014+0000 7f8866ffd640 1 -- 172.21.15.46:0/1371254254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v340) ==== 162+0+0 (secure 0 0 0) 0x7f8868064990 con 0x7f8870163eb0 2024-05-09T03:20:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:20:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88540762d0 msgr2=0x7f88540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88540762d0 0x7f88540787e0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f886c034000 tx=0x7f886c02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 msgr2=0x7f887014f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 0x7f887014f310 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f886800b9e0 tx=0x7f886800bea0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 shutdown_connections 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f88540762d0 0x7f88540787e0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8870163eb0 0x7f887014f310 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88700d2a70 0x7f88700cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 --2- 172.21.15.46:0/1371254254 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f88700bb770 0x7f88700ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 >> 172.21.15.46:0/1371254254 conn(0x7f887006e370 msgr2=0x7f88700bac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 shutdown_connections 2024-05-09T03:20:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.017+0000 7f88782e0640 1 -- 172.21.15.46:0/1371254254 wait complete. 2024-05-09T03:20:40.031 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:40.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.147+0000 7fcf8a9dc640 1 Processor -- start 2024-05-09T03:20:40.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.147+0000 7fcf8a9dc640 1 -- start start 2024-05-09T03:20:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf8a9dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf84162610 0x7fcf84162a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf8a9dc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf840b4ee0 con 0x7fcf840b9200 2024-05-09T03:20:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf8a9dc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf840b62e0 con 0x7fcf84162f60 2024-05-09T03:20:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf8a9dc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcf840b5460 con 0x7fcf84162610 2024-05-09T03:20:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf837fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf84162610 0x7fcf84162a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:40.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf837fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf84162610 0x7fcf84162a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46124/0 (socket says 172.21.15.46:46124) 2024-05-09T03:20:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf83fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcf840b9200 0x7fcf8408d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42450/0 (socket says 172.21.15.46:42450) 2024-05-09T03:20:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.148+0000 7fcf837fe640 1 -- 172.21.15.46:0/3379570742 learned_addr learned my addr 172.21.15.46:0/3379570742 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.149+0000 7fcf837fe640 1 -- 172.21.15.46:0/3379570742 >> v1:172.21.15.88:6789/0 conn(0x7fcf84162f60 legacy=0x7fcf840b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.149+0000 7fcf837fe640 1 -- 172.21.15.46:0/3379570742 >> v1:172.21.15.46:6789/0 conn(0x7fcf840b9200 legacy=0x7fcf8408d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.149+0000 7fcf837fe640 1 -- 172.21.15.46:0/3379570742 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf840b5180 con 0x7fcf84162610 2024-05-09T03:20:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.149+0000 7fcf837fe640 1 --2- 172.21.15.46:0/3379570742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf84162610 0x7fcf84162a20 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009870 tx=0x7fcf7802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbb258df051edd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.149+0000 7fcf82ffd640 1 -- 172.21.15.46:0/3379570742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78037070 con 0x7fcf84162610 2024-05-09T03:20:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf82ffd640 1 -- 172.21.15.46:0/3379570742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf78032440 con 0x7fcf84162610 2024-05-09T03:20:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf82ffd640 1 -- 172.21.15.46:0/3379570742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78032810 con 0x7fcf84162610 2024-05-09T03:20:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3379570742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf84162610 msgr2=0x7fcf84162a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3379570742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf84162610 0x7fcf84162a20 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009870 tx=0x7fcf7802a270 comp rx=0 tx=0).stop 2024-05-09T03:20:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3379570742 shutdown_connections 2024-05-09T03:20:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3379570742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf84162610 0x7fcf84162a20 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3379570742 >> 172.21.15.46:0/3379570742 conn(0x7fcf840b56e0 msgr2=0x7fcf840b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3379570742 shutdown_connections 2024-05-09T03:20:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.150+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3379570742 wait complete. 2024-05-09T03:20:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.152+0000 7fcf8a9dc640 1 Processor -- start 2024-05-09T03:20:40.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.152+0000 7fcf8a9dc640 1 -- start start 2024-05-09T03:20:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf8a9dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 0x7fcf840ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf8a9dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf84162610 0x7fcf840b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf8a9dc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf84162f60 0x7fcf840b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf8a9dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf840cf750 con 0x7fcf840b9200 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf8a9dc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf840cfac0 con 0x7fcf84162610 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf8a9dc640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcf8408d8b0 con 0x7fcf84162f60 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 0x7fcf840ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 0x7fcf840ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46134/0 (socket says 172.21.15.46:46134) 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 -- 172.21.15.46:0/3937978833 learned_addr learned my addr 172.21.15.46:0/3937978833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf837fe640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf84162610 0x7fcf840b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf88f79640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf84162f60 0x7fcf840b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 -- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf84162f60 msgr2=0x7fcf840b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf84162f60 0x7fcf840b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 -- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf84162610 msgr2=0x7fcf840b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf84162610 0x7fcf840b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 -- 172.21.15.46:0/3937978833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf840cd1c0 con 0x7fcf840b9200 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.154+0000 7fcf83fff640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 0x7fcf840ccc80 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fcf74001eb0 tx=0x7fcf7400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.155+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf740074e0 con 0x7fcf840b9200 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.155+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf74007680 con 0x7fcf840b9200 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.155+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8406ec90 con 0x7fcf840b9200 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.155+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf740049e0 con 0x7fcf840b9200 2024-05-09T03:20:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.155+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf8406e6d0 con 0x7fcf840b9200 2024-05-09T03:20:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.156+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf8406a270 con 0x7fcf840b9200 2024-05-09T03:20:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.156+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf7401a020 con 0x7fcf840b9200 2024-05-09T03:20:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.157+0000 7fcf80ff9640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf70076300 0x7fcf70078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.157+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9121+0+0 (secure 0 0 0) 0x7fcf74097a90 con 0x7fcf840b9200 2024-05-09T03:20:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.158+0000 7fcf837fe640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf70076300 0x7fcf70078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.158+0000 7fcf837fe640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf70076300 0x7fcf70078810 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcf78034000 tx=0x7fcf7802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.162+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcf740605b0 con 0x7fcf840b9200 2024-05-09T03:20:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.389+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf840cd3f0 con 0x7fcf840b9200 2024-05-09T03:20:40.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.390+0000 7fcf80ff9640 1 -- 172.21.15.46:0/3937978833 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 340 v340) ==== 95+0+5181 (secure 0 0 0) 0x7fcf7405feb0 con 0x7fcf840b9200 2024-05-09T03:20:40.391 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:40.391 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":340,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":340,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:40.013904+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10416},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10416":{"gid":10416,"name":"c","rank":0,"incarnation":340,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6834/1514520479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1514520479},{"type":"v1","addr":"172.21.15.46:6834","nonce":1514520479}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:40.391 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 340 2024-05-09T03:20:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf70076300 msgr2=0x7fcf70078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf70076300 0x7fcf70078810 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcf78034000 tx=0x7fcf7802e000 comp rx=0 tx=0).stop 2024-05-09T03:20:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 msgr2=0x7fcf840ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 0x7fcf840ccc80 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fcf74001eb0 tx=0x7fcf7400c890 comp rx=0 tx=0).stop 2024-05-09T03:20:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 shutdown_connections 2024-05-09T03:20:40.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcf84162f60 0x7fcf840b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf84162610 0x7fcf840b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcf70076300 0x7fcf70078810 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 --2- 172.21.15.46:0/3937978833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf840b9200 0x7fcf840ccc80 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 >> 172.21.15.46:0/3937978833 conn(0x7fcf840b56e0 msgr2=0x7fcf840b6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 shutdown_connections 2024-05-09T03:20:40.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:40.392+0000 7fcf8a9dc640 1 -- 172.21.15.46:0/3937978833 wait complete. 2024-05-09T03:20:40.404 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-05-09T03:20:43.031 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:43.022+0000 7f11c10ee640 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:43.031 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:20:43.022+0000 7f11c10ee640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-05-09T03:20:43.031 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:43.024+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:20:43.032 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:20:43.024+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-05-09T03:20:43.033 INFO:tasks.ceph.mds.c.smithi046.stderr: -20> 2024-05-09T03:20:43.022+0000 7f11c10ee640 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input [buffer:3] 2024-05-09T03:20:43.033 INFO:tasks.ceph.mds.c.smithi046.stderr: -17> 2024-05-09T03:20:43.022+0000 7f11c10ee640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-05-09T03:20:43.096 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:45.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:45.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.534+0000 7fbcc54f9640 1 Processor -- start 2024-05-09T03:20:45.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.534+0000 7fbcc54f9640 1 -- start start 2024-05-09T03:20:45.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.535+0000 7fbcc54f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 0x7fbcc0160670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.535+0000 7fbcc54f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbcc006e160 0x7fbcc00cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.535+0000 7fbcc54f9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcc00d2d20 con 0x7fbcc00d1ca0 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.535+0000 7fbcc54f9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbcc00d2ad0 con 0x7fbcc006e160 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.535+0000 7fbcc54f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc00d3650 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 0x7fbcc0160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbf7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbcc006e160 0x7fbcc00cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 0x7fbcc0160670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:58432/0 (socket says 172.21.15.46:58432) 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 -- 172.21.15.46:0/929712330 learned_addr learned my addr 172.21.15.46:0/929712330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbe7fc640 1 --1- 172.21.15.46:0/929712330 >> v1:172.21.15.88:6789/0 conn(0x7fbcc00d1ca0 0x7fbcc00d20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:34770/0 (socket says 172.21.15.46:34770) 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbdffb640 1 -- 172.21.15.46:0/929712330 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221793815 0 0) 0x7fbcc00d2d20 con 0x7fbcc00d1ca0 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbdffb640 1 -- 172.21.15.46:0/929712330 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcac003520 con 0x7fbcc00d1ca0 2024-05-09T03:20:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 -- 172.21.15.46:0/929712330 >> v2:172.21.15.88:3300/0 conn(0x7fbcc006e160 msgr2=0x7fbcc00cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 --2- 172.21.15.46:0/929712330 >> v2:172.21.15.88:3300/0 conn(0x7fbcc006e160 0x7fbcc00cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 -- 172.21.15.46:0/929712330 >> v1:172.21.15.88:6789/0 conn(0x7fbcc00d1ca0 legacy=0x7fbcc00d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbeffd640 1 -- 172.21.15.46:0/929712330 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc00d3000 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.536+0000 7fbcbf7fe640 1 --2- 172.21.15.46:0/929712330 >> v2:172.21.15.88:3300/0 conn(0x7fbcc006e160 0x7fbcc00cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.537+0000 7fbcbeffd640 1 --2- 172.21.15.46:0/929712330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 0x7fbcc0160670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8009850 tx=0x7fbcb802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93224db595b0b96e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.537+0000 7fbcbdffb640 1 -- 172.21.15.46:0/929712330 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8004030 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.537+0000 7fbcbdffb640 1 -- 172.21.15.46:0/929712330 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb80041d0 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.537+0000 7fbcbdffb640 1 -- 172.21.15.46:0/929712330 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8003610 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 -- 172.21.15.46:0/929712330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 msgr2=0x7fbcc0160670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/929712330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 0x7fbcc0160670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8009850 tx=0x7fbcb802a250 comp rx=0 tx=0).stop 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 -- 172.21.15.46:0/929712330 shutdown_connections 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/929712330 >> v2:172.21.15.88:3300/0 conn(0x7fbcc006e160 0x7fbcc00cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/929712330 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00cfeb0 0x7fbcc0160670 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 -- 172.21.15.46:0/929712330 >> 172.21.15.46:0/929712330 conn(0x7fbcc009b7d0 msgr2=0x7fbcc009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 -- 172.21.15.46:0/929712330 shutdown_connections 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.538+0000 7fbcc54f9640 1 -- 172.21.15.46:0/929712330 wait complete. 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.540+0000 7fbcc54f9640 1 Processor -- start 2024-05-09T03:20:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.540+0000 7fbcc54f9640 1 -- start start 2024-05-09T03:20:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcc54f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc006e160 0x7fbcc014bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcc54f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 0x7fbcc0162590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcc54f9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00d1ca0 0x7fbcc01f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcc54f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc00cd850 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcc54f9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc00cd6d0 con 0x7fbcc006e160 2024-05-09T03:20:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcc54f9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc00cd9d0 con 0x7fbcc00d1ca0 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcbe7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 0x7fbcc0162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcbe7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 0x7fbcc0162590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46158/0 (socket says 172.21.15.46:46158) 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcbe7fc640 1 -- 172.21.15.46:0/2319004343 learned_addr learned my addr 172.21.15.46:0/2319004343 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.541+0000 7fbcbeffd640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc006e160 0x7fbcc014bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbf7fe640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00d1ca0 0x7fbcc01f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbe7fc640 1 -- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00d1ca0 msgr2=0x7fbcc01f9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbe7fc640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00d1ca0 0x7fbcc01f9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbe7fc640 1 -- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc006e160 msgr2=0x7fbcc014bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbe7fc640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc006e160 0x7fbcc014bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbe7fc640 1 -- 172.21.15.46:0/2319004343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc01638d0 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcbe7fc640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 0x7fbcc0162590 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c3f0 tx=0x7fbca800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca80074e0 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca8007680 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc0163030 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8004a20 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.542+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc01635f0 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.543+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc006a270 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.544+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbca8004bc0 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.545+0000 7fbcb77fe640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca0076300 0x7fbca0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.545+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fbca8097750 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.545+0000 7fbcbeffd640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca0076300 0x7fbca0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.545+0000 7fbcbeffd640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca0076300 0x7fbca0078810 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8033000 tx=0x7fbcb802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.550+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbca805fc60 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.781+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcc00cf600 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.781+0000 7fbcb77fe640 1 -- 172.21.15.46:0/2319004343 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7fbca8063f60 con 0x7fbcc00cfeb0 2024-05-09T03:20:45.782 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:45.782 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:45.782 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca0076300 msgr2=0x7fbca0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca0076300 0x7fbca0078810 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8033000 tx=0x7fbcb802e000 comp rx=0 tx=0).stop 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 msgr2=0x7fbcc0162590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 0x7fbcc0162590 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c3f0 tx=0x7fbca800c7f0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 shutdown_connections 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbcc00d1ca0 0x7fbcc01f9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbca0076300 0x7fbca0078810 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcc00cfeb0 0x7fbcc0162590 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 --2- 172.21.15.46:0/2319004343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc006e160 0x7fbcc014bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 >> 172.21.15.46:0/2319004343 conn(0x7fbcc009b7d0 msgr2=0x7fbcc00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 shutdown_connections 2024-05-09T03:20:45.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.783+0000 7fbcc54f9640 1 -- 172.21.15.46:0/2319004343 wait complete. 2024-05-09T03:20:45.795 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-05-09T03:20:45.795 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.909+0000 7fee43b0d640 1 Processor -- start 2024-05-09T03:20:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.909+0000 7fee43b0d640 1 -- start start 2024-05-09T03:20:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee43b0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 0x7fee3c1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee43b0d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee3c160b20 con 0x7fee3c15dfe0 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee43b0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee3c0d15f0 con 0x7fee3c0d2480 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee43b0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fee3c161150 con 0x7fee3c164c70 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee410a8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fee3c15dfe0 0x7fee3c0d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48478/0 (socket says 172.21.15.46:48478) 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee410a8640 1 -- 172.21.15.46:0/1533028257 learned_addr learned my addr 172.21.15.46:0/1533028257 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee418a9640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 0x7fee3c1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655263254 0 0) 0x7fee3c160b20 con 0x7fee3c15dfe0 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee24003520 con 0x7fee3c15dfe0 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3780045531 0 0) 0x7fee24003520 con 0x7fee3c15dfe0 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 msgr2=0x7fee3c1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 0x7fee3c1615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 >> v1:172.21.15.88:6789/0 conn(0x7fee3c0d2480 legacy=0x7fee3c0d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3c0d18d0 con 0x7fee3c15dfe0 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.911+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fee300016f0 con 0x7fee3c15dfe0 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 >> v1:172.21.15.46:6789/0 conn(0x7fee3c15dfe0 legacy=0x7fee3c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 0x7fee240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24003f90 0x7fee24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee24004d40 0x7fee24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c161150 con 0x7fee24004d40 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fee2400bb80 con 0x7fee24003f90 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fee2400bd80 con 0x7fee3c164c70 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee420aa640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee24004d40 0x7fee24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee418a9640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 0x7fee240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee410a8640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24003f90 0x7fee24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee420aa640 1 -- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 msgr2=0x7fee240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee420aa640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c164c70 0x7fee240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee420aa640 1 -- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24003f90 msgr2=0x7fee24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee420aa640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24003f90 0x7fee24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.912+0000 7fee420aa640 1 -- 172.21.15.46:0/1533028257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee24003520 con 0x7fee24004d40 2024-05-09T03:20:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.913+0000 7fee420aa640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee24004d40 0x7fee24005190 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fee34001810 tx=0x7fee24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618af3b04cdf2d2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.913+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34037070 con 0x7fee24004d40 2024-05-09T03:20:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.913+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3402ac50 con 0x7fee24004d40 2024-05-09T03:20:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.913+0000 7fee408a7640 1 -- 172.21.15.46:0/1533028257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34001430 con 0x7fee24004d40 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee43b0d640 1 -- 172.21.15.46:0/1533028257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee24004d40 msgr2=0x7fee24005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee43b0d640 1 --2- 172.21.15.46:0/1533028257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee24004d40 0x7fee24005190 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fee34001810 tx=0x7fee24005830 comp rx=0 tx=0).stop 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee420aa640 1 -- 172.21.15.46:0/1533028257 reap_dead start 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee43b0d640 1 -- 172.21.15.46:0/1533028257 shutdown_connections 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee43b0d640 1 -- 172.21.15.46:0/1533028257 >> 172.21.15.46:0/1533028257 conn(0x7fee3c073db0 msgr2=0x7fee3c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee43b0d640 1 -- 172.21.15.46:0/1533028257 shutdown_connections 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.914+0000 7fee43b0d640 1 -- 172.21.15.46:0/1533028257 wait complete. 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.915+0000 7fee43b0d640 1 Processor -- start 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.916+0000 7fee43b0d640 1 -- start start 2024-05-09T03:20:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee43b0d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24004d40 0x7fee3c0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee43b0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 0x7fee3c0cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee43b0d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c0cd710 0x7fee3c0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee43b0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c1615e0 con 0x7fee3c151440 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee43b0d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fee3c15e3f0 con 0x7fee24004d40 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee43b0d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fee3c161760 con 0x7fee3c0cd710 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 0x7fee3c0cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 0x7fee3c0cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40192/0 (socket says 172.21.15.46:40192) 2024-05-09T03:20:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 -- 172.21.15.46:0/3335824993 learned_addr learned my addr 172.21.15.46:0/3335824993 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee418a9640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24004d40 0x7fee3c0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee420aa640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c0cd710 0x7fee3c0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 -- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c0cd710 msgr2=0x7fee3c0c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c0cd710 0x7fee3c0c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 -- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24004d40 msgr2=0x7fee3c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24004d40 0x7fee3c0cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.917+0000 7fee410a8640 1 -- 172.21.15.46:0/3335824993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee3c151dd0 con 0x7fee3c151440 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.918+0000 7fee410a8640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 0x7fee3c0cd1d0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fee300072c0 tx=0x7fee30007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.918+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3000f960 con 0x7fee3c151440 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.918+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee30018070 con 0x7fee3c151440 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.918+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee3c0cdb20 con 0x7fee3c151440 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.918+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3000fd80 con 0x7fee3c151440 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.918+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee3c0cb270 con 0x7fee3c151440 2024-05-09T03:20:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.919+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3c08cfd0 con 0x7fee3c151440 2024-05-09T03:20:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.920+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fee3001d460 con 0x7fee3c151440 2024-05-09T03:20:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.920+0000 7fee3a7fc640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fee280762d0 0x7fee280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.920+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fee30098dc0 con 0x7fee3c151440 2024-05-09T03:20:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.921+0000 7fee418a9640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fee280762d0 0x7fee280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.921+0000 7fee418a9640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fee280762d0 0x7fee280787e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fee2c014510 tx=0x7fee2c00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:45.924+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fee3c08cfd0 con 0x7fee3c151440 2024-05-09T03:20:46.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.153+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee3c0ce560 con 0x7fee3c151440 2024-05-09T03:20:46.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.153+0000 7fee3a7fc640 1 -- 172.21.15.46:0/3335824993 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7fee3c0cdb20 con 0x7fee3c151440 2024-05-09T03:20:46.154 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:46.154 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fee280762d0 msgr2=0x7fee280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fee280762d0 0x7fee280787e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fee2c014510 tx=0x7fee2c00b040 comp rx=0 tx=0).stop 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 msgr2=0x7fee3c0cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 0x7fee3c0cd1d0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fee300072c0 tx=0x7fee30007780 comp rx=0 tx=0).stop 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 shutdown_connections 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fee3c0cd710 0x7fee3c0c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fee280762d0 0x7fee280787e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee3c151440 0x7fee3c0cd1d0 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 --2- 172.21.15.46:0/3335824993 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee24004d40 0x7fee3c0cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.155+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 >> 172.21.15.46:0/3335824993 conn(0x7fee3c073db0 msgr2=0x7fee3c0d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.156+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 shutdown_connections 2024-05-09T03:20:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.156+0000 7fee43b0d640 1 -- 172.21.15.46:0/3335824993 wait complete. 2024-05-09T03:20:46.168 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-05-09T03:20:46.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:46.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.282+0000 7f100e429640 1 Processor -- start 2024-05-09T03:20:46.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.282+0000 7f100e429640 1 -- start start 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.283+0000 7f100e429640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10080b7ef0 0x7f100816b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.283+0000 7f100e429640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10080d49e0 con 0x7f10080d57a0 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.283+0000 7f100e429640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10080d4cc0 con 0x7f1008160610 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.283+0000 7f100e429640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f10080d5bb0 con 0x7f10080b7ef0 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f100c9c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10080b7ef0 0x7f100816b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f1007fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f10080d57a0 0x7f10080b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48492/0 (socket says 172.21.15.46:48492) 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f1007fff640 1 -- 172.21.15.46:0/3359863854 learned_addr learned my addr 172.21.15.46:0/3359863854 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f100c9c6640 1 -- 172.21.15.46:0/3359863854 >> v1:172.21.15.88:6789/0 conn(0x7f1008160610 legacy=0x7f10080b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f100c9c6640 1 -- 172.21.15.46:0/3359863854 >> v1:172.21.15.46:6789/0 conn(0x7f10080d57a0 legacy=0x7f10080b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f1006ffd640 1 -- 172.21.15.46:0/3359863854 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728904627 0 0) 0x7f10080d49e0 con 0x7f10080d57a0 2024-05-09T03:20:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.284+0000 7f100c9c6640 1 -- 172.21.15.46:0/3359863854 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10080d4fa0 con 0x7f10080b7ef0 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.285+0000 7f100c9c6640 1 --2- 172.21.15.46:0/3359863854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b7ef0 0x7f100816b1f0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f1000009d90 tx=0x7f100002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb0909e430cb48a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.285+0000 7f1006ffd640 1 -- 172.21.15.46:0/3359863854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000007740 con 0x7f10080b7ef0 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.285+0000 7f1006ffd640 1 -- 172.21.15.46:0/3359863854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10000078e0 con 0x7f10080b7ef0 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.285+0000 7f1006ffd640 1 -- 172.21.15.46:0/3359863854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000007cb0 con 0x7f10080b7ef0 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.285+0000 7f100e429640 1 -- 172.21.15.46:0/3359863854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b7ef0 msgr2=0x7f100816b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.285+0000 7f100e429640 1 --2- 172.21.15.46:0/3359863854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b7ef0 0x7f100816b1f0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f1000009d90 tx=0x7f100002a6f0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.286+0000 7f100e429640 1 -- 172.21.15.46:0/3359863854 shutdown_connections 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.286+0000 7f100e429640 1 --2- 172.21.15.46:0/3359863854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b7ef0 0x7f100816b1f0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.286+0000 7f100e429640 1 -- 172.21.15.46:0/3359863854 >> 172.21.15.46:0/3359863854 conn(0x7f10080743e0 msgr2=0x7f10080d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.286+0000 7f100e429640 1 -- 172.21.15.46:0/3359863854 shutdown_connections 2024-05-09T03:20:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.286+0000 7f100e429640 1 -- 172.21.15.46:0/3359863854 wait complete. 2024-05-09T03:20:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.287+0000 7f100e429640 1 Processor -- start 2024-05-09T03:20:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.288+0000 7f100e429640 1 -- start start 2024-05-09T03:20:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100e429640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10080d57a0 0x7f10081f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100e429640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1008160610 0x7f10081593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100e429640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 0x7f10080b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100e429640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10080d15b0 con 0x7f10080b6de0 2024-05-09T03:20:46.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100e429640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1008160a20 con 0x7f10080d57a0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100e429640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f10080d1920 con 0x7f1008160610 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 0x7f10080b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 0x7f10080b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40206/0 (socket says 172.21.15.46:40206) 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 -- 172.21.15.46:0/4103185831 learned_addr learned my addr 172.21.15.46:0/4103185831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f1007fff640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10080d57a0 0x7f10081f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f10077fe640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1008160610 0x7f10081593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 -- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1008160610 msgr2=0x7f10081593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1008160610 0x7f10081593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 -- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10080d57a0 msgr2=0x7f10081f9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10080d57a0 0x7f10081f9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.289+0000 7f100c9c6640 1 -- 172.21.15.46:0/4103185831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1008152520 con 0x7f10080b6de0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.290+0000 7f100c9c6640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 0x7f10080b71f0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f1000009eb0 tx=0x7f1000004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.290+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100003f070 con 0x7f10080b6de0 2024-05-09T03:20:46.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.290+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1000031c90 con 0x7f10080b6de0 2024-05-09T03:20:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.290+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10080b7730 con 0x7f10080b6de0 2024-05-09T03:20:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.290+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000043630 con 0x7f10080b6de0 2024-05-09T03:20:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.290+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f100815a330 con 0x7f10080b6de0 2024-05-09T03:20:46.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.291+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1008159d00 con 0x7f10080b6de0 2024-05-09T03:20:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.292+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f100003b410 con 0x7f10080b6de0 2024-05-09T03:20:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.292+0000 7f1004ff9640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ff40762d0 0x7f0ff40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.292+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7f10000b9820 con 0x7f10080b6de0 2024-05-09T03:20:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.292+0000 7f1007fff640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ff40762d0 0x7f0ff40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.293+0000 7f1007fff640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ff40762d0 0x7f0ff40787e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8000f60 tx=0x7f0ff8007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:46.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.297+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f10080b7730 con 0x7f10080b6de0 2024-05-09T03:20:46.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.525+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f100806a270 con 0x7f10080b6de0 2024-05-09T03:20:46.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.525+0000 7f1004ff9640 1 -- 172.21.15.46:0/4103185831 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7f1008152520 con 0x7f10080b6de0 2024-05-09T03:20:46.526 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:46.526 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:46.527 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:46.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ff40762d0 msgr2=0x7f0ff40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ff40762d0 0x7f0ff40787e0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8000f60 tx=0x7f0ff8007e40 comp rx=0 tx=0).stop 2024-05-09T03:20:46.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 msgr2=0x7f10080b71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 0x7f10080b71f0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f1000009eb0 tx=0x7f1000004a40 comp rx=0 tx=0).stop 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 shutdown_connections 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f0ff40762d0 0x7f0ff40787e0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10080b6de0 0x7f10080b71f0 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1008160610 0x7f10081593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 --2- 172.21.15.46:0/4103185831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10080d57a0 0x7f10081f9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.527+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 >> 172.21.15.46:0/4103185831 conn(0x7f10080743e0 msgr2=0x7f10080d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.528+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 shutdown_connections 2024-05-09T03:20:46.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.528+0000 7f100e429640 1 -- 172.21.15.46:0/4103185831 wait complete. 2024-05-09T03:20:46.540 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 1.00000000' led to DAMAGED state 2024-05-09T03:20:46.540 INFO:tasks.cephfs.test_damage:All 39 mutations had expected outcomes 2024-05-09T03:20:46.540 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:20:46.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:20:46.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:20:46.619 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:20:46.620 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:20:46.620 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:46.620 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:46.636 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:46.636 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:20:46.637 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:20:46.637 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:46.677 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:20:46.677 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:20:46.678 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:46.678 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:46.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:46.694 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:20:46.694 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:20:46.694 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:46.734 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:20:46.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:46.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.848+0000 7fd16f5a9640 1 Processor -- start 2024-05-09T03:20:46.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.848+0000 7fd16f5a9640 1 -- start start 2024-05-09T03:20:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16f5a9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd168160610 0x7fd1680b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16f5a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1680b7d20 0x7fd1680b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16f5a9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1680b8130 con 0x7fd1680d57a0 2024-05-09T03:20:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16f5a9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd1680d5bb0 con 0x7fd168160610 2024-05-09T03:20:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16f5a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1680d49e0 con 0x7fd1680b7d20 2024-05-09T03:20:46.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16d345640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd1680d57a0 0x7fd1680b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48496/0 (socket says 172.21.15.46:48496) 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16d345640 1 -- 172.21.15.46:0/4173501876 learned_addr learned my addr 172.21.15.46:0/4173501876 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16db46640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1680b7d20 0x7fd1680b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16cb44640 1 --2- 172.21.15.46:0/4173501876 >> v2:172.21.15.88:3300/0 conn(0x7fd168160610 0x7fd1680b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd167fff640 1 -- 172.21.15.46:0/4173501876 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2660550035 0 0) 0x7fd158001490 con 0x7fd1680d57a0 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd167fff640 1 -- 172.21.15.46:0/4173501876 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd150003520 con 0x7fd1680d57a0 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16cb44640 1 -- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1680b7d20 msgr2=0x7fd1680b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16cb44640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1680b7d20 0x7fd1680b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16cb44640 1 -- 172.21.15.46:0/4173501876 >> v1:172.21.15.46:6789/0 conn(0x7fd1680d57a0 legacy=0x7fd1680b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16cb44640 1 -- 172.21.15.46:0/4173501876 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1680d4cc0 con 0x7fd168160610 2024-05-09T03:20:46.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.850+0000 7fd16db46640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1680b7d20 0x7fd1680b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:46.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.851+0000 7fd16cb44640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd168160610 0x7fd1680b77e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd15c0097a0 tx=0x7fd15c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b2bdc654c7709f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:46.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.851+0000 7fd167fff640 1 -- 172.21.15.46:0/4173501876 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c037070 con 0x7fd168160610 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.851+0000 7fd167fff640 1 -- 172.21.15.46:0/4173501876 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15c032440 con 0x7fd168160610 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.851+0000 7fd167fff640 1 -- 172.21.15.46:0/4173501876 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0327f0 con 0x7fd168160610 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 -- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd168160610 msgr2=0x7fd1680b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd168160610 0x7fd1680b77e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fd15c0097a0 tx=0x7fd15c02a180 comp rx=0 tx=0).stop 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 -- 172.21.15.46:0/4173501876 shutdown_connections 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1680b7d20 0x7fd1680b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/4173501876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd168160610 0x7fd1680b77e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 -- 172.21.15.46:0/4173501876 >> 172.21.15.46:0/4173501876 conn(0x7fd1680743e0 msgr2=0x7fd1680d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 -- 172.21.15.46:0/4173501876 shutdown_connections 2024-05-09T03:20:46.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.852+0000 7fd16f5a9640 1 -- 172.21.15.46:0/4173501876 wait complete. 2024-05-09T03:20:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.854+0000 7fd16f5a9640 1 Processor -- start 2024-05-09T03:20:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.854+0000 7fd16f5a9640 1 -- start start 2024-05-09T03:20:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16f5a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 0x7fd168159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16f5a9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680d57a0 0x7fd168152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16f5a9640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd168160610 0x7fd1680b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16f5a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd168160a20 con 0x7fd1680b7d20 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16f5a9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1680d5060 con 0x7fd1680d57a0 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16f5a9640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1680d15b0 con 0x7fd168160610 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16d345640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 0x7fd168159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16d345640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 0x7fd168159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40222/0 (socket says 172.21.15.46:40222) 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.855+0000 7fd16d345640 1 -- 172.21.15.46:0/1178727425 learned_addr learned my addr 172.21.15.46:0/1178727425 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16cb44640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680d57a0 0x7fd168152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16db46640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd168160610 0x7fd1680b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16d345640 1 -- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd168160610 msgr2=0x7fd1680b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16d345640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd168160610 0x7fd1680b6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16d345640 1 -- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680d57a0 msgr2=0x7fd168152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16d345640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680d57a0 0x7fd168152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16d345640 1 -- 172.21.15.46:0/1178727425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1681539f0 con 0x7fd1680b7d20 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16cb44640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680d57a0 0x7fd168152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16db46640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd168160610 0x7fd1680b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16d345640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 0x7fd168159c70 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fd15800caf0 tx=0x7fd15800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15800ac40 con 0x7fd1680b7d20 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15800ade0 con 0x7fd1680b7d20 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1681fa300 con 0x7fd1680b7d20 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd158012620 con 0x7fd1680b7d20 2024-05-09T03:20:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.856+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1681fa8c0 con 0x7fd1680b7d20 2024-05-09T03:20:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.857+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16806a270 con 0x7fd1680b7d20 2024-05-09T03:20:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.858+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd1580127c0 con 0x7fd1680b7d20 2024-05-09T03:20:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.858+0000 7fd165ffb640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1540762d0 0x7fd1540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.859+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fd158098af0 con 0x7fd1680b7d20 2024-05-09T03:20:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.859+0000 7fd16cb44640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1540762d0 0x7fd1540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.859+0000 7fd16cb44640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1540762d0 0x7fd1540787e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd15c034000 tx=0x7fd15c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:46.863+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd1680b8130 con 0x7fd1680b7d20 2024-05-09T03:20:47.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.096+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1680ba1e0 con 0x7fd1680b7d20 2024-05-09T03:20:47.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.097+0000 7fd165ffb640 1 -- 172.21.15.46:0/1178727425 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7fd158061560 con 0x7fd1680b7d20 2024-05-09T03:20:47.098 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:47.098 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:47.098 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:47.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1540762d0 msgr2=0x7fd1540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1540762d0 0x7fd1540787e0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd15c034000 tx=0x7fd15c02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:47.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 msgr2=0x7fd168159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 0x7fd168159c70 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fd15800caf0 tx=0x7fd15800a060 comp rx=0 tx=0).stop 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 shutdown_connections 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd168160610 0x7fd1680b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680d57a0 0x7fd168152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1540762d0 0x7fd1540787e0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 --2- 172.21.15.46:0/1178727425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1680b7d20 0x7fd168159c70 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 >> 172.21.15.46:0/1178727425 conn(0x7fd1680743e0 msgr2=0x7fd1680b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.099+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 shutdown_connections 2024-05-09T03:20:47.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.100+0000 7fd16f5a9640 1 -- 172.21.15.46:0/1178727425 wait complete. 2024-05-09T03:20:47.112 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:47.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.226+0000 7fc544a7e640 1 Processor -- start 2024-05-09T03:20:47.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.226+0000 7fc544a7e640 1 -- start start 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc544a7e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5400b7ed0 0x7fc540163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc544a7e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5401617d0 con 0x7fc5400b5210 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc544a7e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5400d5280 con 0x7fc5401613c0 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc544a7e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc5400b5620 con 0x7fc5400b7ed0 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc53ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5400b7ed0 0x7fc540163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc53e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc5400b5210 0x7fc5400d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48506/0 (socket says 172.21.15.46:48506) 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc53e59c640 1 -- 172.21.15.46:0/1303024614 learned_addr learned my addr 172.21.15.46:0/1303024614 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc53ed9d640 1 -- 172.21.15.46:0/1303024614 >> v1:172.21.15.88:6789/0 conn(0x7fc5401613c0 legacy=0x7fc5400b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc53ed9d640 1 -- 172.21.15.46:0/1303024614 >> v1:172.21.15.46:6789/0 conn(0x7fc5400b5210 legacy=0x7fc5400d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.228+0000 7fc53ed9d640 1 -- 172.21.15.46:0/1303024614 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5400d5560 con 0x7fc5400b7ed0 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.229+0000 7fc53ed9d640 1 --2- 172.21.15.46:0/1303024614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5400b7ed0 0x7fc540163e60 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fc538009d90 tx=0x7fc53802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd6a2f85572a25e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.229+0000 7fc53d59a640 1 -- 172.21.15.46:0/1303024614 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538007740 con 0x7fc5400b7ed0 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.229+0000 7fc53d59a640 1 -- 172.21.15.46:0/1303024614 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5380078e0 con 0x7fc5400b7ed0 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.229+0000 7fc53d59a640 1 -- 172.21.15.46:0/1303024614 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc538007cb0 con 0x7fc5400b7ed0 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 -- 172.21.15.46:0/1303024614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5400b7ed0 msgr2=0x7fc540163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 --2- 172.21.15.46:0/1303024614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5400b7ed0 0x7fc540163e60 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7fc538009d90 tx=0x7fc53802a6f0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 -- 172.21.15.46:0/1303024614 shutdown_connections 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 --2- 172.21.15.46:0/1303024614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5400b7ed0 0x7fc540163e60 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 -- 172.21.15.46:0/1303024614 >> 172.21.15.46:0/1303024614 conn(0x7fc5400d29a0 msgr2=0x7fc5400d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 -- 172.21.15.46:0/1303024614 shutdown_connections 2024-05-09T03:20:47.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.230+0000 7fc544a7e640 1 -- 172.21.15.46:0/1303024614 wait complete. 2024-05-09T03:20:47.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.232+0000 7fc544a7e640 1 Processor -- start 2024-05-09T03:20:47.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.232+0000 7fc544a7e640 1 -- start start 2024-05-09T03:20:47.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc544a7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5400b5210 0x7fc540151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc544a7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 0x7fc5400cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc544a7e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5400cf710 0x7fc5400ba080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc544a7e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc540160ea0 con 0x7fc5401613c0 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc544a7e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5401609a0 con 0x7fc5400cf710 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc544a7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc5400d4f10 con 0x7fc5400b5210 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc53dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 0x7fc5400cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc53dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 0x7fc5400cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40250/0 (socket says 172.21.15.46:40250) 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.233+0000 7fc53dd9b640 1 -- 172.21.15.46:0/2099960401 learned_addr learned my addr 172.21.15.46:0/2099960401 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53e59c640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5400b5210 0x7fc540151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53ed9d640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5400cf710 0x7fc5400ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53dd9b640 1 -- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5400b5210 msgr2=0x7fc540151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53dd9b640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5400b5210 0x7fc540151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53dd9b640 1 -- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5400cf710 msgr2=0x7fc5400ba080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53dd9b640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5400cf710 0x7fc5400ba080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53dd9b640 1 -- 172.21.15.46:0/2099960401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5401e9e60 con 0x7fc5401613c0 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53e59c640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5400b5210 0x7fc540151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc53dd9b640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 0x7fc5400cf1d0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fc52800c3d0 tx=0x7fc52800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528017070 con 0x7fc5401613c0 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5280052d0 con 0x7fc5401613c0 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5401ea140 con 0x7fc5401613c0 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528003830 con 0x7fc5401613c0 2024-05-09T03:20:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.234+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5401ea700 con 0x7fc5401613c0 2024-05-09T03:20:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.235+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc54006a270 con 0x7fc5401613c0 2024-05-09T03:20:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.236+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc52801a070 con 0x7fc5401613c0 2024-05-09T03:20:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.236+0000 7fc536ffd640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5200762d0 0x7fc5200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.237+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fc5280977c0 con 0x7fc5401613c0 2024-05-09T03:20:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.237+0000 7fc53e59c640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5200762d0 0x7fc5200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.237+0000 7fc53e59c640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5200762d0 0x7fc5200787e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc530000f60 tx=0x7fc530009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.241+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc528060290 con 0x7fc5401613c0 2024-05-09T03:20:47.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.475+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5400c9610 con 0x7fc5401613c0 2024-05-09T03:20:47.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.475+0000 7fc536ffd640 1 -- 172.21.15.46:0/2099960401 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7fc528068310 con 0x7fc5401613c0 2024-05-09T03:20:47.476 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:47.477 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:47.477 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5200762d0 msgr2=0x7fc5200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5200762d0 0x7fc5200787e0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc530000f60 tx=0x7fc530009040 comp rx=0 tx=0).stop 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 msgr2=0x7fc5400cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 0x7fc5400cf1d0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fc52800c3d0 tx=0x7fc52800c890 comp rx=0 tx=0).stop 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 shutdown_connections 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5400cf710 0x7fc5400ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc5200762d0 0x7fc5200787e0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5401613c0 0x7fc5400cf1d0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 --2- 172.21.15.46:0/2099960401 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc5400b5210 0x7fc540151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 >> 172.21.15.46:0/2099960401 conn(0x7fc5400d29a0 msgr2=0x7fc5400b9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 shutdown_connections 2024-05-09T03:20:47.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.478+0000 7fc544a7e640 1 -- 172.21.15.46:0/2099960401 wait complete. 2024-05-09T03:20:47.490 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:20:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.604+0000 7fc147bce640 1 Processor -- start 2024-05-09T03:20:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.604+0000 7fc147bce640 1 -- start start 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc147bce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14008d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc147bce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc14008d6e0 0x7fc1400d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc147bce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1400d20a0 0x7fc1400d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc147bce640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc1400d29f0 con 0x7fc1400d20a0 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc147bce640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc1400b5460 con 0x7fc14008d6e0 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc147bce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400cfac0 con 0x7fc1400b9200 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc14616b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1400d20a0 0x7fc1400d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc14616b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1400d20a0 0x7fc1400d24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40258/0 (socket says 172.21.15.46:40258) 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc14616b640 1 -- 172.21.15.46:0/3778865788 learned_addr learned my addr 172.21.15.46:0/3778865788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc14596a640 1 --2- 172.21.15.46:0/3778865788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14008d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.605+0000 7fc145169640 1 --2- 172.21.15.46:0/3778865788 >> v2:172.21.15.88:3300/0 conn(0x7fc14008d6e0 0x7fc1400d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.606+0000 7fc14616b640 1 -- 172.21.15.46:0/3778865788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 msgr2=0x7fc14008d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.606+0000 7fc14616b640 1 --2- 172.21.15.46:0/3778865788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14008d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.606+0000 7fc14616b640 1 -- 172.21.15.46:0/3778865788 >> v2:172.21.15.88:3300/0 conn(0x7fc14008d6e0 msgr2=0x7fc1400d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.606+0000 7fc14616b640 1 --2- 172.21.15.46:0/3778865788 >> v2:172.21.15.88:3300/0 conn(0x7fc14008d6e0 0x7fc1400d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.606+0000 7fc14616b640 1 -- 172.21.15.46:0/3778865788 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1400b4ee0 con 0x7fc1400d20a0 2024-05-09T03:20:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.606+0000 7fc14616b640 1 --2- 172.21.15.46:0/3778865788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc1400d24b0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fc140074a00 tx=0x7fc13802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e545c808fd41e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc144968640 1 -- 172.21.15.46:0/3778865788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138036070 con 0x7fc1400d20a0 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc144968640 1 -- 172.21.15.46:0/3778865788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc138031600 con 0x7fc1400d20a0 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc144968640 1 -- 172.21.15.46:0/3778865788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1380319d0 con 0x7fc1400d20a0 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 -- 172.21.15.46:0/3778865788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 msgr2=0x7fc1400d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 --2- 172.21.15.46:0/3778865788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc1400d24b0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fc140074a00 tx=0x7fc13802a6c0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 -- 172.21.15.46:0/3778865788 shutdown_connections 2024-05-09T03:20:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 --2- 172.21.15.46:0/3778865788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc1400d24b0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 --2- 172.21.15.46:0/3778865788 >> v2:172.21.15.88:3300/0 conn(0x7fc14008d6e0 0x7fc1400d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 --2- 172.21.15.46:0/3778865788 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14008d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 -- 172.21.15.46:0/3778865788 >> 172.21.15.46:0/3778865788 conn(0x7fc1400b56e0 msgr2=0x7fc1400b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 -- 172.21.15.46:0/3778865788 shutdown_connections 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.607+0000 7fc147bce640 1 -- 172.21.15.46:0/3778865788 wait complete. 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.609+0000 7fc147bce640 1 Processor -- start 2024-05-09T03:20:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.609+0000 7fc147bce640 1 -- start start 2024-05-09T03:20:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc14008d6e0 0x7fc140144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14015d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc140147710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400cf470 con 0x7fc1400d20a0 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400b5180 con 0x7fc14008d6e0 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400cf750 con 0x7fc1400b9200 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc140147710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc140147710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40270/0 (socket says 172.21.15.46:40270) 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 -- 172.21.15.46:0/1058636479 learned_addr learned my addr 172.21.15.46:0/1058636479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14596a640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc14008d6e0 0x7fc140144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc145169640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14015d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 -- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 msgr2=0x7fc14015d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14015d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 -- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc14008d6e0 msgr2=0x7fc140144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc14008d6e0 0x7fc140144fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 -- 172.21.15.46:0/1058636479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc140145a10 con 0x7fc1400d20a0 2024-05-09T03:20:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc14616b640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc140147710 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fc138009e50 tx=0x7fc138004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138036050 con 0x7fc1400d20a0 2024-05-09T03:20:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc138042070 con 0x7fc1400d20a0 2024-05-09T03:20:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.611+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc140145cf0 con 0x7fc1400d20a0 2024-05-09T03:20:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.612+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138009510 con 0x7fc1400d20a0 2024-05-09T03:20:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.612+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1401462b0 con 0x7fc1400d20a0 2024-05-09T03:20:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.612+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14006a270 con 0x7fc1400d20a0 2024-05-09T03:20:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.613+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc138007bf0 con 0x7fc1400d20a0 2024-05-09T03:20:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.614+0000 7fc1327fc640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1240762d0 0x7fc1240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.614+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fc1380b8fe0 con 0x7fc1400d20a0 2024-05-09T03:20:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.614+0000 7fc14596a640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1240762d0 0x7fc1240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.615+0000 7fc14596a640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1240762d0 0x7fc1240787e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009aa0 tx=0x7fc13c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.618+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1380814b0 con 0x7fc1400d20a0 2024-05-09T03:20:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.810+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc140071b40 con 0x7fc1400d20a0 2024-05-09T03:20:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.812+0000 7fc1327fc640 1 -- 172.21.15.46:0/1058636479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+21866 (secure 0 0 0) 0x7fc1380857b0 con 0x7fc1400d20a0 2024-05-09T03:20:47.813 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:47.813 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:20:43.024284+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:15:50.144137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-09T03:15:51.401672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.20","mappings":[{"from":1,"to":5}]},{"pgid":"13.24","mappings":[{"from":6,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/1514520479":"2024-05-10T03:20:43.024261+0000","172.21.15.46:6834/2034185175":"2024-05-10T03:20:36.108968+0000","172.21.15.46:6832/2034185175":"2024-05-10T03:20:36.108968+0000","172.21.15.46:6834/2183116503":"2024-05-10T03:20:24.715744+0000","172.21.15.46:6832/2183116503":"2024-05-10T03:20:24.715744+0000","172.21.15.46:6834/134300288":"2024-05-10T03:20:09.183188+0000","172.21.15.46:6832/134300288":"2024-05-10T03:20:09.183188+0000","172.21.15.46:6834/1774919282":"2024-05-10T03:20:03.819873+0000","172.21.15.46:6832/2209678569":"2024-05-10T03:19:52.097765+0000","172.21.15.46:6834/3872337434":"2024-05-10T03:19:35.738958+0000","172.21.15.46:6834/1350199591":"2024-05-10T03:19:30.622459+0000","172.21.15.46:6832/574897435":"2024-05-10T03:20:30.067507+0000","172.21.15.46:6832/3872337434":"2024-05-10T03:19:35.738958+0000","172.21.15.46:6832/925919281":"2024-05-10T03:19:16.273718+0000","172.21.15.46:6834/120572030":"2024-05-10T03:19:12.242288+0000","172.21.15.46:6832/1774919282":"2024-05-10T03:20:03.819873+0000","172.21.15.46:6832/3450505404":"2024-05-10T03:17:52.032707+0000","172.21.15.46:6832/1270051447":"2024-05-10T03:17:29.198075+0000","172.21.15.46:6832/2388442105":"2024-05-10T03:17:44.993790+0000","172.21.15.46:6834/3968811793":"2024-05-10T03:17:16.129644+0000","172.21.15.46:6834/925919281":"2024-05-10T03:19:16.273718+0000","172.21.15.46:6834/1270051447":"2024-05-10T03:17:29.198075+0000","172.21.15.46:6834/2860475391":"2024-05-10T03:16:44.302289+0000","172.21.15.46:6832/3044304069":"2024-05-10T03:17:09.735805+0000","172.21.15.46:6832/1514520479":"2024-05-10T03:20:43.024261+0000","172.21.15.46:6834/2566829830":"2024-05-10T03:16:58.040696+0000","172.21.15.46:6832/3968811793":"2024-05-10T03:17:16.129644+0000","172.21.15.46:6834/2032086476":"2024-05-10T03:17:22.988689+0000","172.21.15.46:6832/1350199591":"2024-05-10T03:19:30.622459+0000","172.21.15.46:6834/2546691383":"2024-05-10T03:18:27.254542+0000","172.21.15.46:6834/2347304023":"2024-05-10T03:16:08.753704+0000","172.21.15.46:6834/2388442105":"2024-05-10T03:17:44.993790+0000","172.21.15.46:6834/1234333269":"2024-05-10T03:17:58.071311+0000","172.21.15.46:6832/2827001018":"2024-05-10T03:19:01.862132+0000","172.21.15.46:6834/2585438921":"2024-05-10T03:18:23.220415+0000","172.21.15.46:6832/2789195040":"2024-05-10T03:18:36.012259+0000","172.21.15.46:6832/1281653036":"2024-05-10T03:16:24.681981+0000","172.21.15.46:6834/2209678569":"2024-05-10T03:19:52.097765+0000","172.21.15.46:6832/1486786164":"2024-05-10T03:17:42.979548+0000","172.21.15.46:6832/2032086476":"2024-05-10T03:17:22.988689+0000","172.21.15.46:6834/3044304069":"2024-05-10T03:17:09.735805+0000","172.21.15.46:6834/2490064583":"2024-05-10T03:18:44.067190+0000","172.21.15.46:6832/782052086":"2024-05-10T03:16:40.263688+0000","172.21.15.46:6832/2566829830":"2024-05-10T03:16:58.040696+0000","172.21.15.46:6834/1281653036":"2024-05-10T03:16:24.681981+0000","172.21.15.46:6834/3297564121":"2024-05-10T03:16:36.237612+0000","172.21.15.46:6832/1234333269":"2024-05-10T03:17:58.071311+0000","172.21.15.46:6832/1033379302":"2024-05-10T03:19:04.179713+0000","172.21.15.46:6832/271467610":"2024-05-10T03:18:02.111947+0000","172.21.15.46:6834/782052086":"2024-05-10T03:16:40.263688+0000","172.21.15.46:6832/3297564121":"2024-05-10T03:16:36.237612+0000","172.21.15.46:6834/271467610":"2024-05-10T03:18:02.111947+0000","172.21.15.46:6832/436596417":"2024-05-10T03:18:06.147085+0000","172.21.15.46:6834/574897435":"2024-05-10T03:20:30.067507+0000","172.21.15.46:6832/905138228":"2024-05-10T03:19:08.214271+0000","172.21.15.46:6834/436596417":"2024-05-10T03:18:06.147085+0000","172.21.15.46:6834/1486786164":"2024-05-10T03:17:42.979548+0000","172.21.15.46:6834/3106969477":"2024-05-10T03:18:20.866657+0000","172.21.15.46:6832/2585438921":"2024-05-10T03:18:23.220415+0000","172.21.15.46:6834/3423945789":"2024-05-10T03:17:12.099885+0000","172.21.15.46:6832/3423945789":"2024-05-10T03:17:12.099885+0000","172.21.15.46:6832/1887013209":"2024-05-10T03:16:30.194822+0000","172.21.15.46:6832/2546691383":"2024-05-10T03:18:27.254542+0000","172.21.15.46:6832/202666403":"2024-05-10T03:18:31.290277+0000","172.21.15.46:6832/935847027":"2024-05-10T03:18:40.005228+0000","172.21.15.46:6834/3450505404":"2024-05-10T03:17:52.032707+0000","172.21.15.46:6832/2347304023":"2024-05-10T03:16:08.753704+0000","172.21.15.46:6834/202666403":"2024-05-10T03:18:31.290277+0000","172.21.15.46:6834/2789195040":"2024-05-10T03:18:36.012259+0000","172.21.15.46:6834/935847027":"2024-05-10T03:18:40.005228+0000","172.21.15.46:6834/905138228":"2024-05-10T03:19:08.214271+0000","172.21.15.46:6832/2490064583":"2024-05-10T03:18:44.067190+0000","172.21.15.46:6834/2827001018":"2024-05-10T03:19:01.862132+0000","172.21.15.46:6832/2638610471":"2024-05-10T03:18:48.102832+0000","172.21.15.46:6832/2860475391":"2024-05-10T03:16:44.302289+0000","172.21.15.46:6832/3106969477":"2024-05-10T03:18:20.866657+0000","172.21.15.46:6834/2638610471":"2024-05-10T03:18:48.102832+0000","172.21.15.46:6834/1033379302":"2024-05-10T03:19:04.179713+0000","172.21.15.46:6834/1887013209":"2024-05-10T03:16:30.194822+0000","172.21.15.46:6832/120572030":"2024-05-10T03:19:12.242288+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:20:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1240762d0 msgr2=0x7fc1240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1240762d0 0x7fc1240787e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009aa0 tx=0x7fc13c009190 comp rx=0 tx=0).stop 2024-05-09T03:20:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 msgr2=0x7fc140147710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc140147710 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fc138009e50 tx=0x7fc138004a40 comp rx=0 tx=0).stop 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 shutdown_connections 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc1240762d0 0x7fc1240787e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1400d20a0 0x7fc140147710 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc1400b9200 0x7fc14015d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 --2- 172.21.15.46:0/1058636479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc14008d6e0 0x7fc140144fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 >> 172.21.15.46:0/1058636479 conn(0x7fc1400b56e0 msgr2=0x7fc1400ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 shutdown_connections 2024-05-09T03:20:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.814+0000 7fc147bce640 1 -- 172.21.15.46:0/1058636479 wait complete. 2024-05-09T03:20:47.827 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:20:47.827 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.940+0000 7f223b384640 1 Processor -- start 2024-05-09T03:20:47.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.941+0000 7f223b384640 1 -- start start 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f223b384640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f22340b7ef0 0x7f223416b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f223b384640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22340d49e0 con 0x7f2234160610 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f223b384640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22340d4cc0 con 0x7f22340d57a0 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f223b384640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f22340d5bb0 con 0x7f22340b7ef0 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f223891f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2234160610 0x7f22340b77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48510/0 (socket says 172.21.15.46:48510) 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f223891f640 1 -- 172.21.15.46:0/2833371416 learned_addr learned my addr 172.21.15.46:0/2833371416 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941847857 0 0) 0x7f22340d49e0 con 0x7f2234160610 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f2239921640 1 --2- 172.21.15.46:0/2833371416 >> v2:172.21.15.88:3300/0 conn(0x7f22340b7ef0 0x7f223416b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.942+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f221c003520 con 0x7f2234160610 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 718517660 0 0) 0x7f221c003520 con 0x7f2234160610 2024-05-09T03:20:47.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 >> v2:172.21.15.88:3300/0 conn(0x7f22340b7ef0 msgr2=0x7f223416b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 --2- 172.21.15.46:0/2833371416 >> v2:172.21.15.88:3300/0 conn(0x7f22340b7ef0 0x7f223416b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 >> v1:172.21.15.88:6789/0 conn(0x7f22340d57a0 legacy=0x7f22340b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22340d4fa0 con 0x7f2234160610 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f22280016f0 con 0x7f2234160610 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 >> v1:172.21.15.46:6789/0 conn(0x7f2234160610 legacy=0x7f22340b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c003a50 0x7f221c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c004800 0x7f221c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f221c005250 0x7f221c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22340d5bb0 con 0x7f221c004800 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f221c00c1e0 con 0x7f221c005250 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f221c00c3e0 con 0x7f221c003a50 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f223891f640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c004800 0x7f221c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2239120640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c003a50 0x7f221c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2239921640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f221c005250 0x7f221c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f223891f640 1 -- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c003a50 msgr2=0x7f221c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f223891f640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c003a50 0x7f221c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f223891f640 1 -- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f221c005250 msgr2=0x7f221c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f223891f640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f221c005250 0x7f221c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f223891f640 1 -- 172.21.15.46:0/2833371416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221c003520 con 0x7f221c004800 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.943+0000 7f2239921640 1 -- 172.21.15.46:0/2833371416 reap_dead start 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.944+0000 7f223891f640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c004800 0x7f221c004c10 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f222800c210 tx=0x7f221c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c685f6d7f1de6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.944+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228007d30 con 0x7f221c004800 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.944+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2228005af0 con 0x7f221c004800 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.944+0000 7f2233fff640 1 -- 172.21.15.46:0/2833371416 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228007070 con 0x7f221c004800 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 -- 172.21.15.46:0/2833371416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c004800 msgr2=0x7f221c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c004800 0x7f221c004c10 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f222800c210 tx=0x7f221c005830 comp rx=0 tx=0).stop 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 -- 172.21.15.46:0/2833371416 shutdown_connections 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 --2- 172.21.15.46:0/2833371416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f221c004800 0x7f221c004c10 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 -- 172.21.15.46:0/2833371416 >> 172.21.15.46:0/2833371416 conn(0x7f22340743e0 msgr2=0x7f22340d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:47.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 -- 172.21.15.46:0/2833371416 shutdown_connections 2024-05-09T03:20:47.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.945+0000 7f223b384640 1 -- 172.21.15.46:0/2833371416 wait complete. 2024-05-09T03:20:47.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.947+0000 7f223b384640 1 Processor -- start 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.947+0000 7f223b384640 1 -- start start 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223b384640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c004800 0x7f22340d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223b384640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 0x7f22341627d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223b384640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2234162d10 0x7f223414d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223b384640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22340d15b0 con 0x7f223415c250 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223b384640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2234160a20 con 0x7f2234162d10 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223b384640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f22340d1920 con 0x7f221c004800 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 0x7f22341627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 0x7f22341627d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40292/0 (socket says 172.21.15.46:40292) 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 -- 172.21.15.46:0/2997114662 learned_addr learned my addr 172.21.15.46:0/2997114662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f2239921640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2234162d10 0x7f223414d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f2239120640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c004800 0x7f22340d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 -- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c004800 msgr2=0x7f22340d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c004800 0x7f22340d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 -- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2234162d10 msgr2=0x7f223414d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.948+0000 7f223891f640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2234162d10 0x7f223414d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f223891f640 1 -- 172.21.15.46:0/2997114662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2234163120 con 0x7f223415c250 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f2239921640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2234162d10 0x7f223414d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f223891f640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 0x7f22341627d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f222800c330 tx=0x7f22340d5170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228040030 con 0x7f223415c250 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2228007d10 con 0x7f223415c250 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228037040 con 0x7f223415c250 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22340caab0 con 0x7f223415c250 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.949+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22340cb070 con 0x7f223415c250 2024-05-09T03:20:47.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.950+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f223406a270 con 0x7f223415c250 2024-05-09T03:20:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.951+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2228043070 con 0x7f223415c250 2024-05-09T03:20:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.951+0000 7f2231ffb640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f22200762d0 0x7f22200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.951+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7f22280b7b60 con 0x7f223415c250 2024-05-09T03:20:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.952+0000 7f2239120640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f22200762d0 0x7f22200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:47.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.952+0000 7f2239120640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f22200762d0 0x7f22200787e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f223406d3b0 tx=0x7f2224009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:47.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:47.956+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2234163120 con 0x7f223415c250 2024-05-09T03:20:48.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.187+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22340d3130 con 0x7f223415c250 2024-05-09T03:20:48.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.188+0000 7f2231ffb640 1 -- 172.21.15.46:0/2997114662 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7f22280016b0 con 0x7f223415c250 2024-05-09T03:20:48.189 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:48.189 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:48.189 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f22200762d0 msgr2=0x7f22200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f22200762d0 0x7f22200787e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f223406d3b0 tx=0x7f2224009040 comp rx=0 tx=0).stop 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 msgr2=0x7f22341627d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 0x7f22341627d0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f222800c330 tx=0x7f22340d5170 comp rx=0 tx=0).stop 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 shutdown_connections 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2234162d10 0x7f223414d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f22200762d0 0x7f22200787e0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223415c250 0x7f22341627d0 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 --2- 172.21.15.46:0/2997114662 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f221c004800 0x7f22340d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 >> 172.21.15.46:0/2997114662 conn(0x7f22340743e0 msgr2=0x7f22341611f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 shutdown_connections 2024-05-09T03:20:48.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.190+0000 7f223b384640 1 -- 172.21.15.46:0/2997114662 wait complete. 2024-05-09T03:20:48.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:48.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.316+0000 7fa2f995f640 1 Processor -- start 2024-05-09T03:20:48.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.316+0000 7fa2f995f640 1 -- start start 2024-05-09T03:20:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f995f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f41615f0 0x7fa2f40b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f995f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2f40d0b80 0x7fa2f40d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f995f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2f40d25e0 0x7fa2f40d29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f995f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f40d15b0 con 0x7fa2f40d25e0 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f995f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f4160b10 con 0x7fa2f40d0b80 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f995f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f4160df0 con 0x7fa2f41615f0 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.317+0000 7fa2f37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2f40d25e0 0x7fa2f40d29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa2f40d25e0 0x7fa2f40d29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40304/0 (socket says 172.21.15.46:40304) 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 -- 172.21.15.46:0/2289162656 learned_addr learned my addr 172.21.15.46:0/2289162656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f27fc640 1 --2- 172.21.15.46:0/2289162656 >> v2:172.21.15.88:3300/0 conn(0x7fa2f40d0b80 0x7fa2f40d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f2ffd640 1 --2- 172.21.15.46:0/2289162656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f41615f0 0x7fa2f40b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 -- 172.21.15.46:0/2289162656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f41615f0 msgr2=0x7fa2f40b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 --2- 172.21.15.46:0/2289162656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f41615f0 0x7fa2f40b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 -- 172.21.15.46:0/2289162656 >> v2:172.21.15.88:3300/0 conn(0x7fa2f40d0b80 msgr2=0x7fa2f40d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 --2- 172.21.15.46:0/2289162656 >> v2:172.21.15.88:3300/0 conn(0x7fa2f40d0b80 0x7fa2f40d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.318+0000 7fa2f37fe640 1 -- 172.21.15.46:0/2289162656 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f41610d0 con 0x7fa2f40d25e0 2024-05-09T03:20:48.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.319+0000 7fa2f37fe640 1 --2- 172.21.15.46:0/2289162656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40d25e0 0x7fa2f40d29f0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40b8870 tx=0x7fa2ec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c41dcd3df0aafeb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.319+0000 7fa2f1ffb640 1 -- 172.21.15.46:0/2289162656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec037070 con 0x7fa2f40d25e0 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.319+0000 7fa2f1ffb640 1 -- 172.21.15.46:0/2289162656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec032440 con 0x7fa2f40d25e0 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.319+0000 7fa2f1ffb640 1 -- 172.21.15.46:0/2289162656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec032810 con 0x7fa2f40d25e0 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.319+0000 7fa2f995f640 1 -- 172.21.15.46:0/2289162656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40d25e0 msgr2=0x7fa2f40d29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.319+0000 7fa2f995f640 1 --2- 172.21.15.46:0/2289162656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40d25e0 0x7fa2f40d29f0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fa2f40b8870 tx=0x7fa2ec02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 -- 172.21.15.46:0/2289162656 shutdown_connections 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 --2- 172.21.15.46:0/2289162656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40d25e0 0x7fa2f40d29f0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 --2- 172.21.15.46:0/2289162656 >> v2:172.21.15.88:3300/0 conn(0x7fa2f40d0b80 0x7fa2f40d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 --2- 172.21.15.46:0/2289162656 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f41615f0 0x7fa2f40b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 -- 172.21.15.46:0/2289162656 >> 172.21.15.46:0/2289162656 conn(0x7fa2f406e190 msgr2=0x7fa2f406e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 -- 172.21.15.46:0/2289162656 shutdown_connections 2024-05-09T03:20:48.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.320+0000 7fa2f995f640 1 -- 172.21.15.46:0/2289162656 wait complete. 2024-05-09T03:20:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.321+0000 7fa2f995f640 1 Processor -- start 2024-05-09T03:20:48.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.322+0000 7fa2f995f640 1 -- start start 2024-05-09T03:20:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f995f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f40d0b80 0x7fa2f414b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f995f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f40d25e0 0x7fa2f415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f995f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 0x7fa2f4149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f995f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f40d48e0 con 0x7fa2f41615f0 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f995f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f40d2f30 con 0x7fa2f40d0b80 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f995f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f40d4a60 con 0x7fa2f40d25e0 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 0x7fa2f4149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 0x7fa2f4149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40316/0 (socket says 172.21.15.46:40316) 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 -- 172.21.15.46:0/4197215468 learned_addr learned my addr 172.21.15.46:0/4197215468 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f2ffd640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f40d0b80 0x7fa2f414b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f27fc640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f40d25e0 0x7fa2f415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 -- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f40d25e0 msgr2=0x7fa2f415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f40d25e0 0x7fa2f415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 -- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f40d0b80 msgr2=0x7fa2f414b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f40d0b80 0x7fa2f414b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.323+0000 7fa2f37fe640 1 -- 172.21.15.46:0/4197215468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f415c7e0 con 0x7fa2f41615f0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.324+0000 7fa2f37fe640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 0x7fa2f4149630 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec02a220 tx=0x7fa2ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.324+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec041020 con 0x7fa2f41615f0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.324+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec03d640 con 0x7fa2f41615f0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.324+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f4148590 con 0x7fa2f41615f0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.324+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec037040 con 0x7fa2f41615f0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.324+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f4148b50 con 0x7fa2f41615f0 2024-05-09T03:20:48.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.325+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f406a270 con 0x7fa2f41615f0 2024-05-09T03:20:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.326+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa2ec044020 con 0x7fa2f41615f0 2024-05-09T03:20:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.326+0000 7fa2dbfff640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2e00762d0 0x7fa2e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.326+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fa2ec0b89c0 con 0x7fa2f41615f0 2024-05-09T03:20:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.326+0000 7fa2f2ffd640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2e00762d0 0x7fa2e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.327+0000 7fa2f2ffd640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2e00762d0 0x7fa2e00787e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4003aa0 tx=0x7fa2e4009830 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.331+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa2ec0bae20 con 0x7fa2f41615f0 2024-05-09T03:20:48.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.559+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2f40d41b0 con 0x7fa2f41615f0 2024-05-09T03:20:48.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.560+0000 7fa2dbfff640 1 -- 172.21.15.46:0/4197215468 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 345 v345) ==== 95+0+5170 (secure 0 0 0) 0x7fa2ec04b030 con 0x7fa2f41615f0 2024-05-09T03:20:48.561 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:48.561 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":345,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[{"mdsmap":{"epoch":344,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:15:52.455242+0000","modified":"2024-05-09T03:20:43.024580+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-05-09T03:20:48.561 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 345 2024-05-09T03:20:48.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2e00762d0 msgr2=0x7fa2e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2e00762d0 0x7fa2e00787e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4003aa0 tx=0x7fa2e4009830 comp rx=0 tx=0).stop 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 msgr2=0x7fa2f4149630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 0x7fa2f4149630 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec02a220 tx=0x7fa2ec0046c0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 shutdown_connections 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa2e00762d0 0x7fa2e00787e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f41615f0 0x7fa2f4149630 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.562+0000 7fa2f995f640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa2f40d25e0 0x7fa2f415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.563+0000 7fa2f995f640 1 --2- 172.21.15.46:0/4197215468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f40d0b80 0x7fa2f414b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.563+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 >> 172.21.15.46:0/4197215468 conn(0x7fa2f406e190 msgr2=0x7fa2f4161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.563+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 shutdown_connections 2024-05-09T03:20:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.563+0000 7fa2f995f640 1 -- 172.21.15.46:0/4197215468 wait complete. 2024-05-09T03:20:48.575 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:20:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.688+0000 7f11f4335640 1 Processor -- start 2024-05-09T03:20:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.688+0000 7f11f4335640 1 -- start start 2024-05-09T03:20:48.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f4335640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec0b5210 0x7f11ec1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f4335640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11ec0b7ed0 0x7f11ec0d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f4335640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11ec0d5280 con 0x7f11ec0d4800 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f4335640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f11ec0b5620 con 0x7f11ec0b7ed0 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f4335640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec1609a0 con 0x7f11ec0b5210 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f18d0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11ec0d4800 0x7f11ec0b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48518/0 (socket says 172.21.15.46:48518) 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f18d0640 1 -- 172.21.15.46:0/3065939160 learned_addr learned my addr 172.21.15.46:0/3065939160 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f28d2640 1 --2- 172.21.15.46:0/3065939160 >> v2:172.21.15.88:3300/0 conn(0x7f11ec0b7ed0 0x7f11ec0d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec0b5210 0x7f11ec1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961145048 0 0) 0x7f11ec0d5280 con 0x7f11ec0d4800 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11d8003520 con 0x7f11ec0d4800 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3128247796 0 0) 0x7f11d8003520 con 0x7f11ec0d4800 2024-05-09T03:20:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec0b5210 msgr2=0x7f11ec1613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.690+0000 7f11f10cf640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec0b5210 0x7f11ec1613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 >> v2:172.21.15.88:3300/0 conn(0x7f11ec0b7ed0 msgr2=0x7f11ec0d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 --2- 172.21.15.46:0/3065939160 >> v2:172.21.15.88:3300/0 conn(0x7f11ec0b7ed0 0x7f11ec0d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec0d5560 con 0x7f11ec0d4800 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f11dc0016f0 con 0x7f11ec0d4800 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 >> v1:172.21.15.46:6789/0 conn(0x7f11ec0d4800 legacy=0x7f11ec0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d8003a00 0x7f11d8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11d8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d8005150 0x7f11d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec0b5620 con 0x7f11d8004800 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1609a0 con 0x7f11d8003a00 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11d800c0b0 con 0x7f11d8005150 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec0b5210 0x7f11ec1613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:48.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f28d2640 1 --2- 172.21.15.46:0/3065939160 >> v2:172.21.15.88:3300/0 conn(0x7f11ec0b7ed0 0x7f11ec0d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11d8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d8003a00 0x7f11d8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f28d2640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d8005150 0x7f11d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f18d0640 1 -- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d8005150 msgr2=0x7f11d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11d8005150 0x7f11d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f18d0640 1 -- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d8003a00 msgr2=0x7f11d8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d8003a00 0x7f11d8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f18d0640 1 -- 172.21.15.46:0/3065939160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d8003520 con 0x7f11d8004800 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.691+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11d8003a00 0x7f11d8003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.692+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11d8004c10 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06ee20 tx=0x7f11d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b337ae68cc5aea60 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.692+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc007d50 con 0x7f11d8004800 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.692+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11dc005ad0 con 0x7f11d8004800 2024-05-09T03:20:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.692+0000 7f11f10cf640 1 -- 172.21.15.46:0/3065939160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc007070 con 0x7f11d8004800 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f4335640 1 -- 172.21.15.46:0/3065939160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 msgr2=0x7f11d8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f4335640 1 --2- 172.21.15.46:0/3065939160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11d8004c10 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06ee20 tx=0x7f11d8005830 comp rx=0 tx=0).stop 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f28d2640 1 -- 172.21.15.46:0/3065939160 reap_dead start 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f4335640 1 -- 172.21.15.46:0/3065939160 shutdown_connections 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f4335640 1 -- 172.21.15.46:0/3065939160 >> 172.21.15.46:0/3065939160 conn(0x7f11ec0d29a0 msgr2=0x7f11ec0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f4335640 1 -- 172.21.15.46:0/3065939160 shutdown_connections 2024-05-09T03:20:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.693+0000 7f11f4335640 1 -- 172.21.15.46:0/3065939160 wait complete. 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.695+0000 7f11f4335640 1 Processor -- start 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.695+0000 7f11f4335640 1 -- start start 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f4335640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11ec0cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f4335640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec1f6500 0x7f11ec1f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f4335640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11ec0cceb0 0x7f11ec0cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f4335640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec0d0ae0 con 0x7f11d8004800 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f4335640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec160ea0 con 0x7f11ec0cceb0 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f4335640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec1613c0 con 0x7f11ec1f6500 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11ec0cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11ec0cecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40324/0 (socket says 172.21.15.46:40324) 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 -- 172.21.15.46:0/3291947001 learned_addr learned my addr 172.21.15.46:0/3291947001 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f28d2640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11ec0cceb0 0x7f11ec0cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec1f6500 0x7f11ec1f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 -- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec1f6500 msgr2=0x7f11ec1f6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec1f6500 0x7f11ec1f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 -- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11ec0cceb0 msgr2=0x7f11ec0cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11ec0cceb0 0x7f11ec0cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f20d1640 1 -- 172.21.15.46:0/3291947001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec0cdc30 con 0x7f11d8004800 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.696+0000 7f11f28d2640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11ec0cceb0 0x7f11ec0cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.697+0000 7f11f20d1640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11ec0cecd0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f11e80116b0 tx=0x7f11e8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.697+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e800f9c0 con 0x7f11d8004800 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.697+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e800fb60 con 0x7f11d8004800 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.697+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec0cdf10 con 0x7f11d8004800 2024-05-09T03:20:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.697+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e8015460 con 0x7f11d8004800 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.697+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec0ce4d0 con 0x7f11d8004800 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.698+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ec06a270 con 0x7f11d8004800 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.698+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f11e8015600 con 0x7f11d8004800 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.699+0000 7f11e2ffd640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11d0076300 0x7f11d0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.699+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7f11e8098e50 con 0x7f11d8004800 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.699+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11d0076300 0x7f11d0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.700+0000 7f11f18d0640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11d0076300 0x7f11d0078810 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f11dc032000 tx=0x7f11dc009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:48.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.704+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11e8019030 con 0x7f11d8004800 2024-05-09T03:20:48.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.893+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11ec0d40d0 con 0x7f11d8004800 2024-05-09T03:20:48.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.894+0000 7f11e2ffd640 1 -- 172.21.15.46:0/3291947001 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+21866 (secure 0 0 0) 0x7f11e80618c0 con 0x7f11d8004800 2024-05-09T03:20:48.895 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:48.896 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":124,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:20:43.024284+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:15:50.144137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-05-09T03:15:51.401672+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.20","mappings":[{"from":1,"to":5}]},{"pgid":"13.24","mappings":[{"from":6,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/1514520479":"2024-05-10T03:20:43.024261+0000","172.21.15.46:6834/2034185175":"2024-05-10T03:20:36.108968+0000","172.21.15.46:6832/2034185175":"2024-05-10T03:20:36.108968+0000","172.21.15.46:6834/2183116503":"2024-05-10T03:20:24.715744+0000","172.21.15.46:6832/2183116503":"2024-05-10T03:20:24.715744+0000","172.21.15.46:6834/134300288":"2024-05-10T03:20:09.183188+0000","172.21.15.46:6832/134300288":"2024-05-10T03:20:09.183188+0000","172.21.15.46:6834/1774919282":"2024-05-10T03:20:03.819873+0000","172.21.15.46:6832/2209678569":"2024-05-10T03:19:52.097765+0000","172.21.15.46:6834/3872337434":"2024-05-10T03:19:35.738958+0000","172.21.15.46:6834/1350199591":"2024-05-10T03:19:30.622459+0000","172.21.15.46:6832/574897435":"2024-05-10T03:20:30.067507+0000","172.21.15.46:6832/3872337434":"2024-05-10T03:19:35.738958+0000","172.21.15.46:6832/925919281":"2024-05-10T03:19:16.273718+0000","172.21.15.46:6834/120572030":"2024-05-10T03:19:12.242288+0000","172.21.15.46:6832/1774919282":"2024-05-10T03:20:03.819873+0000","172.21.15.46:6832/3450505404":"2024-05-10T03:17:52.032707+0000","172.21.15.46:6832/1270051447":"2024-05-10T03:17:29.198075+0000","172.21.15.46:6832/2388442105":"2024-05-10T03:17:44.993790+0000","172.21.15.46:6834/3968811793":"2024-05-10T03:17:16.129644+0000","172.21.15.46:6834/925919281":"2024-05-10T03:19:16.273718+0000","172.21.15.46:6834/1270051447":"2024-05-10T03:17:29.198075+0000","172.21.15.46:6834/2860475391":"2024-05-10T03:16:44.302289+0000","172.21.15.46:6832/3044304069":"2024-05-10T03:17:09.735805+0000","172.21.15.46:6832/1514520479":"2024-05-10T03:20:43.024261+0000","172.21.15.46:6834/2566829830":"2024-05-10T03:16:58.040696+0000","172.21.15.46:6832/3968811793":"2024-05-10T03:17:16.129644+0000","172.21.15.46:6834/2032086476":"2024-05-10T03:17:22.988689+0000","172.21.15.46:6832/1350199591":"2024-05-10T03:19:30.622459+0000","172.21.15.46:6834/2546691383":"2024-05-10T03:18:27.254542+0000","172.21.15.46:6834/2347304023":"2024-05-10T03:16:08.753704+0000","172.21.15.46:6834/2388442105":"2024-05-10T03:17:44.993790+0000","172.21.15.46:6834/1234333269":"2024-05-10T03:17:58.071311+0000","172.21.15.46:6832/2827001018":"2024-05-10T03:19:01.862132+0000","172.21.15.46:6834/2585438921":"2024-05-10T03:18:23.220415+0000","172.21.15.46:6832/2789195040":"2024-05-10T03:18:36.012259+0000","172.21.15.46:6832/1281653036":"2024-05-10T03:16:24.681981+0000","172.21.15.46:6834/2209678569":"2024-05-10T03:19:52.097765+0000","172.21.15.46:6832/1486786164":"2024-05-10T03:17:42.979548+0000","172.21.15.46:6832/2032086476":"2024-05-10T03:17:22.988689+0000","172.21.15.46:6834/3044304069":"2024-05-10T03:17:09.735805+0000","172.21.15.46:6834/2490064583":"2024-05-10T03:18:44.067190+0000","172.21.15.46:6832/782052086":"2024-05-10T03:16:40.263688+0000","172.21.15.46:6832/2566829830":"2024-05-10T03:16:58.040696+0000","172.21.15.46:6834/1281653036":"2024-05-10T03:16:24.681981+0000","172.21.15.46:6834/3297564121":"2024-05-10T03:16:36.237612+0000","172.21.15.46:6832/1234333269":"2024-05-10T03:17:58.071311+0000","172.21.15.46:6832/1033379302":"2024-05-10T03:19:04.179713+0000","172.21.15.46:6832/271467610":"2024-05-10T03:18:02.111947+0000","172.21.15.46:6834/782052086":"2024-05-10T03:16:40.263688+0000","172.21.15.46:6832/3297564121":"2024-05-10T03:16:36.237612+0000","172.21.15.46:6834/271467610":"2024-05-10T03:18:02.111947+0000","172.21.15.46:6832/436596417":"2024-05-10T03:18:06.147085+0000","172.21.15.46:6834/574897435":"2024-05-10T03:20:30.067507+0000","172.21.15.46:6832/905138228":"2024-05-10T03:19:08.214271+0000","172.21.15.46:6834/436596417":"2024-05-10T03:18:06.147085+0000","172.21.15.46:6834/1486786164":"2024-05-10T03:17:42.979548+0000","172.21.15.46:6834/3106969477":"2024-05-10T03:18:20.866657+0000","172.21.15.46:6832/2585438921":"2024-05-10T03:18:23.220415+0000","172.21.15.46:6834/3423945789":"2024-05-10T03:17:12.099885+0000","172.21.15.46:6832/3423945789":"2024-05-10T03:17:12.099885+0000","172.21.15.46:6832/1887013209":"2024-05-10T03:16:30.194822+0000","172.21.15.46:6832/2546691383":"2024-05-10T03:18:27.254542+0000","172.21.15.46:6832/202666403":"2024-05-10T03:18:31.290277+0000","172.21.15.46:6832/935847027":"2024-05-10T03:18:40.005228+0000","172.21.15.46:6834/3450505404":"2024-05-10T03:17:52.032707+0000","172.21.15.46:6832/2347304023":"2024-05-10T03:16:08.753704+0000","172.21.15.46:6834/202666403":"2024-05-10T03:18:31.290277+0000","172.21.15.46:6834/2789195040":"2024-05-10T03:18:36.012259+0000","172.21.15.46:6834/935847027":"2024-05-10T03:18:40.005228+0000","172.21.15.46:6834/905138228":"2024-05-10T03:19:08.214271+0000","172.21.15.46:6832/2490064583":"2024-05-10T03:18:44.067190+0000","172.21.15.46:6834/2827001018":"2024-05-10T03:19:01.862132+0000","172.21.15.46:6832/2638610471":"2024-05-10T03:18:48.102832+0000","172.21.15.46:6832/2860475391":"2024-05-10T03:16:44.302289+0000","172.21.15.46:6832/3106969477":"2024-05-10T03:18:20.866657+0000","172.21.15.46:6834/2638610471":"2024-05-10T03:18:48.102832+0000","172.21.15.46:6834/1033379302":"2024-05-10T03:19:04.179713+0000","172.21.15.46:6834/1887013209":"2024-05-10T03:16:30.194822+0000","172.21.15.46:6832/120572030":"2024-05-10T03:19:12.242288+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:20:48.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11d0076300 msgr2=0x7f11d0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11d0076300 0x7f11d0078810 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f11dc032000 tx=0x7f11dc009030 comp rx=0 tx=0).stop 2024-05-09T03:20:48.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 msgr2=0x7f11ec0cecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11ec0cecd0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f11e80116b0 tx=0x7f11e8011b70 comp rx=0 tx=0).stop 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 shutdown_connections 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11ec0cceb0 0x7f11ec0cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f11ec1f6500 0x7f11ec1f6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11d0076300 0x7f11d0078810 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 --2- 172.21.15.46:0/3291947001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d8004800 0x7f11ec0cecd0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.896+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 >> 172.21.15.46:0/3291947001 conn(0x7f11ec0d29a0 msgr2=0x7f11ec0cb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.897+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 shutdown_connections 2024-05-09T03:20:48.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:48.897+0000 7f11f4335640 1 -- 172.21.15.46:0/3291947001 wait complete. 2024-05-09T03:20:48.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:20:49.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.024+0000 7fafe303b640 1 Processor -- start 2024-05-09T03:20:49.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.024+0000 7fafe303b640 1 -- start start 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe303b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafdc0c78a0 0x7fafdc0b1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe303b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc072500 0x7fafdc163e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe303b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafdc164390 0x7fafdc1461a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe303b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fafdc145d10 con 0x7fafdc0c78a0 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe303b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fafdc0c8720 con 0x7fafdc164390 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe303b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fafdc0c7e80 con 0x7fafdc072500 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.025+0000 7fafe0dd7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafdc0c78a0 0x7fafdc0b1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafdc0c78a0 0x7fafdc0b1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40330/0 (socket says 172.21.15.46:40330) 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 -- 172.21.15.46:0/2979172084 learned_addr learned my addr 172.21.15.46:0/2979172084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe15d8640 1 --2- 172.21.15.46:0/2979172084 >> v2:172.21.15.88:3300/0 conn(0x7fafdc164390 0x7fafdc1461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafd3fff640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc072500 0x7fafdc163e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 -- 172.21.15.46:0/2979172084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc072500 msgr2=0x7fafdc163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc072500 0x7fafdc163e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 -- 172.21.15.46:0/2979172084 >> v2:172.21.15.88:3300/0 conn(0x7fafdc164390 msgr2=0x7fafdc1461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 --2- 172.21.15.46:0/2979172084 >> v2:172.21.15.88:3300/0 conn(0x7fafdc164390 0x7fafdc1461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe0dd7640 1 -- 172.21.15.46:0/2979172084 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc0c8160 con 0x7fafdc0c78a0 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafe15d8640 1 --2- 172.21.15.46:0/2979172084 >> v2:172.21.15.88:3300/0 conn(0x7fafdc164390 0x7fafdc1461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.026+0000 7fafd3fff640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc072500 0x7fafdc163e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.027+0000 7fafe0dd7640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc0c78a0 0x7fafdc0b1dd0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fafdc074250 tx=0x7fafd802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece5d91e988f2373 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.027+0000 7fafd37fe640 1 -- 172.21.15.46:0/2979172084 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd8036070 con 0x7fafdc0c78a0 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.027+0000 7fafd37fe640 1 -- 172.21.15.46:0/2979172084 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd8031600 con 0x7fafdc0c78a0 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.027+0000 7fafd37fe640 1 -- 172.21.15.46:0/2979172084 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd80319d0 con 0x7fafdc0c78a0 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.027+0000 7fafe303b640 1 -- 172.21.15.46:0/2979172084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc0c78a0 msgr2=0x7fafdc0b1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:49.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.027+0000 7fafe303b640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc0c78a0 0x7fafdc0b1dd0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fafdc074250 tx=0x7fafd802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 -- 172.21.15.46:0/2979172084 shutdown_connections 2024-05-09T03:20:49.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 --2- 172.21.15.46:0/2979172084 >> v2:172.21.15.88:3300/0 conn(0x7fafdc164390 0x7fafdc1461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc072500 0x7fafdc163e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 --2- 172.21.15.46:0/2979172084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc0c78a0 0x7fafdc0b1dd0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 -- 172.21.15.46:0/2979172084 >> 172.21.15.46:0/2979172084 conn(0x7fafdc0705c0 msgr2=0x7fafdc0709d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:49.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 -- 172.21.15.46:0/2979172084 shutdown_connections 2024-05-09T03:20:49.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.028+0000 7fafe303b640 1 -- 172.21.15.46:0/2979172084 wait complete. 2024-05-09T03:20:49.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.029+0000 7fafe303b640 1 Processor -- start 2024-05-09T03:20:49.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.030+0000 7fafe303b640 1 -- start start 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe303b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc072500 0x7fafdc1fa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe303b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0c78a0 0x7fafdc0cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe303b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 0x7fafdc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe303b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1466e0 con 0x7fafdc164390 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe303b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc072910 con 0x7fafdc072500 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe303b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fafdc146860 con 0x7fafdc0c78a0 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 0x7fafdc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 0x7fafdc0b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40346/0 (socket says 172.21.15.46:40346) 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 -- 172.21.15.46:0/2651286737 learned_addr learned my addr 172.21.15.46:0/2651286737 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:49.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe0dd7640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc072500 0x7fafdc1fa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafd3fff640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0c78a0 0x7fafdc0cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 -- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0c78a0 msgr2=0x7fafdc0cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0c78a0 0x7fafdc0cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 -- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc072500 msgr2=0x7fafdc1fa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc072500 0x7fafdc1fa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.031+0000 7fafe15d8640 1 -- 172.21.15.46:0/2651286737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc0cbfc0 con 0x7fafdc164390 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.032+0000 7fafe15d8640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 0x7fafdc0b3590 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c5e0 tx=0x7fafd400caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.032+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4015070 con 0x7fafdc164390 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.032+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd4004240 con 0x7fafdc164390 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.032+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafdc1f6a20 con 0x7fafdc164390 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.032+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003610 con 0x7fafdc164390 2024-05-09T03:20:49.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.032+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafdc1f6fe0 con 0x7fafdc164390 2024-05-09T03:20:49.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.033+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafdc069250 con 0x7fafdc164390 2024-05-09T03:20:49.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.034+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fafd40037b0 con 0x7fafdc164390 2024-05-09T03:20:49.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.034+0000 7fafd17fa640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafc8076300 0x7fafc8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:49.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.034+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fafd4098380 con 0x7fafdc164390 2024-05-09T03:20:49.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.034+0000 7fafe0dd7640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafc8076300 0x7fafc8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:49.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.035+0000 7fafe0dd7640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafc8076300 0x7fafc8078810 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fafd8033000 tx=0x7fafd8009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:49.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.039+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fafdc069250 con 0x7fafdc164390 2024-05-09T03:20:49.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:49.277+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fafdc06a270 con 0x7fafdc164390 2024-05-09T03:20:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.158+0000 7fafd17fa640 1 -- 172.21.15.46:0/2651286737 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v346) ==== 161+0+0 (secure 0 0 0) 0x7fafd40608b0 con 0x7fafdc164390 2024-05-09T03:20:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:20:50.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafc8076300 msgr2=0x7fafc8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:50.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafc8076300 0x7fafc8078810 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fafd8033000 tx=0x7fafd8009000 comp rx=0 tx=0).stop 2024-05-09T03:20:50.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 msgr2=0x7fafdc0b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:50.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 0x7fafdc0b3590 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c5e0 tx=0x7fafd400caa0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 shutdown_connections 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fafc8076300 0x7fafc8078810 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafdc164390 0x7fafdc0b3590 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fafdc0c78a0 0x7fafdc0cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 --2- 172.21.15.46:0/2651286737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafdc072500 0x7fafdc1fa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 >> 172.21.15.46:0/2651286737 conn(0x7fafdc0705c0 msgr2=0x7fafdc0ceac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 shutdown_connections 2024-05-09T03:20:50.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.161+0000 7fafe303b640 1 -- 172.21.15.46:0/2651286737 wait complete. 2024-05-09T03:20:50.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:20:50.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.299+0000 7fb601415640 1 Processor -- start 2024-05-09T03:20:50.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.299+0000 7fb601415640 1 -- start start 2024-05-09T03:20:50.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb601415640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc164bf0 0x7fb5fc0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:50.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb601415640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5fc0d0fb0 0x7fb5fc0b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:50.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb601415640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5fc0d17a0 con 0x7fb5fc0b52b0 2024-05-09T03:20:50.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb601415640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb5fc0ba1a0 con 0x7fb5fc0d0fb0 2024-05-09T03:20:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb601415640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb5fc164760 con 0x7fb5fc164bf0 2024-05-09T03:20:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5fc0d0fb0 0x7fb5fc0b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5fc0d0fb0 0x7fb5fc0b7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40348/0 (socket says 172.21.15.46:40348) 2024-05-09T03:20:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 -- 172.21.15.46:0/3208382661 learned_addr learned my addr 172.21.15.46:0/3208382661 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5faffd640 1 --2- 172.21.15.46:0/3208382661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc164bf0 0x7fb5fc0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:50.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 -- 172.21.15.46:0/3208382661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc164bf0 msgr2=0x7fb5fc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:50.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 --2- 172.21.15.46:0/3208382661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc164bf0 0x7fb5fc0d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 -- 172.21.15.46:0/3208382661 >> v1:172.21.15.88:6789/0 conn(0x7fb5fc0b52b0 legacy=0x7fb5fc0d47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:50.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.301+0000 7fb5fa7fc640 1 -- 172.21.15.46:0/3208382661 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5fc0d14c0 con 0x7fb5fc0d0fb0 2024-05-09T03:20:50.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.302+0000 7fb5fa7fc640 1 --2- 172.21.15.46:0/3208382661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc0d0fb0 0x7fb5fc0b7740 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009870 tx=0x7fb5f002a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63412222054cf1e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.302+0000 7fb5f9ffb640 1 -- 172.21.15.46:0/3208382661 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0037070 con 0x7fb5fc0d0fb0 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.302+0000 7fb5f9ffb640 1 -- 172.21.15.46:0/3208382661 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f0032440 con 0x7fb5fc0d0fb0 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.302+0000 7fb5f9ffb640 1 -- 172.21.15.46:0/3208382661 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f0032810 con 0x7fb5fc0d0fb0 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 -- 172.21.15.46:0/3208382661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc0d0fb0 msgr2=0x7fb5fc0b7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 --2- 172.21.15.46:0/3208382661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc0d0fb0 0x7fb5fc0b7740 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0009870 tx=0x7fb5f002a240 comp rx=0 tx=0).stop 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 -- 172.21.15.46:0/3208382661 shutdown_connections 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 --2- 172.21.15.46:0/3208382661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc0d0fb0 0x7fb5fc0b7740 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 --2- 172.21.15.46:0/3208382661 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc164bf0 0x7fb5fc0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 -- 172.21.15.46:0/3208382661 >> 172.21.15.46:0/3208382661 conn(0x7fb5fc0d2920 msgr2=0x7fb5fc0d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:50.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 -- 172.21.15.46:0/3208382661 shutdown_connections 2024-05-09T03:20:50.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.303+0000 7fb601415640 1 -- 172.21.15.46:0/3208382661 wait complete. 2024-05-09T03:20:50.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.305+0000 7fb601415640 1 Processor -- start 2024-05-09T03:20:50.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.305+0000 7fb601415640 1 -- start start 2024-05-09T03:20:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.306+0000 7fb601415640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5fc0b52b0 0x7fb5fc162af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.306+0000 7fb601415640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc0d0fb0 0x7fb5fc15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.306+0000 7fb601415640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 0x7fb5fc0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.306+0000 7fb601415640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc06eb30 con 0x7fb5fc164bf0 2024-05-09T03:20:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.306+0000 7fb601415640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb5fc1643f0 con 0x7fb5fc0b52b0 2024-05-09T03:20:50.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.306+0000 7fb601415640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb5fc06ecb0 con 0x7fb5fc0d0fb0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 0x7fb5fc0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 0x7fb5fc0cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40350/0 (socket says 172.21.15.46:40350) 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 -- 172.21.15.46:0/3717305556 learned_addr learned my addr 172.21.15.46:0/3717305556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fa7fc640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc0d0fb0 0x7fb5fc15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5faffd640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5fc0b52b0 0x7fb5fc162af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 -- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc0d0fb0 msgr2=0x7fb5fc15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc0d0fb0 0x7fb5fc15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 -- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5fc0b52b0 msgr2=0x7fb5fc162af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5fc0b52b0 0x7fb5fc162af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 -- 172.21.15.46:0/3717305556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5fc1ec260 con 0x7fb5fc164bf0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5fb7fe640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 0x7fb5fc0cebd0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c370 tx=0x7fb5f400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4017070 con 0x7fb5fc164bf0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f40052f0 con 0x7fb5fc164bf0 2024-05-09T03:20:50.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5fc1ec540 con 0x7fb5fc164bf0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.307+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4003850 con 0x7fb5fc164bf0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.308+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5fc0cf6d0 con 0x7fb5fc164bf0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.308+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5fc1ebfb0 con 0x7fb5fc164bf0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.309+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb5f401a070 con 0x7fb5fc164bf0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.310+0000 7fb5e37fe640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb5e8076300 0x7fb5e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.310+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7fb5f40979e0 con 0x7fb5fc164bf0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.310+0000 7fb5faffd640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb5e8076300 0x7fb5e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:50.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.311+0000 7fb5faffd640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb5e8076300 0x7fb5e8078810 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec003aa0 tx=0x7fb5ec009830 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:50.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.314+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb5f4021020 con 0x7fb5fc164bf0 2024-05-09T03:20:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:50.546+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb5fc06a270 con 0x7fb5fc164bf0 2024-05-09T03:20:51.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.167+0000 7fb5e37fe640 1 -- 172.21.15.46:0/3717305556 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v347) ==== 104+0+0 (secure 0 0 0) 0x7fb5f405ff70 con 0x7fb5fc164bf0 2024-05-09T03:20:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb5e8076300 msgr2=0x7fb5e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb5e8076300 0x7fb5e8078810 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec003aa0 tx=0x7fb5ec009830 comp rx=0 tx=0).stop 2024-05-09T03:20:51.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 msgr2=0x7fb5fc0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 0x7fb5fc0cebd0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c370 tx=0x7fb5f400c830 comp rx=0 tx=0).stop 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 shutdown_connections 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb5e8076300 0x7fb5e8078810 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5fc164bf0 0x7fb5fc0cebd0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb5fc0d0fb0 0x7fb5fc15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 --2- 172.21.15.46:0/3717305556 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5fc0b52b0 0x7fb5fc162af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 >> 172.21.15.46:0/3717305556 conn(0x7fb5fc0d2920 msgr2=0x7fb5fc15ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 shutdown_connections 2024-05-09T03:20:51.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.170+0000 7fb601415640 1 -- 172.21.15.46:0/3717305556 wait complete. 2024-05-09T03:20:51.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:20:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.310+0000 7f77cbc9a640 1 Processor -- start 2024-05-09T03:20:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.311+0000 7f77cbc9a640 1 -- start start 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77cbc9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b8240 0x7f77c406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77cbc9a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77c406e250 0x7f77c40b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77cbc9a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77c406e660 con 0x7f77c40b5140 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77cbc9a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f77c40b9b60 con 0x7f77c406e250 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77cbc9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77c40b5550 con 0x7f77c40b8240 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77ca237640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77c406e250 0x7f77c40b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77ca237640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f77c406e250 0x7f77c40b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40354/0 (socket says 172.21.15.46:40354) 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77ca237640 1 -- 172.21.15.46:0/476924802 learned_addr learned my addr 172.21.15.46:0/476924802 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.312+0000 7f77c9235640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b8240 0x7f77c406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.313+0000 7f77ca237640 1 -- 172.21.15.46:0/476924802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b8240 msgr2=0x7f77c406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.313+0000 7f77ca237640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b8240 0x7f77c406dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.313+0000 7f77ca237640 1 -- 172.21.15.46:0/476924802 >> v1:172.21.15.88:6789/0 conn(0x7f77c40b5140 legacy=0x7f77c40b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:51.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.313+0000 7f77ca237640 1 -- 172.21.15.46:0/476924802 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c40b9270 con 0x7f77c406e250 2024-05-09T03:20:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.313+0000 7f77c9235640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b8240 0x7f77c406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:51.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.313+0000 7f77ca237640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c40b9ff0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009870 tx=0x7f77bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf7a879a6d0a2c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77c8a34640 1 -- 172.21.15.46:0/476924802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc037070 con 0x7f77c406e250 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77c8a34640 1 -- 172.21.15.46:0/476924802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77bc032440 con 0x7f77c406e250 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77c8a34640 1 -- 172.21.15.46:0/476924802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc032810 con 0x7f77c406e250 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 -- 172.21.15.46:0/476924802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 msgr2=0x7f77c40b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c40b9ff0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009870 tx=0x7f77bc02a270 comp rx=0 tx=0).stop 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 -- 172.21.15.46:0/476924802 shutdown_connections 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c40b9ff0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/476924802 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b8240 0x7f77c406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 -- 172.21.15.46:0/476924802 >> 172.21.15.46:0/476924802 conn(0x7f77c40d5e90 msgr2=0x7f77c40d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 -- 172.21.15.46:0/476924802 shutdown_connections 2024-05-09T03:20:51.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.314+0000 7f77cbc9a640 1 -- 172.21.15.46:0/476924802 wait complete. 2024-05-09T03:20:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.316+0000 7f77cbc9a640 1 Processor -- start 2024-05-09T03:20:51.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.316+0000 7f77cbc9a640 1 -- start start 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.317+0000 7f77cbc9a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c414e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.317+0000 7f77cbc9a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b5140 0x7f77c414fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.317+0000 7f77cbc9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c4150330 0x7f77c40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77cbc9a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77c40b97f0 con 0x7f77c406e250 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77cbc9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f77c40b9510 con 0x7f77c4150330 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77cbc9a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f77c40ba530 con 0x7f77c40b5140 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c414e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:51.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c414e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40370/0 (socket says 172.21.15.46:40370) 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 -- 172.21.15.46:0/3871512781 learned_addr learned my addr 172.21.15.46:0/3871512781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9235640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b5140 0x7f77c414fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77ca237640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c4150330 0x7f77c40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 -- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b5140 msgr2=0x7f77c414fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b5140 0x7f77c414fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 -- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c4150330 msgr2=0x7f77c40ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c4150330 0x7f77c40ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 -- 172.21.15.46:0/3871512781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c40b4660 con 0x7f77c406e250 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77c9a36640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c414e9d0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c760 tx=0x7f77c000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:51.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0003b70 con 0x7f77c406e250 2024-05-09T03:20:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c0003d30 con 0x7f77c406e250 2024-05-09T03:20:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c000aa90 con 0x7f77c406e250 2024-05-09T03:20:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c4150b60 con 0x7f77c406e250 2024-05-09T03:20:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.318+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c40b1ef0 con 0x7f77c406e250 2024-05-09T03:20:51.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.319+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c40b2330 con 0x7f77c406e250 2024-05-09T03:20:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.320+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f77c000ac30 con 0x7f77c406e250 2024-05-09T03:20:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.321+0000 7f77ba7fc640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77a80762d0 0x7f77a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.321+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9207+0+0 (secure 0 0 0) 0x7f77c00982d0 con 0x7f77c406e250 2024-05-09T03:20:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.321+0000 7f77c9235640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77a80762d0 0x7f77a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:51.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.322+0000 7f77c9235640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77a80762d0 0x7f77a80787e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f77c40b6390 tx=0x7f77b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:51.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.325+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f77c0060e60 con 0x7f77c406e250 2024-05-09T03:20:51.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:51.515+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f77c406a270 con 0x7f77c406e250 2024-05-09T03:20:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.169+0000 7f77ba7fc640 1 -- 172.21.15.46:0/3871512781 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v125) ==== 181+0+0 (secure 0 0 0) 0x7f77c0060760 con 0x7f77c406e250 2024-05-09T03:20:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:20:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77a80762d0 msgr2=0x7f77a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77a80762d0 0x7f77a80787e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f77c40b6390 tx=0x7f77b4008040 comp rx=0 tx=0).stop 2024-05-09T03:20:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 msgr2=0x7f77c414e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c414e9d0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c760 tx=0x7f77c000cc20 comp rx=0 tx=0).stop 2024-05-09T03:20:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 shutdown_connections 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77c4150330 0x7f77c40ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f77c40b5140 0x7f77c414fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f77a80762d0 0x7f77a80787e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 --2- 172.21.15.46:0/3871512781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77c406e250 0x7f77c414e9d0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.173+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 >> 172.21.15.46:0/3871512781 conn(0x7f77c40d5e90 msgr2=0x7f77c40b64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.174+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 shutdown_connections 2024-05-09T03:20:52.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.174+0000 7f77cbc9a640 1 -- 172.21.15.46:0/3871512781 wait complete. 2024-05-09T03:20:52.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:20:52.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.316+0000 7f021c08b640 1 Processor -- start 2024-05-09T03:20:52.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.316+0000 7f021c08b640 1 -- start start 2024-05-09T03:20:52.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021c08b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02140b5250 0x7f02140d2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021c08b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d05c0 0x7f02140d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021c08b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0214162020 0x7f0214162430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021c08b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f02140d0f10 con 0x7f02140b5250 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021c08b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f02140cebf0 con 0x7f0214162020 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021c08b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02140ceed0 con 0x7f02140d05c0 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f0219e27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02140b5250 0x7f02140d2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f0219e27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02140b5250 0x7f02140d2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40384/0 (socket says 172.21.15.46:40384) 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f0219e27640 1 -- 172.21.15.46:0/1958530352 learned_addr learned my addr 172.21.15.46:0/1958530352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f021a628640 1 --2- 172.21.15.46:0/1958530352 >> v2:172.21.15.88:3300/0 conn(0x7f0214162020 0x7f0214162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.317+0000 7f0219626640 1 --2- 172.21.15.46:0/1958530352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d05c0 0x7f02140d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.318+0000 7f0219e27640 1 -- 172.21.15.46:0/1958530352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d05c0 msgr2=0x7f02140d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.318+0000 7f0219e27640 1 --2- 172.21.15.46:0/1958530352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d05c0 0x7f02140d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.318+0000 7f0219e27640 1 -- 172.21.15.46:0/1958530352 >> v2:172.21.15.88:3300/0 conn(0x7f0214162020 msgr2=0x7f0214162430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:52.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.318+0000 7f0219e27640 1 --2- 172.21.15.46:0/1958530352 >> v2:172.21.15.88:3300/0 conn(0x7f0214162020 0x7f0214162430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.318+0000 7f0219e27640 1 -- 172.21.15.46:0/1958530352 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02140cf1b0 con 0x7f02140b5250 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.318+0000 7f0219e27640 1 --2- 172.21.15.46:0/1958530352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140b5250 0x7f02140d2650 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f02140b66d0 tx=0x7f021002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10380feb608ebf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f0218e25640 1 -- 172.21.15.46:0/1958530352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210037070 con 0x7f02140b5250 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f0218e25640 1 -- 172.21.15.46:0/1958530352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0210032440 con 0x7f02140b5250 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f0218e25640 1 -- 172.21.15.46:0/1958530352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210032810 con 0x7f02140b5250 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 -- 172.21.15.46:0/1958530352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140b5250 msgr2=0x7f02140d2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 --2- 172.21.15.46:0/1958530352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140b5250 0x7f02140d2650 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f02140b66d0 tx=0x7f021002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 -- 172.21.15.46:0/1958530352 shutdown_connections 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 --2- 172.21.15.46:0/1958530352 >> v2:172.21.15.88:3300/0 conn(0x7f0214162020 0x7f0214162430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 --2- 172.21.15.46:0/1958530352 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140d05c0 0x7f02140d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 --2- 172.21.15.46:0/1958530352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140b5250 0x7f02140d2650 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.319+0000 7f021c08b640 1 -- 172.21.15.46:0/1958530352 >> 172.21.15.46:0/1958530352 conn(0x7f02140d3bd0 msgr2=0x7f02140d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.320+0000 7f021c08b640 1 -- 172.21.15.46:0/1958530352 shutdown_connections 2024-05-09T03:20:52.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.320+0000 7f021c08b640 1 -- 172.21.15.46:0/1958530352 wait complete. 2024-05-09T03:20:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.321+0000 7f021c08b640 1 Processor -- start 2024-05-09T03:20:52.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.321+0000 7f021c08b640 1 -- start start 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021c08b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140b5250 0x7f02140cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021c08b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 0x7f021415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021c08b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0214162020 0x7f02140cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021c08b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02140b9460 con 0x7f02140d05c0 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021c08b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f02140cf770 con 0x7f0214162020 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021c08b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f02140b95e0 con 0x7f02140b5250 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 0x7f021415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 0x7f021415bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40394/0 (socket says 172.21.15.46:40394) 2024-05-09T03:20:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 -- 172.21.15.46:0/2304759626 learned_addr learned my addr 172.21.15.46:0/2304759626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219e27640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140b5250 0x7f02140cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021a628640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0214162020 0x7f02140cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 -- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140b5250 msgr2=0x7f02140cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140b5250 0x7f02140cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 -- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0214162020 msgr2=0x7f02140cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0214162020 0x7f02140cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 -- 172.21.15.46:0/2304759626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02140cd160 con 0x7f02140d05c0 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f021a628640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0214162020 0x7f02140cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f0219626640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 0x7f021415bec0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f020400e870 tx=0x7f020400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204017070 con 0x7f02140d05c0 2024-05-09T03:20:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020400ee60 con 0x7f02140d05c0 2024-05-09T03:20:52.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.323+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020400cb80 con 0x7f02140d05c0 2024-05-09T03:20:52.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.324+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02140cd440 con 0x7f02140d05c0 2024-05-09T03:20:52.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.324+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02140cc070 con 0x7f02140d05c0 2024-05-09T03:20:52.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.325+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f021406a270 con 0x7f02140d05c0 2024-05-09T03:20:52.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.325+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f020401a070 con 0x7f02140d05c0 2024-05-09T03:20:52.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.326+0000 7f020a7fc640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f80762d0 0x7f01f80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:52.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.326+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8769+0+0 (secure 0 0 0) 0x7f0204098370 con 0x7f02140d05c0 2024-05-09T03:20:52.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.327+0000 7f0219e27640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f80762d0 0x7f01f80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:52.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.327+0000 7f0219e27640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f80762d0 0x7f01f80787e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0210034000 tx=0x7f021002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:52.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.331+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0204060a30 con 0x7f02140d05c0 2024-05-09T03:20:52.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:52.522+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f02140cceb0 con 0x7f02140d05c0 2024-05-09T03:20:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.182+0000 7f020a7fc640 1 -- 172.21.15.46:0/2304759626 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v126) ==== 169+0+0 (secure 0 0 0) 0x7f0204064d30 con 0x7f02140d05c0 2024-05-09T03:20:53.183 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:20:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f80762d0 msgr2=0x7f01f80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f80762d0 0x7f01f80787e0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f0210034000 tx=0x7f021002e000 comp rx=0 tx=0).stop 2024-05-09T03:20:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 msgr2=0x7f021415bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 0x7f021415bec0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f020400e870 tx=0x7f020400ed30 comp rx=0 tx=0).stop 2024-05-09T03:20:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 shutdown_connections 2024-05-09T03:20:53.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0214162020 0x7f02140cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01f80762d0 0x7f01f80787e0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02140d05c0 0x7f021415bec0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 --2- 172.21.15.46:0/2304759626 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f02140b5250 0x7f02140cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.186+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 >> 172.21.15.46:0/2304759626 conn(0x7f02140d3bd0 msgr2=0x7f02140725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.187+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 shutdown_connections 2024-05-09T03:20:53.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.187+0000 7f021c08b640 1 -- 172.21.15.46:0/2304759626 wait complete. 2024-05-09T03:20:53.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T03:20:53.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.324+0000 7fc112b08640 1 Processor -- start 2024-05-09T03:20:53.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.325+0000 7fc112b08640 1 -- start start 2024-05-09T03:20:53.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc112b08640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc10c0d47a0 0x7fc10c15de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc112b08640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d0860 0x7fc10c1611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc112b08640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc10c0b7760 0x7fc10c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc112b08640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c0b80b0 con 0x7fc10c0b7760 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc112b08640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c0d10b0 con 0x7fc10c0d47a0 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc112b08640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c0d1390 con 0x7fc10c0d0860 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc10c0b7760 0x7fc10c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc10c0b7760 0x7fc10c0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40410/0 (socket says 172.21.15.46:40410) 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 -- 172.21.15.46:0/2646713694 learned_addr learned my addr 172.21.15.46:0/2646713694 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1108a4640 1 --2- 172.21.15.46:0/2646713694 >> v2:172.21.15.88:3300/0 conn(0x7fc10c0d47a0 0x7fc10c15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc10bfff640 1 --2- 172.21.15.46:0/2646713694 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d0860 0x7fc10c1611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 -- 172.21.15.46:0/2646713694 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d0860 msgr2=0x7fc10c1611c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 --2- 172.21.15.46:0/2646713694 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d0860 0x7fc10c1611c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 -- 172.21.15.46:0/2646713694 >> v2:172.21.15.88:3300/0 conn(0x7fc10c0d47a0 msgr2=0x7fc10c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 --2- 172.21.15.46:0/2646713694 >> v2:172.21.15.88:3300/0 conn(0x7fc10c0d47a0 0x7fc10c15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.326+0000 7fc1110a5640 1 -- 172.21.15.46:0/2646713694 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c0d1670 con 0x7fc10c0b7760 2024-05-09T03:20:53.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.327+0000 7fc1110a5640 1 --2- 172.21.15.46:0/2646713694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0b7b70 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fc10c08ea50 tx=0x7fc10402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a40564f42c56720 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.327+0000 7fc10b7fe640 1 -- 172.21.15.46:0/2646713694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104036070 con 0x7fc10c0b7760 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.327+0000 7fc10b7fe640 1 -- 172.21.15.46:0/2646713694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc104031600 con 0x7fc10c0b7760 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.327+0000 7fc10b7fe640 1 -- 172.21.15.46:0/2646713694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1040319d0 con 0x7fc10c0b7760 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 -- 172.21.15.46:0/2646713694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 msgr2=0x7fc10c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 --2- 172.21.15.46:0/2646713694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0b7b70 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fc10c08ea50 tx=0x7fc10402a610 comp rx=0 tx=0).stop 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 -- 172.21.15.46:0/2646713694 shutdown_connections 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 --2- 172.21.15.46:0/2646713694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0b7b70 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 --2- 172.21.15.46:0/2646713694 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d0860 0x7fc10c1611c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 --2- 172.21.15.46:0/2646713694 >> v2:172.21.15.88:3300/0 conn(0x7fc10c0d47a0 0x7fc10c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 -- 172.21.15.46:0/2646713694 >> 172.21.15.46:0/2646713694 conn(0x7fc10c0d2760 msgr2=0x7fc10c0d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 -- 172.21.15.46:0/2646713694 shutdown_connections 2024-05-09T03:20:53.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.328+0000 7fc112b08640 1 -- 172.21.15.46:0/2646713694 wait complete. 2024-05-09T03:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.330+0000 7fc112b08640 1 Processor -- start 2024-05-09T03:20:53.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.330+0000 7fc112b08640 1 -- start start 2024-05-09T03:20:53.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.331+0000 7fc112b08640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0cf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.331+0000 7fc112b08640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc10c0d0860 0x7fc10c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.331+0000 7fc112b08640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d47a0 0x7fc10c1530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.331+0000 7fc112b08640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c161880 con 0x7fc10c0b7760 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.331+0000 7fc112b08640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c161700 con 0x7fc10c0d0860 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.331+0000 7fc112b08640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c0d0c70 con 0x7fc10c0d47a0 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0cf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0cf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40422/0 (socket says 172.21.15.46:40422) 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 -- 172.21.15.46:0/805383006 learned_addr learned my addr 172.21.15.46:0/805383006 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1110a5640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d47a0 0x7fc10c1530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc10bfff640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc10c0d0860 0x7fc10c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 -- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d47a0 msgr2=0x7fc10c1530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d47a0 0x7fc10c1530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 -- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc10c0d0860 msgr2=0x7fc10c0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc10c0d0860 0x7fc10c0cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 -- 172.21.15.46:0/805383006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c0cb1d0 con 0x7fc10c0b7760 2024-05-09T03:20:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1108a4640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0cf8d0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00d5d0 tx=0x7fc0fc00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc004240 con 0x7fc10c0b7760 2024-05-09T03:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc0043e0 con 0x7fc10c0b7760 2024-05-09T03:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10c0cb4b0 con 0x7fc10c0b7760 2024-05-09T03:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc003830 con 0x7fc10c0b7760 2024-05-09T03:20:53.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.332+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10c0cba70 con 0x7fc10c0b7760 2024-05-09T03:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.333+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10c06a270 con 0x7fc10c0b7760 2024-05-09T03:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.334+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc0fc0039d0 con 0x7fc10c0b7760 2024-05-09T03:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.335+0000 7fc1097fa640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc0f80762d0 0x7fc0f80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.335+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7fc0fc098160 con 0x7fc10c0b7760 2024-05-09T03:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.335+0000 7fc10bfff640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc0f80762d0 0x7fc0f80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.335+0000 7fc10bfff640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc0f80762d0 0x7fc0f80787e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc10c15d590 tx=0x7fc100008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:53.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.339+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc0fc060a00 con 0x7fc10c0b7760 2024-05-09T03:20:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.533+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fc10c0d4070 con 0x7fc10c0b7760 2024-05-09T03:20:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.536+0000 7fc1097fa640 1 -- 172.21.15.46:0/805383006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fc0fc064d00 con 0x7fc10c0b7760 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc0f80762d0 msgr2=0x7fc0f80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc0f80762d0 0x7fc0f80787e0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc10c15d590 tx=0x7fc100008040 comp rx=0 tx=0).stop 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 msgr2=0x7fc10c0cf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0cf8d0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00d5d0 tx=0x7fc0fc00da90 comp rx=0 tx=0).stop 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 shutdown_connections 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc10c0d47a0 0x7fc10c1530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc10c0d0860 0x7fc10c0cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc0f80762d0 0x7fc0f80787e0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 --2- 172.21.15.46:0/805383006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc10c0b7760 0x7fc10c0cf8d0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 >> 172.21.15.46:0/805383006 conn(0x7fc10c0d2760 msgr2=0x7fc10c0b6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 shutdown_connections 2024-05-09T03:20:53.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.539+0000 7fc112b08640 1 -- 172.21.15.46:0/805383006 wait complete. 2024-05-09T03:20:53.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-05-09T03:20:53.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.666+0000 7fabe2054640 1 Processor -- start 2024-05-09T03:20:53.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.666+0000 7fabe2054640 1 -- start start 2024-05-09T03:20:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabe2054640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabdc0b5210 0x7fabdc1613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabe2054640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabdc0d5280 con 0x7fabdc0b7ed0 2024-05-09T03:20:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabe2054640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabdc0d5560 con 0x7fabdc0d4800 2024-05-09T03:20:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabe2054640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fabdc0b5620 con 0x7fabdc0b5210 2024-05-09T03:20:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabdb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fabdc0b5210 0x7fabdc1613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabdbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fabdc0b7ed0 0x7fabdc163e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48524/0 (socket says 172.21.15.46:48524) 2024-05-09T03:20:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.667+0000 7fabdbfff640 1 -- 172.21.15.46:0/2307064364 learned_addr learned my addr 172.21.15.46:0/2307064364 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.668+0000 7fabdb7fe640 1 -- 172.21.15.46:0/2307064364 >> v1:172.21.15.88:6789/0 conn(0x7fabdc0d4800 legacy=0x7fabdc0b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.668+0000 7fabdb7fe640 1 -- 172.21.15.46:0/2307064364 >> v1:172.21.15.46:6789/0 conn(0x7fabdc0b7ed0 legacy=0x7fabdc163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.668+0000 7fabdb7fe640 1 -- 172.21.15.46:0/2307064364 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc160bc0 con 0x7fabdc0b5210 2024-05-09T03:20:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.668+0000 7fabdb7fe640 1 --2- 172.21.15.46:0/2307064364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1613c0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0097a0 tx=0x7fabcc02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890a2250cc7f74e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabdaffd640 1 -- 172.21.15.46:0/2307064364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc037070 con 0x7fabdc0b5210 2024-05-09T03:20:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabdaffd640 1 -- 172.21.15.46:0/2307064364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabcc02adc0 con 0x7fabdc0b5210 2024-05-09T03:20:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabdaffd640 1 -- 172.21.15.46:0/2307064364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc032660 con 0x7fabdc0b5210 2024-05-09T03:20:53.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 -- 172.21.15.46:0/2307064364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 msgr2=0x7fabdc1613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 --2- 172.21.15.46:0/2307064364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1613c0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0097a0 tx=0x7fabcc02a130 comp rx=0 tx=0).stop 2024-05-09T03:20:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 -- 172.21.15.46:0/2307064364 shutdown_connections 2024-05-09T03:20:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 --2- 172.21.15.46:0/2307064364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1613c0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 -- 172.21.15.46:0/2307064364 >> 172.21.15.46:0/2307064364 conn(0x7fabdc0d29a0 msgr2=0x7fabdc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 -- 172.21.15.46:0/2307064364 shutdown_connections 2024-05-09T03:20:53.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.669+0000 7fabe2054640 1 -- 172.21.15.46:0/2307064364 wait complete. 2024-05-09T03:20:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.671+0000 7fabe2054640 1 Processor -- start 2024-05-09T03:20:53.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.671+0000 7fabe2054640 1 -- start start 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabe2054640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabe2054640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc0d4800 0x7fabdc15e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabe2054640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0b6930 0x7fabdc0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabe2054640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc0d0ae0 con 0x7fabdc0b5210 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabe2054640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc1609a0 con 0x7fabdc0d4800 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabe2054640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fabdc0d4c10 con 0x7fabdc0b6930 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40432/0 (socket says 172.21.15.46:40432) 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 -- 172.21.15.46:0/3379470906 learned_addr learned my addr 172.21.15.46:0/3379470906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabd2ffd640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc0d4800 0x7fabdc15e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdbfff640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0b6930 0x7fabdc0ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 -- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0b6930 msgr2=0x7fabdc0ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0b6930 0x7fabdc0ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 -- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc0d4800 msgr2=0x7fabdc15e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc0d4800 0x7fabdc15e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 -- 172.21.15.46:0/3379470906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc0cd1c0 con 0x7fabdc0b5210 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdbfff640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0b6930 0x7fabdc0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.673+0000 7fabdb7fe640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1eada0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0098c0 tx=0x7fabcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.674+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc042070 con 0x7fabdc0b5210 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.674+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabcc004250 con 0x7fabdc0b5210 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.674+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabdc0cd4a0 con 0x7fabdc0b5210 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.674+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc037040 con 0x7fabdc0b5210 2024-05-09T03:20:53.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.674+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabdc0cda60 con 0x7fabdc0b5210 2024-05-09T03:20:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.675+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabdc06a270 con 0x7fabdc0b5210 2024-05-09T03:20:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.675+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fabcc045070 con 0x7fabdc0b5210 2024-05-09T03:20:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.676+0000 7fabd8ff9640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc0762d0 0x7fabbc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.676+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7fabcc0b7c20 con 0x7fabdc0b5210 2024-05-09T03:20:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.676+0000 7fabd2ffd640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc0762d0 0x7fabbc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:53.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.677+0000 7fabd2ffd640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc0762d0 0x7fabbc0787e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fabdc15fd80 tx=0x7fabc4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:53.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.681+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fabcc080430 con 0x7fabdc0b5210 2024-05-09T03:20:53.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:53.953+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) -- 0x7fabdc0d3060 con 0x7fabdc0b5210 2024-05-09T03:20:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.184+0000 7fabd8ff9640 1 -- 172.21.15.46:0/3379470906 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v579) ==== 135+0+0 (secure 0 0 0) 0x7fabcc084730 con 0x7fabdc0b5210 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc0762d0 msgr2=0x7fabbc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc0762d0 0x7fabbc0787e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fabdc15fd80 tx=0x7fabc4009040 comp rx=0 tx=0).stop 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 msgr2=0x7fabdc1eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1eada0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0098c0 tx=0x7fabcc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 shutdown_connections 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fabdc0b6930 0x7fabdc0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabdc0d4800 0x7fabdc15e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fabbc0762d0 0x7fabbc0787e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 --2- 172.21.15.46:0/3379470906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fabdc0b5210 0x7fabdc1eada0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 >> 172.21.15.46:0/3379470906 conn(0x7fabdc0d29a0 msgr2=0x7fabdc1613c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 shutdown_connections 2024-05-09T03:20:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.186+0000 7fabe2054640 1 -- 172.21.15.46:0/3379470906 wait complete. 2024-05-09T03:20:54.199 INFO:tasks.cephfs_test_runner:That the MDS has a clean 'damaged' response to loss of any single metadata object ... ok 2024-05-09T03:20:54.199 INFO:tasks.cephfs_test_runner:Starting test: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) That errors encountered during opening inos are properly propagated 2024-05-09T03:20:54.199 INFO:tasks.cephfs_test_runner:test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-05-09T03:20:54.199 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:20:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.314+0000 7f52db689640 1 Processor -- start 2024-05-09T03:20:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.314+0000 7f52db689640 1 -- start start 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52db689640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52d40d2cf0 0x7f52d415f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52db689640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d40d15b0 con 0x7f52d40b81b0 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52db689640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d4160b10 con 0x7f52d41615f0 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52db689640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f52d4160df0 con 0x7f52d40d2cf0 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d8c24640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52d40b81b0 0x7f52d40d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48528/0 (socket says 172.21.15.46:48528) 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d8c24640 1 -- 172.21.15.46:0/1560015936 learned_addr learned my addr 172.21.15.46:0/1560015936 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d9c26640 1 --2- 172.21.15.46:0/1560015936 >> v2:172.21.15.88:3300/0 conn(0x7f52d40d2cf0 0x7f52d415f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d3fff640 1 -- 172.21.15.46:0/1560015936 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768230465 0 0) 0x7f52d40d15b0 con 0x7f52d40b81b0 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d3fff640 1 -- 172.21.15.46:0/1560015936 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52bc003520 con 0x7f52d40b81b0 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d9c26640 1 -- 172.21.15.46:0/1560015936 >> v1:172.21.15.88:6789/0 conn(0x7f52d41615f0 legacy=0x7f52d40d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d9c26640 1 -- 172.21.15.46:0/1560015936 >> v1:172.21.15.46:6789/0 conn(0x7f52d40b81b0 legacy=0x7f52d40d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.316+0000 7f52d9c26640 1 -- 172.21.15.46:0/1560015936 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d41610d0 con 0x7f52d40d2cf0 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.317+0000 7f52d9c26640 1 --2- 172.21.15.46:0/1560015936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d40d2cf0 0x7f52d415f990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0097a0 tx=0x7f52cc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a367273904ede7c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.317+0000 7f52d3fff640 1 -- 172.21.15.46:0/1560015936 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc037070 con 0x7f52d40d2cf0 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.317+0000 7f52d3fff640 1 -- 172.21.15.46:0/1560015936 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52cc032440 con 0x7f52d40d2cf0 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.317+0000 7f52d3fff640 1 -- 172.21.15.46:0/1560015936 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc0327f0 con 0x7f52d40d2cf0 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 -- 172.21.15.46:0/1560015936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d40d2cf0 msgr2=0x7f52d415f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 --2- 172.21.15.46:0/1560015936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d40d2cf0 0x7f52d415f990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0097a0 tx=0x7f52cc02a230 comp rx=0 tx=0).stop 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 -- 172.21.15.46:0/1560015936 shutdown_connections 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 --2- 172.21.15.46:0/1560015936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d40d2cf0 0x7f52d415f990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 -- 172.21.15.46:0/1560015936 >> 172.21.15.46:0/1560015936 conn(0x7f52d406e190 msgr2=0x7f52d406e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 -- 172.21.15.46:0/1560015936 shutdown_connections 2024-05-09T03:20:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.318+0000 7f52db689640 1 -- 172.21.15.46:0/1560015936 wait complete. 2024-05-09T03:20:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.320+0000 7f52db689640 1 Processor -- start 2024-05-09T03:20:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.320+0000 7f52db689640 1 -- start start 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52db689640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52d40b81b0 0x7f52d41ea190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52db689640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 0x7f52d41fa160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52db689640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d41615f0 0x7f52d41fa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52db689640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52d40d1890 con 0x7f52d40d2cf0 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52db689640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52d40b85c0 con 0x7f52d41615f0 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52db689640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f52d415fed0 con 0x7f52d40b81b0 2024-05-09T03:20:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52d8c24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 0x7f52d41fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52d8c24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 0x7f52d41fa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40442/0 (socket says 172.21.15.46:40442) 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52d8c24640 1 -- 172.21.15.46:0/3683653121 learned_addr learned my addr 172.21.15.46:0/3683653121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52d9c26640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d41615f0 0x7f52d41fa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.321+0000 7f52d9425640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52d40b81b0 0x7f52d41ea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d8c24640 1 -- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52d40b81b0 msgr2=0x7f52d41ea190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d8c24640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52d40b81b0 0x7f52d41ea190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d8c24640 1 -- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d41615f0 msgr2=0x7f52d41fa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d8c24640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d41615f0 0x7f52d41fa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d8c24640 1 -- 172.21.15.46:0/3683653121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d41eae50 con 0x7f52d40d2cf0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d9c26640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d41615f0 0x7f52d41fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d9425640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52d40b81b0 0x7f52d41ea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d8c24640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 0x7f52d41fa160 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c3c0 tx=0x7f52c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8017070 con 0x7f52d40d2cf0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c8005ce0 con 0x7f52d40d2cf0 2024-05-09T03:20:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d41eb130 con 0x7f52d40d2cf0 2024-05-09T03:20:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8007310 con 0x7f52d40d2cf0 2024-05-09T03:20:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.322+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d414c1d0 con 0x7f52d40d2cf0 2024-05-09T03:20:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.323+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d41eabc0 con 0x7f52d40d2cf0 2024-05-09T03:20:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.324+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f52c801a070 con 0x7f52d40d2cf0 2024-05-09T03:20:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.325+0000 7f52d1ffb640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52c0076300 0x7f52c0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.325+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7f52c8097390 con 0x7f52d40d2cf0 2024-05-09T03:20:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.325+0000 7f52d9425640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52c0076300 0x7f52c0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.325+0000 7f52d9425640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52c0076300 0x7f52c0078810 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f52c40012e0 tx=0x7f52c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.329+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f52c8060220 con 0x7f52d40d2cf0 2024-05-09T03:20:54.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.523+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52d406a270 con 0x7f52d40d2cf0 2024-05-09T03:20:54.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.524+0000 7f52d1ffb640 1 -- 172.21.15.46:0/3683653121 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+18000 (secure 0 0 0) 0x7f52c8068210 con 0x7f52d40d2cf0 2024-05-09T03:20:54.525 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:54.525 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":126,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:20:53.177055+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6834/1514520479":"2024-05-10T03:20:43.024261+0000","172.21.15.46:6834/2034185175":"2024-05-10T03:20:36.108968+0000","172.21.15.46:6832/2034185175":"2024-05-10T03:20:36.108968+0000","172.21.15.46:6834/2183116503":"2024-05-10T03:20:24.715744+0000","172.21.15.46:6832/2183116503":"2024-05-10T03:20:24.715744+0000","172.21.15.46:6834/134300288":"2024-05-10T03:20:09.183188+0000","172.21.15.46:6832/134300288":"2024-05-10T03:20:09.183188+0000","172.21.15.46:6834/1774919282":"2024-05-10T03:20:03.819873+0000","172.21.15.46:6832/2209678569":"2024-05-10T03:19:52.097765+0000","172.21.15.46:6834/3872337434":"2024-05-10T03:19:35.738958+0000","172.21.15.46:6834/1350199591":"2024-05-10T03:19:30.622459+0000","172.21.15.46:6832/574897435":"2024-05-10T03:20:30.067507+0000","172.21.15.46:6832/3872337434":"2024-05-10T03:19:35.738958+0000","172.21.15.46:6832/925919281":"2024-05-10T03:19:16.273718+0000","172.21.15.46:6834/120572030":"2024-05-10T03:19:12.242288+0000","172.21.15.46:6832/1774919282":"2024-05-10T03:20:03.819873+0000","172.21.15.46:6832/3450505404":"2024-05-10T03:17:52.032707+0000","172.21.15.46:6832/1270051447":"2024-05-10T03:17:29.198075+0000","172.21.15.46:6832/2388442105":"2024-05-10T03:17:44.993790+0000","172.21.15.46:6834/3968811793":"2024-05-10T03:17:16.129644+0000","172.21.15.46:6834/925919281":"2024-05-10T03:19:16.273718+0000","172.21.15.46:6834/1270051447":"2024-05-10T03:17:29.198075+0000","172.21.15.46:6834/2860475391":"2024-05-10T03:16:44.302289+0000","172.21.15.46:6832/3044304069":"2024-05-10T03:17:09.735805+0000","172.21.15.46:6832/1514520479":"2024-05-10T03:20:43.024261+0000","172.21.15.46:6834/2566829830":"2024-05-10T03:16:58.040696+0000","172.21.15.46:6832/3968811793":"2024-05-10T03:17:16.129644+0000","172.21.15.46:6834/2032086476":"2024-05-10T03:17:22.988689+0000","172.21.15.46:6832/1350199591":"2024-05-10T03:19:30.622459+0000","172.21.15.46:6834/2546691383":"2024-05-10T03:18:27.254542+0000","172.21.15.46:6834/2347304023":"2024-05-10T03:16:08.753704+0000","172.21.15.46:6834/2388442105":"2024-05-10T03:17:44.993790+0000","172.21.15.46:6834/1234333269":"2024-05-10T03:17:58.071311+0000","172.21.15.46:6832/2827001018":"2024-05-10T03:19:01.862132+0000","172.21.15.46:6834/2585438921":"2024-05-10T03:18:23.220415+0000","172.21.15.46:6832/2789195040":"2024-05-10T03:18:36.012259+0000","172.21.15.46:6832/1281653036":"2024-05-10T03:16:24.681981+0000","172.21.15.46:6834/2209678569":"2024-05-10T03:19:52.097765+0000","172.21.15.46:6832/1486786164":"2024-05-10T03:17:42.979548+0000","172.21.15.46:6832/2032086476":"2024-05-10T03:17:22.988689+0000","172.21.15.46:6834/3044304069":"2024-05-10T03:17:09.735805+0000","172.21.15.46:6834/2490064583":"2024-05-10T03:18:44.067190+0000","172.21.15.46:6832/782052086":"2024-05-10T03:16:40.263688+0000","172.21.15.46:6832/2566829830":"2024-05-10T03:16:58.040696+0000","172.21.15.46:6834/1281653036":"2024-05-10T03:16:24.681981+0000","172.21.15.46:6834/3297564121":"2024-05-10T03:16:36.237612+0000","172.21.15.46:6832/1234333269":"2024-05-10T03:17:58.071311+0000","172.21.15.46:6832/1033379302":"2024-05-10T03:19:04.179713+0000","172.21.15.46:6832/271467610":"2024-05-10T03:18:02.111947+0000","172.21.15.46:6834/782052086":"2024-05-10T03:16:40.263688+0000","172.21.15.46:6832/3297564121":"2024-05-10T03:16:36.237612+0000","172.21.15.46:6834/271467610":"2024-05-10T03:18:02.111947+0000","172.21.15.46:6832/436596417":"2024-05-10T03:18:06.147085+0000","172.21.15.46:6834/574897435":"2024-05-10T03:20:30.067507+0000","172.21.15.46:6832/905138228":"2024-05-10T03:19:08.214271+0000","172.21.15.46:6834/436596417":"2024-05-10T03:18:06.147085+0000","172.21.15.46:6834/1486786164":"2024-05-10T03:17:42.979548+0000","172.21.15.46:6834/3106969477":"2024-05-10T03:18:20.866657+0000","172.21.15.46:6832/2585438921":"2024-05-10T03:18:23.220415+0000","172.21.15.46:6834/3423945789":"2024-05-10T03:17:12.099885+0000","172.21.15.46:6832/3423945789":"2024-05-10T03:17:12.099885+0000","172.21.15.46:6832/1887013209":"2024-05-10T03:16:30.194822+0000","172.21.15.46:6832/2546691383":"2024-05-10T03:18:27.254542+0000","172.21.15.46:6832/202666403":"2024-05-10T03:18:31.290277+0000","172.21.15.46:6832/935847027":"2024-05-10T03:18:40.005228+0000","172.21.15.46:6834/3450505404":"2024-05-10T03:17:52.032707+0000","172.21.15.46:6832/2347304023":"2024-05-10T03:16:08.753704+0000","172.21.15.46:6834/202666403":"2024-05-10T03:18:31.290277+0000","172.21.15.46:6834/2789195040":"2024-05-10T03:18:36.012259+0000","172.21.15.46:6834/935847027":"2024-05-10T03:18:40.005228+0000","172.21.15.46:6834/905138228":"2024-05-10T03:19:08.214271+0000","172.21.15.46:6832/2490064583":"2024-05-10T03:18:44.067190+0000","172.21.15.46:6834/2827001018":"2024-05-10T03:19:01.862132+0000","172.21.15.46:6832/2638610471":"2024-05-10T03:18:48.102832+0000","172.21.15.46:6832/2860475391":"2024-05-10T03:16:44.302289+0000","172.21.15.46:6832/3106969477":"2024-05-10T03:18:20.866657+0000","172.21.15.46:6834/2638610471":"2024-05-10T03:18:48.102832+0000","172.21.15.46:6834/1033379302":"2024-05-10T03:19:04.179713+0000","172.21.15.46:6834/1887013209":"2024-05-10T03:16:30.194822+0000","172.21.15.46:6832/120572030":"2024-05-10T03:19:12.242288+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:20:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52c0076300 msgr2=0x7f52c0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52c0076300 0x7f52c0078810 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f52c40012e0 tx=0x7f52c4009040 comp rx=0 tx=0).stop 2024-05-09T03:20:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 msgr2=0x7f52d41fa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 0x7f52d41fa160 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f52c800c3c0 tx=0x7f52c800c880 comp rx=0 tx=0).stop 2024-05-09T03:20:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 shutdown_connections 2024-05-09T03:20:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d41615f0 0x7f52d41fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f52c0076300 0x7f52c0078810 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52d40d2cf0 0x7f52d41fa160 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 --2- 172.21.15.46:0/3683653121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f52d40b81b0 0x7f52d41ea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.526+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 >> 172.21.15.46:0/3683653121 conn(0x7f52d406e190 msgr2=0x7f52d4163e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.527+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 shutdown_connections 2024-05-09T03:20:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.527+0000 7f52db689640 1 -- 172.21.15.46:0/3683653121 wait complete. 2024-05-09T03:20:54.539 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T03:09:22.506928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T03:20:54.539 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T03:20:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.656+0000 7f59ca4cd640 1 Processor -- start 2024-05-09T03:20:54.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.656+0000 7f59ca4cd640 1 -- start start 2024-05-09T03:20:54.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59ca4cd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40d3fc0 0x7f59c416b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59ca4cd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c40b5550 con 0x7f59c40b5140 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59ca4cd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59c40d44e0 con 0x7f59c40b7e70 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59ca4cd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f59c40d3480 con 0x7f59c40d3fc0 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59c3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f59c40b5140 0x7f59c40b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48538/0 (socket says 172.21.15.46:48538) 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59c3fff640 1 -- 172.21.15.46:0/1010799665 learned_addr learned my addr 172.21.15.46:0/1010799665 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59c8a6a640 1 --2- 172.21.15.46:0/1010799665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40d3fc0 0x7f59c416b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.657+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832141050 0 0) 0x7f59c40b5550 con 0x7f59c40b5140 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ac003520 con 0x7f59c40b5140 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431389407 0 0) 0x7f59c40d44e0 con 0x7f59c40b7e70 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59c40b5550 con 0x7f59c40b7e70 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c8a6a640 1 -- 172.21.15.46:0/1010799665 >> v1:172.21.15.88:6789/0 conn(0x7f59c40b7e70 legacy=0x7f59c40d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c8a6a640 1 -- 172.21.15.46:0/1010799665 >> v1:172.21.15.46:6789/0 conn(0x7f59c40b5140 legacy=0x7f59c40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c8a6a640 1 -- 172.21.15.46:0/1010799665 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c40d47c0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.658+0000 7f59c8a6a640 1 --2- 172.21.15.46:0/1010799665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40d3fc0 0x7f59c416b0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009dc0 tx=0x7f59bc02a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6451127c7c1d9d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc007740 con 0x7f59c40d3fc0 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59bc0078e0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59c2ffd640 1 -- 172.21.15.46:0/1010799665 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc007cb0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59ca4cd640 1 -- 172.21.15.46:0/1010799665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40d3fc0 msgr2=0x7f59c416b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/1010799665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40d3fc0 0x7f59c416b0d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f59bc009dc0 tx=0x7f59bc02a760 comp rx=0 tx=0).stop 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59ca4cd640 1 -- 172.21.15.46:0/1010799665 shutdown_connections 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/1010799665 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40d3fc0 0x7f59c416b0d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59ca4cd640 1 -- 172.21.15.46:0/1010799665 >> 172.21.15.46:0/1010799665 conn(0x7f59c408d3f0 msgr2=0x7f59c408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:54.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.659+0000 7f59ca4cd640 1 -- 172.21.15.46:0/1010799665 shutdown_connections 2024-05-09T03:20:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.660+0000 7f59ca4cd640 1 -- 172.21.15.46:0/1010799665 wait complete. 2024-05-09T03:20:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.661+0000 7f59ca4cd640 1 Processor -- start 2024-05-09T03:20:54.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.661+0000 7f59ca4cd640 1 -- start start 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59ca4cd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40b5140 0x7f59c41f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59ca4cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 0x7f59c415b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59ca4cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c4149ea0 0x7f59c4158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59ca4cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59c40d5580 con 0x7f59c40d3fc0 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59ca4cd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f59c40d4d80 con 0x7f59c4149ea0 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59ca4cd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f59c40d5700 con 0x7f59c40b5140 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 0x7f59c415b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 0x7f59c415b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40456/0 (socket says 172.21.15.46:40456) 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 -- 172.21.15.46:0/653459910 learned_addr learned my addr 172.21.15.46:0/653459910 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:54.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c3fff640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40b5140 0x7f59c41f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c8a6a640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c4149ea0 0x7f59c4158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 -- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40b5140 msgr2=0x7f59c41f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40b5140 0x7f59c41f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 -- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c4149ea0 msgr2=0x7f59c4158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c4149ea0 0x7f59c4158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 -- 172.21.15.46:0/653459910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c4156fc0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.663+0000 7f59c37fe640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 0x7f59c415b700 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f59b800c460 tx=0x7f59b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.664+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b80052d0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.664+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b8005490 con 0x7f59c40d3fc0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.664+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c41572a0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.664+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b8012870 con 0x7f59c40d3fc0 2024-05-09T03:20:54.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.664+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c4157860 con 0x7f59c40d3fc0 2024-05-09T03:20:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.665+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c406a270 con 0x7f59c40d3fc0 2024-05-09T03:20:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.666+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f59b8012a10 con 0x7f59c40d3fc0 2024-05-09T03:20:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.666+0000 7f59c0ff9640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f59b00762d0 0x7f59b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.666+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7f59b8097aa0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.666+0000 7f59c3fff640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f59b00762d0 0x7f59b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.667+0000 7f59c3fff640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f59b00762d0 0x7f59b00787e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f59b4007f50 tx=0x7f59b4007ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.671+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f59b8060340 con 0x7f59c40d3fc0 2024-05-09T03:20:54.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.862+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f59c415c9f0 con 0x7f59c40d3fc0 2024-05-09T03:20:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.863+0000 7f59c0ff9640 1 -- 172.21.15.46:0/653459910 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) ==== 93+0+10 (secure 0 0 0) 0x7f59b8064640 con 0x7f59c40d3fc0 2024-05-09T03:20:54.864 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f59b00762d0 msgr2=0x7f59b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f59b00762d0 0x7f59b00787e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f59b4007f50 tx=0x7f59b4007ee0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 msgr2=0x7f59c415b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 0x7f59c415b700 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f59b800c460 tx=0x7f59b800c920 comp rx=0 tx=0).stop 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 shutdown_connections 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c4149ea0 0x7f59c4158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f59b00762d0 0x7f59b00787e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c40d3fc0 0x7f59c415b700 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 --2- 172.21.15.46:0/653459910 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f59c40b5140 0x7f59c41f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 >> 172.21.15.46:0/653459910 conn(0x7f59c408d3f0 msgr2=0x7f59c4163f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:54.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 shutdown_connections 2024-05-09T03:20:54.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.865+0000 7f59ca4cd640 1 -- 172.21.15.46:0/653459910 wait complete. 2024-05-09T03:20:54.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors' 2024-05-09T03:20:54.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.991+0000 7f18bf186640 1 Processor -- start 2024-05-09T03:20:54.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.991+0000 7f18bf186640 1 -- start start 2024-05-09T03:20:54.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18bf186640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18b80d1c80 0x7f18b80d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:54.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18bf186640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b80cfb70 con 0x7f18b80b6180 2024-05-09T03:20:54.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18bf186640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b80cf2d0 con 0x7f18b80d25d0 2024-05-09T03:20:54.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18bf186640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f18b80cf5b0 con 0x7f18b80d1c80 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18b7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18b80d1c80 0x7f18b80d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18b7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18b80d1c80 0x7f18b80d2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40460/0 (socket says 172.21.15.46:40460) 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18b7fff640 1 -- 172.21.15.46:0/1107077740 learned_addr learned my addr 172.21.15.46:0/1107077740 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18b7fff640 1 -- 172.21.15.46:0/1107077740 >> v1:172.21.15.88:6789/0 conn(0x7f18b80d25d0 legacy=0x7f18b80b6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.992+0000 7f18b7fff640 1 -- 172.21.15.46:0/1107077740 >> v1:172.21.15.46:6789/0 conn(0x7f18b80b6180 legacy=0x7f18b80b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.993+0000 7f18b77fe640 1 -- 172.21.15.46:0/1107077740 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154143123 0 0) 0x7f18b80cfb70 con 0x7f18b80b6180 2024-05-09T03:20:54.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.993+0000 7f18b7fff640 1 -- 172.21.15.46:0/1107077740 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b80cf890 con 0x7f18b80d1c80 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.993+0000 7f18b7fff640 1 --2- 172.21.15.46:0/1107077740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80d1c80 0x7f18b80d2090 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009870 tx=0x7f18ac02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c647e7f61d2d29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.993+0000 7f18b77fe640 1 -- 172.21.15.46:0/1107077740 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac037070 con 0x7f18b80d1c80 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18b77fe640 1 -- 172.21.15.46:0/1107077740 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18ac02ae50 con 0x7f18b80d1c80 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18b77fe640 1 -- 172.21.15.46:0/1107077740 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac032860 con 0x7f18b80d1c80 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 -- 172.21.15.46:0/1107077740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80d1c80 msgr2=0x7f18b80d2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 --2- 172.21.15.46:0/1107077740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80d1c80 0x7f18b80d2090 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009870 tx=0x7f18ac02a300 comp rx=0 tx=0).stop 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 -- 172.21.15.46:0/1107077740 shutdown_connections 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 --2- 172.21.15.46:0/1107077740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80d1c80 0x7f18b80d2090 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 -- 172.21.15.46:0/1107077740 >> 172.21.15.46:0/1107077740 conn(0x7f18b80b5660 msgr2=0x7f18b80b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:54.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 -- 172.21.15.46:0/1107077740 shutdown_connections 2024-05-09T03:20:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.994+0000 7f18bf186640 1 -- 172.21.15.46:0/1107077740 wait complete. 2024-05-09T03:20:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.996+0000 7f18bf186640 1 Processor -- start 2024-05-09T03:20:54.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.996+0000 7f18bf186640 1 -- start start 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bf186640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 0x7f18b8160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bf186640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18b80d1c80 0x7f18b8156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bf186640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18b80d25d0 0x7f18b8157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bf186640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f18b80d2b60 con 0x7f18b80b6180 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bf186640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18b80d29e0 con 0x7f18b80d25d0 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bf186640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f18b80b7690 con 0x7f18b80d1c80 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 0x7f18b8160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 0x7f18b8160cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40472/0 (socket says 172.21.15.46:40472) 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 -- 172.21.15.46:0/601012251 learned_addr learned my addr 172.21.15.46:0/601012251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18b7fff640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18b80d1c80 0x7f18b8156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:55.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bd723640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18b80d25d0 0x7f18b8157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 -- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18b80d1c80 msgr2=0x7f18b8156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18b80d1c80 0x7f18b8156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 -- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18b80d25d0 msgr2=0x7f18b8157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18b80d25d0 0x7f18b8157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 -- 172.21.15.46:0/601012251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b815fab0 con 0x7f18b80b6180 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18bcf22640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 0x7f18b8160cf0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f18a800c340 tx=0x7f18a800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.998+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8017070 con 0x7f18b80b6180 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.999+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a80035f0 con 0x7f18b80b6180 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.999+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a80039e0 con 0x7f18b80b6180 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.999+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b815fd90 con 0x7f18b80b6180 2024-05-09T03:20:55.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:54.999+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b8157b50 con 0x7f18b80b6180 2024-05-09T03:20:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.000+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b80b6760 con 0x7f18b80b6180 2024-05-09T03:20:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.000+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18a801a070 con 0x7f18b80b6180 2024-05-09T03:20:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.001+0000 7f18b57fa640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a40762d0 0x7f18a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.001+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7f18a8097290 con 0x7f18b80b6180 2024-05-09T03:20:55.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.002+0000 7f18b7fff640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a40762d0 0x7f18a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:55.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.002+0000 7f18b7fff640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a40762d0 0x7f18a40787e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f18ac034000 tx=0x7f18ac02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:55.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.006+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f18b80b6760 con 0x7f18b80b6180 2024-05-09T03:20:55.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:55.271+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]} v 0) -- 0x7f18b806a270 con 0x7f18b80b6180 2024-05-09T03:20:56.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.190+0000 7f18b57fa640 1 -- 172.21.15.46:0/601012251 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]}]=0 v581) ==== 138+0+0 (secure 0 0 0) 0x7f18a8068030 con 0x7f18b80b6180 2024-05-09T03:20:56.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a40762d0 msgr2=0x7f18a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a40762d0 0x7f18a40787e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f18ac034000 tx=0x7f18ac02e000 comp rx=0 tx=0).stop 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 msgr2=0x7f18b8160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 0x7f18b8160cf0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f18a800c340 tx=0x7f18a800c800 comp rx=0 tx=0).stop 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 shutdown_connections 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18b80d25d0 0x7f18b8157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18b80d1c80 0x7f18b8156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a40762d0 0x7f18a40787e0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 --2- 172.21.15.46:0/601012251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18b80b6180 0x7f18b8160cf0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.193+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 >> 172.21.15.46:0/601012251 conn(0x7f18b80b5660 msgr2=0x7f18b80723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.194+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 shutdown_connections 2024-05-09T03:20:56.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.194+0000 7f18bf186640 1 -- 172.21.15.46:0/601012251 wait complete. 2024-05-09T03:20:56.208 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-05-09T03:20:56.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.332+0000 7fdd07c53640 1 Processor -- start 2024-05-09T03:20:56.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.332+0000 7fdd07c53640 1 -- start start 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd07c53640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd000bb770 0x7fdd000d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd07c53640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd00163eb0 0x7fdd001642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd07c53640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd00164900 con 0x7fdd000d2a70 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd07c53640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdd000d4bb0 con 0x7fdd000bb770 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd07c53640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdd00164be0 con 0x7fdd00163eb0 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd059ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd000bb770 0x7fdd000d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd059ef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd000bb770 0x7fdd000d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33402/0 (socket says 172.21.15.46:33402) 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.333+0000 7fdd059ef640 1 -- 172.21.15.46:0/2517665296 learned_addr learned my addr 172.21.15.46:0/2517665296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:56.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.334+0000 7fdd051ee640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd00163eb0 0x7fdd001642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.334+0000 7fdd059ef640 1 -- 172.21.15.46:0/2517665296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd00163eb0 msgr2=0x7fdd001642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.334+0000 7fdd059ef640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd00163eb0 0x7fdd001642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.334+0000 7fdd059ef640 1 -- 172.21.15.46:0/2517665296 >> v1:172.21.15.88:6789/0 conn(0x7fdd000d2a70 legacy=0x7fdd000d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.334+0000 7fdd059ef640 1 -- 172.21.15.46:0/2517665296 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd000d4840 con 0x7fdd000bb770 2024-05-09T03:20:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.334+0000 7fdd051ee640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd00163eb0 0x7fdd001642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:56.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.335+0000 7fdd059ef640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000bb770 0x7fdd000d5040 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009e50 tx=0x7fdcf002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfdce55d7f4fb68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.335+0000 7fdd049ed640 1 -- 172.21.15.46:0/2517665296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0036070 con 0x7fdd000bb770 2024-05-09T03:20:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.335+0000 7fdd049ed640 1 -- 172.21.15.46:0/2517665296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf0031600 con 0x7fdd000bb770 2024-05-09T03:20:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.335+0000 7fdd049ed640 1 -- 172.21.15.46:0/2517665296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf00319d0 con 0x7fdd000bb770 2024-05-09T03:20:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 -- 172.21.15.46:0/2517665296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000bb770 msgr2=0x7fdd000d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000bb770 0x7fdd000d5040 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0009e50 tx=0x7fdcf002a710 comp rx=0 tx=0).stop 2024-05-09T03:20:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 -- 172.21.15.46:0/2517665296 shutdown_connections 2024-05-09T03:20:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd00163eb0 0x7fdd001642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 --2- 172.21.15.46:0/2517665296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000bb770 0x7fdd000d5040 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 -- 172.21.15.46:0/2517665296 >> 172.21.15.46:0/2517665296 conn(0x7fdd0006e370 msgr2=0x7fdd0006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 -- 172.21.15.46:0/2517665296 shutdown_connections 2024-05-09T03:20:56.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.336+0000 7fdd07c53640 1 -- 172.21.15.46:0/2517665296 wait complete. 2024-05-09T03:20:56.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.337+0000 7fdd07c53640 1 Processor -- start 2024-05-09T03:20:56.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.338+0000 7fdd07c53640 1 -- start start 2024-05-09T03:20:56.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd07c53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd000bb770 0x7fdd0015bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd07c53640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd000d2a70 0x7fdd000ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd07c53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 0x7fdd000ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd07c53640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000d0bc0 con 0x7fdd00163eb0 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd07c53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdd000bbb80 con 0x7fdd000bb770 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd07c53640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdd000d0d40 con 0x7fdd000d2a70 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 0x7fdd000ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 0x7fdd000ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33410/0 (socket says 172.21.15.46:33410) 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 -- 172.21.15.46:0/1780809125 learned_addr learned my addr 172.21.15.46:0/1780809125 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd059ef640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd000bb770 0x7fdd0015bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd051ee640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd000d2a70 0x7fdd000ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 -- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd000d2a70 msgr2=0x7fdd000ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd000d2a70 0x7fdd000ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 -- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd000bb770 msgr2=0x7fdd0015bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd000bb770 0x7fdd0015bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd061f0640 1 -- 172.21.15.46:0/1780809125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0015e0f0 con 0x7fdd00163eb0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.339+0000 7fdd051ee640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd000d2a70 0x7fdd000ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.340+0000 7fdd061f0640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 0x7fdd000ca760 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c370 tx=0x7fdcf800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.340+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8017070 con 0x7fdd00163eb0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.340+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf80052f0 con 0x7fdd00163eb0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.340+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8003850 con 0x7fdd00163eb0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.340+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd001f9e30 con 0x7fdd00163eb0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.340+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd001fa3f0 con 0x7fdd00163eb0 2024-05-09T03:20:56.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.341+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd0015e2c0 con 0x7fdd00163eb0 2024-05-09T03:20:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.342+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdcf801a070 con 0x7fdd00163eb0 2024-05-09T03:20:56.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.342+0000 7fdcfe7fc640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdcec0762d0 0x7fdcec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.342+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7fdcf80973c0 con 0x7fdd00163eb0 2024-05-09T03:20:56.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.342+0000 7fdd059ef640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdcec0762d0 0x7fdcec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.343+0000 7fdd059ef640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdcec0762d0 0x7fdcec0787e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0033000 tx=0x7fdcf0009000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:56.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.347+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdcf805fbd0 con 0x7fdd00163eb0 2024-05-09T03:20:56.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.539+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fdd0006a270 con 0x7fdd00163eb0 2024-05-09T03:20:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.543+0000 7fdcfe7fc640 1 -- 172.21.15.46:0/1780809125 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7fdcf80682a0 con 0x7fdd00163eb0 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdcec0762d0 msgr2=0x7fdcec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdcec0762d0 0x7fdcec0787e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0033000 tx=0x7fdcf0009000 comp rx=0 tx=0).stop 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 msgr2=0x7fdd000ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 0x7fdd000ca760 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c370 tx=0x7fdcf800c830 comp rx=0 tx=0).stop 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 shutdown_connections 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdcec0762d0 0x7fdcec0787e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00163eb0 0x7fdd000ca760 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdd000d2a70 0x7fdd000ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 --2- 172.21.15.46:0/1780809125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd000bb770 0x7fdd0015bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 >> 172.21.15.46:0/1780809125 conn(0x7fdd0006e370 msgr2=0x7fdd000b5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:56.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 shutdown_connections 2024-05-09T03:20:56.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.545+0000 7fdd07c53640 1 -- 172.21.15.46:0/1780809125 wait complete. 2024-05-09T03:20:56.557 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:20:56.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:20:56.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:20:56.655 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:20:56.656 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:56.656 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:56.672 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:56.673 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:20:56.673 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:56.691 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:20:56.691 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:20:56.691 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:56.729 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:20:56.729 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:20:56.729 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:20:56.748 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:20:56.748 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-05-09T03:20:56.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:20:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.882+0000 7f66376c8640 1 Processor -- start 2024-05-09T03:20:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.883+0000 7f66376c8640 1 -- start start 2024-05-09T03:20:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f66376c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 0x7f66300b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f66376c8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66300d2b80 con 0x7f66300b92e0 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f66376c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66300d2e60 con 0x7f66300d0b90 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f66376c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f66300d3140 con 0x7f66300cf6b0 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f6635464640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66300b92e0 0x7f66300d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60630/0 (socket says 172.21.15.46:60630) 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f6635464640 1 -- 172.21.15.46:0/561896618 learned_addr learned my addr 172.21.15.46:0/561896618 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503530920 0 0) 0x7f66300d2b80 con 0x7f66300b92e0 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f6635c65640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 0x7f66300b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.884+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6618003520 con 0x7f66300b92e0 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1906532438 0 0) 0x7f6618003520 con 0x7f66300b92e0 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 msgr2=0x7f66300b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 0x7f66300b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 >> v1:172.21.15.88:6789/0 conn(0x7f66300d0b90 legacy=0x7f663006d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66300d3420 con 0x7f66300b92e0 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f6620001a70 con 0x7f66300b92e0 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 >> v1:172.21.15.46:6789/0 conn(0x7f66300b92e0 legacy=0x7f66300d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6618003a30 0x7f6618003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 0x7f6618004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66180051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66300d3140 con 0x7f6618004d40 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f661800bc20 con 0x7f6618003a30 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f661800be20 con 0x7f66300cf6b0 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635c65640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66180051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635464640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6618003a30 0x7f6618003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6634c63640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 0x7f6618004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635c65640 1 -- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 msgr2=0x7f6618004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635c65640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300cf6b0 0x7f6618004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635c65640 1 -- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6618003a30 msgr2=0x7f6618003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635c65640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6618003a30 0x7f6618003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.885+0000 7f6635c65640 1 -- 172.21.15.46:0/561896618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6618003520 con 0x7f6618004d40 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.886+0000 7f6635c65640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66180051f0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f66300d3740 tx=0x7f6618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1f40690affbc64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.886+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628036070 con 0x7f6618004d40 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.886+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6628031400 con 0x7f6618004d40 2024-05-09T03:20:56.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.886+0000 7f662ffff640 1 -- 172.21.15.46:0/561896618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628009690 con 0x7f6618004d40 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f66376c8640 1 -- 172.21.15.46:0/561896618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 msgr2=0x7f66180051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f66376c8640 1 --2- 172.21.15.46:0/561896618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66180051f0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f66300d3740 tx=0x7f6618005830 comp rx=0 tx=0).stop 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f6635c65640 1 -- 172.21.15.46:0/561896618 reap_dead start 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f66376c8640 1 -- 172.21.15.46:0/561896618 shutdown_connections 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f66376c8640 1 -- 172.21.15.46:0/561896618 >> 172.21.15.46:0/561896618 conn(0x7f66300d3c00 msgr2=0x7f66300d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f66376c8640 1 -- 172.21.15.46:0/561896618 shutdown_connections 2024-05-09T03:20:56.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.887+0000 7f66376c8640 1 -- 172.21.15.46:0/561896618 wait complete. 2024-05-09T03:20:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.889+0000 7f66376c8640 1 Processor -- start 2024-05-09T03:20:56.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.889+0000 7f66376c8640 1 -- start start 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f66376c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66300b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f66376c8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300b8530 0x7f66300b6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f66376c8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66300b6c40 0x7f66300b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f66376c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66300b5530 con 0x7f6618004d40 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f66376c8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f66300b53b0 con 0x7f66300b6c40 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f66376c8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6630169030 con 0x7f66300b8530 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f6635464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66300b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f6635464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66300b7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33426/0 (socket says 172.21.15.46:33426) 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f6635464640 1 -- 172.21.15.46:0/3377946399 learned_addr learned my addr 172.21.15.46:0/3377946399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f6635c65640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66300b6c40 0x7f66300b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.890+0000 7f6634c63640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300b8530 0x7f66300b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635464640 1 -- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300b8530 msgr2=0x7f66300b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635464640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300b8530 0x7f66300b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635464640 1 -- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66300b6c40 msgr2=0x7f66300b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635464640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66300b6c40 0x7f66300b7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635464640 1 -- 172.21.15.46:0/3377946399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6630148f10 con 0x7f6618004d40 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635c65640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66300b6c40 0x7f66300b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6634c63640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300b8530 0x7f66300b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f6635464640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66300b7ad0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f66200079f0 tx=0x7f662000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620017020 con 0x7f6618004d40 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66200133f0 con 0x7f6618004d40 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620001ab0 con 0x7f6618004d40 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66301491f0 con 0x7f6618004d40 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.891+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66301497b0 con 0x7f6618004d40 2024-05-09T03:20:56.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.892+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f663006ecd0 con 0x7f6618004d40 2024-05-09T03:20:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.893+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f662001a020 con 0x7f6618004d40 2024-05-09T03:20:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.893+0000 7f662dffb640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f661c0762d0 0x7f661c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.893+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7f6620098c40 con 0x7f6618004d40 2024-05-09T03:20:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.894+0000 7f6634c63640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f661c0762d0 0x7f661c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:56.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.894+0000 7f6634c63640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f661c0762d0 0x7f661c0787e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f66300b41e0 tx=0x7f6624009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:56.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:56.898+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f663006ecd0 con 0x7f6618004d40 2024-05-09T03:20:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.128+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f663006a270 con 0x7f6618004d40 2024-05-09T03:20:57.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.129+0000 7f662dffb640 1 -- 172.21.15.46:0/3377946399 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 347 v347) ==== 95+0+3869 (secure 0 0 0) 0x7f6620021020 con 0x7f6618004d40 2024-05-09T03:20:57.130 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:57.130 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":347,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5909,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3312873121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3312873121},{"type":"v1","addr":"172.21.15.46:6835","nonce":3312873121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":5915,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/803765962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":803765962},{"type":"v1","addr":"172.21.15.88:6837","nonce":803765962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":6388,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2530248873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2530248873},{"type":"v1","addr":"172.21.15.88:6836","nonce":2530248873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":80},{"gid":10491,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2201711907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2201711907},{"type":"v1","addr":"172.21.15.46:6834","nonce":2201711907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":345}],"filesystems":[]} 2024-05-09T03:20:57.130 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 347 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f661c0762d0 msgr2=0x7f661c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f661c0762d0 0x7f661c0787e0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f66300b41e0 tx=0x7f6624009040 comp rx=0 tx=0).stop 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 msgr2=0x7f66300b7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66300b7ad0 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f66200079f0 tx=0x7f662000a710 comp rx=0 tx=0).stop 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 shutdown_connections 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66300b6c40 0x7f66300b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f66300b8530 0x7f66300b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f661c0762d0 0x7f661c0787e0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 --2- 172.21.15.46:0/3377946399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6618004d40 0x7f66300b7ad0 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 >> 172.21.15.46:0/3377946399 conn(0x7f66300d3c00 msgr2=0x7f66300b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 shutdown_connections 2024-05-09T03:20:57.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.131+0000 7f66376c8640 1 -- 172.21.15.46:0/3377946399 wait complete. 2024-05-09T03:20:57.143 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T03:20:57.144 INFO:tasks.ceph.mds.a:Stopping old one... 2024-05-09T03:20:57.144 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:20:57.144 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:20:57.144 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T03:20:57.144 INFO:tasks.ceph.mds.c:Stopping old one... 2024-05-09T03:20:57.144 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:20:57.144 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:20:57.145 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T03:20:57.145 INFO:tasks.ceph.mds.b:Stopping old one... 2024-05-09T03:20:57.145 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:20:57.145 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:20:57.145 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T03:20:57.145 INFO:tasks.ceph.mds.d:Stopping old one... 2024-05-09T03:20:57.145 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:20:57.145 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:20:57.155 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:20:57.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T03:20:57.157 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:20:57.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T03:20:57.158 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:20:57.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T03:20:57.159 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:20:57.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T03:20:57.200 INFO:tasks.ceph.mds.b:Started 2024-05-09T03:20:57.200 DEBUG:teuthology.parallel:result is None 2024-05-09T03:20:57.201 INFO:tasks.ceph.mds.d:Started 2024-05-09T03:20:57.201 DEBUG:teuthology.parallel:result is None 2024-05-09T03:20:57.201 INFO:tasks.ceph.mds.a:Started 2024-05-09T03:20:57.202 DEBUG:teuthology.parallel:result is None 2024-05-09T03:20:57.203 INFO:tasks.ceph.mds.c:Started 2024-05-09T03:20:57.203 DEBUG:teuthology.parallel:result is None 2024-05-09T03:20:57.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-05-09T03:20:57.293 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-05-09T03:20:57.293 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:20:57.294 INFO:tasks.ceph.mds.b.smithi088.stdout:starting mds.b at 2024-05-09T03:20:57.294 INFO:tasks.ceph.mds.d.smithi088.stdout:starting mds.d at 2024-05-09T03:20:57.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.324+0000 7f449dac6640 1 Processor -- start 2024-05-09T03:20:57.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.324+0000 7f449dac6640 1 -- start start 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.325+0000 7f449dac6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44980bb770 0x7f44980d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.325+0000 7f449dac6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4498163eb0 0x7f44981642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.325+0000 7f449dac6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4498164900 con 0x7f44980d2a70 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.325+0000 7f449dac6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f44980d4bb0 con 0x7f44980bb770 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.325+0000 7f449dac6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4498164be0 con 0x7f4498163eb0 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44980bb770 0x7f44980d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44980bb770 0x7f44980d5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33468/0 (socket says 172.21.15.46:33468) 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 -- 172.21.15.46:0/2778906146 learned_addr learned my addr 172.21.15.46:0/2778906146 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f448effd640 1 --2- 172.21.15.46:0/2778906146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4498163eb0 0x7f44981642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:57.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 -- 172.21.15.46:0/2778906146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4498163eb0 msgr2=0x7f44981642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 --2- 172.21.15.46:0/2778906146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4498163eb0 0x7f44981642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 -- 172.21.15.46:0/2778906146 >> v1:172.21.15.88:6789/0 conn(0x7f44980d2a70 legacy=0x7f44980d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:57.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.326+0000 7f44977fe640 1 -- 172.21.15.46:0/2778906146 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44980d4840 con 0x7f44980bb770 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.327+0000 7f44977fe640 1 --2- 172.21.15.46:0/2778906146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44980d5040 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f4488009870 tx=0x7f448802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed182dd3e88a4cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.327+0000 7f4496ffd640 1 -- 172.21.15.46:0/2778906146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488037070 con 0x7f44980bb770 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.327+0000 7f4496ffd640 1 -- 172.21.15.46:0/2778906146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4488032440 con 0x7f44980bb770 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.327+0000 7f4496ffd640 1 -- 172.21.15.46:0/2778906146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488032810 con 0x7f44980bb770 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 -- 172.21.15.46:0/2778906146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 msgr2=0x7f44980d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 --2- 172.21.15.46:0/2778906146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44980d5040 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f4488009870 tx=0x7f448802a270 comp rx=0 tx=0).stop 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 -- 172.21.15.46:0/2778906146 shutdown_connections 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 --2- 172.21.15.46:0/2778906146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4498163eb0 0x7f44981642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 --2- 172.21.15.46:0/2778906146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44980d5040 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 -- 172.21.15.46:0/2778906146 >> 172.21.15.46:0/2778906146 conn(0x7f449806e370 msgr2=0x7f449806e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 -- 172.21.15.46:0/2778906146 shutdown_connections 2024-05-09T03:20:57.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.328+0000 7f449dac6640 1 -- 172.21.15.46:0/2778906146 wait complete. 2024-05-09T03:20:57.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.330+0000 7f449dac6640 1 Processor -- start 2024-05-09T03:20:57.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.330+0000 7f449dac6640 1 -- start start 2024-05-09T03:20:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f449dac6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44981f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f449dac6640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44980d2a70 0x7f44980ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f449dac6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4498163eb0 0x7f44980cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f449dac6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f44980d0bc0 con 0x7f44980bb770 2024-05-09T03:20:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f449dac6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f44980bbb80 con 0x7f4498163eb0 2024-05-09T03:20:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f449dac6640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f44980d0d40 con 0x7f44980d2a70 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f44977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44981f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f44977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44981f5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33484/0 (socket says 172.21.15.46:33484) 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.331+0000 7f44977fe640 1 -- 172.21.15.46:0/3143933706 learned_addr learned my addr 172.21.15.46:0/3143933706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f448effd640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44980d2a70 0x7f44980ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f4497fff640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4498163eb0 0x7f44980cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f44977fe640 1 -- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44980d2a70 msgr2=0x7f44980ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f44977fe640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44980d2a70 0x7f44980ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f44977fe640 1 -- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4498163eb0 msgr2=0x7f44980cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f44977fe640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4498163eb0 0x7f44980cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f44977fe640 1 -- 172.21.15.46:0/3143933706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44980cf6f0 con 0x7f44980bb770 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f4497fff640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4498163eb0 0x7f44980cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f44977fe640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44981f5fb0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f4488000c00 tx=0x7f44880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:57.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488041020 con 0x7f44980bb770 2024-05-09T03:20:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44880306c0 con 0x7f44980bb770 2024-05-09T03:20:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488037040 con 0x7f44980bb770 2024-05-09T03:20:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44980cf9d0 con 0x7f44980bb770 2024-05-09T03:20:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.332+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f449814ffe0 con 0x7f44980bb770 2024-05-09T03:20:57.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.333+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f449806a270 con 0x7f44980bb770 2024-05-09T03:20:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.334+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4488044020 con 0x7f44980bb770 2024-05-09T03:20:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.335+0000 7f4494ff9640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f44780762d0 0x7f44780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.335+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8297+0+0 (secure 0 0 0) 0x7f44880b8930 con 0x7f44980bb770 2024-05-09T03:20:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.335+0000 7f448effd640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f44780762d0 0x7f44780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:57.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.335+0000 7f448effd640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f44780762d0 0x7f44780787e0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4480006fd0 tx=0x7f4480008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:57.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.339+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f448804b030 con 0x7f44980bb770 2024-05-09T03:20:57.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:57.542+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f44980d4130 con 0x7f44980bb770 2024-05-09T03:20:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.558+0000 7f4494ff9640 1 -- 172.21.15.46:0/3143933706 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v131) ==== 97+0+0 (secure 0 0 0) 0x7f44880817c0 con 0x7f44980bb770 2024-05-09T03:20:58.559 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f44780762d0 msgr2=0x7f44780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f44780762d0 0x7f44780787e0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4480006fd0 tx=0x7f4480008040 comp rx=0 tx=0).stop 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 msgr2=0x7f44981f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44981f5fb0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f4488000c00 tx=0x7f44880046c0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 shutdown_connections 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4498163eb0 0x7f44980cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f44980d2a70 0x7f44980ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f44780762d0 0x7f44780787e0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 --2- 172.21.15.46:0/3143933706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44980bb770 0x7f44981f5fb0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 >> 172.21.15.46:0/3143933706 conn(0x7f449806e370 msgr2=0x7f44980d2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 shutdown_connections 2024-05-09T03:20:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.561+0000 7f449dac6640 1 -- 172.21.15.46:0/3143933706 wait complete. 2024-05-09T03:20:58.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-05-09T03:20:58.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.697+0000 7fcac0042640 1 Processor -- start 2024-05-09T03:20:58.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.698+0000 7fcac0042640 1 -- start start 2024-05-09T03:20:58.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcac0042640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80d0b90 0x7fcab80b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:58.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcac0042640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcab80cf6b0 0x7fcab806d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcac0042640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcab80d2b80 con 0x7fcab80b92e0 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcac0042640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcab80d2e60 con 0x7fcab80cf6b0 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcac0042640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcab80d3140 con 0x7fcab80d0b90 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabe5df640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcab80cf6b0 0x7fcab806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabddde640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcab80b92e0 0x7fcab80d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60658/0 (socket says 172.21.15.46:60658) 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabddde640 1 -- 172.21.15.46:0/1959276118 learned_addr learned my addr 172.21.15.46:0/1959276118 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabe5df640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcab80cf6b0 0x7fcab806d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33490/0 (socket says 172.21.15.46:33490) 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabd5dd640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80d0b90 0x7fcab80b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabe5df640 1 -- 172.21.15.46:0/1959276118 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80d0b90 msgr2=0x7fcab80b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabe5df640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80d0b90 0x7fcab80b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabe5df640 1 -- 172.21.15.46:0/1959276118 >> v1:172.21.15.46:6789/0 conn(0x7fcab80b92e0 legacy=0x7fcab80d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.699+0000 7fcabcddc640 1 -- 172.21.15.46:0/1959276118 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317884070 0 0) 0x7fcab80d2b80 con 0x7fcab80b92e0 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.700+0000 7fcabe5df640 1 -- 172.21.15.46:0/1959276118 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab80d3420 con 0x7fcab80cf6b0 2024-05-09T03:20:58.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.700+0000 7fcabd5dd640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80d0b90 0x7fcab80b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.700+0000 7fcabe5df640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab806d790 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009d90 tx=0x7fcab002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8790e2bcd3ea5696 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.700+0000 7fcabcddc640 1 -- 172.21.15.46:0/1959276118 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0031600 con 0x7fcab80cf6b0 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcabcddc640 1 -- 172.21.15.46:0/1959276118 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab00317a0 con 0x7fcab80cf6b0 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcabcddc640 1 -- 172.21.15.46:0/1959276118 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0031b70 con 0x7fcab80cf6b0 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 -- 172.21.15.46:0/1959276118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 msgr2=0x7fcab806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab806d790 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fcab0009d90 tx=0x7fcab002a5d0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 -- 172.21.15.46:0/1959276118 shutdown_connections 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab806d790 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 --2- 172.21.15.46:0/1959276118 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80d0b90 0x7fcab80b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 -- 172.21.15.46:0/1959276118 >> 172.21.15.46:0/1959276118 conn(0x7fcab80d3c00 msgr2=0x7fcab80d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 -- 172.21.15.46:0/1959276118 shutdown_connections 2024-05-09T03:20:58.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.701+0000 7fcac0042640 1 -- 172.21.15.46:0/1959276118 wait complete. 2024-05-09T03:20:58.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.703+0000 7fcac0042640 1 Processor -- start 2024-05-09T03:20:58.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.703+0000 7fcac0042640 1 -- start start 2024-05-09T03:20:58.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.704+0000 7fcac0042640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80b92e0 0x7fcab8154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:58.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.704+0000 7fcac0042640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:58.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.704+0000 7fcac0042640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcab8156a10 0x7fcab815e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:58.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.704+0000 7fcac0042640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcab806e060 con 0x7fcab80cf6b0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.704+0000 7fcac0042640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcab806dee0 con 0x7fcab8156a10 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.704+0000 7fcac0042640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcab806e1e0 con 0x7fcab80b92e0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab815f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33504/0 (socket says 172.21.15.46:33504) 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 -- 172.21.15.46:0/1834747149 learned_addr learned my addr 172.21.15.46:0/1834747149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabddde640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80b92e0 0x7fcab8154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabe5df640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcab8156a10 0x7fcab815e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 -- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80b92e0 msgr2=0x7fcab8154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80b92e0 0x7fcab8154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 -- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcab8156a10 msgr2=0x7fcab815e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcab8156a10 0x7fcab815e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 -- 172.21.15.46:0/1834747149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab815fd20 con 0x7fcab80cf6b0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcabd5dd640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab815f7e0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800c7f0 tx=0x7fcaa800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:58.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8015070 con 0x7fcab80cf6b0 2024-05-09T03:20:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaa80077c0 con 0x7fcab80cf6b0 2024-05-09T03:20:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab815d280 con 0x7fcab80cf6b0 2024-05-09T03:20:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.705+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8007bd0 con 0x7fcab80cf6b0 2024-05-09T03:20:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.706+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab815d840 con 0x7fcab80cf6b0 2024-05-09T03:20:58.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.707+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab806a270 con 0x7fcab80cf6b0 2024-05-09T03:20:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.707+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcaa8018070 con 0x7fcab80cf6b0 2024-05-09T03:20:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.708+0000 7fcaae7fc640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca9c076300 0x7fca9c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.708+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4857+0+0 (secure 0 0 0) 0x7fcaa8097090 con 0x7fcab80cf6b0 2024-05-09T03:20:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.708+0000 7fcabddde640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca9c076300 0x7fca9c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:58.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.709+0000 7fcabddde640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca9c076300 0x7fca9c078810 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fcab4007fd0 tx=0x7fcab4007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:58.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.713+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcaa8060660 con 0x7fcab80cf6b0 2024-05-09T03:20:58.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.977+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fcab8072690 con 0x7fcab80cf6b0 2024-05-09T03:20:58.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.978+0000 7fcaae7fc640 1 -- 172.21.15.46:0/1834747149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7fcaa8064960 con 0x7fcab80cf6b0 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmHdqcCBAAvWOWo6wqK/XWBOGmuYuD0g==", 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.979 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgurEFRAAqgthaAFzIcGD848NAYKRCw==", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmnrSBEBAARBBF3c5ekurUeVk74R1ZAw==", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:20:58.980 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmgBZZHRAAGTDdR0zLYfGeef8geXyDBw==", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA4PjxmYQbpBBAA8zKNlNTv50O1AKzAqfQvzw==", 2024-05-09T03:20:58.981 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA7PjxmCkocBBAAlQnXq+56kfpa+GqETJx/SA==", 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.982 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA+PjxmS1KNAhAAe66EMDlCZxU4kmi3+LP1Qg==", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBBPjxmvOk/AhAAFqrYNhZimGPANuiffyn0XA==", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.983 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBHPjxmbU+tBRAAIZ5xYmF16/b5PyB4qBIf1g==", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBKPjxmyyOFBBAA+9SplG5bSRxfXWQiCHworA==", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBNPjxmisHZAxAA/ZQoxyw89j42xKi61t/6SQ==", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBQPjxmhqbEAhAAdewdYuO0z2TRGySYcy1ZUw==", 2024-05-09T03:20:58.985 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw==", 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.986 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmkBDOKRAAJHoi+Y6M0GE7AjVFM3vfDQ==", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmiugEJhAAKBdQBzt4nOdsSU96yxZ/Bg==", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:20:58.987 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmdlaLLRAA09ivNGv4dXwiqVHglIZZvg==", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-05-09T03:20:58.988 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA1PjxmuVPDJhAAHOgxPpTSEd3pibPqobcHCg==", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmGJyzBBAAIAnuaSFhboOgrjaUXyBGqg==", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQA2PjxmmzeMABAA8bxelv37xGESTqGo/TLwBQ==", 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-05-09T03:20:58.990 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca9c076300 msgr2=0x7fca9c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca9c076300 0x7fca9c078810 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fcab4007fd0 tx=0x7fcab4007de0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 msgr2=0x7fcab815f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab815f7e0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800c7f0 tx=0x7fcaa800ccb0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 shutdown_connections 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcab8156a10 0x7fcab815e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fca9c076300 0x7fca9c078810 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab80cf6b0 0x7fcab815f7e0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 --2- 172.21.15.46:0/1834747149 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcab80b92e0 0x7fcab8154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:58.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 >> 172.21.15.46:0/1834747149 conn(0x7fcab80d3c00 msgr2=0x7fcab81593a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:58.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 shutdown_connections 2024-05-09T03:20:58.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:58.980+0000 7fcac0042640 1 -- 172.21.15.46:0/1834747149 wait complete. 2024-05-09T03:20:58.993 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T03:20:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.108+0000 7ff3a702b640 1 Processor -- start 2024-05-09T03:20:59.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.108+0000 7ff3a702b640 1 -- start start 2024-05-09T03:20:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a702b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3a00b5210 0x7ff3a01613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a702b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3a00d4800 0x7ff3a00d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a702b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a702b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff3a00b5620 con 0x7ff3a00d4800 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a702b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff3a00b8110 con 0x7ff3a00b5210 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a702b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a00d5280 con 0x7ff3a00b77c0 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3a00d4800 0x7ff3a00d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3a00d4800 0x7ff3a00d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33520/0 (socket says 172.21.15.46:33520) 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 -- 172.21.15.46:0/238415065 learned_addr learned my addr 172.21.15.46:0/238415065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a4dc7640 1 --2- 172.21.15.46:0/238415065 >> v2:172.21.15.88:3300/0 conn(0x7ff3a00b5210 0x7ff3a01613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a55c8640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 -- 172.21.15.46:0/238415065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 msgr2=0x7ff3a00b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 -- 172.21.15.46:0/238415065 >> v2:172.21.15.88:3300/0 conn(0x7ff3a00b5210 msgr2=0x7ff3a01613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 --2- 172.21.15.46:0/238415065 >> v2:172.21.15.88:3300/0 conn(0x7ff3a00b5210 0x7ff3a01613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff39ffff640 1 -- 172.21.15.46:0/238415065 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a00d5560 con 0x7ff3a00d4800 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a4dc7640 1 --2- 172.21.15.46:0/238415065 >> v2:172.21.15.88:3300/0 conn(0x7ff3a00b5210 0x7ff3a01613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.109+0000 7ff3a55c8640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.110+0000 7ff39ffff640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a00d4c10 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7ff3a008ed70 tx=0x7ff38802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad75d8ecf214baa0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.110+0000 7ff39f7fe640 1 -- 172.21.15.46:0/238415065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388037070 con 0x7ff3a00d4800 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.110+0000 7ff39f7fe640 1 -- 172.21.15.46:0/238415065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff388032440 con 0x7ff3a00d4800 2024-05-09T03:20:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.110+0000 7ff39f7fe640 1 -- 172.21.15.46:0/238415065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388032810 con 0x7ff3a00d4800 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 -- 172.21.15.46:0/238415065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 msgr2=0x7ff3a00d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a00d4c10 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7ff3a008ed70 tx=0x7ff38802a140 comp rx=0 tx=0).stop 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 -- 172.21.15.46:0/238415065 shutdown_connections 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 --2- 172.21.15.46:0/238415065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a00d4c10 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 --2- 172.21.15.46:0/238415065 >> v2:172.21.15.88:3300/0 conn(0x7ff3a00b5210 0x7ff3a01613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 -- 172.21.15.46:0/238415065 >> 172.21.15.46:0/238415065 conn(0x7ff3a00d29a0 msgr2=0x7ff3a00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 -- 172.21.15.46:0/238415065 shutdown_connections 2024-05-09T03:20:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.111+0000 7ff3a702b640 1 -- 172.21.15.46:0/238415065 wait complete. 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.113+0000 7ff3a702b640 1 Processor -- start 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.113+0000 7ff3a702b640 1 -- start start 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.114+0000 7ff3a702b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a00b5210 0x7ff3a00b9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.114+0000 7ff3a702b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.114+0000 7ff3a702b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a0152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.114+0000 7ff3a702b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a00d0ae0 con 0x7ff3a00d4800 2024-05-09T03:20:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.114+0000 7ff3a702b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a0160ea0 con 0x7ff3a00b5210 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.114+0000 7ff3a702b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a0163e60 con 0x7ff3a00b77c0 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a0152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a0152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33536/0 (socket says 172.21.15.46:33536) 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 -- 172.21.15.46:0/1594694771 learned_addr learned my addr 172.21.15.46:0/1594694771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a4dc7640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a00b5210 0x7ff3a00b9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff39ffff640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 -- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 msgr2=0x7ff3a00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 -- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a00b5210 msgr2=0x7ff3a00b9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a00b5210 0x7ff3a00b9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 -- 172.21.15.46:0/1594694771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a00ce530 con 0x7ff3a00d4800 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a4dc7640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a00b5210 0x7ff3a00b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff39ffff640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a55c8640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a0152bf0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7ff39800cd80 tx=0x7ff39800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39800f680 con 0x7ff3a00d4800 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff39800f840 con 0x7ff3a00d4800 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a00cdd20 con 0x7ff3a00d4800 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39800fc30 con 0x7ff3a00d4800 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.115+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a00ceab0 con 0x7ff3a00d4800 2024-05-09T03:20:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.116+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a00cda90 con 0x7ff3a00d4800 2024-05-09T03:20:59.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.117+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff3980213f0 con 0x7ff3a00d4800 2024-05-09T03:20:59.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.117+0000 7ff39d7fa640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff380076300 0x7ff380078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.118+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4857+0+0 (secure 0 0 0) 0x7ff398098020 con 0x7ff3a00d4800 2024-05-09T03:20:59.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.118+0000 7ff3a4dc7640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff380076300 0x7ff380078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.118+0000 7ff3a4dc7640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff380076300 0x7ff380078810 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff39000b840 tx=0x7ff390008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:59.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.122+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff398014030 con 0x7ff3a00d4800 2024-05-09T03:20:59.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.315+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff3a006a270 con 0x7ff3a00d4800 2024-05-09T03:20:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.315+0000 7ff39d7fa640 1 -- 172.21.15.46:0/1594694771 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v131) ==== 84+0+65 (secure 0 0 0) 0x7ff398061c40 con 0x7ff3a00d4800 2024-05-09T03:20:59.316 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:20:59.316 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:20:59.316 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:20:59.316 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-05-09T03:20:59.317 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-05-09T03:20:59.317 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:20:59.317 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff380076300 msgr2=0x7ff380078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff380076300 0x7ff380078810 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff39000b840 tx=0x7ff390008040 comp rx=0 tx=0).stop 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 msgr2=0x7ff3a0152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a0152bf0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7ff39800cd80 tx=0x7ff39800a650 comp rx=0 tx=0).stop 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 shutdown_connections 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff380076300 0x7ff380078810 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3a00d4800 0x7ff3a0152bf0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff3a00b77c0 0x7ff3a00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 --2- 172.21.15.46:0/1594694771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3a00b5210 0x7ff3a00b9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 >> 172.21.15.46:0/1594694771 conn(0x7ff3a00d29a0 msgr2=0x7ff3a0164380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 shutdown_connections 2024-05-09T03:20:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.318+0000 7ff3a702b640 1 -- 172.21.15.46:0/1594694771 wait complete. 2024-05-09T03:20:59.330 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T03:20:59.330 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T03:20:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.448+0000 7f5028ae7640 1 Processor -- start 2024-05-09T03:20:59.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.448+0000 7f5028ae7640 1 -- start start 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5028ae7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f50240b5140 0x7f50240b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5028ae7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50240b8070 0x7f502406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5028ae7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f502406e250 0x7f50240d48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5028ae7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f50240b9b60 con 0x7f50240b8070 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5028ae7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f50240b5550 con 0x7f50240b5140 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5028ae7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f50240b8480 con 0x7f502406e250 2024-05-09T03:20:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5021d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50240b8070 0x7f502406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5021d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f50240b8070 0x7f502406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33548/0 (socket says 172.21.15.46:33548) 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5021d9b640 1 -- 172.21.15.46:0/3515011687 learned_addr learned my addr 172.21.15.46:0/3515011687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f502259c640 1 --2- 172.21.15.46:0/3515011687 >> v2:172.21.15.88:3300/0 conn(0x7f50240b5140 0x7f50240b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.449+0000 7f5022d9d640 1 --2- 172.21.15.46:0/3515011687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f502406e250 0x7f50240d48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.450+0000 7f5021d9b640 1 -- 172.21.15.46:0/3515011687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f502406e250 msgr2=0x7f50240d48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.450+0000 7f5021d9b640 1 --2- 172.21.15.46:0/3515011687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f502406e250 0x7f50240d48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.450+0000 7f5021d9b640 1 -- 172.21.15.46:0/3515011687 >> v2:172.21.15.88:3300/0 conn(0x7f50240b5140 msgr2=0x7f50240b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.450+0000 7f5021d9b640 1 --2- 172.21.15.46:0/3515011687 >> v2:172.21.15.88:3300/0 conn(0x7f50240b5140 0x7f50240b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.450+0000 7f5021d9b640 1 -- 172.21.15.46:0/3515011687 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502406e660 con 0x7f50240b8070 2024-05-09T03:20:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.450+0000 7f5021d9b640 1 --2- 172.21.15.46:0/3515011687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502406dd10 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f502408e1e0 tx=0x7f500c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0e14f33acd25f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f502159a640 1 -- 172.21.15.46:0/3515011687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c037070 con 0x7f50240b8070 2024-05-09T03:20:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f502159a640 1 -- 172.21.15.46:0/3515011687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f500c032440 con 0x7f50240b8070 2024-05-09T03:20:59.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f502159a640 1 -- 172.21.15.46:0/3515011687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c032810 con 0x7f50240b8070 2024-05-09T03:20:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 -- 172.21.15.46:0/3515011687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 msgr2=0x7f502406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 --2- 172.21.15.46:0/3515011687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502406dd10 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f502408e1e0 tx=0x7f500c02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 -- 172.21.15.46:0/3515011687 shutdown_connections 2024-05-09T03:20:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 --2- 172.21.15.46:0/3515011687 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f502406e250 0x7f50240d48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 --2- 172.21.15.46:0/3515011687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502406dd10 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 --2- 172.21.15.46:0/3515011687 >> v2:172.21.15.88:3300/0 conn(0x7f50240b5140 0x7f50240b7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 -- 172.21.15.46:0/3515011687 >> 172.21.15.46:0/3515011687 conn(0x7f50240d5e90 msgr2=0x7f50240d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:20:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 -- 172.21.15.46:0/3515011687 shutdown_connections 2024-05-09T03:20:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.451+0000 7f5028ae7640 1 -- 172.21.15.46:0/3515011687 wait complete. 2024-05-09T03:20:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.453+0000 7f5028ae7640 1 Processor -- start 2024-05-09T03:20:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.453+0000 7f5028ae7640 1 -- start start 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5028ae7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f502406e250 0x7f5024163660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5028ae7640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50240b5140 0x7f50240ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5028ae7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502415e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5028ae7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50240b9510 con 0x7f50240b8070 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5028ae7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f50240b9270 con 0x7f502406e250 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5028ae7640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f50240b97f0 con 0x7f50240b5140 2024-05-09T03:20:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502415e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33558/0 (socket says 172.21.15.46:33558) 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 -- 172.21.15.46:0/2545320314 learned_addr learned my addr 172.21.15.46:0/2545320314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f502259c640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f502406e250 0x7f5024163660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5021d9b640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50240b5140 0x7f50240ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 -- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50240b5140 msgr2=0x7f50240ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50240b5140 0x7f50240ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 -- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f502406e250 msgr2=0x7f5024163660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f502406e250 0x7f5024163660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 -- 172.21.15.46:0/2545320314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50240add20 con 0x7f50240b8070 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f502259c640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f502406e250 0x7f5024163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5021d9b640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50240b5140 0x7f50240ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:20:59.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.455+0000 7f5022d9d640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502415e690 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f501400eed0 tx=0x7f501400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.456+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5014018020 con 0x7f50240b8070 2024-05-09T03:20:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.456+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501400bd30 con 0x7f50240b8070 2024-05-09T03:20:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.456+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50241fa560 con 0x7f50240b8070 2024-05-09T03:20:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.456+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5014014640 con 0x7f50240b8070 2024-05-09T03:20:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.456+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50240acfe0 con 0x7f50240b8070 2024-05-09T03:20:59.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.457+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f502406a270 con 0x7f50240b8070 2024-05-09T03:20:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.457+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5014005d10 con 0x7f50240b8070 2024-05-09T03:20:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.458+0000 7f501affd640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50040762d0 0x7f50040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:20:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.458+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4857+0+0 (secure 0 0 0) 0x7f5014097b70 con 0x7f50240b8070 2024-05-09T03:20:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.459+0000 7f502259c640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50040762d0 0x7f50040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:20:59.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.459+0000 7f502259c640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50040762d0 0x7f50040787e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f501c009af0 tx=0x7f501c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:20:59.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.463+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5014022030 con 0x7f50240b8070 2024-05-09T03:20:59.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:20:59.658+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f50241fa2e0 con 0x7f50240b8070 2024-05-09T03:21:00.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.561+0000 7f501affd640 1 -- 172.21.15.46:0/2545320314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v132) ==== 138+0+0 (secure 0 0 0) 0x7f50241fa2e0 con 0x7f50240b8070 2024-05-09T03:21:00.562 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-05-09T03:21:00.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.564+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50040762d0 msgr2=0x7f50040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50040762d0 0x7f50040787e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f501c009af0 tx=0x7f501c008040 comp rx=0 tx=0).stop 2024-05-09T03:21:00.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 msgr2=0x7f502415e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502415e690 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f501400eed0 tx=0x7f501400b480 comp rx=0 tx=0).stop 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 shutdown_connections 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f50040762d0 0x7f50040787e0 secure :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f501c009af0 tx=0x7f501c008040 comp rx=0 tx=0).stop 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50240b8070 0x7f502415e690 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f50240b5140 0x7f50240ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 --2- 172.21.15.46:0/2545320314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f502406e250 0x7f5024163660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.565+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 >> 172.21.15.46:0/2545320314 conn(0x7f50240d5e90 msgr2=0x7f50240b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:00.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.567+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 shutdown_connections 2024-05-09T03:21:00.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.567+0000 7f5028ae7640 1 -- 172.21.15.46:0/2545320314 wait complete. 2024-05-09T03:21:00.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T03:21:00.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.713+0000 7fb2a4fb2640 1 Processor -- start 2024-05-09T03:21:00.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.713+0000 7fb2a4fb2640 1 -- start start 2024-05-09T03:21:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb2a4fb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00b91b0 0x7fb2a00b95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb2a4fb2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a00b9b00 con 0x7fb2a00d1380 2024-05-09T03:21:00.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb2a4fb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a00d45d0 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb2a4fb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a00d1790 con 0x7fb2a00b91b0 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29dd9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2a00d1380 0x7fb2a00d4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60672/0 (socket says 172.21.15.46:60672) 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29dd9b640 1 -- 172.21.15.46:0/158889937 learned_addr learned my addr 172.21.15.46:0/158889937 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29ed9d640 1 --2- 172.21.15.46:0/158889937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00b91b0 0x7fb2a00b95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130255731 0 0) 0x7fb2a00b9b00 con 0x7fb2a00d1380 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb288003520 con 0x7fb2a00d1380 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167805023 0 0) 0x7fb2a00d45d0 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.715+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2a00b9b00 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.716+0000 7fb29ed9d640 1 -- 172.21.15.46:0/158889937 >> v1:172.21.15.88:6789/0 conn(0x7fb2a00bb5b0 legacy=0x7fb2a0163dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.716+0000 7fb29ed9d640 1 -- 172.21.15.46:0/158889937 >> v1:172.21.15.46:6789/0 conn(0x7fb2a00d1380 legacy=0x7fb2a00d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.716+0000 7fb29ed9d640 1 -- 172.21.15.46:0/158889937 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a01645e0 con 0x7fb2a00b91b0 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.716+0000 7fb29ed9d640 1 --2- 172.21.15.46:0/158889937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00b91b0 0x7fb2a00b95c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb298009dc0 tx=0x7fb29802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc83a080796b3bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298007740 con 0x7fb2a00b91b0 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2980078e0 con 0x7fb2a00b91b0 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb29d59a640 1 -- 172.21.15.46:0/158889937 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298007cb0 con 0x7fb2a00b91b0 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/158889937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00b91b0 msgr2=0x7fb2a00b95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/158889937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00b91b0 0x7fb2a00b95c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb298009dc0 tx=0x7fb29802a6d0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/158889937 shutdown_connections 2024-05-09T03:21:00.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/158889937 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00b91b0 0x7fb2a00b95c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/158889937 >> 172.21.15.46:0/158889937 conn(0x7fb2a00d5e80 msgr2=0x7fb2a00d6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/158889937 shutdown_connections 2024-05-09T03:21:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.717+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/158889937 wait complete. 2024-05-09T03:21:00.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.719+0000 7fb2a4fb2640 1 Processor -- start 2024-05-09T03:21:00.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.719+0000 7fb2a4fb2640 1 -- start start 2024-05-09T03:21:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb2a4fb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a00b91b0 0x7fb2a00aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:00.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb2a4fb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 0x7fb2a01f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb2a4fb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00d1380 0x7fb2a00b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb2a4fb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a00d54a0 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb2a4fb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a0164ba0 con 0x7fb2a00b91b0 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb2a4fb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a00d5620 con 0x7fb2a00d1380 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 0x7fb2a01f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 0x7fb2a01f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33570/0 (socket says 172.21.15.46:33570) 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 -- 172.21.15.46:0/3136209260 learned_addr learned my addr 172.21.15.46:0/3136209260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29ed9d640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00d1380 0x7fb2a00b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29e59c640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a00b91b0 0x7fb2a00aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 -- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00d1380 msgr2=0x7fb2a00b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00d1380 0x7fb2a00b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 -- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a00b91b0 msgr2=0x7fb2a00aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a00b91b0 0x7fb2a00aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:00.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 -- 172.21.15.46:0/3136209260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a00ac8d0 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb29dd9b640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 0x7fb2a01f69b0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c410 tx=0x7fb29400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.721+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2940058b0 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.722+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb294019030 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.722+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a00acbb0 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.722+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294005c90 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.722+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a00ad170 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.723+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a00af280 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.723+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb294007070 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.724+0000 7fb28effd640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb280076300 0x7fb280078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.724+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 5227+0+0 (secure 0 0 0) 0x7fb29409ef20 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.724+0000 7fb29e59c640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb280076300 0x7fb280078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:00.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.725+0000 7fb29e59c640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb280076300 0x7fb280078810 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb290000c00 tx=0x7fb290009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:00.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.729+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb294068410 con 0x7fb2a00bb5b0 2024-05-09T03:21:00.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:00.922+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb2a0069140 con 0x7fb2a00bb5b0 2024-05-09T03:21:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.566+0000 7fb28effd640 1 -- 172.21.15.46:0/3136209260 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v133) ==== 170+0+0 (secure 0 0 0) 0x7fb2a0069140 con 0x7fb2a00bb5b0 2024-05-09T03:21:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.570+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb280076300 msgr2=0x7fb280078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.570+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb280076300 0x7fb280078810 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb290000c00 tx=0x7fb290009040 comp rx=0 tx=0).stop 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.570+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 msgr2=0x7fb2a01f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.570+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 0x7fb2a01f69b0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c410 tx=0x7fb29400c8d0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.571+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 shutdown_connections 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.571+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2a00d1380 0x7fb2a00b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.571+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb280076300 0x7fb280078810 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.571+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2a00bb5b0 0x7fb2a01f69b0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.571+0000 7fb2a4fb2640 1 --2- 172.21.15.46:0/3136209260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a00b91b0 0x7fb2a00aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.571+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 >> 172.21.15.46:0/3136209260 conn(0x7fb2a00d5e80 msgr2=0x7fb2a00ba670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.572+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 shutdown_connections 2024-05-09T03:21:01.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.579+0000 7fb2a4fb2640 1 -- 172.21.15.46:0/3136209260 wait complete. 2024-05-09T03:21:01.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T03:21:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.719+0000 7f9e726d1640 1 Processor -- start 2024-05-09T03:21:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.719+0000 7f9e726d1640 1 -- start start 2024-05-09T03:21:01.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e726d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e6c0b7e70 0x7f9e6c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e726d1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6c0b5550 con 0x7f9e6c0b5140 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e726d1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6c0d44e0 con 0x7f9e6c0d3df0 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e726d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9e6c0d3480 con 0x7f9e6c0b7e70 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9e6c0b5140 0x7f9e6c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60674/0 (socket says 172.21.15.46:60674) 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6bfff640 1 -- 172.21.15.46:0/1335787915 learned_addr learned my addr 172.21.15.46:0/1335787915 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1295452708 0 0) 0x7f9e6c0b5550 con 0x7f9e6c0b5140 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e6c0b7e70 0x7f9e6c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e54003520 con 0x7f9e6c0b5140 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1112452833 0 0) 0x7f9e54003520 con 0x7f9e6c0b5140 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e6c0b7e70 msgr2=0x7f9e6c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e6c0b7e70 0x7f9e6c0d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 >> v1:172.21.15.88:6789/0 conn(0x7f9e6c0d3df0 legacy=0x7f9e6c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6c0d47c0 con 0x7f9e6c0b5140 2024-05-09T03:21:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.721+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f9e5c0016f0 con 0x7f9e6c0b5140 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 >> v1:172.21.15.46:6789/0 conn(0x7f9e6c0b5140 legacy=0x7f9e6c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e540039e0 0x7f9e54003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e54004800 0x7f9e54004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e540051a0 0x7f9e5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5400bf00 con 0x7f9e54004800 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5400c080 con 0x7f9e540039e0 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9e5400c280 con 0x7f9e540051a0 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e54004800 0x7f9e54004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6bfff640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e540039e0 0x7f9e54003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e70c6e640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e540051a0 0x7f9e5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e540051a0 msgr2=0x7f9e5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e540051a0 0x7f9e5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e540039e0 msgr2=0x7f9e54003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e540039e0 0x7f9e54003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.722+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/1335787915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e54003520 con 0x7f9e54004800 2024-05-09T03:21:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.723+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e54004800 0x7f9e54004c10 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009ac0 tx=0x7f9e54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e3caddecab9dc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.723+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60036070 con 0x7f9e54004800 2024-05-09T03:21:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.723+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e60031400 con 0x7f9e54004800 2024-05-09T03:21:01.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.723+0000 7f9e6affd640 1 -- 172.21.15.46:0/1335787915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60009690 con 0x7f9e54004800 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e726d1640 1 -- 172.21.15.46:0/1335787915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e54004800 msgr2=0x7f9e54004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e726d1640 1 --2- 172.21.15.46:0/1335787915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e54004800 0x7f9e54004c10 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009ac0 tx=0x7f9e54005830 comp rx=0 tx=0).stop 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e70c6e640 1 -- 172.21.15.46:0/1335787915 reap_dead start 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e726d1640 1 -- 172.21.15.46:0/1335787915 shutdown_connections 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e726d1640 1 -- 172.21.15.46:0/1335787915 >> 172.21.15.46:0/1335787915 conn(0x7f9e6c08d3f0 msgr2=0x7f9e6c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e726d1640 1 -- 172.21.15.46:0/1335787915 shutdown_connections 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.724+0000 7f9e726d1640 1 -- 172.21.15.46:0/1335787915 wait complete. 2024-05-09T03:21:01.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.726+0000 7f9e726d1640 1 Processor -- start 2024-05-09T03:21:01.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.726+0000 7f9e726d1640 1 -- start start 2024-05-09T03:21:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e726d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e54004800 0x7f9e6c0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e726d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 0x7f9e6c0b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e726d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0b72a0 0x7f9e6c1f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e726d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6c0d4200 con 0x7f9e6c15e970 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e726d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6c0d4d80 con 0x7f9e6c0b72a0 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e726d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f9e6c0d5580 con 0x7f9e54004800 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e6b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 0x7f9e6c0b6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e6b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 0x7f9e6c0b6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33602/0 (socket says 172.21.15.46:33602) 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/2833244225 learned_addr learned my addr 172.21.15.46:0/2833244225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e6bfff640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e54004800 0x7f9e6c0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.727+0000 7f9e70c6e640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0b72a0 0x7f9e6c1f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e54004800 msgr2=0x7f9e6c0c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e54004800 0x7f9e6c0c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0b72a0 msgr2=0x7f9e6c1f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0b72a0 0x7f9e6c1f5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:01.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e6b7fe640 1 -- 172.21.15.46:0/2833244225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6c146940 con 0x7f9e6c15e970 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e6b7fe640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 0x7f9e6c0b6d60 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f9e60034040 tx=0x7f9e60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60040020 con 0x7f9e6c15e970 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e60003bc0 con 0x7f9e6c15e970 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6c1f9a90 con 0x7f9e6c15e970 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e60036040 con 0x7f9e6c15e970 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.728+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6c146be0 con 0x7f9e6c15e970 2024-05-09T03:21:01.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.729+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e6c06a270 con 0x7f9e6c15e970 2024-05-09T03:21:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.730+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9e60043020 con 0x7f9e6c15e970 2024-05-09T03:21:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.730+0000 7f9e68ff9640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9e58076300 0x7f9e58078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.731+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 5609+0+0 (secure 0 0 0) 0x7f9e600b7e30 con 0x7f9e6c15e970 2024-05-09T03:21:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.731+0000 7f9e6bfff640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9e58076300 0x7f9e58078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:01.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.731+0000 7f9e6bfff640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9e58076300 0x7f9e58078810 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c001eb0 tx=0x7f9e5c0011c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:01.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.735+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9e6004a070 con 0x7f9e6c15e970 2024-05-09T03:21:01.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:01.964+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f9e6c164bf0 con 0x7f9e6c15e970 2024-05-09T03:21:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.568+0000 7f9e68ff9640 1 -- 172.21.15.46:0/2833244225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v349) ==== 174+0+0 (secure 0 0 0) 0x7f9e600044e0 con 0x7f9e6c15e970 2024-05-09T03:21:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 14 and data pool 15 2024-05-09T03:21:02.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.573+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9e58076300 msgr2=0x7f9e58078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.573+0000 7f9e726d1640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9e58076300 0x7f9e58078810 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c001eb0 tx=0x7f9e5c0011c0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.573+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 msgr2=0x7f9e6c0b6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.573+0000 7f9e726d1640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 0x7f9e6c0b6d60 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f9e60034040 tx=0x7f9e60004730 comp rx=0 tx=0).stop 2024-05-09T03:21:02.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.574+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 shutdown_connections 2024-05-09T03:21:02.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.574+0000 7f9e726d1640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e6c0b72a0 0x7f9e6c1f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.574+0000 7f9e726d1640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f9e58076300 0x7f9e58078810 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.574+0000 7f9e726d1640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e6c15e970 0x7f9e6c0b6d60 secure :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f9e60034040 tx=0x7f9e60004730 comp rx=0 tx=0).stop 2024-05-09T03:21:02.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.574+0000 7f9e726d1640 1 --2- 172.21.15.46:0/2833244225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f9e54004800 0x7f9e6c0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.574+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 >> 172.21.15.46:0/2833244225 conn(0x7f9e6c08d3f0 msgr2=0x7f9e6c0b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:02.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.575+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 shutdown_connections 2024-05-09T03:21:02.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.575+0000 7f9e726d1640 1 -- 172.21.15.46:0/2833244225 wait complete. 2024-05-09T03:21:02.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:21:02.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.728+0000 7fdf50f34640 1 Processor -- start 2024-05-09T03:21:02.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.728+0000 7fdf50f34640 1 -- start start 2024-05-09T03:21:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf50f34640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:02.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf50f34640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf4c0d2480 0x7fdf4c0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf50f34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf4c160b20 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf50f34640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdf4c161150 con 0x7fdf4c0d2480 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf50f34640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdf4c0d15f0 con 0x7fdf4c164c70 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf4ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf4c0d2480 0x7fdf4c0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf4ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdf4c0d2480 0x7fdf4c0d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33604/0 (socket says 172.21.15.46:33604) 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf4ad9d640 1 -- 172.21.15.46:0/1187073183 learned_addr learned my addr 172.21.15.46:0/1187073183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.729+0000 7fdf4a59c640 1 --2- 172.21.15.46:0/1187073183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.730+0000 7fdf4ad9d640 1 -- 172.21.15.46:0/1187073183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 msgr2=0x7fdf4c1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.730+0000 7fdf4ad9d640 1 --2- 172.21.15.46:0/1187073183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.730+0000 7fdf4ad9d640 1 -- 172.21.15.46:0/1187073183 >> v1:172.21.15.88:6789/0 conn(0x7fdf4c15dfe0 legacy=0x7fdf4c0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.730+0000 7fdf4ad9d640 1 -- 172.21.15.46:0/1187073183 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c0d18d0 con 0x7fdf4c0d2480 2024-05-09T03:21:02.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.730+0000 7fdf4ad9d640 1 --2- 172.21.15.46:0/1187073183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c0d2890 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009e50 tx=0x7fdf4402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256fb8187d806a48 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf4959a640 1 -- 172.21.15.46:0/1187073183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf44036070 con 0x7fdf4c0d2480 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf4959a640 1 -- 172.21.15.46:0/1187073183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf44031600 con 0x7fdf4c0d2480 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf4959a640 1 -- 172.21.15.46:0/1187073183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf440319d0 con 0x7fdf4c0d2480 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 -- 172.21.15.46:0/1187073183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c0d2480 msgr2=0x7fdf4c0d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 --2- 172.21.15.46:0/1187073183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c0d2890 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7fdf44009e50 tx=0x7fdf4402a710 comp rx=0 tx=0).stop 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 -- 172.21.15.46:0/1187073183 shutdown_connections 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 --2- 172.21.15.46:0/1187073183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c0d2890 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 --2- 172.21.15.46:0/1187073183 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 -- 172.21.15.46:0/1187073183 >> 172.21.15.46:0/1187073183 conn(0x7fdf4c073db0 msgr2=0x7fdf4c0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 -- 172.21.15.46:0/1187073183 shutdown_connections 2024-05-09T03:21:02.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.731+0000 7fdf50f34640 1 -- 172.21.15.46:0/1187073183 wait complete. 2024-05-09T03:21:02.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.733+0000 7fdf50f34640 1 Processor -- start 2024-05-09T03:21:02.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.733+0000 7fdf50f34640 1 -- start start 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf50f34640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf50f34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 0x7fdf4c1eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf50f34640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf50f34640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c0d2dd0 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf50f34640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c15e3f0 con 0x7fdf4c0d2480 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf50f34640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdf4c0d45c0 con 0x7fdf4c164c70 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 0x7fdf4c1eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 0x7fdf4c1eaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33614/0 (socket says 172.21.15.46:33614) 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 -- 172.21.15.46:0/2478611289 learned_addr learned my addr 172.21.15.46:0/2478611289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf4a59c640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf4ad9d640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:02.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 -- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 msgr2=0x7fdf4c1f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 -- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf4c0d2480 msgr2=0x7fdf4c1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c1524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 -- 172.21.15.46:0/2478611289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c1f7550 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf49d9b640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 0x7fdf4c1eaab0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000c340 tx=0x7fdf4000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.735+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40017070 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.736+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf400052d0 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.736+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4c1e9fd0 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.736+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf40003810 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.736+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4c1ea590 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.736+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4c08cfd0 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.737+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdf4001a070 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.738+0000 7fdf3affd640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdf2c076300 0x7fdf2c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:02.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.738+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5709+0+0 (secure 0 0 0) 0x7fdf40096c80 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.738+0000 7fdf4a59c640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdf2c076300 0x7fdf2c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:02.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.739+0000 7fdf4a59c640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdf2c076300 0x7fdf2c078810 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009a30 tx=0x7fdf3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:02.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.742+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdf4c08cfd0 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.935+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf4c06a270 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.936+0000 7fdf3affd640 1 -- 172.21.15.46:0/2478611289 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+16560 (secure 0 0 0) 0x7fdf40021020 con 0x7fdf4c15dfe0 2024-05-09T03:21:02.937 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:02.937 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":134,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:21:01.965825+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:20:59.663057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-09T03:21:00.927261+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.938+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdf2c076300 msgr2=0x7fdf2c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.938+0000 7fdf50f34640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdf2c076300 0x7fdf2c078810 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c009a30 tx=0x7fdf3c008040 comp rx=0 tx=0).stop 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.938+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 msgr2=0x7fdf4c1eaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.938+0000 7fdf50f34640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 0x7fdf4c1eaab0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7fdf4000c340 tx=0x7fdf4000c800 comp rx=0 tx=0).stop 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.938+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 shutdown_connections 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.938+0000 7fdf50f34640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdf4c164c70 0x7fdf4c1f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.939+0000 7fdf50f34640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdf2c076300 0x7fdf2c078810 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.939+0000 7fdf50f34640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf4c15dfe0 0x7fdf4c1eaab0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.939+0000 7fdf50f34640 1 --2- 172.21.15.46:0/2478611289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf4c0d2480 0x7fdf4c1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.939+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 >> 172.21.15.46:0/2478611289 conn(0x7fdf4c073db0 msgr2=0x7fdf4c0bb040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.939+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 shutdown_connections 2024-05-09T03:21:02.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:02.939+0000 7fdf50f34640 1 -- 172.21.15.46:0/2478611289 wait complete. 2024-05-09T03:21:02.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:21:03.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.065+0000 7ff138bb4640 1 Processor -- start 2024-05-09T03:21:03.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.065+0000 7ff138bb4640 1 -- start start 2024-05-09T03:21:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.066+0000 7ff138bb4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1340b92e0 0x7ff1340b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.066+0000 7ff138bb4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340d0480 0x7ff1340d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.066+0000 7ff138bb4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1340ce6a0 0x7ff1340ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.066+0000 7ff138bb4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff1340d0dd0 con 0x7ff1340ce6a0 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.066+0000 7ff138bb4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff1340d2b80 con 0x7ff1340b92e0 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.066+0000 7ff138bb4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff1340d2e60 con 0x7ff1340d0480 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1340ce6a0 0x7ff1340ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1340ce6a0 0x7ff1340ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33618/0 (socket says 172.21.15.46:33618) 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 -- 172.21.15.46:0/40991030 learned_addr learned my addr 172.21.15.46:0/40991030 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff129d9b640 1 --2- 172.21.15.46:0/40991030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340d0480 0x7ff1340d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff13259c640 1 --2- 172.21.15.46:0/40991030 >> v2:172.21.15.88:3300/0 conn(0x7ff1340b92e0 0x7ff1340b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 -- 172.21.15.46:0/40991030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340d0480 msgr2=0x7ff1340d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 --2- 172.21.15.46:0/40991030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340d0480 0x7ff1340d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 -- 172.21.15.46:0/40991030 >> v2:172.21.15.88:3300/0 conn(0x7ff1340b92e0 msgr2=0x7ff1340b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 --2- 172.21.15.46:0/40991030 >> v2:172.21.15.88:3300/0 conn(0x7ff1340b92e0 0x7ff1340b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.067+0000 7ff132d9d640 1 -- 172.21.15.46:0/40991030 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1340d3140 con 0x7ff1340ce6a0 2024-05-09T03:21:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.068+0000 7ff132d9d640 1 --2- 172.21.15.46:0/40991030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340ce6a0 0x7ff1340ceab0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7ff13406d120 tx=0x7ff12c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01ad93e84f17f94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.068+0000 7ff131d9b640 1 -- 172.21.15.46:0/40991030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c036070 con 0x7ff1340ce6a0 2024-05-09T03:21:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.068+0000 7ff131d9b640 1 -- 172.21.15.46:0/40991030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff12c031600 con 0x7ff1340ce6a0 2024-05-09T03:21:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.068+0000 7ff131d9b640 1 -- 172.21.15.46:0/40991030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c0319d0 con 0x7ff1340ce6a0 2024-05-09T03:21:03.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.068+0000 7ff138bb4640 1 -- 172.21.15.46:0/40991030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340ce6a0 msgr2=0x7ff1340ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.068+0000 7ff138bb4640 1 --2- 172.21.15.46:0/40991030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340ce6a0 0x7ff1340ceab0 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7ff13406d120 tx=0x7ff12c02a610 comp rx=0 tx=0).stop 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 -- 172.21.15.46:0/40991030 shutdown_connections 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 --2- 172.21.15.46:0/40991030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340ce6a0 0x7ff1340ceab0 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 --2- 172.21.15.46:0/40991030 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340d0480 0x7ff1340d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 --2- 172.21.15.46:0/40991030 >> v2:172.21.15.88:3300/0 conn(0x7ff1340b92e0 0x7ff1340b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 -- 172.21.15.46:0/40991030 >> 172.21.15.46:0/40991030 conn(0x7ff1340d3c00 msgr2=0x7ff1340d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 -- 172.21.15.46:0/40991030 shutdown_connections 2024-05-09T03:21:03.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.069+0000 7ff138bb4640 1 -- 172.21.15.46:0/40991030 wait complete. 2024-05-09T03:21:03.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.071+0000 7ff138bb4640 1 Processor -- start 2024-05-09T03:21:03.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.071+0000 7ff138bb4640 1 -- start start 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff138bb4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340b92e0 0x7ff134155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff138bb4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1340ce6a0 0x7ff1340cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff138bb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 0x7ff13415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff138bb4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1340cf170 con 0x7ff1340d0480 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff138bb4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff1340ceff0 con 0x7ff1340ce6a0 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff138bb4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff1340cf2f0 con 0x7ff1340b92e0 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff132d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 0x7ff13415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff132d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 0x7ff13415c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33632/0 (socket says 172.21.15.46:33632) 2024-05-09T03:21:03.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff132d9d640 1 -- 172.21.15.46:0/4121119121 learned_addr learned my addr 172.21.15.46:0/4121119121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff13259c640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340b92e0 0x7ff134155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff129d9b640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1340ce6a0 0x7ff1340cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff132d9d640 1 -- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340b92e0 msgr2=0x7ff134155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff132d9d640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340b92e0 0x7ff134155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.072+0000 7ff132d9d640 1 -- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1340ce6a0 msgr2=0x7ff1340cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff132d9d640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1340ce6a0 0x7ff1340cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff132d9d640 1 -- 172.21.15.46:0/4121119121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff134156aa0 con 0x7ff1340d0480 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff132d9d640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 0x7ff13415c790 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7ff12c034040 tx=0x7ff12c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c036050 con 0x7ff1340d0480 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff12c042070 con 0x7ff1340d0480 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c009510 con 0x7ff1340d0480 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff134156d40 con 0x7ff1340d0480 2024-05-09T03:21:03.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.073+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff134157300 con 0x7ff1340d0480 2024-05-09T03:21:03.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.074+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff13406a270 con 0x7ff1340d0480 2024-05-09T03:21:03.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.075+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff12c007bf0 con 0x7ff1340d0480 2024-05-09T03:21:03.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.075+0000 7ff12b7fe640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1140762d0 0x7ff1140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.075+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5709+0+0 (secure 0 0 0) 0x7ff12c0b8240 con 0x7ff1340d0480 2024-05-09T03:21:03.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.075+0000 7ff13259c640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1140762d0 0x7ff1140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.076+0000 7ff13259c640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1140762d0 0x7ff1140787e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff134071390 tx=0x7ff124009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:03.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.080+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff12c0814c0 con 0x7ff1340d0480 2024-05-09T03:21:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.271+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1340cdef0 con 0x7ff1340d0480 2024-05-09T03:21:03.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.272+0000 7ff12b7fe640 1 -- 172.21.15.46:0/4121119121 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+16560 (secure 0 0 0) 0x7ff12c0857c0 con 0x7ff1340d0480 2024-05-09T03:21:03.273 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:03.273 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":134,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:21:01.965825+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:20:59.663057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-09T03:21:00.927261+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:21:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1140762d0 msgr2=0x7ff1140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1140762d0 0x7ff1140787e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff134071390 tx=0x7ff124009190 comp rx=0 tx=0).stop 2024-05-09T03:21:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 msgr2=0x7ff13415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 0x7ff13415c790 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7ff12c034040 tx=0x7ff12c004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 shutdown_connections 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff1140762d0 0x7ff1140787e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1340d0480 0x7ff13415c790 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1340ce6a0 0x7ff1340cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 --2- 172.21.15.46:0/4121119121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff1340b92e0 0x7ff134155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 >> 172.21.15.46:0/4121119121 conn(0x7ff1340d3c00 msgr2=0x7ff13406e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 shutdown_connections 2024-05-09T03:21:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.274+0000 7ff138bb4640 1 -- 172.21.15.46:0/4121119121 wait complete. 2024-05-09T03:21:03.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T03:21:03.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.404+0000 7f18cbe4c640 1 Processor -- start 2024-05-09T03:21:03.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.405+0000 7f18cbe4c640 1 -- start start 2024-05-09T03:21:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18cbe4c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40d2800 0x7f18c40b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18cbe4c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18c40b3f50 0x7f18c40ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18cbe4c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c406ebe0 con 0x7f18c40b4700 2024-05-09T03:21:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18cbe4c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f18c406eec0 con 0x7f18c40b3f50 2024-05-09T03:21:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18cbe4c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f18c406f1a0 con 0x7f18c40d2800 2024-05-09T03:21:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18ca3e9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18c40b3f50 0x7f18c40ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18ca3e9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18c40b3f50 0x7f18c40ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33642/0 (socket says 172.21.15.46:33642) 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18ca3e9640 1 -- 172.21.15.46:0/1145041285 learned_addr learned my addr 172.21.15.46:0/1145041285 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.406+0000 7f18c93e7640 1 --2- 172.21.15.46:0/1145041285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40d2800 0x7f18c40b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.407+0000 7f18ca3e9640 1 -- 172.21.15.46:0/1145041285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40d2800 msgr2=0x7f18c40b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.407+0000 7f18ca3e9640 1 --2- 172.21.15.46:0/1145041285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40d2800 0x7f18c40b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.407+0000 7f18ca3e9640 1 -- 172.21.15.46:0/1145041285 >> v1:172.21.15.88:6789/0 conn(0x7f18c40b4700 legacy=0x7f18c40d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.407+0000 7f18ca3e9640 1 -- 172.21.15.46:0/1145041285 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c406f480 con 0x7f18c40b3f50 2024-05-09T03:21:03.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.407+0000 7f18ca3e9640 1 --2- 172.21.15.46:0/1145041285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40b3f50 0x7f18c40ce130 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009e50 tx=0x7f18bc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae030c45dc2f31c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:03.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.407+0000 7f18c8be6640 1 -- 172.21.15.46:0/1145041285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc036070 con 0x7f18c40b3f50 2024-05-09T03:21:03.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18c8be6640 1 -- 172.21.15.46:0/1145041285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc031600 con 0x7f18c40b3f50 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18c8be6640 1 -- 172.21.15.46:0/1145041285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0319d0 con 0x7f18c40b3f50 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 -- 172.21.15.46:0/1145041285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40b3f50 msgr2=0x7f18c40ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/1145041285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40b3f50 0x7f18c40ce130 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009e50 tx=0x7f18bc02a680 comp rx=0 tx=0).stop 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 -- 172.21.15.46:0/1145041285 shutdown_connections 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/1145041285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40b3f50 0x7f18c40ce130 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/1145041285 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40d2800 0x7f18c40b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 -- 172.21.15.46:0/1145041285 >> 172.21.15.46:0/1145041285 conn(0x7f18c40b55d0 msgr2=0x7f18c40b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 -- 172.21.15.46:0/1145041285 shutdown_connections 2024-05-09T03:21:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.408+0000 7f18cbe4c640 1 -- 172.21.15.46:0/1145041285 wait complete. 2024-05-09T03:21:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.410+0000 7f18cbe4c640 1 Processor -- start 2024-05-09T03:21:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.410+0000 7f18cbe4c640 1 -- start start 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18cbe4c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c40b3f50 0x7f18c40cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18cbe4c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40b4700 0x7f18c41d76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18cbe4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 0x7f18c4071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18cbe4c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f18c40b51d0 con 0x7f18c40d2800 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18cbe4c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18c40b4360 con 0x7f18c40b3f50 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18cbe4c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f18c40a1580 con 0x7f18c40b4700 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 0x7f18c4071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 0x7f18c4071cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33650/0 (socket says 172.21.15.46:33650) 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 -- 172.21.15.46:0/2793904702 learned_addr learned my addr 172.21.15.46:0/2793904702 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18c93e7640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40b4700 0x7f18c41d76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18c9be8640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c40b3f50 0x7f18c40cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 -- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40b4700 msgr2=0x7f18c41d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40b4700 0x7f18c41d76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 -- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c40b3f50 msgr2=0x7f18c40cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c40b3f50 0x7f18c40cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 -- 172.21.15.46:0/2793904702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c40cc230 con 0x7f18c40d2800 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.412+0000 7f18ca3e9640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 0x7f18c4071cc0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009e50 tx=0x7f18bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.413+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc036050 con 0x7f18c40d2800 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.413+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc042070 con 0x7f18c40d2800 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.413+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c41da0f0 con 0x7f18c40d2800 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.413+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc009510 con 0x7f18c40d2800 2024-05-09T03:21:03.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.413+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c41da6b0 con 0x7f18c40d2800 2024-05-09T03:21:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.414+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c406a270 con 0x7f18c40d2800 2024-05-09T03:21:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.415+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18bc007bf0 con 0x7f18c40d2800 2024-05-09T03:21:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.415+0000 7f18ba7fc640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a80762d0 0x7f18a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.415+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 5709+0+0 (secure 0 0 0) 0x7f18bc0b8ce0 con 0x7f18c40d2800 2024-05-09T03:21:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.416+0000 7f18c9be8640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a80762d0 0x7f18a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.416+0000 7f18c9be8640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a80762d0 0x7f18a80787e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f18c0001720 tx=0x7f18c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:03.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.419+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f18bc0821a0 con 0x7f18c40d2800 2024-05-09T03:21:03.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:03.650+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f18c40cc4d0 con 0x7f18c40d2800 2024-05-09T03:21:04.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.593+0000 7f18ba7fc640 1 -- 172.21.15.46:0/2793904702 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v352) ==== 118+0+0 (secure 0 0 0) 0x7f18bc0864a0 con 0x7f18c40d2800 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a80762d0 msgr2=0x7f18a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a80762d0 0x7f18a80787e0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f18c0001720 tx=0x7f18c0009040 comp rx=0 tx=0).stop 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 msgr2=0x7f18c4071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 0x7f18c4071cc0 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009e50 tx=0x7f18bc004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 shutdown_connections 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f18a80762d0 0x7f18a80787e0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18c40d2800 0x7f18c4071cc0 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f18c40b4700 0x7f18c41d76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 --2- 172.21.15.46:0/2793904702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c40b3f50 0x7f18c40cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 >> 172.21.15.46:0/2793904702 conn(0x7f18c40b55d0 msgr2=0x7f18c4072a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 shutdown_connections 2024-05-09T03:21:04.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.596+0000 7f18cbe4c640 1 -- 172.21.15.46:0/2793904702 wait complete. 2024-05-09T03:21:04.611 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-05-09T03:21:04.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T03:21:04.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.736+0000 7f4073fb2640 1 Processor -- start 2024-05-09T03:21:04.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.736+0000 7f4073fb2640 1 -- start start 2024-05-09T03:21:04.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f4073fb2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f406c0d2e90 0x7f406c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:04.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f4073fb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f406c0b4fc0 0x7f406c0b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:04.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f4073fb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f406c0d3400 con 0x7f406c1605b0 2024-05-09T03:21:04.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f4073fb2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0d4a60 con 0x7f406c0b4fc0 2024-05-09T03:21:04.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f4073fb2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f406c0d4d40 con 0x7f406c0d2e90 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f406c0b4fc0 0x7f406c0b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f406c0b4fc0 0x7f406c0b7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33666/0 (socket says 172.21.15.46:33666) 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 -- 172.21.15.46:0/152310627 learned_addr learned my addr 172.21.15.46:0/152310627 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f4071d4e640 1 --2- 172.21.15.46:0/152310627 >> v2:172.21.15.88:3300/0 conn(0x7f406c0d2e90 0x7f406c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 -- 172.21.15.46:0/152310627 >> v2:172.21.15.88:3300/0 conn(0x7f406c0d2e90 msgr2=0x7f406c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 --2- 172.21.15.46:0/152310627 >> v2:172.21.15.88:3300/0 conn(0x7f406c0d2e90 0x7f406c0b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 -- 172.21.15.46:0/152310627 >> v1:172.21.15.88:6789/0 conn(0x7f406c1605b0 legacy=0x7f406c0d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.738+0000 7f407254f640 1 -- 172.21.15.46:0/152310627 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c0d5020 con 0x7f406c0b4fc0 2024-05-09T03:21:04.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.739+0000 7f407254f640 1 --2- 172.21.15.46:0/152310627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0b7ca0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f406c08e310 tx=0x7f406402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba65ee800a97b42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:04.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.739+0000 7f4070d4c640 1 -- 172.21.15.46:0/152310627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064037070 con 0x7f406c0b4fc0 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.739+0000 7f4070d4c640 1 -- 172.21.15.46:0/152310627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4064032440 con 0x7f406c0b4fc0 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.739+0000 7f4070d4c640 1 -- 172.21.15.46:0/152310627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064032810 con 0x7f406c0b4fc0 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 -- 172.21.15.46:0/152310627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 msgr2=0x7f406c0b7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 --2- 172.21.15.46:0/152310627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0b7ca0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f406c08e310 tx=0x7f406402a140 comp rx=0 tx=0).stop 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 -- 172.21.15.46:0/152310627 shutdown_connections 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 --2- 172.21.15.46:0/152310627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0b7ca0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 --2- 172.21.15.46:0/152310627 >> v2:172.21.15.88:3300/0 conn(0x7f406c0d2e90 0x7f406c0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 -- 172.21.15.46:0/152310627 >> 172.21.15.46:0/152310627 conn(0x7f406c08d390 msgr2=0x7f406c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 -- 172.21.15.46:0/152310627 shutdown_connections 2024-05-09T03:21:04.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.740+0000 7f4073fb2640 1 -- 172.21.15.46:0/152310627 wait complete. 2024-05-09T03:21:04.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.742+0000 7f4073fb2640 1 Processor -- start 2024-05-09T03:21:04.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.742+0000 7f4073fb2640 1 -- start start 2024-05-09T03:21:04.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4073fb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:04.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4073fb2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f406c0d2e90 0x7f406c0cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4073fb2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f406c1605b0 0x7f406c0cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4073fb2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0d5540 con 0x7f406c0b4fc0 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4073fb2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f406c0b4c00 con 0x7f406c1605b0 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4073fb2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f406c0cb660 con 0x7f406c0d2e90 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4071d4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4071d4e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33674/0 (socket says 172.21.15.46:33674) 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4071d4e640 1 -- 172.21.15.46:0/115926954 learned_addr learned my addr 172.21.15.46:0/115926954 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f407254f640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f406c1605b0 0x7f406c0cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f4071d4e640 1 -- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f406c0d2e90 msgr2=0x7f406c0cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.743+0000 7f407154d640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f406c0d2e90 0x7f406c0cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4071d4e640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f406c0d2e90 0x7f406c0cbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4071d4e640 1 -- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f406c1605b0 msgr2=0x7f406c0cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4071d4e640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f406c1605b0 0x7f406c0cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4071d4e640 1 -- 172.21.15.46:0/115926954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c0cd7c0 con 0x7f406c0b4fc0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f407254f640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f406c1605b0 0x7f406c0cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f407154d640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f406c0d2e90 0x7f406c0cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4071d4e640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0cccb0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ccc0 tx=0x7f405c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c003990 con 0x7f406c0b4fc0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405c003b30 con 0x7f406c0b4fc0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c0cdaa0 con 0x7f406c0b4fc0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c004030 con 0x7f406c0b4fc0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.744+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c153540 con 0x7f406c0b4fc0 2024-05-09T03:21:04.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.745+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f406c06a270 con 0x7f406c0b4fc0 2024-05-09T03:21:04.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.746+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f405c0041d0 con 0x7f406c0b4fc0 2024-05-09T03:21:04.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.746+0000 7f406a7fc640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40580762d0 0x7f40580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:04.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.746+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f405c097a00 con 0x7f406c0b4fc0 2024-05-09T03:21:04.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.746+0000 7f407154d640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40580762d0 0x7f40580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:04.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.747+0000 7f407154d640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40580762d0 0x7f40580787e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f406c0cd260 tx=0x7f4060008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:04.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.751+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f405c014070 con 0x7f406c0b4fc0 2024-05-09T03:21:04.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:04.981+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f406c0cdc70 con 0x7f406c0b4fc0 2024-05-09T03:21:05.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.598+0000 7f406a7fc640 1 -- 172.21.15.46:0/115926954 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v353) ==== 122+0+0 (secure 0 0 0) 0x7f405c0613f0 con 0x7f406c0b4fc0 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40580762d0 msgr2=0x7f40580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40580762d0 0x7f40580787e0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f406c0cd260 tx=0x7f4060008040 comp rx=0 tx=0).stop 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 msgr2=0x7f406c0cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0cccb0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ccc0 tx=0x7f405c00a5c0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 shutdown_connections 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f406c1605b0 0x7f406c0cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f406c0d2e90 0x7f406c0cbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40580762d0 0x7f40580787e0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 --2- 172.21.15.46:0/115926954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f406c0b4fc0 0x7f406c0cccb0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 >> 172.21.15.46:0/115926954 conn(0x7f406c08d390 msgr2=0x7f406c0d5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 shutdown_connections 2024-05-09T03:21:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.602+0000 7f4073fb2640 1 -- 172.21.15.46:0/115926954 wait complete. 2024-05-09T03:21:05.617 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T03:21:05.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.739+0000 7f610eaee640 1 Processor -- start 2024-05-09T03:21:05.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.739+0000 7f610eaee640 1 -- start start 2024-05-09T03:21:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f610eaee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080b5210 0x7f61081613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f610eaee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61080d4800 0x7f61080d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f610eaee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61080d5280 con 0x7f61080b77c0 2024-05-09T03:21:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f610eaee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f61080b5620 con 0x7f61080d4800 2024-05-09T03:21:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f610eaee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f61081609a0 con 0x7f61080b5210 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f6107fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61080d4800 0x7f61080d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f6107fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f61080d4800 0x7f61080d4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33686/0 (socket says 172.21.15.46:33686) 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f6107fff640 1 -- 172.21.15.46:0/1723382888 learned_addr learned my addr 172.21.15.46:0/1723382888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.740+0000 7f610c88a640 1 --2- 172.21.15.46:0/1723382888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080b5210 0x7f61081613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.741+0000 7f6107fff640 1 -- 172.21.15.46:0/1723382888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080b5210 msgr2=0x7f61081613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.741+0000 7f6107fff640 1 --2- 172.21.15.46:0/1723382888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080b5210 0x7f61081613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.741+0000 7f6107fff640 1 -- 172.21.15.46:0/1723382888 >> v1:172.21.15.88:6789/0 conn(0x7f61080b77c0 legacy=0x7f61080b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:05.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.741+0000 7f6107fff640 1 -- 172.21.15.46:0/1723382888 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61080d5560 con 0x7f61080d4800 2024-05-09T03:21:05.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.741+0000 7f6107fff640 1 --2- 172.21.15.46:0/1723382888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080d4800 0x7f61080d4c10 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009870 tx=0x7f60f802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fc9c178ae788a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:05.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.741+0000 7f61077fe640 1 -- 172.21.15.46:0/1723382888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8037070 con 0x7f61080d4800 2024-05-09T03:21:05.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f61077fe640 1 -- 172.21.15.46:0/1723382888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f8032440 con 0x7f61080d4800 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f61077fe640 1 -- 172.21.15.46:0/1723382888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8032810 con 0x7f61080d4800 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 -- 172.21.15.46:0/1723382888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080d4800 msgr2=0x7f61080d4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 --2- 172.21.15.46:0/1723382888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080d4800 0x7f61080d4c10 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009870 tx=0x7f60f802a1c0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 -- 172.21.15.46:0/1723382888 shutdown_connections 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 --2- 172.21.15.46:0/1723382888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080d4800 0x7f61080d4c10 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 --2- 172.21.15.46:0/1723382888 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080b5210 0x7f61081613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 -- 172.21.15.46:0/1723382888 >> 172.21.15.46:0/1723382888 conn(0x7f61080d29a0 msgr2=0x7f61080d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 -- 172.21.15.46:0/1723382888 shutdown_connections 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.742+0000 7f610eaee640 1 -- 172.21.15.46:0/1723382888 wait complete. 2024-05-09T03:21:05.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.744+0000 7f610eaee640 1 Processor -- start 2024-05-09T03:21:05.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.744+0000 7f610eaee640 1 -- start start 2024-05-09T03:21:05.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610eaee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 0x7f610815b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610eaee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61080b77c0 0x7f61080c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610eaee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080d4800 0x7f61080b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610eaee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61080d0ae0 con 0x7f61080b5210 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610eaee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6108160ea0 con 0x7f61080b77c0 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610eaee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f6108163e60 con 0x7f61080d4800 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 0x7f610815b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 0x7f610815b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33698/0 (socket says 172.21.15.46:33698) 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 -- 172.21.15.46:0/3285172549 learned_addr learned my addr 172.21.15.46:0/3285172549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f6107fff640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61080b77c0 0x7f61080c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610d08b640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080d4800 0x7f61080b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 -- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080d4800 msgr2=0x7f61080b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080d4800 0x7f61080b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 -- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61080b77c0 msgr2=0x7f61080c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61080b77c0 0x7f61080c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:05.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 -- 172.21.15.46:0/3285172549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610815bb00 con 0x7f61080b5210 2024-05-09T03:21:05.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f610c88a640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 0x7f610815b5c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f60f000d620 tx=0x7f60f000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:05.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.746+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f000b620 con 0x7f61080b5210 2024-05-09T03:21:05.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.747+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f0019070 con 0x7f61080b5210 2024-05-09T03:21:05.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.747+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61080c9d90 con 0x7f61080b5210 2024-05-09T03:21:05.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.747+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f000ba00 con 0x7f61080b5210 2024-05-09T03:21:05.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.747+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61080ca2a0 con 0x7f61080b5210 2024-05-09T03:21:05.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.747+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610806a270 con 0x7f61080b5210 2024-05-09T03:21:05.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.748+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f60f000bba0 con 0x7f61080b5210 2024-05-09T03:21:05.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.749+0000 7f61057fa640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f60e80762d0 0x7f60e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:05.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.749+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f60f0097940 con 0x7f61080b5210 2024-05-09T03:21:05.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.749+0000 7f6107fff640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f60e80762d0 0x7f60e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:05.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.749+0000 7f6107fff640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f60e80762d0 0x7f60e80787e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f60f8034000 tx=0x7f60f802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:05.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.753+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f60f0060bc0 con 0x7f61080b5210 2024-05-09T03:21:05.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.983+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61080d40d0 con 0x7f61080b5210 2024-05-09T03:21:06.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.984+0000 7f61057fa640 1 -- 172.21.15.46:0/3285172549 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) ==== 95+0+5188 (secure 0 0 0) 0x7f60f0064ec0 con 0x7f61080b5210 2024-05-09T03:21:06.126 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:06.126 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":353,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:04.982418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10644},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10644":{"gid":10644,"name":"c","rank":0,"incarnation":349,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2230300620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2230300620},{"type":"v1","addr":"172.21.15.46:6835","nonce":2230300620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10644,"qdb_cluster":[10644]},"id":7}]} 2024-05-09T03:21:06.128 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 353 2024-05-09T03:21:06.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f60e80762d0 msgr2=0x7f60e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f60e80762d0 0x7f60e80787e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f60f8034000 tx=0x7f60f802e000 comp rx=0 tx=0).stop 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 msgr2=0x7f610815b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 0x7f610815b5c0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f60f000d620 tx=0x7f60f000dae0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 shutdown_connections 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f61080d4800 0x7f61080b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61080b77c0 0x7f61080c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f60e80762d0 0x7f60e80787e0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 --2- 172.21.15.46:0/3285172549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61080b5210 0x7f610815b5c0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 >> 172.21.15.46:0/3285172549 conn(0x7f61080d29a0 msgr2=0x7f610815c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:06.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 shutdown_connections 2024-05-09T03:21:06.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:05.986+0000 7f610eaee640 1 -- 172.21.15.46:0/3285172549 wait complete. 2024-05-09T03:21:06.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:21:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.246+0000 7fea37270640 1 Processor -- start 2024-05-09T03:21:06.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.246+0000 7fea37270640 1 -- start start 2024-05-09T03:21:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea37270640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea30160610 0x7fea300b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea37270640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fea300b7d20 0x7fea300b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea37270640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea300b8130 con 0x7fea300d57a0 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea37270640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fea300d5bb0 con 0x7fea30160610 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea37270640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fea300d49e0 con 0x7fea300b7d20 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea3480b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea30160610 0x7fea300b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea3480b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea30160610 0x7fea300b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45644/0 (socket says 172.21.15.46:45644) 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.247+0000 7fea3480b640 1 -- 172.21.15.46:0/2675028225 learned_addr learned my addr 172.21.15.46:0/2675028225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.248+0000 7fea3580d640 1 --2- 172.21.15.46:0/2675028225 >> v2:172.21.15.88:3300/0 conn(0x7fea300b7d20 0x7fea300b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.248+0000 7fea3480b640 1 -- 172.21.15.46:0/2675028225 >> v2:172.21.15.88:3300/0 conn(0x7fea300b7d20 msgr2=0x7fea300b9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.248+0000 7fea3480b640 1 --2- 172.21.15.46:0/2675028225 >> v2:172.21.15.88:3300/0 conn(0x7fea300b7d20 0x7fea300b9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.248+0000 7fea3480b640 1 -- 172.21.15.46:0/2675028225 >> v1:172.21.15.88:6789/0 conn(0x7fea300d57a0 legacy=0x7fea300b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.248+0000 7fea3480b640 1 -- 172.21.15.46:0/2675028225 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea300d4cc0 con 0x7fea30160610 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea3480b640 1 --2- 172.21.15.46:0/2675028225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b77e0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fea301640f0 tx=0x7fea2002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ddc1b22238bc91 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea27fff640 1 -- 172.21.15.46:0/2675028225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20037070 con 0x7fea30160610 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea27fff640 1 -- 172.21.15.46:0/2675028225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea20032440 con 0x7fea30160610 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea27fff640 1 -- 172.21.15.46:0/2675028225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea20032810 con 0x7fea30160610 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea37270640 1 -- 172.21.15.46:0/2675028225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 msgr2=0x7fea300b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea37270640 1 --2- 172.21.15.46:0/2675028225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b77e0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fea301640f0 tx=0x7fea2002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea37270640 1 -- 172.21.15.46:0/2675028225 shutdown_connections 2024-05-09T03:21:06.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea37270640 1 --2- 172.21.15.46:0/2675028225 >> v2:172.21.15.88:3300/0 conn(0x7fea300b7d20 0x7fea300b9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea37270640 1 --2- 172.21.15.46:0/2675028225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b77e0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.249+0000 7fea37270640 1 -- 172.21.15.46:0/2675028225 >> 172.21.15.46:0/2675028225 conn(0x7fea300743e0 msgr2=0x7fea300d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:06.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.250+0000 7fea37270640 1 -- 172.21.15.46:0/2675028225 shutdown_connections 2024-05-09T03:21:06.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.250+0000 7fea37270640 1 -- 172.21.15.46:0/2675028225 wait complete. 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.251+0000 7fea37270640 1 Processor -- start 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.251+0000 7fea37270640 1 -- start start 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea37270640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea300b7d20 0x7fea301ea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea37270640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea300d57a0 0x7fea30154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea37270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea37270640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fea30160a20 con 0x7fea30160610 2024-05-09T03:21:06.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea37270640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fea300d5280 con 0x7fea300d57a0 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea37270640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fea300d15b0 con 0x7fea300b7d20 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45656/0 (socket says 172.21.15.46:45656) 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 -- 172.21.15.46:0/3216614848 learned_addr learned my addr 172.21.15.46:0/3216614848 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3480b640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea300d57a0 0x7fea30154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3500c640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea300b7d20 0x7fea301ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 -- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea300b7d20 msgr2=0x7fea301ea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea300b7d20 0x7fea301ea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 -- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea300d57a0 msgr2=0x7fea30154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea300d57a0 0x7fea30154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 -- 172.21.15.46:0/3216614848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3015a7d0 con 0x7fea30160610 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea3580d640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b6730 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fea28007e90 tx=0x7fea2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.253+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28014420 con 0x7fea30160610 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.254+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea28019070 con 0x7fea30160610 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.254+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28014800 con 0x7fea30160610 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.254+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3015aab0 con 0x7fea30160610 2024-05-09T03:21:06.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.254+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea301e80f0 con 0x7fea30160610 2024-05-09T03:21:06.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.255+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea3006a270 con 0x7fea30160610 2024-05-09T03:21:06.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.255+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fea280149a0 con 0x7fea30160610 2024-05-09T03:21:06.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.256+0000 7fea25ffb640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea140762d0 0x7fea140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.256+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fea28097d00 con 0x7fea30160610 2024-05-09T03:21:06.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.256+0000 7fea3500c640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea140762d0 0x7fea140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.257+0000 7fea3500c640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea140762d0 0x7fea140787e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fea2c006fd0 tx=0x7fea2c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.261+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fea28061010 con 0x7fea30160610 2024-05-09T03:21:06.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.451+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea300ba690 con 0x7fea30160610 2024-05-09T03:21:06.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.452+0000 7fea25ffb640 1 -- 172.21.15.46:0/3216614848 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+16547 (secure 0 0 0) 0x7fea28065310 con 0x7fea30160610 2024-05-09T03:21:06.454 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:06.454 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":135,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:21:03.582672+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:20:59.663057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-09T03:21:00.927261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea140762d0 msgr2=0x7fea140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea140762d0 0x7fea140787e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fea2c006fd0 tx=0x7fea2c009060 comp rx=0 tx=0).stop 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 msgr2=0x7fea300b6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b6730 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7fea28007e90 tx=0x7fea2800e4b0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 shutdown_connections 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fea140762d0 0x7fea140787e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea30160610 0x7fea300b6730 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fea300d57a0 0x7fea30154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 --2- 172.21.15.46:0/3216614848 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fea300b7d20 0x7fea301ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 >> 172.21.15.46:0/3216614848 conn(0x7fea300743e0 msgr2=0x7fea301eb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 shutdown_connections 2024-05-09T03:21:06.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.454+0000 7fea37270640 1 -- 172.21.15.46:0/3216614848 wait complete. 2024-05-09T03:21:06.467 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:21:06.467 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:21:06.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.581+0000 7f339ba7e640 1 Processor -- start 2024-05-09T03:21:06.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.581+0000 7f339ba7e640 1 -- start start 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339ba7e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f33940b5210 0x7f33941613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339ba7e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33940d5280 con 0x7f33940b7ed0 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339ba7e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33940d5560 con 0x7f33940d4800 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339ba7e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f33940b5620 con 0x7f33940b5210 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339a01b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f33940b7ed0 0x7f3394163e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40918/0 (socket says 172.21.15.46:40918) 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339a01b640 1 -- 172.21.15.46:0/3318862033 learned_addr learned my addr 172.21.15.46:0/3318862033 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339981a640 1 --2- 172.21.15.46:0/3318862033 >> v2:172.21.15.88:3300/0 conn(0x7f33940b5210 0x7f33941613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f3398818640 1 -- 172.21.15.46:0/3318862033 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 77193554 0 0) 0x7f33940d5280 con 0x7f33940b7ed0 2024-05-09T03:21:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f3398818640 1 -- 172.21.15.46:0/3318862033 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3380003520 con 0x7f33940b7ed0 2024-05-09T03:21:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339981a640 1 -- 172.21.15.46:0/3318862033 >> v1:172.21.15.88:6789/0 conn(0x7f33940d4800 legacy=0x7f33940b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339981a640 1 -- 172.21.15.46:0/3318862033 >> v1:172.21.15.46:6789/0 conn(0x7f33940b7ed0 legacy=0x7f3394163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.583+0000 7f339981a640 1 -- 172.21.15.46:0/3318862033 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3394160bc0 con 0x7f33940b5210 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.584+0000 7f339981a640 1 --2- 172.21.15.46:0/3318862033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f33941613c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f33900097a0 tx=0x7f339002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b485770cfa82d71e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.584+0000 7f3398818640 1 -- 172.21.15.46:0/3318862033 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3390037070 con 0x7f33940b5210 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.584+0000 7f3398818640 1 -- 172.21.15.46:0/3318862033 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3390032440 con 0x7f33940b5210 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.584+0000 7f3398818640 1 -- 172.21.15.46:0/3318862033 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33900327f0 con 0x7f33940b5210 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 -- 172.21.15.46:0/3318862033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 msgr2=0x7f33941613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 --2- 172.21.15.46:0/3318862033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f33941613c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f33900097a0 tx=0x7f339002a230 comp rx=0 tx=0).stop 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 -- 172.21.15.46:0/3318862033 shutdown_connections 2024-05-09T03:21:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 --2- 172.21.15.46:0/3318862033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f33941613c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 -- 172.21.15.46:0/3318862033 >> 172.21.15.46:0/3318862033 conn(0x7f33940d29a0 msgr2=0x7f33940d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 -- 172.21.15.46:0/3318862033 shutdown_connections 2024-05-09T03:21:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.585+0000 7f339ba7e640 1 -- 172.21.15.46:0/3318862033 wait complete. 2024-05-09T03:21:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.587+0000 7f339ba7e640 1 Processor -- start 2024-05-09T03:21:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.587+0000 7f339ba7e640 1 -- start start 2024-05-09T03:21:06.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339ba7e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f339414aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339ba7e640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33940d4800 0x7f33940b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339ba7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 0x7f33940c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339ba7e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33940d0ae0 con 0x7f339414b570 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339ba7e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f33941609a0 con 0x7f33940b5210 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339ba7e640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f33940d4c10 con 0x7f33940d4800 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339a01b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 0x7f33940c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339a01b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 0x7f33940c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45662/0 (socket says 172.21.15.46:45662) 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.588+0000 7f339a01b640 1 -- 172.21.15.46:0/498637261 learned_addr learned my addr 172.21.15.46:0/498637261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339981a640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f339414aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f3399019640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33940d4800 0x7f33940b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339a01b640 1 -- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33940d4800 msgr2=0x7f33940b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339a01b640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33940d4800 0x7f33940b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339a01b640 1 -- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 msgr2=0x7f339414aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339a01b640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f339414aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339a01b640 1 -- 172.21.15.46:0/498637261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33940b6af0 con 0x7f339414b570 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339a01b640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 0x7f33940c9850 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f338c00caa0 tx=0x7f338c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c00aba0 con 0x7f339414b570 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f338c00ad40 con 0x7f339414b570 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33940b6dd0 con 0x7f339414b570 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338c012600 con 0x7f339414b570 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.589+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33940b7390 con 0x7f339414b570 2024-05-09T03:21:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.590+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f339406a270 con 0x7f339414b570 2024-05-09T03:21:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.591+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f338c0127a0 con 0x7f339414b570 2024-05-09T03:21:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.592+0000 7f338a7fc640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33780762d0 0x7f33780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.592+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f338c097a10 con 0x7f339414b570 2024-05-09T03:21:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.592+0000 7f339981a640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33780762d0 0x7f33780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.592+0000 7f339981a640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33780762d0 0x7f33780787e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3390034000 tx=0x7f339002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.596+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f338c060d20 con 0x7f339414b570 2024-05-09T03:21:06.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.783+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f339414ba50 con 0x7f33780762d0 2024-05-09T03:21:06.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.790+0000 7f338a7fc640 1 -- 172.21.15.46:0/498637261 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252476 (secure 0 0 0) 0x7f3390038040 con 0x7f33780762d0 2024-05-09T03:21:06.795 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:06.797 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:21:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33780762d0 msgr2=0x7f33780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33780762d0 0x7f33780787e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3390034000 tx=0x7f339002e000 comp rx=0 tx=0).stop 2024-05-09T03:21:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 msgr2=0x7f33940c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 0x7f33940c9850 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f338c00caa0 tx=0x7f338c00cf60 comp rx=0 tx=0).stop 2024-05-09T03:21:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 shutdown_connections 2024-05-09T03:21:06.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f33780762d0 0x7f33780787e0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f339414b570 0x7f33940c9850 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f33940d4800 0x7f33940b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 --2- 172.21.15.46:0/498637261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33940b5210 0x7f339414aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 >> 172.21.15.46:0/498637261 conn(0x7f33940d29a0 msgr2=0x7f339415ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:06.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 shutdown_connections 2024-05-09T03:21:06.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.793+0000 7f339ba7e640 1 -- 172.21.15.46:0/498637261 wait complete. 2024-05-09T03:21:06.806 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-05-09T03:21:06.681083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":293412,"kb_used_data":14824,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754681308,"statfs":{"total":773094113280,"available":772793659392,"internally_reserved":0,"allocated":15179776,"data_stored":7920124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":285134385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001409"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589494+0000","last_change":"2024-05-09T03:21:02.596216+0000","last_active":"2024-05-09T03:21:03.589494+0000","last_peered":"2024-05-09T03:21:03.589494+0000","last_clean":"2024-05-09T03:21:03.589494+0000","last_became_active":"2024-05-09T03:21:02.595888+0000","last_became_peered":"2024-05-09T03:21:02.595888+0000","last_unstale":"2024-05-09T03:21:03.589494+0000","last_undegraded":"2024-05-09T03:21:03.589494+0000","last_fullsized":"2024-05-09T03:21:03.589494+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:58:28.100174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806522+0000","last_change":"2024-05-09T03:21:02.604820+0000","last_active":"2024-05-09T03:21:03.806522+0000","last_peered":"2024-05-09T03:21:03.806522+0000","last_clean":"2024-05-09T03:21:03.806522+0000","last_became_active":"2024-05-09T03:21:02.604509+0000","last_became_peered":"2024-05-09T03:21:02.604509+0000","last_unstale":"2024-05-09T03:21:03.806522+0000","last_undegraded":"2024-05-09T03:21:03.806522+0000","last_fullsized":"2024-05-09T03:21:03.806522+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:05.678532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589276+0000","last_change":"2024-05-09T03:21:02.595502+0000","last_active":"2024-05-09T03:21:03.589276+0000","last_peered":"2024-05-09T03:21:03.589276+0000","last_clean":"2024-05-09T03:21:03.589276+0000","last_became_active":"2024-05-09T03:21:02.595191+0000","last_became_peered":"2024-05-09T03:21:02.595191+0000","last_unstale":"2024-05-09T03:21:03.589276+0000","last_undegraded":"2024-05-09T03:21:03.589276+0000","last_fullsized":"2024-05-09T03:21:03.589276+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:34:56.154675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806239+0000","last_change":"2024-05-09T03:21:02.595556+0000","last_active":"2024-05-09T03:21:03.806239+0000","last_peered":"2024-05-09T03:21:03.806239+0000","last_clean":"2024-05-09T03:21:03.806239+0000","last_became_active":"2024-05-09T03:21:02.595251+0000","last_became_peered":"2024-05-09T03:21:02.595251+0000","last_unstale":"2024-05-09T03:21:03.806239+0000","last_undegraded":"2024-05-09T03:21:03.806239+0000","last_fullsized":"2024-05-09T03:21:03.806239+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:34.875542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590314+0000","last_change":"2024-05-09T03:21:02.594854+0000","last_active":"2024-05-09T03:21:03.590314+0000","last_peered":"2024-05-09T03:21:03.590314+0000","last_clean":"2024-05-09T03:21:03.590314+0000","last_became_active":"2024-05-09T03:21:02.594545+0000","last_became_peered":"2024-05-09T03:21:02.594545+0000","last_unstale":"2024-05-09T03:21:03.590314+0000","last_undegraded":"2024-05-09T03:21:03.590314+0000","last_fullsized":"2024-05-09T03:21:03.590314+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:57:55.162362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806127+0000","last_change":"2024-05-09T03:21:02.591379+0000","last_active":"2024-05-09T03:21:03.806127+0000","last_peered":"2024-05-09T03:21:03.806127+0000","last_clean":"2024-05-09T03:21:03.806127+0000","last_became_active":"2024-05-09T03:21:02.591064+0000","last_became_peered":"2024-05-09T03:21:02.591064+0000","last_unstale":"2024-05-09T03:21:03.806127+0000","last_undegraded":"2024-05-09T03:21:03.806127+0000","last_fullsized":"2024-05-09T03:21:03.806127+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:21.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806368+0000","last_change":"2024-05-09T03:21:02.592163+0000","last_active":"2024-05-09T03:21:03.806368+0000","last_peered":"2024-05-09T03:21:03.806368+0000","last_clean":"2024-05-09T03:21:03.806368+0000","last_became_active":"2024-05-09T03:21:02.591851+0000","last_became_peered":"2024-05-09T03:21:02.591851+0000","last_unstale":"2024-05-09T03:21:03.806368+0000","last_undegraded":"2024-05-09T03:21:03.806368+0000","last_fullsized":"2024-05-09T03:21:03.806368+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:28:58.074717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807261+0000","last_change":"2024-05-09T03:21:02.599477+0000","last_active":"2024-05-09T03:21:03.807261+0000","last_peered":"2024-05-09T03:21:03.807261+0000","last_clean":"2024-05-09T03:21:03.807261+0000","last_became_active":"2024-05-09T03:21:02.599152+0000","last_became_peered":"2024-05-09T03:21:02.599152+0000","last_unstale":"2024-05-09T03:21:03.807261+0000","last_undegraded":"2024-05-09T03:21:03.807261+0000","last_fullsized":"2024-05-09T03:21:03.807261+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:14.796836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591763+0000","last_change":"2024-05-09T03:21:02.596464+0000","last_active":"2024-05-09T03:21:03.591763+0000","last_peered":"2024-05-09T03:21:03.591763+0000","last_clean":"2024-05-09T03:21:03.591763+0000","last_became_active":"2024-05-09T03:21:02.594727+0000","last_became_peered":"2024-05-09T03:21:02.594727+0000","last_unstale":"2024-05-09T03:21:03.591763+0000","last_undegraded":"2024-05-09T03:21:03.591763+0000","last_fullsized":"2024-05-09T03:21:03.591763+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:26.766250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590718+0000","last_change":"2024-05-09T03:21:02.590312+0000","last_active":"2024-05-09T03:21:03.590718+0000","last_peered":"2024-05-09T03:21:03.590718+0000","last_clean":"2024-05-09T03:21:03.590718+0000","last_became_active":"2024-05-09T03:21:02.590024+0000","last_became_peered":"2024-05-09T03:21:02.590024+0000","last_unstale":"2024-05-09T03:21:03.590718+0000","last_undegraded":"2024-05-09T03:21:03.590718+0000","last_fullsized":"2024-05-09T03:21:03.590718+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:21.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806031+0000","last_change":"2024-05-09T03:21:02.590850+0000","last_active":"2024-05-09T03:21:03.806031+0000","last_peered":"2024-05-09T03:21:03.806031+0000","last_clean":"2024-05-09T03:21:03.806031+0000","last_became_active":"2024-05-09T03:21:02.590558+0000","last_became_peered":"2024-05-09T03:21:02.590558+0000","last_unstale":"2024-05-09T03:21:03.806031+0000","last_undegraded":"2024-05-09T03:21:03.806031+0000","last_fullsized":"2024-05-09T03:21:03.806031+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:16.316105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591410+0000","last_change":"2024-05-09T03:21:02.584457+0000","last_active":"2024-05-09T03:21:03.591410+0000","last_peered":"2024-05-09T03:21:03.591410+0000","last_clean":"2024-05-09T03:21:03.591410+0000","last_became_active":"2024-05-09T03:21:02.584151+0000","last_became_peered":"2024-05-09T03:21:02.584151+0000","last_unstale":"2024-05-09T03:21:03.591410+0000","last_undegraded":"2024-05-09T03:21:03.591410+0000","last_fullsized":"2024-05-09T03:21:03.591410+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:30:19.274582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806035+0000","last_change":"2024-05-09T03:21:02.587071+0000","last_active":"2024-05-09T03:21:03.806035+0000","last_peered":"2024-05-09T03:21:03.806035+0000","last_clean":"2024-05-09T03:21:03.806035+0000","last_became_active":"2024-05-09T03:21:02.586781+0000","last_became_peered":"2024-05-09T03:21:02.586781+0000","last_unstale":"2024-05-09T03:21:03.806035+0000","last_undegraded":"2024-05-09T03:21:03.806035+0000","last_fullsized":"2024-05-09T03:21:03.806035+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:21.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806590+0000","last_change":"2024-05-09T03:21:02.595240+0000","last_active":"2024-05-09T03:21:03.806590+0000","last_peered":"2024-05-09T03:21:03.806590+0000","last_clean":"2024-05-09T03:21:03.806590+0000","last_became_active":"2024-05-09T03:21:02.594019+0000","last_became_peered":"2024-05-09T03:21:02.594019+0000","last_unstale":"2024-05-09T03:21:03.806590+0000","last_undegraded":"2024-05-09T03:21:03.806590+0000","last_fullsized":"2024-05-09T03:21:03.806590+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:40.633909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805546+0000","last_change":"2024-05-09T03:21:02.588503+0000","last_active":"2024-05-09T03:21:03.805546+0000","last_peered":"2024-05-09T03:21:03.805546+0000","last_clean":"2024-05-09T03:21:03.805546+0000","last_became_active":"2024-05-09T03:21:02.588164+0000","last_became_peered":"2024-05-09T03:21:02.588164+0000","last_unstale":"2024-05-09T03:21:03.805546+0000","last_undegraded":"2024-05-09T03:21:03.805546+0000","last_fullsized":"2024-05-09T03:21:03.805546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:53.391374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805919+0000","last_change":"2024-05-09T03:21:02.600239+0000","last_active":"2024-05-09T03:21:03.805919+0000","last_peered":"2024-05-09T03:21:03.805919+0000","last_clean":"2024-05-09T03:21:03.805919+0000","last_became_active":"2024-05-09T03:21:02.599863+0000","last_became_peered":"2024-05-09T03:21:02.599863+0000","last_unstale":"2024-05-09T03:21:03.805919+0000","last_undegraded":"2024-05-09T03:21:03.805919+0000","last_fullsized":"2024-05-09T03:21:03.805919+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:53.391374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806854+0000","last_change":"2024-05-09T03:21:02.590233+0000","last_active":"2024-05-09T03:21:03.806854+0000","last_peered":"2024-05-09T03:21:03.806854+0000","last_clean":"2024-05-09T03:21:03.806854+0000","last_became_active":"2024-05-09T03:21:02.589923+0000","last_became_peered":"2024-05-09T03:21:02.589923+0000","last_unstale":"2024-05-09T03:21:03.806854+0000","last_undegraded":"2024-05-09T03:21:03.806854+0000","last_fullsized":"2024-05-09T03:21:03.806854+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:12.303990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806136+0000","last_change":"2024-05-09T03:21:02.596891+0000","last_active":"2024-05-09T03:21:03.806136+0000","last_peered":"2024-05-09T03:21:03.806136+0000","last_clean":"2024-05-09T03:21:03.806136+0000","last_became_active":"2024-05-09T03:21:02.596553+0000","last_became_peered":"2024-05-09T03:21:02.596553+0000","last_unstale":"2024-05-09T03:21:03.806136+0000","last_undegraded":"2024-05-09T03:21:03.806136+0000","last_fullsized":"2024-05-09T03:21:03.806136+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:55:39.851421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806397+0000","last_change":"2024-05-09T03:21:02.592836+0000","last_active":"2024-05-09T03:21:03.806397+0000","last_peered":"2024-05-09T03:21:03.806397+0000","last_clean":"2024-05-09T03:21:03.806397+0000","last_became_active":"2024-05-09T03:21:02.592076+0000","last_became_peered":"2024-05-09T03:21:02.592076+0000","last_unstale":"2024-05-09T03:21:03.806397+0000","last_undegraded":"2024-05-09T03:21:03.806397+0000","last_fullsized":"2024-05-09T03:21:03.806397+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:15.448383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591510+0000","last_change":"2024-05-09T03:21:02.592614+0000","last_active":"2024-05-09T03:21:03.591510+0000","last_peered":"2024-05-09T03:21:03.591510+0000","last_clean":"2024-05-09T03:21:03.591510+0000","last_became_active":"2024-05-09T03:21:02.592279+0000","last_became_peered":"2024-05-09T03:21:02.592279+0000","last_unstale":"2024-05-09T03:21:03.591510+0000","last_undegraded":"2024-05-09T03:21:03.591510+0000","last_fullsized":"2024-05-09T03:21:03.591510+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:35.151440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805909+0000","last_change":"2024-05-09T03:21:02.592980+0000","last_active":"2024-05-09T03:21:03.805909+0000","last_peered":"2024-05-09T03:21:03.805909+0000","last_clean":"2024-05-09T03:21:03.805909+0000","last_became_active":"2024-05-09T03:21:02.591237+0000","last_became_peered":"2024-05-09T03:21:02.591237+0000","last_unstale":"2024-05-09T03:21:03.805909+0000","last_undegraded":"2024-05-09T03:21:03.805909+0000","last_fullsized":"2024-05-09T03:21:03.805909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:31:32.262639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806199+0000","last_change":"2024-05-09T03:21:02.600354+0000","last_active":"2024-05-09T03:21:03.806199+0000","last_peered":"2024-05-09T03:21:03.806199+0000","last_clean":"2024-05-09T03:21:03.806199+0000","last_became_active":"2024-05-09T03:21:02.600051+0000","last_became_peered":"2024-05-09T03:21:02.600051+0000","last_unstale":"2024-05-09T03:21:03.806199+0000","last_undegraded":"2024-05-09T03:21:03.806199+0000","last_fullsized":"2024-05-09T03:21:03.806199+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:01.413424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805580+0000","last_change":"2024-05-09T03:21:02.601341+0000","last_active":"2024-05-09T03:21:03.805580+0000","last_peered":"2024-05-09T03:21:03.805580+0000","last_clean":"2024-05-09T03:21:03.805580+0000","last_became_active":"2024-05-09T03:21:02.601038+0000","last_became_peered":"2024-05-09T03:21:02.601038+0000","last_unstale":"2024-05-09T03:21:03.805580+0000","last_undegraded":"2024-05-09T03:21:03.805580+0000","last_fullsized":"2024-05-09T03:21:03.805580+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:09:34.443805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806146+0000","last_change":"2024-05-09T03:21:02.590037+0000","last_active":"2024-05-09T03:21:03.806146+0000","last_peered":"2024-05-09T03:21:03.806146+0000","last_clean":"2024-05-09T03:21:03.806146+0000","last_became_active":"2024-05-09T03:21:02.589041+0000","last_became_peered":"2024-05-09T03:21:02.589041+0000","last_unstale":"2024-05-09T03:21:03.806146+0000","last_undegraded":"2024-05-09T03:21:03.806146+0000","last_fullsized":"2024-05-09T03:21:03.806146+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:55:39.851421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590819+0000","last_change":"2024-05-09T03:21:02.596257+0000","last_active":"2024-05-09T03:21:03.590819+0000","last_peered":"2024-05-09T03:21:03.590819+0000","last_clean":"2024-05-09T03:21:03.590819+0000","last_became_active":"2024-05-09T03:21:02.595959+0000","last_became_peered":"2024-05-09T03:21:02.595959+0000","last_unstale":"2024-05-09T03:21:03.590819+0000","last_undegraded":"2024-05-09T03:21:03.590819+0000","last_fullsized":"2024-05-09T03:21:03.590819+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:55:39.851421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807145+0000","last_change":"2024-05-09T03:21:02.607373+0000","last_active":"2024-05-09T03:21:03.807145+0000","last_peered":"2024-05-09T03:21:03.807145+0000","last_clean":"2024-05-09T03:21:03.807145+0000","last_became_active":"2024-05-09T03:21:02.607032+0000","last_became_peered":"2024-05-09T03:21:02.607032+0000","last_unstale":"2024-05-09T03:21:03.807145+0000","last_undegraded":"2024-05-09T03:21:03.807145+0000","last_fullsized":"2024-05-09T03:21:03.807145+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:00:04.101720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806613+0000","last_change":"2024-05-09T03:21:02.594783+0000","last_active":"2024-05-09T03:21:03.806613+0000","last_peered":"2024-05-09T03:21:03.806613+0000","last_clean":"2024-05-09T03:21:03.806613+0000","last_became_active":"2024-05-09T03:21:02.594119+0000","last_became_peered":"2024-05-09T03:21:02.594119+0000","last_unstale":"2024-05-09T03:21:03.806613+0000","last_undegraded":"2024-05-09T03:21:03.806613+0000","last_fullsized":"2024-05-09T03:21:03.806613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:09:34.443805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590571+0000","last_change":"2024-05-09T03:21:02.597865+0000","last_active":"2024-05-09T03:21:03.590571+0000","last_peered":"2024-05-09T03:21:03.590571+0000","last_clean":"2024-05-09T03:21:03.590571+0000","last_became_active":"2024-05-09T03:21:02.597427+0000","last_became_peered":"2024-05-09T03:21:02.597427+0000","last_unstale":"2024-05-09T03:21:03.590571+0000","last_undegraded":"2024-05-09T03:21:03.590571+0000","last_fullsized":"2024-05-09T03:21:03.590571+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:40.499483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806755+0000","last_change":"2024-05-09T03:21:02.598390+0000","last_active":"2024-05-09T03:21:03.806755+0000","last_peered":"2024-05-09T03:21:03.806755+0000","last_clean":"2024-05-09T03:21:03.806755+0000","last_became_active":"2024-05-09T03:21:02.598081+0000","last_became_peered":"2024-05-09T03:21:02.598081+0000","last_unstale":"2024-05-09T03:21:03.806755+0000","last_undegraded":"2024-05-09T03:21:03.806755+0000","last_fullsized":"2024-05-09T03:21:03.806755+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:52:32.151692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806662+0000","last_change":"2024-05-09T03:21:02.594998+0000","last_active":"2024-05-09T03:21:03.806662+0000","last_peered":"2024-05-09T03:21:03.806662+0000","last_clean":"2024-05-09T03:21:03.806662+0000","last_became_active":"2024-05-09T03:21:02.593677+0000","last_became_peered":"2024-05-09T03:21:02.593677+0000","last_unstale":"2024-05-09T03:21:03.806662+0000","last_undegraded":"2024-05-09T03:21:03.806662+0000","last_fullsized":"2024-05-09T03:21:03.806662+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:40:57.429739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805817+0000","last_change":"2024-05-09T03:21:02.592161+0000","last_active":"2024-05-09T03:21:03.805817+0000","last_peered":"2024-05-09T03:21:03.805817+0000","last_clean":"2024-05-09T03:21:03.805817+0000","last_became_active":"2024-05-09T03:21:02.591810+0000","last_became_peered":"2024-05-09T03:21:02.591810+0000","last_unstale":"2024-05-09T03:21:03.805817+0000","last_undegraded":"2024-05-09T03:21:03.805817+0000","last_fullsized":"2024-05-09T03:21:03.805817+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:16.316105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589880+0000","last_change":"2024-05-09T03:21:02.608498+0000","last_active":"2024-05-09T03:21:03.589880+0000","last_peered":"2024-05-09T03:21:03.589880+0000","last_clean":"2024-05-09T03:21:03.589880+0000","last_became_active":"2024-05-09T03:21:02.607950+0000","last_became_peered":"2024-05-09T03:21:02.607950+0000","last_unstale":"2024-05-09T03:21:03.589880+0000","last_undegraded":"2024-05-09T03:21:03.589880+0000","last_fullsized":"2024-05-09T03:21:03.589880+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:13.370650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806138+0000","last_change":"2024-05-09T03:21:02.593662+0000","last_active":"2024-05-09T03:21:03.806138+0000","last_peered":"2024-05-09T03:21:03.806138+0000","last_clean":"2024-05-09T03:21:03.806138+0000","last_became_active":"2024-05-09T03:21:02.591415+0000","last_became_peered":"2024-05-09T03:21:02.591415+0000","last_unstale":"2024-05-09T03:21:03.806138+0000","last_undegraded":"2024-05-09T03:21:03.806138+0000","last_fullsized":"2024-05-09T03:21:03.806138+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:04.677834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"134'4","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806963+0000","last_change":"2024-05-09T03:21:01.588341+0000","last_active":"2024-05-09T03:21:03.806963+0000","last_peered":"2024-05-09T03:21:03.806963+0000","last_clean":"2024-05-09T03:21:03.806963+0000","last_became_active":"2024-05-09T03:21:01.588033+0000","last_became_peered":"2024-05-09T03:21:01.588033+0000","last_unstale":"2024-05-09T03:21:03.806963+0000","last_undegraded":"2024-05-09T03:21:03.806963+0000","last_fullsized":"2024-05-09T03:21:03.806963+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:34.148962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806967+0000","last_change":"2024-05-09T03:21:02.589024+0000","last_active":"2024-05-09T03:21:03.806967+0000","last_peered":"2024-05-09T03:21:03.806967+0000","last_clean":"2024-05-09T03:21:03.806967+0000","last_became_active":"2024-05-09T03:21:02.588711+0000","last_became_peered":"2024-05-09T03:21:02.588711+0000","last_unstale":"2024-05-09T03:21:03.806967+0000","last_undegraded":"2024-05-09T03:21:03.806967+0000","last_fullsized":"2024-05-09T03:21:03.806967+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:01.413424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806609+0000","last_change":"2024-05-09T03:21:01.590137+0000","last_active":"2024-05-09T03:21:03.806609+0000","last_peered":"2024-05-09T03:21:03.806609+0000","last_clean":"2024-05-09T03:21:03.806609+0000","last_became_active":"2024-05-09T03:21:01.589800+0000","last_became_peered":"2024-05-09T03:21:01.589800+0000","last_unstale":"2024-05-09T03:21:03.806609+0000","last_undegraded":"2024-05-09T03:21:03.806609+0000","last_fullsized":"2024-05-09T03:21:03.806609+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:05.717846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589269+0000","last_change":"2024-05-09T03:21:02.600394+0000","last_active":"2024-05-09T03:21:03.589269+0000","last_peered":"2024-05-09T03:21:03.589269+0000","last_clean":"2024-05-09T03:21:03.589269+0000","last_became_active":"2024-05-09T03:21:02.600093+0000","last_became_peered":"2024-05-09T03:21:02.600093+0000","last_unstale":"2024-05-09T03:21:03.589269+0000","last_undegraded":"2024-05-09T03:21:03.589269+0000","last_fullsized":"2024-05-09T03:21:03.589269+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:38.465838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806544+0000","last_change":"2024-05-09T03:21:01.593090+0000","last_active":"2024-05-09T03:21:03.806544+0000","last_peered":"2024-05-09T03:21:03.806544+0000","last_clean":"2024-05-09T03:21:03.806544+0000","last_became_active":"2024-05-09T03:21:01.592330+0000","last_became_peered":"2024-05-09T03:21:01.592330+0000","last_unstale":"2024-05-09T03:21:03.806544+0000","last_undegraded":"2024-05-09T03:21:03.806544+0000","last_fullsized":"2024-05-09T03:21:03.806544+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:05.717846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591740+0000","last_change":"2024-05-09T03:21:02.590740+0000","last_active":"2024-05-09T03:21:03.591740+0000","last_peered":"2024-05-09T03:21:03.591740+0000","last_clean":"2024-05-09T03:21:03.591740+0000","last_became_active":"2024-05-09T03:21:02.590447+0000","last_became_peered":"2024-05-09T03:21:02.590447+0000","last_unstale":"2024-05-09T03:21:03.591740+0000","last_undegraded":"2024-05-09T03:21:03.591740+0000","last_fullsized":"2024-05-09T03:21:03.591740+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:15.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806978+0000","last_change":"2024-05-09T03:21:01.591363+0000","last_active":"2024-05-09T03:21:03.806978+0000","last_peered":"2024-05-09T03:21:03.806978+0000","last_clean":"2024-05-09T03:21:03.806978+0000","last_became_active":"2024-05-09T03:21:01.591034+0000","last_became_peered":"2024-05-09T03:21:01.591034+0000","last_unstale":"2024-05-09T03:21:03.806978+0000","last_undegraded":"2024-05-09T03:21:03.806978+0000","last_fullsized":"2024-05-09T03:21:03.806978+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:16.915428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806010+0000","last_change":"2024-05-09T03:21:02.599419+0000","last_active":"2024-05-09T03:21:03.806010+0000","last_peered":"2024-05-09T03:21:03.806010+0000","last_clean":"2024-05-09T03:21:03.806010+0000","last_became_active":"2024-05-09T03:21:02.599059+0000","last_became_peered":"2024-05-09T03:21:02.599059+0000","last_unstale":"2024-05-09T03:21:03.806010+0000","last_undegraded":"2024-05-09T03:21:03.806010+0000","last_fullsized":"2024-05-09T03:21:03.806010+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:57:42.234037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589739+0000","last_change":"2024-05-09T03:21:01.578075+0000","last_active":"2024-05-09T03:21:03.589739+0000","last_peered":"2024-05-09T03:21:03.589739+0000","last_clean":"2024-05-09T03:21:03.589739+0000","last_became_active":"2024-05-09T03:21:01.577735+0000","last_became_peered":"2024-05-09T03:21:01.577735+0000","last_unstale":"2024-05-09T03:21:03.589739+0000","last_undegraded":"2024-05-09T03:21:03.589739+0000","last_fullsized":"2024-05-09T03:21:03.589739+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:38:04.349592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806347+0000","last_change":"2024-05-09T03:21:02.588348+0000","last_active":"2024-05-09T03:21:03.806347+0000","last_peered":"2024-05-09T03:21:03.806347+0000","last_clean":"2024-05-09T03:21:03.806347+0000","last_became_active":"2024-05-09T03:21:02.588016+0000","last_became_peered":"2024-05-09T03:21:02.588016+0000","last_unstale":"2024-05-09T03:21:03.806347+0000","last_undegraded":"2024-05-09T03:21:03.806347+0000","last_fullsized":"2024-05-09T03:21:03.806347+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:34.875542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806305+0000","last_change":"2024-05-09T03:21:01.591082+0000","last_active":"2024-05-09T03:21:03.806305+0000","last_peered":"2024-05-09T03:21:03.806305+0000","last_clean":"2024-05-09T03:21:03.806305+0000","last_became_active":"2024-05-09T03:21:01.590693+0000","last_became_peered":"2024-05-09T03:21:01.590693+0000","last_unstale":"2024-05-09T03:21:03.806305+0000","last_undegraded":"2024-05-09T03:21:03.806305+0000","last_fullsized":"2024-05-09T03:21:03.806305+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:14.482325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806266+0000","last_change":"2024-05-09T03:21:02.591399+0000","last_active":"2024-05-09T03:21:03.806266+0000","last_peered":"2024-05-09T03:21:03.806266+0000","last_clean":"2024-05-09T03:21:03.806266+0000","last_became_active":"2024-05-09T03:21:02.589062+0000","last_became_peered":"2024-05-09T03:21:02.589062+0000","last_unstale":"2024-05-09T03:21:03.806266+0000","last_undegraded":"2024-05-09T03:21:03.806266+0000","last_fullsized":"2024-05-09T03:21:03.806266+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:28:58.074717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590365+0000","last_change":"2024-05-09T03:21:01.584386+0000","last_active":"2024-05-09T03:21:03.590365+0000","last_peered":"2024-05-09T03:21:03.590365+0000","last_clean":"2024-05-09T03:21:03.590365+0000","last_became_active":"2024-05-09T03:21:01.582362+0000","last_became_peered":"2024-05-09T03:21:01.582362+0000","last_unstale":"2024-05-09T03:21:03.590365+0000","last_undegraded":"2024-05-09T03:21:03.590365+0000","last_fullsized":"2024-05-09T03:21:03.590365+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:27.616684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590277+0000","last_change":"2024-05-09T03:21:02.590877+0000","last_active":"2024-05-09T03:21:03.590277+0000","last_peered":"2024-05-09T03:21:03.590277+0000","last_clean":"2024-05-09T03:21:03.590277+0000","last_became_active":"2024-05-09T03:21:02.590524+0000","last_became_peered":"2024-05-09T03:21:02.590524+0000","last_unstale":"2024-05-09T03:21:03.590277+0000","last_undegraded":"2024-05-09T03:21:03.590277+0000","last_fullsized":"2024-05-09T03:21:03.590277+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:28.667421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589186+0000","last_change":"2024-05-09T03:21:01.583055+0000","last_active":"2024-05-09T03:21:03.589186+0000","last_peered":"2024-05-09T03:21:03.589186+0000","last_clean":"2024-05-09T03:21:03.589186+0000","last_became_active":"2024-05-09T03:21:01.582183+0000","last_became_peered":"2024-05-09T03:21:01.582183+0000","last_unstale":"2024-05-09T03:21:03.589186+0000","last_undegraded":"2024-05-09T03:21:03.589186+0000","last_fullsized":"2024-05-09T03:21:03.589186+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:06.151109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806860+0000","last_change":"2024-05-09T03:21:02.596214+0000","last_active":"2024-05-09T03:21:03.806860+0000","last_peered":"2024-05-09T03:21:03.806860+0000","last_clean":"2024-05-09T03:21:03.806860+0000","last_became_active":"2024-05-09T03:21:02.595892+0000","last_became_peered":"2024-05-09T03:21:02.595892+0000","last_unstale":"2024-05-09T03:21:03.806860+0000","last_undegraded":"2024-05-09T03:21:03.806860+0000","last_fullsized":"2024-05-09T03:21:03.806860+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:13.221763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589653+0000","last_change":"2024-05-09T03:21:01.579786+0000","last_active":"2024-05-09T03:21:03.589653+0000","last_peered":"2024-05-09T03:21:03.589653+0000","last_clean":"2024-05-09T03:21:03.589653+0000","last_became_active":"2024-05-09T03:21:01.579479+0000","last_became_peered":"2024-05-09T03:21:01.579479+0000","last_unstale":"2024-05-09T03:21:03.589653+0000","last_undegraded":"2024-05-09T03:21:03.589653+0000","last_fullsized":"2024-05-09T03:21:03.589653+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:52.174643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591758+0000","last_change":"2024-05-09T03:21:02.595986+0000","last_active":"2024-05-09T03:21:03.591758+0000","last_peered":"2024-05-09T03:21:03.591758+0000","last_clean":"2024-05-09T03:21:03.591758+0000","last_became_active":"2024-05-09T03:21:02.595467+0000","last_became_peered":"2024-05-09T03:21:02.595467+0000","last_unstale":"2024-05-09T03:21:03.591758+0000","last_undegraded":"2024-05-09T03:21:03.591758+0000","last_fullsized":"2024-05-09T03:21:03.591758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:21.156037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805995+0000","last_change":"2024-05-09T03:21:01.591561+0000","last_active":"2024-05-09T03:21:03.805995+0000","last_peered":"2024-05-09T03:21:03.805995+0000","last_clean":"2024-05-09T03:21:03.805995+0000","last_became_active":"2024-05-09T03:21:01.591162+0000","last_became_peered":"2024-05-09T03:21:01.591162+0000","last_unstale":"2024-05-09T03:21:03.805995+0000","last_undegraded":"2024-05-09T03:21:03.805995+0000","last_fullsized":"2024-05-09T03:21:03.805995+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:14.482325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590660+0000","last_change":"2024-05-09T03:21:02.597970+0000","last_active":"2024-05-09T03:21:03.590660+0000","last_peered":"2024-05-09T03:21:03.590660+0000","last_clean":"2024-05-09T03:21:03.590660+0000","last_became_active":"2024-05-09T03:21:02.597265+0000","last_became_peered":"2024-05-09T03:21:02.597265+0000","last_unstale":"2024-05-09T03:21:03.590660+0000","last_undegraded":"2024-05-09T03:21:03.590660+0000","last_fullsized":"2024-05-09T03:21:03.590660+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:16.316105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"134'3","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806759+0000","last_change":"2024-05-09T03:21:01.585751+0000","last_active":"2024-05-09T03:21:03.806759+0000","last_peered":"2024-05-09T03:21:03.806759+0000","last_clean":"2024-05-09T03:21:03.806759+0000","last_became_active":"2024-05-09T03:21:01.585433+0000","last_became_peered":"2024-05-09T03:21:01.585433+0000","last_unstale":"2024-05-09T03:21:03.806759+0000","last_undegraded":"2024-05-09T03:21:03.806759+0000","last_fullsized":"2024-05-09T03:21:03.806759+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:36:14.219803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806604+0000","last_change":"2024-05-09T03:21:02.600127+0000","last_active":"2024-05-09T03:21:03.806604+0000","last_peered":"2024-05-09T03:21:03.806604+0000","last_clean":"2024-05-09T03:21:03.806604+0000","last_became_active":"2024-05-09T03:21:02.599789+0000","last_became_peered":"2024-05-09T03:21:02.599789+0000","last_unstale":"2024-05-09T03:21:03.806604+0000","last_undegraded":"2024-05-09T03:21:03.806604+0000","last_fullsized":"2024-05-09T03:21:03.806604+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:01.413424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591622+0000","last_change":"2024-05-09T03:21:01.584309+0000","last_active":"2024-05-09T03:21:03.591622+0000","last_peered":"2024-05-09T03:21:03.591622+0000","last_clean":"2024-05-09T03:21:03.591622+0000","last_became_active":"2024-05-09T03:21:01.583997+0000","last_became_peered":"2024-05-09T03:21:01.583997+0000","last_unstale":"2024-05-09T03:21:03.591622+0000","last_undegraded":"2024-05-09T03:21:03.591622+0000","last_fullsized":"2024-05-09T03:21:03.591622+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:08.163668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806550+0000","last_change":"2024-05-09T03:21:02.585939+0000","last_active":"2024-05-09T03:21:03.806550+0000","last_peered":"2024-05-09T03:21:03.806550+0000","last_clean":"2024-05-09T03:21:03.806550+0000","last_became_active":"2024-05-09T03:21:02.585633+0000","last_became_peered":"2024-05-09T03:21:02.585633+0000","last_unstale":"2024-05-09T03:21:03.806550+0000","last_undegraded":"2024-05-09T03:21:03.806550+0000","last_fullsized":"2024-05-09T03:21:03.806550+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:09:34.443805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806180+0000","last_change":"2024-05-09T03:21:01.587181+0000","last_active":"2024-05-09T03:21:03.806180+0000","last_peered":"2024-05-09T03:21:03.806180+0000","last_clean":"2024-05-09T03:21:03.806180+0000","last_became_active":"2024-05-09T03:21:01.586851+0000","last_became_peered":"2024-05-09T03:21:01.586851+0000","last_unstale":"2024-05-09T03:21:03.806180+0000","last_undegraded":"2024-05-09T03:21:03.806180+0000","last_fullsized":"2024-05-09T03:21:03.806180+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:03.154206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589614+0000","last_change":"2024-05-09T03:21:02.601331+0000","last_active":"2024-05-09T03:21:03.589614+0000","last_peered":"2024-05-09T03:21:03.589614+0000","last_clean":"2024-05-09T03:21:03.589614+0000","last_became_active":"2024-05-09T03:21:02.600974+0000","last_became_peered":"2024-05-09T03:21:02.600974+0000","last_unstale":"2024-05-09T03:21:03.589614+0000","last_undegraded":"2024-05-09T03:21:03.589614+0000","last_fullsized":"2024-05-09T03:21:03.589614+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:09.166146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806865+0000","last_change":"2024-05-09T03:21:01.588880+0000","last_active":"2024-05-09T03:21:03.806865+0000","last_peered":"2024-05-09T03:21:03.806865+0000","last_clean":"2024-05-09T03:21:03.806865+0000","last_became_active":"2024-05-09T03:21:01.588425+0000","last_became_peered":"2024-05-09T03:21:01.588425+0000","last_unstale":"2024-05-09T03:21:03.806865+0000","last_undegraded":"2024-05-09T03:21:03.806865+0000","last_fullsized":"2024-05-09T03:21:03.806865+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:31.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591014+0000","last_change":"2024-05-09T03:21:02.587527+0000","last_active":"2024-05-09T03:21:03.591014+0000","last_peered":"2024-05-09T03:21:03.591014+0000","last_clean":"2024-05-09T03:21:03.591014+0000","last_became_active":"2024-05-09T03:21:02.587183+0000","last_became_peered":"2024-05-09T03:21:02.587183+0000","last_unstale":"2024-05-09T03:21:03.591014+0000","last_undegraded":"2024-05-09T03:21:03.591014+0000","last_fullsized":"2024-05-09T03:21:03.591014+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:06.720324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590413+0000","last_change":"2024-05-09T03:21:01.577198+0000","last_active":"2024-05-09T03:21:03.590413+0000","last_peered":"2024-05-09T03:21:03.590413+0000","last_clean":"2024-05-09T03:21:03.590413+0000","last_became_active":"2024-05-09T03:21:01.576834+0000","last_became_peered":"2024-05-09T03:21:01.576834+0000","last_unstale":"2024-05-09T03:21:03.590413+0000","last_undegraded":"2024-05-09T03:21:03.590413+0000","last_fullsized":"2024-05-09T03:21:03.590413+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:03:58.267719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805600+0000","last_change":"2024-05-09T03:21:02.591948+0000","last_active":"2024-05-09T03:21:03.805600+0000","last_peered":"2024-05-09T03:21:03.805600+0000","last_clean":"2024-05-09T03:21:03.805600+0000","last_became_active":"2024-05-09T03:21:02.591655+0000","last_became_peered":"2024-05-09T03:21:02.591655+0000","last_unstale":"2024-05-09T03:21:03.805600+0000","last_undegraded":"2024-05-09T03:21:03.805600+0000","last_fullsized":"2024-05-09T03:21:03.805600+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:09.694627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"134'8","reported_seq":22,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589328+0000","last_change":"2024-05-09T03:21:01.588952+0000","last_active":"2024-05-09T03:21:03.589328+0000","last_peered":"2024-05-09T03:21:03.589328+0000","last_clean":"2024-05-09T03:21:03.589328+0000","last_became_active":"2024-05-09T03:21:01.588569+0000","last_became_peered":"2024-05-09T03:21:01.588569+0000","last_unstale":"2024-05-09T03:21:03.589328+0000","last_undegraded":"2024-05-09T03:21:03.589328+0000","last_fullsized":"2024-05-09T03:21:03.589328+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:34.502908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590930+0000","last_change":"2024-05-09T03:21:02.598414+0000","last_active":"2024-05-09T03:21:03.590930+0000","last_peered":"2024-05-09T03:21:03.590930+0000","last_clean":"2024-05-09T03:21:03.590930+0000","last_became_active":"2024-05-09T03:21:02.598068+0000","last_became_peered":"2024-05-09T03:21:02.598068+0000","last_unstale":"2024-05-09T03:21:03.590930+0000","last_undegraded":"2024-05-09T03:21:03.590930+0000","last_fullsized":"2024-05-09T03:21:03.590930+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:34.875542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591747+0000","last_change":"2024-05-09T03:21:01.584627+0000","last_active":"2024-05-09T03:21:03.591747+0000","last_peered":"2024-05-09T03:21:03.591747+0000","last_clean":"2024-05-09T03:21:03.591747+0000","last_became_active":"2024-05-09T03:21:01.584293+0000","last_became_peered":"2024-05-09T03:21:01.584293+0000","last_unstale":"2024-05-09T03:21:03.591747+0000","last_undegraded":"2024-05-09T03:21:03.591747+0000","last_fullsized":"2024-05-09T03:21:03.591747+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:58:27.097696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806205+0000","last_change":"2024-05-09T03:21:02.593056+0000","last_active":"2024-05-09T03:21:03.806205+0000","last_peered":"2024-05-09T03:21:03.806205+0000","last_clean":"2024-05-09T03:21:03.806205+0000","last_became_active":"2024-05-09T03:21:02.592710+0000","last_became_peered":"2024-05-09T03:21:02.592710+0000","last_unstale":"2024-05-09T03:21:03.806205+0000","last_undegraded":"2024-05-09T03:21:03.806205+0000","last_fullsized":"2024-05-09T03:21:03.806205+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:05:48.022379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590756+0000","last_change":"2024-05-09T03:21:01.580717+0000","last_active":"2024-05-09T03:21:03.590756+0000","last_peered":"2024-05-09T03:21:03.590756+0000","last_clean":"2024-05-09T03:21:03.590756+0000","last_became_active":"2024-05-09T03:21:01.580391+0000","last_became_peered":"2024-05-09T03:21:01.580391+0000","last_unstale":"2024-05-09T03:21:03.590756+0000","last_undegraded":"2024-05-09T03:21:03.590756+0000","last_fullsized":"2024-05-09T03:21:03.590756+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:16.915428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806778+0000","last_change":"2024-05-09T03:21:02.592575+0000","last_active":"2024-05-09T03:21:03.806778+0000","last_peered":"2024-05-09T03:21:03.806778+0000","last_clean":"2024-05-09T03:21:03.806778+0000","last_became_active":"2024-05-09T03:21:02.592074+0000","last_became_peered":"2024-05-09T03:21:02.592074+0000","last_unstale":"2024-05-09T03:21:03.806778+0000","last_undegraded":"2024-05-09T03:21:03.806778+0000","last_fullsized":"2024-05-09T03:21:03.806778+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:05.678532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807093+0000","last_change":"2024-05-09T03:21:01.591388+0000","last_active":"2024-05-09T03:21:03.807093+0000","last_peered":"2024-05-09T03:21:03.807093+0000","last_clean":"2024-05-09T03:21:03.807093+0000","last_became_active":"2024-05-09T03:21:01.591067+0000","last_became_peered":"2024-05-09T03:21:01.591067+0000","last_unstale":"2024-05-09T03:21:03.807093+0000","last_undegraded":"2024-05-09T03:21:03.807093+0000","last_fullsized":"2024-05-09T03:21:03.807093+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:27.664943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806762+0000","last_change":"2024-05-09T03:21:02.594559+0000","last_active":"2024-05-09T03:21:03.806762+0000","last_peered":"2024-05-09T03:21:03.806762+0000","last_clean":"2024-05-09T03:21:03.806762+0000","last_became_active":"2024-05-09T03:21:02.593345+0000","last_became_peered":"2024-05-09T03:21:02.593345+0000","last_unstale":"2024-05-09T03:21:03.806762+0000","last_undegraded":"2024-05-09T03:21:03.806762+0000","last_fullsized":"2024-05-09T03:21:03.806762+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:13.183796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806468+0000","last_change":"2024-05-09T03:21:01.592276+0000","last_active":"2024-05-09T03:21:03.806468+0000","last_peered":"2024-05-09T03:21:03.806468+0000","last_clean":"2024-05-09T03:21:03.806468+0000","last_became_active":"2024-05-09T03:21:01.591952+0000","last_became_peered":"2024-05-09T03:21:01.591952+0000","last_unstale":"2024-05-09T03:21:03.806468+0000","last_undegraded":"2024-05-09T03:21:03.806468+0000","last_fullsized":"2024-05-09T03:21:03.806468+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:30:18.272104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806069+0000","last_change":"2024-05-09T03:21:02.601889+0000","last_active":"2024-05-09T03:21:03.806069+0000","last_peered":"2024-05-09T03:21:03.806069+0000","last_clean":"2024-05-09T03:21:03.806069+0000","last_became_active":"2024-05-09T03:21:02.601591+0000","last_became_peered":"2024-05-09T03:21:02.601591+0000","last_unstale":"2024-05-09T03:21:03.806069+0000","last_undegraded":"2024-05-09T03:21:03.806069+0000","last_fullsized":"2024-05-09T03:21:03.806069+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:46.832807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806125+0000","last_change":"2024-05-09T03:21:01.585788+0000","last_active":"2024-05-09T03:21:03.806125+0000","last_peered":"2024-05-09T03:21:03.806125+0000","last_clean":"2024-05-09T03:21:03.806125+0000","last_became_active":"2024-05-09T03:21:01.585466+0000","last_became_peered":"2024-05-09T03:21:01.585466+0000","last_unstale":"2024-05-09T03:21:03.806125+0000","last_undegraded":"2024-05-09T03:21:03.806125+0000","last_fullsized":"2024-05-09T03:21:03.806125+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:42.528435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805718+0000","last_change":"2024-05-09T03:21:02.591591+0000","last_active":"2024-05-09T03:21:03.805718+0000","last_peered":"2024-05-09T03:21:03.805718+0000","last_clean":"2024-05-09T03:21:03.805718+0000","last_became_active":"2024-05-09T03:21:02.591158+0000","last_became_peered":"2024-05-09T03:21:02.591158+0000","last_unstale":"2024-05-09T03:21:03.805718+0000","last_undegraded":"2024-05-09T03:21:03.805718+0000","last_fullsized":"2024-05-09T03:21:03.805718+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:09.540737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805921+0000","last_change":"2024-05-09T03:21:01.584571+0000","last_active":"2024-05-09T03:21:03.805921+0000","last_peered":"2024-05-09T03:21:03.805921+0000","last_clean":"2024-05-09T03:21:03.805921+0000","last_became_active":"2024-05-09T03:21:01.584229+0000","last_became_peered":"2024-05-09T03:21:01.584229+0000","last_unstale":"2024-05-09T03:21:03.805921+0000","last_undegraded":"2024-05-09T03:21:03.805921+0000","last_fullsized":"2024-05-09T03:21:03.805921+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:40.349991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590062+0000","last_change":"2024-05-09T03:21:02.590312+0000","last_active":"2024-05-09T03:21:03.590062+0000","last_peered":"2024-05-09T03:21:03.590062+0000","last_clean":"2024-05-09T03:21:03.590062+0000","last_became_active":"2024-05-09T03:21:02.589802+0000","last_became_peered":"2024-05-09T03:21:02.589802+0000","last_unstale":"2024-05-09T03:21:03.590062+0000","last_undegraded":"2024-05-09T03:21:03.590062+0000","last_fullsized":"2024-05-09T03:21:03.590062+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:00.709346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805540+0000","last_change":"2024-05-09T03:21:01.593505+0000","last_active":"2024-05-09T03:21:03.805540+0000","last_peered":"2024-05-09T03:21:03.805540+0000","last_clean":"2024-05-09T03:21:03.805540+0000","last_became_active":"2024-05-09T03:21:01.592448+0000","last_became_peered":"2024-05-09T03:21:01.592448+0000","last_unstale":"2024-05-09T03:21:03.805540+0000","last_undegraded":"2024-05-09T03:21:03.805540+0000","last_fullsized":"2024-05-09T03:21:03.805540+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:25.763772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807032+0000","last_change":"2024-05-09T03:21:02.601645+0000","last_active":"2024-05-09T03:21:03.807032+0000","last_peered":"2024-05-09T03:21:03.807032+0000","last_clean":"2024-05-09T03:21:03.807032+0000","last_became_active":"2024-05-09T03:21:02.600936+0000","last_became_peered":"2024-05-09T03:21:02.600936+0000","last_unstale":"2024-05-09T03:21:03.807032+0000","last_undegraded":"2024-05-09T03:21:03.807032+0000","last_fullsized":"2024-05-09T03:21:03.807032+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:54:43.745630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806325+0000","last_change":"2024-05-09T03:21:01.581006+0000","last_active":"2024-05-09T03:21:03.806325+0000","last_peered":"2024-05-09T03:21:03.806325+0000","last_clean":"2024-05-09T03:21:03.806325+0000","last_became_active":"2024-05-09T03:21:01.580605+0000","last_became_peered":"2024-05-09T03:21:01.580605+0000","last_unstale":"2024-05-09T03:21:03.806325+0000","last_undegraded":"2024-05-09T03:21:03.806325+0000","last_fullsized":"2024-05-09T03:21:03.806325+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:28.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589666+0000","last_change":"2024-05-09T03:21:02.602065+0000","last_active":"2024-05-09T03:21:03.589666+0000","last_peered":"2024-05-09T03:21:03.589666+0000","last_clean":"2024-05-09T03:21:03.589666+0000","last_became_active":"2024-05-09T03:21:02.601754+0000","last_became_peered":"2024-05-09T03:21:02.601754+0000","last_unstale":"2024-05-09T03:21:03.589666+0000","last_undegraded":"2024-05-09T03:21:03.589666+0000","last_fullsized":"2024-05-09T03:21:03.589666+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:03:59.270197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"135'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807178+0000","last_change":"2024-05-09T03:21:01.592928+0000","last_active":"2024-05-09T03:21:03.807178+0000","last_peered":"2024-05-09T03:21:03.807178+0000","last_clean":"2024-05-09T03:21:03.807178+0000","last_became_active":"2024-05-09T03:21:01.592598+0000","last_became_peered":"2024-05-09T03:21:01.592598+0000","last_unstale":"2024-05-09T03:21:03.807178+0000","last_undegraded":"2024-05-09T03:21:03.807178+0000","last_fullsized":"2024-05-09T03:21:03.807178+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:20.153559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589284+0000","last_change":"2024-05-09T03:21:02.588027+0000","last_active":"2024-05-09T03:21:03.589284+0000","last_peered":"2024-05-09T03:21:03.589284+0000","last_clean":"2024-05-09T03:21:03.589284+0000","last_became_active":"2024-05-09T03:21:02.587681+0000","last_became_peered":"2024-05-09T03:21:02.587681+0000","last_unstale":"2024-05-09T03:21:03.589284+0000","last_undegraded":"2024-05-09T03:21:03.589284+0000","last_fullsized":"2024-05-09T03:21:03.589284+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:20.121767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806670+0000","last_change":"2024-05-09T03:21:01.591571+0000","last_active":"2024-05-09T03:21:03.806670+0000","last_peered":"2024-05-09T03:21:03.806670+0000","last_clean":"2024-05-09T03:21:03.806670+0000","last_became_active":"2024-05-09T03:21:01.590921+0000","last_became_peered":"2024-05-09T03:21:01.590921+0000","last_unstale":"2024-05-09T03:21:03.806670+0000","last_undegraded":"2024-05-09T03:21:03.806670+0000","last_fullsized":"2024-05-09T03:21:03.806670+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:36:14.219803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590593+0000","last_change":"2024-05-09T03:21:02.597393+0000","last_active":"2024-05-09T03:21:03.590593+0000","last_peered":"2024-05-09T03:21:03.590593+0000","last_clean":"2024-05-09T03:21:03.590593+0000","last_became_active":"2024-05-09T03:21:02.597001+0000","last_became_peered":"2024-05-09T03:21:02.597001+0000","last_unstale":"2024-05-09T03:21:03.590593+0000","last_undegraded":"2024-05-09T03:21:03.590593+0000","last_fullsized":"2024-05-09T03:21:03.590593+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:53.391374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590699+0000","last_change":"2024-05-09T03:21:01.584090+0000","last_active":"2024-05-09T03:21:03.590699+0000","last_peered":"2024-05-09T03:21:03.590699+0000","last_clean":"2024-05-09T03:21:03.590699+0000","last_became_active":"2024-05-09T03:21:01.583736+0000","last_became_peered":"2024-05-09T03:21:01.583736+0000","last_unstale":"2024-05-09T03:21:03.590699+0000","last_undegraded":"2024-05-09T03:21:03.590699+0000","last_fullsized":"2024-05-09T03:21:03.590699+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.775277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806833+0000","last_change":"2024-05-09T03:21:02.597293+0000","last_active":"2024-05-09T03:21:03.806833+0000","last_peered":"2024-05-09T03:21:03.806833+0000","last_clean":"2024-05-09T03:21:03.806833+0000","last_became_active":"2024-05-09T03:21:02.595518+0000","last_became_peered":"2024-05-09T03:21:02.595518+0000","last_unstale":"2024-05-09T03:21:03.806833+0000","last_undegraded":"2024-05-09T03:21:03.806833+0000","last_fullsized":"2024-05-09T03:21:03.806833+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:37.739225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806588+0000","last_change":"2024-05-09T03:21:01.581337+0000","last_active":"2024-05-09T03:21:03.806588+0000","last_peered":"2024-05-09T03:21:03.806588+0000","last_clean":"2024-05-09T03:21:03.806588+0000","last_became_active":"2024-05-09T03:21:01.581023+0000","last_became_peered":"2024-05-09T03:21:01.581023+0000","last_unstale":"2024-05-09T03:21:03.806588+0000","last_undegraded":"2024-05-09T03:21:03.806588+0000","last_fullsized":"2024-05-09T03:21:03.806588+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:08:59.706868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591492+0000","last_change":"2024-05-09T03:21:02.588790+0000","last_active":"2024-05-09T03:21:03.591492+0000","last_peered":"2024-05-09T03:21:03.591492+0000","last_clean":"2024-05-09T03:21:03.591492+0000","last_became_active":"2024-05-09T03:21:02.588445+0000","last_became_peered":"2024-05-09T03:21:02.588445+0000","last_unstale":"2024-05-09T03:21:03.591492+0000","last_undegraded":"2024-05-09T03:21:03.591492+0000","last_fullsized":"2024-05-09T03:21:03.591492+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:36:15.222281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590034+0000","last_change":"2024-05-09T03:21:01.581402+0000","last_active":"2024-05-09T03:21:03.590034+0000","last_peered":"2024-05-09T03:21:03.590034+0000","last_clean":"2024-05-09T03:21:03.590034+0000","last_became_active":"2024-05-09T03:21:01.580048+0000","last_became_peered":"2024-05-09T03:21:01.580048+0000","last_unstale":"2024-05-09T03:21:03.590034+0000","last_undegraded":"2024-05-09T03:21:03.590034+0000","last_fullsized":"2024-05-09T03:21:03.590034+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:59:03.052699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805484+0000","last_change":"2024-05-09T03:21:02.591456+0000","last_active":"2024-05-09T03:21:03.805484+0000","last_peered":"2024-05-09T03:21:03.805484+0000","last_clean":"2024-05-09T03:21:03.805484+0000","last_became_active":"2024-05-09T03:21:02.590786+0000","last_became_peered":"2024-05-09T03:21:02.590786+0000","last_unstale":"2024-05-09T03:21:03.805484+0000","last_undegraded":"2024-05-09T03:21:03.805484+0000","last_fullsized":"2024-05-09T03:21:03.805484+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:57:57.794112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805953+0000","last_change":"2024-05-09T03:21:01.587360+0000","last_active":"2024-05-09T03:21:03.805953+0000","last_peered":"2024-05-09T03:21:03.805953+0000","last_clean":"2024-05-09T03:21:03.805953+0000","last_became_active":"2024-05-09T03:21:01.587059+0000","last_became_peered":"2024-05-09T03:21:01.587059+0000","last_unstale":"2024-05-09T03:21:03.805953+0000","last_undegraded":"2024-05-09T03:21:03.805953+0000","last_fullsized":"2024-05-09T03:21:03.805953+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:34.148962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590256+0000","last_change":"2024-05-09T03:21:02.598051+0000","last_active":"2024-05-09T03:21:03.590256+0000","last_peered":"2024-05-09T03:21:03.590256+0000","last_clean":"2024-05-09T03:21:03.590256+0000","last_became_active":"2024-05-09T03:21:02.597704+0000","last_became_peered":"2024-05-09T03:21:02.597704+0000","last_unstale":"2024-05-09T03:21:03.590256+0000","last_undegraded":"2024-05-09T03:21:03.590256+0000","last_fullsized":"2024-05-09T03:21:03.590256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:31:01.243163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805606+0000","last_change":"2024-05-09T03:21:01.584237+0000","last_active":"2024-05-09T03:21:03.805606+0000","last_peered":"2024-05-09T03:21:03.805606+0000","last_clean":"2024-05-09T03:21:03.805606+0000","last_became_active":"2024-05-09T03:21:01.583951+0000","last_became_peered":"2024-05-09T03:21:01.583951+0000","last_unstale":"2024-05-09T03:21:03.805606+0000","last_undegraded":"2024-05-09T03:21:03.805606+0000","last_fullsized":"2024-05-09T03:21:03.805606+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:13:44.001499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806312+0000","last_change":"2024-05-09T03:21:02.592354+0000","last_active":"2024-05-09T03:21:03.806312+0000","last_peered":"2024-05-09T03:21:03.806312+0000","last_clean":"2024-05-09T03:21:03.806312+0000","last_became_active":"2024-05-09T03:21:02.591041+0000","last_became_peered":"2024-05-09T03:21:02.591041+0000","last_unstale":"2024-05-09T03:21:03.806312+0000","last_undegraded":"2024-05-09T03:21:03.806312+0000","last_fullsized":"2024-05-09T03:21:03.806312+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:00.583934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806586+0000","last_change":"2024-05-09T03:21:01.586379+0000","last_active":"2024-05-09T03:21:03.806586+0000","last_peered":"2024-05-09T03:21:03.806586+0000","last_clean":"2024-05-09T03:21:03.806586+0000","last_became_active":"2024-05-09T03:21:01.586052+0000","last_became_peered":"2024-05-09T03:21:01.586052+0000","last_unstale":"2024-05-09T03:21:03.806586+0000","last_undegraded":"2024-05-09T03:21:03.806586+0000","last_fullsized":"2024-05-09T03:21:03.806586+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:27.664943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":332,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806373+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:21:03.806373+0000","last_peered":"2024-05-09T03:21:03.806373+0000","last_clean":"2024-05-09T03:21:03.806373+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:03.806373+0000","last_undegraded":"2024-05-09T03:21:03.806373+0000","last_fullsized":"2024-05-09T03:21:03.806373+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37628,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334212,"statfs":{"total":96636764160,"available":96598233088,"internally_reserved":0,"allocated":2342912,"data_stored":1432301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1748992,"data_stored":842473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574993,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1744896,"data_stored":841891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574992,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1761280,"data_stored":843557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574991,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1757184,"data_stored":843455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574992,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41204,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330636,"statfs":{"total":96636764160,"available":96594571264,"internally_reserved":0,"allocated":1744896,"data_stored":841855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41208,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330632,"statfs":{"total":96636764160,"available":96594567168,"internally_reserved":0,"allocated":1740800,"data_stored":841821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574992,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29308,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342532,"statfs":{"total":96636764160,"available":96606752768,"internally_reserved":0,"allocated":2338816,"data_stored":1432771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:21:06.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:21:06.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.961+0000 7f90fabf3640 1 Processor -- start 2024-05-09T03:21:06.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.961+0000 7f90fabf3640 1 -- start start 2024-05-09T03:21:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90fabf3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f408cd60 0x7f90f40d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90fabf3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f40d1c80 0x7f90f40cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90fabf3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f40d4900 con 0x7f90f40d2090 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90fabf3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f90f40b5340 con 0x7f90f40d1c80 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90fabf3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f90f40b4fd0 con 0x7f90f408cd60 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90f3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f40d1c80 0x7f90f40cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90f3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f40d1c80 0x7f90f40cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45672/0 (socket says 172.21.15.46:45672) 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.962+0000 7f90f3fff640 1 -- 172.21.15.46:0/1680861022 learned_addr learned my addr 172.21.15.46:0/1680861022 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f898f640 1 --2- 172.21.15.46:0/1680861022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f408cd60 0x7f90f40d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f3fff640 1 -- 172.21.15.46:0/1680861022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f408cd60 msgr2=0x7f90f40d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f3fff640 1 --2- 172.21.15.46:0/1680861022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f408cd60 0x7f90f40d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f3fff640 1 -- 172.21.15.46:0/1680861022 >> v1:172.21.15.46:6789/0 conn(0x7f90f40d2090 legacy=0x7f90f4162310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f3fff640 1 -- 172.21.15.46:0/1680861022 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f40d2c00 con 0x7f90f40d1c80 2024-05-09T03:21:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f37fe640 1 -- 172.21.15.46:0/1680861022 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2625208482 0 0) 0x7f90f40d4900 con 0x7f90f40d2090 2024-05-09T03:21:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.963+0000 7f90f3fff640 1 --2- 172.21.15.46:0/1680861022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f40d1c80 0x7f90f40cf2b0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009870 tx=0x7f90e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68298ed1e01a71cb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90f37fe640 1 -- 172.21.15.46:0/1680861022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0037070 con 0x7f90f40d1c80 2024-05-09T03:21:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90f37fe640 1 -- 172.21.15.46:0/1680861022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e0032440 con 0x7f90f40d1c80 2024-05-09T03:21:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90f37fe640 1 -- 172.21.15.46:0/1680861022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e0032810 con 0x7f90f40d1c80 2024-05-09T03:21:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 -- 172.21.15.46:0/1680861022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f40d1c80 msgr2=0x7f90f40cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 --2- 172.21.15.46:0/1680861022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f40d1c80 0x7f90f40cf2b0 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009870 tx=0x7f90e002a270 comp rx=0 tx=0).stop 2024-05-09T03:21:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 -- 172.21.15.46:0/1680861022 shutdown_connections 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 --2- 172.21.15.46:0/1680861022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f40d1c80 0x7f90f40cf2b0 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 --2- 172.21.15.46:0/1680861022 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f408cd60 0x7f90f40d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 -- 172.21.15.46:0/1680861022 >> 172.21.15.46:0/1680861022 conn(0x7f90f40d3c00 msgr2=0x7f90f40d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 -- 172.21.15.46:0/1680861022 shutdown_connections 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.964+0000 7f90fabf3640 1 -- 172.21.15.46:0/1680861022 wait complete. 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.966+0000 7f90fabf3640 1 Processor -- start 2024-05-09T03:21:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.966+0000 7f90fabf3640 1 -- start start 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 0x7f90f415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f40d1c80 0x7f90f4153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40d2090 0x7f90f40cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90f40b92b0 con 0x7f90f408cd60 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f90f40b9130 con 0x7f90f40d2090 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f90f40d12d0 con 0x7f90f40d1c80 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 0x7f90f415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 0x7f90f415ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45688/0 (socket says 172.21.15.46:45688) 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 -- 172.21.15.46:0/2660219648 learned_addr learned my addr 172.21.15.46:0/2660219648 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f3fff640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f40d1c80 0x7f90f4153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f9190640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40d2090 0x7f90f40cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 -- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f40d1c80 msgr2=0x7f90f4153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f40d1c80 0x7f90f4153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 -- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40d2090 msgr2=0x7f90f40cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40d2090 0x7f90f40cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 -- 172.21.15.46:0/2660219648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f4157290 con 0x7f90f408cd60 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f9190640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40d2090 0x7f90f40cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f898f640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 0x7f90f415ab40 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f90e800d5d0 tx=0x7f90e800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8004240 con 0x7f90f408cd60 2024-05-09T03:21:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e80043e0 con 0x7f90f408cd60 2024-05-09T03:21:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f4157570 con 0x7f90f408cd60 2024-05-09T03:21:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.968+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8003830 con 0x7f90f408cd60 2024-05-09T03:21:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.969+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f4157b30 con 0x7f90f408cd60 2024-05-09T03:21:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.969+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f40b4d20 con 0x7f90f408cd60 2024-05-09T03:21:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.970+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f90e80039d0 con 0x7f90f408cd60 2024-05-09T03:21:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.971+0000 7f90f17fa640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90d8076300 0x7f90d8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.971+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f90e80977e0 con 0x7f90f408cd60 2024-05-09T03:21:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.971+0000 7f90f3fff640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90d8076300 0x7f90d8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.972+0000 7f90f3fff640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90d8076300 0x7f90d8078810 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f90e0034000 tx=0x7f90e002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:06.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:06.975+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f90f40b4d20 con 0x7f90f408cd60 2024-05-09T03:21:07.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.161+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90f406a270 con 0x7f90d8076300 2024-05-09T03:21:07.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.168+0000 7f90f17fa640 1 -- 172.21.15.46:0/2660219648 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252476 (secure 0 0 0) 0x7f90e007fa50 con 0x7f90d8076300 2024-05-09T03:21:07.172 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:07.173 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90d8076300 msgr2=0x7f90d8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90d8076300 0x7f90d8078810 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f90e0034000 tx=0x7f90e002e000 comp rx=0 tx=0).stop 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 msgr2=0x7f90f415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 0x7f90f415ab40 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f90e800d5d0 tx=0x7f90e800da90 comp rx=0 tx=0).stop 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 shutdown_connections 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40d2090 0x7f90f40cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90f40d1c80 0x7f90f4153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90d8076300 0x7f90d8078810 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 --2- 172.21.15.46:0/2660219648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f408cd60 0x7f90f415ab40 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 >> 172.21.15.46:0/2660219648 conn(0x7f90f40d3c00 msgr2=0x7f90f40b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 shutdown_connections 2024-05-09T03:21:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.171+0000 7f90fabf3640 1 -- 172.21.15.46:0/2660219648 wait complete. 2024-05-09T03:21:07.183 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-05-09T03:21:06.681083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":293412,"kb_used_data":14824,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754681308,"statfs":{"total":773094113280,"available":772793659392,"internally_reserved":0,"allocated":15179776,"data_stored":7920124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":285134385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001409"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589494+0000","last_change":"2024-05-09T03:21:02.596216+0000","last_active":"2024-05-09T03:21:03.589494+0000","last_peered":"2024-05-09T03:21:03.589494+0000","last_clean":"2024-05-09T03:21:03.589494+0000","last_became_active":"2024-05-09T03:21:02.595888+0000","last_became_peered":"2024-05-09T03:21:02.595888+0000","last_unstale":"2024-05-09T03:21:03.589494+0000","last_undegraded":"2024-05-09T03:21:03.589494+0000","last_fullsized":"2024-05-09T03:21:03.589494+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:58:28.100174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806522+0000","last_change":"2024-05-09T03:21:02.604820+0000","last_active":"2024-05-09T03:21:03.806522+0000","last_peered":"2024-05-09T03:21:03.806522+0000","last_clean":"2024-05-09T03:21:03.806522+0000","last_became_active":"2024-05-09T03:21:02.604509+0000","last_became_peered":"2024-05-09T03:21:02.604509+0000","last_unstale":"2024-05-09T03:21:03.806522+0000","last_undegraded":"2024-05-09T03:21:03.806522+0000","last_fullsized":"2024-05-09T03:21:03.806522+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:05.678532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589276+0000","last_change":"2024-05-09T03:21:02.595502+0000","last_active":"2024-05-09T03:21:03.589276+0000","last_peered":"2024-05-09T03:21:03.589276+0000","last_clean":"2024-05-09T03:21:03.589276+0000","last_became_active":"2024-05-09T03:21:02.595191+0000","last_became_peered":"2024-05-09T03:21:02.595191+0000","last_unstale":"2024-05-09T03:21:03.589276+0000","last_undegraded":"2024-05-09T03:21:03.589276+0000","last_fullsized":"2024-05-09T03:21:03.589276+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:34:56.154675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806239+0000","last_change":"2024-05-09T03:21:02.595556+0000","last_active":"2024-05-09T03:21:03.806239+0000","last_peered":"2024-05-09T03:21:03.806239+0000","last_clean":"2024-05-09T03:21:03.806239+0000","last_became_active":"2024-05-09T03:21:02.595251+0000","last_became_peered":"2024-05-09T03:21:02.595251+0000","last_unstale":"2024-05-09T03:21:03.806239+0000","last_undegraded":"2024-05-09T03:21:03.806239+0000","last_fullsized":"2024-05-09T03:21:03.806239+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:34.875542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590314+0000","last_change":"2024-05-09T03:21:02.594854+0000","last_active":"2024-05-09T03:21:03.590314+0000","last_peered":"2024-05-09T03:21:03.590314+0000","last_clean":"2024-05-09T03:21:03.590314+0000","last_became_active":"2024-05-09T03:21:02.594545+0000","last_became_peered":"2024-05-09T03:21:02.594545+0000","last_unstale":"2024-05-09T03:21:03.590314+0000","last_undegraded":"2024-05-09T03:21:03.590314+0000","last_fullsized":"2024-05-09T03:21:03.590314+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:57:55.162362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806127+0000","last_change":"2024-05-09T03:21:02.591379+0000","last_active":"2024-05-09T03:21:03.806127+0000","last_peered":"2024-05-09T03:21:03.806127+0000","last_clean":"2024-05-09T03:21:03.806127+0000","last_became_active":"2024-05-09T03:21:02.591064+0000","last_became_peered":"2024-05-09T03:21:02.591064+0000","last_unstale":"2024-05-09T03:21:03.806127+0000","last_undegraded":"2024-05-09T03:21:03.806127+0000","last_fullsized":"2024-05-09T03:21:03.806127+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:21.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806368+0000","last_change":"2024-05-09T03:21:02.592163+0000","last_active":"2024-05-09T03:21:03.806368+0000","last_peered":"2024-05-09T03:21:03.806368+0000","last_clean":"2024-05-09T03:21:03.806368+0000","last_became_active":"2024-05-09T03:21:02.591851+0000","last_became_peered":"2024-05-09T03:21:02.591851+0000","last_unstale":"2024-05-09T03:21:03.806368+0000","last_undegraded":"2024-05-09T03:21:03.806368+0000","last_fullsized":"2024-05-09T03:21:03.806368+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:28:58.074717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807261+0000","last_change":"2024-05-09T03:21:02.599477+0000","last_active":"2024-05-09T03:21:03.807261+0000","last_peered":"2024-05-09T03:21:03.807261+0000","last_clean":"2024-05-09T03:21:03.807261+0000","last_became_active":"2024-05-09T03:21:02.599152+0000","last_became_peered":"2024-05-09T03:21:02.599152+0000","last_unstale":"2024-05-09T03:21:03.807261+0000","last_undegraded":"2024-05-09T03:21:03.807261+0000","last_fullsized":"2024-05-09T03:21:03.807261+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:14.796836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591763+0000","last_change":"2024-05-09T03:21:02.596464+0000","last_active":"2024-05-09T03:21:03.591763+0000","last_peered":"2024-05-09T03:21:03.591763+0000","last_clean":"2024-05-09T03:21:03.591763+0000","last_became_active":"2024-05-09T03:21:02.594727+0000","last_became_peered":"2024-05-09T03:21:02.594727+0000","last_unstale":"2024-05-09T03:21:03.591763+0000","last_undegraded":"2024-05-09T03:21:03.591763+0000","last_fullsized":"2024-05-09T03:21:03.591763+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:26.766250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590718+0000","last_change":"2024-05-09T03:21:02.590312+0000","last_active":"2024-05-09T03:21:03.590718+0000","last_peered":"2024-05-09T03:21:03.590718+0000","last_clean":"2024-05-09T03:21:03.590718+0000","last_became_active":"2024-05-09T03:21:02.590024+0000","last_became_peered":"2024-05-09T03:21:02.590024+0000","last_unstale":"2024-05-09T03:21:03.590718+0000","last_undegraded":"2024-05-09T03:21:03.590718+0000","last_fullsized":"2024-05-09T03:21:03.590718+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:21.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806031+0000","last_change":"2024-05-09T03:21:02.590850+0000","last_active":"2024-05-09T03:21:03.806031+0000","last_peered":"2024-05-09T03:21:03.806031+0000","last_clean":"2024-05-09T03:21:03.806031+0000","last_became_active":"2024-05-09T03:21:02.590558+0000","last_became_peered":"2024-05-09T03:21:02.590558+0000","last_unstale":"2024-05-09T03:21:03.806031+0000","last_undegraded":"2024-05-09T03:21:03.806031+0000","last_fullsized":"2024-05-09T03:21:03.806031+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:16.316105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591410+0000","last_change":"2024-05-09T03:21:02.584457+0000","last_active":"2024-05-09T03:21:03.591410+0000","last_peered":"2024-05-09T03:21:03.591410+0000","last_clean":"2024-05-09T03:21:03.591410+0000","last_became_active":"2024-05-09T03:21:02.584151+0000","last_became_peered":"2024-05-09T03:21:02.584151+0000","last_unstale":"2024-05-09T03:21:03.591410+0000","last_undegraded":"2024-05-09T03:21:03.591410+0000","last_fullsized":"2024-05-09T03:21:03.591410+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:30:19.274582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806035+0000","last_change":"2024-05-09T03:21:02.587071+0000","last_active":"2024-05-09T03:21:03.806035+0000","last_peered":"2024-05-09T03:21:03.806035+0000","last_clean":"2024-05-09T03:21:03.806035+0000","last_became_active":"2024-05-09T03:21:02.586781+0000","last_became_peered":"2024-05-09T03:21:02.586781+0000","last_unstale":"2024-05-09T03:21:03.806035+0000","last_undegraded":"2024-05-09T03:21:03.806035+0000","last_fullsized":"2024-05-09T03:21:03.806035+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:21.171210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806590+0000","last_change":"2024-05-09T03:21:02.595240+0000","last_active":"2024-05-09T03:21:03.806590+0000","last_peered":"2024-05-09T03:21:03.806590+0000","last_clean":"2024-05-09T03:21:03.806590+0000","last_became_active":"2024-05-09T03:21:02.594019+0000","last_became_peered":"2024-05-09T03:21:02.594019+0000","last_unstale":"2024-05-09T03:21:03.806590+0000","last_undegraded":"2024-05-09T03:21:03.806590+0000","last_fullsized":"2024-05-09T03:21:03.806590+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:14:40.633909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805546+0000","last_change":"2024-05-09T03:21:02.588503+0000","last_active":"2024-05-09T03:21:03.805546+0000","last_peered":"2024-05-09T03:21:03.805546+0000","last_clean":"2024-05-09T03:21:03.805546+0000","last_became_active":"2024-05-09T03:21:02.588164+0000","last_became_peered":"2024-05-09T03:21:02.588164+0000","last_unstale":"2024-05-09T03:21:03.805546+0000","last_undegraded":"2024-05-09T03:21:03.805546+0000","last_fullsized":"2024-05-09T03:21:03.805546+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:53.391374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805919+0000","last_change":"2024-05-09T03:21:02.600239+0000","last_active":"2024-05-09T03:21:03.805919+0000","last_peered":"2024-05-09T03:21:03.805919+0000","last_clean":"2024-05-09T03:21:03.805919+0000","last_became_active":"2024-05-09T03:21:02.599863+0000","last_became_peered":"2024-05-09T03:21:02.599863+0000","last_unstale":"2024-05-09T03:21:03.805919+0000","last_undegraded":"2024-05-09T03:21:03.805919+0000","last_fullsized":"2024-05-09T03:21:03.805919+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:53.391374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806854+0000","last_change":"2024-05-09T03:21:02.590233+0000","last_active":"2024-05-09T03:21:03.806854+0000","last_peered":"2024-05-09T03:21:03.806854+0000","last_clean":"2024-05-09T03:21:03.806854+0000","last_became_active":"2024-05-09T03:21:02.589923+0000","last_became_peered":"2024-05-09T03:21:02.589923+0000","last_unstale":"2024-05-09T03:21:03.806854+0000","last_undegraded":"2024-05-09T03:21:03.806854+0000","last_fullsized":"2024-05-09T03:21:03.806854+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:12.303990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806136+0000","last_change":"2024-05-09T03:21:02.596891+0000","last_active":"2024-05-09T03:21:03.806136+0000","last_peered":"2024-05-09T03:21:03.806136+0000","last_clean":"2024-05-09T03:21:03.806136+0000","last_became_active":"2024-05-09T03:21:02.596553+0000","last_became_peered":"2024-05-09T03:21:02.596553+0000","last_unstale":"2024-05-09T03:21:03.806136+0000","last_undegraded":"2024-05-09T03:21:03.806136+0000","last_fullsized":"2024-05-09T03:21:03.806136+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:55:39.851421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806397+0000","last_change":"2024-05-09T03:21:02.592836+0000","last_active":"2024-05-09T03:21:03.806397+0000","last_peered":"2024-05-09T03:21:03.806397+0000","last_clean":"2024-05-09T03:21:03.806397+0000","last_became_active":"2024-05-09T03:21:02.592076+0000","last_became_peered":"2024-05-09T03:21:02.592076+0000","last_unstale":"2024-05-09T03:21:03.806397+0000","last_undegraded":"2024-05-09T03:21:03.806397+0000","last_fullsized":"2024-05-09T03:21:03.806397+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:40:15.448383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591510+0000","last_change":"2024-05-09T03:21:02.592614+0000","last_active":"2024-05-09T03:21:03.591510+0000","last_peered":"2024-05-09T03:21:03.591510+0000","last_clean":"2024-05-09T03:21:03.591510+0000","last_became_active":"2024-05-09T03:21:02.592279+0000","last_became_peered":"2024-05-09T03:21:02.592279+0000","last_unstale":"2024-05-09T03:21:03.591510+0000","last_undegraded":"2024-05-09T03:21:03.591510+0000","last_fullsized":"2024-05-09T03:21:03.591510+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:35.151440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805909+0000","last_change":"2024-05-09T03:21:02.592980+0000","last_active":"2024-05-09T03:21:03.805909+0000","last_peered":"2024-05-09T03:21:03.805909+0000","last_clean":"2024-05-09T03:21:03.805909+0000","last_became_active":"2024-05-09T03:21:02.591237+0000","last_became_peered":"2024-05-09T03:21:02.591237+0000","last_unstale":"2024-05-09T03:21:03.805909+0000","last_undegraded":"2024-05-09T03:21:03.805909+0000","last_fullsized":"2024-05-09T03:21:03.805909+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:31:32.262639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806199+0000","last_change":"2024-05-09T03:21:02.600354+0000","last_active":"2024-05-09T03:21:03.806199+0000","last_peered":"2024-05-09T03:21:03.806199+0000","last_clean":"2024-05-09T03:21:03.806199+0000","last_became_active":"2024-05-09T03:21:02.600051+0000","last_became_peered":"2024-05-09T03:21:02.600051+0000","last_unstale":"2024-05-09T03:21:03.806199+0000","last_undegraded":"2024-05-09T03:21:03.806199+0000","last_fullsized":"2024-05-09T03:21:03.806199+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:01.413424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805580+0000","last_change":"2024-05-09T03:21:02.601341+0000","last_active":"2024-05-09T03:21:03.805580+0000","last_peered":"2024-05-09T03:21:03.805580+0000","last_clean":"2024-05-09T03:21:03.805580+0000","last_became_active":"2024-05-09T03:21:02.601038+0000","last_became_peered":"2024-05-09T03:21:02.601038+0000","last_unstale":"2024-05-09T03:21:03.805580+0000","last_undegraded":"2024-05-09T03:21:03.805580+0000","last_fullsized":"2024-05-09T03:21:03.805580+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:09:34.443805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806146+0000","last_change":"2024-05-09T03:21:02.590037+0000","last_active":"2024-05-09T03:21:03.806146+0000","last_peered":"2024-05-09T03:21:03.806146+0000","last_clean":"2024-05-09T03:21:03.806146+0000","last_became_active":"2024-05-09T03:21:02.589041+0000","last_became_peered":"2024-05-09T03:21:02.589041+0000","last_unstale":"2024-05-09T03:21:03.806146+0000","last_undegraded":"2024-05-09T03:21:03.806146+0000","last_fullsized":"2024-05-09T03:21:03.806146+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:55:39.851421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590819+0000","last_change":"2024-05-09T03:21:02.596257+0000","last_active":"2024-05-09T03:21:03.590819+0000","last_peered":"2024-05-09T03:21:03.590819+0000","last_clean":"2024-05-09T03:21:03.590819+0000","last_became_active":"2024-05-09T03:21:02.595959+0000","last_became_peered":"2024-05-09T03:21:02.595959+0000","last_unstale":"2024-05-09T03:21:03.590819+0000","last_undegraded":"2024-05-09T03:21:03.590819+0000","last_fullsized":"2024-05-09T03:21:03.590819+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:55:39.851421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807145+0000","last_change":"2024-05-09T03:21:02.607373+0000","last_active":"2024-05-09T03:21:03.807145+0000","last_peered":"2024-05-09T03:21:03.807145+0000","last_clean":"2024-05-09T03:21:03.807145+0000","last_became_active":"2024-05-09T03:21:02.607032+0000","last_became_peered":"2024-05-09T03:21:02.607032+0000","last_unstale":"2024-05-09T03:21:03.807145+0000","last_undegraded":"2024-05-09T03:21:03.807145+0000","last_fullsized":"2024-05-09T03:21:03.807145+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:00:04.101720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806613+0000","last_change":"2024-05-09T03:21:02.594783+0000","last_active":"2024-05-09T03:21:03.806613+0000","last_peered":"2024-05-09T03:21:03.806613+0000","last_clean":"2024-05-09T03:21:03.806613+0000","last_became_active":"2024-05-09T03:21:02.594119+0000","last_became_peered":"2024-05-09T03:21:02.594119+0000","last_unstale":"2024-05-09T03:21:03.806613+0000","last_undegraded":"2024-05-09T03:21:03.806613+0000","last_fullsized":"2024-05-09T03:21:03.806613+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:09:34.443805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590571+0000","last_change":"2024-05-09T03:21:02.597865+0000","last_active":"2024-05-09T03:21:03.590571+0000","last_peered":"2024-05-09T03:21:03.590571+0000","last_clean":"2024-05-09T03:21:03.590571+0000","last_became_active":"2024-05-09T03:21:02.597427+0000","last_became_peered":"2024-05-09T03:21:02.597427+0000","last_unstale":"2024-05-09T03:21:03.590571+0000","last_undegraded":"2024-05-09T03:21:03.590571+0000","last_fullsized":"2024-05-09T03:21:03.590571+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:40.499483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806755+0000","last_change":"2024-05-09T03:21:02.598390+0000","last_active":"2024-05-09T03:21:03.806755+0000","last_peered":"2024-05-09T03:21:03.806755+0000","last_clean":"2024-05-09T03:21:03.806755+0000","last_became_active":"2024-05-09T03:21:02.598081+0000","last_became_peered":"2024-05-09T03:21:02.598081+0000","last_unstale":"2024-05-09T03:21:03.806755+0000","last_undegraded":"2024-05-09T03:21:03.806755+0000","last_fullsized":"2024-05-09T03:21:03.806755+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:52:32.151692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806662+0000","last_change":"2024-05-09T03:21:02.594998+0000","last_active":"2024-05-09T03:21:03.806662+0000","last_peered":"2024-05-09T03:21:03.806662+0000","last_clean":"2024-05-09T03:21:03.806662+0000","last_became_active":"2024-05-09T03:21:02.593677+0000","last_became_peered":"2024-05-09T03:21:02.593677+0000","last_unstale":"2024-05-09T03:21:03.806662+0000","last_undegraded":"2024-05-09T03:21:03.806662+0000","last_fullsized":"2024-05-09T03:21:03.806662+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:40:57.429739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805817+0000","last_change":"2024-05-09T03:21:02.592161+0000","last_active":"2024-05-09T03:21:03.805817+0000","last_peered":"2024-05-09T03:21:03.805817+0000","last_clean":"2024-05-09T03:21:03.805817+0000","last_became_active":"2024-05-09T03:21:02.591810+0000","last_became_peered":"2024-05-09T03:21:02.591810+0000","last_unstale":"2024-05-09T03:21:03.805817+0000","last_undegraded":"2024-05-09T03:21:03.805817+0000","last_fullsized":"2024-05-09T03:21:03.805817+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:16.316105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589880+0000","last_change":"2024-05-09T03:21:02.608498+0000","last_active":"2024-05-09T03:21:03.589880+0000","last_peered":"2024-05-09T03:21:03.589880+0000","last_clean":"2024-05-09T03:21:03.589880+0000","last_became_active":"2024-05-09T03:21:02.607950+0000","last_became_peered":"2024-05-09T03:21:02.607950+0000","last_unstale":"2024-05-09T03:21:03.589880+0000","last_undegraded":"2024-05-09T03:21:03.589880+0000","last_fullsized":"2024-05-09T03:21:03.589880+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:25:13.370650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806138+0000","last_change":"2024-05-09T03:21:02.593662+0000","last_active":"2024-05-09T03:21:03.806138+0000","last_peered":"2024-05-09T03:21:03.806138+0000","last_clean":"2024-05-09T03:21:03.806138+0000","last_became_active":"2024-05-09T03:21:02.591415+0000","last_became_peered":"2024-05-09T03:21:02.591415+0000","last_unstale":"2024-05-09T03:21:03.806138+0000","last_undegraded":"2024-05-09T03:21:03.806138+0000","last_fullsized":"2024-05-09T03:21:03.806138+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:04.677834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"134'4","reported_seq":18,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806963+0000","last_change":"2024-05-09T03:21:01.588341+0000","last_active":"2024-05-09T03:21:03.806963+0000","last_peered":"2024-05-09T03:21:03.806963+0000","last_clean":"2024-05-09T03:21:03.806963+0000","last_became_active":"2024-05-09T03:21:01.588033+0000","last_became_peered":"2024-05-09T03:21:01.588033+0000","last_unstale":"2024-05-09T03:21:03.806963+0000","last_undegraded":"2024-05-09T03:21:03.806963+0000","last_fullsized":"2024-05-09T03:21:03.806963+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:34.148962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806967+0000","last_change":"2024-05-09T03:21:02.589024+0000","last_active":"2024-05-09T03:21:03.806967+0000","last_peered":"2024-05-09T03:21:03.806967+0000","last_clean":"2024-05-09T03:21:03.806967+0000","last_became_active":"2024-05-09T03:21:02.588711+0000","last_became_peered":"2024-05-09T03:21:02.588711+0000","last_unstale":"2024-05-09T03:21:03.806967+0000","last_undegraded":"2024-05-09T03:21:03.806967+0000","last_fullsized":"2024-05-09T03:21:03.806967+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:01.413424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806609+0000","last_change":"2024-05-09T03:21:01.590137+0000","last_active":"2024-05-09T03:21:03.806609+0000","last_peered":"2024-05-09T03:21:03.806609+0000","last_clean":"2024-05-09T03:21:03.806609+0000","last_became_active":"2024-05-09T03:21:01.589800+0000","last_became_peered":"2024-05-09T03:21:01.589800+0000","last_unstale":"2024-05-09T03:21:03.806609+0000","last_undegraded":"2024-05-09T03:21:03.806609+0000","last_fullsized":"2024-05-09T03:21:03.806609+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:05.717846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589269+0000","last_change":"2024-05-09T03:21:02.600394+0000","last_active":"2024-05-09T03:21:03.589269+0000","last_peered":"2024-05-09T03:21:03.589269+0000","last_clean":"2024-05-09T03:21:03.589269+0000","last_became_active":"2024-05-09T03:21:02.600093+0000","last_became_peered":"2024-05-09T03:21:02.600093+0000","last_unstale":"2024-05-09T03:21:03.589269+0000","last_undegraded":"2024-05-09T03:21:03.589269+0000","last_fullsized":"2024-05-09T03:21:03.589269+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:34:38.465838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806544+0000","last_change":"2024-05-09T03:21:01.593090+0000","last_active":"2024-05-09T03:21:03.806544+0000","last_peered":"2024-05-09T03:21:03.806544+0000","last_clean":"2024-05-09T03:21:03.806544+0000","last_became_active":"2024-05-09T03:21:01.592330+0000","last_became_peered":"2024-05-09T03:21:01.592330+0000","last_unstale":"2024-05-09T03:21:03.806544+0000","last_undegraded":"2024-05-09T03:21:03.806544+0000","last_fullsized":"2024-05-09T03:21:03.806544+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:05.717846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591740+0000","last_change":"2024-05-09T03:21:02.590740+0000","last_active":"2024-05-09T03:21:03.591740+0000","last_peered":"2024-05-09T03:21:03.591740+0000","last_clean":"2024-05-09T03:21:03.591740+0000","last_became_active":"2024-05-09T03:21:02.590447+0000","last_became_peered":"2024-05-09T03:21:02.590447+0000","last_unstale":"2024-05-09T03:21:03.591740+0000","last_undegraded":"2024-05-09T03:21:03.591740+0000","last_fullsized":"2024-05-09T03:21:03.591740+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:15.484803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806978+0000","last_change":"2024-05-09T03:21:01.591363+0000","last_active":"2024-05-09T03:21:03.806978+0000","last_peered":"2024-05-09T03:21:03.806978+0000","last_clean":"2024-05-09T03:21:03.806978+0000","last_became_active":"2024-05-09T03:21:01.591034+0000","last_became_peered":"2024-05-09T03:21:01.591034+0000","last_unstale":"2024-05-09T03:21:03.806978+0000","last_undegraded":"2024-05-09T03:21:03.806978+0000","last_fullsized":"2024-05-09T03:21:03.806978+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:16.915428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806010+0000","last_change":"2024-05-09T03:21:02.599419+0000","last_active":"2024-05-09T03:21:03.806010+0000","last_peered":"2024-05-09T03:21:03.806010+0000","last_clean":"2024-05-09T03:21:03.806010+0000","last_became_active":"2024-05-09T03:21:02.599059+0000","last_became_peered":"2024-05-09T03:21:02.599059+0000","last_unstale":"2024-05-09T03:21:03.806010+0000","last_undegraded":"2024-05-09T03:21:03.806010+0000","last_fullsized":"2024-05-09T03:21:03.806010+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:57:42.234037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589739+0000","last_change":"2024-05-09T03:21:01.578075+0000","last_active":"2024-05-09T03:21:03.589739+0000","last_peered":"2024-05-09T03:21:03.589739+0000","last_clean":"2024-05-09T03:21:03.589739+0000","last_became_active":"2024-05-09T03:21:01.577735+0000","last_became_peered":"2024-05-09T03:21:01.577735+0000","last_unstale":"2024-05-09T03:21:03.589739+0000","last_undegraded":"2024-05-09T03:21:03.589739+0000","last_fullsized":"2024-05-09T03:21:03.589739+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:38:04.349592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806347+0000","last_change":"2024-05-09T03:21:02.588348+0000","last_active":"2024-05-09T03:21:03.806347+0000","last_peered":"2024-05-09T03:21:03.806347+0000","last_clean":"2024-05-09T03:21:03.806347+0000","last_became_active":"2024-05-09T03:21:02.588016+0000","last_became_peered":"2024-05-09T03:21:02.588016+0000","last_unstale":"2024-05-09T03:21:03.806347+0000","last_undegraded":"2024-05-09T03:21:03.806347+0000","last_fullsized":"2024-05-09T03:21:03.806347+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:34.875542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806305+0000","last_change":"2024-05-09T03:21:01.591082+0000","last_active":"2024-05-09T03:21:03.806305+0000","last_peered":"2024-05-09T03:21:03.806305+0000","last_clean":"2024-05-09T03:21:03.806305+0000","last_became_active":"2024-05-09T03:21:01.590693+0000","last_became_peered":"2024-05-09T03:21:01.590693+0000","last_unstale":"2024-05-09T03:21:03.806305+0000","last_undegraded":"2024-05-09T03:21:03.806305+0000","last_fullsized":"2024-05-09T03:21:03.806305+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:14.482325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806266+0000","last_change":"2024-05-09T03:21:02.591399+0000","last_active":"2024-05-09T03:21:03.806266+0000","last_peered":"2024-05-09T03:21:03.806266+0000","last_clean":"2024-05-09T03:21:03.806266+0000","last_became_active":"2024-05-09T03:21:02.589062+0000","last_became_peered":"2024-05-09T03:21:02.589062+0000","last_unstale":"2024-05-09T03:21:03.806266+0000","last_undegraded":"2024-05-09T03:21:03.806266+0000","last_fullsized":"2024-05-09T03:21:03.806266+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:28:58.074717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590365+0000","last_change":"2024-05-09T03:21:01.584386+0000","last_active":"2024-05-09T03:21:03.590365+0000","last_peered":"2024-05-09T03:21:03.590365+0000","last_clean":"2024-05-09T03:21:03.590365+0000","last_became_active":"2024-05-09T03:21:01.582362+0000","last_became_peered":"2024-05-09T03:21:01.582362+0000","last_unstale":"2024-05-09T03:21:03.590365+0000","last_undegraded":"2024-05-09T03:21:03.590365+0000","last_fullsized":"2024-05-09T03:21:03.590365+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:53:27.616684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590277+0000","last_change":"2024-05-09T03:21:02.590877+0000","last_active":"2024-05-09T03:21:03.590277+0000","last_peered":"2024-05-09T03:21:03.590277+0000","last_clean":"2024-05-09T03:21:03.590277+0000","last_became_active":"2024-05-09T03:21:02.590524+0000","last_became_peered":"2024-05-09T03:21:02.590524+0000","last_unstale":"2024-05-09T03:21:03.590277+0000","last_undegraded":"2024-05-09T03:21:03.590277+0000","last_fullsized":"2024-05-09T03:21:03.590277+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:28.667421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589186+0000","last_change":"2024-05-09T03:21:01.583055+0000","last_active":"2024-05-09T03:21:03.589186+0000","last_peered":"2024-05-09T03:21:03.589186+0000","last_clean":"2024-05-09T03:21:03.589186+0000","last_became_active":"2024-05-09T03:21:01.582183+0000","last_became_peered":"2024-05-09T03:21:01.582183+0000","last_unstale":"2024-05-09T03:21:03.589186+0000","last_undegraded":"2024-05-09T03:21:03.589186+0000","last_fullsized":"2024-05-09T03:21:03.589186+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:53:06.151109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806860+0000","last_change":"2024-05-09T03:21:02.596214+0000","last_active":"2024-05-09T03:21:03.806860+0000","last_peered":"2024-05-09T03:21:03.806860+0000","last_clean":"2024-05-09T03:21:03.806860+0000","last_became_active":"2024-05-09T03:21:02.595892+0000","last_became_peered":"2024-05-09T03:21:02.595892+0000","last_unstale":"2024-05-09T03:21:03.806860+0000","last_undegraded":"2024-05-09T03:21:03.806860+0000","last_fullsized":"2024-05-09T03:21:03.806860+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:02:13.221763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589653+0000","last_change":"2024-05-09T03:21:01.579786+0000","last_active":"2024-05-09T03:21:03.589653+0000","last_peered":"2024-05-09T03:21:03.589653+0000","last_clean":"2024-05-09T03:21:03.589653+0000","last_became_active":"2024-05-09T03:21:01.579479+0000","last_became_peered":"2024-05-09T03:21:01.579479+0000","last_unstale":"2024-05-09T03:21:03.589653+0000","last_undegraded":"2024-05-09T03:21:03.589653+0000","last_fullsized":"2024-05-09T03:21:03.589653+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:32:52.174643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591758+0000","last_change":"2024-05-09T03:21:02.595986+0000","last_active":"2024-05-09T03:21:03.591758+0000","last_peered":"2024-05-09T03:21:03.591758+0000","last_clean":"2024-05-09T03:21:03.591758+0000","last_became_active":"2024-05-09T03:21:02.595467+0000","last_became_peered":"2024-05-09T03:21:02.595467+0000","last_unstale":"2024-05-09T03:21:03.591758+0000","last_undegraded":"2024-05-09T03:21:03.591758+0000","last_fullsized":"2024-05-09T03:21:03.591758+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:21.156037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805995+0000","last_change":"2024-05-09T03:21:01.591561+0000","last_active":"2024-05-09T03:21:03.805995+0000","last_peered":"2024-05-09T03:21:03.805995+0000","last_clean":"2024-05-09T03:21:03.805995+0000","last_became_active":"2024-05-09T03:21:01.591162+0000","last_became_peered":"2024-05-09T03:21:01.591162+0000","last_unstale":"2024-05-09T03:21:03.805995+0000","last_undegraded":"2024-05-09T03:21:03.805995+0000","last_fullsized":"2024-05-09T03:21:03.805995+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:13:14.482325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590660+0000","last_change":"2024-05-09T03:21:02.597970+0000","last_active":"2024-05-09T03:21:03.590660+0000","last_peered":"2024-05-09T03:21:03.590660+0000","last_clean":"2024-05-09T03:21:03.590660+0000","last_became_active":"2024-05-09T03:21:02.597265+0000","last_became_peered":"2024-05-09T03:21:02.597265+0000","last_unstale":"2024-05-09T03:21:03.590660+0000","last_undegraded":"2024-05-09T03:21:03.590660+0000","last_fullsized":"2024-05-09T03:21:03.590660+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:54:16.316105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"134'3","reported_seq":17,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806759+0000","last_change":"2024-05-09T03:21:01.585751+0000","last_active":"2024-05-09T03:21:03.806759+0000","last_peered":"2024-05-09T03:21:03.806759+0000","last_clean":"2024-05-09T03:21:03.806759+0000","last_became_active":"2024-05-09T03:21:01.585433+0000","last_became_peered":"2024-05-09T03:21:01.585433+0000","last_unstale":"2024-05-09T03:21:03.806759+0000","last_undegraded":"2024-05-09T03:21:03.806759+0000","last_fullsized":"2024-05-09T03:21:03.806759+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:36:14.219803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806604+0000","last_change":"2024-05-09T03:21:02.600127+0000","last_active":"2024-05-09T03:21:03.806604+0000","last_peered":"2024-05-09T03:21:03.806604+0000","last_clean":"2024-05-09T03:21:03.806604+0000","last_became_active":"2024-05-09T03:21:02.599789+0000","last_became_peered":"2024-05-09T03:21:02.599789+0000","last_unstale":"2024-05-09T03:21:03.806604+0000","last_undegraded":"2024-05-09T03:21:03.806604+0000","last_fullsized":"2024-05-09T03:21:03.806604+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:58:01.413424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"134'6","reported_seq":20,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591622+0000","last_change":"2024-05-09T03:21:01.584309+0000","last_active":"2024-05-09T03:21:03.591622+0000","last_peered":"2024-05-09T03:21:03.591622+0000","last_clean":"2024-05-09T03:21:03.591622+0000","last_became_active":"2024-05-09T03:21:01.583997+0000","last_became_peered":"2024-05-09T03:21:01.583997+0000","last_unstale":"2024-05-09T03:21:03.591622+0000","last_undegraded":"2024-05-09T03:21:03.591622+0000","last_fullsized":"2024-05-09T03:21:03.591622+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:08.163668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806550+0000","last_change":"2024-05-09T03:21:02.585939+0000","last_active":"2024-05-09T03:21:03.806550+0000","last_peered":"2024-05-09T03:21:03.806550+0000","last_clean":"2024-05-09T03:21:03.806550+0000","last_became_active":"2024-05-09T03:21:02.585633+0000","last_became_peered":"2024-05-09T03:21:02.585633+0000","last_unstale":"2024-05-09T03:21:03.806550+0000","last_undegraded":"2024-05-09T03:21:03.806550+0000","last_fullsized":"2024-05-09T03:21:03.806550+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:09:34.443805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806180+0000","last_change":"2024-05-09T03:21:01.587181+0000","last_active":"2024-05-09T03:21:03.806180+0000","last_peered":"2024-05-09T03:21:03.806180+0000","last_clean":"2024-05-09T03:21:03.806180+0000","last_became_active":"2024-05-09T03:21:01.586851+0000","last_became_peered":"2024-05-09T03:21:01.586851+0000","last_unstale":"2024-05-09T03:21:03.806180+0000","last_undegraded":"2024-05-09T03:21:03.806180+0000","last_fullsized":"2024-05-09T03:21:03.806180+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:05:03.154206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589614+0000","last_change":"2024-05-09T03:21:02.601331+0000","last_active":"2024-05-09T03:21:03.589614+0000","last_peered":"2024-05-09T03:21:03.589614+0000","last_clean":"2024-05-09T03:21:03.589614+0000","last_became_active":"2024-05-09T03:21:02.600974+0000","last_became_peered":"2024-05-09T03:21:02.600974+0000","last_unstale":"2024-05-09T03:21:03.589614+0000","last_undegraded":"2024-05-09T03:21:03.589614+0000","last_fullsized":"2024-05-09T03:21:03.589614+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:45:09.166146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806865+0000","last_change":"2024-05-09T03:21:01.588880+0000","last_active":"2024-05-09T03:21:03.806865+0000","last_peered":"2024-05-09T03:21:03.806865+0000","last_clean":"2024-05-09T03:21:03.806865+0000","last_became_active":"2024-05-09T03:21:01.588425+0000","last_became_peered":"2024-05-09T03:21:01.588425+0000","last_unstale":"2024-05-09T03:21:03.806865+0000","last_undegraded":"2024-05-09T03:21:03.806865+0000","last_fullsized":"2024-05-09T03:21:03.806865+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:31.223171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591014+0000","last_change":"2024-05-09T03:21:02.587527+0000","last_active":"2024-05-09T03:21:03.591014+0000","last_peered":"2024-05-09T03:21:03.591014+0000","last_clean":"2024-05-09T03:21:03.591014+0000","last_became_active":"2024-05-09T03:21:02.587183+0000","last_became_peered":"2024-05-09T03:21:02.587183+0000","last_unstale":"2024-05-09T03:21:03.591014+0000","last_undegraded":"2024-05-09T03:21:03.591014+0000","last_fullsized":"2024-05-09T03:21:03.591014+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:09:06.720324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590413+0000","last_change":"2024-05-09T03:21:01.577198+0000","last_active":"2024-05-09T03:21:03.590413+0000","last_peered":"2024-05-09T03:21:03.590413+0000","last_clean":"2024-05-09T03:21:03.590413+0000","last_became_active":"2024-05-09T03:21:01.576834+0000","last_became_peered":"2024-05-09T03:21:01.576834+0000","last_unstale":"2024-05-09T03:21:03.590413+0000","last_undegraded":"2024-05-09T03:21:03.590413+0000","last_fullsized":"2024-05-09T03:21:03.590413+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:03:58.267719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805600+0000","last_change":"2024-05-09T03:21:02.591948+0000","last_active":"2024-05-09T03:21:03.805600+0000","last_peered":"2024-05-09T03:21:03.805600+0000","last_clean":"2024-05-09T03:21:03.805600+0000","last_became_active":"2024-05-09T03:21:02.591655+0000","last_became_peered":"2024-05-09T03:21:02.591655+0000","last_unstale":"2024-05-09T03:21:03.805600+0000","last_undegraded":"2024-05-09T03:21:03.805600+0000","last_fullsized":"2024-05-09T03:21:03.805600+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:09.694627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"134'8","reported_seq":22,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589328+0000","last_change":"2024-05-09T03:21:01.588952+0000","last_active":"2024-05-09T03:21:03.589328+0000","last_peered":"2024-05-09T03:21:03.589328+0000","last_clean":"2024-05-09T03:21:03.589328+0000","last_became_active":"2024-05-09T03:21:01.588569+0000","last_became_peered":"2024-05-09T03:21:01.588569+0000","last_unstale":"2024-05-09T03:21:03.589328+0000","last_undegraded":"2024-05-09T03:21:03.589328+0000","last_fullsized":"2024-05-09T03:21:03.589328+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:12:34.502908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590930+0000","last_change":"2024-05-09T03:21:02.598414+0000","last_active":"2024-05-09T03:21:03.590930+0000","last_peered":"2024-05-09T03:21:03.590930+0000","last_clean":"2024-05-09T03:21:03.590930+0000","last_became_active":"2024-05-09T03:21:02.598068+0000","last_became_peered":"2024-05-09T03:21:02.598068+0000","last_unstale":"2024-05-09T03:21:03.590930+0000","last_undegraded":"2024-05-09T03:21:03.590930+0000","last_fullsized":"2024-05-09T03:21:03.590930+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:34.875542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591747+0000","last_change":"2024-05-09T03:21:01.584627+0000","last_active":"2024-05-09T03:21:03.591747+0000","last_peered":"2024-05-09T03:21:03.591747+0000","last_clean":"2024-05-09T03:21:03.591747+0000","last_became_active":"2024-05-09T03:21:01.584293+0000","last_became_peered":"2024-05-09T03:21:01.584293+0000","last_unstale":"2024-05-09T03:21:03.591747+0000","last_undegraded":"2024-05-09T03:21:03.591747+0000","last_fullsized":"2024-05-09T03:21:03.591747+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:58:27.097696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806205+0000","last_change":"2024-05-09T03:21:02.593056+0000","last_active":"2024-05-09T03:21:03.806205+0000","last_peered":"2024-05-09T03:21:03.806205+0000","last_clean":"2024-05-09T03:21:03.806205+0000","last_became_active":"2024-05-09T03:21:02.592710+0000","last_became_peered":"2024-05-09T03:21:02.592710+0000","last_unstale":"2024-05-09T03:21:03.806205+0000","last_undegraded":"2024-05-09T03:21:03.806205+0000","last_fullsized":"2024-05-09T03:21:03.806205+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:05:48.022379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590756+0000","last_change":"2024-05-09T03:21:01.580717+0000","last_active":"2024-05-09T03:21:03.590756+0000","last_peered":"2024-05-09T03:21:03.590756+0000","last_clean":"2024-05-09T03:21:03.590756+0000","last_became_active":"2024-05-09T03:21:01.580391+0000","last_became_peered":"2024-05-09T03:21:01.580391+0000","last_unstale":"2024-05-09T03:21:03.590756+0000","last_undegraded":"2024-05-09T03:21:03.590756+0000","last_fullsized":"2024-05-09T03:21:03.590756+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:51:16.915428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806778+0000","last_change":"2024-05-09T03:21:02.592575+0000","last_active":"2024-05-09T03:21:03.806778+0000","last_peered":"2024-05-09T03:21:03.806778+0000","last_clean":"2024-05-09T03:21:03.806778+0000","last_became_active":"2024-05-09T03:21:02.592074+0000","last_became_peered":"2024-05-09T03:21:02.592074+0000","last_unstale":"2024-05-09T03:21:03.806778+0000","last_undegraded":"2024-05-09T03:21:03.806778+0000","last_fullsized":"2024-05-09T03:21:03.806778+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:53:05.678532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807093+0000","last_change":"2024-05-09T03:21:01.591388+0000","last_active":"2024-05-09T03:21:03.807093+0000","last_peered":"2024-05-09T03:21:03.807093+0000","last_clean":"2024-05-09T03:21:03.807093+0000","last_became_active":"2024-05-09T03:21:01.591067+0000","last_became_peered":"2024-05-09T03:21:01.591067+0000","last_unstale":"2024-05-09T03:21:03.807093+0000","last_undegraded":"2024-05-09T03:21:03.807093+0000","last_fullsized":"2024-05-09T03:21:03.807093+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:27.664943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806762+0000","last_change":"2024-05-09T03:21:02.594559+0000","last_active":"2024-05-09T03:21:03.806762+0000","last_peered":"2024-05-09T03:21:03.806762+0000","last_clean":"2024-05-09T03:21:03.806762+0000","last_became_active":"2024-05-09T03:21:02.593345+0000","last_became_peered":"2024-05-09T03:21:02.593345+0000","last_unstale":"2024-05-09T03:21:03.806762+0000","last_undegraded":"2024-05-09T03:21:03.806762+0000","last_fullsized":"2024-05-09T03:21:03.806762+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:13.183796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806468+0000","last_change":"2024-05-09T03:21:01.592276+0000","last_active":"2024-05-09T03:21:03.806468+0000","last_peered":"2024-05-09T03:21:03.806468+0000","last_clean":"2024-05-09T03:21:03.806468+0000","last_became_active":"2024-05-09T03:21:01.591952+0000","last_became_peered":"2024-05-09T03:21:01.591952+0000","last_unstale":"2024-05-09T03:21:03.806468+0000","last_undegraded":"2024-05-09T03:21:03.806468+0000","last_fullsized":"2024-05-09T03:21:03.806468+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:30:18.272104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806069+0000","last_change":"2024-05-09T03:21:02.601889+0000","last_active":"2024-05-09T03:21:03.806069+0000","last_peered":"2024-05-09T03:21:03.806069+0000","last_clean":"2024-05-09T03:21:03.806069+0000","last_became_active":"2024-05-09T03:21:02.601591+0000","last_became_peered":"2024-05-09T03:21:02.601591+0000","last_unstale":"2024-05-09T03:21:03.806069+0000","last_undegraded":"2024-05-09T03:21:03.806069+0000","last_fullsized":"2024-05-09T03:21:03.806069+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:46.832807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806125+0000","last_change":"2024-05-09T03:21:01.585788+0000","last_active":"2024-05-09T03:21:03.806125+0000","last_peered":"2024-05-09T03:21:03.806125+0000","last_clean":"2024-05-09T03:21:03.806125+0000","last_became_active":"2024-05-09T03:21:01.585466+0000","last_became_peered":"2024-05-09T03:21:01.585466+0000","last_unstale":"2024-05-09T03:21:03.806125+0000","last_undegraded":"2024-05-09T03:21:03.806125+0000","last_fullsized":"2024-05-09T03:21:03.806125+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:42.528435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805718+0000","last_change":"2024-05-09T03:21:02.591591+0000","last_active":"2024-05-09T03:21:03.805718+0000","last_peered":"2024-05-09T03:21:03.805718+0000","last_clean":"2024-05-09T03:21:03.805718+0000","last_became_active":"2024-05-09T03:21:02.591158+0000","last_became_peered":"2024-05-09T03:21:02.591158+0000","last_unstale":"2024-05-09T03:21:03.805718+0000","last_undegraded":"2024-05-09T03:21:03.805718+0000","last_fullsized":"2024-05-09T03:21:03.805718+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:09.540737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805921+0000","last_change":"2024-05-09T03:21:01.584571+0000","last_active":"2024-05-09T03:21:03.805921+0000","last_peered":"2024-05-09T03:21:03.805921+0000","last_clean":"2024-05-09T03:21:03.805921+0000","last_became_active":"2024-05-09T03:21:01.584229+0000","last_became_peered":"2024-05-09T03:21:01.584229+0000","last_unstale":"2024-05-09T03:21:03.805921+0000","last_undegraded":"2024-05-09T03:21:03.805921+0000","last_fullsized":"2024-05-09T03:21:03.805921+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:50:40.349991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590062+0000","last_change":"2024-05-09T03:21:02.590312+0000","last_active":"2024-05-09T03:21:03.590062+0000","last_peered":"2024-05-09T03:21:03.590062+0000","last_clean":"2024-05-09T03:21:03.590062+0000","last_became_active":"2024-05-09T03:21:02.589802+0000","last_became_peered":"2024-05-09T03:21:02.589802+0000","last_unstale":"2024-05-09T03:21:03.590062+0000","last_undegraded":"2024-05-09T03:21:03.590062+0000","last_fullsized":"2024-05-09T03:21:03.590062+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:00.709346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805540+0000","last_change":"2024-05-09T03:21:01.593505+0000","last_active":"2024-05-09T03:21:03.805540+0000","last_peered":"2024-05-09T03:21:03.805540+0000","last_clean":"2024-05-09T03:21:03.805540+0000","last_became_active":"2024-05-09T03:21:01.592448+0000","last_became_peered":"2024-05-09T03:21:01.592448+0000","last_unstale":"2024-05-09T03:21:03.805540+0000","last_undegraded":"2024-05-09T03:21:03.805540+0000","last_fullsized":"2024-05-09T03:21:03.805540+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:17:25.763772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807032+0000","last_change":"2024-05-09T03:21:02.601645+0000","last_active":"2024-05-09T03:21:03.807032+0000","last_peered":"2024-05-09T03:21:03.807032+0000","last_clean":"2024-05-09T03:21:03.807032+0000","last_became_active":"2024-05-09T03:21:02.600936+0000","last_became_peered":"2024-05-09T03:21:02.600936+0000","last_unstale":"2024-05-09T03:21:03.807032+0000","last_undegraded":"2024-05-09T03:21:03.807032+0000","last_fullsized":"2024-05-09T03:21:03.807032+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T03:54:43.745630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"134'2","reported_seq":16,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806325+0000","last_change":"2024-05-09T03:21:01.581006+0000","last_active":"2024-05-09T03:21:03.806325+0000","last_peered":"2024-05-09T03:21:03.806325+0000","last_clean":"2024-05-09T03:21:03.806325+0000","last_became_active":"2024-05-09T03:21:01.580605+0000","last_became_peered":"2024-05-09T03:21:01.580605+0000","last_unstale":"2024-05-09T03:21:03.806325+0000","last_undegraded":"2024-05-09T03:21:03.806325+0000","last_fullsized":"2024-05-09T03:21:03.806325+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:57:28.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589666+0000","last_change":"2024-05-09T03:21:02.602065+0000","last_active":"2024-05-09T03:21:03.589666+0000","last_peered":"2024-05-09T03:21:03.589666+0000","last_clean":"2024-05-09T03:21:03.589666+0000","last_became_active":"2024-05-09T03:21:02.601754+0000","last_became_peered":"2024-05-09T03:21:02.601754+0000","last_unstale":"2024-05-09T03:21:03.589666+0000","last_undegraded":"2024-05-09T03:21:03.589666+0000","last_fullsized":"2024-05-09T03:21:03.589666+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:03:59.270197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"135'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.807178+0000","last_change":"2024-05-09T03:21:01.592928+0000","last_active":"2024-05-09T03:21:03.807178+0000","last_peered":"2024-05-09T03:21:03.807178+0000","last_clean":"2024-05-09T03:21:03.807178+0000","last_became_active":"2024-05-09T03:21:01.592598+0000","last_became_peered":"2024-05-09T03:21:01.592598+0000","last_unstale":"2024-05-09T03:21:03.807178+0000","last_undegraded":"2024-05-09T03:21:03.807178+0000","last_fullsized":"2024-05-09T03:21:03.807178+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:49:20.153559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.589284+0000","last_change":"2024-05-09T03:21:02.588027+0000","last_active":"2024-05-09T03:21:03.589284+0000","last_peered":"2024-05-09T03:21:03.589284+0000","last_clean":"2024-05-09T03:21:03.589284+0000","last_became_active":"2024-05-09T03:21:02.587681+0000","last_became_peered":"2024-05-09T03:21:02.587681+0000","last_unstale":"2024-05-09T03:21:03.589284+0000","last_undegraded":"2024-05-09T03:21:03.589284+0000","last_fullsized":"2024-05-09T03:21:03.589284+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:47:20.121767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806670+0000","last_change":"2024-05-09T03:21:01.591571+0000","last_active":"2024-05-09T03:21:03.806670+0000","last_peered":"2024-05-09T03:21:03.806670+0000","last_clean":"2024-05-09T03:21:03.806670+0000","last_became_active":"2024-05-09T03:21:01.590921+0000","last_became_peered":"2024-05-09T03:21:01.590921+0000","last_unstale":"2024-05-09T03:21:03.806670+0000","last_undegraded":"2024-05-09T03:21:03.806670+0000","last_fullsized":"2024-05-09T03:21:03.806670+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:36:14.219803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590593+0000","last_change":"2024-05-09T03:21:02.597393+0000","last_active":"2024-05-09T03:21:03.590593+0000","last_peered":"2024-05-09T03:21:03.590593+0000","last_clean":"2024-05-09T03:21:03.590593+0000","last_became_active":"2024-05-09T03:21:02.597001+0000","last_became_peered":"2024-05-09T03:21:02.597001+0000","last_unstale":"2024-05-09T03:21:03.590593+0000","last_undegraded":"2024-05-09T03:21:03.590593+0000","last_fullsized":"2024-05-09T03:21:03.590593+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:10:53.391374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590699+0000","last_change":"2024-05-09T03:21:01.584090+0000","last_active":"2024-05-09T03:21:03.590699+0000","last_peered":"2024-05-09T03:21:03.590699+0000","last_clean":"2024-05-09T03:21:03.590699+0000","last_became_active":"2024-05-09T03:21:01.583736+0000","last_became_peered":"2024-05-09T03:21:01.583736+0000","last_unstale":"2024-05-09T03:21:03.590699+0000","last_undegraded":"2024-05-09T03:21:03.590699+0000","last_fullsized":"2024-05-09T03:21:03.590699+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.775277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806833+0000","last_change":"2024-05-09T03:21:02.597293+0000","last_active":"2024-05-09T03:21:03.806833+0000","last_peered":"2024-05-09T03:21:03.806833+0000","last_clean":"2024-05-09T03:21:03.806833+0000","last_became_active":"2024-05-09T03:21:02.595518+0000","last_became_peered":"2024-05-09T03:21:02.595518+0000","last_unstale":"2024-05-09T03:21:03.806833+0000","last_undegraded":"2024-05-09T03:21:03.806833+0000","last_fullsized":"2024-05-09T03:21:03.806833+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:55:37.739225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806588+0000","last_change":"2024-05-09T03:21:01.581337+0000","last_active":"2024-05-09T03:21:03.806588+0000","last_peered":"2024-05-09T03:21:03.806588+0000","last_clean":"2024-05-09T03:21:03.806588+0000","last_became_active":"2024-05-09T03:21:01.581023+0000","last_became_peered":"2024-05-09T03:21:01.581023+0000","last_unstale":"2024-05-09T03:21:03.806588+0000","last_undegraded":"2024-05-09T03:21:03.806588+0000","last_fullsized":"2024-05-09T03:21:03.806588+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:08:59.706868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.591492+0000","last_change":"2024-05-09T03:21:02.588790+0000","last_active":"2024-05-09T03:21:03.591492+0000","last_peered":"2024-05-09T03:21:03.591492+0000","last_clean":"2024-05-09T03:21:03.591492+0000","last_became_active":"2024-05-09T03:21:02.588445+0000","last_became_peered":"2024-05-09T03:21:02.588445+0000","last_unstale":"2024-05-09T03:21:03.591492+0000","last_undegraded":"2024-05-09T03:21:03.591492+0000","last_fullsized":"2024-05-09T03:21:03.591492+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T04:36:15.222281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590034+0000","last_change":"2024-05-09T03:21:01.581402+0000","last_active":"2024-05-09T03:21:03.590034+0000","last_peered":"2024-05-09T03:21:03.590034+0000","last_clean":"2024-05-09T03:21:03.590034+0000","last_became_active":"2024-05-09T03:21:01.580048+0000","last_became_peered":"2024-05-09T03:21:01.580048+0000","last_unstale":"2024-05-09T03:21:03.590034+0000","last_undegraded":"2024-05-09T03:21:03.590034+0000","last_fullsized":"2024-05-09T03:21:03.590034+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:59:03.052699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805484+0000","last_change":"2024-05-09T03:21:02.591456+0000","last_active":"2024-05-09T03:21:03.805484+0000","last_peered":"2024-05-09T03:21:03.805484+0000","last_clean":"2024-05-09T03:21:03.805484+0000","last_became_active":"2024-05-09T03:21:02.590786+0000","last_became_peered":"2024-05-09T03:21:02.590786+0000","last_unstale":"2024-05-09T03:21:03.805484+0000","last_undegraded":"2024-05-09T03:21:03.805484+0000","last_fullsized":"2024-05-09T03:21:03.805484+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:57:57.794112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805953+0000","last_change":"2024-05-09T03:21:01.587360+0000","last_active":"2024-05-09T03:21:03.805953+0000","last_peered":"2024-05-09T03:21:03.805953+0000","last_clean":"2024-05-09T03:21:03.805953+0000","last_became_active":"2024-05-09T03:21:01.587059+0000","last_became_peered":"2024-05-09T03:21:01.587059+0000","last_unstale":"2024-05-09T03:21:03.805953+0000","last_undegraded":"2024-05-09T03:21:03.805953+0000","last_fullsized":"2024-05-09T03:21:03.805953+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:05:34.148962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.590256+0000","last_change":"2024-05-09T03:21:02.598051+0000","last_active":"2024-05-09T03:21:03.590256+0000","last_peered":"2024-05-09T03:21:03.590256+0000","last_clean":"2024-05-09T03:21:03.590256+0000","last_became_active":"2024-05-09T03:21:02.597704+0000","last_became_peered":"2024-05-09T03:21:02.597704+0000","last_unstale":"2024-05-09T03:21:03.590256+0000","last_undegraded":"2024-05-09T03:21:03.590256+0000","last_fullsized":"2024-05-09T03:21:03.590256+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:31:01.243163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.805606+0000","last_change":"2024-05-09T03:21:01.584237+0000","last_active":"2024-05-09T03:21:03.805606+0000","last_peered":"2024-05-09T03:21:03.805606+0000","last_clean":"2024-05-09T03:21:03.805606+0000","last_became_active":"2024-05-09T03:21:01.583951+0000","last_became_peered":"2024-05-09T03:21:01.583951+0000","last_unstale":"2024-05-09T03:21:03.805606+0000","last_undegraded":"2024-05-09T03:21:03.805606+0000","last_fullsized":"2024-05-09T03:21:03.805606+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:13:44.001499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806312+0000","last_change":"2024-05-09T03:21:02.592354+0000","last_active":"2024-05-09T03:21:03.806312+0000","last_peered":"2024-05-09T03:21:03.806312+0000","last_clean":"2024-05-09T03:21:03.806312+0000","last_became_active":"2024-05-09T03:21:02.591041+0000","last_became_peered":"2024-05-09T03:21:02.591041+0000","last_unstale":"2024-05-09T03:21:03.806312+0000","last_undegraded":"2024-05-09T03:21:03.806312+0000","last_fullsized":"2024-05-09T03:21:03.806312+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:01.562311+0000","last_clean_scrub_stamp":"2024-05-09T03:21:01.562311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:15:00.583934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"134'1","reported_seq":15,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806586+0000","last_change":"2024-05-09T03:21:01.586379+0000","last_active":"2024-05-09T03:21:03.806586+0000","last_peered":"2024-05-09T03:21:03.806586+0000","last_clean":"2024-05-09T03:21:03.806586+0000","last_became_active":"2024-05-09T03:21:01.586052+0000","last_became_peered":"2024-05-09T03:21:01.586052+0000","last_unstale":"2024-05-09T03:21:03.806586+0000","last_undegraded":"2024-05-09T03:21:03.806586+0000","last_fullsized":"2024-05-09T03:21:03.806586+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":132,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:21:00.559833+0000","last_clean_scrub_stamp":"2024-05-09T03:21:00.559833+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:05:27.664943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":332,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-05-09T03:21:03.806373+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:21:03.806373+0000","last_peered":"2024-05-09T03:21:03.806373+0000","last_clean":"2024-05-09T03:21:03.806373+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:03.806373+0000","last_undegraded":"2024-05-09T03:21:03.806373+0000","last_fullsized":"2024-05-09T03:21:03.806373+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574992,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37628,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334212,"statfs":{"total":96636764160,"available":96598233088,"internally_reserved":0,"allocated":2342912,"data_stored":1432301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1748992,"data_stored":842473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574993,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37044,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334796,"statfs":{"total":96636764160,"available":96598831104,"internally_reserved":0,"allocated":1744896,"data_stored":841891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574992,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1761280,"data_stored":843557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574991,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37056,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334784,"statfs":{"total":96636764160,"available":96598818816,"internally_reserved":0,"allocated":1757184,"data_stored":843455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574992,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41204,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330636,"statfs":{"total":96636764160,"available":96594571264,"internally_reserved":0,"allocated":1744896,"data_stored":841855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41208,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330632,"statfs":{"total":96636764160,"available":96594567168,"internally_reserved":0,"allocated":1740800,"data_stored":841821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574992,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29308,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342532,"statfs":{"total":96636764160,"available":96606752768,"internally_reserved":0,"allocated":2338816,"data_stored":1432771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:21:07.188 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:21:07.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-05-09T03:21:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.345+0000 7f5c3f861640 1 Processor -- start 2024-05-09T03:21:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.345+0000 7f5c3f861640 1 -- start start 2024-05-09T03:21:07.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3f861640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c380b73b0 0x7f5c380b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3f861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c3809cca0 0x7f5c3809d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3f861640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c380cf770 con 0x7f5c380d0160 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3f861640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5c380b9700 con 0x7f5c380b73b0 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3f861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5c380b88f0 con 0x7f5c3809cca0 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3cdfc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c380b73b0 0x7f5c380b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3d5fd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c380d0160 0x7f5c380d2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40934/0 (socket says 172.21.15.46:40934) 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.346+0000 7f5c3d5fd640 1 -- 172.21.15.46:0/3309063487 learned_addr learned my addr 172.21.15.46:0/3309063487 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c3ddfe640 1 --2- 172.21.15.46:0/3309063487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c3809cca0 0x7f5c3809d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/3309063487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c3809cca0 msgr2=0x7f5c3809d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c3cdfc640 1 --2- 172.21.15.46:0/3309063487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c3809cca0 0x7f5c3809d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/3309063487 >> v1:172.21.15.46:6789/0 conn(0x7f5c380d0160 legacy=0x7f5c380d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/3309063487 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c380d1ae0 con 0x7f5c380b73b0 2024-05-09T03:21:07.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c2ffff640 1 -- 172.21.15.46:0/3309063487 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002560208 0 0) 0x7f5c380cf770 con 0x7f5c380d0160 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.347+0000 7f5c3cdfc640 1 --2- 172.21.15.46:0/3309063487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380b77c0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c2802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec213352bedfd1b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c2ffff640 1 -- 172.21.15.46:0/3309063487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28037070 con 0x7f5c380b73b0 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c2ffff640 1 -- 172.21.15.46:0/3309063487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2802ae50 con 0x7f5c380b73b0 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c2ffff640 1 -- 172.21.15.46:0/3309063487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28032660 con 0x7f5c380b73b0 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c3f861640 1 -- 172.21.15.46:0/3309063487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 msgr2=0x7f5c380b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c3f861640 1 --2- 172.21.15.46:0/3309063487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380b77c0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c2802a100 comp rx=0 tx=0).stop 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c3f861640 1 -- 172.21.15.46:0/3309063487 shutdown_connections 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c3f861640 1 --2- 172.21.15.46:0/3309063487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c3809cca0 0x7f5c3809d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c3f861640 1 --2- 172.21.15.46:0/3309063487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380b77c0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.348+0000 7f5c3f861640 1 -- 172.21.15.46:0/3309063487 >> 172.21.15.46:0/3309063487 conn(0x7f5c380d05d0 msgr2=0x7f5c380d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.349+0000 7f5c3f861640 1 -- 172.21.15.46:0/3309063487 shutdown_connections 2024-05-09T03:21:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.349+0000 7f5c3f861640 1 -- 172.21.15.46:0/3309063487 wait complete. 2024-05-09T03:21:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.350+0000 7f5c3f861640 1 Processor -- start 2024-05-09T03:21:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.351+0000 7f5c3f861640 1 -- start start 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3f861640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c3809cca0 0x7f5c381f9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3f861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3f861640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c380d0160 0x7f5c380c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3f861640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5c380cfcf0 con 0x7f5c380b73b0 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3f861640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5c380d2880 con 0x7f5c3809cca0 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3f861640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f5c380b8c70 con 0x7f5c380d0160 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380a1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45704/0 (socket says 172.21.15.46:45704) 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/4132564521 learned_addr learned my addr 172.21.15.46:0/4132564521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3d5fd640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c3809cca0 0x7f5c381f9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3ddfe640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c380d0160 0x7f5c380c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c380d0160 msgr2=0x7f5c380c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c380d0160 0x7f5c380c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c3809cca0 msgr2=0x7f5c381f9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c3809cca0 0x7f5c381f9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.352+0000 7f5c3cdfc640 1 -- 172.21.15.46:0/4132564521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c381f9b40 con 0x7f5c380b73b0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.353+0000 7f5c3cdfc640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380a1460 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f5c280098c0 tx=0x7f5c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.353+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28041020 con 0x7f5c380b73b0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.353+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2803c670 con 0x7f5c380b73b0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.353+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c381fa050 con 0x7f5c380b73b0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.353+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28037040 con 0x7f5c380b73b0 2024-05-09T03:21:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.353+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c381dc900 con 0x7f5c380b73b0 2024-05-09T03:21:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.354+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3806a270 con 0x7f5c380b73b0 2024-05-09T03:21:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.355+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5c28044020 con 0x7f5c380b73b0 2024-05-09T03:21:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.355+0000 7f5c2dffb640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c1c0762d0 0x7f5c1c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.355+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f5c280b78b0 con 0x7f5c380b73b0 2024-05-09T03:21:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.356+0000 7f5c3d5fd640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c1c0762d0 0x7f5c1c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.356+0000 7f5c3d5fd640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c1c0762d0 0x7f5c1c0787e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5c34000f60 tx=0x7f5c34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:07.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.360+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5c2804b020 con 0x7f5c380b73b0 2024-05-09T03:21:07.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.628+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5c380a19a0 con 0x7f5c380b73b0 2024-05-09T03:21:07.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.631+0000 7f5c2dffb640 1 -- 172.21.15.46:0/4132564521 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f5c28089160 con 0x7f5c380b73b0 2024-05-09T03:21:07.631 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-05-09T03:21:07.632 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQA2Pjxmu2knIhAAqrjeOOcwJK3zwGgZNXPdWw== 2024-05-09T03:21:07.632 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-05-09T03:21:07.632 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-05-09T03:21:07.632 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-05-09T03:21:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-05-09T03:21:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c1c0762d0 msgr2=0x7f5c1c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c1c0762d0 0x7f5c1c0787e0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f5c34000f60 tx=0x7f5c34009040 comp rx=0 tx=0).stop 2024-05-09T03:21:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 msgr2=0x7f5c380a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380a1460 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f5c280098c0 tx=0x7f5c280046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 shutdown_connections 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f5c380d0160 0x7f5c380c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5c1c0762d0 0x7f5c1c0787e0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c380b73b0 0x7f5c380a1460 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 --2- 172.21.15.46:0/4132564521 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c3809cca0 0x7f5c381f9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 >> 172.21.15.46:0/4132564521 conn(0x7f5c380d05d0 msgr2=0x7f5c380d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 shutdown_connections 2024-05-09T03:21:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.633+0000 7f5c3f861640 1 -- 172.21.15.46:0/4132564521 wait complete. 2024-05-09T03:21:07.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:07.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.760+0000 7fd77559f640 1 Processor -- start 2024-05-09T03:21:07.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.760+0000 7fd77559f640 1 -- start start 2024-05-09T03:21:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd77559f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7700d0910 0x7fd770160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd77559f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7700d2b40 0x7fd7700b9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd77559f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd770164440 con 0x7fd770163d40 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd77559f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd7700ccaf0 con 0x7fd7700d2b40 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd77559f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd770164ce0 con 0x7fd7700d0910 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd774d9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7700d2b40 0x7fd7700b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd774d9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7700d2b40 0x7fd7700b9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45712/0 (socket says 172.21.15.46:45712) 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd774d9e640 1 -- 172.21.15.46:0/2830758356 learned_addr learned my addr 172.21.15.46:0/2830758356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.761+0000 7fd76ffff640 1 --2- 172.21.15.46:0/2830758356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7700d0910 0x7fd770160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.762+0000 7fd774d9e640 1 -- 172.21.15.46:0/2830758356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7700d0910 msgr2=0x7fd770160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.762+0000 7fd774d9e640 1 --2- 172.21.15.46:0/2830758356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7700d0910 0x7fd770160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.762+0000 7fd774d9e640 1 -- 172.21.15.46:0/2830758356 >> v1:172.21.15.88:6789/0 conn(0x7fd770163d40 legacy=0x7fd7700d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.762+0000 7fd774d9e640 1 -- 172.21.15.46:0/2830758356 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd770164720 con 0x7fd7700d2b40 2024-05-09T03:21:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.762+0000 7fd774d9e640 1 --2- 172.21.15.46:0/2830758356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd7700b9150 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e50 tx=0x7fd76802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11454a2c72fc89bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.762+0000 7fd76effd640 1 -- 172.21.15.46:0/2830758356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768036070 con 0x7fd7700d2b40 2024-05-09T03:21:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd76effd640 1 -- 172.21.15.46:0/2830758356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd768031600 con 0x7fd7700d2b40 2024-05-09T03:21:07.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd76effd640 1 -- 172.21.15.46:0/2830758356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7680319d0 con 0x7fd7700d2b40 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 -- 172.21.15.46:0/2830758356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 msgr2=0x7fd7700b9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 --2- 172.21.15.46:0/2830758356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd7700b9150 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fd768009e50 tx=0x7fd76802a680 comp rx=0 tx=0).stop 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 -- 172.21.15.46:0/2830758356 shutdown_connections 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 --2- 172.21.15.46:0/2830758356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd7700b9150 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 --2- 172.21.15.46:0/2830758356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd7700d0910 0x7fd770160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 -- 172.21.15.46:0/2830758356 >> 172.21.15.46:0/2830758356 conn(0x7fd77006e300 msgr2=0x7fd77006e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 -- 172.21.15.46:0/2830758356 shutdown_connections 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.763+0000 7fd77559f640 1 -- 172.21.15.46:0/2830758356 wait complete. 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.765+0000 7fd77559f640 1 Processor -- start 2024-05-09T03:21:07.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.765+0000 7fd77559f640 1 -- start start 2024-05-09T03:21:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.766+0000 7fd77559f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7700d0910 0x7fd7700cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.766+0000 7fd77559f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd770161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.766+0000 7fd77559f640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd770163d40 0x7fd77015e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.766+0000 7fd77559f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd770164150 con 0x7fd7700d2b40 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.766+0000 7fd77559f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd7700d43b0 con 0x7fd7700d0910 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.766+0000 7fd77559f640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd7700d48d0 con 0x7fd770163d40 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd770161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd770161ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45728/0 (socket says 172.21.15.46:45728) 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 -- 172.21.15.46:0/2952460613 learned_addr learned my addr 172.21.15.46:0/2952460613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76ffff640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7700d0910 0x7fd7700cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd774d9e640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd770163d40 0x7fd77015e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 -- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd770163d40 msgr2=0x7fd77015e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd770163d40 0x7fd77015e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 -- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7700d0910 msgr2=0x7fd7700cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7700d0910 0x7fd7700cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 -- 172.21.15.46:0/2952460613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7701627a0 con 0x7fd7700d2b40 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76f7fe640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd770161ab0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fd76400c370 tx=0x7fd76400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd764017070 con 0x7fd7700d2b40 2024-05-09T03:21:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7640052d0 con 0x7fd7700d2b40 2024-05-09T03:21:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd770162a80 con 0x7fd7700d2b40 2024-05-09T03:21:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd764003830 con 0x7fd7700d2b40 2024-05-09T03:21:07.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.767+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd770163040 con 0x7fd7700d2b40 2024-05-09T03:21:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.768+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd770076c70 con 0x7fd7700d2b40 2024-05-09T03:21:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.769+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd76401a070 con 0x7fd7700d2b40 2024-05-09T03:21:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.770+0000 7fd76cff9640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd75c076300 0x7fd75c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.770+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fd764096810 con 0x7fd7700d2b40 2024-05-09T03:21:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.770+0000 7fd76ffff640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd75c076300 0x7fd75c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:07.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.770+0000 7fd76ffff640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd75c076300 0x7fd75c078810 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd7700bbd40 tx=0x7fd760008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:07.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:07.774+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd76405fb10 con 0x7fd7700d2b40 2024-05-09T03:21:08.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.002+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7700d4060 con 0x7fd7700d2b40 2024-05-09T03:21:08.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.003+0000 7fd76cff9640 1 -- 172.21.15.46:0/2952460613 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) ==== 95+0+5188 (secure 0 0 0) 0x7fd764068150 con 0x7fd7700d2b40 2024-05-09T03:21:08.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:08.004 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":353,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:04.982418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10644},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10644":{"gid":10644,"name":"c","rank":0,"incarnation":349,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2230300620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2230300620},{"type":"v1","addr":"172.21.15.46:6835","nonce":2230300620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10644,"qdb_cluster":[10644]},"id":7}]} 2024-05-09T03:21:08.004 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 353 2024-05-09T03:21:08.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd75c076300 msgr2=0x7fd75c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd75c076300 0x7fd75c078810 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd7700bbd40 tx=0x7fd760008040 comp rx=0 tx=0).stop 2024-05-09T03:21:08.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 msgr2=0x7fd770161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd770161ab0 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fd76400c370 tx=0x7fd76400c830 comp rx=0 tx=0).stop 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 shutdown_connections 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd770163d40 0x7fd77015e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd75c076300 0x7fd75c078810 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7700d2b40 0x7fd770161ab0 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 --2- 172.21.15.46:0/2952460613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd7700d0910 0x7fd7700cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 >> 172.21.15.46:0/2952460613 conn(0x7fd77006e300 msgr2=0x7fd7700b97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 shutdown_connections 2024-05-09T03:21:08.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.005+0000 7fd77559f640 1 -- 172.21.15.46:0/2952460613 wait complete. 2024-05-09T03:21:08.121 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:21:08.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:21:08.202 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:21:08.201+0000 7fee9bfc7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:21:08.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 353, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:21:01.965784+0000', 'modified': '2024-05-09T03:21:04.982418+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10644}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10644': {'gid': 10644, 'name': 'c', 'rank': 0, 'incarnation': 349, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/2230300620', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 2230300620}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2230300620}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10644, 'qdb_cluster': [10644]} 2024-05-09T03:21:08.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:21:08.421 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:21:08.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.538+0000 7f1af8538640 1 Processor -- start 2024-05-09T03:21:08.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.538+0000 7f1af8538640 1 -- start start 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af8538640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1af0163eb0 0x7f1af01642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af8538640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00d2a70 0x7f1af00d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af8538640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af00d33c0 con 0x7f1af00bb770 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af8538640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1af00d4bb0 con 0x7f1af0163eb0 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af8538640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1af0164900 con 0x7f1af00d2a70 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af5ad3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1af0163eb0 0x7f1af01642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af5ad3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1af0163eb0 0x7f1af01642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45740/0 (socket says 172.21.15.46:45740) 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.539+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2886640556 learned_addr learned my addr 172.21.15.46:0/2886640556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00d2a70 0x7f1af00d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2886640556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00d2a70 msgr2=0x7f1af00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00d2a70 0x7f1af00d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2886640556 >> v1:172.21.15.88:6789/0 conn(0x7f1af00bb770 legacy=0x7f1af00d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2886640556 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af0164be0 con 0x7f1af0163eb0 2024-05-09T03:21:08.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00d2a70 0x7f1af00d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.540+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af0163eb0 0x7f1af01642c0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009870 tx=0x7f1ae002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8169fd872561ae50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af52d2640 1 -- 172.21.15.46:0/2886640556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0037070 con 0x7f1af0163eb0 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af52d2640 1 -- 172.21.15.46:0/2886640556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae0032440 con 0x7f1af0163eb0 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af52d2640 1 -- 172.21.15.46:0/2886640556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0032810 con 0x7f1af0163eb0 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 -- 172.21.15.46:0/2886640556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af0163eb0 msgr2=0x7f1af01642c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af0163eb0 0x7f1af01642c0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009870 tx=0x7f1ae002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 -- 172.21.15.46:0/2886640556 shutdown_connections 2024-05-09T03:21:08.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00d2a70 0x7f1af00d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 --2- 172.21.15.46:0/2886640556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af0163eb0 0x7f1af01642c0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 -- 172.21.15.46:0/2886640556 >> 172.21.15.46:0/2886640556 conn(0x7f1af006e370 msgr2=0x7f1af006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.541+0000 7f1af8538640 1 -- 172.21.15.46:0/2886640556 shutdown_connections 2024-05-09T03:21:08.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.542+0000 7f1af8538640 1 -- 172.21.15.46:0/2886640556 wait complete. 2024-05-09T03:21:08.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.543+0000 7f1af8538640 1 Processor -- start 2024-05-09T03:21:08.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.543+0000 7f1af8538640 1 -- start start 2024-05-09T03:21:08.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af8538640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00bb770 0x7f1af01fa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af8538640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 0x7f1af01fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af8538640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1af0163eb0 0x7f1af01496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af8538640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1af00d0bc0 con 0x7f1af00d2a70 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af8538640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1af00bbb80 con 0x7f1af0163eb0 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af8538640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1af00d0d40 con 0x7f1af00bb770 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 0x7f1af01fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 0x7f1af01fa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45756/0 (socket says 172.21.15.46:45756) 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2385454672 learned_addr learned my addr 172.21.15.46:0/2385454672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af62d4640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00bb770 0x7f1af01fa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1af0163eb0 0x7f1af01496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00bb770 msgr2=0x7f1af01fa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00bb770 0x7f1af01fa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1af0163eb0 msgr2=0x7f1af01496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1af0163eb0 0x7f1af01496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 -- 172.21.15.46:0/2385454672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af01eb9a0 con 0x7f1af00d2a70 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af62d4640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00bb770 0x7f1af01fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1af0163eb0 0x7f1af01496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 0x7f1af01fa8f0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009870 tx=0x7f1ae00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1ae6ffd640 1 -- 172.21.15.46:0/2385454672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0041070 con 0x7f1af00d2a70 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1ae6ffd640 1 -- 172.21.15.46:0/2385454672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae003c870 con 0x7f1af00d2a70 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.545+0000 7f1ae6ffd640 1 -- 172.21.15.46:0/2385454672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0037040 con 0x7f1af00d2a70 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.546+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af01ebc80 con 0x7f1af00d2a70 2024-05-09T03:21:08.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.546+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af01ec240 con 0x7f1af00d2a70 2024-05-09T03:21:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.547+0000 7f1ae6ffd640 1 -- 172.21.15.46:0/2385454672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ae0044070 con 0x7f1af00d2a70 2024-05-09T03:21:08.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.548+0000 7f1ae6ffd640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ad4076300 0x7f1ad4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.548+0000 7f1ae6ffd640 1 -- 172.21.15.46:0/2385454672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f1ae00b82d0 con 0x7f1af00d2a70 2024-05-09T03:21:08.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.548+0000 7f1af62d4640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ad4076300 0x7f1ad4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.549+0000 7f1aff3ec740 1 Processor -- start 2024-05-09T03:21:08.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.549+0000 7f1aff3ec740 1 -- start start 2024-05-09T03:21:08.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.549+0000 7f1af62d4640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ad4076300 0x7f1ad4078810 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1aec006fd0 tx=0x7f1aec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e085270 0x55f23e454eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e470120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f23e38d4e0 con 0x55f23e05c3b0 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f23e328ff0 con 0x55f23e064590 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f23e3894a0 con 0x55f23e085270 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1af5ad3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1af5ad3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45766/0 (socket says 172.21.15.46:45766) 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1af5ad3640 1 -- 172.21.15.46:0/4184781879 learned_addr learned my addr 172.21.15.46:0/4184781879 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1af62d4640 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e085270 0x55f23e454eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.550+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e470120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af5ad3640 1 -- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e085270 msgr2=0x55f23e454eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e085270 0x55f23e454eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af5ad3640 1 -- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 msgr2=0x55f23e470120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e470120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af5ad3640 1 -- 172.21.15.46:0/4184781879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23e30f720 con 0x55f23e05c3b0 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e470120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46fbe0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0035040 tx=0x7f1ae007db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5daa5e1641661cae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1ad3fff640 1 -- 172.21.15.46:0/4184781879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae004b040 con 0x55f23e05c3b0 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1ad3fff640 1 -- 172.21.15.46:0/4184781879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae00b80e0 con 0x55f23e05c3b0 2024-05-09T03:21:08.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.551+0000 7f1ad3fff640 1 -- 172.21.15.46:0/4184781879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0081b00 con 0x55f23e05c3b0 2024-05-09T03:21:08.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 -- 172.21.15.46:0/4184781879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 msgr2=0x55f23e46fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46fbe0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0035040 tx=0x7f1ae007db80 comp rx=0 tx=0).stop 2024-05-09T03:21:08.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 -- 172.21.15.46:0/4184781879 shutdown_connections 2024-05-09T03:21:08.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e470120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46fbe0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/4184781879 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e085270 0x55f23e454eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 -- 172.21.15.46:0/4184781879 >> 172.21.15.46:0/4184781879 conn(0x55f23e0793c0 msgr2=0x55f23e42a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 -- 172.21.15.46:0/4184781879 shutdown_connections 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.552+0000 7f1aff3ec740 1 -- 172.21.15.46:0/4184781879 wait complete. 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.553+0000 7f1aff3ec740 1 -- start start 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.553+0000 7f1aff3ec740 1 Processor -- start 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e05c3b0 0x55f23e483060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e467920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e451580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f23e38d4e0 con 0x55f23e085270 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f23e314760 con 0x55f23e05c3b0 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f23e468670 con 0x55f23e064590 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.554+0000 7f1aff3ec740 10 client.0 _ll_register_callbacks cb 0x55f23e3374b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e451580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e451580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45776/0 (socket says 172.21.15.46:45776) 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 -- 172.21.15.46:0/3717112931 learned_addr learned my addr 172.21.15.46:0/3717112931 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af62d4640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e05c3b0 0x55f23e483060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e467920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 -- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 msgr2=0x55f23e467920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e467920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 -- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e05c3b0 msgr2=0x55f23e483060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e05c3b0 0x55f23e483060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 -- 172.21.15.46:0/3717112931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23e355bd0 con 0x55f23e085270 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e451580 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8010b90 tx=0x7f1ae800e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f111271b77f13f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1ad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1ad17fa640 1 -- 172.21.15.46:0/3717112931 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8003da0 con 0x55f23e085270 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae00b80e0 con 0x55f23e085270 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1ad17fa640 1 -- 172.21.15.46:0/3717112931 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f1ae801aaf0 con 0x55f23e085270 2024-05-09T03:21:08.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1ad17fa640 1 -- 172.21.15.46:0/3717112931 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=353) ==== 24+0+0 (secure 0 0 0) 0x7f1ae801acf0 con 0x55f23e085270 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1aff3ec740 10 client.10764 fetch_fsmap learned FSMap version 353 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.555+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=353,monmap=2+}) -- 0x55f23e33b280 con 0x55f23e085270 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1ad17fa640 1 -- 172.21.15.46:0/3717112931 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae00b80e0 con 0x55f23e085270 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1ad17fa640 1 -- 172.21.15.46:0/3717112931 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 353) ==== 3494+0+0 (secure 0 0 0) 0x7f1ae8021050 con 0x55f23e085270 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1aff3ec740 10 client.10764 fetch_fsmap finished waiting for FSMap version 353 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1aff3ec740 10 client.10764 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1aff3ec740 20 client.10764 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e043e40 0x55f23e423ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1aff3ec740 4 client.10764 mds_command: new command op to 10644 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 --> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23e39b0f0 con 0x55f23e043e40 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1af62d4640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e043e40 0x55f23e423ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1ad17fa640 10 client.10764 ms_handle_connect on v2:172.21.15.46:6833/2230300620 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.556+0000 7f1af62d4640 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e043e40 0x55f23e423ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed2f7e23dd7706 server_cookie=a81b0dc1efdef9b4 in_seq=0 out_seq=0 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1ad17fa640 1 -- 172.21.15.46:0/3717112931 <== mds.0 v2:172.21.15.46:6833/2230300620 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f23e39b0f0 con 0x55f23e043e40 2024-05-09T03:21:08.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1ad17fa640 10 client.10764 handle_command_reply: tid=0 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 client.10764 shutdown 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 msgr2=0x55f23e451580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e451580 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8010b90 tx=0x7f1ae800e750 comp rx=0 tx=0).stop 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 shutdown_connections 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e043e40 0x55f23e423ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e451580 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e467920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1ad17fa640 0 client.10764 ms_handle_reset on v2:172.21.15.46:6833/2230300620 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/3717112931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e05c3b0 0x55f23e483060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.557+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 >> 172.21.15.46:0/3717112931 conn(0x55f23e0793c0 msgr2=0x55f23e42a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.558+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 shutdown_connections 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.558+0000 7f1aff3ec740 1 -- 172.21.15.46:0/3717112931 wait complete. 2024-05-09T03:21:08.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.558+0000 7f1aff3ec740 20 client.10764 trim_cache size 0 max 16384 2024-05-09T03:21:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.577+0000 7f1aff3ec740 1 Processor -- start 2024-05-09T03:21:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.577+0000 7f1aff3ec740 1 -- start start 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e05c3b0 0x55f23e467920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e424310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e424850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0202f30 con 0x55f23e085270 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1af014a200 con 0x55f23e064590 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f1af01f7780 con 0x55f23e05c3b0 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e424850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e424850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45784/0 (socket says 172.21.15.46:45784) 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 -- 172.21.15.46:0/1137091484 learned_addr learned my addr 172.21.15.46:0/1137091484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e424310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af62d4640 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e05c3b0 0x55f23e467920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 -- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e05c3b0 msgr2=0x55f23e467920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e05c3b0 0x55f23e467920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 -- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 msgr2=0x55f23e424310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.578+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e424310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.579+0000 7f1af6ad5640 1 -- 172.21.15.46:0/1137091484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23e428560 con 0x55f23e085270 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.579+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e424310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.579+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e424850 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f1ae801bfd0 tx=0x7f1ae8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a270444a6afe5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.579+0000 7f1ad37fe640 1 -- 172.21.15.46:0/1137091484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8028030 con 0x55f23e085270 2024-05-09T03:21:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.579+0000 7f1ad37fe640 1 -- 172.21.15.46:0/1137091484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae800ec50 con 0x55f23e085270 2024-05-09T03:21:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.579+0000 7f1ad37fe640 1 -- 172.21.15.46:0/1137091484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8010030 con 0x55f23e085270 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1137091484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 msgr2=0x55f23e424850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e424850 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7f1ae801bfd0 tx=0x7f1ae8013000 comp rx=0 tx=0).stop 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1137091484 shutdown_connections 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e085270 0x55f23e424850 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e064590 0x55f23e424310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1137091484 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e05c3b0 0x55f23e467920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1137091484 >> 172.21.15.46:0/1137091484 conn(0x55f23e0793c0 msgr2=0x55f23e46a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1137091484 shutdown_connections 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.580+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1137091484 wait complete. 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.581+0000 7f1aff3ec740 1 -- start start 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.581+0000 7f1aff3ec740 1 Processor -- start 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e46c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e085270 0x55f23e4efc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f23e467eb0 con 0x55f23e05c3b0 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f23e44f1b0 con 0x55f23e085270 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55f23e44ec60 con 0x55f23e064590 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1aff3ec740 10 client.0 _ll_register_callbacks cb 0x7f1ae8007740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1af62d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1af62d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45800/0 (socket says 172.21.15.46:45800) 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.582+0000 7f1af62d4640 1 -- 172.21.15.46:0/1239087869 learned_addr learned my addr 172.21.15.46:0/1239087869 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e46c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e085270 0x55f23e4efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af62d4640 1 -- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 msgr2=0x55f23e46c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af62d4640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e46c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af62d4640 1 -- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e085270 msgr2=0x55f23e4efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af62d4640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e085270 0x55f23e4efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af62d4640 1 -- 172.21.15.46:0/1239087869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23e0f0850 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e46c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af6ad5640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e085270 0x55f23e4efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1af62d4640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46c2b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009150 tx=0x7f1aec004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77cf51b7b8eb5da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1ad17fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1ad17fa640 1 -- 172.21.15.46:0/1239087869 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec014b50 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1ad17fa640 1 -- 172.21.15.46:0/1239087869 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f1aec00f040 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae800ec50 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1ad17fa640 1 -- 172.21.15.46:0/1239087869 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=353) ==== 24+0+0 (secure 0 0 0) 0x7f1aec038050 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 10 client.10770 fetch_fsmap learned FSMap version 353 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=353,monmap=2+}) -- 0x55f23e1676b0 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1ad17fa640 1 -- 172.21.15.46:0/1239087869 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae800ec50 con 0x55f23e05c3b0 2024-05-09T03:21:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1ad17fa640 1 -- 172.21.15.46:0/1239087869 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 353) ==== 3494+0+0 (secure 0 0 0) 0x7f1aec03b020 con 0x55f23e05c3b0 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 10 client.10770 fetch_fsmap finished waiting for FSMap version 353 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 10 client.10770 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 20 client.10770 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.583+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e4f0160 0x55f23e46ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1aff3ec740 4 client.10770 mds_command: new command op to 10644 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 --> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] -- command(tid 0: {"prefix": "status"}) -- 0x55f23e394f60 con 0x55f23e4f0160 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e4f0160 0x55f23e46ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1ad17fa640 10 client.10770 ms_handle_connect on v2:172.21.15.46:6833/2230300620 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1af5ad3640 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e4f0160 0x55f23e46ef80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=784fa8cb0a686f55 server_cookie=83bf38d66f5e428a in_seq=0 out_seq=0 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1ad17fa640 1 -- 172.21.15.46:0/1239087869 <== mds.0 v2:172.21.15.46:6833/2230300620 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55f23e394f60 con 0x55f23e4f0160 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1ad17fa640 10 client.10770 handle_command_reply: tid=0 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.584+0000 7f1aff3ec740 1 client.10770 shutdown 2024-05-09T03:21:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 msgr2=0x55f23e46c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46c2b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009150 tx=0x7f1aec004680 comp rx=0 tx=0).stop 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 shutdown_connections 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:6833/2230300620,v1:172.21.15.46:6835/2230300620] conn(0x55f23e4f0160 0x55f23e46ef80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23e085270 0x55f23e4efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55f23e064590 0x55f23e46c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 --2- 172.21.15.46:0/1239087869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f23e05c3b0 0x55f23e46c2b0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1ad17fa640 0 client.10770 ms_handle_reset on v2:172.21.15.46:6833/2230300620 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 >> 172.21.15.46:0/1239087869 conn(0x55f23e0793c0 msgr2=0x55f23e467660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 shutdown_connections 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 1 -- 172.21.15.46:0/1239087869 wait complete. 2024-05-09T03:21:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1aff3ec740 20 client.10770 trim_cache size 0 max 16384 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10644, 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 6.0157025239999999, 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 353, 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 135, 2024-05-09T03:21:08.592 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 135, 2024-05-09T03:21:08.593 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.290017103 2024-05-09T03:21:08.593 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ad4076300 msgr2=0x7f1ad4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ad4076300 0x7f1ad4078810 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f1aec006fd0 tx=0x7f1aec008040 comp rx=0 tx=0).stop 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 msgr2=0x7f1af01fa8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 0x7f1af01fa8f0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009870 tx=0x7f1ae00046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 shutdown_connections 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1af0163eb0 0x7f1af01496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f1ad4076300 0x7f1ad4078810 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1af00d2a70 0x7f1af01fa8f0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 --2- 172.21.15.46:0/2385454672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f1af00bb770 0x7f1af01fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.585+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 >> 172.21.15.46:0/2385454672 conn(0x7f1af006e370 msgr2=0x7f1af00b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.586+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 shutdown_connections 2024-05-09T03:21:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.586+0000 7f1af8538640 1 -- 172.21.15.46:0/2385454672 wait complete. 2024-05-09T03:21:08.598 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:21:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.713+0000 7f2c26e0c640 1 Processor -- start 2024-05-09T03:21:08.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.713+0000 7f2c26e0c640 1 -- start start 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c26e0c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2c200b9200 0x7f2c2008d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c26e0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 0x7f2c200d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c26e0c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c200b4ee0 con 0x7f2c200d20a0 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c26e0c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2c200b5460 con 0x7f2c200b9200 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c26e0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c200cfac0 con 0x7f2c2008d6e0 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c253a9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2c200d20a0 0x7f2c20162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40942/0 (socket says 172.21.15.46:40942) 2024-05-09T03:21:08.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c253a9640 1 -- 172.21.15.46:0/3315722577 learned_addr learned my addr 172.21.15.46:0/3315722577 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046597718 0 0) 0x7f2c200b4ee0 con 0x7f2c200d20a0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c1ffff640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 0x7f2c200d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/3315722577 >> v2:172.21.15.88:3300/0 conn(0x7f2c200b9200 0x7f2c2008d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.714+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c08003520 con 0x7f2c200d20a0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2869900100 0 0) 0x7f2c08003520 con 0x7f2c200d20a0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 msgr2=0x7f2c200d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 0x7f2c200d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 >> v2:172.21.15.88:3300/0 conn(0x7f2c200b9200 msgr2=0x7f2c2008d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 --2- 172.21.15.46:0/3315722577 >> v2:172.21.15.88:3300/0 conn(0x7f2c200b9200 0x7f2c2008d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c200b62e0 con 0x7f2c200d20a0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f2c18001a90 con 0x7f2c200d20a0 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 >> v1:172.21.15.46:6789/0 conn(0x7f2c200d20a0 legacy=0x7f2c20162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c08003a00 0x7f2c08003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 0x7f2c08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08004d40 0x7f2c08005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c200cfac0 con 0x7f2c08003a00 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0800ba70 con 0x7f2c08004d40 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0800bbf0 con 0x7f2c2008d6e0 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c08003a00 0x7f2c08003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c253a9640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08004d40 0x7f2c08005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c1ffff640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 0x7f2c08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c24ba8640 1 -- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 msgr2=0x7f2c08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2008d6e0 0x7f2c08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c24ba8640 1 -- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08004d40 msgr2=0x7f2c08005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08004d40 0x7f2c08005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.715+0000 7f2c24ba8640 1 -- 172.21.15.46:0/3315722577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c08003520 con 0x7f2c08003a00 2024-05-09T03:21:08.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.716+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c08003a00 0x7f2c08003e70 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009ac0 tx=0x7f2c08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b04598cc915c998 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.716+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10036070 con 0x7f2c08003a00 2024-05-09T03:21:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.716+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c10031400 con 0x7f2c08003a00 2024-05-09T03:21:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.716+0000 7f2c1f7fe640 1 -- 172.21.15.46:0/3315722577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10009690 con 0x7f2c08003a00 2024-05-09T03:21:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c26e0c640 1 -- 172.21.15.46:0/3315722577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c08003a00 msgr2=0x7f2c08003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/3315722577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c08003a00 0x7f2c08003e70 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009ac0 tx=0x7f2c08005830 comp rx=0 tx=0).stop 2024-05-09T03:21:08.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c253a9640 1 -- 172.21.15.46:0/3315722577 reap_dead start 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c26e0c640 1 -- 172.21.15.46:0/3315722577 shutdown_connections 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c26e0c640 1 -- 172.21.15.46:0/3315722577 >> 172.21.15.46:0/3315722577 conn(0x7f2c200b56e0 msgr2=0x7f2c200b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c26e0c640 1 -- 172.21.15.46:0/3315722577 shutdown_connections 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.717+0000 7f2c26e0c640 1 -- 172.21.15.46:0/3315722577 wait complete. 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.719+0000 7f2c26e0c640 1 Processor -- start 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.719+0000 7f2c26e0c640 1 -- start start 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c26e0c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08003a00 0x7f2c201584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c26e0c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2015a7d0 0x7f2c20154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c26e0c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 0x7f2c201e9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c26e0c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c200cf750 con 0x7f2c201e8f70 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c26e0c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2c200cf470 con 0x7f2c08003a00 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c26e0c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2c200b4a20 con 0x7f2c2015a7d0 2024-05-09T03:21:08.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c253a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 0x7f2c201e9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c253a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 0x7f2c201e9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45818/0 (socket says 172.21.15.46:45818) 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.720+0000 7f2c253a9640 1 -- 172.21.15.46:0/29518700 learned_addr learned my addr 172.21.15.46:0/29518700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08003a00 0x7f2c201584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c1ffff640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2015a7d0 0x7f2c20154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c253a9640 1 -- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2015a7d0 msgr2=0x7f2c20154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c253a9640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2015a7d0 0x7f2c20154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c253a9640 1 -- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08003a00 msgr2=0x7f2c201584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c253a9640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08003a00 0x7f2c201584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c253a9640 1 -- 172.21.15.46:0/29518700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c201e98c0 con 0x7f2c201e8f70 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08003a00 0x7f2c201584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c253a9640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 0x7f2c201e9380 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f2c200b50b0 tx=0x7f2c18007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1801a070 con 0x7f2c201e8f70 2024-05-09T03:21:08.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c1800ac40 con 0x7f2c201e8f70 2024-05-09T03:21:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c20158b30 con 0x7f2c201e8f70 2024-05-09T03:21:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c180133b0 con 0x7f2c201e8f70 2024-05-09T03:21:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.721+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2015cab0 con 0x7f2c201e8f70 2024-05-09T03:21:08.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.722+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c2006a270 con 0x7f2c201e8f70 2024-05-09T03:21:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.723+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2c1801d070 con 0x7f2c201e8f70 2024-05-09T03:21:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.724+0000 7f2c1d7fa640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c0c0762d0 0x7f2c0c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.724+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f2c18097380 con 0x7f2c201e8f70 2024-05-09T03:21:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.724+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c0c0762d0 0x7f2c0c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:08.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.724+0000 7f2c24ba8640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c0c0762d0 0x7f2c0c0787e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009ac0 tx=0x7f2c10008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:08.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.728+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2c201e98c0 con 0x7f2c201e8f70 2024-05-09T03:21:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.958+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2c200b69c0 con 0x7f2c201e8f70 2024-05-09T03:21:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.958+0000 7f2c1d7fa640 1 -- 172.21.15.46:0/29518700 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v353) ==== 53+0+73 (secure 0 0 0) 0x7f2c18060d70 con 0x7f2c201e8f70 2024-05-09T03:21:08.959 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:21:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c0c0762d0 msgr2=0x7f2c0c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c0c0762d0 0x7f2c0c0787e0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009ac0 tx=0x7f2c10008030 comp rx=0 tx=0).stop 2024-05-09T03:21:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 msgr2=0x7f2c201e9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 0x7f2c201e9380 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f2c200b50b0 tx=0x7f2c18007bd0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 shutdown_connections 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2c0c0762d0 0x7f2c0c0787e0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c201e8f70 0x7f2c201e9380 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2c2015a7d0 0x7f2c20154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 --2- 172.21.15.46:0/29518700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c08003a00 0x7f2c201584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 >> 172.21.15.46:0/29518700 conn(0x7f2c200b56e0 msgr2=0x7f2c200b85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 shutdown_connections 2024-05-09T03:21:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:08.960+0000 7f2c26e0c640 1 -- 172.21.15.46:0/29518700 wait complete. 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:21:08.973 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:21:08.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:08.974 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:21:08.990 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:21:08.991 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41849sec preferred_lft 41849sec 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.992 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:08.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:08.994 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:21:09.055 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:21:09.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:09.057 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:21:09.113 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:21:09.114 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:21:09.114 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:21:09.114 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:21:09.114 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:21:09.114 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:21:09.114 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:21:09.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:21:09.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:21:09.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:21:09.115 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:21:09.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:21:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:09.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:09.243 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:21:09.259 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:21:09.259 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:21:09.260 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:21:09.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:09.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:21:09.325 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:21:09.325 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:21:09.325 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:09.381 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:21:09.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:21:09.382 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:09.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:21:09.504 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:21:09.561 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:21:09.562 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:21:09.563 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:09.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:09.564 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:21:09.617 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:21:09.617 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:21:09.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:21:09.685 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:21:09.744 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:21:09.745 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:09.746 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:09.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:09.747 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:21:09.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:21:09.755+0000 7fcf330cc480 -1 init, newargv = 0x7fcf20001f00 newargc=15 2024-05-09T03:21:09.755 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[88140]: starting ceph client 2024-05-09T03:21:09.760 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[88140]: starting fuse 2024-05-09T03:21:09.771 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:21:09.771 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:21:09.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:21:09.772 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:21:09.773 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:21:09.892 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.88140.asok 2024-05-09T03:21:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:09.903 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.88140.asok 2024-05-09T03:21:09.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:09.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.88140.asok status 2024-05-09T03:21:10.050 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "88140", 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10782, 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:21:10.051 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10782 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3511329982 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3511329982 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10782 192.168.0.1:0/3511329982", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3511329982", 2024-05-09T03:21:10.052 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:21:10.053 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 353, 2024-05-09T03:21:10.053 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 135, 2024-05-09T03:21:10.053 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:21:10.053 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:21:10.053 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:21:10.053 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:10.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.059 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:21:10.059 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:10.076 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:21:10.076 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:10.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:10.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.144 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir1) 2024-05-09T03:21:10.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.206 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir1/file1) 2024-05-09T03:21:10.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.265 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir2) 2024-05-09T03:21:10.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.323 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir2/file2) 2024-05-09T03:21:10.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.385 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-05-09T03:21:10.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.443 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir1/file1 testdir/hardlink1) 2024-05-09T03:21:10.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:10.505 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir2/file2 testdir/hardlink2) 2024-05-09T03:21:10.565 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:21:10.565 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:21:10.565 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:21:10.565 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:10.565 DEBUG:teuthology.orchestra.run.smithi046:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir1/file1").st_ino) 2024-05-09T03:21:10.565 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:21:10.673 INFO:teuthology.orchestra.run.smithi046.stdout:1099511627777 2024-05-09T03:21:10.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:10.682 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:21:10.682 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:21:10.682 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:21:10.682 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:10.682 DEBUG:teuthology.orchestra.run.smithi046:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir2/file2").st_ino) 2024-05-09T03:21:10.682 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:21:10.791 INFO:teuthology.orchestra.run.smithi046.stdout:1099511628279 2024-05-09T03:21:10.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:10.800 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2024-05-09T03:21:10.800 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:21:10.800 DEBUG:teuthology.orchestra.run.smithi046:> import stat 2024-05-09T03:21:10.800 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:10.800 DEBUG:teuthology.orchestra.run.smithi046:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir2").st_ino) 2024-05-09T03:21:10.801 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:21:10.909 INFO:teuthology.orchestra.run.smithi046.stdout:1099511628278 2024-05-09T03:21:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:10.918 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:10.918 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:10.975 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:21:11.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.133+0000 7fb23609a640 1 Processor -- start 2024-05-09T03:21:11.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.133+0000 7fb23609a640 1 -- start start 2024-05-09T03:21:11.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb23609a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300bb770 0x7fb2300d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb23609a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb230163eb0 0x7fb2301642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb23609a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2300d2a70 0x7fb2300d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb23609a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb2300d4bb0 con 0x7fb2300d2a70 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb23609a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb2300d33c0 con 0x7fb230163eb0 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb23609a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb230164900 con 0x7fb2300bb770 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb22ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2300d2a70 0x7fb2300d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb22ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2300d2a70 0x7fb2300d2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45824/0 (socket says 172.21.15.46:45824) 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.134+0000 7fb22ffff640 1 -- 172.21.15.46:0/3575267489 learned_addr learned my addr 172.21.15.46:0/3575267489 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22f7fe640 1 --2- 172.21.15.46:0/3575267489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300bb770 0x7fb2300d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22effd640 1 --2- 172.21.15.46:0/3575267489 >> v2:172.21.15.88:3300/0 conn(0x7fb230163eb0 0x7fb2301642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22ffff640 1 -- 172.21.15.46:0/3575267489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300bb770 msgr2=0x7fb2300d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22ffff640 1 --2- 172.21.15.46:0/3575267489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300bb770 0x7fb2300d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22ffff640 1 -- 172.21.15.46:0/3575267489 >> v2:172.21.15.88:3300/0 conn(0x7fb230163eb0 msgr2=0x7fb2301642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22ffff640 1 --2- 172.21.15.46:0/3575267489 >> v2:172.21.15.88:3300/0 conn(0x7fb230163eb0 0x7fb2301642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22ffff640 1 -- 172.21.15.46:0/3575267489 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb230164be0 con 0x7fb2300d2a70 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.135+0000 7fb22ffff640 1 --2- 172.21.15.46:0/3575267489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2300d2a70 0x7fb2300d2e80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fb2300b88b0 tx=0x7fb22802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbaabd3e9e47b8f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb22e7fc640 1 -- 172.21.15.46:0/3575267489 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228036070 con 0x7fb2300d2a70 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb22e7fc640 1 -- 172.21.15.46:0/3575267489 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb228031600 con 0x7fb2300d2a70 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb22e7fc640 1 -- 172.21.15.46:0/3575267489 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2280319d0 con 0x7fb2300d2a70 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 -- 172.21.15.46:0/3575267489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2300d2a70 msgr2=0x7fb2300d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 --2- 172.21.15.46:0/3575267489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2300d2a70 0x7fb2300d2e80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fb2300b88b0 tx=0x7fb22802a6a0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 -- 172.21.15.46:0/3575267489 shutdown_connections 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 --2- 172.21.15.46:0/3575267489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2300d2a70 0x7fb2300d2e80 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 --2- 172.21.15.46:0/3575267489 >> v2:172.21.15.88:3300/0 conn(0x7fb230163eb0 0x7fb2301642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 --2- 172.21.15.46:0/3575267489 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300bb770 0x7fb2300d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 -- 172.21.15.46:0/3575267489 >> 172.21.15.46:0/3575267489 conn(0x7fb23006e370 msgr2=0x7fb23006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:11.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 -- 172.21.15.46:0/3575267489 shutdown_connections 2024-05-09T03:21:11.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.136+0000 7fb23609a640 1 -- 172.21.15.46:0/3575267489 wait complete. 2024-05-09T03:21:11.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.138+0000 7fb23609a640 1 Processor -- start 2024-05-09T03:21:11.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.138+0000 7fb23609a640 1 -- start start 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb23609a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2300bb770 0x7fb2301f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb23609a640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300d2a70 0x7fb230155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb23609a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 0x7fb2300cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb23609a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2300bbb80 con 0x7fb230163eb0 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb23609a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb2300d5580 con 0x7fb2300bb770 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb23609a640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb2300d0bc0 con 0x7fb2300d2a70 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 0x7fb2300cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 0x7fb2300cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45838/0 (socket says 172.21.15.46:45838) 2024-05-09T03:21:11.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 -- 172.21.15.46:0/3061539623 learned_addr learned my addr 172.21.15.46:0/3061539623 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22effd640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300d2a70 0x7fb230155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22f7fe640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2300bb770 0x7fb2301f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 -- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300d2a70 msgr2=0x7fb230155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300d2a70 0x7fb230155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 -- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2300bb770 msgr2=0x7fb2301f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2300bb770 0x7fb2301f9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 -- 172.21.15.46:0/3061539623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2301fa940 con 0x7fb230163eb0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22effd640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300d2a70 0x7fb230155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22f7fe640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2300bb770 0x7fb2301f9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb22ffff640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 0x7fb2300cde10 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fb22802ac90 tx=0x7fb228004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22803f020 con 0x7fb230163eb0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.140+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb228037040 con 0x7fb230163eb0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.141+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2301fac20 con 0x7fb230163eb0 2024-05-09T03:21:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.141+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228036070 con 0x7fb230163eb0 2024-05-09T03:21:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.141+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2301fa3c0 con 0x7fb230163eb0 2024-05-09T03:21:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.141+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2301fa6b0 con 0x7fb230163eb0 2024-05-09T03:21:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.142+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb228042020 con 0x7fb230163eb0 2024-05-09T03:21:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.143+0000 7fb217fff640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb21c076300 0x7fb21c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.143+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fb2280b7ac0 con 0x7fb230163eb0 2024-05-09T03:21:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.143+0000 7fb22f7fe640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb21c076300 0x7fb21c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.144+0000 7fb22f7fe640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb21c076300 0x7fb21c078810 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:11.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.148+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb228049030 con 0x7fb230163eb0 2024-05-09T03:21:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.354+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb2300d5360 con 0x7fb230163eb0 2024-05-09T03:21:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.355+0000 7fb217fff640 1 -- 172.21.15.46:0/3061539623 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v135) ==== 80+0+0 (secure 0 0 0) 0x7fb228081420 con 0x7fb230163eb0 2024-05-09T03:21:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb21c076300 msgr2=0x7fb21c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb21c076300 0x7fb21c078810 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220008040 comp rx=0 tx=0).stop 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 msgr2=0x7fb2300cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 0x7fb2300cde10 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fb22802ac90 tx=0x7fb228004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 shutdown_connections 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb21c076300 0x7fb21c078810 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb230163eb0 0x7fb2300cde10 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb2300d2a70 0x7fb230155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 --2- 172.21.15.46:0/3061539623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2300bb770 0x7fb2301f9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 >> 172.21.15.46:0/3061539623 conn(0x7fb23006e370 msgr2=0x7fb2300b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 shutdown_connections 2024-05-09T03:21:11.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.357+0000 7fb23609a640 1 -- 172.21.15.46:0/3061539623 wait complete. 2024-05-09T03:21:11.369 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:11.369 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:11.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:21:11.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.541+0000 7f7ec76bd640 1 Processor -- start 2024-05-09T03:21:11.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.541+0000 7f7ec76bd640 1 -- start start 2024-05-09T03:21:11.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.542+0000 7f7ec76bd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ec00b7e70 0x7f7ec00b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.542+0000 7f7ec76bd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ec01605b0 0x7f7ec00b97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec76bd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec00d3400 con 0x7f7ec00d2e90 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec76bd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7ec00d4a60 con 0x7f7ec01605b0 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec76bd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7ec00d4d40 con 0x7f7ec00b7e70 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ec01605b0 0x7f7ec00b97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ec01605b0 0x7f7ec00b97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45846/0 (socket says 172.21.15.46:45846) 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 -- 172.21.15.46:0/55861215 learned_addr learned my addr 172.21.15.46:0/55861215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec4c58640 1 --2- 172.21.15.46:0/55861215 >> v2:172.21.15.88:3300/0 conn(0x7f7ec00b7e70 0x7f7ec00b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 -- 172.21.15.46:0/55861215 >> v2:172.21.15.88:3300/0 conn(0x7f7ec00b7e70 msgr2=0x7f7ec00b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 --2- 172.21.15.46:0/55861215 >> v2:172.21.15.88:3300/0 conn(0x7f7ec00b7e70 0x7f7ec00b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 -- 172.21.15.46:0/55861215 >> v1:172.21.15.88:6789/0 conn(0x7f7ec00d2e90 legacy=0x7f7ec00b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec5c5a640 1 -- 172.21.15.46:0/55861215 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec00d5020 con 0x7f7ec01605b0 2024-05-09T03:21:11.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.543+0000 7f7ec4c58640 1 --2- 172.21.15.46:0/55861215 >> v2:172.21.15.88:3300/0 conn(0x7f7ec00b7e70 0x7f7ec00b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:11.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.544+0000 7f7ec5c5a640 1 --2- 172.21.15.46:0/55861215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec01605b0 0x7f7ec00b97e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f7ec008e880 tx=0x7f7eb802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3582e87ef4d0ebe6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:11.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.544+0000 7f7eb7fff640 1 -- 172.21.15.46:0/55861215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8037070 con 0x7f7ec01605b0 2024-05-09T03:21:11.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.544+0000 7f7eb7fff640 1 -- 172.21.15.46:0/55861215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eb8032440 con 0x7f7ec01605b0 2024-05-09T03:21:11.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.544+0000 7f7eb7fff640 1 -- 172.21.15.46:0/55861215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8032810 con 0x7f7ec01605b0 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 -- 172.21.15.46:0/55861215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec01605b0 msgr2=0x7f7ec00b97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/55861215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec01605b0 0x7f7ec00b97e0 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f7ec008e880 tx=0x7f7eb802a200 comp rx=0 tx=0).stop 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 -- 172.21.15.46:0/55861215 shutdown_connections 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/55861215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec01605b0 0x7f7ec00b97e0 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/55861215 >> v2:172.21.15.88:3300/0 conn(0x7f7ec00b7e70 0x7f7ec00b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 -- 172.21.15.46:0/55861215 >> 172.21.15.46:0/55861215 conn(0x7f7ec008d390 msgr2=0x7f7ec008d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 -- 172.21.15.46:0/55861215 shutdown_connections 2024-05-09T03:21:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.545+0000 7f7ec76bd640 1 -- 172.21.15.46:0/55861215 wait complete. 2024-05-09T03:21:11.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.547+0000 7f7ec76bd640 1 Processor -- start 2024-05-09T03:21:11.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.547+0000 7f7ec76bd640 1 -- start start 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec76bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 0x7f7ec00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec76bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ec01605b0 0x7f7ec0150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec76bd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ec01510a0 0x7f7ec01514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec76bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec00b4d80 con 0x7f7ec00d2e90 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec76bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec00b4c00 con 0x7f7ec01605b0 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec76bd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec00b4fc0 con 0x7f7ec01510a0 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 0x7f7ec00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 0x7f7ec00cb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45850/0 (socket says 172.21.15.46:45850) 2024-05-09T03:21:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 -- 172.21.15.46:0/1674621736 learned_addr learned my addr 172.21.15.46:0/1674621736 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5c5a640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ec01510a0 0x7f7ec01514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec4c58640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ec01605b0 0x7f7ec0150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 -- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ec01510a0 msgr2=0x7f7ec01514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ec01510a0 0x7f7ec01514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 -- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ec01605b0 msgr2=0x7f7ec0150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ec01605b0 0x7f7ec0150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.548+0000 7f7ec5459640 1 -- 172.21.15.46:0/1674621736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec01519f0 con 0x7f7ec00d2e90 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.549+0000 7f7ec5459640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 0x7f7ec00cb560 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00c6f0 tx=0x7f7ebc00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.549+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc016020 con 0x7f7ec00d2e90 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.549+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc006b50 con 0x7f7ec00d2e90 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.549+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc003990 con 0x7f7ec00d2e90 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.549+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec0150c50 con 0x7f7ec00d2e90 2024-05-09T03:21:11.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.549+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec01e99d0 con 0x7f7ec00d2e90 2024-05-09T03:21:11.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.550+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec006a270 con 0x7f7ec00d2e90 2024-05-09T03:21:11.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.551+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7ebc003b30 con 0x7f7ec00d2e90 2024-05-09T03:21:11.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.551+0000 7f7eb5ffb640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ea40762d0 0x7f7ea40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:11.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.551+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f7ebc097190 con 0x7f7ec00d2e90 2024-05-09T03:21:11.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.552+0000 7f7ec4c58640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ea40762d0 0x7f7ea40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:11.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.552+0000 7f7ec4c58640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ea40762d0 0x7f7ea40787e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7ec01eab60 tx=0x7f7eb0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.556+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7ebc020020 con 0x7f7ec00d2e90 2024-05-09T03:21:11.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.758+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7ec0150e80 con 0x7f7ec00d2e90 2024-05-09T03:21:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.758+0000 7f7eb5ffb640 1 -- 172.21.15.46:0/1674621736 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v135) ==== 80+0+0 (secure 0 0 0) 0x7f7ec01519f0 con 0x7f7ec00d2e90 2024-05-09T03:21:11.759 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-05-09T03:21:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ea40762d0 msgr2=0x7f7ea40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ea40762d0 0x7f7ea40787e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7ec01eab60 tx=0x7f7eb0008040 comp rx=0 tx=0).stop 2024-05-09T03:21:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 msgr2=0x7f7ec00cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 0x7f7ec00cb560 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00c6f0 tx=0x7f7ebc00cbb0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 shutdown_connections 2024-05-09T03:21:11.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ec01510a0 0x7f7ec01514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ec01605b0 0x7f7ec0150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7ea40762d0 0x7f7ea40787e0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 --2- 172.21.15.46:0/1674621736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec00d2e90 0x7f7ec00cb560 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 >> 172.21.15.46:0/1674621736 conn(0x7f7ec008d390 msgr2=0x7f7ec015e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 shutdown_connections 2024-05-09T03:21:11.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:11.760+0000 7f7ec76bd640 1 -- 172.21.15.46:0/1674621736 wait complete. 2024-05-09T03:21:11.772 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:21:11.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:11.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:11.797 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[88140]: fuse finished with error 0 and tester_r 0 2024-05-09T03:21:11.798 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:21:17.900 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:21:17.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:17.900 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:21:17.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:18.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.089+0000 7fcdf7d86640 1 Processor -- start 2024-05-09T03:21:18.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.089+0000 7fcdf7d86640 1 -- start start 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf7d86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcdf015f080 0x7fcdf01658f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf7d86640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf0168590 con 0x7fcdf00d0de0 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf7d86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf0168870 con 0x7fcdf00d2e40 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf7d86640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf00d0b30 con 0x7fcdf015f080 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf5321640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcdf00d0de0 0x7fcdf00d2730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43490/0 (socket says 172.21.15.46:43490) 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf5321640 1 -- 172.21.15.46:0/2901138991 learned_addr learned my addr 172.21.15.46:0/2901138991 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf5b22640 1 --2- 172.21.15.46:0/2901138991 >> v2:172.21.15.88:3300/0 conn(0x7fcdf015f080 0x7fcdf01658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546310640 0 0) 0x7fcdf0168590 con 0x7fcdf00d0de0 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.091+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcddc003520 con 0x7fcdf00d0de0 2024-05-09T03:21:18.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1468926769 0 0) 0x7fcddc003520 con 0x7fcdf00d0de0 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 >> v2:172.21.15.88:3300/0 conn(0x7fcdf015f080 msgr2=0x7fcdf01658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 --2- 172.21.15.46:0/2901138991 >> v2:172.21.15.88:3300/0 conn(0x7fcdf015f080 0x7fcdf01658f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 >> v1:172.21.15.88:6789/0 conn(0x7fcdf00d2e40 legacy=0x7fcdf00d3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf0168b50 con 0x7fcdf00d0de0 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7fcde00016f0 con 0x7fcdf00d0de0 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 >> v1:172.21.15.46:6789/0 conn(0x7fcdf00d0de0 legacy=0x7fcdf00d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcddc003a50 0x7fcddc003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcddc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcddc005250 0x7fcddc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf00d0b30 con 0x7fcddc004800 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc00c1e0 con 0x7fcddc005250 2024-05-09T03:21:18.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcddc00c3e0 con 0x7fcddc003a50 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5321640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcddc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5b22640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcddc003a50 0x7fcddc003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf6323640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcddc005250 0x7fcddc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5321640 1 -- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcddc003a50 msgr2=0x7fcddc003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5321640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcddc003a50 0x7fcddc003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5321640 1 -- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcddc005250 msgr2=0x7fcddc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5321640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcddc005250 0x7fcddc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf6323640 1 -- 172.21.15.46:0/2901138991 reap_dead start 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.092+0000 7fcdf5321640 1 -- 172.21.15.46:0/2901138991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc003520 con 0x7fcddc004800 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.093+0000 7fcdf5321640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcddc004c10 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fcde000c240 tx=0x7fcddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf88461bb947bbd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.093+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde0007d30 con 0x7fcddc004800 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.093+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde0005af0 con 0x7fcddc004800 2024-05-09T03:21:18.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.093+0000 7fcdf4b20640 1 -- 172.21.15.46:0/2901138991 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde0007070 con 0x7fcddc004800 2024-05-09T03:21:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 -- 172.21.15.46:0/2901138991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 msgr2=0x7fcddc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcddc004c10 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fcde000c240 tx=0x7fcddc005830 comp rx=0 tx=0).stop 2024-05-09T03:21:18.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 -- 172.21.15.46:0/2901138991 shutdown_connections 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/2901138991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcddc004c10 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 -- 172.21.15.46:0/2901138991 >> 172.21.15.46:0/2901138991 conn(0x7fcdf0073db0 msgr2=0x7fcdf00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 -- 172.21.15.46:0/2901138991 shutdown_connections 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.094+0000 7fcdf7d86640 1 -- 172.21.15.46:0/2901138991 wait complete. 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.096+0000 7fcdf7d86640 1 Processor -- start 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.096+0000 7fcdf7d86640 1 -- start start 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf7d86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcdf00cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf7d86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdf00c8180 0x7fcdf00d4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf7d86640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdf00c8590 0x7fcdf00d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf7d86640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf015f080 con 0x7fcddc004800 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf7d86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf01691a0 con 0x7fcdf00c8180 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf7d86640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf015f200 con 0x7fcdf00c8590 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf5b22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcdf00cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf5b22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcdf00cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46344/0 (socket says 172.21.15.46:46344) 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.097+0000 7fcdf5b22640 1 -- 172.21.15.46:0/1632758957 learned_addr learned my addr 172.21.15.46:0/1632758957 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf5b22640 1 -- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdf00c8590 msgr2=0x7fcdf00d4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf5b22640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdf00c8590 0x7fcdf00d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf5b22640 1 -- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdf00c8180 msgr2=0x7fcdf00d4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf5b22640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdf00c8180 0x7fcdf00d4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf5b22640 1 -- 172.21.15.46:0/1632758957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf01fe170 con 0x7fcddc004800 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf5b22640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcdf00cdad0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fcdec011900 tx=0x7fcdec011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00faf0 con 0x7fcddc004800 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdec00fcb0 con 0x7fcddc004800 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf00ce4b0 con 0x7fcddc004800 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec014680 con 0x7fcddc004800 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.098+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf01fe6f0 con 0x7fcddc004800 2024-05-09T03:21:18.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.099+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf008cfd0 con 0x7fcddc004800 2024-05-09T03:21:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.100+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcdec012040 con 0x7fcddc004800 2024-05-09T03:21:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.100+0000 7fcde67fc640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcdd40762d0 0x7fcdd40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.100+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7fcdec097c90 con 0x7fcddc004800 2024-05-09T03:21:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.101+0000 7fcdf5321640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcdd40762d0 0x7fcdd40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.101+0000 7fcdf5321640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcdd40762d0 0x7fcdd40787e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcde0032000 tx=0x7fcde002c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.105+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcdf008cfd0 con 0x7fcddc004800 2024-05-09T03:21:18.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.333+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdf006a270 con 0x7fcddc004800 2024-05-09T03:21:18.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.334+0000 7fcde67fc640 1 -- 172.21.15.46:0/1632758957 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 353 v353) ==== 95+0+5188 (secure 0 0 0) 0x7fcdec019030 con 0x7fcddc004800 2024-05-09T03:21:18.335 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:18.335 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":353,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":353,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:04.982418+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10644},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10644":{"gid":10644,"name":"c","rank":0,"incarnation":349,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/2230300620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2230300620},{"type":"v1","addr":"172.21.15.46:6835","nonce":2230300620}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10644,"qdb_cluster":[10644]},"id":7}]} 2024-05-09T03:21:18.335 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 353 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcdd40762d0 msgr2=0x7fcdd40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcdd40762d0 0x7fcdd40787e0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcde0032000 tx=0x7fcde002c000 comp rx=0 tx=0).stop 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 msgr2=0x7fcdf00cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcdf00cdad0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fcdec011900 tx=0x7fcdec011dc0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 shutdown_connections 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcdf00c8590 0x7fcdf00d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcdf00c8180 0x7fcdf00d4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcdd40762d0 0x7fcdd40787e0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 --2- 172.21.15.46:0/1632758957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcddc004800 0x7fcdf00cdad0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 >> 172.21.15.46:0/1632758957 conn(0x7fcdf0073db0 msgr2=0x7fcdf015e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 shutdown_connections 2024-05-09T03:21:18.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.336+0000 7fcdf7d86640 1 -- 172.21.15.46:0/1632758957 wait complete. 2024-05-09T03:21:18.348 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-05-09T03:21:18.489 INFO:teuthology.orchestra.run.smithi046.stdout:{"return_code":0} 2024-05-09T03:21:18.489 DEBUG:tasks.cephfs.filesystem:_json_asok output { "return_code": 0 } 2024-05-09T03:21:18.489 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:21:18.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.606+0000 7f9781a42640 1 Processor -- start 2024-05-09T03:21:18.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.606+0000 7f9781a42640 1 -- start start 2024-05-09T03:21:18.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9781a42640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f977c1642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9781a42640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977c164900 con 0x7f977c0bb770 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9781a42640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f977c164be0 con 0x7f977c0d2a70 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9781a42640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f977c0d4bb0 con 0x7f977c163eb0 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f977affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f977c0bb770 0x7f977c0d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43504/0 (socket says 172.21.15.46:43504) 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f977affd640 1 -- 172.21.15.46:0/1815328763 learned_addr learned my addr 172.21.15.46:0/1815328763 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f977a7fc640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f977c1642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053691366 0 0) 0x7f977c164900 con 0x7f977c0bb770 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9764003520 con 0x7f977c0bb770 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1623732723 0 0) 0x7f977c164be0 con 0x7f977c0d2a70 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f977c164900 con 0x7f977c0d2a70 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3526697874 0 0) 0x7f9764003520 con 0x7f977c0bb770 2024-05-09T03:21:18.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 msgr2=0x7f977c1642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.608+0000 7f9779ffb640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f977c1642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 >> v1:172.21.15.88:6789/0 conn(0x7f977c0d2a70 legacy=0x7f977c0d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977c0d4840 con 0x7f977c0bb770 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f976c001710 con 0x7f977c0bb770 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977a7fc640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f977c1642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 >> v1:172.21.15.46:6789/0 conn(0x7f977c0bb770 legacy=0x7f977c0d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f9764003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97640041c0 0x7f9764004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764004d90 0x7f9764005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f977c0d4bb0 con 0x7f9764004d90 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f976400bc20 con 0x7f97640041c0 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f976400be20 con 0x7f977c163eb0 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977b7fe640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764004d90 0x7f9764005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977affd640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f9764003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977a7fc640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97640041c0 0x7f9764004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977b7fe640 1 -- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 msgr2=0x7f9764003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977b7fe640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c163eb0 0x7f9764003c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977b7fe640 1 -- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97640041c0 msgr2=0x7f9764004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977b7fe640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97640041c0 0x7f9764004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.609+0000 7f977b7fe640 1 -- 172.21.15.46:0/1815328763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9764003520 con 0x7f9764004d90 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.610+0000 7f977b7fe640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764004d90 0x7f9764005220 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f9774002000 tx=0x7f9764005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13764cfda9984f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.610+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774001c20 con 0x7f9764004d90 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.610+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9774006d70 con 0x7f9764004d90 2024-05-09T03:21:18.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.610+0000 7f9779ffb640 1 -- 172.21.15.46:0/1815328763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774003bb0 con 0x7f9764004d90 2024-05-09T03:21:18.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f9781a42640 1 -- 172.21.15.46:0/1815328763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764004d90 msgr2=0x7f9764005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f9781a42640 1 --2- 172.21.15.46:0/1815328763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9764004d90 0x7f9764005220 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f9774002000 tx=0x7f9764005830 comp rx=0 tx=0).stop 2024-05-09T03:21:18.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f977b7fe640 1 -- 172.21.15.46:0/1815328763 reap_dead start 2024-05-09T03:21:18.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f9781a42640 1 -- 172.21.15.46:0/1815328763 shutdown_connections 2024-05-09T03:21:18.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f9781a42640 1 -- 172.21.15.46:0/1815328763 >> 172.21.15.46:0/1815328763 conn(0x7f977c06e370 msgr2=0x7f977c06e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f9781a42640 1 -- 172.21.15.46:0/1815328763 shutdown_connections 2024-05-09T03:21:18.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.611+0000 7f9781a42640 1 -- 172.21.15.46:0/1815328763 wait complete. 2024-05-09T03:21:18.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.613+0000 7f9781a42640 1 Processor -- start 2024-05-09T03:21:18.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.613+0000 7f9781a42640 1 -- start start 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f9781a42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9764004d90 0x7f977c14c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f9781a42640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c14b9b0 0x7f977c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f9781a42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 0x7f977c0cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f9781a42640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f977c0d0d40 con 0x7f977c14e6a0 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f9781a42640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f977c0d0bc0 con 0x7f9764004d90 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f9781a42640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f977c1642c0 con 0x7f977c14b9b0 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f977b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 0x7f977c0cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f977b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 0x7f977c0cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46358/0 (socket says 172.21.15.46:46358) 2024-05-09T03:21:18.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f977b7fe640 1 -- 172.21.15.46:0/3269428505 learned_addr learned my addr 172.21.15.46:0/3269428505 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.614+0000 7f977a7fc640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c14b9b0 0x7f977c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977affd640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9764004d90 0x7f977c14c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977b7fe640 1 -- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c14b9b0 msgr2=0x7f977c14e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977b7fe640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c14b9b0 0x7f977c14e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977b7fe640 1 -- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9764004d90 msgr2=0x7f977c14c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977b7fe640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9764004d90 0x7f977c14c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977b7fe640 1 -- 172.21.15.46:0/3269428505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977c14eab0 con 0x7f977c14e6a0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977affd640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9764004d90 0x7f977c14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f977b7fe640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 0x7f977c0cf950 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f9774002120 tx=0x7f9774001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774007d90 con 0x7f977c14e6a0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9774040070 con 0x7f977c14e6a0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977c0ce8b0 con 0x7f977c14e6a0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9774005520 con 0x7f977c14e6a0 2024-05-09T03:21:18.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.615+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f977c0cee70 con 0x7f977c14e6a0 2024-05-09T03:21:18.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.616+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f977c1ec850 con 0x7f977c14e6a0 2024-05-09T03:21:18.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.617+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9774039070 con 0x7f977c14e6a0 2024-05-09T03:21:18.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.617+0000 7f9763fff640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97680762d0 0x7f97680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.617+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5709+0+0 (secure 0 0 0) 0x7f97740b72e0 con 0x7f977c14e6a0 2024-05-09T03:21:18.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.617+0000 7f977affd640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97680762d0 0x7f97680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.618+0000 7f977affd640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97680762d0 0x7f97680787e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f977c0b8880 tx=0x7f976c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.622+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9774080c40 con 0x7f977c14e6a0 2024-05-09T03:21:18.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.860+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f977c15f540 con 0x7f977c14e6a0 2024-05-09T03:21:18.862 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:21:18.860+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:21:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.862+0000 7f9763fff640 1 -- 172.21.15.46:0/3269428505 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v354) ==== 161+0+0 (secure 0 0 0) 0x7f9774080540 con 0x7f977c14e6a0 2024-05-09T03:21:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:21:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97680762d0 msgr2=0x7f97680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97680762d0 0x7f97680787e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f977c0b8880 tx=0x7f976c009040 comp rx=0 tx=0).stop 2024-05-09T03:21:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 msgr2=0x7f977c0cf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 0x7f977c0cf950 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f9774002120 tx=0x7f9774001670 comp rx=0 tx=0).stop 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 shutdown_connections 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f97680762d0 0x7f97680787e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f977c14e6a0 0x7f977c0cf950 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f977c14b9b0 0x7f977c14e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 --2- 172.21.15.46:0/3269428505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9764004d90 0x7f977c14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.865+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 >> 172.21.15.46:0/3269428505 conn(0x7f977c06e370 msgr2=0x7f977c0b4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.866+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 shutdown_connections 2024-05-09T03:21:18.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.866+0000 7f9781a42640 1 -- 172.21.15.46:0/3269428505 wait complete. 2024-05-09T03:21:18.869 INFO:tasks.ceph.mds.c.smithi046.stderr: -5237> 2024-05-09T03:21:08.201+0000 7fee9bfc7640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:21:18.878 DEBUG:teuthology.orchestra.run.smithi046:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-05-09T03:21:18.929 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:21:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.935+0000 7fdd4d5df280 1 -- start start 2024-05-09T03:21:18.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.935+0000 7fdd4d5df280 1 Processor -- start 2024-05-09T03:21:18.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 0x561f51558b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51555200 0x561f51558580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554d80 0x561f51558000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561f522b8000 con 0x561f51557600 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561f522b8180 con 0x561f51555200 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561f522b8300 con 0x561f51554d80 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.936+0000 7fdd4bd8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 0x561f51558b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 0x561f51558b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46400/0 (socket says 172.21.15.46:46400) 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/158696847 learned_addr learned my addr 172.21.15.46:0/158696847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/158696847 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554d80 msgr2=0x561f51558000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554d80 0x561f51558000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/158696847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51555200 msgr2=0x561f51558580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51555200 0x561f51558580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/158696847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4d4aa780 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 0x561f51558b00 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x561f522db950 tx=0x561f515a7ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69245c45ffbc3254 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd49586640 1 -- 172.21.15.46:0/158696847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49ab60 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd49586640 1 -- 172.21.15.46:0/158696847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561f4d4a0700 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f4d4aa000 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=0}) -- 0x561f4d4ab680 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4d5df280 4 waiting for MDS map... 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd49586640 1 -- 172.21.15.46:0/158696847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49a820 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd49586640 1 -- 172.21.15.46:0/158696847 <== mon.0 v2:172.21.15.46:3300/0 4 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x561f5237e600 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd49586640 1 -- 172.21.15.46:0/158696847 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 354) ==== 2885+0+0 (secure 0 0 0) 0x561f514ff180 con 0x561f51557600 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4d5df280 4 Got MDS map 354 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.937+0000 7fdd4d5df280 10 main: JournalTool::main 2024-05-09T03:21:18.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.938+0000 7fdd4d5df280 4 main: JournalTool: connecting to RADOS... 2024-05-09T03:21:18.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.939+0000 7fdd4d5df280 1 Processor -- start 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.939+0000 7fdd4d5df280 1 -- start start 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561f522b8480 con 0x561f51554480 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561f522b8600 con 0x561f51556880 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561f522b8780 con 0x561f51554000 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46406/0 (socket says 172.21.15.46:46406) 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 -- 172.21.15.46:0/925016310 learned_addr learned my addr 172.21.15.46:0/925016310 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4b58a640 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 -- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554000 msgr2=0x561f51559b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 -- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51556880 msgr2=0x561f5155a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 -- 172.21.15.46:0/925016310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4d4ab860 con 0x561f51554480 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x561f52394b40 tx=0x561f52396f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282e031b28e6709 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd47582640 1 -- 172.21.15.46:0/925016310 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49ab60 con 0x561f51554480 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd47582640 1 -- 172.21.15.46:0/925016310 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561f4d4a0c40 con 0x561f51554480 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.940+0000 7fdd47582640 1 -- 172.21.15.46:0/925016310 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49a820 con 0x561f51554480 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 -- 172.21.15.46:0/925016310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 msgr2=0x561f51559600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x561f52394b40 tx=0x561f52396f00 comp rx=0 tx=0).stop 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 -- 172.21.15.46:0/925016310 shutdown_connections 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/925016310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 -- 172.21.15.46:0/925016310 >> 172.21.15.46:0/925016310 conn(0x561f51554900 msgr2=0x561f515ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 -- 172.21.15.46:0/925016310 shutdown_connections 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.941+0000 7fdd4d5df280 1 -- 172.21.15.46:0/925016310 wait complete. 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.942+0000 7fdd4d5df280 1 Processor -- start 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.942+0000 7fdd4d5df280 1 -- start start 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561f522b9e00 con 0x561f51554480 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561f522b9c80 con 0x561f51554000 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561f522b9b00 con 0x561f51556880 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4bd8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4bd8b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46408/0 (socket says 172.21.15.46:46408) 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/3920540566 learned_addr learned my addr 172.21.15.46:0/3920540566 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.943+0000 7fdd4b58a640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51556880 msgr2=0x561f5155a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51554000 msgr2=0x561f51559b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/3920540566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4d4aa780 con 0x561f51554480 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x561f52395890 tx=0x561f515a68a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd45d7f640 1 -- 172.21.15.46:0/3920540566 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49a4e0 con 0x561f51554480 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd45d7f640 1 -- 172.21.15.46:0/3920540566 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561f4d4a01c0 con 0x561f51554480 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561f4d4aa000 con 0x561f51554480 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd45d7f640 1 -- 172.21.15.46:0/3920540566 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49ab60 con 0x561f51554480 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f4d4aba40 con 0x561f51554480 2024-05-09T03:21:18.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.944+0000 7fdd4d5df280 4 main: JournalTool: resolving pool 14 2024-05-09T03:21:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.946+0000 7fdd45d7f640 1 -- 172.21.15.46:0/3920540566 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x561f523e2c00 con 0x561f51554480 2024-05-09T03:21:18.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.946+0000 7fdd45d7f640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x561f51555b00 0x561f5155a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.946+0000 7fdd45d7f640 1 -- 172.21.15.46:0/3920540566 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x561f523e0c00 con 0x561f51554480 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.946+0000 7fdd4d5df280 4 main: JournalTool: creating IoCtx.. 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.946+0000 7fdd4d5df280 4 main: Executing for rank 0 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.946+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x561f51555b00 0x561f5155a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.947+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x561f51555b00 0x561f5155a680 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x561f52394b40 tx=0x561f51568000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.947+0000 7fdd4d5df280 1 -- start start 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.947+0000 7fdd4d5df280 1 Processor -- start 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 0x561f5155ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51557a80 0x561f5155b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4d5df280 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51557180 0x561f5155b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561f522b9800 con 0x561f51555680 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561f522b9680 con 0x561f51557a80 2024-05-09T03:21:18.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4d5df280 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561f522b8480 con 0x561f51557180 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4c58c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 0x561f5155ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4c58c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 0x561f5155ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46420/0 (socket says 172.21.15.46:46420) 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 learned_addr learned my addr 172.21.15.46:0/2661740241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51557a80 0x561f5155b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.948+0000 7fdd4b58a640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51557180 0x561f5155b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51557180 msgr2=0x561f5155b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51557180 0x561f5155b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51557a80 msgr2=0x561f5155b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51557a80 0x561f5155b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4d4abc20 con 0x561f51555680 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 0x561f5155ac00 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x561f5182fb90 tx=0x561f515a6a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc948f14282bfa1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd42d79640 1 -- 172.21.15.46:0/2661740241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f4d49a1a0 con 0x561f51555680 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd42d79640 1 -- 172.21.15.46:0/2661740241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x561f4d4a08c0 con 0x561f51555680 2024-05-09T03:21:18.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd42d79640 1 -- 172.21.15.46:0/2661740241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x561f517ce1a0 con 0x561f51555680 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f4d4ab860 con 0x561f51555680 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=0}) -- 0x561f4d4abc20 con 0x561f51555680 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4d5df280 4 waiting for MDS map... 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd42d79640 1 -- 172.21.15.46:0/2661740241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x561f51900000 con 0x561f51555680 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd42d79640 1 -- 172.21.15.46:0/2661740241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 354) ==== 2885+0+0 (secure 0 0 0) 0x561f514ff500 con 0x561f51555680 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.949+0000 7fdd4d5df280 4 Got MDS map 354 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.950+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x561f51556400 0x561f51916000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.950+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136) -- 0x561f5190b800 con 0x561f51556400 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.950+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x561f51556400 0x561f51916000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.950+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x561f51556400 0x561f51916000 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=2cb3265318663e24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.951+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/2661740241 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x561f5159e000 con 0x561f51556400 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.951+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x561f5191e000 0x561f51916580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.951+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136) -- 0x561f5190bc00 con 0x561f5191e000 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.951+0000 7fdd4b58a640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x561f5191e000 0x561f51916580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.952+0000 7fdd4b58a640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x561f5191e000 0x561f51916580 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=51845dde4c4d167 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.952+0000 7fdd4b58a640 1 -- 172.21.15.46:0/2661740241 <== osd.4 v2:172.21.15.88:6824/902225877 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv22 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x561f5159e500 con 0x561f5191e000 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.952+0000 7fdd41576640 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e136) -- 0x561f51926000 con 0x561f5191e000 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.952+0000 7fdd41576640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x561f51921600 0x561f51917b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.952+0000 7fdd41576640 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e136) -- 0x561f51926400 con 0x561f51921600 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.953+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x561f51921600 0x561f51917b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.953+0000 7fdd4b58a640 1 -- 172.21.15.46:0/2661740241 <== osd.4 v2:172.21.15.88:6824/902225877 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv17 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x561f5159e500 con 0x561f5191e000 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.953+0000 7fdd4c58c640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x561f51921600 0x561f51917b80 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=3f86142a7914583e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.953+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e780 con 0x561f51921600 2024-05-09T03:21:18.958 INFO:teuthology.orchestra.run.smithi046.stdout:old journal was 4220277~892 2024-05-09T03:21:18.959 INFO:teuthology.orchestra.run.smithi046.stdout:new journal start will be 8388608 (4167439 bytes past old end) 2024-05-09T03:21:18.959 INFO:teuthology.orchestra.run.smithi046.stdout:writing journal head 2024-05-09T03:21:18.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.953+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- osd_op(unknown.0.0:5 14.14 14:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f51926c00 con 0x561f5191e000 2024-05-09T03:21:18.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4b58a640 1 -- 172.21.15.46:0/2661740241 <== osd.4 v2:172.21.15.88:6824/902225877 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v136'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x561f5159e500 con 0x561f5191e000 2024-05-09T03:21:18.960 INFO:teuthology.orchestra.run.smithi046.stdout:writing EResetJournal entry 2024-05-09T03:21:18.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:6 14.1b 14:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f51927000 con 0x561f51921600 2024-05-09T03:21:18.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561f5191e480 0x561f51916b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:7 14.b 14:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f51927400 con 0x561f5191e480 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:8 14.1f 14:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f51926000 con 0x561f5191e480 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:9 14.1a 14:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f5190bc00 con 0x561f5191e480 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- osd_op(unknown.0.0:10 14.1e 14:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f4d475800 con 0x561f51556400 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- osd_op(unknown.0.0:11 14.1d 14:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f4d475c00 con 0x561f5191e480 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.955+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561f5191e480 0x561f51916b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.956+0000 7fdd4bd8b640 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561f5191e480 0x561f51916b00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=c1440ba8afa5eef0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.956+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== osd_op_reply(6 200.00000002 [delete] v136'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e780 con 0x561f51921600 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.957+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/2661740241 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== osd_op_reply(10 200.00000006 [delete] v136'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e000 con 0x561f51556400 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.957+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/2661740241 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== osd_op_reply(7 200.00000003 [delete] v136'4 uv3 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e000 con 0x561f5191e480 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.957+0000 7fdd41576640 1 -- 172.21.15.46:0/2661740241 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:12 14.1b 14:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x561f51926800 con 0x561f51921600 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.957+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/2661740241 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== osd_op_reply(9 200.00000005 [delete] v136'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e000 con 0x561f5191e480 2024-05-09T03:21:18.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.957+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/2661740241 <== osd.0 v2:172.21.15.46:6800/1118951116 3 ==== osd_op_reply(11 200.00000007 [delete] v136'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e000 con 0x561f5191e480 2024-05-09T03:21:18.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4bd8b640 1 -- 172.21.15.46:0/2661740241 <== osd.0 v2:172.21.15.46:6800/1118951116 4 ==== osd_op_reply(8 200.00000004 [delete] v136'7 uv6 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x561f5159e000 con 0x561f5191e480 2024-05-09T03:21:18.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4c58c640 1 -- 172.21.15.46:0/2661740241 <== osd.1 v2:172.21.15.46:6824/1118146182 3 ==== osd_op_reply(12 200.00000002 [write 0~48] v136'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x561f5159e780 con 0x561f51921600 2024-05-09T03:21:18.963 INFO:teuthology.orchestra.run.smithi046.stdout:done 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561f5191e480 msgr2=0x561f51916b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x561f5191e480 0x561f51916b00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x561f51921600 msgr2=0x561f51917b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x561f51921600 0x561f51917b80 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x561f5191e000 msgr2=0x561f51916580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x561f5191e000 0x561f51916580 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x561f51556400 msgr2=0x561f51916000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x561f51556400 0x561f51916000 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4b58a640 1 -- 172.21.15.46:0/2661740241 reap_dead start 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4b58a640 1 -- 172.21.15.46:0/2661740241 reap_dead start 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 msgr2=0x561f5155ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 0x561f5155ac00 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x561f5182fb90 tx=0x561f515a6a20 comp rx=0 tx=0).stop 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 shutdown_connections 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/2661740241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51555680 0x561f5155ac00 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 >> 172.21.15.46:0/2661740241 conn(0x561f51556d00 msgr2=0x561f5189e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 shutdown_connections 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/2661740241 wait complete. 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x561f51555b00 msgr2=0x561f5155a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x561f51555b00 0x561f5155a680 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x561f52394b40 tx=0x561f51568000 comp rx=0 tx=0).stop 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 msgr2=0x561f51559600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.959+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x561f52395890 tx=0x561f515a68a0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 shutdown_connections 2024-05-09T03:21:18.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51556880 0x561f5155a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x561f51555b00 0x561f5155a680 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51554480 0x561f51559600 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/3920540566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51554000 0x561f51559b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 >> 172.21.15.46:0/3920540566 conn(0x561f51554900 msgr2=0x561f515ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 shutdown_connections 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/3920540566 wait complete. 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 msgr2=0x561f51558b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 0x561f51558b00 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x561f522db950 tx=0x561f515a7ce0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 shutdown_connections 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561f51554d80 0x561f51558000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561f51555200 0x561f51558580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 --2- 172.21.15.46:0/158696847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561f51557600 0x561f51558b00 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 >> 172.21.15.46:0/158696847 conn(0x561f51555f80 msgr2=0x561f515ef000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 shutdown_connections 2024-05-09T03:21:18.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:18.960+0000 7fdd4d5df280 1 -- 172.21.15.46:0/158696847 wait complete. 2024-05-09T03:21:18.969 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.090319, result: old journal was 4220277~892 new journal start will be 8388608 (4167439 bytes past old end) writing journal head writing EResetJournal entry done 2024-05-09T03:21:18.969 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-05-09T03:21:19.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.086+0000 7f1653f76640 1 Processor -- start 2024-05-09T03:21:19.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.086+0000 7f1653f76640 1 -- start start 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1653f76640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f164c0b7e70 0x7f164c0d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1653f76640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164c0b5550 con 0x7f164c0d3df0 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1653f76640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164c0d44e0 con 0x7f164c0b5140 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1653f76640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f164c0d3480 con 0x7f164c0b7e70 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1651511640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f164c0b7e70 0x7f164c0d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1652513640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f164c0d3df0 0x7f164c0b9380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43514/0 (socket says 172.21.15.46:43514) 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1652513640 1 -- 172.21.15.46:0/3819833165 learned_addr learned my addr 172.21.15.46:0/3819833165 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1651511640 1 -- 172.21.15.46:0/3819833165 >> v1:172.21.15.88:6789/0 conn(0x7f164c0b5140 legacy=0x7f164c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1651511640 1 -- 172.21.15.46:0/3819833165 >> v1:172.21.15.46:6789/0 conn(0x7f164c0d3df0 legacy=0x7f164c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1651511640 1 -- 172.21.15.46:0/3819833165 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164c0d47c0 con 0x7f164c0b7e70 2024-05-09T03:21:19.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.088+0000 7f1650d10640 1 -- 172.21.15.46:0/3819833165 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550754209 0 0) 0x7f164c0b5550 con 0x7f164c0d3df0 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.089+0000 7f1651511640 1 --2- 172.21.15.46:0/3819833165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c0b7e70 0x7f164c0d38b0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f16400097a0 tx=0x7f164002a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3330f22e581484c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.089+0000 7f1650d10640 1 -- 172.21.15.46:0/3819833165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1640037070 con 0x7f164c0b7e70 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.089+0000 7f1650d10640 1 -- 172.21.15.46:0/3819833165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1640032640 con 0x7f164c0b7e70 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.089+0000 7f1650d10640 1 -- 172.21.15.46:0/3819833165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1640032a10 con 0x7f164c0b7e70 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 -- 172.21.15.46:0/3819833165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c0b7e70 msgr2=0x7f164c0d38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 --2- 172.21.15.46:0/3819833165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c0b7e70 0x7f164c0d38b0 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f16400097a0 tx=0x7f164002a2c0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 -- 172.21.15.46:0/3819833165 shutdown_connections 2024-05-09T03:21:19.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 --2- 172.21.15.46:0/3819833165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c0b7e70 0x7f164c0d38b0 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 -- 172.21.15.46:0/3819833165 >> 172.21.15.46:0/3819833165 conn(0x7f164c08d3f0 msgr2=0x7f164c08d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:19.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 -- 172.21.15.46:0/3819833165 shutdown_connections 2024-05-09T03:21:19.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.090+0000 7f1653f76640 1 -- 172.21.15.46:0/3819833165 wait complete. 2024-05-09T03:21:19.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.092+0000 7f1653f76640 1 Processor -- start 2024-05-09T03:21:19.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.092+0000 7f1653f76640 1 -- start start 2024-05-09T03:21:19.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1653f76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f164c0b5140 0x7f164c0c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1653f76640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f164c0d3df0 0x7f164c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1653f76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 0x7f164c148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1653f76640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f164c0d4200 con 0x7f164c15c8a0 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1653f76640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f164c0d4d80 con 0x7f164c0b5140 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1653f76640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f164c0d5580 con 0x7f164c0d3df0 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1652513640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 0x7f164c148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1652513640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 0x7f164c148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46440/0 (socket says 172.21.15.46:46440) 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1652513640 1 -- 172.21.15.46:0/2896262300 learned_addr learned my addr 172.21.15.46:0/2896262300 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.093+0000 7f1651d12640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f164c0b5140 0x7f164c0c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1651511640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f164c0d3df0 0x7f164c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1652513640 1 -- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f164c0d3df0 msgr2=0x7f164c15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1652513640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f164c0d3df0 0x7f164c15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1652513640 1 -- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f164c0b5140 msgr2=0x7f164c0c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1652513640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f164c0b5140 0x7f164c0c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1652513640 1 -- 172.21.15.46:0/2896262300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164c1583f0 con 0x7f164c15c8a0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1652513640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 0x7f164c148610 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c6b0 tx=0x7f163c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c018070 con 0x7f164c15c8a0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163c004da0 con 0x7f164c15c8a0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f164c1586d0 con 0x7f164c15c8a0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c00a880 con 0x7f164c15c8a0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.094+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f164c158c90 con 0x7f164c15c8a0 2024-05-09T03:21:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.095+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f164c06a270 con 0x7f164c15c8a0 2024-05-09T03:21:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.096+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f163c01b070 con 0x7f164c15c8a0 2024-05-09T03:21:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.097+0000 7f164a7fc640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f16380762d0 0x7f16380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.097+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f163c096cd0 con 0x7f164c15c8a0 2024-05-09T03:21:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.097+0000 7f1651d12640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f16380762d0 0x7f16380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:19.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.097+0000 7f1651d12640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f16380762d0 0x7f16380787e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f1644000f60 tx=0x7f1644009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:19.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.101+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f163c05fee0 con 0x7f164c15c8a0 2024-05-09T03:21:19.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.333+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f164c163280 con 0x7f164c15c8a0 2024-05-09T03:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.864+0000 7f164a7fc640 1 -- 172.21.15.46:0/2896262300 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v355) ==== 162+0+0 (secure 0 0 0) 0x7f163c0641e0 con 0x7f164c15c8a0 2024-05-09T03:21:19.865 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-05-09T03:21:19.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f16380762d0 msgr2=0x7f16380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f16380762d0 0x7f16380787e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f1644000f60 tx=0x7f1644009040 comp rx=0 tx=0).stop 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 msgr2=0x7f164c148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 0x7f164c148610 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c6b0 tx=0x7f163c00cb70 comp rx=0 tx=0).stop 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 shutdown_connections 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f16380762d0 0x7f16380787e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f164c15c8a0 0x7f164c148610 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f164c0d3df0 0x7f164c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 --2- 172.21.15.46:0/2896262300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f164c0b5140 0x7f164c0c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.866+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 >> 172.21.15.46:0/2896262300 conn(0x7f164c08d3f0 msgr2=0x7f164c15fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.867+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 shutdown_connections 2024-05-09T03:21:19.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:19.867+0000 7f1653f76640 1 -- 172.21.15.46:0/2896262300 wait complete. 2024-05-09T03:21:19.884 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.002+0000 7ff29aaa8640 1 Processor -- start 2024-05-09T03:21:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.002+0000 7ff29aaa8640 1 -- start start 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.003+0000 7ff29aaa8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff294164c70 0x7ff2941615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.003+0000 7ff29aaa8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff29415dfe0 0x7ff2940d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.003+0000 7ff29aaa8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2940d2480 0x7ff2940d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.003+0000 7ff29aaa8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff294161150 con 0x7ff294164c70 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.003+0000 7ff29aaa8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff294160b20 con 0x7ff29415dfe0 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.003+0000 7ff29aaa8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff2940d15f0 con 0x7ff2940d2480 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff294164c70 0x7ff2941615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff294164c70 0x7ff2941615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46448/0 (socket says 172.21.15.46:46448) 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 -- 172.21.15.46:0/1589966914 learned_addr learned my addr 172.21.15.46:0/1589966914 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff299045640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2940d2480 0x7ff2940d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff293fff640 1 --2- 172.21.15.46:0/1589966914 >> v2:172.21.15.88:3300/0 conn(0x7ff29415dfe0 0x7ff2940d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 -- 172.21.15.46:0/1589966914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2940d2480 msgr2=0x7ff2940d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2940d2480 0x7ff2940d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 -- 172.21.15.46:0/1589966914 >> v2:172.21.15.88:3300/0 conn(0x7ff29415dfe0 msgr2=0x7ff2940d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 --2- 172.21.15.46:0/1589966914 >> v2:172.21.15.88:3300/0 conn(0x7ff29415dfe0 0x7ff2940d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff298844640 1 -- 172.21.15.46:0/1589966914 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2940d18d0 con 0x7ff294164c70 2024-05-09T03:21:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.004+0000 7ff299045640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2940d2480 0x7ff2940d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.005+0000 7ff298844640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294164c70 0x7ff2941615e0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff2940cce00 tx=0x7ff28802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4776165415f33a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.005+0000 7ff2937fe640 1 -- 172.21.15.46:0/1589966914 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288037070 con 0x7ff294164c70 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.005+0000 7ff2937fe640 1 -- 172.21.15.46:0/1589966914 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff288032440 con 0x7ff294164c70 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.005+0000 7ff2937fe640 1 -- 172.21.15.46:0/1589966914 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288032810 con 0x7ff294164c70 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.005+0000 7ff29aaa8640 1 -- 172.21.15.46:0/1589966914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294164c70 msgr2=0x7ff2941615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.005+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294164c70 0x7ff2941615e0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7ff2940cce00 tx=0x7ff28802a1e0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 -- 172.21.15.46:0/1589966914 shutdown_connections 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff2940d2480 0x7ff2940d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/1589966914 >> v2:172.21.15.88:3300/0 conn(0x7ff29415dfe0 0x7ff2940d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/1589966914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff294164c70 0x7ff2941615e0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 -- 172.21.15.46:0/1589966914 >> 172.21.15.46:0/1589966914 conn(0x7ff294073db0 msgr2=0x7ff2940741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 -- 172.21.15.46:0/1589966914 shutdown_connections 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.006+0000 7ff29aaa8640 1 -- 172.21.15.46:0/1589966914 wait complete. 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.007+0000 7ff29aaa8640 1 Processor -- start 2024-05-09T03:21:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.007+0000 7ff29aaa8640 1 -- start start 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff29aaa8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 0x7ff2940cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff29aaa8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff29415dfe0 0x7ff294163810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff29aaa8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294164c70 0x7ff294163d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff29aaa8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff29415e3f0 con 0x7ff2940d2480 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff29aaa8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff2940d1300 con 0x7ff294164c70 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff29aaa8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff2940d2dd0 con 0x7ff29415dfe0 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 0x7ff2940cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 0x7ff2940cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46464/0 (socket says 172.21.15.46:46464) 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 -- 172.21.15.46:0/2192279444 learned_addr learned my addr 172.21.15.46:0/2192279444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:20.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff299045640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294164c70 0x7ff294163d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff293fff640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff29415dfe0 0x7ff294163810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 -- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff29415dfe0 msgr2=0x7ff294163810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff29415dfe0 0x7ff294163810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 -- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294164c70 msgr2=0x7ff294163d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294164c70 0x7ff294163d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.009+0000 7ff298844640 1 -- 172.21.15.46:0/2192279444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2940cb500 con 0x7ff2940d2480 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.010+0000 7ff298844640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 0x7ff2940cafc0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7ff288000c00 tx=0x7ff2880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.010+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288041020 con 0x7ff2940d2480 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.010+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2880306c0 con 0x7ff2940d2480 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.010+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff294162030 con 0x7ff2940d2480 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.010+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff288037040 con 0x7ff2940d2480 2024-05-09T03:21:20.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.010+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff294155db0 con 0x7ff2940d2480 2024-05-09T03:21:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.011+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff29408cfd0 con 0x7ff2940d2480 2024-05-09T03:21:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.012+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff288044020 con 0x7ff2940d2480 2024-05-09T03:21:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.012+0000 7ff2917fa640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2780762d0 0x7ff2780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.012+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7ff2880b7fd0 con 0x7ff2940d2480 2024-05-09T03:21:20.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.013+0000 7ff293fff640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2780762d0 0x7ff2780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:20.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.013+0000 7ff293fff640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2780762d0 0x7ff2780787e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7ff294160750 tx=0x7ff280008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.017+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff29408cfd0 con 0x7ff2940d2480 2024-05-09T03:21:20.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.247+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff294155ba0 con 0x7ff2940d2480 2024-05-09T03:21:20.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.248+0000 7ff2917fa640 1 -- 172.21.15.46:0/2192279444 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 356 v356) ==== 95+0+5181 (secure 0 0 0) 0x7ff28804b020 con 0x7ff2940d2480 2024-05-09T03:21:20.249 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:20.249 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":356,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":356,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:19.867986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-05-09T03:21:20.249 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 356 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2780762d0 msgr2=0x7ff2780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2780762d0 0x7ff2780787e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7ff294160750 tx=0x7ff280008040 comp rx=0 tx=0).stop 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 msgr2=0x7ff2940cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 0x7ff2940cafc0 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7ff288000c00 tx=0x7ff2880046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 shutdown_connections 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294164c70 0x7ff294163d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff29415dfe0 0x7ff294163810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2780762d0 0x7ff2780787e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 --2- 172.21.15.46:0/2192279444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2940d2480 0x7ff2940cafc0 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:20.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 >> 172.21.15.46:0/2192279444 conn(0x7ff294073db0 msgr2=0x7ff2940d5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:20.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 shutdown_connections 2024-05-09T03:21:20.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:20.250+0000 7ff29aaa8640 1 -- 172.21.15.46:0/2192279444 wait complete. 2024-05-09T03:21:20.664 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 356, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:21:01.965784+0000', 'modified': '2024-05-09T03:21:19.867986+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10812': {'gid': 10812, 'name': 'c', 'rank': 0, 'incarnation': 356, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/2236478370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 2236478370}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2236478370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:21:20.664 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10812:up:replay 2024-05-09T03:21:21.665 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:21.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.794+0000 7fdc99927640 1 Processor -- start 2024-05-09T03:21:21.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.794+0000 7fdc99927640 1 -- start start 2024-05-09T03:21:21.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc99927640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 0x7fdc940b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:21.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc99927640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc940b7d20 0x7fdc940b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:21.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc99927640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc940b8130 con 0x7fdc940d57a0 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc99927640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdc940d5bb0 con 0x7fdc940b7d20 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc99927640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdc940d49e0 con 0x7fdc94160610 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc927fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 0x7fdc940b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc940b7d20 0x7fdc940b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc92ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdc940d57a0 0x7fdc940b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:50646/0 (socket says 172.21.15.46:50646) 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc927fc640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 0x7fdc940b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:47758/0 (socket says 172.21.15.46:47758) 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc92ffd640 1 -- 172.21.15.46:0/1988063356 learned_addr learned my addr 172.21.15.46:0/1988063356 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc937fe640 1 -- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 msgr2=0x7fdc940b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:21.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc937fe640 1 --2- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 0x7fdc940b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:21.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc937fe640 1 -- 172.21.15.46:0/1988063356 >> v1:172.21.15.88:6789/0 conn(0x7fdc940d57a0 legacy=0x7fdc940b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:21.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc937fe640 1 -- 172.21.15.46:0/1988063356 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc940d4cc0 con 0x7fdc940b7d20 2024-05-09T03:21:21.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.796+0000 7fdc927fc640 1 --2- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 0x7fdc940b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:21.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.797+0000 7fdc937fe640 1 --2- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc940b9450 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009d90 tx=0x7fdc8c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c1b88e337036157 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:21.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.797+0000 7fdc91ffb640 1 -- 172.21.15.46:0/1988063356 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c007740 con 0x7fdc940b7d20 2024-05-09T03:21:21.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.797+0000 7fdc91ffb640 1 -- 172.21.15.46:0/1988063356 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8c0078e0 con 0x7fdc940b7d20 2024-05-09T03:21:21.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc91ffb640 1 -- 172.21.15.46:0/1988063356 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c007cb0 con 0x7fdc940b7d20 2024-05-09T03:21:21.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 -- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 msgr2=0x7fdc940b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 --2- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc940b9450 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009d90 tx=0x7fdc8c02a660 comp rx=0 tx=0).stop 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 -- 172.21.15.46:0/1988063356 shutdown_connections 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 --2- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc940b9450 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 --2- 172.21.15.46:0/1988063356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc94160610 0x7fdc940b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 -- 172.21.15.46:0/1988063356 >> 172.21.15.46:0/1988063356 conn(0x7fdc940743e0 msgr2=0x7fdc940d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 -- 172.21.15.46:0/1988063356 shutdown_connections 2024-05-09T03:21:21.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.798+0000 7fdc99927640 1 -- 172.21.15.46:0/1988063356 wait complete. 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.800+0000 7fdc99927640 1 Processor -- start 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.800+0000 7fdc99927640 1 -- start start 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc99927640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc94159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc99927640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc940d57a0 0x7fdc9415b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc99927640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 0x7fdc941561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc99927640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94160a20 con 0x7fdc94160610 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc99927640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdc940d5280 con 0x7fdc940b7d20 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc99927640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdc940d15b0 con 0x7fdc940d57a0 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 0x7fdc941561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 0x7fdc941561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46466/0 (socket says 172.21.15.46:46466) 2024-05-09T03:21:21.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 -- 172.21.15.46:0/163304394 learned_addr learned my addr 172.21.15.46:0/163304394 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc92ffd640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc94159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc927fc640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc940d57a0 0x7fdc9415b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 -- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc940d57a0 msgr2=0x7fdc9415b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc940d57a0 0x7fdc9415b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 -- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 msgr2=0x7fdc94159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc94159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 -- 172.21.15.46:0/163304394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9415b630 con 0x7fdc94160610 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.802+0000 7fdc937fe640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 0x7fdc941561e0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009eb0 tx=0x7fdc8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.803+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c03f070 con 0x7fdc94160610 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.803+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8c031bc0 con 0x7fdc94160610 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.803+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9415a030 con 0x7fdc94160610 2024-05-09T03:21:21.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.803+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c009450 con 0x7fdc94160610 2024-05-09T03:21:21.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.803+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9415a610 con 0x7fdc94160610 2024-05-09T03:21:21.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.804+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc940d3220 con 0x7fdc94160610 2024-05-09T03:21:21.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.804+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc8c03b410 con 0x7fdc94160610 2024-05-09T03:21:21.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.805+0000 7fdc7bfff640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:21.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.805+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fdc8c0b8cf0 con 0x7fdc94160610 2024-05-09T03:21:21.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.805+0000 7fdc92ffd640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:21.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.806+0000 7fdc92ffd640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdc84000f60 tx=0x7fdc84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:21.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:21.809+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdc940d3220 con 0x7fdc94160610 2024-05-09T03:21:22.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.043+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc940b4c60 con 0x7fdc94160610 2024-05-09T03:21:22.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.045+0000 7fdc7bfff640 1 -- 172.21.15.46:0/163304394 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 358 v358) ==== 95+0+5181 (secure 0 0 0) 0x7fdc940b4c60 con 0x7fdc94160610 2024-05-09T03:21:22.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:22.046 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":358,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":358,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:20.874991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-05-09T03:21:22.046 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 358 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 msgr2=0x7fdc800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdc84000f60 tx=0x7fdc84009040 comp rx=0 tx=0).stop 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 msgr2=0x7fdc941561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 0x7fdc941561e0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009eb0 tx=0x7fdc8c004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 shutdown_connections 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc94160610 0x7fdc941561e0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:22.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdc940d57a0 0x7fdc9415b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:22.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 --2- 172.21.15.46:0/163304394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc940b7d20 0x7fdc94159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:22.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 >> 172.21.15.46:0/163304394 conn(0x7fdc940743e0 msgr2=0x7fdc941e5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:22.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 shutdown_connections 2024-05-09T03:21:22.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:22.047+0000 7fdc99927640 1 -- 172.21.15.46:0/163304394 wait complete. 2024-05-09T03:21:22.462 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 358, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:21:01.965784+0000', 'modified': '2024-05-09T03:21:20.874991+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10812': {'gid': 10812, 'name': 'c', 'rank': 0, 'incarnation': 356, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.46:6835/2236478370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 2236478370}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2236478370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-05-09T03:21:22.462 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10812:up:rejoin 2024-05-09T03:21:23.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:23.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.592+0000 7f7de18b8640 1 Processor -- start 2024-05-09T03:21:23.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.592+0000 7f7de18b8640 1 -- start start 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7de18b8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7ddc1613c0 0x7f7ddc0b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7de18b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ddc0b8110 con 0x7f7ddc0b7d00 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7de18b8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ddc1617d0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7de18b8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7ddc0b5620 con 0x7f7ddc1613c0 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7ddb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7ddc0b7d00 0x7f7ddc163e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43522/0 (socket says 172.21.15.46:43522) 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7ddb7fe640 1 -- 172.21.15.46:0/2392343532 learned_addr learned my addr 172.21.15.46:0/2392343532 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/2392343532 >> v2:172.21.15.88:3300/0 conn(0x7f7ddc1613c0 0x7f7ddc0b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/2392343532 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108419489 0 0) 0x7f7dd4001490 con 0x7f7ddc0b7d00 2024-05-09T03:21:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/2392343532 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dc4003520 con 0x7f7ddc0b7d00 2024-05-09T03:21:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dda7fc640 1 -- 172.21.15.46:0/2392343532 >> v1:172.21.15.88:6789/0 conn(0x7f7ddc0b5210 legacy=0x7f7ddc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dda7fc640 1 -- 172.21.15.46:0/2392343532 >> v1:172.21.15.46:6789/0 conn(0x7f7ddc0b7d00 legacy=0x7f7ddc163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/2392343532 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053987945 0 0) 0x7f7ddc1617d0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.594+0000 7f7dda7fc640 1 -- 172.21.15.46:0/2392343532 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ddc0d5280 con 0x7f7ddc1613c0 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.595+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/2392343532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc1613c0 0x7f7ddc0b77c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00097a0 tx=0x7f7dd002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b123bbc47d6c9731 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.595+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/2392343532 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0037070 con 0x7f7ddc1613c0 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.595+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/2392343532 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd002ac30 con 0x7f7ddc1613c0 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.595+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/2392343532 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4001c40 con 0x7f7ddc1613c0 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 -- 172.21.15.46:0/2392343532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc1613c0 msgr2=0x7f7ddc0b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 --2- 172.21.15.46:0/2392343532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc1613c0 0x7f7ddc0b77c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00097a0 tx=0x7f7dd002a1a0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 -- 172.21.15.46:0/2392343532 shutdown_connections 2024-05-09T03:21:23.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 --2- 172.21.15.46:0/2392343532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc1613c0 0x7f7ddc0b77c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 -- 172.21.15.46:0/2392343532 >> 172.21.15.46:0/2392343532 conn(0x7f7ddc0d29a0 msgr2=0x7f7ddc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 -- 172.21.15.46:0/2392343532 shutdown_connections 2024-05-09T03:21:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.596+0000 7f7de18b8640 1 -- 172.21.15.46:0/2392343532 wait complete. 2024-05-09T03:21:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.598+0000 7f7de18b8640 1 Processor -- start 2024-05-09T03:21:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.598+0000 7f7de18b8640 1 -- start start 2024-05-09T03:21:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.599+0000 7f7de18b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 0x7f7ddc14f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.599+0000 7f7de18b8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc0b7d00 0x7f7ddc15cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.599+0000 7f7de18b8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ddc1613c0 0x7f7ddc0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.599+0000 7f7de18b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc160bc0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.599+0000 7f7de18b8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ddc1609a0 con 0x7f7ddc0b7d00 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.599+0000 7f7de18b8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f7ddc160ea0 con 0x7f7ddc1613c0 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 0x7f7ddc14f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 0x7f7ddc14f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46468/0 (socket says 172.21.15.46:46468) 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 -- 172.21.15.46:0/3458477253 learned_addr learned my addr 172.21.15.46:0/3458477253 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc0b7d00 0x7f7ddc15cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddb7fe640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ddc1613c0 0x7f7ddc0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 -- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ddc1613c0 msgr2=0x7f7ddc0c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ddc1613c0 0x7f7ddc0c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 -- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc0b7d00 msgr2=0x7f7ddc15cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc0b7d00 0x7f7ddc15cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 -- 172.21.15.46:0/3458477253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ddc15d050 con 0x7f7ddc0b5210 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7ddaffd640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 0x7f7ddc14f1e0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00c3c0 tx=0x7f7dcc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc017070 con 0x7f7ddc0b5210 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dcc0035f0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ddc15d4d0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc003a20 con 0x7f7ddc0b5210 2024-05-09T03:21:23.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.600+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ddc15da90 con 0x7f7ddc0b5210 2024-05-09T03:21:23.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.601+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ddc0d1a50 con 0x7f7ddc0b5210 2024-05-09T03:21:23.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.602+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7dcc01a070 con 0x7f7ddc0b5210 2024-05-09T03:21:23.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.603+0000 7f7dc3fff640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7dc80762d0 0x7f7dc80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:23.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.603+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f7dcc096ac0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.603+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7dc80762d0 0x7f7dc80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:23.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.604+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7dc80762d0 0x7f7dc80787e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0034000 tx=0x7f7dd002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:23.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.608+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7ddc0d1a50 con 0x7f7ddc0b5210 2024-05-09T03:21:23.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.841+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ddc06a270 con 0x7f7ddc0b5210 2024-05-09T03:21:23.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.841+0000 7f7dc3fff640 1 -- 172.21.15.46:0/3458477253 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) ==== 95+0+5190 (secure 0 0 0) 0x7f7dcc05fbc0 con 0x7f7ddc0b5210 2024-05-09T03:21:23.842 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:23.842 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":359,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:22.873173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:23.843 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 359 2024-05-09T03:21:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.843+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7dc80762d0 msgr2=0x7f7dc80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.843+0000 7f7de18b8640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7dc80762d0 0x7f7dc80787e0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0034000 tx=0x7f7dd002e000 comp rx=0 tx=0).stop 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.843+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 msgr2=0x7f7ddc14f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.843+0000 7f7de18b8640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 0x7f7ddc14f1e0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00c3c0 tx=0x7f7dcc00c880 comp rx=0 tx=0).stop 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 shutdown_connections 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7ddc1613c0 0x7f7ddc0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ddc0b7d00 0x7f7ddc15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f7dc80762d0 0x7f7dc80787e0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 --2- 172.21.15.46:0/3458477253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ddc0b5210 0x7f7ddc14f1e0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 >> 172.21.15.46:0/3458477253 conn(0x7f7ddc0d29a0 msgr2=0x7f7ddc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 shutdown_connections 2024-05-09T03:21:23.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:23.844+0000 7f7de18b8640 1 -- 172.21.15.46:0/3458477253 wait complete. 2024-05-09T03:21:24.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 359, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T03:21:01.965784+0000', 'modified': '2024-05-09T03:21:22.873173+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 136, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10812}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10812': {'gid': 10812, 'name': 'c', 'rank': 0, 'incarnation': 356, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6835/2236478370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 2236478370}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2236478370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10812, 'qdb_cluster': [10812]} 2024-05-09T03:21:24.259 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T03:21:24.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T03:21:24.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.378+0000 7faf0e670640 1 Processor -- start 2024-05-09T03:21:24.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.378+0000 7faf0e670640 1 -- start start 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0e670640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf080d2f60 0x7faf080cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0e670640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf0809d470 con 0x7faf080ceba0 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0e670640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf080d11b0 con 0x7faf080d3730 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0e670640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faf0809cfe0 con 0x7faf080d2f60 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0cc0d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf080d2f60 0x7faf080cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf077fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faf080ceba0 0x7faf080cefb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43536/0 (socket says 172.21.15.46:43536) 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf077fe640 1 -- 172.21.15.46:0/211699191 learned_addr learned my addr 172.21.15.46:0/211699191 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0cc0d640 1 -- 172.21.15.46:0/211699191 >> v1:172.21.15.88:6789/0 conn(0x7faf080d3730 legacy=0x7faf0806e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0cc0d640 1 -- 172.21.15.46:0/211699191 >> v1:172.21.15.46:6789/0 conn(0x7faf080ceba0 legacy=0x7faf080cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf0cc0d640 1 -- 172.21.15.46:0/211699191 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0806e9b0 con 0x7faf080d2f60 2024-05-09T03:21:24.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.380+0000 7faf06ffd640 1 -- 172.21.15.46:0/211699191 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910774018 0 0) 0x7faefc001110 con 0x7faf080ceba0 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.381+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/211699191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d2f60 0x7faf080cf6c0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7faf00009d90 tx=0x7faf0002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d7414f9ea84111 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.381+0000 7faf06ffd640 1 -- 172.21.15.46:0/211699191 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00007740 con 0x7faf080d2f60 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.381+0000 7faf06ffd640 1 -- 172.21.15.46:0/211699191 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000078e0 con 0x7faf080d2f60 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.381+0000 7faf06ffd640 1 -- 172.21.15.46:0/211699191 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00007cb0 con 0x7faf080d2f60 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 -- 172.21.15.46:0/211699191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d2f60 msgr2=0x7faf080cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 --2- 172.21.15.46:0/211699191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d2f60 0x7faf080cf6c0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7faf00009d90 tx=0x7faf0002a660 comp rx=0 tx=0).stop 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 -- 172.21.15.46:0/211699191 shutdown_connections 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 --2- 172.21.15.46:0/211699191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d2f60 0x7faf080cf6c0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 -- 172.21.15.46:0/211699191 >> 172.21.15.46:0/211699191 conn(0x7faf080d05f0 msgr2=0x7faf080d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 -- 172.21.15.46:0/211699191 shutdown_connections 2024-05-09T03:21:24.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.382+0000 7faf0e670640 1 -- 172.21.15.46:0/211699191 wait complete. 2024-05-09T03:21:24.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.384+0000 7faf0e670640 1 Processor -- start 2024-05-09T03:21:24.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.384+0000 7faf0e670640 1 -- start start 2024-05-09T03:21:24.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0e670640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf080ceba0 0x7faf0815d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0e670640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf080d2f60 0x7faf080c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0e670640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 0x7faf080c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0e670640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf080d3370 con 0x7faf080d3730 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0e670640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faf080b94a0 con 0x7faf080d2f60 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0e670640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faf080b9160 con 0x7faf080ceba0 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 0x7faf080c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 0x7faf080c9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46484/0 (socket says 172.21.15.46:46484) 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.385+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3963199051 learned_addr learned my addr 172.21.15.46:0/3963199051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf077fe640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf080d2f60 0x7faf080c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf07fff640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf080ceba0 0x7faf0815d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf080ceba0 msgr2=0x7faf0815d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf080ceba0 0x7faf0815d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf080d2f60 msgr2=0x7faf080c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf080d2f60 0x7faf080c8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3963199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0815d970 con 0x7faf080d3730 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf07fff640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf080ceba0 0x7faf0815d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 0x7faf080c9b30 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7faf0002ac50 tx=0x7faf00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf04ff9640 1 -- 172.21.15.46:0/3963199051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0003f070 con 0x7faf080d3730 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf04ff9640 1 -- 172.21.15.46:0/3963199051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf00031c90 con 0x7faf080d3730 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf080c9190 con 0x7faf080d3730 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf04ff9640 1 -- 172.21.15.46:0/3963199051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00043630 con 0x7faf080d3730 2024-05-09T03:21:24.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.386+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0815f460 con 0x7faf080d3730 2024-05-09T03:21:24.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.388+0000 7faf04ff9640 1 -- 172.21.15.46:0/3963199051 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faf0003b410 con 0x7faf080d3730 2024-05-09T03:21:24.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.389+0000 7faf04ff9640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faef4076300 0x7faef4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.389+0000 7faf04ff9640 1 -- 172.21.15.46:0/3963199051 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7faf000b8da0 con 0x7faf080d3730 2024-05-09T03:21:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.389+0000 7faf07fff640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faef4076300 0x7faef4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.389+0000 7faf155fe740 1 Processor -- start 2024-05-09T03:21:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.389+0000 7faf155fe740 1 -- start start 2024-05-09T03:21:24.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.390+0000 7faf07fff640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faef4076300 0x7faef4078810 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faef8007fd0 tx=0x7faef8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6de270 0x564c4eaadeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eac8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf155fe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eac9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf155fe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564c4e9e64e0 con 0x564c4e6bd590 2024-05-09T03:21:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564c4e981ff0 con 0x564c4e6de270 2024-05-09T03:21:24.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564c4e9e24a0 con 0x564c4e6b53b0 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eac9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eac9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46498/0 (socket says 172.21.15.46:46498) 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 -- 172.21.15.46:0/484765041 learned_addr learned my addr 172.21.15.46:0/484765041 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf07fff640 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6de270 0x564c4eaadeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf077fe640 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eac8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 -- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 msgr2=0x564c4eac8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eac8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 -- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6de270 msgr2=0x564c4eaadeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6de270 0x564c4eaadeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 -- 172.21.15.46:0/484765041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c4e968720 con 0x564c4e6bd590 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eac9120 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7faf0007e060 tx=0x7faf00081750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfe3338dfb1abf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.391+0000 7faeedffb640 1 -- 172.21.15.46:0/484765041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00081af0 con 0x564c4e6bd590 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faeedffb640 1 -- 172.21.15.46:0/484765041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf00081fe0 con 0x564c4e6bd590 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faeedffb640 1 -- 172.21.15.46:0/484765041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000862e0 con 0x564c4e6bd590 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 -- 172.21.15.46:0/484765041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 msgr2=0x564c4eac9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eac9120 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7faf0007e060 tx=0x7faf00081750 comp rx=0 tx=0).stop 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 -- 172.21.15.46:0/484765041 shutdown_connections 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eac9120 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eac8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 --2- 172.21.15.46:0/484765041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6de270 0x564c4eaadeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 -- 172.21.15.46:0/484765041 >> 172.21.15.46:0/484765041 conn(0x564c4e6d23c0 msgr2=0x564c4ea83510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 -- 172.21.15.46:0/484765041 shutdown_connections 2024-05-09T03:21:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.392+0000 7faf155fe740 1 -- 172.21.15.46:0/484765041 wait complete. 2024-05-09T03:21:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.393+0000 7faf155fe740 1 -- start start 2024-05-09T03:21:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.393+0000 7faf155fe740 1 Processor -- start 2024-05-09T03:21:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eac4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6bd590 0x564c4eac5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4eac58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564c4e96d760 con 0x564c4e6de270 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564c4eab1510 con 0x564c4e6b53b0 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564c4eac35e0 con 0x564c4e6bd590 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 10 client.0 _ll_register_callbacks cb 0x564c4e9904b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4eac58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4eac58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46502/0 (socket says 172.21.15.46:46502) 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3660291904 learned_addr learned my addr 172.21.15.46:0/3660291904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf077fe640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6bd590 0x564c4eac5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf07fff640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eac4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6bd590 msgr2=0x564c4eac5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6bd590 0x564c4eac5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 msgr2=0x564c4eac4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eac4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 -- 172.21.15.46:0/3660291904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c4e9aebd0 con 0x564c4e6de270 2024-05-09T03:21:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf07fff640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eac4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4eac58a0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7faf000b71d0 tx=0x7faf00009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba57b066bccb10c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faee6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faee6ffd640 1 -- 172.21.15.46:0/3660291904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00081af0 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faee6ffd640 1 -- 172.21.15.46:0/3660291904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7faf0008ab10 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.395+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf00081fe0 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faee6ffd640 1 -- 172.21.15.46:0/3660291904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7faf0008ad90 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 10 client.10866 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x564c4e994280 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faee6ffd640 1 -- 172.21.15.46:0/3660291904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf00081fe0 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faee6ffd640 1 -- 172.21.15.46:0/3660291904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7faf00081240 con 0x564c4e6de270 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 10 client.10866 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 10 client.10866 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 20 client.10866 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4e69ce40 0x564c4ea9b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 4 client.10866 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c4e9f40f0 con 0x564c4e69ce40 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faf07fff640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4e69ce40 0x564c4ea9b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.396+0000 7faee6ffd640 10 client.10866 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.397+0000 7faf07fff640 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4e69ce40 0x564c4ea9b050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5ad4ff7b407cd3c server_cookie=6e02cf82bd017c4f in_seq=0 out_seq=0 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.397+0000 7faee6ffd640 1 -- 172.21.15.46:0/3660291904 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564c4e9f40f0 con 0x564c4e69ce40 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.397+0000 7faee6ffd640 10 client.10866 handle_command_reply: tid=0 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.397+0000 7faf155fe740 1 client.10866 shutdown 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.397+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 msgr2=0x564c4eac58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.397+0000 7faf155fe740 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4eac58a0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7faf000b71d0 tx=0x7faf00009750 comp rx=0 tx=0).stop 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 shutdown_connections 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4e69ce40 0x564c4ea9b050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4eac58a0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6bd590 0x564c4eac5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 --2- 172.21.15.46:0/3660291904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eac4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faee6ffd640 0 client.10866 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 >> 172.21.15.46:0/3660291904 conn(0x564c4e6d23c0 msgr2=0x564c4eaadeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 shutdown_connections 2024-05-09T03:21:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 1 -- 172.21.15.46:0/3660291904 wait complete. 2024-05-09T03:21:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.398+0000 7faf155fe740 20 client.10866 trim_cache size 0 max 16384 2024-05-09T03:21:24.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.417+0000 7faf155fe740 1 Processor -- start 2024-05-09T03:21:24.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.417+0000 7faf155fe740 1 -- start start 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.418+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eaec2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.418+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6bd590 0x564c4ea7cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.418+0000 7faf155fe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4ea7d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.418+0000 7faf155fe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf08202190 con 0x564c4e6de270 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.418+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faf081fa060 con 0x564c4e6bd590 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.418+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faf081fa6b0 con 0x564c4e6b53b0 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4ea7d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4ea7d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46510/0 (socket says 172.21.15.46:46510) 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 -- 172.21.15.46:0/869621475 learned_addr learned my addr 172.21.15.46:0/869621475 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 -- 172.21.15.46:0/869621475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 msgr2=0x564c4eaec2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eaec2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 -- 172.21.15.46:0/869621475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6bd590 msgr2=0x564c4ea7cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6bd590 0x564c4ea7cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 -- 172.21.15.46:0/869621475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c4eaadab0 con 0x564c4e6de270 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4ea7d1c0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7faf00009b10 tx=0x7faf0009d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75645097c43ed499 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faeed7fa640 1 -- 172.21.15.46:0/869621475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000078c0 con 0x564c4e6de270 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faeed7fa640 1 -- 172.21.15.46:0/869621475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000a3070 con 0x564c4e6de270 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.419+0000 7faeed7fa640 1 -- 172.21.15.46:0/869621475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00081fe0 con 0x564c4e6de270 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 -- 172.21.15.46:0/869621475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 msgr2=0x564c4ea7d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4ea7d1c0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7faf00009b10 tx=0x7faf0009d040 comp rx=0 tx=0).stop 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 -- 172.21.15.46:0/869621475 shutdown_connections 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6de270 0x564c4ea7d1c0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6bd590 0x564c4ea7cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 --2- 172.21.15.46:0/869621475 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6b53b0 0x564c4eaec2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 -- 172.21.15.46:0/869621475 >> 172.21.15.46:0/869621475 conn(0x564c4e6d23c0 msgr2=0x564c4eac3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 -- 172.21.15.46:0/869621475 shutdown_connections 2024-05-09T03:21:24.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.420+0000 7faf155fe740 1 -- 172.21.15.46:0/869621475 wait complete. 2024-05-09T03:21:24.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.421+0000 7faf155fe740 1 -- start start 2024-05-09T03:21:24.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.421+0000 7faf155fe740 1 Processor -- start 2024-05-09T03:21:24.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eb46540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eaaec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6de270 0x564c4eaaf140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564c4eac3350 con 0x564c4e6bd590 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564c4eab1340 con 0x564c4e6b53b0 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x564c4eab05c0 con 0x564c4e6de270 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 10 client.0 _ll_register_callbacks cb 0x7faef8001360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eaaec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eaaec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46518/0 (socket says 172.21.15.46:46518) 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 -- 172.21.15.46:0/582962577 learned_addr learned my addr 172.21.15.46:0/582962577 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf0cc0d640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6de270 0x564c4eaaf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf07fff640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eb46540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 -- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6de270 msgr2=0x564c4eaaf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6de270 0x564c4eaaf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 -- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 msgr2=0x564c4eb46540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eb46540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 -- 172.21.15.46:0/582962577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c4e749850 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf07fff640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eb46540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf077fe640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eaaec00 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7faefc000c00 tx=0x7faefc0136a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6068fd079b97cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faee6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faee6ffd640 1 -- 172.21.15.46:0/582962577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc01a020 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faee6ffd640 1 -- 172.21.15.46:0/582962577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7faefc016450 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.423+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf000a3070 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faee6ffd640 1 -- 172.21.15.46:0/582962577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7faefc016650 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 10 client.10872 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x564c4e7c06b0 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faee6ffd640 1 -- 172.21.15.46:0/582962577 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000a3070 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faee6ffd640 1 -- 172.21.15.46:0/582962577 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7faefc027080 con 0x564c4e6bd590 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 10 client.10872 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 10 client.10872 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:24.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 20 client.10872 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4ea83200 0x564c4ea83610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 4 client.10872 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "status"}) -- 0x564c4e9edf60 con 0x564c4ea83200 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf07fff640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4ea83200 0x564c4ea83610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faee6ffd640 10 client.10872 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.424+0000 7faf07fff640 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4ea83200 0x564c4ea83610 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2810cbb62d19b101 server_cookie=2f023bf33b5360c2 in_seq=0 out_seq=0 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faee6ffd640 1 -- 172.21.15.46:0/582962577 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x564c4e9edf60 con 0x564c4ea83200 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faee6ffd640 10 client.10872 handle_command_reply: tid=0 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 client.10872 shutdown 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 msgr2=0x564c4eaaec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eaaec00 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7faefc000c00 tx=0x7faefc0136a0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 shutdown_connections 2024-05-09T03:21:24.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x564c4e6de270 0x564c4eaaf140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x564c4ea83200 0x564c4ea83610 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c4e6bd590 0x564c4eaaec00 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 --2- 172.21.15.46:0/582962577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564c4e6b53b0 0x564c4eb46540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faee6ffd640 0 client.10872 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 >> 172.21.15.46:0/582962577 conn(0x564c4e6d23c0 msgr2=0x564c4eac3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 shutdown_connections 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 1 -- 172.21.15.46:0/582962577 wait complete. 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.425+0000 7faf155fe740 20 client.10872 trim_cache size 0 max 16384 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36d71dd3-2926-4712-824e-80cfbafd76c2", 2024-05-09T03:21:24.431 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10812, 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.5524753159999998, 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 359, 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 136, 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 136, 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 5.4948283590000004 2024-05-09T03:21:24.432 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faef4076300 msgr2=0x7faef4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faef4076300 0x7faef4078810 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faef8007fd0 tx=0x7faef8009040 comp rx=0 tx=0).stop 2024-05-09T03:21:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 msgr2=0x7faf080c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 0x7faf080c9b30 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7faf0002ac50 tx=0x7faf00004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 shutdown_connections 2024-05-09T03:21:24.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faef4076300 0x7faef4078810 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf080d3730 0x7faf080c9b30 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf080d2f60 0x7faf080c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 --2- 172.21.15.46:0/3963199051 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faf080ceba0 0x7faf0815d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 >> 172.21.15.46:0/3963199051 conn(0x7faf080d05f0 msgr2=0x7faf080cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 shutdown_connections 2024-05-09T03:21:24.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.426+0000 7faf0e670640 1 -- 172.21.15.46:0/3963199051 wait complete. 2024-05-09T03:21:24.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T03:21:24.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.551+0000 7f770136c640 1 Processor -- start 2024-05-09T03:21:24.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.551+0000 7f770136c640 1 -- start start 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f770136c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f76fc0b9310 0x7f76fc08cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f770136c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76fc165650 con 0x7f76fc08d2a0 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f770136c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76fc06ebe0 con 0x7f76fc08d6b0 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f770136c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f76fc06f760 con 0x7f76fc0b9310 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76fa7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f76fc08d2a0 0x7f76fc164830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43540/0 (socket says 172.21.15.46:43540) 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76fa7fc640 1 -- 172.21.15.46:0/1973242782 learned_addr learned my addr 172.21.15.46:0/1973242782 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76faffd640 1 --2- 172.21.15.46:0/1973242782 >> v2:172.21.15.88:3300/0 conn(0x7f76fc0b9310 0x7f76fc08cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1966100467 0 0) 0x7f76fc165650 con 0x7f76fc08d2a0 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76e8003520 con 0x7f76fc08d2a0 2024-05-09T03:21:24.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1571817484 0 0) 0x7f76e8003520 con 0x7f76fc08d2a0 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 >> v2:172.21.15.88:3300/0 conn(0x7f76fc0b9310 msgr2=0x7f76fc08cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76f9ffb640 1 --2- 172.21.15.46:0/1973242782 >> v2:172.21.15.88:3300/0 conn(0x7f76fc0b9310 0x7f76fc08cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.553+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 >> v1:172.21.15.88:6789/0 conn(0x7f76fc08d6b0 legacy=0x7f76fc164f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc06eec0 con 0x7f76fc08d2a0 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f76e40016f0 con 0x7f76fc08d2a0 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 >> v1:172.21.15.46:6789/0 conn(0x7f76fc08d2a0 legacy=0x7f76fc164830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76e8003a50 0x7f76e8003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76e8004800 0x7f76e8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc06f760 con 0x7f76e8005250 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f76e800c1e0 con 0x7f76e8004800 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f76e800c3e0 con 0x7f76e8003a50 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76faffd640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76e8003a50 0x7f76e8003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fa7fc640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76e8004800 0x7f76e8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fb7fe640 1 -- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76e8003a50 msgr2=0x7f76e8003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76e8003a50 0x7f76e8003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fb7fe640 1 -- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76e8004800 msgr2=0x7f76e8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76e8004800 0x7f76e8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.554+0000 7f76fb7fe640 1 -- 172.21.15.46:0/1973242782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e8003520 con 0x7f76e8005250 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.555+0000 7f76fb7fe640 1 -- 172.21.15.46:0/1973242782 reap_dead start 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.555+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76e800b9c0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f76f4001790 tx=0x7f76e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e6f08db7da8a24 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.555+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f40013b0 con 0x7f76e8005250 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.555+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f40052d0 con 0x7f76e8005250 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.555+0000 7f76f9ffb640 1 -- 172.21.15.46:0/1973242782 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f402ab20 con 0x7f76e8005250 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 -- 172.21.15.46:0/1973242782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 msgr2=0x7f76e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76e800b9c0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f76f4001790 tx=0x7f76e8005830 comp rx=0 tx=0).stop 2024-05-09T03:21:24.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 -- 172.21.15.46:0/1973242782 shutdown_connections 2024-05-09T03:21:24.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 --2- 172.21.15.46:0/1973242782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76e800b9c0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 -- 172.21.15.46:0/1973242782 >> 172.21.15.46:0/1973242782 conn(0x7f76fc0b5660 msgr2=0x7f76fc0b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 -- 172.21.15.46:0/1973242782 shutdown_connections 2024-05-09T03:21:24.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.556+0000 7f770136c640 1 -- 172.21.15.46:0/1973242782 wait complete. 2024-05-09T03:21:24.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.558+0000 7f770136c640 1 Processor -- start 2024-05-09T03:21:24.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.558+0000 7f770136c640 1 -- start start 2024-05-09T03:21:24.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f770136c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76fc0c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f770136c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc15c100 0x7f76fc164300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f770136c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc164840 0x7f76fc0b69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f770136c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc08dac0 con 0x7f76e8005250 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f770136c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f76fc06f480 con 0x7f76fc15c100 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f770136c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f76fc08cd60 con 0x7f76fc164840 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f76faffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76fc0c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f76faffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76fc0c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46530/0 (socket says 172.21.15.46:46530) 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f76faffd640 1 -- 172.21.15.46:0/1183098279 learned_addr learned my addr 172.21.15.46:0/1183098279 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f76fb7fe640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc164840 0x7f76fc0b69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.559+0000 7f76fa7fc640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc15c100 0x7f76fc164300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76faffd640 1 -- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc164840 msgr2=0x7f76fc0b69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76faffd640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc164840 0x7f76fc0b69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76faffd640 1 -- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc15c100 msgr2=0x7f76fc164300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76faffd640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc15c100 0x7f76fc164300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76faffd640 1 -- 172.21.15.46:0/1183098279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76fc164c50 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76fa7fc640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc15c100 0x7f76fc164300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76faffd640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76fc0c76a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009f40 tx=0x7f76ec00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec019070 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76ec003b90 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76fc0c7be0 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76ec00fea0 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.560+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76fc1f9bf0 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.561+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76fc06a270 con 0x7f76e8005250 2024-05-09T03:21:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.562+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f76ec01c070 con 0x7f76e8005250 2024-05-09T03:21:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.562+0000 7f76f37fe640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76dc076300 0x7f76dc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.562+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f76ec098320 con 0x7f76e8005250 2024-05-09T03:21:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.562+0000 7f76fa7fc640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76dc076300 0x7f76dc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.563+0000 7f76fa7fc640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76dc076300 0x7f76dc078810 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f76fc15ed80 tx=0x7f76e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.567+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f76fc164c50 con 0x7f76e8005250 2024-05-09T03:21:24.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.797+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f76fc0ce4a0 con 0x7f76e8005250 2024-05-09T03:21:24.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.797+0000 7f76f37fe640 1 -- 172.21.15.46:0/1183098279 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v359) ==== 53+0+73 (secure 0 0 0) 0x7f76ec010040 con 0x7f76e8005250 2024-05-09T03:21:24.798 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T03:21:24.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76dc076300 msgr2=0x7f76dc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76dc076300 0x7f76dc078810 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f76fc15ed80 tx=0x7f76e4009040 comp rx=0 tx=0).stop 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 msgr2=0x7f76fc0c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76fc0c76a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009f40 tx=0x7f76ec00f580 comp rx=0 tx=0).stop 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 shutdown_connections 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f76fc164840 0x7f76fc0b69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76fc15c100 0x7f76fc164300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f76dc076300 0x7f76dc078810 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 --2- 172.21.15.46:0/1183098279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76e8005250 0x7f76fc0c76a0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 >> 172.21.15.46:0/1183098279 conn(0x7f76fc0b5660 msgr2=0x7f76fc1627f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 shutdown_connections 2024-05-09T03:21:24.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:24.799+0000 7f770136c640 1 -- 172.21.15.46:0/1183098279 wait complete. 2024-05-09T03:21:24.811 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T03:21:24.811 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T03:21:24.811 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-05-09T03:21:24.811 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T03:21:24.812 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:24.812 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T03:21:24.812 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T03:21:24.812 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T03:21:24.812 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:21:24.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:24.812 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-05-09T03:21:24.828 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T03:21:24.828 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T03:21:24.828 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41833sec preferred_lft 41833sec 2024-05-09T03:21:24.829 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::84b7:58ff:fe6f:66ec/64 scope link 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether c2:9a:3a:40:1e:a4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:24.830 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::c09a:3aff:fe40:1ea4/64 scope link 2024-05-09T03:21:24.831 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.831 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-05-09T03:21:24.831 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether fe:95:3c:87:fe:60 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:24.831 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::fc95:3cff:fe87:fe60/64 scope link 2024-05-09T03:21:24.831 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-05-09T03:21:24.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:24.831 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T03:21:24.874 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:21:24.874+0000 7f7bda2f6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:21:24.892 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-05-09T03:21:24.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:24.894 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-05-09T03:21:24.951 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-05-09T03:21:24.951 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T03:21:24.951 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T03:21:24.951 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T03:21:24.951 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T03:21:24.952 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T03:21:24.952 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-05-09T03:21:24.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T03:21:24.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T03:21:24.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T03:21:24.952 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-05-09T03:21:25.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T03:21:25.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:25.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.083 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:21:25.099 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:21:25.099 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:21:25.099 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:21:25.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-05-09T03:21:25.166 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T03:21:25.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:21:25.166 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:25.222 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T03:21:25.222 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T03:21:25.223 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:25.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:21:25.345 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:21:25.402 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:21:25.403 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:25.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.405 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:21:25.461 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T03:21:25.461 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T03:21:25.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-05-09T03:21:25.528 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-05-09T03:21:25.587 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-05-09T03:21:25.588 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:25.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.589 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-05-09T03:21:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-05-09T03:21:25.599+0000 7fb4ab51a480 -1 init, newargv = 0x7fb498001250 newargc=15 2024-05-09T03:21:25.599 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[89299]: starting ceph client 2024-05-09T03:21:25.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[89299]: starting fuse 2024-05-09T03:21:25.615 INFO:teuthology.orchestra.run.smithi046.stdout:42 2024-05-09T03:21:25.616 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-05-09T03:21:25.616 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> 2024-05-09T03:21:25.617 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-05-09T03:21:25.618 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-05-09T03:21:25.738 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.89299.asok 2024-05-09T03:21:25.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T03:21:25.748 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.89299.asok 2024-05-09T03:21:25.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.89299.asok status 2024-05-09T03:21:25.897 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:25.897 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-05-09T03:21:25.897 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T03:21:25.897 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T03:21:25.897 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-05-09T03:21:25.897 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "89299", 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 10884, 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-05-09T03:21:25.898 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 10884 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3112558098 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3112558098 2024-05-09T03:21:25.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.10884 192.168.0.1:0/3112558098", 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3112558098", 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 359, 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 136, 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-05-09T03:21:25.900 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:25.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.906 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-05-09T03:21:25.906 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:25.923 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-05-09T03:21:25.923 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:25.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:25.923 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:25.988 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-05-09T03:21:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.084+0000 7f26bb645f40 1 Processor -- start 2024-05-09T03:21:26.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.084+0000 7f26bb645f40 1 -- start start 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26bb645f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77158480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26bb645f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771725f0 0x559e771f5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26bb645f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771fda40 0x559e77154de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26bb645f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559e771735f0 con 0x559e7716a3a0 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26bb645f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559e770d0f50 con 0x559e771fda40 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26bb645f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559e771738d0 con 0x559e771725f0 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41372/0 (socket says 172.21.15.46:41372) 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 -- 172.21.15.46:0/3121292995 learned_addr learned my addr 172.21.15.46:0/3121292995 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b590e640 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771fda40 0x559e77154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b490c640 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771725f0 0x559e771f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 -- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771725f0 msgr2=0x559e771f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771725f0 0x559e771f5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 -- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771fda40 msgr2=0x559e77154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771fda40 0x559e77154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 -- 172.21.15.46:0/3121292995 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e771f6030 con 0x559e7716a3a0 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.086+0000 7f26b510d640 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77158480 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f26a8054040 tx=0x7f26a8050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897613d2a848cae4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26af7fe640 1 -- 172.21.15.46:0/3121292995 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8055040 con 0x559e7716a3a0 2024-05-09T03:21:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26af7fe640 1 -- 172.21.15.46:0/3121292995 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a8051ac0 con 0x559e7716a3a0 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26af7fe640 1 -- 172.21.15.46:0/3121292995 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8065480 con 0x559e7716a3a0 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 -- 172.21.15.46:0/3121292995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 msgr2=0x559e77158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77158480 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f26a8054040 tx=0x7f26a8050000 comp rx=0 tx=0).stop 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 -- 172.21.15.46:0/3121292995 shutdown_connections 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771fda40 0x559e77154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771725f0 0x559e771f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 --2- 172.21.15.46:0/3121292995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77158480 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 -- 172.21.15.46:0/3121292995 >> 172.21.15.46:0/3121292995 conn(0x559e7704cac0 msgr2=0x559e7716d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 -- 172.21.15.46:0/3121292995 shutdown_connections 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.087+0000 7f26bb645f40 1 -- 172.21.15.46:0/3121292995 wait complete. 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.088+0000 7f26bb645f40 1 Processor -- start 2024-05-09T03:21:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.088+0000 7f26bb645f40 1 -- start start 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.089+0000 7f26bb645f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77200190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771725f0 0x559e771f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771fda40 0x559e77291a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559e77170e00 con 0x559e7716a3a0 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559e7728f890 con 0x559e771725f0 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559e771fc970 con 0x559e771fda40 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77200190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77200190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41388/0 (socket says 172.21.15.46:41388) 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 -- 172.21.15.46:0/1231798190 learned_addr learned my addr 172.21.15.46:0/1231798190 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b590e640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771fda40 0x559e77291a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b490c640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771725f0 0x559e771f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 -- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771fda40 msgr2=0x559e77291a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771fda40 0x559e77291a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 -- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771725f0 msgr2=0x559e771f4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771725f0 0x559e771f4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 -- 172.21.15.46:0/1231798190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e772006d0 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b590e640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559e771fda40 0x559e77291a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b490c640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e771725f0 0x559e771f4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26b510d640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77200190 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f26a8025980 tx=0x7f26a8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26ae7fc640 1 -- 172.21.15.46:0/1231798190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8055070 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26ae7fc640 1 -- 172.21.15.46:0/1231798190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a8065480 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559e77171540 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26ae7fc640 1 -- 172.21.15.46:0/1231798190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8065850 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.090+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e7728f560 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.092+0000 7f26ae7fc640 1 -- 172.21.15.46:0/1231798190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26a8058020 con 0x559e7716a3a0 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.092+0000 7f26ae7fc640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2698074cb0 0x7f26980771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.092+0000 7f26ae7fc640 1 -- 172.21.15.46:0/1231798190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f26a80db3f0 con 0x559e7716a3a0 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26b490c640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2698074cb0 0x7f26980771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26bb645f40 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x559e77155650 0x559e7729b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 15.7 15:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136) -- 0x559e771f4fb0 con 0x559e77155650 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26b590e640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x559e77155650 0x559e7729b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26b590e640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x559e77155650 0x559e7729b230 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26b490c640 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2698074cb0 0x7f26980771c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f26a0000c00 tx=0x7f26a0001720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.093+0000 7f26b590e640 1 -- 172.21.15.46:0/1231798190 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (91) out=91b] v0'0 uv6 ondisk = 0) ==== 164+0+91 (crc 0 0 0) 0x7f26b0009a60 con 0x559e77155650 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x559e77155650 msgr2=0x559e7729b230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x559e77155650 0x559e7729b230 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2698074cb0 msgr2=0x7f26980771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2698074cb0 0x7f26980771c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f26a0000c00 tx=0x7f26a0001720 comp rx=0 tx=0).stop 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 msgr2=0x559e77200190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 --2- 172.21.15.46:0/1231798190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559e7716a3a0 0x559e77200190 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f26a8025980 tx=0x7f26a8003330 comp rx=0 tx=0).stop 2024-05-09T03:21:26.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26b590e640 1 -- 172.21.15.46:0/1231798190 reap_dead start 2024-05-09T03:21:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 shutdown_connections 2024-05-09T03:21:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 >> 172.21.15.46:0/1231798190 conn(0x559e7704cac0 msgr2=0x559e771fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 shutdown_connections 2024-05-09T03:21:26.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.094+0000 7f26bb645f40 1 -- 172.21.15.46:0/1231798190 wait complete. 2024-05-09T03:21:26.100 INFO:teuthology.orchestra.run.smithi046.stdout:Ufile1dir1 2024-05-09T03:21:26.100 DEBUG:teuthology.orchestra.run.smithi046:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-05-09T03:21:26.207 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:26.207 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511627777, 2024-05-09T03:21:26.207 INFO:teuthology.orchestra.run.smithi046.stdout: "ancestors": [ 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "dirino": 1099511627776, 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "dname": "file1", 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "version": 6 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "dirino": 1, 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "dname": "dir1", 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "version": 19 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "pool": 15, 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: "old_pools": [] 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:26.208 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:26.223 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent rhubarb 2024-05-09T03:21:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.280+0000 7f64bc91ff40 1 Processor -- start 2024-05-09T03:21:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.280+0000 7f64bc91ff40 1 -- start start 2024-05-09T03:21:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64bc91ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 0x5583b4854100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64bc91ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583b47c0d30 0x5583b48388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64bc91ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b4839330 0x5583b4839740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64bc91ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5583b4724f50 con 0x7f64a40022f0 2024-05-09T03:21:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64bc91ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5583b4728f90 con 0x5583b47c0d30 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64bc91ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5583b47c78e0 con 0x5583b4839330 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 0x5583b4854100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 0x5583b4854100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41408/0 (socket says 172.21.15.46:41408) 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 -- 172.21.15.46:0/3474742535 learned_addr learned my addr 172.21.15.46:0/3474742535 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5160640 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583b47c0d30 0x5583b48388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b6162640 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b4839330 0x5583b4839740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 -- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b4839330 msgr2=0x5583b4839740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b4839330 0x5583b4839740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 -- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583b47c0d30 msgr2=0x5583b48388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583b47c0d30 0x5583b48388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.281+0000 7f64b5961640 1 -- 172.21.15.46:0/3474742535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b47bf070 con 0x7f64a40022f0 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64b5961640 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 0x5583b4854100 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f64a8059040 tx=0x7f64a804f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bf730f4138890 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64affff640 1 -- 172.21.15.46:0/3474742535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a80052c0 con 0x7f64a40022f0 2024-05-09T03:21:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64affff640 1 -- 172.21.15.46:0/3474742535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a8066070 con 0x7f64a40022f0 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64affff640 1 -- 172.21.15.46:0/3474742535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a804d910 con 0x7f64a40022f0 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3474742535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 msgr2=0x5583b4854100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 0x5583b4854100 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f64a8059040 tx=0x7f64a804f000 comp rx=0 tx=0).stop 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3474742535 shutdown_connections 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b4839330 0x5583b4839740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5583b47c0d30 0x5583b48388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3474742535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a40022f0 0x5583b4854100 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3474742535 >> 172.21.15.46:0/3474742535 conn(0x5583b46a0ac0 msgr2=0x5583b47c84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3474742535 shutdown_connections 2024-05-09T03:21:26.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.282+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3474742535 wait complete. 2024-05-09T03:21:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.283+0000 7f64bc91ff40 1 Processor -- start 2024-05-09T03:21:26.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.283+0000 7f64bc91ff40 1 -- start start 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b47c0d30 0x5583b48373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583b4839330 0x5583b48da5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a40022f0 0x5583b48eb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5583b47c1da0 con 0x5583b4839330 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5583b47bf350 con 0x7f64a40022f0 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5583b48ec1c0 con 0x5583b47c0d30 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583b4839330 0x5583b48da5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583b4839330 0x5583b48da5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41414/0 (socket says 172.21.15.46:41414) 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 -- 172.21.15.46:0/3085172836 learned_addr learned my addr 172.21.15.46:0/3085172836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b6162640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a40022f0 0x5583b48eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5961640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b47c0d30 0x5583b48373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 -- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b47c0d30 msgr2=0x5583b48373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b47c0d30 0x5583b48373e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 -- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a40022f0 msgr2=0x5583b48eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a40022f0 0x5583b48eb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 -- 172.21.15.46:0/3085172836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b47ab740 con 0x5583b4839330 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b6162640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64a40022f0 0x5583b48eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5961640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x5583b47c0d30 0x5583b48373e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64b5160640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583b4839330 0x5583b48da5f0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f64a0001d10 tx=0x7f64a000f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64aeffd640 1 -- 172.21.15.46:0/3085172836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0016020 con 0x5583b4839330 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64aeffd640 1 -- 172.21.15.46:0/3085172836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a0001e00 con 0x5583b4839330 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64aeffd640 1 -- 172.21.15.46:0/3085172836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0012680 con 0x5583b4839330 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.285+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5583b47aba00 con 0x5583b4839330 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.286+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b48eefe0 con 0x5583b4839330 2024-05-09T03:21:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.287+0000 7f64aeffd640 1 -- 172.21.15.46:0/3085172836 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f64a0019020 con 0x5583b4839330 2024-05-09T03:21:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64aeffd640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6498074ce0 0x7f64980771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64aeffd640 1 -- 172.21.15.46:0/3085172836 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f64a0096450 con 0x5583b4839330 2024-05-09T03:21:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5583b47ac510 0x5583b47ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64b5961640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6498074ce0 0x7f64980771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- osd_op(unknown.0.0:1 15.7 15:e2e07930:::10000000001.00000000:head [setxattr parent (7) in=13b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136) -- 0x5583b4837bb0 con 0x5583b47ac510 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64b6162640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5583b47ac510 0x5583b47ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64b6162640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5583b47ac510 0x5583b47ac920 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.288+0000 7f64b5961640 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6498074ce0 0x7f64980771f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f64a8062000 tx=0x7f64a8051000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64b6162640 1 -- 172.21.15.46:0/3085172836 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (7)] v136'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f64b000b350 con 0x5583b47ac510 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5583b47ac510 msgr2=0x5583b47ac920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x5583b47ac510 0x5583b47ac920 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6498074ce0 msgr2=0x7f64980771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f6498074ce0 0x7f64980771f0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f64a8062000 tx=0x7f64a8051000 comp rx=0 tx=0).stop 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583b4839330 msgr2=0x5583b48da5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 --2- 172.21.15.46:0/3085172836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5583b4839330 0x5583b48da5f0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f64a0001d10 tx=0x7f64a000f6b0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64b6162640 1 -- 172.21.15.46:0/3085172836 reap_dead start 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 shutdown_connections 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 >> 172.21.15.46:0/3085172836 conn(0x5583b46a0ac0 msgr2=0x5583b47c84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 shutdown_connections 2024-05-09T03:21:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.290+0000 7f64bc91ff40 1 -- 172.21.15.46:0/3085172836 wait complete. 2024-05-09T03:21:26.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:26.293 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink1) 2024-05-09T03:21:26.312 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:21:26.311+0000 7f7bd62ee640 -1 mds.0.cache corrupt backtrace on ino x00x10000000001: void inode_backtrace_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 5 < 104: Malformed input [buffer:3] 2024-05-09T03:21:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:21:26.313 INFO:teuthology.orchestra.run.smithi046.stderr:stat: cannot statx 'testdir/hardlink1': Input/output error 2024-05-09T03:21:26.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:26.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.468+0000 7fdca6bb4640 1 Processor -- start 2024-05-09T03:21:26.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.468+0000 7fdca6bb4640 1 -- start start 2024-05-09T03:21:26.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca6bb4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca6bb4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdca00d2480 0x7fdca00d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca6bb4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca0160b20 con 0x7fdca0164c70 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca6bb4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdca015e520 con 0x7fdca00d2480 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca6bb4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdca0161150 con 0x7fdca01615e0 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca4950640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdca0164c70 0x7fdca00d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53478/0 (socket says 172.21.15.46:53478) 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca4950640 1 -- 172.21.15.46:0/3400631645 learned_addr learned my addr 172.21.15.46:0/3400631645 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdc9ffff640 1 --2- 172.21.15.46:0/3400631645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdca5151640 1 --2- 172.21.15.46:0/3400631645 >> v2:172.21.15.88:3300/0 conn(0x7fdca00d2480 0x7fdca00d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.469+0000 7fdc9f7fe640 1 -- 172.21.15.46:0/3400631645 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797245252 0 0) 0x7fdca0160b20 con 0x7fdca0164c70 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.470+0000 7fdc9f7fe640 1 -- 172.21.15.46:0/3400631645 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc8c003520 con 0x7fdca0164c70 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.470+0000 7fdc9ffff640 1 -- 172.21.15.46:0/3400631645 >> v2:172.21.15.88:3300/0 conn(0x7fdca00d2480 msgr2=0x7fdca00d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.470+0000 7fdc9ffff640 1 --2- 172.21.15.46:0/3400631645 >> v2:172.21.15.88:3300/0 conn(0x7fdca00d2480 0x7fdca00d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.470+0000 7fdc9f7fe640 1 -- 172.21.15.46:0/3400631645 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 131102618 0 0) 0x7fdc8c003520 con 0x7fdca0164c70 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.470+0000 7fdc9ffff640 1 -- 172.21.15.46:0/3400631645 >> v1:172.21.15.46:6789/0 conn(0x7fdca0164c70 legacy=0x7fdca00d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.470+0000 7fdc9ffff640 1 -- 172.21.15.46:0/3400631645 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca00d15f0 con 0x7fdca01615e0 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.471+0000 7fdc9ffff640 1 --2- 172.21.15.46:0/3400631645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca015dfe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009d90 tx=0x7fdc9002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d5186f7b1bcf6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.471+0000 7fdc9f7fe640 1 -- 172.21.15.46:0/3400631645 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc880016f0 con 0x7fdca01615e0 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.471+0000 7fdc9f7fe640 1 -- 172.21.15.46:0/3400631645 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc90007540 con 0x7fdca01615e0 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.471+0000 7fdc9f7fe640 1 -- 172.21.15.46:0/3400631645 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc900078f0 con 0x7fdca01615e0 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.471+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3400631645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 msgr2=0x7fdca015dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.471+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3400631645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca015dfe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009d90 tx=0x7fdc9002a5f0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.472+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3400631645 shutdown_connections 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.472+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3400631645 >> v2:172.21.15.88:3300/0 conn(0x7fdca00d2480 0x7fdca00d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.472+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3400631645 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca015dfe0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.472+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3400631645 >> 172.21.15.46:0/3400631645 conn(0x7fdca0073db0 msgr2=0x7fdca00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.472+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3400631645 shutdown_connections 2024-05-09T03:21:26.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.472+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3400631645 wait complete. 2024-05-09T03:21:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.473+0000 7fdca6bb4640 1 Processor -- start 2024-05-09T03:21:26.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.473+0000 7fdca6bb4640 1 -- start start 2024-05-09T03:21:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca6bb4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca00d2480 0x7fdca014aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca6bb4640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca01f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca6bb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 0x7fdca0149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca6bb4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdca01619f0 con 0x7fdca0164c70 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca6bb4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdca0160de0 con 0x7fdca00d2480 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca6bb4640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdca00d2dd0 con 0x7fdca01615e0 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 0x7fdca0149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 0x7fdca0149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41428/0 (socket says 172.21.15.46:41428) 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 -- 172.21.15.46:0/3119802018 learned_addr learned my addr 172.21.15.46:0/3119802018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdc9ffff640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca01f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca4950640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca00d2480 0x7fdca014aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 -- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 msgr2=0x7fdca01f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca01f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 -- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca00d2480 msgr2=0x7fdca014aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca00d2480 0x7fdca014aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 -- 172.21.15.46:0/3119802018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca01485f0 con 0x7fdca0164c70 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.475+0000 7fdca5151640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 0x7fdca0149690 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800e7f0 tx=0x7fdc9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.476+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98017070 con 0x7fdca0164c70 2024-05-09T03:21:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.476+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9800c730 con 0x7fdca0164c70 2024-05-09T03:21:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.476+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca01488d0 con 0x7fdca0164c70 2024-05-09T03:21:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.476+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9800cb20 con 0x7fdca0164c70 2024-05-09T03:21:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.476+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca0148e90 con 0x7fdca0164c70 2024-05-09T03:21:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.477+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca008cfd0 con 0x7fdca0164c70 2024-05-09T03:21:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.477+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdc9801a070 con 0x7fdca0164c70 2024-05-09T03:21:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.478+0000 7fdc9d7fa640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.478+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fdc98097770 con 0x7fdca0164c70 2024-05-09T03:21:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.478+0000 7fdca4950640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.479+0000 7fdca4950640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdc88005880 tx=0x7fdc88001c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.483+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdca008cfd0 con 0x7fdca0164c70 2024-05-09T03:21:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.718+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdca00d43a0 con 0x7fdca0164c70 2024-05-09T03:21:26.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.718+0000 7fdc9d7fa640 1 -- 172.21.15.46:0/3119802018 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) ==== 95+0+5190 (secure 0 0 0) 0x7fdc98060a20 con 0x7fdca0164c70 2024-05-09T03:21:26.719 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:26.720 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":359,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:22.873173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 359 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 msgr2=0x7fdc800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdc88005880 tx=0x7fdc88001c10 comp rx=0 tx=0).stop 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 msgr2=0x7fdca0149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 0x7fdca0149690 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800e7f0 tx=0x7fdc9800ecb0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 shutdown_connections 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fdc800762d0 0x7fdc800787e0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdca0164c70 0x7fdca0149690 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdca01615e0 0x7fdca01f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 --2- 172.21.15.46:0/3119802018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdca00d2480 0x7fdca014aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 >> 172.21.15.46:0/3119802018 conn(0x7fdca0073db0 msgr2=0x7fdca015d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 shutdown_connections 2024-05-09T03:21:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.721+0000 7fdca6bb4640 1 -- 172.21.15.46:0/3119802018 wait complete. 2024-05-09T03:21:26.733 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-05-09T03:21:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.848+0000 7fd1cc829640 1 Processor -- start 2024-05-09T03:21:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.848+0000 7fd1cc829640 1 -- start start 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1cc829640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1c8164bf0 0x7fd1c80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1cc829640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1cc829640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1c80b52b0 0x7fd1c80b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1cc829640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c8164760 con 0x7fd1c8164bf0 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1cc829640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c80d17a0 con 0x7fd1c80b52b0 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1cc829640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c80d14c0 con 0x7fd1c80d0fb0 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1c8164bf0 0x7fd1c80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd1c8164bf0 0x7fd1c80d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41444/0 (socket says 172.21.15.46:41444) 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 -- 172.21.15.46:0/4101927483 learned_addr learned my addr 172.21.15.46:0/4101927483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/4101927483 >> v2:172.21.15.88:3300/0 conn(0x7fd1c80b52b0 0x7fd1c80b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 -- 172.21.15.46:0/4101927483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 msgr2=0x7fd1c80b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 -- 172.21.15.46:0/4101927483 >> v2:172.21.15.88:3300/0 conn(0x7fd1c80b52b0 msgr2=0x7fd1c80b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 --2- 172.21.15.46:0/4101927483 >> v2:172.21.15.88:3300/0 conn(0x7fd1c80b52b0 0x7fd1c80b7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c659c640 1 -- 172.21.15.46:0/4101927483 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c8164110 con 0x7fd1c8164bf0 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.850+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/4101927483 >> v2:172.21.15.88:3300/0 conn(0x7fd1c80b52b0 0x7fd1c80b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.851+0000 7fd1c659c640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c8164bf0 0x7fd1c80d0a70 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80b8f60 tx=0x7fd1c002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb58c00131eb645c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.851+0000 7fd1c559a640 1 -- 172.21.15.46:0/4101927483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0037070 con 0x7fd1c8164bf0 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.851+0000 7fd1c559a640 1 -- 172.21.15.46:0/4101927483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0032440 con 0x7fd1c8164bf0 2024-05-09T03:21:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.851+0000 7fd1c559a640 1 -- 172.21.15.46:0/4101927483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0032810 con 0x7fd1c8164bf0 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 -- 172.21.15.46:0/4101927483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c8164bf0 msgr2=0x7fd1c80d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c8164bf0 0x7fd1c80d0a70 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80b8f60 tx=0x7fd1c002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 -- 172.21.15.46:0/4101927483 shutdown_connections 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 --2- 172.21.15.46:0/4101927483 >> v2:172.21.15.88:3300/0 conn(0x7fd1c80b52b0 0x7fd1c80b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 --2- 172.21.15.46:0/4101927483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c8164bf0 0x7fd1c80d0a70 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 -- 172.21.15.46:0/4101927483 >> 172.21.15.46:0/4101927483 conn(0x7fd1c80d2920 msgr2=0x7fd1c80d2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 -- 172.21.15.46:0/4101927483 shutdown_connections 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.852+0000 7fd1cc829640 1 -- 172.21.15.46:0/4101927483 wait complete. 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.854+0000 7fd1cc829640 1 Processor -- start 2024-05-09T03:21:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.854+0000 7fd1cc829640 1 -- start start 2024-05-09T03:21:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.855+0000 7fd1cc829640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 0x7fd1c80cd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.855+0000 7fd1cc829640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.855+0000 7fd1cc829640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1c8164bf0 0x7fd1c8154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.855+0000 7fd1cc829640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c806ecb0 con 0x7fd1c80b52b0 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.855+0000 7fd1cc829640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c806eb30 con 0x7fd1c8164bf0 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.855+0000 7fd1cc829640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c80d47c0 con 0x7fd1c80d0fb0 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 0x7fd1c80cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 0x7fd1c80cd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41460/0 (socket says 172.21.15.46:41460) 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 -- 172.21.15.46:0/2364847944 learned_addr learned my addr 172.21.15.46:0/2364847944 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1c8164bf0 0x7fd1c8154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 -- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 msgr2=0x7fd1c80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 -- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1c8164bf0 msgr2=0x7fd1c8154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1c8164bf0 0x7fd1c8154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 -- 172.21.15.46:0/2364847944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c80cdab0 con 0x7fd1c80b52b0 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1c8164bf0 0x7fd1c8154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 0x7fd1c80cd410 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fd1c002a220 tx=0x7fd1c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1beffd640 1 -- 172.21.15.46:0/2364847944 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0037040 con 0x7fd1c80b52b0 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1beffd640 1 -- 172.21.15.46:0/2364847944 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0042070 con 0x7fd1c80b52b0 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c80cdd90 con 0x7fd1c80b52b0 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1beffd640 1 -- 172.21.15.46:0/2364847944 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0030bb0 con 0x7fd1c80b52b0 2024-05-09T03:21:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.856+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c80ce350 con 0x7fd1c80b52b0 2024-05-09T03:21:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.858+0000 7fd1beffd640 1 -- 172.21.15.46:0/2364847944 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd1c0047460 con 0x7fd1c80b52b0 2024-05-09T03:21:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.859+0000 7fd1beffd640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1a80762d0 0x7fd1a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.859+0000 7fd1beffd640 1 -- 172.21.15.46:0/2364847944 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fd1c00b83a0 con 0x7fd1c80b52b0 2024-05-09T03:21:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.859+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1a80762d0 0x7fd1a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.859+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1a80762d0 0x7fd1a80787e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0006fd0 tx=0x7fd1b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.860+0000 7fd1d37b2740 1 Processor -- start 2024-05-09T03:21:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.860+0000 7fd1d37b2740 1 -- start start 2024-05-09T03:21:26.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b954350 0x55871bd1c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 0x55871bcf9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bcfa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55871bc5cd10 con 0x55871b92b490 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55871bbe4750 con 0x55871b954350 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55871bc58cd0 con 0x55871b933670 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1c5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 0x55871bcf9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1c5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 0x55871bcf9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41472/0 (socket says 172.21.15.46:41472) 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2999050165 learned_addr learned my addr 172.21.15.46:0/2999050165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b954350 0x55871bd1c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.861+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bcfa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 msgr2=0x55871bcfa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bcfa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b954350 msgr2=0x55871bd1c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b954350 0x55871bd1c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2999050165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55871ba36790 con 0x55871b92b490 2024-05-09T03:21:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b954350 0x55871bd1c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 0x55871bcf9f10 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000db90 tx=0x7fd1b0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a859f8c0b472c83 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1a7fff640 1 -- 172.21.15.46:0/2999050165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0019050 con 0x55871b92b490 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1a7fff640 1 -- 172.21.15.46:0/2999050165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b0023070 con 0x55871b92b490 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.862+0000 7fd1a7fff640 1 -- 172.21.15.46:0/2999050165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0012de0 con 0x55871b92b490 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2999050165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 msgr2=0x55871bcf9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 0x55871bcf9f10 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000db90 tx=0x7fd1b0010000 comp rx=0 tx=0).stop 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2999050165 shutdown_connections 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bcfa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b92b490 0x55871bcf9f10 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2999050165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b954350 0x55871bd1c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2999050165 >> 172.21.15.46:0/2999050165 conn(0x55871b9484a0 msgr2=0x55871bcf76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2999050165 shutdown_connections 2024-05-09T03:21:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.863+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2999050165 wait complete. 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.864+0000 7fd1d37b2740 1 -- start start 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.864+0000 7fd1d37b2740 1 Processor -- start 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd1ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bd1e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd1e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55871bc19bd0 con 0x55871b954350 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55871bd3d960 con 0x55871b92b490 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55871bd202c0 con 0x55871b933670 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1d37b2740 10 client.0 _ll_register_callbacks cb 0x55871bc0e210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd1e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd1e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41484/0 (socket says 172.21.15.46:41484) 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.865+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2537190245 learned_addr learned my addr 172.21.15.46:0/2537190245 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd1ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bd1e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 msgr2=0x55871bd1e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bd1e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 msgr2=0x55871bd1ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd1ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2537190245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55871b905130 con 0x55871b954350 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd1e5a0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fd1b800fb90 tx=0x7fd1b800d740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d0114da50045a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2537190245 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b801c060 con 0x55871b954350 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2537190245 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fd1b800b7f0 con 0x55871b954350 2024-05-09T03:21:26.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1b0023070 con 0x55871b954350 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2537190245 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7fd1b8022050 con 0x55871b954350 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1d37b2740 10 client.10917 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x55871ba365c0 con 0x55871b954350 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.866+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2537190245 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b0023070 con 0x55871b954350 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2537190245 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7fd1b8025020 con 0x55871b954350 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1d37b2740 10 client.10917 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1d37b2740 10 client.10917 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1d37b2740 20 client.10917 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871b913130 0x55871bd1c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1d37b2740 4 client.10917 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55871bc6a920 con 0x55871b913130 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871b913130 0x55871bd1c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1a57fa640 10 client.10917 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.867+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871b913130 0x55871bd1c860 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=589aecdd63ff3203 server_cookie=f7605f16d2881188 in_seq=0 out_seq=0 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2537190245 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55871bc6a920 con 0x55871b913130 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1a57fa640 10 client.10917 handle_command_reply: tid=0 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 client.10917 shutdown 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 msgr2=0x55871bd1e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd1e5a0 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fd1b800fb90 tx=0x7fd1b800d740 comp rx=0 tx=0).stop 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 shutdown_connections 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871b913130 0x55871bd1c860 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd1e5a0 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b933670 0x55871bd1e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2537190245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd1ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1a57fa640 0 client.10917 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 >> 172.21.15.46:0/2537190245 conn(0x55871b9484a0 msgr2=0x55871bcf9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 shutdown_connections 2024-05-09T03:21:26.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2537190245 wait complete. 2024-05-09T03:21:26.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.868+0000 7fd1d37b2740 20 client.10917 trim_cache size 0 max 16384 2024-05-09T03:21:26.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.883+0000 7fd1d37b2740 1 Processor -- start 2024-05-09T03:21:26.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.883+0000 7fd1d37b2740 1 -- start start 2024-05-09T03:21:26.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b92b490 0x55871bd3b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b933670 0x55871bd3b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd4f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c8202db0 con 0x55871b954350 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c8201e20 con 0x55871b933670 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c8162370 con 0x55871b92b490 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd4f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd4f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41492/0 (socket says 172.21.15.46:41492) 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.884+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2168289241 learned_addr learned my addr 172.21.15.46:0/2168289241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b92b490 0x55871bd3b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b933670 0x55871bd3b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b92b490 msgr2=0x55871bd3b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b92b490 0x55871bd3b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b933670 msgr2=0x55871bd3b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b933670 0x55871bd3b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c6d9d640 1 -- 172.21.15.46:0/2168289241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55871b938150 con 0x55871b954350 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b92b490 0x55871bd3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b933670 0x55871bd3b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd4f610 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8008bb0 tx=0x7fd1b8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549036443d761aa4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1a77fe640 1 -- 172.21.15.46:0/2168289241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800ba40 con 0x55871b954350 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1a77fe640 1 -- 172.21.15.46:0/2168289241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b800bd40 con 0x55871b954350 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.885+0000 7fd1a77fe640 1 -- 172.21.15.46:0/2168289241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800b5f0 con 0x55871b954350 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2168289241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 msgr2=0x55871bd4f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd4f610 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8008bb0 tx=0x7fd1b8012000 comp rx=0 tx=0).stop 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2168289241 shutdown_connections 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b954350 0x55871bd4f610 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b933670 0x55871bd3b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2168289241 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b92b490 0x55871bd3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2168289241 >> 172.21.15.46:0/2168289241 conn(0x55871b9484a0 msgr2=0x55871bd1ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2168289241 shutdown_connections 2024-05-09T03:21:26.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.886+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2168289241 wait complete. 2024-05-09T03:21:26.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.887+0000 7fd1d37b2740 1 -- start start 2024-05-09T03:21:26.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.887+0000 7fd1d37b2740 1 Processor -- start 2024-05-09T03:21:26.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd61160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 0x55871bd4f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b954350 0x55871bcf7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55871bd1b610 con 0x55871b933670 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c81629c0 con 0x55871b92b490 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55871bd3caf0 con 0x55871b954350 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1d37b2740 10 client.0 _ll_register_callbacks cb 0x7fd1b80087d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1c5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 0x55871bd4f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1c5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 0x55871bd4f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41496/0 (socket says 172.21.15.46:41496) 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.888+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2241351377 learned_addr learned my addr 172.21.15.46:0/2241351377 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd61160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b954350 0x55871bcf7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b954350 msgr2=0x55871bcf7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b954350 0x55871bcf7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 msgr2=0x55871bd61160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd61160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c5d9b640 1 -- 172.21.15.46:0/2241351377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55871bd616a0 con 0x55871b933670 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c6d9d640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b954350 0x55871bcf7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1c5d9b640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 0x55871bd4f610 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0008010 tx=0x7fd1b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c742234a7136129 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2241351377 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0035060 con 0x55871b933670 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2241351377 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fd1b001e980 con 0x55871b933670 2024-05-09T03:21:26.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1b800bd40 con 0x55871b933670 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2241351377 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7fd1b0019020 con 0x55871b933670 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1d37b2740 10 client.10923 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.889+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x55871bd61930 con 0x55871b933670 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2241351377 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b800bd40 con 0x55871b933670 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2241351377 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7fd1b002f070 con 0x55871b933670 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 10 client.10923 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 10 client.10923 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 20 client.10923 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871bdac820 0x55871bcf9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 4 client.10923 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55871bc64790 con 0x55871bdac820 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871bdac820 0x55871bcf9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:26.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1a57fa640 10 client.10923 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1c659c640 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871bdac820 0x55871bcf9380 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b963041cc1bcd2bc server_cookie=98e2864f9c37e9d8 in_seq=0 out_seq=0 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1a57fa640 1 -- 172.21.15.46:0/2241351377 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+145 (crc 0 0 0) 0x55871bc64790 con 0x55871bdac820 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1a57fa640 10 client.10923 handle_command_reply: tid=0 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.890+0000 7fd1d37b2740 1 client.10923 shutdown 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 msgr2=0x55871bd4f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 0x55871bd4f610 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0008010 tx=0x7fd1b0004730 comp rx=0 tx=0).stop 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 shutdown_connections 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55871b954350 0x55871bcf7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55871bdac820 0x55871bcf9380 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55871b933670 0x55871bd4f610 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 --2- 172.21.15.46:0/2241351377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55871b92b490 0x55871bd61160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1a57fa640 0 client.10923 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 >> 172.21.15.46:0/2241351377 conn(0x55871b9484a0 msgr2=0x55871bd1ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 shutdown_connections 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 1 -- 172.21.15.46:0/2241351377 wait complete. 2024-05-09T03:21:26.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1d37b2740 20 client.10923 trim_cache size 0 max 16384 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 3454863593, 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511627777, 2024-05-09T03:21:26.898 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/testdir/hardlink1" 2024-05-09T03:21:26.899 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:21:26.899 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:21:26.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1a80762d0 msgr2=0x7fd1a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1cc829640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1a80762d0 0x7fd1a80787e0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0006fd0 tx=0x7fd1b0008040 comp rx=0 tx=0).stop 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 msgr2=0x7fd1c80cd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.891+0000 7fd1cc829640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 0x7fd1c80cd410 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7fd1c002a220 tx=0x7fd1c00046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 shutdown_connections 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1c8164bf0 0x7fd1c8154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fd1c80d0fb0 0x7fd1c80cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fd1a80762d0 0x7fd1a80787e0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 --2- 172.21.15.46:0/2364847944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd1c80b52b0 0x7fd1c80cd410 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 >> 172.21.15.46:0/2364847944 conn(0x7fd1c80d2920 msgr2=0x7fd1c80b6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 shutdown_connections 2024-05-09T03:21:26.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:26.892+0000 7fd1cc829640 1 -- 172.21.15.46:0/2364847944 wait complete. 2024-05-09T03:21:26.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.017+0000 7f120aeb1640 1 Processor -- start 2024-05-09T03:21:27.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.017+0000 7f120aeb1640 1 -- start start 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120aeb1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12040aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120aeb1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f12040af0c0 0x7f1204070670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120aeb1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1204070bb0 0x7f1204073870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120aeb1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1204070fc0 con 0x7f1204070bb0 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120aeb1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f120415d980 con 0x7f12040af0c0 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120aeb1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f120415dc60 con 0x7f12040b4ff0 2024-05-09T03:21:27.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1204070bb0 0x7f1204073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1204070bb0 0x7f1204073870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41506/0 (socket says 172.21.15.46:41506) 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 -- 172.21.15.46:0/1080444502 learned_addr learned my addr 172.21.15.46:0/1080444502 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f11fbfff640 1 --2- 172.21.15.46:0/1080444502 >> v2:172.21.15.88:3300/0 conn(0x7f12040af0c0 0x7f1204070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f1208c4d640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12040aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 -- 172.21.15.46:0/1080444502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 msgr2=0x7f12040aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12040aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 -- 172.21.15.46:0/1080444502 >> v2:172.21.15.88:3300/0 conn(0x7f12040af0c0 msgr2=0x7f1204070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 --2- 172.21.15.46:0/1080444502 >> v2:172.21.15.88:3300/0 conn(0x7f12040af0c0 0x7f1204070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f120944e640 1 -- 172.21.15.46:0/1080444502 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120415df40 con 0x7f1204070bb0 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f11fbfff640 1 --2- 172.21.15.46:0/1080444502 >> v2:172.21.15.88:3300/0 conn(0x7f12040af0c0 0x7f1204070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:27.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.019+0000 7f1208c4d640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12040aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:27.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.020+0000 7f120944e640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f1204073870 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f120406f390 tx=0x7f11fc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0114cc5df4c9717 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.020+0000 7f11fb7fe640 1 -- 172.21.15.46:0/1080444502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc036070 con 0x7f1204070bb0 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.020+0000 7f11fb7fe640 1 -- 172.21.15.46:0/1080444502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11fc031600 con 0x7f1204070bb0 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.020+0000 7f11fb7fe640 1 -- 172.21.15.46:0/1080444502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0319d0 con 0x7f1204070bb0 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 -- 172.21.15.46:0/1080444502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 msgr2=0x7f1204073870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f1204073870 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f120406f390 tx=0x7f11fc02a610 comp rx=0 tx=0).stop 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 -- 172.21.15.46:0/1080444502 shutdown_connections 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f1204073870 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 --2- 172.21.15.46:0/1080444502 >> v2:172.21.15.88:3300/0 conn(0x7f12040af0c0 0x7f1204070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 --2- 172.21.15.46:0/1080444502 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12040aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 -- 172.21.15.46:0/1080444502 >> 172.21.15.46:0/1080444502 conn(0x7f12040b3910 msgr2=0x7f12040b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 -- 172.21.15.46:0/1080444502 shutdown_connections 2024-05-09T03:21:27.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.021+0000 7f120aeb1640 1 -- 172.21.15.46:0/1080444502 wait complete. 2024-05-09T03:21:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.023+0000 7f120aeb1640 1 Processor -- start 2024-05-09T03:21:27.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.023+0000 7f120aeb1640 1 -- start start 2024-05-09T03:21:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120aeb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f120415fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120aeb1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12040af0c0 0x7f12041eadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120aeb1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12041eb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120aeb1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f120415d540 con 0x7f1204070bb0 2024-05-09T03:21:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120aeb1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f12040b5400 con 0x7f12040af0c0 2024-05-09T03:21:27.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120aeb1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f120415d6c0 con 0x7f12040b4ff0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f1208c4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f120415fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f1208c4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f120415fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41512/0 (socket says 172.21.15.46:41512) 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f1208c4d640 1 -- 172.21.15.46:0/2252636519 learned_addr learned my addr 172.21.15.46:0/2252636519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f120944e640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12041eb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.024+0000 7f11fbfff640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12040af0c0 0x7f12041eadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f1208c4d640 1 -- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 msgr2=0x7f12041eb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f1208c4d640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12041eb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f1208c4d640 1 -- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12040af0c0 msgr2=0x7f12041eadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f1208c4d640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12040af0c0 0x7f12041eadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f1208c4d640 1 -- 172.21.15.46:0/2252636519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12041407d0 con 0x7f1204070bb0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f1208c4d640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f120415fe90 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f120000d7d0 tx=0x7f120000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200009640 con 0x7f1204070bb0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1200019070 con 0x7f1204070bb0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1204140ab0 con 0x7f1204070bb0 2024-05-09T03:21:27.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200009a20 con 0x7f1204070bb0 2024-05-09T03:21:27.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.025+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1204141070 con 0x7f1204070bb0 2024-05-09T03:21:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.026+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f120406a270 con 0x7f1204070bb0 2024-05-09T03:21:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.027+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1200009bc0 con 0x7f1204070bb0 2024-05-09T03:21:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.027+0000 7f11f97fa640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11f0076300 0x7f11f0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.028+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f12000979b0 con 0x7f1204070bb0 2024-05-09T03:21:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.028+0000 7f11fbfff640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11f0076300 0x7f11f0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.028+0000 7f11fbfff640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11f0076300 0x7f11f0078810 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f12040b6ae0 tx=0x7f11f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.032+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1200060bc0 con 0x7f1204070bb0 2024-05-09T03:21:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.263+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12040b0bb0 con 0x7f1204070bb0 2024-05-09T03:21:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.263+0000 7f11f97fa640 1 -- 172.21.15.46:0/2252636519 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) ==== 95+0+5190 (secure 0 0 0) 0x7f1200069030 con 0x7f1204070bb0 2024-05-09T03:21:27.264 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:27.264 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":359,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:22.873173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 359 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11f0076300 msgr2=0x7f11f0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11f0076300 0x7f11f0078810 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f12040b6ae0 tx=0x7f11f4008040 comp rx=0 tx=0).stop 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 msgr2=0x7f120415fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f120415fe90 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f120000d7d0 tx=0x7f120000dc90 comp rx=0 tx=0).stop 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 shutdown_connections 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f12040b4ff0 0x7f12041eb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12040af0c0 0x7f12041eadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f11f0076300 0x7f11f0078810 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 --2- 172.21.15.46:0/2252636519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1204070bb0 0x7f120415fe90 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 >> 172.21.15.46:0/2252636519 conn(0x7f12040b3910 msgr2=0x7f120413f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 shutdown_connections 2024-05-09T03:21:27.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.266+0000 7f120aeb1640 1 -- 172.21.15.46:0/2252636519 wait complete. 2024-05-09T03:21:27.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 3454863593 2024-05-09T03:21:27.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.395+0000 7f725f384640 1 Processor -- start 2024-05-09T03:21:27.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.395+0000 7f725f384640 1 -- start start 2024-05-09T03:21:27.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725f384640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f72580b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725f384640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72580cfcf0 0x7f725809cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725f384640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72580b9690 con 0x7f72580d1740 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725f384640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f72580d2b00 con 0x7f72580cfcf0 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725f384640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f725808cd60 con 0x7f72580d1110 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725d120640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f72580d1740 0x7f72580d23f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53490/0 (socket says 172.21.15.46:53490) 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725d120640 1 -- 172.21.15.46:0/3061808763 learned_addr learned my addr 172.21.15.46:0/3061808763 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725c91f640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f72580b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725d921640 1 --2- 172.21.15.46:0/3061808763 >> v2:172.21.15.88:3300/0 conn(0x7f72580cfcf0 0x7f725809cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3986518288 0 0) 0x7f72580b9690 con 0x7f72580d1740 2024-05-09T03:21:27.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7240003520 con 0x7f72580d1740 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1956570671 0 0) 0x7f7240003520 con 0x7f72580d1740 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 msgr2=0x7f72580b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f72580b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 >> v2:172.21.15.88:3300/0 conn(0x7f72580cfcf0 msgr2=0x7f725809cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 --2- 172.21.15.46:0/3061808763 >> v2:172.21.15.88:3300/0 conn(0x7f72580cfcf0 0x7f725809cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72580ca9e0 con 0x7f72580d1740 2024-05-09T03:21:27.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f72500016f0 con 0x7f72580d1740 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f725c91f640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f72580b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 >> v1:172.21.15.46:6789/0 conn(0x7f72580d1740 legacy=0x7f72580d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7240003a00 0x7f7240003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7240004800 0x7f7240004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.397+0000 7f7257fff640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f7240005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72580d2b00 con 0x7f7240003a00 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f725808cd60 con 0x7f7240004800 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f724000ba70 con 0x7f72580d1110 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d921640 1 --2- 172.21.15.46:0/3061808763 >> v2:172.21.15.88:3300/0 conn(0x7f72580cfcf0 0x7f725809cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d120640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7240003a00 0x7f7240003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725c91f640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7240004800 0x7f7240004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d921640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f7240005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d120640 1 -- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 msgr2=0x7f7240005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d120640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f72580d1110 0x7f7240005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d120640 1 -- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7240004800 msgr2=0x7f7240004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d120640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7240004800 0x7f7240004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725d120640 1 -- 172.21.15.46:0/3061808763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7240003520 con 0x7f7240003a00 2024-05-09T03:21:27.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.398+0000 7f725c91f640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7240004800 0x7f7240004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:27.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f725d120640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7240003a00 0x7f7240003e70 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f725000c290 tx=0x7f7240005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b73a2eefbceba14 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250007d50 con 0x7f7240003a00 2024-05-09T03:21:27.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7250005af0 con 0x7f7240003a00 2024-05-09T03:21:27.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f7257fff640 1 -- 172.21.15.46:0/3061808763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250007070 con 0x7f7240003a00 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f725f384640 1 -- 172.21.15.46:0/3061808763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7240003a00 msgr2=0x7f7240003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f725f384640 1 --2- 172.21.15.46:0/3061808763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7240003a00 0x7f7240003e70 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f725000c290 tx=0x7f7240005830 comp rx=0 tx=0).stop 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f725d921640 1 -- 172.21.15.46:0/3061808763 reap_dead start 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f725f384640 1 -- 172.21.15.46:0/3061808763 shutdown_connections 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.399+0000 7f725f384640 1 -- 172.21.15.46:0/3061808763 >> 172.21.15.46:0/3061808763 conn(0x7f72580d05f0 msgr2=0x7f72580d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.400+0000 7f725f384640 1 -- 172.21.15.46:0/3061808763 shutdown_connections 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.400+0000 7f725f384640 1 -- 172.21.15.46:0/3061808763 wait complete. 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.401+0000 7f725f384640 1 Processor -- start 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.401+0000 7f725f384640 1 -- start start 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725f384640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7240003a00 0x7f72581f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725f384640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 0x7f72580ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725f384640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580cd2c0 0x7f725815eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725f384640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72580b93b0 con 0x7f725809d8d0 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725f384640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72580d1ca0 con 0x7f72580cd2c0 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725f384640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f725809d1e0 con 0x7f7240003a00 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 0x7f72580ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 0x7f72580ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41536/0 (socket says 172.21.15.46:41536) 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 -- 172.21.15.46:0/1238999014 learned_addr learned my addr 172.21.15.46:0/1238999014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725d921640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580cd2c0 0x7f725815eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725d120640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7240003a00 0x7f72581f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 -- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7240003a00 msgr2=0x7f72581f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7240003a00 0x7f72581f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 -- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580cd2c0 msgr2=0x7f725815eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580cd2c0 0x7f725815eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 -- 172.21.15.46:0/1238999014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72580cd6d0 con 0x7f725809d8d0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725d120640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7240003a00 0x7f72581f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f725c91f640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 0x7f72580ccd80 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f724c011560 tx=0x7f724c011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.403+0000 7f7255ffb640 1 -- 172.21.15.46:0/1238999014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c015870 con 0x7f725809d8d0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.404+0000 7f7255ffb640 1 -- 172.21.15.46:0/1238999014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c015a10 con 0x7f725809d8d0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.404+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72581f7c20 con 0x7f725809d8d0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.404+0000 7f7255ffb640 1 -- 172.21.15.46:0/1238999014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c015dc0 con 0x7f725809d8d0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.404+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72581f81e0 con 0x7f725809d8d0 2024-05-09T03:21:27.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.406+0000 7f7255ffb640 1 -- 172.21.15.46:0/1238999014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f724c01d400 con 0x7f725809d8d0 2024-05-09T03:21:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.407+0000 7f7255ffb640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.407+0000 7f7255ffb640 1 -- 172.21.15.46:0/1238999014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f724c097ed0 con 0x7f725809d8d0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.407+0000 7f725d120640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.407+0000 7f726623a740 1 Processor -- start 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.408+0000 7f726623a740 1 -- start start 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.408+0000 7f725d120640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7250007fd0 tx=0x7f7250009030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x55983126f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f726623a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x55983126ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x559831244d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f726623a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5598311a8130 con 0x559830e763e0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5598311615d0 con 0x559830e7e5c0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x5598311a40f0 con 0x559830e9f2a0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x55983126ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x55983126ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41548/0 (socket says 172.21.15.46:41548) 2024-05-09T03:21:27.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 -- 172.21.15.46:0/479051278 learned_addr learned my addr 172.21.15.46:0/479051278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725d120640 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x55983126f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725d921640 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x559831244d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 -- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 msgr2=0x55983126f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x55983126f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 -- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 msgr2=0x559831244d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x559831244d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 -- 172.21.15.46:0/479051278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559830f816e0 con 0x559830e763e0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.409+0000 7f725c91f640 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x55983126ba70 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f724c0963c0 tx=0x7f724c060fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb18493e0286088 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f723effd640 1 -- 172.21.15.46:0/479051278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c065480 con 0x559830e763e0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f723effd640 1 -- 172.21.15.46:0/479051278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c069ea0 con 0x559830e763e0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f723effd640 1 -- 172.21.15.46:0/479051278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c069a10 con 0x559830e763e0 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 -- 172.21.15.46:0/479051278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 msgr2=0x55983126ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x55983126ba70 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f724c0963c0 tx=0x7f724c060fd0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 -- 172.21.15.46:0/479051278 shutdown_connections 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x559831244d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x55983126ba70 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 --2- 172.21.15.46:0/479051278 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x55983126f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 -- 172.21.15.46:0/479051278 >> 172.21.15.46:0/479051278 conn(0x559830e933f0 msgr2=0x5598312444d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 -- 172.21.15.46:0/479051278 shutdown_connections 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.410+0000 7f726623a740 1 -- 172.21.15.46:0/479051278 wait complete. 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.411+0000 7f726623a740 1 -- start start 2024-05-09T03:21:27.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.411+0000 7f726623a740 1 Processor -- start 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x55983126a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x559831287590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55983112eac0 con 0x559830e763e0 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559831287d50 con 0x559830e7e5c0 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559831299b40 con 0x559830e9f2a0 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f726623a740 10 client.0 _ll_register_callbacks cb 0x559831151110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312898b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41558/0 (socket says 172.21.15.46:41558) 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 -- 172.21.15.46:0/1465996734 learned_addr learned my addr 172.21.15.46:0/1465996734 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725c91f640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x55983126a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d921640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x559831287590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 -- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 msgr2=0x559831287590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x559831287590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 -- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 msgr2=0x55983126a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x55983126a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 -- 172.21.15.46:0/1465996734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559830fe5ed0 con 0x559830e763e0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f725d120640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312898b0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f725002ada0 tx=0x7f7250001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e31eeee27b4319a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.413+0000 7f7237fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f7237fff640 1 -- 172.21.15.46:0/1465996734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725004c070 con 0x559830e763e0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f7237fff640 1 -- 172.21.15.46:0/1465996734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f7250007540 con 0x559830e763e0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f724c069ea0 con 0x559830e763e0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f7237fff640 1 -- 172.21.15.46:0/1465996734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7f7250041040 con 0x559830e763e0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 10 client.10941 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x559830f883e0 con 0x559830e763e0 2024-05-09T03:21:27.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f7237fff640 1 -- 172.21.15.46:0/1465996734 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c069ea0 con 0x559830e763e0 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f7237fff640 1 -- 172.21.15.46:0/1465996734 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7f725005a080 con 0x559830e763e0 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 10 client.10941 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 10 client.10941 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 20 client.10941 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x559830e5e080 0x55983129cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 4 client.10941 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598311b5d40 con 0x559830e5e080 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.414+0000 7f725c91f640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x559830e5e080 0x55983129cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.415+0000 7f7237fff640 10 client.10941 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.415+0000 7f725c91f640 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x559830e5e080 0x55983129cec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16489dbb19b4a190 server_cookie=631de7667410050b in_seq=0 out_seq=0 2024-05-09T03:21:27.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.415+0000 7f7237fff640 1 -- 172.21.15.46:0/1465996734 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5598311b5d40 con 0x559830e5e080 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.415+0000 7f7237fff640 10 client.10941 handle_command_reply: tid=0 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.415+0000 7f726623a740 1 client.10941 shutdown 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 msgr2=0x5598312898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312898b0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f725002ada0 tx=0x7f7250001aa0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 shutdown_connections 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e9f2a0 0x559831287590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e7e5c0 0x55983126a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x559830e5e080 0x55983129cec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 --2- 172.21.15.46:0/1465996734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312898b0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 >> 172.21.15.46:0/1465996734 conn(0x559830e933f0 msgr2=0x55983126f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f7237fff640 0 client.10941 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:27.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 shutdown_connections 2024-05-09T03:21:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 1 -- 172.21.15.46:0/1465996734 wait complete. 2024-05-09T03:21:27.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.416+0000 7f726623a740 20 client.10941 trim_cache size 0 max 16384 2024-05-09T03:21:27.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.430+0000 7f726623a740 1 Processor -- start 2024-05-09T03:21:27.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.430+0000 7f726623a740 1 -- start start 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.431+0000 7f726623a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x559831285e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.431+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x5598312a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.431+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x55983126f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.431+0000 7f726623a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7258200480 con 0x559830e763e0 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.431+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7258161330 con 0x559830e9f2a0 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.431+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f72581ffeb0 con 0x559830e7e5c0 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x559831285e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x559831285e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41570/0 (socket says 172.21.15.46:41570) 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 -- 172.21.15.46:0/1504892806 learned_addr learned my addr 172.21.15.46:0/1504892806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 -- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 msgr2=0x5598312a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x5598312a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 -- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 msgr2=0x55983126f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d921640 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x55983126f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x55983126f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 -- 172.21.15.46:0/1504892806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598312711a0 con 0x559830e763e0 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d921640 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x55983126f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f725d120640 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x559831285e30 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f725000c100 tx=0x7f7250048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37973cecf5e59fd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f723e7fc640 1 -- 172.21.15.46:0/1504892806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725004c040 con 0x559830e763e0 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f723e7fc640 1 -- 172.21.15.46:0/1504892806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7250042040 con 0x559830e763e0 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.432+0000 7f723e7fc640 1 -- 172.21.15.46:0/1504892806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7250007660 con 0x559830e763e0 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 -- 172.21.15.46:0/1504892806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 msgr2=0x559831285e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x559831285e30 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f725000c100 tx=0x7f7250048000 comp rx=0 tx=0).stop 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 -- 172.21.15.46:0/1504892806 shutdown_connections 2024-05-09T03:21:27.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x55983126f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x5598312a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 --2- 172.21.15.46:0/1504892806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x559831285e30 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 -- 172.21.15.46:0/1504892806 >> 172.21.15.46:0/1504892806 conn(0x559830e933f0 msgr2=0x55983126a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 -- 172.21.15.46:0/1504892806 shutdown_connections 2024-05-09T03:21:27.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.433+0000 7f726623a740 1 -- 172.21.15.46:0/1504892806 wait complete. 2024-05-09T03:21:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.434+0000 7f726623a740 1 -- start start 2024-05-09T03:21:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.434+0000 7f726623a740 1 Processor -- start 2024-05-09T03:21:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.435+0000 7f726623a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312ed640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.435+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x55983126fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.435+0000 7f726623a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x559831270270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.435+0000 7f726623a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559831272090 con 0x559830e763e0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.435+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559831271ba0 con 0x559830e9f2a0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.435+0000 7f726623a740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x559831245210 con 0x559830e7e5c0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f726623a740 10 client.0 _ll_register_callbacks cb 0x7f724c009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312ed640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312ed640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41580/0 (socket says 172.21.15.46:41580) 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 -- 172.21.15.46:0/4226555185 learned_addr learned my addr 172.21.15.46:0/4226555185 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d921640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x559831270270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725c91f640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x55983126fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 -- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 msgr2=0x55983126fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x55983126fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 -- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 msgr2=0x559831270270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x559831270270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 -- 172.21.15.46:0/4226555185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55983129f000 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725c91f640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x55983126fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f725d120640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312ed640 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f7250001400 tx=0x7f72500078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ca555bbe2c85c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f7237fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f7237fff640 1 -- 172.21.15.46:0/4226555185 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725004c070 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f7237fff640 1 -- 172.21.15.46:0/4226555185 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f725005db50 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7250042040 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f7237fff640 1 -- 172.21.15.46:0/4226555185 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7f7250007700 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f726623a740 10 client.10947 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.436+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x5598312a16a0 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f7237fff640 1 -- 172.21.15.46:0/4226555185 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7250042040 con 0x559830e763e0 2024-05-09T03:21:27.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f7237fff640 1 -- 172.21.15.46:0/4226555185 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7f725005a070 con 0x559830e763e0 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f726623a740 10 client.10947 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f726623a740 10 client.10947 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f726623a740 20 client.10947 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f726623a740 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x5598312726d0 0x559831289830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f726623a740 4 client.10947 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 3454863593}] 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "damage rm", "damage_id": 3454863593}) -- 0x5598311afbb0 con 0x5598312726d0 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f725c91f640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x5598312726d0 0x559831289830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f7237fff640 10 client.10947 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.437+0000 7f725c91f640 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x5598312726d0 0x559831289830 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f669e247696b580 server_cookie=b2c75c2431c9d2fa in_seq=0 out_seq=0 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f7237fff640 1 -- 172.21.15.46:0/4226555185 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+0 (crc 0 0 0) 0x5598311afbb0 con 0x5598312726d0 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f7237fff640 10 client.10947 handle_command_reply: tid=0 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 client.10947 shutdown 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 msgr2=0x5598312ed640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312ed640 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f7250001400 tx=0x7f72500078f0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 shutdown_connections 2024-05-09T03:21:27.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559830e9f2a0 0x559831270270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x559830e7e5c0 0x55983126fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x5598312726d0 0x559831289830 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 --2- 172.21.15.46:0/4226555185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559830e763e0 0x5598312ed640 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f7237fff640 0 client.10947 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 >> 172.21.15.46:0/4226555185 conn(0x559830e933f0 msgr2=0x559831286b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 shutdown_connections 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 1 -- 172.21.15.46:0/4226555185 wait complete. 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.438+0000 7f726623a740 20 client.10947 trim_cache size 0 max 16384 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 msgr2=0x7f72440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7250007fd0 tx=0x7f7250009030 comp rx=0 tx=0).stop 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 msgr2=0x7f72580ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 0x7f72580ccd80 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f724c011560 tx=0x7f724c011a20 comp rx=0 tx=0).stop 2024-05-09T03:21:27.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 shutdown_connections 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72580cd2c0 0x7f725815eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f72440762d0 0x7f72440787e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f725809d8d0 0x7f72580ccd80 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 --2- 172.21.15.46:0/1238999014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f7240003a00 0x7f72581f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 >> 172.21.15.46:0/1238999014 conn(0x7f72580d05f0 msgr2=0x7f72580b8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 shutdown_connections 2024-05-09T03:21:27.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.439+0000 7f725f384640 1 -- 172.21.15.46:0/1238999014 wait complete. 2024-05-09T03:21:27.451 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100000001f6.00000000 2024-05-09T03:21:27.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.506+0000 7fcd6dd25f40 1 Processor -- start 2024-05-09T03:21:27.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.506+0000 7fcd6dd25f40 1 -- start start 2024-05-09T03:21:27.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6dd25f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a854040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6dd25f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a83d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6dd25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6dd25f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555b6a80f400 con 0x555b6a8e15f0 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6dd25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555b6a7b4f50 con 0x555b6a8e1a00 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6dd25f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555b6a858000 con 0x555b6a858400 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a83d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a83d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41592/0 (socket says 172.21.15.46:41592) 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 -- 172.21.15.46:0/3241966531 learned_addr learned my addr 172.21.15.46:0/3241966531 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 -- 172.21.15.46:0/3241966531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 msgr2=0x555b6a854040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a854040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 -- 172.21.15.46:0/3241966531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 msgr2=0x555b6a83d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.507+0000 7fcd6651e640 1 -- 172.21.15.46:0/3241966531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b6a852f50 con 0x555b6a8e15f0 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6651e640 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a83d140 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009730 tx=0x7fcd5c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2874460331a9e3f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6551c640 1 -- 172.21.15.46:0/3241966531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c0078f0 con 0x555b6a8e15f0 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6551c640 1 -- 172.21.15.46:0/3241966531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd5c007a90 con 0x555b6a8e15f0 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6551c640 1 -- 172.21.15.46:0/3241966531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c007e60 con 0x555b6a8e15f0 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/3241966531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 msgr2=0x555b6a83d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a83d140 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009730 tx=0x7fcd5c005a00 comp rx=0 tx=0).stop 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/3241966531 shutdown_connections 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a83d140 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/3241966531 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a854040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.508+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/3241966531 >> 172.21.15.46:0/3241966531 conn(0x555b6a730ac0 msgr2=0x555b6a838100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.509+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/3241966531 shutdown_connections 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.509+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/3241966531 wait complete. 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.509+0000 7fcd6dd25f40 1 Processor -- start 2024-05-09T03:21:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.509+0000 7fcd6dd25f40 1 -- start start 2024-05-09T03:21:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a851c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a856cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555b6a80ed60 con 0x555b6a8e15f0 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555b6a853b90 con 0x555b6a8e1a00 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x555b6a8dfd00 con 0x555b6a858400 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a856cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a856cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41604/0 (socket says 172.21.15.46:41604) 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 -- 172.21.15.46:0/1959647382 learned_addr learned my addr 172.21.15.46:0/1959647382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd67520640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd66d1f640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a851c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 -- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 msgr2=0x555b6a851c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a851c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 -- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 msgr2=0x555b6a83ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 -- 172.21.15.46:0/1959647382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b6a8e3ec0 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd66d1f640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x555b6a858400 0x555b6a851c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd67520640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b6a8e1a00 0x555b6a83ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6651e640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a856cd0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009cb0 tx=0x7fcd5c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd4ffff640 1 -- 172.21.15.46:0/1959647382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c023070 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd4ffff640 1 -- 172.21.15.46:0/1959647382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd5c017ab0 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.511+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b6a857210 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.512+0000 7fcd4ffff640 1 -- 172.21.15.46:0/1959647382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c017e60 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.512+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b6a8e4130 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.513+0000 7fcd4ffff640 1 -- 172.21.15.46:0/1959647382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcd5c026070 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.514+0000 7fcd4ffff640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd54074ce0 0x7fcd540771f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.514+0000 7fcd4ffff640 1 -- 172.21.15.46:0/1959647382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fcd5c0982a0 con 0x555b6a8e15f0 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.514+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555b6a80d480 0x555b6a80d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.514+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- osd_op(unknown.0.0:1 14.a 14:57fe7535:::100000001f6.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e136) -- 0x555b6a8e4930 con 0x555b6a80d480 2024-05-09T03:21:27.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.514+0000 7fcd66d1f640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd54074ce0 0x7fcd540771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.514+0000 7fcd67520640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555b6a80d480 0x555b6a80d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.515+0000 7fcd66d1f640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd54074ce0 0x7fcd540771f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fcd58002ff0 tx=0x7fcd58005a00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.515+0000 7fcd67520640 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555b6a80d480 0x555b6a80d890 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.516+0000 7fcd67520640 1 -- 172.21.15.46:0/1959647382 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== osd_op_reply(1 100000001f6.00000000 [delete] v136'3 uv3 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fcd600017b0 con 0x555b6a80d480 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.516+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555b6a80d480 msgr2=0x555b6a80d890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.516+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x555b6a80d480 0x555b6a80d890 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd54074ce0 msgr2=0x7fcd540771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fcd54074ce0 0x7fcd540771f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fcd58002ff0 tx=0x7fcd58005a00 comp rx=0 tx=0).stop 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 msgr2=0x555b6a856cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 --2- 172.21.15.46:0/1959647382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555b6a8e15f0 0x555b6a856cd0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009cb0 tx=0x7fcd5c002fb0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd67520640 1 -- 172.21.15.46:0/1959647382 reap_dead start 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 shutdown_connections 2024-05-09T03:21:27.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 >> 172.21.15.46:0/1959647382 conn(0x555b6a730ac0 msgr2=0x555b6a8380e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 shutdown_connections 2024-05-09T03:21:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.517+0000 7fcd6dd25f40 1 -- 172.21.15.46:0/1959647382 wait complete. 2024-05-09T03:21:27.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:27.521 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink2) 2024-05-09T03:21:27.538 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-05-09T03:21:27.538+0000 7f7bd62ee640 -1 log_channel(cluster) log [ERR] : dir 0x100000001f6 object missing on disk; some files may be lost (/dir2) 2024-05-09T03:21:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:21:27.540 INFO:teuthology.orchestra.run.smithi046.stderr:stat: cannot statx 'testdir/hardlink2': Input/output error 2024-05-09T03:21:27.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.694+0000 7fb444ce0640 1 Processor -- start 2024-05-09T03:21:27.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.694+0000 7fb444ce0640 1 -- start start 2024-05-09T03:21:27.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.695+0000 7fb444ce0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 0x7fb4401615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.695+0000 7fb444ce0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb4400d2480 0x7fb4400d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.695+0000 7fb444ce0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb440160b20 con 0x7fb44015dfe0 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.695+0000 7fb444ce0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb440161150 con 0x7fb4400d2480 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.695+0000 7fb444ce0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb4400d15f0 con 0x7fb440164c70 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 0x7fb4401615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb4400d2480 0x7fb4400d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 0x7fb4401615e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:56244/0 (socket says 172.21.15.46:56244) 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 -- 172.21.15.46:0/1465780181 learned_addr learned my addr 172.21.15.46:0/1465780181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb437fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb44015dfe0 0x7fb4400d0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:54820/0 (socket says 172.21.15.46:54820) 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 -- 172.21.15.46:0/1465780181 >> v2:172.21.15.88:3300/0 conn(0x7fb4400d2480 msgr2=0x7fb4400d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 --2- 172.21.15.46:0/1465780181 >> v2:172.21.15.88:3300/0 conn(0x7fb4400d2480 0x7fb4400d2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 -- 172.21.15.46:0/1465780181 >> v1:172.21.15.88:6789/0 conn(0x7fb44015dfe0 legacy=0x7fb4400d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.696+0000 7fb43e59c640 1 -- 172.21.15.46:0/1465780181 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4400d18d0 con 0x7fb440164c70 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.697+0000 7fb43e59c640 1 --2- 172.21.15.46:0/1465780181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 0x7fb4401615e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb43002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e90d0667080e4ff6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.697+0000 7fb43dd9b640 1 -- 172.21.15.46:0/1465780181 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430037070 con 0x7fb440164c70 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.697+0000 7fb43dd9b640 1 -- 172.21.15.46:0/1465780181 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb430032440 con 0x7fb440164c70 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.697+0000 7fb43dd9b640 1 -- 172.21.15.46:0/1465780181 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb430032810 con 0x7fb440164c70 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 -- 172.21.15.46:0/1465780181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 msgr2=0x7fb4401615e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1465780181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 0x7fb4401615e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb43002a270 comp rx=0 tx=0).stop 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 -- 172.21.15.46:0/1465780181 shutdown_connections 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1465780181 >> v2:172.21.15.88:3300/0 conn(0x7fb4400d2480 0x7fb4400d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1465780181 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb440164c70 0x7fb4401615e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 -- 172.21.15.46:0/1465780181 >> 172.21.15.46:0/1465780181 conn(0x7fb440073db0 msgr2=0x7fb4400741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 -- 172.21.15.46:0/1465780181 shutdown_connections 2024-05-09T03:21:27.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.698+0000 7fb444ce0640 1 -- 172.21.15.46:0/1465780181 wait complete. 2024-05-09T03:21:27.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.700+0000 7fb444ce0640 1 Processor -- start 2024-05-09T03:21:27.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.700+0000 7fb444ce0640 1 -- start start 2024-05-09T03:21:27.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb444ce0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4400d2480 0x7fb440151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb444ce0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44015dfe0 0x7fb4401ec100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb444ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 0x7fb4401ec640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb444ce0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb4400d2dd0 con 0x7fb440164c70 2024-05-09T03:21:27.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb444ce0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb44015e3f0 con 0x7fb4400d2480 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb444ce0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb4400d45c0 con 0x7fb44015dfe0 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 0x7fb4401ec640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 0x7fb4401ec640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41620/0 (socket says 172.21.15.46:41620) 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 -- 172.21.15.46:0/1379461608 learned_addr learned my addr 172.21.15.46:0/1379461608 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43e59c640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4400d2480 0x7fb440151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb437fff640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44015dfe0 0x7fb4401ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 -- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44015dfe0 msgr2=0x7fb4401ec100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44015dfe0 0x7fb4401ec100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 -- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4400d2480 msgr2=0x7fb440151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4400d2480 0x7fb440151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.701+0000 7fb43ed9d640 1 -- 172.21.15.46:0/1379461608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb440151bd0 con 0x7fb440164c70 2024-05-09T03:21:27.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.702+0000 7fb43ed9d640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 0x7fb4401ec640 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fb438009d30 tx=0x7fb43800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.702+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb438018020 con 0x7fb440164c70 2024-05-09T03:21:27.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.702+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb438014480 con 0x7fb440164c70 2024-05-09T03:21:27.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.702+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4400cbd10 con 0x7fb440164c70 2024-05-09T03:21:27.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.702+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb438014870 con 0x7fb440164c70 2024-05-09T03:21:27.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.702+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4400cd110 con 0x7fb440164c70 2024-05-09T03:21:27.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.703+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4400cba80 con 0x7fb440164c70 2024-05-09T03:21:27.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.703+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb438005d10 con 0x7fb440164c70 2024-05-09T03:21:27.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.704+0000 7fb436ffd640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4200762d0 0x7fb4200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:27.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.704+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fb4380981b0 con 0x7fb440164c70 2024-05-09T03:21:27.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.704+0000 7fb43e59c640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4200762d0 0x7fb4200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:27.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.705+0000 7fb43e59c640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4200762d0 0x7fb4200787e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb430034000 tx=0x7fb43002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:27.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.709+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb438022070 con 0x7fb440164c70 2024-05-09T03:21:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.937+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb44008cfd0 con 0x7fb440164c70 2024-05-09T03:21:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.938+0000 7fb436ffd640 1 -- 172.21.15.46:0/1379461608 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) ==== 95+0+5190 (secure 0 0 0) 0x7fb44008cfd0 con 0x7fb440164c70 2024-05-09T03:21:27.939 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:27.939 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":359,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:22.873173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 359 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4200762d0 msgr2=0x7fb4200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4200762d0 0x7fb4200787e0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb430034000 tx=0x7fb43002e000 comp rx=0 tx=0).stop 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 msgr2=0x7fb4401ec640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 0x7fb4401ec640 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fb438009d30 tx=0x7fb43800c490 comp rx=0 tx=0).stop 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 shutdown_connections 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb4200762d0 0x7fb4200787e0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb440164c70 0x7fb4401ec640 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb44015dfe0 0x7fb4401ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 --2- 172.21.15.46:0/1379461608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4400d2480 0x7fb440151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 >> 172.21.15.46:0/1379461608 conn(0x7fb440073db0 msgr2=0x7fb4400d49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 shutdown_connections 2024-05-09T03:21:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:27.940+0000 7fb444ce0640 1 -- 172.21.15.46:0/1379461608 wait complete. 2024-05-09T03:21:27.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-05-09T03:21:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.068+0000 7fc866be0640 1 Processor -- start 2024-05-09T03:21:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.068+0000 7fc866be0640 1 -- start start 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc866be0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8600b7eb0 0x7fc8600b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc866be0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8600b4210 con 0x7fc8600cf270 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc866be0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8600ce790 con 0x7fc8600cff90 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc866be0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc8600cede0 con 0x7fc8600b7eb0 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc86517d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc8600b7eb0 0x7fc8600b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc85ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc8600cf270 0x7fc8600b77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53508/0 (socket says 172.21.15.46:53508) 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc85ffff640 1 -- 172.21.15.46:0/1242129800 learned_addr learned my addr 172.21.15.46:0/1242129800 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc86517d640 1 -- 172.21.15.46:0/1242129800 >> v1:172.21.15.88:6789/0 conn(0x7fc8600cff90 legacy=0x7fc8600b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc86517d640 1 -- 172.21.15.46:0/1242129800 >> v1:172.21.15.46:6789/0 conn(0x7fc8600cf270 legacy=0x7fc8600b77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.070+0000 7fc86517d640 1 -- 172.21.15.46:0/1242129800 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8600cea70 con 0x7fc8600b7eb0 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.071+0000 7fc86517d640 1 --2- 172.21.15.46:0/1242129800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600b7eb0 0x7fc8600b23e0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fc858009d90 tx=0x7fc85802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394abdb04eace496 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.071+0000 7fc85f7fe640 1 -- 172.21.15.46:0/1242129800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858007740 con 0x7fc8600b7eb0 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.071+0000 7fc85f7fe640 1 -- 172.21.15.46:0/1242129800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8580078e0 con 0x7fc8600b7eb0 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.071+0000 7fc85f7fe640 1 -- 172.21.15.46:0/1242129800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858007cb0 con 0x7fc8600b7eb0 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 -- 172.21.15.46:0/1242129800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600b7eb0 msgr2=0x7fc8600b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 --2- 172.21.15.46:0/1242129800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600b7eb0 0x7fc8600b23e0 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fc858009d90 tx=0x7fc85802a660 comp rx=0 tx=0).stop 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 -- 172.21.15.46:0/1242129800 shutdown_connections 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 --2- 172.21.15.46:0/1242129800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600b7eb0 0x7fc8600b23e0 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 -- 172.21.15.46:0/1242129800 >> 172.21.15.46:0/1242129800 conn(0x7fc86009b7d0 msgr2=0x7fc86009bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 -- 172.21.15.46:0/1242129800 shutdown_connections 2024-05-09T03:21:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.072+0000 7fc866be0640 1 -- 172.21.15.46:0/1242129800 wait complete. 2024-05-09T03:21:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.073+0000 7fc866be0640 1 Processor -- start 2024-05-09T03:21:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.074+0000 7fc866be0640 1 -- start start 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc866be0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8600b7eb0 0x7fc8601f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc866be0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8600cf270 0x7fc8601f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc866be0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 0x7fc8601f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc866be0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600cf800 con 0x7fc8600cff90 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc866be0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc8600cf680 con 0x7fc8600b7eb0 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc866be0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc86015e450 con 0x7fc8600cf270 2024-05-09T03:21:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 0x7fc8601f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 0x7fc8601f84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41650/0 (socket says 172.21.15.46:41650) 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc86517d640 1 -- 172.21.15.46:0/436841674 learned_addr learned my addr 172.21.15.46:0/436841674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc85ffff640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8600cf270 0x7fc8601f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc86497c640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8600b7eb0 0x7fc8601f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.075+0000 7fc86517d640 1 -- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8600cf270 msgr2=0x7fc8601f7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc86517d640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8600cf270 0x7fc8601f7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc86517d640 1 -- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8600b7eb0 msgr2=0x7fc8601f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc86517d640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8600b7eb0 0x7fc8601f7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc86517d640 1 -- 172.21.15.46:0/436841674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86006db70 con 0x7fc8600cff90 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc86497c640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8600b7eb0 0x7fc8601f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc86517d640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 0x7fc8601f84d0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fc858009eb0 tx=0x7fc858004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc85d7fa640 1 -- 172.21.15.46:0/436841674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85803f070 con 0x7fc8600cff90 2024-05-09T03:21:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc85d7fa640 1 -- 172.21.15.46:0/436841674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858031c90 con 0x7fc8600cff90 2024-05-09T03:21:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86006de50 con 0x7fc8600cff90 2024-05-09T03:21:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc85d7fa640 1 -- 172.21.15.46:0/436841674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858044680 con 0x7fc8600cff90 2024-05-09T03:21:28.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.076+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8600cbdd0 con 0x7fc8600cff90 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.078+0000 7fc85d7fa640 1 -- 172.21.15.46:0/436841674 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc85803b410 con 0x7fc8600cff90 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.078+0000 7fc85d7fa640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc84c0762d0 0x7fc84c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.078+0000 7fc85d7fa640 1 -- 172.21.15.46:0/436841674 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fc8580b8fa0 con 0x7fc8600cff90 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.079+0000 7fc86497c640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc84c0762d0 0x7fc84c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.079+0000 7fc86497c640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc84c0762d0 0x7fc84c0787e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc850007fd0 tx=0x7fc850009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.079+0000 7fc86db6d740 1 Processor -- start 2024-05-09T03:21:28.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.079+0000 7fc86db6d740 1 -- start start 2024-05-09T03:21:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.080+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b10a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.080+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b0e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.080+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 0x55bd1b0e8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86db6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd1b04ad10 con 0x55bd1ad21670 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bd1afd2750 con 0x55bd1ad19490 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bd1b046cd0 con 0x55bd1ad42350 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 0x55bd1b0e8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 0x55bd1b0e8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41664/0 (socket says 172.21.15.46:41664) 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 -- 172.21.15.46:0/3667882120 learned_addr learned my addr 172.21.15.46:0/3667882120 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86497c640 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b10a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc85ffff640 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b0e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 -- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 msgr2=0x55bd1b10a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b10a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 -- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 msgr2=0x55bd1b0e7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b0e7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 -- 172.21.15.46:0/3667882120 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1ae24790 con 0x55bd1ad21670 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86497c640 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b10a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc86517d640 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 0x55bd1b0e8450 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc85807e7c0 tx=0x7fc858081a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecacc6d0d3e431d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc8467fc640 1 -- 172.21.15.46:0/3667882120 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580821d0 con 0x55bd1ad21670 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc8467fc640 1 -- 172.21.15.46:0/3667882120 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858086580 con 0x55bd1ad21670 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.081+0000 7fc8467fc640 1 -- 172.21.15.46:0/3667882120 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85808b030 con 0x55bd1ad21670 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 -- 172.21.15.46:0/3667882120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 msgr2=0x55bd1b0e8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 0x55bd1b0e8450 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fc85807e7c0 tx=0x7fc858081a00 comp rx=0 tx=0).stop 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 -- 172.21.15.46:0/3667882120 shutdown_connections 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad21670 0x55bd1b0e8450 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b0e7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 --2- 172.21.15.46:0/3667882120 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b10a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 -- 172.21.15.46:0/3667882120 >> 172.21.15.46:0/3667882120 conn(0x55bd1ad364a0 msgr2=0x55bd1b0e56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 -- 172.21.15.46:0/3667882120 shutdown_connections 2024-05-09T03:21:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.082+0000 7fc86db6d740 1 -- 172.21.15.46:0/3667882120 wait complete. 2024-05-09T03:21:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.083+0000 7fc86db6d740 1 -- start start 2024-05-09T03:21:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.083+0000 7fc86db6d740 1 Processor -- start 2024-05-09T03:21:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.084+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b10dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad21670 0x55bd1b131620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd1b04ad10 con 0x55bd1ad42350 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bd1b007bd0 con 0x55bd1ad19490 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bd1b13d460 con 0x55bd1ad21670 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 10 client.0 _ll_register_callbacks cb 0x55bd1affc210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41672/0 (socket says 172.21.15.46:41672) 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 -- 172.21.15.46:0/2491845709 learned_addr learned my addr 172.21.15.46:0/2491845709 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc85ffff640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad21670 0x55bd1b131620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 -- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad21670 msgr2=0x55bd1b131620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad21670 0x55bd1b131620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 -- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 msgr2=0x55bd1b10dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86497c640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b10dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b10dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 -- 172.21.15.46:0/2491845709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd1acf3130 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86517d640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131b60 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fc85802a6a0 tx=0x7fc858046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707c3cbc16ff7df8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc83f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2491845709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580b4040 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2491845709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fc858082430 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc858086580 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2491845709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7fc8580a8050 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 10 client.10974 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.085+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x55bd1ae245c0 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2491845709 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858086580 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2491845709 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7fc8580ab020 con 0x55bd1ad42350 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86db6d740 10 client.10974 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86db6d740 10 client.10974 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86db6d740 20 client.10974 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1ad01130 0x55bd1b10a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86db6d740 4 client.10974 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd1b058920 con 0x55bd1ad01130 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86497c640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1ad01130 0x55bd1b10a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc83f7fe640 10 client.10974 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.086+0000 7fc86497c640 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1ad01130 0x55bd1b10a6d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e45acdba7989a3c8 server_cookie=64960d74dc443098 in_seq=0 out_seq=0 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2491845709 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bd1b058920 con 0x55bd1ad01130 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc83f7fe640 10 client.10974 handle_command_reply: tid=0 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 client.10974 shutdown 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 msgr2=0x55bd1b131b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131b60 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7fc85802a6a0 tx=0x7fc858046040 comp rx=0 tx=0).stop 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 shutdown_connections 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1ad01130 0x55bd1b10a6d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131b60 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad21670 0x55bd1b131620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2491845709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad19490 0x55bd1b10dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc83f7fe640 0 client.10974 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 >> 172.21.15.46:0/2491845709 conn(0x55bd1ad364a0 msgr2=0x55bd1b12e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 shutdown_connections 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 1 -- 172.21.15.46:0/2491845709 wait complete. 2024-05-09T03:21:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.087+0000 7fc86db6d740 20 client.10974 trim_cache size 0 max 16384 2024-05-09T03:21:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.101+0000 7fc86db6d740 1 Processor -- start 2024-05-09T03:21:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.101+0000 7fc86db6d740 1 -- start start 2024-05-09T03:21:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad19490 0x55bd1b110cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b10b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86db6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601ffe90 con 0x55bd1ad42350 2024-05-09T03:21:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc8601f4940 con 0x55bd1ad21670 2024-05-09T03:21:28.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fc8601f4f90 con 0x55bd1ad19490 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41680/0 (socket says 172.21.15.46:41680) 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 -- 172.21.15.46:0/1164881121 learned_addr learned my addr 172.21.15.46:0/1164881121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 -- 172.21.15.46:0/1164881121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad19490 msgr2=0x55bd1b110cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad19490 0x55bd1b110cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 -- 172.21.15.46:0/1164881121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 msgr2=0x55bd1b10b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b10b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 -- 172.21.15.46:0/1164881121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd1ad26150 con 0x55bd1ad42350 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc86517d640 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131000 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fc858082160 tx=0x7fc8580b8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e1e12fdd88bd27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc845ffb640 1 -- 172.21.15.46:0/1164881121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580b4040 con 0x55bd1ad42350 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc845ffb640 1 -- 172.21.15.46:0/1164881121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8580b73b0 con 0x55bd1ad42350 2024-05-09T03:21:28.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.103+0000 7fc845ffb640 1 -- 172.21.15.46:0/1164881121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858086580 con 0x55bd1ad42350 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 -- 172.21.15.46:0/1164881121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 msgr2=0x55bd1b131000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131000 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fc858082160 tx=0x7fc8580b8000 comp rx=0 tx=0).stop 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 -- 172.21.15.46:0/1164881121 shutdown_connections 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad42350 0x55bd1b131000 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b10b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 --2- 172.21.15.46:0/1164881121 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad19490 0x55bd1b110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 -- 172.21.15.46:0/1164881121 >> 172.21.15.46:0/1164881121 conn(0x55bd1ad364a0 msgr2=0x55bd1b10ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 -- 172.21.15.46:0/1164881121 shutdown_connections 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.104+0000 7fc86db6d740 1 -- 172.21.15.46:0/1164881121 wait complete. 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.105+0000 7fc86db6d740 1 -- start start 2024-05-09T03:21:28.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.105+0000 7fc86db6d740 1 Processor -- start 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.106+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 0x55bd1b108f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b0e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b0e8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd1b13d240 con 0x55bd1ad19490 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bd1b04ad10 con 0x55bd1ad21670 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55bd1b106420 con 0x55bd1ad42350 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 10 client.0 _ll_register_callbacks cb 0x7fc858081b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 0x55bd1b108f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 0x55bd1b108f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41696/0 (socket says 172.21.15.46:41696) 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 -- 172.21.15.46:0/2664825014 learned_addr learned my addr 172.21.15.46:0/2664825014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc85ffff640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b0e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86517d640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b0e8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 -- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 msgr2=0x55bd1b0e8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b0e8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 -- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 msgr2=0x55bd1b0e7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b0e7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 -- 172.21.15.46:0/2664825014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd1b1a2ae0 con 0x55bd1ad19490 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc85ffff640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b0e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86497c640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 0x55bd1b108f40 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x55bd1b10d7f0 tx=0x7fc850000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab198300836bdc57 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc83f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2664825014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500039b0 con 0x55bd1ad19490 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2664825014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fc850003b50 con 0x55bd1ad19490 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8580b73b0 con 0x55bd1ad19490 2024-05-09T03:21:28.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2664825014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7fc850031050 con 0x55bd1ad19490 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.107+0000 7fc86db6d740 10 client.10980 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x55bd1b109510 con 0x55bd1ad19490 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2664825014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8580b73b0 con 0x55bd1ad19490 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2664825014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7fc850034020 con 0x55bd1ad19490 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 10 client.10980 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 10 client.10980 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 20 client.10980 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1b1a1500 0x55bd1b14eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 4 client.10980 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55bd1b052790 con 0x55bd1b1a1500 2024-05-09T03:21:28.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc85ffff640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1b1a1500 0x55bd1b14eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc83f7fe640 10 client.10980 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.108+0000 7fc85ffff640 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1b1a1500 0x55bd1b14eed0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e44124bb50a9f14 server_cookie=e04f3e4d63cd217a in_seq=0 out_seq=0 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc83f7fe640 1 -- 172.21.15.46:0/2664825014 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+294 (crc 0 0 0) 0x55bd1b052790 con 0x55bd1b1a1500 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc83f7fe640 10 client.10980 handle_command_reply: tid=0 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 client.10980 shutdown 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 msgr2=0x55bd1b108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 0x55bd1b108f40 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x55bd1b10d7f0 tx=0x7fc850000ed0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 shutdown_connections 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55bd1ad42350 0x55bd1b0e8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bd1ad21670 0x55bd1b0e7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55bd1b1a1500 0x55bd1b14eed0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 --2- 172.21.15.46:0/2664825014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd1ad19490 0x55bd1b108f40 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 >> 172.21.15.46:0/2664825014 conn(0x55bd1ad364a0 msgr2=0x55bd1b150320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc83f7fe640 0 client.10980 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 shutdown_connections 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 1 -- 172.21.15.46:0/2664825014 wait complete. 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.109+0000 7fc86db6d740 20 client.10980 trim_cache size 0 max 16384 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "dir_frag", 2024-05-09T03:21:28.116 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1337125749, 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628278, 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "frag": "*", 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/dir2" 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "damage_type": "backtrace", 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 1365243552, 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 1099511628279, 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/testdir/hardlink2" 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-05-09T03:21:28.117 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-05-09T03:21:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc84c0762d0 msgr2=0x7fc84c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc84c0762d0 0x7fc84c0787e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc850007fd0 tx=0x7fc850009040 comp rx=0 tx=0).stop 2024-05-09T03:21:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 msgr2=0x7fc8601f84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 0x7fc8601f84d0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fc858009eb0 tx=0x7fc858004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 shutdown_connections 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fc84c0762d0 0x7fc84c0787e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc8600cff90 0x7fc8601f84d0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fc8600cf270 0x7fc8601f7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 --2- 172.21.15.46:0/436841674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8600b7eb0 0x7fc8601f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 >> 172.21.15.46:0/436841674 conn(0x7fc86009b7d0 msgr2=0x7fc860071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 shutdown_connections 2024-05-09T03:21:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.110+0000 7fc866be0640 1 -- 172.21.15.46:0/436841674 wait complete. 2024-05-09T03:21:28.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.236+0000 7faabbf73640 1 Processor -- start 2024-05-09T03:21:28.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.236+0000 7faabbf73640 1 -- start start 2024-05-09T03:21:28.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faabbf73640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faab4164c70 0x7faab41615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faabbf73640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faabbf73640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab4160b20 con 0x7faab40d2480 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faabbf73640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faab4161150 con 0x7faab4164c70 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faabbf73640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faab40d15f0 con 0x7faab415dfe0 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faaba510640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faab40d2480 0x7faab40d2890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53510/0 (socket says 172.21.15.46:53510) 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faaba510640 1 -- 172.21.15.46:0/1512657307 learned_addr learned my addr 172.21.15.46:0/1512657307 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faab950e640 1 --2- 172.21.15.46:0/1512657307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.237+0000 7faab9d0f640 1 --2- 172.21.15.46:0/1512657307 >> v2:172.21.15.88:3300/0 conn(0x7faab4164c70 0x7faab41615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab8d0d640 1 -- 172.21.15.46:0/1512657307 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3337568124 0 0) 0x7faab4160b20 con 0x7faab40d2480 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab8d0d640 1 -- 172.21.15.46:0/1512657307 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa9c003520 con 0x7faab40d2480 2024-05-09T03:21:28.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab950e640 1 -- 172.21.15.46:0/1512657307 >> v2:172.21.15.88:3300/0 conn(0x7faab4164c70 msgr2=0x7faab41615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab950e640 1 --2- 172.21.15.46:0/1512657307 >> v2:172.21.15.88:3300/0 conn(0x7faab4164c70 0x7faab41615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab950e640 1 -- 172.21.15.46:0/1512657307 >> v1:172.21.15.46:6789/0 conn(0x7faab40d2480 legacy=0x7faab40d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab8d0d640 1 -- 172.21.15.46:0/1512657307 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1089091696 0 0) 0x7faa9c003520 con 0x7faab40d2480 2024-05-09T03:21:28.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.238+0000 7faab950e640 1 -- 172.21.15.46:0/1512657307 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab40d18d0 con 0x7faab415dfe0 2024-05-09T03:21:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faab950e640 1 --2- 172.21.15.46:0/1512657307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40d0dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faaa80097a0 tx=0x7faaa802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef511351aa931a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faab8d0d640 1 -- 172.21.15.46:0/1512657307 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8037070 con 0x7faab415dfe0 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faab8d0d640 1 -- 172.21.15.46:0/1512657307 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa802ab10 con 0x7faab415dfe0 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faab8d0d640 1 -- 172.21.15.46:0/1512657307 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac0016f0 con 0x7faab415dfe0 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faabbf73640 1 -- 172.21.15.46:0/1512657307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 msgr2=0x7faab40d0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faabbf73640 1 --2- 172.21.15.46:0/1512657307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40d0dc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7faaa80097a0 tx=0x7faaa802a080 comp rx=0 tx=0).stop 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faabbf73640 1 -- 172.21.15.46:0/1512657307 shutdown_connections 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faabbf73640 1 --2- 172.21.15.46:0/1512657307 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40d0dc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.239+0000 7faabbf73640 1 --2- 172.21.15.46:0/1512657307 >> v2:172.21.15.88:3300/0 conn(0x7faab4164c70 0x7faab41615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.240+0000 7faabbf73640 1 -- 172.21.15.46:0/1512657307 >> 172.21.15.46:0/1512657307 conn(0x7faab4073db0 msgr2=0x7faab40741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.240+0000 7faabbf73640 1 -- 172.21.15.46:0/1512657307 shutdown_connections 2024-05-09T03:21:28.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.240+0000 7faabbf73640 1 -- 172.21.15.46:0/1512657307 wait complete. 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.241+0000 7faabbf73640 1 Processor -- start 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.241+0000 7faabbf73640 1 -- start start 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faabbf73640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 0x7faab4151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faabbf73640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faabbf73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4164c70 0x7faab40cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faabbf73640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faab415e3f0 con 0x7faab40d2480 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faabbf73640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faab40d1300 con 0x7faab4164c70 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faabbf73640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7faab40d45c0 con 0x7faab415dfe0 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 0x7faab4151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 0x7faab4151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41708/0 (socket says 172.21.15.46:41708) 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 -- 172.21.15.46:0/580578806 learned_addr learned my addr 172.21.15.46:0/580578806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faaba510640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4164c70 0x7faab40cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab950e640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 -- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 msgr2=0x7faab40ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 -- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4164c70 msgr2=0x7faab40cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4164c70 0x7faab40cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 -- 172.21.15.46:0/580578806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab40ce740 con 0x7faab40d2480 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.243+0000 7faab9d0f640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 0x7faab4151d00 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7faaa400efc0 tx=0x7faaa400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.244+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4018020 con 0x7faab40d2480 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.244+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa4014480 con 0x7faab40d2480 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.244+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab40cea20 con 0x7faab40d2480 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.244+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4014850 con 0x7faab40d2480 2024-05-09T03:21:28.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.244+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab40cefe0 con 0x7faab40d2480 2024-05-09T03:21:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.245+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab408cfd0 con 0x7faab40d2480 2024-05-09T03:21:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.246+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faaa4005d10 con 0x7faab40d2480 2024-05-09T03:21:28.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.246+0000 7faab27fc640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faaa0076300 0x7faaa0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.246+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7faaa4098190 con 0x7faab40d2480 2024-05-09T03:21:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.246+0000 7faab950e640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faaa0076300 0x7faaa0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.247+0000 7faab950e640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faaa0076300 0x7faaa0078810 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7faaa8034000 tx=0x7faaa802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.251+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faab408cfd0 con 0x7faab40d2480 2024-05-09T03:21:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.482+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faab406a270 con 0x7faab40d2480 2024-05-09T03:21:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.482+0000 7faab27fc640 1 -- 172.21.15.46:0/580578806 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) ==== 95+0+5190 (secure 0 0 0) 0x7faaa4022070 con 0x7faab40d2480 2024-05-09T03:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:28.483 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":359,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:22.873173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 359 2024-05-09T03:21:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faaa0076300 msgr2=0x7faaa0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faaa0076300 0x7faaa0078810 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7faaa8034000 tx=0x7faaa802e000 comp rx=0 tx=0).stop 2024-05-09T03:21:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 msgr2=0x7faab4151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 0x7faab4151d00 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7faaa400efc0 tx=0x7faaa400c490 comp rx=0 tx=0).stop 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 shutdown_connections 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4164c70 0x7faab40cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7faab415dfe0 0x7faab40ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7faaa0076300 0x7faaa0078810 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 --2- 172.21.15.46:0/580578806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab40d2480 0x7faab4151d00 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 >> 172.21.15.46:0/580578806 conn(0x7faab4073db0 msgr2=0x7faab40bb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 shutdown_connections 2024-05-09T03:21:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.484+0000 7faabbf73640 1 -- 172.21.15.46:0/580578806 wait complete. 2024-05-09T03:21:28.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 1337125749 2024-05-09T03:21:28.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.613+0000 7fbb44900640 1 Processor -- start 2024-05-09T03:21:28.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.613+0000 7fbb44900640 1 -- start start 2024-05-09T03:21:28.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb44900640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbb40163eb0 0x7fbb401642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb44900640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb400d2a70 0x7fbb400d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb44900640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb400d33c0 con 0x7fbb400bb770 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb44900640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbb400d4bb0 con 0x7fbb400d2a70 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb44900640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbb40164900 con 0x7fbb40163eb0 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb3ed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbb400d2a70 0x7fbb400d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb3e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbb400bb770 0x7fbb400d5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53524/0 (socket says 172.21.15.46:53524) 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.614+0000 7fbb3e59c640 1 -- 172.21.15.46:0/2024020040 learned_addr learned my addr 172.21.15.46:0/2024020040 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb37fff640 1 --2- 172.21.15.46:0/2024020040 >> v2:172.21.15.88:3300/0 conn(0x7fbb40163eb0 0x7fbb401642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb3dd9b640 1 -- 172.21.15.46:0/2024020040 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531551777 0 0) 0x7fbb30001110 con 0x7fbb400bb770 2024-05-09T03:21:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb3dd9b640 1 -- 172.21.15.46:0/2024020040 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb2c003520 con 0x7fbb400bb770 2024-05-09T03:21:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2024020040 >> v2:172.21.15.88:3300/0 conn(0x7fbb40163eb0 msgr2=0x7fbb401642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/2024020040 >> v2:172.21.15.88:3300/0 conn(0x7fbb40163eb0 0x7fbb401642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2024020040 >> v1:172.21.15.46:6789/0 conn(0x7fbb400bb770 legacy=0x7fbb400d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.615+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2024020040 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb40164be0 con 0x7fbb400d2a70 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/2024020040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb400d2a70 0x7fbb400d2e80 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fbb400d6d00 tx=0x7fbb3802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451e6033860376e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb3dd9b640 1 -- 172.21.15.46:0/2024020040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38007740 con 0x7fbb400d2a70 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb3dd9b640 1 -- 172.21.15.46:0/2024020040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb380078e0 con 0x7fbb400d2a70 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb3dd9b640 1 -- 172.21.15.46:0/2024020040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38007cb0 con 0x7fbb400d2a70 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb44900640 1 -- 172.21.15.46:0/2024020040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb400d2a70 msgr2=0x7fbb400d2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb44900640 1 --2- 172.21.15.46:0/2024020040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb400d2a70 0x7fbb400d2e80 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fbb400d6d00 tx=0x7fbb3802a680 comp rx=0 tx=0).stop 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb44900640 1 -- 172.21.15.46:0/2024020040 shutdown_connections 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb44900640 1 --2- 172.21.15.46:0/2024020040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb400d2a70 0x7fbb400d2e80 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb44900640 1 --2- 172.21.15.46:0/2024020040 >> v2:172.21.15.88:3300/0 conn(0x7fbb40163eb0 0x7fbb401642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.616+0000 7fbb44900640 1 -- 172.21.15.46:0/2024020040 >> 172.21.15.46:0/2024020040 conn(0x7fbb4006e370 msgr2=0x7fbb4006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.617+0000 7fbb44900640 1 -- 172.21.15.46:0/2024020040 shutdown_connections 2024-05-09T03:21:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.617+0000 7fbb44900640 1 -- 172.21.15.46:0/2024020040 wait complete. 2024-05-09T03:21:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.618+0000 7fbb44900640 1 Processor -- start 2024-05-09T03:21:28.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.619+0000 7fbb44900640 1 -- start start 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb44900640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb400bb770 0x7fbb401f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb44900640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb400d2a70 0x7fbb4014e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb44900640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 0x7fbb4014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb44900640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb400d0bc0 con 0x7fbb40163eb0 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb44900640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbb400bbb80 con 0x7fbb400d2a70 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb44900640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbb400d0d40 con 0x7fbb400bb770 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 0x7fbb4014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 0x7fbb4014ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41718/0 (socket says 172.21.15.46:41718) 2024-05-09T03:21:28.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2023344917 learned_addr learned my addr 172.21.15.46:0/2023344917 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb37fff640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb400d2a70 0x7fbb4014e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb400bb770 0x7fbb401f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb400bb770 msgr2=0x7fbb401f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb400bb770 0x7fbb401f6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb400d2a70 msgr2=0x7fbb4014e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb400d2a70 0x7fbb4014e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.620+0000 7fbb3ed9d640 1 -- 172.21.15.46:0/2023344917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4014f6e0 con 0x7fbb40163eb0 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.621+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 0x7fbb4014ee60 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e80 tx=0x7fbb38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.621+0000 7fbb36ffd640 1 -- 172.21.15.46:0/2023344917 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3803f070 con 0x7fbb40163eb0 2024-05-09T03:21:28.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.621+0000 7fbb36ffd640 1 -- 172.21.15.46:0/2023344917 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb38031c40 con 0x7fbb40163eb0 2024-05-09T03:21:28.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.621+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4014cac0 con 0x7fbb40163eb0 2024-05-09T03:21:28.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.621+0000 7fbb36ffd640 1 -- 172.21.15.46:0/2023344917 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38009450 con 0x7fbb40163eb0 2024-05-09T03:21:28.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.621+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb4014e620 con 0x7fbb40163eb0 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.622+0000 7fbb36ffd640 1 -- 172.21.15.46:0/2023344917 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb38003990 con 0x7fbb40163eb0 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.623+0000 7fbb36ffd640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb20076300 0x7fbb20078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.623+0000 7fbb36ffd640 1 -- 172.21.15.46:0/2023344917 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fbb380b86d0 con 0x7fbb40163eb0 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.623+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb20076300 0x7fbb20078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.624+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb20076300 0x7fbb20078810 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005880 tx=0x7fbb30009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.624+0000 7fbb4b7b5740 1 Processor -- start 2024-05-09T03:21:28.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.624+0000 7fbb4b7b5740 1 -- start start 2024-05-09T03:21:28.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 0x55e7af1c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedc83e0 0x55e7af1bda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af196d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e7af0fa130 con 0x55e7aedf12a0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e7af0b35d0 con 0x55e7aedd05c0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e7af0f60f0 con 0x55e7aedc83e0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 0x55e7af1c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 0x55e7af1c10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41722/0 (socket says 172.21.15.46:41722) 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 -- 172.21.15.46:0/1053615296 learned_addr learned my addr 172.21.15.46:0/1053615296 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb37fff640 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedc83e0 0x55e7af1bda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 -- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedc83e0 msgr2=0x55e7af1bda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedc83e0 0x55e7af1bda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 -- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 msgr2=0x55e7af196d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af196d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 -- 172.21.15.46:0/1053615296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7aeed36e0 con 0x55e7aedf12a0 2024-05-09T03:21:28.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 0x55e7af1c10e0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000fa50 tx=0x7fbb30012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955f2042db04c62 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.626+0000 7fbb1ffff640 1 -- 172.21.15.46:0/1053615296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30025070 con 0x55e7aedf12a0 2024-05-09T03:21:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb1ffff640 1 -- 172.21.15.46:0/1053615296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb3001b050 con 0x55e7aedf12a0 2024-05-09T03:21:28.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb1ffff640 1 -- 172.21.15.46:0/1053615296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30014de0 con 0x55e7aedf12a0 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/1053615296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 msgr2=0x55e7af1c10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 0x55e7af1c10e0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000fa50 tx=0x7fbb30012000 comp rx=0 tx=0).stop 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/1053615296 shutdown_connections 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedc83e0 0x55e7af1bda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/1053615296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedf12a0 0x55e7af1c10e0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.627+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/1053615296 >> 172.21.15.46:0/1053615296 conn(0x55e7aede53f0 msgr2=0x55e7af1964d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.628+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/1053615296 shutdown_connections 2024-05-09T03:21:28.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.628+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/1053615296 wait complete. 2024-05-09T03:21:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.628+0000 7fbb4b7b5740 1 -- start start 2024-05-09T03:21:28.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.629+0000 7fbb4b7b5740 1 Processor -- start 2024-05-09T03:21:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1efba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af1dd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1dd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e7af080ac0 con 0x55e7aedc83e0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e7af1dad40 con 0x55e7aedd05c0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e7af1dbae0 con 0x55e7aedf12a0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb4b7b5740 10 client.0 _ll_register_callbacks cb 0x55e7af0a3110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1efba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41732/0 (socket says 172.21.15.46:41732) 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 -- 172.21.15.46:0/962265912 learned_addr learned my addr 172.21.15.46:0/962265912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb37fff640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af1dd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1dd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 -- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 msgr2=0x55e7af1dd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1dd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 -- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 msgr2=0x55e7af1dd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af1dd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3e59c640 1 -- 172.21.15.46:0/962265912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7aef37ed0 con 0x55e7aedc83e0 2024-05-09T03:21:28.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.630+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1dd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1efba0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fbb30001eb0 tx=0x7fbb30007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90253c0b387f5b94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/962265912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30025070 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/962265912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fbb300208a0 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb3001b050 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/962265912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7fbb30039050 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 10 client.11001 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x55e7aeeda3e0 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/962265912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb3001b050 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/962265912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7fbb3003c020 con 0x55e7aedc83e0 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 10 client.11001 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 10 client.11001 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 20 client.11001 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:28.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7aedb0080 0x55e7af1d9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 4 client.11001 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.631+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7af107d40 con 0x55e7aedb0080 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.632+0000 7fbb37fff640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7aedb0080 0x55e7af1d9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.632+0000 7fbb1d7fa640 10 client.11001 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.632+0000 7fbb37fff640 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7aedb0080 0x55e7af1d9c10 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2629c420676ffd57 server_cookie=dd165865d7b532c1 in_seq=0 out_seq=0 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/962265912 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e7af107d40 con 0x55e7aedb0080 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb1d7fa640 10 client.11001 handle_command_reply: tid=0 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 client.11001 shutdown 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 msgr2=0x55e7af1efba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1efba0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fbb30001eb0 tx=0x7fbb30007b20 comp rx=0 tx=0).stop 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 shutdown_connections 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1dd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedd05c0 0x55e7af1dd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7aedb0080 0x55e7af1d9c10 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/962265912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1efba0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb1d7fa640 0 client.11001 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 >> 172.21.15.46:0/962265912 conn(0x55e7aede53f0 msgr2=0x55e7af196d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 shutdown_connections 2024-05-09T03:21:28.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/962265912 wait complete. 2024-05-09T03:21:28.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.633+0000 7fbb4b7b5740 20 client.11001 trim_cache size 0 max 16384 2024-05-09T03:21:28.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.647+0000 7fbb4b7b5740 1 Processor -- start 2024-05-09T03:21:28.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.647+0000 7fbb4b7b5740 1 -- start start 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedc83e0 0x55e7af1c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 0x55e7af1ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb400ca250 con 0x55e7aedd05c0 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbb400c95b0 con 0x55e7aedc83e0 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbb400c9c00 con 0x55e7aedf12a0 2024-05-09T03:21:28.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 0x55e7af1ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 0x55e7af1ff5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41742/0 (socket says 172.21.15.46:41742) 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 -- 172.21.15.46:0/3642198586 learned_addr learned my addr 172.21.15.46:0/3642198586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 -- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 msgr2=0x55e7af1ffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedc83e0 0x55e7af1c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 -- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedc83e0 msgr2=0x55e7af1c12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedc83e0 0x55e7af1c12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 -- 172.21.15.46:0/3642198586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7af195e00 con 0x55e7aedd05c0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb37fff640 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 0x55e7af1ff5a0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fbb280079c0 tx=0x7fbb28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd423f2e9b7f2681 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb1f7fe640 1 -- 172.21.15.46:0/3642198586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28007cc0 con 0x55e7aedd05c0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb1f7fe640 1 -- 172.21.15.46:0/3642198586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb28004020 con 0x55e7aedd05c0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.649+0000 7fbb1f7fe640 1 -- 172.21.15.46:0/3642198586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28007e60 con 0x55e7aedd05c0 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/3642198586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 msgr2=0x55e7af1ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 0x55e7af1ff5a0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fbb280079c0 tx=0x7fbb28008040 comp rx=0 tx=0).stop 2024-05-09T03:21:28.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/3642198586 shutdown_connections 2024-05-09T03:21:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedf12a0 0x55e7af1ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedd05c0 0x55e7af1ff5a0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/3642198586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedc83e0 0x55e7af1c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/3642198586 >> 172.21.15.46:0/3642198586 conn(0x55e7aede53f0 msgr2=0x55e7af1db100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/3642198586 shutdown_connections 2024-05-09T03:21:28.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.650+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/3642198586 wait complete. 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.651+0000 7fbb4b7b5740 1 -- start start 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.651+0000 7fbb4b7b5740 1 Processor -- start 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedd05c0 0x55e7af1d1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedf12a0 0x55e7af25bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e7af1c3310 con 0x55e7aedc83e0 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e7af1daba0 con 0x55e7aedf12a0 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x55e7af1c3960 con 0x55e7aedd05c0 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb4b7b5740 10 client.0 _ll_register_callbacks cb 0x7fbb30023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:28.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1c4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41750/0 (socket says 172.21.15.46:41750) 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 -- 172.21.15.46:0/2207574245 learned_addr learned my addr 172.21.15.46:0/2207574245 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb37fff640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedd05c0 0x55e7af1d1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3ed9d640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedf12a0 0x55e7af25bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 -- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedd05c0 msgr2=0x55e7af1d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedd05c0 0x55e7af1d1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 -- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedf12a0 msgr2=0x55e7af25bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedf12a0 0x55e7af25bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 -- 172.21.15.46:0/2207574245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7af1bde90 con 0x55e7aedc83e0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb3e59c640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1c4d30 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fbb30007d00 tx=0x7fbb3000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de18b18ed6156b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/2207574245 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3002c890 con 0x55e7aedc83e0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.653+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/2207574245 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fbb3003eb00 con 0x55e7aedc83e0 2024-05-09T03:21:28.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb28004020 con 0x55e7aedc83e0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/2207574245 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7fbb3003ed00 con 0x55e7aedc83e0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 10 client.11007 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x55e7af1dd9f0 con 0x55e7aedc83e0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/2207574245 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb28004020 con 0x55e7aedc83e0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/2207574245 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7fbb30045050 con 0x55e7aedc83e0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 10 client.11007 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 10 client.11007 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 20 client.11007 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7af1da5f0 0x55e7af196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 4 client.11007 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 1337125749}] 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "damage rm", "damage_id": 1337125749}) -- 0x55e7af101bb0 con 0x55e7af1da5f0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb37fff640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7af1da5f0 0x55e7af196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.654+0000 7fbb1d7fa640 10 client.11007 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb37fff640 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7af1da5f0 0x55e7af196cb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=364323d20abd8ba9 server_cookie=e6e3d97cd62b5275 in_seq=0 out_seq=0 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb1d7fa640 1 -- 172.21.15.46:0/2207574245 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+0 (crc 0 0 0) 0x55e7af101bb0 con 0x55e7af1da5f0 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb1d7fa640 10 client.11007 handle_command_reply: tid=0 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 client.11007 shutdown 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 msgr2=0x55e7af1c4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1c4d30 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fbb30007d00 tx=0x7fbb3000a000 comp rx=0 tx=0).stop 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 shutdown_connections 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7aedf12a0 0x55e7af25bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x55e7aedd05c0 0x55e7af1d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x55e7af1da5f0 0x55e7af196cb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 --2- 172.21.15.46:0/2207574245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7aedc83e0 0x55e7af1c4d30 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 >> 172.21.15.46:0/2207574245 conn(0x55e7aede53f0 msgr2=0x55e7af1db100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb1d7fa640 0 client.11007 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 shutdown_connections 2024-05-09T03:21:28.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 1 -- 172.21.15.46:0/2207574245 wait complete. 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.655+0000 7fbb4b7b5740 20 client.11007 trim_cache size 0 max 16384 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb20076300 msgr2=0x7fbb20078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb20076300 0x7fbb20078810 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fbb30005880 tx=0x7fbb30009040 comp rx=0 tx=0).stop 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 msgr2=0x7fbb4014ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 0x7fbb4014ee60 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e80 tx=0x7fbb38004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 shutdown_connections 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbb20076300 0x7fbb20078810 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb40163eb0 0x7fbb4014ee60 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb400d2a70 0x7fbb4014e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 --2- 172.21.15.46:0/2023344917 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbb400bb770 0x7fbb401f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 >> 172.21.15.46:0/2023344917 conn(0x7fbb4006e370 msgr2=0x7fbb400b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 shutdown_connections 2024-05-09T03:21:28.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.656+0000 7fbb44900640 1 -- 172.21.15.46:0/2023344917 wait complete. 2024-05-09T03:21:28.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:28.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.783+0000 7f607fd00640 1 Processor -- start 2024-05-09T03:21:28.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.783+0000 7f607fd00640 1 -- start start 2024-05-09T03:21:28.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.784+0000 7f607fd00640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60780b9200 0x7f607808d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.784+0000 7f607fd00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60780d2270 0x7f60780b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.784+0000 7f607fd00640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60780b4ee0 con 0x7f607808d6e0 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.784+0000 7f607fd00640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f60780b5460 con 0x7f60780b9200 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.784+0000 7f607fd00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f60780cfac0 con 0x7f60780d2270 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60780b9200 0x7f607808d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f60780b9200 0x7f607808d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41762/0 (socket says 172.21.15.46:41762) 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 -- 172.21.15.46:0/2058372082 learned_addr learned my addr 172.21.15.46:0/2058372082 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607e29d640 1 --2- 172.21.15.46:0/2058372082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60780d2270 0x7f60780b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 -- 172.21.15.46:0/2058372082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60780d2270 msgr2=0x7f60780b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 --2- 172.21.15.46:0/2058372082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60780d2270 0x7f60780b6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 -- 172.21.15.46:0/2058372082 >> v1:172.21.15.88:6789/0 conn(0x7f607808d6e0 legacy=0x7f60780d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:28.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.785+0000 7f607da9c640 1 -- 172.21.15.46:0/2058372082 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60780b62e0 con 0x7f60780b9200 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607da9c640 1 --2- 172.21.15.46:0/2058372082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780b9200 0x7f607808d1a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f6074009870 tx=0x7f607402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdd3dba0c8dcdf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607ca9a640 1 -- 172.21.15.46:0/2058372082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074037070 con 0x7f60780b9200 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607ca9a640 1 -- 172.21.15.46:0/2058372082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6074032440 con 0x7f60780b9200 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607ca9a640 1 -- 172.21.15.46:0/2058372082 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074032810 con 0x7f60780b9200 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607fd00640 1 -- 172.21.15.46:0/2058372082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780b9200 msgr2=0x7f607808d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607fd00640 1 --2- 172.21.15.46:0/2058372082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780b9200 0x7f607808d1a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f6074009870 tx=0x7f607402a1c0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607fd00640 1 -- 172.21.15.46:0/2058372082 shutdown_connections 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.786+0000 7f607fd00640 1 --2- 172.21.15.46:0/2058372082 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f60780d2270 0x7f60780b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.787+0000 7f607fd00640 1 --2- 172.21.15.46:0/2058372082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780b9200 0x7f607808d1a0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.787+0000 7f607fd00640 1 -- 172.21.15.46:0/2058372082 >> 172.21.15.46:0/2058372082 conn(0x7f60780b56e0 msgr2=0x7f60780b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.787+0000 7f607fd00640 1 -- 172.21.15.46:0/2058372082 shutdown_connections 2024-05-09T03:21:28.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.787+0000 7f607fd00640 1 -- 172.21.15.46:0/2058372082 wait complete. 2024-05-09T03:21:28.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.788+0000 7f607fd00640 1 Processor -- start 2024-05-09T03:21:28.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.788+0000 7f607fd00640 1 -- start start 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607fd00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f607808d6e0 0x7f6078158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607fd00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60780b9200 0x7f60781e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607fd00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 0x7f60781e8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607fd00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f60780cf750 con 0x7f60780d2270 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607fd00640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60780cf470 con 0x7f60780b9200 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607fd00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f60780b4a20 con 0x7f607808d6e0 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 0x7f60781e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 0x7f60781e8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41768/0 (socket says 172.21.15.46:41768) 2024-05-09T03:21:28.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 -- 172.21.15.46:0/4091413364 learned_addr learned my addr 172.21.15.46:0/4091413364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607d29b640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60780b9200 0x7f60781e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607da9c640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f607808d6e0 0x7f6078158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 -- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f607808d6e0 msgr2=0x7f6078158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f607808d6e0 0x7f6078158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 -- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60780b9200 msgr2=0x7f60781e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60780b9200 0x7f60781e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 -- 172.21.15.46:0/4091413364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6078159450 con 0x7f60780d2270 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f607e29d640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 0x7f60781e8810 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f607000cc00 tx=0x7f607000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.790+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607000ce10 con 0x7f60780d2270 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.791+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6070003990 con 0x7f60780d2270 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.791+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60780a0a40 con 0x7f60780d2270 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.791+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070003d60 con 0x7f60780d2270 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.791+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60780a1000 con 0x7f60780d2270 2024-05-09T03:21:28.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.791+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f607806a270 con 0x7f60780d2270 2024-05-09T03:21:28.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.792+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6070003f00 con 0x7f60780d2270 2024-05-09T03:21:28.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.793+0000 7f606e7fc640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f605c076300 0x7f605c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:28.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.793+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f6070097b30 con 0x7f60780d2270 2024-05-09T03:21:28.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.793+0000 7f607da9c640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f605c076300 0x7f605c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:28.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.794+0000 7f607da9c640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f605c076300 0x7f605c078810 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6074034000 tx=0x7f607402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:28.798+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6070014030 con 0x7f60780d2270 2024-05-09T03:21:29.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.027+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60781e75f0 con 0x7f60780d2270 2024-05-09T03:21:29.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.027+0000 7f606e7fc640 1 -- 172.21.15.46:0/4091413364 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 359 v359) ==== 95+0+5190 (secure 0 0 0) 0x7f6070069380 con 0x7f60780d2270 2024-05-09T03:21:29.028 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:29.028 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":359,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":359,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:22.873173+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:29.029 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 359 2024-05-09T03:21:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f605c076300 msgr2=0x7f605c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f605c076300 0x7f605c078810 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f6074034000 tx=0x7f607402e000 comp rx=0 tx=0).stop 2024-05-09T03:21:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 msgr2=0x7f60781e8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 0x7f60781e8810 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f607000cc00 tx=0x7f607000a5c0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 shutdown_connections 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f605c076300 0x7f605c078810 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60780d2270 0x7f60781e8810 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60780b9200 0x7f60781e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 --2- 172.21.15.46:0/4091413364 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f607808d6e0 0x7f6078158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 >> 172.21.15.46:0/4091413364 conn(0x7f60780b56e0 msgr2=0x7f60780b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 shutdown_connections 2024-05-09T03:21:29.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.029+0000 7f607fd00640 1 -- 172.21.15.46:0/4091413364 wait complete. 2024-05-09T03:21:29.041 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 1365243552 2024-05-09T03:21:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.159+0000 7f27d1514640 1 Processor -- start 2024-05-09T03:21:29.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.159+0000 7f27d1514640 1 -- start start 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27d1514640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc0b9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27d1514640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27cc0d1e70 0x7f27cc0d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27d1514640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27cc0ce730 con 0x7f27cc0cf7d0 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27d1514640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f27cc0d3100 con 0x7f27cc0d1e70 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27d1514640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc0cf340 con 0x7f27cc0ce1e0 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27cb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27cc0d1e70 0x7f27cc0d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27caffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f27cc0cf7d0 0x7f27cc0d1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53538/0 (socket says 172.21.15.46:53538) 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.160+0000 7f27caffd640 1 -- 172.21.15.46:0/1566992002 learned_addr learned my addr 172.21.15.46:0/1566992002 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/1566992002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc0b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27cb7fe640 1 -- 172.21.15.46:0/1566992002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 msgr2=0x7f27cc0b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/1566992002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc0b9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27cb7fe640 1 -- 172.21.15.46:0/1566992002 >> v1:172.21.15.46:6789/0 conn(0x7f27cc0cf7d0 legacy=0x7f27cc0d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27c9ffb640 1 -- 172.21.15.46:0/1566992002 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528727854 0 0) 0x7f27cc0ce730 con 0x7f27cc0cf7d0 2024-05-09T03:21:29.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27cb7fe640 1 -- 172.21.15.46:0/1566992002 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc0cea10 con 0x7f27cc0d1e70 2024-05-09T03:21:29.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/1566992002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc0d2280 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f27c40097a0 tx=0x7f27c402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b5a69fb8c882d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.161+0000 7f27c9ffb640 1 -- 172.21.15.46:0/1566992002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4037070 con 0x7f27cc0d1e70 2024-05-09T03:21:29.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27c9ffb640 1 -- 172.21.15.46:0/1566992002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c4032440 con 0x7f27cc0d1e70 2024-05-09T03:21:29.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27c9ffb640 1 -- 172.21.15.46:0/1566992002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4032810 con 0x7f27cc0d1e70 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 -- 172.21.15.46:0/1566992002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 msgr2=0x7f27cc0d2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 --2- 172.21.15.46:0/1566992002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc0d2280 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f27c40097a0 tx=0x7f27c402a180 comp rx=0 tx=0).stop 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 -- 172.21.15.46:0/1566992002 shutdown_connections 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 --2- 172.21.15.46:0/1566992002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc0d2280 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 --2- 172.21.15.46:0/1566992002 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc0b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 -- 172.21.15.46:0/1566992002 >> 172.21.15.46:0/1566992002 conn(0x7f27cc0d3b90 msgr2=0x7f27cc0d3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 -- 172.21.15.46:0/1566992002 shutdown_connections 2024-05-09T03:21:29.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.162+0000 7f27d1514640 1 -- 172.21.15.46:0/1566992002 wait complete. 2024-05-09T03:21:29.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.164+0000 7f27d1514640 1 Processor -- start 2024-05-09T03:21:29.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.164+0000 7f27d1514640 1 -- start start 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.165+0000 7f27d1514640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.165+0000 7f27d1514640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0cf7d0 0x7f27cc06eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.165+0000 7f27d1514640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc1624a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.165+0000 7f27d1514640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc16a0e0 con 0x7f27cc0d1e70 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27d1514640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc0cefd0 con 0x7f27cc0cf7d0 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27d1514640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc16a260 con 0x7f27cc0ce1e0 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc1624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc1624a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41780/0 (socket says 172.21.15.46:41780) 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 -- 172.21.15.46:0/31392385 learned_addr learned my addr 172.21.15.46:0/31392385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27caffd640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0cf7d0 0x7f27cc06eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 -- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 msgr2=0x7f27cc15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc15c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 -- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0cf7d0 msgr2=0x7f27cc06eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0cf7d0 0x7f27cc06eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 -- 172.21.15.46:0/31392385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc06ff80 con 0x7f27cc0d1e70 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27caffd640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc1624a0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f27c40097a0 tx=0x7f27c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27c37fe640 1 -- 172.21.15.46:0/31392385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4037040 con 0x7f27cc0d1e70 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27c37fe640 1 -- 172.21.15.46:0/31392385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c4042070 con 0x7f27cc0d1e70 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27cc070260 con 0x7f27cc0d1e70 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27c37fe640 1 -- 172.21.15.46:0/31392385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4030b30 con 0x7f27cc0d1e70 2024-05-09T03:21:29.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.166+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27cc070820 con 0x7f27cc0d1e70 2024-05-09T03:21:29.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.168+0000 7f27c37fe640 1 -- 172.21.15.46:0/31392385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f27c4030cd0 con 0x7f27cc0d1e70 2024-05-09T03:21:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.168+0000 7f27c37fe640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27ac0762d0 0x7f27ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.169+0000 7f27c37fe640 1 -- 172.21.15.46:0/31392385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f27c40b8000 con 0x7f27cc0d1e70 2024-05-09T03:21:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.169+0000 7f27caffd640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27ac0762d0 0x7f27ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.169+0000 7f27caffd640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27ac0762d0 0x7f27ac0787e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0012e0 tx=0x7f27bc005b90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.170+0000 7f27d83c9740 1 Processor -- start 2024-05-09T03:21:29.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.170+0000 7f27d83c9740 1 -- start start 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e110e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a183e0 0x561fe2e0da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a205c0 0x561fe2de6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27d83c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561fe2d4a130 con 0x561fe2a412a0 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561fe2d035d0 con 0x561fe2a183e0 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561fe2d460f0 con 0x561fe2a205c0 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e110e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41784/0 (socket says 172.21.15.46:41784) 2024-05-09T03:21:29.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27caffd640 1 -- 172.21.15.46:0/2998267530 learned_addr learned my addr 172.21.15.46:0/2998267530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a205c0 0x561fe2de6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.171+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a183e0 0x561fe2e0da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27caffd640 1 -- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a205c0 msgr2=0x561fe2de6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27caffd640 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a205c0 0x561fe2de6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27caffd640 1 -- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a183e0 msgr2=0x561fe2e0da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27caffd640 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a183e0 0x561fe2e0da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27caffd640 1 -- 172.21.15.46:0/2998267530 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fe2b236e0 con 0x561fe2a412a0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a205c0 0x561fe2de6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27caffd640 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e110e0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f27bc001d00 tx=0x7f27bc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfd082e2659aa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27c0ff9640 1 -- 172.21.15.46:0/2998267530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc015a80 con 0x561fe2a412a0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27c0ff9640 1 -- 172.21.15.46:0/2998267530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27bc015c20 con 0x561fe2a412a0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.172+0000 7f27c0ff9640 1 -- 172.21.15.46:0/2998267530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc021450 con 0x561fe2a412a0 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 -- 172.21.15.46:0/2998267530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 msgr2=0x561fe2e110e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e110e0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f27bc001d00 tx=0x7f27bc012000 comp rx=0 tx=0).stop 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 -- 172.21.15.46:0/2998267530 shutdown_connections 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a205c0 0x561fe2de6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a183e0 0x561fe2e0da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2998267530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e110e0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 -- 172.21.15.46:0/2998267530 >> 172.21.15.46:0/2998267530 conn(0x561fe2a353f0 msgr2=0x561fe2de64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 -- 172.21.15.46:0/2998267530 shutdown_connections 2024-05-09T03:21:29.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.173+0000 7f27d83c9740 1 -- 172.21.15.46:0/2998267530 wait complete. 2024-05-09T03:21:29.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.174+0000 7f27d83c9740 1 -- start start 2024-05-09T03:21:29.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.174+0000 7f27d83c9740 1 Processor -- start 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e3b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e13d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e294e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561fe2cd0ac0 con 0x561fe2a412a0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561fe2e29fb0 con 0x561fe2a205c0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561fe2e3c100 con 0x561fe2a183e0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.175+0000 7f27d83c9740 10 client.0 _ll_register_callbacks cb 0x561fe2cf3110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e294e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41796/0 (socket says 172.21.15.46:41796) 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 -- 172.21.15.46:0/2466237114 learned_addr learned my addr 172.21.15.46:0/2466237114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e13d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27caffd640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e3b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 -- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 msgr2=0x561fe2e3b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e3b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 -- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 msgr2=0x561fe2e13d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e13d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 -- 172.21.15.46:0/2466237114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fe2b87ed0 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e294e0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f27c40b64f0 tx=0x7f27c4081820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a53ad526df5aea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27a9ffb640 1 -- 172.21.15.46:0/2466237114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4089d40 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27a9ffb640 1 -- 172.21.15.46:0/2466237114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f27c40b0d80 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27bc015c20 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27a9ffb640 1 -- 172.21.15.46:0/2466237114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7f27c4081900 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27d83c9740 10 client.11028 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x561fe2b2a3e0 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.176+0000 7f27a9ffb640 1 -- 172.21.15.46:0/2466237114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27bc015c20 con 0x561fe2a412a0 2024-05-09T03:21:29.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27a9ffb640 1 -- 172.21.15.46:0/2466237114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7f27c403a030 con 0x561fe2a412a0 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27d83c9740 10 client.11028 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27d83c9740 10 client.11028 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27d83c9740 20 client.11028 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe2a00080 0x561fe2e3f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27d83c9740 4 client.11028 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561fe2d57d40 con 0x561fe2a00080 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27caffd640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe2a00080 0x561fe2e3f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27a9ffb640 10 client.11028 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.177+0000 7f27caffd640 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe2a00080 0x561fe2e3f250 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2446a12aa0e3fe4b server_cookie=77aaebdc0e1b8496 in_seq=0 out_seq=0 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27a9ffb640 1 -- 172.21.15.46:0/2466237114 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561fe2d57d40 con 0x561fe2a00080 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27a9ffb640 10 client.11028 handle_command_reply: tid=0 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 client.11028 shutdown 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 msgr2=0x561fe2e294e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e294e0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f27c40b64f0 tx=0x7f27c4081820 comp rx=0 tx=0).stop 2024-05-09T03:21:29.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 shutdown_connections 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe2a00080 0x561fe2e3f250 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a412a0 0x561fe2e294e0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e13d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 --2- 172.21.15.46:0/2466237114 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e3b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27a9ffb640 0 client.11028 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 >> 172.21.15.46:0/2466237114 conn(0x561fe2a353f0 msgr2=0x561fe2e2fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 shutdown_connections 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 1 -- 172.21.15.46:0/2466237114 wait complete. 2024-05-09T03:21:29.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.178+0000 7f27d83c9740 20 client.11028 trim_cache size 0 max 16384 2024-05-09T03:21:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.192+0000 7f27d83c9740 1 Processor -- start 2024-05-09T03:21:29.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.192+0000 7f27d83c9740 1 -- start start 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a00080 0x561fe2e0b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e0bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 0x561fe2e22580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27d83c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc202cc0 con 0x561fe2a205c0 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc0cd200 con 0x561fe2a00080 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc0c8d50 con 0x561fe2a183e0 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 0x561fe2e22580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 0x561fe2e22580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41804/0 (socket says 172.21.15.46:41804) 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 -- 172.21.15.46:0/3960135164 learned_addr learned my addr 172.21.15.46:0/3960135164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e0bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27caffd640 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a00080 0x561fe2e0b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 -- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 msgr2=0x561fe2e0bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e0bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 -- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a00080 msgr2=0x561fe2e0b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a00080 0x561fe2e0b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 -- 172.21.15.46:0/3960135164 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fe2cf7900 con 0x561fe2a205c0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27caffd640 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a00080 0x561fe2e0b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 0x561fe2e22580 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f27c407d7a0 tx=0x7f27c4083000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee335e3fa4178353 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27abfff640 1 -- 172.21.15.46:0/3960135164 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4089d40 con 0x561fe2a205c0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27abfff640 1 -- 172.21.15.46:0/3960135164 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c4089fd0 con 0x561fe2a205c0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.194+0000 7f27abfff640 1 -- 172.21.15.46:0/3960135164 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4080660 con 0x561fe2a205c0 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 -- 172.21.15.46:0/3960135164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 msgr2=0x561fe2e22580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 0x561fe2e22580 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7f27c407d7a0 tx=0x7f27c4083000 comp rx=0 tx=0).stop 2024-05-09T03:21:29.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 -- 172.21.15.46:0/3960135164 shutdown_connections 2024-05-09T03:21:29.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a205c0 0x561fe2e22580 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e0bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 --2- 172.21.15.46:0/3960135164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a00080 0x561fe2e0b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 -- 172.21.15.46:0/3960135164 >> 172.21.15.46:0/3960135164 conn(0x561fe2a353f0 msgr2=0x561fe2e291a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 -- 172.21.15.46:0/3960135164 shutdown_connections 2024-05-09T03:21:29.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.195+0000 7f27d83c9740 1 -- 172.21.15.46:0/3960135164 wait complete. 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.196+0000 7f27d83c9740 1 -- start start 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.196+0000 7f27d83c9740 1 Processor -- start 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 0x561fe2e3b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e2b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e29ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561fe2e14240 con 0x561fe2a00080 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc0ccbb0 con 0x561fe2a205c0 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x561fe2e0d180 con 0x561fe2a183e0 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.197+0000 7f27d83c9740 10 client.0 _ll_register_callbacks cb 0x7f27c40810c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T03:21:29.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 0x561fe2e3b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 0x561fe2e3b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41814/0 (socket says 172.21.15.46:41814) 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 -- 172.21.15.46:0/517250416 learned_addr learned my addr 172.21.15.46:0/517250416 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e29ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e2b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 -- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 msgr2=0x561fe2e2b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e2b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 -- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 msgr2=0x561fe2e29ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e29ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 -- 172.21.15.46:0/517250416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fe2de7ce0 con 0x561fe2a00080 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27cb7fe640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e29ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e2b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27caffd640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 0x561fe2e3b060 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f27bc015810 tx=0x7f27bc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d635989b58b9ad1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27a9ffb640 1 -- 172.21.15.46:0/517250416 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc033060 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27a9ffb640 1 -- 172.21.15.46:0/517250416 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f27bc02e420 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c4089fd0 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27a9ffb640 1 -- 172.21.15.46:0/517250416 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=359) ==== 24+0+0 (secure 0 0 0) 0x7f27bc039050 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27d83c9740 10 client.11034 fetch_fsmap learned FSMap version 359 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.198+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=359,monmap=2+}) -- 0x561fe2e2a910 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27a9ffb640 1 -- 172.21.15.46:0/517250416 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c4089fd0 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27a9ffb640 1 -- 172.21.15.46:0/517250416 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 359) ==== 3494+0+0 (secure 0 0 0) 0x7f27bc03c020 con 0x561fe2a00080 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 10 client.11034 fetch_fsmap finished waiting for FSMap version 359 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 10 client.11034 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 20 client.11034 populate_metadata read hostname 'smithi046' 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe29d54c0 0x561fe2e4e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 4 client.11034 mds_command: new command op to 10812 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 1365243552}] 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 --> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] -- command(tid 0: {"prefix": "damage rm", "damage_id": 1365243552}) -- 0x561fe2e23740 con 0x561fe29d54c0 2024-05-09T03:21:29.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe29d54c0 0x561fe2e4e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27a9ffb640 10 client.11034 ms_handle_connect on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27ca7fc640 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe29d54c0 0x561fe2e4e380 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e784fe7bfaadfe9 server_cookie=2cd4d6765f45530a in_seq=0 out_seq=0 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27a9ffb640 1 -- 172.21.15.46:0/517250416 <== mds.0 v2:172.21.15.46:6833/2236478370 1 ==== command_reply(tid 0: 0 ) ==== 8+0+0 (crc 0 0 0) 0x561fe2e23740 con 0x561fe29d54c0 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27a9ffb640 10 client.11034 handle_command_reply: tid=0 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.199+0000 7f27d83c9740 1 client.11034 shutdown 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 msgr2=0x561fe2e3b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 0x561fe2e3b060 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f27bc015810 tx=0x7f27bc001250 comp rx=0 tx=0).stop 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 shutdown_connections 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561fe2a205c0 0x561fe2e29ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x561fe2a183e0 0x561fe2e2b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:6833/2236478370,v1:172.21.15.46:6835/2236478370] conn(0x561fe29d54c0 0x561fe2e4e380 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 --2- 172.21.15.46:0/517250416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561fe2a00080 0x561fe2e3b060 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27a9ffb640 0 client.11034 ms_handle_reset on v2:172.21.15.46:6833/2236478370 2024-05-09T03:21:29.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 >> 172.21.15.46:0/517250416 conn(0x561fe2a353f0 msgr2=0x561fe2e104f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 shutdown_connections 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 1 -- 172.21.15.46:0/517250416 wait complete. 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d83c9740 20 client.11034 trim_cache size 0 max 16384 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27ac0762d0 msgr2=0x7f27ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27ac0762d0 0x7f27ac0787e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0012e0 tx=0x7f27bc005b90 comp rx=0 tx=0).stop 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 msgr2=0x7f27cc1624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc1624a0 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f27c40097a0 tx=0x7f27c40046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 shutdown_connections 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f27ac0762d0 0x7f27ac0787e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27cc0d1e70 0x7f27cc1624a0 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0cf7d0 0x7f27cc06eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 --2- 172.21.15.46:0/31392385 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f27cc0ce1e0 0x7f27cc15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 >> 172.21.15.46:0/31392385 conn(0x7f27cc0d3b90 msgr2=0x7f27cc0b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 shutdown_connections 2024-05-09T03:21:29.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.200+0000 7f27d1514640 1 -- 172.21.15.46:0/31392385 wait complete. 2024-05-09T03:21:29.212 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-05-09T03:21:29.212 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:21:29.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-05-09T03:21:29.295 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-05-09T03:21:29.296 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:21:29.296 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:29.296 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:29.312 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:29.312 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:29.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T03:21:29.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.522+0000 7f95eb2e0640 1 Processor -- start 2024-05-09T03:21:29.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.523+0000 7f95eb2e0640 1 -- start start 2024-05-09T03:21:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95eb2e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95e40d3730 0x7f95e406e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95eb2e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f95e406e7e0 0x7f95e4164980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95eb2e0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e409d470 con 0x7f95e4164ec0 2024-05-09T03:21:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95eb2e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95e409cfe0 con 0x7f95e40d3730 2024-05-09T03:21:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95eb2e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95e40b94a0 con 0x7f95e406e7e0 2024-05-09T03:21:29.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95e907c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95e40d3730 0x7f95e406e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95e907c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95e40d3730 0x7f95e406e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41822/0 (socket says 172.21.15.46:41822) 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95e907c640 1 -- 172.21.15.46:0/1407507312 learned_addr learned my addr 172.21.15.46:0/1407507312 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.524+0000 7f95e887b640 1 --2- 172.21.15.46:0/1407507312 >> v2:172.21.15.88:3300/0 conn(0x7f95e406e7e0 0x7f95e4164980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.525+0000 7f95e907c640 1 -- 172.21.15.46:0/1407507312 >> v2:172.21.15.88:3300/0 conn(0x7f95e406e7e0 msgr2=0x7f95e4164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.525+0000 7f95e907c640 1 --2- 172.21.15.46:0/1407507312 >> v2:172.21.15.88:3300/0 conn(0x7f95e406e7e0 0x7f95e4164980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.525+0000 7f95e907c640 1 -- 172.21.15.46:0/1407507312 >> v1:172.21.15.88:6789/0 conn(0x7f95e4164ec0 legacy=0x7f95e40ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.525+0000 7f95e907c640 1 -- 172.21.15.46:0/1407507312 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e40d11b0 con 0x7f95e40d3730 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.525+0000 7f95e887b640 1 --2- 172.21.15.46:0/1407507312 >> v2:172.21.15.88:3300/0 conn(0x7f95e406e7e0 0x7f95e4164980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.525+0000 7f95e907c640 1 --2- 172.21.15.46:0/1407507312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e406e2a0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f95e40d4c70 tx=0x7f95d402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e255973439210b54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95e3fff640 1 -- 172.21.15.46:0/1407507312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4037070 con 0x7f95e40d3730 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95e3fff640 1 -- 172.21.15.46:0/1407507312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d4032440 con 0x7f95e40d3730 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95e3fff640 1 -- 172.21.15.46:0/1407507312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d4032810 con 0x7f95e40d3730 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 -- 172.21.15.46:0/1407507312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 msgr2=0x7f95e406e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/1407507312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e406e2a0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f95e40d4c70 tx=0x7f95d402a140 comp rx=0 tx=0).stop 2024-05-09T03:21:29.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 -- 172.21.15.46:0/1407507312 shutdown_connections 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/1407507312 >> v2:172.21.15.88:3300/0 conn(0x7f95e406e7e0 0x7f95e4164980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/1407507312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e406e2a0 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 -- 172.21.15.46:0/1407507312 >> 172.21.15.46:0/1407507312 conn(0x7f95e40d05f0 msgr2=0x7f95e40d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 -- 172.21.15.46:0/1407507312 shutdown_connections 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.526+0000 7f95eb2e0640 1 -- 172.21.15.46:0/1407507312 wait complete. 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.528+0000 7f95eb2e0640 1 Processor -- start 2024-05-09T03:21:29.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.528+0000 7f95eb2e0640 1 -- start start 2024-05-09T03:21:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.529+0000 7f95eb2e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95e406e7e0 0x7f95e40c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95eb2e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e4161b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95eb2e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95e4164ec0 0x7f95e409d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95eb2e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95e408cd60 con 0x7f95e40d3730 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95eb2e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95e40b9160 con 0x7f95e406e7e0 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95eb2e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f95e408cee0 con 0x7f95e4164ec0 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e4161b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e4161b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41824/0 (socket says 172.21.15.46:41824) 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 -- 172.21.15.46:0/376190298 learned_addr learned my addr 172.21.15.46:0/376190298 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e907c640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95e406e7e0 0x7f95e40c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e987d640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95e4164ec0 0x7f95e409d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 -- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95e4164ec0 msgr2=0x7f95e409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95e4164ec0 0x7f95e409d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 -- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95e406e7e0 msgr2=0x7f95e40c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95e406e7e0 0x7f95e40c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 -- 172.21.15.46:0/376190298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e40c9420 con 0x7f95e40d3730 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e887b640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e4161b80 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c7f0 tx=0x7f95d800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8015070 con 0x7f95e40d3730 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d80077c0 con 0x7f95e40d3730 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e40c9700 con 0x7f95e40d3730 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.530+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8007b70 con 0x7f95e40d3730 2024-05-09T03:21:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.531+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e409faa0 con 0x7f95e40d3730 2024-05-09T03:21:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.531+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e40b65b0 con 0x7f95e40d3730 2024-05-09T03:21:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.532+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95d8018070 con 0x7f95e40d3730 2024-05-09T03:21:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.533+0000 7f95e1ffb640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d0076300 0x7f95d0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.533+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f95d8097760 con 0x7f95e40d3730 2024-05-09T03:21:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.533+0000 7f95e907c640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d0076300 0x7f95d0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.534+0000 7f95e907c640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d0076300 0x7f95d0078810 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f95d4034000 tx=0x7f95d402e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.538+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95e40b65b0 con 0x7f95e40d3730 2024-05-09T03:21:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.741+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f95e406a270 con 0x7f95e40d3730 2024-05-09T03:21:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.741+0000 7f95e1ffb640 1 -- 172.21.15.46:0/376190298 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v136) ==== 80+0+122 (secure 0 0 0) 0x7f95d8060fd0 con 0x7f95e40d3730 2024-05-09T03:21:29.742 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/2230300620 2024-05-10T03:21:18.861166+0000 2024-05-09T03:21:29.743 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/2230300620 2024-05-10T03:21:18.861166+0000 2024-05-09T03:21:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d0076300 msgr2=0x7f95d0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d0076300 0x7f95d0078810 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f95d4034000 tx=0x7f95d402e000 comp rx=0 tx=0).stop 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 msgr2=0x7f95e4161b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e4161b80 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f95d800c7f0 tx=0x7f95d800ccb0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 shutdown_connections 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f95e4164ec0 0x7f95e409d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f95d0076300 0x7f95d0078810 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95e40d3730 0x7f95e4161b80 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 --2- 172.21.15.46:0/376190298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95e406e7e0 0x7f95e40c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 >> 172.21.15.46:0/376190298 conn(0x7f95e40d05f0 msgr2=0x7f95e40d2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 shutdown_connections 2024-05-09T03:21:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:29.744+0000 7f95eb2e0640 1 -- 172.21.15.46:0/376190298 wait complete. 2024-05-09T03:21:29.755 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-05-09T03:21:29.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:29.756 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T03:21:29.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[89299]: fuse finished with error 0 and tester_r 0 2024-05-09T03:21:29.781 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:21:29.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:29.782 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T03:21:29.949 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:21:29.949 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:21:29.949 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:29.969 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:21:29.969 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:21:29.969 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:29.969 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:30.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:21:30.006 INFO:tasks.cephfs.mount:Terminating background process 2024-05-09T03:21:30.006 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:21:30.006 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:30.027 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:21:30.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:30.067 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:21:30.066+0000 7fafc24c3640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T03:21:30.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.161+0000 7fde47f1c640 1 Processor -- start 2024-05-09T03:21:30.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.161+0000 7fde47f1c640 1 -- start start 2024-05-09T03:21:30.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.162+0000 7fde47f1c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde400d2b90 0x7fde4015dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.162+0000 7fde47f1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde40160b20 con 0x7fde401615e0 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.162+0000 7fde47f1c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde400d15f0 con 0x7fde40164c70 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.162+0000 7fde47f1c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fde4015e520 con 0x7fde400d2b90 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde464b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde400d2b90 0x7fde4015dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde454b7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde401615e0 0x7fde400d2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53550/0 (socket says 172.21.15.46:53550) 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde454b7640 1 -- 172.21.15.46:0/3552932690 learned_addr learned my addr 172.21.15.46:0/3552932690 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde464b9640 1 -- 172.21.15.46:0/3552932690 >> v1:172.21.15.88:6789/0 conn(0x7fde40164c70 legacy=0x7fde400d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde464b9640 1 -- 172.21.15.46:0/3552932690 >> v1:172.21.15.46:6789/0 conn(0x7fde401615e0 legacy=0x7fde400d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde44cb6640 1 -- 172.21.15.46:0/3552932690 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721245289 0 0) 0x7fde40160b20 con 0x7fde401615e0 2024-05-09T03:21:30.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.163+0000 7fde464b9640 1 -- 172.21.15.46:0/3552932690 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde400d18d0 con 0x7fde400d2b90 2024-05-09T03:21:30.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.164+0000 7fde464b9640 1 --2- 172.21.15.46:0/3552932690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde4015dfe0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fde38009d90 tx=0x7fde3802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b210fa1a28e7640 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.164+0000 7fde44cb6640 1 -- 172.21.15.46:0/3552932690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38031600 con 0x7fde400d2b90 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.164+0000 7fde44cb6640 1 -- 172.21.15.46:0/3552932690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde380317a0 con 0x7fde400d2b90 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.164+0000 7fde44cb6640 1 -- 172.21.15.46:0/3552932690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38031b70 con 0x7fde400d2b90 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.164+0000 7fde47f1c640 1 -- 172.21.15.46:0/3552932690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 msgr2=0x7fde4015dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.164+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3552932690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde4015dfe0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fde38009d90 tx=0x7fde3802a5d0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.165+0000 7fde47f1c640 1 -- 172.21.15.46:0/3552932690 shutdown_connections 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.165+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3552932690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde4015dfe0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.165+0000 7fde47f1c640 1 -- 172.21.15.46:0/3552932690 >> 172.21.15.46:0/3552932690 conn(0x7fde40073db0 msgr2=0x7fde400741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.165+0000 7fde47f1c640 1 -- 172.21.15.46:0/3552932690 shutdown_connections 2024-05-09T03:21:30.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.165+0000 7fde47f1c640 1 -- 172.21.15.46:0/3552932690 wait complete. 2024-05-09T03:21:30.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.166+0000 7fde47f1c640 1 Processor -- start 2024-05-09T03:21:30.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.166+0000 7fde47f1c640 1 -- start start 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde47f1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde400ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde47f1c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde401615e0 0x7fde400ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde47f1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40164c70 0x7fde400ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde47f1c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde401619f0 con 0x7fde400d2b90 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde47f1c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde40161150 con 0x7fde40164c70 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde47f1c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fde400d45c0 con 0x7fde401615e0 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde400ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde400ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 -- 172.21.15.46:0/3121379225 learned_addr learned my addr 172.21.15.46:0/3121379225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde464b9640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40164c70 0x7fde400ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde454b7640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde401615e0 0x7fde400ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 -- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde401615e0 msgr2=0x7fde400ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde401615e0 0x7fde400ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 -- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40164c70 msgr2=0x7fde400ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40164c70 0x7fde400ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 -- 172.21.15.46:0/3121379225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde400cc4a0 con 0x7fde400d2b90 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde45cb8640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde400ca330 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00b3f0 tx=0x7fde3c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.168+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c015020 con 0x7fde400d2b90 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.169+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c011830 con 0x7fde400d2b90 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.169+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde40153c10 con 0x7fde400d2b90 2024-05-09T03:21:30.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.169+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c011c20 con 0x7fde400d2b90 2024-05-09T03:21:30.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.169+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde401541d0 con 0x7fde400d2b90 2024-05-09T03:21:30.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.169+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde4008cfd0 con 0x7fde400d2b90 2024-05-09T03:21:30.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.170+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fde3c018020 con 0x7fde400d2b90 2024-05-09T03:21:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.171+0000 7fde367fc640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde240762d0 0x7fde240787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.171+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fde3c097430 con 0x7fde400d2b90 2024-05-09T03:21:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.171+0000 7fde454b7640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde240762d0 0x7fde240787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.172+0000 7fde454b7640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde240762d0 0x7fde240787e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fde400c9ff0 tx=0x7fde30007c00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.176+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fde4008cfd0 con 0x7fde400d2b90 2024-05-09T03:21:30.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.403+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde4006a270 con 0x7fde400d2b90 2024-05-09T03:21:30.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.404+0000 7fde367fc640 1 -- 172.21.15.46:0/3121379225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) ==== 95+0+5190 (secure 0 0 0) 0x7fde3c01f030 con 0x7fde400d2b90 2024-05-09T03:21:30.405 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:30.405 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":360,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":360,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:29.873183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:30.405 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 360 2024-05-09T03:21:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde240762d0 msgr2=0x7fde240787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde240762d0 0x7fde240787e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fde400c9ff0 tx=0x7fde30007c00 comp rx=0 tx=0).stop 2024-05-09T03:21:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 msgr2=0x7fde400ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde400ca330 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00b3f0 tx=0x7fde3c00b8b0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 shutdown_connections 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde40164c70 0x7fde400ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde401615e0 0x7fde400ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde240762d0 0x7fde240787e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 --2- 172.21.15.46:0/3121379225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde400d2b90 0x7fde400ca330 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 >> 172.21.15.46:0/3121379225 conn(0x7fde40073db0 msgr2=0x7fde400b9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 shutdown_connections 2024-05-09T03:21:30.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.406+0000 7fde47f1c640 1 -- 172.21.15.46:0/3121379225 wait complete. 2024-05-09T03:21:30.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:30.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.572+0000 7f38167e0640 1 Processor -- start 2024-05-09T03:21:30.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.572+0000 7f38167e0640 1 -- start start 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f38167e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f38100b9200 0x7f381008d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f38167e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f38100d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f38167e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38100d20a0 0x7f38100d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f38167e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f38100d29f0 con 0x7f38100d20a0 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f38167e0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f38100b5460 con 0x7f38100b9200 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f38167e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38100cfac0 con 0x7f381008d6e0 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38100d20a0 0x7f38100d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f38100d20a0 0x7f38100d24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41860/0 (socket says 172.21.15.46:41860) 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 -- 172.21.15.46:0/1519631719 learned_addr learned my addr 172.21.15.46:0/1519631719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f380f7fe640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f38100d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f380ffff640 1 --2- 172.21.15.46:0/1519631719 >> v2:172.21.15.88:3300/0 conn(0x7f38100b9200 0x7f381008d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 -- 172.21.15.46:0/1519631719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 msgr2=0x7f38100d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:30.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f38100d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 -- 172.21.15.46:0/1519631719 >> v2:172.21.15.88:3300/0 conn(0x7f38100b9200 msgr2=0x7f381008d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 --2- 172.21.15.46:0/1519631719 >> v2:172.21.15.88:3300/0 conn(0x7f38100b9200 0x7f381008d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f3814d7d640 1 -- 172.21.15.46:0/1519631719 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38100b4ee0 con 0x7f38100d20a0 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f380ffff640 1 --2- 172.21.15.46:0/1519631719 >> v2:172.21.15.88:3300/0 conn(0x7f38100b9200 0x7f381008d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.574+0000 7f380f7fe640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f38100d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.575+0000 7f3814d7d640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f38100d24b0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f38100b50b0 tx=0x7f380802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e16051056fa596 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.575+0000 7f380effd640 1 -- 172.21.15.46:0/1519631719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808036070 con 0x7f38100d20a0 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.575+0000 7f380effd640 1 -- 172.21.15.46:0/1519631719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3808031600 con 0x7f38100d20a0 2024-05-09T03:21:30.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.575+0000 7f380effd640 1 -- 172.21.15.46:0/1519631719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38080319d0 con 0x7f38100d20a0 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 -- 172.21.15.46:0/1519631719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 msgr2=0x7f38100d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f38100d24b0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f38100b50b0 tx=0x7f380802a680 comp rx=0 tx=0).stop 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 -- 172.21.15.46:0/1519631719 shutdown_connections 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f38100d24b0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 --2- 172.21.15.46:0/1519631719 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f38100d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 --2- 172.21.15.46:0/1519631719 >> v2:172.21.15.88:3300/0 conn(0x7f38100b9200 0x7f381008d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 -- 172.21.15.46:0/1519631719 >> 172.21.15.46:0/1519631719 conn(0x7f38100b56e0 msgr2=0x7f38100b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 -- 172.21.15.46:0/1519631719 shutdown_connections 2024-05-09T03:21:30.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.576+0000 7f38167e0640 1 -- 172.21.15.46:0/1519631719 wait complete. 2024-05-09T03:21:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.578+0000 7f38167e0640 1 Processor -- start 2024-05-09T03:21:30.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.578+0000 7f38167e0640 1 -- start start 2024-05-09T03:21:30.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f38167e0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f3810072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f38167e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38100b9200 0x7f3810152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f38167e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f3810152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f38167e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f38100cf470 con 0x7f38100d20a0 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f38167e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f38100b5180 con 0x7f38100b9200 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f38167e0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f38100cf750 con 0x7f381008d6e0 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f3814d7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f3810152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.579+0000 7f3814d7d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f3810152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41876/0 (socket says 172.21.15.46:41876) 2024-05-09T03:21:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 -- 172.21.15.46:0/2355218286 learned_addr learned my addr 172.21.15.46:0/2355218286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f380ffff640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f3810072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f380f7fe640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38100b9200 0x7f3810152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 -- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 msgr2=0x7f3810072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f3810072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 -- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38100b9200 msgr2=0x7f3810152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38100b9200 0x7f3810152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 -- 172.21.15.46:0/2355218286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38101510b0 con 0x7f38100d20a0 2024-05-09T03:21:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f3814d7d640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f3810152ff0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f3808009f10 tx=0x7f3808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808041020 con 0x7f38100d20a0 2024-05-09T03:21:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3808007780 con 0x7f38100d20a0 2024-05-09T03:21:30.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808036040 con 0x7f38100d20a0 2024-05-09T03:21:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381015f4d0 con 0x7f38100d20a0 2024-05-09T03:21:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.580+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381015fa90 con 0x7f38100d20a0 2024-05-09T03:21:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.581+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381006a270 con 0x7f38100d20a0 2024-05-09T03:21:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.582+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3808044020 con 0x7f38100d20a0 2024-05-09T03:21:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.583+0000 7f380cff9640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37fc0762d0 0x7f37fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.583+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f38080b86c0 con 0x7f38100d20a0 2024-05-09T03:21:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.583+0000 7f380ffff640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37fc0762d0 0x7f37fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.583+0000 7f380ffff640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37fc0762d0 0x7f37fc0787e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f380000aac0 tx=0x7f3800008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.587+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f380804b030 con 0x7f38100d20a0 2024-05-09T03:21:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.821+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38100b88e0 con 0x7f38100d20a0 2024-05-09T03:21:30.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.822+0000 7f380cff9640 1 -- 172.21.15.46:0/2355218286 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) ==== 95+0+5190 (secure 0 0 0) 0x7f38101510b0 con 0x7f38100d20a0 2024-05-09T03:21:30.823 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:30.823 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":360,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":360,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:29.873183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:30.823 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 360 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37fc0762d0 msgr2=0x7f37fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37fc0762d0 0x7f37fc0787e0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f380000aac0 tx=0x7f3800008040 comp rx=0 tx=0).stop 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 msgr2=0x7f3810152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f3810152ff0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f3808009f10 tx=0x7f3808004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 shutdown_connections 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f37fc0762d0 0x7f37fc0787e0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38100d20a0 0x7f3810152ff0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38100b9200 0x7f3810152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 --2- 172.21.15.46:0/2355218286 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f381008d6e0 0x7f3810072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 >> 172.21.15.46:0/2355218286 conn(0x7f38100b56e0 msgr2=0x7f3810154010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 shutdown_connections 2024-05-09T03:21:30.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.824+0000 7f38167e0640 1 -- 172.21.15.46:0/2355218286 wait complete. 2024-05-09T03:21:30.836 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:21:30.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.950+0000 7fa58ff34640 1 Processor -- start 2024-05-09T03:21:30.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.951+0000 7fa58ff34640 1 -- start start 2024-05-09T03:21:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58ff34640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5880b5210 0x7fa5881613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58ff34640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5880d4800 0x7fa5880d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58ff34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5880d5280 con 0x7fa5880b77c0 2024-05-09T03:21:30.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58ff34640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa5880b5620 con 0x7fa5880b5210 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58ff34640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa5881609a0 con 0x7fa5880d4800 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58dcd0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5880b5210 0x7fa5881613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58dcd0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5880b5210 0x7fa5881613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41884/0 (socket says 172.21.15.46:41884) 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58dcd0640 1 -- 172.21.15.46:0/1796812053 learned_addr learned my addr 172.21.15.46:0/1796812053 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.952+0000 7fa58d4cf640 1 --2- 172.21.15.46:0/1796812053 >> v2:172.21.15.88:3300/0 conn(0x7fa5880d4800 0x7fa5880d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.953+0000 7fa58dcd0640 1 -- 172.21.15.46:0/1796812053 >> v2:172.21.15.88:3300/0 conn(0x7fa5880d4800 msgr2=0x7fa5880d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.953+0000 7fa58dcd0640 1 --2- 172.21.15.46:0/1796812053 >> v2:172.21.15.88:3300/0 conn(0x7fa5880d4800 0x7fa5880d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.953+0000 7fa58dcd0640 1 -- 172.21.15.46:0/1796812053 >> v1:172.21.15.88:6789/0 conn(0x7fa5880b77c0 legacy=0x7fa5880b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:30.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.953+0000 7fa58dcd0640 1 -- 172.21.15.46:0/1796812053 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5880d5560 con 0x7fa5880b5210 2024-05-09T03:21:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.953+0000 7fa58dcd0640 1 --2- 172.21.15.46:0/1796812053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881613c0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fa58806ee20 tx=0x7fa58402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d44b2ebdd9e04d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.953+0000 7fa58ccce640 1 -- 172.21.15.46:0/1796812053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584037070 con 0x7fa5880b5210 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ccce640 1 -- 172.21.15.46:0/1796812053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa584032440 con 0x7fa5880b5210 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ccce640 1 -- 172.21.15.46:0/1796812053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584032810 con 0x7fa5880b5210 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 -- 172.21.15.46:0/1796812053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 msgr2=0x7fa5881613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 --2- 172.21.15.46:0/1796812053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881613c0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fa58806ee20 tx=0x7fa58402a140 comp rx=0 tx=0).stop 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 -- 172.21.15.46:0/1796812053 shutdown_connections 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 --2- 172.21.15.46:0/1796812053 >> v2:172.21.15.88:3300/0 conn(0x7fa5880d4800 0x7fa5880d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 --2- 172.21.15.46:0/1796812053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881613c0 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 -- 172.21.15.46:0/1796812053 >> 172.21.15.46:0/1796812053 conn(0x7fa5880d29a0 msgr2=0x7fa5880d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 -- 172.21.15.46:0/1796812053 shutdown_connections 2024-05-09T03:21:30.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.954+0000 7fa58ff34640 1 -- 172.21.15.46:0/1796812053 wait complete. 2024-05-09T03:21:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.956+0000 7fa58ff34640 1 Processor -- start 2024-05-09T03:21:30.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.956+0000 7fa58ff34640 1 -- start start 2024-05-09T03:21:30.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58ff34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58ff34640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5880b77c0 0x7fa5880cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58ff34640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5880d4800 0x7fa5881fa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58ff34640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5880d0ae0 con 0x7fa5880b5210 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58ff34640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa588160ea0 con 0x7fa5880b77c0 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58ff34640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa588163e60 con 0x7fa5880d4800 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.957+0000 7fa58dcd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881f64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41898/0 (socket says 172.21.15.46:41898) 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 -- 172.21.15.46:0/876420150 learned_addr learned my addr 172.21.15.46:0/876420150 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58e4d1640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5880d4800 0x7fa5881fa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58d4cf640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5880b77c0 0x7fa5880cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 -- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5880d4800 msgr2=0x7fa5881fa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5880d4800 0x7fa5881fa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 -- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5880b77c0 msgr2=0x7fa5880cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5880b77c0 0x7fa5880cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 -- 172.21.15.46:0/876420150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5881f6a00 con 0x7fa5880b5210 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58e4d1640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5880d4800 0x7fa5881fa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58dcd0640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881f64c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fa58402a180 tx=0x7fa5840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584041020 con 0x7fa5880b5210 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5840306c0 con 0x7fa5880b5210 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5880b9dd0 con 0x7fa5880b5210 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584037040 con 0x7fa5880b5210 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.958+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5880ccdf0 con 0x7fa5880b5210 2024-05-09T03:21:30.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.959+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5880cd020 con 0x7fa5880b5210 2024-05-09T03:21:30.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.960+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa584044020 con 0x7fa5880b5210 2024-05-09T03:21:30.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.961+0000 7fa57e7fc640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa56c076300 0x7fa56c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:30.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.961+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fa5840b8150 con 0x7fa5880b5210 2024-05-09T03:21:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.961+0000 7fa58d4cf640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa56c076300 0x7fa56c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:30.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.962+0000 7fa58d4cf640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa56c076300 0x7fa56c078810 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa578006fd0 tx=0x7fa578008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:30.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:30.965+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa5880cd020 con 0x7fa5880b5210 2024-05-09T03:21:31.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.157+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa58806a270 con 0x7fa5880b5210 2024-05-09T03:21:31.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.158+0000 7fa57e7fc640 1 -- 172.21.15.46:0/876420150 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) ==== 74+0+16676 (secure 0 0 0) 0x7fa58404b030 con 0x7fa5880b5210 2024-05-09T03:21:31.159 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:31.159 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":136,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:21:18.861195+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:20:59.663057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-09T03:21:00.927261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/2230300620":"2024-05-10T03:21:18.861166+0000","172.21.15.46:6833/2230300620":"2024-05-10T03:21:18.861166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:21:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa56c076300 msgr2=0x7fa56c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa56c076300 0x7fa56c078810 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fa578006fd0 tx=0x7fa578008040 comp rx=0 tx=0).stop 2024-05-09T03:21:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 msgr2=0x7fa5881f64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881f64c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fa58402a180 tx=0x7fa5840046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 shutdown_connections 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa5880d4800 0x7fa5881fa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5880b77c0 0x7fa5880cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa56c076300 0x7fa56c078810 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 --2- 172.21.15.46:0/876420150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5880b5210 0x7fa5881f64c0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 >> 172.21.15.46:0/876420150 conn(0x7fa5880d29a0 msgr2=0x7fa58815ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 shutdown_connections 2024-05-09T03:21:31.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.160+0000 7fa58ff34640 1 -- 172.21.15.46:0/876420150 wait complete. 2024-05-09T03:21:31.172 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:21:31.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.286+0000 7fe7d2f00640 1 Processor -- start 2024-05-09T03:21:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.286+0000 7fe7d2f00640 1 -- start start 2024-05-09T03:21:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d2f00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 0x7fe7cc0b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d2f00640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 0x7fe7cc163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d2f00640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7cc0b8110 con 0x7fe7cc0b5210 2024-05-09T03:21:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d2f00640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe7cc0b5620 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d2f00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc1609a0 con 0x7fe7cc1613c0 2024-05-09T03:21:31.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d149d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 0x7fe7cc163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 0x7fe7cc0b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d0c9c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe7cc0b5210 0x7fe7cc0d4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:54874/0 (socket says 172.21.15.46:54874) 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d149d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 0x7fe7cc163e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:42944/0 (socket says 172.21.15.46:42944) 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 0x7fe7cc0b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:56514/0 (socket says 172.21.15.46:56514) 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7d0c9c640 1 -- 172.21.15.46:0/2876098721 learned_addr learned my addr 172.21.15.46:0/2876098721 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2876098721 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 msgr2=0x7fe7cc163e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7cbfff640 1 --2- 172.21.15.46:0/2876098721 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 0x7fe7cc163e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.288+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2876098721 >> v1:172.21.15.88:6789/0 conn(0x7fe7cc0b5210 legacy=0x7fe7cc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.289+0000 7fe7cb7fe640 1 -- 172.21.15.46:0/2876098721 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3100800001 0 0) 0x7fe7c4001110 con 0x7fe7cc0b5210 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.289+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2876098721 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc1617d0 con 0x7fe7cc1613c0 2024-05-09T03:21:31.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.289+0000 7fe7d149d640 1 --2- 172.21.15.46:0/2876098721 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 0x7fe7cc163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.289+0000 7fe7cbfff640 1 --2- 172.21.15.46:0/2876098721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 0x7fe7cc0b77c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009d90 tx=0x7fe7bc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2917ab8e7da23024 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7cb7fe640 1 -- 172.21.15.46:0/2876098721 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc007740 con 0x7fe7cc1613c0 2024-05-09T03:21:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7cb7fe640 1 -- 172.21.15.46:0/2876098721 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7bc0078e0 con 0x7fe7cc1613c0 2024-05-09T03:21:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7cb7fe640 1 -- 172.21.15.46:0/2876098721 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc007cb0 con 0x7fe7cc1613c0 2024-05-09T03:21:31.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2876098721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 msgr2=0x7fe7cc0b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2876098721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 0x7fe7cc0b77c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009d90 tx=0x7fe7bc02a660 comp rx=0 tx=0).stop 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2876098721 shutdown_connections 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2876098721 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b7d00 0x7fe7cc163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2876098721 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc1613c0 0x7fe7cc0b77c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2876098721 >> 172.21.15.46:0/2876098721 conn(0x7fe7cc0d29a0 msgr2=0x7fe7cc0d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2876098721 shutdown_connections 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.290+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2876098721 wait complete. 2024-05-09T03:21:31.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.292+0000 7fe7d2f00640 1 Processor -- start 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.292+0000 7fe7d2f00640 1 -- start start 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d2f00640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc0b5210 0x7fe7cc0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d2f00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 0x7fe7cc0c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d2f00640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc1613c0 0x7fe7cc15d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d2f00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc160bc0 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d2f00640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc0d5560 con 0x7fe7cc1613c0 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d2f00640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc160ea0 con 0x7fe7cc0b5210 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 0x7fe7cc0c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 0x7fe7cc0c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41906/0 (socket says 172.21.15.46:41906) 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2603385945 learned_addr learned my addr 172.21.15.46:0/2603385945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d0c9c640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc0b5210 0x7fe7cc0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7d149d640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc1613c0 0x7fe7cc15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc0b5210 msgr2=0x7fe7cc0c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc0b5210 0x7fe7cc0c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc1613c0 msgr2=0x7fe7cc15d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc1613c0 0x7fe7cc15d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 -- 172.21.15.46:0/2603385945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc1faa80 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.294+0000 7fe7cbfff640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 0x7fe7cc0c9d90 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009d90 tx=0x7fe7bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.295+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc03f070 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.295+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7bc031c40 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.295+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc009450 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.295+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc1fad60 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.295+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc15e310 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.296+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc06a270 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.296+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe7bc043410 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.297+0000 7fe7c97fa640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ac0762d0 0x7fe7ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.297+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fe7bc0b8390 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.297+0000 7fe7d0c9c640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ac0762d0 0x7fe7ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.298+0000 7fe7d0c9c640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ac0762d0 0x7fe7ac0787e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4007fd0 tx=0x7fe7c4007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:31.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.302+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe7bc04a070 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.529+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7cc0d40d0 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.530+0000 7fe7c97fa640 1 -- 172.21.15.46:0/2603385945 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) ==== 95+0+5190 (secure 0 0 0) 0x7fe7bc0815b0 con 0x7fe7cc0b7d00 2024-05-09T03:21:31.531 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:31.531 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":360,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":360,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:29.873183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:31.531 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 360 2024-05-09T03:21:31.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ac0762d0 msgr2=0x7fe7ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ac0762d0 0x7fe7ac0787e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4007fd0 tx=0x7fe7c4007de0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 msgr2=0x7fe7cc0c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 0x7fe7cc0c9d90 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009d90 tx=0x7fe7bc004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 shutdown_connections 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc1613c0 0x7fe7cc15d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe7ac0762d0 0x7fe7ac0787e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7cc0b7d00 0x7fe7cc0c9d90 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 --2- 172.21.15.46:0/2603385945 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe7cc0b5210 0x7fe7cc0c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 >> 172.21.15.46:0/2603385945 conn(0x7fe7cc0d29a0 msgr2=0x7fe7cc163df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 shutdown_connections 2024-05-09T03:21:31.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.532+0000 7fe7d2f00640 1 -- 172.21.15.46:0/2603385945 wait complete. 2024-05-09T03:21:31.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:21:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.658+0000 7fa8f7586640 1 Processor -- start 2024-05-09T03:21:31.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.658+0000 7fa8f7586640 1 -- start start 2024-05-09T03:21:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f7586640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b7ed0 0x7fa8f0163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f7586640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f01617d0 con 0x7fa8f01613c0 2024-05-09T03:21:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f7586640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f00d5280 con 0x7fa8f00b5210 2024-05-09T03:21:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f7586640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00b5620 con 0x7fa8f00b7ed0 2024-05-09T03:21:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f4b21640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8f01613c0 0x7fa8f00b77c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53554/0 (socket says 172.21.15.46:53554) 2024-05-09T03:21:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f4b21640 1 -- 172.21.15.46:0/4270673658 learned_addr learned my addr 172.21.15.46:0/4270673658 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:31.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f5b23640 1 --2- 172.21.15.46:0/4270673658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b7ed0 0x7fa8f0163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1898855130 0 0) 0x7fa8f01617d0 con 0x7fa8f01613c0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8d8003520 con 0x7fa8f01613c0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372039554 0 0) 0x7fa8f00d5280 con 0x7fa8f00b5210 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8f01617d0 con 0x7fa8f00b5210 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f5b23640 1 -- 172.21.15.46:0/4270673658 >> v1:172.21.15.88:6789/0 conn(0x7fa8f00b5210 legacy=0x7fa8f00d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f5b23640 1 -- 172.21.15.46:0/4270673658 >> v1:172.21.15.46:6789/0 conn(0x7fa8f01613c0 legacy=0x7fa8f00b77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.660+0000 7fa8f5b23640 1 -- 172.21.15.46:0/4270673658 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f00d5560 con 0x7fa8f00b7ed0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.661+0000 7fa8f5b23640 1 --2- 172.21.15.46:0/4270673658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b7ed0 0x7fa8f0163e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80097d0 tx=0x7fa8e802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a14656cf9ff7d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.661+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8037070 con 0x7fa8f00b7ed0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.661+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8032440 con 0x7fa8f00b7ed0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.661+0000 7fa8effff640 1 -- 172.21.15.46:0/4270673658 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8032810 con 0x7fa8f00b7ed0 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 -- 172.21.15.46:0/4270673658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b7ed0 msgr2=0x7fa8f0163e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 --2- 172.21.15.46:0/4270673658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b7ed0 0x7fa8f0163e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80097d0 tx=0x7fa8e802a210 comp rx=0 tx=0).stop 2024-05-09T03:21:31.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 -- 172.21.15.46:0/4270673658 shutdown_connections 2024-05-09T03:21:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 --2- 172.21.15.46:0/4270673658 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b7ed0 0x7fa8f0163e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 -- 172.21.15.46:0/4270673658 >> 172.21.15.46:0/4270673658 conn(0x7fa8f00d29a0 msgr2=0x7fa8f00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 -- 172.21.15.46:0/4270673658 shutdown_connections 2024-05-09T03:21:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.662+0000 7fa8f7586640 1 -- 172.21.15.46:0/4270673658 wait complete. 2024-05-09T03:21:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.664+0000 7fa8f7586640 1 Processor -- start 2024-05-09T03:21:31.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.664+0000 7fa8f7586640 1 -- start start 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f7586640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b5210 0x7fa8f01fa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f7586640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f01613c0 0x7fa8f01f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f7586640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 0x7fa8f0164200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f7586640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f0160ea0 con 0x7fa8f0163df0 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f7586640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01609a0 con 0x7fa8f01613c0 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f7586640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00d4f10 con 0x7fa8f00b5210 2024-05-09T03:21:31.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f5b23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 0x7fa8f0164200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f5b23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 0x7fa8f0164200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41918/0 (socket says 172.21.15.46:41918) 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.665+0000 7fa8f5b23640 1 -- 172.21.15.46:0/688289670 learned_addr learned my addr 172.21.15.46:0/688289670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5322640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b5210 0x7fa8f01fa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f4b21640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f01613c0 0x7fa8f01f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5b23640 1 -- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b5210 msgr2=0x7fa8f01fa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5b23640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b5210 0x7fa8f01fa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5b23640 1 -- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f01613c0 msgr2=0x7fa8f01f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5b23640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f01613c0 0x7fa8f01f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5b23640 1 -- 172.21.15.46:0/688289670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f0164740 con 0x7fa8f0163df0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f4b21640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f01613c0 0x7fa8f01f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f5b23640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 0x7fa8f0164200 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80097d0 tx=0x7fa8e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8041070 con 0x7fa8f0163df0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8004250 con 0x7fa8f0163df0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f00b86d0 con 0x7fa8f0163df0 2024-05-09T03:21:31.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8037040 con 0x7fa8f0163df0 2024-05-09T03:21:31.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.666+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f00b7910 con 0x7fa8f0163df0 2024-05-09T03:21:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.667+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f00b73a0 con 0x7fa8f0163df0 2024-05-09T03:21:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.668+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa8e8003610 con 0x7fa8f0163df0 2024-05-09T03:21:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.668+0000 7fa8edffb640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8dc0762d0 0x7fa8dc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.668+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fa8e80b7b90 con 0x7fa8f0163df0 2024-05-09T03:21:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.669+0000 7fa8f5322640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8dc0762d0 0x7fa8dc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.669+0000 7fa8f5322640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8dc0762d0 0x7fa8dc0787e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00014a0 tx=0x7fa8e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:31.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.673+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa8e804b020 con 0x7fa8f0163df0 2024-05-09T03:21:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.904+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8f006a270 con 0x7fa8f0163df0 2024-05-09T03:21:31.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.905+0000 7fa8edffb640 1 -- 172.21.15.46:0/688289670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 360 v360) ==== 95+0+5190 (secure 0 0 0) 0x7fa8f0164740 con 0x7fa8f0163df0 2024-05-09T03:21:31.905 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:31.906 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":360,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348}],"filesystems":[{"mdsmap":{"epoch":360,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T03:21:01.965784+0000","modified":"2024-05-09T03:21:29.873183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":136,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10812},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10812":{"gid":10812,"name":"c","rank":0,"incarnation":356,"state":"up:active","state_seq":6,"addr":"172.21.15.46:6835/2236478370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2236478370},{"type":"v1","addr":"172.21.15.46:6835","nonce":2236478370}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10812,"qdb_cluster":[10812]},"id":7}]} 2024-05-09T03:21:31.906 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 360 2024-05-09T03:21:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8dc0762d0 msgr2=0x7fa8dc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8dc0762d0 0x7fa8dc0787e0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00014a0 tx=0x7fa8e0009040 comp rx=0 tx=0).stop 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 msgr2=0x7fa8f0164200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 0x7fa8f0164200 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fa8e80097d0 tx=0x7fa8e80046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 shutdown_connections 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa8dc0762d0 0x7fa8dc0787e0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8f0163df0 0x7fa8f0164200 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8f01613c0 0x7fa8f01f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 --2- 172.21.15.46:0/688289670 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa8f00b5210 0x7fa8f01fa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 >> 172.21.15.46:0/688289670 conn(0x7fa8f00d29a0 msgr2=0x7fa8f015f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 shutdown_connections 2024-05-09T03:21:31.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:31.907+0000 7fa8f7586640 1 -- 172.21.15.46:0/688289670 wait complete. 2024-05-09T03:21:31.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:21:32.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.033+0000 7fb6c87e8640 1 Processor -- start 2024-05-09T03:21:32.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.033+0000 7fb6c87e8640 1 -- start start 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.034+0000 7fb6c87e8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6c0164c70 0x7fb6c01615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.034+0000 7fb6c87e8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 0x7fb6c00d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.034+0000 7fb6c87e8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6c0160b20 con 0x7fb6c015dfe0 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.034+0000 7fb6c87e8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb6c0161150 con 0x7fb6c0164c70 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.034+0000 7fb6c87e8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb6c00d15f0 con 0x7fb6c00d2480 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 0x7fb6c00d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6584640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6c0164c70 0x7fb6c01615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 0x7fb6c00d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:56554/0 (socket says 172.21.15.46:56554) 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 -- 172.21.15.46:0/2414698856 learned_addr learned my addr 172.21.15.46:0/2414698856 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:32.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 -- 172.21.15.46:0/2414698856 >> v2:172.21.15.88:3300/0 conn(0x7fb6c0164c70 msgr2=0x7fb6c01615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:32.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 --2- 172.21.15.46:0/2414698856 >> v2:172.21.15.88:3300/0 conn(0x7fb6c0164c70 0x7fb6c01615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 -- 172.21.15.46:0/2414698856 >> v1:172.21.15.88:6789/0 conn(0x7fb6c015dfe0 legacy=0x7fb6c00d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.035+0000 7fb6c6d85640 1 -- 172.21.15.46:0/2414698856 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c00d18d0 con 0x7fb6c00d2480 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.036+0000 7fb6c6d85640 1 --2- 172.21.15.46:0/2414698856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 0x7fb6c00d2890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009e50 tx=0x7fb6b802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6177b6516ee7d2a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.036+0000 7fb6c5582640 1 -- 172.21.15.46:0/2414698856 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8036070 con 0x7fb6c00d2480 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.036+0000 7fb6c5582640 1 -- 172.21.15.46:0/2414698856 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b8031600 con 0x7fb6c00d2480 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.036+0000 7fb6c5582640 1 -- 172.21.15.46:0/2414698856 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b80319d0 con 0x7fb6c00d2480 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 -- 172.21.15.46:0/2414698856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 msgr2=0x7fb6c00d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/2414698856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 0x7fb6c00d2890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009e50 tx=0x7fb6b802a710 comp rx=0 tx=0).stop 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 -- 172.21.15.46:0/2414698856 shutdown_connections 2024-05-09T03:21:32.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/2414698856 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c00d2480 0x7fb6c00d2890 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/2414698856 >> v2:172.21.15.88:3300/0 conn(0x7fb6c0164c70 0x7fb6c01615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 -- 172.21.15.46:0/2414698856 >> 172.21.15.46:0/2414698856 conn(0x7fb6c0073db0 msgr2=0x7fb6c00741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 -- 172.21.15.46:0/2414698856 shutdown_connections 2024-05-09T03:21:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.037+0000 7fb6c87e8640 1 -- 172.21.15.46:0/2414698856 wait complete. 2024-05-09T03:21:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.039+0000 7fb6c87e8640 1 Processor -- start 2024-05-09T03:21:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.039+0000 7fb6c87e8640 1 -- start start 2024-05-09T03:21:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c87e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6c00d2480 0x7fb6c0151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c87e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 0x7fb6c0159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c87e8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c0164c70 0x7fb6c00cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c87e8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c00d2dd0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c87e8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb6c015e3f0 con 0x7fb6c00d2480 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c87e8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fb6c00d45c0 con 0x7fb6c0164c70 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c5d83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 0x7fb6c0159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c5d83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 0x7fb6c0159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41934/0 (socket says 172.21.15.46:41934) 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c5d83640 1 -- 172.21.15.46:0/3774800191 learned_addr learned my addr 172.21.15.46:0/3774800191 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c6d85640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c0164c70 0x7fb6c00cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.040+0000 7fb6c6584640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6c00d2480 0x7fb6c0151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c5d83640 1 -- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c0164c70 msgr2=0x7fb6c00cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c5d83640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c0164c70 0x7fb6c00cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c5d83640 1 -- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6c00d2480 msgr2=0x7fb6c0151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c5d83640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6c00d2480 0x7fb6c0151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c5d83640 1 -- 172.21.15.46:0/3774800191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6c01572a0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c5d83640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 0x7fb6c0159230 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000c3d0 tx=0x7fb6b000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0017070 con 0x7fb6c015dfe0 2024-05-09T03:21:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b00052d0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6c01577d0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0003830 con 0x7fb6c015dfe0 2024-05-09T03:21:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.041+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6c0163010 con 0x7fb6c015dfe0 2024-05-09T03:21:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.042+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6c0157540 con 0x7fb6c015dfe0 2024-05-09T03:21:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.043+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb6b001a070 con 0x7fb6c015dfe0 2024-05-09T03:21:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.043+0000 7fb6b6ffd640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6a40762d0 0x7fb6a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.043+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7fb6b00968f0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.044+0000 7fb6c6584640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6a40762d0 0x7fb6a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.044+0000 7fb6c6584640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6a40762d0 0x7fb6a40787e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009aa0 tx=0x7fb6bc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.048+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb6b005fba0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.239+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6c008cfd0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.240+0000 7fb6b6ffd640 1 -- 172.21.15.46:0/3774800191 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) ==== 74+0+16676 (secure 0 0 0) 0x7fb6c008cfd0 con 0x7fb6c015dfe0 2024-05-09T03:21:32.241 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:32.241 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":136,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:21:18.861195+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-05-09T03:20:59.663057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"134","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-05-09T03:21:00.927261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/2230300620":"2024-05-10T03:21:18.861166+0000","172.21.15.46:6833/2230300620":"2024-05-10T03:21:18.861166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6a40762d0 msgr2=0x7fb6a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6a40762d0 0x7fb6a40787e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc009aa0 tx=0x7fb6bc009190 comp rx=0 tx=0).stop 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 msgr2=0x7fb6c0159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 0x7fb6c0159230 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000c3d0 tx=0x7fb6b000c890 comp rx=0 tx=0).stop 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 shutdown_connections 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fb6c0164c70 0x7fb6c00cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fb6a40762d0 0x7fb6a40787e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c015dfe0 0x7fb6c0159230 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 --2- 172.21.15.46:0/3774800191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6c00d2480 0x7fb6c0151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 >> 172.21.15.46:0/3774800191 conn(0x7fb6c0073db0 msgr2=0x7fb6c00b9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:32.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 shutdown_connections 2024-05-09T03:21:32.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.242+0000 7fb6c87e8640 1 -- 172.21.15.46:0/3774800191 wait complete. 2024-05-09T03:21:32.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T03:21:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.369+0000 7f6611e2b640 1 Processor -- start 2024-05-09T03:21:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.369+0000 7f6611e2b640 1 -- start start 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f6611e2b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f660c0d1740 0x7f660c0b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f6611e2b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f660c0d1110 0x7f660c0d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f6611e2b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660c0b9690 con 0x7f660c0cfcf0 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f6611e2b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f660c08cd60 con 0x7f660c0d1740 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f6611e2b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f660c0ca9e0 con 0x7f660c0d1110 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f660c0d1740 0x7f660c0b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f660c0cfcf0 0x7f660c09cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53558/0 (socket says 172.21.15.46:53558) 2024-05-09T03:21:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660bfff640 1 -- 172.21.15.46:0/3796116272 learned_addr learned my addr 172.21.15.46:0/3796116272 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660affd640 1 --2- 172.21.15.46:0/3796116272 >> v2:172.21.15.88:3300/0 conn(0x7f660c0d1110 0x7f660c0d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660b7fe640 1 -- 172.21.15.46:0/3796116272 >> v2:172.21.15.88:3300/0 conn(0x7f660c0d1110 msgr2=0x7f660c0d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660b7fe640 1 --2- 172.21.15.46:0/3796116272 >> v2:172.21.15.88:3300/0 conn(0x7f660c0d1110 0x7f660c0d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660b7fe640 1 -- 172.21.15.46:0/3796116272 >> v1:172.21.15.46:6789/0 conn(0x7f660c0cfcf0 legacy=0x7f660c09cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660a7fc640 1 -- 172.21.15.46:0/3796116272 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487770152 0 0) 0x7f660c0b9690 con 0x7f660c0cfcf0 2024-05-09T03:21:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.371+0000 7f660b7fe640 1 -- 172.21.15.46:0/3796116272 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c08cfb0 con 0x7f660c0d1740 2024-05-09T03:21:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.372+0000 7f660b7fe640 1 --2- 172.21.15.46:0/3796116272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1740 0x7f660c0b73d0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f660c0d1b80 tx=0x7f65fc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b84f82edf9bb05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.372+0000 7f660a7fc640 1 -- 172.21.15.46:0/3796116272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc037070 con 0x7f660c0d1740 2024-05-09T03:21:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.372+0000 7f660a7fc640 1 -- 172.21.15.46:0/3796116272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65fc032440 con 0x7f660c0d1740 2024-05-09T03:21:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.372+0000 7f660a7fc640 1 -- 172.21.15.46:0/3796116272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc032810 con 0x7f660c0d1740 2024-05-09T03:21:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 -- 172.21.15.46:0/3796116272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1740 msgr2=0x7f660c0b73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3796116272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1740 0x7f660c0b73d0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f660c0d1b80 tx=0x7f65fc02a1a0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 -- 172.21.15.46:0/3796116272 shutdown_connections 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3796116272 >> v2:172.21.15.88:3300/0 conn(0x7f660c0d1110 0x7f660c0d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3796116272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1740 0x7f660c0b73d0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 -- 172.21.15.46:0/3796116272 >> 172.21.15.46:0/3796116272 conn(0x7f660c0d05f0 msgr2=0x7f660c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 -- 172.21.15.46:0/3796116272 shutdown_connections 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.373+0000 7f6611e2b640 1 -- 172.21.15.46:0/3796116272 wait complete. 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.375+0000 7f6611e2b640 1 Processor -- start 2024-05-09T03:21:32.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.375+0000 7f6611e2b640 1 -- start start 2024-05-09T03:21:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f6611e2b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f660c0cfcf0 0x7f660c156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f6611e2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 0x7f660c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f6611e2b640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f660c0d1740 0x7f660c06f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f6611e2b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f660c09d3b0 con 0x7f660c0d1110 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f6611e2b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0b93b0 con 0x7f660c0cfcf0 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f6611e2b640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0d2930 con 0x7f660c0d1740 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f660affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 0x7f660c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.376+0000 7f660affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 0x7f660c06ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41948/0 (socket says 172.21.15.46:41948) 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 -- 172.21.15.46:0/3135945806 learned_addr learned my addr 172.21.15.46:0/3135945806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660bfff640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f660c0d1740 0x7f660c06f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660b7fe640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f660c0cfcf0 0x7f660c156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 -- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f660c0d1740 msgr2=0x7f660c06f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f660c0d1740 0x7f660c06f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 -- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f660c0cfcf0 msgr2=0x7f660c156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f660c0cfcf0 0x7f660c156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 -- 172.21.15.46:0/3135945806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c156e70 con 0x7f660c0d1110 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f660affd640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 0x7f660c06ec00 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f660000d5d0 tx=0x7f660000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6600004240 con 0x7f660c0d1110 2024-05-09T03:21:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66000043e0 con 0x7f660c0d1110 2024-05-09T03:21:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c1f67d0 con 0x7f660c0d1110 2024-05-09T03:21:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6600003830 con 0x7f660c0d1110 2024-05-09T03:21:32.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.377+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c1f6d90 con 0x7f660c0d1110 2024-05-09T03:21:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.378+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f660c0b65b0 con 0x7f660c0d1110 2024-05-09T03:21:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.379+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f66000039d0 con 0x7f660c0d1110 2024-05-09T03:21:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.379+0000 7f65f3fff640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65f8076300 0x7f65f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.380+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5795+0+0 (secure 0 0 0) 0x7f66000977a0 con 0x7f660c0d1110 2024-05-09T03:21:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.380+0000 7f660b7fe640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65f8076300 0x7f65f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.380+0000 7f660b7fe640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65f8076300 0x7f65f8078810 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f65fc034000 tx=0x7f65fc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.384+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f660c0b65b0 con 0x7f660c0d1110 2024-05-09T03:21:32.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.617+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f660c06a270 con 0x7f660c0d1110 2024-05-09T03:21:32.618 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-05-09T03:21:32.618+0000 7fafbfcbe640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T03:21:32.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.619+0000 7f65f3fff640 1 -- 172.21.15.46:0/3135945806 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v361) ==== 161+0+0 (secure 0 0 0) 0x7f66000609c0 con 0x7f660c0d1110 2024-05-09T03:21:32.620 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T03:21:32.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.622+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65f8076300 msgr2=0x7f65f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.622+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65f8076300 0x7f65f8078810 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f65fc034000 tx=0x7f65fc02e000 comp rx=0 tx=0).stop 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 msgr2=0x7f660c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 0x7f660c06ec00 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f660000d5d0 tx=0x7f660000da90 comp rx=0 tx=0).stop 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 shutdown_connections 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f660c0d1740 0x7f660c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f65f8076300 0x7f65f8078810 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f660c0d1110 0x7f660c06ec00 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 --2- 172.21.15.46:0/3135945806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f660c0cfcf0 0x7f660c156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 >> 172.21.15.46:0/3135945806 conn(0x7f660c0d05f0 msgr2=0x7f660c072200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 shutdown_connections 2024-05-09T03:21:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.623+0000 7f6611e2b640 1 -- 172.21.15.46:0/3135945806 wait complete. 2024-05-09T03:21:32.624 INFO:tasks.ceph.mds.c.smithi046.stderr: -994> 2024-05-09T03:21:24.874+0000 7f7bda2f6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T03:21:32.625 INFO:tasks.ceph.mds.c.smithi046.stderr: -658> 2024-05-09T03:21:26.311+0000 7f7bd62ee640 -1 mds.0.cache corrupt backtrace on ino x00x10000000001: void inode_backtrace_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 5 < 104: Malformed input [buffer:3] 2024-05-09T03:21:32.625 INFO:tasks.ceph.mds.c.smithi046.stderr: -443> 2024-05-09T03:21:27.538+0000 7f7bd62ee640 -1 log_channel(cluster) log [ERR] : dir 0x100000001f6 object missing on disk; some files may be lost (/dir2) 2024-05-09T03:21:32.636 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T03:21:32.682 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-05-09T03:21:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.757+0000 7fa38f0ee640 1 Processor -- start 2024-05-09T03:21:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.757+0000 7fa38f0ee640 1 -- start start 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38f0ee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3880d21a0 0x7fa3880d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38f0ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa3880d2af0 0x7fa3881647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38f0ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3880b90f0 con 0x7fa388164ce0 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38f0ee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa3880d0360 con 0x7fa3880d21a0 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38f0ee640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa3880b93a0 con 0x7fa3880d2af0 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3880d21a0 0x7fa3880d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa3880d21a0 0x7fa3880d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41970/0 (socket says 172.21.15.46:41970) 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 -- 172.21.15.46:0/2024914847 learned_addr learned my addr 172.21.15.46:0/2024914847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 -- 172.21.15.46:0/2024914847 >> v2:172.21.15.88:3300/0 conn(0x7fa3880d2af0 msgr2=0x7fa3881647a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:32.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa387fff640 1 --2- 172.21.15.46:0/2024914847 >> v2:172.21.15.88:3300/0 conn(0x7fa3880d2af0 0x7fa3881647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 --2- 172.21.15.46:0/2024914847 >> v2:172.21.15.88:3300/0 conn(0x7fa3880d2af0 0x7fa3881647a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 -- 172.21.15.46:0/2024914847 >> v1:172.21.15.88:6789/0 conn(0x7fa388164ce0 legacy=0x7fa38806e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.758+0000 7fa38ce8a640 1 -- 172.21.15.46:0/2024914847 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3880d10b0 con 0x7fa3880d21a0 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.759+0000 7fa38ce8a640 1 --2- 172.21.15.46:0/2024914847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d21a0 0x7fa3880d25b0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fa388070110 tx=0x7fa37802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fa853b6098b26b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.759+0000 7fa3877fe640 1 -- 172.21.15.46:0/2024914847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378004030 con 0x7fa3880d21a0 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.759+0000 7fa3877fe640 1 -- 172.21.15.46:0/2024914847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3780041d0 con 0x7fa3880d21a0 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.759+0000 7fa3877fe640 1 -- 172.21.15.46:0/2024914847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa378003610 con 0x7fa3880d21a0 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 -- 172.21.15.46:0/2024914847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d21a0 msgr2=0x7fa3880d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/2024914847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d21a0 0x7fa3880d25b0 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7fa388070110 tx=0x7fa37802a130 comp rx=0 tx=0).stop 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 -- 172.21.15.46:0/2024914847 shutdown_connections 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/2024914847 >> v2:172.21.15.88:3300/0 conn(0x7fa3880d2af0 0x7fa3881647a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/2024914847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d21a0 0x7fa3880d25b0 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 -- 172.21.15.46:0/2024914847 >> 172.21.15.46:0/2024914847 conn(0x7fa3880d04f0 msgr2=0x7fa3880d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:32.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 -- 172.21.15.46:0/2024914847 shutdown_connections 2024-05-09T03:21:32.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.760+0000 7fa38f0ee640 1 -- 172.21.15.46:0/2024914847 wait complete. 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.762+0000 7fa38f0ee640 1 Processor -- start 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.762+0000 7fa38f0ee640 1 -- start start 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.763+0000 7fa38f0ee640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3880d21a0 0x7fa3881596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.763+0000 7fa38f0ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 0x7fa3880b68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.763+0000 7fa38f0ee640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa388164ce0 0x7fa38815f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.763+0000 7fa38f0ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3880d1bf0 con 0x7fa3880d2af0 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.763+0000 7fa38f0ee640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa3880d1a70 con 0x7fa3880d21a0 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.763+0000 7fa38f0ee640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa3880cf8c0 con 0x7fa388164ce0 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 0x7fa3880b68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 0x7fa3880b68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41986/0 (socket says 172.21.15.46:41986) 2024-05-09T03:21:32.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 -- 172.21.15.46:0/3450823272 learned_addr learned my addr 172.21.15.46:0/3450823272 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa38d68b640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa388164ce0 0x7fa38815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa38ce8a640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3880d21a0 0x7fa3881596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 -- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa388164ce0 msgr2=0x7fa38815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa388164ce0 0x7fa38815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 -- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3880d21a0 msgr2=0x7fa3881596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3880d21a0 0x7fa3881596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 -- 172.21.15.46:0/3450823272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3880b7890 con 0x7fa3880d2af0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa38d68b640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa388164ce0 0x7fa38815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa38ce8a640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3880d21a0 0x7fa3881596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa387fff640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 0x7fa3880b68c0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00c610 tx=0x7fa37c00cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c015070 con 0x7fa3880d2af0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa37c004220 con 0x7fa3880d2af0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3880b75b0 con 0x7fa3880d2af0 2024-05-09T03:21:32.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c003610 con 0x7fa3880d2af0 2024-05-09T03:21:32.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.764+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3881ea1f0 con 0x7fa3880d2af0 2024-05-09T03:21:32.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.765+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3880b7320 con 0x7fa3880d2af0 2024-05-09T03:21:32.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.766+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa37c0037b0 con 0x7fa3880d2af0 2024-05-09T03:21:32.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.767+0000 7fa3857fa640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3740762d0 0x7fa3740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:32.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.767+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5881+0+0 (secure 0 0 0) 0x7fa37c097510 con 0x7fa3880d2af0 2024-05-09T03:21:32.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.767+0000 7fa38ce8a640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3740762d0 0x7fa3740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:32.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.767+0000 7fa38ce8a640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3740762d0 0x7fa3740787e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa378033000 tx=0x7fa37802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:32.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:32.771+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa37c060d30 con 0x7fa3880d2af0 2024-05-09T03:21:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.003+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa388159c10 con 0x7fa3880d2af0 2024-05-09T03:21:33.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.624+0000 7fa3857fa640 1 -- 172.21.15.46:0/3450823272 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v362) ==== 104+0+0 (secure 0 0 0) 0x7fa388159c10 con 0x7fa3880d2af0 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3740762d0 msgr2=0x7fa3740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3740762d0 0x7fa3740787e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa378033000 tx=0x7fa37802e000 comp rx=0 tx=0).stop 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 msgr2=0x7fa3880b68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 0x7fa3880b68c0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7fa37c00c610 tx=0x7fa37c00cad0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 shutdown_connections 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa388164ce0 0x7fa38815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa3740762d0 0x7fa3740787e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3880d2af0 0x7fa3880b68c0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 --2- 172.21.15.46:0/3450823272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3880d21a0 0x7fa3881596d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 >> 172.21.15.46:0/3450823272 conn(0x7fa3880d04f0 msgr2=0x7fa388164450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:33.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 shutdown_connections 2024-05-09T03:21:33.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.627+0000 7fa38f0ee640 1 -- 172.21.15.46:0/3450823272 wait complete. 2024-05-09T03:21:33.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T03:21:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.768+0000 7f71b770d640 1 Processor -- start 2024-05-09T03:21:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.768+0000 7f71b770d640 1 -- start start 2024-05-09T03:21:33.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b770d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71b00b79b0 0x7f71b00d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:33.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b770d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b00b9400 con 0x7f71b00d1900 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b770d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71b009cdd0 con 0x7f71b0162f00 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b770d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f71b00b3010 con 0x7f71b00b79b0 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b4ca8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71b00b79b0 0x7f71b00d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b4ca8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71b00b79b0 0x7f71b00d1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42002/0 (socket says 172.21.15.46:42002) 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b54a9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f71b00d1900 0x7f71b01627f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53586/0 (socket says 172.21.15.46:53586) 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b4ca8640 1 -- 172.21.15.46:0/2286765640 learned_addr learned my addr 172.21.15.46:0/2286765640 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:33.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b4ca8640 1 -- 172.21.15.46:0/2286765640 >> v1:172.21.15.88:6789/0 conn(0x7f71b0162f00 legacy=0x7f71b00ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b4ca8640 1 -- 172.21.15.46:0/2286765640 >> v1:172.21.15.46:6789/0 conn(0x7f71b00d1900 legacy=0x7f71b01627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.770+0000 7f71b4ca8640 1 -- 172.21.15.46:0/2286765640 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b00b2c50 con 0x7f71b00b79b0 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.771+0000 7f71b4ca8640 1 --2- 172.21.15.46:0/2286765640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b00d1d10 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f71a0009870 tx=0x7f71a002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccec9e525172c0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.771+0000 7f71a7fff640 1 -- 172.21.15.46:0/2286765640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0037070 con 0x7f71b00b79b0 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.771+0000 7f71a7fff640 1 -- 172.21.15.46:0/2286765640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a0032440 con 0x7f71b00b79b0 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.771+0000 7f71a7fff640 1 -- 172.21.15.46:0/2286765640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0032810 con 0x7f71b00b79b0 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 -- 172.21.15.46:0/2286765640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 msgr2=0x7f71b00d1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:33.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 --2- 172.21.15.46:0/2286765640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b00d1d10 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f71a0009870 tx=0x7f71a002a1c0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 -- 172.21.15.46:0/2286765640 shutdown_connections 2024-05-09T03:21:33.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 --2- 172.21.15.46:0/2286765640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b00d1d10 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 -- 172.21.15.46:0/2286765640 >> 172.21.15.46:0/2286765640 conn(0x7f71b00d05d0 msgr2=0x7f71b00d09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:33.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 -- 172.21.15.46:0/2286765640 shutdown_connections 2024-05-09T03:21:33.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.772+0000 7f71b770d640 1 -- 172.21.15.46:0/2286765640 wait complete. 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.774+0000 7f71b770d640 1 Processor -- start 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.774+0000 7f71b770d640 1 -- start start 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b770d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b01516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b770d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00d1900 0x7f71b00b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b770d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b0162f00 0x7f71b00b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b770d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f71b0074bc0 con 0x7f71b00b79b0 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b770d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71b009d140 con 0x7f71b0162f00 2024-05-09T03:21:33.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b770d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f71b00b96e0 con 0x7f71b00d1900 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b54a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b01516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b54a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b01516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42004/0 (socket says 172.21.15.46:42004) 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.775+0000 7f71b54a9640 1 -- 172.21.15.46:0/4096312413 learned_addr learned my addr 172.21.15.46:0/4096312413 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b54a9640 1 -- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00d1900 msgr2=0x7f71b00b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b54a9640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00d1900 0x7f71b00b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b54a9640 1 -- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b0162f00 msgr2=0x7f71b00b22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b54a9640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b0162f00 0x7f71b00b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b54a9640 1 -- 172.21.15.46:0/4096312413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b0151c20 con 0x7f71b00b79b0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b54a9640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b01516e0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c7a0 tx=0x7f71ac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac017070 con 0x7f71b00b79b0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71ac00a640 con 0x7f71b00b79b0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b014ef30 con 0x7f71b00b79b0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac00a9f0 con 0x7f71b00b79b0 2024-05-09T03:21:33.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.776+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b014f4f0 con 0x7f71b00b79b0 2024-05-09T03:21:33.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.777+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71b006a270 con 0x7f71b00b79b0 2024-05-09T03:21:33.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.778+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71ac01a070 con 0x7f71b00b79b0 2024-05-09T03:21:33.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.778+0000 7f71a5ffb640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71940762d0 0x7f71940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:33.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.778+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5881+0+0 (secure 0 0 0) 0x7f71ac096bb0 con 0x7f71b00b79b0 2024-05-09T03:21:33.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.778+0000 7f71b4ca8640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71940762d0 0x7f71940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:33.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.779+0000 7f71b4ca8640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71940762d0 0x7f71940787e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f71a0034000 tx=0x7f71a002c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:33.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.783+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f71ac05fe10 con 0x7f71b00b79b0 2024-05-09T03:21:33.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:33.974+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f71b00718a0 con 0x7f71b00b79b0 2024-05-09T03:21:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.625+0000 7f71a5ffb640 1 -- 172.21.15.46:0/4096312413 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v138) ==== 181+0+0 (secure 0 0 0) 0x7f71ac064110 con 0x7f71b00b79b0 2024-05-09T03:21:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71940762d0 msgr2=0x7f71940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71940762d0 0x7f71940787e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f71a0034000 tx=0x7f71a002c000 comp rx=0 tx=0).stop 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 msgr2=0x7f71b01516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b01516e0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00c7a0 tx=0x7f71ac00cc60 comp rx=0 tx=0).stop 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 shutdown_connections 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71b0162f00 0x7f71b00b22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f71b00d1900 0x7f71b00b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f71940762d0 0x7f71940787e0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 --2- 172.21.15.46:0/4096312413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71b00b79b0 0x7f71b01516e0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 >> 172.21.15.46:0/4096312413 conn(0x7f71b00d05d0 msgr2=0x7f71b00b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.629+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 shutdown_connections 2024-05-09T03:21:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.630+0000 7f71b770d640 1 -- 172.21.15.46:0/4096312413 wait complete. 2024-05-09T03:21:34.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T03:21:34.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.770+0000 7fe43efea640 1 Processor -- start 2024-05-09T03:21:34.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.770+0000 7fe43efea640 1 -- start start 2024-05-09T03:21:34.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43efea640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe43813fe60 0x7fe438161400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43efea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4380703a0 0x7fe4380707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43efea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe438073290 0x7fe4380736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43efea640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe438070cf0 con 0x7fe4380703a0 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43efea640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe43815d790 con 0x7fe43813fe60 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43efea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe43815da70 con 0x7fe438073290 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4380703a0 0x7fe4380707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4380703a0 0x7fe4380707b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42014/0 (socket says 172.21.15.46:42014) 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 -- 172.21.15.46:0/225610535 learned_addr learned my addr 172.21.15.46:0/225610535 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43d587640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe438073290 0x7fe4380736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43cd86640 1 --2- 172.21.15.46:0/225610535 >> v2:172.21.15.88:3300/0 conn(0x7fe43813fe60 0x7fe438161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 -- 172.21.15.46:0/225610535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe438073290 msgr2=0x7fe4380736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe438073290 0x7fe4380736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 -- 172.21.15.46:0/225610535 >> v2:172.21.15.88:3300/0 conn(0x7fe43813fe60 msgr2=0x7fe438161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 --2- 172.21.15.46:0/225610535 >> v2:172.21.15.88:3300/0 conn(0x7fe43813fe60 0x7fe438161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe437fff640 1 -- 172.21.15.46:0/225610535 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43815dd50 con 0x7fe4380703a0 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43cd86640 1 --2- 172.21.15.46:0/225610535 >> v2:172.21.15.88:3300/0 conn(0x7fe43813fe60 0x7fe438161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.772+0000 7fe43d587640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe438073290 0x7fe4380736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.773+0000 7fe437fff640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380703a0 0x7fe4380707b0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fe438072a60 tx=0x7fe42002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ccf2f52f91cded9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.773+0000 7fe4377fe640 1 -- 172.21.15.46:0/225610535 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420036070 con 0x7fe4380703a0 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.773+0000 7fe4377fe640 1 -- 172.21.15.46:0/225610535 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe420031600 con 0x7fe4380703a0 2024-05-09T03:21:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.773+0000 7fe4377fe640 1 -- 172.21.15.46:0/225610535 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4200319d0 con 0x7fe4380703a0 2024-05-09T03:21:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 -- 172.21.15.46:0/225610535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380703a0 msgr2=0x7fe4380707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380703a0 0x7fe4380707b0 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fe438072a60 tx=0x7fe42002a6a0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 -- 172.21.15.46:0/225610535 shutdown_connections 2024-05-09T03:21:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe438073290 0x7fe4380736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 --2- 172.21.15.46:0/225610535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4380703a0 0x7fe4380707b0 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 --2- 172.21.15.46:0/225610535 >> v2:172.21.15.88:3300/0 conn(0x7fe43813fe60 0x7fe438161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 -- 172.21.15.46:0/225610535 >> 172.21.15.46:0/225610535 conn(0x7fe43806d0a0 msgr2=0x7fe4380b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 -- 172.21.15.46:0/225610535 shutdown_connections 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.774+0000 7fe43efea640 1 -- 172.21.15.46:0/225610535 wait complete. 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.776+0000 7fe43efea640 1 Processor -- start 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.776+0000 7fe43efea640 1 -- start start 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe43efea640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4380703a0 0x7fe43814cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe43efea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 0x7fe4381f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe43efea640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe43813fe60 0x7fe43814dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe43efea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe438160d20 con 0x7fe438073290 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe43efea640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe438140270 con 0x7fe4380703a0 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe43efea640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fe438160ea0 con 0x7fe43813fe60 2024-05-09T03:21:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe437fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 0x7fe4381f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe437fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 0x7fe4381f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42020/0 (socket says 172.21.15.46:42020) 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.777+0000 7fe437fff640 1 -- 172.21.15.46:0/3001102044 learned_addr learned my addr 172.21.15.46:0/3001102044 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe43d587640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe43813fe60 0x7fe43814dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe43cd86640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4380703a0 0x7fe43814cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe437fff640 1 -- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe43813fe60 msgr2=0x7fe43814dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe437fff640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe43813fe60 0x7fe43814dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe437fff640 1 -- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4380703a0 msgr2=0x7fe43814cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe437fff640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4380703a0 0x7fe43814cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe437fff640 1 -- 172.21.15.46:0/3001102044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe43814d210 con 0x7fe438073290 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe437fff640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 0x7fe4381f4470 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7fe420034040 tx=0x7fe420004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420036050 con 0x7fe438073290 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe420042070 con 0x7fe438073290 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420009510 con 0x7fe438073290 2024-05-09T03:21:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe43814d4f0 con 0x7fe438073290 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.778+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43814dab0 con 0x7fe438073290 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.779+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe43806a270 con 0x7fe438073290 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.780+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe420007c30 con 0x7fe438073290 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.781+0000 7fe4357fa640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe418076300 0x7fe418078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.781+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5443+0+0 (secure 0 0 0) 0x7fe4200b8430 con 0x7fe438073290 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.781+0000 7fe43cd86640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe418076300 0x7fe418078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.781+0000 7fe43cd86640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe418076300 0x7fe418078810 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe438073240 tx=0x7fe428008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:34.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.785+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe420081710 con 0x7fe438073290 2024-05-09T03:21:34.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:34.977+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe4380b4a80 con 0x7fe438073290 2024-05-09T03:21:35.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.631+0000 7fe4357fa640 1 -- 172.21.15.46:0/3001102044 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v139) ==== 169+0+0 (secure 0 0 0) 0x7fe420085a10 con 0x7fe438073290 2024-05-09T03:21:35.633 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-05-09T03:21:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe418076300 msgr2=0x7fe418078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe418076300 0x7fe418078810 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe438073240 tx=0x7fe428008040 comp rx=0 tx=0).stop 2024-05-09T03:21:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 msgr2=0x7fe4381f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 0x7fe4381f4470 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7fe420034040 tx=0x7fe420004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 shutdown_connections 2024-05-09T03:21:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fe43813fe60 0x7fe43814dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fe418076300 0x7fe418078810 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe438073290 0x7fe4381f4470 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 --2- 172.21.15.46:0/3001102044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4380703a0 0x7fe43814cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.635+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 >> 172.21.15.46:0/3001102044 conn(0x7fe43806d0a0 msgr2=0x7fe43814bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.636+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 shutdown_connections 2024-05-09T03:21:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.636+0000 7fe43efea640 1 -- 172.21.15.46:0/3001102044 wait complete. 2024-05-09T03:21:35.651 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-05-09T03:21:35.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.772+0000 7f3151061640 1 Processor -- start 2024-05-09T03:21:35.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.772+0000 7f3151061640 1 -- start start 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f3151061640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f314c0d2770 0x7f314c16f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f3151061640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314c1652c0 con 0x7f314c06c680 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f3151061640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314c1655a0 con 0x7f314c06ca90 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f3151061640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f314c165880 con 0x7f314c0d2770 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f314b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f314c0d2770 0x7f314c16f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f314ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f314c06c680 0x7f314c164750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50506/0 (socket says 172.21.15.46:50506) 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f314b59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f314c0d2770 0x7f314c16f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54376/0 (socket says 172.21.15.46:54376) 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.775+0000 7f314ad9d640 1 -- 172.21.15.46:0/4211841766 learned_addr learned my addr 172.21.15.46:0/4211841766 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.776+0000 7f3149d9b640 1 -- 172.21.15.46:0/4211841766 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3188418449 0 0) 0x7f314c1652c0 con 0x7f314c06c680 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.776+0000 7f3149d9b640 1 -- 172.21.15.46:0/4211841766 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3134003520 con 0x7f314c06c680 2024-05-09T03:21:35.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.776+0000 7f314b59e640 1 -- 172.21.15.46:0/4211841766 >> v1:172.21.15.88:6789/0 conn(0x7f314c06ca90 legacy=0x7f314c0d2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:35.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.777+0000 7f314b59e640 1 -- 172.21.15.46:0/4211841766 >> v1:172.21.15.46:6789/0 conn(0x7f314c06c680 legacy=0x7f314c164750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.777+0000 7f314b59e640 1 -- 172.21.15.46:0/4211841766 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c165b60 con 0x7f314c0d2770 2024-05-09T03:21:35.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.778+0000 7f314b59e640 1 --2- 172.21.15.46:0/4211841766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c16f780 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f3144009d90 tx=0x7f314402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d230eca74b418703 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:35.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.779+0000 7f3149d9b640 1 -- 172.21.15.46:0/4211841766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144007740 con 0x7f314c0d2770 2024-05-09T03:21:35.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.779+0000 7f3149d9b640 1 -- 172.21.15.46:0/4211841766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31440078e0 con 0x7f314c0d2770 2024-05-09T03:21:35.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.779+0000 7f3149d9b640 1 -- 172.21.15.46:0/4211841766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144007c90 con 0x7f314c0d2770 2024-05-09T03:21:35.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.780+0000 7f3151061640 1 -- 172.21.15.46:0/4211841766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 msgr2=0x7f314c16f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.780+0000 7f3151061640 1 --2- 172.21.15.46:0/4211841766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c16f780 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f3144009d90 tx=0x7f314402a660 comp rx=0 tx=0).stop 2024-05-09T03:21:35.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.780+0000 7f3151061640 1 -- 172.21.15.46:0/4211841766 shutdown_connections 2024-05-09T03:21:35.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.780+0000 7f3151061640 1 --2- 172.21.15.46:0/4211841766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c16f780 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.780+0000 7f3151061640 1 -- 172.21.15.46:0/4211841766 >> 172.21.15.46:0/4211841766 conn(0x7f314c073d10 msgr2=0x7f314c09b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:35.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.781+0000 7f3151061640 1 -- 172.21.15.46:0/4211841766 shutdown_connections 2024-05-09T03:21:35.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.781+0000 7f3151061640 1 -- 172.21.15.46:0/4211841766 wait complete. 2024-05-09T03:21:35.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.783+0000 7f3151061640 1 Processor -- start 2024-05-09T03:21:35.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.784+0000 7f3151061640 1 -- start start 2024-05-09T03:21:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f3151061640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f314c06c680 0x7f314c0d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f3151061640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f314c06ca90 0x7f314c0cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f3151061640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c14f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f3151061640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f314c164fe0 con 0x7f314c0d2770 2024-05-09T03:21:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f3151061640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f314c164e60 con 0x7f314c06ca90 2024-05-09T03:21:35.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f3151061640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f314c167370 con 0x7f314c06c680 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c14f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c14f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54392/0 (socket says 172.21.15.46:54392) 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 -- 172.21.15.46:0/738069710 learned_addr learned my addr 172.21.15.46:0/738069710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314a59c640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f314c06ca90 0x7f314c0cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314ad9d640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f314c06c680 0x7f314c0d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 -- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f314c06c680 msgr2=0x7f314c0d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f314c06c680 0x7f314c0d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 -- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f314c06ca90 msgr2=0x7f314c0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f314c06ca90 0x7f314c0cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314b59e640 1 -- 172.21.15.46:0/738069710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c0d1c00 con 0x7f314c0d2770 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.785+0000 7f314ad9d640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f314c06c680 0x7f314c0d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.786+0000 7f314b59e640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c14f270 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f3144009eb0 tx=0x7f3144004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.786+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144040070 con 0x7f314c0d2770 2024-05-09T03:21:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.786+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3144031c40 con 0x7f314c0d2770 2024-05-09T03:21:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.786+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c0d1ee0 con 0x7f314c0d2770 2024-05-09T03:21:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.786+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144009450 con 0x7f314c0d2770 2024-05-09T03:21:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.786+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c1fb380 con 0x7f314c0d2770 2024-05-09T03:21:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.787+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314c06e3e0 con 0x7f314c0d2770 2024-05-09T03:21:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.788+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3144039040 con 0x7f314c0d2770 2024-05-09T03:21:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.788+0000 7f31337fe640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31380762d0 0x7f31380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.788+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f31440b8090 con 0x7f314c0d2770 2024-05-09T03:21:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.788+0000 7f314ad9d640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31380762d0 0x7f31380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.789+0000 7f314ad9d640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31380762d0 0x7f31380787e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3140007fd0 tx=0x7f3140009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.793+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f314c06e3e0 con 0x7f314c0d2770 2024-05-09T03:21:35.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.984+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f314c06a270 con 0x7f314c0d2770 2024-05-09T03:21:35.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.988+0000 7f31337fe640 1 -- 172.21.15.46:0/738069710 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7f3144081640 con 0x7f314c0d2770 2024-05-09T03:21:35.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31380762d0 msgr2=0x7f31380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31380762d0 0x7f31380787e0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f3140007fd0 tx=0x7f3140009040 comp rx=0 tx=0).stop 2024-05-09T03:21:35.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 msgr2=0x7f314c14f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:35.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c14f270 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f3144009eb0 tx=0x7f3144004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 shutdown_connections 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f31380762d0 0x7f31380787e0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f314c0d2770 0x7f314c14f270 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f314c06ca90 0x7f314c0cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 --2- 172.21.15.46:0/738069710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f314c06c680 0x7f314c0d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 >> 172.21.15.46:0/738069710 conn(0x7f314c073d10 msgr2=0x7f314c070730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 shutdown_connections 2024-05-09T03:21:35.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:35.990+0000 7f3151061640 1 -- 172.21.15.46:0/738069710 wait complete. 2024-05-09T03:21:36.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors' 2024-05-09T03:21:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.115+0000 7ff313ae0640 1 Processor -- start 2024-05-09T03:21:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.115+0000 7ff313ae0640 1 -- start start 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff313ae0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0d57a0 0x7ff30c160610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff313ae0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff30c0b4af0 0x7ff30c0b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff313ae0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff30c0d49e0 con 0x7ff30c0b77e0 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff313ae0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff30c0d5bb0 con 0x7ff30c0b4af0 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff313ae0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff30c0d4cc0 con 0x7ff30c0d57a0 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31107b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff30c0b4af0 0x7ff30c0b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:36.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31187c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0d57a0 0x7ff30c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31207d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff30c0b77e0 0x7ff30c0b7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:37324/0 (socket says 172.21.15.46:37324) 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31207d640 1 -- 172.21.15.46:0/3613434640 learned_addr learned my addr 172.21.15.46:0/3613434640 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31107b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff30c0b4af0 0x7ff30c0b4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.46:33296/0 (socket says 172.21.15.46:33296) 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31107b640 1 -- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0d57a0 msgr2=0x7ff30c160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31107b640 1 --2- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0d57a0 0x7ff30c160610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31107b640 1 -- 172.21.15.46:0/3613434640 >> v1:172.21.15.88:6789/0 conn(0x7ff30c0b77e0 legacy=0x7ff30c0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31107b640 1 -- 172.21.15.46:0/3613434640 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c0d4fa0 con 0x7ff30c0b4af0 2024-05-09T03:21:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.117+0000 7ff31187c640 1 --2- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0d57a0 0x7ff30c160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.118+0000 7ff31107b640 1 --2- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b4af0 0x7ff30c0b4f00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff3000097a0 tx=0x7ff30002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ddd340c5394d7fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.118+0000 7ff31087a640 1 -- 172.21.15.46:0/3613434640 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff300037070 con 0x7ff30c0b4af0 2024-05-09T03:21:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.118+0000 7ff31087a640 1 -- 172.21.15.46:0/3613434640 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff300032440 con 0x7ff30c0b4af0 2024-05-09T03:21:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.118+0000 7ff31087a640 1 -- 172.21.15.46:0/3613434640 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff300032810 con 0x7ff30c0b4af0 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 -- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b4af0 msgr2=0x7ff30c0b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 --2- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b4af0 0x7ff30c0b4f00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ff3000097a0 tx=0x7ff30002a180 comp rx=0 tx=0).stop 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 -- 172.21.15.46:0/3613434640 shutdown_connections 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 --2- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b4af0 0x7ff30c0b4f00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 --2- 172.21.15.46:0/3613434640 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0d57a0 0x7ff30c160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 -- 172.21.15.46:0/3613434640 >> 172.21.15.46:0/3613434640 conn(0x7ff30c0743e0 msgr2=0x7ff30c0d27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 -- 172.21.15.46:0/3613434640 shutdown_connections 2024-05-09T03:21:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.119+0000 7ff313ae0640 1 -- 172.21.15.46:0/3613434640 wait complete. 2024-05-09T03:21:36.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.121+0000 7ff313ae0640 1 Processor -- start 2024-05-09T03:21:36.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.121+0000 7ff313ae0640 1 -- start start 2024-05-09T03:21:36.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.122+0000 7ff313ae0640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0b4af0 0x7ff30c0d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff313ae0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b77e0 0x7ff30c147030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff313ae0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 0x7ff30c0d3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff313ae0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c0d1920 con 0x7ff30c0d57a0 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff313ae0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c0d15b0 con 0x7ff30c0b77e0 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff313ae0640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff30c145e10 con 0x7ff30c0b4af0 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 0x7ff30c0d3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 0x7ff30c0d3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54404/0 (socket says 172.21.15.46:54404) 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 -- 172.21.15.46:0/2359347178 learned_addr learned my addr 172.21.15.46:0/2359347178 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31107b640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b77e0 0x7ff30c147030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31187c640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0b4af0 0x7ff30c0d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 -- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0b4af0 msgr2=0x7ff30c0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0b4af0 0x7ff30c0d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 -- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b77e0 msgr2=0x7ff30c147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b77e0 0x7ff30c147030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 -- 172.21.15.46:0/2359347178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c147d60 con 0x7ff30c0d57a0 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff31207d640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 0x7ff30c0d3b90 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7ff30400c370 tx=0x7ff30400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304017070 con 0x7ff30c0d57a0 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff304005ce0 con 0x7ff30c0d57a0 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30c1f9b50 con 0x7ff30c0d57a0 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.123+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3040072f0 con 0x7ff30c0d57a0 2024-05-09T03:21:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.124+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff30c145b50 con 0x7ff30c0d57a0 2024-05-09T03:21:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.125+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff30c06a270 con 0x7ff30c0d57a0 2024-05-09T03:21:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.125+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff30401a070 con 0x7ff30c0d57a0 2024-05-09T03:21:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.126+0000 7ff30a7fc640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2f80762d0 0x7ff2f80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.126+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7ff3040965d0 con 0x7ff30c0d57a0 2024-05-09T03:21:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.126+0000 7ff31187c640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2f80762d0 0x7ff2f80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.127+0000 7ff31187c640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2f80762d0 0x7ff2f80787e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009bd0 tx=0x7ff2fc00b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:36.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.131+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff30405fb80 con 0x7ff30c0d57a0 2024-05-09T03:21:36.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.393+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]} v 0) -- 0x7ff30c147af0 con 0x7ff30c0d57a0 2024-05-09T03:21:36.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.632+0000 7ff30a7fc640 1 -- 172.21.15.46:0/2359347178 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]}]=0 v617) ==== 135+0+0 (secure 0 0 0) 0x7ff304063e80 con 0x7ff30c0d57a0 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2f80762d0 msgr2=0x7ff2f80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2f80762d0 0x7ff2f80787e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009bd0 tx=0x7ff2fc00b040 comp rx=0 tx=0).stop 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 msgr2=0x7ff30c0d3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 0x7ff30c0d3b90 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7ff30400c370 tx=0x7ff30400c830 comp rx=0 tx=0).stop 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 shutdown_connections 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff2f80762d0 0x7ff2f80787e0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30c0d57a0 0x7ff30c0d3b90 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff30c0b77e0 0x7ff30c147030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 --2- 172.21.15.46:0/2359347178 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff30c0b4af0 0x7ff30c0d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 >> 172.21.15.46:0/2359347178 conn(0x7ff30c0743e0 msgr2=0x7ff30c0b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 shutdown_connections 2024-05-09T03:21:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:36.635+0000 7ff313ae0640 1 -- 172.21.15.46:0/2359347178 wait complete. 2024-05-09T03:21:36.648 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated ... ok 2024-05-09T03:21:36.649 INFO:tasks.cephfs_test_runner: 2024-05-09T03:21:36.649 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-05-09T03:21:36.649 INFO:tasks.cephfs_test_runner:Ran 6 tests in 701.992s 2024-05-09T03:21:36.649 INFO:tasks.cephfs_test_runner: 2024-05-09T03:21:36.649 INFO:tasks.cephfs_test_runner:OK 2024-05-09T03:21:36.649 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-05-09T03:21:36.657 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-09T03:21:36.665 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-09T03:21:36.666 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:36.666 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:36.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:21:36.684 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-05-09T03:21:36.684 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:36.741 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057103,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20947 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249452k,nr_inodes=812363,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498912k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-05-09T03:21:36.741 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:21:36.741 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:36.761 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:21:36.762 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-05-09T03:21:36.762 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T03:21:36.820 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056333,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19715 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-05-09T03:21:36.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:36.820 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-05-09T03:21:36.879 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-05-09T03:21:36.879 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-05-09T03:21:36.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:36.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-05-09T03:21:36.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:36.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-05-09T03:21:37.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:37.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2024-05-09T03:21:37.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:37.111 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-05-09T03:21:37.130 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-05-09T03:21:37.130 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T03:21:37.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:37.130 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-05-09T03:21:37.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:37.202 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T03:21:37.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T03:21:37.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-05-09T03:21:37.367 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-09T03:21:37.379 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T03:21:37.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:21:37.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.497+0000 7fde6a461640 1 Processor -- start 2024-05-09T03:21:37.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.497+0000 7fde6a461640 1 -- start start 2024-05-09T03:21:37.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde6a461640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde640b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde6a461640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde6408cfc0 0x7fde6408d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde6a461640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde640b0a00 0x7fde640b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde6a461640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fde6408d910 con 0x7fde640b0a00 2024-05-09T03:21:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde6a461640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fde640d2270 con 0x7fde6408cfc0 2024-05-09T03:21:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde6a461640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fde640d2510 con 0x7fde640b8d10 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde689fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde640b0a00 0x7fde640b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde689fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde640b0a00 0x7fde640b0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54414/0 (socket says 172.21.15.46:54414) 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde689fe640 1 -- 172.21.15.46:0/661798598 learned_addr learned my addr 172.21.15.46:0/661798598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde63fff640 1 --2- 172.21.15.46:0/661798598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde640b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.499+0000 7fde637fe640 1 --2- 172.21.15.46:0/661798598 >> v2:172.21.15.88:3300/0 conn(0x7fde6408cfc0 0x7fde6408d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.500+0000 7fde689fe640 1 -- 172.21.15.46:0/661798598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 msgr2=0x7fde640b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.500+0000 7fde689fe640 1 --2- 172.21.15.46:0/661798598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde640b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.500+0000 7fde689fe640 1 -- 172.21.15.46:0/661798598 >> v2:172.21.15.88:3300/0 conn(0x7fde6408cfc0 msgr2=0x7fde6408d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.500+0000 7fde689fe640 1 --2- 172.21.15.46:0/661798598 >> v2:172.21.15.88:3300/0 conn(0x7fde6408cfc0 0x7fde6408d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.500+0000 7fde689fe640 1 -- 172.21.15.46:0/661798598 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde640d27f0 con 0x7fde640b0a00 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.500+0000 7fde689fe640 1 --2- 172.21.15.46:0/661798598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde640b0a00 0x7fde640b0e10 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fde640d4cf0 tx=0x7fde5c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c372be4236ae89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde62ffd640 1 -- 172.21.15.46:0/661798598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c036070 con 0x7fde640b0a00 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde62ffd640 1 -- 172.21.15.46:0/661798598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5c031600 con 0x7fde640b0a00 2024-05-09T03:21:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde62ffd640 1 -- 172.21.15.46:0/661798598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0319d0 con 0x7fde640b0a00 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 -- 172.21.15.46:0/661798598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde640b0a00 msgr2=0x7fde640b0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 --2- 172.21.15.46:0/661798598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde640b0a00 0x7fde640b0e10 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7fde640d4cf0 tx=0x7fde5c02a6a0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 -- 172.21.15.46:0/661798598 shutdown_connections 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 --2- 172.21.15.46:0/661798598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde640b0a00 0x7fde640b0e10 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 --2- 172.21.15.46:0/661798598 >> v2:172.21.15.88:3300/0 conn(0x7fde6408cfc0 0x7fde6408d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 --2- 172.21.15.46:0/661798598 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde640b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 -- 172.21.15.46:0/661798598 >> 172.21.15.46:0/661798598 conn(0x7fde640b56e0 msgr2=0x7fde640b5af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 -- 172.21.15.46:0/661798598 shutdown_connections 2024-05-09T03:21:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.501+0000 7fde6a461640 1 -- 172.21.15.46:0/661798598 wait complete. 2024-05-09T03:21:37.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.503+0000 7fde6a461640 1 Processor -- start 2024-05-09T03:21:37.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.503+0000 7fde6a461640 1 -- start start 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde6a461640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 0x7fde640cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde6a461640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde640b0a00 0x7fde641550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde6a461640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde6406e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde6a461640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde640b9380 con 0x7fde6408cfc0 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde6a461640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde640cfee0 con 0x7fde640b0a00 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde6a461640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fde640b9500 con 0x7fde640b8d10 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 0x7fde640cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 0x7fde640cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54424/0 (socket says 172.21.15.46:54424) 2024-05-09T03:21:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 -- 172.21.15.46:0/2869852784 learned_addr learned my addr 172.21.15.46:0/2869852784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde637fe640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde640b0a00 0x7fde641550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde689fe640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde6406e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 -- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 msgr2=0x7fde6406e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde6406e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 -- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde640b0a00 msgr2=0x7fde641550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde640b0a00 0x7fde641550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 -- 172.21.15.46:0/2869852784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde641555e0 con 0x7fde6408cfc0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde63fff640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 0x7fde640cd140 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fde5400e830 tx=0x7fde5400ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.505+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54017070 con 0x7fde6408cfc0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.506+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5400c940 con 0x7fde6408cfc0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.506+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde640a0660 con 0x7fde6408cfc0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.506+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5400ccf0 con 0x7fde6408cfc0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.506+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde640a0c20 con 0x7fde6408cfc0 2024-05-09T03:21:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.506+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6406a270 con 0x7fde6408cfc0 2024-05-09T03:21:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.507+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fde5401a070 con 0x7fde6408cfc0 2024-05-09T03:21:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.508+0000 7fde60ff9640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde50076300 0x7fde50078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.508+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fde54097660 con 0x7fde6408cfc0 2024-05-09T03:21:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.508+0000 7fde637fe640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde50076300 0x7fde50078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.509+0000 7fde637fe640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde50076300 0x7fde50078810 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fde640b88a0 tx=0x7fde58009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:37.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.513+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fde641555e0 con 0x7fde6408cfc0 2024-05-09T03:21:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.701+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fde6408daf0 con 0x7fde50076300 2024-05-09T03:21:37.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.703+0000 7fde60ff9640 1 -- 172.21.15.46:0/2869852784 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13492 (secure 0 0 0) 0x7fde6408daf0 con 0x7fde50076300 2024-05-09T03:21:37.705 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:37.705 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde50076300 msgr2=0x7fde50078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde50076300 0x7fde50078810 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fde640b88a0 tx=0x7fde58009190 comp rx=0 tx=0).stop 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 msgr2=0x7fde640cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 0x7fde640cd140 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fde5400e830 tx=0x7fde5400ecf0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 shutdown_connections 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fde640b8d10 0x7fde6406e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde640b0a00 0x7fde641550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fde50076300 0x7fde50078810 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 --2- 172.21.15.46:0/2869852784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6408cfc0 0x7fde640cd140 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 >> 172.21.15.46:0/2869852784 conn(0x7fde640b56e0 msgr2=0x7fde640ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 shutdown_connections 2024-05-09T03:21:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.705+0000 7fde6a461640 1 -- 172.21.15.46:0/2869852784 wait complete. 2024-05-09T03:21:37.717 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-05-09T03:21:36.689519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297936,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676784,"statfs":{"total":773094113280,"available":772789026816,"internally_reserved":0,"allocated":15556608,"data_stored":8101557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":289394225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28840,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-88,"num_read_kb":-90,"num_write":-95,"num_write_kb":-61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001394"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":341,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-05-09T03:21:35.770635+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:21:35.770635+0000","last_peered":"2024-05-09T03:21:35.770635+0000","last_clean":"2024-05-09T03:21:35.770635+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:35.770635+0000","last_undegraded":"2024-05-09T03:21:35.770635+0000","last_fullsized":"2024-05-09T03:21:35.770635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37680,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334160,"statfs":{"total":96636764160,"available":96598179840,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574998,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":860158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:21:37.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:21:37.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.831+0000 7f40d416c640 1 Processor -- start 2024-05-09T03:21:37.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.831+0000 7f40d416c640 1 -- start start 2024-05-09T03:21:37.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d416c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40cc0d0b90 0x7f40cc0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d416c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40cc0cf6b0 0x7f40cc06d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d416c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc0d2b80 con 0x7f40cc0b92e0 2024-05-09T03:21:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d416c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc0d2e60 con 0x7f40cc0d0b90 2024-05-09T03:21:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d416c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc0d3140 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40cc0d0b90 0x7f40cc0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40cc0d0b90 0x7f40cc0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54440/0 (socket says 172.21.15.46:54440) 2024-05-09T03:21:37.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 -- 172.21.15.46:0/1495987674 learned_addr learned my addr 172.21.15.46:0/1495987674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d2709640 1 --2- 172.21.15.46:0/1495987674 >> v2:172.21.15.88:3300/0 conn(0x7f40cc0cf6b0 0x7f40cc06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 -- 172.21.15.46:0/1495987674 >> v2:172.21.15.88:3300/0 conn(0x7f40cc0cf6b0 msgr2=0x7f40cc06d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 --2- 172.21.15.46:0/1495987674 >> v2:172.21.15.88:3300/0 conn(0x7f40cc0cf6b0 0x7f40cc06d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 -- 172.21.15.46:0/1495987674 >> v1:172.21.15.88:6789/0 conn(0x7f40cc0b92e0 legacy=0x7f40cc0d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.833+0000 7f40d1707640 1 -- 172.21.15.46:0/1495987674 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc0d3420 con 0x7f40cc0d0b90 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.834+0000 7f40d1707640 1 --2- 172.21.15.46:0/1495987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0d0b90 0x7f40cc0b51a0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0b9720 tx=0x7f40bc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d466bcfff9c845 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.834+0000 7f40d0f06640 1 -- 172.21.15.46:0/1495987674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc037070 con 0x7f40cc0d0b90 2024-05-09T03:21:37.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.834+0000 7f40d0f06640 1 -- 172.21.15.46:0/1495987674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40bc032440 con 0x7f40cc0d0b90 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.834+0000 7f40d0f06640 1 -- 172.21.15.46:0/1495987674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc032810 con 0x7f40cc0d0b90 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 -- 172.21.15.46:0/1495987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0d0b90 msgr2=0x7f40cc0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 --2- 172.21.15.46:0/1495987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0d0b90 0x7f40cc0b51a0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0b9720 tx=0x7f40bc02a1e0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 -- 172.21.15.46:0/1495987674 shutdown_connections 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 --2- 172.21.15.46:0/1495987674 >> v2:172.21.15.88:3300/0 conn(0x7f40cc0cf6b0 0x7f40cc06d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 --2- 172.21.15.46:0/1495987674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0d0b90 0x7f40cc0b51a0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 -- 172.21.15.46:0/1495987674 >> 172.21.15.46:0/1495987674 conn(0x7f40cc0d3c00 msgr2=0x7f40cc0d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:37.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 -- 172.21.15.46:0/1495987674 shutdown_connections 2024-05-09T03:21:37.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.835+0000 7f40d416c640 1 -- 172.21.15.46:0/1495987674 wait complete. 2024-05-09T03:21:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.837+0000 7f40d416c640 1 Processor -- start 2024-05-09T03:21:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.837+0000 7f40d416c640 1 -- start start 2024-05-09T03:21:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d416c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40cc0b92e0 0x7f40cc0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d416c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 0x7f40cc0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d416c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f40cc0b41e0 0x7f40cc0b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d416c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc06e060 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d416c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc06dee0 con 0x7f40cc0b92e0 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d416c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc06e1e0 con 0x7f40cc0b41e0 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d1707640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 0x7f40cc0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d1707640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 0x7f40cc0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54442/0 (socket says 172.21.15.46:54442) 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d1707640 1 -- 172.21.15.46:0/4034985193 learned_addr learned my addr 172.21.15.46:0/4034985193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d1f08640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40cc0b92e0 0x7f40cc0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.838+0000 7f40d2709640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f40cc0b41e0 0x7f40cc0b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1707640 1 -- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f40cc0b41e0 msgr2=0x7f40cc0b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1707640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f40cc0b41e0 0x7f40cc0b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1707640 1 -- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40cc0b92e0 msgr2=0x7f40cc0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1707640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40cc0b92e0 0x7f40cc0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1707640 1 -- 172.21.15.46:0/4034985193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc1e9b10 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1f08640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40cc0b92e0 0x7f40cc0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d1707640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 0x7f40cc0b51a0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f40bc000c00 tx=0x7f40bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc037040 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40bc042070 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc1e7c20 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc030880 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.839+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc15d7e0 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.840+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40cc1e9880 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.841+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f40bc030a20 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.841+0000 7f40c27fc640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40b00762d0 0x7f40b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.841+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f40bc0b7c20 con 0x7f40cc0cf6b0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.841+0000 7f40d1f08640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40b00762d0 0x7f40b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.842+0000 7f40d1f08640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40b00762d0 0x7f40b00787e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f40c8007fd0 tx=0x7f40c8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:37.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:37.846+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f40bc081820 con 0x7f40cc0cf6b0 2024-05-09T03:21:38.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.032+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f40cc0b5c20 con 0x7f40b00762d0 2024-05-09T03:21:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.033+0000 7f40c27fc640 1 -- 172.21.15.46:0/4034985193 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13492 (secure 0 0 0) 0x7f40c8001c50 con 0x7f40b00762d0 2024-05-09T03:21:38.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:38.035 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:21:38.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40b00762d0 msgr2=0x7f40b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40b00762d0 0x7f40b00787e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f40c8007fd0 tx=0x7f40c8009040 comp rx=0 tx=0).stop 2024-05-09T03:21:38.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 msgr2=0x7f40cc0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 0x7f40cc0b51a0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f40bc000c00 tx=0x7f40bc0046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 shutdown_connections 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f40cc0b41e0 0x7f40cc0b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f40b00762d0 0x7f40b00787e0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc0cf6b0 0x7f40cc0b51a0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 --2- 172.21.15.46:0/4034985193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40cc0b92e0 0x7f40cc0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.035+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 >> 172.21.15.46:0/4034985193 conn(0x7f40cc0d3c00 msgr2=0x7f40cc1600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.036+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 shutdown_connections 2024-05-09T03:21:38.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.036+0000 7f40d416c640 1 -- 172.21.15.46:0/4034985193 wait complete. 2024-05-09T03:21:38.050 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-05-09T03:21:36.689519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297936,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676784,"statfs":{"total":773094113280,"available":772789026816,"internally_reserved":0,"allocated":15556608,"data_stored":8101557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":289394225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28840,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-88,"num_read_kb":-90,"num_write":-95,"num_write_kb":-61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001394"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":341,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-05-09T03:21:35.770635+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:21:35.770635+0000","last_peered":"2024-05-09T03:21:35.770635+0000","last_clean":"2024-05-09T03:21:35.770635+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:35.770635+0000","last_undegraded":"2024-05-09T03:21:35.770635+0000","last_fullsized":"2024-05-09T03:21:35.770635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37680,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334160,"statfs":{"total":96636764160,"available":96598179840,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574998,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":860158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:21:38.050 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T03:21:38.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:21:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.165+0000 7f47e2d9d640 1 Processor -- start 2024-05-09T03:21:38.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.165+0000 7f47e2d9d640 1 -- start start 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.166+0000 7f47e2d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc164c70 0x7f47dc1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.166+0000 7f47e2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47dc0d2480 0x7f47dc0d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.166+0000 7f47e2d9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47dc160b20 con 0x7f47dc15dfe0 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.166+0000 7f47e2d9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f47dc161150 con 0x7f47dc0d2480 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.166+0000 7f47e2d9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc0d15f0 con 0x7f47dc164c70 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47dc0d2480 0x7f47dc0d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f47dc0d2480 0x7f47dc0d2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54448/0 (socket says 172.21.15.46:54448) 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 -- 172.21.15.46:0/3300315710 learned_addr learned my addr 172.21.15.46:0/3300315710 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e0b39640 1 --2- 172.21.15.46:0/3300315710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc164c70 0x7f47dc1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 -- 172.21.15.46:0/3300315710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc164c70 msgr2=0x7f47dc1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 --2- 172.21.15.46:0/3300315710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc164c70 0x7f47dc1615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 -- 172.21.15.46:0/3300315710 >> v1:172.21.15.88:6789/0 conn(0x7f47dc15dfe0 legacy=0x7f47dc0d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.167+0000 7f47e133a640 1 -- 172.21.15.46:0/3300315710 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc0d18d0 con 0x7f47dc0d2480 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.168+0000 7f47e133a640 1 --2- 172.21.15.46:0/3300315710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc0d2480 0x7f47dc0d2890 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009e50 tx=0x7f47d402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdeb2e1f3e994a33 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.168+0000 7f47d37fe640 1 -- 172.21.15.46:0/3300315710 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4036070 con 0x7f47dc0d2480 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.168+0000 7f47d37fe640 1 -- 172.21.15.46:0/3300315710 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d4031600 con 0x7f47dc0d2480 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.168+0000 7f47d37fe640 1 -- 172.21.15.46:0/3300315710 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40319d0 con 0x7f47dc0d2480 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.168+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3300315710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc0d2480 msgr2=0x7f47dc0d2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.168+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3300315710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc0d2480 0x7f47dc0d2890 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009e50 tx=0x7f47d402a710 comp rx=0 tx=0).stop 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.169+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3300315710 shutdown_connections 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.169+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3300315710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc0d2480 0x7f47dc0d2890 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.169+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3300315710 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc164c70 0x7f47dc1615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.169+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3300315710 >> 172.21.15.46:0/3300315710 conn(0x7f47dc073db0 msgr2=0x7f47dc0741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:38.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.169+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3300315710 shutdown_connections 2024-05-09T03:21:38.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.169+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3300315710 wait complete. 2024-05-09T03:21:38.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.170+0000 7f47e2d9d640 1 Processor -- start 2024-05-09T03:21:38.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.170+0000 7f47e2d9d640 1 -- start start 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e2d9d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc0d2480 0x7f47dc1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47dc15dfe0 0x7f47dc1eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 0x7f47dc1f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e2d9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc0d2dd0 con 0x7f47dc164c70 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e2d9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc15e3f0 con 0x7f47dc15dfe0 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e2d9d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc0d45c0 con 0x7f47dc0d2480 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 0x7f47dc1f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 0x7f47dc1f6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54450/0 (socket says 172.21.15.46:54450) 2024-05-09T03:21:38.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 -- 172.21.15.46:0/3034766479 learned_addr learned my addr 172.21.15.46:0/3034766479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e0b39640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc0d2480 0x7f47dc1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47d3fff640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47dc15dfe0 0x7f47dc1eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 -- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc0d2480 msgr2=0x7f47dc1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc0d2480 0x7f47dc1524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 -- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47dc15dfe0 msgr2=0x7f47dc1eaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47dc15dfe0 0x7f47dc1eaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.172+0000 7f47e133a640 1 -- 172.21.15.46:0/3034766479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc1f7550 con 0x7f47dc164c70 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.173+0000 7f47e133a640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 0x7f47dc1f6970 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009e50 tx=0x7f47d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.173+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4036050 con 0x7f47dc164c70 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.173+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d4042070 con 0x7f47dc164c70 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.173+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47dc1e9fd0 con 0x7f47dc164c70 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.173+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4009510 con 0x7f47dc164c70 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.173+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47dc1ea590 con 0x7f47dc164c70 2024-05-09T03:21:38.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.174+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47dc08cfd0 con 0x7f47dc164c70 2024-05-09T03:21:38.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.174+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f47d4007bf0 con 0x7f47dc164c70 2024-05-09T03:21:38.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.175+0000 7f47d17fa640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47c8076300 0x7f47c8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.175+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f47d40b8060 con 0x7f47dc164c70 2024-05-09T03:21:38.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.175+0000 7f47e0b39640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47c8076300 0x7f47c8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.176+0000 7f47e0b39640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47c8076300 0x7f47c8078810 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009aa0 tx=0x7f47d8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:38.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.180+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f47dc08cfd0 con 0x7f47dc164c70 2024-05-09T03:21:38.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:21:38.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:21:38.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.369+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f47dc06a270 con 0x7f47c8076300 2024-05-09T03:21:38.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.370+0000 7f47d17fa640 1 -- 172.21.15.46:0/3034766479 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13492 (secure 0 0 0) 0x7f47dc06a270 con 0x7f47c8076300 2024-05-09T03:21:38.371 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:38.372 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:21:38.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47c8076300 msgr2=0x7f47c8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47c8076300 0x7f47c8078810 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009aa0 tx=0x7f47d8009190 comp rx=0 tx=0).stop 2024-05-09T03:21:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 msgr2=0x7f47dc1f6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 0x7f47dc1f6970 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009e50 tx=0x7f47d4004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 shutdown_connections 2024-05-09T03:21:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f47c8076300 0x7f47c8078810 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47dc164c70 0x7f47dc1f6970 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47dc15dfe0 0x7f47dc1eaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 --2- 172.21.15.46:0/3034766479 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f47dc0d2480 0x7f47dc1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 >> 172.21.15.46:0/3034766479 conn(0x7f47dc073db0 msgr2=0x7f47dc0bb040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:38.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 shutdown_connections 2024-05-09T03:21:38.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.372+0000 7f47e2d9d640 1 -- 172.21.15.46:0/3034766479 wait complete. 2024-05-09T03:21:38.385 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-05-09T03:21:36.689519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297936,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676784,"statfs":{"total":773094113280,"available":772789026816,"internally_reserved":0,"allocated":15556608,"data_stored":8101557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":289394225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28840,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-88,"num_read_kb":-90,"num_write":-95,"num_write_kb":-61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001394"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":341,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-05-09T03:21:35.770635+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:21:35.770635+0000","last_peered":"2024-05-09T03:21:35.770635+0000","last_clean":"2024-05-09T03:21:35.770635+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:35.770635+0000","last_undegraded":"2024-05-09T03:21:35.770635+0000","last_fullsized":"2024-05-09T03:21:35.770635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37680,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334160,"statfs":{"total":96636764160,"available":96598179840,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574998,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41240,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330600,"statfs":{"total":96636764160,"available":96594534400,"internally_reserved":0,"allocated":1781760,"data_stored":860158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574998,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:21:38.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T03:21:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.499+0000 7f4cb6653640 1 Processor -- start 2024-05-09T03:21:38.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.499+0000 7f4cb6653640 1 -- start start 2024-05-09T03:21:38.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.500+0000 7f4cb6653640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cb00b5210 0x7f4cb01613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.500+0000 7f4cb6653640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cb00d4800 0x7f4cb00d4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.500+0000 7f4cb6653640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cb00d5280 con 0x7f4cb00b77c0 2024-05-09T03:21:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.500+0000 7f4cb6653640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4cb00b5620 con 0x7f4cb00b5210 2024-05-09T03:21:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.500+0000 7f4cb6653640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4cb01609a0 con 0x7f4cb00d4800 2024-05-09T03:21:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cb00b5210 0x7f4cb01613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4cb4bf0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4cb00b77c0 0x7f4cb00b7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50518/0 (socket says 172.21.15.46:50518) 2024-05-09T03:21:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4cb00b5210 0x7f4cb01613c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54462/0 (socket says 172.21.15.46:54462) 2024-05-09T03:21:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4cb4bf0640 1 -- 172.21.15.46:0/3231525209 learned_addr learned my addr 172.21.15.46:0/3231525209 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caf7fe640 1 --2- 172.21.15.46:0/3231525209 >> v2:172.21.15.88:3300/0 conn(0x7f4cb00d4800 0x7f4cb00d4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caffff640 1 -- 172.21.15.46:0/3231525209 >> v2:172.21.15.88:3300/0 conn(0x7f4cb00d4800 msgr2=0x7f4cb00d4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caffff640 1 --2- 172.21.15.46:0/3231525209 >> v2:172.21.15.88:3300/0 conn(0x7f4cb00d4800 0x7f4cb00d4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caffff640 1 -- 172.21.15.46:0/3231525209 >> v1:172.21.15.46:6789/0 conn(0x7f4cb00b77c0 legacy=0x7f4cb00b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caffff640 1 -- 172.21.15.46:0/3231525209 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb00d5560 con 0x7f4cb00b5210 2024-05-09T03:21:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caeffd640 1 -- 172.21.15.46:0/3231525209 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377922546 0 0) 0x7f4cb00d5280 con 0x7f4cb00b77c0 2024-05-09T03:21:38.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.501+0000 7f4caf7fe640 1 --2- 172.21.15.46:0/3231525209 >> v2:172.21.15.88:3300/0 conn(0x7f4cb00d4800 0x7f4cb00d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:38.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.502+0000 7f4caffff640 1 --2- 172.21.15.46:0/3231525209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00b5210 0x7f4cb01613c0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f4cb006ee20 tx=0x7f4ca002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c3ed83412e638c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:38.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.502+0000 7f4caeffd640 1 -- 172.21.15.46:0/3231525209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0037070 con 0x7f4cb00b5210 2024-05-09T03:21:38.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.502+0000 7f4caeffd640 1 -- 172.21.15.46:0/3231525209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca0032440 con 0x7f4cb00b5210 2024-05-09T03:21:38.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.502+0000 7f4caeffd640 1 -- 172.21.15.46:0/3231525209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca002ae60 con 0x7f4cb00b5210 2024-05-09T03:21:38.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 -- 172.21.15.46:0/3231525209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00b5210 msgr2=0x7f4cb01613c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3231525209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00b5210 0x7f4cb01613c0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f4cb006ee20 tx=0x7f4ca002a110 comp rx=0 tx=0).stop 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 -- 172.21.15.46:0/3231525209 shutdown_connections 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3231525209 >> v2:172.21.15.88:3300/0 conn(0x7f4cb00d4800 0x7f4cb00d4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3231525209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00b5210 0x7f4cb01613c0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 -- 172.21.15.46:0/3231525209 >> 172.21.15.46:0/3231525209 conn(0x7f4cb00d29a0 msgr2=0x7f4cb00d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 -- 172.21.15.46:0/3231525209 shutdown_connections 2024-05-09T03:21:38.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.503+0000 7f4cb6653640 1 -- 172.21.15.46:0/3231525209 wait complete. 2024-05-09T03:21:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.504+0000 7f4cb6653640 1 Processor -- start 2024-05-09T03:21:38.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.505+0000 7f4cb6653640 1 -- start start 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb6653640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00b5210 0x7f4cb014f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb6653640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00b77c0 0x7f4cb00ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb6653640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 0x7f4cb00cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb6653640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb00d0ae0 con 0x7f4cb00d4800 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb6653640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0160ea0 con 0x7f4cb00b5210 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb6653640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f4cb0163e60 con 0x7f4cb00b77c0 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 0x7f4cb00cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 0x7f4cb00cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54478/0 (socket says 172.21.15.46:54478) 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 -- 172.21.15.46:0/3089326672 learned_addr learned my addr 172.21.15.46:0/3089326672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4caffff640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00b5210 0x7f4cb014f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4caf7fe640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00b77c0 0x7f4cb00ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 -- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00b77c0 msgr2=0x7f4cb00ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00b77c0 0x7f4cb00ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 -- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00b5210 msgr2=0x7f4cb014f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00b5210 0x7f4cb014f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.506+0000 7f4cb4bf0640 1 -- 172.21.15.46:0/3089326672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb00c9bc0 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4caf7fe640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00b77c0 0x7f4cb00ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4cb4bf0640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 0x7f4cb00cd330 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800c680 tx=0x7f4ca800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8018070 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca8004d80 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb00c9ea0 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca800a880 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.507+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb00ca460 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.508+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb006a270 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.509+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4ca801b070 con 0x7f4cb00d4800 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.509+0000 7f4cacff9640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c9c0762d0 0x7f4c9c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:38.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.509+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f4ca80968a0 con 0x7f4cb00d4800 2024-05-09T03:21:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.509+0000 7f4caffff640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c9c0762d0 0x7f4c9c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.510+0000 7f4caffff640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c9c0762d0 0x7f4c9c0787e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0034000 tx=0x7f4ca002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:38.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.514+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4ca8060410 con 0x7f4cb00d4800 2024-05-09T03:21:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.706+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4cb00c9950 con 0x7f4cb00d4800 2024-05-09T03:21:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.707+0000 7f4cacff9640 1 -- 172.21.15.46:0/3089326672 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+13088 (secure 0 0 0) 0x7f4ca805fd10 con 0x7f4cb00d4800 2024-05-09T03:21:38.709 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:38.709 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":139,"fsid":"36d71dd3-2926-4712-824e-80cfbafd76c2","created":"2024-05-09T03:09:10.113710+0000","modified":"2024-05-09T03:21:35.627342+0000","last_up_change":"2024-05-09T03:09:21.009309+0000","last_in_change":"2024-05-09T03:09:12.489163+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T03:09:22.506928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2e91e1eb-2bae-45d0-935b-ec079c920ca5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6801","nonce":1118951116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6803","nonce":1118951116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6807","nonce":1118951116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1118951116},{"type":"v1","addr":"172.21.15.46:6805","nonce":1118951116}]},"public_addr":"172.21.15.46:6801/1118951116","cluster_addr":"172.21.15.46:6803/1118951116","heartbeat_back_addr":"172.21.15.46:6807/1118951116","heartbeat_front_addr":"172.21.15.46:6805/1118951116","state":["exists","up"]},{"osd":1,"uuid":"a15e320a-d34c-4d31-9f0d-9cfdd145adcd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6825","nonce":1118146182}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6827","nonce":1118146182}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6831","nonce":1118146182}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1118146182},{"type":"v1","addr":"172.21.15.46:6829","nonce":1118146182}]},"public_addr":"172.21.15.46:6825/1118146182","cluster_addr":"172.21.15.46:6827/1118146182","heartbeat_back_addr":"172.21.15.46:6831/1118146182","heartbeat_front_addr":"172.21.15.46:6829/1118146182","state":["exists","up"]},{"osd":2,"uuid":"84fd4c83-09ea-4eda-b3cc-ddd0aa7adf16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6809","nonce":2404849282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6811","nonce":2404849282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6815","nonce":2404849282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2404849282},{"type":"v1","addr":"172.21.15.46:6813","nonce":2404849282}]},"public_addr":"172.21.15.46:6809/2404849282","cluster_addr":"172.21.15.46:6811/2404849282","heartbeat_back_addr":"172.21.15.46:6815/2404849282","heartbeat_front_addr":"172.21.15.46:6813/2404849282","state":["exists","up"]},{"osd":3,"uuid":"29e7216f-bf54-4c12-9ce2-a567f991c9b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6817","nonce":3177037259}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6819","nonce":3177037259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6823","nonce":3177037259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3177037259},{"type":"v1","addr":"172.21.15.46:6821","nonce":3177037259}]},"public_addr":"172.21.15.46:6817/3177037259","cluster_addr":"172.21.15.46:6819/3177037259","heartbeat_back_addr":"172.21.15.46:6823/3177037259","heartbeat_front_addr":"172.21.15.46:6821/3177037259","state":["exists","up"]},{"osd":4,"uuid":"e2b044e3-c608-40af-b9b7-6f04bf54b508","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6825","nonce":902225877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6827","nonce":902225877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6831","nonce":902225877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":902225877},{"type":"v1","addr":"172.21.15.88:6829","nonce":902225877}]},"public_addr":"172.21.15.88:6825/902225877","cluster_addr":"172.21.15.88:6827/902225877","heartbeat_back_addr":"172.21.15.88:6831/902225877","heartbeat_front_addr":"172.21.15.88:6829/902225877","state":["exists","up"]},{"osd":5,"uuid":"5c1533a5-2622-4320-9458-107694403b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6801","nonce":122581025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6803","nonce":122581025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6807","nonce":122581025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":122581025},{"type":"v1","addr":"172.21.15.88:6805","nonce":122581025}]},"public_addr":"172.21.15.88:6801/122581025","cluster_addr":"172.21.15.88:6803/122581025","heartbeat_back_addr":"172.21.15.88:6807/122581025","heartbeat_front_addr":"172.21.15.88:6805/122581025","state":["exists","up"]},{"osd":6,"uuid":"7ac7f6ad-996f-448d-b0f6-2c055736ef8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6809","nonce":516824467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6811","nonce":516824467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6815","nonce":516824467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":516824467},{"type":"v1","addr":"172.21.15.88:6813","nonce":516824467}]},"public_addr":"172.21.15.88:6809/516824467","cluster_addr":"172.21.15.88:6811/516824467","heartbeat_back_addr":"172.21.15.88:6815/516824467","heartbeat_front_addr":"172.21.15.88:6813/516824467","state":["exists","up"]},{"osd":7,"uuid":"6f3e7095-2ba5-48a0-864d-b1d0c0ef55ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6817","nonce":3368726188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6819","nonce":3368726188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6823","nonce":3368726188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":3368726188},{"type":"v1","addr":"172.21.15.88:6821","nonce":3368726188}]},"public_addr":"172.21.15.88:6817/3368726188","cluster_addr":"172.21.15.88:6819/3368726188","heartbeat_back_addr":"172.21.15.88:6823/3368726188","heartbeat_front_addr":"172.21.15.88:6821/3368726188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.629210+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.715418+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.618187+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.659612+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.736988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.662133+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.690235+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T03:09:17.692348+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/2236478370":"2024-05-10T03:21:32.618958+0000","172.21.15.46:6833/2236478370":"2024-05-10T03:21:32.618958+0000","172.21.15.46:6835/2230300620":"2024-05-10T03:21:18.861166+0000","172.21.15.46:6833/2230300620":"2024-05-10T03:21:18.861166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T03:21:38.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c9c0762d0 msgr2=0x7f4c9c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c9c0762d0 0x7f4c9c0787e0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0034000 tx=0x7f4ca002e000 comp rx=0 tx=0).stop 2024-05-09T03:21:38.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 msgr2=0x7f4cb00cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 0x7f4cb00cd330 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800c680 tx=0x7f4ca800cb40 comp rx=0 tx=0).stop 2024-05-09T03:21:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 shutdown_connections 2024-05-09T03:21:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4c9c0762d0 0x7f4c9c0787e0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4cb00d4800 0x7f4cb00cd330 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4cb00b77c0 0x7f4cb00ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 --2- 172.21.15.46:0/3089326672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cb00b5210 0x7f4cb014f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:38.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 >> 172.21.15.46:0/3089326672 conn(0x7f4cb00d29a0 msgr2=0x7f4cb00b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:38.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 shutdown_connections 2024-05-09T03:21:38.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:38.709+0000 7f4cb6653640 1 -- 172.21.15.46:0/3089326672 wait complete. 2024-05-09T03:21:39.724 INFO:tasks.ceph:Scrubbing osd.0 2024-05-09T03:21:39.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:39.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.856+0000 7f5092108640 1 Processor -- start 2024-05-09T03:21:39.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.856+0000 7f5092108640 1 -- start start 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f5092108640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 0x7f508c0707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f5092108640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508c070cf0 con 0x7f508c13fe60 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f5092108640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508c161b10 con 0x7f508c0cdb30 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f5092108640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f508c15d790 con 0x7f508c0703a0 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f508b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f508c13fe60 0x7f508c161400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50528/0 (socket says 172.21.15.46:50528) 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f508b7fe640 1 -- 172.21.15.46:0/3629581916 learned_addr learned my addr 172.21.15.46:0/3629581916 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:39.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f508affd640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 0x7f508c0707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2334399726 0 0) 0x7f507c001110 con 0x7f508c13fe60 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.858+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5074003520 con 0x7f508c13fe60 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283875972 0 0) 0x7f508c161b10 con 0x7f508c0cdb30 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f507c001110 con 0x7f508c0cdb30 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 972849084 0 0) 0x7f5074003520 con 0x7f508c13fe60 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 msgr2=0x7f508c0707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 0x7f508c0707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 >> v1:172.21.15.88:6789/0 conn(0x7f508c0cdb30 legacy=0x7f508c0cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c15da70 con 0x7f508c13fe60 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f507c0018c0 con 0x7f508c13fe60 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 >> v1:172.21.15.46:6789/0 conn(0x7f508c13fe60 legacy=0x7f508c161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 0x7f5074003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50740041c0 0x7f5074004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5074004d90 0x7f5074005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f508c15d790 con 0x7f50740041c0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f507400bc20 con 0x7f5074004d90 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f507400be20 con 0x7f508c0703a0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508b7fe640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 0x7f5074003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508affd640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50740041c0 0x7f5074004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508bfff640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5074004d90 0x7f5074005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508affd640 1 -- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 msgr2=0x7f5074003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508affd640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c0703a0 0x7f5074003c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508affd640 1 -- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5074004d90 msgr2=0x7f5074005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508affd640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5074004d90 0x7f5074005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.859+0000 7f508affd640 1 -- 172.21.15.46:0/3629581916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5074003520 con 0x7f50740041c0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.860+0000 7f508affd640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50740041c0 0x7f5074004800 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f508c0b4510 tx=0x7f5074005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd44674ed6a6ace1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.860+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080036070 con 0x7f50740041c0 2024-05-09T03:21:39.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.860+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5080031400 con 0x7f50740041c0 2024-05-09T03:21:39.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.860+0000 7f508a7fc640 1 -- 172.21.15.46:0/3629581916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080009690 con 0x7f50740041c0 2024-05-09T03:21:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f5092108640 1 -- 172.21.15.46:0/3629581916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50740041c0 msgr2=0x7f5074004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f5092108640 1 --2- 172.21.15.46:0/3629581916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50740041c0 0x7f5074004800 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f508c0b4510 tx=0x7f5074005830 comp rx=0 tx=0).stop 2024-05-09T03:21:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f508bfff640 1 -- 172.21.15.46:0/3629581916 reap_dead start 2024-05-09T03:21:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f5092108640 1 -- 172.21.15.46:0/3629581916 shutdown_connections 2024-05-09T03:21:39.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f5092108640 1 -- 172.21.15.46:0/3629581916 >> 172.21.15.46:0/3629581916 conn(0x7f508c06d0a0 msgr2=0x7f508c0b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f5092108640 1 -- 172.21.15.46:0/3629581916 shutdown_connections 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.861+0000 7f5092108640 1 -- 172.21.15.46:0/3629581916 wait complete. 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.863+0000 7f5092108640 1 Processor -- start 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.863+0000 7f5092108640 1 -- start start 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.864+0000 7f5092108640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740041c0 0x7f508c13e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.864+0000 7f5092108640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c150e70 0x7f508c152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.864+0000 7f5092108640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c1550d0 0x7f508c1554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.864+0000 7f5092108640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f508c140270 con 0x7f508c150e70 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.864+0000 7f5092108640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f508c15e030 con 0x7f50740041c0 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.864+0000 7f5092108640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f508c160d20 con 0x7f508c1550d0 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c150e70 0x7f508c152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c150e70 0x7f508c152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54494/0 (socket says 172.21.15.46:54494) 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 -- 172.21.15.46:0/3733112988 learned_addr learned my addr 172.21.15.46:0/3733112988 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508bfff640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c1550d0 0x7f508c1554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508b7fe640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740041c0 0x7f508c13e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 -- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c1550d0 msgr2=0x7f508c1554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f508c1550d0 0x7f508c1554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 -- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740041c0 msgr2=0x7f508c13e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740041c0 0x7f508c13e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 -- 172.21.15.46:0/3733112988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c13cf60 con 0x7f508c150e70 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.865+0000 7f508affd640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c150e70 0x7f508c152750 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f5080009b80 tx=0x7f5080004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.866+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080040020 con 0x7f508c150e70 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.866+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c13d240 con 0x7f508c150e70 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.866+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5080003a40 con 0x7f508c150e70 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.866+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080036040 con 0x7f508c150e70 2024-05-09T03:21:39.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.866+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c13d800 con 0x7f508c150e70 2024-05-09T03:21:39.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.866+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f508c06a270 con 0x7f508c150e70 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.868+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5080043020 con 0x7f508c150e70 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f5073fff640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5078076300 0x7f5078078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f50800b7b30 con 0x7f508c150e70 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f5073fff640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f507807c2b0 0x7f507807e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5080046e50 con 0x7f507807c2b0 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f50800b7fc0 con 0x7f508c150e70 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f508b7fe640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5078076300 0x7f5078078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f508bfff640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f507807c2b0 0x7f507807e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:39.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.869+0000 7f508bfff640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f507807c2b0 0x7f507807e700 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.870+0000 7f508b7fe640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5078076300 0x7f5078078810 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f507c007fd0 tx=0x7f507c0012e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:39.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.870+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== osd.0 v2:172.21.15.46:6800/1118951116 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5080046e50 con 0x7f507807c2b0 2024-05-09T03:21:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.883+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 --> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f508c13cd10 con 0x7f507807c2b0 2024-05-09T03:21:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5073fff640 1 -- 172.21.15.46:0/3733112988 <== osd.0 v2:172.21.15.46:6800/1118951116 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f508c13cd10 con 0x7f507807c2b0 2024-05-09T03:21:39.885 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:39.885 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:39.886 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f507807c2b0 msgr2=0x7f507807e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:6800/1118951116,v1:172.21.15.46:6801/1118951116] conn(0x7f507807c2b0 0x7f507807e700 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5078076300 msgr2=0x7f5078078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f5078076300 0x7f5078078810 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f507c007fd0 tx=0x7f507c0012e0 comp rx=0 tx=0).stop 2024-05-09T03:21:39.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c150e70 msgr2=0x7f508c152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 --2- 172.21.15.46:0/3733112988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f508c150e70 0x7f508c152750 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f5080009b80 tx=0x7f5080004730 comp rx=0 tx=0).stop 2024-05-09T03:21:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f508bfff640 1 -- 172.21.15.46:0/3733112988 reap_dead start 2024-05-09T03:21:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 shutdown_connections 2024-05-09T03:21:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 >> 172.21.15.46:0/3733112988 conn(0x7f508c06d0a0 msgr2=0x7f508c06f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 shutdown_connections 2024-05-09T03:21:39.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:39.884+0000 7f5092108640 1 -- 172.21.15.46:0/3733112988 wait complete. 2024-05-09T03:21:39.897 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-09T03:21:40.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.012+0000 7f2f0190d640 1 Processor -- start 2024-05-09T03:21:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.012+0000 7f2f0190d640 1 -- start start 2024-05-09T03:21:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2f0190d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2efc0b3650 0x7f2efc0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2f0190d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2efc0b0d50 0x7f2efc169ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2f0190d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2efc0b7d60 con 0x7f2efc06cd70 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2f0190d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2efc0b77a0 con 0x7f2efc0b0d50 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2f0190d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2efc0b7a80 con 0x7f2efc0b3650 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2efb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2efc0b0d50 0x7f2efc169ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2efb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2efc0b0d50 0x7f2efc169ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54510/0 (socket says 172.21.15.46:54510) 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2efb7fe640 1 -- 172.21.15.46:0/4235600121 learned_addr learned my addr 172.21.15.46:0/4235600121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.013+0000 7f2efaffd640 1 --2- 172.21.15.46:0/4235600121 >> v2:172.21.15.88:3300/0 conn(0x7f2efc0b3650 0x7f2efc0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.014+0000 7f2efb7fe640 1 -- 172.21.15.46:0/4235600121 >> v2:172.21.15.88:3300/0 conn(0x7f2efc0b3650 msgr2=0x7f2efc0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.014+0000 7f2efb7fe640 1 --2- 172.21.15.46:0/4235600121 >> v2:172.21.15.88:3300/0 conn(0x7f2efc0b3650 0x7f2efc0b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.014+0000 7f2efb7fe640 1 -- 172.21.15.46:0/4235600121 >> v1:172.21.15.88:6789/0 conn(0x7f2efc06cd70 legacy=0x7f2efc06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.014+0000 7f2efb7fe640 1 -- 172.21.15.46:0/4235600121 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2efc0af540 con 0x7f2efc0b0d50 2024-05-09T03:21:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.014+0000 7f2efb7fe640 1 --2- 172.21.15.46:0/4235600121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc169ec0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0746f0 tx=0x7f2ef402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91dd64e0f4a0ad34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.014+0000 7f2ef9ffb640 1 -- 172.21.15.46:0/4235600121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4036070 con 0x7f2efc0b0d50 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2ef9ffb640 1 -- 172.21.15.46:0/4235600121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ef4031600 con 0x7f2efc0b0d50 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2ef9ffb640 1 -- 172.21.15.46:0/4235600121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40319d0 con 0x7f2efc0b0d50 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 -- 172.21.15.46:0/4235600121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 msgr2=0x7f2efc169ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 --2- 172.21.15.46:0/4235600121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc169ec0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0746f0 tx=0x7f2ef402a610 comp rx=0 tx=0).stop 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 -- 172.21.15.46:0/4235600121 shutdown_connections 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 --2- 172.21.15.46:0/4235600121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc169ec0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 --2- 172.21.15.46:0/4235600121 >> v2:172.21.15.88:3300/0 conn(0x7f2efc0b3650 0x7f2efc0b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 -- 172.21.15.46:0/4235600121 >> 172.21.15.46:0/4235600121 conn(0x7f2efc073ca0 msgr2=0x7f2efc09b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 -- 172.21.15.46:0/4235600121 shutdown_connections 2024-05-09T03:21:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.015+0000 7f2f0190d640 1 -- 172.21.15.46:0/4235600121 wait complete. 2024-05-09T03:21:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.017+0000 7f2f0190d640 1 Processor -- start 2024-05-09T03:21:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.017+0000 7f2f0190d640 1 -- start start 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2f0190d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2efc06cd70 0x7f2efc163310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2f0190d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc161530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2f0190d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0b3650 0x7f2efc161f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2f0190d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc0b2a90 con 0x7f2efc0b0d50 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2f0190d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2efc160580 con 0x7f2efc0b3650 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2f0190d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2efc162ae0 con 0x7f2efc06cd70 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc161530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc161530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54524/0 (socket says 172.21.15.46:54524) 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 -- 172.21.15.46:0/3926221744 learned_addr learned my addr 172.21.15.46:0/3926221744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efb7fe640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0b3650 0x7f2efc161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efaffd640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2efc06cd70 0x7f2efc163310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 -- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2efc06cd70 msgr2=0x7f2efc163310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2efc06cd70 0x7f2efc163310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 -- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0b3650 msgr2=0x7f2efc161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0b3650 0x7f2efc161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 -- 172.21.15.46:0/3926221744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2efc0cc5a0 con 0x7f2efc0b0d50 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.019+0000 7f2efa7fc640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc161530 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000c370 tx=0x7f2ef000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.020+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0017070 con 0x7f2efc0b0d50 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.020+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ef00052d0 con 0x7f2efc0b0d50 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.020+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2efc0cc2c0 con 0x7f2efc0b0d50 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.020+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef0003810 con 0x7f2efc0b0d50 2024-05-09T03:21:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.020+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2efc160d30 con 0x7f2efc0b0d50 2024-05-09T03:21:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.021+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2efc068820 con 0x7f2efc0b0d50 2024-05-09T03:21:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.021+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2ef001a070 con 0x7f2efc0b0d50 2024-05-09T03:21:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.022+0000 7f2ee3fff640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ee8076300 0x7f2ee8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.022+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f2ef0096610 con 0x7f2efc0b0d50 2024-05-09T03:21:40.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.022+0000 7f2efaffd640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ee8076300 0x7f2ee8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.023+0000 7f2efaffd640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ee8076300 0x7f2ee8078810 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2efc074170 tx=0x7f2eec009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.027+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2efc068820 con 0x7f2efc0b0d50 2024-05-09T03:21:40.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.219+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f2efc163850 con 0x7f2ee8076300 2024-05-09T03:21:40.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.220+0000 7f2ee3fff640 1 -- 172.21.15.46:0/3926221744 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2efc163850 con 0x7f2ee8076300 2024-05-09T03:21:40.222 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2024-05-09T03:21:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ee8076300 msgr2=0x7f2ee8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ee8076300 0x7f2ee8078810 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f2efc074170 tx=0x7f2eec009190 comp rx=0 tx=0).stop 2024-05-09T03:21:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 msgr2=0x7f2efc161530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc161530 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f2ef000c370 tx=0x7f2ef000c830 comp rx=0 tx=0).stop 2024-05-09T03:21:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 shutdown_connections 2024-05-09T03:21:40.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2efc0b3650 0x7f2efc161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2ee8076300 0x7f2ee8078810 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2efc0b0d50 0x7f2efc161530 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 --2- 172.21.15.46:0/3926221744 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2efc06cd70 0x7f2efc163310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 >> 172.21.15.46:0/3926221744 conn(0x7f2efc073ca0 msgr2=0x7f2efc06fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 shutdown_connections 2024-05-09T03:21:40.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.222+0000 7f2f0190d640 1 -- 172.21.15.46:0/3926221744 wait complete. 2024-05-09T03:21:40.235 INFO:tasks.ceph:Scrubbing osd.1 2024-05-09T03:21:40.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:40.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.353+0000 7f81b7ba8640 1 Processor -- start 2024-05-09T03:21:40.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.353+0000 7f81b7ba8640 1 -- start start 2024-05-09T03:21:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b7ba8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81b00b2da0 0x7f81b0070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b7ba8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b0070770 0x7f81b00b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b7ba8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b0070b80 con 0x7f81b00d16b0 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b7ba8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f81b00d2db0 con 0x7f81b00b2da0 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b7ba8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81b00d3090 con 0x7f81b0070770 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81b00b2da0 0x7f81b0070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f81b00b2da0 0x7f81b0070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54532/0 (socket says 172.21.15.46:54532) 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 -- 172.21.15.46:0/2956163874 learned_addr learned my addr 172.21.15.46:0/2956163874 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b6145640 1 --2- 172.21.15.46:0/2956163874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b0070770 0x7f81b00b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 -- 172.21.15.46:0/2956163874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b0070770 msgr2=0x7f81b00b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 --2- 172.21.15.46:0/2956163874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b0070770 0x7f81b00b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 -- 172.21.15.46:0/2956163874 >> v1:172.21.15.88:6789/0 conn(0x7f81b00d16b0 legacy=0x7f81b00d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.355+0000 7f81b5143640 1 -- 172.21.15.46:0/2956163874 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b00d3370 con 0x7f81b00b2da0 2024-05-09T03:21:40.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.356+0000 7f81b5143640 1 --2- 172.21.15.46:0/2956163874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b0070230 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009870 tx=0x7f81ac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa96cbf59a1dd2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.356+0000 7f81b4942640 1 -- 172.21.15.46:0/2956163874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac037070 con 0x7f81b00b2da0 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.356+0000 7f81b4942640 1 -- 172.21.15.46:0/2956163874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ac032440 con 0x7f81b00b2da0 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.356+0000 7f81b4942640 1 -- 172.21.15.46:0/2956163874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac032810 con 0x7f81b00b2da0 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2956163874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 msgr2=0x7f81b0070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 --2- 172.21.15.46:0/2956163874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b0070230 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009870 tx=0x7f81ac02a270 comp rx=0 tx=0).stop 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2956163874 shutdown_connections 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 --2- 172.21.15.46:0/2956163874 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b0070770 0x7f81b00b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 --2- 172.21.15.46:0/2956163874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b0070230 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2956163874 >> 172.21.15.46:0/2956163874 conn(0x7f81b00d3ae0 msgr2=0x7f81b00d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2956163874 shutdown_connections 2024-05-09T03:21:40.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.357+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2956163874 wait complete. 2024-05-09T03:21:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.359+0000 7f81b7ba8640 1 Processor -- start 2024-05-09T03:21:40.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.359+0000 7f81b7ba8640 1 -- start start 2024-05-09T03:21:40.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.360+0000 7f81b7ba8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b0070770 0x7f81b006e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.360+0000 7f81b7ba8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.360+0000 7f81b7ba8640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b00d16b0 0x7f81b006efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.360+0000 7f81b7ba8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f81b00b7c80 con 0x7f81b00b2da0 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.360+0000 7f81b7ba8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81b00b6ed0 con 0x7f81b0070770 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.360+0000 7f81b7ba8640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f81b00b7e00 con 0x7f81b00d16b0 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b006ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54542/0 (socket says 172.21.15.46:54542) 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 -- 172.21.15.46:0/2975129123 learned_addr learned my addr 172.21.15.46:0/2975129123 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5944640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b0070770 0x7f81b006e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 -- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b00d16b0 msgr2=0x7f81b006efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b6145640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b00d16b0 0x7f81b006efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f81b00d16b0 0x7f81b006efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 -- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b0070770 msgr2=0x7f81b006e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b0070770 0x7f81b006e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 -- 172.21.15.46:0/2975129123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b014e370 con 0x7f81b00b2da0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5944640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81b0070770 0x7f81b006e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b5143640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b006ea80 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f81ac000c00 tx=0x7f81ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac037040 con 0x7f81b00b2da0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ac042070 con 0x7f81b00b2da0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b0157840 con 0x7f81b00b2da0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac030880 con 0x7f81b00b2da0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.361+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b0157e00 con 0x7f81b00b2da0 2024-05-09T03:21:40.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.362+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81b006a270 con 0x7f81b00b2da0 2024-05-09T03:21:40.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.363+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81ac030a20 con 0x7f81b00b2da0 2024-05-09T03:21:40.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.363+0000 7f81a27fc640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8194076300 0x7f8194078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f81ac0b7c40 con 0x7f81b00b2da0 2024-05-09T03:21:40.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81b5944640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8194076300 0x7f8194078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81a27fc640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f819407c2b0 0x7f819407e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81ac03be50 con 0x7f819407c2b0 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f81ac0ba340 con 0x7f81b00b2da0 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81b6145640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f819407c2b0 0x7f819407e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81b5944640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8194076300 0x7f8194078810 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f81a4007fd0 tx=0x7f81a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.364+0000 7f81b6145640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f819407c2b0 0x7f819407e700 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.365+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== osd.1 v2:172.21.15.46:6824/1118146182 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81ac03be50 con 0x7f819407c2b0 2024-05-09T03:21:40.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.378+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 --> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f81b014e740 con 0x7f819407c2b0 2024-05-09T03:21:40.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.378+0000 7f81a27fc640 1 -- 172.21.15.46:0/2975129123 <== osd.1 v2:172.21.15.46:6824/1118146182 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f81b014e740 con 0x7f819407c2b0 2024-05-09T03:21:40.380 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:40.380 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:40.380 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:40.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f819407c2b0 msgr2=0x7f819407e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:6824/1118146182,v1:172.21.15.46:6825/1118146182] conn(0x7f819407c2b0 0x7f819407e700 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8194076300 msgr2=0x7f8194078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f8194076300 0x7f8194078810 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f81a4007fd0 tx=0x7f81a4009040 comp rx=0 tx=0).stop 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 msgr2=0x7f81b006ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 --2- 172.21.15.46:0/2975129123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f81b00b2da0 0x7f81b006ea80 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f81ac000c00 tx=0x7f81ac0046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b6145640 1 -- 172.21.15.46:0/2975129123 reap_dead start 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 shutdown_connections 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 >> 172.21.15.46:0/2975129123 conn(0x7f81b00d3ae0 msgr2=0x7f81b00cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 shutdown_connections 2024-05-09T03:21:40.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.379+0000 7f81b7ba8640 1 -- 172.21.15.46:0/2975129123 wait complete. 2024-05-09T03:21:40.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-09T03:21:40.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.507+0000 7f90c8786640 1 Processor -- start 2024-05-09T03:21:40.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.507+0000 7f90c8786640 1 -- start start 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c8786640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f90c00bb770 0x7f90c00d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c8786640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c01642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c8786640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90c0164900 con 0x7f90c00d2a70 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c8786640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f90c00d4bb0 con 0x7f90c00bb770 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c8786640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f90c0164be0 con 0x7f90c0163eb0 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c6d23640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90c00d2a70 0x7f90c00d2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50534/0 (socket says 172.21.15.46:50534) 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.508+0000 7f90c6d23640 1 -- 172.21.15.46:0/3766995600 learned_addr learned my addr 172.21.15.46:0/3766995600 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c6522640 1 --2- 172.21.15.46:0/3766995600 >> v2:172.21.15.88:3300/0 conn(0x7f90c00bb770 0x7f90c00d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c5d21640 1 --2- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c01642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c5520640 1 -- 172.21.15.46:0/3766995600 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208078790 0 0) 0x7f90c0164900 con 0x7f90c00d2a70 2024-05-09T03:21:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c5520640 1 -- 172.21.15.46:0/3766995600 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90ac003520 con 0x7f90c00d2a70 2024-05-09T03:21:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c5520640 1 -- 172.21.15.46:0/3766995600 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 677347109 0 0) 0x7f90ac003520 con 0x7f90c00d2a70 2024-05-09T03:21:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c6522640 1 -- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 msgr2=0x7f90c01642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c6522640 1 --2- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c01642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c6522640 1 -- 172.21.15.46:0/3766995600 >> v1:172.21.15.46:6789/0 conn(0x7f90c00d2a70 legacy=0x7f90c00d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.509+0000 7f90c6522640 1 -- 172.21.15.46:0/3766995600 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c00d4840 con 0x7f90c00bb770 2024-05-09T03:21:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.510+0000 7f90c6522640 1 --2- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00d5040 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c6c41dc1d2d252 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.510+0000 7f90c5520640 1 -- 172.21.15.46:0/3766995600 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8001710 con 0x7f90c00bb770 2024-05-09T03:21:40.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.510+0000 7f90c5520640 1 -- 172.21.15.46:0/3766995600 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc007540 con 0x7f90c00bb770 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.510+0000 7f90c5520640 1 -- 172.21.15.46:0/3766995600 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc0078f0 con 0x7f90c00bb770 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 -- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 msgr2=0x7f90c00d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 --2- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00d5040 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc02a5f0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 -- 172.21.15.46:0/3766995600 shutdown_connections 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 --2- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c01642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 --2- 172.21.15.46:0/3766995600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00d5040 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 -- 172.21.15.46:0/3766995600 >> 172.21.15.46:0/3766995600 conn(0x7f90c006e370 msgr2=0x7f90c006e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 -- 172.21.15.46:0/3766995600 shutdown_connections 2024-05-09T03:21:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.511+0000 7f90c8786640 1 -- 172.21.15.46:0/3766995600 wait complete. 2024-05-09T03:21:40.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.513+0000 7f90c8786640 1 Processor -- start 2024-05-09T03:21:40.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.513+0000 7f90c8786640 1 -- start start 2024-05-09T03:21:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c8786640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c8786640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 0x7f90c0151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c8786640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c00cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c8786640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f90c00d0bc0 con 0x7f90c00d2a70 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c8786640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f90c00bbb80 con 0x7f90c00bb770 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c8786640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f90c00d0d40 con 0x7f90c0163eb0 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c5d21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 0x7f90c0151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c5d21640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 0x7f90c0151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54550/0 (socket says 172.21.15.46:54550) 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.514+0000 7f90c5d21640 1 -- 172.21.15.46:0/3819206356 learned_addr learned my addr 172.21.15.46:0/3819206356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c6522640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c6d23640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c00cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c5d21640 1 -- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 msgr2=0x7f90c00cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c5d21640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c00cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c5d21640 1 -- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 msgr2=0x7f90c00ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c5d21640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00ce770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c5d21640 1 -- 172.21.15.46:0/3819206356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c00c97d0 con 0x7f90c00d2a70 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c5d21640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 0x7f90c0151ed0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f90b000e8a0 tx=0x7f90b000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0017070 con 0x7f90c00d2a70 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b000c940 con 0x7f90c00d2a70 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c00cf1f0 con 0x7f90c00d2a70 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b000cd30 con 0x7f90c00d2a70 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.515+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c01eaf00 con 0x7f90c00d2a70 2024-05-09T03:21:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.516+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90c006a270 con 0x7f90c00d2a70 2024-05-09T03:21:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.517+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f90b001a070 con 0x7f90c00d2a70 2024-05-09T03:21:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.518+0000 7f90b6ffd640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a40762d0 0x7f90a40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.518+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f90b0097ab0 con 0x7f90c00d2a70 2024-05-09T03:21:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.518+0000 7f90c6522640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a40762d0 0x7f90a40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.519+0000 7f90c6522640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a40762d0 0x7f90a40787e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f90bc034000 tx=0x7f90bc02f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.522+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f90b0061620 con 0x7f90c00d2a70 2024-05-09T03:21:40.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.714+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f90c00cef80 con 0x7f90a40762d0 2024-05-09T03:21:40.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.715+0000 7f90b6ffd640 1 -- 172.21.15.46:0/3819206356 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f90c00cef80 con 0x7f90a40762d0 2024-05-09T03:21:40.717 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2024-05-09T03:21:40.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a40762d0 msgr2=0x7f90a40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a40762d0 0x7f90a40787e0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f90bc034000 tx=0x7f90bc02f000 comp rx=0 tx=0).stop 2024-05-09T03:21:40.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 msgr2=0x7f90c0151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 0x7f90c0151ed0 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f90b000e8a0 tx=0x7f90b000ed60 comp rx=0 tx=0).stop 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 shutdown_connections 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f90c0163eb0 0x7f90c00cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f90a40762d0 0x7f90a40787e0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90c00d2a70 0x7f90c0151ed0 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 --2- 172.21.15.46:0/3819206356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c00bb770 0x7f90c00ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 >> 172.21.15.46:0/3819206356 conn(0x7f90c006e370 msgr2=0x7f90c00b69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 shutdown_connections 2024-05-09T03:21:40.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.717+0000 7f90c8786640 1 -- 172.21.15.46:0/3819206356 wait complete. 2024-05-09T03:21:40.730 INFO:tasks.ceph:Scrubbing osd.2 2024-05-09T03:21:40.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.849+0000 7f0193372640 1 Processor -- start 2024-05-09T03:21:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.850+0000 7f0193372640 1 -- start start 2024-05-09T03:21:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f0193372640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f018c0b5140 0x7f018c0b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f0193372640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c06e420 0x7f018c0b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f0193372640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f018c0b8480 con 0x7f018c0b8070 2024-05-09T03:21:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f0193372640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f018c0b9b60 con 0x7f018c0b5140 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f0193372640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f018c0b5550 con 0x7f018c06e420 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f019110e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f018c0b5140 0x7f018c0b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f019110e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f018c0b5140 0x7f018c0b7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54566/0 (socket says 172.21.15.46:54566) 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f019110e640 1 -- 172.21.15.46:0/396663980 learned_addr learned my addr 172.21.15.46:0/396663980 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.851+0000 7f019190f640 1 --2- 172.21.15.46:0/396663980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c06e420 0x7f018c0b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f019110e640 1 -- 172.21.15.46:0/396663980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c06e420 msgr2=0x7f018c0b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f019110e640 1 --2- 172.21.15.46:0/396663980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c06e420 0x7f018c0b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f019110e640 1 -- 172.21.15.46:0/396663980 >> v1:172.21.15.88:6789/0 conn(0x7f018c0b8070 legacy=0x7f018c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f019110e640 1 -- 172.21.15.46:0/396663980 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c0b9270 con 0x7f018c0b5140 2024-05-09T03:21:40.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f019110e640 1 --2- 172.21.15.46:0/396663980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0b7b30 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f0188009870 tx=0x7f018802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f0fcabf6f7d068 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f017ffff640 1 -- 172.21.15.46:0/396663980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188037070 con 0x7f018c0b5140 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.852+0000 7f017ffff640 1 -- 172.21.15.46:0/396663980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0188032440 con 0x7f018c0b5140 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f017ffff640 1 -- 172.21.15.46:0/396663980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188032810 con 0x7f018c0b5140 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 -- 172.21.15.46:0/396663980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 msgr2=0x7f018c0b7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 --2- 172.21.15.46:0/396663980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0b7b30 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f0188009870 tx=0x7f018802a270 comp rx=0 tx=0).stop 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 -- 172.21.15.46:0/396663980 shutdown_connections 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 --2- 172.21.15.46:0/396663980 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c06e420 0x7f018c0b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 --2- 172.21.15.46:0/396663980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0b7b30 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 -- 172.21.15.46:0/396663980 >> 172.21.15.46:0/396663980 conn(0x7f018c0d5e90 msgr2=0x7f018c0d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 -- 172.21.15.46:0/396663980 shutdown_connections 2024-05-09T03:21:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.853+0000 7f0193372640 1 -- 172.21.15.46:0/396663980 wait complete. 2024-05-09T03:21:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.855+0000 7f0193372640 1 Processor -- start 2024-05-09T03:21:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.855+0000 7f0193372640 1 -- start start 2024-05-09T03:21:40.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f0193372640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f018c06e420 0x7f018c15ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f0193372640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f0193372640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c0b8070 0x7f018c0cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f0193372640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0b97f0 con 0x7f018c0b5140 2024-05-09T03:21:40.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f0193372640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f018c0b9510 con 0x7f018c06e420 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f0193372640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f018c0ba530 con 0x7f018c0b8070 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.856+0000 7f019090d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54582/0 (socket says 172.21.15.46:54582) 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 -- 172.21.15.46:0/3458772209 learned_addr learned my addr 172.21.15.46:0/3458772209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019110e640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f018c06e420 0x7f018c15ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019190f640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c0b8070 0x7f018c0cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 -- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c0b8070 msgr2=0x7f018c0cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f018c0b8070 0x7f018c0cf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 -- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f018c06e420 msgr2=0x7f018c15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f018c06e420 0x7f018c15ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 -- 172.21.15.46:0/3458772209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f018c15f190 con 0x7f018c0b5140 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f019090d640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0cefb0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f018000c3a0 tx=0x7f018000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180017070 con 0x7f018c0b5140 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01800052d0 con 0x7f018c0b5140 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f018c0cd2d0 con 0x7f018c0b5140 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.857+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180003830 con 0x7f018c0b5140 2024-05-09T03:21:40.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.858+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f018c0cd890 con 0x7f018c0b5140 2024-05-09T03:21:40.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.858+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f018c06a270 con 0x7f018c0b5140 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.859+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f018001a070 con 0x7f018c0b5140 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f017dffb640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01700762d0 0x7f01700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f0180096730 con 0x7f018c0b5140 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f017dffb640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f017007c280 0x7f017007e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f019110e640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01700762d0 0x7f01700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f017007eda0 con 0x7f017007c280 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f018009d050 con 0x7f018c0b5140 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f019190f640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f017007c280 0x7f017007e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f019110e640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01700762d0 0x7f01700787e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0188034000 tx=0x7f018802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.860+0000 7f019190f640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f017007c280 0x7f017007e6d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:40.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.861+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== osd.2 v2:172.21.15.46:6808/2404849282 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f017007eda0 con 0x7f017007c280 2024-05-09T03:21:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.874+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 --> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f018c0cd080 con 0x7f017007c280 2024-05-09T03:21:40.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f017dffb640 1 -- 172.21.15.46:0/3458772209 <== osd.2 v2:172.21.15.46:6808/2404849282 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f018c0cd080 con 0x7f017007c280 2024-05-09T03:21:40.876 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:40.877 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:40.877 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f017007c280 msgr2=0x7f017007e6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:6808/2404849282,v1:172.21.15.46:6809/2404849282] conn(0x7f017007c280 0x7f017007e6d0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01700762d0 msgr2=0x7f01700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f01700762d0 0x7f01700787e0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0188034000 tx=0x7f018802e000 comp rx=0 tx=0).stop 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 msgr2=0x7f018c0cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 --2- 172.21.15.46:0/3458772209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f018c0b5140 0x7f018c0cefb0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f018000c3a0 tx=0x7f018000c860 comp rx=0 tx=0).stop 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f019190f640 1 -- 172.21.15.46:0/3458772209 reap_dead start 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 shutdown_connections 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 >> 172.21.15.46:0/3458772209 conn(0x7f018c0d5e90 msgr2=0x7f018c160210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 shutdown_connections 2024-05-09T03:21:40.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:40.875+0000 7f0193372640 1 -- 172.21.15.46:0/3458772209 wait complete. 2024-05-09T03:21:40.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-09T03:21:41.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.002+0000 7f877a6d1640 1 Processor -- start 2024-05-09T03:21:41.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.002+0000 7f877a6d1640 1 -- start start 2024-05-09T03:21:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f877a6d1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87740b26b0 0x7f87740cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f877a6d1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87740b41d0 con 0x7f87740b4c20 2024-05-09T03:21:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f877a6d1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f877406ed10 con 0x7f87740b22a0 2024-05-09T03:21:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f877a6d1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f877406f920 con 0x7f87740b26b0 2024-05-09T03:21:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f8778c6e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87740b26b0 0x7f87740cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f87737fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f87740b4c20 0x7f87740cd350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50540/0 (socket says 172.21.15.46:50540) 2024-05-09T03:21:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.003+0000 7f87737fe640 1 -- 172.21.15.46:0/96409474 learned_addr learned my addr 172.21.15.46:0/96409474 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.004+0000 7f8778c6e640 1 -- 172.21.15.46:0/96409474 >> v1:172.21.15.88:6789/0 conn(0x7f87740b22a0 legacy=0x7f87740b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.004+0000 7f8778c6e640 1 -- 172.21.15.46:0/96409474 >> v1:172.21.15.46:6789/0 conn(0x7f87740b4c20 legacy=0x7f87740cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.004+0000 7f8772ffd640 1 -- 172.21.15.46:0/96409474 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3876735237 0 0) 0x7f87740b41d0 con 0x7f87740b4c20 2024-05-09T03:21:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.004+0000 7f8778c6e640 1 -- 172.21.15.46:0/96409474 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877406eff0 con 0x7f87740b26b0 2024-05-09T03:21:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.004+0000 7f8778c6e640 1 --2- 172.21.15.46:0/96409474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87740b26b0 0x7f87740cda60 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f876c009d90 tx=0x7f876c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2220dc182e208d3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f8772ffd640 1 -- 172.21.15.46:0/96409474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c007740 con 0x7f87740b26b0 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f8772ffd640 1 -- 172.21.15.46:0/96409474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876c0078e0 con 0x7f87740b26b0 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f8772ffd640 1 -- 172.21.15.46:0/96409474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c007cb0 con 0x7f87740b26b0 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f877a6d1640 1 -- 172.21.15.46:0/96409474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87740b26b0 msgr2=0x7f87740cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f877a6d1640 1 --2- 172.21.15.46:0/96409474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87740b26b0 0x7f87740cda60 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f876c009d90 tx=0x7f876c02a6f0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f877a6d1640 1 -- 172.21.15.46:0/96409474 shutdown_connections 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f877a6d1640 1 --2- 172.21.15.46:0/96409474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87740b26b0 0x7f87740cda60 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.005+0000 7f877a6d1640 1 -- 172.21.15.46:0/96409474 >> 172.21.15.46:0/96409474 conn(0x7f877409b7d0 msgr2=0x7f877409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.006+0000 7f877a6d1640 1 -- 172.21.15.46:0/96409474 shutdown_connections 2024-05-09T03:21:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.006+0000 7f877a6d1640 1 -- 172.21.15.46:0/96409474 wait complete. 2024-05-09T03:21:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.007+0000 7f877a6d1640 1 Processor -- start 2024-05-09T03:21:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.008+0000 7f877a6d1640 1 -- start start 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f877a6d1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87740b22a0 0x7f8774152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f877a6d1640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87740b26b0 0x7f877415b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f877a6d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 0x7f8774155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f877a6d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f877406ea00 con 0x7f8774155150 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f877a6d1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f877406f5b0 con 0x7f87740b22a0 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f877a6d1640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f87740d3460 con 0x7f87740b26b0 2024-05-09T03:21:41.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 0x7f8774155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 0x7f8774155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54594/0 (socket says 172.21.15.46:54594) 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 -- 172.21.15.46:0/2476506140 learned_addr learned my addr 172.21.15.46:0/2476506140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8773fff640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87740b22a0 0x7f8774152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f87737fe640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87740b26b0 0x7f877415b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 -- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87740b26b0 msgr2=0x7f877415b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87740b26b0 0x7f877415b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 -- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87740b22a0 msgr2=0x7f8774152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.009+0000 7f8778c6e640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87740b22a0 0x7f8774152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f8778c6e640 1 -- 172.21.15.46:0/2476506140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8774155aa0 con 0x7f8774155150 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f87737fe640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87740b26b0 0x7f877415b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f8778c6e640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 0x7f8774155560 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f876c009eb0 tx=0x7f876c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c040070 con 0x7f8774155150 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f876c031c40 con 0x7f8774155150 2024-05-09T03:21:41.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8774157bf0 con 0x7f8774155150 2024-05-09T03:21:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c003990 con 0x7f8774155150 2024-05-09T03:21:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.010+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f877415c730 con 0x7f8774155150 2024-05-09T03:21:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.011+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877415c970 con 0x7f8774155150 2024-05-09T03:21:41.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.013+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f876c039040 con 0x7f8774155150 2024-05-09T03:21:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.013+0000 7f8770ff9640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87600762d0 0x7f87600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.013+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f876c0b85c0 con 0x7f8774155150 2024-05-09T03:21:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.013+0000 7f8773fff640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87600762d0 0x7f87600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.014+0000 7f8773fff640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87600762d0 0x7f87600787e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8764000f60 tx=0x7f8764009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.018+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8774155aa0 con 0x7f8774155150 2024-05-09T03:21:41.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.207+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f877415c1e0 con 0x7f87600762d0 2024-05-09T03:21:41.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.208+0000 7f8770ff9640 1 -- 172.21.15.46:0/2476506140 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f877415c1e0 con 0x7f87600762d0 2024-05-09T03:21:41.210 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87600762d0 msgr2=0x7f87600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87600762d0 0x7f87600787e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f8764000f60 tx=0x7f8764009040 comp rx=0 tx=0).stop 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 msgr2=0x7f8774155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 0x7f8774155560 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f876c009eb0 tx=0x7f876c004a40 comp rx=0 tx=0).stop 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 shutdown_connections 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f87600762d0 0x7f87600787e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8774155150 0x7f8774155560 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f87740b26b0 0x7f877415b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 --2- 172.21.15.46:0/2476506140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87740b22a0 0x7f8774152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 >> 172.21.15.46:0/2476506140 conn(0x7f877409b7d0 msgr2=0x7f877409bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 shutdown_connections 2024-05-09T03:21:41.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.210+0000 7f877a6d1640 1 -- 172.21.15.46:0/2476506140 wait complete. 2024-05-09T03:21:41.223 INFO:tasks.ceph:Scrubbing osd.3 2024-05-09T03:21:41.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.341+0000 7ff63babd640 1 Processor -- start 2024-05-09T03:21:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.341+0000 7ff63babd640 1 -- start start 2024-05-09T03:21:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff63babd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6340d1fd0 0x7ff6340b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff63babd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff6340b2cf0 0x7ff63416a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff63babd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6340d2620 con 0x7ff63406de80 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff63babd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff6340d36c0 con 0x7ff6340d1fd0 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff63babd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff6340d23e0 con 0x7ff6340b2cf0 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff639058640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6340d1fd0 0x7ff6340b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.342+0000 7ff639859640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff63406de80 0x7ff6340d0630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50542/0 (socket says 172.21.15.46:50542) 2024-05-09T03:21:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff639859640 1 -- 172.21.15.46:0/1574691064 learned_addr learned my addr 172.21.15.46:0/1574691064 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff63a05a640 1 --2- 172.21.15.46:0/1574691064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff6340b2cf0 0x7ff63416a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff639058640 1 -- 172.21.15.46:0/1574691064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff6340b2cf0 msgr2=0x7ff63416a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff639058640 1 --2- 172.21.15.46:0/1574691064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff6340b2cf0 0x7ff63416a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff639058640 1 -- 172.21.15.46:0/1574691064 >> v1:172.21.15.46:6789/0 conn(0x7ff63406de80 legacy=0x7ff6340d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff639058640 1 -- 172.21.15.46:0/1574691064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6340d2900 con 0x7ff6340d1fd0 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.343+0000 7ff638857640 1 -- 172.21.15.46:0/1574691064 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232918988 0 0) 0x7ff6340d2620 con 0x7ff63406de80 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff639058640 1 --2- 172.21.15.46:0/1574691064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340d1fd0 0x7ff6340b27b0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7ff6240097a0 tx=0x7ff62402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96221676575b874c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff638857640 1 -- 172.21.15.46:0/1574691064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624037070 con 0x7ff6340d1fd0 2024-05-09T03:21:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff638857640 1 -- 172.21.15.46:0/1574691064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff624032440 con 0x7ff6340d1fd0 2024-05-09T03:21:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff638857640 1 -- 172.21.15.46:0/1574691064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624032810 con 0x7ff6340d1fd0 2024-05-09T03:21:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff63babd640 1 -- 172.21.15.46:0/1574691064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340d1fd0 msgr2=0x7ff6340b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff63babd640 1 --2- 172.21.15.46:0/1574691064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340d1fd0 0x7ff6340b27b0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7ff6240097a0 tx=0x7ff62402a180 comp rx=0 tx=0).stop 2024-05-09T03:21:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff63babd640 1 -- 172.21.15.46:0/1574691064 shutdown_connections 2024-05-09T03:21:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff63babd640 1 --2- 172.21.15.46:0/1574691064 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff6340b2cf0 0x7ff63416a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff63babd640 1 --2- 172.21.15.46:0/1574691064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340d1fd0 0x7ff6340b27b0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.344+0000 7ff63babd640 1 -- 172.21.15.46:0/1574691064 >> 172.21.15.46:0/1574691064 conn(0x7ff6340d3bd0 msgr2=0x7ff6340d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.345+0000 7ff63babd640 1 -- 172.21.15.46:0/1574691064 shutdown_connections 2024-05-09T03:21:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.345+0000 7ff63babd640 1 -- 172.21.15.46:0/1574691064 wait complete. 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.346+0000 7ff63babd640 1 Processor -- start 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.346+0000 7ff63babd640 1 -- start start 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63babd640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff63406de80 0x7ff63415d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63babd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340b2cf0 0x7ff6340a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63babd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6340d1fd0 0x7ff63415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63babd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6340d2ec0 con 0x7ff6340b2cf0 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63babd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff6340d3230 con 0x7ff6340d1fd0 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63babd640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff6340d0d40 con 0x7ff63406de80 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340b2cf0 0x7ff6340a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340b2cf0 0x7ff6340a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54612/0 (socket says 172.21.15.46:54612) 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 -- 172.21.15.46:0/2609850777 learned_addr learned my addr 172.21.15.46:0/2609850777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639859640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff63406de80 0x7ff63415d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63a05a640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6340d1fd0 0x7ff63415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 -- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff63406de80 msgr2=0x7ff63415d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff63406de80 0x7ff63415d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 -- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6340d1fd0 msgr2=0x7ff63415dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6340d1fd0 0x7ff63415dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 -- 172.21.15.46:0/2609850777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6341f66d0 con 0x7ff6340b2cf0 2024-05-09T03:21:41.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff63a05a640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6340d1fd0 0x7ff63415dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.348+0000 7ff639058640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340b2cf0 0x7ff6340a1150 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7ff6240098c0 tx=0x7ff6240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.349+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624041070 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.349+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62403c930 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.349+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6341f69b0 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.349+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff624037040 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.349+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6341f6f70 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.349+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff63406a270 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.350+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff624044070 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff62a7fc640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6180762d0 0x7ff6180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7ff6240b8460 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff62a7fc640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7ff61807c280 0x7ff61807e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff624032440 con 0x7ff61807c280 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7ff6240b88f0 con 0x7ff6340b2cf0 2024-05-09T03:21:41.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff639859640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6180762d0 0x7ff6180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff63a05a640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7ff61807c280 0x7ff61807e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.351+0000 7ff63a05a640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7ff61807c280 0x7ff61807e6d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.352+0000 7ff639859640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6180762d0 0x7ff6180787e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff630007fd0 tx=0x7ff630007de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.352+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== osd.3 v2:172.21.15.46:6816/3177037259 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff624032440 con 0x7ff61807c280 2024-05-09T03:21:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.365+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 --> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff6340d1250 con 0x7ff61807c280 2024-05-09T03:21:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff62a7fc640 1 -- 172.21.15.46:0/2609850777 <== osd.3 v2:172.21.15.46:6816/3177037259 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff6340d1250 con 0x7ff61807c280 2024-05-09T03:21:41.368 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:41.368 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:41.368 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7ff61807c280 msgr2=0x7ff61807e6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:6816/3177037259,v1:172.21.15.46:6817/3177037259] conn(0x7ff61807c280 0x7ff61807e6d0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6180762d0 msgr2=0x7ff6180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6180762d0 0x7ff6180787e0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7ff630007fd0 tx=0x7ff630007de0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340b2cf0 msgr2=0x7ff6340a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 --2- 172.21.15.46:0/2609850777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6340b2cf0 0x7ff6340a1150 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7ff6240098c0 tx=0x7ff6240046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63a05a640 1 -- 172.21.15.46:0/2609850777 reap_dead start 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 shutdown_connections 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 >> 172.21.15.46:0/2609850777 conn(0x7ff6340d3bd0 msgr2=0x7ff6340b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 shutdown_connections 2024-05-09T03:21:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.366+0000 7ff63babd640 1 -- 172.21.15.46:0/2609850777 wait complete. 2024-05-09T03:21:41.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-09T03:21:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.492+0000 7f1891300640 1 Processor -- start 2024-05-09T03:21:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.492+0000 7f1891300640 1 -- start start 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f1891300640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f188c0b7e70 0x7f188c0b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f1891300640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f188c0d3400 con 0x7f188c0d2e90 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f1891300640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f188c0d4a60 con 0x7f188c1605b0 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f1891300640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f188c0d4d40 con 0x7f188c0b7e70 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f188a7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f188c0b7e70 0x7f188c0b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f188affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f188c0d2e90 0x7f188c0b7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50544/0 (socket says 172.21.15.46:50544) 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f188affd640 1 -- 172.21.15.46:0/46791951 learned_addr learned my addr 172.21.15.46:0/46791951 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f188a7fc640 1 -- 172.21.15.46:0/46791951 >> v1:172.21.15.88:6789/0 conn(0x7f188c1605b0 legacy=0x7f188c0b97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f188a7fc640 1 -- 172.21.15.46:0/46791951 >> v1:172.21.15.46:6789/0 conn(0x7f188c0d2e90 legacy=0x7f188c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.494+0000 7f188a7fc640 1 -- 172.21.15.46:0/46791951 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188c0d5020 con 0x7f188c0b7e70 2024-05-09T03:21:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.495+0000 7f188a7fc640 1 --2- 172.21.15.46:0/46791951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b7e70 0x7f188c0b92a0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f18740097a0 tx=0x7f187402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ef4651dc26a3e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.495+0000 7f1889ffb640 1 -- 172.21.15.46:0/46791951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874037070 con 0x7f188c0b7e70 2024-05-09T03:21:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.495+0000 7f1889ffb640 1 -- 172.21.15.46:0/46791951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1874032440 con 0x7f188c0b7e70 2024-05-09T03:21:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.495+0000 7f1889ffb640 1 -- 172.21.15.46:0/46791951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874032810 con 0x7f188c0b7e70 2024-05-09T03:21:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 -- 172.21.15.46:0/46791951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b7e70 msgr2=0x7f188c0b92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 --2- 172.21.15.46:0/46791951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b7e70 0x7f188c0b92a0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f18740097a0 tx=0x7f187402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 -- 172.21.15.46:0/46791951 shutdown_connections 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 --2- 172.21.15.46:0/46791951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c0b7e70 0x7f188c0b92a0 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 -- 172.21.15.46:0/46791951 >> 172.21.15.46:0/46791951 conn(0x7f188c08d390 msgr2=0x7f188c08d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 -- 172.21.15.46:0/46791951 shutdown_connections 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.496+0000 7f1891300640 1 -- 172.21.15.46:0/46791951 wait complete. 2024-05-09T03:21:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.498+0000 7f1891300640 1 Processor -- start 2024-05-09T03:21:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.498+0000 7f1891300640 1 -- start start 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f1891300640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0d2e90 0x7f188c157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f1891300640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 0x7f188c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f1891300640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0cf5d0 0x7f188c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f1891300640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f188c0b4d80 con 0x7f188c1605b0 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f1891300640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f188c0b4c00 con 0x7f188c0cf5d0 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f1891300640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f188c0b4fc0 con 0x7f188c0d2e90 2024-05-09T03:21:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f188a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 0x7f188c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f188a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 0x7f188c0cbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54624/0 (socket says 172.21.15.46:54624) 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.499+0000 7f188a7fc640 1 -- 172.21.15.46:0/1455687420 learned_addr learned my addr 172.21.15.46:0/1455687420 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188affd640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0d2e90 0x7f188c157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188a7fc640 1 -- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0d2e90 msgr2=0x7f188c157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188b7fe640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0cf5d0 0x7f188c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188a7fc640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0d2e90 0x7f188c157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188a7fc640 1 -- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0cf5d0 msgr2=0x7f188c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188a7fc640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0cf5d0 0x7f188c159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188a7fc640 1 -- 172.21.15.46:0/1455687420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188c1584d0 con 0x7f188c1605b0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f188a7fc640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 0x7f188c0cbe70 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f18740098c0 tx=0x7f18740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874037040 con 0x7f188c1605b0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1874030770 con 0x7f188c1605b0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188c1587b0 con 0x7f188c1605b0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1874030b40 con 0x7f188c1605b0 2024-05-09T03:21:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.500+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188c158d70 con 0x7f188c1605b0 2024-05-09T03:21:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.501+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f188c06a270 con 0x7f188c1605b0 2024-05-09T03:21:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.502+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1874044020 con 0x7f188c1605b0 2024-05-09T03:21:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.502+0000 7f18837fe640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f186c0762d0 0x7f186c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.502+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f18740b77f0 con 0x7f188c1605b0 2024-05-09T03:21:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.502+0000 7f188affd640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f186c0762d0 0x7f186c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.503+0000 7f188affd640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f186c0762d0 0x7f186c0787e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f187c005880 tx=0x7f187c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.507+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1874032e20 con 0x7f188c1605b0 2024-05-09T03:21:41.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.698+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f188c158260 con 0x7f186c0762d0 2024-05-09T03:21:41.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.699+0000 7f18837fe640 1 -- 172.21.15.46:0/1455687420 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f188c158260 con 0x7f186c0762d0 2024-05-09T03:21:41.701 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 3 to deep-scrub 2024-05-09T03:21:41.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f186c0762d0 msgr2=0x7f186c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f186c0762d0 0x7f186c0787e0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f187c005880 tx=0x7f187c009040 comp rx=0 tx=0).stop 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 msgr2=0x7f188c0cbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 0x7f188c0cbe70 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f18740098c0 tx=0x7f18740046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 shutdown_connections 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f188c0cf5d0 0x7f188c159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f186c0762d0 0x7f186c0787e0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f188c1605b0 0x7f188c0cbe70 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.701+0000 7f1891300640 1 --2- 172.21.15.46:0/1455687420 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f188c0d2e90 0x7f188c157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.702+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 >> 172.21.15.46:0/1455687420 conn(0x7f188c08d390 msgr2=0x7f188c0b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.702+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 shutdown_connections 2024-05-09T03:21:41.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.702+0000 7f1891300640 1 -- 172.21.15.46:0/1455687420 wait complete. 2024-05-09T03:21:41.714 INFO:tasks.ceph:Scrubbing osd.4 2024-05-09T03:21:41.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.832+0000 7f41a95b2640 1 Processor -- start 2024-05-09T03:21:41.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.832+0000 7f41a95b2640 1 -- start start 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a95b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 0x7f41a40b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a95b2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41a40b8480 con 0x7f41a40b8070 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a95b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41a40b9270 con 0x7f41a406e420 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a95b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f41a40b9b60 con 0x7f41a40b5140 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f41a40b8070 0x7f41a406dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50558/0 (socket says 172.21.15.46:50558) 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a27fc640 1 -- 172.21.15.46:0/3579042534 learned_addr learned my addr 172.21.15.46:0/3579042534 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 0x7f41a40b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.833+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641787258 0 0) 0x7f418c001110 con 0x7f41a40b8070 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4190003520 con 0x7f41a40b8070 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1816238348 0 0) 0x7f41a40b9270 con 0x7f41a406e420 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f418c001110 con 0x7f41a406e420 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3872784001 0 0) 0x7f4190003520 con 0x7f41a40b8070 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 msgr2=0x7f41a40b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 0x7f41a40b7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 >> v1:172.21.15.88:6789/0 conn(0x7f41a406e420 legacy=0x7f41a40b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a406e8f0 con 0x7f41a40b8070 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2891352849 0 0) 0x7f418c0018c0 con 0x7f41a40b8070 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 >> v1:172.21.15.46:6789/0 conn(0x7f41a40b8070 legacy=0x7f41a406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4190003c10 0x7f4190004080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 0x7f4190004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4190004d90 0x7f4190005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40b9b60 con 0x7f4190003c10 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f419000bbd0 con 0x7f4190004d90 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f419000bdd0 con 0x7f41a40b5140 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4190003c10 0x7f4190004080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a27fc640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 0x7f4190004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a37fe640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4190004d90 0x7f4190005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a2ffd640 1 -- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 msgr2=0x7f4190004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f41a40b5140 0x7f4190004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a2ffd640 1 -- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4190004d90 msgr2=0x7f4190005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4190004d90 0x7f4190005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.834+0000 7f41a2ffd640 1 -- 172.21.15.46:0/3579042534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4190003520 con 0x7f4190003c10 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.835+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4190003c10 0x7f4190004080 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f419c009fc0 tx=0x7f4190005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c81472304f973d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.835+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c034020 con 0x7f4190003c10 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.835+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419c0305f0 con 0x7f4190003c10 2024-05-09T03:21:41.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.835+0000 7f41a1ffb640 1 -- 172.21.15.46:0/3579042534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c009c10 con 0x7f4190003c10 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a95b2640 1 -- 172.21.15.46:0/3579042534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4190003c10 msgr2=0x7f4190004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a95b2640 1 --2- 172.21.15.46:0/3579042534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4190003c10 0x7f4190004080 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f419c009fc0 tx=0x7f4190005830 comp rx=0 tx=0).stop 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a37fe640 1 -- 172.21.15.46:0/3579042534 reap_dead start 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a95b2640 1 -- 172.21.15.46:0/3579042534 shutdown_connections 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a95b2640 1 -- 172.21.15.46:0/3579042534 >> 172.21.15.46:0/3579042534 conn(0x7f41a40d5e90 msgr2=0x7f41a40d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a95b2640 1 -- 172.21.15.46:0/3579042534 shutdown_connections 2024-05-09T03:21:41.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.836+0000 7f41a95b2640 1 -- 172.21.15.46:0/3579042534 wait complete. 2024-05-09T03:21:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.838+0000 7f41a95b2640 1 Processor -- start 2024-05-09T03:21:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.838+0000 7f41a95b2640 1 -- start start 2024-05-09T03:21:41.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a95b2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4190003c10 0x7f41a4163270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a95b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a4156130 0x7f41a40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a95b2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41a40cf400 0x7f41a41f9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a95b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40b97f0 con 0x7f41a4156130 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a95b2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40b5550 con 0x7f41a40cf400 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a95b2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f41a40b7b30 con 0x7f4190003c10 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a4156130 0x7f41a40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.839+0000 7f41a27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a4156130 0x7f41a40ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54652/0 (socket says 172.21.15.46:54652) 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 -- 172.21.15.46:0/4215753094 learned_addr learned my addr 172.21.15.46:0/4215753094 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4190003c10 0x7f41a4163270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a37fe640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41a40cf400 0x7f41a41f9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 -- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4190003c10 msgr2=0x7f41a4163270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4190003c10 0x7f41a4163270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 -- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41a40cf400 msgr2=0x7f41a41f9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41a40cf400 0x7f41a41f9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 -- 172.21.15.46:0/4215753094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a40cf810 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a37fe640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41a40cf400 0x7f41a41f9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f4190003c10 0x7f41a4163270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a27fc640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a4156130 0x7f41a40ceec0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f418c001050 tx=0x7f418c007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c00f990 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418c018070 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a41613a0 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c00fd30 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.840+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a4162be0 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.841+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41a40acdc0 con 0x7f41a4156130 2024-05-09T03:21:41.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.842+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f418c01d460 con 0x7f41a4156130 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.842+0000 7f419b7fe640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4184076300 0x7f4184078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.842+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f418c097ff0 con 0x7f41a4156130 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f419b7fe640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7f418407c2b0 0x7f418407e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f418407edd0 con 0x7f418407c2b0 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f418c098440 con 0x7f41a4156130 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4184076300 0x7f4184078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f41a37fe640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7f418407c2b0 0x7f418407e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f41a2ffd640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4184076300 0x7f4184078810 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f419c035000 tx=0x7f419c02f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.843+0000 7f41a37fe640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7f418407c2b0 0x7f418407e700 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.844+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== osd.4 v2:172.21.15.88:6824/902225877 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f418407edd0 con 0x7f418407c2b0 2024-05-09T03:21:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.857+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 --> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f41a4160f30 con 0x7f418407c2b0 2024-05-09T03:21:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f419b7fe640 1 -- 172.21.15.46:0/4215753094 <== osd.4 v2:172.21.15.88:6824/902225877 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f41a4160f30 con 0x7f418407c2b0 2024-05-09T03:21:41.859 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:41.860 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:41.860 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7f418407c2b0 msgr2=0x7f418407e700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6824/902225877,v1:172.21.15.88:6825/902225877] conn(0x7f418407c2b0 0x7f418407e700 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4184076300 msgr2=0x7f4184078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4184076300 0x7f4184078810 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f419c035000 tx=0x7f419c02f000 comp rx=0 tx=0).stop 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a4156130 msgr2=0x7f41a40ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 --2- 172.21.15.46:0/4215753094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a4156130 0x7f41a40ceec0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f418c001050 tx=0x7f418c007720 comp rx=0 tx=0).stop 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a37fe640 1 -- 172.21.15.46:0/4215753094 reap_dead start 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 shutdown_connections 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 >> 172.21.15.46:0/4215753094 conn(0x7f41a40d5e90 msgr2=0x7f41a40d10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 shutdown_connections 2024-05-09T03:21:41.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.858+0000 7f41a95b2640 1 -- 172.21.15.46:0/4215753094 wait complete. 2024-05-09T03:21:41.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-09T03:21:41.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.984+0000 7f29c4c89640 1 Processor -- start 2024-05-09T03:21:41.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.984+0000 7f29c4c89640 1 -- start start 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.985+0000 7f29c4c89640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1740 0x7f29c00b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.985+0000 7f29c4c89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29c00cfcf0 0x7f29c009cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.985+0000 7f29c4c89640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c00b9690 con 0x7f29c00d1110 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.985+0000 7f29c4c89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f29c00d2b00 con 0x7f29c00cfcf0 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.985+0000 7f29c4c89640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f29c008cd60 con 0x7f29c00d1740 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29c00cfcf0 0x7f29c009cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f29c00cfcf0 0x7f29c009cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54662/0 (socket says 172.21.15.46:54662) 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 -- 172.21.15.46:0/3872974989 learned_addr learned my addr 172.21.15.46:0/3872974989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29be59c640 1 --2- 172.21.15.46:0/3872974989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1740 0x7f29c00b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 -- 172.21.15.46:0/3872974989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1740 msgr2=0x7f29c00b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 --2- 172.21.15.46:0/3872974989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1740 0x7f29c00b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 -- 172.21.15.46:0/3872974989 >> v1:172.21.15.88:6789/0 conn(0x7f29c00d1110 legacy=0x7f29c00d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.986+0000 7f29bed9d640 1 -- 172.21.15.46:0/3872974989 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c00ca9e0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.987+0000 7f29bed9d640 1 --2- 172.21.15.46:0/3872974989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009cca0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009e50 tx=0x7f29b802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded6b445485321a8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.987+0000 7f29bd59a640 1 -- 172.21.15.46:0/3872974989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8036070 con 0x7f29c00cfcf0 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.987+0000 7f29bd59a640 1 -- 172.21.15.46:0/3872974989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b8031600 con 0x7f29c00cfcf0 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.987+0000 7f29bd59a640 1 -- 172.21.15.46:0/3872974989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b80319d0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 -- 172.21.15.46:0/3872974989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 msgr2=0x7f29c009cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 --2- 172.21.15.46:0/3872974989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009cca0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009e50 tx=0x7f29b802a710 comp rx=0 tx=0).stop 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 -- 172.21.15.46:0/3872974989 shutdown_connections 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 --2- 172.21.15.46:0/3872974989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009cca0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 --2- 172.21.15.46:0/3872974989 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1740 0x7f29c00b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 -- 172.21.15.46:0/3872974989 >> 172.21.15.46:0/3872974989 conn(0x7f29c00d05f0 msgr2=0x7f29c00d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 -- 172.21.15.46:0/3872974989 shutdown_connections 2024-05-09T03:21:41.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.988+0000 7f29c4c89640 1 -- 172.21.15.46:0/3872974989 wait complete. 2024-05-09T03:21:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.990+0000 7f29c4c89640 1 Processor -- start 2024-05-09T03:21:41.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.990+0000 7f29c4c89640 1 -- start start 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29c4c89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29c4c89640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1110 0x7f29c0152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29c4c89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c00d1740 0x7f29c0161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29c4c89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f29c00b93b0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29c4c89640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f29c00d1ca0 con 0x7f29c00d1740 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29c4c89640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f29c009d1e0 con 0x7f29c00d1110 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54666/0 (socket says 172.21.15.46:54666) 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 -- 172.21.15.46:0/2994083947 learned_addr learned my addr 172.21.15.46:0/2994083947 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29bdd9b640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1110 0x7f29c0152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29bed9d640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c00d1740 0x7f29c0161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 -- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1110 msgr2=0x7f29c0152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1110 0x7f29c0152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 -- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c00d1740 msgr2=0x7f29c0161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c00d1740 0x7f29c0161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.991+0000 7f29be59c640 1 -- 172.21.15.46:0/2994083947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c00a0750 con 0x7f29c00cfcf0 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.992+0000 7f29be59c640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009d860 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f29b000d5f0 tx=0x7f29b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.992+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b00125f0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.992+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b0019070 con 0x7f29c00cfcf0 2024-05-09T03:21:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.992+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c00a0a30 con 0x7f29c00cfcf0 2024-05-09T03:21:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.992+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b00129d0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.992+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c00cabb0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.993+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c00b65b0 con 0x7f29c00cfcf0 2024-05-09T03:21:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.994+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f29b0012b70 con 0x7f29c00cfcf0 2024-05-09T03:21:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.994+0000 7f29aeffd640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29a0076300 0x7f29a0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.994+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f29b0098160 con 0x7f29c00cfcf0 2024-05-09T03:21:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.995+0000 7f29bdd9b640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29a0076300 0x7f29a0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.995+0000 7f29bdd9b640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29a0076300 0x7f29a0078810 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f29c00d2460 tx=0x7f29b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:41.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:41.998+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f29c00b65b0 con 0x7f29c00cfcf0 2024-05-09T03:21:42.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.189+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f29c006a270 con 0x7f29a0076300 2024-05-09T03:21:42.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.190+0000 7f29aeffd640 1 -- 172.21.15.46:0/2994083947 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f29c006a270 con 0x7f29a0076300 2024-05-09T03:21:42.192 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 4 to deep-scrub 2024-05-09T03:21:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29a0076300 msgr2=0x7f29a0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29a0076300 0x7f29a0078810 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f29c00d2460 tx=0x7f29b4009040 comp rx=0 tx=0).stop 2024-05-09T03:21:42.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 msgr2=0x7f29c009d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009d860 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f29b000d5f0 tx=0x7f29b000dab0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 shutdown_connections 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f29c00d1740 0x7f29c0161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f29c00d1110 0x7f29c0152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f29a0076300 0x7f29a0078810 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 --2- 172.21.15.46:0/2994083947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29c00cfcf0 0x7f29c009d860 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 >> 172.21.15.46:0/2994083947 conn(0x7f29c00d05f0 msgr2=0x7f29c0072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 shutdown_connections 2024-05-09T03:21:42.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.192+0000 7f29c4c89640 1 -- 172.21.15.46:0/2994083947 wait complete. 2024-05-09T03:21:42.205 INFO:tasks.ceph:Scrubbing osd.5 2024-05-09T03:21:42.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.322+0000 7f3b2719d640 1 Processor -- start 2024-05-09T03:21:42.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.322+0000 7f3b2719d640 1 -- start start 2024-05-09T03:21:42.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2719d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b20164900 0x7f3b2015dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2719d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b200b75f0 0x7f3b200b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2719d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b201616f0 con 0x7f3b201606e0 2024-05-09T03:21:42.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2719d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3b20164470 con 0x7f3b20164900 2024-05-09T03:21:42.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2719d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3b200b7f40 con 0x7f3b200b75f0 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b20164900 0x7f3b2015dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b20164900 0x7f3b2015dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54676/0 (socket says 172.21.15.46:54676) 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 -- 172.21.15.46:0/3004771966 learned_addr learned my addr 172.21.15.46:0/3004771966 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2599a640 1 --2- 172.21.15.46:0/3004771966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b200b75f0 0x7f3b200b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 -- 172.21.15.46:0/3004771966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b200b75f0 msgr2=0x7f3b200b7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 --2- 172.21.15.46:0/3004771966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b200b75f0 0x7f3b200b7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 -- 172.21.15.46:0/3004771966 >> v1:172.21.15.88:6789/0 conn(0x7f3b201606e0 legacy=0x7f3b20160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b25199640 1 -- 172.21.15.46:0/3004771966 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175913159 0 0) 0x7f3b201616f0 con 0x7f3b201606e0 2024-05-09T03:21:42.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.324+0000 7f3b2619b640 1 -- 172.21.15.46:0/3004771966 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b20161410 con 0x7f3b20164900 2024-05-09T03:21:42.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.325+0000 7f3b2619b640 1 --2- 172.21.15.46:0/3004771966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b2015dd60 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009870 tx=0x7f3b1c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab106921588d44d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.325+0000 7f3b25199640 1 -- 172.21.15.46:0/3004771966 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c037070 con 0x7f3b20164900 2024-05-09T03:21:42.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.325+0000 7f3b25199640 1 -- 172.21.15.46:0/3004771966 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1c032440 con 0x7f3b20164900 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.325+0000 7f3b25199640 1 -- 172.21.15.46:0/3004771966 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c032810 con 0x7f3b20164900 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 -- 172.21.15.46:0/3004771966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 msgr2=0x7f3b2015dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 --2- 172.21.15.46:0/3004771966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b2015dd60 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009870 tx=0x7f3b1c02a270 comp rx=0 tx=0).stop 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 -- 172.21.15.46:0/3004771966 shutdown_connections 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 --2- 172.21.15.46:0/3004771966 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b200b75f0 0x7f3b200b7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 --2- 172.21.15.46:0/3004771966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b2015dd60 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 -- 172.21.15.46:0/3004771966 >> 172.21.15.46:0/3004771966 conn(0x7f3b200d27d0 msgr2=0x7f3b200d2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 -- 172.21.15.46:0/3004771966 shutdown_connections 2024-05-09T03:21:42.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.326+0000 7f3b2719d640 1 -- 172.21.15.46:0/3004771966 wait complete. 2024-05-09T03:21:42.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.328+0000 7f3b2719d640 1 Processor -- start 2024-05-09T03:21:42.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.328+0000 7f3b2719d640 1 -- start start 2024-05-09T03:21:42.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2719d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b200b75f0 0x7f3b200cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2719d640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b201606e0 0x7f3b2015d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2719d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b200ba090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2719d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3b200b45f0 con 0x7f3b20164900 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2719d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20164100 con 0x7f3b200b75f0 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2719d640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f3b200b4770 con 0x7f3b201606e0 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2699c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b200ba090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2699c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b200ba090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54692/0 (socket says 172.21.15.46:54692) 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2699c640 1 -- 172.21.15.46:0/435501474 learned_addr learned my addr 172.21.15.46:0/435501474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2619b640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b200b75f0 0x7f3b200cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.329+0000 7f3b2599a640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b201606e0 0x7f3b2015d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2699c640 1 -- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b201606e0 msgr2=0x7f3b2015d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2699c640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f3b201606e0 0x7f3b2015d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2699c640 1 -- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b200b75f0 msgr2=0x7f3b200cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2699c640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b200b75f0 0x7f3b200cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2699c640 1 -- 172.21.15.46:0/435501474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b200cc970 con 0x7f3b20164900 2024-05-09T03:21:42.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2699c640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b200ba090 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c730 tx=0x7f3b1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b18003b70 con 0x7f3b20164900 2024-05-09T03:21:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b18003d30 con 0x7f3b20164900 2024-05-09T03:21:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b200cef10 con 0x7f3b20164900 2024-05-09T03:21:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b180052b0 con 0x7f3b20164900 2024-05-09T03:21:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.330+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b200cd0d0 con 0x7f3b20164900 2024-05-09T03:21:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.331+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b20076c70 con 0x7f3b20164900 2024-05-09T03:21:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.332+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3b18005450 con 0x7f3b20164900 2024-05-09T03:21:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.332+0000 7f3b16ffd640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3b040762d0 0x7f3b040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f3b18097b10 con 0x7f3b20164900 2024-05-09T03:21:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b2619b640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3b040762d0 0x7f3b040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b16ffd640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f3b0407c280 0x7f3b0407e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b0407eda0 con 0x7f3b0407c280 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f3b18097fa0 con 0x7f3b20164900 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b2599a640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f3b0407c280 0x7f3b0407e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b2619b640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3b040762d0 0x7f3b040787e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c034000 tx=0x7f3b1c02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.333+0000 7f3b2599a640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f3b0407c280 0x7f3b0407e6d0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.334+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== osd.5 v2:172.21.15.88:6800/122581025 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b0407eda0 con 0x7f3b0407c280 2024-05-09T03:21:42.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.347+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 --> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3b200ccba0 con 0x7f3b0407c280 2024-05-09T03:21:42.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b16ffd640 1 -- 172.21.15.46:0/435501474 <== osd.5 v2:172.21.15.88:6800/122581025 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3b200ccba0 con 0x7f3b0407c280 2024-05-09T03:21:42.350 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:42.351 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:42.351 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f3b0407c280 msgr2=0x7f3b0407e6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6800/122581025,v1:172.21.15.88:6801/122581025] conn(0x7f3b0407c280 0x7f3b0407e6d0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3b040762d0 msgr2=0x7f3b040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f3b040762d0 0x7f3b040787e0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c034000 tx=0x7f3b1c02e000 comp rx=0 tx=0).stop 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 msgr2=0x7f3b200ba090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 --2- 172.21.15.46:0/435501474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b20164900 0x7f3b200ba090 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c730 tx=0x7f3b1800cbf0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2699c640 1 -- 172.21.15.46:0/435501474 reap_dead start 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 shutdown_connections 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.348+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 >> 172.21.15.46:0/435501474 conn(0x7f3b200d27d0 msgr2=0x7f3b2015ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.349+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 shutdown_connections 2024-05-09T03:21:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.349+0000 7f3b2719d640 1 -- 172.21.15.46:0/435501474 wait complete. 2024-05-09T03:21:42.361 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-09T03:21:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.475+0000 7f2ddac47640 1 Processor -- start 2024-05-09T03:21:42.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.475+0000 7f2ddac47640 1 -- start start 2024-05-09T03:21:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.476+0000 7f2ddac47640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2dd40d32c0 0x7f2dd40b9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.476+0000 7f2ddac47640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd40d4a60 con 0x7f2dd40b7e70 2024-05-09T03:21:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.476+0000 7f2ddac47640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd40d4d40 con 0x7f2dd40d2eb0 2024-05-09T03:21:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.476+0000 7f2ddac47640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2dd40d5020 con 0x7f2dd40d32c0 2024-05-09T03:21:42.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2dd40b7e70 0x7f2dd40b95a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50574/0 (socket says 172.21.15.46:50574) 2024-05-09T03:21:42.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd3fff640 1 -- 172.21.15.46:0/3353782847 learned_addr learned my addr 172.21.15.46:0/3353782847 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:42.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd91e4640 1 --2- 172.21.15.46:0/3353782847 >> v2:172.21.15.88:3300/0 conn(0x7f2dd40d32c0 0x7f2dd40b9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd37fe640 1 -- 172.21.15.46:0/3353782847 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803898062 0 0) 0x7f2dd40d4a60 con 0x7f2dd40b7e70 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd37fe640 1 -- 172.21.15.46:0/3353782847 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dc4003520 con 0x7f2dd40b7e70 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd37fe640 1 -- 172.21.15.46:0/3353782847 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1389173385 0 0) 0x7f2dd40d4d40 con 0x7f2dd40d2eb0 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd91e4640 1 -- 172.21.15.46:0/3353782847 >> v1:172.21.15.88:6789/0 conn(0x7f2dd40d2eb0 legacy=0x7f2dd40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd91e4640 1 -- 172.21.15.46:0/3353782847 >> v1:172.21.15.46:6789/0 conn(0x7f2dd40b7e70 legacy=0x7f2dd40b95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.477+0000 7f2dd91e4640 1 -- 172.21.15.46:0/3353782847 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd40d5300 con 0x7f2dd40d32c0 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.478+0000 7f2dd91e4640 1 --2- 172.21.15.46:0/3353782847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd40d32c0 0x7f2dd40b9cb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009d90 tx=0x7f2dcc02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bec1ba444674479 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.478+0000 7f2dd37fe640 1 -- 172.21.15.46:0/3353782847 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc007740 con 0x7f2dd40d32c0 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.478+0000 7f2dd37fe640 1 -- 172.21.15.46:0/3353782847 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dcc0078e0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.478+0000 7f2dd37fe640 1 -- 172.21.15.46:0/3353782847 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc007c90 con 0x7f2dd40d32c0 2024-05-09T03:21:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 -- 172.21.15.46:0/3353782847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd40d32c0 msgr2=0x7f2dd40b9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 --2- 172.21.15.46:0/3353782847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd40d32c0 0x7f2dd40b9cb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009d90 tx=0x7f2dcc02a6f0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 -- 172.21.15.46:0/3353782847 shutdown_connections 2024-05-09T03:21:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 --2- 172.21.15.46:0/3353782847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd40d32c0 0x7f2dd40b9cb0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 -- 172.21.15.46:0/3353782847 >> 172.21.15.46:0/3353782847 conn(0x7f2dd408d390 msgr2=0x7f2dd408d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 -- 172.21.15.46:0/3353782847 shutdown_connections 2024-05-09T03:21:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.479+0000 7f2ddac47640 1 -- 172.21.15.46:0/3353782847 wait complete. 2024-05-09T03:21:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.481+0000 7f2ddac47640 1 Processor -- start 2024-05-09T03:21:42.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.481+0000 7f2ddac47640 1 -- start start 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2ddac47640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd40d2eb0 0x7f2dd40c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2ddac47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 0x7f2dd41fa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2ddac47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd414e6d0 0x7f2dd415ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2ddac47640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd40b5360 con 0x7f2dd40d32c0 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2ddac47640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd40b51e0 con 0x7f2dd414e6d0 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2ddac47640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd4163bd0 con 0x7f2dd40d2eb0 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2dd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 0x7f2dd41fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2dd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 0x7f2dd41fa360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54702/0 (socket says 172.21.15.46:54702) 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.482+0000 7f2dd3fff640 1 -- 172.21.15.46:0/1411303786 learned_addr learned my addr 172.21.15.46:0/1411303786 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd89e3640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd40d2eb0 0x7f2dd40c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd91e4640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd414e6d0 0x7f2dd415ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd3fff640 1 -- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd40d2eb0 msgr2=0x7f2dd40c9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd3fff640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd40d2eb0 0x7f2dd40c9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd3fff640 1 -- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd414e6d0 msgr2=0x7f2dd415ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd3fff640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd414e6d0 0x7f2dd415ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd3fff640 1 -- 172.21.15.46:0/1411303786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd41fabc0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd3fff640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 0x7f2dd41fa360 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c410 tx=0x7f2dc000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0017070 con 0x7f2dd40d32c0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc0005ce0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00072d0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd41608f0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.483+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd4160eb0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.484+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd41fa900 con 0x7f2dd40d32c0 2024-05-09T03:21:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.485+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2dc001a070 con 0x7f2dd40d32c0 2024-05-09T03:21:42.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.486+0000 7f2dd17fa640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2db8076300 0x7f2db8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.486+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f2dc0096540 con 0x7f2dd40d32c0 2024-05-09T03:21:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.486+0000 7f2dd89e3640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2db8076300 0x7f2db8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.487+0000 7f2dd89e3640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2db8076300 0x7f2db8078810 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8000f60 tx=0x7f2dc8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.491+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2dc005faf0 con 0x7f2dd40d32c0 2024-05-09T03:21:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.680+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2dd406a270 con 0x7f2db8076300 2024-05-09T03:21:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.681+0000 7f2dd17fa640 1 -- 172.21.15.46:0/1411303786 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2dd406a270 con 0x7f2db8076300 2024-05-09T03:21:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 5 to deep-scrub 2024-05-09T03:21:42.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.683+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2db8076300 msgr2=0x7f2db8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.683+0000 7f2ddac47640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2db8076300 0x7f2db8078810 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8000f60 tx=0x7f2dc8009040 comp rx=0 tx=0).stop 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.683+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 msgr2=0x7f2dd41fa360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.683+0000 7f2ddac47640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 0x7f2dd41fa360 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c410 tx=0x7f2dc000c8d0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 shutdown_connections 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2dd414e6d0 0x7f2dd415ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f2db8076300 0x7f2db8078810 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2dd40d32c0 0x7f2dd41fa360 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 --2- 172.21.15.46:0/1411303786 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f2dd40d2eb0 0x7f2dd40c9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 >> 172.21.15.46:0/1411303786 conn(0x7f2dd408d390 msgr2=0x7f2dd415dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 shutdown_connections 2024-05-09T03:21:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.684+0000 7f2ddac47640 1 -- 172.21.15.46:0/1411303786 wait complete. 2024-05-09T03:21:42.696 INFO:tasks.ceph:Scrubbing osd.6 2024-05-09T03:21:42.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.813+0000 7ff7ab053640 1 Processor -- start 2024-05-09T03:21:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.813+0000 7ff7ab053640 1 -- start start 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7ab053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a4073290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7ab053640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7a40cfc70 0x7ff7a406f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7ab053640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a4073ae0 con 0x7ff7a41472f0 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7ab053640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff7a4161c40 con 0x7ff7a40cfc70 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7ab053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a4161e90 con 0x7ff7a41445a0 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7a8def640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a4073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7a95f0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff7a40cfc70 0x7ff7a406f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7a8def640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a4073290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3301/0 says I am v2:172.21.15.46:33870/0 (socket says 172.21.15.46:33870) 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7a3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff7a41472f0 0x7ff7a40cf560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.46:37426/0 (socket says 172.21.15.46:37426) 2024-05-09T03:21:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.815+0000 7ff7a8def640 1 -- 172.21.15.46:0/1955225561 learned_addr learned my addr 172.21.15.46:0/1955225561 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a37fe640 1 -- 172.21.15.46:0/1955225561 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741597967 0 0) 0x7ff7a4073ae0 con 0x7ff7a41472f0 2024-05-09T03:21:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a37fe640 1 -- 172.21.15.46:0/1955225561 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff790003520 con 0x7ff7a41472f0 2024-05-09T03:21:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 msgr2=0x7ff7a4073290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a95f0640 1 --2- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a4073290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1955225561 >> v1:172.21.15.88:6789/0 conn(0x7ff7a41472f0 legacy=0x7ff7a40cf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1955225561 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a4162170 con 0x7ff7a40cfc70 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.816+0000 7ff7a8def640 1 --2- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a4073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7a95f0640 1 --2- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a40cfc70 0x7ff7a406f290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0097a0 tx=0x7ff79c02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b9b9e035cd837f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7a37fe640 1 -- 172.21.15.46:0/1955225561 <== mon.1 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c037070 con 0x7ff7a40cfc70 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7a37fe640 1 -- 172.21.15.46:0/1955225561 <== mon.1 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79c02ae50 con 0x7ff7a40cfc70 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7a37fe640 1 -- 172.21.15.46:0/1955225561 <== mon.1 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c032640 con 0x7ff7a40cfc70 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7ab053640 1 -- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a40cfc70 msgr2=0x7ff7a406f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7ab053640 1 --2- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a40cfc70 0x7ff7a406f290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0097a0 tx=0x7ff79c02a100 comp rx=0 tx=0).stop 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7ab053640 1 -- 172.21.15.46:0/1955225561 shutdown_connections 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7ab053640 1 --2- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a40cfc70 0x7ff7a406f290 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7ab053640 1 --2- 172.21.15.46:0/1955225561 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a4073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.817+0000 7ff7ab053640 1 -- 172.21.15.46:0/1955225561 >> 172.21.15.46:0/1955225561 conn(0x7ff7a406d0a0 msgr2=0x7ff7a40b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.818+0000 7ff7ab053640 1 -- 172.21.15.46:0/1955225561 shutdown_connections 2024-05-09T03:21:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.818+0000 7ff7ab053640 1 -- 172.21.15.46:0/1955225561 wait complete. 2024-05-09T03:21:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.819+0000 7ff7ab053640 1 Processor -- start 2024-05-09T03:21:42.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.819+0000 7ff7ab053640 1 -- start start 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7ab053640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a40aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7ab053640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a41472f0 0x7ff7a4140bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7ab053640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a415e680 0x7ff7a4145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7ab053640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a4162970 con 0x7ff7a415e680 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7ab053640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a40737d0 con 0x7ff7a41472f0 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7ab053640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a4162af0 con 0x7ff7a41445a0 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a415e680 0x7ff7a4145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a415e680 0x7ff7a4145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54708/0 (socket says 172.21.15.46:54708) 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1013953636 learned_addr learned my addr 172.21.15.46:0/1013953636 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a8def640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a40aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a3fff640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a41472f0 0x7ff7a4140bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 msgr2=0x7ff7a40aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7a41445a0 0x7ff7a40aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a41472f0 msgr2=0x7ff7a4140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7a41472f0 0x7ff7a4140bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.821+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1013953636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a4144f10 con 0x7ff7a415e680 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.822+0000 7ff7a95f0640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a415e680 0x7ff7a4145fb0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7ff79c035040 tx=0x7ff79c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.822+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c037040 con 0x7ff7a415e680 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.822+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79c004030 con 0x7ff7a415e680 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.822+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c006e10 con 0x7ff7a415e680 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.822+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a41451f0 con 0x7ff7a415e680 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.822+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a41457b0 con 0x7ff7a415e680 2024-05-09T03:21:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.823+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff7a406a270 con 0x7ff7a415e680 2024-05-09T03:21:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff79c003610 con 0x7ff7a415e680 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a17fa640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7840762d0 0x7ff7840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7ff79c0b8550 con 0x7ff7a415e680 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a17fa640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7ff78407c280 0x7ff78407e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff79c042ec0 con 0x7ff78407c280 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a8def640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7840762d0 0x7ff7840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.824+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7ff79c0b89e0 con 0x7ff7a415e680 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.825+0000 7ff7a3fff640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7ff78407c280 0x7ff78407e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.825+0000 7ff7a3fff640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7ff78407c280 0x7ff78407e6d0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.825+0000 7ff7a8def640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7840762d0 0x7ff7840787e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff7940072e0 tx=0x7ff79400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.826+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== osd.6 v2:172.21.15.88:6808/516824467 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff79c042ec0 con 0x7ff78407c280 2024-05-09T03:21:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.839+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 --> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff7a415c640 con 0x7ff78407c280 2024-05-09T03:21:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7a17fa640 1 -- 172.21.15.46:0/1013953636 <== osd.6 v2:172.21.15.88:6808/516824467 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff7a415c640 con 0x7ff78407c280 2024-05-09T03:21:42.841 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:42.841 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7ff78407c280 msgr2=0x7ff78407e6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6808/516824467,v1:172.21.15.88:6809/516824467] conn(0x7ff78407c280 0x7ff78407e6d0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7840762d0 msgr2=0x7ff7840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff7840762d0 0x7ff7840787e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff7940072e0 tx=0x7ff79400b040 comp rx=0 tx=0).stop 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a415e680 msgr2=0x7ff7a4145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 --2- 172.21.15.46:0/1013953636 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7a415e680 0x7ff7a4145fb0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7ff79c035040 tx=0x7ff79c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7a95f0640 1 -- 172.21.15.46:0/1013953636 reap_dead start 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 shutdown_connections 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 >> 172.21.15.46:0/1013953636 conn(0x7ff7a406d0a0 msgr2=0x7ff7a40cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 shutdown_connections 2024-05-09T03:21:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.840+0000 7ff7ab053640 1 -- 172.21.15.46:0/1013953636 wait complete. 2024-05-09T03:21:42.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-09T03:21:42.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.966+0000 7fbacc847640 1 Processor -- start 2024-05-09T03:21:42.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.966+0000 7fbacc847640 1 -- start start 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbacc847640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbac80d3730 0x7fbac806e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbacc847640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbac8165090 0x7fbac80d1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbacc847640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac809d470 con 0x7fbac806e7e0 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbacc847640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbac809cfe0 con 0x7fbac8165090 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbacc847640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbac80b94a0 con 0x7fbac80d3730 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbac8165090 0x7fbac80d1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbac8165090 0x7fbac80d1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54716/0 (socket says 172.21.15.46:54716) 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac5d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbac806e7e0 0x7fbac8164980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50586/0 (socket says 172.21.15.46:50586) 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 -- 172.21.15.46:0/1456860743 learned_addr learned my addr 172.21.15.46:0/1456860743 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac659c640 1 --2- 172.21.15.46:0/1456860743 >> v2:172.21.15.88:3300/0 conn(0x7fbac80d3730 0x7fbac806e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 -- 172.21.15.46:0/1456860743 >> v2:172.21.15.88:3300/0 conn(0x7fbac80d3730 msgr2=0x7fbac806e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 --2- 172.21.15.46:0/1456860743 >> v2:172.21.15.88:3300/0 conn(0x7fbac80d3730 0x7fbac806e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 -- 172.21.15.46:0/1456860743 >> v1:172.21.15.46:6789/0 conn(0x7fbac806e7e0 legacy=0x7fbac8164980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac6d9d640 1 -- 172.21.15.46:0/1456860743 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac80d11b0 con 0x7fbac8165090 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac559a640 1 -- 172.21.15.46:0/1456860743 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606727485 0 0) 0x7fbab0001110 con 0x7fbac806e7e0 2024-05-09T03:21:42.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.968+0000 7fbac659c640 1 --2- 172.21.15.46:0/1456860743 >> v2:172.21.15.88:3300/0 conn(0x7fbac80d3730 0x7fbac806e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:42.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.969+0000 7fbac6d9d640 1 --2- 172.21.15.46:0/1456860743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac8165090 0x7fbac80d1f50 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7fbac806d990 tx=0x7fbac002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a59a815feef3bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.969+0000 7fbac559a640 1 -- 172.21.15.46:0/1456860743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0037070 con 0x7fbac8165090 2024-05-09T03:21:42.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.969+0000 7fbac559a640 1 -- 172.21.15.46:0/1456860743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac0032440 con 0x7fbac8165090 2024-05-09T03:21:42.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.969+0000 7fbac559a640 1 -- 172.21.15.46:0/1456860743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac0032810 con 0x7fbac8165090 2024-05-09T03:21:42.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 -- 172.21.15.46:0/1456860743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac8165090 msgr2=0x7fbac80d1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 --2- 172.21.15.46:0/1456860743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac8165090 0x7fbac80d1f50 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7fbac806d990 tx=0x7fbac002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 -- 172.21.15.46:0/1456860743 shutdown_connections 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 --2- 172.21.15.46:0/1456860743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac8165090 0x7fbac80d1f50 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 --2- 172.21.15.46:0/1456860743 >> v2:172.21.15.88:3300/0 conn(0x7fbac80d3730 0x7fbac806e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 -- 172.21.15.46:0/1456860743 >> 172.21.15.46:0/1456860743 conn(0x7fbac80d05f0 msgr2=0x7fbac80d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 -- 172.21.15.46:0/1456860743 shutdown_connections 2024-05-09T03:21:42.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.970+0000 7fbacc847640 1 -- 172.21.15.46:0/1456860743 wait complete. 2024-05-09T03:21:42.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.972+0000 7fbacc847640 1 Processor -- start 2024-05-09T03:21:42.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.972+0000 7fbacc847640 1 -- start start 2024-05-09T03:21:42.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbacc847640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbac806e7e0 0x7fbac80cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbacc847640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 0x7fbac80b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbacc847640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbac8165090 0x7fbac80b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbacc847640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80cfbb0 con 0x7fbac80d3730 2024-05-09T03:21:42.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbacc847640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbac80b9160 con 0x7fbac806e7e0 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbacc847640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80cfd30 con 0x7fbac8165090 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbac5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 0x7fbac80b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbac5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 0x7fbac80b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54732/0 (socket says 172.21.15.46:54732) 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.973+0000 7fbac5d9b640 1 -- 172.21.15.46:0/3284224031 learned_addr learned my addr 172.21.15.46:0/3284224031 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac6d9d640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbac8165090 0x7fbac80b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac659c640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbac806e7e0 0x7fbac80cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac5d9b640 1 -- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbac8165090 msgr2=0x7fbac80b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac5d9b640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbac8165090 0x7fbac80b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac5d9b640 1 -- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbac806e7e0 msgr2=0x7fbac80cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac5d9b640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbac806e7e0 0x7fbac80cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac5d9b640 1 -- 172.21.15.46:0/3284224031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac80cc490 con 0x7fbac80d3730 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac659c640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbac806e7e0 0x7fbac80cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbac5d9b640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 0x7fbac80b77e0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c3a0 tx=0x7fbab000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab00074e0 con 0x7fbac80d3730 2024-05-09T03:21:42.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbab00047e0 con 0x7fbac80d3730 2024-05-09T03:21:42.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac81f6910 con 0x7fbac80d3730 2024-05-09T03:21:42.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0004bd0 con 0x7fbac80d3730 2024-05-09T03:21:42.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.974+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac81f6ed0 con 0x7fbac80d3730 2024-05-09T03:21:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.975+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac81f6680 con 0x7fbac80d3730 2024-05-09T03:21:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.976+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbab0011070 con 0x7fbac80d3730 2024-05-09T03:21:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.977+0000 7fbabeffd640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbaa80762d0 0x7fbaa80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.977+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fbab0097050 con 0x7fbac80d3730 2024-05-09T03:21:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.977+0000 7fbac659c640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbaa80762d0 0x7fbaa80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:42.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.977+0000 7fbac659c640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbaa80762d0 0x7fbaa80787e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fbab8007f40 tx=0x7fbab8007490 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:42.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:42.981+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbac809d470 con 0x7fbac80d3730 2024-05-09T03:21:43.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.176+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fbac806a270 con 0x7fbaa80762d0 2024-05-09T03:21:43.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.177+0000 7fbabeffd640 1 -- 172.21.15.46:0/3284224031 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbac806a270 con 0x7fbaa80762d0 2024-05-09T03:21:43.179 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 6 to deep-scrub 2024-05-09T03:21:43.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbaa80762d0 msgr2=0x7fbaa80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbaa80762d0 0x7fbaa80787e0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fbab8007f40 tx=0x7fbab8007490 comp rx=0 tx=0).stop 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 msgr2=0x7fbac80b77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 0x7fbac80b77e0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c3a0 tx=0x7fbab000c860 comp rx=0 tx=0).stop 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 shutdown_connections 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fbac8165090 0x7fbac80b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fbaa80762d0 0x7fbaa80787e0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbac80d3730 0x7fbac80b77e0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 --2- 172.21.15.46:0/3284224031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbac806e7e0 0x7fbac80cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.179+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 >> 172.21.15.46:0/3284224031 conn(0x7fbac80d05f0 msgr2=0x7fbac8072d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.180+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 shutdown_connections 2024-05-09T03:21:43.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.180+0000 7fbacc847640 1 -- 172.21.15.46:0/3284224031 wait complete. 2024-05-09T03:21:43.192 INFO:tasks.ceph:Scrubbing osd.7 2024-05-09T03:21:43.192 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-09T03:21:43.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.309+0000 7fddec714640 1 Processor -- start 2024-05-09T03:21:43.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.309+0000 7fddec714640 1 -- start start 2024-05-09T03:21:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddec714640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdde40b8240 0x7fdde406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddec714640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdde406e250 0x7fdde40b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddec714640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde406e660 con 0x7fdde40b5140 2024-05-09T03:21:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddec714640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdde40b9b60 con 0x7fdde40b8240 2024-05-09T03:21:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddec714640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdde40b5550 con 0x7fdde406e250 2024-05-09T03:21:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fdde9caf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdde40b8240 0x7fdde406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddea4b0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdde40b5140 0x7fdde40b7b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50594/0 (socket says 172.21.15.46:50594) 2024-05-09T03:21:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddea4b0640 1 -- 172.21.15.46:0/3670543414 learned_addr learned my addr 172.21.15.46:0/3670543414 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:21:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fddeacb1640 1 --2- 172.21.15.46:0/3670543414 >> v2:172.21.15.88:3300/0 conn(0x7fdde406e250 0x7fdde40b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fdde9caf640 1 -- 172.21.15.46:0/3670543414 >> v2:172.21.15.88:3300/0 conn(0x7fdde406e250 msgr2=0x7fdde40b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fdde9caf640 1 --2- 172.21.15.46:0/3670543414 >> v2:172.21.15.88:3300/0 conn(0x7fdde406e250 0x7fdde40b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fdde9caf640 1 -- 172.21.15.46:0/3670543414 >> v1:172.21.15.46:6789/0 conn(0x7fdde40b5140 legacy=0x7fdde40b7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fdde9caf640 1 -- 172.21.15.46:0/3670543414 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde40b9270 con 0x7fdde40b8240 2024-05-09T03:21:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.311+0000 7fdde94ae640 1 -- 172.21.15.46:0/3670543414 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714216652 0 0) 0x7fdde406e660 con 0x7fdde40b5140 2024-05-09T03:21:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.312+0000 7fddeacb1640 1 --2- 172.21.15.46:0/3670543414 >> v2:172.21.15.88:3300/0 conn(0x7fdde406e250 0x7fdde40b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.312+0000 7fdde9caf640 1 --2- 172.21.15.46:0/3670543414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde40b8240 0x7fdde406dd10 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7fdde40c96a0 tx=0x7fddd402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd462d793f917306 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.312+0000 7fdde94ae640 1 -- 172.21.15.46:0/3670543414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4037070 con 0x7fdde40b8240 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.312+0000 7fdde94ae640 1 -- 172.21.15.46:0/3670543414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd4032440 con 0x7fdde40b8240 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.312+0000 7fdde94ae640 1 -- 172.21.15.46:0/3670543414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4032810 con 0x7fdde40b8240 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 -- 172.21.15.46:0/3670543414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde40b8240 msgr2=0x7fdde406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 --2- 172.21.15.46:0/3670543414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde40b8240 0x7fdde406dd10 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7fdde40c96a0 tx=0x7fddd402a1a0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 -- 172.21.15.46:0/3670543414 shutdown_connections 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 --2- 172.21.15.46:0/3670543414 >> v2:172.21.15.88:3300/0 conn(0x7fdde406e250 0x7fdde40b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 --2- 172.21.15.46:0/3670543414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde40b8240 0x7fdde406dd10 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 -- 172.21.15.46:0/3670543414 >> 172.21.15.46:0/3670543414 conn(0x7fdde40d5e90 msgr2=0x7fdde40d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 -- 172.21.15.46:0/3670543414 shutdown_connections 2024-05-09T03:21:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.313+0000 7fddec714640 1 -- 172.21.15.46:0/3670543414 wait complete. 2024-05-09T03:21:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.315+0000 7fddec714640 1 Processor -- start 2024-05-09T03:21:43.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.315+0000 7fddec714640 1 -- start start 2024-05-09T03:21:43.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddec714640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdde406e250 0x7fdde414e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddec714640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde40b5140 0x7fdde4150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddec714640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4154a90 0x7fdde40ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddec714640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdde40b97f0 con 0x7fdde4154a90 2024-05-09T03:21:43.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddec714640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdde40b9510 con 0x7fdde40b5140 2024-05-09T03:21:43.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddec714640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fdde40ba530 con 0x7fdde406e250 2024-05-09T03:21:43.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddeacb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4154a90 0x7fdde40ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.316+0000 7fddeacb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4154a90 0x7fdde40ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54742/0 (socket says 172.21.15.46:54742) 2024-05-09T03:21:43.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 -- 172.21.15.46:0/3553373959 learned_addr learned my addr 172.21.15.46:0/3553373959 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:43.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fdde9caf640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde40b5140 0x7fdde4150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddea4b0640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdde406e250 0x7fdde414e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 -- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdde406e250 msgr2=0x7fdde414e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fdde406e250 0x7fdde414e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 -- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde40b5140 msgr2=0x7fdde4150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde40b5140 0x7fdde4150f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 -- 172.21.15.46:0/3553373959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde4151860 con 0x7fdde4154a90 2024-05-09T03:21:43.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddeacb1640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4154a90 0x7fdde40ceec0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00d5e0 tx=0x7fdddc00daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc018070 con 0x7fdde4154a90 2024-05-09T03:21:43.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdddc004030 con 0x7fdde4154a90 2024-05-09T03:21:43.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc004400 con 0x7fdde4154a90 2024-05-09T03:21:43.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde4151b40 con 0x7fdde4154a90 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.317+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde40ac770 con 0x7fdde4154a90 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.318+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdde406a270 con 0x7fdde4154a90 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.319+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdddc014620 con 0x7fdde4154a90 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.319+0000 7fdddaffd640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddc80762d0 0x7fddc80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fdddc0980d0 con 0x7fdde4154a90 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fdddaffd640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fddc807c280 0x7fddc807e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdddc008f10 con 0x7fddc807c280 2024-05-09T03:21:43.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7fdddc098560 con 0x7fdde4154a90 2024-05-09T03:21:43.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fddea4b0640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddc80762d0 0x7fddc80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fdde9caf640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fddc807c280 0x7fddc807e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fddea4b0640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddc80762d0 0x7fddc80787e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fdde0007ed0 tx=0x7fdde0007e60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.320+0000 7fdde9caf640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fddc807c280 0x7fddc807e6d0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.321+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== osd.7 v2:172.21.15.88:6816/3368726188 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdddc008f10 con 0x7fddc807c280 2024-05-09T03:21:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 --> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdde4151610 con 0x7fddc807c280 2024-05-09T03:21:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fdddaffd640 1 -- 172.21.15.46:0/3553373959 <== osd.7 v2:172.21.15.88:6816/3368726188 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdde4151610 con 0x7fddc807c280 2024-05-09T03:21:43.337 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-05-09T03:21:43.337 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T03:21:43.338 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-05-09T03:21:43.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fddc807c280 msgr2=0x7fddc807e6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6816/3368726188,v1:172.21.15.88:6817/3368726188] conn(0x7fddc807c280 0x7fddc807e6d0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddc80762d0 msgr2=0x7fddc80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fddc80762d0 0x7fddc80787e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fdde0007ed0 tx=0x7fdde0007e60 comp rx=0 tx=0).stop 2024-05-09T03:21:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4154a90 msgr2=0x7fdde40ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.335+0000 7fddec714640 1 --2- 172.21.15.46:0/3553373959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdde4154a90 0x7fdde40ceec0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00d5e0 tx=0x7fdddc00daa0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.336+0000 7fddeacb1640 1 -- 172.21.15.46:0/3553373959 reap_dead start 2024-05-09T03:21:43.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.336+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 shutdown_connections 2024-05-09T03:21:43.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.336+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 >> 172.21.15.46:0/3553373959 conn(0x7fdde40d5e90 msgr2=0x7fdde40b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:43.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.336+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 shutdown_connections 2024-05-09T03:21:43.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.336+0000 7fddec714640 1 -- 172.21.15.46:0/3553373959 wait complete. 2024-05-09T03:21:43.348 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-09T03:21:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.462+0000 7ff70c2e2640 1 Processor -- start 2024-05-09T03:21:43.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.462+0000 7ff70c2e2640 1 -- start start 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70c2e2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70c2e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff7040b3f50 0x7ff7040ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70c2e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff70406ebe0 con 0x7ff7040b4700 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70c2e2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff70406eec0 con 0x7ff7040b3f50 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70c2e2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff70406f1a0 con 0x7ff7040d2800 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff7040b3f50 0x7ff7040ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff7040b3f50 0x7ff7040ce130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54754/0 (socket says 172.21.15.46:54754) 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 -- 172.21.15.46:0/1831833697 learned_addr learned my addr 172.21.15.46:0/1831833697 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:43.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70987d640 1 --2- 172.21.15.46:0/1831833697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 -- 172.21.15.46:0/1831833697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 msgr2=0x7ff7040b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 --2- 172.21.15.46:0/1831833697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 -- 172.21.15.46:0/1831833697 >> v1:172.21.15.88:6789/0 conn(0x7ff7040b4700 legacy=0x7ff7040d20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.464+0000 7ff70a87f640 1 -- 172.21.15.46:0/1831833697 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70406f480 con 0x7ff7040b3f50 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.465+0000 7ff70a87f640 1 --2- 172.21.15.46:0/1831833697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040ce130 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009e50 tx=0x7ff6fc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9a188a5f1dccc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.465+0000 7ff70907c640 1 -- 172.21.15.46:0/1831833697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc036070 con 0x7ff7040b3f50 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.465+0000 7ff70907c640 1 -- 172.21.15.46:0/1831833697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6fc031600 con 0x7ff7040b3f50 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.465+0000 7ff70907c640 1 -- 172.21.15.46:0/1831833697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc0319d0 con 0x7ff7040b3f50 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 -- 172.21.15.46:0/1831833697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 msgr2=0x7ff7040ce130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/1831833697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040ce130 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009e50 tx=0x7ff6fc02a680 comp rx=0 tx=0).stop 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 -- 172.21.15.46:0/1831833697 shutdown_connections 2024-05-09T03:21:43.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/1831833697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040ce130 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/1831833697 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 -- 172.21.15.46:0/1831833697 >> 172.21.15.46:0/1831833697 conn(0x7ff7040b55d0 msgr2=0x7ff7040b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 -- 172.21.15.46:0/1831833697 shutdown_connections 2024-05-09T03:21:43.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.466+0000 7ff70c2e2640 1 -- 172.21.15.46:0/1831833697 wait complete. 2024-05-09T03:21:43.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.468+0000 7ff70c2e2640 1 Processor -- start 2024-05-09T03:21:43.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.468+0000 7ff70c2e2640 1 -- start start 2024-05-09T03:21:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70c2e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70c2e2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7040b4700 0x7ff7040cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70c2e2640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70c2e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7040b51d0 con 0x7ff7040b3f50 2024-05-09T03:21:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70c2e2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7040b4360 con 0x7ff7040b4700 2024-05-09T03:21:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70c2e2640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7ff7040cf340 con 0x7ff7040d2800 2024-05-09T03:21:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54764/0 (socket says 172.21.15.46:54764) 2024-05-09T03:21:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 -- 172.21.15.46:0/110331833 learned_addr learned my addr 172.21.15.46:0/110331833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70987d640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7040b4700 0x7ff7040cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a87f640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 -- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 msgr2=0x7ff7040cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 -- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7040b4700 msgr2=0x7ff7040cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7040b4700 0x7ff7040cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a07e640 1 -- 172.21.15.46:0/110331833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7040cdd60 con 0x7ff7040b3f50 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70987d640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7040b4700 0x7ff7040cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.469+0000 7ff70a87f640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.470+0000 7ff70a07e640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040cee00 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7ff70000c3a0 tx=0x7ff70000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.470+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700017070 con 0x7ff7040b3f50 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.470+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7000052d0 con 0x7ff7040b3f50 2024-05-09T03:21:43.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.470+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7040ce040 con 0x7ff7040b3f50 2024-05-09T03:21:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.470+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700003830 con 0x7ff7040b3f50 2024-05-09T03:21:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.470+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7040ce600 con 0x7ff7040b3f50 2024-05-09T03:21:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.471+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7040cdab0 con 0x7ff7040b3f50 2024-05-09T03:21:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.472+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff70001a070 con 0x7ff7040b3f50 2024-05-09T03:21:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.472+0000 7ff6faffd640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6e8076300 0x7ff6e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.472+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7ff700096a50 con 0x7ff7040b3f50 2024-05-09T03:21:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.472+0000 7ff70987d640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6e8076300 0x7ff6e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.473+0000 7ff70987d640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6e8076300 0x7ff6e8078810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff7040b65b0 tx=0x7ff6f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.476+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff700021020 con 0x7ff7040b3f50 2024-05-09T03:21:43.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.670+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7ff70406a270 con 0x7ff6e8076300 2024-05-09T03:21:43.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.670+0000 7ff6faffd640 1 -- 172.21.15.46:0/110331833 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff70406a270 con 0x7ff6e8076300 2024-05-09T03:21:43.672 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 7 to deep-scrub 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6e8076300 msgr2=0x7ff6e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6e8076300 0x7ff6e8078810 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7ff7040b65b0 tx=0x7ff6f4008040 comp rx=0 tx=0).stop 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 msgr2=0x7ff7040cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040cee00 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7ff70000c3a0 tx=0x7ff70000c860 comp rx=0 tx=0).stop 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 shutdown_connections 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7ff7040d2800 0x7ff7040cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7040b4700 0x7ff7040cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7ff6e8076300 0x7ff6e8078810 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 --2- 172.21.15.46:0/110331833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7040b3f50 0x7ff7040cee00 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 >> 172.21.15.46:0/110331833 conn(0x7ff7040b55d0 msgr2=0x7ff7040b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:43.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 shutdown_connections 2024-05-09T03:21:43.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.673+0000 7ff70c2e2640 1 -- 172.21.15.46:0/110331833 wait complete. 2024-05-09T03:21:43.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:21:43.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.798+0000 7fa605b65640 1 Processor -- start 2024-05-09T03:21:43.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.798+0000 7fa605b65640 1 -- start start 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa605b65640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6000b9310 0x7fa60008cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa605b65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa60008d2a0 0x7fa60008d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa605b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa600164830 0x7fa600164c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa605b65640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa60008dbf0 con 0x7fa6000b9310 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa605b65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa60006f760 con 0x7fa60008d2a0 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa605b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa60006ebe0 con 0x7fa600164830 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6000b9310 0x7fa60008cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6000b9310 0x7fa60008cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54766/0 (socket says 172.21.15.46:54766) 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 -- 172.21.15.46:0/4258970474 learned_addr learned my addr 172.21.15.46:0/4258970474 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:43.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5fffff640 1 --2- 172.21.15.46:0/4258970474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa600164830 0x7fa600164c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5feffd640 1 --2- 172.21.15.46:0/4258970474 >> v2:172.21.15.88:3300/0 conn(0x7fa60008d2a0 0x7fa60008d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 -- 172.21.15.46:0/4258970474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa600164830 msgr2=0x7fa600164c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4258970474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa600164830 0x7fa600164c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 -- 172.21.15.46:0/4258970474 >> v2:172.21.15.88:3300/0 conn(0x7fa60008d2a0 msgr2=0x7fa60008d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4258970474 >> v2:172.21.15.88:3300/0 conn(0x7fa60008d2a0 0x7fa60008d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.800+0000 7fa5ff7fe640 1 -- 172.21.15.46:0/4258970474 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60006eec0 con 0x7fa6000b9310 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.801+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4258970474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000b9310 0x7fa60008cd60 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fa600074740 tx=0x7fa5f002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7f367095a3579d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.801+0000 7fa5fe7fc640 1 -- 172.21.15.46:0/4258970474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0037070 con 0x7fa6000b9310 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.801+0000 7fa5fe7fc640 1 -- 172.21.15.46:0/4258970474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f0032440 con 0x7fa6000b9310 2024-05-09T03:21:43.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.801+0000 7fa5fe7fc640 1 -- 172.21.15.46:0/4258970474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0032810 con 0x7fa6000b9310 2024-05-09T03:21:43.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 -- 172.21.15.46:0/4258970474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000b9310 msgr2=0x7fa60008cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 --2- 172.21.15.46:0/4258970474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000b9310 0x7fa60008cd60 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fa600074740 tx=0x7fa5f002a1e0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 -- 172.21.15.46:0/4258970474 shutdown_connections 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 --2- 172.21.15.46:0/4258970474 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa600164830 0x7fa600164c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 --2- 172.21.15.46:0/4258970474 >> v2:172.21.15.88:3300/0 conn(0x7fa60008d2a0 0x7fa60008d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 --2- 172.21.15.46:0/4258970474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6000b9310 0x7fa60008cd60 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 -- 172.21.15.46:0/4258970474 >> 172.21.15.46:0/4258970474 conn(0x7fa6000b5660 msgr2=0x7fa6000b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 -- 172.21.15.46:0/4258970474 shutdown_connections 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.802+0000 7fa605b65640 1 -- 172.21.15.46:0/4258970474 wait complete. 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.804+0000 7fa605b65640 1 Processor -- start 2024-05-09T03:21:43.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.804+0000 7fa605b65640 1 -- start start 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa605b65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa60008d2a0 0x7fa60015f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa605b65640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6000b9310 0x7fa60015ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa605b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 0x7fa600150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa605b65640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa600165180 con 0x7fa600164830 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa605b65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa60006f480 con 0x7fa60008d2a0 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa605b65640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fa600165300 con 0x7fa6000b9310 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa5fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 0x7fa600150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa5fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 0x7fa600150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54776/0 (socket says 172.21.15.46:54776) 2024-05-09T03:21:43.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.805+0000 7fa5fffff640 1 -- 172.21.15.46:0/4062660146 learned_addr learned my addr 172.21.15.46:0/4062660146 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa60008d2a0 0x7fa60015f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5feffd640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6000b9310 0x7fa60015ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5fffff640 1 -- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6000b9310 msgr2=0x7fa60015ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5fffff640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6000b9310 0x7fa60015ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5fffff640 1 -- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa60008d2a0 msgr2=0x7fa60015f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5fffff640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa60008d2a0 0x7fa60015f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5fffff640 1 -- 172.21.15.46:0/4062660146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa600151490 con 0x7fa600164830 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5feffd640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6000b9310 0x7fa60015ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa60008d2a0 0x7fa60015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5fffff640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 0x7fa600150f50 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800efc0 tx=0x7fa5f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8018020 con 0x7fa600164830 2024-05-09T03:21:43.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f8014480 con 0x7fa600164830 2024-05-09T03:21:43.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60015d980 con 0x7fa600164830 2024-05-09T03:21:43.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8014870 con 0x7fa600164830 2024-05-09T03:21:43.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.806+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa60015df40 con 0x7fa600164830 2024-05-09T03:21:43.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.807+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa60006a270 con 0x7fa600164830 2024-05-09T03:21:43.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.808+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa5f8005d10 con 0x7fa600164830 2024-05-09T03:21:43.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.809+0000 7fa5e7fff640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa5ec076300 0x7fa5ec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:21:43.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.809+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fa5f8097e30 con 0x7fa600164830 2024-05-09T03:21:43.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.809+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa5ec076300 0x7fa5ec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:21:43.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.809+0000 7fa5ff7fe640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa5ec076300 0x7fa5ec078810 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0034000 tx=0x7fa5f002e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:21:43.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:43.814+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa5f8022070 con 0x7fa600164830 2024-05-09T03:21:44.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.002+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa6000b4d40 con 0x7fa5ec076300 2024-05-09T03:21:44.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.004+0000 7fa5e7fff640 1 -- 172.21.15.46:0/4062660146 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13362 (secure 0 0 0) 0x7fa6000b4d40 con 0x7fa5ec076300 2024-05-09T03:21:44.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:21:44.006 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa5ec076300 msgr2=0x7fa5ec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa5ec076300 0x7fa5ec078810 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0034000 tx=0x7fa5f002e000 comp rx=0 tx=0).stop 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 msgr2=0x7fa600150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 0x7fa600150f50 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800efc0 tx=0x7fa5f800c490 comp rx=0 tx=0).stop 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 shutdown_connections 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fa5ec076300 0x7fa5ec078810 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa600164830 0x7fa600150f50 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:44.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fa6000b9310 0x7fa60015ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:44.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 --2- 172.21.15.46:0/4062660146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa60008d2a0 0x7fa60015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:21:44.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 >> 172.21.15.46:0/4062660146 conn(0x7fa6000b5660 msgr2=0x7fa6000cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:21:44.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 shutdown_connections 2024-05-09T03:21:44.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:21:44.006+0000 7fa605b65640 1 -- 172.21.15.46:0/4062660146 wait complete. 2024-05-09T03:21:44.019 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-05-09T03:21:42.690989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297956,"kb_used_data":15208,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676764,"statfs":{"total":773094113280,"available":772789006336,"internally_reserved":0,"allocated":15572992,"data_stored":8107208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":289394225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":-4,"num_read_kb":-4,"num_write":-5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.059295"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":341,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-05-09T03:21:35.770635+0000","last_change":"2024-05-09T03:09:23.684392+0000","last_active":"2024-05-09T03:21:35.770635+0000","last_peered":"2024-05-09T03:21:35.770635+0000","last_clean":"2024-05-09T03:21:35.770635+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:35.770635+0000","last_undegraded":"2024-05-09T03:21:35.770635+0000","last_fullsized":"2024-05-09T03:21:35.770635+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T03:09:22.511667+0000","last_clean_scrub_stamp":"2024-05-09T03:09:22.511667+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:21:56.312075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574999,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37680,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334160,"statfs":{"total":96636764160,"available":96598179840,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574999,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574999,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29356,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342484,"statfs":{"total":96636764160,"available":96606703616,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:21:44.022 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-09T03:09:22.511667+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=3, tm_min=9, tm_sec=22, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=3, tm_min=21, tm_sec=38, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T03:21:44.022 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-09T03:22:04.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T03:22:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.153+0000 7fccd1384640 1 Processor -- start 2024-05-09T03:22:04.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.153+0000 7fccd1384640 1 -- start start 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccd1384640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcccc0b4ff0 0x7fcccc0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccd1384640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcccc0af0c0 0x7fcccc070670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccd1384640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcccc070fc0 con 0x7fcccc070bb0 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccd1384640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcccc15d980 con 0x7fcccc0b4ff0 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccd1384640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcccc15dc60 con 0x7fcccc0af0c0 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccaffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcccc0b4ff0 0x7fcccc0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcccc070bb0 0x7fcccc073870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39682/0 (socket says 172.21.15.46:39682) 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccb7fe640 1 -- 172.21.15.46:0/1570579075 learned_addr learned my addr 172.21.15.46:0/1570579075 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:22:04.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccc3fff640 1 --2- 172.21.15.46:0/1570579075 >> v2:172.21.15.88:3300/0 conn(0x7fcccc0af0c0 0x7fcccc070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccaffd640 1 -- 172.21.15.46:0/1570579075 >> v2:172.21.15.88:3300/0 conn(0x7fcccc0af0c0 msgr2=0x7fcccc070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccaffd640 1 --2- 172.21.15.46:0/1570579075 >> v2:172.21.15.88:3300/0 conn(0x7fcccc0af0c0 0x7fcccc070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccaffd640 1 -- 172.21.15.46:0/1570579075 >> v1:172.21.15.46:6789/0 conn(0x7fcccc070bb0 legacy=0x7fcccc073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fcccaffd640 1 -- 172.21.15.46:0/1570579075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc15df40 con 0x7fcccc0b4ff0 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccca7fc640 1 -- 172.21.15.46:0/1570579075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876556589 0 0) 0x7fccc4001490 con 0x7fcccc070bb0 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.155+0000 7fccc3fff640 1 --2- 172.21.15.46:0/1570579075 >> v2:172.21.15.88:3300/0 conn(0x7fcccc0af0c0 0x7fcccc070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.156+0000 7fcccaffd640 1 --2- 172.21.15.46:0/1570579075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc0aeb80 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0d2d00 tx=0x7fccbc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5976ee23b83ea4c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.156+0000 7fccca7fc640 1 -- 172.21.15.46:0/1570579075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc037070 con 0x7fcccc0b4ff0 2024-05-09T03:22:04.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.156+0000 7fccca7fc640 1 -- 172.21.15.46:0/1570579075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccbc032440 con 0x7fcccc0b4ff0 2024-05-09T03:22:04.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.156+0000 7fccca7fc640 1 -- 172.21.15.46:0/1570579075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc02ae60 con 0x7fcccc0b4ff0 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 -- 172.21.15.46:0/1570579075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0b4ff0 msgr2=0x7fcccc0aeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 --2- 172.21.15.46:0/1570579075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc0aeb80 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fcccc0d2d00 tx=0x7fccbc02a110 comp rx=0 tx=0).stop 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 -- 172.21.15.46:0/1570579075 shutdown_connections 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 --2- 172.21.15.46:0/1570579075 >> v2:172.21.15.88:3300/0 conn(0x7fcccc0af0c0 0x7fcccc070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 --2- 172.21.15.46:0/1570579075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc0aeb80 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 -- 172.21.15.46:0/1570579075 >> 172.21.15.46:0/1570579075 conn(0x7fcccc0b3910 msgr2=0x7fcccc0b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 -- 172.21.15.46:0/1570579075 shutdown_connections 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.157+0000 7fccd1384640 1 -- 172.21.15.46:0/1570579075 wait complete. 2024-05-09T03:22:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.159+0000 7fccd1384640 1 Processor -- start 2024-05-09T03:22:04.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.159+0000 7fccd1384640 1 -- start start 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccd1384640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcccc070bb0 0x7fcccc1e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccd1384640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 0x7fcccc073770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccd1384640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccd1384640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc15d540 con 0x7fcccc0af0c0 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccd1384640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc0b5400 con 0x7fcccc0b4ff0 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccd1384640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7fcccc15d6c0 con 0x7fcccc070bb0 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccc3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 0x7fcccc073770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccc3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 0x7fcccc073770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49908/0 (socket says 172.21.15.46:49908) 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.160+0000 7fccc3fff640 1 -- 172.21.15.46:0/3573568317 learned_addr learned my addr 172.21.15.46:0/3573568317 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fcccaffd640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcccc070bb0 0x7fcccc1e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fcccb7fe640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc3fff640 1 -- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcccc070bb0 msgr2=0x7fcccc1e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc3fff640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcccc070bb0 0x7fcccc1e71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc3fff640 1 -- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcccc0b4ff0 msgr2=0x7fcccc157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc3fff640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc3fff640 1 -- 172.21.15.46:0/3573568317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc09e330 con 0x7fcccc0af0c0 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fcccaffd640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcccc070bb0 0x7fcccc1e71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc3fff640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 0x7fcccc073770 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fccb400d7d0 tx=0x7fccb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4004030 con 0x7fcccc0af0c0 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccb40041d0 con 0x7fcccc0af0c0 2024-05-09T03:22:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcccc09e610 con 0x7fcccc0af0c0 2024-05-09T03:22:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb40096a0 con 0x7fcccc0af0c0 2024-05-09T03:22:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.161+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcccc09ebd0 con 0x7fcccc0af0c0 2024-05-09T03:22:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.162+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcccc09e0a0 con 0x7fcccc0af0c0 2024-05-09T03:22:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.163+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fccb4009840 con 0x7fcccc0af0c0 2024-05-09T03:22:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.163+0000 7fccc37fe640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccac0762d0 0x7fccac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.164+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7fccb4097610 con 0x7fcccc0af0c0 2024-05-09T03:22:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.164+0000 7fcccaffd640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccac0762d0 0x7fccac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.164+0000 7fcccaffd640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccac0762d0 0x7fccac0787e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fccbc034000 tx=0x7fccbc02e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.168+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fccb4061180 con 0x7fcccc0af0c0 2024-05-09T03:22:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.355+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 --> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcccc06a270 con 0x7fccac0762d0 2024-05-09T03:22:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.356+0000 7fccc37fe640 1 -- 172.21.15.46:0/3573568317 <== mgr.4100 v2:172.21.15.88:6832/663422477 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13364 (secure 0 0 0) 0x7fcccc06a270 con 0x7fccac0762d0 2024-05-09T03:22:04.358 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:22:04.358 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-05-09T03:22:04.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccac0762d0 msgr2=0x7fccac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccac0762d0 0x7fccac0787e0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fccbc034000 tx=0x7fccbc02e000 comp rx=0 tx=0).stop 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 msgr2=0x7fcccc073770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 0x7fcccc073770 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fccb400d7d0 tx=0x7fccb400dc90 comp rx=0 tx=0).stop 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 shutdown_connections 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcccc0b4ff0 0x7fcccc157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7fccac0762d0 0x7fccac0787e0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcccc0af0c0 0x7fcccc073770 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 --2- 172.21.15.46:0/3573568317 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7fcccc070bb0 0x7fcccc1e71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 >> 172.21.15.46:0/3573568317 conn(0x7fcccc0b3910 msgr2=0x7fcccc1614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 shutdown_connections 2024-05-09T03:22:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.358+0000 7fccd1384640 1 -- 172.21.15.46:0/3573568317 wait complete. 2024-05-09T03:22:04.371 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-05-09T03:22:02.695336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":297944,"kb_used_data":15208,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676776,"statfs":{"total":773094113280,"available":772789018624,"internally_reserved":0,"allocated":15572992,"data_stored":8107208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":289394225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"1.0","version":"139'74","reported_seq":352,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-05-09T03:21:43.844227+0000","last_change":"2024-05-09T03:21:43.844263+0000","last_active":"2024-05-09T03:21:43.844227+0000","last_peered":"2024-05-09T03:21:43.844227+0000","last_clean":"2024-05-09T03:21:43.844227+0000","last_became_active":"2024-05-09T03:09:23.683264+0000","last_became_peered":"2024-05-09T03:09:23.683264+0000","last_unstale":"2024-05-09T03:21:43.844227+0000","last_undegraded":"2024-05-09T03:21:43.844227+0000","last_fullsized":"2024-05-09T03:21:43.844227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'74","last_scrub_stamp":"2024-05-09T03:21:43.844124+0000","last_deep_scrub":"139'74","last_deep_scrub_stamp":"2024-05-09T03:21:43.844124+0000","last_clean_scrub_stamp":"2024-05-09T03:21:43.844124+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:03:46.687920+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37672,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334168,"statfs":{"total":96636764160,"available":96598188032,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37100,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575003,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575002,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41260,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330580,"statfs":{"total":96636764160,"available":96594513920,"internally_reserved":0,"allocated":1798144,"data_stored":865809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29352,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342488,"statfs":{"total":96636764160,"available":96606707712,"internally_reserved":0,"allocated":2392064,"data_stored":1456177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T03:22:04.372 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-09T03:22:04.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.485+0000 7f489900c640 1 Processor -- start 2024-05-09T03:22:04.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.485+0000 7f489900c640 1 -- start start 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489900c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f48940b5140 0x7f48940b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489900c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b7ca0 0x7f48940d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489900c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48940d3df0 0x7f4894163f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489900c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f48940d3480 con 0x7f48940d3df0 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489900c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f48940b80b0 con 0x7f48940b5140 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489900c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48940b5550 con 0x7f48940b7ca0 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489359e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48940d3df0 0x7f4894163f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.486+0000 7f489359e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f48940d3df0 0x7f4894163f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49914/0 (socket says 172.21.15.46:49914) 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489359e640 1 -- 172.21.15.46:0/2833069329 learned_addr learned my addr 172.21.15.46:0/2833069329 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f4892d9d640 1 --2- 172.21.15.46:0/2833069329 >> v2:172.21.15.88:3300/0 conn(0x7f48940b5140 0x7f48940b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489259c640 1 --2- 172.21.15.46:0/2833069329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b7ca0 0x7f48940d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489359e640 1 -- 172.21.15.46:0/2833069329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b7ca0 msgr2=0x7f48940d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:22:04.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489359e640 1 --2- 172.21.15.46:0/2833069329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b7ca0 0x7f48940d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489359e640 1 -- 172.21.15.46:0/2833069329 >> v2:172.21.15.88:3300/0 conn(0x7f48940b5140 msgr2=0x7f48940b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T03:22:04.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489359e640 1 --2- 172.21.15.46:0/2833069329 >> v2:172.21.15.88:3300/0 conn(0x7f48940b5140 0x7f48940b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.487+0000 7f489359e640 1 -- 172.21.15.46:0/2833069329 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940d44e0 con 0x7f48940d3df0 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489359e640 1 --2- 172.21.15.46:0/2833069329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f4894163f90 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f48940b88d0 tx=0x7f488c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5354653646679d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f4891d9b640 1 -- 172.21.15.46:0/2833069329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c037070 con 0x7f48940d3df0 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f4891d9b640 1 -- 172.21.15.46:0/2833069329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488c032440 con 0x7f48940d3df0 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f4891d9b640 1 -- 172.21.15.46:0/2833069329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c032810 con 0x7f48940d3df0 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 -- 172.21.15.46:0/2833069329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 msgr2=0x7f4894163f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 --2- 172.21.15.46:0/2833069329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f4894163f90 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f48940b88d0 tx=0x7f488c02a270 comp rx=0 tx=0).stop 2024-05-09T03:22:04.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 -- 172.21.15.46:0/2833069329 shutdown_connections 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 --2- 172.21.15.46:0/2833069329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f4894163f90 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 --2- 172.21.15.46:0/2833069329 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b7ca0 0x7f48940d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 --2- 172.21.15.46:0/2833069329 >> v2:172.21.15.88:3300/0 conn(0x7f48940b5140 0x7f48940b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 -- 172.21.15.46:0/2833069329 >> 172.21.15.46:0/2833069329 conn(0x7f489408d3f0 msgr2=0x7f489408d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.488+0000 7f489900c640 1 -- 172.21.15.46:0/2833069329 shutdown_connections 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.489+0000 7f489900c640 1 -- 172.21.15.46:0/2833069329 wait complete. 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.491+0000 7f489900c640 1 Processor -- start 2024-05-09T03:22:04.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.491+0000 7f489900c640 1 -- start start 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489900c640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b5140 0x7f48941ea100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489900c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48940b7ca0 0x7f48941f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489900c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f48941f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489900c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48940d4d80 con 0x7f48940d3df0 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489900c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f48940d4aa0 con 0x7f48940b7ca0 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489900c640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f48940d4200 con 0x7f48940b5140 2024-05-09T03:22:04.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f48941f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f48941f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49928/0 (socket says 172.21.15.46:49928) 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 -- 172.21.15.46:0/156009805 learned_addr learned my addr 172.21.15.46:0/156009805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489259c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48940b7ca0 0x7f48941f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f4892d9d640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b5140 0x7f48941ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 -- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b5140 msgr2=0x7f48941ea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b5140 0x7f48941ea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 -- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48940b7ca0 msgr2=0x7f48941f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48940b7ca0 0x7f48941f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f489359e640 1 -- 172.21.15.46:0/156009805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48941eadc0 con 0x7f48940d3df0 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.492+0000 7f4892d9d640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b5140 0x7f48941ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.493+0000 7f489359e640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f48941f6240 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f488c02a2b0 tx=0x7f488c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.493+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c037040 con 0x7f48940d3df0 2024-05-09T03:22:04.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.493+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488c042070 con 0x7f48940d3df0 2024-05-09T03:22:04.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.493+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c030920 con 0x7f48940d3df0 2024-05-09T03:22:04.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.493+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48941eb0a0 con 0x7f48940d3df0 2024-05-09T03:22:04.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.493+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48941eb660 con 0x7f48940d3df0 2024-05-09T03:22:04.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.494+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f489406a270 con 0x7f48940d3df0 2024-05-09T03:22:04.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.495+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f488c030ac0 con 0x7f48940d3df0 2024-05-09T03:22:04.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.495+0000 7f487b7fe640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4880076300 0x7f4880078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.495+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f488c0b7bb0 con 0x7f48940d3df0 2024-05-09T03:22:04.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.495+0000 7f4892d9d640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4880076300 0x7f4880078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.496+0000 7f4892d9d640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4880076300 0x7f4880078810 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4884004410 tx=0x7f48840039f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.500+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f488c0817b0 con 0x7f48940d3df0 2024-05-09T03:22:04.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.689+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f48940b9900 con 0x7f48940d3df0 2024-05-09T03:22:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.693+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v30)=0 v30) ==== 123+0+0 (secure 0 0 0) 0x7f488c0810b0 con 0x7f48940d3df0 2024-05-09T03:22:04.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.694+0000 7f487b7fe640 1 -- 172.21.15.46:0/156009805 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f488c032440 con 0x7f48940d3df0 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4880076300 msgr2=0x7f4880078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4880076300 0x7f4880078810 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4884004410 tx=0x7f48840039f0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 msgr2=0x7f48941f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f48941f6240 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f488c02a2b0 tx=0x7f488c0046c0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 shutdown_connections 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f4880076300 0x7f4880078810 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48940d3df0 0x7f48941f6240 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48940b7ca0 0x7f48941f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 --2- 172.21.15.46:0/156009805 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f48940b5140 0x7f48941ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 >> 172.21.15.46:0/156009805 conn(0x7f489408d3f0 msgr2=0x7f48940ba340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:22:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 shutdown_connections 2024-05-09T03:22:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.696+0000 7f489900c640 1 -- 172.21.15.46:0/156009805 wait complete. 2024-05-09T03:22:04.710 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T03:22:04.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T03:22:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.823+0000 7f67ee555640 1 Processor -- start 2024-05-09T03:22:04.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.823+0000 7f67ee555640 1 -- start start 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.824+0000 7f67ee555640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e81615f0 0x7f67e80b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.824+0000 7f67ee555640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e80d15b0 con 0x7f67e80d0b80 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67ee555640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e8160b10 con 0x7f67e80d2cf0 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67ee555640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f67e8160df0 con 0x7f67e81615f0 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f67e80d0b80 0x7f67e80d25e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:39688/0 (socket says 172.21.15.46:39688) 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e77fe640 1 -- 172.21.15.46:0/3778571487 learned_addr learned my addr 172.21.15.46:0/3778571487 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-05-09T03:22:04.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e7fff640 1 --2- 172.21.15.46:0/3778571487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e81615f0 0x7f67e80b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176616079 0 0) 0x7f67e80d15b0 con 0x7f67e80d0b80 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d0003520 con 0x7f67e80d0b80 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 <== mon.1 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518038898 0 0) 0x7f67e8160b10 con 0x7f67e80d2cf0 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67e80d15b0 con 0x7f67e80d2cf0 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e7fff640 1 -- 172.21.15.46:0/3778571487 >> v1:172.21.15.88:6789/0 conn(0x7f67e80d2cf0 legacy=0x7f67e8163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e7fff640 1 -- 172.21.15.46:0/3778571487 >> v1:172.21.15.46:6789/0 conn(0x7f67e80d0b80 legacy=0x7f67e80d25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.825+0000 7f67e7fff640 1 -- 172.21.15.46:0/3778571487 --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e81610d0 con 0x7f67e81615f0 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.826+0000 7f67e7fff640 1 --2- 172.21.15.46:0/3778571487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e81615f0 0x7f67e80b81b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f67d80097d0 tx=0x7f67d802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d09c77fc5ea44af6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.826+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 <== mon.2 v2:172.21.15.88:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d8037070 con 0x7f67e81615f0 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.826+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 <== mon.2 v2:172.21.15.88:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f67d8032440 con 0x7f67e81615f0 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.826+0000 7f67e6ffd640 1 -- 172.21.15.46:0/3778571487 <== mon.2 v2:172.21.15.88:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d80327f0 con 0x7f67e81615f0 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 -- 172.21.15.46:0/3778571487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e81615f0 msgr2=0x7f67e80b81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 --2- 172.21.15.46:0/3778571487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e81615f0 0x7f67e80b81b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f67d80097d0 tx=0x7f67d802a210 comp rx=0 tx=0).stop 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 -- 172.21.15.46:0/3778571487 shutdown_connections 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 --2- 172.21.15.46:0/3778571487 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e81615f0 0x7f67e80b81b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 -- 172.21.15.46:0/3778571487 >> 172.21.15.46:0/3778571487 conn(0x7f67e806e190 msgr2=0x7f67e806e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:22:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 -- 172.21.15.46:0/3778571487 shutdown_connections 2024-05-09T03:22:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.827+0000 7f67ee555640 1 -- 172.21.15.46:0/3778571487 wait complete. 2024-05-09T03:22:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.829+0000 7f67ee555640 1 Processor -- start 2024-05-09T03:22:04.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.829+0000 7f67ee555640 1 -- start start 2024-05-09T03:22:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ee555640 1 --2- >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e80d0b80 0x7f67e81f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ee555640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e80d2cf0 0x7f67e815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ee555640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 0x7f67e815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ee555640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67e80d1110 con 0x7f67e81615f0 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ee555640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67e80d0f90 con 0x7f67e80d2cf0 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ee555640 1 -- --> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] -- mon_getmap magic: 0 -- 0x7f67e80d1290 con 0x7f67e80d0b80 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ecaf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 0x7f67e815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ecaf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 0x7f67e815ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49942/0 (socket says 172.21.15.46:49942) 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67ecaf2640 1 -- 172.21.15.46:0/1640702477 learned_addr learned my addr 172.21.15.46:0/1640702477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67e7fff640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e80d0b80 0x7f67e81f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.830+0000 7f67e77fe640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e80d2cf0 0x7f67e815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ecaf2640 1 -- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e80d0b80 msgr2=0x7f67e81f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ecaf2640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e80d0b80 0x7f67e81f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ecaf2640 1 -- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e80d2cf0 msgr2=0x7f67e815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ecaf2640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e80d2cf0 0x7f67e815b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:04.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ecaf2640 1 -- 172.21.15.46:0/1640702477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e815ba00 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ecaf2640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 0x7f67e815ed80 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c730 tx=0x7f67e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0003b70 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f67e0003d50 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e80cab10 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0005330 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.831+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e80cb0d0 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.832+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e806a270 con 0x7f67e81615f0 2024-05-09T03:22:04.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.833+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f67e00056e0 con 0x7f67e81615f0 2024-05-09T03:22:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.833+0000 7f67e4ff9640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d40762d0 0x7f67d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T03:22:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.834+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5029+0+0 (secure 0 0 0) 0x7f67e00973c0 con 0x7f67e81615f0 2024-05-09T03:22:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.834+0000 7f67e7fff640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d40762d0 0x7f67d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T03:22:04.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.834+0000 7f67e7fff640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d40762d0 0x7f67d40787e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f67d8034000 tx=0x7f67d802e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T03:22:04.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:04.838+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f67e00608d0 con 0x7f67e81615f0 2024-05-09T03:22:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.067+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67e81f7760 con 0x7f67e81615f0 2024-05-09T03:22:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.067+0000 7f67e4ff9640 1 -- 172.21.15.46:0/1640702477 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 362 v362) ==== 95+0+3866 (secure 0 0 0) 0x7f67e0064bd0 con 0x7f67e81615f0 2024-05-09T03:22:05.069 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-05-09T03:22:05.069 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":362,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8312,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3239252621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3239252621},{"type":"v1","addr":"172.21.15.88:6836","nonce":3239252621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":8321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/26315009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":26315009},{"type":"v1","addr":"172.21.15.88:6837","nonce":26315009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":9544,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1662582600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1662582600},{"type":"v1","addr":"172.21.15.46:6834","nonce":1662582600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":348},{"gid":11088,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/976606017","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":976606017},{"type":"v1","addr":"172.21.15.46:6835","nonce":976606017}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":362}],"filesystems":[]} 2024-05-09T03:22:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 362 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d40762d0 msgr2=0x7f67d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d40762d0 0x7f67d40787e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f67d8034000 tx=0x7f67d802e000 comp rx=0 tx=0).stop 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 msgr2=0x7f67e815ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 0x7f67e815ed80 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c730 tx=0x7f67e000cbf0 comp rx=0 tx=0).stop 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 shutdown_connections 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:6832/663422477,v1:172.21.15.88:6833/663422477] conn(0x7f67d40762d0 0x7f67d40787e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e81615f0 0x7f67e815ed80 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e80d2cf0 0x7f67e815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 --2- 172.21.15.46:0/1640702477 >> [v2:172.21.15.88:3301/0,v1:172.21.15.88:6790/0] conn(0x7f67e80d0b80 0x7f67e81f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 >> 172.21.15.46:0/1640702477 conn(0x7f67e806e190 msgr2=0x7f67e8161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 shutdown_connections 2024-05-09T03:22:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-05-09T03:22:05.070+0000 7f67ee555640 1 -- 172.21.15.46:0/1640702477 wait complete. 2024-05-09T03:22:05.083 DEBUG:tasks.cephfs.filesystem:already dead... 2024-05-09T03:22:05.083 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-09T03:22:05.083 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T03:22:05.083 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.093 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T03:22:05.093 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T03:22:05.093 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.146 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T03:22:05.146 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T03:22:05.146 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.158 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T03:22:05.158 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T03:22:05.158 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.210 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T03:22:05.210 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-09T03:22:05.210 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-09T03:22:05.211 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.237 INFO:tasks.ceph.osd.0:Stopped 2024-05-09T03:22:05.237 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-09T03:22:05.237 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.309 INFO:tasks.ceph.osd.1:Stopped 2024-05-09T03:22:05.309 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-09T03:22:05.309 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.373 INFO:tasks.ceph.osd.2:Stopped 2024-05-09T03:22:05.374 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-09T03:22:05.374 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.447 INFO:tasks.ceph.osd.3:Stopped 2024-05-09T03:22:05.447 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-09T03:22:05.447 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.478 INFO:tasks.ceph.osd.4:Stopped 2024-05-09T03:22:05.479 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-09T03:22:05.479 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.561 INFO:tasks.ceph.osd.5:Stopped 2024-05-09T03:22:05.562 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-09T03:22:05.562 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.604 INFO:tasks.ceph.osd.6:Stopped 2024-05-09T03:22:05.605 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-09T03:22:05.605 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.647 INFO:tasks.ceph.osd.7:Stopped 2024-05-09T03:22:05.647 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-09T03:22:05.647 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-09T03:22:05.647 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.674 INFO:tasks.ceph.mgr.y:Stopped 2024-05-09T03:22:05.674 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-09T03:22:05.674 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.705 INFO:tasks.ceph.mgr.x:Stopped 2024-05-09T03:22:05.706 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-09T03:22:05.706 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T03:22:05.706 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.727 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T03:22:05.727 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T03:22:05.727 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.759 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T03:22:05.760 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T03:22:05.760 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T03:22:05.812 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T03:22:05.813 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-09T03:22:05.813 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'Error loading MDS rank' | egrep -v 'missing journal object' | egrep -v 'Error recovering journal' | egrep -v 'error decoding table object' | egrep -v 'failed to read JournalPointer' | egrep -v 'Corrupt directory entry' | egrep -v 'Corrupt fnode header' | egrep -v 'corrupt sessionmap header' | egrep -v 'Corrupt dentry' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_READ_ONLY | egrep -v 'force file system read-only' | egrep -v 'with standby daemon mds' | egrep -v 'MDS abort because newly corrupt dentry' | head -n 1 2024-05-09T03:22:05.859 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:22:05.860 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-09T03:22:06.025 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:22:06.026 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-09T03:22:06.089 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:22:06.089 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-09T03:22:06.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:22:06.140 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-09T03:22:06.206 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:22:06.206 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-09T03:22:06.368 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:22:06.368 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-09T03:22:06.442 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:22:06.442 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-09T03:22:06.508 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:22:06.509 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-09T03:22:06.565 INFO:tasks.ceph:Archiving mon data... 2024-05-09T03:22:06.566 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/data/mon.a.tgz 2024-05-09T03:22:06.566 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-05-09T03:22:06.583 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.ibYiwnjxHL 2024-05-09T03:22:06.583 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ibYiwnjxHL 2024-05-09T03:22:06.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.ibYiwnjxHL 2024-05-09T03:22:06.965 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.ibYiwnjxHL is 601KB 2024-05-09T03:22:07.066 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.ibYiwnjxHL 2024-05-09T03:22:07.082 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/data/mon.b.tgz 2024-05-09T03:22:07.082 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-05-09T03:22:07.102 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.Lw5Bopbo0m 2024-05-09T03:22:07.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Lw5Bopbo0m 2024-05-09T03:22:07.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.Lw5Bopbo0m 2024-05-09T03:22:07.410 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.Lw5Bopbo0m is 622KB 2024-05-09T03:22:07.510 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.Lw5Bopbo0m 2024-05-09T03:22:07.526 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/data/mon.c.tgz 2024-05-09T03:22:07.526 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-05-09T03:22:07.580 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.OZkLo16fH2 2024-05-09T03:22:07.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.OZkLo16fH2 2024-05-09T03:22:07.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.OZkLo16fH2 2024-05-09T03:22:07.848 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.OZkLo16fH2 is 622KB 2024-05-09T03:22:07.949 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.OZkLo16fH2 2024-05-09T03:22:07.964 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-09T03:22:07.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T03:22:07.967 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T03:22:08.054 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T03:22:08.084 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T03:22:08.084 INFO:tasks.ceph:Archiving crash dumps... 2024-05-09T03:22:08.086 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/remote/smithi046/crash 2024-05-09T03:22:08.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T03:22:08.164 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/remote/smithi088/crash 2024-05-09T03:22:08.165 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T03:22:08.188 INFO:tasks.ceph:Shutting down logrotate 2024-05-09T03:22:08.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:22:08.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T03:22:08.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T03:22:08.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T03:22:08.347 INFO:tasks.ceph:Compressing logs... 2024-05-09T03:22:08.348 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T03:22:08.366 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T03:22:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85124.log 2024-05-09T03:22:08.429 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35801.log 2024-05-09T03:22:08.429 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42129.log 2024-05-09T03:22:08.429 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35801.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.3.42129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35719.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35801.log.gz 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36010.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.3.42129.log.gz 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35719.log.gz 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41676.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.36010.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36010.log.gz 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.1.41676.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.1.41676.log.gz 2024-05-09T03:22:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35970.log 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35970.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35970.log.gz 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35850.log 2024-05-09T03:22:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.6.log: 92.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-05-09T03:22:08.447 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35930.log 2024-05-09T03:22:08.447 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35890.log 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35850.log.gz 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35930.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35930.log.gz 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35759.log 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35890.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35890.log.gz 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.35759.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-09T03:22:08.448 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.d.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35759.log.gz 2024-05-09T03:22:08.472 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.b.log: 91.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70351.log 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78418.log 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85124.log.gz 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62712.log 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78418.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70351.log.gz 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61636.log 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62712.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78418.log.gz 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67182.log 2024-05-09T03:22:08.564 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62712.log.gz 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45544.log 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61636.log.gz 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.67182.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.75287.log 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45544.log: gzip 87.9% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.67182.log.gz /var/log/ceph/ceph-client.admin.86640.log 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45544.log.gz 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41370.log 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75287.log.gz 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61996.log 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41370.log: gzip 89.2% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.86640.log.gz /var/log/ceph/ceph-client.admin.90473.log 2024-05-09T03:22:08.565 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41370.log.gz 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78665.log 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61996.log.gz 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64271.log 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78665.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90473.log.gz 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72705.log 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64271.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78665.log.gz 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.64271.log.gz 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69682.log 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72705.log: /var/log/ceph/ceph-client.admin.44913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72705.log.gz 2024-05-09T03:22:08.566 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64645.log 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69682.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57653.log 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64645.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44451.log 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64645.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48031.log 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57653.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70581.log 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44451.log: /var/log/ceph/ceph-client.admin.48031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45851.log 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44451.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48031.log.gz 2024-05-09T03:22:08.567 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58301.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70581.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76241.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58301.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45851.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60774.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58301.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90599.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76241.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64691.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90599.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60774.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69192.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.90599.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85443.log 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64691.log.gz 2024-05-09T03:22:08.568 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57695.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69192.log.gz 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83880.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85443.log.gz 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77635.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57695.log.gz 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86083.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83880.log.gz 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76469.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77635.log.gz 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54661.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86083.log.gz 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2024-05-09T03:22:08.569 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76469.log.gz 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53150.log 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55359.log 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54661.log.gz 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85326.log 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53150.log: /var/log/ceph/ceph-client.admin.55359.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.61324.log 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53150.log.gz 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85326.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55359.log.gz 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61121.log 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85326.log.gz 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59768.log 2024-05-09T03:22:08.570 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61324.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75735.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61121.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50930.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59768.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78885.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75735.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88619.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50930.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75142.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78885.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69435.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88619.log.gz 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81420.log 2024-05-09T03:22:08.571 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42546.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40970.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81420.log.gz 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61720.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.75142.log.gz 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42546.log.gz 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51190.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40970.log: /var/log/ceph/ceph-client.admin.61720.log: gzip 92.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.69435.log.gz -- 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.46341.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40970.log.gz 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.61720.log.gz 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.51190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87754.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67066.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46341.log.gz 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53855.log 2024-05-09T03:22:08.572 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87754.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74535.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53855.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67066.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80888.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53855.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89437.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74535.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52081.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89437.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80888.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60900.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.89437.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86892.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.0.51190.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52081.log.gz 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42816.log 2024-05-09T03:22:08.573 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86892.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60900.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65615.log 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86892.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66049.log 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42816.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49335.log 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65615.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60480.log 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66049.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60606.log 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44493.log 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49335.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60480.log.gz 2024-05-09T03:22:08.574 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77789.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60606.log.gz 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71593.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77789.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44493.log.gz 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58641.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77789.log.gz/var/log/ceph/ceph-client.admin.71593.log: 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71635.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71593.log.gz 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46275.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58641.log.gz 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43900.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.46275.log: /var/log/ceph/ceph-client.admin.53645.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63346.log 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43900.log.gz 2024-05-09T03:22:08.575 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53645.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.46275.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64012.log 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53645.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.71635.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79399.log 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63346.log: /var/log/ceph/ceph-client.admin.64012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49774.log 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63346.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79399.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.64012.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87880.log 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79399.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73875.log 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49774.log: /var/log/ceph/ceph-client.admin.87880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86987.log 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49774.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73875.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.87880.log.gz 2024-05-09T03:22:08.576 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53603.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73875.log.gz 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72060.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53603.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86987.log.gz 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73093.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53603.log.gz 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70258.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52933.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.72060.log.gz 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60564.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70258.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73093.log.gz 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81065.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60564.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52933.log.gz 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84039.log 2024-05-09T03:22:08.577 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60564.log.gz 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64049.log 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70258.log.gz 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81065.log.gz 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.84039.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53771.log 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64049.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84039.log.gz 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47664.log 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78299.log 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.64049.log.gz 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53771.log.gz 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.47664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45416.log 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57569.log 2024-05-09T03:22:08.578 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78299.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88973.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45416.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57569.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88973.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57569.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73787.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89984.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.88973.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73787.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.47664.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44315.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73787.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89984.log: /var/log/ceph/ceph-client.admin.44315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66675.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45039.log 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41328.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44315.log.gz 2024-05-09T03:22:08.579 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66675.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.89984.log.gz 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70118.log 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66675.log.gz 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52597.log 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70118.log: gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.45039.log.gz --verbose 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.47136.log 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70118.log.gz 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45670.log 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47136.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52597.log.gz 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57424.log 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45670.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.47136.log.gz -5 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.85611.log 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T03:22:08.580 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45670.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81653.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85611.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57424.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70403.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81653.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.85611.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72393.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81653.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49949.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70403.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66846.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72393.log: /var/log/ceph/ceph-client.admin.49949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72276.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72393.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49949.log.gz 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46065.log 2024-05-09T03:22:08.581 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66846.log.gz 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71840.log 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46065.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72276.log.gz -5 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45374.log 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46065.log.gz 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46900.log 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71840.log.gz 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45374.log.gz 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60648.log 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46900.log.gz 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80590.log 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60648.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-05-09T03:22:08.582 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58118.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60648.log.gz 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.80590.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49125.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.58118.log.gz 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.75235.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.0.80590.log.gz 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49377.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75235.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.49125.log.gz --verbose 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.87544.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75235.log.gz 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50254.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49377.log.gz 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47232.log 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87544.log.gz 2024-05-09T03:22:08.583 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68637.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47232.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50254.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84362.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47232.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.68637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60396.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63901.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84362.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72939.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.68637.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60396.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61804.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63901.log: /var/log/ceph/ceph-client.admin.72939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51474.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63901.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72939.log.gz 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49209.log 2024-05-09T03:22:08.584 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61804.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59250.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51474.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40928.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59250.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49209.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72883.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59250.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41880.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40928.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67235.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84812.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.72883.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41880.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54987.log 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67235.log.gz 2024-05-09T03:22:08.585 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67089.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.54987.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84812.log.gz 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42217.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41548.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55650.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42217.log.gz 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90092.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87712.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55650.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41548.log.gz 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.0.54987.log.gz 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.67089.log.gz 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90092.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55650.log.gz 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51551.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86682.log 2024-05-09T03:22:08.586 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.87712.log.gz -- replaced with /var/log/ceph/ceph-client.admin.90092.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68293.log 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51551.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80176.log 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68293.log: gzip -5 --verbose -- 89.3% /var/log/ceph/ceph-client.admin.88789.log -- replaced with /var/log/ceph/ceph-client.admin.86682.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57382.log 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.68293.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80176.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76917.log 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88789.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87414.log 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57382.log.gz 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50594.log 2024-05-09T03:22:08.587 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76917.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82418.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50594.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.87414.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69580.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50594.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80030.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82418.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56046.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80030.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69580.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60690.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67389.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56046.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62256.log 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67389.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60690.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.80030.log.gz 2024-05-09T03:22:08.588 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45502.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67389.log.gz 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51931.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.62256.log: /var/log/ceph/ceph-client.admin.45502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54131.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45502.log.gz 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63086.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51931.log: /var/log/ceph/ceph-client.admin.54131.log: gzip 88.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.51931.log.gz -- 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.80764.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63086.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54131.log.gz 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53070.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.63086.log.gz 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57340.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53070.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.80764.log.gz -5 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.81706.log 2024-05-09T03:22:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57340.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53070.log.gz 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64499.log 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57340.log.gz 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.81706.log: /var/log/ceph/ceph-client.admin.87067.log 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81706.log.gz/var/log/ceph/ceph-client.admin.64499.log: 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53813.log 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87067.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87067.log.gz 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68096.log 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.54584.log -- replaced with /var/log/ceph/ceph-client.admin.53813.log.gz 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.64499.log.gz 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60984.log 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89942.log/var/log/ceph/ceph-client.admin.54584.log: 2024-05-09T03:22:08.590 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68096.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60984.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54584.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84947.log 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60984.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.0.62256.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/ceph-client.admin.89942.log: --verbose -- /var/log/ceph/ceph-client.admin.55412.log 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47386.log 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89942.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55412.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84947.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75895.log 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55412.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50030.log 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47386.log: /var/log/ceph/ceph-client.admin.75895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67281.log 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47386.log.gz 2024-05-09T03:22:08.591 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75895.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50030.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73327.log 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67281.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72846.log 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63309.log 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72846.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73327.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63568.log 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79071.log 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63568.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63309.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63568.log.gz 2024-05-09T03:22:08.592 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.72846.log.gz 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51877.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91601.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51877.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79071.log.gz -5 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58206.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51877.log.gz 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86766.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58206.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91601.log.gz 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67126.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86766.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58206.log.gz 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65385.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86766.log.gz 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81560.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72458.log 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65385.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67126.log.gz 2024-05-09T03:22:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86167.log 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65385.log.gz 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53557.log 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86598.log 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86167.log.gz 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88681.log 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.81560.log.gz 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53557.log.gz 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73959.log 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86598.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.72458.log.gz 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77348.log 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86598.log.gz 2024-05-09T03:22:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50384.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.88681.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.73959.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77348.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45756.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53939.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50384.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87321.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79619.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87321.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45756.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67898.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79619.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87321.log.gzgzip 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66347.log 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79619.log.gz 2024-05-09T03:22:08.595 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81107.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67898.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82939.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75777.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.66347.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82674.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82939.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63494.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75777.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41551.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82674.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.81107.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56721.log 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82674.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41551.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63494.log.gz 2024-05-09T03:22:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70295.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41551.log.gz 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.56721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55920.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60942.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55920.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70295.log.gz 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78553.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55920.log.gz 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77582.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60942.log: /var/log/ceph/ceph-client.admin.78553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60942.log.gz 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78553.log.gz 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91517.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77582.log.gz 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75389.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44871.log: /var/log/ceph/ceph-client.admin.91517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44567.log 2024-05-09T03:22:08.597 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91517.log.gz/var/log/ceph/ceph-client.admin.75389.log: 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80330.log 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75389.log.gz 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62975.log 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44567.log: 93.3% -- replaced with /var/log/ceph/ceph-client.0.56721.log.gz 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80330.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44567.log.gz 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56177.log 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80330.log.gz 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45981.log 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91391.log 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62975.log.gz 2024-05-09T03:22:08.598 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56177.log.gz 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71473.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91391.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45981.log.gz 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61282.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71473.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91391.log.gz 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53273.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71473.log.gz 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74899.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61282.log.gz/var/log/ceph/ceph-client.admin.53273.log: 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80067.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74899.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53273.log.gz 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69084.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74899.log.gz 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85401.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89761.log 2024-05-09T03:22:08.599 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80067.log.gz 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85401.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.69084.log.gz 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51671.log 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85401.log.gz 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66633.log 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67323.log 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.89761.log.gz 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51671.log.gz 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82873.log 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66633.log.gz 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76417.log 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53473.log 2024-05-09T03:22:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76417.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.67323.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73440.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.82873.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69370.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76417.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53473.log: /var/log/ceph/ceph-client.admin.73440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85965.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53473.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73440.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69370.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75507.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85965.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86262.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64536.log 2024-05-09T03:22:08.601 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86262.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75507.log.gz 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73618.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86262.log.gz 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91220.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73618.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.64536.log.gz -5 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91220.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73618.log.gz 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66792.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91220.log.gz 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65506.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75077.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66792.log.gz 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87586.log 2024-05-09T03:22:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77489.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75077.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65506.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87586.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75077.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52410.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87586.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78707.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77489.log: /var/log/ceph/ceph-client.admin.52410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55454.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52410.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72116.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78707.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56004.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52452.log 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55454.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72116.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.77489.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56004.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72116.log.gz 2024-05-09T03:22:08.603 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62555.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56004.log.gz 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90050.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52452.log: /var/log/ceph/ceph-client.admin.62555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48798.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52452.log.gz 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90050.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62555.log.gz 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49437.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90050.log.gz 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90389.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48798.log: /var/log/ceph/ceph-client.admin.49437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49606.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48798.log.gz 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49437.log.gz/var/log/ceph/ceph-client.admin.90389.log: 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44997.log 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90389.log.gz 2024-05-09T03:22:08.604 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68886.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49606.log.gz 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73917.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44997.log: /var/log/ceph/ceph-client.admin.68886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66109.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44997.log.gz 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73917.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68886.log.gz 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52975.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65308.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73917.log.gz 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66109.log: /var/log/ceph/ceph-client.admin.52975.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66109.log.gz 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54727.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52975.log.gz 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81597.log 2024-05-09T03:22:08.605 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65308.log: /var/log/ceph/ceph-client.admin.54727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71728.log 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65308.log.gz 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54727.log.gz 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91855.log 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.81597.log.gz 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76529.log 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.70856.log 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71728.log.gz 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91855.log: /var/log/ceph/ceph-client.admin.76529.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91855.log.gz 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88843.log 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76529.log.gz 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60858.log 2024-05-09T03:22:08.606 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70856.log: /var/log/ceph/ceph-client.admin.88843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87277.log 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70856.log.gz 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.88843.log.gz 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89564.log 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60858.log.gz 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83922.log 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87277.log: /var/log/ceph/ceph-client.admin.89564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85713.log 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87277.log.gz 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.89564.log.gz 2024-05-09T03:22:08.607 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52681.log 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83922.log.gz 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58431.log 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85713.log: /var/log/ceph/ceph-client.admin.52681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68359.log 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85713.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52681.log.gz 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85909.log 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58431.log.gz 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55962.log 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68359.log: /var/log/ceph/ceph-client.admin.85909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61930.log 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68359.log.gz 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55962.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85909.log.gz 2024-05-09T03:22:08.608 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85872.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55962.log.gz 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41118.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61930.log: /var/log/ceph/ceph-client.admin.85872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68205.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77306.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.61930.log.gz 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41118.log.gz 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48896.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91811.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77306.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68205.log.gz 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48896.log: 88.0%gzip 92.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.85872.log.gz --verbose 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-mds.c.log 2024-05-09T03:22:08.609 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77306.log.gz 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69800.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.48896.log.gz 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79965.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.91811.log.gz 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76324.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69800.log: /var/log/ceph/ceph-client.admin.79965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77723.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69800.log.gz 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79965.log.gz 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82120.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49522.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77723.log: /var/log/ceph/ceph-client.0.82120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71415.log 2024-05-09T03:22:08.610 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49522.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.77723.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75567.log 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.0.82120.log.gz 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% 92.4% -- replaced with /var/log/ceph/ceph-client.admin.76324.log.gz 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49522.log.gz 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71415.log.gz 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73380.log 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88931.log 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.75567.log.gz 2024-05-09T03:22:08.611 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73380.log: gzip 87.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz -- 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.75119.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90211.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73380.log.gz 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73678.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88931.log.gz 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48166.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90211.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75119.log.gz 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66287.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90211.log.gz 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48166.log: gzip 89.5% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.73678.log.gz 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.44703.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69758.log 2024-05-09T03:22:08.612 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66287.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.48166.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76123.log 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69758.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44703.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41884.log 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69758.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76123.log: gzip -5 --verbose -- 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76123.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.56918.log 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58599.log 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41884.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57130.log 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68059.log 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.56918.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57130.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58599.log.gz 2024-05-09T03:22:08.613 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44373.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57130.log.gz 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.68059.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44373.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44373.log.gz 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69823.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52280.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41885.log: /var/log/ceph/ceph-client.admin.69823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66699.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4%/var/log/ceph/ceph-client.admin.52280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56327.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68059.log.gz 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52280.log.gz 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78941.log 2024-05-09T03:22:08.614 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66699.log: /var/log/ceph/ceph-client.admin.56327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68828.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66557.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.69823.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.56327.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78941.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68828.log: /var/log/ceph/ceph-client.admin.66557.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.66699.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68828.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66557.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64476.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79029.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65881.log 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7%/var/log/ceph/ceph-client.admin.79029.log: -- replaced with /var/log/ceph/ceph-client.admin.64476.log.gz 2024-05-09T03:22:08.615 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79029.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43238.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64197.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42462.log: /var/log/ceph/ceph-client.0.43238.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65881.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.0.43238.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69916.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64197.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42084.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50822.log 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69916.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42084.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69916.log.gz 2024-05-09T03:22:08.616 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79642.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42084.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60354.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79642.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50822.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63049.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57996.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60354.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63049.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.79642.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63049.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68017.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59726.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57996.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62938.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68017.log.gz 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-09T03:22:08.617 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58343.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.62938.log: -- replaced with /var/log/ceph/ceph-client.admin.59726.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81200.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.62938.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81495.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58343.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81200.log: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81200.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63827.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81495.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81495.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47039.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46023.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47039.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.63827.log.gz 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79113.log 2024-05-09T03:22:08.618 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47039.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46023.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46023.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.57172.log 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67840.log 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79113.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57172.log: gzip -5 --verbose 89.4% -- /var/log/ceph/ceph-client.admin.58791.log 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57172.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45184.log 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67840.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91025.log 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58791.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89490.log 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45184.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45184.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91025.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91025.log.gz 2024-05-09T03:22:08.619 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65059.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70463.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.65059.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.89490.log.gz 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91306.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72603.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91306.log: /var/log/ceph/ceph-client.admin.70463.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70463.log.gz 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91306.log.gz 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.0.65059.log.gz 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46977.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.72603.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87233.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73051.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72603.log.gz 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75955.log 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46977.log.gz 2024-05-09T03:22:08.620 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87233.log: /var/log/ceph/ceph-client.admin.73051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58160.log 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87233.log.gz 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73051.log.gz 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75955.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75955.log.gz 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52723.log 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68251.log 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58160.log.gz 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.52723.log: /var/log/ceph/ceph-client.admin.47278.log 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62827.log 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52723.log.gz 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68251.log.gz 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73829.log 2024-05-09T03:22:08.621 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47278.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61888.log 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7%/var/log/ceph/ceph-client.admin.73829.log: -- replaced with /var/log/ceph/ceph-client.admin.62827.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73829.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.63975.log 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66245.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61888.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86304.log 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63975.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72663.log 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66245.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61846.log 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86304.log.gz 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63420.log 2024-05-09T03:22:08.622 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72663.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78783.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61846.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55179.log 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8%/var/log/ceph/ceph-client.admin.78783.log: -- replaced with /var/log/ceph/ceph-client.admin.63420.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57298.log 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78783.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90744.log 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55179.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87628.log 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57298.log: /var/log/ceph/ceph-client.admin.90744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87628.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57298.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90744.log.gz 2024-05-09T03:22:08.623 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87628.log.gz 2024-05-09T03:22:08.624 INFO:teuthology.orchestra.run.smithi088.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-09T03:22:08.877 INFO:teuthology.orchestra.run.smithi088.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-09T03:22:08.939 INFO:teuthology.orchestra.run.smithi088.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-09T03:22:08.941 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45586.log 2024-05-09T03:22:08.941 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91128.log 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45586.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63160.log 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.91128.log: -- replaced with /var/log/ceph/ceph-client.admin.45586.log.gz 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59311.log 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91128.log.gz 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61590.log 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.63160.log.gz/var/log/ceph/ceph-client.admin.59311.log: 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63864.log 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42714.log 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.59311.log.gz 2024-05-09T03:22:08.942 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61590.log.gz 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66906.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.63864.log.gz 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62864.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42714.log.gz 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66906.log.gz/var/log/ceph/ceph-client.admin.62864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60732.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85559.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62864.log.gz 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41888.log: /var/log/ceph/ceph-client.admin.60732.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52639.log 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60732.log.gz 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85559.log: 2024-05-09T03:22:08.943 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85559.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63383.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79787.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52639.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76011.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79787.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63383.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67649.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79787.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65469.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76011.log: /var/log/ceph/ceph-client.0.67649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84099.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65469.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76011.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74670.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84099.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.67649.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87372.log 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.65469.log.gz 2024-05-09T03:22:08.944 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84099.log.gz 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59558.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74670.log.gz 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42223.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87372.log: /var/log/ceph/ceph-client.admin.59558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59516.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87372.log.gz 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59558.log.gz 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76959.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42223.log.gz/var/log/ceph/ceph-client.admin.59516.log: gzip 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.85242.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.59516.log.gz /var/log/ceph/ceph-client.admin.66515.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76959.log: gzip/var/log/ceph/ceph-client.admin.85242.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51629.log 2024-05-09T03:22:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66515.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85242.log.gz 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76959.log.gz 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66515.log.gz 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45081.log 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77424.log 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51629.log.gz 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83702.log 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45081.log.gz 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72495.log 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77424.log: /var/log/ceph/ceph-client.admin.83702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70076.log 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72495.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.77424.log.gz 2024-05-09T03:22:08.946 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83702.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.72495.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89606.log 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81023.log 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70076.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79459.log 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45939.log 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81023.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.89606.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79459.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81023.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85068.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79459.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52368.log 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45939.log.gz 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64799.log 2024-05-09T03:22:08.947 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.85068.log.gz 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90941.log 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52368.log: /var/log/ceph/ceph-client.admin.64799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58725.log 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%/var/log/ceph/ceph-client.admin.90941.log: -- replaced with /var/log/ceph/ceph-client.admin.52368.log.gz 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64799.log.gz 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.90941.log.gz 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90347.log 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48854.log 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82595.log 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.58725.log.gz 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.90347.log.gz 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46601.log 2024-05-09T03:22:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45628.log 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48854.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82595.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59642.log 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.46601.log: /var/log/ceph/ceph-client.admin.45628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76571.log 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59642.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45628.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59642.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83997.log 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76571.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66403.log 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83997.log: 92.6% -- replaced with /var/log/ceph/ceph-client.0.46601.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.83997.log.gz 2024-05-09T03:22:08.949 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65904.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69047.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66403.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42222.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69047.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.65904.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81378.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.69047.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89876.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81378.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42222.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83755.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81378.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.89876.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63235.log 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83755.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.89876.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83755.log.gz 2024-05-09T03:22:08.950 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72000.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75675.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.63235.log.gz 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81860.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72000.log.gz/var/log/ceph/ceph-client.admin.75675.log: 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75675.log.gz 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44955.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66310.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%/var/log/ceph/ceph-client.admin.44955.log: -- replaced with /var/log/ceph/ceph-client.admin.81860.log.gz 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63605.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44955.log.gz 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74754.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63605.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.63605.log.gz 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59473.log 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.66310.log.gz 2024-05-09T03:22:08.951 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58076.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59473.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59473.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.84622.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58076.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.74754.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58076.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46842.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.84622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61548.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46842.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49869.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.84622.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44010.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49869.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61548.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82783.log 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49869.log.gz 2024-05-09T03:22:08.952 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46107.log 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82783.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44010.log.gz 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82783.log.gz 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55500.log 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53404.log 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46107.log.gz 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87149.log 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53404.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55500.log.gz 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53404.log.gz 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85284.log 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83609.log 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85284.log: /var/log/ceph/ceph-client.admin.87149.log: 88.9% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87149.log.gz 2024-05-09T03:22:08.953 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85284.log.gz 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60242.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45712.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60242.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60242.log.gz 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70898.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4%/var/log/ceph/ceph-client.admin.45712.log: -- replaced with /var/log/ceph/ceph-client.admin.83609.log.gz 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45712.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75623.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65427.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70898.log: /var/log/ceph/ceph-client.admin.75623.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75623.log.gz 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35770.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.70898.log.gz 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65427.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65427.log.gz 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44787.log 2024-05-09T03:22:08.954 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65764.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35770.log.gz 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71780.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44787.log.gz 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76361.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65764.log.gz 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86125.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71780.log.gz/var/log/ceph/ceph-client.admin.76361.log: 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88885.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5%/var/log/ceph/ceph-client.admin.86125.log: -- replaced with /var/log/ceph/ceph-client.admin.76361.log.gz 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76063.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86125.log.gz 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57821.log 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.76063.log: -- replaced with /var/log/ceph/ceph-client.admin.88885.log.gz 2024-05-09T03:22:08.955 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76063.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75347.log 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41545.log 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75347.log: /var/log/ceph/ceph-client.admin.57821.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57821.log.gz 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75347.log.gz 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49564.log 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77137.log 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41545.log.gz 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84296.log 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77137.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49564.log.gz 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91685.log 2024-05-09T03:22:08.956 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.77137.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85466.log 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91685.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91685.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.84296.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81336.log 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48642.log 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81336.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81336.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43858.log 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.48642.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.85466.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62790.log 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43858.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43858.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29387.log 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62790.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.48642.log.gz 2024-05-09T03:22:08.957 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81537.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.29387.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29387.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.62790.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54065.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71672.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81537.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67024.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54065.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42221.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.71672.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44052.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67024.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83838.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44052.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42221.log.gz 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65722.log 2024-05-09T03:22:08.958 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44052.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79847.log 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65722.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83838.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65722.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78605.log 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70792.log 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79847.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80264.log 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78605.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70739.log 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80264.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70792.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91085.log 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.80264.log.gz 2024-05-09T03:22:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70193.log 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91085.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70739.log.gz 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91937.log 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91085.log.gz 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91937.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70193.log.gz 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91937.log.gz 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83199.log 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85031.log 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63790.log: /var/log/ceph/ceph-client.0.83199.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69412.log 2024-05-09T03:22:08.960 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85031.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.83199.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63272.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69412.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69412.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73271.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63272.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.85031.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91475.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63272.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85671.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91475.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.73271.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.91475.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74712.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85849.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85671.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70964.log 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85849.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74712.log.gz 2024-05-09T03:22:08.961 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85849.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69140.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70964.log: /var/log/ceph/ceph-client.admin.42588.log: 88.3% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70964.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53981.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55608.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69140.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61026.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55608.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53981.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42224.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55608.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55726.log 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42224.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61026.log.gz 2024-05-09T03:22:08.962 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42224.log.gz 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52123.log 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.55726.log.gz 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65839.log 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52123.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72234.log 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52123.log.gz 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59952.log 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72234.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65839.log.gz 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72234.log.gz 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71958.log 2024-05-09T03:22:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/ceph-client.admin.59952.log: --verbose -- /var/log/ceph/ceph-client.admin.73734.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59952.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.71958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50338.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71958.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78460.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50338.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73734.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65562.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50338.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44625.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65562.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65562.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60816.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44625.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.78460.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44625.log.gz 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50426.log 2024-05-09T03:22:08.964 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60816.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72318.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4%/var/log/ceph/ceph-client.admin.50426.log: -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50426.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65941.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84254.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65941.log: /var/log/ceph/ceph-client.admin.72318.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72318.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.65941.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73482.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84254.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50636.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5%/var/log/ceph/ceph-client.admin.73482.log: -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73482.log.gz 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91170.log 2024-05-09T03:22:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78376.log 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91170.log: /var/log/ceph/ceph-client.admin.50636.log: 88.4% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50636.log.gz 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91170.log.gz 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89039.log 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87502.log 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78376.log.gz 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90702.log 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89039.log.gz 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65680.log 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87502.log.gz 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81794.log 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3%/var/log/ceph/ceph-client.admin.65680.log: -- replaced with /var/log/ceph/ceph-client.admin.90702.log.gz 2024-05-09T03:22:08.966 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65680.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84208.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49648.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84208.log: /var/log/ceph/ceph-client.admin.81794.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84208.log.gz 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.81794.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50780.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83390.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49251.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83390.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50780.log.gz 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83390.log.gz 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69472.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52807.log 2024-05-09T03:22:08.967 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69472.log: /var/log/ceph/ceph-client.admin.49251.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49251.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.69472.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57527.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.89299.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52807.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47062.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.89299.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57527.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47062.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.47062.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.89299.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63457.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41543.log: -- /var/log/ceph/ceph-client.admin.63012.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62735.log 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63012.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.63457.log.gz 2024-05-09T03:22:08.968 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63012.log.gz 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75001.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66151.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75001.log: /var/log/ceph/ceph-client.admin.62735.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75001.log.gz 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78241.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66151.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66151.log.gz 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.62735.log.gz 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55256.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79577.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55256.log: /var/log/ceph/ceph-client.admin.78241.log: 88.0% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78241.log.gz 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55256.log.gz 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64434.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77193.log 2024-05-09T03:22:08.969 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79577.log.gz 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90983.log 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77193.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64434.log.gz 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56239.log 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77193.log.gz 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61366.log 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3%/var/log/ceph/ceph-client.admin.56239.log: -- replaced with /var/log/ceph/ceph-client.admin.90983.log.gz 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56239.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81752.log 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61366.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53687.log 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61366.log.gz 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91262.log 2024-05-09T03:22:08.970 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2%/var/log/ceph/ceph-client.admin.53687.log: -- replaced with /var/log/ceph/ceph-client.admin.81752.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53687.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79291.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91262.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91262.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.80123.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79889.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80123.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.79291.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80123.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91643.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54391.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79889.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75530.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.54391.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91643.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75530.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.75530.log.gz 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46149.log 2024-05-09T03:22:08.971 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41552.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.0.54391.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61455.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46149.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64086.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61455.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61455.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59994.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77681.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59994.log: /var/log/ceph/ceph-client.admin.64086.log: 88.4% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64086.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59994.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59889.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91769.log 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77681.log.gz 2024-05-09T03:22:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75465.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91769.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59889.log.gz 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72781.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91769.log.gz 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74847.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75465.log: /var/log/ceph/ceph-client.admin.72781.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72781.log.gz 88.7% 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75465.log.gz 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74593.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70505.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74847.log.gz 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66455.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70505.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74593.log.gz 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89672.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70505.log.gz 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77077.log 2024-05-09T03:22:08.973 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66455.log: /var/log/ceph/ceph-client.admin.89672.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66455.log.gz 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.89672.log.gz 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55298.log 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85789.log 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77077.log.gz 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86724.log 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55298.log.gz 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35667.log 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85789.log.gz 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60522.log 2024-05-09T03:22:08.974 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5%/var/log/ceph/ceph-client.admin.35667.log: -- replaced with /var/log/ceph/ceph-client.admin.86724.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74791.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35667.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50203.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74791.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60522.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74791.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51775.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74959.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50203.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87670.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74959.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51775.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52891.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74959.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79254.log 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52891.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87670.log.gz 2024-05-09T03:22:08.975 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52891.log.gz 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59684.log 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90305.log 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59684.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59684.log.gz 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55711.log 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90305.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.79254.log.gz 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90305.log.gz 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79501.log 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76647.log 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79501.log: 92.0% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79501.log.gz 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55711.log.gz 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88484.log 2024-05-09T03:22:08.976 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48232.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76647.log.gz 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68152.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48232.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.88484.log.gz 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69969.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8%/var/log/ceph/ceph-client.admin.68152.log: -- replaced with /var/log/ceph/ceph-client.admin.48232.log.gz 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68152.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.59051.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83646.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69969.log.gz 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.59051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79347.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83646.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.83646.log.gz 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78497.log 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79347.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79347.log.gz 2024-05-09T03:22:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62497.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78497.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78497.log.gz 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53897.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62497.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62497.log.gz 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51733.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53897.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53897.log.gz 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51733.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51733.log.gz 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46233.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45142.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86514.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46233.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46233.log.gz 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64160.log 2024-05-09T03:22:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86514.log: 89.2% 93.4% -- replaced with /var/log/ceph/ceph-client.0.59051.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86514.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70683.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76749.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.64160.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80806.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76749.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.70683.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62650.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76749.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72823.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80806.log: /var/log/ceph/ceph-client.admin.62650.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62650.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.80806.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70646.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.72823.log: -- /var/log/ceph/ceph-client.admin.82730.log 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72823.log.gz 2024-05-09T03:22:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65250.log 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82730.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82730.log.gz 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77466.log 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65250.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70646.log.gz 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65250.log.gz 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50072.log 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63716.log 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77466.log.gz 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57737.log 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63716.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50072.log.gz 2024-05-09T03:22:08.980 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52238.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63716.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.57737.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.74067.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57737.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61411.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74067.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52238.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49032.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74067.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67975.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61411.log: /var/log/ceph/ceph-client.admin.49032.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61411.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49032.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82830.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.67975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63198.log 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67975.log.gz 2024-05-09T03:22:08.981 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61762.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63198.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82830.log.gz 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86472.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.63198.log.gz 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76301.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61762.log.gz 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69528.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76301.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86472.log.gz 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76301.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64733.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.69528.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44829.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69528.log.gz 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63679.log 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44829.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.64733.log.gz 2024-05-09T03:22:08.982 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44829.log.gz 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79231.log 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63679.log.gz 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53315.log 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%/var/log/ceph/ceph-client.admin.79231.log: -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79231.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41881.log 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53515.log 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41881.log: /var/log/ceph/ceph-client.admin.53315.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53315.log.gz 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41881.log.gz 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76857.log 2024-05-09T03:22:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53515.log.gz 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57779.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76857.log.gz 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73234.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz/var/log/ceph/ceph-client.admin.57779.log: 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84989.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8%/var/log/ceph/ceph-client.admin.73234.log: -- replaced with /var/log/ceph/ceph-client.admin.57779.log.gz 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52039.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84989.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84989.log.gz 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.73234.log.gz 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51835.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69294.log 2024-05-09T03:22:08.984 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52039.log.gz 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52765.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51835.log.gz 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69294.log.gz 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63123.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.42672.log: -- replaced with /var/log/ceph/ceph-client.admin.52765.log.gz 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45458.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81144.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45458.log: /var/log/ceph/ceph-client.admin.63123.log: 88.1% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63123.log.gz 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45458.log.gz 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87814.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85177.log 2024-05-09T03:22:08.985 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.81144.log.gz 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57485.log 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85177.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.87814.log.gz 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42220.log 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.85177.log.gz 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91348.log 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57485.log: /var/log/ceph/ceph-client.admin.42220.log: 88.0% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57485.log.gz 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42220.log.gz 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50296.log 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50678.log 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91348.log.gz 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47855.log 2024-05-09T03:22:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50678.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50296.log.gz 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.50678.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90786.log 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41202.log: /var/log/ceph/ceph-client.admin.47855.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47855.log.gz 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57214.log 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40833.log 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90786.log.gz 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43736.log 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40833.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57214.log.gz 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41886.log 2024-05-09T03:22:08.987 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40833.log.gz 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.2.43736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91433.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41886.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41886.log.gz 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50552.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.2.43736.log.gz 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91433.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42219.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.91433.log.gz 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75853.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4%/var/log/ceph/ceph-client.admin.42219.log: -- replaced with /var/log/ceph/ceph-client.admin.50552.log.gz 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42219.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49293.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79735.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75853.log.gz 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83448.log 2024-05-09T03:22:08.988 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79735.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49293.log.gz 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79735.log.gz 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57916.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83448.log.gz 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57916.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57916.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86430.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47897.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86430.log: /var/log/ceph/ceph-client.admin.49732.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86430.log.gz 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52536.log 2024-05-09T03:22:08.989 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40877.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.47897.log.gz 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86808.log 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53231.log 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52536.log.gz 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86388.log 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53231.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86808.log.gz 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51366.log 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53231.log.gz 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56285.log 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86388.log.gz/var/log/ceph/ceph-client.admin.51366.log: 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75918.log 2024-05-09T03:22:08.990 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56285.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.51366.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56285.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76712.log 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73169.log 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76712.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.75918.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86556.log 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr: 92.3%/var/log/ceph/ceph-client.admin.73169.log: -- replaced with /var/log/ceph/ceph-client.admin.76712.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80222.log 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73169.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86556.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86556.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74025.log 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81253.log 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80222.log.gz 2024-05-09T03:22:08.991 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.74025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84062.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.81253.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74025.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57611.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81253.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73576.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57611.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57611.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50510.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.73576.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.84062.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73576.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80007.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89714.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80007.log: /var/log/ceph/ceph-client.admin.50510.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50510.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80007.log.gz 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46191.log 2024-05-09T03:22:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44220.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46191.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.89714.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46191.log.gz 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48316.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84155.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44220.log.gz 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61506.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84155.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84155.log.gz 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48316.log.gz 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86346.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72991.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61506.log.gz 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42630.log 2024-05-09T03:22:08.993 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72991.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.86346.log.gz 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72991.log.gz 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59600.log 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69860.log 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59600.log: /var/log/ceph/ceph-client.admin.42630.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42630.log.gz 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59600.log.gz 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90660.log 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61678.log 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69860.log.gz 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90660.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90660.log.gz/var/log/ceph/ceph-client.admin.61678.log: 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61678.log.gz 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55542.log 2024-05-09T03:22:08.994 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86018.log 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55542.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55542.log.gz 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43360.log 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86018.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86018.log.gz 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53729.log 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43360.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43360.log.gz 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45800.log 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53729.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53729.log.gz 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88704.log 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45800.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45800.log.gz 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86850.log 2024-05-09T03:22:08.995 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88704.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.88704.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60123.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.86850.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.86850.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82294.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60123.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60123.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69005.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87456.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69005.log: /var/log/ceph/ceph-client.admin.82294.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82294.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69005.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64345.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70623.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87456.log: /var/log/ceph/ceph-client.admin.64345.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87456.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41887.log 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64345.log.gz 2024-05-09T03:22:08.996 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40623.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41887.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70623.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41887.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71882.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47320.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40623.log: /var/log/ceph/ceph-client.admin.71882.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40623.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71882.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54023.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70034.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54023.log: /var/log/ceph/ceph-client.admin.47320.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54023.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.47320.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50468.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48382.log 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50468.log: /var/log/ceph/ceph-client.admin.70034.log: 87.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70034.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50468.log.gz 2024-05-09T03:22:08.997 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63938.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72023.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48382.log: /var/log/ceph/ceph-client.admin.63938.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48382.log.gz 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66948.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63938.log.gz 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50864.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66948.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66948.log.gz 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50864.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.72023.log.gz 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52322.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50864.log.gz 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68963.log 2024-05-09T03:22:08.998 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40705.log: /var/log/ceph/ceph-client.admin.52322.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72435.log 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52322.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.68963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69640.log 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.72435.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68963.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72435.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58385.log 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75179.log 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69640.log: /var/log/ceph/ceph-client.admin.58385.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69640.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49690.log 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58385.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58683.log 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.75179.log: /var/log/ceph/ceph-client.admin.49690.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.75179.log.gz 2024-05-09T03:22:08.999 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78848.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49690.log.gz 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60438.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78848.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58683.log.gz 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63531.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60438.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60438.log.gz 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.78848.log.gz 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91895.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59377.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91895.log: /var/log/ceph/ceph-client.admin.63531.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91895.log.gz 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63531.log.gz 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45293.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.59377.log: -- /var/log/ceph/ceph-client.admin.79679.log 2024-05-09T03:22:09.000 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59377.log.gz/var/log/ceph/ceph-client.admin.45293.log: 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56461.log 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45293.log.gz 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70235.log 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79679.log.gz 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90557.log 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.70235.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56461.log.gz 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77100.log 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70235.log.gz 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47190.log 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90557.log: /var/log/ceph/ceph-client.admin.77100.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90557.log.gz 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82637.log 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47190.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47190.log.gz 2024-05-09T03:22:09.001 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.77100.log.gz 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62901.log 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78825.log 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.62901.log: /var/log/ceph/ceph-client.admin.82637.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62901.log.gz 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79189.log 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.78825.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78825.log.gz 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.82637.log.gz 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76805.log 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.79189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89834.log 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79189.log.gz 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90431.log 2024-05-09T03:22:09.002 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89834.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76805.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64308.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89834.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64234.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1%/var/log/ceph/ceph-client.admin.64308.log: -- replaced with /var/log/ceph/ceph-client.admin.90431.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80946.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.64308.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76165.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.80946.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.64234.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80946.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55818.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66736.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76165.log.gz 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58473.log 2024-05-09T03:22:09.003 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.66736.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55818.log.gz 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72169.log 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66736.log.gz 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58515.log 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58473.log: /var/log/ceph/ceph-client.admin.72169.log: 88.1% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58473.log.gz 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72169.log.gz 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90515.log 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.58515.log: -- /var/log/ceph/ceph-client.admin.42420.log 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58515.log.gz 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64123.log 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90515.log.gz 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78050.log 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42420.log.gz/var/log/ceph/ceph-client.admin.64123.log: 2024-05-09T03:22:09.004 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85503.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1%/var/log/ceph/ceph-client.0.78050.log: -- replaced with /var/log/ceph/ceph-client.admin.64123.log.gz 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59208.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.85503.log: 91.5% -- replaced with /var/log/ceph/ceph-client.0.78050.log.gz 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.85503.log.gz 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91727.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59208.log.gz 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57256.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91727.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91727.log.gz 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71550.log 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57256.log.gz/var/log/ceph/ceph-client.admin.41549.log: 2024-05-09T03:22:09.005 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44745.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4%/var/log/ceph/ceph-client.admin.71550.log: -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71550.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63642.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55878.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44745.log: /var/log/ceph/ceph-client.admin.63642.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44745.log.gz 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82336.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.63642.log.gz 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83567.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82336.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55878.log.gz 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77246.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.82336.log.gz 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74327.log 2024-05-09T03:22:09.006 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77246.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83567.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49167.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77246.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.74327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64592.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49167.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49167.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.64592.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64592.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.74327.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58557.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77526.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.35403.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58557.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gz 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50159.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51973.log 2024-05-09T03:22:09.007 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50159.log: /var/log/ceph/ceph-client.admin.77526.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50159.log.gz 89.6% -- replaced with /var/log/ceph/ceph-client.admin.77526.log.gz 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.88140.log 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51973.log: /var/log/ceph/ceph-client.admin.65997.log 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.88140.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51973.log.gz 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76689.log 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.65997.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65997.log.gz 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82476.log 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.76689.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76689.log.gz 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83525.log 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82476.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82476.log.gz 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87191.log 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.83525.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83525.log.gz 2024-05-09T03:22:09.008 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77035.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.87191.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.87191.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51432.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.77035.log: 92.9% -- replaced with /var/log/ceph/ceph-client.0.88140.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77035.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71208.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82553.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51432.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.71208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88542.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.82553.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82553.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72551.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.88542.log: 91.2% -- replaced with /var/log/ceph/ceph-client.0.71208.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88542.log.gz 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61201.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44094.log 2024-05-09T03:22:09.009 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.61201.log: /var/log/ceph/ceph-client.admin.72551.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72551.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61201.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73641.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44094.log: -- /var/log/ceph/ceph-client.admin.84870.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6%/var/log/ceph/ceph-client.admin.73641.log: -- replaced with /var/log/ceph/ceph-client.admin.44094.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57037.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.84870.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84870.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63753.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57037.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.73641.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57037.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48089.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.63753.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63753.log.gz 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73211.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44136.log 2024-05-09T03:22:09.010 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48089.log: /var/log/ceph/ceph-client.admin.73211.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48089.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73211.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50114.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60036.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50114.log: /var/log/ceph/ceph-client.admin.44136.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50114.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44136.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41883.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.60036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40747.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.41883.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60036.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41883.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52849.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.40747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69252.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40747.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91559.log 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.69252.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52849.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69252.log.gz 2024-05-09T03:22:09.011 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.91559.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91559.log.gz 2024-05-09T03:22:09.012 INFO:teuthology.orchestra.run.smithi046.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-09T03:22:09.037 INFO:teuthology.orchestra.run.smithi088.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-09T03:22:09.098 INFO:teuthology.orchestra.run.smithi088.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-09T03:22:09.188 INFO:teuthology.orchestra.run.smithi088.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-09T03:22:09.205 INFO:teuthology.orchestra.run.smithi088.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-09T03:22:09.207 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-05-09T03:22:09.208 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.805s 2024-05-09T03:22:09.208 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m3.907s 2024-05-09T03:22:09.208 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.165s 2024-05-09T03:22:09.230 INFO:teuthology.orchestra.run.smithi046.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-09T03:22:09.268 INFO:teuthology.orchestra.run.smithi046.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-09T03:22:09.480 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-09T03:22:09.542 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-09T03:22:09.579 INFO:teuthology.orchestra.run.smithi046.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-09T03:22:09.581 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-05-09T03:22:09.581 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m1.200s 2024-05-09T03:22:09.581 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m5.421s 2024-05-09T03:22:09.581 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.755s 2024-05-09T03:22:09.582 INFO:tasks.ceph:Archiving logs... 2024-05-09T03:22:09.582 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/remote/smithi046/log 2024-05-09T03:22:09.583 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T03:22:10.929 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/remote/smithi088/log 2024-05-09T03:22:10.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T03:22:11.466 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T03:22:11.475 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T03:22:11.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T03:22:11.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T03:22:11.569 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T03:22:11.577 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T03:22:11.578 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:22:11.580 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T03:22:11.596 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:22:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-05-09T03:22:11.600 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:22:11.600 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-05-09T03:22:11.600 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv01.front.sepia.ceph.com 3 6 377 4 +2960us[+2965us] +/- 50ms 2024-05-09T03:22:11.600 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -2010us[-2010us] +/- 58ms 2024-05-09T03:22:11.600 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -2395us[-2389us] +/- 86ms 2024-05-09T03:22:11.600 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:22:11.602 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T03:22:11.602 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-05-09T03:22:11.602 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +2898us[+2917us] +/- 50ms 2024-05-09T03:22:11.602 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 -2062us[-2043us] +/- 58ms 2024-05-09T03:22:11.602 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -2435us[-2416us] +/- 86ms 2024-05-09T03:22:11.602 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T03:22:11.602 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T03:22:11.611 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T03:22:11.612 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T03:22:11.620 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T03:22:11.741 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T03:22:11.809 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T03:22:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:22:11.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T03:22:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:22:11.871 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T03:22:11.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T03:22:11.890 INFO:teuthology.task.internal:Duration was 1352.620197 seconds 2024-05-09T03:22:11.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T03:22:11.899 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T03:22:11.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T03:22:11.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T03:22:11.942 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T03:22:11.951 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T03:22:12.209 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T03:22:12.209 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-05-09T03:22:12.209 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T03:22:12.270 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-05-09T03:22:12.271 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T03:22:12.295 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T03:22:12.295 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T03:22:12.313 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T03:22:12.363 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T03:22:12.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T03:22:12.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T03:22:12.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T03:22:12.530 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T03:22:12.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T03:22:12.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T03:22:12.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T03:22:12.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:22:12.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:22:12.620 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-05-09T03:22:12.645 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-05-09T03:22:12.658 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:22:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:22:12.690 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T03:22:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T03:22:12.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T03:22:12.722 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T03:22:12.723 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/remote/smithi046 2024-05-09T03:22:12.723 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T03:22:12.768 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699035/remote/smithi088 2024-05-09T03:22:12.768 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T03:22:12.801 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T03:22:12.802 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T03:22:12.805 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T03:22:12.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T03:22:12.864 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T03:22:12.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T03:22:12.875 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T03:22:12.875 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T03:22:12.878 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T03:22:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 03:22 /home/ubuntu/cephtest 2024-05-09T03:22:12.906 INFO:teuthology.orchestra.run.smithi088.stdout: 265746 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 03:22 /home/ubuntu/cephtest 2024-05-09T03:22:12.907 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T03:22:12.917 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T03:22:12.944 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} duration: 1352.6201968193054 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-05-09T03:22:12.944 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T03:22:13.111 INFO:teuthology.run:pass